POLLS AND THE 2004 PRESIDENTIAL ELECTION by Steve Shaffer
(November 4, 2004) So how accurate are statewide polls in predicting presidential election results in the states?
Answer: pretty accurate, under two conditions.
1) The poll must be conducted as close to the election as possible. Polls conducted within the last five days before the election predicted that the winner’s margin of victory would be 2.1% from the actual results. Polls conducted six days or more before the election were on average 4.4% different from the actual winning margin.
2) One should use as many recent polls as possible. Just average the results together. This increases sample size, and reduces sample error. When using two or more polls, the average difference between actual and predicted winner margins was only 2.7%. When using only one poll, the difference between actual and predicted margins was 4.2%.
What is more important: recency of polls, or using more than one poll?
Answer: use the most recent poll.
When one uses more than one most recent polls, this error margin between actual and predicted margins is on average only 2.2%. When one uses only one poll, and that is conducted more than five days before the election, the error margin is on average 4.9%. The fact that the best case scenario of 2.2% is about the same as the 2.1% error margin when relying on only recent polls and ignoring how many polls are used suggests that recency of polls is more important than the number of polls relied on.
How accurate were Professor Shaffer’s predictions?
Answer: Thanks to this great RealClearPolitics website that lists all state polls, Shaffer correctly predicted who would win 47 of the 50 states, for an accuracy level of 94%. This compares to 50% if he had just flipped a coin. On the other hand, there was so much similarity between states won by Bush in both elections, and those won by Gore and Kerry, that if one had predicted that the states would have voted for the same party as they did four years ago, one’s accuracy level would have also been 94%.
How accurate were the predictions of the winner’s margin of victory?
Answer: Pretty accurate. The average difference between the actual margin and the predicted margin of victory was only 3.3%. To illustrate what this means, take Kansas for example, where the difference between the predicted and actual margin was 3%. The state poll predicted that Bush would get 60% and Kerry 37% of the vote (3% were undecided); in reality, Bush got 62% to Kerry’s 36% (1% went to Nader and 1% to other minor candidates). Hence, this poll was off by only 2% for Bush and only 1% for Kerry, for an average error rate of only 1.5%. Our calculation of differences in margins actually doubles the sample error rates that are reported in polls. Therefore, our average 3.3% margin errors are actually equivalent to an error rate of only 1.65% for any one candidate. State polls generally interview about 600 people, which yields a sample error of 4.1%. Hence, the election results in the 50 states (the 1.65% figure) were generally well within this sample error (4.1%). That is especially impressive when one considers that in the 50 states, the polls that I relied on were conducted on average a full 11 days before the election. Naturally, public opinion can and does shift over time.
Why was Mississippi’s prediction off?
The only recent poll that I was able to use for Mississippi was the American Research Group’s poll, conducted back in mid-September. Hence, it was only one poll, and it was conducted fully six weeks before the election. It correctly predicted that Bush would win the state, but it’s 9% margin of victory was below the actual 20% margin of victory (Bush got 60% to Kerry’s 40%.). I did not rely on our own MSU Mississippi Poll, since it was conducted way back in April. The MSU poll also correctly predicted a Bush victory, but it’s predicted margin was a huge 31%. Interestingly enough, both polls were equally distant from the actual margin, being 11% from the actual margin. The MSU poll predicted that Bush would receive 61% (only 1% off), and that Kerry would receive 30% (10% off). The average error for the two candidates was 5.5%. With only 300 likely voters sampled, the sample error was 5.8%. Therefore, even though public opinion changes over time, and even though the MSU poll was conducted over six months before the election, its results were within sample error of the actual election results. Hence, while the Mississippi prediction for margin was one of the three worst of the 50 states, the poll results were nevertheless impressive. Hawaii’s poll errors were nearly as great, and its polls were conducted only two weeks before the election. Public opinion can change greatly over time, and apparently the tossup predictions for Hawaii greatly energized the state Democratic Party which historically dominated the state.
Do poll affect turnout?
Turnout was highest in states that had the closest pre-election polls,
where one candidate was expected to win with 53% or less of the vote. In
such states, turnout was 62.6% on average. States where a candidate was
expected to win by over 57% of the vote had average turnouts of 57.3%.
However, the competitive nature of the 2004 presidential campaign
nationally and the numerous polls conducted increased turnout in both
types of states. Compared to turnout in 2000, turnout increased by an
average of 2.7% in the least competitive states. Turnout increased even
more in the closely divided states--by 6.2%. Hence, polls may indeed
increase turnout in states where the polls show a close race, but they do
not appear to depress turnout in states where polls predict a landslide.
RAW DATA FILE FOLLOWS:
State Pred. Pred. Act. Act Last No. Pred. Margin
Win Marg Win Mar Poll poll Win. Wrong
Right By
Alabama Bush 18.0 Bush 26 6 1 yes 8.0
Alaska Bush 27.0 Bush 27 52 1 yes .0
Arizona Bush 10.0 Bush 11 5 2 yes 1.0
Arkansas Bush 3.0 Bush 9 1 2 yes 6.0
California Kerry 9.0 Kerry 11 5 2 yes 2.0
Colorado Bush 3.0 Bush 7 2 2 yes 4.0
Connecticut Kerry 8.0 Kerry 10 5 2 yes 2.0
Delaware Kerry 7.0 Kerry 7 38 1 yes .0
Florida Bush .5 Bush 5 2 6 yes 4.5
Georgia Bush 12.0 Bush 18 3 3 yes 6.0
Hawaii Bush 1.0 Kerry 9 14 2 no 10.0
Idaho Bush 29.0 Bush 38 53 1 yes 9.0
Illinois Kerry 13.0 Kerry 10 5 2 yes 3.0
Indiana Bush 19.0 Bush 21 5 1 yes 2.0
Iowa Kerry 1.0 Bush 1 2 3 no 2.0
Kansas Bush 23.0 Bush 26 7 1 yes 3.0
Kentucky Bush 19.0 Bush 20 9 2 yes 1.0
Louisiana Bush 10.0 Bush 15 16 2 yes 5.0
Maine Kerry 8.0 Kerry 8 4 1 yes .0
Maryland Kerry 11.0 Kerry 13 5 1 yes 2.0
Massachusetts Kerry 14.0 Kerry 25 33 1 yes 11.0
Michigan Kerry 4.0 Kerry 3 3 2 yes 1.0
Minnesota Kerry 3.0 Kerry 3 4 7 yes .0
Mississippi Bush 9.0 Bush 20 47 1 yes 11.0
Missouri Bush 5.0 Bush 7 4 2 yes 2.0
Montana Bush 21.0 Bush 21 14 1 yes .0
Nebraska Bush 29.0 Bush 35 15 1 yes 6.0
Nevada Bush 6.0 Bush 2 3 4 yes 4.0
New Hamp. Kerry 1.0 Kerry 1 4 4 yes .0
New Jersey Kerry 3.0 Kerry 7 6 5 yes 4.0
New Mexico Kerry .5 Bush 1 4 3 no 1.5
New York Kerry 17.0 Kerry 18 6 2 yes 1.0
North Car. Bush 8.0 Bush 12 3 1 yes 4.0
North Dakota Bush 20.0 Bush 27 13 1 yes 7.0
Ohio Bush 2.0 Bush 2 4 9 yes .0
Oklahoma Bush 30.0 Bush 32 4 1 yes 2.0
Oregon Kerry 5.0 Kerry 4 6 4 yes 1.0
Pennsylvania Kerry 2.0 Kerry 2 2 2 yes .0
Rhode Island Kerry 15.0 Kerry 21 7 1 yes 6.0
South Car. Bush 16.0 Bush 17 10 1 yes 1.0
South Dakota Bush 15.0 Bush 22 17 2 yes 7.0
Tennessee Bush 16.0 Bush 14 12 3 yes 2.0
Texas Bush 22.0 Bush 23 6 1 yes 1.0
Utah Bush 45.0 Bush 44 9 2 yes 1.0
Vermont Kerry 13.0 Kerry 20 22 1 yes 7.0
Virginia Bush 6.0 Bush 9 8 3 yes 3.0
Washington Kerry 5.0 Kerry 7 4 3 yes 2.0
West Virginia Bush 8.0 Bush 13 6 2 yes 5.0
Wisconsin Kerry .5 Kerry 1 3 3 yes .5
Wyoming Bush 36.0 Bush 40 52 1 yes 4.0
Note: Pred Win means who I predicted to win the state, based on the comprehensive list of state polls provided by: http://www.realclearpolitics.com/index.html. I relied on the most recent polls in each state (just before election day), and attempted to average as many recent polls as possible.
Pred Marg is the predicted margin of victory of these polls, computed as the winner’s percentage of popular vote minus the loser’s vote percentage.
Act Win and Act Mar. are the actual winners of each state, and the margins that they won by, based on preliminary results provided in USA Today, November 4, 2004, p. 4A.
Last Poll is the number of days between when the polls I relied on were conducted, and election day. The midpoint of the days when each poll was conducted was used, and these midpoints for all of the polls used were averaged.
No. Poll is the number of polls I relied on in each state.
Pred. Win. Right is whether the polls correctly predicted who would win the state.
Margin Wrong By is the difference between the actual
margin and the predicted margins.