POLLS AND THE 2004 PRESIDENTIAL ELECTION by Steve Shaffer

 

(November 4, 2004) So how accurate are statewide polls in predicting presidential election results in the states?

Answer: pretty accurate, under two conditions.

1)      The poll must be conducted as close to the election as possible. Polls conducted within the last five days before the election predicted that the winner’s margin of victory would be 2.1% from the actual results. Polls conducted six days or more before the election were on average 4.4% different from the actual winning margin.

2)      One should use as many recent polls as possible. Just average the results together. This increases sample size, and reduces sample error. When using two or more polls, the average difference between actual and predicted winner margins was only 2.7%. When using only one poll, the difference between actual and predicted margins was 4.2%.

 

What is more important: recency of polls, or using more than one poll?

Answer: use the most recent poll.

When one uses more than one most recent polls, this error margin between actual and predicted margins is on average only 2.2%. When one uses only one poll, and that is conducted more than five days before the election, the error margin is on average 4.9%. The fact that the best case scenario of 2.2% is about the same as the 2.1% error margin when relying on only recent polls and ignoring how many polls are used suggests that recency of polls is more important than the number of polls relied on.

 

How accurate were Professor Shaffer’s predictions?

Answer: Thanks to this great RealClearPolitics website that lists all state polls, Shaffer correctly predicted who would win 47 of the 50 states, for an accuracy level of 94%. This compares to 50% if he had just flipped a coin. On the other hand, there was so much similarity between states won by Bush in both elections, and those won by Gore and Kerry, that if one had predicted that the states would have voted for the same party as they did four years ago, one’s accuracy level would have also been 94%.

 

How accurate were the predictions of the winner’s margin of victory?

Answer: Pretty accurate. The average difference between the actual margin and the predicted margin of victory was only 3.3%. To illustrate what this means, take Kansas for example, where the difference between the predicted and actual margin was 3%. The state poll predicted that Bush would get 60% and Kerry 37% of the vote (3% were undecided); in reality, Bush got 62% to Kerry’s 36% (1% went to Nader and 1% to other minor candidates). Hence, this poll was off by only 2% for Bush and only 1% for Kerry, for an average error rate of only 1.5%. Our calculation of differences in margins actually doubles the sample error rates that are reported in polls. Therefore, our average 3.3% margin errors are actually equivalent to an error rate of only 1.65% for any one candidate. State polls generally interview about 600 people, which yields a sample error of 4.1%. Hence, the election results in the 50 states (the 1.65% figure) were generally well within this sample error (4.1%). That is especially impressive when one considers that in the 50 states, the polls that I relied on were conducted on average a full 11 days before the election. Naturally, public opinion can and does shift over time.

 

Why was Mississippi’s prediction off?

The only recent poll that I was able to use for Mississippi was the American Research Group’s poll, conducted back in mid-September. Hence, it was only one poll, and it was conducted fully six weeks before the election. It correctly predicted that Bush would win the state, but it’s 9% margin of victory was below the actual 20% margin of victory (Bush got 60% to Kerry’s 40%.). I did not rely on our own MSU Mississippi Poll, since it was conducted way back in April. The MSU poll also correctly predicted a Bush victory, but it’s predicted margin was a huge 31%. Interestingly enough, both polls were equally distant from the actual margin, being 11% from the actual margin. The MSU poll predicted that Bush would receive 61% (only 1% off), and that Kerry would receive 30% (10% off). The average error for the two candidates was 5.5%. With only 300 likely voters sampled, the sample error was 5.8%. Therefore, even though public opinion changes over time, and even though the MSU poll was conducted over six months before the election, its results were within sample error of the actual election results. Hence, while the Mississippi prediction for margin was one of the three worst of the 50 states, the poll results were nevertheless impressive.  Hawaii’s poll errors were nearly as great, and its polls were conducted only two weeks before the election. Public opinion can change greatly over time, and apparently the tossup predictions for Hawaii greatly energized the state Democratic Party which historically dominated the state. 

Do poll affect turnout?
Turnout was highest in states that had the closest pre-election polls, where one candidate was expected to win with 53% or less of the vote. In such states, turnout was 62.6% on average. States where a candidate was expected to win by over 57% of the vote had average turnouts of 57.3%. However, the competitive nature of the 2004 presidential campaign nationally and the numerous polls conducted increased turnout in both types of states. Compared to turnout in 2000, turnout increased by an average of 2.7% in the least competitive states. Turnout increased even more in the closely divided states--by 6.2%. Hence, polls may indeed increase turnout in states where the polls show a close race, but they do not appear to depress turnout in states where polls predict a landslide.

 

RAW DATA FILE FOLLOWS:

 

State                Pred.            Pred.            Act.            Act            Last            No.            Pred.            Margin

                        Win            Marg            Win            Mar            Poll            poll            Win.    Wrong

                                                                                                Right    By

Alabama            Bush            18.0            Bush            26            6            1            yes            8.0

Alaska              Bush            27.0            Bush            27            52            1            yes            .0

Arizona            Bush            10.0            Bush            11            5            2            yes            1.0

Arkansas            Bush            3.0            Bush            9            1            2            yes            6.0

California            Kerry            9.0            Kerry            11            5            2            yes            2.0

Colorado            Bush            3.0            Bush            7            2            2            yes            4.0

Connecticut            Kerry            8.0            Kerry            10            5            2            yes            2.0

Delaware            Kerry            7.0            Kerry            7            38            1            yes            .0

Florida             Bush            .5            Bush            5            2            6            yes            4.5

Georgia            Bush            12.0            Bush            18            3            3            yes            6.0

Hawaii              Bush            1.0            Kerry            9            14            2            no            10.0

Idaho               Bush            29.0            Bush            38            53            1            yes            9.0

Illinois               Kerry            13.0            Kerry            10            5            2            yes            3.0

Indiana             Bush            19.0            Bush            21            5            1            yes            2.0

Iowa                Kerry            1.0            Bush            1            2            3            no            2.0

Kansas             Bush            23.0            Bush            26            7            1            yes            3.0

Kentucky            Bush            19.0            Bush            20            9            2            yes            1.0

Louisiana            Bush            10.0            Bush            15            16            2            yes            5.0

Maine               Kerry            8.0            Kerry            8            4            1            yes            .0

Maryland            Kerry            11.0            Kerry            13            5            1            yes            2.0

Massachusetts            Kerry            14.0            Kerry            25            33            1            yes            11.0

Michigan            Kerry            4.0            Kerry            3            3            2            yes            1.0

Minnesota            Kerry            3.0            Kerry            3            4            7            yes            .0

Mississippi            Bush            9.0            Bush            20            47            1            yes            11.0

Missouri            Bush            5.0            Bush            7            4            2            yes            2.0

Montana            Bush            21.0            Bush            21            14            1            yes            .0

Nebraska            Bush            29.0            Bush            35            15            1            yes            6.0

Nevada            Bush            6.0            Bush            2            3            4            yes            4.0

New Hamp.            Kerry            1.0            Kerry            1            4            4            yes            .0

New Jersey            Kerry            3.0            Kerry            7            6            5            yes            4.0

New Mexico            Kerry            .5            Bush            1            4            3            no            1.5

New York            Kerry            17.0            Kerry            18            6            2            yes            1.0

North Car.            Bush            8.0            Bush            12            3            1            yes            4.0

North Dakota            Bush            20.0            Bush            27            13            1            yes            7.0

Ohio                 Bush            2.0            Bush            2            4            9            yes            .0

Oklahoma            Bush            30.0            Bush            32            4            1            yes            2.0

Oregon             Kerry            5.0            Kerry            4            6            4            yes            1.0

Pennsylvania            Kerry            2.0            Kerry            2            2            2            yes            .0

Rhode Island            Kerry            15.0            Kerry            21            7            1            yes            6.0

South Car.            Bush            16.0            Bush            17            10            1            yes            1.0

South Dakota            Bush            15.0            Bush            22            17            2            yes            7.0

Tennessee            Bush            16.0            Bush            14            12            3            yes            2.0

Texas               Bush            22.0            Bush            23            6            1            yes            1.0

Utah                 Bush            45.0            Bush            44            9            2            yes            1.0

Vermont            Kerry            13.0            Kerry            20            22            1            yes            7.0

Virginia Bush            6.0            Bush            9            8            3            yes            3.0

Washington            Kerry            5.0            Kerry            7            4            3            yes            2.0

West Virginia            Bush            8.0            Bush            13            6            2            yes            5.0

Wisconsin            Kerry            .5            Kerry            1            3            3            yes            .5

Wyoming            Bush            36.0            Bush            40            52            1            yes            4.0

 

Note: Pred Win means who I predicted to win the state, based on the comprehensive list of state polls provided by: http://www.realclearpolitics.com/index.html. I relied on the most recent polls in each state (just before election day), and attempted to average as many recent polls as possible.

Pred Marg is the predicted margin of victory of these polls, computed as the winner’s percentage of popular vote minus the loser’s vote percentage.

Act Win and Act Mar. are the actual winners of each state, and the margins that they won by, based on preliminary results provided in USA Today, November 4, 2004, p. 4A.

Last Poll is the number of days between when the polls I relied on were conducted, and election day. The midpoint of the days when each poll was conducted was used, and these midpoints for all of the polls used were averaged.

No. Poll is the number of polls I relied on in each state.

Pred. Win. Right is whether the polls correctly predicted who would win the state.

Margin Wrong By is the difference between the actual margin and the predicted margins.