Last week, Nate Silver wrote a lengthy critique of Sam Wang’s midterm election forecast for Political Wire.
Wang responds below.
by Sam Wang
Co-founder, Princeton Election Consortium.
I began what is now the Princeton Election Consortium (PEC) by analyzing the 2004 Presidential race, which came down to Ohio, Pennsylvania, and Florida. The 2014 Senate race is just as suspenseful, and comes down to a handful of states that includes Alaska, Arkansas, Colorado, Iowa, and Louisiana. So before I start, let’s keep in mind that the true stars of this story are not statistical analysts, but the candidates and the voters – and control of the United States Senate.
Recently a well-known commentator, Nate Silver, has taken to criticizing the Princeton Election Consortium approach. However, he has made a number of factual and conceptual errors. If an experienced analyst like him could make those misreadings, so could many people. PEC has a track record equal to (and for the Senate, superior to) commercial outlets such as FiveThirtyEight. In addition to correcting the errors, I will explain how we do it. By “we” I include my readers: PEC’s code has always been free to all, and my commenters compose a small army of experts. In fact, it was their vigorous factchecking of recently published criticism of PEC that encouraged me to respond.
1. The PEC prediction sits at the end of a very narrow range.
The truth is this: neither Nate Silver nor I know which way the Senate will go. Indeed, in the current situation, a few thousand students in Ames, Iowa and Boulder, Colorado could help determine control of the United States Senate by getting out the vote in their states. As a fan of elections and democracy, I find moments like this to be amazing.
In all the major models I see plenty of uncertainty. The Princeton Election Consortium’s long-term estimate has fluctuated between 50% and 70% probability for Democrats and Independents holding 50 or more seats (and never 90% as implied by Silver). This is not much better than a coin toss. This is confirmed if one goes to The Upshot’s “make your own forecast” interactive tool and selects “polls only” and unchecks “house effects” to get a result of 51% for Democratic control.
Accurate predictions in this range can go to the perceived underdog one-third to one-half of the time. Many people make the conceptual error of mentally rounding up to 100% any probability that is even a hair over 50%. For this reason, probability may not be a good way to quantify the current situation. I urge all readers to look beyond probabilities to see an extremely close Senate campaign, in which each side could well end up with 49, 50, or 51 seats. So the question “is the probability above or below 50%?” is excessively simpleminded.
At PEC we prefer to use measures such as the Meta-Margin, defined by how much opinion, as measured by polls, would have to swing to create a perfect tie. It can also be used to estimate how much collective pollster error would be required in order to change the estimate of likely Senate control. With a current Meta-Margin of R+0.9%, this is anybody’s game.
2. How does the PEC approach eliminate pollster bias?
Of perhaps greatest interest is the fact that on Election Eve in 2012, PEC called every close Senate race correctly – 10 out of 10. Silver is protesting against a model that has consistently matched or outperformed his own calls since he came onto the scene (see 2008, 2010, and 2012). PEC does this in a different way than the FiveThirtyEight method of detailed analysis of pollster “house effects” and “fundamentals.” The FiveThirtyEight approach can work in the right hands, but is laborious and mostly converges with a simpler polls-only approach in the closing stages of a campaign.
A key to PEC’s success is our use of medians as a statistical tool. Other sites use averaging, which tends to overly weight outlier polls. PEC’s use of medians reduces the influence of outliers without having to dissect individual pollster performance. Only one recent poll from each pollster is allowed into the daily “snapshot” of current conditions. More generally, the PEC philosophy is to provide a prediction model based entirely on the history of all available likely-voter polls as found in the HuffPollster database. (Oddly, Silver claims that we use a lot of registered-voter polls, which we don’t.)
3. What does day-to-day variation in the PEC daily snapshot teach us?
The PEC daily snapshot is designed to move from day to day. This movability is a feature, not a bug. It provides a sensitive, near-real-time way of identifying true turning points in a campaign – and to rebut false claims of significance touted in the news media. In fact, the snapshot is quite similar to the “nowcast” that Silver himself issued in 2012. Several weeks ago, we noted a sharp shift in the Senate race triggered by changes in Iowa and Colorado. If this shift holds through November, understanding it will be critical to understanding the 2014 campaign for control of the Senate.
Speaking of control of the Senate, PEC’s snapshot also allows us to measure how powerful individual voters are in determining that outcome. For example, we calculate that a vote cast in Iowa has about 100 times as much influence over the probability of Senate control as a vote cast in neighboring Minnesota – and thousands of times the power of my vote in New Jersey.
4. How can polls going back to June help make a long-term prediction?
At PEC, we invite criticism and open discussion of the choice of parameters used in the model. My readers and I have already been going over this topic for weeks.
Whether one starts looking at polls in June or in September, the long-term prediction result is the same, a slight advantage for Senate Democrats. Using a period that starts in June, when polls became more frequent and focused on likely voters, expands the range of outcomes, since June was a particularly good month for Republicans, poll-wise.
Silver claims: “If you’d applied Wang’s technique in the past, you might still have had Republicans favored to win the Senate at this point in 2012 — even though the race had clearly broken toward Democrats by that time.”
Actually, using the “June-to-now” method, at this point in 2012 the predicted probability of Democratic Senate control would have been about 70%. (A time-series of our 2012 Senate tracker is here, with previous published analysis here, here, and here.)
Like FiveThirtyEight and The Upshot, PEC’s prediction increasingly emphasizes recent polls as we get closer to the election. Now, in the home stretch, our November prediction gives less weight to that long-term prediction, exactly as fundamentals-based models do, and more weight to current polling, which as of today favors Republicans. And on Election Eve, our final prediction will be based on current polls alone, just as in past years.
5. How do PEC’s predictions take into account the difference between polls and final election results.
Silver claims that PEC’s predictions are overconfident because they fail to account for differences between polls and final election results. This is at best misleading. The PEC snapshot does not measure this difference because the election is not today. It is designed to be a sensitive measure of the current opinion reflected in polls, not a final Election Day prediction. Agreed, the difference between Election Eve polls and final results is a highly important quantity. PEC uses historical results to calculate how this difference can vary from year to year, and factors the variation into the PEC November prediction. For example, an across-the-board pollster error of as little as 2% would change the current front-runner in Iowa and Colorado. These are prime races for readers to focus their attention and energy.
I do agree that in 2010, PEC did not fully convey the uncertainty in our predictions, mainly because of underestimation of the polls-to-election differential. His complaint is not about the direction of calls, but the exact statistical confidence with which we made the calls. This particular problem was addressed starting in 2012, in part thanks to the efforts of my readers. Over at PEC we have been doing what we always do: openly discuss adjustments that might be made to the model to improve its performance.
6. Does PEC assume that Orman (I) would caucus exclusively with the Democrats?
There is no way to statistically model Orman’s choice based on polls, so PEC simply estimates the likelihood that a certain number of seats will be held by Democrats and Independents combined. The scenario in which Orman’s decision could tip the balance of control of the Senate is broken out in the daily snapshot, so that readers can draw their own conclusions.
Conclusion
Silver compares where we are in the race to the third quarter of a football game. I’d argue that we are still in the pre-game period. The game – or more accurately, an election that will determine the direction of our national government – is still four weeks away. He and I aren’t the players; we’re in the announcer’s booth. Meanwhile, on the field are the candidates, the strategists, and the men and women who are giving their time and money in key races. They’re the ones who are making the 2014 Senate campaign exciting.
Save to Favorites