For those interested in what has become an epic battle of election forecasters, Sam Wang responds to Nate Silver’s extensive criticism of his midterm election forecasting model.
Why is Alaska Polling So Bad?
Sam Wang notes that “in any given year, approximately 10% of Alaska’s population has just arrived or is about to move away.”
“This is reminiscent of Nevada, which has very similar statistics. Nevada is also the site of a spectacularly wrong polling error. In fall of 2010, surveys unanimously showed Republican Sharron Angle favored to unseat Democratic Senator (and Majority Leader) Harry Reid, by a median of four percentage points. Every data-based prognosticator, including me, expected Angle to win. Yet Reid won the election by 5 points, outperforming polls by 9 points. I regard this as the worst miss of Senate polling in the last three election cycles.”
“Based on the extreme mobility of Alaska residents, I suggest a problem of potentially far larger impact. Like Nevada, telephone-based surveys in both states simply fail to capture a sample that is fully representative of voters. In other words, I suggest that ideologically diverse states with lots of recent arrivals are subject to inaccuracy.”
Not All Forecasting Models are Equal
Nate Silver: “I don’t like to call out other forecasters by name unless I have something positive to say about them — and we think most of the other models out there are pretty great. But one is in so much perceived disagreement with FiveThirtyEight’s that it requires some attention. That’s the model put together by Sam Wang, an associate professor of molecular biology at Princeton.”
“That model is wrong — not necessarily because it shows Democrats ahead (ours barely shows any Republican advantage), but because it substantially underestimates the uncertainty associated with polling averages and thereby overestimates the win probabilities for candidates with small leads in the polls.”
Kevin Drum: “If Wang’s prediction of this year’s Senate race turns out to be more accurate than Silver’s, I almost hate to think what might happen. Silver’s head is going to explode or something. In any case, this is far more fun than you normally get from a couple of geeky poll aggregators.”
Is the Polling Industry in Crisis?
Nate Silver: “There is no shortage of reasons to worry about the state of the polling industry. Response rates to political polls are dismal. Even polls that make every effort to contact a representative sample of voters now get no more than 10 percent to complete their surveys — down from about 35 percent in the 1990s.”
“And there are fewer high-quality polls than there used to be. The cost to commission one can run well into five figures, and it has increased as response rates have declined.1 Under budgetary pressure, many news organizations have understandably preferred to trim their polling budgets rather than lay off newsroom staff.”
On Pollsters
Washington Post: “A pollster is one of those jobs — like a football lineman or an oil-tanker captain — that normal people tend to notice only when one of these specialized professionals messes up.”
Are Bad Pollsters Copying Good Pollsters?
Harry Enten: “The political polling industry is a mess. Fewer and fewer people are willing to respond to telephone surveys, particularly automated ones, and the costs of live interviews are climbing ever higher. Meanwhile, polls have gained prominence in the political media… and the Internet seems willing to give a home to almost any survey.”
“Demand is up, and quality supply is down. The result: pollsters that use nontraditional methodologies such as online and automated surveys are getting more press than ever, and they get included in the models of the main polling aggregators…”
“The problem is that many of these nontraditional polls may be cheating, adjusting their results to resemble higher-quality polls. We can see this by looking at polling from the final three weeks of Senate campaigns since 2006: in races without traditional, live-interview surveys (what we’ll call gold-standard polling), nontraditional polls have had significantly higher errors than they’ve had in races with at least one gold-standard poll.”
So Many Bad Polls
First Read: “One more point on the Georgia contest: Yes, Perdue’s win contradicted the existing polls, but they weren’t good polls — which is why, for instance, you never saw us make note of them. We do try and only highlight pollsters who have good methodology and decent track records. In Georgia during this runoff campaign, that didn’t exist. Folks, we’re living in a political age where so much of the data is coming from surveys with either questionable methodology (like not reaching those with cell phones) or with partisan manipulation. And so it’s a reminder to take them with a grain of salt, especially in a low-turnout runoff.”
“And it also means: Be very careful of these aggregation polling sites. They don’t always make the data better… there are actually polling firms out there who seem to have been created for the sole goal of influencing or balancing out these aggregation sites. To the folks who run these aggregation sites, realize that when you mix crap with sirloin, it makes the sirloin taste like &$#% too. ”
Polling Isn’t Broken
Lynn Vavreck: “The science of polling is sound, but if you ask the wrong group of people your poll questions, you can get the wrong answers. Think of it this way: An arrow shot by an expert marksman has some chance of hitting the target depending on the wind, the distance and any number of other things, but if the marksman aims at the wrong target, those other things have nothing to do with why the arrow misses. Saying that 1 out of 20 polls is “wrong,” and therefore polls cannot predict elections conflates the chance that comes from drawing a sample of voters with what happened in Mr. Cantor’s poll — drawing the sample from the wrong set of people.”
McLaughlin Explains How He Got It So Wrong
GOP pollster John McLaughlin put out a memo attempting to explain how he showed Rep. Eric Cantor (R-VA) with a huge lead in Virginia’s 7th congressional district over the ultimate GOP primary winner, David Brat (R).
In short, McLaughlin says many voter who don’t normally vote in GOP primaries turned out this year.
GOP Pollsters Under Fire
Kristen Soltis Anderson: “The last few years have regrettably made the phrase ‘Republican pollster’ less a job title than a punch line. From the 2012 election, where many in the GOP were stunned by the Obama campaign’s victory, to the 2013 closer-than-expected Virginia gubernatorial race, all the way to the present, Republican polling as of late has been a much-maligned sector of the campaign-industrial complex, and not without reason.”
Republicans Warned Not to Use Pollster
“National Republicans are warning candidates to stay away from House Majority Leader Eric Cantor’s (R-VA) pollster, who predicted just weeks before Cantor’s loss that he was up by a huge margin,” The Hill reports.
“Veteran GOP pollster John McLaughlin has a recent history of missing the mark by a wide margin in his top races. Now, it seems some Republicans have had enough. Sources close to the National Republican Congressional Committee hear that the House GOP’s campaign arm will encourage their candidates to reassess whether they should be using McLaughlin in several top races.”
Polling Averages Prove Accurate
Wall Street Journal: “For all the focus on Republican infighting this primary season, one quiet winner has emerged from these intraparty contests: public-opinion polls.”
“Individual survey results have been all over the map in primary after primary, but the averages compiled by Real Clear Politics have predicted the Republican winner in every race so far, save Mississippi, where the aggregate results on Primary Day showed a tie. The contest is now headed to a June 24 runoff because neither candidate eclipsed 50%.”
We Don’t Know Who Likely Voters Are Yet
First Read criticizes news organizations and pollsters that use likely voter models “to describe an election environment before Memorial Day when many Americans haven’t even begun to tune in to the general election.”
“A tip: There is no such thing as a likely voter six months before a general election, and if you think there is one, you’re likely cooking the books. Our rule is to always begin introducing likely voter models after Labor Day.”
New GOP Polling Firm to Launch
“A small group of top Washington Republicans is teaming up to launch a new polling firm, Vox Populi Polling, that will churn out volumes of survey data ahead of the 2014 midterm elections,” Politico reports.
“The firm plans to produce both public and private polling, tracking high-priority 2014 races and issue debates. It will employ a combination of automated polling and live-caller methodology to reach cell phone users, according to strategists involved in the effort.”
One of the partners: Mary Cheney, daughter of former Vice President Dick Cheney.
Harry Reid Rips Nate Silver
Senate Majority Leader Harry Reid (D-NV) blasted Nate Silver’s political forecasts in an interview with the Washington Post.
Said Reid: “He gave me a 16 percent chance of being reelected, he gave Heidi Heitkamp an 8 percent chance of being reelected, he gave Jon Tester a 34 percent chance of being reelected. So all polls are about like Nate Silver’s predictions: good sometimes, bad most of the time.”
Too Early for 2016 Polls
Brendan Nyhan: “We’re still almost three years away from November 2016, but political journalists seem to want to fast-forward past the ongoing Washington stalemate to the next presidential election. How else can we explain the recent flurry of coverage for trial heat polls, which pit possible presidential contenders against each other in hypothetical general election matchups?”
“There’s just one problem: These polls–which exist mainly to generate press coverage for pollsters and fill space in the media–have zero predictive power at this point in the election cycle, don’t tell us anything we can’t learn from other metrics, and distract attention from the real action at this stage of the campaign.”
Pollster Held Poll on Colorado Recall
Public Policy Polling: “We did a poll last weekend in Colorado Senate District 3 and found that voters intended to recall Angela Giron by a 12 point margin, 54/42. In a district that Barack Obama won by almost 20 points I figured there was no way that could be right and made a rare decision not to release the poll. It turns out we should have had more faith in our numbers becaue she was indeed recalled by 12 points.”
Poll Finds Most Think Polls are Biased
A new Kantar poll finds the vast majority of Americans think polls are biased.
Key findings: 75% said they believe the polls are biased in some way, though they couldn’t agree in which direction, while 17% said they thought polls had a liberal bias and 4% said they were slanted toward conservatives.

