Political Wire

  • Front Page
  • Sign in
  • Members
    • Monthly – $6 a month
    • Annual – $60 a year
    • Member Benefits
    • Account
    • Exclusive Analysis
    • Member Library
    • My Favorites
  • Resources
    • Political Calendar
    • Political Job Hunt
    • Political Dictionary
    • Electoral Vote Map
  • Advertise
  • Newsletter
  • Contact Us
You are here: Home / Polling / Nate Silver Rebuts Sam Wang

Nate Silver Rebuts Sam Wang

October 2, 2014 at 8:56 pm EDT By Taegan Goddard Leave a Comment

Exclusive to Political Wire, Nate Silver defends his critique of Sam Wang’s election forecast. It’s definitely worth reading.



by Nate Silver
Editor in Chief, FiveThirtyEight

The most striking feature about the Senate forecasts right now is how much of a consensus there is between them. As of Thursday morning:

  • FiveThirtyEight’s forecast had Republicans with a 59 percent chance of winning.
  • Huffington Post put them at 58 percent.
  • NYT was at 63 percent.
  • Daily Kos was at 66 percent.
  • Charlie Cook was at 60 percent.
  • Predictwise was at 64 percent.
  • Betting markets put them at about 67 percent.
  • And Washington Post had them at 78 percent.

The exception is Sam Wang’s model, which is alone in having Democrats favored (on Thursday AM it had Republicans with a 38 percent chance of winning the majority). So this is really about “Sam Wang vs. the World” and not “Sam Wang vs. FiveThirtyEight”.

Why is Wang’s model so different than the others?

Wang has often claimed that the difference arises because his model uses polls only — whereas others models use polls along with other factors (“fundamentals”) to make their forecasts. But this explanation doesn’t hold up. Among the other models that I mentioned, Huffington Post’s and Daily Kos also use polls only. They show very different results from Wang’s. So do the polls-only Real Clear Politics averages.

So here’s my attempt at explanation. The short version: Wang’s forecast has Democrats ahead because it goes all the way back to June in looking at polls. The long-ish version is below.

Wang’s site offers both a “snapshot” — what he says would happen in an election held today — and a “forecast” — how he assesses the Election Day odds. The snapshot has fluctuated wildly: over the past two weeks, it’s shown everything from a 93 percent chance of a Democratic win to a 76 percent chance of a Republican one. The swings have been even wilder in individual states. His snapshot currently shows the Republican Dan Sullivan with a 94 percent chance of winning the Senate seat in Alaska. Only a week ago, it had Sullivan with just a 1 percent chance.

I’ve critiqued Wang’s snapshot probabilities in detail in the past. They’ve shown super overconfident results — for instance, having Sharron Angle with a 99.997 percent chance of winning the Senate seat in Nevada in 2010. This is because of a central methodological flaw: Wang estimates the uncertainty in his snapshots based on how much the polls differ from one another — and not how much they’ve differed from actual election results.

In reality, polls often err in the same direction. So even if you have a number of polls that show a similar results (as they did in Nevada in 2010) there is still the potential for a big miss. This is perhaps the single most important parameter of an election model, in fact: how much do polls differ from actual results? FWIW, in elections since 1998, the standard error of Election Day polling averages has been about plus or minus 5 percentage points. Polling averages are powerful instruments, but they’re a long way from perfect.

What about Wang’s “forecast” instead of his “snapshot”? Might his forecast be OK even if his snapshot has problems?

Here’s the problem: Wang’s forecast is based on his snapshots. And it uses them in a strange way. In particular, his forecasts are based on an average of his past snapshots since June. Since Wang’s is a “polls only” model, this is equivalent to looking at polls back to June.

Why go all the way back to June? Wang seems to have chosen the date arbitrarily. So how far back should you go? The way to answer this is by looking at the historical data. And the empirical answer is that, while you could debate about whether to go two weeks back or four weeks back or whatever else, you certainly ought not be looking at June polls when trying to forecast a race in October.

If you’d applied Wang’s technique in the past, you might still have had Republicans favored to win the Senate at this point in 2012 — even though the race had clearly broken toward Democrats by that time. Or to use a presidential example, you might still have had Michael Dukakis in a highly competitive race with Bush, even though Bush was well ahead by October 1988.

In football terms, it’s like asserting the Philadelphia Eagles are still favored even after the Dallas Cowboys score a touchdown to go ahead 21-17 because the Eagles had been ahead on average earlier in the game.

Importantly also, a lot of the polling before Labor Day was conducted among registered voters — and those polls tend to exaggerate how well Democrats will do. If you’re only using recent polls, this assumption doesn’t matter so much anymore since almost all polls report likely voter results now. But if you’re going all the way back to June, it’s a problem.

Furthermore, Wang’s model assumes the independent Greg Orman is 100 percent sure to caucus with Democrats if he wins in Kansas. That’s a strange assumption for a model that claims to make simple, neutral assumptions. The simplest assumption would be to take Orman’s word at face value, which is that he doesn’t know who he’ll caucus with in the event his choice would determine the majority (i.e. to treat his choice as 50/50).

All of these things tie together: minor-seeming assumptions matter more in Wang’s model because of its tendency to underestimate the uncertainty in the outlook. Perhaps on some day a few weeks ago or a few months ago, Democrats were slightly favored in a “polls only” model if you were making certain assumptions, like using registered voter polls and treating Orman as certain to caucus with them. The race has always been pretty close. The problem is that Wang’s snapshots will take a tiny advantage and turn it into being a 90 percent favorite to win the election. And those flawed snapshots from months ago are still getting averaged into his forecasts today.

FavoriteLoadingSave to Favorites
TwitterFacebookLinkedInEmail

Filed Under: Polling

Recent Posts

Europe Shuts Schools Again

January 16, 2021 at 1:50 pm EST

“As U.S. authorities debate whether to keep schools open, a consensus is emerging in Europe that children are a considerable factor in the spread of Covid-19—and more countries are shutting…

Trump May Leave a Hole In History

January 16, 2021 at 12:28 pm EST

“The public won’t see President Trump’s White House records for years, but there’s growing concern the collection won’t be complete, leaving a hole in the history of one of America’s…

Donor Backlash Fuels GOP Alarm About Senate

January 16, 2021 at 11:15 am EST

“Republicans are worried that a corporate backlash stirred by the deadly Capitol insurrection could crimp a vital stream of campaign cash, complicating the party’s prospects of retaking the Senate in…

Trump Finishes with Worst First Term Approval Ever

January 16, 2021 at 11:12 am EST

Harry Enten: “A big question during the Trump presidency was whether anything could actually move the public opinion needle. It turns out that the insurrection at the US Capitol last…

Ivanka’s Political Future Comes Into Sharper Focus

January 16, 2021 at 10:44 am ESTJanuary 16, 2021 at 10:45 am EST

Politico: “The senior White House adviser is set to decamp to Florida after her father’s presidency comes to a close. And though talk of her launching a primary challenge to…

Lawmakers Not Told of Police Request for Backup

January 16, 2021 at 10:38 am EST

New York Times: “The Capitol Police asked the sergeants-at-arms to request that the National Guard be placed on standby. But the sergeants-at-arms, Michael C. Stenger of the Senate and Paul…

Follow the Selfies

January 16, 2021 at 10:21 am ESTJanuary 16, 2021 at 10:22 am EST

Bloomberg reports that “selfie-snapping rioters” at the U.S. Capitol last week left the FBI a trail of over 140,000 images. Save to FavoritesTwitterFacebookLinkedInEmail

Prosecutors Expand Probe Into Trump Finances

January 16, 2021 at 10:18 am ESTJanuary 16, 2021 at 10:23 am EST

“The Manhattan district attorney’s office has expanded its criminal investigation into the Trump Organization’s finances to include the family’s compound in Westchester County,” CNN reports. “Prosecutors’ interest in the 212-acre…

The 15 Most Notable Lies of Trump’s Presidency

January 16, 2021 at 9:47 am EST

Daniel Dale: “Trying to pick the most notable lies from Donald Trump’s presidency is like trying to pick the most notable pieces of junk from the town dump.” “There’s just…

Manchin Open to Expelling Hawley and Cruz

January 16, 2021 at 9:43 am ESTJanuary 16, 2021 at 10:41 am EST

Sen. Joe Manchin asked by PBS about using the 14th Amendment to remove sent. Josh Hawley and Ted Cruz from Senate: “That should be a consideration.” Politico: Senate Democrats eye…

We’re Just Finding Out How Bad the Riot Really Was

January 16, 2021 at 9:34 am EST

David Graham: “January 6 not only could have been much worse—it was much worse than was initially apparent. Sometimes real-time coverage of news events leans toward the sensational and overstates…

A Premeditated Election Lie Lit the Fire

January 16, 2021 at 9:29 am ESTJanuary 16, 2021 at 9:29 am EST

Axios begins a series looking into President Trump’s final two months: “For weeks, Trump had been laying the groundwork to declare victory on election night — even if he lost….

U.S. Takes Back Claim Rioters Wanted to ‘Capture’ Officials

January 16, 2021 at 9:26 am EST

“Justice Department prosecutors have formally walked back their assertion in a court filing that said Capitol rioters sought to ‘capture and assassinate elected officials,’” CNN reports. “A federal prosecutor in…

Flashback Quote of the Day

January 16, 2021 at 9:13 am ESTJanuary 16, 2021 at 1:17 pm EST

“The point is that we are all capable of believing things which we know to be untrue, and then, when we are finally proved wrong, impudently twisting the facts so…

Prosecutors Interview Michael Cohen About Trump

January 16, 2021 at 8:44 am EST

“New York prosecutors conducted an hourslong interview Thursday of Michael Cohen, President Trump’s former personal attorney, asking a range of questions about Trump’s business dealings,” the AP reports. “The interview…

Biden Fills Out State Department Team

January 16, 2021 at 8:34 am EST

“President-elect Joe Biden is filling out his State Department team with a group of former career diplomats and veterans of the Obama administration, signaling his desire to return to a…

The Pariah Post-Presidency

January 16, 2021 at 7:48 am EST

New York Times: “In the final days of his presidency, Mr. Trump has been snubbed by foreign allies and banned from social media. Some members of his cabinet fled, and…

Senate GOP Adrift Ahead of Trump Trial

January 16, 2021 at 7:46 am EST

Politico: “Senate Republicans talk as a party as many as three times a week during normal times. But these times are anything but normal. The Senate GOP has not spoken…

Biden Will Push to Legalize 11 Million Immigrants

January 16, 2021 at 7:42 am EST

“During his first days in office, President-elect Joe Biden plans to send a groundbreaking legislative package to Congress to address the long-elusive goal of immigration reform, including what’s certain to…

Republicans Headed for a Bitter Internal Showdown

January 16, 2021 at 7:38 am EST

“As President Trump prepares to leave office with his party in disarray, Republican leaders including Senator Mitch McConnell are maneuvering to thwart his grip on the G.O.P. in future elections,…

« Extra Bonus Quote of the Day
McAuliffe Aide Offered Job to Senator’s Daughter »

Your Account

Sign in

Latest for Members

  • It Comes Down to Mitch McConnell
  • The GOP Is Headed Towards Collapse

About Political Wire

goddard-bw-snapshotTaegan Goddard is the founder of Political Wire, one of the earliest and most influential political web sites. He also runs Political Job Hunt, Electoral Vote Map and the Political Dictionary.

Goddard spent more than a decade as managing director and chief operating officer of a prominent investment firm in New York City. Previously, he was a policy adviser to a U.S. Senator and Governor.

Goddard is also co-author of You Won - Now What? (Scribner, 1998), a political management book hailed by prominent journalists and politicians from both parties. In addition, Goddard's essays on politics and public policy have appeared in dozens of newspapers across the country.

Goddard earned degrees from Vassar College and Harvard University. He lives in New York with his wife and three sons.

Praise for Political Wire

“There are a lot of blogs and news sites claiming to understand politics, but only a few actually do. Political Wire is one of them.”

— Chuck Todd, host of “Meet the Press”

“Concise. Relevant. To the point. Political Wire is the first site I check when I’m looking for the latest political nugget. That pretty much says it all.”

— Stuart Rothenberg, editor of the Rothenberg Political Report

“Political Wire is one of only four or five sites that I check every day and sometimes several times a day, for the latest political news and developments.”

— Charlie Cook, editor of the Cook Political Report

“The big news, delicious tidbits, pearls of wisdom — nicely packaged, constantly updated… What political junkie could ask for more?”

— Larry Sabato, Center for Politics, University of Virginia

“Political Wire is a great, great site.”

— Joe Scarborough, host of MSNBC’s “Morning Joe”

“Taegan Goddard has a knack for digging out political gems that too often get passed over by the mainstream press, and for delivering the latest electoral developments in a sharp, no frills style that makes his Political Wire an addictive blog habit you don’t want to kick.”

— Arianna Huffington, founder of The Huffington Post

“Political Wire is one of the absolute must-read sites in the blogosphere.”

— Glenn Reynolds, founder of Instapundit

“I rely on Taegan Goddard’s Political Wire for straight, fair political news, he gets right to the point. It’s an eagerly anticipated part of my news reading.”

— Craig Newmark, founder of Craigslist.

Copyright © 2021 · Goddard Media LLC | Privacy Policy

Political Wire ® is a registered trademark of Goddard Media LLC