Increasingly, pundits are incorporating statistical models into their analysis of the 2012 election. While I was once a purveyor of such predictive models, I really am not anymore. I don't want to bore you with all the wonky details of my flip-flop, but I do want to give you an example of why you should feel comfortable evaluating these models yourselves, based on common sense.
Writing at Larry Sabato's Center for Politics, Alan Abramowitz uses presidential job approval in May of the election year to derive estimates of final results in November. He concludes:
Whether we base our prediction on President Obama’s 47% approval rating in the Gallup Poll in early May or a more sophisticated forecasting model incorporating economic conditions and the “time for change” factor, it appears likely that we are headed for a very close election in November. Both models make Obama a slight favorite to win a second term. However, the final outcome will depend on the actual performance of the economy and the public’s evaluation of the president’s job performance in the months ahead. Those interested in assessing where the presidential race stands should focus on these two indicators rather than the day-to-day events of the campaign, which tend to dominate media coverage of the election.
Specifically, the model predicts that, with a 47 percent approval rating, Obama will win about 51 percent of the vote.
This does not make sense to me. And there is no reason to suspend common sense when confronted with a statistical model, even if one lacks the technical knowledge to critique it.
For the president to win 51 percent of the vote with a 47 percent approval rating now, either one of three things will have to happen: (a) his approval rating will improve; (b) a larger number of disapprovers will vote for him than approvers will vote against him; (c) the instrument you're using to predict his job approval among the actual electorate (in this case, the Gallup poll) is somehow biased.
Let's leave aside item (c), as it is a major statistical problem, but too wonky to bother with here.
Possibility (b) is exceedingly rare. It only happens in blowout elections like 1972 and 1980, when one party coalition is actually starting to break down. Otherwise, presidents tend to get their job approval. If anything, they tend to do a little worse than their job approval numbers.
What about (a), the idea that presidents tend to see improvement between May and Election Day? Whether or not that is true, there is no way for the model to know that will happen with Obama.
The lesson from all this? You can create highly accurate predictive models that make little theoretical sense. And that is a huge issue. To be behaving in a "scientific" fashion, we cannot just fit a line to a graph with a model. Instead, our model has to be our best guess as to how the world actually works, and it needs to be accurate from every possible angle from which we examine it. That’s how Karl Popper – one of the greatest philosophers of science – conceived of science. It’s a series of “conjectures and refutations.” We make arguments about how the world works, test it against the data, and refute our theories when they don’t measure up. We do not simply try to fit a line to a data array.
That means that statistical analysis and scientific analysis need not be the same thing. Such is the case here. In my opinion, a model that envisions a president picking up a substantial number of their disapprovers does not make sense, and indeed there is good empirical evidence to suggest that this rarely happens.
So moving forward, when somebody tosses a statistical model at you, don't be cowed. Take a careful look at it to see whether it makes sense. If it doesn't, you don't need to suspend your own judgment just because it's a wonky statistical system!
Jay Cost is a staff writer for THE WEEKLY STANDARD and the author of Spoiled Rotten: How the Politics of Patronage Corrupted the Once Noble Democratic Party and Now Threatens the American Republic, available now wherever books are sold.
