Everybody wants to know what will happen in November before November actually comes. To satisfy this demand, there is a cottage industry of statistical modeling that will predict -- with eye-poppingly high degrees of (promised) accuracy -- just what will happen. Nate Silver of the popular FiveThirtyEight blog, now at the New York Times's website, does this, and every election cycle you can usually find a handful of new models in the scholarly journal P.S. -- Political Science and Politics. Earlier today, I blogged on the APSA conference, where political scientists unveiled new predictive models.
I used to be a big fan of these models. I still think they can be helpful when used in a limited fashion, and I also think that Silver and the political scientists who put these APSA models together produce lots of good and interesting work. Still, I've come to believe that these predictive models need to be approached with caution.
Some of my problems are specific to particular models, others have to do with wonky statistical stuff that has no place here. I’ll just focus on my three biggest, relatively non-technical concerns.
(Let me just be clear that my concerns are with predictive modeling of congressional elections, not with the use of statistical analysis to investigate congressional elections, nor with predictive modeling of other phenomena. So, what follows is a critique limited to a pretty small subset of the literature, one that nevertheless gets a lot of publicity from the popular press.)
-There are very few observations to work with. Most models go back only to 1946, at the earliest. According to the laws of statistics, having few observations limits the number of explanatory variables that can be included. That leaves me thinking that these models over-simplify what are really very complex processes. I find it hard to believe that 90 percent or more of the variation in congressional elections over the last 62 years is truly explained by the linear regression of four or five variables.
-It is hard to account for long-run structural changes. Because predictive models date back to 1946, most of their observations were taken prior to the Reagan administration. I think that is a problem. There have been so many structural changes in American politics since the 1940s -- the end of the solid South, the inclusion of African Americans in the voting process, the birth of the modern campaign, the polarization of the parties, the rising importance of party identification, and so on. Any of these changes could influence electoral results independent of the factors typically included in the models. Yet structural variables are too often overlooked.
A great example of this can be seen in the generic ballot. It’s behaving more differently this year than in any previous year. A lot of it has to do with the demise of the Solid South. The GOP is now competitive nationwide, so the generic ballot is showing the Republican party in the lead for the first time ever. That seems to me like a factor that should be included in models like this, but typically it is not.
-Is this a scientific process? Every year, a new batch of models is produced. Each model has slightly different variables than its competitors. Each claims a high degree of historical accuracy. Often times, they predict notably different results for the future.
Isn't that strange?
If they all predict the past so well, why do they disagree about the future? And if researchers are working on models year after year, why haven't they converged on some basic explanatory framework? Beyond the broadest points long since established, I know of no consensus among political scientists about what the true model of congressional elections is, despite decades of modeling.
I think this lack of forward progress is partially due to a combination of several factors: (a) most political variables correlate with one another at least a little bit; (b) there are only a handful of elections; (c) the number of elections in which something "big" happened are fewer still. Taken together, all this means that the same data can support many different models that specify a whole range of theories about how congressional elections really work. Each model will predict the past with accuracy, and will often point toward different future results than its competitors. Unfortunately, the same situation that produces a multiplicity of competing models makes it very hard to eliminate "wrong" ones. It’s easy to tweak models to fit the most recent data, and so the number of potentially correct statistical models never seems to get smaller.
That does not seem like a good scientific process to me. Instead, it seems to me like a situation in which good scientists lack sufficient data to distinctively specify and conclusively test their theories.
The reason I am writing about this on a non-technical page is that statistical analysis implicitly possesses clout because of its technical nature. It’s hard for non-experts to evaluate this kind of wonkish material for themselves, and that can result in an appeal to authority that I don’t think is appropriate. I’d advise caution and care in evaluating these predictive models.