I came across some trash talk (here and here) between Nate Silver at 538 and Sam Wang at Princeton Election Consortium, and thought I'd add some of my own.
First, there's a disagreement about the right question to ask. Wang thinks that it's only possible to use existing polling information to answer questions about the present and past states of the race; Silver thinks that "What would happen in an election held today is a largely meaningless question." I think that both sides are wrong, and their statements largely reflect their own personal interests. As for me, I'm committed to doing a rich enough analysis to answer questions about both the history and the future of the race. Thus, I can confirm Wang's conclusion that the "Celebrity" ad was far more effective than Obama's international tour, and I can strenuously disagree with Silver's conclusion that FL is a key state.
Regarding 538, Wang says:
In other ways, Silver’s methods are generally acceptable, though some of the details add uncertainty more than they improve accuracy. He is a fiend for numerical information and has a good feel for what’s interesting in polling data.
I basically agree with this, and Wang's second sentence is why I glance at 538 every day on my lunch break. However, his methodology has always seemed kludgy to me, and I agree with Wang that the "reverse Bradley effect" adjustment is just the latest hack. A lot of ad-hoc parameters have ended up hidden in Silver's work, and it's not clear to me that they're good or even consistent.
What Wang would probably say about my site is...
I think "win probabilities" are intrinsically inaccurate for an event two months off, mainly because of the difficulties of figuring out the right assumptions.
He'd be right; this is definitely the hardest part of extending current polls into the future. I think I'm just starting to grapple effectively with the problems of quantifying volatility, state-state correlation, and undecided-allocation. However, I think I'm on the right path, and I think the question is too compelling to abandon research so easily.
I haven't seen Silver give any substantive criticisms of PEC (in fact, he described it as "good work" on a "meaningless question"), but I have one. He doesn't do anything to take into account the correlation or coupling between states. Just to be clear, this has obviously occurred to him (see here and here), but I think his conclusion that this is not a big deal is incorrect, and I don't get the sense that he's separated out the issues of volatility and correlation (instead lumping them both into "co-variation").
I do think this is a big deal. (Not in our electoral vote expected values, which are extremely close, but in assessing probabilities of scenarios.) Let's look at the differences between our assessments of the race today. To see a very specific example, here's a chart of our estimated probabilities of 343 and 375 electoral votes:
| EV | Wang % | Schak % |
|---|
| 343 | 7.23% | 0.69% |
| 375 | 1.34% | 10.75% |
As you can see, I give a high probability of 375 and a low probability of 343, while Wang gives the opposite. As far as I can tell, 343 corresponds to Obama = Kerry + {IA,NM,CO,OH,NV,VA,FL,WV}, while 375 corresponds to Obama = Kerry + {IA,NM,CO,OH,NV,VA,FL,MO,IN,NC}.
This difference cannot be explained by saying we have slightly different opinions about individual states. Our orderings of the contentious states are currently:
Schak: MO - IN - NC - WV
Wang: NC - MO - IN - WV
(In other words, we agree that MO/IN/NC are more Obama-friendly than WV is.)
This difference must be explained by different views about how coupled states are to each other. Wang assumes zero correlation between results of different states, while I assume a moderate positive correlation. His method essentially says, "WV may be more Republican than MO/IN/NC, but WV is only one state, so it's easier for WV to flip than MO/IN/NC simultaneously. This is similar to the fact that it's easier to get one die to land on 1 than to get three coins to land on heads simultaneously; even though each coin is separately more likely to land on heads than the die is to land on 1, the independence of the coins makes it more unlikely for all three coins to land on heads."
But this is the wrong argument for the presidential election, because MO/IN/NC/WV are not independent like the straw man's coin-flips. If we take for granted Wang's ordering of NC-MO-IN-WV, then it's far more likely that Obama would win a set like {}, {NC}, {NC,MO}, {NC,MO,IN}, or {NC,MO,IN,WV} out of these four states, than a set like {MO,WV} or {WV}. A set such as {WV} implies that WV has become significantly more pro-Obama, but that Obama's improvement has somehow failed to impact NC, MO, or IN.
I'm so confident that I'm right about this that if the election were held today, I would gladly give Wang 12-1 odds against the election being 343-195, odds which his analysis suggests would be in his favor. In fact, I'd even give him 100-1 odds against the election being 343-195, which his analysis suggests would be outlandishly in his favor.
One more thing. My proposed probability distribution of electoral votes if the election were held today is an objectively stronger hypothesis than Wang's, using the information-theoretic measure of entropy. In loose terms, Wang's distribution leaves 4.9 undecided bits of information, while my distribution leaves 4.4 undecided bits of information. My probability distribution for election day has 5.5; I'm pretty sure from looking at the graphs that this is a much stronger hypothesis than Silver's distribution for election day, but can't be certain because 538 doesn't seem to post the data behind its EV histogram. As a point of comparison, complete uncertainty about the number of electoral votes (that is, a 1/539 chance of any number between 0 and 538) would have 9.1 bits of entropy.
Here is an Excel file showing, among other things, a side-by-side histogram of Wang's EV distribution and my distribution of the election were held today.
Read more (maybe)!
