I think Nate Silver's advice to himself about how to handle fast-breaking polling news is correct. And his advise to the rest of us about how to deal with it is basically wrong.
I should explain. Silver has a post out today defending the way he's handling analysis of convention bounces again those
who correctly note that polling around the party conventions can be volatile. They suggest that we ought to wait for more data before concluding very much about the bounces that the conventions have produced.Silver contends:
Saying “wait for more data” sort of misses the point. What about the data that we have on hand already? Is it compelling enough to suggest that there has been some change in condition of the race? Or isn’t it?Well, here's the thing. Silver normally publishes daily updated projections. Would his site be more useful if he only updated once a week, or when some sort of critical mass of new information was out there? I don't think so. Would his site be more useful if it shut down (or at least the projections were shut down) while the convention bounces came and went? No, not really.
That’s really the central question that we seek to answer with our forecast models. On some days, the trend is a little more obvious than on others. But we make forecasts when it is easy to do so, and we make forecasts when it’s hard.
But I'll stick to what I said before the conventions: the best choice for us as consumers of this information is to be patient about it all. In another week or two we'll know a lot more than we did two weeks ago. What we learn in the meantime, however, is just not particularly reliable. We don't have enough experience with all of it to really know what it means as it's happening (and on top of that, we're really just dealing with the tracking polls; we haven't had very many other polls at all).
Now, I say that without having read Erickson and Wlezien's new book, which I'm certain is excellent and which I can highly recommend sight-unseen, but off the top of my head I can think of half a dozen reasons to be skeptical that the pattern of previous convention bounces will hold this time around. Let's see...the conventions are usually earlier; they usually are not back-to-back as they were this year; Paul Ryan was chosen unusually early; Labor Day may have messed with the polls...well, that's four quick ones, so add increased polarization and the rise of the partisan press (not brand new, of course) and that's six.
Look, we know a few things. We know that horse race head-to-head polling six months out and more is mostly useless. We know that horse race head-to-head polling in October is a very good predictor of election results. We know that traditionally, conventions do a fair amount of the work of moving from that first thing to that second thing. And we also know that traditionally, conventions can move some people temporarily in ways that fizzle out over the next week or two.
What we really don't know are a couple of things: this time, under current conditions, what do conventions do? And: this time, how much if any of that temporary bounce are we seeing?
Fortunately, most of us don't need to know. Unless we're making choices about scarce campaign resources, it just doesn't matter. If we get it wrong during this window in which the polls can't quite be understood, it really doesn't matter.
If we want to get it right, the best advice is to wait another ten days or so and then consult the polling averages out there.
But you know what? We political junkies are generally pretty desperately curious, too. And there's nothing exactly wrong with that. The trick is, if we don't want to misinform ourselves is to just keep in mind that there's an unusual amount of uncertainty right now. And there's nothing at all wrong with the various polling projection sites going ahead and giving us their best estimates right now, especially if they make it clear exactly what it is they're doing.
So: want to know what's going on in the presidential race? My advise is to wait another ten days. But since neither you nor I is going to do that, let's all just remember that the estimates we're seeing right now are quite a bit less reliable than usual, and that in a week or so we'll have estimates that are quite a bit more reliable than any polling-based estimates we've seen so far. Now, I can't expect Silver to tell everyone to wait until a couple of weeks after the convention and tell all of his readers that he doesn't have anything in the meantime, especially when they -- sorry, we -- have a particularly strong craving right now. So he's right to keep updating, and it's certainly more accurate than not updating, anyway. It's just that really what we all should be doing is waiting it out.
What you're saying is that Silver's model should - but doesn't - take into account the extreme volatility of convention-time polls.
ReplyDeleteHis model does assume that each candidate will get a bounce coming out of his convention, and that's built in to the model. But it doesn't seem to actually discount the results of those convention-time polls in any way.
And I think this speaks to the main difference between you and Nate in your thinking on this. You believe that convention-time polls are inherently unreliable and are worth much less than polls taken weeks after the fact. Nate, on the other hand, would agree that these polls are unreliable, but he would say they're unreliable in a predictable way; i.e., we can expect to see surges for each candidate after their respective conventions, and we can expect those surges to dissipate over time.
Given the factors you mentioned for why this year might confound the expectations (and assuming that the data actually do show that convention-time polls are volatile) I think you have the better of the argument. Nate should not only compare the convention-time polls to the historically-expected bounces, but he should also discount the value of these polls somewhat to better reflect their volatility. That way he could put out projections every day - every hour if he wanted to - and there would be no need to wait until more reliable polls come in.
Silver's been publishing his graph of the aggregate "discount factor" for the bounces for a couple of weeks now.
DeleteTo fairly take in our esteemed host's point, he would need to include error bars on that plot. And, like you and Jon, I think those error bars are pretty big. Combine that with the seeming pattern (that Silver notes) of shrinking bounces (which would be consistent with increased polarization), and I think the point of "wait a couple days....we can't quite know right now whether the past is prologue or not" is a solid one.
I think Jon's point is very good, and I like that he realizes that it's mostly both lost on the crowd here and NOT lost on us. We're jonesing for our fix, man!
I think the one point you're missing is that Silver's model "prices in" an average bounce after the convention, and it's going bananas because Obama has appeared to exceed that mark while Romney never met the average bounce threshold.
ReplyDeleteAs to waiting ten days, or waiting until October-- obviously that isn't Silver's field of interest, which he explains in that post and just wrote an entire book about. For that matter, he's building uncertainty into the way he expresses his forecast, which is that Obama has an 80% chance to win reelection. Are there any data points you see out there that would suggest that Romney has more than a 1-in-5 chance to get to 270 electoral votes? It's not like 1-in-5 odds are really even that much of a longshot. (Much better odds than the Pirates making the playoffs, for instance).
Romney has trailed this race basically since it began. One of the key inflection points of the race has passed without him changing that fact. That doesn't end the race, but it has to make him a pretty decent underdog, no?
Andrew,
ReplyDeleteI'd say that there's the known uncertainty, which Silver prices in as best he can, but also unknown uncertainty, which he can't.
Anon,
The problem is that we don't actually know that the conventions have passed without Romney gaining. We won't for a while. It's possible that Romney might have gained 2 points that sticks, while Obama has a 4-6 point bounce that's all temporary.
I definitely like that Silver expresses things as probabilities. But we should be very careful about reading it and saying, ah, the uncertainty is understood and built it.
Point 1. "we also know that traditionally, conventions can move some people temporarily in ways that fizzle out over the next week or two."
ReplyDeletePoint 2. "So he's right to keep updating, and it's certainly more accurate than not updating, anyway"
How can both of these things be true? If point 1 is true, than updating is actively worse than not updating, as it's more likely than not that the forecasting is actively getting the race wrong. If it's true that polling after the conventions is inherently misleading, than it would be the case that using data from that period is wrong, and that you would be actively misleading people by presenting it.
"And there's nothing at all wrong with the various polling projection sites going ahead and giving us their best estimates right now, especially if they make it clear exactly what it is they're doing."
To the best of my knowledge, the non-partisan forecasting sites, like five thirty eight, have gone to great lengths to explain their models, why they believe the models behave as they do, why they believe the models are accurate and can be trusted, etc.
Because there's a difference between accuracy and precision.
DeleteBecause we know that politicians get temporary bumps from conventions, all we have to do is subtract out the expected value of this bump (which 538 does) and we'll have an unbiased estimator of their actual probability of winning. But even with such adjustments, there's still going to be a significant amount of uncertainty. But as long as you indicate that the information is uncertain, you're providing accurate information.
Accurate but imprecise information such as "either Romney or Obama will win the election" is still better than nothing.
I'd say that there's the known uncertainty, which Silver prices in as best he can, but also unknown uncertainty, which he can't.
ReplyDeleteAh. Rummy would be proud!
If point 1 is true, than updating is actively worse than not updating, as it's more likely than not that the forecasting is actively getting the race wrong
I think the problem is that we don't know if it's right or wrong. Just because convention-time polls are volatile doesn't mean they are incorrect. I think that would be cured by taking those polls into account, but discounting their value significantly to reflect that they are less trustworthy than polls taken at other times.
It's worth noting that 538's model does somewhat discount polls taken during the convention period.
ReplyDelete"We do hedge against the convention bounce adjustment in two ways, however. First, in addition to adjusting the polls during the convention period, we’ll also assign them less weight. (Specifically, polls conducted when the absolute value of the convention bounce is at its highest will receive a 50 percent penalty, with everything else scaled accordingly.)" from http://fivethirtyeight.blogs.nytimes.com/2012/08/29/measuring-a-convention-bounce/ which explains the methodology for adjusting for convention bounces. Perhaps this adjustment isn't enough but it's simply not true that 538's model considers polls during the convention period as reliable as other polls.
I stand corrected. Thanks.
DeleteMax --
DeleteYes, I think he handles that as well as he can. There's just no way of knowing, however, whether it's too much, too little, or what.
And if no one ever tries to handle it, obviously obvious settled political science tells us we'll never know? Why ding Silver for doing what he says with candor what he's trying to accomplish.
DeleteCould be, over time, he'll find a model which better picks up on how to judge if a convention is 'successful' or not. Or at least tell us more than we know (or don't know, in your opinion) at this stage.
You kind of sound like Tom Haller shaking his fist at Bill James, and all his funky new measurements to judge player performance.
;)
I have to respond to that one...the people in this who are Tom Haller are (some of) the practitioners, not the political scientists. Oh, not that we can't be wrong, but we're not lacking in empirical research, really. I'm very confident that Nate would agree with that.
DeleteIt all comes down to saying that the data now is particularly noisy. We could ignore it for that reason, and wait till midmonth for less noisy data. But of course we won't.
ReplyDeleteAnd as noisy as the data is, it does seem to show that Obama gained some ground in the convention phase of the campaign. Which we will dine out on, noisily, taking our chances on a serving of crow later on.
I understand your point, but this is a tautology:
ReplyDelete>>In another week or two we'll know a lot more than we did two weeks ago.
I mean...if we wait until November 7, we'll have much more accurate information, too.
Well, the point is that now we don't know (much?) more than we did a couple weeks ago.
DeleteI actually think we do know something we didn't. We know that Obama apparently got a large bounce and Romney apparently got a small bounce and perhaps no bounce at all. That's significant and something we didn't know two weeks ago. Sure in two more weeks we'll know how temporary the change in polling is, which is more information still. But I don't think we can say the current information is irrelevant or meaningless, which is Nate's point.
DeleteIt could be that Romney got a smaller and permanent bounce that is drowned out by a larger but temporary Obama bounce. But ask yourself if this is actually more likely than the possibility that Obama gained some support during the conventions? I think the latter is a more likely scenario, especially given than some modelers have found either no Romney bounce or a negative one (like Sam Wang).
Isn't there some sense in which the appearance of a polling bump, whether sustainable or not, might actually create a bump itself?
ReplyDeleteI mean a lot of what influences the lowest information voters is brief random news bits that only refer tangentially to what's going on. With the MSM's proclivity for covering the horse race over policy it seems like one of the few things these voters might hear about the campaigns in the course of a week is that Obama "took the lead" after the conventions. Whether that is true or just noise, it's the kind of thing that could sway some of these voters due to the whole winner's bias/positive exposure thing.
To bring my point home, CNN.com's headline right now is "Breaking News: Convention bounce puts Obama up by 6 points"
Id venture that all but the most engaged and curious political junkies will actually parse this for what it really means, while most people will assume Obama is actually changing minds with the power of his oratory or the perceived wisdom of his policies.
I think you're precisely incorrect. Political junkies won't parse anything, but rather will hope that the polling results will/won't change in support of their chosen tribe, and they'll be joined by 80-85% of the electorate, who behave much like them in voting tribally, with little thought.
DeleteThe remaining 15-20% of the electorate will ignore the noise and chatter. Some have already made up their minds, even if they're not tribalists. The rest will ignore tribalist yammering as they decide.