A Much Better Defense of "Dumb" Poll Questions

The last Washington Post/ABC poll asked a couple questions that have been ridiculed (well, at least on twitter) all week. For example, “On a ship in a storm, who would you rather have as the captain?” WaPo's Chris Cillizza today attempted a defense of these questions, and only made things worse.

We’ve long maintained that the vote for president, more so than any other vote, is a feel vote.  That is, the up-for-grabs voters don’t simply go to the websites of the two candidates, make a check next to every issue they agree with Obama or Romney on and then add up the columns — voting for whichever of the two men had more checks to his  name.  If they did, George Bush wouldn’t likely have beaten either Al Gore or John Kerry.

The problem is that they can maintain that all that they like, but there's just no evidence that this sort of thing is real. Or, more to the point, the odds are very good that the relationship runs the other way: first voters figure out who they will vote for, and then they go back and make up justifications for it, later mistaking those explanations with the real reasons. Those justifications might be issues-based, or they might be classic retrospective ones (such as the "better off" question), or they might be these personal attribute questions. Unfortunately, it doesn't solve everything if we directly ask people why they decided. For better or worse, we're just not very good at all at understanding the reasons for our candidate preferences.

So of course voters don't vote based on a careful comparison of issue positions, but these kinds of questions don't get at how they do vote -- by party, by group, and yes, by retrospective evaluation.

But that doesn't make "dumb" questions a bad idea! If properly devised (and that's tricky to be sure), they may be able to get at the stories we tell ourselves about why we're supporting one candidate or another. That has to be done carefully, I think; it would be easy to imagine questions which essentially invent those stories, rather than reflect pre-existing ones. But in principle, it should be possible, and while that might not tell us why a candidate was winning, it might tell us something about what was in voters' minds, and that's not a bad thing at all.

Somewhat less seriously, such questions can be fun, and there's nothing at all wrong with having some fun with electoral politics. Just because we might realize that "who would you rather have a beer with?" might not be a question that really illuminates why swing voters went a particular way doesn't mean that it isn't any fun to think about.

In general, I'd say that too much political coverage is based on the idea that it's news if it predicts who will win or if it explains why a candidate is winning. Nothing at all wrong with coverage of how it's happening, even if it doesn't answer that "why?" question. And I'm really not convinced that this is viewer or reader driven; I think it's just a norm that doesn't make sense.

So, pollsters: go ahead and ask "dumb" questions; reporters, have fun talking about them. Just don't think they're explaining stuff that they can't explain, and you'll be fine.

(UPDATED Note: I didn't save the links, but Brendan Nyhan on twitter has been really good about silly polling questions. Also, John Sides had a really good post about knowing our reasons for political choices, but I couldn't find it. [Link added above, but it was a Lee Sigelman post, not John] My apologies for the laziness. I have no idea how either of them would feel about my defense of these questions, however).