Wednesday, September 12, 2012

A Much Better Defense of "Dumb" Poll Questions

The last Washington Post/ABC poll asked a couple questions that have been ridiculed (well, at least on twitter) all week. For example, “On a ship in a storm, who would you rather have as the captain?” WaPo's Chris Cillizza today attempted a defense of these questions, and only made things worse.
We’ve long maintained that the vote for president, more so than any other vote, is a feel vote.  That is, the up-for-grabs voters don’t simply go to the websites of the two candidates, make a check next to every issue they agree with Obama or Romney on and then add up the columns — voting for whichever of the two men had more checks to his  name.  If they did, George Bush wouldn’t likely have beaten either Al Gore or John Kerry.
The problem is that they can maintain that all that they like, but there's just no evidence that this sort of thing is real. Or, more to the point, the odds are very good that the relationship runs the other way: first voters figure out who they will vote for, and then they go back and make up justifications for it, later mistaking those explanations with the real reasons. Those justifications might be issues-based, or they might be classic retrospective ones (such as the "better off" question), or they might be these personal attribute questions. Unfortunately, it doesn't solve everything if we directly ask people why they decided. For better or worse, we're just not very good at all at understanding the reasons for our candidate preferences.

So of course voters don't vote based on a careful comparison of issue positions, but these kinds of questions don't get at how they do vote -- by party, by group, and yes, by retrospective evaluation.

But that doesn't make "dumb" questions a bad idea! If properly devised (and that's tricky to be sure), they may be able to get at the stories we tell ourselves about why we're supporting one candidate or another. That has to be done carefully, I think; it would be easy to imagine questions which essentially invent those stories, rather than reflect pre-existing ones. But in principle, it should be possible, and while that might not tell us why a candidate was winning, it might tell us something about what was in voters' minds, and that's not a bad thing at all.

Somewhat less seriously, such questions can be fun, and there's nothing at all wrong with having some fun with electoral politics. Just because we might realize that "who would you rather have a beer with?" might not be a question that really illuminates why swing voters went a particular way doesn't mean that it isn't any fun to think about.

In general, I'd say that too much political coverage is based on the idea that it's news if it predicts who will win or if it explains why a candidate is winning. Nothing at all wrong with coverage of how it's happening, even if it doesn't answer that "why?" question. And I'm really not convinced that this is viewer or reader driven; I think it's just a norm that doesn't make sense.

So, pollsters: go ahead and ask "dumb" questions; reporters, have fun talking about them. Just don't think they're explaining stuff that they can't explain, and you'll be fine.

(UPDATED Note: I didn't save the links, but Brendan Nyhan on twitter has been really good about silly polling questions. Also, John Sides had a really good post about knowing our reasons for political choices, but I couldn't find it. [Link added above, but it was a Lee Sigelman post, not John] My apologies for the laziness. I have no idea how either of them would feel about my defense of these questions, however).


  1. Ever since you (I think it was you) mentioned how you thought polls showed voters assumed Bill Clinton grew up rich, I've wanted more polls like that. It'd be nice to know how many people really knew that Obama used to be a lecturer and Romney used to be a C.E.O., since in pundit-land these things are assumed to be universally known.

    1. That sounds like me -- and yeah, I agree. I'd love to know how many people know the basic stuff about the candidates that we all think that everyone knows.

  2. Great post. I wonder if pollsters could be more accurate and creative by first gathering data about the associations people self-report. I'm thinking of a large sample, free response survey in which people do simple word association. They might be expected to offer 20 words that they associate with a candidate, and then those words could be used as the basis more creative kinds of questions.

  3. I answered a poll a year or two ago -- not sure if it was ABC, but it was one of the big three. One of the questions included joke "answers" that only existed to make fun of Republicans. It was so juvenile that the woman asking me chuckled nervously. It was at the end of a lengthy and otherwise very serious poll, so maybe they expected insulted people to just hang up at that point. The only thing keeping me from doing so was that the woman was very nice and didn't seem to want to ask me the question (I had already identified as a Republican).

    1. And for the record, I think the "ship in the storm" question is a worthy one. I can't recall the exact question I was asked, but it was so juvenile and anti-Republican that I have a feeling it never saw the light of day.

    2. What were some of the questions? -Just curious.

    3. Anon - There were questions related to 9/11 (it was the ten year anniversary) along with some more general political questions and the usual demographic stuff.


Note: Only a member of this blog may post a comment.

Who links to my website?