Ezra's 7 (4/7): What Do Polls Mean?

Intro to this series here; Ezra Klein's post is here.

Ezra says:

4) Polls are useful for measuring impressions but very bad for measuring beliefs.

Sounds fine to me.

What I'd encourage reporters to know about polling is to always keep in mind that most Americans don't pay very much attention to politics and don't know very much about public policy issues. At the same time, Americans, or at least enough Americans for the math to work, are willing to answer questions from pollsters.

What that means is that what we're getting with a lot of public opinion polling are not "opinions" as we normally think of them. They're reactions to poll questions, which can be a much different thing. Now, the closer a question gets to reporting on real actions, the better the chances that the answers will be meaningful. Thus questions about voting intentions in October of an election year can be pretty accurate predictors of what will happen in a few weeks. Ask someone what they think of the public option or of health care reform generally, and you'll wind up with a mishmash of stuff that often can't really be figured out, especially from just the raw, single-number results.

It helps when we have consistent questions over time. We may not know exactly what it means for people to say they approve of the president's job performance, but we can look at identically worded questions over many decades, and sensibly say that Obama has, say, higher approval ratings than Clinton but lower than Kennedy over his first ten months, or that Bush's approval ratings dwindled steadily from the September 11 spike on, until he had unusually low ratings in his last two years. Or, we may not know exactly what to make of the abortion data, but at least we can say that attitudes seem to be fairly steady over time.

Beyond that, I'd ask reporters to look at more of the data (numerous polls from all pollsters, not just proprietary polls from their own outlet); to learn a basic understanding of probability (two point "swings" between polls are probably noise, nothing else; and to reconcile poll finding about issues with the things that academic specialists on public opinion have learned, since those specialists generally have much more careful studies to work from.