Inspired by my twitter feed going wild with glee from conservatives after the first day of the Gallup general election tracking showed Mitt Romney with a two point lead (sparking junk analysis like this), a couple hours later by a CNN poll showing Obama up by nine points....
We actually in principle know far more now because we have fifty gazillion polls -- without which the very useful polling averages would be impossible. And yet not only is each individual poll just as meaningless as it was when it was just Gallup and Harris and maybe a couple of others, all of them only once a month or less, but the odds of having at least one weird outlier a month go way up, and even more so the odds of some goofy internal splits showing up go way, way up. It's gonna happen: we're going to get some poll that says that African-Americans in the south are suddenly swinging away from Obama, or that LDS members in the West have suddenly tilted away from Romney, and everyone is going to freak out until it eventually turns out that it was just a fluke of one poll.
Which we'll only know because there will be a dozen more polls out there in the next couple of weeks showing that the first one was off base.
I know I talk about this stuff all the time, and most of the regular readers here are probably sick of it, but as long as it continues to be a major issue in how campaigns are (mis-) reported, I'm going to keep repeating it.
I can't help but notice that Romney is leading in quite a few polls lately. Perhaps it's just a temporary bounce from locking up the nomination
ReplyDeleteWhat we have are Gallup with Romney +2, Rasmussen with Romney +3, CNN with Obama +9 and Reuters with Obama +4. If anything that would suggest that probably Obama leads slightly, though it's gotten pretty close.
ReplyDeleteIt's probably just a bounce from locking up the nomination and Republicans uniting around him. It will probably be a dead heat for a few months, and then after the conventions the fundamentals will assert themselves.
DeleteMy article on this received good reviews, including from one person who said he was a college statistics instructor.
ReplyDeleteBasically, for various overlapping reasons, American phone polling is not actually reaching a random sample which matches the characteristics of the overall population. To get 600 finished calls, they have to make 4000 or more phone calls, there is inevitably a sociological selection process leading to an unrepresentative sample. As far as I can tell, this may fluctuate quickly and unpredictably: today's sample may be overly young and conservative, while tomorrow's sample is overly older and liberal.
The much better record of averages of polls performing so much better than any particular poll is a proof that claimed "margins of error" are all wrong, achieving the claim depends on the sample being representative of the population (and they aren't achieving that). If all pollsters were actually reaching their claimed 3 or 4% margin of error, 19 out of 20 polls would already be tracking each other within 3 or 4 points.
The link is http://www.philosophical-ron.com/opinion-polls-arent-scientific/ , thanks for your patience with this self-promotion. IMHO, the site does have much entertaining and profitable material far in advance of the conventional wisdom.