Thursday, October 18, 2012

Yes, Gallup Is Probably Goofy. So What?

Today's Gallup numbers? In their likely voter screen, Mitt Romney leads by a whopping 7 points, 52-45. On the other hand, they have registered voters at 48-47, with Romney barely leading. Meanwhile, Barack Obama's approval rating (measuring over three days, rather than the seven they use for head-to-head) ticks back up to 50%. Huh?

It is highly likely that something is going wrong here -- a six point gap between registered and likely voters is just too high given how many registered voters actually vote, and a 50% approval rating (and the previous three days was at 48%) really isn't consistent with getting crushed by seven points.

So what should we want to do with that number if we want to know who will win? Regular readers know my answer: what we should do with every polling number. Ignore it, and look at the polling averages and the good poll-based predictions. Really; if we were smart, we would all ignore all the individual state and national polls and just keep an eye on, say, the Pollster national average and Nate Silver's current prediction.

Everything else is just going to be running up against the limits of what the polls can really tell us.

In particular, it's a waste of time to try to piece together what's going on with Gallup. Perhaps it's some sort of screwy random effect. Perhaps they're exactly correct, and everyone else is wrong. Perhaps they just have a lousy likely voter screen.

It doesn't really matter. Toss it into the soup, and that's the end of it. And, yes, I definitely do think that those who are doing the polls-of-polls should certainly be including Gallup, even though it's very likely something weird is going on. Of course.

Now, do I take this advice? Well...not exactly. I click over to Gallup every day when the numbers change. But I do feel quite foolish every time I do it. And I am most definitely avoiding as best I can complaints from both sides about polls being slanted against them. Toss it in the pot, and that's the end of it.

21 comments:

  1. At what point do you think it's time to ignore a poll or polling company? If I started a company called "Jason polling," and asked 100 friends who they're voting for, and listed the result as Obama +10, you'd probably advise that this shouldn't go into mix, because it was obviously badly done.

    From what I can tell, Gallup has never been particularly accurate about election results. There has to be some point where it's not worth listening to a pollster, right? What exactly is that threshold for ridiculousness?

    ReplyDelete
    Replies
    1. I was going to put in a paragraph about that, but wanted to keep it short...

      Basically, as long as it uses relatively standard methods and has a track record for honesty, toss it in. IIRC, Nate Silver does give different surveys different weights...can't remember what exactly it's based on, but you want to be very, very careful that you're not cherry picking.

      Delete
    2. Silver bases his weights on track record in previous elections, essentially.

      And, FWIW, Gallup is a perfectly fine pollster, certainly not terrible.

      The Gallup number caused me no small amount of cognitive dissonance this AM. I'm choosing to not think about it. It is odd, but it's possible that it's right.

      What I'm finding really wierd is that the cell phone effect (about +2.5% for Obama if cell phones and live interviewers are part of the protocol) seems to be gone lately...the robopollsters have been tracking slightly higher for O than R over the last couple of weeks.

      Maybe it was all a few pollsters with some wierdness in their different screens leading to some temporarily odd results, because robopolling was pretty much similar to live in 2008/2010, IIRC. Maybe it's really tough to get a nice consistent screen in 2012, what with an incumbent president that isn't feeling the love (bad economy, some deviations from what the base wants {transparency, drones, etc.}), but a base that thinks the other side is clinically insane. Similarly, the out-party has a candidate they dislike running against an incumbent that they loathe with every fiber of their being. So, I wonder if questions on "how interested/excited are you in the election" are not quite asking the right thing, but that we can't ask that ("hey! are you going to vote?") because it never gets us good data.

      Delete
    3. Your statement that "the out-party has a candidate they dislike running against an incumbent that they loathe with every fiber of their being" is hardly true of the entire Republican Party. I have been an active Republican my entire adult life (I am 61), am a member of the Monmouth County, New Jersey Republican county committee, and I can assure you that most active Republicans in the outer New Jersey suburbs of New York City like and respect Romney. They disagree with Obama's tax, spending, and regulatory policies, but hardly "loathe him with every fiber of their being". The Republican Party has tens of millions of adherents, and they represent a broader range of outlooks than you seem to think. Many have moderate views on social issues but have personal experience in the business world of over-regulation; for example, those of us in financial services find the 2,000+ page Dodd-Frank law burdensome.

      Delete
    4. @ Jarvis yeah that's what I was thinking. Looks like the likely voter screens are just terrible, they probably ask things like if your more or less excite about voting this year compared to 2008. Stuff like that. Although to be sure JB's post was about not dissecting individual polls and we are now of course dissecting individual polls. I would just say that every time your look at the cross tabs of a poll to prove it "wrong", the Lannisters win. Obviously we just screwed ourselves.

      Delete
    5. Anonymous at 2:18: First, thanks for a reassuring reminder about the breadth of the Republican Party. I hope your views gain strength within it in the coming years. On the other hand, while I can't vouch for every page of Dodd-Frank, the crash of 2008 resulted in large part from the deregulation of financial markets, and a lot of people found that kind of burdensome, too.

      Delete
  2. These polls drive me crazy...I cant understand how Obama can be behind.

    ReplyDelete
    Replies
    1. He's not, in the polling averages. The race has been stuck as a tie for days.

      Delete
  3. Gallup is certainly an outlier, but there's little doubt that, right now, Romney has a small but significant lead in national polls.

    There is also little doubt that, right now, Obama has small but significant leads in polls of NV, OH, WI, IA, and PA. Those states alone, plus the solid blue ones, give Obama 271 EVs.

    It seems to me that there is a very strong chance (way more than the 5.8% given by Nate Silver) that, if the election were held today, Romney would win the popular vote but lose the electoral college.

    ReplyDelete
    Replies
    1. I conquer. (and hope people get the reference!)

      Delete
    2. Matt Jarvis - Catch me if you can :)

      Delete
    3. 10/17 National Tracking Polls (w/o Gallup) -
      -----------------------

      IBD/TIPP: Obama 46, Romney 46

      Ipsos/Reuters: Obama 47, Romney 44 (LV) (RV 47-39)

      PPP: Obama 48, Romney 48

      Rasmussen: Romney 49, Obama 47

      Rand: Obama 50, Romney 44
      -------------------------------

      I'm not actually seeing a 'small but significant' Romney lead here.

      Delete
  4. Given how many reasonably credible polls are coming out, there's a pretty good chance on any given day that at least one will fall outside the margin of error.

    Which just reinforces the point of focusing on the polling averages, not individual polls.

    ReplyDelete

  5. Is it possible that Gallup over-sampled the south? According to DailyKos regionally President Obama is ahead in the East & Mid-west by 4 points & the West by 6 points, but Romney is inexplicably ahead by a whopping 22 points?


    Nate Silver says, "The first rule of poll analysis is that if a poll looks like an outlier, it usually is."


    I think the best thing to do is to follow your advice & Nate's advice.

    - serena1313

    ReplyDelete
  6. Going back it looks like Gallup's LV screen has a consistent Republican slant over every election of the last decade. Why wouldn't they fix it?

    ReplyDelete
    Replies
    1. LV screens *should* have consistent Republican slants - Republican base voters (old white people) are more likely to vote than Democratic base voters (minorities and young people). That's just a fact of our current political split. But in general it's shown that the RV/LV difference should be more like 2 points than 6 points, and it's also really weird that you would get daily results where LV moves towards one candidate and RV moves toward the other.

      Delete
  7. Case in point about the weirdness of polls: PPP had a national poll out a few days ago that was Romney +5, they released one today that had a tie.
    Pollster had a good summary of the two prevailing ways that pollsters are looking at the numbers: 1) Romney's debate performance shifted the race to a tie by making himself acceptable to previously undecideds, or 2) The debate "bounce" is fading like Obama's post-convention "bounce" faded and the race is settling back to where it was at the end of August... Obama a slight favorite.

    ReplyDelete
  8. I have a potential "Cranky Blogging" subject for the Plain Blogger! Chris Cillizza had a piece in the Washington Post today about how Romney "doesn't have a woman problem" because he's losing women voters by about 7 percentage points whereas McCain '08 lost them by 13 points, W. Bush '00 lost them by 11, Dole '96 lost them by 16 points, etc.

    The problem? All those candidates lost the popular vote! They didn't just lose among woman voters... they lost ground in all demographic groups.

    I didn't do much research but I noticed that Reagan won the vote of women by 16 percentage points in '84. and he won the election by a landslide.

    ReplyDelete
  9. Among all the polling talk, which seems a bit more.. urgent(?) than four years ago.. (understandable in a race perceived to be much closer than the last one)

    Isn't it true that the campaigns also do their own polling, and those are generally accepted to be more prolific and accurate than anything the polling firms generally release to the public? I assume that's why the campaigns will occasionally put out a memo, or otherwise push back against a particularly bad-sounding public poll result - they couldn't really do that if they didn't have their own numbers?

    ReplyDelete
    Replies
    1. Campaigns do their own polls and claim that they're better, but that has been questioned.

      http://themonkeycage.org/blog/2012/10/02/should-you-trust-campaign-pollsters/

      Delete
  10. Are likely voter screens misleading? All year the pollsters have said GOP leaning voters are more motivated and more enthusiastic than Dem voters and yet early voting in Iowa, Ohio and Wisocnsin are showing large Obama leads over Romney.The gound game can't explain all of it

    I did a little research on Rasmussen's polling of the swing states the last weekend before the 2008 election and found Rasmussen had at least a 4-5 point GOP bias in his state polling over the last weekend before the 2008 election, he also incorrectly gave McCain Florida, North Carolina and Indiana and had a tie in Ohio:

    Rasmussen final swing state polls in 2008 election.

    Colorado…………RAS Obama +4, Actual Obama +9).
    Ohio……………… RAS Tie, Actual Obama +4.6).
    Florida……………RAS McCain +1, Actual Obama +2.8).
    Indiana……………RAS McCain +3, Actual Obama +1.1),
    Michigan…………RAS Obama +10, Actual Obama +16).
    Nevada………… RAS Obama +4, Actual Obama +12.5).
    Wisconsin………RAS Obama +7, Actual Obama +13.9).
    Virginia…………RAS Obama +4, Actual Obama +6.3),
    N. Carolina…RAS McCain +1, Actual Obama +0.3).

    ReplyDelete

Who links to my website?