Monday, November 18, 2013

Ignore Those Polls! (Small Sample Size Crosstabs Edition)

Josh Kraushaar also said:
The president isn't just losing his skeptics from the chaotic Obamacare rollout but his allies who stood to gain from the law's benefits -- namely Hispanics, whose approval of the president has dropped more than any demographic subgroup since the problems began. 
Ignore those polls! Or at least, be very careful about subsamples. Kraushaar published this one early today, before last week's Gallup numbers were ready; turns out that Obama's approval among Hispanics, per Gallup, is now at 52% -- up three points over two weeks ago, while the overall rate remained stuck at 41%.

Or whatever. These kinds of subsamples can be all over the place. As it happens, I had noticed the falloff too...and I had been planning to write this item if it turned around, which it more or less has, even before Kraushaar gave me the additional hook.

(Oh, and yes -- the press frenzy of the last two weeks is accompanied by a zero-point decline in approval; Obama is down 2 points overall since the last full week of September, which is the last week before the rollout of the exchanges and the shutdown. Two percentage points. Free fall!).

Here's the difference between Obama's overall approval rating and his rating among Hispanics, going back over time, with this week at 52% overall - 41% Hispanic = 11 points:

11, 9, 8, 15, 13, 10, 14, 10, 16, 14, 16, 15, 18, 18, 17, 20, 18, 17, 18, 18, 19, 19, 20, 15, 20

Going back further, it's around 20 to the beginning of the year, which is as much as Gallup has posted now. So...it does appear that there has probably been a change over time. But the decline started a while ago; that last 20 point gap is at the end of July, and the gap was down to 10 points at the end of September -- before the exchanges, and before the shutdown.

If I had to guess, it would be that it's more about immigration disappearing from the news agenda; as budget and health care dominated the news, Latino voters reacted more or less the same as everyone else. But that's a guess! It's impossible to really get any closer to the truth from these numbers. Other than it's unlikely -- not impossible, but unlikely -- that it's reflecting something about the ACA rollout. The sequence just doesn't seem to suggest that.

The general point: be very, very, very, very wary about using the subsamples in these kinds of polls. It's incredibly easy to get big jumps that are really just random noise -- look, for example, at the 20-15-20 sequence right at the end there. The odds of getting all worked up over something that isn't real are, unfortunately, very high.

2 comments:

  1. Depends, of course, on the subsample.

    Women? Sure...that's half the sample. It's not going to blow the confidence interval all that high.

    Partisans? at about 1/3 of the sample (either Dems or Reps) it's not great, but the larger sample sizes (1000) aren't so terrible here.

    Hispanics? At 10-15% of the sample (usually they're somewhat harder to reach than whites), we're talking about maybe 40-60 people in the smaller samples, and even your full gold-standard 1000 person sample gets you, at most, 150 people. That's not a very big number in your denominator. (technically, the square root of that) Actually, if your sample is 11% Latino/Hispanic, that's works out REALLY nicely to a confidence interval 3 times as large as in the full sample. So, that +/- 3 points? That's +/- 9 points for a Latino subsample. AKA an 18 point range. AKA really big.

    ReplyDelete
  2. Here's the number one reason the Democratic administration has to get the ACA implementation back on track: as far as I can tell, no one, most definitely including all media outlets, will let them pivot to discussing government support for the economy and job creation *until* implementation is going better. US economic confidence continues to be shot, and each day that goes by sequester budgeting seems more normalized.

    ReplyDelete

Note: Only a member of this blog may post a comment.