Context and Interpreting Polls

I hate to be a scold, but...

Dave Weigel noticed yesterday that the number of Americans who think that Barack Obama is helping or will help race relations has fallen dramatically since the first time the Washington Post asked about it (WaPo polling here). Adam Serwer tried to interpret and explain the change in terms of conservative rhetoric surrounding race.

That's fine and all, and I'm a fan of both Weigel and Serwer, but when you see this kind of result, the first question to ask is what else is going on in the polling.  The drop from the first poll, taken during the transition, to the current poll is 23 points. The first question I'd ask is: what was his overall drop in polling? Unfortunately, I can't find an approval number that corresponds with the January 4, 2009 date for the "before" race number, but the two polls surrounding early January asking about the transition recorded 76% and 80% approval numbers. The "after" race number is from a poll which has Obama at 54% overall approval. In other words, 23 point drop on the race question corresponds with an approximately 25 point drop in overall approval.

It is true that Obama is doing a bit worse on the race question than he was in January 2010, despite a flat overall approval trend. So there might be something here. And of course it is possible that at least some of the causation runs the other way, with Americans thinking less of Obama overall because they first decided he was failing to achieve racial harmony. But generally, it's going to be the overall approval rating that drives approval on subquestions, and for all we know the small rally from last January could just be noise.

Polling is a very valuable tool, but always, always ask yourself first whether there are any major factors that call into question the face value of polling numbers.