Seth Masket runs some numbers and winds up updating and supporting the old Tufte conclusion: the things that affect aggregate Congressional results are presidential popularity and change in real per capita income (Gary Jacobson uses one other control variable that seems to be pretty important, "exposure" -- a party is more likely to lose House seats if it has lots of them).
Seth's twist is that he finds pretty much no relationship between unemployment (or unemployment growth) and Congressional election results. His speculative interpretation is that people who lose their jobs are likely to be either nonvoters or strong (Democratic) partisans, so it's not surprising that changes in their relative satisfaction with incumbents wouldn't change voting behavior. Income growth, on the other hand, is more widespread, so marginal voters may change their voting behavior depending on whether they're better off or not.
While that seems plausible to me, I would note that Gary Jacobson's summary of the survey literature (looking at Politics of Congressional Elections, chapter six) suggests that personal economic effects are mild at best. That is, when you try to figure out why individuals vote the way they do, it doesn't seem to have much to do with their own, personal, economic situation. If that's the case, then I don't think Seth's interpretation holds up. It might suggest a different mechanism, however. If general impressions of the economy rather than personal effect of the economy is the thing pushing votes, then it might be that we need to look to media effects -- are media and political elites more likely to be swayed by GDP or personal income numbers than by unemployment data?
Moreover, we know (again from Jacobson) that a lot of what we need to look for are indirect effects. A lot of what matters in Congressional elections happens at elite levels: candidate recruitment especially, but also fundraising and party activist enthusiasm. Again, media effects might be the place to look here: if potential candidates decide whether to run in part by assessing the economy and its likely effect on the election, and they presumably assess how the economy is doing through media reports, then the key question, I would think, is how the media goes about reporting on the state of the economy. I'm sure there have been studies of that -- paging Brendan Nyhan?