Could the polls be wrong? It’s happened, but…

A bunch of people, both live and over email, have asked me why I don’t blog more on poll results, especially given that I worked in the business for a number of years.

Admittedly, I’m blogging a lot less on poll-related topics than I did, say, a month or more ago, which is a bit counterintuitive in light of the avalanche of poll results that comes in the last couple of months before an approaching presidential Election Day.  Since mid-September, pollblogging on WitSnapper has been especially rare, and even then I’ve typically done so only to comment on the misuse or shoddy execution of polls.

There’s a pretty simple reason: taken as a whole, the polls for this race have been entirely unreliable, and my estimation is that it’s because of pollster panic.

In addition, if McCain were to overtake Obama in the last couple of weeks in defiance of the polls, it wouldn’t be the first (or even second) time in modern election history.

Details and further links below the break.

Now, when I say the polls are “unreliable,” that’s not to say that I think McCain actually has a solid lead over Obama which the polls have missed entirely (I’m not going to make some Nixonesque case for the “silent majority”).  What I mean by “unreliable” is that, taken as a whole, the results of these polls have virtually nothing in common. Two national polls just released, by Battleground and NBC News, show Obama with advantages of 2 and 10 points, respectively, during similar time periods. An Associated Press poll, just released (no link yet), shows the race effectively tied at 44 to 43.

DJ Drummond has a lengthy but interestingly argued blog post at Wizbang asserting that polls this year, more than any other, are not to be trusted.  He assigns a good deal of venality to the polling organs, in fact quite a bit more than I’d venture (saying essentially that polling comes into greater demand when the race looks like a wild, unpredictable “roller coaster ride” and therefore improves a pollster’s bottom line), but he and I are in agreement on one important thing: pollsters this year appear to be fiddling with the number-crunching process far too much, in an effort to predict what a cross-section of this year’s eventual voter pool will look like. It’s an X-factor (or, more accurately, a wide array of X-factors) no one can seem to pin down to the satisfaction of any objective observer.

Samples and demographic weighting for a survey of the progress of a political race are determined primarily by three things:  census data, voter rolls, and previous election results.  In this election, all three are problematic:

  • Available census data, regenerated decenially, is now eight years old. We’ll be ditching it for new data in two years; until then, we’re using precinct, county, state, and national census demographics measured during the Clinton administration.
  • Voter rolls are constantly updated, and therefore more current than census data, but difficult to gauge in many states; some states require far less information than others for registration (such as party affiliation, an important demo in any political survey). On top of that, the rash of voter registration fraud uncovered in the news throws voter roll data accuracy into serious doubt, especially in battleground states.
  • Basing turnout expectations (among other data) on previous election results invokes the old cautionary military metaphor of “fighting the last war.”  This year pollsters will be using turnout numbers from the last presidential election in 2004; does anybody think turnout numbers from demographic to demographic will be comparable in 2008?

Now here’s the thing: I don’t think the problem is that pollsters haven’t taken these complications into account.  No, the problem is the opposite: I think they are obsessing over them, especially that last one, to the point where they’re overcompensating with entirely too much “educated guesswork.” Tweaking numbers so that the demographics reflect what the likely voting public will look like is an entirely acceptable practice, when done in moderation. Repeated tweaking, however, has a cumulative effect on the poll’s methodology in which it eventually fails to meet the minimum threshold for scientific randomness.  There’s a line dividing “scientific” and “skewed” that can’t be crossed.  Since that line depends on nuanced degrees of randomness, it’s a somewhat blurry line, which is why reputable pollsters do only minimal weighting in order to avoid even coming close to that line. That ethic appears to have gone out the window this year, as pollsters have gone crazy trying to account for new conditions on the ground (not reflected in available data) that make this election unique.

I did mention earlier that the polls have been comparably wrong recently. In 2000 and 1980, mid-October polls had Gore and Carter, respectively, enjoying comfortable leads, only to see them disappear (Gateway Pundit has a post on the matter.)  I’m skeptical; both of those elections took place under the influence of crushing Clinton and Carter fatigue (again, respectively). This time the fatigue runs in the opposite direction, but I will still be keeping these recent discrepancies in mind.

Another inherent polling problem to keep in mind is the disrepute into which exit polling has fallen in recent elections. Exit polls, taken by volunteers who wait outside polling places and ask people on their way out which candidate they just voted for, have been ridiculously wrong recently, especially since 2000 (in 2002 the entire VNS apparatus crashed, leaving us entirely without exit poll results…it actually made for a much calmer Election Night).  In another post by Gateway Pundit, pollster Mark Blumenthal comments that in 18 of 20 Democratic primaries this year, Obama’s eventual voter share was consistently overestimated by exit polls, by an average error of 7 points. Blumenthal assigns the gap to the fact that Obama’s supporters, who are young, highly educated, and very enthusiastic, were more likely to participate in exit polls.

For my own part, I’ll close with an estimation I gave my brother-in-law last weekend when he asked me point-blank whether I think Sen. McCain can still win this election, despite the polls. I told him that while I won’t go so far as to say he will, I definitely think he can still defy the polls and pull it out. I also said that I think the Electoral College vote margin, and definitely the popular margin in the battleground states, will be extraordinarily thin, and in the end, the outcome will resemble 2000 in that the final electoral vote distribution will be decided by election lawyers, not by hopelessly weight-addled polls.

5 thoughts on “Could the polls be wrong? It’s happened, but…

  1. curiousjoel

    Statistics in the hands of novices are the problem. It is probable fair to say that the margin of error is likely to be greater than in previous years for the reasons mentioned in this post, but taken as a whole, and weighted properly, the polls certainly can certainly show the direction of peoples’ perceptions. I think that does a good job of compiling all of the available data.

  2. Charlie Post author

    Oh, I don’t argue with the assertion that polls “certainly can show the direction of people’s perceptions” (even putting aside my own sentimental bias in favor of the effectiveness of polling).

    Given what I’ve seen this year, however, as described in this post, I can’t imagine that they do, to any credible extent.

  3. hoh

    i heard on the BBC that the polls showed gore being yr next president and look what happened!
    what ever caused bush to win via the courts – has that loop hole been filled?
    if so then the polls could be correct
    although i also heard, that when some democratic white people will not vote for a brown president when in the voting booth.
    between the 2 candidates – there’s no question who to vote for! obama he’s an intellectual who will know who to consult to make the best decisions for the majority
    ps i know naaaathing – have only got info from TV & radio so am not informed in depth & am i’m really not interested to go there 🙂 in fact i can’t wait for it to all be over so that the rest of us on the planet outside of the US is no longer subjected to the circus – the billions that are spent on all the hot air it’s embarrassing don’t you think?

    it’s only because the media is pandering to the US audiance that the rest of us are subjected to it!

    pps cape town south africa

    our ANC government is about to dismantle our scorpion police squad that looks into corruption!
    too many of the top leaders have been investigated and taken to court…power corrupts and absolute power corrupts absolutely [who said this?] oooooh dear…..the slippery slope…

  4. Pingback: Are Murtha’s days numbered? Polls say maybe. « WitSnapper | The Blog

  5. Pingback: Not to bring up polls again, but… « WitSnapper | The Blog

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s