Why I Despise Public Polling

I haven't posted much in the past year, but recent developments have gotten my attention.

I have long regarded public polling with more than a measure of disdain. My distaste is primarily with the flawed sampling and methodology utilized by the polling organizations – antiquated screens, RDD, self-identified partisanship and likelihood of voting with little or no voter file validation - and the trumpeting of facile analysis without fully releasing the underlying data.

Perhaps the worst part of public polling is the complete inability of the talking heads from the polling world to avoid hyperbole or to speak in metaphor-free English, compounding the weakness of their data and diluting the quality of the public discourse.

The Pace University polling efforts struck me as a potentially bright light, with greater disclosure of data, a better sampling methodology and comprehensive and voluminous analysis. But alas, they are gone too soon.

So what triggers my comments today? The release of yet more crappy data that media types will naturally regurgitate for weeks to come.

In the past ten days there have been two public polls (Siena in NY and Quinnipiac in NJ) that have shown Giuliani with a lead over Obama in both states and competitive with Clinton in NY and leading her in NJ.

The results are not surprising, given the relatively high name ID and voter familiarity with the Giuliani and Clinton names and brands, but what do they mean in practical terms for the various candidates?

Well, practically speaking they shouldn't mean anything, but the results could have significant impact on the GOP nominating fight. The findings echo and validate the strategic foundation of the Giuliani candidacy: that he can place New York and New Jersey in play. Part of his donor pitch is that he claims he is the ONLY Republican candidate who can credibly engage in Democratic territory.

If a Republican were to win NY and NJ, the Democrats would be hard pressed to replace those 46 Electoral Votes. The Democrat would have to win Colorado, Florida AND Ohio, and at least two out of three among Iowa, Nevada and New Mexico (while holding New Hampshire and Wisconsin) - an extremely tall order.

Many of those of us who know Giuliani (and yes, we know what familiarity breeds) cannot fathom how the GOP could nominate him, but we also know that they want to win, and know how to win. Electability is a major factor for their activists.

It's a major factor for Democrats as well (perhaps even more so). Other recent public polling (notably the recent Gallup poll) has shown Clinton's favorables sliding (broad-based, they called it), and Obama and Edwards generally frozen in place. The only real movement or curiousity on the Democratic side seems to be with Vice President Gore, who has continued to express a non-Shermanesque disinterest in running.

If Clinton continues to slide, she can’t win. If Obama can't win in the northeast in November, he can’t win, and Edwards remains frozen in a distant third (because someone with his name ID should be doing better) he can’t win the nomination, much less in November. So who emerges? Dodd? Gore?

But Gore isn't running. Why are public pollsters even including a candidate who is not running? How is that even remotely scientific? Are they doing it out of prurience or some other form of curiosity?

The the flawed and potentially false question still gets raised by this "research" - how can a Democrat win if they lose the northeast?

Now I don't think for a second that will end up being the case, but that is potentially the way it will be covered - and sometimes that is all that matters. Maybe that is what the pollsters want - a competitive northeast so that THEY can be relevant.

Most pollsters would say that they only measure opinion. My point is that they should base that measurement on votes or more scientific methods. Since they don't and the media will slavishly cover this so-called public data, all that these polls do is skew and eventually cement public opinion on false data points. The self-fulfilling nature of these polls undercuts the public discourse and public interests the polling institutes purport to represent and serve.

The proof is found in the real source of my pique, the responses to Q2 of the Q-Poll - which show that a majority of respondents are not firm in their support and "might change their minds" about who they are supporting for the Democratic nomination. If that’s the case, then why release the numbers at all? And why no mention of this softness or uncertainty in the press release of the poll?

Pollsters defend this kind of early/non-predictive polling and their craft generally by saying "it is only a snapshot of public opinion at this particular time." The problem with that “defense” is that their data show (at least for the Q-poll) it is a "snapshot" of uncertainty. All it tells us is something that we already know - the race is unformed and volatile, but the conclusions they push are entirely different.

We don't need a poll to tell us the race remains undecided, and we certainly don’t need crappy conclusions, particularly if the spin on this poll doesn’t match the reality of the mood of the electorate.

We live in a rapidly changing and increasingly segmented world. Unless and until public polling institutions reform and refine their methods and motivations, they really aren't worth much. The average voter will probably get more substantive value out of a conversation with their barber or bartender than from this so-called "data." I think I am going to get a haircut.

And that's Why I Despise Public Polling.



Submitted by Anonymous (not verified) on Thu, 04/19/2007 - 3:17pm.

Wow!

Look who's returned from the dead1

Who's next?

Gumbs? Gatey? Beadie Markowitz? Judith Memblatt?


Submitted by Anonymous (not verified) on Thu, 04/19/2007 - 3:30pm.

Polling is a business.  These places release the numbers because they need to sell their polling services.  All of the polling orgnizations want to get paid, and the media outlets aren't going to pay them, sponsor their work, for numbers that don't get released.

Lead Dog, your concerns are valid, but unfortunately with that high minded attitude, you wouldn't stay in business as a pollster very long.  These guys HAVE to churn out the numbers to pay the bills, and to be visible in the media so as to attract new clients.   


Submitted by Larry Littlefield on Thu, 04/19/2007 - 6:16pm.
Back then, the problem was that many people still didn't have phones. Now, it is that young people may not have landlines and fixed addresses. I have no idea how I would do a valid random sample today, other than door to door which no one can afford.

By the way, "segmenting" as you would wish costs truly huge money. To even get a number that can be quoted without it being a joke, you need 50 responses. And it has to be a random sample. So how big a sample do you need to get at least 50 (and preferably a few hundred) of every subgroup you want, if you don't know what subgroup each phone number is in ahead of time so you can do a segmented sample?

In any event, I find all coverage of the 2008 election at this data inane. All about money, and who said what about whom, like middle school. It makes me think our method of selecting leaders screens out anyone worth voting for.

Submitted by whillice (not verified) on Fri, 04/20/2007 - 5:24pm.

I love polls. More than almost anything. But people get suckered in to what they "mean." Reposted from a comment i made over at myDD.com:

"And considering the average journalist reads most polls about as well as I do a schematic to build a nuclear submarine, you get garbage in, garbage out. Less effective poll + Less informed analyst = "HRC and Giuliani may as well be anointed the nominees right now"

If you want detailed analysis, read the mystery pollster blog at pollster.com or Public Opinion Quarterly. But these other guys in the media. . . .

It's not so much the purveyor's fault, either. If they offered a detailed analyis using regressions, multiple variables, and started tossing out terms like "standard deviation" and "multicolinearity" no one would ever read a press release from them ever again.

And so you get the version dumbed down enough that even Bill Schneider can tell you what the poll means. And not like it's in the nature of MSM reporters to ever marginalize their importance or ask if what they're presenting is really "the whole truth". . .

There's a story told about the inventor of the IQ Test, that when asked what specifically his test measures, he respnded "Someone's Intelligence Quotient" to which someone said "And what is Intelligence Quotient?" "It's what my test measures."

The same is true of polls, especially when further biased by flawed or overly broad interpretation.


Submitted by regime rapide (not verified) on Thu, 01/12/2012 - 10:34am.
This web site is my intake, very fantastic layout and Perfect content.

Post new comment

The content of this field is kept private and will not be shown publicly.