Friday, 22 July 2011

Poll Axed

Going Down?: Labour morale, boosted by the generally positive critical reception given to its proposed Capital Gains Tax, sustained a crushing blow when the One News Colmar Brunton poll showed the party plummeting seven percentage points to its worst result in a decade. But are the opinion polls methodologically sound? And, if they're not, who benefits most from the news media's failure to correct them?

TWENTY-SEVEN PERCENT: the results from the One News Colmar Brunton Poll fell upon Labour’s hopes like a sledge-hammer. Party members and supporters who, just days before, had been feeling proud and confident stared at the numbers in blank amazement. Everything they’d experienced since last Thursday’s tax-policy launch had reassured them that the Party’s plans were being well-received by the public. And yet, there they were, those wretched numbers: the worst in a decade.

It was all so demoralising. Why go on fighting? What was the point? When, in spite of most experts hailing Labour’s Capital Gains Tax (CGT) as not only a courageous and much-needed fillip to economic development, but also as a statement of basic fairness, the voting public simply turned their backs.

Such a savage rejection must surely mean that Labour and its supporters constitute an eccentric – almost cultish – subset of the New Zealand electorate? A sort of political Amish, whose beliefs and policies appear as outlandish to “Mainstream New Zealand” as the Amish’s horse-drawn carts and buggies?

That is certainly what Labour’s opponents would like its supporters to think. Which gives rise to the very troubling question: “Are opinion polls being used, quite deliberately, to demoralise the Government’s opponents?”

Prior to the events of the past fortnight in the United Kingdom, such a question would have reeked of the political paranoia usually associated with conspiracy theorists. But, after the revelations exposing the corruption and collusion which has for decades defined the relationship between Britain’s news media and her politicians, it is a question that merits some sort of answer.

Let’s begin with Colmar Brunton. Since at least 2004 there has been persuasive academic evidence that a statistical bias in Colmar Brunton’s sampling methods causes it to consistently overstate the support of the political Right.

The company’s telephone survey is conducted Saturday to Thursday evenings (inclusive). But, by conducting its polling mostly on weeknights, Colmar Brunton’s academic critics (Rob Salmond, Keith Rankin) argue that the company is much more likely to make contact with higher income-earners – men and women whose electoral choices traditionally favour the more conservative political parties.

This statistical bias could be eliminated if Colmar Brunton’s data was weighted to overcome the preponderance of higher income earners among its respondents. But, although Colmar Brunton weights its results to match Statistics New Zealand’s population data on age, gender, household size and ethnic origin, it does not weight them to match the income-spread of New Zealand’s voters.

And neither, it appears, do the other major polling agencies: Reid Research, DigiPoll and Roy Morgan.

There is, however, one polling agency that does weight its data for income – with startling results. The latest HorizonPoll, taken shortly after the Budget (24/5/11) presents a picture of the New Zealand political scene that is radically different from the picture presented by its competitors.

In HorizonPoll’s survey, support for the “Left Bloc” (Labour, Greens, NZ First,) stands at 43 percent; with the “Right Bloc” (National, Act, United Future, Maori Party) only just behind them on 42.7 percent support.

Admittedly, there are more differences in the HorizonPoll’s methodology than simply allowing for voter income. Even so, a news media motivated by a genuine desire to obtain the most accurate description of the electorate’s mood would surely be concerned by the startling discrepancies between the HorizonPoll’s results and the findings obtained using “standard”  methodologies.

Indeed, the only plausible explanation for not being concerned is that, from the perspective of this country’s largest media corporations, the “standard methodologies” are exerting a consistent and positive influence over the public’s political perceptions and voting intentions.

By portraying the contest between Right and Left blocs as a hopelessly one-sided horse-race, the “standard methodologies” obviate the need for a more detailed examination of, and a more even-handed debate about, the political alternatives currently on offer.

But surely, if opinion polls possess the power to alter our perceptions (and they do) then the methodologies sanctioned by their sponsors must be above suspicion?

This essay was originally published in The Dominion Post, The Timaru Herald, The Taranaki Daily News, The Otago Daily Times and The Greymouth Star of Friday, 22 July 2011.

29 comments:

peteremcc said...

Horizon is a farce.

You're showing your mathematical ignorance and political desperation.

Jack Scrivano said...

Chris, I think you are clutching at straws. There is nothing very surprising about the polls. In 21st century New Zealand (and many other places too) one can no longer link policies with electoral prospects. For most people, elections have become popularity contests, not to select the most popular policies, but to select the most popular candidates. And Mr Goff is just not popular.

Chris Trotter said...

To: petermcc.

It's one thing to assert the deficiencies of your opponents, Peter, quite another to back your assertion with evidence.

If, as you claim, HorizonPoll is a "farce", tell us why. Demonstrate your mathematical knowledge. Reveal the grounds for your political complacency.


To: Jack Scrivano.

In rebuttal, Jack.

Australia. Carbon Tax. QED.

Jack Scrivano said...

In rebuttal, Jack.

Australia. Carbon Tax. QED.

A notable exception I think, Chris.

What about that nice man Tony Blair? That nice man Dave Cameron? (Well, until a couple of days ago anyway.) That nice man Barack Obama? And that nice man John Key?

Chris Trotter said...

More required, Jack?

The USA. Healthcare Reform. Stimulus Package.

Baden-Wurttemberg. Nuclear Power.

Where a major issue achieves salience in an election campaign it easily eclipses personalities.

Michael said...

"Such a savage rejection must surely mean that Labour and its supporters constitute an eccentric – almost cultish – subset of the New Zealand electorate? A sort of political Amish, whose beliefs and policies appear as outlandish to “Mainstream New Zealand” as the Amish’s horse-drawn carts and buggies?"
This is certainly how this left-leaning voter came to be voting on the right since 2005.
After some soul-searching I think I have figured out that I am economically on the left, but socially on the right. And it is the social policies of the last Labour government that forced me to vote for social rather than economic reasons. I am fundamentally opposed to asset sales, but I'd have rather live with those than some of the social crap that Labour and/or the Greens will foist upon us.

Sanctuary said...

The difference in Labour's support between the Roy-Morgan and TVNZ polls is over 6% - 6.5% in fact. Now, both polls have a margin or error of around 3.1% so the difference is just outside each polls respective margins of error. You would think such a wide variation in two polls to a point where their different results were actually outside their margins of error would be causing alarm bells to ring in the polling companies themselves, since they stand to lose financially big time if their predictions are proved to be worthless.

There was a lot of talk last election that the polls were wrong, but the final result proved them to be reasonably accurate. But three years is a long time in technology and these wide variations in the polls (together with their failure to accurately pick recent real elections, particularly in Auckland it seems) might indicate that land line polling is indeed reaching the end of its useful life.

The curious thing about the TVNZ Colmar-Brunton poll was the deliberate way it was used to frame a negative narrative around the Capital Gains Tax anouncement. As the Dimpost points out Guyon Espiner

"...blamed this squarely on their new Capital Gains Tax policy, even though it had only been leaked at the start of the polling period and not officially released..."

Yet

"...in the same poll Colmar Brunton explicitly asked voters about a Capital Gains Tax, and 43% were in favor of a hypothetical CGT, 49% opposed. So support for the tax is 16% higher than support for the Labour Party! And these numbers must have been available to Espiner when he blamed Labour’s decline on their new policy – an angle almost every other media outlet regurgitated, because Espiner is such an influential news-leader..."

A "shock" low poll that then sees a leading political editor of the state TV channel deliberately seeking to mis-represent what it might mean? Very Fox News, me thinks.

As you say Mr. Trotter, in the light of the scandal in the UK (delightfully described by Ariana Huffington as revealing "...elite Britain modeled on an organized crime family...") one wonders at the level of coercive editorialising that goes on in our media.

Tiger Mountain said...

Political polls can be predictive, suggestive, self fulfilling, accurate and inaccurate simultaneously it seems to varying degrees. But reality always intrudes eventually.

Can people really feed and educate their kids on aspirational spin? Precarious employment? Dunno, but I can’t. Give me robust collective bargaining, $15 minimum wage, restored ACC and Kiwisaver with no more ‘family silver’ fire sales.

Profit motive pollsters don’t run them for something to do. a) they get paid, b) they get a good shot at entrenching their dirty filthy tory world view. Until Colmar Brunton demonstrates otherwise I shall consider them bent.

Danyl said...

1. Horizon is a self-selecting poll, and is not scientifically valid.

2. As an online poll it's going to be even less representative than telephone based polls

3. You can sign up to Horizon as many times as you like - all you need is an email address. So you can create a bunch of yahoo or gmail addresses and then vote in the Horizon poll multiple times.

Graeme Edgeler said...

That is not "the only plausible explanation for not being concerned".

Basically, I'd suggest status quo bias. I very much believe that TVNZ likes to focus on the Colmar Brunton poll because:

1. that's what they've always done;
2. they pay a heck of a lot of money for it;
3. if they started to wonder whether it was right, they'd wonder if they were wasting money for many years.

This thought is too uncomfortable.

[Also it's a TNVZ poll, so they get to run it first.]

This sort of thinking is common in large organisations - stagnation exists because to question perceived wisdom and change your approach is just too hard.

It's been the same with TV ratings: advertisers pay for TV ads based on audience share and demographic. The people meters work out who watches what. The people meter people once changed their methodology and the results were so different that after a couple of weeks the ad industry told them to change back (even if the answers were wrong): the implied changes in the prices of ads just wouldn't work out for anyone.

Victor said...

Chris

I'm open to the suggestion that there is deliberate bias in the polls.

The question, though, is when does it date from?

In 2002, the polls, from memory, correctly forecast that National was heading for the rocks ......and that's what happened.

jh said...

I think you have answered some of your own questions about labour:
http://bowalleyroad.blogspot.com/2010/07/arrogant-left.html

It used to be: "Mr Rowlding is for the workers and Mr Doon is for business" (as someone put it). Now however the workers are the rednecks and labour represents non workers, gays, lesbians and tangatawhenua.
If labour are going to get on the CGT then they are a little late if this comes as part of a "keep house/land prices down" package. Labour has been in boots and all encouraging an international market in property (eg) "Foreigners don't take it with them!" and Labours Third World Solution (Gareth Morgan). The truth is we have the second highest level of immigration in the OECD and the Savings Working Group has pointed out that when we have an influx of migrants house prices rise and take a long time to fall. The SWG also point out that incomes haven't increased due to immigration as we have to provide infrastructure (tunnels under the harbour etc). Not that immigrants aren't necessarily good people. some are, some aren't but the suspicion is that the realestate industry is importing people with bulging pockets to buy houses.
Those who follow this subject will know about the efforts of the libertarian right to establish zoning as the cause of all our housing wows ("greenies locking up the land supply"). There is some truth to this but it leaves open the question of development levies for new infrastructure, as in do they bring another Benmore with them. Keeping up with Australia aside, a house and land (the Quarter acre Pavlova Paradise .. As Austin Mitchel called us) are arguably more important than the X-box or rubber thingy from the Warehouse.

On another note this poll may be a compared to the reaction to Labours Black Budget, which a TV program about Rogernomics told us was the right thing to do "in hind site" (but governments learn that you don't do unpopular things).

Anonymous said...

The question is whether the polls are produced by and reported by folk who want to use them to convince people that one cause is sensible and the others are a waste of time, or whether the polls are produced and reported by folk who always want to be seen to be backing winners.

that is - do TVNZ and C-B want to eliminate any chance of the Nats losing, or do they want to know exactly who is going to win, and then give them a nudge to help them along the way?

the Murdoch papers always backed the election winner not because they back them which makes people vote for them, but because the analyse the polls to see who will win, and then declare slightly ahead of the rest that they are supporting the front runner.

of course, no one here is foolish enough to assume that the polls are neutral and are reported in a neutral way - you only have to look at the 'special questions' which get commissioned from time to time to see where the real agenda lies behind them.

Anonymous said...

Look, guys, you're trying too hard here, the issue is really simple.
Phil Goff just needs to stop slipping in that silly forced simpering smile every time he's interviewed for tv. It makes him look ineffectual and insincere. Whether he is those things or not is irrelevant.
Its all about image today. Lose the goofy grin and you'll get back in.

Michael said...

Why does a positive poll result for National favour National politically? Are we all such copycats that we always vote with the majority? If Labour thinks that with the CGT they have the best policy for most New Zealanders, should they abandon it if the polls say it is unpopular (it is far from certain they do)? Personally I can say that polls have no effect at all on what I think, and little on how I vote. But Labour may well be unpopular, even if the CGT is recognized as good. The trouble with unpleasant medicine is that it has to be competently administered as part of a sound package. Otherwise it will do no good, and if people think that is the case, they may just vote for whatever personal gain they can get while the economy remains afloat.

Brendan said...

I doubt any readers of this column are influenced by the Polls. They may be upset by them, but not influenced.

The real question is who takes the time to respond to these pollsters? Most of my conservative business friends wouldn't give them the time of day. In fact, my Green and Labour friends wouldn't bother either. Many of the younger 'green' voters don't have landlines, just mobile phones so I'm guessing they are excluded?

It's a bit paranoid to suggest the polls are biased one way or the other. JH (above) hit the nail on the head. Labour has disenfranchised its traditional supporters, choosing instead to align themselves with vocal grievance groups and disadvantaged minorities, gays, beneficiaries, pacifica etc.

I know of retired Labour supporters who abandoned Labour for the first time at the last election. They simply refused to vote. Why, because they were Labour conservatives, just like Michael who responded above. They couldn't stomach the social engineering, and couldn't identify with Labour's new constituency.

Labour MPs were no longer 'like them', and were in parliament to promote 'alien' causes they could not identify with.

I sometimes imagine Phil Goff, who I suspect is actually a decent bloke, casting his eye over caucus and praying they don't win the next election!

The public have woken up to Labour's new identity as an interferring aging drag queen, and for the most part they don't like it. Unless they get back to their conservative socialist roots, there may come a day when they look back nostalgically at 27% popularity.

Anonymous said...

lets not forget that Labours internal polling gives the same results.

The fact that you even try to give any credibility to Horizon polls shows the desperation that the left is spiralling in to.

clearly in the analysis of the final polls that each of the various polling companies, that you malign in this blog, took before the election came very close to the election result.

your post displays the usual reaction that lefties give when polls start going wrong for you... "rogue polls".... accept it....the polls suck for the left. FACT. (not some media conspiracy to help the right retain power.... i notice there was no media conspiracy to keep the left in power when the polls were in favour of Labour).

this post, Chris, is nothing more than grasping at straws.

as for the person comparing the Roy morgan poll to the one news poll....look at the dates they were taken to see why there is a difference...

Andy C said...

Horizon has been well and truely debunked. Honestly, how many "ordinary Kiwis" sign up to a polling company. If you sign up now you could win an IPad2.
If you want to go down this path may I also recommend the TradeMe message boards.

Anonymous said...

Why not just accept that the voters are stupid, Chris? After all, most of them would not be able to explain what a capital gains tax is. What percentage of the population believe in astrology, psychics and the like? How many of them do not believe in the theory of evolution?

In the end, our politicians are incompetent nincompoops because incompetent nincompoops appeal to a sufficient portion of the electorate.

But there's more. Ours, just like elites everywhere, are completely clueless on what to do about the economic situation. The electorate would like to pretend that we can return to the way things were. Hence, whoever promises that most credibly will win the election. A capital gains tax? Too little too late.

The real story of the election will be the low turnout.

Adze said...

How does the latest Horizon poll figure into your calculations?

MPledger said...

Polls taken for news agencies are always going to be a little suspect - the news agency wants headlines to sell advertising - a non-newsworthy polls doesn't get them revenue.

In the states there's a dodgy polling company. Between elections it heavily favours Republicans but as the election approaches it falls more in line with the other polls. People suspect that in the polls between elections it polls more heavily in Republican favoured areas but less so when the election comes up.

It's one of those pieces of wisdom that a tight race means more votes are cast. By misrepresenting the tightness of the race, less votes gets cast and that tends to favour the leaders in the polls.

Horizon has a biased sample of the New Zealand population to draw there polling subsample from. BUT the phone polls are drawing biased samples too (not at homes, no landlines households miss out). And because they don't adjust for household size this can introduce another level of bias - the one adult divorced businessman has views equal in weight to one adult from a 5 adult Polynesian household whereas the former household casts one vote and the latter 5. This typically biases the polls towards the right.

Chris Trotter said...

To: Adze.

The latest HorizonPoll (after dropping out the Choose-Not-to-Votes and the Don't-Knows) has the Right Bloc on 48.1 percent and the Left Bloc (in which I include Mana) on 50.1 percent.

And, yes, I know that the HorizonPoll's reservoir of respondnets is "self-selecting", but I'm assured that the company's weighting formulae allow for that.

Besides, as MPLedger notes above, the methodological shortcomings of the other polling agencies expose them to equally searching criticisms.

I guess it comes down to my own gut feeling that the gap between the Right and Left blocs is nowhere near as wide as Colmar Brunton and Reid Research would have us believe. So, when measured against my gut (which may be large but is not in the least scientific) the HorizonPoll result feels the more credible. (Ditto for their findings on the CGT.)

And here's a question to end with: "Why, when they had all the results available to them, did the Sunday Star-Times not print them?"

Their front-page story was incredibly hard to follow and gave no detailed break-down of the individual party results.

Now, why in the heck would they do that? Unless they were unwilling to let their reader see too clearly how close this election could end up becoming?

Those individual party results are as follows:

Act 4.8; Greens 10.2; Progressive 0.6; Labour 27.5; Mana 1.9; Maori 0.7; National 37.7; NZ First 6.5; United Future 1.1; Others 1.0; Not Voting 0.7; Don't Know 7.6

Cheers.

Don Franks said...

As the polls roll on so does real life.

Groveling in Washington before US president Obama NZ Prime minister John Key used the mass murders in Norway to ingratiate himself, stating:

''If it is an act of global terrorism then I think what it shows is that no country, large or small, is immune from that risk and that's why New Zealand plays its part in Afghanistan as we try and join others like the United States to make the world a safer place.''

Norwegian police apprehended right wing christian Norwegian national Anders Behring Breivik, for the shooting massacre and bombing.

Labour leader Phil Goff said Key should check his facts before making political claims.

''It's obvious the prime minister was keen to justify our presence in Afghanistan and made an assumption that has since been proven to be absolutely incorrect,'' he said.

The comments were ''premature'' and ''unfortunate''.

John Key is a good little servant of imperialism but in relation to Afghanistan he can claim no monopoly of ''premature'' and ''unfortunate''.

Phil Goff was a central member of the Labour government who dispatched troops to Afghanistan within hours of the US invasion.

The killings of innocent people in Norway are a terrible tragedy.

As are the killings of innocent people in Afghanistan by New Zealand troops, courtesy the New Zealand Labour/Alliance and National governments.

Anonymous said...

"Why, when they had all the results available to them, did the Sunday Star-Times not print them?"

Exact-a-freakin-mundo. It's not the journalism, it's how they're journeying. Relentlessly rightward.

And here's another question: "Why, when even the Prime Minister himself has rejected outright the poisonous notion of special privilege for Maori, is the leader of a minor party that has never come near power and influence save as the result of a blatant electoral rort, given at least three prime time interviews on national television to promote those same discredited and rejected views?"

I hope some keen and dedicated researcher is keeping assiduous and detailed records of the reporting in our major dailies and news channels from here to November. Prediction: positive NACT footage and column-inches by around three-to-one.

Martin said...

Re. the capital gains tax on polling. Many people apparently forget that only a few years ago, National was considering a capital gains tax. If it were to do so again, no doubt the same people who think it is a terrible idea coming from Labour would think it is a brilliant idea coming from National. Polls cannot be trusted in this country - they reflect the wishes of vested interest groups to influence people into going with the majority because winners are cool, and losers are.....losers!

Anonymous said...

Act 4.8; Greens 10.2; Progressive 0.6; Labour 27.5; Mana 1.9; Maori 0.7; National 37.7; NZ First 6.5; United Future 1.1; Others 1.0; Not Voting 0.7; Don't Know 7.6

now those really are interesting numbers, when you add in Banks and Dunne to skew them

Anonymous said...

Case in point - the 2002 polling vs actual results posted by Jessica Mutch

http://tvnz.co.nz/politics-news/jessica-mutch-polls-could-get-worse-labour-4328514

suggesting a massive over estimation of the main parties in the months leading up the the actual poll which only narrow at the last possible moment (an still widly out in favour of the at-home-after-work respondent.

rather than a pro Nat bias though, to me it looks like simply poor polling favouring the party which the commoon wisdom expects to win - i.e. the murdoch approach to backing a winner

markus said...

Anonymous (9:17 PM) argues that the final 2008 polls were relatively close to the actual Election result and so couldn't have been skewed in the political Right's favour.

In fact, it's possible for (a) the 2008 polls to have overstated (perhaps quite significantly) Nat/Right-Bloc support and understated Labour/Left-Bloc support AND (b) co-incided relatively closely with the final Election result.

Consider the following two points, the first hypothetical, the second fact:

(1) Hypothetical: Imagine, for arguments sake, that (due to the kind of methodological problems cited by Chris and others) the 2008 polls exaggerated Nat/Right support by, say, 4 percentage points and understated Labour/Left support by the same margin. So, while most of these final polls had the Right-Bloc around 51% and the Left-Bloc around 46%, the real figures (under this scenario) were Right 47% / Left 50%.

(2) Fact: As a number of commentators pointed-out in the run-up to the last Election, almost all print and broadcast journalists were placing total analytical emphasis on the support-gap between the two main parties in their reports of various poll results. FPP-style.

Tracy Watkins in the Dominion Post is a good example. Under hyperbolic headlines like "Labour is Dog-Tucker" and "Nats set for Landslide", Watkins focussed entirely on the 10, 12, 16, 18 point gap between National and Labour. The only time she ever referred to the (much narrower) support gap between the Left/Right Blocs as a whole was to casually dismiss its importance - suggesting this was nothing more than Labour spin. Extraordinarily disingenuous of her. (The fact that the Fairfax Media-Nielsen polls almost always recorded a much larger gap than the other polling companies simply amplified the problem).

As I say, this happened throughout the print and broadcast media, giving the vast majority of voters the impression that the Centre-Left had a far greater gap to bridge in order to hold on to Office than was actually the case.

All of which has the capacity to alter voter behaviour. As Watkins herself suggested a week out from Election Day, if Labour/Left voters believe there's no chance of a win, many will simply move into Non-Voting.

Hence, you could have a situation where the polls are, say, 4, 5, 6 percentage points out but end-up spot-on because (a) the very fact that they overstate Nat/Right support, amplified by (b) the erroneous FPP-style analysis of leading political journalists, changes voter behaviour - encouraging a section of erstwhile Labour/Left voters (particularly the young and the lower-income) to stay home on polling day.

markus said...

Incidently, Chris, I could give you a number of examples of highly misleading Opinion Poll reportage in The Dominion Post over the years.

Two of the examples I'm thinking of follow a roughly similar pattern: (1) A highly misleading headline, followed by (2) an equally deceptive slant in the first 7 or 8 paragraphs and then (3) in paragraph 8 or 9 (at least half-way through the article), an absolutely crucial fact or crucial piece of data that entirely undermines both the headline and the preceding paragraphs finally gets a mention.

Clearly, journalists, sub-editors, and editorial teams are well aware that many (perhaps most) readers only read the headline and skim the first few paragraphs. In these two cases, those who do will have gone away with entirely the wrong impression.

And, by a "quite astonishing co-incidence", the deception in all of the cases I'm thinking of just happens to work in the Political Right's favour.

(I'll also add that the Fairfax-Media polls have a long history of over-stating Nat support/under-stating Labour support. At the 08 Election, National (45%) beat Labour (34%) by 11 percentage points. Yet the final Fairfax Media-Nielsen poll put the Nats a whopping 18 points ahead of Labour. So, I'm more than a little sceptical about the latest Fairfax figures of National 56%/Labour 29%. Which isn't, of course, to deny that those horrendous Tories are well ahead).