This election year, NZ voters should beware of reading too much into the political polls

With a new prime minister sworn in and a cabinet reshuffle imminent, it’s no exaggeration to say the election year has begun with a bang. Already the punditry and speculation are ramping up, with anticipation building for the first opinion polls.

There will be more polls to come, of course, but a word of caution is in order: don’t treat them as gospel, and try not to let them become self-fulfilling prophecies. At this point, we can’t predict who will form New Zealand’s next government, and it could yet be a tight race.

Furthermore, political polling has not had a stellar record in recent times. Former prime minister Jim Bolger’s famous remark from 1993, after he didn’t get the election majority he expected, still resonates: “Bugger the polls.”

It’s not just a local phenomenon, either. The results of the Brexit referendum and the Trump–Clinton presidential contest in 2016, and the 2019 Australian election, were all out of line with preceding opinion polls.

In 2020, the US presidential polls were off by about four percentage points. And the 2022 US midterm elections didn’t produce the landslide (or “red tsunami”) many Republicans had predicted.

The 2020 election miss

It’s a similar story in Aotearoa New Zealand. In 2020, the polls immediately prior to the election overestimated the National vote and underestimated Labour’s.

Taking the averages of the results of all six polls published during the month before election day, National emerged on 30.9% and Labour on 47.2%. In the final three polls during the two weeks when advance voting was open, the averages were National 31.4% and Labour 46.3%.

The gap was closing and Labour would land on about 46%, or so it seemed. As Labour’s trend in the polls since mid-2020 was already downward, 45% looked plausible. But predictions based on the opinion polls were significantly wrong. Labour’s election result was 50%, National’s only 25.6%.

The polls in the final fortnight were overestimating National by an average of 5.8 percentage points. They were underestimating Labour by 3.7 points. The Green and Māori parties were also underestimated (1.1 and 0.7 points, respectively).

There were even bigger failures in polls showing Green candidate Chlöe Swarbrick running third in Auckland Central with about 25% of the vote. Instead, she got 35% and won the seat.

Statistics 101

The opinion polls and the election – the only poll that counts, as the saying goes – use different methods with different samples. They’re intended for different purposes, and hence their results will differ, too.

An opinion poll is a snapshot of a sample of potential voters. By the time it’s published, it’s already in the past. Surveys normally ask which party you’d vote for if the election were held tomorrow. But you may change your mind by the time you actually vote, if you vote at all.

Read more: Here's how to make opinion polls more representative and honest

Furthermore, surveys are prone to random error. So, no matter how scientifically rigorous, they only estimate – and can’t replicate – the relevant population. It’s in the interests of the polling companies to be accurate, of course, especially when close to an election. But we need to read their results critically.

Samples are normally about 1,000 people, and pollsters try to ensure they closely resemble the demographic makeup (ideally by age, gender, ethnicity, education and location) of the eligible population, giving voters of all kinds an equal voice.

Post-survey weighting boosts results from social groups with low response rates. The proportion of the population that holds a specified preference is estimated, and all estimates are subject to variance. This is expressed as a margin of error, which is normally plus or minus three percentage points.

Read more: What is a margin of error? This statistical tool can help you understand vaccine trials and political polling

The margin of error is the range in which the pollster bets the “true” results should probably fall, with the true figures being outside that range only 5% of the time. In other words, pollsters are 95% confident the actual results will fall within that range. It’s only a statistical estimate.

But the quoted margin of error doesn’t apply evenly. If a given party is polling at 50%, then the quoted margin of error applies. If a party is polling higher or lower, then the margin of error narrows – the further you get from 50%, the narrower the margin of error.

Beyond the margin of error

Another concern is whether respondents will give honest answers. Some may be unwilling to reveal their voting intentions or they’ll wilfully mislead the poll.

And often a large proportion of a sample doesn’t know yet whether they’ll actually vote, or for whom they’ll vote. Responsible pollsters will report the percentage of “don’t know” responses.

But the conservative bias in the pre-election 2020 opinion polls was systematically outside of the margins of error, and hence not due only to random variation.

Apparently, pollsters didn’t obtain samples that resembled the population that actually voted. It looks like younger left-wing voters were especially hard to reach or unwilling to participate. Or their election turnout may have been underestimated.

Polling companies are now using online panels to help correct such biases. We’ll have to wait for the next election’s results to judge how it’s working.

Read more: Chris Hipkins becomes NZ’s new prime minister – there are two ways it can go from here

Reading the tea leaves

A series of opinion polls can reveal trends and thus serve a purpose as public information. But they’re not suited for forecasting. One result taken out of context may be misleading, so it’s disappointing when major news organisations over-hype polls.

When party-vote percentages get converted into numbers of seats, journalists are reading tea leaves and not reporting news. Meanwhile, the market research firms are getting massive publicity.

Accurate or not, opinion poll results can have self-fulfilling or “bandwagon” effects on people’s voting behaviour. People might want to back a winner, or not waste their vote on a party that’s polling below 5%. Or some might vote for a party other than their favourite, with an eye to post-electoral negotiations.

Perhaps the best advice for voters is this: when deciding which party to vote for, try not to think about the polls. And poll-watchers should prepare for surprises on election night.