Issues & Ideas | 17th March 2015

Lies, damned lies, and statistics: The margin of error on polls isn’t always 3 points

The error for Labour and the Tories in a typical British poll is more like 4 per cent, but it’s much lower for small parties.

Photo: Getty

By

By now anyone paying attention to ‘the most important election in a generation’ is familiar with the idea that opinion polls come with warning labels. Specifically, they are delivered by their skilled creators with the caveat that each lone poll could be about 3 percentage points (not per cent) adrift.

Somewhere in the back of your subconscious you might also remember hearing that 19 of every 20 poll results ought to fall within this ±3 percentage point margin of error. (So one in 20 is less accurate than this.)

There is good news concerning this received wisdom. If you are a normally busy person with commitments that extend well beyond politics; if you check election forecasts less than once a day; or if you simply have cheese under the grill that might be about to burn, you can stop reading after one more sentence.

The health warning of ±3 percentage points for any individual poll conducted by a reputable pollster leaves you in pretty good shape to make judgements about the election and light dinner party conversation about opinion polls.

If you are a normally busy person you can stop reading after one more sentence.

If you stick to the ‘margin of error rule’ it helps you deal with all manner of headline hyperbole. “Tonight Labour lead by 1” – we don’t know that (The Sun tweets this kind of thing with each new nightly poll). “Labour close the gap in Scotland after a gain of two points” – that we also don’t know.

If a single poll gives Labour a two point lead or a two point gain then statistically speaking they have no lead and we know of no gain. When you read these headlines the editor who wrote them is relaying something about a sample of less than 1000 people whilst tricking you into assuming they reflect the electorate as a whole.

Quite simply, the margin of error inoculates you against this kind of hopeful thinking. It forces you to consider the trend not the outlier. It makes you worship at the altar of polling averages, especially ones which aggregate many results from different companies (as May2015’s does – Ed.).

*

Standard margin of error (henceforth ‘MoE’) is therefore a good rule of thumb, but delving further we discover that there is more to MoE than meets the eye. The version of MoE that has become ubiquitous, like so many of the best clichés, is actually an American import.

The ‘3 percentage point rule’ is designed to describe US elections with only two candidates on the ballot paper. The rule describes the worst case scenario for unbiased random sampling error (the source of MoE), when both candidates are on 50 per cent support in the poll.

But British elections now look nothing like these two-horsed US races. Both the Tories and Labour can only dream of being at 50 per cent in national polls, and smaller parties abound.

The traditional ‘3 percentage point rule’ is, like many of the best clichés, an American import.

Unlike the 50/50 scenario, parties in Britain receive different shares of the vote and so one MoE does not fit all party vote shares. There is no overarching MoE for all parties in one poll, let alone all parties in all polls.

To show what is going on let’s take a very typical Lord Ashcroft poll from last month: Lab 31, Con 30, UKIP 16, LD 9.

As we adjust for the fact that no party is anywhere near 50 per cent, the MoE actually goes down from the standard ‘3 point rule’. The Labour and Conservative numbers now come with a MoE of 2.8 points, meaning that on average for 19 polls out of 20 the ‘true’ figure for Labour’s vote share will lie between 28.2 per cent and 33.8 per cent.*

“That’s better than 3 per cent!” I hear you cry. Exactly, and well worth knowing. Things get even more interesting as we get down to smaller parties. The Lib Dems’ 9 per cent showing comes with just a 0.6 point MoE, and UKIP’s 16 per cent with one of 2.3 points.

All the calculations up to this point assume a sample size of 1,000 people. In fact, in polls a few months out from the election there are still plenty of undecided voters who get removed to generate the headline figures.

The sample size for the final figures in Monday’s Ashcroft poll were a fairly typical in being based on less than 600 respondents. Given this smaller sample size the headline figures for the two largest parties have a MoE of 4 points.

The margin of error is the difference between Labour’s worst ever election and their 2005 victory.

Even if we want to be confident that only 1 in 20 polls will be outside of our MoE in this poll, that means the the actual Labour share of the vote could range from 27 to 35.

This margin is starting to look quite large in such a tight contest, especially when you consider we are still only talking about errors that arise from random sampling. (We haven’t even considered that online pollsters need to use special measures to calculate their sampling errors.)

There is nothing magical about working out our MoE for 19 out of 20 polls, otherwise known as a ‘95% confidence interval’. If we want to be really certain, say 99 per cent certain, then our MoE for this Ashcroft poll is now 5 points for the two main parties. That is we can only be really certain Labour’s share of support lies somewhere between 26 and 36 per cent.

That would be the difference between their worst ever post-war showing and their comfortable 2005 re-election.

You can pick your own confidence level and calculate your preferred MoE for each party. By doing so you will mark yourself out as an elite poll watcher, should you wish to be such a thing. But by simply following the 3 point rule of thumb, you’ll at least be giving the underlying statistics more respect than most journalists.

* The maths to get the 19 out of 20 confidence level is straightforward: p is the proportion of the vote reported for a party and n is the sample size. MoE = 1.96 * (sqrt((p*(1-p))/n)). If you want the 99 out of 100 level of confidence then replace 1.96 with 2.56.

Dr Tom Lubbock is a lecturer in Politics at Regent’s Park College, University of Oxford.