CONSERVATIVE HOME 23.11.09
Beware of Rogue Polls
Posted by Stephan Shakespeare
There’s been a flurry of excitement about an Ipsos Mori poll published on Sunday: a single voting-intention figure from a sample of 449 has led to absurd speculation about a hung parliament. Fortunately this morning some people seem to be regaining their senese of proportion.
The BBC’s Nick Robinson has wisely urged people not to get too excited, reminding them that polls are most fun when most surprising – and unfortunately, surprising usually means ‘rogue’. Remember Twyman’s law: “Any piece of data or evidence that looks interesting or unusual is probably wrong”.
That this particular poll should produce some outlandish figures is even less of a surprise. When pollsters talk about a margin of error of plus or minus 3% it is based on a sample of 1000 people. However, the voting intention figures in this poll weren't based on 1000 people. Firstly only 799 people gave a voting intention. On top of that MORI excluded all of those who were not 10/10 certain they would vote, leaving only 449 people - the margin of error on that is plus or minus 4.6%.
A second issue is that of weighting. Almost every political pollster in the UK these days politically weights their sample to make sure it is properly representative of Britain, not just in simple demographics like age and gender, but also in terms of people's political views. At YouGov we use party identification for this, and most other companies use how people voted at the last election. MORI do not use any political weighting at all. They've got their own reasons for this, and perhaps they’re right and we’re wrong, but the effect is that even the political make up of their samples can change from month to month, further adding to the volatility.
I’m making no negative comment here about the methodology of Ipsos Mori or any other competitor-pollster, just pointing out that which none of us would deny: ‘surprises’ such as those delivered on Sunday are an inevitable effect of the numbers quoted above.
Posts
