Post-Mortem Ontario: Who's right?

Hi,

How did the pollsters fare in this election? There are a number of different ways to look at the situation. The first is to see whether the prediction that I made using all the polls' estimates was accurate.  Here is the graph with the election results:

 It is clear from this graph that the Liberals were globally well predicted by the polls but that, contrary to usual, the PC was overestimated and the NDP underestimated. This a rather rare situation. In the last federal election of 2011, the polls had underestimated the PC by more than 4 points in Ontario.

Who's right? 

There is a debate around methods and pollsters. Who fared better? There are known ways to deal with this. One is to calculate the difference between the results and the estimates from the polls. The relative error (results minus estimates) tells us whether a party tended to be over or underestimated by the polls. A positive result means that the party was underestimated by the polls. The absolute error infoorms on the level of the difference between the results and the estimates. The total absolute error gives the synthetic information on the global level of error. It is computed as the sum of the absolute error for the three main parties. I present the results grouped by method (IVR vs Web) for the polls conducted from June 5 in the table at the end of the message.

What the table and my analyses show is:

• IVR polls got the Liberals right (mean error of -0.34) but they overestimated the PC by 3.6 points and underestimated the NDP by 5.1 points on average.
• Web polls underestimated the Liberal party by 3.7 points on average and overestimated both the PC (2 points) and the NDP (2.3 points).
• The difference between the estimates of IVR polls and Web polls is significant for the Liberals and the NDP. We are still in a situation where Web polls tend to overestimate the NDP.
• As for the absolute difference, it is rather large -- between 2 and 5 points on average. There is no significant difference between the IVR and the WEB polls.
• If we look at the total absolute error, it is 10.8 points on average for the IVR polls and 9.6 for the WEB polls. Again, there is no significant difference between these two averages.

A total absolute error of 10.8 is rather high. It is however lower than what has been seen in BC2013 and Alberta 2012.

In conclusion,

The surprise with this election is the global underestimation of the NDP and the overestimation of the PC. I do not see how to explain this. The only expected bias, i.e. an overestimation of the NDP vote and an underestimation of the PC vote by the WEB surveys indeed happened. The huge underestimation for the NDP vote by the IVR polls will have to be analyzed by the pollsters. In addition, the "likely voter models" did not seem to perform adequately. It is good that pollsters who used these models indeed published also their figures for eligible voters. Analyst and voters could see that polls allowed for different estimates according to different hypotheses about who is participating in the election. However, the likely voter models do not seem to be "the solution" to bias in polls' estimates. Pollsters who use IVR should perhaps think about introducing cell phone numbers in their samples and see whether it improves their estimates.

The final point to notice -- it has been noticed by many analysts -- is the substantial variation between estimates. This means that the methods right now are not at the level of reliability that we would like to see them.

Error of the polls - 2014 Ontario Election
Method	Pollster	mid-date of field period	error LIB	error PC	error NPD	Abs. error LIB	Abs. error PC	Abs. error NDP	Total absolute error
IVR	Forum	06-JUN-2014	-0.3	-5.8	6.7	0.3	5.8	6.7	12.8
	Ekos	07-JUN-2014	2.8	-4.7	6.1	2.8	4.7	6.1	13.6
	Forum	10-JUN-2014	-3.3	-3.8	4.7	3.3	3.8	4.7	11.8
	Ekos	11-JUN-2014	1.4	-0.1	4.5	1.4	0.1	4.5	6.0
	Forum	11-JUN-2014	-2.3	-3.8	3.7	2.3	3.8	3.7	9.8
	Mean		-0.34	-3.64	5.14	2.02	3.64	5.14	10.80
	Median		-0.30	-3.80	4.70	2.30	3.80	4.70	11.80
	Std. error of mean		1.13	0.96	0.55	0.53	0.96	0.55	1.36
WEB	Ipsos	05-JUN-2014	3.7	-3.8	-2.3	3.7	3.8	2.3	9.8
	Abacus	06-JUN-2014	4.7	0.2	-4.3	4.7	0.2	4.3	9.2
	Angus Reid	09-JUN-2014	2.7	-0.8	-2.3	2.7	0.8	2.3	5.8
	Ipsos	09-JUN-2014	5.7	0.2	-6.3	5.7	0.2	6.3	12.2
	Leger	09-JUN-2014	1.7	-5.8	3.7	1.7	5.8	3.7	11.2
	Mean		3.70	-2.00	-2.30	3.70	2.16	3.78	9.64
	Median		3.70	-0.80	-2.30	3.70	0.80	3.70	9.80
	Std. error of mean		0.71	1.20	1.67	0.71	1.13	0.74	1.09
Total
	Mean		1.68	-2.82	1.42	2.86	2.9	4.46	10.22
	Median		2.20	-3.80	3.70	2.75	3.80	4.40	10.50
	Std. error of mean		0.92	0.77	1.49	0.50	0.74	0.49	0.84