Scotland the day after: how did the polls fare?

Hi,

Here is the graph that I published in my previous message on Wednesday afternoon. It shows the likely result when I attribute 67% of non-disclosers to the No side. The results of the vote (55.3% No, 44.7% Yes) tell us that the No vote is underestimated by almost two points, even with this non-proportional attribution. Two polls were published on Election Day: YouGov put the No at 54% and Ipsos-Mori at 53% (this poll was conducted before Election Day however). This may mean that there was some last minute movement. I checked whether my estimate would have been much different if I had given 75% of non-disclosers to the No side. It does not change enough in the prediction to warrant publication.

Were opt-in panel polls responsible for the underestimation of the No side? If I keep only the polls published since the beginning of August, opt-in panels do not differ at all on average from polls conducted using other methods (telephone and face-to-face) in their prediction of the final results. They do differ in the average proportion of non-disclosers: more than five points less on average than other polls. Finally, it is interesting to notice that Survation and ICM  who used to conduct opt-in panel polls both conducted a telephone poll at the end of the campaign. ICM conducted its telephone poll at about the same time (Sept 9-11) as an opt-in panel poll. The telephone poll gave 40% Yes, 42% No and 17% non-disclosers (49% Yes - 51% No after proportional allocation of non-disclosers) while the opt-in panel gave 49% Yes, 42% No and 9% non-disclosers (54% Yes, 46% No). The telephone poll is clearly better. However, we cannot base our evaluation of opt-in panels on only one case.

Conclusion

As with the Quebec 1995 referendum, the polls underestimated the No vote and this, even when we allocate two-thirds of non-disclosers to the No side. This seems like a systematic bias that we will have to take into account in future referendum votes. However, the opt-in panels did not fare better nor worse than other polls at the end of the campaign.