Polls on Scottish Independence: questions of methods

In this post, I first examine the current state of support for Scottish independence and second, I examine whether different methodological poll features are related to estimates, all else being equal.

The current state of support for Scottish independence

The following graphs show the change in support since January 2014. The first graph includes non-disclosers (don't knows & will not vote). Support seems rather stable since March 2014. The first graph shows a possible recent small increase in support for the No concurrent with a decrease in the proportion of non-disclosers. However, when non-disclosers are attributed proportionally to both sides (second graph), no such increase in  support for the No side is present. There is no significant relationship between the proportion of non-disclosers and support for each side (see below). Before attribution of non-disclosers, Yes is at 38% and No at 50%. After proportional attribution, the difference between the two sides is close to 15%, with the Yes slightly over 43% and the No at 57%.

Notice that in the following graphs, the time line is respected, which is not the case usually in the graphs presented on the different web sites and in the media.

Questions of methods

There was an interesting debate in Scotland on the weighting of opt-in web polls according to the recall of previous votes. In short, the question pertained to which election(s)  should be used for weighting. John Curtice's appraisal of the debate -- that took place between Survation and YouGov -- can be found here: Who is right? Survation or YouGov?  I quite share his appraisal that a)  the difference in estimates between YouGov and the three other pollsters using opt-in panels (ICM, Survation and Panelbase) cannot be due only to weighting and b) we do not have sufficient information to assess the cause(s) of this difference.

However, the interesting point here is that we have a systematic difference within the opt-in panel method. Therefore, we may think that there are specific features of opt-in panels (recruitment, selection, administration) that lend to different estimates. Usually, I check whether there is a difference between opt-in panels and other methods (I did find some in most elections held in Canada --see here and here). In the Scottish situation, I will also check whether there is a significant difference between YouGov, Ipsos-Mori (telephone) and TNS-Sofres (face-to-face) on one side and the opt-in panels carried by ICM, Survation and Panelbase on the other side.

I also address the question of whether the wording of the question makes a difference and whether the proportion of non-disclosers is related to support and to methods.



a) Administration mode

Controlling for time, opt-in web polls in general (including YouGov) estimate support for the Yes side, 3.1 points higher, on average, than the telephone and face-to-face polls. However, if we group YouGov with the telephone and face-to-face polls --which is justified by analysis of variance -- we conclude that the "other" opt-in web polls (Survation, ICM and Panelbase), estimate the proportion of Yes 4.6 points higher than the other polls.

In short, there is clearly a difference according to administration mode but YouGov's estimates are closer to the telephone and face-to-face polls' estimates than to the other opt-in panels.

b) Question wording

Question wording is related to mode of administration but it differs also between the different pollsters. I checked for two possibilities, i.e., whether the fact that Yes or No is specifically mentioned in the question makes a difference and whether the fact that the question asks how the respondent would vote now vs on September 18 makes a difference.

Mentioning specifically Yes or No does not make any difference, most probably because it is not relevant in Web Polls and because both Ipsos-Mori (telephone) and TNS-Sofres (face-to-face) mention Yes or No in their question.

The impact on the level of support of asking how the respondent would vote "now" vs "on September 18th varies depending on the variables that are present in the analyses. For now, I conclude that the difference noticed is more related each pollster's "cluster of methods" than to question wording itself.

c) Non disclosers

If there is one main difference between the pollsters, it is in the proportion of non-disclosers. All the firms differ significantly from each other in this regard. TNS-Sofres (face-to-face) has the highest proportion of non-disclosers -- an average of 28% -- while Ipsos-Mori (telephone) has the lowest -- 9.5%. One may ask whether TNS could reduce its very high proportion by improving its methods using for example, a secret ballot or a leaning question. The web polls  have proportions that are in between, from YouGov at 11.7% to ICM at 18.6%. Notice that, during the last month of the campaign in the Quebec 1995 referendum, the level of non-disclosers (telephone polls only) was around 13% on average and that the final turnout reached more than 93.5% (!).

It is also interesting to notice that:

• There is no relationship between time and the proportion of non-disclosers. This proportion is solely related to the pollsters. It is a question of methods.
• However, asking for voting intention "today" instead of "in September" is related to a decrease in the proportion of non-disclosers of more than 5 points. This is related to the different pollsters but is nonetheless very significant.

Conclusion

It is interesting to notice that the difference between opt-in web polls and other methods, and particularly between some web polls and other methods, is systematic and goes in the anticipated direction, i.e., a higher estimation of support for the Yes side by the opt-in web polls. It should also be noticed that there is some research to be carried on why some web polls' estimates differ systematically from others.The other important point is the substantial difference in the proportion of non disclosers. This is clearly a question of methods since the proportion does not decrease with time. However, since it is not related to the proportion of support for each side, it has no clear impact on the estimate of support. Finally, as we get closer to the date of the referendum, the impact of asking voting intention for today vs for referendum day may fade out.

We are getting to the last month of the campaign. I will go on monitoring whether the impact of methods changes in the last stretch. However, in my next post, the focus will be on comparing Scotland with Quebec, and specifically with what happened during the last month (the official campaign) of the Quebec 1995 referendum.