One month before, Scottish Independence & Quebec Sovereignty

Hi,

This is the last month before the referendum on Scottish independence, an appropriate moment to compare the current situation in Scotland and the situation that prevailed in 1995 in Quebec.

Scotland
First, let's examine the current situation in Scotland. With 49 polls published in 2014, one month to go, the likely situation right now is a gap of more than 10 points in favor of the No (12 points if we distribute the non-disclosers proportionally), as the following graphs show. Some have maintained that a rise in support for the Yes side have occurred last week. This is not obvious right now. As we can see, there is variation in the estimates published by the different pollsters. If we take that into account, we are still in a situation of no change in either direction. The impact of the debate held on August 5 may have been to reduce the proportion of non-disclosers (including mostly don't knows).

In evaluating the situation, we have to take into account the fact that half the polls, i.e.,  those conducted using opt-in panels by ICM, Survation and Panelbase, put support for the Yes side 4.5 points higher on average than the other polls (*see methodological notice at the end). If we look at all the elections where opt-in panels were compared to election results, we may conclude that these polls likely overestimate support for the Yes side. Let us now compare this situation with what happened in Quebec during a similar period to see if it can teach us something.

Quebec 1995
What was the situation in Quebec during about the same period? I kept all the polls -- including those conducted by academics, governmental agencies and political parties -- conducted between February 15, 1995 and Referendum Day, October 31th, 1995. I kept the results only when the question asked about vote for Sovereignty with an association with the rest of Canada. Before June 12 (first vertical line on the graph), when an agreement between Parti Québécois, Action Démocratique du Québec and Bloc Québécois -- called the 3-party agreement -- was signed, it was not clear what the question would be and whether a partnership with the rest of Canada would be part of it. Therefore, there are not many polls asking about this option before June 12. I have shown elsewhere that support differs according to the constitutional option -- independence, separation, sovereignty, sovereignty with partnership -- and that only the last option managed to reach majority support during some periods.

The first graph shows the change in support from February 15, 1995 to Referendum Day, including the non-disclosers. It shows that support for sovereignty with an association with the rest of Canada managed to get the support of a plurality of Quebeckers until two months before referendum Day. However, after the official launching of the campaign on September 7 (second vertical line on the graph), Support for the Yes side decreased to 40% and the proportion of non-disclosers increased until the last month. Then support for the Yes side went back up until referendum Day. The final two months are characterized by a roller-coaster type of movement and an increase in the proportion of non-disclosers (including don't knows).

The second graph shows the same information after proportional attribution of non-disclosers (**see methodological notice at the end).  Again, it shows a situation favorable to the Yes side, up until the launching of the campaign on September 7 and a roller-coaster ride afterwards.

Let's now focus on the last stretch. The third graph shows that, when the campaign was launched, the two sides were at par. Support for the Yes side went down during the first month so that one month before referendum Day, support for the Yes side was five to six points lower than support for the No. However, support for the Yes side went back up to finish 6 points higher than support for the No. However, as we can see, with proportional attribution of non-disclosers, support for the Yes was seriously overestimated. The referendum ended with 49.5% Yes and 50.5% No. With realistic (*) attribution of non-disclosers, the prediction of the results was almost perfect.

Conclusion - Scotland vs Quebec
There are a number of differences between the Quebec 1995 campaign and the Scottish 2014 campaign. For instance, the Quebec question pertained to sovereignty accompanied by an offer of  partnership to the rest of Canada and it was officially revealed less than two months before referendum Day. Independence per se (the Scottish option) never managed to get the support of a majority of Quebeckers in the polls.

In terms of poll results, the Quebec question managed to get the support of a short majority of Quebeckers during most of the period, which has not been the case in Scotland. When the Quebec Yes side increased its support during the last month in 1995, it was regaining support from people who had already been inclined to vote Yes at some point during the preceding months. Scotland's Yes side cannot rely on this possibility. It has to convince voters to change a position they held for months. Everything is still possible but, since no movement in support has occurred since March, the Scottish Yes side needs a very major event in order to gain the support of a majority of Scottish people. As the Quebec situation shows, it is possible to reverse the trend. In Quebec, a change in strategy -- i.e. the nomination of Lucien Bouchard as negotiator in chief on October 7th -- seemed to have helped close the gap with the No side. However, Scotland's Yes side would probably need more than that. The current situation in Scotland resembles more to what happened in the first Quebec referendum in 1980 that ended with 59.6 for the No side than to what happened in the second referendum in 1995.

P.S. I will be away from August 24 to September 8. I will be back to monitor the last stretch.

* The other pollsters are TNS-BRMB (face-to-face), Ipsos-Mori (telephone) and YouGov (opt-in panel). These three pollsters had similar estimates since January. Note however that You Gov has just changed some features of its methodology (see here). It announced that it now includes respondents that are 16 and 17 year old (which means that they were not included in preceding polls!!) and that it modified its weighting scheme to take into account country of birth. YouGov states that its results for the last poll are not comparable with previous results and that the proportion of support for Yes Scotland is a bit higher with these new features.
** At that time, in Quebec, media and researchers alike tended to use a "realistic" attribution of non-disclosers that gave 75% of  them to the No side. This practice tended to put both sides almost equal and gave a very good prediction of the final result. Some Media published three figures, i.e., voting intention before attribution of non-disclosers, with proportional attribution and with "realistic" attribution.

Polls on Scottish Independence: questions of methods

In this post, I first examine the current state of support for Scottish independence and second, I examine whether different methodological poll features are related to estimates, all else being equal.

The current state of support for Scottish independence

The following graphs show the change in support since January 2014. The first graph includes non-disclosers (don't knows & will not vote). Support seems rather stable since March 2014. The first graph shows a possible recent small increase in support for the No concurrent with a decrease in the proportion of non-disclosers. However, when non-disclosers are attributed proportionally to both sides (second graph), no such increase in  support for the No side is present. There is no significant relationship between the proportion of non-disclosers and support for each side (see below). Before attribution of non-disclosers, Yes is at 38% and No at 50%. After proportional attribution, the difference between the two sides is close to 15%, with the Yes slightly over 43% and the No at 57%.

Notice that in the following graphs, the time line is respected, which is not the case usually in the graphs presented on the different web sites and in the media.

Questions of methods

There was an interesting debate in Scotland on the weighting of opt-in web polls according to the recall of previous votes. In short, the question pertained to which election(s)  should be used for weighting. John Curtice's appraisal of the debate -- that took place between Survation and YouGov -- can be found here: Who is right? Survation or YouGov?  I quite share his appraisal that a)  the difference in estimates between YouGov and the three other pollsters using opt-in panels (ICM, Survation and Panelbase) cannot be due only to weighting and b) we do not have sufficient information to assess the cause(s) of this difference.

However, the interesting point here is that we have a systematic difference within the opt-in panel method. Therefore, we may think that there are specific features of opt-in panels (recruitment, selection, administration) that lend to different estimates. Usually, I check whether there is a difference between opt-in panels and other methods (I did find some in most elections held in Canada --see here and here). In the Scottish situation, I will also check whether there is a significant difference between YouGov, Ipsos-Mori (telephone) and TNS-Sofres (face-to-face) on one side and the opt-in panels carried by ICM, Survation and Panelbase on the other side.

I also address the question of whether the wording of the question makes a difference and whether the proportion of non-disclosers is related to support and to methods.



a) Administration mode

Controlling for time, opt-in web polls in general (including YouGov) estimate support for the Yes side, 3.1 points higher, on average, than the telephone and face-to-face polls. However, if we group YouGov with the telephone and face-to-face polls --which is justified by analysis of variance -- we conclude that the "other" opt-in web polls (Survation, ICM and Panelbase), estimate the proportion of Yes 4.6 points higher than the other polls.

In short, there is clearly a difference according to administration mode but YouGov's estimates are closer to the telephone and face-to-face polls' estimates than to the other opt-in panels.

b) Question wording

Question wording is related to mode of administration but it differs also between the different pollsters. I checked for two possibilities, i.e., whether the fact that Yes or No is specifically mentioned in the question makes a difference and whether the fact that the question asks how the respondent would vote now vs on September 18 makes a difference.

Mentioning specifically Yes or No does not make any difference, most probably because it is not relevant in Web Polls and because both Ipsos-Mori (telephone) and TNS-Sofres (face-to-face) mention Yes or No in their question.

The impact on the level of support of asking how the respondent would vote "now" vs "on September 18th varies depending on the variables that are present in the analyses. For now, I conclude that the difference noticed is more related each pollster's "cluster of methods" than to question wording itself.

c) Non disclosers

If there is one main difference between the pollsters, it is in the proportion of non-disclosers. All the firms differ significantly from each other in this regard. TNS-Sofres (face-to-face) has the highest proportion of non-disclosers -- an average of 28% -- while Ipsos-Mori (telephone) has the lowest -- 9.5%. One may ask whether TNS could reduce its very high proportion by improving its methods using for example, a secret ballot or a leaning question. The web polls  have proportions that are in between, from YouGov at 11.7% to ICM at 18.6%. Notice that, during the last month of the campaign in the Quebec 1995 referendum, the level of non-disclosers (telephone polls only) was around 13% on average and that the final turnout reached more than 93.5% (!).

It is also interesting to notice that:

• There is no relationship between time and the proportion of non-disclosers. This proportion is solely related to the pollsters. It is a question of methods.
• However, asking for voting intention "today" instead of "in September" is related to a decrease in the proportion of non-disclosers of more than 5 points. This is related to the different pollsters but is nonetheless very significant.

Conclusion

It is interesting to notice that the difference between opt-in web polls and other methods, and particularly between some web polls and other methods, is systematic and goes in the anticipated direction, i.e., a higher estimation of support for the Yes side by the opt-in web polls. It should also be noticed that there is some research to be carried on why some web polls' estimates differ systematically from others.The other important point is the substantial difference in the proportion of non disclosers. This is clearly a question of methods since the proportion does not decrease with time. However, since it is not related to the proportion of support for each side, it has no clear impact on the estimate of support. Finally, as we get closer to the date of the referendum, the impact of asking voting intention for today vs for referendum day may fade out.

We are getting to the last month of the campaign. I will go on monitoring whether the impact of methods changes in the last stretch. However, in my next post, the focus will be on comparing Scotland with Quebec, and specifically with what happened during the last month (the official campaign) of the Quebec 1995 referendum.