Scotland last stretch oh la la

Hi,

One word to come back on my last post. I ended that post saying that the Scottish referendum campaign looked more like the 1980 Quebec campaign, which ended with a 59.6% No vote, than like the 1995 campaign (50.5% No). Obviously, it is not the case anymore. In the last two weeks, we went from a Quebec1980-like campaign to a Quebec1995-like campaign. This has a number of implications in terms of poll analysis, one of them being the question of how to attribute the voting intentions of the so-called undecideds. I prefer to call this group the non-disclosers, since in all the polls, they include both those who say that they don't know how they are going to vote and those who say they are not going to vote. 

Let us first look at whether there has effectively been a change in voting intentions in the last weeks and to what extent. Since the last debate on August 25, which was won by Alex Salmond according to most observers and polls, five new polls were carried, one by Survation, one by Panelbase, two by YouGov and one, published today, by TNS-Sofres. All these polls tend to show that the two sides are coming closer, almost at par. The following graph gives an idea of what it means using all the polls conducted since the beginning of 2014. It shows that the proportion of non-disclosers is decreasing as the proportion of Yes voters is going up and the proportion of No voters is stable.

However, the preceding graph takes into account all the polls published since January. If I analyze only the polls conducted since the beginning of August, it gives the next graph. This analysis gives all the chances to the Yes side. It shows an impressive movement towards the Yes and away from the No concurrent with a decrease in non-disclosure to the point that the two lines cross at 45%. However, we have to remind ourselves that almost all the last polls are opt-in panels and that these polls have given the Yes five points more than the other polls during last year*. In short, this portrait is likely to be inaccurate.

What about the attribution of preferences to the non-disclosers?

As the Yes and No camps become closer, the question of the non-disclosers becomes important. How should we attribute their vote? John Curtice has shown here that it is not likely that these respondents be more favorable to the Yes side than to the No side. 

First, let us stress that if the "undecideds" were just people who still do not know how they will vote,  that proportion would tend to be the same whoever the pollster. However, it is not the case at all. The proportion of non-disclosers varies widely between pollsters, from an average of  9.5% for Ipsos Mori (telephone) to 26.9% for TNS-Sofres (face to face). In between, opt-in panels have average rates that vary from 11% (YouGov) to 18% (ICM). This shows that the proportion of non-disclosers is a feature of the way polls are conducted by each firm at least as much as a feature of the respondents.

Let us first look at what happens when we use a proportional distribution of voting intentions to the non-disclosers, i.e., the method used by all pollsters in Scotland right now. When we use all the polls published since January, we get the following graph. The analysis estimates the Yes at close to 47% with the No at more than 53%.

However, if we use only the polls published since the beginning of August, here is what we get. The two sides are almost equal.

Is there a relationship between the proportion of non-disclosers in the polls and the proportion of Yes or No? I checked and there is none. This does not mean however that the non-disclosers can be attributed to each side proportionally. Why is that? Because the profile of the non-disclosers is usually more similar to the profile of No voters than to the profile of Yes voters.

I did not performed non-proportional attribution of voting intentions before because since the No side was quite ahead, it did not change anything in the estimates. However, in the actual situation, we have to consider non-proportional attribution for at least two reasons. One is that the No voters may be less likely to be reached by pollsters -- and even more by opt-in panels -- or to accept to cooperate with them. The second reason is that, when they are reached, they tend more to hide their opinion. This is even more likely when the Yes side gets ahead because the Yes side tends to be seen as more "popular".

The situation was about the same in the Quebec 1995 referendum campaign so that the media used to publish estimates that gave 75% of the non-disclosers -- around 13% of the respondents at that time -- to the No side. With such a non proportional attribution, the polls' estimates gave 50.5% to the No side and 49.5% to the Yes side, i.e., the exact result of the referendum. In the case of Scotland, it seems relevant to estimate what it would mean to perform a non-proportional attribution. I attributed 67% of undecideds to the No side and 33% to the Yes side because I think that the possible tendency to hide preferences for the No may be less substantial in Scotland than it was in Quebec because of the ethnic concentration of voters. This is the portrait that I get. It estimates the Yes side at 45% and the No side at 55%.

If I use only the polls conducted since the beginning of August, here is what I get. The two sides are no longer at par like in the comparable graph using proportional attribution. The Yes side is at about 47% and the No side at 53%.

Conclusion

Many commentators, particularly in Quebec, have an impression of déjà vu when they see the reaction to a possible change in the polls from the Better Together camp. Polls do not serve only to measure public opinion. At times, they may be used by people to send a message that they want something on the table if they are to stay. Saying that you will vote Yes in the polls has no consequence besides having the other side make moves.  However, the change in voting intentions may also be very real. Only the next polls will tell us how likely it is that the Yes side wins this referendum. In my next post, I will come back on voting intentions according to age groups because this also seems to have changed recently. As in Quebec in 1995, the younger age cohorts are the more likely to move towards the Yes side.

* About methods: The method that I use takes into account the timing of the polls. It may have its drawbacks but it is certainly preferable to taking the average of the polls published during the last two weeks, for example. This is certainly not appropriate when there is a substantial level of change with time. I also see graphs that do not respect the time line i.e., polls that are two weeks apart are put at the same distance than polls that are two days apart. This is also very misleading and it should not be published.