Scotland the day after: how did the polls fare?

Hi,

Here is the graph that I published in my previous message on Wednesday afternoon. It shows the likely result when I attribute 67% of non-disclosers to the No side. The results of the vote (55.3% No, 44.7% Yes) tell us that the No vote is underestimated by almost two points, even with this non-proportional attribution. Two polls were published on Election Day: YouGov put the No at 54% and Ipsos-Mori at 53% (this poll was conducted before Election Day however). This may mean that there was some last minute movement. I checked whether my estimate would have been much different if I had given 75% of non-disclosers to the No side. It does not change enough in the prediction to warrant publication.

Were opt-in panel polls responsible for the underestimation of the No side? If I keep only the polls published since the beginning of August, opt-in panels do not differ at all on average from polls conducted using other methods (telephone and face-to-face) in their prediction of the final results. They do differ in the average proportion of non-disclosers: more than five points less on average than other polls. Finally, it is interesting to notice that Survation and ICM  who used to conduct opt-in panel polls both conducted a telephone poll at the end of the campaign. ICM conducted its telephone poll at about the same time (Sept 9-11) as an opt-in panel poll. The telephone poll gave 40% Yes, 42% No and 17% non-disclosers (49% Yes - 51% No after proportional allocation of non-disclosers) while the opt-in panel gave 49% Yes, 42% No and 9% non-disclosers (54% Yes, 46% No). The telephone poll is clearly better. However, we cannot base our evaluation of opt-in panels on only one case.

Conclusion

As with the Quebec 1995 referendum, the polls underestimated the No vote and this, even when we allocate two-thirds of non-disclosers to the No side. This seems like a systematic bias that we will have to take into account in future referendum votes. However, the opt-in panels did not fare better nor worse than other polls at the end of the campaign.

Scotland Independence: update with the last polls

Hi,

Tomorrow is voting day. I include in my analysis all the polls published yesterday and today -- before 2 PM Montreal time (I do not have the YouGov and the TNS yet, I may update later on if I have time).

Let us examine what the polls tell us of the likely portrait of support for Scottish independence.  First, a graph including the non-disclosers, i.e., don't knows and those who say they will not vote.The graph  shows that the two sides are still close to each other but with the No side still ahead. In addition, the proportion of non-disclosers has stopped declining.

Now, if we use the pollsters' attribution of non-diclosers, i.e., proportional attribution, here is what we get. Voting intentions seem to have frozen. They have been quite stable for the last two weeks, with about five points difference in favor of the No side.

  

 

If I perform a non-proportional attribution of non-disclosers, giving 67% of them to the No side, which, according to many political scientists is realistic, I get the following graph. The analysis estimates the difference between Yes and No at more than seven points.

 

Finally, the following graph is quite interesting. I focus on the difference between support for independence overall and support among young people aged 16-34 years old. As you can see, we could attribute part of the increase in support for the Yes side to young people in recent weeks but polls show that this age group seems to have backed off during the last few days.

Conclusion

If polls are not incredibly wrong, my analysis shows that the No side is likely to win with a good margin. If I am wrong, it will be the pollsters' fault :). And political scientists and survey methodologists will have to revise all what they said in the last 30 years.

Yes Scotland: Ceiling reached?

Hi,

Now, here is the situation, when the polls conducted during the week-end are included. It is interesting to notice that two pollsters -- Survation and ICM -- that used to do only opt-in panel polls conducted telephone polls. The situation right now is that the two sides are close to one another, which explains why we get contradictory estimates from different pollsters. This is quite normal in such circumstances. The graph shows that the Yes side has gained some points and the No side together with the non-disclosers have lost some.The No side is still ahead (barely) but it seems to have lost two points since the beginning of August, while the Yes side has gained close to five points.

Now, if we attribute the non-disclosers proportionally to each side, we get the following graph. It shows that the increase in support for the Yes side seems to have stalled. Support for Yes Scotland is around 48% and support for Better Together at 52%.

If we use a more realistic attribution of non-disclosers' voting intentions, giving 67% of those to the No side, support for the Yes side is closer to 46.5%. The difference between the two sides is estimated at around seven points. The preceding graph and the next one both show rather clearly that support for the Yes side has stopped increasing last week. Voting intentions seem to have stabilized.

A last graph shows the part played by younger people -- 16 to 34 years old -- in the increase in support for the Yes side. Young people's support for the Yes side, already higher than in the rest of the population, has also increased more than that of older Scottish residents. This situation is very similar to what has been observed in the 1995 Quebec referendum. Support for Yes Scotland among young people is around 55% according to the polls.

In conclusion: Are the polls reliable?

The Yes side has gained an average of 1.15 points per week since the beginning of August. Even if it would take another 1.15 points during the last week, it is not sufficient to win. But are the polls reliable? First, let us notice that the difference in estimation between opt-in panel polls and other polls has disappeared. Since August at least, there is no significant difference anymore between web polls and other polls. However, as in Quebec in 1995, the polls may overestimate the Yes side. Why is that? First, it seems that Yes supporters are more likely to answer surveys and to reveal their vote and second, as the Yes becomes more likely to win, some people may back off and finally decide not to "take the risk". As I said before, saying that you will vote Yes to a pollster has much less consequences than doing it for real. A very good analysis of the possible vote of non-disclosers -- by Stephen Fisher -- can be found here. Therefore, I still think that all possible errors of the polls would result in an overestimation of the Yes side.

Scottish independence polls: Update & plus

Hi,

***Please note that I made a mistake. All the analysis are conducted since the beginning of January and not August as I stated in my post. *** Please see my next post for an update.
From now on, I concentrate my analysis on the polls published since the beginning of August in order to give more weight to the recent polls in the estimation of change. In this post, I also come back on the generation gap that I have analyzed in a previous post.

The next graph shows the recent evolution of the situation. It includes the YouGov and ICM polls published in the last hours. We see a concentration of estimates in the last few days: Most of the polls have put the two sides very close.

If we attribute non-disclosers proportionally to each side, we get the following portrait. It is slightly modified compared to my preceding post. Because of the two last posts, it no longer shows the two sides at par. The analysis estimates the Yes side at about 47.5% and the no at 52.5%. The spread of estimates has decreased and the estimates seem more similar across pollsters.

However, there is still a very significant difference between the opt-in web polls and the other polls. Using regression analysis, I have the estimate of support for the Yes side (with proportional attribution of non disclosers) at 42.8% at the beginning of August, and almost three points more when the polls are  opt-in panel polls, and gaining on average .136 points per week for the last 6 weeks (0.8 points total). Therefore, opt-in panels have the Yes at about 46.6% and the other polls at 43.6%.

Like in my preceding post, I estimate the results with a non proportional attribution of non-disclosers, giving 67% of them to the No side and 33% to the Yes side (see here for an explanation). This gives the following graph. With this more realistic attribution, the Yes side is closer to 45%

The generation gap

I have looked at the generation gap in a preceding post. It is very interesting to update on this gap for comparative purposes. In the 1995 referendum in Quebec, people less than 55 years old were more favorable to sovereignty at the beginning of the campaign and they were also the only one to move towards the Yes side during the campaign. The 55 years and older age group did not change its mind -- at least on average -- during the campaign.

The following graph allows to compare the 16-34 age old group to the total. Unfortunately, since pollsters do not use the same age categories, it is the only comparison that I can perform (Notice that for YOUGOV, the age group is in fact 16-39 and was 18-39 in the polls conducted before the August 12-15 poll). The graph confirms that, on average, support for independence is higher and has increased slightly more rapidly among the younger age group. However, notice that the estimates (in light blue) vary much. In August, for example, some polls have put support for independence in the younger age group as low as 37% while others put it at 60%.

Conclusion

We have to be very careful not to interpret random variation as movement. There is no doubt that there has been an increase in support for the Yes side during the last three weeks. However, if we take all the polls into account, the Yes side does not have the support of a majority of voters right now. The next polls will be crucial to estimate whether the increase in support has reached a ceiling or not.

In addition, let us stress the fact that when the difference between the two sides is estimated at less than six points, it is not appropriate -- and even a false assertion -- to say that "The Yes side is ahead for the first time" or "the No side is still ahead". This is what the margin of error means. And it is even more so when polls are not conducted using random samples of respondents like opt-in panels do. In this latter case, some speak about a credibility interval that would be similar to the margin of error. However, empirically, there is evidence that, in addition to that random error, there is some systematic bias in opt-in panels. In this case, the bias is likely to produce an overestimation of the Yes side. If the results show that it is not the case, it will be a very interesting topic for research.

Scotland last stretch oh la la

Hi,

One word to come back on my last post. I ended that post saying that the Scottish referendum campaign looked more like the 1980 Quebec campaign, which ended with a 59.6% No vote, than like the 1995 campaign (50.5% No). Obviously, it is not the case anymore. In the last two weeks, we went from a Quebec1980-like campaign to a Quebec1995-like campaign. This has a number of implications in terms of poll analysis, one of them being the question of how to attribute the voting intentions of the so-called undecideds. I prefer to call this group the non-disclosers, since in all the polls, they include both those who say that they don't know how they are going to vote and those who say they are not going to vote. 

Let us first look at whether there has effectively been a change in voting intentions in the last weeks and to what extent. Since the last debate on August 25, which was won by Alex Salmond according to most observers and polls, five new polls were carried, one by Survation, one by Panelbase, two by YouGov and one, published today, by TNS-Sofres. All these polls tend to show that the two sides are coming closer, almost at par. The following graph gives an idea of what it means using all the polls conducted since the beginning of 2014. It shows that the proportion of non-disclosers is decreasing as the proportion of Yes voters is going up and the proportion of No voters is stable.

However, the preceding graph takes into account all the polls published since January. If I analyze only the polls conducted since the beginning of August, it gives the next graph. This analysis gives all the chances to the Yes side. It shows an impressive movement towards the Yes and away from the No concurrent with a decrease in non-disclosure to the point that the two lines cross at 45%. However, we have to remind ourselves that almost all the last polls are opt-in panels and that these polls have given the Yes five points more than the other polls during last year*. In short, this portrait is likely to be inaccurate.

What about the attribution of preferences to the non-disclosers?

As the Yes and No camps become closer, the question of the non-disclosers becomes important. How should we attribute their vote? John Curtice has shown here that it is not likely that these respondents be more favorable to the Yes side than to the No side. 

First, let us stress that if the "undecideds" were just people who still do not know how they will vote,  that proportion would tend to be the same whoever the pollster. However, it is not the case at all. The proportion of non-disclosers varies widely between pollsters, from an average of  9.5% for Ipsos Mori (telephone) to 26.9% for TNS-Sofres (face to face). In between, opt-in panels have average rates that vary from 11% (YouGov) to 18% (ICM). This shows that the proportion of non-disclosers is a feature of the way polls are conducted by each firm at least as much as a feature of the respondents.

Let us first look at what happens when we use a proportional distribution of voting intentions to the non-disclosers, i.e., the method used by all pollsters in Scotland right now. When we use all the polls published since January, we get the following graph. The analysis estimates the Yes at close to 47% with the No at more than 53%.

However, if we use only the polls published since the beginning of August, here is what we get. The two sides are almost equal.

Is there a relationship between the proportion of non-disclosers in the polls and the proportion of Yes or No? I checked and there is none. This does not mean however that the non-disclosers can be attributed to each side proportionally. Why is that? Because the profile of the non-disclosers is usually more similar to the profile of No voters than to the profile of Yes voters.

I did not performed non-proportional attribution of voting intentions before because since the No side was quite ahead, it did not change anything in the estimates. However, in the actual situation, we have to consider non-proportional attribution for at least two reasons. One is that the No voters may be less likely to be reached by pollsters -- and even more by opt-in panels -- or to accept to cooperate with them. The second reason is that, when they are reached, they tend more to hide their opinion. This is even more likely when the Yes side gets ahead because the Yes side tends to be seen as more "popular".

The situation was about the same in the Quebec 1995 referendum campaign so that the media used to publish estimates that gave 75% of the non-disclosers -- around 13% of the respondents at that time -- to the No side. With such a non proportional attribution, the polls' estimates gave 50.5% to the No side and 49.5% to the Yes side, i.e., the exact result of the referendum. In the case of Scotland, it seems relevant to estimate what it would mean to perform a non-proportional attribution. I attributed 67% of undecideds to the No side and 33% to the Yes side because I think that the possible tendency to hide preferences for the No may be less substantial in Scotland than it was in Quebec because of the ethnic concentration of voters. This is the portrait that I get. It estimates the Yes side at 45% and the No side at 55%.

If I use only the polls conducted since the beginning of August, here is what I get. The two sides are no longer at par like in the comparable graph using proportional attribution. The Yes side is at about 47% and the No side at 53%.

Conclusion

Many commentators, particularly in Quebec, have an impression of déjà vu when they see the reaction to a possible change in the polls from the Better Together camp. Polls do not serve only to measure public opinion. At times, they may be used by people to send a message that they want something on the table if they are to stay. Saying that you will vote Yes in the polls has no consequence besides having the other side make moves.  However, the change in voting intentions may also be very real. Only the next polls will tell us how likely it is that the Yes side wins this referendum. In my next post, I will come back on voting intentions according to age groups because this also seems to have changed recently. As in Quebec in 1995, the younger age cohorts are the more likely to move towards the Yes side.

* About methods: The method that I use takes into account the timing of the polls. It may have its drawbacks but it is certainly preferable to taking the average of the polls published during the last two weeks, for example. This is certainly not appropriate when there is a substantial level of change with time. I also see graphs that do not respect the time line i.e., polls that are two weeks apart are put at the same distance than polls that are two days apart. This is also very misleading and it should not be published.