Scotland the day after: how did the polls fare?

Hi,

Here is the graph that I published in my previous message on Wednesday afternoon. It shows the likely result when I attribute 67% of non-disclosers to the No side. The results of the vote (55.3% No, 44.7% Yes) tell us that the No vote is underestimated by almost two points, even with this non-proportional attribution. Two polls were published on Election Day: YouGov put the No at 54% and Ipsos-Mori at 53% (this poll was conducted before Election Day however). This may mean that there was some last minute movement. I checked whether my estimate would have been much different if I had given 75% of non-disclosers to the No side. It does not change enough in the prediction to warrant publication.

Were opt-in panel polls responsible for the underestimation of the No side? If I keep only the polls published since the beginning of August, opt-in panels do not differ at all on average from polls conducted using other methods (telephone and face-to-face) in their prediction of the final results. They do differ in the average proportion of non-disclosers: more than five points less on average than other polls. Finally, it is interesting to notice that Survation and ICM  who used to conduct opt-in panel polls both conducted a telephone poll at the end of the campaign. ICM conducted its telephone poll at about the same time (Sept 9-11) as an opt-in panel poll. The telephone poll gave 40% Yes, 42% No and 17% non-disclosers (49% Yes - 51% No after proportional allocation of non-disclosers) while the opt-in panel gave 49% Yes, 42% No and 9% non-disclosers (54% Yes, 46% No). The telephone poll is clearly better. However, we cannot base our evaluation of opt-in panels on only one case.

Conclusion

As with the Quebec 1995 referendum, the polls underestimated the No vote and this, even when we allocate two-thirds of non-disclosers to the No side. This seems like a systematic bias that we will have to take into account in future referendum votes. However, the opt-in panels did not fare better nor worse than other polls at the end of the campaign.

Scotland Independence: update with the last polls

Hi,

Tomorrow is voting day. I include in my analysis all the polls published yesterday and today -- before 2 PM Montreal time (I do not have the YouGov and the TNS yet, I may update later on if I have time).

Let us examine what the polls tell us of the likely portrait of support for Scottish independence.  First, a graph including the non-disclosers, i.e., don't knows and those who say they will not vote.The graph  shows that the two sides are still close to each other but with the No side still ahead. In addition, the proportion of non-disclosers has stopped declining.

Now, if we use the pollsters' attribution of non-diclosers, i.e., proportional attribution, here is what we get. Voting intentions seem to have frozen. They have been quite stable for the last two weeks, with about five points difference in favor of the No side.

  

 

If I perform a non-proportional attribution of non-disclosers, giving 67% of them to the No side, which, according to many political scientists is realistic, I get the following graph. The analysis estimates the difference between Yes and No at more than seven points.

 

Finally, the following graph is quite interesting. I focus on the difference between support for independence overall and support among young people aged 16-34 years old. As you can see, we could attribute part of the increase in support for the Yes side to young people in recent weeks but polls show that this age group seems to have backed off during the last few days.

Conclusion

If polls are not incredibly wrong, my analysis shows that the No side is likely to win with a good margin. If I am wrong, it will be the pollsters' fault :). And political scientists and survey methodologists will have to revise all what they said in the last 30 years.

Yes Scotland: Ceiling reached?

Hi,

Now, here is the situation, when the polls conducted during the week-end are included. It is interesting to notice that two pollsters -- Survation and ICM -- that used to do only opt-in panel polls conducted telephone polls. The situation right now is that the two sides are close to one another, which explains why we get contradictory estimates from different pollsters. This is quite normal in such circumstances. The graph shows that the Yes side has gained some points and the No side together with the non-disclosers have lost some.The No side is still ahead (barely) but it seems to have lost two points since the beginning of August, while the Yes side has gained close to five points.

Now, if we attribute the non-disclosers proportionally to each side, we get the following graph. It shows that the increase in support for the Yes side seems to have stalled. Support for Yes Scotland is around 48% and support for Better Together at 52%.

If we use a more realistic attribution of non-disclosers' voting intentions, giving 67% of those to the No side, support for the Yes side is closer to 46.5%. The difference between the two sides is estimated at around seven points. The preceding graph and the next one both show rather clearly that support for the Yes side has stopped increasing last week. Voting intentions seem to have stabilized.

A last graph shows the part played by younger people -- 16 to 34 years old -- in the increase in support for the Yes side. Young people's support for the Yes side, already higher than in the rest of the population, has also increased more than that of older Scottish residents. This situation is very similar to what has been observed in the 1995 Quebec referendum. Support for Yes Scotland among young people is around 55% according to the polls.

In conclusion: Are the polls reliable?

The Yes side has gained an average of 1.15 points per week since the beginning of August. Even if it would take another 1.15 points during the last week, it is not sufficient to win. But are the polls reliable? First, let us notice that the difference in estimation between opt-in panel polls and other polls has disappeared. Since August at least, there is no significant difference anymore between web polls and other polls. However, as in Quebec in 1995, the polls may overestimate the Yes side. Why is that? First, it seems that Yes supporters are more likely to answer surveys and to reveal their vote and second, as the Yes becomes more likely to win, some people may back off and finally decide not to "take the risk". As I said before, saying that you will vote Yes to a pollster has much less consequences than doing it for real. A very good analysis of the possible vote of non-disclosers -- by Stephen Fisher -- can be found here. Therefore, I still think that all possible errors of the polls would result in an overestimation of the Yes side.

Scottish independence polls: Update & plus

Hi,

***Please note that I made a mistake. All the analysis are conducted since the beginning of January and not August as I stated in my post. *** Please see my next post for an update.
From now on, I concentrate my analysis on the polls published since the beginning of August in order to give more weight to the recent polls in the estimation of change. In this post, I also come back on the generation gap that I have analyzed in a previous post.

The next graph shows the recent evolution of the situation. It includes the YouGov and ICM polls published in the last hours. We see a concentration of estimates in the last few days: Most of the polls have put the two sides very close.

If we attribute non-disclosers proportionally to each side, we get the following portrait. It is slightly modified compared to my preceding post. Because of the two last posts, it no longer shows the two sides at par. The analysis estimates the Yes side at about 47.5% and the no at 52.5%. The spread of estimates has decreased and the estimates seem more similar across pollsters.

However, there is still a very significant difference between the opt-in web polls and the other polls. Using regression analysis, I have the estimate of support for the Yes side (with proportional attribution of non disclosers) at 42.8% at the beginning of August, and almost three points more when the polls are  opt-in panel polls, and gaining on average .136 points per week for the last 6 weeks (0.8 points total). Therefore, opt-in panels have the Yes at about 46.6% and the other polls at 43.6%.

Like in my preceding post, I estimate the results with a non proportional attribution of non-disclosers, giving 67% of them to the No side and 33% to the Yes side (see here for an explanation). This gives the following graph. With this more realistic attribution, the Yes side is closer to 45%

The generation gap

I have looked at the generation gap in a preceding post. It is very interesting to update on this gap for comparative purposes. In the 1995 referendum in Quebec, people less than 55 years old were more favorable to sovereignty at the beginning of the campaign and they were also the only one to move towards the Yes side during the campaign. The 55 years and older age group did not change its mind -- at least on average -- during the campaign.

The following graph allows to compare the 16-34 age old group to the total. Unfortunately, since pollsters do not use the same age categories, it is the only comparison that I can perform (Notice that for YOUGOV, the age group is in fact 16-39 and was 18-39 in the polls conducted before the August 12-15 poll). The graph confirms that, on average, support for independence is higher and has increased slightly more rapidly among the younger age group. However, notice that the estimates (in light blue) vary much. In August, for example, some polls have put support for independence in the younger age group as low as 37% while others put it at 60%.

Conclusion

We have to be very careful not to interpret random variation as movement. There is no doubt that there has been an increase in support for the Yes side during the last three weeks. However, if we take all the polls into account, the Yes side does not have the support of a majority of voters right now. The next polls will be crucial to estimate whether the increase in support has reached a ceiling or not.

In addition, let us stress the fact that when the difference between the two sides is estimated at less than six points, it is not appropriate -- and even a false assertion -- to say that "The Yes side is ahead for the first time" or "the No side is still ahead". This is what the margin of error means. And it is even more so when polls are not conducted using random samples of respondents like opt-in panels do. In this latter case, some speak about a credibility interval that would be similar to the margin of error. However, empirically, there is evidence that, in addition to that random error, there is some systematic bias in opt-in panels. In this case, the bias is likely to produce an overestimation of the Yes side. If the results show that it is not the case, it will be a very interesting topic for research.

Scotland last stretch oh la la

Hi,

One word to come back on my last post. I ended that post saying that the Scottish referendum campaign looked more like the 1980 Quebec campaign, which ended with a 59.6% No vote, than like the 1995 campaign (50.5% No). Obviously, it is not the case anymore. In the last two weeks, we went from a Quebec1980-like campaign to a Quebec1995-like campaign. This has a number of implications in terms of poll analysis, one of them being the question of how to attribute the voting intentions of the so-called undecideds. I prefer to call this group the non-disclosers, since in all the polls, they include both those who say that they don't know how they are going to vote and those who say they are not going to vote. 

Let us first look at whether there has effectively been a change in voting intentions in the last weeks and to what extent. Since the last debate on August 25, which was won by Alex Salmond according to most observers and polls, five new polls were carried, one by Survation, one by Panelbase, two by YouGov and one, published today, by TNS-Sofres. All these polls tend to show that the two sides are coming closer, almost at par. The following graph gives an idea of what it means using all the polls conducted since the beginning of 2014. It shows that the proportion of non-disclosers is decreasing as the proportion of Yes voters is going up and the proportion of No voters is stable.

However, the preceding graph takes into account all the polls published since January. If I analyze only the polls conducted since the beginning of August, it gives the next graph. This analysis gives all the chances to the Yes side. It shows an impressive movement towards the Yes and away from the No concurrent with a decrease in non-disclosure to the point that the two lines cross at 45%. However, we have to remind ourselves that almost all the last polls are opt-in panels and that these polls have given the Yes five points more than the other polls during last year*. In short, this portrait is likely to be inaccurate.

What about the attribution of preferences to the non-disclosers?

As the Yes and No camps become closer, the question of the non-disclosers becomes important. How should we attribute their vote? John Curtice has shown here that it is not likely that these respondents be more favorable to the Yes side than to the No side. 

First, let us stress that if the "undecideds" were just people who still do not know how they will vote,  that proportion would tend to be the same whoever the pollster. However, it is not the case at all. The proportion of non-disclosers varies widely between pollsters, from an average of  9.5% for Ipsos Mori (telephone) to 26.9% for TNS-Sofres (face to face). In between, opt-in panels have average rates that vary from 11% (YouGov) to 18% (ICM). This shows that the proportion of non-disclosers is a feature of the way polls are conducted by each firm at least as much as a feature of the respondents.

Let us first look at what happens when we use a proportional distribution of voting intentions to the non-disclosers, i.e., the method used by all pollsters in Scotland right now. When we use all the polls published since January, we get the following graph. The analysis estimates the Yes at close to 47% with the No at more than 53%.

However, if we use only the polls published since the beginning of August, here is what we get. The two sides are almost equal.

Is there a relationship between the proportion of non-disclosers in the polls and the proportion of Yes or No? I checked and there is none. This does not mean however that the non-disclosers can be attributed to each side proportionally. Why is that? Because the profile of the non-disclosers is usually more similar to the profile of No voters than to the profile of Yes voters.

I did not performed non-proportional attribution of voting intentions before because since the No side was quite ahead, it did not change anything in the estimates. However, in the actual situation, we have to consider non-proportional attribution for at least two reasons. One is that the No voters may be less likely to be reached by pollsters -- and even more by opt-in panels -- or to accept to cooperate with them. The second reason is that, when they are reached, they tend more to hide their opinion. This is even more likely when the Yes side gets ahead because the Yes side tends to be seen as more "popular".

The situation was about the same in the Quebec 1995 referendum campaign so that the media used to publish estimates that gave 75% of the non-disclosers -- around 13% of the respondents at that time -- to the No side. With such a non proportional attribution, the polls' estimates gave 50.5% to the No side and 49.5% to the Yes side, i.e., the exact result of the referendum. In the case of Scotland, it seems relevant to estimate what it would mean to perform a non-proportional attribution. I attributed 67% of undecideds to the No side and 33% to the Yes side because I think that the possible tendency to hide preferences for the No may be less substantial in Scotland than it was in Quebec because of the ethnic concentration of voters. This is the portrait that I get. It estimates the Yes side at 45% and the No side at 55%.

If I use only the polls conducted since the beginning of August, here is what I get. The two sides are no longer at par like in the comparable graph using proportional attribution. The Yes side is at about 47% and the No side at 53%.

Conclusion

Many commentators, particularly in Quebec, have an impression of déjà vu when they see the reaction to a possible change in the polls from the Better Together camp. Polls do not serve only to measure public opinion. At times, they may be used by people to send a message that they want something on the table if they are to stay. Saying that you will vote Yes in the polls has no consequence besides having the other side make moves.  However, the change in voting intentions may also be very real. Only the next polls will tell us how likely it is that the Yes side wins this referendum. In my next post, I will come back on voting intentions according to age groups because this also seems to have changed recently. As in Quebec in 1995, the younger age cohorts are the more likely to move towards the Yes side.

* About methods: The method that I use takes into account the timing of the polls. It may have its drawbacks but it is certainly preferable to taking the average of the polls published during the last two weeks, for example. This is certainly not appropriate when there is a substantial level of change with time. I also see graphs that do not respect the time line i.e., polls that are two weeks apart are put at the same distance than polls that are two days apart. This is also very misleading and it should not be published.

One month before, Scottish Independence & Quebec Sovereignty

Hi,

This is the last month before the referendum on Scottish independence, an appropriate moment to compare the current situation in Scotland and the situation that prevailed in 1995 in Quebec.

Scotland
First, let's examine the current situation in Scotland. With 49 polls published in 2014, one month to go, the likely situation right now is a gap of more than 10 points in favor of the No (12 points if we distribute the non-disclosers proportionally), as the following graphs show. Some have maintained that a rise in support for the Yes side have occurred last week. This is not obvious right now. As we can see, there is variation in the estimates published by the different pollsters. If we take that into account, we are still in a situation of no change in either direction. The impact of the debate held on August 5 may have been to reduce the proportion of non-disclosers (including mostly don't knows).

In evaluating the situation, we have to take into account the fact that half the polls, i.e.,  those conducted using opt-in panels by ICM, Survation and Panelbase, put support for the Yes side 4.5 points higher on average than the other polls (*see methodological notice at the end). If we look at all the elections where opt-in panels were compared to election results, we may conclude that these polls likely overestimate support for the Yes side. Let us now compare this situation with what happened in Quebec during a similar period to see if it can teach us something.

Quebec 1995
What was the situation in Quebec during about the same period? I kept all the polls -- including those conducted by academics, governmental agencies and political parties -- conducted between February 15, 1995 and Referendum Day, October 31th, 1995. I kept the results only when the question asked about vote for Sovereignty with an association with the rest of Canada. Before June 12 (first vertical line on the graph), when an agreement between Parti Québécois, Action Démocratique du Québec and Bloc Québécois -- called the 3-party agreement -- was signed, it was not clear what the question would be and whether a partnership with the rest of Canada would be part of it. Therefore, there are not many polls asking about this option before June 12. I have shown elsewhere that support differs according to the constitutional option -- independence, separation, sovereignty, sovereignty with partnership -- and that only the last option managed to reach majority support during some periods.

The first graph shows the change in support from February 15, 1995 to Referendum Day, including the non-disclosers. It shows that support for sovereignty with an association with the rest of Canada managed to get the support of a plurality of Quebeckers until two months before referendum Day. However, after the official launching of the campaign on September 7 (second vertical line on the graph), Support for the Yes side decreased to 40% and the proportion of non-disclosers increased until the last month. Then support for the Yes side went back up until referendum Day. The final two months are characterized by a roller-coaster type of movement and an increase in the proportion of non-disclosers (including don't knows).

The second graph shows the same information after proportional attribution of non-disclosers (**see methodological notice at the end).  Again, it shows a situation favorable to the Yes side, up until the launching of the campaign on September 7 and a roller-coaster ride afterwards.

Let's now focus on the last stretch. The third graph shows that, when the campaign was launched, the two sides were at par. Support for the Yes side went down during the first month so that one month before referendum Day, support for the Yes side was five to six points lower than support for the No. However, support for the Yes side went back up to finish 6 points higher than support for the No. However, as we can see, with proportional attribution of non-disclosers, support for the Yes was seriously overestimated. The referendum ended with 49.5% Yes and 50.5% No. With realistic (*) attribution of non-disclosers, the prediction of the results was almost perfect.

Conclusion - Scotland vs Quebec
There are a number of differences between the Quebec 1995 campaign and the Scottish 2014 campaign. For instance, the Quebec question pertained to sovereignty accompanied by an offer of  partnership to the rest of Canada and it was officially revealed less than two months before referendum Day. Independence per se (the Scottish option) never managed to get the support of a majority of Quebeckers in the polls.

In terms of poll results, the Quebec question managed to get the support of a short majority of Quebeckers during most of the period, which has not been the case in Scotland. When the Quebec Yes side increased its support during the last month in 1995, it was regaining support from people who had already been inclined to vote Yes at some point during the preceding months. Scotland's Yes side cannot rely on this possibility. It has to convince voters to change a position they held for months. Everything is still possible but, since no movement in support has occurred since March, the Scottish Yes side needs a very major event in order to gain the support of a majority of Scottish people. As the Quebec situation shows, it is possible to reverse the trend. In Quebec, a change in strategy -- i.e. the nomination of Lucien Bouchard as negotiator in chief on October 7th -- seemed to have helped close the gap with the No side. However, Scotland's Yes side would probably need more than that. The current situation in Scotland resembles more to what happened in the first Quebec referendum in 1980 that ended with 59.6 for the No side than to what happened in the second referendum in 1995.

P.S. I will be away from August 24 to September 8. I will be back to monitor the last stretch.

* The other pollsters are TNS-BRMB (face-to-face), Ipsos-Mori (telephone) and YouGov (opt-in panel). These three pollsters had similar estimates since January. Note however that You Gov has just changed some features of its methodology (see here). It announced that it now includes respondents that are 16 and 17 year old (which means that they were not included in preceding polls!!) and that it modified its weighting scheme to take into account country of birth. YouGov states that its results for the last poll are not comparable with previous results and that the proportion of support for Yes Scotland is a bit higher with these new features.
** At that time, in Quebec, media and researchers alike tended to use a "realistic" attribution of non-disclosers that gave 75% of  them to the No side. This practice tended to put both sides almost equal and gave a very good prediction of the final result. Some Media published three figures, i.e., voting intention before attribution of non-disclosers, with proportional attribution and with "realistic" attribution.

Polls on Scottish Independence: questions of methods

In this post, I first examine the current state of support for Scottish independence and second, I examine whether different methodological poll features are related to estimates, all else being equal.

The current state of support for Scottish independence

The following graphs show the change in support since January 2014. The first graph includes non-disclosers (don't knows & will not vote). Support seems rather stable since March 2014. The first graph shows a possible recent small increase in support for the No concurrent with a decrease in the proportion of non-disclosers. However, when non-disclosers are attributed proportionally to both sides (second graph), no such increase in  support for the No side is present. There is no significant relationship between the proportion of non-disclosers and support for each side (see below). Before attribution of non-disclosers, Yes is at 38% and No at 50%. After proportional attribution, the difference between the two sides is close to 15%, with the Yes slightly over 43% and the No at 57%.

Notice that in the following graphs, the time line is respected, which is not the case usually in the graphs presented on the different web sites and in the media.

Questions of methods

There was an interesting debate in Scotland on the weighting of opt-in web polls according to the recall of previous votes. In short, the question pertained to which election(s)  should be used for weighting. John Curtice's appraisal of the debate -- that took place between Survation and YouGov -- can be found here: Who is right? Survation or YouGov?  I quite share his appraisal that a)  the difference in estimates between YouGov and the three other pollsters using opt-in panels (ICM, Survation and Panelbase) cannot be due only to weighting and b) we do not have sufficient information to assess the cause(s) of this difference.

However, the interesting point here is that we have a systematic difference within the opt-in panel method. Therefore, we may think that there are specific features of opt-in panels (recruitment, selection, administration) that lend to different estimates. Usually, I check whether there is a difference between opt-in panels and other methods (I did find some in most elections held in Canada --see here and here). In the Scottish situation, I will also check whether there is a significant difference between YouGov, Ipsos-Mori (telephone) and TNS-Sofres (face-to-face) on one side and the opt-in panels carried by ICM, Survation and Panelbase on the other side.

I also address the question of whether the wording of the question makes a difference and whether the proportion of non-disclosers is related to support and to methods.



a) Administration mode

Controlling for time, opt-in web polls in general (including YouGov) estimate support for the Yes side, 3.1 points higher, on average, than the telephone and face-to-face polls. However, if we group YouGov with the telephone and face-to-face polls --which is justified by analysis of variance -- we conclude that the "other" opt-in web polls (Survation, ICM and Panelbase), estimate the proportion of Yes 4.6 points higher than the other polls.

In short, there is clearly a difference according to administration mode but YouGov's estimates are closer to the telephone and face-to-face polls' estimates than to the other opt-in panels.

b) Question wording

Question wording is related to mode of administration but it differs also between the different pollsters. I checked for two possibilities, i.e., whether the fact that Yes or No is specifically mentioned in the question makes a difference and whether the fact that the question asks how the respondent would vote now vs on September 18 makes a difference.

Mentioning specifically Yes or No does not make any difference, most probably because it is not relevant in Web Polls and because both Ipsos-Mori (telephone) and TNS-Sofres (face-to-face) mention Yes or No in their question.

The impact on the level of support of asking how the respondent would vote "now" vs "on September 18th varies depending on the variables that are present in the analyses. For now, I conclude that the difference noticed is more related each pollster's "cluster of methods" than to question wording itself.

c) Non disclosers

If there is one main difference between the pollsters, it is in the proportion of non-disclosers. All the firms differ significantly from each other in this regard. TNS-Sofres (face-to-face) has the highest proportion of non-disclosers -- an average of 28% -- while Ipsos-Mori (telephone) has the lowest -- 9.5%. One may ask whether TNS could reduce its very high proportion by improving its methods using for example, a secret ballot or a leaning question. The web polls  have proportions that are in between, from YouGov at 11.7% to ICM at 18.6%. Notice that, during the last month of the campaign in the Quebec 1995 referendum, the level of non-disclosers (telephone polls only) was around 13% on average and that the final turnout reached more than 93.5% (!).

It is also interesting to notice that:

• There is no relationship between time and the proportion of non-disclosers. This proportion is solely related to the pollsters. It is a question of methods.
• However, asking for voting intention "today" instead of "in September" is related to a decrease in the proportion of non-disclosers of more than 5 points. This is related to the different pollsters but is nonetheless very significant.

Conclusion

It is interesting to notice that the difference between opt-in web polls and other methods, and particularly between some web polls and other methods, is systematic and goes in the anticipated direction, i.e., a higher estimation of support for the Yes side by the opt-in web polls. It should also be noticed that there is some research to be carried on why some web polls' estimates differ systematically from others.The other important point is the substantial difference in the proportion of non disclosers. This is clearly a question of methods since the proportion does not decrease with time. However, since it is not related to the proportion of support for each side, it has no clear impact on the estimate of support. Finally, as we get closer to the date of the referendum, the impact of asking voting intention for today vs for referendum day may fade out.

We are getting to the last month of the campaign. I will go on monitoring whether the impact of methods changes in the last stretch. However, in my next post, the focus will be on comparing Scotland with Quebec, and specifically with what happened during the last month (the official campaign) of the Quebec 1995 referendum.

Scottish Independence & the Young people

 Hi,

The question of young people's vote is interesting sociologically and politically. We expect young people to be more favorable to the Yes side in a referendum on independence because it proposes change. Young people may be more interested in change than in status quo. They may also be more easily convinced since they have not yet made up their mind definitely on many political issues. On the other hand, young people are accustomed to live in a globalized world. We may expect them not to be that interested in national States.

So what happens? I present first the situation in Scotland and then compare it with what happened and is now happening in Quebec regarding support for sovereignty among young people.

Scotland

The following graph shows support for independence from January 2014 to the most recent polls among the whole population and among the 16 to 34 years old (purple line ). I computed a weighted average of the results by age group (age group by sex for Panelbase) presented by the different firms in order to arrive at a combined 16-34 age group*.

The graph shows that support for independence is slightly higher among the 16-34 age group than among the general population. However, the difference is rather stable.

Quebec

The following graph shows the change in support for sovereignty for the three age groups between 1979, just before the first referendum in 1980, and 2014. The analysis is restricted to French-speaking Quebeckers -- usually defined as persons who speak mostly French at home -- since non-French speaking Quebeckers were generally opposed to sovereignty whatever their age.

The graph shows that before the first referendum in 1980, support for sovereignty among the 18-34 age group (47% of the population at that time) stand out compared to support among the two other age groups. It is at 63 percent compared to 36 percent for the 35-54 age group and 22 percent for the 55+ age group. The young people who were favorable to sovereignty in 1980 were still favorable in 1995 -- they constitute most of the 35-54 age group -- and they were joined by the new young people so that both the 18-34 and the 35-54 age groups were very favorable to sovereignty, at more than 63%, while support among the 55+ age group was at 40%.

This gave rise to the "dying federalist hypothesis" which stated that, as older people die, if  young people are still be more sovereigntist than older people and stay so as they grow older, sovereignty will eventually -- mathematically -- get the support of the majority of the population. This is not what happened. As we can see, after 2010, there is no more difference in support for sovereignty according to age which means that a) young people are not more sovereignist than the elder anymore and b) people do not necessarily stay sovereignist all their life. They may change their mind at some point.

In a presentation at IRPP in 2001, I also showed that only the less than 55 years old were influenced by the campaign in 1995. Support among those who were 55 years and older remained stable.

Another way to look at support for sovereignty according to age is illustrated by the following graph. The two upper lines -- blue for the 18-34, green for the 35-54 -- show that support for sovereignty increased among all French speakers after the 1995 referendum where the Yes side was defeated 50.5% to 49.5%. It started to decline around 2002, went up a bit around the Sponsorship scandal in 2004-2005, and finally went on declining. The point is that it declined more among the younger age groups than among the 55+ age group so that, in 2014, there is no difference left between the age groups.

Conclusion

The situation in Scotland seems closer to the actual situation in Quebec than to the situation that prevailed in Quebec on the eve of the two referendums. Support for independence does not differ much between age groups and it changes similarly among the young people than among the general population. The hypothesis that young people do not differ much from older people in their political positions is the most plausible for now. 


* Methodological note: Sample sizes of smaller age groups -- 16-24, 25-34 -- were way too small to use the results with some confidence. These results vary enormously from one poll to the next and from one age group to the next. In addition, there were some differences in the age groups used by the different firms. I computed a weighted average of the support of the 16-24 and 25-34 age groups. For YouGov -- who uses a 25-39 age group -- I considered it to be an estimate of the support among the 25-34 age group and I applied the same weights as for the other firms. For Panelbase, who presents results for males 16-34 and females 16-34, I used a weight according to sex.

Scottish Independence, Quebec Sovereignty, similar or not?

Hi,

First, my apologies to French-Speaking followers since I am going to write my messages on the Scottish referendum solely in English. And apologies to English speakers too: My English writing is not perfect.

The referendum campaign for Scottish independence is under way. It is the first time where it is possible to compare a political campaign with the referendum campaign pertaining to Quebec sovereignty in 1995. The Quebec referendum was held on October 31th 1995 and ended with a very close result, i.e. 50.5% No and 49.5% Yes. The turnout was 93.5%. In both situations, the movement for Independence is taking place in a developed country that is part of the Commonwealth.

For this first post, I will compare the Scottish polls with the Quebec polls during similar periods and I will address the question of attribution of preferences to non disclosers, i.e. respondents who do not reveal a preference or state that they will not vote. In following posts, I will address the question of whether there are differences between polls' estimates according to the methodology used --  mode of administration, question wording, etc. -- and I will deal with the question of whether young people's preferences differ from older people's.

What is the state of public opinion? 

a) Including non disclosers


The following graph shows the evolution of support for Scottish Independence. The non-disclosers -- all those who say they don't know how they will vote or state that they and will not vote -- are  included in the graph.

It shows that although there was a possible increase in support for Independence from January to March, support seems to be going back to what it was at the beginning of the year. In addition, we see that there is much variation between the estimates from the different pollsters. The proportion of Yes is at about 35% but it varies between less than 30% and close to 43%. The proportion of No is close to 50% on average but the range of estimates is also large, i.e. 10 points. Finally, the proportion of non disclosers stays between 10% and 20% except for some specific polls where it reaches 30% or more. Those polls were generally conducted face-to-face.

If we look at a similar period in Quebec, i.e. January to June 1995 -- the referendum was to be held on October 30th -- we get a quite different portrait. Notice that since the question wording had not been decided yet at that time in Quebec, pollsters used different wordings and tested support for different options and according to different wordings. In order to have enough data as well as a relatively homogenous information, I kept the estimates for all the questions pertaining to a vote for either sovereignty per se or sovereignty with an association with the rest of Canada. Here is the portrait:

In the Quebec situation, we notice first that the Yes and No sides were closer than it is in Scotland and second, that the proportion of non-disclosers decreased in the last weeks before summer with a concurrent increase in both Yes and No support.

b) A few words about the non-disclosers

Is there a relationship between the proportion of non-disclosers in the polls and the distribution of Yes and No? The clear answer is that for these periods, there is absolutely no significant  relationship between the proportion of non-disclosers and the distribution of support for Yes and No in Scotland as well as in Quebec. In Quebec however,  following suggestions from researchers -- Maurice Pinard, Pierre Drouilly -- based on empirical findings, there has been a tendency to attribute more non-disclosers to the No side -- between 67% and 75% -- than to the Yes side. The rationale for this practice was that the socio-demographic profile of the non-disclosers is more similar to the profile of No voters -- more likely to be older, for example -- than of Yes voters. This practice allowed for a very good estimate of the referendum results in 1995. Without it, the predictions would not have been accurate.

Should Scottish pollsters proceed to a non proportional attribution of non-disclosers? This question should be examined. In any case, it is highly possible that the No side is underestimated since when polls are not accurate, it is usually the status quo side, the more conservative side of the electorate, that is underestimated.

c) Excluding non-disclosers, with proportional attribution

If we exclude the non-disclosers, we have to decide how to attribute them between Yes and No. The common use is to do it proportionally. The hypothesis here is that the preferences of those who do not reveal a preference are similar to to the preferences of those who do reveal them. Notice however that this means that when the proportion of No is close to 60%, proportional attribution gives to the No side a proportion that is close to the non proportional distribution that was used in Quebec. In short, non proportional attribution has a more substantial effect when the distribution of Yes/No is close to 50%.

Here is the portrait of the evolution of support for Independence in Scotland with proportional attribution of non-disclosers:

 It shows the No  side between 55% and 60% with the difference between Yes and No being close to 15%, and no overlap of Yes and No, whatever the poll.

Now, let's see what was the situation in Quebec at about the same period with a similar proportional attribution of non disclosers.  The graph shows that at most, there was a 10 point gap between Yes and No around the middle of March. By May, the gap had closed and, by the end of June, the difference between Yes and No was 5 to 6 points .

Conclusion:

In Quebec as in Scotland, the campaign started with a deficit for the Yes side. However, the deficit was clearly less substantial in Quebec than it is right now in Scotland. Campaign dynamics vary within and between countries. It will be interesting to follow the Scottish situation and examine the methodological questions that it will raise. Will opt-in panels -- more than two-thirds of the polls -- overestimate the Yes side? Will polls in general be accurate using a proportional attribution of non disclosers? How will young people vote? Will they be much more favorable to Independence like in 1995 in Quebec or will they have similar preferences as the other age groups, like the current situation in Quebec?


P.S. Methodological information for specialists: The lines -- the likely evolution -- are estimated  using weighted local regression (loess) with Epanechnikov estimation using 65% of the points for adjustment. In various trial and error with different "smoothers", it appeared the best ans most flexible compromise and it gave very good results in the last Quebec election in 2014.

Post-Mortem Ontario: Who's right?

Hi,

How did the pollsters fare in this election? There are a number of different ways to look at the situation. The first is to see whether the prediction that I made using all the polls' estimates was accurate.  Here is the graph with the election results:

 It is clear from this graph that the Liberals were globally well predicted by the polls but that, contrary to usual, the PC was overestimated and the NDP underestimated. This a rather rare situation. In the last federal election of 2011, the polls had underestimated the PC by more than 4 points in Ontario.

Who's right? 

There is a debate around methods and pollsters. Who fared better? There are known ways to deal with this. One is to calculate the difference between the results and the estimates from the polls. The relative error (results minus estimates) tells us whether a party tended to be over or underestimated by the polls. A positive result means that the party was underestimated by the polls. The absolute error infoorms on the level of the difference between the results and the estimates. The total absolute error gives the synthetic information on the global level of error. It is computed as the sum of the absolute error for the three main parties. I present the results grouped by method (IVR vs Web) for the polls conducted from June 5 in the table at the end of the message.

What the table and my analyses show is:

• IVR polls got the Liberals right (mean error of -0.34) but they overestimated the PC by 3.6 points and underestimated the NDP by 5.1 points on average.
• Web polls underestimated the Liberal party by 3.7 points on average and overestimated both the PC (2 points) and the NDP (2.3 points).
• The difference between the estimates of IVR polls and Web polls is significant for the Liberals and the NDP. We are still in a situation where Web polls tend to overestimate the NDP.
• As for the absolute difference, it is rather large -- between 2 and 5 points on average. There is no significant difference between the IVR and the WEB polls.
• If we look at the total absolute error, it is 10.8 points on average for the IVR polls and 9.6 for the WEB polls. Again, there is no significant difference between these two averages.

A total absolute error of 10.8 is rather high. It is however lower than what has been seen in BC2013 and Alberta 2012.

In conclusion,

The surprise with this election is the global underestimation of the NDP and the overestimation of the PC. I do not see how to explain this. The only expected bias, i.e. an overestimation of the NDP vote and an underestimation of the PC vote by the WEB surveys indeed happened. The huge underestimation for the NDP vote by the IVR polls will have to be analyzed by the pollsters. In addition, the "likely voter models" did not seem to perform adequately. It is good that pollsters who used these models indeed published also their figures for eligible voters. Analyst and voters could see that polls allowed for different estimates according to different hypotheses about who is participating in the election. However, the likely voter models do not seem to be "the solution" to bias in polls' estimates. Pollsters who use IVR should perhaps think about introducing cell phone numbers in their samples and see whether it improves their estimates.

The final point to notice -- it has been noticed by many analysts -- is the substantial variation between estimates. This means that the methods right now are not at the level of reliability that we would like to see them.

Error of the polls - 2014 Ontario Election
Method	Pollster	mid-date of field period	error LIB	error PC	error NPD	Abs. error LIB	Abs. error PC	Abs. error NDP	Total absolute error
IVR	Forum	06-JUN-2014	-0.3	-5.8	6.7	0.3	5.8	6.7	12.8
	Ekos	07-JUN-2014	2.8	-4.7	6.1	2.8	4.7	6.1	13.6
	Forum	10-JUN-2014	-3.3	-3.8	4.7	3.3	3.8	4.7	11.8
	Ekos	11-JUN-2014	1.4	-0.1	4.5	1.4	0.1	4.5	6.0
	Forum	11-JUN-2014	-2.3	-3.8	3.7	2.3	3.8	3.7	9.8
	Mean		-0.34	-3.64	5.14	2.02	3.64	5.14	10.80
	Median		-0.30	-3.80	4.70	2.30	3.80	4.70	11.80
	Std. error of mean		1.13	0.96	0.55	0.53	0.96	0.55	1.36
WEB	Ipsos	05-JUN-2014	3.7	-3.8	-2.3	3.7	3.8	2.3	9.8
	Abacus	06-JUN-2014	4.7	0.2	-4.3	4.7	0.2	4.3	9.2
	Angus Reid	09-JUN-2014	2.7	-0.8	-2.3	2.7	0.8	2.3	5.8
	Ipsos	09-JUN-2014	5.7	0.2	-6.3	5.7	0.2	6.3	12.2
	Leger	09-JUN-2014	1.7	-5.8	3.7	1.7	5.8	3.7	11.2
	Mean		3.70	-2.00	-2.30	3.70	2.16	3.78	9.64
	Median		3.70	-0.80	-2.30	3.70	0.80	3.70	9.80
	Std. error of mean		0.71	1.20	1.67	0.71	1.13	0.74	1.09
Total
	Mean		1.68	-2.82	1.42	2.86	2.9	4.46	10.22
	Median		2.20	-3.80	3.70	2.75	3.80	4.40	10.50
	Std. error of mean		0.92	0.77	1.49	0.50	0.74	0.49	0.84