US 2020 - no 8. Two days after: did all the polls miss the target?

 HI,

 

My last message, titled "I think Biden will win", was posted on Monday, at midday. It included the polls published before 11AM. From 11AM Monday to Tuesday morning, 16 new polls were published on different sites. These polls confimed that support for Trump was increasing. Therefore, I tweeted on Tuesday that I still thought that Biden would win but that the results would likely be closer than expected.

Now that most of the votes are counted, we can compare the polls and the vote. The counted vote gives 50.4% to Biden and 47.9% to Trump, as I am writing this post. Please note that the proportion of votes for Biden is likely to increase with the counting of the votes and therefore, the performance of the pollsters will eventually appear better than now. In all the graphs, I used the proportional support for each candidate (support for one candidate over the sum of the two) in order to compare likes with likes: Some pollsters do not publish estimates of support for small candidates and even the proportion of undecided. Therefore, the published figures are not comparable. Therefore, I use the following figures for the reported vote: 51.3% for Biden and 48.7% for Trump. This is the proportion of the vote for each over the sum of the vote for the two of them; it allows for comparing likes with likes.

The first graph shows the trends in support for the two candidates when I integrate the 223 polls published from September 1st. As we can see, collectively, the polls missed the target, even if we take into account the late increase in support for Trump. However, as we can also see by the dots what are close to the election results, some polls seem to have performed better than others.

What about modes?

The following graph shows the estimates according to the main mode of administration used by each  pollster. As we can see, not all the polls were wrong. The two upper lines portray the trends traced by IVR polls and the dotted middle line, the trends traced by live interviewer polls. The trends tend to show that these polls were generally more accurate, and within the margin of error. These polls account for only 20% of the polls overall (14% live interviewer and 6% IVR) but they account for almost all the polls that performed better. Not all the telephone polls performed well and not all the web polls performed badly. However, there is a rather strong relationship between mode of administration and estimates of support. In addition, the telephone polls portrayed changes in support that were missed by the web polls. Therefore, a global average of the polls that do not take mode into account is misleading.

Since IVR pollsters cannot call cell phones, they complete their samples using another mode, usually web among cell only users. Other pollsters also combine modes, usually live telephone and web. In the preceding graph, we attributed the mode according to the main mode used. In the following graph, we show the trends according to the use of mixed mode. As the graph shows, there is indeed a tendency for polls using mixed modes -- the upper dotted line -- to be more accurate.

Conclusion

The proportion of web polls went from 50% in 2016 to 80% now. The use of averages that do not take modes into account -- by aggregators for example -- contributed to the impression that all the polls had been wrong. I do not know how and if it can be fixed in the next election but I think it ought to.

The fact that more pollsters resorted to mix modes is good news in my view. It seems difficult to represent the whole population of the United States using only one mode. Web modes cannot reach close to 15% of the population, which does not have internet access. IVR polls cannot reach cell only phone users withour resorting to another mode. The live interviewer polls can reach almost everybody but they are expensive.

What are the factors that explain differences between modes? It is difficult to partial out the impact of the cluster of methods that come with each mode and, sometimes, with each pollster. Modes vary in their sampling source, in their coverage and in their weighting. They also differ in terms of the length of the questionnaires they use, the way they ask their questions, the order of the vote intention question, etc. It remains that the modes that resort to probibilistic or quasi-probabilistic recruitment tended to perform better. The reports show that some web pollsters are trying to improve their methods to integrate some randomness but the estimates for this election show that there is still work to do. The AAPOR committee that will examine the poll performance will no doubt look at all these features.

Most of the methodology reports provided by pollsters were good enough to get sufficient information on what they do, which is good news. There are seven pollsters for which it has been impossible to find any methodological report. Some of them are academic pollsters or work for established media. This is difficult to understand.

Anybody outside the U.S. would be very surprised, I believe, by the proliferation of pollsters. There are 54 pollsters who published polls since September 1st. Among them, 21 conducted only one poll and eight conducted two polls. Five of them appeared in the last week of the campaign. I checked and did not find any difference between their estimates and the estimates of more established pollsters. However, it is more difficult to find information about some of these pollsters and to check on their work. It may become a concern when electoral campaigns are heated. Some may be tempted to publish biased or fake polls because they think it may help their preferred candidate.

We thank Luis Patricio Pena Ibarra for his meticulous work in entering the data and gathering the related information to help me conduct these analyses.

Methodology for the analyses presented in this post.

1) The trends produced are produced by a local regression procedure (Loess). This procedure gives more weigth to estimates that are closer to each other and less to outliers. It is however rather dependent on where the series start. For example, I started this series with the polls conducted after August 1st. This means that all the polls conducted since then influence the trends. If I start on September 1st, the trends are somewhat different because of the absence of the influence of the polls conducted in August. I try to balance the need to have enough polls for analysis and the importance of not having old information influence the trends too much.

2) Every poll is positioned on the middle of its fieldwork, not at the end of it. This seems more appropriate since the information has been gathered over a period of time and reflects the variation in opinion during that period.

3) All the tracking polls are included only once for each period. A poll conducted on three days is included once every three days. In this way, we only include independent data. This is appropriate in statistical terms.

4) The data used comes from the answers to the question about voting intention.

5) For non-Americans, note that, in the USA, IVR ("robopolls") cannot be used to call cell phones.U.S.A. seems to be the only country that has this restriction.

I think Biden will win

 Hi,

I rarely commit myself to predict election results. This time, I am quite confident. And I know I will feel very bad on the 4th -- or when the final results are known -- if I am wrong... Of course, I examined only the national polls but if Biden ends with less electoral votes and a larger proportion of the popular vote than Trump, it would mean that the system really needs to be changed.

I first present the graphs and then, I explain why I think Biden will win. I included all the polls published before 11 this morning. Here is the first graph with all the polls. It shows that there might be a sligth decline in support for Biden recently. If this decline is confirmed however, he would end with the same support as in the beginning of September. As we will see in the next graph, not all the modes estimate that there is such a decline. The difference between the support for Biden over Trump is still at about eight points. Just before the election in 2016, the difference between support for Clinton and for Trump was closer to five points, using the same methodology.

Let us examine now the graph by mode of administration. We still observe a difference in trends by mode. However, with the integration of the recent polls, the difference is not as large. Live telephone  and IVR polls see a recent increase in support for Trump. There are not that many IVR polls so that any new poll may modify the trends. 

In summary, according to the web polls (close to 80% of the polls), proportional support for Trump has stabilized at around 45% since mid-October. Some of these polls have higher estimates, at around 48%, others, lower estimates, at 42%. According to live interviewer polls  (15% of the polls), support for Trump has increased recently, at around 47%. The only polls that estimate the difference in support within the margin of error are the IVR polls (6% of the polls), and they do not estimate that this support is still increasing.

 

Why I think Biden is likely to win

First, the AAPOR report on the 2016 polls has shown that most of the 2016 national polls conducted during the last week before the election in 2016 were within the margin of error. It identified the lack of weighting by education in the state polls as a possible explanation for their not so good performance. The pollsters seem to have adjusted for that. In addition, some pollsters weight for addtional variables like household size, the presence of children, the vote cast at the last elction. The committe concluded that a lower turnout among Clinton supporters was a likely explanation for the difference between the polls and the vote in some states. This does not seem to be happening in this election. The turnout will  likely be much higher than in 2016.

Second, a polling miss can come from two main factors: the poll methodology and people changing their minds (or not revealing their preference). In the 1980's and 1990's, the pollsters used very similar methodologies. Therefore, methodological factors could be a likely explanation for polling misses. Right now, the methods used are so diverse that it is unlikely that methods would lead to a polling miss. We can also exclude a last minute shift -- unless something incredible happens after I finish writing this post. Even then, so many people have already voted and cannot change their vote that a major shift is almost impossible. A last minute shift in voting intention has very rarely been proven. Sturgis -- Report of the British Polling Council -- checked for a late campaign shift to explain the poll failure in the UK in 2015 and concluded that no such movement occurred. The only case I know of is Quebec 2018 polling miss where there was a substantial move from one of the two major parties to the other during the last days of the campaign (Durand and Blais, 2020).

Now, there is one question left. How come IVR polls show such a difference with other polls. Could they be right and all the others wrong? IVR polls have often performed very well. However, in all these cases, their estimates were not far from the estimates produced by the other modes. In this election, after closing the gap with other polls at mid-campaign, they end with estimates that are four points higher than web polls and more than two points higher than live interviewer polls. There are unfortunately not enough polls using mainly this mode to see if it is due to the pollster or to the mode itself.

In summary, since September 1st, we identified 207 national polls conducted by 54 different pollsters with various methodologies. Only 12 of these polls are mainly IVR polls, 30, mainly Live interviewer polls and 16% of the polls combine different modes. It appears to me very unlikely that the large majority of these polls and pollsters are wrong. My only caveat is that we need to keep in mind that web polls seem to have difficulty detecting change.

Conclusion

For Donald Trump to win, it would be necessary that more than 50 different pollsters with different methodologies be wrong. This is highly unlikely. Electoral polls are often the best polls that you can get because pollsters know that their estimates will be compared with the final results (and with their competitors). Their performance testify of their competence. They are therefore very careful and they try to improve their methods all the time. However, there are many "one-shot" pollsters in this election (four new ones last week). I checked on their estimates. They are similar to those of established pollsters. Finally, my precedent blog showed that, even when we "load" the estimates in favor of Trump to simulate a "shy Trump" effect, we are still led to conclude that Biden is sufficiently ahead to win.

We thank Luis Patricio Pena Ibarra for his meticulous work in entering the data and gathering the related information to help me conduct these analyses.

Methodology for the analyses presented in this post.

1) The trends produced are produced by a local regression procedure (Loess). This procedure gives more weigth to estimates that are closer to each other and less to outliers. It is however rather dependent on where the series start. For example, I started this series with the polls conducted after August 1st. This means that all the polls conducted since then influence the trends. If I start on September 1st, the trends are somewhat different because of the absence of the influence of the polls conducted in August. I try to balance the need to have enough polls for analysis and the importance of not having old information influence the trends too much.

2) Every poll is positioned on the middle of its fieldwork, not at the end of it. This seems more appropriate since the information has been gathered over a period of time and reflects the variation in opinion during that period.

3) All the tracking polls are included only once for each period. A poll conducted on three days is included once every three days. In this way, we only include independent data. This is appropriate in statistical terms.

4) The data used comes from the answers to the question about voting intention.

5) For non-Americans, note that, in the USA, IVR ("robopolls") cannot be used to call cell phones.U.S.A. seems to be the only country that has this restriction.

U.S. 2020 no 6 What about the "Shy Trump" hypothesis?

Hi,

 

In 2016, just before the election, I tested the hypothesis of a "Shy Trump" effect. I personnally prefer to speak about a possible underestimation of the support for Trump since I think that it is possible that  different effects are cumulative: a possible under-representation of Trump supporters in the samples, a possible lower tendency of Trump supporters to answer the polls when they are contacted, and a tendency not to reveal their preference. These effects may not play in the same way and at a similar level for different modes of administration.

In order to simulate a "Shy Trump" effect, I proceed in the same way as I did in 2016. This procedure led me to forecast that the two candidates were quite close than we thought at that time and that Trump could win. I computed the sum of all the undecideds, abstainers and those who say that they support other candidates (a very small proportion). I add these three components because it is easy to figure that "shy respondents" may as well say that they will not vote or that they will vote for another candidate. 

This sum varies from 0 to 23%. It is at 7.5%  on average, at 5.3% for IVR polls, 5.9% for live interviews and 8% for web polls. The fact that it is higher for web polls may say something about the plausibility of a shy Trump effect but it could also be due to the composition of the web samples.

I allocate 67% of that sum -- that we will call the non-disclosers -- to Trump and 33% to Biden. Clearly, the hypothesis is heavily "loaded" for Trump. This would probably be the worst situation.

Now here is the graph that I get after proceeding to that allocation. We now have support for Biden at around 53% and for Trump at 47%. We still have a lead of six points for Biden. We see however that some polls have Trump leading over Biden. Let us look at these estimations by mode.

The next graph shows the estimates of support for the two candidates, with the same non-proportional allocation. As we can see, the web polls estimate the support for Trump at close to 47% but the live interviewer polls' estimate is closer to 48% and the IVR estimates are at 50%. We can conclude that, if there is a substantial underestimation of the support for Trump, "objects may be closer than they appear". We will still have to understand, however, why the telephone polls estimate that support for Trump has been increasing lately when WEB polls show, on the contrary, that it has decreased.

Conclusion

These analyses do not take into account the fact that, if there is an underestimation of the support for Trump, whether it is due to a "shy Trump" effect or to other factors, these effects may occur "in context". They may be more substantial in states or areas that are more pro-Biden for example. One could also argue that a similar "shy Biden' effect is present in states or areas that have a concentration of Trump supporters. This would mean that the two phenomena may cancel each other. We should also remind ourselves that "shyness" is not that likely to occur in polls conducted without interviewers, that is, WEB and IVR polls. Those polls account for 85% of the polls published since September 1st.

We thank Luis Patricio Pena Ibarra for his meticulous work in entering the data and the related information to help me conduct these analyses.

Methodology for the analyses presented in this post.

1) The trends produced are produced by a local regression procedure (Loess). This procedure gives more weigth to estimates that are closer to each other and less to outliers. It is however rather dependent on where the series start. For example, I started this series with the polls conducted after August 1st. This means that all the polls conducted since then influence the trends. If I start on September 1st, the trends are somewhat different because of the absence of the influence of the polls conducted in August. I try to balance the need to have enough polls for analysis and the importance of not having old information influence the trends too much.

2) Every poll is positioned on the middle of its fieldwork, not at the end of it. This seems more appropriate since the information has been gathered over a period of time and reflects the variation in opinion during that period.

3) All the tracking polls are included only once for each period. A poll conducted on three days is included once every three days. In this way, we only include independent data. This is appropriate in statistical terms.

4) The data used comes from the answers to the question about voting intention.

5) For non-Americans, note that, in the USA, IVR ("robopolls") cannot be used to call cell phones.U.S.A. seems to be the only country that has this restriction.

U.S. 2020 no 5. Where are we now?

 Hi,

 Five days before the election, and seven days after the second debate, it is time for an update on the polls.  We counted 20 polls that atarted their fieldwork after the debate. Among them, 11 are classified as web polls, 6, IVR and 2, Live phone. We did not find the mode of administration for one poll. This is quite different from the overall distribution since September 1st: 79% Web polls, 14% Live phone and 6% IVR. There are 186 polls total since then.

After reading through each pollster's methodology, I made a few changes in the classification. At first, I had classified all the polls who used IVR as IVR polls. However, some of them have a larger part of their sample coming from web than from IVR. I reclassified them as Web. I also reclassified IBD-TIPP who uses live phone and web as Live phone because the major part of the sample comes from Live phone interviews. NORC also has a small part of its Amerispeak web poll for which interviews are conducted by live interviewers. As you can see, change in methods in this election as compared to 2016, is mostly in the greater reliance on mixed modes. So I decided to compute a new variable to compare the polls using mixed modes and the others.

Where are we now?

The following graph shows that, a few days before the fist debate, support for Biden increased by about one point overall. Since then, support for him is stable or very slightly declining according to the polls. As of yesterday, his support would be at around 55% of voting intentions compared with 45% for Trump.

 

Let us now examine the difference between modes of administration. As we can see in the following graph, we are still in a situation where the web polls do not show the same trends as IVR or Live interviewer polls. There are a lot less telephone polls than web polls, which means that we have to be careful and not jump to conclusions. However, the portrait is consistent. Polls that mainly use the telephone to reach respondents show an increase in support for Trump while web polls don't. On the opposite, these polls estimate that the support for Trump decreased a little lately. In addition, we still observe that the IVR polls estimate the support for Trump at almost five points more than the web polls, at close to 50%. The live interviewer polls are in between. the IVR polls' estimates are significantly different from the other polls' estimates.

We also wanted to examine whether the use of mixed modes made a difference. The portrait is similar to what we see according to modes. Of course, mixed modes are mostly used in polls that are mainly IVR or live phone but some mainly web pollsters also use mixed mode. On average, polls using mixed modes estimate that the support for Trump has been increasing lately while those using only one mode tend to estimate a slight decline.

If I use a regression to predict support for Trump, controlling for the number of days each poll is in the field and for when the poll was conducted, the effect of mode is the following: The web polls estimate the support for Trump 1.9 points lower than the IVR polls and the Live interviewer polls, 1.6 points lower. When polls use mixed mode, their estimate is 1.2 points higher.

Conclusion

A number of reasons can explain differences in trends and in forecasts according to mode, among them, the source of the samples, the weighting used, the question wordings and the presence of a leaning question, the length of the questionnaire and the number and type of questions that are asked before the voting intention question. There are also substantial differences between web pollsters. The methods used to recruit participants vary from probabilistic panels recruited though telephone or face-to-face interviews, to opt-in panels, to platforms like Lucid or MTurk to recruitment via adds on apps. Whether there are differences by mode of recruitment is another area to explore. I was impressed to see that, although we still have methodological reports missing, the majority of the polls now come with rather detailed information on the methodology. 

In my next post, I will look at the "Shy Trump" hypothesis.

US2020 no 4. Who can we trust?

 Hi,

In my fourth post on the polls of this election, 13 days before the election and two days before the second debate, I tried to go deeper into differences between modes, but also between pollsters.

First, let us have a look at the graphs

 The first graph shows the trends in proportional support for Biden and Trump. We need to use this "proportional support", that is the proportion of each on the sum of the two, because there is substantial variation in the proportion of undecideds by pollster and by mode. It ranges from 0% to 12% since September 1st.  It is on average at 2.9% for IVR-misec mode polls, 3.2% for Live interviewer polls and 4.6% for Web polls.

So here are the trends since September 1st.The difference between the support for the two candidates has increased somewhat, from eigth points at the beginning of September to a bit more than 10 points, on average. Biden is clearly ahead in all the polls, except for a few outliers that estimate that the two candidates are very close, this even in October. There are also outliers who estimate the difference between the two candidates at more than 20 points. We will come back to this later on.

 
Let us now look at the trends by mode.

As we can see, there remains a difference in trends by mode of administration but the discrepancy between modes is becoming smaller. We are heading towards a situation that is very similar as 2016, that is, no difference between web and live interviewer polls (estimate at 45%) and estimates of IVR polls that are higher for Trump (around 46%). However, we see some "outliers", particularly among the web polls. We also still see that the web polls trace a much more stable trend than the telephone polls. In addtion, the decline in support for Trump after the first debate now appears very clearly, whatever the mode use.

Of polls and pollsters

Some people -- all people? -- tend to do some "cherry-picking" of polls. They look only at the polls that are good news for their preferred candidate. However, some pollsters' estimates are quite different from the majority of the other estimates. 

There were 42 pollsters conducting polls since the beginning of September. Of those, 19 (42%) conducted only one poll. these "one shot" pollsters are much less likely to comply with AAPOR's transparency guidelines. For six of them, we do not even know the mode of administration used. One pollster states "The survey included mobile phones, landlines and online" as the only information on the mode of administration. For most of them, we do not  have the question wording. One pollster asks: "Would you vote for President Trump or for Joe Biden?", a clearly biased question. So we need to be careful about these pollsters' estimates.

The following Box and Whiskers plot shows which polls are confirmed statistical outliers and how far these outliers are from the other polls. Survey Monkey's poll conducted from September 30 to October 1st estimates the difference between the proportional support for the two main candidates at 21.5 points. This is an extreme outlier. One AP-NORC's  poll and one Opinium's poll, both conducted from October 8 to October 12, are also outliers with an estimated difference of more than 17 points.  Survey Monkey's poll estimated Trump's support at 39% and AP-NORC and Opinium at 41% while the median of the other web polls is 45%. We cannot really rely on these polls because they are too far from the other polls.

On the other end of the spectrum, one Harris poll, conducted between September 22 and 24 and one IBD/TIPP poll conducted from September 30 to October 1st are also outliers in estimating proportional  support for Trump at close to 49%.

The Box and Whiskers plot (information regarding Box and Whiskers plots can be found here) also confirms that the median -- the horizontal line in the middle -- of IVR-mixed mode polls is higher than the median of the polls using other modes.

Why is that? A number of reasons may explain the differences between modes and between pollsters beside the sources of samples. Among them, variation in question wording. In the vote intention question, some pollsters mention the party for all the candidates, others do not. Some mention the vice-presidents, others do not. Some mention the small candidates, others do not. In addition,  telephone polls tend to use a "leaning question" asked from those who say they are undecided. This tends to reduce the proportion of undecideds. In some very good reports, pollsters reveal the vote of the leaners, a very relevant information. Leaners tend to support Trump more than the respondents who state their preference rigth away. This tends to confirm a possible "shy Trump" tendency. Some pollsters rotate the names of the two main candidates - either Trump or Biden is presented first in the question --, others do not. Unfortunately, information about the use of a leaning question and of rotation is not often mentioned in methodological reports.

The number and type of questions asked before the vote intention question may also have a substantial impact since these questions may influence the answers to the subsequent vote intention question by drawing attention on some issues. The 2020 election may be a good occasion to look at all these methodological features to assess whether they have an impact on estimation. However, for this, we will need the cooperation of the pollsters.

In conclusion

In 2016, the difference between Clinton and Trump was smaller (7 points) at about the same time, that is, before the third debate (see my blog post of October 28 2016 here). The main difference is that in 2016, some pollsters had estimates of Trump and Clinton at par or very close, particularly close to the end of the campaign. In addition, support for Clinton started to decline just before the third debate. In this campaign, since September 1st, one IVR poll has put Trump very slightly ahead of Biden in mid-September and two polls have put the two candidates close to each other at the beginning of October. Since then, polls have been quite unanimous in putting Biden systematically and clearly ahead of Trump. Of course, we know that it is not necessarily enough for Biden to win because of the electoral system and ... because there are still two weeks to go. Meanwhile, we should not "jump to conclusions" when we see polls that fit our view when they differ from other polls conducted at the same time, and even more, when they are conducted using the same mode.

 

 We thank Luis Patricio Pena Ibarra for his meticulous work in entering the data and the related information to help me conduct these analyses.

 

Methodology for the analyses presented in this post.

1) The trends produced are produced by a local regression procedure (Loess). This procedure gives more weigth to estimates that are closer to each other and less to outliers. It is however rather dependent on where the series start. For example, I started this series with the polls conducted after August 1st. This means that all the polls conducted since then influence the trends. If I start on September 1st, the trends are somewhat different because of the absence of the influence of the polls conducted in August. I try to balance the need to have enough polls for analysis and the importance of not having old information influence the trends too much.

2) Every poll is positioned on the middle of its fieldwork, not at the end of it. This seems more appropriate since the information has been gathered over a period of time and reflects the variation in opinion during that period.

3) All the tracking polls are included only once for each period. A poll conducted on three days is included once every three days. In this way, we only include independent data. This is appropriate in statistical terms.

4) The data used comes from the answers to the question about voting intention. Undecideds (non-disclosers) are attributed proportionally -- for now.

5) For non-Americans, note that, in the USA, IVR cannot be used to call cell phones.U.S.A. seems to be the only country that has this restriction.

US 2020 - no 3: Let's talk about methods

 Hi,

In this message, I update the graphs presented in my preceding posts that show the different trends according to mode of administration. I perform this analysis for the polls conducted since August 1st, in order to compare with my previous post, and since September 1st -- we now have enough polls to limit the analysis to more recent polls. I also add a graph showing the trend in the difference between the two main candidates. The analyses are based on the proportion of Trump voters on the total of Trump and Biden voters. This is the only way to have comparable estimates.

I will also have a section about the methods used by the pollsters. This will allow for better understanding what modes of administration may mean.

Recent trends by mode

The next two figures show the trends in support for Donald Trump by mode of administration. The first graph uses all the polls published since August 1st (n=192), the second one, since September 1st (n=120). We could not find the mode of administration for two polls.

The first graph confirms the tendency observed in my preceding posts. The telephone polls using live interviewers (the dotted  line) and the Interactive Voice Response (IVR) polls (the upper line) both show that support for Trump was increasing until it reached a plateau at the beginning of September and is decreasing since. The web polls (the straight pink line in the middle) do not detect such a movement. They account for 77% of the polls and therefore, they have a substantial weight in the graphs shown by most aggregators.

The graph also shows that, after the debate -- the vertical dotted line -- there is much variation in the estimates of the different web polls. They estimate Trump support between 39% -- the lowest estimate -- and 48%. Most of these polls estimate proportional support for Trump at around 45%.

Before you conclude too early, I refer you to a similar graph in my post at the end of October 2016 here. In 2016, the same downward trend for Trump occurred after the first debate but he could come back afterwards.

If we now examine only the polls conducted after September 1st, here is what we get. It is rather clear that the IVR polls detected a downward trend at least a week before the debate. It is also clear that the web polls do not detect any change. In addition, the IVR polls' estimates are now very close to the web polls estimates while telephone polls -- the dotted line -- estimate Trump support lower than the other polls. However, we need to be careful because there are not that many Live interviewer and IVR polls. Since September 1st, they account for respectively 12 percent and 9 percent of the published polls.

Another way of showing the same information is to examine the difference between support for Trump and Biden. The following graph shows that the difference between the two candidates has been growing steadily since the beginning of September. It went from 2% to 10% according to IVR polls and from 7% to 13% according to Live Interviewer polls. For web polls however, it stayed almost stable around 10%.

Let's talk about methods
How can we explain these differences? We need to examine how the different polls are conducted in order to at least propose hypotheses of explanation. We need to realize first that our classification in three modes hides substantial differences between pollsters using the same mode of administration.

Pollsters who use Live interviewers may have different sampling frame. In the Appendix of the AAPOR Report on the polls of the 2016 election, we noticed that the proportion of cell phones in the samples varied from 25% to 75%. This information is rarely presented by the pollsters. However, one pollster informs that its polls include 80% of cell phones. We may think that people who cannot be joined by cell phone have specific characteristics that are related to vote intention. However, the 2016 analyses did not show any impact of the proportion of cell phones in the samples on estimates.

IVR polls in the U.S. should probably be renamed Mixed mode polls. I have grouped in this category all the pollster who use Interactive Voice Response -- also called robopolls -- to recruit at least a part of their respondents. However, because of the interdiction to use automated calls to cell phones in the U.S., the pollsters using IVR complete their samples with other sources of respondents. In 2016, their samples usually combined 80% of automated calls to landline phones and 20% of cell-only respondents recuited via web opt-in panels. In 2020, there is much more variation and  respondents joined via automated calls to landline phones are often a minority. 

Web polls also vary incredibly in the methods used to recruit respondents. It is not always easy to find the information on the samples. Opt-in panels may not be the norm anymore. Some pollsters complete their opt-in panels with river sampling. Some use platforms like Lucid where respondents are customers of different major commerces. Some use ads on cell phone apps. Some combine different sources of respondents in order to balance the biases that may be associated with each source. 

Why don't web polls detect movement where other modes do? It may be due to more homogenous pools of respondents. However, we will need to get more information from the web pollsters to assess whether different modes of recruitment explain the observed difficulty in detecting movements. In 2016, we did not find any significant difference between the polls that use river sampling and the others. The difficulty of web polls to detect movement has been observed in other elections in other countries. Web pollsters will have to work on -- and are probably already working on -- finding the source of the problem and eventually correct it. Of course, they could also be right but it nonetheless needs to be analysed. It is of utmost importance when web polls account for two thirds of the published polls.

In conclusion

Modes of administration are only one of the differences between pollsters. In a way, we assume that the more variation we have, the more likely the biases will compensate themselves and the average estimates will be the most reliable. However, if one mode is very dominant, and there are differences between modes, it becomes more difficult to conclude on what it happening. Transparency becomes even more essential.

I am also working on question wording. I will come back on this question in a future post. There is much variation in question wordings. Unfortunately, it is difficult to find the information on the  wording used for about a third of the pollsters who published estimates since the beginning of September. It is therefore difficult to assess whether wordings lead to different estimates.

We thank Luis Patricio Pena Ibarra for his meticulous work in entering the data and the related information to help me conduct these analyses.

 

Methodology for the analyses presented in this post.

1) The trends produced are produced by a local regression procedure (Loess). This procedure gives more weigth to estimates that are closer to each other and less to outliers. It is however rather dependent on where the series start. For example, I started this series with the polls conducted after August 1st. This means that all the polls conducted since then influence the trends. If I start on September 1st, the trends are somewhat different because of the absence of the influence of the polls conducted in August. I try to balance the need to have enough polls for analysis and the importance of not having old information influence the trends too much.

2) Every poll is positioned on the middle of its fieldwork, not at the end of it. This seems more appropriate since the information has been gathered over a period of time and reflects the variation in opinion during that period.

3) All the tracking polls are included only once for each period. A poll conducted on three days is included once every three days. In this way, we only include independent data. This is appropriate in statistical terms.

4) The data used comes from the answers to the question about voting intention. Undecideds (non-disclosers) are attributed proportionally -- for now.

5) For non-Americans, note that, in the USA, IVR cannot be used to call cell phones.U.S.A. seems to be the only country that has this restriction.

It's all about modes - 2020 Edition - no.2

 Hi everybody,

I thought it was important to follow closely the last polls. Here is what I get. The graphs use all the polls conducted from August 1st to October 5.

The first graph shows the relative support for Donald Trump and Joe Biden. The vertical line represents the debate. It shows that, on average, support for Joe Biden seems to be going up since before the debate but even more since the debate. However, this is the average of all the polls. Therefore, it reflects mostly the web polls because they account for 74% of the polls conducted during the period.

If we examine the trends according to each mode of administration, the portrait is somewhat different. As the following graph shows, the polls that use the telephone, whether with live interviewers or using automated dialing see a clear and more substantial downward trend for Trump than the web polls. And this trend seems to have started way before the debate -- around September 15. As in 2016, web polls do not detect movement where telephone polls do. And IVR polls estimate support for Trump systematically higher than the two other modes.

In conclusion, we should be careful before concluding on the poll trends. The difference between modes persists and it is similar to what we have seen in 2016.

It's all about modes - US 2020 edition

 Hi everybody,

As many researchers, I am following the polls for this election. I will personnally focus on aspects of the polls that are not that much covered, that is the possible impact of the methodological features of the polls. In this blog post, I focus on whether there is a difference in the trends portrayed by different modes of administration. 

On average

As the first figure shows, the polls conducted since the beginning of August tend to portray an estimated difference of eigth points in support between Joe Biden and Donald Trump. We use a common scale for all the polls, that is, support for each candidate on the sum of the support for the two of them. The portrait traced here is not different from the portrait traced by well-known agregators and news media.

 

 

 By mode of administration

However, as in 2016, the trends traced by the different modes of administration tend to differ. IVR polls, that use a combination of automated phone calls to landline phones and web polls to cell-only users, estimate the proportional support for Donald Trump two points higher than the other polls. This was also the case in 2016 just before the first debate between Donald Trump and Hillary Clinton as shown in the AAPOR Report on the polls of the 2016 election. 

As in 2016, the web polls tend to trace a more stable portrait of the trends. They do not seem to detect movement when other modes do detect some. Finally, the polls that use live interviewers show an increase in support for Donald Trump in August, an increase that is also detected by IVR polls.Whatever the mode used however, trends in support for Donald Trump appear stable since the end of August.

Conclusion

Web polls account for more than 70 percent of the polls. Therefore, the portrait that we get if we do not take mode into account is heavily determined by web polls estimates. In 2016 however, the IVR polls had a slightly better estimate of the support for Donald Trump than the two other modes. We need to take this into account.

Of course, there are slso differences within modes. Web pollsters do not use the same sources of potential respondents. These sources may be more or less probabilistic and some combine different sources. This may have an impact. IVR polls also differ according to the pollster who use them, particularly in terms of the proportion of the sample coming from automated calls to landline phones. Finally, the polls conducted using live interviewers differ in the proportion of cell phones in their samples. In 2016, this proportion varied from 25 to 75 percent. I presented here an average impact of modes but there may be differences within modes, a question that I will try to cover in future posts.

A final note to pollsters and journalists. Mode of administration is an essential methodological information. It should be presented with the main information regarding the dates of fieldwork and the sample size. Sometimes, it is hidden somewhere in the pollster report. It shouldn't be.