The general election is over a year away, but that has not prohibited the release of several polls from a specious methodology using the internet as measurement. Part of the reason that more news outlets are using this form of polling is because it’s cheap. It also helps organizations get cheap news publicity in a news cycle.
But are these forms of measurement reliable and generalizable? That’s the question because making informed choices and consuming the news based on some truth and verifiability is necessary for the voters. As political junkies it is nice to make informed rational opinions based on information that can he generalized to the entire populace.
At this point I am not convinced internet polling (non-probability polling) does that despite their grandiose claims.
The New York Times is chief among those offering glowing reports of the use of internet polls. This is because the NYT is the chief news organization that has employed the methodology of internet measurements (like from Google and Yougov). Indeed, the NYT hired one of the architects of the YouGov model to conduct their polls and write about them. Conflict of interest?
Yet the NYT claims, for the 2012 & 2014 cycle, internet polling was more accurate. This MAY be true, but one cycle does not constitute reliability over time. And, despite the problems with some probability polls, it does not mean non-probability polls are better or can even stand the test of time. There are problems with this method:
Internet polls, in most cases, use nonprobability sampling. They exclude households without Internet access; these tend to be older and lower-income Americans. Most online polls are also completed by people who opt to participate. Some participants sign up to complete online polls on websites that offer prizes such as gift cards to chain restaurants and movie theaters. Others are responding to ads placed on other websites that may or may not be related to the poll’s subject, a technique known as river sampling.
Self-selection is not random and a poll needs to be random to be generalizable, as Pew makes clear.
But even in this pro-non-probability story in the WaPo, it is noted the standard espoused by the AAPOR criticizes internet polling for several reasons–including the problems related to probability and openness (read ethical here in terms of verifiability from other scientists).
Pew continues to criticize the NYT decision to use non-probability and remains skeptical, but cautious in dismissing completely:
It’s still the case that people who don’t use the internet are different in many ways from those who do – in particular, they are older, poorer and less educated. But their dwindling numbers mean that their absence from a survey won’t make a huge difference in the findings on most questions. Still, we think it’s important to be able to describe our samples as “nationally representative” and try to make sure that they are whenever possible.
Some say internet users are more honest, and that may be, but what we need to replicate in polls is what happens in the voting booth, and every method will have its errors to contend with no matter the method. However, is the argument that more people are online persuasive? Do not those online have cell phones too?!
Online panels can be useful to be sure. And many businesses swear by them, but we are talking measurement of likely voters in an election. Probability polls have been off and have been spot on in the past. Non-Probability polls claim they have been more accurate than probability in the last two cycles:
It’s important to keep in mind that online non-probability panels vary in quality, just as probability-based surveys do. One of the most important points in the AAPOR Task Force report is that there’s no single consensus method for conducting “non-probability sampling.” There are many different approaches, and most of them don’t have the public record of performance that YouGov has. YouGov has been conducting public polls in elections for many years. As a result, they have a track record that can be compared with probability-based polls. Until we have more organizations conducting polls in advance of elections and explaining their methods in detail, I believe that adoption of non-probability sampling for political polling will proceed slowly.
I am highly suspicious of any self-selected non-probability poll even if those chosen in it are chosen at random. Some academics agree. There can be a lot of nefarious uses of the non-probability poll because it is so cheap–and that makes it like a cheap push poll that takes headlines for a day or so, thus driving the political rhetoric to areas the public really is not concerned or believes. That’s unethical.