Why do polls keep missing the mark? It's not because voters are lying—it's about how polls are built and reported in today's newsroom.
Show More Show Less View Video Transcript
0:00
The Associated Press reporting that Clinton is in a commanding position
0:03
Donald Trump will be the 45th president. Nationally, it shows Joe Biden leading President Donald Trump
0:12
The race between President Donald Trump and Joe Biden still too close to call
0:16
Our new poll with Ipsos has Kamala Harris right now back in front
0:20
The next president of the United States is Donald Trump. Three presidential elections in a row
0:26
The polls pointed one way and reality went the other. Many people boil it down to a simple theory
0:31
People lied. But is that really what's wrong with modern polling? To find out, Straight Arrow turned to a swing state pollster
0:37
one of the nation's leading survey researchers, and a veteran TV news producer
0:42
Polling is strategic intelligence. Are you getting good intelligence or bad intelligence
0:47
And that matters, because if not, you might be surprised. That's why I think a lot of people thought the election was stolen
0:53
Mike Noble of Noble Predictive Insights has been polling in Arizona for more than a decade
0:57
Yes, that Arizona. Arizona, are you 100% sure of that call and when you made it and why did you make it
1:06
Absolutely. He says the problem isn't that people are dishonest. For him, it comes down to who is still willing to talk
1:12
People that have, let's say, a college education or white-collar workers, they're taking surveys like it's a hobby
1:18
They have more flexibility in their schedule. They can sit there and do a 5-10 minute interview. However, what you're missing is that people like more blue-collar jobs
1:26
or high school or less education, do you think they got time to stop in the middle of the thing and do a survey
1:31
So basically response rate-wise, you get a big imbalance there. 30 years ago, about one in three Americans would answer a pollster's phone call
1:40
Now, that number is fewer than one in 50. The polling industry says it's adjusted for that
1:45
but Noble says that's harder than it sounds. Making a poll is like a chef A chef you only see the final meal that comes out But again the ingredients used how he prepares it blah blah blah blah blah That how basically polling does So if you follow this recipe you should get this result
2:02
John Krosnick, a professor at Stanford University, has spent his career studying exactly how people answer survey questions
2:09
So when he went searching for evidence that voters were hiding their true opinions from pollsters
2:14
he kept coming up empty. Researchers who want to claim that social desirability is a big problem can point to those studies as long as they don't point to how big the effects are
2:24
The minute they start paying attention to the fact that the effects are pretty small, then it becomes less of a problem
2:30
When Krasnick and his team reviewed every study comparing what people say in surveys to what they do
2:36
What we discovered is that about 85 percent of people's self-reports in surveys are accurate
2:42
And that's often, that's quite a bit higher than many professionals guess before they know what that literature says
2:49
But what about more sensitive questions, where telling the truth might come back with a social cost
2:54
Krosnick set out to explore the merit behind an oft-touted theory on why people conceal their true views, racial attitudes
3:00
We found no more evidence of racial prejudice being expressed, for example, by white people against black people in the anonymous condition than in the oral condition
3:13
And so that's yet another piece of evidence, in my opinion, that really raises serious questions about the claim that you can't trust self-reports because people are lying
3:24
Despite it being a popular theory, there's weak evidence that people are lying to pollsters
3:29
That raises the harder question. If voters are mostly telling the truth
3:33
why do polls keep getting it wrong? It's the abandoning of random sampling that's the problem
3:39
The night before the 2016 election aggregated polls gave Hillary Clinton a 75 to 80 percent chance of winning Most of those polls were opt surveys made up of people who volunteer to take surveys online sometimes for money In the battleground states that decided the election that year
3:54
roughly 70 polls were conducted in the final week. The scene here is so different than it was a few hours ago
4:01
The random probability samples had an average error in predicting the Trump and Clinton share
4:06
of the vote of less than one percentage point. Really accurate. Whereas the average error of those opt-in sample surveys was much higher and sometimes a great deal higher
4:17
We just didn't have good quality telescopes to be able to see the outcome of the selection clearly in those places where it mattered
4:26
A few good telescopes were no match for dozens of blurry ones. And those blurry ones drove most of the headlines, which turned out to be wrong
4:33
Ben Bogardus spent years producing newscasts in Washington, Houston, and Jacksonville before becoming the chair of the journalism department at Quinnipiac University, a school that happens to host a major polling center
4:46
Bogardus says responsible newsrooms will evaluate the source. You want to go down and almost have a checklist. Who is this poll done by? Who paid for it? Who commissioned it? Is it an independent poll? Is it by a network? Or is it by one of the campaigns or an advocacy group
5:00
what is the quality of the poll in terms of the internals
5:05
How did they try to reach people? How much did they weigh it
5:10
What's the sample size? What's the margin of error? And then you have to say, what is the wording of the poll
5:19
Is it phrased in a non-biased, subjective way? Bogartas can rattle this off because of his many years of journalism experience
5:26
something increasingly missing in today's newsrooms. A lot of the people who do have experience, the lifers, are retiring or leaving the profession
5:35
And the younger journalists replacing them are facing even more pressures to be fast be everywhere and produce more stories with fewer people So if a press release comes in with a great headline it looks like a shocking news story based on the poll results you going to be sort of interested in putting that in your newscast
5:54
as something new, something different, something exciting. And on social media, where people are increasingly getting their news
6:00
the polling picture is even less in focus. If someone doesn't click through the article to read the actual specifics of the poll
6:06
they could get a sort of a lopsided view of what the poll actually says because you only have one or two sentences on your social media post
6:16
The problem isn't just polling. It's the system around it. And despite high stakes, flubs and diminishing public confidence in them, polls are still attention grabbers
6:26
Since the beginning of time, there's been this sort of problem in journalism where when you're covering political campaigns, it becomes more about the horse race
6:33
It becomes about who's ahead, who's going down. It's not really about the issues that are out there
6:40
The audience wants a simple answer. The algorithm rewards a simple answer
6:44
And polls, stripped of context, provide the simplest answers out there. Now, polling isn't broken
6:51
John Krosnick's research shows that when the right methodology is used, polls are still remarkably accurate
6:57
Mike Noble has built a career on getting those methods right. And Ben Bogart has spent years in newsrooms that strive to report polling responsibly
7:04
When done correctly, polls are still the best tools we have to understand public opinion
7:09
That is, when you know how to interpret them. I don't care what you say about prediction markets, all this other stuff
7:16
It is still the best tool out there. Is it infallible, perfect? No, because errors can happen in that process of a poll
7:23
However, it's still the best thing out there. The polls are coming
7:27
They always do. But the question isn't whether to pay attention, it's whether you're looking through the right telescope
#news


