The recent Israeli election provides a case study in how political polls—and even exit polls—can get the answer badly wrong, with the result that election winner seemed to flip flop from news outlet to news outlet as the actual count unfolded. The print edition of The Washington Post’s headline on Wednesday morning had gone to bed declaring, “A Virtual Tie in Israeli Election” while The New York Times’ headline announced, “Netanyahu Soundly Defeats Chief Rival.” The Post quickly changed its online version to include an acknowledgment of the poorly informed “virtual tie”, and linking the “Virtual Tie” to the story “Netanyahu Sweeps to Victory”.

The Washington Post based its initial story on exit polls, perhaps biasing their interpretation based on the many polls that suggested Benjamin Netanyahu’s Likud party was behind Isaac Herzog’s Zionist Union Alliance, or at least head-to-head with it. Exit polls by three leading television stations found that the Likud party had only 27-28 parliamentary seats: yet Haaretz reported that he won 30 seats, enough to be the lead party in forming a government for Israel.

Polls are often highly predictive of actual outcomes, but the election result this time around revealed their limitations as well. Here are a few reasons that the Israeli polls missed the boat—all of them are lessons in the thorny nature of predicting votes.

Unrepresentative sampling, without proper adjustments. A poll by its nature reaches only a sample of a population—for a margin of error of about 3 percent, one should poll around 1,000 people. But a sample is just, well, a sample, and it may not accurately reflect the people who will vote. If a sample consists of different demographics with different voting patterns, and those people are not represented proportionally in the poll, pollsters can correct for it. For example, if only 20 percent of Orthodox Jewish Israeli likely voters respond to the opinion poll 50 percent of nonreligious Jewish Israeli likely voters, and 30 percent of Arab Israeli likely voters, the proportions of these different groups in the sample will be different than their representation in the voting population. Pollsters can adjust their numbers to let those who participating in the poll count more or less. If the pollsters do not do this correctly, the estimated results may easily miss their target.

Unexpected turnouts of specific demographics. In the case of this election, about 70 percent of eligible voters turned out—an unusually high turn out for an election. An unprecedented 56 percent of eligible voters of the Arab Israeli population turned up to the polls and, overwhelmingly, casting their votes away from Likud. With Netanyahu publicizing the “threat” to Likud of an expected high number of Arab voters in the closing days of the campaign, this may have resulted in a higher than typical turnout of Likud supporters, which wouldn’t have been accounted for in the earlier polls. 

Turning public opinion about the candidates. It may be that Herzog would have taken the lead had the election been held one week prior. In the week running up to the election, Netanyahu changed his stance on the question of the Palestinian state, stating that he was opposed to any agreement that allowed for a separate, Palestinian state. This may have resulted in more support among Israelis who oppose a Palestinian state, who in turn showed their support at the ballot box.

Bad Sampling Techniques: Israelis are notorious attached to their cell phones. Yet some Israelis, like the Hareda sect of ultra-orthodox Israelis, do not use a cell phone at all. Other sampling techniques include Internet sampling (which underrepresents poor people and older people) and regular phone calling.

Nonreponse to polls: Not everyone appreciates the work of pollsters soliciting their opinion. If Netanyahu supporters, for example, are less likely to talk to those conducting the poll compared to those supporting other parties, the poll will skew away from Netanyahu. This is a form of selection bias in which the polls inadvertently favor the opinion of those who speak to them (and assume all such people are the same).

Exit polls: Perhaps most troubling aspect of the Israeli election, from a polling perspective, is how the exit polls underestimated Netanyahu’s support in the election. We have become accustomed to seeing actual political outcomes mirrored in the exit polling data. But exit polls also have their weaknesses: some polling locations may not have had exit poll representatives at all, and these unrepresented voting locations may have been areas with increased interest in Netanyahu.

Exit polling samples opinion in clusters, and as such has a wider margin of error, as well as and increased likelihood of simple bias. Another explanation for a faulty exit poll is that people who support Netanyahu may have been less likely to talk to pollsters and share their choices on the ballot. And exit polls may also miss the changing demographics of voters over the course of the day: those who are voting during the workday may be more inclined to support a different candidate than those who vote in the evening. Exit polling is often done earlier in the day, since the whole point is to have the results before the votes are actually counted. Finally, exit polling can be plagued with the same problem as pre-election polling: if there is an unexpectedly high turn out of one demographic of voter compared to another, the vote will be tipped in the direction of the high-turn-out voter’s opinion, but the exit polls may not.

Rebecca Goldin is Professor of Mathematical Sciences at George Mason University and Director of STATS.org. She received her undergraduate degree from Harvard University and her Ph.D. from the Massachusetts Institute of Technology. She taught at the University of Maryland as a National Science Foundation postdoctoral fellow before joining George Mason in 2001. Her academic research is in symplectic geometry, group actions and related combinatorics. In 2007, she received the Ruth I. Michler Memorial Prize, presented by the Association for Women in Mathematics. Goldin is supported in part by NSF grant #1201458.

Please note that this is a forum for statisticians and mathematicians to critically evaluate the design and statistical methods used in studies. The subjects (products, procedures, treatments, etc.) of the studies being evaluated are neither endorsed nor rejected by Sense About Science USA. We encourage readers to use these articles as a starting point to discuss better study design and statistical analysis. While we strive for factual accuracy in these posts, they should not be considered journalistic works, but rather pieces of academic writing.