In 2011, political scientists tested an odd hypothesis: that someone staring at you would make you more inclined to vote.
Costas Panagopoulos of Fordham University made "get out the vote" fliers featuring photos of human eyeballs, and sent them to residents of Key West, Florida and Lexington, Kentucky. These creepy fliers, his studies found, increased voting for a new governor in Lexington by more than 2% — enough to turn a close election.
It was a weird and surprising finding, and part of a big trend of social science getting into elections that followed its starring role in Barack Obama's 2012 campaign. But the staring study turns out to be wrong, according to four independent experts.
Last month, a different group reported that sending similar eyeball mailers to people in Minnesota, Ohio, Texas, and Virginia had no effect on voting patterns, aside from a small bump in turnout in Minneapolis. "I don't think there is any extra effect from having a set of eyes plastered on a mailer," Richard Matland of Loyola University of Chicago told BuzzFeed News. "Or if there is, it is very slight."
Scientific studies often can't be replicated, for a wide variety of reasons, from small sample sizes that give fluke results to unrepeatable situations to simple lack of funding. But unfortunately a crisis in replication has stalked both medical and psychology studies for the last decade.
"The checks and balances that once ensured scientific fidelity have been hobbled," Francis Collins, head of the National Institutes of Health, wrote in 2014, in a Nature article outlining new steps to address the crisis.
Psychology experiments tested in the real world have been hit especially hard, with a Dutch psychologist falsifying data in perhaps 55 studies, and more recently, the scandal-ridden retraction of a heralded Science magazine study claiming that gay political canvassers changed voters' opinions on gay marriage.
In response, psychologists have taken steps to replicate more results. In 2014, for example, an international effort of 52 scientists attempted to replicate 13 well-known psychological findings. It found that two findings — the idea that people could be "primed" with pictures of flags, or with pictures of money — were illusory.
In the eyeball study, Panagopoulos borrowed a finding from psychology that priming people with a pair of watching eyes might make them feel as if they are being watched, and therefore behave more conscientiously.
He reasoned that a set of eyes coupled with the exhortation to "DO YOUR CIVIC DUTY AND VOTE!" might increase going to the polls.
But after looking at Matland's failed efforts to replicate, conducted with Gregg Murray of Texas Tech University, independent scientists say that Panagopoulos' original results don't stand up to statistical scrutiny.
"I would tend to bet on the (very small) effect sizes from the new study," psychologist Hal Pashler of the University of California, San Diego, told BuzzFeed News by email, "because we are pretty sure that we are getting an unbiased report."
The big problem with studies like the eyespot one, he adds, is that researchers can't get multiple observations of voters from their experiments, rendering them forever unrepeatable. "People only decide once whether to vote on election day."
On the plus side, this debate between researchers is how replication is supposed to happen, Columbia University political scientist Andrew Gelman told BuzzFeed News. "One thing I really like about Panagopoulos' response is that he does not crawl into an angry, defensive ball." Rather the two sides mostly disagree about how small a statistical difference matters when it comes to voter turnout.
"Statistics aside, this whole research topic is a bit unsavory, isn't it?," Pashler added. Putting eyes on fliers to make people feel watched, "consciously, unconsciously, whatever," gives him the heebie-jeebies.
On the other hand, U.S. voter turnout is notoriously low — around 60% nationwide every Presidential election. With the 2016 elections not too far off, we are likely to see similar psychology experiments on voters, Panagopoulos said. "The reason why we have seen declines in voting since the 1960s may be that it has become less of a social activity, and nobody feels like someone is watching them."