Amazon Rekognition, the controversial facial recognition tool from the Silicon Valley tech giant, has falsely matched 28 members of US Congress with mugshots in its database, a new report from the American Civil Liberties Union revealed today.
"These results demonstrate why Congress should join the ACLU in calling for a moratorium on law enforcement use of face surveillance," said Jacob Snow, an attorney for ACLU of Northern California.
In response to questions from BuzzFeed News about the report, an Amazon Web Services spokesperson emphasized that Rekognition has been used in the past for many useful purposes. It has been deployed in preventing human trafficking, inhibiting child exploitation, and reuniting missing children with their families, the spokesperson claimed, as well as enhancing security "through multi-factor authentication," and preventing package theft. "We remain excited about how image and video analysis can be a driver for good in the world, including in the public sector and law enforcement," the spokesperson told BuzzFeed News in a statement.
But according to the ACLU's report, the technology is far from perfect. Rekognition incorrectly identified more than two dozen lawmakers as people who have been arrested for a crime, and the false matches were disproportionately people of color, the ACLU said. Nearly 40 percent of Rekognition’s false matches in the test were of people of color, even though they make up only 20 percent of Congress. Six members of the Congressional Black Caucus, including noted civil rights leader Rep. John Lewis, were each identified as a match for a mugshot in the Rekognition database.
After the ACLU's report published this morning, Reps. Lewis and Jimmy Gomez wrote a letter to Amazon CEO Jeff Bezos, asking for a meeting to discuss the false matches and the technology's "potential impact on communities of color."
To conduct its test, the ACLU used the standard facial recognition system Amazon offers at a low cost to law enforcement and the public. The ACLU said it spent only $12.33 to build a database of 25,000 publicly available arrest photos and to run a test that included photos of every current member of the US House and Senate. Joshua Kroll, a computer scientist and postdoctoral scholar at the School of Information at UC Berkeley, independently verified the results.
"These results are consistent with a broader pattern of results from the machine learning literature," Kroll told BuzzFeed News. "Not only does face recognition of large sets of individuals remains difficult to do accurately, face recognition systems have been shown to perform much less well for women, people of color, and especially women of color.
"It is important when fielding advanced computer technologies to do so responsibly," Kroll continued. "These results show that Rekognition shouldn’t be used for some applications in law enforcement as it is currently."
In response, the AWS spokesperson said the results of the ACLU's report could probably be improved by "following best practices" around setting the confidence thresholds — the percentage likelihood that Rekognition found a match — used in the test.
"While 80% confidence is an acceptable threshold for photos of hot dogs, chairs, animals, or other social media use cases, it wouldn’t be appropriate for identifying individuals with a reasonable level of certainty," the spokesperson said. "When using facial recognition for law enforcement activities, we guide customers to set a higher threshold of at least 95% or higher."
The ACLU said Amazon did not make any effort to ask clients what they are using Rekognition for. "Instead, the tool sets one default: the same 80% we used in running our test," said Snow. He pointed out that Amazon's website, right now, recommends an 80% confidence for recognizing human faces.
"Amazon seems to have missed, or refuses to acknowledge, the broader point: facial recognition technology in the hands of government is primed for abuse and raises significant civil rights concerns," Snow added. It could allow police to determine who attends protests, ICE to monitor immigrants, and cities to track their own residents, whether or not they have reason to suspect criminal activity, he said. "Changing the threshold from 80 to 95 percent doesn’t change that. In fact, it could exacerbate it."
With this news, lawmakers across party lines expressed concern over how Amazon's technology performed. "Are you sure Amazon isn’t just talking to Donald Trump, Tucker Carlson and Breitbart? Because they think all Latinos are criminals," Rep. Luis Gutiérrez, who was falsely matched with an arrest photo according to the ACLU's report, said in an emailed comment to BuzzFeed News.
"But on a more serious note: If this technology has not been proven effective and has a systematic bias against people of color, then it will hurt, not help law enforcement and anyone else who uses it," Gutiérrez said.
"This technology isn’t ready for prime time, and the risks to Americans’ basic liberties outweigh the benefits," said Rep. Tom Garrett, another lawmaker falsely matched with an arrest photo according to the report.
"Clearly this technology failed to clear an absurdly low hurdle when it comes to accuracy," said Sen. Ron Wyden, who was not falsely matched with a mugshot. "In my view, the government should only deploy facial recognition systems with clear rules and tight controls to prevent the creation of indiscriminate, widespread surveillance of innocent Americans."
"Are you sure Amazon isn’t just talking to Donald Trump, Tucker Carlson and Breitbart? Because they think all Latinos are criminals."
In a statement provided to BuzzFeed News, Rep. Emanuel Cleaver mentioned how previous research has revealed inaccuracies and bias in facial recognition tech — such as a 2012 study coauthored by an FBI technologist that found facial recognition algorithms used by police in California, Maryland, Pennsylvania, and elsewhere consistently performed 5–10% worse for African Americans than for Caucasians. "I remain extremely concerned that Amazon’s Rekognition will further exacerbate issues of racial bias in society, particularly where deployed by law enforcement,” he wrote.
Amazon, for its part, said that in "real-world scenarios," the Rekognition tool is "almost exclusively used to help narrow the field and allow humans to expeditiously review and consider options using their judgment." The AWS spokesperson added that the purpose of the tech is "not to make fully autonomous decisions.”
In May, the ACLU reported that Amazon had been pitching Rekognition to law enforcement agencies to surveil citizens, and that Oregon's Washington County and Orlando had been using it since 2017. The report prompted Amazon employees, shareholders, and a handful of civil rights groups to start a movement and petition Amazon CEO Jeff Bezos to end the contracts with police. Among them was the Congressional Black Caucus, which wrote a letter expressing concern about the "profound negative unintended consequences" face surveillance could have for black people, undocumented immigrants, and protesters.
Bezos has not made a public statement in response. Back in May, an Amazon Web Services spokesperson told BuzzFeed News that "quality of life would be much worse today if we outlawed new technology because some people could choose to abuse the technology."
"Imagine if customers couldn’t buy a computer because it was possible to use that computer for illegal purposes," the spokesperson said, and added that Amazon required customers to comply with the law and to use Rekognition responsibly. Violators of the company's services would be suspended, Amazon said.
Soon after the backlash, the City of Orlando said it would pull the plug on the facial recognition program. But privacy and civil liberties advocates pushing back against the technology still have a long way to go. Representatives of Orlando said in July that the city had reconsidered its position and that the police department would sit down with Amazon to outline a brand new pilot. The city's police chief, John Mina, told the Orlando Sentinel, "It's really to prevent the next tragedy."
