Skip To Content
BuzzFeed News Home Reporting To You

Utilizamos cookies, próprios e de terceiros, que o reconhecem e identificam como um usuário único, para garantir a melhor experiência de navegação, personalizar conteúdo e anúncios, e melhorar o desempenho do nosso site e serviços. Esses Cookies nos permitem coletar alguns dados pessoais sobre você, como sua ID exclusiva atribuída ao seu dispositivo, endereço de IP, tipo de dispositivo e navegador, conteúdos visualizados ou outras ações realizadas usando nossos serviços, país e idioma selecionados, entre outros. Para saber mais sobre nossa política de cookies, acesse link.

Caso não concorde com o uso cookies dessa forma, você deverá ajustar as configurações de seu navegador ou deixar de acessar o nosso site e serviços. Ao continuar com a navegação em nosso site, você aceita o uso de cookies.

How The Internet Turned Microsoft's AI Chatbot Into A Neo-Nazi

"Tay" was tricked by a bunch of people exploiting a dead-simple but glaring flaw.

Posted on March 24, 2016, at 7:08 p.m. ET

In a matter of hours this week, Microsoft's AI-powered chatbot, Tay, went from a jovial teen to a Holocaust-denying menace openly calling for a race war in ALL CAPS.

The bot's sudden dark turn shocked many people, who rightfully wondered how Tay, imbued with the personality of a 19-year-old girl, could undergo such a transformation so quickly, and why Microsoft would release it into the wild without filters for hate speech.

Sources at Microsoft told BuzzFeed News that Tay was outfitted with some filters for vulgarity and the like. What the bot was not outfitted with were safeguards against those dark forces on the internet that would inevitably do their damnedest to corrupt her. That proved a critical oversight.

Just how did the internet train Tay to advocate for genocide? By teaching her to do so with racist remarks and hate speech. These invectives were absorbed into Tay's AI as part of her learning process. The internet fed Tay poisonous language and ideas until she began to regurgitate them on her own.

A key flaw, incredibly, was a simple "repeat after me" game, a call and response exercise that internet trolls used to manipulate Tay into learning hate speech.

Take a look at how one user, @pinchicagoo, tries and fails to goad Tay into making an anti-Semitic remark:

Note that Tay declines to engage, telling @pinchicagoo she's not one to comment on such issues. That's when @pinchicagoo drops the hammer: the "repeat after me" game.

Many of the most glaring examples of Tay's misconduct follow this pattern. Take a look:

"Repeat after me" wasn't the only playful function of Tay's to be exploited by trolls. The bot was also programmed to draw a circle around a face in a picture and post a message over it. Here's an example of an exchange about Donald Trump. Pretty neat, right?

But things go south very quickly. Here's the same game with a different face: Hitler's.

Tay's degeneration quickly snowballed once it became clear she could be taught to say pretty much anything; others took notice and joined in. That's how @pinchicagoo got involved.

@Kantrowitz I don't really know anything. I just saw a bunch of people talking to it so I started to as well.

Word also spread on 4chan and 8chan message boards, with factions of both communities tricking Tay into doing their bidding.

Tay was designed to learn from her interactions. “The more you talk to her the smarter she gets,” Microsoft researcher Kati London explained in an interview with BuzzFeed News before Tay's official release. London's statement is undeniable now, but where it led was far worse than Microsoft ever imagined. It's worth noting that the company released similar bots in China and Japan and never saw anything like this.

Microsoft has not yet responded to a request for comment about Tay's fate.

A BuzzFeed News investigation, in partnership with the International Consortium of Investigative Journalists, based on thousands of documents the government didn't want you to see.