In August 2017, the FBI organized a secret meeting of digital advertising and cybersecurity experts in a secure room in a Manhattan federal building. The roughly 30 people in attendance met to create a coordinated response to a massive ad fraud scheme that posed a risk to the global digital advertising industry and sparked a criminal investigation.
After an introduction by FBI agents, representatives from Google and bot-detection firm White Ops outlined the details of what the tech employees say is one of the largest and most sophisticated digital ad fraud operations they’d encountered.
Sandeep Swadia, the CEO of White Ops, called it a “very complex, ever-shifting maze,” while Scott Spencer, a Google product manager, labeled it a “multiheaded beast” in exclusive interviews with BuzzFeed News.
Eventually, they gave it a name: 3ve (pronounced “eve”).
Today the Department of Justice announced it has unsealed a 13-count indictment against eight men for charges including wire fraud, computer intrusion, aggravated identity theft, and money laundering for their alleged role in masterminding and operating 3ve and Methbot, another large ad fraud scheme exposed in 2016. The government alleges they stole tens of millions of dollars by using "sophisticated computer programming and infrastructure around the world to exploit the digital advertising industry through fraud."
The accused are Aleksandr Zhukov, Boris Timokhin, Mikhail Andreev, Denis Avdeev, Dmitry Novikov, Sergey Ovsyannikov, Aleksandr Isaev, and Yevgeniy Timchenko. Three of the men have been arrested and are awaiting extradition to the United States, while the rest remain at large, according to the DOJ press release.
This comes almost exactly a month after the FBI initiated a takedown of 3ve that saw agents seize Swiss bank accounts, more than 30 web domains, and information from over 80 servers. As that was happening, Google, White Ops, and other tech companies worked to shut down 3ve’s botnets in a coordinated operation.
At its peak, 3ve involved about 1.7 million PCs infected with malware, an array of servers that could generate mountains of fake traffic with bots, roughly 5,000 counterfeit websites created to impersonate legitimate web publishers, and over 60,000 accounts with digital advertising companies to help fraudsters receive ad placements and get paid. The indictment also alleges the fraudsters created their own advertising networks to help facilitate the fraud.
This complex infrastructure served one goal: steal as much money as possible from the $250 billion global digital ad industry while avoiding detection.
Ad fraud is the world’s most lucrative yet low-risk form of criminality today. Billions are stolen every year by people and companies that exploit the complex systems used to buy and sell digital ads. But it’s rare for perpetrators of ad fraud schemes to face criminal charges or serious consequences. That’s one reason why ad fraud is on pace to be eclipsed only by the illicit drug trade in terms of annual revenue, according to the World Federation of Advertisers. An estimated $19 billion will have been stolen this year by ad fraudsters.
“Ad fraud is one of the most profitable crimes with the least amount of risk,” said Tamer Hassan, the chief technology officer of White Ops.
With today’s charges and the takedown of 3ve’s systems, the Department of Justice is sending a message that it now recognizes ad fraud for what it is: a global criminal industry that’s stealing billions with impunity and little fear of prosecution.
“This case sends a powerful message that this Office, together with our law enforcement partners, will use all our available resources to target and dismantle these costly schemes and bring their perpetrators to justice, wherever they are,” said Richard P. Donoghue, United States Attorney for the Eastern District of New York.
The revelation of 3ve and resulting criminal charges for it and Methbot come at a moment when the digital ad industry is facing intense scrutiny from law enforcement and lawmakers. The FBI continues to pursue a criminal investigation into the media buying practices of ad agencies and is asking for brands to cooperate. After BuzzFeed News revealed a different ad fraud scheme last month, Sen. Mark Warner, the ranking member of the Senate Intelligence Committee, called on the Federal Trade Commission to investigate ad fraud and pay close attention to the actions of dominant ad industry players like Google.
Google and White Ops say the fact that roughly 15 companies, some of which are competitors, came together to tackle 3ve shows the industry’s commitment to fighting “invalid traffic,” the technical term used for fake or low-quality traffic.
“What I'm hoping people will take away from the takedown of this is, first and foremost there’s a group actively working to find out when invalid traffic is happening and to try and protect the overall [digital advertising] ecosystem,” Spencer, the Google product manager, said.
But while companies point to 3ve as an example of the industry’s ability to fight fraud, the sophistication and scale of this scheme is also a reminder of how ad fraud has been allowed to become a multibillion-dollar problem that requires FBI intervention. Experts say that other schemes will rise up to take 3ve’s place, a reality that Google and White Ops acknowledge.
“I don’t pretend any single action will take care of the challenge,” Spencer said, though he is optimistic that a push from law enforcement can deter some would-be fraudsters.
In the case of 3ve, he likened the operation to a professionalized company created to steal advertising dollars.
“In many ways this was operating like a corporation or a company where they had diversified tactics where they were trying to maximize their profits and rapidly scale the way they were creating invalid traffic,” he said.
“What stands out about 3ve is how ambitious and global and complex and multilayered it was,” Swadia said.
One of the biggest challenges, according to Swadia and Spencer, was to ensure that the countermeasures used to contain 3ve as the investigation proceeded didn’t tip off the scheme’s perpetrators. If the people running 3ve realized that the industry and law enforcement were onto them, they could have shut it down and made changes to avoid future detection.
“The challenge here was to keep this all under wraps,” Swadia said. “We couldn’t let the [criminals] know their cover was blown, or they would go underground and we would lose all the evidence.”
In early 2017, Google and White Ops began sharing information about a small botnet they were both tracking. The bots were programmed to visit specific websites to generate pageviews and ad impressions that resulted in revenue for the fraudsters.
Google and White Ops identified the fake traffic from the operation and filtered it out of the digital advertising systems they own or are integrated into. The goal was to prevent the websites receiving bot traffic from earning money.
After just a few months, the once-small botnet began to grow quickly. Google and White Ops investigators said they realized this was more than a simple botnet when it began to evade efforts to filter and contain its traffic. Each time they tried to implement measures to filter the traffic out of advertising systems, the ad fraudsters would adapt.
“The thing that was really different here was the number of techniques that they used, their ability to quickly respond when they thought they were being detected, and to evolve the mechanisms they were using in real time,” Spencer said. “We would start to filter traffic and we’d see them change things, and then we’d filter a different way and then they’d change things.”
They dubbed the fraud operation 3ve. By March 2017, Google and White Ops felt they’d gathered enough evidence against 3ve to get in touch with the FBI. “We decided this was worth referring to law enforcement, which is not a normal move for us,” Spencer said.
Meanwhile, 3ve continued to grow and morph. By summer 2017, Google and White Ops were bringing in industry partners to fight it. “It became clear to us that this thing was not a run-of-the-mill bot operation. It was capable of bringing instability and mistrust to the entire [digital ad] ecosystem. And you couldn’t block it,” Swadia said.
Advertising companies including Adobe, the Trade Desk, Amazon Advertising, and Oath pitched in. They were joined by cybersecurity firms such as Malwarebytes, ESET, Proofpoint, Symantec, F-Secure, McAfee, and Trend Micro.
They were all focused on defeating 3ve, but in the short term that meant letting the fraudsters continue earning at least some revenue. Cutting them off completely would signal to 3ve’s operators that they’d been discovered.
Google, for example, let 3ve’s counterfeit websites earn revenue through its ad systems and later refunded its partners for ads that ran. “We actually lose money in this process because we do have to pay out,” Spencer said. (He declined to say how much the company paid out in refunds to partners or to the fraudsters directly.)
What started as a low-level botnet eventually became an operation with three distinct but connected elements. The name 3ve was chosen because of this three-pronged approach.
One part of 3ve involved the use of malware to infect residential computers. These Windows PCs were used to direct fake traffic at counterfeit websites to generate ad impressions and revenue. At its peak, the malware had infected at least 1.7 million Windows computers. These computers provided 3ve’s operators with IP addresses linked to real human users, a valuable commodity that helps evade detection.
The technique used by 3ve’s operators to infect PCs also reveals how the ad industry itself can play a role in enabling fraudsters. Many of the millions of computer users who downloaded the malware became infected after being tricked by misleading ads appearing on websites including the hugely popular adult site Pornhub.
These malicious ads, referred to as malvertising, popped up as users visited the site and tricked them into thinking their browser required a “critical” update, or that Adobe Flash software needed updating, according to samples gathered by security firm Proofpoint.
In October 2017, Proofpoint revealed that Kovter, a type of malware used to power 3ve, infected Pornhub users by running malvertising on the site through an ad network called Traffic Junky. These ads used false claims to manipulate users into downloading and installing the malware.
“The combination of large malvertising campaigns on very high-ranking websites with sophisticated social engineering schemes that convince users to infect themselves means that potential exposure to malware is quite high, reaching millions of web surfers,” Proofpoint wrote at the time. (It noted that Traffic Junky and Pornhub acted swiftly to remove the ads after being notified.)
Proofpoint said that Kovter had been actively infecting people for about a year when it detected the attack on Pornhub. Along with Kovter, 3ve also made use of computers infected with malware called Boaxxe and Miuref.
Google's Spencer said the malware included “anti-forensics” features designed to prevent it from being detected or removed from infected computers. The malware would not run on a computer if it saw that the user had installed security software, and the malware was also programmed not to run if it detected other malicious programs.
“The operators programmed their system to not run [on a PC] if they saw cryptomining software or other malware on the computer, in order to avoid detection,” Spencer said. “So these guys were working very carefully. This was a carefully developed piece of code.”
Along with the malware botnet, 3ve included two operations that used servers in data centers to generate bot traffic. The fraudsters set up or rented servers and then programmed bots to run on them.
3ve’s operators also combined their malware botnet with the data center bots, according to a white paper authored by Google and White Ops, which was provided to BuzzFeed News. The fraudsters used infected computers as gateways to pass high volumes of fake traffic generated by bots hosted on servers. Without users knowing it, their PCs were used as tunnels for bots to pass through on their way to view ads.
In some cases, 3ve’s operators resold fake traffic to third parties who wanted to commit ad fraud, the researchers found. Buying and selling of fake traffic is very common, and 3ve’s bots offered different varieties of fake visitors to customers. The bots could, for example, appear as if they were users on an Android mobile device.
“When combined, the three 3ve sub-operations constituted one of the most widespread ad fraud operations ever uncovered,” the researchers wrote.
They said just one of 3ve’s two data center operations was similar in size to Methbot, a huge ad fraud scheme revealed in late 2016.
In their 3ve white paper, the researchers wrote that Methbot “was likely the largest known ad fraud operation at that time.” Two years later, the DOJ indictment unsealed this week includes charges against five men who are alleged to have run Methbot between 2014 and 2016.
3ve’s three fake traffic operations were used to have bots load and view ads on about 5,000 counterfeit websites set up by the fraudsters. These sites were in many cases nothing more than a skeleton template with little content. The key was that these sites masqueraded as brand-name publishers in advertising exchanges. One example in the white paper juxtaposes a fake 3ve site with the real site for the American cable channel TBS.com:
An agency or advertiser looking to place an ad would only see the name of TBS.com in the ad exchange and not know they were buying ad space on a counterfeit site. Then the bots would visit the fake site and view the ad to generate revenue. (This technique of masquerading as a legitimate publisher is called domain spoofing. It’s a long-running issue, though a new standard called ads.txt is helping curb it.)
Once on the websites, the bots were programmed to mimic human behavior such as mouse movements and clicks, and to click play on videos to ensure lucrative ads were displayed.
As the researchers advanced their understanding of 3ve’s operations, it continued to morph to avoid detection. They said they were constantly concerned that 3ve’s operators would realize they were being watched. This challenge became even greater after Google and White Ops began to recruit other companies into a cross-industry working group.
By then, the FBI investigation was in full swing, and the bureau signaled it wanted to bring everyone together for a meeting in New York in August 2017. Participants were summoned to a secure room, called a Sensitive Compartmented Information Facility, for a briefing and working session. FBI agents told everyone the meeting was confidential and participants could only share details with those in their organizations who needed to know, according to White Ops.
“The first briefing was definitely something — it was memorable; it was very humbling,” Swadia said.
He was struck by the fact that “some of the largest internet companies of our generation, platforms that traditionally compete with each other, [were] coming together and sitting next to each other.”
Along with the industry presence, the room included 10 FBI agents and multiple Department of Justice attorneys, according to some of those in attendance.
“Our favorite moment was at the end of the meeting when an assistant US attorney and prosecutor took on the podium to thank the group and underscore that they were serious about unraveling the incentives of ad fraud,” said Hassan of White Ops.
The working group convened for additional meetings as the operation evolved. Law enforcement participated but did not share details of the people and companies it was zeroing in on, according to White Ops and Google.
Then, in October 2018, the FBI informed the industry working group that the bureau was ready to take down 3ve. The goal was to launch a coordinated effort to free infected PCs from its botnet, block the fake traffic, seize servers and domains, kill the counterfeit websites, and take other action to render 3ve powerless. As part of the takedown, the FBI seized control of 31 internet domains, as well as information from 89 computer servers, according to the DOJ. It also seized "multiple international bank accounts in Switzerland and elsewhere that were associated with the schemes."
FBI agents set Oct. 22 as the date. The white paper includes a graph showing the volume of bids for ads submitted to advertising systems from sites associated with 3ve. It went from a high of roughly 375,000 bids per second on midnight of Oct. 22 to almost zero by 6 p.m.:
“3ve was dead after 18 hours,” Swadia said. With 3ve seemingly dismantled, the industry partners were left to marvel at how an operation involving more than 15 companies and federal law enforcement managed to keep everything under wraps until the signal was given to take 3ve out.
“It was amazing that the [operational security] was good enough that none of the evidence got destroyed,” Hassan said.
In the month since then, Google and White Ops said they were kept in the dark about the charges coming for 3ve’s alleged operators, who these people were, and exactly what the FBI found during its investigation.
When he read today’s press release from the DOJ announcing the charges, Swadia was struck by how hard the FBI and prosecutors had worked to learn about digital advertising, and then to apply this knowledge to secure an indictment.
“Prosecutors and agents across one of the best cybercrime units in the bureau spent two years unraveling the mechanics of programmatic advertising," he said. "This signals a new area in investigation.”●
Added information to indictate that the DOJ indictment includes charges for Methbot.