Part 2 of this series ended with a stalemate. Conficker signed its updates, generated thousands of domains a day and kept a peer-to-peer fallback. Defenders could keep its authors out, but they could never take the botnet over.
GameOver Zeus had the same three defences, and it was run by people who used it every day to steal money. From September 2011 to May 2014 it was one of the most profitable criminal botnets in the world: the FBI put the losses at more than $100 million. Yet on 30 May 2014 an international operation, later known as Operation Tovar, took control of its peer-to-peer network within hours. Its operator tried to win it back and gave up after about five.
This is Part 3 of A Short History of Botnets: how a peer-to-peer botnet was built to survive a takedown, and how defenders found the one layer it could not protect.
A note on the exhibits. As before, the exhibits are reconstructions or illustrations built from published analyses, mainly Dell SecureWorks’ protocol research and Fox-IT’s report on the group behind the botnet. Each one says which kind it is. The takedown itself is described publicly only at a high level, and I have not filled the gaps with guesses.
Lineage: from a toolkit to a private club
GameOver Zeus grew out of ZeuS, the banking trojan that defined the genre. According to Fox-IT’s report, presented at Black Hat 2015 with the FBI’s lead agent on the case, its author, known as Slavik, created ZeuS around 2005 and 2006. From 2007 its “webinjects”, configuration files that rewrote banking pages inside the victim’s browser, became the de facto standard for this kind of fraud.
The key events, in short:
| Date | Event |
|---|---|
| May 2014 | Operation Tovar takes over the peer-to-peer network |
| Spring 2012 | Microsoft’s civil action against ZeuS botnets; GameOver is barely affected |
| September 2011 | GameOver Zeus launches: a private, peer-to-peer version of ZeuS 2.1 |
| 2011 | ZeuS source code leaks, spawning Citadel, Ice-IX and other forks |
| Late 2010 | Slavik announces he is stepping back from selling ZeuS |
The “retirement” in 2010 was not what it looked like. While the public ZeuS kit faded into forks, Slavik moved to a private operation. The group behind it called itself the “business club”: more than 50 people, including fraud specialists, mule recruiters, technical support and outside suppliers. Members paid to join and shared profits. Slavik ran the peer-to-peer network and the back-end infrastructure.
The name came from the server side. Fox-IT traced it to the command gate script, gameover2.php. The botnet kept an average of about 200,000 active infections for nearly three years, and up to 27 separate botnets shared its peer-to-peer network, each with its own back end.
Architecture: three layers between victim and operator
Coreflood had one layer: bots talked straight to a handful of servers. GameOver Zeus put two layers in between, so that no single seizure could reach the operators. The best public description of the design is Dell SecureWorks’ 2012 analysis by Brett Stone-Gross.
The peer-to-peer layer. Every infected machine was a node in a network loosely based on Kademlia, the design behind many file-sharing networks. Each node had a 20-byte ID, listened on a random TCP and UDP port between 10,000 and 30,000, and stored up to 150 neighbours. Short control messages, such as “what version do you have?” and “give me some peers”, went over UDP; a peer-list reply held up to 10 entries. Larger transfers, such as configuration files, binary updates and stolen data, went over TCP.
The proxy layer. A subset of bots, reachable from the Internet, acted as proxy nodes. Ordinary bots handed stolen data, encrypted with RC4, to these proxies, which relayed it upward. The messages that announced which nodes were proxies were themselves signed, so an outsider could not simply declare a machine of its own to be one.
The back end. Behind the proxies sat the command servers, hidden behind further reverse proxies and, by 2014, a bulletproof hosting provider that handed out fresh IP addresses within days when old ones were blocked. Up to 27 separate botnets, each with its own back end, shared the same peer-to-peer network.
Two safety nets completed the design. Configuration files and binaries were digitally signed: Fox-IT notes that pushing a command or update required an RSA private key, and Slavik held it. And if a new bot could not reach any of its hard-coded peers, it fell back to a domain generation algorithm: 1,000 domains a day, 32 to 48 characters long, under .com, .net, .org, .biz, .info and .ru, used only to fetch a fresh list of live peers.
Exhibit 1. The layers between a victim and the operators, and the two places Operation Tovar later went in.
Stolen data travels right, from bots through proxy bots and reverse proxies to the back ends; signed commands travel left. The shaded boxes mark where the defenders entered in 2014, described in the Operation Tovar section.
Compared with Conficker, this was a botnet designed with a takedown in mind. Seizing a command server achieved little, because the proxy layer and the peer network carried on. Registering the fallback domains achieved little, because most bots never needed them. The question for the defenders was where, in all of this, they could get in.
The money: from webinjects to ransomware
Infections came through a supply chain of other criminals’ services. SecureWorks traced a typical 2012 path: spam from the Cutwail botnet linked to compromised websites, which redirected victims to the Blackhole exploit kit, which installed the Pony loader, which in turn downloaded GameOver Zeus.
Once installed, the bot did what ZeuS had always done, but aimed higher. Coreflood stole whatever users typed. GameOver Zeus targeted corporate banking, where a single transfer could be worth more than thousands of stolen personal accounts. Its main high-value technique, called the “World Bank Center” inside the group, was a hybrid attack on two-factor logins that combined webinjects in the victim’s browser with live action by the fraudsters.
The stolen money moved through mules. According to Fox-IT, corporate mule accounts were opened in China, Hong Kong, Cyprus and Latvia, many in the Chinese border towns of Raohe and Suifenhe under the names of trading and shipping companies. Between 2009 and 2014 the group collected an estimated 20 to 30 terabytes of victim data. The FBI’s loss estimate was more than $100 million, and Fox-IT considered the real figure likely higher.
In September 2013 the same infrastructure began delivering a second product: CryptoLocker, one of the first ransomware families to encrypt victims’ files with strong public-key cryptography and demand payment for the key. The US Justice Department’s announcement put the infections at more than 234,000 computers by April 2014, with ransoms “often as much as $700 or more”. The two main sources disagree on the money: the Justice Department cites more than $27 million paid in the first two months, while Fox-IT estimates about $3 million reached Slavik and his affiliates.
CryptoLocker matters for this series because of where it led. It showed criminals that encrypting files paid faster than draining bank accounts, without mules or banks in the middle. Much of the ransomware that followed built on that lesson.
Operation Tovar: turning the network against itself
The operation started small. In June 2012 the FBI’s Pittsburgh office investigated a local business that had lost money to GameOver Zeus. By late 2012, according to an account by Motherboard, the lead agent, Elliott Peterson, and a team of about ten technical experts from companies including CrowdStrike, Dell SecureWorks and Fox-IT had decided to go after the botnet itself. Preparation took more than a year of weekly calls over encrypted channels.
The insight was the one this series has been building towards. GameOver Zeus’s signatures protected what the bots would run: no one without Slavik’s key could push a command or an update. But a peer-to-peer network also has to decide whom to talk to, and that decision was made by the bots themselves, based on what other peers told them. CrowdStrike’s Tillmann Werner described the result: the team added its own peers to the network and turned the peer-to-peer botnet into a centralised one, with themselves at the centre.
The published accounts describe three parts, executed together from Friday 30 May 2014:
- The peer layer. Peers controlled by the defenders were introduced into the network until the bots trusted them and ignored the criminals’ peers, proxies and supernodes. Peterson said fewer than ten such nodes were needed.
- The proxy layer. Internet service providers handed investigators control of the proxy nodes the criminals used to connect their command servers to the botnet.
- The legal layer. Civil and criminal orders from a federal court in Pittsburgh authorised redirecting the bots’ requests to substitute servers and collecting victims’ IP addresses for notification through US-CERT and other national CERTs. The Justice Department stressed that investigators did not access the content of victims’ computers. In parallel, CryptoLocker’s command servers were seized and its domain-generation domains taken over.
The exact protocol messages used have not been published, and I will not guess at them. What matters for defenders is the shape of the change.
Exhibit 2 (illustrative). One bot’s neighbour list before and after the operation. Bot IDs are shortened and invented; addresses come from the reserved documentation ranges.
# BEFORE: neighbours learned from other infected machines
bot_id ip:port role
3f9a..c1 198.51.100.14:18222 peer
81d0..7e 198.51.100.77:24519 peer
0c55..9b 192.0.2.201:11873 proxy <- relays to the criminals' back end
a4e2..03 198.51.100.140:29004 peer
... (up to 150 entries)
# AFTER: the list is dominated by defender-controlled nodes
bot_id ip:port role
5be1..44 203.0.113.10:16031 peer <- defenders' node
5be1..45 203.0.113.11:21877 peer <- defenders' node
5be1..46 203.0.113.12:12409 proxy <- defenders' node
...
# Updates and stolen data now flow only to the substitute servers;
# the bot still runs nothing that lacks Slavik's signature.The counterattack came quickly. Peterson said Slavik noticed within an hour and fought for several hours to regain control; after four or five hours he “didn’t have a chance”. About 300,000 computers were infected when the operation began. More than a year later, roughly 30,000 remained.
The network had been broken, not cleaned. Like Coreflood’s stop command, the takeover cut the operators off. Removing the malware still depended on owners, the US-CERT removal guidance and the security vendors that took part.
The people, and what came after
Conficker’s authors were never identified. GameOver Zeus’s leader was named on the day of the takedown. The Justice Department charged Evgeniy Mikhailovich Bogachev, then 30 and living in Anapa, Russia, with conspiracy, computer hacking, wire fraud, bank fraud and money laundering in a 14-count indictment in Pittsburgh. A separate complaint in Omaha tied him to the earlier Jabber Zeus operation. His aliases included “Slavik”, “Pollingsoon” and “Lucky12345”.
In February 2015 the US government offered a reward of up to $3 million for information leading to his arrest or conviction. He is on the FBI’s Cyber’s Most Wanted list and is believed to be at large in Russia. In December 2016 the US Treasury also placed him on its sanctions list.
Fox-IT’s investigation added a twist that is still debated. Some of the group’s command servers were used to search victims’ computers not for bank data but for political information, using keywords about government classified material. The searches targeted Georgia (106 entries in Fox-IT’s data), Ukraine after its 2014 change of government, and Turkey (11 entries), where the terms focused on classified documents and Syria. Fox-IT speculated that whoever wanted this intelligence may have protected Slavik in return, while stating clearly that this was speculation.
The botnet also tried to come back. On 10 July 2014, six weeks after Tovar, Dell SecureWorks found a new variant spreading through the same Cutwail spam. It had dropped the peer-to-peer network entirely in favour of a centralised design, with a new domain generation algorithm producing 1,000 domains a day. Researchers sinkholed one of its domains the next night, and 177 machines checked in within 12 hours.
I read that retreat as telling. After the operation, the authors chose the design that Conficker had shown defenders could contain, rather than rebuild the one that had just been broken.
Hunting the pattern today
Peer-to-peer malware has a network shape that ordinary office software rarely has. A workstation in a botnet like GameOver Zeus talks to many unrelated external addresses on random high ports, over UDP as well as TCP, at regular intervals. Browsers, mail and business applications mostly talk to a small set of well-known services on standard ports. The rules below look for that difference. The DGA fallback is covered by the queries in Part 2; GameOver’s 32- to 48-character names would also stand out on length alone.
Exhibit 3a: Sigma correlation, peer-to-peer fan-out. A host that sends UDP to 50 or more distinct external addresses on high ports within ten minutes. Uses Zeek connection logs; exclude your VoIP, video-conferencing and gaming ranges first, because they produce the same shape legitimately.
title: Outbound UDP to External High Port
name: outbound_udp_high_port
status: experimental
logsource:
product: zeek
service: conn
detection:
selection:
proto: 'udp'
local_orig: true
local_resp: false
id.resp_p|gte: 10000
filter_dns:
id.resp_p: 53
condition: selection and not filter_dns
---
title: Possible Peer-to-Peer Botnet Fan-Out
id: 7d2c9e14-6b3a-4f81-9c50-2e8a1b4f6d93
status: experimental
description: One internal host sends UDP to many distinct external hosts on high ports in a short window
correlation:
type: value_count
rules:
- outbound_udp_high_port
group-by:
- id.orig_h
timespan: 10m
condition:
field: id.resp_h
gte: 50
falsepositives:
- VoIP and video conferencing, online games, legitimate P2P software
level: mediumExhibit 3b: Splunk, fan-out plus regularity. Finds hosts that talk to many external peers on high ports, then checks how regular the contacts are. A low spread in the intervals between connections suggests automated beaconing rather than a person.
index=zeek sourcetype=zeek:conn local_orig=true local_resp=false
"id.resp_p">=10000
| sort 0 "id.orig_h" _time
| streamstats current=f last(_time) AS prev_time BY "id.orig_h"
| eval gap=_time-prev_time
| stats dc("id.resp_h") AS distinct_peers
dc("id.resp_p") AS distinct_ports
values(proto) AS protocols
avg(gap) AS avg_gap stdev(gap) AS stdev_gap
BY "id.orig_h"
| eval regularity=round(stdev_gap/avg_gap, 2)
| where distinct_peers>=50 AND distinct_ports>=20
| sort - distinct_peersA regularity value well below 1 means the gaps between connections are nearly constant. Treat it as a ranking aid, not a verdict.
Exhibit 3c: KQL (Microsoft Defender for Endpoint). Endpoint data has one advantage over network logs: it names the process. A browser talking to many peers is unusual; a binary in a user’s AppData folder doing it is worth a look.
DeviceNetworkEvents
| where Timestamp > ago(1d)
| where RemoteIPType == "Public" and RemotePort between (10000 .. 65535)
| summarize Peers = dcount(RemoteIP),
Ports = dcount(RemotePort),
Protocols = make_set(Protocol),
FirstSeen = min(Timestamp), LastSeen = max(Timestamp)
by DeviceName, InitiatingProcessFileName, InitiatingProcessFolderPath,
bin(Timestamp, 10m)
| where Peers >= 50 and Ports >= 20
| where InitiatingProcessFolderPath !startswith @"C:\Program Files"
| order by Peers descThe process filter at the end is a starting point, not a rule. Some legitimate software installs in user folders, and some malware hides in Program Files; review what the filter removes before relying on it.
What I take away
Sign the membership, not just the messages. GameOver Zeus signed what mattered most, its commands and updates, and nothing published suggests the defenders ever forged one. They won by changing whom the bots listened to. Any distributed system, from a botnet to a vehicle fleet or a mesh of industrial devices, has to protect both questions: what a node will accept, and which nodes it treats as peers. Securing the first while leaving the second to gossip leaves a door open.
Resilience against seizure is not resilience against everything. The peer-to-peer design made servers irrelevant, which defeated every earlier takedown method. It also made the network depend on the bots’ own judgement of their neighbours, and that judgement could be shaped. Every architectural choice moves the weak point; it rarely removes it.
Patience is a tactic. More than a year of preparation went into a takedown that took a few hours. A partial strike would have warned the operators and taught them what to fix. The same holds in incident response: acting on the first indicator can cost you the whole picture.
Breaking is still not cleaning. Coreflood ended with 19,000 cleaned machines out of millions, Conficker with hundreds of thousands still infected a year later, and GameOver Zeus with about 30,000 of 300,000 still infected after more than a year. Three operations, three eras, the same last mile.
Criminal infrastructure does not stay criminal. The same command servers that stole from banks were also searching for classified documents in Georgia, Turkey and Ukraine. For defenders, the line between “just crime” and espionage is thinner than our threat models often assume.
GameOver Zeus showed that peer-to-peer botnets could be broken. The next part leaves Windows behind: Mirai, the botnet built from cameras, routers and video recorders that nobody had ever thought to update.
Sources
- US Department of Justice: U.S. leads multi-national action against GameOver Zeus botnet and Cryptolocker ransomware (2 June 2014)
- US Department of Justice: Reward announced for cyber fugitive (24 February 2015)
- Michael Sandee (Fox-IT), Tillmann Werner (CrowdStrike) and Elliott Peterson (FBI): GameOver Zeus: Badguys and Backends, Black Hat USA 2015
- Brett Stone-Gross, Dell SecureWorks: The Lifecycle of Peer-to-Peer (Gameover) ZeuS (2012)
- Dell SecureWorks: GameOver Zeus re-emerges without peer-to-peer capability (July 2014)
- Motherboard: How the FBI took down the botnet designed to be impossible to take down
- Dennis Andriesse, Christian Rossow, Brett Stone-Gross, Daniel Plohmann and Herbert Bos: Highly resilient peer-to-peer botnets are here: an analysis of Gameover Zeus, MALWARE 2013
- Earlier parts of this series: Coreflood Under the Hood and Conficker: The Botnet Nobody Could Hijack
Pingback: Conficker: The Botnet Nobody Could Hijack - Taner Güngör