When up to two-thirds of Chinese Internet traffic was suddenly redirected to a handful of websites usually blocked by the Great Firewall, initial reports blamed a hacker attack. Yet experts now say that the most likely cause was an error by administrators of the Great Firewall itself. While the details of what transpired are still widely unknown, Internet security experts have said it would be all but impossible to carry out a hacking attack on that scale.
Yet it remains a mystery why the traffic was routed the way it was. The New York Times discovered that much of the traffic was rerouted to servers owned by a company run out of a small office building in Wyoming:
Technology experts say China’s own Great Firewall — the country’s vast collection of censors and snooping technology used to control Internet traffic in and out of China — was most likely to blame, mistakenly redirecting the country’s traffic to several sites normally blocked inside China, some connected to a company based in the Wyoming building.
[…] The China Internet Network Information Center, a state-run agency that deals with Internet affairs, said it had traced the problem to the country’s domain name system. One of China’s biggest antivirus software vendors, Qihoo 360 Technology, said the problems affected about three-quarters of the country’s domain-name system servers.
“I have never seen a bigger outage,” said Heiko Specht, an Internet analyst at Compuware, a technology company based in Detroit. “Half of the world’s Internet users trying to access the Internet couldn’t.”
Those domain-name servers, which act like an Internet switchboard, routed traffic from some of China’s most popular sites to an Internet address that, according to records, is registered to Sophidea, a company based, at least on paper, in that Wyoming building, in Cheyenne. It is unclear where the company or its servers are physically based, however.
Since much of the traffic was rerouted to a site run by Dynamic Internet Technology, a company affiliated with Falun Gong that designs anti-censorship tools, many assumed they were responsible for the attack. But DIT’s president, Bill Xia, has denied any involvement. Internet users within China have also laid blame on the people making adjustments to the Great Firewall. From Foreign Policy:
China’s state-run media quickly insinuated that foreign interference was involved. The official China News Online stated that a “preliminary investigation” determined an “online attack” had taken place, while Xinhua, China’s state-run news agency, cited an expert who said a “hacker attack” was a “very probable” explanation. An unnamed expert quoted by reliable Communist Party cheerleader Global Times speculated that the prolonged hiccup resulted from a deliberate assault: “We cannot eliminate the possibility that real hackers used this IP address,” the expert said, “as a springboard for their attack.” The article, which carried the Chinese title “Chinese Internet Experiences Mysterious Attack: IP Involved Directs to Censorship Circumvention Software Company,” repeatedly mentioned DIT and the fact that it produces Freegate, a software program that allows users to bypass China’s enormous censorship apparatus, known colloquially as the “Great Firewall.” (DIT founder Bill Xia said in a telephone conversation with Foreign Policy that his company was not responsible for the outage.)
Readers, by and large, had a different theory.
Comments on platforms including news aggregator Netease and Sina Weibo evinced a sense that the protracted glitch was in fact an (accidental) inside job. “I think you messed up while tweaking the Great Firewall,” an anonymous Netease commenter wrote, “and you’re using hackers as a scapegoat.” “You weren’t careful when you were moving a rock and you dropped it on your own foot,” a Weibo user named Wang Rui joked, as if speaking directly to China’s authorities.”Then you said someone else threw it.” One anonymous Weibo user asked, somewhat more obliquely, “Was the Matrix upgrading again?” [Source]
While Global Times and other official media blamed an outside hacking attack, the official CNNIC acknowledged it could be an internal malfunction. From CNN:
The China Internet Network Information Center (CNNIC), a state-run department, blamed a “malfunction in root servers” that blocked access to top-level domain names in China such as .com and .net, according to a post on its Sina Weibo account, the Twitter-like micro-blogging service.
Security analysts quoted by the official Xinhua news agency said this could have been the result of a cyber attack by hackers — though this has not been proved.
Dynamic Internet Technology (DIT) confirmed it owns the web address users were redirected to but denied any involvement. It said the company’s IP address is already blocked in China so users would have been met by a blank web page. [Source]
GreatFire.org provides a technical explanation of how the error could have occurred:
We have conclusive evidence that this outage was caused by the Great Firewall (GFW). DNS poisoning is used extensively by the GFW. Some articles that have appeared about this outage suspected that the root DNS server in China was hacked and all domains hijacked to 65.49.2.178. This could explain why DNS servers in China were poisoned. However, during that time, we see that a lookup to 8.8.8.8, a public DNS operated by Google, returned bogus results if the lookup was done from China. In fact, the Google public DNS was not poisoned; the bogus response 65.49.2.178 could only have been returned by GFW. If the Chinese root DNS server was hacked, a DNS lookup in China via 8.8.8.8 should have returned a correct response.
But why did GFW poison all domains and effectively block all website traffic in China?
This action must have been unintentional. 65.49.2.178 is owned by Dynamic Internet Technology according to an IP lookup, and they are behind the famous circumvention tool FreeGate. Currently, http://65.49.2.178 is a mirror site for dongtaiwang.com, a news portal operated by Falun Gong groups.
One hypothesis is that GFW might have intended to block the IP but accidentally used that IP to poison all domains. [Source]
The Washington Post explains the process in less technical terms:
The Chinese government blocks sites by exploiting a weakness in the infrastructure of the Internet. Let’s say a user is trying to reach a site by entering the domain name — for instance, Facebook.com — into a browser. Ordinarily, that request gets sent to what’s known as a DNS server, which matches the domain name to an IP address, a series of digits that computers can use to identify each other.
Internet experts say China’s Great Firewall works by redirecting traffic to erroneous or fake IP addresses. But in the case of Tuesday’s glitch, something seemed to go wrong.
A massive amount of traffic was diverted to 65.49.2.178, an IP address affiliated with Xia’s Dynamic Internet Technology, a group whose work is routinely censored by the Chinese government.
“The rule was supposed to be, ‘Block everything going to this IP address,’ ” said Nicholas Weaver, a researcher at the International Computer Science Institute, which is affiliated with the University of California at Berkeley. “Instead, they screwed up and probably wrote the rule as ‘Block everything by referring to this IP address.’” [Source]
CDT’s Xiao Qiang explains further to Reuters:
“Our investigation shows very clearly that DNS exclusion happened at servers inside of China,” said Xiao Qiang, an adjunct professor at UC Berkeley School of Information in the U.S. and an expert on China’s Internet controls.
“It all points to the Great Firewall, because that’s where it can simultaneously influence DNS resolutions of all the different networks (in China). But how that happened or why that happened we’re not sure. It’s definitely not the Great Firewall’s normal behavior.” [Source]
