The hackers seemed to be everywhere.
In one of the largest-ever corporate espionage efforts, cyberattackers alleged to be working for China's intelligence services stole volumes of intellectual property, security clearance details and other records from scores of companies over the past several years. They got access to systems with prospecting secrets for mining company Rio Tinto PLC, and sensitive medical research for electronics and health-care giant Philips NV.
They came in through cloud service providers, where companies thought their data was safely stored. Once they got in, they could freely and anonymously hop from client to client, and defied investigators' attempts to kick them out for years.
Cybersecurity investigators first identified aspects of the hack, called Cloud Hopper by the security researchers who first uncovered it, in 2016, and U.S. prosecutors charged two Chinese nationals for the global operation last December. The two men remain at large.
A Wall Street Journal investigation has found that the attack was much bigger than previously known. It goes far beyond the 14 unnamed companies listed in the indictment, stretching across at least a dozen cloud providers, including CGI Group Inc., one of Canada's largest cloud companies; Tieto Oyj, a major Finnish IT services company; and International Business Machines Corp.
The Journal pieced together the hack and the sweeping counteroffensive by security firms and Western governments through interviews with more than a dozen people involved in the investigation, hundreds of pages of internal company and investigative documents, and technical data related to the intrusions.
The Journal found that Hewlett Packard Enterprise Co. was so overrun that the cloud company didn't see the hackers re-enter their clients' networks, even as the company gave customers the all-clear.
Inside the clouds, the hackers, known as APT10 to Western officials and researchers, had access to a vast constellation of clients. The Journal's investigation identified hundreds of firms that had relationships with breached cloud providers, including Rio Tinto, Philips, American Airlines Group Inc., Deutsche Bank AG, Allianz SE and GlaxoSmithKline PLC.
FBI Director Chris Wray called it the hackers' equivalent of stealing the master keys to an entire apartment complex.
It's an open question whether hackers remain inside companies' networks today. The Journal reviewed data provided by Security Scorecard, a cybersecurity firm, and identified thousands of IP addresses globally still reporting back to APT10's network between April and mid-November.
U.S. agencies, including the Justice Department, have worried about their own possible exposure, and whether the hacks now position the Chinese government to access critical infrastructure, current and former U.S. officials said. Reuters earlier this year reported on some aspects of the scope of the Chinese espionage campaign.
The U.S. government now says APT10 took detailed personnel records of more than 100,000 people from the U.S. Navy.
Investigators in and out of government said many of the major cloud companies tried to stonewall clients about what was happening inside their networks. "It was like trying to pin down quicksand," one investigator said.
Officials at the Department of Homeland Security grew so frustrated by resistance by the cloud companies that they are now working to revise federal contracts that would force them to comply with future probes, according to several people familiar with the matter.
A DHS spokeswoman declined to comment when asked if the agency experienced a breach. A Justice Department spokesman didn't respond to requests for comment.
HPE spokesman Adam Bauer said the company "worked diligently to remediate these intrusions for our customers," adding that "the security of customer data is our top priority."
"We strongly dispute any allegation that HPE was anything less than fully cooperative with government authorities from the outset," Mr. Bauer said. "To suggest otherwise is patently false."
IBM spokesman Edward Barbini said that the company worked on the investigation with relevant government agencies, adding, "We have no evidence that any sensitive corporate data was compromised...We have worked individually with clients who have expressed concerns."
The hack illustrates a weakness at the heart of global business, with the biggest companies in the world increasingly storing their most sensitive data with cloud providers, also known as managed service providers, which have long touted their security.
Many firms contacted by the Journal declined to address whether they were targeted in the attack.
American Airlines said it was notified by HPE in 2015 that "their systems were involved in a cybersecurity incident," but "found no evidence to suggest that our systems or data were compromised."
Philips said it was aware of intrusion efforts that could be attributed to APT10, and that "to date, these attempts have been addressed."
An Allianz spokesman said the company had "found no evidence" of APT10 inside its systems.
GlaxoSmithKline, Deutsche Bank, Rio Tinto and Tieto declined to comment. CGI didn't respond to multiple inquiries.
The Chinese government didn't respond to requests for comment. It has denied hacking allegations in the past.
Cloud Hopper was something new for APT10 (short for Advanced Persistent Threat), one of China's most evasive hacking collectives, according to researchers.
"You know the old joke of, why rob a bank?" said Anne Neuberger, the chief of the National Security Agency's cybersecurity directorate. "Because that's where the money is."
Security firms have been tracking APT10 for more than a decade, as they ransacked governments, engineering firms, aerospace companies and telecoms. Much about the team is a mystery, though U.S. prosecutors have alleged at least some are contractors for the Chinese Ministry of State Security.
To break into the cloud, the hackers sometimes sent phishing emails to administrators with high-level access. Other times they cracked in through contractors' systems, according to investigators.
Rio Tinto was among the earliest targets and a kind of test case, according to two people familiar with the matter. The company, whose operations include copper, diamonds, aluminum, iron ore and uranium, was breached through cloud provider CGI as far back as 2013.
What the hackers took is unknown, but one investigator familiar with the case said such information could be used to buy up real estate where mining companies plan to dig.
Orin Paliwoda, an FBI special agent who has been investigating Cloud Hopper, said at a recent cybersecurity conference in New York that the APT10 team operated essentially as ghosts in the clouds. They "basically look like any other traffic," he said. "It is a major, major problem."
Kris McConkey, a top cyber investigator with PricewaterhouseCoopers in London, was one of the first to see the extent of APT10's operation. During a routine security audit of an international consulting firm in early 2016, his monitors began lighting up with red dots signifying a mass attack.
At first, his team thought the attack was just an unusual one-off, given they had come through the cloud, rather than the company's front door. Then they started seeing the same pattern at other clients.
"When you realize there are multiple cases -- and the actor actually understands what they've got access to, and how to abuse it -- you realize the possible severity of it," Mr. McConkey said. He declined to name specific companies or cloud service providers, citing nondisclosure agreements.
Mr. McConkey's team -- one group to clean out the bad guys, another to gather intel about break-ins and where the attackers might go next -- worked out of a secured floor, accessible only by separate elevators.
The hackers, they learned, worked in teams. The "Tuesday team," as Mr. McConkey dubbed it, would come in one day to make sure all their stolen usernames and passwords still worked. Another group would often appear a few days later, whisking away targeted data.
Other times, the hackers used their victims' networks like dropboxes for what they stole. One firm later discovered it had data stashed from at least five separate companies.
In the early months, Mr. McConkey's group began to share intelligence with other security firms that were also starting to see ghosts. At times, the attackers taunted their hunters, registering domain names for its campaign like gostudyantivirus.com and originalspies.com.
"I haven't seen many Chinese APT groups mocking researchers like that," said Mike McLellan, a director of security research at Secureworks. He added that at times APT10 also laced their malware with phrases insulting researchers' abilities.
One of the hackers' most significant targets was HPE, whose enterprise cloud service handled sensitive data for thousands of companies in dozens of countries. One of its clients, Philips, manages 20,000 terabytes of data, including millions of pieces of information about clinical studies and an app for people with diabetes, according to a promotional video posted to HPE's Twitter account in 2016.
APT10 had been a serious issue at HPE since at least early 2014 -- and the company didn't always tell clients the extent of the problem in its cloud, according to people familiar with the matter.
Making matters more complicated, the hackers had gained access to the company's cyber incident response team, according to several people familiar with the matter. As HPE worked to clear infections, the hackers monitored the process -- and sneaked back into the cleaned systems, beginning the cycle again, one of the people said.