It was time to purge the hacker from the U.S. government's computers. After secretly monitoring the hacker's online movements for months, officials worried he was getting too close to critical information and devised a plan, dubbed "the Big Bang," to expel him.
Trouble was, with all their attention focused in that case, they missed the other hacker entirely.
A new congressional report provides previously undisclosed details and a behind-the-scenes chronology of one of the worst-ever cyberattacks on the United States, laying out missed opportunities before the break-in at the Office of Personnel Management exposed security clearances, background checks and fingerprint records. That attack — widely blamed on China's government — compromised personal information of more than 21 million current, former and prospective federal employees, led to the resignation of the OPM director and drew outrage over changing explanations about the hack's seriousness.
The report by the House Committee on Oversight and Government Reform faulted the personnel agency for failing to secure sensitive data despite warnings for years that it was vulnerable to hackers. It concluded that the hacking revealed last year could have been prevented if OPM had put in place basic, required security controls and recognized from an earlier break-in that it was actually dealing with a sophisticated, persistent enemy.
"We have literally tens of millions of Americans whose data was stolen by a nefarious overseas actor, but it was entirely preventable," Rep. Jason Chaffetz, a Utah Republican and committee chairman, said in an interview.
"With some basic hygiene, some good tools, an awareness and some talent, they really could have prevented this," he added.
OPM Acting Director Beth Cobert said in a statement the agency disagrees with much of the report and it "does not fully reflect where this agency stands today." She said the OPM hack "provided a catalyst for accelerated change within our organization," including hiring new cybersecurity experts and strengthening its security.
The government discovered the first OPM hacking in March 2014 when a specialized Homeland Security Department team noticed suspicious streams of data leaving its network between 10 p.m. and 10 a.m. — the online equivalent of moving trucks hauling away filing cabinets containing confidential papers in the middle of the night. The government's so-called Einstein intrusion warning system detected the theft.
"DHS called us and let us know, hey, we think this is bad," Jeff Wagner, OPM's director of information security operations, told officials investigating the hack, according to the hack.
For two months, the personnel office worked with the FBI, National Security Agency and others to monitor the hacker to better understand his movements. Officials developed a plan to expel the hacker over a three-day weekend in May 2014, dubbed "the Big Bang." The effort included resetting administrative accounts, building new accounts for users who had been compromised and taking offline compromised systems.
"The risk of kicking them out too early had come and gone," Wagner said, "and now the risk was becoming having them in too long, and we didn't want to keep them around any longer than we had to."
The problem was far from solved.
Unknown to the experts focused on expelling the hacker, a second intruder posing as an employee of a federal contractor had infiltrated the system weeks before "the Big Bang." That hacker used a contractor's credentials to log into the system, install malicious software and create a backdoor to the network, according to the report.
Over the next several months, roaming unchecked through the system, the hacker stole sensitive security clearance background investigation files, personnel files and, ultimately, fingerprint data.
That breach was not detected until April 2015, when an OPM contract employee traced the flow of stolen material back to an Internet address that had been registered to Steve Rogers, the alter ego of Captain America, indicating a spoof account. By then, sensitive information on millions of American workers had already been compromised.
The report also faulted the personnel office for failing to quickly deploy security tools from an outside firm to detect malicious code and other threats. Once deployed, the tool from Cylance Inc. of Irvine, California, "lit up like a Christmas tree," indicating it found malware throughout the federal computers, an engineer is quoted as saying in the report.
"Could they have done better? Absolutely," said Cylance founder and chief executive Stuart McClure. "But once they had been definitively convinced there was a breach, they took it very seriously."
It said OPM officials misled the public about the scope of the breach and also by saying the two breaches were unrelated when, instead, "they appear to be connected and possibly coordinated," according to the congressional report.
"The two attackers shared the same target, conducted their attacks in a similarly sophisticated manner, and struck with similar timing," the report said.
Though the U.S. suspects the hack was an act of Chinese espionage, the House inquiry did not go into great detail about who was responsible. It mentions that the data breaches discovered in April 2015 were likely perpetrated by the group "Deep Panda," which has been linked to the Chinese military.
Associated Press video journalist Dan Huff contributed to this report.
Follow Eric Tucker on Twitter at http://www.twitter.com/etuckerAP