big russian hack used a technique experts...big russian hack used a technique experts had warned...
TRANSCRIPT
Big Russian hack used a technique experts had warned about for years. Why wasn’t the U.S. government ready? Hackers got long-term, wide-ranging access to government and private networks by manipulating the software that vouches for those allowed inside.
(Washington Post Illustration/iStock) By Craig Timberg Feb. 9, 2021 at 10:21 p.m. GMT+1 The disastrous Russian hack of federal government networks last
year relied on a powerful new trick: Digital spies penetrated so
deeply that they were able to impersonate any user they wanted. It
was the computer network equivalent of sneaking into the State
Department and printing perfectly forged U.S. passports.
Cybersecurity researchers had warned for years that such an attack
was possible. Those from one firm, FireEye, even released hacking
tools in 2019 showing exactly how to do it — in hopes that the
revelation would spur the widespread deployment of better
defenses.
It didn’t.
Now there is urgent debate within cybersecurity circles about how
best to respond to the hack, which was so extensive that experts
describe it as historic.
Some are calling for stronger walls to keep out would-be
intruders or better burglar alarms to alert network administrators
that a hack had begun. Others, arguing that there’s no practical way
to keep the most sophisticated hackers from breaking into
important networks, say the smarter investment would be in
building better tools for hunting and ejecting intruders once they
inevitably get past security perimeters. AD
Meanwhile, questions remain about why this surge of corrective
action didn’t happen earlier for a type of hack that had been
discussed for years within cybersecurity circles and whether, even
now, the potential solutions are being deployed widely enough to
prevent future catastrophes.
Two months after the hack was discovered in December,
cybersecurity researchers say spies are probably still active in some
of the hundreds of breached networks. Victims included the
departments of State, Treasury, Homeland Security, Energy and
Commerce, and the National Institutes of Health and the National
Nuclear Security Administration. Also penetrated were private
companies in the consulting, technology, telecom, and oil and gas
companies in North America, Europe, Asia and the Middle East, as
well as FireEye itself, which first reported the attack on Dec. 8.
Russian hack was ‘classic espionage’ with stealthy, targeted tactics
The Russians used a variety of sophisticated tricks to penetrate the
networks in last year’s attack. But once inside, they often
manipulated a piece of Microsoft software, Active Directory
Federation Services, that vouches for the identities of authorized
users by issuing digital identity documents called “SAML tokens.”
An Israeli researcher had first described this technique, dubbed
a “Golden SAML Attack,” in 2017, but it had not been seen in a
major network intrusion until now, experts say. AD
Such systems for authenticating users are essential to securing the
cloud services used widely by government agencies, corporations,
hospitals, universities and most other places where people
collaborate across long distances. And the ability to forge SAML
tokens lets hackers roam widely among these cloud-based services,
while also minimizing the chances of getting quickly caught.
“All of this outward security doesn’t mean squat if you don’t have
this one thing locked up,” said Matthew D. Green, a Johns Hopkins
cybersecurity and cryptology expert. “This is crazy."
Authenticated SAML (rhymes with “camel”) tokens let intruders
move easily among the computer systems affiliated with an
organization, even if the individual elements are run by different
companies, such as Microsoft, Amazon Web Services or Dropbox.
Hackers can present these tokens as they seek access to different
troves of valuable data — email, document repositories, payroll
systems — while sidestepping common defensive measures, such as
strong passwords and two-factor authentication. AD
There are possible protections against a Golden SAML Attack,
including securing the encryption keys that create the tokens in
their own, well-defended piece of hardware, or sharply limiting who
has high-level access to the computers authorized to issue tokens.
Alerts warning of unusual behavior might help defenders act more
quickly, and more extensive logging could help the detective work
after signs of trouble are detected.
Former National Security Agency hacker Jake Williams said his
security consultancy has been helping clients respond to the recent
Russian attack. But even now, it’s not entirely clear to him which
defenses are best suited to prevent a repeat, given the sophistication
of the attackers, which U.S. officials have said were from the SVR,
Russian’s foreign-intelligence service. He favors bolstering systems
for detecting intruders once they’re inside.
“We are not going to keep a nation-state attacker who has targeted
you out,” said Williams, president of Rendition Infosec. “They are
going to outfox you.” AD
The question then becomes: How best to keep a network intrusion
from becoming a catastrophe?
Why didn’t anyone do something sooner? As this debate plays out within the cybersecurity community, Sen.
Ron Wyden (D-Ore.), a member of the Senate Intelligence
Committee, asked Microsoft and FireEye last month to explain how
a security weakness publicized years ago was not addressed before
the Russians took advantage of it. Microsoft released a tool to help
detect such attacks less than two weeks after the Russian hack was
publicly revealed.
“The American people deserve to know why hackers were able to
steal encryption keys from the U.S. government without anyone
noticing,” Wyden said in a statement to The Washington Post. “I
want to know why Microsoft didn’t provide its customers with tools
to better protect and detect the theft of encryption keys, and why
government agencies failed to deploy their own defenses. I’m also
interested in what steps FireEye took to warn Microsoft, its
customers and the U.S. government about a vulnerability it knew
about nearly two years ago.”
The U.S. government spent billions on a system for detecting
hacks. The Russians outsmarted it.
Both companies defended their handling of the Golden SAML
Attack in replies to Wyden — FireEye by letter, Microsoft by video
call — according to a Wyden aide who spoke on the condition of
anonymity to discuss communications not yet made public. AD
In comments to The Post, the companies noted the multiple
weaknesses the Russians exploited in their intrusions and also the
difficulty creating effective defenses against hackers who already
have penetrated networks so deeply that they can issue their own
SAML tokens. Both companies said better overall security practices
are key to defending against this and other attacks, ideally before
the initial intrusions succeed.
John Lambert, the head of Microsoft’s Threat Intelligence Center,
said in an interview that the company long has recommended
security measures that might have thwarted the Russians, such as
stand-alone hardware to guard encryption keys, and that handling
the issuance of SAML tokens through a cloud service, such as
Microsoft’s Azure, would offer increased protection and potentially
make hacks easier to detect.
He also said that some of the measures now under discussion
among independent cybersecurity experts — such as installing the
hardware modules Microsoft recommends for protecting encryption
keys — would make a Golden SAML Attack harder to execute in the
future. AD
“Defending identity has always been foundational,” Lambert said. “I
think if you go back to any set of attacks at any point of time in the
past, compromise of identities, abuse of identities, has always been
a common element. … Securing identities and the secrets that
underpin them have always been important."
FireEye’s role in publicizing the Golden SAML Attack was
highlighted in a Microsoft post that specifically cited one of the
hacking tools FireEye released in 2019, ADFSDump. The post said
that Microsoft’s Defender software could, as of Dec. 20, send alerts
when it detected ADFSDump and called it “the initial tool used” in
the Russian hacks.
Microsoft later revised this characterization after The Post
questioned FireEye about it, saying that the Russians used a
hacking tool resembling ADFSDump but that it was unclear
whether ADFSDump itself was the one. The company’s updated
version of the post removed the reference to the FireEye tool, saying
instead that Microsoft’s Defender software now had an alert to
“detect techniques used to obtain the information needed in order
to generate security tokens," as happened in last year’s Russian
attack. AD
FireEye acknowledged that its engineers had raised alarm about
Golden SAML Attacks and released a pair of hacking tools to exploit
it during a security conference in Germany in March 2019. But the
company said it found no evidence that these tools were used by the
Russians, though it couldn’t rule out the possibility. The goal of
releasing such tools is to help “red teams” of cybersecurity
researchers probe networks for flaws that can be corrected before
malicious hackers exploit them.
“FireEye develops red team tools to help improve enterprise
cybersecurity by demonstrating the impacts of successful attacks
and by showing the defenders … how to counter them in an
operational environment,” said Dan Wire, vice president of global
communications for FireEye. “Like many security companies, we
have an internal process for responsibly releasing tools, and we
review each release on a case-by-case basis.”
Russian government hackers are behind a broad espionage
campaign that has compromised U.S. agencies, including Treasury
and Commerce
The U.S. government response to the Russian hack, meanwhile,
came under fire Tuesday when the heads of the Senate Intelligence
Committee, Chairman Mark R. Warner (D-Va.) and Vice Chairman
Marco Rubio (R-Fla.), sent a letter to the heads of the FBI, National
Security Agency and other federal agencies demanding the
appointment of “a clear leader” to coordinate the response. AD
“The federal government’s response so far has lacked the leadership
and coordination warranted by a significant cyber event, and we
have little confidence that we are on the shortest path to recovery,”
they wrote.
Russian spies began their attack by hacking SolarWinds, a Texas-
based maker of network-monitoring software, and slipping what
security experts call a “Trojan horse” into the networks of the
company’s many clients during routine software updates. Once
inside, the hackers roamed unchecked for months and might have
stayed even longer had FireEye not found them within their own
systems in December. That discovery triggered detection of the
much wider, more troubling federal hack days later.
Many experienced network defenders point to the introduction of
the Solar Winds trojan — in what’s called a “supply-chain attack” —
as the problem most urgently demanding attention because federal
government systems rely on software produced by many private
companies, each of which offers targets for malicious hackers. Once
they get inside, experts say, there are numerous options, beyond
just a Golden SAML Attack, to exploit a network’s systems for
verifying user identities.
“There are literally dozens upon dozens of ways," said Dmitri
Alperovitch, who co-founded cybersecurity firm CrowdStrike and
now is executive chairman of Silverado Policy Accelerator, a think
tank. “No one can possibly defend against all of them. … The idea
that we should be chasing every single attack vector is a
wrongheaded approach.”
Early alarm brought no response Shaked Reiner, an Israeli cybersecurity expert who described the
Golden SAML Attack in a 2017 blog post, said the method offers
important advantages to hackers — namely its potential to enable
unusually wide-ranging, long-lasting and hard-to-detect intrusions
that may merit more robust defenses.
The initial blog post, made on the site of his employer, CyberArk
Labs, initially generated only modest attention. News of the Russian
hack, three years later, changed that. The National Security Agency
cited Reiner’s post in its advisory on how to detect such
intrusions on Dec. 17.
“Right away, we understood. This is what we were talking about,”
Reiner said.
He added that hackers deploying the Golden SAML Attack “can
pretty much impersonate any user in a network. … Detecting this
type of attack can be extremely difficult.”
Some experts, including Green at Johns Hopkins, argue that
sensitive government networks should invest in computer
equipment called “hardware secure modules” that would house the
encryption keys used to issue SAML tokens, making them almost
impossible to steal. This equipment is expensive, ranging from tens
to hundreds of thousands of dollars, and can add significant
complexity to the operation of cloud-computing networks — factors
that have been barriers to their widespread adoption.
Federal investigators find evidence of previously unknown tactics
used to penetrate government networks
Another approach would be to specify a small number of computers
— perhaps ones at the physical desks of system administrators —
that can gain high-level access to the identity-management software
itself. Even a skilled hacker, for example, would find it much harder
to execute a Golden SAML Attack from Moscow if only a handful of
computers were vulnerable to such manipulation from afar. Even
then, key computers could be left disconnected from the Internet,
adding more barriers to hackers operating remotely.
Some other experts, however, say that even without actually stealing
the encryption keys for issuing SAML tokens, hackers can still find
ways to manipulate network identities in ways that allow them to
expand and prolong intrusions.
Williams, of Rendition Infosec, said, “I agree that Microsoft could
have done a better job of detecting any number of active-directory
weaknesses or the exploitation of those weaknesses.”
But he added that more aggressive action by Microsoft, FireEye or
others would have been unlikely to thwart the Russians, given their
skills and resources.
“I’m confident that wouldn’t have changed the outcome here,”
Williams said.
The most viable solution for the future, some experts say, may be in
better alarms to rapidly alert defenders to suspicious behavior,
along with more extensive network logging of network activities —
preferably activated by default — to assist the detective work after
hacks inevitably occur.
CORRECTION: A previous version of this story said incorrectly
that Sen. Ron Wyden sent letters to FireEye and Microsoft last
month asking for answers related to the Russian attack. But in fact
only FireEye received a letter. The communication with Microsoft
was oral. 111 Comments
Craig Timberg Craig Timberg is a national technology reporter for The Washington Post. Since joining The Post in 1998, he has been a reporter, editor and foreign correspondent, and he contributed to The Post’s Pulitzer Prize-winning coverage of the National Security Agency.Follow
More from The Post
• Capitol riot defendants facing jail have regrets. Judges aren’t buying it.
•
Intermittent fasting and why when you eat makes a difference
• As Texans went without heat, light or water, some companies scored a big payday
• Perspective Carolyn Hax: A father relocating for work wonders how he can show up for his young children
• ‘It’s Donald Trump’s party’: How the former president is building a political operation to cement his hold on the GOP