Security in the face of adversityBecause damnit, security is important

www.brightpearl.com

My brain right now

Who the hell am I?

Dave Tibbs@LowlySysadm1n

l Systems Administrator at Brightpearl Incl Started at Brightpearl UK in October 2010l Back then, only about 20 people in the company

– I was the only Systems Administrator/General IT Dogsbody

l ~7 years experience as Sysadmin working with various flavours of Linux

Security – everyone knows it's important, right?

l Wrong!l In my experience, faced with “more

important” priorities (production issues, delivery deadlines), security is one of the first things to fall by the wayside

l But left unchecked, it has one of the worst damage potentials.

Time for a story...

My thought process...

● First – loads of instances spun up in a very short space of time?

● Very unlikely that someone could manage it that quickly through the UI – even with scripts e.g. Greasemonkey, AWS UI isn't exactly quick

● Got to have been done through the API● What is API auth controlled by? AWS

Keypairs.● ARGGGHHHH● DISABLE ALL THE KEYPAIRS!

Speedy resolution

l Disable all the keypairs on our AWS account. Even if it breaks something in production, with the keypairs (that likely had root privileges), an attacker could do far worse:● Spin up expensive instances for nefarious purposes (they did this bit)● Delete RDS instances● Delete snapshots● Terminate EC2 instances● A whole lot more...l Phone AWS Support and find out what keypair had been used to spin up the

instances so we could keep it disabledl Keep everything else disabled until we could get everything else in IAM (more

about this later)

So how the hell did it happen?

l Genuine badness in our setup:● Our AWS infrastructure and usage of it was likely

set up in a hurry● Our AWS keys in use all over our infrastructure

were attached to the root AWS account (all teh privilegez!)

● Once these were being used and working, no impetus to change them - “Fuck it, it works”

● These keys could well have been used outside the organisation – external services using AWS keys to monitor spend or test Cloudwatch integration, etc

So how did they get that keypair?...

l Unfortunately due to using the same AWS key everywhere (cue more facepalming) it's hard to know. Possibilities:

● Nefarious or even careless ex-employee. Keypairs had never been rotated/regenerated

● The keypair was stored in an environment variable for easy access by everything (as well as other places). Bug found to expose these?

● Keypair used with external service which has been compromised.

Steps for fixage

l Secure AWS account● IAM, IAM, IAM!● The more keypairs the better. Don't share

keypairs between functions – allows you to restrict access per keypair/function

● MFA for user privileged accounts.● Completely remove use of any root account

anywherel Avoid using keypairs outside of AWS

infrastructure (e.g. for external reporting sites)l ROTATE YOUR KEYPAIRS REGULARLY

Leaked keypairs - so you think that's bad?

root

This is always happening to the best of us.

l There are tales of bad security practice everywhere you look.● Adobe hack – hashed but not salted

passwords – 150 million affected● PSN hack – 77 million accounts leaked● Random hackers in Russia appear to be

leaking email/password combos every week, it seems. This means there are STILL companies storing them unencrypted

● #thefappening

Dem blockers though...

l Remember the whole “falling by the wayside” thing? Yeeeaaaahh....● Security is “boring” - often a focus, especially in

startups, of concentrating on the shiny-shiny● People often don't have the time to implement

things “properly”● “Fuck it, it works” - bad security practice is

mostly a temporary test never intended to be permanent, but always ends up that way – root password!

● Change (even to harden security) is seen as “risky”

So what to do?

l FIGHT BACK!● Nobody recognises the value of securing

systems and focusing on security when you're not being hacked

● There's always a higher priority – releases, features, other bugfix

● If you are hacked, it's suddenly TOP PRIORITY and you're strung up for not having done it sooner

l Proactive rather than reactivel Black Ops

● JFDI

Getting people to care about security is a good thing

l If people care, they understand the importance of spending time to implement good security setupsl Get rid of the “But it works now, and X is more

important” mentality. Security isn't only important when it's breached.l More work now to avoid major, major pain later

A final note...

MS14-066 - Vulnerability in SChannel Could Allow Remote Code Execution (2992611)

● Critical vulnerability in Microsoft SChannel – similar to Heartbleed but allows pushing and execution of code

● Patched yesterday● Affects Windows Server 2012, Windows Server 2008 R2 and Windows Server

2003, as well as Windows Vista, 7 and 8● This means every major TLS stack – OpenSSL, GNUTLS, NSS, MS SChannel

and Apple Secure Transport – has had a severe vulnerability this year.