london devops #9 - security at a startup
TRANSCRIPT
1
2
• At a huge competitive advantage compared to enterprises • No RFPs, Long winded procurement contracts • Outsource commodity services to others:
– Most obviously – Hosting – AWS – Monitoring – Dataloop.io – Alerting – Email – Source control and versioning – Messaging
Startups
3
5
6
7
8
10
?
11
12
- A security oversight in one system can often be used to compromise another
- Are you aware when a new email address gets added? - What’s this IP address in our AWS security group? When was it
added? Who added it? If I remove it what will stop working? - Whose Github account is this? - What’s this hook on my github repo? - The list is endless, and is easily work for a dedicated team - That team will never exist in a startup!- I just want to focus on cutting our deploy time in half and
implementing Docker!
Security Jenga
14
One potential solution… - A light weight process that actively audits security and alerts your
“Security team” when something doesn’t “smell” right - Needs to be extensible to keep up with all those new third party
services being added - Adding services needs to take up as little time as possible
15
Single (secure) point of truth - Can be as simple as a text file on github - We chose LDAP - Deserving in its reputation as a PITA to setup: 6 months to learn, build, test, secure,
and populate (More on that later) - It integrates with everything (More on that later too) - Created a custom schema extension for a Beamly employees
- Github ID - Facebook User UID - Public SSH Key
- At first internal only - Now externally accessible
16
Security “unit” tests • Now you’ve got a version controlled state • If you can programmatically extract a list of users from a
service, you can compare against that • Applies to more than just user accounts:
– AWS Security groups – Github hooks
• Store exceptions with explanatory comments • Track changes with git history
17
Beamly Security Scripts • Implemented as Python “unit” tests (py.unit) • Short & quick to write • Executed every 10 minutes via a scheduled CD pipeline • Failures trigger an email to the platform team • Email can trigger whatever else you want (Pagerduty) • Platform team can login and take appropriate action
18
19
20
21
22
collected 1280 items test_all_github_hooks_are_known.py ........................................................................................................................................................................................................................................................................................................................................................................................... test_aws_iam_users_with_passwords_have_2fa.py ..................................................................................... test_aws_security_groups.py ................................................................................................................................................................................................... test_github_users_and_in_ldap_and_2fa.py ............................................................. test_jira_users_not_in_ldap.py ....................................................................................................................................................................................... test_o365_users.py ........................................................................................................... test_onelogin_users_are_in_ldap.py ........................................................................................... test_pagerduty_users_not_in_ldap.py ........................ test_slack_users_not_in_ldap_and_2fa.py ............................................................F.............................................................................................. =================================== FAILURES =================================== _________________________ test_unknown_slack_users[60] _________________________ realname = ’New Joiner', two_factor_enabled = False def two_factor_enabled(realname, two_factor_enabled): > assert two_factor_enabled E assert False test_slack_users_not_in_ldap_and_2fa.py:42: AssertionError =================== 1 failed, 1279 passed in 145.26 seconds ====================
23
24
25
26
27
Systems Integrated with LDAP - VPN access to VPC - Host access (Using SSH Public keys) - Sudo access (Via group) - Office appliances (NAS) - ….any application that supports it – GO, Grafana (Soon)
Couldn’t have done this if we’d have used a text file
28
29
• Herd all these 3rd party applications in to one place • Password manager on steroids • Centrally managed • Every application used by Beamly added by Platform team • Authenticate with LDAP account and 2FA • Single sign on just to services that users have rights to (LDAP groups) • Users never know the password for shared systems • Can bulk rotate shared passwords for all systems
OneLogin
30
31
32
Now what?!
33
“Preppers”!
37
BE A PREPPER!
Part I: Stock your stores - Backups • Taking backups isn’t good enough • 3 -2 -1 • 3 Copies of your data • 2 in different formats • 1 Offsite • S3 mutli-region ISN’T OFF SITE • S3 > (Another cloud provider || On site) • March 31st: World backup day
Part II: Run through restoring from nothing • In the event of the Armageddon, you need to fight to get back up quickly • How long would if take to restore your AWS “Scaffolding” alone
– VPC Configuration – Security groups – Routing tables – Subnet configurations – DNS
• Ideally you’d have all this in Cloudformation…. • ….We set all this stuff up before Cloudformation was available • AWS Cloudformer
41
To sum up: our approach • If a service supports 2FA, it’s mandatory.
– Currently Slack, Mailchimp, Github, AWS, and Onelogin • If a service has an API for exporting a security config: try and script an
audit for it • All third party services in Onelogin • Bulk rotate shared passwords regularly • Prepare for Armageddon:!
– Backup CD config & AWS setup – All data tiers backed up 3-2-1 (One offsite)
42
Lessons Learned • Retro-fitting security is hard • Be prepared to break lots of things (temporarily) • Bring everyone along for the ride
– Explain why it’s important – Everyone gets the access they need – #security for interested parties
• Turn on AWS Cloudtrail and Config everywhere
LONDON
Drury House 34-43 Russell Street
London WC2B 5HA
NEW YORK CITY
84 Wooster Street Suite 703 New York NY 10012
SYDNEY
22-36 Mountain Street Suite 1.10
Ultimo NSW 2007
© Beamly 2014. All Rights Reserved.