devops & security from an enterprise toolsmith's perspective
TRANSCRIPT
Go Fast AND Be Secure?DevOps and Security from an Enterprise
Toolsmith’s Perspective
Alex Honor Damon Edwards
@damonedwards
Damon Edwards Alex Honor
@alexhonor
DevOps ConsultingAutomation Design
OperationsTools
Business Demands
Our #1 priority is moving faster than our competitors!
IT Responds
IT Responds
IT Responds
… but what about security and compliance?
Business Demands
Our #1 priority is moving faster than our competitors!
Our #1 priority is security and compliance! and
IT Under Pressure
Can we go faster and be more secure?
Can we go faster and be more secure?
What gets in the way?
Everything is different
Everything is different● Many servers hand built
Everything is different● Many servers hand built● Custom is the rule
Everything is different● Many servers hand built● Custom is the rule● Inconsistent access control
policy and rules
Everything is different● Many servers hand built● Custom is the rule● Inconsistent access control
policy and rules● Network spaghetti topology
reflects snowflakes
Everything is different● Many servers hand built● Custom is the rule● Inconsistent access control
policy and rules● Network spaghetti topology
reflects snowflakes● … it’s always a network
problem ;-)
Multiplied by Datacenter● Geographically spread ● Generations of
hardware & software ● WAN latencies and
bandwidths ● Sometimes outsourced
Culture clashes between silos
Culture clashes between silos● “Too much change breaks
stuff” - Ops
Culture clashes between silos● “Too much change breaks
stuff” - Ops● “Let me do it myself” - Dev
Culture clashes between silos● “Too much change breaks
stuff” - Ops● “Let me do it myself” - Dev● “This is dangerous!” - Sec
Culture clashes between silos● “Too much change breaks
stuff” - Ops● “Let me do it myself” - Dev● “This is dangerous!” - Sec● “It’s not ready” - QA
Culture clashes between silos● “Too much change breaks
stuff” - Ops● “Let me do it myself” - Dev● “This is dangerous!” - Sec● “It’s not ready” - QA● Finger pointing - everyone
Bureaucracy to get anything delivered“Have you got 27B-6?” - said a guy, in a downstream silo
“I’m a bit of a stickler for paperwork”
“All I need is a ACL/VIP/etc”
It always ends up an escalation● Who yells loudest ● Cube driveby and
who you know ● Crisis at deadline
or outage ● Sometimes still a
rubber stamp
Hard to see how delivery work gets done across the organization
Process Islands Multiple Development teams out here somewhere
Process Islands
“I know there are problems delivering, not sure where, but I know they are outside my island of control”
“We all have the best intentions from our perspective
Process Islands
Process Islands
Process Islands
Process IslandsI really wish to deploy multiple times daily
Friday evening
Process Islands
Monday morning
Process Islands
Process Islands
Everybody on bridge call with the boss
Complicated and self inflicted ● Left hand doesnt know
what the right hand doing
● “Bandaids” and “exception is the rule”
● Telephone and Tribal knowledge
● Low MTTD/MTTR
How do we know when things are getting any better?
You’ll know you are better when...
You’ll know you are better when...● Security policy is applied reliably and consistently
You’ll know you are better when...● Security policy is applied reliably and consistently● Security isn’t the bottleneck
You’ll know you are better when...● Security policy is applied reliably and consistently● Security isn’t the bottleneck● An audit trail is easy to pull together
You’ll know you are better when...● Security policy is applied reliably and consistently● Security isn’t the bottleneck● An audit trail is easy to pull together● Security engineers aren’t left out until the end of the party (or
never consulted)
You’ll know you are better when...● Security policy is applied reliably and consistently● Security isn’t the bottleneck● An audit trail is easy to pull together● Security engineers aren’t left out until the end of the party (or
never consulted)● Everyone has the control they need (without root)
You’ll know you are better when...● Security policy is applied reliably and consistently● Security isn’t the bottleneck● An audit trail is easy to pull together● Security engineers aren’t left out until the end of the party (or
never consulted)● Everyone has the control they need (without root)● Nobody feels like they are having the rug pulled out from
underneath them
Shift left: Host OS SDLC
Collaborate with source code
Artifacts move through the “supply chain”
Bastion host
Bastion host
● centralized access point for authorized access
Bastion host
● centralized access point for authorized access
● disallow home run connections
Bastion host
● centralized access point for authorized access
● disallow home run connections
● dispatcher interfaces remote execution layer
Bastion host
● centralized access point for authorized access
● disallow home run connections
● dispatcher interfaces remote execution layer
● hides network complexity like jump boxes per DC
Bastion host
● centralized access point for authorized access
● disallow home run connections
● dispatcher interfaces remote execution layer
● hides network complexity like jump boxes per DC
User traceability: Delegate account
● User logs in as himself to bastion host ● Remote commands and processes run
under a service account ● Eg, SSH keys used for delegate account
identity
User traceability: End to end
● User logs in as himself to bastion host ● Remote commands executed using
same user account ● Eg., User may raise privilege via sudo
White List and Wrapper
● No ad-hoc interactive logins. ● Use wrapper script and a white list ● Escalate privilege with sudo ● Not foolproof! SELinux still considered too hard for most
eg.: ssh forced command (~/.ssh/authorized_keys: command=wrapper.sh and $SSH_ORIGINAL_COMMAND)
Leverage the toolchain to enforce policy
Leverage the toolchain to enforce policyDesign and code reviews
Leverage the toolchain to enforce policyDesign and code reviews
Code and binary scans
Leverage the toolchain to enforce policyDesign and code reviews
Code and binary scans
“Bake” security tests into your “immune system”
Leverage the toolchain to enforce policyDesign and code reviews
Code and binary scans
“Bake” security tests into your “immune system”
Component vulnerability and governance
Leverage the toolchain to enforce policyDesign and code reviews
Code and binary scans
“Bake” security tests into your “immune system”
Component vulnerability and governance
Access policy and operational security checks
Automate Evidence Collection for Audits
Automate Evidence Collection for AuditsWhat’s the change?
Automate Evidence Collection for AuditsWhat’s the change?
How did you validate the change?
Automate Evidence Collection for AuditsWhat’s the change?
How did you validate the change?
How was the change distributed?
Automate Evidence Collection for AuditsWhat’s the change?
How did you validate the change?
How was the change distributed?
Who did what when and where?
Automate Evidence Collection for AuditsWhat’s the change?
How did you validate the change?
How was the change distributed?
Who did what when and where?
What executed on the node?
Summary
● Shift left ● Bastion host ● User traceability ● White lists and wrappers ● Leverage the toolchain to enforce policy ● Automate evidence collection for audits
● Shift left ● Bastion host ● User traceability ● White lists and wrappers ● Leverage the toolchain to enforce policy ● Automate evidence collection for audits ● ?
Summary