devops & security from an enterprise toolsmith's perspective

Go Fast AND Be Secure?DevOps and Security from an Enterprise

Toolsmith’s Perspective

Alex Honor Damon Edwards

@damonedwards

Damon Edwards Alex Honor

@alexhonor

DevOps ConsultingAutomation Design

OperationsTools

Business Demands

Our #1 priority is moving faster than our competitors!

IT Responds

IT Responds

IT Responds

… but what about security and compliance?

Business Demands

Our #1 priority is moving faster than our competitors!

Our #1 priority is security and compliance! and

IT Under Pressure

Can we go faster and be more secure?

Can we go faster and be more secure?

What gets in the way?

Everything is different

Everything is different● Many servers hand built

Everything is different● Many servers hand built● Custom is the rule

Everything is different● Many servers hand built● Custom is the rule● Inconsistent access control

policy and rules

Everything is different● Many servers hand built● Custom is the rule● Inconsistent access control

policy and rules● Network spaghetti topology

reflects snowflakes

Everything is different● Many servers hand built● Custom is the rule● Inconsistent access control

policy and rules● Network spaghetti topology

reflects snowflakes● … it’s always a network

problem ;-)

Multiplied by Datacenter● Geographically spread ● Generations of

hardware & software ● WAN latencies and

bandwidths ● Sometimes outsourced

Culture clashes between silos

Culture clashes between silos● “Too much change breaks

stuff” - Ops

Culture clashes between silos● “Too much change breaks

stuff” - Ops● “Let me do it myself” - Dev

Culture clashes between silos● “Too much change breaks

stuff” - Ops● “Let me do it myself” - Dev● “This is dangerous!” - Sec

Culture clashes between silos● “Too much change breaks

stuff” - Ops● “Let me do it myself” - Dev● “This is dangerous!” - Sec● “It’s not ready” - QA

Culture clashes between silos● “Too much change breaks

stuff” - Ops● “Let me do it myself” - Dev● “This is dangerous!” - Sec● “It’s not ready” - QA● Finger pointing - everyone

Bureaucracy to get anything delivered“Have you got 27B-6?” - said a guy, in a downstream silo

“I’m a bit of a stickler for paperwork”

“All I need is a ACL/VIP/etc”

It always ends up an escalation● Who yells loudest ● Cube driveby and

who you know ● Crisis at deadline

or outage ● Sometimes still a

rubber stamp

Hard to see how delivery work gets done across the organization

Process Islands Multiple Development teams out here somewhere

Process Islands

“I know there are problems delivering, not sure where, but I know they are outside my island of control”

“We all have the best intentions from our perspective

Process Islands

Process Islands

Process Islands

Process IslandsI really wish to deploy multiple times daily

Friday evening

Process Islands

Monday morning

Process Islands

Process Islands

Everybody on bridge call with the boss

Complicated and self inflicted ● Left hand doesnt know

what the right hand doing

● “Bandaids” and “exception is the rule”

● Telephone and Tribal knowledge

● Low MTTD/MTTR

How do we know when things are getting any better?

You’ll know you are better when...

You’ll know you are better when...● Security policy is applied reliably and consistently

You’ll know you are better when...● Security policy is applied reliably and consistently● Security isn’t the bottleneck

You’ll know you are better when...● Security policy is applied reliably and consistently● Security isn’t the bottleneck● An audit trail is easy to pull together

You’ll know you are better when...● Security policy is applied reliably and consistently● Security isn’t the bottleneck● An audit trail is easy to pull together● Security engineers aren’t left out until the end of the party (or

never consulted)

You’ll know you are better when...● Security policy is applied reliably and consistently● Security isn’t the bottleneck● An audit trail is easy to pull together● Security engineers aren’t left out until the end of the party (or

never consulted)● Everyone has the control they need (without root)

You’ll know you are better when...● Security policy is applied reliably and consistently● Security isn’t the bottleneck● An audit trail is easy to pull together● Security engineers aren’t left out until the end of the party (or

never consulted)● Everyone has the control they need (without root)● Nobody feels like they are having the rug pulled out from

underneath them

Shift left: Host OS SDLC

Collaborate with source code

Artifacts move through the “supply chain”

Bastion host

Bastion host

● centralized access point for authorized access

Bastion host

● centralized access point for authorized access

● disallow home run connections

Bastion host

● centralized access point for authorized access

● disallow home run connections

● dispatcher interfaces remote execution layer

Bastion host

● centralized access point for authorized access

● disallow home run connections

● dispatcher interfaces remote execution layer

● hides network complexity like jump boxes per DC

Bastion host

● centralized access point for authorized access

● disallow home run connections

● dispatcher interfaces remote execution layer

● hides network complexity like jump boxes per DC

User traceability: Delegate account

● User logs in as himself to bastion host ● Remote commands and processes run

under a service account ● Eg, SSH keys used for delegate account

identity

User traceability: End to end

● User logs in as himself to bastion host ● Remote commands executed using

same user account ● Eg., User may raise privilege via sudo

White List and Wrapper

● No ad-hoc interactive logins. ● Use wrapper script and a white list ● Escalate privilege with sudo ● Not foolproof! SELinux still considered too hard for most

eg.: ssh forced command (~/.ssh/authorized_keys: command=wrapper.sh and $SSH_ORIGINAL_COMMAND)

Leverage the toolchain to enforce policy

Leverage the toolchain to enforce policyDesign and code reviews

Leverage the toolchain to enforce policyDesign and code reviews

Code and binary scans

Leverage the toolchain to enforce policyDesign and code reviews

Code and binary scans

“Bake” security tests into your “immune system”

Leverage the toolchain to enforce policyDesign and code reviews

Code and binary scans

“Bake” security tests into your “immune system”

Component vulnerability and governance

Leverage the toolchain to enforce policyDesign and code reviews

Code and binary scans

“Bake” security tests into your “immune system”

Component vulnerability and governance

Access policy and operational security checks

Automate Evidence Collection for Audits

Automate Evidence Collection for AuditsWhat’s the change?

Automate Evidence Collection for AuditsWhat’s the change?

How did you validate the change?

Automate Evidence Collection for AuditsWhat’s the change?

How did you validate the change?

How was the change distributed?

Automate Evidence Collection for AuditsWhat’s the change?

How did you validate the change?

How was the change distributed?

Who did what when and where?

Automate Evidence Collection for AuditsWhat’s the change?

How did you validate the change?

How was the change distributed?

Who did what when and where?

What executed on the node?

Summary

● Shift left ● Bastion host ● User traceability ● White lists and wrappers ● Leverage the toolchain to enforce policy ● Automate evidence collection for audits

● Shift left ● Bastion host ● User traceability ● White lists and wrappers ● Leverage the toolchain to enforce policy ● Automate evidence collection for audits ● ?

Summary