sk8ing on thin ice: a crash course in kubernetes & security · source: verizon dbir - 2016. the...

1The retirement benefit that benefits everyone

Matt Johansen Director of Security

Honest Dollar @mattjay

The retirement benefit that benefits everyone

A Simple, Portable & Affordable Savings Solution

October 2016

Sk8ing on Thin Ice: A Crash Course in Kubernetes & Security


Honest Dollar is a company dedicated to helping people save in

a society built on spending


Information Security Officer

Matt Johansen Long Term Security Guy, Sometimes Developer

SXSW, BlackHat, DEFCON, RSA, more++


Agenda

Security & Kubernetes

What is it? | What is Kubernetes anyway? (This portion will not include how to pronounce it)

Why use it? | “Because $Cool_Unicorn is using it” is not acceptable.

Security Pitfalls | Trust me. There are holes to fall into.

Security Benefits | You mean there are benefits to containerization outside of DevOps?

Examples | No Live Demos™ - But lets walk through some cool security monitoring, logging, alerting, and other tricks. Along with some basic security hygiene.


Kube

Kubernetes


Kube

Kubernetes

Source: Imesh Gunaratne - Intro to Kubernetes


Kubernetes

Things to keep in mind

YAML Land - Config Management is hard

Different Environments - We’re running a company here. Prod, Dev, QA, etc. need processes.

Persistence (and not) - Pods die. Its okay (and sometimes awesome). Manage your data intelligently.

Secret Sauce - Docker is not the magic. Orchestration is the magic. Learn and love your Services, RCs, and Pods.

Kube

Words I might say

Node - Worker VMs.

Pod - Group of Containers.

Replication Controller - Manages Pod lifecycle & config.

Service - Config for a set of Pods.

Cluster - Single network & group of Nodes.


Pitfalls


Cluster


Cluster

Bad Idea

Nearly identical YAMLs. Duplication of work and error prone.


Cluster

Separation of Concerns


Environment Stability

Cluster

Options • Puppet (kinda)

• Shell scripts + templates

• rakefiles + templates (Us Currently)

Lessons Learned • Launch as much as possible, with the fewest

commands possible.

• Track versions and automate container building if possible.

• Automate testing / build pipeline to watch out for regressing over hot fixes.

• Track versions (and dependencies!) between environments


IP

Load Balancing

Watch your logs if you do this. You’ll lose origin IP.

Solution: Break nginx out of cluster (I know, I know)


Loophole

kubectl

Whats wrong with this picture?


Benefits


Security Benefits

Security

Containerization • Patching - Oh you mean `docker build`?

• One process per container - Less vulnerability surface for stuff you don’t use.

• Mobility - Host agnostic

• Segregation - Less pivot surface

• Typically short lifespan - Less old, stale, vulnerable systems.

Kubernetes • Upgrade process - Bring nodes down, bring new

ones up. Pods find their way to a new home.

• Log granularity - Know and prioritize your log events.

• Build pipeline and web hook friendly.*

• Rate of change it enables is incredibly fast.

• Transient by nature.

*Continuous Integration is no longer a ‘nice to have’. For security it is a must.


Fix Fast

Continuous Integration

Good luck with you two week release cycle.

Source: Verizon DBIR - 2016


Watch Everything

Monitoring

Log everything. Watch your logs.

Source: Verizon DBIR - 2016


Monitoring

Log all the things

Logging • ElasticSearch

• rsyslog + RELP

• auditd (go-audit)

• OSQuery

• nginx logs

• Docker logs

Alerting • ElastAlert

• Slack bots

• SSH Connections

• Suspicious commands (curl out of a prod server?)

• File watch

• Anomalies (Geo, IP, data movement, spikes, etc.)


Matt Johansen Director of Security

Honest Dollar @mattjay

The retirement benefit that benefits everyone

A Simple, Portable & Affordable Savings Solution

October 2016

Thank You

sk8ing on thin ice: a crash course in kubernetes & security · source: verizon dbir - 2016. the...

Documents