Join the conversation #DevSecCon

BY ALINA RADU

Permitting agility while enforcing security - a story of making Docker a win-win proposition

Overview

• PaaSTA: Yelp’s open source Platform as a service• Microservices in Docker containers• Jenkins

• build pipelines• multiple steps

• security-check step• runs a suite of security tests

• Notification system for failures & runbook• Take aways

whoami

• Alina• Software Engineer at Yelp London• Security team: Infrastructure security• Politehnica University of Bucharest Alumni• Avid traveller

Yelp’s MissionConnecting people with great

local businesses.

PaaSTA: Platform as a ServiceMesos, Marathon & Chronos

• Platform as a Service• Mesos

• distributed job scheduler• Marathon

• Mesos framework for long running tasks• Chronos

• Mesos framework to schedule batch jobs

PaaSTA: Platform as a ServiceMesos, Marathon & Chronos

• Mesos• distributed job scheduler• master & agents• offers compute resources to frameworks• frameworks provide task - schedules it on an agent

PaaSTA: Platform as a ServiceMesos, Marathon & Chronos

• Marathon• Mesos framework for long running tasks• upstart at datacenter level

PaaSTA: Platform as a ServiceMesos, Marathon & Chronos

• Chronos• Mesos framework to schedule batch jobs

PaaSTA: Platform as a Service

• microservices in Docker containers• autoscaling cluster & resource specification• move from:

• hundreds of specialized servers to• heterogenous Mesos agents

• SOA architecture• operational ownership of individual services

• from operations team to the service authors

PaaSTA contract

• service: 1 git repo and 1 Dockerfile• Docker image

• runs the service• same image multiple use cases: worker daemon vs web task

• contains all the code necessary for the service• Service:

• stateless• filesystem I/O but ephemeral disk• log to external processors (Yelp: Scribe or Kafka)

• all checked, all good

Build Pipeline of a Service

• configuration repository• Jenkins

• orchestrates build and deployment• pipelines of sequential steps• security-check step

PaaSTA security-check

• security status of the service• run a set of tests at every build

• high level security health of the service• something changed?

• actionable alerts for failures• faster response time from team

Security tests

• Ubuntu packages up to date• Docker container best practices• Well known vulnerabilities• No secrets in the service repo• Python/Java dependency check

Ubuntu packages up to date

• Check if the latest packages are installed against our apt repositories

• apt-get update && apt-get --simulate dist-upgrade

Docker container best practices

• container not running as user root• Dockerfile

• Yelp maintained Docker images, no public images• Latest images• no packages pinned to certain versions• .dockerignore contains .git

Whitelisting

• Certain version(s) of a package• Docker images

• public images for open source projects• non-standard Yelp images

Well known vulnerabilities

• bash shellshock• extending the list in the future

• heartbleed - server side applications

No secrets into the service repo

• detect and prevent high entropy strings from entering our code base

• assumes the existing code has no secrets• checks only the new code• solution loosely based off truffleHog

Package vulnerability checking

• look for known vulnerabilities in python/Java packages

• Docker container scanning• vulnerabilities in Ubuntu packages• classified by severity• weekly report via email

Failures & alerts

• Solves: ticket creation to track failures that need to be fixed• security-check failed?

• email• Jira ticket• sensu

• Runbook

How we got here

• run bash tests with goss• get around the libc incompatibilities with rspec• moving everything to python

• race condition between the Ubuntu packages that are updated upstream and the Docker base images we build daily

• some services are not built regularly

You can do it too

• You don’t have to• run our PaaS (PaaSTA)• use our containerisation solution (Docker)

• write high-level security tests• integrate them in your build pipeline• open source our security tests suite next year

Take aways

• PaaSTA• PaaS • microservices in Docker containers• build pipeline: multiple steps

• security-check • runs a set of tests and send notifications

• most important - service owners:• more aware of the security of their service• involved in keeping it safe

@YelpEngineering

fb.com/YelpEngineers

engineeringblog.yelp.com

github.com/yelp

www.yelp.com/careers/

We're Hiring!

Join the conversation #DevSecCon

Thank you

Join the conversation #DevSecCon

Q & A