prepare to defend thyself with blue/green

Fabian Lim

GovTech Singapore

Defending ThyselfWith

Blue

Gree

nAll Day DevOps

2016

My Story-

about.me/Fabian.Lim

● Xxxxxxxxxx Engineer

○ http://tech.gov.sg

● Passionate about CICD

● Cultural Hacking

● Always Thinking about Red Teaming Possibilities

https://about.me/Fabian.Lim




http://tech.gov.sg

http://tech.gov.sg

http://www.geograph.org.uk/photo/3084819

Black Box Deployment?

Who has an automated CICD pipeline?

GitHub

Source Code Check in

Jenkins

Build Tools

S3

Artifact Repository

CodePipeline

Deployment Tools

CodeDeploy

Production Environment

EC2

CICD Pipeline is awesome

GitHub


Jenkins

Build Tools

S3

Artifact Repository

CodePipeline

Deployment Tools

CodeDeploy


EC2

Code change is introduced

GitHub


Jenkins

Build Tools

S3

Artifact Repository

CodePipeline

Deployment Tools

CodeDeploy


EC2

Build starts, and passes!

GitHub


Jenkins

Build Tools

S3

Artifact Repository

CodePipeline

Deployment Tools

CodeDeploy


EC2

Artifact is dropped into repository

GitHub


Jenkins

Build Tools

S3

Artifact Repository

CodePipeline

Deployment Tools

CodeDeploy


EC2

CD Tool picks up new artifact for testing

GitHub


Jenkins

Build Tools

S3

Artifact Repository

CodePipeline

Deployment Tools

CodeDeploy


EC2

CD Tool uses new artifact to deploy

GitHub


Jenkins

Build Tools

S3

Artifact Repository

CodePipeline

Deployment Tools

CodeDeploy


EC2

New/latest code is deployed on an instance!

GitHub


Jenkins

Build Tools

S3

Artifact Repository

CodePipeline

Deployment Tools

CodeDeploy


EC2

Shoot, I need to revert the previous code change! The whole CICD process repeats...

GitHub


Jenkins

Build Tools

S3

Artifact Repository

CodePipeline

Deployment Tools

CodeDeploy


EC2


GitHub


Jenkins

Build Tools

S3

Artifact Repository

CodePipeline

Deployment Tools

CodeDeploy


EC2

Build starts, passes!

GitHub


Jenkins

Build Tools

S3

Artifact Repository

CodePipeline

Deployment Tools

CodeDeploy


EC2


GitHub


Jenkins

Build Tools

S3

Artifact Repository

CodePipeline

Deployment Tools

CodeDeploy


EC2

CD Tool uses new artifact to deploy

GitHub


Jenkins

Build Tools

S3

Artifact Repository

CodePipeline

Deployment Tools

CodeDeploy


EC2

Changes in code is deployed on the same instance

Downsides

● Downtime (SLA)● Previous State of

Deployment is overwritten● Resistance to

Infrastructure Changes● Relatively Sequential

(Traffic Jam Scenario)

Who has an automated CICD pipeline,with blue-green deployment?

Enough TalkShow Me the Money

GitHub


Jenkins

Continuous Integration

S3

Artifact Repository Code

Pipeline

Continuous Deployment

CodeDeploy


CodePipeline Code

Deploy EC2

EC2

Red outline indicates current CICD workflow

Blue-green to the rescue!

(LIV

E)

GitHub


Jenkins


S3


Pipeline


CodeDeploy


CodePipeline Code

Deploy EC2

EC2



(LIV

E)

GitHub


Jenkins


S3


Pipeline


CodeDeploy


CodePipeline Code

Deploy EC2

EC2


Build starts, and passes!

(LIV

E)

GitHub


Jenkins


S3


Pipeline


CodeDeploy


CodePipeline Code

Deploy EC2

EC2



(LIV

E)

GitHub


Jenkins


S3


Pipeline


CodeDeploy


CodePipeline Code

Deploy EC2

EC2


CD Tool picks up new artifact for deployment

(LIV

E)

GitHub


Jenkins


S3


Pipeline


CodeDeploy


CodePipeline Code

Deploy EC2

EC2


Changes in code is deployed on green, going live

(LIV

E)

(LIV

E)

GitHub


Jenkins


S3


Pipeline


CodeDeploy


CodePipeline Code

Deploy EC2

EC2


Completely switch over to green, decommission blue

(LIV

E)

GitHub


Jenkins


S3


Pipeline


CodeDeploy


CodePipeline Code

Deploy EC2

EC2


Known good code / state is preserved on blue

(LIV

E)

GitHub


Jenkins


S3


Pipeline


CodeDeploy


CodePipeline Code

Deploy EC2

EC2


Completely switch over to blue, decommission green

(LIV

E)

How does blue-green defend thy

stack?

● Minimise Downtime● Preserved Last Known

Deployment● Robust Infrastructure● Parallel Pipelines

Minimise Downtime

● Service and Data Availability

● SLA○ Business Critical

Preserved Last Known

Deployment

● Rollback enabled● Debugging● Forensics

○ Take it offline and isolate

○ Selfie* anyone?

*https://alldaydevops2016.sched.org/event/8614/taking-a-selfie-just-try-to-resist-doing-forensics-the-devsecops-way

● Resilient to Security Testing and Fire Drills

● Restore to known good state○ “Refresh” stack

Robust Infrastructure

Robust Infrastructure

● Vulnerability Management○ Quick to patch zero days

- app to infra layers○ One New Zero-Day

Vulnerability Discovered on Average Every Week*

*https://www.symantec.com/en/aa/about/newsroom/press-releases/2016/symantec_0413_01

Parallel Pipelines

● Why stop at blue-green?● Go RAINBOW!● Scale

○ Restore to multiple states on multiple instances

Enough TalkShow Me the RAINBOW

What makes it easier to do blue-green?

Crawl? ● Virtualization

● Infrastructure and Security as Code

● Build a CICD process

● Plan for Security Testing

● Identify where Blue-Green is relevant*

*https://d0.awsstatic.com/whitepapers/AWS_Blue_Green_Deployments.pdf

Walk

Run

Walk.

● Automate Existing CICD Process

● Incorporate Security Testing

● Manually implement Blue-Green where relevant

Crawl

Run

● Automate Blue-Green!

● Automate Security Testing

● Scale RAINBOW!

Walk

Crawl

Run!

GitHub


Jenkins


S3


Pipeline


CodeDeploy


CodePipeline Code

Deploy EC2

EC2

Just securing the end product isn’t enough, it’s the WHOLE pipeline that’s the attack vector

(LIV

E)

Red arrows indicate attack vectors

What do youmean secure CICD?

How would you feel if your build tools and deployment tools are down / hacked?

(╯°□°）╯︵ ┻━┻

Yeah, me too.

GitHub


Jenkins


S3


Pipeline


CodeDeploy


CodePipeline Code

Deploy EC2

EC2

Emphasize on securing pipeline as much as end-product

(LIV

E)

Convincing your team to go secure blue-green?

Red team* it; to fix it.

GoalMethod

*https://alldaydevops2016.sched.org/event/861E/operationalizing-red-team-for-fun-and-profit

https://github.com/fabianlim1989/DefendThyselfBlueGreen



Time: 25 man-hoursMoney: ~ $10Engineer: 1Software: FreeSecurity: Built-In, Priceless

AppreciateYourKindAttention

Thank you

Gracias

谢谢

Terima Kasih

Gam Xia