(cmp406) amazon ecs at coursera: a general-purpose microservice

© 2015, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

Frank Chen, Coursera

Brennan Saeta, Coursera

October 2015

CMP406

Amazon ECS at CourseraPowering a general-purpose near-line execution

microservice, while defending against untrusted code

What to Expect from the Session

• Techniques for a unified near-line, batch, and scheduled

micro-service powered by Amazon ECS

• Security vulnerabilities and countermeasures when

running untrusted code in Docker with Amazon ECS

• Reasons to modify the Amazon ECS agent

Session Outline

• Introduction to Coursera

• Near-line, batch and scheduled job execution framework

• Motivations and background

• Amazon ECS benefits and limitations

• Iguazú and its architecture

• Evaluating programming assignments

• System requirements

• Security threat model

• Attacks and defenses

Education at Scale

15 million learners worldwide

2.5 millioncourse completions

1,300+courses

125+partners

A unified execution framework

Batch Processing Enables…

Reporting

Instructor Reports

• Grade exports

• Learner demographics

• Course progress

statistics

Internal Reports

• Business metrics

• Payments

reconciliation

Scheduled Processing Enables…

Marketing

• Recommendation emails

• Targeted marketing / reactivation emails

Nearline Processing Enables…

Pedagogical Innovations

• Peer-review matching & analysis

• Auto-graded programming assignments

The early days…

January 2012

Bad Old Days of Batch Processing @ Coursera

Cascade

• PHP-based job runner

• Originally ran in screen sessions

• Polled APIs for new jobs

• Forced restarts on regular basis

due to unidentified memory leaks

• Fragile and unreliable

The early

days…

Bad Old Days of Batch Processing @ Coursera

Saturn

• Scala scheduled batch job runner• Powered by Quartz Scheduler library

• Better than Cascade, but…

• All jobs ran on same JVM, causing

interference

The not-

so early

days?

Looking for something better…

What We Wanted

Reliable Easy Development Easy Deployment

High Efficiency Low Ops Load Cost Effective

What Else Did We Look At?

Home-grown Tech

• Tried, but proved

to be unreliable

• Difficult to

handle

coordination and

synchronization

• Powerful, but

hard to

productionize

• Needs

developers with

experience

• Designed for

GCE first

• Not a managed

service, higher

Ops load

Amazon ECS to the Rescue

Amazon re:Invent 2014 – Dr. Werner Vogels introducing Amazon ECS

Screenshot from https://www.youtube.com/watch?v=LE5uBqNp2Ds by Amazon Web Services

https://www.youtube.com/watch?v=LE5uBqNp2Ds

Amazon ECS to the Rescue

Little

maintenance

Integrated with

rest of AWSEasy to

develop for

However…

Amazon ECS is a great building block,

but we still need to build tools around it

for our purposes.

What We Built: Iguazú

Marissa Strniste (https://www.flickr.com/photos/mstrniste/5999464924) CC-BY-2.0

• Batch Job Scheduler for Amazon ECS

• Immediately

• Deferred (run once at X time)

• Scheduled recurring (cron-like)

• Programmatically accessible internally via

our standard APIs and clients

• Named for Iguazú falls

• World’s largest waterfall by volume

• We hope Iguazú handles a similar volume of jobs

Iguazú

Frontend

Iguazú

SchedulerIguazú

Backend

Iguazú: Architecture

CassandraServices Services

Iguazú

Admin

ECS

Workers

SQS

ECS API

Devs

Users

Developing Iguazú Jobs

class Job extends AbstractJob with StrictLogging {

override val reservedCpu = 1024 // 1 CPU core

override val reservedMemory = 1024 // 1 GB RAM

def run(parameters: JsValue) = {

logger.info("I am running my job! ")

expensiveComputationHere()

}

}

Running Jobs from Other Services

// invoking a job with one function call

// from another service via Naptime RPC/REST framework

val invocationId = IguazuJobInvocationClient

.create(IguazuJobInvocationRequest(

jobName = "exportQuizGrades",

parameters = quizParams))

Iguazú: Developer / Ops User Interface

Deploying Jobs

Easy Deployment

1. Developers Merge into master. Done!

Jenkins Build Steps:

1. Builds zip package from master

2. Prepares Docker image with zip file

3. Pushes image into Docker registry

4. Registers updated jobs with

Amazon ECS API

Logs

• Logs are in /var/lib/docker/containers/*

• Upload into log analysis service (Sumologic)

• Wrapper prints out job name and job ID

at the start for easy searching

• Good enough for now

Metrics

• Using third-party metrics collector (Datadog)

• Metrics for both jobs and container instances

• So long as the worker machines can talk to Internet,

things will work out pretty well

Since April 2015…

65 jobs in

production

>1000 runs

per day

44 different

scheduled jobs

Evaluating

Programming Assignments

Programming Assignments at Coursera

The Security Challenge

Compiling and running untrusted, arbitrary code in

Amazon EC2

Would you like to compile and run C code from random

people on the Internet on your servers?

1st Generation System

Class graders in

separate AWS acct

Custom grader systems

on cloud providers

Course grader under the

instructor’s desk

Learners Coursera Servers Queue Service

1st Generation System: Weaknesses

No Auto Scaling No standard security Graders crashed

Design Goals

Cost Savings No Maintenance Near Real-time Secure Infrastructure

Threat Model

Prevent submitted code from:

• impacting the evaluation of other submissions.

• disrupting the grading environment (e.g., DoS)

• affecting the rest of the Coursera learning platform

Additional goals:

• Minimize exfiltration of information

• Test cases, solutions, etc…

• Minimize risk of submissions changing own scores

• Avoid turning into bitcoin miners or part of botnet

Threat Model - Assumptions

• Run arbitrary binaries

• Instructor grading scripts may have vulnerabilities

• ∴ Grading code is untrusted

• Unknown vulnerabilities in Docker and Linux name-

spacing and/or container implementation

Attack / Vulnerability Classes

Divided into 2 main categories:

• Assuming basic containers are secure, prevent any

negative impacts to running arbitrary code.

• Assuming basic container technology is vulnerable,

mitigate negative impacts as much as possible.

What We Built: GrID

Patrick Hoesly (https://www.flickr.com/photos/zooboing/5665221326/) CC-BY-2.0

• Service + architecture for grading

programming assignments

• Builds on Amazon ECS and Iguazú

• Named for Tron’s “digital frontier”

• Backronym: Grading Inside Docker

High-level GrID Architecture

Learners

GrID

Iguazú

S3 Bucket

ECS APIs

Grading MachinesVPC Firewalls

Coursera Production Account Coursera GrID Grading Account


Learners

GrID

Iguazú

S3 Bucket

ECS API

Grading MachinesVPC Firewalls

Production Acct GrID Grading Account


Learners

GrID

Iguazú

S3 Bucket

ECS API

Grading

Machines

VPC

Firewalls

Production Acct GrID Grading Account

Attacks: Resource Exhaustion

Defenses:

• Docker / CGroups:

• CPU quotas

• Memory limits

• Swap limits

• Hard timeouts for container execution

• btrfs limits

• file system storage quotas

• IOPS throttling

Attacks: Kernel Resource Exhaustion

Defenses:

• Open file limits per container (nofile)

• nproc Process limits

• Limit kernel memory per cgroup

• Limit execution time

Attacks: Network attacks

Attacks:

• Bitcoin mining

• DoS attacks on third-party systems

• Access Amazon S3 and other AWS

APIs

Defense:

• Deny network access

Modifying the ECS Agent: Network Modes

• NetworkDisabled too restrictive

• Some graders require local loopback

• Feature also deprecated

• --net=none + deny net_admin + audit network• Isolation via Docker creating an

independent network stack for each

container

• github.com/coursera/amazon-ecs-agent

https://github.com/coursera/amazon-ecs-agent

Attacks: Namespace / Container Vulnerabilities

• App Armor & Mandatory Access Control

• Required modifying the Amazon ECS Agent

• Allows auditing or denying access to a

variety of subsystems

• Drop capabilities

• No need for NET_BIND_SERVICE, CAP_FOWNER

• No root within container

Attacks: Root escalations within the container

• We modify instructor grader images

before allowing them to be run

• Clears setuid

• Inserts C wrapper to drop privileges from

root and redirect stdin/stdout/stderr

• Required Amazon ECS Agent

modification

• Grant root privileges

• Map Docker socket into Docker

containers to run Docker in Docker!

Attacks: If all else fails…

• Utilizes VPC security measures to

further restrict network access

• No public internet access

• Security group to restrict

inbound/outbound access

• Network flow logs for auditing

• Separate AWS account

• Run in an Auto Scaling group

• Regularly terminate all grading EC2

instances

Other Security Measures

• Utilize AWS CloudTrail for audit logs

• Third-party security monitoring

(Threat Stack)• No one should log in, so any TTY is an alert

• Penetration testing by third-party red

team (Synack)

Technique: Co-process

• Environment has no network, but has to

get submissions in and results out

• Python co-process watches Amazon ECS

/ Docker

• Python co-process then:• Mounts a shared folder containing submission

• Reads back the grade from the shared folder

after container exits

• Monitors and cleans up

Future Improvements

• Priority queues for different grading

priorities

• Re-grades vs on-demand grades

• Better instructor tooling

• Automated “unit-testing” for new graders

• Better simulation of production

environment on instructor machines

• Support scheduling GPUs

Lessons Learned

• Run the latest kernels

• Latest security patches

• btrfs wedging on older kernels

• Default Ubuntu 14.04 kernel not new

enough!

• Carefully monitor disk usage

• Docker-in-docker can’t clean up after

itself (yet).

• Reliable deploy tooling pays for itself

Related Sessions

Also from Coursera:

• BDT404 - Building and Managing Large-Scale ETL Data

Flows with AWS Data Pipeline and Dataduct - Friday

Containers and Amazon ECS:

• CMP302 - Amazon EC2 Container Service: Distributed

Applications at Scale – Next timeslot in Venetian H

Thank you!

Questions?

Also, we are hiring!

www.coursera.org/jobs

tech.coursera.org

Brennan Saetagithub/saeta

@bsaeta

[email protected]

Frank Chengithub/frankchn

@frankchn

[email protected]

http://www.coursera.org/jobs

https://tech.coursera.org/

Remember to complete

your evaluations!

(cmp406) amazon ecs at coursera: a general-purpose microservice

Technology