![Page 1: AWS re:Invent 2016: Securing Container-Based Applications (CON402)](https://reader034.vdocuments.site/reader034/viewer/2022042907/587126161a28abe4448b6285/html5/thumbnails/1.jpg)
© 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
December 2016
CON402
Securing
Container-Based Applications
Henrik Johansson, Security Solutions Architect, AWS
Michael Capicotto, Solutions Architect, AWS
![Page 2: AWS re:Invent 2016: Securing Container-Based Applications (CON402)](https://reader034.vdocuments.site/reader034/viewer/2022042907/587126161a28abe4448b6285/html5/thumbnails/2.jpg)
What to Expect from the Session
OS & Container Security
• Intro to container security and best practices
• Container lifecycle security (including vulnerability analysis)
• Managing secrets
Infrastructure security
• Enforce governance with Amazon ECS
• Using containers to automate security deployments
![Page 3: AWS re:Invent 2016: Securing Container-Based Applications (CON402)](https://reader034.vdocuments.site/reader034/viewer/2022042907/587126161a28abe4448b6285/html5/thumbnails/3.jpg)
Intro to Container Security
![Page 4: AWS re:Invent 2016: Securing Container-Based Applications (CON402)](https://reader034.vdocuments.site/reader034/viewer/2022042907/587126161a28abe4448b6285/html5/thumbnails/4.jpg)
Container vs VM isolation
![Page 5: AWS re:Invent 2016: Securing Container-Based Applications (CON402)](https://reader034.vdocuments.site/reader034/viewer/2022042907/587126161a28abe4448b6285/html5/thumbnails/5.jpg)
Isolation
• Implemented via Linux namespaces
• Weaker than VM isolation
• Containers run on single kernel
• Containers share files, sockets, memory areas,
devices, etc.
![Page 6: AWS re:Invent 2016: Securing Container-Based Applications (CON402)](https://reader034.vdocuments.site/reader034/viewer/2022042907/587126161a28abe4448b6285/html5/thumbnails/6.jpg)
Secure by default
pid namespace
nmt namespace
net namespace
uts namespace
user namespace
pivot_root
uid/gid drop
cap drop
all cgroups
selinux
apparmor
seccomp
Out-of-the-box default settings and profiles
Granular controls to customize settings
No SSH to containers
No system users
Immutable infrastructure
Secure
by
default
![Page 7: AWS re:Invent 2016: Securing Container-Based Applications (CON402)](https://reader034.vdocuments.site/reader034/viewer/2022042907/587126161a28abe4448b6285/html5/thumbnails/7.jpg)
Defense through segmentation
• Run containers on top of virtual instances
• Keep instance and container up to date
• IAM roles for tasks to restrict host permissions
• Segregate containers
• Role/customer
• Risk/trust/exposure
![Page 8: AWS re:Invent 2016: Securing Container-Based Applications (CON402)](https://reader034.vdocuments.site/reader034/viewer/2022042907/587126161a28abe4448b6285/html5/thumbnails/8.jpg)
Reduce attack surface area
• Container attack surface much smaller than VM
• Service, not system
• Emulate drivers
• VENOM attack / CVE-2015-3456
(http://venom.crowdstrike.com/)
• Best practice: run small images
• Only contain static binary
• Harden the cluster instance
• NIST, SANS, etc.
• Use many but smaller instances to limit blast radius
![Page 9: AWS re:Invent 2016: Securing Container-Based Applications (CON402)](https://reader034.vdocuments.site/reader034/viewer/2022042907/587126161a28abe4448b6285/html5/thumbnails/9.jpg)
Best practices
• Limit memory, CPU allocation and resource limits
• Set filesystems to be read-only
• Limit container networking
• Remove setuid/setgid binaries from images
• Set containers to run as non-root user
• Leverage Linux kernel security features
![Page 10: AWS re:Invent 2016: Securing Container-Based Applications (CON402)](https://reader034.vdocuments.site/reader034/viewer/2022042907/587126161a28abe4448b6285/html5/thumbnails/10.jpg)
Setting resource limits in ECS
Configured in the ECS task definition
• CPU
• Minimum number of CPU units to reserve for container
• RAM
• The number of MiB of memory reserved for container
• Ulimits
• Reduce for example filesize locks and memlocks allowed
"ulimits": [
{
"name": "core"|"cpu"|"data"|"fsize"|"locks"|"memlock"|"msgqueue"|"nice"|"nprock"…
"softLimit": integer,
"hardLimit": integer
}
...
]
![Page 11: AWS re:Invent 2016: Securing Container-Based Applications (CON402)](https://reader034.vdocuments.site/reader034/viewer/2022042907/587126161a28abe4448b6285/html5/thumbnails/11.jpg)
• Configured in the ECS task definition
• Setting root file system to be read-only
• Maps to Docker ReadOnlyRootfs and --read-only
• Set volumes to be read-only
Read-only filesystem access
"readonlyRootFileSystem": true
"mountPoints": [
{
"sourceVolume": "string",
"containerPath": "string",
"readOnly": true
}
]
![Page 12: AWS re:Invent 2016: Securing Container-Based Applications (CON402)](https://reader034.vdocuments.site/reader034/viewer/2022042907/587126161a28abe4448b6285/html5/thumbnails/12.jpg)
Securing container networking
• Containers should only open ports it needs
• Governance on task definition
• Pre deploy assessment
• Know expected traffic patterns
• North/South, East/West
• Disable inter-container communication
• Set Docker option flag --icc=false and --iptables=true
• Use --link to connect containers
![Page 13: AWS re:Invent 2016: Securing Container-Based Applications (CON402)](https://reader034.vdocuments.site/reader034/viewer/2022042907/587126161a28abe4448b6285/html5/thumbnails/13.jpg)
Securing container networking -
• Detects threats from external and internal networks
• Attacks such as DDOS and XSS on containers
• Protects your container network from L3 to L7
• Detects application threats
• Has built-in security policies for 30+ applications and
protocols
• Prevents unauthorized connections between containers
• Monitors all ‘east-west’ container traffic
• Integrate logs with SIEM/Log aggregator
![Page 14: AWS re:Invent 2016: Securing Container-Based Applications (CON402)](https://reader034.vdocuments.site/reader034/viewer/2022042907/587126161a28abe4448b6285/html5/thumbnails/14.jpg)
Remove setuid/setgid binaries
• Most apps don’t need access to setuid/setgid binaries
• Remove to avoid privilege escalation attacks
• Example Debian “defanged ” image Dockerfile:
FROM debian:wheezy
RUN find / -perm +6000 –type f -exec chmod a-s {} \; || true
![Page 15: AWS re:Invent 2016: Securing Container-Based Applications (CON402)](https://reader034.vdocuments.site/reader034/viewer/2022042907/587126161a28abe4448b6285/html5/thumbnails/15.jpg)
Set a non-root user
Multiple methods
• Create the USER in Dockerfile
• Change to the user via USER command or sudo/gosu/su
• Set USER in Docker command or ECS task def
Don’t use root as default!
![Page 16: AWS re:Invent 2016: Securing Container-Based Applications (CON402)](https://reader034.vdocuments.site/reader034/viewer/2022042907/587126161a28abe4448b6285/html5/thumbnails/16.jpg)
Seccomp
• Secure computing mode (Seccomp) is a Linux kernel
feature
• Supported from Docker 1.10
• Docker has default profile limiting many system calls (44
system calls out of 300+)
• Can also be customized for more blocks
• Can provide custom seccomp profile to containers via
Docker security options
![Page 17: AWS re:Invent 2016: Securing Container-Based Applications (CON402)](https://reader034.vdocuments.site/reader034/viewer/2022042907/587126161a28abe4448b6285/html5/thumbnails/17.jpg)
SELinux
• Implementation of Mandatory Access Control (MAC)
• Default SELinux policy for Docker designed to protect host
from containers and containers from each other• Uses tag to restrict usage
• Containers assigned default process type: svirt_lxc_net_t
and files accessible to container: svirt_sandbox_file_t.
• Policy enforces only read/execute container permissions to
/usr directory
• Assigns unique MCS category number to prevent cross
container access to files or resources
![Page 18: AWS re:Invent 2016: Securing Container-Based Applications (CON402)](https://reader034.vdocuments.site/reader034/viewer/2022042907/587126161a28abe4448b6285/html5/thumbnails/18.jpg)
Security options in ECS
![Page 19: AWS re:Invent 2016: Securing Container-Based Applications (CON402)](https://reader034.vdocuments.site/reader034/viewer/2022042907/587126161a28abe4448b6285/html5/thumbnails/19.jpg)
Security options in ECS
ECS_SELINUX_CAPABLE=true
![Page 20: AWS re:Invent 2016: Securing Container-Based Applications (CON402)](https://reader034.vdocuments.site/reader034/viewer/2022042907/587126161a28abe4448b6285/html5/thumbnails/20.jpg)
Container Lifecycle Security
![Page 21: AWS re:Invent 2016: Securing Container-Based Applications (CON402)](https://reader034.vdocuments.site/reader034/viewer/2022042907/587126161a28abe4448b6285/html5/thumbnails/21.jpg)
![Page 22: AWS re:Invent 2016: Securing Container-Based Applications (CON402)](https://reader034.vdocuments.site/reader034/viewer/2022042907/587126161a28abe4448b6285/html5/thumbnails/22.jpg)
Lifecycle risks
• Are my developers
introducing new
vulnerabilities
during
development?
• Are my container
images in my
repository
vulnerable?
• Do I have a runtime
defence if containers are
compromised?
• How do I ensure by
Docker platform is
secure over time?
![Page 23: AWS re:Invent 2016: Securing Container-Based Applications (CON402)](https://reader034.vdocuments.site/reader034/viewer/2022042907/587126161a28abe4448b6285/html5/thumbnails/23.jpg)
Mitigation approaches
• Do image
vulnerability analysis
in the CI/CD pipeline
• Only allow
developers to use
“approved” images
• Configuration
governance
• Regularly analyse
images in your
private repository
• Only allow compliant
images to run in
production
• Have runtime defence
in place
• Regular checks of
Docker platform
against industry
benchmarks
Own your repo!
![Page 24: AWS re:Invent 2016: Securing Container-Based Applications (CON402)](https://reader034.vdocuments.site/reader034/viewer/2022042907/587126161a28abe4448b6285/html5/thumbnails/24.jpg)
Secure Host via Docker Bench
• Recommendations provided by Center for Internet
Security’s Benchmark for Docker Engine
• Best practice: use Docker Bench to validate host
against recommendations
• www.dockerbench.com
![Page 25: AWS re:Invent 2016: Securing Container-Based Applications (CON402)](https://reader034.vdocuments.site/reader034/viewer/2022042907/587126161a28abe4448b6285/html5/thumbnails/25.jpg)
Run DockerBench via EC2 Run Cmd
Output report to
CloudWatch logs
Create CloudWatch alarm
on non-compliance
Automating Docker Bench
Invoke Lambda on schedule
![Page 26: AWS re:Invent 2016: Securing Container-Based Applications (CON402)](https://reader034.vdocuments.site/reader034/viewer/2022042907/587126161a28abe4448b6285/html5/thumbnails/26.jpg)
![Page 27: AWS re:Invent 2016: Securing Container-Based Applications (CON402)](https://reader034.vdocuments.site/reader034/viewer/2022042907/587126161a28abe4448b6285/html5/thumbnails/27.jpg)
Scan running containers for vulnerabilities in dev, test, staging, and
production.
• Public CVE vulnerabilities
• Some application specific vulnerabilities which may or may not have
CVE
• Scan the host file system, not just containers
• Can to tell whether the host is safe or not to load application
containers.
Visualize services, containers, and network behavior easily in staging
and production
Protects running containers against violations, threats, and
vulnerabilities
![Page 28: AWS re:Invent 2016: Securing Container-Based Applications (CON402)](https://reader034.vdocuments.site/reader034/viewer/2022042907/587126161a28abe4448b6285/html5/thumbnails/28.jpg)
Continuous integration
![Page 29: AWS re:Invent 2016: Securing Container-Based Applications (CON402)](https://reader034.vdocuments.site/reader034/viewer/2022042907/587126161a28abe4448b6285/html5/thumbnails/29.jpg)
CI/CD pipeline / Container lifecycle
CodeCommit
Source Build CodePipeline
Jenkins
1. Build Docker image
2. Check image for vulnerabilities
3. Push image to ECR
4. Update ECS service via CloudFormation
Run
Live scan
![Page 30: AWS re:Invent 2016: Securing Container-Based Applications (CON402)](https://reader034.vdocuments.site/reader034/viewer/2022042907/587126161a28abe4448b6285/html5/thumbnails/30.jpg)
Managing Secrets
![Page 31: AWS re:Invent 2016: Securing Container-Based Applications (CON402)](https://reader034.vdocuments.site/reader034/viewer/2022042907/587126161a28abe4448b6285/html5/thumbnails/31.jpg)
Bake into image
![Page 32: AWS re:Invent 2016: Securing Container-Based Applications (CON402)](https://reader034.vdocuments.site/reader034/viewer/2022042907/587126161a28abe4448b6285/html5/thumbnails/32.jpg)
Bake into image
![Page 33: AWS re:Invent 2016: Securing Container-Based Applications (CON402)](https://reader034.vdocuments.site/reader034/viewer/2022042907/587126161a28abe4448b6285/html5/thumbnails/33.jpg)
Environment variables
• Suggested by 12-factor apps
• Environment variables can be seen in
too many places
• linked containers, ECS API calls, docker
inspect
• Can’t be deleted
![Page 34: AWS re:Invent 2016: Securing Container-Based Applications (CON402)](https://reader034.vdocuments.site/reader034/viewer/2022042907/587126161a28abe4448b6285/html5/thumbnails/34.jpg)
S3-based secrets storage
• Secrets stored in S3 bucket
• Accessed via IAM roles for EC2
• Enforce encryption at rest and flight via IAM
policies and KMS
• Use VPC endpoint for S3 to lock down access
from certain VPCs
https://blogs.aws.amazon.com/security/post/Tx2B3QUWAA7KOU/
![Page 35: AWS re:Invent 2016: Securing Container-Based Applications (CON402)](https://reader034.vdocuments.site/reader034/viewer/2022042907/587126161a28abe4448b6285/html5/thumbnails/35.jpg)
Third-party secrets management solutions
• Vault from HashiCorp
• https://www.vaultproject.io/
• Keywhiz from Square
• https://square.github.io/keywhiz/
• Secrets accessed via leases
• Good for Dynamic Secrets generation
![Page 36: AWS re:Invent 2016: Securing Container-Based Applications (CON402)](https://reader034.vdocuments.site/reader034/viewer/2022042907/587126161a28abe4448b6285/html5/thumbnails/36.jpg)
Temporary credentials for RDS databases
Vault for RDS credentials
![Page 37: AWS re:Invent 2016: Securing Container-Based Applications (CON402)](https://reader034.vdocuments.site/reader034/viewer/2022042907/587126161a28abe4448b6285/html5/thumbnails/37.jpg)
Enforce Governance with ECS
![Page 38: AWS re:Invent 2016: Securing Container-Based Applications (CON402)](https://reader034.vdocuments.site/reader034/viewer/2022042907/587126161a28abe4448b6285/html5/thumbnails/38.jpg)
ECS primer
ECS cluster
…Task 2
Task N
Task 1
Task definition 1 Amazon ECR
Docker images
![Page 39: AWS re:Invent 2016: Securing Container-Based Applications (CON402)](https://reader034.vdocuments.site/reader034/viewer/2022042907/587126161a28abe4448b6285/html5/thumbnails/39.jpg)
Enforcing governance… is simple!
Task definition controls
• Which Docker images to use (app_latest)
• CPU & memory allocation
• Container links
• Host-to-container port mappings
• **NEW** IAM roles for tasks
![Page 40: AWS re:Invent 2016: Securing Container-Based Applications (CON402)](https://reader034.vdocuments.site/reader034/viewer/2022042907/587126161a28abe4448b6285/html5/thumbnails/40.jpg)
IAM roles for tasks
Benefits
• Simplify usage of AWS SDKs in containers
• Credential isolation between tasks
• Authorization per task
• Auditability in CloudTrail with taskArn
![Page 41: AWS re:Invent 2016: Securing Container-Based Applications (CON402)](https://reader034.vdocuments.site/reader034/viewer/2022042907/587126161a28abe4448b6285/html5/thumbnails/41.jpg)
IAM roles for tasks - example
Web UI
Container
Cluster Instance
Data Insights
Container
ECS Task 2ECS Task 1
IAM role
The old way…
Amazon
DynamoDBAmazon S3
Undesired
permissionUndesired
permission
![Page 42: AWS re:Invent 2016: Securing Container-Based Applications (CON402)](https://reader034.vdocuments.site/reader034/viewer/2022042907/587126161a28abe4448b6285/html5/thumbnails/42.jpg)
IAM roles for tasks - example
Web UI
Container
Cluster Instance
Data Insights
Container
ECS Task 2ECS Task 1
IAM Role 1
The new way!
IAM Role 2
Amazon
DynamoDBAmazon S3
![Page 43: AWS re:Invent 2016: Securing Container-Based Applications (CON402)](https://reader034.vdocuments.site/reader034/viewer/2022042907/587126161a28abe4448b6285/html5/thumbnails/43.jpg)
IAM roles for tasks – explained
1. ECS agent periodically queries ECS control plane
2. Control plane generates ID token
Auto-rotated
3. ECS agent:
Constructs HTTP URL for each container
Sets AWS_CREDENTIALS_ENDPOINT in HostConfig
4. AWS SDK extracts URL
![Page 44: AWS re:Invent 2016: Securing Container-Based Applications (CON402)](https://reader034.vdocuments.site/reader034/viewer/2022042907/587126161a28abe4448b6285/html5/thumbnails/44.jpg)
IAM roles for tasks - demo
![Page 45: AWS re:Invent 2016: Securing Container-Based Applications (CON402)](https://reader034.vdocuments.site/reader034/viewer/2022042907/587126161a28abe4448b6285/html5/thumbnails/45.jpg)
Automate Security Deployments
![Page 46: AWS re:Invent 2016: Securing Container-Based Applications (CON402)](https://reader034.vdocuments.site/reader034/viewer/2022042907/587126161a28abe4448b6285/html5/thumbnails/46.jpg)
Governance during continuous integration
Docker image
Developers Security Engineers Ops EngineersAmazon ECR
Task definition
![Page 47: AWS re:Invent 2016: Securing Container-Based Applications (CON402)](https://reader034.vdocuments.site/reader034/viewer/2022042907/587126161a28abe4448b6285/html5/thumbnails/47.jpg)
Governance during continuous deploymentDevelopers Ops Engineers Security Engineers
App Team
Developers Ops Engineers Security Engineers
InfoSec Team
![Page 48: AWS re:Invent 2016: Securing Container-Based Applications (CON402)](https://reader034.vdocuments.site/reader034/viewer/2022042907/587126161a28abe4448b6285/html5/thumbnails/48.jpg)
Docker Images
Web Application Host-Based Intrusion Detection
Securely merge…
…and deploy
Governance during continuous deployment
App Team InfoSec Team
Task Definition Docker Images Task Definition
![Page 49: AWS re:Invent 2016: Securing Container-Based Applications (CON402)](https://reader034.vdocuments.site/reader034/viewer/2022042907/587126161a28abe4448b6285/html5/thumbnails/49.jpg)
Deploying security functionality - example
Rate limiting proxy
Web Application
Container
Cluster instance
Reverse-Proxy
Container
Outbound
network
access
ECS Task
![Page 50: AWS re:Invent 2016: Securing Container-Based Applications (CON402)](https://reader034.vdocuments.site/reader034/viewer/2022042907/587126161a28abe4448b6285/html5/thumbnails/50.jpg)
Deploying security functionality - demo
![Page 51: AWS re:Invent 2016: Securing Container-Based Applications (CON402)](https://reader034.vdocuments.site/reader034/viewer/2022042907/587126161a28abe4448b6285/html5/thumbnails/51.jpg)
Why should I care?
• Remove accidental conflicts
• Make security processes continuous and automatic
• Encapsulate software artifacts and implement controls
one level up
• Control changes to this framework via IAM
![Page 52: AWS re:Invent 2016: Securing Container-Based Applications (CON402)](https://reader034.vdocuments.site/reader034/viewer/2022042907/587126161a28abe4448b6285/html5/thumbnails/52.jpg)
Not just containers!
Amazon Machine
Images (AMIs)
Docker Images
OS Packages
Amazon EC2
Container Service
AWS
CloudFormation
AWS CodeDeploy
![Page 53: AWS re:Invent 2016: Securing Container-Based Applications (CON402)](https://reader034.vdocuments.site/reader034/viewer/2022042907/587126161a28abe4448b6285/html5/thumbnails/53.jpg)
Amazon Machine
Images (AMIs)
Docker Images
OS Packages
Amazon EC2
Container Service
AWS
CloudFormation
AWS CodeDeploy
Software Artifacts Deployment Services
Not just containers!
![Page 54: AWS re:Invent 2016: Securing Container-Based Applications (CON402)](https://reader034.vdocuments.site/reader034/viewer/2022042907/587126161a28abe4448b6285/html5/thumbnails/54.jpg)
Not just containers!
CloudFormation
TemplateTask Definition
Application Specification
File (AppSpec.yml)
…applies to any
deployment
process.
![Page 55: AWS re:Invent 2016: Securing Container-Based Applications (CON402)](https://reader034.vdocuments.site/reader034/viewer/2022042907/587126161a28abe4448b6285/html5/thumbnails/55.jpg)
Thank you!
![Page 56: AWS re:Invent 2016: Securing Container-Based Applications (CON402)](https://reader034.vdocuments.site/reader034/viewer/2022042907/587126161a28abe4448b6285/html5/thumbnails/56.jpg)
Remember to complete
your evaluations!