summit - amazon web services marketing/summit-berlin...summit © 2019, amazon web services, inc. or...

SUMMITBerlin

© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.SUMMIT

Making S3 more resilientusing Lambda@Edge

Júlia Biró, Yann HamonReliability TeamContentful

SessionID


Agenda

Introduction

Proof of Concept

Going live

Improving our Lambda@Edge software platform

Review


Our file delivery infrastructure


Our goal: multi-region active-active

https://www.youtube.com/watch?v=2e29I3dA8o4


Current state-of-the-art (?)

“Highly available multi region S3 website

Cloudfront distributions” - Derek Higgins (2017)

Could work but:

Failover solution

No guaranteed propagation time for configuration changes in Cloudfront

Manual reset

https://medium.com/cloudaholics/highly-available-multi-region-s3-website-cloudfront-distributions-29fe07195a3


A possible solution... that doesn’t work

Use an origin with DNS Round-Robin?

Does not work:

The request's Host needs to match the name of the S3 bucket


There must be a way...

"Dynamically Route Viewer Requests

to Any Origin Using Lambda@Edge"

Jake Wells, AWS Blog (Nov. 2017)


A/B Testing with Lambda@Edge


Load-balancing with Lambda@Edge

dns.resolveCname('files-origin.contentful.com', (function(err, result) {

if (result[0].includes('us-east-1')) {

bucketName = 'cf-files.s3.us-east-1.amazonaws.com';

region = 'us-east-1';

else {

bucketName = 'cf-files.s3.us-west-2.amazonaws.com';

region = 'us-west-2';

}

request.origin.s3.region = region;

request.origin.s3.domain = bucketName;

request.headers['host'] = [{key: 'host', value: bucketName}];

}


Technology validation

We build a small proof-of-concept. Only 20 lines of code!

Proof: different image with the same path in both regions.

Learnings:Our Javascript is not great - we don't do this every day!

We do a DNS resolution on every cache miss

us-east-1 us-west-2


DNS Caching

static async ResolveCname(fqdn) {

let now = Date.now();

let cachedEntry = this.cache[fqdn];

if (cachedEntry && now - cachedEntry.updatedAt < this.defaultTTL) {

return cachedEntry.answers;

}

[...]

DNS lookups add latency to our requests...

But we can cache the results [1]

https://aws.amazon.com/blogs/networking-and-content-delivery/lambdaedge-design-best-practices/


It works!

Thank you for your attention.


Leap of faith

Our team is not used to writing Javascript

Lambda@Edge is a new technology to the company

High cost of failure


High cost of failure

contractual SLAs

100s of requests/second

no graceful degradation

We need safety gear.


Why do we trust our current software platform?

version control

dash-boards


Production-readiness list

1. Solution-agnostic criteria2. Translated for the specific solution 3. Gap analysis

To add an image, select Click to insert image, and find the

image you want to use.


Production-readiness list is an ideal

No need to meet all requirements but..It needs to be a conscious and

documented decision

Our existing software platforms did

not meet all requirements

The goal is to reduce uncertainty and risk

RUNNIN' RHINO

Design by Allan Faustino

https://www.threadless.com/@alanis


Risk-aversion


Going live… gradually

1. Attach the Lambda - new feature turned off completely 2. Feature-flag / dark release3. Whitelist for some internal test customers4. Gradually roll out to all traffic 5. Monitor at all steps

Big red button: ability to quickly revertat all time

Expect to meet unknowns.


Our new file delivery infrastructure


Migrating a larger service to Lambda@Edge

Apply what we learned to our first Lambda project

Rewrite in Typescript

Backport all the tooling

Lambda@Edge becoming a first class citizen in our software platform


Working with lambda@Edge

Near-immediate scale-up

Marginal costs

Highly available

But..Challenging development environment (esp. integration testing)

Logs saved in each region

Our deployment workflow still manual


Working with lambda@Edge

TypeScript helped us write safer codewe also contributed to the Cloudfront specific package


Future improvements

Use Geo-routing to forward requests to the closest S3 bucket

Run the DNS resolution outside of the main event loop

Automate deployments


A few months later...

Similar implementations emerged:Using Amazon CloudFront with Multi-Region Amazon S3 Origins (Sept. 30th, Seldam)

Amazon S3 Region Failover — Part 2: CloudFront S3 origin failover (Oct. 30th, Frias)

Cloudfront origin failover was introduced:Amazon CloudFront announces support for Origin Failover (Nov. 20th, AWS)

But…Current solution has proven cheap, fast and stable

Has laid the groundworks for other multi-region projects

https://aws.amazon.com/blogs/apn/using-amazon-cloudfront-with-multi-region-amazon-s3-origins/

https://medium.com/@alejandro.millan.frias/amazon-s3-region-failover-part-2-cloudfront-origin-d66e6886b9b7

https://aws.amazon.com/about-aws/whats-new/2018/11/amazon-cloudfront-announces-support-for-origin-failover/


Takeaways

Lambda@Edge is a cheap, scalable & highly reliable platform to build stateless APIs

When the cost of failure is high, use production-readiness lists

Feature-flagging, canarying, gradual rollouts are easy to use with Lambda@Edge to reduce risk of large-scale changes

summit - amazon web services marketing/summit-berlin...summit © 2019, amazon web services, inc. or...

Documents