(spot302) under the covers of aws: core distributed systems primitives that power our platform | aws...

November 13, 2014 | Las Vegas, NV

Al Vermeulen and Swami Sivasubramanian

Trend #1: The race between Computing power

and expectations

Computing systems keep getting more capableBut… expectations are going up even faster

Trend #2: Every application is a distributed app

The number of computers is going up fast

Many applications are distributed

Distributed systems is not a niche field anymore

Hardware and software trends

Specialized (expensive)

hardware

Built in redundancy

Simple Software

Commodity hardware

Smarter software

With commodity hardware and scale - server failures are inevitable!

But by using smarter software,

we can build more robust systems

Trend #3: Commodity Hardware and Smarter Software

OnlineOnline

Online

OnlineOnline

OnlineOnline

Online

OnlineOnline

OnlineOnline

Online

OnlineOnline

Cloud – Elasticity is the new normalThis results in fleets being dynamic

Our World

Challenges

Addressing Distributed Computing challenges

primitives

Core Distributed Systems Primitives

Group

MembershipDiscovery

Metadata

Store

Failure

DetectionWorkflows

Group Membership

Group Membership

–

Amazon RDS Multi-AZ

Amazon ElastiCache Group – List of caches in a Memcache

group

An example…

Replica B

Replica C

Writes from

client AReplica A

Replica D

New member in the

group

Should I continue to serve reads?

Should I start a new quorum?

Replica E Replica F

Reads and

Writes from

client B

Classic Split Brain Issue in Replicated systems leading to lost writes!

Group Membership Fundamentals

Adding a new member to the group

Removing a member from the group

Discovering when the group membership changes

Discovering roles within the group

Discovery

Discovery

Discovery – Configuration File

Discovery - DNS

Discovery – DNS (cons)

Discovery – Gossip Protocol

Discovery – Gossip Protocol (Cons)Discovery – Gossip Protocol (Con)

Discovery – Metadata store/consensus

Amazon DynamoDB

Metadata Store

Metadata Store

Metadata Store - what are good characteristics?

Simplicity

Availability

Scalability

Amazon DynamoDB – top

choice for metadata storage in

Amazon

Metadata Store – Lessons Learned

Failure Detection

Failure Detection - Challenges

Failure Detection - Techniques

Failure detection: Lessons Learned

Workflows

Workflow – What is it?

To execute a series of actions asynchronously

What is a workflow?

What is not a workflow?

synchronous

asynchronous

Workflow – A simple script

Workflow – Recommended Approach

Activity

1Activity

2Activity

3

Activity

4

Workflow – Lessons Learned

Idempotent

metadata

Amazon Simple Workflow

Service

What is the underlying problem?

Group

MembershipDiscovery

Metadata

Store

Failure

DetectionWorkflows

Consensus

Paxos and consensus

single point

of failure

Paxos at

the bottom

broken

Consensus – Lessons Learned

consensus..

Paxos at Amazon

Group

Membership

Discovery

Metadata

Store

Failure

Detection

Workflows

Lock

ManagementAmazon

Kinesis

Amazon

DynamoDB

Streams

???

Summary

http://bit.ly/awsevals