re-platforming all the things

Re-Platforming All the Things

© 2015 Mesosphere, Inc.

Who in the world are you?

Thomas Rampelberg

Product Manager

thomas@mesosphere.io

@grampelberg

2

© 2015 Mesosphere, Inc.

Who do you work for?

3

© 2015 Mesosphere, Inc.

Interested in trying it out now?

• Mesosphere DCOS• Apache Mesos + Marathon• https://mesosphere.com/amazon/

4

© 2015 Mesosphere, Inc.

• The datacenter of yesterday• The datacenter of today• The datacenter of tomorrow

Agenda

5

© 2015 Mesosphere, Inc. 6

Yesterday

© 2015 Mesosphere, Inc.

History of the World

7

The Mainframe

Linux + x86

Virtual Machines / Cloud

Containers

© 2015 Mesosphere, Inc.

Mainframes

• Applications - 1 service• Hardware - 1 box• Redundancy - Systems Architecture• Reboot - Never• Services - 100s/”host”

8

© 2015 Mesosphere, Inc.

• Applications - 2(ish) services• Hardware - 10s of boxes• Redundancy - Multiple boxes• Reboot - Occasionally• Services - 10s/”host”

Linux + x86 - Commoditization

9

© 2015 Mesosphere, Inc.

Virtual Machines / Cloud

• Applications - 3+ service• Hardware - 100s of “hosts”• Redundancy - Hardware + Software• Reboot - Occasionally• Services - A couple/”host”

10

© 2015 Mesosphere, Inc.

• Applications - Microservices• Hardware - 1000s of containers• Redundancy - Multiple boxes• Reboot - Often• Services - 1 per container

Containers

11

© 2015 Mesosphere, Inc. 12

Today

© 2015 Mesosphere, Inc.

Pets

• They’re Individuals• You have to clean up• They fail eventually

13

© 2015 Mesosphere, Inc.

• Treated as a herd• Cow pies happen• They fail eventually

Cattle

14

© 2015 Mesosphere, Inc.

How are these different?

• 144 Cores• 1.15 TB Memory

15

• 144 Cores• 1.15 TB Memory

© 2015 Mesosphere, Inc.

“Sixty-four cores or 128 cores in a single computer looks a lot like 64 or 128

machines in a datcenter” - Benjamin Hindman

16

© 2015 Mesosphere, Inc.

One Big Computer

• Manage Resources Globally• Manage Failures Globally• Scale Across Hosts

17

© 2015 Mesosphere, Inc.

• Scalability• Multi-tennancy• Efficiency• Reliability & availability• On-demand resources

What do you get?

18

© 2015 Mesosphere, Inc.

Who’s doing it?

Closed Source

• Google Borg• Google Omega

19

© 2015 Mesosphere, Inc.

Who’s doing it?

20

Open Source

• Apache Mesos• Apache Hadoop (YARN)• Kubernetes

© 2015 Mesosphere, Inc.

Apache Mesos

21

© 2015 Mesosphere, Inc.

Overview

• Borg is awesome, let’s do something better

• Came out of UC Berkeley's AMP Lab• First paper in 2009• Running at Twitter and Apple

22

© 2015 Mesosphere, Inc.

• Separate resources from placement• Focus on running batch jobs and

services• All about the scheduler

What’s Different?

23

© 2015 Mesosphere, Inc.

Apache Hadoop (YARN)

24

© 2015 Mesosphere, Inc.

Overview

• Hadoop needed a better scheduler• YARN became the foundation for

Hadoop 2.0• Paper came out in 2013• Running at Yahoo and LinkedIn

25

© 2015 Mesosphere, Inc.

• Centralize the resource and placement decisions

• Optimize for somewhat long batch jobs (eg. MR)

• Bake the storage layer into the solution

What’s Different?

26

© 2015 Mesosphere, Inc.

Kubernetes

27

© 2015 Mesosphere, Inc.

Overview

• Google took everything they learned from Borg and Omega

• Kubernetes was open source in 2014, just went 1.0 last month

• Running in production ….. ?

28

© 2015 Mesosphere, Inc.

• Containers first, only containers• Pods are a first class citizen• Labels, labels everywhere

What’s Different?

29

© 2015 Mesosphere, Inc. 30

Tomorrow

© 2015 Mesosphere, Inc.

Stateful Applications

• Persistent Volumes• Durable Pods• Distributed File Systems

31

© 2015 Mesosphere, Inc.

• Networking virtualization• Flannel• Weave• Calico

• Dynamic load balancing• IPv6

Service Discovery

32

© 2015 Mesosphere, Inc.

Security

• Not everyone is a small startup• Compliance is important• Long tail of process• Containers are still early• Schedule policy, automatically

33

© 2015 Mesosphere, Inc.

• Sizing is hard• Not all processes need guarantees• Let the scheduler sort it out

Oversubscription

34

© 2015 Mesosphere, Inc. 35

Next Steps

© 2015 Mesosphere, Inc.

Getting Started

• Apache Mesos - http://mesosphere.com/product

• Apache Hadoop (YARN) - http://bit.ly/1K13E4S

• Kubernetes - http://kubernetes.io/gettingstarted/

36

© 2015 Mesosphere, Inc.

• http://chat.mesosphere.com/• http://mesos.apache.org/community/• https://wiki.apache.

org/hadoop/HowToContribute• http://kubernetes.io/community/

Get Involved!

37

© 2015 Mesosphere, Inc.

Get Awesome

38