journey to the modern app with containers, microservices and big data

© 2017 Mesosphere, Inc. All Rights Reserved.

Eryn Muetzel, Product Marketing for Big Data@errbear3

JOURNEY TO THE MODERN APP WITH CONTAINERS, MICROSERVICES AND BIG DATA

Datacenter Operating System (DC/OS)The simplest way to build, run, scale and manage modern enterprise apps


INDUSTRY TRANSITIONS

Customer analytics2000s

Electronic customer records1980s

Online customer engagement1990s

Real-time & predictive customer engagement2013+

WE LIVE IN THE ALWAYS CONNECTED ECONOMY.

3.7BMobile Broadband Users

80%New Data is Unstructured

13Connected Devices per Household

(N. Amer)

40%Annual Enterprise Data Growth

IN THE ALWAYS CONNECTED ECONOMY SOFTWARE IS YOUR BUSINESS.

Build more apps.Personal and meaningful customer experiences

Innovate faster.Capture value from new value streams and actionable data insights

Modernize ITAlways-available services that are secure and cost effective

1M trips daily across 311 cities, disrupting multiple industries.

Personal behavior-based insurance using IoT.

Real-time, personalized clinical care.

MODERN ENTERPRISE APPS IN ACTION.


ARCHITECTURAL SHIFT

TRADITIONAL ENTERPRISE APPLICATIONS

MODERN ENTERPRISE APPLICATIONS

Data

Code

Latency

Users

2+ Billion smartphone users

3+ Billion internet users

Enterprise data growth (CAGR): 40%+

Source: KPCB Internet Trends 2017, EMC Digital Universe 2014


ARCHITECTING FOR SCALE AT TWITTERBEFORE AFTER


Streaming

11

THE MODERN ENTERPRISE APPPLATFORM SERVICESMICROSERVICES

STATELESS CONTAINERS ANALYTICS

Batch Machine Learning

Search

DATABASES

Time Series

SQL / NoSQL


Web Apps and Microservices

Connected Devices

Machine Learning and Simulation

Industrial IoT Connected CarFraud DetectionSocial Media

DIGITAL TRANSFORMATION REQUIRES MODERN APPS

Modern Apps

Operational expertise

Continuous integration

and delivery

Big Data Analytics

Message Queue

Distributed Database Search Future

services...

Platform Services Container

Orchestration

How do you develop these skills in your organization?

MODERN ENTERPRISE APPS REQUIRE NEW CAPABILITIES. Distributed computing

expertise.

Engineering and operations of secure & highly reliable services at scale

DevOps process & culture.

Frequent & reliable releases supported by an automated CI/CD toolchain

Cloud-native technologies.

Containers, microservices, data servies (e.g., Spark, Kafka, Cassandra)


MODERN ENTERPRISE APPS

Containers Data Services

MicroservicesOpen source & Social coding1

2

3

4


OPEN SOURCE SOFTWARE PROVIDES LEVERAGE1


CONTAINERS SIMPLIFY PACKAGING AND DEPLOYMENT

Private Copy

Shared

App 3

Libraries

Guest Operating

System

Infrastructure

App 1

Libs

Host Operating System

Start / stop time One minute to few seconds Milliseconds

Workload density 10 - 100x1x

App 2

Libraries

Guest Operating

System

App 1

Libraries

Guest Operating

SystemApp 2

Libs

App 3

Libs

Virtual Machines Containers

Host Operating System

Hypervisor

Infrastructure

App 4

Libs

App 5

Libs

2


JUST HOW FAST ARE CONTAINERS?

Larry Rau from @Verizon with @flo Launching 50,000 containers in seconds with @mesosphere #DC/OS

https://twitter.com/Verizon



https://twitter.com/flo



https://twitter.com/mesosphere



https://twitter.com/hashtag/DCOS?src=hash





MICROSERVICES ENABLE FAST AND FREQUENT RELEASES

Traditional Architecture

Many functions in a single process

Siloed teams

REST

API

s

Microservices Architecture

Cross-functional teams organized around capabilities

Scales individually

Each element of functionality defined as “microservices”

Scales monolithically

3


DATA SERVICES PROVIDE CONNECTION AND PERSISTENCE

Data Ingestion

Response

Devices

ClientSensors

Event Bus

Reactive App

Database

Analytics

Use Case Examples

● Anomaly detection

● Personalization

● IoT Applications

● Predictive Analytics

● Machine Learning

4

© 2017 Mesosphere, Inc. All Rights Reserved. 20

BUT WHAT ABOUT THE INFRASTRUCTURE?

CaaSPaaS

Container AppContainer App

Stateful Service #1

V1

Big Data Analytics #1

V1

Stateful Service #2

V1

Stateful Service #2

V2

PaaS

Container App

Team A


V2

Big Data Analytics

#2V1

Team B


BUT WHAT ABOUT THE INFRASTRUCTURE?

CaaSPaaS

Container AppContainer App

Stateful Service #1

V1


V1

Stateful Service #2

V1

Stateful Service #2

V2

PaaS

Container App

Team A


V2

Big Data Analytics

#2V1

Team B

● Extremely low utilization - less than 10%, due to static partitioning

● Manual operations - Weeks to provision, wasted time managing individual machines

● Difficult to experiment with new tech - months to provision environments to evaluate new technologies

● High risk - Failure from manual operations and difficulties maintaining high availability


WHERE HAVE WE SEEN THIS PROBLEM BEFORE?

You are launching Google Chrome.Which core would you like to use?

Core 1 Core 2

Core 3 Core 4

You are launching a distributed app requiring 500 cores.Which of your 50,000 cores would you like to use?

Core 1 Core 2 Core 3 Core 4 Core 5

Core 6 Core 8 Core 9 Core 10Core 7










MODERN APPS REQUIRE OPERATIONS AT THE DC LEVEL

Traditional App Modern Enterprise App


YES, THIS TOO CAN BE SOLVED WITH ABSTRACTION

Scheduler

Executor

Task Launch

Task Launch

Task Status

Task Status

Resource Offer

ExecutorExecutorExecutor

Scheduler

Executor

Task Launch

ExecutorExecutor

Task Status

Scheduler

Executor

Task Launch

ExecutorExecutor

Task Status

Distributed System A Distributed System B Datacenter Operating System Approach

Distributed Systems A+B+C+...

Apache Mesos Two-level Scheduling


MODERN APPS WITH THE DC/OS MODEL Traditional Approach

Big Data Analytics

Stateful Service

DC/OS Model

Container App

Container App

CaaS PaaS

● Turnkey install of datacenter-wide services● Simplified operations & efficiency● Lowers barriers to experiment with new tech (e.g., Spark)

Container App

Datacenter-scale operating system


BRINGING MODERN APPS TO EVERYONE

● 100% Open source

● No limits

27THE DC/OS COMMUNITY

YOUR DATACENTER OR CLOUD IS NOW ONE GIANT COMPUTER


DC/OS SERVICE INSTALL


DC/OS SERVICE OPS

PLATFORM SERVICES INSTALLED ACROSS DATACENTER OR CLOUD IN A SINGLE COMMAND


TRY DC/OS IN UNDER 20 MINUTES

dcos.io

BUSINESS APPS

MESOSPHERE ENTERPRISE DC/OS

CONTAINERORCHESTRATION CI/CD BIG DATA

ANALYTICSMESSAGE

QUEUEDISTRIBUTED

DATABASE SEARCH

PLATFORM SERVICES

RESOURCEMANAGEMENT OPERATIONS SECURITY MULTI-TENANCY

DATACENTER-CLOUD OPERATING SYSTEM

APACHE MESOS

PRIVATE CLOUD AWS INFRASTRUCTURE (EC2) GOOGLE CLOUD MICROSOFT AZURE

MESOSPHERE ENTERPRISE

DC/OS

Container Orchestration

PaaS

Stateful Data Services

Analytics

Hybrid Cloud

Platform Services Ecosystem

Production Proven at Scale

DATACENTER

MESOSPHERE DC/OS CUSTOMER IMPACT

Accelerate Time-to-Market.Capture value from data insights and quickly roll out new services to stay competitive.

Ensure Resilience and Automation.Provide always available and secure business services on hybrid clouds.

Save Money.

Get the most out of infrastructure with high utilization and cost predictability.


“We’re running real time analytics and visualization of data from connected sensors

Before DC/OS we had no where near the scale we had today, from thousands of to millions of events per second.

Next, we’re doing predictive GIS”

Adam Mollenkopf, Real Time Geospatial Information Systems Capability Lead


“We are building a PaaS and services on an existing OpenStack cloud to enable developers to build microservices on Docker containers and use big data technologies...

DC/OS was a force multiplier for us to get these new services out the door fast...

We transitioned the full enterprise infrastructure at 2X the speed of previous efforts.”

Tim PletcherEngineering Director, Application Services


“We came across Mesosphere while planning for the coming IoT market..

If I build everything in a silo, I have no chance...

With DC/OS, I have one single contiguous cluster and I can bring my data and store it and I can run all my applications as well..

and I think that's a huge advantage”

Larry Rau, Director Architecture & Infrastructure, Verizon Labs

© 2017 Mesosphere, Inc. All Rights Reserved.McKinsey Quarterly 2015: “The perils of ignoring software development”Based on assessment of software development projects at 1,300 companies

Top companies bring new software to market

3x faster


Thank YouQuestions?

@errbear3 Learn more at mesosphere.com, including our whitepaper:

Modern Enterprise App Operations with DC/OSLessons from Running Containers, Microservices, and Stateful Big Data Services in Production

journey to the modern app with containers, microservices and big data

Technology