powering microservices with docker, kubernetes, kafka, and mongodb

#MDBW17

Andrew Morgan – MongoDB Product Marketing

POWERING MICROSERVICES WITH DOCKER, KUBERNETES, KAFKA, & MONGODB

andrewmorgan

#MDBW17

AGENDA

Microservice

s

What, Why,

How?

Containers

Docker, Kafka

Orchestratio

n

Kubernetes,

Mesos, …

MongoDB

Why, How?

When to use Use cases

Who, Why?

1 2 3 4 5 6

MICROSERVICES

#MDBW17

WHY USE MICROSERVICES? (TL;DRWEBSCALE)

Speed Change Maintain Scale Empower

Build MVP quickly Rapid iterations Simple components Product Team == Component

React to market Isolated impact Team Committees

#MDBW17

Monolithic

Cross-Team

Tight Coupling

Small Change

Huge Impact

System Retests

#MDBW17

Microservices

Decoupled

Independent Dev

Isolated Impact

#MDBW17

MICROSERVICES EXAMPLE

Twitter

IngestGoogle+

Ingest

Snapcha

t Ingest

Feed

merge

Faceboo

k Ingest

#MDBW17


Twitter

Ingest

Snapcha

t Ingest

Feed

merge

Faceboo

k Ingest

#MDBW17


Twitter

Ingest

Snapcha

t Ingest

Feed

merge

Faceboo

k Ingest

Whatsap

p Ingest

#MDBW17


Twitter

Ingest

Snapcha

t Ingest

Feed

merge

Faceboo

k Ingest

Whatsap

p Ingest

Snapcha

t IngestSnapcha

t Ingest

#MDBW17

DEVELOPMENT TEAMS

CONTAINERS

#MDBW17

CONTAINERS – POWERING MICROSERVICES

Real world shipping containers

• Road, rail & sea

• Contents untouched

• Ubiquitous & standardized

• Simple

• Contents protected

• Constraints

#MDBW17

CONTAINERS – POWERING MICROSERVICES

Software containers

• 1 image -> Many containers‒ Laptop, DC, cloud

‒ Dev, QA, production, support

• Simple, efficient

• Isolation

• Constraints

#MDBW17

VMS VS. CONTAINERS

VM VMVM

Bare Metal

Host Operating System

Hypervisor

Guest OS

Libraries

Apps

Service

Guest OS

Libraries

Apps

Service

Guest OS

Libraries

Apps

Service

Container ContainerContainer

Bare Metal

Host Operating System

Docker Engine

Libraries

Libraries

Apps

Libraries

Apps

Service ServiceService

#MDBW17

DOCKER

• Simple to use

• 100K+ images on Docker Hub

• Build images from images

• Platforms‒ Linux, OS X, Windows

‒ Laptop, VM, Cloud,…

‒ Cloud services

#MDBW17

RUN MONGODB

docker run -d mongo

#MDBW17

TITLE ONLY

#MDBW17

MICROSERVICE ARCHITECTURES BUILT ON CONTAINERS

Many small, focused containers ->

sophisticated services

• Well defined APIs

• Independent languages &

libraries

• Modular: easy maintenance +

reuse

• Fault tolerant

• Scalable

#MDBW17

CONNECTING THE MICROSERVICES –KAFKA

Producer

9

8

7

123...

Topic A

Consumer

#MDBW17


Producer

9

8

7

123...

Topic A

Consumer

Producer Consumer

#MDBW17


Producer

9

8

7

123...

Partition 0

Topic A

Consumer

Producer Consumer

4

3

5

123...

Partition 1

#MDBW17


Producer

LEADER

Topic A / Partition 0

Broker 1

FOLLOWER


FOLLOWER


Broker 2

LEADER


#MDBW17


Producer

Producer

Producer

9

8

7

123...

Partition 0

4

3

5

123...

Partition 1

7

3

2

123...

Partition N

Topic A

Topic B

7

6

5

123...

Partition 0

New Old

Consumer

Consumer

ORCHESTRATION

#MDBW17

ORCHESTRATION

Automated deployment, connecting, and maintenance of multiple containers

• Provision hosts

• Containers‒ Instantiate

‒ Reschedule

‒ Link

‒ Scale Out/In

• Expose services

#MDBW17

KUBERNETES

Created by Google, feature-rich and widely adopted

• Deployment and ‘replication’

• On-line scale out/in

• Rolling upgrades

• High Availability

• Persistence

• Ports

• Load balancing

• Google Compute Engine

#MDBW17

APACHE MESOS

10,000s of physical servers; used by

Twitter, Airbnb & Apple

• Code (“frameworks) vs.

declarative

• Less feature rich than Kubernetes

• Kubernetes as a Mesos

framework

• Foundation for distributed systems‒ Apache Aurora, Chronos, Marathon

#MDBW17

CHOOSING AN ORCHESTRATION FRAMEWORK

• What you have:‒ Skills?

‒ DevOps frameworks?

‒ Number of hosts?

‒ Bare metal, VMs, or cloud?

• Lifecycle

• Features‒ Automated High Availability?

‒ Grouping and load balancing?

‒ As a service?

MONGODB

#MDBW17

WHY MONGODB IS A GOOD FIT FOR MICROSERVICES

Monitoring

&

Automation

Flexible Data

Model

Redundancy Scalability Simplicity

#MDBW17

ORCHESTRATING MONGODB USING KUBERNETES

Distributed, stateful application

• Persistent volumes

• External IP addresses for internal

comms

• Init MongoDB replica set

• Monitor

• Backup

Volumename:

mongo-persistent-storage1

pdName: mongodb-disk1

mongodb

-disk1

Containername: mongo-node1

image: mongo

command: mongod –replSet my_replica_set

containerPort: 27017

volumeMounts:

name: mongo-persistent-storage1

mountPath: /data/db

DockerHubRegistry

Podlabels: [name: mongo-node1; instance: rod]

ReplicationControllername: mongo-rc1

labels: [name: mongo-rc]

replicas: 1

selector: [name: mongo-node1]

Service:LoadBalancername: mongo-svc-a

labels: [name: mongo-svc-a]

ports: [port: 27017, targetPort: 27017]]

selector: [name: mongo-node, instance: rod]ExternalIP

Address

104.1.1.1

App

104.1.1.1:27017

ReplicationControlername: mongo-rc-europe

labels: [name: mongo-europe]

replicas: 1

selector: [name: mongo-node]

Volumename:

mongo-persistent-storage

pdName: mongodb-disk-europe

mongodb-

disk-europe

Containername: mongo-node

image: mongo



volumeMounts:

name: mongo-persistent-storage

mountPath: /data/db

Podlabels: [name: mongo-node]

Service:LoadBalancername: mongo-svc-europe

labels: [name: mongo-svc-europe]



ReplicationControlername: mongo-rc-asia

labels: [name: mongo-asia]

replicas: 1


Volumename:


pdName: mongodb-disk-asia

mongodb-

disk-asia


image: mongo



volumeMounts:


mountPath: /data/db


Service:LoadBalancername: mongo-svc-asia

labels: [name: mongo-svc-asia]



ReplicationControlername: mongo-rc-us

labels: [name: mongo-us]

replicas: 1


Volumename:


pdName: mongodb-disk-us

mongodb-

disk-us


image: mongo



volumeMounts:


mountPath: /data/db


Service:LoadBalancername: mongo-svc-us

labels: [name: mongo-svc-us]



#MDBW17

STATEFULSETS

Beta in Kubernetes 1.5/6

• Stable, predictable, unique network identifiers.‒ IP addresses may change

• Stable, persistent storage

• Ordered, graceful deployment and scaling (0 N-1)

• Ordered, graceful deletion and termination (N-1 0)

WHEN TO USE

MICROSERVICES

#MDBW17

WHEN TO USE MICROSERVICES

USE CASES

#MDBW17

MONGODB & MICROSERVICES IN THE WILD

#MDBW17

REFERENCES

• Enabling Microservices – Containers & Orchestration Explained

https://www.mongodb.com/collateral/microservices-containers-and-orchestration-explained

• Microservices: The Evolution of Building Modern Applications

https://www.mongodb.com/collateral/microservices-the-evolution-of-building-modern-

applications

• Data Streaming with Apache Kafka & MongoDB

https://www.mongodb.com/collateral/data-streaming-with-apache-kafka-and-mongodb

https://www.mongodb.com/collateral/microservices-containers-and-orchestration-explained

https://www.mongodb.com/collateral/microservices-the-evolution-of-building-modern-applications

https://www.mongodb.com/collateral/data-streaming-with-apache-kafka-and-mongodb

#MDBW17

RUNNING MONGODB AS A MICROSERVICE

Conway’s Law1967

Any organization that designs a system will inevitably produce a design whose structure is a copy of the organization's communication structure.”

#MDBW17

ALTERNATE MICROSERVICES EXAMPLE

• Much larger Microservices1. User Account

2. Product Catalog

3. Inventory

4. Orders

• Best practice is each

microservice has its own

database

#MDBW17

SECURITY

Opportunities

• Isolation

• Constrain resources

• Narrow roles

• Keep images/containers current

• root: so restrict access

powering microservices with docker, kubernetes, kafka, and mongodb

Technology