Andrew Morgan – MongoDB Product Marketing
POWERING MICROSERVICES
WITH DOCKER,
KUBERNETES, KAFKA, &
MONGODB
andrewmorgan
AGENDA
Microservice
s
What, Why,
How?
Containers
Docker, Kafka
Orchestratio
n
Kubernetes,
Mesos, …
MongoDB
Why, How?
When to use Use cases
Who, Why?
1 2 3 4 5 6
MICROSERVICES
WHY USE MICROSERVICES? (TL;DRWEBSCALE)
Speed Change Maintain Scale Empower
Build MVP quickly Rapid iterations Simple components Product Team == Component
React to market Isolated impact Team Committees
WHY USE MICROSERVICES? (TL;DRWEBSCALE)
Speed Change Maintain Scale Empower
Build MVP quickly Rapid iterations Simple components Product Team == Component
React to market Isolated impact Team Committees
WHY USE MICROSERVICES? (TL;DRWEBSCALE)
Speed Change Maintain Scale Empower
Build MVP quickly Rapid iterations Simple components Product Team == Component
React to market Isolated impact Team Committees
WHY USE MICROSERVICES? (TL;DRWEBSCALE)
Speed Change Maintain Scale Empower
Build MVP quickly Rapid iterations Simple components Product Team == Component
React to market Isolated impact Team Committees
WHY USE MICROSERVICES? (TL;DRWEBSCALE)
Speed Change Maintain Scale Empower
Build MVP quickly Rapid iterations Simple components Product Team == Component
React to market Isolated impact Team Committees
WHY USE MICROSERVICES? (TL;DRWEBSCALE)
Speed Change Maintain Scale Empower
Build MVP quickly Rapid iterations Simple components Product Team == Component
React to market Isolated impact Team Committees
Monolithic
Cross-Team
Tight Coupling
Small Change
Huge Impact
System Retests
Microservices
Decoupled
Independent Dev
Isolated Impact
MICROSERVICES EXAMPLE
IngestGoogle+
Ingest
Snapcha
t Ingest
Feed
merge
Faceboo
k Ingest
MICROSERVICES EXAMPLE
Ingest
Snapcha
t Ingest
Feed
merge
Faceboo
k Ingest
MICROSERVICES EXAMPLE
Ingest
Snapcha
t Ingest
Feed
merge
Faceboo
k Ingest
MICROSERVICES EXAMPLE
Ingest
Snapcha
t Ingest
Feed
merge
Faceboo
k Ingest
Whatsap
p Ingest
MICROSERVICES EXAMPLE
Ingest
Snapcha
t Ingest
Feed
merge
Faceboo
k Ingest
Whatsap
p Ingest
Snapcha
t IngestSnapcha
t Ingest
DEVELOPMENT TEAMS
CONTAINERS
CONTAINERS – POWERING MICROSERVICES
Real world shipping containers
• Road, rail & sea
• Contents untouched
• Ubiquitous & standardized
• Simple
• Contents protected
• Constraints
CONTAINERS – POWERING MICROSERVICES
Software containers
• 1 image -> Many containers‒ Laptop, DC, cloud
‒ Dev, QA, production, support
• Simple, efficient
• Isolation
• Constraints
VMS VS. CONTAINERS
VM VMVM
Bare Metal
Host Operating System
Hypervisor
Guest OS
Libraries
Apps
Service
Guest OS
Libraries
Apps
Service
Guest OS
Libraries
Apps
Service
Container ContainerContainer
Bare Metal
Host Operating System
Docker Engine
Libraries
Libraries
Apps
Libraries
Apps
Service ServiceService
VMS VS. CONTAINERS
VM VMVM
Bare Metal
Host Operating System
Hypervisor
Guest OS
Libraries
Apps
Service
Guest OS
Libraries
Apps
Service
Guest OS
Libraries
Apps
Service
Container ContainerContainer
Bare Metal
Host Operating System
Docker Engine
Libraries
Libraries
Apps
Libraries
Apps
Service ServiceService
VMS VS. CONTAINERS
VM VMVM
Bare Metal
Host Operating System
Hypervisor
Guest OS
Libraries
Apps
Service
Guest OS
Libraries
Apps
Service
Guest OS
Libraries
Apps
Service
Container ContainerContainer
Bare Metal
Host Operating System
Docker Engine
Libraries
Libraries
Apps
Libraries
Apps
Service ServiceService
DOCKER
• Simple to use
• 100K+ images on Docker Hub
• Build images from images
• Platforms‒ Linux, OS X, Windows
‒ Laptop, VM, Cloud,…
‒ Cloud services
RUN MONGODB
docker run -d mongo
TITLE ONLY
TITLE ONLY
MICROSERVICE ARCHITECTURES BUILT ON CONTAINERS
Many small, focused containers ->
sophisticated services
• Well defined APIs
• Independent languages &
libraries
• Modular: easy maintenance +
reuse
• Fault tolerant
• Scalable
CONNECTING THE MICROSERVICES –KAFKA
Producer
9
8
7
123...
Topic A
Consumer
CONNECTING THE MICROSERVICES –KAFKA
Producer
9
8
7
123...
Topic A
Consumer
Producer Consumer
CONNECTING THE MICROSERVICES –KAFKA
Producer
9
8
7
123...
Partition 0
Topic A
Consumer
Producer Consumer
4
3
5
123...
Partition 1
CONNECTING THE MICROSERVICES –KAFKA
Producer
LEADER
Topic A / Partition 0
Broker 1
FOLLOWER
Topic A / Partition 1
FOLLOWER
Topic A / Partition 0
Broker 2
LEADER
Topic A / Partition 1
CONNECTING THE MICROSERVICES –KAFKA
Producer
Producer
Producer
9
8
7
123...
Partition 0
4
3
5
123...
Partition 1
7
3
2
123...
Partition N
Topic A
Topic B
7
6
5
123...
Partition 0
New Old
Consumer
Consumer
ORCHESTRATION
ORCHESTRATION
Automated deployment, connecting, and maintenance of multiple containers
• Provision hosts
• Containers‒ Instantiate
‒ Reschedule
‒ Link
‒ Scale Out/In
• Expose services
KUBERNETES
Created by Google, feature-rich and widely adopted
• Deployment and ‘replication’
• On-line scale out/in
• Rolling upgrades
• High Availability
• Persistence
• Ports
• Load balancing
• Google Compute Engine
APACHE MESOS
10,000s of physical servers; used by
Twitter, Airbnb & Apple
• Code (“frameworks) vs.
declarative
• Less feature rich than Kubernetes
• Kubernetes as a Mesos
framework
• Foundation for distributed systems‒ Apache Aurora, Chronos, Marathon
CHOOSING AN ORCHESTRATION FRAMEWORK
• What you have:‒ Skills?
‒ DevOps frameworks?
‒ Number of hosts?
‒ Bare metal, VMs, or cloud?
• Lifecycle
• Features‒ Automated High Availability?
‒ Grouping and load balancing?
‒ As a service?
MONGODB
WHY MONGODB IS A GOOD FIT FOR MICROSERVICES
Monitoring
&
Automation
Flexible Data
Model
Redundancy Scalability Simplicity
WHY MONGODB IS A GOOD FIT FOR MICROSERVICES
Monitoring
&
Automation
Flexible Data
Model
Redundancy Scalability Simplicity
WHY MONGODB IS A GOOD FIT FOR MICROSERVICES
Monitoring
&
Automation
Flexible Data
Model
Redundancy Scalability Simplicity
WHY MONGODB IS A GOOD FIT FOR MICROSERVICES
Monitoring
&
Automation
Flexible Data
Model
Redundancy Scalability Simplicity
WHY MONGODB IS A GOOD FIT FOR MICROSERVICES
Monitoring
&
Automation
Flexible Data
Model
Redundancy Scalability Simplicity
WHY MONGODB IS A GOOD FIT FOR MICROSERVICES
Monitoring
&
Automation
Flexible Data
Model
Redundancy Scalability Simplicity
HOW TO USE MONGODB WITH CONTAINERS
Ingest
Snapch
at Ingest
Feed
merge
Faceboo
k Ingest
Whatsap
p Ingest
Snapch
at IngestSnapch
at Ingest
MongoDB Atlas
HOW TO USE MONGODB WITH CONTAINERS
Ingest
Snapch
at Ingest
Feed
merge
Faceboo
k Ingest
Whatsap
p Ingest
Snapch
at IngestSnapch
at Ingest
Kubernetes
Ops Mgr
agent
Ops Mgr
agent
Ops Mgr
agent
HOW TO USE MONGODB WITH CONTAINERS
Ingest
Snapch
at Ingest
Feed
merge
Faceboo
k Ingest
Whatsap
p Ingest
Snapch
at IngestSnapch
at Ingest
Kubernetes
mongod
mongod
mongod
ORCHESTRATING MONGODB USING KUBERNETES
Distributed, stateful application
• Persistent volumes
• External IP addresses for internal
comms
• Init MongoDB replica set
• Monitor
• Backup
Volumename:
mongo-persistent-storage1
pdName: mongodb-disk1
mongodb
-disk1
Containername: mongo-node1
image: mongo
command: mongod –replSet my_replica_set
containerPort: 27017
volumeMounts:
name: mongo-persistent-storage1
mountPath: /data/db
DockerHubRegistry
Podlabels: [name: mongo-node1; instance: rod]
ReplicationControllername: mongo-rc1
labels: [name: mongo-rc]
replicas: 1
selector: [name: mongo-node1]
Service:LoadBalancername: mongo-svc-a
labels: [name: mongo-svc-a]
ports: [port: 27017, targetPort: 27017]]
selector: [name: mongo-node, instance: rod]ExternalIP
Address
104.1.1.1
App
104.1.1.1:27017
ReplicationControlername: mongo-rc-europe
labels: [name: mongo-europe]
replicas: 1
selector: [name: mongo-node]
Volumename:
mongo-persistent-storage
pdName: mongodb-disk-europe
mongodb-
disk-europe
Containername: mongo-node
image: mongo
command: mongod –replSet my_replica_set
containerPort: 27017
volumeMounts:
name: mongo-persistent-storage
mountPath: /data/db
Podlabels: [name: mongo-node]
Service:LoadBalancername: mongo-svc-europe
labels: [name: mongo-svc-europe]
ports: [port: 27017, targetPort: 27017]]
selector: [name: mongo-node]
ReplicationControlername: mongo-rc-asia
labels: [name: mongo-asia]
replicas: 1
selector: [name: mongo-node]
Volumename:
mongo-persistent-storage
pdName: mongodb-disk-asia
mongodb-
disk-asia
Containername: mongo-node
image: mongo
command: mongod –replSet my_replica_set
containerPort: 27017
volumeMounts:
name: mongo-persistent-storage
mountPath: /data/db
Podlabels: [name: mongo-node]
Service:LoadBalancername: mongo-svc-asia
labels: [name: mongo-svc-asia]
ports: [port: 27017, targetPort: 27017]]
selector: [name: mongo-node]
ReplicationControlername: mongo-rc-us
labels: [name: mongo-us]
replicas: 1
selector: [name: mongo-node]
Volumename:
mongo-persistent-storage
pdName: mongodb-disk-us
mongodb-
disk-us
Containername: mongo-node
image: mongo
command: mongod –replSet my_replica_set
containerPort: 27017
volumeMounts:
name: mongo-persistent-storage
mountPath: /data/db
Podlabels: [name: mongo-node]
Service:LoadBalancername: mongo-svc-us
labels: [name: mongo-svc-us]
ports: [port: 27017, targetPort: 27017]]
selector: [name: mongo-node]
STATEFULSETS
Beta in Kubernetes 1.5/6
• Stable, predictable, unique network identifiers.‒ IP addresses may change
• Stable, persistent storage
• Ordered, graceful deployment and scaling (0 N-1)
• Ordered, graceful deletion and termination (N-1 0)
PUTTING IT INTO PRACTICE
WHEN TO USE
MICROSERVICES
WHEN TO USE MICROSERVICES
USE CASES
MONGODB & MICROSERVICES IN THE WILD
REFERENCES
• Enabling Microservices – Containers & Orchestration Explained
https://www.mongodb.com/collateral/microservices-containers-and-orchestration-explained
• Microservices: The Evolution of Building Modern Applications
https://www.mongodb.com/collateral/microservices-the-evolution-of-building-modern-
applications
• Data Streaming with Apache Kafka & MongoDB
https://www.mongodb.com/collateral/data-streaming-with-apache-kafka-and-mongodb
PUTTING IT INTO PRACTICE