powering microservices with mongodb, docker, kubernetes & kafka – mongodb europe 2016
TRANSCRIPT
Powering Microservices with Docker, Kubernetes & KafkaAndrew MorganMongoDB Product Marketing@andrewmorgan
#MDBW16
Agenda
What, Why, How?
Microservices01Kubernetes, Mesos,…
Orchestration03DockerKafka
Containers02
Why, How?
MongoDB04Who, Why?
Use Cases06When to Use05
Microservices
#MDBW16
Why Use Microservices? (tl;dr WebScale)
Team = Components
No committees
EmpowerBuild MVP quickly
SpeedSimple
componentsIsolated impact
MaintainRapid iterationsReact to market
ChangeProductTeam
Scale
#MDBW16
1990s & Earlier
MonolithicCross-TeamTight Coupling
Small Chang
e
Huge Impac
t
System
Retests
#MDBW16
2000sSOA
CoordinationLooser
CouplingSystem Retests
#MDBW16
2010sMicroservices
DecoupledIndependent
DevIsolated Impact
#MDBW16
Microservices Example
IngestGoogl
e+ Ingest
Snapchat
Ingest
Feed merge
Ingest
#MDBW16
Microservices Example
Ingest
Snapchat
Ingest
Feed merge
Ingest
#MDBW16
Microservices Example
Ingest
Snapchat
Ingest
Feed merge
Ingest
#MDBW16
Microservices Example
Ingest
Snapchat
Ingest
Feed merge
Ingest
Ingest
#MDBW16
Microservices Example
Ingest
Snapchat
Ingest
Feed merge
Ingest
Ingest
Snapchat
IngestSnapchat
Ingest
#MDBW16
Alternate Microservices Example
• Much larger Microservices1. User Account2. Product Catalog3. Inventory4. Orders
• Best practice is each microservice has its own database
“ Conway’s Law1967
Any organization that designs a system will inevitably produce a design whose structure is a copy of the organization's communication structure.”
#MDBW16
Development Teams
Containers
#MDBW16
Containers – Powering Microservices
Real world shipping containers• Road, rail & sea• Contents untouched• Ubiquitous & standardized• Simple• Contents protected• Constraints
#MDBW16
Containers – Powering Microservices
Software containers• 1 image -> Many
containers• Laptop, DC, cloud• Dev, QA, production, support
• Simple, efficient• Isolation• Constraints
#MDBW16
VMs vs. ContainersVM VMVM
Bare MetalHost Operating System
HypervisorGuest OSLibraries
AppsService
Guest OSLibraries
AppsService
Guest OSLibraries
AppsService
Container ContainerContainer
Bare MetalHost Operating System
Docker EngineLibraries
LibrariesApps
LibrariesApps
Service ServiceService
#MDBW16
VMs vs. ContainersVM VMVM
Bare MetalHost Operating System
HypervisorGuest OSLibraries
AppsService
Guest OSLibraries
AppsService
Guest OSLibraries
AppsService
Container ContainerContainer
Bare MetalHost Operating System
Docker EngineLibraries
LibrariesApps
LibrariesApps
Service ServiceService
#MDBW16
VMs vs. ContainersVM VMVM
Bare MetalHost Operating System
HypervisorGuest OSLibraries
AppsService
Guest OSLibraries
AppsService
Guest OSLibraries
AppsService
Container ContainerContainer
Bare MetalHost Operating System
Docker EngineLibraries
LibrariesApps
LibrariesApps
Service ServiceService
#MDBW16
Docker
• Simple to use• 100K+ images on Docker
Hub• Build images from images• Platforms• Linux, OS X, Windows• Laptop, VM, Cloud,…• Cloud services
#MDBW16
Run MongoDB
docker run -d mongo
#MDBW16
Security
Opportunities
• Isolation• Constrain resources• Narrow roles• Keep images/containers
current• root: so restrict access
#MDBW16
Title only
#MDBW16
Title only
#MDBW16
Microservice Architectures Built on Containers
Many small, focused containers -> sophisticated services
• Well defined APIs• Independent languages &
libraries• Modular: easy maintenance
+ reuse• Fault tolerant• Scalable
#MDBW16
Connecting the Microservices – Apache Kafka
Producer987
123...
Topic A
Consumer
#MDBW16
Connecting the Microservices – Apache Kafka
Producer
987
123...
Topic A
Consumer
Producer Consumer
#MDBW16
Connecting the Microservices – Apache Kafka
Producer
987
123...Partition 0
Topic A
Consumer
Producer Consumer
435
123...Partition 1
#MDBW16
Connecting the Microservices – Apache Kafka
Producer
LEADER
Topic A / Partition 0
Broker 1
FOLLOWER
Topic A / Partition 1
FOLLOWER
Topic A / Partition 0
Broker 2
LEADER
Topic A / Partition 1
#MDBW16
Connecting the Microservices – Apache Kafka
Producer
Producer
Producer
987
123...
Partition 0
435
123...
Partition 1
732
123...
Partition N
Topic A
Topic B
765
123...
Partition 0
New Old
Consumer
Consumer
Orchestration
#MDBW16
Orchestration
Automated deployment, connecting, and maintenance of multiple containers
• Provision hosts• Containers
• Instantiate• Reschedule• Link• Scale Out/In
• Expose services
#MDBW16
Kubernetes
Created by Google, feature-rich and widely adopted
• Deployment and ‘replication’• On-line scale out/in• Rolling upgrades• High Availability• Persistence• Ports• Load balancing• Google Compute Engine
#MDBW16
Apache Mesos
10,000s of physical servers; used by Twitter, Airbnb & Apple
• Code (“frameworks) vs. declarative• Less feature rich than Kubernetes• Kubernetes as a Mesos
framework• Foundation for distributed
systems• Apache Aurora, Chronos, Marathon
#MDBW16
Choosing an Orchestration Framework
• What you have:• Skills?• DevOps frameworks?• Number of hosts?• Bare metal, VMs, or cloud?
• Lifecycle• Features• Automated High Availability?• Grouping and load balancing?• As a service??
MongoDB
#MDBW16
Why MongoDB is a Good Fit For Microservices
ScalabilityMonitoring&
AutomationRedundancyFlexible
DataModel
Simplicity
#MDBW16
Orchestrating MongoDB Using Kubernetes
Distributed, stateful application
• Persistent volumes• External IP addresses for internal comms• Init MongoDB replica set• Monitor• Backup
When to use Microservices
#MDBW16
When to use Microservices
Use Cases
#MDBW16
MongoDB & Microservices in the Wild
#MDBW16
References
• Enabling Microservices – Containers & Orchestration Explainedhttps://www.mongodb.com/collateral/microservices-containers-and-orchestration-explained
• Microservices: The Evolution of Building Modern Applicationshttps://www.mongodb.com/collateral/microservices-the-evolution-of-building-modern-applications
• Data Streaming with Apache Kafka & MongoDBhttps://www.mongodb.com/collateral/data-streaming-with-apache-kafka-and-mongodb
