Download - SF HashiCorp User Group at GitHub
HASHICORP
Deploying and Discovering at Scale with Consul & Nomad
HASHICORP
Jon Benson @jm_benson
HASHICORP
Agenda
• Nomad Overview
• Nomad Architecture
HASHICORP
Agenda
• Consul Overview
• Consul Architecture
• Prepared Queries
HASHICORP
Agenda
• Demo
• Questions!
HASHICORP
Advantages of a Scheduler
Higher Resource Utilization
Decouple Work from Resources
Better Quality of Service
HASHICORP
Advantages of a Scheduler
Bin Packing
Over-Subscription
Job Queueing
Higher Resource Utilization
Decouple Work from Resources
Better Quality of Service
HASHICORP
Advantages of a Scheduler
Abstraction
API Contracts
Standardization
Higher Resource Utilization
Decouple Work from Resources
Better Quality of Service
HASHICORP
Advantages of a Scheduler
Priorities
Resource Isolation
Pre-emption
Higher Resource Utilization
Decouple Work from Resources
Better Quality of Service
HASHICORP
NomadHASHICORP
Cluster Scheduler
Easily Deploy Applications
Job Specification
HASHICORP
example.nomad
# Define our simple redis jobjob "redis" {
# Run only in us-east-1 datacenters = ["us-east-1"]
# Define the single redis task using Docker task "redis" { driver = "docker"
config { image = "redis:latest" }
resources { cpu = 500 # Mhz memory = 256 # MB network { mbits = 10 dynamic_ports = ["redis"] } } }}
HASHICORP
Job Specification
Declares what to run
HASHICORP
Job Specification
Nomad determines where and manages how to run
HASHICORP
Job Specification
Abstract work from resources
NomadHASHICORP
Higher Resource Utilization
Decouple Work from Resources
Better Quality of Service
NomadHASHICORP
Multi-Datacenter
Multi-Region
Flexible Workloads
Job Priorities
Bin Packing
Large Scale
Operationally Simple
HASHICORP
Thousands of regions
Tens of thousands of clients per region
Thousands of jobs per region
HASHICORP
Built on Experience
gossip consensus
HASHICORP
• Cluster Management
• Gossip Based (P2P)
• Membership
• Failure Detection
• Event System
HASHICORP
• Gossip Protocol
• Large Scale
• Production Hardened
• Operationally Simple
HASHICORP
• Service configuration and discovery
• Monitoring at scale
• High-availability
HASHICORP
• Service configuration and discovery
• Monitoring at scale
• High-availability
HASHICORP
• Service configuration and discovery
• Monitoring at scale
• High-availability
HASHICORP
• Service configuration and discovery
• Monitoring at scale
• High-availability
HASHICORP
HASHICORP
• Multi-Datacenter
• Raft Consensus
• Large Scale
• Production Hardened
• Coordination (Locking)
• Central Servers + Distributed Clients
• Network Tomography
• Prepared Queries
HASHICORP
Prepared Queries
• Multiple instances of a given service exist in multiple
datacenters
• Clients can talk to any of them, and always prefer the
instances with lowest latency
• Policies can change, desire to not have the clients know
the details of how to locate a healthy service
HASHICORP
Prepared Queries
• New query namespace, similar to services
• Register queries to answer for parts of this namespace
• Clients use APIs, or “.query.consul” DNS lookups to run
queries
HASHICORP
gossip consensus
Mature Libraries
Design Patterns
No Scheduling Logic
HASHICORP
Built on Research
gossip consensus
NomadHASHICORP
Inspired by Google Omega
Optimistic Concurrency
Internal State and Coordination
Service and Batch workloads
Pluggable Architecture
HASHICORP
Single Region Architecture
SERVER SERVER SERVER
CLIENT CLIENT CLIENTDC1 DC2 DC3
FOLLOWER LEADER FOLLOWER
REPLICATIONFORWARDING
REPLICATIONFORWARDING
RPC RPC RPC
HASHICORP
Multi Region Architecture
SERVER SERVER SERVERFOLLOWER LEADER FOLLOWER
REPLICATIONFORWARDING
REPLICATION
REGION B GOSSIP
REPLICATION REPLICATIONFORWARDING
REGION FORWARDING
REGION A
SERVERFOLLOWER
SERVER SERVERLEADER FOLLOWER
NomadHASHICORP
Region is Isolation Domain
1-N Datacenters Per Region
Flexibility to do 1:1 (Consul)
Scheduling Boundary
HASHICORP
Data Model
HASHICORP
Evaluations ~= State Change Event
HASHICORP
Create / Update / Delete JobNode Up / Node Down
Allocation Failed
HASHICORP
External Event
Evalua?on Crea?on
Evalua?on Queuing
Evalua?on Processing
Op?mis?c Coordina?on
State Updates
HASHICORP
Server Architecture
Omega Class Scheduler
Pluggable Logic
Internal Coordination and State
Multi-Region / Multi-Datacenter
HASHICORP
Client Architecture
Broad OS Support
Host Fingerprinting
Pluggable Drivers
HASHICORP
Fingerprinting
Operating System
Hardware
Applications
Environment
Type Examples
Kernel, OS, Versions
CPU, Memory, Disk
Java, Docker, Consul
AWS, GCE
HASHICORP
Fingerprinting
Constrain Placement and Bin Pack
HASHICORP
Fingerprinting
“Task Requires Linux, Docker, and PCI-Compliant Hardware” expressed as Constraints
HASHICORP
Fingerprinting
“Task needs 512MB RAM and 1 Core” expressed as Resource Ask
HASHICORP
Drivers
Execute Tasks Provide Resource Isolation
HASHICORP
Containerized
Virtualized
Standalone
Docker
Qemu / KVM
Java Jar
Static Binaries
Rocket
HASHICORP
Containerized
Virtualized
Standalone
Docker
Rocket
Windows Server Containers
Qemu / KVM
Hyper-V
Xen
Java Jar
Static Binaries
C#
NomadHASHICORP
Workload Flexibility:
Schedulers
Fingerprints
Drivers
Job Specification
NomadHASHICORP
Operational Simplicity:
Single Binary
No Dependencies
Highly Available
NomadHASHICORP
Cluster Scheduler
Easily Deploy Applications
Job Specification
NomadHASHICORP
Higher Resource Utilization
Decouple Work from Resources
Better Quality of Service
NomadHASHICORP
• Million Container Challenge
• hashicorp.com/c1m.html
• github.com/hashicorp/c1m
• Nomad 0.4
• Volume support across drivers
• Advanced networking
HASHICORP
Thanks!We’ll do a quick demo then answer questions…
Jon Benson @jm_benson