BOSH&

DiegoBenjamin Gandon

@BenjGa

2016-04-04

@BenjGa

Founder & CEO @ Gstack.io

Formerly: CTO @ Alvarum.com

Benjamin Gandon

BOSH

At:

We do:

And:

BOSHDeploys Distributed Systems

Argh! It’s again those Americans! Don’t worry, Herr General, we will find out soon!

What is BOSH

BOSH is a flexible and resilient technology that ………………………

complete ecosystems of distributed software.

1. Package2. Deploy3. Monitor4. Manage

(start/stop, scale, resurrect, update, …)

5. Upgrade

Instance in a BOSH Deployment

Container (or VM)

Filesystem + BOSH Agent

NginxHAproxy Pcre

HAproxy Nginx

Stemcell

Packages

Jobs

Instance Stemcell

+ Jobs

+ Packages

= Instance

Equation

/var/vcap/jobs/*

/var/vcap/packages/*

Availability Zone #3

AZ2AZ1

What we do with those “instances”

HAproxy

Stemcell

Pkg1 Pkg2

Nginx

Stemcell

Pkg3 Pkg2

HAproxy

Stemcell

Pkg1 Pkg2

Nginx

Stemcell

Pkg3 Pkg2

Nginx

Stemcell

Pkg3 Pkg2

● Garden containers (in a VitrualBox VM)● Docker (experimental)● AWS● OpenStack● CloudStack● Azure● vSphere

BOSH secret? → Use abstract Cloud Provider Interfaces (CPIs)

Deploy where ?

DiegoFlexible Container Engine

From

Try at home! Because…

● BOSH

● Cloud Foundry

● Diego

Open Source

Why “diego”?

● Since 2019, intuition for stateless “not-yet-containers” → the Droplets

● Their engine: the Droplet Execution Agent

→ DEA

● When rewritten in Go: DEA-GO

→ Diego!

Diego architecture

(logs)

(controller)

(http routes)

Source: https://github.com/cloudfoundry-incubator/diego-design-notes

Not just about containers

● Long Running Processes

→ LRPs

● Tasks

→ Servers

→ Cron jobs

Two ways to use Diego

● Multi-tenant Diego

→ as Cloud Foundry central container engine

● Single-tenant Diego

→ as Lattice, a single-tenant Diego cluster

Dude, go to http://lattice.cf

Lattice is definitely cool

● Containerized workloads○ Long running, or temporary tasks○ Dynamically scaled○ Dynamically balanced across cells (but no live re-balancing)

● Cluster scheduler● HTTP load balancing● Log aggregation● Health management

Some code!

Diego Auctions Scoring

// See: github.com/cloudfoundry-incubator/rep/…/resources.go#L15-L24type CellState struct {

RootFSProviders RootFSProvidersAvailableResources ResourcesTotalResources ResourcesLRPs []LRPTasks []TaskStartingContainerCount intZone stringEvacuating boolVolumeDrivers []string

}// See: github.com/cloudfoundry-incubator/rep/…/resources.go#L74-L79func (c CellState) ComputeScore(res *Resource, startingContainerWeight float64) float64 {

remainingResources := c.AvailableResources.Copy()remainingResources.Subtract(res)startingContainerScore := float64(c.StartingContainerCount) * startingContainerWeightreturn remainingResources.ComputeScore(&c.TotalResources) + startingContainerScore

}// See: github.com/cloudfoundry-incubator/rep/…/resources.go#L90-L93type Resources struct {

MemoryMB int32DiskMB int32Containers int

}// See: github.com/cloudfoundry-incubator/rep/…/resources.go#L110-L115func (r *Resources) ComputeScore(total *Resources) float64 {

fractionUsedMemory := 1.0 - float64(r.MemoryMB)/float64(total.MemoryMB)fractionUsedDisk := 1.0 - float64(r.DiskMB)/float64(total.DiskMB)fractionUsedContainers := 1.0 - float64(r.Containers)/float64(total.Containers)return (fractionUsedMemory + fractionUsedDisk + fractionUsedContainers) / 3.0

}

// See: github.com/cloudfoundry-incubator/auction/…/auctionrunner/scheduler.go#L206-L258

func (s *Scheduler) scheduleLRPAuction(lrpAuction *auctiontypes.LRPAuction) (*auctiontypes.LRPAuction, error) {var winnerCell *CellwinnerScore := 1e20

// Lists all zones, and count how many times the LRP is already in each of themzones := accumulateZonesByInstances(s.zones, lrpAuction.ProcessGuid)

// Retain only zones with compatible filesystem (e.g. Linux or Windows)filteredZones := filterZones(zones, lrpAuction)

if len(filteredZones) == 0 {return nil, auctiontypes.ErrorCellMismatch

}

sortedZones := sortZonesByInstances(filteredZones)

// …

}

Diego Auctions Evaluation (1/3)

// See: github.com/cloudfoundry-incubator/auction/…/auctionrunner/scheduler.go#L206-L258

func (s *Scheduler) scheduleLRPAuction(lrpAuction *auctiontypes.LRPAuction) (*auctiontypes.LRPAuction, error) {// …for zoneIndex, lrpByZone := range sortedZones {

for _, cell := range lrpByZone.zone {score, err := cell.ScoreForLRP(&lrpAuction.LRP, s.startingContainerWeight)if err != nil {

continue}

// Look for the Diego cell with minimal overall utilizationif score < winnerScore {

winnerScore = scorewinnerCell = cell

}}

// if (not last zone) && (this zone has the same # of instances as the next sorted zone)// acts as a tie breakerif zoneIndex+1 < len(sortedZones) &&

lrpByZone.instances == sortedZones[zoneIndex+1].instances {continue

}

if winnerCell != nil {break

}}// …

}

Diego Auctions Evaluation (2/3)

// See: github.com/cloudfoundry-incubator/auction/…/auctionrunner/scheduler.go#L206-L258

func (s *Scheduler) scheduleLRPAuction(lrpAuction *auctiontypes.LRPAuction) (*auctiontypes.LRPAuction, error) {// …

if winnerCell == nil {return nil, rep.ErrorInsufficientResources

}

err := winnerCell.ReserveLRP(&lrpAuction.LRP)if err != nil {

s.logger.Error("lrp-failed-to-reserve-cell", err, lager.Data{"cell-guid": winnerCell.Guid, "lrp-guid": lrpAuction.Identifier()})return nil, err

}

winningAuction := lrpAuction.Copy()

// Return a copy of the auction containing the winning cellwinningAuction.Winner = winnerCell.Guidreturn &winningAuction, nil

}

Diego Auctions Evaluation (3/3)

→ Simple as that!

Thank you!

www.gstack.io (not yet in Google)

www.bosh.iowww.lattice.cfwww.cloudfoundry.org

Questions?↘ ↙

→ @BenjGa ←↗ ↖