@ WeaveworksTom Wilkie, 14/03/2017

Weave Online Users Group

Who Am I?

• Director Software Engineering at Weaveworks

• Home brewer

Previously:

• SRE for Google Analytics

• Founder/CTO at Acunu

• Engineer at XenSource

@tom_wilkie

https://github.com/tomwilkie

Work on https://cloud.weave.works, our service for exploring, deploying and monitoring Micro Services…

1. gRPC for Prometheus

Retriever

scraping

your jobs

Your DC

Weave Cloud

Frontend, Authenticator

Distributor

Ingester

Distributor…

IngesterIngester

DynamoDB S3 http://goo.gl/qsJlrQ

gRPC?

1. gRPC for Prometheus

Prometheus AWS ELB Nginx Cortexweavecortex

2. gRPC intra service

Retriever

scraping

your jobs

Your DC

Weave Cloud

Frontend, Authenticator

Distributor

Ingester

Distributor…

IngesterIngester

DynamoDB S3

gRPC?

gRPC?

gRPC?

3. gRPC monitoringhttps://github.com/weaveworks/common/blob/master/middleware/

grpc_instrumentation.go

3. gRPC monitoring

4. HTTP over gRPC

https://www.weave.works/turtles-way-http-grpc/https://github.com/weaveworks/common/tree/master/httpgrpc

FrontendBackendweavecortex

Client

HTTP

gRPC

- Frontend parses HTTP request - Sends generic gRPC request - Backend turns it back in HTTP

4. HTTP over gRPC

5. gRPC Streaming

• Cortex rolling upgrades

• Flush all data to S3

• Took >10mins per instance

5. gRPC Streaming

New system uses gRPC streaming to send data to other ingester.

Now takes ~14s per instance.

leaving instance

weavecortex joining instance

weavecortex

gRPC stream

1. gRPC for Prometheus

2. gRPC from Weave Cloud

3. gRPC monitoring

4. HTTP over gRPC

5. gRPC streaming

We’re hiring!London BerlinSan Francisco