local testing of containerized distributed systems
TRANSCRIPT
![Page 1: Local testing of Containerized Distributed Systems](https://reader030.vdocuments.site/reader030/viewer/2022032711/587c1b4c1a28abb5068b543f/html5/thumbnails/1.jpg)
Local testing of Containerized Distributed Systems
Strange Group Meetup, Justin Carter (@bodymindarts)
Berlin, Oct. 27. 2016 Stark & Wayne
![Page 2: Local testing of Containerized Distributed Systems](https://reader030.vdocuments.site/reader030/viewer/2022032711/587c1b4c1a28abb5068b543f/html5/thumbnails/2.jpg)
Delmo
Delmo is a tool to test systems running within multiple docker containers
https://github.com/bodymindarts/delmo
![Page 3: Local testing of Containerized Distributed Systems](https://reader030.vdocuments.site/reader030/viewer/2022032711/587c1b4c1a28abb5068b543f/html5/thumbnails/3.jpg)
30 staff, 7 countries
PaaS experts
Cloud Foundry experts
![Page 4: Local testing of Containerized Distributed Systems](https://reader030.vdocuments.site/reader030/viewer/2022032711/587c1b4c1a28abb5068b543f/html5/thumbnails/4.jpg)
Plattforms
Dingo PostgreSQL
Delmo
![Page 5: Local testing of Containerized Distributed Systems](https://reader030.vdocuments.site/reader030/viewer/2022032711/587c1b4c1a28abb5068b543f/html5/thumbnails/5.jpg)
http://www.slideshare.net/KarlIsenberg/cloud-surfing-kubernetes-on-mesos
![Page 6: Local testing of Containerized Distributed Systems](https://reader030.vdocuments.site/reader030/viewer/2022032711/587c1b4c1a28abb5068b543f/html5/thumbnails/6.jpg)
The sophistication of our platform bounds the granularity of decomposition in our
systems
(also means more complexity to setup locally)
![Page 7: Local testing of Containerized Distributed Systems](https://reader030.vdocuments.site/reader030/viewer/2022032711/587c1b4c1a28abb5068b543f/html5/thumbnails/7.jpg)
How much of the cloud stack do we want to run locally (if any)?
Platform on dev workstation? Dev env in the cloud?
Apps decoupled from platform?
![Page 8: Local testing of Containerized Distributed Systems](https://reader030.vdocuments.site/reader030/viewer/2022032711/587c1b4c1a28abb5068b543f/html5/thumbnails/8.jpg)
Contract Driven Tests / Mocks add a lot of code to maintain.
Dealing with deps?
Do we need them?
![Page 9: Local testing of Containerized Distributed Systems](https://reader030.vdocuments.site/reader030/viewer/2022032711/587c1b4c1a28abb5068b543f/html5/thumbnails/9.jpg)
Cloud dev workflowHow much of the stack do we want to run?
How do we want to run our deps?
How do we automate this and integrate into an efficient workflow?
![Page 10: Local testing of Containerized Distributed Systems](https://reader030.vdocuments.site/reader030/viewer/2022032711/587c1b4c1a28abb5068b543f/html5/thumbnails/10.jpg)
![Page 11: Local testing of Containerized Distributed Systems](https://reader030.vdocuments.site/reader030/viewer/2022032711/587c1b4c1a28abb5068b543f/html5/thumbnails/11.jpg)
When you can deploy 1000s of apps, you’d better be able to deploy 1000s
of dbs!
Platform consideration
![Page 12: Local testing of Containerized Distributed Systems](https://reader030.vdocuments.site/reader030/viewer/2022032711/587c1b4c1a28abb5068b543f/html5/thumbnails/12.jpg)
PostgreSQL in Docker
Clustered with streaming replication
Continuously backed up
Automatic failover
Dingo PostgreSQL
https://github.com/dingotiles/dingo-postgresql-release
![Page 13: Local testing of Containerized Distributed Systems](https://reader030.vdocuments.site/reader030/viewer/2022032711/587c1b4c1a28abb5068b543f/html5/thumbnails/13.jpg)
Dingo PostgreSQL Architecture
H A P r o x y
Deployed by BOSH
Client 1
Client 2
![Page 14: Local testing of Containerized Distributed Systems](https://reader030.vdocuments.site/reader030/viewer/2022032711/587c1b4c1a28abb5068b543f/html5/thumbnails/14.jpg)
Critical FeaturesRe-configuration of PostgreSQL
Restart PostgreSQL
Streaming backups to S3
![Page 15: Local testing of Containerized Distributed Systems](https://reader030.vdocuments.site/reader030/viewer/2022032711/587c1b4c1a28abb5068b543f/html5/thumbnails/15.jpg)
Changing Docker Image
![Page 16: Local testing of Containerized Distributed Systems](https://reader030.vdocuments.site/reader030/viewer/2022032711/587c1b4c1a28abb5068b543f/html5/thumbnails/16.jpg)
![Page 17: Local testing of Containerized Distributed Systems](https://reader030.vdocuments.site/reader030/viewer/2022032711/587c1b4c1a28abb5068b543f/html5/thumbnails/17.jpg)
Test independently of BOSH
Test smaller system boundary
Build and test containers locally
Goals
![Page 18: Local testing of Containerized Distributed Systems](https://reader030.vdocuments.site/reader030/viewer/2022032711/587c1b4c1a28abb5068b543f/html5/thumbnails/18.jpg)
Postgres Leader
Postgres Replica
Example TestWill failover work?
Dingo PG
HAProxy
Query
Leader
etcd
![Page 19: Local testing of Containerized Distributed Systems](https://reader030.vdocuments.site/reader030/viewer/2022032711/587c1b4c1a28abb5068b543f/html5/thumbnails/19.jpg)
AssertDid data get replicated?
Did reconfiguration / promotion work?
Are connections being accepted?
![Page 20: Local testing of Containerized Distributed Systems](https://reader030.vdocuments.site/reader030/viewer/2022032711/587c1b4c1a28abb5068b543f/html5/thumbnails/20.jpg)
Given: System definition
When: Some failure
Then: Assert expected Behaviour
Test Case
![Page 21: Local testing of Containerized Distributed Systems](https://reader030.vdocuments.site/reader030/viewer/2022032711/587c1b4c1a28abb5068b543f/html5/thumbnails/21.jpg)
Delmo
Declare system with docker-compose
Declare test cases
Run tests (locally or remotely)
![Page 22: Local testing of Containerized Distributed Systems](https://reader030.vdocuments.site/reader030/viewer/2022032711/587c1b4c1a28abb5068b543f/html5/thumbnails/22.jpg)
patroni1
Docker Compose
patroni2
etcd
System Tasks
Tasks are packaged bash scripts that execute commands against the running system
![Page 23: Local testing of Containerized Distributed Systems](https://reader030.vdocuments.site/reader030/viewer/2022032711/587c1b4c1a28abb5068b543f/html5/thumbnails/23.jpg)
System
Tasks
Cache
![Page 24: Local testing of Containerized Distributed Systems](https://reader030.vdocuments.site/reader030/viewer/2022032711/587c1b4c1a28abb5068b543f/html5/thumbnails/24.jpg)
Delmo Config
![Page 25: Local testing of Containerized Distributed Systems](https://reader030.vdocuments.site/reader030/viewer/2022032711/587c1b4c1a28abb5068b543f/html5/thumbnails/25.jpg)
Meta
Tests
Tasks
![Page 27: Local testing of Containerized Distributed Systems](https://reader030.vdocuments.site/reader030/viewer/2022032711/587c1b4c1a28abb5068b543f/html5/thumbnails/27.jpg)
Initialise
Start SystemRun TasksCleanup
Summary
![Page 28: Local testing of Containerized Distributed Systems](https://reader030.vdocuments.site/reader030/viewer/2022032711/587c1b4c1a28abb5068b543f/html5/thumbnails/28.jpg)
MoreRun on a remote docker-machine
Run tasks in parallel
delmo-concourse image (for executing in CI/CD)
![Page 29: Local testing of Containerized Distributed Systems](https://reader030.vdocuments.site/reader030/viewer/2022032711/587c1b4c1a28abb5068b543f/html5/thumbnails/29.jpg)
Next Steps
Better error output / validation
Network failure / partitioning
Support for other backends (bosh)?
![Page 30: Local testing of Containerized Distributed Systems](https://reader030.vdocuments.site/reader030/viewer/2022032711/587c1b4c1a28abb5068b543f/html5/thumbnails/30.jpg)
Questions?
Justin Carter @bodymindarts github.com/bodymindarts github.com/bodymindarts/delmo