The zoo

* Scheduling and Orchestration * Mesos * Marathon * Conductor

* Persistence * AWS RDS * S3

* Service Discovery and Routing * Consul * haproxy * DNS

* Networking * AWS VPC * weave net

* Caching * AWS ElastiCache * AWS CloudFront * CloudFlare

* Logging * custom logspout daemon * logstash * ElasticSearch * Kibana

* Monitoring * Datadog

visibility

play

noun vis·i·bil·i·ty \ˌvi-zə-ˈbi-lə-tē\

: the ability to see or be seen

Visibility for

● Processes● System

kernel: [4838261.071917] haproxy[610]: segfault at 7fdb0426e000 ip 00007fdb0403305b sp 00007ffee5ff8988 error 4 in ld-musl-x86_64.so.1[7fdb03fe2000+87000]

1. Set /proc/sys/kernel/core_pattern

$ echo 'core.%e.%t.%p' | sudo tee /proc/sys/kernel/core_pattern

2. Se default ulimit

docker daemon [...] --default-ulimit core=-1

(gdb) backtrace#0 in memcpy () from /lib/ld-musl-x86_64.so.1...#5 in http_compression_buffer_end (...) at src/compression.c:249#6 in http_response_forward_body (...) at src/proto_http.c:7173...#9 in run_poll_loop () at src/haproxy.c:1573#10 in main (...) at src/haproxy.c:1933(gdb) frame 6#6 in http_response_forward_body (...) at src/proto_http.c:7173

(gdb) print txn->uri$3 = 0x55e95aff7a00 "POST /webhook/_REDACTED_FOUR_GREAT_SECURITAH__/ HTTP/1.1"

(gdb) frame 5#5 in http_compression_buffer_end (...) at src/compression.c:249(gdb) print *s->comp_algo$6 = {cfg_name = 0x55e955ced5b0 "gzip", ...}

logging aggregation

● Aggregated by metadata tags○ service name○ version○ environment

● Stream level aggregation○ multi-line○ json

http://www.youtube.com/watch?v=fymb0yBbGnk

System visibility

Breaking the monolith breaks visibility

● Breaking the monolith breaks our visibility tools● Targeted tools, automated ad-hoc queries

○ Metrics systems - Datadog, Prometheus○ Core dumps○ Profile graphs

● System-wide visibility○ Tracing / Application Performance Monitoring○ Network flow visualizations○ Big Data ™