Download - distributed tracing in 5 minutes
![Page 1: distributed tracing in 5 minutes](https://reader034.vdocuments.site/reader034/viewer/2022042713/547e05c05906b5a1718b461f/html5/thumbnails/1.jpg)
distributed tracing
![Page 2: distributed tracing in 5 minutes](https://reader034.vdocuments.site/reader034/viewer/2022042713/547e05c05906b5a1718b461f/html5/thumbnails/2.jpg)
twitter zipkingoogle dapper
x-tracetracelytics... more!
![Page 3: distributed tracing in 5 minutes](https://reader034.vdocuments.site/reader034/viewer/2022042713/547e05c05906b5a1718b461f/html5/thumbnails/3.jpg)
motivation
![Page 4: distributed tracing in 5 minutes](https://reader034.vdocuments.site/reader034/viewer/2022042713/547e05c05906b5a1718b461f/html5/thumbnails/4.jpg)
what is slow?
![Page 5: distributed tracing in 5 minutes](https://reader034.vdocuments.site/reader034/viewer/2022042713/547e05c05906b5a1718b461f/html5/thumbnails/5.jpg)
what is slow?
![Page 6: distributed tracing in 5 minutes](https://reader034.vdocuments.site/reader034/viewer/2022042713/547e05c05906b5a1718b461f/html5/thumbnails/6.jpg)
causal flow of control
![Page 7: distributed tracing in 5 minutes](https://reader034.vdocuments.site/reader034/viewer/2022042713/547e05c05906b5a1718b461f/html5/thumbnails/7.jpg)
causal flow of control
![Page 8: distributed tracing in 5 minutes](https://reader034.vdocuments.site/reader034/viewer/2022042713/547e05c05906b5a1718b461f/html5/thumbnails/8.jpg)
how to
![Page 9: distributed tracing in 5 minutes](https://reader034.vdocuments.site/reader034/viewer/2022042713/547e05c05906b5a1718b461f/html5/thumbnails/9.jpg)
possible approaches
![Page 10: distributed tracing in 5 minutes](https://reader034.vdocuments.site/reader034/viewer/2022042713/547e05c05906b5a1718b461f/html5/thumbnails/10.jpg)
possible approaches•Unique identifier
![Page 11: distributed tracing in 5 minutes](https://reader034.vdocuments.site/reader034/viewer/2022042713/547e05c05906b5a1718b461f/html5/thumbnails/11.jpg)
possible approaches•Unique identifier•propagate throughout
![Page 12: distributed tracing in 5 minutes](https://reader034.vdocuments.site/reader034/viewer/2022042713/547e05c05906b5a1718b461f/html5/thumbnails/12.jpg)
possible approaches•Unique identifier•propagate throughout•write instrumentation for various
transports
![Page 13: distributed tracing in 5 minutes](https://reader034.vdocuments.site/reader034/viewer/2022042713/547e05c05906b5a1718b461f/html5/thumbnails/13.jpg)
possible approaches•Unique identifier•propagate throughout•write instrumentation for various
transports
![Page 14: distributed tracing in 5 minutes](https://reader034.vdocuments.site/reader034/viewer/2022042713/547e05c05906b5a1718b461f/html5/thumbnails/14.jpg)
possible approaches•Unique identifier•propagate throughout•write instrumentation for various
transports
•Observe and correlate
![Page 15: distributed tracing in 5 minutes](https://reader034.vdocuments.site/reader034/viewer/2022042713/547e05c05906b5a1718b461f/html5/thumbnails/15.jpg)
possible approaches•Unique identifier•propagate throughout•write instrumentation for various
transports
•Observe and correlate•always on the outside - black box
![Page 16: distributed tracing in 5 minutes](https://reader034.vdocuments.site/reader034/viewer/2022042713/547e05c05906b5a1718b461f/html5/thumbnails/16.jpg)
possible approaches•Unique identifier•propagate throughout•write instrumentation for various
transports
•Observe and correlate•always on the outside - black box•difficult to get threaded + evented
processes right
![Page 17: distributed tracing in 5 minutes](https://reader034.vdocuments.site/reader034/viewer/2022042713/547e05c05906b5a1718b461f/html5/thumbnails/17.jpg)
1BD57B58AE7E315BBEAB6795F0BDC198296357
![Page 18: distributed tracing in 5 minutes](https://reader034.vdocuments.site/reader034/viewer/2022042713/547e05c05906b5a1718b461f/html5/thumbnails/18.jpg)
![Page 19: distributed tracing in 5 minutes](https://reader034.vdocuments.site/reader034/viewer/2022042713/547e05c05906b5a1718b461f/html5/thumbnails/19.jpg)
nginx
pythoncache
db internet
the java
t = start
![Page 20: distributed tracing in 5 minutes](https://reader034.vdocuments.site/reader034/viewer/2022042713/547e05c05906b5a1718b461f/html5/thumbnails/20.jpg)
nginx
pythoncache
db internet
the java
t = start
![Page 21: distributed tracing in 5 minutes](https://reader034.vdocuments.site/reader034/viewer/2022042713/547e05c05906b5a1718b461f/html5/thumbnails/21.jpg)
nginx
pythoncache
db internet
the java
t = start
![Page 22: distributed tracing in 5 minutes](https://reader034.vdocuments.site/reader034/viewer/2022042713/547e05c05906b5a1718b461f/html5/thumbnails/22.jpg)
nginx
pythoncache
db internet
the java
t = start
![Page 23: distributed tracing in 5 minutes](https://reader034.vdocuments.site/reader034/viewer/2022042713/547e05c05906b5a1718b461f/html5/thumbnails/23.jpg)
nginx
pythoncache
db internet
the java
t = start
![Page 24: distributed tracing in 5 minutes](https://reader034.vdocuments.site/reader034/viewer/2022042713/547e05c05906b5a1718b461f/html5/thumbnails/24.jpg)
nginx
pythoncache
db internet
the java
t = start
![Page 25: distributed tracing in 5 minutes](https://reader034.vdocuments.site/reader034/viewer/2022042713/547e05c05906b5a1718b461f/html5/thumbnails/25.jpg)
nginx
pythoncache
db internet
the java
t = start
![Page 26: distributed tracing in 5 minutes](https://reader034.vdocuments.site/reader034/viewer/2022042713/547e05c05906b5a1718b461f/html5/thumbnails/26.jpg)
nginx
pythoncache
db internet
the java
t = start
![Page 27: distributed tracing in 5 minutes](https://reader034.vdocuments.site/reader034/viewer/2022042713/547e05c05906b5a1718b461f/html5/thumbnails/27.jpg)
nginx
pythoncache
db internet
the java
t = start
![Page 28: distributed tracing in 5 minutes](https://reader034.vdocuments.site/reader034/viewer/2022042713/547e05c05906b5a1718b461f/html5/thumbnails/28.jpg)
nginx
pythoncache
db internet
the java
t = start
![Page 29: distributed tracing in 5 minutes](https://reader034.vdocuments.site/reader034/viewer/2022042713/547e05c05906b5a1718b461f/html5/thumbnails/29.jpg)
nginx
pythoncache
db internet
the java
t = start
![Page 30: distributed tracing in 5 minutes](https://reader034.vdocuments.site/reader034/viewer/2022042713/547e05c05906b5a1718b461f/html5/thumbnails/30.jpg)
nginx
pythoncache
db internet
the java
t = start
![Page 31: distributed tracing in 5 minutes](https://reader034.vdocuments.site/reader034/viewer/2022042713/547e05c05906b5a1718b461f/html5/thumbnails/31.jpg)
nginx
pythoncache
db internet
the java
t = start
![Page 32: distributed tracing in 5 minutes](https://reader034.vdocuments.site/reader034/viewer/2022042713/547e05c05906b5a1718b461f/html5/thumbnails/32.jpg)
nginx
pythoncache
db internet
the java
t = start
![Page 33: distributed tracing in 5 minutes](https://reader034.vdocuments.site/reader034/viewer/2022042713/547e05c05906b5a1718b461f/html5/thumbnails/33.jpg)
nginx
pythoncache
db internet
the java
t = start
t = end
![Page 34: distributed tracing in 5 minutes](https://reader034.vdocuments.site/reader034/viewer/2022042713/547e05c05906b5a1718b461f/html5/thumbnails/34.jpg)
piggyback rides•More Doable•HTTP: x-headers• Thrift: secret argument• Internal RPC protocol: you’re the
boss
• Less Doable• SQL: one way ticket, also you’re
not percona•memcache: not extensible so not
backwards compatible
![Page 35: distributed tracing in 5 minutes](https://reader034.vdocuments.site/reader034/viewer/2022042713/547e05c05906b5a1718b461f/html5/thumbnails/35.jpg)
nginx
pythoncache
db internet
the java
t = start
t = end
![Page 36: distributed tracing in 5 minutes](https://reader034.vdocuments.site/reader034/viewer/2022042713/547e05c05906b5a1718b461f/html5/thumbnails/36.jpg)
timing and structure• Timing• distributed = clock skew
• Structure -- two approaches• Encode in ID• Encode in back-pointers
![Page 37: distributed tracing in 5 minutes](https://reader034.vdocuments.site/reader034/viewer/2022042713/547e05c05906b5a1718b461f/html5/thumbnails/37.jpg)
encode in ID?• nginx1• nginx1python1• nginx1python1cache1• nginx1python1cache1python2• nginx1python1cache1python2sql
1• nginx1python1cache1python2sql
1python3• ...
![Page 38: distributed tracing in 5 minutes](https://reader034.vdocuments.site/reader034/viewer/2022042713/547e05c05906b5a1718b461f/html5/thumbnails/38.jpg)
encode in back-pointer?
nginx python cache python
![Page 39: distributed tracing in 5 minutes](https://reader034.vdocuments.site/reader034/viewer/2022042713/547e05c05906b5a1718b461f/html5/thumbnails/39.jpg)
reporting
![Page 40: distributed tracing in 5 minutes](https://reader034.vdocuments.site/reader034/viewer/2022042713/547e05c05906b5a1718b461f/html5/thumbnails/40.jpg)
reporting
![Page 41: distributed tracing in 5 minutes](https://reader034.vdocuments.site/reader034/viewer/2022042713/547e05c05906b5a1718b461f/html5/thumbnails/41.jpg)
![Page 42: distributed tracing in 5 minutes](https://reader034.vdocuments.site/reader034/viewer/2022042713/547e05c05906b5a1718b461f/html5/thumbnails/42.jpg)
![Page 43: distributed tracing in 5 minutes](https://reader034.vdocuments.site/reader034/viewer/2022042713/547e05c05906b5a1718b461f/html5/thumbnails/43.jpg)
![Page 44: distributed tracing in 5 minutes](https://reader034.vdocuments.site/reader034/viewer/2022042713/547e05c05906b5a1718b461f/html5/thumbnails/44.jpg)
![Page 45: distributed tracing in 5 minutes](https://reader034.vdocuments.site/reader034/viewer/2022042713/547e05c05906b5a1718b461f/html5/thumbnails/45.jpg)
![Page 46: distributed tracing in 5 minutes](https://reader034.vdocuments.site/reader034/viewer/2022042713/547e05c05906b5a1718b461f/html5/thumbnails/46.jpg)
![Page 47: distributed tracing in 5 minutes](https://reader034.vdocuments.site/reader034/viewer/2022042713/547e05c05906b5a1718b461f/html5/thumbnails/47.jpg)
![Page 48: distributed tracing in 5 minutes](https://reader034.vdocuments.site/reader034/viewer/2022042713/547e05c05906b5a1718b461f/html5/thumbnails/48.jpg)
![Page 49: distributed tracing in 5 minutes](https://reader034.vdocuments.site/reader034/viewer/2022042713/547e05c05906b5a1718b461f/html5/thumbnails/49.jpg)
other things worth figuring out
• sampling
• reporting
• aggregate analysis
![Page 51: distributed tracing in 5 minutes](https://reader034.vdocuments.site/reader034/viewer/2022042713/547e05c05906b5a1718b461f/html5/thumbnails/51.jpg)
resources• X-Trace: http://x-trace.net• http://x-trace.net/pubs/xtr-nsdi07.pdf
• Google Dapper: http://research.google.com/pubs/pub36356.html
• Twitter Zipkin: https://github.com/twitter/zipkin
• CMU PDL: www.pdl.cmu.edu• StarDust: http://www.pdl.cmu.edu/PDL-FTP/
SelfStar/thereska_sigmetrics06.pdf• Trace Diff: http://www.pdl.cmu.edu/PDL-FTP/
SelfStar/NSDI11.pdf