beyond averages - web performance meetup
DESCRIPTION
When raw data becomes overwhelming, we turn to abstraction to understand our world. In examining the performance of our systems, the data is always overwhelming. Solutions like summary statistics have come to our rescue, and they are good—up to a point. In order to truly understand our systems, we need to know when and how to sidestep those abstractions,to get deep, detailed performance insight. At this meetup, I’ll explore techniques for visualizing the underlying structure of performance data and how this empowers drilling down to populations and individual samples in the data set.TRANSCRIPT
![Page 1: Beyond Averages - Web Performance Meetup](https://reader033.vdocuments.site/reader033/viewer/2022060121/5593ba401a28ab22548b45f5/html5/thumbnails/1.jpg)
BEYOND AVERAGESDan Kuebrich / @dankosaur
![Page 2: Beyond Averages - Web Performance Meetup](https://reader033.vdocuments.site/reader033/viewer/2022060121/5593ba401a28ab22548b45f5/html5/thumbnails/2.jpg)
Beyond Averages
• Abstraction: summary statistics for performance data
• Why performance data is hard
• Visualizing data distributions
![Page 3: Beyond Averages - Web Performance Meetup](https://reader033.vdocuments.site/reader033/viewer/2022060121/5593ba401a28ab22548b45f5/html5/thumbnails/3.jpg)
![Page 4: Beyond Averages - Web Performance Meetup](https://reader033.vdocuments.site/reader033/viewer/2022060121/5593ba401a28ab22548b45f5/html5/thumbnails/4.jpg)
![Page 5: Beyond Averages - Web Performance Meetup](https://reader033.vdocuments.site/reader033/viewer/2022060121/5593ba401a28ab22548b45f5/html5/thumbnails/5.jpg)
A few of my favorite abstractions
![Page 6: Beyond Averages - Web Performance Meetup](https://reader033.vdocuments.site/reader033/viewer/2022060121/5593ba401a28ab22548b45f5/html5/thumbnails/6.jpg)
A few of my favorite abstractions• Abstraction lets us trade information for
actionability
![Page 7: Beyond Averages - Web Performance Meetup](https://reader033.vdocuments.site/reader033/viewer/2022060121/5593ba401a28ab22548b45f5/html5/thumbnails/7.jpg)
A few of my favorite abstractions• Abstraction lets us trade information for
actionability
• Min, max, average (“mean”), quantiles, stdev
![Page 8: Beyond Averages - Web Performance Meetup](https://reader033.vdocuments.site/reader033/viewer/2022060121/5593ba401a28ab22548b45f5/html5/thumbnails/8.jpg)
A few of my favorite abstractions• Abstraction lets us trade information for
actionability
• Min, max, average (“mean”), quantiles, stdev
• That’s a great trade! • ... right?
![Page 9: Beyond Averages - Web Performance Meetup](https://reader033.vdocuments.site/reader033/viewer/2022060121/5593ba401a28ab22548b45f5/html5/thumbnails/9.jpg)
Averages: average at best
![Page 10: Beyond Averages - Web Performance Meetup](https://reader033.vdocuments.site/reader033/viewer/2022060121/5593ba401a28ab22548b45f5/html5/thumbnails/10.jpg)
Averages: average at best
![Page 11: Beyond Averages - Web Performance Meetup](https://reader033.vdocuments.site/reader033/viewer/2022060121/5593ba401a28ab22548b45f5/html5/thumbnails/11.jpg)
Averages: average at best
![Page 12: Beyond Averages - Web Performance Meetup](https://reader033.vdocuments.site/reader033/viewer/2022060121/5593ba401a28ab22548b45f5/html5/thumbnails/12.jpg)
Averages: average at best
![Page 13: Beyond Averages - Web Performance Meetup](https://reader033.vdocuments.site/reader033/viewer/2022060121/5593ba401a28ab22548b45f5/html5/thumbnails/13.jpg)
Averages vs Percentiles
[1, 16, 17, 19, 13, 5, 20, 3, 10, 14, 8]
![Page 14: Beyond Averages - Web Performance Meetup](https://reader033.vdocuments.site/reader033/viewer/2022060121/5593ba401a28ab22548b45f5/html5/thumbnails/14.jpg)
Averages vs Percentiles
[1, 3, 5, 8, 10, 13, 14, 16, 17, 19, 20]
![Page 15: Beyond Averages - Web Performance Meetup](https://reader033.vdocuments.site/reader033/viewer/2022060121/5593ba401a28ab22548b45f5/html5/thumbnails/15.jpg)
Averages vs Percentiles
[1, 3, 5, 8, 10, 13, 14, 16, 17, 19, 20]
Average (Mean): 11.54
Median (50th Percentile): 13 90th Percentile: 19
![Page 16: Beyond Averages - Web Performance Meetup](https://reader033.vdocuments.site/reader033/viewer/2022060121/5593ba401a28ab22548b45f5/html5/thumbnails/16.jpg)
Percentiles: 1 of 100 slices95%
X
![Page 17: Beyond Averages - Web Performance Meetup](https://reader033.vdocuments.site/reader033/viewer/2022060121/5593ba401a28ab22548b45f5/html5/thumbnails/17.jpg)
Percentiles: 2 of 100 slices95%
10%
X
Y
![Page 18: Beyond Averages - Web Performance Meetup](https://reader033.vdocuments.site/reader033/viewer/2022060121/5593ba401a28ab22548b45f5/html5/thumbnails/18.jpg)
Percentiles: 2 of 100 slices95%
10%
![Page 19: Beyond Averages - Web Performance Meetup](https://reader033.vdocuments.site/reader033/viewer/2022060121/5593ba401a28ab22548b45f5/html5/thumbnails/19.jpg)
Percentiles: 2 of 100 slices95%
10%
![Page 20: Beyond Averages - Web Performance Meetup](https://reader033.vdocuments.site/reader033/viewer/2022060121/5593ba401a28ab22548b45f5/html5/thumbnails/20.jpg)
Percentiles: 2 of 100 slices95%
10%
![Page 21: Beyond Averages - Web Performance Meetup](https://reader033.vdocuments.site/reader033/viewer/2022060121/5593ba401a28ab22548b45f5/html5/thumbnails/21.jpg)
Computers are hard• Rarely do we have a single distribution underlying the
data
• Different users, different requests, different resources, different instances, different times
![Page 22: Beyond Averages - Web Performance Meetup](https://reader033.vdocuments.site/reader033/viewer/2022060121/5593ba401a28ab22548b45f5/html5/thumbnails/22.jpg)
Percentiles vs Distributions
http://en.wikipedia.org/wiki/Percentile
![Page 23: Beyond Averages - Web Performance Meetup](https://reader033.vdocuments.site/reader033/viewer/2022060121/5593ba401a28ab22548b45f5/html5/thumbnails/23.jpg)
Percentiles vs Distributions
[13, 13, 13, 13, 13, 13, 19, 19, 19, 19, 19]
Median (50th Percentile): 13 90th Percentile: 19
![Page 24: Beyond Averages - Web Performance Meetup](https://reader033.vdocuments.site/reader033/viewer/2022060121/5593ba401a28ab22548b45f5/html5/thumbnails/24.jpg)
Rarely do we have a single normal distribution underlying the data
![Page 25: Beyond Averages - Web Performance Meetup](https://reader033.vdocuments.site/reader033/viewer/2022060121/5593ba401a28ab22548b45f5/html5/thumbnails/25.jpg)
![Page 26: Beyond Averages - Web Performance Meetup](https://reader033.vdocuments.site/reader033/viewer/2022060121/5593ba401a28ab22548b45f5/html5/thumbnails/26.jpg)
Median Mean 90th
![Page 27: Beyond Averages - Web Performance Meetup](https://reader033.vdocuments.site/reader033/viewer/2022060121/5593ba401a28ab22548b45f5/html5/thumbnails/27.jpg)
![Page 28: Beyond Averages - Web Performance Meetup](https://reader033.vdocuments.site/reader033/viewer/2022060121/5593ba401a28ab22548b45f5/html5/thumbnails/28.jpg)
![Page 29: Beyond Averages - Web Performance Meetup](https://reader033.vdocuments.site/reader033/viewer/2022060121/5593ba401a28ab22548b45f5/html5/thumbnails/29.jpg)
![Page 30: Beyond Averages - Web Performance Meetup](https://reader033.vdocuments.site/reader033/viewer/2022060121/5593ba401a28ab22548b45f5/html5/thumbnails/30.jpg)
Median Mean 90th
![Page 31: Beyond Averages - Web Performance Meetup](https://reader033.vdocuments.site/reader033/viewer/2022060121/5593ba401a28ab22548b45f5/html5/thumbnails/31.jpg)
The Log-Normal Distribution
(source: http://www.geo.mtu.edu/volcanoes/vc_web/background/S_chem.html)
![Page 32: Beyond Averages - Web Performance Meetup](https://reader033.vdocuments.site/reader033/viewer/2022060121/5593ba401a28ab22548b45f5/html5/thumbnails/32.jpg)
Log-Normal Distribution
(source: http://en.wikipedia.org/wiki/File:Comparison_mean_median_mode.svg)
![Page 33: Beyond Averages - Web Performance Meetup](https://reader033.vdocuments.site/reader033/viewer/2022060121/5593ba401a28ab22548b45f5/html5/thumbnails/33.jpg)
Log-Normal Distribution
(source: http://en.wikipedia.org/wiki/File:Comparison_mean_median_mode.svg)
![Page 34: Beyond Averages - Web Performance Meetup](https://reader033.vdocuments.site/reader033/viewer/2022060121/5593ba401a28ab22548b45f5/html5/thumbnails/34.jpg)
Log-Normal Distribution
![Page 35: Beyond Averages - Web Performance Meetup](https://reader033.vdocuments.site/reader033/viewer/2022060121/5593ba401a28ab22548b45f5/html5/thumbnails/35.jpg)
Is there a place between Averageland and “A Beautiful Mind”?
http://now-here-this.timeout.com/2012/10/07/crazy-walls-of-clues-from-tv-film-reviewed-by-carrie-from-homeland/
![Page 36: Beyond Averages - Web Performance Meetup](https://reader033.vdocuments.site/reader033/viewer/2022060121/5593ba401a28ab22548b45f5/html5/thumbnails/36.jpg)
Histograms
Freq
uenc
y (e
g. #
of c
alls
)
Value (eg. latency)
![Page 37: Beyond Averages - Web Performance Meetup](https://reader033.vdocuments.site/reader033/viewer/2022060121/5593ba401a28ab22548b45f5/html5/thumbnails/37.jpg)
Populations revisited95%
10%
![Page 38: Beyond Averages - Web Performance Meetup](https://reader033.vdocuments.site/reader033/viewer/2022060121/5593ba401a28ab22548b45f5/html5/thumbnails/38.jpg)
HistogramsFr
eque
ncy
(eg.
# o
f cal
ls)
Value (eg. latency)
![Page 39: Beyond Averages - Web Performance Meetup](https://reader033.vdocuments.site/reader033/viewer/2022060121/5593ba401a28ab22548b45f5/html5/thumbnails/39.jpg)
Populations re-revisited95%
10%?
![Page 40: Beyond Averages - Web Performance Meetup](https://reader033.vdocuments.site/reader033/viewer/2022060121/5593ba401a28ab22548b45f5/html5/thumbnails/40.jpg)
3d Histograms?
Freq
uenc
y (e
g. #
of c
alls
)
Value (eg. latency)
![Page 41: Beyond Averages - Web Performance Meetup](https://reader033.vdocuments.site/reader033/viewer/2022060121/5593ba401a28ab22548b45f5/html5/thumbnails/41.jpg)
3d Histograms?Fr
eque
ncy
(eg.
# o
f cal
ls)
Value (eg. latency)
Time
![Page 42: Beyond Averages - Web Performance Meetup](https://reader033.vdocuments.site/reader033/viewer/2022060121/5593ba401a28ab22548b45f5/html5/thumbnails/42.jpg)
Heatmaps
Freq
uenc
y (e
g. #
of c
alls
)
Value (eg. latency)
![Page 43: Beyond Averages - Web Performance Meetup](https://reader033.vdocuments.site/reader033/viewer/2022060121/5593ba401a28ab22548b45f5/html5/thumbnails/43.jpg)
HeatmapsFr
eque
ncy
(eg.
# o
f cal
ls)
Value (eg. latency)
![Page 44: Beyond Averages - Web Performance Meetup](https://reader033.vdocuments.site/reader033/viewer/2022060121/5593ba401a28ab22548b45f5/html5/thumbnails/44.jpg)
HeatmapsFr
eque
ncy
(eg.
# o
f cal
ls)
Value (eg. latency)
![Page 45: Beyond Averages - Web Performance Meetup](https://reader033.vdocuments.site/reader033/viewer/2022060121/5593ba401a28ab22548b45f5/html5/thumbnails/45.jpg)
HeatmapsVa
lue
(eg.
late
ncy)
Time
![Page 46: Beyond Averages - Web Performance Meetup](https://reader033.vdocuments.site/reader033/viewer/2022060121/5593ba401a28ab22548b45f5/html5/thumbnails/46.jpg)
HeatmapsVa
lue
(eg.
late
ncy)
Time
![Page 47: Beyond Averages - Web Performance Meetup](https://reader033.vdocuments.site/reader033/viewer/2022060121/5593ba401a28ab22548b45f5/html5/thumbnails/47.jpg)
Latency in the wild…
http://sciencefiction.com/2013/10/24/throwback-thursday-jurassic-park/
![Page 48: Beyond Averages - Web Performance Meetup](https://reader033.vdocuments.site/reader033/viewer/2022060121/5593ba401a28ab22548b45f5/html5/thumbnails/48.jpg)
Average, or Absolute?
![Page 49: Beyond Averages - Web Performance Meetup](https://reader033.vdocuments.site/reader033/viewer/2022060121/5593ba401a28ab22548b45f5/html5/thumbnails/49.jpg)
Average, or Absolute?
![Page 50: Beyond Averages - Web Performance Meetup](https://reader033.vdocuments.site/reader033/viewer/2022060121/5593ba401a28ab22548b45f5/html5/thumbnails/50.jpg)
Multi-modal Data
![Page 51: Beyond Averages - Web Performance Meetup](https://reader033.vdocuments.site/reader033/viewer/2022060121/5593ba401a28ab22548b45f5/html5/thumbnails/51.jpg)
Multi-modal Data
![Page 52: Beyond Averages - Web Performance Meetup](https://reader033.vdocuments.site/reader033/viewer/2022060121/5593ba401a28ab22548b45f5/html5/thumbnails/52.jpg)
Multi-modal Drill-down
![Page 53: Beyond Averages - Web Performance Meetup](https://reader033.vdocuments.site/reader033/viewer/2022060121/5593ba401a28ab22548b45f5/html5/thumbnails/53.jpg)
Multi-modal Drill-down
![Page 54: Beyond Averages - Web Performance Meetup](https://reader033.vdocuments.site/reader033/viewer/2022060121/5593ba401a28ab22548b45f5/html5/thumbnails/54.jpg)
Multi-modal Drill-down
![Page 55: Beyond Averages - Web Performance Meetup](https://reader033.vdocuments.site/reader033/viewer/2022060121/5593ba401a28ab22548b45f5/html5/thumbnails/55.jpg)
Long Tails and Outliers
![Page 56: Beyond Averages - Web Performance Meetup](https://reader033.vdocuments.site/reader033/viewer/2022060121/5593ba401a28ab22548b45f5/html5/thumbnails/56.jpg)
bottom 98%
Long Tails and Outliers
![Page 57: Beyond Averages - Web Performance Meetup](https://reader033.vdocuments.site/reader033/viewer/2022060121/5593ba401a28ab22548b45f5/html5/thumbnails/57.jpg)
bottom 98%
Long Tails and Outliers
![Page 58: Beyond Averages - Web Performance Meetup](https://reader033.vdocuments.site/reader033/viewer/2022060121/5593ba401a28ab22548b45f5/html5/thumbnails/58.jpg)
all of it
Long Tails and Outliers
![Page 59: Beyond Averages - Web Performance Meetup](https://reader033.vdocuments.site/reader033/viewer/2022060121/5593ba401a28ab22548b45f5/html5/thumbnails/59.jpg)
Added Population
![Page 60: Beyond Averages - Web Performance Meetup](https://reader033.vdocuments.site/reader033/viewer/2022060121/5593ba401a28ab22548b45f5/html5/thumbnails/60.jpg)
Added Population
![Page 61: Beyond Averages - Web Performance Meetup](https://reader033.vdocuments.site/reader033/viewer/2022060121/5593ba401a28ab22548b45f5/html5/thumbnails/61.jpg)
Added Population
![Page 62: Beyond Averages - Web Performance Meetup](https://reader033.vdocuments.site/reader033/viewer/2022060121/5593ba401a28ab22548b45f5/html5/thumbnails/62.jpg)
Thanks!
Sign up for a free TraceView account at appneta.com/products/traceview-free-account