do it in-production-seth_eliot_2013_03
TRANSCRIPT
![Page 1: Do it in-production-seth_eliot_2013_03](https://reader035.vdocuments.site/reader035/viewer/2022070315/5554aeabb4c905fd608b5410/html5/thumbnails/1.jpg)
Do It In ProductionTesting Where It Counts
Seth EliotSenior Knowledge Engineer, Test Excellence
![Page 2: Do it in-production-seth_eliot_2013_03](https://reader035.vdocuments.site/reader035/viewer/2022070315/5554aeabb4c905fd608b5410/html5/thumbnails/2.jpg)
Services and Cloud
A/B Testing of Services
Digital Media Services
Petabytes Processed
About Seth
Part 3July 2012The Cloud
Part 2March 2012TestOps
Part 1Nov 2011Testing in Production
The Future of Software Testing
![Page 3: Do it in-production-seth_eliot_2013_03](https://reader035.vdocuments.site/reader035/viewer/2022070315/5554aeabb4c905fd608b5410/html5/thumbnails/3.jpg)
Measurement
A quantitatively expressed reduction of uncertainty
…based on one or more observations
Testing
…about quality of a system under test…
![Page 4: Do it in-production-seth_eliot_2013_03](https://reader035.vdocuments.site/reader035/viewer/2022070315/5554aeabb4c905fd608b5410/html5/thumbnails/4.jpg)
Data Driven Decision Making
Data Driven Validation
Testing in Production
TiPReal Users
Production Environments
![Page 5: Do it in-production-seth_eliot_2013_03](https://reader035.vdocuments.site/reader035/viewer/2022070315/5554aeabb4c905fd608b5410/html5/thumbnails/5.jpg)
![Page 6: Do it in-production-seth_eliot_2013_03](https://reader035.vdocuments.site/reader035/viewer/2022070315/5554aeabb4c905fd608b5410/html5/thumbnails/6.jpg)
Real World Performance Monitoring
500 Million measurements per month
JSIJavaScript Instrumentation
![Page 7: Do it in-production-seth_eliot_2013_03](https://reader035.vdocuments.site/reader035/viewer/2022070315/5554aeabb4c905fd608b5410/html5/thumbnails/7.jpg)
What is Big Data?
Value
VelocityVolume
MB GB TBPB EB ZB
Variety
![Page 8: Do it in-production-seth_eliot_2013_03](https://reader035.vdocuments.site/reader035/viewer/2022070315/5554aeabb4c905fd608b5410/html5/thumbnails/8.jpg)
TwitterXbox Kinect
![Page 9: Do it in-production-seth_eliot_2013_03](https://reader035.vdocuments.site/reader035/viewer/2022070315/5554aeabb4c905fd608b5410/html5/thumbnails/9.jpg)
What Does Data Have To Do With TiP?
Biggest and Best Data is in Production
Testing in Production (TiP)
Use Data for Quality Assessment
Production is Truth
![Page 10: Do it in-production-seth_eliot_2013_03](https://reader035.vdocuments.site/reader035/viewer/2022070315/5554aeabb4c905fd608b5410/html5/thumbnails/10.jpg)
Why Testing in Production (TiP)?
Find this with a unit test….
“Walking Directions”
This route may be missing sidewalks
![Page 11: Do it in-production-seth_eliot_2013_03](https://reader035.vdocuments.site/reader035/viewer/2022070315/5554aeabb4c905fd608b5410/html5/thumbnails/11.jpg)
5 Million Metrics
Grid Report:CPU, Network…
Operational Data Store (ODS)
System…Application…Business……Metrics
Historical trending and analysisHadoop
![Page 12: Do it in-production-seth_eliot_2013_03](https://reader035.vdocuments.site/reader035/viewer/2022070315/5554aeabb4c905fd608b5410/html5/thumbnails/12.jpg)
Metrics find…Feature broken for significant percent of users
Metrics find…Problems at Scale
Constant dogfooding with reporting tools
Engineers stay with code every step of the way
“This process works for Facebook partly because Facebook does not, by and large, need to produce particularly high-quality software” Really?
![Page 13: Do it in-production-seth_eliot_2013_03](https://reader035.vdocuments.site/reader035/viewer/2022070315/5554aeabb4c905fd608b5410/html5/thumbnails/13.jpg)
Just Throw it in Production?
Integration Testing
Functional TestingUnit Testing
Metrics and Optics
…maybe less of thisTesting in Production
Functional Testing
Integration Testing
Performance and ScalabilityTDD
InstrumentEverything
![Page 14: Do it in-production-seth_eliot_2013_03](https://reader035.vdocuments.site/reader035/viewer/2022070315/5554aeabb4c905fd608b5410/html5/thumbnails/14.jpg)
Real … Users… Applications… Infrastructure
Synthetic Transactions
Active ValidationPassive Validation
AvailabilityReliabilityPerformance
Operational Intelligence
Business Intelligence
Two Types of Data-Driven Validation
![Page 15: Do it in-production-seth_eliot_2013_03](https://reader035.vdocuments.site/reader035/viewer/2022070315/5554aeabb4c905fd608b5410/html5/thumbnails/15.jpg)
AvailabilityPerformance
From Azure
![Page 16: Do it in-production-seth_eliot_2013_03](https://reader035.vdocuments.site/reader035/viewer/2022070315/5554aeabb4c905fd608b5410/html5/thumbnails/16.jpg)
Fault Injection
Chaos Monkey
Latency Monkey
Game Day
Chaos Gorilla
![Page 17: Do it in-production-seth_eliot_2013_03](https://reader035.vdocuments.site/reader035/viewer/2022070315/5554aeabb4c905fd608b5410/html5/thumbnails/17.jpg)
Synthetics Have Risks
Who's Got the Monkey?
Who's Got the Monkey Now?
Monkey
![Page 18: Do it in-production-seth_eliot_2013_03](https://reader035.vdocuments.site/reader035/viewer/2022070315/5554aeabb4c905fd608b5410/html5/thumbnails/18.jpg)
Synthetics Have Risks
![Page 19: Do it in-production-seth_eliot_2013_03](https://reader035.vdocuments.site/reader035/viewer/2022070315/5554aeabb4c905fd608b5410/html5/thumbnails/19.jpg)
Provides insight into real usage
Reproducible and well understood scenarios
Covers a vast variety of environments
Requires proper handling of Personally Identifiable Information (PII)
May adversely alter production and production data
P
P
P
A
A
You’ve learned a lot already…
![Page 20: Do it in-production-seth_eliot_2013_03](https://reader035.vdocuments.site/reader035/viewer/2022070315/5554aeabb4c905fd608b5410/html5/thumbnails/20.jpg)
Experimentation
Exposure Control
Dogfood and Beta
A/B Testing
“To have a great idea, have a lot of them”
-Thomas Edison
![Page 21: Do it in-production-seth_eliot_2013_03](https://reader035.vdocuments.site/reader035/viewer/2022070315/5554aeabb4c905fd608b5410/html5/thumbnails/21.jpg)
Experimentation at Google
“…dice and slice in any way you can possibly fathom”
1% launches
Shadow launches
1/3
2/3
![Page 22: Do it in-production-seth_eliot_2013_03](https://reader035.vdocuments.site/reader035/viewer/2022070315/5554aeabb4c905fd608b5410/html5/thumbnails/22.jpg)
Experimentation at Netflix
1B API requests per day
Canary Deployment
![Page 23: Do it in-production-seth_eliot_2013_03](https://reader035.vdocuments.site/reader035/viewer/2022070315/5554aeabb4c905fd608b5410/html5/thumbnails/23.jpg)
A brief look at data science
Beware of averages
Even small data is useful data
Science!
Watch out for sample bias
![Page 24: Do it in-production-seth_eliot_2013_03](https://reader035.vdocuments.site/reader035/viewer/2022070315/5554aeabb4c905fd608b5410/html5/thumbnails/24.jpg)
How many years have you worked in software?
![Page 25: Do it in-production-seth_eliot_2013_03](https://reader035.vdocuments.site/reader035/viewer/2022070315/5554aeabb4c905fd608b5410/html5/thumbnails/25.jpg)
Averages are Your Enemy
![Page 26: Do it in-production-seth_eliot_2013_03](https://reader035.vdocuments.site/reader035/viewer/2022070315/5554aeabb4c905fd608b5410/html5/thumbnails/26.jpg)
Tools of Big Data: Hadoop
Hadoop HDFS
ADDFCAECC
ADD FCA
ECC
ADDADD
ADD
FCAFCA
FCA
ECCECC
ECC
Map-Reduce
1xA1xC1xF
1xA2xD
1xE2xC
2xA0xB3xC2xD1xE1xF
![Page 27: Do it in-production-seth_eliot_2013_03](https://reader035.vdocuments.site/reader035/viewer/2022070315/5554aeabb4c905fd608b5410/html5/thumbnails/27.jpg)
Tools of Big Data: Cosmos
Processing 2PB per day
On tens of thousands of computers
Stores hundreds of petabyte
![Page 28: Do it in-production-seth_eliot_2013_03](https://reader035.vdocuments.site/reader035/viewer/2022070315/5554aeabb4c905fd608b5410/html5/thumbnails/28.jpg)
“We know we can't anticipate the 101 things that will go wrong, The only thing we can control is ensuring our team responds appropriately to those situations.“ – Jerry Hook, Executive Producer Halo
…Hundreds of thousands of requests per second
Hack using modified Xbox
Halo ServersXbox Console SharepointPower Pivot Action
Target
“But even if you’re following the law, you can do things where people get queasy.”
HDInsight
Hadoop on Azure
HDInsight
![Page 29: Do it in-production-seth_eliot_2013_03](https://reader035.vdocuments.site/reader035/viewer/2022070315/5554aeabb4c905fd608b5410/html5/thumbnails/29.jpg)
Availability (y) over time (x)
Predict 75% of dips 24 hours ahead of timeData
c
Machine Learning
c
Cosmos
c
![Page 30: Do it in-production-seth_eliot_2013_03](https://reader035.vdocuments.site/reader035/viewer/2022070315/5554aeabb4c905fd608b5410/html5/thumbnails/30.jpg)
Passive
Active
The best observations are often in production
A set of observations to reduce uncertainty about quality of a system under testTesting
![Page 31: Do it in-production-seth_eliot_2013_03](https://reader035.vdocuments.site/reader035/viewer/2022070315/5554aeabb4c905fd608b5410/html5/thumbnails/31.jpg)
blog: http://bit.ly/seth_qa
Do It In Production
Testing Where it Counts
me….
@setheliot
?
Thank you!