continuous delivery while minimizing performance risks
DESCRIPTION
These are the slides to my talk at Velocity Europe 2012: http://velocityconf.com/velocityeu2012/public/schedule/detail/26162TRANSCRIPT
![Page 1: Continuous delivery while minimizing performance risks](https://reader033.vdocuments.site/reader033/viewer/2022052821/55493b4bb4c905194d8b4c8c/html5/thumbnails/1.jpg)
CONTINUOUS
DELIVERY WHILE
MINIMISING
PERFORMANCE
RISKS
![Page 2: Continuous delivery while minimizing performance risks](https://reader033.vdocuments.site/reader033/viewer/2022052821/55493b4bb4c905194d8b4c8c/html5/thumbnails/2.jpg)
INTRODUCTION
![Page 3: Continuous delivery while minimizing performance risks](https://reader033.vdocuments.site/reader033/viewer/2022052821/55493b4bb4c905194d8b4c8c/html5/thumbnails/3.jpg)
OBJECTIVE
Put working software into production as quickly as possible,
whilst minimising risk of load-related problems:
› Bad response times
› Too small capacity
› Availability too low
› Excessive system resource use
![Page 4: Continuous delivery while minimizing performance risks](https://reader033.vdocuments.site/reader033/viewer/2022052821/55493b4bb4c905194d8b4c8c/html5/thumbnails/4.jpg)
RISK PREVENTION IS A BIG SUBJECT
Photo by chillihead: www.flickr.com/photos/chillihead/1778980935
PREVENTING RISK IS A BIG SUBJECT,
WHAT FOLLOWS IS TAKEN FROM OUR EXPERIENCE
![Page 5: Continuous delivery while minimizing performance risks](https://reader033.vdocuments.site/reader033/viewer/2022052821/55493b4bb4c905194d8b4c8c/html5/thumbnails/5.jpg)
CONTINUOUS DELIVERY LITERATURE PROVIDES METHODS
THAT HELP REDUCE RISK
› Blue-green deployments
› Dark launching
› Feature toggles
› Canary releasing
› Production immune systems
Jez Humble, http://continuousdelivery.com
![Page 6: Continuous delivery while minimizing performance risks](https://reader033.vdocuments.site/reader033/viewer/2022052821/55493b4bb4c905194d8b4c8c/html5/thumbnails/6.jpg)
BUT LEGACY SYSTEMS OFTEN LACK THE REQUIRED
RESILIENCE
![Page 7: Continuous delivery while minimizing performance risks](https://reader033.vdocuments.site/reader033/viewer/2022052821/55493b4bb4c905194d8b4c8c/html5/thumbnails/7.jpg)
WHILE WE WORK ON OUR RESILIENCE, WE USE LOAD TESTS
TO HELP IDENTIFY THE BIGGEST RISKS
![Page 8: Continuous delivery while minimizing performance risks](https://reader033.vdocuments.site/reader033/viewer/2022052821/55493b4bb4c905194d8b4c8c/html5/thumbnails/8.jpg)
PRE-PROD LOAD TESTING IS NOT FREE
› Extra code to maintain
› Usually test runs last several hours
› A production-like environment is expensive
› Realistic testing is hard
› Not all developers like writing (performance) tests
![Page 9: Continuous delivery while minimizing performance risks](https://reader033.vdocuments.site/reader033/viewer/2022052821/55493b4bb4c905194d8b4c8c/html5/thumbnails/9.jpg)
![Page 10: Continuous delivery while minimizing performance risks](https://reader033.vdocuments.site/reader033/viewer/2022052821/55493b4bb4c905194d8b4c8c/html5/thumbnails/10.jpg)
USE IT WISELY, WHERE PRODUCTION TESTING IS STILL
INAPPROPRIATE
› It provides no guarantee
› Use it to find any showstoppers you can
› Essentially, an optional service that teams can use
![Page 11: Continuous delivery while minimizing performance risks](https://reader033.vdocuments.site/reader033/viewer/2022052821/55493b4bb4c905194d8b4c8c/html5/thumbnails/11.jpg)
Photo by vastateparksstaff: www.flickr.com/photos/vastateparksstaff/5330257235
USE IT AS A PLAYGROUND TO TRY RISKY CHANGES
![Page 12: Continuous delivery while minimizing performance risks](https://reader033.vdocuments.site/reader033/viewer/2022052821/55493b4bb4c905194d8b4c8c/html5/thumbnails/12.jpg)
Functional integration
tests
Load tests
Build, unit test, etc.
Very often Less often About once a day
(at night)
![Page 13: Continuous delivery while minimizing performance risks](https://reader033.vdocuments.site/reader033/viewer/2022052821/55493b4bb4c905194d8b4c8c/html5/thumbnails/13.jpg)
Functional integration
tests
Load tests
Build, unit test, etc.
Very often Less often About once a day
(at night)
Load test script check
![Page 14: Continuous delivery while minimizing performance risks](https://reader033.vdocuments.site/reader033/viewer/2022052821/55493b4bb4c905194d8b4c8c/html5/thumbnails/14.jpg)
THE AIM IS NOT PERFECTION, GO FOR “AS REALISTIC AS
NEEDED”
![Page 15: Continuous delivery while minimizing performance risks](https://reader033.vdocuments.site/reader033/viewer/2022052821/55493b4bb4c905194d8b4c8c/html5/thumbnails/15.jpg)
SET UP TEST DATA IN THE WEEKEND, TO MINIMIZE
DISRUPTION
![Page 16: Continuous delivery while minimizing performance risks](https://reader033.vdocuments.site/reader033/viewer/2022052821/55493b4bb4c905194d8b4c8c/html5/thumbnails/16.jpg)
WHEN IS A PROBLEM REALLY A PROBLEM?
![Page 17: Continuous delivery while minimizing performance risks](https://reader033.vdocuments.site/reader033/viewer/2022052821/55493b4bb4c905194d8b4c8c/html5/thumbnails/17.jpg)
FIND AN OBJECTIVE WAY TO JUDGE YOUR FINDINGS
![Page 18: Continuous delivery while minimizing performance risks](https://reader033.vdocuments.site/reader033/viewer/2022052821/55493b4bb4c905194d8b4c8c/html5/thumbnails/18.jpg)
ESTABLISH REQUIREMENTS TO MAKE CLEAR WHAT IS
ACCEPTABLE
› Seen from the main stakeholders’ perspective
– Response time: users
– System resources: ops
– Capacity: business
› Specific
› Measurable
› Achievable
› Relevant
![Page 19: Continuous delivery while minimizing performance risks](https://reader033.vdocuments.site/reader033/viewer/2022052821/55493b4bb4c905194d8b4c8c/html5/thumbnails/19.jpg)
Concurrent users
Scale: Maximum load in a day, while
response times are still according to
spec.
Meter: Session table row count.
Intention: The website should at least be
able to manage our typical daily load, but
we would like some margin for growth
and marketing campaigns.
Stakeholder: Business
Fail: < 100k
Target: 200k
Now: 150k
![Page 20: Continuous delivery while minimizing performance risks](https://reader033.vdocuments.site/reader033/viewer/2022052821/55493b4bb4c905194d8b4c8c/html5/thumbnails/20.jpg)
FOR RESPONSE TIMES TOO, FOCUS ON THE MAIN
STAKEHOLDER!
FOR RESPONSE TIMES TOO, FOCUS ON THE
MAIN STAKEHOLDER!
![Page 21: Continuous delivery while minimizing performance risks](https://reader033.vdocuments.site/reader033/viewer/2022052821/55493b4bb4c905194d8b4c8c/html5/thumbnails/21.jpg)
SO USE A REAL BROWSER TO TEST
A REAL USER’S EXPERIENCE
![Page 22: Continuous delivery while minimizing performance risks](https://reader033.vdocuments.site/reader033/viewer/2022052821/55493b4bb4c905194d8b4c8c/html5/thumbnails/22.jpg)
Response time Fail [Today] Target
Homepage.FV > 6 sec 3.9 sec 2 sec
Homepage.RV > 5 sec 2.8 sec 1 sec
Checkout.FV > 8 sec 6.5 sec 2 sec
Details.FV > 6 sec 1.9 sec 2 sec
Details.RV > 5 sec 1.7 sec 1 sec
Search.FV > 6 sec 4.8 sec 2 sec
Search.RV > 5 sec 3.7 sec 1 sec
Cart.FV > 6 sec 4.4 sec 2 sec
Cart.RV > 5 sec 3.4 sec 1 sec
LoginForm.FV > 6 sec 3.5 sec 2 sec
LoginForm.RV > 5 sec 2.5 sec 1 sec
![Page 23: Continuous delivery while minimizing performance risks](https://reader033.vdocuments.site/reader033/viewer/2022052821/55493b4bb4c905194d8b4c8c/html5/thumbnails/23.jpg)
![Page 24: Continuous delivery while minimizing performance risks](https://reader033.vdocuments.site/reader033/viewer/2022052821/55493b4bb4c905194d8b4c8c/html5/thumbnails/24.jpg)
TO MAKE COMPARING SENSIBLE, MAKE YOUR TESTS
DETERMINISTIC
Stub systems that you have no control over
![Page 25: Continuous delivery while minimizing performance risks](https://reader033.vdocuments.site/reader033/viewer/2022052821/55493b4bb4c905194d8b4c8c/html5/thumbnails/25.jpg)
LOAD TESTING SHOULD BE OPTIONAL, THE ONLY THING
THAT COUNTS IS PRODUCTION!
› Your definition of done should reflect that
› The aim is to get early feedback from a safe environment
![Page 26: Continuous delivery while minimizing performance risks](https://reader033.vdocuments.site/reader033/viewer/2022052821/55493b4bb4c905194d8b4c8c/html5/thumbnails/26.jpg)
ANYTHING YOU FIND IS AN OPPORTUNITY TO FIX MORE
THAN ONE PROBLEM
![Page 27: Continuous delivery while minimizing performance risks](https://reader033.vdocuments.site/reader033/viewer/2022052821/55493b4bb4c905194d8b4c8c/html5/thumbnails/27.jpg)
SO WHAT MONITORING IS TYPICALLY NEEDED?
› Be able to localise where latency is coming from!
– For every system, all incoming and outgoing calls (count and
time spent stats)
› Finite resources (pools, CPU, I/O, etc.)
› Number of active users
› Response size, where possible
› Add whatever you need
It should be identical on all environments!
![Page 28: Continuous delivery while minimizing performance risks](https://reader033.vdocuments.site/reader033/viewer/2022052821/55493b4bb4c905194d8b4c8c/html5/thumbnails/28.jpg)
SET CLEAR TARGETS, SO YOU KNOW YOUR SITUATION
› How many errors would be OK in production?
› What kind of errors do we care about?
![Page 29: Continuous delivery while minimizing performance risks](https://reader033.vdocuments.site/reader033/viewer/2022052821/55493b4bb4c905194d8b4c8c/html5/thumbnails/29.jpg)
0
50
100
150
200
250
30000:0
0
01:0
0
02:0
0
03
:00
04:0
0
05:0
0
06:0
0
07:0
0
08
:00
09:0
0
10:0
0
11:0
0
12:0
0
13:0
0
14:0
0
15:0
0
16:0
0
17:0
0
18:0
0
19
:00
20:0
0
21:0
0
22:0
0
23:0
0
00
:00
Nu
mb
er
of
sta
le s
erv
er
sessio
n r
eq
uests
/
min
Other servers taken out of LB Other servers
back in LB
![Page 30: Continuous delivery while minimizing performance risks](https://reader033.vdocuments.site/reader033/viewer/2022052821/55493b4bb4c905194d8b4c8c/html5/thumbnails/30.jpg)
CONCLUSION
Support your continuous delivery process with optional load
tests and strong specs.
Use the load tests to identify some pain points, so you can
modify the code and add monitoring, making it safer to do
dark releases and canary testing in production.
Constantly ask yourself: what would it take to only do this in
production? Is it adding value?