continuous improvement devops day mountain view 2012
DESCRIPTION
The past few years have seen a phenomenon of software organizations abandoning traditional release cycles in favor of daily or even hourly deployments. The emergence of a new, rapid software development workflow has raised questions regarding the role of test and QA in a product's life cycle. When there is no QA phase, can there still be QA?Indeed, as the speed of product development changes, risk must be assessed differently. A monthly release cycle provides time to assure a product is reasonably free of defects. In a rapid release cycle, such assurance is impossible. Quality instead comes to mean assuring that the product is capable of recovering from defects.I will discuss how software risk mitigation practice changes as the speed of development increases. And I will explore the idea that recovering from failure is a far more pragmatic goal than preventing failure in the first place.TRANSCRIPT
Continuous Improvement
How rapid release cycles alter QA and testing.
Noah Sussman
DevopsDay Mountain View, 2012@noahsussman
#devopsdays
The Canonical Agile Release Cycle
Sprints of two weeks or more in length.Start deployment process at the end of the sprint.
QA is part of the deployment process.QA must be complete before new code goes live.
The Continuous Release Cycle
Minimum viable feature set.Deployment is decoupled from release.
Real-time data on how releases impact revenue.Constant tweaks to live features.
Releasing a feature is decoupled from deploying code.
David E. Smith http://www.flickr.com/photos/david_e_smith/3566697122
An airport without an air traffic controller.
—Chad Dickerson
This Part Really Is Different From Agile
Large features are deployed piecemeal over time.Every feature is part of an A/B campaign.
Dark and Admin-Only launches.Wire-Offs and Config Flags.
There is no “Done Done.”
Observed Behavior Of Complex Systems
Emergent behaviors require unplanned responses.Improvements, too are discovered not designed.
Users of the system have complex expectations.Such systems are never “complete.”
QA Happens When?First of all, what is “Quality Assurance?”Authoritatively assuring that there are no defects?
That’s impossible.
Testing is everyone’s job.
Myths About Bug Detection
There are a finite number of bugs.There are a finite number of detectable bugs.
All severity one bugs can be found before releaseSoftware is built to specifications.
At some point, software is finished.
The Biggest MythBugs have complex, unpredictable causes.
In fact, most errors in software are the results of incorrect assumptions made by programmers.
Many Small Anomalies CombinedThe “Swiss Cheese” model of risk presents us with a strong case for prioritizing the elimination of
small errors rather than focusing on the mitigation of large catastrophic failures.
Unit testing is great at eliminating small errors.
The whole time I’m programming, I’m constantly checking my assumptions.
—Rasmus Lerdorf
Resilience, Not “Quality”
Readable code.Reasonable test coverage.
Sane architecture.Good debugging tools.
An engineering culture that values refactoring.These are measureable goals.
Manual TestingIt doesn’t always look like you think it looks.
Real-Time Monitoring is the new face of testing.
Anomaly detection is hard.
Watching The GraphsEtsy collects well over a quarter million metrics.Deciding which ones matter is a human problem.
Everyone watches some subset of the graphs.Human vision excels at anomaly detection.
QA Happens When??Exploratory testing can be performed any time.Rigorous, scientific approach.
Focus on customer satisfaction rather than a spec.Equally useful before or after a release.
Just Quality“Assurance” is a terrible word. Let’s discard it.Quality exists, we just can’t assure or prove it.
There Is No Such Thing As A Formal Proof Of Quality.
Yet most of us would agree it exists.
I propose that “customer experience” is a better term-of-art-than “quality.”
Though there’s no formal proof for that either.
Exploratory TestingAddresses areas that Developer Testing can’t.Developer Testing validates assumptions.
The Tester’s job is to invalidate assumptions.
Technology Informs Customer ExperienceExploratory Testing requires an understanding of the ways in which a whole system is intended to
serve a community of users.
The problem space has as much to do with technology as it does with product requirements.
Most bugs, most of the time, are easily nailed given even an incomplete but suggestive characterization of their error conditions at source-code level.
—Eric S. Raymond
Source, diffs, logs.If your QA Analysts don’t look at these, teach them.
Customer SupportYour customer support operators spend more time talking to your users than anyone else.
Customer Support interface with users as
individuals rather than as aggregate data.
Keep the feedback loop short.
Manage Your Culture.
Effeciency ToThoroughness
Trade-OffRapid release cycles have different risks thanslower release cycles.
But nothing about risk itself has changed.
Test EverywhereForeseeable errors can be worked out in dev.Unforeseeable errors must be worked out in prod.
Fail ForwardLet go of the idea of “last stable release.”Software exists in context.
Networks, services and people are always in flux.
Forget About Satisfying The Requirements
Watch your graphs.Listen to your customers.
Adhere to your protocols.Improve your product.
Further Reading“How Google Tests Software,” James Whittaker (especially chapter 5)
“Look At Your Data,” John Rausser
“Optimizing For Developer Happiness,” Chad Dickerson
“Outages, Postmortems and Human Error,” John Allspaw
http://en.wikipedia.org/wiki/Swiss_cheese_model
“What Is Exploratory Testing?,” James Bach
“How Many Eyeballs Tame Complexity,” ESR
“The Timeless Way of Building,” Christopher Alexander