transform your agile test process to ship fast with high quality
DESCRIPTION
Until 2009, the Atlassian JIRA team shipped a major release of its software every nine to twelve months. Everything was tested—every story and every bug fix—and everything still contained serious bugs. Story development moved quickly, but after the feature-complete date, several month-long hardening periods were required to make the software actually shippable. Integrating the release into Atlassian’s hosted platform took another three or four months. Now, the team ships JIRA from master directly to the hosted platform every two weeks, with features being written and tested within an iteration, and then bundled into a shippable release at the end. Penny Wyatt describes how to move the responsibility for quality back to developers, find bugs before development work even starts through early communication, use an active feedback loop to prevent classes of bugs, and reinforce the concept that “done means done”—ruthlessly backing out changes that aren’t fully completed by the end of the iteration. You, too, can learn how to ship fast with high quality.TRANSCRIPT
AW12 Session 6/5/2013 3:45 PM
"Transforming Your Agile Test Process"
Presented by:
Penny Wyatt Atlassian Software
Brought to you by:
340 Corporate Way, Suite 300, Orange Park, FL 32073 888‐268‐8770 ∙ 904‐278‐0524 ∙ [email protected] ∙ www.sqe.com
Penny Wyatt Atlassian
With ten years of software industry experience, Penny Wyatt works at Atlassian as the QA team lead for JIRA, an issue tracker used by more than 11,000 organizations worldwide. Penny started her career as a software developer, but after joining Microsoft in Redmond as a developer in test, she discovered that breaking software is much more enjoyable than building it. After a few years of developing testing tools, Penny realized that hiring a small test team to educate developers and prevent bugs is much more efficient than hiring a large test team to find bugs. Penny focuses on white-box testing, security training, and ruining developers' fun.
Transform Your Agile Process –Ship Fast with High Quality
Penny WyattAtlassian
Who am I?
0 5 10
DevDev‐in‐TestTestQA
Experience (Years)
What do I do?
Why am I here?
JIRA, 2010
• About 15 Developers• 2 QA Engineers
JIRA, 2010
Development QA Hardening Release
9‐12 months
Everything I code gets rejected over and over,
months laterI want to see increased sales now, not in 9
months
The harder I work, the people criticise
Why can’t you add this feature? Surely
it’s not hard.
me for slowing down shipping
Let’s hire more QA!
Quality AssuranceAssistance
JIRA, 2011
• About 20 Developers• 2 QA Engineers
DoTing
• DoS – Developer on Support• DoT – Developer on Test
DoTing
• Aims– Remove QA bottleneck– Make developers responsible for quality– Remove the dev/QA wall
JIRA, 2011
Development
QA Hardening Release
QA
DoT
6‐9 months
The results
• QA was more scalableBut…• Bugs found later• Hardening became more important• Stories in “Done” state weren’t really done• Inconsistent quality across JIRA
Time
Scope Quality
Improving DoTing
• Better DoTing means less hardening• We were testing experts, they weren’t• “No one asks the developers to teach the tech writers how to code”
DoTing Training
• Testing technique• JIRA‐specific hints• Practical examples
JIRA, 2012
• About 35 Developers• 2 QA Engineers
Crisis Time!
• QA team becoming irrelevant• Hardening phase getting longer• Focus on SaaS platform• Goal – to ship from master to production every two weeks
Goals
• Scalability of QA• Better usability• Consistent high quality• Developer education• Done means Done• Early intervention
Where are the bugs?QA DoT DoT DoT DoT DoT
Usability issues QAUsability issues QA
Non‐obvious bugs QA
D T D T D T D T D T
Story 1 Story 2 Story 3 Story 4 Story 5 Story 6
Obvious bugs QA DoT DoT DoT DoT DoT
DoT DoT DoT DoT DoT
Breadth, not DepthDoT DoT DoT DoT DoTDoT
QA QA QA QA QA QAUsability issues QA QA QA QA QA QAUsability issues
Non‐obvious bugs
D T D T D T D T D T
Story 1 Story 2 Story 3 Story 4 Story 5 Story 6
Obvious bugs DoT DoT DoT DoT DoT DoT
DoT DoT DoT DoT DoTDoT
Goals
Scalability of QABetter usability
• Consistent high quality• Developer education• Done means Done• Early intervention
QA Magic
Developers can’t test?
• “QA Magic”– Detailed product knowledge– SaaS knowledge– Keeping all testing aspects in mind– Feedback loop from bug history– Exploratory testing skillsExploratory testing skills
Testing Notes
• Hints for the testing of a feature• Specific to a story• Written by a QA engineer after a quick pairing session with the dev
• Consumed by a DoT while testing
Testing Notes
• Hints and tips• Not checklists• Assume the competence and intelligence of the reader
Testing Notes
“As an administrator, I want to invite users via email to sign up to JIRA”
• Click “Invite”. • Check that the invite popup appears. • Insert an email address. • Click “Send”. • Check that the email arrives.
users via email to sign up to JIRA
• Click “Invite”. • Check that the invite popup appears. • Insert an invalid email address. • Click “Send”. • Check that an error appears.
Testing Notes
“As an administrator, I want to invite users via email to sign up to JIRA”
• What if we’re connected to a read‐only LDAP server?
• What is outgoing mail is disabled?• Check security of the invite tokens – can th b d? D th i ?
users via email to sign up to JIRA
they be reused? Do they expire?
Benefits of Testing Notes
• No more “QA Magic”• Easy dissemination of knowledge• More efficient DoTing
Developers can test!DoT DoT DoT DoT DoTDoT
QA QA QA QA QA QAUsability issues QA QA QA QA QA QAUsability issues
Non‐obvious bugs DoT DoT DoT DoT DoTDoT
Story 1 Story 2 Story 3 Story 4 Story 5 Story 6
Obvious bugs DoT DoT DoT DoT DoT DoT
Done means Done
• Story is Done ‐> Ready to release• Release is always in a known state• Miss the deadline?
– Finish the work– Or back it out
Goals
Scalability of QABetter usabilityConsistent high qualityDeveloper educationDone means Done
• Early intervention
JIRA, 2012
Development
Dev/QA Pairing Release
DoT
2 weeks
Room for Improvement
Development Dev/QA Pairing ReleaseDoT
I’m going to add a new action!
I bet you forget to handle session timeouts, again…
It’s ready for DoTing!
Hey DoT, please test session timeouts
Session timeouts are broken, please fix.
OK! Fixed now.
I’m going to add a new action!
Don’t forget session timeouts!
It’s ready for DoTing! I made sure session timeouts work.
SweetSweet.
Pre‐ to Post‐Dev Testing Notes
• Implementation choices affect risks– Need to add some testing hints– Need to remove some testing hints
• QA/Dev Pairing sessions still critical• Avoid duplication of testing
Goals
Scalability of QABetter usabilityConsistent high qualityDeveloper educationDone means DoneEarly intervention
A Welcome Surprise
• Empty post‐dev testing notes!
Development Dev/QA Pairing ReleaseDoT
Goals
Scalability of QABetter usabilityConsistent high qualityDeveloper educationDone means DoneEarly interventionUnicorns
JIRA, 2013
• 60 Developers • 4 QA Engineers
JIRA, 2013
Development Dev/QA Pairing Release
DoT
Usability
Pre‐devtesting notes
Usabilityfeedback
Post‐devtesting notes
My features don’t get rejected! I see results in weeks,
not months!
I get to make people happier, not angrier!
We get high‐quality features fast!
not angrier!
What else do we do?
What else do we do?
• Spot‐check features• Pick up stories from the DoTing backlog
What else do we do?
• Write testing tools• Set up automation frameworks• Create test data sets
What else do we do?
• ShipIt days– Quarterly programming competition– 24 hours to “ship” a feature– QA team works on product features or test tools– Many QA victories
What else do we do?
• Work with Support• Analyse bug history• Collect metrics
What else do we do?
• Speak at conferences ☺
Take‐Away Lessons
• Question your assumptions• Constraints encourage efficiency• Unreasonable goals can trigger great change• Don’t be afraid to reinvent yourselves• Teach people to fish, don’t hand them fish• Software QA is about people, not code
