agile san diego: testing as exploration (continuous delivery w/o automation)
TRANSCRIPT
Testing as Explorationaka. Continuous Delivery without
Automation
Maaret Pyhäjärvi
Email: <[email protected]> | Twitter: maaretp
Maaret Pyhäjärvi
Nimeä | Attribution (Finland)
http://creativecommons.org/licenses/by/1.0/fi/
http://creativecommons.org/licenses/by/1.0/fi/deed.en
Exploratory Testing:
Better tests, better testers! • An approach, not a technique
• Find unknown unknowns
• Disciplined
• Test is a performance, not artifact
– Artifacts support human
memory
– Many forms: e.g. checklists and
automation
• Exploratory performance testing,
Exploratory test automation,
Exploratory regression testingTest-related
learning
Design of new
tests
Test executionResult
interpretation
5
Exploratory Testing:
Learning & Modeling
”A day’s work”
Vision (“Sandbox”) Current Charter
Other Charters Details
Bug
Reports
Perception of
quality and
coverage
Quality
ReportDebriefing
Tester
Test
Manager
Past
Results
Obstacles
Outlook
Feelings
?
#
xCharter backlog of the future
testing
Out of
budget
Next in
importance!#, ?, x, +
20:20:60
Session sheets of the past testing
Idea of
exploration
Metrics
summary
Coachin
g
6
Playbooks
Coverage outlines
A Bit of Background
• 3 years into a web based (.NET/C#) product
• 1:8 tester:developer ratio
• Missing agile technical practices: test automation non-existent
• Scrum-like monthly releases for first 2 years– Releases late
– Releases not tested well enough
• Negative cycle of failing with estimates eating team morale
• Jira in a central role with estimates and burndown
The Main Driver for Change:
Testing
Scheduled release
Feature 1
Feature 2
Feature 3
Feature 4
Testing
Feature 1
Feature 2
Feature 3
Feature 4
R1 R2 R3
Done
includes
Tested
Schedule
over
Quality
SCRUM + SCHEDULED DELIVERY with continuous integration
KANBAN + CONTINUOUS DELIVERY
Enablers that Made Timing Just
Right
• Henri Karhatsu and #NoEstimates –
experience report
• Availability of Git in Visual Studio without
plugins
• Problems with schedules in Scrum
• Lean and value delivery focus in Product
Management
Changing How We Managed Our
Work: Setting up Jira Kanban
Removed hour &
story point estimates
Agreed in WIP for
each phase
Agreed on swarming
Updated issue types
Branches and Test
Environments
Feature Test Environment
Feature branches on-demand builds
“Developers think this can be tested”
Development Test
Environment
Integration branch build-on-checkin
“Developers think this can be released”
Acceptance Test
Environment
Master branch with fixes and merges from integration
on-demand builds
“Release next morning”
Continuous Deployment but
Significant Lead Times
Mon Tue Wed Thu Fri Mon Tue Wed Thu Fri
Change to
Master
Integration
to master
Feature to
integration
Developers drive the decision on what
they want feedback on
Exploratory Testing Enables
Continuous Deployment• Every Jira task gets planned for
– Sometimes we go with developer testing only
– Sometimes we test extensively
• Exploratory Planning– No set test cases
– Talking to developers and reflecting to a model of of use created through earlier explorations
– Actionable information first –principle
– Production monitoring is an option for getting information
Changes
• Active discussion about schedules and merging, and needs of testing in the branches
• More pairing for testing the features
• More group work on defining the features
• Introducing Flowdock due to increased need to collaborate; integrating logs, emails
• Focus on getting better (scope of test automation; refactoring; pairing and group work; individuals’ skills)