testing within an agile environment - beyza sakir and chris gollop

37
Testing within an agile environment Chris Gollop & Beyza Sakir

Upload: jaxlondon2014

Post on 10-Aug-2015

157 views

Category:

Presentations & Public Speaking


0 download

TRANSCRIPT

Page 1: Testing within an Agile Environment - Beyza Sakir and Chris Gollop

Testing within an agile environmentChris Gollop & Beyza Sakir

Page 2: Testing within an Agile Environment - Beyza Sakir and Chris Gollop

Overview

History Changing Testing Concerns Adaptations Future Conclusion Further Reading

Page 3: Testing within an Agile Environment - Beyza Sakir and Chris Gollop

History – what we do

Release every 2 weeks Trade $billions/day Average 0.4ms execution time for MTF members 10,000 orders second 1 tester per 5-6 developers Very low rate of production issues

Page 4: Testing within an Agile Environment - Beyza Sakir and Chris Gollop

History - company

Conceived as an Agile company Agile has matured but we’re still adapting Agile testing:

Acceptance Tests Integration Tests Exploratory Testing

Page 5: Testing within an Agile Environment - Beyza Sakir and Chris Gollop

History - company

• This has allowed us to experiment and adapt:• Time freed up• Quick feedback on changes• Historic data available to analyse

Page 6: Testing within an Agile Environment - Beyza Sakir and Chris Gollop

History – test team

4 test specialists with different strengths complement each other as a 'team‘ Rotate through all teams• No Test Manager• Part of development team though test interests have a separate iteration

meeting• Testing is responsibility of everyone

Page 7: Testing within an Agile Environment - Beyza Sakir and Chris Gollop

History – team structure

l

Team One Team ThreeTeam Two

Test Specialists

Business Specialists

Development Specialists

Page 8: Testing within an Agile Environment - Beyza Sakir and Chris Gollop

History - innovate

Keep changing, even if something appears to work there may be something better

“If I had asked people what they wanted, they would have said faster horses. I gave them a car” (Henry Ford)

Page 9: Testing within an Agile Environment - Beyza Sakir and Chris Gollop

History – measure and feedback

Measure To know if it is effective Many monitoring and measuring tools

Feedback Tying feedback to ideas Quicker this is the faster you can confidently adapt Can be numerical or social

Page 10: Testing within an Agile Environment - Beyza Sakir and Chris Gollop

Changing Testing Concerns Over Time

Traditional Testing Maturity Models start at “ad-hoc” and move through structured testing before finishing at “Continuous Improvement”

Some of our test concerns (over time) Decrease Manual Testing & increase automated Acceptance Tests Performance DSL Speed of Feedback Increasing number of Edge Cases as system becomes more complex

Page 11: Testing within an Agile Environment - Beyza Sakir and Chris Gollop

Changing Testing Concerns Over Time

10,000 Behavioural Tests 8,700 Acceptance Tests (UI, API) 1,300 Integration Tests (subsets of services using internal interfaces)

Acceptance Test would test a rejection Integration Tests would test the different rejection types

Page 12: Testing within an Agile Environment - Beyza Sakir and Chris Gollop

Changing Testing Concerns Over Time

Was right at the time as needed Acceptance Tests to get good coverage quickly, trade-off is they are slower to run repeatedly.

Focus is now quicker feedback.

Page 13: Testing within an Agile Environment - Beyza Sakir and Chris Gollop

Adaptations – Fitness Landscape

Page 14: Testing within an Agile Environment - Beyza Sakir and Chris Gollop

Adaptations – Fitness Landscape

Each square is similar to it's neighbour Higher it is the better the solution – valleys are bad, mountaintops are good

You don't fully know where you are on the landscape You feel your way by trying out new ideas and recognising failure You may be half-way up the mountain rather than at the top And the mountains move!

Some peaks move faster than others: Google v McDonalds

Evolution isn't just biological it's everywhere

Page 15: Testing within an Agile Environment - Beyza Sakir and Chris Gollop

Adaptations – Palchinsky Principles

Peter Palchinsky was a Russian engineer who advised on (against) Lenin Dam and Magnitogorsk – large projects which essentially failed due to being too big and unable to adapt.

Palchinsky Principles are shaped to encourage stronger innovation, better leadership and more effective policies: Variation - seek out new ideas and try new ideas Survivability - when trying something new do it on a scale where failure is

survivable Selection - seek out feedback and learn from mistakes as you go along, avoid an

instinctive reaction of denial

Page 16: Testing within an Agile Environment - Beyza Sakir and Chris Gollop

Adaptations – Recognise failure quickly

Being able to recognize a failure means you can re-cast it into something more likely to succeed.

Accept something is failing, don’t deny or ignore it.

"Hard to see what changes would improve current process when they’re currently sweeping all problems under the ‘our context’ rug." (Anthony Green)

The essential first step is being willing to fail.

Page 17: Testing within an Agile Environment - Beyza Sakir and Chris Gollop

Adaptations - Google v Apple

Both considered as innovative companies.

Apple innovates in things that are core to its business Google innovates on things that indirectly contribute to its business

Apple releases products that are polished and well-scoped Google releases many different products (Buzz, Wave, Docs, AppEngine) early in

their development and lets the community develop and adapt them.

Apple create new mountains on the fitness landscape Google is surfing along on, or near, the moving fitness landscape mountain-top by

adapting its strategy and offerings.

Page 18: Testing within an Agile Environment - Beyza Sakir and Chris Gollop

Adaptations – a brief history of what we have tried

Page 19: Testing within an Agile Environment - Beyza Sakir and Chris Gollop

Adaptations – what worked

Production Data Testing in Live Big Feedback Remove Intermittent Tests Tester Pairing

Page 20: Testing within an Agile Environment - Beyza Sakir and Chris Gollop

Adaptations – Sanitised Production Data

Raw data (database) Patterns of use Data profile patterns

Migration (automated) Real world data Larger than we can create Many edge cases Try and reincorporate into our Acceptance Tests Performance Staging

Progression in what and how we use the data

Page 21: Testing within an Agile Environment - Beyza Sakir and Chris Gollop

Adaptations – Testing in Live

Uses same hardware and code as it is just an isolated venue Runs a subset of the Acceptance Tests

When we expanded our Exchange by adding a multi-tenanted concept known as a venue we took advantage for testing

Verifies that the various components of the exchange are hooked up and configured correctly and that the deployment process worked correctly

Runs automatically throughout the day as one of our monitoring systems

Page 22: Testing within an Agile Environment - Beyza Sakir and Chris Gollop

Adaptations – Big Feedback

Page 23: Testing within an Agile Environment - Beyza Sakir and Chris Gollop

Adaptations – Big Feedback

Important CI pipeline builds at the top Red (failed) items rise to the top Continuously and easily expanded Useful features survive

Failure ownership/failures at a glance - AutoTrish Intermittent tests - Magic Eight Ball Auto Embargo Performance metrics

Build sheriff to keep check-in embargo

Page 24: Testing within an Agile Environment - Beyza Sakir and Chris Gollop

Adaptations – Intermittent Tests

Tests that can fail even if feature is working Now have <1 intermittent tests per run (out of 8,200) Each about 1 test run in 20

Hides genuine failures Have previously missed a release as couldn't tell if genuine

Can result from Asynchronous messaging Tests changing global settings/lack of isolation End of day transitions Date/time

Page 25: Testing within an Agile Environment - Beyza Sakir and Chris Gollop

Adaptations – Intermittent Tests

anon

anon

Page 26: Testing within an Agile Environment - Beyza Sakir and Chris Gollop

Adaptations – Intermittent Tests

All Failures run twice Fail twice = failure Fail once, pass once = intermittent Accepted as something we need to deal with

Work to remove DSL improvement Fix a genuine issue (race condition) End of iteration is given over to removing intermittency Build Sheriff also works on this

Page 27: Testing within an Agile Environment - Beyza Sakir and Chris Gollop

Adaptations – Intermittent Tests

Page 28: Testing within an Agile Environment - Beyza Sakir and Chris Gollop

Adaptations – Intermittent Tests

Page 29: Testing within an Agile Environment - Beyza Sakir and Chris Gollop

Adaptations – Tester Pairing

Each tester has their own strengths

Noticed that at rotation handover the new tester generally found a bug Issue with the testers (denial) Something we can harness (acceptance)

Already do development pairing (Dev & Dev and Dev & Tester)

When do we pair? At the right time for the story

Initial brainstorming when creating a mindmap When we feel we need to (pull) When someone has time (push)

Page 30: Testing within an Agile Environment - Beyza Sakir and Chris Gollop

Adaptations – mindmap

Page 31: Testing within an Agile Environment - Beyza Sakir and Chris Gollop

Adaptations – what didn't work

Not many absolute failures

Although many reasons it can fail

Even if something fails you learn something

Natural selection will continue

You can't stay still, unless you are improving you are losing ground as others will innovate or adapt

Page 32: Testing within an Agile Environment - Beyza Sakir and Chris Gollop

Adaptations – facts

Of the top 25 industrial corporations in the United States in 1900, only two remained on that list at the start of the 1960s. And of the top 25 companies on the Fortune 500 in 1961, only six remain there today (IBM)

Average life expectancy of a Fortune 500 company has declined from around 75 years half a century ago to less than 15 years today, and heading towards 5 years (Deloitte)

Page 33: Testing within an Agile Environment - Beyza Sakir and Chris Gollop

Future

Expanding the way we test

Speed up feedback Increasing integration test coverage Pushing UI to API – testing at the right layer Change AT run order

Improving feedback/communication Test team standups

Documentation What is just enough How to get people up to speed in a complex system

Page 34: Testing within an Agile Environment - Beyza Sakir and Chris Gollop

Future

Is about the people

Team make-up is varied and we recruit to keep it this way

“’Testing is a strictly formal process that by its nature should not vary’ - this is a typical belief by people who know nothing about testing.” (James Bach).

Page 35: Testing within an Agile Environment - Beyza Sakir and Chris Gollop

Conclusion

Need to experiment and adapt to stay ahead

Even if something works it does not mean there isn't something better

Make changes on a scale where failure can be survived

Do not be afraid of failure but know how to recognise it quickly

Page 36: Testing within an Agile Environment - Beyza Sakir and Chris Gollop

Further Reading

• Tim Harford• Richard Wiseman• Malcolm Gladwell• Daniel Kahneman• Dan Ariely• Richard Thaler

• LMAX Staff blogs - http://blogs.lmax.com/staff-blogs/

Page 37: Testing within an Agile Environment - Beyza Sakir and Chris Gollop