Post on 01-Dec-2014
Embed Size (px)
- 1. Agile Testing Challengesbret@pettichord.comwww.pettichord.comwww.thoughtworks.com Pacific Northwest Software Quality Conference, 13 October 2004 Copyright 2004 Bret Pettichord. Copying permitted with attribution.
2. Agenda What is Agile Development? Challenges for Testers Strategies for Dealing with Them Agile Testing Tools 3. Context-Driven Testing Principles The value of a practice depends on context. Practices may be good in context, but there are no best practices. People, working together, are the most important part of any projects context. Projects unfold are often not predictable. The product is a solution. If the problem isnt solved, the product doesnt work. Good software testing is a challenging intellectual process. Only through judgment and skill, exercised cooperatively throughout the entire project, are we able to do the right things at the right times to effectively test our products.Context-driven testing is biased towards rapid feedbackand adaptabilitywww.context-driven-testing.com 4. What is Agile Development? Incremental, Iterative, Adaptive Incremental Build a system gradually See the system grow Demonstrating progress Iterative Multiple releases or check points during a project, each closer to the target Iterations include requirements development and testing Typical iterations are two weeks long Plan as you go Adaptive Goals change based on lessons from prior iterations, feedback and business opportunities 5. What is Agile Development? Regularly delivers business value Work broken down into stories Sometimes called features or use-cases Consists of multiple tasks, often assigned to differentprogrammers Has defined acceptance criteria Not done until the unit and acceptance tests pass 6. What is Agile Development? Collaborative Pairing, with changing pairs Avoids specialized knowledge Team code ownership (optimistic locking) No Backsliding Continuous Integration Unit Testing (usually using test-driven development) Constant regression testing 7. The Old Strategies Wont Work Detailed test planning Not enough time Too many changes Dedicating a phase for testing You dont get to define entry and exit criteria Iterations will move on without you Change control Changes are now commonplace Being in a position of authority You need to learn how to express concerns without reallybeing able to judge software correctness authoritatively 8. What are Testers Good for Anyway? In business, any worthwhile function either provides productsor services. What product or service does testing provide? Tested, Debugged Software Wrong Answer Testing providesinformation about the status of software under development to inform decisions. 9. Challenge: Are Testers Obsolete? With developers doing unit testing, do we still need to have QA testers? Some teams have fired testers when adopting agile. They have then regretted this. Testing may be done by people who arent called QA or Tester: e.g. Business Analysts. Some teams are using developers for acceptance testing. Dedicated testers bring two benefits: Focus on customer usage over technical implementation Focus on uncovering flaws over confirming completeness 10. Challenge: Testing Half-Baked Code With frequent iterations delivered to testing, how can testers test incomplete code? Stories must be defined in terms of business value. Stories should not span iterations. Good story definition is difficult to do. Clumsy story definition often impacts testers more than developers. Testers may need to help with story definition (i.e. become analysts). Dont test the stories, just help get them right. There will always be gaps. Testers need to learn and adapt as they find them. 11. Challenge: Arent Story Acceptance Tests Simplistic? How can it be good testing if your acceptance tests are only verifying that a story is complete? Isnt this happy-path testing? Each iteration requires additional tests other than those that simply focus on accepting completion of a story. 12. Iteration Testing Dev 1 23Test12 At the end of an iteration, promote code for further testing: Exploratory Testing Endurance Testing Learn, plan and execute at once Execute software over long Look for bugs, missing features periods of time and opportunities for improvement Combination/Interaction Testing Business Cycle Testing Focus on interactions between Execute scenarios based on end- featuresof-day, end-of-month, end-of- Scenario Testing quarter and other business cycles Use real-world scenarios that Load Testing exercise multiple stories Replicate a full load on the system 13. Challenge: Getting Testers to be Part of the Team How can we get testers to be part of a team? Doesnt this force them to sacrifice their integrity? Testers have been an oppressed group and have often stuck together to provide mutual support. Its time to let go of this. Testers should co-locate with developers and analysts. Agile only works when there is lots of both formal and informal communication within a team. 14. Challenge: When is Testing Done? Without the time devoted to complete testing, how do we know when testing is done? Useful testing addresses important product risks. Agile testers need to be able to justify tests in terms of risk. What risks would these tests inform? Ultimately testing must be prioritized just like stories. Bug metrics also provide an indicator of completeness. A good tester is never done. 15. Pairing Testers with Developers Why? Facilitate Grey Box Testing Developers gain insight into Understand relationships betweenpotential errorsparts of the system Testers gain insight into Analyze impact of changesconstraints and opportunities What needs retesting? Together can succeed with What can be left alone?automated testing Understand bugs Automate Acceptance Testing What were the root causes? Where did the problems surface? Write acceptance tests using thesame programming environment Understand riskused for development Develop test strategy that targetsrisk Reuse unit testing frameworks Justify and articulate testing Make the software more testable objectives Learn to diagnose bugs Identify source of errors 16. Pairing Testers with Analysts Why? Testers need to understand the business There are always hidden requirements Analysts need to understand how to test their requirements Involve testers earlier Defining requirements is difficult Often harder to specify requirements than design Easy to become out of date Specify by Example Defining them by example is incremental, easier and more concrete Acceptance tests Ground concepts Define goals and expectations Demonstrate progress Drive development Good acceptance tests are: Clear so any one can understand Specific so it can be executed 17. Challenge: Dont We Need Bug Tracking? Some have declared that agile teams shouldnttrack bugs and just fix them as soon as they arefound. This works well when you are testing in Dev. When you are testing a completed iteration in a Testenvironment, youll need to track bugs because youwont see fixes for a while (even if they are fixed rightaway). Ideally, bugs will be fixed right away, but some may bedeferred. Ultimately, bugs can be prioritized with stories. 18. Challenge: Collecting Useful Metrics What are useful quality metrics for agile projects? One of the best quality metrics is the number of bugs that escape development and are found after deployment. Unfortunately, this is a trailing indicator. Counting escapes by iteration can give a leading indicator of how good the code is. We can also learn from bugs in other ways: Are there unit tests to catch the kinds of problems that arebeing found by acceptance tests? Add them. Can we make bugs easier to find and diagnose? Can we make it so that programmers are less likely tomake common mistakes? 19. Challenge: Regression Testing With frequent iterations, we need to retest often. And unit tests arent enough. How do we get effective user-level regression tests? You dont necessarily need to do a full regression test with each iteration. You may run parts in each iteration cycling through. But youll still need some level of automated user-level regression tests. 20. Challenge: Regression Test Tools Most commercial test tools work poorly in an agile environment. Most have these flaws: Vendor-specific languages (vendorscripts) Poor integration with source control Hard to use with continuous integration Impractical to install on every workstation These problems make them impractical for use by the team as a whole. Agile teams are building their own test tools and releasing many of them as open-source 21. Problems with Commercial Test Tools Proprietary Scripting Languages Winrunner (TSL), SilkTest (4test), Robot (Test Basic) http://www.stickyminds.com/se/S2326.asp But newer tools are now using standard languages Astra QuickTest (VB Script), XDE Tester (Java), Incompatibility with Source Control Temporary files and directories (WinRunner) http://paulhammant.com/blog/000245.html Key information stored in repositories (Rational) Lack of External Calling APIs They refuse to allow themselves to be used as a library. Generally, you can only launch complete scripts with limited access toresults information. Therefore difficult to integrate with Continuous Integration Some new low-cost and shareware tools are exceptions E.g. TestComplete Restrictive and Expensive Licensing Developers cant run test suites. These features encourage vendor-lock and frustrate seriousprogramming Open-Source Tools almost always avoid these shortcomings. 22. Open-Source Tools for Acceptance Testing Many tools for Windows, Java and Web GUIs Today well focus on the Web HttpUnit and Other Protocol Drivers WTR and other COM/DOM Drivers 23. Web Protocol Drivers for Functional Testing Tool Tests HttpUnit, popularJava Java http://www.httpunit.org/jWebU