test automation is no longer optional. automated exploratory tests or structured automated tests

23
© 2010 LogiGear Corporation. All Rights Reserved Test Automation is No Longer Optional. Automated Exploratory Tests or Structured Automated Tests Which One is Better and Can You Do Both? June 16 th , 2010 – San Mateo, CA Cem Kaner - Hung Q. Nguyen Moderated by Michael Hackett Large-scale Test Automation Production

Upload: essien

Post on 11-Feb-2016

51 views

Category:

Documents


0 download

DESCRIPTION

Large-scale Test Automation Production. Test Automation is No Longer Optional. Automated Exploratory Tests or Structured Automated Tests Which One is Better and Can You Do Both? June 16 th , 2010 – San Mateo, CA Cem Kaner - Hung Q. Nguyen Moderated by Michael Hackett. - PowerPoint PPT Presentation

TRANSCRIPT

Page 1: Test Automation is No Longer Optional. Automated Exploratory Tests or Structured Automated Tests

© 2010 LogiGear Corporation. All Rights Reserved

Test Automation is No Longer Optional.

Automated Exploratory Tests or Structured Automated Tests

Which One is Better and Can You Do Both?

June 16th, 2010 – San Mateo, CA

Cem Kaner - Hung Q. Nguyen

Moderated by Michael Hackett

Large-scale Test Automation Production

Page 2: Test Automation is No Longer Optional. Automated Exploratory Tests or Structured Automated Tests

© 2010 LogiGear Corporation. All Rights Reserved

Common Challenges

• Automation Technology– Multiplatform (cross-platform) support

• Ability to interface with multiple platforms• Ease of extensibility to new platforms• Reusability of test suites/scripts across multiple

platforms

– Output/Object validation support• High rate-of-change

– Interface changes– Functionality changes

Page 3: Test Automation is No Longer Optional. Automated Exploratory Tests or Structured Automated Tests

© 2010 LogiGear Corporation. All Rights Reserved

Automation Cost of Ownership

1. Technology = Tool Licensing + Development/Customization Cost

2. Production Cost = *One-time Cost + **Recurring Cost*NEW–One-time Cost of Design, Implementation & Execution

– Test Case or Test Script Creation » Test line creation » Sequencing test logic or order of test lines (test steps) » Testing and debugging test script

– Function or Keyword Creation – Interface Capturing or Mapping – Test Dataset Creation

**EXISTING—Recurring Cost Test Execution & Maintenance Cost – Monitoring test suit execution – Investigating and troubleshooting false negatives – Maintaining Test Case or Test Script

» Test line modification » Updating test logic or re-sequencing order of test lines (test steps) » Testing and debugging updated test script » Function or Keyword Modification » Interface Recapturing or Remapping » Test Dataset Modification

Page 4: Test Automation is No Longer Optional. Automated Exploratory Tests or Structured Automated Tests

© 2010 LogiGear Corporation. All Rights Reserved

Large-scale Automationis Method Dependent

Automation is Method-Centric—it is more than about the technology; it is about the effective application of technology– Maintainability—handling high rate-of-change– Reusability—ability to reuse common test

components without having to program/code– Scalability—ability to automate high volume of tests

through reusability (and team-based cost efficiency)– Visibility

• Tests auditable by management and non-programmer staff• Productivity is structurally measurable

Page 5: Test Automation is No Longer Optional. Automated Exploratory Tests or Structured Automated Tests

© 2010 LogiGear Corporation. All Rights Reserved

Efficiency is Key!Efficiency Key Contributors• Test design

Definition:

A test (case) creation activity with intent to serve one or both of the following objectives:

1. Exposing bugs/errors (or show otherwise, it works as intended), and

2. Optimizing maintainability and scalability (especially for automation-readiness).

• Test automation (large-scale)– Method-centric, not tool-centric– Team-based, not individual

coders

Page 6: Test Automation is No Longer Optional. Automated Exploratory Tests or Structured Automated Tests

© 2010 LogiGear Corporation. All Rights Reserved

Rules of Automation Built-to-Last

1. Tests are treated as product asset, along with the source code.

2. Tests, good or bad are dependent on the design.3. Tests, manual or automated must be optimized for

visibility, reusability, scalability and maintainability.4. Tests must be automation-ready.5. Tests, if they are worth automating, should follow the

5% rule:· No more than 5% of all tests should be executed manually· No more than 5% of all efforts around testing should involve

automating the tests· No more than 5% of coded test scripts against non-coded

test scripts.

Page 7: Test Automation is No Longer Optional. Automated Exploratory Tests or Structured Automated Tests

© 2010 LogiGear Corporation. All Rights Reserved

Automation Strategy Must Include

• Efficient Practices– Technology

• High extensibility• Object validation capability

– Methodology• High maintainability (low maintenance cost)• High reusability

• People and process– High scalability (high-volume and team-based methodology)– Just-in-time automation—test engineers are skilled at test

design and automation can be done before code-complete and/or concurrent to exploratory testing

– High manageability (high visibility)

Page 8: Test Automation is No Longer Optional. Automated Exploratory Tests or Structured Automated Tests

© 2010 LogiGear Corporation. All Rights Reserved

"Agile" System Development

• Establish objectives (global)• Develop tests• Develop the system

– in iterations, one subsystem at the time– test-driven

• Release

Agile “Home Ground*”• Low mission-critical• Senior developers• Requirements change often• Small number of developers• Culture that thrives on chaos

* Barry Boehm and Richard Turner Balancing Agility and Discipline: A Guide for the Perplexed

Addison-Wesley, 2004

Page 9: Test Automation is No Longer Optional. Automated Exploratory Tests or Structured Automated Tests

© 2010 LogiGear Corporation. All Rights Reserved

Role of Automation

• Effective automation can allow tests to support agile system development

• Automation should not dominate. Don't make it into an "agile automation" project

• A keyword driven method is, in my view, essential for successful test development w/ automation in agile development– Automation is separate from test development– Usually doesn’t need user interaction (only some with testers)– Automation focuses on actions, not on tests

Page 10: Test Automation is No Longer Optional. Automated Exploratory Tests or Structured Automated Tests

© 2010 LogiGear Corporation. All Rights Reserved

Agile Test Development

Developers Stakeholders

SYSTEM DEVELOPMENT

Testers Stakeholders

TEST DEVELOPMENT

Tests can drive the development

AUTOMATION

Page 11: Test Automation is No Longer Optional. Automated Exploratory Tests or Structured Automated Tests

© 2010 LogiGear Corporation. All Rights Reserved

Recommendations• Have a global test-design scheme

– Keep it short and simple: focused on breakdown of the tests– This global test design can (and usually should) be revisited throughout

the project cycle• Separate (1) test objectives and (2) test cases

– Test objectives are easier to create/describe than test cases• Use your stakeholders (specialists, power-users, etc.) wisely

– Focus the efforts on the relevant tests– Don’t bother them more than needed

• Avoid involving them in low- and medium-level tests– Observe various types of input:

• How does this work?• What do we need to test?• What is interesting to test/How can we break the system?

• Keep automation separate– Test developers are not consumed by automation– Use keyword-driven approach (e.g., Action Based Testing)

Page 12: Test Automation is No Longer Optional. Automated Exploratory Tests or Structured Automated Tests

© 2010 LogiGear Corporation. All Rights Reserved

EAS1 EAS2 EAS3 EAS4 EASx 3rd-Party Tools

Device x

Low-level / System Action

High-level / Application

Action

FR

AM

EW

OR

K A

RC

HIT

EC

TU

RE

PE

OP

LE &

PR

OC

ES

S

Test Layer

AutomationLayer

• Communication protocol• Interface/Object recognition• Object content validation

Extensibility Integratability

ProductionLayer

SUT /AUT

EAS: Extended Automation Support

Page 13: Test Automation is No Longer Optional. Automated Exploratory Tests or Structured Automated Tests

© 2010 LogiGear Corporation. All Rights Reserved

Scalability Illustration

3000

200

20

6000

250

22

TestsApplication (High-level)

Actions

Built-in/System(Low-level)

Actions

Page 14: Test Automation is No Longer Optional. Automated Exploratory Tests or Structured Automated Tests

© 2010 LogiGear Corporation. All Rights Reserved

Approach

Workload Allocation

Automation Engineering

Test Case Production

Test Data Construction /

Selection

Record & Playback

N/A Test Engineer N/A

Scripting Automation Engineer

Automation Engineer N/A

Data-driven Automation Engineer

Automation Engineer Test Engineer

Action-driven Automation Engineer Test Engineer Test Engineer

Note: Programming skill is required for Automation Engineer

Division of Work

Page 15: Test Automation is No Longer Optional. Automated Exploratory Tests or Structured Automated Tests

© 2010 LogiGear Corporation. All Rights Reserved

Key Takeaways

1. Fully understand automation cost-of-ownership

2. Don’t underestimate the challenge of keeping maintenance costs low

3. You need to get efficient—optimize your volume of test to exceed 50% coverage

4. Efficiency is key, and it will come from excellent test design and automation methodology (e.g., action-driven), and a well architected framework technology

5. Minimize programming tests

6. High scalability comes from high reusability of common “actions” and team-based staffing model

7. High maintainability comes from keeping maintenance activities at the lowest level

8. High visibility in your automation program to give you control and measurability, which ultimately leads to manageability

9. Practice just-in-time automation

10. Have a global test-design scheme that separate test objectives from test cases; and automation from test development

Page 16: Test Automation is No Longer Optional. Automated Exploratory Tests or Structured Automated Tests

© 2010 LogiGear Corporation. All Rights Reserved

Appendix

Action Based Testing example

Page 17: Test Automation is No Longer Optional. Automated Exploratory Tests or Structured Automated Tests

© 2010 LogiGear Corporation. All Rights Reserved

Example withTestArchitect™ (TA)

TA

ClientTA

AgentTCP/IP

WirelessMobile API

Page 18: Test Automation is No Longer Optional. Automated Exploratory Tests or Structured Automated Tests

© 2010 LogiGear Corporation. All Rights Reserved

Test Example

Page 19: Test Automation is No Longer Optional. Automated Exploratory Tests or Structured Automated Tests

© 2010 LogiGear Corporation. All Rights Reserved

Action Definition for “send sms message and check” action

Page 20: Test Automation is No Longer Optional. Automated Exploratory Tests or Structured Automated Tests

© 2010 LogiGear Corporation. All Rights Reserved

TA Connects to Window Mobile via TCP/IP

TATCP/IP

Wireless

Page 21: Test Automation is No Longer Optional. Automated Exploratory Tests or Structured Automated Tests

© 2010 LogiGear Corporation. All Rights Reserved

Mobile API

Agent Connects to Device via Mobile API

Page 22: Test Automation is No Longer Optional. Automated Exploratory Tests or Structured Automated Tests

© 2010 LogiGear Corporation. All Rights Reserved

Interface Definition for “message editor” Entity

Page 23: Test Automation is No Longer Optional. Automated Exploratory Tests or Structured Automated Tests

© 2010 LogiGear Corporation. All Rights Reserved

About LogiGear Corporation

LogiGear is a world leader in high-volume software test automation production, serving hundreds of companies, from Fortune 100 to startups.

Draw from 15+ years of research and practice, 5 books, including one bestseller, 18 pragmatic training courses, a proven automation methodology and a software framework that elevates automation test coverage to exceed 95% on a diverse set of technology platforms, and 400+ skilled engineers in the US and Vietnam, it delivers unbeatable value that precisely meets each customer’s specific needs.

[email protected]