testing, mathias olausson

QWise software engineering – refactored! www.qwise.se

Testing, testingA first-look at the new testing

capabilities in Visual Studio 2010

Mathias OlaussonCallista Knowledgebase | QWise


About Mathias...


About the Presentation...

• What’s on the menu?– Test planning and execution– Issue tracking with rich bugs – Eliminating the "works on my machine / cannot

reproduce" scenario– Automated UI Testing– Virtual Lab Management


TEST PLANNING AND EXECUTION


Agile Testing


Core Practices


Agile Testing Quadrants

Functional Tests Exploratory Tests

Unit TestsComponent Tests

Performance TestsLoad Tests

Security Tests”-ilitets” Tests

Business-facing

Technology-facing

Supp

ortin

g th

e te

amCritisizing the product

Automated &Manual

Manual

AutomatedAutomated


Test TypesManual Tests

UI Tests

Acceptance Tests

Unit Tests / Component Tests


Testing in an Iteration

sprint plan

sprint plan

write tests for us1

write tests for us1

write tests for us2

write tests for us2

sprint plan

sprint plan

build 1build 1 build 2build 2 build 3build 3

implement user story ‘1’

(us1)

implement user story ‘1’

(us1)

test us1 & file bugs


Implement US2

Implement US2

build 4build 4

fix bugsfix bugs

build 6build 6 build 7build 7

regress impacted

tests

regress impacted

tests

build 5build 5

implement us2

fix bugsfix bugs



verify fixesverify fixes

verify fixesverify fixes


Microsoft Test Manager and Visual Studio

Team Foundation Server Reporting

Development (Teams)Testing (Teams)

Web Test

Load Test

Coded UI Test Unit Testing

Test Case Management

Lab Management

Microsoft Test Runner

Fast Forward For Manual Testing

GeneralistSpecialistEngineering Support

Data Collectors (IntelliTrace, Test Impact, Metrics)


The Fundamentals


Defining and Running Tests


ISSUE TRACKING WITH RICH BUGS


Should We Do Bug Tracking?

Pros– Tracability– Priority of work– Visibility

Cons– Overhead

Should we log bugs within an iteration?Source: Pixar ”A bugs’ life”


Defect Tracking in MTM and MTR


No-more-no-repro

• Automatic data collection

• Test settings– Configuration of

diagnostic data collectors

– Assigned to test plan• Or per test run


Diagnostic Data Adapters


No-more-no-repro


AUTOMATED UI TESTING


Why Automated Testing?

Källa: Tools for Agility, Kent Beck


iteration 1iteration 1 iteration 2iteration 2 release iterationrelease iteration

feature afeature a feature bfeature b feature cfeature c feature dfeature d

unit testing

manual testing

regression testing

performance testing

feature afeature a

Automation Strategy


Coded UI Tests

• Unit test for UI• High-level framework• Create test from– Recording– Test result (action

log)– Using API


Automation Platform Support


Coded UI Tests


LAB MANAGEMENT


Motivation


Lab Management Aims To...


Lab Management Infrastructure

Team Foundation Server (TFS)

Lab Management

Test Case management

Build management

Work Item Tracking

Source Control

Hyper-V HostsVirtual

Machine1

Virtual Machine2

Virtual Machinen

Library SharesLibrary Share1

Library Share2

Library Sharen

System Center Virtual Machine Manager (SCVMM)

Team Foundation Server (TFS)

Lab Management

Test Case management

Build management

Work Item Tracking

Source Control

Hyper-V HostsVirtual

Machine1

Virtual Machine2

Virtual Machinen

Library SharesLibrary Share1

Library Share2

Library Sharen

System Center Virtual Machine Manager (SCVMM)


Environments

• An environment is a set of machines with different roles

• An environment can– Be deployed, saved, removed– Started, stopped– Snapsshot

VM1

Lab Agent

VM2

Lab Agent


Manual Tests in Lab

• Testing from the outside– Easy for web applications– Limitations in data collection

• Testing from the inside– Run application in lab via environment viewer


Automatic Tests in Lab

• Testing in VM– Test case connected to test environment– Test component comes from selected build

• Testing in build– Test case connected to build definition– Runs test list or test component


Lab Management Build Workflow


Build and Deploy


Summary

• Structured test planning with MTM• Integrated test execution• Effective defect tracking with rich bugs• Seamless integration with automated tests• Powerful lab management


Questions?

[email protected]://msmvps.com/blogs/molausson