test management in a changing world · • increasing public awareness of risks • power of social...

Test Management in a changing world

Iris PinksterBob van de Burgt

www.eurostarconferences.com

Agenda

• Introduction

• Agile & Scrum

• Lean & Six Sigma

• Focus on non-functionals

• Virtualisation

• Continuous….& DevOps



Something weird is happening …

• This year, there will be more than 10 times as many photos taken than in 1993, and yet Kodak is bankrupt

• More news is being generated on a daily basis than ever, and yet…– The Los Angeles Times is bankrupt

– The New York Times’ debt has junk status

– The New Orleans Times Picayune publishes just three days a week

• People are watching more movies at home than ever, yet Blockbuster is bankrupt

• Book sales increased 27% in 2011, and yet Borders collapsed


Widget winners Digit winners


A long time ago….

…there were only cowboys.

They were developing and testing.

Everything went well, until…


…systems got more complex,

and more structure was needed.

Methods were developed, training was given and development and test became two separate professions.

And that came with all possible problems.


In other words….


And then we got Agile…


Agility & Scrum

Agility*:1. The ability to rapidly and deliberately respond to changing demand, while

controlling risk.

2. Flexibility, the capacity and capability of rapidly and efficiently adapting.

Scrum*:A framework within which people can address complex problems, and productively and creatively deliver products of the highest possible value.

Scrum is a tool used to become Agile.* © 1993 – 2012 Scrum.org


Characteristics of an Agile method

• Increments/iterations

• ‘Reduce the waste’

• Continuous feedback


Content in Scrum

Roles

• Product Owner

• Development Team

• Scrum Master

Artefacts

• Increment

• Product Backlog

• Sprint Backlog

Events

• Sprint

• Sprint Planning

• Daily Scrum

• Sprint Review

• Retrospective

© 1993 – 2012 Scrum.org


Roles

• Optimizes value of the Product

• Manages the Product BacklogProduct Owner

• Manages itself

• Delivers Done Increments

Development Team

• Manages the Scrum Process

• Removes ImpedimentsScrum Master

© 1993 – 2012 Scrum.org


In which roles is test embedded?

• Group discussion – 10 minutes


Artefacts - Increment

Each increment must be:

• Done, with no work remaining

• Potentially usable by a customer

• Transparent – all viewers of the increment have the same understanding of it

© 1993 – 2012 Scrum.org


How is test part of an increment?



Product Backlog

• Product backlog holds the requirements for Scrum

• Transparent

• Inventory of “desirements”

• Ordered based on value, dependencies & risk

• Work “to do” is estimated

• A vehicle for starting conversations

© 1993 – 2012 Scrum.org


What test items are part of Product Backlog?



Agile is not easy to implement

• Multidisciplinary team

• Delivered products must have added value to the customer

• Continuous integration

• Team is working in one location


Case 1Building an aircraft


Wrap - up: Agile

Prepare: 10 minutes

What are the consequences of Agile:

• For a Tester

• For a Test Manager

• For a (Test) Organization


Agenda

• Introduction

• Agile & Scrum



• Virtualisation




• Increments/iterations

• ‘Reduce the waste’

• Continuous feedback

Characteristics of an Agile method


Lean & Six Sigma

• Six Sigma – Higher quality and smarter

• 4 key elements *) :

– Delight the customer

– Improve processes

– Teamwork

– Data and facts

• Toyota Production System –Quicker and more efficient (Lean)

• 7 wastes:

– Overproduction

– Waiting

– Transporting

– Inappropriate processing

– Unnecessary inventory

– Unnecessary / excess motion

– Defects

*) George, Mike, Rowlands, Dave, and Kastle, Bill. What is Lean Six Sigma? New York. 2004.


Lean versus Agile

Lean• Complete operation /

management

• Interaction between people

• Operation – stability

• Top-down implementation

Agile• Software development

• Interaction between people

• Innovation – flexibility

• Bottom-up implementation

• Has Lean principles

Maybe a combination works best !


Case 2

How can we make testing more Agile and more Lean?


Roadmap to Lean Test Management

DMAIC:• Define• Measure• Analyse• Improve• Control

Between every step is a tollgate


Tools for Lean Test Management

Define

• SIPOC

• Value Stream Map

Measure• Pareto chart

Analyse

• 5 Why’s

• Ishikawa diagram

Improve• Pick chart

Control• Control chart


Wrap - up: Lean Test Management

Prepare: 10 minutes

What is the added value of Lean Test Management:

• For Testers

• For Test Managers

• For (Test) Organization


Agenda

• Introduction

• Agile & Scrum



• Virtualisation




Shift in focus


Reasons for shift in focus

• More external presence of company systems (web sites & Apps)

• Increasing interactivity in web applications

• Higher change of attacks (external & internal)

• Protection of personal information

• Protection against espionage

• Increasing public awareness of risks

• Power of social media


Definitions

• A non-functional requirement is a requirement that specifies criteria that can be used to judge the operation of a system, rather than specific behaviours. The plan for implementing non-functional requirements is detailed in the system architecture.

• A functional requirement defines specific behaviour or functions. The plan for implementing functional requirements is detailed in the system design.


Case 3

• How can we organize testing of non-functionals in an Agile environment?


Wrap - up: non-functionals

• Out-of-the-box thinking

• Create awareness to get sponsorship

• Select the proper tools & environments

• Hire experts

• Change of mind-set

• Don’t just focus on requirements, but also on risks


Agenda

• Introduction

• Agile & Scrum



• Virtualisation




Different levels of virtualisation

• Server

• Desktop

• Application

• Service


Server Virtualisation

Hardware (CPU, Memory, Disk, etc)

Hypervisor (Hyper-V, Xen, ESX Server)

Application

Guest OS

Virtual Hardware

Application

Guest OS

Virtual Hardware

Application

Guest OS

Virtual Hardware


Desktop Virtualisation

Hardware (CPU, Memory, Disk, etc)Hypervisor (Hyper-V, Xen, ESX Server)

Application

Guest OS

Virtual Hardware

Application

Guest OS

Virtual Hardware

Application

Guest OS

Virtual Hardware


Application Virtualisation

Hardware (CPU, Memory, Disk, etc)Hypervisor (Hyper-V, Xen, ESX Server)

Application

Guest OS

Virtual Hardware

Application

Guest OS

Virtual Hardware

Application

Guest OS

Virtual HardwareApplication Server


Experiences with SCRUM & Application virtualisation


The Environment


The Users


The approach of the project

• General (conform Prince2)– Project manager

– Technical coordinator

– Test manager

• Applications (SCRUM)– SCRUM Master

– Product Owner (architect PZH)

– Developers (testers & sequencers)

CO

MB

I


The approach for applications

3 weeks

24

hours

Daily

SCRUM

meeting

Application

Backlog

Sprint

Application

Backlog

Applications

to be tested

in Acceptance

environment


Rules of the game

• Daily SCRUM (pick a time)

• Presence

• Team responsible for result

• Use the walls! (transparency)

• …..


Product Backlog

• 357 applications in scope (initially)

• Where to start?– First applications from the Office2010 project image

– Usage last three months

• Dependencies?– Middleware

– Cooperation with other front-end

applications

– Etc…

Naam Aantal

Intranet 6332

Idms 4480

Microsoft Outlook 2412

Microsoft Word 1954

Microsoft Internet Explorer 1874

Adobe Reader X 1676

Windows Verkenner 928

[unknown] 480

Sagitta Tijdregistratie 409

Beeldschermtachograaf 3 391


Sprint planning

• Product Owner explains PBI’s

• Development team decides what can be delivered

• Development team decides on the approach


The Scrum Board

• To do, script, test, done

• App-v, XenApp, Silo server, XenDesktop


The Scrum BoardTODO SCRIPT/

sequenceTEST DONE

APP-V

XenApp(local)

Silo Server

XenDesktop

= Application

= Task


Points of attention virtualisation (from POC)

• Hardware drivers

• 16-bits applications

• Hard-coded paths

• License security (MAC, server names, dongles)

• …etc.


Approach per application

• Look for pre-requisites in installation manuals

• Check which pre-requisites already in golden image– Adjust golden image when necessary

• Sequence

• Test in DEV

• Deploy in TEST

• Test in TEST

• Deploy in ACC


Testing

• Part of development team

• OTAP

• System test by development team using Test Charters

• Acceptance testing with:

• Functional managers

• User

• Issues management using JIRA


Definition of Done

• Installation guide

• Application

• Present in RES

• Test approved

• CMDB updated?

• Backlog updated

• Golden Image updated


Points of attention (1)


Points of attention (2)

• JAVA

• ActiveX

• Heavy Graphical applications (GIS and AutoCAD)

• Browsers

• Control local applications op thin client (extender)

• Screen resolutions

• Migration strategy i.r.t. thin client updates


Lessons learned

• SCRUM is working very well in situations where you just have to start….

• When the Acceptance environment is available on time include the acceptance test in the sprint

• Pay attention to the cooperation between the new and the old environment when using a phased migration strategy

• The role of test manager can be combined with the role of SCRUM Master very well


Different levels of virtualisation

• Server

• Desktop

• Application

• Service


Service Virtualisation


Case 4

• How can virtualisation be used within various test levels and test types?


Wrap-up: Virtualisation

• Rethink your test approach

• Plan for flexibility

• Practice controlled integration

• Test continuously from development to production

• Externalize your test data

• Explore advanced test scenarios

• Avoid reinventing the wheel

• Service virtualization isn’t just for testers

• Share virtual components across the enterprise

• Enhancing team productivity by building skills


Agenda

• Introduction

• Agile & Scrum



• Virtualisation




DevOps

• Short for Development and Operations

• Based on Agile and Lean principles

• Collaboration between business, development team and operations

• Deliver software in a continuous manner:

– Shorter time to market

– Quicker customer feedback

• DevOps is a business process and NOT an IT capability


DevOps principles

• Develop and test against production-like systems (“shift left”)

• Deploy with repeatable, reliable processes

• Monitor and validate operational quality

• Amplify feedback loops

Sdarchitect.wordpress.com


Will DevOps lead to higher quality?



Introducing DevOps

www.ibm.com


Develop and Test

• Merging development constantly

• Test code as often as possible

• Automated tests from Unit Test onwards

Continuous Integration

• Testing earlier and continuously in the SDLC

• Earlier and continuous feedback on quality

• By test automation and service virtualisation

Continuous Testing


What is necessary for Continuous Testing?

• Test environment management and configuration

• Test data management

• Integration, functional, performance, and security testing tools

• Service VirtualisationVideosnpictures.com


What is the consequence of introducing DevOps?

Prepare: 10 minutes

• For Testers & Test Managers

• For Test Environments

• For (Test) Organization


Continuous…

• Merging development constantly

• Test code as often as possible

• Automated tests from Unit Test onwards


• Moving ideas as quickly as possible to production

• Validate business assumptions quickly

• Constant flow of changes into production

Continuous Delivery

• Release code to Production when ready

• Needs Continuous Integration & delivery

Continuous Deployment


DevOps practices

• Release planning


• Continuous delivery

• Continuous testing

• Continuous monitoring and feedback


Increasing level of automation


Continuous Delivery

Continuous Deployment

Mo

re /

all

auto

mat

ion

For verification

try to automate as

much as possible and optimal as possible

Never forget to execute

manual testing for validation based on risk


Case 5

How can manual testing be incorporated in DevOps?


Wrap-up: DevOps

• Active stakeholder participation

• Automated testing

• Integrated configuration management

• Integrated change management


• Integrated deployment planning

• Continuous deployment

• Production support

• Application monitoring

• Automated dashboards


Conclusions

• Trends seem to focus on process in stead of tooling

• The test process is able to adjust to all new trends: maybe testing is the most Agile of all professions!

• The test organization will change

• The test profession will change

• The test profession will get more interesting (“Shift left”)

• Test is ready for the future


Questions

Contact:e: [email protected]: [email protected]

mailto:[email protected]

mailto:[email protected]

test management in a changing world · • increasing public awareness of risks • power of social...

Documents