realistic test process improvement ... - eurostar conference€¦ · the case study organisation...
TRANSCRIPT
Realistic Test Process Improvement by Professionals:
A Case Study
Brian Wells,Experimentus,
UKW14
Realistic Test Process Improvement by Professionals: a Case Study
Brian WellsExperimentus
UK
www.experimentus.com
Session W14EuroSTAR 2007 5 Dec 2007
What will we cover?
Test Process Improvement – what are the issues!
The case study organisation: today’s example!
The (recent) Past
Current Improvement activities
The main “Enablers”
Professional resources as well as a Test Profession!
Communication, culture change & skills/knowledge
acquisition
Measurement!
Test Process Improvement Programme Customer Survey
How many of you have been part of process improvement
programmes?
How many programmes have said they will do things so much
better?
And save lots of money/effort?
How many of you have seen PREDICTED, DEMONSTRABLE,
CONTINUING savings/improvements?
Why do Test Process Improvement Initiatives Fail…
Usually becomes “shelf life” after a short
period as people do not understand why and what
the benefits are
Senior Mgt fully supports change but difficult at
operational level
Lack of clear direction AND acceptance
/understanding of the changesat all levels
Who really understandstest
Management?
Inconsistent test skills within organisation
Lack of Acceptance of testing
as a profession and a formal discipline
Process Improvement often driven by local “heroes”
Test Process Improvement Initiatives Fail……
Insufficient effort to ensure the right professional resources are
placed in a structured profession
Full Communication, awareness, training, mentoring and monitoring
of implementation
Inaccurate monitoring, control and measurement of benefits
accruing to all levels
Less than effective, efficient and continuing change
Retail IT – HBOS
A case study
The Case Study Organisation
Major UK Banking & Financial institution
4 Business Divisions; Retail Banking the division under the spotlight
today with offices in Leeds, Halifax and Cardiff
Retail IT covers all technology deliverables (mix of on and off shore
resources for development and increasingly test)
Central Test Authority with ownership of test processes etc within
Business Division
Head of Central Testing plus 4 permanent resources from different
backgrounds
Owns the test methodology and standards
Improvement driven by need to be smarter, more efficient and reduce
costs while improving quality!
Retail IT, HBOS – The Starting Point
Formal TPI® assessment undertaken in 2004/2005
Results and the input from Retail IT defined objectives for 2-
year Test Improvement project
Central Test Authority managed implementation of specific
activities to:
Standardise Reporting
Implement Test Metrics Programme
Implement Career Paths & Test Organisation
Undertake Test Life Cycle Improvements
Improve Test Data Management and make visible
Consolidate Test Tools Inventory & Best Practice
Overall objective was to ensure Test is <=40% of total project
What Progress?
Commissioned a TMM Assessment in Q3-Q4, 2006
Evaluated competitive TMMi® and TPI® bids to undertake
appraisal of Test and Test-related Quality Process Maturity
TMMi appraisal to Levels 2 & 3 demonstrated best combination of:
Professional, Industry recognised, Assessment approach
Speed (of appraisal and elapsed time)
Breadth (of appraisal across organisation)
Alignment to Organisation going forward
Transparent to all
Structured delivery of Findings and recommended, prioritised
Process Improvement Plan
TMMi Assessment Objectives
Objectives of the TMMi Assessment exercise were to:
Have the 2-year project process improvements delivered as
defined?
Accurately and succinctly measure current Test Process capability
Validate priority of their existing planned improvement activities
Identify and prioritise any gaps
Identify what's been achieved, what now needed to be done, and
why
In other words…..
Doing the right things?
Doing them in the right way?
The Results – The original 2-year Project Objectives
General feeling was that they had succeeded
TMM Appraisal demonstrated that most 2-year Project
deliverables had been broadly achieved except:
Test Automation (and associated Regression Test Guidance)
Standard approach to risk-based testing
Real progress on alleviating issues around test environments
& data
Estimate 75% of Project deliverables implemented
BUT BASE ACCURATE MEASUREMENT OF ACHIEVEMENT OF
BENEFITS IN INFANCY!
The Results – TMMi Assessment
Appraisal findings indicate achievement of approximately 70%
for level 2 and 75% for level 3
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
2.1-T
DGP
2.2-T
PE
2.3-T
TM
3.1-T
O
3.2-T
TP
3.3_TIS
DLC
3.4-C
MTP
KPA Maturity Achievement
Partially/Not Deployed
Mostly Deployed
Actively Performed
What was changed
Approach to Implementation
Change of emphasis - Central Test Authority manage and
control all change - Business as Usual now
Identified 68 specific, detailed improvement opportunities
grouped into strategic Improvement Work Packets
Improvement Work Packet delivery process defined as:
Definition of Process, Procedures, Templates
Structured Reviews andWalkthroughs
Defined specific training modules
Run pilots and evaluate Phased implementation
Strategic Work Packet 1 - Stronger Policies & Framework
Aligned with TMM
Clear mandatory direction from top
Clear Exemption procedures
Consolidated test framework elements by:
Restructuring Intranet Portal
Defining clear integration into Test and
delivery Life Cycles
Ensuring it is easy for all to understand ,
access and use!
E2E Test Life Cycle with clear Test Quality
Gates
Strategic Work Packet 2 - Test Deliverable Matrix
Test Deliverables Matrix
List of Testware with Owners, Reviewers and Informed roles
Covers entire Test Life Cycle activities
Mandatory and Optional deliverables linked to Test Life Cycle, project
size etc with integrated project controls
Linked to Test Quality Gates, Structured Reviews
Benefits accruing from these three Work Packets are:
Clarity on mandatory, standard, integrated framework for all Business
Areas – removed confusion
Easy to understand, follow and use – streamlined effort and removed
duplication
Critical enablers to realise tangible benefits
Strategic Work Packet 3 - Structured Reviews Procedures
Fully defined Reviews Procedures and Templates which:
Covers informal and structured reviews
Gathers critical metrics data
Includes full training programme
Fault Distribution Curve
0
50
100
150
Test Stages
Faults
Current
Improving
Target
Benefits are:
Improving early fault
detection and removal!
Reduction in the cost of
fixing faults in execution
Strategic Work Packet 4 - Risk Based Testing
Risk Analysis
0
39
78
0 39 78
Liklihood
Imp
act
SMS Server
Interface 1
Database1
Web Server 1
Data Archive 1
Telephony S1
Broadcast 1
Device 1
Implemented robust approach to Product Risk
using a Boston Square approach
Benefits are:
Identification of areas of high risk
early and first
Critical fault detection and resolution
earlier
In parallel, reducing test effort overall
“down stream”
Greatly improved communication and
understanding between all roles on
project
Strategic Work Packet 5 - Measurement Programme
Defined and published detailed Test & Debugging goals
Standardised test management and fault tracking structure, work flow
Ensured common reporting and analysis at test activity/project level
Implemented formal Test Quality Gates and enhanced, minimum
mandatory Entry/Exit criteria
Formalised Central Test Authority Process Audit/Quality Assurance
responsibilities within programme
Improvement activities capable of being measured!
Measurement Objectives & Benefits
Approved Test & Debugging Goals provide specific objectives/targets to:
To improve Test Efficiency & Effectiveness
Measure accruing, anticipated benefits of process improvements
Accruing benefits are:
Developing accurate picture of where they are today
Various elements already leading to more efficient and effective planning
through to implementation
Identifying additional opportunities for improvement already
High % of faults Rejected as “Not a Fault”
High % of faults Re-Opened (i.e. not fixed 1st, 2nd and more times)
Large test effort finding few faults
Test Process Improvement by Professional Testers!
Profession versus Professional?
“Profession” defined as “a paid occupation, especially one
involving training and a formal qualification”
“Professional” defined as “worthy of or appropriate to a
professional person; competent” – “a person having impressive
competence in a particular activity”
The main enabler identified was (radically) to:
Create and embed a Test Profession career structure
AND
Use these dedicated, increasingly skilled and professional test
resources to be instrumental in ensuring robust, long term
implementation of test process initiatives by Professionals
The Rationale
A Profession can be created; becoming a Professional takes time!
Organisational culture change is easy to impose from above but hard to
get working operationally
To address these, the key enabler identified to “plant” key test
professional in every business area:
Single, qualified point of contact: easy communication and manage
culture change effectively
Effective link between Method and practise
Ensure test integration with all other elements of SDLC
Streamline implementation and early, continuous benefits process
improvement initiatives
The Progressive Approach
Test career structure institutionalised within HR
ISTQB Foundation Qualifications as part of minimum Test skill set
Area Test Champions appointed and Testers Forum active
Pressure for Area Test Champions to be Central Test Authority
“lieutenants”
Responsibility for operational implementation of all aspects of
testing – including change
Test resources maturing skill set and reputation/respect
Enhanced training capability linked to Test Process Improvement Work
Packets
Organisation actively accept and embrace dedicated test professionals
AND CHANGE
The Delivery Stream Test Managers
Intended to enhance Test Organisation capability
Highly skilled: both in Test and “soft”
Primary responsibilities are to:
“Own” Testing within Delivery Stream
Provide Delivery Stream Test Management
and communication
Ensures early and continuous test
engagement
Other benefits being realised
Greatly improving communication, awareness and acceptance
of all things test!
Monitoring and enhancing maturing test skills and capabilities
Represents and promotes Professional testers and testing
Facilitates internal, standard processes implemented in
controlled and robust way
Culture change much reduced “blocker”
Allowing other enabler process changes to be implemented
quicker, more effectively and robustly
Ensuring changes, and the demonstrable measurement of
change, happens at all levels
The Key Message!
Evaluated “traditional” approach to implementing change and common
problems
Central Test Authority identified key enabler of effective, efficient, robust
change: their Test Professionals!
Identified, trained Delivery Stream Test Managers highly experienced in
Test and soft skills
Change is removing common problems encountered with change!
It is Working!
Any Questions?
Thank You!
Brian Wells:
www.experimentus.com
+44 (0)7725 709262