stephen k. allott - people challenges for test teams - softtest ireland
DESCRIPTION
Visit SoftTest Ireland www.softtest.ie and sign up for access to free Irish Software Testing events.TRANSCRIPT
SoftTest Ireland Copyright 2008 ElectroMind Ltd 1
People Challenges for Test Teams
Stephen K. Allott
SoftTest Ireland Copyright 2008 ElectroMind Ltd 2
SoftTest Ireland Copyright 2008 ElectroMind Ltd 3
SoftTest Ireland Copyright 2008 ElectroMind Ltd 4
SoftTest Ireland Copyright 2008 ElectroMind Ltd 5
Introductions
My backgroundAims of this workshop
Explain some common testing challenges based on recent experienceAsk for your experiencesAgree a list of challenges we all face as testersTry to come up with some quick wins & longer term approaches to solutions
SoftTest Ireland Copyright 2008 ElectroMind Ltd 6
My recent clients
Investment bankAirline web siteTravel company web siteMobile phone operator Retail web site Travel company back officeRecruitment web site
Includes assignments done in association with TestHouse (London) and Neueda Technologies (Antrim & Dublin)
SoftTest Ireland Copyright 2008 ElectroMind Ltd 7
Their technologies
Microsoft Dot (.) NET 1/2/3Microsoft VSTS / TFSOracleMS SQL ServerWindows XPCitrix MetaframeQuality Centre / QTP
SoftTest Ireland Copyright 2008 ElectroMind Ltd 8
Number of testers
Typically 12 to 14 in the test teamA couple of clients had only 1, some have none!
SoftTest Ireland Copyright 2008 ElectroMind Ltd 9
Scale 0 1 2 3 4 5 6 7 8 9 10 11 12 13
Key Area Controlled Efficient Optimising
Test strategy A B C D
Life-cycle model A B
Moment of involvement A B C D
Estimating and planning A B
Test specification techniques A B
Static test techniques A B
Metrics A B C D
Test automation A B C
Test environment A B C
Office environment A
Commitment and motivation A B C
Test functions and training A B C
Scope of methodology A B C
Communication A B C
Reporting A B C D
Defect management A B C
Testware management A B C D
Test process management A B C
Evaluation A B
Low-level testing A B C
Composite TPI Test Maturity Matrix from several clients
SoftTest Ireland Copyright 2008 ElectroMind Ltd 10
Summary of TPI results
No documented test processNo formal test policy, strategy or planMany tests, no formal designNo metricsNo proper test environment
Some tested in “live”
No documented test basisNot involved early enough
SoftTest Ireland Copyright 2008 ElectroMind Ltd 11
Some common observations
Constantly changing business prioritiesIncreasing (frantic) demand for new functionalityWeekly, sometimes daily release cyclesAttempt to use automation to solve all the problemsNo numbers to explain testing’s added valueSmall important teams yet with little influence
SoftTest Ireland Copyright 2008 ElectroMind Ltd 12
People challenges – behaviour
MotivationA few are resigned to how it is, no hope?Some expect spoon feedingSome adopt a negative attitudeMany are willing to make improvements, if there was time
Turnover – very high offshoreCulture
Some managers get too involved in detailCo-dependency – testers must push backIt hasn’t broken yet so . . .
SoftTest Ireland Copyright 2008 ElectroMind Ltd 13
People challenges – skills
Testing SkillsProcessesStandardsTechniquesTools
IT SkillsProgrammingDatabasesInfrastructureWeb
Domain KnowledgeIndustry specificApplication knowledgeE-Business
Soft SkillsCommunicationNegotiationManaging technical teams
SoftTest Ireland Copyright 2008 ElectroMind Ltd 14
People challenges – perception
In general testers are not valuedTesters are often blamed
Project delaysPoor qualityManaging expectations with RBT
Lack of awareness of what the test team does
SoftTest Ireland Copyright 2008 ElectroMind Ltd 15
People challenges – managers
Demand for certified staffDemand for skills & knowledgePressure to cut costs and use offshore resourcesDeveloping good test leadsRole of testers in an agile projectSelling the value of testing to the organisation
SoftTest Ireland Copyright 2008 ElectroMind Ltd 16
Themes
Facilitated discussion bearing in mind the following themes:
1. Relentless business pressure for new functionality2. Constant and last minute changes to requirements3. Introduction of new development / testing methodologies e.g. agile 4. Very frequent releases, often weekly, sometimes daily!5. Increasing application complexity over time6. Knowledge gap (e.g. what lies beyond the browser?)7. Steep learning curve (business as well as technical)8. Use of contract and offshore resources (loss of in-house expertise)9. Little or no documentation against which to test10. Lack of attention to all testing requirements (e.g. middleware, back
end database, non-functional quality characteristics)
SoftTest Ireland Copyright 2008 ElectroMind Ltd 17
Discussion – how do we respond?
Business PressuresNew functionalityFrequent changeMaturity
TechnologyComplexRapid changeInterfaces
OffshoreCheaperPlentifulExpertise
BehaviourMotivationAttitudeCulture
SkillsAnalysisIT / TechnicalBusinessTestingSoft skills
ManagersCostsTime to marketAgile
SoftTest Ireland Copyright 2008 ElectroMind Ltd 18
Conclusions
In my opinion testers of the future must . . .Measure / appreciate application complexityBe able to accept and manage last minute changesBe flexible but push back with estimatesIncrease their technical knowledgeLearn more about testing, especially techniquesApply the tried and tested techniquesCapture metrics to help prove the business caseGet involved in the projects much earlier
Recruit / train / certify / develop / coach = RetainShare your views with the group
SoftTest Ireland Copyright 2008 ElectroMind Ltd 19
Final Thoughts / Questions?
SoftTest Ireland Copyright 2008 ElectroMind Ltd 20
People Challenges for Test Teams
[email protected]+44(0) 773 476 1363www.electromind.com
SoftTest Ireland Copyright 2008 ElectroMind Ltd 21
People Challenges for Test Teams
Appendix – Bonus Slides
SoftTest Ireland Copyright 2008 ElectroMind Ltd 22
A typical project in trouble
CR Churn by Month - AveCR=47 AveChurn=150 AveChurnCR=3
1427
58 5565
40
116
279
229213
0 0
254
124
263
0
50
100
150
200
250
300
June July August September October
Months
CR
cou
nt
0
50
100
150
200
250
300
Cha
nges
to C
Rs
CRs Churn Bugs
SoftTest Ireland Copyright 2008 ElectroMind Ltd 23
Bug snapshot by priority
Sev 1 Sev 2 Sev 3 Sev 4 Total
Pri 1 57 81 14 152
Pri 2 19 52 12 1 84
Pri 3 6 14 10 30
Pri 4 2 1 3
None 68 288 114 10 480
Total 146 428 154 21 749
SoftTest Ireland Copyright 2008 ElectroMind Ltd 24
Bug snapshot by resolution
Sev 1 Sev 2 Sev 3 Sev 4 Total
By design 10 37 29 4 80
Duplicate 6 33 7 1 47
External 2 16 4 0 22
Fixed 86 254 95 12 447
Not repro 29 56 4 0 89
Postponed 12 17 6 4 39
Will not fix 1 15 9 0 25
Total 146 428 154 21 749
SoftTest Ireland Copyright 2008 ElectroMind Ltd 25
Impact of the improvements
CR Churn by Month - AveCR=47 AveChurn=150 AveChurnCR=3
1427
58 5565
30
61
17 15 10 5
40
116
279
229213
6686
18 15 10 50 0
254
124
263
13
90
12 9 5 10
50
100
150
200
250
300
June July
Augus
tSep
tembe
r
Octobe
r
Novembe
rDece
mber
Janu
ary
Februa
ry
March
April
Months
CR c
ount
0
50
100
150
200
250
300
Cha
nges
to C
Rs
CRs Churn Bugs
Events Systems1. Vendor2. User Reg.3. User Shop
Sales and Marketing Systems1. Consulting Practice Mgt.2. Vendor Account Mgt.3. Event Mgt.4. Leads and Referral
Databases1. Profile2. Common3. Event OLTP4. Transaction OLTP5. Contact History OLTP6. Domain Reference
World-wide Marketing DB1. Profile OLAP2. Event OLAP3. Analytical Reporting4. Contract History OLAP5. Domain
Corporate Sales and Marketing Systems
System Interfaces1. Event Orders2. Billing & Financials3. Credit Card (real-time)4. Event Communication / FAX5. Direct email service6. Direct mail Postal interface7. Leads & Referrals8. OLTP Staging Interface9. OLAP Staging Interface10. SAP11. Licensing12. Call Center Source: Reality Test Inc.
SoftTest Ireland Copyright 2008 ElectroMind Ltd 27
Basic complexity calculation
Characteristic Value Factor Total
Windows 37 8 296
Objects 23 1 23
Tables 47 1 47
Apps Impacted 3 10 30
Factors are weighted from a testing perspective
SoftTest Ireland Copyright 2008 ElectroMind Ltd 28
Comparison across applications
Application UI Back End
Env Test Code
Middle Tier
Total
Sales Mgr. 1360 2250 152 0 0 3762Call Center 960 824 90 129 0 2003Web Sales 389 183 109 0 2 681 Middleware component
0 146 23 27 232 428
SoftTest Ireland Copyright 2008 ElectroMind Ltd 29
0
1000
2000
3000
4000
PSS CTR
Knowled
ge B
ase
Look
ing G
lass
Web
Sale
s
Accou
nt Man
ager
Delta
Call C
enter
Sales M
anag
er
Application Name
Com
plex
ity V
alue
Complexity comparison
SoftTest Ireland Copyright 2008 ElectroMind Ltd 30
Actual staffing comparison
0
5
10
15
20
PSS CTR
Knowled
ge B
ase
Look
ing G
lass
Web
Sale
sAcc
ount
Manag
er
Delta
Call C
enter
Sales M
anag
er
Applications
Num
ber o
f Tes
ters
Estimated
Actual
SoftTest Ireland Copyright 2008 ElectroMind Ltd 31
Organisation evaluation
Organisation within the companyProcess definition and executionTeamworkAbility to do the job
SoftTest Ireland Copyright 2008 ElectroMind Ltd 32
Organisation within the company
Section Item being evaluated Max Score1.1 Reporting structure 501.2 Professionalism 1701.3 Organisational churn 1351.4a Staffing – permanence 301.4b Staffing – stability 351.4c Staffing – morale 501.4d Staffing – number of testers 401.4e Staffing – organisation type 401.5 Budget control 30
SoftTest Ireland Copyright 2008 ElectroMind Ltd 33
Process definition and execution
Section Item being evaluated Max Score2.1 Software development life-cycle 602.2 Design reviews and documentation 702.3 Early participation in development 502.4 Ability to influence schedule deadlines 602.5 Development standards 602.6 Defined testing approach & procedures 1902.7 Metrics reporting 602.8 Churn 902.9 Patch releases 402.10 Post-project reviews 60
SoftTest Ireland Copyright 2008 ElectroMind Ltd 34
Teamwork
Section Item being evaluated Max Score3.1 Support from production support 603.2 Support from management 603.3 Correct level of quality 903.4 Resource allocation 80
SoftTest Ireland Copyright 2008 ElectroMind Ltd 35
Ability to do the job
Section Item being evaluated Max Score4.1 Adequate equipment / lab 2004.2 Automated testing 704.3 Programming ability 704.4 Non-testing activities 50
SoftTest Ireland Copyright 2008 ElectroMind Ltd 36
Total evaluation score
900
1000
1100
1200
1300
Q1/00 Q2/00 Q3/00 Q4/00 Q1/01 Q2/01 Q3/01
Date
Eval
uatio
n
Total Evaluation Score
SoftTest Ireland Copyright 2008 ElectroMind Ltd 37
Organisational Evaluation
100
200
300
400
500
Q1/00 Q2/00 Q3/00 Q4/00 Q1/01 Q2/01 Q3/01
Date
Eva
luat
ion
Organization
Process
Teaming
Ability
SoftTest Ireland Copyright 2008 ElectroMind Ltd 38
Components of a centre of excellence
Resource pool Test lab
Test Managers SIG
ConsultantsKnowledge Base
R&D Centre
Tool Experts
Project 1
Project 2
Project 3
Project 4
SoftTest Ireland Copyright 2008 ElectroMind Ltd 39
Benefits of a centre of excellence
Independent testingBreadth & depthFewer product recallsFewer production failuresCentral reference pointKnowledgeable staffExpertise across org.Structured career path
Variety of workIdentity, prideMorale, professionalismExtra revenue streamConsistent approachMulti-methodologiesEarlier involvement