do you test the way your customer acts?

Madrid, 26th to 28th May 2014

Do you test the way your customer acts?

Real-life test; Real-life situations

A practical example of the Statistical Usage Test


Customer satisfaction

– Number one need:Involvement• Customers want to feel like

you care about their needs,and that you’re emotionallyinvested in helping themsolve their problems(Source: http://www.businesspundit.com/top-10-things-that-all-customers-want/)

2Real-life test; Real-life situations

http://www.businesspundit.com/top-10-things-that-all-customers-want/

http://www.businesspundit.com/top-10-things-that-all-customers-want/


Therefore, we test

– Because testing …

… is a process that provides insight into, and advice on, quality and the related risks

– And quality …

… is the totality of features and characteristics of a product or service that bear on its ability to satisfy stated or implied needs

– Thus, …

… the customer gets what he really needs, right? 3Real-life test; Real-life situations


What is happening


Focus shift


Does that satisfy the customer?

– “On system level we do exploratory testing”

– Does that really address the customer needs …

… and does it solves his/her problems?



What about test design techniques?


Test Design Technique

Useful for Test Design Technique

Useful for

Decision Table Verification Evolutionary Algorithms

Verification

Classification Tree Method

Verification State Transition Test Verification

Elementary Comparison Test

Verification Real-life test: Statistical usage Test

Verification & Validation

Error Guessing Verification Rare Event Test Verification

Exploratory Testing Verification & Validation

Semantic Test Verification

Data Cycle Test Verification Syntactic Test Verification

Process Cycle test Verification Use Case Test Verification & Validation


Validation Techniques

– Exploratory Testing• Only explores what the testers has interest in• Tester in center stage, instead of user• Actually verifying, not validating

– Use Case Testing• Fixed and pre-defined workflows to represent user

behavior• Based upon defined actors/users

– Real-life test: Statistical Usage Test• Statistically correct user behavior randomly applied• Based upon operational usage of all users



Statistical Usage Test – A Real Life Test

– Operational Usage is captured in Operational Profiles• Which operation is most likely to be performed under which

circumstances?– ‘Behaviors’ (events) – ‘States’ (history classes)– Probability of HistoryClass-Event combination


Event 1 Event 2 … Event m

History Class 1 P1,1 P1,2 .. P1,m

History Class 2 P2,1 P2,2 .. P2,m

… .. .. .. ..

History Class n Pn,1 Pn,2 .. Pn,m


Real-life test; Real-life situations 9

Statistical Usage Test

– Example of an operational profile (VCR)Events ►

History classes ▼evStop evRewind

(X tics)evPlay(X tics)

evFastForward(X tics)

evRecord(X tics)

Standby 0 0.2 0.5 0.2 0.1

Rewind 0.3 0 0.6 0.1 0

Play(counter < 15000)

0.5 0.3 0 0.2 0

Fast forward 0.5 0.1 0.4 0 0

Record 1 0 0 0 0

Play(counter ≥ 15000)

0.8 0.15 0 0.05 0



Gather the Operational Profile (1)

– Formal analysis / specification• Differentiate customers• Differentiate users• Differentiate system modes• Differentiate functions• Specify operational profile

for each [customer, user, system mode, function]

– Learn by example• Watch and record• Interviews



Gather the Operational Profile (2)

– Better: Use production data• Log files• Transaction data• Usage data

– Most systems are evolutions instead of revolutions• Evaluate predecessors• Evaluate comparable systems


– Everything your users do is done prove it

– No fixed sequence, butrandom selection at runtime statistically correct

– Thus, caring about the users’needs

– But what if the real user finds an anomaly? 13Real-life test; Real-life situations

Run the test


OK, there’s more to it

– Given a system (mobile X-ray system) with:• Intermittent failures• Unclear problem descriptions like

– “Sometimes the connection between stand and Mobile Viewing Station is lost”

– “Sometimes images are lost”

• Problems could not be reproduced by functional tests

– Root cause of system failure not yet found– Log file contains not enough information– Data analysis assumes hw errors



Solving a customers problem

– Increasing number of repairs

– Wear-out effects and error pareto shift


1% 2%1%6%

61%

24%

2%4%

1st timeBattery

CF card defect

ETX

Mechanical damage

No repair performed

Other

PSU

SCSI disk

1% 2%

17%

52%

23%

2%

2%

2nd timeBattery

CF card defect

ETX

Mechanical damage

No repair performed

Other

PSU

SCSI disk

4%

26%

30%

26%

15%

3+ timeBattery

CF card defect

ETX

Mechanical damage

No repair performed

Other

PSU

SCSI disk


Gather the Operational Profile

– Log file does contain usage information

– Define most important parameters:• Switch on / switch off

system• Start / stop low dose

fluoroscopy• Start / stop high

definition fluoroscopy• Store images

Log file excerpt10-02-10 14:05:34 S INIT > > > > INIT_Main:

Starting application < < < < 10-02-10 14:05:34 S INIT Build by nlybstqvp4-

sys24 on NLYBSTQVP4DT037, 10-02-10 14:05:34 M IPDR DFI Unit Serial Number:

[00-00-10-6B-22-03-01]10-02-10 14:05:34 M IPDR DFI Unit Versions: IN [0x1] PROC [0x1] CTRL

[0x5] DISP [0x1]10-02-10 14:05:34 S IPDR Starting DFI selftest

10-02-10 14:38:55 M SYS USR switch on RIGHT10-02-10 14:38:55 M SYS Xray mode continuous

fluoro10-02-10 14:38:55 M USR START FLUORO RUN RIGHT 10-02-10 14:38:55 M SYS USR switch off at 187683 10-02-10 14:38:56 M USR STOP FLUORO RUN 10-02-10 14:38:56 M FILE Stored [1] new image(s)

on disk10-02-10 14:38:56 M FILE Deleted [0] image(s)

from disk




– Data from log file


Case#

Start datedd-mm-yy

Start Timehh:mm:ss

End Datedd-mm-

yyEnd Timehh:mm:ss

Run Timehh:mm:ss

Left Fluo

#

Right Fluo

#

Images Stored

#

Total Fluo time (sec)

Avg Fluo time (sec)

1 10-02-10 14:05:34 10-02-10 14:08:16 0:02:42 0 0 0 0 0

2 10-02-10 14:08:16 10-02-10 14:57:08 0:48:52 2 24 24 15 0.576923077

3 12-02-10 7:12:11 12-02-10 10:27:44 3:15:33 9 74 75 63 0.759036145

38 9-03-10 7:15:20 09-03-10 13:57:47 6:42:27 60 1 18 58 0.950819672

39 10-03-10 7:37:01 10-03-10 7:37:37 0:00:36 0 0 0

Totals 61:27:51 565 480 521 1065

Average 1:34:34 14 12 13 28



– Interpreting the data determine history classes – Basic profile

• A ‘case’ is from system start to system stop• Sometimes no fluoroscopy is done at all• Most of the times a case consist of some Fluoroscopy

(LDF, HDF or both)18Real-life test; Real-life situations

History class Start system

Stop system

Make fluoroscopy run(s)

System switched off 1 0 0

System started 0 0.08 0.92

Fluoro Runs completed 0 1 0



– Add more details for fluoroscopy history classes• During a case the following figures apply for low

dose fluoroscopy (LDF)– Average number of LDF runs is known: 11.63– Assume normal distribution applies: sigma 1 = 66% - 133% of

average– “normal” maximum: 45 LDF runs– Real maximum: 270 LDF runs

• Same analysis for high definition fluoroscopy (HDF)




– Final profile


History class Start system

Stop system

Make fluoroscopy run(s)

System switched off 1 0 0

System started 0 0.08 0.92

Fluoro Runs completed 0 1 0

1 – 7 * LDF 8 – 16 * LDF 17 – 45 * LDF 46 – 270 * LDF

Standby 45% 29% 24% 2%

+1 – 7 * HDF 8 – 16 * HDF 17 – 78 * HDF

Standby 40% 40% 20%


Executing the test: AutomateSwitch on the systemDecide on the number of fluoroscopy runs @nr_fluoL = 0 @nr_fluoR = 0 # start throwing the dice for the left

fluoro runs @leftSel = rand() case @leftSel when 0 .. 0.41 @nr_fluoL = 0 when 0.42 .. 0.67 @nr_fluoL = 1 + rand(7) when 0.68 .. 0.84 @nr_fluoL = 8 + rand(9) when 0.85 .. 0.99 @nr_fluoL = 17 + rand(28) else @nr_fluoL = 46 + rand(225) end

# then for the right fluoro runs (= images being stored)

@rightSel = rand() case @rightSel when 0 .. 0.20 @nr_fluoR = 0 when 0.21 .. 0.59 @nr_fluoR = 1 + rand(7) when 0.60 .. 0.78 @nr_fluoR = 8 + rand(9) else @nr_fluoR = 17 + rand(62) end

Perform the selected number of runsSwitch off the system

And start all over again (endless loop)



Statistical Usage Test results

– System starts showing intermittent failures– When continuing, test results in a Fatal Error

(hard disk broken)

– 13 days of test versus 1 month of real-life usage• Images stored: 80719 versus 521• Total Fluoroscopy time: 135995 sec versus 1065 sec


Run Timehh:mm:ss # Left Fluo

# Right Fluo

# Images Stored

Total Fluo time (sec)

Customer system 61:27:51 565 480 521 1065

Test system 198:19:31 71873 89548 80719 135995


Statistical Usage Test results

– Problem reproduced– Test results have proven the data analysis

– Possible improvements on the Operational Profile and/or the test• Combine more user data (e.g. more log files, more

customers)

• Define actual contribution on available data (no assumption of normal distribution)

– Automation is mandatory, e.g. using FitNesse 23Real-life test; Real-life situations


Conclusion

– Real-life testing is doing what your customer wants to do Paying attention to thecustomer needs

– Real-life testing helps findingissues Help solving thecustomers’ problems



IS YOUR CUSTOMER SATISFIED?

Unless you have 100% customer satisfaction…you must improve.Horst Schulze

do you test the way your customer acts?

Software

test design techniques

statistical usage test

event example test case

given operational profile

operational profile

customer needs

operational profiles

use case testing