operational testing – walking a mile in the users boots' by gitte ottosen
DESCRIPTION
When testing is based on the requirements and focusing solely on the features being implemented, we tend to forget an important perspective – whether the system fits for its purpose. We need to take testing one step further – to walk a mile in the user’s boots – to understand and to test based on intensive domain knowledge When developing software, we base our work on requirements - being line items, use cases or scenarios. But often there is a long way from requirements to what the user really wants, and often the user is far away from where the requirements specification is being defined.When testing is based on the requirements and focusing solely on the features being implemented, we tend to forget an important perspective – whether the system fits for its purpose. We need to take testing one step further – to walk a mile in the user’s boots – to understand and to test based on intensive domain knowledge. In Systematic we especially discovered this need when developing our product, a command and control system for the army, and we therefore decided to take testing to the trenches – to implement operational testing of the system. The company already used domain advisors extensively when designing the system in order to ensure the users voice being heard, but now we wanted to take it yet another step towards REAL use and REAL users, designing and executing large operational scenarios with the use of REAL operational users. In this presentation you will be introduced to the process we went through and the results this lead to. In the presentation I will describe the process we have gone through in order to implement the concept, describing both successes and challenges. I will try to ensure that I also include reactions from the users that are involved as well as feedback from development.TRANSCRIPT
Operational Testing:
Walking a Mile in the User's Boots
© Sogeti
Systematic
• Established in 1985• Approx. 500 employees;
58% hold a MSc og PhD in software engineering
• Microsoft, ESRI and SAP partner
• 97% of our customers would recommend Systematic to other customers
• Customers in 35 countries
• CMMI Level 5 certified
• For more info visit:
www.systematic.com
Aarhus – HeadquartersSoftware and systems development
TampereProducts and services
LondonProducts and services
WashingtonProducts and services
Page 2
© Sogeti
Systematic’s Business Areas
A leading international company in delivering reliable and simplesolutions to people who make critical decisions every day
Vision:
Mission: Simplifying Critical Decision Making
Inte
ropera
bili
tyC
om
mand a
nd C
ontr
ol
Sit
uati
on A
ware
ness
Ele
ctro
nic
Warf
are
Support
Clin
ical In
form
ati
on S
yst
em
s
Managem
ent
Info
rmati
on S
yst
em
sB
oard
er
Contr
ol
Vari
ous
Finance
Transp
ort
Agri
cult
ure
Govern
ment
Defe
nce
Healt
hC
are
Inte
lligence
and
Nati
onal Secu
rity
Inte
gra
tion S
erv
ices
© Sogeti
The Sitaware Product Suite
© Sogeti
OVERVIEW
SitaWare Headquarters
Usable at multiple organisational levels
Improving situational awareness
Speeding up and supporting the Military Decision Making Process
Reducing planning preparation time
Swift andimproved ability to adjust on-going operations
© Sogeti
Product vs Project
SitaWare COTS Product
SitaWare Product Add-in
SitaWare Product Add-in
SitaWare Product Add-in
SitaWare Product Add-in
SitaWare Product Add-in
SitaWare Product Add-in
Customer specific
extension
Customer specific
extension
Customer specific
extension
Customer specific
extension
© Sogeti
The Background – How we Work
GUI
Communication Layer
Database
Business logic
Featu
re 1
Featu
re 2
The System
© Sogeti
How we Test
© Sogeti
Test in the Whole Lifecycle
Feature Kick-off
Feature A
Feature B
Code cut-off
Feature C
System test
Testing Stories
Test ”Qualify Feature”
Bug hunts
UnitIntegrationSystem
Sprint 3Sprint 1 Sprint 2
Increment
“Sprint test”
System Integration test
© Sogeti
The Challenges
• The domain• The complexity • The workflows• The environment - Interoperability
Sensors
Communications
WiFi
HF
UHFVHF
Works with many different comunication products
GyroGPS
Laser RangeFinder
Integrates to a variety of sensor products
SitaWare Headquarters
© Sogeti
Operational Test
© Sogeti
Moving the Focus
Test that the concept holds
Test the SYSTEM as a whole
Attack the system as a user would
Transfer fresh domain knowledge to the project
© Sogeti
An Iterative Process
Execute opr. test
Defining the
concept
Scenario and environment
ready
Execute test
Learning
Refining the
concept
Defining TSP and
environment
Project Product
Time
© Sogeti
Defining the Concept – Take One
• A project context• The operational test specification• The test environment• Internal test execution
© Sogeti
The Project Environment
© Sogeti
The Result and the Challenges
New bugs foundSystem used in a whole new way/contextDomain knowledge was transferred to the
developer/tester÷ Used as regression test – or misused÷ Stopped reading – executing by memory÷ Went faster and faster÷ Running the same scenario every time
© Sogeti
Defining the Concept – Take Two
• The real concept - Product• Defining the operational basis• Defining the organization
OPORDExercise Lightning
Fist1. Situation2. Mission3. Execution4. …5. …
© Sogeti
The First Execution – Getting Ready
Establishing the
environment
Defining the scenario
Getting the right people
Getting the training
© Sogeti
The Execution
© Sogeti
The Result of First Execution
• The concept was validated• New defects were identified• Knowledge was transferred into the organization• Testers were trained (Internal as well as external)
© Sogeti
What did we learn?
• Need for structure• Needs a lot of organisation• Clear division of responsibilities• Communication - communication – communication
© Sogeti
Following Executions
• Getting more people with the right profile• A better environment – a ”real organization”• Inviting UX to participate as observers - get to know the
users
• Inviting test to participate as assistants - get to know the domain
• What else:
– Usability test with the end users– Conceptual discussions
© Sogeti
Experiences
– Best for systems with larger work flows– Needs a lot of preparation– Need a lot of resources
• People• Environment• Test data setup• Scenarios• ….. And time
© Sogeti
The Way Ahead
• Extending the scenarios – Special operations– Including more applications from the product suite
• Finding new profiles – Fit the changing operational profiles– Other nationalities?
• More active involvement of operational testers in the planning phase
• The focus – from concept validation to bug hunting
• The schedule– Split up into shorter scenarios – max 2-4 hours– Feedback more often
© Sogeti
So What’s in it for us?
• Value on many levels– Verification of the concept
– Finding new defects– More operational knowledge gathered
– Testers get a better domain knowledge– Operational testers used for sparring as well, input for
the roadmap
© Sogeti