thomas loveday - nsw clinical excellence commission
TRANSCRIPT
Considering the ‘Human Factor’ in Medical Device Procurement
Thomas LovedaySnr Manager, Organisational Development and Human Factors
October 2016
Presentation OverviewThis presentation will:• A brief review of why ‘usability’ is a core concern
in medical device procurement• How Human Factors provides a framework for
evaluating usability• Designing for usability OR evaluating usability• The usability evaluation processes/tools being
developed within HealthShare NSW
Clinical Engagement in Procurement• Healthcare staff are now formally involved in:
– Specification writing
– Ongoing contract management
– Desk top evaluation products
– Local site trials
3
Includes one usability item
Why is usability critical?
4
What is Human Factors?• Human Factors is the practice of studying and
designing systems so that they fit the peoplewithin them. It is also known as:
• User-Centred Design (UCD)
• Ergonomics (physical, cognitive, organisational)
The Human Factor in Health• Doctors
• Nurses
• Pharmacists
• Clerical staff
• Cleaners
• Patients
Human Factors• Physiological characteristics
– including vision, strength, flexibility, height
• Psychological characteristics– Including perception, memory, intuition, stress,
fatigue, values
• Proprioceptive characteristics– Including hand-eye coordination, spatial awareness,
balance
System Factors• Communication and interaction between
individuals, teams and services
• The formal structure and the informal culture of the organisation
• Interactions with equipment and physical environment
Change the System or the Human?• Hard (impossible) to change many human factors
– Universal; and/or
– Fixed (i.e. height, perception, memory)
• We can procure devices that accommodate the “human factor” and reduce the incidence of use error.
Comprehensive Device Usability Evaluation Process
10
(The Goal)
HF Specifications in Contracts• Request compliance with:
– AAMI/ANSI HE75:2009;
– ISO/IEC 62366-1:2015;
– ANSI/AAMI/ISO 14971:2007; AND/OR
– ISO 60601-1-8:2006.
• Additional HF specifications identified by:– HF Specialist
– Human Factors Advisory Committee
Heuristic Evaluation• A type of usability inspection method, where
3+ evaluators examine a device for consistency with usability principles
• Designed for interfaces, but has been adapted for devices (Zhang et al, 2004)
Zhang et al (2004) Heuristics1. Consistency and standards – Users should be informed about what is going on
with the system
2. Visibility of system state. Users should be informed about what is going on with the system.
3. Match between system and world – Clear conceptual link between controls and outcomes
4. Minimalist – Any extraneous information is a distraction and a slow-down.
5. Minimize memory load – Users should not have to memorize a lot of information to carry out tasks.
6. Informative feedback – Users should be given prompt and informative feedback about their actions.
7. Flexibility and efficiency – Allow users to create shortcuts to accelerate their performance (where appropriate).
Zhang et al (2004) Heuristics8. Good error messages – Device messages/alerts should be informative enough that
users can understand the nature of errors, learn from errors, and recover from errors.
9. Prevent errors – It is always better to design interfaces that prevent errors from happening.
10. Clear closure – Every task has a beginning and an end.
11. Reversible actions – Users should be allowed to recover from errors.
12. Use users’ language – The language should be always presented in a form understandable by the intended users.
13. Users in control – Do not give users that impression that they are controlled by the systems.
14. Help and documentation – Always provide help when needed.
Method• Evaluator completes key device tasks several
times to identify key user interactions
– For ‘complex’ devices ask an expert to help
• Compares each ‘interaction’ against usability principles (if they apply)
Heuristic Evaluation Cheap
Fast
Requires minimal training
Not specific
Subjective
No user input/involvement
Effectiveness dependent on HF expertise
Simulation-based usability testing
• Control risk;• Observe rare events;• Provide experimental control;• Incorporate qualitative AND
quantitative metrics.
17
Usability Testing can identifying impediments to “ease of use”
and “goal completion” by testing it on real users.
What happens in a usability test?
Courtesy of Dr Cate McIntosh
What are the results of a test?• Identify impediments to ease of use based on:
– Errors
– Time to completion
– Verbal statements during task
– Where users got “stuck”
– Etc
• There metrics provide a basis for comparing devices (comparative testing)
Usability testing Otherwise difficult to predict what a first-time user will experience
Provides more detailed information
As few as five ‘users’ can reveal >90% of design issues
Scenario development requires information gathering
Comparative testing is resource intensive
(2 facilitators x 5 sessions x No. of devices in a simulation suite)
Local site trials
Relatively resource light Real-world evidence May not encounter rare critical events Present risk to patients
Requires rigour and metrics for comparison of devices
Which devices?High-risk categories?
• Infusion Pumps
• PCA Pumps
• Syringe drivers
• Defibrillators
• Beds
The NSW Health Approach
23
(the reality)
Raj Ratwani quote“Summative testing requirements occur at the end of development of the product and any design flaws that are identified are unlikely to be addressed since the product has already been fully developed.”Testimony of Raj M. Ratwani, PhD to U.S. Senate Committee on Health, Education, Labor and Pensions
24
Vendor self-assessment
Vendor asked to identify user-centred design (UCD) methods used during development of device
2.1With which of the following Human Factors standard(s) does the device design and/or evaluation comply?
2.2Has the device or a similar device (see Note 2) ever undergone formal Human Factors assessment as part of the design process?
2.2.1Did the Human Factors Assessment include a review of prior assessments and studies of similar devices to identify and manage potential usability risks?
2.2.2Did the Human Factors Assessment include a 'Heuristic Evaluation' (i.e. a panel of experts reviews the device and judges its compliance with recognised usability principles)?
2.2.3Did the Human Factors Assessment include 'Usability Testing?'
Vendor self-assessment Feasible and sustainable
Increase pressure on manufacturers to adopt UCD practices
UCD processes may be superficial
May not capture unique local critical tasks.
26
