User Satisfaction: Illusion or Paradox?

Charlene R. Weir, PhDProfessorDepartment of Biomedical Informatics

3515B

OVERVIEW

Current state of the literature (2015 only) Instruments used Correlation with adoption

Purpose and Related Theories Proxy for adoption Proxy for effectiveness

Alternative View Focus on Behavior (usage) Motivation Task-based assessments of effectiveness

EXAMPLE (VA)

Recent Studies(2015)Assessing User Satisfaction

# of studies 32 Intention to use 12 Adoption Behavior (only usage) 7 Information Performance 5 Theories 15

What is the question that User Satisfaction is supposed to answer?

A proxy for adoption?

A proxy for effectiveness?

A Proxy for Adoption

What IS Adoption? Intention to UseUsage/BehaviorCustomization

What are relevant theories?Technology Adoption ModelUniversal Theory of ***

DIFFUSION OF INNOVATIONA staged communication process

Early Innovators

Innovation Characteristics

Relative Advantage – compared to other things

Compatibility - How consistent the innovation is with the values, experiences of adopters.

Complexity - How difficult the innovation is to understand

Triability - The extent to which the innovation can be tested or experimented with

Observability - The extent to which the innovation provides tangible results.

MOTIVATIONAL THEORIES

Theory of Planned Behavior Behavioral belief / Perceptions of Outcomes: beliefs

regarding outcomes of XXX PLUS the subjective probability that the behavior will produce that outcome. (CPOE decreases errors, improves communication, improves quality of care)

Attitude toward behavior: Perceived value for the outcome (good thing/bad thing)

Normative beliefs: perception of social influence Perceived behavioral control: beliefs about it actually

working (e.g. support, mandated, resources). Control beliefs: The concept of perceived behavioral

control is conceptually related to self-efficacy.

UTAUT (Unified Theory of Acceptance and Use of Technology)Venkatesh, V., Morris, M.G., Davis, F.D., and Davis, G.B.(2003) “User Acceptance of Information Technology: Toward a Unified View,” MIS Quarterly

MODERATORS

TAMTechnology Acceptance Model

SUS (System Usability Scale) I think that I would like to use this system frequently I found the system unnecessarily complex I thought the system was easy to use I think that I would need the support of a technical

person to be able to use this system I found the various functions were integrated I thought there was too much inconsistency in the

system  I would imagine that most people would learn to use this

system very quickly I found the system very cumbersome to use

I felt very confident using the system I needed to learn a lot of things before I could get going

What is WRONG with this picture?Why do we care?

Most usage is mandated in clinical information systems

Adoption measures are not correlated with outcomes

Adoption is not correlated to implementation strategies

Models developed to PREDICT usage do not predict “making decisions better,” “workflow more efficient” or “enhanced coordination”

Can’t be used for design

Results are always in the 3.2 to 4.2 range (1 to 5 scale)

We already KNOW the theories are empirically validated

A Proxy for Effectiveness What is effectiveness?

System performance IT support Information Management tasksDocumentation GoalsCommunication/Coordination

What are relative theories?DeLone and McLeonTask/Technology Fit Information Theory

DeLone W, McLean E. (2003) The DeLone and McLean Model of Information Systems Success: A Ten-Year Update

Task/Technology FitGoodhue & Ammenworth

WHAT ARE the NET BENEFITS of Clinical IT?

Example: Assessing Satisfaction and User Behavior for CPRS Adoption

Methods – Task Instrument Interviews of 24 Primary Care Practitioners Week-long observation of 2 primary clinics Action Identification Interviews

“Describe what you are doing” Qualitative Analysis Generation of 34 items Factor Analysis with 360 providers Reduction to 22 items (TASK)

Theory of Action IdentificationVallacher R, Wegner D. What do people think they are doing? Action identification and human behavior. Psych Rev. 1987;94:3–15

Action identities “what are you doing?”“ You are doing XX in order to do XX?”“ You are doing XX BY DOING YY”?

Hierarchical constructionGoals/Values on topActions at bottom

Experts report high levels; novices low levels

Typical Action “Identities”

Reporting required information Making a decision about the best course of

treatment Looking for patient data Checking over the patient status Becoming familiar with the patient Finding info to teach pt. Looking for the evidence on the best treatment Determining the # of procedures done per day Looking up local policies

Results

Methods – Validation

N=124 26 Primary Care Practitioners 18 Other specialists 14 Pharmacists 44 RNs 22 Residents

Pre/Post CPRS Implementation 92% response rate

Measures of User Satisfaction

Task Effectiveness (TASK; n= 22 items) “Information Tasks” Three questions each on ratings of effectiveness

Communication • Documentation Decision-Making • Workload Tracking Policy Compliance • Data Review

Total Summed: alpha = 0.90 Usability (3 items)

Easy to use, minimal effort; easy to navigate Total Summed: alpha = 0.93

Measures of User Satisfaction Intrinsic Motivation (AFFECT)

Three items assessing affective experience Interest Enjoyment Fun

Items summed to form scale: alpha = 0.68 User Satisfaction (GENERAL)

9 items (Doll, et al) (comprehensive, accurate, easy to use, useful, improves my work)

1 factor (alpha= 0.84)

Correlation Results

RESULTS: User Satisfaction Measures

Early vs Late Adopters: Progress Notes in Primary Care

Level of AdoptionPersonal Customization

Sum of 12 customization options used

RESISTANCE RATINGS

Taken from Reactance Theory Reactance results from a threat to control Results in demeaning the messenger; dismissing the

message, avoiding the threat, and reasserting control (refusals, hording paper)

Four members of the implementation team rated 24 primary care providers on:

1 (none) to 7 (very high) scale Ratings were averaged.

RESISTANCE RATINGSCorrelation Results

STAGES OF CHANGEProchaska & DiClemente

Combines many motivational theories Proposes four stages in change process,

characterized by attitude, intentions and behavior. Pre-Contemplation - not concerned, little attention Contemplation - thinking about the pros & cons Action - formed an intention; making plans Maintenance - behavior in place; well-learned

STAGES OF CHANGEThey checked one (out of 5)

I use the system only when I have to and I do not intend to try to expand my use in any way. I see no reason to learn much more about it.

I have begun trying to integrate using the computer in my daily work processes ad am now partially successful. I am learning more about it all the time.

Even if the system were not mandated, I would use it. My work is extensively dependent on the computer.

Stage of ChangeMean values across stage for each measure (ANOVA)A good measure “picks up” the change across stages

Involvement/Volunteer

Asked if they wanted to get more information and would like to volunteer.

Involvement Group = either question (n=57)

Non-involvement group (n=20) = checked none

INVOLVEMENT

So What? These results had NO impact on developers

Did not inform implementation

Might be used to compare across settings

Need to develop metrics that can relate Usability measures to User Satisfaction

ANSWER:

Focus on Tasks – relate them to goals Link Usability Assessments to these Tasks Link tasks to implementation strategies Link tasks to clinical outcomes

Questions?