Innovative Use of Multiple Methods to Assess Health Informatics System Standards

APHA 2013 -- Boston, MANovember 6, 2013

Linda Piccinino, Andrew Westdorp, Jennifer Kwan, Lydia Rogers, Jean Wilson, & Sue Griffey

2

Presentation Outline

• Background• Purpose• Framework• Evaluation Data Collection• Results• Conclusion

BACKGROUND

3

4

NCS Vanguard Study

• The National Children’s Study (NCS)– Large-scale, dynamic health study – 20-year time horizon

• NCS Vanguard Study– Pilot study for the NCS main study– Uses multiple information management systems

(IMS) to support one research data collection effort

5

NCS Vanguard Study IMS Evaluation

• Assessment of IMS to see how well meeting the study’s needs– Current needs– Future projected needs

PURPOSE

6

7

IMS Evaluation

• Develop and implement an evidence-based evaluation approach

• Answer fundamental questions:– Does each IMS support the study’s basic

functional requirements?– Does each IMS perform as expected?

• Leverage evaluation framework and methodology

8

Evaluation Phases

• Phase 1 – Short-term: 2-3 months– High level assessment of basic functionality of

IMS solutions – Completed in Fall 2012

9

Evaluation Phases (continued)

• Phase 2 – Longer-term: 9-12 months– Identify/refine preferred practices, processes,

and functionality of an IMS solution used to support NCS Vanguard protocol

– Identify gaps in IMS solutions with respect to evaluation criteria

FRAMEWORK

10

11

Phase 1 Evaluation Framework

Theory IndicatorsData

Collection Methods

Data Sources

EVALUATION DATA COLLECTION

12

13

Phase 1 IMS Evaluation

Benchmarking System Characteristics

Real-Case Scenario-Testing

• Reviewed system and user documentation

• Performed system stand up for available solutions

• Developed standardized scenarios

• Conducted testing using host-provided environment

14

Example: Role-Based Scenario Functions Tested

Administrator Data Collector and Supervisor

Supervisor/Manager

Perform Study Center setup

Assign user roles Preload pregnancy

screeners

Consent mother and child

Conduct Pregnancy Visit 1 event

Complete participant record of visit

Enter Operational Data Elements

Enter linkages

Assign cases Manage field work Provide time and

expense information View staff

certifications View and edit

instrument data Obtain basic

cooperation rates Obtain basic refusal

rates Obtain rates by

disposition code

15

Phase 1 Overview: IMS Functional Requirements Tested

IMS Functional RequirementsDoc.

ReviewScenario Testing

Architecture 1. Is the IMS system open-source? √ √2. Does the system utilize open architecture and use of modern non-proprietary programming languages? √ √

3. Does the system have an open API for easy integration with other products? √

4. Does the system have any special hardware requirements? √ √

5. Does the system have the support 2-factor authentication? √

Case Management 1. Cases can be flagged for supervisor review √2. System allows for creation of new person records to be associated with each case √

RESULTS

16

17

Benchmarking

• Benchmarking results showed where systems were strongest and where they needed additional functionality

18

Review of Documentation on IMS Characteristics

Category IMS 1 IMS 2 IMS 3Documentation Attributes All All All

General System Characteristics ≥ ½ All < ½

Architecture< ½ ≥ ½ ≥ ½

Case Management≥ ½ All ≥ ½

IMS Performance< ½ ≥ ½ < ½

IMS Data Collection< ½ All ≥ ½

Data Management< ½ ≥ ½ ≥ ½

CATI< ½ ≥ ½ ≥ ½

Sample Tracking*< ½ < ½ < ½

All All characteristics documented

≥ ½ Half or more characteristics documented

< ½ Less than half of characteristics documented

*Few sites collecting biospecimens at the time.

19

Real-Case Scenario Functional Testing

• Results provided:– Objective information on functionality– Qualitative input from IMS testers

20

Data Collector User Role Scenario Testing

Functionality IMS 1 IMS 2 IMS 31. Can understand how to enter data and use system

based on documentation available?Y Y Y

2. Can view prior events? Y Y Y3. Can view contact records? Y Y Y4. Next event needed is indicated in system? Y Y N5. Can schedule events? Y Y Y6. Requires Pregnancy Screener records to allow

consent entry? N N n/a

7. Can enter Study consent information for mother?Y Y Y

8. Can view Study consent information for mother?Y Y Y

21

Functionality of IMS Solution Characteristics

> ½ More than half of characteristics tested were functional

≤ ½ Half or fewer of characteristics tested were functional

N Characteristics were not observed

IMS 1 IMS 2 IMS 3

Case Management > ½ > ½ > ½

IMS Performance > ½ > ½ > ½

IMS Data Collection > ½ > ½ > ½

Data Management > ½ N ≤ ½

22

Translating Results into Use

• Results used to:– Identify high-level areas already meeting

requirements – Identify specific gaps or areas needing

improvement– Inform Phase 2 assessment

CONCLUSION

23

24

NCS Vanguard Study IMS Evaluation

• Framework – Grounded in theory

• Collaboration– Involving and communicating with multiple

stakeholders– Critical when testing systems

• Innovation – Moved beyond traditional IT evaluation to include

aspects of program evaluation– Implemented ways to test systems, not users, using

protocol-driven real-life standardized scenarios

25

Thank You!

Authors• Linda Piccinino, Senior Evaluator, SSS, lpiccinino@s-3.com• Andrew Westdorp, Center Director, SSS, awestdorp@s-3.com• Jennifer Kwan, Evaluation Team Co-Lead, Program Office,

National Children’s Study, Eunice Kennedy Shriver National Institute of Child Health and Human Development, jennifer.kwan@nih.gov

• Lydia Rogers, Project Director, SSS, lrogers@s-3.com• Jean Wilson, Senior Study Director, SSS, jwilson@s-3.com• Sue Griffey, Vice President and Director, Evaluation Center,

SSS, sgriffey@s-3.com