Data Validation for CER in SAFTINetMarion Sills12 September 2011

Overview

Data certificationData validation

Data Certification

Data certification: definitions/semantics Structured quality process to ensure data

meets standards established by its users (e.g., range checks, consistency checks)

Ensuring that data inserted into an application satisfies defined formats and other input criteria

Computer science: “data validation”

Data Certification

Data certification: SAFTINet Primarily performed by the Tech team, drawing

on DARTNet and other prior efforts for methods Development of a test plan with a formal

checklist CER team may be consulted regarding specific

CER-required elements excluded from prior certifications

Not the focus of today’s meeting

Data Validation

Data validation A definition: extent to which a concept,

conclusion or measurement is well-founded and corresponds accurately to the real world

For CER team: what are the key concepts in our CER analyses we should validate? Cohort definition Outcomes Delivery system characteristics Other variables

Cohort Definition Validation

Start with cohort definition validationLiterature review of sample approaches to

validation of asthma cohort definitionSuggestions for how to approach picking a

validation method

CER Team Asthma Cohort Definition

At least 2 diagnosis codes of 493 in 18 months, during visits at least 4 weeks apart, not restricted to outpatient visits

Validation of Asthma Cohort Definitions: Literature

OverviewMost validation studies on asthma case

definitions used survey or chart review data as a “gold standard”

Patient survey for cohort validation

Definition validated: HEDIS definition of persistent asthma

Standards: Mailed patient survey Asthma-related utilization

Data source: administrative data, survey Schatz M, Zeiger RS, Yang SJ, et al. Persistent asthma defined using

HEDIS versus survey criteria. Am J Manag Care. 2010 Nov 1;16(11):e281-8.

Huzel L, Roos LL, Anthonisen NR, Manfreda J. Diagnosing asthma: the fit between survey and administrative database. Can Respir J. 2002 Nov-Dec;9(6):407-12.

Chart review (expert panel) for cohort

validationDefinition validated: > 2 ambulatory visits and/or

> 1 hospitalization(s) for asthma in 2 yearsStandard:

patient charts reviewed by a blinded expert panel Data source: administrative data, chart review Gershon AS, Wang C, Guan J, et al. Identifying patients with

physician-diagnosed asthma in health administrative databases. Can Respir J. 2009 Nov-Dec;16(6):183-8.

Triangulation for cohort validation

Definition validated: HEDIS measure for persistent asthma

Standard: ED visits

Data source: administrative data Fuhlbrigge AL, Carey VJ, Finkelstein JA, et al. Validity of the

HEDIS criteria to identify children with persistent asthma and sustained high utilization. Am J Manag Care. 2005 May;11(5):325-30.

Longitudinal concordance for cohort

validation Definition validated: 6 different definitions Standard:

concordance in individual patient classification over sequential years

Data source: claims data Dombkowski KJ, Wasilevich EA, Lyon-Callo SK. Pediatric asthma

surveillance using Medicaid claims. Public Health Rep. 2005 Sep-Oct;120(5):515-24.

Similar (only 1 definition studied): Mosen DM, Macy E, Schatz M, et al. How well do the HEDIS asthma inclusion criteria identify persistent asthma? Am J Manag Care. 2005 Oct;11(10):650-4.

Latent class modeling for cohort validation

Standard: latent class modeling-based definition of treated asthma

LCM: used to find subtypes of cases (“latent classes”) in multivariate categorical data based on patterns of associations in the variables

Definitions validated: 3 standard definitions (including modified HEDIS)

Data source: administrative data Prosser RJ, Carleton BC, Smith MA. Identifying persons with treated

asthma using administrative data via latent class modelling. Health Serv Res. 2008 Apr;43(2):733-54.

Marker variables for LCM study

Findings for LCM study

Methods for Validation in SAFTINet

What is the best validation method for each SAFTINet cohort? Why?

What are best validation methods for other SAFTINet CER measures?