Session 3: Session 3: Assessment & EvaluationAssessment & Evaluation

Lisa Jacques-Carroll, MSWLisa Jacques-Carroll, MSW

NCIRD, CDCNCIRD, CDC

Learning ObjectivesLearning Objectives

Describe key components of evaluation of Describe key components of evaluation of a perinatal hepatitis B prevention program a perinatal hepatitis B prevention program describe different methods for assessing and describe different methods for assessing and

evaluating a perinatal hepatitis B prevention evaluating a perinatal hepatitis B prevention programprogram

identify several tools that are availableidentify several tools that are available

Assessment & EvaluationAssessment & Evaluation

Program evaluation vs. program Program evaluation vs. program monitoring or assessmentmonitoring or assessment

The focus of this session is on assessing The focus of this session is on assessing performance measures, which is a performance measures, which is a component of program evaluation component of program evaluation

For more information on program For more information on program evaluation see:  evaluation see: 

www.cdc.gov/vaccines/programs/progeval/www.cdc.gov/vaccines/programs/progeval/

Activities Required by CDCActivities Required by CDC

Requirements for assessment of the Requirements for assessment of the perinatal hepatitis B include:perinatal hepatitis B include: frequency of assessmentfrequency of assessment performance measuresperformance measures target levels target levels

For more information: For more information: http://www.cdc.gov/vaccines/vac-gen/policies/ http://www.cdc.gov/vaccines/vac-gen/policies/ ipom/downloads/chp-05-perinatal-hepb.pdfipom/downloads/chp-05-perinatal-hepb.pdf

OverviewOverview

Laboratory reporting auditsLaboratory reporting audits Hospital record reviewsHospital record reviews Hospital policy surveysHospital policy surveys Expected births to HBsAg+ womenExpected births to HBsAg+ women Assessing case management Assessing case management Monitoring birth doseMonitoring birth dose

Laboratory Reporting AuditsLaboratory Reporting Audits

Use to assess:Use to assess: completeness of reporting of HBsAg+ resultscompleteness of reporting of HBsAg+ results timeliness of reportingtimeliness of reporting

Include delivery hospital labsInclude delivery hospital labs Collaborate with communicable disease Collaborate with communicable disease

and state laboratory staff to help plan and and state laboratory staff to help plan and execute reviewsexecute reviews

Prioritize Labs to EvaluatePrioritize Labs to Evaluate

Priority labs:Priority labs: labs serving high-morbidity areas or populationslabs serving high-morbidity areas or populations labs reporting large volume of hepatitis serologylabs reporting large volume of hepatitis serology labs serving prenatal clinicslabs serving prenatal clinics delivery hospital labsdelivery hospital labs

How often?How often? ideally, once/year for priority labsideally, once/year for priority labs every 2–3 years for othersevery 2–3 years for others minimally every 5 yearsminimally every 5 years

Laboratory Audit ToolLaboratory Audit Tool

A CDC tool available at:A CDC tool available at:

http://www.cdc.gov/ncidod/diseases/hepatitis/resource/LabReportingAssessment.doc

Completeness of Lab ReportingCompleteness of Lab Reporting

Pick a time periodPick a time period (at least 3 months) (at least 3 months) Calculate:Calculate:

Total # HBsAg+ test results reported

Total # specimens that were HBsAg+

=Proportion of HBsAg+

tests reported, or completeness of

reporting

Timeliness of Lab ReportingTimeliness of Lab Reporting

Date of HBsAg Date of HBsAg serologyserology

Date HBsAg+ Date HBsAg+ report rec’d by report rec’d by

health depthealth dept

# Days between # Days between serology and serology and

reportreport

8-18-20078-18-2007 8-23-20078-23-2007 55

8-23-20078-23-2007 8-29-20078-29-2007 66

9-7-20079-7-2007 9-12-20079-12-2007 55

11-18-200711-18-2007 11-27-200711-27-2007 99

SUMSUM 2525

Timeliness of Lab ReportingTimeliness of Lab Reporting - cont’d - cont’d

=Mean reporting time for

the laboratory, or timeliness of reporting

Time period for reporting is typically set by state statute

Sum of column “# days Sum of column “# days between serology and between serology and

report”report”

Total # of serologies Total # of serologies evaluatedevaluated

25

4= 6.25 dayse.g. from the

previous slide

Tips for Working with LaboratoriesTips for Working with Laboratories

Contact other areas of health department Contact other areas of health department to see if activity is already being doneto see if activity is already being done

See if an existing lab report can be used See if an existing lab report can be used for the audit—ensure it is source datafor the audit—ensure it is source data

When making contact with labs, include:When making contact with labs, include: lab directorlab director techniciantechnician IT staffIT staff

Hospital Record ReviewsHospital Record Reviews Use to assess:Use to assess:

maternal HBsAg screeningmaternal HBsAg screening birth dose administrationbirth dose administration post-exposure prophylaxis (PEP) for infants post-exposure prophylaxis (PEP) for infants

born to HBsAg+ and unknown-status womenborn to HBsAg+ and unknown-status women Audit paired Audit paired maternalmaternal and and infantinfant records records Partner with health dept programs to collect Partner with health dept programs to collect

other perinatal data other perinatal data rubella antibodyrubella antibody • HIV• HIV group B strepgroup B strep • • early hearingearly hearing syphilissyphilis

Determining Sample SizeDetermining Sample Size

Tool available in Tool available in Guide to Life (Table 2.1)Guide to Life (Table 2.1)

http://www.cdc.gov/ncidod/diseases/hepatitis/resource/perinatalhepB.htm

Using the Using the Guide to LifeGuide to Life table table

Determine hospital’s annual number of Determine hospital’s annual number of deliveries deliveries

Decide where it falls in the “Birth Cohort Decide where it falls in the “Birth Cohort Size” categorySize” category

Estimate HBsAg screening and hepatitis B Estimate HBsAg screening and hepatitis B birth dose coverage for the hospitalbirth dose coverage for the hospital

Use the Use the lowerlower coverage to determine the coverage to determine the sample sizesample size

Sample Size ExampleSample Size Example

Hospital X had 648 deliveries last yearHospital X had 648 deliveries last year

The expected HBsAg screening rate is 95%The expected HBsAg screening rate is 95%

The state National Immunization Survey The state National Immunization Survey (NIS) birth dose rate is 65%(NIS) birth dose rate is 65%

Using the 65% coverage level, the sample Using the 65% coverage level, the sample size for Hospital X would be 111 recordssize for Hospital X would be 111 records

Hospital Data to CollectHospital Data to Collect

Delivery date/timeDelivery date/time Mother’s HBsAg test date/result/type in:Mother’s HBsAg test date/result/type in:

maternal recordmaternal record infant recordinfant record

Infant hepatitis B vaccination (yes/no)Infant hepatitis B vaccination (yes/no) Date/time of vaccinationDate/time of vaccination Infant HBIG administration (yes/no)Infant HBIG administration (yes/no)

Hospital Data to CollectHospital Data to Collect - cont’d- cont’d

Date/time of HBIGDate/time of HBIG Other variables as appropriateOther variables as appropriate

Hospital Policy SurveysHospital Policy Surveys

Use to assess whether hospitals have Use to assess whether hospitals have written policies/standing orders in place:written policies/standing orders in place: universal birth dose of hep Buniversal birth dose of hep B HBsAg screening of pregnant womenHBsAg screening of pregnant women PEP to infants born to HBsAg+ and unknown-PEP to infants born to HBsAg+ and unknown-

status womenstatus women documentation of maternal HBsAg status, documentation of maternal HBsAg status,

birth dose, and HBIG (if appropriate) in birth dose, and HBIG (if appropriate) in medical recordsmedical records

Conducting Policy SurveysConducting Policy Surveys

Mail surveyMail survey Telephone surveyTelephone survey In-person survey In-person survey

during hospitalduring hospital

medical record medical record

reviewsreviews

Keep in mind, policies do not always match practice, Keep in mind, policies do not always match practice,

so medical record reviews are essentialso medical record reviews are essential

Expected Births to HBsAg+ WomenExpected Births to HBsAg+ Women

CDC expected birth CDC expected birth estimates are a benchmark estimates are a benchmark for programsfor programs

CDC provides estimates for CDC provides estimates for states and some cities states and some cities

Estimates are calculated Estimates are calculated using:using: birth databirth data prevalence of chronic prevalence of chronic HBV infection by HBV infection by race/ethnicityrace/ethnicity

Expected BirthsExpected Births - cont’d- cont’d

Nationally, about 24,000 births expected Nationally, about 24,000 births expected annually, only 50% identified in 2005annually, only 50% identified in 2005

CDC encourages programs to improve CDC encourages programs to improve estimates of births to HBsAg+ womenestimates of births to HBsAg+ women

More information in presentation at:More information in presentation at:

http://www.cdc.gov/ncidod/diseases/hepatitis/http://www.cdc.gov/ncidod/diseases/hepatitis/resource/presentations/2007-04PeriGrantees/PDFs/resource/presentations/2007-04PeriGrantees/PDFs/May1-p0130-1LJ_2007PeriCoordMtg-May1-p0130-1LJ_2007PeriCoordMtg-

ExpectedBirths.pdfExpectedBirths.pdf

Assessing Case ManagementAssessing Case Management

Number of infants born to HBsAg+ womenNumber of infants born to HBsAg+ women Proportion of infants receiving Proportion of infants receiving

hep B vaccine & HBIG within 12 hours of birthhep B vaccine & HBIG within 12 hours of birth on-time completion of hep B vaccine serieson-time completion of hep B vaccine series post-vaccination testingpost-vaccination testing

Reasons for cases lost to follow-upReasons for cases lost to follow-up Proportion of screened and vaccinated Proportion of screened and vaccinated

household and sexual contactshousehold and sexual contacts

Assessing Case Mgmt - cont’dAssessing Case Mgmt - cont’d

Monitor case management outcomes:Monitor case management outcomes: compare program-wide rates (entire state or compare program-wide rates (entire state or

city) to national rates (CDC peritable)city) to national rates (CDC peritable) regionallyregionally locally (county level)locally (county level) by case workerby case worker quarterly or monthlyquarterly or monthly over time (trends)over time (trends)

Monitoring Birth DoseMonitoring Birth Dose

National Immunization Survey (NIS) data National Immunization Survey (NIS) data measures hepatitis B birth dose ratesmeasures hepatitis B birth dose rates

Hospital medical record reviewsHospital medical record reviews Immunization Information Systems (IIS)Immunization Information Systems (IIS)

hep B vaccine birth certificate data hep B vaccine birth certificate data automatically uploaded into IIS automatically uploaded into IIS

calculate birth dose coverage for cities, calculate birth dose coverage for cities, regions, and individual hospitalsregions, and individual hospitals

Continuous MonitoringContinuous Monitoring

Some aspects of the Some aspects of the program require program require continuous monitoringcontinuous monitoring case management case management

outcomes outcomes laboratory reportinglaboratory reporting

Plan ahead for Plan ahead for assessments your assessments your program will conduct program will conduct over the next year/sover the next year/s

Using Your ResultsUsing Your Results

Determine future direction and focus of Determine future direction and focus of program effortsprogram efforts

Provide feedback to:Provide feedback to: hospitalshospitals laboratorieslaboratories providersproviders health departmentshealth departments case workerscase workers