DATA VALIDATION

NORTH CAROLINA’S EXPERIENCE

NC Division of Employment & Training 2

Program Year 2002 Data Validation

We contracted with our former Finance Director (retired) to complete PY2002 data validation

There were no electronic records provided to the performance or MIS staff after the project was completed

No records were sent to DOL A sparse state report was provided and

was disseminated to the Local Areas

NC Division of Employment & Training 3

Program Year 2003 Data Validation

Challenges:

No raw data from prior year No state policies or procedures No practical experience No training (beyond DOL

workshops) No staff

NC Division of Employment & Training 4

Program Year 2003 Data Validation

First Steps: Convene a Data Validation Committee

to establish policies, procedures and training materials (August 2004) Committee Membership:

DET Staff MIS Super Users Case Managers

NC Division of Employment & Training 5

Data Validation Committee

DET Staff MIS Director, Planners (2), Field Staff (Monitor),

Technical Writer, Performance Manager MIS Super Users (7)

MIS Super Users are MIS and performance specialists in each Local Area

Case Managers (6) For adult, dislocated worker and youth

programs

NC Division of Employment & Training 6

MIS Super Users

MIS Super Users are specialists in each Local Area responsible for: Attaining a high-level proficiency with the

WorkforcePlus™ system and functioning as the Local Area’s expert user of the participant MIS

Attending and actively participating in all MIS User’s Group meetings

Training Local Area end users in the proper use of WorkforcePlus™

Providing ongoing technical assistance and remedial training on WorkforcePlus™ issues

NC Division of Employment & Training 7

MIS Super Users Providing feedback to DET concerning MIS

issues Conducting continual review of Local Area

data, monitoring the quality of data entry and the proficiency of users and promptly addressing identified problem areas

Understanding the relationship between the Local Area’s performance goals, program design, and program outcomes and ensuring the proper recording of program outcomes to maximize performance levels

Attaining proficiency in extracting performance and management reports from WorkforcePlus™

NC Division of Employment & Training 8

Data Validation Committee

First Steps: Train DV Committee members Agree on the WorkforcePlus™ source

documentation that will be acceptable for NC data validation and amend DOL source list to be NC specific

Determine what state policies and procedures need to be put in place specific to data validation

NC Division of Employment & Training 9

Data Validation Committee

Source Documentation Developed a revised DV source document

chart, deleting references in DOL’s chart that were not applicable to NC and adding specific WorkforcePlus™ (MIS) references for data entry screens and printouts that could be used for DV documentation

A WorkforcePlus™ ‘definitions and instructions’ document that had been drafted by our Super Users group was adopted as state policy

NC Division of Employment & Training 10

Data Validation Committee State Policies and Procedures Three Local Area Issuances were developed

and released in December 2004 Entering Activities and Dates for First WIA

Activity Clarified service requirements for initial WIA

registration in NC

WIA Application Time Limit

WIA Data Element Validation and Participant File Structure Formalized a case file structure and content

requirements across all Local Areas

NC Division of Employment & Training 11

Data Validation Policies Entering Activities and Dates for First

WIA Activity (Local Area Issuance 2004-15) Established specific employment activities as the

initial service to be provided for every WIA participant

Provides statewide consistency in setting the WIA registration date

The “start date” must equal the “creation date” in WorkforcePlus™ for the initial activities

For Adults and Dislocated Workers, the initial enrollment must be “case management”

For Youth, the initial enrollment must be “comprehensive guidance and counseling”

NC Division of Employment & Training 12

Data Validation Policies WIA Application Time Limit (Local

Area Issuance 2004-15) Reinforces that individuals must be eligible at

the time of enrollment into WIA, not just at application

Resolves concerns we had regarding a lengthy time lapse between eligibility determination and enrollment into the WIA program.

Requires a new application to be completed if more than 60 days have lapsed since the application date/eligibility date

NC Division of Employment & Training 13

Data Validation Policies WIA Data Element Validation and

Participant File Structure (Local Areas Issuance 2004-17) Intended to improve compliance with data

validation requirements Requires Local Areas to develop a formal file

structure policy (or amend an existing file structure policy) for all participant files

Requires each file to have a file checklist at the front of the file indicating where supporting documentation can be found in the file

Requires each DV data element to be labeled within the file

NC Division of Employment & Training 14

Data Validation Policies File Structure (continued)

Each DV data element has a ‘field code’ consisting of a cohort, category, and DOL’s reference number

Cohorts: A (adult), DW (dislocated worker), OY (older youth)

or YY (younger youth) Category:

E – Eligibility/Intake/Application A – Employment Activities XP – Exit and Post-Program Activities

DOL Reference Number: corresponds to the elements’ reference numbers in the DOL WIA Data Validation Handbook

NC Division of Employment & Training 15

Data Validation Policies File Structure (continued)

Field codes must be placed by each data elements in the file checklist(s) and the field codes added to the source documentation in the files

The policy also requires that certain documents/screen prints also be placed in each file: Signed and dated application 1st activity enrollment screen Case profiles Case/activity notes Supplemental data screens (where applicable) Leave Program screen Employment Plan screen

NC Division of Employment & Training 16

Data Validation Policies

File Structure (continued) All files for participants active in WIA as of July

1, 2004 are required to be in the file structure format and contain all source documentation for data validation

Some Local Areas had file structure ‘parties’ where the case managers met and assembled participants’ files in the new structure

Feedback from the case managers has been overwhelmingly positive

Most Local Areas use sectioned file folders, although it is not required by the state

NC Division of Employment & Training 17

Data Validation Committee Training

Developed a data validation training outline and materials

Created two training manuals For local case managers For state data validation staff

Initial DV training for MIS Super Users and select case managers was held over two days in February 2005

Super Users in each Local Area were responsible for training their case managers

Quarterly MIS Users Groups meetings provide a forum for continued dialogue between the state and Local Area staff

NC Division of Employment & Training 18

Data Validation Committee Additional Training

Case notes may be used in North Carolina to validate select data elements

During Program Year 2003 data validation, it was found that some of the case notes in the files were marginal, at best

An RFP was released to solicit bids for the provision of training for data validation specific “Writing Effective Case Notes”

We provided 16 training sessions and trained 643 case managers at the Workforce Development Training center

Local Areas also arranged for additional “Writing Effective Case Notes” training

NC Division of Employment & Training 19

Data Validation Committee

Additional Training, continued There are plans to offer ongoing data

validation training to end users and case managers (after the revised reporting requirements are released by DOL)

Data validation training will probably be combined with common measures training

One day sessions with half of the day devoted to common measures and the other half devoted to data validation

Case Managers tend to want training specific to the individuals they serve (i.e., youth separate from adult and dislocated workers)

NC Division of Employment & Training 20

Staffing DET has used 4 time-limited/temporary staff

to complete data validation for the last 2 years Pay is between $11.18 - $16.62 per hour No fringe benefits Receive reimbursement for mileage, meals

and lodging There are two data validation teams – East

and West Both teams go as a group to Local Areas with

more than 100 files in the sample to minimize disruption to the Local Area staff

NC Division of Employment & Training 21

Staff Training

The DV staff receive a week of training on WIA programs, the NC participant database, and federal and state DV policies and procedures

25 participant files are ‘validated’ in house to allow staff to become familiar with the information they will be seeing in participant file folders and to practice validating the elements

Once the files are validated, additional instruction and clarification are provided where needed

NC Division of Employment & Training 22

Staff Training

All validators and the manager go to the first Local Area

A select group of files from each cohort at the Local Area (adult, dislocated worker, older youth, and younger youth) is reviewed by all 5 individuals

DV results for all of these select files are compared and discussed

The file(s) are re-reviewed by the group to determine if a questioned data element is supported by source documentation

NC Division of Employment & Training 23

Actual Data Validation

North Carolina has an official sample and an unofficial sample

Every Local Area has a data validation visit The data validation schedule is arranged by the

manager The validators are required to call Local Area

contacts the week prior to the visit to confirm and to inform them when they will arrive (day and time)

Validators hold entrance and exit conferences with the Local Area staff

Validators complete hard copy worksheets at the Local Area.

For every data element that is failed, the validator has to document why the element was failed

NC Division of Employment & Training 24

Actual Data Validation Upon return to the state office:

Each validator writes an individual report about his/her general findings at the Local Area and summarizes recommendations for the Local Area

Validators enter DV findings into the DOL software

Validators create a comment sheet for each Local Area containing the individuals’ names, data elements failed and why the elements failed (by cohort)

State staff compiles reports for each Local Area Numerical report Narrative report (including comments on all

failed data elements

NC Division of Employment & Training 25

Actual Data Validation

After the completion of all data validation for the state Individual reports are completed for each Local

Area DOL “WIA Data Element Validation

Summary and Analytical Report” DET Report Narrative report (including comments on all

failed data elements

NC Division of Employment & Training 26

Example 1 of Local Area ReportLAXXX - Dislocated Worker (DOL Report)

Wrong SSN 0 Invalid Records 0

Missing Records 0 Unable to Locate Records 0

# of Errors

OverallError Rate

Reported DataError Rate

02 Date of Birth 1.00 2.30% 2.30%

03 Disability 0.00 0.00% 0.00%

04 Veteran 2.00 4.60% 100.00%

06 UC Claimant 5.00 10.65% 17.46%

08 TANF 0.00 0.00% 0.00%

09 Cash Assistance 0.00 0.00% 0.00%

10 Displaced Homemaker 0.00 0.00% 0.00%

11 Dislocation Date 3.00 6.90% 13.32%

18 WIA IB Registration Date 5.00 11.50% 11.50%

19 WIA Exit Date 35.00 86.20% 86.20%

21 Intensive Service Date 5.00 11.50% 11.50%

22 Training Service Date 4.00 9.20% 14.02%

NC Division of Employment & Training 27

Example 2 of Local Area Report

LAXXX - Dislocated Worker (DET Report)

Wrong SSN 0 Invalid Records 0

Missing Records 0 Unable to Locate Records 0

# ElementsSelected

#Pass

#Fail

%Pass

%Fail

02 Date of Birth 41 40 1 97.56% 2.44%

03 Disability 0 0 0 0.00% 0.00%

04 Veteran 2 0 2 0.00% 100.00%

06 UC Claimant 25 20 5 80.00% 20.00%

08 TANF 0 0 0 0.00% 0.00%

09 Cash Assistance 0 0 0 0.00% 0.00%

10 Displaced Homemaker 0 0 0 0.00% 0.00%

11 Dislocation Date 21 18 3 85.71% 14.29%

18 WIA IB Registration Date 41 36 5 87.80% 12.20%

19 WIA Exit Date 41 6 35 14.63% 85.37%

21 Intensive Service Date 41 36 5 87.80% 12.20%

22 Training Service Date 27 23 4 85.19% 14.81%

NC Division of Employment & Training 28

Reports

DOL’s DV report may tell them what they want to know, but it is confusing to the Local Areas How many of you know (without looking) what

‘Overall Error Rate’ and ‘Reported Data Error Rate’ mean?

How many of your Local Areas’ staff understand the meaning of the error rates?

The DET DV report puts information in perspective for the Local Area: how many elements were selected for validation and how many passed or failed?

NC Division of Employment & Training 29

What Have We Learned?

Local Areas want training, structure and guidance

For the most part, case managers like the file structure requirements (once they got their files in order with all requisite information)

Program Year 2004 data looks better than prior years’ data

We anticipate that as the exit date range for DV catches up with our DV policies, our data validation performance will improve significantly.

NC Division of Employment & Training 30

QUESTIONS?

NC VALIDATION EXPERIENCE

NC Division of Employment & Training 31

Contacts and References

Martha CranfordNC Division of Employment & Training313 Chapanoke Road, Suite 1204316 Mail Service CenterRaleigh, NC 27699-4316Phone: 919-329-5290Fax: 919-661-4770mcranford@nccommerce.com

NC DET State Policieswww.ncdet.com

Case Notes TrainingMary Ann Lawrence (Center for Workforce Learning) MALaw528@aol.com www.mylearningchain.com