training program quality assurance and evaluation€¦ · training program quality assurance and...
TRANSCRIPT
Training Program QualityAssurance and Evaluation
Best Practices for Worker Training
Foundations
ÿMinimum Criteria Document (1910.120
Appendix E)
ÿCooperative Agreement Requirements
ÿAwardee Evaluations
Minimum Criteria DocumentSuggested Program Quality Control
Criteria1. Training Plan
2. Program Management
3. Facilities and Resources
4. Quality Control and Evaluation
5. Students
6. Institutional Environment andAdministrative Support
Quality Control andEvaluation
ÿAdvisory Committee and/or OutsideReviewers for overall policy guidance
ÿAdequate and appropriate qualitycontrol and evaluation program toaccount for …ÿ Instructor Performanceÿ Course evaluation for improvementÿ Student evaluations
Key Evaluation Questions
ÿ Quality and appropriateness of …ÿ Program objectives (clarity and
achievement)ÿ Facilities and staffÿ Course material and mix of classroom and
hands-on trainingÿ Assessment of program strengths and
weaknesses and needed improvements
Cooperative Agreements
ÿ NIEHS – Stewardship and Oversight Rolesÿ Guiding Language for Quality Assurance
and Evaluationÿ NIEHS - Review Panel - team of outside
experts and agency staffÿ Awardees – Lead responsibility in quality
control and internal evaluationÿ Established own evaluation systems
Cooperative AgreementRequirements
ÿ Independent Board of Advisorsÿ Appropriate training and expertise to
evaluate and oversee the proposed workertraining program
ÿFormal Quality Control and EvaluationPlanÿ Different forms depending on the the
nature of the student population andawardee’s program culture
Why EvaluateÿUse positive feedback to build and
expand programsÿLearn how to improve programsÿDetermine need of additional trainingÿDocument learning, confidence building
and workplace changesÿAccountability (legal/program
requirements)
Multi-Program Evaluation: ADescriptive Review February 1996
ÿReview of over 50 awardee evaluationreports and 13 grant related journalarticles and publications
ÿUsed in overall program review
Awardee Evaluations - ManyForms
ÿFocus on individuals and groupsÿQualitative (how and why) and
quantitative (how much and how many)ÿDescriptive (non-experimental designs)
and trying to infer cause (quasi-experimental designs)
Awardee Evaluations -FocusÿStudent perceptions of training
ÿ Many thousands of positive student ratings
ÿCourse materialsÿ Perceptions of usefulnessÿ Post-training use
ÿKnowledge, skills and decision-makingÿ Student self-assessmentsÿ Testing and performance assessments
Awardee Evaluations -Focusÿ Changes in awareness, concerns and
attitudesÿ Improvements in post-training response
actions to HAZMAT incidentsÿ Changes in personal protective practicesÿ Systematic changes in worksite programs,
policies, preparedness and equipmentÿ Catalyzing of additional site-based training –
training and sharing of information
An Evaluation of the NIEHSWETP - External Panel Report
“Not only has the NIEHS grant programprovided training to hundreds ofthousands of workers, managers andhealth and safety professionals, it hasalso made a substantial contribution to amore systematic, analytical and scientificapproach to training programdevelopment, delivery and evaluation interms of advancing the state of the art.”
December 28, 1995
Resource Guide forEvaluating Worker TrainingPurposesÿTo provide both general and specific
step by step guidance for bothexperienced and novice evaluators onhow to design and carry out anevaluation
ÿTo provide examples of evaluationinstruments
Resource Guide Content
ÿEvaluation overviewÿSpecial challenges to an evaluation
teamÿEvaluation methodsÿAnnotated bibliographyÿEvaluation instruments (for off-the-
shelf use or adaptation)
Evaluation: Building theCapacity to Learn
The Self-Sufficiency Researchand Evaluation Project (SREP) -A Participatory EvaluationModel
SREP Partnersÿ AFSCME - American Federation of State,
County and Municipal Employees with theUniversity of Massachusetts Lowell
ÿ PACE - Paper, Allied Industrial, Chemical andEnergy Workers International Union with theLabor Institute and New PerspectivesConsulting Group
ÿ UAW - United Auto Workers with University ofMichigan
SREP - A Multi-OrganizationalCollaborative
ÿThree partners—union-basedoccupational safety and healtheducation programs
ÿTeam-based—composed ofworker-trainers, program staffand/or evaluators
SREP OverviewWorkshops Team Evaluation Projects
Developing Evaluation PlansGathering and Analyzing Data
Aug 1998
Collective ReflectionsDeveloping Lessons Learned
May 1999
Research and Evaluation OverviewIntroduction to Program Theories
May 1998
Developing Meaning andPromoting the Use of Findings
Jan 1999
Refine data collection plans ,begin data and analysis
Develop program diffusion,prepare lessons learned report
Develop and refine evaluationquestions and designs
Ongoing data collection, analysisand report generation
Description of SREP TeamProjects
ÿSchool district: Short survey and focusgroup (AFSCME)
ÿMunicipality: Pilot individual interviews(AFSCME)
ÿOil Refinery: In-plant labor managementrefinery team use of “Charting HowYour Program Works:” and monitoringnew safety and health initiative (PACE)
Project Description (continued)
ÿProgram-WideÿWorker understanding of systems of
safety—card sort focus group (PACE)ÿWorkplace Impact—phone interviews
(UAW)ÿWeek-long Training Conference
ÿQuick feedback from and back totraining program participants (UAW)
Model of Worker-Led, Team-based Participatory Evaluation
1. Builds a community united in a sharedcommitment to the rights of all workersto safe and healthy workplaces.
2. Actively involves workers in all aspectsof evaluation.
3. Is a collective effort—within andamong partner organizations—thatdraws upon each other’s insights,strengths and experiences
Model (continued)
4. Understands evaluation as a processof continuous learning, rather thanbeing an end product
5. Provides important ways to measureand document program successes.
6. Recognizes the importance ofidentifying program values and goalsto guide evaluations.
Worker-Led, Team-BasedEvaluation
TraditionalWho:ÿ Evaluation consultant, program
administratorWhat they do:ÿ Consultant designs, conducts,
analyzes and writes report
ÿ Worker trainers and trainersmay distribute and collectevaluation forms
ÿ Consultant recommendschanges and future directionsfor programs
ParticipatoryWho:ÿ Team of worker trainers,
trainers, evaluation consultant,program administrator and staff
What they do:ÿ Team decides evaluation focus,
design, data collectioninstruments, analysis, etc.
ÿ Consultant may provide morehands-on work while thoseinternal to program provideideas and feedback
ÿ Team reflects on findings anddecides implications for futureprogram directions
Worker-Led, Team-BasedEvaluation (continued)
TraditionalWhen:ÿ At the end of projectHow/Who:ÿ Formal written report for
program administrators,funders
Use:ÿ To make judgments
ParticipatoryWhen:ÿ Throughout projectHow/Who:ÿ Variety of formats—formal
written reports, group activities,newsletters—for workertrainers, programadministrators, funders, staff
ÿ Use:ÿ Learn how program works to
guide ongoing improvements
ÿ Expand original learning
SREP Is OngoingÿBeginning May 30 SREP partners will
begin a new round of three-dayworkshops that will involve participantsin training, planning and organizing tocarry out participatory evaluations oftheir programs
ÿWe welcome inquiries about joining us