the measurement and evaluation of the ppsi oregon pilot program paint product stewardship initiative...
TRANSCRIPT
The Measurement and Evaluation of the PPSI Oregon
Pilot Program
Paint Product Stewardship Initiative
Portland, Oregon
December 10, 2009
Matt KeeneOffice of Policy, Evaluation Support DivisionUnited States Environmental Protection Agency
Jennifer NashDirector of Policy and ProgramsProduct Stewardship Institute, Inc.
Purpose of the Presentation
Provide an overview of the process of integrating measurement and evaluation into the Oregon Pilot Program and describe the work of the evaluation committee
Presentation Outline
1. Introduction to Program Evaluation
2. Participatory, Utility-Focused Evaluation
3. Integrating Evaluation into the OR Pilot
• Evaluation Questions
• Data and Measures
4. Questions, Comments, Discussion
Program Evaluation
Definition• A systematic study that uses measurement and
analysis to answer specific questions about how well a program is working to achieve its outcomes and why.
Orientation/Approaches to Evaluation• Accountability
External Audience
• Learning & Program ImprovementInternal/External Audiences
Conduct a program evaluation to gain value-added insight:• Learn what works well, what does not, and why• Learn how a program could be improved• Help inform front-end program design• Provide performance information for
accountability purposes
Measurement and Evaluation
Program Evaluation:
A systematic study with a definable methodology that uses measurement and analysis to answer specific questions about how well a program is working to achieve its outcomes and why.
Performance Measurement:
A basic and necessary component of program evaluation that consists of the ongoing monitoring and reporting of program progress, using pre-selected performance measures.
The Evaluation Committee: Participatory, Utility-Focused Participatory, Utility-Focused
Evaluation
Evaluation Committee
• Funding and Support
• Purpose
• Progress until now
Evaluation Template
• Evaluation Questions
• Audiences
• Approach
• Data Sources
1. Team 2. Mission3. Goals & Objectives4. Logic Model
Building Measurement & Evaluation into the OR Pilot
Program
PPSI Oregon Pilot Program
Program
QuestionsDocumentation
Measures
1. Context2. Audience3. Communication4. Use
1. Data Sources2. Collection Methods & Strategy3. Analysis Tools4. Data Collection5. Data Management
1. Evaluation Methodology2. Evaluation Policy
is our program
Describing the Program
Evaluation Team
Mission
Goals and objectives
Logic model
Program
The Program
1. Team 2. Mission3. Goals & Objectives4. Logic Model
Integrating Evaluation into Program Design
1. Team 2. Mission3. Goals & Objectives4. Logic Model
Integrating Evaluation into Program Design
Program
Questions
What are the critical questions to understanding the success of the OR program?
What contextual factors may influence the answers to each question?
Who are the audiences for each question?
•What’s the best way to communicate with each audience?
•How might each audience use the answer to each question?
Evaluation Questions
1. Context2. Audience 3. Communication4. Use
1. Team 2. Mission3. Goals & Objectives4. Logic Model
Integrating Evaluation into Program Design
Program
Questions
1. Context2. Audience 3. Communication4. Use
What can we measure to answer each question?
Where is the information for each measure?
How can we collect the information? What will be our strategy?
What analytical tools will give us the most useful information?
How will we implement the collection strategy?
How will we manage the data?
Measures
1. Data Sources2. Collection Methods & Strategy3. Analysis Tools4. Data Collection5. Data Management
Measures
1. Team 2. Mission3. Goals & Objectives4. Logic Model
Integrating Evaluation into Program Design
Program
Questions
1. Context2. Audience 3. Communication4. Use
1. Data Sources2. Collection Methods
& Strategy3. Analysis Tools4. Data Collection5. Data Management
Documentation
Evaluation Methodology
Systematic approach structures a methodology
Evaluation Policy
Guides strategy and planning for evaluation and program management
Method and Policy become part of the OR Program…and the National System?
1. Evaluation Methodology2. Evaluation Policy
Measures
Documentation
1. Team 2. Mission3. Goals & Objectives4. Logic Model
Building Measurement & Evaluation into the OR Pilot Program
PPSI Oregon Pilot Program
Program
QuestionsDocumentation
Measures
1. Context2. Audience3. Communication4. Use
1. Data Sources2. Collection Methods & Strategy3. Analysis Tools4. Data Collection5. Data Management
1. Evaluation Methodology2. Evaluation Policy
General Evaluation Questions
Was the Pilot program collaborative & cooperative?
What is the Paint Stewardship Organization and how did the financing mechanism work?
Did consumer awareness and/or behavior change? Why?
Was the Oregon paint management system environmentally beneficial, economical, and convenient?
What are alternative products and markets for post-consumer paint?
How is the evaluation useful to the rollout of a national paint management system?
Measures and Data
Potential Measures
• Stakeholder ratings of collaboration
• Inventory of outreach materials
• Change in consumer recycling behavior
• Volume collected• Cost/unit collected• Change in demand
for paint products
Data Sources
• Consumers
• PPSI
• OR DEQ
• PSO
Collection Methods
• Focus Group
• Ongoing reporting
• Document review
• Survey & Interview
Discussion
Challenges ahead
• Timeframe
• Baselines
• Reporting
• Interpreting results
What is most important to understand about how the program is working?
What are your concerns?
Thank You!!
Environmental Evaluators NetworkLinkedIn Group: environmental evaluators networkwww.nfwf.org/een
Jennifer Nash Matt Keene617.236.4853 [email protected]
Director of Policy and Programs Evaluation Support DivisionProduct Stewardship Institute, Inc. Office of Policy
U.S. Environmental Protection Agency