Getting Started With Your ATE Evaluation

ATE PI ConferenceOctober 24, 2012

This material is based upon work supported by the National Science Foundation under Grant Nos. 0802245 0802245 and 1204683. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of NSF.

To promote the goals of the ATE program by- partnering with ATE projects and centers to

strengthen the program's evaluation knowledge base

- expand the use of exemplary evaluation practices

- support the continuous improvement of technician education throughout the nation

’s Mission

- Understand the role evaluation should play in your ATE project

- Have clear expectations for your evaluator

- Know where to turn if you need more guidance about your ATE evaluation

Session Objectives

Formative EvaluationLearn how the project could be improved as it is implemented

Summative EvaluationProvide evidence of the quality and impact of your achievements

AccountabilityDocument what you did with the grant money

Evaluation Purposes

The most important purpose of evaluation is not to prove, but to improve.

―Daniel Stufflebeam

Formative Evaluation

Project staff…- make time for evaluation- are open to hearing

negative findings - use feedback to inform

decision making

Formative Evaluation

Evaluator…- learns about the project,

its context, and stakeholders

- gathers data in sound, efficient ways

- provides timely feedback- guides project staff in

understanding and using results

Formative Evaluation

What is the project’s quality? What are the project’s impacts?

Summative Evaluation

What is the project’s quality?What are the project’s impacts?- changes in knowledge, skills, abilities, attitudes,

performance, practices, or policies- show up at the individual, program, institution,

organization, regional, or national level- occur in the short term, long term, or in between

Summative Evaluation

Document… - What you did- Who you reached- Who you worked with - What you produced

(outputs)

Accountability

Accountability

www.evalu-ate.org/about_us

Create a project vita- Mission- Goals- Funding- Activities- Products- Staff- Students- Contributors

& collaborators- Consultants

Formative Evaluation

How is the project doing and how can

it improve?

AccountabilityWhat was done, who was served, who contributed,

what are the outputs?

Summative EvaluationWhat is the quality and

impact of the project?

Evaluation Purposes & Uses

Formative Evaluation

How is the project doing and how can

it improve?

AccountabilityWhat was done, who was served, who contributed,

what are the outputs?

Summative EvaluationWhat is the quality and

impact of the project?

Evaluation Purposes & Uses

- Feedback to internal stakeholders

- Reflection and redirection

Formative Evaluation

How is the project doing and how can

it improve?

AccountabilityWhat was done, who was served, who contributed,

what are the outputs?

- Summative report to key stakeholders- FastLane/annual report to NSF - ATE Impacts book - Annual ATE survey - New proposals- Promotion and dissemination

Summative EvaluationWhat is the quality and

impact of the project?

Evaluation Purposes & Uses

- Feedback to internal stakeholders

- Reflection and redirection

Report Sections- Participants- Activities & findings- Publications & products- Contributions

Annual Reports to NSF

www.fastlane.nsf.gov

See also: EvaluATE’s summary of FastLane report components

evalu-ate.org/resources Keyword: FastLane

Annual Reports to NSF

For example, # of students and their

demographic characteristics

# of professional development participants

# of materials developed

www.evalu-ate.org/annual_survey

Annual ATE Survey

Student impact data

Workforce impact data

www.atecenters.orgwww.ateprojectimpact.org

ATE Projects/Centers Impact Book & Website

“The Project Description must begin with the subsection on Results of Prior Support…. This subsection must contain specific outcomes and results including metrics to demonstrate the impact of the activities undertaken including evidence of the quality and effectiveness of the project's deliverables.

—ATE Program Solicitation

“

Results of Prior NSF Support

Be an Informed ConsumerJoint Committee on Standards for

Educational EvaluationThe Program Evaluation Standards

www.jcsee.orgAmerican Evaluation Association Guiding Principles for Evaluators

www.eval.org

Canadian Evaluation SocietyCompetencies for Evaluation Practice

www.evaluationcanada.ca

What should we expect to receive from our evaluator?

― Participant in the 2010 ATE PI Conference

Getting Started Workshop

- Detailed evaluation plan- Opportunities to review

and provide feedback on draft plans and materials

- Interim reports- Annual reports- Knowledge and insights

you could not have obtained on your own

Evaluation Deliverables

EVAL REPORT

EVALUATOR PI

PROGRAM OFFICER

OTHER PROJECT STAKEHOLDERS, E.G.:partners, advisors, participants,

host administrators

Reporting

EVAL REPORT

EVALUATOR PI

PROGRAM OFFICER

OTHER PROJECT STAKEHOLDERS, E.G.:partners, advisors, participants,

host administrators

Reporting

EVAL REPORT

EVALUATOR PI

PROGRAM OFFICER

OTHER PROJECT STAKEHOLDERS, E.G.:partners, advisors, participants,

host administrators

Reporting

EVAL REPORT

EVALUATOR PI

PROGRAM OFFICER

OTHER PROJECT STAKEHOLDERS, E.G.:partners, advisors, participants,

host administrators

Reporting

EVAL REPORT

Ways of reporting evaluation results to your program officer

PI PROGRAM OFFICER

Ask your program officer how he/she prefers the evaluation results to be conveyed.

Complete original report

Originalexecutive summary

Evaluators’ summary in

annual report

Reporting

EVAL REPORT

Ways of reporting evaluation results to your program officer

PI PROGRAM OFFICER

Ask your program officer how he/she prefers the evaluation results to be conveyed.

Complete original report

Originalexecutive summary

Evaluators’ summary in

annual report

ASK!

Reporting

Using EvaluATE

www.evalu-ate.org

NewslettersFeature articles by experienced ATE PIs and evaluators

Q & A

Definitions & discussion of evaluation terms

Resource highlights

Resource Library

Search by - Evaluation task- ATE topic- Keyword- Author

Evaluator Directory

Search by - location- discipline- project type- keyword

Annual Survey

Survey form FAQsReports

Webinars

Upcoming eventsMaterials from past events:- Recordings- Slides- Handouts

Next Webinar

Register at www.evalu-ate.org/events

Conference ActivitiesConcurrent with this sessionPI and evaluator workshop on evaluationTonight Showcase #209-210 Tomorrow Afternoon Birds of a Feather Session:Beyond Satisfaction and Short-Term Self-Reports: Evaluating the Impact of Your ATE Grant

Thank You!

Advice from the Trenches Elaine’s Golden Rule:

The one who has the gold rules!i.e., don’t disperse grant funds to contractors or partners until obligations have been met

“ “

Elaine CraftATE PI & Evaluator

Advice from the Trenches Set up regular data

collection milestones or checkpoints so that data is being collected as part of the normal process of carrying out the grant and doesn't have to be ‘bolted’ onto the process of the grant after it is underway.

““

Karl KappATE Evaluator

Advice from the Trenches The evaluator is the expert

in evaluation but you are the expert on your project. If you feel that the evalua-tion is not providing the information you need, then talk to your evaluator. Do it immediately. … You can change evaluators but first, communicate, communicate. communicate.

““Jane Ostrander

ATE PI & National Visiting Committee chairperson