TAH Project Evaluation
Great - Yet Realistic - ExpectationsJeff Sun, Sun Associates
Our BackgroundInitiative-wide Qualitative Evaluation
District-level initiatives
State-level initiatives
Strategic Planning
STEM, Science, Instructional Technology, Social Studies, New Media
Evaluation of Teacher Professional Development
10 TAH projects (7 of these are current)
Qualitative is the EmphasisMakes the connection between what
projects say they want to do and what participants say that projects have done.
This data is invariably qualitative and accounts for a wide range of effects
Quantitative data helps paint the picture in broad outlines...
...but qualitative data helps you understand what the quantitative data means and to use the evaluation strategically
This Presentation
We want to talk about how to
Create a good evaluation
Use evaluation strategically
Work effectively with evaluators
Four Functions for Evaluation
Evaluation performs four meaningful functions in a project’s life...
As a part of the proposal
To clarifying a project’s work
To provide formative assessment of Progress and Implementation
Summatively, to account for the project’s success (or lack thereof)
First Function - The Proposal
As expressed in the logic map, evaluation helps explain:
Need for the project
The project’s actions in response to those needs
Who the project will serve
What the project expects to produce
All in one tidy package in the proposal text!
Issues Related to Function 1We need time to develop the evaluation plan
Ideally, we come in when the basic work is set, but the details are not fully fixed
We need space to fully describe the evaluation process
It’s hard to create this plan in 2 paragraphs or a half page
We need connect with those who will be actually implementing the project if funded
Proposal-writers will help you get funded...but it’s often no coincidence that they’re nowhere to be found once you get the grant.
Second Function - ClarificationOnce you’re funded, the real work begins
Use the evaluation plan as an organizer for talking with partners about
what the project will be doing
expectations for partners and participants
This discussion clarifies goals and develops a shared vision for project success
Out of this discussion come the project’s performance indicators...which we organize into rubrics
It would be nice to have all of your indicators before funding, as part of the proposal
But realistically, there’s never time for that
Projects need their own indicators
ED’s indicators are not the same and cannot replace your own indicators
Issues Related to Function 2
We can’t help if we aren’t at the table
Often, we have to really bug projects to bring us on board early
Evaluation is too often seen as the thing that happens after the work is done...not as the thing that can get work done.
We read the proposal and base the actual evaluation on what is written
This can be a problem when the proposal was written by the now-absent proposal-writer
The clarification discussion - assuming it happens early enough - is where this problem is resolved
Third Function - Formative
Collecting the data and sharing it as the project progresses
Measuring what the project is doing against what it said that it would do
Using indicator rubrics to account for data that is highly qualitative
Issues Related to Function 3
Our challenge is to identify data collection points that get at the essence of the project and its impact.
This can be one of the most important parts of our work.
Questions continue the process of clarifying the project work through participant feedback on impact
Next most difficult is connecting directly with participants
Fourth Function - Summative
Annual comparison of progress to goals
Sometimes more reporting than “annual” is important
More reporting might be useful in the beginning, less reporting might be necessary as the project matures
Summative data should play a role in the development of new proposals
Issues Related to Function 4
Qualitative data is valuable for projects, but does not seem to be something that ED is interested in.
This is a problem because projects cannot really produce meaningful data that ED is interested in.
So, do we spend extra time (money) producing two sets of data?
Or do we sacrifice the meaningful data to concentrate on producing what we’re told to create?
This has been the on-going debate among TAH evaluators on a national level
Questions? Comments?
What challenges have you faced when considering project evaluation?
How have you used evaluation in different stages of your project? (from planning to reporting)
[email protected] or 978-251-1600 x 204