focusing an evaluation ben silliman, youth development specialist nc 4-h youth development
Post on 27-Mar-2015
218 Views
Preview:
TRANSCRIPT
Focusing an EvaluationBen Silliman, Youth Development Specialist
NC 4-H Youth Development
Keys to Focusing an Evaluation
Discerning readiness Determining purpose Deciding on level/type Distilling questions Developing plan Directing implementation
Discerning ReadinessEvaluate no program before its time
Internal ChemistryClear objectivesCapable, trained staffRecruitment of target audienceTime to implement and refine programTime and money to conduct evaluationExpertise to plan, gather, analyze data
Discerning ReadinessEvaluate no program before its time
External ChemistryClear objectives—understood by stakeholdersConstructive relationships, staff and stakeholdersReasonable timeline for program developmentResources and assistance to evaluateSupport for gathering, analyzing, using data
Determining PurposeSweeten to taste
Stakeholder PerspectivesCitizens: Does it make a difference?Funding organization: Is it cost-effective?Parents: Is my child safe? learning?Staff: Are we running a quality program?Youth: Are we having a good time?
Determining PurposeSweeten to taste
Evaluation Standards that shape purposeUtility, serving the needs of intended usersFeasibility, examining if the evaluation is realistic,
prudent, diplomatic, practical Propriety, assessing legal, ethical, in the best
interests of all stakeholders Accuracy, technically and broadly conveying the
worth of the program
Deciding on Level/TypeChecking the vintage: Taste test
Formative EvaluationOngoing feedback about performance needs assessment to determine target audience, needs,
best strategies evaluability assessment to determines feasiblliiy, usefulness structured conceptualization to define the program or
technology, the target population, and the possible outcomes implementation evaluation monitors the fidelity of the
program or technology delivery process evaluation investigates the process of delivering the
program or technology, including alternative delivery procedures
Source: Research Methods Knowledge Base
Deciding on Level/TypeChecking the vintage: Taste test
Summative EvaluationLong-term effects or outcomes of a program outcome evaluations explore whether and how much a program
produces targeted results impact evaluation examines the broader effects of the program or
technology as a whole cost-effectiveness and cost-benefit analysis address questions
of efficiency by standardizing outcomes in dollar costs and values secondary analysis reexamines existing data to address new
questions or use methods not previously employed meta-analysis integrates the outcome estimates from multiple
studies to arrive at an overall or summary judgement on an evaluation question procedures
Source: Research Methods Knowledge Base
Distilling QuestionsMade to Order
Program Quality and ProcessBest practices/Youth experienceProgram improvement
Distilling QuestionsMade to Order
Youth OutcomesShort-term: Knowledge-Attitude-Skill-AspirationsMid-term: Behavior Change-ApplicationLong-term: Social-Economic-Environment-Cultural
Change
Developing a PlanTapping the Casks
Selecting Indicators (examples)Health: healthy weight; appropriate exerciseScience: skill in soldering, selecting seeds, calf careCitizenship: knowledge/skill in parliamentary
procedure
Developing a PlanTapping the Casks
Selecting IndicatorsHealth
diet/calorie journalmeal plan for retreat
Scienceaspirations for learningself-efficacy
Citizenship: communityservice plantime priorities
Developing a PlanTapping the Casks
Data SourcesChildYouth leaderParentsOthers impacted by child’s work (team members or
helpees)Generic data (School records, community
Developing a PlanTapping the Casks
Data Collection OptionsTest-Survey-InterviewPortfolio-DemonstrationObservation
Developing a PlanTapping the Casks
Data Collection ProtocolEthicalEducationalEffective
Developing a PlanTapping the Casks
Analyzing dataQualitative: Trends, anecdotes, commentsQuantitative: Individual growth, group patterns;
correlated elements
Directing ImplementationManaging the Vineyard
Step-by-Step Procedures Budgeting Timeline Monitoring roles Critique
You can’t find out what you want to know until you know what you want to find.
top related