11

Instruments and Data Collection

New Mexico AmeriCorpsApril 20, 2006

Sue Hyatt, Project STAR Coach

22

AgendaAgenda

What are instruments?What are instruments?

Measuring outputs and Measuring outputs and outcomesoutcomes

Linking indicators and targets to Linking indicators and targets to instrumentsinstruments

Data Collection IssuesData Collection Issues

PM PlanPM Plan

33

Performance Measurement Six Steps

1. Develop Logic Model

& Project Plan

2. DevelopPM Plan

3. Developand Test

Instruments4. Collect

Data

5. AnalyzeData

6. ReportResults

ContinuousProgram

Improvement

44

Types of Performance Measures

OutputsOutputs Intermediate OutcomesIntermediate Outcomes End OutcomesEnd Outcomes

55

What is an Instrument?What is an Instrument?

Document or form used to collectDocument or form used to collectinformation from data sourceinformation from data source

Data source: Person, place or group that Data source: Person, place or group that provides you with information to measure provides you with information to measure indicatorsindicators

Examples: Attendance rosters, tally sheets, Examples: Attendance rosters, tally sheets, questionnaires, checklistsquestionnaires, checklists

Adapt existing instruments or develop your ownAdapt existing instruments or develop your own

66

Advice about InstrumentsAdvice about Instruments

Make sure instrument will measure Make sure instrument will measure

indicatorsindicators

Consider who will Consider who will administeradminister instrument instrument

Consider who will Consider who will complete complete instrumentinstrument

Consider Consider whenwhen to administer instrument to administer instrument

Consider data Consider data accessibilityaccessibility

77

Instrument Considerations

Connect to results– Measures the identified PM

Aim for appropriate method– Measures the correct population (data source)

Lower burden of use– e.g. Captive audience, (part of project)– Balance between “best” method and

“manageable” method Assure that sites have instruments or summary

tool

88

ExerciseExercise

What Does the What Does the Instrument Measure?Instrument Measure?

99

Indicators and TargetsIndicators and Targets

For each result (output, intermediate outcome and end outcome), you need to specify:

Indicator Target Instrument

COMMON ISSUE: Indicators and targets are NOT tied to the specific instrument being used!

1010

What is an Indicator?What is an Indicator?

Information you collect to determineInformation you collect to determineprogress toward achieving resultsprogress toward achieving results

Concrete and measurableConcrete and measurable Answers the question, “What is myAnswers the question, “What is my

evidence that something has occurred?”that something has occurred?” Output indicator usually begins with Output indicator usually begins with

“Number of...”“Number of...” Outcome indicator usually begins with Outcome indicator usually begins with

“Percent of...”“Percent of...”

1111

What is a Target?What is a Target?

Describes Describes amount of change you of change youanticipate achieving each yearanticipate achieving each year

Based on indicator – specific, measurable Based on indicator – specific, measurable amount of change that will signify amount of change that will signify success

Outcome Target:Outcome Target:– Who will experience change? will experience change?– How many people will experience change? people will experience change?– How much change will occur (compared to change will occur (compared to

baseline)?baseline)?– Over Over what time period will change occur (one will change occur (one

year)?year)?

1212

Advice about TargetsAdvice about Targets

Consider available programConsider available programresources.resources.

Consider community context.Consider community context.

First year targets may be “educated guesses”.First year targets may be “educated guesses”.

Be prepared to adjust targets based on Be prepared to adjust targets based on experience.experience.

Targets should be ambitious but also realistic.Targets should be ambitious but also realistic.

Targets are based on indicators and the content Targets are based on indicators and the content of the instrument.of the instrument.

1313

ExerciseExercise

How Targets are Built How Targets are Built from Instruments:from Instruments:

Tutoring Data Tutoring Data Aggregation ExerciseAggregation Exercise

1414

Data Collection Issues

Can you access the data source (beneficiaries, organizations)?

Does an instrument (site or project) exist? Are they willing to give you data? (burden,

capacity) Can you get the data when you need it? Are your helpers (e.g. members) trained? Do you have a mechanism to collect

summarized data from sites?

1515

Methods

Logs, Tally Sheets, Attendance Rosters Survey, Questionnaire (Check list, Index, Scale) Interview (Interview Guide)

Service Recipients Interview Expert/Provider/Key Informant

Observation (Observation Guide) Case Study (Measurable change of a case over time) Focus Group (Focus Group Guide) Secondary Data (Existing records and information) Test / Experimental Designs

1616

ExerciseExercise

Program Instrument and Program Instrument and Data Collection Review Data Collection Review

and Discussionand Discussion

1717

Additional Additional Resources::

Contact Project STARContact Project STAR• http://nationalserviceresources.org//siteshttp://nationalserviceresources.org//sites

/star//star/

• 1-800-548-36561-800-548-3656• star@aiweb.comstar@aiweb.com