evaluation and measurement

Evaluation and Measurement

Spring 2010Quality Training Certificate Program

Quality Training Certificate ProgramCertificate Details:

Participants have the option to attend individual classes, or to participate in 6 classes and obtain a certificate in quality training.  Classes will be held twice yearly for a total of three years to complete the certificate.  Classes will continue to cycle, so new participants will be able to complete the certificate no matter when they begin.  Classes can also be taken at any time as stand-alone experiences.  There will be no charge for attending the classes, but supervisor permission must be given for attendance during regular work hours. 

Attendees from previously held classes will be given credit toward their certificate for attending. 

Classes will last 3 hours each, for a total of 18 total contact hours.

Attendance credit will continue to be given for individual classes taken. When attendees complete six classes, they will fill out an application for certification, and send it to BPA, who will verify the information and award the certificate.

Today’s Power Point and HandoutsWill be available online:http://www3.uwstout.edu/bpa/training.cfm

Overview Introductions Goals for Today Opportunity to suggest additions to today’s

agenda Course Content, including:

Practice Designing an Evaluation Discussion Questions Q&A

Breaks as needed

Introductions Tell us:

Who you are What worries you most about evaluation and

measurement?

Goals for Today To understand when evaluation can be used To understand the basic questions that guide

the evaluation process To identify appropriate evaluation

methodologies To implement basic evaluation methodologies To utilize evaluation results

Usage of Today’s Curriculum To apply the skills that you learn today in your

everyday practice Within the next month, to practice at least

one thing that you learned today

Questions/Topics you Want Covered Write down anything you want to be sure we

cover on the notecards in front of you I will collect these when we take our first

break

To understand when eval. can be used

What can be evaluated: 3 P’s:

Programs Processes Products

Evaluations can be “formal” or “informal” and time consuming or simple.

All faculty and staff typically engage in some sort of evaluation in their everyday work – although not all apply it in a formal way.

To understand when eval. can be used

Two major purposes: To answer questions about changes that need

to be made to programs/processes/products (formative),

To answer questions about continuation/discontinuation of programs/processes/products (summative)

Most of the evaluations we do at UW-Stout are formative.

To understand when eval. can be usedExamples: Most grants require an evaluation section The surveys that you fill out at the end of a

training session are a form of evaluation UW-Stout has a formal evaluation plan for our

e-Scholar program, first year experience program, customized instruction program, etc..

Evaluation can be used to determine if you are meeting customer needs

To understand when eval. can be usedDiscussion question: What other examples do you have of when

evaluation can or has been used? What was your experience with these

evaluations?

To understand the basic questions that guide the evaluation process

Two basic questions:• What is the purpose of the process/product/

program?• How will you use the results of the evaluation?

To understand the basic questions that guide the evaluation process

Considerations:• It is sometimes hard to identify the goals and usage

of the results, but you cannot develop a good evaluation without this information. Other ways to ask the question:

• What does success look like? • If this program/process/product did was wildly successful,

describe what that would look like.• Do not proceed with the evaluation if you cannot

articulate how you will use the results.

To understand the basic questions that guide the evaluation process

Discussion Question: How often have you been asked to participate

in a survey, focus group, or other data collection method when it was not clear how the results would be used?

To understand the basic questions that guide the evaluation process

Goal: How to know if goal is achieved:

Evaluation methods:

Target: Target date:

Goal 1

Goal 2

Goal 3

Goal 4

• All evaluation methods, data, and analysis should tie back to the goals and how the data will be used• A matrix can facilitate this:

Example evaluation plan

•

To understand the basic questions that guide the evaluation process

Practice: Think of an evaluation that you have done, or

that you would like to do, or that you anticipate you will be involved with.

If you don’t have one, use one of the examples on the flip chart.

Fill in: Evaluation question Goals of the process/product or program How the results of the evaluation will be used

To understand the basic questions that guide the evaluation process

Once you have identified your goals and how the results can be used, you need to think about how you will know if the goal is achieved.

At this point, do not think about methods.

Example: Goal: To identify appropriate evaluation

methodologies How achieved: Level of knowledge

participants have of evaluation methodologies

To understand the basic questions that guide the evaluation process

Discussion question: Who has a goal from their sheet that they are

comfortable sharing? How will we know if this goal is achieved?

To understand the basic questions that guide the evaluation process

Practice: Fill in the “how to know if goal is achieved”

column on your sheet

To identify appropriate evaluation methodologies

Considerations: Don’t let methods constrain initial discussion –

start with ideas. People like to challenge methodologies if they

don’t like the results. Too much data is a common problem. All data

must tie back to goals. It often takes more time to collect, manage and

analyze data than you think it will. Utilize multiple methods. Available time and resources will often be the

driving factors for selecting methods. Also your audience.

To identify appropriate evaluation methodologies

Major evaluation methods: Existing data Surveys Focus Groups/Interviews Inventories/Usage data/Database creation

To identify appropriate evaluation methodologies

Discussion questions: What do you think we see most often at

Stout? What do you think is the most underutilized

method at Stout?

To identify appropriate evaluation methodologies

Existing dataYou want to use existing data wherever

possible. Pros:

Minimal time commitment Readily available Ensures data collected is used for multiple

purposes Cons:

May be limited on what is available

To identify appropriate evaluation methodologies

Surveys Use surveys when:

The information you need to obtain is not available from existing data, and cannot easily be obtained through inventories or databases

You don’t need detailed qualitative data You have enough time to do it right

To identify appropriate evaluation methodologies

Surveys Pros:

Can get at factors not easily obtainable from other sources – for example, motivation, attitudes

Typically does not require a large time commitment from participants

Best for likert-scale, checklist short answer-type questions

Cons: Relies on self-report Not good if you want in-depth, qualitative comments Language, response rates, etc… often challenged

Purpose of a focus group:

“The purpose of a focus group is to listen and gather information. It is a way to better understand how people feel or think about an issue, product, or service. Participants are selected because they have certain characteristics in common that relate to the topic of the focus group”

Krueger, Richard, and Casey. Focus Groups. 3rd ed. Thousand Oaks, USA: Sage Publications, 2000.

What a focus group is A carefully planned series of discussions designed

to obtain perceptions on a defined area of interest Discussions in a permissive, nonthreatening,

environment Groups of 5-12 facilitated by a skilled facilitator Repeated groups – need to hold at least 3-4

sessions

When to use focus groups:

Looking for a range of ideas or perspectives on an issue

Trying to determine differences in perspectives between groups of people “A group possesses the capacity to become

more than the sum of its parts” (Krueger and Casey, 2000)

Researcher needs information to aid existing quantitative data

When NOT to use focus groups:

Want people to come to a consensus Want to educate people Don’t intend to use results Other methodologies can produce better

results Other methodologies can produce same

results more efficiently Don’t have enough time to follow standard

focus group procedures

To identify appropriate evaluation methodologies

Focus Group/Interviews Pros

Allows you to go in-depth in an issue Allows for a range of ideas – and you hear from

people in their own voice Cons

Most time-consuming approach

To identify appropriate evaluation methodologies

Inventories/Usage data/Observations Pros:

Typically more accurate than self-report data Good for tracking participation in events,

trainings, programs, etc.. Cons:

Only works for things you can count or observe Often have to rely on other people to obtain the

data When multiple people are involved, need to make

sure that everything is counted in the same way

To identify appropriate evaluation methodologiesPractice: Circle the evaluation methods that you feel

can be utilized to obtain the information in the second column of your sheet. Circle multiple methods if you can use more than one method.

Tying it all together You have two more columns on your sheet – a

target and target date. It is important that you identify specific

targets and dates before you collect any data – as after you collect the data, people will use it to prove their point.

Often you cannot identify a target until you have your evaluation methods more fully developed – so we will not be filling in these columns today.

Tying it all together What you have on your sheets now is typically

your ideal scenario for how you can go about evaluating your process/product/program.

Usually, you don’t have time to do it all, so next you need to determine what is feasible given your time, resources, and who will be reviewing the results.

Tying it all togetherStart by sharing your plan with your key

stakeholders to get their input. Explain why you have chosen the methods that

you have. Convince them that you have good reason for choosing the methods that you did.

What concerns do they have about the plan? Given your available time and resources, you need

their help to prioritize what is most important to them.

It is often helpful to obtain approval of the plan in writing.

Tying it all togetherPractice: Share your plan with a neighbor. Pretend

they are one of your key stakeholders. Get their input.

Tying it all togetherDiscussion question: Is anyone willing to share their plan with the

group?

To implement basic evaluation methodologies

Now that you have approval on your plan, you need to implement it.

Next, you need to develop your instruments and implement them.

To implement basic evaluation methodologies

Considerations: Before collecting your data, develop a data

analysis plan. Just as it is important to get approval on your

evaluation plan before starting, it is also important to obtain approval on the instrument and data analysis plan.

Pilot testing is important – administer the survey/focus group/form to people similar to those who will receive it.

To implement basic evaluation methodologies

Existing Data Can sometimes find a survey, focus group or

other study that has recently been administered that covers the same topics that you are assessing

Several ways to access existing data at Stout: Contact the BPA office Survey Clearinghouse (in development) Information Portal (in development) Qualitative results:

http://www.uwstout.edu/static/bpa/ir/surveylistqual.html

To implement basic evaluation methodologies

Surveys Qualtrics is the online survey instrument that

Stout uses: http://www3.uwstout.edu/bpa/survey/

Qualtrics question library in development. Will be available from the above link. Will provide suggested demographic questions Will also provide suggestions for other standard

questions Typically don’t want it to take more than 10

minutes to complete – time to complete is more important than the number of questions.

To implement basic evaluation methodologies

Surveys, cont. Look for other surveys you can model after –

internally developed and externally developed. Request permission from the author to use some of the same questions. Survey Clearinghouse (in development) Surveys webpage:

http://www.uwstout.edu/static/bpa/ir/surveylist.html

Sample survey guide: http://www.uwstout.edu/static/bpa/ir/afu/information/instruction.pdf

To implement basic evaluation methodologies

Focus Groups/Interviews Different than a “listening session”

Discussion question:How are focus groups different than listening

sessions?

To implement basic evaluation methodologies

Core Ingredients Continue groups until you hear same themes

repeated – the ARC typically holds 8 sessions. Sometimes it is possible to do less.

No more than 12 people in a group Circle seating Typically requires formal training for

facilitators/ moderators Pre-determined questions Systematic analysis

To implement basic evaluation methodologies

Inventories/Usage data/Observations Must provide training to the people who will

be entering data into databases or inventories. Include specific instructions.

Examples: Sign-in sheets outside of a tutor center (sample in

separate document) Appointment logs Formal observation forms, with checklists, open-

ended questions, tally marks, etc… (sample in separate document)

To utilize evaluation results

Evaluation results are primarily used for two purposes:

Program/process/product improvement (formative)

Decisions about program/process/product continuation/discontinuation (summative)

Both are important

To utilize evaluation results

Formative evaluation: Typically done early in on program. Usually

identifies improvements that can be made related to program implementation.

To utilize evaluation results

Formative evaluation, examples: Decisions about improvements that can be

made to get more people to attend a pre-college program.

Decisions about improvements that can be made to get more people to apply what they’ve learned from training programs.

Decisions about new ways to present or deliver departmental newsletters.

To utilize evaluation results

Summative evaluation: Typically done later on in the program – after

all of your target dates have passed. Usually used to make decisions about continuation/discontinuation.

Typically summative decisions are the most difficult to make. If you establish specific targets and target due dates, and have done formative evaluation prior to the summative evaluation, these decisions are easier.

To utilize evaluation results

Summative Evaluation, examples: Decisions about if funding for a grant will

continue Decisions about if various initiatives to

improve retention will continue Decisions about if on campus funding will

become permanent Decisions to keep or eliminate items from the

campus-wide training and development plan

To utilize evaluation results We often hear from people that nothing is

done with the feedback they provide on surveys, focus groups, etc…

It is important to invest time into sharing how you have used the results from the evaluation. People will not often make the connection between feedback and their actions on their own.

Questions/Comments?