program evaluation and assessment · academic lines of reporting (academic or student affairs)...

Program Evaluation andProgram Evaluation and

AssessmentAssessment

David Parker, University of ConnecticutDavid Parker, University of Connecticut

Elaine Elaine ManglitzManglitz, Clayton State University, Clayton State University

Sam Goodin, University of MichiganSam Goodin, University of Michigan

Steve Robillard, SR-PS, Inc.Steve Robillard, SR-PS, Inc.

Richard Riccardi, Southern Connecticut State UniversityRichard Riccardi, Southern Connecticut State University

AHEAD 2007, Charlotte, North Carolina, USA July 17-21, 2007

TodayToday’’s Agendas Agenda

2:00 Introductions2:00 Introductions

2:10 Overview; Reasons for Conducting a Program2:10 Overview; Reasons for Conducting a Program

Evaluation (David) Evaluation (David)

2:35 What to Evaluate: CAS and AHEAD Program2:35 What to Evaluate: CAS and AHEAD Program Standards (Elaine) Standards (Elaine)

2:55 Using Survey Tools to Gather Data (Sam)2:55 Using Survey Tools to Gather Data (Sam)

3:15 Break3:15 Break

3:35 Use of Non-Survey Tools to Collect Data (Steve)3:35 Use of Non-Survey Tools to Collect Data (Steve)

3:55 Tools for Storing, Analyzing, and Presenting Data3:55 Tools for Storing, Analyzing, and Presenting Data

(Richard) (Richard)

4:15 Interactive Segment (Suggestions for Utilizing4:15 Interactive Segment (Suggestions for Utilizing

Evaluation Results; Final Q&A) Evaluation Results; Final Q&A)


Reasons for Conducting a ProgramReasons for Conducting a Program

EvaluationEvaluation

David ParkerDavid Parker

University of ConnecticutUniversity of Connecticut


Accompanying MaterialsAccompanying Materials

Materials accompanying this symposium session are available

online at AHEAD’s learning management system:

http://ahead.coursewebs.com

Enter username: assessment

Password: ahead2007

Click on AHEADSYM6 Login

This will bring you to a page with instructions for accessing your

copy of Program Evaluation of Postsecondary Disability

Services: From Theory to Practice and accompanying

materials.


Reasons for ConductingReasons for Conducting

a Program Evaluationa Program Evaluation

Ensure Accountability

“Given the fiscal, administrative, and

legal accountability of OSDs, it is vital

that the personnel responsible for

ensuring equal educational access

conduct a periodic self-assessment of

their performance.”

- Dukes & Shaw(1999)




Inform Decision Making

Conduct an LD program evaluation to:

Justify program needs (staff, space, equipment) and

academic lines of reporting (academic or student

affairs)

Monitor student achievement

Document compliance with Section 504/ADA

Analyze effectiveness of program services

Assist in developing institutional policy (e.g., course

substitution policy)

- Brinckerhoff, McGuire, & Shaw (2002)




3) Promote Self-determination

When sensitive to context, program evaluations provide datathat help DSS professionals establish, implement, andevaluate a programmatic mission:

Establish descriptive data/baselines

Clarify problems

Document successes

Meet peer-driven standards of “best practice”

- Goodin, Parker, Shaw, & McGuire (2004)




“What are our peer institutions doing?”

Program Evaluation of Postsecondary Disability Services:From Theory to Practice

2) DSS Program Evaluations (Questionnaire Results)

Use of data

Planning to evaluate

Anticipated benefits


Using Survey Tools to Gather DataUsing Survey Tools to Gather Data

Sam GoodinSam GoodinUniversity of MichiganUniversity of Michigan


The The ““RealReal”” Reasons to do Reasons to do

Program EvaluationProgram Evaluation

DSS staff do not have the time to reconfigure supportDSS staff do not have the time to reconfigure supportservice mechanisms on the whims of concernedservice mechanisms on the whims of concernedstudents, staff, supervisors or officials from the Office ofstudents, staff, supervisors or officials from the Office ofCivil Rights.Civil Rights.

We do not have the blood pressure and cholesterolWe do not have the blood pressure and cholesterollevels that will allow us to spend our entire professionallevels that will allow us to spend our entire professionallives putting out fires.lives putting out fires.

We do not have spouses, children and significantWe do not have spouses, children and significantothers who will forever tolerate our spending weekendsothers who will forever tolerate our spending weekendsand evenings creating pilot projects for which we findand evenings creating pilot projects for which we findno demand.no demand.


Modular Mailed SurveyModular Mailed SurveyModule I General questions for all students

Modules II A-D Disability specific questions

Module III Demographic data

Please read each item and rate the availability and quality of each

service.

1 = poor 2 = fair 3 = adequate 4 = good 5 = excellent

___ Reader services (availability)

___ Reader services (quality)

___ Loan of Special equipment (e.g., recorder) (availability)

___ Loan of Special equipment (e.g., recorder) (quality)

___ Test/quiz administration (availability)

___ Test/quiz administration (quality)


Phone SurveyPhone Survey

Its much easier to terminate a telephone interview abruptly,

saying something like, “Whoops! someone's at the door. I gotta

go.” or “OMIGOD! The pigs are eating my Volvo!”

- Babbie, 1992


Evaluating Other ServicesEvaluating Other Services

When people ask “how big is your staff?” I am fond of telling

them I have a staff of 2,500. That usually gets a look (and a

stir) out of the questioner, but in fact, on our campus, that is the

approximate number of our faculty and staff.

- Harris, 1984


Instruments for EvaluatingInstruments for Evaluating

Other ServicesOther Services

Other Campus Offices:

Admissions

Counseling

Parking Office

Health Service

Housing

Library

Career Center


Evaluation of Service ProvidersEvaluation of Service Providers

…formal performance evaluation probably began in United

States in 1813 (Bellows & Estep, 1954) when army general

Lewis Cass submitted to the War Department an evaluation of

each of his men using such terms as “a good natured man” or

“knave despised by all.”

- Murphy & Clevland, 1995


Instruments for EvaluatingInstruments for Evaluating

Service ProvidersService Providers

Readers

Interpreters

Notetakers

Professional and Administrative Staff


StudentVoiceStudentVoice.com

Each StudentVoice toolkit provides each studentaffairs unit with specific assessment programsfrom which they can choose. These programsinclude common instruments for comparison withpeer institutions and the opportunity to addcampus-specific questions to address preciseissues or concerns that help Directors monitor theusage, satisfaction and outcomes.


Need a Title

How do you rate the ease of finding out about

Paratransit/Yellow Cab service?

How do you rate the ease of registering with SSD to be eligible

for these services?

Have you used the Yellow Cabs?

Please rate the following regarding your experience with the

Yellow Cabs: - The cab consistently came within the promised

time.



Elaine Elaine ManglitzManglitzClayton State UniversityClayton State University


What to Evaluate: CAS andWhat to Evaluate: CAS and

AHEAD Program StandardsAHEAD Program Standards


–– http://www.ahead.org/resources.phphttp://www.ahead.org/resources.php

CAS Professional Standards for HigherCAS Professional Standards for Higher

Education, 6Education, 6thth edition edition

–– http://www.cas.edu/http://www.cas.edu/


Revised professional standards andRevised professional standards and

performance indicators for our fieldperformance indicators for our field

Developed with input from experts withinDeveloped with input from experts within

the fieldthe field

Can direct program evaluation andCan direct program evaluation and

development effortsdevelopment efforts

Supports staff development effortsSupports staff development efforts

CAS Professional StandardsCAS Professional Standards

Consistent format across all functionalConsistent format across all functional

areas, including disability support servicesareas, including disability support services

Tied to student learning and developmentTied to student learning and development

Self-assessment guidesSelf-assessment guides

Use to gauge program effectivenessUse to gauge program effectiveness

Use to create an action plan forUse to create an action plan for

improvementimprovement

Supports professional staff developmentSupports professional staff development

Recommendations for Using theRecommendations for Using the

Assessment ToolsAssessment Tools

Using both AHEAD program standardsUsing both AHEAD program standards

and CAS professional standardsand CAS professional standards

Tie to institutionTie to institution’’s vision and missions vision and mission

Implementation of evaluation efforts: noImplementation of evaluation efforts: no

one one rightright way way

Making the tools work for your office andMaking the tools work for your office and

your needsyour needs

Additional RecommendationsAdditional Recommendations

Assess/evaluate how well the disabilitiesAssess/evaluate how well the disabilities

service office meets the mission of theservice office meets the mission of the

UniversityUniversity

Assess/evaluate how well the disabilitiesAssess/evaluate how well the disabilities

service office meets the vision and missionservice office meets the vision and mission

of the general Division of which it is a partof the general Division of which it is a part

(Academic or Student Affairs)(Academic or Student Affairs)


Steve RobillardSteve Robillard

SR-PS, Inc.SR-PS, Inc.


Why Use Non-Survey Data?Why Use Non-Survey Data?

If the only tool you have is a hammer, everyIf the only tool you have is a hammer, every

problem is a nail.problem is a nail.

TrendsTrends

Overall shift from qualitative toOverall shift from qualitative to

quantitative, or better yet, both.quantitative, or better yet, both.

Graduation, retention, student learnerGraduation, retention, student learner

outcomesoutcomes

What is available?What is available?

Looking in your own backyardLooking in your own backyard

The UniversityThe University

StateState

FederalFederal

Peer institutionsPeer institutions

ExamplesExamples

Staff activityStaff activity

CSP 100CSP 100

Designated MathDesignated Math

Extended TimeExtended Time


Richard RiccardiRichard Riccardi

Southern Connecticut State UniversitySouthern Connecticut State University


Think Before You DigThink Before You Dig……

Identify the problem to be addressedIdentify the problem to be addressed

–– ExistenceExistenceHave you used the Yellow Cabs?

–– Opinion based (Satisfaction/Agreement)Opinion based (Satisfaction/Agreement)

How do you rate the easeHow do you rate the ease……

–– Time basedTime based

How long have you used the service?How long have you used the service?

–– CombinationCombination

Is the service Is the service betterbetter than before? than before?

What is What is ““better?better?””

The Old Wheel Is RoundThe Old Wheel Is Round……

The New Wheel Is RoundThe New Wheel Is Round

Your own system (electronic, paper)Your own system (electronic, paper)

Student Information SystemStudent Information System

–– Extracts (IT, IR, Assessment, self)Extracts (IT, IR, Assessment, self)

School profiles (own, peer schools)School profiles (own, peer schools)

State/Federal data (IPEDS, US News)State/Federal data (IPEDS, US News)

Noel Levitz, NSSENoel Levitz, NSSE

Collecting DataCollecting Data

Survey ToolsSurvey Tools

–– SurveyMonkeySurveyMonkey

–– Snap SurveySnap Survey

–– ZoomerangZoomerang

–– ExploranceExplorance

–– StudentVoiceStudentVoice

–– Paper and #2 pencilsPaper and #2 pencils

–– TelephoneTelephone

Issues with Survey ToolsIssues with Survey Tools

Ease of useEase of use

AccessibilityAccessibility

Web-basedWeb-based

AudienceAudience

SecuritySecurity

Data ownershipData ownership

ITIT

Survey design (reliability, validity)Survey design (reliability, validity)

Analyzing DataAnalyzing Data

Microsoft ExcelMicrosoft Excel

–– Universal (PC, Macintosh)Universal (PC, Macintosh)

–– Easy to useEasy to use

–– Some statistical power (Pivot Tables, t-Test,Some statistical power (Pivot Tables, t-Test,

z-Test, Chi-Square, Pearson correlation)z-Test, Chi-Square, Pearson correlation)

Power ToolsPower Tools

–– Microsoft Access, Microsoft Access, FilemakerFilemaker, Oracle, , Oracle, MySQLMySQL

–– SPSS, SASSPSS, SAS

Excel Excel –– Pivot Tables Pivot Tables

Outcome Data

Fail No Grade Pass Withdraw Total Students

Term Subject Course Section Students % Students % Students % Students %

Fall 1993 CSP 100 02 2 9.1% 0.0% 19 86.4% 1 4.5% 22

Fall 1994 CSP 100 01 2 9.5% 0.0% 18 85.7% 1 4.8% 21

Fall 1995 CSP 100 01 1 4.5% 1 4.6% 20 90.9% 0.0% 22

Fall 1996 CSP 100 01 3 13.0% 0.0% 20 87.0% 0.0% 23

Fall 1997 CSP 100 01 3 12.5% 0.0% 20 83.3% 1 4.2% 24

Fall 1998 CSP 100 01 4 19.0% 0.0% 17 81.0% 0.0% 21

Fall 1999 CSP 100 01 2 13.3% 1 6.7% 12 80.0% 0.0% 15

Fall 2000 CSP 100 01 2 7.7% 0.0% 23 88.5% 1 3.8% 26

Fall 2001 CSP 100 02 0.0% 0.0% 22 95.7% 1 4.3% 23

Fall 2002 CSP 100 01 0.0% 0.0% 17 94.4% 1 5.6% 18

Fall 2003 CSP 100 01 1 4.3% 0.0% 21 91.3% 1 4.3% 23

Fall 2004 CSP 100 01 1 5.0% 0.0% 19 95.0% 0.0% 20

Grand Total 21 8.1% 2 0.8% 228 88.4% 7 2.7% 258

Legend (groupings by grade): Fail = D- or F, Incomplete = I or I+, No Grade = N or R, Pass = D or better, Withdraw = W, WP, or WF

Outcomes for CSP 100 - Fall 1993 to Fall 2004

A Picture Can Be Worth an FTEA Picture Can Be Worth an FTE……

Student/FT Staff Ratio

0

50

100

150

200

250

2003 2004 2005 2006 2007

Year

Stu

den

ts

Year FT Staff Students Served Student Ratio

2003 6 450 75

2004 5 575 115

2005 5 600 120

2006 4 700 175

2007 4 800 200

Presenting InformationPresenting Information

Know Your AudienceKnow Your Audience

–– ““Just give me the bullets.Just give me the bullets.””

–– How does your data fit with the mission and vision ofHow does your data fit with the mission and vision ofuniversity?university?

Know Your PurposeKnow Your Purpose

–– Set baselineSet baseline

–– Improve serviceImprove service

–– Increase fundingIncrease funding

KISS MethodKISS Method

–– If it needs explaining, it needs to be simplified.If it needs explaining, it needs to be simplified.

The Assessment CycleThe Assessment Cycle

Problem + Data + Standards =Problem + Data + Standards =

InformationInformation

Information + Context =Information + Context =

KnowledgeKnowledge

Appling the Knowledge Gained =Appling the Knowledge Gained =

AssessmentAssessment

program evaluation and assessment · academic lines of reporting (academic or student affairs)...

Documents