Implementing Implementing Good Practices in Outcomes-Good Practices in Outcomes-

Based Program ReviewBased Program Review

International Assessment & Retention ConferenceInternational Assessment & Retention ConferenceJune 8, 2007 - St Louis, MOJune 8, 2007 - St Louis, MO

Marilee Bresciani, San Diego State UniversityMarilee Bresciani, San Diego State UniversityCyd Jenefsky, John F. Kennedy UniversityCyd Jenefsky, John F. Kennedy University

marilee.bresciani@mail.sdsu.edumarilee.bresciani@mail.sdsu.edu & & jenefsky@jfku.edujenefsky@jfku.edu

Outcomes of SessionOutcomes of Session

Evaluate good-practice principles in your Evaluate good-practice principles in your institution’s program review processesinstitution’s program review processes

Analyze good-practice principles applied Analyze good-practice principles applied to a case study to a case study

Identify next steps for applying/ improving Identify next steps for applying/ improving good-practices in your own institutiongood-practices in your own institution

Inverted FunnelInverted Funnel

Criteria for Good Practices for Criteria for Good Practices for Outcomes-Based Assessment Program Outcomes-Based Assessment Program

ReviewReview

Criterion 1: Criterion 1: Clear understanding of Clear understanding of goals & expectations for program goals & expectations for program reviewreview

Criterion 2: CollaborationCriterion 2: Collaboration

Criterion 3: Use of resultsCriterion 3: Use of results

Criterion 4: Awards and recognitionCriterion 4: Awards and recognition

Criterion 5:Criterion 5: Resources to support Resources to support program review program review

Criterion 6: Coordination of the Criterion 6: Coordination of the process process

Criterion 7:Criterion 7: Flexibility Flexibility

Criterion 8Criterion 8: Addressing barriers : Addressing barriers

Criterion 9:Criterion 9: Evaluation of the Evaluation of the program review processprogram review process

What good practice criteria do What good practice criteria do you already have in place at you already have in place at

your institution?your institution?

Use the sample criteria the sample criteria checklist to apply to your checklist to apply to your

institution.institution.

Inverted FunnelInverted FunnelReminderReminder

Case Study: JFK UniversityCase Study: JFK University

Illustration of implementing criteria Illustration of implementing criteria 1, 2, 5, 6, 7 & 81, 2, 5, 6, 7 & 8

JFKU: Process DesignJFKU: Process Design

Groundwork:Groundwork:

Univ-wide key stakeholder involvementUniv-wide key stakeholder involvementHigh-credibility, active facultyHigh-credibility, active facultyExplicit leadership commitmentExplicit leadership commitmentAccreditation review as leverAccreditation review as leverPoint person in charge of developmentPoint person in charge of development

Process DesignProcess Design

Research-basedResearch-based– Good-practice researchGood-practice research– Models from good-practice institutionsModels from good-practice institutions

IterativeIterativeStart with ‘what matters’ to stakeholdersStart with ‘what matters’ to stakeholdersTrain-the-trainer modelTrain-the-trainer modelClear committee goals and timelineClear committee goals and timeline

ImplementationImplementation

Actively cultivate buy-inActively cultivate buy-inStart with “champions”Start with “champions”Professional development at each stepProfessional development at each stepAddress fears, anxieties, concernsAddress fears, anxieties, concerns– inc. one-on-one conversationsinc. one-on-one conversations

Stakeholder representativesStakeholder representatives

Guiding QuestionsGuiding Questions

What do we want our students to learn by the time they complete our program?– Intended student learning outcomes

What are they actually learning?– Actual student learning results

Guiding Questions Guiding Questions (cont’d)(cont’d)

How well are they learning these outcomes?– Levels of achievement (based on explicit

criteria)

How do we know?– Evidence of student learning

Guiding Questions Guiding Questions (cont’d)(cont’d)

Closing the loop:

How are we using the evidence to guide decisions for improvement?

Do the improvements we make work?

Capacity-BuildingCapacity-Building

Process Process builds capacitybuilds capacity for assessment for assessment– does not presume capacitydoes not presume capacity– workforce retrainingworkforce retraining

Built into conceptual frameworkBuilt into conceptual frameworkTrain-the-trainerTrain-the-trainer– expertexpert– peer-to-peer trainingpeer-to-peer training

““Zone of proximal development”Zone of proximal development”

FlexibilityFlexibility

Constantly self-assessing & adaptingConstantly self-assessing & adaptingProcess serves “end-users”Process serves “end-users”Disparate cultural groupsDisparate cultural groupsWork with multiple accreditation standardsWork with multiple accreditation standardsLearning as we goLearning as we go

Apply JFK’s ProcessApply JFK’s Process

What criteria can you easily What criteria can you easily identify in their program manual identify in their program manual

that you could adapt and apply to that you could adapt and apply to your institution? your institution?

How do we move our How do we move our institutions toward institutions toward

implementation of good implementation of good practice criteria?practice criteria?

Suggestions for ImplementationSuggestions for Implementation

Create a well-represented, Create a well-represented, well-respectedwell-respected OBAPR committeeOBAPR committeeOrganize the committee’s role and Organize the committee’s role and responsibilitiesresponsibilitiesArticulate expectationsArticulate expectations for outcomes-based for outcomes-based assessment program reviewassessment program reviewPlan short-range and long-range Plan short-range and long-range goalsgoals

Suggestions Suggestions (cont’d)(cont’d)

Identify existing resources and processes Identify existing resources and processes and identify new resourcesand identify new resourcesEstablish a communication planEstablish a communication planDiscussDiscuss implementation barriers and implementation barriers and strategies to overcome themstrategies to overcome themMove forward with Move forward with flexibilityflexibility

What are your organization’s What are your organization’s next steps for implementing next steps for implementing good practices in OBAPR?good practices in OBAPR?

What other questions do you What other questions do you need to ask yourselves?need to ask yourselves?

The process is not a means The process is not a means unto its own end;unto its own end;

it is a way to systematically engage it is a way to systematically engage

in daily critical inquiry in daily critical inquiry about discovering what works well and about discovering what works well and

what needs to be improved. what needs to be improved.

(Maki, 2004).(Maki, 2004).

ReferencesReferences

Bresciani, M.J. (2006). Bresciani, M.J. (2006). Outcomes-Based Outcomes-Based Academic and Co-Curricular Program Academic and Co-Curricular Program Review: A Compilation of Institutional Review: A Compilation of Institutional Good Practices.Good Practices. Sterling, VA: Stylus Sterling, VA: Stylus PublishingPublishing

Criterion Sample Selection Criterion Sample Selection ReferencesReferences

American Association of Higher Education (1994). American Association of Higher Education (1994). Nine Nine principles of good practice for assessing student learningprinciples of good practice for assessing student learning . . Retrieved March 17, 2006, from Retrieved March 17, 2006, from http://www.aahe.org/assessment/principl.htmlhttp://www.aahe.org/assessment/principl.html..Eckel, P., Green, M., and Hill, B. (2001). Eckel, P., Green, M., and Hill, B. (2001). On change V—On change V—Riding the waves of change: Insights from transforming Riding the waves of change: Insights from transforming institutionsinstitutions.. Washington DC: American Council of Washington DC: American Council of Education.Education.Lopez, C. (1997). Lopez, C. (1997). Opportunities for improvement: Advice Opportunities for improvement: Advice from consultant-evaluators on assessing student learning.from consultant-evaluators on assessing student learning. Evidence of strong institutional support for assessing Evidence of strong institutional support for assessing student learning. Retrieved March 16, 2006, from student learning. Retrieved March 16, 2006, from http://www.ncahigherlearningcommission.org/resources/ashttp://www.ncahigherlearningcommission.org/resources/assessment/index.html.sessment/index.html.

One Minute EvaluationOne Minute Evaluation

What is the most valuable What is the most valuable lesson that you learned from lesson that you learned from this workshop?this workshop?

What is one question that you What is one question that you still have?still have?

Contact info:Contact info:

marilee.bresciani@mail.sdsu.edumarilee.bresciani@mail.sdsu.edu

jenefsky@jfku.edujenefsky@jfku.edu