Institutional Effectiveness at the Institutional Effectiveness at the University of North AlabamaUniversity of North Alabama

Dr. Andrew L. Luna

Institutional Research, Planning, and Assessment

IE Presentation, November 20, 2008

Connections to IE?Connections to IE?

Hawthorne Works, Bell Laboratories

Telephones and IE?Telephones and IE?

Walter Shewhart

Edwards Deming

Joseph Juran

Shewhart CycleShewhart Cycle

FACT...

The Shewhart Cycle is the foundation for all quality and

Continuous improvement processes that we use today

Plan

Do

Check

ActContinuous

Improvement

Points of DiscussionPoints of Discussion

Similarities between the Shewhart Cycle and Institutional Effectiveness

Overview of Institutional Effectiveness at UNA

Review of Outcomes and Improvement Processes

Review of AssessmentQuestions

More on the Shewhart CycleMore on the Shewhart Cycle

Plan – Create a strategy as to what you want to do and how you will measure success

Do – Follow the plan and do what you say you will do

Check – Assess the effectiveness of the current plan by looking at the success outcomes measures

Act – Make changes to the strategies to improve the measured outcomes

Repeat the Cycle!

Why is the Shewhart Cycle Important?Why is the Shewhart Cycle Important?

If you can’t measure something, you can’t understand it…

If you can’t understand it, you can’t control it… If you can’t control it, you can’t improve it… If you can’t improve it…then why the heck are

you doing it?

So, What is Institutional Effectiveness?So, What is Institutional Effectiveness?

A sharpened statement of institutional mission and objectives

Identification of intended departmental/programmatic outcomes or results (Plan)

Establishment of effective means of assessing the accomplishments outcomes and results (Do, Act, Check)

FACT...

Institutional Effectiveness is primarily undertaken to improve what we do…not just to pass accreditation.

Shewhart Cycle and SACSShewhart Cycle and SACSMacro IEMacro IE

Core Requirement 2.5:

“The institution engages in ongoing, integrated, and institution-wide research-based planning and evaluation processes that (1) incorporate a systematic review of institutional mission, goals, and outcomes; (2) result in continuing improvement in institutional quality; (3) demonstrate the institution is effectively accomplishing its mission.”

PlanCheck and

Act

Do

Key Points to Core Requirement 2.5Key Points to Core Requirement 2.5

Emphasizes an expectation that the institution is the primary focal point for compliance

Sets expectations for the description of planning and evaluation processes that are active and continuous rather than static or single occurrences.

Points to a clear and strong expectation for documentation of the systematic review of institutional mission, goals and accomplishments consistent with its mission

Sets expectations for the documented use of results of institutional planning and evaluation to achieve institutional improvements

Shewhart and SACS, Cont.Shewhart and SACS, Cont.Micro IEMicro IE

Comprehensive Standard 3.3.1:

“The institution identifies expected outcomes for its education programs … and its administrative and educational support services; assesses whether it achieves those outcomes; and provides evidence of improvement based on analysis of those results.”

Plan

CheckDo and Act

Key Points to Comprehensive Standard Key Points to Comprehensive Standard 3.3.13.3.1 Emphasizes the unit level of individual educational

programs and support services The expected achievements of educational programs and

support services should be articulated, and evidence presented concerning accomplishments

Distinguishes between program outcomes and learning outcomes

Sets expectations that improvement is guided by the establishment and evaluation of program and learning outcomes

Shewhart and SACS, Cont.Shewhart and SACS, Cont.General Education and IEGeneral Education and IE

Comprehensive Standards 3.5.1

“The institution identifies college-level competencies within the general education core and provides evidence that graduates have attained those competencies.”

Plan

Do, Check, Act

Key Points to Comprehensive Standard Key Points to Comprehensive Standard 3.5.13.5.1 General Education should be part of the institutional

mission The expected achievements of the General Education

program should be articulated, and evidence presented concerning accomplishments

Improvement should be guided by the establishment and evaluation of learning outcomes

Overview of Institutional EffectivenessOverview of Institutional Effectiveness

Focus on Assessment

Comprehensive Dept./Program Review

Program Outcomes•Quality Indicators•Productivity Indicators•Viability Indicators

Continuous Improvement of Programs and Departments

Institutional Effectiveness

MissionStrategic Goals

Evaluation of LearningLearning Outcomes

•What graduates know•What graduates can do•What attitudes/values graduates possess

Continuous Improvement of Student Learning

Institutional Effectiveness System at Institutional Effectiveness System at UNAUNA

Annual Report - Annual Action Plan and Assessment Report

Comprehensive Program and Department review – Five-year Review

Review of General Education – Five-year cycle of General Education assessment

Schematic of Institutional Effectiveness Schematic of Institutional Effectiveness ProcessProcess

Year 1Annual Reports

Year 2Annual Reports

Year 3Annual Reports

Year 4Annual Reports

Year 5Annual Reports

Review of Strategic Goals

Five-Year Review for Selected Depts.

Five-Year Review for Selected Depts.

Five-Year Review for Selected Depts.

Five-Year Review for Selected Depts.

Five-Year Review for Selected Depts.

Area IFive-Year Assessment

Area IIFive-Year Assessment

Area IIIFive-Year Assessment

Area IVFive-Year Assessment

OverallGen. Ed. Assessment

•OIRPA•IE Committee•Gen. Ed. Committee

5-Year Cycle?

No

Yes

Five-Year Program/Department Review Five-Year Program/Department Review Timeline Timeline (pending IE Committee Approval)(pending IE Committee Approval)

September October November December January February March April May June July August September

Last year’s Depts. that underwent 5-year review submits outcomes of review as AAPAR priority initiatives

OIRPA submits Five-Year Enrollment report to academic departments

OIRPA conducts assessment workshop for UNA campus

OIRPA meets with Deans/VP for overview

Deans/VPs meet with departments to discuss review

OIRPA meets with departments up for review

OIRPA initiates individual departments meetings

Five-Year Reviews completed and sent to Dean/VP

OIRPA submits overview of Five-Year process to IE Committee

Annual Action Plan and Assessment Annual Action Plan and Assessment Report Timeline Report Timeline (pending IE Committee Approval)(pending IE Committee Approval)

September October November December January February March April May June July August September

President, VP, and Dean Initiatives Due

1st part of AAPAR due for current fiscal year w/ one Priority Initiative for next FY

Next FY Priority Initiatives by Deans

Next FY Priority Initiatives by VPs

SPBS reviews Next FY Priority Initiatives

2nd Part of AAPAR completed by depts.

OIRPA submits AAPAR overview to IE Committee

Budget initiatives based on Priority Initiatives are established

OutcomesOutcomes

Operational Outcomes - measures of how well the institution/division/department is meeting/exceeding requirements

Learning Outcomes - statements of the knowledge, skills, and abilities the individual student possesses and can demonstrate upon completion of a learning experience or sequence of learning experiences (e.g., course, program, degree).

Problems with OutcomesProblems with Outcomes

Outcomes are too broad Outcomes do not address core

requirements/competencies or mission Outcomes are not measurable

Types of MeasurementTypes of Measurement

Discrete or Attributes data Binary data with only two

values

Continuous or Variable data

Information that can be measured on a continuum or scale

Yes/No Good Bad On/Off Male/Female Pass/Fail Height/Weight Temperature Test Scores Time Distance

Bloom’s Taxonomy of Learning Bloom’s Taxonomy of Learning OutcomesOutcomes

Category Definition Related Behaviors

Knowledge

recalling or remembering something without necessarily understanding, using, or changing it

define, describe, identify, label, list, match, memorize, point to, recall, select, state

Comprehension

understanding something that has been communicated without necessarily relating it to anything else

alter, account for, annotate, calculate, change, convert, group, explain, generalize, give examples, infer, interpret, paraphrase, predict, review, summarize, translate

Application

using a general concept to solve problems in a particular situation; using learned material in new and concrete situations

apply, adopt, collect, construct, demonstrate, discover, illustrate, interview, make use of, manipulate, relate, show, solve, use

Bloom’s Taxonomy, Cont.Bloom’s Taxonomy, Cont.Category Definition Related Behaviors

Analysis

breaking something down into its parts; may focus on identification of parts or analysis of relationships between parts, or recognition of organizational principles

analyze, compare, contrast, diagram, differentiate, dissect, distinguish, identify, illustrate, infer, outline, point out, select, separate, sort, subdivide

Synthesis

relating something new by putting parts of different ideas together to make a whole

blend, build, change, combine, compile, compose, conceive, create, design, formulate, generate, hypothesize, plan, predict, produce, reorder, revise, tell, write

Evaluation

judging the value of material or methods as they might be applied in a particular situation; judging with the use of definite criteria

accept, appraise, assess, arbitrate, award, choose, conclude, criticize, grade, judge, prioritize, recommend, referee, select, support

 

Forms of MeasurementForms of Measurement

Longitudinal data is gathered over an extended period

Semester1 , ,Semester2 …Semester3 Semestert

Forms of Measurement, Cont.Forms of Measurement, Cont.

Cross-sectional data represent a snapshot of one point in time

What is Improvement?What is Improvement?

Measurable actions that increase learning, efficiency, effectiveness, and/or the bottom line

Decrease the BadIncrease the GoodDecrease Variability

Decrease Variability?Decrease Variability? What the heck is that? What the heck is that?

Class A

100, 100 99, 98 88, 77 72, 68 67, 52 43, 42

Mean = 75.5

Class B

91, 85 81, 79 78, 77 73, 75 72, 70 65, 60

Mean = 75.5STD = 21.93 STD = 8.42

Mean

Inputs, Processes, and OutcomesInputs, Processes, and Outcomes

OutcomesInput

Measurement Materials Methods

Environment People Machines

X’s

X’s

X’s

X’s X’s

X’sX’s Y’s

Assessment StepsAssessment Steps

Develop learning/operational objectives Check for alignment between the

curriculum/business process and the objectives Develop an assessment plan Collect assessment data Use results to improve programs/department Routinely examine the assessment process and

correct, as needed

Types of Assessment – DirectTypes of Assessment – DirectAcademicAcademic

Published Tests Locally Developed Tests Embedded Assignments and Course Activities Competence Interviews Portfolios

Types of Assessment – DirectTypes of Assessment – DirectEducational Support/AdministrativeEducational Support/Administrative

People enrolled/participating/served Work accomplished Revenue generated Turnaround time Decrease in nonconformities

Types of Assessment - IndirectTypes of Assessment - Indirect

Surveys Interviews Focus Groups Reflective Essays

How Can OIRPA Assist?How Can OIRPA Assist?

Create university wide reports – Five-year departmental reports

Analyze university-wide assessment data – NSSE, CAAP

Hold workshops on assessment and IE Work with individual departments on annual reports,

program review, and outcomes assessment Provide ad hoc data reports for departments Work with committees to develop assessment plans –

IE Committee, Gen. Ed. Committee

Questions or Comments?Questions or Comments?