Institutional Effectiveness:Developing an Institutional Philosophy

for the Quest for Quality Terri M. Manning, EdD.

Central Piedmont Community College

National Council for Instructional AdministratorsApril 4, 2008

Institutional Effectiveness (IE)

• A philosophy with a set of processes

• A combination of the content from the following fields:

– Assessment

– Program Evaluation

– Quality Assurance

– Organizational Development

• You combine these areas and apply the result to higher education and you get IE

A Definition• Institutional Effectiveness is:

• an ongoing, integrated and systematic set of institutional processes that include planning, the evaluation of programs and services, the identification and measurement of learning outcomes, the use of data and assessment results for decision-making that results in improvements in programs, service and institutional quality.

You have had a speaker on this topic for the last several years – yet you ask for additional training on this topic.

Why is this hard for institutions to accomplish?

Why This is Hard for Many Colleges• Colleges want someone to tell them exactly

how to do this – what is expected.• They think they are doing it for their

accrediting agencies.• They don’t create processes that are

valuable to the institution.• They are either too simple or too complex.• They try to drive it from the top down –

administration tries to tell the faculty what needs to be done.

• It has no central coordination of efforts – no natural connection between processes.

Why It’s Hard, cont.

• Their approach is like students in the classroom - “what exactly do we have to do to get an A on the test.” They completely miss the concept. They don’t “get it.”

• We do not measure learning outcomes so we can say we did it.

• This is not an end into itself.• It is part of a process leading

to a much greater end…. Improvements in institutional quality.

What You Are Really Doing • Is attempting to create institutional change

by:– Getting faculty to value something different– Measure themselves differently– Have a different philosophy– Reward themselves for

something different– Shake up their power

structure

• Is it any wonder we have trouble with this?

Lewen’s Organizational Change Model

• Unfreezing – reducing the forces that maintain the organization’s current behavior. One introduces information that points out discrepancies between “what we have” and “what we want.” This is supposed to motivate members of the organization to engage in change behaviors.

• Moving – shifts the desired behavior to a new level by intervening in the system to develop new values, attitudes and behaviors through changes in processes.

• Refreezing – stabilizing the new behaviors by the use of support mechanisms such as a new culture, policies, norms and structures.

Source: Essentials of Organization Development & Change, Cummings & Worley

What Holds Us Back

• Barriers to Individual and Organizational Change– Failure to recognize the need for change

• We are fine like we are – no need for improvement

– Habit• That is not how we have always done it

– Security• My position or rank or the department’s

standing could be affected

– Fear of the unknown• What if we don’t meet our outcomes (our students aren’t

learning), we might be held accountable for it

– Previous unsuccessful efforts• We tried that before and it failed

What Holds Us Back, cont.

– Threats to expertise• Assessment results are not going to force me to teach my class

differently or to improvement

– Threats to social and power relationships• The one on top may no longer be on top, no longer the favorite

or in control

– Threats to resource allocation• My department or program might lose funding if we do not

perform as well as other departments or programs

(from Greenberg & Baron, 2000; Jones, 2004; Robbins 2000)

Mandates for Institutional Effectiveness• In 2006, U.S. Secretary of Education

Margaret Spellings released “A Test of Leadership: Charting the Future of U.S. Higher Education.” This report addresses the need to transform higher education. – “There is inadequate transparency and accountability for

measuring institutional performance, which is more and more necessary to maintaining public trust in higher education.” (p. 14)

– “To meet the challenges of the 21st century, higher education must change from a system primarily based on reputation to one based on performance. We urge the creation of a robust culture of accountability and transparency throughout higher education.” (p. 21)

Mandates for Institutional Effectiveness

– “The commission supports the development of a privacy-protected higher education information system that collects, analyzes and uses student-level data as a vital tool for accountability, policy-making, and consumer choice.” (p. 22)

– “Faculty must be at the forefront of defining educational objectives for students and developing meaningful, evidence-based measures of their progress toward these goals.” (p. 24)

– “The results of student learning assessments, including value-added measurements that indicate how students’ skills have improved over time should be made available to students and reported in the aggregate publicly.” (p. 24)

Accreditation Has Changed• Over the past 3-5 years, all six accrediting

agencies have made significant changes in their processes – all requiring assessment and evaluation and the inclusion of student learning outcomes. The words “institutional effectiveness” are mentioned in the criteria for most accrediting agencies.

• College and universities can no longer wait until 1-2 years before an accreditation visit and “gear up” for it.

• Colleges and universities are starting to get in trouble with their accrediting agencies.

• It’s going to get worse before it gets better .

What Do The Accrediting Agencies Want

Mandate for Identifying, Measuring and Using Outcome Data

  Identify Assess Analyze Use For

Accrediting Body Outcomes Outcomes Results Improvement

Middle States7, 11, 12, 13, 14 X X   X

New England4.18, 4.28, 4.44, 4.45 X X   X

North Central2c, 3a, 4b, 4c X X   X

Northwest2b, policy 2.2 X X   X

Southern2.5, 3.3.1, 3.5.1 X X X X

Western4.6, 4.7, 4.8 X X X X

Source: Gita Wijesinghe, Florida A&M

The Reality

• At our institutions, our policies and practices are set up to perfectly deliver the results we are currently achieving.

• Are they good enough?

I ask you:… if you are going to ask your

faculty to participate in and embrace institutional effectiveness, to invest the time to develop effective processes, to assess, analyze and use results - - -

….once it is all done, what would you want to know?

Here is what others have said…• What characteristics should our graduates have and

do they have them?• Are our students learning?• Are our students succeeding?• Are some programs having trouble?• What will programs need to succeed in the future?• What is new and innovative coming our way (by

discipline)?• Where should our enrollments be (growing vs.

declining)?• What are the costs?

Institutional Effectiveness• Is a way of thinking…a philosophy• Is the way an institution keeps its finger to

the pulse of its various communities.• It is about student success and student

learning.• It is mostly about an institution’s continuous

quest for quality, efficiency, effectiveness and innovation.

Institutional Effectiveness• It answers the questions:

– Are our students learning what we intended for them to learn

– Can they apply what they learned in the real world

– Are we serving our students well– How can we improve, innovate and create– What is the real value of an education obtained

from this institution

Institutional Effectiveness• There is no prescription – no commonly

accepted set of practices or processes

• Can be culturally changing to serious institutions

• Is a unique system of questions and inquiry

• Means looking at the questions before we come up with answers

• Means looking at the data

• It facilitates a culture of evidence

Educating Students• Is a developmental process• Unique to each institution• Yet with some common core values• It is like parenting

– Would you want anyone to give your children an exam that measured how good of a parent you were, the quality of your family life…

– Your first question would be: who made the decision on what a “good parent” is or what qualities are acceptable in families. While we all hold some common values about families and children, we are very different.

Difficulty for Faculty

• Issues to discuss/work on:

• What do we value?

• How do we operationally define it?

• Can we all agree on a few characteristics we value and measure them?

• Is that enough?

• Who decides if it is enough?

• How much more are we going to add to their plate without removing anything?

Effective Institutions• Don’t make decisions, change policy or practice

without the evidence to give them direction• When they start something new, they evaluate it

heavily to make sure it produces the desired effect• When things don’t work they fix it• Don’t set students up to fail (policy and practice)• They create a continuous flow of information to the

opinion leaders, decision-makers and stakeholders• Anticipate change and are prepared• Listen to their front-line employees – those most in

touch with their students/customers/clients (faculty and counselors/advisors)

Good Practices• Institutions who fully participate in IE:

– Figure out why students are not progressing and make changes

– Discover student barriers to success and remove them

– Assess effective teaching modalities and expand and enhance them

– Openly uncover their weaknesses and work toward strengths

It is all about…• Your students

• Their success

• Faculty and staff as facilitators

• The quality of your institution

• The responsibility we have to tax payers, students, our community and state

• A desire for continuous improvement

It is Not About• Doing what we have always done but in a

different format

• A prescribed set assessment tools

• Filling out forms and putting documents in order

• Paying lip-service to processes

• Getting through an accreditation visit

• This is so much bigger than an accreditation process or agency

Four Most Important Elements • Support from the administration

• “Buy in” and trust from the grassroots level – the development of relationships

• Freedom and fairness for those being reviewed

• Effective processes that work for your institution - that are supported and followed

Support from the Administration

• If you don’t have it nothing works• It is hardest where there is the unspoken rule “no

bad news”• Faculty/staff can’t fight like salmon swimming up-

stream• You have to be creative and develop perseverance• You are there to get the job done• Some times it takes more than once for your

message to get across

Support from the Administration• To obtain this support, you have to:

– Know what you are doing– Develop simple and clear processes– Fit the “administrative response” into your

processes– Report results to gain visibility– Show how this relates to the bottom line and will

increase enrollment growth and retention

Buy In From the Grassroots Level

• Allow the faculty to create the process themselves (as much as possible)

• You have to believe that what they are doing is critical to the success of the institution

• You have to understand and validate that “they don’t have time for this”

• Build friendships and trust. Function as a helper.

• Be a faculty advocate and be proactive for their issues

Buy In From the Grassroots Level

• Never be perceived as the one who is “grading their performance”

• Roll up your sleeves and get involved

• Seek to understand their issues

• Give them tools and templates – put things online

• Create rewards

• Realize their #1 role is to teach, not to be assessment experts

Freedom and Fairness

• Never use data against them• Give them as much input as possible – give

them flexibility when you can• Allow them to measure what they believe is

important• Aggregate data when necessary• Try to get the message across “This is a

process with the goal of a continuous quest for quality. It is okay if things are not perfect – then we know where strategies for improvement need to be utilized.”

Effective Processes

• Why develop a process?– There is an expectation of a due date– There is a format to follow that utilizes standards– Processes usually have outlines and calendars – Processes can be revised to include everything

important to the institution

Typical Processes• A Mission Statement• A Strategic Plan

– Institutional Goals and Outcomes

• Annual Goal/Objective Cycle for All College Units/Departments

• Annual Research and Assessment Reports• Program, Learning and Administrative Outcomes

– Collect through processes• General Education Assessment

• Program/Unit Review

• Outcome Assessment Matrices

Today we are going to talk about:

• Two areas critical to instruction/academic affairs:

– The Development and Assessment of Learning Outcomes

– The Development and Assessment of General Education Learning Outcomes

Why are We Moving from Goals to Outcomes?

• Outcomes are program-specific• They measure the effect of classroom activities

and services provided.• Outcomes represent a new way of thinking• Outcomes have become widely accepted by

our various publics• They are here to stay• They are skills-based variables you can

observe, measure, scale or score

Program Outcome Model

Resources Services Products or Results of Activities

Staff Education (classes) Numbers servedFTE Services FTE (input next year)Facilities Counseling # Classes taughtState funds Student activities # Students recruitedAbility of Students

ConstraintsLawsState regulations

INPUTS ACTIVITIES OUTPUTS

Theory of Change Model

Program Outcomes Model

Benefits for People

*New knowledge *Increased skills

*Changes in values *Modified behavior *Improved condition *Altered status *New opportunities

ACTIVITIES OUTCOMES

(Outcomes answer the “so what” question)

OUTPUTSINPUTS

Inputs through Outcomes: The Conceptual Chain

Inputs

Activities

Long-range

Outputs

Intermediate

Initial OUTCOMES

Different Types of Outcomes

• Learning Outcomes (can be at course, program or institutional level)

• Program Outcomes

• Administrative Outcomes

Definitions and Examples• Learning Outcomes:

– What changes in knowledge, skills, attitude, awareness, condition, position (etc.) occur as a result of the learning that takes place in the classroom. These are direct benefits to students.

– Examples: general learning skills (e.g. improved writing and speaking abilities), ability to apply learning to the work environment (e.g. demonstrate skills in co-op), program-specific skills developed or enhanced (e.g. take blood pressure.)

Definitions and Examples• Program Outcomes:

– The benefits that results from the completion of an entire program or series of courses. Are there benefits for students who get the entire degree versus those who take a few courses? If so what are they?

– Typical examples are: licensure pass rates, employment rates, acceptance into 4-year schools or graduate programs, lifelong learning issues, content mastery, contributions to society, the profession, etc.)

Definitions and Examples

• Administrative Outcomes – Units/programs want to improve services or approach an

old problem in a new way.– They want to become more efficient and effective.– They establish an outcome objective for the administration.

• Typical examples are:– All faculty will attend one professional meeting annually so

they can stay up-to-date in their field, or:– Counseling wants to recruit a new counselor with expertise

in working with first-generation students, or:– Facilities services wants students, faculty and staff

to feel that they are safe on campus.

Why is This Hard?

• Because it is education

• Because the best results may not happen for years

• Because we are so busy doing what we are doing…. we forget why we are doing it

Assessment

• “The assessment of student learning can be defined as the systematic collection of information about student learning, using the time, knowledge, expertise, and resources available, in order to inform decisions about how to improve learning. (p.2)”

Source:Assessment Clear and Simple: A Practical Guide for Institutions, Departments and General Education by Barbara E. Walvoord, 2004

Assessment Characteristics• It is intended not to generate broad theories but to inform

action.• Educational situations contain too many variables to make

“proof” possible. • Assessment gathers indicators that will be useful for decision

making.• It is not limited to learning that can be objectively tested. A

department can state its highest goals and seek the best available indicators about whether those goals are being met.

• It does not require standardized test or objective measures.• Faculty regularly assess complex work in their fields and make

judgments about its quality.• Faculty can make informed professional judgments about

critical thinking, scientific reasoning, or other qualities in student work, and use those judgments to inform departmental and institutional decisions. (page 2)

Source:Assessment Clear and Simple: A Practical Guide for Institutions, Departments and General Education by Barbara E. Walvoord, 2004

Assessment Characteristics• Assessment means basing decisions about curriculum,

pedagogy, staffing, advising and student support on the best possible data about student learning and the factors that affect it.

• A lot of assessment is already going on in responsible classrooms, departments, and institutions, though we have not always called it that.

• Assessment can move beyond the classroom to become program assessment:– Classroom assessment – faculty evaluates her own students’

assignments in the capstone course and uses the information to improve her own teaching the next semester

– Program assessment – faculty evaluated her own students’ assignments in the capstone course, outlining the strengths and weaknesses of the students’ work in relationship to departmental learning goals. The department uses the data to inform decisions about curriculum and other factors that affect student learning. (page 2-3)

Source:Assessment Clear and Simple: A Practical Guide for Institutions, Departments and General Education by Barbara E. Walvoord, 2004

Methods of Assessment

• Course imbedded assessments• Written works• Student journals• Speeches• Skills-based assessment (demonstrated skills)• Observation checklists• Teamwork assessments• Surveys that ask about specific behaviors

indicative of changes in values and attitudes– Not self-evaluation of actual learning

Great Fallacy #1• Grades

– In this day of social promotion, grade inflation and different teaching/learning philosophies, grades tell you virtually nothing.

– They are not a measure of outcome achievement.– Two teachers will grade a student

differently for the exact same work.– They cannot be used!

Great Fallacy #2• The evaluation of teaching

– Only tells you who is happy…”happiness coefficient”– Measures more about the business and science of teaching

– not learning– Learning outcomes are not measures of teacher

effectiveness or students satisfaction with the teaching/learning process

– Learning outcomes are skills based– Students opinions are poor indicators if asked to self-assess

• “In your opinion, are you more accepting of cultural differences in people since completing your coursework at the college?”

• It is why we don’t ask faculty “On a scale of 1-5, how good of a teacher do you feel you are?”

Great Fallacy #3

• I have 65 learning objectives on my syllabus

• The faculty has established that when students complete ENG 111, they will have met these 65 objectives – so we are establishing and measuring outcomes

• What is the problem with this?

How to Get Faculty On-board• Get them involved at the first step• Give them as much control as possible• Allow for a few tantrums – then hit it again• Allow them to create over an adequate period of time• Make use of opinion leaders• Help them understand “why”• Expect them to do an excellent job• Start with what you have – what you’re already doing• There is no magic bullet• There is no one-size-fits-all with outcomes

To Move Quickly• Create Several Committees made up of mostly

faculty and your IR/IE/Assessment officers– Examples

• General education committee• Learning outcomes committee• Program review committee

• Give them an assignment:– Find as many good examples of learning outcomes

from various sources as you can…. other colleges, accrediting agencies, the literature, standardized tests (my least favorite), national organizations and bring it back to the group.

From That List – Establish Values• Are any of these relevant to our students?• Can we reword them to make them relevant to

our students?• Are we teaching these concepts/constructs?• If so, where?• How can we assess them?• How can we use the results?• Of this list, what is the most important to us?• What would it be best to know?• How much can we do with our current

staffing and resources?

Sources of Ideas for Outcomes* program documents* program faculty and staff* national associations/credentialing boards* key volunteers* former students* parents of students* records of complaints* programs/agencies/employers that are the

next step for your students* other colleges with similar programs,

services and students as yours* outside observers of your program in action

How Often • Should we measure student learning

outcomes every year – every semester?• Benefits and barriers with timing?

• When does measurement become too time consuming?

• Units need time to put into effect the changes made as a result of outcome assessment before they are thrown back into another cycle. They need time to reflect on changes and results.

Disappointing Outcome Findings: Why Didn’t We Meet Our Objectives?

Internal Factors:

* Sudden faculty/staff turnover* New teaching philosophy/strategy* Curricular change (campus move)* Unrealistic outcome targets* Measurement problems (lack of follow-through, no effective tracking)

Disappointing Outcome Findings

External Factors:

* Community unemployment increases* State funding changes* Related programs (BS or MS programs) close* Public transportation increases fares or shuts down some routes serving your campus or time slot* Employment trends change

Failing to meet your learning outcome objectives is sometimes the best thing that can happen to you!

Why?

Use Your Findings

Internal Uses for Outcome Findings

• Provide direction for curricular changes

• Improve educational and support programs

• Identify training needs for staff and students

• Support annual and long-range planning

• Guide budgets and justify resource allocations

• Suggest outcome targets (expected change)

• Focus board members’ attention on programmatic issues

• Help the college expand its most effective services

• Facilitates an atmosphere of change within the institution

An Example from One College Program

• Workplace Literacy Program– This program is a literacy initiative that goes directly

into the worksite and teaches ESL classes, GED prep and GED classes. Serves mostly immigrants.

• During their first attempt at outcome assessment, they surveyed both employers and students.

• This was the first time they had ever done this.

What They Learned• Employers said:

– 43.8% of employers reported increases in employee performance as a result of participation in the program.

– 31.3% reported a reduction in absenteeism by participants.

– 87.5% said classes improved the morale of their employees

– 37.5% said participants received raises– 50% said communication had improved.

What Students Said

• 70.2% reported being able to fill out job forms better

• 35.5% said they could now help their children with their homework

• 91.1% said they felt better about themselves

• 44.4% said they had received a raise, promotion or opportunity as a result of the courses

• 86.3% said their ability to communicate in the workplace had improved

What Has Happened Since

• Their assessment data has shown up in their marketing brochures to employers.

• Their enrollment has grown dramatically.• They have received funding and marketing

support from their local Chamber and are considered a model adult literacy program.

External Uses of Outcome Findings

• Recruit talented faculty and staff• Promote college programs to potential

students• Identify partners for collaboration (hospitals,

businesses, etc....)• Enhance the college’s public image• Retain and increase funding• Garner support for innovative efforts• Win designation as a model or

demonstration site

Good Sources for Outcomes (besides your own faculty)

If you can’t come up with anything good…..• Steal it from someone who else who did.• Two Good Sources:

– Longwood University 201 High Street Farmville, Virginia 23909 Phone: (434) 395-2000 http://www.longwood.edu/gened/15goals.html

– Pellissippi State Technical Community College10915 Hardin Valley RoadKnoxville, TN 37933-0990865-694-6400 http://www.pstcc.edu/departments/curriculum_and_ instruction/currinfo/general-ed-outcomes.html

Typical General Education Goals• Basic computer/technology skills• Information literacy• Critical thinking/analytical thinking• Effective communication (both oral and written)• Moral character and values• Cultural diversity• Workforce skills• Reading and comprehension• Appreciation for the fine arts• Computational skills• Oral presentation skills• Teamwork skills• Leadership skills• Social awareness Interpersonal skills• Self management• Understand scientific reasoning/the scientific process or way of

knowing• Lifelong learning skills

Look at Two General Education Processesin Your Handouts

Top Ten Skills for the Future • Work ethic, including self-motivation and time management.• Physical skills, e.g., maintaining one's health and good appearance.• Verbal (oral) communication, including one-on-one and in a group• Written communication, including editing and proofing one's work.• Working directly with people, relationship building, and team work.• Influencing people, including effective salesmanship and leadership.• Gathering information through various media and keeping it

organized.• Using quantitative tools, e.g., statistics, graphs, or spreadsheets.• Asking and answering the right questions, evaluating information, and

applying knowledge.• Solving problems, including identifying problems, developing possible

solutions, and launching solutions.

The Futurist Update (Vol. 5, No. 2), an e-newsletter from the World Future Society, quotes Bill Coplin on the “ten things employers want [young people] to learn in college”

Learning Outcomes for the 21st Century Students in the 21st Century will need to be proficient in:• Reading, writing, speaking and listening• Applying concepts and reasoning• Analyzing and using numerical data• Citizenship, diversity/pluralism • Local, community, global, environmental awareness

• Analysis, synthesis, evaluation, decision-making, creative thinking• Collecting, analyzing and organizing information• Teamwork, relationship management, conflict resolution and

workplace skills• Learning to learn, understand and manage self, management of

change, personal responsibility, aesthetic responsiveness and wellness

• Computer literacy, internet skills, information retrieval and information management

(The League for Innovation’s 21st Century Learning Outcomes Project.)

Challenges• Identifying and defining outcomes is the easy

part.

• The devil is in the details.

• How do we track it, where does it all go, how do we score it, compile it, turn it into a comprehensive report.

• How do we “demonstrate improvement in institutional quality.”

Things to Remember• Outcome measurement must be initiated from the

unit/department level (promotes ownership of process).

• Measure only what you are teaching or facilitating.

• Measure what is “important” to you or your program.

• Be selective (2-3 outcomes only for a course, a select list for programs and institutional outcomes).

• Put as much time in to “thinking through” the tracking process as you do into the definition of outcomes.

• Spend the time up front in planning and the process will flow smoothly.

• It will prove to be energy well spent.

Remember

• We do not do outcome assessment/evaluation so we can say we did it.

• We do it only for one reason:

–To Improve Programs and Services

Where Colleges Get In Trouble• Overkill – they evaluate everything that

walks and breaths every semester in every area.

• No time to “reflect” before they enter back into another assessment cycle.

• No focus on “use of results.”

• No ability to track results and tally them across the College.

Contact Information• Copy of presentation:

– www.cpcc.edu/planning– Click on “studies and reports”– Posted as “AACC NCIA session”– CPCC’s IE site– www.cpcc.edu/IE– Terri Manning

• terri.manning@cpcc.edu• (704) 330-6592