Assessment and Institutional Effectiveness: What are the links?

Where are the disconnects?Terry L. Allison, Provost and Vice President

of Academic Affairs

Imagine a world without assessment… Not “non-traditional grading”—no grades No goals, no rubrics, no accreditation No evaluation or reflection on teaching to

see what works No assessment of student services…or of business services What would that look like to you?

Assessment and Institutional Effectiveness

Much class time is taken in evaluating the institution

Students are evaluated on their exhaustive memory of small detail (and not on concepts)

We spend part of each departmental meeting examining weekly costs of supplies

All work (and vacation) stops to assess

Or a world of assessment to the extreme

Finding the proper balance:Exercise How much time should you spend per week

in assessing student work? In assessing your program? In assessing other parts of the university?

Why do we assess?

Parent/child Master/apprentice Teacher/pupil Professor/student

We assess memory, applied knowledge, communication skills, reliability, validity, quality

We “always” have assessed

Think about the first time you remember being assessed. What was your experience? Who was the teacher? Why do you think you recall the assessment?

Exercise

Built on formal assessment Grading is only about 200 years old Some universities in the U.S. and elsewhere

have experimented in NOT giving grades. Governors State University, established in 1969 as an experimental university, was one such university.

Exercise: Why would a university NOT wish to give grades?

Modern educational systems

Credentialing is a LARGE part of assessment Why are credentials so NECESSARY?

We assess to credential

It is clear for dentists, doctors—for any healthcare provider.

How about engineers? Teachers? And funeral home directors? Sociologists? Historians?

Ethics is often a factor in Credentialing.

Is credentialing equally important in all disciplines?

How do we know that one credential is as good as another? (Especially as I can order one on the web?)

Now we finally arrive at the Bologna Process!

Credentialing = Quality Assurance LEAP Value Rubric Controversy about for-profit institutions

What about portability of credentials?

Establishing norms of superior, good, acceptable and unacceptable performance

Providing advice on how to improve (a restaurant, a film, a book)

Can we assess without an aim to improve quality? (The worst critics and the worst teachers may do so.)

Exercise: Is assessment ever harmful?

We assess to improve quality

Complaining is classic assessment talk In an ideal system of university assessment,

what would happen? What would it look like?

We assess to talk with each other

Michel Foucault and modern institutions Universities as modern institutions where

we assess to exercise control

Assessment as a modern technology of control

We need regular assessment of: what students are learning how well faculty, staff, and administrators

perform whether our student and business services

meet the needs of students and other clients

whether our programs meet community needs and what communities think of us

whether our physical assets are adequate to meet our expectations and aspirations

Can we all agree?

what can be measured and with what level of sophistication

appropriate methods and instruments how much time, effort, and money

assessment should take whether those being assessed are

performing at their top level during assessments

We tend to disagree about:

whether results should be shared, compared and how

who decides what should be measured and how

what to do once we have assessed and who’s responsible for change

what is assessment for and who is the audience

We tend to disagree about…

Think of the last time you and a colleague disagreed about assessment. Why did you disagree? Were you able to resolve the difference of opinion? What do you learn from this?

Exercise

Components of effective systems of assessment

We have defined “institutional effectiveness”

In other words, we understand and (generally) agree upon what would make the institution effective

That recognition and agreement is widely distributed in some key documents including the mission, vision, and values statements and the strategic plans of the university and its major units

The most critical component

How has Sakarya University defined Institutional Effectiveness?

How do you know that your university is successful?

And your program?

Exercise

When we disagree about what constitutes institutional effectiveness:◦ Even when we use the same evaluation tools,

results will be cherry picked to prove my view◦ The same result will be interpreted very

differently◦ We will sometimes use different assessment tools

and dismiss the tools that others choose◦ We often won’t choose to

evaluate results or won’t share our results

We sometimes disagree about the Big Picture

Stakeholders contribute to and care about the outcomes and assessment processes

The audience is us; we don’t do assessment merely as a compliance function for regulatory agencies on their 10-year timeline

Assessments are performed in regular and reasonable cycles in which information can be analyzed, discussed, and used to improve

We discuss and make meaning of assessment results and choose to act or not to act on findings

Some signs of effective evaluation systems

We collectively define effectiveness and have chosen or designed the best instruments to measure what we define as significant

Some measures are longer term and we’re willing to pursue longer term follow-up to get meaningful data

Our reports are honest, accessible, and trigger action

Costs are known, discussed, and reasonable The parts relate to the whole

More signs of effective assessment

We use the most sophisticated instruments and analysis that we can afford

We try to capture some of the most difficult things we want to measure

Our assessment system includes honest external validation which we value

We may choose to employ an integrated assessment system that evaluates the whole of our efforts

And more signs…

Reflect on whether you have an effective system of evaluation for the university and for your program.

Describe how your program of assessment is effective or ineffective.

Exercise

The Provost’s Job

Keep the focus on defining institutional effectiveness—particularly but not exclusively what defines success of academic programs

Recognize assessment as a microcosm of organizational behavior (people, money, space)

Lead processes to choose which areas of strength and weakness to prioritize, and how to build on strengths and address weaknesses

Inspire, celebrate successes, &challenge to reach higher goals

Provide academic leadership

Connect the university to the national or international dialogue on assessment

Track national or international special accreditation

Lead the regional accreditation effort Focus on both assessment of student

learning and institutional effectiveness Shared regional efforts Connect the different parts of the university

Track (inter)national and local developments

Work with the other leaders of the institution to develop, choose, and implement effective processes to create, choose, and implement effective systems of evaluation.

Consider integrated evaluation systems (e.g., Baldridge) or tools (e.g., pbViews, LiveText)

Understand what works in strategic planning and how assessment is linked to mission, vision and values

Contribute knowledge and experience

What knowledge and experience can you contribute to your campus assessment effort?

How have you been a leader in assessment?

Exercise

Promote pathways so that people talk with each other respectfully and productively.

Be a GREAT listener and attentive responder Support the students, faculty, staff, and

community Support the President’s administrative team

Create and participate in effective teams