october 31, 2006sonoma state university, rohnert park, ca 1 (un)intended consequences: wasc...

October 31, 2006 Sonoma State University, Rohnert Park, CA

1

(Un)Intended Consequences: WASC Accreditation and Implications of the Report of the Secretary of Education’s Commission on the Future of Higher Education

Barbara D. Wright,Associate DirectorWASC

[email protected]


2

An overview …

Where assessment is today Where WASC is today Report of the Spellings Commission Implications, possibilities Discussion


3

The pendulum has swung . . .

1983-1990: heavy focus on assessment for accountability, ascertaining level of achievement

1990-2003: more focus on assessment for quality improvement, less on quality assurance

2003 – present: renewed focus on quality assurance, standard-setting


4

Shifts in our understanding of assessment

Isolated facts, skills

Memorization, reproduction

Comparing performance against other students

A full range of knowledge, skills, dispositions

Problem solving, investigating, reasoning, applying, communicating

Comparing performance to established criteria

from . . . to


5

Shifts in assessment, cont. Scoring right,

wrong answers

a single way to demonstrate knowledge, e.g. m/c or short-answer test

Simplified evidence

Looking at the whole reasoning process

Multiple methods & opportunities, e.g., open-ended tasks, projects, observations

Complex evidence


6

Shifts in assessment, cont. A secret,

exclusive & fixed process

Reporting only group means, normed scores

Scientific A filter An add-on

open, public & participatory

Disaggregation, analysis, feedback

Educative A pump Embedded


7

Shifts in assessment, cont. “teacher-proof”

assessment

Students as objects of measurement

episodic, conclusive

Reliability

Respect, support for faculty & their judgments

Students as participants, beneficiaries of feedback

continual, integrative, developmental

Validity


8

My pet assessment peeve . . .

Lots of focus on how, on process Some examples of use Nothing on the actual FINDINGS

Why not?


9

The 2001 WASC Standards:

I: Defining Institutional Purposes and Ensuring Educational Objectives

II: Achieving Educational Objectives Through Core Functions

III: Developing and Applying Resources and Organizational Structures to Ensure Sustainability

IV: Creating An Organization Committed to Learning and Improvement


10

What does WASC mean by EE?

Educational infrastructure (leadership, expertise, processes, resources)

Educational outcomes at all levels (student, program, institution, organization)

A culture of inquiry and evidence

A system of quality assurance (intentional, holistic, aligned) for student and organizational learning that demonstrates . . .


11

An evolution in WASC thinking:

FROM Standards I & III as capacity issues, II & IV as educational effectiveness . . .

TO seeing ALL four Standards as having both a capacity and an educational effectiveness dimension (see “Two Lenses on Two Reviews”)


12

An evolution . . .

FROM requesting a lot of different kinds of data about EE . . .

TO privileging direct evidence (i.e., student work and performances), although descriptive data and indirect evidence are still important


13

An evolution . . .

FROM an expectation of program review . . .

TO insistence on robust program review, including an assessment plan and a focus on student learning as a central component of a broader review


14

An evolution . . .

FROM expecting programs to define standards for student learning . . .

TO asking for evidence that students achieve those standards


15

And finally. . .

Growing awareness of the need for member institutions – and WASC – to communicate more fully with the broad public About student learning About the results of accreditation

reviews


16

The Spellings Commission Report --

A Test of Leadership. Charting the Future of U.S. Higher Education

Fall 2006


17

The context:

Boomers – “the most highly educated generation in US history” – are about to retire

Replacement workers will need higher educational levels for a knowledge-based global economy

College costs are rising, results declining The “education gap” and “ambition gap” (Th.

Friedman) put US at competitive disadvantage Prosperity, national security are at risk


18

The Commission on the Future of Higher Education – Issue Paper #2

National Assessment of Adult Literacy: less than 1/3 of college graduates could read complex texts, make inferences

National Survey of America’s College Students: significant numbers of college grads (20-30%) have only basic quantitative skills

Employers: college grads lack skills for the workplace

“Accountability/Assessment” by Charles Miller & Geri Malandra


19


Colleges’ standards are “fuzzy” HE is “focused inward,” doesn’t

communicate goals, outcomes to the public “There are no commonly used tests or

other assessments” “College courses are not designed to

foster critical thinking and problem solving, which are hard to address with lecture formats and multiple-choice tests.”


20


AAC&U’s approach (Our Students’ Best Work, 2004; and Liberal Education Outcomes, 2005)

Accreditation’s focus on student learning SHEEO’s endorsement of a focus on student

learning Business Higher Education Forum’s endorsement

of multiple methods and linking with K-12 education

New instruments (NSSE, CLA, ETS’ MAPP)

“Promising efforts” include:


21

Spellings Commission concerns . . . HE grads too often deficient in

Reading and writing Math Critical thinking Problem solving Ability to be life-long learners

Other countries overtaking the US in HS graduation rates College-going rates Baccalaureate graduation rates


22

Concerns, cont. . . .

Lack of alignment between HS and HE expectations for learning

Lack of information available to students, parents about costs, comparative quality of colleges

Achievement gaps (racial, ethnic, SES) HE insufficiently innovative, adaptable,

efficient, focused on outcomes Accreditation part of the problem, not a

solution


23

On quality, transparency, and accountability:

“Despite increased attention to student learning results by colleges and universities and accreditation agencies, parents and students have no solid evidence, comparable across institutions, of how much students learn in colleges or whether they learn more at one college than another… Accreditation reviews are typically kept private, and those that are made public still focus on process reviews more than bottom-line results for learning or costs” (p.13-14).


24

Commission’s Recommendations …

Seamless alignment of high school and college curricula

Performance outcomes as “the core” of accreditation, over inputs or processes

Technology and new pedagogies used to get more learning at lower cost

A “culture of accountability and transparency” with “consumer–friendly” information


25

Recommendations, cont. “Institutions should measure and report

meaningful student learning outcomes” (CLA, NSSE, MAPP noted)

Availability of test scores, certification and licensure rates, time to degree, other data should be “a condition of accreditation”

User-friendly data should allow inter-institutional and interstate comparisons


26

The challenge . . .

To find ways to assess student learning – and report the findings – in ways that support education and do no harm.

Standardized tests are a legitimate part of the picture, but they can’t be the whole picture.


27

Choice of assessment method matters.

Students value and learn what we teach and test.

How we teach and test matters as much as what

What and how we assess also matters.

We get more of what we test or assess, less of what we don’t.


28

Higher-order thinking …( adapted from L. Resnick, 1987)

It’s nonalgorithmic, i.e., the path of action is not fully specified in advance.

It’s complex, i.e., the total path is not “visible” from any single vantage point.

It often yields multiple solutions, each with costs and benefits.

It requires nuanced judgment and interpretation

It involves application of multiple criteria, which may conflict with one another


29

Higher order thinking, cont …

It often involves uncertainty; not everything about the task is known or can be.

It requires self-regulation; someone else is not giving directions.

It involves making meaning, discerning patterns in apparent disorder.

It is effortful: the elaborations and judgments required entail considerable mental work and are likely to take time.


30

Other ways to think about higher-level learning . . .

Bloom’s taxonomy (knowledge, comprehension, application, analysis, synthesis, evaluation)

Perry Scheme of Intellectual Development (dualism, multiplicity, relativism, commitment)

Biggs’, Entwistle’s “surface” vs. “deep” learning

Constructivist approaches ???


31

Methods for complex outcomes …

are direct are open-ended focus on essentials, principles pose authentic, engaging tasks require meaning-making, judgment require active expression are scored for understanding, not just

regurgitation


32

Methods for complex outcomes include …

Portfolios Capstones Performances Common assignments Secondary readings Course management programs Local tests, comps in the major The CLA


33

Four dimensions of learning --

What students learn (cognitive as well as affective, social, civic, professional, spiritual and other dimensions)

How well (thoroughness, complexity, subtlety, agility, transferability)

What happens over time (cumulative, developmental effects)

Is this good enough?


34

Thinking About Standards . . . Absolute standards: the knowledge/skill level

of champions, award winners, top experts Contextual standards: appropriate

expectations for, e.g., a 10-year old, a college student, an experienced professional

Developmental standards: amount of growth, progress over time, e.g., 2 years of college, 5 years

(Institutional, regional, national standards?)

We know “what,” but do we know “how well,” or what is “good enough”? How might we know?


35

Once assessment is in place. . .

We need to not only use our findings to make improvements;

we need to reveal our findings and use them to set standards for

college-level learning

This is the next frontier in assessing and improving students’ learning and development. If we don’t do it, others will.


36

We’re not starting from zero – assessment is a foundation to build on:

Outcomes – pervasive Gathering student work – in process Analysis and use – coming along Public sharing of findings – ? Developed institutional standards – a

handful? Cross-institutional standards, rubrics

and examples of student work – not yet, but why not?


37

We all have implicit standards, e.g., for

Remedial writing, math Freshman writing, math Course-level judgments about grades,

especially Ds, Fs Competency-based curricula

As we did with outcomes, now we need to make the implicit explicit and collectively owned.


38

What do students know, and how do we – all – know they know it? Together we …

What do students know? Is it good enough? A course of action might be to…

Analyze student work for quality, proficiency Determine the effective ceiling & floor of our

students’ learning at different points Compare it to work at other institutions Begin to reach a collective sense of “college-

level learning” in a few key areas And communicate that level of learning to

students, parents high school faculty and students policy makers the general public


39

It won’t be easy. Plenty of factors can work against the use of standards.

Traditional proxies: requirements, credits, seat time

Grade inflation The unspoken faculty/student truce (I won’t

bother you if you don’t bother me.) Insufficiently rigorous AP and dual

enrollment courses Lack of shared culture, trust The complexity of the whole issue: what

standards, for whom, applied how, with what consequences?


40

Are parents & students really interested in learning?

“Research suggests … otherwise… College students tend to look upon knowledge and ideas less as ends in themselves and more as a means toward accomplishing other goals, such as … achieving success in their careers … Rather than measures of student learning, indicators such as graduation rates, employment rates, and cost of attendance are some of the “consumer information” colleges and universities should make readily available” (NASULGC, 2006).


41

Having those standards means We can have substantive

conversations with students, parents, employers, policy makers, and the public.

We have not just numbers (data) but specific, substantive, outcomes-based information to inform conversations across the educational spectrum, K-16 and beyond.


42

Parade Magazine? Are you kidding?

respond to public need, interest inform the public, build intellectual

capital in the wider society – and among policy makers

allow parents, high school, others to internalize and reinforce our message

No. Communication is essential. It will


43

Communication, cont. be educative for students, show them

what they have mastered or still need to do

shift some responsibility for education to these others

free us in HE from taking ALL the responsibility for a complex process that can never succeed if it is ours alone.


44

Questions for discussion …

How could your institution use assessment to set standards? What efforts could you build on?

How might commercial and local approaches be combined?

What are the benefits/drawbacks of Institutional standards? Regional standards? Consortial standards? State or national Standards?

Should we be doing this at all?


45

What about WASC?

A user group for campuses using NSSE A user group for campuses using the CLA A group to explore cross-institutional

rubrics for Capstones Portfolios

A group to inform WASC about institutional experience with standardized tests

WASC plans several projects in the coming year:

october 31, 2006sonoma state university, rohnert park, ca 1 (un)intended consequences: wasc...

Documents

sonoma state university

rohnert park

improvement slide

developmental validity

pump embedded slide

possibilities discussion

pet assessment peeve

quality improvement