Making Student Services Assessment MeaningfulJoanna M. Oxendine, MEd

Overview Intentionality: The Meaningful-Assessment Foundation Outcome Articulation: An Assessment Tipping Point Authentic—and Manageable!—Assessment Responding Intentionally: Meaningful Change and Continual Improvement

The Intentional Assessment Cycle

Articulate Intended

Outcomes

Work Toward Intended

Outcomes

Gather Evidence

Determine Extent to

Which Outcomes Were Met

Plan for Intentional

Improvements

STUDENTS

The Culture of Intentionality

Is student-centered Seeks information about how well students are

learning and/or how well various areas of the college are supporting the student experience

Reflects on what we teach or do and how we teach or do it

Accepts (some) responsibility for student learning and the student experience Experiments with new strategies for student

success

Students become the primary focus of the assessment process

A Process of Questions

1. Why are you here? What need do you meet and for whom? (Mission)

2. What do you hope to accomplish? (Goals)

3. What measurable or observable thing(s) do you expect students to do after interacting with your office/department? (Student Learning Outcomes) What measurable or observable thing(s) does your office/department do to support the institution’s learning environment? (Service Area Outcomes)

4. Where/how will students they learn it? How will you do it? (Mapping of Activities to Outcomes)

5. How will you know if and how well students learned what you expected them to learn? How will you know if and how well you supported the learning environment? (Alignment of Measures to Outcomes)

6. What does your evidence tell you? How can you use it to make appropriate decisions for continual improvement? (Closing the Loop/Making Meaningful Change)

The Culture of Intentionality opens a clear approach to program design and assessment.

It all begins with intentionality…

and mutuality…

and hinges upon well-articulated outcomes.

Outcome Articulation:AN ASSESSMENT TIPPING POINT

Articulating Clear Outcomes Begin with the end in mind.

What is it, exactly, you want your students to be able to do? What is it, exactly, you want your office, department, or program to do with regards to administrative

process or procedure?

Experiment with articulation… SUBJECT - VERB - OBJECT Who will do what?

…until you get it right. Does your outcome accurately reflect the end you have in mind? Does it utilize active verbs that clearly articulate the expected action? Is it specific? Is it measurable or observable?

Features of Effective Outcomes Clear and succinct

Under the control of the responsible unit

Central to the unit’s mission and goals

Lend themselves to improvement

Focus on meaningful results

Focus on measuring levels of effectiveness

Utilize active verbs

Common Outcome Pitfalls Wordy

Vague

Stacked or double-barreled

Unmeasurable

Too many outcomes/unmanageable

A ReminderOutcome articulation: Directs efforts to evaluate how successful activities have been. Helps to identify what kind of information to collect.

Outcomes drive measures.The clearer, more descriptive the outcome (read “the verb used in the outcome”), the easier it becomes to authentically assess the outcome. The easier it is to authentically assess the outcome, the more useful the data becomes in making meaningful change.

Operational Verbs

Verbs describing student acquisition and preparation of tools, materials, and texts of various types (including digital and archival)

Verbs indicating what students do to certify information, materials, texts, etc.

Verbs indicating the modes of student characterization of the objects of knowledge or materials of production, performance, exhibit

Verbs describing what students do in processing data and allied information

access, acquire, collect, accumulate, extract, gather, locate, obtain, retrieve

cite, document, record, reference, source

categorize, classify, define, describe, determine, frame, identify, prioritize, specify

calculate, determine, estimate, manipulate, measure, solve, test

Verbs further describing the ways in which students format data, information, materials

Verbs describing what students do in explaining a position, creation, set of observations, or a text

Verbs falling under the cognitive activities we group under “analyze”

Verbs describing what students do when they “inquire”

arrange, assemble, collate, organize, sort

articulate, clarify, explicate, illustrate, interpret, outline, translate, elaborate, elucidate

compare, contrast, differentiate, distinguish, formulate, map, match, equate

examine, experiment, explore, hypothesize, investigate, research, test

Verbs describing what students do when they combine ideas, materials, observations

Verbs that describe what students do in various forms of “making”

Verbs that describe the various ways in which students utilize the materials of learning

Verbs that describe various executive functions students perform

assimilate, consolidate, merge, connect, integrate, link, synthesize, summarize

build, compose, construct, craft, create, design, develop, generate, model, shape, simulate

apply, carry out, conduct, demonstrate, employ, implement, perform, produce, use

operate, administer, control, coordinate, engage, lead, maintain, manage, navigate, optimize, plan

Verbs that describe forms of deliberative activity in which students engage

Verbs that indicate how students valuate objects, experiences, texts, productions, etc.

Verbs that reference the types of communication in which we ask students to engage

Verbs that describe what students do in rethinking or reconstructing

argue, challenge, debate, defend, justify, resolve, dispute, advocate, persuade

audit, appraise, assess, evaluate, judge, rank

report, edit, encode/decode, pantomime (v), map, display, draw/diagram, collaborate, contribute, negotiate, feed back

accommodate, adapt, adjust, improve, modify, refine, reflect, review

Adapted from Adelman, C. (2015). To imagine a verb: The language and syntax of learning outcomes statements. (Occasional Paper No. 24). Urbana, IL: University of Illinois and Indiana University, National Institute for Learning Outcomes Assessment.

Authentic—and Manageable!—Assessment

Identifying Measures to Gather Evidence

In general, assessment should be:

Meaningful

Measurable

Manageable

Clear and careful outcome articulation ensures it’s measureable.

The critical analysis and use of evidence for continual improvement strategies ensures it’s meaningful.

But how can we make assessment more manageable?

Let your outcomes be your guide…

Identifying Measures to Gather Evidence

What, exactly, do you intend students be able to do? What, exactly, do you intend to do?

How do you know they’re able to do it (and how well)? How do you know you did it (and how well)?

Take inventory!What are you already doing that provides evidence as to whether or not your articulated outcomes are being met?

Can you tweak any of your existing tools to serve as meaningful measures?

Let your outcomes be your guide…

It all begins with intentionality…

and hinges upon well-articulated outcomes.

Self-Diagnostic

To what extent does your department/area have a mission statement that accurately reflects who you are and what you do?

To what extent do your outcomes accurately reflect that which you expect students to do or that which you do to

support the learning environment?

To what extent do your outcomes help you identify what kind of information to collect or measures/tools to use?

To what extent do you know how well your students learned what you intended them to learn or how well your effortssupport the learning environment?

To what extent does your assessment data provide feedback and evidence to inform change and guide your continual improvement efforts?

Take 20 minutes to discuss with your colleagues the degree of your satisfaction in each of the areas of intentionality.

Example Let your outcomes be your guide… What kind of information needs to be collected?

What kinds of measures/tools might be used? Will said measure(s) provide information as to the

extent to which students met the outcome? Will the assessment data provide feedback and

evidence to inform change and guide continual improvement efforts?

Let your outcomes be your guide…

Your Turn Articulate an intended outcome. Identify the various ways in which you might gather evidence for

that outcome. Determine whether or not said measure(s) will provide information as to the extent to which the outcome was met. Evaluate the extent to which the measure(s) will provide feedback

and evidence to inform change and guide continual improvement efforts

Responding IntentionallyMEANINGFUL CHANGE AND CONTINUAL IMPROVEMENT

Responding to the Results

If an outcome meets expectations at the standard:

Take no action

Identify this area as a program strength

Consider increasing expectations outlined in the outcome

Raise the standard of attainment

Consider surveying students about their experience of the program or service area

Assessment may find that student learning or a program meets expectations at the determined standard for some outcomes.

Responding to the Results

If results for an outcome do not meet expectations:

ASK QUESTIONS!! Are our outcomes aligned with our expectations? What aspect of our program or activities might be in need of

revision? Are our teaching and learning methods in need of revision? With whom might we need to work to strengthen our

outcomes? What crazy idea might we come up with to tackle the problem?

Assessment may find that student learning or a program does not meet expectations at the determined standard for some outcomes.

Not all data is created equally…Performance Percentages? Percentages of students meeting outcomes or not reveals overall performance but does not highlight HOW students do well or go wrong.

Descriptions of Performance? Descriptions of patterns of strength and patterns of weakness can be the most revealing information, but percentages can help to define the extent of a particular problem.

Surveys? Surveys often provide a snapshot or overview of satisfaction or awareness of services, but they rarely provide authentic or direct evidence as to whether learning outcomes have been met or highlight HOW students do well or go wrong.

Let your outcomes be your guide…and your measures provide actionable data to inform decisions and plans for intentional improvement.

Let your outcomes be your guide…and your measures provide actionable data to inform decisions and plans for intentional improvement.

Finding Some Takeaways

1. What (if anything) do these exercises reveal about your own program’s assessment practices and needs?

2. Identify one thing from this workshop that you think you might be able to take back to your program to make assessment more meaningful.

Responding to the ResultsStudents benefit from an institution’s thoughtful response to an honestly undertaken attempt to determine a program’s strengths and weaknesses.

Articulate Intended

Outcomes

Work Toward Intended

Outcomes

Gather Evidence

Determine Extent to

Which Outcomes Were Met

Plan for Intentional

Improvements

STUDENTS