PREPARING FACULTY FOR PROGRAM AND LEARNING OUTCOMES ASSESSMENT
Terri Manning and Denise WellsCentral Piedmont Community College
Community College Planning and Research Organization
My Opinion (or bias) About Assessment
Learning outcomes assessment is a means to an end – not the end in itself.
It should exist within an overall evaluative process at colleges/universities where assessment is done for improvement of programs and services.
There is no such thing as a “one size fits all” model where we can all measure the same thing.
Institutions must have the freedom to determine their own outcomes based on their mission and the student body they serve.
Opinion, cont.
We must do a good job of establishing appropriate levels of effectiveness or government agencies will do it for us. We don’t need another IPEDS type of structure for learning outcomes.
The focus should be on outcomes, not just learning outcomes. All units and staff impact learning and student success therefore all units should define, measure and use outcome data to improve services and success.
The Problem with Continuous Meaningful
Assessment Institutions get the idea they are doing it for
their accrediting agencies which creates the wrong foundation
Faculty and staff don’t respond well to time-consuming processes being forced on them and being told “SACS says we have to do this.”
The accrediting agencies are requiring it because good institutions do assessment and act/react appropriately.
This is not an end into itself. It is part of a process leading to a much greater
end.
Why Continuation is Hard
Often it is driven from the top down – administration tries to tell the faculty what needs to be done.
Only have support in small pockets. It becomes one more thing we place on the
backs of the faculty. Keep adding more to their plates and taking
nothing away. My philosophy is “do as little assessment as
you can and get meaningful, useful results.” The “if 2 is good then 16 must be better”
approach must be tossed out.
The Trick: Getting Faculty to Commit to Assessment
Assessment tells you what is going on with your students
What works and what doesn’t work What skills are fully developed,
improving or absent Makes sure students have the critical
skills Makes sure students have the building
blocks from which to improve
Good Assessment
Is meaningful to faculty and students Should be authentic Should be imbedded in the classroom Becomes a critical tool for faculty Informs good teaching
What is a Learning Outcome? Learning Outcomes:
What changes in knowledge, skills, attitude, awareness, condition, position (etc.) occur as a result of the learning that takes place in the classroom. These are direct benefits to students.
Examples: general learning skills (e.g. improved writing and speaking abilities), ability to apply learning to the work environment (e.g. demonstrate skills in co-op), program-specific skills developed or enhanced (e.g. take blood pressure.)
What is a Program Outcome? Program Outcomes:
The benefits that results from the completion of an entire program or series of courses. Are there benefits for students who get the entire degree versus those who take a few courses? If so what are they?
Typical examples are: licensure pass rates, employment rates, acceptance into 4-year schools or graduate programs, lifelong learning issues, contributions to society, the profession, etc.)
What is Assessment?
Assessment – the ability to appraise or estimate the attributes of a person, group or program. Tools of assessment can include checklists, inventories, observations, needs assessments, rating scales and all types of tests.
Measurement – the process used to assign numerals to objects or constructs according to rules so that the numbers have quantitative meaning.
Tests – tools used in measurement of knowledge, attitude and behavior. Tests and testing are specific to a defined circumstances, a period of time or set of outcomes.
Direct and Indirect Measures
Direct assessment involves looking at actual samples of student work produced in our programs.
Indirect assessment is gathering information through means other than looking at actual samples of student work, such as surveys, exit interviews, and focus groups.
What is Student Learning Outcomes Assessment?
“The assessment of student learning can be defined as the systematic collection of information about student learning, using the time, knowledge, expertise, and resources available, in order to inform decisions about how to improve learning. (p.2)”
Source: Assessment Clear and Simple: A Practical Guide for Institutions, Departments and General Education by Barbara E. Walvoord, 2004
Resource
National Institute for Learning Outcomes Assessment
http://www.learningoutcomeassessment.org/
New Publication Opening Doors to Faculty Involvement in
Assessment Author: Pat Hutchings
Obstacles to Faculty Engagement
1. The language of assessment as been less than welcoming. The language (accounting, testing, evaluation, measurement, benchmarking, etc.) come from business and education – not necessarily respected disciplines on campus.
Obstacles, cont.
2. Faculty are not trained in assessment. More focus has been placed on scholarly expertise in one’s field in doctoral programs with less attention paid to reflecting on educational purposes, formulating learning goals, developing assessments and exams and using data for improvement. Not enough professional development training offered. When the conversation turns technical, faculty bow out not wanting to appear amateurs. When colleges establish an assessment office with specialized staff to manage it, it can marginalize regular faculty.
Obstacles, cont.
3. The work of assessment is not a match with internal reward systems (promotion, tenure and merit). It is more valued in an environment where teaching is the first and foremost activity. In universities where research is valued more than teaching, assessment is undervalued and not part of the promotion and tenure conversation.
Obstacles, cont.
4. Faculty have not seen evidence that assessment makes a difference. Based on the demands on faculty time and energy – faculty must make a choice – and the rational choice is often “no assessment.” Institutions have a responsibility to use assessment results to improve the educational experiences of their students. The simple presence of data does not lead to improved outcomes.
A few other issues….
In the early days the players in assessment were primarily outside of academe which made it look to faculty like “someone else’s agenda.” When faculty get the idea that this is not their job – you have a problem.
Much of what has been done with assessment has not involved large numbers of faculty in a significant way.
Faculty need to be significant participants in the assessment process – not just token members of a committee pulled together for an accreditation visit or an after-the-fact audience for assessment results they had no part in shaping.
A few other issues……
Faculty must ask (both individually and collectively) about their students: What purposes and goals are most
important for students? Are these goals being met and what
methods can we use to determine this? How can we shape and share feedback that
can strengthen student learning?
Resource Assessment Clear and Simple: A
Practical Guide for Institutions, Departments, and Genera Education
Jossey-Bass Publishers Authors: Barbara Walvoord and Trudy
Banta $26.51 on Amazon.com
Assessment Characteristics
It is intended to: to inform action not to provide “proof” (too many variable) provide the best available indicators about the
attainment of department learning goals include a variety of tools and not require
standardized test or objective measures. provide evidence on which to base decisions
about curriculum, pedagogy, staffing, advising and student support on the best possible data about student learning and the factors that affect it.
Assessment Characteristics
Faculty regularly: assess complex work in their fields
make informed professional judgments about critical thinking, scientific reasoning, or other qualities in student work
use those judgments to inform departmental and institutional decisions (p. 2)
Source: Assessment Clear and Simple: A Practical Guide for Institutions, Departments and General Education by Barbara E. Walvoord, 2004
Assessment Characteristics A lot of assessment is already going on
in responsible classrooms, departments, and institutions, though we have not always called it that.
We are asking faculty to take something they casually do and make it formal.
Assessment can move beyond the classroom to become program assessment
Assessment Characteristics
Classroom assessment – faculty evaluates her own students’ assignments in the capstone course and uses the information to improve her own teaching the next semester
Program assessment – faculty evaluates her own students’ assignments in the capstone course, outlining the strengths and weaknesses of the students’ work in relationship to departmental learning goals. The department uses the data to inform decisions about curriculum and other factors that affect student learning.
Source: Assessment Clear and Simple: A Practical Guide for Institutions, Departments and General Education by Barbara E. Walvoord, 2004
Methods of Assessment
Course imbedded assessments for learning outcomes which can be (direct): Written works Student journals Speeches Skills-based assessment (demonstrated skills) Observation checklists Tests for specific skills Teamwork assessments
Methods of Assessment
Indirect Assessment Methods Surveys that ask about specific behaviors
indicative of changes in values and attitudes Not self-evaluation of perceptions of learning Surveys that ask about effectiveness and
efficiency Qualitative methods Focus groups, key informant interviews, open ended
surveys, journaling Portfolio analysis
Authentic Assessment
A form of assessment in which students are asked to perform real-world tasks that demonstrate meaningful application of essential knowledge and skills. An authentic assessment usually includes a task for students to perform and a rubric by which their performance on the task will be evaluated.www.oaklandcc.edu/assessment/terminology.htm
Authentic Assessment Examples
Authentic assessment tends to focus on complex tasks, enabling students to demonstrate their competency in a more 'authentic' setting. Examples of authentic assessments include: performance of the skills, or demonstrating
use of a particular knowledge simulations and role plays studio portfolios, strategically selecting items exhibitions and displays
Toughest Concept
Difference between grading and assessing learning outcomes.
Course Grades are a measure of the percentage of all the requirements for a course that a student completes.
Course requirements often include things like attendance, class participation, turning work in on time, following the rules on an assignment, etc.
Grades vs. Outcome Assessment
Grades vs. Outcome Assessment
It is possible for a student to make an A and have learned little or nothing. It is possible for a student to earn an F and have learned more than anyone in the class.
Grades are - "An inadequate report of an inaccurate judgment by a biased and variable judge of the extent to which a student has attained an undefined level of mastery of an unknown proportion of an indefinite material.“ Ohmer Milton, Howard Pollio and James Eison, Making Sense of College Grades, (Jossey-Bass, 1986).
Measurements of Learning Outcomes
Poor example Student writing skills will improve
70% of students will pass the midterm - or 70% of students will pass the final – or 70% of students will make A grade of C or better in ENG 111
Good example Students will demonstrate the ability to locate,
critically evaluate, and present information 70% of student persuasive speeches evaluated will receive at
least a score of 3 or better on a 5-point evaluation rubric designed to measure effective speaking along a continuum on six construct scales.
Methods of Assessment
Surveys that ask about specific behaviors indicative of changes in values and attitudes Not self-evaluation of perceptions of learning
Surveys that ask about effectiveness and efficiency
Qualitative methods Focus groups, key informant interviews, open ended
surveys, journaling Portfolio analysis
Assessments of Behavior Changes
In what behaviors do we observe changes in students? Teamwork Cultural awareness Fear of pubic speaking Ethic issues Math phobias
How might we measure these?
Assessment
Is a developmental, iterative process Like testing swimming pool water
A Seven Step Process
What We Do With Our Faculty
First - Orientation
Instructional administration attends to show support
Review process Review of timelines Preview of program review Introduce players and their skills Have added marketing and recruitment
Second: Provide Training
Nuts and Bolts of Outcome Assessment Defines types of outcomes Establishes reasons for assessment Identifies sources for outcomes Give examples
How to Read Your Data Faculty need to see their student success data (grades, retention,
graduation, etc.) but they are not necessarily good with numbers The assessment just establishes that
students are mastering the learning and program outcomes so they can move forward to the items above
Seven Step Assessment Process
Lots of hand-holding and assistance1. Discovery - During this phase, faculty members are involved in
a working session where they are trained on learning outcomes assessment. They move into an activity on identifying “reasons for being, what we want them to get out of it and overall benefits for students.” Outcome Identification Form
2. Prioritization - During this phase they begin the narrowing of a long list of potential outcomes to the critical ones. This process involves them working through 1) what would be useful for faculty and departments to know, 2) what the faculty/department values, 3) what would inform decision-making and 4) what is most critical for student success. Outcome Prioritization Form
Seven Steps (cont.)
3. Operationalization - During this phase they create operational definitions of each outcome based on the curriculum course or program (e.g. effective writing is defined differently for English faculty than it is for early childhood faculty). Outcome Statements Form
4. Mapping - During this phase, faculty look at the natural fit between 1) courses and outcomes and 2) assessments/projects and outcomes. They create assessment processes where good ones are lacking. Outcome Assessment Form
Seven Steps (cont.)
5. Measurement - This phase involves setting realistic outcome targets, developing/using assessment tools, the process of distribution and collection of data and the reporting chain of command.
6. Analysis and Use - This phase involves analysis among appropriate faculty, determining a plan of action and writing up results.
7. Follow-up one year later on action plans (strongly recommended.)
What’s Needed to Get the Job Done?
Training Time Saving Short-cuts Templates and processes Website support and
assistance Survey/assessment
assistance Coaching and mentoring
from IE Getting faculty buy-in
Bring Faculty On-board
Pat Hutchings suggests six ways to bring the purposes of assessment and the regular work of faculty closer together in the publication: Opening Doors to Faculty Involvement in Assessment
1. Build assessment around the regular, ongoing work of teaching and learning
Build on the process of grading in every course every semester. This can bring forth questions about course design, assignments and exams and feedback to students. This is where faculty talents and interests lie. Embedding assessment in the classroom sets the stage for discussions at the program level and draws on what faculty care most about: their discipline or field. When assessment reflects and respects disciplinary interest, it is more likely to lead to important faculty engagement.
2. Integrate assessment into the preparation of graduate students;
Finding this is the exception rather than the rule but it is occurring more often.
3. Reframe assessment as scholarship
Student learning should be seen as an important phenomena for investigators. The work should not be seen as service to the college (serving on the assessment committee) but as an important intellectual exercise. Attention should be paid to the use of new forms formulas, and genres for capturing the scholarly work of teaching, learning and assessment.
4. Make a place for assessment in faculty
development Many institutions of higher education
have created centers for teaching and learning and assessment can be part of these centers. Bringing faculty together and facilitating constructive conversations around assessments, meaning, implications for pedagogy and a commitment to evidence can increase faculty engagement.
5. Create campus spaces & occasions for constructive conversations and
action
Teaching and learning have been traditionally seen and undertaken as private activities. Faculty assume responsibility for their own students and classrooms. It is foreign for them to come together in groups and discuss student learning, difficulties, strategies and possible actions as groups. Institutions have done various things from faculty learning communities, setting aside time in department meetings, multidisciplinary reading groups and inquiry groups to foster these conversations.
6. Involve students in assessment.
Student self-assessment where they monitor and direct their own development is gaining groups. New products exist such as e-portfolios, rubrics to guide student work and involve students in campus conversations about learning and assessment.
Working through the process…
Mentoring Reviewing data, reviewing what research is,
inputting and formatting data Calendars
Reviewing deadlines Putting Learning and Program outcomes
into action Using data for results
How Do We Use What We Learn
You have spent time creating a list of agreed-upon outcomes
You have created an assessment tool You have created a grading process and
established a benchmark for “success” You use the assessment tool for multiple
sections, score it and pull together the results
Your students do not pass at the level you expected
Why didn’t the pass? What do you do?
Use of Results
What processes are in place to make sure you use results to improve programs, services and student learning.
Just because you have data showing that students are having difficulty doesn’t mean you know what to do to fix it.
Can you measure, report and forget it? Then why do it?
Using Results
If faculty can easily figure it out – make changes and move on
Do some research Methods that have been proven to work by
discipline – to improve retention, engage students, help explain content, create authentic assignments, improve content mastery, etc.
May have to make some curricular changes May need more info – to drill down further
Questions We Should Ask Ourselves
1. We’re spending time and resources trying to achieve student learning – is it working?
2. When we claim to be graduating students with qualifications, do they actually have them?
3. We think they have problems with certain concepts or are weak in certain areas, do we have data to back that up?
4. When we identify weaknesses, how can we best address the problem?
5. What are our most cost effective methods of improving learning?
Things to Remember
Outcome measurement must be initiated from the unit/department level (promotes ownership of process).
Measure only what you are teaching or facilitating.
Measure what is “important” to you or your program.
Prioritize your outcomes. Be selective (2-3 outcomes only for a course
or programs). Use what you find.
Support and Contact Info:
http://www.cpcc.edu/planning
http://www.cpcc.edu/IE Presentation Posted in “studies
and reports” Terri Manning [email protected] (704) 330-6592