What’s Going On? Five Year Reviews and Assessment: Sharing Experiences so Far
Ruth Guzley, Mike Ward and Bill Loker, AURA
Five Year Program Reviews:The Broader Context …
Were routine, episodic efforts of little utility, much futility
What do we want to know about our programs? What are we trying to accomplish? How well are we meeting our goals? Academic success, good use of
resources
Assessment: Doing the Right Thing for the Right Reasons
The Intentional Curriculum: Do individual courses add up to a program?
The Reflective Department: Are we working together to provide a quality educational experience for our students?
The Culture of Evidence: How do we know the answers to the above questions?
Assessment can provide these answers …
The Assessment Juggernaut
WASC reaccredidation … educational effectiveness reviews …
http://www.wascsenior.org/wasc/Doc_Lib/2001%20Handbook.pdf
Cornerstones … CSU system-wide accountability effort focused on quality of Baccalaureate programs …
http://www.csuchico.edu/prs/AccountabilityReport_2002.doc
Educational effectiveness and other stakeholders … students, parents, legislators, etc
Aligning Five Year Review with Assessment Demands …
New Five Year Review reflects assessment concerns
Transforms FYR from an episodic time sink into a cumulative exercise in evidence gathering to improve educational outcomes and satisfy stakeholder accountability
demands Assessment only one component of
FYR
From the FYR template …
Articulating a Collective Vision: Mission and Program goalsCriteria for Review
1.1 The program has a clear and published mission statement that is appropriate for higher education and consonant with the mission and strategic priorities of the university.
1.3 The program periodically reviews and revises the mission statement as appropriate. The review process involves appropriate stakeholders.
Program Performance (Hint: Insert mission statement here.)
From the FYR template … (cont’d)
Organizing For Learning: Curriculum
Criteria for Review
2.2 The program has specified its expected learning outcomes and they have been widely shared among its members, including faculty, students, staff, and … external stakeholders.
2.5 The program has established processes for assessing student learning outcomes and for assuring that students are achieving core competencies for completion of the program.
Program Performance … (Hint: you don’t want a blank here.)
From the FYR template … (cont’d)
Organizing For Learning: Faculty Resources
Criteria for Review
3.1 Faculty recruitment, incentives and evaluation practices are aligned with … program missions and educational objectives.
3.3 The deployment of faculty reflects the mission and program goals …
Program Performance … (Hint: Need mission, goals and outcomes statements to answer these questions.)
The FYR is consequential
3. Final Review and Approval of Continuous Approval Plan
“The Review Team – consisting of an Office of the Provost representative, College Dean, Department Chair, Program Director, and an undergraduate student – will make one of four recommendations to the Provost ….
1. Recommend continuation …2. Recommend continuation , but cite specific
concerns …3. Recommend the program remain under
continuing review ….4. Recommend suspension ….
(Hint: Outcome #4 not so good …)
The Right Thing, Right Reasons, & Get a Jump on FYR Aligning Mission, Goals, Student Learning
Outcomes and Curriculum is Good Practice Best Practice engages in a process of
continuous assessment & improvement SLO assessment is a necessary component
of FYR … you don’t want blanks! Departments do things other than teach,
but excellence in undergraduate education is our #1 strategic priority …
Assessment of SLOs: evidence of success
Toward A Comprehensive Assessment Plan:
Program Mission Goals Student Learning Outcomes (SLOs) Course Alignment Matrices
An Outcomes / Course Alignment Matrix
Course SLO 1 SLO 2 SLO 3 SLO 4
151 I I
177 P I
201 P I
277 P P
340 M - oral presentation
M
477 P - term paper
Stage 1 – Ensuring a Cohesive CurriculumIn which courses are particular outcomes introduced (I), practiced (P), mastered (M)?
Assessment of SLOs
Direct Assessment Methods: Published Tests Local Exams Embedded Assignments & Activities Portfolios
Indirect Assessment Methods: Surveys Interviews Focus Groups
Assessment of SLOs
Direct Assessment Methods: Examine Student Work …
Indirect Assessment Methods: Examine Student / Alumni Perceptions
…
Effective Assessment Programs Use Multiple Measures … Triangulate
Using Rubrics to “Objectively” Evaluate Student Work …
Explicit Scoring for Student Work: Holistic Scoring
Analytical Rubrics
Program Faculty Should Discuss, Debate, Develop Some Consensus on What Methods and Measures (Rubrics) to Use …
Evaluation and Use of Data for Continuous Improvement
Assessment Data Must be Evaluated Systematically, Periodically Recommend Annual Collection / Assessment
/ Action Plan Modify Curriculum and Pedagogy to Improve
Student Learning … Typical Actions:
Modify Content, Sequencing of Course(s) Modify Pedagogical Approaches
Follow-up Focus Group or Survey to Triangulate …
Moving Toward Continuous Improvement:
The Entire Process (Goals, Outcomes, Assessment Measures) Must be Evaluated Periodically Recommend Five Year Re-evaluation to
synchronize with FYR process Are we measuring reasonable work samples? Is our process meaningful? Are we better
informed about what our students are learning or not?
Is our process sustainable? Can we be more efficient and still have valid measures?
The Assessment Plan Design Process … Wherever you begin developing your
Assessment Program will NOT be the ideal place to start …
You Must Begin Somewhere …
Iteration will save you ... Continuous Refinement of Methods Consensus and Team Building Alignment of Program Resources with
Program Goals and Faculty Values …
Case Study: Mechanical Engineering Program Mission
Mechanical Engineering Program Mission Statement
The Mechanical Engineering Program has the primary mission of providing a high-quality undergraduate engineering education with
1.A curriculum that is firmly grounded in engineering fundamentals2.A faculty that provides superior teaching and mentoring both in and out of the classroom3.A faculty whose focus is undergraduate education4.Class sizes that encourage student participation5.Project experiences that build on fundamentals and develop team skills6.Facilities and equipment that are readily accessible7.An environment that is conducive to learning and encourages students from different genders and backgrounds
Note: Not Goals or Outcomes for Student Learning !
Mechanical Engineering Program graduates demonstrate the ability to:
A. Apply knowledge of math, science, and mechanical engineering to formulate and solve problems.
B. Conduct experiments, and analyze and interpret the resultant data.C. Design mechanical components and systems to meet desired
needs, and design experiments to evaluate system performance with respect to specifications.
D. Use current mechanical engineering tools.E. Communicate technical matters effectively in oral, written, and
graphical form.F. Function effectively as members of multi-disciplinary teams.G. Understand their professional and ethical responsibilities, the
impact of their activities on society and the environment, and appreciate contemporary issues facing society.
H. Use information resources and recognize the importance of continued learning.
Mechanical Engineering Student Learning Outcomes
Mechanical Engineering Embedded Assessment Matrix:
SLOs: A B C D E F G H
Courses where student work is measured
Embedded Assessment of Outcome C in ME 138: Design mechanical components and systems to meet desired needs …
•Student grades on design projects are based on assessment of 6 design projects: 3 individual, 3 group
•A standardized rubric is used to score each design
•The performance metric:•A cumulative grade of C or better on design projects must be attained to demonstrate the ability to design mechanical components and systems (competence).
Other Measures of Mechanical Engineering Outcome C (Triangulation!)
•Senior Exit Survey Question: How well prepared are you to design a component or system to meet desired needs?
•Mean Responses (4.29 Spring 2002, 4.0 Spring 2003)
•Scale: 5=Very Well Prepared – 1=Very Unprepared
•A mean response of 4.0 is considered acceptable to faculty
Outcome C: Design mechanical components and systems to meet desired needs …