program outcome assessment: case study from three engineering programs presented by toni l. doolen,...

PROGRAM OUTCOME ASSESSMENT: CASE STUDY FROM THREE ENGINEERING PROGRAMS Presented by Toni L. Doolen, Professor School of Mechanical, Industrial & Manufacturing Engineering Oregon State University 09/21/2011

Abstract This presentation will introduce a framework for a continuous improvement process for programs driven by program outcome assessment. A case study illustrating the application of this framework to programs in the College of Engineering will be presented. Program outcome data collected from different sources (courses, students, and alumni) is being used to provide feedback to evaluate whether or not changes are needed in programs. A set of processes has been implemented to create an annual review cycle with a focus on continuous improvement of programs. Specific examples and templates for program outcome assessment and reporting will be provided.

Agenda Overview of the School of Mechanical, Industrial, and Manufacturing Engineering (MIME) Assessment and Evaluation in MIME Program Educational Objectives Student Outcomes Course Learning Outcomes Continuous Improvement process

MIME Demographics ~35 full-time faculty members and instructors ~185 graduate students in three programs (industrial engineering, material science mechanical engineering) ~1,375 undergraduates in five programs (energy engineering management, general, industrial, manufacturing, and mechanical) ~ $7.3 million in research funding in 2010

Need for Assessment and Evaluation Undergraduate programs accredited under ABET Many employers require accreditation Students must graduate from accredited engineering program to sit for PE (Professional Engineering exam)

Three Levels of Assessment and Evaluation Program Educational Objectives (PEOs) Student Outcomes (SOs) Course Learning Outcomes (CLOs)

Assessment Data collected from Multiple Stakeholders Students Alumni Industrial Advisory Board Members Employers (local, regional, national, international)

Levels of Assessment in MIME MIME Program Educational Objectives (PEOs) Student Outcomes Course Learning Outcomes

Program Educational Objectives Program educational objectives are broad statements that describe what graduates are expected to attain within a few years of graduation. Program educational objectives are based on the needs of the programs constituencies. Program constituencies: students, faculty, employers/graduate schools, alumni, state of Oregon,

PEOs A & E Process Defined for programs Our graduates will be systems thinkers. Our graduates will be global collaborators. Our graduates will be innovative designers and problem solvers. Assessed and evaluated using alumni (3 5 years after graduation) survey data, employer survey data, and IAB input. Responsibility for A&E and CI process is MIME Head/AH, ABET coordinator, and undergraduate program committees

MIME Program Educational Objectives MIME graduates will be able to analyze, evaluate, improve, and design engineered systems and processes using modern engineering tools (hardware and software) and approaches. They will demonstrate in-depth knowledge of mechanical, industrial, and/or manufacturing systems. Our graduates will be systems thinkers. MIME graduates will be able to communicate effectively across disciplines and cultures. They will provide management and leadership skills within their organizations and work effectively in diverse environments. Our graduates will be global collaborators. MIME graduates will use both structured and unstructured methodologies to innovate systems and processes. They will apply technical know-how and creativity to real-world problems. They will demonstrate a broad understanding of engineering design and problem-solving processes. Our graduates will be innovative designers and problem solvers.

PEO Assessment Data from Alumni Assessment of MIME PEOs is completed using an annual Alumni Survey that is administered to alumni 2 5 years after graduation. Items include specific questions about PEOs. Rated on a 7-point scale (1=not at all; 7=extremely). Respondents also rate importance of each PEO. Survey provides an assessment of PEO achievement directly and also uses aggregated Student Outcome achievement ratings provided by alumni for a second measure. Other demographic details are collected related to employment and general satisfaction with educational experiences.

PEO Evaluation PEOs and assessment results reviewed with IAB annually. PEO data analysis and recommendations prepared by ABET coordinator. Analysis and recommendations reviewed with UPCs on an annual basis and actions for improvement identified.

Linkage of PEOs to Student Outcomes Our graduates will be systems thinkers. Ability to apply mathematics, science, and engineering Ability to design and conduct experiments, as well as to analyze and interpret data Ability to identify, formulate, and solve engineering problems Ability to use the techniques, skills, and modern engineering tools necessary for engineering practice Our graduates will be global collaborators. Ability to function on multi-disciplinary teams Understanding of professional and ethical responsibility Ability to communicate effectively Knowledge of contemporary issues Our graduates will be innovative designers and problem solvers. Ability to design a system, component, or process to meet desired needs within realistic constraints Broad education necessary to understand the impact of engineering solutions in a global and societal context Recognition of the need for, and an ability to engage in life-long learning

MIME Program Educational Objectives (PEOs) Student Outcomes Course Learning Outcomes

Student Outcomes Student outcomes describe what students are expected to know and be able to do by the time of graduation. These relate to the skills, knowledge, and behaviors that students acquire as they progress through the program

Student Outcomes A&E Process General and Program-specific student outcomes defined e.g. Assessed and evaluated using exit surveys completed by students in the quarter they graduate and through Course Learning Outcome mapping Responsibility for A&E and CI process is ABET coordinator and UPC

Student Outcomes Examples General outcomes for all programs Ability to design and conduct experiments, as well as to analyze and interpret data Ability to design a system, component, or process to meet desired needs Ability to function on multi-disciplinary teams Program-specific outcomes Ability to design, develop, implement and improve integrated systems that include people, materials, information, equipment, and people Ability to work professionally in mechanical systems area including the design & realization of such systems

Student Outcome Assessment via Exit Survey Assessment of student outcomes is completed using an annual Exit Survey that is administered to graduates in the term that they graduate, i.e. administered all 4 quarters. Items cover Student Outcomes that are rated on a 7-point scale (1=not at all; 7=extremely). Survey provides a direct assessment of program specific students outcomes in addition to a-k. Other demographic details are collected related to coop experiences, number of interviews, employment status, and general satisfaction with their educational experiences (labs, faculty, advising, facilities, classes, etc.)

Student Outcome Evaluation Process Student Outcome data analysis and recommendations are prepared by ABET coordinator. Analysis and recommendations reviewed with UPCs on an annual basis and actions for improvement identified.

MIME Program Educational Objectives (PEOs) Student Outcomes Course Learning Outcomes

Course-level Assessment and Evaluation CLOs

Course Learning Outcomes Course learning outcomes are statements that describe what students are expected to know and/or be able to do at the completion of a specific course.

CLOs 3 7 LOs typical for a course Examples of LOs Be able to identify and describe (in writing) the most significant challenges faced by engineering managers in organizations in today's global environment. Be able to identify and facilitate solutions to ethical dilemmas faced by engineering managers

Student Outcome Assessment via Course-Level Assessment and Evaluation Faculty must also identify two other direct measures to evaluate the achievement of Course Learning Outcomes and ultimately of the linked Program Outcome. Specific assignments, exams, project reports, and grading rubrics are used by faculty for direct measures as appropriate to a given course. Faculty members are responsible for collecting the necessary data for assessment and for completing appropriate analyses of the data to determine whether or not Program Outcomes have been achieved. Faculty must also provide a narrative summary of their evaluation. This process is completed annually for all required courses (or every other year for courses taught on an every other year basis). Faculty members are responsible for reporting any significant issues in meeting Program Outcomes to the UPC. Evidence of this process is provided through the course notebooks.

CLO Assessment Plan Example

Evaluation of Learning Outcomes

CLO Evaluation Example Expectation is for at least three distinct measures for each learning outcome, one of these measures can be student evaluation of CLO achievement for SET

Continuous Improvement Process

Continuous Improvement Cycle for all A&E Processes PlanDoStudyAct Review & Update Desired Outcomes A&E Performance against Outcomes Measure Impact of Changes Implement Changes to Address Gaps

PEO CI Process Examples DoAct

PEO CI Process Examples Plan

SO CI Process Study Plan Do Study Act

CI Process: Plan In response to the ABET reviewers concerns on thermal design and the persistent low achievement rating of Outcome c, the ME ABET coordinator as well as the UPC began discussions on possibility of enhancing the design experience for students in Fall 2009. A recommendation from the coordinator and the UPC on this matter to the ME faculty is tentatively scheduled for Fall 2010. Plan

CI Process: Do Considering the low achievement rating on Outcome c and d, ME 382 was revised in its Fall 2009 offering to include a greater emphasis on designing real world products and forming diverse teams (using personality and skills as metrics). This change in project focus, while retaining the same design methodology topics, is expected to increase student awareness of the relevance of design to the mechanical engineering profession through is concrete experience with real mechanical systems (covering all domains of mechanical systems). Any potential impacts from these revisions will be first captured in the 2011 exit surveys (since ME382 is a junior- level course). Since Fall 2009, ME382 has been made a required course for MfgE and an elective for IE students. This ensures that the teams are more multi-disciplinary. Any potential impacts from these revisions will be first captured in the 2011 exit surveys. Do

CI Process: Study and Act StudyAct

CLO CI Process PlanDoStudyAct

program outcome assessment: case study from three engineering programs presented by toni l. doolen,...

Documents

engineering programs

program outcome assessment

accredited engineering

program constituencies

program outcome data

assessment data

international slide

programs constituencies