assessment, evaluation & continuous improvement the experience of king abdulaziz university

Assessment, Evaluation & Continuous Improvement

The Experience of King Abdulaziz University

Jubail Industrial College - Oct. 23, 2011

Ali M. Al-BahiProfessor of Aeronautical Engineering

Director of the Academic Accreditation Unit

األنبياء أشرف على والسالم والصالةوالمرسلين

وصحبه آله وعلى محمد سيدناأجمعين

JIC Presentation

Who are we? ABET TAC Accreditation Assessment and Evaluation of Program

Educational Objectives Assessment and Evaluation of Student

Outcomes Conclusions

Contents

Oct. 2011 3

Who Are We?

4Oct. 2011 JIC Presentation

King Abdulaziz University Profile KAU is Saudi Government financed institution. Founded in 1967 in Jeddah, 70 Km west of Makah holy city 18 Faculties/Colleges & 9 Deanships 110+ Departments offering a diversity of specializations. 4100+ Faculty members & 2100+ Administrative and support

personnel 95000+ Students. 30 Scientific Chairs 6 Centers of Excellence 16 Scientific Research Centers

5JIC PresentationOct. 2011

College of Engineering Profile Founded in 1975 G 2700+ Students 7000+ Alumni to date 9 Departments offering 12 programs. 240+ Faculty members 100+ Administrative and support personnel


ABET Substantial Equivalency The 12 Programs started introducing themselves to the world

of accreditation and quality education in 2002 through ABET SUBSTANTIAL EQUIVALENCY.

Although this was not A FULL ACCREDITATION, it represented a huge step forward.

This was the only available international evaluation at that time and allowed some deviations from in-land accreditation:

“Program is comparable in educational outcomes but may differ in format or method of delivery”

Nevertheless the wheel of quality started to rotate


ABET Full AccreditationIn 2003, once the successful ABET visit to evaluate the 12 engineering programs was completed, The College started to prepare for the next evaluation visit in 2008 which coincided with:

•the introduction of the new outcome-based Engineering Criteria EC 2000, •the phase-out of the substantial equivalency in favor of full accreditation.

EC 2000 represented a major paradigm shift influenced by pressures from industry and global competition. It addressed the effectiveness of engineering programs by focusing on assessment & evaluation process that assures the achievement of educational objectives & outcomes.

In August 2008 the 12 KAU Engineering Programs obtained the full ABET Accreditation under EC 2000


Academic Accreditation Unit (AAU) Established in May 2005 4 faculty members (working

part time) Initial mission: Qualify the

education system in the Faculty of Engineering to Meet ABET EC 2000 Standards.

9Oct. 2011

AAU: Quality Assurance In KAU Engineering Education

JIC Presentation

ABET TAC Accreditation


ABET The Accreditation Board of Engineering and Technology ,

actually known as ABET Inc. The recognized accreditation entity for college and university

programs in applied science, computing, engineering, and technology in the US.

A federation of 30 professional and technical societies in these fields.

For over 77 years, ABET has provided leadership and quality assurance in higher education.

Currently ABET accredits some 2,800 programs in more than 600 colleges and universities inside the US and worldwide.


Elements of ABET Criteria1. Definitions2. General Criteria: apply to all programs accredited by

an ABET commission. Each program accredited by an ABET commission must satisfy every Criterion in the General Criteria for that commission.

3. Program Criteria: provide discipline-specific accreditation criteria. Programs must show that they satisfy all of the specific Program Criteria implied by the program title. Any overlapping requirements need be satisfied only once.


ABET-TAC Definitions

Program Educational Objectives: Broad statements that describe what graduates are

expected to attain within a few years of graduation. Program educational objectives are based on the needs of the program’s constituencies.

Student Outcomes: Describe what students are expected to know and be

able to do by the time of graduation. These relate to the knowledge, skills, and behaviors that students acquire as they progress through the program.


ABET-TAC Definitions (Continued) Assessment : one or more processes that identify, collect, and

prepare data to evaluate the attainment of student outcomes and program educational objectives. Effective assessment uses relevant direct, indirect, quantitative and qualitative measures as appropriate to the outcome or objective being measured. Appropriate sampling methods may be used as part of an assessment process.

Evaluation: one or more processes for interpreting the data and evidence accumulated through assessment processes. Evaluation determines the extent to which student outcomes and program educational objectives are being attained. Evaluation results in decisions and actions regarding program improvement


ABET-TAC Continuous Improvement

Criterion 4

The program must regularly use appropriate, documented processes for assessing and evaluating the extent to which both the program educational objectives and the student outcomes are being attained. The results of these evaluations must be systematically utilized as input for the continuous improvement of the program. Other available information may also be used to assist in the continuous improvement of the program.


Assessment & Evaluation of Program Educational Objectives (PEOs)


Establishment of PEOs• Broad statements that describe what graduates are expected

to attain within a few years of graduation. • They should be consistent with the mission of the institution,

the needs of the program’s various constituencies, and ABET criteria.

• They should be documented (website, publications, posters, etc.)

• They should be established with the full involvement of all program constituencies.

• Program constituencies include Alumni, Employers, Students/ parents, faculty, institution management.


18Oct. 2011

ABET Committee

defines/redefines PEOs

DraftsO.K.?O.K.?

Dept. Council

Approval

Publish PEOs

Other Similar Programs Worldwide

Program Mission

ABET Criteria

Final DraftStart

Input from all Constituencies

Assess PEOs Adequacy

NoNo

O.K.?

No Assess PEOs Achievement

O.K.?Modify Outcomes and/or Curriculum

Assess Outcomes

Define/redefine Outcomes

Yes

No

3-YearCycle

Yes


Departmental Council

Yes

Outcomes Achievement

Cycle

PEOs Adequacy Cycle

Outcomes Adequacy

Cycle3-YearCycle

AnnualCycle

Input from off campus Constituencies and/or

alumni data base

Establishment & Achievement of Program Educational Objectives

Establishment of PEOs -Continued

JIC Presentation

Assessment of PEOs

19

Effective assessment uses relevant direct, indirect, quantitative and qualitative measures as appropriate to the objective being measured.• Assessment of PEOs should be carried outside the institution -

since you need to evaluate the achievement of the graduates.

• You should target your graduates (alumni) and their employers (and not the students or the faculty).

• You should assess and evaluate:1. Adequacy of PEOs to the needs of your constituencies.2. Achievement of PEOs by your graduates.

Oct. 2011 JIC Presentation

Modifying PEOsPublish PEOs

Assess PEOs Adequacy

O.K.?

No Assess PEOs Achievement

O.K.?Modify Outcomes and/or Curriculum

Assess Outcomes

Define/redefine Outcomes

Yes

No

3-YearCycle

Yes



Cycle

PEOs Adequacy Cycle

Outcomes Adequacy

Cycle3-YearCycle

AnnualCycle

Input from off campus Constituencies and/or

alumni data base

Assessment & Evaluation of Program Educational Objectives

JIC PresentationOct. 2011 20

Adequacy and Achievement of PEOs

21

PEOs Adequacy cycle (3 year cycle, twice during an ABET accreditation cycle). It may result in redefinition of PEOs.

PEOs achievement cycle (normally carried in the same time as the adequacy cycle). It may result in modifying the student outcomes or enhancing of their implementations.


Direct Assessment of PEOs

22Oct. 2011

Facts Finding Questionnaire (Versus opinion-based surveys) Employer survey (every four years) Alumni survey (every three years)

Archival Data Placement records (mean time to find a job) Admissions to and success in graduate programs

(if applicable)

Inputs from the Industrial Advisory Board

JIC Presentation

Direct Assessment of PEOs - Continued

Define for each Program Education Objective a set of:

1. Indicators.2. Assessment tools.3. Performance criteria for each assessment

tool.4. Time line for assessment and evaluation.


Assessment and Achievement of Student Outcomes (SOs)


25

Establishing Student Outcomes

Oct. 2011

• ABET is an outcome-based approach.• Your Student Outcomes should be documented.• They should foster the attainment of your PEOs.• There should be a documented process to review

and revise your SOs.• The students should demonstrate achieving SOs

before graduation.JIC Presentation

26Oct. 2011

Assess Outcomes Through Courses & Constituencies

Define Assessment Tools

SOs Achieved?

Improve Courses/ Curriculum

No

AnnualCycle

Yes

Implement Outcomes into

Courses & Curriculum

3-YearCycle

Outcomes Adequacy

Cycle


Cycle

3 cycles Completed?

Yes

No

Results from Previous PEOs

3-Year Assessment Cycle

Input from Constituencies

Define/modify Outcomes

Define Outcome

Elements & Attributes

New or Modified PEOs, if Any

JIC Presentation

Outcome Elements and Attributes

Development of each outcome intoOutcome Elements: different abilities specified in a single outcome that would generally require different assessment measures.

For each outcome element define outcome attributes or indicators: measurable actions that explicitly demonstrate mastery of the abilities specified in an outcome element, at each level of learning.


IntendedLearning

Outcomes ofthe Lesson

IntendedLearning

Outcomes ofthe Unit

IntendedLearning

Outcomes ofthe Course

Deliver Forward

Design Backward

Infusion of Student Outcomes into Courses

StudentOutcomes ofthe Program


Assessment of Student Outcomes • Direct Measures

– Course assessment (every semester)– Internship evaluations.

• Indirect Measures– Exit interviews – Focus groups (every year)– Exit survey (every semester)

• Yearly evaluations


Assessment and Grading• Both intend to identify what the students have learned. • Grades don’t identify which specific student learning outcomes

and at what levels students have learned. • Some course grades include additional student behaviors that

are not related to student learning outcomes .• For example, a grade of B in a mathematics course indicates a

student learned an acceptable level of math, but not specific math knowledge or skills or what areas need improvement.

• Grades need to be clearly linked and aligned to learning outcomes and/or rubrics to be sufficient as direct evidence for assessment purposes.


Key Courses Approach

31

• Assessment of SOs is the task of the program.• The program is required to collect SOs assessment data supported

by convincing evidences to demonstrate that the students achieved all SOs.

• Each course is responsible to introduce some of these outcomes, enhance the students abilities to achieve them, and collect convincing evidences transferable to the program committee responsible for evaluation and continuous improvement actions.

• It is sufficient that each course will do this for only a limited number (2 or 3) of the SOs (Known as critical outcomes, the course is a key course for these SOs) if and only if compulsory courses together cover all outcomes with reasonable redundancy.

JIC PresentationOct. 2011

Course-level assessment of key outcomes


What is a rubric A rubric is a descriptive scoring guide composed of

• Things you are looking for in an assignment• Guidelines for evaluating each of those things.

A rubric is some kind of scorecard that breaks down a written or demonstrated assignment into manageable, observable pieces.

A rubric is a descriptive scoring/grading tool that is generally used for subjective assignments.


Workshop Assessment Rubric

34

LevelCriteria

4 3 2 1

Intended Outcomes

Intended outcomes were manageable, clearly evident throughout and strongly supported by learning activities.

Intended outcomes were clearly stated, evident in lecture and supported by learning activities.

Intended outcomes were stated, but lecture and learning activities were not clearly organized to achieve them.

Intended outcomes were not stated and unclear. Lecture was unfocussed, and any learning activities did not seem to directly support any particular outcome.

Expertise

Speaker’s expertise in content area was remarkable.

Speaker’s expertise in content area was evident.

Speaker’s expertise in content area was unclear or minimal.

Speaker’s expertise in content was not evident.

Presentation

Presentation was well-organized and easy to follow. It was interesting.

Presentation was clear, and easy to follow, using a standard format.

Presentation followed a main idea, but was repetitious or tangential to no purpose.

Presentation was disorganized and therefore difficult to follow.

Logistics

Media and learning materials were used effectively to enhance the presentation.

Media and learning materials supported the presentation appropriately.

Some media and learning materials did not contribute to the presentation.

Media and learning materials were missing, distracted from the presentation or poorly used.

Learning Engagement

I was meaningfully engaged in learning, making connections to my current professional practice and seeking new learning for continuous improvement.

I was actively engaged in learning and sought to find aspects to contribute to my professional practice.

I chose to be involved, and followed instructions to some degree.

My involvement in learning was passive. I chose not to be engaged in the presentation.


35

Conclusions To get accredited we were required to demonstrate

to ABET evaluators that:1. WE KNOW WHAT WE DO.2. WE KNOW WHY WE DO IT.3. WE KNOW HOW TO DO IT.4. WE DO IT WELL.5. WE CAN PROVE IT.6. WE RECEIVE INPUT AND FEEDBCAK.7. WE HAVE A PROCESS TO MAKE CONTINUOUS IMPROVEMENTS.

To get accredited you are required to have:1. WELL LAID-OUT PROCESS, and 2. INVOLVEMENT OF THE ENTIRE FACULTY IN THE PROCESS.

JIC PresentationOct. 2011

رب لله الحمد أن دعوانا رب وآخر لله الحمد أن دعوانا وآخرالعالمينالعالمين

assessment, evaluation & continuous improvement the experience of king abdulaziz university

Documents

kau engineering programs

faculty of engineering

land accreditation

world of accreditation

jic presentationoct

case studywho

university programs

evaluation visit