بسم الله الرحمن الرحيم

Outcome-BasedProgram Assessment

An Overview

Dr. Mohammad S. Al-HomoudDirector

Program Assessment CenterDeanship of Academic Development

King Fahd University of Petroleum & MineralsDhahran 31261

assess@kfupm.edu.sa

What is Assessment?

Assessment is a systematic ongoing, iterative process of monitoring (recording and analyzing) student learning outcomes and processes in order to determine what we are doing well and what we must improve.

“Processes that identify, collect, use and prepare data for evaluation of the achievements of program outcomes, educational objectives, or for program improvement”

(ABET)

What is the aim of program assessment?

The aim of program assessment is to understand how educational programs are working and to determine whether they are contributing to student growth and development. It focuses on programs rather than on individual students. It provides information on whether the curriculum as a whole provides students with the knowledge, skills and values that graduates should possess in accordance with its mission and set goals and learning objectives.

What is Evaluation?

“Processes for the interpretation of the data and evidence obtained from assessment practices that a) determine the extent to which program outcomes or educational objectives are being achieved; or b) result in decisions and actions to improve the program.”

(ABET)

Evaluation is an event with common checkpoints by an evaluator(s) that involves determining the quality of past efforts and provides a reward or punishment (accountability). It constitutes any effort to use assessment evidence or findings to apply judgments that result in a grade, score or value.

Assessment or Evaluation ?

Assessment is meant for quality assurance and involves:

• Analyzing and interpreting information

• Activities aimed at improvement

• Strong connection between assessment results and instructional plans.

Assessment or Evaluation ?

Evaluation is meant for accountability and involves:• Applying judgments to assessment efforts• Activities aimed at demonstration of accountability• Little connection with instructional plans

Measurement involves: • Gathering and quantifying information

What is Accreditation?

A procedure by which an authoritative body gives formal recognition that a body (organization) or a person is competent to carry out specific tasks.

(ISO/IEC guide 2:96)

Traditional emphasis• What we give

students?• Inputs• Faculty teaching• Curriculum• Educational activities

as an end• More quantitative (how

much)• Prescriptive based

New emphasis• What they can/are able to

do?• Outcomes• Student learning• Education• Educational activities as

means to an end• More qualitative (how well)

• Outcome-based

New Philosophy

Process Output(Anticipated Outcomes)

Input

Standards

Traditional Educational Process(Prescriptive, teaching focused)

Traditional Educational Process

Input Process Output(Actual

outcomes)

Measurement/ Comparison

Intended Outcomes

Feedback

Outcome-Based Assessment(Outcome-based, learning focused, continuous

feedback/improvement)

Outcome-Based Assessment

Characteristics of Continuous Quality Improvement:•Systematic• Dynamic • Continuous• Motivating• Collaborative • Integrative not in isolation• Focus on education not on curriculum• Satisfy the needs of constituents

Continuous Quality Improvement

Assessment Key Questions

Program assessment system should answer the following questions:

• Who are our constituents/suppliers?• What are we trying to do (objectives/

outcomes)?• Do our constituents/suppliers understand

our objectives?• How well we are achieving what we are

doing?• What is the process of doing it?

Assessment Key Questions

• What qualities and level of qualities that we want?

• How are we going to measure these qualities?• How do we utilize the results to continuously

improve?• Are we achieving our objectives?• Are we satisfying the needs of our

constituents?• Does it work?

Push/need for program

assessment

Employers/ industry

Accrediting

Agencies Government authorities

Administrators

Faculty

Students

Assessment Driving Forces

Assessment Driving Forces

Assessment Benefits

• Making the institution more self-conscious about what its programs are accomplishing

• Providing information for recruitment • Providing information for planning • Providing information for accreditation• Improvements in teaching and learning• Improving institutional image• Enhancing funding opportunities• Attracting better students

Approaches to Assessment

What to assess (educational outcomes)?• Cognitive (knowledge-based) :

what should students know?

• Affective (attitudes and values) : what should students value?

• Psychomotor (skills) : what should students do or be able to do?

Approaches to Assessment

Whom to assess (objects)?• Students • Faculty • Course• Program

Why assess (purpose)?• Formative: on going (e.g. to ascertain progress or

development)

• Summative: at the end (e.g. to ascertain mastery level of achievement)

Assessment Steps

• Recognition/commitment• Define program constituents • Define program /institution mission statements• Define educational objectives that meet

program / institution constituents• Define program outcomes• Select appropriate assessment tools• Collect necessary data• Emphasis on outcomes (graduates

preparation for professional practice)

Assessment Steps

• Demonstrate how criteria and program educational objectives/outcomes are being met

• Seek external review• Implement actions for improvement• Refine and repeat the process

Goals

Assessment Steps

Mission

Actions for improvement

Objectives

Outcomes

Assessment

toolsAssessment

activities

Goals and Objectives

• Goals are used to express intended results in general terms (broad learning concepts such as clear communication, problem solving, ethical awareness, etc.)

• Objectives are used to express intended results in precise terms (specific behaviors students should exhibit). Objectives tell us more specifically what needs to be assessed, and thus a more accurate guide to suitable assessment tools.

Goals and Objectives

• Both terms are used interchangeably

• Both terms describe the intended results of educational activities

• Both terms provide direction for assessment

Program Educational Objectives• Statements that describe the expected

accomplishments of graduates during their first few years of career after graduation as a result of their educational preparation.

Types of program educational objectives:• Objectives that all graduates are expected to

accomplish;• Objectives that some graduates are expected

to accomplish.

Goals and Objectives

Audiences for objective statements:Constituents external to the institution including:• Prospective students• Parents• Student sponsors, donors, and contributors• Employers• Transfer institutions, graduate programs• Industry advisors• Accrediting agencies• Government organizations

Goals and Objective

Features of Good Statements of Program Educational Objectives

• Relevant to the mission of the institution;• Each addresses one or more needs of one or

more of the program constituents;• Understandable by the program constituents;• Clear, concise, and consistent;• Realistic/achievable;• Measurable;• Forward looking (not short term);• Manageable list (limited number of statements);• Focus on student learning.

Goals and Objectives

Program Outcomes

• Statements that describe what students are expected to know and be able to do by the time of graduation. These relate to the knowledge, skills and behaviors that students acquire in the program.

• Intended outcomes (from described statements of expected learning outcomes)

Program Outcomes

• Actual outcomes (form the results of assessment activities)

• Achievements of all program outcomes should indicate that the graduate is prepared to achieve the program educational objectives and should be verified before certification for graduation.

Program Outcomes:

Features of Good Statements of Intended Outcomes:

• Relative to the missions of the program and institution;

• Describe the expected learning outcomes for students from individual courses/the program as a whole;

• Reflect what the students are expected to know and be able to do rather than what the course/program intends to cover;

• Each describes an area of knowledge and/or skill that a person is expected to possess;

• Supportive of one or more educational objectives;

• Describe an observable behavior using action verbs (i.e. apply, identify, solve, construct, etc.);

• Avoid vague words that might be open to interpretation (i.e. understand, appreciate, etc.);

• Assessable (outcomes can be demonstrated in terms of performance).

Program Outcomes:

Assessment Methods

I. Direct Assessment Methods:• Directly determine whether students have

mastered the content of their academic programs.

• Require students to display their knowledge and skills as they respond to the instrument itself (i.e. objective tests, essays, presentation, and classroom assignments).

Assessment Methods

• Direct assessment methods include:Standardized examsLocally developed examsOral examsPortfolios (work collected over time)Performance appraisal

Assessment Methods

• Direct assessment methods include (Cont.):

Oral presentationsProjects, demonstrations, case studies,

simulationsCapstone experience (embodied in capstone

courses)Juried activitiesEvaluation of field work.Behavioral observations

Assessment Methods

II. Indirect Assessment Methods:• Ask students to reflect on their learning, what

they have learned and experienced, rather than to demonstrate it (i.e. surveys and interviews).

• Details about instructional or curricular strengths that can not be provided by direct methods alone.

Assessment Methods

• Indirect assessment methods include:Written surveys and questionnaires

o Entering studentso Current studentso Graduating seniorso Faculty o Alumnio Employerso Parents

Exit interviewsFocus groups External examinerArchival records

Pre-College Experience

Curriculum:MajorCoursesGE

Educational Outcomes

Educational Objectives

Surrounding Environment:Campus lifeSocial life

Classroom Experience:PedagogyFacultyFacilities

Out-of-Class Experiences:Co-opInternship

College Educational Experience Post-College Experience

Pre-College Educationa

l Experience

“Primary” Educational Outcomes Assessment Zone

Educational Objectives

Assessment Zone

Program Outcomes Influencing Factors

Assessment Key Success Factors

Assessment Key Success Factors Include:• Commitment • Participation (involvement) of all constituents• Listening to voices• Actions for improvement• Feedback• Relating assessment to the world of work• Continuity and flexibility• Consistency of reporting format over time• History of results• Responsibility• Resources• Rewards

THANK YOU