Assessment of Student

Learning Outcomes

Handbook December 2014

Written by Lorri Stiles and Kaylie Becker under the direction of Dr. Caryl Sheffield

1

TABLE OF CONTENTS

WHAT IS ASSESSMENT OF STUDENT LEARNING OUTCOMES (ASLO)? ................................ 2

COMPLETING THE ASLO YEAR-END REPORT .............................................................................. 3

USE OF RESULTS FROM PREVIOUS YEAR .................................................................................................................... 3 Example of Use of Results from Previous Year ................................................................................................................... 3

I. DEPARTMENTAL PHILOSOPHY OR MISSION ........................................................................................................... 5 Example of Mission Statement ........................................................................................................................................... 5

II. EDUCATIONAL GOALS ............................................................................................................................................ 6 Example of Educational Goals ............................................................................................................................................ 7

III. LIST OF EDUCATIONAL OBJECTIVES ...................................................................................................................... 8 Example of Educational Objectives .................................................................................................................................... 9

IV. ASSESSMENT RECORD ........................................................................................................................................ 10

Means of Assessment (A-1-a, A-2-a; B-1-a, B-2-a) .............................................................................................. 10 Possible Means of Assessment: ........................................................................................................................................ 11

Criteria for Success (A-1-b, A-2-b; B-1-b, B-2-b) .................................................................................................. 12 Examples of Means of Assessment and Criteria for Success ............................................................................................ 13

Assessment Results (A-1-c, A-2-c; B-1-c, B-2-c) ................................................................................................... 14 Data Collection ................................................................................................................................................................. 14 Data Analysis .................................................................................................................................................................... 14 Results .............................................................................................................................................................................. 15 Example of Assessment Results ....................................................................................................................................... 16

Use of Results (A-1-d, A-2d; B-1-d, B-2-d) ........................................................................................................... 16 Example of Use of Results ................................................................................................................................................ 18

APPENDICES ............................................................................................................................................................ 19

Appendix A .......................................................................................................................................................... 19

Appendix B .......................................................................................................................................................... 19 Example of Appendix B ..................................................................................................................................................... 19

FORMATTING REPORT ............................................................................................................................................. 20 Example for Formatting the Report.................................................................................................................................. 20

APPENDIX ................................................................................................................................................ 21

APPENDIX I .............................................................................................................................................................. 22

APPENDIX II ............................................................................................................................................................. 23

APPENDIX III ............................................................................................................................................................ 26

APPENDIX IV ............................................................................................................................................................ 31

2

“The most important purpose of assessment should be not improvement or accountability

but their common aim: everyone wants students to get the best possible education.

Everyone wants them to learn what’s most important. A college’s mission statement and

goals are essentially promises that the college is making to its students, their families,

employers, and society. Today’s world needs people with the attributes we promise. We

need skilled writers, thinkers, problem-solvers and leaders. We need people who are

prepared to act ethically, to help those in need, and to participate meaningfully in an

increasingly diverse and global society. Imagine what the world would be like if every

one of our graduates achieved the goals we promise them! We need people with those

traits, and we need them now. Assessment is simply a vital tool to help us make sure we

fulfill the crucial promises we make to our students and society.” –Linda Suskie

https://www.insidehighered.com/views/2010/10/26/suskie

WHAT IS ASSESSMENT OF STUDENT LEARNING OUTCOMES (ASLO)?

Throughout the year, California University of Pennsylvania faculty are engaged in a

standardized assessment process to determine if students in their programs are actually learning

what faculty say the students should be learning. This practice is called Assessment of Student

Learning Outcomes (ASLO) and is part of the bigger process of assessment of institutional

effectiveness and Middle States Commission on Higher Education accreditation. Additionally,

ASLO is aligned with other academic reporting, such as the university strategic plan, department

annual reports, program accreditation, and five-year program reviews.

Cal U has been conducting university-wide assessment of student learning outcomes

since 1990. A university-wide ASLO committee, consisting of faculty from each college and an

associate provost, coordinate the ASLO process. They develop timetables, forms, the report

evaluation rubric, and other useful materials. The committee also provides individualized

assistance for program facilitators and evaluates the year-end reports. See Appendix 1 for the list

of committee members.

3

“Colleges and universities increasingly are emphasizing the assessment of student

learning for two primary reasons: Improvement. Assessment can help improve the

quality of teaching, learning, programs and services, and planning and decision making.

Accountability. Assessment can validate current programs, services, and teaching and

learning efforts and thereby secure continued support for them by demonstrating their

effectiveness to concerned audiences. In other words, assessment can demonstrate

accountability to these audiences.”

Suskie, L. (2009). Assessing student learning: A common sense guide (2nd

ed). San

Francisco, CA: Jossey-Bass.

COMPLETING THE ASLO YEAR-END REPORT

This part of the handbook provides guidance for completing each section of the ASLO

year-end report. It might be helpful to have a copy of the year-end report template as you read

each section.

USE OF RESULTS FROM PREVIOUS YEAR

In this section of the year-end report, program faculty report how they used the results

from the previous year’s assessment to make changes to the program. Use the expandable

textbox provided to address concerns that emerged in the data from the previous year’s

assessment of student learning and to note any adjustments that were made. For example,

program adjustments might include a plan for curriculum revision, changing course objectives,

or revising assessments.

If the program is new or if this is the first time that a year-end report is written, there

would be nothing to report in this section. Use the expandable textbox to make a brief statement

documenting the reason there was nothing to report.

Example of Use of Results from Previous Year

Begin with the “use of results” from the previous year. Highlight the changes which were

identified as needed.

4

Then, indicate if these changes have been made in the program and if they have been

effective. State how the changes are designed to increase the meeting of criteria of success

or to increase the overall success of the program.

Use of Results from Previous Year

As stated in the previous year’s outcomes report, additional opportunities to write behavioral

objectives have been added to the program starting in lower level courses. Opportunities to

write behavioral objectives for a variety of clinical populations are now offered in lower and

upper level course to broaden the scope of expertise of the students graduating from the

program. In addition, students in lower level courses are exposed to correctly written,

effective behavioral objectives to increase their ability to identify changes that need to be

made to correct them.

These changes are designed to offer more support to students in the task of writing

behavioral objectives for treatment plans through offering them more opportunities to

experience and create ones that are well written.

These changes appear to be increasing the students’ ability to write effective behavioral

objectives as shown by the increase in scores students are receiving on assignments in CMD

450. In addition, a larger number of students are receiving higher grades in the class than

prior to the change in curriculum.

Identify changes

that have been

made in program.

How changes

support objective

Effect of

changes

5

I. Brief Statement of Departmental Philosophy or Mission

Distinct way program

will impact careers

I. DEPARTMENTAL PHILOSOPHY OR MISSION

Every department/program offered at California University of Pennsylvania should have

a distinct and specific philosophy or mission statement that reflects the program’s purpose and

aligns with the mission statement of the university. The program philosophy/mission statement

should be a broad statement that is more theoretical than measurable. It should reflect the

ultimate goals of the program or department.

Cal U mission statement:

“To advance its ultimate mission of building the character and careers of students,

the University shall focus its efforts on three goals: student achievement and

success, institutional excellence, and community service. These interrelated ends

will be facilitated by the following means: high quality faculty, students,

programs and facilities. These means, in turn, will be funded through an energetic

program of resource acquisition and stewardship.”

The following steps may help program faculty develop a departmental philosophy or

mission statement:

1. Review California University of Pennsylvania’s Identity, Mission Statement and

Vision Statements at http://www.calu.edu/about-us/mission-statement-and-

vision/index.htm.

2. Describe how the program aligns with the university’s mission statement.

3. Describe distinct ways that the program will impact the character and careers of

the graduates.

Example of Mission Statement

Alignment with the

University’s Mission

How program will

impact character

The History and Political Science department promotes the university’s goal of increasing academic

excellence through rigorous academic standards relative to course content and theoretical or

methodological understanding. We believe that our graduates’ career prospects are enhanced through

developing the intellectual and communicative skills to conduct research and write effectively. We

embrace the university’s goal of fostering diversity, broadly defined, through our course content and

research methods. We support the university’s goal of civic engagement through service activities and

intend to instill character in our graduates by adhering to standards of academic integrity.

http://www.calu.edu/about-us/mission-statement-and-vision/index.htmhttp://www.calu.edu/about-us/mission-statement-and-vision/index.htm

6

II. EDUCATIONAL GOALS

Educational Goals describe the knowledge, skills, and values expected of graduates from

the program and answer the following questions:

What does the student know (knowledge)?

What can the student do (skills)?

What does the student care about (values/dispositions)?

Educational Goals relate directly to the discipline and are consistent with external

standards. Goals should be related directly to the underlying foundations and skills of the subject

matter pertinent to the program and department. The educational goals should also be consistent

with external accrediting bodies.

Educational goals should be comprehensive and realistic. Educational goals for a

department or program should encompass the major expectations of graduates from the

department and program, in broad terms. They are based on the honest expectations for the

graduates of the program.

Between three and seven educational goals are acceptable. The number of goals should

reflect the complexity (i.e., baccalaureate vs. master’s degree) of the program and the

requirements of the accrediting bodies.

7

Example of Educational Goals

II. Educational Goals

The ___English Department____ program aims to produce graduates who can:

G-1. Understand and articulate the connections between language and culture and the ways we and our

worldviews are shaped by language.

G-2. Demonstrate effective ways to employ written and oral language as tools of self-discovery and

communication.

G-3. Analyze, critique and evaluate texts, both literary and nonliterary.

G-4. Use methods for investigating, evaluating, synthesizing and documenting relevant information

sources, both on paper and online.

G-5. Apply ways to fashion a career using the skills gained in the major.

G-6. Specialize in at least one area of English studies and demonstrate knowledge and skill in that area.

Tied to specific

discipline or

program

Highlights the

discipline’s

core concepts

What students will know,

be able to do, and be after

graduation.

8

III. LIST OF EDUCATIONAL OBJECTIVES

Educational objectives, or intended student learning outcomes, are specific, measureable

statements describing what graduates of the program are expected to know and be able to do. In

order to recognize if planned learning was accomplished, student learning outcomes identify the

knowledge, values, and skills students must demonstrate upon finishing the program. Each

educational objective should be directly linked to the program content. The objectives should

guide the choosing of assessment tools or means.

Tips for writing educational objectives:

Unlike Educational Goals above, educational objectives should be written in measureable

terms. Bloom’s taxonomy can help define measurable terms that are appropriate for each

objective. Educational objectives should only contain one measureable verb per

objective. This allows for clear measurement of the objective.

Educational objectives should be obviously linked to educational goals. If the goal states

that a graduate should be able to effectively communicate, then the objective should give

an example of how the graduate will effectively communicate, (for example, writing in

APA format or presenting in front of a class.).

Finally, educational objectives should be short, easy to read, clear and concise.

To complete this section, list the program’s educational objectives. Six to twelve

objectives will suffice. Indicate to which goal each objective is linked. Because every objective

is measured in a 5-year cycle, list the specific years in which each objective will be measured,

rather than listing ‘every two years’, or ‘every year’.

9

Example of Educational Objectives

Objectives: Intended Student Learning Outcomes Planned Academic Year

(over 5 year period)

For Measurement*

Ob. – 1. The student will be able to explain the foundational

principles of legal theory and the nature of law.

(linked to G – 6 from 2012-13 Year End Report)

2013-2014

Ob. – 2. The student will be able to identify practices that generate

conflicts of interest and/or are unethical.

(linked to G – 7 from 2012-13 Year End Report)

2013-2014

Ob. – 3. The student will be able to compare the differences in

historical models of jurisprudence.

(linked to G – 3)

2014-2015

Ob. – 4. The student will be able to locate, through legal research,

and correctly apply the law to a fact pattern in order to resolve the

legal question.

(linked to G-1, G-3).

2014-2015

Objectives are

program specific

Only one

measureable verb

per objective

Year objective is

measured. All

objectives measured

once in five year

cycle. Current

objective highlighted

Use of Bloom’s

Taxonomy for

measureable

verbs Linked

to goals

10

IV. ASSESSMENT RECORD

The assessment record is a detailed account of the two objectives (A and B) that are

measured for the year. Each objective is assessed by 2 different measures (A-1 and A-2; B-1 and

B-2).

Means of Assessment (A-1-a, A-2-a; B-1-a, B-2-a)

Each objective is measured in two different ways or, stated differently, by two different

means of assessment. The means of assessment must clearly measure the indicated learning

objective. Assessments can be interviews, oral presentation (rated by a rubric) -the possibilities

are endless (see p.11 for more examples).

Assessments can be embedded in courses as assignments or tests, which is convenient for

gathering and analyzing data. Assessments can also be standardized tests, comprehensive exams

or licensure exams.

Assessments should provide quantitative data. This means that the assessments, when

completed, should provide faculty with a number value. This can include a percentage, number

correct, total points or number of students that chose a specific item. If a rubric is used, it will

need to have a number value associated with the scoring categories. Course grade point averages

(GPAs), letter grades and overall GPA should not be used as they are not an acceptable means of

assessing a program’s effectiveness.

While 2 means of assessment are required, one of them MUST be an objective

assessment. The second method of measuring the learning objective may either be objective or

subjective. An objective measure is one that is free from opinion. Examples of objective

measurements are standardized tests, portfolios, and course assignments. Subjective measures

are based on the opinions of others such as graduate surveys and employer surveys. If rubrics

11

are used to assess papers, portfolios, presentations or other assignments, there must be at least

two raters. The raters must be identified by at least their position within the university or

department. Inter-rater reliability (the correlation between the raters’ scoring) must be reported

in the means of assessment.

Include all assessment measures, i.e., copies of surveys, rubrics, assignment instructions

and/or information about standardized tests. Including the assessments allows for the ASLO

committee to fully understand assessments that are outside of their field of expertise.

Possible Means of Assessment:

Objective Means of Assessment

Comprehensive examinations

Performance demonstration for graduating seniors

Written proficiency exams

National Major Field Achievement Tests

Graduate Record Exams

Certification examination, licensure examinations

Locally developed pre- and post- tests

Senior thesis / major project

Portfolio evaluation

Reflective journals

Capstone courses assessments

Internship evaluations

Grading with scoring rubrics

Graduate school acceptance rates

Subjective Means of Assessment

Surveys of graduating seniors

Exit interviews of graduating students

Alumni Surveys

Focus Group Discussions

Employer Surveys

Internship Supervisor Surveys

12

Criteria for Success (A-1-b, A-2-b; B-1-b, B-2-b)

Declare a criterion for success for each means of assessment. The criteria should state

how many of the students assessed will achieve a specific level on the assessment. Criteria

should be stated in concrete, quantitative terms that correspond to the means of measurement.

The level of achievement on the assessment as established by the criterion for success

should reflect the requirements of any accrediting bodies, the high standard of education that is

required by the university’s mission statement, and the competitive nature of the occupational

fields that graduates will be entering when they leave the university. The criterion for success or

level of achievement must be stated in numeric terms such as percentage correct, average, etc.

The number of students that need to attain the criterion of success or level of achievement

should reflect the university’s dedication to both excellence in education and to the students. It is

important for criteria for success to reflect expectations. A criterion should be an honest measure

of students successfully demonstrating that they have mastered the skills, knowledge, and

dispositions required from the program. Setting a criterion at a high level challenges a

department to ensure that the students are given every opportunity possible to gain the

knowledge, skills and dispositions needed to be successful.

The number of students who are to attain the level of achievement may be stated in a

percentage of all students taking the assessment, percentage of respondents, or a specific number

of students in programs that limit their enrollment. The number of students may also be stated in

a whole group average, stating that the scores of all students taking the assessment must average

a certain percentage/score or higher.

13

IV. ASSESSMENT RECORD

(2012-2013)

Examples of Means of Assessment and Criteria for Success

A-1. First measurement

Students will submit a map portfolio.

The Departmental Outcomes

Assessment Committee will evaluate

the portfolios comprised of map

projects and assignments based on the

rubric provided. (Please see Appendix

A). The inter-rater reliability is 0.76.

85% of all senior level students in the program will

score an 80% or better on the portfolio when graded

using the rubric included in Appendix A.

Reports a percentage

of students that need to

achieve the criteria for

success

Inter-rater reliability

and the raters are

identified

Score is above

the level of

excellence set by

the university

A-1-a Means of Assessment A-1-b Criteria for Success

Means of

assessment is

clearly tied to

objective

Exam is a requirement for

graduation, increases

number of students who

take exam

Exam is an

objective

measure, second

measure can be

subjective

Exam gives quantitative

data; number of answers

correct.

Information about the CPCE (Validity,

Reliability and example questions) will

be listed in the appendix

14

Assessment Results (A-1-c, A-2-c; B-1-c, B-2-c)

The discussion of assessment results should describe data collection and analysis.

Data Collection

Collecting data can be done in several ways. Using the Desire2Learn system, instructors

are able to collect data for assessments that are completed using the system. Instructors may also

input scores into the system for assessments that are completed in class, or as assignments. For

instructors or programs that use standardized testing such as the Master’s Comprehensive

Exams, Desire2Learn is still an option as again, scores can be entered into the system.

Alternatively, data can be collected in an excel spreadsheet format. Some departments have

specific tools that are designed to collect and store the data from assessments and assignments

completed by the students, such as LiveText.

Data should be collected from all students completing the assessment, and specified by

the criterion for success. This may mean sorting some students out who are not in the program,

or integrating the data from several classes in the program. For instance, if the assessment is

being given in a course where there may be students from other majors or programs, or students

of different ranks, it may be necessary to remove their data from summative analysis.

Data Analysis

In order to analyze the data, it is important to fully understand what the criterion for

success is looking for. If the criterion for success is “80% of the students will achieve a specific

score”, it is necessary to know the number of students that are taking the assessment (what 80%

of that number is), and if that specific number of students achieved that level.

In order to accurately analyze the data certain information needs to be known:

The number of students taking the assessment

15

The score, in numeric form, that each student received on the assessment

The level of achievement that is required for meeting the criterion of success

The number or percentage of students required to meet that level of achievement

The mean or average of all of the grades

The standard deviation for the grades

Using this data, the facilitator can report if the students that took the assessment have or have not

met the criterion for success. Help with the formulas and calculations to find this information is

available through the ASLO committee.

Results

When reporting the results, it is important to use as much data as possible rather than just

subsets of the data. This will increase the validity of the results being used in the report. To

describe the sample, it is important to report the sample size, the mean, and the standard

deviation. Raw data should not be included in the report.

The results should be reported in the same terms as the criterion for success. To do this,

the criterion for success should be repeated with either the number of students or the level of

achievement changed, to reflect the results. The results section must include a statement

regarding the achievement of the criterion for success. This is a simple sentence that states either

the criterion for success was not met or the criterion for success was met.

16

Example of Assessment Results

A-2-c. Assessment Results

Use of Results (A-1-d, A-2d; B-1-d, B-2-d)

Using assessment results to make program modifications and adjustments is what the

Middle States Commission on Higher Education refers to as “closing the loop.” This critical step

in the ASLO process helps program faculty make the necessary changes to improve student

learning outcomes.

For the overall assessment of this competency in Advanced Field Education, 94.7% (18

out of 19 students) achieved a rating of 4 or higher. The mean score for this assessment

was 4.21667 with a standard deviation of .3329.

The criterion for success was met.

Percentage of students that achieved

level of success is stated. Included

is the number of total students

taking the assessment (n value)

Simple statement of whether the criterion for success was met

Standard deviation

is reported

Mean score

is reported

Level of achievement restated

“Does assessment really improve student learning?... It is not the

assessment itself, but how the faculty, staff, and institutional leaders

use it that leads to improvements in student learning.”

Suskie, L. (2009). Assessing student learning: A common sense guide

(2nd

ed). San Francisco, CA: Jossey-Bass.

17

In order to complete this section, all program faculty should be involved in a discussion

of the assessment results. It would be beneficial to state when and how the faculty discussed the

results, such as “at the spring department meeting.” Discussing the results as a department

reinforces the idea that outcomes assessment is a departmental responsibility. It will also allow

for the generation of ideas to improve or maintain the results.

If results meet the criterion of success, faculty might consider a plan to maintain the

success and create a plan to address those students who did not meet the criterion. This plan can

also include ways in which the department will attempt to raise the criterion of success by either

increasing the number of students required to attain the achievement level or by raising the

achievement level itself, especially if a majority of the students are achieving it.

If results do not meet the criterion of success, a statement of how the department will

respond must be reported. This can include an outlined plan of curriculum changes, increased

emphasis on the learning objective, or more time given to practice the objective prior to the

assessment as examples. This plan will be reported on in the following year’s assessment of

student learning report.

http://www.lacitycollege.edu

18

Example of Use of Results

Includes discussion

of department

Discusses changes to be

made in the future to improve

or increase the success rate.

Briefly discusses the meaning of

the results and how they are

interpreted by the department

A-2-d. Use of Results:

The faculty and staff reviewed the data and results as part of the departmental meeting where

strengths, weaknesses and the need for changes were identified. The data will also be reviewed at

TEAM day.

The department is not surprised to see the students rating themselves slightly higher than the faculty

did. The data showed a discrepancy in how the students rated themselves in the area of classroom

management and partnerships, when compared to the faculty members. Students identified

assessment as the strongest area for the group which did not match the faculty’s viewpoint. The

department feels that the faculty needs to make students more aware of the strengths that are seen in

certain area and make expectations more clear. This will be accomplished through increased written

positive feedback for students in the program and clarification in areas such as rubrics and

classroom objectives.

APPENDIX A:

Appendix A should include a copy of all the assessments used in the current years report.

This can include copies of the surveys, rubrics, assessments and information given on

standardized tests that cannot be included in the report due to copyright issues.

Include labels that allow readers to know which assessment goes with which objective

being measured.

APPENDIX B:

Appendix B is set up exactly like the actual report, but without the results and use of results

sections. This means that the objective to be measured must be stated, the means of

assessment that will be used must be stated and included and the criteria for success must

be stated. Include the copies of the assessments at the end of the Appendix.

19

A-1-d. Use of Results:

APPENDICES

Appendix A Include in Appendix A a copy of all the assessments used in the current years report.

This can include copies of surveys, rubrics, assessment tools, and information given on

standardized tests that cannot be included in the report due to copyright issues. Include labels

that allow readers to know which assessment goes with which objective being measured.

Appendix B Appendix B is set up exactly like the actual report, but without the results and use of

results sections. Include the objective to be measured, the means of assessment that will be used,

and the criteria for success. Also, include the copies of the assessments your program will use in

the next academic year at the end of Appendix B.

Example of Appendix B

All graduating seniors during either their junior

or senior year, will produce either a research

poster or research paper on a topic of their

choice within their program tract and in

consultation with their advisor that

demonstrates good organizational skills, logical

thinking, and grammatically sound.

All graduating seniors who produced a research

poster will receive a mean score of not less than

30/40 points (or 75%) (based on the scores from

three faculty judges from other departments)

using the presentation evaluation poster form

created by the FPDC; all graduating seniors who

produced a research paper will receive a mean

score not less than 3/5 for each item on the oral

presentation evaluation form.

A-1-c. Assessment Results:

Describe the means of

assessment that will be

used- Include validity and

inter-rater reliability

information, identify how

the assessment will be

scored and by who

Clearly state the criteria

for success including who

will be assessed, the

number of students that

need to meet criteria and

the score required for the

criteria to be met.

Leave Assessment

Results and Use of

Results blank

State the objective that

will be measured in

the following year

Ob. – 1. Students will be able to demonstrate good writing and speaking skills

A-1-a Means of Assessment A-1-b Criteria for Success

20

FORMATTING REPORT

The report should be a paginated PDF document that includes the actual report and both

appendices. Reports should only include the information for one program. If a department has

several programs, submit a report for each program. While many of the missions, goals,

objectives and assessments for the individual programs may be the same or similar, each

program probably has a unique set of objectives and results from the assessment that make it

different from others in the department.

Document naming convention: name the PDF file as

CatalogueNameofProgram_year_DeparmentName_LastNameofFacilitator

Example for Formatting the Report

CED_2013_CounselorEducation_Smith

Catalogue Name

of Program- Three

letter initial

Reporting

Year

Department

Name

Last name of

report writer

21

Appendix I. Committee Members

II. Guidelines

III. Template

IV. Checklist

22

APPENDIX I

2014-2015 ASLO Committee

College of Education

and Human Services

Chris Harman

Health Sciences Department

harman@calu.edu

Bonnie Laing

Social Work Department

laing@calu.edu

College of Liberal Arts Justin Hackett

Psychology Department

hackett@calu.edu

Christina Fisanick

English Department

fisanick@calu.edu

Eberly College of

Science and Technology

Joan Clites

Nursing Department

clites@calu.edu

Gary DeLorenzo

Math, Computer Science, and

Computer Information Systems

Department

delorenzo@calu.edu

School of Graduate

Studies and Research

Grafton Eliason

Counselor Education

Department

eliason@calu.edu

Peter Heh

Special Education Department

heh@calu.edu

Academic Affairs Caryl Sheffield

Associate Provost

sheffield@calu.edu

23

APPENDIX II

Guidelines

Program-level Assessment of Student Learning and Outcomes (ASLO) Year-End Reports

(Last Updated: May, 2014)

Year-End Report Content – All Year-End Reports will be submitted on the “Year-end Report Form:

Undergraduate/Graduate Assessment of Student Learning Outcomes.” This report should clearly identify

the program’s mission statement, all program goals and objectives, and when the objectives will be

assessed within a five- year timeframe. The Five-year Plan includes:

Heading Information – Identify the academic year when data was collected, program facilitator

(the people/person who managed the assessment and outcomes process), department, and degree

program assessed.

Use of Results from Previous Year – List and describe programmatic adjustments made related

to the ASLO Committee’s comments (See previous year’s ASLO rating letter).

Mission Statement – Include a departmental statement of educational philosophy/Mission

Statement which is integrated with university’s mission of character and careers.

Goals (3 – 6) – Learning goals are broad statements derived from the educational philosophy

which are not necessarily measurable.

Measureable Learning Objectives (5 – 10) – List measurable learning objectives developed

from and directly aligned with the goals (e.g., Obj.1 – Goal I, etc.).

Objectives Measured – Identify two learning objectives that will be assessed each academic year

within the five-year time frame. All objectives must be measured at least once in the five-year

period. To satisfy other requirements, some programs may choose to measure several learning

objectives every year (e.g., accrediting associations, etc.). The Assessment Committee, however,

will only review and evaluate the two learning objectives identify for the given Year-End Report

(e.g., 2012-2013).

Means of Assessment – Two means of measuring each learning objective are required annually.

Objective - One method needs to be a direct and objective measurement (e.g.,

competency-based data demonstrating what students have learned).

Subjective - The other method is an indirect method of measurement. This method

provides data from the subject’s perspective (e.g., student satisfaction or post-graduate

surveys).

Data for each Year-End Report must be collected and analyzed prior to September 30th. Include

in Appendix A copies of all instruments (survey forms, rubrics, etc.) used to collect data.

Criteria for Success – Clearly state the criterion for determining success for each Means of

Assessment measuring two student learning objective. This criterion must be in quantified terms

24

(e.g., “75% of students measured will achieve an 80% or better”). Criterion for success stated in

qualitative terms is not permitted for this report (e.g., “Most students earn a letter grade of “B” on

the project”).

Assessment Results - For the year of the report, include the assessment data collected for each

learning objective. Assessment results using percentages or averages must also indicate the

number (N) of students on which the results were based.

Use of Results - It must be evident from the Year-End Report that assessment results were

analyzed and a departmental and/or program-level “use of results” plan was created. Based on the

assessment results, identify planned program-level strategies to either improve or maintain

student learning outcomes.

Plans for Next Year - Two objectives for the forthcoming academic year are to be submitted

with the Year-End Report. The objectives are taken from your five-year plan Also, Means of

Assessment and Criteria for Success must be included with these objectives.

Appendices – Include one copy of each instrument used to assess each student learning objective.

Appendix A – Instruments used for the current report.

Appendix B – Instruments your program will use in the next academic year.

If a standardized instrument is used annually, please include it once and make a notation in the

subsequent appendix.

Submission of Report – Year-end Reports are to be submitted electronically to D2L >Program-Level

Assessment of Student Learning by September 30 of each year (e.g., Year-End reports for academic year

2013-2014 are due September 30, 2014).

**All information (Year-end Report and Appendices) must be merged into one document in the

accepted file formats (.doc, .docx, or .pdf) and attached in the D2L Drop Box.

**The document must be labelled “Department call letters-program name-(full name)-last name of

report author- year” If the program is GO, please add these letters after the last name.

Example: CED-Adult Clinical Mental Health-Patrick-2014.docx

Assessment of Year-End Reports – All Year-End Reports are evaluated by members of the University-

Wide Committee for the Assessment of Student Learning and Outcomes (UW – ASLO). Results include

ratings for the Year-End Report and approval of the Plan for Next Year (means of measurement and

criteria for success for the identified objectives). Future modifications of the Plan will be expected as

deemed necessary.

Notification of Results – Results will be made available to the program facilitators, department chairs

and deans during the Fall semester.

Ratings Categories –Approved; Approved with Reservations; Needs Improvement; Not Submitted.

25

Resources – Additional materials and forms are available on D2L > Program-Level Assessment of

Student Learning> Content. If you need assistance or have questions, please contact your College

Coordinator for Assessment:

Education and Human Services (GRAD and UG): Chris Harman Harman@calu.edu or Bonnie Laing laing@calu.edu

Liberal Arts (UG): Christina Fisanick fisanick@calu.edu or Justin Hackett hackett@calu.edu Science & Technology (UG):

Joan Clites Clites@calu.edu Graduate School (GRAD and Global Online):

Grafton Eliason Eliason@cal.edu or Peter Heh heh@calu.edu

mailto:Harman@calu.edumailto:laing@calu.edumailto:fisanick@calu.edumailto:hackett@calu.edumailto:Clites@calu.edumailto:Eliason@cal.edumailto:heh@calu.edu

26

APPENDIX III

YEAR END REPORT FORM UNDERGRADUATE/GRADUATE ASSESSMENT OF STUDENT LEARNING OUTCOMES

Form Last Updated: January 2, 2014

Program

College Program Facilitator

Academic Year 2013 - 2014 Report Submitted by

Phone/email Date Submitted

Directions: Please complete one form for each of the degree programs within your department; boxes will expand to accommodate unlimited data. Follow the Year-End Report Instructions and Checklist to complete this form. If you have any questions, please contact your College Coordinator: Education and Human Services (GRAD and UG):

Chris Harman Harman@calu.edu or Bonnie Laing laing@calu.edu Liberal Arts (UG): Christina Fisanick fisanick@calu.edu or Justin Hackett hackett@calu.edu Science & Technology (UG):

Joan Clites Clites@calu.edu Graduate School (GRAD and Global Online):

Grafton Eliason Eliason@cal.edu or Peter Heh heh@calu.edu Please submit this report with merged appendices as ONE document to D2L > Program-Level Assessment of Student Learning by the announced due date.

Use of Results from Previous Year Please describe how your program made adjustments and addressed the concerns that emerged in the data from the previous year’s assessment of student learning. This might include a plan for curriculum revision to enhance student performance to meet criteria, enhancing criteria to improve success of the program, etc.

I. Brief Statement of Departmental Philosophy or Mission

II. Educational Goals

The _(Name of program)______ program aims to produce graduates who can: G – 1.

mailto:Harman@calu.edumailto:laing@calu.edumailto:fisanick@calu.edumailto:hackett@calu.edumailto:Clites@calu.edumailto:Eliason@cal.edumailto:heh@calu.edu

27

G – 2. G – 3. G – 4. G – 5. G – 6. G – 7. (Insert/delete rows as needed for program’s goals.)

III. List of Educational Objectives: Intended Student Learning Outcomes

The list of measurable learning objectives (6-12 should suffice) is developed from the goals; and should be clearly linked to goals, e.g. Obj.1 – Goal I, etc. Use measurable verbs to describe the outcomes (see Bloom’s Taxonomy). Accomplishment of intended student learning objectives should be observable and measurable.

By the time of graduation _(Name of program)______ students will demonstrate the following:

Objectives: Intended Student Learning Outcomes Planned Academic Year (over 5 year period) For Measurement*

Ob. – 1. (linked to G – )

Ob. – 2. . (linked to G – )

Ob. – 3. . (linked to G – )

Ob. – 4. . (linked to G – )

Ob. – 5. . (linked to G – )

Ob. – 6. . (linked to G – )

Ob. – 7. . (linked to G – )

Ob. – 8. . (linked to G – )

Ob. – 9. . (linked to G – )

Ob. – 10. . (linked to G – )

(Insert/delete rows as needed for program’s objectives.) * NOTE: Each of the Intended Student Learning Objectives (Outcomes) should be measured at least once within the 5-year program review cycle.

IV. ASSESSMENT RECORD

2013 - 2014

http://www.wku.edu/sacs/bloomstaxonomy.htm

28

A. Intended Student Learning Objective (Outcome) A-1. First measurement A-1-a Means of Assessment A-1-b Criteria for Success

A-1-c. Assessment Results:

A-1-d. Use of Results: A-2. Second measurement A-2-a Means of Assessment A-2-b Criteria for Success

A-2-c. Assessment Results

A-2-d. Use of Results: B. Intended Student Learning Objective (Outcome) B-1. First measurement B-1-a Means of Assessment B-1-b Criteria for Success

B-1-c. Assessment Results:

29

B-1-d. Use of Results: B-2. Second measurement B-2-a Means of Assessment B-2-b Criteria for Success

B-2-c. Assessment Results:

B-2-d. Use of Results: APPENDIX A: Measures Used for THIS report APPENDIX B: OUTCOMES ASSESSMENT PLAN FOR NEXT ACADEMIC YEAR A. Intended Student Learning Objective (Outcome

A-1. First measurement A-1-a Means of Assessment A-1-b Criteria for Success

A-2. Second measurement A-2-a Means of Assessment A-2-b Criteria for Success

B. Intended Student Learning Objective (Outcome)

30

B-1. First measurement B-1-a Means of Assessment B-1-b Criteria for Success

B-2. Second measurement B-2-a Means of Assessment B-2-b Criteria for Success

A copy of the assessment instruments to be used for next academic year is appended here:

31

APPENDIX IV

ASSESSMENT OF STUDENT LEARNING OUTCOMES CHECKLIST -

YEAR END REPORT Last Revised: January 2, 2014

Program: ___________ Program Facilitator ___________Academic Year:

I. USE OF RESULTS FROM PREVIOUS YEAR:

Are the issues/concerns that were raised by the previous year’s data addressed (e.g., a plan for improvement, revision to the program, etc.)? Has a plan been described that addresses the student performance that did not meet the criterion?

Has a plan been described that advances/maintains successes in the program?

II. MISSION STATEMENT: A departmental statement of educational philosophy

Is the Mission Statement included?

Is the Mission Statement obviously and appropriately linked with the University’s Mission

Statement of “Careers and Character?”

Does the Mission Statement address BOTH aspects of the University Mission: Careers and

Character?

III. GOALS:

Are the goals appropriate for the program content?

Are the goals appropriate for the program level (undergraduate, graduate)?

Is the content of the Mission Statement reflected in the goals? Are at least three (3) goals listed?

Do the goals report what students who graduate from this program should know (knowledge), be

able to do (skills), and think (dispositions)?

IV. OBJECTIVES: Are objectives written in measurable terms? DO NOT use words like “understand,” “know.” Use

words like: “demonstrate knowledge of,” “apply,” “evaluate,” “critique,” words that are behaviors a student can do that can be measured. Is each objective clearly linked to at least one learning goal (and goal number is next to the objective)? Are ALL the goals linked to at least one objective?

Are at least six (6) and no more than ten (10) objectives listed?

Are two (2) objectives assessed each year?

Are all objectives assessed at least once within a 5 year period?

V. ASSESSMENT RECORD

Means of Assessment: Are at least but no more than two (2) objectives measured? FOR EACH OBJECTIVE ASSESSED IN THE REPORT:

Are two types of measurement (Means of Assessment) for each objective measured? You may use subjective or indirect measures (e.g., Asking students how well they think they have

learned some knowledge base, a skill or acquired a disposition) as ONE (1) of the measures,

BUT the other measure MUST be objective or direct (an exam, a presentation, a portfolio

item, etc.)

Are the Means of Assessment described to clearly indicate what is used to measure the objective? Is the plan of assessment clearly explained?

32

Are the means of the assessment a valid measure of that objective and is this indicated in the

description (e.g., if a standardized test is used, like the ETS Major Field Tests, does the

content of that test reflect the content of the objective. If a rubric is used, do the items that

get ratings reflect the objective content)? Is evidence provided that the means of measure is a valid measure of the objective? Are ALL measures numerical?

Are copies of ALL measures used included in Appendix A? If raters are used (for rubrics, for example), are there at least two? If raters are used, is inter-rater reliability measured (compute a correlation between the two

sets of ratings, then report the inter rater correlation; should be .80 or higher)

Are the raters identified (e.g., “Departmental ASLO committee” or “two outside observers”)? If rubrics are used, are all of the numerical levels behaviorally anchored

Example: Employer survey asks if the graduate can make ethical decisions

1= is not aware of the profession’s ethics

2 = is aware of the ethics but does not connect that to decision-making

3= makes connections between profession’s ethics and decision-making Example: Rubric for a paper that expects the student to evaluate information/critique/ and form an

Opinion or conclusion

1 = others’ ideas presented with no evidence of evaluation of those ideas, conclusion missing

2 = others’ ideas presented and critiqued in obvious ways (authors of those ideas themselves point out the flaws); conclusion is a reiteration of others’ ideas

3 = others’ ideas critiqued in student’s words, conclusion is student’s but does not seem connected to

the critique

4 = ideas critiqued by student, conclusion is original (in student’s words) and linked to critique

AVOID single word anchors that can be idiosyncratically interpreted by the raters (like “exemplary”, “adequate”, “below average”, or “needs remediation”)

Criterion for Success: FOR EACH OBJECTIVE

Is each criterion defined in a numerical manner?

Is it clear exactly what is meant by each criterion? Are use of grades avoided as a criterion? (Use percentage correct, average, etc. DO NOT USE

GRADES OR GPA)

Is it indicated the number of students that need to achieve success for each criterion? (Do a

certain percentage of students have to accomplish the criterion; is it a group average, will it be

within a range)?

Is the criterion a reflection of Cal U’s level of excellence (that is, is the criterion set high enough)?

Results: FOR EACH OBJECTIVE

Are sample sizes reported?

Is all available appropriate data used (not just a sampling of senior, but all available seniors)?

Are the reported measure of success in the terms of the criterion for success? (e.g., If 80% of the sample needed to achieve a 75% on an examination, then report the percentage who achieved a

75% on the exam) If means (averages) are reported, are standard deviations also included? Was a statement included regarding the achievement of each criterion for success?

Use of Results: FOR EACH OBJECTIVE

Were the results discussed as a department?

33

Were the results connected to future program objectives and goals? Were potential changes

discussed in response to the results?

Was a plan outlined to maintain the achieved results (if they met or exceeded the criterion for success)? If each criterion was not met, is a plan for program changes included?

Was a plan described that addressed student performance for those who did not meet the criterion?

ENTIRE REPORT: Are all of the above reported in clear and understandable ways?

Are all of the above completed for EACH objective measured this year?

Are copies of ALL assessment instruments included in Appendix A?

Are the following included in Appendix B:

The objectives plan to measure for the next academic year

The means of assessment and criterion for success according to the

check lists above?

Are copies included of all assessment instruments that are the

means of assessment?

Is the report ONE document – including the appendices? (use Adobe and save as .pdf)

Have raw data (and any student names) been eliminated from the report?

Is the report paginated?

Has the document been labeled and submitted as: CatalogNameofProgram_year_DepartmentName_LastNameofFacilitator

Does the report ONLY pertain to ONE program (do not report findings from all programs in the

department in one document – each program is to have its own report)

Program Facilitator: Date: Signature

College Coordinator: Date: Signature

Please submit your completed Annual Year-end Report to: D2L > Program-Level Assessment of Student

Learning