April 8, 2010: Drafted by Perry Deess, Fadi Deek, and Norbert Elliot

May 11, 2010: Presented to President Altenkirch, Provost Gatley; Committee for Department and Program Assessment (Steve Tricamo, Chair as Associate Provost, Joel Bloom, Sunil Saigal, Urs Gauchat, Fadi Deek, Narain Gehani, Bob English, Rich Sweeney, Jack Gentul); and Sub-Committee on Assessment (Perry Deess, Co-chair as IRP Director, John Bechtold [Member as Master Teacher], James Geller, Norbert Elliot [Member as MSCHE Chair], John Cays, Shanti Gopalakrishnan, Bruce Bukiet, Ala Saadeghvaziri, Barry Cohen)

Updated: September, 2012: Perry Deess, Judy Redling

Updated Guidelines reflect Committee structure changes, edits to Table of Contents (including Part III-A-iv, details on external assessment plan), and additional requirements for the Appendix: syllabi for core courses (with states learning outcomes) and NJIT Quality in E-learning rubric for all online courses.

NJIT Program Review Process: Guidelines

Background

The enhanced program review process is responsive to increased demands for accountability. A synthesis of program review guidelines from accreditation agencies such as the Accreditation Board for Engineering and Technology, the Association to Advance Collegiate Schools of Business, the National Architectural Accrediting Board, and the Middle States Commission on Higher Education—as well as university strategic planning initiatives—the program review process allows both transparency and collaboration: In reviewing all NJIT program, the process allows NJIT curriculum directors to learn from each other and to create innovative strategies for achieving enhanced program outcomes.

Introduction

Beyond a mere audit, the Program Review Process has four innovative objectives: 1) To provide a forum for the assessment and improvement of all degree and university programs; 2) To demonstrate continuous improvement in the delivery of educational curricula; 3) To promote a culture of assessment by building a cohesive assessment strategy; 4) To create a central, web-based repository for assessment design and supporting documents. These four objectives will yield a university-wide forum for the following: 1) collaboration on establishing program vision, strategies, tactics, and metrics; 2) investigation of new ways of understanding student achievement; 3)

collaboration on common ways of reporting student outcomes; and 4) creation of a web-based system of promoting program assessment models.

While all degree-granting programs will participate in program review to advance the key objectives, programs accredited through the Accreditation Board for Engineering and Technology, the Association to Advance Collegiate Schools of Business, and the National Architectural Accrediting Board will not be required to complete a separate program review report. Program directors, working with a review team within their academic units, will be asked to submit their existing accreditation reports with key sections highlighted that address the NJIT program review guidelines. Program directors will also submit a brief cover memo, of no more than 2 pages, identifying the key sections and describing changes made in response to the accreditation self-study. The reason for requesting review of accredited programs is to allow a sense of university-wide collaboration, to study new ways of coming to terms with the complexities of student achievement, to encourage collaboration on strategies of reporting student outcomes, and to create a new model by which we may promote our innovative assessment models.

Vision

The NJIT Program Review Process will answer the following questions:

How are institutional and program level goals integrated and articulated? How have planned assessment processes been implemented? How do assessment results provide convincing evidence of student

learning? How do program administrators use assessment in decision making? What changes have been made based on assessment? How have program assessment results been communicated to

shareholders? What is the cohesive framework of program assessment at NJIT?

Governance and the Program Review Process

The NJIT Program Review Process is governed by a standing university committee, The Committee on Assessment of Student Learning, reports to the Deans’ Council (the Provost and deans from Newark College of Engineering, the College of Architecture and Design, the School of Management, the College of Computing Sciences, the College of Science and Liberal Arts, and the Albert Dorman Honors College. Committee members include the Associate Provost for Academic Affairs, The Director of Institutional Research and Planning, Associate Provost for Graduate Studies, Associate Deans, identified faculty, and identified administrators (UCRC Chair, TLT Chair, Graduate Council Chair).

The process begins with the selection of the university’s degree programs to move through review on a five-year rotating basis. That is, the process calls for approximately

2

22 programs to be reviewed each year. Once these programs are identified by the Committee on Assessment of Student Learning within the five year cycle, selected programs will work collaboratively with the assessment team in Institutional Research and Planning to add available measures to the existing review strategy.

Program directors will then develop an analysis focusing on student learning outcomes. Directors will include a detailed description of how data have been used to make improvements in the past and how they will use the self-study process to drive future improvements. Once the report is ready, the Committee on Assessment of Student Learning will undertake the review on behalf of the Dean’s Council.

After the program review is completed, the Committee makes one of three levels of recommendations: an exemplary recommendation; a qualified program recommendation; or review for monitoring. If a program receives an exemplary recommendation, a concerted effort will be made to strengthen other degree programs by advocating process similar to those practiced by the exemplar program. If a program receives a qualified program recommendation, a concerted effort will be made to identify the standards of achievement of the qualified program so that all programs may achieve similar standards; if a program needs further monitoring, the committee will document the case, work with the program director to improve program effectiveness, and report that process to the Committee for Assessment of Student Learning. The Committee may then refer the program to the provost for further action.

Timeline

The timeline consists of five phases that begin with the agenda established by the Committee on Assessment of Student Learning.

1. Initiate Process Operating within the five year cycle, the annual process begins with a review of the yearly roster of the programs to be reviewed in the five year process. Concurrently, the NJIT Office of Institutional Research and Planning plans to provide technical assistance to directors of the programs to be reviewed. The fall process also includes the charge to the Committee.

2. Orientation of the Program Directors Members of the Committee orient the identified program director to the review process. The orientation consists of a review of guidelines, a review of procedures, and an overview of resources available to assist the director.

3. Assemble Data The Office of Institutional Research and Planning provides the program director with the necessary data to needed to complete or enhance the program review document.

3

4. Submit the ReportWith the support of Institutional Research and Planning, the program director submits the report to the Committee.

5. Committee ReviewThe Committee reviews the report and makes recommendation with a 1 page summary of program strengths and weaknesses to the Dean’s Council.

6. The Deans’ Council Makes Final RecommendationsThe Deans’ Council makes final recommendations and issues a brief report citing program strengths and weaknesses to the provost.

The timeline is described in the figure below:

Program Review Process: Timeline of Annual Activities

4

Committee on Assessment of Student Learning initiates process (Co-chairs)

Orientation of the program director to the NJIT Program Review Process (Co-chairs)

Data Assembly (Office of Institutional Research and Planning)

Program Review Report submitted

Committee on Assessment of Student Learning reviews Program Review Report

Charge to Program Directors

In preparing the self-study, the program and the academic unit’s program review team director should keep two goals of this process in mind:

1. To ensure that NJIT is offering quality academic programs (e.g. has appropriate educational objectives, and is accomplishing those objectives as evidenced by student learning outcomes assessment)

2. To demonstrate continuous improvement in the delivery of the educational curriculum.

In preparation for the review, the program director should review the following:

• Program oversight and management• Program objectives• Core course objectives• Relationship between curriculum and objectives• Resources—human, physical, financial—required to support the curriculum• Evaluation student learning outcomes according to a previously established

assessment plan• Measurement of student performance in courses• Measurement of student performance across courses• General measurement of student learning outcomes• Measurement of student attitudes

• Obstacles to meeting objectives• Continuous curricular improvement• Signature features, such as undergraduate research experience• Submission of the Institute Information Literacy Report• If eLearning is involved, review the Institute eLearning Quality Assurance Report

Charge to Institutional Research and Planning

The Office of Institutional Research and Planning will provide each director with the following information:

Graduation rate Retention rate Time to degree Five year enrollment Most recent diversity measures Average GPA by student level

Collaboration on the NJIT Program Reporting Process

Program directors, members of the academic unit review team, and representatives from Institutional Research and Planning will work together to assemble the following evidence related to program review.

5

Deans review Committee report and make final recommendations to the Provost

Describe relationship of program outcomes and educational objectives; Describe the processes used to assure that graduates have achieved the

program outcomes: identify direct and indirect measures of the learning outcomes.

Provide qualitative and quantitative data to demonstrate that graduates satisfy planned program outcomes;

Describe how the assessment results are implemented to improve the program.

The NJIT Program Report Document

The NJIT program report provides an overview of the program in general and it’s the assessment of student learning outcomes in particular. It directly addresses the twin objectives of assessment and continuous improvement by demonstrating how the program meets its own educational objectives and how it has made changes to improve the program. The program review report should employ Middle States terminology, unless the program has external accreditation. For additional assistance see the Middle States guide to assessing student learning: http://www.msche.org/publications/Assessment_Expectations051222081842.pdf

For a more comprehensive discussion see:http://www.msche.org/publications/SLA_Book_0808080728085320.pdf

Programs participating in an external accreditation process such as ABET, NAAB, or AACSB will be asked to submit existing reports with a detailed cover letter and relevant sections marked.

An additional program review report will be scheduled for the assessment of student learning outcomes in the GUR. This report will follow a similar model, but with a different structure because it involves the assessment of learning across the curriculum.

Definitions:

Institutional Learning Goals: As students seek disciplinary mastery, they will achieve skills and knowledge in:

Research-based Inquiry:  Students employ investigative methods Ethical Conduct:  Students understand professional and civic

responsibility Economic Opportunity:  Students understand economic reasoning and

demonstrate that they are able to allocate resources effectively and logically under operating constraints

Collaboration:  Students work effectively in teams to engage multidisciplinary perspectives

Engagement:  Students are active and committed learners

6

Program Learning Goals:Statements that describe what students are expected to know and be able

to do by the time of graduation. These relate to skills, knowledge, and behaviors that students acquire as they progress through the program.

Course Learning Goals: Formally defined expectations for what students should know and be able

to do after they complete a course. These relate to skills, knowledge, and behaviors that students acquire as they progress through the course.

Learning Outcomes:The knowledge, skills, and abilities that students have attained as a result

of their involvement in program based educational experiences.

Assessment of student learning:Assessing student achievement employs evidence from student courses,

projects, activities, or performances to determine the extent to which student learning achieves the stated program goals

http://www.msche.org/publications/examples-of-evidence-of-student-learning.pdf

Sample Table of Contents

I. General program information (2 page)A. Program background information: B. Specify program title(s): all degrees awarded for the program under review,

including options, as specified in transcripts;C. Indicate program mode: (e.g., face-to-face or distance learning; percentage of

students enrolled part-time).D. Program details: discuss metrics and implications for program, including how

data have been used to make improvements in the past and how they will use the self-study process to drive future improvements.

1. Graduation rate2. Retention rate3. Time to degree4. Five year enrollment5. Most recent diversity measures6. Average GPA by student level

II Program mission and goals (1 page)A. Describe program mission and program learning goalsB. Provide a matrix showing how the institutional learning goals are mapped to the

Program Learning Goals (See Table 1)Table 1. Matrix of Institutional Learning Goals and Program Goals

7

  Institutional Learning Goal 1: Researched-based Inquiry

Institutional Learning Goal 2: Ethical Conduct

Institutional Learning Goal 3: Economic Opportunity

Institutional Learning Goal 4: Collaboration

Institutional Learning Goal 5: Engagement

Program Learning Goal:1

         

Program Learning Goal: 2

         

Program Learning Goal: ETC.

         

III Learning outcomes (4 pages)A. Discuss how the student achievement of program learning goals is

assessedi. Describe the program assessment plan:

How are student learning outcomes assessed over the course of the program? Indicate measures of student learning and criterion-based standards (rubrics) against which learning is evaluated.

ii. If the assessment plan includes assessment at the course level, how are course learning outcomes related to the achievement of program learning goals.(see Table 2). Syllabi with learning outcomes for all core courses should be provided in the appendix.

iii. If the assessment employs portfolio analysis or the application of rubrics to specific student work describe the process, including how standards are defined.

8

iv. Provide details on external assessment plan for comparative quality assurance: discipline-specific exam, external review of final project, portfolio, or exam.

v. For programs where there are no well defined core courses, assessment should be based on a senior project, thesis, qualifying exam, publications or dissertation.

vi. For undergraduate programs, identify core courses with an information literacy component. Describe the information literacy assessment process and results.

vii. Undergraduate programs should address the contribution of the GUR to learning outcomes in the program. (For GUR program review guidelines see Appendix 1)

viii. If students can enroll in a online courses, include in the appendix a completed NJIT Quality in eLearning Rubric for each course:www.njit.edu/tlt/quality-assurance/quality-assurance.pdf

Table 2. Matrix of Program Learning Goals to Core Course Goals

  Program Learning Goal:1

Program Learning Goal:3

Program Learning Goal: ETC.

Core Course 1Goals

     

Core Course 2Goals

     

Core Course 3 Goals, etc.

     

IV Using student learning assessment (1 page)A. Discuss the results of the assessment and to what extent these results

provide evidence that program learning goals have been achieved.B. Describe what specific program strengths and weaknesses are revealed

by the assessment of student learning outcomes.C. Show how the assessment results are being used to implement

changes. (What are those changes?)V. Assessment in the next cycle (1 page)

9

A. Discuss the sustainability of assessment and how it is integrated into the program.

B. Describe planned improvements in the assessment process.

VI Conclusion (1 page)

A. SummaryB. Future improvements

Appendices—Supporting Documents (possible examples shown below)

A) Syllabi with learning outcomes for all core coursesB) Program assessment supporting data including rubricsC) Summary of test resultsD) Summary of ePortfolio evaluationE) NJIT Quality in E-learning rubrics and evaluation reportF) Presentations and publications associated with student learning assessment

Implementation

The guidelines described in this document are effective for program review reports written after Spring 2012 and will remain in effect until superceded by a new set of guidelines.

10

APPENDIX I

ASSESSMENT OF THE GENERAL UNIVERSITY REQUIREMENTS: A fundamental guiding principle in the development of the General University Requirements (GUR) is the formulation of a foundational curriculum encompassing the necessary preconditions for success in undergraduate disciplines, a curriculum that establishes a devotion to lifetime intellectual discovery and personal development. Learning Outcomes relevant to this guiding principle are articulated in the NJIT Core Competencies. Assessment of the Core Competencies is achieved through the use of locally-developed methods and nationally-based tests.

The NJIT Program Review Process for GUR will answer the following questions:

What learning outcomes are specific to a thematic area within the GUR? Which Core Competencies are addressed? How is a given Core Competency

integrated and articulated in the GUR thematic area? How have planned assessment processes been implemented? How do assessment results provide convincing evidence of student learning? How do program administrators use assessment results in decision making? How have actions been taken as a result of assessment? How have assessment results been communicated to a variety of shareholders?

Review Process

The GUR program review report will be prepared by the Program Review Committee as a whole, based on separate contributions from each of the disciplines offering GUR courses. The review will follow the overall structure discussed in the Program Review Guide, but with separate sections for each discipline and will cover all standard GUR courses in the GUR disciplines. The process of aligning and assessing goals from the institutional goal to the program goal, to the discipline goal, to the course level will be followed with an additional focus on the extent to which core competency benchmarks have been achieved.

The assessment of the GUR differs in several ways from the other program reviews. First, it must add a discipline level for learning goals because the GUR includes one step between the program goal and the course goal and this is the goal of the specific GUR discipline. Second, the emphasis for the review will be on the core competencies. This is not to suggest that other goals of the GUR including general education, engaged citizenship, and a love of life-long learning are of lesser importance, but they are notoriously difficult to assess. The extent to which they are addressed by individual courses will be considered but beyond this, assessment of these goals will belimited.

11

Table 1. The NJIT General University Requirements

GUR DISCIPLINE DESCRIPTION NUMBER OF

CREDITS

Computing Sciences

An understanding of the nature of computing, its impact on society, and the driving forces behind its pervasive deployment is integral to effective functioning as a professional and as a citizen.

3 credits

Cultural History

All educated individuals are expected to understand and appreciate history and the world’s cultures. 3 credits

EnglishThe ability to communicate ideas is an essential characteristic of educated individuals. All students are expected to achieve proficiency in both oral and written English.

6 credits

HumanitiesThe ideals of a liberal education transcend particular major fields and career goals. All students are expected to develop an interest in specific areas within the humanities.

6 credits of junior-level electives;3 credits of senior seminar

Management

All students are expected to develop the management skills needed to function effectively in an organizational setting.

3 credits

MathematicsThe ability to reason qualitatively and quantitatively, to understand probability, and to apply mathematical models to a variety of circumstances is fundamental to making informed decisions in the modern world.

6 credits, including at least 1 credit of probability and statistics

Natural Sciences

The natural sciences provide the basis for our knowledge of the physical universe and for technological progress. All students are expected to develop a thorough understanding of at least one laboratory science.

7 credits, including at least 1 credit of lab

Social Sciences

An understanding of the social sciences is essential in order to understand the economic, social, and political forces at work in our world.

6 credits

Core Competencies:As all students, regardless of discipline, experience General University Requirements, we establish benchmarks and measure these three Core Competencies to support the improvement of student learning:

A. Writing, Reading, and Critical ThinkingB. Quantitative ReasoningC. Information Literacy

12

Table of Contents

I. General program information (2 pages) Brief description of GUR and its relationship to the institutional learning goals

II Program mission and goals (3 pages)C. Describe GUR mission program learning goals, and discipline level learning goalsD. Provide a matrix showing how the institutional learning goals are mapped to the

Program Learning Goals (See Table 1) and how the Program Goals are mapped to discipline goals (Table 2)

Table 1. Matrix of Institutional Learning Goals and Program Goals

  Institutional Learning Goal 1: Researched-based Inquiry

Institutional Learning Goal 2: Ethical Conduct

Institutional Learning Goal 3: Economic Opportunity

Institutional Learning Goal 4: Collaboration

Institutional Learning Goal 5: Engagement

Program Learning Goal:1

         

Program Learning

         

13

Goal: 2Program Learning Goal: ETC.

         

Table 2. Matrix of Program Goals and Discipline Goals

  Program Goal 1: Researched-based Inquiry

Program Goal 2: Ethical Conduct

Program Goal 3: Economic Opportunity

Program Goal 4: Collaboration

Program Goal 5: Engagement

Discipline Goal:1

         

Discipline Goal: 2

         

Discipline Goal: ETC.

         

III Learning outcomes (10 pages)B. Discuss how the student achievement of program learning goals is

assessed in each disciplineix. Describe the program assessment plan:

14

How are student learning outcomes assessed over the course of the program? Indicate measures of student learning and

criterion-based standards (rubrics) against which learning is evaluated.

x. Most assessment will be conducted through the analysis of results from of nationally normed tests. If the assessment plan in a discipline includes additional assessment at the course level, how are course learning outcomes related to the achievement of program learning goals.(see Table 3).

xi. If theadditional assessment employs portfolio analysis or the application of rubrics to specific student work describe the process, including how standards are defined.

xii. Identify core courses with an information literacy component. Describe the information literacy assessment process and results. (Most of this will be conducted through the iSkills exam, but additional evidence may also be provided.)

xiii. If students can enroll in an online course, include a completed NJIT Quality Matters Rubric www.njit.edu/tlt/quality-assurance/quality-assurance.pdf

Table 3. Matrix of Discipline Goals to Course Goals

  Discipline Goal:1

Discipline Goal:3

Discipline Goal: ETC.

Course Goal 1

     

Course Goal 2

     

Course Goal 3 etc.

     

E. Provide a matrix showing mapping of Core Competencies to GUR courses (Table 5)

Table 5. Matrix of Core Competencies and Courses

15

  CC1: Writing, Reading, Critical Thinking

CC2: Quantitative Reasoning

CC 3: Information Literacy

(Other)Learning Outcome 1

(Other)Learning Outcome 2

Course1      Course 2      Course 3ETC.      

F. For the overall GUR program the program review committee will establish core competency benchmarks that indicate a target achievement level. For example, at least 70% of students must achieve Core Competency #1.

IV Using student learning outcomes assessment overall

D. Discuss the results of the assessment and to what extent these results provide evidence core competency benchmarks and program goals have been achieved

E. Describe what strengths and weaknesses are revealed F. Show how the assessment results are being used to implement changes. (What are

those changes?)

IV Using student learning assessment, for each separate discipline (4 page)A. Discuss the results of the assessment and to what extent these results

provide evidence that program and discipline learning goals have been achieved.

B. Describe what specific program strengths and weaknesses are revealed by the assessment of student learning outcomes.

C. Show how the assessment results are being used to implement changes. (What are those changes?)

V. Assessment in the next cycle (2 page)A. Discuss the sustainability of assessment and how it is integrated into

each disciplineB. Describe planned improvements in the assessment process.

VI Conclusion (1 page)C. SummaryD. Future improvements

16