Using Assessment Data for Using Assessment Data for Continuous Improvement in Continuous Improvement in

General EducationGeneral Education

AAC&U ConferenceAAC&U Conference

Atlanta, GeorgiaAtlanta, Georgia

February 19, 2005February 19, 2005

Presented by…Presented by…

Linda Pomerantz-Zhang, Ph.D.Linda Pomerantz-Zhang, Ph.D.

lpomerantz@csudh.edulpomerantz@csudh.edu

C. Edward Zoerner, Jr., Ph.D.C. Edward Zoerner, Jr., Ph.D.

ezoerner@csudh.eduezoerner@csudh.edu

Sue Fellwock-Schaar, Ed.D.Sue Fellwock-Schaar, Ed.D.

sschaar@csudh.edusschaar@csudh.edu

California State University, Dominguez HillsCalifornia State University, Dominguez Hills

Is your institution collecting data for Is your institution collecting data for your General Education Program?your General Education Program?

What are you hoping to gain from this What are you hoping to gain from this session on assessment?session on assessment?

CSUDH CSUDH ProfileProfile~ One of 24 CSU campuses, 4 in LA metroOne of 24 CSU campuses, 4 in LA metro

~ 13,000 students, 61% undergrad13,000 students, 61% undergrad

~ Small but growing number of FTF (5900 are upper Small but growing number of FTF (5900 are upper division transfer students)division transfer students)

~ Approx 90% of incoming frosh need remediation in math, Approx 90% of incoming frosh need remediation in math, English, or bothEnglish, or both

~ High degree of cultural and racial diversityHigh degree of cultural and racial diversity

~ ECLP grant to improve critical student skills in writing, ECLP grant to improve critical student skills in writing, reading, critical reasoningreading, critical reasoning

Background to GE Assessment Background to GE Assessment Activities…Activities…

~ Administrators unable to encourage faculty to initiate & Administrators unable to encourage faculty to initiate & sustain program review for GEsustain program review for GE

~ System-wide pressure to reduce size of GE package & System-wide pressure to reduce size of GE package & streamline curriculumstreamline curriculum

~ Fear of “turf wars”Fear of “turf wars”

~ Inconsistent oversight from departmental level on upInconsistent oversight from departmental level on up

~ General distrust of administration motivesGeneral distrust of administration motives

~ 15 years of unsuccessful administrative efforts to reduce 15 years of unsuccessful administrative efforts to reduce size of GE & BA/BSsize of GE & BA/BS

And then came…And then came…~ Growing faculty concern about student achievement levelsGrowing faculty concern about student achievement levels

~ Creation of Student Learning Outcomes Assessment Creation of Student Learning Outcomes Assessment CommitteeCommittee

~ Creation of half-time Assessment Coordinator (faculty)Creation of half-time Assessment Coordinator (faculty)

~ WASC focus on student learning outcomes and “culture of WASC focus on student learning outcomes and “culture of evidence”evidence”

~ ECLP grant aimed at improving student writingECLP grant aimed at improving student writing

~ GE Syllabus Analysis sponsored by ECLPGE Syllabus Analysis sponsored by ECLP

~ Embedded Assessment initiativesEmbedded Assessment initiatives

~ Willingness to review GE in 5-year cycleWillingness to review GE in 5-year cycle

The tensions…The tensions…

Who controls the process?Who controls the process?(Administration vs. Faculty)(Administration vs. Faculty)

How are the data used?How are the data used?(Course improvement vs. (Course improvement vs.

faculty/program faultfinding)faculty/program faultfinding)

The Review Procedure—Year 1The Review Procedure—Year 1

Departments were notified in Departments were notified in Spring, 2003, that materials should Spring, 2003, that materials should

be gathered in Fall, 2003 and be gathered in Fall, 2003 and submitted in February, 2004submitted in February, 2004

Questions we asked in planning our Questions we asked in planning our GE Assessment…GE Assessment…

~ What types of assessment data should/could be What types of assessment data should/could be collected?collected?

~ What types of data analyses & who should conduct What types of data analyses & who should conduct them?them?

~ Who should see the collected data and analyses?Who should see the collected data and analyses?

~ How should assessment data be utilized and by whom?How should assessment data be utilized and by whom?

~ What types of obstacles may be encountered in utilizing What types of obstacles may be encountered in utilizing data?data?

~ How can the tension between assessment and faculty How can the tension between assessment and faculty fears be managed?fears be managed?

~ How can an operative feedback loop be insured?How can an operative feedback loop be insured?

Initial procedures in Fall, 2003…Initial procedures in Fall, 2003…

Forms for the department review Forms for the department review coordinators (chair or other as appointed) coordinators (chair or other as appointed) were distributed at initial meetingwere distributed at initial meeting

~Overview of the processOverview of the process~Course Assessment Record form for each Course Assessment Record form for each

Area A ObjectiveArea A Objective~Coordinator Report formsCoordinator Report forms~ Instructor Forms to be attached to graded Instructor Forms to be attached to graded

student workstudent work

Data collected from facultyData collected from faculty

TypesTypes~ SyllabiSyllabi

~ Student work with Student work with gradesgrades

~ Sample exams Sample exams with gradeswith grades

Analyzed for…Analyzed for…~ Alignment w/ GE Alignment w/ GE

objectivesobjectives

~ Student centered, Student centered, measurable course measurable course objectivesobjectives

~ Appropriate materialsAppropriate materials

~ Academic rigorAcademic rigor

~ Commensurability Commensurability across sectionsacross sections

Data collected from departments…Data collected from departments…

~ Description of ways assessment methods were Description of ways assessment methods were selected & aligned with GE objectivesselected & aligned with GE objectives

~ Description of how department ensured Description of how department ensured commensurability of course reqs & student commensurability of course reqs & student learning across sectionslearning across sections

~ Description of how the results of the review will Description of how the results of the review will be used to improve the instructional programbe used to improve the instructional program

~ Recommendations for changes in the Area A Recommendations for changes in the Area A learning outcome objectiveslearning outcome objectives

~ Feedback about the GE review process Feedback about the GE review process

Collected from RegistrarCollected from Registrar

Grades of all courses Grades of all courses under review, by section under review, by section but without faculty names but without faculty names

attachedattached

Review Team (4-5 members) met to analyze Review Team (4-5 members) met to analyze documentation and data in March, 2004documentation and data in March, 2004

Rubric was developed and review Rubric was developed and review procedures were designedprocedures were designed

Team reviewed findings with Administrator Team reviewed findings with Administrator who was working with us (LPZ)who was working with us (LPZ)

Administrator wrote letter regarding Team’s Administrator wrote letter regarding Team’s findings to GE Committeefindings to GE Committee

GE Committee reviewed findings of Review GE Committee reviewed findings of Review TeamTeam

GE Chair wrote letter to Department and GE Chair wrote letter to Department and attached letter from the Teamattached letter from the Team

(Only the letter from the GE Committee went (Only the letter from the GE Committee went to the Deans and Provost)to the Deans and Provost)

Team requested preliminary update in Fall, Team requested preliminary update in Fall, more complete update in Spring 2005more complete update in Spring 2005

Department’s Original ResponseDepartment’s Original Response

~Fear that some would employ results to Fear that some would employ results to argue for only one semester of Freshman argue for only one semester of Freshman CompComp

~General mild resentment over process; General mild resentment over process; some duplication of laborsome duplication of labor

~Mild hope that some good could come Mild hope that some good could come from the reviewfrom the review

Morals…Morals…

~Make sure things are non-threatening—Make sure things are non-threatening—people will see it that way despite efforts people will see it that way despite efforts to the contraryto the contrary

~Ask about local assessment practices and Ask about local assessment practices and consider using (parts of) them before consider using (parts of) them before implementing a system-wide practiceimplementing a system-wide practice

Original Department Actions Original Department Actions and Results…and Results…

~Chair attended meeting to learn about Chair attended meeting to learn about process and formsprocess and forms

~Chair attempted to gather required data Chair attempted to gather required data and provide analysisand provide analysis

~Chair underestimated time and energy Chair underestimated time and energy required and turned in an originally required and turned in an originally inadequate reportinadequate report

Moral…Moral…

~ If possible, provide a model response to If possible, provide a model response to guide those writing reportsguide those writing reports

Departmental next steps…Departmental next steps…

~Director of Composition assumed Director of Composition assumed responsibility for data accumulation and responsibility for data accumulation and analysisanalysis

~Report much more closely conformed to Report much more closely conformed to desired parametersdesired parameters

Moral…Moral…

~Get the right person for the job!Get the right person for the job!

Principal responses from Principal responses from Review Team…Review Team…

~ Concern over course objectives in Freshman Concern over course objectives in Freshman Comp IIComp II

~ Concern about syllabus uniformity and match to Concern about syllabus uniformity and match to catalog description, especially in Freshman catalog description, especially in Freshman Comp IIComp II

~ Concern about grading practices—higher than Concern about grading practices—higher than expected percentages in A and B rangeexpected percentages in A and B range

~ Concern about the way individual instructors Concern about the way individual instructors mark papersmark papers

Departmental response…Departmental response…~ Director of Comp and Chair review syllabi more Director of Comp and Chair review syllabi more

carefully; return to faculty for revision as neededcarefully; return to faculty for revision as needed

~ Memo of “Review of Objectives” and “Learning Memo of “Review of Objectives” and “Learning Outcomes” given to all instructorsOutcomes” given to all instructors

~ Director of Comp held Grading standards WorkshopDirector of Comp held Grading standards Workshop

~ Memo to/conferencing with instructors regarding Memo to/conferencing with instructors regarding grading standardsgrading standards

~ Rebuttal to Review Team’s concern regarding Rebuttal to Review Team’s concern regarding instructors’ marking of papersinstructors’ marking of papers

Morals…Morals…

~Seemingly large problems have Seemingly large problems have (potentially) simple solutions(potentially) simple solutions

~Reviewers need to be careful about Reviewers need to be careful about commenting on things outside their commenting on things outside their professional expertiseprofessional expertise

~Dialog can be fruitfulDialog can be fruitful

Short-term results…Short-term results…

~Generally improved syllabi, with greater Generally improved syllabi, with greater clarity and more explicit focus on analytic clarity and more explicit focus on analytic writingwriting

~Little change in grade distributionLittle change in grade distribution

Morals…Morals…

~Things Things cancan improve as a result of improve as a result of the assessment processthe assessment process

~Attempts at improvement must be Attempts at improvement must be sustainedsustained

Longer-range results…Longer-range results…

~Heightened awareness of need to simplify Heightened awareness of need to simplify Basic Skills Objectives in CatalogueBasic Skills Objectives in Catalogue

~General departmental satisfaction with General departmental satisfaction with Freshman Composition sequenceFreshman Composition sequence

~Understanding of need to move students Understanding of need to move students to analytical writing more quicklyto analytical writing more quickly

Changes in way pre-Freshman course Changes in way pre-Freshman course is runis run

Longer-range results… Longer-range results… (cont.)(cont.)

~Understanding that department needs to Understanding that department needs to remain vigilant against grade inflationremain vigilant against grade inflation

~Department needs to consider more Department needs to consider more systematic mechanism of review to systematic mechanism of review to ensure that course offerings uniformly ensure that course offerings uniformly meet desired objectivesmeet desired objectives

Things we learned about Things we learned about the process…the process…

~Although not perfect, the process was Although not perfect, the process was workableworkable

~Rubrics were not a perfect fit but a Rubrics were not a perfect fit but a reasonable guidereasonable guide

~Provide rubrics BEFORE departments Provide rubrics BEFORE departments begin the reviewbegin the review

~ Inter-rater training is importantInter-rater training is important~Consider what kind of report you will give Consider what kind of report you will give

to departments—quantitative or narrativeto departments—quantitative or narrative

Things we learned about Things we learned about the process… the process… (cont.)(cont.)

~ Five years is too long to wait for recommended Five years is too long to wait for recommended changes to be reviewedchanges to be reviewed

~ Departments who used coordinators generally Departments who used coordinators generally turned in stronger portfoliosturned in stronger portfolios

~ Need to increase awareness of the expectation Need to increase awareness of the expectation that the departments should be doing their own that the departments should be doing their own internal review and evaluation as a part of the internal review and evaluation as a part of the processprocess

Things we learned about Things we learned about the process… the process… (cont.)(cont.)

~ Start early enough to catch the exceptions to Start early enough to catch the exceptions to typical scheduling typical scheduling

~ Plan for contingencies regarding team Plan for contingencies regarding team members members

~ Should consider ways to have direct contact Should consider ways to have direct contact with students or information from themwith students or information from them

~ Mixed results are likely, but waiting until Mixed results are likely, but waiting until everyone is ready is not in the best interest of everyone is ready is not in the best interest of the studentsthe students