assessment of the first year experienceexperience ...fys coordinator, fys faculty, sr pra first year...

10/12/2010

1

Assessment of the First Year Assessment of the First Year ExperienceExperience-- Covering All the BasesCovering All the Bases

Dr. Mary Lou D’Allegro: Senior Director, Planning, Research & AssessmentPenn State BerksPenn State Berks

Dr. Tami Mysliwiec: Associate Professor, Biology & First Year Seminar CoordinatorPenn State Berks

The 2010 Assessment Institute- IUPUI - IUPUI October 24-26, 2010 1

Objectives for PresentationObjectives for Presentation

(1) Identify the critical first year experiences that should be assessed,

(2) Identify the assessments that will aptly capture these critical first year experiences,

(3) Develop a framework in which the assessment data will be analyzed,

(4) Explore the implications of both the improvement of the first year experience and subsequently student success, and

2

(5) Introduce a multi-dimensional model that enables assessment professionals to ensure that all that should be considered is assessed.

The 2010 Assessment Institute- IUPUI - IUPUI October 24-26, 2010

10/12/2010

2

What this Presentation is NOTWhat this Presentation is NOT

Creative methods to assess the First Year Experience

Innovative use of existing technology or adaption of new faculty

New ways to engage faculty & staff

3


Admittedly,Admittedly,This Presentation is Just Plain Boring!

4


10/12/2010

3

BackgroundBackground

Penn State Berks is a college campus of the Pennsylvania State University (PSU) system

2800 students926 New Baccalaureate in fall 2008 741 New Baccalaureate in fall 2009

SAT-Critical Reading + SAT-Quantitative hovers around+/- 1000

30% of students are in residence halls Almost all are full-time

5

Require First Year Seminar (FYS) for all first time students


YouTube Video: http://www.youtube.com/watch?v=DaMX-QmdSE0

BackgroundBackground By Spring 2009, University required all campuses (19) to

have a First Year Experience (FYE) plan in place

Must address PSU goals for FYE (set in stone)• G l 1 d i l i d i h h h l l i f • Goal 1: to engage students in learning and orient them to the scholarly community from

the outset of their undergraduate studies in a way that will bridge to later experiences in their chosen majors, and

• Goal 2: to facilitate students’ adjustment to the high expectations, demanding workload, increased liberties, and other aspects of the transition to college life.

Objective 1: to introduce students to University study

Objective 2: to introduce students to Penn State as an academic community, including fields of study and areas of interest available to students

6

g y

Objective 3: to acquaint students with the learning tools and resources available

Objective 4: to provide an opportunity for students to develop relationships with full-time faculty and other students in an academic area of interest to them

Objective 5: to introduce students to their responsibilities as part of the University community.


10/12/2010

4

BackgroundBackgroundAssessment of First Year Engagement Plan

Semester

Assessment

Assessment Activity

When Administered

Who is Responsible?

Cohort

Purpose

Spring 2008

NSSE Survey Feb 08-Apr 08

SR PRA* First Year/ Seniors

Baseline

Summer 2009

Survey of FTCAP, Survey Student

Orientation Experience

Last week in June, July, 3rd

week in August

SR PRA, DR AA**, AS CL***

All new incoming students

Baseline

rdFall 2009 Survey of FYEP Learning Outcomes

3rd Week,12th Week

FYS Coordinator,FYS Faculty,

SR PRA,

First Year Seminar Students

Assess student gains in outcomes between

pre/postSpring 2010

Survey of FYEP Learning Outcomes

3rd Week, 12th Week

FYS Coordinator, FYS Faculty,

SR PRA



pre/postSummer

2010 Survey of FTCAP,

Survey Student Orientation Experience

Make changes to FTCAP, Student

Orientation, and FYS based on results of

Fall 2009/Spring 2010 Surveys

AS CL, DR AA,

FYS Coordinator, Associate Dean

All new incoming students

Assess improvement of FTCAP and Student Orientation Experience with regards to FYEP

student outcomes

Fall 2010 Survey of FYEP Learning Outcomes

3rd Week, 12th Week


SR PRA



pre/post

7


Spring 2011

NSSE Survey

Survey of FYEP Learning Outcomes

Feb 11-Apr 11

3rd Week, 12th Week

SR PRA


SR PRA

First Year/ Seniors


Ascertain gains in composite scores

between 2008-2011 scores for BOTH

Cohorts


pre/postSummer

2011 Look to see if changes

to FYS improved student outcomes

FYS Coordinator, SR PRA,

Associate Dean

First Year Assessment ComponentsFirst Year Assessment Components

(0) Assessment of Placement Testing Experience (FTCAP),

(1) Assessment of the New Student Orientation,(1) Assessment of the New Student Orientation,

(2) Survey of the First Year Seminar,

(3) Student success indicators, and

(4) Examination of NSSE results for first year students.

8


10/12/2010

5

First Year Assessment ComponentsFirst Year Assessment ComponentsConceptual ModelConceptual Model

Orientation (1)

Placement testing

(0)

FYS (2)

Extent of College

Engagement (3)

( )

9


Student Success Indicators

(4) Ultrinstic, 2010

FTCAP SurveyFTCAP Survey

•Ask students & parents questions about FTCAP activities

•Results VERY positive so there has been no drill-down results to:

Orientation, FYS, NSSE, Student Success Indicators

10

• Comparison to previous FTCAP Survey administrations


10/12/2010

6

New Student Orientation SurveyNew Student Orientation Survey•Asks questions about orientationactivities

•Asks questions that connects the orientation activities to the FYE orientation activities to the FYE goals & objectives

• If students don’t respond to “connection” questions, revamp orientation

11

•Working on student learning (ex. Common Reading FYS score on Common Reading assessments)


New Student Orientation SurveyNew Student Orientation Survey

12

•Results focused around orientation activities


•Results focused around FYE goals

10/12/2010

7

New Student OrientationNew Student Orientation(New International Students Survey)(New International Students Survey)

13


Pre/Post Survey of the FYSPre/Post Survey of the FYS

• Asks questions about the FYSactivities How often engage in “good behaviors” How important good behaviors arep g Behavior categories

Classroom (ex. attentive in class)Academic (ex. think critically)

Personal (ex. network with other majors)

• Asks questions that connect theFYS and FYE goals & objectives

14


• If responses are poor on POST or minimal positive movement from PRE POST revise FYS

10/12/2010

8

NSSE Results for First Year StudentsNSSE Results for First Year StudentsStep 1: Look at five composites for First Year vs SeniorsStep 2: Look at (-) comparisons to three peer sets

(1) PSU college campuses(2) “Local Competitors”-regional area schools that Berks compete(3) “National Benchmark ” schools with better retention graduation rates than Berks

First Year (FY) vs. Seniors

(SR)First Year

(FY)Seniors

(SR)TOTAL

PLUS (+)

2003-2006-2008

NSSE 2008 Results

PSU

Peer Comparisons

National BenchmarkLocal Competitive Set

(3) National Benchmark- schools with better retention graduation rates than Berks

Step 3: Look at previous administrations

15


Composite FY FY SR FY SRLevel of Academic Challenge (LAC) - + - - - + - - 2

Active & Collaborative Learning (ACL) + + - + - + + - 6Student-Faculty Interaction (SFI) + + + + + + + - 8

Enriching Educational Experience (EEE) - - - - - + + - 2Supportive Campus Environment (SCE) + + - + - - - - 3

p

NSSE Results for First Year StudentsNSSE Results for First Year StudentsStep 4: Look at individual item effect sizes (with other PSU Campuses)

both (-) & (+) within composites that peers outperformed Berks

Composite Item #First Year Senior First Year Senior First Year Senior

Item Effect Size (ES)^ ES Significant?MeanFirst Year Senior First Year Senior First Year Senior

1l.

2.58 2.71 -0.08 -0.28 *

1u. 2.48 2.57 -0.07 -0.04

1v. 2.56 2.47 -0.13 -0.25 * *

7a. 0.10 0.64 0.25 0.01 **

7b. 0.25 0.48 -0.18 -0.28 *** *

7c. 0.11 0.26 -0.01 0.047e. 0.18 0.37 -0.23 -0.22 *** *

7f. 0.02 0.04 0.01 -0.31 ***7g. 0.04 0.21 0.03 0.08

Enrich

ing E

duca

tional E

xperie

nc

Foreign language coursework

Independent study or self-designed majorStudy abroad

Had serious conversations with students of a different race or ethnicity than your own

Had serious conversations with students who are very different from you in terms of their religious beliefs, political opinions, or personal values

Used an electronic medium (listserv, chat group, Internet, instant messaging, etc.) to discuss or complete an assignment

Practicum, internship, field experience, co-op experience, or clinical assignment

Community service or volunteer worka t c pate a ea g co u ty o so e ot eformal program where groups of students take two or more classes together

16


7h. 0.04 0.31 0.16 -0.07

9d. 2.13 1.53 -0.15 -0.56 * ***

10c. 2.79 2.45 0.11 0.07

ce (E

EE

)

Encouraging contact among students from different economic, social, and racial or ethnic backgrounds

Culminating senior experience (capstone course, senior project or thesis, comprehensive exam, etc.)

Participating in co-curricular activities (organizations, campus publications, student government, fraternity or sorority, intercollegiate or intramural sports, etc.)

10/12/2010

9

NSSE Results for First Year StudentsNSSE Results for First Year Students

Step 5: Should be looking at NSSE results by(1) Orientation participation(2) FYS type (Linked or not linked)( ) yp ( )

(3) FYS grade categories (A-C, D-F, W)

17


Student Success IndicatorsStudent Success Indicators

•Student Success IndicatorsFirst Semester Credits EnrolledFirst Semester Credits EarnedFirst Year Credits Earned First Year GPA One-year retention rateSix-year graduation rate

18


10/12/2010

10

Student Success IndicatorsStudent Success Indicators

• Over 40 cohorts identified by Enrollment Management Team & First Year Experience Committee

Group *Club/activities/recreation

*Commuters and off-campus housing students *Veterans *STEM *Intercollegiate athletics Ethnicity Gender Committee

• Also examine student success indicators (SSI) by: (1) Orientation participation(2) FYS type (Linked or not linked)(3) FYS grade categories

(A-C, D-F, W)

• Comparison of SSI of cohort

Gender Parent education level Non-traditional Learning Community Discovery Learning Center Aspire students ESL Apart from the Start Student “leaders” Remedial courses Gateway courses Honors students Hybrid classes 1st Year assessment Judicial issues Financial Aid

19


• Comparison of SSI of cohort to overall baccalaureate or associate cohort

Financial Aid Division/Majors Instate/Out of State Service area Alumni CHOA Clearing house “Crossing the Finish Line” Early Intervention Students

Student Success IndicatorsStudent Success IndicatorsStandard Report of Student Success Indicators for overall, & each of the eight that were completed this summer

Veterans Count Fall 2003 Level

1st Semester Credits Enrolled

1st Semester Credits Earned

1st Semester

GPAOne Year Retention

3 (Asc)/ 6 (Bac) Year Graduation

ALL Veterans

CountALL B h l 14 68 14 35 2 68 83 2% 63 6%ALL Bachelors 14.68 14.35 2.68 83.2% 63.6%ALL Associate 13.05 11.87 2.02 82.8% 12.6%

17 Veterans Bachelors 12.56 12.56 3.12 82.4% 82.4% 898 Veterans Associate 9.86 9.86 2.72 75.0% 37.5%

20


10/12/2010

11

RecommendationsRecommendations

Sometimes letting the 400 lb gorilla stay in the room is preferable to reporting assessment results

That said…

21


RecommendationsRecommendationsBefore you start the assessment• Have articulated goals or objectives or both in place

Assessment• Assess to those goals,

Assess the intangibles, “the means,” but

Do not loose site of the “ends”- student success

22


10/12/2010

12

RecommendationsRecommendationsData Analysis Considerations:• Disaggregate results by specific student groups

• Use comparative data if available (historic, peer institutions)p ( , p )

Use of Assessment• Be able to connect the “ends” to the “means” (slide 9)to make assessment results meaningful and valuable

23


to make assessment results meaningful and valuable

• Use assessment of the “means” to identify milestones or activities that affirm student success or fall short

QuestionsQuestions

24


assessment of the first year experienceexperience ...fys coordinator, fys faculty, sr pra first year...

Documents