Holistic Course Placement and Whole-Student Model of Early

Intervention

Kevin Li, Dean of InstructionSara Schupack, Director of Developmental EducationLawrence Buonaguidi, Quality Assurance Coordinator

Steve Robbins, Director of Research Innovation, Academic and Workforce Success

The Innovations 2013 ConferenceMarch 12, 2013; Dallas, Texas

Part 1: Holistic Course Placement

ETS/CCC Collaboration Objectives• Combine COMPASS and Psychosocial Measure to accelerate

student progress and college completion– Students placed within the upper-band and combined with

high psychosocial scores can enroll in the next level course• Allow more individualized, actionable plans for students tied to

curricular goals• Non-cognitive survey and in-class behavioral ratings will be used

to trigger Early Alert and intervention strategies • Create supplementary services tied to Early Alert and to

behavioral monitoring

• Two new studies from the Community College Research Center (CCRC) at Teachers College, Columbia University suggest that a significant number of students may be accelerated if provided proper supports

• Placement tests alone do not yield strong predictions of how students will perform in college

• Success requires intervening with at-risk students to ensure completion and success

Background

An Alternative Measure Based on Grounded Research: Non-Cognitive Assessment

• A meta-analysis was conducted as an attempt to integrate the psychological and educational literatures regarding the postsecondary outcomes of achievement and retention (Robbins, S., Lauver, K., Le, H., Davis, D., Langley, R., & Carlstrom, A., 2004)

• Examined the relationship between psychosocial and study skill factors (PSFs) across 109 studies

• Results indicated relationships between retention and academic goals, academic self-efficacy, and academic-related skills

An Alternative Measure Based on Grounded Research: Non-Cognitive Assessment

• Respect the whole student (examine cognitive, non-cognitive, behavioral factors)

• Focus on the unique individuality of students

• Focus on factors educators can control and empower students to modify: such as motivation, social connectedness, as opposed to socio-economic, situational factors

• CCC found that behavioral monitoring and intervention was key to student success

[HAND-OUT for group exercise 1]

Group interaction 1: 5 min.

• Form a 3-4 person group• Look over the hand-outs• Discuss what “non-cognitive” or “psycho-social skills” mean to

you and how they are manifested.• How would you assess, monitor, and address these

components at your campus?• Report out

[HAND-OUT for group exercise 1]

Domain Facet Definition Example Items

Academic SkillsBehaviors, beliefs, and skills that directly facilitate academic success.

OrganizationBeing planful in one’s actions, including effective use of time management and organizational tools and strategies.

I make a schedule for getting my school work done.I take due dates seriously.

Behavioral Compliance Complying with expected, academic-related behaviors that directly impact academic success.

I attend almost all of my classes.I complete the reading that is assigned to me.

Academic Self-Efficacy Belief in one’s ability to perform and achieve in an academic setting.

I'm confident that I will succeed in my courses this semester.I can do well in college if I apply myself.

MotivationCommitment to, drive toward, and perceived importance of academic success.

Degree Goals Clear goal commitment and plan of action to achieve post secondary goals

One of my life goals is to graduate college.The benefit of a college education outweighs the cost.

Institutional Commitment Attitudes toward and attachment to college or the institution.

This is the right school for me.I’m proud to say I attend this school.

Energy & Determination Propensity to work hard in attaining academic goals. Nothing can stop me from graduating from college.I push myself very hard to succeed.

Self-Management Able to anticipate and respond to pressure and stress related to college life.

Stress Sensitivity Tendency to react to daily hassles academic stress I get stressed out easily when things don't go my way.I am easily frustrated.

Coping Using adaptive strategies to deal with academic stressors or pressures.

When feeling stressed about the amount of homework I have, I try to get organized to get on top of my homework.When worried about a test, I try to get prepared for the test regardless.

Test Anxiety Negative reactions to the pressures and experience of academic testing.

When taking a test, I think about what happens if I don't do well.Before a test, my stomach gets upset.

Social SupportPerceived availability of resources to support academic success.

Connectedness A general sense of belonging and engagement in college life.

I feel connected to my peers.People understand me.

Institutional Support The ability to identify and use resources and supports that lead to success in college.

If I don't understand something in class, I ask the instructor for help.I know how to find out what's expected of me in classes.

Instrumental Support External factors that could enable or impede academic success.

Family pressures make it hard for me to commit to school.People close to me support me going to college.

Background Information: Communicate key student information from both SuccessNavigator and SIS to faculty/advisor.

Risk Indices: Separate indices for both classroom and enrollment success. Based on background, cognitive , and psycho-social information and supported by statistical relationships with success.

Domain Scores: Four general areas of student strengths and weaknesses. Scores are presented normatively.

Feedback: Determine by more specific “facet” scores (see next page).

Action Plans: Suggested interaction with programs and services on campus.

Background Information: Communicate key student information from both SuccessNavigator and SIS to faculty/advisor.

Risk Indices: Separate indices for both classroom and enrollment success. Based on background, cognitive , and psycho-social information and supported by statistical relationships with success.

Domain Scores: Four general areas of student strengths and weaknesses. Scores are presented normatively.

Feedback: Determine by more specific “facet” scores (see next page).

Action Plans: Suggested interaction with programs and services on campus.

Background Information: Communicate key student information from both SuccessNavigator and SIS to faculty/advisor.

Risk Indices: Separate indices for both classroom and enrollment success. Based on background, cognitive , and psycho-social information and supported by statistical relationships with success.

Domain Scores: Four general areas of student strengths and weaknesses. Scores are presented normatively.

Feedback: Determine by more specific “facet” scores (see next page).

Action Plans: Suggested interaction with programs and services on campus.

The “Success Navigator” - Purpose

• Low Stakes designed to assess the dispositions and noncognitive skills of incoming college students

• Can be used with or without academic markers (SAT, HS GPA, course placement score)

• Three primary intended uses:– Identifying likelihood of persistence to degree and

academic failure– Informing course placement decisions– Developmental Feedback for advising, FYE, etc

The “Success Navigator” - Logistics

• Length: 20-25 minute on-line assessment

• Items– Initially, ~120 self-report items– Incorporating anchoring vignettes and forced-choice items– Demographic and life event information

• Administration: Varies per institution: could be at home, during new student orientation, or in first-year experience course

Sample of Holistic Course Placement ChartExisting Reading Placement

COMPASS0 - 20 Conference Required

21 - 37 Foundational Studies 138 - 49 Foundational Studies 250 - 64 Reading 09965 - 79 Reading 125

80 and above Reading Not Required

Future Holistic Reading PlacementCOMPASS Success Navigator Course placement

0 - 20 NA Conference Required21 - 29 NA Foundational Studies 130-37 >= .8 Foundational Studies 238 - 42 NA Foundational Studies 243-49 >= .8 Reading 99

50 – 57 NA Reading 09958-64 >= .8 Reading 12565 - 72 NA Reading 12573-79 >= .8 Reading Not Required

80 and above Reading Not Required

• More accurate placement• Acceleration: Less time spent in developmental education• Help us get to know our incoming students early and

efficiently• More nuanced understanding of the whole student in a

meaningful way• Provide data to help develop placement and intervention

programs

Expected Outcomes of Holistic Placement

Part 2: Whole-Person Model of Early Alert

Whole-person model: 3 domains of assessment

Whole-student/Whole-person Early Alert Model

Math Readiness

Effort Level

High Medium Low

High .96 .91 --

Medium .94 .77 .82

Low .86 .61 .38

Note. N = 411. Success rates were not calculated for cells with insufficient sample sizes (denoted by a dash).

Research shows that behavioral monitoring and intervention are key to student success

Group interaction 2: 10 minutes• Form a group of 3-4• Look over the hand-outs.• Think about your students and your programs: What are ways

that you could best support the whole student on your campus? What collaborations across departments and units could you encourage?

• What are different ways that you can reach out to at-risk students?

• What are different ways that you can monitor students’ behaviors in order to intervene with a robust support program?

• Report out

Use the Whole Student Model to Drive Intervention Strategies

• Map support services to individual students’ needs based on non-cognitive scores

• Feedback loop to share non-cognitive report with students to arrive at action plans for students

• Consider other strategies to further student success, such as supplementary instruction, early alert and support other new and existing initiatives

[HAND-OUT for group exercise 2]

Comprehensive Early Interventions• Model and coach at-risk students toward a set of desirable behaviors.• Based on previous research (Li et al., 2012), low-level dev ed math students can

increase the likelihood of passing the course by 30% if they exhibit desirable in-class behaviors:– Active participation in groupwork (Student is actively engaged during group

work; helps other students with assignments; does his/her fair share of the work, etc)

– Active participation in lecture (Student's alert, attentive during class, asks/answers questions, etc)

– Attendance (Attends class, stays for whole period, etc)– Completion of homework assignments (Student completes assignments

thoroughly and completely, turns assignments in on time)

[HAND-OUT for group exercise 2]

Interventions: Yes we can!

• Three basic implications on early interventions:• As educators, we can motivate our students, shape and

reinforce their intentionality and academic determination. (Surrounding non-cognitive assessment)– Outward behaviors are indicators of inwards states such as

motivation and engagement– We have the opportunity and capability to coach our

students towards a set of desirable classroom/academic behaviors.

[HAND-OUT for group exercise 2]

Comprehensive Early Interventions• Map support services to non-cognitive psychological constructs:

Academic Skills, Self Management, Motivation, Social SupportNon-Cognitive construct Facet

Psycho-social Skills Refer Student to Existing natural fit support services at CCC

Academic Skills (Behaviors, beliefs and skills that facilitate success)

•Organization•Behavioral Compliance•Academic Self-Efficacy

Study skills Study skills workshop

Time management skills Study skills workshop

Self Management(Able to anticipate and respond to pressure and stress)

•Stress Sensitivity•Coping•Test Anxiety

Emotional control Wellness Center

Resilience Meeting with advisors, tutors and instructors

Motivation(Commitment to, drive towards, and perceived importance of success.)

•Degree Goals•Institutional Commitment•Energy & Determination

Degree intention Transfer center, advising

Class attendance Advisors, tutors, instructors

Engagement Advisors, tutors, instructors

Social support(Perceived availability of resources)

•Connectedness•Institutional Support•Instrumental Support

Help seeking New student orientations, advisors, service referrals

Peer support Student clubs and organizations

Recommendations: The 3 Pillars of Success

• Place students more accurately using multiple measures, including non-cognitive skills for possible acceleration

• Smarter, strategic interventions: coordinate efforts to support the whole student• Institutional commitment: Faculty, advisors, and staff collaborate in early alert

program• Generate fine-grained reporting to empower frontline workers and drive further

research and program development

Behavioral Factors

Instruction and Institutional Resources

Academic Readiness

ReferencesACT, Inc. (2002). Creating seamless educational transitions for urban African American and Hispanic students. Iowa City, IA: Author.Scott-Clayton, J. (2012). Do High-Stakes Placement Exams Predict College Success? Predicting Success in College: The Importance of

Placement Tests and High School Transcripts. CCRC Working Paper No. 41. New York: Community College Research Center, Teachers College, Columbia University.

ACT, Inc. (2004). Schools involving parents in early postsecondary planning. Iowa City, IA: Author.ACT, Inc. (2008). Student Readiness Inventory user’s guide. Retrieved from ACT, Inc. website:

http://www.act.org/sri/pdf/UserGuide.pdf

ACT, Inc. (2008). What we know about college success: Using ACT data to inform educational issues. Iowa City, IA: Author.

ACT, Inc. (2008). Student Readiness Inventory Toolshop. Retrieved from ACT, Inc. website: http://www.act.org/sri/studentguide/toolshop.html

Allen, J. & Robbins, S. (2010). Effects of interest-major congruence, motivation, and academic performance on timely degree attainment. Journal of Counseling Psychology, 57, 23-35.

Allen, J., Robbins, S., Casillas, A., & Oh, I.-S. (2008). Third-year college retention and transfer: effects of academic performance, motivation, and social connectedness. Research in Higher Education, 49(7), 647-664.

American Association of Community Colleges (AACC). (2009). Community college fast facts. Retrieved July 10, 2009 from http://www2.aacc.nche.edu/research/index.htm

Belfield, C.R., Crosta, P.M. (2012). Predicting Success in College: The Importance of Placement Tests and High School Transcripts. CCRC Working Paper No. 42. New York: Community College Research Center, Teachers College, Columbia University.

References (cont.)Berkner, L., & Choy, S. (2008). Descriptive summary of 2003-04 beginning postsecondary students: Three years later. Washington,

DC: U.S. Department of Education, Institute of Education Sciences.

Bragg, D. D. (2001). Community college access, mission and outcomes: Considering intriguing intersections and challenges. Peabody Journal of Education. 76(1), 93-116.

Community College Research Center. (2002a). Accountability and learning outcomes in community colleges. Retrieved July 7, 2008, from http://ccrc.tc.columbia.edu/Seminar.asp?uid=8

Community College Research Center. (2002b). Accountability and learning outcomes in community colleges. Retrieved July 7, 2008, 2008, from http://ccrc.tc.columbia.edu/Seminar.asp?uid=8

Habley, W. & McClanahan, R. (2004). What Works in student retention – All survey colleges. Iowa City, IA: ACT, Inc.Harris, S. (1991) Can’t you guys read? Cartoons on academia. New Brunswick, NJ: Rutgers University Press.Herzog, K. & Miller, M. P. (Eds.). (1985). Scholarship: More great cartoons from the Kappan. Bloomington, IN: Phi Beta Kappa

International Inc.Le, H., Casillas, A., Robbins, S., & Langley, R. (2005). Motivational and skills, social, and self-management predictors of college

outcomes: Constructing the Student Readiness Inventory. Educational and Psychological Measurement, 65, 482-508.Li, K, & Cordes, C. (2010). Individualizing the Delivery of Math Developmental Education. Talk presented at the 34th Annual National

Association of Developmental Education Conference, Columbus, OH.Lotkowski, V., Robbins, S., & Noeth, R. (2004). The role of academic and non-academic factors in improving college retention.

Retrieved from ACT, Inc. website: www.act.org/research/policy/index.html. McCabe, R.H. (2000). No one to waste. Washington, DC: Community College Press.NCES (2004). Remedial education at degree-granting postsecondary institutions in fall 2000 (NCES 2004-010). NISOD (2012). Assessing College-readiness assessments. Webinar delivered on 3/28/2012.

References (cont.)Pascarella, E. T., & Terenzini, P. T. (2005). How college affects students: A third decade of research. San Francisco: Jossey-Bass.Peterson, C. H., Casillas, A., & Robbins, S. B. (2006). The Student Readiness Inventory and the Big Five: Examining social

desirability and college academic performance. Personality and Individual Difference, 41, 663-673.Porchea, S., Allen, J., Robbins, S., Phelps, R. (in press). Predictors of long-term enrollment and degree outcomes for community

college students: Integrating academic, psychosocial, socio-demographic, and situational factors. Journal of Higher Education.

President’s Commission on Higher Education. (1947). Organizing higher education. Washington, DC: Author.Robbins, S. B., Allen, J. Casillas, A., Akamigbo, A., Saltonstall, M., Campbell, R., Mahoney, E. & Gore, P.A. (2009). Associations

of resource and service utilization, risk level, and college outcomes. Research in Higher Education, 50(1), 101-118. Robbins, S., Allen, J., Casillas, A., Peterson, C., & Le, H. (2006). Unraveling the differential effects of motivational and skills, social,

and self-management measures from traditional predictors of college outcomes. Journal of Educational Psychology, 98, 598-616.

Robbins, S. B., Lauver, K., Le, H., David, D., Langley, R., & Carlstrom, A. (2004). Do psychosocial and study skill factors predict college outcomes? A meta-analysis. Psychological Bulletin, 130, 261-288.

Robbins, S., Oh, I., Le, H., & Button, C. (2009). Intervention effects on college performance and retention as mediated by motivational, emotional, and social control factors: Integrated meta-analytic path analyses. Journal of Applied Psychology, 94, 1163-1184.

Swail, W. S. (2004, January 23). Legislation to improve graduation rates could have the opposite effect. The Chronicle of Higher Education, B16.

Refining Course Placement Practices and Driving toward

Student Success using a Holistic Model

Correspondence regarding this presentationshould be addressed to:

Steve Robbins, Director, Research Innovation, ETS srobbins@ets.org

Kevin Li, Dean of Instruction Wilbur Wright College,

One of the City Colleges of Chicagokli@ccc.edu