2b c6 common assessment - accca conference/2014/2b c6... · 2014. 3. 3. · c6 development process...

Developing a Developing a Next Generation Common Placement Next Generation Common Placement Assessment for the C6 ConsortiumAssessment for the C6 Consortium

1

Assessment for the C6 ConsortiumAssessment for the C6 Consortium

Barbara Hioco, Ed.D, CVHECRyen Hirata, Reedley College-Willow InternationalSaundra McGlothlin, MSW, West Hills College CoalingaMai Meidinger, Merced CollegeMatthew Schultz, PhD, McCann Associates

I. Significance of Common AssessmentII. Historical Overview & Background

III. The Drawing Board: Guidelines

Presentation OverviewPresentation Overview

III. The Drawing Board: Guidelines

IV.The Process

V. Conclusion

Historical Overview & BackgroundHistorical Overview & Background

3

The Process of Developing a Next Generation Common Placement Assessment

for the C6 Consortium

▶ Bakerfield*▶ Cerro Coso*▶ College of the Sequoias▶ Columbia▶ Fresno City*▶ Merced*▶ Modesto*

C6 ConsortiumC6 Consortium

▶ Modesto*▶ Porterville*▶ Reedley – Willow Intl*▶ SJ Delta*▶ Taft▶ WHC Coalinga*▶ WHC Lemoore*

*Identify active participants

�Address Central Valley Colleges’ placement test concerns

�New state legislation and mandates

Why Common Assessment? Why Common Assessment?

�New state legislation and mandates

�No definitive timeline for a statewide common assessment (at that time)

What are Current Test Challenges?What are Current Test Challenges?�Inaccurate placement of students

�High costs

�Not compatible with college data systems

�No diagnostic test

�No college curriculum alignment

�No common placement between sites/colleges

�Little to no support with cut score setting

�Overall, colleges are not satisfied with current test

▶ Community colleges in the central valley

▶ Instructional faculty (subject matter experts)

CollaborationCollaboration

▶ Assessment/Matriculation specialists

▶ Administration

▶ Test vendor

The Drawing Board: GuidelinesThe Drawing Board: Guidelines

8

The Process of Developing a Next Generation Common Placement Assessment

for the C6 Consortium

Drawing BoardDrawing Board

� Framework

� Development process

Functionality� Functionality

� Adaptability

� Common Cut

Test FrameworkTest Framework▶Web-based▶ Adaptive & diagnostic▶ Accessible: individuals with disabilities▶ Timeline goal: 2 hours▶ Common cut scores (Uh Oh!)▶ Adaptability (How?) ▶ Price point▶ Data warehouse

Development process typically used by vendors producing test for CCC.

Designed to be “off the shelf” tests

Primarily Adaptive (diagnostic available for additional cost)

Diagnostic typically not part of course placement

Current Vendor Development ProcessCurrent Vendor Development Process

Subject matter solely determined by test developer (really unknown to faculty/staff/college)

Developed to address general subject matter and High School standardized tests/core standards

No commonality: All cut scores vary between colleges

Background questions set by test developer and are customizable

C6 created a new development process based on faculty input and community college curriculum.

Placement test will include adaptive and diagnostic test portions

Adaptive and diagnostic will be used to determine course placement

Subject matter reviewed and selected by faculty within specified discipline

C6 Development ProcessC6 Development Process

Subject matter reviewed and selected by faculty within specified discipline (i.e., English-R/W, Math, and ESL)

Developed based on community college curriculum standards (C-ID) and course objectives

Common cut scores for transfer level English and degree level math

Background questions developed by CC to have standard and set specific questions

C6 Test FunctionalityC6 Test Functionality

▶Score availability, portability, comparability

▶Web-based

▶Reduce self-selection of appropriate test

▶Seamless transition ▶Seamless transition

▶Accessible for all students

▶Compatible with college data systems

▶Provides data for reports (i.e., MIS, SS, & validation)

Test Common FeaturesTest Common Features

▶The test questions, criteria, scoring

▶The required background questions

▶The required data on results page▶The required data on results page

▶Common cut scores (transfer level English; degree applicable math only)

Test CustomizationTest CustomizationWhat can be customized?

▶Slide background design

▶Additional background questions

▶Information & comments on results page▶Information & comments on results page

▶Local cut scores (except transfer level English & degree applicable math)

Cut ScoresCut Scores▶ Determined by C6 faculty

▶ Common cut scores: Transfer level English and degree level math

▶ Common cut scores accepted by all participating colleges

▶ Local cut scores determined by individual colleges to align with local curriculum

The ProcessThe Process

17

The Process of Developing a Next Generation Common Placement Assessment

for the C6 Consortium

Determine AssignmentsDetermine AssignmentsC6 & College Faculty/Staff/ Admin. McCann

Utilize Faculty/Staff input for test development

Provide a test Platform: “College Success”

Identify objectives and test items related to curriculum objectives (C-ID)

Provide test question banks

Determine required background questions to be used as standard

Create a new test based on C6 needsto be used as standard

Determine common cut scores for transfer level English and associate degree level Math

Provide automated scoring for test, including AET (Automated Essay Scoring)

Collect and house test data

1. Identify project guidelines

2. Create test flow

3. Identify adaptable test components (Blue-Prints)

4. Identify diagnostic test components (Blue-Prints)

5. Compose a working draft of test

Project OutlineProject Outline

6. Collect data and analyze

7. State Chancellor’s Office approval

8. Establish common cut scores & local cut scores

9. Pilot test

10. Continuous field testing & psychometric monitoring

� 2012▶ C6 began discussions with McCann

▶ C6 Open forum

▶ C6 Placement test development workshop

Project ActivitiesProject Activities

� 2013▶ Established lead faculty to plan/coordinate pilot project

▶ Pilot project test development

▶ Pilot test platform developed by McCann

▶ Data collection and review

Test Flow & Blue Print DevelopmentTest Flow & Blue Print Development

English/ESL Writing/Reading

Math

� Faculty/Staff driven

� Aligned with curriculum/course (C-ID)

Test ComponentsTest ComponentsTest Subject Method of Test Descriptors

Adaptive Engl/ESL Writing Essay Writing Sample Utilize Automated Essay Scoring (AES) Technology

Engl/ESL Reading Multiple Choice • 300+ items per bank

• Items calibrated • Items calibrated using 3 parameter IRT Math Multiple Choice

Diagnostic ESL Writing Mutiple Choice Provides score reporting on competency of specific objectives

Engl/ESL Reading Multiple Choice

Math Multiple Choice

Test SequenceTest Sequence

ADAPTIVE TEST

LINEAR DIAGNOSTIC TEST

PLACEMENT

C6 Test: Latest VersionC6 Test: Latest Versionhttp://college.measuredsuccess.com/

24

What Next?What Next?

� Data collection for validation

� Submit for approval from the Chancellor’s Office

� Set cut scores

� Continuous field testing & ongoing psychometric monitoring

ConclusionConclusion

29

The Process of Developing a Next Generation Common Placement Assessment

for the C6 Consortium

Test Comparison: Current vs. C6Test Comparison: Current vs. C6

Characteristics of current products Characteristics of C6

All colleges use different placement test products

All colleges can use a common placement test product

Primarily Adaptive (diagnostic available for additional cost)

Placement test will include adaptive and diagnostic test portions

Diagnostic typically not part of course Adaptive and diagnostic will be used to Diagnostic typically not part of course placement

Adaptive and diagnostic will be used to determine course placement

Subject matter solely determined by test developer (really unknown to faculty/staff/college)

Subject matter reviewed and selected by faculty within specified discipline (i.e., English-R/W, Math, and ESL)

Developed based on general subject matter and High School standardized tests/core standards

Developed based on community college curriculum standards (C-ID) and course objectives

No commonality: All cut scores vary between colleges

Common cut scores for transfer level English and degree level math

Background questions set by test developer

Background questions developed by CC to have standard and site specific

BENEFITS OF C6 PROJECT

Faculty

drive

Utilized the

Creation of

Provide a

uniform

Uniform

interface for

Provide

Facilitate

Improve

availability, portability,

Vision:

Utilize our drive

n test developme

nt

the expertise of professionals

on of a new test

model

m assessmen

t instrument

for stude

nt testing and administration

de faculty with class data

ate cost savin

gs

bility, comparability of data

between

colleges

e our experience

& expertise

32

The Process of Developing a Next Generation Common Placement Assessment

for the C6 Consortium