ETS Proficiency Profile Test Pilot Summary Report

Pima Community College, Planning and Institutional Research

Spring 2012

Page 1 of 12

Executive Summary

As part of the College’s ongoing efforts to directly measure its student’s performance in general education proficiency,

Pima has adopted the ETS Proficiency Profile Test to begin standardized testing across the district.

The ETS Proficiency Profile Test is an exam intended to measure general education outcomes at institutions of higher

education. The test is specifically intended to measure four skill areas: critical thinking, reading, writing and

mathematics. According to ETS, the Proficiency Profile Test allows a college or university to: a) gain a full perspective of

the effectiveness of your general education program to meet requirements for accreditation and performance funding,

b) inform teaching and learning with actionable score reports you can use to pinpoint strengths and areas of

improvement, c) take your institution to the next level by providing comparative data on more than 400 institutions and

over 500,000 students nationwide, and d) create greater flexibility in your testing program by adding 50 locally authored

questions and using both paper-and-pencil and online formats, as well as choosing to use proctored and non-proctored

versions. The Proficiency Profile Test is administered to different student cohorts as a means of institutional

effectiveness assessment and for funding and accreditation purposes1.

ETS at Pima Community College

After examining different ways of directly measuring student performance, Pima administrators decided to pilot the ETS

Proficiency Profile Test in spring 2012. Pima chose the ETS Proficiency Profile Test for several reasons. First, ETS is a well-

known and well-trusted testing service provider. Second, the specific test is well-suited for the needs of the College, the

faculty and the student body. Third, the College had previously administered the ETS Academic Profile (an older ETS test

which measured performance in general undergraduate core curricula) which will consequently allow the college to

compare the results from the ETS Proficiency Profile Test to the ETS Academic Profile Pima in order to render a

longitudinal representation of the College’s performance based on direct assessment measures. The Planning and

Institutional Research Department (PIR) of the College was/ is responsible for the implementation of this pilot project.

Pima elected to use the forty-minute (compared to the two-hour), abbreviated version of the test to mitigate impact on

normal classroom activities. ETS uses Criterion-referenced scores for the abbreviated version of the test. Criterion-

referenced scores measure proficiency levels associated with specific skill sets in mathematics, writing and reading,

creating proficiency classifications. These scores are generally used for measuring individual student progress as well as

curriculum and program improvement.

PIR created a random sample to test one-hundred courses at or above the two-hundred level. The courses chosen were

from multiple disciplines across all of the Pima campuses. The test was administered via two delivery methods. The first

method was a proctored, paper-and-pencil, face to face administration. The second method was an unproctored online

administration. The test administration period spanned from 02/10/2012-0310/2012.

Spring 2012 Test Details Number of students tested: 1121 Number of students included in these statistics: 1027 Number of students excluded: 94 (These students were excluded because they completed less than 75% of the test.)

1 http://www.ets.org/proficiencyprofile/about/

Page 2 of 12

Summary of Scaled Scores

Possible Range

Mean Score

95% Confidence Limits2 for Mean

Standard Deviation

25th Percentile

50th Percentile

75th Percentile

Total Score 400-500 438.94 438-440 18.83 425 435 450

Skills Subscores

Critical Thinking

100-130 110.83 110-112 6.07 106 110 115

Reading 100-130 116.49 116-117 6.78 112 116 121

Writing 100-130 112.79 112-114 4.81 109 113 115

Mathematics 100-130 112.09 111-113 6.05 108 111 116

Context-Based Subscores

Humanities 100-130 114.01 113-115 6.34 109 113 118

Social Sciences

100-130 112.60 112-113 6.06 109 112 116

Natural Sciences

100-130 114.29 114-115 5.69 111 114 118

2 The confidence limits are based on the assumption that the questions contributing to each scaled score are a sample from a much larger set of possible questions

that could have been used to measure those same skills. If the group of students taking the test is a sample from some larger population of students eligible to be tested, the confidence limits include both sampling of students and sampling of questions as factors that could cause the mean score to vary. The confidence limits indicate the precision of the mean score of the students actually tested, as an estimate of the "true population mean" - the mean score that would result if all the students in the population could somehow be tested with all possible questions. These confidence limits were computed by a procedure that has a 95 percent probability of producing upper and lower limits that will surround the true population mean. The population size used in the calculation of the confidence limits for the mean scores in this report is 1027.

Page 3 of 12

Scaled Score Distribution by Context (Subject Area)

Page 4 of 12

Total Scaled Score Distribution

Page 5 of 12

Summary of Proficiency Classifications3 (shows the percentage of students proficient at each level4)

Note: Criterion-referenced scores (proficiency classifications) measure the level of proficiency students obtained on a certain skill set. These scores have meaning of their own — without other scores to compare against — but can be used to make comparisons between students or groups of students. The ETS® Proficiency Profile provides criterion-referenced scores of proficient, marginal and not proficient for three skill areas: reading/critical thinking, writing and mathematics.

Demographic Analysis Reports Age:

3 The skills measured by the ETS® Proficiency Profile test are grouped into proficiency levels - three proficiency levels for writing, three for mathematics, and three for the combined set of skills involved in reading and critical thinking. The table and graph show the number and percentage of students who are proficient, marginal, and not proficient at each proficiency level in reading and critical thinking, writing, and mathematics. A student classified as marginal is one whose test results do not provide enough evidence to classify the student either as proficient or as not proficient. See the User's Guide for more information about these classifications, including a list of the specific skills associated with each proficiency level in each skill area. 4 Explanations describing skills by level can be found in Appendix 1

Page 6 of 12

Best Language:

Credit Hours:

Curriculum Completed:

Enrollment Status:

Page 7 of 12

Ethnicity:

Gender:

GPA:

Hours Working:

Page 8 of 12

Major:

Page 9 of 12

Program Enrolled:

Transfer Status:

Supplemental Questions (Created by Pima Community College) Question 1: What is your reason for attending PCC? 0. PCC degree or certificate 1. Transfer to a four year institution 2. Co-enrolled at UA 3. Job skills 4. Personal interest

0 1 2 3 4 5 6 7 8 9

302 (27%) 595 (53%) 93 (8%) 23 (2%) 35 (3%) 0 (0%) 0 (0%) 0 (0%) 0 (0%) 0 (0%)

No Response: 73 (7%) Question 2: Do you anticipate achieving your educational goal this year? 0. Yes 1. No 2. Not sure

0 1 2 3 4 5 6 7 8 9

518 (46%) 423 (38%) 110 (10%) 1 (0%) 0 (0%) 0 (0%) 0 (0%) 0 (0%) 0 (0%) 0 (0%)

No Response: 69 (6%)

Page 10 of 12

Question 3: How long have you been taking courses at PCC? 0. Less than 1 year 1. 1 year or longer, but less than 2 years 2. 2 years or longer, but less than 3 years 3. 3 years or longer, but less than 4 years 4. 4 years or longer, but less than 5 years 5. 5 years or longer, but less than 6 years 6. 6 years or longer 7. Not sure

0 1 2 3 4 5 6 7 8 9

263 (23%) 252 (22%) 244 (22%) 154 (14%) 52 (5%) 23 (2%) 46 (4%) 17 (2%) 0 (0%) 0 (0%)

No Response: 70 (6%) Question 4: Do you consider yourself to be Hispanic/Latino? 0. Yes 1. No 2. Prefer not to answer

0 1 2 3 4 5 6 7 8 9

411 (37%) 601 (54%) 38 (3%) 0 (0%) 2 (0%) 0 (0%) 0 (0%) 0 (0%) 0 (0%) 0 (0%)

No Response: 69 (6%) Question 5: In addition, select one or more of the following racial categories to describe yourself: 0. American Indian or Alaska Native 1. Asian 2. Black or African American 3. Native Hawaiian or Pacific Islander 4. White

0 1 2 3 4 5 6 7 8 9

47 (4%) 48 (4%) 58 (5%) 16 (1%) 692 (62%) 5 (0%) 1 (0%) 0 (0%) 0 (0%) 2 (0%)

No Response: 252 (22%) Question 6: Please select from the following age ranges: 0. 13-17 1. 18-19 2. 20-21 3. 22-24 4. 25-29 5. 30-39 6. 40-49 7. 50-59 8. 60-100 9. Prefer not to answer

0 1 2 3 4 5 6 7 8 9

3 (0%) 201 (18%) 286 (26%) 173 (15%) 154 (14%) 118 (11%) 53 (5%) 27 (2%) 7 (1%) 8 (1%)

No Response: 91 (8%)

Page 11 of 12

Appendix 1: ETS Explanation of Proficiency Levels

Skills tested at each level (ETS User’s Guide, pages 9-11) Reading and Critical Thinking To be considered proficient at Level 1, a student should be able to:

recognize factual material explicitly presented in a reading passage

understand the meaning of particular words or phrases in the context of a reading passage To be considered proficient at Level 2, a student should be able to:

synthesize material from different sections of a passage

recognize valid inferences derived from material in the passage

identify accurate summaries of a passage or of significant sections of the passage

understand and interpret figurative language

discern the main idea, purpose of focus of a passage or a significant portion of the passage To be considered proficient at Level 3, a student should be able to:

evaluate competing causal explanations

evaluate hypotheses for consistency with known facts

determine the relevance of information for evaluating an argument or conclusion

determine whether an artistic interpretation is supported by evidence contained in a work

recognize the salient features or themes in a work of art

evaluate the appropriateness of procedures for investigating a question of causation

evaluate data for consistency with known facts, hypotheses or methods Writing To be considered proficient at Level 1, a student should be able to:

recognize agreement among basic grammatical elements (e.g., nouns, verbs, pronouns and conjunctions)

recognize appropriate transition words

recognize incorrect word choice

order sentences in a paragraph

order elements in an outline To be considered proficient at Level 2, a student should be able to:

incorporate new material into a passage

recognize agreement among basic grammatical elements (e.g., nouns, verbs, pronouns and conjunctions) when these elements are complicated by intervening words or phrases

combine simple clauses into single, more complex combinations

recast existing sentences into new syntactic combinations To be considered proficient at Level 3, a student should be able to:

discriminate between appropriate and inappropriate use of parallelism

discriminate between appropriate and inappropriate use of idiomatic language

recognize redundancy

discriminate between correct and incorrect constructions

recognize the most effective revision of a sentence

Page 12 of 12

Mathematics To be considered proficient at Level 1, a student should be able to:

solve word problems that would most likely be solved by arithmetic and do not involve conversion of units or of proportionality (These problems can be multi-step if the steps are repeated rather than embedded.)

solve problems involving the informational properties of numbers and operations, often involving the Number Line, including positive and negative numbers, whole numbers and fractions (including conversions of common fractions to percent, such as converting ¼ to 25%)

solve problems requiring a general understanding of square roots and the squares of numbers

solve a simple equation or substitute numbers into a algebraic expression

find information from a graph (This task may involve finding a specified piece of information in a graph that also contains other information.)

To be considered proficient at Level 2, a student should be able to:

solve arithmetic problems with some complications, such as complex wording, maximizing or minimizing and embedded ratios (These problems include algebra problems that can be solved by arithmetic [the answer choices are numeric].)

simplify algebraic expressions, perform basic translations and draw conclusions from algebraic expressions and inequalities (These tasks are more complicated than solving a simple equation, though they may be approached arithmetically by substituting numbers.)

interpret a trend represented in a graph, or choose a graph that reflects a trend

solve problems involving sets (The problems would have numeric answer choices) To be considered proficient at Level 3, a student should be able to:

solve word problems that would be unlikely to be solved by arithmetic; the answer choices are either algebraic expressions or are numbers that do not lend themselves to back-solving

solve problems involving difficult arithmetic concepts such as exponents and roots other than squares and square roots and percent of increase or decrease

generalize about numbers, e.g., identify the values of (x) for which an expression increases as (x) increases

solve problems requiring an understanding of the properties of integers, rational numbers, etc.

interpret a graph in which the trends are to be expresses algebraically or in which one of the following is involved: exponents and roots other than squares and square roots, percent of increase or decrease

solve problems requiring insight or logical reasoning