ets proficiency profile summary results - florida a&m ... report (final).pdf · ets proficiency...

ETS Proficiency Profile

Summary Results

Prepared by:

Office of University Assessment

Florida Agricultural and Mechanical University

1

Table of Contents

Executive Summary ...................................................................................................................................... 2

Introduction .................................................................................................................................................. 4

Overview of ETS Proficiency Profile ......................................................................................................... 4

Historical Overview of ETS Proficiency Profile at FAMU ......................................................................... 5

Guiding Questions .................................................................................................................................... 5

Methodology ................................................................................................................................................ 6

Findings ......................................................................................................................................................... 6

Trends in Incoming Freshmen Performance ............................................................................................ 6

Trends in Graduating Seniors Performance............................................................................................. 9

EPP Freshmen Comparative Group Analysis ......................................................................................... 13

EPP Graduating Seniors Comparative Group Analysis .......................................................................... 17

Recommendations for Improvement ........................................................................................................ 20

FAMU EPP Top Performers ........................................................................................................................ 21

Limitations of Report ................................................................................................................................. 21

APPENDIX ................................................................................................................................................... 23

2

Executive Summary The ETS Proficiency Profile (EPP) is a General Education knowledge and skills test that is designed to

measure critical thinking and college-level reading, writing, and mathematical skills in the contexts of

the humanities, social sciences, and natural sciences. In an effort to evaluate General Education

learning outcomes, the University currently requires students to take the EPP at entry (freshmen) and

exit (senior) levels. The EPP is administered twice a year, to incoming freshmen in the fall

semester and to graduating seniors in the spring semester. Within the past six years (fall 2006 –

spring 2012), (n=2,918) freshmen and (n=2,100) seniors have participated in the EPP test.

In fall 2013, (n=334) incoming freshmen attempted to take the EPP. Results were calculated for

(n=308) of those students who completed 75% or more of the test items. The total mean score for

the incoming freshmen was (M=431.68, SD=17.20) based on a 400-500 range. The results revealed

that one student received a perfect score of 500 among the five top performers on the test. In spring

2014, (n=600) out of (N=981) graduating seniors took the EPP. Results were calculated for (n=537)

students who completed 75% or more of the test. Results revealed that the total mean score for the

graduating seniors was (M=430.56, SD=18.31) based on a 400-500 range.

A comparative analysis of Florida Agricultural and Mechanical University (FAMU) scores to those of

the ETS provided National Averages, Carnegie classification of similar institutions, Historically Black

Colleges and Universities (HBCUs) taken as group, and institutions in the State University System

(SUS) of Florida to include institutional and aspirational peers, revealed that the average scaled scores

for both incoming freshmen and graduating seniors were below the national average and the group

of institutions in the SUS who participated in the test. In both instances, FAMU incoming freshmen

and graduating seniors’ overall mean scaled scores were above HBCUs taken as a group. See

summary table below for a comparative synopsis of the results for incoming freshmen and graduating

seniors.

3

Comparative Aggregate Summary of Student Performance on ETS Proficiency Profile for period (2009-2014)

Skill Dimension

National Carnegie HBCU SUS/Inst./

Aspirational

FAMU

M SD M SD M SD M SD M SD

Incoming Freshmen Total Score

432.7 17.6 429.0 15.3 427.6 14.6 441.8 19.2 430.8

16.0

Graduating Seniors Total Score

443.5 21.0 437.9 19.5 431.9 17.6 447.3 21.9 433.2

18.9

Note: All numbers are rounded to 1 decimal place.

The results of the ETS Proficiency Profile continues to confirm that our students are entering the

university with significant deficiencies in all skill areas as reported by the proportion of incoming

freshmen who were classified as proficient on the test. Equally concerning, is the proportion of

graduating seniors who were classified as proficient in the skill areas measured by the test. Since

2009 less than 55% of FAMU graduating seniors were classified as proficient in the skill areas

measured by the test. Additionally, less than 10% of graduating seniors completing the test were

scored proficient in Critical Thinking, Writing Level 3 and Mathematics Level 3. It is clear that

more emphasis across degree programs should be placed on remediating and

developing/enhancing the competencies measured by the test. In general education courses in

addition to upper division courses, these skill sets should be emphasized as a way of addressing

what students should know and/or be able to do at the end of the degree program.

There is a great deal of information presented in this report and it is our hope that you will find the

report meaningful and empowering in efforts to improve the academic performance of our students.

4

Introduction

The 2013-2014 Office of University Assessment (OUA) ETS Proficiency Profile (EPP) Summary Report was

prepared to provide you insights into the performance of Florida A&M University students (i.e. incoming

freshmen and graduating seniors) on this general education knowledge and skills test. The EPP is

designed to measure critical thinking and college-level reading, writing, and mathematical skills in the

contexts of the humanities, social sciences, and natural sciences. This report is divided into several key

areas which will provide you key insights into the performance of FAMU students employing data from

2006 through 2014.

Overview of ETS Proficiency Profile

The ETS Proficiency Profile is designed by the Educational Testing Service (ETS) as a measure of college-

level reading, mathematics, writing, and critical thinking in the context of the humanities, social sciences,

and natural sciences. Colleges and universities across the United States (US) use the test to assess general

education outcomes. The ETS test provides invaluable data for accreditation, strategic planning,

curriculum improvement, performance-based funding, benchmarking, and for determining “value added"

or learning gains. The test is also used by the Voluntary System of Accountability (VSA) for the College

Portrait, which allows stakeholders (i.e. potential students, parents, university systems, accrediting bodies

etc.) to access and compare the performance of students on the test from across participating institutions

in the US.

Questions on the EPP are multiple choice and are arranged in blocks of three to eight. Each section test

the same types of skills. This integrated design prevents a particular skill area from appearing all at once

or late in the test when fatigue can affect student performance. The total score on the test is reported

on a scale of 400-500. There are seven sub-scores that are reported on a scale of 100-130 for each skill

area (i.e. Critical Thinking, Reading, Writing, Mathematics, Humanities, Social Sciences, and Natural

Sciences). In addition to a total score, institutions receive a proficiency classifications (i.e. proficient,

marginal or not proficient) for each skill level identified simply as Level 1, Level 2 and Level 3 based on

students’ performance taken as a group in each skill dimension. Please refer to Appendix A for a

comprehensive overview of each proficiency level which is defined in terms of competencies expected of

students. Proficiency classifications capture how well students have mastered each skill area.

Scaled scores are computed by means of a two-stage process.

5

The student's "raw score" is computed by counting the number of questions the student

answered correctly (there is no penalty for incorrect guessing).

A raw-to-scale conversion table converts the raw score to a scaled score.

The scaled scores are norm-referenced, i.e., intended for comparing individual students with a group and

for comparing a group of students with other groups of students. These comparisons can be entirely

within the institution, or they can involve data from other institutions.

Historical Overview of ETS Proficiency Profile at FAMU

In an effort to evaluate General Education outcomes at FAMU, all incoming freshmen and graduating

seniors are asked to participate in the test. Participation is critical in helping the University to evaluate

general education outcomes. The test was first administered at FAMU in 2006. The University has since

participated in the test on an annual basis. Starting in 2009 every fall and spring semester, incoming

freshmen and graduating seniors respectively, participate in the test. For freshmen, the test is

administered one week before classes begin in the fall and remains open until the end of September.

Freshmen are solicited for participation in the test by means of email, FAMUINFO, Summer Reading Book

Review Sessions, Resident Advisors in on-campus housing facilities, and freshmen orientation courses.

Similarly, the test is administered to graduating seniors in the spring. The test is administered the week

following spring break and remains open until commencement. Graduating students are solicited for

participation in the test my means of: email, FAMUINFO, correspondence from Deans, Department Chairs,

Institutional Level Assessment Committee (ILAC) members, and the Registrar’s office.

Guiding Questions

The preparation of this report was guided by the follow questions:

1. How have incoming freshmen at FAMU performed on the ETS Proficiency Profile?

2. How have graduating seniors at FAMU performed on the ETS Proficiency Profile?

3. How does the performance of incoming freshmen at FAMU compare to the National

Average, institutions in our Carnegie Classification, HBCU’s, and institutions in the State University

System of Florida taken as a group?

4. How does the performance of graduating seniors at FAMU compare to the National Average,

institutions in our Carnegie Classification, HBCU’s, and institutions in the State University System

of Florida taken as a group?

6

Methodology

Descriptive and inferential techniques were employed in addressing each of the guiding questions which

informed the preparation of this report.

Findings

This section of the report highlights the findings for each of the guiding questions which informed the

preparation of this report.

Trends in Incoming Freshmen Performance

Guiding Question 1: How have incoming freshmen at FAMU performed on the ETS Proficiency Profile?

Descriptive techniques were employed in addressing this question. Specifically, mean and standard

deviation of scores were computed for each skill dimension.

In fall 2013, (n=334) incoming freshmen attempted to take the EPP test. Results were calculated for

(n=308) of those students who completed 75% or more of the test items. Table 1 provides a summary of

the results for the scaled scores. As can be discerned from Table 1, in 2013 the mean overall score for

incoming freshmen was (M=431.68, SD=17.20). The means overall performance of FAMU freshmen was

significantly below the national average for 2013 (M=447.89, SD=8.87). Similarly, the mean performance

of FAMU freshmen in each skill dimension fell below the national average. Please refer to Table 1 for a

summary of these results.

Table 1 FAMU Incoming Freshmen Average Scaled Scores for 2013

Possible Range

of Scores FAMU 2013

National Average & Standard Deviation

Difference Indicator

M SD M SD

Total Score 400 to 500 431.68 17.20 435.31 10.18

Critical Thinking 100 to 130 107.97 5.36 108.70 2.56

Reading 100 to 130 113.57 7.10 114.30 3.07

Writing 100 to 130 112.43 5.32 112.20 2.14

Mathematics 100 to 130 110.97 5.06 110.50 2.82

Humanities 100 to 130 113.17 6.13 112.80 2.39

Social Sciences 100 to 130 110.29 5.99 111.00 2.45

Natural Sciences 100 to 130 111.54 5.86 112.60 2.48 FAMU Mean Performance below National Average FAMU Mean Performance above National Average

7

A review of the trends associated with the performance of FAMU incoming freshmen on the EPP revealed much variability in overall mean

performance. Table 2 provides a comprehensive overview of the performance of FAMU incoming freshmen on the EPP spanning 2006-2013.

Consistent with the findings previously discussed for 2013, since 2006 the mean performance of FAMU freshmen has fallen below the national

average (see Figure 1).

Table 2 Trends in Incoming Freshmen Average Performance on ETS Proficiency Profile for 2006-2013

Skill Dimension

2006 2007 2008 2009 2010 2011 2012 2013

N =124 N =714 N =142 N =613 N =751 N =508 N =354 N =308

Mean SD Mean SD Mean SD Mean SD Mean SD Mean SD Mean SD Mean SD

Total Score* 431.20 12.70 436.16 14.40 434.8 17 430.41 14.15 429.67 16.23 431.61 16.07 431.26 17.75 431.68 17.20

Critical Thinking

108.40 4.70 109.38 5.18 108.85 5.45 108.08 5.13 107.56 4.97 108.28 5.23 107.65 5.35 107.97 5.36

Reading 114 5.50 116.24 5.89 115.40 6.11 113.59 6.12 113.07 6.78 113.52 7.12 113.54 7.16 113.57 7.10

Writing 112 4.15 112.91 4.23 113.67 4.76 112.17 4.58 111.95 5.26 112.36 4.96 112.12 5.15 112.43 5.32

Mathematics 110.10 4.58 111.22 5.01 110.04 5.63 109.88 4.73 110.49 5.12 110.77 4.88 111.15 5.33 110.97 5.06

Humanities 111.30 4.72 113.37 5.66 112.5 5.96 111.62 5.60 112.43 5.92 112.57 6 112.64 6.17 113.17 6.13

Social Sciences

110.80 4.57 111.88 5.15 111.37 5.41 110.51 5.18 110.03 5.75 110.72 5.84 110.42 5.54 110.29 5.99

Natural Sciences

112.20 5.20 113.48 5.09 113.01 5.06 111.96 5.10 111.17 5.80 111.91 5.61 111.05 6.22 111.54 5.86

8

Figure 1 Historical Overview of FAMU Freshmen Performance on ETS Proficiency Profile Compared to National Average

A review of each skill dimension measured on the ETS Proficiency Profile revealed that since 2009, less

than 41% of incoming freshmen who participated in the test were classified as proficient. The results of

all skill dimensions measured by the test suggest serious concerns as it relates to the preparedness of

incoming students entering FAMU. Please refer to Table 3 for a comprehensive overview of these results.

Table 3 Trends in Incoming Freshmen Classified as Proficient (2006-2013)

Proficiency Levels 2006 2007 2008 2009 2010 2011 2012 2013

N=124 N=714 N=142 N=613 N=751 N=508 N=354 N=308

Reading, Level 1 41% 55% 51% 34% 29% 32% 32% 31%

Reading, Level 2 13% 23% 20% 12% 10% 12% 12% 13%

Critical Thinking 1% 2% 3% 1% 1% 1% 3% 2%

Writing, Level 1 43% 52% 53% 39% 35% 34% 37% 40%

Writing, Level 2 7% 10% 19% 8% 9% 9% 10% 11%

Writing, Level 3 2% 3% 7% 2% 3% 4% 3% 3%

Mathematics, Level 1 37% 42% 29% 28% 28% 28% 29% 32%



2006 2007 2008 2009 2010 2011 2012 2013 2014

Mean Score* 431.19 436.16 434.8 430.41 429.67 431.61 431.26 431.68 429.13

National Avg. 440.50 440.50 440.50 440.50 440.50 440.50 435.31 435.31 435.31

422

424

426

428

430

432

434

436

438

440

442M

ean

Sca

led

Sco

re

9

Trends in Graduating Seniors Performance

Guiding Question 2: How have graduating seniors at FAMU performed on the ETS Proficiency Profile?

Descriptive techniques were employed in addressing this question. Specifically, mean and standard

deviation of scores were computed for each skill dimension.

In spring 2014, (n=600) out of (n=981) graduating seniors took the EPP. Results were calculated for (n=537)

of those students who completed 75% or more of the test. Table 4 provides a summary of the results for

the scaled scores. As can be discerned from Table 4, in 2014 the mean overall score for graduating seniors

was (M=430.56, SD=18.31). Similar to the results for incoming freshmen, FAMU graduating seniors’ total

score was significantly below the national average for 2014 (M=447.89, SD=8.87). A review of FAMU

graduating seniors’ performance in each skill dimension also revealed that performance fell below the

national average. Please refer to Table 4 for a summary of these results.

Table 4 FAMU Graduating Seniors Average Scaled Scores for 2014

Possible Range

of Scores FAMU 2014

National Mean & Standard Deviation

Difference Indicator

M SD M SD

Total Score 400 to 500 430.56 18.31 447.89 8.87

Critical Thinking 100 to 130 108.09 5.37 112.75 2.28

Reading 100 to 130 113.69 7.34 118.98 2.42

Writing 100 to 130 111.45 5.61 114.86 1.75

Mathematics 100 to 130 110.90 5.90 114.31 2.54

Humanities 100 to 130 112.88 6.32 115.68 2.08

Social Sciences 100 to 130 110.55 5.74 114.37 2.03

Natural Sciences 100 to 130 111.76 5.84 116.00 2.01

FAMU Mean Performance below National Average FAMU Mean Performance above National Average

A review of the trends associated with the performance of FAMU graduating seniors on the EPP revealed

some variability in terms of overall mean performance. Between 2009 and 2014 there has been a

significant drop in terms of the overall performance of graduating seniors on the test [2009 (M=438.08,

SD=16.01), 2014 (M=430.56, SD=18.31)]. Please refer to Table 5 for a summary of these results.

10

Further analysis also revealed that since 2009 the mean performance of FAMU graduating seniors has fallen below the national average (see Figure

2).

Table 5 Trends in Graduating Seniors Average Performance on ETS Proficiency Profile for 2009-2014

Skill Dimension 2009 2010 2011 2012 2013 2014

N =613 N =751 N =242 N =409 N =287 N =537

Mean SD Mean SD Mean SD Mean SD Mean SD Mean SD

Total Score 438.08 16.01 432.97 19.16 430.84 17.22 434.64 20.47 434.28 20.24 430.56 18.31

Critical Thinking 109.88 5.14 108.55 5.54 108.28 4.95 109.07 6.13 109.18 5.87 108.09 5.37

Reading 115.81 6.48 114.80 7.32 113.79 7.05 114.85 7.15 114.53 7.50 113.69 7.34

Writing 113.48 4.81 112.14 5.49 111.54 5.46 112.35 5.41 112.61 5.76 111.45 5.61

Mathematics 112.40 6.06 111.04 5.86 110.75 5.61 111.53 5.91 111.20 5.72 110.90 5.90

Humanities 113.47 5.87 113.74 6.38 112.67 6.19 114.03 6.51 113.51 6.54 112.88 6.32

Social Sciences 111.88 5.31 111.11 6.10 111.01 5.70 111.39 6.37 111.34 6.43 110.55 5.74

Natural Sciences 113.72 5.28 112.34 6.07 112.02 5.47 112.56 6.12 112.98 6.05 111.76 5.84

11

Figure 2 Historical Overview of FAMU Graduating Seniors Performance on ETS Proficiency Profile Compared to National Average

A review of each skill dimension measured on the EPP for graduating seniors revealed serious

concerns with FAMU students’ proficiency with all skill areas. Since 2009 less than 10% of

graduating seniors completing the test were classified as proficient in Critical Thinking, Writing

Level 3 and Mathematics Level 3. Please refer to Table 6 for a comprehensive overview of these

results.

Table 6 Trends in Graduating Seniors Classified as Proficient (2009-2014)

Proficiency Levels 2009 2010 2011 2012 2013 2014

N=613 N=751 N=242 N=409 N=287 N=537 Reading, Level 1 51% 39% 32% 37% 37% 35%

Reading, Level 2 24% 18% 13% 17% 16% 13%

Critical Thinking 2% 1% 0% 3% 4% 1%

Writing, Level 1 54% 36% 32% 39% 41% 32%

Writing, Level 2 15% 11% 10% 11% 13% 10%

Writing, Level 3 5% 4% 3% 4% 4% 2%

Mathematics, Level 1 47% 31% 31% 33% 31% 31%



2009 2010 2011 2012 2013 2014

Mean Score* 438.08 432.97 430.84 434.64 434.28 430.56

National Avg. 448.46 448.46 448.46 448.46 448.46 447.89

420

425

430

435

440

445

450M

ean

Sca

led

Sco

re

12

Freshmen/Senior Comparative Analysis

Figures 4 and 5 provides a comparison of freshmen results to those of seniors assuming four and

six years to time of graduation. Assuming a four year graduation for the cohort of incoming

freshmen who took the test in 2006, overall mean performance between incoming freshmen

(M=431.19) and graduating seniors (M=432.97) showed a marginal increase (i.e. 1.78 scaled

score points). Assuming a six year graduation for the 2006 cohort of incoming freshmen who

participated in the test, the performance of graduating seniors was slightly higher (M=434.64).

Please refer to figures 4 and 5 for a longitudinal overview of these results.

Figure 4 Comparison assuming 4-year Graduation

2006/2010 2007/2011 2008/2012 2009/2013 2010/2014

Freshmen 431.19 436.16 434.8 430.41 429.67

Seniors 432.97 430.84 434.64 434.28 430.56

428

429

430

431

432

433

434

435

436

437

13

Figure 5 Comparison assuming 6-year Graduation

EPP Freshmen Comparative Group Analysis

Guiding Question 3: How does the performance of incoming freshmen at FAMU compare to the National

Average, institutions in our Carnegie Classification, HBCU’s, and institutions in the State University System


First, comparison groups had to be identified then descriptive techniques were employed in addressing

this guiding question. Specifically, mean and standard deviation of scores were computed for each skill

dimension.

The Carnegie classification of similar institutions were developed based on institutions basic classification

(DRU/Doctoral/Research University) and enrollment profile. Once institutions were identified, further

work was required in an effort to ascertain whether or not they participated in the EPP. This process

helped to narrow the relevant group of institutions for comparison. A similar process was employed in

identifying the HBCU comparison group. Relative to institutions in the SUS, all participating institutions

were included in the comparison group along with institutional and aspirational peers as defined by

Institutional Effectiveness. This was necessary as a minimum of ten institutions were required for

comparative analyses. Please refer to Appendix B for a list of all institutions in the comparison groups.

2006/2012 2007/2013 2008/2014

Freshmen 431.19 436.16 434.8

Seniors 434.64 434.28 430.56

427

428

429

430

431

432

433

434

435

436

437

14

Following, is a comparative summary of the performance of FAMU freshmen and that of the ETS provided National Averages, Carnegie

classification of similar institutions, HBCUs taken as a group, and participating institutions in the State University System of Florida to include

institutional and aspirational peers. The national mean provided in Tables 7 was taken from the annual comparative data guide provided by ETS.

The report provided descriptive statistics based on the number of students who completed an un-proctored version of the ETS Proficiency Profile

between July 1, 2008 and June 30, 2013. A five year rolling average is utilized to normalize the scores.

As can be discerned from the data provided in Table 7, the mean score of FAMU freshmen taken as a group for the period July 1, 2008 through

June 30, 2013 (M=430.75, SD 16.0) fell below the national average (M=432.70, SD 17.6) and the SUS/Institutional/Aspirational comparison group

(M=441.8, M=19.2), but was above that of the Carnegie comparison group (M=429.0, SD 15.3) and the HBCU comparison group (M=427.6,

SD=14.6). Please refer to Table 7 for a comprehensive overview of these results to include the results for each skill dimension which outlines the

proportion of the test sample who were classified as P (Proficient), M (Marginally Proficient) and N (Not Proficient).

Table 7 Comparative Summary of the Performance of FAMU Freshmen to Comparison Groups

Skill Dimension

National Average Carnegie Comparison Group

HBCU Comparison Group

SUS Comparison Group FAMU Cumulative Results

N=27,171 N=3,696 N=18,335 N=5,813 N=2,952


Total Score 432.70 17.60 429.00 15.30 427.60 14.60 441.80 19.20 430.75 16.00

Critical Thinking 108.70 5.60 107.80 5.00 107.30 4.80 111.00 6.30 107.88 5.15

Reading 114.30 7.10 113.10 6.40 112.50 6.20 117.00 7.20 113.50 6.77

Writing 112.20 5.20 111.30 4.90 110.90 4.80 114.20 5.10 112.20 5.02

15

Mathematics 110.50 5.50 110.00 4.90 109.30 4.60 113.10 5.90 110.51 5.06

Humanities 112.80 6.10 111.70 5.60 110.80 5.30 114.50 6.40 112.38 5.92

Social Sciences 111.00 5.90 110.00 5.50 109.30 5.10 112.90 6.30 110.37 5.61

Natural Sciences 112.60 6.00 111.30 5.40 110.80 5.10 114.60 6.20 111.60 5.67

National Average Carnegie Peer Group

HBCU Peer Group SUS/ Institutional/ Aspirational Peer Group

FAMU Cumulative Results

Proficiency Levels P M N P M N P M N P M N P M N

Reading, Level 1 38% 22% 40% 32% 26% 42% 31% 27% 42% 57% 20% 24% 32% 23% 45%

Reading, Level 2 15% 14% 71% 11% 12% 77% 9% 13% 78% 31% 17% 52% 12% 12% 76%

Critical Thinking 2% 5% 93% 1% 4% 95% 1% 4% 96% 4% 16% 80% 1% 4% 95%

Writing, Level 1 39% 32% 28% 34% 36% 30% 33% 40% 28% 58% 27% 14% 37% 34% 29%

Writing, Level 2 10% 24% 66% 7% 21% 72% 5% 21% 74% 18% 34% 48% 9% 23% 68%

Writing, Level 3 3% 14% 83% 2% 12% 86% 2% 10% 88% 6% 24% 69% 3% 14% 83%

Mathematics, Level 1 29% 24% 47% 27% 27% 45% 23% 31% 47% 50% 25% 26% 29% 24% 48%



Note: P = % of students who were Proficient M = % of students who were Marginally Proficient N = % of students who were Not-Proficient

Figures 6-8 provide a graphical representation of these results.

16

Figure 6 Incoming Freshmen Total Mean Score Comparison

Figure 7 Incoming Freshmen Skill Dimensions Comparison

Figure 8 Incoming Freshmen Proficiency Classification Comparison (i.e. Percent of Students classified as proficient)

420

425

430

435

440

445National Average

Carnegie Peer Group

HBCU Peer Group

SUS/ Institutional/ AspirationalPeer Group


102104106108110112114116118

National Average

Carnegie Peer Group

HBCU Peer Group



0%10%20%30%40%50%60%70%

National Average

Carnegie Peer Group

HBCU Peer Group



17

EPP Graduating Seniors Comparative Group Analysis

Guiding Question 4: How does the performance of graduating seniors at FAMU compare to the national

average, institutions in our Carnegie Classification, HBCU’s, and institutions in the State University System


The same approach outlined to address the third guiding question was employed in the identification and

development of the comparison group of institutions. See Appendix B for a list of institutions that

comprised the comparison groups. Descriptive techniques were employed in addressing this question.

Specifically, mean and standard deviation of scores were computed for each skill dimension.

Following, is a comparative summary of the performance of FAMU graduating seniors and that of the ETS

provided National Averages, Carnegie classification of similar institutions, HBCUs taken as a group, and

participating institutions in the State University System (SUS) of Florida to include institutional and

aspirational peers. The national mean provided in Table 8 was taken from the annual comparative data

guide provided by ETS. The report provided descriptive statistics based on the number of students who

completed an un-proctored version of the ETS Proficiency Profile between July 1, 2008 and June 30, 2014.

A five year rolling average is utilized to normalize the scores.

As can be discerned from the data provided in Table 8, the mean overall performance of FAMU graduating

seniors taken as a group for the period July 1, 2009 through June 30, 2014 (M=433.18, SD=18.95) fell

below the national average (M=443.50, SD 21.0), the Carnegie comparison group (M=437.9, SD=19.5) and

the SUS/Institutional/Aspirational comparison group (M=447.3, SD=21.9), but was above that of the HBCU

comparison group (M=431.9, SD=17.6). Please refer to Table 8 for a comprehensive overview of these

results to include the results for each skill dimension which outlines the proportion of the test sample

who were classified as P (Proficient), M (Marginally Proficient) and N (Not Proficient).

18

Table 8 Graduating Seniors Comparative Data

Skill Dimension

National Average

Carnegie Peer Group

HBCU Peer Group

SUS/ Institutional/

Aspirational Peer Group


N=35,063 N=6,081 N=7,829 N=4,738 N=2,370


Total Score 443.50 21.00 437.90 19.50 431.90 17.60 447.30 21.90 433.18 18.95

Critical Thinking 111.70 6.70 109.90 6.10 108.50 5.50 112.50 6.80 108.72 5.58

Reading 118.00 7.50 116.20 7.10 114.50 6.90 118.60 7.30 114.53 7.24

Writing 114.30 5.20 113.10 5.20 111.70 5.10 114.70 5.30 112.15 5.50

Mathematics 112.80 6.30 111.90 6.00 110.10 5.40 114.80 6.80 111.22 5.87

Humanities 115.90 6.70 113.70 6.30 112.10 5.80 116.00 6.70 113.43 6.36

Social Sciences 114.00 6.60 112.20 6.20 110.70 5.90 114.40 6.70 111.13 6.00

Natural Sciences 115.20 6.40 113.70 5.90 112.20 5.60 115.90 6.40 112.44 5.92

National Average Carnegie Peer Group

HBCU Peer Group SUS/ Institutional/

Aspirational Peer Group


Proficiency Levels P M N P M N P M N P M N P M N

Reading, Level 1 58% 18% 24% 52% 21% 27% 42% 25% 33% 65% 16% 19% 38% 22% 40%

Reading, Level 2 33% 17% 50% 25% 17% 58% 17% 16% 67% 40% 17% 43% 16% 13% 70%

Critical Thinking 6% 13% 80% 3% 11% 86% 3% 7% 91% 9% 20% 71% 2% 5% 93%

Writing, Level 1 55% 28% 17% 49% 30% 21% 39% 34% 27% 62% 24% 14% 38% 29% 33%

19

Writing, Level 2 19% 31% 50% 14% 29% 57% 9% 25% 66% 23% 34% 44% 12% 23% 65%

Writing, Level 3 6% 23% 70% 5% 19% 76% 3% 13% 84% 8% 27% 65% 4% 15% 81%




Note: P = % of students who were Proficient M = % of students who were Marginally Proficient N = % of students who were Not-Proficient

Figures 9-11 provides a graphical representation of these results.

Figure 9 Graduating Seniors Total Mean Score Comparison

420

425

430

435

440

445

450

Total Score

National Average

Carnegie Peer Group

HBCU Peer Group



20

Figure 10 Graduating Seniors Skill Dimension Comparison

Figure 11 Graduating Seniors Proficiency Classification Comparison (i.e. Percent of Students classified as proficient)

Recommendations for Improvement

The results of the EPP confirm that our students are entering the university with serious deficits in all skill

areas as reported by the proportion of incoming freshmen who were classified as proficient on the test.

Equally concerning, is the proportion of graduating seniors who were classified as proficient in the skill

areas measured by the test. Since 2009 less than 55% of FAMU graduating seniors were classified as

proficient in the skill areas measured by the test. Additionally, less than 10% of graduating seniors

completing the test were classified as proficient in Critical Thinking, Writing Level 3 and Mathematics Level

102

104

106

108

110

112

114

116

118

120

National Average

Carnegie Peer Group

HBCU Peer Group

SUS/ Institutional/ Aspirational PeerGroup


0%

10%

20%

30%

40%

50%

60%

70%

National Average

Carnegie Peer Group

HBCU Peer Group

SUS/ Institutional/ Aspirational PeerGroup


21

3. It is clear that more emphasis should be placed on remediating and developing/enhancing the

competencies measured by the test. In general education courses in addition to upper division courses,

these skill sets should be emphasized as a way of speaking to what students should know and or be able

to do at the end of the degree program.

FAMU EPP Top Performers

This section of the report highlights the top performers on the ETS Proficiency Profile for fall 2013 and

spring 2014. Please refer to Tables 9 & 10 for a list of the top performers.

Table 9 Fall 2013 ETS Top Performers (incoming freshmen)

Name Intended Major Total Score

Fralin, Phylesia, A Accounting 500

McGowan, Calvin, E Biological Sciences 493

Roy, Rais Health & Medical Sciences 493

Gonzalez, Jorge Biological Sciences 490

Onubogu, Ugoma, C Biological Sciences 486

Table 10 Spring 2014 ETS Top Performers (graduating seniors)

Name Major Total Score

Moore, Jason, N Pharmacy 493

Shaw, Kelvin, D Psychology 493

Clark, Jerry, L Physics 490

Okany, Chinelo, Pharmacy 489

Harris, Mitchell, T Pharmacy 483

Limitations of Report

The following considerations should be kept in mind when interpreting the comparative results:

This data should be considered comparative rather than normative because the institutions included in the data do not represent proportionally the various types of higher education institutions. The data are drawn entirely from institutions that choose to participate in the ETS Proficiency Profile. Such a self-selected sample may not be representative of all institutions.

22

The number of students tested and sampling procedures vary from one institution to another.

Therefore, it is impossible to verify that the students tested at each institution are representative

of all that institution's students.

It is helpful when these comparisons involve students at approximately the same point in their educational careers. Students who have not identified their credit status are excluded from these calculations.

The tables report data for institutions that have tested 30 or more students at the selected class level or number of credit hours attained. Institutions with fewer than 30 test takers at that class level are excluded from these calculations.

23

APPENDIX

24

Appendix A

Proficiency Measures

In addition to a total score, proficiency classifications (proficient, marginal or not proficient) measure how well your students have mastered each level of proficiency within three skill areas:

Reading/Critical Thinking

Writing

Mathematics

Reading/Critical Thinking

Level 1: Students who are proficient can:

recognize factual material explicitly presented in a reading passage

understand the meaning of particular words or phrases in the context of a reading passage


synthesize material from different sections of a passage

recognize valid inferences derived from material in the passage

identify accurate summaries of a passage or of significant sections of the passage

understand and interpret figurative language

discern the main idea, purpose or focus of a passage or a significant portion of the passage

Level 3/Critical Thinking: Students who are proficient can:

evaluate competing causal explanations

evaluate hypotheses for consistency with known facts

determine the relevance of information for evaluating an argument or conclusion

determine whether an artistic interpretation is supported by evidence contained in a work

recognize the salient features or themes in a work of art

evaluate the appropriateness of procedures for investigating a question of causation

evaluate data for consistency with known facts, hypotheses or methods

recognize flaws and inconsistencies in an argument

25

Writing Skills


recognize agreement among basic grammatical elements (e.g., nouns, verbs, pronouns and conjunctions)

recognize appropriate transition words

recognize incorrect word choice

order sentences in a paragraph

order elements in an outline


incorporate new material into a passage

recognize agreement among basic grammatical elements (e.g., nouns, verbs, pronouns and conjunctions) when these elements are complicated by intervening words or phrases

combine simple clauses into single, more complex combinations

recast existing sentences into new syntactic combinations


discriminate between appropriate and inappropriate use of parallelism

discriminate between appropriate and inappropriate use of idiomatic language

recognize redundancy

discriminate between correct and incorrect constructions

recognize the most effective revision of a sentence

Mathematics


Solve word problems that would most likely be solved by arithmetic and do not involve conversion of units or proportionality. These problems can be multistep if the steps are repeated rather than embedded

Solve problems involving the informal properties of numbers and operations, often involving the Number Line, including positive and negative numbers, whole numbers and fractions (including conversions of common fractions to percent, such as converting "1/4" to 25%)

solve problems requiring a general understanding of square roots and the squares of numbers

solve a simple equation or substitute numbers into an algebraic expression

Find information from a graph. This task may involve finding a specific piece of

26

information in a graph that also contains other information


Solve arithmetic problems with some complications, such as complex wording, maximizing or minimizing, and embedded ratios. These problems include algebra problems that can be solved by arithmetic (the answer choices are numeric)

Simplify algebraic expressions, perform basic translations, and draw conclusions from algebraic equations and inequalities. These tasks are more complicated than solving a simple equation, though they may be approached arithmetically by substituting numbers

interpret a trend represented in a graph, or choose a graph that reflects a trend

solve problems involving sets; problems have numeric answer choices


solve word problems that would be unlikely to be solved by arithmetic; the answer choices are either algebraic expressions or numbers that do not lend themselves to back-solving

solve problems involving difficult arithmetic concepts, such as exponents and roots other than squares and square roots, and percent of increase or decrease

generalize about numbers (e.g., identify the values of (x) for which an expression increases as (x) increases)

Solve problems requiring an understanding of the properties of integers, rational numbers, etc.

interpret a graph in which the trends are to be expressed algebraically or one of the following is involved: exponents and roots other than squares and square roots, percent of increase or decrease

solve problems requiring insight or logical reasoning

27

Appendix B

Following are the comparison groups used in the analyses. Table A Carnegie Comparison Group

Freshmen Seniors

Bowie State University Bowie State University

Florida Agricultural and Mechanical University Florida Agricultural and Mechanical University

Indiana State University Indiana State University

Lamar University Lamar University

Morgan State University Morgan State University

North Carolina A & T State University North Carolina A & T State University

Sam Houston State University Sam Houston State University

South Carolina State University South Carolina State University

Texas A & M University-Commerce Tennessee State University

Texas A & M University-Kingsville Texas A & M University-Commerce

Texas A & M University-Kingsville

University of Arkansas at Little Rock

Table B HBCU Comparison Group

Freshmen Seniors

Alabama A & M University Alabama A & M University

Alabama State University Bennett College for Women

Alcorn State University Bowie State University

Benedict College Cheyney University of Pennsylvania

Bennett College for Women Clark Atlanta University

Bowie State University Coppin State University

Cheyney University of Pennsylvania Fisk University

Clark Atlanta University Florida Agricultural and Mechanical University

Coppin State University Grambling State University

Elizabeth City State University Huston-Tillotson University

Fisk University Jarvis Christian College

Florida Agricultural and Mechanical University Mississippi Valley State University

Huston-Tillotson University Morgan State University

Mississippi Valley State University Norfolk State University

Morehouse College North Carolina A & T State University

Morgan State University Paul Quinn College

Norfolk State University Philander Smith College

28

Freshmen Seniors

North Carolina A & T State University Prairie View A & M University

Prairie View A & M University Saint Augustine’s College

South Carolina State University South Carolina State University

Spelman College Spelman College

Winston-Salem State University St Philips College

Talladega College

Tennessee State University

University of Maryland Eastern Shore

Wiley College

Winston-Salem State University

Table C SUS/Institutional/Aspirational Peers Comparison Group

Freshmen Seniors

Florida Agricultural and Mechanical University Florida Agricultural and Mechanical University

Florida Gulf Coast University Florida International University

Florida International University North Carolina A&T State University

North Carolina A&T State University Temple University

Nova Southeastern University - Law School University of North Florida

Temple University University of South Florida - Sarasota-Manatee

University of North Florida University of South Florida - St. Petersburg

University of South Florida - Sarasota-Manatee University of South Florida - Tampa

University of South Florida - St. Petersburg University of South Florida Polytechnic

University of South Florida - Tampa University of Arkansas - Little Rock Note: Given the small proportion of institution in the SUS that participates in the ETS Proficiency Profile, Institutional and Aspirational Peers as defined by Institutional Effectiveness had to be added in an effort to run the analysis.

ets proficiency profile summary results - florida a&m ... report (final).pdf · ets proficiency...

Documents