response: postsecondary institution ratings system [docket id ed–2013–ies–0151] prepared by...

Response: Postsecondary Institution Ratings System [Docket ID ED–2013–IES–0151]

Prepared by Tom Benghauser

for

The National Center for Education Statistics, the Institute of Education Sciences, and the Department of Education

Tom Benghauser609 542-0738

[email protected]

mailto:[email protected]

CONTENTS

Background and Overview

Data Needs, Sources, and Issues

Appendix

BACKGROUND AND OVERVIEWThe Education Department’s Postsecondary Institution Rating System is being developed in support of President Obama’s broader goals of reversing the growth of income inequality and increasing the possibilities for upward mobility in the United States.

As the President stated in a speech at the University of Buffalo last August:

“It will be my firm principle that our ratings have tobe carefully designed to increase, not decrease, the opportunities for higher education for students whoface economic or other disadvantages.”

BACKGROUND AND OVERVIEWBut it is not sufficient that such additional opportunities merely exist.

The individuals who will benefit from them must first be made aware of and then motivated to take advantage of them.

As President Obama subsequently said at the recent White House Higher Education Summit:

“We also know that too many students don’t apply to the schools that are right for them. They may sometimes underestimate where they could succeed, where they could go. There may be a mismatch in terms of what their aspirations are and the nature of what's offered at the school that's close by.”

BACKGROUND AND OVERVIEWAnd as Robert Kelchen, one of the data experts invited by NCES to present at the Technical Symposium on Postsecondary Institution Ratings on February 6, recently wrote in the Washington Monthly:

“Once the ratings get released, there is no guarantee that students [will] use the ratings in any meaningful way (although it’s possible).”

BACKGROUND AND OVERVIEW

Research carried out by Turner and Hoxby has demonstrated that proactively drawing high-potential/low-income high school students’ attention to colleges they otherwise wouldn’t have considered, providing them with key financial aid and other information on these schools, even sending them application fee waivers, dramatically increases the likelihood they will apply to, enroll at, and then thrive at the best match/best fit institutions where they are most likely to derive the greatest benefit.


The PIRS being developed by the Department of Education must accordingly be made available to college intenders in general and socio-economically disadvantaged candidates in particular as part of an inviting, easy to use, well-publicized resource – a “presentation framework” that will not only

• draw their attention to the college(s) they can get into and afford and where they will be neither over-matched nor under-matched relative to their academic potential

but also

• maximize the probability they will go on to actually apply to, enroll at, and graduate from a best match/best fit institution


The approach we urge the Department of Education to adopt as a structure- and content-template for the “presentation framework” for the PIRS both

• provides college intenders with step-by-step guidance in systematically narrowing their search for and identifying their own best-match/best-fit colleges and

• gives them personalized hand-holding clear though the application, financial aid, final decision, paperwork, and matriculation processes.


Research of our approach with numerous members of our nation’s cadre of overworked, overstretched high school guidance counselors makes clear that the vast majority will immediately recognize it as an exceptionally valuable tool that they will quickly bring to the attention of their advisees.

We are confident that at least one of its several overtly entertaining aspects – in particular its highlighting of “fun findings” from the college alumni satisfaction and outcomes surveys that will be a source of data for numerous key metrics to be included in our scores and ratings – will immediately engage active college intenders

And we are equally certain that – as a result of its ease of use and most of all usefulness - these college intenders will continue to rely on it throughout their journey to college.


Finally, the resource we have developed and are recommending for ED’s adoption as a prototype also resolves a problem raised by multiple members of the higher education establishment. One of them, Christopher Nelson, wrote in a recent Huffington Post editorial (‘Flawed From the Start: The President's New Plan for Higher Education’) that

“Rating of any kind suggests that all students are looking for something common; it also assumes that colleges are more alike than they actually are. Better instead to find ways of getting as much information as possible about each school to each student and let them make their own ratings - ones that will suit their own individual needs!”

Christopher Nelson is President of St. John’s College Annapolis and a founding member of the Annapolis Group, a consortium of more than 120 liberal arts colleges,


President Nelson will presumably be pleased that our resource enables students to browse and compare different colleges’ profiles and then to generate short lists of the colleges that best match up with their individual resources, abilities, and aspirations.

This calculation excludes need-based aid tied to on-campus jobs and/or summer employment, based on the conviction that it deprives its recipients

of invaluable time for study and social acclimatization and also labels them as socioeconomically disadvantaged.


We then help them to zero in on their best match + best fit college, by letting them calculate comparative scores for each of their short listed colleges on the basis of

how important they indicate a number of more subjective, less tangible college characteristics are to them

weighted by

how highly recent alumni from the short-list colleges have rated their alma maters on these same factors.


Identifying a best-match/best-fit college – an affordable institution that can make the most of a college intender’s academic potential and where, based on its ability to meet her more subjective interests and preferences, she will feel sufficiently “comfortable” to make the most of her opportunity – is just the start for many students.

It is well documented, for instance, that many students are so unfamiliar with the application process in general and the FASFA completion ordeal in particular that they simple give up.

Others, unable to afford more than a handful of application fees, choose not to apply to colleges they still fear may be too great a reach to waste money on.


This is why our presentation framework

• provides college intenders with step-by-step guidance in completing the common application and even sends them emails/text message reminding them of key deadlines for all the institutions they apply to

• informs/reminds college intenders that application fee waivers are available and also makes it possible for them to complete a fee waiver request form - and for their guidance counselor to authenticate it – just once yet still have it sent to all the colleges they decide to apply to.


Our presentation framework will also enable intenders to not only read what affinity alumni from their short-listed colleges have said about their alma maters but also get real-time mentoring and personal reassurances from them.


Finally, for a variety of reasons that include the fear of leaving family and community, students not infrequently chicken out after being accepted at a best-match/best-fit college and instead decide to attend a local institution where they are tragically under matched.

The insider jargon for this phenomenon is “summer melt” and it is to prevent it that our presentation framework includes mechanisms that let students

• Compare the institutions they’ve been admitted to side by side• Network in real time with others who’ve also been admitted • Get additional mentoring from their affinity alumni• Chat with undergrads who are already there

It also provides assistance in completing all of the paperwork for their best-match/best fit college.

DATA NEEDS, SOURCES, AND ISSUES

The data displayed in our presentation framework include several metrics whose values are derived from values contained in the IPEDS and/or Common Data Set data sets while others are obtained from alumni satisfaction and outcomes research.

One of the derived values is many colleges’ Dirty Little Secret:

The percentage of freshmen determined to have needwho are not actually granted any need-based aid

This new metric reveals the cynicism underlying many institutions’ Net Price Calculators.


Another derived variable – the average amount of need based aid per student determined to have need – is often significantly lower for the overall undergraduate student body than it is for first- time full-time freshmen only.

This suggests the possibility that some colleges may be using exceptionally generous aid offers to lure in desirable prospects and then, once switching to an other institution has become problematic, reducing the amount of aid they receive.

It is something prospective students should be made aware of.


Other derived variables are required for the calculation of two major indicators:

• our derived ‘bang for the buck/value for money/return on investment’ index, which is intended for students’ use as a key if not final criterion for differentiating among the mission-comparable colleges on their short lists

This calculation is in turn a key component of:

• our derived measure of the relative ‘value added’ by the various colleges - their ‘institutional effectiveness’, a metric that is intended for use primarily by the government and probably doesn’t need to be included in the presentation framework we’ve designed for college intenders

Bang for the Buck/Value for Money/Return on Investment Index:

Variable Source

Overall graduation rates IPEDS/CDS

Average years required to obtain bachelors Derived

Annual cost of attendance (sticker price) IPEDS/CDS

Average need based aid per undergraduate Derived

Average grad’s annual cost of obtaining degree net of need based aid Derived

Average grad’s total cost of obtaining bachelors net of need based aid Calculation

Average grad’s loan obligation at graduation Derived

Average grad’s total cost of obtaining bachelor's degree including loans outstanding at graduation

Calculation

Average starting salary of graduates in their first full-time paying job (excluding volunteer/charitable work)

Alumni Satisfaction and Outcomes Research

Details of Derived Variables Using for Illustrative PurposesThe 2012 - 2013 Common Data Set for Kenyon College

H2 a) Number of degree-seeking undergraduate students (CDS Item B1 if reporting on Fall 2012 cohort)

445 1817H2 b) Number of students in line a who applied for need-based financial aid 257 870H2 c) Number of students in line b who were determined to have financial need 190 720H2 d) Number of students in line c who were awarded any financial aid 189 712H2 e) Number of students in line d who were awarded any need-based scholarship or grant aid 185 698H2 f) Number of students in line d who were awarded any need-based self-help aid 106 527H2 g) Number of students in line d who were awarded any non-need-based scholarship or grant aid 36 120H2 h) Number of students in line d whose need was fully met (exclude PLUS loans, unsubsidized

loans, and private alternative loans)142 466

H2i) On average, the percentage of need that was met of students who were awarded any need-based aid. Exclude any aid that was awarded in excess of need as well as any resources that were awarded to replace EFC (PLUS loans, unsubsidized loans, and private alternative loans)

95.8% 94.9%

H2 j) The average financial aid package of those in line d. Exclude any resources that were awarded to replace EFC (PLUS loans, unsubsidized loans, and private alternative loans)

$ 37,805 $ 37,283

H2 k) Average need-based scholarship and grant award of those in line e $ 36,339 $ 34,245H2 l) Average need-based self-help award (excluding PLUS loans, unsubsidized loans, and private

alternative loans) of those in line f$ 3,983 $ 5,015

H2 m) Average need-based loan (excluding PLUS loans, unsubsidized loans, and private alternative loans) of those in line f who were awarded a need-based loan

$ 2,911 $ 4,119

D1 Percent of students who applied for need-based aid = b) divided by a) 58% 48%D2 Percent of students applying for need-based aid who were determined to need it = c) divided by

b)74% 83%

D3 Percent of students determined to have need who received some need based scholarship/grant aid = e) divided by c)

97% 97%D4 Percent of students determined to have need but who receive NO need-based scholarship/grant

aid = 100 minus D33% 3%

D5 Total Need Based Scholarship/Grant Aid Awarded = e) times k) $6,722,715 $23,903,010D6 Average need based grant/scholarship per student awarded any need based aid = D2 divided

by e)$36,339 $34,245

D7 Avg need based aid per student determined to have need = D2 divided by c) $35,383 $33,199D8 Avg need-based aid per undergraduate = D5 divided by a) -- $13,155

Undergrads (Incl. Fresh.)

First-time Full-time Freshmen

Continued…

B3 410H4 41%H5 $15,894D9 168

D10 $2,671,702D11 $6,516

B4 439

B5Of the initial 2005 cohort, how many did not persist and did not graduate for the following reasons: death, permanent disability, service in the armed forces, foreign aid service of the federal government, or official church missions; total allowable exclusions:

0

B6 Final 2005 cohort, after adjusting for allowable exclusions: (subtract question B5 from question B4) 439

B7 Of the initial 2005 cohort, how many completed the program in four years or less (by August 31, 2009): 364

B8 Of the initial 2005 cohort, how many completed the program in more than four years but in five years or less (after August 31, 2009 and by August 31, 2010): 16

B9 1B10 Total graduating within six years (sum of questions B7, B8, and B9): 381B11 Six-year graduation rate for 2005 cohort (question B10 divided by question B6): 87%D12 Percent Graduating in 4 years = B7 divided by B6 82.9%D13 Percent Graduating in 5 years = B8 divided by B6 3.6%D14 Percent Graduating in 6 years = B9 divided by B6 0.2%D16 Average Years to Graduate = weighted average of D12, D13, and D14 4.05

Of the initial 2005 cohort, how many completed the program in more than five years but in six years or less (after

Total Amount Owed By Graduating Class = H5 times D1Amount Owed by Average Graduate = D3 divided by B3Initial 2005 cohort of first-time, full-time bachelor's (or equivalent) degree-seeking undergraduate students; total all students:

Bachelor's degrees awarded from July 1, 2011 to June 30, 2012Provide the percentage of the graduating class (defined above) who borrowed at any time through any loan Report the average per-undergraduate-borrower cumulative principal borrowed of those in line H4.Number of Graduating Class Who Borrowed = B3 times H4

Kenyon Data Source

Annual Cost of Attendance $59,960 CDS/IPEDSMinus Avg Need Based Aid Per Undergraduate $13,155 Derived Variable D8Equals Average Annual Cost of Attendance Net of Need Based Aid $46,805Times Average Years Required to Obtain a Bachelor's Degree 4.83 Derived Variable D16Equals Average Total Cost of Obtaining Bachelor's Net of Need Based Aid $226,067Plus Average $ Amount of Undergrad Loan Obligation at Graduation $6,516 Derived Variable D11Equals Total $ Amount of Obtaining Bachelor's Degree Including Loans Outstanding at Graduation$232,583Divided by Average Starting Salary of Graduates in Their First Full-Time Paying Job $33,703 CTS Alumni SurveyEquals Average Years 6.9

Using Derived Variables to Calculate Years at Starting Salary for College Cost "Pay Back"

Kenyon Data Source



Details of Derived Variables Using for Illustrative PurposesThe 2012 - 2013 Common Data Set for Kenyon College

H2 a) Number of degree-seeking undergraduate students (CDS Item B1 if reporting on Fall 2012 cohort) 445 1817

H2 b) Number of students in line a who applied for need-based financial aid 257 870

H2 c) Number of students in line b who were determined to have financial need 190 720

H2 d) Number of students in line c who were awarded any financial aid 189 712

H2 e) Number of students in line d who were awarded any need-based scholarship or grant aid 185 698

H2 f) Number of students in line d who were awarded any need-based self-help aid 106 527

H2 g) Number of students in line d who were awarded any non-need-based scholarship or grant aid 36 120

H2 h) Number of students in line d whose need was fully met (exclude PLUS loans, unsubsidized loans, and private alternative loans)

142 466

H2i) On average, the percentage of need that was met of students who were awarded any need-based aid. Exclude any aid that was awarded in excess of need as well as any resources that were awarded to replace EFC (PLUS loans, unsubsidized loans, and private alternative loans)

95.8% 94.9%

H2j) The average financial aid package of those in line d. Exclude any resources that were awarded to replace EFC (PLUS loans, unsubsidized loans, and private alternative loans)

$ 37,805 $ 37,283

H2 k) Average need-based scholarship and grant award of those in line e $ 36,339 $ 34,245

H2 l) Average need-based self-help award (excluding PLUS loans, unsubsidized loans, and private alternative loans) of those in line f

$ 3,983 $ 5,015

H2 m) Average need-based loan (excluding PLUS loans, unsubsidized loans, and private alternative loans) of those in line f who were awarded a need-based loan

$ 2,911 $ 4,119

D1 Percent of students who applied for need-based aid = b) divided by a) 58% 48%D2 Percent of students applying for need-based aid who were determined to need it = c) divided by b) 74% 83%D3 Percent of students determined to have need who received some need based aid = e) divided by c) 97% 97%D4 Percent of students determined to have need but who receive NO need-based aid = 100 minus D3 3% 3%D5 Total Need Based Aid Awarded = e) times k) $6,722,715 $23,903,010D6 Average need based grant/scholarship per student awarded any need based aid = D2 divided by e) $36,339 $34,245D7 Avg need based aid per student determined to have need = D2 divided by c) $35,383 $33,199D8 Avg need-based aid per undergraduate = D5 divided by a) -- $13,155

First-time Full-time

FreshmenUndergrads (Incl. Fresh.)

Kenyon Data Source



PERCENT GRADUATING IN Kenyon U of Mass Amherst

Mount Holyoke Princeton U of Penn

4 Years 83 52 80 87 875 Years 4 13 4 7 7

6 Years 1 2 1 2 2

Never (weighted as 10) 12 33 15 4 4

Average Years To Graduate 4.8 6.2 5.0 4.4 4.4

Kenyon Data Source



B3 410

H4 41%

H5 $15,894

D9 168D10 $2,671,702D11 $6,516

Total Amount Owed By Graduating Class = H5 times D1Amount Owed by Average Graduate = D3 divided by B3

Bachelor's degrees awarded from July 1, 2011 to June 30, 2012

Provide the percentage of the graduating class (defined above) who borrowed at any time through any loan programs (institutional, state, Federal Perkins, Federal Stafford Subsidized and Unsubsidized, private loans that were certified by Report the average per-undergraduate-borrower cumulative principal borrowed of those in line H4.

Number of Graduating Class Who Borrowed = B3 times H4

Bang for the Buck/Value for Money/Return on Investment Index:

Importantly, for college prospects who already know their goals are not primarily pecuniary, bang/value/return could also be expressed in terms of other outputs measured by our alumni surveys – e.g. the chances of getting into grad school or of gaining employment in a particular field.

Added Value/Institutional Effectiveness Index:

The Difficulty of Producing the Outcomes OutcomesInstitutional Effectiveness

Percent of full-time first-time

undergraduates receiving Pell grants

(SFA0910_RV)Average SAT

(2-Part)1600 Minus

Avg SAT

Years to Pay Off Cost of Education Total Score Index

Upenn 17% 1433 168 3.6 787 98Princeton 12% 1452 148 3.3 540 67Umass Amherst 23% 1208 392 5.9 1530 190Mount Holyoke College 18% 1329 271 5.4 906 112Kenyon College 7% 1334 266 7.0 268 33

Average = 806 100





(2-Part)1600 Minus

Avg SAT



Average = 806 100

The larger the percentage, the

greater the difficulty of the

task

The lower the score, the

greater the difficulty of the

task

The fewer the years, the

better





(2-Part)1600 Minus

Avg SAT



Average = 806 100

Percent of full-time first-time undergraduates receiving Pell grants

UPenn 17Princeton 12UMass Amherst 23Mount Holyoke 18Kenyon College 7

Source: IPEDS





(2-Part)1600 Minus

Avg SAT



Average = 806 100

Percent of full-time first-time undergraduates receiving Pell grants

UPenn 17Princeton 12UMass Amherst 23Mount Holyoke 18Kenyon College 7

Reading Math Reading Math Reading Math700-800 62.00% 73.00% 700-800 7.50% 12.60% 700-800 44.00% 22.00%600-699 33.00% 24.00% 600-699 35.90% 47.50% 600-699 42.00% 62.00%500-599 6.00% 3.00% 500-599 46.60% 35.40% 500-599 13.00% 15.00%400-499 0.00% 0.00% 400-499 9.40% 4.40% 400-499 1.00% 1.00%300-399 0.00% 0.00% 300-399 0.60% 0.10% 300-399 0.00% 0.00%200-299 0.00% 0.00% 200-299 0.00% 0.00% 200-299 0.00% 0.00%Avg 713 720 Avg 590 618 Avg 679 655

TOTAL 1433 TOTAL 1208 TOTAL 1334

Penn KenyonUmass

Reading Math Reading Math700-800 37.33% 30.67% 700-800 76.00% 80.30%600-699 45.00% 49.67% 600-699 21.00% 18.40%500-599 17.00% 19.00% 500-599 3.00% 1.30%400-499 0.67% 0.67% 400-499 0.00% 0.00%300-399 0.00% 0.00% 300-399 0.00% 0.00%200-299 0.00% 0.00% 200-299 0.00% 0.00%Avg 669 660 Avg 723 729

TOTAL 1329 TOTAL 1452

Mount Holyoke Princeton

Source: Averages Derived from

Common Data Set Percentages





(2-Part)1600 Minus

Avg SAT



Average = 806 100(Percent Pell Grantees) x (1600 – Average SAT)÷

(Average Years Earnings to Pay Off Cost x 0.3) =

Total Score





(2-Part)1600 Minus

Avg SAT


UMass Amherst 23% 1208 392 5.9 510 190Mount Holyoke College 18% 1329 271 5.4 302 112UPenn 17% 1433 168 3.6 262 98Princeton 12% 1452 148 3.3 180 67Kenyon College 7% 1334 266 7.0 89 33

Average = 269 100


Alumni Satisfaction and Outcomes Surveys:An Essential Source of Essential Data

• Proof of concept in the form of a successful pilot survey carried out in 2011 exists for the value and feasibility of utilizing alumni satisfaction and outcomes research to supplement IPEDS, the College Board’s Common Data Sets, and the previously unpublished but highly revealing new variables that have now been derived from them.

• This pilot survey gathered information on the following variables, all of which will be of great utility both for inclusion in ED’s PIRS and as data for academic researchand analysis.

Analytical and problem-solving skills/ability to think criticallyKnowledge of a particular field or disciplineLeadership abilitiesActive interest in community serviceAbility to work independentlyAbility to form and retain friendshipsAbililty to have a good rapport with people holding different beliefsAbility to work effectively with different kinds of peopleReligious valuesAbility to communicate well orallyCompetitivenessAbility to work cooperativelyAbility to relax and enjoy leisure timeAbility to write clearly and effectivelyAbility to adjust to changeAbility to think creatively, approach problems from unconventional perspectivesAbility to think like an entrepreneur, willingness to take risksSelf-confidence

First NameLastNameCustom Data

RespondentIDStartDateEndDateIP AddressEmail Address

From what 4-year college or university did you get your undergraduate (bachelors) degree? (Please select from drop-down What year did you graduate?How satisfied are you overall with your undergraduate experience at [Q1] Why do you say that?What degree did you earn?What was your undergraduate major? (Please make your selection from the drop-down list)How much would you say you gained in each of the following areas from your time at [Q1]?

Students of different races, ethnicities, and nationalities socialized outside of classesOn-campus housing was comfortableStudents were open to opinions and viewpoints different from their own.When I arrived the faculty and administration helped me to feel welcome and “at home”.I made close friends with other students and am still in regular contact with themI made close friends with other students during my undergraduate yearsThere was extensive use of marijuana on campus during my time thereThere was extensive consumption of alcoholic beverages on campus during my time thereThe college offered many social opportunitiesThere was very little cheating during exams and tests and on writing assignments and other homework.I was able to choose from a large number of extracurricular activitiesOverall the student body was more interested in having fun than in studying

Please indicate the extent to which you agree - or disagree - with the following statements about [Q1].

Please provide your best recollection of how much the training cost in US$.Have you ever made a donation to [Q1]?When did you make your most recent donation?

As part of your college application process, did you take either the ACT or SAT? Please provide your best recollection of how you scored on the SAT 2-PartPlease provide your best recollection of how you scored on the SAT 3-PartPlease provide your best recollection of how you scored on the ACTDid you receive any special training in preparation for the ACT or SAT? Did you or your family have to pay for this training?

Thinking back to when you originally applied to college, which other colleges did you apply to?Was [Q1] the college you most wanted to attend when you were applying to college?Which other college was your first choice?Were you admitted to [Q11]?You said that [Q11] was your first choice and that you were admitted. Why did you attend [Q1] instead?Knowing what you know now, do you think you would be happier and/or more successful today if you had instead gone to a college other than [Q1]?

What kind of high school did you attend before enrolling at [Q1]? Where is that secondary school located?

Professors and other faculty were readily available to give me help and/or adviceThere was extensive sexual activity between studentsMost of my classmates were smarter than I amI frequently had discussions with faculty members outside of classThere was a lot of "school spirit"The overall ‘atmosphere’ on campus was serious and encouraged learningStudents tended to room and socialize with members of their own race, nationality, religion, sexual orientation, and/or social class.The student body was diverse in terms of race, nationality, religion, gender, sexual orientation, and social class.There was hostility on campus towards students of homosexual and transgender sexual orientationsOverall the political climate on campus was liberalThere was extensive use of so-called hard drugs such as cocaine and methamphetemine on campus during my time thereI spent most of my time reading, writing, and studyingHomosexuals were treated just like anybody else by students, faculty, and the administration.

I did not receive financial aidNeed-based grants from my collegePell grantSubsidized loansOther loansWork-study employmentOther forms of aid (e.g., merit scholarships and prizes)

Working full-timeLooking for workObtaining a graduate degreeOther

What university are you now attending for your graduate or professional studies?

Here are a few more statements about [Q1] that we'd appreciate your indicating how much you agree with.

During your undergraduate years did you belong to a fraternity, sorority, or similar social organization? Did you receive financial aid as an undergraduate? Mark all that apply.

How satisfied are you with the level of financial aid [Q1] provided?At the time you graduated, approximately what was the total amount you had borrowed to finance your undergraduate

What are you doing now?

Did your parents or other relatives attend [Q1]?

Did the benefits you received from attending [Q1] outweigh the financial costs to you and your family? Mark the best answer

Finally, feel free to use the comment box below to tell us anything you'd care to about our survey and its objectives. And if you have any questions, you may contact us via email at [email protected]

If you are a citizen of the United States, what is your race or ethnicity?What is your gender?What is your sexual orientation? What is your current personal status? Mark the one best answer.

And what is your annual salary now?Please tell us about the graduate and professional degrees, if any, that you have either already received or for which your are How well did [Q1] prepare you for graduate or professional school? How well do you feel [Q1] prepared you for what you are doing now? Where do you live now? What is your nationality? What country or countries do you hold a passport from?

For what type of organization are you now working on a paid or volunteer basis? Mark the one where you spend the most time.What branch of the military are you in?What is your rank?Which of the following best describes your current position? What was your per annum starting salary in the first full-time job you took after graduating from [Q1]?

Additional Metrics:

One criterion of community colleges’ effectiveness should be how many of their degree recipients or other attendees go on to four-year institutions and then how successful they are there.

Data germane to this issue could be easily obtained by asking alumni survey respondents whether they originally matriculated to their alma maters directly from high school or instead transferred in. Transfer students could be asked to identify the post-secondary institution they’d transferred in from and then be asked to rate it on parameters such as how well they felt it prepared them for study at a four-year college.

Data Availability Timetables

Although there is significant overlap in the variables reported on by the IPEDS database and the College Board’s Common Data Sets, the latter do not include data for Pell recipients and are also limited to a fraction of all Title IV postsecondary institutions.

On the other hand, CDS data are available much sooner than IPEDS data: as of 3 February 2014, CDS data for the 2013 – 2014 academic year were already appearing while “provisional final” IPEDS data for the 2012 – 2013 year had appeared just weeks earlier, at the end of December 2013.

RTI reports that much of the time lag in IPEDS release dates results from requirements for multiple quality control reviews.

Data Availability Timetables:

It is reasonable to assume that a primary purpose of this built in redundancy is to identity fraudulent data submissions.

Given the importance of timely, up-to-data to the college evaluation and selection process, we recommend that

• the IPEDS publication schedule be significantly accelerated by delaying internal data reviews until after publication

• the submission of false data be controlled either through

- legislation mandating that violating institutions become ineligible for Title IV aid

and/or

- highly publicized “name and shame” lists of cheaters- exclusion from the ED’s college search resource

APPENDIX

Responder’s Biography

Review of Pilot Alumni Survey

A graduate of Princeton (AB cum laude) and Penn (JD plus MBA studies at Wharton), Tom Benghauser spent twenty-plus years in the US, Germany, South Africa, and the UK in high level management positions with ad agency giants such as JWT.

In 1986 he co-founded and was managing director of the Brand Dynamics Group, a London-based consultancy specializing in the development of predictive models of brand behavior for clients that included Kodak, Glaxo SmithKline, RJ Reynolds, J. Walter Thompson, and Philip Morris.

In 1993 he founded and until 2004 was CEO of a highly successful UK-based loyalty enhancement agency that carried out its unique satisfaction research-based customer-dialogue programs for marketers such as Vauxhall Motors, Ikea-owned Habitat, General Motors Europe, Fiat-Alfa Romeo, and Saab.

He returned to the U.S. – Denver – in 2005 with the intention of retiring but never got around to it and instead almost immediately began meddling in a lot of matters that at the time he knew absolutely nothing about.

College Straight Talk™

Pilot Satisfaction and Outcomes Survey

Recent Alumni:Kenyon, Mount Holyoke,

Penn, Princeton, Tufts

Findings

Methodology/Survey Instrument

CONTENTS

© 2011 – 2013 Tom Benghauser

Pilot Survey: Methodology

May – June 2011

Kenyon, Mount Holyoke, University of Pennsylvania, Princeton, Tufts

Bachelors recipients who graduated in 2006 - 2010

Sample respondents along with email addresses and other contact

information randomly extracted from the colleges’ on-line alumni

directories.

Pilot Survey: Methodology On-line using email invitations containing direct links to the survey instrument

52 questions/3 rating matrices (44 attributes) = 96 total (details page 12)

Opens with overall satisfaction rating (1 – 10 scale) followed by open ended “Why do

you say that?”

• Avoids the possibility of question-sequence induced response error on the single

most important question

• Maximizes responses to the single most important question

• Extensive verbatims (average word count = 41) provide especially rich insights

The final question is an open-ended one that invites respondents to provide

observations about the survey, including questions they did not understand,

questions they thought should have been included, etc.

Universe/population

Sample frameKenyon Holyoke Princeton Penn Tufts Total

Email Invitations sent 460 995 816 1,416 802 4,489Bounced 5 22 7 39 143 216Email invitations received 455 973 809 1,377 659 4,273Responses 192 240 192 275 205 1,104Reponse rate invitations delivered 42.2% 24.7% 23.7% 20.0% 31.1% 25.8%Reponse rate total invitations sent 41.7% 24.1% 23.5% 19.4% 25.6% 24.6%

All alumni from classes of 2006 - 2010All alumni from classes of 2006 - 2010 who provided email

addresses to their college's on-line alumni directory

Response rates are in-line with current trends for on-line surveying despite

• the absence of the participation incentives (e.g. chances to win major donations to

respondents’ educational institution of choice) that will be included in the email

participation-invitations for future surveys

• only one round of email reminders

Pilot Survey: Methodological Findings

Kenyon Holyoke Princeton Penn TuftsEntering Class Sizes 1362 1563 4734 12020 5244Graduation Rates 86.1% 82.4% 96.0% 95.3% 86.0%Universe: Graduates 1172 1288 4545 11459 4510Responses 343 335 337 374 359% Margin of Error 4.58 4.61 5.15 4.98 4.96


The resulting response volumes resulted in respectable margins of error at

the 95% level of confidence. Again, these were obtained without the use of

incentives or extensive reminders cum cajoling.

Comparisons between a key, independently established characteristic of the

universe as defined and that characteristic as measured in the survey

strongly suggest that our findings are fully representative of the universe.

• The Common Data Sets developed and employed by the College Board

and posted on numerous colleges’ website were the sources of the data

on this key demographic variable.


Number Pct Pct NumberNonresident alien 339 14.3% 11.7% 18Hispanic 126 5.3% 3.2% 5Black, non-Hispanic 115 4.8% 1.9% 3White, non-Hispanic 1445 60.9% 68.8% 106American Indian or Alaska Native 23 1.0% 0.0% 0Asian or Pacific Islander 325 13.7% 14.3% 22Total 2373 100.0% 100.0% 154

r = 0.997

Number Pct Pct NumberNonresident alien 2.6% 4.3% 11Hispanic 53 3.1% 3.5% 9Black, non-Hispanic 57 3.3% 0.8% 2White, non-Hispanic 1485 86.9% 88.7% 227American Indian or Alaska Native 8 0.5% 0.0% 0Asian or Pacific Islander 60 3.5% 2.7% 7Total 1708 100.0% 100.0% 256

r = 0.999


r = 0.999

Mount Holyoke Classes of 2006 - 2010

Kenyon Classes of 2006 - 2010Universe: Common Data

Sets CST Survey

Universe: Common Data Sets CST Survey

Princeton Classes of 2006 - 2010


The high correlations between the ethnic compositions measured by our

survey and the independently established census-based compositions of the

alumni universes is especially significant given both

• the great explanatory power of the ethnicity variable in analyzing our

other measurements

• the high interest in the variable within the higher education community

Pilot Survey: Methodological Outcomes


r = 0.997

Number Pct Pct NumberNonresident alien 2.6% 4.3% 11Hispanic 53 3.1% 3.5% 9Black, non-Hispanic 57 3.3% 0.8% 2White, non-Hispanic 1485 86.9% 88.7% 227American Indian or Alaska Native 8 0.5% 0.0% 0Asian or Pacific Islander 60 3.5% 2.7% 7Total 1708 100.0% 100.0% 256

r = 0.999


r = 0.999

Mount Holyoke Classes of 2006 - 2010

Kenyon Classes of 2006 - 2010Universe: Common Data

Sets CST Survey


Princeton Classes of 2006 - 2010


Notes on Self Selection-Induced Error

Given the high stakes that are not infrequently involved in customer

satisfaction and outcomes research, it is hardly surprising that, from

virtually the beginning of the discipline, hypotheses have been put forward

that members who choose to take part in such surveys are “different” in

ways that make them collectively unrepresentative of the overall

populations of which they are members.

The reality may indeed be that, say, extremely satisfied or dissatisfied

customers are in general more likely to find the time or go to the trouble of

taking part in satisfaction surveys than are other members of a particular

body of customers or users.

The issue, however, is not whether response bias exists but rather whether

it manifests itself with the alumni bodies of certain colleges to a greater or

lesser degree than it does with the alumni bodies of other colleges.

This is because

• what we are attempting to do is to compare colleges relative to one

another and not to measure absolute Truth with a capital T.

• nothing in the literature documents that response bias

disproportionately effects some brands or even product/ service

sectors more than it does others.


Finally, the reason most frequently posited for hypothesized differences in

response rates among invitees whose satisfaction levels differ is that

especially unhappy or happy ones have much stronger emotional or other

motivations for taking the time and/or going to the trouble of taking part.

To the extent that this reason is valid, response differentials will logically

have much less impact on on-line surveys - where the “effort” involved in

participating is significantly lower than it is for in-person, hard-copy, or

telephone surveys.


The Survey Questions

Analytical and problem-solving skills/ability to think criticallyKnowledge of a particular field or disciplineLeadership abilitiesActive interest in community serviceAbility to work independentlyAbility to form and retain friendshipsAbililty to have a good rapport with people holding different beliefsAbility to work effectively with different kinds of peopleReligious valuesAbility to communicate well orallyCompetitivenessAbility to work cooperativelyAbility to relax and enjoy leisure timeAbility to write clearly and effectivelyAbility to adjust to changeAbility to think creatively, approach problems from unconventional perspectivesAbility to think like an entrepreneur, willingness to take risksSelf-confidence

First NameLastNameCustom Data

RespondentIDStartDateEndDateIP AddressEmail Address

From what 4-year college or university did you get your undergraduate (bachelors) degree? (Please select from drop-down What year did you graduate?How satisfied are you overall with your undergraduate experience at [Q1] Why do you say that?What degree did you earn?What was your undergraduate major? (Please make your selection from the drop-down list)How much would you say you gained in each of the following areas from your time at [Q1]?

Students of different races, ethnicities, and nationalities socialized outside of classesOn-campus housing was comfortableStudents were open to opinions and viewpoints different from their own.When I arrived the faculty and administration helped me to feel welcome and “at home”.I made close friends with other students and am still in regular contact with themI made close friends with other students during my undergraduate yearsThere was extensive use of marijuana on campus during my time thereThere was extensive consumption of alcoholic beverages on campus during my time thereThe college offered many social opportunitiesThere was very little cheating during exams and tests and on writing assignments and other homework.I was able to choose from a large number of extracurricular activitiesOverall the student body was more interested in having fun than in studying

Please indicate the extent to which you agree - or disagree - with the following statements about [Q1].

Please provide your best recollection of how much the training cost in US$.Have you ever made a donation to [Q1]?When did you make your most recent donation?

As part of your college application process, did you take either the ACT or SAT? Please provide your best recollection of how you scored on the SAT 2-PartPlease provide your best recollection of how you scored on the SAT 3-PartPlease provide your best recollection of how you scored on the ACTDid you receive any special training in preparation for the ACT or SAT? Did you or your family have to pay for this training?

Thinking back to when you originally applied to college, which other colleges did you apply to?Was [Q1] the college you most wanted to attend when you were applying to college?Which other college was your first choice?Were you admitted to [Q11]?You said that [Q11] was your first choice and that you were admitted. Why did you attend [Q1] instead?Knowing what you know now, do you think you would be happier and/or more successful today if you had instead gone to a college other than [Q1]?

What kind of high school did you attend before enrolling at [Q1]? Where is that secondary school located?

Professors and other faculty were readily available to give me help and/or adviceThere was extensive sexual activity between studentsMost of my classmates were smarter than I amI frequently had discussions with faculty members outside of classThere was a lot of "school spirit"The overall ‘atmosphere’ on campus was serious and encouraged learningStudents tended to room and socialize with members of their own race, nationality, religion, sexual orientation, and/or social class.The student body was diverse in terms of race, nationality, religion, gender, sexual orientation, and social class.There was hostility on campus towards students of homosexual and transgender sexual orientationsOverall the political climate on campus was liberalThere was extensive use of so-called hard drugs such as cocaine and methamphetemine on campus during my time thereI spent most of my time reading, writing, and studyingHomosexuals were treated just like anybody else by students, faculty, and the administration.

I did not receive financial aidNeed-based grants from my collegePell grantSubsidized loansOther loansWork-study employmentOther forms of aid (e.g., merit scholarships and prizes)

Working full-timeLooking for workObtaining a graduate degreeOther

What university are you now attending for your graduate or professional studies?

Here are a few more statements about [Q1] that we'd appreciate your indicating how much you agree with.

During your undergraduate years did you belong to a fraternity, sorority, or similar social organization? Did you receive financial aid as an undergraduate? Mark all that apply.

How satisfied are you with the level of financial aid [Q1] provided?At the time you graduated, approximately what was the total amount you had borrowed to finance your undergraduate


Did your parents or other relatives attend [Q1]?

Did the benefits you received from attending [Q1] outweigh the financial costs to you and your family? Mark the best answer

Finally, feel free to use the comment box below to tell us anything you'd care to about our survey and its objectives. And if you have any questions, you may contact us via email at [email protected]

If you are a citizen of the United States, what is your race or ethnicity?What is your gender?What is your sexual orientation? What is your current personal status? Mark the one best answer.

And what is your annual salary now?Please tell us about the graduate and professional degrees, if any, that you have either already received or for which your are How well did [Q1] prepare you for graduate or professional school? How well do you feel [Q1] prepared you for what you are doing now? Where do you live now? What is your nationality? What country or countries do you hold a passport from?

For what type of organization are you now working on a paid or volunteer basis? Mark the one where you spend the most time.What branch of the military are you in?What is your rank?Which of the following best describes your current position? What was your per annum starting salary in the first full-time job you took after graduating from [Q1]?

Pilot Survey:Substantive Findings

Highlights

Overall Satisfaction with the Undergraduate Experience

8.39

8.06

8.498.58

8.76

7.707.80

7.908.00

8.108.20

8.308.40

8.508.60

8.708.80

8.90

Kenyon Holyoke Princeton Penn Tufts

The vertical red bars represent 90% confidence intervals, meaning that there is only a 1 in 10 chance that the true average satisfaction score for a particular college falls outside the range represented by its bar.

Nationality

98.8%

83.6%92.5% 88.7% 92.5%

16.4%7.5% 11.2% 7.6%

1.2%

0.0%

100.0%

Non-US

US

Holyoke TuftsPrincetonPennKenyon

Type of Secondary School

28.7%19.8%

35.6% 32.0% 31.5%

0.0%

100.0%

Private Secondary SchoolOutside US

State Secondary SchoolOutside US

Home Schooling

Charter Secondary School

Public Secondary School

Private Parochial SecondarySchool

Private Secondary School


Race/Ethnicity

91.2%

75.2% 76.6%68.6%

78.1%

0.0%

100.0%

Two or more races, nonHispanic/Latino

Native Hawaiian or other PacificIslander, non Hispanic/Latino

Hispanic/Latino

Asian, non Hispanic/Latino

Black or African American, nonHispanic/Latino

White, non Hispanic/Latino


Kenyon Holyoke Penn Princeton TuftsHispanic/Latino 4.4% 3.5% 6.0% 8.3% 4.7%Black or African American, non Hispanic/Latino 0.7% 2.1% 3.6% 7.4% 3.9%White, non Hispanic/Latino 91.2% 75.2% 76.6% 68.6% 78.1%Asian, non Hispanic/Latino 1.5% 14.9% 10.8% 9.9% 6.3%Native Hawaiian or other Pacific Islander, non Hispanic/Latino 0.7% 0.0% 0.0% 0.0% 0.0%Two or more races, non Hispanic/Latino 1.5% 4.3% 3.0% 5.8% 7.0%

Received Financial Aid

65.0%

74.6%

46.9%

60.1%

44.3%

30.0%

80.0%

Kenyon Mt Holyoke Penn Princeton Tufts

Satisfaction With Financial Aid

7.827.33

7.93

9.08

7.98

5.00

10.00

Kenyon Holyoke Penn Princeton Tufts

1 = Not Satisfied at All, 10 = Completely Satisfied

Amount of Loans at Graduation

$25,730

$33,372

$24,829

$8,277

$24,217

$5,000

$35,000


Yes: Benefits Outweighed Costs

69.5% 66.1%72.2%

85.3%

63.6%

0.0%

50.0%


Received special training in preparation for the SAT or ACT

36.4% 39.1%

52.9%

39.6%

50.7%

0.0%

100.0%


Paid for Special SAT or ACT Training

83.9%76.5%

86.0% 87.7%78.1%

0.0%

100.0%


Cost of Special SAT or ACT Training

$501

$1,138

$994

$1,105

$889

$400

$700

$1,000


Kenyon Holyoke Penn Princeton Tufts TotalsWorking fulltime 52.4% 57.6% 63.4% 54.3% 60.8% 57.9%Working parttime 8.4% 5.8% 2.2% 4.3% 2.1% 4.5%Looking for work 3.5% 4.7% 6.0% 4.3% 3.5% 4.5%Obtaining a graduate degree 32.9% 27.9% 26.8% 30.4% 28.5% 29.1%Raising a family 0.7% 0.0% 0.0% 0.0% 0.2% 0.3%Doing volunteer work 1.4% 1.7% 0.5% 2.2% 2.8% 1.7%Being retired 0.0% 0.0% 0.0% 0.0% 0.7% 0.1%Other 0.7% 2.3% 1.1% 4.5% 1.4% 1.9%Total 100.0% 100.0% 100.0% 100.0% 100.0% 100.0%


Now Working Full Time

52.4%57.6%

63.4%54.3%

60.8%

0.0%

50.0%

100.0%


Starting Salary in First Job(Now Working Full Time)

$33,703 $35,744

$56,183$48,356

$37,756

$20,000

$40,000

$60,000

$80,000


Current Salary(Working Full Time)

$43,792$39,004

$81,898

$58,384$50,424

$20,000

$40,000

$60,000

$80,000


Starting Salary in First Full-Time Job

$30,587$34,609

$53,692

$44,727

$38,225

$20,000

$40,000

$60,000


Degree Earned

100.00% 98.50%

78.50% 82.70% 83.10%

21.50% 17.30% 16.90%

0.00%

100.00%

Kenyon Holyoke Penn Princeton Tufts

BS

BA

Kenyon Holyoke Penn Princeton Tufts TotalArchitecture 0.0% 0.4% 0.0% 0.7% 0.0% 0.2%Area and ethnic studies 1.3% 2.1% 0.6% 1.5% 0.7% 1.2%Biological/life sciences 11.6% 14.0% 11.1% 9.4% 13.3% 11.9%

Business/marketing 0.6% 0.0% 21.2% 1.1% 1.7% 5.2%Communication/journalism 0.0% 0.7% 5.7% 0.0% 0.0% 1.4%Computer and information sciences 0.0% 1.8% 0.6% 2.2% 0.3% 1.0%Education 0.0% 0.7% 0.3% 0.0% 0.7% 0.3%Engineering 0.0% 0.0% 3.8% 13.5% 0.3% 3.3%Engineering technologies 0.0% 0.0% 0.0% 0.4% 0.3% 0.1%English 17.7% 4.9% 7.6% 6.0% 8.5% 9.1%Foreign languages and literature 4.8% 4.9% 2.2% 6.0% 4.4% 4.4%Health professions and related sciences 0.0% 0.7% 1.9% 0.0% 1.0% 0.7%History 8.4% 6.3% 7.9% 12.4% 5.1% 7.9%Interdisciplinary studies 5.8% 7.0% 3.8% 1.1% 6.1% 4.8%Liberal arts/general studies 4.2% 4.2% 2.2% 1.9% 5.8% 3.7%Mathematics and statistics 3.9% 3.5% 1.6% 2.2% 2.4% 2.7%Natural resources/environmental science 0.0% 1.4% 0.3% 0.0% 0.0% 0.3%Philosophy and religious studies 2.3% 2.1% 1.3% 3.4% 3.4% 2.4%Physical sciences 2.6% 3.2% 1.9% 4.5% 1.0% 2.6%Politics, Political Science 6.1% 11.2% 8.5% 6.4% 10.2% 8.5%Psychology 10.3% 8.4% 6.0% 6.0% 13.6% 8.9%Public administration and social services 0.0% 0.0% 0.0% 1.1% 1.0% 0.4%Science technologies 0.3% 0.7% 0.3% 0.0% 0.0% 0.3%Social sciences 12.6% 16.5% 9.2% 19.9% 16.7% 14.7%Theology and religious vocations 0.6% 0.7% 0.0% 0.4% 0.0% 0.3%Visual and performing arts 6.8% 4.9% 1.6% 0.7% 3.4% 3.5%

100.0% 100.0% 100.0% 100.0% 100.0% 100.0%

Major

Pilot Survey:Selected Detailed Findings

Gained in Analytic Skills

9.00 8.93 8.748.46

7.92

7.00

8.00

9.00

10.00

Holyoke Kenyon Princeton Penn Tufts

1 = Gained Nothing, 10 = Gained a Lot

How much would you say you gained in each of the following

areas from your time at [Q1]?

Gained in the Ability to Write Clearly and Effectively

9.188.78 8.77

8.237.80

7.00

8.00

9.00

10.00

Kenyon Holyoke Princeton Penn Tufts




Gained in the Ability to Communicate Well Orally

8.45 8.45

7.97 7.87

7.387.00

8.00

9.00

10.00

Holyoke Kenyon Princeton Penn Tufts




Gained in Competitiveness

7.89

7.46 7.44

6.306.116.00

7.00

8.00

9.00

Penn Mt Holyoke Princeton Tufts Kenyon




Gained in the Ability to Adjust to Change

8.35 8.20 8.15 8.077.76

6.00

7.00

8.00

9.00

Mt Holyoke Kenyon Princeton Penn Tufts




Gained in the Ability to Think Creatively, Approach Problems from Unconventional Perspectives

8.64 8.61

8.13 8.04

7.36

6.00

7.00

8.00

9.00

Kenyon Mt Holyoke Princeton Penn Tufts




Gained in Self Confidence

8.758.49

7.837.64 7.55

6.00

7.00

8.00

9.00

Mt Holyoke Kenyon Penn Tufts Princeton




I frequently had discussions with facultymembers outside of class

7.78 7.53

5.85 5.74 5.50

0.50

1.50

2.50

3.50

4.50

5.50

6.50

7.50

8.50

9.50

Holyoke Tufts Princeton PennKenyon

Please indicate the extent to which you agree - or disagree - with

the following statements about [Q1].

1 = Don’t Agree at All, 10 = Agree Completely

Professors and other faculty were readily availableto give me help and/or advice

9.188.87

7.76 7.737.35

7.00

8.00

9.00

10.00

Kenyon Holyoke Princeton Tufts Penn

Holyoke TuftsPrinceton PennKenyon

Please indicate the extent to which you agree - or disagree - with

the following statements about [Q1].

1 = Don’t Agree at All, 10 = Agree Completely

response: postsecondary institution ratings system [docket id ed–2013–ies–0151] prepared by...

Documents

education statistics

overview research

higher education establishment

contents background

overview data needs

issues appendix slide

ratings ones

president of