2008 FAEIS Triennial Evaluation

The purpose of the FAEIS Triennial Evaluation is to conduct a comprehensive assessment of FAEIS once every three years in order to improve the usability of the information provided by FAEIS, monitor progress during the three-year grant cycle and plan for the future.

Objectiveso Describe selected demographic and other variables that have potential association with FAEIS use.

o Determine the level of usefulness of FAEIS data resources and applications.

o Determine the level of satisfaction with selected components of FAEIS.

o Determine levels of adoption of FAEIS and factors that explain the levels of adoption.

o Determine the relationship of selected demographic and usage variables (involvement, use, experience, primary job title) with perceived usefulness and satisfaction with FAEIS.

Methodology

o FAEIS Team developed first draft of questionso Peer Panel and USDA partners reviewed and made suggestions to draft questionso Peer Panel members were used to pilot test the on-line survey instrument. Changes made accordingly.o Web survey administered by the VT Center for Survey Researcho 1,145 email addresses from FAEIS users used as survey sample group. Reduced to 1,120 by request of sample members.o Follow-up reminders sent to sample memberso 396 respondents completed surveys for a response rate of 35 %

Question 1: My professional affiliation is best described as ?

2007 Percent of

Total (N=308)

2008 Percent of Total (N=285)

2009 Percent of

Total (N=397)

Staff 49 46.7 30.7

Faculty 18 7.7 11.8

Department Head 6 19.3 19.6

College Associate/Assistant Dean 16 15.4 15.1

College Dean or VP 5 7.7 9.3

University-level Administration 5 2.5 4.5

Other College-level Administration 5.0

Government -- 0.7 0.0

Non-Govt., Organization 0.0

Institutional Research 3.5

No answer 1 --

Analysis – fewer staff respondents in ‘09; greater number of faculty and administrator respondents in 2009 compared to 2007 & 2008.

# of Respondents by Institution Type

1862 – 1831890 – 651994 – 6PNLG – 109Private – 21Did not report -12

n=396

Question 2 2009 Respondents by Institution Type

n=384

Question 3 Indicate which of the following organizations, if any, is your

primary affiliation.

Organization Percentage of Responses

APS 10.7 %

AASCARR 9.6 %

AEA/ARD 1.3 %

BOHS 8.6 %

SAF 3.4 %

AAVMC 0.8 %

CAFCS 6.5 %

NAUFRP 4.7 %

APLU 16.1 %

USDA 1.8 %

Other 10.2 %

Does not apply 26.3 %

Question 45Years in your current position?

N=383

Questions 43 & 44 Demographics

Female 55.70 %

Male 44.30 %

Asian 1.60 %

African American/ Black 8.80 %

American Indian / Alaskan Native 0.50 %

Hispanic 0.80 %

Native Hawaiian/ other Pacific Islander

0.50 %

White 85.20 %

Other 2.60 %

Three-Year Trends: Question 4 : My total experience using FAEIS is ?

Analysis – Users with > 3 yrs experience is trending upward while the other two groups are trending down.

Question 5My Involvement with FAEIS Includes:

(select as many as apply; all possible combinations displayed)

N=396

2009 Results

Three-Year Trends:Question 5: Involvement with FAEIS of Survey Respondents

( “ none” and “ no answer” responses eliminated in ’07 & ‘08 )

Analysis – Data Entry and Product Developer survey respondents trending downward; Users of FAEIS products is trending upward.

Three-Year Trends: Question 6: How often have you used FAEIS in the past 12 months ?

Analysis –The more frequent use categories (3-4 times & >4 times) are trending downward. The less frequent use categories (not at all & 1-2 times) are trending upward.

Three-Year Trends:Question 7: “My level of adoption (use) of FAEIS is best described as:”

(no answer – eliminated from analysis in ‘07 & ‘08)

Analysis – Only “unaware” and “use it but not sure of long-term use” are trending upwards. “Do not use” trending downward.

“ My level of adoption (use) of FAEIS is best described as:” (no answer – eliminated from analysis in ‘07 & ‘08)

2007Mean

2008Mean

2009Mean

Data Entry Users Only 2.49(N=159)

2.30(N=152)

2.32(N=157)

All Other Users 3.16(N=117)

3.24(N=125)

2.93(N=240)

Total 2.78(N=276)

2.73(N=277)

2.69(N=396)

1=unaware

2=aware of it but do not use it

3=Use it but not sure of long term use

4=Use it and it is integral to my job

5=Integral to my job and expect long term use

6=So important as to never be without FAEIS

Analysis – Average level of adoption of FAEIS decreased in ’09 for non-data entry respondents

Question 21Who are the users of FAEIS data at your institution/organization/agency ?

Figure 13. Users of FAEIS at Survey Respondent

Institutions/Organizations/Agencies

1.1%

16.6%

2.4%

39.2%

40.6%

0% 5% 10% 15% 20% 25% 30% 35% 40% 45%

Policy makers

Other

Faculty/researchers

Do not know

University administrators

n=391

Question 9Is FAEIS collecting data from the institutions which you consider

your peers?

n=395

n=271

Question 10If you answered no to:

“Is FAEIS collecting data from the institutions which you consider your peers?,” Which Institutions are missing ?

o Private schools with agriculture programs o Family and Consumer Sciences programs at NLG

Question 11Do a sufficient number of institutions report data to allow for you to

have a meaningful and helpful peer analysis?

n=389

n=210

Three-Year Trends: Question 13: Describe your level of satisfaction with FAEIS components.

(i.e., newsletter, help desk, data entry, report builder, instructions) (1=not satisfied; 5 = very satisfied)

Analysis – Higher and lower satisfaction ratings are trending downward. The neutral rating (3) is trending upward.

Question 14How useful to you are the following FAEIS products and resources?

Figure 9. Respondent Ratings of Utility of FAEIS Products and Resources

5.8

6.6

8.6

5.3

6.1

8.1

7.6

3.5

5.3

5.3

6.8

5.1

7.3

5.1

24.7

30.1

11.9

23.7

18.2

9.1

11.9

18.9

13.4

8.1

19.7

11.6

5.3

7.3

0.0 10.0 20.0 30.0 40.0

FAEIS website

FAEIS electronic newsletter

FAEIS data CD

Classification of Instructional Programs (CIP) code lists

Resource links on the FAEIS website

Teaching Workshop and Awards website

USDA/Science and Education Resources Development(SERD) Funded Projects information

Not Useful At All Not Very Useful Somewhat Useful Very Useful

Highest Ratings:

o CIP listso Newslettero Web site

Lowest Ratings:

o Teaching Awards / Workshops websiteo FAEIS Data CD

n=396

Question 14 hAre there any other FAEIS products or resources you would like to

add to the list ?

Some Responses:

o Salary data for placement data o Trend data such as degree programs that are changing over a 10 year cycle

o Summary of IPEDS major trends that influence food and agricultural sciences o Similar data from 1994 land grant universities obtained from sister institutions.

Question 8 Does the CIP system adequately allow you to classify your

instructional programs and/or address your reporting needs?

n=392

Question 12What aspect of FAEIS is most important to you ?

Most Common Responses: o Student Enrollment data

o Comparisons between institutions

o Trends in enrollment, degrees awarded

o Faculty salaries

Question 15For which of the following purposes have you used FAEIS data

and/or reports? ( Please check all that apply)

N=393

2009 Results

Analysis – Benchmarking and trend analysis are the two most common uses of FAEIS data.

Question 16 Please rate the usefulness of the data resources in FAEIS (for student

enrollment, degrees awarded and placement, disciplines, gender, ethnicity, faculty salaries, etc.) (1 = not useful, 5 = very useful)

2009 Results

N=397

Three-Year Trends: Question 16 : Please rate the usefulness of the data resources in

FAEIS (for student enrollment, degrees awarded and placement, disciplines, gender, ethnicity, faculty salaries, etc.).

Analysis – The higher usefulness scores (4, 5) are trending upward. The lower usefulness scores (1, 2) are stable & trending downward.

Question 17Please indicate how much confidence you have in the following

FAEIS data resources.

Table 15. Respondent Confidence in Types of FAEIS Data

Not Confident

At AllNot Very Confident Neutral

Somewhat Confident

Very Confident

Do Not Know

Total college enrollment 2.3 3.1 19.4 28.2 26.7 20.2

Student enrollment by CIP 2.6 6.5 18.8 29.8 19.6 22.7

Student demographics 1.8 3.4 20.7 32.1 21.5 20.5

Degrees awarded by CIP 2.3 3.4 20.0 29.9 22.3 22.1

Student placement by CIP 2.6 8.1 24.4 26.0 15.1 23.9

Faculty headcounts 1.3 3.1 19.8 28.9 25.3 21.6

Faculty demographics 1.3 2.9 21.1 27.9 24.7 22.1

Faculty salaries 1.8 4.1 20.7 28.2 23.1 22.0

International programs 1.3 3.4 26.2 24.1 15.3 29.8

1st

3rd

2nd

= Least confident

= Most confident n=386

Question 18Is FAEIS the primary place you go to seek higher education data and other information relating to the food and agricultural sciences, natural resources

and conservation, and family and consumer sciences/human sciences ?

n=391

Question 19Do you use any of the following databases ?

Figure 12. Databases Utilized by FAEIS Survey Respondents(Percentage of Respondents Selecting Database on Survey)

11.9

14.6

17.7

17.9

22.7

44.9

0 5 10 15 20 25 30 35 40 45 50

Other

Oklahoma State Faculty Salary Data

Department of Education NationalCenter for Education Statistics (NCES)

College and University ProfessionalAssociation for Human Resources

National Science Foundation (NSF) -Survey of Earned Doctorate Data

Food and Agricultural EducationInformation System (FAEIS)

Other:o CRIS, o NAUFRP, o USDA databases n=280

Question 20To what extent does FAEIS add value to other information sources that

are available ?

n=384

Question 24 Have you contacted the FAEIS Help Desk for assistance ?

Figure 14. Contact with FAEIS Help Desk Among

Survey Respondents

Yes, emailed the help desk

14.4%

Yes, telephoned the help desk

11.3%

No, have not contacted the help

desk61.6%

Yes, telephoned and emailed the

help desk12.6%

n=388

Question 25How would you rate the support you received from the Help Desk ?

Figure 15. Respondent Ratings of FAEIS Help Desk

Excellent62.1%

Good29.4%

Poor0.7%

Fair5.2%

Do not know2.6%

n=153

Question 26If you have contacted the FAEIS Help desk , how could its operation be

improved ? (n=146)

22 % - No improvement needed. 1 % - Increase hours of operation 0.3 % - Improve email response time12.4 % - Do not know 1.8 % - Other suggestions:

o Help Desk staff is limited by the lack of thorough, well-edited, accurate and consistent documentation, web screen content and communications.o Information provided by the Help Desk is not consistent from time-to-time.o Develop databases more thoroughlyo Help Desk employees need better trainingo Clear understanding of the parameters of the data being enteredo More understanding of the difficulties institutions are having entering the data

Question 27How would you rate the ease of navigation of the FAEIS web site?

Figure 16. Respondent Ratings of Ease of Navigating the FAEIS Web Site

Very difficult1.3%

Difficult7.4%

Neither difficult nor easy34.9%

Easy26.2%

Very easy5.4%

Do not know24.9%

n=390

Question 28Have you used customized FAEIS data reports produced by the FAEIS

Report Builder application ?

Figure 17. Levels and Types of Usage of the FAEIS Report Builder Application for Customized Data Reports

No, I have not used any

customized FAEIS data

reports70.7%

Yes, I developed the customized FAEIS reports

myself9.7%

Yes, my staff have developed

customized reports for me

2.3%

Yes, the FAEIS Help Desk

developed the customized

reports for us4.1%

Do not know6.4%

Does not apply6.9%

n=393

Question 29 Have you personally used the FAEIS Report Builder application ?

n=390

Question 30 How well does the FAEIS Report Builder meet your needs ?

Figure 18. Survey Respondent Ratings of How Well the FAEIS Report Builder Meets Their Needs

Excellent11.3%

Do not know5.6%

Poor5.6%

Fair25.4%

Good52.1%n=71

Question 31How easy is it to use the FAEIS Report Builder ?

Figure 19. Respondent Ratings of the Ease of Use of the FAEIS Report Builder (Percentages Reporting)

51.4

24.3

17.1

2.9

2.9

1.4

0 10 20 30 40 50 60

Very difficult

Very easy

Do not know

Difficult

Easy

Neither difficult nor easy

n=70

Question 32Are the instructions for the FAEIS Report Builder clear ?

Figure 20. Percentages of Survey Responses For Survey Item:

"Are the Instructions for the FAEIS Report Builder Clear?"

Yes59.2%

No23.9%

Do not know16.9%

n=71

Question 33Would training on how to use the FAEIS Report Builder be useful to you?

Figure 21. Percentage of Responses to "Would Training on How to Use the FAEIS Report Builder be Useful to You?"

No26.8%

Yes54.9%

Do not know18.3%

n=71

Question 34Do you enter FAEIS data, or are you responsible for data

entered into FAEIS ?

n=389

Question 35How would you rate your ability to obtain the following data in the format

required by FAEIS ?

Enrollments by degree level, discipline,

gender, and ethnicityDegrees awarded by degree level, discipline, gender, and ethnicity

N=262 N=260

Question 35How would you rate your ability to obtain the following data in the format

required by FAEIS?

Faculty by discipline, rank, gender, and

ethnicityPlacement by degree level into various CIP

disciplines and work categories

N=257 N=260

Question 36 Are data on placement of students collected at your institution ?

Figure 22. Survey Findings Related to Collection of Student Placement Data at Respondent Institutions (Respondents Could Select All Response Options

That Applied)

1.1%

1.5%

1.9%

13.7%

16.0%

22.5%

26.3%

30.2%

Yes, data collected but not available

Does not apply

Other

Do not know

Yes, other office in college collects these data

No, placement data are not collected

Yes, our office collects these data

Yes, another office at institution collects these data

N=134

Question 37At your institution, from where do you acquire the data for the various

surveys ?

Figure 23. Sources Within Institution for Acquiring Data for Various Surveys (Respondents Could Select All Survey Response Options That Applied)

18%

15%

32%

1%

18%

16%

28%

1%

12%

21%21%

15%

1%

5%

22%

14%

9%

0%

12%13%

10%

17%

12%11%

0%

0%

5%

10%

15%

20%

25%

30%

35%

Department College Institutional Research Human Resources Other Office onCampus

Student Enrollment Degrees Awarded Faculty Student Placement International Programs

N=257-261

Question 38What time of year would be best to enter the following types

of FAEIS data ?

Figure 25. Respondent Perceptions of Best Time of Year to Enter FAEIS Data by Type of Data

19.2%

11.6%

6.1%

13.4%

8.6%

6.1%

10.1%

25.5%

15.4%

8.6%

5.8%

15.4%

8.3%

5.8%

6.6%

21.0%

6.8%

5.1%

3.8%

12.6%

0.0% 5.0% 10.0% 15.0% 20.0% 25.0% 30.0%

Fall

Winter

Spring

Summer

Student enrollment Degrees Awarded Faculty Student Placement International programs

N=253-260

Question 9What data entry method do you use for the following surveys ?

N=244-256

Figure 24. Respondent Institution Data Entry Methods for Different Data Types

12%

6%

9%

6% 5%6%

10%

6% 7%

4%

13%

26%

15%

21%

13%

15%

20%

24%

36%

15%

0%

5%

10%

15%

20%

25%

30%

35%

40%

Student enrollment Degrees Awarded Faculty Student Placement International programs

Online data entry Spreadsheet data entry Email Telephone call to FAEIS help desk

Question 41Are the instructions clear for FAEIS data entry methods that you use ?

N=240

If answered no, which instructions need to be improved?

o Would prefer questionnaire by mailo Navigation on screen a paino Difficult to discern how to move from in-progress to completedo Finishing an entry is difficulto Difficult to navigate and decide when to saveo Not sure what majors you want to be collectedo Submission instructions are vagueo Time frames for info requested needs to be stated clearly

Question 22How can FAEIS be more useful to your institution or be improved ?

Most Common Responses:

o Data input is cumbersome, needs to be easier, bothersome. o Need more information about what FAEIS is and what it can do for us.

o I need to use it more. o Concerned about the accuracy and credibility of the data.

o Complaints about CIP codes

Question 23 Please share any comments about when FAEIS was helpful to you or

about when you had a problem with FAEIS.

Most common “Helpful” Responses:

o Staff was helpful. (21)o Useful for faculty salary comparisons.(4)o Helpful in grant boilerplates (1)o Student diversity by CIP info (1)o Dropping / adding programs (1)o Growth of programs, trends in majors (2)o Long-term planning (1)

Most Common “Problem” Responses:

o Data not dependable or accurate.(6)o Data entry not user friendly (5)o Report builder not user friendly. (2)o Poor job of marketing / facilitating use of FAEIS (1)

SUMMARY

o The more years using FAEIS the greater the satisfaction with FAEISo Greatest satisfaction of FAEIS data and its use:  enrollment, institutional benchmarking, trend analysis, salary comparisons o Greatest satisfaction of FAEIS products:  CIP code list, e-newsletter, websiteo FAEIS report builder—good, but room for improvemento Most confident in FAEIS data:  college enrollment, faculty headcount, student/faculty demographics, degrees awarded:  least confident: international/student placemento Most users believe FAEIS adds-value & is used twice as often as any other similar databaseo Most users have not contacted help desk; but of those that have---90% rated it excellent or good.o Users get FAEIS data from many places—Depts/Colleges/IR/HR