office of budget, planning and analysis applied research center

Office of Budget, Planning and Analysis

Applied Research Center

UW-Stout Alumni Follow-Up Study

Study last conducted Spring 2008

Implemented in 1979

Currently conducted every other year

Purpose of Study:• Assess student satisfaction• Measure educational

effectiveness• Identify areas for improvement

Study Background

University of Wisconsin - StoutBudget, Planning & Analysis

UW-Stout Alumni Who Graduated in the Year 2002• Five years after graduation• 423 responded out of 1,381

surveys sent (31% response)

UW-Stout Alumni Who Graduated in the Year 2006• One year after graduation• 525 responded out of 1,513

surveys sent (35% response)

Participation


2002

2006

Types of Questions:Forty-four Likert-scale questions

& three qualitative questions• Personal development in skill

areas• Preparation for employment• Satisfaction with employment• Quality of education at UW-Stout• Effectiveness of programs at UW-

Stout• Current employment information

Survey Questions


Means for all Likert questions with t-test or ANOVA analysis (Student-Newman-Keuls post hoc test).

Representative themes were developed for comments to qualitative questions.

Data Analysis


Trends back to 1979

University-level data shared with leadership group for decision-making

Program data provided to program director for program improvement

Data Analysis (cont.)


20091999

19891979

Alumni received:Postcard to watch for survey in near future Survey packet mailed with letter from

Chancelloroption given to complete survey on web or

return in business-reply envelopeAdditional postcard sent to non-

respondents as reminderSurvey packet sent again to non-

respondents

Employer survey sent to employers of alumni

2008 Study Design


• Letter from Chancellor

• Undergraduate or graduate survey

• Program specific survey (if available)

• Business-reply envelope

2008 Study Design (cont.)


Alumni Survey Packets Included :

http://upload.wikimedia.org/wikipedia/commons/c/ce/Mail-envelope.png

Results of 2008 Study

UndergraduatesMean(5 pt. scale)

If you had it to do over again, would you attend UW-Stout?

4.3

How important UW-Stout was in contributing to:

Working in teams 4.2

Leadership 4.1

Availability of faculty in program courses

4.2

Graduates

Mean(4 pt. scale)

If you had it to do over again, would you attend UW-Stout?

3.6

Education at UW-Stout prepared me to consider the ethics of my profession

3.5

Availability of faculty in program courses

3.5

Highest Ratings


Results of 2008 Study

Undergraduates

Mean(5 pt. scale

)

Value of senior year course work activities in developing:

Preparation for community, civic and political roles

3.2

Financial management 3.0

How important UW-Stout was in contributing to appreciating:

The value of literature 3.2

History in context to current issues

3.2

Graduates

Mean(4 pt. scale)

How well co-curricular and extra-curricular activities prepared me for employment

2.8

Education experience at UW-Stout prepared me to:

Utilize computing and digital technology

2.8

View community service as my social responsibility

2.8

Lowest Ratings


One- and five-year responses from 2008 study were compared.

Comparison of 5-Year and 1-Year Alumni

Significant Findings

Undergraduates 2002 Graduates(5 Yrs After Grad)

2006 Graduates(1 Yr After Grad)

How did Stout contribute to:

Writing effectively 3.7* 3.6

Creative problem solving 4.0* 3.9

Rate aspects of education:

General educ. instruction 3.6* 3.5

Program instruction 4.0* 3.8

Availability of faculty in gen. ed. courses 3.8* 3.6

Availability of faculty in program courses 4.2* 4.0

Course availability (according to program sequence) 4.0* 3.8

Lab facilities and equipment 3.8* 3.7

Digital environment 3.6 3.8*

* Statistically higher ratings


One-, three-, and five-year responses from alumni who graduated in 2000 were compared.

Comparison of Alumni in 2000

Significant Findings

Undergraduates of 2000

Spring 2006Survey

Spring 2004Survey

Spring 2000 Survey

5 Yrs After Graduation

3 Yrs After Graduation

1 Yr After Graduation

Importance of experience at Stout contributing to: Organizing information 3.8 4.0* 4.0*

Rate aspects of education:

General educ. instruction 3.6 4.1* 3.6*

Program instruction 4.0 4.2* 4.1

Availability of faculty in general educ. courses 3.7 4.1* 4.1*

Would you enroll in the same program 3.8 3.9 4.1*

* Statistically higher ratings


Additional results from the 2008 study can be found at: http://www.uwstout.edu/bpa/ir/afu/2008index.html

Additional Results


Went through Lean Process December 2008Learned the concepts of Administrative

Lean (benefits of one piece flow & elimination of waste)

Brought different areas together on campusMailroom, alumni office, program director

Identified efficiencies

Lean Process


Why did we do this . . .

• Reduce process Lead Time• Get a better handle on end user due dates• Better defined customer requirements• Understand how better to schedule work to

meet due dates (don’t fall behind)• Identify potential technological solutions

(automation)

Lean Process (cont.)



Current State vs. Future State

Large Process

Smaller Process

UW Stout Alumni Survey ProcessFuture State Map

December 14, 2008 (page 1 of 1)

Update the Index

page

Meetings w/PD’s who requested

one.Obtain email

addressesOrder Post

Cards order

General Surveys

P/T= 15 min

C/A= 100%

On Hold until Mid Dec

P/T= 60 min

C/A= 100%

Distribution package (survey

info, Q/A Timeline,

Examples, Mktg)

Request alumni mailing list #

IN

1 day

4670 min 360 min

50 day

1275 min

14 day

360 min

21 day

1920 min

7 day

Outsource the mailing process

Send Postcards and List to Mailroom

Notify PD’s the survey is coming.

Get in the mail.

Send email to

all alumni on list

Process undeliverables update list &

remove responses

Obtain mailing list

from Alumni office, get ID

#’s

Use Senior Salute as a mailing tool

P/T= 120 min

C/A= 100%

Repeat Post Card Process

Send Program

Specifics to Duplicating.

IN IN

21 day

1 day

P/T= 15 min

C/A= 100%

P/T= 5 min

C/A= 100%

Edits to online survey

Personalize notification

to PD’s

P/T= 60 min

C/A= 100%

P/T= 750 min

C/A= 100%

Send Mailing

to Printer

P/T= 600 min

C/A= 100%P/T= 600 min

C/A= 100%

P/T= 5 min

C/A= 100%

IN

14 day

IN

3 day

IN

28 day

IN

3 day

IN

28 day

P/T= 60 min

C/A= 100%P/T= 15 min

C/A= 100% P/T= 30 min

C/A= 100%

Program Directors

and Senior Mgmt

P/T= 60 min

C/A= 100%

P/T= 105 min

C/A= 100%

Repeat Post Card Process Again

P/T= 30 min

C/A= 100%

P/T= 105 min

C/A= 100%

P/T= 30 min

C/A= 100%

20 min 1410 min

1 day

65 min

21 day

15 min

1 day

180 min

0

15 min

3 day

90 min

3 day

935 min

28 day

30 min

14 day

105 min

28 day

Do Qualitative & Statistical analysis on General & Employer surveys

Provide update to

PD’s

P/T= 360 min

C/A= 100%

IN

14 day

Export survey results to do data entry

Do data entry from paper survey &

employer survey

Send to all PD’s

electronically

Run Statistical Analysis on

Program Spec Surveys,

Program spec reports, SPSS Report version

P/T= 750 min

C/A= 100%

IN

50 day

P/T= 30 min

C/A= 100%

IN

21 day

P/T= 720 min

C/A= 100%

P/T=360 min

C/A= 100%

IN

7 day

Narrative on

Employer & General surveys

P/T= 1200 min

C/A= 100%

IN

56 day

Process Time= 11,450 minutes or 190.8 hours

Lead Time= 247 daysTotal

Complete & accurate = 100 %

End of Oct

Mid Sept 1st wk

in Jan

1st of Feb

1st of Mar

End of Mar

Create Employer

survey & send out

P/T= 200 min

C/A= 100%

End of May

P/T= 4640 min

C/A= 100%

P/T= 525 min

C/A= 100%

Develop Web Surveys & index page

Develop PD help

form

Develop PD Packet

Develop our marketing plan

Contact IT about

retaining student email

addresses

Prepare a piece for

Senior Salute

Link to ID #

Create a new postcard

Confirm now to add ID to

Postcard

Develop Web Survey & index

page

Investigate an email service to

find emails

Contact provost about

SPSS report

Develop format for

SPSS report

Clean Database

Mid June

Mid Sept

Report out

56 day


Future State - Diagram

* Current state was under-estimated on actual time it has taken to complete the process.


Project Metrics & Goals

Metric Current FutureNet

Improvement

Lead Time 269.5 days* 247 days 22.5 days

Process Time 67.6 days 23.8 days 43.7 days

% Complete & Accurate

12.4% 95% 82.6%

# of Process Steps 32 20 12

Reduced Process Time by 60% or 20,000 minutes

Converted paper surveys to electronicReduced the # of copies

Potential cost reduction for assembly of mailing

Better connect Alumni with Stout

Efficiencies/Improvements


Efficiencies/Improvements (cont.)


Program specific results available in a more timely fashion

More time available to write the final report

Potential to eliminate the mailing if online response rate is high enough.

More involvement of program directorsDeveloped webpage with helpful information

Instructions for developing program specific survey

Examples of other program specific surveysTimeline of study

Give updates throughout processRequest any e-mail addresses they might haveAsk to follow up with non-respondentsTimeline better aligns with program directors’

needs

Efficiencies/Improvements (cont.)


All surveys, including program specifics, put on-line

Use “Alumni Finder” to locate accurate alumni e-mail addresses

Developed step-by-step process for paper survey mailing

Developed “gateway” on web so alumni are given correct survey

Examples of Improvements


Survey Gateway


Alumni received: Postcard with short

web address to access survey

Two additional postcards sent later to non-respondents

If still no response, actual survey packet sent through the mail

Study Design - 2010


Comments?

Questions?

Thank You!



Contact info: [email protected]

www.uwstout.edu/bpa/arc

University of Wisconsin-StoutApplied Research Center

Conducting Applied Research for Decision-Making

Services provided include:• Benchmark surveys• Customized surveys• On-site presentation of research results• Formulation and implementation of data-based action plan• Design & implementation of faculty development workshops• Data sharing in higher educational research (PolyDASHER)

office of budget, planning and analysis applied research center

Documents