assessment 101 - unt division of student affairs 101 assessment presented by: jan hillman, ed.d....

ASSESSMENT 101 - UNT DIVISION OF STUDENT AFFAIRS

101Assessment

Presented by:Jan Hillman, Ed.D.

University of North Texas

101Assessment


101Assessment

Our Session Today• What is Assessment Vs. Research?• Process of Assessment Activity: Problem to

Report• Tips on Literature Reviews and Types of

Literature To Draw Upon• Groupwork on developing an assessment and

on reporting results


101Assessment

How do you see it?Research Component / Characteristic % for

Assessment% for

Research

Study focuses on practical problems

Importance of institutional context to study

Contributes to changes in practice

Influence of institutional politics

Activity grounded in established theory

Contributes to formal knowledge base

Note: This is an arbitrary graphical representation of the cultural differences between assessment and formal research. ALL SCORES NEED TO ADD UP TO 100


101Assessment

Research Components Defining Assessment







Note: This is an arbitrary graphical representation of the cultural differences between assessment and formal research.


101Assessment

Research Components Defining Research









101Assessment





Activitiy grounded in established theory


Cultural Difference Between Assessment and Research

Components Defining True ResearchComponents Defining Assessment



101Assessment Which is which?Study on male identity development for

fraternity members at large public research institutions.RESEARCH

Study on UNT Greek Men and their perceptions of the campus climate for Greeks.

ASSESSMENT


101Assessment Which is which?Analyzing usage data by a population of 18-22

year old users of campus recreation center.ASSESSMENT

Analyzing usage data by a population of 18-22 year old users of campus recreation centers

throughout the State of Texas to identify connections between BMI and GPA.

RESEARCH


101Assessment

According to Schuh & Upcraft (2001)“Assessment guides good practice, while research guides theory development and tests concepts…Assessment typically has implications for a single institution while research typically has broader implications for student affairs and higher education” (p.5).


101Assessment

Assessment from Problem to Report

Source: Schuh, J. H., Upcraft, M. L. & Associates. (2001)


101Assessment

Step 1: Identify the problem• What specific circumstances or situations are

driving the need for assessment?• What external pressures are driving the need

for assessment?• What internal circumstances are driving the

need for assessment?


101Assessment

Step 2: Determine the purpose of the study

• What information do we need to help solve the problems identified in Step 1?

• Rely on this information to then develop the purpose of the study – then stick with it!

• Resist the temptation to earn a Nobel Peace Prize or find the cure to all of the woes at UNT – keep the study focused.


101Assessment

Step 3: Determine where to get the information needed

• Is the population students? What types of students? Be specific and focused.

• What does the literature say (more on that later)?

• Do we have previous institutional/department reports, findings or datasets?

• Do we have access to benchmarking studies of our peer institutions?

• Would faculty, staff or alumni be helpful?


101Assessment

Step 4: Determine the best assessment methods

• What is the best way to get the information I need?

• Quantitative, Qualitative or Both Oh My!• How to decide?– “What” questions are best answered by

Quantitative Research Methods– “Why” questions are best answered by

Qualitative Research Methods


101Assessment

Assessment Designs• Satisfaction Surveys - doesn’t tell you enough• Program Reviews• Needs Assessment • Cost Effectiveness Study• Benchmarking Study• Campus Culture/Environmental Assessment• Demographic Profiles (local, regional and

national)


101Assessment

Step 5: Determine whom to study

• Do we survey the entire student population?• Do we want to sample a segment of the

student population?– Gender, age, race and ethnicity– Class standing, GPA, Full-Time, Part-Time– Commuter, resident, distance learner– Service user, student leader, student employee


101Assessment

Step 6: Determine how data will be collected

• The method(s) selected MUST be consistent with the purpose of the study– Paper questionnaires (remember to scantron it!) – Telephone survey with trained interviewer– Web-based surveys– Individual interviews– Focus Group interviews


101Assessment

Step 7: Determine what instruments will be used

• The instrument must be able to yield results that can be statistically analyzed!

• Do we want to use a survey from a national source where validity and reliability is assured?


101Assessment

Reliability and Validity

Not Reliable, Not Valid Reliable, Not Valid

Not Reliable, Valid Reliable, Valid

Source: Adapted from presentation for EPSY 6020 by Dr. Axelson.


101Assessment ASSESSMENT TERM ALERT:RELIABILITY

Reliability refers to the extent to which assessments are consistent. Just as we enjoy having reliable cars (cars that start every time we need them), we strive to have reliable, consistent instruments to measure student achievement. Another way to think of reliability is to imagine a kitchen scale. If you weigh five pounds of potatoes in the morning, and the scale is reliable, the same scale should register five pounds for the potatoes an hour later (unless, of course, you peeled and cooked them). Likewise, instruments such as classroom tests and national standardized exams should be reliable – it should not make any difference whether a student takes the assessment in the morning or afternoon; one day or the next.

Source: Classroom Assessment Course, Florida Center for Instructional Technology and University of South Florida: http://fcit.usf.edu/assessment/basic/basicc.html


101Assessment ASSESSMENT TERM ALERT:VALIDITY

Validity refers to the accuracy of an assessment -- whether or not it measures what it is supposed to measure. Even if a test is reliable, it may not provide a valid measure. Let’s imagine a bathroom scale that consistently tells you that you weigh 130 pounds. The reliability (consistency) of this scale is very good, but it is not accurate (valid) because you actually weigh 145 pounds (perhaps you re-set the scale in a weak moment)! Since teachers, parents, and school districts make decisions about students based on assessments (such as grades, promotions, and graduation), the validity inferred from the assessments is essential -- even more crucial than the reliability. Also, if a test is valid, it is almost always reliable.

Source: Classroom Assessment Course, Florida Center for Instructional Technology and University of South Florida: http://fcit.usf.edu/assessment/basic/basicc.html


101Assessment

Step 7: Determine what instruments will be used

• The instrument must be able to yield results that can be statistically analyzed!

• Do we want to use a test from a national source where validity and reliability is assured?

• Do we want to develop our own instrument despite the fact validity and reliability will be a concern?

• Qualitative – design open ended questions which are standardized, as well as any follow up questions or items of clarification


101Assessment

Step 8: Determine who should collect the data

• Qualified and trained individuals are a good start for all assessment

• Need to be cautious about bias– Can we really trust a study that was done exclusively

by a person with a personal stake in the outcomes?– Can we really trust a study that was done exclusively

be outside experts with no context or awareness of the nuances of the institution?

– Bias can be reduced when outside experts in assessment review the process and methods used


101Assessment

Groupwork: Develop an Assessment• What is the feasibility of establishing an Alumni

Student Scholarship Program involving alumni and recruiting student leaders from high school?

• You think that there may be over duplication of mentoring/leadership programs across campus.

• You are unsure how student affairs specifically affects retention of your students.

• You and your staff think that the campus needs a recreation center. How would you provide evidence to the various decision-makers to make your case?


101Assessment

Step 9: Determine how the data will be analyzed

• Analysis of quantitative data depends on the purpose of the study: Are the respondents representative of the population? If yes, descriptive and differential statistics can then be applied in the quantitative arena

• Themes, trends and variations can be explored in the qualitative study arena and a plan developed to record and analyze this data


101Assessment

Step 10: Determine the implications of the study for policy and practice

• We suggest you rely on your findings and present how your findings impact the campus or program area.– What are the implications of this study?– What approaches to solving the problem should be

considered in light of this study?– What policies and procedures need to be created or

overhauled?– What is the call for action?


101Assessment

Step 11: Report the results effectively

• Understand we must package the findings carefully so that it motivates change

• How a study is distributed and formatted may be more important than the results found

• Segment your audience and tailor specific reports to specific audiences

• Make sure the results get into the right hands of people who can make a difference


101AssessmentChanges Made

What are some examples of changes made at your institution due to assessment findings?


101Assessment

Keys To A Good Literature Review• Where does it come from?• Sources– Peer reviewed journal articles

• Journal of College Student Development• Review of Higher Education• Research in Higher Education• Journal of Higher Education

– Reputable edited journal articles• “New Directions” series• Change magazine, Liberal Education, Peer Review, Phi Delta

Kappan, Assessment Update– Books from reputable authors and publishers


101Assessment

Keys To A Good Literature ReviewOther sources:• Conference presentations, proceedings• Published reports by universities, federal, state,

or local government, or independent research or policy institutions

• Dissertations or theses• ERIC ED documents• Reputable magazines or newspaper (for facts)• Reference books


101Assessment

Keys To A Good Literature Review

Don’t quote these…• Wikipedia• Individual web pages and blogs• Personal communications• Newspaper or magazines (for research

findings)• Secondary sources


101Assessment

Keys To A Good Literature ReviewWhere do I start my search?• Education Resources Information Center

(ERIC) http://www.eric.ed.gov/• PsycInfo(psychology related research)• LexisNexis (policy, law, business news)• Your campus library resources• Google Scholar http://scholar.google.com• Book or journal article reference list


101Assessment

Keys To A Good Literature Review

Alternative sources of information:• Ask around (professors and co-workers)• Online discussion forums• Amazon.com• Librarians • Internet search engines (as a starting point)


101AssessmentGroupwork: Reporting Scenarios

Who gets what, when and in what format?• Your latest campus climate survey revealed that your campus is not welcoming

to 1st generation students and your strategic plan specifically states you want to increase graduation rates for first generation students. What do you do with the results?

• Your campus participated in a national student profile (like the CIRP Freshman Survey). In what ways should the information be distributed and to whom?

• You completed a student experience survey on a random sample of your students and found some significant differences in service needs with this population than first time in college students. 79.8% of the transfers do not use any student affairs services.


101Assessment Contact Us:

Dr. Jan HillmanExecutive Director of Planning and Assessment

[email protected](940) 565-4909

Jason F. SimonDirector, Research, Assessment and Planning

[email protected](940) 369-8054


101AssessmentReferences

Chen, P.S. D. (2008). EDHE 6530 Research on Students in Higher Education Course. Slides 24-28 reproduced with permission of instructor.

Classroom Assessment. Collaboration between Florida Center for Information Technology and University of South Florida. Retrieved September 25. 2008: http://fcit.usf.edu/assessment/basic/basicc.html.

Schuh, J. H., Upcraft, M. L. & Associates. (2001). Assessment practice in studentaffairs: An applications manual. Jossey-Bass. San Francisco, CA.


101Assessment

Q&A

assessment 101 - unt division of student affairs 101 assessment presented by: jan hillman, ed.d....

Documents