10/22/2019

1

Maybe We Should Ask Them: When It’s Time for 

Meta‐AssessmentIUPUI Assessment Institute

October 14, 2019

Cynthia Groover, EdDAssessment and Accreditation Specialist

cgroover@georgiasouthern.edu

How We Do What We Do

• Business and Finance• Information Technology• Student Affairs and Enrollment Management• Academic Affairs• Office of the President

–Advancement –External/Government Affairs

10/22/2019

2

Who Are We Targeting?

Resources and Processes in Question

• OIE Website 

• OIE Rubric

• Electronic Feedback

• Face to Face Feedback

• Consultation with IE Review Team Off Cycle

• Consultation with OIE Staff Member Off Cycle

• Divisional Example

10/22/2019

3

Programmatic Assessment Focus

What are the strengths and weaknesses of the processes and resources in place in support of administrative and student affairs assessment? 

10/22/2019

4

Programmatic Assessment

Shutt, Garrett, Lynch, and Dean (2012) 

Any program “should continue to undergo evaluation where it can be modified to ensure that every element contributes to the program’s outcomes” (p. 78).

Fink (2013) 

Institutions need to look at the impact of multiple resources and processes in place to determine if they have the “right set of activities” in place to positively impact assessment practices across campus (p. 47). 

Getting Uncomfortable

Meyer and Murrell (2014)

“ask the tough questions and to get the news that something is not working (or working as assumed) and should therefore be revised or eliminated” (p. 4).

10/22/2019

5

Guiding Questions

1. What are the perceived strengths and weaknesses of the processes and resources in place to develop knowledge of and confidence in assessment and how does perceived utility differ among divisions of the institution?

2. How do assessment coordinators perceive their own knowledge of and confidence in assessment?

3. What is the relationship between knowledge of and confidence in assessment and the utility of resources in place? 

4. What is the relationship between knowledge of and confidence in assessment and number of assessment cycles in which participants have engaged?

Data Collection

Why E‐mail ‐ Social Desirability Bias

Saturation sampling

Population           

N = 85

n = 61

71.7%

10/22/2019

6

Research Question One

1. What are the perceived strengths and weaknesses of the resources in place to develop knowledge of and confidence in assessment and how does perceived utility differ among divisions of the institution?

This resource was very helpful.

This resource was quite helpful.

This resource was a little helpful.

This resource was not at all helpful.

I knew about this resource but did not

use it.

I did not know about this resource.

Utility of Resources

OIE Website

External Resources

Face to Face

FeedbackElectronic Feedback

Review Team

Off Cycle

OIE Off Cycle

OIE Rubric

Divisional Example

Mean 3.21 3.00 5.11 4.92 4.05 4.21 3.54 3.70

Median 4.00 3.00 5.00 5.00 4.00 5.00 4.00 4.00

Mode 1.00 1.00 6.00 5.00 6.00 6.00 4.00 5.00

Std. Deviation 1.77 1.81 0.92 1.01 1.72 1.77 1.76 1.80

Range 4.00 5.00 5.00 4.00 5.00 5.00 5.00 5.00

Note. n = 61

10/22/2019

7

Research Question Two

2. How do assessment coordinators perceive their own knowledge of and confidence in assessment? 

a) I have a solid understanding of what constitutes good assessment   practice.

b) I am confident I can successfully conduct assessment activities in my unit.

c) I am confident I can successfully report assessment activities in my unit.

Very true Somewhat true Neither true nor untrue Somewhat untrue Very untrue

10/22/2019

8

I have a solid understanding of what

constitutes good assessment practice.

I am confident I can successfully conduct

assessment activities in my unit.

I am confident I can successfully report

assessment activities in my unit.

Mean 4.05 4.08 4.05

Median 4.00 4.00 4.00

Mode 4.00 4.00 4.00

Range 3.00 4.00 3.00

Note. n = 61.

Knowledge and Confidence

“Somewhat True”

10/22/2019

9

By Division

10/22/2019

10

Research Question Three

3. What is the relationship between knowledge of and confidence in assessment and the utility of resources in place? 

Knowledge Confidence Composite

Understanding        +         Conducting            +           ReportingConductingUnderstanding Reporting+ +

10/22/2019

11

Knowledge Confidence Composite & Resource Utility

OIE Website

External Resources

Face to Face

FeedbackElectronic Feedback

Review Team

Off Cycle

OIE Off Cycle

OIE Rubric

Divisional Example

KCC 0.38* 0.22 0.25 0.32* 0.04 0.18 0.24 0.17

Note. n = 61. *Denotes significant at the p < 0.05 level

Knowledge Confidence Composite and Resource Utility Take 2

OIE Website

External Resources

Face to Face

FeedbackElectronic Feedback

Review Team Off

Cycle

OIE Off

Cycle OIE RubricDivisional Example

KCC Pearson Correlation

0.33 0.29 0.35** 0.34** 0.54** 0.55** 0.42** 0.14

N 35 33 60 60 43 44 44 45

10/22/2019

12

Research Question Four

4. What is the relationship between knowledge of and confidence in assessment and the number of assessment cycles in which participants have engaged?

Knowledge Confidence Composite and Experience

Assessment Cycles

KCC 0.11Note. n = 60. *Denotes significant at the p < .05 level.

10/22/2019

13

Using What We Know Now

• We have some disconnects to address

– Perception versus all the data

• We have some relationships to build

– Functioning within a very different environment 

– Posting resources may help some

– Being the resource will help more

• We need to keep asking those questions

– And be ready to respond

References

Fink, L. D. (2013). Innovative ways of assessing faculty development. New Directions for Teaching & Learning, 2013(133), 47‐59. doi: 10.1002/tl.20045

Meyer, K. A., & Murrell, V. S. (2014). A national survey of faculty development evaluation outcome measures and procedures. Online Learning, 18(3). 

Rodgers, M., Grays, M., Fulcher, K., & Jurich, D. (2013). Improving academic program assessment: A mixed methods study. Innovative Higher Education, 38(5), 383‐395. doi: 10.1007/s10755‐012‐9245‐9

Shutt, M. D., Garrett, J. M., Lynch, J. W., & Dean, L. A. (2012). An assessment model as best practice in student affairs. Journal of Student Affairs Research and Practice, 49(1), 65‐82.