using examsoft data to prepare for and ease the accreditation process

1

Use of ExamSoft Data to Prepare For and Ease the

Accreditation ProcessKimberly K. Daugherty, PharmD, BCPS

Ashley Castleberry, PharmD, MAEdLeah Simpson, EdD, MPA

Polling Question: Who is in the audience?• Please select the category that best fits you:

1. Assistant/Associate Dean2. Director of Assessment3. Chair of Curriculum and/or Assessment Committee4. Faculty member5. Staff Member

Session Learning Outcomes• Discuss how the autonomy supportive feedback system can be used

by students and faculty to enhance student learning and metacognitive development

• Demonstrate how longitudinal assessment data run from ExamSoft can be used to provide students with assessment report cards showing their progress across a curriculum

• Model how ExamSoft longitudinal data can be aggregated for use in accreditation reports

Autonomy and ExamSoft• Self-Determination Theory (SDT) states that all humans have three

basic psychological needs: autonomy, competence, and relatedness (Deci & Ryan, 1985, 1994, 2000)

• Autonomy in learning has been linked to greater student satisfaction, persistence, and deeper learning (Seiver & Troja, 2014; Wielenga-Meijer, Taris, Wigboldus & Kompier, 2012)

Autonomy-Support• By meeting students’ intrinsic desires, autonomy-support has been

shown to increase student satisfaction, persistence, creativity, and curiosity (Patrick & Williams, 2009; Seiver & Troja, 2014)

• Perception of autonomy-support leads to greater school achievement (Mih & Mih, 2013)

• Feedback can be autonomy-supportive when it Feeds Up, Feeds Back, and Feeds Forward (Hattie & Timperley, 2007)

University of KentuckyStudent End of Exam Report Cards

Feed Up, Feed Back, and Feed Forward with ExamSoft Reports

Students LOVE these reports!

Does your institution use end of exam reports?

1. Yes2. No

University of Arkansas for Medical Sciences

Longitudinal Reports-Students-Course

-Curriculum

Reports > Longitudinal Analysis

Student and Course Reports

Student and Course Reports

64.17%

Student and Course Reports

How Longitudinal Reports Fit Autonomy Support Model• Feed Up

• Provide students with data needed to formulate goals in specified areas

• Feed Back• Provides students with data on achievement of their goals

• Feed Forward• Students meet with faculty to discuss data and make plans for improvement

How Longitudinal Reports Fit Autonomy Support Model• Programmatic Assessment

• Data from courses provide aggregate data on course effectiveness• Data from the curriculum provide aggregate data on curriculum effectiveness

Curriculum ReportsBloom’s Taxonomy

Curriculum ReportsDidactic Topics

Does your institution use longitudinal reporting at the course level?

1. Yes2. No

Sullivan UniversityComprehensive Student Report Cards

SUCOP Comprehensive Student Report Card• Background on Sullivan University College of Pharmacy

• 3-year year around program• Quarter system

• Comprehensive Student Report Cards• Reason for implementation: to provide students and their advisors

information on overall GPA, progression on pharmacy knowledge and skills, and progression on program outcome attainment

• Data on report cards is aggregated from multiple sources• E-value-experiential software• ExamSoft-didactic testing software• Campus Vue-Student Information System• CHARM: OSCE data

Student Report Card • Required didactic elements: data is rolled

up from exam question results tagged in ExamSoft

• Program outcome elements: data rolled up from exam question results tagged in ExamSoft

• Report cards are compiled after the Fall and Spring quarters are completed for the first and second professional year classes

Required Didactic Elements

Assessment BS PS SABS CS

ExamSoft Student Average ** 63.92 77.73 77.55 76.41

Number of Questions 888 1748 674 1291

Class Average Class SD 75.29 12.07 86.27 11.43 85.69 17.34 84.98 13.86

PCOA Aug 2015 Student Averag 248 277 263 246

SUCOP Avg-National Avg 328-350 324-334 327-339 318-326

Program Outcomes

Community IPPE*** 3 3.00 3.00 3 NA 3.00 3.00

Hospital IPPE*** 3 3 3 NA 3 3 NA

ExamSoft Student Average** 75.87 78.52 71.4 82.67 76.37 81.63 67.65

# of Questions 3063 933 222 70 125 686 34 Class Average 85.46 83.66 85.55 84.39 82.92 89.06 84.88 Class SD 6.41 9.06 9.41 12.69 8.61 11.6 14.96

Assessment Areas 1.1 2.1 2.2 2.3 2.4 3.1 3.2

How Report Cards Fit Autonomy Support Model• Feed Up

• Provide students with data needed to be able to formulate goals in each of the specified areas

• Feed Back• Provides students with data on achievement of their goals

• Feed Forward• Students meet with their advisors to discuss the data and make plans for

improvement

• Programmatic assessment• Data from the individual report cards is “rolled-up” to provide aggregate data on

element attainment

How data can be combined with other data to tell accreditation story PY1 Didactic PY1 and PY2 Didactic

EO # Questions Mean SD # Questions Mean SD 1.1 1820 84.5% 0.1 4896 85.9% 0.1 2.1 300 78.5% 0.1 809 86.7% 0.1 2.2 270 86.5% 0.1 270 86.6% 0.1 2.3 46 77.9% 0.2 98 85.8% 0.1 2.4 14 78.2% 0.1 212 85.7% 0.1 3.1 293 90.2% 0.1 871 84.3% 0.1 3.2 34 93.0% 0.1 62 92.7% 0.1 3.3 90 85.1% 0.2 97 81.4% 0.1 3.4 8 93.7% 0.1 9 89.6% 0.1 3.5 3 77.0% 0.2 7 90.2% 0.1 3.6 149 87.0% 0.1 176 90.4% 0.1 4.1 25 87.2% 0.1 25 92.7% 0.1 4.2 5 76.1% 0.2 6 87.9% 0.1 4.3 NA NA NA NA NA NA 4.4 79 85.9% 0.1 577 87.8% 0.1

JCPP-PPCP % of Exam Mean SD % of Exam Mean SD

C NA NA NA 2.6% 36.0% 0.2 H NA NA NA 55.7% 80.1% 0.1 A NA NA NA 17% 93.8% 0.1 R NA NA NA 19% 80.3% 0.2 M NA NA NA 5% 61.5% 0.2

Blocks 1 and 2* Blocks 3 and 4** EO Mean SD Mean SD 1.1 3.20 0.58 3.63 1.6 2.1 3.07 0.73 4.04 1.19 2.2 2.91 1.32 3.84 1.54 2.3 NA NA 3.21 2.04 2.4 2.64 1.5 3.41 1.82 3.1 3.35 0.51 4.18 0.86 3.2 2.98 0.8 3.64 1.6 3.3 NA NA 2.84 2.11 3.4 3.72 0.47 4.45 0.73 3.5 NA NA 3.69 1.62 3.6 2.97 0.73 3.98 1.29 4.1 3.61 0.61 4.37 0.73 4.2 3.29 0.58 4.01 1.32 4.3 NA NA 3.12 1.93 4.4 3.40 0.52 4.53 0.66

JCPP-PCPP Mean SD Mean SD

C NA NA 4.12 1.21 H NA NA 4.11 1.17 A NA NA 4.02 1.14 R NA NA 4.02 1.17 M NA NA 3.92 1.25

Blocks 1 and 2: 4-point Likert scale (0=NA-4=Exceed); Blocks 3 and 4: 5-point Likert scale (0=NA-1=Beyond entry level)

Does your institution use longitudinal reporting at the program level?

1. Yes2. No

QUESTIONS