assessment consistency: is everyone on the same page?

AssessmentConsistency:IsEveryoneontheSamePage?

DepartmentofPharmaceuticalSciencesWegmansSchoolofPharmacy

St.JohnFisherCollege,Rochester,NY

MindyLull,Ph.D.AssociateProfessor

JenniferMathews,Ph.D.AssociateProfessor

Participantswilllearn:

1. Howtousefacultydevelopmentworkshopstoprovidethefoundationofwritingqualityexams.

2. Howtoeffectivelyusethepeer-reviewprocess.3. Howtocreateconsensusandguidelinesfor

creatingandlaunchingexams.

Whyisconsistencyimportant?

• Consistencycontributestoexamquality• Examsshouldreflectcontentlearning• Reducesstudentconfusionorfrustration• Examsarehigh-stakes• Dataareusedformanypurposes

FacultyDevelopmentforConsistency

• Topreparefacultyforwritinghighqualityexams,weconductedaseriesofworkshops:– learningoutcomes– examblueprints– codingtoBloom’sTaxonomy– peer-reviewprocess

• Eachoftheseworkshopshelpedtoformthefoundationofconstructingexamswhichtrulyreflectedstudentlearning.

Developclearlywrittenstatementsofkeylearning

outcomes.

Useresultsofassessmentsto

improveteachingandlearning.

Designexperiencesthatprovide

opportunitiestoachieveoutcomes.

Assessstudentachievementof

outcomes

Cycleas described byMiddlesStatesCommission on HigherEducation

CycleofAssessment

Clearlywrittenoutcomesandexammaps

• WemadethedecisiontouseExamSoft®asourtoolforcourseassessments.

• Beforeanyexamswerecreatedusingthetechnologywehadseveralfacultydevelopmentworkshops.

• DevelopmentsessionsledbyNABP– Revisionofcourselearningoutcomesbyfacultyinthe

course– Techniquesforwritinggoodexamquestions– UseofexamMaps– PropercodingtoBloomsTaxonomy

• Allfacultyarewritingandcodingtheirownexamquestions.

Exammap:ismyexamreflectiveofwhatItaught?

Course learningoutcome:Basedonthesiteandmechanismofaction,predictthetherapeuticandsideeffectsassociatedwithpharmacologicalagentsusedtotreatvariousdisease states. CourseActivity LectureLO Knowledge Application Synthesis

Lecture2(9.11.12– 9.13.12)APPElectures

Lecture,TP,patientscenario

EPOREMS· pg.;Bank question 1420(MC)

AnemiaOutcome4 1

Iron+VitaminC· pg.9;Bank question 1525(FITB)

AnemiaOutcome4 1

AnemiafromRBCdestruction• pg.; Bankquestion 1527(SA)

AnemiaOutcome1,2,3

1

Continuedupdatesandquicktips

• Usetimeduringfacultyretreatsandmeetingstopracticecodingsamplequestions,toaddresscodinginconsistencies,holesinourcurricularcoverage,questions/concerns

• Emailreminderswith“howto”– Ex.HowtowriteagoodFITBquestion

• Bindersforallfacultywithsamplematerials– AppendixBtopics,WSOPprogramoutcomes,Bloom’sTaxonomyverbs

Resources:Sample“Howto”

Thepeerreviewprocess

• “Peer-reviewedarticlesprovideatrusted formofscientificcommunication.”

• “Evenifyouareunfamiliarwiththetopicorthescientistswhoauthoredaparticularstudy,youcantrustpeer-reviewedworktomeetcertainstandardsofscientificquality.”

Diagramandquotesfrom:http://undsci.berkeley.edu/article/howscienceworks_16

Peerreview

• Asacademicsweplacevalueinthepeerreviewprocess.

• Weshouldalsoplacevalueinwritinghighquality,peer-reviewedassessments.

OurProcess

§ Thisisavoluntaryprocessforfacultywhowanttoparticipate

§ 2-3reviewers/exam§ Facultyareinvestedintheprocess§ ReviewersDONOTmakechangestoexams,anditemwriterscanchoosenottomakesuggestedchanges

§ Studentdataismuchmorereflectiveofwhatwaslearnedversuscouldmytestitembedeciphered

Qualityofquestions:Peerreviewedtestitem

YES NO Comments/Suggestions

1 Isthequestion cleartothereader?2. Arethere typographicalerrors?3. Isthequestion codedtothe

appropriatelevelofBloom’sTaxonomy?

4. Isthequestion codedappropriatelytothe learningoutcome(s)?

5. If thequestionhasahistory,diditperformwellinthepast?

6. Is thecontent levelappropriateforthecourse?

7. Otherconsiderations:“positive”language,jargonfree,limitedabbreviations,double-jeopardy,examtime

PRTI:PeerReviewedTestItems

Howtolearnfromothers

• Problemareas:1. FITBquestions2. Shortanswer/essay(characterlimits)3. EliminatingK-type4. Selectallthatapply

Wenowhaveavalidatedtestbank

• Wenowhaveavalidatedtestbank.• Over3000questionshavebeenpeer-reviewed!

CreatingFormattingGuidelinesforConsistency

• Ensuresaconsistentlooktoallexamsandexamreports

• Considerhavinga“QuickTips”documentthatoutlinestheseforallfaculty

• Guidelines– Questionformat– Examcreationandpostings– Examadministration– Examreports

ConsistencyinQuestionFormat

• Canthestudentgetaquickideaofthequestion?

ConsistencyinQuestionFormat

• Arecommonlyusedelementsformattedthesameway?

ConsistencyinExamCreation

• Preventlogisticalerrorsinexamwithachecklist:– Correct#ofpoints– Checkquestiongroupings– Overridefontandfontsize– Checksettingsonpartialcredit,select-all-that-applyandk-typequestion

– Exportcopyofexamforlast-minutespellingerrors– Previewexamsbeforeposting

ConsistencyinExamPosting

ConsistencyinExamAdministration• Examdetails• Assessmentdetails• Policies• Contactinformation

ConsistencyinExamReports

Conclusions

• Goodexamconsistencyinvolvesquestionwriting,examcreation,examposting,examadministration,andexamreports.

• Trainingisrequiredtogetallfacultyonthesamepage,andtoemphasizetheimportanceofconsistency.

• A“Quick-Tips”guideisusefulforlogisticalrecommendationsforconsistency.

Thankyou!

Contactus:MelindaLull:[email protected]

JenniferMathews:[email protected]