Youprobablywouldn’tbehereifyoudidn’tbelievethatComputerizedAdap9veTes9ngshouldtakeaprominentroleinthefutureofmeasurement.Withthebenefitsofimprovedefficiency,less‐discouraging(orboring)testsforexaminees,more‐completeitembankusage,andsoon,onemightwonderwhyCAThasn’tcaughtonfaster.Thereasonsbehindtheseeminglyslowadop9onofadap9vetes9ngaremany,buttheonehindranceIwouldliketofocusontodayistherela9velackofreadilyavailableCATsoIware.MostcurrentCATsystemsareeitherhome‐grownorproprietary,meaningthatnewcomerstothefieldfindthemselvesintheposi9onofeitherneedingtodeveloptheirownsoIwarefromscratch(assumingthey’restrongcomputerprogrammers)orpaylicensingfees,whichmaybeprohibi9veforsmallerscaleinnova9on.Afree,off‐the‐shelfsoIwarelibraryforCATcouldallowmoredeveloperstobeginworkingwithadap9vetes9ngwithouttheveryredundantburdenofwri9ng,tes9ng,anddebuggingcomputercodeforcommonCATfunc9ons.SuchalibrarywouldstreamlineentryintoCATresearchanddevelopment,andwouldfreeresearcherstofocusontheirsubstan9veques9onsofinterest,insteadofwas9ng9metryingtodiagnosewhytheirFisherInforma9onisexplodinginsteadofmaximizing.Today,Iwouldliketopresenttoyouarela9velynew,freeandopensourcesoIwarelibrarythathopestoreducetheburdenofgeOngstartedinCATdevelopment.It’scalled,simplyenough,theOpenSourceComputerizedAdap9veTes9ngSystem,orOSCATSforshort.I’lldescribehowthesystemisputtogether;thenwe’lltakealookathowtouseOSCATStoconstructaCATsimula9ontoansweraprac9calresearchques9on.

1

Beforedelvingintothedetails,I’dliketomen9onafewofthedesiredcharacteris9csofaCATprogramminglibrarythatdriveOSCATSdevelopment,allofwhichareintendedtofacilitaterapidCATdevelopmentforresearchoropera9onalprototyping.First,OSCATSaimstoprovidecommonCATrou9nes—popularitemresponsemodels,itemselec9onalgorithms,andsta9s9calanalysistoolsoIenusedtostudytheproper9esofanadap9vetest.Someofthesehavealreadybeenimplementedforthemostcommonchoices,andmoreareunderdevelopment.Second,OSCATSisdesignedtobemodular.Itshouldbeeasytopluginandpulloutormix‐and‐matchdifferentfeaturesoftheCAT,suchasmodelsandalgorithms.Tothisend,OSCATSiswriUenundertheobject‐orientedprogrammingparadigm.Featuressuchasmodelsandalgorithmsarerepresentedasobjects.TheseobjectsdefinehowtheCATfeaturesbehaveinstandardizedwayssothatoneIRTmodel,say,canbeeasilyreplacedwithanotherinoneloca9onwithouthavingtomodifyallthecodethatmakesuseofthemodel.Third,OSCATSisdesignedtobeextensible.AlthoughOSCATSalreadyprovidessomeofthemostcommonCATcomponentsandaimstoeventuallyhaveaverybroadselec9onofmodelsandalgorithms,CATdevelopersshouldhavethefreedomtocomeupwiththeirowntwistonamodeloralgorithmandhaveitworkeasilyandrela9velyseamlesslywiththerestofOSCATS.Finally,OSCATSaimstobefamiliarinthesenseofallowingdeveloperstoworkinaprogramminglanguagethattheyalreadyknow.Programmersdon’tallhavethesameexperience,anditwouldbesomewhatlimi9ngtoforceCATdeveloperstoallusethesamelanguagethatOSCATSwaswriUenin.So,OSCATSprovidesbindingsinotherlanguages;thatis,OSCATScomeswithaminimalamountgluethatallowsuserstomanipulateOSCATSfromseveraldifferentprogramminglanguages,allowingprogrammerstoworkintheirmostfamiliarlanguage.

2

So,whichlanguagedoesOSCATSuse?C.Now,givensomeofthegoalsIjustmen9oned,Cmaynotseemlikethebestchoice.Cplacesmuchmoreoftheburdenofmemorymanagementontheprogrammer,whichcanmakedebuggingmorechallenging,anditssta9ctypingsystemandcompila9onrequirementsdon’tusuallymakepeoplethink“rapid”and“flexible.”But,Cwasareasonablechoiceforseveralreasons:it’sfast,light‐weight,portable,andnearlyubiquitous.Butmoreimportantly,itis(rela9vely)straigh`orwardtointerfacemostprogramminglanguagesouttherewithlibrarieswriUeninC.It’snotatallstraigh`orwardtowriteaprograminPythonandmanipulateitfromPHP,butyoucanfairlywellthroughC.Now,youmaybethinkingthatCisn’tanobject‐orientedprogramminglanguage,likeC++orJava.But,object‐orientedprogramingisreallyaphilosophyoraparadigm,notanintrinsicfeatureofaprogramminglanguage.Itjustsohappensthatsomelanguages(likeC++andJava)havesyntac9calfeaturesthatmakeusingtheobject‐orientedparadigmmoreconvenient.SincetheClanguagedoesn’tprovidemuchbuilt‐inhelpforobjectorienta9on,OSCATSmakesuseoftheGObjectframework.Perhapsyou’veheardofGTK+(theGUItoolkit)orGIMP(imagemanipula9onprogram)—bothofthesearewriUeninobject‐orientedCusingGObject.Now,tobehonest,programmingwithGObjectcanbealiUlecumbersome.However,GObjectisinvaluableforOSCATS’sgoaloffamiliaritybecauseitwasexplicitlydesignedwithlanguagebindingsinmind.Thatis,GObjecthascertainbuilt‐infeaturesthatmakeiteasytowritethegluethatallowsyoutouseaClibraryfromotherlanguages.Moreover,GObjectbindingsalreadyexistforoveradozenlanguages,makingiteveneasiertogeneratebindingsfortheOSCATSlibrary.Infact,OSCATSbindingsalreadyexistforPython,Perl,PHP,andJava,andit’spossibletoaccessOSCATSinMATLABthroughtheJavabindings.So,eventhoughCandGObjectthemselvescanbesomewhatcumbersome,OSCATSusersdon’tactuallyhavetodealwiththemdirectly,butcanuseamoreconvenient,higher‐levellanguageforeaseofdevelopment.Finally,Iwouldmen9onthatOSCATSiscurrentlygearedtowardCATsimula9on.Thatis,itdoesn’tincludeauserinterfaceforadministeringitemstoactualexaminees.OSCATScertainlycouldbeusedasthepsychometricenginebehindsuchasysteminthefuture,butthisjusthasn’tbeenthefocusoftheprojectsofar.(Ibelievewe’llbehearingaboutanotherprojectlaterinthesessionthatprovidesatestadministra9oninterfaceforliveexaminees.)Instead,theflexibilityofOSCATSisdesignedtofacilitateresearchanddevelopmentofCATmethodsandtherapidprototypingofpoten9alopera9onaladap9vetests—forexample,studyingtheproper9esofapar9cularitembankordeterminingtheeffectofdifferentcontentbalancingtechniquesandthelike.

3

TobegintogetintothedetailsofhowOSCATSworks,let’sfirstthinkabouthowadap9vetestsruningeneral.Iknowyou’reallquitefamiliarwiththis,soIwon’tspendmuch9mehere.AgenericCATcyclestartswithsomeini9aliza9on.Then,theCATchoosesanitem—thismightinvolvefilteringoutineligibleitems(sayinmul9stagetes9ngorforcertaincontentbalancingalgorithms),selec9ngaproposalitem(withMaximumFisherInforma9onorwhathaveyou),andpoten9allyapprovingthechoice(e.g.inaSHorcontentbalancingalgorithm).Thentheitemisadministered(orsimulated),andtheresponseisanalyzed(interaliaobtaininganewabilityes9mate).Thenyoucheckatermina9oncriterionandloop,orifthetestiscomplete,youreporttheresultsandanyassociatedsta9s9cs.Alladap9vetestsfollowthissamebasicoutline;however,thespecificdetailsofeachstepwilldependonthepar9cularalgorithmschosen‐‐‐somemightuseMFI,othersKLI,somehavecontentbalancing,othersincludenon‐sta9s9calconstraints.Usually,aCATprogrammerwouldhavetocustomizethecodethatimplementstheadap9vetes9ngcyclebyhandtoincludeorexcludealgorithmsaccordingtothepar9culardesignofthegiventest,whichcouldpoten9allygetabitmessyifaresearcheristryingtoaddandsubtractanumberofdifferentalgorithmstodeterminetheireffectonCATperformance.InOSCATS,ontheotherhand,thereisasinglepieceofcodethatimplementstheadap9vetes9ngcyclefor*all*OSCATStests.

4

Thisisachievedusingsomethingcalledsignals.Asignalisinessenceawayforoneobjecttobroadcasttootherobjectsthatsomethinghashappened.SignalsareoIenusedinevent‐basedprogramming,par9cularlyforsystemssuchasgraphicaluserinterfacesthathavetoexecutecodeinanunknown,arbitraryorder(namely,intheorderthattheuserclickbuUons).Oneobjectwilladver9sethatitprovidesacertainsignal,andotherobjecthavetheopportunityto“connect”tothesignalforthatobject.Then,thefirstobjectswillsendamessage(thesignal)toallobjectsthathave“signedup”toreceivethesignalwheneverthegiveneventoccurs.SinceOSCATSisobjectoriented,thereisaTestobjectthatincludesareferencetotheitembankandcoordinatestheadministra9onofthetest.Therearealso“algorithm”objects.Itmaybeabitstrangetothinkofanalgorithmasanobject,sincealgorithmsaren’ttangiblethings.Butthesealgorithmobjectssimplyencapsulatethecode(andanyopera9ngparameters)necessarytoperformonepieceoftheCAT(suchastheitemselec9onviaMFI,orsimula9ngaresponse,ores9ma9ngabilityviatheMLE).SeOngupanOSCATStestinvolvescrea9ngtheTestobjectandanAlgorithmobjectforeachcomponenttobeincludedinthetest.ThentheAlgorithmsconnecttovarioussignalsofferedbytheTestthatcorrespondtoeachstepoftheadap9vetes9ngcycle.

5

ThisishowOSCATSachievesmodularity.TheTestobjectdoesn’tactuallyneedtoknowwhatalgorithmsareavailabletoit,orevenwhichalgorithmshaveconnectedtoitssignals—allofthatishandledautoma9callybytheGObjecttypesystem.TheTestsimplyiteratesthrougheachofthestagesoftheadap9vetes9ngcycle,emiOngtheappropriatesignalsinturn.ThecodethatcontrolsOSCATStestadministra9onreallyisn’tmuchmorecomplicatedthanwhat’slistedhere(plusaliUlebookkeeping).Changingthepar9cularsetofalgorithmsfromonetesttothenextissimplyamaUerofconnec9ngadifferentsetofAlgorithmobjectstotheTest.Moreover,wri9ngnewalgorithmsissimplified,sincetheAlgorithmobjectonlyhastofocusonitsonetaskand“plugin”totheTestobjectattheappropriatepoints.

6

Let’sseehowthiswouldworkforanexampleCAT.It’saknownphenomenon(e.g.RulisonandLokenAPM‐2009‐33‐83)thatunderMFIitemselec9on,examineeswhohappentoanswerthefirstfewofitemsincorrectlyhavedifficultyrecovering,eveniftheyhaveahighabilitylevel.Supposeyou’reinterestedinexaminingbiasines9matedabilityundervariousitemselec9oncondi9ons,butonlyforthoseitemswhoanswerthefirstfewitemseitherallcorrectorallincorrect.You’dchoosesomethinglikethissetofOSCATSalgorithms.OSCATScomeswithanalgorithmtosimulateexamineeresponses,butherewewanttodosomethingspecial(forthefirsttworesponses,wewanttospecifythattheexamineegetsthembothcorrectorincorrect),sowe’llhavetowriteourownadministra9onalgorithm,CustomSimulateAlg.But,thisisn’ttoodifficultsinceOSCATSwasdesignedforextensibility.Moreover,eventhoughOSCATSiswriUeninC,we’llworkthroughthisexampleinPython(includingwri9ngthenewitemadministra9on/simula9onalgorithm).

7

Aswe’llseeinamoment,OSCATStestconstruc9oninvolvescrea9nganinstanceofeachoftheseobjectsand“registering”themforaTestobject.Onregistra9on,theindividualAlgorithmstakecareofconnec9ngthemselvestotheappropriatesignalsoftheTestobject.OscatsAlgMaxFisherconnectstotheini9alizesignal(toini9alizesomeworkingmemory)andtheselectsignal,etc.AstheTestproceedsthroughtheadap9vetes9ngcycle,itemitseachsignalinturn,whichcallstheappropriatecodeforwhicheveralgorithmisconnectedtothesignal.EachAlgorithmobjectonlyhastobeconcernedwithperformingitsintendedfunc9onwhenitssignalisemiUed.Inthisexample,weonlyhaveonealgorithmconnectedtoeachsignal;but,inprinciple,anynumberofalgorithmscouldbeconnectedtoasignal(thoughforsomesignals,suchas“administer,”itreallyonlymakessensetohaveone).

8

FirststepistocreatetheTestobject.Provideahelpfuliden9fier,indicatetheitembank(thiscodesnippetassumesavariablemybankhasalreadybeencreated),andgivethetestahintofabouthowlongthetestwillbe(justpreallocatessomearrays).Then,createeachoftheAlgorithmobjectsinturnandhavethemregisterwithourTestobject.Here,haveMaxFisherselectrandomlybetweenthetop5items.Ourcustomsimula9onalgorithmistoldtoforcethefirsttworesponsescorrect.UseEAP.Specifythelengthas20.Inthelastcase,westoretheCustomBiasAlginthevariablefi_resultinordertoaccesstheresultsaIerthesimula9on.

9

Now,ifwewanttochangetheconfigura9onfromMFItoKLI,duetothemodularityofOSCATS,it’ssimplyamaUerofreplacingoneline.Similarly,no9cethatwedidn’thavetotellthealgorithms(suchasforsimula9onandes9ma9on)whatkindofIRTmodelswe’reusing.Thereareobjectsforthemodelofeachitem(createdbytheprogrammerelsewhereandstoredinthemybankvariable,here),andeverykindofModelobjectfollowsastandardinterface,soalgorithmscanbemoreorlessignorantofthedetailsofanypar9cularmodel.Theysimplyaskamodel“giventhislatentability,what’stheprobabilityofansweringcorrectly”(orthelike)andleavethedetailsoftheimplementa9ontothemodelobject.Ifwewantedtoswitchfroma2PLtoa3PL,wedon’thavetotouchanyofthealgorithmcode—wesimplymodifythesec9onthatsetsuptheitems(simulatesrandomparametersorreadsparametersfromafile—notshownhere).

10

Wes9llhavetoimplementourcustomsimula9onalgorithm.We’lllookatthefullpythonimplementaitonofCustomSimulateAlgoverthenext3slides.

11

Thisistheheartofthealgorithm—thepieceofcodethatwillbecalledeach9meTestemitstheadministersignal.

12

13

Forthesakeof9me,won’tshowthecodeforcrea9ngtheItemsandExaminees.Oncethey’reallini9alized,torunthetest,yousimplycalltest.administer()oneachexaminee.

14

15

Note:SomeofthefeaturesdemonstratedtodayareinthedevelopmentversionofOSCATS(coderepository),soifyoutrytodownloadthereleasedversion,itwon’tquiteworklikeshown.But,anewreleaseisplannedtohappeninthenextfewweeks.Thecompletecodeforthisexamplewillbeincludedinthatrelease.

16