Towards Understanding the Impact of Real-Time

AI-Powered Educational Dashboards (RAED) on Providing Guidance to Instructors

Ajay KulkarniGeorge Mason University

Fairfax, VAakulkar8@gmu.edu

Michael EagleGeorge Mason University

Fairfax, VAmeagle@gmu.edu

ABSTRACT TheobjectivesofthisongoingresearcharetobuildRealTimeAI-Powered Educational Dashboard (RAED) as a decisionsupporttoolforinstructors,andtomeasureitsimpactonthemwhilemaking decisions. Current developments in AI can becombinedwiththeeducationaldashboardstomakethemAI-Powered. Thus, AI can help in providing recommendationsbasedonthestudents’performances.AIPowerededucationaldashboards can also assist instructors in tracking real-timestudent activities. In this ongoing research, our aim is todevelop the AI component as well as improve the existingdesign component of the RAED. Further, we will conductexperimentstostudyitsimpactoninstructors,andunderstandhow much they trust RAED to guide them while makingdecisions.Thispaperelaboratesontheongoingresearchandfuturedirection.

Keywords Decisionsupporttool,Educationaldashboard,Interactivevisualizations,Impact,Unsupervisedlearning,Recommendations

1. INTRODUCTION A dashboard is a collection of wisely selected visualizationsthat assists in understanding raw information stored indatabases,whichhelpshumancognition[6].Adashboardcanbeviewedasacontainerof indicators[13],butBronusetal.provided the most accurate definition of the dashboard.Bronusetal.definedthedashboardas“aneasytoread,oftensingle page, real-time user interface, showing a graphicalpresentation of the current status (snapshot) and historicaltrendsofanorganizationkeyperformanceindicators(KPIs)toenableinstantaneousandinformeddecisionstobemadeataglance” [5].This typeofvisualdisplaysare critical in sense-makingashumansareabletoprocesslargeamountsofdataifpresented in a meaningful way [17]. The use of learninganalyticstoolsandvisualizationshavethepotentialtoprovideeffective support to instructors by helping them to keepstudentsengagedandachievelearningobjectives[15].Yooetal.[21]conductedareviewofeducationaldashboardsinwhichthey underline the usefulness by mentioning dashboardspresenttheresultsoftheeducationaldataminingprocessandhelp teachers to monitor and understand their student’slearningpatterns.Wecanapplythesameprincipletothedata

collected from a student’s quiz questions. The responsesreceived from the quiz can be used for understandingconceptualandmeta-cognitiveknowledgecomponents[4].Ithas also been noted that very few of the deployed learningdashboardsaddressed theactual impacton instructors [20].Thus,weseeaneed foraReal-TimeAIPoweredEducationalDashboard (RAED) that isdesigned for assisting instructors.Therearetwomainobjectivesoftheproposedresearch.

Objective1:BuildaRAED,whichwillactasadecisionsupporttoolforinstructors.

Objective2:Measurethe impactof theRAEDoninstructorsand understand their trust in using theRAEDwhilemakingdecisions.

The proposed dashboard consists of two components - thevisualization component and the AI component. Thevisualization component will present an entire classroom’sactions in real-time on the dashboard. This will helpinstructorstoanswertwoquestions:(i)wherearemostofthestudentsstruggling?and(ii)onwhichquestionsaremostofthestudentsusinghints?Answers to thesequestionscanbeuseful forproviding furtherexplanationsof certainconceptsimmediatelyafterthequiz.TheAIcomponentofthedashboardwillperformunsupervisedlearningonthecollectedresponses.It will produce clusters of students and also generaterecommendations based on the results, which will bedisplayedonthedashboard.TheserecommendationsmadebytheAIcomponentwillalsobeincludedinthevisualizationsofthe dashboard. These visuals will facilitate the instructors’decision-making process. For instance, an instructor maydecide to give additional questions or teach a particularconceptagainaftergettingrecommendations.Therefore, thecurrent research will also be useful for understanding theusability,impact,andtrustinthefusionofvisualizationsandAIinreal-time.

Figure1:ProposedarchitectureoftheRAED

2. RELATED WORK Thereisagrowinginterestinthedesignanddevelopmentofreal-time systems, such as [14] and [19], that can provideactionableteachinganalyticsinreal-timefordecisionmaking.These real-time systems are also beneficial from students’perspectivebecauseitgivesmoretimetoteacherstoprovideone-on-one support to students [11]. A user-centric teacherdashboard has been developed by Aleven et al. [2] forunderstandinginteractiondataandanalyticsfromintelligenttutoringsystems(ITS).Alevenetal.[2]notedthatadashboardcouldbeusefultoteachersforhelpingtheclasswhileteaching,and for preparation for the next classes. Diana et al. [7]displayed real-time analytics of interactive programmingassignments on an instructor dashboard. From the results,Diana et al. [7] concluded that student outcomes could beaccurately predicted from the student’s program states. Inaddition to that, for helping more students in a classroom,Diana et al. [8] also used themachine learningmodel alongwithapproachmapsforidentifyingandgroupingstudentswhoneedsimilarhelp inreal-time.Holsteinetal. [12]developedthe Luna dashboard by collecting data from interpretationsessions and affinity diagramming from middle-schoolteachers.Thegoalwastounderstandthedashboard’susabilityfrom the teacher’s aspect, as well as its effect on studentslearning. In a recent paper [10], a wearable classroomorchestraltoolforK-12teacherswastestedbyHolsteinetal.The classroom was represented as a dashboard. In thatresearch,mixed-realitysmartglasseswereconnectedtoITSsfor understanding real-time student learning and behaviorwithin the ITSs. A framework consisting of five dimensions(Target, Attention, Social visibility, Presence over time, andInteroperation) has also been proposed for the design andanalysisofteachingaugmentationin[3].

3. PROPOSED CONTRIBUTIONS This section presents the architecture and design of ourproposedRAED.ItfurtherdescribesthefeaturesoftheRAEDanddiscussesitsdesirableproperties,suchasportabilityandexplainability.ItalsoincludesinformationonthecurrentstateofourRAEDdevelopment.

3.1 Architecture and design ThearchitectureofRAEDisshowninFigure1.Studentswillgetaquizinterfaceonwhichtheywillseethequestionsandrespondtothem.Theresponseswillgetstoredinadatabase,whichcanbequeriedbythedashboardengine.Thedashboardengine will be responsible for data preprocessing and datacleaning.Theresultingcleandatawillthenbegivenasinputtothe visualization and AI components. The visualizationcomponentwillproducevisualizationson

1 https://tinyurl.com/qnp46y9

Figure2:DesigncomponentoftheRAED

thedashboard,andtheAIcomponentwillperformclusteringofthedatainreal-time.TheresultsfromtheAIcomponentwillbe visualized and interpreted in terms of recommendations.Currently, we have developed a quiz interface 1 forexperimental purposes, using R and Shiny. This interfacestores results on a Google sheet, and currently this Googlesheetactsasourdatabase.ThedashboardengineisconnectedtotheGooglesheet,whichqueriesdataevery6seconds.Thus,thedashboardisrefreshedevery6seconds.

In this on-going research, we have implemented a designcomponent of the dashboard (shown in Figure 2), whichdisplays real-time visualizations 2 . The essentialcharacteristicsofthedashboardsnotedbyFew[9]aretakenintoconsiderationwhiledesigningourdashboard.Wealsowillbe following four elements of the learning analytics processmodel [20]asa foundation for the conceptualdesign.Thesefour elements are awareness, reflection, sensemaking, andimpact.Atthecurrentstate,ourdesignincludesthefirstthree.Awareness refers to the data, which can be visualized orrepresented in tables as it streams. Reflection focuses onmirroringteachingpractices,andsensemakingcandealwiththeunderstandingat-riskstudents[20].

3.2 Features of the dashboard We propose five unique features of the RAED. We haveimplemented a majority of the features, and details are asfollows.

1)Interactivevisualizations–Thevisualizationsgeneratedonthedashboardarefullyinteractive(showninFigure3)andcan be downloaded in Portable Network Graphics (PNG)format. Instructors can interact with them by zooming in,zooming out, selecting different components of thevisualizations, etc. These visualizations can also providemeaningful information if the cursor hovers over them.Currently,ourdashboardvisualizationsincludescatterplots,barplots,andhistograms.Thescatterplotandbarplotsareused for understanding the quiz score and number of hintsrequestedbystudents.Histogramsareusedtounderstandthe

2 https://tinyurl.com/yx3pht5e

score distribution of the class. Another essential role of theRAED isenhancing theperceptionof instructorsas theycandecidewhattofocuson.

Figure3:InteractivevisualizationsoftheRAED

Figure4:DynamictablesontheRAED

2) Dynamictables–OneoftheuniquefeaturesoftheRAEDisitsdynamictables.Wehaveprovidedasummaryoftheclassandscorecardintheformoftables.Thesetablesgetupdatedinreal-time,andinstructorscansearchaswellassortthetables.TheRAEDalsoprovidesfunctionalitytodownloadthese tables into CSV format for further analysis (shown inFigure4).

3) Portability –The dashboard is designed in R anddeployedontheShinyserver.Thisdashboardisportableandcanbeconnectedtoanydatabaseortool.

4) Real-time–Thedashboardprovidesupdatesofthedataevery6seconds,whichcanhelptocapturestudent’sreal-time interaction during the quiz. Further, we provide theadditional feature of pausing and resuming real-timestreaming. This feature can be especially useful whenanalyzingthedashboard.

5) AI and explainability – This feature is currentlyunderdevelopment.WeplantoemployexplainableAIonthedashboard. It will help instructors to understand how AIprovides results to the dashboard, i.e., how it chooses thenumberofclustersandhowitproducesrecommendations.

4. FUTURE DIRECTION ThefuturedirectionofthisresearchistodevelopaprototypeoftheRAED,testitinclassrooms,andthenconductsurveysto

measureitsimpactandtrust.Futureresearchwillbeheldinthefollowingfourphases.

• Phase1(AIcomponent):Currently,wecanstorestudentids,namesofthecoursetopics,responses,scores,whetherhintsareused,andwhat is the totalnumberof requestedhints. We will be using this information for clusteringstudents and generating recommendations for them. Theprocessofclusteringcanbeusefulforfocusingattentiononstudentswithsimilarcharacteristicsandlearningrates.Thisinformation can help instructors to form support groupswithin the class and to provide personalized guidance toparticular students. For example, students from thehighperformancegroupcanbepairedwithstudentsinthelow-performance group, which can help to improveperformance.

In the first step, similar students will be identified byperformingclusteringonthedata.Thegoalofthisstepistoidentify three clusters (high performance, averageperformance, and low performance). It is essential tovisualize the process of clustering for implementingexplainableAI.Thus,theimplementationofAgglomerativeHierarchicalClusteringmakesasuitablechoice.Usingthisapproach, clustering process will begin with points asindividualclusters,andateachstep,thesimilarpointswillbemergedintoalargercluster[18].Thisentireprocessofclusteringcanbevisualisedbyplottingadendrogram,whichfulfills our goal of explainability. The other advantage ofusing Agglomerative Hierarchical Clustering is that itprovides good resultswhen given small datasets as input[1],asistheexpectednumberofstudentsinaclass.Inthenextstep,informationofstudentsfromtheseclusterswillbeobtained, and a list of concepts that students need toimprove will be derived from the responses. It will alsoprovide suggestions on pairing students during in-classactivities.Thisinformationwillactasrecommendationstoinstructorsandcanalsohelp tounderstandconceptualaswellasmetacognitiveknowledgecomponentsoftheclass.

• Phase2(Design):WewillfocusonthedesignaspectoftheRAED using the iterative design process. In this step, theprototype will be shown, and functionalities will beexplained to the instructors for getting their insights onRAED. Surveys will be provided to the instructors forevaluationsandtoknowtheiradditionalneeds.Questionsinthe surveywill be based on the questionnaire created byParketal.[16].

The results will help us to get inputs on informationusefulness, visual effectiveness, appropriateness of visualrepresentation,user-friendliness,andunderstandingoftheinformation. Changes will be made in the design afteranalyzingresponsesfromthesurvey.Inthenextiteration,RAED will be shown again to the instructors, and theirfeedbackwillbe requested. In thisway, at theendof thisphase,theprototypewillbereadyfortesting.

• Phase 3 (Testing): In this phase, the prototype will betestedinclassrooms.Thequizinterfacewillbeprovidedtostudents, andRAEDwill bemade available to instructors.

Thisphasewillhelpusunderstandthetechnicalproblemsthatmayoccur,suchasserverissues.Improvementswillbemadeasnecessary.

• Phase 4 (Survey): In this final phase, surveys will beprovided to instructors to understand their changes inbehavior, the achievement of the objective, trust in thesystem,effectonmotivationanddecisionmakingduetotheRAED.TheresponseswillhelpustomeasuretheimpactoftheRAEDon the instructor’s decisionmaking. Itwill alsogiveusinsightsaboutthetrustinstructorshaveintheRAED.

5. ACKNOWLEDGMENTS The authors would like to thank DataLab at George MasonUniversity for their support. The authors alsowould like tothankDr.OlgaGkountounaforusefulfeedbackonthiswork.

6. REFERENCES [1] O. A. Abbas. Comparisons between data clustering

algorithms. International Arab Journal of InformationTechnology(IAJIT),5(3),2008.

[2] V. Aleven, F. Xhakaj, K. Holstein, and B. M. McLaren.Developinga teacherdashboard forusewith intelligenttutoringsystems.technology,34:44,2010.

[3] P.An,K.Holstein,B.d’Anjou,B.Eggen,andS.Bakker.Theta framework: Designing real-time teachingaugmentation for k-12 classrooms. arXiv preprintarXiv:2001.02985,2020.

[4] L.W.AndersonandL.A.Sosniak.Bloom’staxonomy.Univ.ChicagoPressChicago,IL,1994.

[5] F.Brouns,M.E.ZorrillaPantaleo´n,E.E. AlvarezSaiz,P.Solana-Gonza´lez,´ A.CoboOrtega,´

E. R.RochaBlanco,M.CollantesVian˜a,C.Rodr´ıguezHoyos,M.DeLimaSilva,C.Marta-Lazo,etal.Ecod2.5learninganalyticsrequirementsandmetricsreport.2015.

[6] S. K. Card, J. D. Mackinlay, and B. Shneiderman. Usingvision to think. InReadings in informationvisualization:usingvisiontothink,pages579–581.1999.

[7] N.Diana,M.Eagle, J.Stamper,S.Grover,M.Bienkowski,and S. Basu. An instructor dashboard for real-timeanalytics in interactive programming assignments. InProceedings of the Seventh International LearningAnalytics&KnowledgeConference,pages272–279,2017.

[8] N.Diana,M.Eagle, J.Stamper,S.Grover,M.Bienkowski,and S. Basu. Peer tutor matching for introductoryprogramming: Data-driven methods to enable newopportunities for help. International Society of theLearningSciences,Inc.[ISLS].,2018.

[9] S.Few.InformationDashboardDesign:Displayingdataforat-a-glance monitoring, volume 81. Analytics PressBurlingame,CA,2013.

[10] K. Holstein, G. Hong, M. Tegene, B. M. McLaren, and V.Aleven. The classroom as a dashboard: co-designingwearable cognitive augmentation for k-12 teachers. InProceedingsofthe8thinternationalconferenceonlearningAnalyticsandknowledge,pages79–88,2018.

[11] K.Holstein,B.M.McLaren,andV.Aleven.

Intelligent tutors as teachers’ aides: exploring teacherneeds for real-time analytics in blended classrooms. InProceedings of the Seventh International LearningAnalytics&KnowledgeConference,pages257–266,2017.

[12] K.Holstein,F.Xhakaj,V.Aleven,andB.McLaren.Luna:adashboardforteachersusingintelligenttutoringsystems.Education,60(1):159–171,2010.

[13] M.Ji,C.Michel,E.Lavou´e,andS.George.Ddart,adynamicdashboard for collection, analysis and visualization ofactivityandreportingtraces.InEuropeanConferenceonTechnologyEnhancedLearning,pages440–445.Springer,2014.

[14] R.Martinez-Maldonado,J.Kay,K.Yacef,M.T.Edbauer,andY. Dimitriadis. Mtclassroom and mtdashboard:supporting analysis of teacher attention in anorchestrated multi-tabletop classroom. In 10thInternational Conference on Computer-SupportedCollaborativeLearning,CSCL2013,pages320–327.InternationalSocietyoftheLearningSciences,2013.

[15] R.Mazza andV.Dimitrova.Visualising student trackingdata to support instructors in web-based distanceeducation.InProceedingsofthe13thinternationalWorldWideWebconferenceonAlternatetrackpapers&posters,pages154–161,2004.

[16] Y. Park and I.-H. Jo. Factors that affect the success oflearning analytics dashboards. Educational TechnologyResearchandDevelopment,67(6):1547–1571,2019.

[17] S. Shemwell. Futuristic decision-making. ExecutiveBriefingBusinessValuefrom,2005.

[18] P.-N.Tan,M.Steinbach,andV.Kumar.Introductiontodatamining.PearsonEducationIndia,2016.

[19] M.Tissenbaum,C.Matuk,M.Berland,L.Lyons,F. Cocco,M.Linn,J.L.Plass,N.Hajny,A.Olsen,B.Schwendimann,etal.Real-timevisualizationofstudentactivitiestosupportclassroomorchestration.Singapore:InternationalSocietyoftheLearningSciences,2016.

[20] K.Verbert,E.Duval,J.Klerkx,S.Govaerts,andJ.L.Santos.Learning analytics dashboard applications. AmericanBehavioralScientist,57(10):1500–1509,2013.

[21] Y.Yoo,H.Lee,I.-H.Jo,andY.Park.Educationaldashboardsforsmartlearning:Reviewofcasestudies.In Emerging issues in smart learning, pages 145–155.Springer,2015.