Paper Architecture Paper Architecture and and Exam Scoring ApplicationExam Scoring Application

Narcis LozanoNarcis LozanoHidetoHideto OdaOda

Masaki NakagawaMasaki NakagawaTokyo University of Agriculture and Tokyo University of Agriculture and

TechnologyTechnology

Table of contentsTable of contents

Pen interfacesPen interfacesPaper architecturePaper architectureExam scoring applicationExam scoring applicationConclusion Conclusion Future workFuture work

Pen InterfacesPen Interfaces

Devices:Devices:–– Graphic tabletsGraphic tablets–– MonitorsMonitors–– Tablet PCsTablet PCs–– PDAPDA

Characteristics:Characteristics:–– Constantly connected to Constantly connected to

the computerthe computer–– Instant visual feedbackInstant visual feedback–– High interactivityHigh interactivity

Pen InterfacesPen Interfaces

CharacteristicsCharacteristics–– Use paperUse paper–– Online data stored in memoryOnline data stored in memory–– Delayed data processDelayed data process–– Low interactivityLow interactivity

DevicesDevices–– UltrasoundsUltrasounds–– InfraredInfrared–– Patterned PaperPatterned Paper

Need for an unified architecture for Need for an unified architecture for pen and paper devicespen and paper devices

Several brands and technologies.Several brands and technologies.No standard neither de facto standard for pen No standard neither de facto standard for pen devices that write on paper.devices that write on paper.Several versions must be developed for each Several versions must be developed for each pen device that must be supported by the pen device that must be supported by the system.system.

Application

Application’

Application’’

Paper ArchitecturePaper ArchitectureUnified software architecture.Unified software architecture.Following Following purepure lazy recognition (do not show lazy recognition (do not show recognition until needed). recognition until needed). Allows using several types of pen devices with Allows using several types of pen devices with the same application.the same application.

ApplicationInkML PaperInterface

XML

Template

Paper Interface InputPaper Interface Input

Server

Client

Client

Client

InkMLFile

<Ink><Trace>

<Point X = 1 Y = 1><Point X = 31 Y = 11>

</Trace><Trace>

<Point X = 14 Y = 61>…

The users fulfils the printed document using the device of their choice. When they connect the device to the computer, the data, which is in a proprietary format, is converted into InkML using a client application. This InkML file is then sent to the server.

TemplatesTemplatesInput by writing in a Input by writing in a printed documentprinted documentAreas:Areas:–– QuestionsQuestions–– CheckboxesCheckboxes–– Input with recognitionInput with recognition–– Raw inputRaw input–– Virtual keyboardVirtual keyboard

Contextual information Contextual information neededneeded–– Template filesTemplate files

Template creationTemplate creation

ORIGINAL DOCUMENT EDITOR

TEMPLATEFILE

SERVER

IdentifierSegmentation

SharedIdentifierTemplate

Identifier２３０

Segmentationand

data process

Data process in ServerData process in Server

Template２３０

Template Folder

ＸＭＬResult

File

＜ＸＭＬ＞<seqField>

<content>Student Id</content><content>05646211</content>

</seqField><question>

<usrAnswer>1196</usrAnswer><result>CORRECT</ result >

</question1>。。。

InkＭＬOnlineData

OutputOutput

SERVER

ＸＭＬResult

File

＜ＸＭＬ＞<seqField>

<content>Student id</content><content>05646211</content>

</seqField><question>

<usrAnswer>1196</usrAnswer><result>CORRECT</ result >

</question1>。。。

ＸＭL Viewer

Web Page

Application

Pen Devices in EducationPen Devices in EducationDevices:Devices:–– Electronic whiteboardElectronic whiteboard–– Tablet PCsTablet PCs–– PDAsPDAs

Applications:Applications:–– Use on lecturesUse on lectures–– Video lecturesVideo lectures–– Collaborative learningCollaborative learning–– Learning by gameLearning by game--like competitionlike competition–– Etc.Etc.

Pen & Paper Interface ApplicationPen & Paper Interface Application

Systems designSystems design–– Students fulfill exams using a pen & paper Students fulfill exams using a pen & paper

devicedevice–– Teachers score the answers using the exam Teachers score the answers using the exam

scoring applicationscoring application–– Results and statistics are calculated and sent to Results and statistics are calculated and sent to

the studentsthe studentsExam scoring applicationExam scoring application–– Not fully automatic scoring but hybrid approach Not fully automatic scoring but hybrid approach

Problems withProblems withautomatic evaluationautomatic evaluation

Segmentation and Segmentation and recognition errorsrecognition errorsQuestions with Questions with several possible several possible answers:answers:–– Free text questionsFree text questions–– Even short questions,Even short questions,

There are synonyms.There are synonyms.

Hybrid evaluationHybrid evaluation

Boolean questions and questions with one Boolean questions and questions with one answer:answer:–– Automatic scoring.Automatic scoring.–– Handwriting recognition when necessary.Handwriting recognition when necessary.Rest of the questions:Rest of the questions:–– TeacherTeacher’’s scoring assisted by computers.s scoring assisted by computers.–– Handwriting recognition is not performed.Handwriting recognition is not performed.

Application FlowApplication Flow

PaperArchitecture

ServerScoringResult

Computer AssistedScoring Application

Xml ResultFiles

Computer Assisted Exam Scoring InterfaceComputer Assisted Exam Scoring Interface

Video of the system runningVideo of the system running

Answer reorganizationAnswer reorganization

Question 1Question 2Question 3Question 4

Question 1Question 1Question 1Question 1

Question 2 Question 3 Question 4Question 2Question 2Question 2

Question 3Question 3Question 3

Question 4Question 4Question 4

Video of marking by student or by answerVideo of marking by student or by answer

Video of correctionVideo of correction

Preliminary EvaluationPreliminary Evaluation

Exam scoring of 15 exam answer sheets.Exam scoring of 15 exam answer sheets.Exam consisting of vocabulary and fillExam consisting of vocabulary and fill--inin--thethe--gaps questions.gaps questions.Processed by the Paper InterfaceProcessed by the Paper InterfaceWe asked 7 people to score the answer We asked 7 people to score the answer sheets using the scoring application.sheets using the scoring application.

AdvantagesAdvantages

Answer reorganization Answer reorganization seems efficient for seems efficient for teachersteachers

Relatively low costRelatively low costGeneration of detailed Generation of detailed statisticsstatisticsPaper versions of the Paper versions of the original exam available.original exam available.

Table 1. Experimental result.

40.001220No. 7

52.171123No. 6

41.671424No. 5

50.001224No. 4

45.451222No. 3

18.18911No. 2

45.831324No. 1

Time reduction ratio (%)

Comp. assistedmarking time (min.)

Manual marking time (min.)

Partici-pant

Preliminary EvaluationPreliminary Evaluation

ProblemsProblems–– Missing strokesMissing strokes–– Incorrect segmentation of the answers caused Incorrect segmentation of the answers caused

by displacement of the document.by displacement of the document.

ConclusionConclusion

Paper architecturePaper architecture–– Unified platform for pen and paper devicesUnified platform for pen and paper devicesExam scoring applicationExam scoring application–– Automatic scoring for selection type questions Automatic scoring for selection type questions –– TeacherTeacher’’s scoring for free text questions.s scoring for free text questions.–– Question reorganizationQuestion reorganization–– OneOne--click question evaluation, statistics click question evaluation, statistics

export and automatic eexport and automatic e--mailing of exam mailing of exam results to the students were also implemented. results to the students were also implemented.

Future workFuture work

Remaining issues:Remaining issues:–– Students and teachers benefits. Added effect Students and teachers benefits. Added effect

v.sv.s. mental burden, human error rates, etc.. mental burden, human error rates, etc.–– Unexploited information from digital ink. Unexploited information from digital ink.

StudentsStudents’’ answering speed or fluency of answering speed or fluency of writing might signal something about the writing might signal something about the student confidence.student confidence.

Improvement of the segmentation.Improvement of the segmentation.Evaluation in real situations for a larger Evaluation in real situations for a larger number of students.number of students.

Paper Architecture Paper Architecture and and Exam Scoring ApplicationExam Scoring Application

Narcis LozanoNarcis LozanoHidetoHideto OdaOda

Masaki NakagawaMasaki NakagawaTokyo University of Agriculture and Tokyo University of Agriculture and

TechnologyTechnology