project document cover sheet -...

Project Acronym: AQuRate Version: 1.0 Contact: Graham Alsop Date: 19/03/08

Project Document Cover Sheet

Project Information Project Acronym AQuRate Project Title AQuRate: A QTI Authoring Tool Start Date 1st March 2007 End Date 31st March 2008 Lead Institution Kingston University Project Director Graham Alsop Project Manager & contact details

Graham Alsop, Learning Technology Research Group, Kingston University, KT1 2EE, [email protected], 02085477889

Partner Institutions CARET, University of Cambridge Project Web URL aqurate.kingston.ac.uk Programme Name (and number)

JISC Capital Programme, Call II Assessment, a). Item Authoring

Programme Manager Myles Danson

Document Name Document Title Final Report Reporting Period N/A Author(s) & project role

Project team (Final edit by Graham Alsop and James Orwell)

Date 31 March 2008 Filename aquratefinalreport0308.doc URL N/A Access Project and JISC internal General dissemination

Document History Version Date Comments

Revision 26 31 March 2008

Edited final version sourced from Google Docs

mailto:[email protected]


JISC Final Report

AQuRate

Final Report

Authors: Graham Alsop, James Annesley, Zhen Cai, Alicia Campos, Martin Colbert, James Orwell (Editors: Graham Alsop and James Orwell).

Contact Person: Graham Alsop ([email protected])

31 March 2008

mailto:[email protected]


Table of Contents

1. Acknowledgements 4 2. Executive Summary 5 3. Background 6 4. Aims and Objectives 8 5. Methodology 8 6. Implementation 12 7. Outputs and Results 18 8. Outcomes 22 9. Conclusions 23 10. Implications 23 11. References 24 12. Appendices 25


1. Acknowledgements

This project was funded under the JISC eLearning Capital Programme, Call II Assessment, Strand a) Item Authoring. Close collaboration took place among the other two funded projects in this call: AsDel (University of Southampton managed by Dr Gary Wills) and Minibix (CARET managed by Dr Steve Lay). In addition useful input was made by the Programme manager Myles Danson, the Advisory Group and OSS Watch. Many thanks to the project team. Without their work this would not have been possible (Graham Alsop, James Annesley, Zhen Cai, Alicia Campos, Steve Lay (Consultant,) David Livingstone, James Orwell, Martin Colbert and Graham Smith (Consultant).


2. Executive Summary

This final report describes the design and implementation of software for the creation and editing of assessment questions. The software was designed to work in conjunction with two other projects, Minibix and Asdel. The assessment questions can be stored in (and retrieved from) a Minibix repository. The Asdel project provides components to deliver a test composed of such assessment questions.

A major motivation for the trio of projects is version 2.1 of the QTI (Question Test Interoperability) standard. This standard has addressed several shortcomings in versions 1.x that limit the interoperability of software and content used by different institutions and companies. By implementing several important components of software that use QTI 2.1, the projects have provided a platform for further adoption of the standard. Furthermore, in using the standard to implement a complex ‘Assessment Scenario’ (creation, editing, storage, selection and delivery), some minor flaws in the standard were identified. The associated remedies were also identified and earmarked for inclusion in the next release of the standard.

For the AQuRate project, two key requirements were that the software is ‘open source’, and that the software can be used across the common platforms (Windows, Mac, Linux, etc). The identification of the ‘New BSD’ form of license, to be appropriate for all projects, was an early project activity. Towards the end of the project, the source and executable releases were examined to ensure that all elements complied with the various third party licences. The requirement to make the project results accessible from multiple platforms was one factor in deciding to use Java as the main software development environment. In the main part, this requirement was satisfied: the software could be executed on any platform for which there is a Java Run-Time Environment. However, several small issues and problems persisted on some platforms. This was partly as a consequence of detailed testing taking place largely on Windows platforms only.

For the other two projects, the main integration requirement was to provide input and output functions with the Minibix repository. This was achieved, and demonstrated at the Assessment Special Interest Group meeting in Cambridge on 15th February 2008. The joint result of the two projects is an open source system that can create, store, retrieve, and edit Assessment questions, using a web-based repository and a desktop-based authoring tool. It is perhaps not unexpected that some of the requirements for this interaction were identified fairly late in the project timeline. In particular, the AQuRate team had initially failed to appreciate the requirements for meta-data (such as author name, institution etc) and appropriate elements were subsequently added to the authoring tool interface.

All three projects used a two-phase planning process. A progress review, between phases, identified the most useful directions for the second phase. It was decided that during this second phase the number of supported interaction types would not be extended beyond the commonly used ‘core’ set of interactions. The extended types could be accommodated at a future development stage. For the second phase, the focus was developing the interaction with the Minibix repository, and eliminating the small problems associated with the graphical user interface controls. Collaborative Design Sessions provided usability testing.

In conjunction with Dr. Graham Smith, The AQuRate project successfully re-used the ‘JAssess’ software to provide ‘rendering and processing’ of Assessment Items, in a browser, without requiring any server (i.e. using the AQuRate desktop application alone). In addition to the Desktop tool, server-based prototypes of the Authoring Tool were developed, all using a common API.

By the end of the project, there were over 1,500 downloads of the software from the project site, with numerous expressions of interest to continue the development. The project team were pleased with the final results, and look forward to continued involvement in the development of tools using the QTI standard.


3. Background

The AQuRate project looked to develop an open source, standards compliant and platform-independent tool to enable the authoring of question items conforming to the current IMS QTI 2.1 Specification. From the start of the project close collaboration took place with both the Assessment Delivery (AsDel) and Item Banking (Minibix) projects. Computer Aided Assessment is still a fast developing area and QTI Ver 2.1 to date has had limited implementation. This document should be read in conjunction with the Project Proposal and Interim Reports.

The QTI domain

An overview of the QTI standard is given in [ref: IMS Question and Test Interoperability Overview Version 2.1 Public Draft (revision 2) Specification http://www.imsglobal.org/question/qtiv2p1pd2/imsqti_oviewv2p1pd2.html]. The AQuRate project provides an implementation of the AuthoringTool component. The output elements of the AuthoringTool are stored in an ItemBank. A TestConstructionTool downloads items or questions from the ItemBank to build an Assessment. The TestDeliveryTool is used to then deploy the Assessment to students. The LearningSystem or VLE (Virtual Learning Environment) could be used in all four of these steps, linking into the other systems, e.g. student databases or course information and also providing management and security facilities.

Multiple choice questions and QTI terminology

The JISC CAA (Computer Assisted Assessment) Centre define a multiple choice question as comprising of a Stem, Options, Key, and Distracters. In QTI terms, the Stem is called the Prompt. This is different from any other text within an ItemBody. The ItemBody contains not only the Interactions but also the set of HTML 4.0 elements. Technically these other types could contain the Options, but are intended to provide the formatting controls. Writing a good multiple choice question is a difficult task and the CAA Centre provides guidelines on the topic.

Question Test Interoperability

Question Test Interoperability (QTI) is an IMS (Instructional Management Systems) specification for multiple choice questions design. The specifications are implemented as XML schemas. The aim is to provide a standard form for these questions and for them to be exchanged between computer-based assessment systems. Two QTI specifications exist, version one [ref:] and version two [ref]. A description of version one can be found at here [ref: IMS Question & Test Interoperability: An Overview Final Specification Version 1.2 http://www.imsglobal.org/question/qtiv1p2/imsqti_oviewv1p2.html#1398934] and the main structure are an Assessment, a Section and an Item (ASI). Essentially the problems with version 1 are based around the flexibility in its design. This created a situation where a question with the same semantic meaning could be implemented in a variety of ways. This limited the ability of the documents to be shared via computer processes. Version 2 is designed to address this issue. A QTI question is defined as an AssessmentItem and the questions which it can contain, potentially multiple in number or composite, are known as Interactions. The Interactions are displayed to the user within an ItemBody. Other important components are feedback description components, template components, response components, formatting components (any more). Many different Interactions are defined and a subset is given in the Method Section. A composite question has implicitations of complexity in terms of designing questions which answering the question. Templates exist to handle standard response types e.g. single or multiple response

http://www.imsglobal.org/question/qtiv2p1pd2/imsqti_oviewv2p1pd2.html

http://www.imsglobal.org/question/qtiv1p2/imsqti_oviewv1p2.html#1398934


questions.

LOM

IEEE (Institute of Electrical and Electronics Engineers) have developed a standard meta-data Classification Scheme called LOM (Learning Object Model). QTI version two uses LOM meta-data to describe resources associated with a question. This meta-data and the resources, e.g. the questions and images, are packaged using the IMS CP (Content Package). A CP is a zip file which contains a manifest of the resources. CPs are a subset of the SCORM (Shareable Content Object Reference Model) specification for learning object delivery designed by the Advanced Distributed Learning (ADL). The manifest contains the meta-data about the resources, e.g. Author, organisation etc. The use of CPs resolve issues related to resource location and the specification of meta-data about the resources.

JASSESS

JASSESS is an online rendering tool for QTI Ver 2.0. This offers the opportunity for questions to be seen as they would be delivered, responded and offered feedback to. This work was developed using Java and was considered due to the development team looking to use an offline/desktop renderer that used Java (http://qtitools.caret.cam.ac.uk/index.php?option=com_content&task=category&sectionid=4&id=32&Itemid=32).

Existing JISC Work

The project proposal included the following list of JISC projects and tools that, in the course of the AQuRate project, would be reviewed and considered for use:

1. R2Q2, QTItools (distribution point for JASSSESS, PyAssess and Migration tool) APIS rendering services for viewing and testing QTI questions

2. Using the PyAssess and QTITools toolkit for QTI Specification testing 3. Using SPAID for item banking and packaging 4. Using TOIA / E3AN for pre-existing QTI content and item banking 5. Using TOIA for assessment delivery and item exchange 6. Supporting the RQP protocol for communicating with other QTI assessment services

At project initiation and commencement these resources were reviewed and assessed. The following points were noted as a consequence:

1. At the start of the project R2Q2 and APIS offered online rendering of assessments rather than items (using QTI 1 and 2 respectively). PyAssess used Python rather than Java

2. As above. 3. Minibix was working on Item Banking, so it was decided to wait until the integration at Phase 2 and

work with Minibix’s solution to this issue. 4. The need for example content was fulfilled through the use of JASSESS examples for considering the

development of the authoring tool. 5. AsDel and Minibix provided this need. 6. The decision to offer a desktop service initially negated the initial need to consider RQP.

http://qtitools.caret.cam.ac.uk/index.php?option=com_content&task=category&sectionid=4&id=32&Itemid=32

http://qtitools.caret.cam.ac.uk/index.php?option=com_content&task=category&sectionid=4&id=32&Itemid=32


4. Aims and Objectives

The aim of the project was to produce a desktop-based system that enables the production of QTI Items that represent as wide a range of QTI 2.1 question types as possible within the project timescale. The system followed an open source development and RAD model and was designed to work on a wide a range of software platforms (PC, Linux and Mac). In addition the system architecture allows the core functionality of the software to be extensible to other user environments and interfaces, or to developers who wish extend the system capabilities. The original aims and objectives as stated in the project proposal were: “The core functionality of the system will be to:

• Create new QTI 2.1 items • Edit existing QTI 2.1 items • Import and Export QTI 2.1 items (from / to other pre-existing systems) • Package QTI 2.1 Items as IMS Content Packages that are SCORM 2004 compliant • Use external web services for testing the rendering and response of questions using one or more of

Assessment Delivery, Assessment Rendering and QTI validation systems • Use external web services for depositing and retrieving questions from Item Banks • Dual layer architecture separating the presentation and core logic components of the system (for

extensibility) The system will be designed with the following aims in mind:

• Offline creation of assessments • Offline rendering of assessments • Production of new system interfaces e.g. plug-ins, web services, server pages • Extension and Development of new System Capabilities e.g. extra question types, use of items with

sequenced content, support for custom interactions”. These will be revisited and analysed later in the section entitles “Outcomes”.

5. Methodology

The project used a Rapid Application Development approach using a Dynamic Systems Development Methodology and was divided into two Phases. Phase 1 represented working alone for the first half of the project developing the application whilst Phase 2 was the focus of implementation and interface development among the two other projects under this call dealing with item banking and assessment delivery. During the Functional Iteration Phase it was recognized that AQuRate needed to conform to IMS QTI 2.1, IMS CP 1.1 and SCORM 2004 standards. Due to the complexity of these standards, work began by analysing and prioritising the components of each specification with a view to developing a tool with as complete a set of functionalities as possible. Initially, research into existing commercial and Open Source authoring tools was carried out. This research aimed:

• To observe the grade of accomplishment of these tools with both IMS QTI and IMS CP specifications and gather information about how previous tools faced the implementation of complex QTI tasks

• To define and establish additional functional and non functional requirements for the tool. In this regard, local rendering functionality for testing assessment items was considered useful for end users, as was an HTML editor for creating assessment items with basic styles and format, and images.


• To identify the strengths and weakness of graphical user interface of existing authoring tool: appropriate vocabulary to avoid confusing less expert users, graphic components to represent and show QTI content, means to allow the user to carry out specific QTI functionalities, such as partial scoring or the definition of modal and integrated feedback.

There are 21 Question Types in the QTI 2.1 standard. These were divided into two sets: core and extended. The first (‘core’) subset were be implemented in the first phase of development. The extended set were decided not to be pursued after consultation with the Advisory Group and JISC. Essentially it was decided that it was better to offer a more polished authoring tool, rather than one that was spread widely as regards question types, but thinly in terms of usability. The core set is listed and described below:

Phase 1 (core set)

Description

Choice Single, multiple answer responses Facilitates possible of giving credit for some wrong answers

Order Ordering and interaction Needs a special render to deal with the reordering

Associate Associating interaction Associate pairs direction unspecified, needs specialist rendering s/w

Inline Choice Drop down box (choice) match interaction

Text Entry Text entry interaction

Hotspot Hot spot interaction Select hot areas in a graphic illustration

Graphic Order Graphic order interaction A hot spot interaction with ordering element similar to ordering and interaction

Slider Slider interaction A graphical slider used to


specify a particular value within a range of values

A technology was required that allow the processing of the schemas provided in the specifications by the authoring tool. JAXB was considered and chosen for this. Since AQURATE had to be an open source project it was required to adopt an appropriate license. OSS Watch group guided the team during the process of selection. A New BSD license was finally adopted. In the Design and Build Phase a user-centred approach was naturally adopted for the design of the authoring tool’s Graphical User Interface (GUI). Based on the review of existing systems, a design for the user interface of the initial prototype was produced in MS VISIO, and covered the authoring of choice type questions. As interactive prototypes became available, collaborative design sessions with example users were conducted to give early evaluative feedback on the GUI implemented, and to suggest design improvements. Two GUI design iterations were conducted. Illustrative text and images for the “New ...” dialog was also created, when the second design iteration showed that this content was required.

Application Environment (developed from the project proposal)

It was stated that a desktop application was going to be developed that deployable on a range of platforms. It was decided that the initial prototype would be stand-alone and create/save xml without reference to an outside service (leaving content packaging and interaction with outside services to a later timebox during Phase 2). The architecture stated in the proposal is represented in the diagram below.

Figure 1: Planned Architecture

The initial prototype had a tightly coupled interface and the API was defined and isolated in a later phase of development. The proposal stated that “the Desktop Application should support extensibility, to e.g. allow an authoring component to be ‘plugged in’ to the ‘Eclipse’ integrated development environment. The requirement for extensibility suggests that an ‘authoring API’ should be identified and abstracted from the Desktop Application, to be used in (as yet undetermined) extended environments such as an Eclipse plug-in, an authoring web-service or possibly Server Pages to allow a Web-Based Authoring tool. “ The outcome is illustrated in Figure 2.


Figure 2:Actual Architecture

The tool is able to manage and process an assessment item in two different “formats”: 1. As a single QTI2.1 xml file with associated resources (e.g. image files) 2. As a content package, a zipped archive containing the QTI2.1 xml file, the associated resources, and

the ‘imsmanifest’ xml file containing the metadata associated with this question. The choice of these two “formats” for the AQuRate application was motivated by considerations of software development and interoperability standardization. The first format is a basic development tool for staged implementation of loading and saving of QTI content. The second format is a key component of the SCORM 2004 standard. Figure 2 includes a web-based perspective of the authoring tool. It consists of a Java Applet running in the client-side which is able to work with items directly retrieved or deposited in a remote bank. This service could be also implemented by offer the user interacting with Java Server Pages and Servlets which would be responsible for communicating with the external repository. Both approaches require the client side security issues to be addressed, e.g. to allow access to client file systems. The usage of content packages to share and exchange information among AQURATE and MINIBIX systems involves to work with two different specifications: the IMS Content Packaging 1.1.4 standard, intended to describe the assessment item and resources structure, and the IEEE Learning Object Metadata specification used to describe the item in multiple categories such as: General, Lifecycle, Metadata, Technical, Educational, Rights, Annotation and Classification. However these meta-data are scenario-specific and generated by other means. AQURATE reduces the scope of these categories to the item authoring application. To produce and load assessment items with metadata associated involves working with IMS CP and LOM schemas. Once more, JAXB technology is used to bind these schemas allowing the validation of metadata elements. The interoperability existing with MINIBIX requires establishing and place in common some of these meta-data.

AQURATE and MINIBIX Integration

There were two important elements required to develop the necessary communication between the AQuRate


desktop application and the Minibix item bank. The first element is the capability to export SCORM 2004 Content Packages correctly. The second element is the provision of a web interface to the Minibix server, which can deposit and retrieve the authored questions. The questions must be packaged as content packages and authoring these content packages presented a challenge. At the start of integration phase AQuRate loaded and stored its questions in XML format, the raw format of the QTIv2 questions. The packaging of resources is handled by the content package and these resources are commonly images. This avoids problems of storage location and provides a mechanism to supply additional meta-data. The additional meta-data provides identification and cataloguing meta-data. One use is to allow an item bank to index the content packages, which provides advanced searching capability. The second step relies upon developing a communication link between AQuRate and Minibix. The developers for these projects exchanged ideas as to how the interface should be implemented, e.g. CGI (Common Gateway Interface), Web Services, SOAP (Simple Object Access Protocol). Once the method is decided, the details of the implementation must be shared, e.g. how the items are streamed over the Internet, what the extra meta-data will be, how the items can be identified in the bank and how we can download a particular item, etc.

Graphical User Interface

The overall problem was to reveal and control many useful functionalities, without requiring authors to comprehend the idiosyncratic detail and complex structure of QTIv2 specification – authors just want to write good questions quickly. Also, the user group included both ‘novice’ and ‘experienced‘ authors, who want to write both simple and complex questions. The eventual design was a hybrid of traditional ‘wizard’ and ‘document’ based user interfaces. It offered a clear sequence of steps to guide novice users through the authoring process and which experienced authors can use to work systematically. However, it also let users act directly upon the question they are creating, and also encouraged authors to explore the educational potential of QTIv2, and so learn to write more complex, and educationally more valuable questions.

The relevant user interface styleguide is the standard and advanced volumes from SUN (Java Look & Feel Design Guidelines, 2nd Edition, SUN Inc. 2001, and Java Look & Feel design Guidelines: Advanced Topics, Sun Inc. 2002). Relevant guidelines for online content is provided in Horton (1990), Designing and Writing online Documentation, Wiley, and Redish (2007), Letting Go fo the Words, Morgan Kaufman.

6. Implementation

RAD and MVC

In the initial stage of development, a RAD (rapid application development) method was used. Its purpose was to quickly develop a simple application to prove the correctness of some key design concepts, in particular, the design of the GUI (graphical user interface). Basic functionalities and the GUI of a few core question types were developed with this method. There was no API developed at this stage, and all the java code evolved around the GUI components. In this “Phase 1”/Functional Model Iteration of the application development stage, better understanding of the QTIv2 specification was achieved and it was very clear that a different development method needed to be used in order to maintain the scalability and robustness of the author tool. The phase 2 of the development process was designed to address those issues encountered during phase 1. MVC (model view controller) architecture was the perfect solution to the flexibility problem with the phase 1 prototype application. It separates the functional Java classes from the GUI classes, thus allowing independent development of the two major components. This also created an opportunity to start the


development of the an API as part of the authoring tool. The AQuRate API was based on set of JAXB (java architecture for XML binding) generated from the QTIv2 XML schema. Along with the all the methods allowing the creation/loading/modification of QTI components, it also provides real-time syntatic validation check against the QTIv2 schema. AQuRate document models were built on top of the API, each model classes is responsible for the initialization of a new question type, loading an existing question and carrying out semantics check of each components of a question. There is a controller class for each of the document model classes, their purpose is to pass information from the view classes (GUI classes) to the document model classes or visa-versa. This passed information reflects changes to the QTI document by the user via the GUI. The diagram below shows the MVC approach.

JAXBclasses

AQuRate APIQTI

Document Controller

Editing View

XML View

Browser

MODEL VIEW CONTROLLER

html files

Figure 3:Model View Controller Approach

Licensing

The AQuRate project is licensed under a New BSD license, which gives user full rights to modify and redistribute the code. This also means if the tool is redistributed, all the third party libraries which may or may not have a BSD license must be distributed in accordance to their own license agreements. Two of the


major third party libraries required by AQuRate are JAXB libraries and “BrowserLauncher” libraries. JAXB libraries are needed because the java version 1.5 does not contain these libraries, therefore these libraries needed to be redistributed in the binary runnable version of AQuRate. The license for the JAXB classes is the none-open source Sun license. Providing there are no changes to the jar packages downloaded from the Sun website, these libraries can be redistributed with their original license included. The other third party library is the “Browser Launcher” library, it provides the authoring tool with the ability to launch a browser on any computer platform when previewing a question. Its license is LGPL, which is not completely compatible with the AQuRate license. This means its source code could not be included in the AQuRate source tree. The solution was to use the binary version of the library during development, this binary version is provided as a separate download.

Web-based Authoring

A web-based authoring tool provides a question author with the ability to create QTI questions via a web-browser. Potential benefits include no product deployment processes, moving the code to allow authoring within a VLE. The issues here involve the sandbox within which the Java VM (Virtual Machine) must operate. This means reading and writing to disk is prohibited. The authoring tool must be altered, so either no reading or writing occurs or so it runs within a Java server, which allows reading (and potentially writing). The former requires the removal of the standalone render and it replacement with an online rendering service, e.g. The University of Southampton R2Q2 tool. There is now a prototype in action that offers no reading or writing to disk and runs locally. The only way to read is using URLs. The current desktop application requires a lot of disk activity so in the loger term it would be best to consider writing a client server application where resources are loaded and saved on the server. Such a serve could be deployed as a a JSP or servlet or as a simple web application (like Minibix). Alternatively off -the-shelf code that converts the java application to AJAX web code could be used. This is done in realtime. For example, using ajaxswing from “cremetec”.

Eclipse plugin

An ‘Eclipse plug-in’ version of the project is now available. http://code.google.com/p/qtidevelopment/downloads/list

Multiple Interaction

In order for Mutiple Interaction to be implemented some further re-engineering needs considering that is out of scope for this project, but would offer an interesting potential redesign for the interface. A report on this will be available via the project website.

Use-Case Analysis

Along the first phase of this project the team centred in the development of the core question types proposed in the plan. The set of use cases covers the initial functional requirements demanded and the most relevant QTI functionalities identified. They are enumerated and explained as follows:

1. Create an instance of a core assessment type The authoring process provided by AQURATE is form based. In the interface there are implemented two views of an assessment item:

http://code.google.com/p/qtidevelopment/downloads/list


a) The Form View, which shows assessment item fields to edit the question. This view tries to group the UI components according with the QTI specification. Therefore, it presents a wording declaration part, which is common to most of question types and, a responses definition part, specific to each particular type. This allows, by one hand, the familiarisation with QTI elements for novices users and, by the other hand, a systematic workflow for those who are more advanced.

b) The XML View, which shows a QTI 2.1 assessment item XML document. This view is intended those users who are interested in or those who have knowledge about it. It reflects the information which is being edited in the form view and other QTI elements discarded in the Form View.

During the authoring process, formatting is very time-consuming task and is highly demanded by users. To achieve this feature, it is implemented a HTML editor with a tool bar for inserting the most frequently used formatting tags (bold, italic, underline).

2. Save this instance in the local machine. It is implemented is such way that the tool saves the assessment item currently edited and the associated resources (images).

3. Load and modify an assessment items. The tool must be able to work with assessment items compliant with previous versions of QTI specification. Therefore the loading process requires as much flexibility as possible. Initially the tool carries out a testing process of the QTI elements which the item contains. This process validates the compulsory QTI elements (response declaration, item body, etc.). The format in the content it is also significant not admitting the inclusion of special characters. This is especially important in the identifier attribute common to different QTI elements.

4. Edit the QTI elements associated with an assessment item. QTI 2.1 specification provides a wide variety of attributes to associate with an assessment item. It was needed a process of understanding of these elements to determine their relevance to the end user. There were distinguished and separated two groups: by one hand, general attributes belonging to the question (title, shuffle, time dependant, etc.), by other hand, specific attributes related with a particular response (score and feedback).

5. Allow the user to provide a partial scoring for the assessment. The scoring of an assessment item can be performed through two different methods: In the first place, by using the element response processing. This is considered the most complex QTI element to implement in terms of user interface and in terms of control it through the Model View-Controller approach adopted in this project. The response processing is not implemented for none of the question types; instead there are used provided templates which contains this element but don’t allow partial scoring. In the second place, the partial scoring can be performed through the mapping element. This QTI component permits to establish a score for a particular response within a question. It is suitable to be used in questions whose response is compound by a set of responses and is implemented for: Choice, Associate, Hotspot, Inline interactions. Text Entry and Slider requires extending their interface with an extra component to define the mapping of the possible answers given by candidates. Order and Graphic Order are problematic regarding with the implementation of this element due to the response to be evaluated is the whole set of choices in a particular order. A provided template is used to score these questions. This template does not provide partial scoring.


During the second phase of the project, the set of use cases was extended according with the requirements of integration with MINIBIX Item Bank. Such integration process and the implementation details are described bellow.

AQURATE and MINIBIX Integration

Cambridge already had developed a content package reading API (Application Programmers Interface) and to help Kingston quickly develop content package generation code, this was extended to include writing support. These packages were delivered via a concurrent source code management system (Subversion) and a configuration management system (Apache Maven). Once the APIs were integrated into the AQuRate source tree, the content packages were built. The new items should be inserted into a content package, with the associated resources and meta-data. The writing functionality was developed in collaboration with AQuRate and the minimum meta-data specification required by Minibix was decided as the development progressed. At the projects main demonstration at 18

th February 2008, the meta-data supported was “question title” and

“author”. The format of the meta-data information was important, e.g. the meta-data containing the author identification must have had a vCard (Versitcard by the Versit consortium) format. Some obstacles arose during the storage of image resources associated with a question. The references to these resources are made in the content package’s meta-data file or ‘manifest’, and these had to be amended to reflect the relative paths according to the structure of directories and files within the package. The graphic interactions caused further problems, e.g. Hotspot and Graphic Order. These types involved an extra process to examine their interaction meta-data, e.g. the image upon which the user selects the ‘hotspots’ is contained in a different area. This also affected the ‘prompt’ meta-data. A common identifier for a question was important for AQuRate and Minibix to allow a particular question to be exchanged. Minibix provided the identifier name, ticket, and format, string. This is provided when the item is first stored and associated with the question inside AQuRate. AQuRate added question Import and Export menu items, which provided the user with the ability to enter the details about the question e.g. item bank address and the meta-data and to optionally provide a ticket. To upload, download and amend questions from the item bank, simple CGI technology is used. This provides a simple mechanism for “Get” and “Set” functionality over the web. A Set request is made by supplying the Minibix web server web address or URL (Uniform Resource Locator). The end of the address specifies the repository. Once the connection is initiated, a further request deposits the content package. A request to Get a content package is using the same URL and repository name but also supplying the ticket number. The server responds with a package of meta-data with a binary component, which is the content package. The meta-data contains information such as the creation time, title, etc. An amendment to a question is made with a Set request and a ticket number. The ticket number is supplied in the Response Header. The Response Header is part of the CGI standard and used to provide a channel of communication between the server and the client. If a request for a non-existent item is made, the error is handled appropriately. The process of integration promoted to re-evaluate the set of functionalities and its redefinition: • The tool must be able to author, save and load assessment items as content pakages. This involved the

establishment of a new requirement: the authoring and loading of metadata associated with a content package. For this purpose JAXB technology is again considered as the most appropiated regarding with the authoring, however the Cambridge API used for creating manifest files is also considered. (ac, modified or extended. depending on results in next week)

• Since end users shouldn’t be forced to work with metadata and content packages, the tool allow the to import and export in the local machine assessment items as xml files with the resources associated.


User Studies - Collaborative Design Sessions

Collaborative Design Sessions took place in our usability lab. They involved an example user unrelated to the project, but from Kingston University. The questions created were either of the project’s chosing (we selected it from past Electronics Exam questions) or hypothetical questions devised on the spot by the participant (not real questions, but sufficient to try out the tool). Each session was in two parts: in part (i) participants input into the tool existing paper questions; in part (ii) participants were asked to create a new question of a type they were unfamiliar with. The session moderator sat next to the participant and prompted the participant to talk aloud as they worked. At critical moments during the task, for example, the participant made a mistake, or an interesting comment, the moderator interrupted to task to pursue the point, for example, to clarify the participant’s thought and to carry forward the thought process to redesign suggestions. If the participant ‘got stuck’, the moderator asked the participant to keep trying, but if the participant was still stuck, the moderator told the participant which user interface element to use.

Study 1 - August 07 Using AQuRate v1 The participant was a psychology lecturer. He has written various multiple-choice tests for psychology modules, some of which were based on question banks provided by course textbooks, using Word or the test editor on the CD.

Part (i) Input Electronics Exam questions The task was to enter 3 multiple choice, single correct response questions on Electronics. Figures for the questions were provided as .GIF files on the desktop. A choice type question was input correctly. There were inefficiencies with regard to input in tables, and the participant actually created a multiple correct response, multiple choice type question, rather specifying a single correct response muti-choice question. Special characters in the original question were replaced with text in AQuRate e.g. “ Ω “ on paper was input as “ohms”. Question 2 was partially created - the prompt and stem were input, but the participant did not add the image, because he was unable to find to command button. He did not preview either question, again, because he was unable to find the command button

Part(ii) Create Order and Association question about psychology The participant found the structure and labelling of the response tables for association and order questions unintuitive, and was needed assistance from the moderator to complete the task.

Study 2 - August 08 using AQuRate v2 The participant was an office administrator. She has never written and examination before, but does enter student, timetabling and financial information onto various university systems.

Part (i) Input Electronics Exam questions Like study 1, the task was to enter the same 3 multiple choice, single correct response questions on Electronics. Figures for the questions were provided as .GIF files on the desktop. All three questions were input and previewed successfully, although there were some presentational discrepancies between the original question and the question input on AQuRate. Some inefficiencies in interaction were detected. Note the improvement compared to AQuRate v1.

Part(ii) Create Hotspot question for open days At first, the participant created a multiple choice question by mistake, because she did not appreciate how it


was different to a hotspot question. The manner of creating hotspots was also unintuitive, but the participant did figure it out by trial and error.

User Studies - Field Studies The field studies involved real end-users (not related to Kingston university), who were interested in using AQuRate for real, future e-assessments. Consequently, field study sessions took place off-site and the task was to create real questions of the participants chosing with AQuRate.

Collaboration

From day one close collaboration took place among the projects funded under this call. During both Phases weekly management meetings took place using Skype as a means of conference calling. Here development issues were discussed and inter-project issues managed. Additionally joint face-to-face project meetings were used to iron out development work. These were in addition to Advisory Group and JISC meetings. Skype proved very helpful by alerting managers to when others were available to talk, and offering a means of taking meeting motes on the fly. Technical collaboration initially was better on a face-to-face basis. There was a greater need for this during Phase 2, and Google Groups was used initially facilitate this. Later these discussions began to be moved over to Source Forge.

Peer Review

The peer review process was undertaken between the 21st January and 5th February. The project was reviewed by developers from the ‘Asdel’ project. In addition, the Aqurate staff reviewed the Minibix project. The peer review was useful in identifying areas of the source code that could benefit from additional documentation and where the non-identical licenses required acknowledging and packaging appropriately.

7. Outputs and Results

Project Website

The project website was created in order to: provide general description and documentation of this project, allow the contact with both managers and developers, give instructions to make easier further collaborations and receive feedback. This website permits the access to a general QTI projects repository where all technical documentation, libraries and source code are available. Links to documentation, project contact details, the application and source are also available.

More information can be found in http://aqurate.kingston.ac.uk

Desktop Application

An application is available for download from http://code.google.com/p/qtidevelopment/downloads/list

http://aqurate.kingston.ac.uk/



Figure 4: AQuRate Desktop Interface

General Features of the Application

1. Loading/saving of 8 core question types with exception of multi-interaction “Inline Choice” and “Text Entry” question types

2. Loading/saving questions in IMS content package format, exporting to QTIv2 XML format 3. Downloading/uploading content package from “Minibix” item bank 4. Full HTML editor for “Preamble”, “Postscript” and “Prompt” (if available)

i. Undo/Redo ii. Copy/Cut/Paste

iii. Bold/Italic/Code styles iv. Paragraph alignments v. Insert symbol (Greek) in html symbol code

vi. Attach image from local file system 5. List of currently opened questions, each question accessible by clicking. Each question can be saved

or closed via right click menu 6. Local preview of questions, the tool launches rendered questions in default browser. User will be able

to test the question interactively as well with full responses options 7. Metadata creation/loading/modification for each question type (author name, date created, language)

QTI Question Type Specific Features

1) Choice a) Creation of single or multiple correct answer questions b) Correct choices have automatic score of 1 and incorrect choices have automatic score of 0, all score

can be changed by the user, negative score can also be given c) Presentation of the choices can be fixed to their position in the authoring environment , or randomized

2) Order a) Ordering choices are defined by the user, an ordering list is generated, user then can rearrange the list

to the correct order. Presentation position of each ordering choices can be fixed or randomized 3) Associate


a) All associate items need to be defined by the user; the presentation of the items in the end delivery engine can be fixed or randomly generated. User can add a pair and choose the 2 items from the list of all items for pairing. User then can define this pair to be correct or incorrect, and a partial scoring can be given to the answer pair

4) Inline Choice a) Single interaction model, which means 1 single choice field per question b) Users input the text before the choice field, and the text after, then they put in the number of choices

for the choice field c) Correct choice is chosen by the user (Note: there can be only 1 correct choice for this type of

question), partial scoring can be defined for each of the choices 5) Text Entry

a) Single interaction model, which means 1 single text entry field per question b) User add text before and after the text entry field c) Correct text is inputted into the text entry field

6) Hotspot a) User first upload a main image, the image is then automatically resized to fit the viewing window

(Note: this does not change the original image or the presentation of the image in the delivery engine, only scale the image in the authoring tool for usability purpose)

b) Move the hotspot cursor around the image, define hotspot image choices (Note: the position defined on the scaled image will be translated to position in the original image, defined hotspots do not have a visual representation on the original image, therefore it is the author’s responsibility to draw the visual indicators on the original image)

c) There can only be 1 correct hotspot choice, and scores can be given to each hotspot choices 7) Graphical Ordering

a) Main image is defined in the same manor as “Hotpot” question, image scaling takes place automatically

b) All choices are added automatically to an ordering list, author moves each choice into their correct position

8) Slider a) There are 3 field values to set for the slider, “Lower Bound”, “Upper Bound” and “Step”.

Upper/Lower bound defines the range of values the slider can take, and step defines the value between each slider movements.

b) To pick the correct answer for the question, the author moves the slider to the desired value. This value is indicated in a “Correct Value” field

Prototype Web Application


Figure 5: AQuRate Web Interface

This is in development and a prototype will be available for download from http://code.google.com/p/qtidevelopment/downloads/list

Source Code

Eclipse projects are available of: 1) The QTI_API 2) AQuRate Source

These are also available from http://code.google.com/p/qtidevelopment/downloads/list

User Documentation

Documents which describe and introduce this tool distinguish between different categories of users. Academic organizations and students who are considered the end users and possess different grades of knowledge about QTI 2.1 and IMS CP specifications. For them, there is available a User Guide where instructions about the tool’s usage and information about different question types can be found. Given the condition of AQURATE as open source software, the other branch of users are developers and other institutions working on similar or related tools. Technical documentation about the libraries applied and




the API built is available through AQURATE website. Licensing information is also provided. To expert QTI and IMS CP academic users, there is also available a document which describes AQURATE profile. There are defined and explained its behaviour as authoring tool as well as supported and non-supported features. This document is also useful for developers due to it describe at the same time the AQURATE API profile.

Documents are available for download from http://code.google.com/p/qtidevelopment/downloads/list

Conference Papers

A paper was submitted to CAA 2008

Download Analytical Data

Data was kept through using Google Analytics for the Google Code site used for the project. A summary of this is provided in the Appendices.

Presentations

Presentations were made at three Assessment SIGs during the project.

8. Outcomes

Aims and Objectives Achieved Create new QTI 2.1 items Edit existing QTI 2.1 items Import and Export QTI 2.1 items (from / to other pre-existing systems)

To and from Minibix

Package QTI 2.1 Items as IMS Content Packages that are SCORM 2004 compliant

Use external web services for testing the rendering and response of questions using one or more of Assessment Delivery, Assessment Rendering and QTI validation systems

Uploaded to Minibix, and AsDel interfaced with Minibix

Use external web services for depositing and retrieving questions from Item Banks

To and from Minibix

Dual layer architecture separating the presentation and core logic components of the system (for extensibility)

Offline creation of assessments Items not assessments created.

Offline rendering of assessments Items not assessments rendered.

Production of new system interfaces e.g. Prototype web interface



plug-ins, web services, server pages created. Extension and Development of new System Capabilities e.g. extra question types, use of items with sequenced content, support for custom interactions.

Eclipse projects available.

9. Conclusions

The AQuRate project has successfully delivered an open-source, multi-platform Desktop authoring tool. This tool loads and saves QTI2.1 content as content-packages, with a choice of using a local file-system or the Minibix repository as the storage device. The tool includes the facility to preview and test each authored question in a browser, without requiring any further programs or servers to be installed or running.

The AQuRate project successfully investigated and delivered several variations on this desktop authoring tool. It has been re-cast as a web-tool (an applet) and as an Eclipse plug-in. A core API of software components facilitates this flexibility.

The tool will input (and output) a subset only of the total QTI2.1 specification. In particular, only the eight 'core' interaction types are supported, and the tool currently supports exactly one 'interaction' per assessment item. The reason for the limitation is requirement to make the best use of the available project resources. The complexity and extensive scope of the QTI2.1 specification creates challenges for both developers and users. One challenge is to design the user interface to allow easy adoption by users, and yet also provide control over the advanced features provided by the standard. Another challenge is to design the data model and user interface to remain appropriate no matter what combination of elements are (validly) included in an assessment item.

The relationship between question and test authoring needs careful consideration. The JISC programme funding the three related projects covered an item authoring tool, item bank and assessment delivery engine, but not a test/assessment authoring tool. In a Higher Education context, tests are often authored alongside items, and the rationale for separating these tasks needs more to be re-examined.

Finally, a useful and unexpected outcome of the project was the development of the 'standalone use case', in which the 'preview' function of the Desktop authoring tool is extended to include viewing and processing of sequences of items (i.e. tests). This facility may be useful in some assessment scenarios.

The implications of these project results are discussed in the following Section.

10. Implications

The main implication of the completed project is that there is now a freely available desktop authoring tool for QTI2.1 Assessment Items. This has led to some 1,500 downloads of the software, and several expressions of interest for continuation of work. Notwithstanding this (and other) successful project outcomes, there are a small number of open issues that are worthy of further consideration. One issue is the degree of unification and coherence of the data models used, both in AQuRate and also in related projects. The software uses one data model (generated by JAXB) for the authoring functions and


another data model (derived from JASSESS work) for the rendering and processing functions. In addition, other data models are available (such as R2Q2 project results) and are used elsewhere (e.g. the Asdel project). While it can be argued that multiple data models and source trees are useful in some respects, one implication of this fragmention are that subsequent development work will not benefit all fragments equally and that this may result in a slower than otherwise uptake of the standard. Another issue is the appropriate degree of flexibility granted to users to edit and create assessment items. The choice has implications for the usability of the software (for novices and for advanced users) and also for promoting good practise by supporting a standard approach for the representation of assessment items. One example is nested assessment items; another example is the use of style sheets in the html to enable flexible and consistent rendering of the content.

11. References

• Alsop et al., Data Modelling, GUI Design and Use Case Issues in Developing AQuRate, CAA Conference Paper (subject to acceptance) 2008

• AQuRate, Website and associated documents, http://aqurate.kingston.ac.uk, Last Accessed 28 March 2008

• IMS Question and Test Interoperability Version 2.1 http://www.imsglobal.org/question/, Last Accessed 17 March 2008

• IMS Content Package Version 1.1.4 http://www.imsglobal.org/content/packaging/, Last Accessed 10 March 2008

• IMS Learning Resource Meta-Data XML Binding, Version 1.2.1 http://www.imsglobal.org/metadata/imsmdv1p2p1/imsmd_bindv1p2p1.html, Last Accessed 5 March 2008

• IEEE 1484.12.1-2002 Standard for Learning Object Meta-data (LOM) http://www.imsglobal.org/metadata/mdv1p3pd/imsmd_bestv1p3pd.html, Last Accessed 1o March 2008

• JASSESS, Hosted at CARET, http://qtitools.caret.cam.ac.uk/index.php?option=com_content&task=view&id=19&Itemid=2, Last accessed 26 February 2008

• QTITools , CARET, University of Cambridge, http://qtitools.caret.cam.ac.uk/, Last Accessed 19 March 2008


http://www.imsglobal.org/question/

http://www.imsglobal.org/content/packaging/

http://www.imsglobal.org/metadata/imsmdv1p2p1/imsmd_bindv1p2p1.html

http://www.imsglobal.org/metadata/mdv1p3pd/imsmd_bestv1p3pd.html

http://qtitools.caret.cam.ac.uk/index.php?option=com_content&task=view&id=19&Itemid=2

http://qtitools.caret.cam.ac.uk/


12. Appendices


Multi-interaction Report

Report on implementing multiple interactions for Aqurate – looking to the future Author: James Annesley Date: 13/03/2008

Project: Aqurate – http://aqurate.kingston.ac.uk

Institution: Kingston University

Introduction

Aqurate is a multiple choice question authoring tool which implements questions conforming to the IMS QTI (Question & Test Interoperability) v2.1 specification. By the end of the project the Aqurate tool was able to author eight of the IMS QTI interaction types (Interaction). Each question (AssessmentItem) could contain a single interaction. This only provides a partial implementation of the QTI v2.1 specification, which can allow, amongst others, multiple interactions, of different types, in a single question. Effort was expended on enabling the tool to provide multiple interaction functionality, but which wasn’t included in the release. This report attempts to provide the developer with useful information in extending the Aqurate tool, if desired in the future.

Design

The Aqurate tool uses an API (Application Programmers Interface), called AqurateAPI, and designed for use with an application by performing the backend QTI XML processing tasks. The Aqurate application uses a MVC (Model View Controller) which provides a scalable solution to complexity management.

AqurateAPI perspective

The scope of the QTI v2.1 was restricted to allow the specification to be understood, to manage complexity and for development to progress satisfactorily. As discussed in the introduction, the question format was constrained to a single interaction per question. The API was in fact extended to support multiple interactions, but the Aqurate user-interface required an extensive overhaul in order to provide this functionality. These changes to the API were put to one side because the application did not work with the extended API and the user-interface was continued to be developed for single interactions only. QTIv2 question interactions are housed in an ‘itembody’ element similar to a ‘body’ element in HTML. The contents of the Itembody are presented in a user-interface and the QTIv2 specification supports HTML 4.0, so in theory a standard web page can be displayed which contains the interactions. At first the Itembody was restricted to contain an interaction without any other text or formatting. This design was extended to allow a paragraph, <P>, to be added before and after the interaction, known as a ‘preamble’ and a ‘postscript’. This approach worked fine for six of the eight interactions, but failed for two of them. The ‘InlineChoiceInteraction’ and ‘TextEntryInteraction’ were the interactions to cause the problems. These interactions are designed to be included within a question’s block of text. Their use is often characterised by there being multiple interactions within the text. Additionally, there are designed to not be implemented at the top-level in the Itembody, as the other interactions can, reinforcing their objective as inline interactions



within blocks of text. Aqurate user-interface design also restricted the format of the question’s content. In order to load a question, the question must have the Aqurate formatting characteristics, e.g. a single interaction contained either solely within the Itembody or sandwiched between two paragraphs, which is a severe restrictive. In order for the API to be able to accept these ‘InlineChoiceInteraction’ and ‘TextEntryInteraction’ interactions three approaches are possible. First is to implement them as single interactions and supply the extra formatting tags required by Aqurate, e.g. <P> tags sandwiching the interactions. Second is to add special conditions for these types. Third is to make the entire loading and modelling process more flexible, i.e. to support multiple interactions in a single question. The first approach was feasible and worked within the existing framework and this was the method adopted for the release. The second option is sub-optimal. The third approach requires work to redesign both the API and the application, but ultimately provides a tool with the flexibility to present the majority of QTI Itembody meta-data. The third approach is described below.

Loading

The process of loading involves creating API instances and transferring these to the application’s MVC. The Documents (Model) each contain an AqurateAPI question instance and have a corresponding user-interface (View) to display the contents, e.g. the View’s HTML components display the contents of the Itembody in a HTML panel. Since this is a standard Java component (JEditorPane), HTML 3.1 is supported but not the QTIv2 interactions. One solution is to present the interactions as images within an HTML block, with the interaction’s details given in the images’ attributes. In order to edit this interaction, a corresponding button is displayed. This layout is shown in Figure 1.


Figure 1: The main screen. The steps to be taken are:

1) Extract the Itembody element as a string using JAXB, see Snippet 1. 2) Maintain a separate ItembodyHTML within the AqurateAPI containing the HTML. 3) Extract interactions and create the AqurateAPI interactions, e.g. aqurateChoiceInteraction, etc. For each

interaction, create the appropriate API instance, replacing all interactions with images (<img>) and store in the ItembodyHTML.

4) A new AssessmentItem MVC is required in the application. 5) The AssessmentItem Document creates the appropriate interaction Models for the application. The Itembody

HTML block is parsed and the interactions are created via the ID attribute of the image element. The interaction name and id are joined together. The name is the interaction type and the id is the interaction within the list of interactions maintained by the AqurateAPI.

6) Render the HTML tags within the user‐interface HTML panel (JEditorPane). This HTML panel now exists in the AssessmentItem user‐interface.

The existing code requires modifying to remove the “before” and “after” HTML blocks, i.e. “preamble” and


“postscript”. StringWriter sw = null; String sXML = “”; Iterator<Object> it = null; Object o = null; try aai.getMarshaller().setProperty(Marshaller.JAXB_FRAGMENT, true); catch (PropertyException prope) prope.printStackTrace(); sw = new StringWriter(); it = prompt.getContent().iterator(); while (it.hasNext()) o = it.next(); if (o.getClass().getSimpleName().equals(“String”)) sw.append((String) o); else try aai.getMarshaller().marshal(o, sw); catch (JAXBException jaxbe) jaxbe.printStackTrace(); try aai.getMarshaller().setProperty(Marshaller.JAXB_FRAGMENT, false); catch (PropertyException prope) prope.printStackTrace(); sXML = sw.toString(); try sw.close(); catch (IOException e) e.printStackTrace(); return sXML;

Snippet 1: Extracting a portion of XML from a JAXB object

Creating

When a new question is created, the application creates an AssessmentItem Model and the user interface for that question. The interactions are created when the buttons are clicked. The corresponding AqurateAPI components of AssessmentItem and interactions Models are created. The suggested method for creating interactions is suggested as parsing the HTML panel and creating the interactions by parsing each image. To parse the Itembody HTML the following code snippet, Snippet 2, illustrates the technique. The HTMLEditorKit parser is used. As the individual tags are parsed, each is checked to see whether it of an interaction type. If it is then the appropriate type is created. This method is suggested, as opposed to using the button click event, for example, because the MVC architecture means the HTML is continually parsed in order to create the correct QTI XML. This means the interactions is better modelled via a parsing process, which can be easily automated, rather than a one off event. if (tag == HTML.Tag.IMG) // create an image item Img img = objectFactory.createImg(); img.setAlt(sAlt); img.setHeight(sHeight); img.setId(sID); img.setLang(sLang); img.setSrc(sSRC); img.setWidth(sWidth); if (sID.startsWith(“InlineChoice”) || sID.startsWith(“Choice”) ||


sID.startsWith(“Order”) || sID.startsWith(“Associate”) || sID.startsWith(“Hotspot”)) if (aib == null) System.err.println(“AqurateItemBody is null - Aqurate API error”); else AqurateInteraction i =null; try i = aib.getAqurateInteraction(sID); if (!sIMS.isEmpty()) Object adder = sIMS.peek(); Object addee = i.getInteraction(); addcorrect(adder, addee); catch (AqurateException ae) if (i==null) else ae.printStackTrace();

Snippet 2: Parsing the HTML to create Interactions. The AqurateAPI maintains a list of interactions. The parser extracts the interaction’s name and ID from the image ID and matches it appropriately.

Application User Interface

The user interface requires modifying so that a single question could contain many interactions. The idea is that the interface remains WYSIWYG (what you see is what you get). The main screen has a list of question on the left with a single HTML pane in its centre, see Figure 1. The left hand side of the screen is currently used to present the different questions in the same way, but at present the perspective is at Interaction level. An extra user-interface MVC is required for the AssessmentItem, as stated above. The HTML window has a tool bar above it which has one button for each interaction type. When the button is clicked, an interaction is created and an image appears in the HTML pane and a corresponding button below it. The image is the interaction’s placeholder within the text. The button allows the interaction to be edited. Clicking the button brings up a dialog, which has a similar look as currently offered by the program for each interaction, shown in Figure 2.


Figure 2: A Choice Interaction is being edited. The question can be edited and the screen reverts back to the main screen when editing is complete. Text and images can be added before or after the placeholder. The user can also cut and paste the image placeholders. When the image is deleted a DocumentListener is used to also remove the document. An example is given in the code snippet below, Snippet 3. public void insertUpdate(DocumentEvent arg0) public void removeUpdate(DocumentEvent arg0) // find which one has gone ElementChange elemChg = null; Element[] removedElements = null; String sID = “”; HTMLDocument htmldoc; Element root; ElementIterator eli; Element thisElement; if (!bIgnoreRemoveUpdate) htmldoc = (HTMLDocument) arg0.getDocument(); root = htmldoc.getDefaultRootElement(); eli = new ElementIterator(root);


thisElement = eli.first(); while (thisElement != null) elemChg = arg0.getChange(thisElement); if (elemChg != null) removedElements = elemChg.getChildrenRemoved(); for (int i = 0; i < removedElements.length; i++) if (removedElements[i].getName().equals(“img”)) sID = (String) removedElements[i] .getAttributes().getAttribute( HTML.Attribute.ID); ici.changeDelInlineChoiceInteraction(sID); thisElement = eli.next(); void setIgnoreRemoveUpdate(boolean b) this.bIgnoreRemoveUpdate = b;

Snippet 3: Reacting to editing changes and removing the corresponding models and AquateAPI interactions.

When adding an interaction the image is created and the appropriate entry added to the HTML. This code is interpreted by the parser, which when it finds an ‘img’ with the id of an Interaction, it creates the necessary document. This is shown in the following code snippet, Snippet 4. void insertPlaceholder(String sText) String sHTMLimg = “”; String sWorkingDir = “images/temp”; Image imagePlaceholder = null; BufferedImage bufferedImagePlaceholder = null; Graphics2D graphics2D = null; File filePlaceholder = null; URL imgURLPlaceholder = null; BufferedOutputStream imageoutPlaceholder = null; JPEGImageEncoder encoder = null; JPEGEncodeParam encoderParam = null; FileOutputStream fos = null; int w = 150; int h = 15; int pix[] = new int[w * h]; int index = 0; for (int y = 0; y < h; y++) int red = (y * 255) / (h - 1); for (int x = 0; x < w; x++) int blue = (x * 255) / (w - 1); pix[index++] = (255 << 24) | (red << 16) | blue; imagePlaceholder = createImage(new MemoryImageSource(w, h, pix, 0, w)); bufferedImagePlaceholder = new BufferedImage(imagePlaceholder .getWidth(this), imagePlaceholder.getHeight(this), BufferedImage.TYPE_INT_RGB); graphics2D = bufferedImagePlaceholder.createGraphics(); graphics2D.setRenderingHint(RenderingHints.KEY_INTERPOLATION, RenderingHints.VALUE_INTERPOLATION_BILINEAR); graphics2D.drawImage(imagePlaceholder, null, null); filePlaceholder = new File(sWorkingDir + File.separator + sText + “.jpg”); try imgURLPlaceholder = filePlaceholder.toURL(); catch (MalformedURLException e) e.printStackTrace(); graphics2D.drawString(sText, 5, imagePlaceholder.getHeight(this) - 5); try fos = new FileOutputStream(filePlaceholder); catch (FileNotFoundException e) e.printStackTrace();


imageoutPlaceholder = new BufferedOutputStream(fos, 4092); encoder = JPEGCodec.createJPEGEncoder(imageoutPlaceholder); encoderParam = encoder.getDefaultJPEGEncodeParam(bufferedImagePlaceholder); encoderParam.setQuality(1, false); encoder.setJPEGEncodeParam(encoderParam); try encoder.encode(bufferedImagePlaceholder); imageoutPlaceholder.close(); fos.close(); catch (IOException e) e.printStackTrace(); sHTMLimg = “<br></br><img src=\”” + imgURLPlaceholder + “\” alt=\”” + sText + “\” id=\”” + sText + “\” class=\”attach\”></img>”; try /* * HTMLDocument htmldoc = (HTMLDocument) * jEditorPaneInlineChoiceInteraction.getDocument(); Element * elementHTML = htmldoc.getRootElements()[0]; Element elementBODY = * elementHTML.getElement(0); int ij = * elementBODY.getElementCount(); Element elementContent = * elementBODY.getElement(ij-1); * * htmldoc.insertBeforeEnd(elementContent, sHTMLimg); */ htmlEditorKitInline.insertHTML(htmlDocumentInline, jEditorPaneInlineChoiceInteraction.getCaretPosition(), sHTMLimg, 0, 0, HTML.Tag.IMG); catch (BadLocationException be) be.printStackTrace(); catch (IOException ie) ie.printStackTrace();

Snippet 4: Creating the image placeholders.

Concluding remarks on API and User Interface

The steps drafted above give some information about a possible solution to giving Aqurate a multiple interaction interface and the API multiple interaction capabilities. There will be addition steps to be taken, e.g. handling response declarations for each interaction, handling adaptive questions. Having implemented multiple interactions, it is suggested that further improvements will be possible, e.g. giving the interaction choices more scope for editing and not just a plain text field. The fact that the JEditorPane only supports HTML 3.1 is also a problem, since QTIv2p1 supports HTML 4.0. There are other problems with the JEditorPane, such as poor parsing tools. These are apparently being addressed in Java 7.0. If Java 7.0 becomes supported by Apple and Linux then the project could be moved to support it. Otherwise, there may be alternative HTML WYSIWYG controls available, with improved editing ability, HTML support and parsing functionality. It may also be possible to extend the JEditorPane or JTextArea so buttons could be used as placeholders. This requires some advanced programming, but the extra interaction buttons can be removed from the user-interface.


Google Analytics Data for Google Code Site

See included pdf file.