online quiz tool review for statistics 10 - ucla · online quiz tool review for statistics 10 ......

Online Quiz Tool Review For Statistics 10

Created by the Teaching Enhancement Center

In the UCLA Office of Instructional Development As part of the Blended Instruction Case Studies (BICS)

TEC Review of Online Quiz/Assessment Tools for BICS

PURPOSE OF REVIEW:............................................................................................................................ 3 REQUIREMENTS:....................................................................................................................................... 3 TOOL: ARTIST ...................................................................................................................................... 4 TOOL: MOODLE 1.2.1 ......................................................................................................................... 6 TOOL: ELEDGE VERSION 3.1.0........................................................................................................ 9 TOOL: CLAROLINE............................................................................................................................ 11 TOOL: QUIAWEB................................................................................................................................ 13 TOOL: RESPONDUS.......................................................................................................................... 15 TOOL: PERCEPTION FOR WEB/ PERCEPTION FOR WINDOWS .......................................... 17 TOOL: HOTPOTATOES V.6 ............................................................................................................. 19 TOOL: BLACKBOARD (VARIOUS VERSIONS) ........................................................................... 21 TOOL: TESTPILOT (CLASSIC - 3.2.5 ENTERPRISE - 4.4.2)..................................................... 23 TOOL: ATUTOR V. 1.3.3 ................................................................................................................... 25 TOOL: ICREATE POWERPOINT PLUG-IN.................................................................................... 27 APPENDIX: REJECTED TOOLS ..................................................................................................... 29 APPENDIX: TOOL REVIEW FORM................................................................................................. 30

. 2

PURPOSE OF REVIEW: The purpose of this document is to describe and compare the functionality, usability, and technical requirements for a selection of software tools that provide online quiz capabilities. Software that were evaluated range from standalone quiz software to entire course management systems. This is not meant to be an exhaustive review of all available tools, but rather a review of likely candidates for use on a particular project with a specific set of requirements. REQUIREMENTS: The quiz tool review was based upon the following requirements presented by the Statistics department:

Quiz Tool Features - Requirements from Statistics team

Should be Linux / UNIX / OS X compatible to fit with existing department technology.

Prefer to use “Off the shelf” product and not reinvent the wheel

Must be scaleable to ~1700 students (11 sections of ~160 students; each divided into 4 discussion sections of ~40 students)

Eight quizzes will be administered during the quarter with 10 questions per quiz.

Questions can be randomly selected based on content.

Level of difficulty and the approach of the questions vary and include conceptual understanding, calculations, applications, and interpretation of results and computer printouts.

Provide instant feedback to students about performance and records scores.

Provide summary of class performance to instructor.

Should evaluate written responses sufficiently well to let the instructor know whether student has responded correctly.

Enable instructors to assign students to groups based on exam scoring (top, middle, bottom) and groups will then re-do quiz together.

Enable instructors to create mid-term and final exams aligned with course objectives.

Should allow for anonymous surveys.

If a multiple-choice question, partial credit may be given.

Questions can include graphics.

Students can download data sets, manipulate data and return answer.

Provide students with quick feedback and challenging and realistic problems that help them know whether or not they understand the material.

Provide students with practice on more routine ‘skill-based’ problems.

Provide faculty with quick assessment so that they know what to emphasize in class.

Has interactive homework tool with guided feedback for right / wrong answers.

. 3

Tool: ARTIST Last Updated: February 2004 Organization: University of Minnesota Web address: http://www.gen.umn.edu/artist/index.html Cost: Free use; not open source – resides on UM servers Authoring requirements: web browser to select items; word processing for .rtf output Server requirements: none – test bank hosted by UM End user (student) requirements: none Question types:

multiple-choice (number) multiple-answer true/false fill-in-the-blank/Cloze ordering/ranking matching short answer/essay drag-n-drop numerical

Graphics: Equations: SCORM compliant: Administration & Tracking:

student name student ID batch user upload time stamp IP logging multiple attempts feedback branching keyword analysis track questions used randomization gradebook email results to prof sort by difficulty sort by objective track individual answers export gradebook statistics on tests track access groups student file upload

Import formats: none; test-bank is closed. Export formats: .rtf paper-based text Comments: (interface, usability, look-and-feel, etc)

• ARTIST is a closed test bank of statistics assessment items (currently 1075) organized by topic (objective) and learning outcome (literacy, reasoning, thinking).

• Output is a paper-based file of all items selected using web-based form. • Permission is given to use ARTIST test items for non-commercial use with proper

attribution. Questions would need to be entered by hand into another format or database for use within an online testing or course management system.

. 4

http://www.gen.umn.edu/artist/index.html

ARTIST Screenshots Sample Assessment Manager item view:

. 5

Tool: Moodle 1.2.1 Last Updated: March 2004 Organization: The Moodle Trust Web address: http://moodle.orgCost: Free/GNU Authoring requirements: web browser (detailed specifications not listed) Server requirements: Apache/PHP/MySQL (or other databases); see: http://moodle.org/doc/?file=install.htmlEnd user (student) requirements: web browser (detailed specifications not listed) Question types:

multiple-choice (2 - 10) multiple-answer (2 - 10) true/false fill-in-the-blank/Cloze ordering/ranking matching short answer/essay drag-n-drop numerical

Graphics: Equations: SCORM compliant: (in-progress) Administration & Tracking:


Import formats: Several, including WebCT, Blackboard, custom (some coding required) Export formats: export stats in .txt or .xls format; no question export. Comments: (interface, usability, look-and-feel, etc)

• Modular design of this open source CMS allows functionality to be customized. • Clean, usable interface. • Can authenticate against LDAP and other systems

(see http://moodle.org/doc/?file=features.html) • User levels include: administrator, course creator, teacher, and student. • Branching questions enabled for lessons but not quizzes. • Questions form ‘test bank’ for quizzes but cannot be added as questions in lessons. • Has peer review functionality. • Very extensive log files, sortable by user, date, and activity; includes IP • Scripts to enable communication with WebWork under development. • Three course formats: weekly, topics and social. Custom format may be developed.

. 6

http://moodle.org/

http://moodle.org/doc/?file=install.html

Moodle Screenshots Sample Quiz Student View:

Sample Quiz Feedback:

. 7

Sample Course Interface:

. 8

Tool: Eledge version 3.1.0 Organization: University of Utah Last Updated: October 2003 Web address: http://eledge.sourceforge.net/Cost: Free/GNU Authoring requirements: web browser (detailed specifications not listed) Server requirements: Linux/Apache/Tomcat/Java 2/MySQL/JDBC driver End user (student) requirements: web browser (detailed specifications not listed) Question types:

multiple-choice (5) multiple-answer (5) true/false fill-in-the-blank/Cloze ordering/ranking matching short answer/essay drag-n-drop numerical

Graphics: (workaround > URL) Equations: SCORM compliant: Administration & Tracking:


Import formats: none Export formats: none Comments: (interface, usability, look-and-feel, etc)

• Open source course management system including content, quiz, calendar, etc. • CMS modules can be enabled/disabled & interface style customized. • Allows for TA access, peer reviews, journal. • Feedback limited to showing which questions answered incorrectly (no correct answer or

comment shown). • Student self-enrollment poses big security issue; need to directly access MySQL tables to

batch create users. No other authentication but uses SSL to encrypt logins. • No test bank functionality or sharing of items between quizzes. • User statuses: Current, Frozen, Visitor, TA, Instructor • Very basic interface

. 9

http://eledge.sourceforge.net/

Eledge Screenshots Sample Student Quiz View:

View of gradebook section (Instructors):

. 10

Tool: Claroline Last Updated: 1.4.2 stable – June 2003; 1.5 beta – April 2004 Organization: University of Louvain (Belgium) Web address: http://www.claroline.netCost: Free/GPL Authoring requirements: web browser (detailed specifications not listed) Server requirements: Apache (or other), PHP 4+, MySQL 3.23, optional Mail Transport agent End user (student) requirements: web browser (detailed specifications not listed) Question types:

multiple-choice (10+) multiple-answer true/false fill-in-the-blank/Cloze ordering/ranking matching short answer/essay drag-n-drop numerical

Graphics: Equations: SCORM compliant: in 1.5 beta Administration & Tracking:


Import formats: Export formats: Comments: (interface, usability, look-and-feel, etc)

• Database admin must do batch uploads of users • Admin rights are all-or-none; i.e. tutors, managers, teachers may all have same rights. • Very clean interface plus help guides for students & teachers. • Very straightforward file management. Create directories, move, visible/invisible • “Exercises” not quizzes –feedback to students and completion/score to teacher but

limited administrative control (no timing, quiz password, multiple attempts, answer tracking, ip tracking, etc)

• Stats show results, downloads, page views, by user or content. • Modules can be activated/deactivated. Modules visible from front page. • Can be configured to authenticate via LDAP server • Question pool allows questions to be reused among exercises.

. 11

http://www.claroline.net/

Claroline Screenshots Sample student exercise screen:

Sample course login screen:

. 12

Tool: QuiaWeb Organization: Quia Web address: http://www.quia.comCost: $49/year per instructor, students always free. Authoring requirements: web browser (NS 4+, IE5+, not AOL) Server requirements: none – hosted solution End user (student) requirements: web browser (NS 4+, IE5+, not AOL) Question types:

multiple-choice (1-5) multiple-answer true/false fill-in-the-blank/Cloze ordering/ranking matching short answer/essay drag-n-drop numerical

Graphics: Equations: LATEX SCORM compliant: Administration & Tracking:


Import formats: word lists can be imported Export formats: student and quiz summaries - .csv format Comments: (interface, usability, look-and-feel, etc)

• Easy to use, clean interface • No real ‘content’ sections – just surveys, quizzes, and activities geared toward younger

audiences (jumbles, crossword puzzles, etc). • Students must self-enroll (major security issue)

. 13

http://www.quia.com/

Quia Screenshots Sample quiz screen from student view:

Sample stats from instructor view:

. 14

Tool: Respondus Organization: Respondus, Inc. Web address: http://www.respondus.comCost: $79 academic/ $139 commercial; free print only edition; site license $2500+. Authoring requirements: Software installed on author workstation; Windows 95*, 98, NT4*, ME, 2000, XP. Apple users can use Connectix Virtual PC to run program. Server requirements: none for authoring. CMS for exporting questions to online format. End user (student) requirements: depends on delivery system Question types:


Graphics: Equations: EquationEditor or MathType SCORM compliant: Administration & Tracking:


Import formats: Import .txt, .rtf and Respondus .rsp files Export formats:

• Publish online via: Blackboard 5 and higher (Basic and Enterprise); WebCT 2.1 and higher; WebCT Vista 2.0 and higher**; eCollege AU+, IMS QTI 1.x;

• Archive to a Respondus .rsa file. • Printable version saved as .doc, .rtf, .txt file

Comments: (interface, usability, look-and-feel, etc)

• This is a exam creation tool. • Though not a question pool per se, items can be imported from one exam into another. • NO test administration through this tool • Tests can be output to paper format or upload to various CMS’s. (see above). • Some publisher test banks available • Test item creation is offline • Interface is easy to use and intuitive with the exception of no way to sort/organize

questions except for by name of item. • Feedback is for right/wrong only (not specific to which wrong answer selected). • May be able to retrieve stats from CMS • Much of the tracking/administration/security issues can be handled through whatever

CMS is used to deliver the Respondus-created exams.

. 15

http://www.respondus.com/

Respondus Screenshots Question authoring interface:

. 16

Tool: Perception for Web/ Perception for Windows Organization: Questionmark Web address: http://www.questionmark.comCost: quotations must be requested – very expensive Authoring requirements: Windows 98/NT/2000/ME/XP – web and PC versions available Server requirements: hosting available; Perception server - End user (student) requirements: Internet Explorer, AOL and Netscape; or Questionmark Secure environment Question types:

multiple-choice (no limit) multiple-answer(no limit) true/false fill-in-the-blank/Cloze ordering/ranking matching short answer/essay drag-n-drop numerical



Import formats: Import ASCII text, QML, or QTI XML questions; Export formats: Reports output to Excel, ASCII text files; Results stored in Access, SQL, or Oracle database (add-on); Questions export to QML or QTI XML; Comments: (interface, usability, look-and-feel, etc)

• Wizards can be used to create questions • Assessments interface very complex for average non-technical user • More applicable to large-scale, high-stakes testing than small-scale higher ed use. • All question types listed above, plus: Java, Flash, matrix, pull-down, explanation, hotspot,

Likert, select a blank, text matching (not very effective), yes/no • To use with Blackboard, a ‘bridge’ must be installed on BB server; for WebCT a

‘powerlink’ must be installed on WebCT server • ‘Jumps’ allow for branching based on user responses.

. 17

http://www.questionmark.com/

Perception Screenshots Assessment Manager Interface:

Item editing view:

. 18

Tool: HotPotatoes v.6 Last Updated: March 2004 Organization: University of Victoria/Half-Baked Software Web address: http://www.halfbakedsoftware.com / http://web.uvic.ca/hrd/hotpot/Cost: Free if non-profit and make materials freely available on web; or $120/single user Authoring requirements: Software installed on Mac: PowerMac/G3/G4 OS 8+; PC: Windows 98/ME/NT4/2000/XP Server requirements: web server or hotpotatoes.net hosting (not free) End user (student) requirements: IE6, Netscape 7, Mozilla, Opera 7.2, Safari (prior version of HotPotatoes can save pages for earlier browsers). Question types:


Graphics: Equations: MathML, Unicode SCORM compliant: IMS-future? Administration & Tracking:


Import formats: none Export formats: exports exercises as HTML/javascript web pages; can export to WebCT. Comments: (interface, usability, look-and-feel, etc)

• Useful as a free tool for non-credit/non-graded exercises and reviewing. • Easy to use interface but very limited options • Web page output could be modified in Dreamweaver, etc. to improve layout, style. • Javascript provides no security (ie. smart students can ‘view source’ to find answers).

. 19

http://www.halfbakedsoftware.com/

http://web.uvic.ca/hrd/hotpot/

HotPotatoes Screenshots Quiz Authoring interface:

Sample quiz interface:

. 20

Tool: Blackboard (various versions) Organization: Blackboard, Inc. Web address: http://www.blackboard.com Cost: Purchased at dept/institutional level – expensive; 1-yr course hosted by BB $295 Authoring requirements: web browser (detailed specifications not listed) Server requirements: see site; extensive install/support needed End user (student) requirements: web browser (detailed specifications not listed) Question types:


Graphics: Equations: WebEQ (add-on $99/user) SCORM compliant: Administration & Tracking:


Import formats: Export formats: gradebook can be exported Comments: (interface, usability, look-and-feel, etc)

• Blackboard is a widely used CMS. • Security/authentication features built-in • Process to create test items is cumbersome but can be saved into pools and

reused/exported/shared with other BB users. • Courses can be archived, recycled. • Publisher cartridges usually include question pools. • Can be used with Respondus if Building Block installed.

. 21

Blackboard Screenshots Partial view of item authoring screen:

Quiz feedback screen

. 22

Tool: TestPilot (Classic - 3.2.5 Enterprise - 4.4.2) Organization: ClearLearning, Inc. Web address: http://www.clearlearning.comCost: Perpetual license ranging from $750 (<200 students) to $15000 (up to 20,000 students) Authoring requirements: v4+ browser with Javascript Server requirements: MacOSx/Linux/Unix/Windows End user (student) requirements: Question types:

multiple-choice (no limit) multiple-answer(no limit) true/false fill-in-the-blank/Cloze ordering/ranking matching short answer/essay drag-n-drop numerical

Graphics: Equations: MathML,LaTex, Tex SCORM compliant: Administration & Tracking:


Import formats: Export formats: download scores as .xls file Comments: (interface, usability, look-and-feel, etc)

• Hideous interface filled with bugs • Ridiculously expensive

. 23

http://www.clearlearning.com/

TestPilot Screenshots This screenshot shows confusing interface - unclear what user should be doing or in what order.

. 24

Tool: ATutor v. 1.3.3 Last Updated: February 2004 Organization: University of Toronto Web address: http://www.atutor.caCost: Free/GNU Authoring requirements: recommended NS7+, IE 4+, Opera 5+ Server requirements: Web server (Apache), PHP 4.2.0+, MySQL 3.23+ End user (student) requirements: recommended NS7+, IE 4+, Opera 5+ Question types:

multiple-choice (10) multiple-answer true/false fill-in-the-blank/Cloze ordering/ranking matching short answer/essay drag-n-drop numerical




• Limited quiz functionality • Confusing navigation • Click (navigation) tracking and page usage statistics

. 25

http://www.atutor.ca/

ATutor Screenshots Sample test screen:

. 26

Tool: iCreate PowerPoint plug-in Last Updated: February 2004 Organization: Latitude Communications (now owned by Cisco) Web address: http://www.latitude.com Cost: $299 for desktop version Authoring requirements: PowerPoint 2000/2002/XP, web browser, Flash Player 5+ Server requirements: Windows 2000/XP server for live presentations End user (student) requirements: web browser, Flash Player 5+ Question types:

multiple-choice (3) multiple-answer true/false fill-in-the-blank/Cloze ordering/ranking matching short answer/essay drag-n-drop numerical




• All tracking capabilities appear to require integration with a AICC-compliant course management system.

• Interface doesn’t appear to be customizable in terms of colors, fonts, layouts, etc. • Feedback pop-up is sloooow. • Narrations can be added to slides. • iCreate converts PowerPoints into Flash .swfs that can be published directly to an iCreate

server or saved and put in a web page. • Text can be very small / hard to read when converted to Flash format in default mode;

larger slide mode is better but navigation is less obvious (slide list turns into drop down list only visible on rollover.)

• PowerPoints can be narrated slide by slide.

. 27

iCreate Screenshots Sample screen in published mode:

. 28

APPENDIX: Rejected Tools These are tools which did not meet enough requirements upon initial review and thus were not reviewed in-depth using the Tool Review worksheet.

• PHPTestManager – individual developer, minimal documentation, no live demo (only BAD screenshots)

• QuizShock – discussion board software tweaked to be ‘quiz’ tool with non-customizable Flash interface; very limited question format, tracking, registration, security functionality.

• LON-CAPA – awkward interface & very slow page loading, no teacher version of demo

• PHPQuest – programmed by 10th grader, no authoring demo avail, in development

• wpQuiz – broken demo

• starXText v 2.0 – very limited functionality

• Sunk Quiz Script – no demo, limited functionality

• WebQuiz/GuruQuiz – more of a game/contest tool than quiz tool

• Quiz Manager – very limited functionality

• ETest – demo not functional

. 29

APPENDIX: Tool Review Form Tool: Organization: Web address: Cost: Authoring requirements: Server requirements: End user (student) requirements: Question types:




Import formats: Export formats: Comments: (interface, usability, look-and-feel, etc) Add sample screenshots below.

. 30

online quiz tool review for statistics 10 - ucla · online quiz tool review for statistics 10 ......

Documents