© blue coat systems, inc. 2009. all rights reserved. blue coat systems confidential – internal...

© Blue Coat Systems, Inc. 2009. All Rights Reserved. Blue Coat Systems Confidential – Internal Use Only

Blue Coat Certification Exams Part 2: Creating Objectives, Items

Education and Training ServicesMay 2009


Introduction

Requirements on exam content

The structure of exam data to be created

Developing objectives

Developing items

Versioning and releases

2


Requirements

Exam content should be under version control (our SVN rather than Prometric)– Prometric Website allows accessing and editing anything,

but it is difficult to ensure version history. Master copy should be on our server.

Exam content should be imported to Prometric from XML rather than entered manually– Avoid double effort and reduce human-caused errors

Exam content should be single-sourced (http://en.wikipedia.org/wiki/Single_source_publishing )– We create items in one place and then use them wherever

it is appropriate.


High-Level Content Hierarchy: Sections

Subdivide the chapters into 3-8 large groups.

Give each group a short title (up to 25 characters) – it is a section

Decide, how many questions should be from each section

Ideally – formulate “3-8 most important things” covered by the given course. (You cannot expect the learner to remember, say, “20 most important things” – even if the course has 20 chapters).

Currently – we mostly do some ad-hoc grouping of chapters into somewhat related “clusters”. Also works fine


Our courses and sections

svnroot/proxysg/trunk/v311/exams/conf/section-chapter-index-v321.xml


Lower-Level Content Hierarchy: Objectives

Option 1: Exam measures the knowledge of the course book

Option 2: Exam measures the skills related to ProxySG (PacketShaper) that are deemed importand by SMEs


“General skills” and “specific skills” (or objectives)

Chapters are divided into “general skills” (denoted by A,B,C, ... ) The expert opinion on skill importance could be used to assign weights to the exam items (not currently done). Skills could have different degrees of importance — they could range from very important (weight=4), to important (weight=3), to somewhat important (weight=2), to of little importance (weight=1)).

“General skills” are further divided into separate “specific skills” or objectives. Each objective is clearly stated by a verbal phrase (“Describe ...”, “Identify ...”, “Configure ...”, etc.). Each objective is assigned to a specific slide in the textbook.


A Fragment of the “Objective Tree”

Refer to individual objectives as PROD.B.4 etc.

Traceability to things that do not change (as slide numbers sometimes do)


Creating Items

After objectives (general and specific skills) are clear, can write items.

Prometric exam import requires specifically structured XML (accordingly to a DTD – Document Type Definition)

It does not matter, how we create this XML as long as it conforms to this structure.

Validating against DTD is not a guarantee that exam can be imported – some text fields may have limited length. (Prometric gives some unfriendly error messages in this case)


FrameMaker – XML Editor of Choice

Prometric allows import of XML, so we need to create or maintain data in XML format. – There is an alternative – create our own database

application. More time-consuming (we would eventually creating our own mini-Prometric application).

Any good open-source XML editors?

FrameMaker is a strange XML editor, but it works– File > Preferences > General > Product Interface – can

set this to “Structural Frame Maker” (allows editing any XML according to its XML Schema or DTD).

– Use regular WYSIWYG FrameMaker (this is our approach).


The Way of FrameMaker

Item entry is in fact easy due to FrameMaker paragraph sequencing. Not following this order may cause invalid XML files. (A better XML editor? A Web form?)


Template for a Single Response Item

Stem

Stepa

Stepb

Alternativea

Alternativeb

Answer

Optional meta-info fields:Version, Hide, Tags

May insert here an optional “Story”: (1) text paragraph, (2) code sample, (3) anchored frame with an image

Points


What is Stepa/Stepb?

After the “Stem” (i.e. the formulated question for an item) there is an optional list of “Stepa” followed by one or more “Stepb”. Answer alternatives are something like “a & b only”, “a, b & c only”, etc.

This is our workaround to serve questions with multiple correct answers (emulated by questions with single response)


Template for TRUE/FALSE Item

Same as “Single response” items; use only two alternatives – TRUE and FALSE


Metainfo Fields (optional)

All the fields not case sensitive; can omit.

Analysis (one or more paragraphs explaining, why the correct answer is correct)

ConformsTo (all, *-v311, v303, v311, v311-*)

Hide (No/yes)– “yes”, to exclude item from production and mock exams.

Default is “no”.

Tags (comma separated list)– If tag “sample” is used, then the item is used for

sample/mock exam.

– Short strings that identify item’s topic. Items involving HTTP can have tag “http”. Anything related to byte cache can have tag “byte-cache”. Will use to search.


Item Types

Only three item types are recommended:

Single Response: Question stem has 2-5 answer alternatives; one of them is correct

TRUE/FALSE – Question should be answered as either TRUE or FALSE

Multiple Response (currently only in PacketShaper exams)


Assigning Points for an Item

Author assigns points for items. Possible strategies: – Currently used: All items are worth 1 point. In the limit it is

normal distribution anyway. No partial credit.– More difficult items get more points (accordingly to

previously observed statistics or beta-test results)– Harder-to-guess items in probability terms get more points.

TRUE/FALSE is easier to guess correctly than to pick 3 from 5. (We do not play with this – we do not punish for guessing either)

– More important skills get more points (This is recommended practice). Problem: We do not know yet, which skills are more important.

– Any other strategy?

© blue coat systems, inc. 2009. all rights reserved. blue coat systems confidential – internal...

Documents