BSD Certification: How to Create a

Psychometrically ValidCertification Examination

Dru LavigneChair, BSD Certification Group Inc.

EuroBSDCon, 2011

Outline

What is psychometrics? (and why should I care?)

The Job Task Analysis (JTA)

Certification Requirements

Item Writing

Beta Period

Angoff Session

Exam Maintenance

Becoming Involved

Psychometrics

Science of assessment

Determines the skills and knowledge to be assessed

Assesses understanding rather than memorization

Increases the value of the assessment

Psychometrics

Value to:

Certificant: since the exam matches the requirements, even if additional study aides are not available, examinees can gauge their exam preparedness by comparing their skillset to the listed objectives

Employer: has a yardstick that validates the minimum skillset of certified employees

Psychometrics

Value to:

Certifying Body: is in a position to successfully defend the reliability of the assessment should an examinee dispute the results of their examination

Community: the certificate is worth more than the paper it is written on and certificants are not able to pass the exam by memorizing the content available from "exam cram" websites

Psychometrics

Signs of pyschometric validity:

Published exam requirements

Exam questions match the exam requirements

Assessment is repeatable and defendable

Exam does not try to “trick” the examinee or introduce bias

Exam does not contain inaccuracies

JTA

First step in creating a new certification

Determines which tasks will be assessed

Goal is to define the content areas (domains) and the importance and frequency of each task

Survey sample should be large and should include working members of the target audience and their employers

JTA

Results are analyzed and used to create the Certification Requirements

BSDA: http://www.bsdcertification.org/downloads/sr1_links.pdf

BSDP: http://www.bsdcertification.org/downloads/delivery_survey.pdf

Certification Requirements

Also known as the exam blueprint

Contains the following information:

1. A clear definition of the intended audience

2. The knowledge domains and their percentages (Table of Specifications)

3. Detailed exam objectives that clearly identify the tasks to be assessed

Certification Requirements

4. The depth of knowledge to be assessed according to Bloom's Taxonomy(http://en.wikipedia.org/wiki/Blooms_taxonomy)

This document should be published and freely available to examinees, employers, and the creators of study aides

BSDA: http://www.bsdcertification.org/downloads/pr_20051005_certreq_bsda_en_en.pdf

BSDP: http://www.bsdcertification.org/downloads/certreq_bsdp_en.pdf

Item Writing

SMEs (Subject Matter Experts) use the blueprint to create exam questions (items)

Item writing is an art and a skill

Items must be written to the skill level of intended audience

Items must be clearly written and understandable, even to non-native English speakers

An item can only assess one exam objective

Item Writing

Items must be consistent in punctuation, style, and formatting to avoid distractions

Items can't contain cut/pastes from instructional materials

Items shouldn't contain double negatives

Correct answers should not be in same position or follow a pattern

Item Writing

Multiple choice item contains:

Stem: the question being asked

Key: the correct answer(s); appears as an option

Distractors: 3-4 incorrect answers which appear as options; must be plausible and not contain contextual cues

Item Writing

First draft of an item rarely looks like the final version of question that appears on exam

Items undergo much scrutiny, rewriting, and review by both the SMEs and the psychometrician

An exam form can not contain enemies (a question that helps to answer another question)

Beta Period

Once the items have finished their review, the psychometrician creates one or more exam forms

Goal of the beta period is to generate statistical information on how examinees respond to the exam forms and to identify anomalies

You can't pass or fail a beta as the passing (cut) score can't be determined until the stastics of the beta are analyzed, required questions are rewritten or removed from the form, and the Angoff session has occurred

Angoff Session

Goal of theAngoff session is to set the cut score (passing score) by determining the number of items a minimally competent examinee would answer correctly

Each SME individually rates every item

Ratings are compared and any discrepancies are discussed until concensus is achieved

Psychometrician uses final ratings to determine exam's passing score; exam can now be published

Exam Maintenance

Once exam is published, psychometrics does not end

Statistics continue to be gathered for analysis on how examinees respond to each exam item

Psychometrician will alert exam body of any anomalies that may indicate cheating is occurring

Regular review period is scheduled (by time or number of completed exams)

Exam Maintenance

For each established review period, the psychometrician will identify which items should be reviewed by the SMEs

Some items may be rewritten, removed, or replaced, and pilot items may be introduced

New version of form may be published (usually not advertised)

Period must be set for when to redo JTA and start process all over again

Becoming Involved

Members of intended audience can act as SMEs and participate in item writing, item review, and Angoff sessions

Employers who include BSD certification in job postings help to increase awareness of the certification program while assisting the organization in finding suitable candidates

Anyone who is interested in promoting BSD can assist the BSDCG in spreading the word about BSD certification

Additional Resources

What it Takes: 3 part series in the June to August Issues of BSD Magazine (http://bsdmag.org)

A Case Study in Open Source Certificationhttp://blip.tv/bsdtv

The Importance of Psychometricshttp://www.bsdcertification.org/downloads/psychometric.pdf

Contact & Questions

Contact:dru@freebsd.org

URL to slides online: http://www.slideshare.net/dlavigne/ eurobsdcon-2011

URL to formal paper:http://www.slideshare.net/dlavigne/lavigne-eurobsdcon11