eurobsdcon 2011
Post on 07-Nov-2014
815 Views
Preview:
DESCRIPTION
TRANSCRIPT
BSD Certification: How to Create a
Psychometrically ValidCertification Examination
Dru LavigneChair, BSD Certification Group Inc.
EuroBSDCon, 2011
Outline
What is psychometrics? (and why should I care?)
The Job Task Analysis (JTA)
Certification Requirements
Item Writing
Beta Period
Angoff Session
Exam Maintenance
Becoming Involved
Psychometrics
Science of assessment
Determines the skills and knowledge to be assessed
Assesses understanding rather than memorization
Increases the value of the assessment
Psychometrics
Value to:
Certificant: since the exam matches the requirements, even if additional study aides are not available, examinees can gauge their exam preparedness by comparing their skillset to the listed objectives
Employer: has a yardstick that validates the minimum skillset of certified employees
Psychometrics
Value to:
Certifying Body: is in a position to successfully defend the reliability of the assessment should an examinee dispute the results of their examination
Community: the certificate is worth more than the paper it is written on and certificants are not able to pass the exam by memorizing the content available from "exam cram" websites
Psychometrics
Signs of pyschometric validity:
Published exam requirements
Exam questions match the exam requirements
Assessment is repeatable and defendable
Exam does not try to “trick” the examinee or introduce bias
Exam does not contain inaccuracies
JTA
First step in creating a new certification
Determines which tasks will be assessed
Goal is to define the content areas (domains) and the importance and frequency of each task
Survey sample should be large and should include working members of the target audience and their employers
JTA
Results are analyzed and used to create the Certification Requirements
BSDA: http://www.bsdcertification.org/downloads/sr1_links.pdf
BSDP: http://www.bsdcertification.org/downloads/delivery_survey.pdf
Certification Requirements
Also known as the exam blueprint
Contains the following information:
1. A clear definition of the intended audience
2. The knowledge domains and their percentages (Table of Specifications)
3. Detailed exam objectives that clearly identify the tasks to be assessed
Certification Requirements
4. The depth of knowledge to be assessed according to Bloom's Taxonomy(http://en.wikipedia.org/wiki/Blooms_taxonomy)
This document should be published and freely available to examinees, employers, and the creators of study aides
BSDA: http://www.bsdcertification.org/downloads/pr_20051005_certreq_bsda_en_en.pdf
BSDP: http://www.bsdcertification.org/downloads/certreq_bsdp_en.pdf
Item Writing
SMEs (Subject Matter Experts) use the blueprint to create exam questions (items)
Item writing is an art and a skill
Items must be written to the skill level of intended audience
Items must be clearly written and understandable, even to non-native English speakers
An item can only assess one exam objective
Item Writing
Items must be consistent in punctuation, style, and formatting to avoid distractions
Items can't contain cut/pastes from instructional materials
Items shouldn't contain double negatives
Correct answers should not be in same position or follow a pattern
Item Writing
Multiple choice item contains:
Stem: the question being asked
Key: the correct answer(s); appears as an option
Distractors: 3-4 incorrect answers which appear as options; must be plausible and not contain contextual cues
Item Writing
First draft of an item rarely looks like the final version of question that appears on exam
Items undergo much scrutiny, rewriting, and review by both the SMEs and the psychometrician
An exam form can not contain enemies (a question that helps to answer another question)
Beta Period
Once the items have finished their review, the psychometrician creates one or more exam forms
Goal of the beta period is to generate statistical information on how examinees respond to the exam forms and to identify anomalies
You can't pass or fail a beta as the passing (cut) score can't be determined until the stastics of the beta are analyzed, required questions are rewritten or removed from the form, and the Angoff session has occurred
Angoff Session
Goal of theAngoff session is to set the cut score (passing score) by determining the number of items a minimally competent examinee would answer correctly
Each SME individually rates every item
Ratings are compared and any discrepancies are discussed until concensus is achieved
Psychometrician uses final ratings to determine exam's passing score; exam can now be published
Exam Maintenance
Once exam is published, psychometrics does not end
Statistics continue to be gathered for analysis on how examinees respond to each exam item
Psychometrician will alert exam body of any anomalies that may indicate cheating is occurring
Regular review period is scheduled (by time or number of completed exams)
Exam Maintenance
For each established review period, the psychometrician will identify which items should be reviewed by the SMEs
Some items may be rewritten, removed, or replaced, and pilot items may be introduced
New version of form may be published (usually not advertised)
Period must be set for when to redo JTA and start process all over again
Becoming Involved
Members of intended audience can act as SMEs and participate in item writing, item review, and Angoff sessions
Employers who include BSD certification in job postings help to increase awareness of the certification program while assisting the organization in finding suitable candidates
Anyone who is interested in promoting BSD can assist the BSDCG in spreading the word about BSD certification
Additional Resources
What it Takes: 3 part series in the June to August Issues of BSD Magazine (http://bsdmag.org)
A Case Study in Open Source Certificationhttp://blip.tv/bsdtv
The Importance of Psychometricshttp://www.bsdcertification.org/downloads/psychometric.pdf
Contact & Questions
Contact:dru@freebsd.org
URL to slides online: http://www.slideshare.net/dlavigne/ eurobsdcon-2011
URL to formal paper:http://www.slideshare.net/dlavigne/lavigne-eurobsdcon11
top related