caveon webinar series - learning and teaching best practices in test security - june 2015
TRANSCRIPT
www.caveon.com 1
Caveon Webinar Series
June 17, 2015
Aimée Rhodes, CFA InstituteJoe Kamell, American Board of Radiology
Dennis Maynes, Caveon Test Security
Learning About and Teaching Best Practices in Test Security;
A Panel Discussion
www.caveon.com 2
Why this topic?
Training and education are essential for
improving test security
Respond
Protect
DetectImprove
www.caveon.com 3
Agenda for Today
• Educational Experiences
• Educational Resources
• Panel Discussion
• Summary
www.caveon.com 4
Educational Experiences
Pioneer: One who is among the first in a field of endeavor
www.caveon.com 5
Aimée RhodesDirector of Exam Security at CFA Institute, Aimée is tasked with maintaining the public confidence in CFA-sponsored designations by ensuring that:
1. exam content is secure;
2. no candidate has an unfair advantage over another;
3. only those individuals truly qualified to pass the exam do so; and
4. the risks inherent in our exam programs are identified and addressed appropriately.
www.caveon.com 6
Aimée’s Story• A lawyer by education, Aimée served on
active duty in the US Army JAG Corps before entering private practice.
• Aimée moved to ACT as Director, Test Security in 2005. – She oversaw the security of all of ACT’s resident
and contract programs
• Joined CFA Institute in 2008 and has built their exam security program from scratch, including:– A data forensic program– Web patrol program– Unusual candidate behavior tracking– Site visit program
www.caveon.com 7
Joe Kamell
1. exam content is secure,
2. Exams are delivered in a consistent and fair exam environment,
3. all security related events are investigated, and,
4. risks to the security of all ABR exams are identified and addressed
As a manager of exam delivery and security at the American Board of Radiology, Joe is tasked with maintaining the integrity of ABR Certification by ensuring that:
www.caveon.com 8
Joe’s Story
• BS – Health Education / MBA in Technology Mgmt.
• Several years in retail loss prevention• Training in interviewing and interrogation
techniques (Wicklander-Zulawski)• As a result of a public event in 2011, ABR
added the responsibilities of exam security to my duties, where the program was built from the ground up, including:– Web patrolling– Data forensics– Site visit program– Investigative process
www.caveon.com 9
Dennis MaynesChief Scientist
Dennis oversees the statistical analysis of exams for detecting potential test fraud and strengthening the security of client exams.
Other responsibilities:
1. Present and discuss test security topics at conferences, workshops, and webinars
2. Provide expert testimony support as necessary
3. Conduct research for furthering the state of the art of data forensics
www.caveon.com 10
Dennis’ Story
• With several others, founded Caveon in 2003
• Developed data forensics methodologies
• Presented in dozens of conferences• Testified as an expert witness• Favorite tag lines:– The answer’s in the data!– You can’t manage what you don’t
measure!
www.caveon.com 11
Educational Resources
www.caveon.com 12
Reference Works
• https://cespcert.org/resources• Handbook of Test Security– CCSSO TILSA Guidebook– Operational Best Practices for Statewide
Large-Scale Assessment Programs– Standards for Educational and
Psychological Testing– The ITC Guidelines on Test Security
www.caveon.com 13
Topic-specific Monographs
• ATP Test Security Committee– ATP Guidelines for Legal Protection of
Assessment Content, – ATP Online Piracy Enforcement
Strategies: Guidelines and Best Practices, – ATP Security Survey Report 2013, – Assessment Security Options:
Considerations by Delivery Channel and Assessment Model
www.caveon.com 14
Journals and Publications
• Journal of Educational and Behavioral Statistics
• Applied Measurement in Education
www.caveon.com 15
Conferences
• ATP – www.testpublishers.org
• COTS - http://cete.ku.edu/conference-test-security
• ICE - http://www.credentialingexcellence.org
• NCSA - http://www.ccsso.org/ncsa.html
www.caveon.com 16
Workshops
• Caveon Test Security Workshop• http://www.caveon.com/were-taking-t
est-security-on-the-road/
• Various specialty webinars offered by ICE and Caveon
www.caveon.com 17
CESP
• Certified Exam Security Professional• Training and Credential• www.cespcert.org
www.caveon.com 18
Other Online Resources
• Blogs: Caveon• Candidate Agreements• Test Security Procedures
www.caveon.com 19
Panel Discussion
www.caveon.com 20
Q1
• How do you stay current with test security developments and ideas?
www.caveon.com 21
Q1: How do you stay current with test security developments and ideas?
• Conferences are a good source of information
• Benchmarking with colleagues in the industry– Vendors– Testing partners– Other Directors, Exam Security
• Test security is ALWAYS in the news!
Aimée Rhodes
www.caveon.com 22
• Warning – blatant plug!– Caveon newsletters provide perspective
and news on current issues– Caveon often discusses current issues in
webinars
• Get engaged on Social Media– Current trends are often discussed on
LinkedIn– Some folks regularly “tweet” on test
security news
Q1: How do you stay current with test security developments and ideas?Dennis Maynes
www.caveon.com 23
Q2
• Is there a special vocabulary for test security? If so, how do I learn it?
www.caveon.com 24
• Vocabulary and language are very important• You need to know the differences between– Vulnerabilities, Threats, Risks– Anomalies, Irregularities, Violations,
Breaches• You need to be able to talk about “loss”• Watch how you talk about potential test
fraud– Are you worried about “cheating” or
“validity?”• Learn how to precisely discuss issues
without stereotyping or implying malfeasance
Q2: Is there a special vocabulary for test security? If so, how do I learn it?Dennis Maynes
www.caveon.com 25
Q3
• What advice would you give for someone who is new to this area and needs to create a test security program from scratch?
www.caveon.com 26
• NETWORK!
• Reach out to other exam security professionals and spend some time learning from them.
• Ask for examples of various documentation (policies, letters, etc.).
• Conduct audits to understand where you are and attack the major issues first.
• Attend training sessions, webinars and conferences as much as you can.
Q3: What advice would you give for someone who is new to this area and needs to create a test security program from scratch?
Joe Kamell
www.caveon.com 27
• Build relationships with folks who have been there before!
• Conduct a security audit or peer review to identify and prioritize your needs
• Identify resources from within your organization that you can rely on for help:– Exam Development - Exam Administration– Legal - Risk Management– Internal Audit - Professional Conduct
Q3: What advice would you give for someone who is new to this area and needs to create a test security program from scratch?
Aimée Rhodes
www.caveon.com 28
Q4
• What things can you do to network with others who are learning and teaching about test security?
www.caveon.com 29
• The most important thing is to pick up the phone and don’t be shy asking questions.
• Partner with companies such as Caveon and see what services are available that would be useful to you. Realize that you may not need the full menu of options they provide.
Q4: What things can you do to network with others who are learning and teaching about test security?
Joe Kamell
www.caveon.com 30
• Find the right contacts among your vendors
• Attend conferences• Seek out others who are in your
position• Establish peer groups– Non-disclosure agreements– Regular meetings– Share information and experiences
Q4: What things can you do to network with others who are learning and teaching about test security?
Aimée Rhodes
www.caveon.com 31
Q5
• How can I get involved in promoting test security education and providing training materials and experiences for others in test security?
www.caveon.com 32
• Be willing to speak up and share your experiences. Every testing program is unique.
• Regardless of expertise, there is always an opportunity to learn from others how they have handled different security-related events.
Q5: How can I get involved in promoting test security education and providing training materials and experiences for others in test security?
Joe Kamell
www.caveon.com 33
Q6
• What has been your biggest challenge and how have you handled it?
www.caveon.com 34
• Educating myself and others what constitutes a complete or comprehensive test security program
• The field is still developing and resources are not unified
• I read a lot. I read from other disciplines and I listen to what others are saying.
• I think about how writings like “The Art of War” apply to our discipline.
Q6: What has been your biggest challenge and how have you handled it?
Dennis Maynes
www.caveon.com 35
Summary
• Education is critical – an uneducated individual is one who can’t make choices
• We need to learn from others if we do not wish to repeat the mistakes that others have made.
www.caveon.com 36
Thank You!
Aimée RhodesDirector, Exam [email protected]
Dennis MaynesChief [email protected]
Follow Caveon on twitter @caveonCheck out our blog www.caveon.com/blogLinkedIn Group “Caveon Test Security”
Joe KamellExam Delivery and
Security [email protected]