2007 annual conference in pursuit of testing excellence the experts respond to your practical...

2007 Annual Conference

In Pursuit of Testing Excellence

The Experts Respond To Your Practical Testing Questions

Council on Licensure, Enforcement and RegulationAtlanta, Georgia2007 Annual Conference

Moderator and Coordinator

Chuck Friedman

Consultant


PanelistReid Klion

Chief Science Officer

pan-a TALX Company


Panelist

Rose McCallin

Director of Examination Services

Colorado Department of Regulatory Agencies


PanelistPaul Naylor

Consultant


SpeakersYOU………the audience!!!!


Credentialing and Exam Issues at CLEAR

• Challenges to Quality: Electronic and Candidate Integrity and Its Implications

• Subject Matter Experts: Friends or Foes?

• Getting the Most Out of Your Face-to-Face and Virtual Group Meetings

• Get Ready for Continuous Computer- Based Testing: Examination Quality and Item Security


Credentialing and Exam Issues at CLEAR

• Best Practice in Job Analysis and Validation of Test Specifications

• Promoting Excellence in Managing ADA Accommodations

• Quality Assurance and Regulatory Excellence

• Vital Implementation Issues in Translation

• Innovations and Trends in Testing


Responding to YOUR Testing Questions


ValidityReliability

Item Analysis

Item Review

Item Writing

Passing ScoreScoring

Job Analysis

Test Blueprint

Standard Administration


Valid

ity

Reliability

Item

Anal

ysis

Item

Writin

g

Pass

ing

Scor

e

Scorin

g

Test Blueprint

Standard

Administ

ratio

n

Job Analysis Item

Rev

iew


What are some alternative ways to conduct a job analysis?

Job analysis


What are some alternative methods to conduct a job

analysis?• Full Survey

• Literature Search

• In-Depth Interviews

• Focus Groups


What are examples of some advanced, alternative item formats?

What are the benefits and drawbacks of using these formats?


Innovative Item TypesOptions are Limitless…..

• Performance Testing • Analogue Work Tasks

- Visual Refraction

- Medical Diagnostic Processes

- Plant Operators

• Streaming Video and Audio• On-Screen Manipulables


…..but Reality Constrains

• Pragmatics– Usability– Logistics– Security ~ Salience of Rich Media– Development Costs Often High

What are we getting for our money?

• Psychometric Considerations Measures More Difficult to Develop than

Methods


If the testing process is disrupted, (lost answer sheets/data, power failure, computer

crash), what should my Board tell the candidate and what action should we take?


Stuff Happens…• Force majeure (“a greater force”)

Tornado Strikes Brooklyn—August 2007


What Could Happen? • Natural Disaster

• Power/Internet Failure

• Local Issue/Emergency

• Many Events Unavoidable

Can Only Plan to Mitigate Impact

• Policy to Deal with Issues


Mitigation of Computer-Based

Issues• Test Provider Issues- Redundant Systems- Backups

• End-User (or Test Taker) Problems- Backups of Test Responses

- Transmission to Servers After Each Screen Movement

-Ability to Resume Test Session


What feedback should be provided to candidates on score reports and results?


What feedback should be provided to candidates on score reports and results?• Scoring Methods

• Content Domain Scores– Interpretations

• Reliability Issues– Cautions

• Aggregate Scores

• How to Reapply


Should my Board or program sponsor allow candidates to review test results including

test questions?


Disclosure May Threaten Validity of Interpretations on Future Exams

Candidate Is Entitled to Fair & Reasonable Appeals Process

AERA, APA, NCME Standard 8.2


Allow Candidate Review?


Discourage Over-Interpretation of Information Subject to Considerable Error

Candidate Interprets P/F Status on Basis of Isolated Items






• Candidate Right To Fair, Meaningful Appeals Process– Some Programs Allow Limited Access to

Certain Test Items and At High Costs• Only Review M/C Items• Cost per M/C Item = $100+• Process for Review Not Realistic• If Credit Granted, “Not Accredited”



• How Minimize Validity Threats of Exposure & Provide Candidates Meaningful Appeals Process?– Independent Review by Qualified 3rd Party

• Non-Disclosure Contract• Supervised Environment• Authority to Challenge


What should I tell a candidate who failed the examination by one point?


• Regulatory Exams: Aims of Measurement & Validity of Interpretations

Maximize Maximize Accuracy of P/F Accuracy of P/F

Classification Classification DecisionsDecisions

Failed by “1 Point”?


Failed by “1 Point”? • Validity of Interpretations of Performance

on Regulatory Exams– Maximize Consist, Accurate Classification

Decisions– Consequences of Errors of Measurement

• Candidates Whose Performance Is Far Above or Far Below P/F Decision Point

• Candidates Whose Performance Is Close to P/F Decision Point (False Positives & False Negatives)


Failed by “1 Point”? Provide Results Reports that Encourage Interpretations Warranted by Precision of

Classification Consistency

0

5

10

15

20

25

30

35

A. B. C. D. E. F. G. H.

0%

20%

40%

60%

80%

100%

# Items on Exam N Correct %Msty by Part


Failed by “1 Point”? Exam Part

Exam Items

% of Test

N Correct

% Msty

A. 31 21% 24 77% B. 15 10% 11 73% C. 21 14% 16 76% D. 15 10% 11 73% E. 9 6% 5 56% F. 30 21% 22 73% G. 10 7% 6 60% H. 14 10% 11 79%

Totals 145 100% 106 73%


What are the practicalities and economics of maintaining a small licensing program?

Quality


PIA & Result

Post-Audits

Pre

limn

. T

ry O

uts

Cut S

core

s

ItemDevel.

Test

Specs.

Entry- Level Reqmts.

Small Programs: Practicalities/Economi

cs

JA for

Reg

Strong As Weakest Lnk


Small Programs: Practicalities &

Economics• Measurement Models: CTT or Rasch

• Problems of Dimensionality– Carefully Consider Logic of Test

Specifications, Content Dimensions

• Best Practices– Good Practice Analysis, Test Specs. &

Appropriate, Well-Formulated Test Items & Try-Outs


Validity As A Unified Concept


What are the benefits and drawbacks of computer based testing as opposed to paper

and pencil delivery?


Computerized Assessment - Why?

Convenience- Test Wherever There is a Computer

- Logistics Vastly Simplified

- Management of Errata

- Possibility of Continuous Testing

- Immediate and Accurate Scoring “Hanging” Chads OpScan Issues


Facilitation of Data Management

• Data in Electronic Format

• Web-Based Data = Immediate Availability

- Centralized Management of Assessment Programs- Access to Array of Psychometric Data

(e.g., response latencies, test taker activity)

- Facilitate Integration with Other IT Systems


Candidate Perception• Candidate/Member Preference

- Familiarity/Perceived Ease of Use

- Demographic Issues

• Organizational Appearance


Why Continue to Use Paper-Pencil?

• Programs That Must Test Massive Numbers Concurrently in One Locality

• Where Economics Dictate– Very Small Programs– Program Costs “Hidden”

• “Immovable” Impediments


We are starting a new licensing program which does not allow grandfathering. What issues need to be considered?


We are starting a new licensing program which does not allow grandfathering. What issues

need to be considered?• Security

• SMEs

• Limited Disclosure

• Portfolio Review


What are the issues and potential pitfalls in adapting/translating our examination?

See you soon!


What are the issues and potential pitfalls in

adapting/translating our examination?• Adaptation

– Process– Beyond Translation

• Back Translation– Intuitively Valid


Adaptation/Translation• Translators

– Language Skills– Cultural Skills– Subject Matter Knowledge– Trained in Item Writing/Review

• Field Testing

• Ron Hambleton/Stephan Sireci


What is the relationship between the examination process and reciprocity

between states? Between countries?


Relationship Between Exam Process & Mobility

• Reciprocity– Jurisdictions Agree to Recognize Each

Other’s Practitioners without Require to Demonstrate Competence

– License May or May Not Be Required from Other Jurisdiction



• Endorsement– Jurisdiction Recognizes License from Another

Jurisdiction– Licensee Relieved of Full Burden of Obtaining

License in New Jurisdiction



• Major Barriers – Standards for Entry to Practice Are Not Universally

Focused On Public Protection– Professional Interests, Scope-of-Practice Protection,

etc. = Barriers to Entry

• Change Agents – Mobility– Labor Supply/Demand– Communications Innovations


What is the impact of test industry consolidation (mergers, acquisitions) and

privatization for state licensing programs?


Consolidation & Privatization

• Media Landscape: Who Owns What?

• Testing Industry Legacies: Then & Now

• Licensure Exam Industry: Impact of Privatization by Regulatory Authorities

• Independent Auditing Mechanisms


Consolidation & Privatization

• Media Landscape: Who Owns What? (Columbia Journalism Review: http://www.cjr.org/resources )

– Three of Top 5 “Print” Corporations Affiliated with Testing Companies

• Reed Elsevier: Harcourt (The Psychological Corp.)

• NCS Pearson: Pearson Education (Pearson Assessments–Pearson VUE, Promissor, Riverside)

• McGraw-Hill: CTB


Testing Industry Legacies

“Big Four” Educational Marketer (2001) http://www.pbs.org/

Harcourt

CTB/McGraw Hill

Riverside Publishing

NCS-Pearson


Testing Industry Legacies: Then & Now

Reed Elsevier2001

007Pearson

Buying Harcourt(Pending SEC)

Harcourt 19051921: TPC



CTB/McGraw-Hill1926

2007CTB/McGraw-Hill

McGraw-Hill 1909



Houghton Mifflin1880

2006Pearson

Buys Riverside

Riverside Press1852



Pearson Education2000

2006Pearson Assessments

NCS1962



Thomson Prometric (CapStar)2000-2007

2007ETS Buys

Thomson Prometric

Drake 1980s-1997Sylvan 1980s-1999


Testing Industry Today

CASTLEWW

Kry-terion

CTS

Caveon

ACT

PCS SMT

CTB

AMP

CollegeBoard

Comira

ETSProme-

tric

Pear-son

More!


Privatization & Independent Auditing

MechanismsMarket Demand(Regulatory Agencies)

Who Is EnforcingCompliance with the Standards?

Suppliers(Testing Industry)


How do we combat threats to security for both paper-and-pencil test delivery and CBT

in the face of new technology?


Security• Any Exposure of Item is a Security Risk

• Issues Common Across Delivery Modalities– Content Management Handling– Item Presentation and Usage – Candidate Item Review Policies– Need for Good Proctoring

• Need to Analyze Risk Factors in Program Population


Item Security in CBT• SSL Encryption

• Randomized Forms

• Locked-Down Browsers

• Candidate Education (May be Particularly Useful for Certain Certification Programs…)

• Signed Attestations


Besides test delivery, how can technology impact licensing programs and state

Boards?


Impact of Technology• Data Are Electronic

• Web 2.0 is Real (Internet as Platform)

- Immediate Availability of Assessment Data

- Centralized Management of Assessment Programs

- Facilitate Integration of Test Data with Other IT Systems and Processes


What is on the horizon in the next five year that will affect the testing process for

licensure and certification?


What is on the horizon in the next five years that will affect

the testing process for licensure and certification? • Increased Dependence on Accreditation

• Increased Consolidation in Test Industry

• P&P Hangin’ In There


Speaker Contact Information

• Rose C. McCallin, Ph.D.– Director, Office of Examination Services– Colorado Department of Regulatory

Agencies– Division of Registrations

• 1560 Broadway, Suite 880• Denver, CO 80202• 303.894.7710• [email protected]


Speaker Contact InformationReid E. Klion, PhD

Chief Science Officer

pan—A TALX Corporation

[email protected]


Speaker Contact Information

• Paul D Naylor, Ph.D.

• Psychometric Consultant

• [email protected]

• (919) 730-1593

2007 annual conference in pursuit of testing excellence the experts respond to your practical...

Documents

regulation slide

annual conference council

testing slide

job analysis slide

item security slide

regulation credentialing

regulation moderator

alternative item formats