“You Can’t Always Get What You Want:

Accountability and Limits in Score Reports

to Boards”Troy Elliott

Association of Social Work Boards

2006 Annual Conference

Alexandria, Virginia

Council on Licensure, Enforcement and Regulation

Expect the Unexpected: Are We Clearly Prepared?

Censored image of psychometrician in Mick Jagger style tight leather pants

Presented at the 2006 CLEAR Annual ConferenceSeptember 14-16 Alexandria, Virginia

Who We Are

Association of Social Work Boards (ASWB)

49 states, DC, USVI, 7 provincesPublic protection missionSocial work licensure examinations—about 25,000

administrations annuallyOther services: Disciplinary databank, continuing

education provider approval, credentials registry, communication, etc.

Presented at the 2006 CLEAR Annual ConferenceSeptember 14-16 Alexandria, Virginia

The ASWB Examinations

• Five categories (Associate, Bachelors, Masters, Advanced Generalist, Clinical)

• 4 option multiple choice• 150 scored, 20 pretest• CBT• 4 hour time limit

Psychometric and administration support through ACT, Inc.

Presented at the 2006 CLEAR Annual ConferenceSeptember 14-16 Alexandria, Virginia

The Score

IRT process – no single raw score applicable to all forms

Slight variations to maintain consistent overall difficulty – anchor exam

Presented at the 2006 CLEAR Annual ConferenceSeptember 14-16 Alexandria, Virginia

The score, continued, unfortunately

So what everyone gets is the equated score, right?

If only it were that easy…

Presented at the 2006 CLEAR Annual ConferenceSeptember 14-16 Alexandria, Virginia

The score, continued, unfortunately

The Scaled Score Zone, or Ketchup vs. Catsup—the debate rages on

“70” versus “75”Where does it exist? Why does it exist?

How many English majors does it take to develop an apt simile?

70 75

Presented at the 2006 CLEAR Annual ConferenceSeptember 14-16 Alexandria, Virginia

Now that the basics are covered…

Who gets what right away: Passing Candidates

Presented at the 2006 CLEAR Annual ConferenceSeptember 14-16 Alexandria, Virginia

Now that the basics are covered…

Who gets what right away: Failing Candidates

Presented at the 2006 CLEAR Annual ConferenceSeptember 14-16 Alexandria, Virginia

Now that the basics are covered…

Who gets what right away: Regulatory Boards

Presented at the 2006 CLEAR Annual ConferenceSeptember 14-16 Alexandria, Virginia

Summary reportsEvery year, to boards

Presented at the 2006 CLEAR Annual ConferenceSeptember 14-16 Alexandria, Virginia

Choose your cliché The greatest thing since sliced bread….or

something else entirely?

Our great idea: reports to schools

Presented at the 2006 CLEAR Annual ConferenceSeptember 14-16 Alexandria, Virginia

School dataWhat the schools get

Presented at the 2006 CLEAR Annual ConferenceSeptember 14-16 Alexandria, Virginia

How could anything go wrong?

• Misuse – program outcome, marketing• Requests for more data• Fingerpointing

Presented at the 2006 CLEAR Annual ConferenceSeptember 14-16 Alexandria, Virginia

What does this have to do with my topic?

Reporting service highlighted relationship between regulatory boards and schools

Some boards interested in school data

Presented at the 2006 CLEAR Annual ConferenceSeptember 14-16 Alexandria, Virginia

Data salad, hold the context

Requests and questions from ASWB member boards

• Subscores?• Jurisdictional pass rates?• School data shared with everyone?

Presented at the 2006 CLEAR Annual ConferenceSeptember 14-16 Alexandria, Virginia

Data salad, hold the context

ASWB’s initial answers:

• No• No• No

Presented at the 2006 CLEAR Annual ConferenceSeptember 14-16 Alexandria, Virginia

What’s our problem anyway?

The three Ps: philosophy, psychometrics, potential litigation

Presented at the 2006 CLEAR Annual ConferenceSeptember 14-16 Alexandria, Virginia

What’s our problem anyway?

Philosophy

Whose data is it?

Presented at the 2006 CLEAR Annual ConferenceSeptember 14-16 Alexandria, Virginia

What’s our problem anyway?

Psychometrics

150-item test with content areas ranging from 3 – 22 percent of the examination.

Schools—and sometimes, entire jurisdictions—with small testing populations, even smaller first-timer numbers.

Presented at the 2006 CLEAR Annual ConferenceSeptember 14-16 Alexandria, Virginia

What’s our problem anyway?

Potential litigation

• Misuse/misrepresentation of data• Hiring/firing/program changes based on

poor data

Would we prevail? Probably. But at what cost?

Presented at the 2006 CLEAR Annual ConferenceSeptember 14-16 Alexandria, Virginia

Coming soon to an association near you

Protection of the exam meets service to members

Presented at the 2006 CLEAR Annual ConferenceSeptember 14-16 Alexandria, Virginia

Under consideration…

Compilation of jurisdictional pass rates, with identification where allowed

All school data reported to boards (in addition to existing program)

Presented at the 2006 CLEAR Annual ConferenceSeptember 14-16 Alexandria, Virginia

What did we learn?

A victim of our own success?Licensing (and the exam)more

important than ever• Fewer exemptions• More importance among employers• Recognition by educational

programs

Presented at the 2006 CLEAR Annual ConferenceSeptember 14-16 Alexandria, Virginia

What did we learn?

Looking for outcomes in all the wrong places

• Test data can get caught up in political and economic pressures faced by schools

Presented at the 2006 CLEAR Annual ConferenceSeptember 14-16 Alexandria, Virginia

What did we learn?Member education never stops

• Turnover not just limited to board members

• Multidisciplinary boards increase the need for education (“If they can do it why can’t you?”)

• Culture of profession regulated impacts expectations, available options

Presented at the 2006 CLEAR Annual ConferenceSeptember 14-16 Alexandria, Virginia

What do we still need to learn?

Information culture – has the data train left the station?

• Expectations• “Good” data vs. “transparency”• Whose data is it anyway?

Presented at the 2006 CLEAR Annual ConferenceSeptember 14-16 Alexandria, Virginia

What do we still need to learn?

Association culture – when is service a disservice?

• Long range planning, goal identification: don’t forget overall profile

• Tough decisions: sometimes you have to make them

Presented at the 2006 CLEAR Annual ConferenceSeptember 14-16 Alexandria, Virginia

And you thought I’d never shut up

Troy Elliott, Communications Director

Association of Social Work Boards

400 South Ridge Parkway, Suite BCulpeper, VA 22701

800-225-6880

Email: info@aswb.orgwww.aswb.org