Assessing Practice: OSCE Adapted for Social Work

Marion Bogo

King’s College, London May 2015

Holistic Competence

Meta-competencies

Procedural competencies

Meta-competencies Matter

Meta-competencies are higher order, overarching qualities, abilities and capacities:

• Cognitive • Affective • Reflective

Related to ability to use

Procedural competencies – operational, behavioral techniques and performance skills needed to carry out professional tasks. (Bogo et al., 2006; 2010; 2013; 2014).

A Model of Holistic Competence in Social Work

Skills

Self-regulation Emotions, reflection,

self-awareness

Knowledge Generic and specialist

Theoretical and empirical

Judgment Assumptions, critical thinking,

decision making

Complex Practice Behavior

PROFESSIONAL CONTEXT – VALUES (Bogo, Rawlings, Katz, & Logie, 2014)

ORGANIZATION AND COMMUNITY CONTEXT

The OSCE Adapted for Social Work

Simulated Interview

Assess Procedural Competencies

-Relationship building

-Assessment

-Interviewing

Scenarios depicting social work situations

Trained actors simulating clients

Rated by trained instructors

Reflection

Assess Meta-competencies

-Conceptualize practice

Assess/Judgment

-Relationship/Use of self

-Learn from reflecting on their practice

Questions

Reflective dialogue

Written reflections

Course assignment

What are OSCE’s

• An Objective Structured Clinical Exam • 15 minute interview with a standardized client directly

evaluated by an evaluator using a rating scale. • Standardized Client: An actor or actress trained to

enact the role of a client scenario consistently each time.

• Objective: Skills directly observed • Structured: A set scenario and format • Clinical: Assesses practice skill implementation. • Exam: Performance is evaluated on a set scale.

Post Interview Reflection

• Traditional OSCE only measures behaviors • A post-interview reflection with standardized

questions taps into meta-competencies: – what and how students are thinking and feeling

about their practice • Link theory to practice, critical thinking, judgment,

decision making • Subjective experience • Self assessment and learning

Rationale: Why OSCE’s

• Limitations of self-report and field evaluations for assessing direct practice skill.

• Need for reliable and valid evaluation methods in social work that directly assess skill.

• Post interview reflection questions demonstrate students’ ability to: – Accurately recall their practice – Use theory in practice – Think critically about the way they make decisions,

use of self in practice.

OSCE in Health Professions

• Extensive research literature exists • Method has acceptable validity and reliability • Accepted by students and faculty members, as

fair and authentic • Frequently used for episodic performance-based

assessment in undergraduate medical education, specialty areas, and in licensing examinations in some countries.

(Hodges, 2006)

OSCE in Social Work Critical Appraisal

• Found 14 research studies reporting intervention and observational data

• Used standardized clients/actors; trained actors, faculty or drama students.

• Educational interventions included individual, group or classroom interactions with a SC.

(Logie, Bogo, Regehr & Regehr (2013).

Limitations and Needs

• Need • replication – build on each other’s work

• standardized, reliable and valid tools

• studies focusing on evaluation of performance rather than satisfaction

• include diversity and cultural competence

Toronto Team Studies

• Reliability and validity of OSCE adapted method of performance and reflection, scales, scenarios

• Variation in student scores: – On performance – On reflection

• (Bogo et al., 2011; 2012; 2013)

Variation OSCE Performance

Variation OSCE Reflection

Toronto Team Studies

• OSCE correlated with field performance – to some extent (Bogo et al., 2012)

• Reflective dialogue highlighted needs that informed curriculum design.

Extremely well received by students

– Fair method for assessing competence

– Contributes to learning through practice

– More confident re: entering practicum

Azusa Pacific Studies: Summary

• Reliability – Internal Consistency strong for 5 point scale and

strong inter-rater reliability. – Overall item is strongly correlated between raters,

and strongest predictor of field performance.

• Validity – Potential for predicting field. – Good correlation with EPAS items, however, EPAS not

predictive of field.

• Rater bias risk and considerations.

The Process of Developing an OSCE

Identify competencies and practice behaviors

Map competencies, issues and content for

scenarios

Design scenarios

Construct measures

The Process of Developing an OSCE • Conceptualizing competence

What practice behaviors and meta-competencies do you want to measure

• Mapping competence Identify issues and abilities in potential scenarios

• Designing scenarios Practice situations with material for students to demonstrate competence

• Defining specific behaviors for a rating scale What should the student be able to do?

Conclusion… an iterative process….

Challenges to Conceptualization and Definition of Competence

• Context: Diverse settings and a range of practice approaches - general descriptions

• To assess practice behaviors need a level of specificity • Reduction to numerous discrete items - as if practice

were mechanistic • Need definitions and methods that capture the holistic

nature of practice: what type of knowledge is used and how it is used (internal cognitive and emotional processes that contribute to decision making and performance)

Selected References • Bogo, M., Rawlings, M., Katz, E., & Logie, C. (2014). Using Simulation in

Assessment and Teaching: OSCE Adapted for Social Work (Objective Structured Clinical Examination). CSWE: Alexandria, VI.

• Bogo, M., Regehr, C., Woodford, M., Hughes, J., Power, R., & Regehr, G. (2006).

Beyond competencies: Field instructors' descriptions of student performance. Journal of Social Work Education, 42(3), 191-205.

• Bogo, M., Katz, E., Regehr, C., Logie, C., Mylopoulos, M., & Tufford, L. (2013). Toward understanding meta-competence: An analysis of students’ reflection on their simulated interviews. Social Work Education 32(2): 259-273. DOI: 10.1080/02615479.2012.738662

• Bogo, M., Shlonsky, A., Lee. B., & Serbinski, S. (2014). Acting like it matters: A scoping review of simulation in child welfare training. Journal of Public Child Welfare 8(1)70-93. DOI: 10.1080/15548732.2013.818610

• Bogo, M., Regehr, C., Katz, E., Logie, C., Tufford, L., & Litvack, A. (2012). Evaluating the use of an objective structured clinical examination (OSCE) adapted for social work. Research on Social Work Practice. 22(4), 428 - 436. DOI: 10.1177/1049731512437557

Selected References

• Bogo, M., Regehr, C., Logie, C., Katz, E., Mylopoulos, M., & Regehr, G. (2011). Adapting objective structured clinical examinations to assess social work students’ performance and reflections. Journal of Social Work Education (47)1, 5-18.

• Katz, E., Tufford, L., Bogo, M., & Regehr, C. (2014online). Illuminating students’ pre-practicum conceptual and emotional states: Implications for field education. Journal of Teaching in Social Work 34, 96-108.

• Logie, C., Bogo, M., & Katz, E. (2015online). “I didn’t feel equipped: Social work students’ reflections on a simulated client ‘coming out’. Journal of Social Work Education.

• Logie, C., Bogo, M., Regehr, C., & Regehr, G. (2013). A critical appraisal of the use of standardized client simulations in social work education. Journal of Social Work Education, 49(1): 66-80. DOI: 10.1080/10437797.2013.755377

• Tufford, L., Bogo, M., & Asakura, K. (2015). How do social workers respond to potential child neglect? Social Work Education 34(2): 229-243.DOI: 10.1080/02615479.2014.958985