Phillip W. Boswell MA, B.Ed, RRP

R. Stephen Russell, MA. Sc, P. Eng

Common Tools in Vocational Evaluation for Marginalized Client Populations

Learning Objectives

Introduction Identify the difference between vocational

assessment and vocational evaluation Recognize the three levels of vocational

assessment Ascertain the factors involved in the appraisal of

clients Identify useful voc assessment/evaluation resources Develop a working knowledge of common tools

used in vocational evaluations Work samples

Assessment - Defined

Systematic procedures to obtain information from a variety of sources to draw inferences

about people

(Standards for Educational and Psychological Testing, 1999)

Five Phases of Vocational Rehabilitation Same job with the same employer. Same employer, modified job (light duties) Different employer, similar or closely related

job Different employer, alternative employment in

the same or related industry Development of new occupational skills

What can be assessed?

Functional skills (basic skills & functional capacity)

Interest Achievement Aptitude Intelligence Learning style

Work Samples Personality Assistive technology Neuropsychological Sensory impairments Career Assessments Dexterity

Career maturity, competency, employment barriers, emotional and multiple intelligence, job readiness, personal factors, personal style, prior learning and values

Factors in Appraisal of the Individual

Education & Training

Acquired Skills

PotentialSkills

Leisure Time

Activities

Personal Traits

Physical Capacities

Social &Economic

Factors

Interests

Pearl E.White

Vocational Evaluation Tools

Achievement Tests

Work Samples

Aptitude Tests

Neuro-psychologicalAssessment

Personality Assessments

Functional Skills

Assessment

Dexterity Tests

InterestInventories

Pearl E.Gates

Individual Appraisal/Tools

Education & Training

Acquired Skills

PotentialSkills Leisure

Time Activities

Personal Traits

Physical Capacities

Social &Economic

Factors

Interests

Pearl E.White

Achievement Tests

Work Samples

Aptitude Tests Neuro-

psychologicalAssessment

Personality Assessments

Functional Skills

Assessment

Dexterity Tests

InterestInventories

Pearl E.Gates

Vocational Assessment - Defined

A general term for the process of identifying and appraising an individual’s level of

functioning in relation to vocational preparation and employment decision making

Why Vocational Assessment?

Clients often do not have precise occupational goals and aspirations. They frequently have insufficient or inaccurate information on the labour market and limited self-knowledge of their own interests and abilities.

Purpose of Vocational Assessment Plan a course of action Enhance client self-knowledge and

vocational decision-making abilities Predict realistic employment outcomes that

result in successful client vocational rehabilitation

Role of the Rehabilitation Professional Collect enough information or ‘data’ about the

client to diagnose and make predictive statements about his/her potential to obtain a successful rehabilitation outcome

Levels of Vocational Assessment

Level 1 – Screening/Needs Assessment

Level 2 – Clinical or Exploratory

Level 3 – Comprehensive Vocational Evaluation

Level 1 – Screening/Needs AssessmentInitial Process

Needs Assessment Determine what is necessary to develop a plan of

action (e.g. vocational plan)

Level 1 – Screening/Needs AssessmentIncludes

Interview with client Collect routine background information (demographics)

Reliance on subjective consumer statements Vocational choice/interest Self-estimates of competence Reported work history

Functional Assessment (e.g. personal capacity questionnaire)

Limited, if any, standardized testing (e.g. interest)

Level 2 – Clinical or ExploratoryIntermediate processIn depth exploration or case study approach to the client and

vocationally related circumstances. Builds on level 1 information

through the use of: Additional interviews Collect/analyze documents (schools records, medical records, etc.)

Career exploration Vocational and/or adjustment counselling Psychometric/standardized testing Transferable skills analysis Computerized job matching Job Analysis and/or environmental assessment Assistive technology considerations

Level 3 – Vocational Evaluation

“Final” Process Comprehensive individualized holistic process of assessment

that utilizes specific instruments, procedures, and behavioural observation

Designed to measure, observe, and document interests, values, temperaments, work-related behaviours, aptitudes, skills, and physical capacities, in order to predict viable employment and/or training outcomes

Used when more in depth information about the client is necessary and not available from information in level 1 and 2

Level 3 – Vocational EvaluationReal or simulated work used as the focal point of the evaluation Systematic observation and recording of work behaviour &

performance Occurs over time and uses multiple methods and techniques to

validate finds. Some combination of the following methods are used: Work samples, job samples Situational assessments, community based assessments Standardized testing Interview Transferable skills, job matching Analysis of background information Career exploration/job shadow Assistive technology considerations Prescriptive recommendations

Distinction Between VA & VEVE is holistic: considers disability/medical, psychological, social, vocational,

educational, cultural and economic issues

Ideally conducted by a certified vocational evaluator (CVE)

VE occurs over time (days/week +) evaluated over time with varying work-related demands and environments

Systematic Behaviour Observation and Recording Work performance Work behaviour Adaptive and/or transferable skills are questionable or unknown

Work is the specific focus of VE Work samples Situational assessments Community-based assessments Occupational resources Career exploration

If we are facing in the right direction, all we have to do is keep on walking Buddhist proverb

Vocational evaluation identifies….

Transferable skills Vocational Interests Worker Characteristics

and Behaviours Academic Abilities and

Potential

Learning Styles Work Aptitudes and

Abilities Suitable Employment

Alternatives

Evaluating Assessments

When assessing an individual, the first question is always……Should we use tests at all?

Testing may not always be appropriate Extreme anxiety Sensory impairments Minimal education Limited language abilities

Questions to ask

Is the test reliable (repeatable) and valid (does it measure what its suppose to?)

Does my client meet the reading level required by the test?

Is my client disadvantaged in any way by completing this test?

Will the test provide the information that I am looking for?

Questions to ask

Can the test administration procedures be modified to accommodate the client?

Can clients answer the question on the test form rather than a separate score sheet?

Are all the questions relevant and correct for my client?

Questions to ask

Are there relevant norms/criteria to compare my client’s performance?

Issue in rehabilitation is the lack of representation of marginalized individuals in sample populations

Marginalized client interested in competitive employment is better served using “general working population norms” rather than “disabled norms”

Transferable Skills

Can be achieved through formal or informal training

Is work related Should have a measurable component Should have the capacity to be matched to

other/alternative work options

Vocational Interests

Subjective measure of an individual’s preferences in work activities

Can add validity by incorporating variety of surveys with similar outcomes

Does not confirm that skills exist

Worker Characteristics and Behaviours

Vocational Evaluation relies heavily on the observational skills of the Counsellor

The Counsellor “takes on” different roles throughout the evaluation

Allows the inclusion of subjective information in the process

Academic Abilities and Potential Understanding current skills Can evaluate need for upgrading

To competitively seek employment To pursue further training

Can predict potential Timed vs. untimed testing

Needed to insure proper skill profile is created

Learning Styles

Generally subjective in nature

Can be helpful in identifying training environment

Can be helpful in identifying suitable work environment

Work Aptitudes and Abilities

Computerized assessment Work Samples Academic Achievement Aptitude testing (paper/pencil) Work history (transferable skills)

All are used to create a composite profile

Worker Qualification Profile

General Education Development (GED) Reasoning Math Language

Aptitudes General Learning, Verbal Aptitude, Numerical

Aptitude, Spatial Aptitude, Form Perception, Clerical Perception, Motor Co-ordination, Finger Dexterity, Manual Dexterity, Eye-Hand-Foot Co-ordination, Colour Discrimination

General Education Development-GED Embraces those aspects of education (formal

or informal) which contribute to a a worker’s reasoning development and ability to follow instructions; and, the acquisition of “tool knowledge” such as language and mathematical skills. (Revised Handbook for Analyzing Jobs - 1991)

GED FACTORS

Reasoning Development Math Development Language Development

1 to 6 scale

1 = Low 6 = High

Aptitudes

The capacity to acquire proficiency in and activity with a given amount of formal or informal training

Aptitudes may be general, such as learning aptitude or special, such as mechanical aptitude (Pruitt, 1986)

Aptitudes

General Learning Verbal Aptitude Numerical Aptitude Spatial Aptitude Form Perception Clerical Perception Motor Co-ordination

Finger Dexterity Manual Dexterity Eye-Hand-Foot Co-

ordination Colour Discrimination

1-5 scale

1=High 5=Low

The Tools of Vocational Evaluation Interest Surveys Academic Achievement and Aptitude Testing Computerized Assessment Work Samples Observation Other Surveys

Interest Surveys

Strong Interest inventory (SII) Canadian Work Preference Inventory (CWPI) Jackson Vocational Interest Survey (JVIS) Career Occupational Preference System

Interest Inventory (COPS) Reading Free Vocational Interest Inventory: 2

(R-FVII:2) Career Assessment Inventory (CAI)

Ability=Achievement+Aptitude Achievement Tests - What a person has

already done! Canadian Academic Achievement Test (CAAT) Wide Range Achievement Test (WRAT-4)

Aptitude Tests – What a person has the potential to do! General Aptitude Test Battery (GATB) Differential Aptitude Test (DAT)

Work Samples

Simulated work activity Criterion Referenced Test (CRT) Method Time Measurement (MTM) Standardized tasks Directly related to aptitude factors Valpar has over 30 work samples for

functional and job specific assessments

Valpar Work Samples

VCWS03 Numerical Sorting

VCWS06 Independent Problem Solving

VCWS07 Multi-Level Sorting

Norm-Referenced test & scores Compares the evaluee’s performance to the

performance of members of a particular norm group Selected norm group population should be well

defined, relatively current and appropriate to the evaluee’s goal

Results are reported in terms of percentile rankings

Norms

Frequently referred to as ‘moving targets’ as an individual’s ‘relative’ rank can shift significantly depending on the norm group

Pearl scored at the 87th percentile in arithmetic (grade 4 students)

Pearl scored at the 11th percentile in arithmetic (grade 10 students)

When norms are not enough

The world of work expects people to be able to perform some job tasks based on criterion of success or precision rather than compared only to how well others perform Expect airline pilots to take off and land an

airplane with 100% level of success, not just a little better than their pilot classmates!

Criterion references tests & scores Criterion-referenced tests have a predetermined

external standard or criterion Work related criterion generally come from industry

standards Number of widgets assembled per hour Keyboarding speed 60 wpm 0 errors MTM (Valpar)

Measures a specific knowledge, skill, or trait and compares person’s performance against the criterion

Focus and importance of on performance rather than relative rank of how others performed

Norm vs criterion

The “Standards for Educational and Psychological Testing” provide a basis on which to evaluate tests. established by APA, National Council on Measurement in Education, & American Education Research Association

The samples used for test validation and norming must be of adequate size and must be sufficiently representative to substantiate validity statements, to establish appropriate norms, and to support conclusions regarding the use of the instrument for the intended purpose

A Huge Issue---Norm vs. Criterion Most norm referenced tests do not use

representative norm samples that include individuals with disabilities

This often results in labels of disability and the identification of weakness

Important to note that in some instances it is helpful to understand how an individual is functioning compared to the “average” population when assessing for competitive employment

Norms Criterion

Competitive: compares people to other people

Relative skills reported Content difficulty fixed High scores inferred

from high performance on tasks with lower scores

Non-competitive: compares people directly to jobs

Actual skills reported Content difficulty

adjusts to evaluee’s skills

Scores measured directly at corresponding content difficulty

Norms Criterion

Validity depends on appropriateness of norm group

Strictly valid only for members of the norm group

Translation from percentiles by traditional methods distorts DOT levels

Validated by Method Time Measurement (MTM). No norm groups

Accuracy verified by data from over thousands of assessments

Measures DOT levels directly: no translation required

Best Norm Group (when available) Norm to the environment you are predicting to

e.g. employed workers, general working population, general

population, education, training norms 62nd percentile Form & Spatial GATB (GWP) 12th percentile MN Paper Form Board (employed drafting

technicians) Obviously, criterion-referenced tests are better

indicators of performance, so if you have a choice between a norm referenced and criterion referenced measure – choose criterion-referenced.

Computerized Assessment

Bridges Ability Profile Multi-Aptitude Assessment

Bridges Interest Profiler Career Cruising - Career Matchmaker Valpar Professional (PRO 3000)

(Magellan and Aviator)

PRO3000

System Manager Computerized Assessment (COMPASS) with 3

specialized work samples DOT Descriptions and database Work History for transferable skills analysis Pictorial/Audio Interest Survey Spatial/Non-verbal PET Survey Norm Referenced Test Conversion (TECO)

Suitable Employment Alternatives

Counsellor directed selection of vocational options

Objective listing based on testing results Can be combined with career exploration

The Client Perspective

Generally 3 to 6 hours Includes intake session Pre-screening Academic achievement testing Interest / Aptitude Testing Discussion of test results

VALPAR International

Started in 1973 in Tucson, Arizona Product Developer with over 6000 customers world

wide Component Work Samples Series (VCWS) JOULE Functional Capacity Evaluation PRO3000 Professional Vocational Assessment MAGELLAN Career Self Assessment/Explorer AVIATOR Aptitude & Interest Assessment SIGI PLUS Values-Based Career Planning

Career Systems Canada Limited

Steve Russell

P. O. Box 255

Buckhorn, ON. K0L1J0

tel: 705 657-8646

fax: 705 657-1572

cell: 416.721.5759

steverussell@necicom.net

www.valparint.com

P.W. Boswell & Associates

Phil Boswell

9534 Riverbend Road

Black Creek, BC  V9J 1E8

tel 250.337.5087

fax 250.337.5867

toll  877.790.7074

info@employment-training.org

www.employment-training.org