Streamlined job analysis for exam and training manual development

Andrew MartinPurdue Pesticide Programs

What is job analysis?

Job analysis refers to the systematic collection and organization of data that describes what workers do and what [worker] qualities are necessary to perform a job.

Definitions

Tasks - Discrete units of work-related behavior

Competencies - The collective knowledge, skills, and abilities necessary to perform a job

Knowledge - An article of information that jobholders need to know in order to perform the jobSkill - An acquired proficiency needed to perform a function or task Ability - A stable attribute necessary for performing a range of tasks

Licensure testing: A balance between public protection and individual rights1

Licensure tests focus on competencies critical to public protection.

Testing standards need to be sufficiently strict to protect the public, but not so strict as to be unreasonably demanding on the practitioner.

1 In this presentation licensure is synonymous with certification.

Why is job analysis the appropriate basis for licensure test development?

The proper approach to licensure test development is the adoption of standards advanced by the testing community.

Professional standards are articulated in Standards for Educational and Psychological Testing (American Educational Research Association, American Psychological Association, & National Council on Measurement in Education, 1999).

Standard 14.14 stipulates that job analysis is the primary basis for defining a credentialing test content domain.

A generalized job analysis model

Establish an SME committee.

Conduct worker interviews, observation, and document analysis.

Draft a survey instrument.

Pilot the job analysis survey.

Administer the job analysis survey to the survey sample.

Analyze the survey data and prepare a job analysis report.

National Railroad Contractors Association job analysis

NRCA was formed in1975.

There are currently five member companies.

The five member companies employ approximately 165 hi-rail and spray train operators.

The five member companies employ approximately 45 persons who supervise operators.

Railroad vegetation control operator interviews

Participant interviews yielded 119 job tasks organized under five functions:

I. Planning and Initiating Work ActivitiesII. Operating Spray EquipmentIII. Monitoring Work Site ConditionsIV. Documenting Work Processes and ProductsV. Maintaining a Clean Workplace and Functional

Equipment.

Interviews also identified 27 broad competencies organized as 17 knowledge statements, four skills, four abilities, and two [other] characteristics.

Interview data formed the basis for a first round Delphi instrument mailed to 12 subject matter experts.

Delphi round one results

Delphi panelist responses to the round one survey instrument refined the preliminary draft of tasks and competencies to include 103 tasks and 27 competencies by recommending the

deletion of 23 tasks,revision of 38 tasks,addition of 7 tasks,removal of 2 tasks to different functions,deletion of 3 knowledge statements,addition of 2 knowledge statements,revision of 4 knowledge statements, and theaddition of 1 ability statement.

Delphi, round one results formed the basis for a second-round instrument mailed to [the same] 12 subject matter experts.

Delphi round two results

Delphi panelists rated each task on a five-point, Likert-type, importance scale.

Means and standard deviations were calculated for each task and tasks were prioritized accordingly within each function.

Delphi panelists assigned points, also according to importance, to the knowledge, skill, and ability statements, such that the sum of the points equaled 100.

Means were calculated for each knowledge, skill, and ability statement and statements were prioritized accordingly within knowledge, skill, and ability areas.

Implications for testing and training

Sufficient data was collected to develop a test plan.

There was a marked cost and time savings to analyst and industry participants.

Program instructors revised and reinvigorated a long-standing program.

Some thoughts on job analysis and test development

Remember that there is no single best job analysis method.

Don’t let concerns about generalized job analysis models become an impediment to collecting job-related data.

Adopt a job analysis method that meets your agency’s needs and accommodates the idiosyncrasies of the regulated occupation.