IAEA Training in Emergency Preparedness and Response

Exercise Evaluation

LectureLecture

Module L-056Module L-056

L-056: Exercise Evaluation 2

Objectives

• Understand the importance of evaluation for the success of the exercise

• Understand performance-based evaluation

• Know the role of evaluators and their procedures

• Be able to develop evaluation criteria

• Know how to grade and report evaluations

L-056: Exercise Evaluation 3

Principles of Evaluation

• Performance required by plans and procedures is being tested

• Evaluations are results-oriented to enhance the plan and the supporting elements, such as proper organization, training, and equipment

L-056: Exercise Evaluation 4

Aim of Evaluations

• The aim of an evaluation is to identify strengths and weakness in the emergency response capability

L-056: Exercise Evaluation 5

What is a Performance-based Evaluation?

• Focus on results, not on process

• Based on response objectives

• It answers the question: was the result achieved?

L-056: Exercise Evaluation 6

How to Evaluate Based on Performance (1)

• Define the response or exercise objective• e.g. Protection of facility personnel

• Define the criteria• e.g. The personnel are promptly alerted

• e.g. The dose to personnel is kept within established limits

L-056: Exercise Evaluation 7

How to Evaluate Based on Performance (2)

• For each performance objective, define the evaluation criteria• e.g. The personnel are promptly alerted

• Was a public address message broadcast immediately upon discovery of an emergency situation?

• Did all personnel hear the message?

• e.g. The dose to personnel is kept within established limits

• Were personnel relocated to safe areas?

• Was the dose rate monitored at personnel assembly areas?

L-056: Exercise Evaluation 8

How to Evaluate Based on Performance (3)

• After the exercise, based on the FACTS, answer the question: “was the performance objective met”?

• It is not absolutely necessary that all criteria be satisfied• The actual response will depend on the situation

L-056: Exercise Evaluation 9

How to Evaluate Based on Performance (4)

• If the performance objective was met• Was the procedure followed?

• Was the result achieved in good faith or by emergency?

• If the performance objective was not met, why?• Was the procedure followed?

• Was training lacking?

• Was equipment lacking?

• Was the exercise at fault?

L-056: Exercise Evaluation 10

Evaluation Team

• Organization• Chief evaluator

• Evaluators for all key groups or locations

L-056: Exercise Evaluation 11

Evaluator Role and Responsibilities

• Chief evaluator• Coordinate the evaluation team

• Instruct evaluators on the schedule, method and criteria

• Ensure consistency between evaluators

• Instruct the evaluators on the format and schedule of the evaluation reports

• Produce the final evaluation report

L-056: Exercise Evaluation 12

Evaluator Role and Responsibilities

• Evaluators• Observe and record

• Do not evaluate during the exercise

• Stick to facts

• Evaluate after the exercise

• Provide a report to the chief evaluator

L-056: Exercise Evaluation 13

Choice of Evaluators

• Evaluators must be qualified for the function they evaluate

• Evaluators must be trained• Evaluation techniques

• Procedures of the groups being evaluated

• Evaluators do not need to know the detailed scenario

L-056: Exercise Evaluation 14

Preparation of the Evaluation Team

• Brief the evaluation team

• Provide clear written instructions

• Organize a site orientation tour

L-056: Exercise Evaluation 15

Evaluator Instructions

• Before the exercise:• Review the scenario timeline and objectives

• Review the procedures used by the participants

• Review the applicable evaluation criteria

• Understand the actions required to demonstrate a performance objective

L-056: Exercise Evaluation 16

Evaluator Instructions (Cont’d)

• During the exercise• Arrive before an event

• Locate a telephone

• Wear Evaluator’s identification

• Do not interfere with the participants

• Never criticize the participants

• Minimize communications

L-056: Exercise Evaluation 17

Evaluator Instructions

• Following the exercise• Attend debriefing and critique

• Observe the effectiveness of the debriefing and critique

• Obtain exercise documents

• Ensure exercise location is in a safe state

• Finalize your chronology notes

L-056: Exercise Evaluation 18

Observation Techniques

• Record arrival times

• Record names and functions

• Record times of major events

• Record repetitive interventions

• Listen to commands and instructions

• Listen and record Controller inputs

• Record exercise observations

L-056: Exercise Evaluation 19

Guide for Evaluators

• After the exercise, based on the FACTS, answer the question: “was the performance objective met”

• It is not absolutely necessary that all criteria be satisfied:• The actual response will depend on the

situation

L-056: Exercise Evaluation 20

Drillmanship

• Drillmanship is the art of exercise delivery

• Record facts on:• Simulation

• The exercise scenario

• Safety

• Participant feedback

L-056: Exercise Evaluation 21

On the Use of Evaluation Checklists...

• Several organizations use checklists for each group being evaluated

• This focuses the evaluation on each group in isolation rather than on the overall coordinated response performance

• Focus on recording important facts rather than “checking” a list

• Evaluate the overall performance on the basis of exercise objectives using all the facts collected by the evaluators

L-056: Exercise Evaluation 22

Debriefings

• Hot Wash-up

• Exercise Team Debriefing

• Executive Debriefing• Brief the management staff

• Give credit to the players for identifying deficiencies and give them a grace period to fix the problems before forcing compliance or imposing penalties

L-056: Exercise Evaluation 23

Safety Responsibilities

• Evaluators are also responsible for monitoring exercise play to ensure a safe environment

L-056: Exercise Evaluation 24

Grading Criteria

• Excellent

• Satisfactory

• Unsatisfactory

L-056: Exercise Evaluation 25

Grading Criteria (cont’d)

• Grading must be supported by facts (if it was not observed, it didn’t happen)

• Trivial mistakes remain trivial even when they are observed

L-056: Exercise Evaluation 26

Classification of Unsatisfactory Grades

• Critical Deficiency

• Major Deficiency

• Minor Deficiency

L-056: Exercise Evaluation 27

Deficiency Classification

• Planning

• Training

• Resources

L-056: Exercise Evaluation 28

Exercise Report

• A useful report is most difficult to write

• Based on facts, not impressions

• Focus on key areas

• Use the exercise objectives as the basis for the report

• Prepare and publish in a timely fashion

• Not everybody will agree with the findings

L-056: Exercise Evaluation 29

Post Exercise Actions

• Report clarification• Ensure there are no inaccuracies in the report

• Critical deficiencies• Take corrective actions as soon as possible, e.g. one

month

• Major deficiencies• Take corrective actions within a reasonable time frame,

e.g. months

• Minor deficiencies• Take corrective actions during the annual revision of the

plan or procedures

L-056: Exercise Evaluation 30

Summary

• The evaluation should be performance-based

• Evaluators and controllers are not the same persons

• An evaluator does NOT evaluate DURING an exercise, but after

• Base the evaluation report on the exercise objectives

• Look for corrective actions