Running head: Methods for evaluation 1

The Methods Used to Evaluate the NextEra Energy Instructor Continuing Training Program

Form

Sherry Cox

Walden University

Author Note

EDIT 6130-4, Program Evaluation.

Methods for evaluation 2

The Methods Used to Evaluate the NextEra Energy Instructor Continuing Training Program

Form

A necessary component of performing a program evaluation is determining what data is

needed to meet the project objectives and how that will be evaluated. Using a systematic

methodology, the evaluation methods used for the evaluation of the NextEra Energy Instructor

Training Program Continuing Training form are considered.

Evaluation Methods

This program evaluation is narrowly focused on the form used to collect and validate all

information required to meet the annual continuing training requirements for an instructor of any

of the accredited training programs. To meet this evaluation scope, this project will target the

inputs and outputs of the Instructor CT form and determine if all required information is

collected and validated at the time designated by the NextEra Fleet Instructor Training Program

Description, TR-AA-101. Five evaluation methods were considered for use in the design of this

evaluation; advantages and disadvantages for each method were considered (see table 1). A

mixed model approach comprised of the consumer, program, and participant-oriented approaches

was selected. This selection was based upon the advantages of these models and the ability to

minimize the disadvantages of each as they relate to the evaluation questions.

Data Collection Design and Sampling Strategy

The design for this evaluation project will use a mixed data collection methodology;

combining a review of data and face-to-fact interviews. A review of completed Instructor CT

forms will be reviewed to determine the following: Completeness and Pre-approvals & approvals

of inputs at the specified times. The LMS will also be queried to determine if any instructors

were statused as Unqualified due to failure to complete continuing training requirements. Face-

Methods for evaluation 3

to-face interviews will be performed to include a minimum of two-thirds of all qualified

instructors (n=30) and all training supervisors (n=4); interviews will be semi-structured in nature.

The interviews will include the evaluation questions listed in figure 2 and will allow for

unstructured questions, as deemed fitting by the interviewer, to gather information required to

satisfy the purpose of this evaluation. If the data review reveals any incomplete forms or

unqualified instructors under the conditions specified above, additional questions may be

included for those interviews involving affected individuals.

The evaluation design will satisfy the needs of all stakeholders. The instructors and

training supervisors, who provide the overwhelming majority of the inputs and outputs of this

form, will have the opportunity to provide input of its effectiveness. These two groups, being the

most intimate with the form, will provide the best indication of its value to act in its intended

purpose. As the designers of the form; the ITs will not be interviewed; information pertinent to

this stakeholder group, will be gathered through the review of the data.

There is minimal bias anticipated during the implementation of this evaluation plan.

Interviewers should begin each interview by reminding the interviewers that the goal of this

project is to ensure the Instructor CT form is the best vehicle to gather the required data; the

interviewee should also be reminded that the collection of relevant continuing information

satisfies Federal Code of Regulations, Academy/Accreditation and procedural requirements.

Interview Questions. To effectively evaluate the use of the Instructor CT form and

ensure it is meeting the needs of all stakeholders; this evaluation will answer five questions that

to determine the effectiveness of the form and make any necessary adjustments, five questions

Methods for evaluation 4

will be asked during this project. The evaluation questions, proposed by this project, are shown

in figure 2.

Two primary stakeholders should be included in the determination of which questions

should be included in this evaluation; the instructors and the training supervisors. Both groups of

stakeholders are vested in this form being accurate and easy to use. An Instructor CT form that is

difficult to use and does not collect all of the required information will lead to both instructors

and supervisors creating workarounds which could lead to noncompliance of the Training

Program Description.

The Instructional Technologist/Instructor Trainer (IT) at each NextEra Energy nuclear site

should also be included in the question selection. As the Program Administrators for the

Instructor Training Program at their respective sites, they use this form to enter completion of

continuing training requirements, by the instructors in their program, into the SuccessFactors

Learning Management System (LMS). The IT has to know, with certainty, that the information

contained on the form is accurate and compete.

Representatives from the above mentioned stakeholder groups should be asked to review

the proposed questions put forth by this evaluation plan and will be asked to validate that the

questions will satisfy the needs of this project.

Methods for evaluation 5

References

Fitzpatrick, J. L., Sanders, J. R., Worthen, B. R., & Worthen, B. R. (2011). Program evaluation:

Alternative approaches and practical guidelines. Upper Saddle River, NJ: Pearson

Education.

TR-AA-101, Fleet Instructor Training Program Description [NextEra Energy Fleet Training

Procedure]. (2015, March 9). Revision 3, Approved by W. Goodes for C. Sizemore

Methods for evaluation 6

Tables

Table 1

Table of Evaluation Models

Evaluation Model

Advantages Disadvantages

EXPERTISE AND CONSUMER-ORIENTED APPROACHES

EXPERTISEThe expertise method leverages the experience of the field’s experts.The expertise method allows for the evaluation of more complex programs and products.

CONSUMER-ORIENTEDThe consumer-orientated approaches are best used to evaluate products, services or organizations.Helpful in determining what to buy.The evaluation is unbiased.The consumer of the product or service is the beneficiary of the evaluationEasy to use and understand

EXPERTISEThe expertise method is subject to the prejudices and politics of the experts on the evaluation panel.CONSUMER-ORIENTED

May be narrowly focusedNot open to debate

PROGRAM-ORIENTED EVALUATION APPROACHES

Can be used to explain the program outcomeAvoids unknowns in outcome

Heavily concentrated on research and less on stakeholdersMay over emphasize outcomes

DECISION-ORIENTED EVALUATION APPROACHES

More comprehensive than other modelsUses a systematic approachSensitive to the needs of those who will use the evaluation

There is an assumption that decision making is rational and predictableThe focus is on the needs of program managers

PARTICIPANT-ORIENTED EVALUATION APPROACHES

This approach pluralistic; it focuses on description and judgementThe model has multiple uses; it can be used to evaluate an organization or individual learning

There is a potential to be cost and labor intensive.Higher potential for less knowledgeable stakeholders to lead the study inappropriatelyLesser ability to replicate

Explain your choice of model for your program evaluation: The model employed for this program selection is a mix of the consumer, program, and participant-oriented approaches. The decision to use a mixed-model approach over a single method is to leverage the advantages of each method and minimize the disadvantages of all in order to produce a strong and meaningful product (Fitzpatrick, 2011).

Methods for evaluation 7

Figures

Methods for evaluation 8

Methods for evaluation 9

Figure 1. Example of the NextEra Instructor Continuing Training and Development Form

(Revision 3,2015).

Methods for evaluation 10

Figure 2. Proposed Evaluation Questions 1. Are all required

inputs obtained at

the time specified?

a. What are

the barriers

to

obtaining

this

information

?

2. How is the

instructor’s

technical

continuing training

validated?

a. Are there

any

difficulties

in

determinin

g the

technical

requiremen

ts? (for

Operations

Training

Instructors