Safety Training-Evaluation

[Type text][Type text][Type text]

10

Safety Training-Evaluation

Safety Training—Evaluation

Daniella Krantz

Purdue University – EDCI 577

October 11, 2015

Identification of Evaluation Goals and Scope

Overview

ArcelorMittal is the leading international steel manufacturer. The company employs hundreds of thousands of employees in 60 different countries. Great satisfaction is taken in the quality of steel produced and distributed. In many companies safety is a high priority. This can be due to many factors some of which include funding or reimbursements from insurance or shareholder confidence in the quality of the product. For ArcelorMittal safety is a highly important part of day-to-day production as a non-safe environment can be detrimental to an increase of quality production and could affect shareholder and other external views of the company as a desired place to work.

Each plant requires all employees to participate in a 12-module program that provides them with the safety training necessary as a part of yearly competencies. The current requirement is to complete the 12 modules by the end of the calendar year.

Over the past year, the accident rate at I/N Tek and I/N Kote has exceeded the goal set by management; in fact, the number has been increasing at an alarming speed. At one time the location was considered to be the number one safest plant for the organization; however, now the plant is falling behind in this category. To combat this predicament, the safety department has teamed up with the training department to pinpoint the gap in training. The Safety Department has asked the Training Department to evaluate the current method of safety training and suggest possible improvements. The current process requires employees to complete the annual training by the end of the year—but it can be done at their own speed. There are a series of modules that are done online for all employees and a separate continuation of the safety training that is monitored through software required of management. Each compilation of training will be looked at separately and as a whole to evaluate the best way to improve the overall process.

Objectives and Audience

The primary objective is to reach the specified safety goal, but ultimately, management would like to see zero work-related injuries. The audience for these safety modules is all operational support functions, non-operational support functions, first line managers, and middle managers.

Evaluation Summary

Because of the joint effort between the Safety and Training departments, there is a bit more being offered by way of resources. Since there is already a “template” of safety training in place, management has given a preemptive approval to proceed with this evaluation in order to correct the current accident rate. The evaluation plan will follow Kirkpatrick’s four levels of evaluation model as outlined in Evaluating Training Programs. The data collection will primarily occur through online resources since that is the current method of training delivery as well as being the most cost effective due to the fact a physical instructor is not needed to be trained nor be present to educate employees . Also, by collecting the data electronically, this will allow for quicker turn around of results and ease of analyzing the data.

Description of Process

Note: The process description will be utilizing dates beginning November 2015 in order provide a clear idea of the prototype.

After the decision is made to proceed, a formal evaluation will be commenced November 2, 2015. The goals and scope will be researched and planned out within the first month. After these two elements have been reviewed adequately, a timeline will be prepared and provided around the first of the new year to the training and safety departments in order to allow them a basic idea of how the evaluation will proceed. A review of each of the instruments used in the four levels will be done within the first quarter of the year. This can be useful to the training department so they can make any modifications on the modules before the company has completed the full training program evaluation. Since there is currently no Level 1 evaluation provided to the participants, this data will be collected in two groupings. The first will be provided at 6 months and the second will be completed and presented at the conclusion of the full evaluation. This will allow stakeholders and management to see if there are any imminent changes that need to be made and corrected.

Since the current method only requires employees to complete the modules by the end of the calendar year, a recommendation would be in order to place a requirement designating employees to complete one module per month. This type of process will allow data to be managed and analyzed accurately and timely. The Level 2 data collected in the current year can be used as a basic control group as the exams themselves have not had any changes made to them. Beginning January 4, 2016, the various levels of Kirkpatrick’s 4-level model will be implemented and reviewed extensively to assess for necessary modifications. While those in management positions monitor Level 3 data, a clearer and more standardized process would be suggested to limit the variables and possible non-standardized results.

One of the last details to be reviewed will be the analyzed results from the data of Level 1 through Level 3. Although the data will be gathered throughout the year and presented as a part of the safety meetings, the full data will be processed towards the end of the year; this will be commenced in November 2016 and formally presented along with the final conclusions. The final detail of the process is the data analysis, this will be completed and presented in three parts the first will be once 25% of data has been completed, the second at 50% and the third at 100%. A formal report will be compiled in early January 2017 and presented to stakeholders and management in February 2017. Recommendations will be based on the data analysis and made midyear and within the final report.

Please see the Appendix A for a Gantt chart review and associated table of the described process.

Instruments & Instrument Descriptions

Since there is not a current practice of Level 1 data collection, the description for this instrument is a basic example and may be modified according to the organization’s needs. Each participant who has completed the module will receive a prompt to complete the post-learning survey. This will then be submitted electronically and compiled through the use of software utilized by the training department. Although it will be recorded that each participant has completed the survey, no names will be associated with individual surveys. The evaluation will focus on the effectiveness of the training, the content, and the ease of using technology chosen to present the information. An evaluation for this level is important because it will allow shareholders and management to see effectiveness in meeting the objectives.

Please see Appendix B for a sample evaluation of Level 1.

Level 2 data will also be compiled electronically and at the completion of each module. The evaluation will consist of 10 multiple-choice questions related to the content of the specific module completed. The participant will be required to pass each exam with a minimum of an 80% in order to receive credit for finishing the module. To evaluate at this level is necessary in order to determine how much knowledge and learning was transferred to the participant. To some extent, it would be suggested to the training department to create exam questions that are not repetitive year after year in order to prevent participants from just memorizing answers. Also, the current model is fully embeds the exam within the module; it would be more favorable to create a post-test that provides a post-test that reflects a more traditional testing format.

Please see Appendix C for a sample presented at the end of each module. For the sake of space, an evaluation for each of the 12 modules is not provided.

A behavior change noted by a Level 3 evaluation is monitored by management at various times throughout the year through a process called shop floor audits. This occurs mainly through interviews with employees on a randomly timed basis. Depending on the level of management, the responsibility to complete a specified amount of evaluations differs, yet by the end of the year each employee will have been interviewed. The content covered will determine if the employee is using proper protective equipment, employees will be encouraged to voice any concerns, suggestions, and interviews are conducted on a one on one basis. Once the interview has been completed the manager is required to input the information using a software monitoring program. The employee interviewed is registered by the manager through the use of employee ID numbers. While the interview is recorded, the answers and level of knowledge retained is not recorded making this instrument somewhat lacking in value. Should there be a method to rating the employee on his answers, this Level 3 evaluation would be an invaluable tool of great use to the company.

Please see Appendix D for the various management level requirements for shop floor audits and a visual of the Level 3 evaluation data collection method.

The final instrument used to complete a Level 4 evaluation is a table that is presented at the monthly safety meetings. It is a compilation of the data collected from Levels 1, 2, and 3; this is mainly presented with a bottom-line score showing the number of work related injuries. Using the current method, only management is aware of the results as they are not effectively communicated to employees as a part of their jobs. A recommendation would be made to present this to all employees in order to aid in giving them a sense of ownership in their part of the data. This will also be listed in the Executive Summary in more detail.

Please see Appendix E for a visual of the monthly presentation of Level 4 compilation.

Data Collection

Level 1-Reaction

With the current model practiced for this location, there is not a Level 1 evaluation for the completed modules. Therefore, there is no data to be collected to be evaluated. A recommendation to implement this level of evaluation would be strongly suggested in order to better manage the program and assess module effectiveness. This recommendation is based on Kirkpatrick and Kirkpatrick (2006) saying, “If training is going to be effective, it is important that trainees react favorably to it” (Kirkpatrick and Kirkpatrick, 2006 p. 27).

Level 2-Learning

The learning evaluation is embedded within the module and will take place immediately at the end. The employee will receive a prompt to complete the exam. The results of this exam will be compiled once all employees have completed the module at the end of each month. Since the current method of electronic collection is adequate, the software utilized will be continued. A recommended modification would be to place a deadline on each module and link this to the employee’s evaluation. This would also provide some motivation to the employees to complete the modules in a timely manner.

Level 3-Behavior Change

A change in behavior will be recognized at monthly safety meetings as accidents are reported. There will also be reports compiled in accordance with the practice performed by managers known as the shop floor audits. These are completed at various times as determined by the safety committee. Once shop floor audits are completed, information is inputted into a performance management site and the data will then be analyzed in conjunction with the other levels. The completion rate of management in the responsibility of shop floor audits will also be recorded and collected to be discussed as a potential factor in the overall training program results. This data will be dynamic as the training program progresses throughout the year. A decrease in job related accidents would indicate a degree of success for the safety training program.

Level 4-Results

All results will be compiled and analyzed electronically. This will ensure accuracy in data collection and anonymity in order that employees will feel encouraged to answer evaluation questions honestly. Level 3 results will be an ongoing compilation throughout the year and answers may change as behavior changes and employees begin to feel a sense of ownership in their part of the company. The results will indicate the degree of effectiveness of the program. Should there be changes necessary, these should be addressed in the safety meetings and the evaluation instruments should be modified to produce effective change.

Analysis Process

For each of the following evaluation levels, stakeholder involvement is appreciated at the local level meaning the plant president, management, and individual employees. The plan president is responsible to answer to corporate authority for the plant safety rate; on the other hand, the division safety rates are a reflection on management and the care they take to ensure the subordinates are compliant with safety techniques and safety measures. Lastly, each individual is responsible for his/her own part in the division safety rate and the overall safety rate of the plant. One method that should be modified and utilized is a monthly requirement of each module. This method of control was mentioned in the Data Collection section—linking module completion to employee yearly performance evaluations.

Level 1-Reaction

Since there is no data being collected currently, the analyzing process described here will reflect a hypothetical situation. Using the electronic recording method, each survey question will be compiled with the associated answer. The training department expects a 100% completion rate of the post-survey evaluation; in order to achieve this, the module will not be marked “complete” in the system unless the post-survey evaluation is submitted. They also would like 60% of the survey questions to reflect the answer of “Agree” or “Strongly Agree.” If any question should not reflect this goal, the corresponding part of the module will be researched and modified if necessary.

Level 2-Learning

Level 2 will be evaluated using a software program that collates the data by question as well as by employee job function. This could be useful to analyze whether there is a gap in training based on specific job type rather than an overall training problem. With the use of technology, the process of analyzing can be done quickly and cost effectively. Specific Level 2 evaluation exams hold a required minimum passing score of 80%, per management and the Safety Department. The answers provided could also be compiled and compared to determine if there are questions that are poorly worded, do not directly reflect the learning, or other various problems with the questions. This would allow for a full review of the post-test to resolve any potential problems. For the safety audits (completed by management), a better tracking system of completed audits will be developed; one such method would be to allow management to see their completed progress as it advances.

Level 3-Behavior Change

Safety audits will be the hallmark of a Level 3 analysis. This type of evaluation will be done by interviewing employees at various times during the year; management will fill out the online form with the responses they have been given and this level can be combined with the Level 2 evaluations to determine if the initial learned knowledge is being retained. Kirkpatrick and Kirkpatrick (2006) have suggested the method of interviews as the more appropriate approach to gaining more information (Kirkpatrick and Kirkpatrick, 2006 p. 56). As is the case in many methods of data collection, repetition is another recommendation made by Kirkpatrick and Kirkpatrick (2006), which is completed with each employee based on the company policy. The performance management site will arrange and compare the answers in order to allow for the results to be discussed at the monthly safety meetings. This will permit changes in the system or method to be made as determined necessary by the safety committees.

Level 4-Results

The results will be communicated on a monthly basis to at the safety meetings. This is displayed in a tabular format as shown in Appendix E; the data is to evaluate and analyze the results will be based on whether the desired safety goal is being met. In order to control this collection a timely completion of Levels 1-3 would be recommended. This will then ensure that data is reliable and methods can be modified as necessary. Kirkpatrick and Kirkpatrick (2006) mention three recommendations that are in line with the current methods of result acquisition the company uses. By allowing a full year to obtain results an appropriate amount of time will be allotted to gaining data, also, by discussing the statistics monthly, this can allow management to see real time changes and deliberate and examine the cost of the training and its associate benefits.

Executive Summary

Overview and Objectives

This preliminary evaluation plan was prepared in order to ascertain whether the current safety program has been effective and to seek an answer as to the training and learning gap causing the increasing incidence of work related injuries. With this evaluation the company will be able to modify the current training program to create a safer work environment and potentially increase production and material output with the potential to have greater returns for the company. As stated earlier, the primary objective is to reach the specified safety goal, but ultimately, management would like to see zero work-related injuries.

Key Findings

While the basics of the current training program and the methods of data collection are a good template, there were many things noted and that could be hindering the safety goals management and corporate would like to see. While there are results being observed to follow the model of Kirkpatrick and Kirkpatrick (2006), Levels 1-3 should drive the overall improvement in safety of the company and be shown in the Level 4 data. The data summary as described in the Gantt chart at the 25%, 50%, and 100% timing indicates a target to cause change to occur sooner rather than later by having to wait for the full evaluation to be completed which is proposed to take a year. The year is necessary and chosen in order to allow for an appropriate amount of time to pass to see change take place as well as to gather a suitable amount of data (Kirkpatrick and Kirkpatrick, 2006).

Currently, management sees completion rates of the online training modules; however, these rates are not communicated to employees as a part of staff meetings. For all those completing training modules, they are aware of modules being completed, yet there is not a visual tracking method; employees are not able to see individual progression. The current system requires employees to complete a total of 12 modules by the end of the year, as is often the case, many employees wait until the end of the year and then rush through each module in order to complete them on time. Currently management is evaluated on shop floor audits just as completion rate—without the occurrence of a progress note or the value of results achieved.

Recommendations

Some recommendations would include the following:

· An awards program that shows

· Department safety achievement

· Timely completion of modules—by department and individual

· Shop floor audit completion—by department and individual

· Suggestion additions by employees causing positive behavior change

· Safety council attendance

· Monthly or weekly short “meetings” with associates to

· Quarterly staff meetings to communicate progress

· Valued awards system

· To encourage participation from all employees

· To encourage individual ownership in the company safety achievement

· Chart for completed programs to improve visual tracking of training completion and test scores

· Comparison of safety results to other departments within the plant in addition to other plants so as to allow for progress to be shown to employees as a part of their monthly meetings

References

ArcelorMittal: Who we are. (n.d.). Retrieved September 2015, from ArcelorMittal: http://corporate.arcelormittal.com/

Kirkpatrick, D.L. & Kirkpatrick, J.D. (2006). Evaluating training programs (3rd ed.). San Francisco: Berrett-Koehler Publishers, Inc.

Appendix

Appendix A

Process Description

Figure A1. Gantt chart descriptor of process timeline

Table A1. Associated table describing tasks and details of Gantt chart

Appendix B

Level 1 example survey

This survey would serve as a recommendation for a template of a Level 1 evaluation to be completed at the end of each module.

Reaction measurement-survey

Module 1: Class Content

This survey is anonymous; while it will be recorded that you have completed the survey, your name will not be associated with any answers. Please answer honestly as this will help us to improve the program as necessary. Select the appropriate number below each question that most closely corresponds to your reaction to the course. The scale is demonstrated below.

1 = Strongly Disagree 2 = Disagree 3 = Neutral 4 = Agree 5 = Strongly Agree

1)The subject matter was covered well and in depth.

1. Strongly Disagree

2. Disagree

3. Neutral

4. Agree

5. Strongly Agree

2)The material was presented in a clear concise method.

1. Strongly Disagree

2. Disagree

3. Neutral

4. Agree

5. Strongly Agree

3)The subject material was interesting

1. Strongly Disagree

2. Disagree

3. Neutral

4. Agree

5. Strongly Agree

4)The course was useful to my position or future position

1. Strongly Disagree

2. Disagree

3. Neutral

4. Agree

5. Strongly Agree

5)The course material covered the stated objectives.

1. Strongly Disagree

2. Disagree

3. Neutral

4. Agree

5. Strongly Agree

6)The course was organized and cohesive, allowing for me to learn the material adequately.

1. Strongly Disagree

2. Disagree

3. Neutral

4. Agree

5. Strongly Agree

7)The exams covered material learned in the modules.

1. Strongly Disagree

2. Disagree

3. Neutral

4. Agree

5. Strongly Agree

8)The exam questions tested my learning of the new material.

1. Strongly Disagree

2. Disagree

3. Neutral

4. Agree

5. Strongly Agree

9)The course was easy to navigate.

1. Strongly Disagree

2. Disagree

3. Neutral

4. Agree

5. Strongly Agree

10) Please offer any comments you may have regarding the course content, instructor, and relevance to your job below.

Appendix C

Figures C1-C5. Sample of safety module progression from training, to post-test, score, and certificate prompt

Appendix D

Figure D1. Shop floor audit requirements based on management position

Figure D2. Online software managing program to monitor shop floor audit completion

Appendix E

Figure 4. Table of Results that will change monthly as data is collected