Training Evaluation

Karishma DhagePGDM – C

Roll No - 131

Training Evaluation

“Any attempt to obtain information (feedback) on the effects of training program and to assess the value of training in the light of that information for improving

further training” – By Hamblin (1970)

Need for Evaluation ?

• Cost Benefit Returns from Training Investment (Major stakeholders like Training Department, Senior Manager, Sponsors, Clients, Trainees)

• Enable Improvements in the assessment of Training Needs (check suitability & feasibility of the objectives set for training)

• Self-correcting feedback (evaluation will help decide area of improvement)

• Feedback on the performance of the Trainees (aims at improving knowledge & attitude of the trainee)

Trading Cycle

Identification of Training Needs

Develop Training Objectives

Measure Training Results

Conduct Training Programme

Design Training Curriculum

Design/Select Training Methods

Design Training Methodology

Models of Evaluation

• Hamblin Model of Evaluation

• Kirkpatrick’s Design of Evaluation

• Warr’s Framework of Evaluation

• Virmani & Premil’a Model of Evaluation

• Peter Bramely’s Model of Evaluation

Hamblin Model of Evaluation

• We can evaluate at any of the 4 levels namely: Reaction, Learning, Job Behavior & Functioning

• Objectives of the training should be set at each of the 4 levels

• According to Hamblin, to be effective, any scheme of evaluation has to be tailor made to fit the needs of the users of training

• These approaches are possible for external training as well as in company training

Approaches to Evaluating External & In–Company Training

Ultimate Value Approach

Trainee Centered Approach

Trainee Centered Approach

Client Centered Approach

Trainee Centered A

Trainee Centered A

Cost Benefit Criteria

Organization Development

criteria

Job Behavior Criteria

Learning Criteria

Reach criteriaTransfer Criteria

In Company Training External Training

Kirkpatricks Design of Evaluation

Warr’s Framework of Evaluation

• Context Evaluation (C) : Obtaining & using information about the current operational context, that is, about individual differences & organisational deficiencies

• Input Evaluation (I) : determining & using facts & opinion about the available human & material training resources in order to choose between alternative training methods

• Process Evaluation (P) : monitoring the training as it is in progress. This involves continuous examination of administrative arrangements & feedback from trainees

• Outcome Evaluation (O) : Measuring the consequences of training

3 levels of outcome evaluation has been distinguished

1. Immediate Outcome : the changes in trainees knowledge, skills & attitudes which can be identified immediately after the completion of training

2. Intermediate Outcome : the changes in trainee’s actual work behavior which result from training assessment, involves monitoring performance on the job

3. Long Term Outcome : the changes in the functioning of part or all of the organization which have resulted from changes in the work behavior initiated through training

Warr’s Framework of Evaluation contd

Vermani & Premila’s Model of Evaluation

3 STAGES

1. Pre-Training Evaluation (Period before training during which trainees have expectations from the course)

2. Context & Input Evaluation ( Teaching & Learning stage)

3. Post-Training Evaluation (the stage after training, where trainee’s are supposed to integrate training with their job performance

Peter Bramely’s Model of Evaluation

• If the intention is to improve performance by changing behavior, some evaluation should be carried out before a learning activity is designed & run

• Knowledge & theory may be necessary but, if behavioral change is expected, some practice of the type of behavior expected will be required

Evaluation Before Designing Learning Event

• Evaluation during a learning activity can be a useful source of information with which we can improve the quality of the event

• A focus on the utility of the learning & action planning for its application in the workplace will reinforce the link between objectives of the event & organizational requirements

Peter Bramely’s Model of Evaluation contd

Evaluation During the Event

• Most difficult stage in the art of evaluation• After the event, analysis would measure changes in

effectiveness at the 1. Organizational Level2. Team Level3. Individual Level• Change in Behavior (to assess whether change has

taken place)• Change in Learning

Evaluation After the Event

Peter Bramely’s Model of Evaluation contd

David Reay’s Approach to Evaluation

STAGE 1 – THE TRAIL PHASE

• Early Tentative phase of development.

1. Development Stage – involves an informal trail of a sample set of materials, exercises or activities

2. Pilot Testing - more refined operation & tells the evaluation whether the training being evaluated is suitable for the target population

STAGE 2 – THE ONGOING PHASE

1. Validation is used to measure the effectiveness of a whole training design

2. Formative evaluation is the monitoring of the effectiveness of different aspects of training on a continuous basis with a view to modification

David Reay’s Approach to Evaluation contd

STAGE 3 – THE FINAL PHASE• Seeks answers to the following questions1. Were the training needs properly identified2. Were the learning objectives relevant3. Were the performance standards identified4. Were the right priorities established5. Did the training meet the objectives set for it6. How did the learners feel about the training7. Were people prepared for training8. Was the training pitched at the right level9. Was the trainee able to transfer the training to do the

job10.Did the training result in some benefit to the

organization

David Reay’s Approach to Evaluation contd

Pre Training Evaluation

IDENTIFICATION OF TRAINING NEEDS• Trainer carries out Training needs Survey• Uses semi structured interviews• Constructs & administers diagnostic questionnaires

EVALUATE PERFORMANCE STANDARDS• Directly linked to training needs• Performance standards identified as KRA’s focus on

Productivity, profit, Achieving deadlines, Machine Down time, coping with external pressure etc

• Method of evaluation would include detailed description of each behavior & method of measuring quality of behavior

EVALUATE TRAINING OBJECTIVES• To assess whether there is a goal congruence

between the trainee’s needs & training objectives

EVALUATE TRAINEE’S PROFILE• Trainee’s level of knowledge, attitude & skill prior

& post training• Not all but some measures of pre training

knowledge, skill & attitudes is desirable to compare them with post training performance

Pre Training Evaluation contd.

INPUT EVALUATION• Committee approach, brainstorming session

would help in situation where feedback on the subject is already available

• Helps to establish the validity of the training design

EVALUATION DURING THE TRAINING• Both evaluation & subsequent modification need

careful handling• Some methods are examined below

Pre Training Evaluation contd.

1. Observation- trainer’s perceptiveness & observation skills cannot be discounted2. Behavior Analysis- relevant for imbibing certain skills & attitudes such as team orientation & attitudes towards learners3. Course Audits- can be conducted at the end of the day or midway through the course4. Session Assessment- done by using semantic differential scale, 3 point or 5 point & trainee is supposed to only tick mark

Pre Training Evaluation contd.

REACTION EVALUATION• Useful for restructuring training programmes, so

long as it is not confused with an evaluation of learning/job behavior

• Learning’s divided into 3 sections

1. Knowledge learning2. Skill learning3. Attitude learning

Post Training Evaluation

EVALUATING THE INDIVIDUAL TRAINER• When responsibilities are grouped under a

common accountability, measurement of performance against accountability becomes the first means of evaluation

• Impact of trainers activities on solving human performance provides means of evaluation

• Elements of the reputation of the trainer includes comments by line manager about the trainer or the results of training

• External reputation is a measure of trainer performance

Evaluating training staff performance

EVALUATING THE TOTAL STAFF1. Accomplishment of departmental objectives 2. Economic accomplishment of the department.

Training should not cost an organization anything. Results of its activities should be worth more than the cost of staffing & running it

3. How effectively the department uses its resources is a third basis of evaluation. One method of measuring this is to have a system for establishing priorities of departmental activities

4. The department’s reputation & the budget request trend can be evaluated

Evaluating training staff performance

•Modify training to meet learner’s needs by altering exercises, changing training methods & reorganizing the sequence of topics•Document any changes you make & explain these changes to the instructional designers•Use the end of course evaluation to generate ideas for making the training better•Keep in mind that most forms of training evaluations are subjective. Hence, it will be important for you to look at the overall picture & watch trends rather than get bogged down with feedback from one source

Evaluation of the effectiveness of Training

•It helps to justify the existence of the training department, showing the unique value added by the training function to the organization•Second reason is to make a “go” or “no go” decision regarding continuation of a training programme in the future•To gather information to make programme improvements•To meet the unique needs of different audiences

Evaluation Reporting Strategy

If in written form it should be organized as follows1. Cover Page: this provides the title of the report, the date,

the company name, and your contact information2. Table of Contents: this provides title of all section & sub

section, with page numbers3. Executive Summary: this is a one to two page summary of

the report that is aimed at busy senior managers who want to skim a short summary & get all the key facts

4. Evaluation Report: provides specific & detailed information about the training & the learner’s performance. It contains a brief background description of the training, purpose of the evaluation, key question answered by the study, any assumptions/limitations on the study, methods used to carry out the evaluation & the results

5. Conclusion & Recommendation: draws conclusion from evaluation & offers recommendation for improving training

Format of the Evaluation Report

Be sure to include in any report such information as:•The needs that the training was intended to meet•The training objectives for the course•Results of end of course reaction surveys, if they were administered•Results of tests taken by learner’s in aggregate or individual form, if tests were administered•Results of on the job follow up studies to assess transfer of training to the job, if they have been conducted•Impact of training outcome study results, if they have been conducted

Contents of the Evaluation Report

•Most difficult evaluating process of all in the review of training•Problem lies in the ability to quantify changes before & after training•It is essential to have evidence of changes in approach, style & adaptability as a result of the learning process•Changes in the affective area of behavior are concerned with experiences of individual persons learning to act in their roles in the concern•A caring company that trains its staff notices the outward & visible signs & is not afraid to comment on non-quantifiable aspects of performance such as motivation, attitudes to work, team building & morale as aresult of its commitment to training

Evaluating attitude change

•During observation information can be difficult to interpret

•During surveys & questionnaires they are more impersonal for participants, & usually not good for getting quotes in participant’s own words

Training Evaluation Challenges

Bibliography

• Enriching Human Capital through T & D by P L Rao

• www.citehr.com

• www. Uwex.edu