What is training evaluation? Methods to evaluate training. Purpose of evaluation Why training evaluation? Benefits of evaluation. Evaluation process. Evaluation Methods 1.Expert and Peer Review 2.Quality Review 3.One-to-one observation test 4.Pilot Test

Evaluation Conclusion Example and final Review.

Assessing the effectiveness of the training program in terms of the benefits to the trainees and the company. It is a process of collecting outcomes to determine if

the training program was effective from whom, what, when, and how information should be collected

It is a process of establishing a worth of something.

The ‘worth’, which means the value, merit or excellence of the thing

State of mind, rather than a set of techniques

Why evaluate training?

Formative evaluation –Formative evaluation – evaluation conducted to improve the training process.

Summative evaluation –Summative evaluation – evaluation conducted to determine the extent to which trainees have changed as a result of participating in the training program.

Feedback - on the effectiveness of the training activities.

Control - over the provision of training.

Intervention - into the organizational processes that affect training

Companies are investing millions of dollars in training programs to help gain a competitive advantage.

Training investment is increasing because learning creates knowledge which differentiates between those companies and employees who are successful and

those who are not.

Because companies have made large dollar investments in training and education and view training as a strategy to be successful, they expect the outcomes or benefits related to training to be measurable.

Improved quality of training activities Improved ability of the trainers to relate inputs to outputs Better discrimination of training activities between those

that are worthy of support and those that should be dropped

Better integration of training offered and on-the job development

Better co-operation between trainers and line-managers in the development of staff

Evidence of the contribution that training and development are making to the organization

Cognitive Learning Skills Learning Affect ‘Objective’ results

quantitative, experimentalobjectivebroadnumerical measurementsquestionnaires on large groupsdata collection, loggingchecklists

qualitative, richsubjectivein-depth qualitativeinterviewspersonal viewsdescriptivenaturalistic

Review by peers and experts on content, pedagogy and interface

Editorial review of the whole course for inaccuracies, omissions, inconsistencies

Provide simple methods for busy experts: give the background, analysis, and a checklist and leave them to do it

Or talk-through with the evaluator using the material as if he/she were a novice - a cooperative evaluation

systematic exercise on a complete draft before pilot use, using checklists

(also for ready made packages, for selection) ready made checklists exist

e.g. Alessi & Trollip, MEDA modify these to create an appropriate evaluation tool score package against list, y/n or numbers (see SB’s web site documents for checklists)

developer observes learners using it record learner’s activity on paper, user log, video... students ‘think aloud’ their activities, which can also be

recorded and then compared to expectation evaluator is a learner who has taken a similar course

already time-consuming but valuable once substantial material is

available

a small group of experienced students, in the environment of use

complete the course to identify omissions, inappropriate examples, poor questions, weak text comment on content and usability

done in student pairs or singly collect data by interview, questionnaires, software logs,

learning assessment, manual record sheets, video

use by a representative sample of target group, including a range of ability, whose role is user rather than evaluator

instruments should be non-intrusive observations, recording, automatic logs diaries, record sheet of progress.

interviews, questionnaires afterwards for user views and assess learning achieved.

similar to pilot tests but with real learners so only a short explanation of the evaluation being done and less data collected per user.

• Uirle-Patrick Approach.

• Bells Systems Approach.

• CIRO Approach.

• Sorologa Institute Approach.

• IBM Approach.

• Xerox Corporation Approach.

• CIPP Model.

• KIRKE PATRICK MODEL FOR EVALUATION

LEVEL QUESTIONS

REACTION Were the participants pleased with the program?

LEARNING What did the participants learn in the program?

BEHAVIOR Did the participants change their behavior based on what was learnt?

RESULTS Did the change in plan positively effect the organization?

REACTION OUTCOMES:

Contents, materials, method, activity.

CAPABILITY OUTCOMES:

Outcomes against participant’s expectation.

APPLICATION OUTCOME:

Application of training in work setting.

WORTH OUTCOME: Organizations’ benefits in terms of money, efforts, time and resources.

• Developed by Warr, Bird and Rachal.

• It gives Evaluation in terms of – CONTEXT,– INPUT,– REACTION, – OUTCOME.

IMMEDIATE OBJECTIVE:

New knowledge, skills and attitudes required to reach intermediate objective.

INTERMEDIATE OBJECTIVE:

Change in employees work balance necessary for ultimate objective.

ULTIMATE OBJECTIVE:

Particular deficiency in the organization that will be eliminated.

Following relevant questions are considered:• What are the relative merits of the different

HRD methods?• Is it feasible for an outside organization to be

more efficient at conducting the programme?• Should it be developed with the internal

resources?• Should the line managers be involved?• How much time is available for HRD?• What results were achieved when a similar

programme was conducted in the past?

• Includes subjective reports of the participants about the whole pogramme.

STEPS:

• Defining trained objectives.

• Selecting and constructing some measures of those objectives.

• Making the measurements in the appropriate time.

• Assessing the results and using them to improve future programme.

• This approach highlights that the evaluation of the programme should judge:– The Satisfaction,– Learning Change,– Change in behavior,– organizational change.

REACTION: A satisfaction rating that asks the trainees how valuable they found the program?

TESTING: Pre- and Post-programme measurements in terms of knowledge and skills improvement.

APPLICATION: Extent to which skills applied on the job and the results achieved.

BUSINESS RESULTS:

What IBM expected from the programme in the form of a return that?

ENTRY CAPABILITY: Prerequisites for the program evaluated.

END OF COURSE PERFORMANCE:

Whether trainees achieved the desired outcomes?

MASELEY JOB PERFORMANCE:

Whether trainees exhibit mastery performance under normal job condition?

ORGANIZATIONAL PERFORMANCE:

Which programme participants meet or exceed the organizational targets?

CONTEXT EVALUATION:

Needs analysis, this assists in forming goals.

INPUT EVALUATION: Policies, budgets, schedules, proposals and procedures aids in programme planning.

PROCESS EVALUATION:

Reaction sheets, rating scales and analysis of existing records- guide’s implementation.

PRODUCT EVALUATION:

Measures and interprets the attainment of objective- helps in recycling decisions.

Brainstorming is a group creativity technique designed to generate a large number of ideas for the solution to a problem. Although brainstorming has become a popular group technique, researchers have generally failed to find evidence of its effectiveness for enhancing either quantity or quality of ideas generated. brainstorming groups are little more effective than other types of groups, and they are actually less effective than individuals working independently.

34

Advantages

➔ Listening exercise that allows creative thinking for new ideas.

➔ Encourages full participation because all ideas are equally recorded.

➔ Draws on group's knowledge and experience.

➔ Spirit of cooperation is created.

➔ One idea can spark off other ideas.

35

Disadvantages

➔ Can be unfocused.

➔ Needs to be limited to 5-7 minutes.

➔ Students may have difficulty getting away from known reality.

➔ If not managed well, criticism and negative evaluation may occur.

➔ Value to students depends in part on their maturity level.

36