.

THE TAXONOMY OF TRAINING AND DEVELOPMENT OUTCOMES (TOTADO)

EVALUATION FRAMEWORK

Dr Kamal BirdiInstitute of Work Psychology

Management SchoolUniversity of Sheffield

Sheffield S10 2TNUnited Kingdom

Email: K.Birdi@Sheffield.ac.ukTel: +44 (0)114 222 3288

This document contains a description of the TOTADO training and development evaluation framework and two tools to help with its implementation and data collection.

1. A paper summarising the TOTADO framework

2. A TOTADO evaluation strategy planning template

3. A TOTADO evaluation interview schedule

1

.

THE TAXONOMY OF TRAINING AND DEVELOPMENT OUTCOMES (TOTADO): A NEW MODEL OF TRAINING EVALUATION

Kamal BirdiInstitute of Work Psychology, University of Sheffield

Paper presented at the Annual BPS Division of Occupational Psychology Conference, 13th-15th

January 2010, Brighton, UK.

INTRODUCTIONThe effectiveness of training and development activities can be assessed by the extent to which those

activities produce desirable outcomes and reduce undesirable ones. Despite organisations expending a

great deal of effort setting up training and development programmes, comparatively less attention is

paid to systematically evaluating their effectiveness (Sadler-Smith, 2006). This lack of assessment

may be partly due to the extra cost and effort involved but it also seems that many Human Resource

practitioners are unaware of how and what to evaluate (Bates, 2004). The objectives of this paper are

therefore to briefly review and critique past models of evaluation and then offer a new model of

evaluation (the Taxonomy of Training and Development Outcomes – TOTADO) which addresses

these shortcomings.

A review of the literature indicates that three types of approaches can be described when considering

how outcome evaluation has been previously addressed. First, the predominant traditional training

evaluation approach (e.g. Kirkpatrick, 1959; Warr et al., 1970) offers a practical framework for

assessing outcomes in terms of trainee reactions, learning of knowledge and skills, on-the-job

behaviour and organisational performance. Although popular in organisations, these models tend to

be theoretically and practically vague in their specification of different types of learning outcomes,

work behaviours and organisational performance criteria (Phillips & Phillips, 2001). A second

approach is illustrated by Kraiger et al. (1993), who have proposed a classification scheme for

individual learning outcomes (cognitive, skill-based or affective) based on psychological theory. This

framework is useful for defining the specific types of criteria which indicate learning and it allows

theoretical rationales to be developed when choosing predictors of learning outcomes. However, this

second approach has rarely been used in subsequent research studies (Beech & Leather, 2006) and,

since it only focuses on learning outcomes, it also needs to be incorporated into the wider frameworks

offered by Kirkpatrick (1959) and Warr et al. (1970). Finally, the employee development literature

(e.g. Benson et al., 2004; Maurer et al., 2002) has empirically assessed experienced or perceived

development outcomes from different types of learning and development activities. The findings from

these studies indicates that such activities can lead to changes in a variety of areas not considered by

traditional training evaluation approaches, such as changing employees’ status, relationship to

colleagues or even health. A broader perspective on types of outcomes beyond individual learning of

knowledge and skills, work behaviour and organisational performance is therefore needed if training

and development effectiveness is to be accurately assessed. The Taxonomy of Training and

Development Outcomes (TOTADO) attempts to do this (see Figure 1) and will be outlined.

2

.

DESCRIPTION OF THE TAXONOMY OF TRAINING AND DEVELOPMENT OUTCOMES

TOTADO integrates and elaborates on previous evaluation approaches to define a taxonomy of

outcomes that allows us to capture and describe the diverse range of outcomes experienced from

participation in both training and other development activities. The research described earlier revealed

that although the impact of development activities can be assessed at differing levels, much of the

work has concentrated on the individual. Kirkpatrick’s model of evaluation has been mainly used to

consider the individual level of analysis from Level 1 to Level 3; Kraiger et al.’s (1993) taxonomy

considers solely individual level learning outcomes; most studies from the employee development

literature have used surveys to assess the impact of training and development activities on their

participants. Consequently, the outcomes described in the previous section refer predominantly to the

individual, whereas other levels of outcomes are equally important. Within the TOTADO framework,

it is therefore said that outcomes can be measured at four basic levels: individual, team (or work

group), organisational and societal.

Figure 1. The Taxonomy of Training and Development Outcomes (TOTADO) Framework.

1. Individual-level outcomes

3

.

With TOTADO we extend previous evaluation models by elaborating five different types of

individual-level outcomes where the trainee is the unit of analysis: Affective (e.g. participant reactions

to training, motivation, self-efficacy, attitudes, mental well-being); Cognitive (verbal knowledge,

knowledge representation); Behavioural (off-the job task performance, on-the-job task performance);

Physical (e.g. health, fitness, injuries): and Instrumental (events, actions or status changes resulting

from participation in training and development activities that are work intrinsic (increased job control),

work extrinsic (pay rise, promotion) or work relational (forming new relationships or networks)).

2. Team-level outcomes

With the increasing implementation of teamworking in the workplace, there has been a concurrent rise

in team training and development activities (Wilson et al., 2007). For example, many organisations

send work groups on external “outward-bound” courses where team members are required to co-

operate and work closely to achieve given objectives. Evidence to date of the impact of such activities

tends to be anecdotal, unsystematic and, in particular, ignores whether there is any change in team

functioning back in the workplace (Salas et al., 2004). Hence, we need to consider using the team or

work group as the unit of analysis for assessing the impact of these types of opportunities. The

majority of the outcomes which will be described for individuals can also be applied by aggregation to

the team context, although there is more focus on measuring interpersonal behaviours and group

cognition and affect. The four dimensions of impact here where the team or group is the unit of

analysis are therefore: Affective (e.g. changes in average team identity or trust); Cognitive (verbal

knowledge, knowledge representation in terms of shared cognition); Behavioural (team task processes,

intra-team processes); and Instrumental (events, actions or status changes for the team as a whole

such as increased team autonomy or gaining team bonuses).

3. Organisational-level outcomes

The next level takes the unit of analysis to be the organisation, or major functional components of it

(e.g. departments) and outcomes relate to those directly involved with the organization in a

transactional format (e.g. employees, customers, shareholders). Although individual- or team-level

outcomes can be aggregated to provide organisational-level outcomes (e.g. proportion of company

employees with a certain technical qualification), some variables can only be measured at the

organisational level (e.g. company profitability, share values). This is the domain that Kirkpatrick

(1959) refers to as Level 4 or Business Results, although he remains vague in terms of what aspects

should be included in this level. It would be conceptually very helpful to identify key categories into

which we can place organisational-level outcomes, much as was done with the individual and team

levels. The organisational effectiveness literature highlights the complex nature of measuring

performance but does attempt to provide some broad criteria. We therefore draw on the work of

management researchers (Cameron, 1980; Kaplan & Norton, 1996; and Richard et al., 2009) to specify

four dimensions of organisational performance outcomes: Financial (e.g. turnover, profit, share price);

Outputs (quantity, quality, variety of components, products or services); Processes (how well the

4

.

organisation functions e.g. time to complete tasks, stoppages, communication system efficiency); and

Resources (human and non-human).

4. Societal-level outcomes

The final level is rarely considered but relates to the impact of training beyond the organisational level

where the area or group outside the organisation is the unit of analysis (Watkins et al, 1998). The

majority of past models focus solely on impacts within the organisation, yet a wider perspective is

sometimes needed. For example, training employees in carbon-friendly practices within a

manufacturing company may translate to decreased waste and pollution levels for surrounding

communities. At this level, we can therefore also assess the impact of training on a sectoral, regional

or even national bases. For instance, does training all Job Centre Plus staff in coaching techniques to

help the long-term unemployed significantly improve regional employment rates? The five broad

dimensions of societal impact we can consider are therefore: Economic (e.g. £ investment in region);

Health and Welfare (e.g. mortality rates from heart disease); Educational (e.g. level of qualifications of

local populations); Law and Order (e.g. local crime rates): and Environmental (e.g. pollution levels).

DISCUSSION

The TOTADO framework therefore presents an approach which integrates and enhances past models

of evaluation by presenting multiple levels of impact and specifying within each level the

key dimensions of outcome change. Key features worth noting include:

• It acknowledges that the effectiveness of training and development participation should be

assessed on a number of levels and using a range of relevant criteria within each level. Conceptually,

it integrates the focused view of the traditional Kirkpatrick training evaluation approach with the wider

evaluation perspective offered by other literatures. By using the taxonomy in the early stages of

developing an evaluation strategy, researchers and practitioners can become fully aware of the variety

of salient outcome criteria that may need to be measured.

• The taxonomy clearly distinguishes between individual, team, organisational and societal

measures of organisational effectiveness. Approaches such as Kirkpatrick’s have tended to remain

vague about measures of individual work behaviour and organisational results and seemed to have

ignored team-level analysis altogether.

• The nature of the taxonomy also allows detailed and comparative examination of the types of

outcomes produced by different learning activities. For instance, how much impact does participation

in formal training compared to work-based development activities have on employees’ general job

satisfaction? In situations where several studies have examined the same development activity but

assessed different types of outcomes, the taxonomy provides a framework to integrate the findings in

order to provide a more complete overview of the impact of that activity. It would also highlight

which types of outcomes had not yet been assessed and hence drive future research to address those

gaps.

5

.

• Since the framework is a taxonomy, it does not make any claims of causality between outcomes.

Although Kirkpatrick (1996) claims that his framework is solely a taxonomy, it has widely been

interpreted as a training effectiveness model which specifies causal linkages between individual and

organisational outcomes (Alliger et al., 1997). If we are truly to identify the factors that influence

training outcomes, just considering other outcomes is clearly insufficient. Previous research has

consistently shown that a wide range of individual, training-related and organisational factors can also

have an impact on different types of outcomes (Noe, 2008). Such factors need to be included in any

good training effectiveness model. The evaluation taxonomy presented here, however, does provide

the building blocks for creating models which propose causal linkages between different types of

outcomes and other types of variables.

• Finally, the evaluation taxonomy provides a common format for assessing learning and

development impact from a diverse range of theoretical approaches. For example, cognitive scientists

conducting laboratory studies on learning complex tasks can be regarded as examining learning

effectiveness in terms of individual-level cognitive or behavioural outcomes that are measured in an

off-the-job context. Field studies conducted by occupational psychologists examining the transfer of

training to the workplace may focus on individual work behavioural outcomes. Team researchers can

be considered as assessing the impact of development activities in terms of changes in team

behaviours such as communication and co-operation. Organisational researchers go beyond individual

and team levels and relate the amount of training offered by companies to organisational-level goal-

directed outcomes such as sales or turnover (Wright et al., 2005). The framework should therefore be

flexible enough to cater for a wide variety of paradigms where learning effectiveness is to be assessed.

The presentation will illustrate the utility of TOTADO by providing examples from research

conducted by the author including:

• An evaluation study which shows how three types of creativity training programme run within a

government department can generate different levels of individual-level affective, cognitive and

behavioural outcomes for participants.

• A study of 500 UK organisations showing the relationship between organisations’ use of

individual training and development practices and different organisational performance outcomes.

REFERENCES

Alliger, G. et al. (1997). A meta-analysis of the relations among training criteria. Personnel

Psychology, 50, 341-358.

Bates, R. (2004). A critical analysis of evaluation practice: the Kirkpatrick model and the

principle of beneficence. Evaluation and Program Planning, 27, 371-347.

6

.

Beech, B., & Leather, P. (2006). Workplace violence in the health care sector: A review of

staff training and integration of training evaluation models. Aggression and Violent

Behavior, 11(1), 27-43.

Benson, G. S., et al. (2004). You paid for the skills, now keep them: Tuition reimbursement

and voluntary turnover. Academy of Management Journal, 47(3), 315-331.

Cameron, K. (1980). Critical questions in assessing organizational effectiveness.

Organizational Dynamics, Autumn, 66-80.

Kaplan, R., & Norton, D. (1996). Linking the balanced scorecard to strategy. California

Management Review, 39, 53-79.

Kirkpatrick, D. (1959). Techniques for evaluating training programs. Journal for the

American Society for Training and Development, 13, 3-9.

Kraiger, K., et al. (1993). Application of cognitive, skill-based and affective theories of

learning outcomes to new methods of training evaluation. Journal of Applied

Psychology, 78(2), 311-328.

Maurer, T. e al. (2002). Perceived beneficiary of employee development activity:A three-

dimensional social exchange model. . Academy of Management Review, 27(3), 432-

444.

Noe, R. A. (2008). Employee Training and Development (Fourth Edition). New York:

McGraw-Hill.

Phillips, P. & Phillips, J. (2001). Symposium on the Evaluation of Training: Editorial.

International Journal of Training and Development, 5(4), 240-247.

Richard, P. et al. (2009). Measuring Organizational Performance: Towards Methodological

Best Practice. Journal of Management, 35(3), 718-804.

Sadler-Smith, E. (2006). Learning and Development for Managers. Oxford: Blackwell

Publishing.

Salas, E. et al. (2004). 25 years of team effectiveness in organizations: Research themes and

emerging needs. In C. Cooper & I. Robertson (Eds.), International Review of

Industrial and Organizational Psychology (Vol. 19, pp. 47-91). Chichester: John

Wiley & Sons.

Warr, P. et al. (1970). Evaluation of management training. London: Gower Press.

Watkins, R et al. (1998). Kirkpatrick plus: Evaluation and continuous improvement with a

community focus. Educational Technology, Research & Development, 46(4), 90-95.

Wilson, J. et al (2007). Group Learning. Academy of Management Review, 32(4), 1041-1059.

Wright, P. et al. (2005). The relationship between HR practices and firm performance:

Examining causal order. Personnel Psychology, 58, 409-446.

7

.

TOTADO EVALUATION STRATEGY FRAMEWORKNAME OF TRAINING / DEVELOPMENT PROGRAMME:

Level Dimension of effectiveness

Type of information Methods of collecting information

Who provides the information

When to collect information

TIME 0 TIME 1 TIME 2 TIME 3

period before the course

at the start of the course

at the end of the course

period after the course

IND

IVID

UA

L Affective Reactions, attitudes, motivation, self-efficacy, mental well-being

Cognitive Verbal Knowledge,Knowledge Representation

Physical Health, Fitness, Injuries

Behavioural Off-the-job, on-the-job task performance

Instrumental Work intrinsic, work extrinsic, work relational, nonwork

TE

AM Affective Shared team attitudes

Cognitive Shared cognition

Behavioural Team performance, team interaction processes

Instrumental Work intrinsic, work extrinsic, work relational,

8

.

NAME OF TRAINING / DEVELOPMENT PROGRAMME:O

RG

AN

ISA

TIO

NA

L Outputs Quantity, Quality, Variety of products / services

Resources Material, Personnel

Processes Efficiency, effectiveness of work systems

Financial Turnover, budget deficits, profit

SOC

IET

AL Economic e.g. £ investment in geographic

region, unemployment rates in area

Health and Welfare e.g. no. of deaths from heart disease, alcohol-related illnesses in local population

Educational e.g. level of qualifications of local population, % social inclusion of minorities

Law and order no. of robberies in the area, % reduction in drug crime

Environmental pollution and waste levels in the region

© K.Birdi, University of Sheffield 2006

Level Dimension of effectiveness

Type of information Methods of collecting information

Who provides the information

When to collect information

TIME 0 TIME 1 TIME 2 TIME 3

period before the course

at the start of the course

at the end of the course

period after the course

9

.

TOTADO INTERVIEW SCHEDULE

“In the following interview, I shall be asking you to describe any changes that you think have arisen because of programme X.”

A. INDIVIDUAL OUTCOMES

GENERAL LEAD INAs a result of taking part in X, how do you think you have changed?

AFFECTIVEHow has taking part in the X changed the way you FEEL about things?

COGNITIVEWhat type of things have you LEARNED as a result of taking part in X?

PHYSICALHow has taking part in the X changed your LEVELS OF HEALTH AND FITNESS?

BEHAVIOURAL What can you DO now that you couldn’t do before taking part in X?How has taking part in X changed your JOB PERFORMANCE?

INSTRUMENTALHas taking part in X led to any changes in the type of work you do?Has taking part in X led to any changes in your job status or conditions of employment?Has taking part in X helped you form new relationships at work or changed existing relationships?Has taking part in X led to any other changes in your work that we haven’t discussed yet?Has taking part in X led to any changes for you outside work?

IDENTIFYING BARRIERS TO CHANGE IN INDIVIDUAL OUTCOMESHas anything in particular stopped or hindered you changing personally the way you wanted after the training / taking part in Programme X?

Is there anything that could be changed about Programme X, your organisation or other aspects that could help you achieve these desired changes?

10

.

B. TEAM / GROUP OUTCOMES

GENERAL LEAD INAs a result of taking part in the X programme, how do you think your team has changed?

AFFECTIVEHow has taking part in the X changed the way your team FEELS about things?

COGNITIVEWhat type of things has your team LEARNED as a result of taking part in X?

BEHAVIOURAL What can your team DO now that you couldn’t do before taking part in X?How has taking part in X changed your team’s WORK PERFORMANCE?

INSTRUMENTALHas taking part in X led to any changes in the type of work your team does?Has taking part in X led to any other changes in the type of work your team does?Has taking part in X helped your team form new relationships at work or changed existing relationships?Has taking part in X led to any other changes in your team’s work that we haven’t discussed yet?

IDENTIFYING BARRIERS TO CHANGE IN TEAM/GROUP OUTCOMESHas anything in particular stopped or hindered your team or teams changing the way that was wanted after the training / taking part in Programme X?

Is there anything that could be changed about Programme X, your organisation or other aspects that could help you achieve these desired changes?

11

.

C. ORGANISATIONAL OUTCOMES

GENERAL LEAD INAs a result of running the X programme, how do you think your organisation has changed?

OUTPUTSHas running X led to any changes in your organisation’s outputs e.g. its productivity, quality levels, variety of products/sevices offered etc.?

RESOURCESHas running X led to any changes in your organisation’s employees e.g. their performance, morale, absenteeism?Has running X led to any changes in your organisation’s material resources e.g. reductions in wastage?

PROCESSES / OPERATIONSHas running X led to any changes in your organisation’s efficiency or the way work is carried out e.g. speed of production, turnaround on project times?

FINANCIALHas running X led to any changes in your organisation’s financial performance e.g. its profitability, balancing of budgets?

Has taking part in X led to any other changes in your organisation that we haven’t discussed yet?

IDENTIFYING BARRIERS TO CHANGE IN ORGANISATIONAL OUTCOMES

Has anything in particular stopped your department or organisation changing the way that was wanted after the training / taking part in Programme X?

Is there anything that could be changed about Programme X, your organisation or other aspects that could help you achieve these desired changes?

12

.

D. SOCIETAL OUTCOMES

GENERAL LEAD INWhat types of general impact would you say the X programme has had on communities or groups outside the organisation?

ECONOMICWhat types of general impact would you say the X programme has had on the economy in your region ? e.g. £ investment in geographic region, unemployment rates in area

HEALTH AND WELFAREWhat types of general impact would you say the X programme has had on the health or welfare of the community in your region? e.g. no. of deaths from heart disease, alcohol-related illnesses in local population

EDUCATIONALWhat types of general impact would you say the X programme has had on the education levels of the community in your region ? e.g. level of qualifications of local population, % social inclusion of minorities

LAW AND ORDERWhat types of general impact would you say the X programme has had on law and order issues in the community in your region? e.g. no. of robberies in the area, % reduction in drug crime

ENVIRONMENTALWhat types of general impact would you say the X programme has had on the geographical environment of your region? e.g. pollution and waste levels in the region

Has taking part in X had any other impacts on the local or national community that we haven’t discussed yet?

IDENTIFYING BARRIERS TO CHANGE IN SOCIETAL OUTCOMES

Has anything in particular stopped or hindered your organisation’s impact on the community or society in the way you wanted after the training / taking part in Programme X?

Is there anything that could be changed about Programme X, your organisation or other aspects that could help you achieve these desired changes?

13

.

NOTE 1: The above questions can be tailored to different stakeholders (trainees, line

managers, senior managers, customers, regulatory bodies etc) and different types of

programmes.

NOTE 2: Where desired changes have not occurred or undesirable changes have occurred,

then probe for the reasons for this situation (see BARRIERS question). Identifying

barriers to desired change in outcomes can help improve the effectiveness of future

activities.

© K.Birdi, University of Sheffield 2006

14