the effect of performance support and training on performer attitudes

The Effect of Performance Supportand Training on Performer Attitudes

Frank Nguyen

While training has been a proven and heavilyrelied on intervention to impart job-en-abling information to performers (Arthur,

Bennett, Edens, & Bell, 2003; Burke & Day, 1986), itsability to have a positive effect on job performance hasbeen demonstrated to diminish over time. Accordingto a study conducted by the Research Institute ofAmerica, performers retain only 58% of the contentdelivered by training interventions just 33 minutesafter course completion (Snipes, 2005). Retentioncontinues to decline from there. After the secondday, just 33% of the information learned in training isretained. Three weeks later, only 15% can be recalled.Rackham (1979) found that 87% of the learning ac-quired from training is lost within 30 days if it is notsupported by an additional intervention.

One intervention that has been adopted by perfor-mance technologists to provide support is an electro-nic performance support system (EPSS). These systems provide users with"individualized on-line access to the full range of y systems to permit jobperformance" (Gery, 1991, p. 21). In other words, performance supportsystems seek to provide the right information to the right user at the righttime.

Past research has examined the intersection of these two fields. Bastiaens,Nijhof, Streumer, and Abma (1997) explored the effectiveness of differentcombinations of computer-based and paper-based performance supportwith computer-based and instructor-led training. They found that partici-pants preferred paper-based forms of performance support over electronicversions as well as instructor-led training over computer-based training.They found no significant difference on test achievement scores, perfor-mance, or sales results over a one-year period. Mao and Brown (2005)

95

P E R F O R M A N C E I M P R O V E M E N T Q U A R T E R L Y , 2 2 ( 1 ) P P . 9 5 – 1 1 4

& 2009 International Society for Performance Improvement

Published online in Wiley InterScience (www.interscience.wiley.com). DOI: 10.1002/piq.20047

While training has been a provenand heavily relied on intervention toimpart job-enabling information toperformers, its ability to have a positiveeffect on job performance has beendemonstrated to diminish over time.One intervention that has beenadopted by performance technologiststo provide ongoing support is an elec-tronic performance support system(EPSS). The study presented here exam-ined the effect of EPSS and training onuser attitudes. Results revealed thatparticipants receiving only EPSS andthose receiving training and EPSS hadsignificantly higher attitudes than par-ticipants who received only training.Recommendations on how to bestcombine and implement these perfor-mance interventions based on thesedata are discussed.

conducted a study comparing the effect of instructor-led training andperformance support on learning how to use Microsoft Access. They foundthat users provided with an EPSS performed significantly better on anachievement test than those provided with training. They found no sig-nificant difference between the two groups on a procedural task. Nguyen andKlein (2008) examined the effect of training and performance supportindividually and as combined interventions. They reported that participantswho received only EPSS and both training and EPSS together performedsignificantly better on a procedural task than those who received justtraining.

These findings provide some evidence to support the widely held beliefthat implementing a performance support intervention can reduce orperhaps even eliminate the amount of training necessary to address aperformance problem (Chase, 1998; Desmarais, Leclair, Fiset, & Talbi1997; Foster, 1997). However, they do not provide any evidence-basedguidelines to assist performance technologists to determine when a parti-cular problem can be safely addressed by just training, EPSS, or a combina-tion of EPSS and training and what proportion of those two interventionsmay be required.

The research study examined here sought to address this gap byexploring the following research question: What combination of perfor-mance support and training do users prefer? By revealing performers’attitudes toward these two interventions, the psychological factors thatinfluence the use of such systems to support performance should be revealed.

Method

Design and Participants

This research study employed an identical design to the study previouslyreported by Nguyen and Klein (2008). A posttest-only control group designwas used. Participants were randomly assigned to one of three treatmentgroups: training only, EPSS only, or training and EPSS. One group receivedtraining prior to completing the performance task. Another group had accessto an EPSS while completing the task but did not receive any prior training. Athird treatment group received both the pretask training and access to theEPSS. The primary dependent measure was user attitudes.

Seventy-eight employees from multiple companies completed the study.Some participants were identified by their direct managers at variouscompanies. Additional volunteers for the study were also solicited fromlocal chapters of the American Society for Training and Development(ASTD) and the International Society for Performance Improvement (ISPI).Participants involved in the study represented a broad array of educationalbackgrounds: 39 had a master’s degree, 28 held a bachelor’s degree, 6 hadobtained a doctoral degree, 3 were high school graduates, and 2 held anassociate degree. The participants represented a diverse range of job roles: 33were involved in the education and training industry, 16 identified them-

96 DOI: 10.1002/piq Performance Improvement Quarterly

selves as software developers or information technology professionals, 15were in human resources, 6 were involved in manufacturing, 3 worked incustomer service, and 5 worked in other professions.

Materials

The materials in this study were a tax software application, a Web-basedtraining course, a performance support system, a task scenario, a posttaskattitude survey, and an interview questionnaire.

Tax Software Application. A Web-based software application based on acorporate tax return form was used by all participants in the study. As part ofthe process to submit a tax return, companies are required to submit dataregarding revenue, profit, costs, and other financial information. While thesedata are typically recorded on paper-based forms, participants in this studywere asked to record data and calculations into an online tax form asillustrated in Figure 1.

Web-Based Training Course. In addition to the tax form, a Web-basedtraining course was used to teach processes, procedures, and principles thatare required as part of the corporate tax preparation task. If the participantswere assigned to the training-only or training and EPSS groups, the taxsoftware application required them to complete the Web-based trainingactivity before attempting the corporate tax performance task.

The Web-based training course consisted of 9 introductory screens, 49information screens, 24 practice screens, and 5 concluding screens. In total,

FIGURE 1. SOFTWARE APPLICATION

Volume 22, Number 1 / 2009 DOI: 10.1002/piq 97

the course contained 87 screens and took approximately 1 hour to complete.The course was divided into five modules, each addressing a specific instruc-tional objective. Each module began with an introductory screen informingthe learner of the objective for the module. In addition, this screen referenceda diagram of the corporate tax process, which served as an advance organizerforthe content (see Figure 2).Each line inthe taxform wasaddressed byoneormore instructional screens. Instructional screens included a brief amount ofcontent about tax concepts, rules, and procedures that must be completed intax form and examples where relevant. After the instructional sequence, eachmodule provided scenario-based practice activities, with the exception ofmodule 1, which provided matching and multiple-choice practice activitiesfor factual objectives. All practice activities included appropriate feedback forcorrect responses or remediation feedback for incorrect responses.

Performance Support System. The tax software application was also equippedwith a performance support system for participants in the EPSS-only and thetraining and EPSS treatments. The EPSS is illustrated in Figure 3. The EPSSused was a context-sensitive help system that was found in previous studiesto be an effective method to deliver on-the-job support (Bailey, 2003;Nguyen, Klein, & Sullivan, 2005). The opening screen of the applicationprovided a brief set of instructions demonstrating how to access the supportsystem. Help buttons in the form of a question mark were insertedthroughout the tax software application. When participants clicked on thebuttons, their request was recorded in a database, and a new window openeddisplaying support information associated with the task.

FIGURE 2. WEB-BASED TRAINING COURSE


To avoid any effects due to content differences between the training andEPSS, the content used for the EPSS was derived from the training course.Web-based training courses can be developed into modular, reusablelearning objects. These learning objects are granular components of atraining course, such as individual modules, lessons, screens, practiceactivities, or media elements. These objects can exist independent of theoriginal training course, which then allows them to be accessed as singlelearning offerings or be combined in different ways to create new trainingcourses. These information objects were linked directly to individual helpbuttons embedded in the tax software application. With this approach,identical components from the Web-based training course could be reusedfor performance support purposes. Since the actual learning objects wereidentical between the two treatments, any differences due to the quality of thecontent could be eliminated.

Task Scenario. The task scenario portrayed a realistic issue that a newemployee might face. It included information that a manager mightfurnish to a new finance employee in preparing federal tax submission fora company. Corporate tax preparation was chosen as the basis of the scenariobecause of the complexity of the task.

Figure 4 shows an excerpt of the task scenario text. The first sectionprompted the participant to imagine having recently been hired as a financialanalyst for a small manufacturing company. The second portion containedan e-mail sent to the participant from the imaginary new manager. In the

FIGURE 3. ELECTRONIC PERFORMANCE SUPPORT SYSTEM


e-mail, the manager asked the participant to prepare a tax return for thecompany. To support this task, the e-mail contained detailed financialinformation about income, expenses, and payroll and other companyinformation. The participant used these data and any training and supportinformation to complete the tax return using the tax software application.

Posttask Attitude Survey. An eight-item attitude survey was administeredafter participants completed the task. The survey is shown in Exhibit 1. Thesurvey was divided into three sections: usefulness of training and support,quantity of learning, and satisfaction with the system.

The first section of the survey examined the usefulness of the training andsupport content. The first question asked participants how easy it was to findthe information they needed to perform the task. The second questionexamined whether participants felt training and support content had theappropriate level of detail.

The next section addressed the quantity of learning provided for the task.The third question asked participants if the amount of training provided wassufficient to complete the task, and the fourth question asked if the amount ofhelp during the task was sufficient.

The third set of questions gauged the participants’ satisfaction with thesystem. The fifth question asked participants if they felt confident complet-ing the task. The sixth examined whether the participants would like to usesoftware applications such as the one demonstrated in the study in the future,and the seventh asked if they would recommend one for their organization.

The final question prompted participants to rate the amount of time theyspent learning how to perform the task as about right, too much, or too little.The Cronbach’s alpha reliability coefficient was .81 for the attitude survey.

Posttask Interview Questionnaire. A 10-item questionnaire was used tointerview selected participants 1 to 5 days after they completed the

FIGURE 4. TASK SCENARIO


EXHIBIT 1 POSTTASK ATTITUDE SURVEY

This brief survey is the last part of the study. It will be used to compare your preferences between differenttypes of learning support.

1. It was easy to find the information I needed to complete the tax scenario.

� Strongly agree

� Agree

� Disagree

� Strongly disagree

2. The learning support provided was at the appropriate level of detail to aid in completing the tax scenario.

� Strongly agree

� Agree

� Disagree


3. I received enough training to successfully complete the tax scenario.

� Strongly agree

� Agree

� Disagree


4. I received enough help to successfully complete the tax scenario.

� Strongly agree

� Agree

� Disagree


5. I felt confident that I could successfully complete the tax scenario.

� Strongly agree

� Agree

� Disagree


6. In the future, I would like to use learning support such as the one demonstrated in this study.

� Strongly agree

� Agree

� Disagree


7. I would recommend the type of learning support used in this study to my organization to help employeesperform better.

� Strongly agree

� Agree

� Disagree


8. The total amount of time I spent learning how to perform the tax scenario was:

� Too little

� About right

� Too much


research study. The questionnaire is shown in Exhibit 2. The interviewquestions were divided into three sections: the first three questions weretargeted for participants who completed the study in the training-only andthe training and EPSS groups, questions 4 to 6 targeted participants whocompleted the study in the EPSS-only and the training and EPSS treatment

EXHIBIT 2 INTERVIEW QUESTIONNAIRE

Interview questions for treatment groups receiving Training (Training-only, Training & EPSS)

Procedure: Provide interview participants with a link to the web-based training course, ask them to

launch the course, and then ask the following questions.

1. What did you like about the training course?

2. What did you not like about the training course?

3. Explain how you went through the course.

Interview questions for treatment groups receiving EPSS (EPSS-only, Training & EPSS)

Procedure: Provide interview participants with a link to the tax software with EPSS, ask them to launch

the application, and then ask the following questions.

4. What did you like about the online help?

5. What did you not like about the online help?

6. Explain how you used the online help.

Interview questions for all treatment groups

7. Did you feel the learning support help you complete the task? Why?

8. What motivated you to complete the task?

9. What kinds of issues did you run into while trying to complete the research study?

10. What would you recommend changing to encourage people to complete a study like this in

the future?

FIGURE 5. TRAINING AND PERFORMANCE SUPPORT MATRIX


groups, and questions 7 to 10 were common questions for all participantswho completed the research study.

Criterion Measures

The primary measure in this study was the users’ attitudes toward theperformance support and/or training intervention provided.

The eight-item survey shown in Exhibit 1 was administered to measureparticipant attitudes toward the effectiveness of the training and supportinterventions provided in aiding them to complete the task. For mostquestions, respondents used a 4-point Likert scale (4 5 strongly agree, 1 5

strongly disagree) to rate their attitudes regarding the effectiveness of theinterventions. On question 8, respondents were asked to rate the amount oftime they spent learning how to perform the task. As a result, respondentswere provided with only three options: "about right," "too much," and "toolittle." The internal consistency reliability of this survey was determinedusing a Cronbach alpha procedure.

To collect qualitative data regarding user attitudes, seven participantsfrom each of the three treatment groups were selected to participate inposttask interviews. Interview participants were identified using interest datacollected using a pretask demographic survey. In an attempt to ensure thatthe participants’ comments were recent and relevant, participants wereinterviewed 1 to 5 days after they participated in the study. Study participantswere sent hyperlinks to view the Web-based training course or performancesupport system based on their treatment group. Participant feedback wascollected by the lead researcher using the posttask interview questionnaire.These data were compiled and used to inform the data from the user attitudesurvey.

FIGURE 6. THE RELATIONSHIP BETWEEN TRAINING AND PERFORMANCE SUPPORT

CONTENT


Procedures

Since the participants in the study worked in different companies andwere geographically dispersed, various corporate training managers andlocal chapters of ASTD and ISPI were asked to recruit participants from theirrespective organizations. An e-mail invitation was sent to study participants.The e-mail instructions directed participants to the location of the researchstudy on the Internet. Prior to the assignment of treatments, participantscompleted a demographic survey that was used to screen for prior knowledgeof corporate tax preparation. Any individual currently working in a finance-related role, with a finance-related degree, or with tax or accountingcertifications was not selected to participate in the study.

The system randomly assigned participants who did not have any financebackground into one of three treatment groups (training-only, EPSS only, andtraining and EPSS) and displayed the appropriate intervention. Participantswere not aware that they were assigned to a different treatment group or that

TABLE 1 USER ATTITUDE SCORES BY TREATMENT

ITEM

TRAINING-

ONLY EPSS-ONLY

TRAINING

& EPSS

OVERALL

MEANS

(n 5 26) (n 5 26) (n 5 26) (n 5 78)

It was easy to find the information I

needed to complete the tax scenario.

1.73a b 2.62a c 3.15b c 2.50

The learning support provided was at

the appropriate level of detail to aid

in completing the tax scenario.

1.54a b 2.58a c 3.12b c 2.41

I received enough training to success-

fully complete the tax scenario.

2.27d b 1.92dc 2.77b c 2.32

I received enough help to successfully

complete the tax scenario.

1.77a b 2.58a 2.96b 2.44

I felt confident that I could successfully

complete the tax scenario.

2.04a b 2.54a 2.85b 2.47

In the future, I would like to use

learning support such as the one

demonstrated in this study.

2.23a b 2.81a c 3.35b c 2.79

I would recommend the type of

learning support used in this study

to my organization to help

employees perform better.

1.77a b 3.12a 3.31b 2.73

Overall Means 1.91 2.59 3.07 2.52

Note. Items were measured on a 4-point Likert scale (1 5 strongly disagree, 4 5 strongly agree).

aSignificant difference between Training-only and EPSS-only at po.01.

bSignificant difference between Training-only and Training & EPSS at po.01.

cSignificant difference between EPSS-only and Training and EPSS at po.01.

dSignificant difference between Training-only and EPSS-only at po.05.


their system was configured with a different training or EPSS intervention. Ifthe participants were part of the training-only group, they werefirstdirected totake the Web-based training course. If the participants were part of the EPSS-only group, the opening screen of the tax software application was presented,providing a brief set of instructions demonstrating how to access the supportsystem. If the participants were part of the training and EPSS group, they firsttookthe trainingcourse and werethenprovided with the performancesupportsystem instructions. Once the participants completed the task and submittedthe tax information, they were automatically directed by the system tocomplete the user attitude survey.

Data Analysis

Multivariate analysis of variance (MANOVA) was conducted on aportion of the data from the attitude survey, followed by univariate ANOVAswhere appropriate. A chi square test was conducted on nonparametric datafrom the attitude survey. A Cronbach alpha procedure was conducted on theuser attitude data to compute the reliability of the posttask survey.

Results

Survey Data

The primary research question investigated the effect of EPSS andtraining interventions on the attitudes of participants. An eight-item surveywas administered after completion of the tax procedure task. The first sevenitems included a 4-point Likert-type scale for participants to respond tostatements about the training and performance support interventions (4 5

strongly agree, 1 5 strongly disagree). Data reported in Table 1 reveal that theaverage overall attitude rating was 3.07 for the training and EPSS group, 2.59for the EPSS-only group, and 1.91 for the training-only group.

A 3–by-7 MANOVA was conducted on these seven items to test forsignificant differences. The overall means were significantly different acrossthe three treatment groups: Wilks’ l5 .18, F (18, 134) 5 13.14, po.01. Thestrength of the relationship between the treatments and user attitude scoreswas strong: e121 5 .57.

Follow-up one-way analyses of variance revealed significant differencesamong treatment groups on all seven of the survey items. Post hoc tests were

TABLE 2 USER ATTITUDE FREQUENCIES FOR TIME SPENTLEARNING HOW TO COMPLETE THE TASK

RATING TOO LITTLE ABOUT RIGHT TOO MUCH

Training-only 16 0 10

EPSS-only 4 17 5

Training and EPSS 3 21 2

Total 23 38 17


TABLE 3 SUMMARY OF PARTICIPANT POST-TASK INTERVIEW RESPONSES

ITEM RESPONSES

Interview questions for training-only and training and EPSS groups

1. What did you like about the training course?

Practice activities 6

Tax process flow diagram 4

Navigation 2

Explanation of navigation 1

Organization 1

2. What did you not like about the training course?

Tax topic 6

Too much detail 5

Did not provide a job aid 2

Could not apply the learning immediately 1

Too much information 1

3. Explain how you went through the training course.

Proceeded through course in a linear fashion 8

Read each topic thoroughly at first, skimmed towards the end 7

Read certain topics thoroughly, skimmed other topics 4

Skimmed all topics 3

Interview questions for EPSS-only and training and EPSS groups

4. What did you like about the online help? 6

Information available at point of need 5

Did not have to remember the information 2

Information chunked 1

Did not have to learn the information in a training course

5. What did you not like about the online help?

Information did not provide enough detail to support successful task completion 8

Information was not clear 2

Tax topic 2

Could not use external sources for information (IRS, Google) 1

Wait time while help content loaded 1


conducted on these items. Pairwise comparisons revealed 20 significantdifferences among groups. On six questions, participants in the EPSS-onlygroup and those in the training and EPSS group had significantly morepositive attitudes than participants in the training-only group. In addition,participants in the training and EPSS group had significantly more positiveattitudes than those in the EPSS-only group on four questions.

6. Explain how you used the online help.

Used when did not know the answer 8

Used to validate prior knowledge 5

Used for each line item 1

Interview questions for all participants

7. Did you feel the learning support helped you complete the task?

Why?

No, could not remember information from training 8

No, not enough detail in help 5

No, tax content was confusing 5

Yes, prefer to learn by doing 3

8. What motivated you to complete the task?

Want to help lead researcher 15

Want to complete any tasks that are started 5

Want to figure out the task 1

9. What kinds of issues did you run into while trying to complete the research study?

No issues reported 11

Distractions in work environment 5

Information on certain help screens disappeared after 30 seconds 2

Went back to training course and lost data in tax software 2

Popup blocker prevented help screens from opening 1

10. What would you recommend changing to improve this study in the future?

No suggestions provided 15

Select a different topic that is more valuable to participants 4

Change online help to an automated wizard 1

Conduct survey research 1

TABLE 3 SUMMARY OF PARTICIPANT POST-TASK INTERVIEW RESPONSES(CONTINUED)

ITEM RESPONSES


The eighth item on the survey presented three choices that measured theparticipants’ perception toward the amount of time they spent learning howto perform the tax scenario. The response frequency for this item is shown inTable 2. The table reveals that 38 participants (49%) thought that the amountof time they spent learning was about right, 23 participants (29%) thoughtthat they had spent too little time, and 17 (22%) thought that they had spenttoo much time. Closer examination of the data shows that the majority ofparticipants in both the training and EPSS group (81%) and in the EPSS-onlygroup (65%) thought that the time they spent learning how to do the task wasabout right. But no participants in the training-only group indicated that theamount of time they spent learning how to perform the task was about right.Sixteen participants in the training-only group (62%) thought that they spenttoo much time learning, and the remaining 10 participants (38%) thoughtthat they did not spend enough time. A chi square test revealed that thedifference in proportions between the treatment groups was significant:w121(2, N 5 78) 5 9.00, po05.

Interview Data

Seven participants from each treatment condition were interviewed todetermine their opinions of the program (n 5 21). User interview responsesto the ten questions are summarized and presented in Table 3.

Participants who received training as part of their treatment were firstasked what they liked about the Web-based training course. Six of the 14training participants indicated that they liked the practice activities. Fourparticipants identified the tax process flow diagram used at the beginning ofeach unit of instruction as a helpful advanced organizer.

When asked about what they did not like about the Web-based course,six participants who received training indicated that they did not like the taxtopic used as the basis of course. Five participants complained that the coursehad too much detailed information.

Training-only and training and EPSS group participants were then askedto explain how they proceeded through the training course. Seven partici-pants indicated that they started out reading the pages of the coursethoroughly at first but then skimmed the content as they progressed. Fourparticipants reported that they selected topics that they felt were importantand read them thoroughly; they skimmed less meaningful topics. Threeparticipants admitted that they skimmed through the course without readingthe topics thoroughly. In addition, eight participants added that theyproceeded through the course in a linear fashion.

Participants who received the EPSS as part of their treatment were askedwhat they liked about the online help system. Six of the 14 EPSS participantsindicated that they liked the fact that support information was available at thepoint of need. Five participants liked the fact that they did not have toremember the information required to complete the tax preparation task.

When asked what they did not like about the EPSS, eight participants feltthat the online help system did not provide information that was detailedenough to support successful completion of the task. Two participants found


the information unclear, and two others complained that they did not like thetopic of the tax scenario.

EPSS-only and training and EPSS group participants were then asked toexplain how they decided to use the online help system during the task. Eightparticipants reported that they used the EPSS when they did not know theanswer. Five indicated that they used the EPSS when they felt they knew theanswer but wanted to validate their assumptions.

All 21 interview participants were then asked if they felt that sufficientlearning support was provided to help them complete the task. Eight of 14training participants indicated that they could not remember enough informa-tion from the training course to successfully complete the tax preparationscenario.Five of 14 EPSS participants felt that they did not have enoughsupportbecause the help content did not provide enough detail. Three of 14 EPSSparticipants found the learning support sufficient. Five participants from all 21interviewees reported that the topic of taxes was confusing to them.

When asked what motivated them to complete the tax scenario, 15 of 21interviewees across all treatment groups noted that they wanted to help thelead researcher complete the study. Meanwhile, five indicated that theirpreference was to complete any task that they start.

Participants were asked to identify any issues that they may haveexperienced while completing the study. While most participants did notidentify any problems, five participants mentioned that they had difficultyfocusing on the tax scenario due to distractions in their work environmentsuch as e-mail, phone calls, and interruptions from coworkers. Two EPSSparticipants pointed out that content in one of the help screens in the EPSSdisappeared after 30 seconds. In addition, two training participants com-plained that they attempted to return to the Web-based training course forreference while they were completing the tax preparation task and lost all oftheir data in the process. One participant noted that a pop-up blockerinstalled on the computer prevented the online help windows from appear-ing at first, and the participant corrected the issue.

Finally, participants were asked for recommendations to improve theresearch study. Fifteen of the 21 interviewees did not offer any suggestions.Four participants suggested that the research be focused on a topic thatwould be more meaningful to potential participants. One participantsuggested that the EPSS be changed from an online help system to a wizardthat automated the tax preparation task. Another participant noted thatresearch in industry is typically conducted using survey instruments thatparticipants can spend a short amount of time completing. The participantdid not realize that she would be asked to complete a task.

Discussion

Key Themes

Several key themes can be interpreted from the results of the attitudesurvey and participant interviews.


Too Much Training. It is clear that participants provided only with trainingwere generally less satisfied than the other groups were. During follow-upinterviews, six participants said they felt overwhelmed by the amount ofinformation or detail provided in the training course. The Web-basedtraining course used in this study contained 87 navigation, instructional,practice, and transitional screens. Participants spent an average of38 minutes 44 seconds reviewing the content in the training course prior

to completing the tax procedure. This volume ofinformation likely contributed to the lowerattitudes of the training-only group. Oneparticipant noted, "It was very easy to get lost inthe explanations." Because of the overwhelmingamount of information, some participants foundit difficult to remember the content when they wereasked to complete the tax preparation task. Said

one, "It was difficult for me to fill out all of the information in the formbecause I could not remember all the details [from the training course]."Another participant complained, "This is one of the problems I have withtypical training. Sometimes I miss details that are critical. It’s hard todifferentiate what’s important from what I already know."

A few training-only participants even asked for some type of supportsystem to provide information during task completion. Typical responseswere: "It would have been nice to have some cheat sheet to find the things that Iremembered reading,but I didn’t remember what the details were" and "I thinkthe biggest thing [that would have assisted me in this task] would be some typeof help during the task because the steps can get rather convoluted."

One side effect of the overwhelming amount of information providedduring training was the participants’ tendency to process the information at asuperficial level. Findings revealed that 13 of 21 participants interviewedadmitted that they skimmed through portions of or, in a few cases, all of thetraining course. One participant noted, "I started out reading everything andthen towards the end, or for specific sections, I found myself skimming."Another participant flatly stated, "I didn’t pay attention to the last half of thetraining course."

When asked if their course-taking habits in the study were different fromwhat they do in a normal work or training setting, participants felt that theirbehaviors were typical: "I approached this [training course] the way as I wouldnormally," said one. Others provided more insight into their online traininghabits: "In general, I usually go through it fairly quickly. I tend to skim andread things at a high-level." One participant admitted, "When I take requiredcourses, I sometimes skip to the end to get credit and meet the requirements."Another participant felt that the amount of content was problematic: "Ifound myself skimming through some of it just because there was no way Icould possibly remember all of the detail."

The Strengths of Training. Despite calls from some authors (Chase, 1998;Desmarais et al., 1997; Foster, 1997) to reduce or eliminate

It is clear that participantsprovided only with training

were generally lesssatisfied than the other

groups were.


training, participants did identify attributes of their training intervention thatseemed to add value. Some participants commented that they liked the taxprocess flow diagram used at the beginning of each unit of instruction. Theprocess served not only as an advanced organizer for the training course butalso as a guide or checklist for the participants as they completed the task.Others indicated that more than any other aspect of the training course, thepractice activities helped to prepare them to perform the tax procedure. Dueto the high amount of user control provided by EPSS, it would be difficult toprovide performers with comparable prior knowledge about the process oropportunities to practice the task if a training intervention was not provided.

Not Enough Support. While training-only participants complained that theyreceived an overwhelming amount of information, eight participants whoreceived EPSS felt that the system did not provide information that wasdetailed enough to support successful completion of the task. This finding isparticularly interesting as the content delivered to participants in both thetraining and EPSS treatments was identical. This suggests that while thevolume of information delivered during pretask training should be minimized,access to support content during job performance should be much broader.

Recommendations for Training and Performance Support

Interventions

Based on the findings of this study, several recommendations can beoffered to assist performance technologists when implementing training andperformance support as performance interventions:

Limit the amount of information delivered during training. Participantsreported feeling overwhelmed by the quantity of content they received andskimmed through the training course as a result. Performance technologistsand instructional designers should carefully identify the content to bedelivered using a training intervention in the analysis phase. Two potentialfactors to consider are frequency and criticality.

As shown in Figure 5, tasks and information that are used frequently andare mission critical should be delivered primarily through a training interven-tion. A performer who did not master this content prior to task performanceprobably would not be able to complete the task correctly, potentially harmingself, others, or even the organization as a whole. Tasks and information thatperformers do not routinely work with and are not critical to the organizationwould likely serve as distracters in a training intervention. As such, they can besafely delivered exclusively through performance support. Objectives that areeither high frequency and low criticality or low frequency and high criticalityshould also be trained, although the level of mastery required may be variable.Since such tasks may not be performed routinely or the consequences of errorsignificant, it is also important to provide redundant performance support. Bydoing so, performers can refresh what they may have learned during trainingwhile on the job.

Provide performers with robust practice during training that incorporatesthe EPSS. Practice has been regarded as a powerful instructional method


(Merrill & Salisbury, 1984, Vinsonhaler & Bass, 1972). Participants in thisstudy confirmed that the practice provided them an opportunity to preparefor and build confidence toward successfully completing the task. If possible,performance technologists and instructional designers should incorporateany performance support systems that will be available to performers on thejob into training practice activities, particularly activities that may require theperformer to solve realistic problems. This will provide the performers theopportunity to learn how to use the EPSS in a safe environment andpotentially increase adoption as performers may experience firsthand thevalue of using the system to solve problems.

Make training materials available for performance support. In this study,participants became familiar with the support content during the trainingtreatment but were not able to accurately recall the information that theylearned. Those provided with both training and EPSS could quickly accessthe information through the performance support system, allowing them torefresh or validate what they learned during the training phase. Oneparticipant noted, "I could cross-check whatever I was putting in the item,I could feel confident that it was hopefully the right stuff." Extending thispractice to other performance problems could benefit performers.

Provide access to a broader range of content for performance support.While performance technologists should strive to minimize training contentto objectives high in frequency or criticality, they should at the same timemaximize the amount of relevant support content available to performers onthe job. As shown in Figure 6, EPSS content can be considered the superset ofinformation required for task performance, while training can be a carefullyprescribed subset of that body of information (in addition to instructionalmethods that may not be relevant for performance support, such asintroductory materials, practice, and assessment).

It is important to note, however, that providing a dearth of performancesupport information without regard to limiting the scope or providingintegrated, intuitive ways to access the information could also have a negativeeffect on performance and attitudes. A performance technologist must stillbe judicious in selecting the appropriate EPSS type and support content toaddress a particular performance problem.

Limitations

Several limitations should be considered when interpreting the results ofthis study. The task used in this research was tax preparation, a procedurelargely supported by requisite background facts, concepts, and principles.One potential limitation is that these findings may not extend well to otherwork contexts. In addition, some participants did not like the topic of taxpreparation, which may have had a negative impact on their attitudes fromthe outset.

Another limitation is that the study employed an extrinsic context helpEPSS (Gery, 1995). This extrinsic performance support system was selectedfor use in this study based on its demonstrated strength in previous research(Bailey, 2003; Nguyen et al., 2005). If a different type of EPSS was used, such as


an external search engine or intrinsic automatedwizard, participant attitudes could be affected.

Finally, the training intervention used in thisstudy was a Web-based training course that parti-cipants completed individually with minimal super-vision or control. If another method to delivertraining was adopted, such as instructor-led or peertraining, participant attitudes could be affected.

Future Research

Although the results of this study support thenotion that one should strive to rely less on train-ing and more on performance support interventions for content of certainfrequency and criticality, this practice could create a new problem ofinformation overload during job performance instead of during training.To mitigate such issues, additional research should focus on how to bestpresent EPSS content, techniques to present text-based information, navi-gation strategies, and the efficacy of rich media such as audio and video forperformance support.

The availability of content originally designed for training to beused for performance support also creates instructional design anddevelopment challenges. Research on how to best design materials forreuse across training and EPSS would ensure its effectiveness for bothinterventions.

Conclusion

While training and performance support may have been initially con-ceived to be mutually exclusive interventions, this study provides insight intoperformers’ attitudes toward both and how they could potentially becombined into an effective solution to address problems before and duringjob performance.

References

Arthur, W., Bennett, W., Edens, P. S., & Bell, S. T. (2003). Effectiveness of training in

organizations: A meta-analysis of design and evaluation features. Journal of Applied

Psychology, 88(2), 234–245.

Bailey, B. (2003). Linking vs searching: guidelines for use [Electronic version]. Retrieved

January 17, 2006, from http://www.Webusability.com/article_linking_vs_

searching_2_2003.htm

Bastiaens, T. J., Nijhof, W. J., Streumer, J. N., & Abma, H. J. (1997). Working and learning with

electronic performance support systems: An effectiveness study. International

Journal of Training and Development, 1(1), 72–78.

Burke, M. J., & Day, R. R. (1986). A cumulative study of the effectiveness of managerial

training. Journal of Applied Psychology, 71(2), 232–245.

Chase, N. (1998). Electronic support cuts training time [Electronic version]. Quality

Magazine. Retrieved January 12, 2005, from http://openacademy.mindef.gov.sg/

OpenAcademy/Learning Resources/EPSS/c16.htm

Desmarais, M. C., Leclair, R., Fiset, J. V., & Talbi, H. (1997). Cost-justifying electronic

performance support systems. Communications of the ACM, 40(7), 39–48.

While performancetechnologists should striveto minimize trainingcontent to objectives highin frequency or criticality,they should at the sametime maximize the amountof relevant support contentavailable to performers onthe job.


Foster, E. (1997). Training when you need it [Electronic version]. Info World. Retrieved

November 17, 2004, from http://openacademy.mindef.gov.sg/OpenAcademy/

Learning%20Resources/EPSS/c1.htm

Gery, G. (1991). Electronic performance support systems. Tolland, MA: Gery Associates.

Gery, G. (1995). Attributes and behaviors of performance-centered systems. Performance

Improvement Quarterly, 8(1), 47–93.

Mao, J., & Brown, B. (2005). The effectiveness of online task support versus instructor-led

training. Journal of Organizational and End User Computing, 17(3), 27–46.

Merrill, P. F., & Salisbury, D. (1984). Research on drill and practice strategies. Journal of

Computer-Based Instruction, 11(1), 19–21.

Nguyen, F., & Klein, J. D. (2008). The effect of performance support and training as

performance interventions. Performance Improvement Quarterly, 21(1), 95–114.

Nguyen, F., Klein, J. D., & Sullivan, H. (2005). A comparative study of electronic performance

support systems. Performance Improvement Quarterly, 18(4), 71–86.

Rackham, N. (1979). The coaching controversy. Training and Development Journal, 33(11),

12–16.

Snipes, J. (2005). Blended learning: Reinforcing results. CLO Magazine. Retrieved June 29,

2008, from http://www.ninthhouse.com/papers/4%20CLO%20Reinforcing%20

Results.pdf

Vinsonhaler, J. F., & Bass, R. K. (1972). A summary of ten major studies on CAI drill and

practice. Educational Technology, 12(7), 29–32.

FRANK NGUYEN

Frank Nguyen is an assistant professor at San Diego State University.Mailing address: San Diego State University, 5500 Campanile Drive, SanDiego, CA 92182. E-mail: [email protected]


the effect of performance support and training on performer attitudes

Documents