effectiveness of role playing, case studies, and...

Effectiveness Of Role Playing,Case Studies, and Simulation

Games in TeachingAgricultural Economics

Steven C. Blank

This paper assesses the impact of introducing role playing, case studies, and simulation

games into undergraduate courses in agricultural economics. An educational impact model is

used in a qualitative evaluation of the teaching aids and to generate hypotheses. Quantitative

experimental results are used to test the hypotheses. Results indicate that each aid can improve

students' performance when used in appropriate situations. However, there is a definite trade-

off between student contact time requirements and the effect of these aids.

Role playing, case studies, and simula-tion games are three of the many teachingaids being used by university instructorsto supplement the lecture format forteaching (O'Connor and Osterman; Sieg-fried and Fels; French; Kendrick). Allthree of these techniques are simulationswhich can be classified as learning tools(Blank, p. 215). A simulation is an oper-ating model which demonstrates thestructure of a system. Simulations de-signed as learning tools are those that pro-vide participants with a new or improvedunderstanding of the system which hasbeen modeled. These techniques cannotcompare with the lecture format when theobjective is to convey large volumes of in-formation, but they can improve the levelof understanding concerning specificpoints (Kolb). Often techniques such as

Steven C. Blank is currently an Extension Economistin the Agricultural Economics Department at the

University of Arizona. When this research was un-

dertaken he was with the Agricultural ManagementDepartment at California Polytechnic State Univer-

sity, San Luis Obispo.

The author thanks Brian Schmiesing and Emerson

Babb for helpful comments on an earlier draft of this

paper.

these are used because the learning pro-cess of typical students in some disciplinesnecessitates using teaching methods thatrely less on intuition and reading and moreon sensing and factual materials (Robertsand Lee). Yet, despite the increased useof these teaching aids, studies of their abil-ity to increase the effectiveness of learn-ing have had mixed results (Siegfried andFels).

This study was developed to contributeinsight into these issues. Presentation ofthe findings is organized into four majorsections. First, the objectives and meth-odology are discussed, emphasizing theexperimental design used. Next, a briefdescription of the three teaching aids ispresented. Third, the analysis of experi-mental results for each aid is presented.Finally, conclusions and limitations of thestudy are outlined.

Objectives and Methodology

The purpose of this paper is to assessthe impact of introducing role playing,case studies, and simulation games intoundergraduate courses in agricultural eco-nomics. The educational impact modeldeveloped by Joyce and Showers will be

Western Journal of Agricultural Economics, 10(1): 55-62© 1985 by the Western Agricultural Economics Association

Western Journal of Agricultural Economics

used in a qualitative evaluation of thecontents and expected effectiveness of theteaching aids. In addition, the results ofan experiment designed to quantify eachtechnique's actual educational impact willbe presented.

Educational Impact Model

Joyce and Showers state that when stu-dents use what has been learned to solveproblems they are demonstrating that theirtraining has had the highest level of im-pact possible. The level of impact a teach-ing program will have, in turn, is affectedby the following training components:

1. presentation of theory or descriptionof skill or strategy,

2. model or demonstration of skills ormodels of teaching,

3. practice in simulated and classroomsettings,

4. structured and open-ended feedback,and

5. coaching for application.

Joyce and Showers indicate that compo-nents 1 through 5 above have increasinglygreater levels of impact on students' abil-ities to solve problems. When all five com-ponents are included in a teaching pro-gram, up to 75 percent of students areable to apply what has been learned.

The model by Joyce and Showers im-plies that teaching aids which incorporatemore of the five components will havegreater impact than aids involving fewertraining components. This hypothesis willbe considered here by using the resultsfrom an experiment conducted over a pe-riod of one academic year. The experi-ment uses average test scores as an indi-cator of student performance.

Experiment Used

Student test scores were evaluated dur-ing a three-year period. During the firstyear, three different courses were taught

56

by a single instructor using primarily thestandard lecture format. During the year-end review of each course the instructordecided that the level of student perfor-mance was not satisfactory. Therefore, theinstructor decided to incorporate theteaching aids into the course presentation.During the second academic year the aidswere used (as described in the next sec-tion) and evaluated informally.

At the end of the second year the in-structor decided to eliminate doubts con-cerning the results of the informal analysisby conducting a formal experiment. Dur-ing one term of the third year, each of thethree aids was tested using a Posttest-OnlyControl Group (POCG) design.' This de-sign, considered a "true experiment"(Blank, p. 198), is diagrammed

R X O0

R 02

where an "X" represents the introductionof an experimental stimulus to a group,an "0" identifies a measurement or ob-servation event, and an "R" indicates thatthe group members have been selectedrandomly. The Xs and Os are read fromleft to right in terms of time periods, andsymbols which are vertical to each othertake place simultaneously. Threats to bothinternal and external validity are handledadequately by this design. The effect ofeach teaching aid (X) is measured by thedifference 0, - 02 in separate experi-mental iterations.

The POCG experiments involved com-paring the performance of two separatesections of the same course. During thesame academic term, the instructor hadtwo sections of the course, one which wastaught using only the lecture style, theother taught by using the relevant teach-ing aid in addition to the lecture. The sec-tions were offered on the same days and

'Each aid was tested during a different term in thethree-quarter academic year.

July 1985

Teaching Aid Effectiveness

the control groups were the second (later)section so as to benefit from any improve-ments in instructor performance and todownward-bias the observed improve-ment of the experimental groups. Studentperformance was measured using scoresfrom the relevant course examinations.Each section was given the same exam onthe same day. The form of each exam(about 2/3 multiple choice and 1/3 short an-swer questions) facilitated objective grad-ing.

The Aids as Teaching Devices

The three aids (role playing, case stud-ies, and simulation games) are all used bythe Agricultural Management (AM) de-partment of California Polytechnic StateUniversity, San Luis Obispo (CPSU). Thatdepartment will serve as the source for allexamples presented here. A brief descrip-tion of how each aid was used follows.

Role Playing

Role playing, as most often used in aclassroom, requires physical involvementon the part of students. Two or more peo-ple "act out" the part of individuals in ahypothetical situation (Black). One ex-ample comes from an introductory agri-cultural economics course. While present-ing the subject of price determination, arole-playing exercise patterned around theopen-outcry market typical of commodityfutures markets was used. Sixteen studentsparticipated in three successive auctions,some acting as farmers and others actingas wholesalers. Information concerningquantities to be bought or sold and profit/cost levels per pound was provided on in-dex cards given to each participant by theinstructor. During each three-minute auc-tion the students had to negotiate trans-action prices with one another. After thethird auction, the three sets of resultingprices were analyzed by the entire class.After the instructor told the class which

set of prices were generated in an equilib-rium, surplus, and shortage situation thestudents could see that the theory pre-sented in their textbooks did, in fact, de-scribe what happened in their exercise.

Using role playing ensures that studentswill be exposed to training components 1through 3 of Joyce and Showers' impactmodel. Whereas lectures utilize only thefirst two components, role playing is de-signed to give students classroom practicein solving particular types of problems.Component 4 is often excluded from beingpart of this teaching aid.

Case Studies

Case studies are used widely. An un-dergraduate research methods course atCPSU used cases analyzed by small teamsof students in an effort to tie together theskills learned in a number of other cours-es.

The purpose of studying business casesin the research methods course is not al-ways to learn specific answers to specificproblems, but to become familiar withanalysis and decision making, with theprocess of arriving at answers rather thanwith answers themselves (Ladd). In thissituation students analyzed each case inteams of 3 to 5 with the entire group beingresponsible for submitting a single writtenreport. This method forced each studentto come in contact with others in the class.During the academic term the composi-tion of the teams was continually changedso that each student worked with a differ-ent group on each case. The teams workedtogether both in and out of class to pre-pare the written report. That forced themto deal with the real problems of allocat-ing their scarce time and resources tocompleting the case assignment whileworking with people that they did not se-lect themselves.

Using case studies incorporates compo-nents 1 through 4 of the impact model.The students practice applying their new

57

Blank


TABLE 1. Impact of Teaching Aids on Exam Scores in Different Agricultural Economics Cours-es.

Average Exam Scorea % Change

Control TestAid (Course) Group Group Topicb Total % A/hr

Role-playingd (Introductory) 65 68 3 4.6 4.6Case studiese (Research Methods) 70 78 -11.4 2.3Computer game' (Marketing) 72 83 -15.3 1.7

a Score is from a maximum possible of 100.b Change in scores related to the topic of price determination only.c The average student contact time was one, five and nine hours per term, respectively, for role playing, case

studies, and the computer game.d Sample size for each group: 45.e Sample size for each group: 30.f Sample size for each group: 32.

problem-solving skills both inside and out-side the classroom. They receive struc-tured feedback from the instructor as wellas open-ended feedback from other stu-dents while the teams are working togeth-er.

Simulation Games

Simulation games in economics andmanagement tend to be computerized(Litzenberg). One of the most compre-hensive games used in the AM depart-ment of CPSU was part of an advancedmarketing course. In that course studentswere divided into three-person teams.Each team performed as the managementof a separate company in a computerizedsimulation game. All of the teams werepart of a single industry, competing withone another. Decisions concerning all as-pects of company operations were sub-mitted by each team on a regular basisand the results of those actions were cal-culated. Having to make the decisionsgives students valuable insight into howvarious aspects of a company are interre-lated (Boehlje and Eidman). This perspec-tive could not be given to students usingonly a lecture format (Baker and Babb).

Computerized simulation games com-bine the contents of case studies and role-playing exercises (White). As a result sim-

ulation games expose students to trainingcomponents 1 through 4 and provide moredepth of exposure than either of the othertechniques. This is due principally to therelatively long period of time that simu-lation games require to complete. There-fore, simulation games provide studentswith many more opportunities to practiceproblem solving and to receive feedbackfrom both instructors and other students.

Hypotheses Implied by the Model

Implied by the qualitative assessmentof the aids above is the hypothesis thateach of the three teaching devices will im-prove student performance over that ob-served when a lecture-only approach isused. All three aids are expected to im-prove on the lecture-only approach be-cause lecturing involves just the first twotraining components of Joyce and Show-ers' model while role playing, case studies,and simulation games each include at leastthree components. A quantitative assess-ment of the hypothesis is presented below.

Analysis of the Aids

The effects that role playing, case stud-ies, and simulation games had on the per-formance of students in the experiment

58

July 1985


described above are presented in Table 1.Average test scores improved in all cases,as did student evaluations of the courseswhen the teaching aids were introduced.

The results for the experiment concern-ing use of a role-playing exercise providesome insight into the impact of that teach-ing aid and possibly others. Although roleplaying had a small direct effect on examscores, covering only the relevant topic (a3 percent increase on scores concerningprice determination), course total scoresfrom the experiment improved 4.6 per-cent, on average (Table 1). An analysis ofvariance (ANOVA) of the scores from thecontrol and experimental groups found astatistical difference at the 90% signifi-cance level. Written comments made bystudents evaluating the course indicate thatrole playing may have raised student en-joyment enough to have had a significantindirect effect on course grades. Seventy-three percent of participants mentionedthe exercise in their answer to the open-ended question, "What were the most fa-vorable attributes of the course?" Only 3percent listed the exercise as one of the"least favorable" attributes in response tothe next question on the course evaluationform.

There was an 11.4 percent increase inthe average test score in the researchmethods course after case studies were in-troduced. An ANOVA found this differ-ence between control and experimentalgroups to be significant at the 95% level.In general, there was improvement acrossthe range of students: more "A" and few-er "F" course grades were given.

In the marketing course there was a 15.3percent increase in exam scores when thecomputer simulation game was used. As aresult, the average course grade issued rosefrom a "C" to a "B." The difference inscores was found to be significant at the95% level using an ANOVA.

It is likely that some of the improve-ments made in the research and market-

ing courses were due to the indirect ef-fects of the teaching aids, as was the casein the introductory course. Students eval-uating each of the two courses often notedthat the cases and the game presentedthem with an interesting challenge whichthey enjoyed.2

Care must be taken when interpretingthe results of these experiments. Althoughthe total percent change in test scores re-ported in Table 1 support the hypothesisposed, it must be kept in mind that thelevel of impact noted here for each teach-ing aid is not necessarily indicative of thelevel of impact that the technique wouldhave if used in other courses. Also, theresults for the three teaching aids are notdirectly comparable because they camefrom three different courses. The changesin test scores were used here simply toindicate whether the teaching aids had anyimpact on the effectiveness of learning,not to assess which technique is "best."

To make the results between aids morecomparable, the impacts must be modi-fied to allow for differences in studentcontact time. Each teaching aid requireda different amount of students' time peracademic term (1, 5, and 9 hours, respec-tively, for role playing, case studies, andthe computer game). 3 As a result, it is notsurprising that the aid requiring the mosttime, the computer game, had the highestabsolute level of impact on students' per-formance. This result corresponds to thatof Siegfried and Fels. A better measure ofthe relative impact a teaching aid had onlearning would be one which reflected the"return on investment" received from stu-

2 The percentage of "most favorable" and "least fa-vorable" comments for case studies were 61 and 6,respectively. The responses for the computer gamewere 80% favorable, 2% unfavorable.

3 These are the average amounts of student contacttime reported by students for case studies and thecomputer game, and from observation by the in-structor for the role-playing exercise.

59

Blank


dent contact time. This modification ledto the last column in Table 1, which re-ports the percent change in exam scoresper hour of average student contact time. 4

Such a modification still does not makethe results from the three aids directlycomparable, but it does give a differentimpression of the impacts generated. Tobe comparable the results would have toall come from applications in the samecourse (not three different courses).Nevertheless, by shifting the focus of aquantitative assessment of the aids fromthe absolute level of impact to a return ontime invested framework, the modifiedresults of this study raise some questionsfor future research.

For example, the relationship betweeninputs (time) required of students/facultyand resulting outputs (educational im-pacts) for particular teaching aids may bean important factor in determining whichlearning devices are most appropriate inspecific circumstances. Equally importantmight be the choice of how outputs are tobe measured-on a total or marginal ba-sis. Plotting the experimental results re-ported here for all three aids on a graphwith total student contact hours on thehorizontal axis and total percent changein test scores on the vertical axis illustratesa positive relationship (nearly a straightline sloping upward to the right). How-ever, placing percent change in test scoresper hour of contact time on the verticalaxis transforms the relationship into a neg-atively sloped curvilinear figure. In otherwords, it appears that use of the marginalphysical product concept may be appro-priate in selecting teaching aids. First, itwill be necessary to test an assortment ofdevices requiring varying amounts of stu-dent contact time to provide a represen-tative sample. From such a sample, an in-

4Essentially the same results were obtained in aninformal experiment conducted during the first yearthat the aids were used.

put-output relationship for teaching aidsmight be estimated to allow relative ed-ucational impacts to be measured.

Conclusions

The general conclusion reached in thisbrief analysis is that role playing, casestudies, and simulation games can all im-prove the effectiveness of learning whenused as teaching aids. The results of theexperiment reported here indicate thatthese techniques improve students' under-standing of complicated material whencombined with lecture presentations.However, the amount of impact eachtechnique has on students varies directlywith the amount of time required, aswould be expected.

Role playing helps in giving students abetter understanding of the decision-mak-ing environment, but does not allow formuch detail in the information presentedor in the variables being studied. Studentsare exposed to the stresses of a particularsituation of interest, yet the exercise lastsonly a short time.

Case studies can present detailed infor-mation about situations, but often do notallow students to "feel" the decision mak-er's position. It is difficult for students tofully understand the nature of the envi-ronment being described in a case if thosestudents are analyzing the case data in arelaxed, academic atmosphere while hav-ing no personal stake in the outcome ofdecisions being made. Therefore, casestudies can be useful when an instructor'sgoals include presenting large amounts oftechnical data in the form of a problemto be solved, or when placing some em-phasis on the mechanics of the manage-ment/decision-making process itself.

Computer-based simulation games givestudents both repeated exposure to the de-cision-making environment and the op-portunity to deal with a large amount ofdetailed information. While participating

60

July 1985


in a simulation game, students must worktogether to make decisions concerning awide range of variables over a period oftime in a competitive environment. Thismeans that students must analyze detailedinformation, as in case studies, but theymust do so in an environment filled withreal stresses, just as in role-playing situa-tions.

The results of this study do not meanthat simulation games are considered tobe more valuable than either case studiesor role playing as teaching aids. There isa definite trade-off between student con-tact time requirements and the effect ofthese teaching aids. The simulation gamehad the greatest total effect here, but sim-ulations often require previous exposureto certain topics and/or methods of anal-ysis. As a result, games such as the onedescribed here may not be appropriate foruse in introductory level courses. Also, thetime required to conduct a simulationgame may not be justified when attempt-ing to illustrate relatively minor points.Case studies and role playing, however,require less time and can be used in manydifferent situations with little difficulty,but have less total effect. Yet, the returnon time invested was found to be highestfor role playing and lowest for the game.This result is the reverse of the orderingof the impacts for the aids when total ef-fects were measured. Therefore, it is clearthat all three teaching aids can be valu-able additions to a program of instructionwhen used in appropriate situations.

The limitations of this study center onthe unresolved problems of measuring alloutputs and assigning them weights. Also,different teaching aids have different im-pacts on different students, so the distri-bution of outputs needs to be measured.As suggested in the analysis section, re-search efforts need to be focused on theinput-output relationship for teaching aidsso that guidelines can be developed to as-sist in choosing effective devices for par-

ticular situations. In this way improvededucational impact models will becomeavailable to teachers as tools for use in im-proving the quality of the learning envi-ronment.

References

Baker, G. and E. Babb. "Managerial Goals and FirmPerformance: A Laboratory Experiment." NorthCentral Journal of Agricultural Economics,6(1984): 88-94.

Black, R. "Psychodrama in Classroom Teaching."Improving College and University Teaching,26(Spring 1978): 118-20.

Blank, S. Practical Business Research Methods. AVIPublishing Company, Westport, Connecticut, 1984.

Boehlje, M. and V. Eidman. "Simulation and Gam-ing Models: Application in Teaching and Exten-sion Programs." American Journal of AgriculturalEconomics, 60(1978): 987-92.

French, C. "Selected Alternative Programs forBringing the Real World to the UndergraduateClassroom." American Journal of AgriculturalEconomics, 56(1974): 1163-75.

Joyce, B. and B. Showers. "Improving InserviceTraining: The Messages of Research." EducationalLeadership, 37(1980): 379-85.

Kendrick, J. "Techniques for Motivating Students."American Journal of Agricultural Economics,55(1973): 762-66.

Kolb, D. "Individual Learning Styles and the Learn-ing Process." Sloan School Working Papers No.535-7, Massachusetts Institute of Technology, 1971.

Ladd, D. "Notes on Use of the Case Method." Paperdistributed by the Intercollegiate Case ClearingHouse, 1978.

Litzenberg, K. "Computer Use in the AgriculturalEconomics Classroom." American Journal of Ag-ricultural Economics, 64(1982): 970-77.

O'Connor, C. and D. Osterman. "Innovations inTeaching: An Introduction." American Journal ofAgricultural Economics, 60(1978): 985-86.

61

Blank


Roberts, D. and H. Lee. "Personalized Learning Pro-cesses in Agricultural Economics." American Jour-nal of Agricultural Economics, 59(1977): 1022-26.

Siegfried, J. and R. Fels. "Research of Teaching Col-

lege Economics: A Survey." Journal of EconomicLiterature, 17(1979): 923-69.

White, F. "Simulation of Agricultural Policy For-mation as a Teaching Approach." American Jour-nal of Agricultural Economics, 59(1977): 225-27.

62

July 1985

effectiveness of role playing, case studies, and...

Documents