Copyright 2002 Psychonomic Society Inc 672

Psychonomic Bulletin amp Review2002 9 (4) 672-705

Behaving in an unusual variable or unpredictable man-ner is sometimes functional An individual may samplenew paths to reach a goal invoke varied strategies whencompeting with an opponent and generate novel combina-tions of images and concepts when doing scientific or artis-tic work Where does such variability come from Why cansome individuals ldquobe looserdquo avoid ruts and engage innovel actions whereas others seem set in their ways Howdo we increase or decrease variability when it is importantto do one or the othermdashto behave in a nontraditional or cre-ative way when that is needed and to repeat a practicedpredictable response when situations so demand One an-swer to these questions has rarely been examinedmdashnamelythat behavioral variability is controlled by its conse-quences In other words variability can be directly rein-forcedmdashincreased maintained or decreasedmdashthrough thepresentation of reinforcers contingent on levels of variabil-ity To state it yet another way variability is an operant di-mension of behavior molded by its consequences learnedor emitted voluntarily Of course there are other influenceson variability including noise inherent in all physical thingsand accidents in the environment However an under-standing of operant variability may help us to explain manyfascinating aspects of thought and behavior

In this article evidence for operant variability its rela-tions to other sources of variation and its functions ex-planations and implications are examined Some of the

topics covered are the following Variability is controlledby discriminative stimuli and reinforcing consequenceswhich are characteristics of other operant dimensionsThe precise contingencies make a difference in the lev-els and types of variation that are produced People andanimals choose whether when and how much to varymuch as they choose among any other set of operant al-ternatives Reinforcement of variations facilitates acqui-sition of new responses especially difficult-to-learn onesCreativity exploration and problem solving may dependpartly on operant variability Some psychopathologiessuch as attention deficit hyperactivity disorder (ADHD)depression and autism may manifest abnormal variabil-ity but reinforcement can produce more normal levelsReinforced variability may also help to explain the volun-tary nature of operant behavior These and other topicswill be explored but first some terms will be defined

DefinitionsOperant dimensions are response attributes or param-

eters that both influence reinforcers and are influenced bythem Examples include response rate force duration lo-cation topography and as will be seen variability Al-though we often refer to ldquooperant responsesrdquo in fact whatis generally meant is a combination of dimensionsmdashforexample a press of the left lever of at least a 10-g forcewith an interresponse time (IRT) of no more than 1 secVariability implies dispersion and unpredictability but itcan also refer to a continuum ranging from repetitive atone end to stochastic at the other Context will indicatethe intended meaning The terms stochastic and randomwhich will be used interchangeably imply the highestvariability with the probability of a response being in-dependent of prior responses Both terms require pre-cisely defined sets of possibilities for example a coin

Much of the research from my laboratory was supported by the Na-tional Science Foundation I thank my many Reed College studentco-workers for their skill creativity persistence and good humor I alsothank Armando Machado and Seth Roberts for helpful comments andJohn Wixted for his gentle persistence in getting me to write this paperCorrespondence concerning this article should be sent to A NeuringerReed College Portland OR 97202 (e-mail allenneuringerreededu)

Operant variability Evidence functions and theory

ALLEN NEURINGERReed College Portland Oregon

Although responses are sometimes easy to predict at other times responding seems highly variableunpredictable or even random The inability to predict is generally attributed to ignorance of control-ling variables but this article is a review of research showing that the highest levels of behavioral vari-ability may result from identifiable reinforcers contingent on such variability That is variability is anoperant Discriminative stimuli and reinforcers control it resulting in low or high variability depend-ing on the contingencies Schedule-of-reinforcement effects are orderly and choosing to vary or repeatis lawfully governed by relative reinforcement frequencies The operant nature of variability has im-portant implications For example learning exploring creating and problem solving may partly de-pend on it Abnormal levels of variability including those found in psychopathologies such as autismdepression and attention deficit hyperactivity disorder may be modified through reinforcement Op-erant variability may also help to explain some of the unique attributes of voluntary action

OPERANT VARIABILITY 673

has two possible states heads up and tails up Novel re-sponses are those that have not previously been ob-served and the term generally refers to the behavior of agiven organism although sometimes it refers to that of agroup or culture Creative responses are novel or vari-able ones that meet an additional criterion often an aes-thetic (eg artistic) one but sometimes a functional (egscientific) one Lastly many of the experiments to be re-viewed compare variable and repeated operants an exam-ple of the latter being repeated responses by a rat on a sin-gle lever under the control of reinforcement schedules

ObjectionsObjections are often raised to the suggestion that vari-

ability can be reinforced For example a reinforcer af-fects the response that produces it but variability refersto a set of responses not a single one How can a set ofevents possibly extending far back in time be rein-forced Another objection is that reinforcement by itsnature organizes and strengthens responses and rein-forcement of variability is therefore not possible An-other is that attempting to reinforce variability is incon-sistent with two basic goals of behavioral psychologistsmdashnamely to predict behavior and to control it Another isthat good experimental procedures should minimizevariability not increase it Yet another is that variabilityis perceived or posited only when an observer is igno-rant of its underlying causes Finally adherents of a de-terminist philosophy may argue that variability does notexist For these reasons reinforcement of variability issaid to be undesirable perhaps impossible and forsome not meaningful The evidence as we will see in-dicates otherwise The study of reinforced variabilityprovides a new way to view operant behavior and opensnew areas of research

INITIAL STUDIES

Reinforcement of Variable and Novel ResponsesThe proposal that variable behaviors can be reinforced

goes back at least 40 years but its history has been a con-tentious one In this section some of that history and thesurrounding debate will be described Maltzman (1960)was among the first to suggest that human subjects couldlearn to generate low-probability responses in this caseverbal responses to stimulus words Experimental sub-jects were asked to devise many different associates tothe same word whereas a control group had to produceonly one Because of their training the experimentalsubjects came to generate lower probability associatesthan did the controls Although these findings imply thatlow-probability responding can be trained there was noclear evidence that reinforcement played a role

However reinforcement was clearly involved whenPryor Haag and OrsquoReilly (1969) rewarded porpoises forjumping turning or swimming in novel ways In eachsession the trainer waited for a response that had notpreviously been observed and that response became the

reinforced target throughout the session The porpoisescame to emit novel and complex behaviors wheneverthey entered the experimental context Indeed they be-haved in ways never before seen in the species This wasan influential study and it led to additional research

Goetz and Baer (1973) for example rewarded preschoolchildren for block-building constructions that differedfrom one another A particular form was reinforced uponits first occurrence within a session but in contrast tothe procedures used by Pryor et al (1969) many differ-ent forms were reinforced during each session As train-ing proceeded the children constructed increasingly dif-ferent forms including novel ones When reinforcerswere later contingent upon repetitions the children re-peatedly constructed a single form Thus variability andrepetition were controlled by consequences

Although Maltzman (1960) Pryor et al (1969) andGoetz and Baer (1973) advantageously studied naturallyoccurring behaviors some problems were also identi-fied In Pryor et al the responses became so complexthat the experimenters had diff iculty distinguishingamong them In both Pryor et al and Goetz and Baer theexperimenters had to identify each form or response pat-tern and decide when a new one had occurred prior toeach reinforcement In all three studies novel responsesbecame increasingly diff icult since the ldquoeasyrdquo oneswere emitted first Also in the Maltzman and the Goetzand Baer studies verbal feedback was provided and thecontribution of descriptive information was difficult todistinguish from that of reinforcement Alternativemethods were needed especially if the goal was to ana-lyze reinforced variability experimentally and Blough(1966) provided an ingenious one

Reinforcement of Variable IRTsBlough (1966) rewarded pigeons for pecking a re-

sponse key if the time between consecutive pecks orIRT had occurred least frequently in the recent past Thisconstituted an attempt to see if pigeons could learn to re-spond randomly over time similarly to what takes placeduring the emission of atomic particles at random inter-vals during radioactive decay The birds were surpris-ingly successful To get a sense of the reinforcementcontingencies imagine a stochastic process in which re-sponses occurred with a 05 probability during each sec-ond since the previous response Response probabilitywould be 05 during the first second and if a responsehad not occurred it would be 05 during the next secondand so on Given 1000 responses approximately 500would occur during the first second since the previousresponse that is in the 0ndash1-sec time frame or whatBlough referred to as a ldquobinrdquo 250 responses would occurin the 1ndash2-sec period since the previous response 125 inthe 2ndash3-sec bin and so on Blough created 16 IRT binsadjusting their sizes so that a random generator wouldproduce equal numbers in the counters associated witheach bin Because as has just been described manymore short than long IRTs are emitted bin size was small

674 NEURINGER

at the short end and large at the long end and increasedsystematically across the IRT range For the response tobe reinforced a pigeonrsquos IRT had to fall in the bin thatcontained the fewest prior entries across a moving win-dow of 150 responses Fine-grained analyses of the pi-geonsrsquo performances showed that successive responseswere not truly independent as would be expected from arandom generatormdashthe tendency for long IRTs to followlong ones and for short IRTs to follow short ones wasgreater than would be expected from a random emittermdashbut this might have been due to particular aspects of theprocedure including the facts that short IRTs were rein-forced somewhat more frequently than longer ones andthat if there were few entries in a bin (because the birdhad not been responding at that IRT) then successiverepetitions of the same IRT could be reinforced repeat-edly Despite these problems the data were impressivelyclose to those of a random model as is shown in the toppanel of Figure 1 To the extent that the functions are lin-ear on the semilogarithmic coordinates the behavior ap-proximates that of a stochastic process The advantagesof Bloughrsquos procedure were that response classes weredefined objectively and independently of a human ob-server reinforcers were administered automatically thou-sands of responses were emitted per session and ses-sions could continue indef initely without changes inresponse difficulty or contingency

Shimp (1967) attempted to extend Bloughrsquos (1966) pro-cedure to pigeon choices Because choices are currently amajor focus of interest and because there were problemswith Shimprsquos procedure it will be discussed in some de-tail Pigeons pecked two keys left (L) and right (R) in acontinuous stream such as LLLLRRLLRRRLRLLL thedata were analyzed on line in terms of overlapping setsof four responses In the example just given assume thateach new response is added to the right of the string Themost recent set was composed of RLLL the second mostrecent (moving to the left by one response) of LRLL thethird of RLRL and so on Given two alternatives L andR and sets consisting of four responses there are 24 or16 possible sequences LLLL LLLR LLRL RRRRReinforcement was provided if the most recent responsehad terminated a sequence that had occurred less fre-quently than all other 15 possible sequences over a mov-ing window of 96 responses Although the resultingchoice distributions were quite variable in some respectsthe results did not match those of a random model andwere diff icult to analyze possibly because of the fol-lowing problem

Assume that 16 counters represent the 16 sequences ofresponses and that one count is added to each counterwhenever its associated sequence is emitted If LLLLhas just been emitted one count will be added to thatcounter Another L will increase the same counter andeach successive L will add another If LLRR has justbeen emitted however thereby adding one count to itsassociated counter four additional responses (not 1 as inthe LLLL case) will be required to add again to the same

counter Therefore 16 successive responses will be re-quired for four instances of LLRR (namely LLRRLLR-RLLRRLLRR) as opposed to seven for L (LLLLLLL)Although a random process would generate equal fre-quencies of all sequences including LLLL and LLRRthe distribution of successive occurrences would differthereby affecting the distribution of reinforcements Thatis LLLL and RRRR would tend to be more ldquoclumpedrdquothan would LLRR and RRLL which would be more dis-tributed Similar considerations apply to all other se-quences That these differences might affect choice dis-tributions is indicated by the many studies showing thatthe distribution of reinforcements over time given equal

Figure 1 (A) Data from one pigeon in Blough (1966) under dif-ferent parameter conditions (the three sets of lines) with replica-tion at each parameter Shown are the mean numbers of re-sponses emitted in each half-second IRT bin a straight line beingexpected from a stochastic responder Adapted from ldquoThe rein-forcement of least frequent interresponse timesrdquo by D S Blough1966 Journal of the Experimental Analysis of Behavior 9 p 587(B) Data (points) from one pigeon at the end of training in Machado(1989) The line represents the frequencies expected of a randommodel From ldquoOperant conditioning of behavioral variabilityusing a percentile reinforcement schedulerdquo by A Machado 1989Journal of the Experimental Analysis of Behavior 52 p 161 Bothfigures copyrighted by the Society for Experimental Analysis ofBehavior Reprinted with permission

OPERANT VARIABILITY 675

average reinforcement frequencies affects choices for ex-ample choices are affected differently by variable-interval(VI) than by fixed-interval (FI) schedules of reinforcement

Wilson (1986) devised one solution to this problemWilsonrsquos procedure is complex but in brief responding wasanalyzed in a manner that avoided overlapping sequences(which was the cause of the clumping in Shimprsquos 1967procedure) Four sets of counters were created Using thesame example given above in the first set one count wasadded to each counter representing RLLL RRRL RRLLand LLLL so that successive trials were completely inde-pendent (Note the absence of overlap) When the next re-sponse was emitted sequences began with that responsemdashlet us assume that the next response added to the rightof the series was an Lmdashand therefore in the examplegiven the second set was created by offsetting responsesone place to the left Now one was added to LLLL withprevious counts added to RRLR RLLR LLLR and soon A similar procedure was followed for each of the re-maining two sets Each response therefore terminated afour-response sequence in one and only one of the foursets and reinforcement was based on satisfaction of thecontingency within that set only Wilsonrsquos procedure hadother aspects but for present purposes it suffices to notethat the dependencies in Shimprsquos procedure were avoidedand when pigeons were tested under these contingen-cies they generated choices that more closely approxi-mated those of a random model Hopefully this lengthyexplanation indicates the importance of precise contin-gencies when one is attempting to reinforce variabilityAn advantage of Shimprsquos and Wilsonrsquos procedures wasthat every response could be reinforced at least poten-tially as was the case for each of the other procedures dis-cussed until now However a simpler way to avoid theproblems just described is to require a fixed number ofresponses in each trial with reinforcement possible onlyat completion of the trial As will be seen most recentstudies have used the latter tactic

The evidence generally supported the claim that be-havioral variability could be reinforced but the agree-ment about these findings was short lived Schwartz (19801982) reached the opposite conclusion and his researchwill be considered in detail

Schwartzrsquos ResearchSchwartz (1980 1982) used fixed-length trials con-

sisting of eight responses per trial by pigeons across tworesponse keys During an initial control phase a trial ter-minated with food if it contained exactly four L and fourR responses regardless of the particular sequence of Lsand Rs However if either key was pecked a fifth timethe trial terminated immediately with timeout A 5 3 5light matrix was provided to assist the birds in this taskAt the beginning of a trial the upper-left corner light wasilluminated and all others were off An L peck movedthe light one column to the right and an R peck movedit one row down Any sequence of four L and four R re-sponses would therefore move the light from upper left

to lower right and the trial would end with food Each ofthe birds came to emit a single sequence with high prob-ability in most cases LLLLRRRR or RRRRLLLL Arelatively low success rate at the end of 40 training ses-sions indicated that the task was harder than it mighthave appeared Only 66 of trials ended with food andthe remaining 34 ended with timeout because withina given trial one of the keys had been pecked a fifth time

Schwartz (1982) then required that the sequence in thecurrent trial differ from that of the just preceding trialThis will be referred to as a lag 1 variability + constraintcontingency (ldquoconstraintrdquo indicating the required fourLs + four Rs) If a bird had emitted LLLLRRRR in theprevious trial for example any other sequence would bereinforced in the next trial assuming that it containedfour Ls and four Rs but a repetition led to timeout Thebirds performed poorly only 36 of the trials now end-ing with food because a dominant sequence was oftenrepeated or one of the keys was pecked five times Hy-pothesizing that this failure to reinforce variability mayhave been due to the birdsrsquo extended experience underthe initial control condition (in which variations were notrequired and any sequence of four Ls and four Rs wasreinforced) Schwartz (1982 Experiment 1) again triedthe variability-requiring procedure with naive subjectsAgain the birds failed In another experiment Schwartz(1982 Experiment 2) explicitly trained birds to alternatebetween LLLLRRRR and RRRRLLLL sequences andthen introduced the lag 1 variability + constraint contin-gency but again the birds failed to vary Schwartz (1982)concluded that variability could not be reinforced andoffered a number of possible explanations for why his re-sults differed from the previous findings Perhaps Maltz-manrsquos (1960) and Goetz and Baerrsquos (1973) findings weredue to the information providedmdashverbal indicationsfrom the experimenters that subjects should varymdashrather than to reinforcement Perhaps Pryor et alrsquos (1969and by implication Goetz amp Baerrsquos 1973) results weredue to brief extinction periods in which reinforcers werewithheld (because variations were not forthcoming)rather than to direct reinforcement of variability PerhapsBloughrsquos (1966) results were specific to IRT and not rel-evant to response forms or choices These explanationsall imply that it is not possible to reinforce variability orthat it is possible only in narrowly defined cases As willbe seen Schwartzrsquos (1982) results readily met the test ofreplication but there was a problem nevertheless withhis conclusion

Schwartz ExplainedPage and Neuringer (1985) simulated Schwartzrsquos (1982)

lag 1 variability + constraint procedure with random ldquore-sponsesrdquo generated by a computer Although the simula-tion generated L and R with equal probabilities (05) ldquore-inforcementrdquo was obtained on only 29 of its trials whichwas somewhat lower than the level attained by pigeonsThe reason is that of the 256 possible eight-response se-quences only 72 contained exactly four Rs and four Ls

676 NEURINGER

and therefore the random simulation often failed tomeet the 4L + 4R constraint The implication was that ifthe birds attempted to respond randomly they would bereinforced only rarely

To test whether the penalty for fifth pecks was respon-sible for the birdsrsquo failure to vary Page and Neuringer(1985) permitted eight responses across L and R keyswithout the four responses-per-key requirement If thecurrent sequence of eight responses differed from that inthe last trial a lag 1 variability contingency (without con-straint) reinforcement was provided Now the pigeonssucceeded in varying attaining reinforcement on morethan 90 of the trials a rate significantly higher thanthat observed in Schwartz (1982) and close to that of thesimulating random responder under the new procedurePage and Neuringer also replicated Schwartzrsquos (1982)variability + constraint procedure and again the pigeonsfailed just as did Schwartzrsquos The evidence was clearUnder Schwartzrsquos (1982) procedure pigeons failed tovary but penalty for fifth responses was responsible

One additional aspect of Schwartzrsquos (1982) proceduremight also have contributed to low variability If thebirds intermixed a single preferred sequencemdashone thatmet the 4L + 4R constraintmdashwith any other then 50 ofthe trials would end with food this level of intermittentreinforcement being sufficient to maintain responding inmany other situations For example if a bird alternatedbetween LLLLL (the fifth L producing a timeout) and adominant LLLLRRRR sequence the latter would alwaysresult in reinforcement Birds under the variability +constraint contingencies in fact emitted a dominant se-quence with high probability and many of their rewardswere obtained following that sequence (because it hadbeen preceded by a different sequence often one con-taining five Ls or five Rs) Thus failure to vary could beexplained by the fact that variability was not differen-tially reinforced whereas a single dominant sequencewas Schwartz (1988) later reviewed this new evidenceand concluded that pigeons could indeed be reinforcedfor responding in random-like fashion

Yoke ControlOne of Schwartzrsquos (1982) hypotheses is worth further

consideration When variability is reinforced there areperiods during which reinforcement is withheld becauseresponses did not satisfy the contingencies As will bediscussed below many studies have shown that extinctionelicits variability Might short-term extinction periodswhen reinforcers were not forthcoming have elicited thevariability attributed to reinforcement

Page and Neuringer (1985) compared two conditionsIn one referred to as Var reinforcement depended on se-quence variations similar to the lag 1 variability contin-gencies described above (The matrix of lights was nolonger used because it was found not to contribute to re-sponse variability) In the other condition referred to asYoke exactly the same frequency and distribution of re-inforcements were provided but independently of se-

quence variations This was accomplished by having thecomputer maintain a record of which trials were rein-forced and which led to timeout in the Var phase andusing this record to program reinforcers during Yoke Fora particular bird for example the first as well as the sec-ond trials in a Var session might have been reinforcedbut in the third trial the bird may have repeated a se-quence causing a timeout and so forth In the Yoke phasethe first and second trials would be reinforced but thethird trial would lead to timeout whether the birdrsquos re-sponses varied or not Each birdrsquos own sequence of rein-forcements and timeouts in Var was therefore replicatedin Yoke If short-term extinction or withholding of rein-forcement was responsible then variability should beequal in the Var and in the Yoke phases That was not theresult Variability was significantly higher in Var than inYoke a finding that has been confirmed many times inour laboratory and in those of others (eg Barba amp Hun-ziker 2002 Machado 1992) We conclude that variabil-ity was in fact reinforced

VARIABILITY IS AN OPERANT

Control by reinforcement is the most notable charac-teristic of an operant response but there are others includ-ing precise influence of contingencies effects of rein-forcement frequencies control by discriminative stimulilawful distribution of choices and extinction Researchin each of these areas supports the hypothesis that be-havioral variability is an operant dimension similar toother operant dimensions An organism can be rein-forced not only for repeating a particular behavior but fordistributing responses across a wide array of possibilitiesin a way that may be quite unpredictable The evidencewill be reviewed

Contingencies and Frequencies of ReinforcementContingencies specify what must be done to earn re-

inforcement When applied to variability contingenciesspecify a set of responses and in most studies to date aminimal level of variability within that set That controlby contingencies does not depend on reinforcement fre-quency was demonstrated by Blough (1966) for pigeonIRTs and by Machado (1989 1992) for pigeon responsesequences across L and R keys In both cases when re-inforcement depended on high variability response vari-ability was high when intermediate variability was re-quired the level generated was intermediate and whenlower levels of variability were reinforced variabilitywas relatively low Reinforcement frequencies were heldconstant in these studies and therefore were not respon-sible for the changes in the levels

With respect to other operant dimensions such as re-sponse rate reinforcement frequency often has complexand sometimes inconsistent effects As reinforcementfrequency increases response rates sometimes increasesometimes decrease and sometimes are unchanged Vari-ability is also affected in seemingly inconsistent ways

OPERANT VARIABILITY 677

Some studies report that variability increases when rein-forcement frequencies decrease (Boren Moerschbaecheramp Whyte 1978 Tatham Wanchisen amp Hineline 1993Tremont 1984) but others report small or no effects(Blough 1966 Eckerman amp Lanson 1969 Herrnstein1961 Machado 1989)

Grunow and Neuringer (2002 Experiment 1) inde-pendently manipulated variability contingencies and re-inforcement frequencies and their results may help toexplain the previous inconsistencies Four groups of ratswere studied with the contingencies for each group re-quiring a different minimum level of variability (A thresh-old contingency was employed the details of which willbe described below) In the initial phase reinforcementwas provided whenever the respective contingencieswere met (reinforcement frequency was relatively highin this phase) and as can be expected from the Blough(1966) and Machado (1989) studies described above lev-els of variability were directly controlled by the contin-gencies In a second phase with the same variabilitycontingencies held constant reinforcement was limitedto no more than once per minute (VI 1 or intermediatefrequency) In a third phase meeting of the variabilitycontingency was reinforced on an average of only once

every 5 min (VI 5 or low frequency) We observed a sig-nificant interaction between reinforcement frequencyand variability contingency (Figure 2) which helps toexplain some of the previous inconsistencies In thegroup that was reinforced for low variability representedby the triangles in Figure 2 as reinforcement frequencydecreased variability increased slightly in the group re-inforced for high variability represented by the squareswhen reinforcement decreased variability decreasedand in the intermediate groups variability was unchangedAlthough these effects were small they reached statisti-cal significance As can be seen in Figure 2 variabilitycontingencies affected variability more than did rein-forcement frequencies and effects of the latter were in-fluenced by the contingencies

An example might help to illustrate these complexfindings When low variability is required perhaps as in assembly-line work low variability emerges When highvariability is required perhaps as in artistic work re-sponse variability is high Now what happens if the normalrate of reinforcement for each of these tasks is loweredAccording to Grunow and Neuringerrsquos (2002) findingsthe assembly-line workerrsquos variability would increasebut only slightly and the artistrsquos variability would de-

Figure 2 Average U-values across groups of rats in Grunow and Neuringer (2002)The highest function (squares) is from a group reinforced for highest sequence vari-ability the lowest function (triangles) is from a group reinforced under a lenient con-tingencymdashthat is low variability sufficed for reinforcement the other groups (Xs andcircles) were reinforced for intermediate levels of variability Each group was rein-forced at three frequencies each time the contingency was satisfied (continuous rein-forcement ie CRF) on an average of not more than once per minute (VI 1) and onan average of not more than once every 5 min (VI 5) High U-values indicate high lev-els of sequence variability and low U-values indicate relatively repetitive respondingError bars represent standard errors of the mean

678 NEURINGER

crease somewhat Importantly the artist would continueto behave much more variably than the assembly-lineworker Both contingency and frequency affect responsevariability but contingency appears to be more influen-tial at least over the range of values studied to date Thesefindings if general are important for those who withholdreinforcement in order to encourage response variationsas is often the case when behavior modifiers attempt toshape new behaviors Effects of withholding may dependon baseline contingencies and a more productive proce-dure might be to reinforce variability directly We willconsider this suggestion further below

Discriminative ControlThe operant response as defined by Skinner is con-

trolled by discriminative stimuli as well as by conse-quences For example red traffic lights are the discrimi-native stimulus for depressing the automobile brake thedistance of the golf ball from the hole provides a discrim-inative stimulus for how hard to hit the ball In the labora-tory discriminative control is often studied with multipleschedules in which one reinforcement contingency suchas fixed ratio is operative in the presence of one stimulussuch as a red keylight and a different contingencymdashforexample extinctionmdashis operative in a second stimulusmdashfor example a green keylight Discriminative control isshown by differences in responding Variability is alsocontrolled by discriminative cues

Exteroceptive discriminative stimulus controlPage and Neuringer (1985 Experiment 6) reinforced pi-geons for repeating a single sequence LRRLL in thepresence of blue keylights and for variable sequences inthe presence of red Blue and red alternated after every10 reinforcements The birds learned to repeat in thepresence of blue and to vary in the presence of red andwhen the stimulus relationships were reversed the birdsreversed their patterns Cohen Neuringer and Rhodes(1990) obtained similar results with rats Denney andNeuringer (1998) extended this research to show thatdiscriminative control did not depend on reinforced repe-titions in one of the components Again within the contextof a multiple schedule rats were rewarded for variablesequences under one stimulus (Var) and independentlyof variability under a second stimulus (Yoke) Frequen-cies of reinforcement were identical in the two stimulusconditions That the stimuli exerted significant controlover variability was indicated by three main findingsVariability was higher in Var than in Yoke differences invariability were greatest immediately following stimuluschange and when the discriminative stimuli were re-moved and all other aspects of the program kept constantvariability immediately converged on an intermediate andidentical level in the two components The conclusion isthat operant variability is controlled by discriminativestimuli These findings are important in terms of both ex-plaining observed levels of variability and controllingthem For tasks in which response variations are impor-tant such as when new operants are being shaped or when

an individual is attempting to behave creatively or solveproblems cues that indicate ldquoVaryrdquo can be helpful

Discriminative control without exteroceptive stim-uli The relationship between a response and a reinforcercan itself provide a discriminative stimulus even whenexternal cues such as lights or tones are unavailable Forexample under an FI schedule there is a period duringwhich reinforcers cannot be obtained The subject learns topause following each reinforcement although no cuesother than the reinforcer are provided Another case is themixed schedule identical to the multiple schedule describedabove but with no external cues Animals sometimeslearn to respond differentially to the consequences oftheir behaviors even under such stimulus-impoverishedconditions (Ferster amp Skinner 1957) The question stud-ied next was whether responsendashreinforcer relationshipscan provide discriminative cues sufficient to control lev-els of variability This is an important question becauseas was indicated above it has been assumed that with-holding reinforcement elicits variability The present hy-pothesis is that absence of reinforcement can sometimesserve as a discriminative cue for operant variability Thedistinction between elicited (respondent) and reinforced(operant) variability will be revisited shortly

Hopson Burt and Neuringer (2002) rewarded ratsunder a mixed schedule in which two periods of time orcomponents Var and Repetition (Rep) alternated with-out any external cues In the Var component four-responsesequences of L and R leverpresses were reinforced ifthey met a threshold variability contingency in the Repcomponent only repetitions of LLLL were reinforced(Probabilities of reinforcement were equalized in the twocomponents by intermittent reinforcement of LLLL inRep) The results demonstrated discriminative controldespite the absence of exteroceptive cues (see Figure 3)With trials since the unmarked switch into the Var com-ponent shown in the left column variability increasedand LLLL sequences decreased with trials since the unmarked switch into Rep shown in the right columnresponding became increasingly repetitive until LLLLsequences predominated The two functions were sim-ilar which is an important finding in that it indicatesdiscriminative control rather than elicitation If variabil-ity had been elicited because reinforcement was tem-porarily withheld (when the contingencies were switched)then variability should have increased following entryinto both Rep and Var components but that was not theresult

Therefore variability that is observed when rein-forcement is withheld for short periods as when a newresponse is being shaped may be partly discriminativelycontrolled despite the absence of external cues that isanimals and people may learn when it is functional tovary and some of the cues may come from responsendashoutcome relationships Control over this operant vari-ability may be more precise than that elicited by extinc-tion because as was noted above operant contingenciesdefine both the set of acceptable variations and the level

OPERANT VARIABILITY 679

of required variability It may also be maintained muchlonger because as will be seen shortly extinction weakensresponding

ChoiceThe frequency of one response relative to others or

response probability is used as a measure of choice orpreference Choices are found to match the probabilitiesof their consequences For example relative frequenciesof pecking one of two keys approximate relative fre-

quencies of reinforcement Neuringer (1992) showedthat the same matching relationship also governs choicesby pigeons to vary or repeat The more variations werereinforced the more the pigeons chose to vary and sim-ilarly for repetitions Each trial consisted of four L andR responses and reinforcements for varying and for re-peating were systematically manipulated The Var con-tingency was met whenever the current sequence dif-fered from each of the previous three sequencesmdasha lag 3criterion The Rep contingency was satisfied if the cur-

Figure 3 The left column shows the probabilities ( y-axis) that each of five rats met a variability contin-gency (solid circles) versus a repetition contingency (open circles) when they were being reinforced for vary-ing (Hopson Burt amp Neuringer 2002) Along the x-axis are trials since the mixed schedule (no cues pro-vided to indicate the component) had switched into the variability-reinforcing component The right columnshows the corresponding data for performances when reinforcement was contingent upon repetitions thex-axis here indicating trials since the unmarked switch into the repetition-reinforcing component

680 NEURINGER

rent sequence repeated at least one of the previous threeOn an average of once every 30 sec (VI 30 sec) the com-puter probabilistically primed reinforcement for Var orRep the prime being maintained until the reinforcementhad been collected Thus if the computer primed Var re-inforcement was delivered whenever the lag 3 variabilitycontingency was next met Similarly if Rep had beenprimed the next Rep sequence was reinforced Proba-bilities of Var and Rep reinforcements were systemati-cally manipulated the two always summing to 10 and amatching-type relationship was obtained (Figure 4panel A) The relationship was more complex than sim-ple matching possibly because if the pigeons chose tovary by responding stochastically there was some prob-ability due to chance that the current sequence wouldrepeat a previous sequence This was shown by a com-parison of pigeon performances to the performance of acomputer-simulated stochastic model programmed tomatch its Var choices to the probability of Var reinforce-ments The pigeonsrsquo choice distributions were found tobe the same as the modelrsquos (Figure 4 panel B) Thuschoices to vary or repeat were governed by relative fre-quencies of reinforcement An animal can choose whetherwhen and how much to vary a result consistent withthose described above from Blough (1966) Machado(1989) and Grunow and Neuringer (2002) Choices ofwhether or not to vary appear to be governed similarly toall other operant choices

These f indings may have important applicationsWhen an individual repeatedly responds in a way that isnonfunctional reinforcement of choices to vary mayhelp the individual to emerge from the ineffective pat-tern This can be true for those experiencing everydayproblems but feeling helpless to change and it may beespecially important for those experiencing depressionor who respond in otherwise abnormally stereotypedmanners such as those with autism These topics will beconsidered in a later section

ExtinctionAs was indicated above extinction is often found to

increase variability but almost all studies of this effectinvolve responses that had previously been repetitiveAntonitis (1951) for example reinforced rats for pokingtheir noses anywhere along a 50-cm horizontal openingAlthough the exact location of pokes did not matter withcontinued training responses became limited to one or afew locations along the strip When reinforcement waswithheld variability of location increased (see also Eck-erman amp Lanson 1969) Variability produced by extinc-tion has been reported for many other response dimen-sions as well including force (Notterman amp Mintz1965) number (Mechner 1958) topography (Stokes1995) and sequence (Balsam Paterniti Zechowy ampStokes 2002 Mechner Hyten Field amp Madden 1997)

The issue is complicated however by other studiesthat show that previously reinforced responses and re-sponse patterns are maintained intact during and afterextinction That is extinction does not break down

learned patterns For example when Schwartz (1981) re-inforced pigeons for any sequence of four L and four Rresponses each bird developed a dominant sequence asdescribed above and these sequences were maintainedduring extinction Despite decreasing response ratesduring extinction response patterns were retained Sim-ilar maintenance of response structure has been ob-served during extinction following reinforcement of

Figure 4 Average performance across a group of pigeons inNeuringerrsquos (1992) study of choices to vary or repeat (A) Loga-rithms of the ratio of number of times the pigeons met the Varcontingency divided by the times they met the Rep contingencyas a function of the analogous numbers of reinforcements re-ceived for each Points represent group averages in two separateexperiments Also shown is the least squares best-fitting line(B) Changes in U-value an index of response variability as afunction of the obtained percentage of reinforcements for vary-ing The solid line in panel B represents the performance of acomputer-based model that matched its choices to vary (vs re-peat) to its obtained relative frequency of reinforcement for vary-ing From ldquoChoosing to vary and repeatrdquo by A Neuringer 1992Psychological Science 3 p 248 Copyright 1992 by the AmericanPsychological Society Reprinted with permission

OPERANT VARIABILITY 681

choices (Myerson amp Hale 1988) responding underfixed-interval contingencies (Machado amp Cevik 1998)visual discriminations (Nevin 1967) and matching tosample (Cumming Berryman Cohen amp Lanson 1967)How can stability during extinction be reconciled withincreased variability

Neuringer Kornell and Olufs (2001) wrestled withthis question in a study of extinction following rein-forcement under three different contingencies An oper-ant chamber contained two levers left (L) and right (R)on either side of a reinforcement dispenser and a key (K)in the rear of the chamber distant from the dispenserOne group of rats Rep was reinforced for repeating afixed sequence of responses LKR A second group Varwas reinforced for variations across the three operandaunder a threshold contingency Reinforcement for a thirdgroup Yoke was matched to those of the Var animalsand was therefore independent of the particular sequencesBecause of the unusual configuration of operandaskewed distributions of sequence probabilities were ob-served in all of the groups an outcome that was antici-pated and desired in order to document possible in-creases in variability during extinctionmdasha flattening ofthe distributionmdashas well as possible decreases That isduring the reinforcement phase the Rep group came torepeat the LKR sequence with high probability the Yokegroup tended to emit the easiest sequences such as LLLand RRR but distributed their responses over more se-quences than the Rep group did and the Var group dis-tributed most widely although because of differences inthe operanda and in their locations some sequences weremore preferred than others and in particular sequencesending with a K response were least likely (the key beingfarthest from the reinforcement dispenser) Thus thecontingencies resulted in three different distributions orhierarchies of sequence probabilities very steep forRep intermediate for Yoke and flattest for Var

The main finding was that withholding reinforcementhad similar effects in the three groups In each responserates fell to low levels across the four sessions of extinc-tion Furthermore sequences that were most frequentduring the reinforcement phase continued to be most fre-quent throughout extinction A slight flattening of thedistributions was observed however with the lowestprobability sequences increasing relative to the rein-forcement phase Thus because the overall hierarchy ofsequence frequencies was maintained the results indi-cated stability but because the distributions were some-what flattened the results also showed slightly increasedvariability It is as if the strategy during extinction wereto keep doing what had worked in the past but occa-sionally to do something very different This compli-cated story integrates two effects of extinction increasedvariability and maintained stability Most importantly forthe present discussion extinction had similar effects onvariable yoked and repetitive operants

One additional comparison is of interest Recall thatGrunow and Neuringer (2002) found that when contin-

gencies demanded high variability obtained levels ofvariability decreased slightly as reinforcement frequen-cies were lowered This result is opposite to the Vargrouprsquos slight increase in variability just described underextinction conditions The reason may be that extinction-induced variability is short lived lasting only as long asresponding continues perhaps for hours or a few ses-sions at most whereas the variability engendered by theVI 5 intermittent schedule of reinforcement in Grunowand Neuringer can be maintained indefinitely The de-crease in Grunow and Neuringerrsquos operant variabilitymay be due to the weakening of the reinforcement effect(ie reinforcement is less frequent) whereas the in-crease following extinction may be a short-term elicitedeffect This distinction between reinforced and elicitedvariability is important and will be discussed further inthe next section

Operant and Respondent InteractionsElicited or what is sometimes referred to as respondent

effects occur when environmental changes that affectbehavior do not depend on (are not caused by) the behav-ior Pavlovian conditioning being the most commonlystudied example Operant effects are observed when a re-inforcer depends on the response to produce it Althoughfor historical reasons respondent and operant effects areoften studied separately most behaviors result from acombination or interaction of the two Variability is nodifferent and we turn to an example of such an interaction

As reinforcement is approached under FI or fixed-ratio (FR) schedules response rates are commonly ob-served to increase Similarly when reinforcement de-pends on accuracy as in discrimination learning ormatching to sample it too is found to increase within theFI or the FR (Nelson 1978 Nevin 1967) Cherot Jonesand Neuringer (1996) asked whether the same patternheld for reinforced variability Does operant variabilityincrease as reinforcement for varying is approachedThe question may be related to such real-world cases aswhen a composer is rushing to complete a compositionbefore a scheduled performance or when a programmeris trying to solve a problem before an upcoming meetingThese activities involve at least to some extent non-repetitive original or variable thought and behaviorThe question is whether variability increases with prox-imity to or anticipation of reinforcement for varying

Cherot et al (1996) employed trials consisting of fourresponses across L and R operanda and a lag 4 variabil-ity contingency with rats in one experiment and pigeonsin another Unlike in any of the cases described so farhowever the lag-variability contingency had to be satis-fied four times in order for a single reinforcer to be ob-tained With ldquordquo indicating successful completion of theVar contingency ldquo2rdquo indicating failure to meet that con-tingency and PELLET indicating the food reinforcer anexample follows LRLL RRRL LRLL2 LRLL2RLLR RRRL2 LRRR+PELLET Thus food reinforce-ment was provided only after the subject had satisfied

682 NEURINGER

the variability contingency four times FR 4 The resultswere that as the food reinforcer was approached (ie asprogress was made through the ratio) the subjects wereless and less successful at varying The probability ofmeeting the variability contingency in the first trial afterreinforcement was higher than it was in the second whichin turn was higher than that in the third trial and so onFor comparison a Rep group was reinforced for repeatingsequences once again with reinforcement only after foursuccessful repetitions Probability of repeating increasedwith proximity to reinforcement a finding consistentwith the literature but opposite that for the variabilitygroup Thus approach to reinforcement facilitated oper-ant repetition but interfered with operant variabilitySimilar variability-interfering effects have been docu-mented for within-trials responses The probability that aresponse repeats the just prior response increases within atrial even when sequence variability is reinforced (McElroyamp Neuringer 1990 Neuringer 1991) To return to the ex-ample of a composer rushing to complete her composi-tion she may indeed have increasing difficulty respond-ing creatively under the pressure of a deadline On theother hand the soloist may find it easier to repeat thecorrect performance during practice periods as the con-cert nears How can these results be reconciled with themany findings that variability can be reinforced

An additional result of Cherot et al (1996) provides ananswer The Var animals responded much more variablyoverall than did the Rep animals Although variability inthe Var groups decreased with approach to the reinforceroverall variability was always much higher when it was re-inforced (Var group) than when it was not (Rep group)Reinforcement therefore exerts two influences Overallvariability is elevated sometimes to the highest levelswhen it is reinforced but approach to (and possibly focuson) the reinforcers constrains or lowers variability Sim-ilar reinforcement-produced interference effects had beenreported previously for example when raccoons are re-inforced for dropping a ldquocoinrdquo into a slot the reinforce-ment results in the animalsrsquo holding and rubbing the coinrather than dropping it (Breland amp Breland 1961) Theoverall enhancement of variability was of much greatermagnitude in Cherot et al than the decrease with approachwas although both effects were statistically significant(The notes created by the composer under pressure ofdeadline will be less readily predicted than will thoseplayed by the performer practicing a well-known piece)This study highlights the importance of interactions be-tween operant effects of reinforcement and elicited effectsproduced by anticipation of reinforcement and the resultsmay be relevant to the debate concerning possible detri-mental effects of reinforcement on creative behaviors (seethe discussion below and Neuringer in press)

SummaryReinforcement controls behavioral variability as it

does other operant dimensions Levels of variability areprecisely governed by schedule contingencies and dis-

criminative stimuli Choices to vary or repeat match rel-ative frequencies of reinforcement Extinction increasesvariability slightly whereas the hierarchies of sequenceprobabilities are retained and operant and respondenteffects interact The evidence shows that behavioral vari-ability is an operant controlled by its consequences

GENERALITY AND PARAMETERS

Many different methods measures species and re-sponses have been used in the study of operant variabil-ity and this diversity supports the generality of the phe-nomenon As is the case for repeated operants howeverthere are parametric constraints This section briefly re-views the generality and some of the parameters studiedto date

GeneralityMethods Throughout this section ldquoresponserdquo will

signify a response unit including both an individual re-sponse and a multiresponse sequence Each of the meth-ods to be described increases or maintains variability byreinforcing it

Novel response procedures reinforce responses upontheir first occurrence ldquoeverrdquo (Pryor et al 1969) or upontheir first occurrence within a session (Goetz amp Baer1973) Radial-arm maze procedures also reinforce ini-tial (within a given session) entries into arms of a mazeand result in a distribution of choices across the availableoptions the sequence or pattern of choices often beingquite variable

Lag procedures reinforce a response if it has not oc-curred across a given number of previous trials (Page ampNeuringer 1985) For example under lag 5 the currentsequence is reinforced only if it has not occurred duringany of the previous five trials A variant is the recurrence-time procedure which specifies a minimum number of in-tervening trials before a previously emitted response canagain be reinforced Machado (1989) combined recurrencetime with a percentile-reinforcement contingency to gen-erate high levels of variability (see also Machado 1992)Percentile-reinforcement contingencies base the criterionfor reinforcement on the subjectrsquos own performance over aprevious set of responses (see Galbicka 1994) For exam-ple a sequence will be reinforced only if it has not oc-curred over the previous n trials the value of n continuallychanging so that 50 of the trials are reinforced

Least-frequent response procedures (Blough 1966Schoenfeld Harris amp Farmer 1966 Shimp 1967) rein-force responses that are currently least frequent across amoving window

Threshold procedures reinforce responses whose rela-tive frequencies fall below a given value Because thistype of procedure was used in the research describedabove and in many other more recent studies details willbe provided In Denney and Neuringer (1998) which willserve as an example trials consisted of four responses byrats across L and R levers A running tally was kept of the

OPERANT VARIABILITY 683

number of occurrences of each of the 16 possible se-quences LLLL LLLR LLRL and so on If the relativefrequency of the current sequencemdashthe number of its oc-currences divided by the sum of all 16 sequencesmdashwasbelow a specified threshold value in this case 05 the ratwas rewarded Recently emitted sequences contributedmore to the maintained tally than did those in the distantpast because after each reinforcement all 16 counterswere multiplied by a weighting coefficient in this case95 Therefore the influence of a particular trial decreasedexponentially with successive reinforcements Thresholdprocedures differ from Bloughrsquos (1966) least-frequentprocedure in that more than one sequence (or response)can be reinforced at any given timemdashnamely all those se-quences that have relative frequencies below the thresholdvalue The procedure differs from lag in permitting agiven sequence to be reinforced many times in successionif the associated relative frequency is sufficiently low

The frequency dependence procedure (Machado 19921993) is a variant of threshold in which the probability ofreinforcement is a continuous function of relative re-sponse frequency The more frequent a response is thelower the probability is that it will be reinforced

Statistical feedback procedures provide subjects withcomparisons between their performances and that of amodel Neuringer (1986) showed human subjects feedbackfrom 10 statistical tests thereby enabling them to comparetheir own distributions of responses to that of a randommodel Neuringer and Voss (1993) provided feedbackbased on a chaotic model The goal in each of these caseswas for the subjects to learn to approximate the model andthe results of these experiments will be discussed below

Differential reinforcement of switches versus repetitionsbalance a subjectrsquos tendency to engage in each of these pat-terns Rather than reinforcement for infrequent responsesor random-like distributions reinforcement is providedsometimes for repeating a response and other times forswitching from one response to another In the two-operandum case a switch is L followed by R or R by La repetition is L then L or R then R Bryant and Church(1974) found that stochastic-like responding was gener-ated by rats when switches were reinforced with a 75probability and repetitions with a 25 probability that isthe rats had a natural tendency to repeat at least in theenvironment of this study Machado (1997) reinforcedpigeons for eight pecks across two keys if the sequenceof pecks contained at least one (one group of birds) ortwo (another group) switches Although the birds couldhave satisfied these contingencies by repeating a singlesequencemdashin the ldquoone switchrdquo condition for examplerepetition of LRRRRRR would produce reinforcement on100 of the trialsmdashthey tended to vary their sequencesBarba and Hunziker (2002) showed however that vari-ability was higher when it was directly reinforced thanwhen reinforcement was based on proportions of switchesa finding consistent with that of Machado (1997)

The evidence from each of these methods supports thehypothesis that variability can be reinforced

Measures Variability must be interpreted with refer-ence to a particular measure The reason is that there canbe high variability (and low predictability) at one level ofanalysis but order (and certainty of prediction) at a dif-ferent level For example the sequence 1 1 2 2 3 3 44 5 5 1 1 2 2 has the same variance (a commonlyemployed measure of variability) as 2 3 2 1 5 1 4 41 1 3 2 5 2 but an analysis of conditional proba-bilities in which the unit contains pairs of instances in-dicates that the two sequences have different levels ofvariability In general high variability observed with onemeasure does not rule out the possibility of order ob-served with another and vice versa That is why evalua-tions of random-number generators often employ manydifferent measures and indeed there are hundreds of dif-ferent measures of the randomness of a sequence (Knuth1969) All behavioral studies make simplifying assump-tions and rely on one measure or at the most a few

U-value has been the most commonly employed mea-sure of operant variability (Evans amp Graham 1980Machado 1989 Page amp Neuringer 1985 Stokes 1995)U-value measures the distribution of relative frequen-cies or probabilities of a set of responses For a set of16 possible responses i = 1ndash16 U-value is given by

where RFi = the relative frequency (or probability) ofeach of the 16 responses U-values approach 10 whenrelative frequencies approach equality as would be ex-pected over the long run from a random generator andthey approach 00 when a single instance is repeated

Other measures include the percentage of trials or fre-quency at which variability contingencies are met (Page ampNeuringer 1985) percentages of alternations versus stays(Machado 1992) conditional probabilities of responses(Machado 1992) the number or frequency of novel re-sponses (Goetz amp Baer 1973 Schwartz 1982) the num-ber or frequency of different responses (Machado 1997Schwartz 1982) Markov analyses (Machado 1992) andstatistical tests of distributions of responses (Neuringer1986) All of the measures employed to date indicate thatvariability can be reinforced

Responses and species Generality has further beendemonstrated by the use of many different responses andspecies These include Siamese fighting fish (Betta splen-dens) swimming in quadrants of a tank (Roots 2000)young chicks pecking keys (Neuringer 2002) pigeonspecking keys (Page amp Neuringer 1985) budgerigarssinging songs (Manabe Staddon amp Cleaveland 1997)rats pressing levers (Van Hest van Haaren amp van de Poll1989) porpoises swimming and jumping (Pryor et al1969) rhesus monkeys pressing levers (Sydney Reisbickpersonal communication) children drawing and build-ing with blocks (Holman Goetz amp Baer 1977) andhuman adults and children pressing buttons (Saldana ampNeuringer 1998 Stokes amp Balsam 2001) There havebeen few attempts to compare species or responses but

- ( )[ ] ( )=aring RF RFi ii

log log 2 21

1616

684 NEURINGER

in one study lowest baseline variability was observedwhen rats pulled trapezes intermediate levels were ob-served when they pushed keys and highest levels wereobserved when they pressed levers (Morgan amp Neuringer1990) with variability increasing in all cases when it wasreinforced In another study operant variability wascompared across rats and people with similar effects ob-served in the two species (Neuringer Deiss amp Imig 2000)

ParametersWe turn now to some parameters studied to date Many

other parameters as yet unstudied are no doubt also im-portantmdashfor example motivation and types of reinforcers

Age Older people (see eg Van der Linden Beertenamp Pesenti 1998) and older rats (see eg Lamberty ampGower 1990) respond less variably than their youngercounterparts when variable responding is reinforced or re-quested but in most cases it is not clear to what extent thevariability is elicited as opposed to reinforced Neuringerand Huntley (1992) compared reinforced variability in2-month-old (young) and 10-month-old (mature) Long-Evans rats Under Yoke contingencies the young rats re-sponded significantly more variably (two levers four re-sponses per trial) than the mature rats which constitutesan elicited effect When reinforcement was contingenton variability (lag 4) variability increased significantlyin both groups thereby overcoming the initial differencecorrelated with age Thus explicitly reinforcing vari-ability in older animals caused them to respond no dif-ferently than younger animals did a finding potentiallyrelevant to real-world human aging However when Wells(1999) compared 8-month-old (mature) to 24-month-old(old) rats although the mature rats responded more vari-ably than the old rats did under Yoke contingencies thedifference was not statistically significant Additionalstudies are needed especially of whether reinforcingvariable behaviors in older populations can facilitatelearning and performance

Gender Male and female rats are equally sensitive toreinforcement of variability (Neuringer amp Huntley 1992van Hest et al 1989) However there was a tendency(not statistically significant) for males to respond over-all more variably than females did and because thesestudies did not control for differences in size and weightof subjects or for other aspects of motivation additionalstudies are necessary

Stage of training Stokes and Balsam (2001) reporteddifferences in variability as a function of whether variabil-ity was initially reinforced early or later in training (seealso Gottlieb 1992) Human subjects generated sequencesof 10 L and R responses per trial on a computer keyboardunder lag 0 and lag 25 contingencies Under lag 0 everytrial ended with reinforcement under lag 25 the currentsequence had to differ from each of the previous 25 Whenlag 25 was experienced at the very beginning of traininglevels of variability at the end of training (when lag 0 wasoperative) were lower than they were when lag 25 was ex-perienced after 50 trials of initial lag 0 These results were

interpreted to suggest that there is an ldquooptimal period forsetting variability levelsrdquo (Stokes amp Balsam 2001 p 181)but that general conclusion needs further testing and anumber of questions need to be answered One has to dowith the contribution of reinforcement frequency Underlag 0 every sequence leads to reinforcement but rein-forcement is intermittent under lag 25 Another has to dowith whether learning to vary differs from learning to repeatwith respect to optimal periods (see eg Bouton 1994)

Related to stage of training are effects of prior experi-ence Hunziker Lee Ferreira da Silva and Caramori(2002) showed that for human subjects if Var contingen-cies were experienced just prior to Yoke contingencieslevels of response variation during the Yoke phase werehigher than they would have been if the Var contingencieshad not been previously experienced The time delay be-tween Var and Yoke experiences mattered with a long in-terval (3 months) producing only a small effect Saldanaand Neuringer (1998) also found effects of prior Var train-ing on performances in Yoke (increased variability inYoke) but the opposite did not hold that is training underYoke had relatively little effect on Var Hunziker Caramorida Silva and Barba (1998) reported similar effects withrats Thus there may be an asymmetry Reinforcement forvarying has relatively strong influences but reinforce-ment without regard to variations does not adversely affectlearning to vary These different effects are not surprisinggiven that Yoke contingencies are permissivemdashreinforce-ment is given whether or not responses varymdashwhereascontingencies under Var can be quite demanding How-ever the results motivate a number of additional ques-tions One is whether effects of Var training differ fromthose observed when repetitions are reinforced Another iswhether direct reinforcement of variable behaviors canhelp animals or people to overcome the detrimental effectsof prior noncontingent experiences such as those that pro-duce learned helplessness (Seligman 1975)

Environmental constraints Stokes (1995) arguedalso that variability increases when behavior is con-strained The term ldquoconstraintrdquo is ambiguous because allreinforcement contingencies exert a form of constraintIn one study Stokes (1995) compared different strate-gies of shaping leverpresses Rats in one group were re-inforced initially for pressing the lever with only theright paw (right only) and eventually for any press (withright or left paw or with any other part of the body) Theorder of training was reversed for a second group ldquoanyrdquofirst and then right only Behavior sequences were morevariable in the right-only-first group leading Stokes(1995) to conclude that early constraints lead to highvariability (see also Stokes Mechner amp Balsam 1999)That is an important hypothesis with many implicationsbut controls are needed before the results can be attrib-uted to the contingencies For example ldquoright onlyrdquo andldquoanyrdquo differed in terms of reinforcement distributions aswell as in the precision of the reinforcement operationthe ldquoanyrdquo reinforcers were provided immediately and au-tomatically by the programming equipment whereas in

OPERANT VARIABILITY 685

the right-only condition a human observer had to decidewhether a response had occurred and then present the re-inforcer necessarily leading to longer and more variedlatencies and a less consistent contingency Furthermorethere are many cases in which increasing constraints suchas those due to a decrease in the number of responses pertrial (Page amp Neuringer 1985) or the requirement of ex-actly four responses on each of two keys (Schwartz 1980)results in decreased variability Thus the contributionsof response-definition constraints and environmentallimitations must be explored further

SummaryThe generality of operant variability is shown by the

robustness of the phenomenon across procedures mea-sures responses and species As with other operant phe-nomena parameters influence the effects

FUNCTIONS

All operants including variability are functional Op-erant responses produce reinforcing consequences Be-cause other behavioral competencies depend on it oper-ant variability is functional in a broader sense as wellSome of these functions will now be described

Operant ConditioningThe potential of a reinforcer to change behavior is lim-

ited by the range of available variations (Hull Langmanamp Glenn 2001 Skinner 1981) and thus whatever pro-duces those variations contributes to the conditioning ofnew operants Much work has been done to identifysources of variation (Balsam Deich Ohyama amp Stokes1998 Segal 1972 Staddon amp Simmelhag 1971) butNeuringer (1993) was first to show that the same rein-forcers that strengthen individual responses can also pro-duce the necessary variability and that these two functionsof reinforcement may work simultaneously Implied isthat when an operant response is being conditioned re-inforcement works not only to strengthen an individualresponse but at the same time to engender the variabilityfrom which the response emerges (That is variability isnot solely a baseline phenomenon elicited or noise)Concurrent reinforcement of variations and individualtarget sequences will be the topic of this section

Pigeons in one experiment and rats in another were re-warded for varying among 16 possible response sequences(four responses per trial L and R operanda lag contin-gencies Neuringer 1993) Concurrently one of the 16designated as the target sequence was reinforced when-ever it occurred Thus reinforcements were given bothfor sequence variations and for a single selected targeta reinforced-variation-and-selection (RVS) procedureWhen the target was ldquoeasyrdquo (eg LLLL) it emergedrapidly from the baseline variations Given that the tar-get was always reinforced reinforcers were obtained morefrequently when the subjects did not vary and only re-peated the target When the target was ldquodifficultrdquo (eg

LLLR) it was not learned As in the choice experimentdescribed above (Neuringer 1992) the relative valuesor difficulties of target versus variations governed theprobability that the target would be learned

However even difficult targets could be conditionedunder the RVS procedure as was shown by another ex-periment in the same series (Neuringer 1993) Sequencevariations were again reinforced concurrently with a dif-ficult target RRLR but now the frequency of reinforce-ments for varying was systematically decreased Satis-fying the variability contingency was intermittentlyreinforced according to a VI schedule and over sessionsreinforcement for varying became less and less frequentThis technique resulted in acquisition of even the mostdifficult target sequence whereas a control group rein-forced for the target alone (without the concurrent rein-forcement for variations) never learned

Reinforcers were obtained more frequently in the RVScondition than in the control condition (since the RVSsubjects were rewarded both for varying and for emittingthe target sequence) and the additional reinforcers mighthave generated greater motivation to respond (Killeen1981) Thus motivation not the direct reinforcement ofvariability might have been responsible for the RVS ef-fect To test this possibility Neuringer Deiss and Olson(2000) reinforced three groups of rats for emitting a par-ticularly difficult target sequence LLRLR The experi-mental group was reinforced concurrently for sequencevariability on the average of once per minute (VI 1 min)The target-only control group received no reinforce-ments other than for the LLRLR target A third groupwas yoked to the experimental animalsmdashreinforced onceper minute for any sequence variability not requiredmdashand as for the other two groups whenever the target oc-curred If the additional reinforcement was responsiblefor the facilitated learning under RVS then because theexperimental and yoked groups received equal reinforce-ments they should have learned equally well but thatwas not the result Only the experimental group learnedto emit the LLRLR target The yoked controls respondedat high rates throughout the experiment but they did notvary sufficiently to make contact with the target contin-gency and never learned The target-only control groupreceived insufficient reinforcement and their responsesextinguished Therefore the added reinforcers helped tomaintain motivation as suggested above but reinforce-ment of variations was necessary for learning of a diffi-cult sequence The RVS procedure maintains both highmotivation and high variation and the two together re-sult in the acquisition of difficult-to-learn responses

Grunow and Neuringer (2002 Experiment 2) studiedone possibly important parameter of the RVS proceduremdashnamely the level of variability When a difficult target re-sponse is being trained does it matter whether reinforcedvariability is low intermediate or high As describedabove each of four groups of rats was reinforced for vari-able response sequences the first group for high vari-ability the fourth group for low variability and the other

686 NEURINGER

two for intermediate levels In other ways the procedurewas similar to that of the Neuringer Deiss and Olson(2000) RVS contingencies just described with a singledifficult target sequence always reinforced The resultswere that the higher the level of reinforced baseline vari-ability the more quickly the target was learned Thusvariability levels indeed mattered and if the just-re-ported effect is found to be general it will be most help-ful to reinforce high variability when difficult-to-learnnew responses are being shaped Other parameters re-quiring study are class definitionsmdashwhich responses areincluded in the class of acceptable variationsmdashand num-ber of instances within the class

One additional effect should be noted When varia-tions among a set of responses are reinforced if one vari-ant is selected for extinction (no longer reinforced) thefrequency of that instance decreases Neuringer (1993)showed this for pigeons responding under a lag-variabilityschedule When one of the 16 sequences was no longerreinforced (with no other change to indicate its status) thefrequency of that sequence decreased Thus reinforce-ment of variations provides the baseline to select for (iestrengthen) and select against (ie weaken) individualinstances

These findings have potentially important implicationsFirst as has been indicated the same reinforcers thatstrengthen particular instances can also reinforce varia-tions That a binary eventmdashreinforce or not reinforcemdashsimultaneously affects variations and selected instancesindicates both the power of reinforcement and a concep-tual problem How reinforcers exert such diverse effectsis a question to which we will return

Second shaping procedures may implicitly reinforcebaseline variations An approximation to a goal responseis reinforced for a period of time but eventually rein-forcement is withheld until a response closer to the goalis obtained It is commonly thought that withholding re-inforcement or mini-extinction induces the necessaryvariations for shaping to proceed as has been indicatedabove An alternative suggested by the present studiesis that organisms learn both to vary (within a set) and to re-peat (selected instances) until one instance is always rein-forced That is the process of shaping might involve re-inforcement both of variations (generally done implicitly)and of targeted instances If so a useful contributor tothe shaping process might be explicit reinforcement ofvariations

Third reinforcement of variations could facilitate ac-quisition of other competencies such as cognitive andmotor skills Siegler (1996) has shown in an important se-ries of studies that childrenrsquos acquisition of complex cog-nitive skills is correlated with high variability Anotherexample is that when motor skills are practiced in a vari-able manner high quality of performance results (Ma-noel amp Connolly 1995 1997 Mechner 1992 Schmidtamp Lee 1999) However it is rare that variability in thesedomains is directly reinforced as part of the training oreducational process

Fourth the RVS procedure might be used in therapeu-tic settings to modify maladaptive and self-injurioushabits Therapists sometimes employ noncontingent re-inforcement procedures to weaken undesirable behaviors(Vollmer Iwata Zarcone Smith amp Mazaleski 1993)The reasoning is that many maladaptive behaviors aremaintained by unintended consequences and that non-contingent rewards may compete with those conse-quences thereby lowering the strength of the unwantedhabit For example when a staff personrsquos attention toself-injurious behavior helps to maintain it providing at-tention noncontingently may decrease the strength of theunwanted target behavior However the RVS proceduremight accomplish the same goal more effectively andwithout undesirable side effects (Vollmer RingdahlRoane amp Marcus 1997) Noncontingent rewards some-times result in unwanted ldquosuperstitiousrdquo behaviors andthat would be avoided under RVS Furthermore RVSwould help to provide the set of baseline behaviors nec-essary to establish desirable new responses

Finally RVS procedures may help to gain control overand eventually to weaken other undesirable target behav-iors The target can be incorporated within a set of moredesirable responsesmdasha new operant class established withthe target as one membermdashand variable instances rein-forced Once variable responding is successfully main-tained including the undesirable target response the lattercan be weakened by selectively withholding reinforcement

Summary The conditioning of operant responses de-pends on both baseline variations and selection of suc-cessive approximations to a goal response The same re-inforcers used to select the approximations can reinforcebaseline variations thereby defining the precise set ofappropriate variants and the desired levels of variationSuch concurrent reinforcement of variations and selec-tions may facilitate the acquisition of difficult-to-learnoperants and selective withholding of reinforcementmay weaken undesirable instances

Creativity and Problem SolvingBehaving in a novel unusual or nonrepeating manner is

part of creative activities (Campbell 1960) Novelty andvariation may arise through combinations of previouslylearned behaviors (Epstein 1996) transformations of thesebehaviors (Boden 1994) direct reinforcement of variabil-ity as described above or some combination thereof Ofcourse neither novelty nor variability alone suffices forcreative outputmdashthe throw of the dice or the toss of acoin are not creative nor are haphazard strokes of a paint-brush on a canvasmdashbut without variation creativity islow Whether the variability required for creativity canbe reinforced is relevant to whether creativity itself canbe reinforced The latter question has led to contentiousdebate with some arguing that reinforcement facilitatescreativity (Cameron amp Pierce 1994 Eisenberger ampCameron 1996 Stokes 2001 Winston amp Baker 1985)and others focusing on detrimental effects of reinforce-ment (Amabile 1983 Deci Koestner amp Ryan 1999)

OPERANT VARIABILITY 687

The debate goes well beyond the present discussion butsome of the research described above is relevant Firstreinforcement contingent on variability generates higherlevels of behavioral variability than does reinforcementcontingent simply on responding as has been shown bythe many studies described above in which Var contin-gencies were compared to Yoke contingencies Similarlyprecise reinforcement contingencies matter for creativity(Cameron amp Pierce 1994 Lepper amp Henderlong 2000)If any product rather than only creative ones is rein-forced creativity tends to fall (just as variability decreasesunder yoke conditions) Second approach to or antici-pation of reinforcement interferes with operant variabil-ity (Cherot et al 1996 McElroy amp Neuringer 1990Neuringer 1991) which again might be related to effectson creativity when an individual focuses on the rein-forcer Note however that although variability decreaseswith proximity to the reinforcer overall levels of vari-ability are much higher when variability is reinforced thanwhen it is not The same may be true for creativity aswell anticipation of reward may interfere with creativeoutput to some extent but at the same time high creativ-ity may depend on its reinforcement

Operant variability may be important for problemsolving as well Here too there is disagreement as towhether training is helpful or not Early studies by Maier(1933) indicated that it is and these found support inlater work (eg Lung amp Dominowski 1985) Somestudies showed that those individuals most likely toldquobreak the setrdquo are also most likely to solve a problem(Dover amp Shore 1991) However a number of studiesfound that practice had no influence (Duncan 1961)

Arnesen (2000) explored a related phenomenon witha rat model One group Var was reinforced for variableinteractions with novel objects and another group Yokewas reinforced independently of such interactions EachVar rat received training with 12 different objects oneper session For example in the presence of a soup canthe rat was reinforced the first time it entered the canbut thereafter a different response was required such asrolling it climbing on it and so forth This procedure issimilar to that of Goetz and Baerrsquos (1973) described aboveexcept that objects changed from session to session Ratsin the Yoke control group had identical experiences in anadjoining identical chamber that contained identical ob-jects but now rewards were not contingent on object ma-nipulation After this training the rats were tested in adifferent room with 30 never-before-experienced ob-jects scattered on the floor Hidden in each was a smallpiece of food When compared to their Yoke controls theVar animals were more active during this test exploredmore manipulated objects more and most importantlydiscovered significantly more food pellets Another con-trol group received no preliminary training with objectsand their performance in the test session did not differfrom that of the yoked controls Therefore when ratswere reinforced for variable interactions with objectsthe result was an increased tendency to explore manip-ulate and discover new sources of reinforcement It

would be of considerable interest to determine whethera similar effect occurs with children

Assessment and Treatment of PsychopathologiesAbnormal levels of behavioral variability characterize

some human psychopathologies as has been shown byclinical observations as well as by research with humansubjects and animal models Few studies have attemptedto reinforce variability directly in such problem popula-tions The main question is whether reinforcement canmodify abnormal levels of variability in the direction ofnormalcy and whether such changes might have associ-ated beneficial effects

Autism Stereotyped responding characterizes indi-viduals with autism (Hertzig amp Shapiro 1990) For ex-ample Baron-Cohen (1992) asked children with autismand others in a control group to hide a penny in one handso that the experimenter could not guess which hand heldthe penny Those with autism were more likely thanthose without to generate a simple predictable patternsuch as repeatedly switching back and forth from left toright hands Such repetitive responding can be maladap-tive For example Mullins and Rincover (1985) askedchildren with and without autism to pick one of five cardsFood was occasionally found in a cup behind the cardEach of the five cards was a discriminative stimulus fora different schedule of reinforcement including contin-uous reinforcement (CRF every choice reinforced) FR 2(fixed ratio 2 in which every second choice was rein-forced) FR 4 (every fourth choice reinforced) FR 7 andFR 11 Control participants sampled all five alternativesand quickly learned to choose the most frequently rein-forced CRF card Autistic children sampled only a lim-ited number of cards and therefore often preferred anonoptimal alternative Thus children with autism tendto respond less variably than normal children and thiscan result in failure to adapt

Miller and Neuringer (2000) asked whether direct re-inforcement could influence the variability of individu-als with autism Sequences of responses on two largebuttons were first reinforced probabilistically and inde-pendently of variability (Prob phase) after which rein-forcement was contingent upon satisfying a thresholdvariability contingency (Var phase) The autistic partic-ipantsrsquo variability was significantly lower than that ofmatched controls during both Prob and Var phases butwhen variability was reinforced its levels increased sig-nificantly in both groups This is an important findingbecause it suggests that direct reinforcement of variabil-ity may help to modify some of the behavioral stereo-types that are characteristic of autism Lee McComasand Jawor (in press) offer support for this hypothesisThree individuals with autism were reinforced for variedverbal responses to social questions the responses had tobe appropriate given the context Variability increased intwo of the three individuals In a related procedureBaker and Koegel (1999) reinforced children with autismfor varied social interaction in a game-playing contextAlthough the reinforcement was permission to engage in

688 NEURINGER

the stereotyped behaviors preferred by the child the re-sult was increased social interactions both during theperiod of the experimental manipulation and in 1- and2-month follow-ups These three studies taken togethersuggest that reinforcement of varied behaviors may facil-itate modification of nonfunctional ritualistic and stereo-typed behaviors

Depression Depression provides another example ofmaladaptive consequences of low variability Lapp Mar-inier and Pihl (1982) observed that depressed womenproduced fewer alternative solutions to hypotheticalproblems than did nondepressed women Channon andBaker (1996) found that attempts by moderately de-pressed college students to identify faults in a series ofinterconnected circuits were less varied and less success-ful that were those of nondepressed controls HorneEvans and Orne (1982) found that when depressed pa-tients were asked to generate a random sequence of num-bers their responses were less variable than were thoseof controls These experimental results are consistentwith clinical observations

Can depressed individuals be rewarded for increasingvariation Hopkinson and Neuringer (in press) dividedcollege students into mildly depressed and nondepressedgroups on the basis of the CES-D scale a paper-and-pencil self-evaluative index of depression In the first oftwo phases sequences of responses on a computer key-board were reinforced independently of variability (Prob)following which only highly variable sequences were rein-forced (Var) Under Prob depressed students respondedsignificantly less variably than did the controls in agree-ment with previous findings Under Var variability in-creased significantly in both groups Indeed by the endof the experiment depressed and nondepressed subjectswere responding equally variably (Instructions were alsoeffective in increasing variability in the depressed par-ticipants) Thus even in a simple computer-game envi-ronment mildly depressed subjects manifested lowervariability than controls did but direct reinforcementmodified their behavior to control levels Can reinforce-ment of variability be used in real-world conditions tohelp depressed individuals An affirmative answer comesfrom cognitive behavior modification procedures thatwork to increase alternative solutions to problems (Beck1976) A more general conclusion from these studies ofautistic and mildly depressed individuals is that direct-reinforcement-of-variability procedures may benefitthose who manifest abnormally low levels of variability

Attention Deficit Hyperactivity Disorder (ADHD)ADHD may provide an example of the opposite stateabnormally high levels of variability (Barkley 1990)Wultz Sagvolden Moser and Moser (1990) proposedthat the spontaneously hypertensive rat (SHR) may serveas a good animal model of ADHD the SHR shows manyof the behavioral characteristics of that disorder includ-ing hyperactivity poor learning ability poor ability totolerate delayed reinforcement and poor self-control

The next study to be described asked about operant vari-ability in SHRs

Mook Jeffrey and Neuringer (1993) compared vari-ability of SHRs and of a control strain Wystar Kyoto(WKY) under two conditions Prob and Var The mainresult was that SHRs tended to respond variably whetheror not the contingencies required it consistently with theresults from the human ADHD literature When vari-ability was reinforced the two strains responded at ap-proximately equally high levels of variation These re-sults were confirmed in two other studies (HunzikerSaldana amp Neuringer 1996 Mook amp Neuringer 1994)Thus control rats varied only when the contingencies re-quired variations whereas SHRs varied whether or notthe contingencies required it As might be expected fromthese results when reinforcement depended on repeti-tions the SHRs performed poorly (Mook et al 1993)

Another similarity between SHRs and ADHD humanswas seen when amphetamine a drug that has effects simi-lar to those of Ritalin often prescribed for ADHD was ad-ministered (Mook amp Neuringer 1994) When repetitionswere required for reinforcement amphetamine improvedperformances by the SHRs so that they were as accurate assaline-injected WKYs a finding again consistent withthose in the ADHD literature These studies support SHRsas a model of ADHD and suggest that additional attemptsshould be made to assess whether or not individuals withADHD are less able to change levels of variability thanare controls Although as indicated above there is someevidence in support of this hypothesis (Barkley 1990)when Saldana and Neuringer (1998) directly comparedhuman ADHD and control participantsrsquo operant variabil-ity within the context of a computer game no differenceswere observed Most of those with ADHD were main-tained on Ritalin and other drugs however a fact unfor-tunately not documented in the original article

Drug EffectsAlthough many drugs are known to influence behavioral

variability (Brugger 1997) rarely have attempts beenmade to analyze explain or identify the sources of theeffects As will be discussed below operant variabilitycan result from the memory-based process of remember-ing previous responses and then behaving differently orcan result from responding stochastically a process thatappears to be quite different McElroy and Neuringer(1990) attempted to distinguish the effects of alcohol onmemory from its effects on stochastic processes in astudy in which one group of rats was reinforced for vary-ing four-response sequences under a lag 5 contingency(Var) and another was reinforced for repeating a singlesequence LLRR (Rep) Ethanol injections of 075 and20 gkg affected Var and Rep performances differentlyWhereas the Rep animals were much less able to satisfythe LLRR contingency at both doses the Var rats re-sponded variably whether or not alcohol had been ad-ministered Similar results were obtained by Cohen et al

OPERANT VARIABILITY 689

(1990) with a multiple schedule in which each rat servedas its own control In the same animals and within thesame sessions alcohol degraded Rep performances butleft Var intact Thus alcohol appears to interfere with atask that requires working memory as presumably is thecase for the Rep task but not with one that reinforcesvariability The memory-interfering effect of alcoholmight account for why alcohol also degrades radial-armmaze performance (Devenport 1983)

Alcohol also affects choices to vary or to repeat For ex-ample Gorka (1993) reinforced rats concurrently for re-peating and varying and monitored percentage of choicesas a function of time following injection of ethanol Inthe control sessions when the rats were injected withsaline the probabilities of varying and repeating remainedapproximately constant across the 45-min session witha slight bias in favor of varying After injection with 15gkg of ethanol however there was a significant increasein choices to vary at the beginning of the session andthen a later decrease to below 50 That is alcohol ledinitially to a markedly increased preference for varyingbut later to a preference for repeating (Figure 5) This re-search should be replicated and extended especially be-cause the contingencies ldquopressuredrdquo the animals to re-main at approximately 50 preference but thesuggestion is that alcoholrsquos effects on response variabil-ity may depend in part on the duration of its influence Amore general implication is that drugs may affect re-peated and variable operants differently and that effectsdepend on time following administration

SummaryReinforced variability may contribute to many types

of learning including operant shaping and acquisition ofcognitive and complex motor skills Similarly creativityand problem solving depend on variable responding andin these areas as well explicit reinforcement of variationsmay be functional Abnormal levels of variability aresometimes caused by psychopathologies and drugs andoperant variability may be useful in these areas in twoways first as a way of assessing the variability that is pos-sible and second to modify variability toward more nor-mal levels Studies show that individuals with autism andmild depression vary less than do controls but that ex-plicit reinforcement of variability moves the levels towardsnormalcy Other studies show that drugs such as alcoholmay affect repeated and variable operants differently

EXPLANATIONS

Two questions must be answered in order to explainoperant variability First what are the sources of vari-ability that permit its reinforcement Second how is itmaintained Related questions including how instancesbecome members of a set from which variations emergeand whether operant variability is a generalizable com-petency will also be considered

Origination From Nonoperant SourcesSkinner (1966) reasoned that operant responses orig-

inate from nonoperant sources with reinforcement con-tacting a behavior shaping and strengthening it onlyafter the behavior had occurred for reasons other than re-inforcement To put it simply a behavior must occur be-fore it can be reinforced (Balsam amp Silver 1994) I willconsider three initial sources of variable behavior en-dogenous variability noncontingent environmental eventsand extinction These appear to parallel evolutionaryprocesses and they too will be identified

Endogenous variability All physical phenomenavary and behavior is no exception This fact can providethe opportunity for reinforcement to select particularlevels of variation Examples of spontaneous variabilityinclude spontaneous alternations in mazes (Dember ampRichman 1989) nondifferentially reinforced variationsin operant responses (Guthrie amp Horton 1946) and ex-ploration (Archer amp Birke 1983) Endogenous muta-tions provide a parallel in genetics Base-pair changes inDNA molecules occur spontaneously intron-bearinggenes mix and match segments to encode new proteinsjumping genes introduce new DNA sequences and atthe level of the chromosome random crossings and as-sortment occur as a natural part of reproductive meiosisThese sources of random genetic change occur indepen-dently of external influences

Figure 5 The relative frequency of variations by rats as a func-tion of 5-min blocks of time since the beginning of the sessionThe open circles represent performance after the rats were in-jected with saline indicating a slight tendency to vary more thanrepeat but one that is approximately constant throughout thesession The filled circles represent performance after injectionwith 15 gkg ethanol indicating a much higher likelihood ofvarying soon after the alcohol is injected followed by a decreasein variations to below 50 and then a return to baseline at theend of the session Group averages are shown Data are fromGorka (1993)

690 NEURINGER

Noncontingent events Noncontingent events some-times referred to as chance accidents or luck also affectbehavior One happens upon a particular passage in a bookthat leads to a new thought one happens to sit next to aninteresting woman or man on an airplane which leads tomarriage (Bandura 1982) In science as well as in every-day life accidents or serendipity are an important part ofthe process of discovery (Beveridge 1957) In each of thecases presented above an accidental event leads to new be-haviors that are then strengthened by consequences Thesame is potentially true for variability itself Particular lev-els of variability might occur for adventitious reasons butthen become functionally related to consequences

That organisms are particularly sensitive to such unan-ticipated noncontingent events is shown by two basicprinciples of learning habituation and Pavlovian condi-tioning Organisms attend preferentially to unexpectedevents and stop attending or habituate to repeated onesThe conditioning of new Pavlovian responses dependsaccording to influential theories upon unexpected rein-forcing events (eg Rescorla amp Wagner 1972)

Noncontingent events such as X-rays and other pro-cesses that can act as mutagens play an important role ingenetic variation as well The particular genes affectedand the type of changes are random there is no neces-sary relationship or contingency between the externalevent and the affected gene However sensitivity to mu-tations is not random but has been selected presumablybecause it is functional Thus in both behavior and ge-netics sensitivity to noncontingent events is conserved

Extinction Extinction increases behavioral variabilityat least to some extent as was described above Distance inspace or time from the reinforcer does the same (Cherotet al 1996) A possible parallel at the genetic level in-volves high levels of deprivation at least in simple lifeforms For example when E coli bacteria are deprivedof nutrients unusually high mutation rates are observedThe exact mechanism by which such hypermutations areinduced is not known but they increase the likelihoodthat a population can survive such environmental insults(Moxon 1997) That is deprivation-induced variability isfunctional at the genetic level (The behavioral evidenceregarding long-term deprivation is less clear some stud-ies showing increased variability and others increasedstereotypy eg Carlton 1962 Senkowski Vogel amp Po-zulp 1978)

Endogenous variations external noncontingent eventsand withholding of reinforcement are three sources forthe variability that precedes control by reinforcementBut once control by reinforcement is established a dif-ferent question emerges What behavioral processes areinvolved when variability is maintained as an operantThe sources just discussed generate uncontrolled non-specific and often short-lived variability and cannot ex-plain the precisely controlled operant type

Operant Variability ProcessesThree processes are hypothesized to underlie operant

variability The first involves the use of random environ-

mental events such as the toss of a coin The second in-volves memory for responses such as when an animallearns not to return to previously visited locations Thethird will be referred to as an endogenous stochasticgenerator

Random environmental events Throughout historymany cultures have used random events to influencechoices and decisions such as throws of dice and randomselections of sticks cards bones or organs Today coinflips are used by a referee at the beginning of a game to de-cide which team kicks the ball a computerrsquos random-number generator is employed by scientists to avoid bi-ases in assigning subjects to experimental groups andaleatoric events are sometimes employed in modern artmusic and literature The Dice Man by Rhinehart (1998)provides a fictional example in which the protagonistbored with life writes a number of possible actions onslips of paper and then periodically selects one blindlyand behaves accordingly These examples show that randomexternal events may be used to avoid biases engender un-likely responses and break out of behavioral niches

Memory-based variability Many methods used tostudy operant variability differentially reinforce nonrepe-titions at least over the recent past Therefore one strat-egy that subjects can employ is to cycle through few re-sponses or sequences especially when the variabilitycontingencies are relatively permissive For exampleunder a lag 2 contingency cycling repeatedly throughthree sequences will result in 100 reinforcement Ani-mals and people sometimes do this apparently basingtheir current response on memory for or discriminativecontrol by the just emitted responses One such instancewas shown by Schoenfeld et al (1966) who rewardedrats for IRTs that fell into one of 10 interval classes onlyif the current IRT differed from the interval class of theprevious response The rats came to alternate betweenlong and short IRTs thus behaving systematically to meetthis lenient variability contingency Machado (1993)studied an analogous effect having pigeons peck L and Rkeys under a frequency-dependent variability contin-gency related to Shimprsquos (1967) procedure describedabove When the trial unit consisted of two responses thebirds developed memory-based repetitive sequencessuch as RRLLRRLL and thereby satisfied the variabilitycontingency When the trial unit was three responses inlength the birds did not develop the optimal fixed pat-tern of RRRLRLLLmdashwhich was too complexmdashbut in-stead developed ldquorandom-like behaviorrdquo (Machado1993 p 103) Thus a memory-based strategy was em-ployed when it was possible but when the memory loadbecame too high stochastic responding emerged A sim-ilar pattern was seen when variability of budgerigarsongs was reinforced (Manabe et al 1997) Under lag 1contingencies the birds sang mainly two songs underlag 2 they increased to three songs Only when the lagwas increased to 3 did song diversity increase apprecia-bly in one of the birds In radial-arm mazes as well whencontextual cues are available memory again appears toplay an important role leading to much lower probabil-

OPERANT VARIABILITY 691

ities of repetition than would be expected from a sto-chastic generator (McElroy amp Neuringer 1990)

Note that under lag schedules memory-based strate-gies result in higher reinforcement rates than respondingstochastically does The reason is that a stochastic gen-erator repeats by chance That animals and people oftendo not employ memory-based strategies probably indi-cates the difficulty of remembering long sequences andpossibly other advantages of unpredictable responding

If a memory strategy is the only way to satisfy vari-ability contingencies then it can be used even when thememory requirements are demanding as was shown byNeuringer and Voss (1993) Human subjects were trainedto generate on a computer keyboard a chaotic-like se-quence of responses that matched iterations of the logistic-difference equation

where Rn refers to the nth iteration in a series with R be-tween 00 and 10 and t is a constant between 10 and40 When the t parameter approaches 40 iterations ofthis equation result in very ldquonoisyrdquo sequences As withother chaotic processes however there is an orderlinessunderlying the surface noise In this case if responsesare autocorrelated with Rn drawn as a function of Rn21the resulting function is a parabola In the Neuringer andVoss (1993) study following each response college stu-dents were shown how close their predicted values wereto that from iterations of the logistic-difference model Inone experiment the students made digital predictions inanother they placed a pointer along a horizontal lineand this analogue response was then translated into dig-ital values for computation With training the studentsbecame increasingly adept at responding in chaotic-likefashionmdashthe studentsrsquo predictions matched closely theiterations of the logistic functionmdashand their autocorrela-tions became parabolic The left column in Figure 6 showsperformances by 4 subjects (rows) early in training thatis not well approximated by the logistic-differenceparabola and the right column shows performances at theend of training in which r 2 values were 72 99 98 andapproaching 10 How was this accomplished Sinceeach iteration in the logistic-difference sequence is basedon the prior outputmdashthat is the sequence is generated bya deterministic although nonlinear processmdashit was hy-pothesized that the human subjectsrsquo chaotic-like responseswere also based on memory for prior responses To put itsimply the subjects may have memorized a long series ofpairs in the form ldquoIf the previous response was value Athen the current response must be value Brdquo as Metzger(1994) Ward and West (1994) and Ward (2002) have sug-gested

To test whether subjects in fact relied on memory forprior responses Neuringer and Voss (2002) systemati-cally slowed responding by interposing pausesmdashthat isby requiring an IRT As IRT increased the difference be-tween the subjectsrsquo sequences and the modelrsquos chaotic

Rn n nt R R= -( )- -1 11

Figure 6 Autocorrelations (responses on trial n on the y-axis asa function of n21 on the x-axis) for 4 human subjects at the be-ginning (left column) and end (right column) of training inNeuringer and Voss (1993) Equations for the best-fitting curveand amount of variance accounted for (r2) are shown at the topof each panel To the extent that the circles (responses) approxi-mate the equation y = 40x 2 40x 2 the subjects were respondingin a way that matched the logistic-difference chaotic functionFrom ldquoApproximating Chaotic Behaviorrdquo by A Neuringer andC Voss 1993 Psychological Science 4 p 115 Copyright 1993 bythe American Psychological Society Reprinted with permission

692 NEURINGER

output increased and the underlying parabolic structurewas broken That is when pauses increased subjects wereless and less able to approximate the modelrsquos chaotic dis-tribution (see Figure 7) The important point here is thathighly variable responses which in this case are chaoticin nature can be based on memorial processes

Endogenous stochastic generator The evidence tobe reviewed next indicates that animals and people canrespond stochastically and that this competency is basedneither on chance environmental events nor on memoryfor prior events or responses It is impossible to provethat an endogenous stochastic process underlies operantvariability that would be analogous to proving the nullhypothesis However much evidence is consistent withthe stochastic hypothesis Included are the facts thatmemory limitations do not permit adequate respondingunder many variability contingencies that response dis-tributions match those expected from a stochastic gener-ator that changes in parameter values affect operantvariability as they would a stochastic generator and thatinterference with memory processes leaves stochasti-cally based operant variability unaffected As a wholethe evidence is consistent with an endogenous stochas-tic source of operant variability

Memory limitations One type of evidence is nega-tive in character and is based on the presumed inability ofmemory for prior responses or sequences to satisfy sometypes of variability contingencies Two examples weregiven above (Machado 1993 Manabe et al 1997) andanother was provided by a lag 50 contingency in Pageand Neuringer (1985) In the latter case the sequence in

the current trial (eight responses across L and R keys) wasrequired to differ from those in each of the previous 50 tri-als including at the beginning of a session the last 50trials from the previous session Pigeons responded wellunder these contingencies and in a way similar to that ofa stochastic model It is unlikely that pigeons can remem-ber each of their previous 50 eight-response sequences

Response slowing As was shown above for chaotic-like responding performances are often degraded by de-lays imposed between a stimulus and a response pre-sumably because of limitations in working or short-termmemory (Berryman Cumming amp Nevin 1963 Williams1971) Neuringer (1991) relied on this effect to test whethermemorial processes could explain the generation of vari-able sequences under lag schedules Two groups of ratswere reinforced for L and R leverpresses producing fourresponses per trial One group (Var) was trained under alag 5 contingency and the other (Rep) was trained to asingle sequence of responses LLRR It was assumedthat accurate Rep performance depended on workingmemory After performances had stabilized IRTs weresystematically varied by the imposition of timeouts of05 to 20 sec between consecutive responses in the se-quence During these forced pauses the chamber wasdark and responses were not counted toward completionof the sequence If the animals had used the previous re-sponse(s) as a discriminative cue for the current responsethen performances should have been degraded by theseinterpolated pauses

The results were that for the Rep group as the re-quired pauses increased from 0 to 6 sec percentages ofcorrect sequences were relatively unaffected but at pausesgreater than 6 sec probability of a correct LLRR se-quence fell sharply This pattern of results suggests in-volvement of working memory of prior responses withthe rats able to remember responses within a 6-sec win-dow The Var grouprsquos results were quite different Aspauses increased between 0 and 6 sec the rats were moreand more likely to satisfy the variability contingenciesand with pauses greater than 6 sec percent correct re-mained at a high and constant level Thus the Var grouprsquosresults were opposite to those of the Rep group and op-posite to those predicted from a memory-based strategySimilar results showing that operant variability increasesor is maintained as responding is slowed have been re-ported by Morris (1987) with pigeons and by Baddeley(1966) with random response generation by human sub-jects The results reported above concerning alcoholrsquos ef-fects on variability and repetition were also consistentwith those of Neuringer (1991)

Memory for prior responses may not have controlled theVar grouprsquos performance but why did variability increaseEach of three answers is consistent with a stochastic-generator interpretation Weiss (1964 1965) hypothe-sized that random responding requires that current re-sponses be independent of previous ones memory for orcontrol by prior responses interferes with such indepen-

Figure 7 Approximation to a chaotic sequence (given by delta)as a function of the imposed interresponse times Shown are theaverages of 5 subjects

OPERANT VARIABILITY 693

dence and therefore pauses would be expected to facil-itate variability A second possibility is that at shortIRTs animals tend to repeat responses on the sameoperandum Blough (1966) found this in pigeons and ex-cluded such double pecks from his analyses because theyappeared not to be under the control of reinforcement con-tingencies Morris (1987) also found a tendency for birdsto repeat when no interresponse timeouts were imposedA third hypothesis is that there were two contributors tothe observed variability One was a stochastic processcontrolled by reinforcement and the other elicited vari-ability generated by interposition of pauses The high vari-ability elicited as an effect of slowed responding is ageneral phenomenon supported in many other cases Ac-cording to this interpretation operant variability in the Vargroup was governed by a stochastic-based process operantrepetition in the Rep group was governed by a memory-based process and pauses elicited variability under bothcontingencies Thus Rep performance was interferedwith whereas Var was facilitated Each of these hypothe-ses is consistent with the conclusion that memory for (ordiscriminative control by) prior responses does not con-tribute to and possibly interferes with variable respond-ing when an organism is reinforced for variability andthis in turn is consistent with the stochastic-generatorhypothesis

Stochastic response distributions Other evidence iscorrelational Response distributions are similar to thosefrom a stochastic generator The data from two experi-ments already discussed are shown in Figure 1 In thesecases reinforcement of least frequent IRTs (Blough 1966)and of nonrepetitive choice sequences (Machado 1989)resulted in stochastic-like responding

Neuringer (1986) reported another striking case Highschool and college students generated sequences of 1sand 2s on a computer keyboard with each set of 100 re-sponses constituting one trial A baseline phase lastedfor 60 such trials for a total of 6000 responses duringwhich the students were asked to behave as randomly aspossible as if they were reporting the outcome of a se-ries of coin tosses As in the many previous studies ofthis sort the subjectsrsquo responses differed significantlyfrom those of a stochastic model (Brugger 1997 Wage-naar 1972) During the training phase the subjects re-ceived feedback which enabled them to compare their per-formances with that of a stochastic model first accordingto one statistical test and then according to another untilfollowing each trial feedback was provided from 10 dif-ferent statistics The results were that all of the subjectslearned to approximate the stochastic model accordingto the 10 statistics as well as some (but not all) other sta-tistical tests That is the distributions of the statisticswhich clearly did not match a stochastic model at the be-ginning of training approximated the stochastic modelat the end of it (see Figure 8)

The previous failures of other researchers to obtain ev-idence for random responding had led some to concludethat humans did not have the capacity to behave ran-

domly What would account for the difference betweenthe present positive finding and the many other negativeones Neuringerrsquos (1986) study was the first in which sto-chastic responding was explicitly reinforced in this casewith feedback from 10 statistical tests An analogy mighthelp to explain the difference in procedures and resultsImagine an experimenter walking around a city and ask-ing people if they know what a violin sounds like (analo-gous to asking people whether they know what a series ofcoin tosses or dice throws is like) Most would answeryes If the same individuals were then asked to play theviolin very few if any would succeed The experimentermight conclude after many replications with differentsubjects in different cities that people did not have thecapacity to play the violin The point of course is thatplaying the violin is a skill requiring training and prac-tice and the same may be true for responding randomlyIt is easy to vary but practice and reinforcement may benecessary to approximate a random distribution

Response number Page and Neuringer (1985) testedthe stochastic-generator hypothesis in another way byvarying number of responses per trial with pigeons work-ing under a lag 3 contingency The reasoning was that ifprevious responses served as a discriminative cue forwhat not to domdasha memory-based strategymdashthen perfor-mance should be degraded when the number of re-sponses was increased Eight responses per trial requiresubjects to remember more than four responses The sto-chastic hypothesis predicts the opposite outcome as isdemonstrated by the following example If each trialwere two responses in length with responses determinedby the toss of a coin then the probability that a given trialwould repeat the previous one would be 25 (There arefour possible sequences in the first trial RR RL LRand LL Thus the second trial has a 1-in-4 chance ofmatching the first) If a trial consists of four coin tossesthe probability of a repetition by chance is 0625 or 1 in16 Thus if subjects used a stochastic process to gener-ate Ls and Rs performances should improve with in-creasing responses per trial Results were precisely thosepredicted by the stochastic hypothesis and inconsistentwith a memory strategy Probability of meeting the con-tingencies and therefore probability of reinforcementincreased as responses per trial increased In otherwords eight-response trials were easier for pigeons thanfour-response trials

Response interference Neuringer and Voss (2002)attempted to distinguish between memorial and stochas-tic processes in yet another way They hypothesized thatwhereas chaotic responding was based on memory forprior responses (as was discussed above) responding inNeuringerrsquos (1986) study with feedback from 10 statisti-cal tests indicated involvement of an endogenous sto-chastic generator To show that human subjects could en-gage in both processes at will three college studentswere trained to generate chaotic sequences in one compo-nent of a multiple schedule as in Neuringer and Voss(1993) and to generate stochastic sequences in a sepa-

694 NEURINGER

rate component as in Neuringer (1986) The wordCHAOTICS was on the screen in the first component theword STOCHASTICS was on during the other and thesewords alternated throughout the session After the sub-jects succeeded at each of the component tasks test ses-sions were introduced in which memory for prior re-sponses was subject to interference in both the chaoticsand stochastics components as follows During the

chaotics component four separate chaotic sequenceswere cycled one after the other each cued by differentscreen colors The subjects attempted to approximate achaotic sequence in one color (eg light green) follow-ing which they would have to match a second chaotic se-quence (lavender screen color) and so on the fourchaotic contingencies following one another across thesession Although all four chaotic sequences used the

Figure 8 Performance by one subject at the end of training to respond stochastically in Neuringer(1986) Shown are comparisons on 6 different statistics between the subject (solid lines and Xs) andthe random model (dashed lines and closed circles) At the end of training the subjectrsquos perfor-mance did not differ statistically from that of the random model on these and four additional sta-tistics From ldquoCan people behave lsquorandomlyrsquo The role of feedbackrdquo by A Neuringer 1986 Jour-nal of Experimental Psychology General 115 p 71 Copyright 1986 by the American PsychologicalAssociation Reprinted with permission

OPERANT VARIABILITY 695

same logistic-difference equation that a model did theywere out of phase and independent If chaotic respond-ing is memory based then the four independent se-quences would be expected to interfere with one anotherthat is approximations to the four chaotic models wouldbe lower than when only a single chaotic output was re-quired as in Neuringer and Voss (1993) That was ex-actly the finding Performances by the three subjectswere signif icantly degraded during the four-segmentphase To put it simply chaotic responses interfered withone another

A different pattern of results was seen in the stochasticsportion of the experiment The same interference proce-dure was used with four independent stochastic sequencesrequired in the four different color conditions As inNeuringer (1986) subjects were required to satisfy eightdifferent statistical tests of randomness and to do this in-dependently in each of the four phases (If responses in anytwo or more of the four stochastic phases became corre-lated above a minimal level responses were considered tobe incorrect thereby prohibiting repetition of a single se-quence in two or more of the phases) For one subjectseven of eight statistics were closer to those of a randommodel during this interference phase than they were at theend of the original training with only a single sequenceand for a second subject all eight statistics were closerThe third subject showed closer-to-random approximationunder the four-segment condition on only two of the eightstatistics but most of her changes were also in the directionof a random generator For example if a random generatoremits equal numbers of ldquouprdquo and ldquodownrdquo sequences andthis subject in the baseline phase had emitted only 47ldquouprdquo sequences then during the four-segment tests shecame to emit ldquouprdquo 55 of the time She changed in the di-rection of the random model but overshot the mark Thuswhereas interference degraded chaotic performance sig-nificantly in all three subjects it improved stochastic per-formances in at least two of them These results are con-sistent with the hypothesis that a memory-based processcontrols chaotic variability and that a stochastic processnot dependent on memory controls stochastic variability

Summary of Tripartite TheoryThe following three processes constitute a tripartite

theory of operant variability attention to random events inthe environment (coin toss) memorial processes (chaosradial maze) and stochastic processes (response se-quences and random-number generation with feedback)Random environmental events assure unpredictabilitymdashfor example in making unbiased selections Remember-ing what was done previously will enable one to avoidrepetitions to vary the students called upon in class tovary the locations when looking for a hidden object orto try different approaches when attempting to solve aproblem External aids such as notes to oneself or sci-entific records help to enhance memory capacities

An endogenous stochastic generator provides the thirdand most controversial source of behavioral variability

When an opponent is attempting to predict your behav-ior in a game or at war it may be functional to behavestochastically Similarly operant conditioning of new re-sponses may depend on stochastic variations It is im-possible to prove stochasticity some presently unknowndeterministic system may ultimately explain the resultsHowever the evidence is consistent with the stochastichypothesis and inconsistent with the alternatives For ex-ample memory is not sufficiently powerful to explain alloperant variability memory-interference proceduresleave stochastic responding intact and behaviors corre-late well with those of stochastic models

Both genetics and reinforcement play important rolesin all three processes Reinforcement is always involvedin that learning provides information about when whatwhere and how much to vary Genetic influences areshown in that not all responses are possible and if theyare possible they are not all equiprobable (eg as inNeuringer et al 2001) Stochastic responding may be bi-ased due to experiences and knowledge that is biasesmay be established by reinforcement contingencies or bythe interface between the organismrsquos propensities (or be-havior systems see Timberlake amp Lucas 1989) and theenvironment In the case of random-number generationby human subjects biases may be due to misconceptionsconcerning what ldquorandomrdquo means However some bi-ases can be overcome through feedback and instructions(Lopes amp Oden 1987 Neuringer 1986)

The importance of variations for evolutionary changeis supported by the many sources of genetic variabilityand by the fact that genetic variability is conserved Asimilar argument holds for operant variability That thereare at least three independent sources of operant varia-tions as well as other elicitors such as environmentalnoise and response extinction supports its importancefor normal adaptive functioning

Structure of VariabilityEvery operant ldquoresponserdquo is actually an instance of a

class with each member of the class exerting the sameeffect on the environment (Skinner 1959) When a rat isreinforced for pressing a lever for example the class isdefined by closure of a microswitch and any way of ac-complishing the closure whether with the left paw theright paw or the mouth is as effective in producing re-inforcement as any other If the contingencies of rein-forcement so specify however the response class can beredefined and circumscribed which is another charac-teristic of the operant For example if only right-pawpresses are acceptable members of the reinforced classresponding will tend to occur only with the right paw(Stokes 1995) Of course the capabilities of the organismplay a role for example including ldquotail pressesrdquo in the op-erant class does not mean that the lever would be or couldbe pressed by the ratrsquos tail The structure of the operantclass therefore emerges from the current reinforcementcontingencies and the organismrsquos capability derivedfrom genetics development and prior experiences

696 NEURINGER

Operant variability has structure as well and as for alloperants that structure is due partly to the organism it-self and partly to the reinforcement contingencies Theresearch described throughout this article has providedexamples of the structure of variability generally in theform of sets of four or eight responses across L and Roperanda But the structure of variations can be speci-f ied more precisely For example Mook and her col-leagues (Mook et al 1993 Mook amp Neuringer 1994)reinforced rats for varying four response sequences acrossL and R levers under a lag 1 contingency Unlike in pre-vious studies however the acceptable class was furtherconstrained so that only sequences beginning with two Rresponses (RRLL RRLR RRRL and RRRR) were rein-forced The rats learned to vary their sequences withinthe imposed limitations that is they generally begantheir sequences with two R responses and varied the se-quences within those limitations Similar results wereobtained in the context of a radial-arm maze in whichonly a subset of arms was baited

The precision with which reinforcement contingen-cies can specify class was highlighted in research per-formed by Burns (1997) and Ross and Neuringer (2002)In the latter case college students drew rectangles on acomputer screen with the computerrsquos mouse Each rec-tangle constituted one trial (the screen cleared at the endof each trial) with reinforcement (points) depending onsimultaneous variations along three dimensions of therectangle location on the screen area of the rectangleand shape with the latter defined as the ratio of the rec-tanglersquos height to its width Although the subjects weregiven no instruction other than to ldquogain pointsrdquo theylearned to vary along all three dimensions with levels ofvariability significantly higher than those of yoked controlsubjects who were reinforced independently of variations

A second part of the Ross and Neuringer (2002) studydemonstrated even greater precision of control To gainpoints three new groups of subjects were required to re-peat along one of the dimensions while simultaneouslyvarying along the other two One group was required torepeatedly draw rectangles of approximately the samesize while simultaneously varying location and shapeanother group had to repeat the shape while varying areaand location and the third to repeat location while vary-ing size and shape Verbal instructions were minimal asthey were in the first part of the study Subjects learnedto satisfy these contingencies as well varying two at-tributes while simultaneously repeating one and signif-icant differences were obtained rapidly within 300 trials

These demonstrations raise questions as to how binaryfeedbackmdashreinforcement presented or notmdashcan simul-taneously control independent behavioral dimensionsThe issue is similar to that discussed above concerningsimultaneous reinforcement of variations and selectionsIt is similar also to those issues faced by researchers at-tempting to understand how the nervous system strength-ens particular synapses when many different neurons firesimultaneously and to ldquoallocation of creditrdquo in artificial

learning models For present purposes suffice it to notethe exquisite precision with which reinforcement controlsvariability and repetition

GeneralizationDo animals and people learn to ldquovaryrdquo implying a

generalizable skill or do they learn a specific distributionof specific responses across particular operanda Thequestion of generalization has been asked of many otherbehavioral competencies such as imitation When chil-dren learn to imitate do they learn stimulusndashresponseassociations (if mother moves her right hand the childlearns to move his right hand) or do they learn moregenerally to ldquoimitaterdquo Evidence indicates that reinforce-ment of imitative instances leads to generalized imitation(see eg Baer amp Sherman 1964 Bandura 1986) A sim-ilar question has been raised when pigeons rats or chil-dren learn to choose a stimulus that matches a previouslyshown sample For example if a pigeon is shown a reddisk as a sample it must then choose red when given achoice between that and blue Does the bird learn to matchIn this case the answer appears to depend on the train-ing procedures If trained with few stimuli such as redgreen and blue then the pigeon learns simple associa-tions (if red sample choose red if blue sample chooseblue) but if hundreds of different stimuli (pictures) areemployed during training then generalized ldquomatchingrdquois learned (Wright 1997) In each of these examples thelearning of a general competency or strategy was shownby transfer-of-training tests What was learned with oneset of stimuli or in one situation transferred to new stim-uli and situations We turn now to the question of whetherlearning to vary transfers

After the subjects in Maltzmanrsquos (1960) experimentdescribed above learned to generate remote verbal as-sociates they were asked to devise unusual uses of com-mon objects In comparison with those in the controlgroup these subjects generated significantly more un-usual uses indicating transfer of training to a differenttask However transfer was not observed in the oppositedirection from unusual uses to verbal associates

Holman et al (1977) using a procedure similar to thatof the Goetz and Baer (1973) research described aboverewarded preschool children verbally for novel drawingsand found increased novelty Unrewarded block construc-tions were also monitored and weak evidence was ob-tained for transfer of novel responding from rewardeddrawing to the unrewarded constructions In a second ex-periment preschoolers engaged in four independent taskspainting pen drawing wood block construction and Legoconstruction When the children were reinforced for vary-ing their pen drawings the novelty of paintings increasedalong with the pen drawings indicating transfer from onetask to the other However transfer to the topographicallydifferent block- or Lego-building tasks was not observed

Eisenberger and Armeli (1997) gave monetary rewardsto one group of fifth- and sixth-grade students the vari-ability group for describing unusual uses of common ob-

OPERANT VARIABILITY 697

jects A second group the repeat group received thesame rewards for describing common uses of the sameobjects In a later test all of the children were asked todraw pictures incorporating a circle and the group that hadpreviously been reinforced for varied responses drewmore unusual (or lower frequency) pictures than did therepeat group Thus there was transfer of novelty trainingfrom one domain to another with additional studies in theseries showing similar effects (Eisenberger Armeli ampPretz 1998 Eisenberger Haskins amp Gambleton 1999Eisenberger amp Selbst 1994) However Eisenberger et al(1999) failed to replicate the transfer-of-training effectThe variability and repeat groups responded similarly dur-ing the test phase In this study separate experimenterstrained and tested the subjects thereby avoiding possibleexperimenter-expectancy effects whereas the same ex-perimenter did both training and testing in Eisenbergerrsquosother studies Across the series of experiments there weremany other cases in which transfer was not observed andmore research is necessary to explain why

In Lee et alrsquos (in press) study with autistic individualsdescribed above after variable answers were trained toquestions verbal variability transferred to new experi-menters and new contexts

Arnesenrsquos (2000) results described above are consis-tent with transfer of training in an animal model Rats inthe Var group were trained to interact variably with oneset of objects When later tested in a new situation withnew objects the Var animals showed (in comparisonwith controls) higher levels of exploration manipulationof the novel objects and discovery of hidden reinforcers

Much of the evidence reviewed in this section there-fore indicates transfer of training and supports a generalldquovaryrdquo competency However there remain unresolvedquestions the most important of which is the extent towhich transfer occurs Note also a methodological pointMany of the studies tested whether transfer occurs whenthe contingencies are neutral with respect to the to-be-transferred behavior Holman et al (1977) is a case inpoint Variations were reinforced in one domain draw-ing and the question was whether variations wouldoccur in another domainmdashblock constructionmdashin whichreinforcement did not depend on varying A negativef indingmdashin this case absence of transfermdashindicatesnothing about the ability to generalize but only aboutthe tendency to do so given the absence of a reason (orcontingency) A better test would be to compare experi-mental and control performances when variability in thenew task is in fact functional as in Arnesen (2000)

SummaryBefore variability can be reinforced it must occur for

reasons other than reinforcement Some of the reasonsinclude variability inherent in all behavioral phenomenaldquoaccidentalrdquo environmental events that are unrelated toparticular behaviors and withholding of reinforcementTransfer from other situations may serve a similar functionFor example if an organism can discriminate shaping-

of-new-response periods from others then it may learngenerally to ldquovaryrdquo during shaping episodes After rein-forcement makes initial contact with response variationsthere are at least three behavioral strategies that can beemployed in order to maintain functionally high levelsof variability respond based on external random eventssuch as the toss of a coin avoid repetitions by remem-bering what responses had previously been made and re-spond stochastically Evidence for an endogenous sto-chastic process was seen in the similarity of responsedistributions to those generated by stochastic sourcesand by findings that interfering with memory for priorresponses either improves or leaves unaffected stochas-tic behaviors Stochastic responding as well as othertypes of variability-generating strategies depend uponcontingencies to define the required level of variabilityand the set from which variations emerge

IMPLICATIONS

One implication of reinforced variability is that oper-ant responses may differ in the extent to which their vari-ability is affected by reinforcement contingenciesmdashthatis operants may be ordered along a continuum of sensi-tivity of variability to reinforcement Another is that re-inforced variability causes us to question the generalityof some basic goals of behavioral psychology includingprediction and control A third is that reinforced vari-ability may help us to provide a scientifically useful wayto describe voluntary action In particular the variabilityof all voluntary behaviors can readily be changed mo-ment to moment by discriminative stimuli and conse-quences Implications for everyday life also will be noted

Operant ContinuumAs was indicated above the operant is composed of

instances that is the operant is a class of many differentresponses all functionally equivalent but often topo-graphically distinct Skinner suggested and I hypothe-size that instances within the class emerge stochastically(see Moxley 1997 for a discussion of Skinnerrsquos posi-tion) Although reinforcement contingencies and contextinfluence the individual probabilities and their distribu-tions emission is stochastic nonetheless The particularmember of the operant class generally cannot be predictedwith a high degree of certainty For example a particu-lar rat may be more likely to press the lever with its leftpaw than its right but left- and right-paw responses areemitted probabilistically This class-nature of the operantsuggests one way to conceptualize operant variability interms of size of class and within-classes probability dis-tributions High variability implies large classes equalprobability distributions or both low variability impliesthe opposite

Variability may be modified more readily in some op-erants than in others implying that operant variability isnot an all-or-none phenomenon but rather a continuumwith reinforced variations in variability (RVV) differing

698 NEURINGER

for different operants RVV might best be understood byanalogy to the concepts of speed and acceleration Bothvariability and speed range from low to high RVV isanalogous to acceleration the rate of change in speedHighly sensitive RVV is indicated by the possibility ofreinforcing many different levels of variability from rep-etition to random-like responding and by the fact thatchanges in level occur readily and rapidly

Different operants manifest differences in RVV Forexample variability of chain pulls changed more thanthat of leverpresses when variability was reinforced inrats (Morgan amp Neuringer 1990) Some operant-like re-sponses may show relatively low RVV For example fir-ing rates increase in single hippocampal nerve cellswhen dopaminergic substances are contingent upon theresponse (Stein Xue amp Belluzzi 1994) suggesting thatnerve f iring is controlled by reinforcement To datehowever there is no evidence that variability of nervefiring is similarly controlled Other cases might includereinforcement of nondiscriminable (subconscious) re-sponses such as Hefferline and Pererarsquos (1963) reinforce-ment of subaware thumb twitches and biofeedback ofmuscle and cardiovascular activity In these cases rein-forcement may modify performance but it is not knownwhether variability is also sensitive to reinforcement andif so to what degree Human verbal variability may bethe most readily manipulable of all behaviors the rangeof predictable to unpredictable verbal responses beingimmense with extraordinary sensitivity to context andconsequence RVV may also differ across organisms (per-haps from simple to complex) individuals (perhaps asan index of intelligence or learning ability) and physio-logical and psychological states (WKY vs SHR depres-sion and autism)

RVV may help to distinguish operant responses fromother superficially similar cases of selection by conse-quences That is the ldquooperantrdquo may also best be concep-tualized as a continuum with RVV an important deter-miner of position along the operant continuum Oneexample of selection by consequences but low ldquooperant-nessrdquo is genetic change Variability is essential for evolu-tionary change of course and levels of genetic variabil-ity are sometimes affected by environmental conditions(Moxon 1997) but there are few examples of levels ofvariability being influenced by selection pressures Thatis the source of much genetic variability is thought to berandom not influenced by current pressures and needsThus although selected by consequences genetic changemay be low along the operant continuum Other exam-ples of operant-like phenomena for which variability although important may be relatively insensitive to con-sequencesmdashvariability in these cases is elicited not se-lectedmdashinclude the development of the central nervoussystem and immune system responses (Hull et al 2001)In both of these cases change depends upon selection byconsequences but it is not clear whether variability canalso be selected Most commonly studied operant re-sponses on the other hand show high sensitivity of vari-

ability to reinforcing events But to reiterate the mainpoint of this section operants differ in terms of RVV andtherefore in terms of their ldquooperantnessrdquo from low sensi-tivity (possibly responses of single nerve cells) to extra-ordinarily high sensitivity (verbal behavior) The operantnature of variability as well as that of every other oper-ant dimension is best conceptualized as a continuum

Prediction and ControlSmith (1992) described Skinnerian operant condition-

ing theory as Baconian in nature a primary goal of Skin-nerian behaviorists being to control behavior and hecontrasted this with the more Aristotelian goal of de-scription Skinner and his students (eg Zuriff 1985) in-deed emphasized prediction and control Many of thestudies reviewed above however involved reinforce-ment procedures that increase variability and conse-quently decrease predictability It is not only ignorancetherefore that restricts prediction and control as Skin-ner suggested but reinforcement itself Operant meth-ods and theories when applied to variability lead togreater emphasis on description with control over pre-cise response characteristics possible under some butnot all conditions

In summary an operant is a class of responses underthe control of contingencies of reinforcement The classcan be controlled and predicted given knowledge ofconditioning history Indeed the size of the class andpossibly the distribution of within-classes probabilities orlevel of variability can also be controlled and predictedBut to the extent that instances emerge stochastically aparticular instance cannot be predicted The instant re-sponse can be explained as a member of a particularclass but prediction of the instance will be diff icultsometimes to the extreme This point will be developedin the next section

Voluntary BehaviorI hypothesize that voluntary behaviors are operants for

which RVVs are highly sensitive to influence by conse-quences One way to test this hypothesis would be togather empirical data concerning how voluntary versusinvoluntary actions are classified by human observers Asfar as I know there are no psychophysical data concern-ing this question but the hypothesis implies that highsensitivity of variability to operant contingencies RVVswill be present in all actions perceived to be voluntary

This operant-variability theory of voluntary action isbased on the supposition of two attributes First volun-tary acts must be potentially explainable (given suff i-cient knowledge) on the basis of genetics conditioninghistories current contingencies anticipated conse-quences intentions goals and the like That is volun-tary behaviors are lawful The second attribute is that thevoluntary act must be potentially unpredictable That isvoluntary acts are ldquofreerdquo of control by identifiable stim-uli If behavior is perceived necessarily to result frommdashto be predicted on the basis ofmdashexternal events includ-

OPERANT VARIABILITY 699

ing reinforcement contingencies it will not be perceivedas voluntary ldquoVoluntaryrdquo implies potential indepen-dence from stimulusndashresponse or responsendashreinforcerdetermination According to this theory both explain-ability and unpredictability are necessary attributes ofvoluntary acts

How though can behaviors be simultaneously ex-plainable and unpredictable Reinforcement contingen-cies play two essential roles Reinforcement (togetherwith genes prior experiences and current environment)helps to determine the set of possible behaviors fromwhich a voluntary act emergesmdashthat is the operant classReinforcement (together with the other contributorsspecified above) also helps to determine the level of vari-ability within the class Thus both instances comprisinga class and level of variability within that class can bepredictedmdashagain given sufficient knowledge Howeverthe individual response is at least sometimes (or relatively)unpredictable The reason is that the operant class fromwhich the response stochastically emerges is sometimesextremely large (eg ldquoname a wordrdquo or ldquodo somethingrdquo)and the contingencies sometimes call for equiprobablestochastic emergence (eg ldquofool merdquo) In the latter casea voluntary act may appear to be foolish meaninglessor unwarranted but if it is voluntary it can potentially beexplained (For example in response to the questionldquoWhy did you do that silly actrdquo the answer could beldquoJust to show that you canrsquot predict my behaviorrdquo seeScriven 1965) In these cases prediction of instancesfalls to very low levels indeed and I hypothesize evenan all-knowing observer will be unable to predict thenext response at greater-than-chance levels which aregiven by the size of the operant class and by within-classesprobability distribution Even within small classes (egin the game rockndashscissorsndashpaper in which unpredictabil-ity among the three possible responses is functional) be-havioral stochasticity may lead to unpredictability of theinstant response Note however that although voluntarybehaviors are sometimes unpredictable the opposite isoften the case When contingencies call for highly con-strained particular responses predictions can easily bemade For example if a police officer asks your nameyour response can most likely be predicted Voluntary re-sponses are therefore described as potentially unpre-dictable with variability contingencies engendering dif-ferent levels of unpredictability

According to this theory behaviors low in the sensitivity-of-variability dimension are described as relatively in-voluntary and those that are high in said dimension areperceived as voluntary Low RVV may be shown at anylevel along the variability continuum For example al-ways responding in the same unchanging way will beperceived as involuntary but so will always respondingrandomly (The coin does not voluntarily come up withheads or tails) This point is essential More than 2000years ago Epicurus argued that occasional randomswerves of atoms provided the source of voluntary be-havior in an otherwise determined universe but al-

though this conjecture was prescient it was not quitecorrect ldquoFree-willedrdquo or voluntary action whether it isbased on random swerves or on quantum mechanics isnot explained by variability per se but by functionalchanges in levels of variabilitymdashthat is by rapid andprecise changes in response to environmental demandsResponses can be placed along a voluntaryndashinvoluntarycontinuum isomorphic to the operantndashrespondent con-tinuum with voluntary behavior being functional pre-cisely in the same way as operant behavior is ThusSkinnerrsquos (1974) suggestion that voluntary behavior isoperant behavior was almost correct Only those oper-ants for which variability is highly sensitive to conse-quences will be perceived as voluntary

One additional example might help Operant classes(sets of potential responses) are created ldquoon linerdquo as we goabout our daily activities As we walk from car to officewe tend to walk in a characteristic way take the samepath with tightly constrained operant classes followingone after the other in rapid succession Class sizes aresmall and within-classes probability distributions arehighly peaked Predictions of next responses by a knowl-edgeable observer can readily be made However classsizes can be increased and probability distributions flat-tened we can walk in quite strange ways indeed dancealong a new path sing hand candy to passers-by Strangebehavior indeed but made possible by the ability of op-erant sets to be modified moment to moment not only interms of class definition but in terms of class size lev-els of within-classes variations and so on The same istrue for conversation Moment-to-moment changes inverbal class definition size and predictability of within-classes instances are hallmarks of voluntary behavior

GenerativityRelated to the definition of the operant and to our discus-

sion of voluntary action is that fact that operant behaviormdashleverpressing no less than languagemdashis by its naturegenerative This claim follows from the functional na-ture of the class of responses (all members of the classare effective in producing an outcome) and the stochas-tic emission of instances From these flow never-before-emitted instances patterns and sequences (see Epstein1996) Operants especially those high on the RVV con-tinuum are inherently generative

Self-ExperimentationVariations systematic or otherwise are a necessary part

of scientific work Neuringer (1981) and Roberts and Neu-ringer (1998) described how variations can be brought to bear explicitly on everyday life in the form of self-experiments Individuals vary aspects of their lives suchas the foods they eat social contacts and methods ofmemorizing they keep records concerning the effects ofthese manipulations and form hypotheses concerningcontrolling variables Using self-experimental methodsindividuals can learn how what when where and howmuch to vary Explicitly varying onersquos everyday behav-

700 NEURINGER

iors and thoughts may help to engender what Langer(1989 2000) has referred to as mindful livingmdashthat isnonautomatic voluntary engagement with beneficialeffects in different areas including learning The impor-tant point here is that variability can be voluntarily cho-sen maintained and modified in everyday life and thatexplicit variations may have important consequences

Implications for SocietyConsideration should be given to the importance of re-

inforcement contingencies to support variability in so-cial milieus A basic point made throughout the presentarticle is that merely permitting behavior to vary doesnot necessarily engender or maximize variability (as isshown by Yoke and Prob schedules) Analogously merelypermitting varied social behavior as in a free or laissez-faire society may not suffice to produce the desired di-versity If the results shown for individual organismscan provide useful information for groups (see eg Grottamp Neuringer 1974) then when variations are desiredsocieties might reward individuals for varying not sim-ply permit it

SummaryOperants differ in the extent to which reinforcers in-

fluence levels of variation When variability is highlysensitive to reinforcement the operant may be perceivedto be voluntary Voluntary responses have two attributesThey are functional implying that the class of responsesmdashthe operantmdashcan be predicted given sufficient knowl-edge But instances emerge stochastically implyingsome degree of unpredictability Because reinforcementof variability can generate extremely large classes withstochastic emergence of instances it suggests limitationsof the two basic goals of Skinnerian behavioral theorymdashnamely prediction and control Operant variability also hasimplications for self-experimentation and for society

RELATED AREAS

Operant variability may be important in many otherareas and I will briefly review some of these

Negative reinforcement To my knowledge there havebeen no published studies in which variability is nega-tively reinforced High variability might sometimesavoid or reduce a punisher such as loss of goods ormoney but at other times low variability could be func-tional Effects of positive and negative reinforcement onvariability may have different evolutionary bases onederived from increasing probabilities of locating sourcesof food or mates the other from avoiding confrontationsor predation (Driver amp Humphries 1988) One questionconcerns how quickly variability can be changed by neg-ative as opposed to positive reinforcement There isconsiderable evidence that loss functions differ fromgain functions (Kahneman amp Tversky 1984) possiblyindicating that positive reinforcement-of-variabilityfunctions will differ from negative ones as well

Resistance to change In some of the studies describedabove variability appeared to be more resistant tochange than repetitions were for example alcohol hadlittle effect on variable operant responses but interferedwith repeated ones Doughty and Lattal (2001) tested thegenerality of these f indings using four-response se-quences with pigeons a threshold variability contin-gency in the Var component of a multiple schedule andLRLR in the Rep component Doughty and Lattal chal-lenged stability of responding by prefeeding the birds inone experiment and providing reinforcers independentlyof responding in the other (see Nevin 1992) Operantvariability was again more resistant to change than wasoperant repetition Thus reinforcement of variabilitymight increase the stability of responding (that is its re-sistance to change) a seemingly paradoxical findingHowever Mechner (1992) suggested that this is the casefor complex skills such as piano performance Varyingpractice routines will make the performance relativelyinsensitive to unforeseen changes in the environmentCare must be taken however when variation is comparedwith repetition because they have different measuresAlso the results reported by Doughty and Lattal werebased on the performance of three pigeons and were notconsistent throughout the experiment The hypothesiswarrants further study

Attraction In some bird species complex male songsattract females (for a review see Catchpole amp Slater1995) There is evidence that male song complexity isunder the discriminative stimulus control of a receptive fe-male (Searcy amp Yasukawa 1990) That explicit reinforce-ment may play a role in such song complexity is suggestedby the Manabe et al (1997) study described above An in-teresting question for further research is the extent towhich operant contingencies contribute to the attractioncreated by behavioral variability in these and other species

Discrimination of variability Animals and people canuse the variability of a stimulus array or entropy as a dis-criminative stimulus (Young amp Wasserman 2001) Whatis the relationship between perception of variability onthe one hand and emission of variable responses on theother Do individuals attempt to match their own levelsof variation to those of others in their environments whichwould involve both perception and emission How mightdiscrimination of onersquos own level of variability influenceonersquos ability to modify such levels and do influences thatobscure discriminations therefore affect ability to change(see eg van der Meer van der Weel amp Lee 1995)Hunziker et al (2002) report that although variabilitywas successfully reinforced their human subjects couldnot correctly describe the contingencies of reinforce-ment a finding similar to those of informal observationsin other studies (eg Neuringer Deiss amp Imig 2000)but whether subjects could discriminate variability levelsrather than contingencies was not studied

Attention and awareness When attention is subject tointerference or is withdrawn from the task at hand abilityto vary is degraded (eg Baddeley Emslie Kolodny amp

OPERANT VARIABILITY 701

Duncan 1998 Evans amp Graham 1980 Neuringer amp Voss2002) It is unclear at present how attention influencesthe different processes underlying operant variabilityand in particular memorial and stochastic processesRich areas for additional study are the relationships be-tween attention awareness and consciousness on theone hand and operant variability on the other Voluntaryaction and consciousness are often related (see egLibet 1999)

Concepts Concepts are thought by many to be sets ofinstances (much like operant sets) whose structures aregraded meaning that there are more or less typical in-stances of such concepts as birds (robin is typical ostrichis not) Barsalou (1987) showed that the gradation ofthese cognitive structures is manipulable or as he putsit not stable A number of variables influence structureincluding context individual differences and points ofview Barsalou argued that these structures are createdas the need arisesmdashthat is on line by the thinking indi-vidual The structure of concepts may parallel that of op-erants and in both cases levels of within-structures di-versity or variability may be modified by consequencesOperant variability may be found in the cognitive as wellas in the behavioral domain

Play and humor Highly variable behaviors are ob-served when animals and people especially children en-gage in play and humor (see eg Fagen 1981) The re-sult of play may be acquisition of new responses (seeSutton-Smith 1975) Although not currently needed re-sponses learned during play may provide the source ofvariations for successful shaping of behaviors at anothertime This is quite analogous to the role of conserved ge-netic variants which are passed along to progeny but donot currently serve any identifiable role Play may alsoinvolve learning how to vary operantly

Reinforcement by variability The self-maintainingnature of play suggests that engaging in variable behav-iors may itself be reinforcing Indirect support comesfrom findings that varied stimulation can serve as a re-inforcer and that stimulus repetition becomes aversive(Fiske amp Maddi 1961) Possibly also related are findingsthat varying reinforcers and permitting choice among re-inforcers increase their value (Bowman Piazza FisherHagopian amp Kogan 1997) and that permitting choicesamong varied responses is also preferred to demandingfixed responses (Catania 1980)

Foraging Two issues regarding variability are presentin the foraging literature One has to do with risk prefer-ence versus avoidance (see eg Shettleworth 1998) Asecond has to do with search strategies Most reportedcases of foraging involve systematic search patterns(Stephens amp Krebs 1986) However search patterns aresometimes stochastic or highly variable (see eg Peter-son 1997) One question is whether some species can varybetween these two strategies and the challenge is to iden-tify the pressures leading to and the advantages of each

Animal training Wilkes (1996) has applied operantvariability techniques in developing training proceduresfor pet dogs More generally reinforcement of variabil-

ity may enable the gaining of control over behaviors ofanimals and as has been suggested above may be an im-portant part of the process of shaping

Models of learning Reinforced variations in variabil-ity may be important when human and animal learningis being modeled whether the models are mathematicalin form computer based or machine based (Blumberget al 2002 Brooks 1999 Sutton amp Barto 1998)

CONCLUDING COMMENT

Genes interact with environments to produce diverseanatomical physiological and behavioral phenotypesGenes and environments also provide for another level ofdiversitymdashnamely operant variability There are manysources of operant variability including random environ-mental events memory for past responses and stochasticgeneration each of which ultimately derives from ge-netic and environmental influences To the extent thatoperant variability is based on a stochastic source how-ever the behavioral results cannot be predicted fromknowledge of these influences That is organisms haveevolved to behave unpredictably The positing of sto-chasticity to explain behavior has a history extending atleast as far back as Epicurus with major contributors in-cluding Gustav Fechner (see Heidelberger 1987) EgonBrunswik (see Gigerenzer 1987) Skinner (1959) andSuppes (1984) Knowledge of the stochastic nature ofthe operant and of how it is controlled leads to a worldview that differs from genetic and environmental deter-minism with roots in the works of the scholars just men-tioned At the scientific level this view leads to the ex-perimental analysis of consequence-influenced variabilityAt the philosophical level it leads to explanations of vol-untary action At the personal level it can lead to change

REFERENCES

Amabile T M (1983) The social psychology of creativity New YorkSpringer-Verlag

Antonitis J J (1951) Response variability in the white rat duringconditioning extinction and reconditioning Journal of Experimen-tal Psychology 42 273-281

Archer J amp Birke L (1983) Exploration in animals and humansCambridge Van Nostrand Reinhold

Arnesen E M (2000) Reinforcement of object manipulation in-creases discovery Unpublished undergraduate thesis Reed College

Baddeley A D (1966) The capacity for generating information byrandomization Quarterly Journal of Experimental Psychology 18119-129

Baddeley A [D] Emslie H Kolodny J amp Duncan J (1998)Random generation and the executive control of working memoryQuarterly Journal of Experimental Psychology 51A 819-852

Baer D M amp Sherman J A (1964) Reinforcement control of gen-eralized imitation in young children Journal of Experimental ChildPsychology 1 37-49

Baker M J amp Koegel R L (1999) Transforming children withautismrsquos ritualistic behaviors into positive play interactions with sib-lings Paper presented at the meeting of the California Association ofBehavior Analysis San Francisco

Balsam P D Deich J D Ohyama T amp Stokes P D (1998) Ori-gins of new behavior In W OrsquoDonohue (Ed) Learning and behav-ior therapy (pp 403-420) Boston Allyn amp Bacon

Balsam P D Paterniti A Zechowy K amp Stokes P D (2002)

OPERANT VARIABILITY 673

has two possible states heads up and tails up Novel re-sponses are those that have not previously been ob-served and the term generally refers to the behavior of agiven organism although sometimes it refers to that of agroup or culture Creative responses are novel or vari-able ones that meet an additional criterion often an aes-thetic (eg artistic) one but sometimes a functional (egscientific) one Lastly many of the experiments to be re-viewed compare variable and repeated operants an exam-ple of the latter being repeated responses by a rat on a sin-gle lever under the control of reinforcement schedules

ObjectionsObjections are often raised to the suggestion that vari-

ability can be reinforced For example a reinforcer af-fects the response that produces it but variability refersto a set of responses not a single one How can a set ofevents possibly extending far back in time be rein-forced Another objection is that reinforcement by itsnature organizes and strengthens responses and rein-forcement of variability is therefore not possible An-other is that attempting to reinforce variability is incon-sistent with two basic goals of behavioral psychologistsmdashnamely to predict behavior and to control it Another isthat good experimental procedures should minimizevariability not increase it Yet another is that variabilityis perceived or posited only when an observer is igno-rant of its underlying causes Finally adherents of a de-terminist philosophy may argue that variability does notexist For these reasons reinforcement of variability issaid to be undesirable perhaps impossible and forsome not meaningful The evidence as we will see in-dicates otherwise The study of reinforced variabilityprovides a new way to view operant behavior and opensnew areas of research

INITIAL STUDIES

Reinforcement of Variable and Novel ResponsesThe proposal that variable behaviors can be reinforced

goes back at least 40 years but its history has been a con-tentious one In this section some of that history and thesurrounding debate will be described Maltzman (1960)was among the first to suggest that human subjects couldlearn to generate low-probability responses in this caseverbal responses to stimulus words Experimental sub-jects were asked to devise many different associates tothe same word whereas a control group had to produceonly one Because of their training the experimentalsubjects came to generate lower probability associatesthan did the controls Although these findings imply thatlow-probability responding can be trained there was noclear evidence that reinforcement played a role

However reinforcement was clearly involved whenPryor Haag and OrsquoReilly (1969) rewarded porpoises forjumping turning or swimming in novel ways In eachsession the trainer waited for a response that had notpreviously been observed and that response became the

reinforced target throughout the session The porpoisescame to emit novel and complex behaviors wheneverthey entered the experimental context Indeed they be-haved in ways never before seen in the species This wasan influential study and it led to additional research

Goetz and Baer (1973) for example rewarded preschoolchildren for block-building constructions that differedfrom one another A particular form was reinforced uponits first occurrence within a session but in contrast tothe procedures used by Pryor et al (1969) many differ-ent forms were reinforced during each session As train-ing proceeded the children constructed increasingly dif-ferent forms including novel ones When reinforcerswere later contingent upon repetitions the children re-peatedly constructed a single form Thus variability andrepetition were controlled by consequences

Although Maltzman (1960) Pryor et al (1969) andGoetz and Baer (1973) advantageously studied naturallyoccurring behaviors some problems were also identi-fied In Pryor et al the responses became so complexthat the experimenters had diff iculty distinguishingamong them In both Pryor et al and Goetz and Baer theexperimenters had to identify each form or response pat-tern and decide when a new one had occurred prior toeach reinforcement In all three studies novel responsesbecame increasingly diff icult since the ldquoeasyrdquo oneswere emitted first Also in the Maltzman and the Goetzand Baer studies verbal feedback was provided and thecontribution of descriptive information was difficult todistinguish from that of reinforcement Alternativemethods were needed especially if the goal was to ana-lyze reinforced variability experimentally and Blough(1966) provided an ingenious one

Reinforcement of Variable IRTsBlough (1966) rewarded pigeons for pecking a re-

sponse key if the time between consecutive pecks orIRT had occurred least frequently in the recent past Thisconstituted an attempt to see if pigeons could learn to re-spond randomly over time similarly to what takes placeduring the emission of atomic particles at random inter-vals during radioactive decay The birds were surpris-ingly successful To get a sense of the reinforcementcontingencies imagine a stochastic process in which re-sponses occurred with a 05 probability during each sec-ond since the previous response Response probabilitywould be 05 during the first second and if a responsehad not occurred it would be 05 during the next secondand so on Given 1000 responses approximately 500would occur during the first second since the previousresponse that is in the 0ndash1-sec time frame or whatBlough referred to as a ldquobinrdquo 250 responses would occurin the 1ndash2-sec period since the previous response 125 inthe 2ndash3-sec bin and so on Blough created 16 IRT binsadjusting their sizes so that a random generator wouldproduce equal numbers in the counters associated witheach bin Because as has just been described manymore short than long IRTs are emitted bin size was small

674 NEURINGER

at the short end and large at the long end and increasedsystematically across the IRT range For the response tobe reinforced a pigeonrsquos IRT had to fall in the bin thatcontained the fewest prior entries across a moving win-dow of 150 responses Fine-grained analyses of the pi-geonsrsquo performances showed that successive responseswere not truly independent as would be expected from arandom generatormdashthe tendency for long IRTs to followlong ones and for short IRTs to follow short ones wasgreater than would be expected from a random emittermdashbut this might have been due to particular aspects of theprocedure including the facts that short IRTs were rein-forced somewhat more frequently than longer ones andthat if there were few entries in a bin (because the birdhad not been responding at that IRT) then successiverepetitions of the same IRT could be reinforced repeat-edly Despite these problems the data were impressivelyclose to those of a random model as is shown in the toppanel of Figure 1 To the extent that the functions are lin-ear on the semilogarithmic coordinates the behavior ap-proximates that of a stochastic process The advantagesof Bloughrsquos procedure were that response classes weredefined objectively and independently of a human ob-server reinforcers were administered automatically thou-sands of responses were emitted per session and ses-sions could continue indef initely without changes inresponse difficulty or contingency

Shimp (1967) attempted to extend Bloughrsquos (1966) pro-cedure to pigeon choices Because choices are currently amajor focus of interest and because there were problemswith Shimprsquos procedure it will be discussed in some de-tail Pigeons pecked two keys left (L) and right (R) in acontinuous stream such as LLLLRRLLRRRLRLLL thedata were analyzed on line in terms of overlapping setsof four responses In the example just given assume thateach new response is added to the right of the string Themost recent set was composed of RLLL the second mostrecent (moving to the left by one response) of LRLL thethird of RLRL and so on Given two alternatives L andR and sets consisting of four responses there are 24 or16 possible sequences LLLL LLLR LLRL RRRRReinforcement was provided if the most recent responsehad terminated a sequence that had occurred less fre-quently than all other 15 possible sequences over a mov-ing window of 96 responses Although the resultingchoice distributions were quite variable in some respectsthe results did not match those of a random model andwere diff icult to analyze possibly because of the fol-lowing problem

Assume that 16 counters represent the 16 sequences ofresponses and that one count is added to each counterwhenever its associated sequence is emitted If LLLLhas just been emitted one count will be added to thatcounter Another L will increase the same counter andeach successive L will add another If LLRR has justbeen emitted however thereby adding one count to itsassociated counter four additional responses (not 1 as inthe LLLL case) will be required to add again to the same

counter Therefore 16 successive responses will be re-quired for four instances of LLRR (namely LLRRLLR-RLLRRLLRR) as opposed to seven for L (LLLLLLL)Although a random process would generate equal fre-quencies of all sequences including LLLL and LLRRthe distribution of successive occurrences would differthereby affecting the distribution of reinforcements Thatis LLLL and RRRR would tend to be more ldquoclumpedrdquothan would LLRR and RRLL which would be more dis-tributed Similar considerations apply to all other se-quences That these differences might affect choice dis-tributions is indicated by the many studies showing thatthe distribution of reinforcements over time given equal

Figure 1 (A) Data from one pigeon in Blough (1966) under dif-ferent parameter conditions (the three sets of lines) with replica-tion at each parameter Shown are the mean numbers of re-sponses emitted in each half-second IRT bin a straight line beingexpected from a stochastic responder Adapted from ldquoThe rein-forcement of least frequent interresponse timesrdquo by D S Blough1966 Journal of the Experimental Analysis of Behavior 9 p 587(B) Data (points) from one pigeon at the end of training in Machado(1989) The line represents the frequencies expected of a randommodel From ldquoOperant conditioning of behavioral variabilityusing a percentile reinforcement schedulerdquo by A Machado 1989Journal of the Experimental Analysis of Behavior 52 p 161 Bothfigures copyrighted by the Society for Experimental Analysis ofBehavior Reprinted with permission

OPERANT VARIABILITY 675

average reinforcement frequencies affects choices for ex-ample choices are affected differently by variable-interval(VI) than by fixed-interval (FI) schedules of reinforcement

Wilson (1986) devised one solution to this problemWilsonrsquos procedure is complex but in brief responding wasanalyzed in a manner that avoided overlapping sequences(which was the cause of the clumping in Shimprsquos 1967procedure) Four sets of counters were created Using thesame example given above in the first set one count wasadded to each counter representing RLLL RRRL RRLLand LLLL so that successive trials were completely inde-pendent (Note the absence of overlap) When the next re-sponse was emitted sequences began with that responsemdashlet us assume that the next response added to the rightof the series was an Lmdashand therefore in the examplegiven the second set was created by offsetting responsesone place to the left Now one was added to LLLL withprevious counts added to RRLR RLLR LLLR and soon A similar procedure was followed for each of the re-maining two sets Each response therefore terminated afour-response sequence in one and only one of the foursets and reinforcement was based on satisfaction of thecontingency within that set only Wilsonrsquos procedure hadother aspects but for present purposes it suffices to notethat the dependencies in Shimprsquos procedure were avoidedand when pigeons were tested under these contingen-cies they generated choices that more closely approxi-mated those of a random model Hopefully this lengthyexplanation indicates the importance of precise contin-gencies when one is attempting to reinforce variabilityAn advantage of Shimprsquos and Wilsonrsquos procedures wasthat every response could be reinforced at least poten-tially as was the case for each of the other procedures dis-cussed until now However a simpler way to avoid theproblems just described is to require a fixed number ofresponses in each trial with reinforcement possible onlyat completion of the trial As will be seen most recentstudies have used the latter tactic

The evidence generally supported the claim that be-havioral variability could be reinforced but the agree-ment about these findings was short lived Schwartz (19801982) reached the opposite conclusion and his researchwill be considered in detail

Schwartzrsquos ResearchSchwartz (1980 1982) used fixed-length trials con-

sisting of eight responses per trial by pigeons across tworesponse keys During an initial control phase a trial ter-minated with food if it contained exactly four L and fourR responses regardless of the particular sequence of Lsand Rs However if either key was pecked a fifth timethe trial terminated immediately with timeout A 5 3 5light matrix was provided to assist the birds in this taskAt the beginning of a trial the upper-left corner light wasilluminated and all others were off An L peck movedthe light one column to the right and an R peck movedit one row down Any sequence of four L and four R re-sponses would therefore move the light from upper left

to lower right and the trial would end with food Each ofthe birds came to emit a single sequence with high prob-ability in most cases LLLLRRRR or RRRRLLLL Arelatively low success rate at the end of 40 training ses-sions indicated that the task was harder than it mighthave appeared Only 66 of trials ended with food andthe remaining 34 ended with timeout because withina given trial one of the keys had been pecked a fifth time

Schwartz (1982) then required that the sequence in thecurrent trial differ from that of the just preceding trialThis will be referred to as a lag 1 variability + constraintcontingency (ldquoconstraintrdquo indicating the required fourLs + four Rs) If a bird had emitted LLLLRRRR in theprevious trial for example any other sequence would bereinforced in the next trial assuming that it containedfour Ls and four Rs but a repetition led to timeout Thebirds performed poorly only 36 of the trials now end-ing with food because a dominant sequence was oftenrepeated or one of the keys was pecked five times Hy-pothesizing that this failure to reinforce variability mayhave been due to the birdsrsquo extended experience underthe initial control condition (in which variations were notrequired and any sequence of four Ls and four Rs wasreinforced) Schwartz (1982 Experiment 1) again triedthe variability-requiring procedure with naive subjectsAgain the birds failed In another experiment Schwartz(1982 Experiment 2) explicitly trained birds to alternatebetween LLLLRRRR and RRRRLLLL sequences andthen introduced the lag 1 variability + constraint contin-gency but again the birds failed to vary Schwartz (1982)concluded that variability could not be reinforced andoffered a number of possible explanations for why his re-sults differed from the previous findings Perhaps Maltz-manrsquos (1960) and Goetz and Baerrsquos (1973) findings weredue to the information providedmdashverbal indicationsfrom the experimenters that subjects should varymdashrather than to reinforcement Perhaps Pryor et alrsquos (1969and by implication Goetz amp Baerrsquos 1973) results weredue to brief extinction periods in which reinforcers werewithheld (because variations were not forthcoming)rather than to direct reinforcement of variability PerhapsBloughrsquos (1966) results were specific to IRT and not rel-evant to response forms or choices These explanationsall imply that it is not possible to reinforce variability orthat it is possible only in narrowly defined cases As willbe seen Schwartzrsquos (1982) results readily met the test ofreplication but there was a problem nevertheless withhis conclusion

Schwartz ExplainedPage and Neuringer (1985) simulated Schwartzrsquos (1982)

lag 1 variability + constraint procedure with random ldquore-sponsesrdquo generated by a computer Although the simula-tion generated L and R with equal probabilities (05) ldquore-inforcementrdquo was obtained on only 29 of its trials whichwas somewhat lower than the level attained by pigeonsThe reason is that of the 256 possible eight-response se-quences only 72 contained exactly four Rs and four Ls

676 NEURINGER

and therefore the random simulation often failed tomeet the 4L + 4R constraint The implication was that ifthe birds attempted to respond randomly they would bereinforced only rarely

To test whether the penalty for fifth pecks was respon-sible for the birdsrsquo failure to vary Page and Neuringer(1985) permitted eight responses across L and R keyswithout the four responses-per-key requirement If thecurrent sequence of eight responses differed from that inthe last trial a lag 1 variability contingency (without con-straint) reinforcement was provided Now the pigeonssucceeded in varying attaining reinforcement on morethan 90 of the trials a rate significantly higher thanthat observed in Schwartz (1982) and close to that of thesimulating random responder under the new procedurePage and Neuringer also replicated Schwartzrsquos (1982)variability + constraint procedure and again the pigeonsfailed just as did Schwartzrsquos The evidence was clearUnder Schwartzrsquos (1982) procedure pigeons failed tovary but penalty for fifth responses was responsible

One additional aspect of Schwartzrsquos (1982) proceduremight also have contributed to low variability If thebirds intermixed a single preferred sequencemdashone thatmet the 4L + 4R constraintmdashwith any other then 50 ofthe trials would end with food this level of intermittentreinforcement being sufficient to maintain responding inmany other situations For example if a bird alternatedbetween LLLLL (the fifth L producing a timeout) and adominant LLLLRRRR sequence the latter would alwaysresult in reinforcement Birds under the variability +constraint contingencies in fact emitted a dominant se-quence with high probability and many of their rewardswere obtained following that sequence (because it hadbeen preceded by a different sequence often one con-taining five Ls or five Rs) Thus failure to vary could beexplained by the fact that variability was not differen-tially reinforced whereas a single dominant sequencewas Schwartz (1988) later reviewed this new evidenceand concluded that pigeons could indeed be reinforcedfor responding in random-like fashion

Yoke ControlOne of Schwartzrsquos (1982) hypotheses is worth further

consideration When variability is reinforced there areperiods during which reinforcement is withheld becauseresponses did not satisfy the contingencies As will bediscussed below many studies have shown that extinctionelicits variability Might short-term extinction periodswhen reinforcers were not forthcoming have elicited thevariability attributed to reinforcement

Page and Neuringer (1985) compared two conditionsIn one referred to as Var reinforcement depended on se-quence variations similar to the lag 1 variability contin-gencies described above (The matrix of lights was nolonger used because it was found not to contribute to re-sponse variability) In the other condition referred to asYoke exactly the same frequency and distribution of re-inforcements were provided but independently of se-

quence variations This was accomplished by having thecomputer maintain a record of which trials were rein-forced and which led to timeout in the Var phase andusing this record to program reinforcers during Yoke Fora particular bird for example the first as well as the sec-ond trials in a Var session might have been reinforcedbut in the third trial the bird may have repeated a se-quence causing a timeout and so forth In the Yoke phasethe first and second trials would be reinforced but thethird trial would lead to timeout whether the birdrsquos re-sponses varied or not Each birdrsquos own sequence of rein-forcements and timeouts in Var was therefore replicatedin Yoke If short-term extinction or withholding of rein-forcement was responsible then variability should beequal in the Var and in the Yoke phases That was not theresult Variability was significantly higher in Var than inYoke a finding that has been confirmed many times inour laboratory and in those of others (eg Barba amp Hun-ziker 2002 Machado 1992) We conclude that variabil-ity was in fact reinforced

VARIABILITY IS AN OPERANT

Control by reinforcement is the most notable charac-teristic of an operant response but there are others includ-ing precise influence of contingencies effects of rein-forcement frequencies control by discriminative stimulilawful distribution of choices and extinction Researchin each of these areas supports the hypothesis that be-havioral variability is an operant dimension similar toother operant dimensions An organism can be rein-forced not only for repeating a particular behavior but fordistributing responses across a wide array of possibilitiesin a way that may be quite unpredictable The evidencewill be reviewed

Contingencies and Frequencies of ReinforcementContingencies specify what must be done to earn re-

inforcement When applied to variability contingenciesspecify a set of responses and in most studies to date aminimal level of variability within that set That controlby contingencies does not depend on reinforcement fre-quency was demonstrated by Blough (1966) for pigeonIRTs and by Machado (1989 1992) for pigeon responsesequences across L and R keys In both cases when re-inforcement depended on high variability response vari-ability was high when intermediate variability was re-quired the level generated was intermediate and whenlower levels of variability were reinforced variabilitywas relatively low Reinforcement frequencies were heldconstant in these studies and therefore were not respon-sible for the changes in the levels

With respect to other operant dimensions such as re-sponse rate reinforcement frequency often has complexand sometimes inconsistent effects As reinforcementfrequency increases response rates sometimes increasesometimes decrease and sometimes are unchanged Vari-ability is also affected in seemingly inconsistent ways

OPERANT VARIABILITY 677

Some studies report that variability increases when rein-forcement frequencies decrease (Boren Moerschbaecheramp Whyte 1978 Tatham Wanchisen amp Hineline 1993Tremont 1984) but others report small or no effects(Blough 1966 Eckerman amp Lanson 1969 Herrnstein1961 Machado 1989)

Grunow and Neuringer (2002 Experiment 1) inde-pendently manipulated variability contingencies and re-inforcement frequencies and their results may help toexplain the previous inconsistencies Four groups of ratswere studied with the contingencies for each group re-quiring a different minimum level of variability (A thresh-old contingency was employed the details of which willbe described below) In the initial phase reinforcementwas provided whenever the respective contingencieswere met (reinforcement frequency was relatively highin this phase) and as can be expected from the Blough(1966) and Machado (1989) studies described above lev-els of variability were directly controlled by the contin-gencies In a second phase with the same variabilitycontingencies held constant reinforcement was limitedto no more than once per minute (VI 1 or intermediatefrequency) In a third phase meeting of the variabilitycontingency was reinforced on an average of only once

every 5 min (VI 5 or low frequency) We observed a sig-nificant interaction between reinforcement frequencyand variability contingency (Figure 2) which helps toexplain some of the previous inconsistencies In thegroup that was reinforced for low variability representedby the triangles in Figure 2 as reinforcement frequencydecreased variability increased slightly in the group re-inforced for high variability represented by the squareswhen reinforcement decreased variability decreasedand in the intermediate groups variability was unchangedAlthough these effects were small they reached statisti-cal significance As can be seen in Figure 2 variabilitycontingencies affected variability more than did rein-forcement frequencies and effects of the latter were in-fluenced by the contingencies

An example might help to illustrate these complexfindings When low variability is required perhaps as in assembly-line work low variability emerges When highvariability is required perhaps as in artistic work re-sponse variability is high Now what happens if the normalrate of reinforcement for each of these tasks is loweredAccording to Grunow and Neuringerrsquos (2002) findingsthe assembly-line workerrsquos variability would increasebut only slightly and the artistrsquos variability would de-

Figure 2 Average U-values across groups of rats in Grunow and Neuringer (2002)The highest function (squares) is from a group reinforced for highest sequence vari-ability the lowest function (triangles) is from a group reinforced under a lenient con-tingencymdashthat is low variability sufficed for reinforcement the other groups (Xs andcircles) were reinforced for intermediate levels of variability Each group was rein-forced at three frequencies each time the contingency was satisfied (continuous rein-forcement ie CRF) on an average of not more than once per minute (VI 1) and onan average of not more than once every 5 min (VI 5) High U-values indicate high lev-els of sequence variability and low U-values indicate relatively repetitive respondingError bars represent standard errors of the mean

678 NEURINGER

crease somewhat Importantly the artist would continueto behave much more variably than the assembly-lineworker Both contingency and frequency affect responsevariability but contingency appears to be more influen-tial at least over the range of values studied to date Thesefindings if general are important for those who withholdreinforcement in order to encourage response variationsas is often the case when behavior modifiers attempt toshape new behaviors Effects of withholding may dependon baseline contingencies and a more productive proce-dure might be to reinforce variability directly We willconsider this suggestion further below

Discriminative ControlThe operant response as defined by Skinner is con-

trolled by discriminative stimuli as well as by conse-quences For example red traffic lights are the discrimi-native stimulus for depressing the automobile brake thedistance of the golf ball from the hole provides a discrim-inative stimulus for how hard to hit the ball In the labora-tory discriminative control is often studied with multipleschedules in which one reinforcement contingency suchas fixed ratio is operative in the presence of one stimulussuch as a red keylight and a different contingencymdashforexample extinctionmdashis operative in a second stimulusmdashfor example a green keylight Discriminative control isshown by differences in responding Variability is alsocontrolled by discriminative cues

Exteroceptive discriminative stimulus controlPage and Neuringer (1985 Experiment 6) reinforced pi-geons for repeating a single sequence LRRLL in thepresence of blue keylights and for variable sequences inthe presence of red Blue and red alternated after every10 reinforcements The birds learned to repeat in thepresence of blue and to vary in the presence of red andwhen the stimulus relationships were reversed the birdsreversed their patterns Cohen Neuringer and Rhodes(1990) obtained similar results with rats Denney andNeuringer (1998) extended this research to show thatdiscriminative control did not depend on reinforced repe-titions in one of the components Again within the contextof a multiple schedule rats were rewarded for variablesequences under one stimulus (Var) and independentlyof variability under a second stimulus (Yoke) Frequen-cies of reinforcement were identical in the two stimulusconditions That the stimuli exerted significant controlover variability was indicated by three main findingsVariability was higher in Var than in Yoke differences invariability were greatest immediately following stimuluschange and when the discriminative stimuli were re-moved and all other aspects of the program kept constantvariability immediately converged on an intermediate andidentical level in the two components The conclusion isthat operant variability is controlled by discriminativestimuli These findings are important in terms of both ex-plaining observed levels of variability and controllingthem For tasks in which response variations are impor-tant such as when new operants are being shaped or when

an individual is attempting to behave creatively or solveproblems cues that indicate ldquoVaryrdquo can be helpful

Discriminative control without exteroceptive stim-uli The relationship between a response and a reinforcercan itself provide a discriminative stimulus even whenexternal cues such as lights or tones are unavailable Forexample under an FI schedule there is a period duringwhich reinforcers cannot be obtained The subject learns topause following each reinforcement although no cuesother than the reinforcer are provided Another case is themixed schedule identical to the multiple schedule describedabove but with no external cues Animals sometimeslearn to respond differentially to the consequences oftheir behaviors even under such stimulus-impoverishedconditions (Ferster amp Skinner 1957) The question stud-ied next was whether responsendashreinforcer relationshipscan provide discriminative cues sufficient to control lev-els of variability This is an important question becauseas was indicated above it has been assumed that with-holding reinforcement elicits variability The present hy-pothesis is that absence of reinforcement can sometimesserve as a discriminative cue for operant variability Thedistinction between elicited (respondent) and reinforced(operant) variability will be revisited shortly

Hopson Burt and Neuringer (2002) rewarded ratsunder a mixed schedule in which two periods of time orcomponents Var and Repetition (Rep) alternated with-out any external cues In the Var component four-responsesequences of L and R leverpresses were reinforced ifthey met a threshold variability contingency in the Repcomponent only repetitions of LLLL were reinforced(Probabilities of reinforcement were equalized in the twocomponents by intermittent reinforcement of LLLL inRep) The results demonstrated discriminative controldespite the absence of exteroceptive cues (see Figure 3)With trials since the unmarked switch into the Var com-ponent shown in the left column variability increasedand LLLL sequences decreased with trials since the unmarked switch into Rep shown in the right columnresponding became increasingly repetitive until LLLLsequences predominated The two functions were sim-ilar which is an important finding in that it indicatesdiscriminative control rather than elicitation If variabil-ity had been elicited because reinforcement was tem-porarily withheld (when the contingencies were switched)then variability should have increased following entryinto both Rep and Var components but that was not theresult

Therefore variability that is observed when rein-forcement is withheld for short periods as when a newresponse is being shaped may be partly discriminativelycontrolled despite the absence of external cues that isanimals and people may learn when it is functional tovary and some of the cues may come from responsendashoutcome relationships Control over this operant vari-ability may be more precise than that elicited by extinc-tion because as was noted above operant contingenciesdefine both the set of acceptable variations and the level

OPERANT VARIABILITY 679

of required variability It may also be maintained muchlonger because as will be seen shortly extinction weakensresponding

ChoiceThe frequency of one response relative to others or

response probability is used as a measure of choice orpreference Choices are found to match the probabilitiesof their consequences For example relative frequenciesof pecking one of two keys approximate relative fre-

quencies of reinforcement Neuringer (1992) showedthat the same matching relationship also governs choicesby pigeons to vary or repeat The more variations werereinforced the more the pigeons chose to vary and sim-ilarly for repetitions Each trial consisted of four L andR responses and reinforcements for varying and for re-peating were systematically manipulated The Var con-tingency was met whenever the current sequence dif-fered from each of the previous three sequencesmdasha lag 3criterion The Rep contingency was satisfied if the cur-

Figure 3 The left column shows the probabilities ( y-axis) that each of five rats met a variability contin-gency (solid circles) versus a repetition contingency (open circles) when they were being reinforced for vary-ing (Hopson Burt amp Neuringer 2002) Along the x-axis are trials since the mixed schedule (no cues pro-vided to indicate the component) had switched into the variability-reinforcing component The right columnshows the corresponding data for performances when reinforcement was contingent upon repetitions thex-axis here indicating trials since the unmarked switch into the repetition-reinforcing component

680 NEURINGER

rent sequence repeated at least one of the previous threeOn an average of once every 30 sec (VI 30 sec) the com-puter probabilistically primed reinforcement for Var orRep the prime being maintained until the reinforcementhad been collected Thus if the computer primed Var re-inforcement was delivered whenever the lag 3 variabilitycontingency was next met Similarly if Rep had beenprimed the next Rep sequence was reinforced Proba-bilities of Var and Rep reinforcements were systemati-cally manipulated the two always summing to 10 and amatching-type relationship was obtained (Figure 4panel A) The relationship was more complex than sim-ple matching possibly because if the pigeons chose tovary by responding stochastically there was some prob-ability due to chance that the current sequence wouldrepeat a previous sequence This was shown by a com-parison of pigeon performances to the performance of acomputer-simulated stochastic model programmed tomatch its Var choices to the probability of Var reinforce-ments The pigeonsrsquo choice distributions were found tobe the same as the modelrsquos (Figure 4 panel B) Thuschoices to vary or repeat were governed by relative fre-quencies of reinforcement An animal can choose whetherwhen and how much to vary a result consistent withthose described above from Blough (1966) Machado(1989) and Grunow and Neuringer (2002) Choices ofwhether or not to vary appear to be governed similarly toall other operant choices

These f indings may have important applicationsWhen an individual repeatedly responds in a way that isnonfunctional reinforcement of choices to vary mayhelp the individual to emerge from the ineffective pat-tern This can be true for those experiencing everydayproblems but feeling helpless to change and it may beespecially important for those experiencing depressionor who respond in otherwise abnormally stereotypedmanners such as those with autism These topics will beconsidered in a later section

ExtinctionAs was indicated above extinction is often found to

increase variability but almost all studies of this effectinvolve responses that had previously been repetitiveAntonitis (1951) for example reinforced rats for pokingtheir noses anywhere along a 50-cm horizontal openingAlthough the exact location of pokes did not matter withcontinued training responses became limited to one or afew locations along the strip When reinforcement waswithheld variability of location increased (see also Eck-erman amp Lanson 1969) Variability produced by extinc-tion has been reported for many other response dimen-sions as well including force (Notterman amp Mintz1965) number (Mechner 1958) topography (Stokes1995) and sequence (Balsam Paterniti Zechowy ampStokes 2002 Mechner Hyten Field amp Madden 1997)

The issue is complicated however by other studiesthat show that previously reinforced responses and re-sponse patterns are maintained intact during and afterextinction That is extinction does not break down

learned patterns For example when Schwartz (1981) re-inforced pigeons for any sequence of four L and four Rresponses each bird developed a dominant sequence asdescribed above and these sequences were maintainedduring extinction Despite decreasing response ratesduring extinction response patterns were retained Sim-ilar maintenance of response structure has been ob-served during extinction following reinforcement of

Figure 4 Average performance across a group of pigeons inNeuringerrsquos (1992) study of choices to vary or repeat (A) Loga-rithms of the ratio of number of times the pigeons met the Varcontingency divided by the times they met the Rep contingencyas a function of the analogous numbers of reinforcements re-ceived for each Points represent group averages in two separateexperiments Also shown is the least squares best-fitting line(B) Changes in U-value an index of response variability as afunction of the obtained percentage of reinforcements for vary-ing The solid line in panel B represents the performance of acomputer-based model that matched its choices to vary (vs re-peat) to its obtained relative frequency of reinforcement for vary-ing From ldquoChoosing to vary and repeatrdquo by A Neuringer 1992Psychological Science 3 p 248 Copyright 1992 by the AmericanPsychological Society Reprinted with permission

OPERANT VARIABILITY 681

choices (Myerson amp Hale 1988) responding underfixed-interval contingencies (Machado amp Cevik 1998)visual discriminations (Nevin 1967) and matching tosample (Cumming Berryman Cohen amp Lanson 1967)How can stability during extinction be reconciled withincreased variability

Neuringer Kornell and Olufs (2001) wrestled withthis question in a study of extinction following rein-forcement under three different contingencies An oper-ant chamber contained two levers left (L) and right (R)on either side of a reinforcement dispenser and a key (K)in the rear of the chamber distant from the dispenserOne group of rats Rep was reinforced for repeating afixed sequence of responses LKR A second group Varwas reinforced for variations across the three operandaunder a threshold contingency Reinforcement for a thirdgroup Yoke was matched to those of the Var animalsand was therefore independent of the particular sequencesBecause of the unusual configuration of operandaskewed distributions of sequence probabilities were ob-served in all of the groups an outcome that was antici-pated and desired in order to document possible in-creases in variability during extinctionmdasha flattening ofthe distributionmdashas well as possible decreases That isduring the reinforcement phase the Rep group came torepeat the LKR sequence with high probability the Yokegroup tended to emit the easiest sequences such as LLLand RRR but distributed their responses over more se-quences than the Rep group did and the Var group dis-tributed most widely although because of differences inthe operanda and in their locations some sequences weremore preferred than others and in particular sequencesending with a K response were least likely (the key beingfarthest from the reinforcement dispenser) Thus thecontingencies resulted in three different distributions orhierarchies of sequence probabilities very steep forRep intermediate for Yoke and flattest for Var

The main finding was that withholding reinforcementhad similar effects in the three groups In each responserates fell to low levels across the four sessions of extinc-tion Furthermore sequences that were most frequentduring the reinforcement phase continued to be most fre-quent throughout extinction A slight flattening of thedistributions was observed however with the lowestprobability sequences increasing relative to the rein-forcement phase Thus because the overall hierarchy ofsequence frequencies was maintained the results indi-cated stability but because the distributions were some-what flattened the results also showed slightly increasedvariability It is as if the strategy during extinction wereto keep doing what had worked in the past but occa-sionally to do something very different This compli-cated story integrates two effects of extinction increasedvariability and maintained stability Most importantly forthe present discussion extinction had similar effects onvariable yoked and repetitive operants

One additional comparison is of interest Recall thatGrunow and Neuringer (2002) found that when contin-

gencies demanded high variability obtained levels ofvariability decreased slightly as reinforcement frequen-cies were lowered This result is opposite to the Vargrouprsquos slight increase in variability just described underextinction conditions The reason may be that extinction-induced variability is short lived lasting only as long asresponding continues perhaps for hours or a few ses-sions at most whereas the variability engendered by theVI 5 intermittent schedule of reinforcement in Grunowand Neuringer can be maintained indefinitely The de-crease in Grunow and Neuringerrsquos operant variabilitymay be due to the weakening of the reinforcement effect(ie reinforcement is less frequent) whereas the in-crease following extinction may be a short-term elicitedeffect This distinction between reinforced and elicitedvariability is important and will be discussed further inthe next section

Operant and Respondent InteractionsElicited or what is sometimes referred to as respondent

effects occur when environmental changes that affectbehavior do not depend on (are not caused by) the behav-ior Pavlovian conditioning being the most commonlystudied example Operant effects are observed when a re-inforcer depends on the response to produce it Althoughfor historical reasons respondent and operant effects areoften studied separately most behaviors result from acombination or interaction of the two Variability is nodifferent and we turn to an example of such an interaction

As reinforcement is approached under FI or fixed-ratio (FR) schedules response rates are commonly ob-served to increase Similarly when reinforcement de-pends on accuracy as in discrimination learning ormatching to sample it too is found to increase within theFI or the FR (Nelson 1978 Nevin 1967) Cherot Jonesand Neuringer (1996) asked whether the same patternheld for reinforced variability Does operant variabilityincrease as reinforcement for varying is approachedThe question may be related to such real-world cases aswhen a composer is rushing to complete a compositionbefore a scheduled performance or when a programmeris trying to solve a problem before an upcoming meetingThese activities involve at least to some extent non-repetitive original or variable thought and behaviorThe question is whether variability increases with prox-imity to or anticipation of reinforcement for varying

Cherot et al (1996) employed trials consisting of fourresponses across L and R operanda and a lag 4 variabil-ity contingency with rats in one experiment and pigeonsin another Unlike in any of the cases described so farhowever the lag-variability contingency had to be satis-fied four times in order for a single reinforcer to be ob-tained With ldquordquo indicating successful completion of theVar contingency ldquo2rdquo indicating failure to meet that con-tingency and PELLET indicating the food reinforcer anexample follows LRLL RRRL LRLL2 LRLL2RLLR RRRL2 LRRR+PELLET Thus food reinforce-ment was provided only after the subject had satisfied

682 NEURINGER

the variability contingency four times FR 4 The resultswere that as the food reinforcer was approached (ie asprogress was made through the ratio) the subjects wereless and less successful at varying The probability ofmeeting the variability contingency in the first trial afterreinforcement was higher than it was in the second whichin turn was higher than that in the third trial and so onFor comparison a Rep group was reinforced for repeatingsequences once again with reinforcement only after foursuccessful repetitions Probability of repeating increasedwith proximity to reinforcement a finding consistentwith the literature but opposite that for the variabilitygroup Thus approach to reinforcement facilitated oper-ant repetition but interfered with operant variabilitySimilar variability-interfering effects have been docu-mented for within-trials responses The probability that aresponse repeats the just prior response increases within atrial even when sequence variability is reinforced (McElroyamp Neuringer 1990 Neuringer 1991) To return to the ex-ample of a composer rushing to complete her composi-tion she may indeed have increasing difficulty respond-ing creatively under the pressure of a deadline On theother hand the soloist may find it easier to repeat thecorrect performance during practice periods as the con-cert nears How can these results be reconciled with themany findings that variability can be reinforced

An additional result of Cherot et al (1996) provides ananswer The Var animals responded much more variablyoverall than did the Rep animals Although variability inthe Var groups decreased with approach to the reinforceroverall variability was always much higher when it was re-inforced (Var group) than when it was not (Rep group)Reinforcement therefore exerts two influences Overallvariability is elevated sometimes to the highest levelswhen it is reinforced but approach to (and possibly focuson) the reinforcers constrains or lowers variability Sim-ilar reinforcement-produced interference effects had beenreported previously for example when raccoons are re-inforced for dropping a ldquocoinrdquo into a slot the reinforce-ment results in the animalsrsquo holding and rubbing the coinrather than dropping it (Breland amp Breland 1961) Theoverall enhancement of variability was of much greatermagnitude in Cherot et al than the decrease with approachwas although both effects were statistically significant(The notes created by the composer under pressure ofdeadline will be less readily predicted than will thoseplayed by the performer practicing a well-known piece)This study highlights the importance of interactions be-tween operant effects of reinforcement and elicited effectsproduced by anticipation of reinforcement and the resultsmay be relevant to the debate concerning possible detri-mental effects of reinforcement on creative behaviors (seethe discussion below and Neuringer in press)

SummaryReinforcement controls behavioral variability as it

does other operant dimensions Levels of variability areprecisely governed by schedule contingencies and dis-

criminative stimuli Choices to vary or repeat match rel-ative frequencies of reinforcement Extinction increasesvariability slightly whereas the hierarchies of sequenceprobabilities are retained and operant and respondenteffects interact The evidence shows that behavioral vari-ability is an operant controlled by its consequences

GENERALITY AND PARAMETERS

Many different methods measures species and re-sponses have been used in the study of operant variabil-ity and this diversity supports the generality of the phe-nomenon As is the case for repeated operants howeverthere are parametric constraints This section briefly re-views the generality and some of the parameters studiedto date

GeneralityMethods Throughout this section ldquoresponserdquo will

signify a response unit including both an individual re-sponse and a multiresponse sequence Each of the meth-ods to be described increases or maintains variability byreinforcing it

Novel response procedures reinforce responses upontheir first occurrence ldquoeverrdquo (Pryor et al 1969) or upontheir first occurrence within a session (Goetz amp Baer1973) Radial-arm maze procedures also reinforce ini-tial (within a given session) entries into arms of a mazeand result in a distribution of choices across the availableoptions the sequence or pattern of choices often beingquite variable

Lag procedures reinforce a response if it has not oc-curred across a given number of previous trials (Page ampNeuringer 1985) For example under lag 5 the currentsequence is reinforced only if it has not occurred duringany of the previous five trials A variant is the recurrence-time procedure which specifies a minimum number of in-tervening trials before a previously emitted response canagain be reinforced Machado (1989) combined recurrencetime with a percentile-reinforcement contingency to gen-erate high levels of variability (see also Machado 1992)Percentile-reinforcement contingencies base the criterionfor reinforcement on the subjectrsquos own performance over aprevious set of responses (see Galbicka 1994) For exam-ple a sequence will be reinforced only if it has not oc-curred over the previous n trials the value of n continuallychanging so that 50 of the trials are reinforced

Least-frequent response procedures (Blough 1966Schoenfeld Harris amp Farmer 1966 Shimp 1967) rein-force responses that are currently least frequent across amoving window

Threshold procedures reinforce responses whose rela-tive frequencies fall below a given value Because thistype of procedure was used in the research describedabove and in many other more recent studies details willbe provided In Denney and Neuringer (1998) which willserve as an example trials consisted of four responses byrats across L and R levers A running tally was kept of the

OPERANT VARIABILITY 683

number of occurrences of each of the 16 possible se-quences LLLL LLLR LLRL and so on If the relativefrequency of the current sequencemdashthe number of its oc-currences divided by the sum of all 16 sequencesmdashwasbelow a specified threshold value in this case 05 the ratwas rewarded Recently emitted sequences contributedmore to the maintained tally than did those in the distantpast because after each reinforcement all 16 counterswere multiplied by a weighting coefficient in this case95 Therefore the influence of a particular trial decreasedexponentially with successive reinforcements Thresholdprocedures differ from Bloughrsquos (1966) least-frequentprocedure in that more than one sequence (or response)can be reinforced at any given timemdashnamely all those se-quences that have relative frequencies below the thresholdvalue The procedure differs from lag in permitting agiven sequence to be reinforced many times in successionif the associated relative frequency is sufficiently low

The frequency dependence procedure (Machado 19921993) is a variant of threshold in which the probability ofreinforcement is a continuous function of relative re-sponse frequency The more frequent a response is thelower the probability is that it will be reinforced

Statistical feedback procedures provide subjects withcomparisons between their performances and that of amodel Neuringer (1986) showed human subjects feedbackfrom 10 statistical tests thereby enabling them to comparetheir own distributions of responses to that of a randommodel Neuringer and Voss (1993) provided feedbackbased on a chaotic model The goal in each of these caseswas for the subjects to learn to approximate the model andthe results of these experiments will be discussed below

Differential reinforcement of switches versus repetitionsbalance a subjectrsquos tendency to engage in each of these pat-terns Rather than reinforcement for infrequent responsesor random-like distributions reinforcement is providedsometimes for repeating a response and other times forswitching from one response to another In the two-operandum case a switch is L followed by R or R by La repetition is L then L or R then R Bryant and Church(1974) found that stochastic-like responding was gener-ated by rats when switches were reinforced with a 75probability and repetitions with a 25 probability that isthe rats had a natural tendency to repeat at least in theenvironment of this study Machado (1997) reinforcedpigeons for eight pecks across two keys if the sequenceof pecks contained at least one (one group of birds) ortwo (another group) switches Although the birds couldhave satisfied these contingencies by repeating a singlesequencemdashin the ldquoone switchrdquo condition for examplerepetition of LRRRRRR would produce reinforcement on100 of the trialsmdashthey tended to vary their sequencesBarba and Hunziker (2002) showed however that vari-ability was higher when it was directly reinforced thanwhen reinforcement was based on proportions of switchesa finding consistent with that of Machado (1997)

The evidence from each of these methods supports thehypothesis that variability can be reinforced

Measures Variability must be interpreted with refer-ence to a particular measure The reason is that there canbe high variability (and low predictability) at one level ofanalysis but order (and certainty of prediction) at a dif-ferent level For example the sequence 1 1 2 2 3 3 44 5 5 1 1 2 2 has the same variance (a commonlyemployed measure of variability) as 2 3 2 1 5 1 4 41 1 3 2 5 2 but an analysis of conditional proba-bilities in which the unit contains pairs of instances in-dicates that the two sequences have different levels ofvariability In general high variability observed with onemeasure does not rule out the possibility of order ob-served with another and vice versa That is why evalua-tions of random-number generators often employ manydifferent measures and indeed there are hundreds of dif-ferent measures of the randomness of a sequence (Knuth1969) All behavioral studies make simplifying assump-tions and rely on one measure or at the most a few

U-value has been the most commonly employed mea-sure of operant variability (Evans amp Graham 1980Machado 1989 Page amp Neuringer 1985 Stokes 1995)U-value measures the distribution of relative frequen-cies or probabilities of a set of responses For a set of16 possible responses i = 1ndash16 U-value is given by

where RFi = the relative frequency (or probability) ofeach of the 16 responses U-values approach 10 whenrelative frequencies approach equality as would be ex-pected over the long run from a random generator andthey approach 00 when a single instance is repeated

Other measures include the percentage of trials or fre-quency at which variability contingencies are met (Page ampNeuringer 1985) percentages of alternations versus stays(Machado 1992) conditional probabilities of responses(Machado 1992) the number or frequency of novel re-sponses (Goetz amp Baer 1973 Schwartz 1982) the num-ber or frequency of different responses (Machado 1997Schwartz 1982) Markov analyses (Machado 1992) andstatistical tests of distributions of responses (Neuringer1986) All of the measures employed to date indicate thatvariability can be reinforced

Responses and species Generality has further beendemonstrated by the use of many different responses andspecies These include Siamese fighting fish (Betta splen-dens) swimming in quadrants of a tank (Roots 2000)young chicks pecking keys (Neuringer 2002) pigeonspecking keys (Page amp Neuringer 1985) budgerigarssinging songs (Manabe Staddon amp Cleaveland 1997)rats pressing levers (Van Hest van Haaren amp van de Poll1989) porpoises swimming and jumping (Pryor et al1969) rhesus monkeys pressing levers (Sydney Reisbickpersonal communication) children drawing and build-ing with blocks (Holman Goetz amp Baer 1977) andhuman adults and children pressing buttons (Saldana ampNeuringer 1998 Stokes amp Balsam 2001) There havebeen few attempts to compare species or responses but

- ( )[ ] ( )=aring RF RFi ii

log log 2 21

1616

684 NEURINGER

in one study lowest baseline variability was observedwhen rats pulled trapezes intermediate levels were ob-served when they pushed keys and highest levels wereobserved when they pressed levers (Morgan amp Neuringer1990) with variability increasing in all cases when it wasreinforced In another study operant variability wascompared across rats and people with similar effects ob-served in the two species (Neuringer Deiss amp Imig 2000)

ParametersWe turn now to some parameters studied to date Many

other parameters as yet unstudied are no doubt also im-portantmdashfor example motivation and types of reinforcers

Age Older people (see eg Van der Linden Beertenamp Pesenti 1998) and older rats (see eg Lamberty ampGower 1990) respond less variably than their youngercounterparts when variable responding is reinforced or re-quested but in most cases it is not clear to what extent thevariability is elicited as opposed to reinforced Neuringerand Huntley (1992) compared reinforced variability in2-month-old (young) and 10-month-old (mature) Long-Evans rats Under Yoke contingencies the young rats re-sponded significantly more variably (two levers four re-sponses per trial) than the mature rats which constitutesan elicited effect When reinforcement was contingenton variability (lag 4) variability increased significantlyin both groups thereby overcoming the initial differencecorrelated with age Thus explicitly reinforcing vari-ability in older animals caused them to respond no dif-ferently than younger animals did a finding potentiallyrelevant to real-world human aging However when Wells(1999) compared 8-month-old (mature) to 24-month-old(old) rats although the mature rats responded more vari-ably than the old rats did under Yoke contingencies thedifference was not statistically significant Additionalstudies are needed especially of whether reinforcingvariable behaviors in older populations can facilitatelearning and performance

Gender Male and female rats are equally sensitive toreinforcement of variability (Neuringer amp Huntley 1992van Hest et al 1989) However there was a tendency(not statistically significant) for males to respond over-all more variably than females did and because thesestudies did not control for differences in size and weightof subjects or for other aspects of motivation additionalstudies are necessary

Stage of training Stokes and Balsam (2001) reporteddifferences in variability as a function of whether variabil-ity was initially reinforced early or later in training (seealso Gottlieb 1992) Human subjects generated sequencesof 10 L and R responses per trial on a computer keyboardunder lag 0 and lag 25 contingencies Under lag 0 everytrial ended with reinforcement under lag 25 the currentsequence had to differ from each of the previous 25 Whenlag 25 was experienced at the very beginning of traininglevels of variability at the end of training (when lag 0 wasoperative) were lower than they were when lag 25 was ex-perienced after 50 trials of initial lag 0 These results were

interpreted to suggest that there is an ldquooptimal period forsetting variability levelsrdquo (Stokes amp Balsam 2001 p 181)but that general conclusion needs further testing and anumber of questions need to be answered One has to dowith the contribution of reinforcement frequency Underlag 0 every sequence leads to reinforcement but rein-forcement is intermittent under lag 25 Another has to dowith whether learning to vary differs from learning to repeatwith respect to optimal periods (see eg Bouton 1994)

Related to stage of training are effects of prior experi-ence Hunziker Lee Ferreira da Silva and Caramori(2002) showed that for human subjects if Var contingen-cies were experienced just prior to Yoke contingencieslevels of response variation during the Yoke phase werehigher than they would have been if the Var contingencieshad not been previously experienced The time delay be-tween Var and Yoke experiences mattered with a long in-terval (3 months) producing only a small effect Saldanaand Neuringer (1998) also found effects of prior Var train-ing on performances in Yoke (increased variability inYoke) but the opposite did not hold that is training underYoke had relatively little effect on Var Hunziker Caramorida Silva and Barba (1998) reported similar effects withrats Thus there may be an asymmetry Reinforcement forvarying has relatively strong influences but reinforce-ment without regard to variations does not adversely affectlearning to vary These different effects are not surprisinggiven that Yoke contingencies are permissivemdashreinforce-ment is given whether or not responses varymdashwhereascontingencies under Var can be quite demanding How-ever the results motivate a number of additional ques-tions One is whether effects of Var training differ fromthose observed when repetitions are reinforced Another iswhether direct reinforcement of variable behaviors canhelp animals or people to overcome the detrimental effectsof prior noncontingent experiences such as those that pro-duce learned helplessness (Seligman 1975)

Environmental constraints Stokes (1995) arguedalso that variability increases when behavior is con-strained The term ldquoconstraintrdquo is ambiguous because allreinforcement contingencies exert a form of constraintIn one study Stokes (1995) compared different strate-gies of shaping leverpresses Rats in one group were re-inforced initially for pressing the lever with only theright paw (right only) and eventually for any press (withright or left paw or with any other part of the body) Theorder of training was reversed for a second group ldquoanyrdquofirst and then right only Behavior sequences were morevariable in the right-only-first group leading Stokes(1995) to conclude that early constraints lead to highvariability (see also Stokes Mechner amp Balsam 1999)That is an important hypothesis with many implicationsbut controls are needed before the results can be attrib-uted to the contingencies For example ldquoright onlyrdquo andldquoanyrdquo differed in terms of reinforcement distributions aswell as in the precision of the reinforcement operationthe ldquoanyrdquo reinforcers were provided immediately and au-tomatically by the programming equipment whereas in

OPERANT VARIABILITY 685

the right-only condition a human observer had to decidewhether a response had occurred and then present the re-inforcer necessarily leading to longer and more variedlatencies and a less consistent contingency Furthermorethere are many cases in which increasing constraints suchas those due to a decrease in the number of responses pertrial (Page amp Neuringer 1985) or the requirement of ex-actly four responses on each of two keys (Schwartz 1980)results in decreased variability Thus the contributionsof response-definition constraints and environmentallimitations must be explored further

SummaryThe generality of operant variability is shown by the

robustness of the phenomenon across procedures mea-sures responses and species As with other operant phe-nomena parameters influence the effects

FUNCTIONS

All operants including variability are functional Op-erant responses produce reinforcing consequences Be-cause other behavioral competencies depend on it oper-ant variability is functional in a broader sense as wellSome of these functions will now be described

Operant ConditioningThe potential of a reinforcer to change behavior is lim-

ited by the range of available variations (Hull Langmanamp Glenn 2001 Skinner 1981) and thus whatever pro-duces those variations contributes to the conditioning ofnew operants Much work has been done to identifysources of variation (Balsam Deich Ohyama amp Stokes1998 Segal 1972 Staddon amp Simmelhag 1971) butNeuringer (1993) was first to show that the same rein-forcers that strengthen individual responses can also pro-duce the necessary variability and that these two functionsof reinforcement may work simultaneously Implied isthat when an operant response is being conditioned re-inforcement works not only to strengthen an individualresponse but at the same time to engender the variabilityfrom which the response emerges (That is variability isnot solely a baseline phenomenon elicited or noise)Concurrent reinforcement of variations and individualtarget sequences will be the topic of this section

Pigeons in one experiment and rats in another were re-warded for varying among 16 possible response sequences(four responses per trial L and R operanda lag contin-gencies Neuringer 1993) Concurrently one of the 16designated as the target sequence was reinforced when-ever it occurred Thus reinforcements were given bothfor sequence variations and for a single selected targeta reinforced-variation-and-selection (RVS) procedureWhen the target was ldquoeasyrdquo (eg LLLL) it emergedrapidly from the baseline variations Given that the tar-get was always reinforced reinforcers were obtained morefrequently when the subjects did not vary and only re-peated the target When the target was ldquodifficultrdquo (eg

LLLR) it was not learned As in the choice experimentdescribed above (Neuringer 1992) the relative valuesor difficulties of target versus variations governed theprobability that the target would be learned

However even difficult targets could be conditionedunder the RVS procedure as was shown by another ex-periment in the same series (Neuringer 1993) Sequencevariations were again reinforced concurrently with a dif-ficult target RRLR but now the frequency of reinforce-ments for varying was systematically decreased Satis-fying the variability contingency was intermittentlyreinforced according to a VI schedule and over sessionsreinforcement for varying became less and less frequentThis technique resulted in acquisition of even the mostdifficult target sequence whereas a control group rein-forced for the target alone (without the concurrent rein-forcement for variations) never learned

Reinforcers were obtained more frequently in the RVScondition than in the control condition (since the RVSsubjects were rewarded both for varying and for emittingthe target sequence) and the additional reinforcers mighthave generated greater motivation to respond (Killeen1981) Thus motivation not the direct reinforcement ofvariability might have been responsible for the RVS ef-fect To test this possibility Neuringer Deiss and Olson(2000) reinforced three groups of rats for emitting a par-ticularly difficult target sequence LLRLR The experi-mental group was reinforced concurrently for sequencevariability on the average of once per minute (VI 1 min)The target-only control group received no reinforce-ments other than for the LLRLR target A third groupwas yoked to the experimental animalsmdashreinforced onceper minute for any sequence variability not requiredmdashand as for the other two groups whenever the target oc-curred If the additional reinforcement was responsiblefor the facilitated learning under RVS then because theexperimental and yoked groups received equal reinforce-ments they should have learned equally well but thatwas not the result Only the experimental group learnedto emit the LLRLR target The yoked controls respondedat high rates throughout the experiment but they did notvary sufficiently to make contact with the target contin-gency and never learned The target-only control groupreceived insufficient reinforcement and their responsesextinguished Therefore the added reinforcers helped tomaintain motivation as suggested above but reinforce-ment of variations was necessary for learning of a diffi-cult sequence The RVS procedure maintains both highmotivation and high variation and the two together re-sult in the acquisition of difficult-to-learn responses

Grunow and Neuringer (2002 Experiment 2) studiedone possibly important parameter of the RVS proceduremdashnamely the level of variability When a difficult target re-sponse is being trained does it matter whether reinforcedvariability is low intermediate or high As describedabove each of four groups of rats was reinforced for vari-able response sequences the first group for high vari-ability the fourth group for low variability and the other

686 NEURINGER

two for intermediate levels In other ways the procedurewas similar to that of the Neuringer Deiss and Olson(2000) RVS contingencies just described with a singledifficult target sequence always reinforced The resultswere that the higher the level of reinforced baseline vari-ability the more quickly the target was learned Thusvariability levels indeed mattered and if the just-re-ported effect is found to be general it will be most help-ful to reinforce high variability when difficult-to-learnnew responses are being shaped Other parameters re-quiring study are class definitionsmdashwhich responses areincluded in the class of acceptable variationsmdashand num-ber of instances within the class

One additional effect should be noted When varia-tions among a set of responses are reinforced if one vari-ant is selected for extinction (no longer reinforced) thefrequency of that instance decreases Neuringer (1993)showed this for pigeons responding under a lag-variabilityschedule When one of the 16 sequences was no longerreinforced (with no other change to indicate its status) thefrequency of that sequence decreased Thus reinforce-ment of variations provides the baseline to select for (iestrengthen) and select against (ie weaken) individualinstances

These findings have potentially important implicationsFirst as has been indicated the same reinforcers thatstrengthen particular instances can also reinforce varia-tions That a binary eventmdashreinforce or not reinforcemdashsimultaneously affects variations and selected instancesindicates both the power of reinforcement and a concep-tual problem How reinforcers exert such diverse effectsis a question to which we will return

Second shaping procedures may implicitly reinforcebaseline variations An approximation to a goal responseis reinforced for a period of time but eventually rein-forcement is withheld until a response closer to the goalis obtained It is commonly thought that withholding re-inforcement or mini-extinction induces the necessaryvariations for shaping to proceed as has been indicatedabove An alternative suggested by the present studiesis that organisms learn both to vary (within a set) and to re-peat (selected instances) until one instance is always rein-forced That is the process of shaping might involve re-inforcement both of variations (generally done implicitly)and of targeted instances If so a useful contributor tothe shaping process might be explicit reinforcement ofvariations

Third reinforcement of variations could facilitate ac-quisition of other competencies such as cognitive andmotor skills Siegler (1996) has shown in an important se-ries of studies that childrenrsquos acquisition of complex cog-nitive skills is correlated with high variability Anotherexample is that when motor skills are practiced in a vari-able manner high quality of performance results (Ma-noel amp Connolly 1995 1997 Mechner 1992 Schmidtamp Lee 1999) However it is rare that variability in thesedomains is directly reinforced as part of the training oreducational process

Fourth the RVS procedure might be used in therapeu-tic settings to modify maladaptive and self-injurioushabits Therapists sometimes employ noncontingent re-inforcement procedures to weaken undesirable behaviors(Vollmer Iwata Zarcone Smith amp Mazaleski 1993)The reasoning is that many maladaptive behaviors aremaintained by unintended consequences and that non-contingent rewards may compete with those conse-quences thereby lowering the strength of the unwantedhabit For example when a staff personrsquos attention toself-injurious behavior helps to maintain it providing at-tention noncontingently may decrease the strength of theunwanted target behavior However the RVS proceduremight accomplish the same goal more effectively andwithout undesirable side effects (Vollmer RingdahlRoane amp Marcus 1997) Noncontingent rewards some-times result in unwanted ldquosuperstitiousrdquo behaviors andthat would be avoided under RVS Furthermore RVSwould help to provide the set of baseline behaviors nec-essary to establish desirable new responses

Finally RVS procedures may help to gain control overand eventually to weaken other undesirable target behav-iors The target can be incorporated within a set of moredesirable responsesmdasha new operant class established withthe target as one membermdashand variable instances rein-forced Once variable responding is successfully main-tained including the undesirable target response the lattercan be weakened by selectively withholding reinforcement

Summary The conditioning of operant responses de-pends on both baseline variations and selection of suc-cessive approximations to a goal response The same re-inforcers used to select the approximations can reinforcebaseline variations thereby defining the precise set ofappropriate variants and the desired levels of variationSuch concurrent reinforcement of variations and selec-tions may facilitate the acquisition of difficult-to-learnoperants and selective withholding of reinforcementmay weaken undesirable instances

Creativity and Problem SolvingBehaving in a novel unusual or nonrepeating manner is

part of creative activities (Campbell 1960) Novelty andvariation may arise through combinations of previouslylearned behaviors (Epstein 1996) transformations of thesebehaviors (Boden 1994) direct reinforcement of variabil-ity as described above or some combination thereof Ofcourse neither novelty nor variability alone suffices forcreative outputmdashthe throw of the dice or the toss of acoin are not creative nor are haphazard strokes of a paint-brush on a canvasmdashbut without variation creativity islow Whether the variability required for creativity canbe reinforced is relevant to whether creativity itself canbe reinforced The latter question has led to contentiousdebate with some arguing that reinforcement facilitatescreativity (Cameron amp Pierce 1994 Eisenberger ampCameron 1996 Stokes 2001 Winston amp Baker 1985)and others focusing on detrimental effects of reinforce-ment (Amabile 1983 Deci Koestner amp Ryan 1999)

OPERANT VARIABILITY 687

The debate goes well beyond the present discussion butsome of the research described above is relevant Firstreinforcement contingent on variability generates higherlevels of behavioral variability than does reinforcementcontingent simply on responding as has been shown bythe many studies described above in which Var contin-gencies were compared to Yoke contingencies Similarlyprecise reinforcement contingencies matter for creativity(Cameron amp Pierce 1994 Lepper amp Henderlong 2000)If any product rather than only creative ones is rein-forced creativity tends to fall (just as variability decreasesunder yoke conditions) Second approach to or antici-pation of reinforcement interferes with operant variabil-ity (Cherot et al 1996 McElroy amp Neuringer 1990Neuringer 1991) which again might be related to effectson creativity when an individual focuses on the rein-forcer Note however that although variability decreaseswith proximity to the reinforcer overall levels of vari-ability are much higher when variability is reinforced thanwhen it is not The same may be true for creativity aswell anticipation of reward may interfere with creativeoutput to some extent but at the same time high creativ-ity may depend on its reinforcement

Operant variability may be important for problemsolving as well Here too there is disagreement as towhether training is helpful or not Early studies by Maier(1933) indicated that it is and these found support inlater work (eg Lung amp Dominowski 1985) Somestudies showed that those individuals most likely toldquobreak the setrdquo are also most likely to solve a problem(Dover amp Shore 1991) However a number of studiesfound that practice had no influence (Duncan 1961)

Arnesen (2000) explored a related phenomenon witha rat model One group Var was reinforced for variableinteractions with novel objects and another group Yokewas reinforced independently of such interactions EachVar rat received training with 12 different objects oneper session For example in the presence of a soup canthe rat was reinforced the first time it entered the canbut thereafter a different response was required such asrolling it climbing on it and so forth This procedure issimilar to that of Goetz and Baerrsquos (1973) described aboveexcept that objects changed from session to session Ratsin the Yoke control group had identical experiences in anadjoining identical chamber that contained identical ob-jects but now rewards were not contingent on object ma-nipulation After this training the rats were tested in adifferent room with 30 never-before-experienced ob-jects scattered on the floor Hidden in each was a smallpiece of food When compared to their Yoke controls theVar animals were more active during this test exploredmore manipulated objects more and most importantlydiscovered significantly more food pellets Another con-trol group received no preliminary training with objectsand their performance in the test session did not differfrom that of the yoked controls Therefore when ratswere reinforced for variable interactions with objectsthe result was an increased tendency to explore manip-ulate and discover new sources of reinforcement It

would be of considerable interest to determine whethera similar effect occurs with children

Assessment and Treatment of PsychopathologiesAbnormal levels of behavioral variability characterize

some human psychopathologies as has been shown byclinical observations as well as by research with humansubjects and animal models Few studies have attemptedto reinforce variability directly in such problem popula-tions The main question is whether reinforcement canmodify abnormal levels of variability in the direction ofnormalcy and whether such changes might have associ-ated beneficial effects

Autism Stereotyped responding characterizes indi-viduals with autism (Hertzig amp Shapiro 1990) For ex-ample Baron-Cohen (1992) asked children with autismand others in a control group to hide a penny in one handso that the experimenter could not guess which hand heldthe penny Those with autism were more likely thanthose without to generate a simple predictable patternsuch as repeatedly switching back and forth from left toright hands Such repetitive responding can be maladap-tive For example Mullins and Rincover (1985) askedchildren with and without autism to pick one of five cardsFood was occasionally found in a cup behind the cardEach of the five cards was a discriminative stimulus fora different schedule of reinforcement including contin-uous reinforcement (CRF every choice reinforced) FR 2(fixed ratio 2 in which every second choice was rein-forced) FR 4 (every fourth choice reinforced) FR 7 andFR 11 Control participants sampled all five alternativesand quickly learned to choose the most frequently rein-forced CRF card Autistic children sampled only a lim-ited number of cards and therefore often preferred anonoptimal alternative Thus children with autism tendto respond less variably than normal children and thiscan result in failure to adapt

Miller and Neuringer (2000) asked whether direct re-inforcement could influence the variability of individu-als with autism Sequences of responses on two largebuttons were first reinforced probabilistically and inde-pendently of variability (Prob phase) after which rein-forcement was contingent upon satisfying a thresholdvariability contingency (Var phase) The autistic partic-ipantsrsquo variability was significantly lower than that ofmatched controls during both Prob and Var phases butwhen variability was reinforced its levels increased sig-nificantly in both groups This is an important findingbecause it suggests that direct reinforcement of variabil-ity may help to modify some of the behavioral stereo-types that are characteristic of autism Lee McComasand Jawor (in press) offer support for this hypothesisThree individuals with autism were reinforced for variedverbal responses to social questions the responses had tobe appropriate given the context Variability increased intwo of the three individuals In a related procedureBaker and Koegel (1999) reinforced children with autismfor varied social interaction in a game-playing contextAlthough the reinforcement was permission to engage in

688 NEURINGER

the stereotyped behaviors preferred by the child the re-sult was increased social interactions both during theperiod of the experimental manipulation and in 1- and2-month follow-ups These three studies taken togethersuggest that reinforcement of varied behaviors may facil-itate modification of nonfunctional ritualistic and stereo-typed behaviors

Depression Depression provides another example ofmaladaptive consequences of low variability Lapp Mar-inier and Pihl (1982) observed that depressed womenproduced fewer alternative solutions to hypotheticalproblems than did nondepressed women Channon andBaker (1996) found that attempts by moderately de-pressed college students to identify faults in a series ofinterconnected circuits were less varied and less success-ful that were those of nondepressed controls HorneEvans and Orne (1982) found that when depressed pa-tients were asked to generate a random sequence of num-bers their responses were less variable than were thoseof controls These experimental results are consistentwith clinical observations

Can depressed individuals be rewarded for increasingvariation Hopkinson and Neuringer (in press) dividedcollege students into mildly depressed and nondepressedgroups on the basis of the CES-D scale a paper-and-pencil self-evaluative index of depression In the first oftwo phases sequences of responses on a computer key-board were reinforced independently of variability (Prob)following which only highly variable sequences were rein-forced (Var) Under Prob depressed students respondedsignificantly less variably than did the controls in agree-ment with previous findings Under Var variability in-creased significantly in both groups Indeed by the endof the experiment depressed and nondepressed subjectswere responding equally variably (Instructions were alsoeffective in increasing variability in the depressed par-ticipants) Thus even in a simple computer-game envi-ronment mildly depressed subjects manifested lowervariability than controls did but direct reinforcementmodified their behavior to control levels Can reinforce-ment of variability be used in real-world conditions tohelp depressed individuals An affirmative answer comesfrom cognitive behavior modification procedures thatwork to increase alternative solutions to problems (Beck1976) A more general conclusion from these studies ofautistic and mildly depressed individuals is that direct-reinforcement-of-variability procedures may benefitthose who manifest abnormally low levels of variability

Attention Deficit Hyperactivity Disorder (ADHD)ADHD may provide an example of the opposite stateabnormally high levels of variability (Barkley 1990)Wultz Sagvolden Moser and Moser (1990) proposedthat the spontaneously hypertensive rat (SHR) may serveas a good animal model of ADHD the SHR shows manyof the behavioral characteristics of that disorder includ-ing hyperactivity poor learning ability poor ability totolerate delayed reinforcement and poor self-control

The next study to be described asked about operant vari-ability in SHRs

Mook Jeffrey and Neuringer (1993) compared vari-ability of SHRs and of a control strain Wystar Kyoto(WKY) under two conditions Prob and Var The mainresult was that SHRs tended to respond variably whetheror not the contingencies required it consistently with theresults from the human ADHD literature When vari-ability was reinforced the two strains responded at ap-proximately equally high levels of variation These re-sults were confirmed in two other studies (HunzikerSaldana amp Neuringer 1996 Mook amp Neuringer 1994)Thus control rats varied only when the contingencies re-quired variations whereas SHRs varied whether or notthe contingencies required it As might be expected fromthese results when reinforcement depended on repeti-tions the SHRs performed poorly (Mook et al 1993)

Another similarity between SHRs and ADHD humanswas seen when amphetamine a drug that has effects simi-lar to those of Ritalin often prescribed for ADHD was ad-ministered (Mook amp Neuringer 1994) When repetitionswere required for reinforcement amphetamine improvedperformances by the SHRs so that they were as accurate assaline-injected WKYs a finding again consistent withthose in the ADHD literature These studies support SHRsas a model of ADHD and suggest that additional attemptsshould be made to assess whether or not individuals withADHD are less able to change levels of variability thanare controls Although as indicated above there is someevidence in support of this hypothesis (Barkley 1990)when Saldana and Neuringer (1998) directly comparedhuman ADHD and control participantsrsquo operant variabil-ity within the context of a computer game no differenceswere observed Most of those with ADHD were main-tained on Ritalin and other drugs however a fact unfor-tunately not documented in the original article

Drug EffectsAlthough many drugs are known to influence behavioral

variability (Brugger 1997) rarely have attempts beenmade to analyze explain or identify the sources of theeffects As will be discussed below operant variabilitycan result from the memory-based process of remember-ing previous responses and then behaving differently orcan result from responding stochastically a process thatappears to be quite different McElroy and Neuringer(1990) attempted to distinguish the effects of alcohol onmemory from its effects on stochastic processes in astudy in which one group of rats was reinforced for vary-ing four-response sequences under a lag 5 contingency(Var) and another was reinforced for repeating a singlesequence LLRR (Rep) Ethanol injections of 075 and20 gkg affected Var and Rep performances differentlyWhereas the Rep animals were much less able to satisfythe LLRR contingency at both doses the Var rats re-sponded variably whether or not alcohol had been ad-ministered Similar results were obtained by Cohen et al

OPERANT VARIABILITY 689

(1990) with a multiple schedule in which each rat servedas its own control In the same animals and within thesame sessions alcohol degraded Rep performances butleft Var intact Thus alcohol appears to interfere with atask that requires working memory as presumably is thecase for the Rep task but not with one that reinforcesvariability The memory-interfering effect of alcoholmight account for why alcohol also degrades radial-armmaze performance (Devenport 1983)

Alcohol also affects choices to vary or to repeat For ex-ample Gorka (1993) reinforced rats concurrently for re-peating and varying and monitored percentage of choicesas a function of time following injection of ethanol Inthe control sessions when the rats were injected withsaline the probabilities of varying and repeating remainedapproximately constant across the 45-min session witha slight bias in favor of varying After injection with 15gkg of ethanol however there was a significant increasein choices to vary at the beginning of the session andthen a later decrease to below 50 That is alcohol ledinitially to a markedly increased preference for varyingbut later to a preference for repeating (Figure 5) This re-search should be replicated and extended especially be-cause the contingencies ldquopressuredrdquo the animals to re-main at approximately 50 preference but thesuggestion is that alcoholrsquos effects on response variabil-ity may depend in part on the duration of its influence Amore general implication is that drugs may affect re-peated and variable operants differently and that effectsdepend on time following administration

SummaryReinforced variability may contribute to many types

of learning including operant shaping and acquisition ofcognitive and complex motor skills Similarly creativityand problem solving depend on variable responding andin these areas as well explicit reinforcement of variationsmay be functional Abnormal levels of variability aresometimes caused by psychopathologies and drugs andoperant variability may be useful in these areas in twoways first as a way of assessing the variability that is pos-sible and second to modify variability toward more nor-mal levels Studies show that individuals with autism andmild depression vary less than do controls but that ex-plicit reinforcement of variability moves the levels towardsnormalcy Other studies show that drugs such as alcoholmay affect repeated and variable operants differently

EXPLANATIONS

Two questions must be answered in order to explainoperant variability First what are the sources of vari-ability that permit its reinforcement Second how is itmaintained Related questions including how instancesbecome members of a set from which variations emergeand whether operant variability is a generalizable com-petency will also be considered

Origination From Nonoperant SourcesSkinner (1966) reasoned that operant responses orig-

inate from nonoperant sources with reinforcement con-tacting a behavior shaping and strengthening it onlyafter the behavior had occurred for reasons other than re-inforcement To put it simply a behavior must occur be-fore it can be reinforced (Balsam amp Silver 1994) I willconsider three initial sources of variable behavior en-dogenous variability noncontingent environmental eventsand extinction These appear to parallel evolutionaryprocesses and they too will be identified

Endogenous variability All physical phenomenavary and behavior is no exception This fact can providethe opportunity for reinforcement to select particularlevels of variation Examples of spontaneous variabilityinclude spontaneous alternations in mazes (Dember ampRichman 1989) nondifferentially reinforced variationsin operant responses (Guthrie amp Horton 1946) and ex-ploration (Archer amp Birke 1983) Endogenous muta-tions provide a parallel in genetics Base-pair changes inDNA molecules occur spontaneously intron-bearinggenes mix and match segments to encode new proteinsjumping genes introduce new DNA sequences and atthe level of the chromosome random crossings and as-sortment occur as a natural part of reproductive meiosisThese sources of random genetic change occur indepen-dently of external influences

Figure 5 The relative frequency of variations by rats as a func-tion of 5-min blocks of time since the beginning of the sessionThe open circles represent performance after the rats were in-jected with saline indicating a slight tendency to vary more thanrepeat but one that is approximately constant throughout thesession The filled circles represent performance after injectionwith 15 gkg ethanol indicating a much higher likelihood ofvarying soon after the alcohol is injected followed by a decreasein variations to below 50 and then a return to baseline at theend of the session Group averages are shown Data are fromGorka (1993)

690 NEURINGER

Noncontingent events Noncontingent events some-times referred to as chance accidents or luck also affectbehavior One happens upon a particular passage in a bookthat leads to a new thought one happens to sit next to aninteresting woman or man on an airplane which leads tomarriage (Bandura 1982) In science as well as in every-day life accidents or serendipity are an important part ofthe process of discovery (Beveridge 1957) In each of thecases presented above an accidental event leads to new be-haviors that are then strengthened by consequences Thesame is potentially true for variability itself Particular lev-els of variability might occur for adventitious reasons butthen become functionally related to consequences

That organisms are particularly sensitive to such unan-ticipated noncontingent events is shown by two basicprinciples of learning habituation and Pavlovian condi-tioning Organisms attend preferentially to unexpectedevents and stop attending or habituate to repeated onesThe conditioning of new Pavlovian responses dependsaccording to influential theories upon unexpected rein-forcing events (eg Rescorla amp Wagner 1972)

Noncontingent events such as X-rays and other pro-cesses that can act as mutagens play an important role ingenetic variation as well The particular genes affectedand the type of changes are random there is no neces-sary relationship or contingency between the externalevent and the affected gene However sensitivity to mu-tations is not random but has been selected presumablybecause it is functional Thus in both behavior and ge-netics sensitivity to noncontingent events is conserved

Extinction Extinction increases behavioral variabilityat least to some extent as was described above Distance inspace or time from the reinforcer does the same (Cherotet al 1996) A possible parallel at the genetic level in-volves high levels of deprivation at least in simple lifeforms For example when E coli bacteria are deprivedof nutrients unusually high mutation rates are observedThe exact mechanism by which such hypermutations areinduced is not known but they increase the likelihoodthat a population can survive such environmental insults(Moxon 1997) That is deprivation-induced variability isfunctional at the genetic level (The behavioral evidenceregarding long-term deprivation is less clear some stud-ies showing increased variability and others increasedstereotypy eg Carlton 1962 Senkowski Vogel amp Po-zulp 1978)

Endogenous variations external noncontingent eventsand withholding of reinforcement are three sources forthe variability that precedes control by reinforcementBut once control by reinforcement is established a dif-ferent question emerges What behavioral processes areinvolved when variability is maintained as an operantThe sources just discussed generate uncontrolled non-specific and often short-lived variability and cannot ex-plain the precisely controlled operant type

Operant Variability ProcessesThree processes are hypothesized to underlie operant

variability The first involves the use of random environ-

mental events such as the toss of a coin The second in-volves memory for responses such as when an animallearns not to return to previously visited locations Thethird will be referred to as an endogenous stochasticgenerator

Random environmental events Throughout historymany cultures have used random events to influencechoices and decisions such as throws of dice and randomselections of sticks cards bones or organs Today coinflips are used by a referee at the beginning of a game to de-cide which team kicks the ball a computerrsquos random-number generator is employed by scientists to avoid bi-ases in assigning subjects to experimental groups andaleatoric events are sometimes employed in modern artmusic and literature The Dice Man by Rhinehart (1998)provides a fictional example in which the protagonistbored with life writes a number of possible actions onslips of paper and then periodically selects one blindlyand behaves accordingly These examples show that randomexternal events may be used to avoid biases engender un-likely responses and break out of behavioral niches

Memory-based variability Many methods used tostudy operant variability differentially reinforce nonrepe-titions at least over the recent past Therefore one strat-egy that subjects can employ is to cycle through few re-sponses or sequences especially when the variabilitycontingencies are relatively permissive For exampleunder a lag 2 contingency cycling repeatedly throughthree sequences will result in 100 reinforcement Ani-mals and people sometimes do this apparently basingtheir current response on memory for or discriminativecontrol by the just emitted responses One such instancewas shown by Schoenfeld et al (1966) who rewardedrats for IRTs that fell into one of 10 interval classes onlyif the current IRT differed from the interval class of theprevious response The rats came to alternate betweenlong and short IRTs thus behaving systematically to meetthis lenient variability contingency Machado (1993)studied an analogous effect having pigeons peck L and Rkeys under a frequency-dependent variability contin-gency related to Shimprsquos (1967) procedure describedabove When the trial unit consisted of two responses thebirds developed memory-based repetitive sequencessuch as RRLLRRLL and thereby satisfied the variabilitycontingency When the trial unit was three responses inlength the birds did not develop the optimal fixed pat-tern of RRRLRLLLmdashwhich was too complexmdashbut in-stead developed ldquorandom-like behaviorrdquo (Machado1993 p 103) Thus a memory-based strategy was em-ployed when it was possible but when the memory loadbecame too high stochastic responding emerged A sim-ilar pattern was seen when variability of budgerigarsongs was reinforced (Manabe et al 1997) Under lag 1contingencies the birds sang mainly two songs underlag 2 they increased to three songs Only when the lagwas increased to 3 did song diversity increase apprecia-bly in one of the birds In radial-arm mazes as well whencontextual cues are available memory again appears toplay an important role leading to much lower probabil-

OPERANT VARIABILITY 691

ities of repetition than would be expected from a sto-chastic generator (McElroy amp Neuringer 1990)

Note that under lag schedules memory-based strate-gies result in higher reinforcement rates than respondingstochastically does The reason is that a stochastic gen-erator repeats by chance That animals and people oftendo not employ memory-based strategies probably indi-cates the difficulty of remembering long sequences andpossibly other advantages of unpredictable responding

If a memory strategy is the only way to satisfy vari-ability contingencies then it can be used even when thememory requirements are demanding as was shown byNeuringer and Voss (1993) Human subjects were trainedto generate on a computer keyboard a chaotic-like se-quence of responses that matched iterations of the logistic-difference equation

where Rn refers to the nth iteration in a series with R be-tween 00 and 10 and t is a constant between 10 and40 When the t parameter approaches 40 iterations ofthis equation result in very ldquonoisyrdquo sequences As withother chaotic processes however there is an orderlinessunderlying the surface noise In this case if responsesare autocorrelated with Rn drawn as a function of Rn21the resulting function is a parabola In the Neuringer andVoss (1993) study following each response college stu-dents were shown how close their predicted values wereto that from iterations of the logistic-difference model Inone experiment the students made digital predictions inanother they placed a pointer along a horizontal lineand this analogue response was then translated into dig-ital values for computation With training the studentsbecame increasingly adept at responding in chaotic-likefashionmdashthe studentsrsquo predictions matched closely theiterations of the logistic functionmdashand their autocorrela-tions became parabolic The left column in Figure 6 showsperformances by 4 subjects (rows) early in training thatis not well approximated by the logistic-differenceparabola and the right column shows performances at theend of training in which r 2 values were 72 99 98 andapproaching 10 How was this accomplished Sinceeach iteration in the logistic-difference sequence is basedon the prior outputmdashthat is the sequence is generated bya deterministic although nonlinear processmdashit was hy-pothesized that the human subjectsrsquo chaotic-like responseswere also based on memory for prior responses To put itsimply the subjects may have memorized a long series ofpairs in the form ldquoIf the previous response was value Athen the current response must be value Brdquo as Metzger(1994) Ward and West (1994) and Ward (2002) have sug-gested

To test whether subjects in fact relied on memory forprior responses Neuringer and Voss (2002) systemati-cally slowed responding by interposing pausesmdashthat isby requiring an IRT As IRT increased the difference be-tween the subjectsrsquo sequences and the modelrsquos chaotic

Rn n nt R R= -( )- -1 11

Figure 6 Autocorrelations (responses on trial n on the y-axis asa function of n21 on the x-axis) for 4 human subjects at the be-ginning (left column) and end (right column) of training inNeuringer and Voss (1993) Equations for the best-fitting curveand amount of variance accounted for (r2) are shown at the topof each panel To the extent that the circles (responses) approxi-mate the equation y = 40x 2 40x 2 the subjects were respondingin a way that matched the logistic-difference chaotic functionFrom ldquoApproximating Chaotic Behaviorrdquo by A Neuringer andC Voss 1993 Psychological Science 4 p 115 Copyright 1993 bythe American Psychological Society Reprinted with permission

692 NEURINGER

output increased and the underlying parabolic structurewas broken That is when pauses increased subjects wereless and less able to approximate the modelrsquos chaotic dis-tribution (see Figure 7) The important point here is thathighly variable responses which in this case are chaoticin nature can be based on memorial processes

Endogenous stochastic generator The evidence tobe reviewed next indicates that animals and people canrespond stochastically and that this competency is basedneither on chance environmental events nor on memoryfor prior events or responses It is impossible to provethat an endogenous stochastic process underlies operantvariability that would be analogous to proving the nullhypothesis However much evidence is consistent withthe stochastic hypothesis Included are the facts thatmemory limitations do not permit adequate respondingunder many variability contingencies that response dis-tributions match those expected from a stochastic gener-ator that changes in parameter values affect operantvariability as they would a stochastic generator and thatinterference with memory processes leaves stochasti-cally based operant variability unaffected As a wholethe evidence is consistent with an endogenous stochas-tic source of operant variability

Memory limitations One type of evidence is nega-tive in character and is based on the presumed inability ofmemory for prior responses or sequences to satisfy sometypes of variability contingencies Two examples weregiven above (Machado 1993 Manabe et al 1997) andanother was provided by a lag 50 contingency in Pageand Neuringer (1985) In the latter case the sequence in

the current trial (eight responses across L and R keys) wasrequired to differ from those in each of the previous 50 tri-als including at the beginning of a session the last 50trials from the previous session Pigeons responded wellunder these contingencies and in a way similar to that ofa stochastic model It is unlikely that pigeons can remem-ber each of their previous 50 eight-response sequences

Response slowing As was shown above for chaotic-like responding performances are often degraded by de-lays imposed between a stimulus and a response pre-sumably because of limitations in working or short-termmemory (Berryman Cumming amp Nevin 1963 Williams1971) Neuringer (1991) relied on this effect to test whethermemorial processes could explain the generation of vari-able sequences under lag schedules Two groups of ratswere reinforced for L and R leverpresses producing fourresponses per trial One group (Var) was trained under alag 5 contingency and the other (Rep) was trained to asingle sequence of responses LLRR It was assumedthat accurate Rep performance depended on workingmemory After performances had stabilized IRTs weresystematically varied by the imposition of timeouts of05 to 20 sec between consecutive responses in the se-quence During these forced pauses the chamber wasdark and responses were not counted toward completionof the sequence If the animals had used the previous re-sponse(s) as a discriminative cue for the current responsethen performances should have been degraded by theseinterpolated pauses

The results were that for the Rep group as the re-quired pauses increased from 0 to 6 sec percentages ofcorrect sequences were relatively unaffected but at pausesgreater than 6 sec probability of a correct LLRR se-quence fell sharply This pattern of results suggests in-volvement of working memory of prior responses withthe rats able to remember responses within a 6-sec win-dow The Var grouprsquos results were quite different Aspauses increased between 0 and 6 sec the rats were moreand more likely to satisfy the variability contingenciesand with pauses greater than 6 sec percent correct re-mained at a high and constant level Thus the Var grouprsquosresults were opposite to those of the Rep group and op-posite to those predicted from a memory-based strategySimilar results showing that operant variability increasesor is maintained as responding is slowed have been re-ported by Morris (1987) with pigeons and by Baddeley(1966) with random response generation by human sub-jects The results reported above concerning alcoholrsquos ef-fects on variability and repetition were also consistentwith those of Neuringer (1991)

Memory for prior responses may not have controlled theVar grouprsquos performance but why did variability increaseEach of three answers is consistent with a stochastic-generator interpretation Weiss (1964 1965) hypothe-sized that random responding requires that current re-sponses be independent of previous ones memory for orcontrol by prior responses interferes with such indepen-

Figure 7 Approximation to a chaotic sequence (given by delta)as a function of the imposed interresponse times Shown are theaverages of 5 subjects

OPERANT VARIABILITY 693

dence and therefore pauses would be expected to facil-itate variability A second possibility is that at shortIRTs animals tend to repeat responses on the sameoperandum Blough (1966) found this in pigeons and ex-cluded such double pecks from his analyses because theyappeared not to be under the control of reinforcement con-tingencies Morris (1987) also found a tendency for birdsto repeat when no interresponse timeouts were imposedA third hypothesis is that there were two contributors tothe observed variability One was a stochastic processcontrolled by reinforcement and the other elicited vari-ability generated by interposition of pauses The high vari-ability elicited as an effect of slowed responding is ageneral phenomenon supported in many other cases Ac-cording to this interpretation operant variability in the Vargroup was governed by a stochastic-based process operantrepetition in the Rep group was governed by a memory-based process and pauses elicited variability under bothcontingencies Thus Rep performance was interferedwith whereas Var was facilitated Each of these hypothe-ses is consistent with the conclusion that memory for (ordiscriminative control by) prior responses does not con-tribute to and possibly interferes with variable respond-ing when an organism is reinforced for variability andthis in turn is consistent with the stochastic-generatorhypothesis

Stochastic response distributions Other evidence iscorrelational Response distributions are similar to thosefrom a stochastic generator The data from two experi-ments already discussed are shown in Figure 1 In thesecases reinforcement of least frequent IRTs (Blough 1966)and of nonrepetitive choice sequences (Machado 1989)resulted in stochastic-like responding

Neuringer (1986) reported another striking case Highschool and college students generated sequences of 1sand 2s on a computer keyboard with each set of 100 re-sponses constituting one trial A baseline phase lastedfor 60 such trials for a total of 6000 responses duringwhich the students were asked to behave as randomly aspossible as if they were reporting the outcome of a se-ries of coin tosses As in the many previous studies ofthis sort the subjectsrsquo responses differed significantlyfrom those of a stochastic model (Brugger 1997 Wage-naar 1972) During the training phase the subjects re-ceived feedback which enabled them to compare their per-formances with that of a stochastic model first accordingto one statistical test and then according to another untilfollowing each trial feedback was provided from 10 dif-ferent statistics The results were that all of the subjectslearned to approximate the stochastic model accordingto the 10 statistics as well as some (but not all) other sta-tistical tests That is the distributions of the statisticswhich clearly did not match a stochastic model at the be-ginning of training approximated the stochastic modelat the end of it (see Figure 8)

The previous failures of other researchers to obtain ev-idence for random responding had led some to concludethat humans did not have the capacity to behave ran-

domly What would account for the difference betweenthe present positive finding and the many other negativeones Neuringerrsquos (1986) study was the first in which sto-chastic responding was explicitly reinforced in this casewith feedback from 10 statistical tests An analogy mighthelp to explain the difference in procedures and resultsImagine an experimenter walking around a city and ask-ing people if they know what a violin sounds like (analo-gous to asking people whether they know what a series ofcoin tosses or dice throws is like) Most would answeryes If the same individuals were then asked to play theviolin very few if any would succeed The experimentermight conclude after many replications with differentsubjects in different cities that people did not have thecapacity to play the violin The point of course is thatplaying the violin is a skill requiring training and prac-tice and the same may be true for responding randomlyIt is easy to vary but practice and reinforcement may benecessary to approximate a random distribution

Response number Page and Neuringer (1985) testedthe stochastic-generator hypothesis in another way byvarying number of responses per trial with pigeons work-ing under a lag 3 contingency The reasoning was that ifprevious responses served as a discriminative cue forwhat not to domdasha memory-based strategymdashthen perfor-mance should be degraded when the number of re-sponses was increased Eight responses per trial requiresubjects to remember more than four responses The sto-chastic hypothesis predicts the opposite outcome as isdemonstrated by the following example If each trialwere two responses in length with responses determinedby the toss of a coin then the probability that a given trialwould repeat the previous one would be 25 (There arefour possible sequences in the first trial RR RL LRand LL Thus the second trial has a 1-in-4 chance ofmatching the first) If a trial consists of four coin tossesthe probability of a repetition by chance is 0625 or 1 in16 Thus if subjects used a stochastic process to gener-ate Ls and Rs performances should improve with in-creasing responses per trial Results were precisely thosepredicted by the stochastic hypothesis and inconsistentwith a memory strategy Probability of meeting the con-tingencies and therefore probability of reinforcementincreased as responses per trial increased In otherwords eight-response trials were easier for pigeons thanfour-response trials

Response interference Neuringer and Voss (2002)attempted to distinguish between memorial and stochas-tic processes in yet another way They hypothesized thatwhereas chaotic responding was based on memory forprior responses (as was discussed above) responding inNeuringerrsquos (1986) study with feedback from 10 statisti-cal tests indicated involvement of an endogenous sto-chastic generator To show that human subjects could en-gage in both processes at will three college studentswere trained to generate chaotic sequences in one compo-nent of a multiple schedule as in Neuringer and Voss(1993) and to generate stochastic sequences in a sepa-

694 NEURINGER

rate component as in Neuringer (1986) The wordCHAOTICS was on the screen in the first component theword STOCHASTICS was on during the other and thesewords alternated throughout the session After the sub-jects succeeded at each of the component tasks test ses-sions were introduced in which memory for prior re-sponses was subject to interference in both the chaoticsand stochastics components as follows During the

chaotics component four separate chaotic sequenceswere cycled one after the other each cued by differentscreen colors The subjects attempted to approximate achaotic sequence in one color (eg light green) follow-ing which they would have to match a second chaotic se-quence (lavender screen color) and so on the fourchaotic contingencies following one another across thesession Although all four chaotic sequences used the

Figure 8 Performance by one subject at the end of training to respond stochastically in Neuringer(1986) Shown are comparisons on 6 different statistics between the subject (solid lines and Xs) andthe random model (dashed lines and closed circles) At the end of training the subjectrsquos perfor-mance did not differ statistically from that of the random model on these and four additional sta-tistics From ldquoCan people behave lsquorandomlyrsquo The role of feedbackrdquo by A Neuringer 1986 Jour-nal of Experimental Psychology General 115 p 71 Copyright 1986 by the American PsychologicalAssociation Reprinted with permission

OPERANT VARIABILITY 695

same logistic-difference equation that a model did theywere out of phase and independent If chaotic respond-ing is memory based then the four independent se-quences would be expected to interfere with one anotherthat is approximations to the four chaotic models wouldbe lower than when only a single chaotic output was re-quired as in Neuringer and Voss (1993) That was ex-actly the finding Performances by the three subjectswere signif icantly degraded during the four-segmentphase To put it simply chaotic responses interfered withone another

A different pattern of results was seen in the stochasticsportion of the experiment The same interference proce-dure was used with four independent stochastic sequencesrequired in the four different color conditions As inNeuringer (1986) subjects were required to satisfy eightdifferent statistical tests of randomness and to do this in-dependently in each of the four phases (If responses in anytwo or more of the four stochastic phases became corre-lated above a minimal level responses were considered tobe incorrect thereby prohibiting repetition of a single se-quence in two or more of the phases) For one subjectseven of eight statistics were closer to those of a randommodel during this interference phase than they were at theend of the original training with only a single sequenceand for a second subject all eight statistics were closerThe third subject showed closer-to-random approximationunder the four-segment condition on only two of the eightstatistics but most of her changes were also in the directionof a random generator For example if a random generatoremits equal numbers of ldquouprdquo and ldquodownrdquo sequences andthis subject in the baseline phase had emitted only 47ldquouprdquo sequences then during the four-segment tests shecame to emit ldquouprdquo 55 of the time She changed in the di-rection of the random model but overshot the mark Thuswhereas interference degraded chaotic performance sig-nificantly in all three subjects it improved stochastic per-formances in at least two of them These results are con-sistent with the hypothesis that a memory-based processcontrols chaotic variability and that a stochastic processnot dependent on memory controls stochastic variability

Summary of Tripartite TheoryThe following three processes constitute a tripartite

theory of operant variability attention to random events inthe environment (coin toss) memorial processes (chaosradial maze) and stochastic processes (response se-quences and random-number generation with feedback)Random environmental events assure unpredictabilitymdashfor example in making unbiased selections Remember-ing what was done previously will enable one to avoidrepetitions to vary the students called upon in class tovary the locations when looking for a hidden object orto try different approaches when attempting to solve aproblem External aids such as notes to oneself or sci-entific records help to enhance memory capacities

An endogenous stochastic generator provides the thirdand most controversial source of behavioral variability

When an opponent is attempting to predict your behav-ior in a game or at war it may be functional to behavestochastically Similarly operant conditioning of new re-sponses may depend on stochastic variations It is im-possible to prove stochasticity some presently unknowndeterministic system may ultimately explain the resultsHowever the evidence is consistent with the stochastichypothesis and inconsistent with the alternatives For ex-ample memory is not sufficiently powerful to explain alloperant variability memory-interference proceduresleave stochastic responding intact and behaviors corre-late well with those of stochastic models

Both genetics and reinforcement play important rolesin all three processes Reinforcement is always involvedin that learning provides information about when whatwhere and how much to vary Genetic influences areshown in that not all responses are possible and if theyare possible they are not all equiprobable (eg as inNeuringer et al 2001) Stochastic responding may be bi-ased due to experiences and knowledge that is biasesmay be established by reinforcement contingencies or bythe interface between the organismrsquos propensities (or be-havior systems see Timberlake amp Lucas 1989) and theenvironment In the case of random-number generationby human subjects biases may be due to misconceptionsconcerning what ldquorandomrdquo means However some bi-ases can be overcome through feedback and instructions(Lopes amp Oden 1987 Neuringer 1986)

The importance of variations for evolutionary changeis supported by the many sources of genetic variabilityand by the fact that genetic variability is conserved Asimilar argument holds for operant variability That thereare at least three independent sources of operant varia-tions as well as other elicitors such as environmentalnoise and response extinction supports its importancefor normal adaptive functioning

Structure of VariabilityEvery operant ldquoresponserdquo is actually an instance of a

class with each member of the class exerting the sameeffect on the environment (Skinner 1959) When a rat isreinforced for pressing a lever for example the class isdefined by closure of a microswitch and any way of ac-complishing the closure whether with the left paw theright paw or the mouth is as effective in producing re-inforcement as any other If the contingencies of rein-forcement so specify however the response class can beredefined and circumscribed which is another charac-teristic of the operant For example if only right-pawpresses are acceptable members of the reinforced classresponding will tend to occur only with the right paw(Stokes 1995) Of course the capabilities of the organismplay a role for example including ldquotail pressesrdquo in the op-erant class does not mean that the lever would be or couldbe pressed by the ratrsquos tail The structure of the operantclass therefore emerges from the current reinforcementcontingencies and the organismrsquos capability derivedfrom genetics development and prior experiences

696 NEURINGER

Operant variability has structure as well and as for alloperants that structure is due partly to the organism it-self and partly to the reinforcement contingencies Theresearch described throughout this article has providedexamples of the structure of variability generally in theform of sets of four or eight responses across L and Roperanda But the structure of variations can be speci-f ied more precisely For example Mook and her col-leagues (Mook et al 1993 Mook amp Neuringer 1994)reinforced rats for varying four response sequences acrossL and R levers under a lag 1 contingency Unlike in pre-vious studies however the acceptable class was furtherconstrained so that only sequences beginning with two Rresponses (RRLL RRLR RRRL and RRRR) were rein-forced The rats learned to vary their sequences withinthe imposed limitations that is they generally begantheir sequences with two R responses and varied the se-quences within those limitations Similar results wereobtained in the context of a radial-arm maze in whichonly a subset of arms was baited

The precision with which reinforcement contingen-cies can specify class was highlighted in research per-formed by Burns (1997) and Ross and Neuringer (2002)In the latter case college students drew rectangles on acomputer screen with the computerrsquos mouse Each rec-tangle constituted one trial (the screen cleared at the endof each trial) with reinforcement (points) depending onsimultaneous variations along three dimensions of therectangle location on the screen area of the rectangleand shape with the latter defined as the ratio of the rec-tanglersquos height to its width Although the subjects weregiven no instruction other than to ldquogain pointsrdquo theylearned to vary along all three dimensions with levels ofvariability significantly higher than those of yoked controlsubjects who were reinforced independently of variations

A second part of the Ross and Neuringer (2002) studydemonstrated even greater precision of control To gainpoints three new groups of subjects were required to re-peat along one of the dimensions while simultaneouslyvarying along the other two One group was required torepeatedly draw rectangles of approximately the samesize while simultaneously varying location and shapeanother group had to repeat the shape while varying areaand location and the third to repeat location while vary-ing size and shape Verbal instructions were minimal asthey were in the first part of the study Subjects learnedto satisfy these contingencies as well varying two at-tributes while simultaneously repeating one and signif-icant differences were obtained rapidly within 300 trials

These demonstrations raise questions as to how binaryfeedbackmdashreinforcement presented or notmdashcan simul-taneously control independent behavioral dimensionsThe issue is similar to that discussed above concerningsimultaneous reinforcement of variations and selectionsIt is similar also to those issues faced by researchers at-tempting to understand how the nervous system strength-ens particular synapses when many different neurons firesimultaneously and to ldquoallocation of creditrdquo in artificial

learning models For present purposes suffice it to notethe exquisite precision with which reinforcement controlsvariability and repetition

GeneralizationDo animals and people learn to ldquovaryrdquo implying a

generalizable skill or do they learn a specific distributionof specific responses across particular operanda Thequestion of generalization has been asked of many otherbehavioral competencies such as imitation When chil-dren learn to imitate do they learn stimulusndashresponseassociations (if mother moves her right hand the childlearns to move his right hand) or do they learn moregenerally to ldquoimitaterdquo Evidence indicates that reinforce-ment of imitative instances leads to generalized imitation(see eg Baer amp Sherman 1964 Bandura 1986) A sim-ilar question has been raised when pigeons rats or chil-dren learn to choose a stimulus that matches a previouslyshown sample For example if a pigeon is shown a reddisk as a sample it must then choose red when given achoice between that and blue Does the bird learn to matchIn this case the answer appears to depend on the train-ing procedures If trained with few stimuli such as redgreen and blue then the pigeon learns simple associa-tions (if red sample choose red if blue sample chooseblue) but if hundreds of different stimuli (pictures) areemployed during training then generalized ldquomatchingrdquois learned (Wright 1997) In each of these examples thelearning of a general competency or strategy was shownby transfer-of-training tests What was learned with oneset of stimuli or in one situation transferred to new stim-uli and situations We turn now to the question of whetherlearning to vary transfers

After the subjects in Maltzmanrsquos (1960) experimentdescribed above learned to generate remote verbal as-sociates they were asked to devise unusual uses of com-mon objects In comparison with those in the controlgroup these subjects generated significantly more un-usual uses indicating transfer of training to a differenttask However transfer was not observed in the oppositedirection from unusual uses to verbal associates

Holman et al (1977) using a procedure similar to thatof the Goetz and Baer (1973) research described aboverewarded preschool children verbally for novel drawingsand found increased novelty Unrewarded block construc-tions were also monitored and weak evidence was ob-tained for transfer of novel responding from rewardeddrawing to the unrewarded constructions In a second ex-periment preschoolers engaged in four independent taskspainting pen drawing wood block construction and Legoconstruction When the children were reinforced for vary-ing their pen drawings the novelty of paintings increasedalong with the pen drawings indicating transfer from onetask to the other However transfer to the topographicallydifferent block- or Lego-building tasks was not observed

Eisenberger and Armeli (1997) gave monetary rewardsto one group of fifth- and sixth-grade students the vari-ability group for describing unusual uses of common ob-

OPERANT VARIABILITY 697

jects A second group the repeat group received thesame rewards for describing common uses of the sameobjects In a later test all of the children were asked todraw pictures incorporating a circle and the group that hadpreviously been reinforced for varied responses drewmore unusual (or lower frequency) pictures than did therepeat group Thus there was transfer of novelty trainingfrom one domain to another with additional studies in theseries showing similar effects (Eisenberger Armeli ampPretz 1998 Eisenberger Haskins amp Gambleton 1999Eisenberger amp Selbst 1994) However Eisenberger et al(1999) failed to replicate the transfer-of-training effectThe variability and repeat groups responded similarly dur-ing the test phase In this study separate experimenterstrained and tested the subjects thereby avoiding possibleexperimenter-expectancy effects whereas the same ex-perimenter did both training and testing in Eisenbergerrsquosother studies Across the series of experiments there weremany other cases in which transfer was not observed andmore research is necessary to explain why

In Lee et alrsquos (in press) study with autistic individualsdescribed above after variable answers were trained toquestions verbal variability transferred to new experi-menters and new contexts

Arnesenrsquos (2000) results described above are consis-tent with transfer of training in an animal model Rats inthe Var group were trained to interact variably with oneset of objects When later tested in a new situation withnew objects the Var animals showed (in comparisonwith controls) higher levels of exploration manipulationof the novel objects and discovery of hidden reinforcers

Much of the evidence reviewed in this section there-fore indicates transfer of training and supports a generalldquovaryrdquo competency However there remain unresolvedquestions the most important of which is the extent towhich transfer occurs Note also a methodological pointMany of the studies tested whether transfer occurs whenthe contingencies are neutral with respect to the to-be-transferred behavior Holman et al (1977) is a case inpoint Variations were reinforced in one domain draw-ing and the question was whether variations wouldoccur in another domainmdashblock constructionmdashin whichreinforcement did not depend on varying A negativef indingmdashin this case absence of transfermdashindicatesnothing about the ability to generalize but only aboutthe tendency to do so given the absence of a reason (orcontingency) A better test would be to compare experi-mental and control performances when variability in thenew task is in fact functional as in Arnesen (2000)

SummaryBefore variability can be reinforced it must occur for

reasons other than reinforcement Some of the reasonsinclude variability inherent in all behavioral phenomenaldquoaccidentalrdquo environmental events that are unrelated toparticular behaviors and withholding of reinforcementTransfer from other situations may serve a similar functionFor example if an organism can discriminate shaping-

of-new-response periods from others then it may learngenerally to ldquovaryrdquo during shaping episodes After rein-forcement makes initial contact with response variationsthere are at least three behavioral strategies that can beemployed in order to maintain functionally high levelsof variability respond based on external random eventssuch as the toss of a coin avoid repetitions by remem-bering what responses had previously been made and re-spond stochastically Evidence for an endogenous sto-chastic process was seen in the similarity of responsedistributions to those generated by stochastic sourcesand by findings that interfering with memory for priorresponses either improves or leaves unaffected stochas-tic behaviors Stochastic responding as well as othertypes of variability-generating strategies depend uponcontingencies to define the required level of variabilityand the set from which variations emerge

IMPLICATIONS

One implication of reinforced variability is that oper-ant responses may differ in the extent to which their vari-ability is affected by reinforcement contingenciesmdashthatis operants may be ordered along a continuum of sensi-tivity of variability to reinforcement Another is that re-inforced variability causes us to question the generalityof some basic goals of behavioral psychology includingprediction and control A third is that reinforced vari-ability may help us to provide a scientifically useful wayto describe voluntary action In particular the variabilityof all voluntary behaviors can readily be changed mo-ment to moment by discriminative stimuli and conse-quences Implications for everyday life also will be noted

Operant ContinuumAs was indicated above the operant is composed of

instances that is the operant is a class of many differentresponses all functionally equivalent but often topo-graphically distinct Skinner suggested and I hypothe-size that instances within the class emerge stochastically(see Moxley 1997 for a discussion of Skinnerrsquos posi-tion) Although reinforcement contingencies and contextinfluence the individual probabilities and their distribu-tions emission is stochastic nonetheless The particularmember of the operant class generally cannot be predictedwith a high degree of certainty For example a particu-lar rat may be more likely to press the lever with its leftpaw than its right but left- and right-paw responses areemitted probabilistically This class-nature of the operantsuggests one way to conceptualize operant variability interms of size of class and within-classes probability dis-tributions High variability implies large classes equalprobability distributions or both low variability impliesthe opposite

Variability may be modified more readily in some op-erants than in others implying that operant variability isnot an all-or-none phenomenon but rather a continuumwith reinforced variations in variability (RVV) differing

698 NEURINGER

for different operants RVV might best be understood byanalogy to the concepts of speed and acceleration Bothvariability and speed range from low to high RVV isanalogous to acceleration the rate of change in speedHighly sensitive RVV is indicated by the possibility ofreinforcing many different levels of variability from rep-etition to random-like responding and by the fact thatchanges in level occur readily and rapidly

Different operants manifest differences in RVV Forexample variability of chain pulls changed more thanthat of leverpresses when variability was reinforced inrats (Morgan amp Neuringer 1990) Some operant-like re-sponses may show relatively low RVV For example fir-ing rates increase in single hippocampal nerve cellswhen dopaminergic substances are contingent upon theresponse (Stein Xue amp Belluzzi 1994) suggesting thatnerve f iring is controlled by reinforcement To datehowever there is no evidence that variability of nervefiring is similarly controlled Other cases might includereinforcement of nondiscriminable (subconscious) re-sponses such as Hefferline and Pererarsquos (1963) reinforce-ment of subaware thumb twitches and biofeedback ofmuscle and cardiovascular activity In these cases rein-forcement may modify performance but it is not knownwhether variability is also sensitive to reinforcement andif so to what degree Human verbal variability may bethe most readily manipulable of all behaviors the rangeof predictable to unpredictable verbal responses beingimmense with extraordinary sensitivity to context andconsequence RVV may also differ across organisms (per-haps from simple to complex) individuals (perhaps asan index of intelligence or learning ability) and physio-logical and psychological states (WKY vs SHR depres-sion and autism)

RVV may help to distinguish operant responses fromother superficially similar cases of selection by conse-quences That is the ldquooperantrdquo may also best be concep-tualized as a continuum with RVV an important deter-miner of position along the operant continuum Oneexample of selection by consequences but low ldquooperant-nessrdquo is genetic change Variability is essential for evolu-tionary change of course and levels of genetic variabil-ity are sometimes affected by environmental conditions(Moxon 1997) but there are few examples of levels ofvariability being influenced by selection pressures Thatis the source of much genetic variability is thought to berandom not influenced by current pressures and needsThus although selected by consequences genetic changemay be low along the operant continuum Other exam-ples of operant-like phenomena for which variability although important may be relatively insensitive to con-sequencesmdashvariability in these cases is elicited not se-lectedmdashinclude the development of the central nervoussystem and immune system responses (Hull et al 2001)In both of these cases change depends upon selection byconsequences but it is not clear whether variability canalso be selected Most commonly studied operant re-sponses on the other hand show high sensitivity of vari-

ability to reinforcing events But to reiterate the mainpoint of this section operants differ in terms of RVV andtherefore in terms of their ldquooperantnessrdquo from low sensi-tivity (possibly responses of single nerve cells) to extra-ordinarily high sensitivity (verbal behavior) The operantnature of variability as well as that of every other oper-ant dimension is best conceptualized as a continuum

Prediction and ControlSmith (1992) described Skinnerian operant condition-

ing theory as Baconian in nature a primary goal of Skin-nerian behaviorists being to control behavior and hecontrasted this with the more Aristotelian goal of de-scription Skinner and his students (eg Zuriff 1985) in-deed emphasized prediction and control Many of thestudies reviewed above however involved reinforce-ment procedures that increase variability and conse-quently decrease predictability It is not only ignorancetherefore that restricts prediction and control as Skin-ner suggested but reinforcement itself Operant meth-ods and theories when applied to variability lead togreater emphasis on description with control over pre-cise response characteristics possible under some butnot all conditions

In summary an operant is a class of responses underthe control of contingencies of reinforcement The classcan be controlled and predicted given knowledge ofconditioning history Indeed the size of the class andpossibly the distribution of within-classes probabilities orlevel of variability can also be controlled and predictedBut to the extent that instances emerge stochastically aparticular instance cannot be predicted The instant re-sponse can be explained as a member of a particularclass but prediction of the instance will be diff icultsometimes to the extreme This point will be developedin the next section

Voluntary BehaviorI hypothesize that voluntary behaviors are operants for

which RVVs are highly sensitive to influence by conse-quences One way to test this hypothesis would be togather empirical data concerning how voluntary versusinvoluntary actions are classified by human observers Asfar as I know there are no psychophysical data concern-ing this question but the hypothesis implies that highsensitivity of variability to operant contingencies RVVswill be present in all actions perceived to be voluntary

This operant-variability theory of voluntary action isbased on the supposition of two attributes First volun-tary acts must be potentially explainable (given suff i-cient knowledge) on the basis of genetics conditioninghistories current contingencies anticipated conse-quences intentions goals and the like That is volun-tary behaviors are lawful The second attribute is that thevoluntary act must be potentially unpredictable That isvoluntary acts are ldquofreerdquo of control by identifiable stim-uli If behavior is perceived necessarily to result frommdashto be predicted on the basis ofmdashexternal events includ-

OPERANT VARIABILITY 699

ing reinforcement contingencies it will not be perceivedas voluntary ldquoVoluntaryrdquo implies potential indepen-dence from stimulusndashresponse or responsendashreinforcerdetermination According to this theory both explain-ability and unpredictability are necessary attributes ofvoluntary acts

How though can behaviors be simultaneously ex-plainable and unpredictable Reinforcement contingen-cies play two essential roles Reinforcement (togetherwith genes prior experiences and current environment)helps to determine the set of possible behaviors fromwhich a voluntary act emergesmdashthat is the operant classReinforcement (together with the other contributorsspecified above) also helps to determine the level of vari-ability within the class Thus both instances comprisinga class and level of variability within that class can bepredictedmdashagain given sufficient knowledge Howeverthe individual response is at least sometimes (or relatively)unpredictable The reason is that the operant class fromwhich the response stochastically emerges is sometimesextremely large (eg ldquoname a wordrdquo or ldquodo somethingrdquo)and the contingencies sometimes call for equiprobablestochastic emergence (eg ldquofool merdquo) In the latter casea voluntary act may appear to be foolish meaninglessor unwarranted but if it is voluntary it can potentially beexplained (For example in response to the questionldquoWhy did you do that silly actrdquo the answer could beldquoJust to show that you canrsquot predict my behaviorrdquo seeScriven 1965) In these cases prediction of instancesfalls to very low levels indeed and I hypothesize evenan all-knowing observer will be unable to predict thenext response at greater-than-chance levels which aregiven by the size of the operant class and by within-classesprobability distribution Even within small classes (egin the game rockndashscissorsndashpaper in which unpredictabil-ity among the three possible responses is functional) be-havioral stochasticity may lead to unpredictability of theinstant response Note however that although voluntarybehaviors are sometimes unpredictable the opposite isoften the case When contingencies call for highly con-strained particular responses predictions can easily bemade For example if a police officer asks your nameyour response can most likely be predicted Voluntary re-sponses are therefore described as potentially unpre-dictable with variability contingencies engendering dif-ferent levels of unpredictability

According to this theory behaviors low in the sensitivity-of-variability dimension are described as relatively in-voluntary and those that are high in said dimension areperceived as voluntary Low RVV may be shown at anylevel along the variability continuum For example al-ways responding in the same unchanging way will beperceived as involuntary but so will always respondingrandomly (The coin does not voluntarily come up withheads or tails) This point is essential More than 2000years ago Epicurus argued that occasional randomswerves of atoms provided the source of voluntary be-havior in an otherwise determined universe but al-

though this conjecture was prescient it was not quitecorrect ldquoFree-willedrdquo or voluntary action whether it isbased on random swerves or on quantum mechanics isnot explained by variability per se but by functionalchanges in levels of variabilitymdashthat is by rapid andprecise changes in response to environmental demandsResponses can be placed along a voluntaryndashinvoluntarycontinuum isomorphic to the operantndashrespondent con-tinuum with voluntary behavior being functional pre-cisely in the same way as operant behavior is ThusSkinnerrsquos (1974) suggestion that voluntary behavior isoperant behavior was almost correct Only those oper-ants for which variability is highly sensitive to conse-quences will be perceived as voluntary

One additional example might help Operant classes(sets of potential responses) are created ldquoon linerdquo as we goabout our daily activities As we walk from car to officewe tend to walk in a characteristic way take the samepath with tightly constrained operant classes followingone after the other in rapid succession Class sizes aresmall and within-classes probability distributions arehighly peaked Predictions of next responses by a knowl-edgeable observer can readily be made However classsizes can be increased and probability distributions flat-tened we can walk in quite strange ways indeed dancealong a new path sing hand candy to passers-by Strangebehavior indeed but made possible by the ability of op-erant sets to be modified moment to moment not only interms of class definition but in terms of class size lev-els of within-classes variations and so on The same istrue for conversation Moment-to-moment changes inverbal class definition size and predictability of within-classes instances are hallmarks of voluntary behavior

GenerativityRelated to the definition of the operant and to our discus-

sion of voluntary action is that fact that operant behaviormdashleverpressing no less than languagemdashis by its naturegenerative This claim follows from the functional na-ture of the class of responses (all members of the classare effective in producing an outcome) and the stochas-tic emission of instances From these flow never-before-emitted instances patterns and sequences (see Epstein1996) Operants especially those high on the RVV con-tinuum are inherently generative

Self-ExperimentationVariations systematic or otherwise are a necessary part

of scientific work Neuringer (1981) and Roberts and Neu-ringer (1998) described how variations can be brought to bear explicitly on everyday life in the form of self-experiments Individuals vary aspects of their lives suchas the foods they eat social contacts and methods ofmemorizing they keep records concerning the effects ofthese manipulations and form hypotheses concerningcontrolling variables Using self-experimental methodsindividuals can learn how what when where and howmuch to vary Explicitly varying onersquos everyday behav-

700 NEURINGER

iors and thoughts may help to engender what Langer(1989 2000) has referred to as mindful livingmdashthat isnonautomatic voluntary engagement with beneficialeffects in different areas including learning The impor-tant point here is that variability can be voluntarily cho-sen maintained and modified in everyday life and thatexplicit variations may have important consequences

Implications for SocietyConsideration should be given to the importance of re-

inforcement contingencies to support variability in so-cial milieus A basic point made throughout the presentarticle is that merely permitting behavior to vary doesnot necessarily engender or maximize variability (as isshown by Yoke and Prob schedules) Analogously merelypermitting varied social behavior as in a free or laissez-faire society may not suffice to produce the desired di-versity If the results shown for individual organismscan provide useful information for groups (see eg Grottamp Neuringer 1974) then when variations are desiredsocieties might reward individuals for varying not sim-ply permit it

SummaryOperants differ in the extent to which reinforcers in-

fluence levels of variation When variability is highlysensitive to reinforcement the operant may be perceivedto be voluntary Voluntary responses have two attributesThey are functional implying that the class of responsesmdashthe operantmdashcan be predicted given sufficient knowl-edge But instances emerge stochastically implyingsome degree of unpredictability Because reinforcementof variability can generate extremely large classes withstochastic emergence of instances it suggests limitationsof the two basic goals of Skinnerian behavioral theorymdashnamely prediction and control Operant variability also hasimplications for self-experimentation and for society

RELATED AREAS

Operant variability may be important in many otherareas and I will briefly review some of these

Negative reinforcement To my knowledge there havebeen no published studies in which variability is nega-tively reinforced High variability might sometimesavoid or reduce a punisher such as loss of goods ormoney but at other times low variability could be func-tional Effects of positive and negative reinforcement onvariability may have different evolutionary bases onederived from increasing probabilities of locating sourcesof food or mates the other from avoiding confrontationsor predation (Driver amp Humphries 1988) One questionconcerns how quickly variability can be changed by neg-ative as opposed to positive reinforcement There isconsiderable evidence that loss functions differ fromgain functions (Kahneman amp Tversky 1984) possiblyindicating that positive reinforcement-of-variabilityfunctions will differ from negative ones as well

Resistance to change In some of the studies describedabove variability appeared to be more resistant tochange than repetitions were for example alcohol hadlittle effect on variable operant responses but interferedwith repeated ones Doughty and Lattal (2001) tested thegenerality of these f indings using four-response se-quences with pigeons a threshold variability contin-gency in the Var component of a multiple schedule andLRLR in the Rep component Doughty and Lattal chal-lenged stability of responding by prefeeding the birds inone experiment and providing reinforcers independentlyof responding in the other (see Nevin 1992) Operantvariability was again more resistant to change than wasoperant repetition Thus reinforcement of variabilitymight increase the stability of responding (that is its re-sistance to change) a seemingly paradoxical findingHowever Mechner (1992) suggested that this is the casefor complex skills such as piano performance Varyingpractice routines will make the performance relativelyinsensitive to unforeseen changes in the environmentCare must be taken however when variation is comparedwith repetition because they have different measuresAlso the results reported by Doughty and Lattal werebased on the performance of three pigeons and were notconsistent throughout the experiment The hypothesiswarrants further study

Attraction In some bird species complex male songsattract females (for a review see Catchpole amp Slater1995) There is evidence that male song complexity isunder the discriminative stimulus control of a receptive fe-male (Searcy amp Yasukawa 1990) That explicit reinforce-ment may play a role in such song complexity is suggestedby the Manabe et al (1997) study described above An in-teresting question for further research is the extent towhich operant contingencies contribute to the attractioncreated by behavioral variability in these and other species

Discrimination of variability Animals and people canuse the variability of a stimulus array or entropy as a dis-criminative stimulus (Young amp Wasserman 2001) Whatis the relationship between perception of variability onthe one hand and emission of variable responses on theother Do individuals attempt to match their own levelsof variation to those of others in their environments whichwould involve both perception and emission How mightdiscrimination of onersquos own level of variability influenceonersquos ability to modify such levels and do influences thatobscure discriminations therefore affect ability to change(see eg van der Meer van der Weel amp Lee 1995)Hunziker et al (2002) report that although variabilitywas successfully reinforced their human subjects couldnot correctly describe the contingencies of reinforce-ment a finding similar to those of informal observationsin other studies (eg Neuringer Deiss amp Imig 2000)but whether subjects could discriminate variability levelsrather than contingencies was not studied

Attention and awareness When attention is subject tointerference or is withdrawn from the task at hand abilityto vary is degraded (eg Baddeley Emslie Kolodny amp

OPERANT VARIABILITY 701

Duncan 1998 Evans amp Graham 1980 Neuringer amp Voss2002) It is unclear at present how attention influencesthe different processes underlying operant variabilityand in particular memorial and stochastic processesRich areas for additional study are the relationships be-tween attention awareness and consciousness on theone hand and operant variability on the other Voluntaryaction and consciousness are often related (see egLibet 1999)

Concepts Concepts are thought by many to be sets ofinstances (much like operant sets) whose structures aregraded meaning that there are more or less typical in-stances of such concepts as birds (robin is typical ostrichis not) Barsalou (1987) showed that the gradation ofthese cognitive structures is manipulable or as he putsit not stable A number of variables influence structureincluding context individual differences and points ofview Barsalou argued that these structures are createdas the need arisesmdashthat is on line by the thinking indi-vidual The structure of concepts may parallel that of op-erants and in both cases levels of within-structures di-versity or variability may be modified by consequencesOperant variability may be found in the cognitive as wellas in the behavioral domain

Play and humor Highly variable behaviors are ob-served when animals and people especially children en-gage in play and humor (see eg Fagen 1981) The re-sult of play may be acquisition of new responses (seeSutton-Smith 1975) Although not currently needed re-sponses learned during play may provide the source ofvariations for successful shaping of behaviors at anothertime This is quite analogous to the role of conserved ge-netic variants which are passed along to progeny but donot currently serve any identifiable role Play may alsoinvolve learning how to vary operantly

Reinforcement by variability The self-maintainingnature of play suggests that engaging in variable behav-iors may itself be reinforcing Indirect support comesfrom findings that varied stimulation can serve as a re-inforcer and that stimulus repetition becomes aversive(Fiske amp Maddi 1961) Possibly also related are findingsthat varying reinforcers and permitting choice among re-inforcers increase their value (Bowman Piazza FisherHagopian amp Kogan 1997) and that permitting choicesamong varied responses is also preferred to demandingfixed responses (Catania 1980)

Foraging Two issues regarding variability are presentin the foraging literature One has to do with risk prefer-ence versus avoidance (see eg Shettleworth 1998) Asecond has to do with search strategies Most reportedcases of foraging involve systematic search patterns(Stephens amp Krebs 1986) However search patterns aresometimes stochastic or highly variable (see eg Peter-son 1997) One question is whether some species can varybetween these two strategies and the challenge is to iden-tify the pressures leading to and the advantages of each

Animal training Wilkes (1996) has applied operantvariability techniques in developing training proceduresfor pet dogs More generally reinforcement of variabil-

ity may enable the gaining of control over behaviors ofanimals and as has been suggested above may be an im-portant part of the process of shaping

Models of learning Reinforced variations in variabil-ity may be important when human and animal learningis being modeled whether the models are mathematicalin form computer based or machine based (Blumberget al 2002 Brooks 1999 Sutton amp Barto 1998)

CONCLUDING COMMENT

Genes interact with environments to produce diverseanatomical physiological and behavioral phenotypesGenes and environments also provide for another level ofdiversitymdashnamely operant variability There are manysources of operant variability including random environ-mental events memory for past responses and stochasticgeneration each of which ultimately derives from ge-netic and environmental influences To the extent thatoperant variability is based on a stochastic source how-ever the behavioral results cannot be predicted fromknowledge of these influences That is organisms haveevolved to behave unpredictably The positing of sto-chasticity to explain behavior has a history extending atleast as far back as Epicurus with major contributors in-cluding Gustav Fechner (see Heidelberger 1987) EgonBrunswik (see Gigerenzer 1987) Skinner (1959) andSuppes (1984) Knowledge of the stochastic nature ofthe operant and of how it is controlled leads to a worldview that differs from genetic and environmental deter-minism with roots in the works of the scholars just men-tioned At the scientific level this view leads to the ex-perimental analysis of consequence-influenced variabilityAt the philosophical level it leads to explanations of vol-untary action At the personal level it can lead to change

REFERENCES

Amabile T M (1983) The social psychology of creativity New YorkSpringer-Verlag

Antonitis J J (1951) Response variability in the white rat duringconditioning extinction and reconditioning Journal of Experimen-tal Psychology 42 273-281

Archer J amp Birke L (1983) Exploration in animals and humansCambridge Van Nostrand Reinhold

Arnesen E M (2000) Reinforcement of object manipulation in-creases discovery Unpublished undergraduate thesis Reed College

Baddeley A D (1966) The capacity for generating information byrandomization Quarterly Journal of Experimental Psychology 18119-129

Baddeley A [D] Emslie H Kolodny J amp Duncan J (1998)Random generation and the executive control of working memoryQuarterly Journal of Experimental Psychology 51A 819-852

Baer D M amp Sherman J A (1964) Reinforcement control of gen-eralized imitation in young children Journal of Experimental ChildPsychology 1 37-49

Baker M J amp Koegel R L (1999) Transforming children withautismrsquos ritualistic behaviors into positive play interactions with sib-lings Paper presented at the meeting of the California Association ofBehavior Analysis San Francisco

Balsam P D Deich J D Ohyama T amp Stokes P D (1998) Ori-gins of new behavior In W OrsquoDonohue (Ed) Learning and behav-ior therapy (pp 403-420) Boston Allyn amp Bacon

Balsam P D Paterniti A Zechowy K amp Stokes P D (2002)

674 NEURINGER

at the short end and large at the long end and increasedsystematically across the IRT range For the response tobe reinforced a pigeonrsquos IRT had to fall in the bin thatcontained the fewest prior entries across a moving win-dow of 150 responses Fine-grained analyses of the pi-geonsrsquo performances showed that successive responseswere not truly independent as would be expected from arandom generatormdashthe tendency for long IRTs to followlong ones and for short IRTs to follow short ones wasgreater than would be expected from a random emittermdashbut this might have been due to particular aspects of theprocedure including the facts that short IRTs were rein-forced somewhat more frequently than longer ones andthat if there were few entries in a bin (because the birdhad not been responding at that IRT) then successiverepetitions of the same IRT could be reinforced repeat-edly Despite these problems the data were impressivelyclose to those of a random model as is shown in the toppanel of Figure 1 To the extent that the functions are lin-ear on the semilogarithmic coordinates the behavior ap-proximates that of a stochastic process The advantagesof Bloughrsquos procedure were that response classes weredefined objectively and independently of a human ob-server reinforcers were administered automatically thou-sands of responses were emitted per session and ses-sions could continue indef initely without changes inresponse difficulty or contingency

Shimp (1967) attempted to extend Bloughrsquos (1966) pro-cedure to pigeon choices Because choices are currently amajor focus of interest and because there were problemswith Shimprsquos procedure it will be discussed in some de-tail Pigeons pecked two keys left (L) and right (R) in acontinuous stream such as LLLLRRLLRRRLRLLL thedata were analyzed on line in terms of overlapping setsof four responses In the example just given assume thateach new response is added to the right of the string Themost recent set was composed of RLLL the second mostrecent (moving to the left by one response) of LRLL thethird of RLRL and so on Given two alternatives L andR and sets consisting of four responses there are 24 or16 possible sequences LLLL LLLR LLRL RRRRReinforcement was provided if the most recent responsehad terminated a sequence that had occurred less fre-quently than all other 15 possible sequences over a mov-ing window of 96 responses Although the resultingchoice distributions were quite variable in some respectsthe results did not match those of a random model andwere diff icult to analyze possibly because of the fol-lowing problem

Assume that 16 counters represent the 16 sequences ofresponses and that one count is added to each counterwhenever its associated sequence is emitted If LLLLhas just been emitted one count will be added to thatcounter Another L will increase the same counter andeach successive L will add another If LLRR has justbeen emitted however thereby adding one count to itsassociated counter four additional responses (not 1 as inthe LLLL case) will be required to add again to the same

counter Therefore 16 successive responses will be re-quired for four instances of LLRR (namely LLRRLLR-RLLRRLLRR) as opposed to seven for L (LLLLLLL)Although a random process would generate equal fre-quencies of all sequences including LLLL and LLRRthe distribution of successive occurrences would differthereby affecting the distribution of reinforcements Thatis LLLL and RRRR would tend to be more ldquoclumpedrdquothan would LLRR and RRLL which would be more dis-tributed Similar considerations apply to all other se-quences That these differences might affect choice dis-tributions is indicated by the many studies showing thatthe distribution of reinforcements over time given equal

Figure 1 (A) Data from one pigeon in Blough (1966) under dif-ferent parameter conditions (the three sets of lines) with replica-tion at each parameter Shown are the mean numbers of re-sponses emitted in each half-second IRT bin a straight line beingexpected from a stochastic responder Adapted from ldquoThe rein-forcement of least frequent interresponse timesrdquo by D S Blough1966 Journal of the Experimental Analysis of Behavior 9 p 587(B) Data (points) from one pigeon at the end of training in Machado(1989) The line represents the frequencies expected of a randommodel From ldquoOperant conditioning of behavioral variabilityusing a percentile reinforcement schedulerdquo by A Machado 1989Journal of the Experimental Analysis of Behavior 52 p 161 Bothfigures copyrighted by the Society for Experimental Analysis ofBehavior Reprinted with permission

OPERANT VARIABILITY 675

average reinforcement frequencies affects choices for ex-ample choices are affected differently by variable-interval(VI) than by fixed-interval (FI) schedules of reinforcement

Wilson (1986) devised one solution to this problemWilsonrsquos procedure is complex but in brief responding wasanalyzed in a manner that avoided overlapping sequences(which was the cause of the clumping in Shimprsquos 1967procedure) Four sets of counters were created Using thesame example given above in the first set one count wasadded to each counter representing RLLL RRRL RRLLand LLLL so that successive trials were completely inde-pendent (Note the absence of overlap) When the next re-sponse was emitted sequences began with that responsemdashlet us assume that the next response added to the rightof the series was an Lmdashand therefore in the examplegiven the second set was created by offsetting responsesone place to the left Now one was added to LLLL withprevious counts added to RRLR RLLR LLLR and soon A similar procedure was followed for each of the re-maining two sets Each response therefore terminated afour-response sequence in one and only one of the foursets and reinforcement was based on satisfaction of thecontingency within that set only Wilsonrsquos procedure hadother aspects but for present purposes it suffices to notethat the dependencies in Shimprsquos procedure were avoidedand when pigeons were tested under these contingen-cies they generated choices that more closely approxi-mated those of a random model Hopefully this lengthyexplanation indicates the importance of precise contin-gencies when one is attempting to reinforce variabilityAn advantage of Shimprsquos and Wilsonrsquos procedures wasthat every response could be reinforced at least poten-tially as was the case for each of the other procedures dis-cussed until now However a simpler way to avoid theproblems just described is to require a fixed number ofresponses in each trial with reinforcement possible onlyat completion of the trial As will be seen most recentstudies have used the latter tactic

The evidence generally supported the claim that be-havioral variability could be reinforced but the agree-ment about these findings was short lived Schwartz (19801982) reached the opposite conclusion and his researchwill be considered in detail

Schwartzrsquos ResearchSchwartz (1980 1982) used fixed-length trials con-

sisting of eight responses per trial by pigeons across tworesponse keys During an initial control phase a trial ter-minated with food if it contained exactly four L and fourR responses regardless of the particular sequence of Lsand Rs However if either key was pecked a fifth timethe trial terminated immediately with timeout A 5 3 5light matrix was provided to assist the birds in this taskAt the beginning of a trial the upper-left corner light wasilluminated and all others were off An L peck movedthe light one column to the right and an R peck movedit one row down Any sequence of four L and four R re-sponses would therefore move the light from upper left

to lower right and the trial would end with food Each ofthe birds came to emit a single sequence with high prob-ability in most cases LLLLRRRR or RRRRLLLL Arelatively low success rate at the end of 40 training ses-sions indicated that the task was harder than it mighthave appeared Only 66 of trials ended with food andthe remaining 34 ended with timeout because withina given trial one of the keys had been pecked a fifth time

Schwartz (1982) then required that the sequence in thecurrent trial differ from that of the just preceding trialThis will be referred to as a lag 1 variability + constraintcontingency (ldquoconstraintrdquo indicating the required fourLs + four Rs) If a bird had emitted LLLLRRRR in theprevious trial for example any other sequence would bereinforced in the next trial assuming that it containedfour Ls and four Rs but a repetition led to timeout Thebirds performed poorly only 36 of the trials now end-ing with food because a dominant sequence was oftenrepeated or one of the keys was pecked five times Hy-pothesizing that this failure to reinforce variability mayhave been due to the birdsrsquo extended experience underthe initial control condition (in which variations were notrequired and any sequence of four Ls and four Rs wasreinforced) Schwartz (1982 Experiment 1) again triedthe variability-requiring procedure with naive subjectsAgain the birds failed In another experiment Schwartz(1982 Experiment 2) explicitly trained birds to alternatebetween LLLLRRRR and RRRRLLLL sequences andthen introduced the lag 1 variability + constraint contin-gency but again the birds failed to vary Schwartz (1982)concluded that variability could not be reinforced andoffered a number of possible explanations for why his re-sults differed from the previous findings Perhaps Maltz-manrsquos (1960) and Goetz and Baerrsquos (1973) findings weredue to the information providedmdashverbal indicationsfrom the experimenters that subjects should varymdashrather than to reinforcement Perhaps Pryor et alrsquos (1969and by implication Goetz amp Baerrsquos 1973) results weredue to brief extinction periods in which reinforcers werewithheld (because variations were not forthcoming)rather than to direct reinforcement of variability PerhapsBloughrsquos (1966) results were specific to IRT and not rel-evant to response forms or choices These explanationsall imply that it is not possible to reinforce variability orthat it is possible only in narrowly defined cases As willbe seen Schwartzrsquos (1982) results readily met the test ofreplication but there was a problem nevertheless withhis conclusion

Schwartz ExplainedPage and Neuringer (1985) simulated Schwartzrsquos (1982)

lag 1 variability + constraint procedure with random ldquore-sponsesrdquo generated by a computer Although the simula-tion generated L and R with equal probabilities (05) ldquore-inforcementrdquo was obtained on only 29 of its trials whichwas somewhat lower than the level attained by pigeonsThe reason is that of the 256 possible eight-response se-quences only 72 contained exactly four Rs and four Ls

676 NEURINGER

and therefore the random simulation often failed tomeet the 4L + 4R constraint The implication was that ifthe birds attempted to respond randomly they would bereinforced only rarely

To test whether the penalty for fifth pecks was respon-sible for the birdsrsquo failure to vary Page and Neuringer(1985) permitted eight responses across L and R keyswithout the four responses-per-key requirement If thecurrent sequence of eight responses differed from that inthe last trial a lag 1 variability contingency (without con-straint) reinforcement was provided Now the pigeonssucceeded in varying attaining reinforcement on morethan 90 of the trials a rate significantly higher thanthat observed in Schwartz (1982) and close to that of thesimulating random responder under the new procedurePage and Neuringer also replicated Schwartzrsquos (1982)variability + constraint procedure and again the pigeonsfailed just as did Schwartzrsquos The evidence was clearUnder Schwartzrsquos (1982) procedure pigeons failed tovary but penalty for fifth responses was responsible

One additional aspect of Schwartzrsquos (1982) proceduremight also have contributed to low variability If thebirds intermixed a single preferred sequencemdashone thatmet the 4L + 4R constraintmdashwith any other then 50 ofthe trials would end with food this level of intermittentreinforcement being sufficient to maintain responding inmany other situations For example if a bird alternatedbetween LLLLL (the fifth L producing a timeout) and adominant LLLLRRRR sequence the latter would alwaysresult in reinforcement Birds under the variability +constraint contingencies in fact emitted a dominant se-quence with high probability and many of their rewardswere obtained following that sequence (because it hadbeen preceded by a different sequence often one con-taining five Ls or five Rs) Thus failure to vary could beexplained by the fact that variability was not differen-tially reinforced whereas a single dominant sequencewas Schwartz (1988) later reviewed this new evidenceand concluded that pigeons could indeed be reinforcedfor responding in random-like fashion

Yoke ControlOne of Schwartzrsquos (1982) hypotheses is worth further

consideration When variability is reinforced there areperiods during which reinforcement is withheld becauseresponses did not satisfy the contingencies As will bediscussed below many studies have shown that extinctionelicits variability Might short-term extinction periodswhen reinforcers were not forthcoming have elicited thevariability attributed to reinforcement

Page and Neuringer (1985) compared two conditionsIn one referred to as Var reinforcement depended on se-quence variations similar to the lag 1 variability contin-gencies described above (The matrix of lights was nolonger used because it was found not to contribute to re-sponse variability) In the other condition referred to asYoke exactly the same frequency and distribution of re-inforcements were provided but independently of se-

quence variations This was accomplished by having thecomputer maintain a record of which trials were rein-forced and which led to timeout in the Var phase andusing this record to program reinforcers during Yoke Fora particular bird for example the first as well as the sec-ond trials in a Var session might have been reinforcedbut in the third trial the bird may have repeated a se-quence causing a timeout and so forth In the Yoke phasethe first and second trials would be reinforced but thethird trial would lead to timeout whether the birdrsquos re-sponses varied or not Each birdrsquos own sequence of rein-forcements and timeouts in Var was therefore replicatedin Yoke If short-term extinction or withholding of rein-forcement was responsible then variability should beequal in the Var and in the Yoke phases That was not theresult Variability was significantly higher in Var than inYoke a finding that has been confirmed many times inour laboratory and in those of others (eg Barba amp Hun-ziker 2002 Machado 1992) We conclude that variabil-ity was in fact reinforced

VARIABILITY IS AN OPERANT

Control by reinforcement is the most notable charac-teristic of an operant response but there are others includ-ing precise influence of contingencies effects of rein-forcement frequencies control by discriminative stimulilawful distribution of choices and extinction Researchin each of these areas supports the hypothesis that be-havioral variability is an operant dimension similar toother operant dimensions An organism can be rein-forced not only for repeating a particular behavior but fordistributing responses across a wide array of possibilitiesin a way that may be quite unpredictable The evidencewill be reviewed

Contingencies and Frequencies of ReinforcementContingencies specify what must be done to earn re-

inforcement When applied to variability contingenciesspecify a set of responses and in most studies to date aminimal level of variability within that set That controlby contingencies does not depend on reinforcement fre-quency was demonstrated by Blough (1966) for pigeonIRTs and by Machado (1989 1992) for pigeon responsesequences across L and R keys In both cases when re-inforcement depended on high variability response vari-ability was high when intermediate variability was re-quired the level generated was intermediate and whenlower levels of variability were reinforced variabilitywas relatively low Reinforcement frequencies were heldconstant in these studies and therefore were not respon-sible for the changes in the levels

With respect to other operant dimensions such as re-sponse rate reinforcement frequency often has complexand sometimes inconsistent effects As reinforcementfrequency increases response rates sometimes increasesometimes decrease and sometimes are unchanged Vari-ability is also affected in seemingly inconsistent ways

OPERANT VARIABILITY 677

Some studies report that variability increases when rein-forcement frequencies decrease (Boren Moerschbaecheramp Whyte 1978 Tatham Wanchisen amp Hineline 1993Tremont 1984) but others report small or no effects(Blough 1966 Eckerman amp Lanson 1969 Herrnstein1961 Machado 1989)

Grunow and Neuringer (2002 Experiment 1) inde-pendently manipulated variability contingencies and re-inforcement frequencies and their results may help toexplain the previous inconsistencies Four groups of ratswere studied with the contingencies for each group re-quiring a different minimum level of variability (A thresh-old contingency was employed the details of which willbe described below) In the initial phase reinforcementwas provided whenever the respective contingencieswere met (reinforcement frequency was relatively highin this phase) and as can be expected from the Blough(1966) and Machado (1989) studies described above lev-els of variability were directly controlled by the contin-gencies In a second phase with the same variabilitycontingencies held constant reinforcement was limitedto no more than once per minute (VI 1 or intermediatefrequency) In a third phase meeting of the variabilitycontingency was reinforced on an average of only once

every 5 min (VI 5 or low frequency) We observed a sig-nificant interaction between reinforcement frequencyand variability contingency (Figure 2) which helps toexplain some of the previous inconsistencies In thegroup that was reinforced for low variability representedby the triangles in Figure 2 as reinforcement frequencydecreased variability increased slightly in the group re-inforced for high variability represented by the squareswhen reinforcement decreased variability decreasedand in the intermediate groups variability was unchangedAlthough these effects were small they reached statisti-cal significance As can be seen in Figure 2 variabilitycontingencies affected variability more than did rein-forcement frequencies and effects of the latter were in-fluenced by the contingencies

An example might help to illustrate these complexfindings When low variability is required perhaps as in assembly-line work low variability emerges When highvariability is required perhaps as in artistic work re-sponse variability is high Now what happens if the normalrate of reinforcement for each of these tasks is loweredAccording to Grunow and Neuringerrsquos (2002) findingsthe assembly-line workerrsquos variability would increasebut only slightly and the artistrsquos variability would de-

Figure 2 Average U-values across groups of rats in Grunow and Neuringer (2002)The highest function (squares) is from a group reinforced for highest sequence vari-ability the lowest function (triangles) is from a group reinforced under a lenient con-tingencymdashthat is low variability sufficed for reinforcement the other groups (Xs andcircles) were reinforced for intermediate levels of variability Each group was rein-forced at three frequencies each time the contingency was satisfied (continuous rein-forcement ie CRF) on an average of not more than once per minute (VI 1) and onan average of not more than once every 5 min (VI 5) High U-values indicate high lev-els of sequence variability and low U-values indicate relatively repetitive respondingError bars represent standard errors of the mean

678 NEURINGER

crease somewhat Importantly the artist would continueto behave much more variably than the assembly-lineworker Both contingency and frequency affect responsevariability but contingency appears to be more influen-tial at least over the range of values studied to date Thesefindings if general are important for those who withholdreinforcement in order to encourage response variationsas is often the case when behavior modifiers attempt toshape new behaviors Effects of withholding may dependon baseline contingencies and a more productive proce-dure might be to reinforce variability directly We willconsider this suggestion further below

Discriminative ControlThe operant response as defined by Skinner is con-

trolled by discriminative stimuli as well as by conse-quences For example red traffic lights are the discrimi-native stimulus for depressing the automobile brake thedistance of the golf ball from the hole provides a discrim-inative stimulus for how hard to hit the ball In the labora-tory discriminative control is often studied with multipleschedules in which one reinforcement contingency suchas fixed ratio is operative in the presence of one stimulussuch as a red keylight and a different contingencymdashforexample extinctionmdashis operative in a second stimulusmdashfor example a green keylight Discriminative control isshown by differences in responding Variability is alsocontrolled by discriminative cues

Exteroceptive discriminative stimulus controlPage and Neuringer (1985 Experiment 6) reinforced pi-geons for repeating a single sequence LRRLL in thepresence of blue keylights and for variable sequences inthe presence of red Blue and red alternated after every10 reinforcements The birds learned to repeat in thepresence of blue and to vary in the presence of red andwhen the stimulus relationships were reversed the birdsreversed their patterns Cohen Neuringer and Rhodes(1990) obtained similar results with rats Denney andNeuringer (1998) extended this research to show thatdiscriminative control did not depend on reinforced repe-titions in one of the components Again within the contextof a multiple schedule rats were rewarded for variablesequences under one stimulus (Var) and independentlyof variability under a second stimulus (Yoke) Frequen-cies of reinforcement were identical in the two stimulusconditions That the stimuli exerted significant controlover variability was indicated by three main findingsVariability was higher in Var than in Yoke differences invariability were greatest immediately following stimuluschange and when the discriminative stimuli were re-moved and all other aspects of the program kept constantvariability immediately converged on an intermediate andidentical level in the two components The conclusion isthat operant variability is controlled by discriminativestimuli These findings are important in terms of both ex-plaining observed levels of variability and controllingthem For tasks in which response variations are impor-tant such as when new operants are being shaped or when

an individual is attempting to behave creatively or solveproblems cues that indicate ldquoVaryrdquo can be helpful

Discriminative control without exteroceptive stim-uli The relationship between a response and a reinforcercan itself provide a discriminative stimulus even whenexternal cues such as lights or tones are unavailable Forexample under an FI schedule there is a period duringwhich reinforcers cannot be obtained The subject learns topause following each reinforcement although no cuesother than the reinforcer are provided Another case is themixed schedule identical to the multiple schedule describedabove but with no external cues Animals sometimeslearn to respond differentially to the consequences oftheir behaviors even under such stimulus-impoverishedconditions (Ferster amp Skinner 1957) The question stud-ied next was whether responsendashreinforcer relationshipscan provide discriminative cues sufficient to control lev-els of variability This is an important question becauseas was indicated above it has been assumed that with-holding reinforcement elicits variability The present hy-pothesis is that absence of reinforcement can sometimesserve as a discriminative cue for operant variability Thedistinction between elicited (respondent) and reinforced(operant) variability will be revisited shortly

Hopson Burt and Neuringer (2002) rewarded ratsunder a mixed schedule in which two periods of time orcomponents Var and Repetition (Rep) alternated with-out any external cues In the Var component four-responsesequences of L and R leverpresses were reinforced ifthey met a threshold variability contingency in the Repcomponent only repetitions of LLLL were reinforced(Probabilities of reinforcement were equalized in the twocomponents by intermittent reinforcement of LLLL inRep) The results demonstrated discriminative controldespite the absence of exteroceptive cues (see Figure 3)With trials since the unmarked switch into the Var com-ponent shown in the left column variability increasedand LLLL sequences decreased with trials since the unmarked switch into Rep shown in the right columnresponding became increasingly repetitive until LLLLsequences predominated The two functions were sim-ilar which is an important finding in that it indicatesdiscriminative control rather than elicitation If variabil-ity had been elicited because reinforcement was tem-porarily withheld (when the contingencies were switched)then variability should have increased following entryinto both Rep and Var components but that was not theresult

Therefore variability that is observed when rein-forcement is withheld for short periods as when a newresponse is being shaped may be partly discriminativelycontrolled despite the absence of external cues that isanimals and people may learn when it is functional tovary and some of the cues may come from responsendashoutcome relationships Control over this operant vari-ability may be more precise than that elicited by extinc-tion because as was noted above operant contingenciesdefine both the set of acceptable variations and the level

OPERANT VARIABILITY 679

of required variability It may also be maintained muchlonger because as will be seen shortly extinction weakensresponding

ChoiceThe frequency of one response relative to others or

response probability is used as a measure of choice orpreference Choices are found to match the probabilitiesof their consequences For example relative frequenciesof pecking one of two keys approximate relative fre-

quencies of reinforcement Neuringer (1992) showedthat the same matching relationship also governs choicesby pigeons to vary or repeat The more variations werereinforced the more the pigeons chose to vary and sim-ilarly for repetitions Each trial consisted of four L andR responses and reinforcements for varying and for re-peating were systematically manipulated The Var con-tingency was met whenever the current sequence dif-fered from each of the previous three sequencesmdasha lag 3criterion The Rep contingency was satisfied if the cur-

Figure 3 The left column shows the probabilities ( y-axis) that each of five rats met a variability contin-gency (solid circles) versus a repetition contingency (open circles) when they were being reinforced for vary-ing (Hopson Burt amp Neuringer 2002) Along the x-axis are trials since the mixed schedule (no cues pro-vided to indicate the component) had switched into the variability-reinforcing component The right columnshows the corresponding data for performances when reinforcement was contingent upon repetitions thex-axis here indicating trials since the unmarked switch into the repetition-reinforcing component

680 NEURINGER

rent sequence repeated at least one of the previous threeOn an average of once every 30 sec (VI 30 sec) the com-puter probabilistically primed reinforcement for Var orRep the prime being maintained until the reinforcementhad been collected Thus if the computer primed Var re-inforcement was delivered whenever the lag 3 variabilitycontingency was next met Similarly if Rep had beenprimed the next Rep sequence was reinforced Proba-bilities of Var and Rep reinforcements were systemati-cally manipulated the two always summing to 10 and amatching-type relationship was obtained (Figure 4panel A) The relationship was more complex than sim-ple matching possibly because if the pigeons chose tovary by responding stochastically there was some prob-ability due to chance that the current sequence wouldrepeat a previous sequence This was shown by a com-parison of pigeon performances to the performance of acomputer-simulated stochastic model programmed tomatch its Var choices to the probability of Var reinforce-ments The pigeonsrsquo choice distributions were found tobe the same as the modelrsquos (Figure 4 panel B) Thuschoices to vary or repeat were governed by relative fre-quencies of reinforcement An animal can choose whetherwhen and how much to vary a result consistent withthose described above from Blough (1966) Machado(1989) and Grunow and Neuringer (2002) Choices ofwhether or not to vary appear to be governed similarly toall other operant choices

These f indings may have important applicationsWhen an individual repeatedly responds in a way that isnonfunctional reinforcement of choices to vary mayhelp the individual to emerge from the ineffective pat-tern This can be true for those experiencing everydayproblems but feeling helpless to change and it may beespecially important for those experiencing depressionor who respond in otherwise abnormally stereotypedmanners such as those with autism These topics will beconsidered in a later section

ExtinctionAs was indicated above extinction is often found to

increase variability but almost all studies of this effectinvolve responses that had previously been repetitiveAntonitis (1951) for example reinforced rats for pokingtheir noses anywhere along a 50-cm horizontal openingAlthough the exact location of pokes did not matter withcontinued training responses became limited to one or afew locations along the strip When reinforcement waswithheld variability of location increased (see also Eck-erman amp Lanson 1969) Variability produced by extinc-tion has been reported for many other response dimen-sions as well including force (Notterman amp Mintz1965) number (Mechner 1958) topography (Stokes1995) and sequence (Balsam Paterniti Zechowy ampStokes 2002 Mechner Hyten Field amp Madden 1997)

The issue is complicated however by other studiesthat show that previously reinforced responses and re-sponse patterns are maintained intact during and afterextinction That is extinction does not break down

learned patterns For example when Schwartz (1981) re-inforced pigeons for any sequence of four L and four Rresponses each bird developed a dominant sequence asdescribed above and these sequences were maintainedduring extinction Despite decreasing response ratesduring extinction response patterns were retained Sim-ilar maintenance of response structure has been ob-served during extinction following reinforcement of

Figure 4 Average performance across a group of pigeons inNeuringerrsquos (1992) study of choices to vary or repeat (A) Loga-rithms of the ratio of number of times the pigeons met the Varcontingency divided by the times they met the Rep contingencyas a function of the analogous numbers of reinforcements re-ceived for each Points represent group averages in two separateexperiments Also shown is the least squares best-fitting line(B) Changes in U-value an index of response variability as afunction of the obtained percentage of reinforcements for vary-ing The solid line in panel B represents the performance of acomputer-based model that matched its choices to vary (vs re-peat) to its obtained relative frequency of reinforcement for vary-ing From ldquoChoosing to vary and repeatrdquo by A Neuringer 1992Psychological Science 3 p 248 Copyright 1992 by the AmericanPsychological Society Reprinted with permission

OPERANT VARIABILITY 681

choices (Myerson amp Hale 1988) responding underfixed-interval contingencies (Machado amp Cevik 1998)visual discriminations (Nevin 1967) and matching tosample (Cumming Berryman Cohen amp Lanson 1967)How can stability during extinction be reconciled withincreased variability

Neuringer Kornell and Olufs (2001) wrestled withthis question in a study of extinction following rein-forcement under three different contingencies An oper-ant chamber contained two levers left (L) and right (R)on either side of a reinforcement dispenser and a key (K)in the rear of the chamber distant from the dispenserOne group of rats Rep was reinforced for repeating afixed sequence of responses LKR A second group Varwas reinforced for variations across the three operandaunder a threshold contingency Reinforcement for a thirdgroup Yoke was matched to those of the Var animalsand was therefore independent of the particular sequencesBecause of the unusual configuration of operandaskewed distributions of sequence probabilities were ob-served in all of the groups an outcome that was antici-pated and desired in order to document possible in-creases in variability during extinctionmdasha flattening ofthe distributionmdashas well as possible decreases That isduring the reinforcement phase the Rep group came torepeat the LKR sequence with high probability the Yokegroup tended to emit the easiest sequences such as LLLand RRR but distributed their responses over more se-quences than the Rep group did and the Var group dis-tributed most widely although because of differences inthe operanda and in their locations some sequences weremore preferred than others and in particular sequencesending with a K response were least likely (the key beingfarthest from the reinforcement dispenser) Thus thecontingencies resulted in three different distributions orhierarchies of sequence probabilities very steep forRep intermediate for Yoke and flattest for Var

The main finding was that withholding reinforcementhad similar effects in the three groups In each responserates fell to low levels across the four sessions of extinc-tion Furthermore sequences that were most frequentduring the reinforcement phase continued to be most fre-quent throughout extinction A slight flattening of thedistributions was observed however with the lowestprobability sequences increasing relative to the rein-forcement phase Thus because the overall hierarchy ofsequence frequencies was maintained the results indi-cated stability but because the distributions were some-what flattened the results also showed slightly increasedvariability It is as if the strategy during extinction wereto keep doing what had worked in the past but occa-sionally to do something very different This compli-cated story integrates two effects of extinction increasedvariability and maintained stability Most importantly forthe present discussion extinction had similar effects onvariable yoked and repetitive operants

One additional comparison is of interest Recall thatGrunow and Neuringer (2002) found that when contin-

gencies demanded high variability obtained levels ofvariability decreased slightly as reinforcement frequen-cies were lowered This result is opposite to the Vargrouprsquos slight increase in variability just described underextinction conditions The reason may be that extinction-induced variability is short lived lasting only as long asresponding continues perhaps for hours or a few ses-sions at most whereas the variability engendered by theVI 5 intermittent schedule of reinforcement in Grunowand Neuringer can be maintained indefinitely The de-crease in Grunow and Neuringerrsquos operant variabilitymay be due to the weakening of the reinforcement effect(ie reinforcement is less frequent) whereas the in-crease following extinction may be a short-term elicitedeffect This distinction between reinforced and elicitedvariability is important and will be discussed further inthe next section

Operant and Respondent InteractionsElicited or what is sometimes referred to as respondent

effects occur when environmental changes that affectbehavior do not depend on (are not caused by) the behav-ior Pavlovian conditioning being the most commonlystudied example Operant effects are observed when a re-inforcer depends on the response to produce it Althoughfor historical reasons respondent and operant effects areoften studied separately most behaviors result from acombination or interaction of the two Variability is nodifferent and we turn to an example of such an interaction

As reinforcement is approached under FI or fixed-ratio (FR) schedules response rates are commonly ob-served to increase Similarly when reinforcement de-pends on accuracy as in discrimination learning ormatching to sample it too is found to increase within theFI or the FR (Nelson 1978 Nevin 1967) Cherot Jonesand Neuringer (1996) asked whether the same patternheld for reinforced variability Does operant variabilityincrease as reinforcement for varying is approachedThe question may be related to such real-world cases aswhen a composer is rushing to complete a compositionbefore a scheduled performance or when a programmeris trying to solve a problem before an upcoming meetingThese activities involve at least to some extent non-repetitive original or variable thought and behaviorThe question is whether variability increases with prox-imity to or anticipation of reinforcement for varying

Cherot et al (1996) employed trials consisting of fourresponses across L and R operanda and a lag 4 variabil-ity contingency with rats in one experiment and pigeonsin another Unlike in any of the cases described so farhowever the lag-variability contingency had to be satis-fied four times in order for a single reinforcer to be ob-tained With ldquordquo indicating successful completion of theVar contingency ldquo2rdquo indicating failure to meet that con-tingency and PELLET indicating the food reinforcer anexample follows LRLL RRRL LRLL2 LRLL2RLLR RRRL2 LRRR+PELLET Thus food reinforce-ment was provided only after the subject had satisfied

682 NEURINGER

the variability contingency four times FR 4 The resultswere that as the food reinforcer was approached (ie asprogress was made through the ratio) the subjects wereless and less successful at varying The probability ofmeeting the variability contingency in the first trial afterreinforcement was higher than it was in the second whichin turn was higher than that in the third trial and so onFor comparison a Rep group was reinforced for repeatingsequences once again with reinforcement only after foursuccessful repetitions Probability of repeating increasedwith proximity to reinforcement a finding consistentwith the literature but opposite that for the variabilitygroup Thus approach to reinforcement facilitated oper-ant repetition but interfered with operant variabilitySimilar variability-interfering effects have been docu-mented for within-trials responses The probability that aresponse repeats the just prior response increases within atrial even when sequence variability is reinforced (McElroyamp Neuringer 1990 Neuringer 1991) To return to the ex-ample of a composer rushing to complete her composi-tion she may indeed have increasing difficulty respond-ing creatively under the pressure of a deadline On theother hand the soloist may find it easier to repeat thecorrect performance during practice periods as the con-cert nears How can these results be reconciled with themany findings that variability can be reinforced

An additional result of Cherot et al (1996) provides ananswer The Var animals responded much more variablyoverall than did the Rep animals Although variability inthe Var groups decreased with approach to the reinforceroverall variability was always much higher when it was re-inforced (Var group) than when it was not (Rep group)Reinforcement therefore exerts two influences Overallvariability is elevated sometimes to the highest levelswhen it is reinforced but approach to (and possibly focuson) the reinforcers constrains or lowers variability Sim-ilar reinforcement-produced interference effects had beenreported previously for example when raccoons are re-inforced for dropping a ldquocoinrdquo into a slot the reinforce-ment results in the animalsrsquo holding and rubbing the coinrather than dropping it (Breland amp Breland 1961) Theoverall enhancement of variability was of much greatermagnitude in Cherot et al than the decrease with approachwas although both effects were statistically significant(The notes created by the composer under pressure ofdeadline will be less readily predicted than will thoseplayed by the performer practicing a well-known piece)This study highlights the importance of interactions be-tween operant effects of reinforcement and elicited effectsproduced by anticipation of reinforcement and the resultsmay be relevant to the debate concerning possible detri-mental effects of reinforcement on creative behaviors (seethe discussion below and Neuringer in press)

SummaryReinforcement controls behavioral variability as it

does other operant dimensions Levels of variability areprecisely governed by schedule contingencies and dis-

criminative stimuli Choices to vary or repeat match rel-ative frequencies of reinforcement Extinction increasesvariability slightly whereas the hierarchies of sequenceprobabilities are retained and operant and respondenteffects interact The evidence shows that behavioral vari-ability is an operant controlled by its consequences

GENERALITY AND PARAMETERS

Many different methods measures species and re-sponses have been used in the study of operant variabil-ity and this diversity supports the generality of the phe-nomenon As is the case for repeated operants howeverthere are parametric constraints This section briefly re-views the generality and some of the parameters studiedto date

GeneralityMethods Throughout this section ldquoresponserdquo will

signify a response unit including both an individual re-sponse and a multiresponse sequence Each of the meth-ods to be described increases or maintains variability byreinforcing it

Novel response procedures reinforce responses upontheir first occurrence ldquoeverrdquo (Pryor et al 1969) or upontheir first occurrence within a session (Goetz amp Baer1973) Radial-arm maze procedures also reinforce ini-tial (within a given session) entries into arms of a mazeand result in a distribution of choices across the availableoptions the sequence or pattern of choices often beingquite variable

Lag procedures reinforce a response if it has not oc-curred across a given number of previous trials (Page ampNeuringer 1985) For example under lag 5 the currentsequence is reinforced only if it has not occurred duringany of the previous five trials A variant is the recurrence-time procedure which specifies a minimum number of in-tervening trials before a previously emitted response canagain be reinforced Machado (1989) combined recurrencetime with a percentile-reinforcement contingency to gen-erate high levels of variability (see also Machado 1992)Percentile-reinforcement contingencies base the criterionfor reinforcement on the subjectrsquos own performance over aprevious set of responses (see Galbicka 1994) For exam-ple a sequence will be reinforced only if it has not oc-curred over the previous n trials the value of n continuallychanging so that 50 of the trials are reinforced

Least-frequent response procedures (Blough 1966Schoenfeld Harris amp Farmer 1966 Shimp 1967) rein-force responses that are currently least frequent across amoving window

Threshold procedures reinforce responses whose rela-tive frequencies fall below a given value Because thistype of procedure was used in the research describedabove and in many other more recent studies details willbe provided In Denney and Neuringer (1998) which willserve as an example trials consisted of four responses byrats across L and R levers A running tally was kept of the

OPERANT VARIABILITY 683

number of occurrences of each of the 16 possible se-quences LLLL LLLR LLRL and so on If the relativefrequency of the current sequencemdashthe number of its oc-currences divided by the sum of all 16 sequencesmdashwasbelow a specified threshold value in this case 05 the ratwas rewarded Recently emitted sequences contributedmore to the maintained tally than did those in the distantpast because after each reinforcement all 16 counterswere multiplied by a weighting coefficient in this case95 Therefore the influence of a particular trial decreasedexponentially with successive reinforcements Thresholdprocedures differ from Bloughrsquos (1966) least-frequentprocedure in that more than one sequence (or response)can be reinforced at any given timemdashnamely all those se-quences that have relative frequencies below the thresholdvalue The procedure differs from lag in permitting agiven sequence to be reinforced many times in successionif the associated relative frequency is sufficiently low

The frequency dependence procedure (Machado 19921993) is a variant of threshold in which the probability ofreinforcement is a continuous function of relative re-sponse frequency The more frequent a response is thelower the probability is that it will be reinforced

Statistical feedback procedures provide subjects withcomparisons between their performances and that of amodel Neuringer (1986) showed human subjects feedbackfrom 10 statistical tests thereby enabling them to comparetheir own distributions of responses to that of a randommodel Neuringer and Voss (1993) provided feedbackbased on a chaotic model The goal in each of these caseswas for the subjects to learn to approximate the model andthe results of these experiments will be discussed below

Differential reinforcement of switches versus repetitionsbalance a subjectrsquos tendency to engage in each of these pat-terns Rather than reinforcement for infrequent responsesor random-like distributions reinforcement is providedsometimes for repeating a response and other times forswitching from one response to another In the two-operandum case a switch is L followed by R or R by La repetition is L then L or R then R Bryant and Church(1974) found that stochastic-like responding was gener-ated by rats when switches were reinforced with a 75probability and repetitions with a 25 probability that isthe rats had a natural tendency to repeat at least in theenvironment of this study Machado (1997) reinforcedpigeons for eight pecks across two keys if the sequenceof pecks contained at least one (one group of birds) ortwo (another group) switches Although the birds couldhave satisfied these contingencies by repeating a singlesequencemdashin the ldquoone switchrdquo condition for examplerepetition of LRRRRRR would produce reinforcement on100 of the trialsmdashthey tended to vary their sequencesBarba and Hunziker (2002) showed however that vari-ability was higher when it was directly reinforced thanwhen reinforcement was based on proportions of switchesa finding consistent with that of Machado (1997)

The evidence from each of these methods supports thehypothesis that variability can be reinforced

Measures Variability must be interpreted with refer-ence to a particular measure The reason is that there canbe high variability (and low predictability) at one level ofanalysis but order (and certainty of prediction) at a dif-ferent level For example the sequence 1 1 2 2 3 3 44 5 5 1 1 2 2 has the same variance (a commonlyemployed measure of variability) as 2 3 2 1 5 1 4 41 1 3 2 5 2 but an analysis of conditional proba-bilities in which the unit contains pairs of instances in-dicates that the two sequences have different levels ofvariability In general high variability observed with onemeasure does not rule out the possibility of order ob-served with another and vice versa That is why evalua-tions of random-number generators often employ manydifferent measures and indeed there are hundreds of dif-ferent measures of the randomness of a sequence (Knuth1969) All behavioral studies make simplifying assump-tions and rely on one measure or at the most a few

U-value has been the most commonly employed mea-sure of operant variability (Evans amp Graham 1980Machado 1989 Page amp Neuringer 1985 Stokes 1995)U-value measures the distribution of relative frequen-cies or probabilities of a set of responses For a set of16 possible responses i = 1ndash16 U-value is given by

where RFi = the relative frequency (or probability) ofeach of the 16 responses U-values approach 10 whenrelative frequencies approach equality as would be ex-pected over the long run from a random generator andthey approach 00 when a single instance is repeated

Other measures include the percentage of trials or fre-quency at which variability contingencies are met (Page ampNeuringer 1985) percentages of alternations versus stays(Machado 1992) conditional probabilities of responses(Machado 1992) the number or frequency of novel re-sponses (Goetz amp Baer 1973 Schwartz 1982) the num-ber or frequency of different responses (Machado 1997Schwartz 1982) Markov analyses (Machado 1992) andstatistical tests of distributions of responses (Neuringer1986) All of the measures employed to date indicate thatvariability can be reinforced

Responses and species Generality has further beendemonstrated by the use of many different responses andspecies These include Siamese fighting fish (Betta splen-dens) swimming in quadrants of a tank (Roots 2000)young chicks pecking keys (Neuringer 2002) pigeonspecking keys (Page amp Neuringer 1985) budgerigarssinging songs (Manabe Staddon amp Cleaveland 1997)rats pressing levers (Van Hest van Haaren amp van de Poll1989) porpoises swimming and jumping (Pryor et al1969) rhesus monkeys pressing levers (Sydney Reisbickpersonal communication) children drawing and build-ing with blocks (Holman Goetz amp Baer 1977) andhuman adults and children pressing buttons (Saldana ampNeuringer 1998 Stokes amp Balsam 2001) There havebeen few attempts to compare species or responses but

- ( )[ ] ( )=aring RF RFi ii

log log 2 21

1616

684 NEURINGER

in one study lowest baseline variability was observedwhen rats pulled trapezes intermediate levels were ob-served when they pushed keys and highest levels wereobserved when they pressed levers (Morgan amp Neuringer1990) with variability increasing in all cases when it wasreinforced In another study operant variability wascompared across rats and people with similar effects ob-served in the two species (Neuringer Deiss amp Imig 2000)

ParametersWe turn now to some parameters studied to date Many

other parameters as yet unstudied are no doubt also im-portantmdashfor example motivation and types of reinforcers

Age Older people (see eg Van der Linden Beertenamp Pesenti 1998) and older rats (see eg Lamberty ampGower 1990) respond less variably than their youngercounterparts when variable responding is reinforced or re-quested but in most cases it is not clear to what extent thevariability is elicited as opposed to reinforced Neuringerand Huntley (1992) compared reinforced variability in2-month-old (young) and 10-month-old (mature) Long-Evans rats Under Yoke contingencies the young rats re-sponded significantly more variably (two levers four re-sponses per trial) than the mature rats which constitutesan elicited effect When reinforcement was contingenton variability (lag 4) variability increased significantlyin both groups thereby overcoming the initial differencecorrelated with age Thus explicitly reinforcing vari-ability in older animals caused them to respond no dif-ferently than younger animals did a finding potentiallyrelevant to real-world human aging However when Wells(1999) compared 8-month-old (mature) to 24-month-old(old) rats although the mature rats responded more vari-ably than the old rats did under Yoke contingencies thedifference was not statistically significant Additionalstudies are needed especially of whether reinforcingvariable behaviors in older populations can facilitatelearning and performance

Gender Male and female rats are equally sensitive toreinforcement of variability (Neuringer amp Huntley 1992van Hest et al 1989) However there was a tendency(not statistically significant) for males to respond over-all more variably than females did and because thesestudies did not control for differences in size and weightof subjects or for other aspects of motivation additionalstudies are necessary

Stage of training Stokes and Balsam (2001) reporteddifferences in variability as a function of whether variabil-ity was initially reinforced early or later in training (seealso Gottlieb 1992) Human subjects generated sequencesof 10 L and R responses per trial on a computer keyboardunder lag 0 and lag 25 contingencies Under lag 0 everytrial ended with reinforcement under lag 25 the currentsequence had to differ from each of the previous 25 Whenlag 25 was experienced at the very beginning of traininglevels of variability at the end of training (when lag 0 wasoperative) were lower than they were when lag 25 was ex-perienced after 50 trials of initial lag 0 These results were

interpreted to suggest that there is an ldquooptimal period forsetting variability levelsrdquo (Stokes amp Balsam 2001 p 181)but that general conclusion needs further testing and anumber of questions need to be answered One has to dowith the contribution of reinforcement frequency Underlag 0 every sequence leads to reinforcement but rein-forcement is intermittent under lag 25 Another has to dowith whether learning to vary differs from learning to repeatwith respect to optimal periods (see eg Bouton 1994)

Related to stage of training are effects of prior experi-ence Hunziker Lee Ferreira da Silva and Caramori(2002) showed that for human subjects if Var contingen-cies were experienced just prior to Yoke contingencieslevels of response variation during the Yoke phase werehigher than they would have been if the Var contingencieshad not been previously experienced The time delay be-tween Var and Yoke experiences mattered with a long in-terval (3 months) producing only a small effect Saldanaand Neuringer (1998) also found effects of prior Var train-ing on performances in Yoke (increased variability inYoke) but the opposite did not hold that is training underYoke had relatively little effect on Var Hunziker Caramorida Silva and Barba (1998) reported similar effects withrats Thus there may be an asymmetry Reinforcement forvarying has relatively strong influences but reinforce-ment without regard to variations does not adversely affectlearning to vary These different effects are not surprisinggiven that Yoke contingencies are permissivemdashreinforce-ment is given whether or not responses varymdashwhereascontingencies under Var can be quite demanding How-ever the results motivate a number of additional ques-tions One is whether effects of Var training differ fromthose observed when repetitions are reinforced Another iswhether direct reinforcement of variable behaviors canhelp animals or people to overcome the detrimental effectsof prior noncontingent experiences such as those that pro-duce learned helplessness (Seligman 1975)

Environmental constraints Stokes (1995) arguedalso that variability increases when behavior is con-strained The term ldquoconstraintrdquo is ambiguous because allreinforcement contingencies exert a form of constraintIn one study Stokes (1995) compared different strate-gies of shaping leverpresses Rats in one group were re-inforced initially for pressing the lever with only theright paw (right only) and eventually for any press (withright or left paw or with any other part of the body) Theorder of training was reversed for a second group ldquoanyrdquofirst and then right only Behavior sequences were morevariable in the right-only-first group leading Stokes(1995) to conclude that early constraints lead to highvariability (see also Stokes Mechner amp Balsam 1999)That is an important hypothesis with many implicationsbut controls are needed before the results can be attrib-uted to the contingencies For example ldquoright onlyrdquo andldquoanyrdquo differed in terms of reinforcement distributions aswell as in the precision of the reinforcement operationthe ldquoanyrdquo reinforcers were provided immediately and au-tomatically by the programming equipment whereas in

OPERANT VARIABILITY 685

the right-only condition a human observer had to decidewhether a response had occurred and then present the re-inforcer necessarily leading to longer and more variedlatencies and a less consistent contingency Furthermorethere are many cases in which increasing constraints suchas those due to a decrease in the number of responses pertrial (Page amp Neuringer 1985) or the requirement of ex-actly four responses on each of two keys (Schwartz 1980)results in decreased variability Thus the contributionsof response-definition constraints and environmentallimitations must be explored further

SummaryThe generality of operant variability is shown by the

robustness of the phenomenon across procedures mea-sures responses and species As with other operant phe-nomena parameters influence the effects

FUNCTIONS

All operants including variability are functional Op-erant responses produce reinforcing consequences Be-cause other behavioral competencies depend on it oper-ant variability is functional in a broader sense as wellSome of these functions will now be described

Operant ConditioningThe potential of a reinforcer to change behavior is lim-

ited by the range of available variations (Hull Langmanamp Glenn 2001 Skinner 1981) and thus whatever pro-duces those variations contributes to the conditioning ofnew operants Much work has been done to identifysources of variation (Balsam Deich Ohyama amp Stokes1998 Segal 1972 Staddon amp Simmelhag 1971) butNeuringer (1993) was first to show that the same rein-forcers that strengthen individual responses can also pro-duce the necessary variability and that these two functionsof reinforcement may work simultaneously Implied isthat when an operant response is being conditioned re-inforcement works not only to strengthen an individualresponse but at the same time to engender the variabilityfrom which the response emerges (That is variability isnot solely a baseline phenomenon elicited or noise)Concurrent reinforcement of variations and individualtarget sequences will be the topic of this section

Pigeons in one experiment and rats in another were re-warded for varying among 16 possible response sequences(four responses per trial L and R operanda lag contin-gencies Neuringer 1993) Concurrently one of the 16designated as the target sequence was reinforced when-ever it occurred Thus reinforcements were given bothfor sequence variations and for a single selected targeta reinforced-variation-and-selection (RVS) procedureWhen the target was ldquoeasyrdquo (eg LLLL) it emergedrapidly from the baseline variations Given that the tar-get was always reinforced reinforcers were obtained morefrequently when the subjects did not vary and only re-peated the target When the target was ldquodifficultrdquo (eg

LLLR) it was not learned As in the choice experimentdescribed above (Neuringer 1992) the relative valuesor difficulties of target versus variations governed theprobability that the target would be learned

However even difficult targets could be conditionedunder the RVS procedure as was shown by another ex-periment in the same series (Neuringer 1993) Sequencevariations were again reinforced concurrently with a dif-ficult target RRLR but now the frequency of reinforce-ments for varying was systematically decreased Satis-fying the variability contingency was intermittentlyreinforced according to a VI schedule and over sessionsreinforcement for varying became less and less frequentThis technique resulted in acquisition of even the mostdifficult target sequence whereas a control group rein-forced for the target alone (without the concurrent rein-forcement for variations) never learned

Reinforcers were obtained more frequently in the RVScondition than in the control condition (since the RVSsubjects were rewarded both for varying and for emittingthe target sequence) and the additional reinforcers mighthave generated greater motivation to respond (Killeen1981) Thus motivation not the direct reinforcement ofvariability might have been responsible for the RVS ef-fect To test this possibility Neuringer Deiss and Olson(2000) reinforced three groups of rats for emitting a par-ticularly difficult target sequence LLRLR The experi-mental group was reinforced concurrently for sequencevariability on the average of once per minute (VI 1 min)The target-only control group received no reinforce-ments other than for the LLRLR target A third groupwas yoked to the experimental animalsmdashreinforced onceper minute for any sequence variability not requiredmdashand as for the other two groups whenever the target oc-curred If the additional reinforcement was responsiblefor the facilitated learning under RVS then because theexperimental and yoked groups received equal reinforce-ments they should have learned equally well but thatwas not the result Only the experimental group learnedto emit the LLRLR target The yoked controls respondedat high rates throughout the experiment but they did notvary sufficiently to make contact with the target contin-gency and never learned The target-only control groupreceived insufficient reinforcement and their responsesextinguished Therefore the added reinforcers helped tomaintain motivation as suggested above but reinforce-ment of variations was necessary for learning of a diffi-cult sequence The RVS procedure maintains both highmotivation and high variation and the two together re-sult in the acquisition of difficult-to-learn responses

Grunow and Neuringer (2002 Experiment 2) studiedone possibly important parameter of the RVS proceduremdashnamely the level of variability When a difficult target re-sponse is being trained does it matter whether reinforcedvariability is low intermediate or high As describedabove each of four groups of rats was reinforced for vari-able response sequences the first group for high vari-ability the fourth group for low variability and the other

686 NEURINGER

two for intermediate levels In other ways the procedurewas similar to that of the Neuringer Deiss and Olson(2000) RVS contingencies just described with a singledifficult target sequence always reinforced The resultswere that the higher the level of reinforced baseline vari-ability the more quickly the target was learned Thusvariability levels indeed mattered and if the just-re-ported effect is found to be general it will be most help-ful to reinforce high variability when difficult-to-learnnew responses are being shaped Other parameters re-quiring study are class definitionsmdashwhich responses areincluded in the class of acceptable variationsmdashand num-ber of instances within the class

One additional effect should be noted When varia-tions among a set of responses are reinforced if one vari-ant is selected for extinction (no longer reinforced) thefrequency of that instance decreases Neuringer (1993)showed this for pigeons responding under a lag-variabilityschedule When one of the 16 sequences was no longerreinforced (with no other change to indicate its status) thefrequency of that sequence decreased Thus reinforce-ment of variations provides the baseline to select for (iestrengthen) and select against (ie weaken) individualinstances

These findings have potentially important implicationsFirst as has been indicated the same reinforcers thatstrengthen particular instances can also reinforce varia-tions That a binary eventmdashreinforce or not reinforcemdashsimultaneously affects variations and selected instancesindicates both the power of reinforcement and a concep-tual problem How reinforcers exert such diverse effectsis a question to which we will return

Second shaping procedures may implicitly reinforcebaseline variations An approximation to a goal responseis reinforced for a period of time but eventually rein-forcement is withheld until a response closer to the goalis obtained It is commonly thought that withholding re-inforcement or mini-extinction induces the necessaryvariations for shaping to proceed as has been indicatedabove An alternative suggested by the present studiesis that organisms learn both to vary (within a set) and to re-peat (selected instances) until one instance is always rein-forced That is the process of shaping might involve re-inforcement both of variations (generally done implicitly)and of targeted instances If so a useful contributor tothe shaping process might be explicit reinforcement ofvariations

Third reinforcement of variations could facilitate ac-quisition of other competencies such as cognitive andmotor skills Siegler (1996) has shown in an important se-ries of studies that childrenrsquos acquisition of complex cog-nitive skills is correlated with high variability Anotherexample is that when motor skills are practiced in a vari-able manner high quality of performance results (Ma-noel amp Connolly 1995 1997 Mechner 1992 Schmidtamp Lee 1999) However it is rare that variability in thesedomains is directly reinforced as part of the training oreducational process

Fourth the RVS procedure might be used in therapeu-tic settings to modify maladaptive and self-injurioushabits Therapists sometimes employ noncontingent re-inforcement procedures to weaken undesirable behaviors(Vollmer Iwata Zarcone Smith amp Mazaleski 1993)The reasoning is that many maladaptive behaviors aremaintained by unintended consequences and that non-contingent rewards may compete with those conse-quences thereby lowering the strength of the unwantedhabit For example when a staff personrsquos attention toself-injurious behavior helps to maintain it providing at-tention noncontingently may decrease the strength of theunwanted target behavior However the RVS proceduremight accomplish the same goal more effectively andwithout undesirable side effects (Vollmer RingdahlRoane amp Marcus 1997) Noncontingent rewards some-times result in unwanted ldquosuperstitiousrdquo behaviors andthat would be avoided under RVS Furthermore RVSwould help to provide the set of baseline behaviors nec-essary to establish desirable new responses

Finally RVS procedures may help to gain control overand eventually to weaken other undesirable target behav-iors The target can be incorporated within a set of moredesirable responsesmdasha new operant class established withthe target as one membermdashand variable instances rein-forced Once variable responding is successfully main-tained including the undesirable target response the lattercan be weakened by selectively withholding reinforcement

Summary The conditioning of operant responses de-pends on both baseline variations and selection of suc-cessive approximations to a goal response The same re-inforcers used to select the approximations can reinforcebaseline variations thereby defining the precise set ofappropriate variants and the desired levels of variationSuch concurrent reinforcement of variations and selec-tions may facilitate the acquisition of difficult-to-learnoperants and selective withholding of reinforcementmay weaken undesirable instances

Creativity and Problem SolvingBehaving in a novel unusual or nonrepeating manner is

part of creative activities (Campbell 1960) Novelty andvariation may arise through combinations of previouslylearned behaviors (Epstein 1996) transformations of thesebehaviors (Boden 1994) direct reinforcement of variabil-ity as described above or some combination thereof Ofcourse neither novelty nor variability alone suffices forcreative outputmdashthe throw of the dice or the toss of acoin are not creative nor are haphazard strokes of a paint-brush on a canvasmdashbut without variation creativity islow Whether the variability required for creativity canbe reinforced is relevant to whether creativity itself canbe reinforced The latter question has led to contentiousdebate with some arguing that reinforcement facilitatescreativity (Cameron amp Pierce 1994 Eisenberger ampCameron 1996 Stokes 2001 Winston amp Baker 1985)and others focusing on detrimental effects of reinforce-ment (Amabile 1983 Deci Koestner amp Ryan 1999)

OPERANT VARIABILITY 687

The debate goes well beyond the present discussion butsome of the research described above is relevant Firstreinforcement contingent on variability generates higherlevels of behavioral variability than does reinforcementcontingent simply on responding as has been shown bythe many studies described above in which Var contin-gencies were compared to Yoke contingencies Similarlyprecise reinforcement contingencies matter for creativity(Cameron amp Pierce 1994 Lepper amp Henderlong 2000)If any product rather than only creative ones is rein-forced creativity tends to fall (just as variability decreasesunder yoke conditions) Second approach to or antici-pation of reinforcement interferes with operant variabil-ity (Cherot et al 1996 McElroy amp Neuringer 1990Neuringer 1991) which again might be related to effectson creativity when an individual focuses on the rein-forcer Note however that although variability decreaseswith proximity to the reinforcer overall levels of vari-ability are much higher when variability is reinforced thanwhen it is not The same may be true for creativity aswell anticipation of reward may interfere with creativeoutput to some extent but at the same time high creativ-ity may depend on its reinforcement

Operant variability may be important for problemsolving as well Here too there is disagreement as towhether training is helpful or not Early studies by Maier(1933) indicated that it is and these found support inlater work (eg Lung amp Dominowski 1985) Somestudies showed that those individuals most likely toldquobreak the setrdquo are also most likely to solve a problem(Dover amp Shore 1991) However a number of studiesfound that practice had no influence (Duncan 1961)

Arnesen (2000) explored a related phenomenon witha rat model One group Var was reinforced for variableinteractions with novel objects and another group Yokewas reinforced independently of such interactions EachVar rat received training with 12 different objects oneper session For example in the presence of a soup canthe rat was reinforced the first time it entered the canbut thereafter a different response was required such asrolling it climbing on it and so forth This procedure issimilar to that of Goetz and Baerrsquos (1973) described aboveexcept that objects changed from session to session Ratsin the Yoke control group had identical experiences in anadjoining identical chamber that contained identical ob-jects but now rewards were not contingent on object ma-nipulation After this training the rats were tested in adifferent room with 30 never-before-experienced ob-jects scattered on the floor Hidden in each was a smallpiece of food When compared to their Yoke controls theVar animals were more active during this test exploredmore manipulated objects more and most importantlydiscovered significantly more food pellets Another con-trol group received no preliminary training with objectsand their performance in the test session did not differfrom that of the yoked controls Therefore when ratswere reinforced for variable interactions with objectsthe result was an increased tendency to explore manip-ulate and discover new sources of reinforcement It

would be of considerable interest to determine whethera similar effect occurs with children

Assessment and Treatment of PsychopathologiesAbnormal levels of behavioral variability characterize

some human psychopathologies as has been shown byclinical observations as well as by research with humansubjects and animal models Few studies have attemptedto reinforce variability directly in such problem popula-tions The main question is whether reinforcement canmodify abnormal levels of variability in the direction ofnormalcy and whether such changes might have associ-ated beneficial effects

Autism Stereotyped responding characterizes indi-viduals with autism (Hertzig amp Shapiro 1990) For ex-ample Baron-Cohen (1992) asked children with autismand others in a control group to hide a penny in one handso that the experimenter could not guess which hand heldthe penny Those with autism were more likely thanthose without to generate a simple predictable patternsuch as repeatedly switching back and forth from left toright hands Such repetitive responding can be maladap-tive For example Mullins and Rincover (1985) askedchildren with and without autism to pick one of five cardsFood was occasionally found in a cup behind the cardEach of the five cards was a discriminative stimulus fora different schedule of reinforcement including contin-uous reinforcement (CRF every choice reinforced) FR 2(fixed ratio 2 in which every second choice was rein-forced) FR 4 (every fourth choice reinforced) FR 7 andFR 11 Control participants sampled all five alternativesand quickly learned to choose the most frequently rein-forced CRF card Autistic children sampled only a lim-ited number of cards and therefore often preferred anonoptimal alternative Thus children with autism tendto respond less variably than normal children and thiscan result in failure to adapt

Miller and Neuringer (2000) asked whether direct re-inforcement could influence the variability of individu-als with autism Sequences of responses on two largebuttons were first reinforced probabilistically and inde-pendently of variability (Prob phase) after which rein-forcement was contingent upon satisfying a thresholdvariability contingency (Var phase) The autistic partic-ipantsrsquo variability was significantly lower than that ofmatched controls during both Prob and Var phases butwhen variability was reinforced its levels increased sig-nificantly in both groups This is an important findingbecause it suggests that direct reinforcement of variabil-ity may help to modify some of the behavioral stereo-types that are characteristic of autism Lee McComasand Jawor (in press) offer support for this hypothesisThree individuals with autism were reinforced for variedverbal responses to social questions the responses had tobe appropriate given the context Variability increased intwo of the three individuals In a related procedureBaker and Koegel (1999) reinforced children with autismfor varied social interaction in a game-playing contextAlthough the reinforcement was permission to engage in

688 NEURINGER

the stereotyped behaviors preferred by the child the re-sult was increased social interactions both during theperiod of the experimental manipulation and in 1- and2-month follow-ups These three studies taken togethersuggest that reinforcement of varied behaviors may facil-itate modification of nonfunctional ritualistic and stereo-typed behaviors

Depression Depression provides another example ofmaladaptive consequences of low variability Lapp Mar-inier and Pihl (1982) observed that depressed womenproduced fewer alternative solutions to hypotheticalproblems than did nondepressed women Channon andBaker (1996) found that attempts by moderately de-pressed college students to identify faults in a series ofinterconnected circuits were less varied and less success-ful that were those of nondepressed controls HorneEvans and Orne (1982) found that when depressed pa-tients were asked to generate a random sequence of num-bers their responses were less variable than were thoseof controls These experimental results are consistentwith clinical observations

Can depressed individuals be rewarded for increasingvariation Hopkinson and Neuringer (in press) dividedcollege students into mildly depressed and nondepressedgroups on the basis of the CES-D scale a paper-and-pencil self-evaluative index of depression In the first oftwo phases sequences of responses on a computer key-board were reinforced independently of variability (Prob)following which only highly variable sequences were rein-forced (Var) Under Prob depressed students respondedsignificantly less variably than did the controls in agree-ment with previous findings Under Var variability in-creased significantly in both groups Indeed by the endof the experiment depressed and nondepressed subjectswere responding equally variably (Instructions were alsoeffective in increasing variability in the depressed par-ticipants) Thus even in a simple computer-game envi-ronment mildly depressed subjects manifested lowervariability than controls did but direct reinforcementmodified their behavior to control levels Can reinforce-ment of variability be used in real-world conditions tohelp depressed individuals An affirmative answer comesfrom cognitive behavior modification procedures thatwork to increase alternative solutions to problems (Beck1976) A more general conclusion from these studies ofautistic and mildly depressed individuals is that direct-reinforcement-of-variability procedures may benefitthose who manifest abnormally low levels of variability

Attention Deficit Hyperactivity Disorder (ADHD)ADHD may provide an example of the opposite stateabnormally high levels of variability (Barkley 1990)Wultz Sagvolden Moser and Moser (1990) proposedthat the spontaneously hypertensive rat (SHR) may serveas a good animal model of ADHD the SHR shows manyof the behavioral characteristics of that disorder includ-ing hyperactivity poor learning ability poor ability totolerate delayed reinforcement and poor self-control

The next study to be described asked about operant vari-ability in SHRs

Mook Jeffrey and Neuringer (1993) compared vari-ability of SHRs and of a control strain Wystar Kyoto(WKY) under two conditions Prob and Var The mainresult was that SHRs tended to respond variably whetheror not the contingencies required it consistently with theresults from the human ADHD literature When vari-ability was reinforced the two strains responded at ap-proximately equally high levels of variation These re-sults were confirmed in two other studies (HunzikerSaldana amp Neuringer 1996 Mook amp Neuringer 1994)Thus control rats varied only when the contingencies re-quired variations whereas SHRs varied whether or notthe contingencies required it As might be expected fromthese results when reinforcement depended on repeti-tions the SHRs performed poorly (Mook et al 1993)

Another similarity between SHRs and ADHD humanswas seen when amphetamine a drug that has effects simi-lar to those of Ritalin often prescribed for ADHD was ad-ministered (Mook amp Neuringer 1994) When repetitionswere required for reinforcement amphetamine improvedperformances by the SHRs so that they were as accurate assaline-injected WKYs a finding again consistent withthose in the ADHD literature These studies support SHRsas a model of ADHD and suggest that additional attemptsshould be made to assess whether or not individuals withADHD are less able to change levels of variability thanare controls Although as indicated above there is someevidence in support of this hypothesis (Barkley 1990)when Saldana and Neuringer (1998) directly comparedhuman ADHD and control participantsrsquo operant variabil-ity within the context of a computer game no differenceswere observed Most of those with ADHD were main-tained on Ritalin and other drugs however a fact unfor-tunately not documented in the original article

Drug EffectsAlthough many drugs are known to influence behavioral

variability (Brugger 1997) rarely have attempts beenmade to analyze explain or identify the sources of theeffects As will be discussed below operant variabilitycan result from the memory-based process of remember-ing previous responses and then behaving differently orcan result from responding stochastically a process thatappears to be quite different McElroy and Neuringer(1990) attempted to distinguish the effects of alcohol onmemory from its effects on stochastic processes in astudy in which one group of rats was reinforced for vary-ing four-response sequences under a lag 5 contingency(Var) and another was reinforced for repeating a singlesequence LLRR (Rep) Ethanol injections of 075 and20 gkg affected Var and Rep performances differentlyWhereas the Rep animals were much less able to satisfythe LLRR contingency at both doses the Var rats re-sponded variably whether or not alcohol had been ad-ministered Similar results were obtained by Cohen et al

OPERANT VARIABILITY 689

(1990) with a multiple schedule in which each rat servedas its own control In the same animals and within thesame sessions alcohol degraded Rep performances butleft Var intact Thus alcohol appears to interfere with atask that requires working memory as presumably is thecase for the Rep task but not with one that reinforcesvariability The memory-interfering effect of alcoholmight account for why alcohol also degrades radial-armmaze performance (Devenport 1983)

Alcohol also affects choices to vary or to repeat For ex-ample Gorka (1993) reinforced rats concurrently for re-peating and varying and monitored percentage of choicesas a function of time following injection of ethanol Inthe control sessions when the rats were injected withsaline the probabilities of varying and repeating remainedapproximately constant across the 45-min session witha slight bias in favor of varying After injection with 15gkg of ethanol however there was a significant increasein choices to vary at the beginning of the session andthen a later decrease to below 50 That is alcohol ledinitially to a markedly increased preference for varyingbut later to a preference for repeating (Figure 5) This re-search should be replicated and extended especially be-cause the contingencies ldquopressuredrdquo the animals to re-main at approximately 50 preference but thesuggestion is that alcoholrsquos effects on response variabil-ity may depend in part on the duration of its influence Amore general implication is that drugs may affect re-peated and variable operants differently and that effectsdepend on time following administration

SummaryReinforced variability may contribute to many types

of learning including operant shaping and acquisition ofcognitive and complex motor skills Similarly creativityand problem solving depend on variable responding andin these areas as well explicit reinforcement of variationsmay be functional Abnormal levels of variability aresometimes caused by psychopathologies and drugs andoperant variability may be useful in these areas in twoways first as a way of assessing the variability that is pos-sible and second to modify variability toward more nor-mal levels Studies show that individuals with autism andmild depression vary less than do controls but that ex-plicit reinforcement of variability moves the levels towardsnormalcy Other studies show that drugs such as alcoholmay affect repeated and variable operants differently

EXPLANATIONS

Two questions must be answered in order to explainoperant variability First what are the sources of vari-ability that permit its reinforcement Second how is itmaintained Related questions including how instancesbecome members of a set from which variations emergeand whether operant variability is a generalizable com-petency will also be considered

Origination From Nonoperant SourcesSkinner (1966) reasoned that operant responses orig-

inate from nonoperant sources with reinforcement con-tacting a behavior shaping and strengthening it onlyafter the behavior had occurred for reasons other than re-inforcement To put it simply a behavior must occur be-fore it can be reinforced (Balsam amp Silver 1994) I willconsider three initial sources of variable behavior en-dogenous variability noncontingent environmental eventsand extinction These appear to parallel evolutionaryprocesses and they too will be identified

Endogenous variability All physical phenomenavary and behavior is no exception This fact can providethe opportunity for reinforcement to select particularlevels of variation Examples of spontaneous variabilityinclude spontaneous alternations in mazes (Dember ampRichman 1989) nondifferentially reinforced variationsin operant responses (Guthrie amp Horton 1946) and ex-ploration (Archer amp Birke 1983) Endogenous muta-tions provide a parallel in genetics Base-pair changes inDNA molecules occur spontaneously intron-bearinggenes mix and match segments to encode new proteinsjumping genes introduce new DNA sequences and atthe level of the chromosome random crossings and as-sortment occur as a natural part of reproductive meiosisThese sources of random genetic change occur indepen-dently of external influences

Figure 5 The relative frequency of variations by rats as a func-tion of 5-min blocks of time since the beginning of the sessionThe open circles represent performance after the rats were in-jected with saline indicating a slight tendency to vary more thanrepeat but one that is approximately constant throughout thesession The filled circles represent performance after injectionwith 15 gkg ethanol indicating a much higher likelihood ofvarying soon after the alcohol is injected followed by a decreasein variations to below 50 and then a return to baseline at theend of the session Group averages are shown Data are fromGorka (1993)

690 NEURINGER

Noncontingent events Noncontingent events some-times referred to as chance accidents or luck also affectbehavior One happens upon a particular passage in a bookthat leads to a new thought one happens to sit next to aninteresting woman or man on an airplane which leads tomarriage (Bandura 1982) In science as well as in every-day life accidents or serendipity are an important part ofthe process of discovery (Beveridge 1957) In each of thecases presented above an accidental event leads to new be-haviors that are then strengthened by consequences Thesame is potentially true for variability itself Particular lev-els of variability might occur for adventitious reasons butthen become functionally related to consequences

That organisms are particularly sensitive to such unan-ticipated noncontingent events is shown by two basicprinciples of learning habituation and Pavlovian condi-tioning Organisms attend preferentially to unexpectedevents and stop attending or habituate to repeated onesThe conditioning of new Pavlovian responses dependsaccording to influential theories upon unexpected rein-forcing events (eg Rescorla amp Wagner 1972)

Noncontingent events such as X-rays and other pro-cesses that can act as mutagens play an important role ingenetic variation as well The particular genes affectedand the type of changes are random there is no neces-sary relationship or contingency between the externalevent and the affected gene However sensitivity to mu-tations is not random but has been selected presumablybecause it is functional Thus in both behavior and ge-netics sensitivity to noncontingent events is conserved

Extinction Extinction increases behavioral variabilityat least to some extent as was described above Distance inspace or time from the reinforcer does the same (Cherotet al 1996) A possible parallel at the genetic level in-volves high levels of deprivation at least in simple lifeforms For example when E coli bacteria are deprivedof nutrients unusually high mutation rates are observedThe exact mechanism by which such hypermutations areinduced is not known but they increase the likelihoodthat a population can survive such environmental insults(Moxon 1997) That is deprivation-induced variability isfunctional at the genetic level (The behavioral evidenceregarding long-term deprivation is less clear some stud-ies showing increased variability and others increasedstereotypy eg Carlton 1962 Senkowski Vogel amp Po-zulp 1978)

Endogenous variations external noncontingent eventsand withholding of reinforcement are three sources forthe variability that precedes control by reinforcementBut once control by reinforcement is established a dif-ferent question emerges What behavioral processes areinvolved when variability is maintained as an operantThe sources just discussed generate uncontrolled non-specific and often short-lived variability and cannot ex-plain the precisely controlled operant type

Operant Variability ProcessesThree processes are hypothesized to underlie operant

variability The first involves the use of random environ-

mental events such as the toss of a coin The second in-volves memory for responses such as when an animallearns not to return to previously visited locations Thethird will be referred to as an endogenous stochasticgenerator

Random environmental events Throughout historymany cultures have used random events to influencechoices and decisions such as throws of dice and randomselections of sticks cards bones or organs Today coinflips are used by a referee at the beginning of a game to de-cide which team kicks the ball a computerrsquos random-number generator is employed by scientists to avoid bi-ases in assigning subjects to experimental groups andaleatoric events are sometimes employed in modern artmusic and literature The Dice Man by Rhinehart (1998)provides a fictional example in which the protagonistbored with life writes a number of possible actions onslips of paper and then periodically selects one blindlyand behaves accordingly These examples show that randomexternal events may be used to avoid biases engender un-likely responses and break out of behavioral niches

Memory-based variability Many methods used tostudy operant variability differentially reinforce nonrepe-titions at least over the recent past Therefore one strat-egy that subjects can employ is to cycle through few re-sponses or sequences especially when the variabilitycontingencies are relatively permissive For exampleunder a lag 2 contingency cycling repeatedly throughthree sequences will result in 100 reinforcement Ani-mals and people sometimes do this apparently basingtheir current response on memory for or discriminativecontrol by the just emitted responses One such instancewas shown by Schoenfeld et al (1966) who rewardedrats for IRTs that fell into one of 10 interval classes onlyif the current IRT differed from the interval class of theprevious response The rats came to alternate betweenlong and short IRTs thus behaving systematically to meetthis lenient variability contingency Machado (1993)studied an analogous effect having pigeons peck L and Rkeys under a frequency-dependent variability contin-gency related to Shimprsquos (1967) procedure describedabove When the trial unit consisted of two responses thebirds developed memory-based repetitive sequencessuch as RRLLRRLL and thereby satisfied the variabilitycontingency When the trial unit was three responses inlength the birds did not develop the optimal fixed pat-tern of RRRLRLLLmdashwhich was too complexmdashbut in-stead developed ldquorandom-like behaviorrdquo (Machado1993 p 103) Thus a memory-based strategy was em-ployed when it was possible but when the memory loadbecame too high stochastic responding emerged A sim-ilar pattern was seen when variability of budgerigarsongs was reinforced (Manabe et al 1997) Under lag 1contingencies the birds sang mainly two songs underlag 2 they increased to three songs Only when the lagwas increased to 3 did song diversity increase apprecia-bly in one of the birds In radial-arm mazes as well whencontextual cues are available memory again appears toplay an important role leading to much lower probabil-

OPERANT VARIABILITY 691

ities of repetition than would be expected from a sto-chastic generator (McElroy amp Neuringer 1990)

Note that under lag schedules memory-based strate-gies result in higher reinforcement rates than respondingstochastically does The reason is that a stochastic gen-erator repeats by chance That animals and people oftendo not employ memory-based strategies probably indi-cates the difficulty of remembering long sequences andpossibly other advantages of unpredictable responding

If a memory strategy is the only way to satisfy vari-ability contingencies then it can be used even when thememory requirements are demanding as was shown byNeuringer and Voss (1993) Human subjects were trainedto generate on a computer keyboard a chaotic-like se-quence of responses that matched iterations of the logistic-difference equation

where Rn refers to the nth iteration in a series with R be-tween 00 and 10 and t is a constant between 10 and40 When the t parameter approaches 40 iterations ofthis equation result in very ldquonoisyrdquo sequences As withother chaotic processes however there is an orderlinessunderlying the surface noise In this case if responsesare autocorrelated with Rn drawn as a function of Rn21the resulting function is a parabola In the Neuringer andVoss (1993) study following each response college stu-dents were shown how close their predicted values wereto that from iterations of the logistic-difference model Inone experiment the students made digital predictions inanother they placed a pointer along a horizontal lineand this analogue response was then translated into dig-ital values for computation With training the studentsbecame increasingly adept at responding in chaotic-likefashionmdashthe studentsrsquo predictions matched closely theiterations of the logistic functionmdashand their autocorrela-tions became parabolic The left column in Figure 6 showsperformances by 4 subjects (rows) early in training thatis not well approximated by the logistic-differenceparabola and the right column shows performances at theend of training in which r 2 values were 72 99 98 andapproaching 10 How was this accomplished Sinceeach iteration in the logistic-difference sequence is basedon the prior outputmdashthat is the sequence is generated bya deterministic although nonlinear processmdashit was hy-pothesized that the human subjectsrsquo chaotic-like responseswere also based on memory for prior responses To put itsimply the subjects may have memorized a long series ofpairs in the form ldquoIf the previous response was value Athen the current response must be value Brdquo as Metzger(1994) Ward and West (1994) and Ward (2002) have sug-gested

To test whether subjects in fact relied on memory forprior responses Neuringer and Voss (2002) systemati-cally slowed responding by interposing pausesmdashthat isby requiring an IRT As IRT increased the difference be-tween the subjectsrsquo sequences and the modelrsquos chaotic

Rn n nt R R= -( )- -1 11

Figure 6 Autocorrelations (responses on trial n on the y-axis asa function of n21 on the x-axis) for 4 human subjects at the be-ginning (left column) and end (right column) of training inNeuringer and Voss (1993) Equations for the best-fitting curveand amount of variance accounted for (r2) are shown at the topof each panel To the extent that the circles (responses) approxi-mate the equation y = 40x 2 40x 2 the subjects were respondingin a way that matched the logistic-difference chaotic functionFrom ldquoApproximating Chaotic Behaviorrdquo by A Neuringer andC Voss 1993 Psychological Science 4 p 115 Copyright 1993 bythe American Psychological Society Reprinted with permission

692 NEURINGER

output increased and the underlying parabolic structurewas broken That is when pauses increased subjects wereless and less able to approximate the modelrsquos chaotic dis-tribution (see Figure 7) The important point here is thathighly variable responses which in this case are chaoticin nature can be based on memorial processes

Endogenous stochastic generator The evidence tobe reviewed next indicates that animals and people canrespond stochastically and that this competency is basedneither on chance environmental events nor on memoryfor prior events or responses It is impossible to provethat an endogenous stochastic process underlies operantvariability that would be analogous to proving the nullhypothesis However much evidence is consistent withthe stochastic hypothesis Included are the facts thatmemory limitations do not permit adequate respondingunder many variability contingencies that response dis-tributions match those expected from a stochastic gener-ator that changes in parameter values affect operantvariability as they would a stochastic generator and thatinterference with memory processes leaves stochasti-cally based operant variability unaffected As a wholethe evidence is consistent with an endogenous stochas-tic source of operant variability

Memory limitations One type of evidence is nega-tive in character and is based on the presumed inability ofmemory for prior responses or sequences to satisfy sometypes of variability contingencies Two examples weregiven above (Machado 1993 Manabe et al 1997) andanother was provided by a lag 50 contingency in Pageand Neuringer (1985) In the latter case the sequence in

the current trial (eight responses across L and R keys) wasrequired to differ from those in each of the previous 50 tri-als including at the beginning of a session the last 50trials from the previous session Pigeons responded wellunder these contingencies and in a way similar to that ofa stochastic model It is unlikely that pigeons can remem-ber each of their previous 50 eight-response sequences

Response slowing As was shown above for chaotic-like responding performances are often degraded by de-lays imposed between a stimulus and a response pre-sumably because of limitations in working or short-termmemory (Berryman Cumming amp Nevin 1963 Williams1971) Neuringer (1991) relied on this effect to test whethermemorial processes could explain the generation of vari-able sequences under lag schedules Two groups of ratswere reinforced for L and R leverpresses producing fourresponses per trial One group (Var) was trained under alag 5 contingency and the other (Rep) was trained to asingle sequence of responses LLRR It was assumedthat accurate Rep performance depended on workingmemory After performances had stabilized IRTs weresystematically varied by the imposition of timeouts of05 to 20 sec between consecutive responses in the se-quence During these forced pauses the chamber wasdark and responses were not counted toward completionof the sequence If the animals had used the previous re-sponse(s) as a discriminative cue for the current responsethen performances should have been degraded by theseinterpolated pauses

The results were that for the Rep group as the re-quired pauses increased from 0 to 6 sec percentages ofcorrect sequences were relatively unaffected but at pausesgreater than 6 sec probability of a correct LLRR se-quence fell sharply This pattern of results suggests in-volvement of working memory of prior responses withthe rats able to remember responses within a 6-sec win-dow The Var grouprsquos results were quite different Aspauses increased between 0 and 6 sec the rats were moreand more likely to satisfy the variability contingenciesand with pauses greater than 6 sec percent correct re-mained at a high and constant level Thus the Var grouprsquosresults were opposite to those of the Rep group and op-posite to those predicted from a memory-based strategySimilar results showing that operant variability increasesor is maintained as responding is slowed have been re-ported by Morris (1987) with pigeons and by Baddeley(1966) with random response generation by human sub-jects The results reported above concerning alcoholrsquos ef-fects on variability and repetition were also consistentwith those of Neuringer (1991)

Memory for prior responses may not have controlled theVar grouprsquos performance but why did variability increaseEach of three answers is consistent with a stochastic-generator interpretation Weiss (1964 1965) hypothe-sized that random responding requires that current re-sponses be independent of previous ones memory for orcontrol by prior responses interferes with such indepen-

Figure 7 Approximation to a chaotic sequence (given by delta)as a function of the imposed interresponse times Shown are theaverages of 5 subjects

OPERANT VARIABILITY 693

dence and therefore pauses would be expected to facil-itate variability A second possibility is that at shortIRTs animals tend to repeat responses on the sameoperandum Blough (1966) found this in pigeons and ex-cluded such double pecks from his analyses because theyappeared not to be under the control of reinforcement con-tingencies Morris (1987) also found a tendency for birdsto repeat when no interresponse timeouts were imposedA third hypothesis is that there were two contributors tothe observed variability One was a stochastic processcontrolled by reinforcement and the other elicited vari-ability generated by interposition of pauses The high vari-ability elicited as an effect of slowed responding is ageneral phenomenon supported in many other cases Ac-cording to this interpretation operant variability in the Vargroup was governed by a stochastic-based process operantrepetition in the Rep group was governed by a memory-based process and pauses elicited variability under bothcontingencies Thus Rep performance was interferedwith whereas Var was facilitated Each of these hypothe-ses is consistent with the conclusion that memory for (ordiscriminative control by) prior responses does not con-tribute to and possibly interferes with variable respond-ing when an organism is reinforced for variability andthis in turn is consistent with the stochastic-generatorhypothesis

Stochastic response distributions Other evidence iscorrelational Response distributions are similar to thosefrom a stochastic generator The data from two experi-ments already discussed are shown in Figure 1 In thesecases reinforcement of least frequent IRTs (Blough 1966)and of nonrepetitive choice sequences (Machado 1989)resulted in stochastic-like responding

Neuringer (1986) reported another striking case Highschool and college students generated sequences of 1sand 2s on a computer keyboard with each set of 100 re-sponses constituting one trial A baseline phase lastedfor 60 such trials for a total of 6000 responses duringwhich the students were asked to behave as randomly aspossible as if they were reporting the outcome of a se-ries of coin tosses As in the many previous studies ofthis sort the subjectsrsquo responses differed significantlyfrom those of a stochastic model (Brugger 1997 Wage-naar 1972) During the training phase the subjects re-ceived feedback which enabled them to compare their per-formances with that of a stochastic model first accordingto one statistical test and then according to another untilfollowing each trial feedback was provided from 10 dif-ferent statistics The results were that all of the subjectslearned to approximate the stochastic model accordingto the 10 statistics as well as some (but not all) other sta-tistical tests That is the distributions of the statisticswhich clearly did not match a stochastic model at the be-ginning of training approximated the stochastic modelat the end of it (see Figure 8)

The previous failures of other researchers to obtain ev-idence for random responding had led some to concludethat humans did not have the capacity to behave ran-

domly What would account for the difference betweenthe present positive finding and the many other negativeones Neuringerrsquos (1986) study was the first in which sto-chastic responding was explicitly reinforced in this casewith feedback from 10 statistical tests An analogy mighthelp to explain the difference in procedures and resultsImagine an experimenter walking around a city and ask-ing people if they know what a violin sounds like (analo-gous to asking people whether they know what a series ofcoin tosses or dice throws is like) Most would answeryes If the same individuals were then asked to play theviolin very few if any would succeed The experimentermight conclude after many replications with differentsubjects in different cities that people did not have thecapacity to play the violin The point of course is thatplaying the violin is a skill requiring training and prac-tice and the same may be true for responding randomlyIt is easy to vary but practice and reinforcement may benecessary to approximate a random distribution

Response number Page and Neuringer (1985) testedthe stochastic-generator hypothesis in another way byvarying number of responses per trial with pigeons work-ing under a lag 3 contingency The reasoning was that ifprevious responses served as a discriminative cue forwhat not to domdasha memory-based strategymdashthen perfor-mance should be degraded when the number of re-sponses was increased Eight responses per trial requiresubjects to remember more than four responses The sto-chastic hypothesis predicts the opposite outcome as isdemonstrated by the following example If each trialwere two responses in length with responses determinedby the toss of a coin then the probability that a given trialwould repeat the previous one would be 25 (There arefour possible sequences in the first trial RR RL LRand LL Thus the second trial has a 1-in-4 chance ofmatching the first) If a trial consists of four coin tossesthe probability of a repetition by chance is 0625 or 1 in16 Thus if subjects used a stochastic process to gener-ate Ls and Rs performances should improve with in-creasing responses per trial Results were precisely thosepredicted by the stochastic hypothesis and inconsistentwith a memory strategy Probability of meeting the con-tingencies and therefore probability of reinforcementincreased as responses per trial increased In otherwords eight-response trials were easier for pigeons thanfour-response trials

Response interference Neuringer and Voss (2002)attempted to distinguish between memorial and stochas-tic processes in yet another way They hypothesized thatwhereas chaotic responding was based on memory forprior responses (as was discussed above) responding inNeuringerrsquos (1986) study with feedback from 10 statisti-cal tests indicated involvement of an endogenous sto-chastic generator To show that human subjects could en-gage in both processes at will three college studentswere trained to generate chaotic sequences in one compo-nent of a multiple schedule as in Neuringer and Voss(1993) and to generate stochastic sequences in a sepa-

694 NEURINGER

rate component as in Neuringer (1986) The wordCHAOTICS was on the screen in the first component theword STOCHASTICS was on during the other and thesewords alternated throughout the session After the sub-jects succeeded at each of the component tasks test ses-sions were introduced in which memory for prior re-sponses was subject to interference in both the chaoticsand stochastics components as follows During the

chaotics component four separate chaotic sequenceswere cycled one after the other each cued by differentscreen colors The subjects attempted to approximate achaotic sequence in one color (eg light green) follow-ing which they would have to match a second chaotic se-quence (lavender screen color) and so on the fourchaotic contingencies following one another across thesession Although all four chaotic sequences used the

Figure 8 Performance by one subject at the end of training to respond stochastically in Neuringer(1986) Shown are comparisons on 6 different statistics between the subject (solid lines and Xs) andthe random model (dashed lines and closed circles) At the end of training the subjectrsquos perfor-mance did not differ statistically from that of the random model on these and four additional sta-tistics From ldquoCan people behave lsquorandomlyrsquo The role of feedbackrdquo by A Neuringer 1986 Jour-nal of Experimental Psychology General 115 p 71 Copyright 1986 by the American PsychologicalAssociation Reprinted with permission

OPERANT VARIABILITY 695

same logistic-difference equation that a model did theywere out of phase and independent If chaotic respond-ing is memory based then the four independent se-quences would be expected to interfere with one anotherthat is approximations to the four chaotic models wouldbe lower than when only a single chaotic output was re-quired as in Neuringer and Voss (1993) That was ex-actly the finding Performances by the three subjectswere signif icantly degraded during the four-segmentphase To put it simply chaotic responses interfered withone another

A different pattern of results was seen in the stochasticsportion of the experiment The same interference proce-dure was used with four independent stochastic sequencesrequired in the four different color conditions As inNeuringer (1986) subjects were required to satisfy eightdifferent statistical tests of randomness and to do this in-dependently in each of the four phases (If responses in anytwo or more of the four stochastic phases became corre-lated above a minimal level responses were considered tobe incorrect thereby prohibiting repetition of a single se-quence in two or more of the phases) For one subjectseven of eight statistics were closer to those of a randommodel during this interference phase than they were at theend of the original training with only a single sequenceand for a second subject all eight statistics were closerThe third subject showed closer-to-random approximationunder the four-segment condition on only two of the eightstatistics but most of her changes were also in the directionof a random generator For example if a random generatoremits equal numbers of ldquouprdquo and ldquodownrdquo sequences andthis subject in the baseline phase had emitted only 47ldquouprdquo sequences then during the four-segment tests shecame to emit ldquouprdquo 55 of the time She changed in the di-rection of the random model but overshot the mark Thuswhereas interference degraded chaotic performance sig-nificantly in all three subjects it improved stochastic per-formances in at least two of them These results are con-sistent with the hypothesis that a memory-based processcontrols chaotic variability and that a stochastic processnot dependent on memory controls stochastic variability

Summary of Tripartite TheoryThe following three processes constitute a tripartite

theory of operant variability attention to random events inthe environment (coin toss) memorial processes (chaosradial maze) and stochastic processes (response se-quences and random-number generation with feedback)Random environmental events assure unpredictabilitymdashfor example in making unbiased selections Remember-ing what was done previously will enable one to avoidrepetitions to vary the students called upon in class tovary the locations when looking for a hidden object orto try different approaches when attempting to solve aproblem External aids such as notes to oneself or sci-entific records help to enhance memory capacities

An endogenous stochastic generator provides the thirdand most controversial source of behavioral variability

When an opponent is attempting to predict your behav-ior in a game or at war it may be functional to behavestochastically Similarly operant conditioning of new re-sponses may depend on stochastic variations It is im-possible to prove stochasticity some presently unknowndeterministic system may ultimately explain the resultsHowever the evidence is consistent with the stochastichypothesis and inconsistent with the alternatives For ex-ample memory is not sufficiently powerful to explain alloperant variability memory-interference proceduresleave stochastic responding intact and behaviors corre-late well with those of stochastic models

Both genetics and reinforcement play important rolesin all three processes Reinforcement is always involvedin that learning provides information about when whatwhere and how much to vary Genetic influences areshown in that not all responses are possible and if theyare possible they are not all equiprobable (eg as inNeuringer et al 2001) Stochastic responding may be bi-ased due to experiences and knowledge that is biasesmay be established by reinforcement contingencies or bythe interface between the organismrsquos propensities (or be-havior systems see Timberlake amp Lucas 1989) and theenvironment In the case of random-number generationby human subjects biases may be due to misconceptionsconcerning what ldquorandomrdquo means However some bi-ases can be overcome through feedback and instructions(Lopes amp Oden 1987 Neuringer 1986)

The importance of variations for evolutionary changeis supported by the many sources of genetic variabilityand by the fact that genetic variability is conserved Asimilar argument holds for operant variability That thereare at least three independent sources of operant varia-tions as well as other elicitors such as environmentalnoise and response extinction supports its importancefor normal adaptive functioning

Structure of VariabilityEvery operant ldquoresponserdquo is actually an instance of a

class with each member of the class exerting the sameeffect on the environment (Skinner 1959) When a rat isreinforced for pressing a lever for example the class isdefined by closure of a microswitch and any way of ac-complishing the closure whether with the left paw theright paw or the mouth is as effective in producing re-inforcement as any other If the contingencies of rein-forcement so specify however the response class can beredefined and circumscribed which is another charac-teristic of the operant For example if only right-pawpresses are acceptable members of the reinforced classresponding will tend to occur only with the right paw(Stokes 1995) Of course the capabilities of the organismplay a role for example including ldquotail pressesrdquo in the op-erant class does not mean that the lever would be or couldbe pressed by the ratrsquos tail The structure of the operantclass therefore emerges from the current reinforcementcontingencies and the organismrsquos capability derivedfrom genetics development and prior experiences

696 NEURINGER

Operant variability has structure as well and as for alloperants that structure is due partly to the organism it-self and partly to the reinforcement contingencies Theresearch described throughout this article has providedexamples of the structure of variability generally in theform of sets of four or eight responses across L and Roperanda But the structure of variations can be speci-f ied more precisely For example Mook and her col-leagues (Mook et al 1993 Mook amp Neuringer 1994)reinforced rats for varying four response sequences acrossL and R levers under a lag 1 contingency Unlike in pre-vious studies however the acceptable class was furtherconstrained so that only sequences beginning with two Rresponses (RRLL RRLR RRRL and RRRR) were rein-forced The rats learned to vary their sequences withinthe imposed limitations that is they generally begantheir sequences with two R responses and varied the se-quences within those limitations Similar results wereobtained in the context of a radial-arm maze in whichonly a subset of arms was baited

The precision with which reinforcement contingen-cies can specify class was highlighted in research per-formed by Burns (1997) and Ross and Neuringer (2002)In the latter case college students drew rectangles on acomputer screen with the computerrsquos mouse Each rec-tangle constituted one trial (the screen cleared at the endof each trial) with reinforcement (points) depending onsimultaneous variations along three dimensions of therectangle location on the screen area of the rectangleand shape with the latter defined as the ratio of the rec-tanglersquos height to its width Although the subjects weregiven no instruction other than to ldquogain pointsrdquo theylearned to vary along all three dimensions with levels ofvariability significantly higher than those of yoked controlsubjects who were reinforced independently of variations

A second part of the Ross and Neuringer (2002) studydemonstrated even greater precision of control To gainpoints three new groups of subjects were required to re-peat along one of the dimensions while simultaneouslyvarying along the other two One group was required torepeatedly draw rectangles of approximately the samesize while simultaneously varying location and shapeanother group had to repeat the shape while varying areaand location and the third to repeat location while vary-ing size and shape Verbal instructions were minimal asthey were in the first part of the study Subjects learnedto satisfy these contingencies as well varying two at-tributes while simultaneously repeating one and signif-icant differences were obtained rapidly within 300 trials

These demonstrations raise questions as to how binaryfeedbackmdashreinforcement presented or notmdashcan simul-taneously control independent behavioral dimensionsThe issue is similar to that discussed above concerningsimultaneous reinforcement of variations and selectionsIt is similar also to those issues faced by researchers at-tempting to understand how the nervous system strength-ens particular synapses when many different neurons firesimultaneously and to ldquoallocation of creditrdquo in artificial

learning models For present purposes suffice it to notethe exquisite precision with which reinforcement controlsvariability and repetition

GeneralizationDo animals and people learn to ldquovaryrdquo implying a

generalizable skill or do they learn a specific distributionof specific responses across particular operanda Thequestion of generalization has been asked of many otherbehavioral competencies such as imitation When chil-dren learn to imitate do they learn stimulusndashresponseassociations (if mother moves her right hand the childlearns to move his right hand) or do they learn moregenerally to ldquoimitaterdquo Evidence indicates that reinforce-ment of imitative instances leads to generalized imitation(see eg Baer amp Sherman 1964 Bandura 1986) A sim-ilar question has been raised when pigeons rats or chil-dren learn to choose a stimulus that matches a previouslyshown sample For example if a pigeon is shown a reddisk as a sample it must then choose red when given achoice between that and blue Does the bird learn to matchIn this case the answer appears to depend on the train-ing procedures If trained with few stimuli such as redgreen and blue then the pigeon learns simple associa-tions (if red sample choose red if blue sample chooseblue) but if hundreds of different stimuli (pictures) areemployed during training then generalized ldquomatchingrdquois learned (Wright 1997) In each of these examples thelearning of a general competency or strategy was shownby transfer-of-training tests What was learned with oneset of stimuli or in one situation transferred to new stim-uli and situations We turn now to the question of whetherlearning to vary transfers

After the subjects in Maltzmanrsquos (1960) experimentdescribed above learned to generate remote verbal as-sociates they were asked to devise unusual uses of com-mon objects In comparison with those in the controlgroup these subjects generated significantly more un-usual uses indicating transfer of training to a differenttask However transfer was not observed in the oppositedirection from unusual uses to verbal associates

Holman et al (1977) using a procedure similar to thatof the Goetz and Baer (1973) research described aboverewarded preschool children verbally for novel drawingsand found increased novelty Unrewarded block construc-tions were also monitored and weak evidence was ob-tained for transfer of novel responding from rewardeddrawing to the unrewarded constructions In a second ex-periment preschoolers engaged in four independent taskspainting pen drawing wood block construction and Legoconstruction When the children were reinforced for vary-ing their pen drawings the novelty of paintings increasedalong with the pen drawings indicating transfer from onetask to the other However transfer to the topographicallydifferent block- or Lego-building tasks was not observed

Eisenberger and Armeli (1997) gave monetary rewardsto one group of fifth- and sixth-grade students the vari-ability group for describing unusual uses of common ob-

OPERANT VARIABILITY 697

jects A second group the repeat group received thesame rewards for describing common uses of the sameobjects In a later test all of the children were asked todraw pictures incorporating a circle and the group that hadpreviously been reinforced for varied responses drewmore unusual (or lower frequency) pictures than did therepeat group Thus there was transfer of novelty trainingfrom one domain to another with additional studies in theseries showing similar effects (Eisenberger Armeli ampPretz 1998 Eisenberger Haskins amp Gambleton 1999Eisenberger amp Selbst 1994) However Eisenberger et al(1999) failed to replicate the transfer-of-training effectThe variability and repeat groups responded similarly dur-ing the test phase In this study separate experimenterstrained and tested the subjects thereby avoiding possibleexperimenter-expectancy effects whereas the same ex-perimenter did both training and testing in Eisenbergerrsquosother studies Across the series of experiments there weremany other cases in which transfer was not observed andmore research is necessary to explain why

In Lee et alrsquos (in press) study with autistic individualsdescribed above after variable answers were trained toquestions verbal variability transferred to new experi-menters and new contexts

Arnesenrsquos (2000) results described above are consis-tent with transfer of training in an animal model Rats inthe Var group were trained to interact variably with oneset of objects When later tested in a new situation withnew objects the Var animals showed (in comparisonwith controls) higher levels of exploration manipulationof the novel objects and discovery of hidden reinforcers

Much of the evidence reviewed in this section there-fore indicates transfer of training and supports a generalldquovaryrdquo competency However there remain unresolvedquestions the most important of which is the extent towhich transfer occurs Note also a methodological pointMany of the studies tested whether transfer occurs whenthe contingencies are neutral with respect to the to-be-transferred behavior Holman et al (1977) is a case inpoint Variations were reinforced in one domain draw-ing and the question was whether variations wouldoccur in another domainmdashblock constructionmdashin whichreinforcement did not depend on varying A negativef indingmdashin this case absence of transfermdashindicatesnothing about the ability to generalize but only aboutthe tendency to do so given the absence of a reason (orcontingency) A better test would be to compare experi-mental and control performances when variability in thenew task is in fact functional as in Arnesen (2000)

SummaryBefore variability can be reinforced it must occur for

reasons other than reinforcement Some of the reasonsinclude variability inherent in all behavioral phenomenaldquoaccidentalrdquo environmental events that are unrelated toparticular behaviors and withholding of reinforcementTransfer from other situations may serve a similar functionFor example if an organism can discriminate shaping-

of-new-response periods from others then it may learngenerally to ldquovaryrdquo during shaping episodes After rein-forcement makes initial contact with response variationsthere are at least three behavioral strategies that can beemployed in order to maintain functionally high levelsof variability respond based on external random eventssuch as the toss of a coin avoid repetitions by remem-bering what responses had previously been made and re-spond stochastically Evidence for an endogenous sto-chastic process was seen in the similarity of responsedistributions to those generated by stochastic sourcesand by findings that interfering with memory for priorresponses either improves or leaves unaffected stochas-tic behaviors Stochastic responding as well as othertypes of variability-generating strategies depend uponcontingencies to define the required level of variabilityand the set from which variations emerge

IMPLICATIONS

One implication of reinforced variability is that oper-ant responses may differ in the extent to which their vari-ability is affected by reinforcement contingenciesmdashthatis operants may be ordered along a continuum of sensi-tivity of variability to reinforcement Another is that re-inforced variability causes us to question the generalityof some basic goals of behavioral psychology includingprediction and control A third is that reinforced vari-ability may help us to provide a scientifically useful wayto describe voluntary action In particular the variabilityof all voluntary behaviors can readily be changed mo-ment to moment by discriminative stimuli and conse-quences Implications for everyday life also will be noted

Operant ContinuumAs was indicated above the operant is composed of

instances that is the operant is a class of many differentresponses all functionally equivalent but often topo-graphically distinct Skinner suggested and I hypothe-size that instances within the class emerge stochastically(see Moxley 1997 for a discussion of Skinnerrsquos posi-tion) Although reinforcement contingencies and contextinfluence the individual probabilities and their distribu-tions emission is stochastic nonetheless The particularmember of the operant class generally cannot be predictedwith a high degree of certainty For example a particu-lar rat may be more likely to press the lever with its leftpaw than its right but left- and right-paw responses areemitted probabilistically This class-nature of the operantsuggests one way to conceptualize operant variability interms of size of class and within-classes probability dis-tributions High variability implies large classes equalprobability distributions or both low variability impliesthe opposite

Variability may be modified more readily in some op-erants than in others implying that operant variability isnot an all-or-none phenomenon but rather a continuumwith reinforced variations in variability (RVV) differing

698 NEURINGER

for different operants RVV might best be understood byanalogy to the concepts of speed and acceleration Bothvariability and speed range from low to high RVV isanalogous to acceleration the rate of change in speedHighly sensitive RVV is indicated by the possibility ofreinforcing many different levels of variability from rep-etition to random-like responding and by the fact thatchanges in level occur readily and rapidly

Different operants manifest differences in RVV Forexample variability of chain pulls changed more thanthat of leverpresses when variability was reinforced inrats (Morgan amp Neuringer 1990) Some operant-like re-sponses may show relatively low RVV For example fir-ing rates increase in single hippocampal nerve cellswhen dopaminergic substances are contingent upon theresponse (Stein Xue amp Belluzzi 1994) suggesting thatnerve f iring is controlled by reinforcement To datehowever there is no evidence that variability of nervefiring is similarly controlled Other cases might includereinforcement of nondiscriminable (subconscious) re-sponses such as Hefferline and Pererarsquos (1963) reinforce-ment of subaware thumb twitches and biofeedback ofmuscle and cardiovascular activity In these cases rein-forcement may modify performance but it is not knownwhether variability is also sensitive to reinforcement andif so to what degree Human verbal variability may bethe most readily manipulable of all behaviors the rangeof predictable to unpredictable verbal responses beingimmense with extraordinary sensitivity to context andconsequence RVV may also differ across organisms (per-haps from simple to complex) individuals (perhaps asan index of intelligence or learning ability) and physio-logical and psychological states (WKY vs SHR depres-sion and autism)

RVV may help to distinguish operant responses fromother superficially similar cases of selection by conse-quences That is the ldquooperantrdquo may also best be concep-tualized as a continuum with RVV an important deter-miner of position along the operant continuum Oneexample of selection by consequences but low ldquooperant-nessrdquo is genetic change Variability is essential for evolu-tionary change of course and levels of genetic variabil-ity are sometimes affected by environmental conditions(Moxon 1997) but there are few examples of levels ofvariability being influenced by selection pressures Thatis the source of much genetic variability is thought to berandom not influenced by current pressures and needsThus although selected by consequences genetic changemay be low along the operant continuum Other exam-ples of operant-like phenomena for which variability although important may be relatively insensitive to con-sequencesmdashvariability in these cases is elicited not se-lectedmdashinclude the development of the central nervoussystem and immune system responses (Hull et al 2001)In both of these cases change depends upon selection byconsequences but it is not clear whether variability canalso be selected Most commonly studied operant re-sponses on the other hand show high sensitivity of vari-

ability to reinforcing events But to reiterate the mainpoint of this section operants differ in terms of RVV andtherefore in terms of their ldquooperantnessrdquo from low sensi-tivity (possibly responses of single nerve cells) to extra-ordinarily high sensitivity (verbal behavior) The operantnature of variability as well as that of every other oper-ant dimension is best conceptualized as a continuum

Prediction and ControlSmith (1992) described Skinnerian operant condition-

ing theory as Baconian in nature a primary goal of Skin-nerian behaviorists being to control behavior and hecontrasted this with the more Aristotelian goal of de-scription Skinner and his students (eg Zuriff 1985) in-deed emphasized prediction and control Many of thestudies reviewed above however involved reinforce-ment procedures that increase variability and conse-quently decrease predictability It is not only ignorancetherefore that restricts prediction and control as Skin-ner suggested but reinforcement itself Operant meth-ods and theories when applied to variability lead togreater emphasis on description with control over pre-cise response characteristics possible under some butnot all conditions

In summary an operant is a class of responses underthe control of contingencies of reinforcement The classcan be controlled and predicted given knowledge ofconditioning history Indeed the size of the class andpossibly the distribution of within-classes probabilities orlevel of variability can also be controlled and predictedBut to the extent that instances emerge stochastically aparticular instance cannot be predicted The instant re-sponse can be explained as a member of a particularclass but prediction of the instance will be diff icultsometimes to the extreme This point will be developedin the next section

Voluntary BehaviorI hypothesize that voluntary behaviors are operants for

which RVVs are highly sensitive to influence by conse-quences One way to test this hypothesis would be togather empirical data concerning how voluntary versusinvoluntary actions are classified by human observers Asfar as I know there are no psychophysical data concern-ing this question but the hypothesis implies that highsensitivity of variability to operant contingencies RVVswill be present in all actions perceived to be voluntary

This operant-variability theory of voluntary action isbased on the supposition of two attributes First volun-tary acts must be potentially explainable (given suff i-cient knowledge) on the basis of genetics conditioninghistories current contingencies anticipated conse-quences intentions goals and the like That is volun-tary behaviors are lawful The second attribute is that thevoluntary act must be potentially unpredictable That isvoluntary acts are ldquofreerdquo of control by identifiable stim-uli If behavior is perceived necessarily to result frommdashto be predicted on the basis ofmdashexternal events includ-

OPERANT VARIABILITY 699

ing reinforcement contingencies it will not be perceivedas voluntary ldquoVoluntaryrdquo implies potential indepen-dence from stimulusndashresponse or responsendashreinforcerdetermination According to this theory both explain-ability and unpredictability are necessary attributes ofvoluntary acts

How though can behaviors be simultaneously ex-plainable and unpredictable Reinforcement contingen-cies play two essential roles Reinforcement (togetherwith genes prior experiences and current environment)helps to determine the set of possible behaviors fromwhich a voluntary act emergesmdashthat is the operant classReinforcement (together with the other contributorsspecified above) also helps to determine the level of vari-ability within the class Thus both instances comprisinga class and level of variability within that class can bepredictedmdashagain given sufficient knowledge Howeverthe individual response is at least sometimes (or relatively)unpredictable The reason is that the operant class fromwhich the response stochastically emerges is sometimesextremely large (eg ldquoname a wordrdquo or ldquodo somethingrdquo)and the contingencies sometimes call for equiprobablestochastic emergence (eg ldquofool merdquo) In the latter casea voluntary act may appear to be foolish meaninglessor unwarranted but if it is voluntary it can potentially beexplained (For example in response to the questionldquoWhy did you do that silly actrdquo the answer could beldquoJust to show that you canrsquot predict my behaviorrdquo seeScriven 1965) In these cases prediction of instancesfalls to very low levels indeed and I hypothesize evenan all-knowing observer will be unable to predict thenext response at greater-than-chance levels which aregiven by the size of the operant class and by within-classesprobability distribution Even within small classes (egin the game rockndashscissorsndashpaper in which unpredictabil-ity among the three possible responses is functional) be-havioral stochasticity may lead to unpredictability of theinstant response Note however that although voluntarybehaviors are sometimes unpredictable the opposite isoften the case When contingencies call for highly con-strained particular responses predictions can easily bemade For example if a police officer asks your nameyour response can most likely be predicted Voluntary re-sponses are therefore described as potentially unpre-dictable with variability contingencies engendering dif-ferent levels of unpredictability

According to this theory behaviors low in the sensitivity-of-variability dimension are described as relatively in-voluntary and those that are high in said dimension areperceived as voluntary Low RVV may be shown at anylevel along the variability continuum For example al-ways responding in the same unchanging way will beperceived as involuntary but so will always respondingrandomly (The coin does not voluntarily come up withheads or tails) This point is essential More than 2000years ago Epicurus argued that occasional randomswerves of atoms provided the source of voluntary be-havior in an otherwise determined universe but al-

though this conjecture was prescient it was not quitecorrect ldquoFree-willedrdquo or voluntary action whether it isbased on random swerves or on quantum mechanics isnot explained by variability per se but by functionalchanges in levels of variabilitymdashthat is by rapid andprecise changes in response to environmental demandsResponses can be placed along a voluntaryndashinvoluntarycontinuum isomorphic to the operantndashrespondent con-tinuum with voluntary behavior being functional pre-cisely in the same way as operant behavior is ThusSkinnerrsquos (1974) suggestion that voluntary behavior isoperant behavior was almost correct Only those oper-ants for which variability is highly sensitive to conse-quences will be perceived as voluntary

One additional example might help Operant classes(sets of potential responses) are created ldquoon linerdquo as we goabout our daily activities As we walk from car to officewe tend to walk in a characteristic way take the samepath with tightly constrained operant classes followingone after the other in rapid succession Class sizes aresmall and within-classes probability distributions arehighly peaked Predictions of next responses by a knowl-edgeable observer can readily be made However classsizes can be increased and probability distributions flat-tened we can walk in quite strange ways indeed dancealong a new path sing hand candy to passers-by Strangebehavior indeed but made possible by the ability of op-erant sets to be modified moment to moment not only interms of class definition but in terms of class size lev-els of within-classes variations and so on The same istrue for conversation Moment-to-moment changes inverbal class definition size and predictability of within-classes instances are hallmarks of voluntary behavior

GenerativityRelated to the definition of the operant and to our discus-

sion of voluntary action is that fact that operant behaviormdashleverpressing no less than languagemdashis by its naturegenerative This claim follows from the functional na-ture of the class of responses (all members of the classare effective in producing an outcome) and the stochas-tic emission of instances From these flow never-before-emitted instances patterns and sequences (see Epstein1996) Operants especially those high on the RVV con-tinuum are inherently generative

Self-ExperimentationVariations systematic or otherwise are a necessary part

of scientific work Neuringer (1981) and Roberts and Neu-ringer (1998) described how variations can be brought to bear explicitly on everyday life in the form of self-experiments Individuals vary aspects of their lives suchas the foods they eat social contacts and methods ofmemorizing they keep records concerning the effects ofthese manipulations and form hypotheses concerningcontrolling variables Using self-experimental methodsindividuals can learn how what when where and howmuch to vary Explicitly varying onersquos everyday behav-

700 NEURINGER

iors and thoughts may help to engender what Langer(1989 2000) has referred to as mindful livingmdashthat isnonautomatic voluntary engagement with beneficialeffects in different areas including learning The impor-tant point here is that variability can be voluntarily cho-sen maintained and modified in everyday life and thatexplicit variations may have important consequences

Implications for SocietyConsideration should be given to the importance of re-

inforcement contingencies to support variability in so-cial milieus A basic point made throughout the presentarticle is that merely permitting behavior to vary doesnot necessarily engender or maximize variability (as isshown by Yoke and Prob schedules) Analogously merelypermitting varied social behavior as in a free or laissez-faire society may not suffice to produce the desired di-versity If the results shown for individual organismscan provide useful information for groups (see eg Grottamp Neuringer 1974) then when variations are desiredsocieties might reward individuals for varying not sim-ply permit it

SummaryOperants differ in the extent to which reinforcers in-

fluence levels of variation When variability is highlysensitive to reinforcement the operant may be perceivedto be voluntary Voluntary responses have two attributesThey are functional implying that the class of responsesmdashthe operantmdashcan be predicted given sufficient knowl-edge But instances emerge stochastically implyingsome degree of unpredictability Because reinforcementof variability can generate extremely large classes withstochastic emergence of instances it suggests limitationsof the two basic goals of Skinnerian behavioral theorymdashnamely prediction and control Operant variability also hasimplications for self-experimentation and for society

RELATED AREAS

Operant variability may be important in many otherareas and I will briefly review some of these

Negative reinforcement To my knowledge there havebeen no published studies in which variability is nega-tively reinforced High variability might sometimesavoid or reduce a punisher such as loss of goods ormoney but at other times low variability could be func-tional Effects of positive and negative reinforcement onvariability may have different evolutionary bases onederived from increasing probabilities of locating sourcesof food or mates the other from avoiding confrontationsor predation (Driver amp Humphries 1988) One questionconcerns how quickly variability can be changed by neg-ative as opposed to positive reinforcement There isconsiderable evidence that loss functions differ fromgain functions (Kahneman amp Tversky 1984) possiblyindicating that positive reinforcement-of-variabilityfunctions will differ from negative ones as well

Resistance to change In some of the studies describedabove variability appeared to be more resistant tochange than repetitions were for example alcohol hadlittle effect on variable operant responses but interferedwith repeated ones Doughty and Lattal (2001) tested thegenerality of these f indings using four-response se-quences with pigeons a threshold variability contin-gency in the Var component of a multiple schedule andLRLR in the Rep component Doughty and Lattal chal-lenged stability of responding by prefeeding the birds inone experiment and providing reinforcers independentlyof responding in the other (see Nevin 1992) Operantvariability was again more resistant to change than wasoperant repetition Thus reinforcement of variabilitymight increase the stability of responding (that is its re-sistance to change) a seemingly paradoxical findingHowever Mechner (1992) suggested that this is the casefor complex skills such as piano performance Varyingpractice routines will make the performance relativelyinsensitive to unforeseen changes in the environmentCare must be taken however when variation is comparedwith repetition because they have different measuresAlso the results reported by Doughty and Lattal werebased on the performance of three pigeons and were notconsistent throughout the experiment The hypothesiswarrants further study

Attraction In some bird species complex male songsattract females (for a review see Catchpole amp Slater1995) There is evidence that male song complexity isunder the discriminative stimulus control of a receptive fe-male (Searcy amp Yasukawa 1990) That explicit reinforce-ment may play a role in such song complexity is suggestedby the Manabe et al (1997) study described above An in-teresting question for further research is the extent towhich operant contingencies contribute to the attractioncreated by behavioral variability in these and other species

Discrimination of variability Animals and people canuse the variability of a stimulus array or entropy as a dis-criminative stimulus (Young amp Wasserman 2001) Whatis the relationship between perception of variability onthe one hand and emission of variable responses on theother Do individuals attempt to match their own levelsof variation to those of others in their environments whichwould involve both perception and emission How mightdiscrimination of onersquos own level of variability influenceonersquos ability to modify such levels and do influences thatobscure discriminations therefore affect ability to change(see eg van der Meer van der Weel amp Lee 1995)Hunziker et al (2002) report that although variabilitywas successfully reinforced their human subjects couldnot correctly describe the contingencies of reinforce-ment a finding similar to those of informal observationsin other studies (eg Neuringer Deiss amp Imig 2000)but whether subjects could discriminate variability levelsrather than contingencies was not studied

Attention and awareness When attention is subject tointerference or is withdrawn from the task at hand abilityto vary is degraded (eg Baddeley Emslie Kolodny amp

OPERANT VARIABILITY 701

Duncan 1998 Evans amp Graham 1980 Neuringer amp Voss2002) It is unclear at present how attention influencesthe different processes underlying operant variabilityand in particular memorial and stochastic processesRich areas for additional study are the relationships be-tween attention awareness and consciousness on theone hand and operant variability on the other Voluntaryaction and consciousness are often related (see egLibet 1999)

Concepts Concepts are thought by many to be sets ofinstances (much like operant sets) whose structures aregraded meaning that there are more or less typical in-stances of such concepts as birds (robin is typical ostrichis not) Barsalou (1987) showed that the gradation ofthese cognitive structures is manipulable or as he putsit not stable A number of variables influence structureincluding context individual differences and points ofview Barsalou argued that these structures are createdas the need arisesmdashthat is on line by the thinking indi-vidual The structure of concepts may parallel that of op-erants and in both cases levels of within-structures di-versity or variability may be modified by consequencesOperant variability may be found in the cognitive as wellas in the behavioral domain

Play and humor Highly variable behaviors are ob-served when animals and people especially children en-gage in play and humor (see eg Fagen 1981) The re-sult of play may be acquisition of new responses (seeSutton-Smith 1975) Although not currently needed re-sponses learned during play may provide the source ofvariations for successful shaping of behaviors at anothertime This is quite analogous to the role of conserved ge-netic variants which are passed along to progeny but donot currently serve any identifiable role Play may alsoinvolve learning how to vary operantly

Reinforcement by variability The self-maintainingnature of play suggests that engaging in variable behav-iors may itself be reinforcing Indirect support comesfrom findings that varied stimulation can serve as a re-inforcer and that stimulus repetition becomes aversive(Fiske amp Maddi 1961) Possibly also related are findingsthat varying reinforcers and permitting choice among re-inforcers increase their value (Bowman Piazza FisherHagopian amp Kogan 1997) and that permitting choicesamong varied responses is also preferred to demandingfixed responses (Catania 1980)

Foraging Two issues regarding variability are presentin the foraging literature One has to do with risk prefer-ence versus avoidance (see eg Shettleworth 1998) Asecond has to do with search strategies Most reportedcases of foraging involve systematic search patterns(Stephens amp Krebs 1986) However search patterns aresometimes stochastic or highly variable (see eg Peter-son 1997) One question is whether some species can varybetween these two strategies and the challenge is to iden-tify the pressures leading to and the advantages of each

Animal training Wilkes (1996) has applied operantvariability techniques in developing training proceduresfor pet dogs More generally reinforcement of variabil-

ity may enable the gaining of control over behaviors ofanimals and as has been suggested above may be an im-portant part of the process of shaping

Models of learning Reinforced variations in variabil-ity may be important when human and animal learningis being modeled whether the models are mathematicalin form computer based or machine based (Blumberget al 2002 Brooks 1999 Sutton amp Barto 1998)

CONCLUDING COMMENT

Genes interact with environments to produce diverseanatomical physiological and behavioral phenotypesGenes and environments also provide for another level ofdiversitymdashnamely operant variability There are manysources of operant variability including random environ-mental events memory for past responses and stochasticgeneration each of which ultimately derives from ge-netic and environmental influences To the extent thatoperant variability is based on a stochastic source how-ever the behavioral results cannot be predicted fromknowledge of these influences That is organisms haveevolved to behave unpredictably The positing of sto-chasticity to explain behavior has a history extending atleast as far back as Epicurus with major contributors in-cluding Gustav Fechner (see Heidelberger 1987) EgonBrunswik (see Gigerenzer 1987) Skinner (1959) andSuppes (1984) Knowledge of the stochastic nature ofthe operant and of how it is controlled leads to a worldview that differs from genetic and environmental deter-minism with roots in the works of the scholars just men-tioned At the scientific level this view leads to the ex-perimental analysis of consequence-influenced variabilityAt the philosophical level it leads to explanations of vol-untary action At the personal level it can lead to change

REFERENCES

Amabile T M (1983) The social psychology of creativity New YorkSpringer-Verlag

Antonitis J J (1951) Response variability in the white rat duringconditioning extinction and reconditioning Journal of Experimen-tal Psychology 42 273-281

Archer J amp Birke L (1983) Exploration in animals and humansCambridge Van Nostrand Reinhold

Arnesen E M (2000) Reinforcement of object manipulation in-creases discovery Unpublished undergraduate thesis Reed College

Baddeley A D (1966) The capacity for generating information byrandomization Quarterly Journal of Experimental Psychology 18119-129

Baddeley A [D] Emslie H Kolodny J amp Duncan J (1998)Random generation and the executive control of working memoryQuarterly Journal of Experimental Psychology 51A 819-852

Baer D M amp Sherman J A (1964) Reinforcement control of gen-eralized imitation in young children Journal of Experimental ChildPsychology 1 37-49

Baker M J amp Koegel R L (1999) Transforming children withautismrsquos ritualistic behaviors into positive play interactions with sib-lings Paper presented at the meeting of the California Association ofBehavior Analysis San Francisco

Balsam P D Deich J D Ohyama T amp Stokes P D (1998) Ori-gins of new behavior In W OrsquoDonohue (Ed) Learning and behav-ior therapy (pp 403-420) Boston Allyn amp Bacon

Balsam P D Paterniti A Zechowy K amp Stokes P D (2002)

OPERANT VARIABILITY 675

average reinforcement frequencies affects choices for ex-ample choices are affected differently by variable-interval(VI) than by fixed-interval (FI) schedules of reinforcement

Wilson (1986) devised one solution to this problemWilsonrsquos procedure is complex but in brief responding wasanalyzed in a manner that avoided overlapping sequences(which was the cause of the clumping in Shimprsquos 1967procedure) Four sets of counters were created Using thesame example given above in the first set one count wasadded to each counter representing RLLL RRRL RRLLand LLLL so that successive trials were completely inde-pendent (Note the absence of overlap) When the next re-sponse was emitted sequences began with that responsemdashlet us assume that the next response added to the rightof the series was an Lmdashand therefore in the examplegiven the second set was created by offsetting responsesone place to the left Now one was added to LLLL withprevious counts added to RRLR RLLR LLLR and soon A similar procedure was followed for each of the re-maining two sets Each response therefore terminated afour-response sequence in one and only one of the foursets and reinforcement was based on satisfaction of thecontingency within that set only Wilsonrsquos procedure hadother aspects but for present purposes it suffices to notethat the dependencies in Shimprsquos procedure were avoidedand when pigeons were tested under these contingen-cies they generated choices that more closely approxi-mated those of a random model Hopefully this lengthyexplanation indicates the importance of precise contin-gencies when one is attempting to reinforce variabilityAn advantage of Shimprsquos and Wilsonrsquos procedures wasthat every response could be reinforced at least poten-tially as was the case for each of the other procedures dis-cussed until now However a simpler way to avoid theproblems just described is to require a fixed number ofresponses in each trial with reinforcement possible onlyat completion of the trial As will be seen most recentstudies have used the latter tactic

The evidence generally supported the claim that be-havioral variability could be reinforced but the agree-ment about these findings was short lived Schwartz (19801982) reached the opposite conclusion and his researchwill be considered in detail

Schwartzrsquos ResearchSchwartz (1980 1982) used fixed-length trials con-

sisting of eight responses per trial by pigeons across tworesponse keys During an initial control phase a trial ter-minated with food if it contained exactly four L and fourR responses regardless of the particular sequence of Lsand Rs However if either key was pecked a fifth timethe trial terminated immediately with timeout A 5 3 5light matrix was provided to assist the birds in this taskAt the beginning of a trial the upper-left corner light wasilluminated and all others were off An L peck movedthe light one column to the right and an R peck movedit one row down Any sequence of four L and four R re-sponses would therefore move the light from upper left

to lower right and the trial would end with food Each ofthe birds came to emit a single sequence with high prob-ability in most cases LLLLRRRR or RRRRLLLL Arelatively low success rate at the end of 40 training ses-sions indicated that the task was harder than it mighthave appeared Only 66 of trials ended with food andthe remaining 34 ended with timeout because withina given trial one of the keys had been pecked a fifth time

Schwartz (1982) then required that the sequence in thecurrent trial differ from that of the just preceding trialThis will be referred to as a lag 1 variability + constraintcontingency (ldquoconstraintrdquo indicating the required fourLs + four Rs) If a bird had emitted LLLLRRRR in theprevious trial for example any other sequence would bereinforced in the next trial assuming that it containedfour Ls and four Rs but a repetition led to timeout Thebirds performed poorly only 36 of the trials now end-ing with food because a dominant sequence was oftenrepeated or one of the keys was pecked five times Hy-pothesizing that this failure to reinforce variability mayhave been due to the birdsrsquo extended experience underthe initial control condition (in which variations were notrequired and any sequence of four Ls and four Rs wasreinforced) Schwartz (1982 Experiment 1) again triedthe variability-requiring procedure with naive subjectsAgain the birds failed In another experiment Schwartz(1982 Experiment 2) explicitly trained birds to alternatebetween LLLLRRRR and RRRRLLLL sequences andthen introduced the lag 1 variability + constraint contin-gency but again the birds failed to vary Schwartz (1982)concluded that variability could not be reinforced andoffered a number of possible explanations for why his re-sults differed from the previous findings Perhaps Maltz-manrsquos (1960) and Goetz and Baerrsquos (1973) findings weredue to the information providedmdashverbal indicationsfrom the experimenters that subjects should varymdashrather than to reinforcement Perhaps Pryor et alrsquos (1969and by implication Goetz amp Baerrsquos 1973) results weredue to brief extinction periods in which reinforcers werewithheld (because variations were not forthcoming)rather than to direct reinforcement of variability PerhapsBloughrsquos (1966) results were specific to IRT and not rel-evant to response forms or choices These explanationsall imply that it is not possible to reinforce variability orthat it is possible only in narrowly defined cases As willbe seen Schwartzrsquos (1982) results readily met the test ofreplication but there was a problem nevertheless withhis conclusion

Schwartz ExplainedPage and Neuringer (1985) simulated Schwartzrsquos (1982)

lag 1 variability + constraint procedure with random ldquore-sponsesrdquo generated by a computer Although the simula-tion generated L and R with equal probabilities (05) ldquore-inforcementrdquo was obtained on only 29 of its trials whichwas somewhat lower than the level attained by pigeonsThe reason is that of the 256 possible eight-response se-quences only 72 contained exactly four Rs and four Ls

676 NEURINGER

and therefore the random simulation often failed tomeet the 4L + 4R constraint The implication was that ifthe birds attempted to respond randomly they would bereinforced only rarely

To test whether the penalty for fifth pecks was respon-sible for the birdsrsquo failure to vary Page and Neuringer(1985) permitted eight responses across L and R keyswithout the four responses-per-key requirement If thecurrent sequence of eight responses differed from that inthe last trial a lag 1 variability contingency (without con-straint) reinforcement was provided Now the pigeonssucceeded in varying attaining reinforcement on morethan 90 of the trials a rate significantly higher thanthat observed in Schwartz (1982) and close to that of thesimulating random responder under the new procedurePage and Neuringer also replicated Schwartzrsquos (1982)variability + constraint procedure and again the pigeonsfailed just as did Schwartzrsquos The evidence was clearUnder Schwartzrsquos (1982) procedure pigeons failed tovary but penalty for fifth responses was responsible

One additional aspect of Schwartzrsquos (1982) proceduremight also have contributed to low variability If thebirds intermixed a single preferred sequencemdashone thatmet the 4L + 4R constraintmdashwith any other then 50 ofthe trials would end with food this level of intermittentreinforcement being sufficient to maintain responding inmany other situations For example if a bird alternatedbetween LLLLL (the fifth L producing a timeout) and adominant LLLLRRRR sequence the latter would alwaysresult in reinforcement Birds under the variability +constraint contingencies in fact emitted a dominant se-quence with high probability and many of their rewardswere obtained following that sequence (because it hadbeen preceded by a different sequence often one con-taining five Ls or five Rs) Thus failure to vary could beexplained by the fact that variability was not differen-tially reinforced whereas a single dominant sequencewas Schwartz (1988) later reviewed this new evidenceand concluded that pigeons could indeed be reinforcedfor responding in random-like fashion

Yoke ControlOne of Schwartzrsquos (1982) hypotheses is worth further

consideration When variability is reinforced there areperiods during which reinforcement is withheld becauseresponses did not satisfy the contingencies As will bediscussed below many studies have shown that extinctionelicits variability Might short-term extinction periodswhen reinforcers were not forthcoming have elicited thevariability attributed to reinforcement

Page and Neuringer (1985) compared two conditionsIn one referred to as Var reinforcement depended on se-quence variations similar to the lag 1 variability contin-gencies described above (The matrix of lights was nolonger used because it was found not to contribute to re-sponse variability) In the other condition referred to asYoke exactly the same frequency and distribution of re-inforcements were provided but independently of se-

quence variations This was accomplished by having thecomputer maintain a record of which trials were rein-forced and which led to timeout in the Var phase andusing this record to program reinforcers during Yoke Fora particular bird for example the first as well as the sec-ond trials in a Var session might have been reinforcedbut in the third trial the bird may have repeated a se-quence causing a timeout and so forth In the Yoke phasethe first and second trials would be reinforced but thethird trial would lead to timeout whether the birdrsquos re-sponses varied or not Each birdrsquos own sequence of rein-forcements and timeouts in Var was therefore replicatedin Yoke If short-term extinction or withholding of rein-forcement was responsible then variability should beequal in the Var and in the Yoke phases That was not theresult Variability was significantly higher in Var than inYoke a finding that has been confirmed many times inour laboratory and in those of others (eg Barba amp Hun-ziker 2002 Machado 1992) We conclude that variabil-ity was in fact reinforced

VARIABILITY IS AN OPERANT

Control by reinforcement is the most notable charac-teristic of an operant response but there are others includ-ing precise influence of contingencies effects of rein-forcement frequencies control by discriminative stimulilawful distribution of choices and extinction Researchin each of these areas supports the hypothesis that be-havioral variability is an operant dimension similar toother operant dimensions An organism can be rein-forced not only for repeating a particular behavior but fordistributing responses across a wide array of possibilitiesin a way that may be quite unpredictable The evidencewill be reviewed

Contingencies and Frequencies of ReinforcementContingencies specify what must be done to earn re-

inforcement When applied to variability contingenciesspecify a set of responses and in most studies to date aminimal level of variability within that set That controlby contingencies does not depend on reinforcement fre-quency was demonstrated by Blough (1966) for pigeonIRTs and by Machado (1989 1992) for pigeon responsesequences across L and R keys In both cases when re-inforcement depended on high variability response vari-ability was high when intermediate variability was re-quired the level generated was intermediate and whenlower levels of variability were reinforced variabilitywas relatively low Reinforcement frequencies were heldconstant in these studies and therefore were not respon-sible for the changes in the levels

With respect to other operant dimensions such as re-sponse rate reinforcement frequency often has complexand sometimes inconsistent effects As reinforcementfrequency increases response rates sometimes increasesometimes decrease and sometimes are unchanged Vari-ability is also affected in seemingly inconsistent ways

OPERANT VARIABILITY 677

Some studies report that variability increases when rein-forcement frequencies decrease (Boren Moerschbaecheramp Whyte 1978 Tatham Wanchisen amp Hineline 1993Tremont 1984) but others report small or no effects(Blough 1966 Eckerman amp Lanson 1969 Herrnstein1961 Machado 1989)

Grunow and Neuringer (2002 Experiment 1) inde-pendently manipulated variability contingencies and re-inforcement frequencies and their results may help toexplain the previous inconsistencies Four groups of ratswere studied with the contingencies for each group re-quiring a different minimum level of variability (A thresh-old contingency was employed the details of which willbe described below) In the initial phase reinforcementwas provided whenever the respective contingencieswere met (reinforcement frequency was relatively highin this phase) and as can be expected from the Blough(1966) and Machado (1989) studies described above lev-els of variability were directly controlled by the contin-gencies In a second phase with the same variabilitycontingencies held constant reinforcement was limitedto no more than once per minute (VI 1 or intermediatefrequency) In a third phase meeting of the variabilitycontingency was reinforced on an average of only once

every 5 min (VI 5 or low frequency) We observed a sig-nificant interaction between reinforcement frequencyand variability contingency (Figure 2) which helps toexplain some of the previous inconsistencies In thegroup that was reinforced for low variability representedby the triangles in Figure 2 as reinforcement frequencydecreased variability increased slightly in the group re-inforced for high variability represented by the squareswhen reinforcement decreased variability decreasedand in the intermediate groups variability was unchangedAlthough these effects were small they reached statisti-cal significance As can be seen in Figure 2 variabilitycontingencies affected variability more than did rein-forcement frequencies and effects of the latter were in-fluenced by the contingencies

An example might help to illustrate these complexfindings When low variability is required perhaps as in assembly-line work low variability emerges When highvariability is required perhaps as in artistic work re-sponse variability is high Now what happens if the normalrate of reinforcement for each of these tasks is loweredAccording to Grunow and Neuringerrsquos (2002) findingsthe assembly-line workerrsquos variability would increasebut only slightly and the artistrsquos variability would de-

Figure 2 Average U-values across groups of rats in Grunow and Neuringer (2002)The highest function (squares) is from a group reinforced for highest sequence vari-ability the lowest function (triangles) is from a group reinforced under a lenient con-tingencymdashthat is low variability sufficed for reinforcement the other groups (Xs andcircles) were reinforced for intermediate levels of variability Each group was rein-forced at three frequencies each time the contingency was satisfied (continuous rein-forcement ie CRF) on an average of not more than once per minute (VI 1) and onan average of not more than once every 5 min (VI 5) High U-values indicate high lev-els of sequence variability and low U-values indicate relatively repetitive respondingError bars represent standard errors of the mean

678 NEURINGER

crease somewhat Importantly the artist would continueto behave much more variably than the assembly-lineworker Both contingency and frequency affect responsevariability but contingency appears to be more influen-tial at least over the range of values studied to date Thesefindings if general are important for those who withholdreinforcement in order to encourage response variationsas is often the case when behavior modifiers attempt toshape new behaviors Effects of withholding may dependon baseline contingencies and a more productive proce-dure might be to reinforce variability directly We willconsider this suggestion further below

Discriminative ControlThe operant response as defined by Skinner is con-

trolled by discriminative stimuli as well as by conse-quences For example red traffic lights are the discrimi-native stimulus for depressing the automobile brake thedistance of the golf ball from the hole provides a discrim-inative stimulus for how hard to hit the ball In the labora-tory discriminative control is often studied with multipleschedules in which one reinforcement contingency suchas fixed ratio is operative in the presence of one stimulussuch as a red keylight and a different contingencymdashforexample extinctionmdashis operative in a second stimulusmdashfor example a green keylight Discriminative control isshown by differences in responding Variability is alsocontrolled by discriminative cues

Exteroceptive discriminative stimulus controlPage and Neuringer (1985 Experiment 6) reinforced pi-geons for repeating a single sequence LRRLL in thepresence of blue keylights and for variable sequences inthe presence of red Blue and red alternated after every10 reinforcements The birds learned to repeat in thepresence of blue and to vary in the presence of red andwhen the stimulus relationships were reversed the birdsreversed their patterns Cohen Neuringer and Rhodes(1990) obtained similar results with rats Denney andNeuringer (1998) extended this research to show thatdiscriminative control did not depend on reinforced repe-titions in one of the components Again within the contextof a multiple schedule rats were rewarded for variablesequences under one stimulus (Var) and independentlyof variability under a second stimulus (Yoke) Frequen-cies of reinforcement were identical in the two stimulusconditions That the stimuli exerted significant controlover variability was indicated by three main findingsVariability was higher in Var than in Yoke differences invariability were greatest immediately following stimuluschange and when the discriminative stimuli were re-moved and all other aspects of the program kept constantvariability immediately converged on an intermediate andidentical level in the two components The conclusion isthat operant variability is controlled by discriminativestimuli These findings are important in terms of both ex-plaining observed levels of variability and controllingthem For tasks in which response variations are impor-tant such as when new operants are being shaped or when

an individual is attempting to behave creatively or solveproblems cues that indicate ldquoVaryrdquo can be helpful

Discriminative control without exteroceptive stim-uli The relationship between a response and a reinforcercan itself provide a discriminative stimulus even whenexternal cues such as lights or tones are unavailable Forexample under an FI schedule there is a period duringwhich reinforcers cannot be obtained The subject learns topause following each reinforcement although no cuesother than the reinforcer are provided Another case is themixed schedule identical to the multiple schedule describedabove but with no external cues Animals sometimeslearn to respond differentially to the consequences oftheir behaviors even under such stimulus-impoverishedconditions (Ferster amp Skinner 1957) The question stud-ied next was whether responsendashreinforcer relationshipscan provide discriminative cues sufficient to control lev-els of variability This is an important question becauseas was indicated above it has been assumed that with-holding reinforcement elicits variability The present hy-pothesis is that absence of reinforcement can sometimesserve as a discriminative cue for operant variability Thedistinction between elicited (respondent) and reinforced(operant) variability will be revisited shortly

Hopson Burt and Neuringer (2002) rewarded ratsunder a mixed schedule in which two periods of time orcomponents Var and Repetition (Rep) alternated with-out any external cues In the Var component four-responsesequences of L and R leverpresses were reinforced ifthey met a threshold variability contingency in the Repcomponent only repetitions of LLLL were reinforced(Probabilities of reinforcement were equalized in the twocomponents by intermittent reinforcement of LLLL inRep) The results demonstrated discriminative controldespite the absence of exteroceptive cues (see Figure 3)With trials since the unmarked switch into the Var com-ponent shown in the left column variability increasedand LLLL sequences decreased with trials since the unmarked switch into Rep shown in the right columnresponding became increasingly repetitive until LLLLsequences predominated The two functions were sim-ilar which is an important finding in that it indicatesdiscriminative control rather than elicitation If variabil-ity had been elicited because reinforcement was tem-porarily withheld (when the contingencies were switched)then variability should have increased following entryinto both Rep and Var components but that was not theresult

Therefore variability that is observed when rein-forcement is withheld for short periods as when a newresponse is being shaped may be partly discriminativelycontrolled despite the absence of external cues that isanimals and people may learn when it is functional tovary and some of the cues may come from responsendashoutcome relationships Control over this operant vari-ability may be more precise than that elicited by extinc-tion because as was noted above operant contingenciesdefine both the set of acceptable variations and the level

OPERANT VARIABILITY 679

of required variability It may also be maintained muchlonger because as will be seen shortly extinction weakensresponding

ChoiceThe frequency of one response relative to others or

response probability is used as a measure of choice orpreference Choices are found to match the probabilitiesof their consequences For example relative frequenciesof pecking one of two keys approximate relative fre-

quencies of reinforcement Neuringer (1992) showedthat the same matching relationship also governs choicesby pigeons to vary or repeat The more variations werereinforced the more the pigeons chose to vary and sim-ilarly for repetitions Each trial consisted of four L andR responses and reinforcements for varying and for re-peating were systematically manipulated The Var con-tingency was met whenever the current sequence dif-fered from each of the previous three sequencesmdasha lag 3criterion The Rep contingency was satisfied if the cur-

Figure 3 The left column shows the probabilities ( y-axis) that each of five rats met a variability contin-gency (solid circles) versus a repetition contingency (open circles) when they were being reinforced for vary-ing (Hopson Burt amp Neuringer 2002) Along the x-axis are trials since the mixed schedule (no cues pro-vided to indicate the component) had switched into the variability-reinforcing component The right columnshows the corresponding data for performances when reinforcement was contingent upon repetitions thex-axis here indicating trials since the unmarked switch into the repetition-reinforcing component

680 NEURINGER

rent sequence repeated at least one of the previous threeOn an average of once every 30 sec (VI 30 sec) the com-puter probabilistically primed reinforcement for Var orRep the prime being maintained until the reinforcementhad been collected Thus if the computer primed Var re-inforcement was delivered whenever the lag 3 variabilitycontingency was next met Similarly if Rep had beenprimed the next Rep sequence was reinforced Proba-bilities of Var and Rep reinforcements were systemati-cally manipulated the two always summing to 10 and amatching-type relationship was obtained (Figure 4panel A) The relationship was more complex than sim-ple matching possibly because if the pigeons chose tovary by responding stochastically there was some prob-ability due to chance that the current sequence wouldrepeat a previous sequence This was shown by a com-parison of pigeon performances to the performance of acomputer-simulated stochastic model programmed tomatch its Var choices to the probability of Var reinforce-ments The pigeonsrsquo choice distributions were found tobe the same as the modelrsquos (Figure 4 panel B) Thuschoices to vary or repeat were governed by relative fre-quencies of reinforcement An animal can choose whetherwhen and how much to vary a result consistent withthose described above from Blough (1966) Machado(1989) and Grunow and Neuringer (2002) Choices ofwhether or not to vary appear to be governed similarly toall other operant choices

These f indings may have important applicationsWhen an individual repeatedly responds in a way that isnonfunctional reinforcement of choices to vary mayhelp the individual to emerge from the ineffective pat-tern This can be true for those experiencing everydayproblems but feeling helpless to change and it may beespecially important for those experiencing depressionor who respond in otherwise abnormally stereotypedmanners such as those with autism These topics will beconsidered in a later section

ExtinctionAs was indicated above extinction is often found to

increase variability but almost all studies of this effectinvolve responses that had previously been repetitiveAntonitis (1951) for example reinforced rats for pokingtheir noses anywhere along a 50-cm horizontal openingAlthough the exact location of pokes did not matter withcontinued training responses became limited to one or afew locations along the strip When reinforcement waswithheld variability of location increased (see also Eck-erman amp Lanson 1969) Variability produced by extinc-tion has been reported for many other response dimen-sions as well including force (Notterman amp Mintz1965) number (Mechner 1958) topography (Stokes1995) and sequence (Balsam Paterniti Zechowy ampStokes 2002 Mechner Hyten Field amp Madden 1997)

The issue is complicated however by other studiesthat show that previously reinforced responses and re-sponse patterns are maintained intact during and afterextinction That is extinction does not break down

learned patterns For example when Schwartz (1981) re-inforced pigeons for any sequence of four L and four Rresponses each bird developed a dominant sequence asdescribed above and these sequences were maintainedduring extinction Despite decreasing response ratesduring extinction response patterns were retained Sim-ilar maintenance of response structure has been ob-served during extinction following reinforcement of

Figure 4 Average performance across a group of pigeons inNeuringerrsquos (1992) study of choices to vary or repeat (A) Loga-rithms of the ratio of number of times the pigeons met the Varcontingency divided by the times they met the Rep contingencyas a function of the analogous numbers of reinforcements re-ceived for each Points represent group averages in two separateexperiments Also shown is the least squares best-fitting line(B) Changes in U-value an index of response variability as afunction of the obtained percentage of reinforcements for vary-ing The solid line in panel B represents the performance of acomputer-based model that matched its choices to vary (vs re-peat) to its obtained relative frequency of reinforcement for vary-ing From ldquoChoosing to vary and repeatrdquo by A Neuringer 1992Psychological Science 3 p 248 Copyright 1992 by the AmericanPsychological Society Reprinted with permission

OPERANT VARIABILITY 681

choices (Myerson amp Hale 1988) responding underfixed-interval contingencies (Machado amp Cevik 1998)visual discriminations (Nevin 1967) and matching tosample (Cumming Berryman Cohen amp Lanson 1967)How can stability during extinction be reconciled withincreased variability

Neuringer Kornell and Olufs (2001) wrestled withthis question in a study of extinction following rein-forcement under three different contingencies An oper-ant chamber contained two levers left (L) and right (R)on either side of a reinforcement dispenser and a key (K)in the rear of the chamber distant from the dispenserOne group of rats Rep was reinforced for repeating afixed sequence of responses LKR A second group Varwas reinforced for variations across the three operandaunder a threshold contingency Reinforcement for a thirdgroup Yoke was matched to those of the Var animalsand was therefore independent of the particular sequencesBecause of the unusual configuration of operandaskewed distributions of sequence probabilities were ob-served in all of the groups an outcome that was antici-pated and desired in order to document possible in-creases in variability during extinctionmdasha flattening ofthe distributionmdashas well as possible decreases That isduring the reinforcement phase the Rep group came torepeat the LKR sequence with high probability the Yokegroup tended to emit the easiest sequences such as LLLand RRR but distributed their responses over more se-quences than the Rep group did and the Var group dis-tributed most widely although because of differences inthe operanda and in their locations some sequences weremore preferred than others and in particular sequencesending with a K response were least likely (the key beingfarthest from the reinforcement dispenser) Thus thecontingencies resulted in three different distributions orhierarchies of sequence probabilities very steep forRep intermediate for Yoke and flattest for Var

The main finding was that withholding reinforcementhad similar effects in the three groups In each responserates fell to low levels across the four sessions of extinc-tion Furthermore sequences that were most frequentduring the reinforcement phase continued to be most fre-quent throughout extinction A slight flattening of thedistributions was observed however with the lowestprobability sequences increasing relative to the rein-forcement phase Thus because the overall hierarchy ofsequence frequencies was maintained the results indi-cated stability but because the distributions were some-what flattened the results also showed slightly increasedvariability It is as if the strategy during extinction wereto keep doing what had worked in the past but occa-sionally to do something very different This compli-cated story integrates two effects of extinction increasedvariability and maintained stability Most importantly forthe present discussion extinction had similar effects onvariable yoked and repetitive operants

One additional comparison is of interest Recall thatGrunow and Neuringer (2002) found that when contin-

gencies demanded high variability obtained levels ofvariability decreased slightly as reinforcement frequen-cies were lowered This result is opposite to the Vargrouprsquos slight increase in variability just described underextinction conditions The reason may be that extinction-induced variability is short lived lasting only as long asresponding continues perhaps for hours or a few ses-sions at most whereas the variability engendered by theVI 5 intermittent schedule of reinforcement in Grunowand Neuringer can be maintained indefinitely The de-crease in Grunow and Neuringerrsquos operant variabilitymay be due to the weakening of the reinforcement effect(ie reinforcement is less frequent) whereas the in-crease following extinction may be a short-term elicitedeffect This distinction between reinforced and elicitedvariability is important and will be discussed further inthe next section

Operant and Respondent InteractionsElicited or what is sometimes referred to as respondent

effects occur when environmental changes that affectbehavior do not depend on (are not caused by) the behav-ior Pavlovian conditioning being the most commonlystudied example Operant effects are observed when a re-inforcer depends on the response to produce it Althoughfor historical reasons respondent and operant effects areoften studied separately most behaviors result from acombination or interaction of the two Variability is nodifferent and we turn to an example of such an interaction

As reinforcement is approached under FI or fixed-ratio (FR) schedules response rates are commonly ob-served to increase Similarly when reinforcement de-pends on accuracy as in discrimination learning ormatching to sample it too is found to increase within theFI or the FR (Nelson 1978 Nevin 1967) Cherot Jonesand Neuringer (1996) asked whether the same patternheld for reinforced variability Does operant variabilityincrease as reinforcement for varying is approachedThe question may be related to such real-world cases aswhen a composer is rushing to complete a compositionbefore a scheduled performance or when a programmeris trying to solve a problem before an upcoming meetingThese activities involve at least to some extent non-repetitive original or variable thought and behaviorThe question is whether variability increases with prox-imity to or anticipation of reinforcement for varying

Cherot et al (1996) employed trials consisting of fourresponses across L and R operanda and a lag 4 variabil-ity contingency with rats in one experiment and pigeonsin another Unlike in any of the cases described so farhowever the lag-variability contingency had to be satis-fied four times in order for a single reinforcer to be ob-tained With ldquordquo indicating successful completion of theVar contingency ldquo2rdquo indicating failure to meet that con-tingency and PELLET indicating the food reinforcer anexample follows LRLL RRRL LRLL2 LRLL2RLLR RRRL2 LRRR+PELLET Thus food reinforce-ment was provided only after the subject had satisfied

682 NEURINGER

the variability contingency four times FR 4 The resultswere that as the food reinforcer was approached (ie asprogress was made through the ratio) the subjects wereless and less successful at varying The probability ofmeeting the variability contingency in the first trial afterreinforcement was higher than it was in the second whichin turn was higher than that in the third trial and so onFor comparison a Rep group was reinforced for repeatingsequences once again with reinforcement only after foursuccessful repetitions Probability of repeating increasedwith proximity to reinforcement a finding consistentwith the literature but opposite that for the variabilitygroup Thus approach to reinforcement facilitated oper-ant repetition but interfered with operant variabilitySimilar variability-interfering effects have been docu-mented for within-trials responses The probability that aresponse repeats the just prior response increases within atrial even when sequence variability is reinforced (McElroyamp Neuringer 1990 Neuringer 1991) To return to the ex-ample of a composer rushing to complete her composi-tion she may indeed have increasing difficulty respond-ing creatively under the pressure of a deadline On theother hand the soloist may find it easier to repeat thecorrect performance during practice periods as the con-cert nears How can these results be reconciled with themany findings that variability can be reinforced

An additional result of Cherot et al (1996) provides ananswer The Var animals responded much more variablyoverall than did the Rep animals Although variability inthe Var groups decreased with approach to the reinforceroverall variability was always much higher when it was re-inforced (Var group) than when it was not (Rep group)Reinforcement therefore exerts two influences Overallvariability is elevated sometimes to the highest levelswhen it is reinforced but approach to (and possibly focuson) the reinforcers constrains or lowers variability Sim-ilar reinforcement-produced interference effects had beenreported previously for example when raccoons are re-inforced for dropping a ldquocoinrdquo into a slot the reinforce-ment results in the animalsrsquo holding and rubbing the coinrather than dropping it (Breland amp Breland 1961) Theoverall enhancement of variability was of much greatermagnitude in Cherot et al than the decrease with approachwas although both effects were statistically significant(The notes created by the composer under pressure ofdeadline will be less readily predicted than will thoseplayed by the performer practicing a well-known piece)This study highlights the importance of interactions be-tween operant effects of reinforcement and elicited effectsproduced by anticipation of reinforcement and the resultsmay be relevant to the debate concerning possible detri-mental effects of reinforcement on creative behaviors (seethe discussion below and Neuringer in press)

SummaryReinforcement controls behavioral variability as it

does other operant dimensions Levels of variability areprecisely governed by schedule contingencies and dis-

criminative stimuli Choices to vary or repeat match rel-ative frequencies of reinforcement Extinction increasesvariability slightly whereas the hierarchies of sequenceprobabilities are retained and operant and respondenteffects interact The evidence shows that behavioral vari-ability is an operant controlled by its consequences

GENERALITY AND PARAMETERS

Many different methods measures species and re-sponses have been used in the study of operant variabil-ity and this diversity supports the generality of the phe-nomenon As is the case for repeated operants howeverthere are parametric constraints This section briefly re-views the generality and some of the parameters studiedto date

GeneralityMethods Throughout this section ldquoresponserdquo will

signify a response unit including both an individual re-sponse and a multiresponse sequence Each of the meth-ods to be described increases or maintains variability byreinforcing it

Novel response procedures reinforce responses upontheir first occurrence ldquoeverrdquo (Pryor et al 1969) or upontheir first occurrence within a session (Goetz amp Baer1973) Radial-arm maze procedures also reinforce ini-tial (within a given session) entries into arms of a mazeand result in a distribution of choices across the availableoptions the sequence or pattern of choices often beingquite variable

Lag procedures reinforce a response if it has not oc-curred across a given number of previous trials (Page ampNeuringer 1985) For example under lag 5 the currentsequence is reinforced only if it has not occurred duringany of the previous five trials A variant is the recurrence-time procedure which specifies a minimum number of in-tervening trials before a previously emitted response canagain be reinforced Machado (1989) combined recurrencetime with a percentile-reinforcement contingency to gen-erate high levels of variability (see also Machado 1992)Percentile-reinforcement contingencies base the criterionfor reinforcement on the subjectrsquos own performance over aprevious set of responses (see Galbicka 1994) For exam-ple a sequence will be reinforced only if it has not oc-curred over the previous n trials the value of n continuallychanging so that 50 of the trials are reinforced

Least-frequent response procedures (Blough 1966Schoenfeld Harris amp Farmer 1966 Shimp 1967) rein-force responses that are currently least frequent across amoving window

Threshold procedures reinforce responses whose rela-tive frequencies fall below a given value Because thistype of procedure was used in the research describedabove and in many other more recent studies details willbe provided In Denney and Neuringer (1998) which willserve as an example trials consisted of four responses byrats across L and R levers A running tally was kept of the

OPERANT VARIABILITY 683

number of occurrences of each of the 16 possible se-quences LLLL LLLR LLRL and so on If the relativefrequency of the current sequencemdashthe number of its oc-currences divided by the sum of all 16 sequencesmdashwasbelow a specified threshold value in this case 05 the ratwas rewarded Recently emitted sequences contributedmore to the maintained tally than did those in the distantpast because after each reinforcement all 16 counterswere multiplied by a weighting coefficient in this case95 Therefore the influence of a particular trial decreasedexponentially with successive reinforcements Thresholdprocedures differ from Bloughrsquos (1966) least-frequentprocedure in that more than one sequence (or response)can be reinforced at any given timemdashnamely all those se-quences that have relative frequencies below the thresholdvalue The procedure differs from lag in permitting agiven sequence to be reinforced many times in successionif the associated relative frequency is sufficiently low

The frequency dependence procedure (Machado 19921993) is a variant of threshold in which the probability ofreinforcement is a continuous function of relative re-sponse frequency The more frequent a response is thelower the probability is that it will be reinforced

Statistical feedback procedures provide subjects withcomparisons between their performances and that of amodel Neuringer (1986) showed human subjects feedbackfrom 10 statistical tests thereby enabling them to comparetheir own distributions of responses to that of a randommodel Neuringer and Voss (1993) provided feedbackbased on a chaotic model The goal in each of these caseswas for the subjects to learn to approximate the model andthe results of these experiments will be discussed below

Differential reinforcement of switches versus repetitionsbalance a subjectrsquos tendency to engage in each of these pat-terns Rather than reinforcement for infrequent responsesor random-like distributions reinforcement is providedsometimes for repeating a response and other times forswitching from one response to another In the two-operandum case a switch is L followed by R or R by La repetition is L then L or R then R Bryant and Church(1974) found that stochastic-like responding was gener-ated by rats when switches were reinforced with a 75probability and repetitions with a 25 probability that isthe rats had a natural tendency to repeat at least in theenvironment of this study Machado (1997) reinforcedpigeons for eight pecks across two keys if the sequenceof pecks contained at least one (one group of birds) ortwo (another group) switches Although the birds couldhave satisfied these contingencies by repeating a singlesequencemdashin the ldquoone switchrdquo condition for examplerepetition of LRRRRRR would produce reinforcement on100 of the trialsmdashthey tended to vary their sequencesBarba and Hunziker (2002) showed however that vari-ability was higher when it was directly reinforced thanwhen reinforcement was based on proportions of switchesa finding consistent with that of Machado (1997)

The evidence from each of these methods supports thehypothesis that variability can be reinforced

Measures Variability must be interpreted with refer-ence to a particular measure The reason is that there canbe high variability (and low predictability) at one level ofanalysis but order (and certainty of prediction) at a dif-ferent level For example the sequence 1 1 2 2 3 3 44 5 5 1 1 2 2 has the same variance (a commonlyemployed measure of variability) as 2 3 2 1 5 1 4 41 1 3 2 5 2 but an analysis of conditional proba-bilities in which the unit contains pairs of instances in-dicates that the two sequences have different levels ofvariability In general high variability observed with onemeasure does not rule out the possibility of order ob-served with another and vice versa That is why evalua-tions of random-number generators often employ manydifferent measures and indeed there are hundreds of dif-ferent measures of the randomness of a sequence (Knuth1969) All behavioral studies make simplifying assump-tions and rely on one measure or at the most a few

U-value has been the most commonly employed mea-sure of operant variability (Evans amp Graham 1980Machado 1989 Page amp Neuringer 1985 Stokes 1995)U-value measures the distribution of relative frequen-cies or probabilities of a set of responses For a set of16 possible responses i = 1ndash16 U-value is given by

where RFi = the relative frequency (or probability) ofeach of the 16 responses U-values approach 10 whenrelative frequencies approach equality as would be ex-pected over the long run from a random generator andthey approach 00 when a single instance is repeated

Other measures include the percentage of trials or fre-quency at which variability contingencies are met (Page ampNeuringer 1985) percentages of alternations versus stays(Machado 1992) conditional probabilities of responses(Machado 1992) the number or frequency of novel re-sponses (Goetz amp Baer 1973 Schwartz 1982) the num-ber or frequency of different responses (Machado 1997Schwartz 1982) Markov analyses (Machado 1992) andstatistical tests of distributions of responses (Neuringer1986) All of the measures employed to date indicate thatvariability can be reinforced

Responses and species Generality has further beendemonstrated by the use of many different responses andspecies These include Siamese fighting fish (Betta splen-dens) swimming in quadrants of a tank (Roots 2000)young chicks pecking keys (Neuringer 2002) pigeonspecking keys (Page amp Neuringer 1985) budgerigarssinging songs (Manabe Staddon amp Cleaveland 1997)rats pressing levers (Van Hest van Haaren amp van de Poll1989) porpoises swimming and jumping (Pryor et al1969) rhesus monkeys pressing levers (Sydney Reisbickpersonal communication) children drawing and build-ing with blocks (Holman Goetz amp Baer 1977) andhuman adults and children pressing buttons (Saldana ampNeuringer 1998 Stokes amp Balsam 2001) There havebeen few attempts to compare species or responses but

- ( )[ ] ( )=aring RF RFi ii

log log 2 21

1616

684 NEURINGER

in one study lowest baseline variability was observedwhen rats pulled trapezes intermediate levels were ob-served when they pushed keys and highest levels wereobserved when they pressed levers (Morgan amp Neuringer1990) with variability increasing in all cases when it wasreinforced In another study operant variability wascompared across rats and people with similar effects ob-served in the two species (Neuringer Deiss amp Imig 2000)

ParametersWe turn now to some parameters studied to date Many

other parameters as yet unstudied are no doubt also im-portantmdashfor example motivation and types of reinforcers

Age Older people (see eg Van der Linden Beertenamp Pesenti 1998) and older rats (see eg Lamberty ampGower 1990) respond less variably than their youngercounterparts when variable responding is reinforced or re-quested but in most cases it is not clear to what extent thevariability is elicited as opposed to reinforced Neuringerand Huntley (1992) compared reinforced variability in2-month-old (young) and 10-month-old (mature) Long-Evans rats Under Yoke contingencies the young rats re-sponded significantly more variably (two levers four re-sponses per trial) than the mature rats which constitutesan elicited effect When reinforcement was contingenton variability (lag 4) variability increased significantlyin both groups thereby overcoming the initial differencecorrelated with age Thus explicitly reinforcing vari-ability in older animals caused them to respond no dif-ferently than younger animals did a finding potentiallyrelevant to real-world human aging However when Wells(1999) compared 8-month-old (mature) to 24-month-old(old) rats although the mature rats responded more vari-ably than the old rats did under Yoke contingencies thedifference was not statistically significant Additionalstudies are needed especially of whether reinforcingvariable behaviors in older populations can facilitatelearning and performance

Gender Male and female rats are equally sensitive toreinforcement of variability (Neuringer amp Huntley 1992van Hest et al 1989) However there was a tendency(not statistically significant) for males to respond over-all more variably than females did and because thesestudies did not control for differences in size and weightof subjects or for other aspects of motivation additionalstudies are necessary

Stage of training Stokes and Balsam (2001) reporteddifferences in variability as a function of whether variabil-ity was initially reinforced early or later in training (seealso Gottlieb 1992) Human subjects generated sequencesof 10 L and R responses per trial on a computer keyboardunder lag 0 and lag 25 contingencies Under lag 0 everytrial ended with reinforcement under lag 25 the currentsequence had to differ from each of the previous 25 Whenlag 25 was experienced at the very beginning of traininglevels of variability at the end of training (when lag 0 wasoperative) were lower than they were when lag 25 was ex-perienced after 50 trials of initial lag 0 These results were

interpreted to suggest that there is an ldquooptimal period forsetting variability levelsrdquo (Stokes amp Balsam 2001 p 181)but that general conclusion needs further testing and anumber of questions need to be answered One has to dowith the contribution of reinforcement frequency Underlag 0 every sequence leads to reinforcement but rein-forcement is intermittent under lag 25 Another has to dowith whether learning to vary differs from learning to repeatwith respect to optimal periods (see eg Bouton 1994)

Related to stage of training are effects of prior experi-ence Hunziker Lee Ferreira da Silva and Caramori(2002) showed that for human subjects if Var contingen-cies were experienced just prior to Yoke contingencieslevels of response variation during the Yoke phase werehigher than they would have been if the Var contingencieshad not been previously experienced The time delay be-tween Var and Yoke experiences mattered with a long in-terval (3 months) producing only a small effect Saldanaand Neuringer (1998) also found effects of prior Var train-ing on performances in Yoke (increased variability inYoke) but the opposite did not hold that is training underYoke had relatively little effect on Var Hunziker Caramorida Silva and Barba (1998) reported similar effects withrats Thus there may be an asymmetry Reinforcement forvarying has relatively strong influences but reinforce-ment without regard to variations does not adversely affectlearning to vary These different effects are not surprisinggiven that Yoke contingencies are permissivemdashreinforce-ment is given whether or not responses varymdashwhereascontingencies under Var can be quite demanding How-ever the results motivate a number of additional ques-tions One is whether effects of Var training differ fromthose observed when repetitions are reinforced Another iswhether direct reinforcement of variable behaviors canhelp animals or people to overcome the detrimental effectsof prior noncontingent experiences such as those that pro-duce learned helplessness (Seligman 1975)

Environmental constraints Stokes (1995) arguedalso that variability increases when behavior is con-strained The term ldquoconstraintrdquo is ambiguous because allreinforcement contingencies exert a form of constraintIn one study Stokes (1995) compared different strate-gies of shaping leverpresses Rats in one group were re-inforced initially for pressing the lever with only theright paw (right only) and eventually for any press (withright or left paw or with any other part of the body) Theorder of training was reversed for a second group ldquoanyrdquofirst and then right only Behavior sequences were morevariable in the right-only-first group leading Stokes(1995) to conclude that early constraints lead to highvariability (see also Stokes Mechner amp Balsam 1999)That is an important hypothesis with many implicationsbut controls are needed before the results can be attrib-uted to the contingencies For example ldquoright onlyrdquo andldquoanyrdquo differed in terms of reinforcement distributions aswell as in the precision of the reinforcement operationthe ldquoanyrdquo reinforcers were provided immediately and au-tomatically by the programming equipment whereas in

OPERANT VARIABILITY 685

the right-only condition a human observer had to decidewhether a response had occurred and then present the re-inforcer necessarily leading to longer and more variedlatencies and a less consistent contingency Furthermorethere are many cases in which increasing constraints suchas those due to a decrease in the number of responses pertrial (Page amp Neuringer 1985) or the requirement of ex-actly four responses on each of two keys (Schwartz 1980)results in decreased variability Thus the contributionsof response-definition constraints and environmentallimitations must be explored further

SummaryThe generality of operant variability is shown by the

robustness of the phenomenon across procedures mea-sures responses and species As with other operant phe-nomena parameters influence the effects

FUNCTIONS

All operants including variability are functional Op-erant responses produce reinforcing consequences Be-cause other behavioral competencies depend on it oper-ant variability is functional in a broader sense as wellSome of these functions will now be described

Operant ConditioningThe potential of a reinforcer to change behavior is lim-

ited by the range of available variations (Hull Langmanamp Glenn 2001 Skinner 1981) and thus whatever pro-duces those variations contributes to the conditioning ofnew operants Much work has been done to identifysources of variation (Balsam Deich Ohyama amp Stokes1998 Segal 1972 Staddon amp Simmelhag 1971) butNeuringer (1993) was first to show that the same rein-forcers that strengthen individual responses can also pro-duce the necessary variability and that these two functionsof reinforcement may work simultaneously Implied isthat when an operant response is being conditioned re-inforcement works not only to strengthen an individualresponse but at the same time to engender the variabilityfrom which the response emerges (That is variability isnot solely a baseline phenomenon elicited or noise)Concurrent reinforcement of variations and individualtarget sequences will be the topic of this section

Pigeons in one experiment and rats in another were re-warded for varying among 16 possible response sequences(four responses per trial L and R operanda lag contin-gencies Neuringer 1993) Concurrently one of the 16designated as the target sequence was reinforced when-ever it occurred Thus reinforcements were given bothfor sequence variations and for a single selected targeta reinforced-variation-and-selection (RVS) procedureWhen the target was ldquoeasyrdquo (eg LLLL) it emergedrapidly from the baseline variations Given that the tar-get was always reinforced reinforcers were obtained morefrequently when the subjects did not vary and only re-peated the target When the target was ldquodifficultrdquo (eg

LLLR) it was not learned As in the choice experimentdescribed above (Neuringer 1992) the relative valuesor difficulties of target versus variations governed theprobability that the target would be learned

However even difficult targets could be conditionedunder the RVS procedure as was shown by another ex-periment in the same series (Neuringer 1993) Sequencevariations were again reinforced concurrently with a dif-ficult target RRLR but now the frequency of reinforce-ments for varying was systematically decreased Satis-fying the variability contingency was intermittentlyreinforced according to a VI schedule and over sessionsreinforcement for varying became less and less frequentThis technique resulted in acquisition of even the mostdifficult target sequence whereas a control group rein-forced for the target alone (without the concurrent rein-forcement for variations) never learned

Reinforcers were obtained more frequently in the RVScondition than in the control condition (since the RVSsubjects were rewarded both for varying and for emittingthe target sequence) and the additional reinforcers mighthave generated greater motivation to respond (Killeen1981) Thus motivation not the direct reinforcement ofvariability might have been responsible for the RVS ef-fect To test this possibility Neuringer Deiss and Olson(2000) reinforced three groups of rats for emitting a par-ticularly difficult target sequence LLRLR The experi-mental group was reinforced concurrently for sequencevariability on the average of once per minute (VI 1 min)The target-only control group received no reinforce-ments other than for the LLRLR target A third groupwas yoked to the experimental animalsmdashreinforced onceper minute for any sequence variability not requiredmdashand as for the other two groups whenever the target oc-curred If the additional reinforcement was responsiblefor the facilitated learning under RVS then because theexperimental and yoked groups received equal reinforce-ments they should have learned equally well but thatwas not the result Only the experimental group learnedto emit the LLRLR target The yoked controls respondedat high rates throughout the experiment but they did notvary sufficiently to make contact with the target contin-gency and never learned The target-only control groupreceived insufficient reinforcement and their responsesextinguished Therefore the added reinforcers helped tomaintain motivation as suggested above but reinforce-ment of variations was necessary for learning of a diffi-cult sequence The RVS procedure maintains both highmotivation and high variation and the two together re-sult in the acquisition of difficult-to-learn responses

Grunow and Neuringer (2002 Experiment 2) studiedone possibly important parameter of the RVS proceduremdashnamely the level of variability When a difficult target re-sponse is being trained does it matter whether reinforcedvariability is low intermediate or high As describedabove each of four groups of rats was reinforced for vari-able response sequences the first group for high vari-ability the fourth group for low variability and the other

686 NEURINGER

two for intermediate levels In other ways the procedurewas similar to that of the Neuringer Deiss and Olson(2000) RVS contingencies just described with a singledifficult target sequence always reinforced The resultswere that the higher the level of reinforced baseline vari-ability the more quickly the target was learned Thusvariability levels indeed mattered and if the just-re-ported effect is found to be general it will be most help-ful to reinforce high variability when difficult-to-learnnew responses are being shaped Other parameters re-quiring study are class definitionsmdashwhich responses areincluded in the class of acceptable variationsmdashand num-ber of instances within the class

One additional effect should be noted When varia-tions among a set of responses are reinforced if one vari-ant is selected for extinction (no longer reinforced) thefrequency of that instance decreases Neuringer (1993)showed this for pigeons responding under a lag-variabilityschedule When one of the 16 sequences was no longerreinforced (with no other change to indicate its status) thefrequency of that sequence decreased Thus reinforce-ment of variations provides the baseline to select for (iestrengthen) and select against (ie weaken) individualinstances

These findings have potentially important implicationsFirst as has been indicated the same reinforcers thatstrengthen particular instances can also reinforce varia-tions That a binary eventmdashreinforce or not reinforcemdashsimultaneously affects variations and selected instancesindicates both the power of reinforcement and a concep-tual problem How reinforcers exert such diverse effectsis a question to which we will return

Second shaping procedures may implicitly reinforcebaseline variations An approximation to a goal responseis reinforced for a period of time but eventually rein-forcement is withheld until a response closer to the goalis obtained It is commonly thought that withholding re-inforcement or mini-extinction induces the necessaryvariations for shaping to proceed as has been indicatedabove An alternative suggested by the present studiesis that organisms learn both to vary (within a set) and to re-peat (selected instances) until one instance is always rein-forced That is the process of shaping might involve re-inforcement both of variations (generally done implicitly)and of targeted instances If so a useful contributor tothe shaping process might be explicit reinforcement ofvariations

Third reinforcement of variations could facilitate ac-quisition of other competencies such as cognitive andmotor skills Siegler (1996) has shown in an important se-ries of studies that childrenrsquos acquisition of complex cog-nitive skills is correlated with high variability Anotherexample is that when motor skills are practiced in a vari-able manner high quality of performance results (Ma-noel amp Connolly 1995 1997 Mechner 1992 Schmidtamp Lee 1999) However it is rare that variability in thesedomains is directly reinforced as part of the training oreducational process

Fourth the RVS procedure might be used in therapeu-tic settings to modify maladaptive and self-injurioushabits Therapists sometimes employ noncontingent re-inforcement procedures to weaken undesirable behaviors(Vollmer Iwata Zarcone Smith amp Mazaleski 1993)The reasoning is that many maladaptive behaviors aremaintained by unintended consequences and that non-contingent rewards may compete with those conse-quences thereby lowering the strength of the unwantedhabit For example when a staff personrsquos attention toself-injurious behavior helps to maintain it providing at-tention noncontingently may decrease the strength of theunwanted target behavior However the RVS proceduremight accomplish the same goal more effectively andwithout undesirable side effects (Vollmer RingdahlRoane amp Marcus 1997) Noncontingent rewards some-times result in unwanted ldquosuperstitiousrdquo behaviors andthat would be avoided under RVS Furthermore RVSwould help to provide the set of baseline behaviors nec-essary to establish desirable new responses

Finally RVS procedures may help to gain control overand eventually to weaken other undesirable target behav-iors The target can be incorporated within a set of moredesirable responsesmdasha new operant class established withthe target as one membermdashand variable instances rein-forced Once variable responding is successfully main-tained including the undesirable target response the lattercan be weakened by selectively withholding reinforcement

Summary The conditioning of operant responses de-pends on both baseline variations and selection of suc-cessive approximations to a goal response The same re-inforcers used to select the approximations can reinforcebaseline variations thereby defining the precise set ofappropriate variants and the desired levels of variationSuch concurrent reinforcement of variations and selec-tions may facilitate the acquisition of difficult-to-learnoperants and selective withholding of reinforcementmay weaken undesirable instances

Creativity and Problem SolvingBehaving in a novel unusual or nonrepeating manner is

part of creative activities (Campbell 1960) Novelty andvariation may arise through combinations of previouslylearned behaviors (Epstein 1996) transformations of thesebehaviors (Boden 1994) direct reinforcement of variabil-ity as described above or some combination thereof Ofcourse neither novelty nor variability alone suffices forcreative outputmdashthe throw of the dice or the toss of acoin are not creative nor are haphazard strokes of a paint-brush on a canvasmdashbut without variation creativity islow Whether the variability required for creativity canbe reinforced is relevant to whether creativity itself canbe reinforced The latter question has led to contentiousdebate with some arguing that reinforcement facilitatescreativity (Cameron amp Pierce 1994 Eisenberger ampCameron 1996 Stokes 2001 Winston amp Baker 1985)and others focusing on detrimental effects of reinforce-ment (Amabile 1983 Deci Koestner amp Ryan 1999)

OPERANT VARIABILITY 687

The debate goes well beyond the present discussion butsome of the research described above is relevant Firstreinforcement contingent on variability generates higherlevels of behavioral variability than does reinforcementcontingent simply on responding as has been shown bythe many studies described above in which Var contin-gencies were compared to Yoke contingencies Similarlyprecise reinforcement contingencies matter for creativity(Cameron amp Pierce 1994 Lepper amp Henderlong 2000)If any product rather than only creative ones is rein-forced creativity tends to fall (just as variability decreasesunder yoke conditions) Second approach to or antici-pation of reinforcement interferes with operant variabil-ity (Cherot et al 1996 McElroy amp Neuringer 1990Neuringer 1991) which again might be related to effectson creativity when an individual focuses on the rein-forcer Note however that although variability decreaseswith proximity to the reinforcer overall levels of vari-ability are much higher when variability is reinforced thanwhen it is not The same may be true for creativity aswell anticipation of reward may interfere with creativeoutput to some extent but at the same time high creativ-ity may depend on its reinforcement

Operant variability may be important for problemsolving as well Here too there is disagreement as towhether training is helpful or not Early studies by Maier(1933) indicated that it is and these found support inlater work (eg Lung amp Dominowski 1985) Somestudies showed that those individuals most likely toldquobreak the setrdquo are also most likely to solve a problem(Dover amp Shore 1991) However a number of studiesfound that practice had no influence (Duncan 1961)

Arnesen (2000) explored a related phenomenon witha rat model One group Var was reinforced for variableinteractions with novel objects and another group Yokewas reinforced independently of such interactions EachVar rat received training with 12 different objects oneper session For example in the presence of a soup canthe rat was reinforced the first time it entered the canbut thereafter a different response was required such asrolling it climbing on it and so forth This procedure issimilar to that of Goetz and Baerrsquos (1973) described aboveexcept that objects changed from session to session Ratsin the Yoke control group had identical experiences in anadjoining identical chamber that contained identical ob-jects but now rewards were not contingent on object ma-nipulation After this training the rats were tested in adifferent room with 30 never-before-experienced ob-jects scattered on the floor Hidden in each was a smallpiece of food When compared to their Yoke controls theVar animals were more active during this test exploredmore manipulated objects more and most importantlydiscovered significantly more food pellets Another con-trol group received no preliminary training with objectsand their performance in the test session did not differfrom that of the yoked controls Therefore when ratswere reinforced for variable interactions with objectsthe result was an increased tendency to explore manip-ulate and discover new sources of reinforcement It

would be of considerable interest to determine whethera similar effect occurs with children

Assessment and Treatment of PsychopathologiesAbnormal levels of behavioral variability characterize

some human psychopathologies as has been shown byclinical observations as well as by research with humansubjects and animal models Few studies have attemptedto reinforce variability directly in such problem popula-tions The main question is whether reinforcement canmodify abnormal levels of variability in the direction ofnormalcy and whether such changes might have associ-ated beneficial effects

Autism Stereotyped responding characterizes indi-viduals with autism (Hertzig amp Shapiro 1990) For ex-ample Baron-Cohen (1992) asked children with autismand others in a control group to hide a penny in one handso that the experimenter could not guess which hand heldthe penny Those with autism were more likely thanthose without to generate a simple predictable patternsuch as repeatedly switching back and forth from left toright hands Such repetitive responding can be maladap-tive For example Mullins and Rincover (1985) askedchildren with and without autism to pick one of five cardsFood was occasionally found in a cup behind the cardEach of the five cards was a discriminative stimulus fora different schedule of reinforcement including contin-uous reinforcement (CRF every choice reinforced) FR 2(fixed ratio 2 in which every second choice was rein-forced) FR 4 (every fourth choice reinforced) FR 7 andFR 11 Control participants sampled all five alternativesand quickly learned to choose the most frequently rein-forced CRF card Autistic children sampled only a lim-ited number of cards and therefore often preferred anonoptimal alternative Thus children with autism tendto respond less variably than normal children and thiscan result in failure to adapt

Miller and Neuringer (2000) asked whether direct re-inforcement could influence the variability of individu-als with autism Sequences of responses on two largebuttons were first reinforced probabilistically and inde-pendently of variability (Prob phase) after which rein-forcement was contingent upon satisfying a thresholdvariability contingency (Var phase) The autistic partic-ipantsrsquo variability was significantly lower than that ofmatched controls during both Prob and Var phases butwhen variability was reinforced its levels increased sig-nificantly in both groups This is an important findingbecause it suggests that direct reinforcement of variabil-ity may help to modify some of the behavioral stereo-types that are characteristic of autism Lee McComasand Jawor (in press) offer support for this hypothesisThree individuals with autism were reinforced for variedverbal responses to social questions the responses had tobe appropriate given the context Variability increased intwo of the three individuals In a related procedureBaker and Koegel (1999) reinforced children with autismfor varied social interaction in a game-playing contextAlthough the reinforcement was permission to engage in

688 NEURINGER

the stereotyped behaviors preferred by the child the re-sult was increased social interactions both during theperiod of the experimental manipulation and in 1- and2-month follow-ups These three studies taken togethersuggest that reinforcement of varied behaviors may facil-itate modification of nonfunctional ritualistic and stereo-typed behaviors

Depression Depression provides another example ofmaladaptive consequences of low variability Lapp Mar-inier and Pihl (1982) observed that depressed womenproduced fewer alternative solutions to hypotheticalproblems than did nondepressed women Channon andBaker (1996) found that attempts by moderately de-pressed college students to identify faults in a series ofinterconnected circuits were less varied and less success-ful that were those of nondepressed controls HorneEvans and Orne (1982) found that when depressed pa-tients were asked to generate a random sequence of num-bers their responses were less variable than were thoseof controls These experimental results are consistentwith clinical observations

Can depressed individuals be rewarded for increasingvariation Hopkinson and Neuringer (in press) dividedcollege students into mildly depressed and nondepressedgroups on the basis of the CES-D scale a paper-and-pencil self-evaluative index of depression In the first oftwo phases sequences of responses on a computer key-board were reinforced independently of variability (Prob)following which only highly variable sequences were rein-forced (Var) Under Prob depressed students respondedsignificantly less variably than did the controls in agree-ment with previous findings Under Var variability in-creased significantly in both groups Indeed by the endof the experiment depressed and nondepressed subjectswere responding equally variably (Instructions were alsoeffective in increasing variability in the depressed par-ticipants) Thus even in a simple computer-game envi-ronment mildly depressed subjects manifested lowervariability than controls did but direct reinforcementmodified their behavior to control levels Can reinforce-ment of variability be used in real-world conditions tohelp depressed individuals An affirmative answer comesfrom cognitive behavior modification procedures thatwork to increase alternative solutions to problems (Beck1976) A more general conclusion from these studies ofautistic and mildly depressed individuals is that direct-reinforcement-of-variability procedures may benefitthose who manifest abnormally low levels of variability

Attention Deficit Hyperactivity Disorder (ADHD)ADHD may provide an example of the opposite stateabnormally high levels of variability (Barkley 1990)Wultz Sagvolden Moser and Moser (1990) proposedthat the spontaneously hypertensive rat (SHR) may serveas a good animal model of ADHD the SHR shows manyof the behavioral characteristics of that disorder includ-ing hyperactivity poor learning ability poor ability totolerate delayed reinforcement and poor self-control

The next study to be described asked about operant vari-ability in SHRs

Mook Jeffrey and Neuringer (1993) compared vari-ability of SHRs and of a control strain Wystar Kyoto(WKY) under two conditions Prob and Var The mainresult was that SHRs tended to respond variably whetheror not the contingencies required it consistently with theresults from the human ADHD literature When vari-ability was reinforced the two strains responded at ap-proximately equally high levels of variation These re-sults were confirmed in two other studies (HunzikerSaldana amp Neuringer 1996 Mook amp Neuringer 1994)Thus control rats varied only when the contingencies re-quired variations whereas SHRs varied whether or notthe contingencies required it As might be expected fromthese results when reinforcement depended on repeti-tions the SHRs performed poorly (Mook et al 1993)

Another similarity between SHRs and ADHD humanswas seen when amphetamine a drug that has effects simi-lar to those of Ritalin often prescribed for ADHD was ad-ministered (Mook amp Neuringer 1994) When repetitionswere required for reinforcement amphetamine improvedperformances by the SHRs so that they were as accurate assaline-injected WKYs a finding again consistent withthose in the ADHD literature These studies support SHRsas a model of ADHD and suggest that additional attemptsshould be made to assess whether or not individuals withADHD are less able to change levels of variability thanare controls Although as indicated above there is someevidence in support of this hypothesis (Barkley 1990)when Saldana and Neuringer (1998) directly comparedhuman ADHD and control participantsrsquo operant variabil-ity within the context of a computer game no differenceswere observed Most of those with ADHD were main-tained on Ritalin and other drugs however a fact unfor-tunately not documented in the original article

Drug EffectsAlthough many drugs are known to influence behavioral

variability (Brugger 1997) rarely have attempts beenmade to analyze explain or identify the sources of theeffects As will be discussed below operant variabilitycan result from the memory-based process of remember-ing previous responses and then behaving differently orcan result from responding stochastically a process thatappears to be quite different McElroy and Neuringer(1990) attempted to distinguish the effects of alcohol onmemory from its effects on stochastic processes in astudy in which one group of rats was reinforced for vary-ing four-response sequences under a lag 5 contingency(Var) and another was reinforced for repeating a singlesequence LLRR (Rep) Ethanol injections of 075 and20 gkg affected Var and Rep performances differentlyWhereas the Rep animals were much less able to satisfythe LLRR contingency at both doses the Var rats re-sponded variably whether or not alcohol had been ad-ministered Similar results were obtained by Cohen et al

OPERANT VARIABILITY 689

(1990) with a multiple schedule in which each rat servedas its own control In the same animals and within thesame sessions alcohol degraded Rep performances butleft Var intact Thus alcohol appears to interfere with atask that requires working memory as presumably is thecase for the Rep task but not with one that reinforcesvariability The memory-interfering effect of alcoholmight account for why alcohol also degrades radial-armmaze performance (Devenport 1983)

Alcohol also affects choices to vary or to repeat For ex-ample Gorka (1993) reinforced rats concurrently for re-peating and varying and monitored percentage of choicesas a function of time following injection of ethanol Inthe control sessions when the rats were injected withsaline the probabilities of varying and repeating remainedapproximately constant across the 45-min session witha slight bias in favor of varying After injection with 15gkg of ethanol however there was a significant increasein choices to vary at the beginning of the session andthen a later decrease to below 50 That is alcohol ledinitially to a markedly increased preference for varyingbut later to a preference for repeating (Figure 5) This re-search should be replicated and extended especially be-cause the contingencies ldquopressuredrdquo the animals to re-main at approximately 50 preference but thesuggestion is that alcoholrsquos effects on response variabil-ity may depend in part on the duration of its influence Amore general implication is that drugs may affect re-peated and variable operants differently and that effectsdepend on time following administration

SummaryReinforced variability may contribute to many types

of learning including operant shaping and acquisition ofcognitive and complex motor skills Similarly creativityand problem solving depend on variable responding andin these areas as well explicit reinforcement of variationsmay be functional Abnormal levels of variability aresometimes caused by psychopathologies and drugs andoperant variability may be useful in these areas in twoways first as a way of assessing the variability that is pos-sible and second to modify variability toward more nor-mal levels Studies show that individuals with autism andmild depression vary less than do controls but that ex-plicit reinforcement of variability moves the levels towardsnormalcy Other studies show that drugs such as alcoholmay affect repeated and variable operants differently

EXPLANATIONS

Two questions must be answered in order to explainoperant variability First what are the sources of vari-ability that permit its reinforcement Second how is itmaintained Related questions including how instancesbecome members of a set from which variations emergeand whether operant variability is a generalizable com-petency will also be considered

Origination From Nonoperant SourcesSkinner (1966) reasoned that operant responses orig-

inate from nonoperant sources with reinforcement con-tacting a behavior shaping and strengthening it onlyafter the behavior had occurred for reasons other than re-inforcement To put it simply a behavior must occur be-fore it can be reinforced (Balsam amp Silver 1994) I willconsider three initial sources of variable behavior en-dogenous variability noncontingent environmental eventsand extinction These appear to parallel evolutionaryprocesses and they too will be identified

Endogenous variability All physical phenomenavary and behavior is no exception This fact can providethe opportunity for reinforcement to select particularlevels of variation Examples of spontaneous variabilityinclude spontaneous alternations in mazes (Dember ampRichman 1989) nondifferentially reinforced variationsin operant responses (Guthrie amp Horton 1946) and ex-ploration (Archer amp Birke 1983) Endogenous muta-tions provide a parallel in genetics Base-pair changes inDNA molecules occur spontaneously intron-bearinggenes mix and match segments to encode new proteinsjumping genes introduce new DNA sequences and atthe level of the chromosome random crossings and as-sortment occur as a natural part of reproductive meiosisThese sources of random genetic change occur indepen-dently of external influences

Figure 5 The relative frequency of variations by rats as a func-tion of 5-min blocks of time since the beginning of the sessionThe open circles represent performance after the rats were in-jected with saline indicating a slight tendency to vary more thanrepeat but one that is approximately constant throughout thesession The filled circles represent performance after injectionwith 15 gkg ethanol indicating a much higher likelihood ofvarying soon after the alcohol is injected followed by a decreasein variations to below 50 and then a return to baseline at theend of the session Group averages are shown Data are fromGorka (1993)

690 NEURINGER

Noncontingent events Noncontingent events some-times referred to as chance accidents or luck also affectbehavior One happens upon a particular passage in a bookthat leads to a new thought one happens to sit next to aninteresting woman or man on an airplane which leads tomarriage (Bandura 1982) In science as well as in every-day life accidents or serendipity are an important part ofthe process of discovery (Beveridge 1957) In each of thecases presented above an accidental event leads to new be-haviors that are then strengthened by consequences Thesame is potentially true for variability itself Particular lev-els of variability might occur for adventitious reasons butthen become functionally related to consequences

That organisms are particularly sensitive to such unan-ticipated noncontingent events is shown by two basicprinciples of learning habituation and Pavlovian condi-tioning Organisms attend preferentially to unexpectedevents and stop attending or habituate to repeated onesThe conditioning of new Pavlovian responses dependsaccording to influential theories upon unexpected rein-forcing events (eg Rescorla amp Wagner 1972)

Noncontingent events such as X-rays and other pro-cesses that can act as mutagens play an important role ingenetic variation as well The particular genes affectedand the type of changes are random there is no neces-sary relationship or contingency between the externalevent and the affected gene However sensitivity to mu-tations is not random but has been selected presumablybecause it is functional Thus in both behavior and ge-netics sensitivity to noncontingent events is conserved

Extinction Extinction increases behavioral variabilityat least to some extent as was described above Distance inspace or time from the reinforcer does the same (Cherotet al 1996) A possible parallel at the genetic level in-volves high levels of deprivation at least in simple lifeforms For example when E coli bacteria are deprivedof nutrients unusually high mutation rates are observedThe exact mechanism by which such hypermutations areinduced is not known but they increase the likelihoodthat a population can survive such environmental insults(Moxon 1997) That is deprivation-induced variability isfunctional at the genetic level (The behavioral evidenceregarding long-term deprivation is less clear some stud-ies showing increased variability and others increasedstereotypy eg Carlton 1962 Senkowski Vogel amp Po-zulp 1978)

Endogenous variations external noncontingent eventsand withholding of reinforcement are three sources forthe variability that precedes control by reinforcementBut once control by reinforcement is established a dif-ferent question emerges What behavioral processes areinvolved when variability is maintained as an operantThe sources just discussed generate uncontrolled non-specific and often short-lived variability and cannot ex-plain the precisely controlled operant type

Operant Variability ProcessesThree processes are hypothesized to underlie operant

variability The first involves the use of random environ-

mental events such as the toss of a coin The second in-volves memory for responses such as when an animallearns not to return to previously visited locations Thethird will be referred to as an endogenous stochasticgenerator

Random environmental events Throughout historymany cultures have used random events to influencechoices and decisions such as throws of dice and randomselections of sticks cards bones or organs Today coinflips are used by a referee at the beginning of a game to de-cide which team kicks the ball a computerrsquos random-number generator is employed by scientists to avoid bi-ases in assigning subjects to experimental groups andaleatoric events are sometimes employed in modern artmusic and literature The Dice Man by Rhinehart (1998)provides a fictional example in which the protagonistbored with life writes a number of possible actions onslips of paper and then periodically selects one blindlyand behaves accordingly These examples show that randomexternal events may be used to avoid biases engender un-likely responses and break out of behavioral niches

Memory-based variability Many methods used tostudy operant variability differentially reinforce nonrepe-titions at least over the recent past Therefore one strat-egy that subjects can employ is to cycle through few re-sponses or sequences especially when the variabilitycontingencies are relatively permissive For exampleunder a lag 2 contingency cycling repeatedly throughthree sequences will result in 100 reinforcement Ani-mals and people sometimes do this apparently basingtheir current response on memory for or discriminativecontrol by the just emitted responses One such instancewas shown by Schoenfeld et al (1966) who rewardedrats for IRTs that fell into one of 10 interval classes onlyif the current IRT differed from the interval class of theprevious response The rats came to alternate betweenlong and short IRTs thus behaving systematically to meetthis lenient variability contingency Machado (1993)studied an analogous effect having pigeons peck L and Rkeys under a frequency-dependent variability contin-gency related to Shimprsquos (1967) procedure describedabove When the trial unit consisted of two responses thebirds developed memory-based repetitive sequencessuch as RRLLRRLL and thereby satisfied the variabilitycontingency When the trial unit was three responses inlength the birds did not develop the optimal fixed pat-tern of RRRLRLLLmdashwhich was too complexmdashbut in-stead developed ldquorandom-like behaviorrdquo (Machado1993 p 103) Thus a memory-based strategy was em-ployed when it was possible but when the memory loadbecame too high stochastic responding emerged A sim-ilar pattern was seen when variability of budgerigarsongs was reinforced (Manabe et al 1997) Under lag 1contingencies the birds sang mainly two songs underlag 2 they increased to three songs Only when the lagwas increased to 3 did song diversity increase apprecia-bly in one of the birds In radial-arm mazes as well whencontextual cues are available memory again appears toplay an important role leading to much lower probabil-

OPERANT VARIABILITY 691

ities of repetition than would be expected from a sto-chastic generator (McElroy amp Neuringer 1990)

Note that under lag schedules memory-based strate-gies result in higher reinforcement rates than respondingstochastically does The reason is that a stochastic gen-erator repeats by chance That animals and people oftendo not employ memory-based strategies probably indi-cates the difficulty of remembering long sequences andpossibly other advantages of unpredictable responding

If a memory strategy is the only way to satisfy vari-ability contingencies then it can be used even when thememory requirements are demanding as was shown byNeuringer and Voss (1993) Human subjects were trainedto generate on a computer keyboard a chaotic-like se-quence of responses that matched iterations of the logistic-difference equation

where Rn refers to the nth iteration in a series with R be-tween 00 and 10 and t is a constant between 10 and40 When the t parameter approaches 40 iterations ofthis equation result in very ldquonoisyrdquo sequences As withother chaotic processes however there is an orderlinessunderlying the surface noise In this case if responsesare autocorrelated with Rn drawn as a function of Rn21the resulting function is a parabola In the Neuringer andVoss (1993) study following each response college stu-dents were shown how close their predicted values wereto that from iterations of the logistic-difference model Inone experiment the students made digital predictions inanother they placed a pointer along a horizontal lineand this analogue response was then translated into dig-ital values for computation With training the studentsbecame increasingly adept at responding in chaotic-likefashionmdashthe studentsrsquo predictions matched closely theiterations of the logistic functionmdashand their autocorrela-tions became parabolic The left column in Figure 6 showsperformances by 4 subjects (rows) early in training thatis not well approximated by the logistic-differenceparabola and the right column shows performances at theend of training in which r 2 values were 72 99 98 andapproaching 10 How was this accomplished Sinceeach iteration in the logistic-difference sequence is basedon the prior outputmdashthat is the sequence is generated bya deterministic although nonlinear processmdashit was hy-pothesized that the human subjectsrsquo chaotic-like responseswere also based on memory for prior responses To put itsimply the subjects may have memorized a long series ofpairs in the form ldquoIf the previous response was value Athen the current response must be value Brdquo as Metzger(1994) Ward and West (1994) and Ward (2002) have sug-gested

To test whether subjects in fact relied on memory forprior responses Neuringer and Voss (2002) systemati-cally slowed responding by interposing pausesmdashthat isby requiring an IRT As IRT increased the difference be-tween the subjectsrsquo sequences and the modelrsquos chaotic

Rn n nt R R= -( )- -1 11

Figure 6 Autocorrelations (responses on trial n on the y-axis asa function of n21 on the x-axis) for 4 human subjects at the be-ginning (left column) and end (right column) of training inNeuringer and Voss (1993) Equations for the best-fitting curveand amount of variance accounted for (r2) are shown at the topof each panel To the extent that the circles (responses) approxi-mate the equation y = 40x 2 40x 2 the subjects were respondingin a way that matched the logistic-difference chaotic functionFrom ldquoApproximating Chaotic Behaviorrdquo by A Neuringer andC Voss 1993 Psychological Science 4 p 115 Copyright 1993 bythe American Psychological Society Reprinted with permission

692 NEURINGER

output increased and the underlying parabolic structurewas broken That is when pauses increased subjects wereless and less able to approximate the modelrsquos chaotic dis-tribution (see Figure 7) The important point here is thathighly variable responses which in this case are chaoticin nature can be based on memorial processes

Endogenous stochastic generator The evidence tobe reviewed next indicates that animals and people canrespond stochastically and that this competency is basedneither on chance environmental events nor on memoryfor prior events or responses It is impossible to provethat an endogenous stochastic process underlies operantvariability that would be analogous to proving the nullhypothesis However much evidence is consistent withthe stochastic hypothesis Included are the facts thatmemory limitations do not permit adequate respondingunder many variability contingencies that response dis-tributions match those expected from a stochastic gener-ator that changes in parameter values affect operantvariability as they would a stochastic generator and thatinterference with memory processes leaves stochasti-cally based operant variability unaffected As a wholethe evidence is consistent with an endogenous stochas-tic source of operant variability

Memory limitations One type of evidence is nega-tive in character and is based on the presumed inability ofmemory for prior responses or sequences to satisfy sometypes of variability contingencies Two examples weregiven above (Machado 1993 Manabe et al 1997) andanother was provided by a lag 50 contingency in Pageand Neuringer (1985) In the latter case the sequence in

the current trial (eight responses across L and R keys) wasrequired to differ from those in each of the previous 50 tri-als including at the beginning of a session the last 50trials from the previous session Pigeons responded wellunder these contingencies and in a way similar to that ofa stochastic model It is unlikely that pigeons can remem-ber each of their previous 50 eight-response sequences

Response slowing As was shown above for chaotic-like responding performances are often degraded by de-lays imposed between a stimulus and a response pre-sumably because of limitations in working or short-termmemory (Berryman Cumming amp Nevin 1963 Williams1971) Neuringer (1991) relied on this effect to test whethermemorial processes could explain the generation of vari-able sequences under lag schedules Two groups of ratswere reinforced for L and R leverpresses producing fourresponses per trial One group (Var) was trained under alag 5 contingency and the other (Rep) was trained to asingle sequence of responses LLRR It was assumedthat accurate Rep performance depended on workingmemory After performances had stabilized IRTs weresystematically varied by the imposition of timeouts of05 to 20 sec between consecutive responses in the se-quence During these forced pauses the chamber wasdark and responses were not counted toward completionof the sequence If the animals had used the previous re-sponse(s) as a discriminative cue for the current responsethen performances should have been degraded by theseinterpolated pauses

The results were that for the Rep group as the re-quired pauses increased from 0 to 6 sec percentages ofcorrect sequences were relatively unaffected but at pausesgreater than 6 sec probability of a correct LLRR se-quence fell sharply This pattern of results suggests in-volvement of working memory of prior responses withthe rats able to remember responses within a 6-sec win-dow The Var grouprsquos results were quite different Aspauses increased between 0 and 6 sec the rats were moreand more likely to satisfy the variability contingenciesand with pauses greater than 6 sec percent correct re-mained at a high and constant level Thus the Var grouprsquosresults were opposite to those of the Rep group and op-posite to those predicted from a memory-based strategySimilar results showing that operant variability increasesor is maintained as responding is slowed have been re-ported by Morris (1987) with pigeons and by Baddeley(1966) with random response generation by human sub-jects The results reported above concerning alcoholrsquos ef-fects on variability and repetition were also consistentwith those of Neuringer (1991)

Memory for prior responses may not have controlled theVar grouprsquos performance but why did variability increaseEach of three answers is consistent with a stochastic-generator interpretation Weiss (1964 1965) hypothe-sized that random responding requires that current re-sponses be independent of previous ones memory for orcontrol by prior responses interferes with such indepen-

Figure 7 Approximation to a chaotic sequence (given by delta)as a function of the imposed interresponse times Shown are theaverages of 5 subjects

OPERANT VARIABILITY 693

dence and therefore pauses would be expected to facil-itate variability A second possibility is that at shortIRTs animals tend to repeat responses on the sameoperandum Blough (1966) found this in pigeons and ex-cluded such double pecks from his analyses because theyappeared not to be under the control of reinforcement con-tingencies Morris (1987) also found a tendency for birdsto repeat when no interresponse timeouts were imposedA third hypothesis is that there were two contributors tothe observed variability One was a stochastic processcontrolled by reinforcement and the other elicited vari-ability generated by interposition of pauses The high vari-ability elicited as an effect of slowed responding is ageneral phenomenon supported in many other cases Ac-cording to this interpretation operant variability in the Vargroup was governed by a stochastic-based process operantrepetition in the Rep group was governed by a memory-based process and pauses elicited variability under bothcontingencies Thus Rep performance was interferedwith whereas Var was facilitated Each of these hypothe-ses is consistent with the conclusion that memory for (ordiscriminative control by) prior responses does not con-tribute to and possibly interferes with variable respond-ing when an organism is reinforced for variability andthis in turn is consistent with the stochastic-generatorhypothesis

Stochastic response distributions Other evidence iscorrelational Response distributions are similar to thosefrom a stochastic generator The data from two experi-ments already discussed are shown in Figure 1 In thesecases reinforcement of least frequent IRTs (Blough 1966)and of nonrepetitive choice sequences (Machado 1989)resulted in stochastic-like responding

Neuringer (1986) reported another striking case Highschool and college students generated sequences of 1sand 2s on a computer keyboard with each set of 100 re-sponses constituting one trial A baseline phase lastedfor 60 such trials for a total of 6000 responses duringwhich the students were asked to behave as randomly aspossible as if they were reporting the outcome of a se-ries of coin tosses As in the many previous studies ofthis sort the subjectsrsquo responses differed significantlyfrom those of a stochastic model (Brugger 1997 Wage-naar 1972) During the training phase the subjects re-ceived feedback which enabled them to compare their per-formances with that of a stochastic model first accordingto one statistical test and then according to another untilfollowing each trial feedback was provided from 10 dif-ferent statistics The results were that all of the subjectslearned to approximate the stochastic model accordingto the 10 statistics as well as some (but not all) other sta-tistical tests That is the distributions of the statisticswhich clearly did not match a stochastic model at the be-ginning of training approximated the stochastic modelat the end of it (see Figure 8)

The previous failures of other researchers to obtain ev-idence for random responding had led some to concludethat humans did not have the capacity to behave ran-

domly What would account for the difference betweenthe present positive finding and the many other negativeones Neuringerrsquos (1986) study was the first in which sto-chastic responding was explicitly reinforced in this casewith feedback from 10 statistical tests An analogy mighthelp to explain the difference in procedures and resultsImagine an experimenter walking around a city and ask-ing people if they know what a violin sounds like (analo-gous to asking people whether they know what a series ofcoin tosses or dice throws is like) Most would answeryes If the same individuals were then asked to play theviolin very few if any would succeed The experimentermight conclude after many replications with differentsubjects in different cities that people did not have thecapacity to play the violin The point of course is thatplaying the violin is a skill requiring training and prac-tice and the same may be true for responding randomlyIt is easy to vary but practice and reinforcement may benecessary to approximate a random distribution

Response number Page and Neuringer (1985) testedthe stochastic-generator hypothesis in another way byvarying number of responses per trial with pigeons work-ing under a lag 3 contingency The reasoning was that ifprevious responses served as a discriminative cue forwhat not to domdasha memory-based strategymdashthen perfor-mance should be degraded when the number of re-sponses was increased Eight responses per trial requiresubjects to remember more than four responses The sto-chastic hypothesis predicts the opposite outcome as isdemonstrated by the following example If each trialwere two responses in length with responses determinedby the toss of a coin then the probability that a given trialwould repeat the previous one would be 25 (There arefour possible sequences in the first trial RR RL LRand LL Thus the second trial has a 1-in-4 chance ofmatching the first) If a trial consists of four coin tossesthe probability of a repetition by chance is 0625 or 1 in16 Thus if subjects used a stochastic process to gener-ate Ls and Rs performances should improve with in-creasing responses per trial Results were precisely thosepredicted by the stochastic hypothesis and inconsistentwith a memory strategy Probability of meeting the con-tingencies and therefore probability of reinforcementincreased as responses per trial increased In otherwords eight-response trials were easier for pigeons thanfour-response trials

Response interference Neuringer and Voss (2002)attempted to distinguish between memorial and stochas-tic processes in yet another way They hypothesized thatwhereas chaotic responding was based on memory forprior responses (as was discussed above) responding inNeuringerrsquos (1986) study with feedback from 10 statisti-cal tests indicated involvement of an endogenous sto-chastic generator To show that human subjects could en-gage in both processes at will three college studentswere trained to generate chaotic sequences in one compo-nent of a multiple schedule as in Neuringer and Voss(1993) and to generate stochastic sequences in a sepa-

694 NEURINGER

rate component as in Neuringer (1986) The wordCHAOTICS was on the screen in the first component theword STOCHASTICS was on during the other and thesewords alternated throughout the session After the sub-jects succeeded at each of the component tasks test ses-sions were introduced in which memory for prior re-sponses was subject to interference in both the chaoticsand stochastics components as follows During the

chaotics component four separate chaotic sequenceswere cycled one after the other each cued by differentscreen colors The subjects attempted to approximate achaotic sequence in one color (eg light green) follow-ing which they would have to match a second chaotic se-quence (lavender screen color) and so on the fourchaotic contingencies following one another across thesession Although all four chaotic sequences used the

Figure 8 Performance by one subject at the end of training to respond stochastically in Neuringer(1986) Shown are comparisons on 6 different statistics between the subject (solid lines and Xs) andthe random model (dashed lines and closed circles) At the end of training the subjectrsquos perfor-mance did not differ statistically from that of the random model on these and four additional sta-tistics From ldquoCan people behave lsquorandomlyrsquo The role of feedbackrdquo by A Neuringer 1986 Jour-nal of Experimental Psychology General 115 p 71 Copyright 1986 by the American PsychologicalAssociation Reprinted with permission

OPERANT VARIABILITY 695

same logistic-difference equation that a model did theywere out of phase and independent If chaotic respond-ing is memory based then the four independent se-quences would be expected to interfere with one anotherthat is approximations to the four chaotic models wouldbe lower than when only a single chaotic output was re-quired as in Neuringer and Voss (1993) That was ex-actly the finding Performances by the three subjectswere signif icantly degraded during the four-segmentphase To put it simply chaotic responses interfered withone another

A different pattern of results was seen in the stochasticsportion of the experiment The same interference proce-dure was used with four independent stochastic sequencesrequired in the four different color conditions As inNeuringer (1986) subjects were required to satisfy eightdifferent statistical tests of randomness and to do this in-dependently in each of the four phases (If responses in anytwo or more of the four stochastic phases became corre-lated above a minimal level responses were considered tobe incorrect thereby prohibiting repetition of a single se-quence in two or more of the phases) For one subjectseven of eight statistics were closer to those of a randommodel during this interference phase than they were at theend of the original training with only a single sequenceand for a second subject all eight statistics were closerThe third subject showed closer-to-random approximationunder the four-segment condition on only two of the eightstatistics but most of her changes were also in the directionof a random generator For example if a random generatoremits equal numbers of ldquouprdquo and ldquodownrdquo sequences andthis subject in the baseline phase had emitted only 47ldquouprdquo sequences then during the four-segment tests shecame to emit ldquouprdquo 55 of the time She changed in the di-rection of the random model but overshot the mark Thuswhereas interference degraded chaotic performance sig-nificantly in all three subjects it improved stochastic per-formances in at least two of them These results are con-sistent with the hypothesis that a memory-based processcontrols chaotic variability and that a stochastic processnot dependent on memory controls stochastic variability

Summary of Tripartite TheoryThe following three processes constitute a tripartite

theory of operant variability attention to random events inthe environment (coin toss) memorial processes (chaosradial maze) and stochastic processes (response se-quences and random-number generation with feedback)Random environmental events assure unpredictabilitymdashfor example in making unbiased selections Remember-ing what was done previously will enable one to avoidrepetitions to vary the students called upon in class tovary the locations when looking for a hidden object orto try different approaches when attempting to solve aproblem External aids such as notes to oneself or sci-entific records help to enhance memory capacities

An endogenous stochastic generator provides the thirdand most controversial source of behavioral variability

When an opponent is attempting to predict your behav-ior in a game or at war it may be functional to behavestochastically Similarly operant conditioning of new re-sponses may depend on stochastic variations It is im-possible to prove stochasticity some presently unknowndeterministic system may ultimately explain the resultsHowever the evidence is consistent with the stochastichypothesis and inconsistent with the alternatives For ex-ample memory is not sufficiently powerful to explain alloperant variability memory-interference proceduresleave stochastic responding intact and behaviors corre-late well with those of stochastic models

Both genetics and reinforcement play important rolesin all three processes Reinforcement is always involvedin that learning provides information about when whatwhere and how much to vary Genetic influences areshown in that not all responses are possible and if theyare possible they are not all equiprobable (eg as inNeuringer et al 2001) Stochastic responding may be bi-ased due to experiences and knowledge that is biasesmay be established by reinforcement contingencies or bythe interface between the organismrsquos propensities (or be-havior systems see Timberlake amp Lucas 1989) and theenvironment In the case of random-number generationby human subjects biases may be due to misconceptionsconcerning what ldquorandomrdquo means However some bi-ases can be overcome through feedback and instructions(Lopes amp Oden 1987 Neuringer 1986)

The importance of variations for evolutionary changeis supported by the many sources of genetic variabilityand by the fact that genetic variability is conserved Asimilar argument holds for operant variability That thereare at least three independent sources of operant varia-tions as well as other elicitors such as environmentalnoise and response extinction supports its importancefor normal adaptive functioning

Structure of VariabilityEvery operant ldquoresponserdquo is actually an instance of a

class with each member of the class exerting the sameeffect on the environment (Skinner 1959) When a rat isreinforced for pressing a lever for example the class isdefined by closure of a microswitch and any way of ac-complishing the closure whether with the left paw theright paw or the mouth is as effective in producing re-inforcement as any other If the contingencies of rein-forcement so specify however the response class can beredefined and circumscribed which is another charac-teristic of the operant For example if only right-pawpresses are acceptable members of the reinforced classresponding will tend to occur only with the right paw(Stokes 1995) Of course the capabilities of the organismplay a role for example including ldquotail pressesrdquo in the op-erant class does not mean that the lever would be or couldbe pressed by the ratrsquos tail The structure of the operantclass therefore emerges from the current reinforcementcontingencies and the organismrsquos capability derivedfrom genetics development and prior experiences

696 NEURINGER

Operant variability has structure as well and as for alloperants that structure is due partly to the organism it-self and partly to the reinforcement contingencies Theresearch described throughout this article has providedexamples of the structure of variability generally in theform of sets of four or eight responses across L and Roperanda But the structure of variations can be speci-f ied more precisely For example Mook and her col-leagues (Mook et al 1993 Mook amp Neuringer 1994)reinforced rats for varying four response sequences acrossL and R levers under a lag 1 contingency Unlike in pre-vious studies however the acceptable class was furtherconstrained so that only sequences beginning with two Rresponses (RRLL RRLR RRRL and RRRR) were rein-forced The rats learned to vary their sequences withinthe imposed limitations that is they generally begantheir sequences with two R responses and varied the se-quences within those limitations Similar results wereobtained in the context of a radial-arm maze in whichonly a subset of arms was baited

The precision with which reinforcement contingen-cies can specify class was highlighted in research per-formed by Burns (1997) and Ross and Neuringer (2002)In the latter case college students drew rectangles on acomputer screen with the computerrsquos mouse Each rec-tangle constituted one trial (the screen cleared at the endof each trial) with reinforcement (points) depending onsimultaneous variations along three dimensions of therectangle location on the screen area of the rectangleand shape with the latter defined as the ratio of the rec-tanglersquos height to its width Although the subjects weregiven no instruction other than to ldquogain pointsrdquo theylearned to vary along all three dimensions with levels ofvariability significantly higher than those of yoked controlsubjects who were reinforced independently of variations

A second part of the Ross and Neuringer (2002) studydemonstrated even greater precision of control To gainpoints three new groups of subjects were required to re-peat along one of the dimensions while simultaneouslyvarying along the other two One group was required torepeatedly draw rectangles of approximately the samesize while simultaneously varying location and shapeanother group had to repeat the shape while varying areaand location and the third to repeat location while vary-ing size and shape Verbal instructions were minimal asthey were in the first part of the study Subjects learnedto satisfy these contingencies as well varying two at-tributes while simultaneously repeating one and signif-icant differences were obtained rapidly within 300 trials

These demonstrations raise questions as to how binaryfeedbackmdashreinforcement presented or notmdashcan simul-taneously control independent behavioral dimensionsThe issue is similar to that discussed above concerningsimultaneous reinforcement of variations and selectionsIt is similar also to those issues faced by researchers at-tempting to understand how the nervous system strength-ens particular synapses when many different neurons firesimultaneously and to ldquoallocation of creditrdquo in artificial

learning models For present purposes suffice it to notethe exquisite precision with which reinforcement controlsvariability and repetition

GeneralizationDo animals and people learn to ldquovaryrdquo implying a

generalizable skill or do they learn a specific distributionof specific responses across particular operanda Thequestion of generalization has been asked of many otherbehavioral competencies such as imitation When chil-dren learn to imitate do they learn stimulusndashresponseassociations (if mother moves her right hand the childlearns to move his right hand) or do they learn moregenerally to ldquoimitaterdquo Evidence indicates that reinforce-ment of imitative instances leads to generalized imitation(see eg Baer amp Sherman 1964 Bandura 1986) A sim-ilar question has been raised when pigeons rats or chil-dren learn to choose a stimulus that matches a previouslyshown sample For example if a pigeon is shown a reddisk as a sample it must then choose red when given achoice between that and blue Does the bird learn to matchIn this case the answer appears to depend on the train-ing procedures If trained with few stimuli such as redgreen and blue then the pigeon learns simple associa-tions (if red sample choose red if blue sample chooseblue) but if hundreds of different stimuli (pictures) areemployed during training then generalized ldquomatchingrdquois learned (Wright 1997) In each of these examples thelearning of a general competency or strategy was shownby transfer-of-training tests What was learned with oneset of stimuli or in one situation transferred to new stim-uli and situations We turn now to the question of whetherlearning to vary transfers

After the subjects in Maltzmanrsquos (1960) experimentdescribed above learned to generate remote verbal as-sociates they were asked to devise unusual uses of com-mon objects In comparison with those in the controlgroup these subjects generated significantly more un-usual uses indicating transfer of training to a differenttask However transfer was not observed in the oppositedirection from unusual uses to verbal associates

Holman et al (1977) using a procedure similar to thatof the Goetz and Baer (1973) research described aboverewarded preschool children verbally for novel drawingsand found increased novelty Unrewarded block construc-tions were also monitored and weak evidence was ob-tained for transfer of novel responding from rewardeddrawing to the unrewarded constructions In a second ex-periment preschoolers engaged in four independent taskspainting pen drawing wood block construction and Legoconstruction When the children were reinforced for vary-ing their pen drawings the novelty of paintings increasedalong with the pen drawings indicating transfer from onetask to the other However transfer to the topographicallydifferent block- or Lego-building tasks was not observed

Eisenberger and Armeli (1997) gave monetary rewardsto one group of fifth- and sixth-grade students the vari-ability group for describing unusual uses of common ob-

OPERANT VARIABILITY 697

jects A second group the repeat group received thesame rewards for describing common uses of the sameobjects In a later test all of the children were asked todraw pictures incorporating a circle and the group that hadpreviously been reinforced for varied responses drewmore unusual (or lower frequency) pictures than did therepeat group Thus there was transfer of novelty trainingfrom one domain to another with additional studies in theseries showing similar effects (Eisenberger Armeli ampPretz 1998 Eisenberger Haskins amp Gambleton 1999Eisenberger amp Selbst 1994) However Eisenberger et al(1999) failed to replicate the transfer-of-training effectThe variability and repeat groups responded similarly dur-ing the test phase In this study separate experimenterstrained and tested the subjects thereby avoiding possibleexperimenter-expectancy effects whereas the same ex-perimenter did both training and testing in Eisenbergerrsquosother studies Across the series of experiments there weremany other cases in which transfer was not observed andmore research is necessary to explain why

In Lee et alrsquos (in press) study with autistic individualsdescribed above after variable answers were trained toquestions verbal variability transferred to new experi-menters and new contexts

Arnesenrsquos (2000) results described above are consis-tent with transfer of training in an animal model Rats inthe Var group were trained to interact variably with oneset of objects When later tested in a new situation withnew objects the Var animals showed (in comparisonwith controls) higher levels of exploration manipulationof the novel objects and discovery of hidden reinforcers

Much of the evidence reviewed in this section there-fore indicates transfer of training and supports a generalldquovaryrdquo competency However there remain unresolvedquestions the most important of which is the extent towhich transfer occurs Note also a methodological pointMany of the studies tested whether transfer occurs whenthe contingencies are neutral with respect to the to-be-transferred behavior Holman et al (1977) is a case inpoint Variations were reinforced in one domain draw-ing and the question was whether variations wouldoccur in another domainmdashblock constructionmdashin whichreinforcement did not depend on varying A negativef indingmdashin this case absence of transfermdashindicatesnothing about the ability to generalize but only aboutthe tendency to do so given the absence of a reason (orcontingency) A better test would be to compare experi-mental and control performances when variability in thenew task is in fact functional as in Arnesen (2000)

SummaryBefore variability can be reinforced it must occur for

reasons other than reinforcement Some of the reasonsinclude variability inherent in all behavioral phenomenaldquoaccidentalrdquo environmental events that are unrelated toparticular behaviors and withholding of reinforcementTransfer from other situations may serve a similar functionFor example if an organism can discriminate shaping-

of-new-response periods from others then it may learngenerally to ldquovaryrdquo during shaping episodes After rein-forcement makes initial contact with response variationsthere are at least three behavioral strategies that can beemployed in order to maintain functionally high levelsof variability respond based on external random eventssuch as the toss of a coin avoid repetitions by remem-bering what responses had previously been made and re-spond stochastically Evidence for an endogenous sto-chastic process was seen in the similarity of responsedistributions to those generated by stochastic sourcesand by findings that interfering with memory for priorresponses either improves or leaves unaffected stochas-tic behaviors Stochastic responding as well as othertypes of variability-generating strategies depend uponcontingencies to define the required level of variabilityand the set from which variations emerge

IMPLICATIONS

One implication of reinforced variability is that oper-ant responses may differ in the extent to which their vari-ability is affected by reinforcement contingenciesmdashthatis operants may be ordered along a continuum of sensi-tivity of variability to reinforcement Another is that re-inforced variability causes us to question the generalityof some basic goals of behavioral psychology includingprediction and control A third is that reinforced vari-ability may help us to provide a scientifically useful wayto describe voluntary action In particular the variabilityof all voluntary behaviors can readily be changed mo-ment to moment by discriminative stimuli and conse-quences Implications for everyday life also will be noted

Operant ContinuumAs was indicated above the operant is composed of

instances that is the operant is a class of many differentresponses all functionally equivalent but often topo-graphically distinct Skinner suggested and I hypothe-size that instances within the class emerge stochastically(see Moxley 1997 for a discussion of Skinnerrsquos posi-tion) Although reinforcement contingencies and contextinfluence the individual probabilities and their distribu-tions emission is stochastic nonetheless The particularmember of the operant class generally cannot be predictedwith a high degree of certainty For example a particu-lar rat may be more likely to press the lever with its leftpaw than its right but left- and right-paw responses areemitted probabilistically This class-nature of the operantsuggests one way to conceptualize operant variability interms of size of class and within-classes probability dis-tributions High variability implies large classes equalprobability distributions or both low variability impliesthe opposite

Variability may be modified more readily in some op-erants than in others implying that operant variability isnot an all-or-none phenomenon but rather a continuumwith reinforced variations in variability (RVV) differing

698 NEURINGER

for different operants RVV might best be understood byanalogy to the concepts of speed and acceleration Bothvariability and speed range from low to high RVV isanalogous to acceleration the rate of change in speedHighly sensitive RVV is indicated by the possibility ofreinforcing many different levels of variability from rep-etition to random-like responding and by the fact thatchanges in level occur readily and rapidly

Different operants manifest differences in RVV Forexample variability of chain pulls changed more thanthat of leverpresses when variability was reinforced inrats (Morgan amp Neuringer 1990) Some operant-like re-sponses may show relatively low RVV For example fir-ing rates increase in single hippocampal nerve cellswhen dopaminergic substances are contingent upon theresponse (Stein Xue amp Belluzzi 1994) suggesting thatnerve f iring is controlled by reinforcement To datehowever there is no evidence that variability of nervefiring is similarly controlled Other cases might includereinforcement of nondiscriminable (subconscious) re-sponses such as Hefferline and Pererarsquos (1963) reinforce-ment of subaware thumb twitches and biofeedback ofmuscle and cardiovascular activity In these cases rein-forcement may modify performance but it is not knownwhether variability is also sensitive to reinforcement andif so to what degree Human verbal variability may bethe most readily manipulable of all behaviors the rangeof predictable to unpredictable verbal responses beingimmense with extraordinary sensitivity to context andconsequence RVV may also differ across organisms (per-haps from simple to complex) individuals (perhaps asan index of intelligence or learning ability) and physio-logical and psychological states (WKY vs SHR depres-sion and autism)

RVV may help to distinguish operant responses fromother superficially similar cases of selection by conse-quences That is the ldquooperantrdquo may also best be concep-tualized as a continuum with RVV an important deter-miner of position along the operant continuum Oneexample of selection by consequences but low ldquooperant-nessrdquo is genetic change Variability is essential for evolu-tionary change of course and levels of genetic variabil-ity are sometimes affected by environmental conditions(Moxon 1997) but there are few examples of levels ofvariability being influenced by selection pressures Thatis the source of much genetic variability is thought to berandom not influenced by current pressures and needsThus although selected by consequences genetic changemay be low along the operant continuum Other exam-ples of operant-like phenomena for which variability although important may be relatively insensitive to con-sequencesmdashvariability in these cases is elicited not se-lectedmdashinclude the development of the central nervoussystem and immune system responses (Hull et al 2001)In both of these cases change depends upon selection byconsequences but it is not clear whether variability canalso be selected Most commonly studied operant re-sponses on the other hand show high sensitivity of vari-

ability to reinforcing events But to reiterate the mainpoint of this section operants differ in terms of RVV andtherefore in terms of their ldquooperantnessrdquo from low sensi-tivity (possibly responses of single nerve cells) to extra-ordinarily high sensitivity (verbal behavior) The operantnature of variability as well as that of every other oper-ant dimension is best conceptualized as a continuum

Prediction and ControlSmith (1992) described Skinnerian operant condition-

ing theory as Baconian in nature a primary goal of Skin-nerian behaviorists being to control behavior and hecontrasted this with the more Aristotelian goal of de-scription Skinner and his students (eg Zuriff 1985) in-deed emphasized prediction and control Many of thestudies reviewed above however involved reinforce-ment procedures that increase variability and conse-quently decrease predictability It is not only ignorancetherefore that restricts prediction and control as Skin-ner suggested but reinforcement itself Operant meth-ods and theories when applied to variability lead togreater emphasis on description with control over pre-cise response characteristics possible under some butnot all conditions

In summary an operant is a class of responses underthe control of contingencies of reinforcement The classcan be controlled and predicted given knowledge ofconditioning history Indeed the size of the class andpossibly the distribution of within-classes probabilities orlevel of variability can also be controlled and predictedBut to the extent that instances emerge stochastically aparticular instance cannot be predicted The instant re-sponse can be explained as a member of a particularclass but prediction of the instance will be diff icultsometimes to the extreme This point will be developedin the next section

Voluntary BehaviorI hypothesize that voluntary behaviors are operants for

which RVVs are highly sensitive to influence by conse-quences One way to test this hypothesis would be togather empirical data concerning how voluntary versusinvoluntary actions are classified by human observers Asfar as I know there are no psychophysical data concern-ing this question but the hypothesis implies that highsensitivity of variability to operant contingencies RVVswill be present in all actions perceived to be voluntary

This operant-variability theory of voluntary action isbased on the supposition of two attributes First volun-tary acts must be potentially explainable (given suff i-cient knowledge) on the basis of genetics conditioninghistories current contingencies anticipated conse-quences intentions goals and the like That is volun-tary behaviors are lawful The second attribute is that thevoluntary act must be potentially unpredictable That isvoluntary acts are ldquofreerdquo of control by identifiable stim-uli If behavior is perceived necessarily to result frommdashto be predicted on the basis ofmdashexternal events includ-

OPERANT VARIABILITY 699

ing reinforcement contingencies it will not be perceivedas voluntary ldquoVoluntaryrdquo implies potential indepen-dence from stimulusndashresponse or responsendashreinforcerdetermination According to this theory both explain-ability and unpredictability are necessary attributes ofvoluntary acts

How though can behaviors be simultaneously ex-plainable and unpredictable Reinforcement contingen-cies play two essential roles Reinforcement (togetherwith genes prior experiences and current environment)helps to determine the set of possible behaviors fromwhich a voluntary act emergesmdashthat is the operant classReinforcement (together with the other contributorsspecified above) also helps to determine the level of vari-ability within the class Thus both instances comprisinga class and level of variability within that class can bepredictedmdashagain given sufficient knowledge Howeverthe individual response is at least sometimes (or relatively)unpredictable The reason is that the operant class fromwhich the response stochastically emerges is sometimesextremely large (eg ldquoname a wordrdquo or ldquodo somethingrdquo)and the contingencies sometimes call for equiprobablestochastic emergence (eg ldquofool merdquo) In the latter casea voluntary act may appear to be foolish meaninglessor unwarranted but if it is voluntary it can potentially beexplained (For example in response to the questionldquoWhy did you do that silly actrdquo the answer could beldquoJust to show that you canrsquot predict my behaviorrdquo seeScriven 1965) In these cases prediction of instancesfalls to very low levels indeed and I hypothesize evenan all-knowing observer will be unable to predict thenext response at greater-than-chance levels which aregiven by the size of the operant class and by within-classesprobability distribution Even within small classes (egin the game rockndashscissorsndashpaper in which unpredictabil-ity among the three possible responses is functional) be-havioral stochasticity may lead to unpredictability of theinstant response Note however that although voluntarybehaviors are sometimes unpredictable the opposite isoften the case When contingencies call for highly con-strained particular responses predictions can easily bemade For example if a police officer asks your nameyour response can most likely be predicted Voluntary re-sponses are therefore described as potentially unpre-dictable with variability contingencies engendering dif-ferent levels of unpredictability

According to this theory behaviors low in the sensitivity-of-variability dimension are described as relatively in-voluntary and those that are high in said dimension areperceived as voluntary Low RVV may be shown at anylevel along the variability continuum For example al-ways responding in the same unchanging way will beperceived as involuntary but so will always respondingrandomly (The coin does not voluntarily come up withheads or tails) This point is essential More than 2000years ago Epicurus argued that occasional randomswerves of atoms provided the source of voluntary be-havior in an otherwise determined universe but al-

though this conjecture was prescient it was not quitecorrect ldquoFree-willedrdquo or voluntary action whether it isbased on random swerves or on quantum mechanics isnot explained by variability per se but by functionalchanges in levels of variabilitymdashthat is by rapid andprecise changes in response to environmental demandsResponses can be placed along a voluntaryndashinvoluntarycontinuum isomorphic to the operantndashrespondent con-tinuum with voluntary behavior being functional pre-cisely in the same way as operant behavior is ThusSkinnerrsquos (1974) suggestion that voluntary behavior isoperant behavior was almost correct Only those oper-ants for which variability is highly sensitive to conse-quences will be perceived as voluntary

One additional example might help Operant classes(sets of potential responses) are created ldquoon linerdquo as we goabout our daily activities As we walk from car to officewe tend to walk in a characteristic way take the samepath with tightly constrained operant classes followingone after the other in rapid succession Class sizes aresmall and within-classes probability distributions arehighly peaked Predictions of next responses by a knowl-edgeable observer can readily be made However classsizes can be increased and probability distributions flat-tened we can walk in quite strange ways indeed dancealong a new path sing hand candy to passers-by Strangebehavior indeed but made possible by the ability of op-erant sets to be modified moment to moment not only interms of class definition but in terms of class size lev-els of within-classes variations and so on The same istrue for conversation Moment-to-moment changes inverbal class definition size and predictability of within-classes instances are hallmarks of voluntary behavior

GenerativityRelated to the definition of the operant and to our discus-

sion of voluntary action is that fact that operant behaviormdashleverpressing no less than languagemdashis by its naturegenerative This claim follows from the functional na-ture of the class of responses (all members of the classare effective in producing an outcome) and the stochas-tic emission of instances From these flow never-before-emitted instances patterns and sequences (see Epstein1996) Operants especially those high on the RVV con-tinuum are inherently generative

Self-ExperimentationVariations systematic or otherwise are a necessary part

of scientific work Neuringer (1981) and Roberts and Neu-ringer (1998) described how variations can be brought to bear explicitly on everyday life in the form of self-experiments Individuals vary aspects of their lives suchas the foods they eat social contacts and methods ofmemorizing they keep records concerning the effects ofthese manipulations and form hypotheses concerningcontrolling variables Using self-experimental methodsindividuals can learn how what when where and howmuch to vary Explicitly varying onersquos everyday behav-

700 NEURINGER

iors and thoughts may help to engender what Langer(1989 2000) has referred to as mindful livingmdashthat isnonautomatic voluntary engagement with beneficialeffects in different areas including learning The impor-tant point here is that variability can be voluntarily cho-sen maintained and modified in everyday life and thatexplicit variations may have important consequences

Implications for SocietyConsideration should be given to the importance of re-

inforcement contingencies to support variability in so-cial milieus A basic point made throughout the presentarticle is that merely permitting behavior to vary doesnot necessarily engender or maximize variability (as isshown by Yoke and Prob schedules) Analogously merelypermitting varied social behavior as in a free or laissez-faire society may not suffice to produce the desired di-versity If the results shown for individual organismscan provide useful information for groups (see eg Grottamp Neuringer 1974) then when variations are desiredsocieties might reward individuals for varying not sim-ply permit it

SummaryOperants differ in the extent to which reinforcers in-

fluence levels of variation When variability is highlysensitive to reinforcement the operant may be perceivedto be voluntary Voluntary responses have two attributesThey are functional implying that the class of responsesmdashthe operantmdashcan be predicted given sufficient knowl-edge But instances emerge stochastically implyingsome degree of unpredictability Because reinforcementof variability can generate extremely large classes withstochastic emergence of instances it suggests limitationsof the two basic goals of Skinnerian behavioral theorymdashnamely prediction and control Operant variability also hasimplications for self-experimentation and for society

RELATED AREAS

Operant variability may be important in many otherareas and I will briefly review some of these

Negative reinforcement To my knowledge there havebeen no published studies in which variability is nega-tively reinforced High variability might sometimesavoid or reduce a punisher such as loss of goods ormoney but at other times low variability could be func-tional Effects of positive and negative reinforcement onvariability may have different evolutionary bases onederived from increasing probabilities of locating sourcesof food or mates the other from avoiding confrontationsor predation (Driver amp Humphries 1988) One questionconcerns how quickly variability can be changed by neg-ative as opposed to positive reinforcement There isconsiderable evidence that loss functions differ fromgain functions (Kahneman amp Tversky 1984) possiblyindicating that positive reinforcement-of-variabilityfunctions will differ from negative ones as well

Resistance to change In some of the studies describedabove variability appeared to be more resistant tochange than repetitions were for example alcohol hadlittle effect on variable operant responses but interferedwith repeated ones Doughty and Lattal (2001) tested thegenerality of these f indings using four-response se-quences with pigeons a threshold variability contin-gency in the Var component of a multiple schedule andLRLR in the Rep component Doughty and Lattal chal-lenged stability of responding by prefeeding the birds inone experiment and providing reinforcers independentlyof responding in the other (see Nevin 1992) Operantvariability was again more resistant to change than wasoperant repetition Thus reinforcement of variabilitymight increase the stability of responding (that is its re-sistance to change) a seemingly paradoxical findingHowever Mechner (1992) suggested that this is the casefor complex skills such as piano performance Varyingpractice routines will make the performance relativelyinsensitive to unforeseen changes in the environmentCare must be taken however when variation is comparedwith repetition because they have different measuresAlso the results reported by Doughty and Lattal werebased on the performance of three pigeons and were notconsistent throughout the experiment The hypothesiswarrants further study

Attraction In some bird species complex male songsattract females (for a review see Catchpole amp Slater1995) There is evidence that male song complexity isunder the discriminative stimulus control of a receptive fe-male (Searcy amp Yasukawa 1990) That explicit reinforce-ment may play a role in such song complexity is suggestedby the Manabe et al (1997) study described above An in-teresting question for further research is the extent towhich operant contingencies contribute to the attractioncreated by behavioral variability in these and other species

Discrimination of variability Animals and people canuse the variability of a stimulus array or entropy as a dis-criminative stimulus (Young amp Wasserman 2001) Whatis the relationship between perception of variability onthe one hand and emission of variable responses on theother Do individuals attempt to match their own levelsof variation to those of others in their environments whichwould involve both perception and emission How mightdiscrimination of onersquos own level of variability influenceonersquos ability to modify such levels and do influences thatobscure discriminations therefore affect ability to change(see eg van der Meer van der Weel amp Lee 1995)Hunziker et al (2002) report that although variabilitywas successfully reinforced their human subjects couldnot correctly describe the contingencies of reinforce-ment a finding similar to those of informal observationsin other studies (eg Neuringer Deiss amp Imig 2000)but whether subjects could discriminate variability levelsrather than contingencies was not studied

Attention and awareness When attention is subject tointerference or is withdrawn from the task at hand abilityto vary is degraded (eg Baddeley Emslie Kolodny amp

OPERANT VARIABILITY 701

Duncan 1998 Evans amp Graham 1980 Neuringer amp Voss2002) It is unclear at present how attention influencesthe different processes underlying operant variabilityand in particular memorial and stochastic processesRich areas for additional study are the relationships be-tween attention awareness and consciousness on theone hand and operant variability on the other Voluntaryaction and consciousness are often related (see egLibet 1999)

Concepts Concepts are thought by many to be sets ofinstances (much like operant sets) whose structures aregraded meaning that there are more or less typical in-stances of such concepts as birds (robin is typical ostrichis not) Barsalou (1987) showed that the gradation ofthese cognitive structures is manipulable or as he putsit not stable A number of variables influence structureincluding context individual differences and points ofview Barsalou argued that these structures are createdas the need arisesmdashthat is on line by the thinking indi-vidual The structure of concepts may parallel that of op-erants and in both cases levels of within-structures di-versity or variability may be modified by consequencesOperant variability may be found in the cognitive as wellas in the behavioral domain

Play and humor Highly variable behaviors are ob-served when animals and people especially children en-gage in play and humor (see eg Fagen 1981) The re-sult of play may be acquisition of new responses (seeSutton-Smith 1975) Although not currently needed re-sponses learned during play may provide the source ofvariations for successful shaping of behaviors at anothertime This is quite analogous to the role of conserved ge-netic variants which are passed along to progeny but donot currently serve any identifiable role Play may alsoinvolve learning how to vary operantly

Reinforcement by variability The self-maintainingnature of play suggests that engaging in variable behav-iors may itself be reinforcing Indirect support comesfrom findings that varied stimulation can serve as a re-inforcer and that stimulus repetition becomes aversive(Fiske amp Maddi 1961) Possibly also related are findingsthat varying reinforcers and permitting choice among re-inforcers increase their value (Bowman Piazza FisherHagopian amp Kogan 1997) and that permitting choicesamong varied responses is also preferred to demandingfixed responses (Catania 1980)

Foraging Two issues regarding variability are presentin the foraging literature One has to do with risk prefer-ence versus avoidance (see eg Shettleworth 1998) Asecond has to do with search strategies Most reportedcases of foraging involve systematic search patterns(Stephens amp Krebs 1986) However search patterns aresometimes stochastic or highly variable (see eg Peter-son 1997) One question is whether some species can varybetween these two strategies and the challenge is to iden-tify the pressures leading to and the advantages of each

Animal training Wilkes (1996) has applied operantvariability techniques in developing training proceduresfor pet dogs More generally reinforcement of variabil-

ity may enable the gaining of control over behaviors ofanimals and as has been suggested above may be an im-portant part of the process of shaping

Models of learning Reinforced variations in variabil-ity may be important when human and animal learningis being modeled whether the models are mathematicalin form computer based or machine based (Blumberget al 2002 Brooks 1999 Sutton amp Barto 1998)

CONCLUDING COMMENT

Genes interact with environments to produce diverseanatomical physiological and behavioral phenotypesGenes and environments also provide for another level ofdiversitymdashnamely operant variability There are manysources of operant variability including random environ-mental events memory for past responses and stochasticgeneration each of which ultimately derives from ge-netic and environmental influences To the extent thatoperant variability is based on a stochastic source how-ever the behavioral results cannot be predicted fromknowledge of these influences That is organisms haveevolved to behave unpredictably The positing of sto-chasticity to explain behavior has a history extending atleast as far back as Epicurus with major contributors in-cluding Gustav Fechner (see Heidelberger 1987) EgonBrunswik (see Gigerenzer 1987) Skinner (1959) andSuppes (1984) Knowledge of the stochastic nature ofthe operant and of how it is controlled leads to a worldview that differs from genetic and environmental deter-minism with roots in the works of the scholars just men-tioned At the scientific level this view leads to the ex-perimental analysis of consequence-influenced variabilityAt the philosophical level it leads to explanations of vol-untary action At the personal level it can lead to change

REFERENCES

Amabile T M (1983) The social psychology of creativity New YorkSpringer-Verlag

Antonitis J J (1951) Response variability in the white rat duringconditioning extinction and reconditioning Journal of Experimen-tal Psychology 42 273-281

Archer J amp Birke L (1983) Exploration in animals and humansCambridge Van Nostrand Reinhold

Arnesen E M (2000) Reinforcement of object manipulation in-creases discovery Unpublished undergraduate thesis Reed College

Baddeley A D (1966) The capacity for generating information byrandomization Quarterly Journal of Experimental Psychology 18119-129

Baddeley A [D] Emslie H Kolodny J amp Duncan J (1998)Random generation and the executive control of working memoryQuarterly Journal of Experimental Psychology 51A 819-852

Baer D M amp Sherman J A (1964) Reinforcement control of gen-eralized imitation in young children Journal of Experimental ChildPsychology 1 37-49

Baker M J amp Koegel R L (1999) Transforming children withautismrsquos ritualistic behaviors into positive play interactions with sib-lings Paper presented at the meeting of the California Association ofBehavior Analysis San Francisco

Balsam P D Deich J D Ohyama T amp Stokes P D (1998) Ori-gins of new behavior In W OrsquoDonohue (Ed) Learning and behav-ior therapy (pp 403-420) Boston Allyn amp Bacon

Balsam P D Paterniti A Zechowy K amp Stokes P D (2002)

676 NEURINGER

and therefore the random simulation often failed tomeet the 4L + 4R constraint The implication was that ifthe birds attempted to respond randomly they would bereinforced only rarely

To test whether the penalty for fifth pecks was respon-sible for the birdsrsquo failure to vary Page and Neuringer(1985) permitted eight responses across L and R keyswithout the four responses-per-key requirement If thecurrent sequence of eight responses differed from that inthe last trial a lag 1 variability contingency (without con-straint) reinforcement was provided Now the pigeonssucceeded in varying attaining reinforcement on morethan 90 of the trials a rate significantly higher thanthat observed in Schwartz (1982) and close to that of thesimulating random responder under the new procedurePage and Neuringer also replicated Schwartzrsquos (1982)variability + constraint procedure and again the pigeonsfailed just as did Schwartzrsquos The evidence was clearUnder Schwartzrsquos (1982) procedure pigeons failed tovary but penalty for fifth responses was responsible

One additional aspect of Schwartzrsquos (1982) proceduremight also have contributed to low variability If thebirds intermixed a single preferred sequencemdashone thatmet the 4L + 4R constraintmdashwith any other then 50 ofthe trials would end with food this level of intermittentreinforcement being sufficient to maintain responding inmany other situations For example if a bird alternatedbetween LLLLL (the fifth L producing a timeout) and adominant LLLLRRRR sequence the latter would alwaysresult in reinforcement Birds under the variability +constraint contingencies in fact emitted a dominant se-quence with high probability and many of their rewardswere obtained following that sequence (because it hadbeen preceded by a different sequence often one con-taining five Ls or five Rs) Thus failure to vary could beexplained by the fact that variability was not differen-tially reinforced whereas a single dominant sequencewas Schwartz (1988) later reviewed this new evidenceand concluded that pigeons could indeed be reinforcedfor responding in random-like fashion

Yoke ControlOne of Schwartzrsquos (1982) hypotheses is worth further

consideration When variability is reinforced there areperiods during which reinforcement is withheld becauseresponses did not satisfy the contingencies As will bediscussed below many studies have shown that extinctionelicits variability Might short-term extinction periodswhen reinforcers were not forthcoming have elicited thevariability attributed to reinforcement

Page and Neuringer (1985) compared two conditionsIn one referred to as Var reinforcement depended on se-quence variations similar to the lag 1 variability contin-gencies described above (The matrix of lights was nolonger used because it was found not to contribute to re-sponse variability) In the other condition referred to asYoke exactly the same frequency and distribution of re-inforcements were provided but independently of se-

quence variations This was accomplished by having thecomputer maintain a record of which trials were rein-forced and which led to timeout in the Var phase andusing this record to program reinforcers during Yoke Fora particular bird for example the first as well as the sec-ond trials in a Var session might have been reinforcedbut in the third trial the bird may have repeated a se-quence causing a timeout and so forth In the Yoke phasethe first and second trials would be reinforced but thethird trial would lead to timeout whether the birdrsquos re-sponses varied or not Each birdrsquos own sequence of rein-forcements and timeouts in Var was therefore replicatedin Yoke If short-term extinction or withholding of rein-forcement was responsible then variability should beequal in the Var and in the Yoke phases That was not theresult Variability was significantly higher in Var than inYoke a finding that has been confirmed many times inour laboratory and in those of others (eg Barba amp Hun-ziker 2002 Machado 1992) We conclude that variabil-ity was in fact reinforced

VARIABILITY IS AN OPERANT

Control by reinforcement is the most notable charac-teristic of an operant response but there are others includ-ing precise influence of contingencies effects of rein-forcement frequencies control by discriminative stimulilawful distribution of choices and extinction Researchin each of these areas supports the hypothesis that be-havioral variability is an operant dimension similar toother operant dimensions An organism can be rein-forced not only for repeating a particular behavior but fordistributing responses across a wide array of possibilitiesin a way that may be quite unpredictable The evidencewill be reviewed

Contingencies and Frequencies of ReinforcementContingencies specify what must be done to earn re-

inforcement When applied to variability contingenciesspecify a set of responses and in most studies to date aminimal level of variability within that set That controlby contingencies does not depend on reinforcement fre-quency was demonstrated by Blough (1966) for pigeonIRTs and by Machado (1989 1992) for pigeon responsesequences across L and R keys In both cases when re-inforcement depended on high variability response vari-ability was high when intermediate variability was re-quired the level generated was intermediate and whenlower levels of variability were reinforced variabilitywas relatively low Reinforcement frequencies were heldconstant in these studies and therefore were not respon-sible for the changes in the levels

With respect to other operant dimensions such as re-sponse rate reinforcement frequency often has complexand sometimes inconsistent effects As reinforcementfrequency increases response rates sometimes increasesometimes decrease and sometimes are unchanged Vari-ability is also affected in seemingly inconsistent ways

OPERANT VARIABILITY 677

Some studies report that variability increases when rein-forcement frequencies decrease (Boren Moerschbaecheramp Whyte 1978 Tatham Wanchisen amp Hineline 1993Tremont 1984) but others report small or no effects(Blough 1966 Eckerman amp Lanson 1969 Herrnstein1961 Machado 1989)

Grunow and Neuringer (2002 Experiment 1) inde-pendently manipulated variability contingencies and re-inforcement frequencies and their results may help toexplain the previous inconsistencies Four groups of ratswere studied with the contingencies for each group re-quiring a different minimum level of variability (A thresh-old contingency was employed the details of which willbe described below) In the initial phase reinforcementwas provided whenever the respective contingencieswere met (reinforcement frequency was relatively highin this phase) and as can be expected from the Blough(1966) and Machado (1989) studies described above lev-els of variability were directly controlled by the contin-gencies In a second phase with the same variabilitycontingencies held constant reinforcement was limitedto no more than once per minute (VI 1 or intermediatefrequency) In a third phase meeting of the variabilitycontingency was reinforced on an average of only once

every 5 min (VI 5 or low frequency) We observed a sig-nificant interaction between reinforcement frequencyand variability contingency (Figure 2) which helps toexplain some of the previous inconsistencies In thegroup that was reinforced for low variability representedby the triangles in Figure 2 as reinforcement frequencydecreased variability increased slightly in the group re-inforced for high variability represented by the squareswhen reinforcement decreased variability decreasedand in the intermediate groups variability was unchangedAlthough these effects were small they reached statisti-cal significance As can be seen in Figure 2 variabilitycontingencies affected variability more than did rein-forcement frequencies and effects of the latter were in-fluenced by the contingencies

An example might help to illustrate these complexfindings When low variability is required perhaps as in assembly-line work low variability emerges When highvariability is required perhaps as in artistic work re-sponse variability is high Now what happens if the normalrate of reinforcement for each of these tasks is loweredAccording to Grunow and Neuringerrsquos (2002) findingsthe assembly-line workerrsquos variability would increasebut only slightly and the artistrsquos variability would de-

Figure 2 Average U-values across groups of rats in Grunow and Neuringer (2002)The highest function (squares) is from a group reinforced for highest sequence vari-ability the lowest function (triangles) is from a group reinforced under a lenient con-tingencymdashthat is low variability sufficed for reinforcement the other groups (Xs andcircles) were reinforced for intermediate levels of variability Each group was rein-forced at three frequencies each time the contingency was satisfied (continuous rein-forcement ie CRF) on an average of not more than once per minute (VI 1) and onan average of not more than once every 5 min (VI 5) High U-values indicate high lev-els of sequence variability and low U-values indicate relatively repetitive respondingError bars represent standard errors of the mean

678 NEURINGER

crease somewhat Importantly the artist would continueto behave much more variably than the assembly-lineworker Both contingency and frequency affect responsevariability but contingency appears to be more influen-tial at least over the range of values studied to date Thesefindings if general are important for those who withholdreinforcement in order to encourage response variationsas is often the case when behavior modifiers attempt toshape new behaviors Effects of withholding may dependon baseline contingencies and a more productive proce-dure might be to reinforce variability directly We willconsider this suggestion further below

Discriminative ControlThe operant response as defined by Skinner is con-

trolled by discriminative stimuli as well as by conse-quences For example red traffic lights are the discrimi-native stimulus for depressing the automobile brake thedistance of the golf ball from the hole provides a discrim-inative stimulus for how hard to hit the ball In the labora-tory discriminative control is often studied with multipleschedules in which one reinforcement contingency suchas fixed ratio is operative in the presence of one stimulussuch as a red keylight and a different contingencymdashforexample extinctionmdashis operative in a second stimulusmdashfor example a green keylight Discriminative control isshown by differences in responding Variability is alsocontrolled by discriminative cues

Exteroceptive discriminative stimulus controlPage and Neuringer (1985 Experiment 6) reinforced pi-geons for repeating a single sequence LRRLL in thepresence of blue keylights and for variable sequences inthe presence of red Blue and red alternated after every10 reinforcements The birds learned to repeat in thepresence of blue and to vary in the presence of red andwhen the stimulus relationships were reversed the birdsreversed their patterns Cohen Neuringer and Rhodes(1990) obtained similar results with rats Denney andNeuringer (1998) extended this research to show thatdiscriminative control did not depend on reinforced repe-titions in one of the components Again within the contextof a multiple schedule rats were rewarded for variablesequences under one stimulus (Var) and independentlyof variability under a second stimulus (Yoke) Frequen-cies of reinforcement were identical in the two stimulusconditions That the stimuli exerted significant controlover variability was indicated by three main findingsVariability was higher in Var than in Yoke differences invariability were greatest immediately following stimuluschange and when the discriminative stimuli were re-moved and all other aspects of the program kept constantvariability immediately converged on an intermediate andidentical level in the two components The conclusion isthat operant variability is controlled by discriminativestimuli These findings are important in terms of both ex-plaining observed levels of variability and controllingthem For tasks in which response variations are impor-tant such as when new operants are being shaped or when

an individual is attempting to behave creatively or solveproblems cues that indicate ldquoVaryrdquo can be helpful

Discriminative control without exteroceptive stim-uli The relationship between a response and a reinforcercan itself provide a discriminative stimulus even whenexternal cues such as lights or tones are unavailable Forexample under an FI schedule there is a period duringwhich reinforcers cannot be obtained The subject learns topause following each reinforcement although no cuesother than the reinforcer are provided Another case is themixed schedule identical to the multiple schedule describedabove but with no external cues Animals sometimeslearn to respond differentially to the consequences oftheir behaviors even under such stimulus-impoverishedconditions (Ferster amp Skinner 1957) The question stud-ied next was whether responsendashreinforcer relationshipscan provide discriminative cues sufficient to control lev-els of variability This is an important question becauseas was indicated above it has been assumed that with-holding reinforcement elicits variability The present hy-pothesis is that absence of reinforcement can sometimesserve as a discriminative cue for operant variability Thedistinction between elicited (respondent) and reinforced(operant) variability will be revisited shortly

Hopson Burt and Neuringer (2002) rewarded ratsunder a mixed schedule in which two periods of time orcomponents Var and Repetition (Rep) alternated with-out any external cues In the Var component four-responsesequences of L and R leverpresses were reinforced ifthey met a threshold variability contingency in the Repcomponent only repetitions of LLLL were reinforced(Probabilities of reinforcement were equalized in the twocomponents by intermittent reinforcement of LLLL inRep) The results demonstrated discriminative controldespite the absence of exteroceptive cues (see Figure 3)With trials since the unmarked switch into the Var com-ponent shown in the left column variability increasedand LLLL sequences decreased with trials since the unmarked switch into Rep shown in the right columnresponding became increasingly repetitive until LLLLsequences predominated The two functions were sim-ilar which is an important finding in that it indicatesdiscriminative control rather than elicitation If variabil-ity had been elicited because reinforcement was tem-porarily withheld (when the contingencies were switched)then variability should have increased following entryinto both Rep and Var components but that was not theresult

Therefore variability that is observed when rein-forcement is withheld for short periods as when a newresponse is being shaped may be partly discriminativelycontrolled despite the absence of external cues that isanimals and people may learn when it is functional tovary and some of the cues may come from responsendashoutcome relationships Control over this operant vari-ability may be more precise than that elicited by extinc-tion because as was noted above operant contingenciesdefine both the set of acceptable variations and the level

OPERANT VARIABILITY 679

of required variability It may also be maintained muchlonger because as will be seen shortly extinction weakensresponding

ChoiceThe frequency of one response relative to others or

response probability is used as a measure of choice orpreference Choices are found to match the probabilitiesof their consequences For example relative frequenciesof pecking one of two keys approximate relative fre-

quencies of reinforcement Neuringer (1992) showedthat the same matching relationship also governs choicesby pigeons to vary or repeat The more variations werereinforced the more the pigeons chose to vary and sim-ilarly for repetitions Each trial consisted of four L andR responses and reinforcements for varying and for re-peating were systematically manipulated The Var con-tingency was met whenever the current sequence dif-fered from each of the previous three sequencesmdasha lag 3criterion The Rep contingency was satisfied if the cur-

Figure 3 The left column shows the probabilities ( y-axis) that each of five rats met a variability contin-gency (solid circles) versus a repetition contingency (open circles) when they were being reinforced for vary-ing (Hopson Burt amp Neuringer 2002) Along the x-axis are trials since the mixed schedule (no cues pro-vided to indicate the component) had switched into the variability-reinforcing component The right columnshows the corresponding data for performances when reinforcement was contingent upon repetitions thex-axis here indicating trials since the unmarked switch into the repetition-reinforcing component

680 NEURINGER

rent sequence repeated at least one of the previous threeOn an average of once every 30 sec (VI 30 sec) the com-puter probabilistically primed reinforcement for Var orRep the prime being maintained until the reinforcementhad been collected Thus if the computer primed Var re-inforcement was delivered whenever the lag 3 variabilitycontingency was next met Similarly if Rep had beenprimed the next Rep sequence was reinforced Proba-bilities of Var and Rep reinforcements were systemati-cally manipulated the two always summing to 10 and amatching-type relationship was obtained (Figure 4panel A) The relationship was more complex than sim-ple matching possibly because if the pigeons chose tovary by responding stochastically there was some prob-ability due to chance that the current sequence wouldrepeat a previous sequence This was shown by a com-parison of pigeon performances to the performance of acomputer-simulated stochastic model programmed tomatch its Var choices to the probability of Var reinforce-ments The pigeonsrsquo choice distributions were found tobe the same as the modelrsquos (Figure 4 panel B) Thuschoices to vary or repeat were governed by relative fre-quencies of reinforcement An animal can choose whetherwhen and how much to vary a result consistent withthose described above from Blough (1966) Machado(1989) and Grunow and Neuringer (2002) Choices ofwhether or not to vary appear to be governed similarly toall other operant choices

These f indings may have important applicationsWhen an individual repeatedly responds in a way that isnonfunctional reinforcement of choices to vary mayhelp the individual to emerge from the ineffective pat-tern This can be true for those experiencing everydayproblems but feeling helpless to change and it may beespecially important for those experiencing depressionor who respond in otherwise abnormally stereotypedmanners such as those with autism These topics will beconsidered in a later section

ExtinctionAs was indicated above extinction is often found to

increase variability but almost all studies of this effectinvolve responses that had previously been repetitiveAntonitis (1951) for example reinforced rats for pokingtheir noses anywhere along a 50-cm horizontal openingAlthough the exact location of pokes did not matter withcontinued training responses became limited to one or afew locations along the strip When reinforcement waswithheld variability of location increased (see also Eck-erman amp Lanson 1969) Variability produced by extinc-tion has been reported for many other response dimen-sions as well including force (Notterman amp Mintz1965) number (Mechner 1958) topography (Stokes1995) and sequence (Balsam Paterniti Zechowy ampStokes 2002 Mechner Hyten Field amp Madden 1997)

The issue is complicated however by other studiesthat show that previously reinforced responses and re-sponse patterns are maintained intact during and afterextinction That is extinction does not break down

learned patterns For example when Schwartz (1981) re-inforced pigeons for any sequence of four L and four Rresponses each bird developed a dominant sequence asdescribed above and these sequences were maintainedduring extinction Despite decreasing response ratesduring extinction response patterns were retained Sim-ilar maintenance of response structure has been ob-served during extinction following reinforcement of

Figure 4 Average performance across a group of pigeons inNeuringerrsquos (1992) study of choices to vary or repeat (A) Loga-rithms of the ratio of number of times the pigeons met the Varcontingency divided by the times they met the Rep contingencyas a function of the analogous numbers of reinforcements re-ceived for each Points represent group averages in two separateexperiments Also shown is the least squares best-fitting line(B) Changes in U-value an index of response variability as afunction of the obtained percentage of reinforcements for vary-ing The solid line in panel B represents the performance of acomputer-based model that matched its choices to vary (vs re-peat) to its obtained relative frequency of reinforcement for vary-ing From ldquoChoosing to vary and repeatrdquo by A Neuringer 1992Psychological Science 3 p 248 Copyright 1992 by the AmericanPsychological Society Reprinted with permission

OPERANT VARIABILITY 681

choices (Myerson amp Hale 1988) responding underfixed-interval contingencies (Machado amp Cevik 1998)visual discriminations (Nevin 1967) and matching tosample (Cumming Berryman Cohen amp Lanson 1967)How can stability during extinction be reconciled withincreased variability

Neuringer Kornell and Olufs (2001) wrestled withthis question in a study of extinction following rein-forcement under three different contingencies An oper-ant chamber contained two levers left (L) and right (R)on either side of a reinforcement dispenser and a key (K)in the rear of the chamber distant from the dispenserOne group of rats Rep was reinforced for repeating afixed sequence of responses LKR A second group Varwas reinforced for variations across the three operandaunder a threshold contingency Reinforcement for a thirdgroup Yoke was matched to those of the Var animalsand was therefore independent of the particular sequencesBecause of the unusual configuration of operandaskewed distributions of sequence probabilities were ob-served in all of the groups an outcome that was antici-pated and desired in order to document possible in-creases in variability during extinctionmdasha flattening ofthe distributionmdashas well as possible decreases That isduring the reinforcement phase the Rep group came torepeat the LKR sequence with high probability the Yokegroup tended to emit the easiest sequences such as LLLand RRR but distributed their responses over more se-quences than the Rep group did and the Var group dis-tributed most widely although because of differences inthe operanda and in their locations some sequences weremore preferred than others and in particular sequencesending with a K response were least likely (the key beingfarthest from the reinforcement dispenser) Thus thecontingencies resulted in three different distributions orhierarchies of sequence probabilities very steep forRep intermediate for Yoke and flattest for Var

The main finding was that withholding reinforcementhad similar effects in the three groups In each responserates fell to low levels across the four sessions of extinc-tion Furthermore sequences that were most frequentduring the reinforcement phase continued to be most fre-quent throughout extinction A slight flattening of thedistributions was observed however with the lowestprobability sequences increasing relative to the rein-forcement phase Thus because the overall hierarchy ofsequence frequencies was maintained the results indi-cated stability but because the distributions were some-what flattened the results also showed slightly increasedvariability It is as if the strategy during extinction wereto keep doing what had worked in the past but occa-sionally to do something very different This compli-cated story integrates two effects of extinction increasedvariability and maintained stability Most importantly forthe present discussion extinction had similar effects onvariable yoked and repetitive operants

One additional comparison is of interest Recall thatGrunow and Neuringer (2002) found that when contin-

gencies demanded high variability obtained levels ofvariability decreased slightly as reinforcement frequen-cies were lowered This result is opposite to the Vargrouprsquos slight increase in variability just described underextinction conditions The reason may be that extinction-induced variability is short lived lasting only as long asresponding continues perhaps for hours or a few ses-sions at most whereas the variability engendered by theVI 5 intermittent schedule of reinforcement in Grunowand Neuringer can be maintained indefinitely The de-crease in Grunow and Neuringerrsquos operant variabilitymay be due to the weakening of the reinforcement effect(ie reinforcement is less frequent) whereas the in-crease following extinction may be a short-term elicitedeffect This distinction between reinforced and elicitedvariability is important and will be discussed further inthe next section

Operant and Respondent InteractionsElicited or what is sometimes referred to as respondent

effects occur when environmental changes that affectbehavior do not depend on (are not caused by) the behav-ior Pavlovian conditioning being the most commonlystudied example Operant effects are observed when a re-inforcer depends on the response to produce it Althoughfor historical reasons respondent and operant effects areoften studied separately most behaviors result from acombination or interaction of the two Variability is nodifferent and we turn to an example of such an interaction

As reinforcement is approached under FI or fixed-ratio (FR) schedules response rates are commonly ob-served to increase Similarly when reinforcement de-pends on accuracy as in discrimination learning ormatching to sample it too is found to increase within theFI or the FR (Nelson 1978 Nevin 1967) Cherot Jonesand Neuringer (1996) asked whether the same patternheld for reinforced variability Does operant variabilityincrease as reinforcement for varying is approachedThe question may be related to such real-world cases aswhen a composer is rushing to complete a compositionbefore a scheduled performance or when a programmeris trying to solve a problem before an upcoming meetingThese activities involve at least to some extent non-repetitive original or variable thought and behaviorThe question is whether variability increases with prox-imity to or anticipation of reinforcement for varying

Cherot et al (1996) employed trials consisting of fourresponses across L and R operanda and a lag 4 variabil-ity contingency with rats in one experiment and pigeonsin another Unlike in any of the cases described so farhowever the lag-variability contingency had to be satis-fied four times in order for a single reinforcer to be ob-tained With ldquordquo indicating successful completion of theVar contingency ldquo2rdquo indicating failure to meet that con-tingency and PELLET indicating the food reinforcer anexample follows LRLL RRRL LRLL2 LRLL2RLLR RRRL2 LRRR+PELLET Thus food reinforce-ment was provided only after the subject had satisfied

682 NEURINGER

the variability contingency four times FR 4 The resultswere that as the food reinforcer was approached (ie asprogress was made through the ratio) the subjects wereless and less successful at varying The probability ofmeeting the variability contingency in the first trial afterreinforcement was higher than it was in the second whichin turn was higher than that in the third trial and so onFor comparison a Rep group was reinforced for repeatingsequences once again with reinforcement only after foursuccessful repetitions Probability of repeating increasedwith proximity to reinforcement a finding consistentwith the literature but opposite that for the variabilitygroup Thus approach to reinforcement facilitated oper-ant repetition but interfered with operant variabilitySimilar variability-interfering effects have been docu-mented for within-trials responses The probability that aresponse repeats the just prior response increases within atrial even when sequence variability is reinforced (McElroyamp Neuringer 1990 Neuringer 1991) To return to the ex-ample of a composer rushing to complete her composi-tion she may indeed have increasing difficulty respond-ing creatively under the pressure of a deadline On theother hand the soloist may find it easier to repeat thecorrect performance during practice periods as the con-cert nears How can these results be reconciled with themany findings that variability can be reinforced

An additional result of Cherot et al (1996) provides ananswer The Var animals responded much more variablyoverall than did the Rep animals Although variability inthe Var groups decreased with approach to the reinforceroverall variability was always much higher when it was re-inforced (Var group) than when it was not (Rep group)Reinforcement therefore exerts two influences Overallvariability is elevated sometimes to the highest levelswhen it is reinforced but approach to (and possibly focuson) the reinforcers constrains or lowers variability Sim-ilar reinforcement-produced interference effects had beenreported previously for example when raccoons are re-inforced for dropping a ldquocoinrdquo into a slot the reinforce-ment results in the animalsrsquo holding and rubbing the coinrather than dropping it (Breland amp Breland 1961) Theoverall enhancement of variability was of much greatermagnitude in Cherot et al than the decrease with approachwas although both effects were statistically significant(The notes created by the composer under pressure ofdeadline will be less readily predicted than will thoseplayed by the performer practicing a well-known piece)This study highlights the importance of interactions be-tween operant effects of reinforcement and elicited effectsproduced by anticipation of reinforcement and the resultsmay be relevant to the debate concerning possible detri-mental effects of reinforcement on creative behaviors (seethe discussion below and Neuringer in press)

SummaryReinforcement controls behavioral variability as it

does other operant dimensions Levels of variability areprecisely governed by schedule contingencies and dis-

criminative stimuli Choices to vary or repeat match rel-ative frequencies of reinforcement Extinction increasesvariability slightly whereas the hierarchies of sequenceprobabilities are retained and operant and respondenteffects interact The evidence shows that behavioral vari-ability is an operant controlled by its consequences

GENERALITY AND PARAMETERS

Many different methods measures species and re-sponses have been used in the study of operant variabil-ity and this diversity supports the generality of the phe-nomenon As is the case for repeated operants howeverthere are parametric constraints This section briefly re-views the generality and some of the parameters studiedto date

GeneralityMethods Throughout this section ldquoresponserdquo will

signify a response unit including both an individual re-sponse and a multiresponse sequence Each of the meth-ods to be described increases or maintains variability byreinforcing it

Novel response procedures reinforce responses upontheir first occurrence ldquoeverrdquo (Pryor et al 1969) or upontheir first occurrence within a session (Goetz amp Baer1973) Radial-arm maze procedures also reinforce ini-tial (within a given session) entries into arms of a mazeand result in a distribution of choices across the availableoptions the sequence or pattern of choices often beingquite variable

Lag procedures reinforce a response if it has not oc-curred across a given number of previous trials (Page ampNeuringer 1985) For example under lag 5 the currentsequence is reinforced only if it has not occurred duringany of the previous five trials A variant is the recurrence-time procedure which specifies a minimum number of in-tervening trials before a previously emitted response canagain be reinforced Machado (1989) combined recurrencetime with a percentile-reinforcement contingency to gen-erate high levels of variability (see also Machado 1992)Percentile-reinforcement contingencies base the criterionfor reinforcement on the subjectrsquos own performance over aprevious set of responses (see Galbicka 1994) For exam-ple a sequence will be reinforced only if it has not oc-curred over the previous n trials the value of n continuallychanging so that 50 of the trials are reinforced

Least-frequent response procedures (Blough 1966Schoenfeld Harris amp Farmer 1966 Shimp 1967) rein-force responses that are currently least frequent across amoving window

Threshold procedures reinforce responses whose rela-tive frequencies fall below a given value Because thistype of procedure was used in the research describedabove and in many other more recent studies details willbe provided In Denney and Neuringer (1998) which willserve as an example trials consisted of four responses byrats across L and R levers A running tally was kept of the

OPERANT VARIABILITY 683

number of occurrences of each of the 16 possible se-quences LLLL LLLR LLRL and so on If the relativefrequency of the current sequencemdashthe number of its oc-currences divided by the sum of all 16 sequencesmdashwasbelow a specified threshold value in this case 05 the ratwas rewarded Recently emitted sequences contributedmore to the maintained tally than did those in the distantpast because after each reinforcement all 16 counterswere multiplied by a weighting coefficient in this case95 Therefore the influence of a particular trial decreasedexponentially with successive reinforcements Thresholdprocedures differ from Bloughrsquos (1966) least-frequentprocedure in that more than one sequence (or response)can be reinforced at any given timemdashnamely all those se-quences that have relative frequencies below the thresholdvalue The procedure differs from lag in permitting agiven sequence to be reinforced many times in successionif the associated relative frequency is sufficiently low

The frequency dependence procedure (Machado 19921993) is a variant of threshold in which the probability ofreinforcement is a continuous function of relative re-sponse frequency The more frequent a response is thelower the probability is that it will be reinforced

Statistical feedback procedures provide subjects withcomparisons between their performances and that of amodel Neuringer (1986) showed human subjects feedbackfrom 10 statistical tests thereby enabling them to comparetheir own distributions of responses to that of a randommodel Neuringer and Voss (1993) provided feedbackbased on a chaotic model The goal in each of these caseswas for the subjects to learn to approximate the model andthe results of these experiments will be discussed below

Differential reinforcement of switches versus repetitionsbalance a subjectrsquos tendency to engage in each of these pat-terns Rather than reinforcement for infrequent responsesor random-like distributions reinforcement is providedsometimes for repeating a response and other times forswitching from one response to another In the two-operandum case a switch is L followed by R or R by La repetition is L then L or R then R Bryant and Church(1974) found that stochastic-like responding was gener-ated by rats when switches were reinforced with a 75probability and repetitions with a 25 probability that isthe rats had a natural tendency to repeat at least in theenvironment of this study Machado (1997) reinforcedpigeons for eight pecks across two keys if the sequenceof pecks contained at least one (one group of birds) ortwo (another group) switches Although the birds couldhave satisfied these contingencies by repeating a singlesequencemdashin the ldquoone switchrdquo condition for examplerepetition of LRRRRRR would produce reinforcement on100 of the trialsmdashthey tended to vary their sequencesBarba and Hunziker (2002) showed however that vari-ability was higher when it was directly reinforced thanwhen reinforcement was based on proportions of switchesa finding consistent with that of Machado (1997)

The evidence from each of these methods supports thehypothesis that variability can be reinforced

Measures Variability must be interpreted with refer-ence to a particular measure The reason is that there canbe high variability (and low predictability) at one level ofanalysis but order (and certainty of prediction) at a dif-ferent level For example the sequence 1 1 2 2 3 3 44 5 5 1 1 2 2 has the same variance (a commonlyemployed measure of variability) as 2 3 2 1 5 1 4 41 1 3 2 5 2 but an analysis of conditional proba-bilities in which the unit contains pairs of instances in-dicates that the two sequences have different levels ofvariability In general high variability observed with onemeasure does not rule out the possibility of order ob-served with another and vice versa That is why evalua-tions of random-number generators often employ manydifferent measures and indeed there are hundreds of dif-ferent measures of the randomness of a sequence (Knuth1969) All behavioral studies make simplifying assump-tions and rely on one measure or at the most a few

U-value has been the most commonly employed mea-sure of operant variability (Evans amp Graham 1980Machado 1989 Page amp Neuringer 1985 Stokes 1995)U-value measures the distribution of relative frequen-cies or probabilities of a set of responses For a set of16 possible responses i = 1ndash16 U-value is given by

where RFi = the relative frequency (or probability) ofeach of the 16 responses U-values approach 10 whenrelative frequencies approach equality as would be ex-pected over the long run from a random generator andthey approach 00 when a single instance is repeated

Other measures include the percentage of trials or fre-quency at which variability contingencies are met (Page ampNeuringer 1985) percentages of alternations versus stays(Machado 1992) conditional probabilities of responses(Machado 1992) the number or frequency of novel re-sponses (Goetz amp Baer 1973 Schwartz 1982) the num-ber or frequency of different responses (Machado 1997Schwartz 1982) Markov analyses (Machado 1992) andstatistical tests of distributions of responses (Neuringer1986) All of the measures employed to date indicate thatvariability can be reinforced

Responses and species Generality has further beendemonstrated by the use of many different responses andspecies These include Siamese fighting fish (Betta splen-dens) swimming in quadrants of a tank (Roots 2000)young chicks pecking keys (Neuringer 2002) pigeonspecking keys (Page amp Neuringer 1985) budgerigarssinging songs (Manabe Staddon amp Cleaveland 1997)rats pressing levers (Van Hest van Haaren amp van de Poll1989) porpoises swimming and jumping (Pryor et al1969) rhesus monkeys pressing levers (Sydney Reisbickpersonal communication) children drawing and build-ing with blocks (Holman Goetz amp Baer 1977) andhuman adults and children pressing buttons (Saldana ampNeuringer 1998 Stokes amp Balsam 2001) There havebeen few attempts to compare species or responses but

- ( )[ ] ( )=aring RF RFi ii

log log 2 21

1616

684 NEURINGER

in one study lowest baseline variability was observedwhen rats pulled trapezes intermediate levels were ob-served when they pushed keys and highest levels wereobserved when they pressed levers (Morgan amp Neuringer1990) with variability increasing in all cases when it wasreinforced In another study operant variability wascompared across rats and people with similar effects ob-served in the two species (Neuringer Deiss amp Imig 2000)

ParametersWe turn now to some parameters studied to date Many

other parameters as yet unstudied are no doubt also im-portantmdashfor example motivation and types of reinforcers

Age Older people (see eg Van der Linden Beertenamp Pesenti 1998) and older rats (see eg Lamberty ampGower 1990) respond less variably than their youngercounterparts when variable responding is reinforced or re-quested but in most cases it is not clear to what extent thevariability is elicited as opposed to reinforced Neuringerand Huntley (1992) compared reinforced variability in2-month-old (young) and 10-month-old (mature) Long-Evans rats Under Yoke contingencies the young rats re-sponded significantly more variably (two levers four re-sponses per trial) than the mature rats which constitutesan elicited effect When reinforcement was contingenton variability (lag 4) variability increased significantlyin both groups thereby overcoming the initial differencecorrelated with age Thus explicitly reinforcing vari-ability in older animals caused them to respond no dif-ferently than younger animals did a finding potentiallyrelevant to real-world human aging However when Wells(1999) compared 8-month-old (mature) to 24-month-old(old) rats although the mature rats responded more vari-ably than the old rats did under Yoke contingencies thedifference was not statistically significant Additionalstudies are needed especially of whether reinforcingvariable behaviors in older populations can facilitatelearning and performance

Gender Male and female rats are equally sensitive toreinforcement of variability (Neuringer amp Huntley 1992van Hest et al 1989) However there was a tendency(not statistically significant) for males to respond over-all more variably than females did and because thesestudies did not control for differences in size and weightof subjects or for other aspects of motivation additionalstudies are necessary

Stage of training Stokes and Balsam (2001) reporteddifferences in variability as a function of whether variabil-ity was initially reinforced early or later in training (seealso Gottlieb 1992) Human subjects generated sequencesof 10 L and R responses per trial on a computer keyboardunder lag 0 and lag 25 contingencies Under lag 0 everytrial ended with reinforcement under lag 25 the currentsequence had to differ from each of the previous 25 Whenlag 25 was experienced at the very beginning of traininglevels of variability at the end of training (when lag 0 wasoperative) were lower than they were when lag 25 was ex-perienced after 50 trials of initial lag 0 These results were

interpreted to suggest that there is an ldquooptimal period forsetting variability levelsrdquo (Stokes amp Balsam 2001 p 181)but that general conclusion needs further testing and anumber of questions need to be answered One has to dowith the contribution of reinforcement frequency Underlag 0 every sequence leads to reinforcement but rein-forcement is intermittent under lag 25 Another has to dowith whether learning to vary differs from learning to repeatwith respect to optimal periods (see eg Bouton 1994)

Related to stage of training are effects of prior experi-ence Hunziker Lee Ferreira da Silva and Caramori(2002) showed that for human subjects if Var contingen-cies were experienced just prior to Yoke contingencieslevels of response variation during the Yoke phase werehigher than they would have been if the Var contingencieshad not been previously experienced The time delay be-tween Var and Yoke experiences mattered with a long in-terval (3 months) producing only a small effect Saldanaand Neuringer (1998) also found effects of prior Var train-ing on performances in Yoke (increased variability inYoke) but the opposite did not hold that is training underYoke had relatively little effect on Var Hunziker Caramorida Silva and Barba (1998) reported similar effects withrats Thus there may be an asymmetry Reinforcement forvarying has relatively strong influences but reinforce-ment without regard to variations does not adversely affectlearning to vary These different effects are not surprisinggiven that Yoke contingencies are permissivemdashreinforce-ment is given whether or not responses varymdashwhereascontingencies under Var can be quite demanding How-ever the results motivate a number of additional ques-tions One is whether effects of Var training differ fromthose observed when repetitions are reinforced Another iswhether direct reinforcement of variable behaviors canhelp animals or people to overcome the detrimental effectsof prior noncontingent experiences such as those that pro-duce learned helplessness (Seligman 1975)

Environmental constraints Stokes (1995) arguedalso that variability increases when behavior is con-strained The term ldquoconstraintrdquo is ambiguous because allreinforcement contingencies exert a form of constraintIn one study Stokes (1995) compared different strate-gies of shaping leverpresses Rats in one group were re-inforced initially for pressing the lever with only theright paw (right only) and eventually for any press (withright or left paw or with any other part of the body) Theorder of training was reversed for a second group ldquoanyrdquofirst and then right only Behavior sequences were morevariable in the right-only-first group leading Stokes(1995) to conclude that early constraints lead to highvariability (see also Stokes Mechner amp Balsam 1999)That is an important hypothesis with many implicationsbut controls are needed before the results can be attrib-uted to the contingencies For example ldquoright onlyrdquo andldquoanyrdquo differed in terms of reinforcement distributions aswell as in the precision of the reinforcement operationthe ldquoanyrdquo reinforcers were provided immediately and au-tomatically by the programming equipment whereas in

OPERANT VARIABILITY 685

the right-only condition a human observer had to decidewhether a response had occurred and then present the re-inforcer necessarily leading to longer and more variedlatencies and a less consistent contingency Furthermorethere are many cases in which increasing constraints suchas those due to a decrease in the number of responses pertrial (Page amp Neuringer 1985) or the requirement of ex-actly four responses on each of two keys (Schwartz 1980)results in decreased variability Thus the contributionsof response-definition constraints and environmentallimitations must be explored further

SummaryThe generality of operant variability is shown by the

robustness of the phenomenon across procedures mea-sures responses and species As with other operant phe-nomena parameters influence the effects

FUNCTIONS

All operants including variability are functional Op-erant responses produce reinforcing consequences Be-cause other behavioral competencies depend on it oper-ant variability is functional in a broader sense as wellSome of these functions will now be described

Operant ConditioningThe potential of a reinforcer to change behavior is lim-

ited by the range of available variations (Hull Langmanamp Glenn 2001 Skinner 1981) and thus whatever pro-duces those variations contributes to the conditioning ofnew operants Much work has been done to identifysources of variation (Balsam Deich Ohyama amp Stokes1998 Segal 1972 Staddon amp Simmelhag 1971) butNeuringer (1993) was first to show that the same rein-forcers that strengthen individual responses can also pro-duce the necessary variability and that these two functionsof reinforcement may work simultaneously Implied isthat when an operant response is being conditioned re-inforcement works not only to strengthen an individualresponse but at the same time to engender the variabilityfrom which the response emerges (That is variability isnot solely a baseline phenomenon elicited or noise)Concurrent reinforcement of variations and individualtarget sequences will be the topic of this section

Pigeons in one experiment and rats in another were re-warded for varying among 16 possible response sequences(four responses per trial L and R operanda lag contin-gencies Neuringer 1993) Concurrently one of the 16designated as the target sequence was reinforced when-ever it occurred Thus reinforcements were given bothfor sequence variations and for a single selected targeta reinforced-variation-and-selection (RVS) procedureWhen the target was ldquoeasyrdquo (eg LLLL) it emergedrapidly from the baseline variations Given that the tar-get was always reinforced reinforcers were obtained morefrequently when the subjects did not vary and only re-peated the target When the target was ldquodifficultrdquo (eg

LLLR) it was not learned As in the choice experimentdescribed above (Neuringer 1992) the relative valuesor difficulties of target versus variations governed theprobability that the target would be learned

However even difficult targets could be conditionedunder the RVS procedure as was shown by another ex-periment in the same series (Neuringer 1993) Sequencevariations were again reinforced concurrently with a dif-ficult target RRLR but now the frequency of reinforce-ments for varying was systematically decreased Satis-fying the variability contingency was intermittentlyreinforced according to a VI schedule and over sessionsreinforcement for varying became less and less frequentThis technique resulted in acquisition of even the mostdifficult target sequence whereas a control group rein-forced for the target alone (without the concurrent rein-forcement for variations) never learned

Reinforcers were obtained more frequently in the RVScondition than in the control condition (since the RVSsubjects were rewarded both for varying and for emittingthe target sequence) and the additional reinforcers mighthave generated greater motivation to respond (Killeen1981) Thus motivation not the direct reinforcement ofvariability might have been responsible for the RVS ef-fect To test this possibility Neuringer Deiss and Olson(2000) reinforced three groups of rats for emitting a par-ticularly difficult target sequence LLRLR The experi-mental group was reinforced concurrently for sequencevariability on the average of once per minute (VI 1 min)The target-only control group received no reinforce-ments other than for the LLRLR target A third groupwas yoked to the experimental animalsmdashreinforced onceper minute for any sequence variability not requiredmdashand as for the other two groups whenever the target oc-curred If the additional reinforcement was responsiblefor the facilitated learning under RVS then because theexperimental and yoked groups received equal reinforce-ments they should have learned equally well but thatwas not the result Only the experimental group learnedto emit the LLRLR target The yoked controls respondedat high rates throughout the experiment but they did notvary sufficiently to make contact with the target contin-gency and never learned The target-only control groupreceived insufficient reinforcement and their responsesextinguished Therefore the added reinforcers helped tomaintain motivation as suggested above but reinforce-ment of variations was necessary for learning of a diffi-cult sequence The RVS procedure maintains both highmotivation and high variation and the two together re-sult in the acquisition of difficult-to-learn responses

Grunow and Neuringer (2002 Experiment 2) studiedone possibly important parameter of the RVS proceduremdashnamely the level of variability When a difficult target re-sponse is being trained does it matter whether reinforcedvariability is low intermediate or high As describedabove each of four groups of rats was reinforced for vari-able response sequences the first group for high vari-ability the fourth group for low variability and the other

686 NEURINGER

two for intermediate levels In other ways the procedurewas similar to that of the Neuringer Deiss and Olson(2000) RVS contingencies just described with a singledifficult target sequence always reinforced The resultswere that the higher the level of reinforced baseline vari-ability the more quickly the target was learned Thusvariability levels indeed mattered and if the just-re-ported effect is found to be general it will be most help-ful to reinforce high variability when difficult-to-learnnew responses are being shaped Other parameters re-quiring study are class definitionsmdashwhich responses areincluded in the class of acceptable variationsmdashand num-ber of instances within the class

One additional effect should be noted When varia-tions among a set of responses are reinforced if one vari-ant is selected for extinction (no longer reinforced) thefrequency of that instance decreases Neuringer (1993)showed this for pigeons responding under a lag-variabilityschedule When one of the 16 sequences was no longerreinforced (with no other change to indicate its status) thefrequency of that sequence decreased Thus reinforce-ment of variations provides the baseline to select for (iestrengthen) and select against (ie weaken) individualinstances

These findings have potentially important implicationsFirst as has been indicated the same reinforcers thatstrengthen particular instances can also reinforce varia-tions That a binary eventmdashreinforce or not reinforcemdashsimultaneously affects variations and selected instancesindicates both the power of reinforcement and a concep-tual problem How reinforcers exert such diverse effectsis a question to which we will return

Second shaping procedures may implicitly reinforcebaseline variations An approximation to a goal responseis reinforced for a period of time but eventually rein-forcement is withheld until a response closer to the goalis obtained It is commonly thought that withholding re-inforcement or mini-extinction induces the necessaryvariations for shaping to proceed as has been indicatedabove An alternative suggested by the present studiesis that organisms learn both to vary (within a set) and to re-peat (selected instances) until one instance is always rein-forced That is the process of shaping might involve re-inforcement both of variations (generally done implicitly)and of targeted instances If so a useful contributor tothe shaping process might be explicit reinforcement ofvariations

Third reinforcement of variations could facilitate ac-quisition of other competencies such as cognitive andmotor skills Siegler (1996) has shown in an important se-ries of studies that childrenrsquos acquisition of complex cog-nitive skills is correlated with high variability Anotherexample is that when motor skills are practiced in a vari-able manner high quality of performance results (Ma-noel amp Connolly 1995 1997 Mechner 1992 Schmidtamp Lee 1999) However it is rare that variability in thesedomains is directly reinforced as part of the training oreducational process

Fourth the RVS procedure might be used in therapeu-tic settings to modify maladaptive and self-injurioushabits Therapists sometimes employ noncontingent re-inforcement procedures to weaken undesirable behaviors(Vollmer Iwata Zarcone Smith amp Mazaleski 1993)The reasoning is that many maladaptive behaviors aremaintained by unintended consequences and that non-contingent rewards may compete with those conse-quences thereby lowering the strength of the unwantedhabit For example when a staff personrsquos attention toself-injurious behavior helps to maintain it providing at-tention noncontingently may decrease the strength of theunwanted target behavior However the RVS proceduremight accomplish the same goal more effectively andwithout undesirable side effects (Vollmer RingdahlRoane amp Marcus 1997) Noncontingent rewards some-times result in unwanted ldquosuperstitiousrdquo behaviors andthat would be avoided under RVS Furthermore RVSwould help to provide the set of baseline behaviors nec-essary to establish desirable new responses

Finally RVS procedures may help to gain control overand eventually to weaken other undesirable target behav-iors The target can be incorporated within a set of moredesirable responsesmdasha new operant class established withthe target as one membermdashand variable instances rein-forced Once variable responding is successfully main-tained including the undesirable target response the lattercan be weakened by selectively withholding reinforcement

Summary The conditioning of operant responses de-pends on both baseline variations and selection of suc-cessive approximations to a goal response The same re-inforcers used to select the approximations can reinforcebaseline variations thereby defining the precise set ofappropriate variants and the desired levels of variationSuch concurrent reinforcement of variations and selec-tions may facilitate the acquisition of difficult-to-learnoperants and selective withholding of reinforcementmay weaken undesirable instances

Creativity and Problem SolvingBehaving in a novel unusual or nonrepeating manner is

part of creative activities (Campbell 1960) Novelty andvariation may arise through combinations of previouslylearned behaviors (Epstein 1996) transformations of thesebehaviors (Boden 1994) direct reinforcement of variabil-ity as described above or some combination thereof Ofcourse neither novelty nor variability alone suffices forcreative outputmdashthe throw of the dice or the toss of acoin are not creative nor are haphazard strokes of a paint-brush on a canvasmdashbut without variation creativity islow Whether the variability required for creativity canbe reinforced is relevant to whether creativity itself canbe reinforced The latter question has led to contentiousdebate with some arguing that reinforcement facilitatescreativity (Cameron amp Pierce 1994 Eisenberger ampCameron 1996 Stokes 2001 Winston amp Baker 1985)and others focusing on detrimental effects of reinforce-ment (Amabile 1983 Deci Koestner amp Ryan 1999)

OPERANT VARIABILITY 687

The debate goes well beyond the present discussion butsome of the research described above is relevant Firstreinforcement contingent on variability generates higherlevels of behavioral variability than does reinforcementcontingent simply on responding as has been shown bythe many studies described above in which Var contin-gencies were compared to Yoke contingencies Similarlyprecise reinforcement contingencies matter for creativity(Cameron amp Pierce 1994 Lepper amp Henderlong 2000)If any product rather than only creative ones is rein-forced creativity tends to fall (just as variability decreasesunder yoke conditions) Second approach to or antici-pation of reinforcement interferes with operant variabil-ity (Cherot et al 1996 McElroy amp Neuringer 1990Neuringer 1991) which again might be related to effectson creativity when an individual focuses on the rein-forcer Note however that although variability decreaseswith proximity to the reinforcer overall levels of vari-ability are much higher when variability is reinforced thanwhen it is not The same may be true for creativity aswell anticipation of reward may interfere with creativeoutput to some extent but at the same time high creativ-ity may depend on its reinforcement

Operant variability may be important for problemsolving as well Here too there is disagreement as towhether training is helpful or not Early studies by Maier(1933) indicated that it is and these found support inlater work (eg Lung amp Dominowski 1985) Somestudies showed that those individuals most likely toldquobreak the setrdquo are also most likely to solve a problem(Dover amp Shore 1991) However a number of studiesfound that practice had no influence (Duncan 1961)

Arnesen (2000) explored a related phenomenon witha rat model One group Var was reinforced for variableinteractions with novel objects and another group Yokewas reinforced independently of such interactions EachVar rat received training with 12 different objects oneper session For example in the presence of a soup canthe rat was reinforced the first time it entered the canbut thereafter a different response was required such asrolling it climbing on it and so forth This procedure issimilar to that of Goetz and Baerrsquos (1973) described aboveexcept that objects changed from session to session Ratsin the Yoke control group had identical experiences in anadjoining identical chamber that contained identical ob-jects but now rewards were not contingent on object ma-nipulation After this training the rats were tested in adifferent room with 30 never-before-experienced ob-jects scattered on the floor Hidden in each was a smallpiece of food When compared to their Yoke controls theVar animals were more active during this test exploredmore manipulated objects more and most importantlydiscovered significantly more food pellets Another con-trol group received no preliminary training with objectsand their performance in the test session did not differfrom that of the yoked controls Therefore when ratswere reinforced for variable interactions with objectsthe result was an increased tendency to explore manip-ulate and discover new sources of reinforcement It

would be of considerable interest to determine whethera similar effect occurs with children

Assessment and Treatment of PsychopathologiesAbnormal levels of behavioral variability characterize

some human psychopathologies as has been shown byclinical observations as well as by research with humansubjects and animal models Few studies have attemptedto reinforce variability directly in such problem popula-tions The main question is whether reinforcement canmodify abnormal levels of variability in the direction ofnormalcy and whether such changes might have associ-ated beneficial effects

Autism Stereotyped responding characterizes indi-viduals with autism (Hertzig amp Shapiro 1990) For ex-ample Baron-Cohen (1992) asked children with autismand others in a control group to hide a penny in one handso that the experimenter could not guess which hand heldthe penny Those with autism were more likely thanthose without to generate a simple predictable patternsuch as repeatedly switching back and forth from left toright hands Such repetitive responding can be maladap-tive For example Mullins and Rincover (1985) askedchildren with and without autism to pick one of five cardsFood was occasionally found in a cup behind the cardEach of the five cards was a discriminative stimulus fora different schedule of reinforcement including contin-uous reinforcement (CRF every choice reinforced) FR 2(fixed ratio 2 in which every second choice was rein-forced) FR 4 (every fourth choice reinforced) FR 7 andFR 11 Control participants sampled all five alternativesand quickly learned to choose the most frequently rein-forced CRF card Autistic children sampled only a lim-ited number of cards and therefore often preferred anonoptimal alternative Thus children with autism tendto respond less variably than normal children and thiscan result in failure to adapt

Miller and Neuringer (2000) asked whether direct re-inforcement could influence the variability of individu-als with autism Sequences of responses on two largebuttons were first reinforced probabilistically and inde-pendently of variability (Prob phase) after which rein-forcement was contingent upon satisfying a thresholdvariability contingency (Var phase) The autistic partic-ipantsrsquo variability was significantly lower than that ofmatched controls during both Prob and Var phases butwhen variability was reinforced its levels increased sig-nificantly in both groups This is an important findingbecause it suggests that direct reinforcement of variabil-ity may help to modify some of the behavioral stereo-types that are characteristic of autism Lee McComasand Jawor (in press) offer support for this hypothesisThree individuals with autism were reinforced for variedverbal responses to social questions the responses had tobe appropriate given the context Variability increased intwo of the three individuals In a related procedureBaker and Koegel (1999) reinforced children with autismfor varied social interaction in a game-playing contextAlthough the reinforcement was permission to engage in

688 NEURINGER

the stereotyped behaviors preferred by the child the re-sult was increased social interactions both during theperiod of the experimental manipulation and in 1- and2-month follow-ups These three studies taken togethersuggest that reinforcement of varied behaviors may facil-itate modification of nonfunctional ritualistic and stereo-typed behaviors

Depression Depression provides another example ofmaladaptive consequences of low variability Lapp Mar-inier and Pihl (1982) observed that depressed womenproduced fewer alternative solutions to hypotheticalproblems than did nondepressed women Channon andBaker (1996) found that attempts by moderately de-pressed college students to identify faults in a series ofinterconnected circuits were less varied and less success-ful that were those of nondepressed controls HorneEvans and Orne (1982) found that when depressed pa-tients were asked to generate a random sequence of num-bers their responses were less variable than were thoseof controls These experimental results are consistentwith clinical observations

Can depressed individuals be rewarded for increasingvariation Hopkinson and Neuringer (in press) dividedcollege students into mildly depressed and nondepressedgroups on the basis of the CES-D scale a paper-and-pencil self-evaluative index of depression In the first oftwo phases sequences of responses on a computer key-board were reinforced independently of variability (Prob)following which only highly variable sequences were rein-forced (Var) Under Prob depressed students respondedsignificantly less variably than did the controls in agree-ment with previous findings Under Var variability in-creased significantly in both groups Indeed by the endof the experiment depressed and nondepressed subjectswere responding equally variably (Instructions were alsoeffective in increasing variability in the depressed par-ticipants) Thus even in a simple computer-game envi-ronment mildly depressed subjects manifested lowervariability than controls did but direct reinforcementmodified their behavior to control levels Can reinforce-ment of variability be used in real-world conditions tohelp depressed individuals An affirmative answer comesfrom cognitive behavior modification procedures thatwork to increase alternative solutions to problems (Beck1976) A more general conclusion from these studies ofautistic and mildly depressed individuals is that direct-reinforcement-of-variability procedures may benefitthose who manifest abnormally low levels of variability

Attention Deficit Hyperactivity Disorder (ADHD)ADHD may provide an example of the opposite stateabnormally high levels of variability (Barkley 1990)Wultz Sagvolden Moser and Moser (1990) proposedthat the spontaneously hypertensive rat (SHR) may serveas a good animal model of ADHD the SHR shows manyof the behavioral characteristics of that disorder includ-ing hyperactivity poor learning ability poor ability totolerate delayed reinforcement and poor self-control

The next study to be described asked about operant vari-ability in SHRs

Mook Jeffrey and Neuringer (1993) compared vari-ability of SHRs and of a control strain Wystar Kyoto(WKY) under two conditions Prob and Var The mainresult was that SHRs tended to respond variably whetheror not the contingencies required it consistently with theresults from the human ADHD literature When vari-ability was reinforced the two strains responded at ap-proximately equally high levels of variation These re-sults were confirmed in two other studies (HunzikerSaldana amp Neuringer 1996 Mook amp Neuringer 1994)Thus control rats varied only when the contingencies re-quired variations whereas SHRs varied whether or notthe contingencies required it As might be expected fromthese results when reinforcement depended on repeti-tions the SHRs performed poorly (Mook et al 1993)

Another similarity between SHRs and ADHD humanswas seen when amphetamine a drug that has effects simi-lar to those of Ritalin often prescribed for ADHD was ad-ministered (Mook amp Neuringer 1994) When repetitionswere required for reinforcement amphetamine improvedperformances by the SHRs so that they were as accurate assaline-injected WKYs a finding again consistent withthose in the ADHD literature These studies support SHRsas a model of ADHD and suggest that additional attemptsshould be made to assess whether or not individuals withADHD are less able to change levels of variability thanare controls Although as indicated above there is someevidence in support of this hypothesis (Barkley 1990)when Saldana and Neuringer (1998) directly comparedhuman ADHD and control participantsrsquo operant variabil-ity within the context of a computer game no differenceswere observed Most of those with ADHD were main-tained on Ritalin and other drugs however a fact unfor-tunately not documented in the original article

Drug EffectsAlthough many drugs are known to influence behavioral

variability (Brugger 1997) rarely have attempts beenmade to analyze explain or identify the sources of theeffects As will be discussed below operant variabilitycan result from the memory-based process of remember-ing previous responses and then behaving differently orcan result from responding stochastically a process thatappears to be quite different McElroy and Neuringer(1990) attempted to distinguish the effects of alcohol onmemory from its effects on stochastic processes in astudy in which one group of rats was reinforced for vary-ing four-response sequences under a lag 5 contingency(Var) and another was reinforced for repeating a singlesequence LLRR (Rep) Ethanol injections of 075 and20 gkg affected Var and Rep performances differentlyWhereas the Rep animals were much less able to satisfythe LLRR contingency at both doses the Var rats re-sponded variably whether or not alcohol had been ad-ministered Similar results were obtained by Cohen et al

OPERANT VARIABILITY 689

(1990) with a multiple schedule in which each rat servedas its own control In the same animals and within thesame sessions alcohol degraded Rep performances butleft Var intact Thus alcohol appears to interfere with atask that requires working memory as presumably is thecase for the Rep task but not with one that reinforcesvariability The memory-interfering effect of alcoholmight account for why alcohol also degrades radial-armmaze performance (Devenport 1983)

Alcohol also affects choices to vary or to repeat For ex-ample Gorka (1993) reinforced rats concurrently for re-peating and varying and monitored percentage of choicesas a function of time following injection of ethanol Inthe control sessions when the rats were injected withsaline the probabilities of varying and repeating remainedapproximately constant across the 45-min session witha slight bias in favor of varying After injection with 15gkg of ethanol however there was a significant increasein choices to vary at the beginning of the session andthen a later decrease to below 50 That is alcohol ledinitially to a markedly increased preference for varyingbut later to a preference for repeating (Figure 5) This re-search should be replicated and extended especially be-cause the contingencies ldquopressuredrdquo the animals to re-main at approximately 50 preference but thesuggestion is that alcoholrsquos effects on response variabil-ity may depend in part on the duration of its influence Amore general implication is that drugs may affect re-peated and variable operants differently and that effectsdepend on time following administration

SummaryReinforced variability may contribute to many types

of learning including operant shaping and acquisition ofcognitive and complex motor skills Similarly creativityand problem solving depend on variable responding andin these areas as well explicit reinforcement of variationsmay be functional Abnormal levels of variability aresometimes caused by psychopathologies and drugs andoperant variability may be useful in these areas in twoways first as a way of assessing the variability that is pos-sible and second to modify variability toward more nor-mal levels Studies show that individuals with autism andmild depression vary less than do controls but that ex-plicit reinforcement of variability moves the levels towardsnormalcy Other studies show that drugs such as alcoholmay affect repeated and variable operants differently

EXPLANATIONS

Two questions must be answered in order to explainoperant variability First what are the sources of vari-ability that permit its reinforcement Second how is itmaintained Related questions including how instancesbecome members of a set from which variations emergeand whether operant variability is a generalizable com-petency will also be considered

Origination From Nonoperant SourcesSkinner (1966) reasoned that operant responses orig-

inate from nonoperant sources with reinforcement con-tacting a behavior shaping and strengthening it onlyafter the behavior had occurred for reasons other than re-inforcement To put it simply a behavior must occur be-fore it can be reinforced (Balsam amp Silver 1994) I willconsider three initial sources of variable behavior en-dogenous variability noncontingent environmental eventsand extinction These appear to parallel evolutionaryprocesses and they too will be identified

Endogenous variability All physical phenomenavary and behavior is no exception This fact can providethe opportunity for reinforcement to select particularlevels of variation Examples of spontaneous variabilityinclude spontaneous alternations in mazes (Dember ampRichman 1989) nondifferentially reinforced variationsin operant responses (Guthrie amp Horton 1946) and ex-ploration (Archer amp Birke 1983) Endogenous muta-tions provide a parallel in genetics Base-pair changes inDNA molecules occur spontaneously intron-bearinggenes mix and match segments to encode new proteinsjumping genes introduce new DNA sequences and atthe level of the chromosome random crossings and as-sortment occur as a natural part of reproductive meiosisThese sources of random genetic change occur indepen-dently of external influences

Figure 5 The relative frequency of variations by rats as a func-tion of 5-min blocks of time since the beginning of the sessionThe open circles represent performance after the rats were in-jected with saline indicating a slight tendency to vary more thanrepeat but one that is approximately constant throughout thesession The filled circles represent performance after injectionwith 15 gkg ethanol indicating a much higher likelihood ofvarying soon after the alcohol is injected followed by a decreasein variations to below 50 and then a return to baseline at theend of the session Group averages are shown Data are fromGorka (1993)

690 NEURINGER

Noncontingent events Noncontingent events some-times referred to as chance accidents or luck also affectbehavior One happens upon a particular passage in a bookthat leads to a new thought one happens to sit next to aninteresting woman or man on an airplane which leads tomarriage (Bandura 1982) In science as well as in every-day life accidents or serendipity are an important part ofthe process of discovery (Beveridge 1957) In each of thecases presented above an accidental event leads to new be-haviors that are then strengthened by consequences Thesame is potentially true for variability itself Particular lev-els of variability might occur for adventitious reasons butthen become functionally related to consequences

That organisms are particularly sensitive to such unan-ticipated noncontingent events is shown by two basicprinciples of learning habituation and Pavlovian condi-tioning Organisms attend preferentially to unexpectedevents and stop attending or habituate to repeated onesThe conditioning of new Pavlovian responses dependsaccording to influential theories upon unexpected rein-forcing events (eg Rescorla amp Wagner 1972)

Noncontingent events such as X-rays and other pro-cesses that can act as mutagens play an important role ingenetic variation as well The particular genes affectedand the type of changes are random there is no neces-sary relationship or contingency between the externalevent and the affected gene However sensitivity to mu-tations is not random but has been selected presumablybecause it is functional Thus in both behavior and ge-netics sensitivity to noncontingent events is conserved

Extinction Extinction increases behavioral variabilityat least to some extent as was described above Distance inspace or time from the reinforcer does the same (Cherotet al 1996) A possible parallel at the genetic level in-volves high levels of deprivation at least in simple lifeforms For example when E coli bacteria are deprivedof nutrients unusually high mutation rates are observedThe exact mechanism by which such hypermutations areinduced is not known but they increase the likelihoodthat a population can survive such environmental insults(Moxon 1997) That is deprivation-induced variability isfunctional at the genetic level (The behavioral evidenceregarding long-term deprivation is less clear some stud-ies showing increased variability and others increasedstereotypy eg Carlton 1962 Senkowski Vogel amp Po-zulp 1978)

Endogenous variations external noncontingent eventsand withholding of reinforcement are three sources forthe variability that precedes control by reinforcementBut once control by reinforcement is established a dif-ferent question emerges What behavioral processes areinvolved when variability is maintained as an operantThe sources just discussed generate uncontrolled non-specific and often short-lived variability and cannot ex-plain the precisely controlled operant type

Operant Variability ProcessesThree processes are hypothesized to underlie operant

variability The first involves the use of random environ-

mental events such as the toss of a coin The second in-volves memory for responses such as when an animallearns not to return to previously visited locations Thethird will be referred to as an endogenous stochasticgenerator

Random environmental events Throughout historymany cultures have used random events to influencechoices and decisions such as throws of dice and randomselections of sticks cards bones or organs Today coinflips are used by a referee at the beginning of a game to de-cide which team kicks the ball a computerrsquos random-number generator is employed by scientists to avoid bi-ases in assigning subjects to experimental groups andaleatoric events are sometimes employed in modern artmusic and literature The Dice Man by Rhinehart (1998)provides a fictional example in which the protagonistbored with life writes a number of possible actions onslips of paper and then periodically selects one blindlyand behaves accordingly These examples show that randomexternal events may be used to avoid biases engender un-likely responses and break out of behavioral niches

Memory-based variability Many methods used tostudy operant variability differentially reinforce nonrepe-titions at least over the recent past Therefore one strat-egy that subjects can employ is to cycle through few re-sponses or sequences especially when the variabilitycontingencies are relatively permissive For exampleunder a lag 2 contingency cycling repeatedly throughthree sequences will result in 100 reinforcement Ani-mals and people sometimes do this apparently basingtheir current response on memory for or discriminativecontrol by the just emitted responses One such instancewas shown by Schoenfeld et al (1966) who rewardedrats for IRTs that fell into one of 10 interval classes onlyif the current IRT differed from the interval class of theprevious response The rats came to alternate betweenlong and short IRTs thus behaving systematically to meetthis lenient variability contingency Machado (1993)studied an analogous effect having pigeons peck L and Rkeys under a frequency-dependent variability contin-gency related to Shimprsquos (1967) procedure describedabove When the trial unit consisted of two responses thebirds developed memory-based repetitive sequencessuch as RRLLRRLL and thereby satisfied the variabilitycontingency When the trial unit was three responses inlength the birds did not develop the optimal fixed pat-tern of RRRLRLLLmdashwhich was too complexmdashbut in-stead developed ldquorandom-like behaviorrdquo (Machado1993 p 103) Thus a memory-based strategy was em-ployed when it was possible but when the memory loadbecame too high stochastic responding emerged A sim-ilar pattern was seen when variability of budgerigarsongs was reinforced (Manabe et al 1997) Under lag 1contingencies the birds sang mainly two songs underlag 2 they increased to three songs Only when the lagwas increased to 3 did song diversity increase apprecia-bly in one of the birds In radial-arm mazes as well whencontextual cues are available memory again appears toplay an important role leading to much lower probabil-

OPERANT VARIABILITY 691

ities of repetition than would be expected from a sto-chastic generator (McElroy amp Neuringer 1990)

Note that under lag schedules memory-based strate-gies result in higher reinforcement rates than respondingstochastically does The reason is that a stochastic gen-erator repeats by chance That animals and people oftendo not employ memory-based strategies probably indi-cates the difficulty of remembering long sequences andpossibly other advantages of unpredictable responding

If a memory strategy is the only way to satisfy vari-ability contingencies then it can be used even when thememory requirements are demanding as was shown byNeuringer and Voss (1993) Human subjects were trainedto generate on a computer keyboard a chaotic-like se-quence of responses that matched iterations of the logistic-difference equation

where Rn refers to the nth iteration in a series with R be-tween 00 and 10 and t is a constant between 10 and40 When the t parameter approaches 40 iterations ofthis equation result in very ldquonoisyrdquo sequences As withother chaotic processes however there is an orderlinessunderlying the surface noise In this case if responsesare autocorrelated with Rn drawn as a function of Rn21the resulting function is a parabola In the Neuringer andVoss (1993) study following each response college stu-dents were shown how close their predicted values wereto that from iterations of the logistic-difference model Inone experiment the students made digital predictions inanother they placed a pointer along a horizontal lineand this analogue response was then translated into dig-ital values for computation With training the studentsbecame increasingly adept at responding in chaotic-likefashionmdashthe studentsrsquo predictions matched closely theiterations of the logistic functionmdashand their autocorrela-tions became parabolic The left column in Figure 6 showsperformances by 4 subjects (rows) early in training thatis not well approximated by the logistic-differenceparabola and the right column shows performances at theend of training in which r 2 values were 72 99 98 andapproaching 10 How was this accomplished Sinceeach iteration in the logistic-difference sequence is basedon the prior outputmdashthat is the sequence is generated bya deterministic although nonlinear processmdashit was hy-pothesized that the human subjectsrsquo chaotic-like responseswere also based on memory for prior responses To put itsimply the subjects may have memorized a long series ofpairs in the form ldquoIf the previous response was value Athen the current response must be value Brdquo as Metzger(1994) Ward and West (1994) and Ward (2002) have sug-gested

To test whether subjects in fact relied on memory forprior responses Neuringer and Voss (2002) systemati-cally slowed responding by interposing pausesmdashthat isby requiring an IRT As IRT increased the difference be-tween the subjectsrsquo sequences and the modelrsquos chaotic

Rn n nt R R= -( )- -1 11

Figure 6 Autocorrelations (responses on trial n on the y-axis asa function of n21 on the x-axis) for 4 human subjects at the be-ginning (left column) and end (right column) of training inNeuringer and Voss (1993) Equations for the best-fitting curveand amount of variance accounted for (r2) are shown at the topof each panel To the extent that the circles (responses) approxi-mate the equation y = 40x 2 40x 2 the subjects were respondingin a way that matched the logistic-difference chaotic functionFrom ldquoApproximating Chaotic Behaviorrdquo by A Neuringer andC Voss 1993 Psychological Science 4 p 115 Copyright 1993 bythe American Psychological Society Reprinted with permission

692 NEURINGER

output increased and the underlying parabolic structurewas broken That is when pauses increased subjects wereless and less able to approximate the modelrsquos chaotic dis-tribution (see Figure 7) The important point here is thathighly variable responses which in this case are chaoticin nature can be based on memorial processes

Endogenous stochastic generator The evidence tobe reviewed next indicates that animals and people canrespond stochastically and that this competency is basedneither on chance environmental events nor on memoryfor prior events or responses It is impossible to provethat an endogenous stochastic process underlies operantvariability that would be analogous to proving the nullhypothesis However much evidence is consistent withthe stochastic hypothesis Included are the facts thatmemory limitations do not permit adequate respondingunder many variability contingencies that response dis-tributions match those expected from a stochastic gener-ator that changes in parameter values affect operantvariability as they would a stochastic generator and thatinterference with memory processes leaves stochasti-cally based operant variability unaffected As a wholethe evidence is consistent with an endogenous stochas-tic source of operant variability

Memory limitations One type of evidence is nega-tive in character and is based on the presumed inability ofmemory for prior responses or sequences to satisfy sometypes of variability contingencies Two examples weregiven above (Machado 1993 Manabe et al 1997) andanother was provided by a lag 50 contingency in Pageand Neuringer (1985) In the latter case the sequence in

the current trial (eight responses across L and R keys) wasrequired to differ from those in each of the previous 50 tri-als including at the beginning of a session the last 50trials from the previous session Pigeons responded wellunder these contingencies and in a way similar to that ofa stochastic model It is unlikely that pigeons can remem-ber each of their previous 50 eight-response sequences

Response slowing As was shown above for chaotic-like responding performances are often degraded by de-lays imposed between a stimulus and a response pre-sumably because of limitations in working or short-termmemory (Berryman Cumming amp Nevin 1963 Williams1971) Neuringer (1991) relied on this effect to test whethermemorial processes could explain the generation of vari-able sequences under lag schedules Two groups of ratswere reinforced for L and R leverpresses producing fourresponses per trial One group (Var) was trained under alag 5 contingency and the other (Rep) was trained to asingle sequence of responses LLRR It was assumedthat accurate Rep performance depended on workingmemory After performances had stabilized IRTs weresystematically varied by the imposition of timeouts of05 to 20 sec between consecutive responses in the se-quence During these forced pauses the chamber wasdark and responses were not counted toward completionof the sequence If the animals had used the previous re-sponse(s) as a discriminative cue for the current responsethen performances should have been degraded by theseinterpolated pauses

The results were that for the Rep group as the re-quired pauses increased from 0 to 6 sec percentages ofcorrect sequences were relatively unaffected but at pausesgreater than 6 sec probability of a correct LLRR se-quence fell sharply This pattern of results suggests in-volvement of working memory of prior responses withthe rats able to remember responses within a 6-sec win-dow The Var grouprsquos results were quite different Aspauses increased between 0 and 6 sec the rats were moreand more likely to satisfy the variability contingenciesand with pauses greater than 6 sec percent correct re-mained at a high and constant level Thus the Var grouprsquosresults were opposite to those of the Rep group and op-posite to those predicted from a memory-based strategySimilar results showing that operant variability increasesor is maintained as responding is slowed have been re-ported by Morris (1987) with pigeons and by Baddeley(1966) with random response generation by human sub-jects The results reported above concerning alcoholrsquos ef-fects on variability and repetition were also consistentwith those of Neuringer (1991)

Memory for prior responses may not have controlled theVar grouprsquos performance but why did variability increaseEach of three answers is consistent with a stochastic-generator interpretation Weiss (1964 1965) hypothe-sized that random responding requires that current re-sponses be independent of previous ones memory for orcontrol by prior responses interferes with such indepen-

Figure 7 Approximation to a chaotic sequence (given by delta)as a function of the imposed interresponse times Shown are theaverages of 5 subjects

OPERANT VARIABILITY 693

dence and therefore pauses would be expected to facil-itate variability A second possibility is that at shortIRTs animals tend to repeat responses on the sameoperandum Blough (1966) found this in pigeons and ex-cluded such double pecks from his analyses because theyappeared not to be under the control of reinforcement con-tingencies Morris (1987) also found a tendency for birdsto repeat when no interresponse timeouts were imposedA third hypothesis is that there were two contributors tothe observed variability One was a stochastic processcontrolled by reinforcement and the other elicited vari-ability generated by interposition of pauses The high vari-ability elicited as an effect of slowed responding is ageneral phenomenon supported in many other cases Ac-cording to this interpretation operant variability in the Vargroup was governed by a stochastic-based process operantrepetition in the Rep group was governed by a memory-based process and pauses elicited variability under bothcontingencies Thus Rep performance was interferedwith whereas Var was facilitated Each of these hypothe-ses is consistent with the conclusion that memory for (ordiscriminative control by) prior responses does not con-tribute to and possibly interferes with variable respond-ing when an organism is reinforced for variability andthis in turn is consistent with the stochastic-generatorhypothesis

Stochastic response distributions Other evidence iscorrelational Response distributions are similar to thosefrom a stochastic generator The data from two experi-ments already discussed are shown in Figure 1 In thesecases reinforcement of least frequent IRTs (Blough 1966)and of nonrepetitive choice sequences (Machado 1989)resulted in stochastic-like responding

Neuringer (1986) reported another striking case Highschool and college students generated sequences of 1sand 2s on a computer keyboard with each set of 100 re-sponses constituting one trial A baseline phase lastedfor 60 such trials for a total of 6000 responses duringwhich the students were asked to behave as randomly aspossible as if they were reporting the outcome of a se-ries of coin tosses As in the many previous studies ofthis sort the subjectsrsquo responses differed significantlyfrom those of a stochastic model (Brugger 1997 Wage-naar 1972) During the training phase the subjects re-ceived feedback which enabled them to compare their per-formances with that of a stochastic model first accordingto one statistical test and then according to another untilfollowing each trial feedback was provided from 10 dif-ferent statistics The results were that all of the subjectslearned to approximate the stochastic model accordingto the 10 statistics as well as some (but not all) other sta-tistical tests That is the distributions of the statisticswhich clearly did not match a stochastic model at the be-ginning of training approximated the stochastic modelat the end of it (see Figure 8)

The previous failures of other researchers to obtain ev-idence for random responding had led some to concludethat humans did not have the capacity to behave ran-

domly What would account for the difference betweenthe present positive finding and the many other negativeones Neuringerrsquos (1986) study was the first in which sto-chastic responding was explicitly reinforced in this casewith feedback from 10 statistical tests An analogy mighthelp to explain the difference in procedures and resultsImagine an experimenter walking around a city and ask-ing people if they know what a violin sounds like (analo-gous to asking people whether they know what a series ofcoin tosses or dice throws is like) Most would answeryes If the same individuals were then asked to play theviolin very few if any would succeed The experimentermight conclude after many replications with differentsubjects in different cities that people did not have thecapacity to play the violin The point of course is thatplaying the violin is a skill requiring training and prac-tice and the same may be true for responding randomlyIt is easy to vary but practice and reinforcement may benecessary to approximate a random distribution

Response number Page and Neuringer (1985) testedthe stochastic-generator hypothesis in another way byvarying number of responses per trial with pigeons work-ing under a lag 3 contingency The reasoning was that ifprevious responses served as a discriminative cue forwhat not to domdasha memory-based strategymdashthen perfor-mance should be degraded when the number of re-sponses was increased Eight responses per trial requiresubjects to remember more than four responses The sto-chastic hypothesis predicts the opposite outcome as isdemonstrated by the following example If each trialwere two responses in length with responses determinedby the toss of a coin then the probability that a given trialwould repeat the previous one would be 25 (There arefour possible sequences in the first trial RR RL LRand LL Thus the second trial has a 1-in-4 chance ofmatching the first) If a trial consists of four coin tossesthe probability of a repetition by chance is 0625 or 1 in16 Thus if subjects used a stochastic process to gener-ate Ls and Rs performances should improve with in-creasing responses per trial Results were precisely thosepredicted by the stochastic hypothesis and inconsistentwith a memory strategy Probability of meeting the con-tingencies and therefore probability of reinforcementincreased as responses per trial increased In otherwords eight-response trials were easier for pigeons thanfour-response trials

Response interference Neuringer and Voss (2002)attempted to distinguish between memorial and stochas-tic processes in yet another way They hypothesized thatwhereas chaotic responding was based on memory forprior responses (as was discussed above) responding inNeuringerrsquos (1986) study with feedback from 10 statisti-cal tests indicated involvement of an endogenous sto-chastic generator To show that human subjects could en-gage in both processes at will three college studentswere trained to generate chaotic sequences in one compo-nent of a multiple schedule as in Neuringer and Voss(1993) and to generate stochastic sequences in a sepa-

694 NEURINGER

rate component as in Neuringer (1986) The wordCHAOTICS was on the screen in the first component theword STOCHASTICS was on during the other and thesewords alternated throughout the session After the sub-jects succeeded at each of the component tasks test ses-sions were introduced in which memory for prior re-sponses was subject to interference in both the chaoticsand stochastics components as follows During the

chaotics component four separate chaotic sequenceswere cycled one after the other each cued by differentscreen colors The subjects attempted to approximate achaotic sequence in one color (eg light green) follow-ing which they would have to match a second chaotic se-quence (lavender screen color) and so on the fourchaotic contingencies following one another across thesession Although all four chaotic sequences used the

Figure 8 Performance by one subject at the end of training to respond stochastically in Neuringer(1986) Shown are comparisons on 6 different statistics between the subject (solid lines and Xs) andthe random model (dashed lines and closed circles) At the end of training the subjectrsquos perfor-mance did not differ statistically from that of the random model on these and four additional sta-tistics From ldquoCan people behave lsquorandomlyrsquo The role of feedbackrdquo by A Neuringer 1986 Jour-nal of Experimental Psychology General 115 p 71 Copyright 1986 by the American PsychologicalAssociation Reprinted with permission

OPERANT VARIABILITY 695

same logistic-difference equation that a model did theywere out of phase and independent If chaotic respond-ing is memory based then the four independent se-quences would be expected to interfere with one anotherthat is approximations to the four chaotic models wouldbe lower than when only a single chaotic output was re-quired as in Neuringer and Voss (1993) That was ex-actly the finding Performances by the three subjectswere signif icantly degraded during the four-segmentphase To put it simply chaotic responses interfered withone another

A different pattern of results was seen in the stochasticsportion of the experiment The same interference proce-dure was used with four independent stochastic sequencesrequired in the four different color conditions As inNeuringer (1986) subjects were required to satisfy eightdifferent statistical tests of randomness and to do this in-dependently in each of the four phases (If responses in anytwo or more of the four stochastic phases became corre-lated above a minimal level responses were considered tobe incorrect thereby prohibiting repetition of a single se-quence in two or more of the phases) For one subjectseven of eight statistics were closer to those of a randommodel during this interference phase than they were at theend of the original training with only a single sequenceand for a second subject all eight statistics were closerThe third subject showed closer-to-random approximationunder the four-segment condition on only two of the eightstatistics but most of her changes were also in the directionof a random generator For example if a random generatoremits equal numbers of ldquouprdquo and ldquodownrdquo sequences andthis subject in the baseline phase had emitted only 47ldquouprdquo sequences then during the four-segment tests shecame to emit ldquouprdquo 55 of the time She changed in the di-rection of the random model but overshot the mark Thuswhereas interference degraded chaotic performance sig-nificantly in all three subjects it improved stochastic per-formances in at least two of them These results are con-sistent with the hypothesis that a memory-based processcontrols chaotic variability and that a stochastic processnot dependent on memory controls stochastic variability

Summary of Tripartite TheoryThe following three processes constitute a tripartite

theory of operant variability attention to random events inthe environment (coin toss) memorial processes (chaosradial maze) and stochastic processes (response se-quences and random-number generation with feedback)Random environmental events assure unpredictabilitymdashfor example in making unbiased selections Remember-ing what was done previously will enable one to avoidrepetitions to vary the students called upon in class tovary the locations when looking for a hidden object orto try different approaches when attempting to solve aproblem External aids such as notes to oneself or sci-entific records help to enhance memory capacities

An endogenous stochastic generator provides the thirdand most controversial source of behavioral variability

When an opponent is attempting to predict your behav-ior in a game or at war it may be functional to behavestochastically Similarly operant conditioning of new re-sponses may depend on stochastic variations It is im-possible to prove stochasticity some presently unknowndeterministic system may ultimately explain the resultsHowever the evidence is consistent with the stochastichypothesis and inconsistent with the alternatives For ex-ample memory is not sufficiently powerful to explain alloperant variability memory-interference proceduresleave stochastic responding intact and behaviors corre-late well with those of stochastic models

Both genetics and reinforcement play important rolesin all three processes Reinforcement is always involvedin that learning provides information about when whatwhere and how much to vary Genetic influences areshown in that not all responses are possible and if theyare possible they are not all equiprobable (eg as inNeuringer et al 2001) Stochastic responding may be bi-ased due to experiences and knowledge that is biasesmay be established by reinforcement contingencies or bythe interface between the organismrsquos propensities (or be-havior systems see Timberlake amp Lucas 1989) and theenvironment In the case of random-number generationby human subjects biases may be due to misconceptionsconcerning what ldquorandomrdquo means However some bi-ases can be overcome through feedback and instructions(Lopes amp Oden 1987 Neuringer 1986)

The importance of variations for evolutionary changeis supported by the many sources of genetic variabilityand by the fact that genetic variability is conserved Asimilar argument holds for operant variability That thereare at least three independent sources of operant varia-tions as well as other elicitors such as environmentalnoise and response extinction supports its importancefor normal adaptive functioning

Structure of VariabilityEvery operant ldquoresponserdquo is actually an instance of a

class with each member of the class exerting the sameeffect on the environment (Skinner 1959) When a rat isreinforced for pressing a lever for example the class isdefined by closure of a microswitch and any way of ac-complishing the closure whether with the left paw theright paw or the mouth is as effective in producing re-inforcement as any other If the contingencies of rein-forcement so specify however the response class can beredefined and circumscribed which is another charac-teristic of the operant For example if only right-pawpresses are acceptable members of the reinforced classresponding will tend to occur only with the right paw(Stokes 1995) Of course the capabilities of the organismplay a role for example including ldquotail pressesrdquo in the op-erant class does not mean that the lever would be or couldbe pressed by the ratrsquos tail The structure of the operantclass therefore emerges from the current reinforcementcontingencies and the organismrsquos capability derivedfrom genetics development and prior experiences

696 NEURINGER

Operant variability has structure as well and as for alloperants that structure is due partly to the organism it-self and partly to the reinforcement contingencies Theresearch described throughout this article has providedexamples of the structure of variability generally in theform of sets of four or eight responses across L and Roperanda But the structure of variations can be speci-f ied more precisely For example Mook and her col-leagues (Mook et al 1993 Mook amp Neuringer 1994)reinforced rats for varying four response sequences acrossL and R levers under a lag 1 contingency Unlike in pre-vious studies however the acceptable class was furtherconstrained so that only sequences beginning with two Rresponses (RRLL RRLR RRRL and RRRR) were rein-forced The rats learned to vary their sequences withinthe imposed limitations that is they generally begantheir sequences with two R responses and varied the se-quences within those limitations Similar results wereobtained in the context of a radial-arm maze in whichonly a subset of arms was baited

The precision with which reinforcement contingen-cies can specify class was highlighted in research per-formed by Burns (1997) and Ross and Neuringer (2002)In the latter case college students drew rectangles on acomputer screen with the computerrsquos mouse Each rec-tangle constituted one trial (the screen cleared at the endof each trial) with reinforcement (points) depending onsimultaneous variations along three dimensions of therectangle location on the screen area of the rectangleand shape with the latter defined as the ratio of the rec-tanglersquos height to its width Although the subjects weregiven no instruction other than to ldquogain pointsrdquo theylearned to vary along all three dimensions with levels ofvariability significantly higher than those of yoked controlsubjects who were reinforced independently of variations

A second part of the Ross and Neuringer (2002) studydemonstrated even greater precision of control To gainpoints three new groups of subjects were required to re-peat along one of the dimensions while simultaneouslyvarying along the other two One group was required torepeatedly draw rectangles of approximately the samesize while simultaneously varying location and shapeanother group had to repeat the shape while varying areaand location and the third to repeat location while vary-ing size and shape Verbal instructions were minimal asthey were in the first part of the study Subjects learnedto satisfy these contingencies as well varying two at-tributes while simultaneously repeating one and signif-icant differences were obtained rapidly within 300 trials

These demonstrations raise questions as to how binaryfeedbackmdashreinforcement presented or notmdashcan simul-taneously control independent behavioral dimensionsThe issue is similar to that discussed above concerningsimultaneous reinforcement of variations and selectionsIt is similar also to those issues faced by researchers at-tempting to understand how the nervous system strength-ens particular synapses when many different neurons firesimultaneously and to ldquoallocation of creditrdquo in artificial

learning models For present purposes suffice it to notethe exquisite precision with which reinforcement controlsvariability and repetition

GeneralizationDo animals and people learn to ldquovaryrdquo implying a

generalizable skill or do they learn a specific distributionof specific responses across particular operanda Thequestion of generalization has been asked of many otherbehavioral competencies such as imitation When chil-dren learn to imitate do they learn stimulusndashresponseassociations (if mother moves her right hand the childlearns to move his right hand) or do they learn moregenerally to ldquoimitaterdquo Evidence indicates that reinforce-ment of imitative instances leads to generalized imitation(see eg Baer amp Sherman 1964 Bandura 1986) A sim-ilar question has been raised when pigeons rats or chil-dren learn to choose a stimulus that matches a previouslyshown sample For example if a pigeon is shown a reddisk as a sample it must then choose red when given achoice between that and blue Does the bird learn to matchIn this case the answer appears to depend on the train-ing procedures If trained with few stimuli such as redgreen and blue then the pigeon learns simple associa-tions (if red sample choose red if blue sample chooseblue) but if hundreds of different stimuli (pictures) areemployed during training then generalized ldquomatchingrdquois learned (Wright 1997) In each of these examples thelearning of a general competency or strategy was shownby transfer-of-training tests What was learned with oneset of stimuli or in one situation transferred to new stim-uli and situations We turn now to the question of whetherlearning to vary transfers

After the subjects in Maltzmanrsquos (1960) experimentdescribed above learned to generate remote verbal as-sociates they were asked to devise unusual uses of com-mon objects In comparison with those in the controlgroup these subjects generated significantly more un-usual uses indicating transfer of training to a differenttask However transfer was not observed in the oppositedirection from unusual uses to verbal associates

Holman et al (1977) using a procedure similar to thatof the Goetz and Baer (1973) research described aboverewarded preschool children verbally for novel drawingsand found increased novelty Unrewarded block construc-tions were also monitored and weak evidence was ob-tained for transfer of novel responding from rewardeddrawing to the unrewarded constructions In a second ex-periment preschoolers engaged in four independent taskspainting pen drawing wood block construction and Legoconstruction When the children were reinforced for vary-ing their pen drawings the novelty of paintings increasedalong with the pen drawings indicating transfer from onetask to the other However transfer to the topographicallydifferent block- or Lego-building tasks was not observed

Eisenberger and Armeli (1997) gave monetary rewardsto one group of fifth- and sixth-grade students the vari-ability group for describing unusual uses of common ob-

OPERANT VARIABILITY 697

jects A second group the repeat group received thesame rewards for describing common uses of the sameobjects In a later test all of the children were asked todraw pictures incorporating a circle and the group that hadpreviously been reinforced for varied responses drewmore unusual (or lower frequency) pictures than did therepeat group Thus there was transfer of novelty trainingfrom one domain to another with additional studies in theseries showing similar effects (Eisenberger Armeli ampPretz 1998 Eisenberger Haskins amp Gambleton 1999Eisenberger amp Selbst 1994) However Eisenberger et al(1999) failed to replicate the transfer-of-training effectThe variability and repeat groups responded similarly dur-ing the test phase In this study separate experimenterstrained and tested the subjects thereby avoiding possibleexperimenter-expectancy effects whereas the same ex-perimenter did both training and testing in Eisenbergerrsquosother studies Across the series of experiments there weremany other cases in which transfer was not observed andmore research is necessary to explain why

In Lee et alrsquos (in press) study with autistic individualsdescribed above after variable answers were trained toquestions verbal variability transferred to new experi-menters and new contexts

Arnesenrsquos (2000) results described above are consis-tent with transfer of training in an animal model Rats inthe Var group were trained to interact variably with oneset of objects When later tested in a new situation withnew objects the Var animals showed (in comparisonwith controls) higher levels of exploration manipulationof the novel objects and discovery of hidden reinforcers

Much of the evidence reviewed in this section there-fore indicates transfer of training and supports a generalldquovaryrdquo competency However there remain unresolvedquestions the most important of which is the extent towhich transfer occurs Note also a methodological pointMany of the studies tested whether transfer occurs whenthe contingencies are neutral with respect to the to-be-transferred behavior Holman et al (1977) is a case inpoint Variations were reinforced in one domain draw-ing and the question was whether variations wouldoccur in another domainmdashblock constructionmdashin whichreinforcement did not depend on varying A negativef indingmdashin this case absence of transfermdashindicatesnothing about the ability to generalize but only aboutthe tendency to do so given the absence of a reason (orcontingency) A better test would be to compare experi-mental and control performances when variability in thenew task is in fact functional as in Arnesen (2000)

SummaryBefore variability can be reinforced it must occur for

reasons other than reinforcement Some of the reasonsinclude variability inherent in all behavioral phenomenaldquoaccidentalrdquo environmental events that are unrelated toparticular behaviors and withholding of reinforcementTransfer from other situations may serve a similar functionFor example if an organism can discriminate shaping-

of-new-response periods from others then it may learngenerally to ldquovaryrdquo during shaping episodes After rein-forcement makes initial contact with response variationsthere are at least three behavioral strategies that can beemployed in order to maintain functionally high levelsof variability respond based on external random eventssuch as the toss of a coin avoid repetitions by remem-bering what responses had previously been made and re-spond stochastically Evidence for an endogenous sto-chastic process was seen in the similarity of responsedistributions to those generated by stochastic sourcesand by findings that interfering with memory for priorresponses either improves or leaves unaffected stochas-tic behaviors Stochastic responding as well as othertypes of variability-generating strategies depend uponcontingencies to define the required level of variabilityand the set from which variations emerge

IMPLICATIONS

One implication of reinforced variability is that oper-ant responses may differ in the extent to which their vari-ability is affected by reinforcement contingenciesmdashthatis operants may be ordered along a continuum of sensi-tivity of variability to reinforcement Another is that re-inforced variability causes us to question the generalityof some basic goals of behavioral psychology includingprediction and control A third is that reinforced vari-ability may help us to provide a scientifically useful wayto describe voluntary action In particular the variabilityof all voluntary behaviors can readily be changed mo-ment to moment by discriminative stimuli and conse-quences Implications for everyday life also will be noted

Operant ContinuumAs was indicated above the operant is composed of

instances that is the operant is a class of many differentresponses all functionally equivalent but often topo-graphically distinct Skinner suggested and I hypothe-size that instances within the class emerge stochastically(see Moxley 1997 for a discussion of Skinnerrsquos posi-tion) Although reinforcement contingencies and contextinfluence the individual probabilities and their distribu-tions emission is stochastic nonetheless The particularmember of the operant class generally cannot be predictedwith a high degree of certainty For example a particu-lar rat may be more likely to press the lever with its leftpaw than its right but left- and right-paw responses areemitted probabilistically This class-nature of the operantsuggests one way to conceptualize operant variability interms of size of class and within-classes probability dis-tributions High variability implies large classes equalprobability distributions or both low variability impliesthe opposite

Variability may be modified more readily in some op-erants than in others implying that operant variability isnot an all-or-none phenomenon but rather a continuumwith reinforced variations in variability (RVV) differing

698 NEURINGER

for different operants RVV might best be understood byanalogy to the concepts of speed and acceleration Bothvariability and speed range from low to high RVV isanalogous to acceleration the rate of change in speedHighly sensitive RVV is indicated by the possibility ofreinforcing many different levels of variability from rep-etition to random-like responding and by the fact thatchanges in level occur readily and rapidly

Different operants manifest differences in RVV Forexample variability of chain pulls changed more thanthat of leverpresses when variability was reinforced inrats (Morgan amp Neuringer 1990) Some operant-like re-sponses may show relatively low RVV For example fir-ing rates increase in single hippocampal nerve cellswhen dopaminergic substances are contingent upon theresponse (Stein Xue amp Belluzzi 1994) suggesting thatnerve f iring is controlled by reinforcement To datehowever there is no evidence that variability of nervefiring is similarly controlled Other cases might includereinforcement of nondiscriminable (subconscious) re-sponses such as Hefferline and Pererarsquos (1963) reinforce-ment of subaware thumb twitches and biofeedback ofmuscle and cardiovascular activity In these cases rein-forcement may modify performance but it is not knownwhether variability is also sensitive to reinforcement andif so to what degree Human verbal variability may bethe most readily manipulable of all behaviors the rangeof predictable to unpredictable verbal responses beingimmense with extraordinary sensitivity to context andconsequence RVV may also differ across organisms (per-haps from simple to complex) individuals (perhaps asan index of intelligence or learning ability) and physio-logical and psychological states (WKY vs SHR depres-sion and autism)

RVV may help to distinguish operant responses fromother superficially similar cases of selection by conse-quences That is the ldquooperantrdquo may also best be concep-tualized as a continuum with RVV an important deter-miner of position along the operant continuum Oneexample of selection by consequences but low ldquooperant-nessrdquo is genetic change Variability is essential for evolu-tionary change of course and levels of genetic variabil-ity are sometimes affected by environmental conditions(Moxon 1997) but there are few examples of levels ofvariability being influenced by selection pressures Thatis the source of much genetic variability is thought to berandom not influenced by current pressures and needsThus although selected by consequences genetic changemay be low along the operant continuum Other exam-ples of operant-like phenomena for which variability although important may be relatively insensitive to con-sequencesmdashvariability in these cases is elicited not se-lectedmdashinclude the development of the central nervoussystem and immune system responses (Hull et al 2001)In both of these cases change depends upon selection byconsequences but it is not clear whether variability canalso be selected Most commonly studied operant re-sponses on the other hand show high sensitivity of vari-

ability to reinforcing events But to reiterate the mainpoint of this section operants differ in terms of RVV andtherefore in terms of their ldquooperantnessrdquo from low sensi-tivity (possibly responses of single nerve cells) to extra-ordinarily high sensitivity (verbal behavior) The operantnature of variability as well as that of every other oper-ant dimension is best conceptualized as a continuum

Prediction and ControlSmith (1992) described Skinnerian operant condition-

ing theory as Baconian in nature a primary goal of Skin-nerian behaviorists being to control behavior and hecontrasted this with the more Aristotelian goal of de-scription Skinner and his students (eg Zuriff 1985) in-deed emphasized prediction and control Many of thestudies reviewed above however involved reinforce-ment procedures that increase variability and conse-quently decrease predictability It is not only ignorancetherefore that restricts prediction and control as Skin-ner suggested but reinforcement itself Operant meth-ods and theories when applied to variability lead togreater emphasis on description with control over pre-cise response characteristics possible under some butnot all conditions

In summary an operant is a class of responses underthe control of contingencies of reinforcement The classcan be controlled and predicted given knowledge ofconditioning history Indeed the size of the class andpossibly the distribution of within-classes probabilities orlevel of variability can also be controlled and predictedBut to the extent that instances emerge stochastically aparticular instance cannot be predicted The instant re-sponse can be explained as a member of a particularclass but prediction of the instance will be diff icultsometimes to the extreme This point will be developedin the next section

Voluntary BehaviorI hypothesize that voluntary behaviors are operants for

which RVVs are highly sensitive to influence by conse-quences One way to test this hypothesis would be togather empirical data concerning how voluntary versusinvoluntary actions are classified by human observers Asfar as I know there are no psychophysical data concern-ing this question but the hypothesis implies that highsensitivity of variability to operant contingencies RVVswill be present in all actions perceived to be voluntary

This operant-variability theory of voluntary action isbased on the supposition of two attributes First volun-tary acts must be potentially explainable (given suff i-cient knowledge) on the basis of genetics conditioninghistories current contingencies anticipated conse-quences intentions goals and the like That is volun-tary behaviors are lawful The second attribute is that thevoluntary act must be potentially unpredictable That isvoluntary acts are ldquofreerdquo of control by identifiable stim-uli If behavior is perceived necessarily to result frommdashto be predicted on the basis ofmdashexternal events includ-

OPERANT VARIABILITY 699

ing reinforcement contingencies it will not be perceivedas voluntary ldquoVoluntaryrdquo implies potential indepen-dence from stimulusndashresponse or responsendashreinforcerdetermination According to this theory both explain-ability and unpredictability are necessary attributes ofvoluntary acts

How though can behaviors be simultaneously ex-plainable and unpredictable Reinforcement contingen-cies play two essential roles Reinforcement (togetherwith genes prior experiences and current environment)helps to determine the set of possible behaviors fromwhich a voluntary act emergesmdashthat is the operant classReinforcement (together with the other contributorsspecified above) also helps to determine the level of vari-ability within the class Thus both instances comprisinga class and level of variability within that class can bepredictedmdashagain given sufficient knowledge Howeverthe individual response is at least sometimes (or relatively)unpredictable The reason is that the operant class fromwhich the response stochastically emerges is sometimesextremely large (eg ldquoname a wordrdquo or ldquodo somethingrdquo)and the contingencies sometimes call for equiprobablestochastic emergence (eg ldquofool merdquo) In the latter casea voluntary act may appear to be foolish meaninglessor unwarranted but if it is voluntary it can potentially beexplained (For example in response to the questionldquoWhy did you do that silly actrdquo the answer could beldquoJust to show that you canrsquot predict my behaviorrdquo seeScriven 1965) In these cases prediction of instancesfalls to very low levels indeed and I hypothesize evenan all-knowing observer will be unable to predict thenext response at greater-than-chance levels which aregiven by the size of the operant class and by within-classesprobability distribution Even within small classes (egin the game rockndashscissorsndashpaper in which unpredictabil-ity among the three possible responses is functional) be-havioral stochasticity may lead to unpredictability of theinstant response Note however that although voluntarybehaviors are sometimes unpredictable the opposite isoften the case When contingencies call for highly con-strained particular responses predictions can easily bemade For example if a police officer asks your nameyour response can most likely be predicted Voluntary re-sponses are therefore described as potentially unpre-dictable with variability contingencies engendering dif-ferent levels of unpredictability

According to this theory behaviors low in the sensitivity-of-variability dimension are described as relatively in-voluntary and those that are high in said dimension areperceived as voluntary Low RVV may be shown at anylevel along the variability continuum For example al-ways responding in the same unchanging way will beperceived as involuntary but so will always respondingrandomly (The coin does not voluntarily come up withheads or tails) This point is essential More than 2000years ago Epicurus argued that occasional randomswerves of atoms provided the source of voluntary be-havior in an otherwise determined universe but al-

though this conjecture was prescient it was not quitecorrect ldquoFree-willedrdquo or voluntary action whether it isbased on random swerves or on quantum mechanics isnot explained by variability per se but by functionalchanges in levels of variabilitymdashthat is by rapid andprecise changes in response to environmental demandsResponses can be placed along a voluntaryndashinvoluntarycontinuum isomorphic to the operantndashrespondent con-tinuum with voluntary behavior being functional pre-cisely in the same way as operant behavior is ThusSkinnerrsquos (1974) suggestion that voluntary behavior isoperant behavior was almost correct Only those oper-ants for which variability is highly sensitive to conse-quences will be perceived as voluntary

One additional example might help Operant classes(sets of potential responses) are created ldquoon linerdquo as we goabout our daily activities As we walk from car to officewe tend to walk in a characteristic way take the samepath with tightly constrained operant classes followingone after the other in rapid succession Class sizes aresmall and within-classes probability distributions arehighly peaked Predictions of next responses by a knowl-edgeable observer can readily be made However classsizes can be increased and probability distributions flat-tened we can walk in quite strange ways indeed dancealong a new path sing hand candy to passers-by Strangebehavior indeed but made possible by the ability of op-erant sets to be modified moment to moment not only interms of class definition but in terms of class size lev-els of within-classes variations and so on The same istrue for conversation Moment-to-moment changes inverbal class definition size and predictability of within-classes instances are hallmarks of voluntary behavior

GenerativityRelated to the definition of the operant and to our discus-

sion of voluntary action is that fact that operant behaviormdashleverpressing no less than languagemdashis by its naturegenerative This claim follows from the functional na-ture of the class of responses (all members of the classare effective in producing an outcome) and the stochas-tic emission of instances From these flow never-before-emitted instances patterns and sequences (see Epstein1996) Operants especially those high on the RVV con-tinuum are inherently generative

Self-ExperimentationVariations systematic or otherwise are a necessary part

of scientific work Neuringer (1981) and Roberts and Neu-ringer (1998) described how variations can be brought to bear explicitly on everyday life in the form of self-experiments Individuals vary aspects of their lives suchas the foods they eat social contacts and methods ofmemorizing they keep records concerning the effects ofthese manipulations and form hypotheses concerningcontrolling variables Using self-experimental methodsindividuals can learn how what when where and howmuch to vary Explicitly varying onersquos everyday behav-

700 NEURINGER

iors and thoughts may help to engender what Langer(1989 2000) has referred to as mindful livingmdashthat isnonautomatic voluntary engagement with beneficialeffects in different areas including learning The impor-tant point here is that variability can be voluntarily cho-sen maintained and modified in everyday life and thatexplicit variations may have important consequences

Implications for SocietyConsideration should be given to the importance of re-

inforcement contingencies to support variability in so-cial milieus A basic point made throughout the presentarticle is that merely permitting behavior to vary doesnot necessarily engender or maximize variability (as isshown by Yoke and Prob schedules) Analogously merelypermitting varied social behavior as in a free or laissez-faire society may not suffice to produce the desired di-versity If the results shown for individual organismscan provide useful information for groups (see eg Grottamp Neuringer 1974) then when variations are desiredsocieties might reward individuals for varying not sim-ply permit it

SummaryOperants differ in the extent to which reinforcers in-

fluence levels of variation When variability is highlysensitive to reinforcement the operant may be perceivedto be voluntary Voluntary responses have two attributesThey are functional implying that the class of responsesmdashthe operantmdashcan be predicted given sufficient knowl-edge But instances emerge stochastically implyingsome degree of unpredictability Because reinforcementof variability can generate extremely large classes withstochastic emergence of instances it suggests limitationsof the two basic goals of Skinnerian behavioral theorymdashnamely prediction and control Operant variability also hasimplications for self-experimentation and for society

RELATED AREAS

Operant variability may be important in many otherareas and I will briefly review some of these

Negative reinforcement To my knowledge there havebeen no published studies in which variability is nega-tively reinforced High variability might sometimesavoid or reduce a punisher such as loss of goods ormoney but at other times low variability could be func-tional Effects of positive and negative reinforcement onvariability may have different evolutionary bases onederived from increasing probabilities of locating sourcesof food or mates the other from avoiding confrontationsor predation (Driver amp Humphries 1988) One questionconcerns how quickly variability can be changed by neg-ative as opposed to positive reinforcement There isconsiderable evidence that loss functions differ fromgain functions (Kahneman amp Tversky 1984) possiblyindicating that positive reinforcement-of-variabilityfunctions will differ from negative ones as well

Resistance to change In some of the studies describedabove variability appeared to be more resistant tochange than repetitions were for example alcohol hadlittle effect on variable operant responses but interferedwith repeated ones Doughty and Lattal (2001) tested thegenerality of these f indings using four-response se-quences with pigeons a threshold variability contin-gency in the Var component of a multiple schedule andLRLR in the Rep component Doughty and Lattal chal-lenged stability of responding by prefeeding the birds inone experiment and providing reinforcers independentlyof responding in the other (see Nevin 1992) Operantvariability was again more resistant to change than wasoperant repetition Thus reinforcement of variabilitymight increase the stability of responding (that is its re-sistance to change) a seemingly paradoxical findingHowever Mechner (1992) suggested that this is the casefor complex skills such as piano performance Varyingpractice routines will make the performance relativelyinsensitive to unforeseen changes in the environmentCare must be taken however when variation is comparedwith repetition because they have different measuresAlso the results reported by Doughty and Lattal werebased on the performance of three pigeons and were notconsistent throughout the experiment The hypothesiswarrants further study

Attraction In some bird species complex male songsattract females (for a review see Catchpole amp Slater1995) There is evidence that male song complexity isunder the discriminative stimulus control of a receptive fe-male (Searcy amp Yasukawa 1990) That explicit reinforce-ment may play a role in such song complexity is suggestedby the Manabe et al (1997) study described above An in-teresting question for further research is the extent towhich operant contingencies contribute to the attractioncreated by behavioral variability in these and other species

Discrimination of variability Animals and people canuse the variability of a stimulus array or entropy as a dis-criminative stimulus (Young amp Wasserman 2001) Whatis the relationship between perception of variability onthe one hand and emission of variable responses on theother Do individuals attempt to match their own levelsof variation to those of others in their environments whichwould involve both perception and emission How mightdiscrimination of onersquos own level of variability influenceonersquos ability to modify such levels and do influences thatobscure discriminations therefore affect ability to change(see eg van der Meer van der Weel amp Lee 1995)Hunziker et al (2002) report that although variabilitywas successfully reinforced their human subjects couldnot correctly describe the contingencies of reinforce-ment a finding similar to those of informal observationsin other studies (eg Neuringer Deiss amp Imig 2000)but whether subjects could discriminate variability levelsrather than contingencies was not studied

Attention and awareness When attention is subject tointerference or is withdrawn from the task at hand abilityto vary is degraded (eg Baddeley Emslie Kolodny amp

OPERANT VARIABILITY 701

Duncan 1998 Evans amp Graham 1980 Neuringer amp Voss2002) It is unclear at present how attention influencesthe different processes underlying operant variabilityand in particular memorial and stochastic processesRich areas for additional study are the relationships be-tween attention awareness and consciousness on theone hand and operant variability on the other Voluntaryaction and consciousness are often related (see egLibet 1999)

Concepts Concepts are thought by many to be sets ofinstances (much like operant sets) whose structures aregraded meaning that there are more or less typical in-stances of such concepts as birds (robin is typical ostrichis not) Barsalou (1987) showed that the gradation ofthese cognitive structures is manipulable or as he putsit not stable A number of variables influence structureincluding context individual differences and points ofview Barsalou argued that these structures are createdas the need arisesmdashthat is on line by the thinking indi-vidual The structure of concepts may parallel that of op-erants and in both cases levels of within-structures di-versity or variability may be modified by consequencesOperant variability may be found in the cognitive as wellas in the behavioral domain

Play and humor Highly variable behaviors are ob-served when animals and people especially children en-gage in play and humor (see eg Fagen 1981) The re-sult of play may be acquisition of new responses (seeSutton-Smith 1975) Although not currently needed re-sponses learned during play may provide the source ofvariations for successful shaping of behaviors at anothertime This is quite analogous to the role of conserved ge-netic variants which are passed along to progeny but donot currently serve any identifiable role Play may alsoinvolve learning how to vary operantly

Reinforcement by variability The self-maintainingnature of play suggests that engaging in variable behav-iors may itself be reinforcing Indirect support comesfrom findings that varied stimulation can serve as a re-inforcer and that stimulus repetition becomes aversive(Fiske amp Maddi 1961) Possibly also related are findingsthat varying reinforcers and permitting choice among re-inforcers increase their value (Bowman Piazza FisherHagopian amp Kogan 1997) and that permitting choicesamong varied responses is also preferred to demandingfixed responses (Catania 1980)

Foraging Two issues regarding variability are presentin the foraging literature One has to do with risk prefer-ence versus avoidance (see eg Shettleworth 1998) Asecond has to do with search strategies Most reportedcases of foraging involve systematic search patterns(Stephens amp Krebs 1986) However search patterns aresometimes stochastic or highly variable (see eg Peter-son 1997) One question is whether some species can varybetween these two strategies and the challenge is to iden-tify the pressures leading to and the advantages of each

Animal training Wilkes (1996) has applied operantvariability techniques in developing training proceduresfor pet dogs More generally reinforcement of variabil-

ity may enable the gaining of control over behaviors ofanimals and as has been suggested above may be an im-portant part of the process of shaping

Models of learning Reinforced variations in variabil-ity may be important when human and animal learningis being modeled whether the models are mathematicalin form computer based or machine based (Blumberget al 2002 Brooks 1999 Sutton amp Barto 1998)

CONCLUDING COMMENT

Genes interact with environments to produce diverseanatomical physiological and behavioral phenotypesGenes and environments also provide for another level ofdiversitymdashnamely operant variability There are manysources of operant variability including random environ-mental events memory for past responses and stochasticgeneration each of which ultimately derives from ge-netic and environmental influences To the extent thatoperant variability is based on a stochastic source how-ever the behavioral results cannot be predicted fromknowledge of these influences That is organisms haveevolved to behave unpredictably The positing of sto-chasticity to explain behavior has a history extending atleast as far back as Epicurus with major contributors in-cluding Gustav Fechner (see Heidelberger 1987) EgonBrunswik (see Gigerenzer 1987) Skinner (1959) andSuppes (1984) Knowledge of the stochastic nature ofthe operant and of how it is controlled leads to a worldview that differs from genetic and environmental deter-minism with roots in the works of the scholars just men-tioned At the scientific level this view leads to the ex-perimental analysis of consequence-influenced variabilityAt the philosophical level it leads to explanations of vol-untary action At the personal level it can lead to change

REFERENCES

Amabile T M (1983) The social psychology of creativity New YorkSpringer-Verlag

Antonitis J J (1951) Response variability in the white rat duringconditioning extinction and reconditioning Journal of Experimen-tal Psychology 42 273-281

Archer J amp Birke L (1983) Exploration in animals and humansCambridge Van Nostrand Reinhold

Arnesen E M (2000) Reinforcement of object manipulation in-creases discovery Unpublished undergraduate thesis Reed College

Baddeley A D (1966) The capacity for generating information byrandomization Quarterly Journal of Experimental Psychology 18119-129

Baddeley A [D] Emslie H Kolodny J amp Duncan J (1998)Random generation and the executive control of working memoryQuarterly Journal of Experimental Psychology 51A 819-852

Baer D M amp Sherman J A (1964) Reinforcement control of gen-eralized imitation in young children Journal of Experimental ChildPsychology 1 37-49

Baker M J amp Koegel R L (1999) Transforming children withautismrsquos ritualistic behaviors into positive play interactions with sib-lings Paper presented at the meeting of the California Association ofBehavior Analysis San Francisco

Balsam P D Deich J D Ohyama T amp Stokes P D (1998) Ori-gins of new behavior In W OrsquoDonohue (Ed) Learning and behav-ior therapy (pp 403-420) Boston Allyn amp Bacon

Balsam P D Paterniti A Zechowy K amp Stokes P D (2002)

OPERANT VARIABILITY 677

Some studies report that variability increases when rein-forcement frequencies decrease (Boren Moerschbaecheramp Whyte 1978 Tatham Wanchisen amp Hineline 1993Tremont 1984) but others report small or no effects(Blough 1966 Eckerman amp Lanson 1969 Herrnstein1961 Machado 1989)

Grunow and Neuringer (2002 Experiment 1) inde-pendently manipulated variability contingencies and re-inforcement frequencies and their results may help toexplain the previous inconsistencies Four groups of ratswere studied with the contingencies for each group re-quiring a different minimum level of variability (A thresh-old contingency was employed the details of which willbe described below) In the initial phase reinforcementwas provided whenever the respective contingencieswere met (reinforcement frequency was relatively highin this phase) and as can be expected from the Blough(1966) and Machado (1989) studies described above lev-els of variability were directly controlled by the contin-gencies In a second phase with the same variabilitycontingencies held constant reinforcement was limitedto no more than once per minute (VI 1 or intermediatefrequency) In a third phase meeting of the variabilitycontingency was reinforced on an average of only once

every 5 min (VI 5 or low frequency) We observed a sig-nificant interaction between reinforcement frequencyand variability contingency (Figure 2) which helps toexplain some of the previous inconsistencies In thegroup that was reinforced for low variability representedby the triangles in Figure 2 as reinforcement frequencydecreased variability increased slightly in the group re-inforced for high variability represented by the squareswhen reinforcement decreased variability decreasedand in the intermediate groups variability was unchangedAlthough these effects were small they reached statisti-cal significance As can be seen in Figure 2 variabilitycontingencies affected variability more than did rein-forcement frequencies and effects of the latter were in-fluenced by the contingencies

An example might help to illustrate these complexfindings When low variability is required perhaps as in assembly-line work low variability emerges When highvariability is required perhaps as in artistic work re-sponse variability is high Now what happens if the normalrate of reinforcement for each of these tasks is loweredAccording to Grunow and Neuringerrsquos (2002) findingsthe assembly-line workerrsquos variability would increasebut only slightly and the artistrsquos variability would de-

Figure 2 Average U-values across groups of rats in Grunow and Neuringer (2002)The highest function (squares) is from a group reinforced for highest sequence vari-ability the lowest function (triangles) is from a group reinforced under a lenient con-tingencymdashthat is low variability sufficed for reinforcement the other groups (Xs andcircles) were reinforced for intermediate levels of variability Each group was rein-forced at three frequencies each time the contingency was satisfied (continuous rein-forcement ie CRF) on an average of not more than once per minute (VI 1) and onan average of not more than once every 5 min (VI 5) High U-values indicate high lev-els of sequence variability and low U-values indicate relatively repetitive respondingError bars represent standard errors of the mean

678 NEURINGER

crease somewhat Importantly the artist would continueto behave much more variably than the assembly-lineworker Both contingency and frequency affect responsevariability but contingency appears to be more influen-tial at least over the range of values studied to date Thesefindings if general are important for those who withholdreinforcement in order to encourage response variationsas is often the case when behavior modifiers attempt toshape new behaviors Effects of withholding may dependon baseline contingencies and a more productive proce-dure might be to reinforce variability directly We willconsider this suggestion further below

Discriminative ControlThe operant response as defined by Skinner is con-

trolled by discriminative stimuli as well as by conse-quences For example red traffic lights are the discrimi-native stimulus for depressing the automobile brake thedistance of the golf ball from the hole provides a discrim-inative stimulus for how hard to hit the ball In the labora-tory discriminative control is often studied with multipleschedules in which one reinforcement contingency suchas fixed ratio is operative in the presence of one stimulussuch as a red keylight and a different contingencymdashforexample extinctionmdashis operative in a second stimulusmdashfor example a green keylight Discriminative control isshown by differences in responding Variability is alsocontrolled by discriminative cues

Exteroceptive discriminative stimulus controlPage and Neuringer (1985 Experiment 6) reinforced pi-geons for repeating a single sequence LRRLL in thepresence of blue keylights and for variable sequences inthe presence of red Blue and red alternated after every10 reinforcements The birds learned to repeat in thepresence of blue and to vary in the presence of red andwhen the stimulus relationships were reversed the birdsreversed their patterns Cohen Neuringer and Rhodes(1990) obtained similar results with rats Denney andNeuringer (1998) extended this research to show thatdiscriminative control did not depend on reinforced repe-titions in one of the components Again within the contextof a multiple schedule rats were rewarded for variablesequences under one stimulus (Var) and independentlyof variability under a second stimulus (Yoke) Frequen-cies of reinforcement were identical in the two stimulusconditions That the stimuli exerted significant controlover variability was indicated by three main findingsVariability was higher in Var than in Yoke differences invariability were greatest immediately following stimuluschange and when the discriminative stimuli were re-moved and all other aspects of the program kept constantvariability immediately converged on an intermediate andidentical level in the two components The conclusion isthat operant variability is controlled by discriminativestimuli These findings are important in terms of both ex-plaining observed levels of variability and controllingthem For tasks in which response variations are impor-tant such as when new operants are being shaped or when

an individual is attempting to behave creatively or solveproblems cues that indicate ldquoVaryrdquo can be helpful

Discriminative control without exteroceptive stim-uli The relationship between a response and a reinforcercan itself provide a discriminative stimulus even whenexternal cues such as lights or tones are unavailable Forexample under an FI schedule there is a period duringwhich reinforcers cannot be obtained The subject learns topause following each reinforcement although no cuesother than the reinforcer are provided Another case is themixed schedule identical to the multiple schedule describedabove but with no external cues Animals sometimeslearn to respond differentially to the consequences oftheir behaviors even under such stimulus-impoverishedconditions (Ferster amp Skinner 1957) The question stud-ied next was whether responsendashreinforcer relationshipscan provide discriminative cues sufficient to control lev-els of variability This is an important question becauseas was indicated above it has been assumed that with-holding reinforcement elicits variability The present hy-pothesis is that absence of reinforcement can sometimesserve as a discriminative cue for operant variability Thedistinction between elicited (respondent) and reinforced(operant) variability will be revisited shortly

Hopson Burt and Neuringer (2002) rewarded ratsunder a mixed schedule in which two periods of time orcomponents Var and Repetition (Rep) alternated with-out any external cues In the Var component four-responsesequences of L and R leverpresses were reinforced ifthey met a threshold variability contingency in the Repcomponent only repetitions of LLLL were reinforced(Probabilities of reinforcement were equalized in the twocomponents by intermittent reinforcement of LLLL inRep) The results demonstrated discriminative controldespite the absence of exteroceptive cues (see Figure 3)With trials since the unmarked switch into the Var com-ponent shown in the left column variability increasedand LLLL sequences decreased with trials since the unmarked switch into Rep shown in the right columnresponding became increasingly repetitive until LLLLsequences predominated The two functions were sim-ilar which is an important finding in that it indicatesdiscriminative control rather than elicitation If variabil-ity had been elicited because reinforcement was tem-porarily withheld (when the contingencies were switched)then variability should have increased following entryinto both Rep and Var components but that was not theresult

Therefore variability that is observed when rein-forcement is withheld for short periods as when a newresponse is being shaped may be partly discriminativelycontrolled despite the absence of external cues that isanimals and people may learn when it is functional tovary and some of the cues may come from responsendashoutcome relationships Control over this operant vari-ability may be more precise than that elicited by extinc-tion because as was noted above operant contingenciesdefine both the set of acceptable variations and the level

OPERANT VARIABILITY 679

of required variability It may also be maintained muchlonger because as will be seen shortly extinction weakensresponding

ChoiceThe frequency of one response relative to others or

response probability is used as a measure of choice orpreference Choices are found to match the probabilitiesof their consequences For example relative frequenciesof pecking one of two keys approximate relative fre-

quencies of reinforcement Neuringer (1992) showedthat the same matching relationship also governs choicesby pigeons to vary or repeat The more variations werereinforced the more the pigeons chose to vary and sim-ilarly for repetitions Each trial consisted of four L andR responses and reinforcements for varying and for re-peating were systematically manipulated The Var con-tingency was met whenever the current sequence dif-fered from each of the previous three sequencesmdasha lag 3criterion The Rep contingency was satisfied if the cur-

Figure 3 The left column shows the probabilities ( y-axis) that each of five rats met a variability contin-gency (solid circles) versus a repetition contingency (open circles) when they were being reinforced for vary-ing (Hopson Burt amp Neuringer 2002) Along the x-axis are trials since the mixed schedule (no cues pro-vided to indicate the component) had switched into the variability-reinforcing component The right columnshows the corresponding data for performances when reinforcement was contingent upon repetitions thex-axis here indicating trials since the unmarked switch into the repetition-reinforcing component

680 NEURINGER

rent sequence repeated at least one of the previous threeOn an average of once every 30 sec (VI 30 sec) the com-puter probabilistically primed reinforcement for Var orRep the prime being maintained until the reinforcementhad been collected Thus if the computer primed Var re-inforcement was delivered whenever the lag 3 variabilitycontingency was next met Similarly if Rep had beenprimed the next Rep sequence was reinforced Proba-bilities of Var and Rep reinforcements were systemati-cally manipulated the two always summing to 10 and amatching-type relationship was obtained (Figure 4panel A) The relationship was more complex than sim-ple matching possibly because if the pigeons chose tovary by responding stochastically there was some prob-ability due to chance that the current sequence wouldrepeat a previous sequence This was shown by a com-parison of pigeon performances to the performance of acomputer-simulated stochastic model programmed tomatch its Var choices to the probability of Var reinforce-ments The pigeonsrsquo choice distributions were found tobe the same as the modelrsquos (Figure 4 panel B) Thuschoices to vary or repeat were governed by relative fre-quencies of reinforcement An animal can choose whetherwhen and how much to vary a result consistent withthose described above from Blough (1966) Machado(1989) and Grunow and Neuringer (2002) Choices ofwhether or not to vary appear to be governed similarly toall other operant choices

These f indings may have important applicationsWhen an individual repeatedly responds in a way that isnonfunctional reinforcement of choices to vary mayhelp the individual to emerge from the ineffective pat-tern This can be true for those experiencing everydayproblems but feeling helpless to change and it may beespecially important for those experiencing depressionor who respond in otherwise abnormally stereotypedmanners such as those with autism These topics will beconsidered in a later section

ExtinctionAs was indicated above extinction is often found to

increase variability but almost all studies of this effectinvolve responses that had previously been repetitiveAntonitis (1951) for example reinforced rats for pokingtheir noses anywhere along a 50-cm horizontal openingAlthough the exact location of pokes did not matter withcontinued training responses became limited to one or afew locations along the strip When reinforcement waswithheld variability of location increased (see also Eck-erman amp Lanson 1969) Variability produced by extinc-tion has been reported for many other response dimen-sions as well including force (Notterman amp Mintz1965) number (Mechner 1958) topography (Stokes1995) and sequence (Balsam Paterniti Zechowy ampStokes 2002 Mechner Hyten Field amp Madden 1997)

The issue is complicated however by other studiesthat show that previously reinforced responses and re-sponse patterns are maintained intact during and afterextinction That is extinction does not break down

learned patterns For example when Schwartz (1981) re-inforced pigeons for any sequence of four L and four Rresponses each bird developed a dominant sequence asdescribed above and these sequences were maintainedduring extinction Despite decreasing response ratesduring extinction response patterns were retained Sim-ilar maintenance of response structure has been ob-served during extinction following reinforcement of

Figure 4 Average performance across a group of pigeons inNeuringerrsquos (1992) study of choices to vary or repeat (A) Loga-rithms of the ratio of number of times the pigeons met the Varcontingency divided by the times they met the Rep contingencyas a function of the analogous numbers of reinforcements re-ceived for each Points represent group averages in two separateexperiments Also shown is the least squares best-fitting line(B) Changes in U-value an index of response variability as afunction of the obtained percentage of reinforcements for vary-ing The solid line in panel B represents the performance of acomputer-based model that matched its choices to vary (vs re-peat) to its obtained relative frequency of reinforcement for vary-ing From ldquoChoosing to vary and repeatrdquo by A Neuringer 1992Psychological Science 3 p 248 Copyright 1992 by the AmericanPsychological Society Reprinted with permission

OPERANT VARIABILITY 681

choices (Myerson amp Hale 1988) responding underfixed-interval contingencies (Machado amp Cevik 1998)visual discriminations (Nevin 1967) and matching tosample (Cumming Berryman Cohen amp Lanson 1967)How can stability during extinction be reconciled withincreased variability

Neuringer Kornell and Olufs (2001) wrestled withthis question in a study of extinction following rein-forcement under three different contingencies An oper-ant chamber contained two levers left (L) and right (R)on either side of a reinforcement dispenser and a key (K)in the rear of the chamber distant from the dispenserOne group of rats Rep was reinforced for repeating afixed sequence of responses LKR A second group Varwas reinforced for variations across the three operandaunder a threshold contingency Reinforcement for a thirdgroup Yoke was matched to those of the Var animalsand was therefore independent of the particular sequencesBecause of the unusual configuration of operandaskewed distributions of sequence probabilities were ob-served in all of the groups an outcome that was antici-pated and desired in order to document possible in-creases in variability during extinctionmdasha flattening ofthe distributionmdashas well as possible decreases That isduring the reinforcement phase the Rep group came torepeat the LKR sequence with high probability the Yokegroup tended to emit the easiest sequences such as LLLand RRR but distributed their responses over more se-quences than the Rep group did and the Var group dis-tributed most widely although because of differences inthe operanda and in their locations some sequences weremore preferred than others and in particular sequencesending with a K response were least likely (the key beingfarthest from the reinforcement dispenser) Thus thecontingencies resulted in three different distributions orhierarchies of sequence probabilities very steep forRep intermediate for Yoke and flattest for Var

The main finding was that withholding reinforcementhad similar effects in the three groups In each responserates fell to low levels across the four sessions of extinc-tion Furthermore sequences that were most frequentduring the reinforcement phase continued to be most fre-quent throughout extinction A slight flattening of thedistributions was observed however with the lowestprobability sequences increasing relative to the rein-forcement phase Thus because the overall hierarchy ofsequence frequencies was maintained the results indi-cated stability but because the distributions were some-what flattened the results also showed slightly increasedvariability It is as if the strategy during extinction wereto keep doing what had worked in the past but occa-sionally to do something very different This compli-cated story integrates two effects of extinction increasedvariability and maintained stability Most importantly forthe present discussion extinction had similar effects onvariable yoked and repetitive operants

One additional comparison is of interest Recall thatGrunow and Neuringer (2002) found that when contin-

gencies demanded high variability obtained levels ofvariability decreased slightly as reinforcement frequen-cies were lowered This result is opposite to the Vargrouprsquos slight increase in variability just described underextinction conditions The reason may be that extinction-induced variability is short lived lasting only as long asresponding continues perhaps for hours or a few ses-sions at most whereas the variability engendered by theVI 5 intermittent schedule of reinforcement in Grunowand Neuringer can be maintained indefinitely The de-crease in Grunow and Neuringerrsquos operant variabilitymay be due to the weakening of the reinforcement effect(ie reinforcement is less frequent) whereas the in-crease following extinction may be a short-term elicitedeffect This distinction between reinforced and elicitedvariability is important and will be discussed further inthe next section

Operant and Respondent InteractionsElicited or what is sometimes referred to as respondent

effects occur when environmental changes that affectbehavior do not depend on (are not caused by) the behav-ior Pavlovian conditioning being the most commonlystudied example Operant effects are observed when a re-inforcer depends on the response to produce it Althoughfor historical reasons respondent and operant effects areoften studied separately most behaviors result from acombination or interaction of the two Variability is nodifferent and we turn to an example of such an interaction

As reinforcement is approached under FI or fixed-ratio (FR) schedules response rates are commonly ob-served to increase Similarly when reinforcement de-pends on accuracy as in discrimination learning ormatching to sample it too is found to increase within theFI or the FR (Nelson 1978 Nevin 1967) Cherot Jonesand Neuringer (1996) asked whether the same patternheld for reinforced variability Does operant variabilityincrease as reinforcement for varying is approachedThe question may be related to such real-world cases aswhen a composer is rushing to complete a compositionbefore a scheduled performance or when a programmeris trying to solve a problem before an upcoming meetingThese activities involve at least to some extent non-repetitive original or variable thought and behaviorThe question is whether variability increases with prox-imity to or anticipation of reinforcement for varying

Cherot et al (1996) employed trials consisting of fourresponses across L and R operanda and a lag 4 variabil-ity contingency with rats in one experiment and pigeonsin another Unlike in any of the cases described so farhowever the lag-variability contingency had to be satis-fied four times in order for a single reinforcer to be ob-tained With ldquordquo indicating successful completion of theVar contingency ldquo2rdquo indicating failure to meet that con-tingency and PELLET indicating the food reinforcer anexample follows LRLL RRRL LRLL2 LRLL2RLLR RRRL2 LRRR+PELLET Thus food reinforce-ment was provided only after the subject had satisfied

682 NEURINGER

the variability contingency four times FR 4 The resultswere that as the food reinforcer was approached (ie asprogress was made through the ratio) the subjects wereless and less successful at varying The probability ofmeeting the variability contingency in the first trial afterreinforcement was higher than it was in the second whichin turn was higher than that in the third trial and so onFor comparison a Rep group was reinforced for repeatingsequences once again with reinforcement only after foursuccessful repetitions Probability of repeating increasedwith proximity to reinforcement a finding consistentwith the literature but opposite that for the variabilitygroup Thus approach to reinforcement facilitated oper-ant repetition but interfered with operant variabilitySimilar variability-interfering effects have been docu-mented for within-trials responses The probability that aresponse repeats the just prior response increases within atrial even when sequence variability is reinforced (McElroyamp Neuringer 1990 Neuringer 1991) To return to the ex-ample of a composer rushing to complete her composi-tion she may indeed have increasing difficulty respond-ing creatively under the pressure of a deadline On theother hand the soloist may find it easier to repeat thecorrect performance during practice periods as the con-cert nears How can these results be reconciled with themany findings that variability can be reinforced

An additional result of Cherot et al (1996) provides ananswer The Var animals responded much more variablyoverall than did the Rep animals Although variability inthe Var groups decreased with approach to the reinforceroverall variability was always much higher when it was re-inforced (Var group) than when it was not (Rep group)Reinforcement therefore exerts two influences Overallvariability is elevated sometimes to the highest levelswhen it is reinforced but approach to (and possibly focuson) the reinforcers constrains or lowers variability Sim-ilar reinforcement-produced interference effects had beenreported previously for example when raccoons are re-inforced for dropping a ldquocoinrdquo into a slot the reinforce-ment results in the animalsrsquo holding and rubbing the coinrather than dropping it (Breland amp Breland 1961) Theoverall enhancement of variability was of much greatermagnitude in Cherot et al than the decrease with approachwas although both effects were statistically significant(The notes created by the composer under pressure ofdeadline will be less readily predicted than will thoseplayed by the performer practicing a well-known piece)This study highlights the importance of interactions be-tween operant effects of reinforcement and elicited effectsproduced by anticipation of reinforcement and the resultsmay be relevant to the debate concerning possible detri-mental effects of reinforcement on creative behaviors (seethe discussion below and Neuringer in press)

SummaryReinforcement controls behavioral variability as it

does other operant dimensions Levels of variability areprecisely governed by schedule contingencies and dis-

criminative stimuli Choices to vary or repeat match rel-ative frequencies of reinforcement Extinction increasesvariability slightly whereas the hierarchies of sequenceprobabilities are retained and operant and respondenteffects interact The evidence shows that behavioral vari-ability is an operant controlled by its consequences

GENERALITY AND PARAMETERS

Many different methods measures species and re-sponses have been used in the study of operant variabil-ity and this diversity supports the generality of the phe-nomenon As is the case for repeated operants howeverthere are parametric constraints This section briefly re-views the generality and some of the parameters studiedto date

GeneralityMethods Throughout this section ldquoresponserdquo will

signify a response unit including both an individual re-sponse and a multiresponse sequence Each of the meth-ods to be described increases or maintains variability byreinforcing it

Novel response procedures reinforce responses upontheir first occurrence ldquoeverrdquo (Pryor et al 1969) or upontheir first occurrence within a session (Goetz amp Baer1973) Radial-arm maze procedures also reinforce ini-tial (within a given session) entries into arms of a mazeand result in a distribution of choices across the availableoptions the sequence or pattern of choices often beingquite variable

Lag procedures reinforce a response if it has not oc-curred across a given number of previous trials (Page ampNeuringer 1985) For example under lag 5 the currentsequence is reinforced only if it has not occurred duringany of the previous five trials A variant is the recurrence-time procedure which specifies a minimum number of in-tervening trials before a previously emitted response canagain be reinforced Machado (1989) combined recurrencetime with a percentile-reinforcement contingency to gen-erate high levels of variability (see also Machado 1992)Percentile-reinforcement contingencies base the criterionfor reinforcement on the subjectrsquos own performance over aprevious set of responses (see Galbicka 1994) For exam-ple a sequence will be reinforced only if it has not oc-curred over the previous n trials the value of n continuallychanging so that 50 of the trials are reinforced

Least-frequent response procedures (Blough 1966Schoenfeld Harris amp Farmer 1966 Shimp 1967) rein-force responses that are currently least frequent across amoving window

Threshold procedures reinforce responses whose rela-tive frequencies fall below a given value Because thistype of procedure was used in the research describedabove and in many other more recent studies details willbe provided In Denney and Neuringer (1998) which willserve as an example trials consisted of four responses byrats across L and R levers A running tally was kept of the

OPERANT VARIABILITY 683

number of occurrences of each of the 16 possible se-quences LLLL LLLR LLRL and so on If the relativefrequency of the current sequencemdashthe number of its oc-currences divided by the sum of all 16 sequencesmdashwasbelow a specified threshold value in this case 05 the ratwas rewarded Recently emitted sequences contributedmore to the maintained tally than did those in the distantpast because after each reinforcement all 16 counterswere multiplied by a weighting coefficient in this case95 Therefore the influence of a particular trial decreasedexponentially with successive reinforcements Thresholdprocedures differ from Bloughrsquos (1966) least-frequentprocedure in that more than one sequence (or response)can be reinforced at any given timemdashnamely all those se-quences that have relative frequencies below the thresholdvalue The procedure differs from lag in permitting agiven sequence to be reinforced many times in successionif the associated relative frequency is sufficiently low

The frequency dependence procedure (Machado 19921993) is a variant of threshold in which the probability ofreinforcement is a continuous function of relative re-sponse frequency The more frequent a response is thelower the probability is that it will be reinforced

Statistical feedback procedures provide subjects withcomparisons between their performances and that of amodel Neuringer (1986) showed human subjects feedbackfrom 10 statistical tests thereby enabling them to comparetheir own distributions of responses to that of a randommodel Neuringer and Voss (1993) provided feedbackbased on a chaotic model The goal in each of these caseswas for the subjects to learn to approximate the model andthe results of these experiments will be discussed below

Differential reinforcement of switches versus repetitionsbalance a subjectrsquos tendency to engage in each of these pat-terns Rather than reinforcement for infrequent responsesor random-like distributions reinforcement is providedsometimes for repeating a response and other times forswitching from one response to another In the two-operandum case a switch is L followed by R or R by La repetition is L then L or R then R Bryant and Church(1974) found that stochastic-like responding was gener-ated by rats when switches were reinforced with a 75probability and repetitions with a 25 probability that isthe rats had a natural tendency to repeat at least in theenvironment of this study Machado (1997) reinforcedpigeons for eight pecks across two keys if the sequenceof pecks contained at least one (one group of birds) ortwo (another group) switches Although the birds couldhave satisfied these contingencies by repeating a singlesequencemdashin the ldquoone switchrdquo condition for examplerepetition of LRRRRRR would produce reinforcement on100 of the trialsmdashthey tended to vary their sequencesBarba and Hunziker (2002) showed however that vari-ability was higher when it was directly reinforced thanwhen reinforcement was based on proportions of switchesa finding consistent with that of Machado (1997)

The evidence from each of these methods supports thehypothesis that variability can be reinforced

Measures Variability must be interpreted with refer-ence to a particular measure The reason is that there canbe high variability (and low predictability) at one level ofanalysis but order (and certainty of prediction) at a dif-ferent level For example the sequence 1 1 2 2 3 3 44 5 5 1 1 2 2 has the same variance (a commonlyemployed measure of variability) as 2 3 2 1 5 1 4 41 1 3 2 5 2 but an analysis of conditional proba-bilities in which the unit contains pairs of instances in-dicates that the two sequences have different levels ofvariability In general high variability observed with onemeasure does not rule out the possibility of order ob-served with another and vice versa That is why evalua-tions of random-number generators often employ manydifferent measures and indeed there are hundreds of dif-ferent measures of the randomness of a sequence (Knuth1969) All behavioral studies make simplifying assump-tions and rely on one measure or at the most a few

U-value has been the most commonly employed mea-sure of operant variability (Evans amp Graham 1980Machado 1989 Page amp Neuringer 1985 Stokes 1995)U-value measures the distribution of relative frequen-cies or probabilities of a set of responses For a set of16 possible responses i = 1ndash16 U-value is given by

where RFi = the relative frequency (or probability) ofeach of the 16 responses U-values approach 10 whenrelative frequencies approach equality as would be ex-pected over the long run from a random generator andthey approach 00 when a single instance is repeated

Other measures include the percentage of trials or fre-quency at which variability contingencies are met (Page ampNeuringer 1985) percentages of alternations versus stays(Machado 1992) conditional probabilities of responses(Machado 1992) the number or frequency of novel re-sponses (Goetz amp Baer 1973 Schwartz 1982) the num-ber or frequency of different responses (Machado 1997Schwartz 1982) Markov analyses (Machado 1992) andstatistical tests of distributions of responses (Neuringer1986) All of the measures employed to date indicate thatvariability can be reinforced

Responses and species Generality has further beendemonstrated by the use of many different responses andspecies These include Siamese fighting fish (Betta splen-dens) swimming in quadrants of a tank (Roots 2000)young chicks pecking keys (Neuringer 2002) pigeonspecking keys (Page amp Neuringer 1985) budgerigarssinging songs (Manabe Staddon amp Cleaveland 1997)rats pressing levers (Van Hest van Haaren amp van de Poll1989) porpoises swimming and jumping (Pryor et al1969) rhesus monkeys pressing levers (Sydney Reisbickpersonal communication) children drawing and build-ing with blocks (Holman Goetz amp Baer 1977) andhuman adults and children pressing buttons (Saldana ampNeuringer 1998 Stokes amp Balsam 2001) There havebeen few attempts to compare species or responses but

- ( )[ ] ( )=aring RF RFi ii

log log 2 21

1616

684 NEURINGER

in one study lowest baseline variability was observedwhen rats pulled trapezes intermediate levels were ob-served when they pushed keys and highest levels wereobserved when they pressed levers (Morgan amp Neuringer1990) with variability increasing in all cases when it wasreinforced In another study operant variability wascompared across rats and people with similar effects ob-served in the two species (Neuringer Deiss amp Imig 2000)

ParametersWe turn now to some parameters studied to date Many

other parameters as yet unstudied are no doubt also im-portantmdashfor example motivation and types of reinforcers

Age Older people (see eg Van der Linden Beertenamp Pesenti 1998) and older rats (see eg Lamberty ampGower 1990) respond less variably than their youngercounterparts when variable responding is reinforced or re-quested but in most cases it is not clear to what extent thevariability is elicited as opposed to reinforced Neuringerand Huntley (1992) compared reinforced variability in2-month-old (young) and 10-month-old (mature) Long-Evans rats Under Yoke contingencies the young rats re-sponded significantly more variably (two levers four re-sponses per trial) than the mature rats which constitutesan elicited effect When reinforcement was contingenton variability (lag 4) variability increased significantlyin both groups thereby overcoming the initial differencecorrelated with age Thus explicitly reinforcing vari-ability in older animals caused them to respond no dif-ferently than younger animals did a finding potentiallyrelevant to real-world human aging However when Wells(1999) compared 8-month-old (mature) to 24-month-old(old) rats although the mature rats responded more vari-ably than the old rats did under Yoke contingencies thedifference was not statistically significant Additionalstudies are needed especially of whether reinforcingvariable behaviors in older populations can facilitatelearning and performance

Gender Male and female rats are equally sensitive toreinforcement of variability (Neuringer amp Huntley 1992van Hest et al 1989) However there was a tendency(not statistically significant) for males to respond over-all more variably than females did and because thesestudies did not control for differences in size and weightof subjects or for other aspects of motivation additionalstudies are necessary

Stage of training Stokes and Balsam (2001) reporteddifferences in variability as a function of whether variabil-ity was initially reinforced early or later in training (seealso Gottlieb 1992) Human subjects generated sequencesof 10 L and R responses per trial on a computer keyboardunder lag 0 and lag 25 contingencies Under lag 0 everytrial ended with reinforcement under lag 25 the currentsequence had to differ from each of the previous 25 Whenlag 25 was experienced at the very beginning of traininglevels of variability at the end of training (when lag 0 wasoperative) were lower than they were when lag 25 was ex-perienced after 50 trials of initial lag 0 These results were

interpreted to suggest that there is an ldquooptimal period forsetting variability levelsrdquo (Stokes amp Balsam 2001 p 181)but that general conclusion needs further testing and anumber of questions need to be answered One has to dowith the contribution of reinforcement frequency Underlag 0 every sequence leads to reinforcement but rein-forcement is intermittent under lag 25 Another has to dowith whether learning to vary differs from learning to repeatwith respect to optimal periods (see eg Bouton 1994)

Related to stage of training are effects of prior experi-ence Hunziker Lee Ferreira da Silva and Caramori(2002) showed that for human subjects if Var contingen-cies were experienced just prior to Yoke contingencieslevels of response variation during the Yoke phase werehigher than they would have been if the Var contingencieshad not been previously experienced The time delay be-tween Var and Yoke experiences mattered with a long in-terval (3 months) producing only a small effect Saldanaand Neuringer (1998) also found effects of prior Var train-ing on performances in Yoke (increased variability inYoke) but the opposite did not hold that is training underYoke had relatively little effect on Var Hunziker Caramorida Silva and Barba (1998) reported similar effects withrats Thus there may be an asymmetry Reinforcement forvarying has relatively strong influences but reinforce-ment without regard to variations does not adversely affectlearning to vary These different effects are not surprisinggiven that Yoke contingencies are permissivemdashreinforce-ment is given whether or not responses varymdashwhereascontingencies under Var can be quite demanding How-ever the results motivate a number of additional ques-tions One is whether effects of Var training differ fromthose observed when repetitions are reinforced Another iswhether direct reinforcement of variable behaviors canhelp animals or people to overcome the detrimental effectsof prior noncontingent experiences such as those that pro-duce learned helplessness (Seligman 1975)

Environmental constraints Stokes (1995) arguedalso that variability increases when behavior is con-strained The term ldquoconstraintrdquo is ambiguous because allreinforcement contingencies exert a form of constraintIn one study Stokes (1995) compared different strate-gies of shaping leverpresses Rats in one group were re-inforced initially for pressing the lever with only theright paw (right only) and eventually for any press (withright or left paw or with any other part of the body) Theorder of training was reversed for a second group ldquoanyrdquofirst and then right only Behavior sequences were morevariable in the right-only-first group leading Stokes(1995) to conclude that early constraints lead to highvariability (see also Stokes Mechner amp Balsam 1999)That is an important hypothesis with many implicationsbut controls are needed before the results can be attrib-uted to the contingencies For example ldquoright onlyrdquo andldquoanyrdquo differed in terms of reinforcement distributions aswell as in the precision of the reinforcement operationthe ldquoanyrdquo reinforcers were provided immediately and au-tomatically by the programming equipment whereas in

OPERANT VARIABILITY 685

the right-only condition a human observer had to decidewhether a response had occurred and then present the re-inforcer necessarily leading to longer and more variedlatencies and a less consistent contingency Furthermorethere are many cases in which increasing constraints suchas those due to a decrease in the number of responses pertrial (Page amp Neuringer 1985) or the requirement of ex-actly four responses on each of two keys (Schwartz 1980)results in decreased variability Thus the contributionsof response-definition constraints and environmentallimitations must be explored further

SummaryThe generality of operant variability is shown by the

robustness of the phenomenon across procedures mea-sures responses and species As with other operant phe-nomena parameters influence the effects

FUNCTIONS

All operants including variability are functional Op-erant responses produce reinforcing consequences Be-cause other behavioral competencies depend on it oper-ant variability is functional in a broader sense as wellSome of these functions will now be described

Operant ConditioningThe potential of a reinforcer to change behavior is lim-

ited by the range of available variations (Hull Langmanamp Glenn 2001 Skinner 1981) and thus whatever pro-duces those variations contributes to the conditioning ofnew operants Much work has been done to identifysources of variation (Balsam Deich Ohyama amp Stokes1998 Segal 1972 Staddon amp Simmelhag 1971) butNeuringer (1993) was first to show that the same rein-forcers that strengthen individual responses can also pro-duce the necessary variability and that these two functionsof reinforcement may work simultaneously Implied isthat when an operant response is being conditioned re-inforcement works not only to strengthen an individualresponse but at the same time to engender the variabilityfrom which the response emerges (That is variability isnot solely a baseline phenomenon elicited or noise)Concurrent reinforcement of variations and individualtarget sequences will be the topic of this section

Pigeons in one experiment and rats in another were re-warded for varying among 16 possible response sequences(four responses per trial L and R operanda lag contin-gencies Neuringer 1993) Concurrently one of the 16designated as the target sequence was reinforced when-ever it occurred Thus reinforcements were given bothfor sequence variations and for a single selected targeta reinforced-variation-and-selection (RVS) procedureWhen the target was ldquoeasyrdquo (eg LLLL) it emergedrapidly from the baseline variations Given that the tar-get was always reinforced reinforcers were obtained morefrequently when the subjects did not vary and only re-peated the target When the target was ldquodifficultrdquo (eg

LLLR) it was not learned As in the choice experimentdescribed above (Neuringer 1992) the relative valuesor difficulties of target versus variations governed theprobability that the target would be learned

However even difficult targets could be conditionedunder the RVS procedure as was shown by another ex-periment in the same series (Neuringer 1993) Sequencevariations were again reinforced concurrently with a dif-ficult target RRLR but now the frequency of reinforce-ments for varying was systematically decreased Satis-fying the variability contingency was intermittentlyreinforced according to a VI schedule and over sessionsreinforcement for varying became less and less frequentThis technique resulted in acquisition of even the mostdifficult target sequence whereas a control group rein-forced for the target alone (without the concurrent rein-forcement for variations) never learned

Reinforcers were obtained more frequently in the RVScondition than in the control condition (since the RVSsubjects were rewarded both for varying and for emittingthe target sequence) and the additional reinforcers mighthave generated greater motivation to respond (Killeen1981) Thus motivation not the direct reinforcement ofvariability might have been responsible for the RVS ef-fect To test this possibility Neuringer Deiss and Olson(2000) reinforced three groups of rats for emitting a par-ticularly difficult target sequence LLRLR The experi-mental group was reinforced concurrently for sequencevariability on the average of once per minute (VI 1 min)The target-only control group received no reinforce-ments other than for the LLRLR target A third groupwas yoked to the experimental animalsmdashreinforced onceper minute for any sequence variability not requiredmdashand as for the other two groups whenever the target oc-curred If the additional reinforcement was responsiblefor the facilitated learning under RVS then because theexperimental and yoked groups received equal reinforce-ments they should have learned equally well but thatwas not the result Only the experimental group learnedto emit the LLRLR target The yoked controls respondedat high rates throughout the experiment but they did notvary sufficiently to make contact with the target contin-gency and never learned The target-only control groupreceived insufficient reinforcement and their responsesextinguished Therefore the added reinforcers helped tomaintain motivation as suggested above but reinforce-ment of variations was necessary for learning of a diffi-cult sequence The RVS procedure maintains both highmotivation and high variation and the two together re-sult in the acquisition of difficult-to-learn responses

Grunow and Neuringer (2002 Experiment 2) studiedone possibly important parameter of the RVS proceduremdashnamely the level of variability When a difficult target re-sponse is being trained does it matter whether reinforcedvariability is low intermediate or high As describedabove each of four groups of rats was reinforced for vari-able response sequences the first group for high vari-ability the fourth group for low variability and the other

686 NEURINGER

two for intermediate levels In other ways the procedurewas similar to that of the Neuringer Deiss and Olson(2000) RVS contingencies just described with a singledifficult target sequence always reinforced The resultswere that the higher the level of reinforced baseline vari-ability the more quickly the target was learned Thusvariability levels indeed mattered and if the just-re-ported effect is found to be general it will be most help-ful to reinforce high variability when difficult-to-learnnew responses are being shaped Other parameters re-quiring study are class definitionsmdashwhich responses areincluded in the class of acceptable variationsmdashand num-ber of instances within the class

One additional effect should be noted When varia-tions among a set of responses are reinforced if one vari-ant is selected for extinction (no longer reinforced) thefrequency of that instance decreases Neuringer (1993)showed this for pigeons responding under a lag-variabilityschedule When one of the 16 sequences was no longerreinforced (with no other change to indicate its status) thefrequency of that sequence decreased Thus reinforce-ment of variations provides the baseline to select for (iestrengthen) and select against (ie weaken) individualinstances

These findings have potentially important implicationsFirst as has been indicated the same reinforcers thatstrengthen particular instances can also reinforce varia-tions That a binary eventmdashreinforce or not reinforcemdashsimultaneously affects variations and selected instancesindicates both the power of reinforcement and a concep-tual problem How reinforcers exert such diverse effectsis a question to which we will return

Second shaping procedures may implicitly reinforcebaseline variations An approximation to a goal responseis reinforced for a period of time but eventually rein-forcement is withheld until a response closer to the goalis obtained It is commonly thought that withholding re-inforcement or mini-extinction induces the necessaryvariations for shaping to proceed as has been indicatedabove An alternative suggested by the present studiesis that organisms learn both to vary (within a set) and to re-peat (selected instances) until one instance is always rein-forced That is the process of shaping might involve re-inforcement both of variations (generally done implicitly)and of targeted instances If so a useful contributor tothe shaping process might be explicit reinforcement ofvariations

Third reinforcement of variations could facilitate ac-quisition of other competencies such as cognitive andmotor skills Siegler (1996) has shown in an important se-ries of studies that childrenrsquos acquisition of complex cog-nitive skills is correlated with high variability Anotherexample is that when motor skills are practiced in a vari-able manner high quality of performance results (Ma-noel amp Connolly 1995 1997 Mechner 1992 Schmidtamp Lee 1999) However it is rare that variability in thesedomains is directly reinforced as part of the training oreducational process

Fourth the RVS procedure might be used in therapeu-tic settings to modify maladaptive and self-injurioushabits Therapists sometimes employ noncontingent re-inforcement procedures to weaken undesirable behaviors(Vollmer Iwata Zarcone Smith amp Mazaleski 1993)The reasoning is that many maladaptive behaviors aremaintained by unintended consequences and that non-contingent rewards may compete with those conse-quences thereby lowering the strength of the unwantedhabit For example when a staff personrsquos attention toself-injurious behavior helps to maintain it providing at-tention noncontingently may decrease the strength of theunwanted target behavior However the RVS proceduremight accomplish the same goal more effectively andwithout undesirable side effects (Vollmer RingdahlRoane amp Marcus 1997) Noncontingent rewards some-times result in unwanted ldquosuperstitiousrdquo behaviors andthat would be avoided under RVS Furthermore RVSwould help to provide the set of baseline behaviors nec-essary to establish desirable new responses

Finally RVS procedures may help to gain control overand eventually to weaken other undesirable target behav-iors The target can be incorporated within a set of moredesirable responsesmdasha new operant class established withthe target as one membermdashand variable instances rein-forced Once variable responding is successfully main-tained including the undesirable target response the lattercan be weakened by selectively withholding reinforcement

Summary The conditioning of operant responses de-pends on both baseline variations and selection of suc-cessive approximations to a goal response The same re-inforcers used to select the approximations can reinforcebaseline variations thereby defining the precise set ofappropriate variants and the desired levels of variationSuch concurrent reinforcement of variations and selec-tions may facilitate the acquisition of difficult-to-learnoperants and selective withholding of reinforcementmay weaken undesirable instances

Creativity and Problem SolvingBehaving in a novel unusual or nonrepeating manner is

part of creative activities (Campbell 1960) Novelty andvariation may arise through combinations of previouslylearned behaviors (Epstein 1996) transformations of thesebehaviors (Boden 1994) direct reinforcement of variabil-ity as described above or some combination thereof Ofcourse neither novelty nor variability alone suffices forcreative outputmdashthe throw of the dice or the toss of acoin are not creative nor are haphazard strokes of a paint-brush on a canvasmdashbut without variation creativity islow Whether the variability required for creativity canbe reinforced is relevant to whether creativity itself canbe reinforced The latter question has led to contentiousdebate with some arguing that reinforcement facilitatescreativity (Cameron amp Pierce 1994 Eisenberger ampCameron 1996 Stokes 2001 Winston amp Baker 1985)and others focusing on detrimental effects of reinforce-ment (Amabile 1983 Deci Koestner amp Ryan 1999)

OPERANT VARIABILITY 687

The debate goes well beyond the present discussion butsome of the research described above is relevant Firstreinforcement contingent on variability generates higherlevels of behavioral variability than does reinforcementcontingent simply on responding as has been shown bythe many studies described above in which Var contin-gencies were compared to Yoke contingencies Similarlyprecise reinforcement contingencies matter for creativity(Cameron amp Pierce 1994 Lepper amp Henderlong 2000)If any product rather than only creative ones is rein-forced creativity tends to fall (just as variability decreasesunder yoke conditions) Second approach to or antici-pation of reinforcement interferes with operant variabil-ity (Cherot et al 1996 McElroy amp Neuringer 1990Neuringer 1991) which again might be related to effectson creativity when an individual focuses on the rein-forcer Note however that although variability decreaseswith proximity to the reinforcer overall levels of vari-ability are much higher when variability is reinforced thanwhen it is not The same may be true for creativity aswell anticipation of reward may interfere with creativeoutput to some extent but at the same time high creativ-ity may depend on its reinforcement

Operant variability may be important for problemsolving as well Here too there is disagreement as towhether training is helpful or not Early studies by Maier(1933) indicated that it is and these found support inlater work (eg Lung amp Dominowski 1985) Somestudies showed that those individuals most likely toldquobreak the setrdquo are also most likely to solve a problem(Dover amp Shore 1991) However a number of studiesfound that practice had no influence (Duncan 1961)

Arnesen (2000) explored a related phenomenon witha rat model One group Var was reinforced for variableinteractions with novel objects and another group Yokewas reinforced independently of such interactions EachVar rat received training with 12 different objects oneper session For example in the presence of a soup canthe rat was reinforced the first time it entered the canbut thereafter a different response was required such asrolling it climbing on it and so forth This procedure issimilar to that of Goetz and Baerrsquos (1973) described aboveexcept that objects changed from session to session Ratsin the Yoke control group had identical experiences in anadjoining identical chamber that contained identical ob-jects but now rewards were not contingent on object ma-nipulation After this training the rats were tested in adifferent room with 30 never-before-experienced ob-jects scattered on the floor Hidden in each was a smallpiece of food When compared to their Yoke controls theVar animals were more active during this test exploredmore manipulated objects more and most importantlydiscovered significantly more food pellets Another con-trol group received no preliminary training with objectsand their performance in the test session did not differfrom that of the yoked controls Therefore when ratswere reinforced for variable interactions with objectsthe result was an increased tendency to explore manip-ulate and discover new sources of reinforcement It

would be of considerable interest to determine whethera similar effect occurs with children

Assessment and Treatment of PsychopathologiesAbnormal levels of behavioral variability characterize

some human psychopathologies as has been shown byclinical observations as well as by research with humansubjects and animal models Few studies have attemptedto reinforce variability directly in such problem popula-tions The main question is whether reinforcement canmodify abnormal levels of variability in the direction ofnormalcy and whether such changes might have associ-ated beneficial effects

Autism Stereotyped responding characterizes indi-viduals with autism (Hertzig amp Shapiro 1990) For ex-ample Baron-Cohen (1992) asked children with autismand others in a control group to hide a penny in one handso that the experimenter could not guess which hand heldthe penny Those with autism were more likely thanthose without to generate a simple predictable patternsuch as repeatedly switching back and forth from left toright hands Such repetitive responding can be maladap-tive For example Mullins and Rincover (1985) askedchildren with and without autism to pick one of five cardsFood was occasionally found in a cup behind the cardEach of the five cards was a discriminative stimulus fora different schedule of reinforcement including contin-uous reinforcement (CRF every choice reinforced) FR 2(fixed ratio 2 in which every second choice was rein-forced) FR 4 (every fourth choice reinforced) FR 7 andFR 11 Control participants sampled all five alternativesand quickly learned to choose the most frequently rein-forced CRF card Autistic children sampled only a lim-ited number of cards and therefore often preferred anonoptimal alternative Thus children with autism tendto respond less variably than normal children and thiscan result in failure to adapt

Miller and Neuringer (2000) asked whether direct re-inforcement could influence the variability of individu-als with autism Sequences of responses on two largebuttons were first reinforced probabilistically and inde-pendently of variability (Prob phase) after which rein-forcement was contingent upon satisfying a thresholdvariability contingency (Var phase) The autistic partic-ipantsrsquo variability was significantly lower than that ofmatched controls during both Prob and Var phases butwhen variability was reinforced its levels increased sig-nificantly in both groups This is an important findingbecause it suggests that direct reinforcement of variabil-ity may help to modify some of the behavioral stereo-types that are characteristic of autism Lee McComasand Jawor (in press) offer support for this hypothesisThree individuals with autism were reinforced for variedverbal responses to social questions the responses had tobe appropriate given the context Variability increased intwo of the three individuals In a related procedureBaker and Koegel (1999) reinforced children with autismfor varied social interaction in a game-playing contextAlthough the reinforcement was permission to engage in

688 NEURINGER

the stereotyped behaviors preferred by the child the re-sult was increased social interactions both during theperiod of the experimental manipulation and in 1- and2-month follow-ups These three studies taken togethersuggest that reinforcement of varied behaviors may facil-itate modification of nonfunctional ritualistic and stereo-typed behaviors

Depression Depression provides another example ofmaladaptive consequences of low variability Lapp Mar-inier and Pihl (1982) observed that depressed womenproduced fewer alternative solutions to hypotheticalproblems than did nondepressed women Channon andBaker (1996) found that attempts by moderately de-pressed college students to identify faults in a series ofinterconnected circuits were less varied and less success-ful that were those of nondepressed controls HorneEvans and Orne (1982) found that when depressed pa-tients were asked to generate a random sequence of num-bers their responses were less variable than were thoseof controls These experimental results are consistentwith clinical observations

Can depressed individuals be rewarded for increasingvariation Hopkinson and Neuringer (in press) dividedcollege students into mildly depressed and nondepressedgroups on the basis of the CES-D scale a paper-and-pencil self-evaluative index of depression In the first oftwo phases sequences of responses on a computer key-board were reinforced independently of variability (Prob)following which only highly variable sequences were rein-forced (Var) Under Prob depressed students respondedsignificantly less variably than did the controls in agree-ment with previous findings Under Var variability in-creased significantly in both groups Indeed by the endof the experiment depressed and nondepressed subjectswere responding equally variably (Instructions were alsoeffective in increasing variability in the depressed par-ticipants) Thus even in a simple computer-game envi-ronment mildly depressed subjects manifested lowervariability than controls did but direct reinforcementmodified their behavior to control levels Can reinforce-ment of variability be used in real-world conditions tohelp depressed individuals An affirmative answer comesfrom cognitive behavior modification procedures thatwork to increase alternative solutions to problems (Beck1976) A more general conclusion from these studies ofautistic and mildly depressed individuals is that direct-reinforcement-of-variability procedures may benefitthose who manifest abnormally low levels of variability

Attention Deficit Hyperactivity Disorder (ADHD)ADHD may provide an example of the opposite stateabnormally high levels of variability (Barkley 1990)Wultz Sagvolden Moser and Moser (1990) proposedthat the spontaneously hypertensive rat (SHR) may serveas a good animal model of ADHD the SHR shows manyof the behavioral characteristics of that disorder includ-ing hyperactivity poor learning ability poor ability totolerate delayed reinforcement and poor self-control

The next study to be described asked about operant vari-ability in SHRs

Mook Jeffrey and Neuringer (1993) compared vari-ability of SHRs and of a control strain Wystar Kyoto(WKY) under two conditions Prob and Var The mainresult was that SHRs tended to respond variably whetheror not the contingencies required it consistently with theresults from the human ADHD literature When vari-ability was reinforced the two strains responded at ap-proximately equally high levels of variation These re-sults were confirmed in two other studies (HunzikerSaldana amp Neuringer 1996 Mook amp Neuringer 1994)Thus control rats varied only when the contingencies re-quired variations whereas SHRs varied whether or notthe contingencies required it As might be expected fromthese results when reinforcement depended on repeti-tions the SHRs performed poorly (Mook et al 1993)

Another similarity between SHRs and ADHD humanswas seen when amphetamine a drug that has effects simi-lar to those of Ritalin often prescribed for ADHD was ad-ministered (Mook amp Neuringer 1994) When repetitionswere required for reinforcement amphetamine improvedperformances by the SHRs so that they were as accurate assaline-injected WKYs a finding again consistent withthose in the ADHD literature These studies support SHRsas a model of ADHD and suggest that additional attemptsshould be made to assess whether or not individuals withADHD are less able to change levels of variability thanare controls Although as indicated above there is someevidence in support of this hypothesis (Barkley 1990)when Saldana and Neuringer (1998) directly comparedhuman ADHD and control participantsrsquo operant variabil-ity within the context of a computer game no differenceswere observed Most of those with ADHD were main-tained on Ritalin and other drugs however a fact unfor-tunately not documented in the original article

Drug EffectsAlthough many drugs are known to influence behavioral

variability (Brugger 1997) rarely have attempts beenmade to analyze explain or identify the sources of theeffects As will be discussed below operant variabilitycan result from the memory-based process of remember-ing previous responses and then behaving differently orcan result from responding stochastically a process thatappears to be quite different McElroy and Neuringer(1990) attempted to distinguish the effects of alcohol onmemory from its effects on stochastic processes in astudy in which one group of rats was reinforced for vary-ing four-response sequences under a lag 5 contingency(Var) and another was reinforced for repeating a singlesequence LLRR (Rep) Ethanol injections of 075 and20 gkg affected Var and Rep performances differentlyWhereas the Rep animals were much less able to satisfythe LLRR contingency at both doses the Var rats re-sponded variably whether or not alcohol had been ad-ministered Similar results were obtained by Cohen et al

OPERANT VARIABILITY 689

(1990) with a multiple schedule in which each rat servedas its own control In the same animals and within thesame sessions alcohol degraded Rep performances butleft Var intact Thus alcohol appears to interfere with atask that requires working memory as presumably is thecase for the Rep task but not with one that reinforcesvariability The memory-interfering effect of alcoholmight account for why alcohol also degrades radial-armmaze performance (Devenport 1983)

Alcohol also affects choices to vary or to repeat For ex-ample Gorka (1993) reinforced rats concurrently for re-peating and varying and monitored percentage of choicesas a function of time following injection of ethanol Inthe control sessions when the rats were injected withsaline the probabilities of varying and repeating remainedapproximately constant across the 45-min session witha slight bias in favor of varying After injection with 15gkg of ethanol however there was a significant increasein choices to vary at the beginning of the session andthen a later decrease to below 50 That is alcohol ledinitially to a markedly increased preference for varyingbut later to a preference for repeating (Figure 5) This re-search should be replicated and extended especially be-cause the contingencies ldquopressuredrdquo the animals to re-main at approximately 50 preference but thesuggestion is that alcoholrsquos effects on response variabil-ity may depend in part on the duration of its influence Amore general implication is that drugs may affect re-peated and variable operants differently and that effectsdepend on time following administration

SummaryReinforced variability may contribute to many types

of learning including operant shaping and acquisition ofcognitive and complex motor skills Similarly creativityand problem solving depend on variable responding andin these areas as well explicit reinforcement of variationsmay be functional Abnormal levels of variability aresometimes caused by psychopathologies and drugs andoperant variability may be useful in these areas in twoways first as a way of assessing the variability that is pos-sible and second to modify variability toward more nor-mal levels Studies show that individuals with autism andmild depression vary less than do controls but that ex-plicit reinforcement of variability moves the levels towardsnormalcy Other studies show that drugs such as alcoholmay affect repeated and variable operants differently

EXPLANATIONS

Two questions must be answered in order to explainoperant variability First what are the sources of vari-ability that permit its reinforcement Second how is itmaintained Related questions including how instancesbecome members of a set from which variations emergeand whether operant variability is a generalizable com-petency will also be considered

Origination From Nonoperant SourcesSkinner (1966) reasoned that operant responses orig-

inate from nonoperant sources with reinforcement con-tacting a behavior shaping and strengthening it onlyafter the behavior had occurred for reasons other than re-inforcement To put it simply a behavior must occur be-fore it can be reinforced (Balsam amp Silver 1994) I willconsider three initial sources of variable behavior en-dogenous variability noncontingent environmental eventsand extinction These appear to parallel evolutionaryprocesses and they too will be identified

Endogenous variability All physical phenomenavary and behavior is no exception This fact can providethe opportunity for reinforcement to select particularlevels of variation Examples of spontaneous variabilityinclude spontaneous alternations in mazes (Dember ampRichman 1989) nondifferentially reinforced variationsin operant responses (Guthrie amp Horton 1946) and ex-ploration (Archer amp Birke 1983) Endogenous muta-tions provide a parallel in genetics Base-pair changes inDNA molecules occur spontaneously intron-bearinggenes mix and match segments to encode new proteinsjumping genes introduce new DNA sequences and atthe level of the chromosome random crossings and as-sortment occur as a natural part of reproductive meiosisThese sources of random genetic change occur indepen-dently of external influences

Figure 5 The relative frequency of variations by rats as a func-tion of 5-min blocks of time since the beginning of the sessionThe open circles represent performance after the rats were in-jected with saline indicating a slight tendency to vary more thanrepeat but one that is approximately constant throughout thesession The filled circles represent performance after injectionwith 15 gkg ethanol indicating a much higher likelihood ofvarying soon after the alcohol is injected followed by a decreasein variations to below 50 and then a return to baseline at theend of the session Group averages are shown Data are fromGorka (1993)

690 NEURINGER

Noncontingent events Noncontingent events some-times referred to as chance accidents or luck also affectbehavior One happens upon a particular passage in a bookthat leads to a new thought one happens to sit next to aninteresting woman or man on an airplane which leads tomarriage (Bandura 1982) In science as well as in every-day life accidents or serendipity are an important part ofthe process of discovery (Beveridge 1957) In each of thecases presented above an accidental event leads to new be-haviors that are then strengthened by consequences Thesame is potentially true for variability itself Particular lev-els of variability might occur for adventitious reasons butthen become functionally related to consequences

That organisms are particularly sensitive to such unan-ticipated noncontingent events is shown by two basicprinciples of learning habituation and Pavlovian condi-tioning Organisms attend preferentially to unexpectedevents and stop attending or habituate to repeated onesThe conditioning of new Pavlovian responses dependsaccording to influential theories upon unexpected rein-forcing events (eg Rescorla amp Wagner 1972)

Noncontingent events such as X-rays and other pro-cesses that can act as mutagens play an important role ingenetic variation as well The particular genes affectedand the type of changes are random there is no neces-sary relationship or contingency between the externalevent and the affected gene However sensitivity to mu-tations is not random but has been selected presumablybecause it is functional Thus in both behavior and ge-netics sensitivity to noncontingent events is conserved

Extinction Extinction increases behavioral variabilityat least to some extent as was described above Distance inspace or time from the reinforcer does the same (Cherotet al 1996) A possible parallel at the genetic level in-volves high levels of deprivation at least in simple lifeforms For example when E coli bacteria are deprivedof nutrients unusually high mutation rates are observedThe exact mechanism by which such hypermutations areinduced is not known but they increase the likelihoodthat a population can survive such environmental insults(Moxon 1997) That is deprivation-induced variability isfunctional at the genetic level (The behavioral evidenceregarding long-term deprivation is less clear some stud-ies showing increased variability and others increasedstereotypy eg Carlton 1962 Senkowski Vogel amp Po-zulp 1978)

Endogenous variations external noncontingent eventsand withholding of reinforcement are three sources forthe variability that precedes control by reinforcementBut once control by reinforcement is established a dif-ferent question emerges What behavioral processes areinvolved when variability is maintained as an operantThe sources just discussed generate uncontrolled non-specific and often short-lived variability and cannot ex-plain the precisely controlled operant type

Operant Variability ProcessesThree processes are hypothesized to underlie operant

variability The first involves the use of random environ-

mental events such as the toss of a coin The second in-volves memory for responses such as when an animallearns not to return to previously visited locations Thethird will be referred to as an endogenous stochasticgenerator

Random environmental events Throughout historymany cultures have used random events to influencechoices and decisions such as throws of dice and randomselections of sticks cards bones or organs Today coinflips are used by a referee at the beginning of a game to de-cide which team kicks the ball a computerrsquos random-number generator is employed by scientists to avoid bi-ases in assigning subjects to experimental groups andaleatoric events are sometimes employed in modern artmusic and literature The Dice Man by Rhinehart (1998)provides a fictional example in which the protagonistbored with life writes a number of possible actions onslips of paper and then periodically selects one blindlyand behaves accordingly These examples show that randomexternal events may be used to avoid biases engender un-likely responses and break out of behavioral niches

Memory-based variability Many methods used tostudy operant variability differentially reinforce nonrepe-titions at least over the recent past Therefore one strat-egy that subjects can employ is to cycle through few re-sponses or sequences especially when the variabilitycontingencies are relatively permissive For exampleunder a lag 2 contingency cycling repeatedly throughthree sequences will result in 100 reinforcement Ani-mals and people sometimes do this apparently basingtheir current response on memory for or discriminativecontrol by the just emitted responses One such instancewas shown by Schoenfeld et al (1966) who rewardedrats for IRTs that fell into one of 10 interval classes onlyif the current IRT differed from the interval class of theprevious response The rats came to alternate betweenlong and short IRTs thus behaving systematically to meetthis lenient variability contingency Machado (1993)studied an analogous effect having pigeons peck L and Rkeys under a frequency-dependent variability contin-gency related to Shimprsquos (1967) procedure describedabove When the trial unit consisted of two responses thebirds developed memory-based repetitive sequencessuch as RRLLRRLL and thereby satisfied the variabilitycontingency When the trial unit was three responses inlength the birds did not develop the optimal fixed pat-tern of RRRLRLLLmdashwhich was too complexmdashbut in-stead developed ldquorandom-like behaviorrdquo (Machado1993 p 103) Thus a memory-based strategy was em-ployed when it was possible but when the memory loadbecame too high stochastic responding emerged A sim-ilar pattern was seen when variability of budgerigarsongs was reinforced (Manabe et al 1997) Under lag 1contingencies the birds sang mainly two songs underlag 2 they increased to three songs Only when the lagwas increased to 3 did song diversity increase apprecia-bly in one of the birds In radial-arm mazes as well whencontextual cues are available memory again appears toplay an important role leading to much lower probabil-

OPERANT VARIABILITY 691

ities of repetition than would be expected from a sto-chastic generator (McElroy amp Neuringer 1990)

Note that under lag schedules memory-based strate-gies result in higher reinforcement rates than respondingstochastically does The reason is that a stochastic gen-erator repeats by chance That animals and people oftendo not employ memory-based strategies probably indi-cates the difficulty of remembering long sequences andpossibly other advantages of unpredictable responding

If a memory strategy is the only way to satisfy vari-ability contingencies then it can be used even when thememory requirements are demanding as was shown byNeuringer and Voss (1993) Human subjects were trainedto generate on a computer keyboard a chaotic-like se-quence of responses that matched iterations of the logistic-difference equation

where Rn refers to the nth iteration in a series with R be-tween 00 and 10 and t is a constant between 10 and40 When the t parameter approaches 40 iterations ofthis equation result in very ldquonoisyrdquo sequences As withother chaotic processes however there is an orderlinessunderlying the surface noise In this case if responsesare autocorrelated with Rn drawn as a function of Rn21the resulting function is a parabola In the Neuringer andVoss (1993) study following each response college stu-dents were shown how close their predicted values wereto that from iterations of the logistic-difference model Inone experiment the students made digital predictions inanother they placed a pointer along a horizontal lineand this analogue response was then translated into dig-ital values for computation With training the studentsbecame increasingly adept at responding in chaotic-likefashionmdashthe studentsrsquo predictions matched closely theiterations of the logistic functionmdashand their autocorrela-tions became parabolic The left column in Figure 6 showsperformances by 4 subjects (rows) early in training thatis not well approximated by the logistic-differenceparabola and the right column shows performances at theend of training in which r 2 values were 72 99 98 andapproaching 10 How was this accomplished Sinceeach iteration in the logistic-difference sequence is basedon the prior outputmdashthat is the sequence is generated bya deterministic although nonlinear processmdashit was hy-pothesized that the human subjectsrsquo chaotic-like responseswere also based on memory for prior responses To put itsimply the subjects may have memorized a long series ofpairs in the form ldquoIf the previous response was value Athen the current response must be value Brdquo as Metzger(1994) Ward and West (1994) and Ward (2002) have sug-gested

To test whether subjects in fact relied on memory forprior responses Neuringer and Voss (2002) systemati-cally slowed responding by interposing pausesmdashthat isby requiring an IRT As IRT increased the difference be-tween the subjectsrsquo sequences and the modelrsquos chaotic

Rn n nt R R= -( )- -1 11

Figure 6 Autocorrelations (responses on trial n on the y-axis asa function of n21 on the x-axis) for 4 human subjects at the be-ginning (left column) and end (right column) of training inNeuringer and Voss (1993) Equations for the best-fitting curveand amount of variance accounted for (r2) are shown at the topof each panel To the extent that the circles (responses) approxi-mate the equation y = 40x 2 40x 2 the subjects were respondingin a way that matched the logistic-difference chaotic functionFrom ldquoApproximating Chaotic Behaviorrdquo by A Neuringer andC Voss 1993 Psychological Science 4 p 115 Copyright 1993 bythe American Psychological Society Reprinted with permission

692 NEURINGER

output increased and the underlying parabolic structurewas broken That is when pauses increased subjects wereless and less able to approximate the modelrsquos chaotic dis-tribution (see Figure 7) The important point here is thathighly variable responses which in this case are chaoticin nature can be based on memorial processes

Endogenous stochastic generator The evidence tobe reviewed next indicates that animals and people canrespond stochastically and that this competency is basedneither on chance environmental events nor on memoryfor prior events or responses It is impossible to provethat an endogenous stochastic process underlies operantvariability that would be analogous to proving the nullhypothesis However much evidence is consistent withthe stochastic hypothesis Included are the facts thatmemory limitations do not permit adequate respondingunder many variability contingencies that response dis-tributions match those expected from a stochastic gener-ator that changes in parameter values affect operantvariability as they would a stochastic generator and thatinterference with memory processes leaves stochasti-cally based operant variability unaffected As a wholethe evidence is consistent with an endogenous stochas-tic source of operant variability

Memory limitations One type of evidence is nega-tive in character and is based on the presumed inability ofmemory for prior responses or sequences to satisfy sometypes of variability contingencies Two examples weregiven above (Machado 1993 Manabe et al 1997) andanother was provided by a lag 50 contingency in Pageand Neuringer (1985) In the latter case the sequence in

the current trial (eight responses across L and R keys) wasrequired to differ from those in each of the previous 50 tri-als including at the beginning of a session the last 50trials from the previous session Pigeons responded wellunder these contingencies and in a way similar to that ofa stochastic model It is unlikely that pigeons can remem-ber each of their previous 50 eight-response sequences

Response slowing As was shown above for chaotic-like responding performances are often degraded by de-lays imposed between a stimulus and a response pre-sumably because of limitations in working or short-termmemory (Berryman Cumming amp Nevin 1963 Williams1971) Neuringer (1991) relied on this effect to test whethermemorial processes could explain the generation of vari-able sequences under lag schedules Two groups of ratswere reinforced for L and R leverpresses producing fourresponses per trial One group (Var) was trained under alag 5 contingency and the other (Rep) was trained to asingle sequence of responses LLRR It was assumedthat accurate Rep performance depended on workingmemory After performances had stabilized IRTs weresystematically varied by the imposition of timeouts of05 to 20 sec between consecutive responses in the se-quence During these forced pauses the chamber wasdark and responses were not counted toward completionof the sequence If the animals had used the previous re-sponse(s) as a discriminative cue for the current responsethen performances should have been degraded by theseinterpolated pauses

The results were that for the Rep group as the re-quired pauses increased from 0 to 6 sec percentages ofcorrect sequences were relatively unaffected but at pausesgreater than 6 sec probability of a correct LLRR se-quence fell sharply This pattern of results suggests in-volvement of working memory of prior responses withthe rats able to remember responses within a 6-sec win-dow The Var grouprsquos results were quite different Aspauses increased between 0 and 6 sec the rats were moreand more likely to satisfy the variability contingenciesand with pauses greater than 6 sec percent correct re-mained at a high and constant level Thus the Var grouprsquosresults were opposite to those of the Rep group and op-posite to those predicted from a memory-based strategySimilar results showing that operant variability increasesor is maintained as responding is slowed have been re-ported by Morris (1987) with pigeons and by Baddeley(1966) with random response generation by human sub-jects The results reported above concerning alcoholrsquos ef-fects on variability and repetition were also consistentwith those of Neuringer (1991)

Memory for prior responses may not have controlled theVar grouprsquos performance but why did variability increaseEach of three answers is consistent with a stochastic-generator interpretation Weiss (1964 1965) hypothe-sized that random responding requires that current re-sponses be independent of previous ones memory for orcontrol by prior responses interferes with such indepen-

Figure 7 Approximation to a chaotic sequence (given by delta)as a function of the imposed interresponse times Shown are theaverages of 5 subjects

OPERANT VARIABILITY 693

dence and therefore pauses would be expected to facil-itate variability A second possibility is that at shortIRTs animals tend to repeat responses on the sameoperandum Blough (1966) found this in pigeons and ex-cluded such double pecks from his analyses because theyappeared not to be under the control of reinforcement con-tingencies Morris (1987) also found a tendency for birdsto repeat when no interresponse timeouts were imposedA third hypothesis is that there were two contributors tothe observed variability One was a stochastic processcontrolled by reinforcement and the other elicited vari-ability generated by interposition of pauses The high vari-ability elicited as an effect of slowed responding is ageneral phenomenon supported in many other cases Ac-cording to this interpretation operant variability in the Vargroup was governed by a stochastic-based process operantrepetition in the Rep group was governed by a memory-based process and pauses elicited variability under bothcontingencies Thus Rep performance was interferedwith whereas Var was facilitated Each of these hypothe-ses is consistent with the conclusion that memory for (ordiscriminative control by) prior responses does not con-tribute to and possibly interferes with variable respond-ing when an organism is reinforced for variability andthis in turn is consistent with the stochastic-generatorhypothesis

Stochastic response distributions Other evidence iscorrelational Response distributions are similar to thosefrom a stochastic generator The data from two experi-ments already discussed are shown in Figure 1 In thesecases reinforcement of least frequent IRTs (Blough 1966)and of nonrepetitive choice sequences (Machado 1989)resulted in stochastic-like responding

Neuringer (1986) reported another striking case Highschool and college students generated sequences of 1sand 2s on a computer keyboard with each set of 100 re-sponses constituting one trial A baseline phase lastedfor 60 such trials for a total of 6000 responses duringwhich the students were asked to behave as randomly aspossible as if they were reporting the outcome of a se-ries of coin tosses As in the many previous studies ofthis sort the subjectsrsquo responses differed significantlyfrom those of a stochastic model (Brugger 1997 Wage-naar 1972) During the training phase the subjects re-ceived feedback which enabled them to compare their per-formances with that of a stochastic model first accordingto one statistical test and then according to another untilfollowing each trial feedback was provided from 10 dif-ferent statistics The results were that all of the subjectslearned to approximate the stochastic model accordingto the 10 statistics as well as some (but not all) other sta-tistical tests That is the distributions of the statisticswhich clearly did not match a stochastic model at the be-ginning of training approximated the stochastic modelat the end of it (see Figure 8)

The previous failures of other researchers to obtain ev-idence for random responding had led some to concludethat humans did not have the capacity to behave ran-

domly What would account for the difference betweenthe present positive finding and the many other negativeones Neuringerrsquos (1986) study was the first in which sto-chastic responding was explicitly reinforced in this casewith feedback from 10 statistical tests An analogy mighthelp to explain the difference in procedures and resultsImagine an experimenter walking around a city and ask-ing people if they know what a violin sounds like (analo-gous to asking people whether they know what a series ofcoin tosses or dice throws is like) Most would answeryes If the same individuals were then asked to play theviolin very few if any would succeed The experimentermight conclude after many replications with differentsubjects in different cities that people did not have thecapacity to play the violin The point of course is thatplaying the violin is a skill requiring training and prac-tice and the same may be true for responding randomlyIt is easy to vary but practice and reinforcement may benecessary to approximate a random distribution

Response number Page and Neuringer (1985) testedthe stochastic-generator hypothesis in another way byvarying number of responses per trial with pigeons work-ing under a lag 3 contingency The reasoning was that ifprevious responses served as a discriminative cue forwhat not to domdasha memory-based strategymdashthen perfor-mance should be degraded when the number of re-sponses was increased Eight responses per trial requiresubjects to remember more than four responses The sto-chastic hypothesis predicts the opposite outcome as isdemonstrated by the following example If each trialwere two responses in length with responses determinedby the toss of a coin then the probability that a given trialwould repeat the previous one would be 25 (There arefour possible sequences in the first trial RR RL LRand LL Thus the second trial has a 1-in-4 chance ofmatching the first) If a trial consists of four coin tossesthe probability of a repetition by chance is 0625 or 1 in16 Thus if subjects used a stochastic process to gener-ate Ls and Rs performances should improve with in-creasing responses per trial Results were precisely thosepredicted by the stochastic hypothesis and inconsistentwith a memory strategy Probability of meeting the con-tingencies and therefore probability of reinforcementincreased as responses per trial increased In otherwords eight-response trials were easier for pigeons thanfour-response trials

Response interference Neuringer and Voss (2002)attempted to distinguish between memorial and stochas-tic processes in yet another way They hypothesized thatwhereas chaotic responding was based on memory forprior responses (as was discussed above) responding inNeuringerrsquos (1986) study with feedback from 10 statisti-cal tests indicated involvement of an endogenous sto-chastic generator To show that human subjects could en-gage in both processes at will three college studentswere trained to generate chaotic sequences in one compo-nent of a multiple schedule as in Neuringer and Voss(1993) and to generate stochastic sequences in a sepa-

694 NEURINGER

rate component as in Neuringer (1986) The wordCHAOTICS was on the screen in the first component theword STOCHASTICS was on during the other and thesewords alternated throughout the session After the sub-jects succeeded at each of the component tasks test ses-sions were introduced in which memory for prior re-sponses was subject to interference in both the chaoticsand stochastics components as follows During the

chaotics component four separate chaotic sequenceswere cycled one after the other each cued by differentscreen colors The subjects attempted to approximate achaotic sequence in one color (eg light green) follow-ing which they would have to match a second chaotic se-quence (lavender screen color) and so on the fourchaotic contingencies following one another across thesession Although all four chaotic sequences used the

Figure 8 Performance by one subject at the end of training to respond stochastically in Neuringer(1986) Shown are comparisons on 6 different statistics between the subject (solid lines and Xs) andthe random model (dashed lines and closed circles) At the end of training the subjectrsquos perfor-mance did not differ statistically from that of the random model on these and four additional sta-tistics From ldquoCan people behave lsquorandomlyrsquo The role of feedbackrdquo by A Neuringer 1986 Jour-nal of Experimental Psychology General 115 p 71 Copyright 1986 by the American PsychologicalAssociation Reprinted with permission

OPERANT VARIABILITY 695

same logistic-difference equation that a model did theywere out of phase and independent If chaotic respond-ing is memory based then the four independent se-quences would be expected to interfere with one anotherthat is approximations to the four chaotic models wouldbe lower than when only a single chaotic output was re-quired as in Neuringer and Voss (1993) That was ex-actly the finding Performances by the three subjectswere signif icantly degraded during the four-segmentphase To put it simply chaotic responses interfered withone another

A different pattern of results was seen in the stochasticsportion of the experiment The same interference proce-dure was used with four independent stochastic sequencesrequired in the four different color conditions As inNeuringer (1986) subjects were required to satisfy eightdifferent statistical tests of randomness and to do this in-dependently in each of the four phases (If responses in anytwo or more of the four stochastic phases became corre-lated above a minimal level responses were considered tobe incorrect thereby prohibiting repetition of a single se-quence in two or more of the phases) For one subjectseven of eight statistics were closer to those of a randommodel during this interference phase than they were at theend of the original training with only a single sequenceand for a second subject all eight statistics were closerThe third subject showed closer-to-random approximationunder the four-segment condition on only two of the eightstatistics but most of her changes were also in the directionof a random generator For example if a random generatoremits equal numbers of ldquouprdquo and ldquodownrdquo sequences andthis subject in the baseline phase had emitted only 47ldquouprdquo sequences then during the four-segment tests shecame to emit ldquouprdquo 55 of the time She changed in the di-rection of the random model but overshot the mark Thuswhereas interference degraded chaotic performance sig-nificantly in all three subjects it improved stochastic per-formances in at least two of them These results are con-sistent with the hypothesis that a memory-based processcontrols chaotic variability and that a stochastic processnot dependent on memory controls stochastic variability

Summary of Tripartite TheoryThe following three processes constitute a tripartite

theory of operant variability attention to random events inthe environment (coin toss) memorial processes (chaosradial maze) and stochastic processes (response se-quences and random-number generation with feedback)Random environmental events assure unpredictabilitymdashfor example in making unbiased selections Remember-ing what was done previously will enable one to avoidrepetitions to vary the students called upon in class tovary the locations when looking for a hidden object orto try different approaches when attempting to solve aproblem External aids such as notes to oneself or sci-entific records help to enhance memory capacities

An endogenous stochastic generator provides the thirdand most controversial source of behavioral variability

When an opponent is attempting to predict your behav-ior in a game or at war it may be functional to behavestochastically Similarly operant conditioning of new re-sponses may depend on stochastic variations It is im-possible to prove stochasticity some presently unknowndeterministic system may ultimately explain the resultsHowever the evidence is consistent with the stochastichypothesis and inconsistent with the alternatives For ex-ample memory is not sufficiently powerful to explain alloperant variability memory-interference proceduresleave stochastic responding intact and behaviors corre-late well with those of stochastic models

Both genetics and reinforcement play important rolesin all three processes Reinforcement is always involvedin that learning provides information about when whatwhere and how much to vary Genetic influences areshown in that not all responses are possible and if theyare possible they are not all equiprobable (eg as inNeuringer et al 2001) Stochastic responding may be bi-ased due to experiences and knowledge that is biasesmay be established by reinforcement contingencies or bythe interface between the organismrsquos propensities (or be-havior systems see Timberlake amp Lucas 1989) and theenvironment In the case of random-number generationby human subjects biases may be due to misconceptionsconcerning what ldquorandomrdquo means However some bi-ases can be overcome through feedback and instructions(Lopes amp Oden 1987 Neuringer 1986)

The importance of variations for evolutionary changeis supported by the many sources of genetic variabilityand by the fact that genetic variability is conserved Asimilar argument holds for operant variability That thereare at least three independent sources of operant varia-tions as well as other elicitors such as environmentalnoise and response extinction supports its importancefor normal adaptive functioning

Structure of VariabilityEvery operant ldquoresponserdquo is actually an instance of a

class with each member of the class exerting the sameeffect on the environment (Skinner 1959) When a rat isreinforced for pressing a lever for example the class isdefined by closure of a microswitch and any way of ac-complishing the closure whether with the left paw theright paw or the mouth is as effective in producing re-inforcement as any other If the contingencies of rein-forcement so specify however the response class can beredefined and circumscribed which is another charac-teristic of the operant For example if only right-pawpresses are acceptable members of the reinforced classresponding will tend to occur only with the right paw(Stokes 1995) Of course the capabilities of the organismplay a role for example including ldquotail pressesrdquo in the op-erant class does not mean that the lever would be or couldbe pressed by the ratrsquos tail The structure of the operantclass therefore emerges from the current reinforcementcontingencies and the organismrsquos capability derivedfrom genetics development and prior experiences

696 NEURINGER

Operant variability has structure as well and as for alloperants that structure is due partly to the organism it-self and partly to the reinforcement contingencies Theresearch described throughout this article has providedexamples of the structure of variability generally in theform of sets of four or eight responses across L and Roperanda But the structure of variations can be speci-f ied more precisely For example Mook and her col-leagues (Mook et al 1993 Mook amp Neuringer 1994)reinforced rats for varying four response sequences acrossL and R levers under a lag 1 contingency Unlike in pre-vious studies however the acceptable class was furtherconstrained so that only sequences beginning with two Rresponses (RRLL RRLR RRRL and RRRR) were rein-forced The rats learned to vary their sequences withinthe imposed limitations that is they generally begantheir sequences with two R responses and varied the se-quences within those limitations Similar results wereobtained in the context of a radial-arm maze in whichonly a subset of arms was baited

The precision with which reinforcement contingen-cies can specify class was highlighted in research per-formed by Burns (1997) and Ross and Neuringer (2002)In the latter case college students drew rectangles on acomputer screen with the computerrsquos mouse Each rec-tangle constituted one trial (the screen cleared at the endof each trial) with reinforcement (points) depending onsimultaneous variations along three dimensions of therectangle location on the screen area of the rectangleand shape with the latter defined as the ratio of the rec-tanglersquos height to its width Although the subjects weregiven no instruction other than to ldquogain pointsrdquo theylearned to vary along all three dimensions with levels ofvariability significantly higher than those of yoked controlsubjects who were reinforced independently of variations

A second part of the Ross and Neuringer (2002) studydemonstrated even greater precision of control To gainpoints three new groups of subjects were required to re-peat along one of the dimensions while simultaneouslyvarying along the other two One group was required torepeatedly draw rectangles of approximately the samesize while simultaneously varying location and shapeanother group had to repeat the shape while varying areaand location and the third to repeat location while vary-ing size and shape Verbal instructions were minimal asthey were in the first part of the study Subjects learnedto satisfy these contingencies as well varying two at-tributes while simultaneously repeating one and signif-icant differences were obtained rapidly within 300 trials

These demonstrations raise questions as to how binaryfeedbackmdashreinforcement presented or notmdashcan simul-taneously control independent behavioral dimensionsThe issue is similar to that discussed above concerningsimultaneous reinforcement of variations and selectionsIt is similar also to those issues faced by researchers at-tempting to understand how the nervous system strength-ens particular synapses when many different neurons firesimultaneously and to ldquoallocation of creditrdquo in artificial

learning models For present purposes suffice it to notethe exquisite precision with which reinforcement controlsvariability and repetition

GeneralizationDo animals and people learn to ldquovaryrdquo implying a

generalizable skill or do they learn a specific distributionof specific responses across particular operanda Thequestion of generalization has been asked of many otherbehavioral competencies such as imitation When chil-dren learn to imitate do they learn stimulusndashresponseassociations (if mother moves her right hand the childlearns to move his right hand) or do they learn moregenerally to ldquoimitaterdquo Evidence indicates that reinforce-ment of imitative instances leads to generalized imitation(see eg Baer amp Sherman 1964 Bandura 1986) A sim-ilar question has been raised when pigeons rats or chil-dren learn to choose a stimulus that matches a previouslyshown sample For example if a pigeon is shown a reddisk as a sample it must then choose red when given achoice between that and blue Does the bird learn to matchIn this case the answer appears to depend on the train-ing procedures If trained with few stimuli such as redgreen and blue then the pigeon learns simple associa-tions (if red sample choose red if blue sample chooseblue) but if hundreds of different stimuli (pictures) areemployed during training then generalized ldquomatchingrdquois learned (Wright 1997) In each of these examples thelearning of a general competency or strategy was shownby transfer-of-training tests What was learned with oneset of stimuli or in one situation transferred to new stim-uli and situations We turn now to the question of whetherlearning to vary transfers

After the subjects in Maltzmanrsquos (1960) experimentdescribed above learned to generate remote verbal as-sociates they were asked to devise unusual uses of com-mon objects In comparison with those in the controlgroup these subjects generated significantly more un-usual uses indicating transfer of training to a differenttask However transfer was not observed in the oppositedirection from unusual uses to verbal associates

Holman et al (1977) using a procedure similar to thatof the Goetz and Baer (1973) research described aboverewarded preschool children verbally for novel drawingsand found increased novelty Unrewarded block construc-tions were also monitored and weak evidence was ob-tained for transfer of novel responding from rewardeddrawing to the unrewarded constructions In a second ex-periment preschoolers engaged in four independent taskspainting pen drawing wood block construction and Legoconstruction When the children were reinforced for vary-ing their pen drawings the novelty of paintings increasedalong with the pen drawings indicating transfer from onetask to the other However transfer to the topographicallydifferent block- or Lego-building tasks was not observed

Eisenberger and Armeli (1997) gave monetary rewardsto one group of fifth- and sixth-grade students the vari-ability group for describing unusual uses of common ob-

OPERANT VARIABILITY 697

jects A second group the repeat group received thesame rewards for describing common uses of the sameobjects In a later test all of the children were asked todraw pictures incorporating a circle and the group that hadpreviously been reinforced for varied responses drewmore unusual (or lower frequency) pictures than did therepeat group Thus there was transfer of novelty trainingfrom one domain to another with additional studies in theseries showing similar effects (Eisenberger Armeli ampPretz 1998 Eisenberger Haskins amp Gambleton 1999Eisenberger amp Selbst 1994) However Eisenberger et al(1999) failed to replicate the transfer-of-training effectThe variability and repeat groups responded similarly dur-ing the test phase In this study separate experimenterstrained and tested the subjects thereby avoiding possibleexperimenter-expectancy effects whereas the same ex-perimenter did both training and testing in Eisenbergerrsquosother studies Across the series of experiments there weremany other cases in which transfer was not observed andmore research is necessary to explain why

In Lee et alrsquos (in press) study with autistic individualsdescribed above after variable answers were trained toquestions verbal variability transferred to new experi-menters and new contexts

Arnesenrsquos (2000) results described above are consis-tent with transfer of training in an animal model Rats inthe Var group were trained to interact variably with oneset of objects When later tested in a new situation withnew objects the Var animals showed (in comparisonwith controls) higher levels of exploration manipulationof the novel objects and discovery of hidden reinforcers

Much of the evidence reviewed in this section there-fore indicates transfer of training and supports a generalldquovaryrdquo competency However there remain unresolvedquestions the most important of which is the extent towhich transfer occurs Note also a methodological pointMany of the studies tested whether transfer occurs whenthe contingencies are neutral with respect to the to-be-transferred behavior Holman et al (1977) is a case inpoint Variations were reinforced in one domain draw-ing and the question was whether variations wouldoccur in another domainmdashblock constructionmdashin whichreinforcement did not depend on varying A negativef indingmdashin this case absence of transfermdashindicatesnothing about the ability to generalize but only aboutthe tendency to do so given the absence of a reason (orcontingency) A better test would be to compare experi-mental and control performances when variability in thenew task is in fact functional as in Arnesen (2000)

SummaryBefore variability can be reinforced it must occur for

reasons other than reinforcement Some of the reasonsinclude variability inherent in all behavioral phenomenaldquoaccidentalrdquo environmental events that are unrelated toparticular behaviors and withholding of reinforcementTransfer from other situations may serve a similar functionFor example if an organism can discriminate shaping-

of-new-response periods from others then it may learngenerally to ldquovaryrdquo during shaping episodes After rein-forcement makes initial contact with response variationsthere are at least three behavioral strategies that can beemployed in order to maintain functionally high levelsof variability respond based on external random eventssuch as the toss of a coin avoid repetitions by remem-bering what responses had previously been made and re-spond stochastically Evidence for an endogenous sto-chastic process was seen in the similarity of responsedistributions to those generated by stochastic sourcesand by findings that interfering with memory for priorresponses either improves or leaves unaffected stochas-tic behaviors Stochastic responding as well as othertypes of variability-generating strategies depend uponcontingencies to define the required level of variabilityand the set from which variations emerge

IMPLICATIONS

One implication of reinforced variability is that oper-ant responses may differ in the extent to which their vari-ability is affected by reinforcement contingenciesmdashthatis operants may be ordered along a continuum of sensi-tivity of variability to reinforcement Another is that re-inforced variability causes us to question the generalityof some basic goals of behavioral psychology includingprediction and control A third is that reinforced vari-ability may help us to provide a scientifically useful wayto describe voluntary action In particular the variabilityof all voluntary behaviors can readily be changed mo-ment to moment by discriminative stimuli and conse-quences Implications for everyday life also will be noted

Operant ContinuumAs was indicated above the operant is composed of

instances that is the operant is a class of many differentresponses all functionally equivalent but often topo-graphically distinct Skinner suggested and I hypothe-size that instances within the class emerge stochastically(see Moxley 1997 for a discussion of Skinnerrsquos posi-tion) Although reinforcement contingencies and contextinfluence the individual probabilities and their distribu-tions emission is stochastic nonetheless The particularmember of the operant class generally cannot be predictedwith a high degree of certainty For example a particu-lar rat may be more likely to press the lever with its leftpaw than its right but left- and right-paw responses areemitted probabilistically This class-nature of the operantsuggests one way to conceptualize operant variability interms of size of class and within-classes probability dis-tributions High variability implies large classes equalprobability distributions or both low variability impliesthe opposite

Variability may be modified more readily in some op-erants than in others implying that operant variability isnot an all-or-none phenomenon but rather a continuumwith reinforced variations in variability (RVV) differing

698 NEURINGER

for different operants RVV might best be understood byanalogy to the concepts of speed and acceleration Bothvariability and speed range from low to high RVV isanalogous to acceleration the rate of change in speedHighly sensitive RVV is indicated by the possibility ofreinforcing many different levels of variability from rep-etition to random-like responding and by the fact thatchanges in level occur readily and rapidly

Different operants manifest differences in RVV Forexample variability of chain pulls changed more thanthat of leverpresses when variability was reinforced inrats (Morgan amp Neuringer 1990) Some operant-like re-sponses may show relatively low RVV For example fir-ing rates increase in single hippocampal nerve cellswhen dopaminergic substances are contingent upon theresponse (Stein Xue amp Belluzzi 1994) suggesting thatnerve f iring is controlled by reinforcement To datehowever there is no evidence that variability of nervefiring is similarly controlled Other cases might includereinforcement of nondiscriminable (subconscious) re-sponses such as Hefferline and Pererarsquos (1963) reinforce-ment of subaware thumb twitches and biofeedback ofmuscle and cardiovascular activity In these cases rein-forcement may modify performance but it is not knownwhether variability is also sensitive to reinforcement andif so to what degree Human verbal variability may bethe most readily manipulable of all behaviors the rangeof predictable to unpredictable verbal responses beingimmense with extraordinary sensitivity to context andconsequence RVV may also differ across organisms (per-haps from simple to complex) individuals (perhaps asan index of intelligence or learning ability) and physio-logical and psychological states (WKY vs SHR depres-sion and autism)

RVV may help to distinguish operant responses fromother superficially similar cases of selection by conse-quences That is the ldquooperantrdquo may also best be concep-tualized as a continuum with RVV an important deter-miner of position along the operant continuum Oneexample of selection by consequences but low ldquooperant-nessrdquo is genetic change Variability is essential for evolu-tionary change of course and levels of genetic variabil-ity are sometimes affected by environmental conditions(Moxon 1997) but there are few examples of levels ofvariability being influenced by selection pressures Thatis the source of much genetic variability is thought to berandom not influenced by current pressures and needsThus although selected by consequences genetic changemay be low along the operant continuum Other exam-ples of operant-like phenomena for which variability although important may be relatively insensitive to con-sequencesmdashvariability in these cases is elicited not se-lectedmdashinclude the development of the central nervoussystem and immune system responses (Hull et al 2001)In both of these cases change depends upon selection byconsequences but it is not clear whether variability canalso be selected Most commonly studied operant re-sponses on the other hand show high sensitivity of vari-

ability to reinforcing events But to reiterate the mainpoint of this section operants differ in terms of RVV andtherefore in terms of their ldquooperantnessrdquo from low sensi-tivity (possibly responses of single nerve cells) to extra-ordinarily high sensitivity (verbal behavior) The operantnature of variability as well as that of every other oper-ant dimension is best conceptualized as a continuum

Prediction and ControlSmith (1992) described Skinnerian operant condition-

ing theory as Baconian in nature a primary goal of Skin-nerian behaviorists being to control behavior and hecontrasted this with the more Aristotelian goal of de-scription Skinner and his students (eg Zuriff 1985) in-deed emphasized prediction and control Many of thestudies reviewed above however involved reinforce-ment procedures that increase variability and conse-quently decrease predictability It is not only ignorancetherefore that restricts prediction and control as Skin-ner suggested but reinforcement itself Operant meth-ods and theories when applied to variability lead togreater emphasis on description with control over pre-cise response characteristics possible under some butnot all conditions

In summary an operant is a class of responses underthe control of contingencies of reinforcement The classcan be controlled and predicted given knowledge ofconditioning history Indeed the size of the class andpossibly the distribution of within-classes probabilities orlevel of variability can also be controlled and predictedBut to the extent that instances emerge stochastically aparticular instance cannot be predicted The instant re-sponse can be explained as a member of a particularclass but prediction of the instance will be diff icultsometimes to the extreme This point will be developedin the next section

Voluntary BehaviorI hypothesize that voluntary behaviors are operants for

which RVVs are highly sensitive to influence by conse-quences One way to test this hypothesis would be togather empirical data concerning how voluntary versusinvoluntary actions are classified by human observers Asfar as I know there are no psychophysical data concern-ing this question but the hypothesis implies that highsensitivity of variability to operant contingencies RVVswill be present in all actions perceived to be voluntary

This operant-variability theory of voluntary action isbased on the supposition of two attributes First volun-tary acts must be potentially explainable (given suff i-cient knowledge) on the basis of genetics conditioninghistories current contingencies anticipated conse-quences intentions goals and the like That is volun-tary behaviors are lawful The second attribute is that thevoluntary act must be potentially unpredictable That isvoluntary acts are ldquofreerdquo of control by identifiable stim-uli If behavior is perceived necessarily to result frommdashto be predicted on the basis ofmdashexternal events includ-

OPERANT VARIABILITY 699

ing reinforcement contingencies it will not be perceivedas voluntary ldquoVoluntaryrdquo implies potential indepen-dence from stimulusndashresponse or responsendashreinforcerdetermination According to this theory both explain-ability and unpredictability are necessary attributes ofvoluntary acts

How though can behaviors be simultaneously ex-plainable and unpredictable Reinforcement contingen-cies play two essential roles Reinforcement (togetherwith genes prior experiences and current environment)helps to determine the set of possible behaviors fromwhich a voluntary act emergesmdashthat is the operant classReinforcement (together with the other contributorsspecified above) also helps to determine the level of vari-ability within the class Thus both instances comprisinga class and level of variability within that class can bepredictedmdashagain given sufficient knowledge Howeverthe individual response is at least sometimes (or relatively)unpredictable The reason is that the operant class fromwhich the response stochastically emerges is sometimesextremely large (eg ldquoname a wordrdquo or ldquodo somethingrdquo)and the contingencies sometimes call for equiprobablestochastic emergence (eg ldquofool merdquo) In the latter casea voluntary act may appear to be foolish meaninglessor unwarranted but if it is voluntary it can potentially beexplained (For example in response to the questionldquoWhy did you do that silly actrdquo the answer could beldquoJust to show that you canrsquot predict my behaviorrdquo seeScriven 1965) In these cases prediction of instancesfalls to very low levels indeed and I hypothesize evenan all-knowing observer will be unable to predict thenext response at greater-than-chance levels which aregiven by the size of the operant class and by within-classesprobability distribution Even within small classes (egin the game rockndashscissorsndashpaper in which unpredictabil-ity among the three possible responses is functional) be-havioral stochasticity may lead to unpredictability of theinstant response Note however that although voluntarybehaviors are sometimes unpredictable the opposite isoften the case When contingencies call for highly con-strained particular responses predictions can easily bemade For example if a police officer asks your nameyour response can most likely be predicted Voluntary re-sponses are therefore described as potentially unpre-dictable with variability contingencies engendering dif-ferent levels of unpredictability

According to this theory behaviors low in the sensitivity-of-variability dimension are described as relatively in-voluntary and those that are high in said dimension areperceived as voluntary Low RVV may be shown at anylevel along the variability continuum For example al-ways responding in the same unchanging way will beperceived as involuntary but so will always respondingrandomly (The coin does not voluntarily come up withheads or tails) This point is essential More than 2000years ago Epicurus argued that occasional randomswerves of atoms provided the source of voluntary be-havior in an otherwise determined universe but al-

though this conjecture was prescient it was not quitecorrect ldquoFree-willedrdquo or voluntary action whether it isbased on random swerves or on quantum mechanics isnot explained by variability per se but by functionalchanges in levels of variabilitymdashthat is by rapid andprecise changes in response to environmental demandsResponses can be placed along a voluntaryndashinvoluntarycontinuum isomorphic to the operantndashrespondent con-tinuum with voluntary behavior being functional pre-cisely in the same way as operant behavior is ThusSkinnerrsquos (1974) suggestion that voluntary behavior isoperant behavior was almost correct Only those oper-ants for which variability is highly sensitive to conse-quences will be perceived as voluntary

One additional example might help Operant classes(sets of potential responses) are created ldquoon linerdquo as we goabout our daily activities As we walk from car to officewe tend to walk in a characteristic way take the samepath with tightly constrained operant classes followingone after the other in rapid succession Class sizes aresmall and within-classes probability distributions arehighly peaked Predictions of next responses by a knowl-edgeable observer can readily be made However classsizes can be increased and probability distributions flat-tened we can walk in quite strange ways indeed dancealong a new path sing hand candy to passers-by Strangebehavior indeed but made possible by the ability of op-erant sets to be modified moment to moment not only interms of class definition but in terms of class size lev-els of within-classes variations and so on The same istrue for conversation Moment-to-moment changes inverbal class definition size and predictability of within-classes instances are hallmarks of voluntary behavior

GenerativityRelated to the definition of the operant and to our discus-

sion of voluntary action is that fact that operant behaviormdashleverpressing no less than languagemdashis by its naturegenerative This claim follows from the functional na-ture of the class of responses (all members of the classare effective in producing an outcome) and the stochas-tic emission of instances From these flow never-before-emitted instances patterns and sequences (see Epstein1996) Operants especially those high on the RVV con-tinuum are inherently generative

Self-ExperimentationVariations systematic or otherwise are a necessary part

of scientific work Neuringer (1981) and Roberts and Neu-ringer (1998) described how variations can be brought to bear explicitly on everyday life in the form of self-experiments Individuals vary aspects of their lives suchas the foods they eat social contacts and methods ofmemorizing they keep records concerning the effects ofthese manipulations and form hypotheses concerningcontrolling variables Using self-experimental methodsindividuals can learn how what when where and howmuch to vary Explicitly varying onersquos everyday behav-

700 NEURINGER

iors and thoughts may help to engender what Langer(1989 2000) has referred to as mindful livingmdashthat isnonautomatic voluntary engagement with beneficialeffects in different areas including learning The impor-tant point here is that variability can be voluntarily cho-sen maintained and modified in everyday life and thatexplicit variations may have important consequences

Implications for SocietyConsideration should be given to the importance of re-

inforcement contingencies to support variability in so-cial milieus A basic point made throughout the presentarticle is that merely permitting behavior to vary doesnot necessarily engender or maximize variability (as isshown by Yoke and Prob schedules) Analogously merelypermitting varied social behavior as in a free or laissez-faire society may not suffice to produce the desired di-versity If the results shown for individual organismscan provide useful information for groups (see eg Grottamp Neuringer 1974) then when variations are desiredsocieties might reward individuals for varying not sim-ply permit it

SummaryOperants differ in the extent to which reinforcers in-

fluence levels of variation When variability is highlysensitive to reinforcement the operant may be perceivedto be voluntary Voluntary responses have two attributesThey are functional implying that the class of responsesmdashthe operantmdashcan be predicted given sufficient knowl-edge But instances emerge stochastically implyingsome degree of unpredictability Because reinforcementof variability can generate extremely large classes withstochastic emergence of instances it suggests limitationsof the two basic goals of Skinnerian behavioral theorymdashnamely prediction and control Operant variability also hasimplications for self-experimentation and for society

RELATED AREAS

Operant variability may be important in many otherareas and I will briefly review some of these

Negative reinforcement To my knowledge there havebeen no published studies in which variability is nega-tively reinforced High variability might sometimesavoid or reduce a punisher such as loss of goods ormoney but at other times low variability could be func-tional Effects of positive and negative reinforcement onvariability may have different evolutionary bases onederived from increasing probabilities of locating sourcesof food or mates the other from avoiding confrontationsor predation (Driver amp Humphries 1988) One questionconcerns how quickly variability can be changed by neg-ative as opposed to positive reinforcement There isconsiderable evidence that loss functions differ fromgain functions (Kahneman amp Tversky 1984) possiblyindicating that positive reinforcement-of-variabilityfunctions will differ from negative ones as well

Resistance to change In some of the studies describedabove variability appeared to be more resistant tochange than repetitions were for example alcohol hadlittle effect on variable operant responses but interferedwith repeated ones Doughty and Lattal (2001) tested thegenerality of these f indings using four-response se-quences with pigeons a threshold variability contin-gency in the Var component of a multiple schedule andLRLR in the Rep component Doughty and Lattal chal-lenged stability of responding by prefeeding the birds inone experiment and providing reinforcers independentlyof responding in the other (see Nevin 1992) Operantvariability was again more resistant to change than wasoperant repetition Thus reinforcement of variabilitymight increase the stability of responding (that is its re-sistance to change) a seemingly paradoxical findingHowever Mechner (1992) suggested that this is the casefor complex skills such as piano performance Varyingpractice routines will make the performance relativelyinsensitive to unforeseen changes in the environmentCare must be taken however when variation is comparedwith repetition because they have different measuresAlso the results reported by Doughty and Lattal werebased on the performance of three pigeons and were notconsistent throughout the experiment The hypothesiswarrants further study

Attraction In some bird species complex male songsattract females (for a review see Catchpole amp Slater1995) There is evidence that male song complexity isunder the discriminative stimulus control of a receptive fe-male (Searcy amp Yasukawa 1990) That explicit reinforce-ment may play a role in such song complexity is suggestedby the Manabe et al (1997) study described above An in-teresting question for further research is the extent towhich operant contingencies contribute to the attractioncreated by behavioral variability in these and other species

Discrimination of variability Animals and people canuse the variability of a stimulus array or entropy as a dis-criminative stimulus (Young amp Wasserman 2001) Whatis the relationship between perception of variability onthe one hand and emission of variable responses on theother Do individuals attempt to match their own levelsof variation to those of others in their environments whichwould involve both perception and emission How mightdiscrimination of onersquos own level of variability influenceonersquos ability to modify such levels and do influences thatobscure discriminations therefore affect ability to change(see eg van der Meer van der Weel amp Lee 1995)Hunziker et al (2002) report that although variabilitywas successfully reinforced their human subjects couldnot correctly describe the contingencies of reinforce-ment a finding similar to those of informal observationsin other studies (eg Neuringer Deiss amp Imig 2000)but whether subjects could discriminate variability levelsrather than contingencies was not studied

Attention and awareness When attention is subject tointerference or is withdrawn from the task at hand abilityto vary is degraded (eg Baddeley Emslie Kolodny amp

OPERANT VARIABILITY 701

Duncan 1998 Evans amp Graham 1980 Neuringer amp Voss2002) It is unclear at present how attention influencesthe different processes underlying operant variabilityand in particular memorial and stochastic processesRich areas for additional study are the relationships be-tween attention awareness and consciousness on theone hand and operant variability on the other Voluntaryaction and consciousness are often related (see egLibet 1999)

Concepts Concepts are thought by many to be sets ofinstances (much like operant sets) whose structures aregraded meaning that there are more or less typical in-stances of such concepts as birds (robin is typical ostrichis not) Barsalou (1987) showed that the gradation ofthese cognitive structures is manipulable or as he putsit not stable A number of variables influence structureincluding context individual differences and points ofview Barsalou argued that these structures are createdas the need arisesmdashthat is on line by the thinking indi-vidual The structure of concepts may parallel that of op-erants and in both cases levels of within-structures di-versity or variability may be modified by consequencesOperant variability may be found in the cognitive as wellas in the behavioral domain

Play and humor Highly variable behaviors are ob-served when animals and people especially children en-gage in play and humor (see eg Fagen 1981) The re-sult of play may be acquisition of new responses (seeSutton-Smith 1975) Although not currently needed re-sponses learned during play may provide the source ofvariations for successful shaping of behaviors at anothertime This is quite analogous to the role of conserved ge-netic variants which are passed along to progeny but donot currently serve any identifiable role Play may alsoinvolve learning how to vary operantly

Reinforcement by variability The self-maintainingnature of play suggests that engaging in variable behav-iors may itself be reinforcing Indirect support comesfrom findings that varied stimulation can serve as a re-inforcer and that stimulus repetition becomes aversive(Fiske amp Maddi 1961) Possibly also related are findingsthat varying reinforcers and permitting choice among re-inforcers increase their value (Bowman Piazza FisherHagopian amp Kogan 1997) and that permitting choicesamong varied responses is also preferred to demandingfixed responses (Catania 1980)

Foraging Two issues regarding variability are presentin the foraging literature One has to do with risk prefer-ence versus avoidance (see eg Shettleworth 1998) Asecond has to do with search strategies Most reportedcases of foraging involve systematic search patterns(Stephens amp Krebs 1986) However search patterns aresometimes stochastic or highly variable (see eg Peter-son 1997) One question is whether some species can varybetween these two strategies and the challenge is to iden-tify the pressures leading to and the advantages of each

Animal training Wilkes (1996) has applied operantvariability techniques in developing training proceduresfor pet dogs More generally reinforcement of variabil-

ity may enable the gaining of control over behaviors ofanimals and as has been suggested above may be an im-portant part of the process of shaping

Models of learning Reinforced variations in variabil-ity may be important when human and animal learningis being modeled whether the models are mathematicalin form computer based or machine based (Blumberget al 2002 Brooks 1999 Sutton amp Barto 1998)

CONCLUDING COMMENT

Genes interact with environments to produce diverseanatomical physiological and behavioral phenotypesGenes and environments also provide for another level ofdiversitymdashnamely operant variability There are manysources of operant variability including random environ-mental events memory for past responses and stochasticgeneration each of which ultimately derives from ge-netic and environmental influences To the extent thatoperant variability is based on a stochastic source how-ever the behavioral results cannot be predicted fromknowledge of these influences That is organisms haveevolved to behave unpredictably The positing of sto-chasticity to explain behavior has a history extending atleast as far back as Epicurus with major contributors in-cluding Gustav Fechner (see Heidelberger 1987) EgonBrunswik (see Gigerenzer 1987) Skinner (1959) andSuppes (1984) Knowledge of the stochastic nature ofthe operant and of how it is controlled leads to a worldview that differs from genetic and environmental deter-minism with roots in the works of the scholars just men-tioned At the scientific level this view leads to the ex-perimental analysis of consequence-influenced variabilityAt the philosophical level it leads to explanations of vol-untary action At the personal level it can lead to change

REFERENCES

Amabile T M (1983) The social psychology of creativity New YorkSpringer-Verlag

Antonitis J J (1951) Response variability in the white rat duringconditioning extinction and reconditioning Journal of Experimen-tal Psychology 42 273-281

Archer J amp Birke L (1983) Exploration in animals and humansCambridge Van Nostrand Reinhold

Arnesen E M (2000) Reinforcement of object manipulation in-creases discovery Unpublished undergraduate thesis Reed College

Baddeley A D (1966) The capacity for generating information byrandomization Quarterly Journal of Experimental Psychology 18119-129

Baddeley A [D] Emslie H Kolodny J amp Duncan J (1998)Random generation and the executive control of working memoryQuarterly Journal of Experimental Psychology 51A 819-852

Baer D M amp Sherman J A (1964) Reinforcement control of gen-eralized imitation in young children Journal of Experimental ChildPsychology 1 37-49

Baker M J amp Koegel R L (1999) Transforming children withautismrsquos ritualistic behaviors into positive play interactions with sib-lings Paper presented at the meeting of the California Association ofBehavior Analysis San Francisco

Balsam P D Deich J D Ohyama T amp Stokes P D (1998) Ori-gins of new behavior In W OrsquoDonohue (Ed) Learning and behav-ior therapy (pp 403-420) Boston Allyn amp Bacon

Balsam P D Paterniti A Zechowy K amp Stokes P D (2002)

678 NEURINGER

crease somewhat Importantly the artist would continueto behave much more variably than the assembly-lineworker Both contingency and frequency affect responsevariability but contingency appears to be more influen-tial at least over the range of values studied to date Thesefindings if general are important for those who withholdreinforcement in order to encourage response variationsas is often the case when behavior modifiers attempt toshape new behaviors Effects of withholding may dependon baseline contingencies and a more productive proce-dure might be to reinforce variability directly We willconsider this suggestion further below

Discriminative ControlThe operant response as defined by Skinner is con-

trolled by discriminative stimuli as well as by conse-quences For example red traffic lights are the discrimi-native stimulus for depressing the automobile brake thedistance of the golf ball from the hole provides a discrim-inative stimulus for how hard to hit the ball In the labora-tory discriminative control is often studied with multipleschedules in which one reinforcement contingency suchas fixed ratio is operative in the presence of one stimulussuch as a red keylight and a different contingencymdashforexample extinctionmdashis operative in a second stimulusmdashfor example a green keylight Discriminative control isshown by differences in responding Variability is alsocontrolled by discriminative cues

Exteroceptive discriminative stimulus controlPage and Neuringer (1985 Experiment 6) reinforced pi-geons for repeating a single sequence LRRLL in thepresence of blue keylights and for variable sequences inthe presence of red Blue and red alternated after every10 reinforcements The birds learned to repeat in thepresence of blue and to vary in the presence of red andwhen the stimulus relationships were reversed the birdsreversed their patterns Cohen Neuringer and Rhodes(1990) obtained similar results with rats Denney andNeuringer (1998) extended this research to show thatdiscriminative control did not depend on reinforced repe-titions in one of the components Again within the contextof a multiple schedule rats were rewarded for variablesequences under one stimulus (Var) and independentlyof variability under a second stimulus (Yoke) Frequen-cies of reinforcement were identical in the two stimulusconditions That the stimuli exerted significant controlover variability was indicated by three main findingsVariability was higher in Var than in Yoke differences invariability were greatest immediately following stimuluschange and when the discriminative stimuli were re-moved and all other aspects of the program kept constantvariability immediately converged on an intermediate andidentical level in the two components The conclusion isthat operant variability is controlled by discriminativestimuli These findings are important in terms of both ex-plaining observed levels of variability and controllingthem For tasks in which response variations are impor-tant such as when new operants are being shaped or when

an individual is attempting to behave creatively or solveproblems cues that indicate ldquoVaryrdquo can be helpful

Discriminative control without exteroceptive stim-uli The relationship between a response and a reinforcercan itself provide a discriminative stimulus even whenexternal cues such as lights or tones are unavailable Forexample under an FI schedule there is a period duringwhich reinforcers cannot be obtained The subject learns topause following each reinforcement although no cuesother than the reinforcer are provided Another case is themixed schedule identical to the multiple schedule describedabove but with no external cues Animals sometimeslearn to respond differentially to the consequences oftheir behaviors even under such stimulus-impoverishedconditions (Ferster amp Skinner 1957) The question stud-ied next was whether responsendashreinforcer relationshipscan provide discriminative cues sufficient to control lev-els of variability This is an important question becauseas was indicated above it has been assumed that with-holding reinforcement elicits variability The present hy-pothesis is that absence of reinforcement can sometimesserve as a discriminative cue for operant variability Thedistinction between elicited (respondent) and reinforced(operant) variability will be revisited shortly

Hopson Burt and Neuringer (2002) rewarded ratsunder a mixed schedule in which two periods of time orcomponents Var and Repetition (Rep) alternated with-out any external cues In the Var component four-responsesequences of L and R leverpresses were reinforced ifthey met a threshold variability contingency in the Repcomponent only repetitions of LLLL were reinforced(Probabilities of reinforcement were equalized in the twocomponents by intermittent reinforcement of LLLL inRep) The results demonstrated discriminative controldespite the absence of exteroceptive cues (see Figure 3)With trials since the unmarked switch into the Var com-ponent shown in the left column variability increasedand LLLL sequences decreased with trials since the unmarked switch into Rep shown in the right columnresponding became increasingly repetitive until LLLLsequences predominated The two functions were sim-ilar which is an important finding in that it indicatesdiscriminative control rather than elicitation If variabil-ity had been elicited because reinforcement was tem-porarily withheld (when the contingencies were switched)then variability should have increased following entryinto both Rep and Var components but that was not theresult

Therefore variability that is observed when rein-forcement is withheld for short periods as when a newresponse is being shaped may be partly discriminativelycontrolled despite the absence of external cues that isanimals and people may learn when it is functional tovary and some of the cues may come from responsendashoutcome relationships Control over this operant vari-ability may be more precise than that elicited by extinc-tion because as was noted above operant contingenciesdefine both the set of acceptable variations and the level

OPERANT VARIABILITY 679

of required variability It may also be maintained muchlonger because as will be seen shortly extinction weakensresponding

ChoiceThe frequency of one response relative to others or

response probability is used as a measure of choice orpreference Choices are found to match the probabilitiesof their consequences For example relative frequenciesof pecking one of two keys approximate relative fre-

quencies of reinforcement Neuringer (1992) showedthat the same matching relationship also governs choicesby pigeons to vary or repeat The more variations werereinforced the more the pigeons chose to vary and sim-ilarly for repetitions Each trial consisted of four L andR responses and reinforcements for varying and for re-peating were systematically manipulated The Var con-tingency was met whenever the current sequence dif-fered from each of the previous three sequencesmdasha lag 3criterion The Rep contingency was satisfied if the cur-

Figure 3 The left column shows the probabilities ( y-axis) that each of five rats met a variability contin-gency (solid circles) versus a repetition contingency (open circles) when they were being reinforced for vary-ing (Hopson Burt amp Neuringer 2002) Along the x-axis are trials since the mixed schedule (no cues pro-vided to indicate the component) had switched into the variability-reinforcing component The right columnshows the corresponding data for performances when reinforcement was contingent upon repetitions thex-axis here indicating trials since the unmarked switch into the repetition-reinforcing component

680 NEURINGER

rent sequence repeated at least one of the previous threeOn an average of once every 30 sec (VI 30 sec) the com-puter probabilistically primed reinforcement for Var orRep the prime being maintained until the reinforcementhad been collected Thus if the computer primed Var re-inforcement was delivered whenever the lag 3 variabilitycontingency was next met Similarly if Rep had beenprimed the next Rep sequence was reinforced Proba-bilities of Var and Rep reinforcements were systemati-cally manipulated the two always summing to 10 and amatching-type relationship was obtained (Figure 4panel A) The relationship was more complex than sim-ple matching possibly because if the pigeons chose tovary by responding stochastically there was some prob-ability due to chance that the current sequence wouldrepeat a previous sequence This was shown by a com-parison of pigeon performances to the performance of acomputer-simulated stochastic model programmed tomatch its Var choices to the probability of Var reinforce-ments The pigeonsrsquo choice distributions were found tobe the same as the modelrsquos (Figure 4 panel B) Thuschoices to vary or repeat were governed by relative fre-quencies of reinforcement An animal can choose whetherwhen and how much to vary a result consistent withthose described above from Blough (1966) Machado(1989) and Grunow and Neuringer (2002) Choices ofwhether or not to vary appear to be governed similarly toall other operant choices

These f indings may have important applicationsWhen an individual repeatedly responds in a way that isnonfunctional reinforcement of choices to vary mayhelp the individual to emerge from the ineffective pat-tern This can be true for those experiencing everydayproblems but feeling helpless to change and it may beespecially important for those experiencing depressionor who respond in otherwise abnormally stereotypedmanners such as those with autism These topics will beconsidered in a later section

ExtinctionAs was indicated above extinction is often found to

increase variability but almost all studies of this effectinvolve responses that had previously been repetitiveAntonitis (1951) for example reinforced rats for pokingtheir noses anywhere along a 50-cm horizontal openingAlthough the exact location of pokes did not matter withcontinued training responses became limited to one or afew locations along the strip When reinforcement waswithheld variability of location increased (see also Eck-erman amp Lanson 1969) Variability produced by extinc-tion has been reported for many other response dimen-sions as well including force (Notterman amp Mintz1965) number (Mechner 1958) topography (Stokes1995) and sequence (Balsam Paterniti Zechowy ampStokes 2002 Mechner Hyten Field amp Madden 1997)

The issue is complicated however by other studiesthat show that previously reinforced responses and re-sponse patterns are maintained intact during and afterextinction That is extinction does not break down

learned patterns For example when Schwartz (1981) re-inforced pigeons for any sequence of four L and four Rresponses each bird developed a dominant sequence asdescribed above and these sequences were maintainedduring extinction Despite decreasing response ratesduring extinction response patterns were retained Sim-ilar maintenance of response structure has been ob-served during extinction following reinforcement of

Figure 4 Average performance across a group of pigeons inNeuringerrsquos (1992) study of choices to vary or repeat (A) Loga-rithms of the ratio of number of times the pigeons met the Varcontingency divided by the times they met the Rep contingencyas a function of the analogous numbers of reinforcements re-ceived for each Points represent group averages in two separateexperiments Also shown is the least squares best-fitting line(B) Changes in U-value an index of response variability as afunction of the obtained percentage of reinforcements for vary-ing The solid line in panel B represents the performance of acomputer-based model that matched its choices to vary (vs re-peat) to its obtained relative frequency of reinforcement for vary-ing From ldquoChoosing to vary and repeatrdquo by A Neuringer 1992Psychological Science 3 p 248 Copyright 1992 by the AmericanPsychological Society Reprinted with permission

OPERANT VARIABILITY 681

choices (Myerson amp Hale 1988) responding underfixed-interval contingencies (Machado amp Cevik 1998)visual discriminations (Nevin 1967) and matching tosample (Cumming Berryman Cohen amp Lanson 1967)How can stability during extinction be reconciled withincreased variability

Neuringer Kornell and Olufs (2001) wrestled withthis question in a study of extinction following rein-forcement under three different contingencies An oper-ant chamber contained two levers left (L) and right (R)on either side of a reinforcement dispenser and a key (K)in the rear of the chamber distant from the dispenserOne group of rats Rep was reinforced for repeating afixed sequence of responses LKR A second group Varwas reinforced for variations across the three operandaunder a threshold contingency Reinforcement for a thirdgroup Yoke was matched to those of the Var animalsand was therefore independent of the particular sequencesBecause of the unusual configuration of operandaskewed distributions of sequence probabilities were ob-served in all of the groups an outcome that was antici-pated and desired in order to document possible in-creases in variability during extinctionmdasha flattening ofthe distributionmdashas well as possible decreases That isduring the reinforcement phase the Rep group came torepeat the LKR sequence with high probability the Yokegroup tended to emit the easiest sequences such as LLLand RRR but distributed their responses over more se-quences than the Rep group did and the Var group dis-tributed most widely although because of differences inthe operanda and in their locations some sequences weremore preferred than others and in particular sequencesending with a K response were least likely (the key beingfarthest from the reinforcement dispenser) Thus thecontingencies resulted in three different distributions orhierarchies of sequence probabilities very steep forRep intermediate for Yoke and flattest for Var

The main finding was that withholding reinforcementhad similar effects in the three groups In each responserates fell to low levels across the four sessions of extinc-tion Furthermore sequences that were most frequentduring the reinforcement phase continued to be most fre-quent throughout extinction A slight flattening of thedistributions was observed however with the lowestprobability sequences increasing relative to the rein-forcement phase Thus because the overall hierarchy ofsequence frequencies was maintained the results indi-cated stability but because the distributions were some-what flattened the results also showed slightly increasedvariability It is as if the strategy during extinction wereto keep doing what had worked in the past but occa-sionally to do something very different This compli-cated story integrates two effects of extinction increasedvariability and maintained stability Most importantly forthe present discussion extinction had similar effects onvariable yoked and repetitive operants

One additional comparison is of interest Recall thatGrunow and Neuringer (2002) found that when contin-

gencies demanded high variability obtained levels ofvariability decreased slightly as reinforcement frequen-cies were lowered This result is opposite to the Vargrouprsquos slight increase in variability just described underextinction conditions The reason may be that extinction-induced variability is short lived lasting only as long asresponding continues perhaps for hours or a few ses-sions at most whereas the variability engendered by theVI 5 intermittent schedule of reinforcement in Grunowand Neuringer can be maintained indefinitely The de-crease in Grunow and Neuringerrsquos operant variabilitymay be due to the weakening of the reinforcement effect(ie reinforcement is less frequent) whereas the in-crease following extinction may be a short-term elicitedeffect This distinction between reinforced and elicitedvariability is important and will be discussed further inthe next section

Operant and Respondent InteractionsElicited or what is sometimes referred to as respondent

effects occur when environmental changes that affectbehavior do not depend on (are not caused by) the behav-ior Pavlovian conditioning being the most commonlystudied example Operant effects are observed when a re-inforcer depends on the response to produce it Althoughfor historical reasons respondent and operant effects areoften studied separately most behaviors result from acombination or interaction of the two Variability is nodifferent and we turn to an example of such an interaction

As reinforcement is approached under FI or fixed-ratio (FR) schedules response rates are commonly ob-served to increase Similarly when reinforcement de-pends on accuracy as in discrimination learning ormatching to sample it too is found to increase within theFI or the FR (Nelson 1978 Nevin 1967) Cherot Jonesand Neuringer (1996) asked whether the same patternheld for reinforced variability Does operant variabilityincrease as reinforcement for varying is approachedThe question may be related to such real-world cases aswhen a composer is rushing to complete a compositionbefore a scheduled performance or when a programmeris trying to solve a problem before an upcoming meetingThese activities involve at least to some extent non-repetitive original or variable thought and behaviorThe question is whether variability increases with prox-imity to or anticipation of reinforcement for varying

Cherot et al (1996) employed trials consisting of fourresponses across L and R operanda and a lag 4 variabil-ity contingency with rats in one experiment and pigeonsin another Unlike in any of the cases described so farhowever the lag-variability contingency had to be satis-fied four times in order for a single reinforcer to be ob-tained With ldquordquo indicating successful completion of theVar contingency ldquo2rdquo indicating failure to meet that con-tingency and PELLET indicating the food reinforcer anexample follows LRLL RRRL LRLL2 LRLL2RLLR RRRL2 LRRR+PELLET Thus food reinforce-ment was provided only after the subject had satisfied

682 NEURINGER

the variability contingency four times FR 4 The resultswere that as the food reinforcer was approached (ie asprogress was made through the ratio) the subjects wereless and less successful at varying The probability ofmeeting the variability contingency in the first trial afterreinforcement was higher than it was in the second whichin turn was higher than that in the third trial and so onFor comparison a Rep group was reinforced for repeatingsequences once again with reinforcement only after foursuccessful repetitions Probability of repeating increasedwith proximity to reinforcement a finding consistentwith the literature but opposite that for the variabilitygroup Thus approach to reinforcement facilitated oper-ant repetition but interfered with operant variabilitySimilar variability-interfering effects have been docu-mented for within-trials responses The probability that aresponse repeats the just prior response increases within atrial even when sequence variability is reinforced (McElroyamp Neuringer 1990 Neuringer 1991) To return to the ex-ample of a composer rushing to complete her composi-tion she may indeed have increasing difficulty respond-ing creatively under the pressure of a deadline On theother hand the soloist may find it easier to repeat thecorrect performance during practice periods as the con-cert nears How can these results be reconciled with themany findings that variability can be reinforced

An additional result of Cherot et al (1996) provides ananswer The Var animals responded much more variablyoverall than did the Rep animals Although variability inthe Var groups decreased with approach to the reinforceroverall variability was always much higher when it was re-inforced (Var group) than when it was not (Rep group)Reinforcement therefore exerts two influences Overallvariability is elevated sometimes to the highest levelswhen it is reinforced but approach to (and possibly focuson) the reinforcers constrains or lowers variability Sim-ilar reinforcement-produced interference effects had beenreported previously for example when raccoons are re-inforced for dropping a ldquocoinrdquo into a slot the reinforce-ment results in the animalsrsquo holding and rubbing the coinrather than dropping it (Breland amp Breland 1961) Theoverall enhancement of variability was of much greatermagnitude in Cherot et al than the decrease with approachwas although both effects were statistically significant(The notes created by the composer under pressure ofdeadline will be less readily predicted than will thoseplayed by the performer practicing a well-known piece)This study highlights the importance of interactions be-tween operant effects of reinforcement and elicited effectsproduced by anticipation of reinforcement and the resultsmay be relevant to the debate concerning possible detri-mental effects of reinforcement on creative behaviors (seethe discussion below and Neuringer in press)

SummaryReinforcement controls behavioral variability as it

does other operant dimensions Levels of variability areprecisely governed by schedule contingencies and dis-

criminative stimuli Choices to vary or repeat match rel-ative frequencies of reinforcement Extinction increasesvariability slightly whereas the hierarchies of sequenceprobabilities are retained and operant and respondenteffects interact The evidence shows that behavioral vari-ability is an operant controlled by its consequences

GENERALITY AND PARAMETERS

Many different methods measures species and re-sponses have been used in the study of operant variabil-ity and this diversity supports the generality of the phe-nomenon As is the case for repeated operants howeverthere are parametric constraints This section briefly re-views the generality and some of the parameters studiedto date

GeneralityMethods Throughout this section ldquoresponserdquo will

signify a response unit including both an individual re-sponse and a multiresponse sequence Each of the meth-ods to be described increases or maintains variability byreinforcing it

Novel response procedures reinforce responses upontheir first occurrence ldquoeverrdquo (Pryor et al 1969) or upontheir first occurrence within a session (Goetz amp Baer1973) Radial-arm maze procedures also reinforce ini-tial (within a given session) entries into arms of a mazeand result in a distribution of choices across the availableoptions the sequence or pattern of choices often beingquite variable

Lag procedures reinforce a response if it has not oc-curred across a given number of previous trials (Page ampNeuringer 1985) For example under lag 5 the currentsequence is reinforced only if it has not occurred duringany of the previous five trials A variant is the recurrence-time procedure which specifies a minimum number of in-tervening trials before a previously emitted response canagain be reinforced Machado (1989) combined recurrencetime with a percentile-reinforcement contingency to gen-erate high levels of variability (see also Machado 1992)Percentile-reinforcement contingencies base the criterionfor reinforcement on the subjectrsquos own performance over aprevious set of responses (see Galbicka 1994) For exam-ple a sequence will be reinforced only if it has not oc-curred over the previous n trials the value of n continuallychanging so that 50 of the trials are reinforced

Least-frequent response procedures (Blough 1966Schoenfeld Harris amp Farmer 1966 Shimp 1967) rein-force responses that are currently least frequent across amoving window

Threshold procedures reinforce responses whose rela-tive frequencies fall below a given value Because thistype of procedure was used in the research describedabove and in many other more recent studies details willbe provided In Denney and Neuringer (1998) which willserve as an example trials consisted of four responses byrats across L and R levers A running tally was kept of the

OPERANT VARIABILITY 683

number of occurrences of each of the 16 possible se-quences LLLL LLLR LLRL and so on If the relativefrequency of the current sequencemdashthe number of its oc-currences divided by the sum of all 16 sequencesmdashwasbelow a specified threshold value in this case 05 the ratwas rewarded Recently emitted sequences contributedmore to the maintained tally than did those in the distantpast because after each reinforcement all 16 counterswere multiplied by a weighting coefficient in this case95 Therefore the influence of a particular trial decreasedexponentially with successive reinforcements Thresholdprocedures differ from Bloughrsquos (1966) least-frequentprocedure in that more than one sequence (or response)can be reinforced at any given timemdashnamely all those se-quences that have relative frequencies below the thresholdvalue The procedure differs from lag in permitting agiven sequence to be reinforced many times in successionif the associated relative frequency is sufficiently low

The frequency dependence procedure (Machado 19921993) is a variant of threshold in which the probability ofreinforcement is a continuous function of relative re-sponse frequency The more frequent a response is thelower the probability is that it will be reinforced

Statistical feedback procedures provide subjects withcomparisons between their performances and that of amodel Neuringer (1986) showed human subjects feedbackfrom 10 statistical tests thereby enabling them to comparetheir own distributions of responses to that of a randommodel Neuringer and Voss (1993) provided feedbackbased on a chaotic model The goal in each of these caseswas for the subjects to learn to approximate the model andthe results of these experiments will be discussed below

Differential reinforcement of switches versus repetitionsbalance a subjectrsquos tendency to engage in each of these pat-terns Rather than reinforcement for infrequent responsesor random-like distributions reinforcement is providedsometimes for repeating a response and other times forswitching from one response to another In the two-operandum case a switch is L followed by R or R by La repetition is L then L or R then R Bryant and Church(1974) found that stochastic-like responding was gener-ated by rats when switches were reinforced with a 75probability and repetitions with a 25 probability that isthe rats had a natural tendency to repeat at least in theenvironment of this study Machado (1997) reinforcedpigeons for eight pecks across two keys if the sequenceof pecks contained at least one (one group of birds) ortwo (another group) switches Although the birds couldhave satisfied these contingencies by repeating a singlesequencemdashin the ldquoone switchrdquo condition for examplerepetition of LRRRRRR would produce reinforcement on100 of the trialsmdashthey tended to vary their sequencesBarba and Hunziker (2002) showed however that vari-ability was higher when it was directly reinforced thanwhen reinforcement was based on proportions of switchesa finding consistent with that of Machado (1997)

The evidence from each of these methods supports thehypothesis that variability can be reinforced

Measures Variability must be interpreted with refer-ence to a particular measure The reason is that there canbe high variability (and low predictability) at one level ofanalysis but order (and certainty of prediction) at a dif-ferent level For example the sequence 1 1 2 2 3 3 44 5 5 1 1 2 2 has the same variance (a commonlyemployed measure of variability) as 2 3 2 1 5 1 4 41 1 3 2 5 2 but an analysis of conditional proba-bilities in which the unit contains pairs of instances in-dicates that the two sequences have different levels ofvariability In general high variability observed with onemeasure does not rule out the possibility of order ob-served with another and vice versa That is why evalua-tions of random-number generators often employ manydifferent measures and indeed there are hundreds of dif-ferent measures of the randomness of a sequence (Knuth1969) All behavioral studies make simplifying assump-tions and rely on one measure or at the most a few

U-value has been the most commonly employed mea-sure of operant variability (Evans amp Graham 1980Machado 1989 Page amp Neuringer 1985 Stokes 1995)U-value measures the distribution of relative frequen-cies or probabilities of a set of responses For a set of16 possible responses i = 1ndash16 U-value is given by

where RFi = the relative frequency (or probability) ofeach of the 16 responses U-values approach 10 whenrelative frequencies approach equality as would be ex-pected over the long run from a random generator andthey approach 00 when a single instance is repeated

Other measures include the percentage of trials or fre-quency at which variability contingencies are met (Page ampNeuringer 1985) percentages of alternations versus stays(Machado 1992) conditional probabilities of responses(Machado 1992) the number or frequency of novel re-sponses (Goetz amp Baer 1973 Schwartz 1982) the num-ber or frequency of different responses (Machado 1997Schwartz 1982) Markov analyses (Machado 1992) andstatistical tests of distributions of responses (Neuringer1986) All of the measures employed to date indicate thatvariability can be reinforced

Responses and species Generality has further beendemonstrated by the use of many different responses andspecies These include Siamese fighting fish (Betta splen-dens) swimming in quadrants of a tank (Roots 2000)young chicks pecking keys (Neuringer 2002) pigeonspecking keys (Page amp Neuringer 1985) budgerigarssinging songs (Manabe Staddon amp Cleaveland 1997)rats pressing levers (Van Hest van Haaren amp van de Poll1989) porpoises swimming and jumping (Pryor et al1969) rhesus monkeys pressing levers (Sydney Reisbickpersonal communication) children drawing and build-ing with blocks (Holman Goetz amp Baer 1977) andhuman adults and children pressing buttons (Saldana ampNeuringer 1998 Stokes amp Balsam 2001) There havebeen few attempts to compare species or responses but

- ( )[ ] ( )=aring RF RFi ii

log log 2 21

1616

684 NEURINGER

in one study lowest baseline variability was observedwhen rats pulled trapezes intermediate levels were ob-served when they pushed keys and highest levels wereobserved when they pressed levers (Morgan amp Neuringer1990) with variability increasing in all cases when it wasreinforced In another study operant variability wascompared across rats and people with similar effects ob-served in the two species (Neuringer Deiss amp Imig 2000)

ParametersWe turn now to some parameters studied to date Many

other parameters as yet unstudied are no doubt also im-portantmdashfor example motivation and types of reinforcers

Age Older people (see eg Van der Linden Beertenamp Pesenti 1998) and older rats (see eg Lamberty ampGower 1990) respond less variably than their youngercounterparts when variable responding is reinforced or re-quested but in most cases it is not clear to what extent thevariability is elicited as opposed to reinforced Neuringerand Huntley (1992) compared reinforced variability in2-month-old (young) and 10-month-old (mature) Long-Evans rats Under Yoke contingencies the young rats re-sponded significantly more variably (two levers four re-sponses per trial) than the mature rats which constitutesan elicited effect When reinforcement was contingenton variability (lag 4) variability increased significantlyin both groups thereby overcoming the initial differencecorrelated with age Thus explicitly reinforcing vari-ability in older animals caused them to respond no dif-ferently than younger animals did a finding potentiallyrelevant to real-world human aging However when Wells(1999) compared 8-month-old (mature) to 24-month-old(old) rats although the mature rats responded more vari-ably than the old rats did under Yoke contingencies thedifference was not statistically significant Additionalstudies are needed especially of whether reinforcingvariable behaviors in older populations can facilitatelearning and performance

Gender Male and female rats are equally sensitive toreinforcement of variability (Neuringer amp Huntley 1992van Hest et al 1989) However there was a tendency(not statistically significant) for males to respond over-all more variably than females did and because thesestudies did not control for differences in size and weightof subjects or for other aspects of motivation additionalstudies are necessary

Stage of training Stokes and Balsam (2001) reporteddifferences in variability as a function of whether variabil-ity was initially reinforced early or later in training (seealso Gottlieb 1992) Human subjects generated sequencesof 10 L and R responses per trial on a computer keyboardunder lag 0 and lag 25 contingencies Under lag 0 everytrial ended with reinforcement under lag 25 the currentsequence had to differ from each of the previous 25 Whenlag 25 was experienced at the very beginning of traininglevels of variability at the end of training (when lag 0 wasoperative) were lower than they were when lag 25 was ex-perienced after 50 trials of initial lag 0 These results were

interpreted to suggest that there is an ldquooptimal period forsetting variability levelsrdquo (Stokes amp Balsam 2001 p 181)but that general conclusion needs further testing and anumber of questions need to be answered One has to dowith the contribution of reinforcement frequency Underlag 0 every sequence leads to reinforcement but rein-forcement is intermittent under lag 25 Another has to dowith whether learning to vary differs from learning to repeatwith respect to optimal periods (see eg Bouton 1994)

Related to stage of training are effects of prior experi-ence Hunziker Lee Ferreira da Silva and Caramori(2002) showed that for human subjects if Var contingen-cies were experienced just prior to Yoke contingencieslevels of response variation during the Yoke phase werehigher than they would have been if the Var contingencieshad not been previously experienced The time delay be-tween Var and Yoke experiences mattered with a long in-terval (3 months) producing only a small effect Saldanaand Neuringer (1998) also found effects of prior Var train-ing on performances in Yoke (increased variability inYoke) but the opposite did not hold that is training underYoke had relatively little effect on Var Hunziker Caramorida Silva and Barba (1998) reported similar effects withrats Thus there may be an asymmetry Reinforcement forvarying has relatively strong influences but reinforce-ment without regard to variations does not adversely affectlearning to vary These different effects are not surprisinggiven that Yoke contingencies are permissivemdashreinforce-ment is given whether or not responses varymdashwhereascontingencies under Var can be quite demanding How-ever the results motivate a number of additional ques-tions One is whether effects of Var training differ fromthose observed when repetitions are reinforced Another iswhether direct reinforcement of variable behaviors canhelp animals or people to overcome the detrimental effectsof prior noncontingent experiences such as those that pro-duce learned helplessness (Seligman 1975)

Environmental constraints Stokes (1995) arguedalso that variability increases when behavior is con-strained The term ldquoconstraintrdquo is ambiguous because allreinforcement contingencies exert a form of constraintIn one study Stokes (1995) compared different strate-gies of shaping leverpresses Rats in one group were re-inforced initially for pressing the lever with only theright paw (right only) and eventually for any press (withright or left paw or with any other part of the body) Theorder of training was reversed for a second group ldquoanyrdquofirst and then right only Behavior sequences were morevariable in the right-only-first group leading Stokes(1995) to conclude that early constraints lead to highvariability (see also Stokes Mechner amp Balsam 1999)That is an important hypothesis with many implicationsbut controls are needed before the results can be attrib-uted to the contingencies For example ldquoright onlyrdquo andldquoanyrdquo differed in terms of reinforcement distributions aswell as in the precision of the reinforcement operationthe ldquoanyrdquo reinforcers were provided immediately and au-tomatically by the programming equipment whereas in

OPERANT VARIABILITY 685

the right-only condition a human observer had to decidewhether a response had occurred and then present the re-inforcer necessarily leading to longer and more variedlatencies and a less consistent contingency Furthermorethere are many cases in which increasing constraints suchas those due to a decrease in the number of responses pertrial (Page amp Neuringer 1985) or the requirement of ex-actly four responses on each of two keys (Schwartz 1980)results in decreased variability Thus the contributionsof response-definition constraints and environmentallimitations must be explored further

SummaryThe generality of operant variability is shown by the

robustness of the phenomenon across procedures mea-sures responses and species As with other operant phe-nomena parameters influence the effects

FUNCTIONS

All operants including variability are functional Op-erant responses produce reinforcing consequences Be-cause other behavioral competencies depend on it oper-ant variability is functional in a broader sense as wellSome of these functions will now be described

Operant ConditioningThe potential of a reinforcer to change behavior is lim-

ited by the range of available variations (Hull Langmanamp Glenn 2001 Skinner 1981) and thus whatever pro-duces those variations contributes to the conditioning ofnew operants Much work has been done to identifysources of variation (Balsam Deich Ohyama amp Stokes1998 Segal 1972 Staddon amp Simmelhag 1971) butNeuringer (1993) was first to show that the same rein-forcers that strengthen individual responses can also pro-duce the necessary variability and that these two functionsof reinforcement may work simultaneously Implied isthat when an operant response is being conditioned re-inforcement works not only to strengthen an individualresponse but at the same time to engender the variabilityfrom which the response emerges (That is variability isnot solely a baseline phenomenon elicited or noise)Concurrent reinforcement of variations and individualtarget sequences will be the topic of this section

Pigeons in one experiment and rats in another were re-warded for varying among 16 possible response sequences(four responses per trial L and R operanda lag contin-gencies Neuringer 1993) Concurrently one of the 16designated as the target sequence was reinforced when-ever it occurred Thus reinforcements were given bothfor sequence variations and for a single selected targeta reinforced-variation-and-selection (RVS) procedureWhen the target was ldquoeasyrdquo (eg LLLL) it emergedrapidly from the baseline variations Given that the tar-get was always reinforced reinforcers were obtained morefrequently when the subjects did not vary and only re-peated the target When the target was ldquodifficultrdquo (eg

LLLR) it was not learned As in the choice experimentdescribed above (Neuringer 1992) the relative valuesor difficulties of target versus variations governed theprobability that the target would be learned

However even difficult targets could be conditionedunder the RVS procedure as was shown by another ex-periment in the same series (Neuringer 1993) Sequencevariations were again reinforced concurrently with a dif-ficult target RRLR but now the frequency of reinforce-ments for varying was systematically decreased Satis-fying the variability contingency was intermittentlyreinforced according to a VI schedule and over sessionsreinforcement for varying became less and less frequentThis technique resulted in acquisition of even the mostdifficult target sequence whereas a control group rein-forced for the target alone (without the concurrent rein-forcement for variations) never learned

Reinforcers were obtained more frequently in the RVScondition than in the control condition (since the RVSsubjects were rewarded both for varying and for emittingthe target sequence) and the additional reinforcers mighthave generated greater motivation to respond (Killeen1981) Thus motivation not the direct reinforcement ofvariability might have been responsible for the RVS ef-fect To test this possibility Neuringer Deiss and Olson(2000) reinforced three groups of rats for emitting a par-ticularly difficult target sequence LLRLR The experi-mental group was reinforced concurrently for sequencevariability on the average of once per minute (VI 1 min)The target-only control group received no reinforce-ments other than for the LLRLR target A third groupwas yoked to the experimental animalsmdashreinforced onceper minute for any sequence variability not requiredmdashand as for the other two groups whenever the target oc-curred If the additional reinforcement was responsiblefor the facilitated learning under RVS then because theexperimental and yoked groups received equal reinforce-ments they should have learned equally well but thatwas not the result Only the experimental group learnedto emit the LLRLR target The yoked controls respondedat high rates throughout the experiment but they did notvary sufficiently to make contact with the target contin-gency and never learned The target-only control groupreceived insufficient reinforcement and their responsesextinguished Therefore the added reinforcers helped tomaintain motivation as suggested above but reinforce-ment of variations was necessary for learning of a diffi-cult sequence The RVS procedure maintains both highmotivation and high variation and the two together re-sult in the acquisition of difficult-to-learn responses

Grunow and Neuringer (2002 Experiment 2) studiedone possibly important parameter of the RVS proceduremdashnamely the level of variability When a difficult target re-sponse is being trained does it matter whether reinforcedvariability is low intermediate or high As describedabove each of four groups of rats was reinforced for vari-able response sequences the first group for high vari-ability the fourth group for low variability and the other

686 NEURINGER

two for intermediate levels In other ways the procedurewas similar to that of the Neuringer Deiss and Olson(2000) RVS contingencies just described with a singledifficult target sequence always reinforced The resultswere that the higher the level of reinforced baseline vari-ability the more quickly the target was learned Thusvariability levels indeed mattered and if the just-re-ported effect is found to be general it will be most help-ful to reinforce high variability when difficult-to-learnnew responses are being shaped Other parameters re-quiring study are class definitionsmdashwhich responses areincluded in the class of acceptable variationsmdashand num-ber of instances within the class

One additional effect should be noted When varia-tions among a set of responses are reinforced if one vari-ant is selected for extinction (no longer reinforced) thefrequency of that instance decreases Neuringer (1993)showed this for pigeons responding under a lag-variabilityschedule When one of the 16 sequences was no longerreinforced (with no other change to indicate its status) thefrequency of that sequence decreased Thus reinforce-ment of variations provides the baseline to select for (iestrengthen) and select against (ie weaken) individualinstances

These findings have potentially important implicationsFirst as has been indicated the same reinforcers thatstrengthen particular instances can also reinforce varia-tions That a binary eventmdashreinforce or not reinforcemdashsimultaneously affects variations and selected instancesindicates both the power of reinforcement and a concep-tual problem How reinforcers exert such diverse effectsis a question to which we will return

Second shaping procedures may implicitly reinforcebaseline variations An approximation to a goal responseis reinforced for a period of time but eventually rein-forcement is withheld until a response closer to the goalis obtained It is commonly thought that withholding re-inforcement or mini-extinction induces the necessaryvariations for shaping to proceed as has been indicatedabove An alternative suggested by the present studiesis that organisms learn both to vary (within a set) and to re-peat (selected instances) until one instance is always rein-forced That is the process of shaping might involve re-inforcement both of variations (generally done implicitly)and of targeted instances If so a useful contributor tothe shaping process might be explicit reinforcement ofvariations

Third reinforcement of variations could facilitate ac-quisition of other competencies such as cognitive andmotor skills Siegler (1996) has shown in an important se-ries of studies that childrenrsquos acquisition of complex cog-nitive skills is correlated with high variability Anotherexample is that when motor skills are practiced in a vari-able manner high quality of performance results (Ma-noel amp Connolly 1995 1997 Mechner 1992 Schmidtamp Lee 1999) However it is rare that variability in thesedomains is directly reinforced as part of the training oreducational process

Fourth the RVS procedure might be used in therapeu-tic settings to modify maladaptive and self-injurioushabits Therapists sometimes employ noncontingent re-inforcement procedures to weaken undesirable behaviors(Vollmer Iwata Zarcone Smith amp Mazaleski 1993)The reasoning is that many maladaptive behaviors aremaintained by unintended consequences and that non-contingent rewards may compete with those conse-quences thereby lowering the strength of the unwantedhabit For example when a staff personrsquos attention toself-injurious behavior helps to maintain it providing at-tention noncontingently may decrease the strength of theunwanted target behavior However the RVS proceduremight accomplish the same goal more effectively andwithout undesirable side effects (Vollmer RingdahlRoane amp Marcus 1997) Noncontingent rewards some-times result in unwanted ldquosuperstitiousrdquo behaviors andthat would be avoided under RVS Furthermore RVSwould help to provide the set of baseline behaviors nec-essary to establish desirable new responses

Finally RVS procedures may help to gain control overand eventually to weaken other undesirable target behav-iors The target can be incorporated within a set of moredesirable responsesmdasha new operant class established withthe target as one membermdashand variable instances rein-forced Once variable responding is successfully main-tained including the undesirable target response the lattercan be weakened by selectively withholding reinforcement

Summary The conditioning of operant responses de-pends on both baseline variations and selection of suc-cessive approximations to a goal response The same re-inforcers used to select the approximations can reinforcebaseline variations thereby defining the precise set ofappropriate variants and the desired levels of variationSuch concurrent reinforcement of variations and selec-tions may facilitate the acquisition of difficult-to-learnoperants and selective withholding of reinforcementmay weaken undesirable instances

Creativity and Problem SolvingBehaving in a novel unusual or nonrepeating manner is

part of creative activities (Campbell 1960) Novelty andvariation may arise through combinations of previouslylearned behaviors (Epstein 1996) transformations of thesebehaviors (Boden 1994) direct reinforcement of variabil-ity as described above or some combination thereof Ofcourse neither novelty nor variability alone suffices forcreative outputmdashthe throw of the dice or the toss of acoin are not creative nor are haphazard strokes of a paint-brush on a canvasmdashbut without variation creativity islow Whether the variability required for creativity canbe reinforced is relevant to whether creativity itself canbe reinforced The latter question has led to contentiousdebate with some arguing that reinforcement facilitatescreativity (Cameron amp Pierce 1994 Eisenberger ampCameron 1996 Stokes 2001 Winston amp Baker 1985)and others focusing on detrimental effects of reinforce-ment (Amabile 1983 Deci Koestner amp Ryan 1999)

OPERANT VARIABILITY 687

The debate goes well beyond the present discussion butsome of the research described above is relevant Firstreinforcement contingent on variability generates higherlevels of behavioral variability than does reinforcementcontingent simply on responding as has been shown bythe many studies described above in which Var contin-gencies were compared to Yoke contingencies Similarlyprecise reinforcement contingencies matter for creativity(Cameron amp Pierce 1994 Lepper amp Henderlong 2000)If any product rather than only creative ones is rein-forced creativity tends to fall (just as variability decreasesunder yoke conditions) Second approach to or antici-pation of reinforcement interferes with operant variabil-ity (Cherot et al 1996 McElroy amp Neuringer 1990Neuringer 1991) which again might be related to effectson creativity when an individual focuses on the rein-forcer Note however that although variability decreaseswith proximity to the reinforcer overall levels of vari-ability are much higher when variability is reinforced thanwhen it is not The same may be true for creativity aswell anticipation of reward may interfere with creativeoutput to some extent but at the same time high creativ-ity may depend on its reinforcement

Operant variability may be important for problemsolving as well Here too there is disagreement as towhether training is helpful or not Early studies by Maier(1933) indicated that it is and these found support inlater work (eg Lung amp Dominowski 1985) Somestudies showed that those individuals most likely toldquobreak the setrdquo are also most likely to solve a problem(Dover amp Shore 1991) However a number of studiesfound that practice had no influence (Duncan 1961)

Arnesen (2000) explored a related phenomenon witha rat model One group Var was reinforced for variableinteractions with novel objects and another group Yokewas reinforced independently of such interactions EachVar rat received training with 12 different objects oneper session For example in the presence of a soup canthe rat was reinforced the first time it entered the canbut thereafter a different response was required such asrolling it climbing on it and so forth This procedure issimilar to that of Goetz and Baerrsquos (1973) described aboveexcept that objects changed from session to session Ratsin the Yoke control group had identical experiences in anadjoining identical chamber that contained identical ob-jects but now rewards were not contingent on object ma-nipulation After this training the rats were tested in adifferent room with 30 never-before-experienced ob-jects scattered on the floor Hidden in each was a smallpiece of food When compared to their Yoke controls theVar animals were more active during this test exploredmore manipulated objects more and most importantlydiscovered significantly more food pellets Another con-trol group received no preliminary training with objectsand their performance in the test session did not differfrom that of the yoked controls Therefore when ratswere reinforced for variable interactions with objectsthe result was an increased tendency to explore manip-ulate and discover new sources of reinforcement It

would be of considerable interest to determine whethera similar effect occurs with children

Assessment and Treatment of PsychopathologiesAbnormal levels of behavioral variability characterize

some human psychopathologies as has been shown byclinical observations as well as by research with humansubjects and animal models Few studies have attemptedto reinforce variability directly in such problem popula-tions The main question is whether reinforcement canmodify abnormal levels of variability in the direction ofnormalcy and whether such changes might have associ-ated beneficial effects

Autism Stereotyped responding characterizes indi-viduals with autism (Hertzig amp Shapiro 1990) For ex-ample Baron-Cohen (1992) asked children with autismand others in a control group to hide a penny in one handso that the experimenter could not guess which hand heldthe penny Those with autism were more likely thanthose without to generate a simple predictable patternsuch as repeatedly switching back and forth from left toright hands Such repetitive responding can be maladap-tive For example Mullins and Rincover (1985) askedchildren with and without autism to pick one of five cardsFood was occasionally found in a cup behind the cardEach of the five cards was a discriminative stimulus fora different schedule of reinforcement including contin-uous reinforcement (CRF every choice reinforced) FR 2(fixed ratio 2 in which every second choice was rein-forced) FR 4 (every fourth choice reinforced) FR 7 andFR 11 Control participants sampled all five alternativesand quickly learned to choose the most frequently rein-forced CRF card Autistic children sampled only a lim-ited number of cards and therefore often preferred anonoptimal alternative Thus children with autism tendto respond less variably than normal children and thiscan result in failure to adapt

Miller and Neuringer (2000) asked whether direct re-inforcement could influence the variability of individu-als with autism Sequences of responses on two largebuttons were first reinforced probabilistically and inde-pendently of variability (Prob phase) after which rein-forcement was contingent upon satisfying a thresholdvariability contingency (Var phase) The autistic partic-ipantsrsquo variability was significantly lower than that ofmatched controls during both Prob and Var phases butwhen variability was reinforced its levels increased sig-nificantly in both groups This is an important findingbecause it suggests that direct reinforcement of variabil-ity may help to modify some of the behavioral stereo-types that are characteristic of autism Lee McComasand Jawor (in press) offer support for this hypothesisThree individuals with autism were reinforced for variedverbal responses to social questions the responses had tobe appropriate given the context Variability increased intwo of the three individuals In a related procedureBaker and Koegel (1999) reinforced children with autismfor varied social interaction in a game-playing contextAlthough the reinforcement was permission to engage in

688 NEURINGER

the stereotyped behaviors preferred by the child the re-sult was increased social interactions both during theperiod of the experimental manipulation and in 1- and2-month follow-ups These three studies taken togethersuggest that reinforcement of varied behaviors may facil-itate modification of nonfunctional ritualistic and stereo-typed behaviors

Depression Depression provides another example ofmaladaptive consequences of low variability Lapp Mar-inier and Pihl (1982) observed that depressed womenproduced fewer alternative solutions to hypotheticalproblems than did nondepressed women Channon andBaker (1996) found that attempts by moderately de-pressed college students to identify faults in a series ofinterconnected circuits were less varied and less success-ful that were those of nondepressed controls HorneEvans and Orne (1982) found that when depressed pa-tients were asked to generate a random sequence of num-bers their responses were less variable than were thoseof controls These experimental results are consistentwith clinical observations

Can depressed individuals be rewarded for increasingvariation Hopkinson and Neuringer (in press) dividedcollege students into mildly depressed and nondepressedgroups on the basis of the CES-D scale a paper-and-pencil self-evaluative index of depression In the first oftwo phases sequences of responses on a computer key-board were reinforced independently of variability (Prob)following which only highly variable sequences were rein-forced (Var) Under Prob depressed students respondedsignificantly less variably than did the controls in agree-ment with previous findings Under Var variability in-creased significantly in both groups Indeed by the endof the experiment depressed and nondepressed subjectswere responding equally variably (Instructions were alsoeffective in increasing variability in the depressed par-ticipants) Thus even in a simple computer-game envi-ronment mildly depressed subjects manifested lowervariability than controls did but direct reinforcementmodified their behavior to control levels Can reinforce-ment of variability be used in real-world conditions tohelp depressed individuals An affirmative answer comesfrom cognitive behavior modification procedures thatwork to increase alternative solutions to problems (Beck1976) A more general conclusion from these studies ofautistic and mildly depressed individuals is that direct-reinforcement-of-variability procedures may benefitthose who manifest abnormally low levels of variability

Attention Deficit Hyperactivity Disorder (ADHD)ADHD may provide an example of the opposite stateabnormally high levels of variability (Barkley 1990)Wultz Sagvolden Moser and Moser (1990) proposedthat the spontaneously hypertensive rat (SHR) may serveas a good animal model of ADHD the SHR shows manyof the behavioral characteristics of that disorder includ-ing hyperactivity poor learning ability poor ability totolerate delayed reinforcement and poor self-control

The next study to be described asked about operant vari-ability in SHRs

Mook Jeffrey and Neuringer (1993) compared vari-ability of SHRs and of a control strain Wystar Kyoto(WKY) under two conditions Prob and Var The mainresult was that SHRs tended to respond variably whetheror not the contingencies required it consistently with theresults from the human ADHD literature When vari-ability was reinforced the two strains responded at ap-proximately equally high levels of variation These re-sults were confirmed in two other studies (HunzikerSaldana amp Neuringer 1996 Mook amp Neuringer 1994)Thus control rats varied only when the contingencies re-quired variations whereas SHRs varied whether or notthe contingencies required it As might be expected fromthese results when reinforcement depended on repeti-tions the SHRs performed poorly (Mook et al 1993)

Another similarity between SHRs and ADHD humanswas seen when amphetamine a drug that has effects simi-lar to those of Ritalin often prescribed for ADHD was ad-ministered (Mook amp Neuringer 1994) When repetitionswere required for reinforcement amphetamine improvedperformances by the SHRs so that they were as accurate assaline-injected WKYs a finding again consistent withthose in the ADHD literature These studies support SHRsas a model of ADHD and suggest that additional attemptsshould be made to assess whether or not individuals withADHD are less able to change levels of variability thanare controls Although as indicated above there is someevidence in support of this hypothesis (Barkley 1990)when Saldana and Neuringer (1998) directly comparedhuman ADHD and control participantsrsquo operant variabil-ity within the context of a computer game no differenceswere observed Most of those with ADHD were main-tained on Ritalin and other drugs however a fact unfor-tunately not documented in the original article

Drug EffectsAlthough many drugs are known to influence behavioral

variability (Brugger 1997) rarely have attempts beenmade to analyze explain or identify the sources of theeffects As will be discussed below operant variabilitycan result from the memory-based process of remember-ing previous responses and then behaving differently orcan result from responding stochastically a process thatappears to be quite different McElroy and Neuringer(1990) attempted to distinguish the effects of alcohol onmemory from its effects on stochastic processes in astudy in which one group of rats was reinforced for vary-ing four-response sequences under a lag 5 contingency(Var) and another was reinforced for repeating a singlesequence LLRR (Rep) Ethanol injections of 075 and20 gkg affected Var and Rep performances differentlyWhereas the Rep animals were much less able to satisfythe LLRR contingency at both doses the Var rats re-sponded variably whether or not alcohol had been ad-ministered Similar results were obtained by Cohen et al

OPERANT VARIABILITY 689

(1990) with a multiple schedule in which each rat servedas its own control In the same animals and within thesame sessions alcohol degraded Rep performances butleft Var intact Thus alcohol appears to interfere with atask that requires working memory as presumably is thecase for the Rep task but not with one that reinforcesvariability The memory-interfering effect of alcoholmight account for why alcohol also degrades radial-armmaze performance (Devenport 1983)

Alcohol also affects choices to vary or to repeat For ex-ample Gorka (1993) reinforced rats concurrently for re-peating and varying and monitored percentage of choicesas a function of time following injection of ethanol Inthe control sessions when the rats were injected withsaline the probabilities of varying and repeating remainedapproximately constant across the 45-min session witha slight bias in favor of varying After injection with 15gkg of ethanol however there was a significant increasein choices to vary at the beginning of the session andthen a later decrease to below 50 That is alcohol ledinitially to a markedly increased preference for varyingbut later to a preference for repeating (Figure 5) This re-search should be replicated and extended especially be-cause the contingencies ldquopressuredrdquo the animals to re-main at approximately 50 preference but thesuggestion is that alcoholrsquos effects on response variabil-ity may depend in part on the duration of its influence Amore general implication is that drugs may affect re-peated and variable operants differently and that effectsdepend on time following administration

SummaryReinforced variability may contribute to many types

of learning including operant shaping and acquisition ofcognitive and complex motor skills Similarly creativityand problem solving depend on variable responding andin these areas as well explicit reinforcement of variationsmay be functional Abnormal levels of variability aresometimes caused by psychopathologies and drugs andoperant variability may be useful in these areas in twoways first as a way of assessing the variability that is pos-sible and second to modify variability toward more nor-mal levels Studies show that individuals with autism andmild depression vary less than do controls but that ex-plicit reinforcement of variability moves the levels towardsnormalcy Other studies show that drugs such as alcoholmay affect repeated and variable operants differently

EXPLANATIONS

Two questions must be answered in order to explainoperant variability First what are the sources of vari-ability that permit its reinforcement Second how is itmaintained Related questions including how instancesbecome members of a set from which variations emergeand whether operant variability is a generalizable com-petency will also be considered

Origination From Nonoperant SourcesSkinner (1966) reasoned that operant responses orig-

inate from nonoperant sources with reinforcement con-tacting a behavior shaping and strengthening it onlyafter the behavior had occurred for reasons other than re-inforcement To put it simply a behavior must occur be-fore it can be reinforced (Balsam amp Silver 1994) I willconsider three initial sources of variable behavior en-dogenous variability noncontingent environmental eventsand extinction These appear to parallel evolutionaryprocesses and they too will be identified

Endogenous variability All physical phenomenavary and behavior is no exception This fact can providethe opportunity for reinforcement to select particularlevels of variation Examples of spontaneous variabilityinclude spontaneous alternations in mazes (Dember ampRichman 1989) nondifferentially reinforced variationsin operant responses (Guthrie amp Horton 1946) and ex-ploration (Archer amp Birke 1983) Endogenous muta-tions provide a parallel in genetics Base-pair changes inDNA molecules occur spontaneously intron-bearinggenes mix and match segments to encode new proteinsjumping genes introduce new DNA sequences and atthe level of the chromosome random crossings and as-sortment occur as a natural part of reproductive meiosisThese sources of random genetic change occur indepen-dently of external influences

Figure 5 The relative frequency of variations by rats as a func-tion of 5-min blocks of time since the beginning of the sessionThe open circles represent performance after the rats were in-jected with saline indicating a slight tendency to vary more thanrepeat but one that is approximately constant throughout thesession The filled circles represent performance after injectionwith 15 gkg ethanol indicating a much higher likelihood ofvarying soon after the alcohol is injected followed by a decreasein variations to below 50 and then a return to baseline at theend of the session Group averages are shown Data are fromGorka (1993)

690 NEURINGER

Noncontingent events Noncontingent events some-times referred to as chance accidents or luck also affectbehavior One happens upon a particular passage in a bookthat leads to a new thought one happens to sit next to aninteresting woman or man on an airplane which leads tomarriage (Bandura 1982) In science as well as in every-day life accidents or serendipity are an important part ofthe process of discovery (Beveridge 1957) In each of thecases presented above an accidental event leads to new be-haviors that are then strengthened by consequences Thesame is potentially true for variability itself Particular lev-els of variability might occur for adventitious reasons butthen become functionally related to consequences

That organisms are particularly sensitive to such unan-ticipated noncontingent events is shown by two basicprinciples of learning habituation and Pavlovian condi-tioning Organisms attend preferentially to unexpectedevents and stop attending or habituate to repeated onesThe conditioning of new Pavlovian responses dependsaccording to influential theories upon unexpected rein-forcing events (eg Rescorla amp Wagner 1972)

Noncontingent events such as X-rays and other pro-cesses that can act as mutagens play an important role ingenetic variation as well The particular genes affectedand the type of changes are random there is no neces-sary relationship or contingency between the externalevent and the affected gene However sensitivity to mu-tations is not random but has been selected presumablybecause it is functional Thus in both behavior and ge-netics sensitivity to noncontingent events is conserved

Extinction Extinction increases behavioral variabilityat least to some extent as was described above Distance inspace or time from the reinforcer does the same (Cherotet al 1996) A possible parallel at the genetic level in-volves high levels of deprivation at least in simple lifeforms For example when E coli bacteria are deprivedof nutrients unusually high mutation rates are observedThe exact mechanism by which such hypermutations areinduced is not known but they increase the likelihoodthat a population can survive such environmental insults(Moxon 1997) That is deprivation-induced variability isfunctional at the genetic level (The behavioral evidenceregarding long-term deprivation is less clear some stud-ies showing increased variability and others increasedstereotypy eg Carlton 1962 Senkowski Vogel amp Po-zulp 1978)

Endogenous variations external noncontingent eventsand withholding of reinforcement are three sources forthe variability that precedes control by reinforcementBut once control by reinforcement is established a dif-ferent question emerges What behavioral processes areinvolved when variability is maintained as an operantThe sources just discussed generate uncontrolled non-specific and often short-lived variability and cannot ex-plain the precisely controlled operant type

Operant Variability ProcessesThree processes are hypothesized to underlie operant

variability The first involves the use of random environ-

mental events such as the toss of a coin The second in-volves memory for responses such as when an animallearns not to return to previously visited locations Thethird will be referred to as an endogenous stochasticgenerator

Random environmental events Throughout historymany cultures have used random events to influencechoices and decisions such as throws of dice and randomselections of sticks cards bones or organs Today coinflips are used by a referee at the beginning of a game to de-cide which team kicks the ball a computerrsquos random-number generator is employed by scientists to avoid bi-ases in assigning subjects to experimental groups andaleatoric events are sometimes employed in modern artmusic and literature The Dice Man by Rhinehart (1998)provides a fictional example in which the protagonistbored with life writes a number of possible actions onslips of paper and then periodically selects one blindlyand behaves accordingly These examples show that randomexternal events may be used to avoid biases engender un-likely responses and break out of behavioral niches

Memory-based variability Many methods used tostudy operant variability differentially reinforce nonrepe-titions at least over the recent past Therefore one strat-egy that subjects can employ is to cycle through few re-sponses or sequences especially when the variabilitycontingencies are relatively permissive For exampleunder a lag 2 contingency cycling repeatedly throughthree sequences will result in 100 reinforcement Ani-mals and people sometimes do this apparently basingtheir current response on memory for or discriminativecontrol by the just emitted responses One such instancewas shown by Schoenfeld et al (1966) who rewardedrats for IRTs that fell into one of 10 interval classes onlyif the current IRT differed from the interval class of theprevious response The rats came to alternate betweenlong and short IRTs thus behaving systematically to meetthis lenient variability contingency Machado (1993)studied an analogous effect having pigeons peck L and Rkeys under a frequency-dependent variability contin-gency related to Shimprsquos (1967) procedure describedabove When the trial unit consisted of two responses thebirds developed memory-based repetitive sequencessuch as RRLLRRLL and thereby satisfied the variabilitycontingency When the trial unit was three responses inlength the birds did not develop the optimal fixed pat-tern of RRRLRLLLmdashwhich was too complexmdashbut in-stead developed ldquorandom-like behaviorrdquo (Machado1993 p 103) Thus a memory-based strategy was em-ployed when it was possible but when the memory loadbecame too high stochastic responding emerged A sim-ilar pattern was seen when variability of budgerigarsongs was reinforced (Manabe et al 1997) Under lag 1contingencies the birds sang mainly two songs underlag 2 they increased to three songs Only when the lagwas increased to 3 did song diversity increase apprecia-bly in one of the birds In radial-arm mazes as well whencontextual cues are available memory again appears toplay an important role leading to much lower probabil-

OPERANT VARIABILITY 691

ities of repetition than would be expected from a sto-chastic generator (McElroy amp Neuringer 1990)

Note that under lag schedules memory-based strate-gies result in higher reinforcement rates than respondingstochastically does The reason is that a stochastic gen-erator repeats by chance That animals and people oftendo not employ memory-based strategies probably indi-cates the difficulty of remembering long sequences andpossibly other advantages of unpredictable responding

If a memory strategy is the only way to satisfy vari-ability contingencies then it can be used even when thememory requirements are demanding as was shown byNeuringer and Voss (1993) Human subjects were trainedto generate on a computer keyboard a chaotic-like se-quence of responses that matched iterations of the logistic-difference equation

where Rn refers to the nth iteration in a series with R be-tween 00 and 10 and t is a constant between 10 and40 When the t parameter approaches 40 iterations ofthis equation result in very ldquonoisyrdquo sequences As withother chaotic processes however there is an orderlinessunderlying the surface noise In this case if responsesare autocorrelated with Rn drawn as a function of Rn21the resulting function is a parabola In the Neuringer andVoss (1993) study following each response college stu-dents were shown how close their predicted values wereto that from iterations of the logistic-difference model Inone experiment the students made digital predictions inanother they placed a pointer along a horizontal lineand this analogue response was then translated into dig-ital values for computation With training the studentsbecame increasingly adept at responding in chaotic-likefashionmdashthe studentsrsquo predictions matched closely theiterations of the logistic functionmdashand their autocorrela-tions became parabolic The left column in Figure 6 showsperformances by 4 subjects (rows) early in training thatis not well approximated by the logistic-differenceparabola and the right column shows performances at theend of training in which r 2 values were 72 99 98 andapproaching 10 How was this accomplished Sinceeach iteration in the logistic-difference sequence is basedon the prior outputmdashthat is the sequence is generated bya deterministic although nonlinear processmdashit was hy-pothesized that the human subjectsrsquo chaotic-like responseswere also based on memory for prior responses To put itsimply the subjects may have memorized a long series ofpairs in the form ldquoIf the previous response was value Athen the current response must be value Brdquo as Metzger(1994) Ward and West (1994) and Ward (2002) have sug-gested

To test whether subjects in fact relied on memory forprior responses Neuringer and Voss (2002) systemati-cally slowed responding by interposing pausesmdashthat isby requiring an IRT As IRT increased the difference be-tween the subjectsrsquo sequences and the modelrsquos chaotic

Rn n nt R R= -( )- -1 11

Figure 6 Autocorrelations (responses on trial n on the y-axis asa function of n21 on the x-axis) for 4 human subjects at the be-ginning (left column) and end (right column) of training inNeuringer and Voss (1993) Equations for the best-fitting curveand amount of variance accounted for (r2) are shown at the topof each panel To the extent that the circles (responses) approxi-mate the equation y = 40x 2 40x 2 the subjects were respondingin a way that matched the logistic-difference chaotic functionFrom ldquoApproximating Chaotic Behaviorrdquo by A Neuringer andC Voss 1993 Psychological Science 4 p 115 Copyright 1993 bythe American Psychological Society Reprinted with permission

692 NEURINGER

output increased and the underlying parabolic structurewas broken That is when pauses increased subjects wereless and less able to approximate the modelrsquos chaotic dis-tribution (see Figure 7) The important point here is thathighly variable responses which in this case are chaoticin nature can be based on memorial processes

Endogenous stochastic generator The evidence tobe reviewed next indicates that animals and people canrespond stochastically and that this competency is basedneither on chance environmental events nor on memoryfor prior events or responses It is impossible to provethat an endogenous stochastic process underlies operantvariability that would be analogous to proving the nullhypothesis However much evidence is consistent withthe stochastic hypothesis Included are the facts thatmemory limitations do not permit adequate respondingunder many variability contingencies that response dis-tributions match those expected from a stochastic gener-ator that changes in parameter values affect operantvariability as they would a stochastic generator and thatinterference with memory processes leaves stochasti-cally based operant variability unaffected As a wholethe evidence is consistent with an endogenous stochas-tic source of operant variability

Memory limitations One type of evidence is nega-tive in character and is based on the presumed inability ofmemory for prior responses or sequences to satisfy sometypes of variability contingencies Two examples weregiven above (Machado 1993 Manabe et al 1997) andanother was provided by a lag 50 contingency in Pageand Neuringer (1985) In the latter case the sequence in

the current trial (eight responses across L and R keys) wasrequired to differ from those in each of the previous 50 tri-als including at the beginning of a session the last 50trials from the previous session Pigeons responded wellunder these contingencies and in a way similar to that ofa stochastic model It is unlikely that pigeons can remem-ber each of their previous 50 eight-response sequences

Response slowing As was shown above for chaotic-like responding performances are often degraded by de-lays imposed between a stimulus and a response pre-sumably because of limitations in working or short-termmemory (Berryman Cumming amp Nevin 1963 Williams1971) Neuringer (1991) relied on this effect to test whethermemorial processes could explain the generation of vari-able sequences under lag schedules Two groups of ratswere reinforced for L and R leverpresses producing fourresponses per trial One group (Var) was trained under alag 5 contingency and the other (Rep) was trained to asingle sequence of responses LLRR It was assumedthat accurate Rep performance depended on workingmemory After performances had stabilized IRTs weresystematically varied by the imposition of timeouts of05 to 20 sec between consecutive responses in the se-quence During these forced pauses the chamber wasdark and responses were not counted toward completionof the sequence If the animals had used the previous re-sponse(s) as a discriminative cue for the current responsethen performances should have been degraded by theseinterpolated pauses

The results were that for the Rep group as the re-quired pauses increased from 0 to 6 sec percentages ofcorrect sequences were relatively unaffected but at pausesgreater than 6 sec probability of a correct LLRR se-quence fell sharply This pattern of results suggests in-volvement of working memory of prior responses withthe rats able to remember responses within a 6-sec win-dow The Var grouprsquos results were quite different Aspauses increased between 0 and 6 sec the rats were moreand more likely to satisfy the variability contingenciesand with pauses greater than 6 sec percent correct re-mained at a high and constant level Thus the Var grouprsquosresults were opposite to those of the Rep group and op-posite to those predicted from a memory-based strategySimilar results showing that operant variability increasesor is maintained as responding is slowed have been re-ported by Morris (1987) with pigeons and by Baddeley(1966) with random response generation by human sub-jects The results reported above concerning alcoholrsquos ef-fects on variability and repetition were also consistentwith those of Neuringer (1991)

Memory for prior responses may not have controlled theVar grouprsquos performance but why did variability increaseEach of three answers is consistent with a stochastic-generator interpretation Weiss (1964 1965) hypothe-sized that random responding requires that current re-sponses be independent of previous ones memory for orcontrol by prior responses interferes with such indepen-

Figure 7 Approximation to a chaotic sequence (given by delta)as a function of the imposed interresponse times Shown are theaverages of 5 subjects

OPERANT VARIABILITY 693

dence and therefore pauses would be expected to facil-itate variability A second possibility is that at shortIRTs animals tend to repeat responses on the sameoperandum Blough (1966) found this in pigeons and ex-cluded such double pecks from his analyses because theyappeared not to be under the control of reinforcement con-tingencies Morris (1987) also found a tendency for birdsto repeat when no interresponse timeouts were imposedA third hypothesis is that there were two contributors tothe observed variability One was a stochastic processcontrolled by reinforcement and the other elicited vari-ability generated by interposition of pauses The high vari-ability elicited as an effect of slowed responding is ageneral phenomenon supported in many other cases Ac-cording to this interpretation operant variability in the Vargroup was governed by a stochastic-based process operantrepetition in the Rep group was governed by a memory-based process and pauses elicited variability under bothcontingencies Thus Rep performance was interferedwith whereas Var was facilitated Each of these hypothe-ses is consistent with the conclusion that memory for (ordiscriminative control by) prior responses does not con-tribute to and possibly interferes with variable respond-ing when an organism is reinforced for variability andthis in turn is consistent with the stochastic-generatorhypothesis

Stochastic response distributions Other evidence iscorrelational Response distributions are similar to thosefrom a stochastic generator The data from two experi-ments already discussed are shown in Figure 1 In thesecases reinforcement of least frequent IRTs (Blough 1966)and of nonrepetitive choice sequences (Machado 1989)resulted in stochastic-like responding

Neuringer (1986) reported another striking case Highschool and college students generated sequences of 1sand 2s on a computer keyboard with each set of 100 re-sponses constituting one trial A baseline phase lastedfor 60 such trials for a total of 6000 responses duringwhich the students were asked to behave as randomly aspossible as if they were reporting the outcome of a se-ries of coin tosses As in the many previous studies ofthis sort the subjectsrsquo responses differed significantlyfrom those of a stochastic model (Brugger 1997 Wage-naar 1972) During the training phase the subjects re-ceived feedback which enabled them to compare their per-formances with that of a stochastic model first accordingto one statistical test and then according to another untilfollowing each trial feedback was provided from 10 dif-ferent statistics The results were that all of the subjectslearned to approximate the stochastic model accordingto the 10 statistics as well as some (but not all) other sta-tistical tests That is the distributions of the statisticswhich clearly did not match a stochastic model at the be-ginning of training approximated the stochastic modelat the end of it (see Figure 8)

The previous failures of other researchers to obtain ev-idence for random responding had led some to concludethat humans did not have the capacity to behave ran-

domly What would account for the difference betweenthe present positive finding and the many other negativeones Neuringerrsquos (1986) study was the first in which sto-chastic responding was explicitly reinforced in this casewith feedback from 10 statistical tests An analogy mighthelp to explain the difference in procedures and resultsImagine an experimenter walking around a city and ask-ing people if they know what a violin sounds like (analo-gous to asking people whether they know what a series ofcoin tosses or dice throws is like) Most would answeryes If the same individuals were then asked to play theviolin very few if any would succeed The experimentermight conclude after many replications with differentsubjects in different cities that people did not have thecapacity to play the violin The point of course is thatplaying the violin is a skill requiring training and prac-tice and the same may be true for responding randomlyIt is easy to vary but practice and reinforcement may benecessary to approximate a random distribution

Response number Page and Neuringer (1985) testedthe stochastic-generator hypothesis in another way byvarying number of responses per trial with pigeons work-ing under a lag 3 contingency The reasoning was that ifprevious responses served as a discriminative cue forwhat not to domdasha memory-based strategymdashthen perfor-mance should be degraded when the number of re-sponses was increased Eight responses per trial requiresubjects to remember more than four responses The sto-chastic hypothesis predicts the opposite outcome as isdemonstrated by the following example If each trialwere two responses in length with responses determinedby the toss of a coin then the probability that a given trialwould repeat the previous one would be 25 (There arefour possible sequences in the first trial RR RL LRand LL Thus the second trial has a 1-in-4 chance ofmatching the first) If a trial consists of four coin tossesthe probability of a repetition by chance is 0625 or 1 in16 Thus if subjects used a stochastic process to gener-ate Ls and Rs performances should improve with in-creasing responses per trial Results were precisely thosepredicted by the stochastic hypothesis and inconsistentwith a memory strategy Probability of meeting the con-tingencies and therefore probability of reinforcementincreased as responses per trial increased In otherwords eight-response trials were easier for pigeons thanfour-response trials

Response interference Neuringer and Voss (2002)attempted to distinguish between memorial and stochas-tic processes in yet another way They hypothesized thatwhereas chaotic responding was based on memory forprior responses (as was discussed above) responding inNeuringerrsquos (1986) study with feedback from 10 statisti-cal tests indicated involvement of an endogenous sto-chastic generator To show that human subjects could en-gage in both processes at will three college studentswere trained to generate chaotic sequences in one compo-nent of a multiple schedule as in Neuringer and Voss(1993) and to generate stochastic sequences in a sepa-

694 NEURINGER

rate component as in Neuringer (1986) The wordCHAOTICS was on the screen in the first component theword STOCHASTICS was on during the other and thesewords alternated throughout the session After the sub-jects succeeded at each of the component tasks test ses-sions were introduced in which memory for prior re-sponses was subject to interference in both the chaoticsand stochastics components as follows During the

chaotics component four separate chaotic sequenceswere cycled one after the other each cued by differentscreen colors The subjects attempted to approximate achaotic sequence in one color (eg light green) follow-ing which they would have to match a second chaotic se-quence (lavender screen color) and so on the fourchaotic contingencies following one another across thesession Although all four chaotic sequences used the

Figure 8 Performance by one subject at the end of training to respond stochastically in Neuringer(1986) Shown are comparisons on 6 different statistics between the subject (solid lines and Xs) andthe random model (dashed lines and closed circles) At the end of training the subjectrsquos perfor-mance did not differ statistically from that of the random model on these and four additional sta-tistics From ldquoCan people behave lsquorandomlyrsquo The role of feedbackrdquo by A Neuringer 1986 Jour-nal of Experimental Psychology General 115 p 71 Copyright 1986 by the American PsychologicalAssociation Reprinted with permission

OPERANT VARIABILITY 695

same logistic-difference equation that a model did theywere out of phase and independent If chaotic respond-ing is memory based then the four independent se-quences would be expected to interfere with one anotherthat is approximations to the four chaotic models wouldbe lower than when only a single chaotic output was re-quired as in Neuringer and Voss (1993) That was ex-actly the finding Performances by the three subjectswere signif icantly degraded during the four-segmentphase To put it simply chaotic responses interfered withone another

A different pattern of results was seen in the stochasticsportion of the experiment The same interference proce-dure was used with four independent stochastic sequencesrequired in the four different color conditions As inNeuringer (1986) subjects were required to satisfy eightdifferent statistical tests of randomness and to do this in-dependently in each of the four phases (If responses in anytwo or more of the four stochastic phases became corre-lated above a minimal level responses were considered tobe incorrect thereby prohibiting repetition of a single se-quence in two or more of the phases) For one subjectseven of eight statistics were closer to those of a randommodel during this interference phase than they were at theend of the original training with only a single sequenceand for a second subject all eight statistics were closerThe third subject showed closer-to-random approximationunder the four-segment condition on only two of the eightstatistics but most of her changes were also in the directionof a random generator For example if a random generatoremits equal numbers of ldquouprdquo and ldquodownrdquo sequences andthis subject in the baseline phase had emitted only 47ldquouprdquo sequences then during the four-segment tests shecame to emit ldquouprdquo 55 of the time She changed in the di-rection of the random model but overshot the mark Thuswhereas interference degraded chaotic performance sig-nificantly in all three subjects it improved stochastic per-formances in at least two of them These results are con-sistent with the hypothesis that a memory-based processcontrols chaotic variability and that a stochastic processnot dependent on memory controls stochastic variability

Summary of Tripartite TheoryThe following three processes constitute a tripartite

theory of operant variability attention to random events inthe environment (coin toss) memorial processes (chaosradial maze) and stochastic processes (response se-quences and random-number generation with feedback)Random environmental events assure unpredictabilitymdashfor example in making unbiased selections Remember-ing what was done previously will enable one to avoidrepetitions to vary the students called upon in class tovary the locations when looking for a hidden object orto try different approaches when attempting to solve aproblem External aids such as notes to oneself or sci-entific records help to enhance memory capacities

An endogenous stochastic generator provides the thirdand most controversial source of behavioral variability

When an opponent is attempting to predict your behav-ior in a game or at war it may be functional to behavestochastically Similarly operant conditioning of new re-sponses may depend on stochastic variations It is im-possible to prove stochasticity some presently unknowndeterministic system may ultimately explain the resultsHowever the evidence is consistent with the stochastichypothesis and inconsistent with the alternatives For ex-ample memory is not sufficiently powerful to explain alloperant variability memory-interference proceduresleave stochastic responding intact and behaviors corre-late well with those of stochastic models

Both genetics and reinforcement play important rolesin all three processes Reinforcement is always involvedin that learning provides information about when whatwhere and how much to vary Genetic influences areshown in that not all responses are possible and if theyare possible they are not all equiprobable (eg as inNeuringer et al 2001) Stochastic responding may be bi-ased due to experiences and knowledge that is biasesmay be established by reinforcement contingencies or bythe interface between the organismrsquos propensities (or be-havior systems see Timberlake amp Lucas 1989) and theenvironment In the case of random-number generationby human subjects biases may be due to misconceptionsconcerning what ldquorandomrdquo means However some bi-ases can be overcome through feedback and instructions(Lopes amp Oden 1987 Neuringer 1986)

The importance of variations for evolutionary changeis supported by the many sources of genetic variabilityand by the fact that genetic variability is conserved Asimilar argument holds for operant variability That thereare at least three independent sources of operant varia-tions as well as other elicitors such as environmentalnoise and response extinction supports its importancefor normal adaptive functioning

Structure of VariabilityEvery operant ldquoresponserdquo is actually an instance of a

class with each member of the class exerting the sameeffect on the environment (Skinner 1959) When a rat isreinforced for pressing a lever for example the class isdefined by closure of a microswitch and any way of ac-complishing the closure whether with the left paw theright paw or the mouth is as effective in producing re-inforcement as any other If the contingencies of rein-forcement so specify however the response class can beredefined and circumscribed which is another charac-teristic of the operant For example if only right-pawpresses are acceptable members of the reinforced classresponding will tend to occur only with the right paw(Stokes 1995) Of course the capabilities of the organismplay a role for example including ldquotail pressesrdquo in the op-erant class does not mean that the lever would be or couldbe pressed by the ratrsquos tail The structure of the operantclass therefore emerges from the current reinforcementcontingencies and the organismrsquos capability derivedfrom genetics development and prior experiences

696 NEURINGER

Operant variability has structure as well and as for alloperants that structure is due partly to the organism it-self and partly to the reinforcement contingencies Theresearch described throughout this article has providedexamples of the structure of variability generally in theform of sets of four or eight responses across L and Roperanda But the structure of variations can be speci-f ied more precisely For example Mook and her col-leagues (Mook et al 1993 Mook amp Neuringer 1994)reinforced rats for varying four response sequences acrossL and R levers under a lag 1 contingency Unlike in pre-vious studies however the acceptable class was furtherconstrained so that only sequences beginning with two Rresponses (RRLL RRLR RRRL and RRRR) were rein-forced The rats learned to vary their sequences withinthe imposed limitations that is they generally begantheir sequences with two R responses and varied the se-quences within those limitations Similar results wereobtained in the context of a radial-arm maze in whichonly a subset of arms was baited

The precision with which reinforcement contingen-cies can specify class was highlighted in research per-formed by Burns (1997) and Ross and Neuringer (2002)In the latter case college students drew rectangles on acomputer screen with the computerrsquos mouse Each rec-tangle constituted one trial (the screen cleared at the endof each trial) with reinforcement (points) depending onsimultaneous variations along three dimensions of therectangle location on the screen area of the rectangleand shape with the latter defined as the ratio of the rec-tanglersquos height to its width Although the subjects weregiven no instruction other than to ldquogain pointsrdquo theylearned to vary along all three dimensions with levels ofvariability significantly higher than those of yoked controlsubjects who were reinforced independently of variations

A second part of the Ross and Neuringer (2002) studydemonstrated even greater precision of control To gainpoints three new groups of subjects were required to re-peat along one of the dimensions while simultaneouslyvarying along the other two One group was required torepeatedly draw rectangles of approximately the samesize while simultaneously varying location and shapeanother group had to repeat the shape while varying areaand location and the third to repeat location while vary-ing size and shape Verbal instructions were minimal asthey were in the first part of the study Subjects learnedto satisfy these contingencies as well varying two at-tributes while simultaneously repeating one and signif-icant differences were obtained rapidly within 300 trials

These demonstrations raise questions as to how binaryfeedbackmdashreinforcement presented or notmdashcan simul-taneously control independent behavioral dimensionsThe issue is similar to that discussed above concerningsimultaneous reinforcement of variations and selectionsIt is similar also to those issues faced by researchers at-tempting to understand how the nervous system strength-ens particular synapses when many different neurons firesimultaneously and to ldquoallocation of creditrdquo in artificial

learning models For present purposes suffice it to notethe exquisite precision with which reinforcement controlsvariability and repetition

GeneralizationDo animals and people learn to ldquovaryrdquo implying a

generalizable skill or do they learn a specific distributionof specific responses across particular operanda Thequestion of generalization has been asked of many otherbehavioral competencies such as imitation When chil-dren learn to imitate do they learn stimulusndashresponseassociations (if mother moves her right hand the childlearns to move his right hand) or do they learn moregenerally to ldquoimitaterdquo Evidence indicates that reinforce-ment of imitative instances leads to generalized imitation(see eg Baer amp Sherman 1964 Bandura 1986) A sim-ilar question has been raised when pigeons rats or chil-dren learn to choose a stimulus that matches a previouslyshown sample For example if a pigeon is shown a reddisk as a sample it must then choose red when given achoice between that and blue Does the bird learn to matchIn this case the answer appears to depend on the train-ing procedures If trained with few stimuli such as redgreen and blue then the pigeon learns simple associa-tions (if red sample choose red if blue sample chooseblue) but if hundreds of different stimuli (pictures) areemployed during training then generalized ldquomatchingrdquois learned (Wright 1997) In each of these examples thelearning of a general competency or strategy was shownby transfer-of-training tests What was learned with oneset of stimuli or in one situation transferred to new stim-uli and situations We turn now to the question of whetherlearning to vary transfers

After the subjects in Maltzmanrsquos (1960) experimentdescribed above learned to generate remote verbal as-sociates they were asked to devise unusual uses of com-mon objects In comparison with those in the controlgroup these subjects generated significantly more un-usual uses indicating transfer of training to a differenttask However transfer was not observed in the oppositedirection from unusual uses to verbal associates

Holman et al (1977) using a procedure similar to thatof the Goetz and Baer (1973) research described aboverewarded preschool children verbally for novel drawingsand found increased novelty Unrewarded block construc-tions were also monitored and weak evidence was ob-tained for transfer of novel responding from rewardeddrawing to the unrewarded constructions In a second ex-periment preschoolers engaged in four independent taskspainting pen drawing wood block construction and Legoconstruction When the children were reinforced for vary-ing their pen drawings the novelty of paintings increasedalong with the pen drawings indicating transfer from onetask to the other However transfer to the topographicallydifferent block- or Lego-building tasks was not observed

Eisenberger and Armeli (1997) gave monetary rewardsto one group of fifth- and sixth-grade students the vari-ability group for describing unusual uses of common ob-

OPERANT VARIABILITY 697

jects A second group the repeat group received thesame rewards for describing common uses of the sameobjects In a later test all of the children were asked todraw pictures incorporating a circle and the group that hadpreviously been reinforced for varied responses drewmore unusual (or lower frequency) pictures than did therepeat group Thus there was transfer of novelty trainingfrom one domain to another with additional studies in theseries showing similar effects (Eisenberger Armeli ampPretz 1998 Eisenberger Haskins amp Gambleton 1999Eisenberger amp Selbst 1994) However Eisenberger et al(1999) failed to replicate the transfer-of-training effectThe variability and repeat groups responded similarly dur-ing the test phase In this study separate experimenterstrained and tested the subjects thereby avoiding possibleexperimenter-expectancy effects whereas the same ex-perimenter did both training and testing in Eisenbergerrsquosother studies Across the series of experiments there weremany other cases in which transfer was not observed andmore research is necessary to explain why

In Lee et alrsquos (in press) study with autistic individualsdescribed above after variable answers were trained toquestions verbal variability transferred to new experi-menters and new contexts

Arnesenrsquos (2000) results described above are consis-tent with transfer of training in an animal model Rats inthe Var group were trained to interact variably with oneset of objects When later tested in a new situation withnew objects the Var animals showed (in comparisonwith controls) higher levels of exploration manipulationof the novel objects and discovery of hidden reinforcers

Much of the evidence reviewed in this section there-fore indicates transfer of training and supports a generalldquovaryrdquo competency However there remain unresolvedquestions the most important of which is the extent towhich transfer occurs Note also a methodological pointMany of the studies tested whether transfer occurs whenthe contingencies are neutral with respect to the to-be-transferred behavior Holman et al (1977) is a case inpoint Variations were reinforced in one domain draw-ing and the question was whether variations wouldoccur in another domainmdashblock constructionmdashin whichreinforcement did not depend on varying A negativef indingmdashin this case absence of transfermdashindicatesnothing about the ability to generalize but only aboutthe tendency to do so given the absence of a reason (orcontingency) A better test would be to compare experi-mental and control performances when variability in thenew task is in fact functional as in Arnesen (2000)

SummaryBefore variability can be reinforced it must occur for

reasons other than reinforcement Some of the reasonsinclude variability inherent in all behavioral phenomenaldquoaccidentalrdquo environmental events that are unrelated toparticular behaviors and withholding of reinforcementTransfer from other situations may serve a similar functionFor example if an organism can discriminate shaping-

of-new-response periods from others then it may learngenerally to ldquovaryrdquo during shaping episodes After rein-forcement makes initial contact with response variationsthere are at least three behavioral strategies that can beemployed in order to maintain functionally high levelsof variability respond based on external random eventssuch as the toss of a coin avoid repetitions by remem-bering what responses had previously been made and re-spond stochastically Evidence for an endogenous sto-chastic process was seen in the similarity of responsedistributions to those generated by stochastic sourcesand by findings that interfering with memory for priorresponses either improves or leaves unaffected stochas-tic behaviors Stochastic responding as well as othertypes of variability-generating strategies depend uponcontingencies to define the required level of variabilityand the set from which variations emerge

IMPLICATIONS

One implication of reinforced variability is that oper-ant responses may differ in the extent to which their vari-ability is affected by reinforcement contingenciesmdashthatis operants may be ordered along a continuum of sensi-tivity of variability to reinforcement Another is that re-inforced variability causes us to question the generalityof some basic goals of behavioral psychology includingprediction and control A third is that reinforced vari-ability may help us to provide a scientifically useful wayto describe voluntary action In particular the variabilityof all voluntary behaviors can readily be changed mo-ment to moment by discriminative stimuli and conse-quences Implications for everyday life also will be noted

Operant ContinuumAs was indicated above the operant is composed of

instances that is the operant is a class of many differentresponses all functionally equivalent but often topo-graphically distinct Skinner suggested and I hypothe-size that instances within the class emerge stochastically(see Moxley 1997 for a discussion of Skinnerrsquos posi-tion) Although reinforcement contingencies and contextinfluence the individual probabilities and their distribu-tions emission is stochastic nonetheless The particularmember of the operant class generally cannot be predictedwith a high degree of certainty For example a particu-lar rat may be more likely to press the lever with its leftpaw than its right but left- and right-paw responses areemitted probabilistically This class-nature of the operantsuggests one way to conceptualize operant variability interms of size of class and within-classes probability dis-tributions High variability implies large classes equalprobability distributions or both low variability impliesthe opposite

Variability may be modified more readily in some op-erants than in others implying that operant variability isnot an all-or-none phenomenon but rather a continuumwith reinforced variations in variability (RVV) differing

698 NEURINGER

for different operants RVV might best be understood byanalogy to the concepts of speed and acceleration Bothvariability and speed range from low to high RVV isanalogous to acceleration the rate of change in speedHighly sensitive RVV is indicated by the possibility ofreinforcing many different levels of variability from rep-etition to random-like responding and by the fact thatchanges in level occur readily and rapidly

Different operants manifest differences in RVV Forexample variability of chain pulls changed more thanthat of leverpresses when variability was reinforced inrats (Morgan amp Neuringer 1990) Some operant-like re-sponses may show relatively low RVV For example fir-ing rates increase in single hippocampal nerve cellswhen dopaminergic substances are contingent upon theresponse (Stein Xue amp Belluzzi 1994) suggesting thatnerve f iring is controlled by reinforcement To datehowever there is no evidence that variability of nervefiring is similarly controlled Other cases might includereinforcement of nondiscriminable (subconscious) re-sponses such as Hefferline and Pererarsquos (1963) reinforce-ment of subaware thumb twitches and biofeedback ofmuscle and cardiovascular activity In these cases rein-forcement may modify performance but it is not knownwhether variability is also sensitive to reinforcement andif so to what degree Human verbal variability may bethe most readily manipulable of all behaviors the rangeof predictable to unpredictable verbal responses beingimmense with extraordinary sensitivity to context andconsequence RVV may also differ across organisms (per-haps from simple to complex) individuals (perhaps asan index of intelligence or learning ability) and physio-logical and psychological states (WKY vs SHR depres-sion and autism)

RVV may help to distinguish operant responses fromother superficially similar cases of selection by conse-quences That is the ldquooperantrdquo may also best be concep-tualized as a continuum with RVV an important deter-miner of position along the operant continuum Oneexample of selection by consequences but low ldquooperant-nessrdquo is genetic change Variability is essential for evolu-tionary change of course and levels of genetic variabil-ity are sometimes affected by environmental conditions(Moxon 1997) but there are few examples of levels ofvariability being influenced by selection pressures Thatis the source of much genetic variability is thought to berandom not influenced by current pressures and needsThus although selected by consequences genetic changemay be low along the operant continuum Other exam-ples of operant-like phenomena for which variability although important may be relatively insensitive to con-sequencesmdashvariability in these cases is elicited not se-lectedmdashinclude the development of the central nervoussystem and immune system responses (Hull et al 2001)In both of these cases change depends upon selection byconsequences but it is not clear whether variability canalso be selected Most commonly studied operant re-sponses on the other hand show high sensitivity of vari-

ability to reinforcing events But to reiterate the mainpoint of this section operants differ in terms of RVV andtherefore in terms of their ldquooperantnessrdquo from low sensi-tivity (possibly responses of single nerve cells) to extra-ordinarily high sensitivity (verbal behavior) The operantnature of variability as well as that of every other oper-ant dimension is best conceptualized as a continuum

Prediction and ControlSmith (1992) described Skinnerian operant condition-

ing theory as Baconian in nature a primary goal of Skin-nerian behaviorists being to control behavior and hecontrasted this with the more Aristotelian goal of de-scription Skinner and his students (eg Zuriff 1985) in-deed emphasized prediction and control Many of thestudies reviewed above however involved reinforce-ment procedures that increase variability and conse-quently decrease predictability It is not only ignorancetherefore that restricts prediction and control as Skin-ner suggested but reinforcement itself Operant meth-ods and theories when applied to variability lead togreater emphasis on description with control over pre-cise response characteristics possible under some butnot all conditions

In summary an operant is a class of responses underthe control of contingencies of reinforcement The classcan be controlled and predicted given knowledge ofconditioning history Indeed the size of the class andpossibly the distribution of within-classes probabilities orlevel of variability can also be controlled and predictedBut to the extent that instances emerge stochastically aparticular instance cannot be predicted The instant re-sponse can be explained as a member of a particularclass but prediction of the instance will be diff icultsometimes to the extreme This point will be developedin the next section

Voluntary BehaviorI hypothesize that voluntary behaviors are operants for

which RVVs are highly sensitive to influence by conse-quences One way to test this hypothesis would be togather empirical data concerning how voluntary versusinvoluntary actions are classified by human observers Asfar as I know there are no psychophysical data concern-ing this question but the hypothesis implies that highsensitivity of variability to operant contingencies RVVswill be present in all actions perceived to be voluntary

This operant-variability theory of voluntary action isbased on the supposition of two attributes First volun-tary acts must be potentially explainable (given suff i-cient knowledge) on the basis of genetics conditioninghistories current contingencies anticipated conse-quences intentions goals and the like That is volun-tary behaviors are lawful The second attribute is that thevoluntary act must be potentially unpredictable That isvoluntary acts are ldquofreerdquo of control by identifiable stim-uli If behavior is perceived necessarily to result frommdashto be predicted on the basis ofmdashexternal events includ-

OPERANT VARIABILITY 699

ing reinforcement contingencies it will not be perceivedas voluntary ldquoVoluntaryrdquo implies potential indepen-dence from stimulusndashresponse or responsendashreinforcerdetermination According to this theory both explain-ability and unpredictability are necessary attributes ofvoluntary acts

How though can behaviors be simultaneously ex-plainable and unpredictable Reinforcement contingen-cies play two essential roles Reinforcement (togetherwith genes prior experiences and current environment)helps to determine the set of possible behaviors fromwhich a voluntary act emergesmdashthat is the operant classReinforcement (together with the other contributorsspecified above) also helps to determine the level of vari-ability within the class Thus both instances comprisinga class and level of variability within that class can bepredictedmdashagain given sufficient knowledge Howeverthe individual response is at least sometimes (or relatively)unpredictable The reason is that the operant class fromwhich the response stochastically emerges is sometimesextremely large (eg ldquoname a wordrdquo or ldquodo somethingrdquo)and the contingencies sometimes call for equiprobablestochastic emergence (eg ldquofool merdquo) In the latter casea voluntary act may appear to be foolish meaninglessor unwarranted but if it is voluntary it can potentially beexplained (For example in response to the questionldquoWhy did you do that silly actrdquo the answer could beldquoJust to show that you canrsquot predict my behaviorrdquo seeScriven 1965) In these cases prediction of instancesfalls to very low levels indeed and I hypothesize evenan all-knowing observer will be unable to predict thenext response at greater-than-chance levels which aregiven by the size of the operant class and by within-classesprobability distribution Even within small classes (egin the game rockndashscissorsndashpaper in which unpredictabil-ity among the three possible responses is functional) be-havioral stochasticity may lead to unpredictability of theinstant response Note however that although voluntarybehaviors are sometimes unpredictable the opposite isoften the case When contingencies call for highly con-strained particular responses predictions can easily bemade For example if a police officer asks your nameyour response can most likely be predicted Voluntary re-sponses are therefore described as potentially unpre-dictable with variability contingencies engendering dif-ferent levels of unpredictability

According to this theory behaviors low in the sensitivity-of-variability dimension are described as relatively in-voluntary and those that are high in said dimension areperceived as voluntary Low RVV may be shown at anylevel along the variability continuum For example al-ways responding in the same unchanging way will beperceived as involuntary but so will always respondingrandomly (The coin does not voluntarily come up withheads or tails) This point is essential More than 2000years ago Epicurus argued that occasional randomswerves of atoms provided the source of voluntary be-havior in an otherwise determined universe but al-

though this conjecture was prescient it was not quitecorrect ldquoFree-willedrdquo or voluntary action whether it isbased on random swerves or on quantum mechanics isnot explained by variability per se but by functionalchanges in levels of variabilitymdashthat is by rapid andprecise changes in response to environmental demandsResponses can be placed along a voluntaryndashinvoluntarycontinuum isomorphic to the operantndashrespondent con-tinuum with voluntary behavior being functional pre-cisely in the same way as operant behavior is ThusSkinnerrsquos (1974) suggestion that voluntary behavior isoperant behavior was almost correct Only those oper-ants for which variability is highly sensitive to conse-quences will be perceived as voluntary

One additional example might help Operant classes(sets of potential responses) are created ldquoon linerdquo as we goabout our daily activities As we walk from car to officewe tend to walk in a characteristic way take the samepath with tightly constrained operant classes followingone after the other in rapid succession Class sizes aresmall and within-classes probability distributions arehighly peaked Predictions of next responses by a knowl-edgeable observer can readily be made However classsizes can be increased and probability distributions flat-tened we can walk in quite strange ways indeed dancealong a new path sing hand candy to passers-by Strangebehavior indeed but made possible by the ability of op-erant sets to be modified moment to moment not only interms of class definition but in terms of class size lev-els of within-classes variations and so on The same istrue for conversation Moment-to-moment changes inverbal class definition size and predictability of within-classes instances are hallmarks of voluntary behavior

GenerativityRelated to the definition of the operant and to our discus-

sion of voluntary action is that fact that operant behaviormdashleverpressing no less than languagemdashis by its naturegenerative This claim follows from the functional na-ture of the class of responses (all members of the classare effective in producing an outcome) and the stochas-tic emission of instances From these flow never-before-emitted instances patterns and sequences (see Epstein1996) Operants especially those high on the RVV con-tinuum are inherently generative

Self-ExperimentationVariations systematic or otherwise are a necessary part

of scientific work Neuringer (1981) and Roberts and Neu-ringer (1998) described how variations can be brought to bear explicitly on everyday life in the form of self-experiments Individuals vary aspects of their lives suchas the foods they eat social contacts and methods ofmemorizing they keep records concerning the effects ofthese manipulations and form hypotheses concerningcontrolling variables Using self-experimental methodsindividuals can learn how what when where and howmuch to vary Explicitly varying onersquos everyday behav-

700 NEURINGER

iors and thoughts may help to engender what Langer(1989 2000) has referred to as mindful livingmdashthat isnonautomatic voluntary engagement with beneficialeffects in different areas including learning The impor-tant point here is that variability can be voluntarily cho-sen maintained and modified in everyday life and thatexplicit variations may have important consequences

Implications for SocietyConsideration should be given to the importance of re-

inforcement contingencies to support variability in so-cial milieus A basic point made throughout the presentarticle is that merely permitting behavior to vary doesnot necessarily engender or maximize variability (as isshown by Yoke and Prob schedules) Analogously merelypermitting varied social behavior as in a free or laissez-faire society may not suffice to produce the desired di-versity If the results shown for individual organismscan provide useful information for groups (see eg Grottamp Neuringer 1974) then when variations are desiredsocieties might reward individuals for varying not sim-ply permit it

SummaryOperants differ in the extent to which reinforcers in-

fluence levels of variation When variability is highlysensitive to reinforcement the operant may be perceivedto be voluntary Voluntary responses have two attributesThey are functional implying that the class of responsesmdashthe operantmdashcan be predicted given sufficient knowl-edge But instances emerge stochastically implyingsome degree of unpredictability Because reinforcementof variability can generate extremely large classes withstochastic emergence of instances it suggests limitationsof the two basic goals of Skinnerian behavioral theorymdashnamely prediction and control Operant variability also hasimplications for self-experimentation and for society

RELATED AREAS

Operant variability may be important in many otherareas and I will briefly review some of these

Negative reinforcement To my knowledge there havebeen no published studies in which variability is nega-tively reinforced High variability might sometimesavoid or reduce a punisher such as loss of goods ormoney but at other times low variability could be func-tional Effects of positive and negative reinforcement onvariability may have different evolutionary bases onederived from increasing probabilities of locating sourcesof food or mates the other from avoiding confrontationsor predation (Driver amp Humphries 1988) One questionconcerns how quickly variability can be changed by neg-ative as opposed to positive reinforcement There isconsiderable evidence that loss functions differ fromgain functions (Kahneman amp Tversky 1984) possiblyindicating that positive reinforcement-of-variabilityfunctions will differ from negative ones as well

Resistance to change In some of the studies describedabove variability appeared to be more resistant tochange than repetitions were for example alcohol hadlittle effect on variable operant responses but interferedwith repeated ones Doughty and Lattal (2001) tested thegenerality of these f indings using four-response se-quences with pigeons a threshold variability contin-gency in the Var component of a multiple schedule andLRLR in the Rep component Doughty and Lattal chal-lenged stability of responding by prefeeding the birds inone experiment and providing reinforcers independentlyof responding in the other (see Nevin 1992) Operantvariability was again more resistant to change than wasoperant repetition Thus reinforcement of variabilitymight increase the stability of responding (that is its re-sistance to change) a seemingly paradoxical findingHowever Mechner (1992) suggested that this is the casefor complex skills such as piano performance Varyingpractice routines will make the performance relativelyinsensitive to unforeseen changes in the environmentCare must be taken however when variation is comparedwith repetition because they have different measuresAlso the results reported by Doughty and Lattal werebased on the performance of three pigeons and were notconsistent throughout the experiment The hypothesiswarrants further study

Attraction In some bird species complex male songsattract females (for a review see Catchpole amp Slater1995) There is evidence that male song complexity isunder the discriminative stimulus control of a receptive fe-male (Searcy amp Yasukawa 1990) That explicit reinforce-ment may play a role in such song complexity is suggestedby the Manabe et al (1997) study described above An in-teresting question for further research is the extent towhich operant contingencies contribute to the attractioncreated by behavioral variability in these and other species

Discrimination of variability Animals and people canuse the variability of a stimulus array or entropy as a dis-criminative stimulus (Young amp Wasserman 2001) Whatis the relationship between perception of variability onthe one hand and emission of variable responses on theother Do individuals attempt to match their own levelsof variation to those of others in their environments whichwould involve both perception and emission How mightdiscrimination of onersquos own level of variability influenceonersquos ability to modify such levels and do influences thatobscure discriminations therefore affect ability to change(see eg van der Meer van der Weel amp Lee 1995)Hunziker et al (2002) report that although variabilitywas successfully reinforced their human subjects couldnot correctly describe the contingencies of reinforce-ment a finding similar to those of informal observationsin other studies (eg Neuringer Deiss amp Imig 2000)but whether subjects could discriminate variability levelsrather than contingencies was not studied

Attention and awareness When attention is subject tointerference or is withdrawn from the task at hand abilityto vary is degraded (eg Baddeley Emslie Kolodny amp

OPERANT VARIABILITY 701

Duncan 1998 Evans amp Graham 1980 Neuringer amp Voss2002) It is unclear at present how attention influencesthe different processes underlying operant variabilityand in particular memorial and stochastic processesRich areas for additional study are the relationships be-tween attention awareness and consciousness on theone hand and operant variability on the other Voluntaryaction and consciousness are often related (see egLibet 1999)

Concepts Concepts are thought by many to be sets ofinstances (much like operant sets) whose structures aregraded meaning that there are more or less typical in-stances of such concepts as birds (robin is typical ostrichis not) Barsalou (1987) showed that the gradation ofthese cognitive structures is manipulable or as he putsit not stable A number of variables influence structureincluding context individual differences and points ofview Barsalou argued that these structures are createdas the need arisesmdashthat is on line by the thinking indi-vidual The structure of concepts may parallel that of op-erants and in both cases levels of within-structures di-versity or variability may be modified by consequencesOperant variability may be found in the cognitive as wellas in the behavioral domain

Play and humor Highly variable behaviors are ob-served when animals and people especially children en-gage in play and humor (see eg Fagen 1981) The re-sult of play may be acquisition of new responses (seeSutton-Smith 1975) Although not currently needed re-sponses learned during play may provide the source ofvariations for successful shaping of behaviors at anothertime This is quite analogous to the role of conserved ge-netic variants which are passed along to progeny but donot currently serve any identifiable role Play may alsoinvolve learning how to vary operantly

Reinforcement by variability The self-maintainingnature of play suggests that engaging in variable behav-iors may itself be reinforcing Indirect support comesfrom findings that varied stimulation can serve as a re-inforcer and that stimulus repetition becomes aversive(Fiske amp Maddi 1961) Possibly also related are findingsthat varying reinforcers and permitting choice among re-inforcers increase their value (Bowman Piazza FisherHagopian amp Kogan 1997) and that permitting choicesamong varied responses is also preferred to demandingfixed responses (Catania 1980)

Foraging Two issues regarding variability are presentin the foraging literature One has to do with risk prefer-ence versus avoidance (see eg Shettleworth 1998) Asecond has to do with search strategies Most reportedcases of foraging involve systematic search patterns(Stephens amp Krebs 1986) However search patterns aresometimes stochastic or highly variable (see eg Peter-son 1997) One question is whether some species can varybetween these two strategies and the challenge is to iden-tify the pressures leading to and the advantages of each

Animal training Wilkes (1996) has applied operantvariability techniques in developing training proceduresfor pet dogs More generally reinforcement of variabil-

ity may enable the gaining of control over behaviors ofanimals and as has been suggested above may be an im-portant part of the process of shaping

Models of learning Reinforced variations in variabil-ity may be important when human and animal learningis being modeled whether the models are mathematicalin form computer based or machine based (Blumberget al 2002 Brooks 1999 Sutton amp Barto 1998)

CONCLUDING COMMENT

Genes interact with environments to produce diverseanatomical physiological and behavioral phenotypesGenes and environments also provide for another level ofdiversitymdashnamely operant variability There are manysources of operant variability including random environ-mental events memory for past responses and stochasticgeneration each of which ultimately derives from ge-netic and environmental influences To the extent thatoperant variability is based on a stochastic source how-ever the behavioral results cannot be predicted fromknowledge of these influences That is organisms haveevolved to behave unpredictably The positing of sto-chasticity to explain behavior has a history extending atleast as far back as Epicurus with major contributors in-cluding Gustav Fechner (see Heidelberger 1987) EgonBrunswik (see Gigerenzer 1987) Skinner (1959) andSuppes (1984) Knowledge of the stochastic nature ofthe operant and of how it is controlled leads to a worldview that differs from genetic and environmental deter-minism with roots in the works of the scholars just men-tioned At the scientific level this view leads to the ex-perimental analysis of consequence-influenced variabilityAt the philosophical level it leads to explanations of vol-untary action At the personal level it can lead to change

REFERENCES

Amabile T M (1983) The social psychology of creativity New YorkSpringer-Verlag

Antonitis J J (1951) Response variability in the white rat duringconditioning extinction and reconditioning Journal of Experimen-tal Psychology 42 273-281

Archer J amp Birke L (1983) Exploration in animals and humansCambridge Van Nostrand Reinhold

Arnesen E M (2000) Reinforcement of object manipulation in-creases discovery Unpublished undergraduate thesis Reed College

Baddeley A D (1966) The capacity for generating information byrandomization Quarterly Journal of Experimental Psychology 18119-129

Baddeley A [D] Emslie H Kolodny J amp Duncan J (1998)Random generation and the executive control of working memoryQuarterly Journal of Experimental Psychology 51A 819-852

Baer D M amp Sherman J A (1964) Reinforcement control of gen-eralized imitation in young children Journal of Experimental ChildPsychology 1 37-49

Baker M J amp Koegel R L (1999) Transforming children withautismrsquos ritualistic behaviors into positive play interactions with sib-lings Paper presented at the meeting of the California Association ofBehavior Analysis San Francisco

Balsam P D Deich J D Ohyama T amp Stokes P D (1998) Ori-gins of new behavior In W OrsquoDonohue (Ed) Learning and behav-ior therapy (pp 403-420) Boston Allyn amp Bacon

Balsam P D Paterniti A Zechowy K amp Stokes P D (2002)

OPERANT VARIABILITY 679

of required variability It may also be maintained muchlonger because as will be seen shortly extinction weakensresponding

ChoiceThe frequency of one response relative to others or

response probability is used as a measure of choice orpreference Choices are found to match the probabilitiesof their consequences For example relative frequenciesof pecking one of two keys approximate relative fre-

quencies of reinforcement Neuringer (1992) showedthat the same matching relationship also governs choicesby pigeons to vary or repeat The more variations werereinforced the more the pigeons chose to vary and sim-ilarly for repetitions Each trial consisted of four L andR responses and reinforcements for varying and for re-peating were systematically manipulated The Var con-tingency was met whenever the current sequence dif-fered from each of the previous three sequencesmdasha lag 3criterion The Rep contingency was satisfied if the cur-

Figure 3 The left column shows the probabilities ( y-axis) that each of five rats met a variability contin-gency (solid circles) versus a repetition contingency (open circles) when they were being reinforced for vary-ing (Hopson Burt amp Neuringer 2002) Along the x-axis are trials since the mixed schedule (no cues pro-vided to indicate the component) had switched into the variability-reinforcing component The right columnshows the corresponding data for performances when reinforcement was contingent upon repetitions thex-axis here indicating trials since the unmarked switch into the repetition-reinforcing component

680 NEURINGER

rent sequence repeated at least one of the previous threeOn an average of once every 30 sec (VI 30 sec) the com-puter probabilistically primed reinforcement for Var orRep the prime being maintained until the reinforcementhad been collected Thus if the computer primed Var re-inforcement was delivered whenever the lag 3 variabilitycontingency was next met Similarly if Rep had beenprimed the next Rep sequence was reinforced Proba-bilities of Var and Rep reinforcements were systemati-cally manipulated the two always summing to 10 and amatching-type relationship was obtained (Figure 4panel A) The relationship was more complex than sim-ple matching possibly because if the pigeons chose tovary by responding stochastically there was some prob-ability due to chance that the current sequence wouldrepeat a previous sequence This was shown by a com-parison of pigeon performances to the performance of acomputer-simulated stochastic model programmed tomatch its Var choices to the probability of Var reinforce-ments The pigeonsrsquo choice distributions were found tobe the same as the modelrsquos (Figure 4 panel B) Thuschoices to vary or repeat were governed by relative fre-quencies of reinforcement An animal can choose whetherwhen and how much to vary a result consistent withthose described above from Blough (1966) Machado(1989) and Grunow and Neuringer (2002) Choices ofwhether or not to vary appear to be governed similarly toall other operant choices

These f indings may have important applicationsWhen an individual repeatedly responds in a way that isnonfunctional reinforcement of choices to vary mayhelp the individual to emerge from the ineffective pat-tern This can be true for those experiencing everydayproblems but feeling helpless to change and it may beespecially important for those experiencing depressionor who respond in otherwise abnormally stereotypedmanners such as those with autism These topics will beconsidered in a later section

ExtinctionAs was indicated above extinction is often found to

increase variability but almost all studies of this effectinvolve responses that had previously been repetitiveAntonitis (1951) for example reinforced rats for pokingtheir noses anywhere along a 50-cm horizontal openingAlthough the exact location of pokes did not matter withcontinued training responses became limited to one or afew locations along the strip When reinforcement waswithheld variability of location increased (see also Eck-erman amp Lanson 1969) Variability produced by extinc-tion has been reported for many other response dimen-sions as well including force (Notterman amp Mintz1965) number (Mechner 1958) topography (Stokes1995) and sequence (Balsam Paterniti Zechowy ampStokes 2002 Mechner Hyten Field amp Madden 1997)

The issue is complicated however by other studiesthat show that previously reinforced responses and re-sponse patterns are maintained intact during and afterextinction That is extinction does not break down

learned patterns For example when Schwartz (1981) re-inforced pigeons for any sequence of four L and four Rresponses each bird developed a dominant sequence asdescribed above and these sequences were maintainedduring extinction Despite decreasing response ratesduring extinction response patterns were retained Sim-ilar maintenance of response structure has been ob-served during extinction following reinforcement of

Figure 4 Average performance across a group of pigeons inNeuringerrsquos (1992) study of choices to vary or repeat (A) Loga-rithms of the ratio of number of times the pigeons met the Varcontingency divided by the times they met the Rep contingencyas a function of the analogous numbers of reinforcements re-ceived for each Points represent group averages in two separateexperiments Also shown is the least squares best-fitting line(B) Changes in U-value an index of response variability as afunction of the obtained percentage of reinforcements for vary-ing The solid line in panel B represents the performance of acomputer-based model that matched its choices to vary (vs re-peat) to its obtained relative frequency of reinforcement for vary-ing From ldquoChoosing to vary and repeatrdquo by A Neuringer 1992Psychological Science 3 p 248 Copyright 1992 by the AmericanPsychological Society Reprinted with permission

OPERANT VARIABILITY 681

choices (Myerson amp Hale 1988) responding underfixed-interval contingencies (Machado amp Cevik 1998)visual discriminations (Nevin 1967) and matching tosample (Cumming Berryman Cohen amp Lanson 1967)How can stability during extinction be reconciled withincreased variability

Neuringer Kornell and Olufs (2001) wrestled withthis question in a study of extinction following rein-forcement under three different contingencies An oper-ant chamber contained two levers left (L) and right (R)on either side of a reinforcement dispenser and a key (K)in the rear of the chamber distant from the dispenserOne group of rats Rep was reinforced for repeating afixed sequence of responses LKR A second group Varwas reinforced for variations across the three operandaunder a threshold contingency Reinforcement for a thirdgroup Yoke was matched to those of the Var animalsand was therefore independent of the particular sequencesBecause of the unusual configuration of operandaskewed distributions of sequence probabilities were ob-served in all of the groups an outcome that was antici-pated and desired in order to document possible in-creases in variability during extinctionmdasha flattening ofthe distributionmdashas well as possible decreases That isduring the reinforcement phase the Rep group came torepeat the LKR sequence with high probability the Yokegroup tended to emit the easiest sequences such as LLLand RRR but distributed their responses over more se-quences than the Rep group did and the Var group dis-tributed most widely although because of differences inthe operanda and in their locations some sequences weremore preferred than others and in particular sequencesending with a K response were least likely (the key beingfarthest from the reinforcement dispenser) Thus thecontingencies resulted in three different distributions orhierarchies of sequence probabilities very steep forRep intermediate for Yoke and flattest for Var

The main finding was that withholding reinforcementhad similar effects in the three groups In each responserates fell to low levels across the four sessions of extinc-tion Furthermore sequences that were most frequentduring the reinforcement phase continued to be most fre-quent throughout extinction A slight flattening of thedistributions was observed however with the lowestprobability sequences increasing relative to the rein-forcement phase Thus because the overall hierarchy ofsequence frequencies was maintained the results indi-cated stability but because the distributions were some-what flattened the results also showed slightly increasedvariability It is as if the strategy during extinction wereto keep doing what had worked in the past but occa-sionally to do something very different This compli-cated story integrates two effects of extinction increasedvariability and maintained stability Most importantly forthe present discussion extinction had similar effects onvariable yoked and repetitive operants

One additional comparison is of interest Recall thatGrunow and Neuringer (2002) found that when contin-

gencies demanded high variability obtained levels ofvariability decreased slightly as reinforcement frequen-cies were lowered This result is opposite to the Vargrouprsquos slight increase in variability just described underextinction conditions The reason may be that extinction-induced variability is short lived lasting only as long asresponding continues perhaps for hours or a few ses-sions at most whereas the variability engendered by theVI 5 intermittent schedule of reinforcement in Grunowand Neuringer can be maintained indefinitely The de-crease in Grunow and Neuringerrsquos operant variabilitymay be due to the weakening of the reinforcement effect(ie reinforcement is less frequent) whereas the in-crease following extinction may be a short-term elicitedeffect This distinction between reinforced and elicitedvariability is important and will be discussed further inthe next section

Operant and Respondent InteractionsElicited or what is sometimes referred to as respondent

effects occur when environmental changes that affectbehavior do not depend on (are not caused by) the behav-ior Pavlovian conditioning being the most commonlystudied example Operant effects are observed when a re-inforcer depends on the response to produce it Althoughfor historical reasons respondent and operant effects areoften studied separately most behaviors result from acombination or interaction of the two Variability is nodifferent and we turn to an example of such an interaction

As reinforcement is approached under FI or fixed-ratio (FR) schedules response rates are commonly ob-served to increase Similarly when reinforcement de-pends on accuracy as in discrimination learning ormatching to sample it too is found to increase within theFI or the FR (Nelson 1978 Nevin 1967) Cherot Jonesand Neuringer (1996) asked whether the same patternheld for reinforced variability Does operant variabilityincrease as reinforcement for varying is approachedThe question may be related to such real-world cases aswhen a composer is rushing to complete a compositionbefore a scheduled performance or when a programmeris trying to solve a problem before an upcoming meetingThese activities involve at least to some extent non-repetitive original or variable thought and behaviorThe question is whether variability increases with prox-imity to or anticipation of reinforcement for varying

Cherot et al (1996) employed trials consisting of fourresponses across L and R operanda and a lag 4 variabil-ity contingency with rats in one experiment and pigeonsin another Unlike in any of the cases described so farhowever the lag-variability contingency had to be satis-fied four times in order for a single reinforcer to be ob-tained With ldquordquo indicating successful completion of theVar contingency ldquo2rdquo indicating failure to meet that con-tingency and PELLET indicating the food reinforcer anexample follows LRLL RRRL LRLL2 LRLL2RLLR RRRL2 LRRR+PELLET Thus food reinforce-ment was provided only after the subject had satisfied

682 NEURINGER

the variability contingency four times FR 4 The resultswere that as the food reinforcer was approached (ie asprogress was made through the ratio) the subjects wereless and less successful at varying The probability ofmeeting the variability contingency in the first trial afterreinforcement was higher than it was in the second whichin turn was higher than that in the third trial and so onFor comparison a Rep group was reinforced for repeatingsequences once again with reinforcement only after foursuccessful repetitions Probability of repeating increasedwith proximity to reinforcement a finding consistentwith the literature but opposite that for the variabilitygroup Thus approach to reinforcement facilitated oper-ant repetition but interfered with operant variabilitySimilar variability-interfering effects have been docu-mented for within-trials responses The probability that aresponse repeats the just prior response increases within atrial even when sequence variability is reinforced (McElroyamp Neuringer 1990 Neuringer 1991) To return to the ex-ample of a composer rushing to complete her composi-tion she may indeed have increasing difficulty respond-ing creatively under the pressure of a deadline On theother hand the soloist may find it easier to repeat thecorrect performance during practice periods as the con-cert nears How can these results be reconciled with themany findings that variability can be reinforced

An additional result of Cherot et al (1996) provides ananswer The Var animals responded much more variablyoverall than did the Rep animals Although variability inthe Var groups decreased with approach to the reinforceroverall variability was always much higher when it was re-inforced (Var group) than when it was not (Rep group)Reinforcement therefore exerts two influences Overallvariability is elevated sometimes to the highest levelswhen it is reinforced but approach to (and possibly focuson) the reinforcers constrains or lowers variability Sim-ilar reinforcement-produced interference effects had beenreported previously for example when raccoons are re-inforced for dropping a ldquocoinrdquo into a slot the reinforce-ment results in the animalsrsquo holding and rubbing the coinrather than dropping it (Breland amp Breland 1961) Theoverall enhancement of variability was of much greatermagnitude in Cherot et al than the decrease with approachwas although both effects were statistically significant(The notes created by the composer under pressure ofdeadline will be less readily predicted than will thoseplayed by the performer practicing a well-known piece)This study highlights the importance of interactions be-tween operant effects of reinforcement and elicited effectsproduced by anticipation of reinforcement and the resultsmay be relevant to the debate concerning possible detri-mental effects of reinforcement on creative behaviors (seethe discussion below and Neuringer in press)

SummaryReinforcement controls behavioral variability as it

does other operant dimensions Levels of variability areprecisely governed by schedule contingencies and dis-

criminative stimuli Choices to vary or repeat match rel-ative frequencies of reinforcement Extinction increasesvariability slightly whereas the hierarchies of sequenceprobabilities are retained and operant and respondenteffects interact The evidence shows that behavioral vari-ability is an operant controlled by its consequences

GENERALITY AND PARAMETERS

Many different methods measures species and re-sponses have been used in the study of operant variabil-ity and this diversity supports the generality of the phe-nomenon As is the case for repeated operants howeverthere are parametric constraints This section briefly re-views the generality and some of the parameters studiedto date

GeneralityMethods Throughout this section ldquoresponserdquo will

signify a response unit including both an individual re-sponse and a multiresponse sequence Each of the meth-ods to be described increases or maintains variability byreinforcing it

Novel response procedures reinforce responses upontheir first occurrence ldquoeverrdquo (Pryor et al 1969) or upontheir first occurrence within a session (Goetz amp Baer1973) Radial-arm maze procedures also reinforce ini-tial (within a given session) entries into arms of a mazeand result in a distribution of choices across the availableoptions the sequence or pattern of choices often beingquite variable

Lag procedures reinforce a response if it has not oc-curred across a given number of previous trials (Page ampNeuringer 1985) For example under lag 5 the currentsequence is reinforced only if it has not occurred duringany of the previous five trials A variant is the recurrence-time procedure which specifies a minimum number of in-tervening trials before a previously emitted response canagain be reinforced Machado (1989) combined recurrencetime with a percentile-reinforcement contingency to gen-erate high levels of variability (see also Machado 1992)Percentile-reinforcement contingencies base the criterionfor reinforcement on the subjectrsquos own performance over aprevious set of responses (see Galbicka 1994) For exam-ple a sequence will be reinforced only if it has not oc-curred over the previous n trials the value of n continuallychanging so that 50 of the trials are reinforced

Least-frequent response procedures (Blough 1966Schoenfeld Harris amp Farmer 1966 Shimp 1967) rein-force responses that are currently least frequent across amoving window

Threshold procedures reinforce responses whose rela-tive frequencies fall below a given value Because thistype of procedure was used in the research describedabove and in many other more recent studies details willbe provided In Denney and Neuringer (1998) which willserve as an example trials consisted of four responses byrats across L and R levers A running tally was kept of the

OPERANT VARIABILITY 683

number of occurrences of each of the 16 possible se-quences LLLL LLLR LLRL and so on If the relativefrequency of the current sequencemdashthe number of its oc-currences divided by the sum of all 16 sequencesmdashwasbelow a specified threshold value in this case 05 the ratwas rewarded Recently emitted sequences contributedmore to the maintained tally than did those in the distantpast because after each reinforcement all 16 counterswere multiplied by a weighting coefficient in this case95 Therefore the influence of a particular trial decreasedexponentially with successive reinforcements Thresholdprocedures differ from Bloughrsquos (1966) least-frequentprocedure in that more than one sequence (or response)can be reinforced at any given timemdashnamely all those se-quences that have relative frequencies below the thresholdvalue The procedure differs from lag in permitting agiven sequence to be reinforced many times in successionif the associated relative frequency is sufficiently low

The frequency dependence procedure (Machado 19921993) is a variant of threshold in which the probability ofreinforcement is a continuous function of relative re-sponse frequency The more frequent a response is thelower the probability is that it will be reinforced

Statistical feedback procedures provide subjects withcomparisons between their performances and that of amodel Neuringer (1986) showed human subjects feedbackfrom 10 statistical tests thereby enabling them to comparetheir own distributions of responses to that of a randommodel Neuringer and Voss (1993) provided feedbackbased on a chaotic model The goal in each of these caseswas for the subjects to learn to approximate the model andthe results of these experiments will be discussed below

Differential reinforcement of switches versus repetitionsbalance a subjectrsquos tendency to engage in each of these pat-terns Rather than reinforcement for infrequent responsesor random-like distributions reinforcement is providedsometimes for repeating a response and other times forswitching from one response to another In the two-operandum case a switch is L followed by R or R by La repetition is L then L or R then R Bryant and Church(1974) found that stochastic-like responding was gener-ated by rats when switches were reinforced with a 75probability and repetitions with a 25 probability that isthe rats had a natural tendency to repeat at least in theenvironment of this study Machado (1997) reinforcedpigeons for eight pecks across two keys if the sequenceof pecks contained at least one (one group of birds) ortwo (another group) switches Although the birds couldhave satisfied these contingencies by repeating a singlesequencemdashin the ldquoone switchrdquo condition for examplerepetition of LRRRRRR would produce reinforcement on100 of the trialsmdashthey tended to vary their sequencesBarba and Hunziker (2002) showed however that vari-ability was higher when it was directly reinforced thanwhen reinforcement was based on proportions of switchesa finding consistent with that of Machado (1997)

The evidence from each of these methods supports thehypothesis that variability can be reinforced

Measures Variability must be interpreted with refer-ence to a particular measure The reason is that there canbe high variability (and low predictability) at one level ofanalysis but order (and certainty of prediction) at a dif-ferent level For example the sequence 1 1 2 2 3 3 44 5 5 1 1 2 2 has the same variance (a commonlyemployed measure of variability) as 2 3 2 1 5 1 4 41 1 3 2 5 2 but an analysis of conditional proba-bilities in which the unit contains pairs of instances in-dicates that the two sequences have different levels ofvariability In general high variability observed with onemeasure does not rule out the possibility of order ob-served with another and vice versa That is why evalua-tions of random-number generators often employ manydifferent measures and indeed there are hundreds of dif-ferent measures of the randomness of a sequence (Knuth1969) All behavioral studies make simplifying assump-tions and rely on one measure or at the most a few

U-value has been the most commonly employed mea-sure of operant variability (Evans amp Graham 1980Machado 1989 Page amp Neuringer 1985 Stokes 1995)U-value measures the distribution of relative frequen-cies or probabilities of a set of responses For a set of16 possible responses i = 1ndash16 U-value is given by

where RFi = the relative frequency (or probability) ofeach of the 16 responses U-values approach 10 whenrelative frequencies approach equality as would be ex-pected over the long run from a random generator andthey approach 00 when a single instance is repeated

Other measures include the percentage of trials or fre-quency at which variability contingencies are met (Page ampNeuringer 1985) percentages of alternations versus stays(Machado 1992) conditional probabilities of responses(Machado 1992) the number or frequency of novel re-sponses (Goetz amp Baer 1973 Schwartz 1982) the num-ber or frequency of different responses (Machado 1997Schwartz 1982) Markov analyses (Machado 1992) andstatistical tests of distributions of responses (Neuringer1986) All of the measures employed to date indicate thatvariability can be reinforced

Responses and species Generality has further beendemonstrated by the use of many different responses andspecies These include Siamese fighting fish (Betta splen-dens) swimming in quadrants of a tank (Roots 2000)young chicks pecking keys (Neuringer 2002) pigeonspecking keys (Page amp Neuringer 1985) budgerigarssinging songs (Manabe Staddon amp Cleaveland 1997)rats pressing levers (Van Hest van Haaren amp van de Poll1989) porpoises swimming and jumping (Pryor et al1969) rhesus monkeys pressing levers (Sydney Reisbickpersonal communication) children drawing and build-ing with blocks (Holman Goetz amp Baer 1977) andhuman adults and children pressing buttons (Saldana ampNeuringer 1998 Stokes amp Balsam 2001) There havebeen few attempts to compare species or responses but

- ( )[ ] ( )=aring RF RFi ii

log log 2 21

1616

684 NEURINGER

in one study lowest baseline variability was observedwhen rats pulled trapezes intermediate levels were ob-served when they pushed keys and highest levels wereobserved when they pressed levers (Morgan amp Neuringer1990) with variability increasing in all cases when it wasreinforced In another study operant variability wascompared across rats and people with similar effects ob-served in the two species (Neuringer Deiss amp Imig 2000)

ParametersWe turn now to some parameters studied to date Many

other parameters as yet unstudied are no doubt also im-portantmdashfor example motivation and types of reinforcers

Age Older people (see eg Van der Linden Beertenamp Pesenti 1998) and older rats (see eg Lamberty ampGower 1990) respond less variably than their youngercounterparts when variable responding is reinforced or re-quested but in most cases it is not clear to what extent thevariability is elicited as opposed to reinforced Neuringerand Huntley (1992) compared reinforced variability in2-month-old (young) and 10-month-old (mature) Long-Evans rats Under Yoke contingencies the young rats re-sponded significantly more variably (two levers four re-sponses per trial) than the mature rats which constitutesan elicited effect When reinforcement was contingenton variability (lag 4) variability increased significantlyin both groups thereby overcoming the initial differencecorrelated with age Thus explicitly reinforcing vari-ability in older animals caused them to respond no dif-ferently than younger animals did a finding potentiallyrelevant to real-world human aging However when Wells(1999) compared 8-month-old (mature) to 24-month-old(old) rats although the mature rats responded more vari-ably than the old rats did under Yoke contingencies thedifference was not statistically significant Additionalstudies are needed especially of whether reinforcingvariable behaviors in older populations can facilitatelearning and performance

Gender Male and female rats are equally sensitive toreinforcement of variability (Neuringer amp Huntley 1992van Hest et al 1989) However there was a tendency(not statistically significant) for males to respond over-all more variably than females did and because thesestudies did not control for differences in size and weightof subjects or for other aspects of motivation additionalstudies are necessary

Stage of training Stokes and Balsam (2001) reporteddifferences in variability as a function of whether variabil-ity was initially reinforced early or later in training (seealso Gottlieb 1992) Human subjects generated sequencesof 10 L and R responses per trial on a computer keyboardunder lag 0 and lag 25 contingencies Under lag 0 everytrial ended with reinforcement under lag 25 the currentsequence had to differ from each of the previous 25 Whenlag 25 was experienced at the very beginning of traininglevels of variability at the end of training (when lag 0 wasoperative) were lower than they were when lag 25 was ex-perienced after 50 trials of initial lag 0 These results were

interpreted to suggest that there is an ldquooptimal period forsetting variability levelsrdquo (Stokes amp Balsam 2001 p 181)but that general conclusion needs further testing and anumber of questions need to be answered One has to dowith the contribution of reinforcement frequency Underlag 0 every sequence leads to reinforcement but rein-forcement is intermittent under lag 25 Another has to dowith whether learning to vary differs from learning to repeatwith respect to optimal periods (see eg Bouton 1994)

Related to stage of training are effects of prior experi-ence Hunziker Lee Ferreira da Silva and Caramori(2002) showed that for human subjects if Var contingen-cies were experienced just prior to Yoke contingencieslevels of response variation during the Yoke phase werehigher than they would have been if the Var contingencieshad not been previously experienced The time delay be-tween Var and Yoke experiences mattered with a long in-terval (3 months) producing only a small effect Saldanaand Neuringer (1998) also found effects of prior Var train-ing on performances in Yoke (increased variability inYoke) but the opposite did not hold that is training underYoke had relatively little effect on Var Hunziker Caramorida Silva and Barba (1998) reported similar effects withrats Thus there may be an asymmetry Reinforcement forvarying has relatively strong influences but reinforce-ment without regard to variations does not adversely affectlearning to vary These different effects are not surprisinggiven that Yoke contingencies are permissivemdashreinforce-ment is given whether or not responses varymdashwhereascontingencies under Var can be quite demanding How-ever the results motivate a number of additional ques-tions One is whether effects of Var training differ fromthose observed when repetitions are reinforced Another iswhether direct reinforcement of variable behaviors canhelp animals or people to overcome the detrimental effectsof prior noncontingent experiences such as those that pro-duce learned helplessness (Seligman 1975)

Environmental constraints Stokes (1995) arguedalso that variability increases when behavior is con-strained The term ldquoconstraintrdquo is ambiguous because allreinforcement contingencies exert a form of constraintIn one study Stokes (1995) compared different strate-gies of shaping leverpresses Rats in one group were re-inforced initially for pressing the lever with only theright paw (right only) and eventually for any press (withright or left paw or with any other part of the body) Theorder of training was reversed for a second group ldquoanyrdquofirst and then right only Behavior sequences were morevariable in the right-only-first group leading Stokes(1995) to conclude that early constraints lead to highvariability (see also Stokes Mechner amp Balsam 1999)That is an important hypothesis with many implicationsbut controls are needed before the results can be attrib-uted to the contingencies For example ldquoright onlyrdquo andldquoanyrdquo differed in terms of reinforcement distributions aswell as in the precision of the reinforcement operationthe ldquoanyrdquo reinforcers were provided immediately and au-tomatically by the programming equipment whereas in

OPERANT VARIABILITY 685

the right-only condition a human observer had to decidewhether a response had occurred and then present the re-inforcer necessarily leading to longer and more variedlatencies and a less consistent contingency Furthermorethere are many cases in which increasing constraints suchas those due to a decrease in the number of responses pertrial (Page amp Neuringer 1985) or the requirement of ex-actly four responses on each of two keys (Schwartz 1980)results in decreased variability Thus the contributionsof response-definition constraints and environmentallimitations must be explored further

SummaryThe generality of operant variability is shown by the

robustness of the phenomenon across procedures mea-sures responses and species As with other operant phe-nomena parameters influence the effects

FUNCTIONS

All operants including variability are functional Op-erant responses produce reinforcing consequences Be-cause other behavioral competencies depend on it oper-ant variability is functional in a broader sense as wellSome of these functions will now be described

Operant ConditioningThe potential of a reinforcer to change behavior is lim-

ited by the range of available variations (Hull Langmanamp Glenn 2001 Skinner 1981) and thus whatever pro-duces those variations contributes to the conditioning ofnew operants Much work has been done to identifysources of variation (Balsam Deich Ohyama amp Stokes1998 Segal 1972 Staddon amp Simmelhag 1971) butNeuringer (1993) was first to show that the same rein-forcers that strengthen individual responses can also pro-duce the necessary variability and that these two functionsof reinforcement may work simultaneously Implied isthat when an operant response is being conditioned re-inforcement works not only to strengthen an individualresponse but at the same time to engender the variabilityfrom which the response emerges (That is variability isnot solely a baseline phenomenon elicited or noise)Concurrent reinforcement of variations and individualtarget sequences will be the topic of this section

Pigeons in one experiment and rats in another were re-warded for varying among 16 possible response sequences(four responses per trial L and R operanda lag contin-gencies Neuringer 1993) Concurrently one of the 16designated as the target sequence was reinforced when-ever it occurred Thus reinforcements were given bothfor sequence variations and for a single selected targeta reinforced-variation-and-selection (RVS) procedureWhen the target was ldquoeasyrdquo (eg LLLL) it emergedrapidly from the baseline variations Given that the tar-get was always reinforced reinforcers were obtained morefrequently when the subjects did not vary and only re-peated the target When the target was ldquodifficultrdquo (eg

LLLR) it was not learned As in the choice experimentdescribed above (Neuringer 1992) the relative valuesor difficulties of target versus variations governed theprobability that the target would be learned

However even difficult targets could be conditionedunder the RVS procedure as was shown by another ex-periment in the same series (Neuringer 1993) Sequencevariations were again reinforced concurrently with a dif-ficult target RRLR but now the frequency of reinforce-ments for varying was systematically decreased Satis-fying the variability contingency was intermittentlyreinforced according to a VI schedule and over sessionsreinforcement for varying became less and less frequentThis technique resulted in acquisition of even the mostdifficult target sequence whereas a control group rein-forced for the target alone (without the concurrent rein-forcement for variations) never learned

Reinforcers were obtained more frequently in the RVScondition than in the control condition (since the RVSsubjects were rewarded both for varying and for emittingthe target sequence) and the additional reinforcers mighthave generated greater motivation to respond (Killeen1981) Thus motivation not the direct reinforcement ofvariability might have been responsible for the RVS ef-fect To test this possibility Neuringer Deiss and Olson(2000) reinforced three groups of rats for emitting a par-ticularly difficult target sequence LLRLR The experi-mental group was reinforced concurrently for sequencevariability on the average of once per minute (VI 1 min)The target-only control group received no reinforce-ments other than for the LLRLR target A third groupwas yoked to the experimental animalsmdashreinforced onceper minute for any sequence variability not requiredmdashand as for the other two groups whenever the target oc-curred If the additional reinforcement was responsiblefor the facilitated learning under RVS then because theexperimental and yoked groups received equal reinforce-ments they should have learned equally well but thatwas not the result Only the experimental group learnedto emit the LLRLR target The yoked controls respondedat high rates throughout the experiment but they did notvary sufficiently to make contact with the target contin-gency and never learned The target-only control groupreceived insufficient reinforcement and their responsesextinguished Therefore the added reinforcers helped tomaintain motivation as suggested above but reinforce-ment of variations was necessary for learning of a diffi-cult sequence The RVS procedure maintains both highmotivation and high variation and the two together re-sult in the acquisition of difficult-to-learn responses

Grunow and Neuringer (2002 Experiment 2) studiedone possibly important parameter of the RVS proceduremdashnamely the level of variability When a difficult target re-sponse is being trained does it matter whether reinforcedvariability is low intermediate or high As describedabove each of four groups of rats was reinforced for vari-able response sequences the first group for high vari-ability the fourth group for low variability and the other

686 NEURINGER

two for intermediate levels In other ways the procedurewas similar to that of the Neuringer Deiss and Olson(2000) RVS contingencies just described with a singledifficult target sequence always reinforced The resultswere that the higher the level of reinforced baseline vari-ability the more quickly the target was learned Thusvariability levels indeed mattered and if the just-re-ported effect is found to be general it will be most help-ful to reinforce high variability when difficult-to-learnnew responses are being shaped Other parameters re-quiring study are class definitionsmdashwhich responses areincluded in the class of acceptable variationsmdashand num-ber of instances within the class

One additional effect should be noted When varia-tions among a set of responses are reinforced if one vari-ant is selected for extinction (no longer reinforced) thefrequency of that instance decreases Neuringer (1993)showed this for pigeons responding under a lag-variabilityschedule When one of the 16 sequences was no longerreinforced (with no other change to indicate its status) thefrequency of that sequence decreased Thus reinforce-ment of variations provides the baseline to select for (iestrengthen) and select against (ie weaken) individualinstances

These findings have potentially important implicationsFirst as has been indicated the same reinforcers thatstrengthen particular instances can also reinforce varia-tions That a binary eventmdashreinforce or not reinforcemdashsimultaneously affects variations and selected instancesindicates both the power of reinforcement and a concep-tual problem How reinforcers exert such diverse effectsis a question to which we will return

Second shaping procedures may implicitly reinforcebaseline variations An approximation to a goal responseis reinforced for a period of time but eventually rein-forcement is withheld until a response closer to the goalis obtained It is commonly thought that withholding re-inforcement or mini-extinction induces the necessaryvariations for shaping to proceed as has been indicatedabove An alternative suggested by the present studiesis that organisms learn both to vary (within a set) and to re-peat (selected instances) until one instance is always rein-forced That is the process of shaping might involve re-inforcement both of variations (generally done implicitly)and of targeted instances If so a useful contributor tothe shaping process might be explicit reinforcement ofvariations

Third reinforcement of variations could facilitate ac-quisition of other competencies such as cognitive andmotor skills Siegler (1996) has shown in an important se-ries of studies that childrenrsquos acquisition of complex cog-nitive skills is correlated with high variability Anotherexample is that when motor skills are practiced in a vari-able manner high quality of performance results (Ma-noel amp Connolly 1995 1997 Mechner 1992 Schmidtamp Lee 1999) However it is rare that variability in thesedomains is directly reinforced as part of the training oreducational process

Fourth the RVS procedure might be used in therapeu-tic settings to modify maladaptive and self-injurioushabits Therapists sometimes employ noncontingent re-inforcement procedures to weaken undesirable behaviors(Vollmer Iwata Zarcone Smith amp Mazaleski 1993)The reasoning is that many maladaptive behaviors aremaintained by unintended consequences and that non-contingent rewards may compete with those conse-quences thereby lowering the strength of the unwantedhabit For example when a staff personrsquos attention toself-injurious behavior helps to maintain it providing at-tention noncontingently may decrease the strength of theunwanted target behavior However the RVS proceduremight accomplish the same goal more effectively andwithout undesirable side effects (Vollmer RingdahlRoane amp Marcus 1997) Noncontingent rewards some-times result in unwanted ldquosuperstitiousrdquo behaviors andthat would be avoided under RVS Furthermore RVSwould help to provide the set of baseline behaviors nec-essary to establish desirable new responses

Finally RVS procedures may help to gain control overand eventually to weaken other undesirable target behav-iors The target can be incorporated within a set of moredesirable responsesmdasha new operant class established withthe target as one membermdashand variable instances rein-forced Once variable responding is successfully main-tained including the undesirable target response the lattercan be weakened by selectively withholding reinforcement

Summary The conditioning of operant responses de-pends on both baseline variations and selection of suc-cessive approximations to a goal response The same re-inforcers used to select the approximations can reinforcebaseline variations thereby defining the precise set ofappropriate variants and the desired levels of variationSuch concurrent reinforcement of variations and selec-tions may facilitate the acquisition of difficult-to-learnoperants and selective withholding of reinforcementmay weaken undesirable instances

Creativity and Problem SolvingBehaving in a novel unusual or nonrepeating manner is

part of creative activities (Campbell 1960) Novelty andvariation may arise through combinations of previouslylearned behaviors (Epstein 1996) transformations of thesebehaviors (Boden 1994) direct reinforcement of variabil-ity as described above or some combination thereof Ofcourse neither novelty nor variability alone suffices forcreative outputmdashthe throw of the dice or the toss of acoin are not creative nor are haphazard strokes of a paint-brush on a canvasmdashbut without variation creativity islow Whether the variability required for creativity canbe reinforced is relevant to whether creativity itself canbe reinforced The latter question has led to contentiousdebate with some arguing that reinforcement facilitatescreativity (Cameron amp Pierce 1994 Eisenberger ampCameron 1996 Stokes 2001 Winston amp Baker 1985)and others focusing on detrimental effects of reinforce-ment (Amabile 1983 Deci Koestner amp Ryan 1999)

OPERANT VARIABILITY 687

The debate goes well beyond the present discussion butsome of the research described above is relevant Firstreinforcement contingent on variability generates higherlevels of behavioral variability than does reinforcementcontingent simply on responding as has been shown bythe many studies described above in which Var contin-gencies were compared to Yoke contingencies Similarlyprecise reinforcement contingencies matter for creativity(Cameron amp Pierce 1994 Lepper amp Henderlong 2000)If any product rather than only creative ones is rein-forced creativity tends to fall (just as variability decreasesunder yoke conditions) Second approach to or antici-pation of reinforcement interferes with operant variabil-ity (Cherot et al 1996 McElroy amp Neuringer 1990Neuringer 1991) which again might be related to effectson creativity when an individual focuses on the rein-forcer Note however that although variability decreaseswith proximity to the reinforcer overall levels of vari-ability are much higher when variability is reinforced thanwhen it is not The same may be true for creativity aswell anticipation of reward may interfere with creativeoutput to some extent but at the same time high creativ-ity may depend on its reinforcement

Operant variability may be important for problemsolving as well Here too there is disagreement as towhether training is helpful or not Early studies by Maier(1933) indicated that it is and these found support inlater work (eg Lung amp Dominowski 1985) Somestudies showed that those individuals most likely toldquobreak the setrdquo are also most likely to solve a problem(Dover amp Shore 1991) However a number of studiesfound that practice had no influence (Duncan 1961)

Arnesen (2000) explored a related phenomenon witha rat model One group Var was reinforced for variableinteractions with novel objects and another group Yokewas reinforced independently of such interactions EachVar rat received training with 12 different objects oneper session For example in the presence of a soup canthe rat was reinforced the first time it entered the canbut thereafter a different response was required such asrolling it climbing on it and so forth This procedure issimilar to that of Goetz and Baerrsquos (1973) described aboveexcept that objects changed from session to session Ratsin the Yoke control group had identical experiences in anadjoining identical chamber that contained identical ob-jects but now rewards were not contingent on object ma-nipulation After this training the rats were tested in adifferent room with 30 never-before-experienced ob-jects scattered on the floor Hidden in each was a smallpiece of food When compared to their Yoke controls theVar animals were more active during this test exploredmore manipulated objects more and most importantlydiscovered significantly more food pellets Another con-trol group received no preliminary training with objectsand their performance in the test session did not differfrom that of the yoked controls Therefore when ratswere reinforced for variable interactions with objectsthe result was an increased tendency to explore manip-ulate and discover new sources of reinforcement It

would be of considerable interest to determine whethera similar effect occurs with children

Assessment and Treatment of PsychopathologiesAbnormal levels of behavioral variability characterize

some human psychopathologies as has been shown byclinical observations as well as by research with humansubjects and animal models Few studies have attemptedto reinforce variability directly in such problem popula-tions The main question is whether reinforcement canmodify abnormal levels of variability in the direction ofnormalcy and whether such changes might have associ-ated beneficial effects

Autism Stereotyped responding characterizes indi-viduals with autism (Hertzig amp Shapiro 1990) For ex-ample Baron-Cohen (1992) asked children with autismand others in a control group to hide a penny in one handso that the experimenter could not guess which hand heldthe penny Those with autism were more likely thanthose without to generate a simple predictable patternsuch as repeatedly switching back and forth from left toright hands Such repetitive responding can be maladap-tive For example Mullins and Rincover (1985) askedchildren with and without autism to pick one of five cardsFood was occasionally found in a cup behind the cardEach of the five cards was a discriminative stimulus fora different schedule of reinforcement including contin-uous reinforcement (CRF every choice reinforced) FR 2(fixed ratio 2 in which every second choice was rein-forced) FR 4 (every fourth choice reinforced) FR 7 andFR 11 Control participants sampled all five alternativesand quickly learned to choose the most frequently rein-forced CRF card Autistic children sampled only a lim-ited number of cards and therefore often preferred anonoptimal alternative Thus children with autism tendto respond less variably than normal children and thiscan result in failure to adapt

Miller and Neuringer (2000) asked whether direct re-inforcement could influence the variability of individu-als with autism Sequences of responses on two largebuttons were first reinforced probabilistically and inde-pendently of variability (Prob phase) after which rein-forcement was contingent upon satisfying a thresholdvariability contingency (Var phase) The autistic partic-ipantsrsquo variability was significantly lower than that ofmatched controls during both Prob and Var phases butwhen variability was reinforced its levels increased sig-nificantly in both groups This is an important findingbecause it suggests that direct reinforcement of variabil-ity may help to modify some of the behavioral stereo-types that are characteristic of autism Lee McComasand Jawor (in press) offer support for this hypothesisThree individuals with autism were reinforced for variedverbal responses to social questions the responses had tobe appropriate given the context Variability increased intwo of the three individuals In a related procedureBaker and Koegel (1999) reinforced children with autismfor varied social interaction in a game-playing contextAlthough the reinforcement was permission to engage in

688 NEURINGER

the stereotyped behaviors preferred by the child the re-sult was increased social interactions both during theperiod of the experimental manipulation and in 1- and2-month follow-ups These three studies taken togethersuggest that reinforcement of varied behaviors may facil-itate modification of nonfunctional ritualistic and stereo-typed behaviors

Depression Depression provides another example ofmaladaptive consequences of low variability Lapp Mar-inier and Pihl (1982) observed that depressed womenproduced fewer alternative solutions to hypotheticalproblems than did nondepressed women Channon andBaker (1996) found that attempts by moderately de-pressed college students to identify faults in a series ofinterconnected circuits were less varied and less success-ful that were those of nondepressed controls HorneEvans and Orne (1982) found that when depressed pa-tients were asked to generate a random sequence of num-bers their responses were less variable than were thoseof controls These experimental results are consistentwith clinical observations

Can depressed individuals be rewarded for increasingvariation Hopkinson and Neuringer (in press) dividedcollege students into mildly depressed and nondepressedgroups on the basis of the CES-D scale a paper-and-pencil self-evaluative index of depression In the first oftwo phases sequences of responses on a computer key-board were reinforced independently of variability (Prob)following which only highly variable sequences were rein-forced (Var) Under Prob depressed students respondedsignificantly less variably than did the controls in agree-ment with previous findings Under Var variability in-creased significantly in both groups Indeed by the endof the experiment depressed and nondepressed subjectswere responding equally variably (Instructions were alsoeffective in increasing variability in the depressed par-ticipants) Thus even in a simple computer-game envi-ronment mildly depressed subjects manifested lowervariability than controls did but direct reinforcementmodified their behavior to control levels Can reinforce-ment of variability be used in real-world conditions tohelp depressed individuals An affirmative answer comesfrom cognitive behavior modification procedures thatwork to increase alternative solutions to problems (Beck1976) A more general conclusion from these studies ofautistic and mildly depressed individuals is that direct-reinforcement-of-variability procedures may benefitthose who manifest abnormally low levels of variability

Attention Deficit Hyperactivity Disorder (ADHD)ADHD may provide an example of the opposite stateabnormally high levels of variability (Barkley 1990)Wultz Sagvolden Moser and Moser (1990) proposedthat the spontaneously hypertensive rat (SHR) may serveas a good animal model of ADHD the SHR shows manyof the behavioral characteristics of that disorder includ-ing hyperactivity poor learning ability poor ability totolerate delayed reinforcement and poor self-control

The next study to be described asked about operant vari-ability in SHRs

Mook Jeffrey and Neuringer (1993) compared vari-ability of SHRs and of a control strain Wystar Kyoto(WKY) under two conditions Prob and Var The mainresult was that SHRs tended to respond variably whetheror not the contingencies required it consistently with theresults from the human ADHD literature When vari-ability was reinforced the two strains responded at ap-proximately equally high levels of variation These re-sults were confirmed in two other studies (HunzikerSaldana amp Neuringer 1996 Mook amp Neuringer 1994)Thus control rats varied only when the contingencies re-quired variations whereas SHRs varied whether or notthe contingencies required it As might be expected fromthese results when reinforcement depended on repeti-tions the SHRs performed poorly (Mook et al 1993)

Another similarity between SHRs and ADHD humanswas seen when amphetamine a drug that has effects simi-lar to those of Ritalin often prescribed for ADHD was ad-ministered (Mook amp Neuringer 1994) When repetitionswere required for reinforcement amphetamine improvedperformances by the SHRs so that they were as accurate assaline-injected WKYs a finding again consistent withthose in the ADHD literature These studies support SHRsas a model of ADHD and suggest that additional attemptsshould be made to assess whether or not individuals withADHD are less able to change levels of variability thanare controls Although as indicated above there is someevidence in support of this hypothesis (Barkley 1990)when Saldana and Neuringer (1998) directly comparedhuman ADHD and control participantsrsquo operant variabil-ity within the context of a computer game no differenceswere observed Most of those with ADHD were main-tained on Ritalin and other drugs however a fact unfor-tunately not documented in the original article

Drug EffectsAlthough many drugs are known to influence behavioral

variability (Brugger 1997) rarely have attempts beenmade to analyze explain or identify the sources of theeffects As will be discussed below operant variabilitycan result from the memory-based process of remember-ing previous responses and then behaving differently orcan result from responding stochastically a process thatappears to be quite different McElroy and Neuringer(1990) attempted to distinguish the effects of alcohol onmemory from its effects on stochastic processes in astudy in which one group of rats was reinforced for vary-ing four-response sequences under a lag 5 contingency(Var) and another was reinforced for repeating a singlesequence LLRR (Rep) Ethanol injections of 075 and20 gkg affected Var and Rep performances differentlyWhereas the Rep animals were much less able to satisfythe LLRR contingency at both doses the Var rats re-sponded variably whether or not alcohol had been ad-ministered Similar results were obtained by Cohen et al

OPERANT VARIABILITY 689

(1990) with a multiple schedule in which each rat servedas its own control In the same animals and within thesame sessions alcohol degraded Rep performances butleft Var intact Thus alcohol appears to interfere with atask that requires working memory as presumably is thecase for the Rep task but not with one that reinforcesvariability The memory-interfering effect of alcoholmight account for why alcohol also degrades radial-armmaze performance (Devenport 1983)

Alcohol also affects choices to vary or to repeat For ex-ample Gorka (1993) reinforced rats concurrently for re-peating and varying and monitored percentage of choicesas a function of time following injection of ethanol Inthe control sessions when the rats were injected withsaline the probabilities of varying and repeating remainedapproximately constant across the 45-min session witha slight bias in favor of varying After injection with 15gkg of ethanol however there was a significant increasein choices to vary at the beginning of the session andthen a later decrease to below 50 That is alcohol ledinitially to a markedly increased preference for varyingbut later to a preference for repeating (Figure 5) This re-search should be replicated and extended especially be-cause the contingencies ldquopressuredrdquo the animals to re-main at approximately 50 preference but thesuggestion is that alcoholrsquos effects on response variabil-ity may depend in part on the duration of its influence Amore general implication is that drugs may affect re-peated and variable operants differently and that effectsdepend on time following administration

SummaryReinforced variability may contribute to many types

of learning including operant shaping and acquisition ofcognitive and complex motor skills Similarly creativityand problem solving depend on variable responding andin these areas as well explicit reinforcement of variationsmay be functional Abnormal levels of variability aresometimes caused by psychopathologies and drugs andoperant variability may be useful in these areas in twoways first as a way of assessing the variability that is pos-sible and second to modify variability toward more nor-mal levels Studies show that individuals with autism andmild depression vary less than do controls but that ex-plicit reinforcement of variability moves the levels towardsnormalcy Other studies show that drugs such as alcoholmay affect repeated and variable operants differently

EXPLANATIONS

Two questions must be answered in order to explainoperant variability First what are the sources of vari-ability that permit its reinforcement Second how is itmaintained Related questions including how instancesbecome members of a set from which variations emergeand whether operant variability is a generalizable com-petency will also be considered

Origination From Nonoperant SourcesSkinner (1966) reasoned that operant responses orig-

inate from nonoperant sources with reinforcement con-tacting a behavior shaping and strengthening it onlyafter the behavior had occurred for reasons other than re-inforcement To put it simply a behavior must occur be-fore it can be reinforced (Balsam amp Silver 1994) I willconsider three initial sources of variable behavior en-dogenous variability noncontingent environmental eventsand extinction These appear to parallel evolutionaryprocesses and they too will be identified

Endogenous variability All physical phenomenavary and behavior is no exception This fact can providethe opportunity for reinforcement to select particularlevels of variation Examples of spontaneous variabilityinclude spontaneous alternations in mazes (Dember ampRichman 1989) nondifferentially reinforced variationsin operant responses (Guthrie amp Horton 1946) and ex-ploration (Archer amp Birke 1983) Endogenous muta-tions provide a parallel in genetics Base-pair changes inDNA molecules occur spontaneously intron-bearinggenes mix and match segments to encode new proteinsjumping genes introduce new DNA sequences and atthe level of the chromosome random crossings and as-sortment occur as a natural part of reproductive meiosisThese sources of random genetic change occur indepen-dently of external influences

Figure 5 The relative frequency of variations by rats as a func-tion of 5-min blocks of time since the beginning of the sessionThe open circles represent performance after the rats were in-jected with saline indicating a slight tendency to vary more thanrepeat but one that is approximately constant throughout thesession The filled circles represent performance after injectionwith 15 gkg ethanol indicating a much higher likelihood ofvarying soon after the alcohol is injected followed by a decreasein variations to below 50 and then a return to baseline at theend of the session Group averages are shown Data are fromGorka (1993)

690 NEURINGER

Noncontingent events Noncontingent events some-times referred to as chance accidents or luck also affectbehavior One happens upon a particular passage in a bookthat leads to a new thought one happens to sit next to aninteresting woman or man on an airplane which leads tomarriage (Bandura 1982) In science as well as in every-day life accidents or serendipity are an important part ofthe process of discovery (Beveridge 1957) In each of thecases presented above an accidental event leads to new be-haviors that are then strengthened by consequences Thesame is potentially true for variability itself Particular lev-els of variability might occur for adventitious reasons butthen become functionally related to consequences

That organisms are particularly sensitive to such unan-ticipated noncontingent events is shown by two basicprinciples of learning habituation and Pavlovian condi-tioning Organisms attend preferentially to unexpectedevents and stop attending or habituate to repeated onesThe conditioning of new Pavlovian responses dependsaccording to influential theories upon unexpected rein-forcing events (eg Rescorla amp Wagner 1972)

Noncontingent events such as X-rays and other pro-cesses that can act as mutagens play an important role ingenetic variation as well The particular genes affectedand the type of changes are random there is no neces-sary relationship or contingency between the externalevent and the affected gene However sensitivity to mu-tations is not random but has been selected presumablybecause it is functional Thus in both behavior and ge-netics sensitivity to noncontingent events is conserved

Extinction Extinction increases behavioral variabilityat least to some extent as was described above Distance inspace or time from the reinforcer does the same (Cherotet al 1996) A possible parallel at the genetic level in-volves high levels of deprivation at least in simple lifeforms For example when E coli bacteria are deprivedof nutrients unusually high mutation rates are observedThe exact mechanism by which such hypermutations areinduced is not known but they increase the likelihoodthat a population can survive such environmental insults(Moxon 1997) That is deprivation-induced variability isfunctional at the genetic level (The behavioral evidenceregarding long-term deprivation is less clear some stud-ies showing increased variability and others increasedstereotypy eg Carlton 1962 Senkowski Vogel amp Po-zulp 1978)

Endogenous variations external noncontingent eventsand withholding of reinforcement are three sources forthe variability that precedes control by reinforcementBut once control by reinforcement is established a dif-ferent question emerges What behavioral processes areinvolved when variability is maintained as an operantThe sources just discussed generate uncontrolled non-specific and often short-lived variability and cannot ex-plain the precisely controlled operant type

Operant Variability ProcessesThree processes are hypothesized to underlie operant

variability The first involves the use of random environ-

mental events such as the toss of a coin The second in-volves memory for responses such as when an animallearns not to return to previously visited locations Thethird will be referred to as an endogenous stochasticgenerator

Random environmental events Throughout historymany cultures have used random events to influencechoices and decisions such as throws of dice and randomselections of sticks cards bones or organs Today coinflips are used by a referee at the beginning of a game to de-cide which team kicks the ball a computerrsquos random-number generator is employed by scientists to avoid bi-ases in assigning subjects to experimental groups andaleatoric events are sometimes employed in modern artmusic and literature The Dice Man by Rhinehart (1998)provides a fictional example in which the protagonistbored with life writes a number of possible actions onslips of paper and then periodically selects one blindlyand behaves accordingly These examples show that randomexternal events may be used to avoid biases engender un-likely responses and break out of behavioral niches

Memory-based variability Many methods used tostudy operant variability differentially reinforce nonrepe-titions at least over the recent past Therefore one strat-egy that subjects can employ is to cycle through few re-sponses or sequences especially when the variabilitycontingencies are relatively permissive For exampleunder a lag 2 contingency cycling repeatedly throughthree sequences will result in 100 reinforcement Ani-mals and people sometimes do this apparently basingtheir current response on memory for or discriminativecontrol by the just emitted responses One such instancewas shown by Schoenfeld et al (1966) who rewardedrats for IRTs that fell into one of 10 interval classes onlyif the current IRT differed from the interval class of theprevious response The rats came to alternate betweenlong and short IRTs thus behaving systematically to meetthis lenient variability contingency Machado (1993)studied an analogous effect having pigeons peck L and Rkeys under a frequency-dependent variability contin-gency related to Shimprsquos (1967) procedure describedabove When the trial unit consisted of two responses thebirds developed memory-based repetitive sequencessuch as RRLLRRLL and thereby satisfied the variabilitycontingency When the trial unit was three responses inlength the birds did not develop the optimal fixed pat-tern of RRRLRLLLmdashwhich was too complexmdashbut in-stead developed ldquorandom-like behaviorrdquo (Machado1993 p 103) Thus a memory-based strategy was em-ployed when it was possible but when the memory loadbecame too high stochastic responding emerged A sim-ilar pattern was seen when variability of budgerigarsongs was reinforced (Manabe et al 1997) Under lag 1contingencies the birds sang mainly two songs underlag 2 they increased to three songs Only when the lagwas increased to 3 did song diversity increase apprecia-bly in one of the birds In radial-arm mazes as well whencontextual cues are available memory again appears toplay an important role leading to much lower probabil-

OPERANT VARIABILITY 691

ities of repetition than would be expected from a sto-chastic generator (McElroy amp Neuringer 1990)

Note that under lag schedules memory-based strate-gies result in higher reinforcement rates than respondingstochastically does The reason is that a stochastic gen-erator repeats by chance That animals and people oftendo not employ memory-based strategies probably indi-cates the difficulty of remembering long sequences andpossibly other advantages of unpredictable responding

If a memory strategy is the only way to satisfy vari-ability contingencies then it can be used even when thememory requirements are demanding as was shown byNeuringer and Voss (1993) Human subjects were trainedto generate on a computer keyboard a chaotic-like se-quence of responses that matched iterations of the logistic-difference equation

where Rn refers to the nth iteration in a series with R be-tween 00 and 10 and t is a constant between 10 and40 When the t parameter approaches 40 iterations ofthis equation result in very ldquonoisyrdquo sequences As withother chaotic processes however there is an orderlinessunderlying the surface noise In this case if responsesare autocorrelated with Rn drawn as a function of Rn21the resulting function is a parabola In the Neuringer andVoss (1993) study following each response college stu-dents were shown how close their predicted values wereto that from iterations of the logistic-difference model Inone experiment the students made digital predictions inanother they placed a pointer along a horizontal lineand this analogue response was then translated into dig-ital values for computation With training the studentsbecame increasingly adept at responding in chaotic-likefashionmdashthe studentsrsquo predictions matched closely theiterations of the logistic functionmdashand their autocorrela-tions became parabolic The left column in Figure 6 showsperformances by 4 subjects (rows) early in training thatis not well approximated by the logistic-differenceparabola and the right column shows performances at theend of training in which r 2 values were 72 99 98 andapproaching 10 How was this accomplished Sinceeach iteration in the logistic-difference sequence is basedon the prior outputmdashthat is the sequence is generated bya deterministic although nonlinear processmdashit was hy-pothesized that the human subjectsrsquo chaotic-like responseswere also based on memory for prior responses To put itsimply the subjects may have memorized a long series ofpairs in the form ldquoIf the previous response was value Athen the current response must be value Brdquo as Metzger(1994) Ward and West (1994) and Ward (2002) have sug-gested

To test whether subjects in fact relied on memory forprior responses Neuringer and Voss (2002) systemati-cally slowed responding by interposing pausesmdashthat isby requiring an IRT As IRT increased the difference be-tween the subjectsrsquo sequences and the modelrsquos chaotic

Rn n nt R R= -( )- -1 11

Figure 6 Autocorrelations (responses on trial n on the y-axis asa function of n21 on the x-axis) for 4 human subjects at the be-ginning (left column) and end (right column) of training inNeuringer and Voss (1993) Equations for the best-fitting curveand amount of variance accounted for (r2) are shown at the topof each panel To the extent that the circles (responses) approxi-mate the equation y = 40x 2 40x 2 the subjects were respondingin a way that matched the logistic-difference chaotic functionFrom ldquoApproximating Chaotic Behaviorrdquo by A Neuringer andC Voss 1993 Psychological Science 4 p 115 Copyright 1993 bythe American Psychological Society Reprinted with permission

692 NEURINGER

output increased and the underlying parabolic structurewas broken That is when pauses increased subjects wereless and less able to approximate the modelrsquos chaotic dis-tribution (see Figure 7) The important point here is thathighly variable responses which in this case are chaoticin nature can be based on memorial processes

Endogenous stochastic generator The evidence tobe reviewed next indicates that animals and people canrespond stochastically and that this competency is basedneither on chance environmental events nor on memoryfor prior events or responses It is impossible to provethat an endogenous stochastic process underlies operantvariability that would be analogous to proving the nullhypothesis However much evidence is consistent withthe stochastic hypothesis Included are the facts thatmemory limitations do not permit adequate respondingunder many variability contingencies that response dis-tributions match those expected from a stochastic gener-ator that changes in parameter values affect operantvariability as they would a stochastic generator and thatinterference with memory processes leaves stochasti-cally based operant variability unaffected As a wholethe evidence is consistent with an endogenous stochas-tic source of operant variability

Memory limitations One type of evidence is nega-tive in character and is based on the presumed inability ofmemory for prior responses or sequences to satisfy sometypes of variability contingencies Two examples weregiven above (Machado 1993 Manabe et al 1997) andanother was provided by a lag 50 contingency in Pageand Neuringer (1985) In the latter case the sequence in

the current trial (eight responses across L and R keys) wasrequired to differ from those in each of the previous 50 tri-als including at the beginning of a session the last 50trials from the previous session Pigeons responded wellunder these contingencies and in a way similar to that ofa stochastic model It is unlikely that pigeons can remem-ber each of their previous 50 eight-response sequences

Response slowing As was shown above for chaotic-like responding performances are often degraded by de-lays imposed between a stimulus and a response pre-sumably because of limitations in working or short-termmemory (Berryman Cumming amp Nevin 1963 Williams1971) Neuringer (1991) relied on this effect to test whethermemorial processes could explain the generation of vari-able sequences under lag schedules Two groups of ratswere reinforced for L and R leverpresses producing fourresponses per trial One group (Var) was trained under alag 5 contingency and the other (Rep) was trained to asingle sequence of responses LLRR It was assumedthat accurate Rep performance depended on workingmemory After performances had stabilized IRTs weresystematically varied by the imposition of timeouts of05 to 20 sec between consecutive responses in the se-quence During these forced pauses the chamber wasdark and responses were not counted toward completionof the sequence If the animals had used the previous re-sponse(s) as a discriminative cue for the current responsethen performances should have been degraded by theseinterpolated pauses

The results were that for the Rep group as the re-quired pauses increased from 0 to 6 sec percentages ofcorrect sequences were relatively unaffected but at pausesgreater than 6 sec probability of a correct LLRR se-quence fell sharply This pattern of results suggests in-volvement of working memory of prior responses withthe rats able to remember responses within a 6-sec win-dow The Var grouprsquos results were quite different Aspauses increased between 0 and 6 sec the rats were moreand more likely to satisfy the variability contingenciesand with pauses greater than 6 sec percent correct re-mained at a high and constant level Thus the Var grouprsquosresults were opposite to those of the Rep group and op-posite to those predicted from a memory-based strategySimilar results showing that operant variability increasesor is maintained as responding is slowed have been re-ported by Morris (1987) with pigeons and by Baddeley(1966) with random response generation by human sub-jects The results reported above concerning alcoholrsquos ef-fects on variability and repetition were also consistentwith those of Neuringer (1991)

Memory for prior responses may not have controlled theVar grouprsquos performance but why did variability increaseEach of three answers is consistent with a stochastic-generator interpretation Weiss (1964 1965) hypothe-sized that random responding requires that current re-sponses be independent of previous ones memory for orcontrol by prior responses interferes with such indepen-

Figure 7 Approximation to a chaotic sequence (given by delta)as a function of the imposed interresponse times Shown are theaverages of 5 subjects

OPERANT VARIABILITY 693

dence and therefore pauses would be expected to facil-itate variability A second possibility is that at shortIRTs animals tend to repeat responses on the sameoperandum Blough (1966) found this in pigeons and ex-cluded such double pecks from his analyses because theyappeared not to be under the control of reinforcement con-tingencies Morris (1987) also found a tendency for birdsto repeat when no interresponse timeouts were imposedA third hypothesis is that there were two contributors tothe observed variability One was a stochastic processcontrolled by reinforcement and the other elicited vari-ability generated by interposition of pauses The high vari-ability elicited as an effect of slowed responding is ageneral phenomenon supported in many other cases Ac-cording to this interpretation operant variability in the Vargroup was governed by a stochastic-based process operantrepetition in the Rep group was governed by a memory-based process and pauses elicited variability under bothcontingencies Thus Rep performance was interferedwith whereas Var was facilitated Each of these hypothe-ses is consistent with the conclusion that memory for (ordiscriminative control by) prior responses does not con-tribute to and possibly interferes with variable respond-ing when an organism is reinforced for variability andthis in turn is consistent with the stochastic-generatorhypothesis

Stochastic response distributions Other evidence iscorrelational Response distributions are similar to thosefrom a stochastic generator The data from two experi-ments already discussed are shown in Figure 1 In thesecases reinforcement of least frequent IRTs (Blough 1966)and of nonrepetitive choice sequences (Machado 1989)resulted in stochastic-like responding

Neuringer (1986) reported another striking case Highschool and college students generated sequences of 1sand 2s on a computer keyboard with each set of 100 re-sponses constituting one trial A baseline phase lastedfor 60 such trials for a total of 6000 responses duringwhich the students were asked to behave as randomly aspossible as if they were reporting the outcome of a se-ries of coin tosses As in the many previous studies ofthis sort the subjectsrsquo responses differed significantlyfrom those of a stochastic model (Brugger 1997 Wage-naar 1972) During the training phase the subjects re-ceived feedback which enabled them to compare their per-formances with that of a stochastic model first accordingto one statistical test and then according to another untilfollowing each trial feedback was provided from 10 dif-ferent statistics The results were that all of the subjectslearned to approximate the stochastic model accordingto the 10 statistics as well as some (but not all) other sta-tistical tests That is the distributions of the statisticswhich clearly did not match a stochastic model at the be-ginning of training approximated the stochastic modelat the end of it (see Figure 8)

The previous failures of other researchers to obtain ev-idence for random responding had led some to concludethat humans did not have the capacity to behave ran-

domly What would account for the difference betweenthe present positive finding and the many other negativeones Neuringerrsquos (1986) study was the first in which sto-chastic responding was explicitly reinforced in this casewith feedback from 10 statistical tests An analogy mighthelp to explain the difference in procedures and resultsImagine an experimenter walking around a city and ask-ing people if they know what a violin sounds like (analo-gous to asking people whether they know what a series ofcoin tosses or dice throws is like) Most would answeryes If the same individuals were then asked to play theviolin very few if any would succeed The experimentermight conclude after many replications with differentsubjects in different cities that people did not have thecapacity to play the violin The point of course is thatplaying the violin is a skill requiring training and prac-tice and the same may be true for responding randomlyIt is easy to vary but practice and reinforcement may benecessary to approximate a random distribution

Response number Page and Neuringer (1985) testedthe stochastic-generator hypothesis in another way byvarying number of responses per trial with pigeons work-ing under a lag 3 contingency The reasoning was that ifprevious responses served as a discriminative cue forwhat not to domdasha memory-based strategymdashthen perfor-mance should be degraded when the number of re-sponses was increased Eight responses per trial requiresubjects to remember more than four responses The sto-chastic hypothesis predicts the opposite outcome as isdemonstrated by the following example If each trialwere two responses in length with responses determinedby the toss of a coin then the probability that a given trialwould repeat the previous one would be 25 (There arefour possible sequences in the first trial RR RL LRand LL Thus the second trial has a 1-in-4 chance ofmatching the first) If a trial consists of four coin tossesthe probability of a repetition by chance is 0625 or 1 in16 Thus if subjects used a stochastic process to gener-ate Ls and Rs performances should improve with in-creasing responses per trial Results were precisely thosepredicted by the stochastic hypothesis and inconsistentwith a memory strategy Probability of meeting the con-tingencies and therefore probability of reinforcementincreased as responses per trial increased In otherwords eight-response trials were easier for pigeons thanfour-response trials

Response interference Neuringer and Voss (2002)attempted to distinguish between memorial and stochas-tic processes in yet another way They hypothesized thatwhereas chaotic responding was based on memory forprior responses (as was discussed above) responding inNeuringerrsquos (1986) study with feedback from 10 statisti-cal tests indicated involvement of an endogenous sto-chastic generator To show that human subjects could en-gage in both processes at will three college studentswere trained to generate chaotic sequences in one compo-nent of a multiple schedule as in Neuringer and Voss(1993) and to generate stochastic sequences in a sepa-

694 NEURINGER

rate component as in Neuringer (1986) The wordCHAOTICS was on the screen in the first component theword STOCHASTICS was on during the other and thesewords alternated throughout the session After the sub-jects succeeded at each of the component tasks test ses-sions were introduced in which memory for prior re-sponses was subject to interference in both the chaoticsand stochastics components as follows During the

chaotics component four separate chaotic sequenceswere cycled one after the other each cued by differentscreen colors The subjects attempted to approximate achaotic sequence in one color (eg light green) follow-ing which they would have to match a second chaotic se-quence (lavender screen color) and so on the fourchaotic contingencies following one another across thesession Although all four chaotic sequences used the

Figure 8 Performance by one subject at the end of training to respond stochastically in Neuringer(1986) Shown are comparisons on 6 different statistics between the subject (solid lines and Xs) andthe random model (dashed lines and closed circles) At the end of training the subjectrsquos perfor-mance did not differ statistically from that of the random model on these and four additional sta-tistics From ldquoCan people behave lsquorandomlyrsquo The role of feedbackrdquo by A Neuringer 1986 Jour-nal of Experimental Psychology General 115 p 71 Copyright 1986 by the American PsychologicalAssociation Reprinted with permission

OPERANT VARIABILITY 695

same logistic-difference equation that a model did theywere out of phase and independent If chaotic respond-ing is memory based then the four independent se-quences would be expected to interfere with one anotherthat is approximations to the four chaotic models wouldbe lower than when only a single chaotic output was re-quired as in Neuringer and Voss (1993) That was ex-actly the finding Performances by the three subjectswere signif icantly degraded during the four-segmentphase To put it simply chaotic responses interfered withone another

A different pattern of results was seen in the stochasticsportion of the experiment The same interference proce-dure was used with four independent stochastic sequencesrequired in the four different color conditions As inNeuringer (1986) subjects were required to satisfy eightdifferent statistical tests of randomness and to do this in-dependently in each of the four phases (If responses in anytwo or more of the four stochastic phases became corre-lated above a minimal level responses were considered tobe incorrect thereby prohibiting repetition of a single se-quence in two or more of the phases) For one subjectseven of eight statistics were closer to those of a randommodel during this interference phase than they were at theend of the original training with only a single sequenceand for a second subject all eight statistics were closerThe third subject showed closer-to-random approximationunder the four-segment condition on only two of the eightstatistics but most of her changes were also in the directionof a random generator For example if a random generatoremits equal numbers of ldquouprdquo and ldquodownrdquo sequences andthis subject in the baseline phase had emitted only 47ldquouprdquo sequences then during the four-segment tests shecame to emit ldquouprdquo 55 of the time She changed in the di-rection of the random model but overshot the mark Thuswhereas interference degraded chaotic performance sig-nificantly in all three subjects it improved stochastic per-formances in at least two of them These results are con-sistent with the hypothesis that a memory-based processcontrols chaotic variability and that a stochastic processnot dependent on memory controls stochastic variability

Summary of Tripartite TheoryThe following three processes constitute a tripartite

theory of operant variability attention to random events inthe environment (coin toss) memorial processes (chaosradial maze) and stochastic processes (response se-quences and random-number generation with feedback)Random environmental events assure unpredictabilitymdashfor example in making unbiased selections Remember-ing what was done previously will enable one to avoidrepetitions to vary the students called upon in class tovary the locations when looking for a hidden object orto try different approaches when attempting to solve aproblem External aids such as notes to oneself or sci-entific records help to enhance memory capacities

An endogenous stochastic generator provides the thirdand most controversial source of behavioral variability

When an opponent is attempting to predict your behav-ior in a game or at war it may be functional to behavestochastically Similarly operant conditioning of new re-sponses may depend on stochastic variations It is im-possible to prove stochasticity some presently unknowndeterministic system may ultimately explain the resultsHowever the evidence is consistent with the stochastichypothesis and inconsistent with the alternatives For ex-ample memory is not sufficiently powerful to explain alloperant variability memory-interference proceduresleave stochastic responding intact and behaviors corre-late well with those of stochastic models

Both genetics and reinforcement play important rolesin all three processes Reinforcement is always involvedin that learning provides information about when whatwhere and how much to vary Genetic influences areshown in that not all responses are possible and if theyare possible they are not all equiprobable (eg as inNeuringer et al 2001) Stochastic responding may be bi-ased due to experiences and knowledge that is biasesmay be established by reinforcement contingencies or bythe interface between the organismrsquos propensities (or be-havior systems see Timberlake amp Lucas 1989) and theenvironment In the case of random-number generationby human subjects biases may be due to misconceptionsconcerning what ldquorandomrdquo means However some bi-ases can be overcome through feedback and instructions(Lopes amp Oden 1987 Neuringer 1986)

The importance of variations for evolutionary changeis supported by the many sources of genetic variabilityand by the fact that genetic variability is conserved Asimilar argument holds for operant variability That thereare at least three independent sources of operant varia-tions as well as other elicitors such as environmentalnoise and response extinction supports its importancefor normal adaptive functioning

Structure of VariabilityEvery operant ldquoresponserdquo is actually an instance of a

class with each member of the class exerting the sameeffect on the environment (Skinner 1959) When a rat isreinforced for pressing a lever for example the class isdefined by closure of a microswitch and any way of ac-complishing the closure whether with the left paw theright paw or the mouth is as effective in producing re-inforcement as any other If the contingencies of rein-forcement so specify however the response class can beredefined and circumscribed which is another charac-teristic of the operant For example if only right-pawpresses are acceptable members of the reinforced classresponding will tend to occur only with the right paw(Stokes 1995) Of course the capabilities of the organismplay a role for example including ldquotail pressesrdquo in the op-erant class does not mean that the lever would be or couldbe pressed by the ratrsquos tail The structure of the operantclass therefore emerges from the current reinforcementcontingencies and the organismrsquos capability derivedfrom genetics development and prior experiences

696 NEURINGER

Operant variability has structure as well and as for alloperants that structure is due partly to the organism it-self and partly to the reinforcement contingencies Theresearch described throughout this article has providedexamples of the structure of variability generally in theform of sets of four or eight responses across L and Roperanda But the structure of variations can be speci-f ied more precisely For example Mook and her col-leagues (Mook et al 1993 Mook amp Neuringer 1994)reinforced rats for varying four response sequences acrossL and R levers under a lag 1 contingency Unlike in pre-vious studies however the acceptable class was furtherconstrained so that only sequences beginning with two Rresponses (RRLL RRLR RRRL and RRRR) were rein-forced The rats learned to vary their sequences withinthe imposed limitations that is they generally begantheir sequences with two R responses and varied the se-quences within those limitations Similar results wereobtained in the context of a radial-arm maze in whichonly a subset of arms was baited

The precision with which reinforcement contingen-cies can specify class was highlighted in research per-formed by Burns (1997) and Ross and Neuringer (2002)In the latter case college students drew rectangles on acomputer screen with the computerrsquos mouse Each rec-tangle constituted one trial (the screen cleared at the endof each trial) with reinforcement (points) depending onsimultaneous variations along three dimensions of therectangle location on the screen area of the rectangleand shape with the latter defined as the ratio of the rec-tanglersquos height to its width Although the subjects weregiven no instruction other than to ldquogain pointsrdquo theylearned to vary along all three dimensions with levels ofvariability significantly higher than those of yoked controlsubjects who were reinforced independently of variations

A second part of the Ross and Neuringer (2002) studydemonstrated even greater precision of control To gainpoints three new groups of subjects were required to re-peat along one of the dimensions while simultaneouslyvarying along the other two One group was required torepeatedly draw rectangles of approximately the samesize while simultaneously varying location and shapeanother group had to repeat the shape while varying areaand location and the third to repeat location while vary-ing size and shape Verbal instructions were minimal asthey were in the first part of the study Subjects learnedto satisfy these contingencies as well varying two at-tributes while simultaneously repeating one and signif-icant differences were obtained rapidly within 300 trials

These demonstrations raise questions as to how binaryfeedbackmdashreinforcement presented or notmdashcan simul-taneously control independent behavioral dimensionsThe issue is similar to that discussed above concerningsimultaneous reinforcement of variations and selectionsIt is similar also to those issues faced by researchers at-tempting to understand how the nervous system strength-ens particular synapses when many different neurons firesimultaneously and to ldquoallocation of creditrdquo in artificial

learning models For present purposes suffice it to notethe exquisite precision with which reinforcement controlsvariability and repetition

GeneralizationDo animals and people learn to ldquovaryrdquo implying a

generalizable skill or do they learn a specific distributionof specific responses across particular operanda Thequestion of generalization has been asked of many otherbehavioral competencies such as imitation When chil-dren learn to imitate do they learn stimulusndashresponseassociations (if mother moves her right hand the childlearns to move his right hand) or do they learn moregenerally to ldquoimitaterdquo Evidence indicates that reinforce-ment of imitative instances leads to generalized imitation(see eg Baer amp Sherman 1964 Bandura 1986) A sim-ilar question has been raised when pigeons rats or chil-dren learn to choose a stimulus that matches a previouslyshown sample For example if a pigeon is shown a reddisk as a sample it must then choose red when given achoice between that and blue Does the bird learn to matchIn this case the answer appears to depend on the train-ing procedures If trained with few stimuli such as redgreen and blue then the pigeon learns simple associa-tions (if red sample choose red if blue sample chooseblue) but if hundreds of different stimuli (pictures) areemployed during training then generalized ldquomatchingrdquois learned (Wright 1997) In each of these examples thelearning of a general competency or strategy was shownby transfer-of-training tests What was learned with oneset of stimuli or in one situation transferred to new stim-uli and situations We turn now to the question of whetherlearning to vary transfers

After the subjects in Maltzmanrsquos (1960) experimentdescribed above learned to generate remote verbal as-sociates they were asked to devise unusual uses of com-mon objects In comparison with those in the controlgroup these subjects generated significantly more un-usual uses indicating transfer of training to a differenttask However transfer was not observed in the oppositedirection from unusual uses to verbal associates

Holman et al (1977) using a procedure similar to thatof the Goetz and Baer (1973) research described aboverewarded preschool children verbally for novel drawingsand found increased novelty Unrewarded block construc-tions were also monitored and weak evidence was ob-tained for transfer of novel responding from rewardeddrawing to the unrewarded constructions In a second ex-periment preschoolers engaged in four independent taskspainting pen drawing wood block construction and Legoconstruction When the children were reinforced for vary-ing their pen drawings the novelty of paintings increasedalong with the pen drawings indicating transfer from onetask to the other However transfer to the topographicallydifferent block- or Lego-building tasks was not observed

Eisenberger and Armeli (1997) gave monetary rewardsto one group of fifth- and sixth-grade students the vari-ability group for describing unusual uses of common ob-

OPERANT VARIABILITY 697

jects A second group the repeat group received thesame rewards for describing common uses of the sameobjects In a later test all of the children were asked todraw pictures incorporating a circle and the group that hadpreviously been reinforced for varied responses drewmore unusual (or lower frequency) pictures than did therepeat group Thus there was transfer of novelty trainingfrom one domain to another with additional studies in theseries showing similar effects (Eisenberger Armeli ampPretz 1998 Eisenberger Haskins amp Gambleton 1999Eisenberger amp Selbst 1994) However Eisenberger et al(1999) failed to replicate the transfer-of-training effectThe variability and repeat groups responded similarly dur-ing the test phase In this study separate experimenterstrained and tested the subjects thereby avoiding possibleexperimenter-expectancy effects whereas the same ex-perimenter did both training and testing in Eisenbergerrsquosother studies Across the series of experiments there weremany other cases in which transfer was not observed andmore research is necessary to explain why

In Lee et alrsquos (in press) study with autistic individualsdescribed above after variable answers were trained toquestions verbal variability transferred to new experi-menters and new contexts

Arnesenrsquos (2000) results described above are consis-tent with transfer of training in an animal model Rats inthe Var group were trained to interact variably with oneset of objects When later tested in a new situation withnew objects the Var animals showed (in comparisonwith controls) higher levels of exploration manipulationof the novel objects and discovery of hidden reinforcers

Much of the evidence reviewed in this section there-fore indicates transfer of training and supports a generalldquovaryrdquo competency However there remain unresolvedquestions the most important of which is the extent towhich transfer occurs Note also a methodological pointMany of the studies tested whether transfer occurs whenthe contingencies are neutral with respect to the to-be-transferred behavior Holman et al (1977) is a case inpoint Variations were reinforced in one domain draw-ing and the question was whether variations wouldoccur in another domainmdashblock constructionmdashin whichreinforcement did not depend on varying A negativef indingmdashin this case absence of transfermdashindicatesnothing about the ability to generalize but only aboutthe tendency to do so given the absence of a reason (orcontingency) A better test would be to compare experi-mental and control performances when variability in thenew task is in fact functional as in Arnesen (2000)

SummaryBefore variability can be reinforced it must occur for

reasons other than reinforcement Some of the reasonsinclude variability inherent in all behavioral phenomenaldquoaccidentalrdquo environmental events that are unrelated toparticular behaviors and withholding of reinforcementTransfer from other situations may serve a similar functionFor example if an organism can discriminate shaping-

of-new-response periods from others then it may learngenerally to ldquovaryrdquo during shaping episodes After rein-forcement makes initial contact with response variationsthere are at least three behavioral strategies that can beemployed in order to maintain functionally high levelsof variability respond based on external random eventssuch as the toss of a coin avoid repetitions by remem-bering what responses had previously been made and re-spond stochastically Evidence for an endogenous sto-chastic process was seen in the similarity of responsedistributions to those generated by stochastic sourcesand by findings that interfering with memory for priorresponses either improves or leaves unaffected stochas-tic behaviors Stochastic responding as well as othertypes of variability-generating strategies depend uponcontingencies to define the required level of variabilityand the set from which variations emerge

IMPLICATIONS

One implication of reinforced variability is that oper-ant responses may differ in the extent to which their vari-ability is affected by reinforcement contingenciesmdashthatis operants may be ordered along a continuum of sensi-tivity of variability to reinforcement Another is that re-inforced variability causes us to question the generalityof some basic goals of behavioral psychology includingprediction and control A third is that reinforced vari-ability may help us to provide a scientifically useful wayto describe voluntary action In particular the variabilityof all voluntary behaviors can readily be changed mo-ment to moment by discriminative stimuli and conse-quences Implications for everyday life also will be noted

Operant ContinuumAs was indicated above the operant is composed of

instances that is the operant is a class of many differentresponses all functionally equivalent but often topo-graphically distinct Skinner suggested and I hypothe-size that instances within the class emerge stochastically(see Moxley 1997 for a discussion of Skinnerrsquos posi-tion) Although reinforcement contingencies and contextinfluence the individual probabilities and their distribu-tions emission is stochastic nonetheless The particularmember of the operant class generally cannot be predictedwith a high degree of certainty For example a particu-lar rat may be more likely to press the lever with its leftpaw than its right but left- and right-paw responses areemitted probabilistically This class-nature of the operantsuggests one way to conceptualize operant variability interms of size of class and within-classes probability dis-tributions High variability implies large classes equalprobability distributions or both low variability impliesthe opposite

Variability may be modified more readily in some op-erants than in others implying that operant variability isnot an all-or-none phenomenon but rather a continuumwith reinforced variations in variability (RVV) differing

698 NEURINGER

for different operants RVV might best be understood byanalogy to the concepts of speed and acceleration Bothvariability and speed range from low to high RVV isanalogous to acceleration the rate of change in speedHighly sensitive RVV is indicated by the possibility ofreinforcing many different levels of variability from rep-etition to random-like responding and by the fact thatchanges in level occur readily and rapidly

Different operants manifest differences in RVV Forexample variability of chain pulls changed more thanthat of leverpresses when variability was reinforced inrats (Morgan amp Neuringer 1990) Some operant-like re-sponses may show relatively low RVV For example fir-ing rates increase in single hippocampal nerve cellswhen dopaminergic substances are contingent upon theresponse (Stein Xue amp Belluzzi 1994) suggesting thatnerve f iring is controlled by reinforcement To datehowever there is no evidence that variability of nervefiring is similarly controlled Other cases might includereinforcement of nondiscriminable (subconscious) re-sponses such as Hefferline and Pererarsquos (1963) reinforce-ment of subaware thumb twitches and biofeedback ofmuscle and cardiovascular activity In these cases rein-forcement may modify performance but it is not knownwhether variability is also sensitive to reinforcement andif so to what degree Human verbal variability may bethe most readily manipulable of all behaviors the rangeof predictable to unpredictable verbal responses beingimmense with extraordinary sensitivity to context andconsequence RVV may also differ across organisms (per-haps from simple to complex) individuals (perhaps asan index of intelligence or learning ability) and physio-logical and psychological states (WKY vs SHR depres-sion and autism)

RVV may help to distinguish operant responses fromother superficially similar cases of selection by conse-quences That is the ldquooperantrdquo may also best be concep-tualized as a continuum with RVV an important deter-miner of position along the operant continuum Oneexample of selection by consequences but low ldquooperant-nessrdquo is genetic change Variability is essential for evolu-tionary change of course and levels of genetic variabil-ity are sometimes affected by environmental conditions(Moxon 1997) but there are few examples of levels ofvariability being influenced by selection pressures Thatis the source of much genetic variability is thought to berandom not influenced by current pressures and needsThus although selected by consequences genetic changemay be low along the operant continuum Other exam-ples of operant-like phenomena for which variability although important may be relatively insensitive to con-sequencesmdashvariability in these cases is elicited not se-lectedmdashinclude the development of the central nervoussystem and immune system responses (Hull et al 2001)In both of these cases change depends upon selection byconsequences but it is not clear whether variability canalso be selected Most commonly studied operant re-sponses on the other hand show high sensitivity of vari-

ability to reinforcing events But to reiterate the mainpoint of this section operants differ in terms of RVV andtherefore in terms of their ldquooperantnessrdquo from low sensi-tivity (possibly responses of single nerve cells) to extra-ordinarily high sensitivity (verbal behavior) The operantnature of variability as well as that of every other oper-ant dimension is best conceptualized as a continuum

Prediction and ControlSmith (1992) described Skinnerian operant condition-

ing theory as Baconian in nature a primary goal of Skin-nerian behaviorists being to control behavior and hecontrasted this with the more Aristotelian goal of de-scription Skinner and his students (eg Zuriff 1985) in-deed emphasized prediction and control Many of thestudies reviewed above however involved reinforce-ment procedures that increase variability and conse-quently decrease predictability It is not only ignorancetherefore that restricts prediction and control as Skin-ner suggested but reinforcement itself Operant meth-ods and theories when applied to variability lead togreater emphasis on description with control over pre-cise response characteristics possible under some butnot all conditions

In summary an operant is a class of responses underthe control of contingencies of reinforcement The classcan be controlled and predicted given knowledge ofconditioning history Indeed the size of the class andpossibly the distribution of within-classes probabilities orlevel of variability can also be controlled and predictedBut to the extent that instances emerge stochastically aparticular instance cannot be predicted The instant re-sponse can be explained as a member of a particularclass but prediction of the instance will be diff icultsometimes to the extreme This point will be developedin the next section

Voluntary BehaviorI hypothesize that voluntary behaviors are operants for

which RVVs are highly sensitive to influence by conse-quences One way to test this hypothesis would be togather empirical data concerning how voluntary versusinvoluntary actions are classified by human observers Asfar as I know there are no psychophysical data concern-ing this question but the hypothesis implies that highsensitivity of variability to operant contingencies RVVswill be present in all actions perceived to be voluntary

This operant-variability theory of voluntary action isbased on the supposition of two attributes First volun-tary acts must be potentially explainable (given suff i-cient knowledge) on the basis of genetics conditioninghistories current contingencies anticipated conse-quences intentions goals and the like That is volun-tary behaviors are lawful The second attribute is that thevoluntary act must be potentially unpredictable That isvoluntary acts are ldquofreerdquo of control by identifiable stim-uli If behavior is perceived necessarily to result frommdashto be predicted on the basis ofmdashexternal events includ-

OPERANT VARIABILITY 699

ing reinforcement contingencies it will not be perceivedas voluntary ldquoVoluntaryrdquo implies potential indepen-dence from stimulusndashresponse or responsendashreinforcerdetermination According to this theory both explain-ability and unpredictability are necessary attributes ofvoluntary acts

How though can behaviors be simultaneously ex-plainable and unpredictable Reinforcement contingen-cies play two essential roles Reinforcement (togetherwith genes prior experiences and current environment)helps to determine the set of possible behaviors fromwhich a voluntary act emergesmdashthat is the operant classReinforcement (together with the other contributorsspecified above) also helps to determine the level of vari-ability within the class Thus both instances comprisinga class and level of variability within that class can bepredictedmdashagain given sufficient knowledge Howeverthe individual response is at least sometimes (or relatively)unpredictable The reason is that the operant class fromwhich the response stochastically emerges is sometimesextremely large (eg ldquoname a wordrdquo or ldquodo somethingrdquo)and the contingencies sometimes call for equiprobablestochastic emergence (eg ldquofool merdquo) In the latter casea voluntary act may appear to be foolish meaninglessor unwarranted but if it is voluntary it can potentially beexplained (For example in response to the questionldquoWhy did you do that silly actrdquo the answer could beldquoJust to show that you canrsquot predict my behaviorrdquo seeScriven 1965) In these cases prediction of instancesfalls to very low levels indeed and I hypothesize evenan all-knowing observer will be unable to predict thenext response at greater-than-chance levels which aregiven by the size of the operant class and by within-classesprobability distribution Even within small classes (egin the game rockndashscissorsndashpaper in which unpredictabil-ity among the three possible responses is functional) be-havioral stochasticity may lead to unpredictability of theinstant response Note however that although voluntarybehaviors are sometimes unpredictable the opposite isoften the case When contingencies call for highly con-strained particular responses predictions can easily bemade For example if a police officer asks your nameyour response can most likely be predicted Voluntary re-sponses are therefore described as potentially unpre-dictable with variability contingencies engendering dif-ferent levels of unpredictability

According to this theory behaviors low in the sensitivity-of-variability dimension are described as relatively in-voluntary and those that are high in said dimension areperceived as voluntary Low RVV may be shown at anylevel along the variability continuum For example al-ways responding in the same unchanging way will beperceived as involuntary but so will always respondingrandomly (The coin does not voluntarily come up withheads or tails) This point is essential More than 2000years ago Epicurus argued that occasional randomswerves of atoms provided the source of voluntary be-havior in an otherwise determined universe but al-

though this conjecture was prescient it was not quitecorrect ldquoFree-willedrdquo or voluntary action whether it isbased on random swerves or on quantum mechanics isnot explained by variability per se but by functionalchanges in levels of variabilitymdashthat is by rapid andprecise changes in response to environmental demandsResponses can be placed along a voluntaryndashinvoluntarycontinuum isomorphic to the operantndashrespondent con-tinuum with voluntary behavior being functional pre-cisely in the same way as operant behavior is ThusSkinnerrsquos (1974) suggestion that voluntary behavior isoperant behavior was almost correct Only those oper-ants for which variability is highly sensitive to conse-quences will be perceived as voluntary

One additional example might help Operant classes(sets of potential responses) are created ldquoon linerdquo as we goabout our daily activities As we walk from car to officewe tend to walk in a characteristic way take the samepath with tightly constrained operant classes followingone after the other in rapid succession Class sizes aresmall and within-classes probability distributions arehighly peaked Predictions of next responses by a knowl-edgeable observer can readily be made However classsizes can be increased and probability distributions flat-tened we can walk in quite strange ways indeed dancealong a new path sing hand candy to passers-by Strangebehavior indeed but made possible by the ability of op-erant sets to be modified moment to moment not only interms of class definition but in terms of class size lev-els of within-classes variations and so on The same istrue for conversation Moment-to-moment changes inverbal class definition size and predictability of within-classes instances are hallmarks of voluntary behavior

GenerativityRelated to the definition of the operant and to our discus-

sion of voluntary action is that fact that operant behaviormdashleverpressing no less than languagemdashis by its naturegenerative This claim follows from the functional na-ture of the class of responses (all members of the classare effective in producing an outcome) and the stochas-tic emission of instances From these flow never-before-emitted instances patterns and sequences (see Epstein1996) Operants especially those high on the RVV con-tinuum are inherently generative

Self-ExperimentationVariations systematic or otherwise are a necessary part

of scientific work Neuringer (1981) and Roberts and Neu-ringer (1998) described how variations can be brought to bear explicitly on everyday life in the form of self-experiments Individuals vary aspects of their lives suchas the foods they eat social contacts and methods ofmemorizing they keep records concerning the effects ofthese manipulations and form hypotheses concerningcontrolling variables Using self-experimental methodsindividuals can learn how what when where and howmuch to vary Explicitly varying onersquos everyday behav-

700 NEURINGER

iors and thoughts may help to engender what Langer(1989 2000) has referred to as mindful livingmdashthat isnonautomatic voluntary engagement with beneficialeffects in different areas including learning The impor-tant point here is that variability can be voluntarily cho-sen maintained and modified in everyday life and thatexplicit variations may have important consequences

Implications for SocietyConsideration should be given to the importance of re-

inforcement contingencies to support variability in so-cial milieus A basic point made throughout the presentarticle is that merely permitting behavior to vary doesnot necessarily engender or maximize variability (as isshown by Yoke and Prob schedules) Analogously merelypermitting varied social behavior as in a free or laissez-faire society may not suffice to produce the desired di-versity If the results shown for individual organismscan provide useful information for groups (see eg Grottamp Neuringer 1974) then when variations are desiredsocieties might reward individuals for varying not sim-ply permit it

SummaryOperants differ in the extent to which reinforcers in-

fluence levels of variation When variability is highlysensitive to reinforcement the operant may be perceivedto be voluntary Voluntary responses have two attributesThey are functional implying that the class of responsesmdashthe operantmdashcan be predicted given sufficient knowl-edge But instances emerge stochastically implyingsome degree of unpredictability Because reinforcementof variability can generate extremely large classes withstochastic emergence of instances it suggests limitationsof the two basic goals of Skinnerian behavioral theorymdashnamely prediction and control Operant variability also hasimplications for self-experimentation and for society

RELATED AREAS

Operant variability may be important in many otherareas and I will briefly review some of these

Negative reinforcement To my knowledge there havebeen no published studies in which variability is nega-tively reinforced High variability might sometimesavoid or reduce a punisher such as loss of goods ormoney but at other times low variability could be func-tional Effects of positive and negative reinforcement onvariability may have different evolutionary bases onederived from increasing probabilities of locating sourcesof food or mates the other from avoiding confrontationsor predation (Driver amp Humphries 1988) One questionconcerns how quickly variability can be changed by neg-ative as opposed to positive reinforcement There isconsiderable evidence that loss functions differ fromgain functions (Kahneman amp Tversky 1984) possiblyindicating that positive reinforcement-of-variabilityfunctions will differ from negative ones as well

Resistance to change In some of the studies describedabove variability appeared to be more resistant tochange than repetitions were for example alcohol hadlittle effect on variable operant responses but interferedwith repeated ones Doughty and Lattal (2001) tested thegenerality of these f indings using four-response se-quences with pigeons a threshold variability contin-gency in the Var component of a multiple schedule andLRLR in the Rep component Doughty and Lattal chal-lenged stability of responding by prefeeding the birds inone experiment and providing reinforcers independentlyof responding in the other (see Nevin 1992) Operantvariability was again more resistant to change than wasoperant repetition Thus reinforcement of variabilitymight increase the stability of responding (that is its re-sistance to change) a seemingly paradoxical findingHowever Mechner (1992) suggested that this is the casefor complex skills such as piano performance Varyingpractice routines will make the performance relativelyinsensitive to unforeseen changes in the environmentCare must be taken however when variation is comparedwith repetition because they have different measuresAlso the results reported by Doughty and Lattal werebased on the performance of three pigeons and were notconsistent throughout the experiment The hypothesiswarrants further study

Attraction In some bird species complex male songsattract females (for a review see Catchpole amp Slater1995) There is evidence that male song complexity isunder the discriminative stimulus control of a receptive fe-male (Searcy amp Yasukawa 1990) That explicit reinforce-ment may play a role in such song complexity is suggestedby the Manabe et al (1997) study described above An in-teresting question for further research is the extent towhich operant contingencies contribute to the attractioncreated by behavioral variability in these and other species

Discrimination of variability Animals and people canuse the variability of a stimulus array or entropy as a dis-criminative stimulus (Young amp Wasserman 2001) Whatis the relationship between perception of variability onthe one hand and emission of variable responses on theother Do individuals attempt to match their own levelsof variation to those of others in their environments whichwould involve both perception and emission How mightdiscrimination of onersquos own level of variability influenceonersquos ability to modify such levels and do influences thatobscure discriminations therefore affect ability to change(see eg van der Meer van der Weel amp Lee 1995)Hunziker et al (2002) report that although variabilitywas successfully reinforced their human subjects couldnot correctly describe the contingencies of reinforce-ment a finding similar to those of informal observationsin other studies (eg Neuringer Deiss amp Imig 2000)but whether subjects could discriminate variability levelsrather than contingencies was not studied

Attention and awareness When attention is subject tointerference or is withdrawn from the task at hand abilityto vary is degraded (eg Baddeley Emslie Kolodny amp

OPERANT VARIABILITY 701

Duncan 1998 Evans amp Graham 1980 Neuringer amp Voss2002) It is unclear at present how attention influencesthe different processes underlying operant variabilityand in particular memorial and stochastic processesRich areas for additional study are the relationships be-tween attention awareness and consciousness on theone hand and operant variability on the other Voluntaryaction and consciousness are often related (see egLibet 1999)

Concepts Concepts are thought by many to be sets ofinstances (much like operant sets) whose structures aregraded meaning that there are more or less typical in-stances of such concepts as birds (robin is typical ostrichis not) Barsalou (1987) showed that the gradation ofthese cognitive structures is manipulable or as he putsit not stable A number of variables influence structureincluding context individual differences and points ofview Barsalou argued that these structures are createdas the need arisesmdashthat is on line by the thinking indi-vidual The structure of concepts may parallel that of op-erants and in both cases levels of within-structures di-versity or variability may be modified by consequencesOperant variability may be found in the cognitive as wellas in the behavioral domain

Play and humor Highly variable behaviors are ob-served when animals and people especially children en-gage in play and humor (see eg Fagen 1981) The re-sult of play may be acquisition of new responses (seeSutton-Smith 1975) Although not currently needed re-sponses learned during play may provide the source ofvariations for successful shaping of behaviors at anothertime This is quite analogous to the role of conserved ge-netic variants which are passed along to progeny but donot currently serve any identifiable role Play may alsoinvolve learning how to vary operantly

Reinforcement by variability The self-maintainingnature of play suggests that engaging in variable behav-iors may itself be reinforcing Indirect support comesfrom findings that varied stimulation can serve as a re-inforcer and that stimulus repetition becomes aversive(Fiske amp Maddi 1961) Possibly also related are findingsthat varying reinforcers and permitting choice among re-inforcers increase their value (Bowman Piazza FisherHagopian amp Kogan 1997) and that permitting choicesamong varied responses is also preferred to demandingfixed responses (Catania 1980)

Foraging Two issues regarding variability are presentin the foraging literature One has to do with risk prefer-ence versus avoidance (see eg Shettleworth 1998) Asecond has to do with search strategies Most reportedcases of foraging involve systematic search patterns(Stephens amp Krebs 1986) However search patterns aresometimes stochastic or highly variable (see eg Peter-son 1997) One question is whether some species can varybetween these two strategies and the challenge is to iden-tify the pressures leading to and the advantages of each

Animal training Wilkes (1996) has applied operantvariability techniques in developing training proceduresfor pet dogs More generally reinforcement of variabil-

ity may enable the gaining of control over behaviors ofanimals and as has been suggested above may be an im-portant part of the process of shaping

Models of learning Reinforced variations in variabil-ity may be important when human and animal learningis being modeled whether the models are mathematicalin form computer based or machine based (Blumberget al 2002 Brooks 1999 Sutton amp Barto 1998)

CONCLUDING COMMENT

Genes interact with environments to produce diverseanatomical physiological and behavioral phenotypesGenes and environments also provide for another level ofdiversitymdashnamely operant variability There are manysources of operant variability including random environ-mental events memory for past responses and stochasticgeneration each of which ultimately derives from ge-netic and environmental influences To the extent thatoperant variability is based on a stochastic source how-ever the behavioral results cannot be predicted fromknowledge of these influences That is organisms haveevolved to behave unpredictably The positing of sto-chasticity to explain behavior has a history extending atleast as far back as Epicurus with major contributors in-cluding Gustav Fechner (see Heidelberger 1987) EgonBrunswik (see Gigerenzer 1987) Skinner (1959) andSuppes (1984) Knowledge of the stochastic nature ofthe operant and of how it is controlled leads to a worldview that differs from genetic and environmental deter-minism with roots in the works of the scholars just men-tioned At the scientific level this view leads to the ex-perimental analysis of consequence-influenced variabilityAt the philosophical level it leads to explanations of vol-untary action At the personal level it can lead to change

REFERENCES

Amabile T M (1983) The social psychology of creativity New YorkSpringer-Verlag

Antonitis J J (1951) Response variability in the white rat duringconditioning extinction and reconditioning Journal of Experimen-tal Psychology 42 273-281

Archer J amp Birke L (1983) Exploration in animals and humansCambridge Van Nostrand Reinhold

Arnesen E M (2000) Reinforcement of object manipulation in-creases discovery Unpublished undergraduate thesis Reed College

Baddeley A D (1966) The capacity for generating information byrandomization Quarterly Journal of Experimental Psychology 18119-129

Baddeley A [D] Emslie H Kolodny J amp Duncan J (1998)Random generation and the executive control of working memoryQuarterly Journal of Experimental Psychology 51A 819-852

Baer D M amp Sherman J A (1964) Reinforcement control of gen-eralized imitation in young children Journal of Experimental ChildPsychology 1 37-49

Baker M J amp Koegel R L (1999) Transforming children withautismrsquos ritualistic behaviors into positive play interactions with sib-lings Paper presented at the meeting of the California Association ofBehavior Analysis San Francisco

Balsam P D Deich J D Ohyama T amp Stokes P D (1998) Ori-gins of new behavior In W OrsquoDonohue (Ed) Learning and behav-ior therapy (pp 403-420) Boston Allyn amp Bacon

Balsam P D Paterniti A Zechowy K amp Stokes P D (2002)

680 NEURINGER

rent sequence repeated at least one of the previous threeOn an average of once every 30 sec (VI 30 sec) the com-puter probabilistically primed reinforcement for Var orRep the prime being maintained until the reinforcementhad been collected Thus if the computer primed Var re-inforcement was delivered whenever the lag 3 variabilitycontingency was next met Similarly if Rep had beenprimed the next Rep sequence was reinforced Proba-bilities of Var and Rep reinforcements were systemati-cally manipulated the two always summing to 10 and amatching-type relationship was obtained (Figure 4panel A) The relationship was more complex than sim-ple matching possibly because if the pigeons chose tovary by responding stochastically there was some prob-ability due to chance that the current sequence wouldrepeat a previous sequence This was shown by a com-parison of pigeon performances to the performance of acomputer-simulated stochastic model programmed tomatch its Var choices to the probability of Var reinforce-ments The pigeonsrsquo choice distributions were found tobe the same as the modelrsquos (Figure 4 panel B) Thuschoices to vary or repeat were governed by relative fre-quencies of reinforcement An animal can choose whetherwhen and how much to vary a result consistent withthose described above from Blough (1966) Machado(1989) and Grunow and Neuringer (2002) Choices ofwhether or not to vary appear to be governed similarly toall other operant choices

These f indings may have important applicationsWhen an individual repeatedly responds in a way that isnonfunctional reinforcement of choices to vary mayhelp the individual to emerge from the ineffective pat-tern This can be true for those experiencing everydayproblems but feeling helpless to change and it may beespecially important for those experiencing depressionor who respond in otherwise abnormally stereotypedmanners such as those with autism These topics will beconsidered in a later section

ExtinctionAs was indicated above extinction is often found to

increase variability but almost all studies of this effectinvolve responses that had previously been repetitiveAntonitis (1951) for example reinforced rats for pokingtheir noses anywhere along a 50-cm horizontal openingAlthough the exact location of pokes did not matter withcontinued training responses became limited to one or afew locations along the strip When reinforcement waswithheld variability of location increased (see also Eck-erman amp Lanson 1969) Variability produced by extinc-tion has been reported for many other response dimen-sions as well including force (Notterman amp Mintz1965) number (Mechner 1958) topography (Stokes1995) and sequence (Balsam Paterniti Zechowy ampStokes 2002 Mechner Hyten Field amp Madden 1997)

The issue is complicated however by other studiesthat show that previously reinforced responses and re-sponse patterns are maintained intact during and afterextinction That is extinction does not break down

learned patterns For example when Schwartz (1981) re-inforced pigeons for any sequence of four L and four Rresponses each bird developed a dominant sequence asdescribed above and these sequences were maintainedduring extinction Despite decreasing response ratesduring extinction response patterns were retained Sim-ilar maintenance of response structure has been ob-served during extinction following reinforcement of

Figure 4 Average performance across a group of pigeons inNeuringerrsquos (1992) study of choices to vary or repeat (A) Loga-rithms of the ratio of number of times the pigeons met the Varcontingency divided by the times they met the Rep contingencyas a function of the analogous numbers of reinforcements re-ceived for each Points represent group averages in two separateexperiments Also shown is the least squares best-fitting line(B) Changes in U-value an index of response variability as afunction of the obtained percentage of reinforcements for vary-ing The solid line in panel B represents the performance of acomputer-based model that matched its choices to vary (vs re-peat) to its obtained relative frequency of reinforcement for vary-ing From ldquoChoosing to vary and repeatrdquo by A Neuringer 1992Psychological Science 3 p 248 Copyright 1992 by the AmericanPsychological Society Reprinted with permission

OPERANT VARIABILITY 681

choices (Myerson amp Hale 1988) responding underfixed-interval contingencies (Machado amp Cevik 1998)visual discriminations (Nevin 1967) and matching tosample (Cumming Berryman Cohen amp Lanson 1967)How can stability during extinction be reconciled withincreased variability

Neuringer Kornell and Olufs (2001) wrestled withthis question in a study of extinction following rein-forcement under three different contingencies An oper-ant chamber contained two levers left (L) and right (R)on either side of a reinforcement dispenser and a key (K)in the rear of the chamber distant from the dispenserOne group of rats Rep was reinforced for repeating afixed sequence of responses LKR A second group Varwas reinforced for variations across the three operandaunder a threshold contingency Reinforcement for a thirdgroup Yoke was matched to those of the Var animalsand was therefore independent of the particular sequencesBecause of the unusual configuration of operandaskewed distributions of sequence probabilities were ob-served in all of the groups an outcome that was antici-pated and desired in order to document possible in-creases in variability during extinctionmdasha flattening ofthe distributionmdashas well as possible decreases That isduring the reinforcement phase the Rep group came torepeat the LKR sequence with high probability the Yokegroup tended to emit the easiest sequences such as LLLand RRR but distributed their responses over more se-quences than the Rep group did and the Var group dis-tributed most widely although because of differences inthe operanda and in their locations some sequences weremore preferred than others and in particular sequencesending with a K response were least likely (the key beingfarthest from the reinforcement dispenser) Thus thecontingencies resulted in three different distributions orhierarchies of sequence probabilities very steep forRep intermediate for Yoke and flattest for Var

The main finding was that withholding reinforcementhad similar effects in the three groups In each responserates fell to low levels across the four sessions of extinc-tion Furthermore sequences that were most frequentduring the reinforcement phase continued to be most fre-quent throughout extinction A slight flattening of thedistributions was observed however with the lowestprobability sequences increasing relative to the rein-forcement phase Thus because the overall hierarchy ofsequence frequencies was maintained the results indi-cated stability but because the distributions were some-what flattened the results also showed slightly increasedvariability It is as if the strategy during extinction wereto keep doing what had worked in the past but occa-sionally to do something very different This compli-cated story integrates two effects of extinction increasedvariability and maintained stability Most importantly forthe present discussion extinction had similar effects onvariable yoked and repetitive operants

One additional comparison is of interest Recall thatGrunow and Neuringer (2002) found that when contin-

gencies demanded high variability obtained levels ofvariability decreased slightly as reinforcement frequen-cies were lowered This result is opposite to the Vargrouprsquos slight increase in variability just described underextinction conditions The reason may be that extinction-induced variability is short lived lasting only as long asresponding continues perhaps for hours or a few ses-sions at most whereas the variability engendered by theVI 5 intermittent schedule of reinforcement in Grunowand Neuringer can be maintained indefinitely The de-crease in Grunow and Neuringerrsquos operant variabilitymay be due to the weakening of the reinforcement effect(ie reinforcement is less frequent) whereas the in-crease following extinction may be a short-term elicitedeffect This distinction between reinforced and elicitedvariability is important and will be discussed further inthe next section

Operant and Respondent InteractionsElicited or what is sometimes referred to as respondent

effects occur when environmental changes that affectbehavior do not depend on (are not caused by) the behav-ior Pavlovian conditioning being the most commonlystudied example Operant effects are observed when a re-inforcer depends on the response to produce it Althoughfor historical reasons respondent and operant effects areoften studied separately most behaviors result from acombination or interaction of the two Variability is nodifferent and we turn to an example of such an interaction

As reinforcement is approached under FI or fixed-ratio (FR) schedules response rates are commonly ob-served to increase Similarly when reinforcement de-pends on accuracy as in discrimination learning ormatching to sample it too is found to increase within theFI or the FR (Nelson 1978 Nevin 1967) Cherot Jonesand Neuringer (1996) asked whether the same patternheld for reinforced variability Does operant variabilityincrease as reinforcement for varying is approachedThe question may be related to such real-world cases aswhen a composer is rushing to complete a compositionbefore a scheduled performance or when a programmeris trying to solve a problem before an upcoming meetingThese activities involve at least to some extent non-repetitive original or variable thought and behaviorThe question is whether variability increases with prox-imity to or anticipation of reinforcement for varying

Cherot et al (1996) employed trials consisting of fourresponses across L and R operanda and a lag 4 variabil-ity contingency with rats in one experiment and pigeonsin another Unlike in any of the cases described so farhowever the lag-variability contingency had to be satis-fied four times in order for a single reinforcer to be ob-tained With ldquordquo indicating successful completion of theVar contingency ldquo2rdquo indicating failure to meet that con-tingency and PELLET indicating the food reinforcer anexample follows LRLL RRRL LRLL2 LRLL2RLLR RRRL2 LRRR+PELLET Thus food reinforce-ment was provided only after the subject had satisfied

682 NEURINGER

the variability contingency four times FR 4 The resultswere that as the food reinforcer was approached (ie asprogress was made through the ratio) the subjects wereless and less successful at varying The probability ofmeeting the variability contingency in the first trial afterreinforcement was higher than it was in the second whichin turn was higher than that in the third trial and so onFor comparison a Rep group was reinforced for repeatingsequences once again with reinforcement only after foursuccessful repetitions Probability of repeating increasedwith proximity to reinforcement a finding consistentwith the literature but opposite that for the variabilitygroup Thus approach to reinforcement facilitated oper-ant repetition but interfered with operant variabilitySimilar variability-interfering effects have been docu-mented for within-trials responses The probability that aresponse repeats the just prior response increases within atrial even when sequence variability is reinforced (McElroyamp Neuringer 1990 Neuringer 1991) To return to the ex-ample of a composer rushing to complete her composi-tion she may indeed have increasing difficulty respond-ing creatively under the pressure of a deadline On theother hand the soloist may find it easier to repeat thecorrect performance during practice periods as the con-cert nears How can these results be reconciled with themany findings that variability can be reinforced

An additional result of Cherot et al (1996) provides ananswer The Var animals responded much more variablyoverall than did the Rep animals Although variability inthe Var groups decreased with approach to the reinforceroverall variability was always much higher when it was re-inforced (Var group) than when it was not (Rep group)Reinforcement therefore exerts two influences Overallvariability is elevated sometimes to the highest levelswhen it is reinforced but approach to (and possibly focuson) the reinforcers constrains or lowers variability Sim-ilar reinforcement-produced interference effects had beenreported previously for example when raccoons are re-inforced for dropping a ldquocoinrdquo into a slot the reinforce-ment results in the animalsrsquo holding and rubbing the coinrather than dropping it (Breland amp Breland 1961) Theoverall enhancement of variability was of much greatermagnitude in Cherot et al than the decrease with approachwas although both effects were statistically significant(The notes created by the composer under pressure ofdeadline will be less readily predicted than will thoseplayed by the performer practicing a well-known piece)This study highlights the importance of interactions be-tween operant effects of reinforcement and elicited effectsproduced by anticipation of reinforcement and the resultsmay be relevant to the debate concerning possible detri-mental effects of reinforcement on creative behaviors (seethe discussion below and Neuringer in press)

SummaryReinforcement controls behavioral variability as it

does other operant dimensions Levels of variability areprecisely governed by schedule contingencies and dis-

criminative stimuli Choices to vary or repeat match rel-ative frequencies of reinforcement Extinction increasesvariability slightly whereas the hierarchies of sequenceprobabilities are retained and operant and respondenteffects interact The evidence shows that behavioral vari-ability is an operant controlled by its consequences

GENERALITY AND PARAMETERS

Many different methods measures species and re-sponses have been used in the study of operant variabil-ity and this diversity supports the generality of the phe-nomenon As is the case for repeated operants howeverthere are parametric constraints This section briefly re-views the generality and some of the parameters studiedto date

GeneralityMethods Throughout this section ldquoresponserdquo will

signify a response unit including both an individual re-sponse and a multiresponse sequence Each of the meth-ods to be described increases or maintains variability byreinforcing it

Novel response procedures reinforce responses upontheir first occurrence ldquoeverrdquo (Pryor et al 1969) or upontheir first occurrence within a session (Goetz amp Baer1973) Radial-arm maze procedures also reinforce ini-tial (within a given session) entries into arms of a mazeand result in a distribution of choices across the availableoptions the sequence or pattern of choices often beingquite variable

Lag procedures reinforce a response if it has not oc-curred across a given number of previous trials (Page ampNeuringer 1985) For example under lag 5 the currentsequence is reinforced only if it has not occurred duringany of the previous five trials A variant is the recurrence-time procedure which specifies a minimum number of in-tervening trials before a previously emitted response canagain be reinforced Machado (1989) combined recurrencetime with a percentile-reinforcement contingency to gen-erate high levels of variability (see also Machado 1992)Percentile-reinforcement contingencies base the criterionfor reinforcement on the subjectrsquos own performance over aprevious set of responses (see Galbicka 1994) For exam-ple a sequence will be reinforced only if it has not oc-curred over the previous n trials the value of n continuallychanging so that 50 of the trials are reinforced

Least-frequent response procedures (Blough 1966Schoenfeld Harris amp Farmer 1966 Shimp 1967) rein-force responses that are currently least frequent across amoving window

Threshold procedures reinforce responses whose rela-tive frequencies fall below a given value Because thistype of procedure was used in the research describedabove and in many other more recent studies details willbe provided In Denney and Neuringer (1998) which willserve as an example trials consisted of four responses byrats across L and R levers A running tally was kept of the

OPERANT VARIABILITY 683

number of occurrences of each of the 16 possible se-quences LLLL LLLR LLRL and so on If the relativefrequency of the current sequencemdashthe number of its oc-currences divided by the sum of all 16 sequencesmdashwasbelow a specified threshold value in this case 05 the ratwas rewarded Recently emitted sequences contributedmore to the maintained tally than did those in the distantpast because after each reinforcement all 16 counterswere multiplied by a weighting coefficient in this case95 Therefore the influence of a particular trial decreasedexponentially with successive reinforcements Thresholdprocedures differ from Bloughrsquos (1966) least-frequentprocedure in that more than one sequence (or response)can be reinforced at any given timemdashnamely all those se-quences that have relative frequencies below the thresholdvalue The procedure differs from lag in permitting agiven sequence to be reinforced many times in successionif the associated relative frequency is sufficiently low

The frequency dependence procedure (Machado 19921993) is a variant of threshold in which the probability ofreinforcement is a continuous function of relative re-sponse frequency The more frequent a response is thelower the probability is that it will be reinforced

Statistical feedback procedures provide subjects withcomparisons between their performances and that of amodel Neuringer (1986) showed human subjects feedbackfrom 10 statistical tests thereby enabling them to comparetheir own distributions of responses to that of a randommodel Neuringer and Voss (1993) provided feedbackbased on a chaotic model The goal in each of these caseswas for the subjects to learn to approximate the model andthe results of these experiments will be discussed below

Differential reinforcement of switches versus repetitionsbalance a subjectrsquos tendency to engage in each of these pat-terns Rather than reinforcement for infrequent responsesor random-like distributions reinforcement is providedsometimes for repeating a response and other times forswitching from one response to another In the two-operandum case a switch is L followed by R or R by La repetition is L then L or R then R Bryant and Church(1974) found that stochastic-like responding was gener-ated by rats when switches were reinforced with a 75probability and repetitions with a 25 probability that isthe rats had a natural tendency to repeat at least in theenvironment of this study Machado (1997) reinforcedpigeons for eight pecks across two keys if the sequenceof pecks contained at least one (one group of birds) ortwo (another group) switches Although the birds couldhave satisfied these contingencies by repeating a singlesequencemdashin the ldquoone switchrdquo condition for examplerepetition of LRRRRRR would produce reinforcement on100 of the trialsmdashthey tended to vary their sequencesBarba and Hunziker (2002) showed however that vari-ability was higher when it was directly reinforced thanwhen reinforcement was based on proportions of switchesa finding consistent with that of Machado (1997)

The evidence from each of these methods supports thehypothesis that variability can be reinforced

Measures Variability must be interpreted with refer-ence to a particular measure The reason is that there canbe high variability (and low predictability) at one level ofanalysis but order (and certainty of prediction) at a dif-ferent level For example the sequence 1 1 2 2 3 3 44 5 5 1 1 2 2 has the same variance (a commonlyemployed measure of variability) as 2 3 2 1 5 1 4 41 1 3 2 5 2 but an analysis of conditional proba-bilities in which the unit contains pairs of instances in-dicates that the two sequences have different levels ofvariability In general high variability observed with onemeasure does not rule out the possibility of order ob-served with another and vice versa That is why evalua-tions of random-number generators often employ manydifferent measures and indeed there are hundreds of dif-ferent measures of the randomness of a sequence (Knuth1969) All behavioral studies make simplifying assump-tions and rely on one measure or at the most a few

U-value has been the most commonly employed mea-sure of operant variability (Evans amp Graham 1980Machado 1989 Page amp Neuringer 1985 Stokes 1995)U-value measures the distribution of relative frequen-cies or probabilities of a set of responses For a set of16 possible responses i = 1ndash16 U-value is given by

where RFi = the relative frequency (or probability) ofeach of the 16 responses U-values approach 10 whenrelative frequencies approach equality as would be ex-pected over the long run from a random generator andthey approach 00 when a single instance is repeated

Other measures include the percentage of trials or fre-quency at which variability contingencies are met (Page ampNeuringer 1985) percentages of alternations versus stays(Machado 1992) conditional probabilities of responses(Machado 1992) the number or frequency of novel re-sponses (Goetz amp Baer 1973 Schwartz 1982) the num-ber or frequency of different responses (Machado 1997Schwartz 1982) Markov analyses (Machado 1992) andstatistical tests of distributions of responses (Neuringer1986) All of the measures employed to date indicate thatvariability can be reinforced

Responses and species Generality has further beendemonstrated by the use of many different responses andspecies These include Siamese fighting fish (Betta splen-dens) swimming in quadrants of a tank (Roots 2000)young chicks pecking keys (Neuringer 2002) pigeonspecking keys (Page amp Neuringer 1985) budgerigarssinging songs (Manabe Staddon amp Cleaveland 1997)rats pressing levers (Van Hest van Haaren amp van de Poll1989) porpoises swimming and jumping (Pryor et al1969) rhesus monkeys pressing levers (Sydney Reisbickpersonal communication) children drawing and build-ing with blocks (Holman Goetz amp Baer 1977) andhuman adults and children pressing buttons (Saldana ampNeuringer 1998 Stokes amp Balsam 2001) There havebeen few attempts to compare species or responses but

- ( )[ ] ( )=aring RF RFi ii

log log 2 21

1616

684 NEURINGER

in one study lowest baseline variability was observedwhen rats pulled trapezes intermediate levels were ob-served when they pushed keys and highest levels wereobserved when they pressed levers (Morgan amp Neuringer1990) with variability increasing in all cases when it wasreinforced In another study operant variability wascompared across rats and people with similar effects ob-served in the two species (Neuringer Deiss amp Imig 2000)

ParametersWe turn now to some parameters studied to date Many

other parameters as yet unstudied are no doubt also im-portantmdashfor example motivation and types of reinforcers

Age Older people (see eg Van der Linden Beertenamp Pesenti 1998) and older rats (see eg Lamberty ampGower 1990) respond less variably than their youngercounterparts when variable responding is reinforced or re-quested but in most cases it is not clear to what extent thevariability is elicited as opposed to reinforced Neuringerand Huntley (1992) compared reinforced variability in2-month-old (young) and 10-month-old (mature) Long-Evans rats Under Yoke contingencies the young rats re-sponded significantly more variably (two levers four re-sponses per trial) than the mature rats which constitutesan elicited effect When reinforcement was contingenton variability (lag 4) variability increased significantlyin both groups thereby overcoming the initial differencecorrelated with age Thus explicitly reinforcing vari-ability in older animals caused them to respond no dif-ferently than younger animals did a finding potentiallyrelevant to real-world human aging However when Wells(1999) compared 8-month-old (mature) to 24-month-old(old) rats although the mature rats responded more vari-ably than the old rats did under Yoke contingencies thedifference was not statistically significant Additionalstudies are needed especially of whether reinforcingvariable behaviors in older populations can facilitatelearning and performance

Gender Male and female rats are equally sensitive toreinforcement of variability (Neuringer amp Huntley 1992van Hest et al 1989) However there was a tendency(not statistically significant) for males to respond over-all more variably than females did and because thesestudies did not control for differences in size and weightof subjects or for other aspects of motivation additionalstudies are necessary

Stage of training Stokes and Balsam (2001) reporteddifferences in variability as a function of whether variabil-ity was initially reinforced early or later in training (seealso Gottlieb 1992) Human subjects generated sequencesof 10 L and R responses per trial on a computer keyboardunder lag 0 and lag 25 contingencies Under lag 0 everytrial ended with reinforcement under lag 25 the currentsequence had to differ from each of the previous 25 Whenlag 25 was experienced at the very beginning of traininglevels of variability at the end of training (when lag 0 wasoperative) were lower than they were when lag 25 was ex-perienced after 50 trials of initial lag 0 These results were

interpreted to suggest that there is an ldquooptimal period forsetting variability levelsrdquo (Stokes amp Balsam 2001 p 181)but that general conclusion needs further testing and anumber of questions need to be answered One has to dowith the contribution of reinforcement frequency Underlag 0 every sequence leads to reinforcement but rein-forcement is intermittent under lag 25 Another has to dowith whether learning to vary differs from learning to repeatwith respect to optimal periods (see eg Bouton 1994)

Related to stage of training are effects of prior experi-ence Hunziker Lee Ferreira da Silva and Caramori(2002) showed that for human subjects if Var contingen-cies were experienced just prior to Yoke contingencieslevels of response variation during the Yoke phase werehigher than they would have been if the Var contingencieshad not been previously experienced The time delay be-tween Var and Yoke experiences mattered with a long in-terval (3 months) producing only a small effect Saldanaand Neuringer (1998) also found effects of prior Var train-ing on performances in Yoke (increased variability inYoke) but the opposite did not hold that is training underYoke had relatively little effect on Var Hunziker Caramorida Silva and Barba (1998) reported similar effects withrats Thus there may be an asymmetry Reinforcement forvarying has relatively strong influences but reinforce-ment without regard to variations does not adversely affectlearning to vary These different effects are not surprisinggiven that Yoke contingencies are permissivemdashreinforce-ment is given whether or not responses varymdashwhereascontingencies under Var can be quite demanding How-ever the results motivate a number of additional ques-tions One is whether effects of Var training differ fromthose observed when repetitions are reinforced Another iswhether direct reinforcement of variable behaviors canhelp animals or people to overcome the detrimental effectsof prior noncontingent experiences such as those that pro-duce learned helplessness (Seligman 1975)

Environmental constraints Stokes (1995) arguedalso that variability increases when behavior is con-strained The term ldquoconstraintrdquo is ambiguous because allreinforcement contingencies exert a form of constraintIn one study Stokes (1995) compared different strate-gies of shaping leverpresses Rats in one group were re-inforced initially for pressing the lever with only theright paw (right only) and eventually for any press (withright or left paw or with any other part of the body) Theorder of training was reversed for a second group ldquoanyrdquofirst and then right only Behavior sequences were morevariable in the right-only-first group leading Stokes(1995) to conclude that early constraints lead to highvariability (see also Stokes Mechner amp Balsam 1999)That is an important hypothesis with many implicationsbut controls are needed before the results can be attrib-uted to the contingencies For example ldquoright onlyrdquo andldquoanyrdquo differed in terms of reinforcement distributions aswell as in the precision of the reinforcement operationthe ldquoanyrdquo reinforcers were provided immediately and au-tomatically by the programming equipment whereas in

OPERANT VARIABILITY 685

the right-only condition a human observer had to decidewhether a response had occurred and then present the re-inforcer necessarily leading to longer and more variedlatencies and a less consistent contingency Furthermorethere are many cases in which increasing constraints suchas those due to a decrease in the number of responses pertrial (Page amp Neuringer 1985) or the requirement of ex-actly four responses on each of two keys (Schwartz 1980)results in decreased variability Thus the contributionsof response-definition constraints and environmentallimitations must be explored further

SummaryThe generality of operant variability is shown by the

robustness of the phenomenon across procedures mea-sures responses and species As with other operant phe-nomena parameters influence the effects

FUNCTIONS

All operants including variability are functional Op-erant responses produce reinforcing consequences Be-cause other behavioral competencies depend on it oper-ant variability is functional in a broader sense as wellSome of these functions will now be described

Operant ConditioningThe potential of a reinforcer to change behavior is lim-

ited by the range of available variations (Hull Langmanamp Glenn 2001 Skinner 1981) and thus whatever pro-duces those variations contributes to the conditioning ofnew operants Much work has been done to identifysources of variation (Balsam Deich Ohyama amp Stokes1998 Segal 1972 Staddon amp Simmelhag 1971) butNeuringer (1993) was first to show that the same rein-forcers that strengthen individual responses can also pro-duce the necessary variability and that these two functionsof reinforcement may work simultaneously Implied isthat when an operant response is being conditioned re-inforcement works not only to strengthen an individualresponse but at the same time to engender the variabilityfrom which the response emerges (That is variability isnot solely a baseline phenomenon elicited or noise)Concurrent reinforcement of variations and individualtarget sequences will be the topic of this section

Pigeons in one experiment and rats in another were re-warded for varying among 16 possible response sequences(four responses per trial L and R operanda lag contin-gencies Neuringer 1993) Concurrently one of the 16designated as the target sequence was reinforced when-ever it occurred Thus reinforcements were given bothfor sequence variations and for a single selected targeta reinforced-variation-and-selection (RVS) procedureWhen the target was ldquoeasyrdquo (eg LLLL) it emergedrapidly from the baseline variations Given that the tar-get was always reinforced reinforcers were obtained morefrequently when the subjects did not vary and only re-peated the target When the target was ldquodifficultrdquo (eg

LLLR) it was not learned As in the choice experimentdescribed above (Neuringer 1992) the relative valuesor difficulties of target versus variations governed theprobability that the target would be learned

However even difficult targets could be conditionedunder the RVS procedure as was shown by another ex-periment in the same series (Neuringer 1993) Sequencevariations were again reinforced concurrently with a dif-ficult target RRLR but now the frequency of reinforce-ments for varying was systematically decreased Satis-fying the variability contingency was intermittentlyreinforced according to a VI schedule and over sessionsreinforcement for varying became less and less frequentThis technique resulted in acquisition of even the mostdifficult target sequence whereas a control group rein-forced for the target alone (without the concurrent rein-forcement for variations) never learned

Reinforcers were obtained more frequently in the RVScondition than in the control condition (since the RVSsubjects were rewarded both for varying and for emittingthe target sequence) and the additional reinforcers mighthave generated greater motivation to respond (Killeen1981) Thus motivation not the direct reinforcement ofvariability might have been responsible for the RVS ef-fect To test this possibility Neuringer Deiss and Olson(2000) reinforced three groups of rats for emitting a par-ticularly difficult target sequence LLRLR The experi-mental group was reinforced concurrently for sequencevariability on the average of once per minute (VI 1 min)The target-only control group received no reinforce-ments other than for the LLRLR target A third groupwas yoked to the experimental animalsmdashreinforced onceper minute for any sequence variability not requiredmdashand as for the other two groups whenever the target oc-curred If the additional reinforcement was responsiblefor the facilitated learning under RVS then because theexperimental and yoked groups received equal reinforce-ments they should have learned equally well but thatwas not the result Only the experimental group learnedto emit the LLRLR target The yoked controls respondedat high rates throughout the experiment but they did notvary sufficiently to make contact with the target contin-gency and never learned The target-only control groupreceived insufficient reinforcement and their responsesextinguished Therefore the added reinforcers helped tomaintain motivation as suggested above but reinforce-ment of variations was necessary for learning of a diffi-cult sequence The RVS procedure maintains both highmotivation and high variation and the two together re-sult in the acquisition of difficult-to-learn responses

Grunow and Neuringer (2002 Experiment 2) studiedone possibly important parameter of the RVS proceduremdashnamely the level of variability When a difficult target re-sponse is being trained does it matter whether reinforcedvariability is low intermediate or high As describedabove each of four groups of rats was reinforced for vari-able response sequences the first group for high vari-ability the fourth group for low variability and the other

686 NEURINGER

two for intermediate levels In other ways the procedurewas similar to that of the Neuringer Deiss and Olson(2000) RVS contingencies just described with a singledifficult target sequence always reinforced The resultswere that the higher the level of reinforced baseline vari-ability the more quickly the target was learned Thusvariability levels indeed mattered and if the just-re-ported effect is found to be general it will be most help-ful to reinforce high variability when difficult-to-learnnew responses are being shaped Other parameters re-quiring study are class definitionsmdashwhich responses areincluded in the class of acceptable variationsmdashand num-ber of instances within the class

One additional effect should be noted When varia-tions among a set of responses are reinforced if one vari-ant is selected for extinction (no longer reinforced) thefrequency of that instance decreases Neuringer (1993)showed this for pigeons responding under a lag-variabilityschedule When one of the 16 sequences was no longerreinforced (with no other change to indicate its status) thefrequency of that sequence decreased Thus reinforce-ment of variations provides the baseline to select for (iestrengthen) and select against (ie weaken) individualinstances

These findings have potentially important implicationsFirst as has been indicated the same reinforcers thatstrengthen particular instances can also reinforce varia-tions That a binary eventmdashreinforce or not reinforcemdashsimultaneously affects variations and selected instancesindicates both the power of reinforcement and a concep-tual problem How reinforcers exert such diverse effectsis a question to which we will return

Second shaping procedures may implicitly reinforcebaseline variations An approximation to a goal responseis reinforced for a period of time but eventually rein-forcement is withheld until a response closer to the goalis obtained It is commonly thought that withholding re-inforcement or mini-extinction induces the necessaryvariations for shaping to proceed as has been indicatedabove An alternative suggested by the present studiesis that organisms learn both to vary (within a set) and to re-peat (selected instances) until one instance is always rein-forced That is the process of shaping might involve re-inforcement both of variations (generally done implicitly)and of targeted instances If so a useful contributor tothe shaping process might be explicit reinforcement ofvariations

Third reinforcement of variations could facilitate ac-quisition of other competencies such as cognitive andmotor skills Siegler (1996) has shown in an important se-ries of studies that childrenrsquos acquisition of complex cog-nitive skills is correlated with high variability Anotherexample is that when motor skills are practiced in a vari-able manner high quality of performance results (Ma-noel amp Connolly 1995 1997 Mechner 1992 Schmidtamp Lee 1999) However it is rare that variability in thesedomains is directly reinforced as part of the training oreducational process

Fourth the RVS procedure might be used in therapeu-tic settings to modify maladaptive and self-injurioushabits Therapists sometimes employ noncontingent re-inforcement procedures to weaken undesirable behaviors(Vollmer Iwata Zarcone Smith amp Mazaleski 1993)The reasoning is that many maladaptive behaviors aremaintained by unintended consequences and that non-contingent rewards may compete with those conse-quences thereby lowering the strength of the unwantedhabit For example when a staff personrsquos attention toself-injurious behavior helps to maintain it providing at-tention noncontingently may decrease the strength of theunwanted target behavior However the RVS proceduremight accomplish the same goal more effectively andwithout undesirable side effects (Vollmer RingdahlRoane amp Marcus 1997) Noncontingent rewards some-times result in unwanted ldquosuperstitiousrdquo behaviors andthat would be avoided under RVS Furthermore RVSwould help to provide the set of baseline behaviors nec-essary to establish desirable new responses

Finally RVS procedures may help to gain control overand eventually to weaken other undesirable target behav-iors The target can be incorporated within a set of moredesirable responsesmdasha new operant class established withthe target as one membermdashand variable instances rein-forced Once variable responding is successfully main-tained including the undesirable target response the lattercan be weakened by selectively withholding reinforcement

Summary The conditioning of operant responses de-pends on both baseline variations and selection of suc-cessive approximations to a goal response The same re-inforcers used to select the approximations can reinforcebaseline variations thereby defining the precise set ofappropriate variants and the desired levels of variationSuch concurrent reinforcement of variations and selec-tions may facilitate the acquisition of difficult-to-learnoperants and selective withholding of reinforcementmay weaken undesirable instances

Creativity and Problem SolvingBehaving in a novel unusual or nonrepeating manner is

part of creative activities (Campbell 1960) Novelty andvariation may arise through combinations of previouslylearned behaviors (Epstein 1996) transformations of thesebehaviors (Boden 1994) direct reinforcement of variabil-ity as described above or some combination thereof Ofcourse neither novelty nor variability alone suffices forcreative outputmdashthe throw of the dice or the toss of acoin are not creative nor are haphazard strokes of a paint-brush on a canvasmdashbut without variation creativity islow Whether the variability required for creativity canbe reinforced is relevant to whether creativity itself canbe reinforced The latter question has led to contentiousdebate with some arguing that reinforcement facilitatescreativity (Cameron amp Pierce 1994 Eisenberger ampCameron 1996 Stokes 2001 Winston amp Baker 1985)and others focusing on detrimental effects of reinforce-ment (Amabile 1983 Deci Koestner amp Ryan 1999)

OPERANT VARIABILITY 687

The debate goes well beyond the present discussion butsome of the research described above is relevant Firstreinforcement contingent on variability generates higherlevels of behavioral variability than does reinforcementcontingent simply on responding as has been shown bythe many studies described above in which Var contin-gencies were compared to Yoke contingencies Similarlyprecise reinforcement contingencies matter for creativity(Cameron amp Pierce 1994 Lepper amp Henderlong 2000)If any product rather than only creative ones is rein-forced creativity tends to fall (just as variability decreasesunder yoke conditions) Second approach to or antici-pation of reinforcement interferes with operant variabil-ity (Cherot et al 1996 McElroy amp Neuringer 1990Neuringer 1991) which again might be related to effectson creativity when an individual focuses on the rein-forcer Note however that although variability decreaseswith proximity to the reinforcer overall levels of vari-ability are much higher when variability is reinforced thanwhen it is not The same may be true for creativity aswell anticipation of reward may interfere with creativeoutput to some extent but at the same time high creativ-ity may depend on its reinforcement

Operant variability may be important for problemsolving as well Here too there is disagreement as towhether training is helpful or not Early studies by Maier(1933) indicated that it is and these found support inlater work (eg Lung amp Dominowski 1985) Somestudies showed that those individuals most likely toldquobreak the setrdquo are also most likely to solve a problem(Dover amp Shore 1991) However a number of studiesfound that practice had no influence (Duncan 1961)

Arnesen (2000) explored a related phenomenon witha rat model One group Var was reinforced for variableinteractions with novel objects and another group Yokewas reinforced independently of such interactions EachVar rat received training with 12 different objects oneper session For example in the presence of a soup canthe rat was reinforced the first time it entered the canbut thereafter a different response was required such asrolling it climbing on it and so forth This procedure issimilar to that of Goetz and Baerrsquos (1973) described aboveexcept that objects changed from session to session Ratsin the Yoke control group had identical experiences in anadjoining identical chamber that contained identical ob-jects but now rewards were not contingent on object ma-nipulation After this training the rats were tested in adifferent room with 30 never-before-experienced ob-jects scattered on the floor Hidden in each was a smallpiece of food When compared to their Yoke controls theVar animals were more active during this test exploredmore manipulated objects more and most importantlydiscovered significantly more food pellets Another con-trol group received no preliminary training with objectsand their performance in the test session did not differfrom that of the yoked controls Therefore when ratswere reinforced for variable interactions with objectsthe result was an increased tendency to explore manip-ulate and discover new sources of reinforcement It

would be of considerable interest to determine whethera similar effect occurs with children

Assessment and Treatment of PsychopathologiesAbnormal levels of behavioral variability characterize

some human psychopathologies as has been shown byclinical observations as well as by research with humansubjects and animal models Few studies have attemptedto reinforce variability directly in such problem popula-tions The main question is whether reinforcement canmodify abnormal levels of variability in the direction ofnormalcy and whether such changes might have associ-ated beneficial effects

Autism Stereotyped responding characterizes indi-viduals with autism (Hertzig amp Shapiro 1990) For ex-ample Baron-Cohen (1992) asked children with autismand others in a control group to hide a penny in one handso that the experimenter could not guess which hand heldthe penny Those with autism were more likely thanthose without to generate a simple predictable patternsuch as repeatedly switching back and forth from left toright hands Such repetitive responding can be maladap-tive For example Mullins and Rincover (1985) askedchildren with and without autism to pick one of five cardsFood was occasionally found in a cup behind the cardEach of the five cards was a discriminative stimulus fora different schedule of reinforcement including contin-uous reinforcement (CRF every choice reinforced) FR 2(fixed ratio 2 in which every second choice was rein-forced) FR 4 (every fourth choice reinforced) FR 7 andFR 11 Control participants sampled all five alternativesand quickly learned to choose the most frequently rein-forced CRF card Autistic children sampled only a lim-ited number of cards and therefore often preferred anonoptimal alternative Thus children with autism tendto respond less variably than normal children and thiscan result in failure to adapt

Miller and Neuringer (2000) asked whether direct re-inforcement could influence the variability of individu-als with autism Sequences of responses on two largebuttons were first reinforced probabilistically and inde-pendently of variability (Prob phase) after which rein-forcement was contingent upon satisfying a thresholdvariability contingency (Var phase) The autistic partic-ipantsrsquo variability was significantly lower than that ofmatched controls during both Prob and Var phases butwhen variability was reinforced its levels increased sig-nificantly in both groups This is an important findingbecause it suggests that direct reinforcement of variabil-ity may help to modify some of the behavioral stereo-types that are characteristic of autism Lee McComasand Jawor (in press) offer support for this hypothesisThree individuals with autism were reinforced for variedverbal responses to social questions the responses had tobe appropriate given the context Variability increased intwo of the three individuals In a related procedureBaker and Koegel (1999) reinforced children with autismfor varied social interaction in a game-playing contextAlthough the reinforcement was permission to engage in

688 NEURINGER

the stereotyped behaviors preferred by the child the re-sult was increased social interactions both during theperiod of the experimental manipulation and in 1- and2-month follow-ups These three studies taken togethersuggest that reinforcement of varied behaviors may facil-itate modification of nonfunctional ritualistic and stereo-typed behaviors

Depression Depression provides another example ofmaladaptive consequences of low variability Lapp Mar-inier and Pihl (1982) observed that depressed womenproduced fewer alternative solutions to hypotheticalproblems than did nondepressed women Channon andBaker (1996) found that attempts by moderately de-pressed college students to identify faults in a series ofinterconnected circuits were less varied and less success-ful that were those of nondepressed controls HorneEvans and Orne (1982) found that when depressed pa-tients were asked to generate a random sequence of num-bers their responses were less variable than were thoseof controls These experimental results are consistentwith clinical observations

Can depressed individuals be rewarded for increasingvariation Hopkinson and Neuringer (in press) dividedcollege students into mildly depressed and nondepressedgroups on the basis of the CES-D scale a paper-and-pencil self-evaluative index of depression In the first oftwo phases sequences of responses on a computer key-board were reinforced independently of variability (Prob)following which only highly variable sequences were rein-forced (Var) Under Prob depressed students respondedsignificantly less variably than did the controls in agree-ment with previous findings Under Var variability in-creased significantly in both groups Indeed by the endof the experiment depressed and nondepressed subjectswere responding equally variably (Instructions were alsoeffective in increasing variability in the depressed par-ticipants) Thus even in a simple computer-game envi-ronment mildly depressed subjects manifested lowervariability than controls did but direct reinforcementmodified their behavior to control levels Can reinforce-ment of variability be used in real-world conditions tohelp depressed individuals An affirmative answer comesfrom cognitive behavior modification procedures thatwork to increase alternative solutions to problems (Beck1976) A more general conclusion from these studies ofautistic and mildly depressed individuals is that direct-reinforcement-of-variability procedures may benefitthose who manifest abnormally low levels of variability

Attention Deficit Hyperactivity Disorder (ADHD)ADHD may provide an example of the opposite stateabnormally high levels of variability (Barkley 1990)Wultz Sagvolden Moser and Moser (1990) proposedthat the spontaneously hypertensive rat (SHR) may serveas a good animal model of ADHD the SHR shows manyof the behavioral characteristics of that disorder includ-ing hyperactivity poor learning ability poor ability totolerate delayed reinforcement and poor self-control

The next study to be described asked about operant vari-ability in SHRs

Mook Jeffrey and Neuringer (1993) compared vari-ability of SHRs and of a control strain Wystar Kyoto(WKY) under two conditions Prob and Var The mainresult was that SHRs tended to respond variably whetheror not the contingencies required it consistently with theresults from the human ADHD literature When vari-ability was reinforced the two strains responded at ap-proximately equally high levels of variation These re-sults were confirmed in two other studies (HunzikerSaldana amp Neuringer 1996 Mook amp Neuringer 1994)Thus control rats varied only when the contingencies re-quired variations whereas SHRs varied whether or notthe contingencies required it As might be expected fromthese results when reinforcement depended on repeti-tions the SHRs performed poorly (Mook et al 1993)

Another similarity between SHRs and ADHD humanswas seen when amphetamine a drug that has effects simi-lar to those of Ritalin often prescribed for ADHD was ad-ministered (Mook amp Neuringer 1994) When repetitionswere required for reinforcement amphetamine improvedperformances by the SHRs so that they were as accurate assaline-injected WKYs a finding again consistent withthose in the ADHD literature These studies support SHRsas a model of ADHD and suggest that additional attemptsshould be made to assess whether or not individuals withADHD are less able to change levels of variability thanare controls Although as indicated above there is someevidence in support of this hypothesis (Barkley 1990)when Saldana and Neuringer (1998) directly comparedhuman ADHD and control participantsrsquo operant variabil-ity within the context of a computer game no differenceswere observed Most of those with ADHD were main-tained on Ritalin and other drugs however a fact unfor-tunately not documented in the original article

Drug EffectsAlthough many drugs are known to influence behavioral

variability (Brugger 1997) rarely have attempts beenmade to analyze explain or identify the sources of theeffects As will be discussed below operant variabilitycan result from the memory-based process of remember-ing previous responses and then behaving differently orcan result from responding stochastically a process thatappears to be quite different McElroy and Neuringer(1990) attempted to distinguish the effects of alcohol onmemory from its effects on stochastic processes in astudy in which one group of rats was reinforced for vary-ing four-response sequences under a lag 5 contingency(Var) and another was reinforced for repeating a singlesequence LLRR (Rep) Ethanol injections of 075 and20 gkg affected Var and Rep performances differentlyWhereas the Rep animals were much less able to satisfythe LLRR contingency at both doses the Var rats re-sponded variably whether or not alcohol had been ad-ministered Similar results were obtained by Cohen et al

OPERANT VARIABILITY 689

(1990) with a multiple schedule in which each rat servedas its own control In the same animals and within thesame sessions alcohol degraded Rep performances butleft Var intact Thus alcohol appears to interfere with atask that requires working memory as presumably is thecase for the Rep task but not with one that reinforcesvariability The memory-interfering effect of alcoholmight account for why alcohol also degrades radial-armmaze performance (Devenport 1983)

Alcohol also affects choices to vary or to repeat For ex-ample Gorka (1993) reinforced rats concurrently for re-peating and varying and monitored percentage of choicesas a function of time following injection of ethanol Inthe control sessions when the rats were injected withsaline the probabilities of varying and repeating remainedapproximately constant across the 45-min session witha slight bias in favor of varying After injection with 15gkg of ethanol however there was a significant increasein choices to vary at the beginning of the session andthen a later decrease to below 50 That is alcohol ledinitially to a markedly increased preference for varyingbut later to a preference for repeating (Figure 5) This re-search should be replicated and extended especially be-cause the contingencies ldquopressuredrdquo the animals to re-main at approximately 50 preference but thesuggestion is that alcoholrsquos effects on response variabil-ity may depend in part on the duration of its influence Amore general implication is that drugs may affect re-peated and variable operants differently and that effectsdepend on time following administration

SummaryReinforced variability may contribute to many types

of learning including operant shaping and acquisition ofcognitive and complex motor skills Similarly creativityand problem solving depend on variable responding andin these areas as well explicit reinforcement of variationsmay be functional Abnormal levels of variability aresometimes caused by psychopathologies and drugs andoperant variability may be useful in these areas in twoways first as a way of assessing the variability that is pos-sible and second to modify variability toward more nor-mal levels Studies show that individuals with autism andmild depression vary less than do controls but that ex-plicit reinforcement of variability moves the levels towardsnormalcy Other studies show that drugs such as alcoholmay affect repeated and variable operants differently

EXPLANATIONS

Two questions must be answered in order to explainoperant variability First what are the sources of vari-ability that permit its reinforcement Second how is itmaintained Related questions including how instancesbecome members of a set from which variations emergeand whether operant variability is a generalizable com-petency will also be considered

Origination From Nonoperant SourcesSkinner (1966) reasoned that operant responses orig-

inate from nonoperant sources with reinforcement con-tacting a behavior shaping and strengthening it onlyafter the behavior had occurred for reasons other than re-inforcement To put it simply a behavior must occur be-fore it can be reinforced (Balsam amp Silver 1994) I willconsider three initial sources of variable behavior en-dogenous variability noncontingent environmental eventsand extinction These appear to parallel evolutionaryprocesses and they too will be identified

Endogenous variability All physical phenomenavary and behavior is no exception This fact can providethe opportunity for reinforcement to select particularlevels of variation Examples of spontaneous variabilityinclude spontaneous alternations in mazes (Dember ampRichman 1989) nondifferentially reinforced variationsin operant responses (Guthrie amp Horton 1946) and ex-ploration (Archer amp Birke 1983) Endogenous muta-tions provide a parallel in genetics Base-pair changes inDNA molecules occur spontaneously intron-bearinggenes mix and match segments to encode new proteinsjumping genes introduce new DNA sequences and atthe level of the chromosome random crossings and as-sortment occur as a natural part of reproductive meiosisThese sources of random genetic change occur indepen-dently of external influences

Figure 5 The relative frequency of variations by rats as a func-tion of 5-min blocks of time since the beginning of the sessionThe open circles represent performance after the rats were in-jected with saline indicating a slight tendency to vary more thanrepeat but one that is approximately constant throughout thesession The filled circles represent performance after injectionwith 15 gkg ethanol indicating a much higher likelihood ofvarying soon after the alcohol is injected followed by a decreasein variations to below 50 and then a return to baseline at theend of the session Group averages are shown Data are fromGorka (1993)

690 NEURINGER

Noncontingent events Noncontingent events some-times referred to as chance accidents or luck also affectbehavior One happens upon a particular passage in a bookthat leads to a new thought one happens to sit next to aninteresting woman or man on an airplane which leads tomarriage (Bandura 1982) In science as well as in every-day life accidents or serendipity are an important part ofthe process of discovery (Beveridge 1957) In each of thecases presented above an accidental event leads to new be-haviors that are then strengthened by consequences Thesame is potentially true for variability itself Particular lev-els of variability might occur for adventitious reasons butthen become functionally related to consequences

That organisms are particularly sensitive to such unan-ticipated noncontingent events is shown by two basicprinciples of learning habituation and Pavlovian condi-tioning Organisms attend preferentially to unexpectedevents and stop attending or habituate to repeated onesThe conditioning of new Pavlovian responses dependsaccording to influential theories upon unexpected rein-forcing events (eg Rescorla amp Wagner 1972)

Noncontingent events such as X-rays and other pro-cesses that can act as mutagens play an important role ingenetic variation as well The particular genes affectedand the type of changes are random there is no neces-sary relationship or contingency between the externalevent and the affected gene However sensitivity to mu-tations is not random but has been selected presumablybecause it is functional Thus in both behavior and ge-netics sensitivity to noncontingent events is conserved

Extinction Extinction increases behavioral variabilityat least to some extent as was described above Distance inspace or time from the reinforcer does the same (Cherotet al 1996) A possible parallel at the genetic level in-volves high levels of deprivation at least in simple lifeforms For example when E coli bacteria are deprivedof nutrients unusually high mutation rates are observedThe exact mechanism by which such hypermutations areinduced is not known but they increase the likelihoodthat a population can survive such environmental insults(Moxon 1997) That is deprivation-induced variability isfunctional at the genetic level (The behavioral evidenceregarding long-term deprivation is less clear some stud-ies showing increased variability and others increasedstereotypy eg Carlton 1962 Senkowski Vogel amp Po-zulp 1978)

Endogenous variations external noncontingent eventsand withholding of reinforcement are three sources forthe variability that precedes control by reinforcementBut once control by reinforcement is established a dif-ferent question emerges What behavioral processes areinvolved when variability is maintained as an operantThe sources just discussed generate uncontrolled non-specific and often short-lived variability and cannot ex-plain the precisely controlled operant type

Operant Variability ProcessesThree processes are hypothesized to underlie operant

variability The first involves the use of random environ-

mental events such as the toss of a coin The second in-volves memory for responses such as when an animallearns not to return to previously visited locations Thethird will be referred to as an endogenous stochasticgenerator

Random environmental events Throughout historymany cultures have used random events to influencechoices and decisions such as throws of dice and randomselections of sticks cards bones or organs Today coinflips are used by a referee at the beginning of a game to de-cide which team kicks the ball a computerrsquos random-number generator is employed by scientists to avoid bi-ases in assigning subjects to experimental groups andaleatoric events are sometimes employed in modern artmusic and literature The Dice Man by Rhinehart (1998)provides a fictional example in which the protagonistbored with life writes a number of possible actions onslips of paper and then periodically selects one blindlyand behaves accordingly These examples show that randomexternal events may be used to avoid biases engender un-likely responses and break out of behavioral niches

Memory-based variability Many methods used tostudy operant variability differentially reinforce nonrepe-titions at least over the recent past Therefore one strat-egy that subjects can employ is to cycle through few re-sponses or sequences especially when the variabilitycontingencies are relatively permissive For exampleunder a lag 2 contingency cycling repeatedly throughthree sequences will result in 100 reinforcement Ani-mals and people sometimes do this apparently basingtheir current response on memory for or discriminativecontrol by the just emitted responses One such instancewas shown by Schoenfeld et al (1966) who rewardedrats for IRTs that fell into one of 10 interval classes onlyif the current IRT differed from the interval class of theprevious response The rats came to alternate betweenlong and short IRTs thus behaving systematically to meetthis lenient variability contingency Machado (1993)studied an analogous effect having pigeons peck L and Rkeys under a frequency-dependent variability contin-gency related to Shimprsquos (1967) procedure describedabove When the trial unit consisted of two responses thebirds developed memory-based repetitive sequencessuch as RRLLRRLL and thereby satisfied the variabilitycontingency When the trial unit was three responses inlength the birds did not develop the optimal fixed pat-tern of RRRLRLLLmdashwhich was too complexmdashbut in-stead developed ldquorandom-like behaviorrdquo (Machado1993 p 103) Thus a memory-based strategy was em-ployed when it was possible but when the memory loadbecame too high stochastic responding emerged A sim-ilar pattern was seen when variability of budgerigarsongs was reinforced (Manabe et al 1997) Under lag 1contingencies the birds sang mainly two songs underlag 2 they increased to three songs Only when the lagwas increased to 3 did song diversity increase apprecia-bly in one of the birds In radial-arm mazes as well whencontextual cues are available memory again appears toplay an important role leading to much lower probabil-

OPERANT VARIABILITY 691

ities of repetition than would be expected from a sto-chastic generator (McElroy amp Neuringer 1990)

Note that under lag schedules memory-based strate-gies result in higher reinforcement rates than respondingstochastically does The reason is that a stochastic gen-erator repeats by chance That animals and people oftendo not employ memory-based strategies probably indi-cates the difficulty of remembering long sequences andpossibly other advantages of unpredictable responding

If a memory strategy is the only way to satisfy vari-ability contingencies then it can be used even when thememory requirements are demanding as was shown byNeuringer and Voss (1993) Human subjects were trainedto generate on a computer keyboard a chaotic-like se-quence of responses that matched iterations of the logistic-difference equation

where Rn refers to the nth iteration in a series with R be-tween 00 and 10 and t is a constant between 10 and40 When the t parameter approaches 40 iterations ofthis equation result in very ldquonoisyrdquo sequences As withother chaotic processes however there is an orderlinessunderlying the surface noise In this case if responsesare autocorrelated with Rn drawn as a function of Rn21the resulting function is a parabola In the Neuringer andVoss (1993) study following each response college stu-dents were shown how close their predicted values wereto that from iterations of the logistic-difference model Inone experiment the students made digital predictions inanother they placed a pointer along a horizontal lineand this analogue response was then translated into dig-ital values for computation With training the studentsbecame increasingly adept at responding in chaotic-likefashionmdashthe studentsrsquo predictions matched closely theiterations of the logistic functionmdashand their autocorrela-tions became parabolic The left column in Figure 6 showsperformances by 4 subjects (rows) early in training thatis not well approximated by the logistic-differenceparabola and the right column shows performances at theend of training in which r 2 values were 72 99 98 andapproaching 10 How was this accomplished Sinceeach iteration in the logistic-difference sequence is basedon the prior outputmdashthat is the sequence is generated bya deterministic although nonlinear processmdashit was hy-pothesized that the human subjectsrsquo chaotic-like responseswere also based on memory for prior responses To put itsimply the subjects may have memorized a long series ofpairs in the form ldquoIf the previous response was value Athen the current response must be value Brdquo as Metzger(1994) Ward and West (1994) and Ward (2002) have sug-gested

To test whether subjects in fact relied on memory forprior responses Neuringer and Voss (2002) systemati-cally slowed responding by interposing pausesmdashthat isby requiring an IRT As IRT increased the difference be-tween the subjectsrsquo sequences and the modelrsquos chaotic

Rn n nt R R= -( )- -1 11

Figure 6 Autocorrelations (responses on trial n on the y-axis asa function of n21 on the x-axis) for 4 human subjects at the be-ginning (left column) and end (right column) of training inNeuringer and Voss (1993) Equations for the best-fitting curveand amount of variance accounted for (r2) are shown at the topof each panel To the extent that the circles (responses) approxi-mate the equation y = 40x 2 40x 2 the subjects were respondingin a way that matched the logistic-difference chaotic functionFrom ldquoApproximating Chaotic Behaviorrdquo by A Neuringer andC Voss 1993 Psychological Science 4 p 115 Copyright 1993 bythe American Psychological Society Reprinted with permission

692 NEURINGER

output increased and the underlying parabolic structurewas broken That is when pauses increased subjects wereless and less able to approximate the modelrsquos chaotic dis-tribution (see Figure 7) The important point here is thathighly variable responses which in this case are chaoticin nature can be based on memorial processes

Endogenous stochastic generator The evidence tobe reviewed next indicates that animals and people canrespond stochastically and that this competency is basedneither on chance environmental events nor on memoryfor prior events or responses It is impossible to provethat an endogenous stochastic process underlies operantvariability that would be analogous to proving the nullhypothesis However much evidence is consistent withthe stochastic hypothesis Included are the facts thatmemory limitations do not permit adequate respondingunder many variability contingencies that response dis-tributions match those expected from a stochastic gener-ator that changes in parameter values affect operantvariability as they would a stochastic generator and thatinterference with memory processes leaves stochasti-cally based operant variability unaffected As a wholethe evidence is consistent with an endogenous stochas-tic source of operant variability

Memory limitations One type of evidence is nega-tive in character and is based on the presumed inability ofmemory for prior responses or sequences to satisfy sometypes of variability contingencies Two examples weregiven above (Machado 1993 Manabe et al 1997) andanother was provided by a lag 50 contingency in Pageand Neuringer (1985) In the latter case the sequence in

the current trial (eight responses across L and R keys) wasrequired to differ from those in each of the previous 50 tri-als including at the beginning of a session the last 50trials from the previous session Pigeons responded wellunder these contingencies and in a way similar to that ofa stochastic model It is unlikely that pigeons can remem-ber each of their previous 50 eight-response sequences

Response slowing As was shown above for chaotic-like responding performances are often degraded by de-lays imposed between a stimulus and a response pre-sumably because of limitations in working or short-termmemory (Berryman Cumming amp Nevin 1963 Williams1971) Neuringer (1991) relied on this effect to test whethermemorial processes could explain the generation of vari-able sequences under lag schedules Two groups of ratswere reinforced for L and R leverpresses producing fourresponses per trial One group (Var) was trained under alag 5 contingency and the other (Rep) was trained to asingle sequence of responses LLRR It was assumedthat accurate Rep performance depended on workingmemory After performances had stabilized IRTs weresystematically varied by the imposition of timeouts of05 to 20 sec between consecutive responses in the se-quence During these forced pauses the chamber wasdark and responses were not counted toward completionof the sequence If the animals had used the previous re-sponse(s) as a discriminative cue for the current responsethen performances should have been degraded by theseinterpolated pauses

The results were that for the Rep group as the re-quired pauses increased from 0 to 6 sec percentages ofcorrect sequences were relatively unaffected but at pausesgreater than 6 sec probability of a correct LLRR se-quence fell sharply This pattern of results suggests in-volvement of working memory of prior responses withthe rats able to remember responses within a 6-sec win-dow The Var grouprsquos results were quite different Aspauses increased between 0 and 6 sec the rats were moreand more likely to satisfy the variability contingenciesand with pauses greater than 6 sec percent correct re-mained at a high and constant level Thus the Var grouprsquosresults were opposite to those of the Rep group and op-posite to those predicted from a memory-based strategySimilar results showing that operant variability increasesor is maintained as responding is slowed have been re-ported by Morris (1987) with pigeons and by Baddeley(1966) with random response generation by human sub-jects The results reported above concerning alcoholrsquos ef-fects on variability and repetition were also consistentwith those of Neuringer (1991)

Memory for prior responses may not have controlled theVar grouprsquos performance but why did variability increaseEach of three answers is consistent with a stochastic-generator interpretation Weiss (1964 1965) hypothe-sized that random responding requires that current re-sponses be independent of previous ones memory for orcontrol by prior responses interferes with such indepen-

Figure 7 Approximation to a chaotic sequence (given by delta)as a function of the imposed interresponse times Shown are theaverages of 5 subjects

OPERANT VARIABILITY 693

dence and therefore pauses would be expected to facil-itate variability A second possibility is that at shortIRTs animals tend to repeat responses on the sameoperandum Blough (1966) found this in pigeons and ex-cluded such double pecks from his analyses because theyappeared not to be under the control of reinforcement con-tingencies Morris (1987) also found a tendency for birdsto repeat when no interresponse timeouts were imposedA third hypothesis is that there were two contributors tothe observed variability One was a stochastic processcontrolled by reinforcement and the other elicited vari-ability generated by interposition of pauses The high vari-ability elicited as an effect of slowed responding is ageneral phenomenon supported in many other cases Ac-cording to this interpretation operant variability in the Vargroup was governed by a stochastic-based process operantrepetition in the Rep group was governed by a memory-based process and pauses elicited variability under bothcontingencies Thus Rep performance was interferedwith whereas Var was facilitated Each of these hypothe-ses is consistent with the conclusion that memory for (ordiscriminative control by) prior responses does not con-tribute to and possibly interferes with variable respond-ing when an organism is reinforced for variability andthis in turn is consistent with the stochastic-generatorhypothesis

Stochastic response distributions Other evidence iscorrelational Response distributions are similar to thosefrom a stochastic generator The data from two experi-ments already discussed are shown in Figure 1 In thesecases reinforcement of least frequent IRTs (Blough 1966)and of nonrepetitive choice sequences (Machado 1989)resulted in stochastic-like responding

Neuringer (1986) reported another striking case Highschool and college students generated sequences of 1sand 2s on a computer keyboard with each set of 100 re-sponses constituting one trial A baseline phase lastedfor 60 such trials for a total of 6000 responses duringwhich the students were asked to behave as randomly aspossible as if they were reporting the outcome of a se-ries of coin tosses As in the many previous studies ofthis sort the subjectsrsquo responses differed significantlyfrom those of a stochastic model (Brugger 1997 Wage-naar 1972) During the training phase the subjects re-ceived feedback which enabled them to compare their per-formances with that of a stochastic model first accordingto one statistical test and then according to another untilfollowing each trial feedback was provided from 10 dif-ferent statistics The results were that all of the subjectslearned to approximate the stochastic model accordingto the 10 statistics as well as some (but not all) other sta-tistical tests That is the distributions of the statisticswhich clearly did not match a stochastic model at the be-ginning of training approximated the stochastic modelat the end of it (see Figure 8)

The previous failures of other researchers to obtain ev-idence for random responding had led some to concludethat humans did not have the capacity to behave ran-

domly What would account for the difference betweenthe present positive finding and the many other negativeones Neuringerrsquos (1986) study was the first in which sto-chastic responding was explicitly reinforced in this casewith feedback from 10 statistical tests An analogy mighthelp to explain the difference in procedures and resultsImagine an experimenter walking around a city and ask-ing people if they know what a violin sounds like (analo-gous to asking people whether they know what a series ofcoin tosses or dice throws is like) Most would answeryes If the same individuals were then asked to play theviolin very few if any would succeed The experimentermight conclude after many replications with differentsubjects in different cities that people did not have thecapacity to play the violin The point of course is thatplaying the violin is a skill requiring training and prac-tice and the same may be true for responding randomlyIt is easy to vary but practice and reinforcement may benecessary to approximate a random distribution

Response number Page and Neuringer (1985) testedthe stochastic-generator hypothesis in another way byvarying number of responses per trial with pigeons work-ing under a lag 3 contingency The reasoning was that ifprevious responses served as a discriminative cue forwhat not to domdasha memory-based strategymdashthen perfor-mance should be degraded when the number of re-sponses was increased Eight responses per trial requiresubjects to remember more than four responses The sto-chastic hypothesis predicts the opposite outcome as isdemonstrated by the following example If each trialwere two responses in length with responses determinedby the toss of a coin then the probability that a given trialwould repeat the previous one would be 25 (There arefour possible sequences in the first trial RR RL LRand LL Thus the second trial has a 1-in-4 chance ofmatching the first) If a trial consists of four coin tossesthe probability of a repetition by chance is 0625 or 1 in16 Thus if subjects used a stochastic process to gener-ate Ls and Rs performances should improve with in-creasing responses per trial Results were precisely thosepredicted by the stochastic hypothesis and inconsistentwith a memory strategy Probability of meeting the con-tingencies and therefore probability of reinforcementincreased as responses per trial increased In otherwords eight-response trials were easier for pigeons thanfour-response trials

Response interference Neuringer and Voss (2002)attempted to distinguish between memorial and stochas-tic processes in yet another way They hypothesized thatwhereas chaotic responding was based on memory forprior responses (as was discussed above) responding inNeuringerrsquos (1986) study with feedback from 10 statisti-cal tests indicated involvement of an endogenous sto-chastic generator To show that human subjects could en-gage in both processes at will three college studentswere trained to generate chaotic sequences in one compo-nent of a multiple schedule as in Neuringer and Voss(1993) and to generate stochastic sequences in a sepa-

694 NEURINGER

rate component as in Neuringer (1986) The wordCHAOTICS was on the screen in the first component theword STOCHASTICS was on during the other and thesewords alternated throughout the session After the sub-jects succeeded at each of the component tasks test ses-sions were introduced in which memory for prior re-sponses was subject to interference in both the chaoticsand stochastics components as follows During the

chaotics component four separate chaotic sequenceswere cycled one after the other each cued by differentscreen colors The subjects attempted to approximate achaotic sequence in one color (eg light green) follow-ing which they would have to match a second chaotic se-quence (lavender screen color) and so on the fourchaotic contingencies following one another across thesession Although all four chaotic sequences used the

Figure 8 Performance by one subject at the end of training to respond stochastically in Neuringer(1986) Shown are comparisons on 6 different statistics between the subject (solid lines and Xs) andthe random model (dashed lines and closed circles) At the end of training the subjectrsquos perfor-mance did not differ statistically from that of the random model on these and four additional sta-tistics From ldquoCan people behave lsquorandomlyrsquo The role of feedbackrdquo by A Neuringer 1986 Jour-nal of Experimental Psychology General 115 p 71 Copyright 1986 by the American PsychologicalAssociation Reprinted with permission

OPERANT VARIABILITY 695

same logistic-difference equation that a model did theywere out of phase and independent If chaotic respond-ing is memory based then the four independent se-quences would be expected to interfere with one anotherthat is approximations to the four chaotic models wouldbe lower than when only a single chaotic output was re-quired as in Neuringer and Voss (1993) That was ex-actly the finding Performances by the three subjectswere signif icantly degraded during the four-segmentphase To put it simply chaotic responses interfered withone another

A different pattern of results was seen in the stochasticsportion of the experiment The same interference proce-dure was used with four independent stochastic sequencesrequired in the four different color conditions As inNeuringer (1986) subjects were required to satisfy eightdifferent statistical tests of randomness and to do this in-dependently in each of the four phases (If responses in anytwo or more of the four stochastic phases became corre-lated above a minimal level responses were considered tobe incorrect thereby prohibiting repetition of a single se-quence in two or more of the phases) For one subjectseven of eight statistics were closer to those of a randommodel during this interference phase than they were at theend of the original training with only a single sequenceand for a second subject all eight statistics were closerThe third subject showed closer-to-random approximationunder the four-segment condition on only two of the eightstatistics but most of her changes were also in the directionof a random generator For example if a random generatoremits equal numbers of ldquouprdquo and ldquodownrdquo sequences andthis subject in the baseline phase had emitted only 47ldquouprdquo sequences then during the four-segment tests shecame to emit ldquouprdquo 55 of the time She changed in the di-rection of the random model but overshot the mark Thuswhereas interference degraded chaotic performance sig-nificantly in all three subjects it improved stochastic per-formances in at least two of them These results are con-sistent with the hypothesis that a memory-based processcontrols chaotic variability and that a stochastic processnot dependent on memory controls stochastic variability

Summary of Tripartite TheoryThe following three processes constitute a tripartite

theory of operant variability attention to random events inthe environment (coin toss) memorial processes (chaosradial maze) and stochastic processes (response se-quences and random-number generation with feedback)Random environmental events assure unpredictabilitymdashfor example in making unbiased selections Remember-ing what was done previously will enable one to avoidrepetitions to vary the students called upon in class tovary the locations when looking for a hidden object orto try different approaches when attempting to solve aproblem External aids such as notes to oneself or sci-entific records help to enhance memory capacities

An endogenous stochastic generator provides the thirdand most controversial source of behavioral variability

When an opponent is attempting to predict your behav-ior in a game or at war it may be functional to behavestochastically Similarly operant conditioning of new re-sponses may depend on stochastic variations It is im-possible to prove stochasticity some presently unknowndeterministic system may ultimately explain the resultsHowever the evidence is consistent with the stochastichypothesis and inconsistent with the alternatives For ex-ample memory is not sufficiently powerful to explain alloperant variability memory-interference proceduresleave stochastic responding intact and behaviors corre-late well with those of stochastic models

Both genetics and reinforcement play important rolesin all three processes Reinforcement is always involvedin that learning provides information about when whatwhere and how much to vary Genetic influences areshown in that not all responses are possible and if theyare possible they are not all equiprobable (eg as inNeuringer et al 2001) Stochastic responding may be bi-ased due to experiences and knowledge that is biasesmay be established by reinforcement contingencies or bythe interface between the organismrsquos propensities (or be-havior systems see Timberlake amp Lucas 1989) and theenvironment In the case of random-number generationby human subjects biases may be due to misconceptionsconcerning what ldquorandomrdquo means However some bi-ases can be overcome through feedback and instructions(Lopes amp Oden 1987 Neuringer 1986)

The importance of variations for evolutionary changeis supported by the many sources of genetic variabilityand by the fact that genetic variability is conserved Asimilar argument holds for operant variability That thereare at least three independent sources of operant varia-tions as well as other elicitors such as environmentalnoise and response extinction supports its importancefor normal adaptive functioning

Structure of VariabilityEvery operant ldquoresponserdquo is actually an instance of a

class with each member of the class exerting the sameeffect on the environment (Skinner 1959) When a rat isreinforced for pressing a lever for example the class isdefined by closure of a microswitch and any way of ac-complishing the closure whether with the left paw theright paw or the mouth is as effective in producing re-inforcement as any other If the contingencies of rein-forcement so specify however the response class can beredefined and circumscribed which is another charac-teristic of the operant For example if only right-pawpresses are acceptable members of the reinforced classresponding will tend to occur only with the right paw(Stokes 1995) Of course the capabilities of the organismplay a role for example including ldquotail pressesrdquo in the op-erant class does not mean that the lever would be or couldbe pressed by the ratrsquos tail The structure of the operantclass therefore emerges from the current reinforcementcontingencies and the organismrsquos capability derivedfrom genetics development and prior experiences

696 NEURINGER

Operant variability has structure as well and as for alloperants that structure is due partly to the organism it-self and partly to the reinforcement contingencies Theresearch described throughout this article has providedexamples of the structure of variability generally in theform of sets of four or eight responses across L and Roperanda But the structure of variations can be speci-f ied more precisely For example Mook and her col-leagues (Mook et al 1993 Mook amp Neuringer 1994)reinforced rats for varying four response sequences acrossL and R levers under a lag 1 contingency Unlike in pre-vious studies however the acceptable class was furtherconstrained so that only sequences beginning with two Rresponses (RRLL RRLR RRRL and RRRR) were rein-forced The rats learned to vary their sequences withinthe imposed limitations that is they generally begantheir sequences with two R responses and varied the se-quences within those limitations Similar results wereobtained in the context of a radial-arm maze in whichonly a subset of arms was baited

The precision with which reinforcement contingen-cies can specify class was highlighted in research per-formed by Burns (1997) and Ross and Neuringer (2002)In the latter case college students drew rectangles on acomputer screen with the computerrsquos mouse Each rec-tangle constituted one trial (the screen cleared at the endof each trial) with reinforcement (points) depending onsimultaneous variations along three dimensions of therectangle location on the screen area of the rectangleand shape with the latter defined as the ratio of the rec-tanglersquos height to its width Although the subjects weregiven no instruction other than to ldquogain pointsrdquo theylearned to vary along all three dimensions with levels ofvariability significantly higher than those of yoked controlsubjects who were reinforced independently of variations

A second part of the Ross and Neuringer (2002) studydemonstrated even greater precision of control To gainpoints three new groups of subjects were required to re-peat along one of the dimensions while simultaneouslyvarying along the other two One group was required torepeatedly draw rectangles of approximately the samesize while simultaneously varying location and shapeanother group had to repeat the shape while varying areaand location and the third to repeat location while vary-ing size and shape Verbal instructions were minimal asthey were in the first part of the study Subjects learnedto satisfy these contingencies as well varying two at-tributes while simultaneously repeating one and signif-icant differences were obtained rapidly within 300 trials

These demonstrations raise questions as to how binaryfeedbackmdashreinforcement presented or notmdashcan simul-taneously control independent behavioral dimensionsThe issue is similar to that discussed above concerningsimultaneous reinforcement of variations and selectionsIt is similar also to those issues faced by researchers at-tempting to understand how the nervous system strength-ens particular synapses when many different neurons firesimultaneously and to ldquoallocation of creditrdquo in artificial

learning models For present purposes suffice it to notethe exquisite precision with which reinforcement controlsvariability and repetition

GeneralizationDo animals and people learn to ldquovaryrdquo implying a

generalizable skill or do they learn a specific distributionof specific responses across particular operanda Thequestion of generalization has been asked of many otherbehavioral competencies such as imitation When chil-dren learn to imitate do they learn stimulusndashresponseassociations (if mother moves her right hand the childlearns to move his right hand) or do they learn moregenerally to ldquoimitaterdquo Evidence indicates that reinforce-ment of imitative instances leads to generalized imitation(see eg Baer amp Sherman 1964 Bandura 1986) A sim-ilar question has been raised when pigeons rats or chil-dren learn to choose a stimulus that matches a previouslyshown sample For example if a pigeon is shown a reddisk as a sample it must then choose red when given achoice between that and blue Does the bird learn to matchIn this case the answer appears to depend on the train-ing procedures If trained with few stimuli such as redgreen and blue then the pigeon learns simple associa-tions (if red sample choose red if blue sample chooseblue) but if hundreds of different stimuli (pictures) areemployed during training then generalized ldquomatchingrdquois learned (Wright 1997) In each of these examples thelearning of a general competency or strategy was shownby transfer-of-training tests What was learned with oneset of stimuli or in one situation transferred to new stim-uli and situations We turn now to the question of whetherlearning to vary transfers

After the subjects in Maltzmanrsquos (1960) experimentdescribed above learned to generate remote verbal as-sociates they were asked to devise unusual uses of com-mon objects In comparison with those in the controlgroup these subjects generated significantly more un-usual uses indicating transfer of training to a differenttask However transfer was not observed in the oppositedirection from unusual uses to verbal associates

Holman et al (1977) using a procedure similar to thatof the Goetz and Baer (1973) research described aboverewarded preschool children verbally for novel drawingsand found increased novelty Unrewarded block construc-tions were also monitored and weak evidence was ob-tained for transfer of novel responding from rewardeddrawing to the unrewarded constructions In a second ex-periment preschoolers engaged in four independent taskspainting pen drawing wood block construction and Legoconstruction When the children were reinforced for vary-ing their pen drawings the novelty of paintings increasedalong with the pen drawings indicating transfer from onetask to the other However transfer to the topographicallydifferent block- or Lego-building tasks was not observed

Eisenberger and Armeli (1997) gave monetary rewardsto one group of fifth- and sixth-grade students the vari-ability group for describing unusual uses of common ob-

OPERANT VARIABILITY 697

jects A second group the repeat group received thesame rewards for describing common uses of the sameobjects In a later test all of the children were asked todraw pictures incorporating a circle and the group that hadpreviously been reinforced for varied responses drewmore unusual (or lower frequency) pictures than did therepeat group Thus there was transfer of novelty trainingfrom one domain to another with additional studies in theseries showing similar effects (Eisenberger Armeli ampPretz 1998 Eisenberger Haskins amp Gambleton 1999Eisenberger amp Selbst 1994) However Eisenberger et al(1999) failed to replicate the transfer-of-training effectThe variability and repeat groups responded similarly dur-ing the test phase In this study separate experimenterstrained and tested the subjects thereby avoiding possibleexperimenter-expectancy effects whereas the same ex-perimenter did both training and testing in Eisenbergerrsquosother studies Across the series of experiments there weremany other cases in which transfer was not observed andmore research is necessary to explain why

In Lee et alrsquos (in press) study with autistic individualsdescribed above after variable answers were trained toquestions verbal variability transferred to new experi-menters and new contexts

Arnesenrsquos (2000) results described above are consis-tent with transfer of training in an animal model Rats inthe Var group were trained to interact variably with oneset of objects When later tested in a new situation withnew objects the Var animals showed (in comparisonwith controls) higher levels of exploration manipulationof the novel objects and discovery of hidden reinforcers

Much of the evidence reviewed in this section there-fore indicates transfer of training and supports a generalldquovaryrdquo competency However there remain unresolvedquestions the most important of which is the extent towhich transfer occurs Note also a methodological pointMany of the studies tested whether transfer occurs whenthe contingencies are neutral with respect to the to-be-transferred behavior Holman et al (1977) is a case inpoint Variations were reinforced in one domain draw-ing and the question was whether variations wouldoccur in another domainmdashblock constructionmdashin whichreinforcement did not depend on varying A negativef indingmdashin this case absence of transfermdashindicatesnothing about the ability to generalize but only aboutthe tendency to do so given the absence of a reason (orcontingency) A better test would be to compare experi-mental and control performances when variability in thenew task is in fact functional as in Arnesen (2000)

SummaryBefore variability can be reinforced it must occur for

reasons other than reinforcement Some of the reasonsinclude variability inherent in all behavioral phenomenaldquoaccidentalrdquo environmental events that are unrelated toparticular behaviors and withholding of reinforcementTransfer from other situations may serve a similar functionFor example if an organism can discriminate shaping-

of-new-response periods from others then it may learngenerally to ldquovaryrdquo during shaping episodes After rein-forcement makes initial contact with response variationsthere are at least three behavioral strategies that can beemployed in order to maintain functionally high levelsof variability respond based on external random eventssuch as the toss of a coin avoid repetitions by remem-bering what responses had previously been made and re-spond stochastically Evidence for an endogenous sto-chastic process was seen in the similarity of responsedistributions to those generated by stochastic sourcesand by findings that interfering with memory for priorresponses either improves or leaves unaffected stochas-tic behaviors Stochastic responding as well as othertypes of variability-generating strategies depend uponcontingencies to define the required level of variabilityand the set from which variations emerge

IMPLICATIONS

One implication of reinforced variability is that oper-ant responses may differ in the extent to which their vari-ability is affected by reinforcement contingenciesmdashthatis operants may be ordered along a continuum of sensi-tivity of variability to reinforcement Another is that re-inforced variability causes us to question the generalityof some basic goals of behavioral psychology includingprediction and control A third is that reinforced vari-ability may help us to provide a scientifically useful wayto describe voluntary action In particular the variabilityof all voluntary behaviors can readily be changed mo-ment to moment by discriminative stimuli and conse-quences Implications for everyday life also will be noted

Operant ContinuumAs was indicated above the operant is composed of

instances that is the operant is a class of many differentresponses all functionally equivalent but often topo-graphically distinct Skinner suggested and I hypothe-size that instances within the class emerge stochastically(see Moxley 1997 for a discussion of Skinnerrsquos posi-tion) Although reinforcement contingencies and contextinfluence the individual probabilities and their distribu-tions emission is stochastic nonetheless The particularmember of the operant class generally cannot be predictedwith a high degree of certainty For example a particu-lar rat may be more likely to press the lever with its leftpaw than its right but left- and right-paw responses areemitted probabilistically This class-nature of the operantsuggests one way to conceptualize operant variability interms of size of class and within-classes probability dis-tributions High variability implies large classes equalprobability distributions or both low variability impliesthe opposite

Variability may be modified more readily in some op-erants than in others implying that operant variability isnot an all-or-none phenomenon but rather a continuumwith reinforced variations in variability (RVV) differing

698 NEURINGER

for different operants RVV might best be understood byanalogy to the concepts of speed and acceleration Bothvariability and speed range from low to high RVV isanalogous to acceleration the rate of change in speedHighly sensitive RVV is indicated by the possibility ofreinforcing many different levels of variability from rep-etition to random-like responding and by the fact thatchanges in level occur readily and rapidly

Different operants manifest differences in RVV Forexample variability of chain pulls changed more thanthat of leverpresses when variability was reinforced inrats (Morgan amp Neuringer 1990) Some operant-like re-sponses may show relatively low RVV For example fir-ing rates increase in single hippocampal nerve cellswhen dopaminergic substances are contingent upon theresponse (Stein Xue amp Belluzzi 1994) suggesting thatnerve f iring is controlled by reinforcement To datehowever there is no evidence that variability of nervefiring is similarly controlled Other cases might includereinforcement of nondiscriminable (subconscious) re-sponses such as Hefferline and Pererarsquos (1963) reinforce-ment of subaware thumb twitches and biofeedback ofmuscle and cardiovascular activity In these cases rein-forcement may modify performance but it is not knownwhether variability is also sensitive to reinforcement andif so to what degree Human verbal variability may bethe most readily manipulable of all behaviors the rangeof predictable to unpredictable verbal responses beingimmense with extraordinary sensitivity to context andconsequence RVV may also differ across organisms (per-haps from simple to complex) individuals (perhaps asan index of intelligence or learning ability) and physio-logical and psychological states (WKY vs SHR depres-sion and autism)

RVV may help to distinguish operant responses fromother superficially similar cases of selection by conse-quences That is the ldquooperantrdquo may also best be concep-tualized as a continuum with RVV an important deter-miner of position along the operant continuum Oneexample of selection by consequences but low ldquooperant-nessrdquo is genetic change Variability is essential for evolu-tionary change of course and levels of genetic variabil-ity are sometimes affected by environmental conditions(Moxon 1997) but there are few examples of levels ofvariability being influenced by selection pressures Thatis the source of much genetic variability is thought to berandom not influenced by current pressures and needsThus although selected by consequences genetic changemay be low along the operant continuum Other exam-ples of operant-like phenomena for which variability although important may be relatively insensitive to con-sequencesmdashvariability in these cases is elicited not se-lectedmdashinclude the development of the central nervoussystem and immune system responses (Hull et al 2001)In both of these cases change depends upon selection byconsequences but it is not clear whether variability canalso be selected Most commonly studied operant re-sponses on the other hand show high sensitivity of vari-

ability to reinforcing events But to reiterate the mainpoint of this section operants differ in terms of RVV andtherefore in terms of their ldquooperantnessrdquo from low sensi-tivity (possibly responses of single nerve cells) to extra-ordinarily high sensitivity (verbal behavior) The operantnature of variability as well as that of every other oper-ant dimension is best conceptualized as a continuum

Prediction and ControlSmith (1992) described Skinnerian operant condition-

ing theory as Baconian in nature a primary goal of Skin-nerian behaviorists being to control behavior and hecontrasted this with the more Aristotelian goal of de-scription Skinner and his students (eg Zuriff 1985) in-deed emphasized prediction and control Many of thestudies reviewed above however involved reinforce-ment procedures that increase variability and conse-quently decrease predictability It is not only ignorancetherefore that restricts prediction and control as Skin-ner suggested but reinforcement itself Operant meth-ods and theories when applied to variability lead togreater emphasis on description with control over pre-cise response characteristics possible under some butnot all conditions

In summary an operant is a class of responses underthe control of contingencies of reinforcement The classcan be controlled and predicted given knowledge ofconditioning history Indeed the size of the class andpossibly the distribution of within-classes probabilities orlevel of variability can also be controlled and predictedBut to the extent that instances emerge stochastically aparticular instance cannot be predicted The instant re-sponse can be explained as a member of a particularclass but prediction of the instance will be diff icultsometimes to the extreme This point will be developedin the next section

Voluntary BehaviorI hypothesize that voluntary behaviors are operants for

which RVVs are highly sensitive to influence by conse-quences One way to test this hypothesis would be togather empirical data concerning how voluntary versusinvoluntary actions are classified by human observers Asfar as I know there are no psychophysical data concern-ing this question but the hypothesis implies that highsensitivity of variability to operant contingencies RVVswill be present in all actions perceived to be voluntary

This operant-variability theory of voluntary action isbased on the supposition of two attributes First volun-tary acts must be potentially explainable (given suff i-cient knowledge) on the basis of genetics conditioninghistories current contingencies anticipated conse-quences intentions goals and the like That is volun-tary behaviors are lawful The second attribute is that thevoluntary act must be potentially unpredictable That isvoluntary acts are ldquofreerdquo of control by identifiable stim-uli If behavior is perceived necessarily to result frommdashto be predicted on the basis ofmdashexternal events includ-

OPERANT VARIABILITY 699

ing reinforcement contingencies it will not be perceivedas voluntary ldquoVoluntaryrdquo implies potential indepen-dence from stimulusndashresponse or responsendashreinforcerdetermination According to this theory both explain-ability and unpredictability are necessary attributes ofvoluntary acts

How though can behaviors be simultaneously ex-plainable and unpredictable Reinforcement contingen-cies play two essential roles Reinforcement (togetherwith genes prior experiences and current environment)helps to determine the set of possible behaviors fromwhich a voluntary act emergesmdashthat is the operant classReinforcement (together with the other contributorsspecified above) also helps to determine the level of vari-ability within the class Thus both instances comprisinga class and level of variability within that class can bepredictedmdashagain given sufficient knowledge Howeverthe individual response is at least sometimes (or relatively)unpredictable The reason is that the operant class fromwhich the response stochastically emerges is sometimesextremely large (eg ldquoname a wordrdquo or ldquodo somethingrdquo)and the contingencies sometimes call for equiprobablestochastic emergence (eg ldquofool merdquo) In the latter casea voluntary act may appear to be foolish meaninglessor unwarranted but if it is voluntary it can potentially beexplained (For example in response to the questionldquoWhy did you do that silly actrdquo the answer could beldquoJust to show that you canrsquot predict my behaviorrdquo seeScriven 1965) In these cases prediction of instancesfalls to very low levels indeed and I hypothesize evenan all-knowing observer will be unable to predict thenext response at greater-than-chance levels which aregiven by the size of the operant class and by within-classesprobability distribution Even within small classes (egin the game rockndashscissorsndashpaper in which unpredictabil-ity among the three possible responses is functional) be-havioral stochasticity may lead to unpredictability of theinstant response Note however that although voluntarybehaviors are sometimes unpredictable the opposite isoften the case When contingencies call for highly con-strained particular responses predictions can easily bemade For example if a police officer asks your nameyour response can most likely be predicted Voluntary re-sponses are therefore described as potentially unpre-dictable with variability contingencies engendering dif-ferent levels of unpredictability

According to this theory behaviors low in the sensitivity-of-variability dimension are described as relatively in-voluntary and those that are high in said dimension areperceived as voluntary Low RVV may be shown at anylevel along the variability continuum For example al-ways responding in the same unchanging way will beperceived as involuntary but so will always respondingrandomly (The coin does not voluntarily come up withheads or tails) This point is essential More than 2000years ago Epicurus argued that occasional randomswerves of atoms provided the source of voluntary be-havior in an otherwise determined universe but al-

though this conjecture was prescient it was not quitecorrect ldquoFree-willedrdquo or voluntary action whether it isbased on random swerves or on quantum mechanics isnot explained by variability per se but by functionalchanges in levels of variabilitymdashthat is by rapid andprecise changes in response to environmental demandsResponses can be placed along a voluntaryndashinvoluntarycontinuum isomorphic to the operantndashrespondent con-tinuum with voluntary behavior being functional pre-cisely in the same way as operant behavior is ThusSkinnerrsquos (1974) suggestion that voluntary behavior isoperant behavior was almost correct Only those oper-ants for which variability is highly sensitive to conse-quences will be perceived as voluntary

One additional example might help Operant classes(sets of potential responses) are created ldquoon linerdquo as we goabout our daily activities As we walk from car to officewe tend to walk in a characteristic way take the samepath with tightly constrained operant classes followingone after the other in rapid succession Class sizes aresmall and within-classes probability distributions arehighly peaked Predictions of next responses by a knowl-edgeable observer can readily be made However classsizes can be increased and probability distributions flat-tened we can walk in quite strange ways indeed dancealong a new path sing hand candy to passers-by Strangebehavior indeed but made possible by the ability of op-erant sets to be modified moment to moment not only interms of class definition but in terms of class size lev-els of within-classes variations and so on The same istrue for conversation Moment-to-moment changes inverbal class definition size and predictability of within-classes instances are hallmarks of voluntary behavior

GenerativityRelated to the definition of the operant and to our discus-

sion of voluntary action is that fact that operant behaviormdashleverpressing no less than languagemdashis by its naturegenerative This claim follows from the functional na-ture of the class of responses (all members of the classare effective in producing an outcome) and the stochas-tic emission of instances From these flow never-before-emitted instances patterns and sequences (see Epstein1996) Operants especially those high on the RVV con-tinuum are inherently generative

Self-ExperimentationVariations systematic or otherwise are a necessary part

of scientific work Neuringer (1981) and Roberts and Neu-ringer (1998) described how variations can be brought to bear explicitly on everyday life in the form of self-experiments Individuals vary aspects of their lives suchas the foods they eat social contacts and methods ofmemorizing they keep records concerning the effects ofthese manipulations and form hypotheses concerningcontrolling variables Using self-experimental methodsindividuals can learn how what when where and howmuch to vary Explicitly varying onersquos everyday behav-

700 NEURINGER

iors and thoughts may help to engender what Langer(1989 2000) has referred to as mindful livingmdashthat isnonautomatic voluntary engagement with beneficialeffects in different areas including learning The impor-tant point here is that variability can be voluntarily cho-sen maintained and modified in everyday life and thatexplicit variations may have important consequences

Implications for SocietyConsideration should be given to the importance of re-

inforcement contingencies to support variability in so-cial milieus A basic point made throughout the presentarticle is that merely permitting behavior to vary doesnot necessarily engender or maximize variability (as isshown by Yoke and Prob schedules) Analogously merelypermitting varied social behavior as in a free or laissez-faire society may not suffice to produce the desired di-versity If the results shown for individual organismscan provide useful information for groups (see eg Grottamp Neuringer 1974) then when variations are desiredsocieties might reward individuals for varying not sim-ply permit it

SummaryOperants differ in the extent to which reinforcers in-

fluence levels of variation When variability is highlysensitive to reinforcement the operant may be perceivedto be voluntary Voluntary responses have two attributesThey are functional implying that the class of responsesmdashthe operantmdashcan be predicted given sufficient knowl-edge But instances emerge stochastically implyingsome degree of unpredictability Because reinforcementof variability can generate extremely large classes withstochastic emergence of instances it suggests limitationsof the two basic goals of Skinnerian behavioral theorymdashnamely prediction and control Operant variability also hasimplications for self-experimentation and for society

RELATED AREAS

Operant variability may be important in many otherareas and I will briefly review some of these

Negative reinforcement To my knowledge there havebeen no published studies in which variability is nega-tively reinforced High variability might sometimesavoid or reduce a punisher such as loss of goods ormoney but at other times low variability could be func-tional Effects of positive and negative reinforcement onvariability may have different evolutionary bases onederived from increasing probabilities of locating sourcesof food or mates the other from avoiding confrontationsor predation (Driver amp Humphries 1988) One questionconcerns how quickly variability can be changed by neg-ative as opposed to positive reinforcement There isconsiderable evidence that loss functions differ fromgain functions (Kahneman amp Tversky 1984) possiblyindicating that positive reinforcement-of-variabilityfunctions will differ from negative ones as well

Resistance to change In some of the studies describedabove variability appeared to be more resistant tochange than repetitions were for example alcohol hadlittle effect on variable operant responses but interferedwith repeated ones Doughty and Lattal (2001) tested thegenerality of these f indings using four-response se-quences with pigeons a threshold variability contin-gency in the Var component of a multiple schedule andLRLR in the Rep component Doughty and Lattal chal-lenged stability of responding by prefeeding the birds inone experiment and providing reinforcers independentlyof responding in the other (see Nevin 1992) Operantvariability was again more resistant to change than wasoperant repetition Thus reinforcement of variabilitymight increase the stability of responding (that is its re-sistance to change) a seemingly paradoxical findingHowever Mechner (1992) suggested that this is the casefor complex skills such as piano performance Varyingpractice routines will make the performance relativelyinsensitive to unforeseen changes in the environmentCare must be taken however when variation is comparedwith repetition because they have different measuresAlso the results reported by Doughty and Lattal werebased on the performance of three pigeons and were notconsistent throughout the experiment The hypothesiswarrants further study

Attraction In some bird species complex male songsattract females (for a review see Catchpole amp Slater1995) There is evidence that male song complexity isunder the discriminative stimulus control of a receptive fe-male (Searcy amp Yasukawa 1990) That explicit reinforce-ment may play a role in such song complexity is suggestedby the Manabe et al (1997) study described above An in-teresting question for further research is the extent towhich operant contingencies contribute to the attractioncreated by behavioral variability in these and other species

Discrimination of variability Animals and people canuse the variability of a stimulus array or entropy as a dis-criminative stimulus (Young amp Wasserman 2001) Whatis the relationship between perception of variability onthe one hand and emission of variable responses on theother Do individuals attempt to match their own levelsof variation to those of others in their environments whichwould involve both perception and emission How mightdiscrimination of onersquos own level of variability influenceonersquos ability to modify such levels and do influences thatobscure discriminations therefore affect ability to change(see eg van der Meer van der Weel amp Lee 1995)Hunziker et al (2002) report that although variabilitywas successfully reinforced their human subjects couldnot correctly describe the contingencies of reinforce-ment a finding similar to those of informal observationsin other studies (eg Neuringer Deiss amp Imig 2000)but whether subjects could discriminate variability levelsrather than contingencies was not studied

Attention and awareness When attention is subject tointerference or is withdrawn from the task at hand abilityto vary is degraded (eg Baddeley Emslie Kolodny amp

OPERANT VARIABILITY 701

Duncan 1998 Evans amp Graham 1980 Neuringer amp Voss2002) It is unclear at present how attention influencesthe different processes underlying operant variabilityand in particular memorial and stochastic processesRich areas for additional study are the relationships be-tween attention awareness and consciousness on theone hand and operant variability on the other Voluntaryaction and consciousness are often related (see egLibet 1999)

Concepts Concepts are thought by many to be sets ofinstances (much like operant sets) whose structures aregraded meaning that there are more or less typical in-stances of such concepts as birds (robin is typical ostrichis not) Barsalou (1987) showed that the gradation ofthese cognitive structures is manipulable or as he putsit not stable A number of variables influence structureincluding context individual differences and points ofview Barsalou argued that these structures are createdas the need arisesmdashthat is on line by the thinking indi-vidual The structure of concepts may parallel that of op-erants and in both cases levels of within-structures di-versity or variability may be modified by consequencesOperant variability may be found in the cognitive as wellas in the behavioral domain

Play and humor Highly variable behaviors are ob-served when animals and people especially children en-gage in play and humor (see eg Fagen 1981) The re-sult of play may be acquisition of new responses (seeSutton-Smith 1975) Although not currently needed re-sponses learned during play may provide the source ofvariations for successful shaping of behaviors at anothertime This is quite analogous to the role of conserved ge-netic variants which are passed along to progeny but donot currently serve any identifiable role Play may alsoinvolve learning how to vary operantly

Reinforcement by variability The self-maintainingnature of play suggests that engaging in variable behav-iors may itself be reinforcing Indirect support comesfrom findings that varied stimulation can serve as a re-inforcer and that stimulus repetition becomes aversive(Fiske amp Maddi 1961) Possibly also related are findingsthat varying reinforcers and permitting choice among re-inforcers increase their value (Bowman Piazza FisherHagopian amp Kogan 1997) and that permitting choicesamong varied responses is also preferred to demandingfixed responses (Catania 1980)

Foraging Two issues regarding variability are presentin the foraging literature One has to do with risk prefer-ence versus avoidance (see eg Shettleworth 1998) Asecond has to do with search strategies Most reportedcases of foraging involve systematic search patterns(Stephens amp Krebs 1986) However search patterns aresometimes stochastic or highly variable (see eg Peter-son 1997) One question is whether some species can varybetween these two strategies and the challenge is to iden-tify the pressures leading to and the advantages of each

Animal training Wilkes (1996) has applied operantvariability techniques in developing training proceduresfor pet dogs More generally reinforcement of variabil-

ity may enable the gaining of control over behaviors ofanimals and as has been suggested above may be an im-portant part of the process of shaping

Models of learning Reinforced variations in variabil-ity may be important when human and animal learningis being modeled whether the models are mathematicalin form computer based or machine based (Blumberget al 2002 Brooks 1999 Sutton amp Barto 1998)

CONCLUDING COMMENT

Genes interact with environments to produce diverseanatomical physiological and behavioral phenotypesGenes and environments also provide for another level ofdiversitymdashnamely operant variability There are manysources of operant variability including random environ-mental events memory for past responses and stochasticgeneration each of which ultimately derives from ge-netic and environmental influences To the extent thatoperant variability is based on a stochastic source how-ever the behavioral results cannot be predicted fromknowledge of these influences That is organisms haveevolved to behave unpredictably The positing of sto-chasticity to explain behavior has a history extending atleast as far back as Epicurus with major contributors in-cluding Gustav Fechner (see Heidelberger 1987) EgonBrunswik (see Gigerenzer 1987) Skinner (1959) andSuppes (1984) Knowledge of the stochastic nature ofthe operant and of how it is controlled leads to a worldview that differs from genetic and environmental deter-minism with roots in the works of the scholars just men-tioned At the scientific level this view leads to the ex-perimental analysis of consequence-influenced variabilityAt the philosophical level it leads to explanations of vol-untary action At the personal level it can lead to change

REFERENCES

Amabile T M (1983) The social psychology of creativity New YorkSpringer-Verlag

Antonitis J J (1951) Response variability in the white rat duringconditioning extinction and reconditioning Journal of Experimen-tal Psychology 42 273-281

Archer J amp Birke L (1983) Exploration in animals and humansCambridge Van Nostrand Reinhold

Arnesen E M (2000) Reinforcement of object manipulation in-creases discovery Unpublished undergraduate thesis Reed College

Baddeley A D (1966) The capacity for generating information byrandomization Quarterly Journal of Experimental Psychology 18119-129

Baddeley A [D] Emslie H Kolodny J amp Duncan J (1998)Random generation and the executive control of working memoryQuarterly Journal of Experimental Psychology 51A 819-852

Baer D M amp Sherman J A (1964) Reinforcement control of gen-eralized imitation in young children Journal of Experimental ChildPsychology 1 37-49

Baker M J amp Koegel R L (1999) Transforming children withautismrsquos ritualistic behaviors into positive play interactions with sib-lings Paper presented at the meeting of the California Association ofBehavior Analysis San Francisco

Balsam P D Deich J D Ohyama T amp Stokes P D (1998) Ori-gins of new behavior In W OrsquoDonohue (Ed) Learning and behav-ior therapy (pp 403-420) Boston Allyn amp Bacon

Balsam P D Paterniti A Zechowy K amp Stokes P D (2002)

OPERANT VARIABILITY 681

choices (Myerson amp Hale 1988) responding underfixed-interval contingencies (Machado amp Cevik 1998)visual discriminations (Nevin 1967) and matching tosample (Cumming Berryman Cohen amp Lanson 1967)How can stability during extinction be reconciled withincreased variability

Neuringer Kornell and Olufs (2001) wrestled withthis question in a study of extinction following rein-forcement under three different contingencies An oper-ant chamber contained two levers left (L) and right (R)on either side of a reinforcement dispenser and a key (K)in the rear of the chamber distant from the dispenserOne group of rats Rep was reinforced for repeating afixed sequence of responses LKR A second group Varwas reinforced for variations across the three operandaunder a threshold contingency Reinforcement for a thirdgroup Yoke was matched to those of the Var animalsand was therefore independent of the particular sequencesBecause of the unusual configuration of operandaskewed distributions of sequence probabilities were ob-served in all of the groups an outcome that was antici-pated and desired in order to document possible in-creases in variability during extinctionmdasha flattening ofthe distributionmdashas well as possible decreases That isduring the reinforcement phase the Rep group came torepeat the LKR sequence with high probability the Yokegroup tended to emit the easiest sequences such as LLLand RRR but distributed their responses over more se-quences than the Rep group did and the Var group dis-tributed most widely although because of differences inthe operanda and in their locations some sequences weremore preferred than others and in particular sequencesending with a K response were least likely (the key beingfarthest from the reinforcement dispenser) Thus thecontingencies resulted in three different distributions orhierarchies of sequence probabilities very steep forRep intermediate for Yoke and flattest for Var

The main finding was that withholding reinforcementhad similar effects in the three groups In each responserates fell to low levels across the four sessions of extinc-tion Furthermore sequences that were most frequentduring the reinforcement phase continued to be most fre-quent throughout extinction A slight flattening of thedistributions was observed however with the lowestprobability sequences increasing relative to the rein-forcement phase Thus because the overall hierarchy ofsequence frequencies was maintained the results indi-cated stability but because the distributions were some-what flattened the results also showed slightly increasedvariability It is as if the strategy during extinction wereto keep doing what had worked in the past but occa-sionally to do something very different This compli-cated story integrates two effects of extinction increasedvariability and maintained stability Most importantly forthe present discussion extinction had similar effects onvariable yoked and repetitive operants

One additional comparison is of interest Recall thatGrunow and Neuringer (2002) found that when contin-

gencies demanded high variability obtained levels ofvariability decreased slightly as reinforcement frequen-cies were lowered This result is opposite to the Vargrouprsquos slight increase in variability just described underextinction conditions The reason may be that extinction-induced variability is short lived lasting only as long asresponding continues perhaps for hours or a few ses-sions at most whereas the variability engendered by theVI 5 intermittent schedule of reinforcement in Grunowand Neuringer can be maintained indefinitely The de-crease in Grunow and Neuringerrsquos operant variabilitymay be due to the weakening of the reinforcement effect(ie reinforcement is less frequent) whereas the in-crease following extinction may be a short-term elicitedeffect This distinction between reinforced and elicitedvariability is important and will be discussed further inthe next section

Operant and Respondent InteractionsElicited or what is sometimes referred to as respondent

effects occur when environmental changes that affectbehavior do not depend on (are not caused by) the behav-ior Pavlovian conditioning being the most commonlystudied example Operant effects are observed when a re-inforcer depends on the response to produce it Althoughfor historical reasons respondent and operant effects areoften studied separately most behaviors result from acombination or interaction of the two Variability is nodifferent and we turn to an example of such an interaction

As reinforcement is approached under FI or fixed-ratio (FR) schedules response rates are commonly ob-served to increase Similarly when reinforcement de-pends on accuracy as in discrimination learning ormatching to sample it too is found to increase within theFI or the FR (Nelson 1978 Nevin 1967) Cherot Jonesand Neuringer (1996) asked whether the same patternheld for reinforced variability Does operant variabilityincrease as reinforcement for varying is approachedThe question may be related to such real-world cases aswhen a composer is rushing to complete a compositionbefore a scheduled performance or when a programmeris trying to solve a problem before an upcoming meetingThese activities involve at least to some extent non-repetitive original or variable thought and behaviorThe question is whether variability increases with prox-imity to or anticipation of reinforcement for varying

Cherot et al (1996) employed trials consisting of fourresponses across L and R operanda and a lag 4 variabil-ity contingency with rats in one experiment and pigeonsin another Unlike in any of the cases described so farhowever the lag-variability contingency had to be satis-fied four times in order for a single reinforcer to be ob-tained With ldquordquo indicating successful completion of theVar contingency ldquo2rdquo indicating failure to meet that con-tingency and PELLET indicating the food reinforcer anexample follows LRLL RRRL LRLL2 LRLL2RLLR RRRL2 LRRR+PELLET Thus food reinforce-ment was provided only after the subject had satisfied

682 NEURINGER

the variability contingency four times FR 4 The resultswere that as the food reinforcer was approached (ie asprogress was made through the ratio) the subjects wereless and less successful at varying The probability ofmeeting the variability contingency in the first trial afterreinforcement was higher than it was in the second whichin turn was higher than that in the third trial and so onFor comparison a Rep group was reinforced for repeatingsequences once again with reinforcement only after foursuccessful repetitions Probability of repeating increasedwith proximity to reinforcement a finding consistentwith the literature but opposite that for the variabilitygroup Thus approach to reinforcement facilitated oper-ant repetition but interfered with operant variabilitySimilar variability-interfering effects have been docu-mented for within-trials responses The probability that aresponse repeats the just prior response increases within atrial even when sequence variability is reinforced (McElroyamp Neuringer 1990 Neuringer 1991) To return to the ex-ample of a composer rushing to complete her composi-tion she may indeed have increasing difficulty respond-ing creatively under the pressure of a deadline On theother hand the soloist may find it easier to repeat thecorrect performance during practice periods as the con-cert nears How can these results be reconciled with themany findings that variability can be reinforced

An additional result of Cherot et al (1996) provides ananswer The Var animals responded much more variablyoverall than did the Rep animals Although variability inthe Var groups decreased with approach to the reinforceroverall variability was always much higher when it was re-inforced (Var group) than when it was not (Rep group)Reinforcement therefore exerts two influences Overallvariability is elevated sometimes to the highest levelswhen it is reinforced but approach to (and possibly focuson) the reinforcers constrains or lowers variability Sim-ilar reinforcement-produced interference effects had beenreported previously for example when raccoons are re-inforced for dropping a ldquocoinrdquo into a slot the reinforce-ment results in the animalsrsquo holding and rubbing the coinrather than dropping it (Breland amp Breland 1961) Theoverall enhancement of variability was of much greatermagnitude in Cherot et al than the decrease with approachwas although both effects were statistically significant(The notes created by the composer under pressure ofdeadline will be less readily predicted than will thoseplayed by the performer practicing a well-known piece)This study highlights the importance of interactions be-tween operant effects of reinforcement and elicited effectsproduced by anticipation of reinforcement and the resultsmay be relevant to the debate concerning possible detri-mental effects of reinforcement on creative behaviors (seethe discussion below and Neuringer in press)

SummaryReinforcement controls behavioral variability as it

does other operant dimensions Levels of variability areprecisely governed by schedule contingencies and dis-

criminative stimuli Choices to vary or repeat match rel-ative frequencies of reinforcement Extinction increasesvariability slightly whereas the hierarchies of sequenceprobabilities are retained and operant and respondenteffects interact The evidence shows that behavioral vari-ability is an operant controlled by its consequences

GENERALITY AND PARAMETERS

Many different methods measures species and re-sponses have been used in the study of operant variabil-ity and this diversity supports the generality of the phe-nomenon As is the case for repeated operants howeverthere are parametric constraints This section briefly re-views the generality and some of the parameters studiedto date

GeneralityMethods Throughout this section ldquoresponserdquo will

signify a response unit including both an individual re-sponse and a multiresponse sequence Each of the meth-ods to be described increases or maintains variability byreinforcing it

Novel response procedures reinforce responses upontheir first occurrence ldquoeverrdquo (Pryor et al 1969) or upontheir first occurrence within a session (Goetz amp Baer1973) Radial-arm maze procedures also reinforce ini-tial (within a given session) entries into arms of a mazeand result in a distribution of choices across the availableoptions the sequence or pattern of choices often beingquite variable

Lag procedures reinforce a response if it has not oc-curred across a given number of previous trials (Page ampNeuringer 1985) For example under lag 5 the currentsequence is reinforced only if it has not occurred duringany of the previous five trials A variant is the recurrence-time procedure which specifies a minimum number of in-tervening trials before a previously emitted response canagain be reinforced Machado (1989) combined recurrencetime with a percentile-reinforcement contingency to gen-erate high levels of variability (see also Machado 1992)Percentile-reinforcement contingencies base the criterionfor reinforcement on the subjectrsquos own performance over aprevious set of responses (see Galbicka 1994) For exam-ple a sequence will be reinforced only if it has not oc-curred over the previous n trials the value of n continuallychanging so that 50 of the trials are reinforced

Least-frequent response procedures (Blough 1966Schoenfeld Harris amp Farmer 1966 Shimp 1967) rein-force responses that are currently least frequent across amoving window

Threshold procedures reinforce responses whose rela-tive frequencies fall below a given value Because thistype of procedure was used in the research describedabove and in many other more recent studies details willbe provided In Denney and Neuringer (1998) which willserve as an example trials consisted of four responses byrats across L and R levers A running tally was kept of the

OPERANT VARIABILITY 683

number of occurrences of each of the 16 possible se-quences LLLL LLLR LLRL and so on If the relativefrequency of the current sequencemdashthe number of its oc-currences divided by the sum of all 16 sequencesmdashwasbelow a specified threshold value in this case 05 the ratwas rewarded Recently emitted sequences contributedmore to the maintained tally than did those in the distantpast because after each reinforcement all 16 counterswere multiplied by a weighting coefficient in this case95 Therefore the influence of a particular trial decreasedexponentially with successive reinforcements Thresholdprocedures differ from Bloughrsquos (1966) least-frequentprocedure in that more than one sequence (or response)can be reinforced at any given timemdashnamely all those se-quences that have relative frequencies below the thresholdvalue The procedure differs from lag in permitting agiven sequence to be reinforced many times in successionif the associated relative frequency is sufficiently low

The frequency dependence procedure (Machado 19921993) is a variant of threshold in which the probability ofreinforcement is a continuous function of relative re-sponse frequency The more frequent a response is thelower the probability is that it will be reinforced

Statistical feedback procedures provide subjects withcomparisons between their performances and that of amodel Neuringer (1986) showed human subjects feedbackfrom 10 statistical tests thereby enabling them to comparetheir own distributions of responses to that of a randommodel Neuringer and Voss (1993) provided feedbackbased on a chaotic model The goal in each of these caseswas for the subjects to learn to approximate the model andthe results of these experiments will be discussed below

Differential reinforcement of switches versus repetitionsbalance a subjectrsquos tendency to engage in each of these pat-terns Rather than reinforcement for infrequent responsesor random-like distributions reinforcement is providedsometimes for repeating a response and other times forswitching from one response to another In the two-operandum case a switch is L followed by R or R by La repetition is L then L or R then R Bryant and Church(1974) found that stochastic-like responding was gener-ated by rats when switches were reinforced with a 75probability and repetitions with a 25 probability that isthe rats had a natural tendency to repeat at least in theenvironment of this study Machado (1997) reinforcedpigeons for eight pecks across two keys if the sequenceof pecks contained at least one (one group of birds) ortwo (another group) switches Although the birds couldhave satisfied these contingencies by repeating a singlesequencemdashin the ldquoone switchrdquo condition for examplerepetition of LRRRRRR would produce reinforcement on100 of the trialsmdashthey tended to vary their sequencesBarba and Hunziker (2002) showed however that vari-ability was higher when it was directly reinforced thanwhen reinforcement was based on proportions of switchesa finding consistent with that of Machado (1997)

The evidence from each of these methods supports thehypothesis that variability can be reinforced

Measures Variability must be interpreted with refer-ence to a particular measure The reason is that there canbe high variability (and low predictability) at one level ofanalysis but order (and certainty of prediction) at a dif-ferent level For example the sequence 1 1 2 2 3 3 44 5 5 1 1 2 2 has the same variance (a commonlyemployed measure of variability) as 2 3 2 1 5 1 4 41 1 3 2 5 2 but an analysis of conditional proba-bilities in which the unit contains pairs of instances in-dicates that the two sequences have different levels ofvariability In general high variability observed with onemeasure does not rule out the possibility of order ob-served with another and vice versa That is why evalua-tions of random-number generators often employ manydifferent measures and indeed there are hundreds of dif-ferent measures of the randomness of a sequence (Knuth1969) All behavioral studies make simplifying assump-tions and rely on one measure or at the most a few

U-value has been the most commonly employed mea-sure of operant variability (Evans amp Graham 1980Machado 1989 Page amp Neuringer 1985 Stokes 1995)U-value measures the distribution of relative frequen-cies or probabilities of a set of responses For a set of16 possible responses i = 1ndash16 U-value is given by

where RFi = the relative frequency (or probability) ofeach of the 16 responses U-values approach 10 whenrelative frequencies approach equality as would be ex-pected over the long run from a random generator andthey approach 00 when a single instance is repeated

Other measures include the percentage of trials or fre-quency at which variability contingencies are met (Page ampNeuringer 1985) percentages of alternations versus stays(Machado 1992) conditional probabilities of responses(Machado 1992) the number or frequency of novel re-sponses (Goetz amp Baer 1973 Schwartz 1982) the num-ber or frequency of different responses (Machado 1997Schwartz 1982) Markov analyses (Machado 1992) andstatistical tests of distributions of responses (Neuringer1986) All of the measures employed to date indicate thatvariability can be reinforced

Responses and species Generality has further beendemonstrated by the use of many different responses andspecies These include Siamese fighting fish (Betta splen-dens) swimming in quadrants of a tank (Roots 2000)young chicks pecking keys (Neuringer 2002) pigeonspecking keys (Page amp Neuringer 1985) budgerigarssinging songs (Manabe Staddon amp Cleaveland 1997)rats pressing levers (Van Hest van Haaren amp van de Poll1989) porpoises swimming and jumping (Pryor et al1969) rhesus monkeys pressing levers (Sydney Reisbickpersonal communication) children drawing and build-ing with blocks (Holman Goetz amp Baer 1977) andhuman adults and children pressing buttons (Saldana ampNeuringer 1998 Stokes amp Balsam 2001) There havebeen few attempts to compare species or responses but

- ( )[ ] ( )=aring RF RFi ii

log log 2 21

1616

684 NEURINGER

in one study lowest baseline variability was observedwhen rats pulled trapezes intermediate levels were ob-served when they pushed keys and highest levels wereobserved when they pressed levers (Morgan amp Neuringer1990) with variability increasing in all cases when it wasreinforced In another study operant variability wascompared across rats and people with similar effects ob-served in the two species (Neuringer Deiss amp Imig 2000)

ParametersWe turn now to some parameters studied to date Many

other parameters as yet unstudied are no doubt also im-portantmdashfor example motivation and types of reinforcers

Age Older people (see eg Van der Linden Beertenamp Pesenti 1998) and older rats (see eg Lamberty ampGower 1990) respond less variably than their youngercounterparts when variable responding is reinforced or re-quested but in most cases it is not clear to what extent thevariability is elicited as opposed to reinforced Neuringerand Huntley (1992) compared reinforced variability in2-month-old (young) and 10-month-old (mature) Long-Evans rats Under Yoke contingencies the young rats re-sponded significantly more variably (two levers four re-sponses per trial) than the mature rats which constitutesan elicited effect When reinforcement was contingenton variability (lag 4) variability increased significantlyin both groups thereby overcoming the initial differencecorrelated with age Thus explicitly reinforcing vari-ability in older animals caused them to respond no dif-ferently than younger animals did a finding potentiallyrelevant to real-world human aging However when Wells(1999) compared 8-month-old (mature) to 24-month-old(old) rats although the mature rats responded more vari-ably than the old rats did under Yoke contingencies thedifference was not statistically significant Additionalstudies are needed especially of whether reinforcingvariable behaviors in older populations can facilitatelearning and performance

Gender Male and female rats are equally sensitive toreinforcement of variability (Neuringer amp Huntley 1992van Hest et al 1989) However there was a tendency(not statistically significant) for males to respond over-all more variably than females did and because thesestudies did not control for differences in size and weightof subjects or for other aspects of motivation additionalstudies are necessary

Stage of training Stokes and Balsam (2001) reporteddifferences in variability as a function of whether variabil-ity was initially reinforced early or later in training (seealso Gottlieb 1992) Human subjects generated sequencesof 10 L and R responses per trial on a computer keyboardunder lag 0 and lag 25 contingencies Under lag 0 everytrial ended with reinforcement under lag 25 the currentsequence had to differ from each of the previous 25 Whenlag 25 was experienced at the very beginning of traininglevels of variability at the end of training (when lag 0 wasoperative) were lower than they were when lag 25 was ex-perienced after 50 trials of initial lag 0 These results were

interpreted to suggest that there is an ldquooptimal period forsetting variability levelsrdquo (Stokes amp Balsam 2001 p 181)but that general conclusion needs further testing and anumber of questions need to be answered One has to dowith the contribution of reinforcement frequency Underlag 0 every sequence leads to reinforcement but rein-forcement is intermittent under lag 25 Another has to dowith whether learning to vary differs from learning to repeatwith respect to optimal periods (see eg Bouton 1994)

Related to stage of training are effects of prior experi-ence Hunziker Lee Ferreira da Silva and Caramori(2002) showed that for human subjects if Var contingen-cies were experienced just prior to Yoke contingencieslevels of response variation during the Yoke phase werehigher than they would have been if the Var contingencieshad not been previously experienced The time delay be-tween Var and Yoke experiences mattered with a long in-terval (3 months) producing only a small effect Saldanaand Neuringer (1998) also found effects of prior Var train-ing on performances in Yoke (increased variability inYoke) but the opposite did not hold that is training underYoke had relatively little effect on Var Hunziker Caramorida Silva and Barba (1998) reported similar effects withrats Thus there may be an asymmetry Reinforcement forvarying has relatively strong influences but reinforce-ment without regard to variations does not adversely affectlearning to vary These different effects are not surprisinggiven that Yoke contingencies are permissivemdashreinforce-ment is given whether or not responses varymdashwhereascontingencies under Var can be quite demanding How-ever the results motivate a number of additional ques-tions One is whether effects of Var training differ fromthose observed when repetitions are reinforced Another iswhether direct reinforcement of variable behaviors canhelp animals or people to overcome the detrimental effectsof prior noncontingent experiences such as those that pro-duce learned helplessness (Seligman 1975)

Environmental constraints Stokes (1995) arguedalso that variability increases when behavior is con-strained The term ldquoconstraintrdquo is ambiguous because allreinforcement contingencies exert a form of constraintIn one study Stokes (1995) compared different strate-gies of shaping leverpresses Rats in one group were re-inforced initially for pressing the lever with only theright paw (right only) and eventually for any press (withright or left paw or with any other part of the body) Theorder of training was reversed for a second group ldquoanyrdquofirst and then right only Behavior sequences were morevariable in the right-only-first group leading Stokes(1995) to conclude that early constraints lead to highvariability (see also Stokes Mechner amp Balsam 1999)That is an important hypothesis with many implicationsbut controls are needed before the results can be attrib-uted to the contingencies For example ldquoright onlyrdquo andldquoanyrdquo differed in terms of reinforcement distributions aswell as in the precision of the reinforcement operationthe ldquoanyrdquo reinforcers were provided immediately and au-tomatically by the programming equipment whereas in

OPERANT VARIABILITY 685

the right-only condition a human observer had to decidewhether a response had occurred and then present the re-inforcer necessarily leading to longer and more variedlatencies and a less consistent contingency Furthermorethere are many cases in which increasing constraints suchas those due to a decrease in the number of responses pertrial (Page amp Neuringer 1985) or the requirement of ex-actly four responses on each of two keys (Schwartz 1980)results in decreased variability Thus the contributionsof response-definition constraints and environmentallimitations must be explored further

SummaryThe generality of operant variability is shown by the

robustness of the phenomenon across procedures mea-sures responses and species As with other operant phe-nomena parameters influence the effects

FUNCTIONS

All operants including variability are functional Op-erant responses produce reinforcing consequences Be-cause other behavioral competencies depend on it oper-ant variability is functional in a broader sense as wellSome of these functions will now be described

Operant ConditioningThe potential of a reinforcer to change behavior is lim-

ited by the range of available variations (Hull Langmanamp Glenn 2001 Skinner 1981) and thus whatever pro-duces those variations contributes to the conditioning ofnew operants Much work has been done to identifysources of variation (Balsam Deich Ohyama amp Stokes1998 Segal 1972 Staddon amp Simmelhag 1971) butNeuringer (1993) was first to show that the same rein-forcers that strengthen individual responses can also pro-duce the necessary variability and that these two functionsof reinforcement may work simultaneously Implied isthat when an operant response is being conditioned re-inforcement works not only to strengthen an individualresponse but at the same time to engender the variabilityfrom which the response emerges (That is variability isnot solely a baseline phenomenon elicited or noise)Concurrent reinforcement of variations and individualtarget sequences will be the topic of this section

Pigeons in one experiment and rats in another were re-warded for varying among 16 possible response sequences(four responses per trial L and R operanda lag contin-gencies Neuringer 1993) Concurrently one of the 16designated as the target sequence was reinforced when-ever it occurred Thus reinforcements were given bothfor sequence variations and for a single selected targeta reinforced-variation-and-selection (RVS) procedureWhen the target was ldquoeasyrdquo (eg LLLL) it emergedrapidly from the baseline variations Given that the tar-get was always reinforced reinforcers were obtained morefrequently when the subjects did not vary and only re-peated the target When the target was ldquodifficultrdquo (eg

LLLR) it was not learned As in the choice experimentdescribed above (Neuringer 1992) the relative valuesor difficulties of target versus variations governed theprobability that the target would be learned

However even difficult targets could be conditionedunder the RVS procedure as was shown by another ex-periment in the same series (Neuringer 1993) Sequencevariations were again reinforced concurrently with a dif-ficult target RRLR but now the frequency of reinforce-ments for varying was systematically decreased Satis-fying the variability contingency was intermittentlyreinforced according to a VI schedule and over sessionsreinforcement for varying became less and less frequentThis technique resulted in acquisition of even the mostdifficult target sequence whereas a control group rein-forced for the target alone (without the concurrent rein-forcement for variations) never learned

Reinforcers were obtained more frequently in the RVScondition than in the control condition (since the RVSsubjects were rewarded both for varying and for emittingthe target sequence) and the additional reinforcers mighthave generated greater motivation to respond (Killeen1981) Thus motivation not the direct reinforcement ofvariability might have been responsible for the RVS ef-fect To test this possibility Neuringer Deiss and Olson(2000) reinforced three groups of rats for emitting a par-ticularly difficult target sequence LLRLR The experi-mental group was reinforced concurrently for sequencevariability on the average of once per minute (VI 1 min)The target-only control group received no reinforce-ments other than for the LLRLR target A third groupwas yoked to the experimental animalsmdashreinforced onceper minute for any sequence variability not requiredmdashand as for the other two groups whenever the target oc-curred If the additional reinforcement was responsiblefor the facilitated learning under RVS then because theexperimental and yoked groups received equal reinforce-ments they should have learned equally well but thatwas not the result Only the experimental group learnedto emit the LLRLR target The yoked controls respondedat high rates throughout the experiment but they did notvary sufficiently to make contact with the target contin-gency and never learned The target-only control groupreceived insufficient reinforcement and their responsesextinguished Therefore the added reinforcers helped tomaintain motivation as suggested above but reinforce-ment of variations was necessary for learning of a diffi-cult sequence The RVS procedure maintains both highmotivation and high variation and the two together re-sult in the acquisition of difficult-to-learn responses

Grunow and Neuringer (2002 Experiment 2) studiedone possibly important parameter of the RVS proceduremdashnamely the level of variability When a difficult target re-sponse is being trained does it matter whether reinforcedvariability is low intermediate or high As describedabove each of four groups of rats was reinforced for vari-able response sequences the first group for high vari-ability the fourth group for low variability and the other

686 NEURINGER

two for intermediate levels In other ways the procedurewas similar to that of the Neuringer Deiss and Olson(2000) RVS contingencies just described with a singledifficult target sequence always reinforced The resultswere that the higher the level of reinforced baseline vari-ability the more quickly the target was learned Thusvariability levels indeed mattered and if the just-re-ported effect is found to be general it will be most help-ful to reinforce high variability when difficult-to-learnnew responses are being shaped Other parameters re-quiring study are class definitionsmdashwhich responses areincluded in the class of acceptable variationsmdashand num-ber of instances within the class

One additional effect should be noted When varia-tions among a set of responses are reinforced if one vari-ant is selected for extinction (no longer reinforced) thefrequency of that instance decreases Neuringer (1993)showed this for pigeons responding under a lag-variabilityschedule When one of the 16 sequences was no longerreinforced (with no other change to indicate its status) thefrequency of that sequence decreased Thus reinforce-ment of variations provides the baseline to select for (iestrengthen) and select against (ie weaken) individualinstances

These findings have potentially important implicationsFirst as has been indicated the same reinforcers thatstrengthen particular instances can also reinforce varia-tions That a binary eventmdashreinforce or not reinforcemdashsimultaneously affects variations and selected instancesindicates both the power of reinforcement and a concep-tual problem How reinforcers exert such diverse effectsis a question to which we will return

Second shaping procedures may implicitly reinforcebaseline variations An approximation to a goal responseis reinforced for a period of time but eventually rein-forcement is withheld until a response closer to the goalis obtained It is commonly thought that withholding re-inforcement or mini-extinction induces the necessaryvariations for shaping to proceed as has been indicatedabove An alternative suggested by the present studiesis that organisms learn both to vary (within a set) and to re-peat (selected instances) until one instance is always rein-forced That is the process of shaping might involve re-inforcement both of variations (generally done implicitly)and of targeted instances If so a useful contributor tothe shaping process might be explicit reinforcement ofvariations

Third reinforcement of variations could facilitate ac-quisition of other competencies such as cognitive andmotor skills Siegler (1996) has shown in an important se-ries of studies that childrenrsquos acquisition of complex cog-nitive skills is correlated with high variability Anotherexample is that when motor skills are practiced in a vari-able manner high quality of performance results (Ma-noel amp Connolly 1995 1997 Mechner 1992 Schmidtamp Lee 1999) However it is rare that variability in thesedomains is directly reinforced as part of the training oreducational process

Fourth the RVS procedure might be used in therapeu-tic settings to modify maladaptive and self-injurioushabits Therapists sometimes employ noncontingent re-inforcement procedures to weaken undesirable behaviors(Vollmer Iwata Zarcone Smith amp Mazaleski 1993)The reasoning is that many maladaptive behaviors aremaintained by unintended consequences and that non-contingent rewards may compete with those conse-quences thereby lowering the strength of the unwantedhabit For example when a staff personrsquos attention toself-injurious behavior helps to maintain it providing at-tention noncontingently may decrease the strength of theunwanted target behavior However the RVS proceduremight accomplish the same goal more effectively andwithout undesirable side effects (Vollmer RingdahlRoane amp Marcus 1997) Noncontingent rewards some-times result in unwanted ldquosuperstitiousrdquo behaviors andthat would be avoided under RVS Furthermore RVSwould help to provide the set of baseline behaviors nec-essary to establish desirable new responses

Finally RVS procedures may help to gain control overand eventually to weaken other undesirable target behav-iors The target can be incorporated within a set of moredesirable responsesmdasha new operant class established withthe target as one membermdashand variable instances rein-forced Once variable responding is successfully main-tained including the undesirable target response the lattercan be weakened by selectively withholding reinforcement

Summary The conditioning of operant responses de-pends on both baseline variations and selection of suc-cessive approximations to a goal response The same re-inforcers used to select the approximations can reinforcebaseline variations thereby defining the precise set ofappropriate variants and the desired levels of variationSuch concurrent reinforcement of variations and selec-tions may facilitate the acquisition of difficult-to-learnoperants and selective withholding of reinforcementmay weaken undesirable instances

Creativity and Problem SolvingBehaving in a novel unusual or nonrepeating manner is

part of creative activities (Campbell 1960) Novelty andvariation may arise through combinations of previouslylearned behaviors (Epstein 1996) transformations of thesebehaviors (Boden 1994) direct reinforcement of variabil-ity as described above or some combination thereof Ofcourse neither novelty nor variability alone suffices forcreative outputmdashthe throw of the dice or the toss of acoin are not creative nor are haphazard strokes of a paint-brush on a canvasmdashbut without variation creativity islow Whether the variability required for creativity canbe reinforced is relevant to whether creativity itself canbe reinforced The latter question has led to contentiousdebate with some arguing that reinforcement facilitatescreativity (Cameron amp Pierce 1994 Eisenberger ampCameron 1996 Stokes 2001 Winston amp Baker 1985)and others focusing on detrimental effects of reinforce-ment (Amabile 1983 Deci Koestner amp Ryan 1999)

OPERANT VARIABILITY 687

The debate goes well beyond the present discussion butsome of the research described above is relevant Firstreinforcement contingent on variability generates higherlevels of behavioral variability than does reinforcementcontingent simply on responding as has been shown bythe many studies described above in which Var contin-gencies were compared to Yoke contingencies Similarlyprecise reinforcement contingencies matter for creativity(Cameron amp Pierce 1994 Lepper amp Henderlong 2000)If any product rather than only creative ones is rein-forced creativity tends to fall (just as variability decreasesunder yoke conditions) Second approach to or antici-pation of reinforcement interferes with operant variabil-ity (Cherot et al 1996 McElroy amp Neuringer 1990Neuringer 1991) which again might be related to effectson creativity when an individual focuses on the rein-forcer Note however that although variability decreaseswith proximity to the reinforcer overall levels of vari-ability are much higher when variability is reinforced thanwhen it is not The same may be true for creativity aswell anticipation of reward may interfere with creativeoutput to some extent but at the same time high creativ-ity may depend on its reinforcement

Operant variability may be important for problemsolving as well Here too there is disagreement as towhether training is helpful or not Early studies by Maier(1933) indicated that it is and these found support inlater work (eg Lung amp Dominowski 1985) Somestudies showed that those individuals most likely toldquobreak the setrdquo are also most likely to solve a problem(Dover amp Shore 1991) However a number of studiesfound that practice had no influence (Duncan 1961)

Arnesen (2000) explored a related phenomenon witha rat model One group Var was reinforced for variableinteractions with novel objects and another group Yokewas reinforced independently of such interactions EachVar rat received training with 12 different objects oneper session For example in the presence of a soup canthe rat was reinforced the first time it entered the canbut thereafter a different response was required such asrolling it climbing on it and so forth This procedure issimilar to that of Goetz and Baerrsquos (1973) described aboveexcept that objects changed from session to session Ratsin the Yoke control group had identical experiences in anadjoining identical chamber that contained identical ob-jects but now rewards were not contingent on object ma-nipulation After this training the rats were tested in adifferent room with 30 never-before-experienced ob-jects scattered on the floor Hidden in each was a smallpiece of food When compared to their Yoke controls theVar animals were more active during this test exploredmore manipulated objects more and most importantlydiscovered significantly more food pellets Another con-trol group received no preliminary training with objectsand their performance in the test session did not differfrom that of the yoked controls Therefore when ratswere reinforced for variable interactions with objectsthe result was an increased tendency to explore manip-ulate and discover new sources of reinforcement It

would be of considerable interest to determine whethera similar effect occurs with children

Assessment and Treatment of PsychopathologiesAbnormal levels of behavioral variability characterize

some human psychopathologies as has been shown byclinical observations as well as by research with humansubjects and animal models Few studies have attemptedto reinforce variability directly in such problem popula-tions The main question is whether reinforcement canmodify abnormal levels of variability in the direction ofnormalcy and whether such changes might have associ-ated beneficial effects

Autism Stereotyped responding characterizes indi-viduals with autism (Hertzig amp Shapiro 1990) For ex-ample Baron-Cohen (1992) asked children with autismand others in a control group to hide a penny in one handso that the experimenter could not guess which hand heldthe penny Those with autism were more likely thanthose without to generate a simple predictable patternsuch as repeatedly switching back and forth from left toright hands Such repetitive responding can be maladap-tive For example Mullins and Rincover (1985) askedchildren with and without autism to pick one of five cardsFood was occasionally found in a cup behind the cardEach of the five cards was a discriminative stimulus fora different schedule of reinforcement including contin-uous reinforcement (CRF every choice reinforced) FR 2(fixed ratio 2 in which every second choice was rein-forced) FR 4 (every fourth choice reinforced) FR 7 andFR 11 Control participants sampled all five alternativesand quickly learned to choose the most frequently rein-forced CRF card Autistic children sampled only a lim-ited number of cards and therefore often preferred anonoptimal alternative Thus children with autism tendto respond less variably than normal children and thiscan result in failure to adapt

Miller and Neuringer (2000) asked whether direct re-inforcement could influence the variability of individu-als with autism Sequences of responses on two largebuttons were first reinforced probabilistically and inde-pendently of variability (Prob phase) after which rein-forcement was contingent upon satisfying a thresholdvariability contingency (Var phase) The autistic partic-ipantsrsquo variability was significantly lower than that ofmatched controls during both Prob and Var phases butwhen variability was reinforced its levels increased sig-nificantly in both groups This is an important findingbecause it suggests that direct reinforcement of variabil-ity may help to modify some of the behavioral stereo-types that are characteristic of autism Lee McComasand Jawor (in press) offer support for this hypothesisThree individuals with autism were reinforced for variedverbal responses to social questions the responses had tobe appropriate given the context Variability increased intwo of the three individuals In a related procedureBaker and Koegel (1999) reinforced children with autismfor varied social interaction in a game-playing contextAlthough the reinforcement was permission to engage in

688 NEURINGER

the stereotyped behaviors preferred by the child the re-sult was increased social interactions both during theperiod of the experimental manipulation and in 1- and2-month follow-ups These three studies taken togethersuggest that reinforcement of varied behaviors may facil-itate modification of nonfunctional ritualistic and stereo-typed behaviors

Depression Depression provides another example ofmaladaptive consequences of low variability Lapp Mar-inier and Pihl (1982) observed that depressed womenproduced fewer alternative solutions to hypotheticalproblems than did nondepressed women Channon andBaker (1996) found that attempts by moderately de-pressed college students to identify faults in a series ofinterconnected circuits were less varied and less success-ful that were those of nondepressed controls HorneEvans and Orne (1982) found that when depressed pa-tients were asked to generate a random sequence of num-bers their responses were less variable than were thoseof controls These experimental results are consistentwith clinical observations

Can depressed individuals be rewarded for increasingvariation Hopkinson and Neuringer (in press) dividedcollege students into mildly depressed and nondepressedgroups on the basis of the CES-D scale a paper-and-pencil self-evaluative index of depression In the first oftwo phases sequences of responses on a computer key-board were reinforced independently of variability (Prob)following which only highly variable sequences were rein-forced (Var) Under Prob depressed students respondedsignificantly less variably than did the controls in agree-ment with previous findings Under Var variability in-creased significantly in both groups Indeed by the endof the experiment depressed and nondepressed subjectswere responding equally variably (Instructions were alsoeffective in increasing variability in the depressed par-ticipants) Thus even in a simple computer-game envi-ronment mildly depressed subjects manifested lowervariability than controls did but direct reinforcementmodified their behavior to control levels Can reinforce-ment of variability be used in real-world conditions tohelp depressed individuals An affirmative answer comesfrom cognitive behavior modification procedures thatwork to increase alternative solutions to problems (Beck1976) A more general conclusion from these studies ofautistic and mildly depressed individuals is that direct-reinforcement-of-variability procedures may benefitthose who manifest abnormally low levels of variability

Attention Deficit Hyperactivity Disorder (ADHD)ADHD may provide an example of the opposite stateabnormally high levels of variability (Barkley 1990)Wultz Sagvolden Moser and Moser (1990) proposedthat the spontaneously hypertensive rat (SHR) may serveas a good animal model of ADHD the SHR shows manyof the behavioral characteristics of that disorder includ-ing hyperactivity poor learning ability poor ability totolerate delayed reinforcement and poor self-control

The next study to be described asked about operant vari-ability in SHRs

Mook Jeffrey and Neuringer (1993) compared vari-ability of SHRs and of a control strain Wystar Kyoto(WKY) under two conditions Prob and Var The mainresult was that SHRs tended to respond variably whetheror not the contingencies required it consistently with theresults from the human ADHD literature When vari-ability was reinforced the two strains responded at ap-proximately equally high levels of variation These re-sults were confirmed in two other studies (HunzikerSaldana amp Neuringer 1996 Mook amp Neuringer 1994)Thus control rats varied only when the contingencies re-quired variations whereas SHRs varied whether or notthe contingencies required it As might be expected fromthese results when reinforcement depended on repeti-tions the SHRs performed poorly (Mook et al 1993)

Another similarity between SHRs and ADHD humanswas seen when amphetamine a drug that has effects simi-lar to those of Ritalin often prescribed for ADHD was ad-ministered (Mook amp Neuringer 1994) When repetitionswere required for reinforcement amphetamine improvedperformances by the SHRs so that they were as accurate assaline-injected WKYs a finding again consistent withthose in the ADHD literature These studies support SHRsas a model of ADHD and suggest that additional attemptsshould be made to assess whether or not individuals withADHD are less able to change levels of variability thanare controls Although as indicated above there is someevidence in support of this hypothesis (Barkley 1990)when Saldana and Neuringer (1998) directly comparedhuman ADHD and control participantsrsquo operant variabil-ity within the context of a computer game no differenceswere observed Most of those with ADHD were main-tained on Ritalin and other drugs however a fact unfor-tunately not documented in the original article

Drug EffectsAlthough many drugs are known to influence behavioral

variability (Brugger 1997) rarely have attempts beenmade to analyze explain or identify the sources of theeffects As will be discussed below operant variabilitycan result from the memory-based process of remember-ing previous responses and then behaving differently orcan result from responding stochastically a process thatappears to be quite different McElroy and Neuringer(1990) attempted to distinguish the effects of alcohol onmemory from its effects on stochastic processes in astudy in which one group of rats was reinforced for vary-ing four-response sequences under a lag 5 contingency(Var) and another was reinforced for repeating a singlesequence LLRR (Rep) Ethanol injections of 075 and20 gkg affected Var and Rep performances differentlyWhereas the Rep animals were much less able to satisfythe LLRR contingency at both doses the Var rats re-sponded variably whether or not alcohol had been ad-ministered Similar results were obtained by Cohen et al

OPERANT VARIABILITY 689

(1990) with a multiple schedule in which each rat servedas its own control In the same animals and within thesame sessions alcohol degraded Rep performances butleft Var intact Thus alcohol appears to interfere with atask that requires working memory as presumably is thecase for the Rep task but not with one that reinforcesvariability The memory-interfering effect of alcoholmight account for why alcohol also degrades radial-armmaze performance (Devenport 1983)

Alcohol also affects choices to vary or to repeat For ex-ample Gorka (1993) reinforced rats concurrently for re-peating and varying and monitored percentage of choicesas a function of time following injection of ethanol Inthe control sessions when the rats were injected withsaline the probabilities of varying and repeating remainedapproximately constant across the 45-min session witha slight bias in favor of varying After injection with 15gkg of ethanol however there was a significant increasein choices to vary at the beginning of the session andthen a later decrease to below 50 That is alcohol ledinitially to a markedly increased preference for varyingbut later to a preference for repeating (Figure 5) This re-search should be replicated and extended especially be-cause the contingencies ldquopressuredrdquo the animals to re-main at approximately 50 preference but thesuggestion is that alcoholrsquos effects on response variabil-ity may depend in part on the duration of its influence Amore general implication is that drugs may affect re-peated and variable operants differently and that effectsdepend on time following administration

SummaryReinforced variability may contribute to many types

of learning including operant shaping and acquisition ofcognitive and complex motor skills Similarly creativityand problem solving depend on variable responding andin these areas as well explicit reinforcement of variationsmay be functional Abnormal levels of variability aresometimes caused by psychopathologies and drugs andoperant variability may be useful in these areas in twoways first as a way of assessing the variability that is pos-sible and second to modify variability toward more nor-mal levels Studies show that individuals with autism andmild depression vary less than do controls but that ex-plicit reinforcement of variability moves the levels towardsnormalcy Other studies show that drugs such as alcoholmay affect repeated and variable operants differently

EXPLANATIONS

Two questions must be answered in order to explainoperant variability First what are the sources of vari-ability that permit its reinforcement Second how is itmaintained Related questions including how instancesbecome members of a set from which variations emergeand whether operant variability is a generalizable com-petency will also be considered

Origination From Nonoperant SourcesSkinner (1966) reasoned that operant responses orig-

inate from nonoperant sources with reinforcement con-tacting a behavior shaping and strengthening it onlyafter the behavior had occurred for reasons other than re-inforcement To put it simply a behavior must occur be-fore it can be reinforced (Balsam amp Silver 1994) I willconsider three initial sources of variable behavior en-dogenous variability noncontingent environmental eventsand extinction These appear to parallel evolutionaryprocesses and they too will be identified

Endogenous variability All physical phenomenavary and behavior is no exception This fact can providethe opportunity for reinforcement to select particularlevels of variation Examples of spontaneous variabilityinclude spontaneous alternations in mazes (Dember ampRichman 1989) nondifferentially reinforced variationsin operant responses (Guthrie amp Horton 1946) and ex-ploration (Archer amp Birke 1983) Endogenous muta-tions provide a parallel in genetics Base-pair changes inDNA molecules occur spontaneously intron-bearinggenes mix and match segments to encode new proteinsjumping genes introduce new DNA sequences and atthe level of the chromosome random crossings and as-sortment occur as a natural part of reproductive meiosisThese sources of random genetic change occur indepen-dently of external influences

Figure 5 The relative frequency of variations by rats as a func-tion of 5-min blocks of time since the beginning of the sessionThe open circles represent performance after the rats were in-jected with saline indicating a slight tendency to vary more thanrepeat but one that is approximately constant throughout thesession The filled circles represent performance after injectionwith 15 gkg ethanol indicating a much higher likelihood ofvarying soon after the alcohol is injected followed by a decreasein variations to below 50 and then a return to baseline at theend of the session Group averages are shown Data are fromGorka (1993)

690 NEURINGER

Noncontingent events Noncontingent events some-times referred to as chance accidents or luck also affectbehavior One happens upon a particular passage in a bookthat leads to a new thought one happens to sit next to aninteresting woman or man on an airplane which leads tomarriage (Bandura 1982) In science as well as in every-day life accidents or serendipity are an important part ofthe process of discovery (Beveridge 1957) In each of thecases presented above an accidental event leads to new be-haviors that are then strengthened by consequences Thesame is potentially true for variability itself Particular lev-els of variability might occur for adventitious reasons butthen become functionally related to consequences

That organisms are particularly sensitive to such unan-ticipated noncontingent events is shown by two basicprinciples of learning habituation and Pavlovian condi-tioning Organisms attend preferentially to unexpectedevents and stop attending or habituate to repeated onesThe conditioning of new Pavlovian responses dependsaccording to influential theories upon unexpected rein-forcing events (eg Rescorla amp Wagner 1972)

Noncontingent events such as X-rays and other pro-cesses that can act as mutagens play an important role ingenetic variation as well The particular genes affectedand the type of changes are random there is no neces-sary relationship or contingency between the externalevent and the affected gene However sensitivity to mu-tations is not random but has been selected presumablybecause it is functional Thus in both behavior and ge-netics sensitivity to noncontingent events is conserved

Extinction Extinction increases behavioral variabilityat least to some extent as was described above Distance inspace or time from the reinforcer does the same (Cherotet al 1996) A possible parallel at the genetic level in-volves high levels of deprivation at least in simple lifeforms For example when E coli bacteria are deprivedof nutrients unusually high mutation rates are observedThe exact mechanism by which such hypermutations areinduced is not known but they increase the likelihoodthat a population can survive such environmental insults(Moxon 1997) That is deprivation-induced variability isfunctional at the genetic level (The behavioral evidenceregarding long-term deprivation is less clear some stud-ies showing increased variability and others increasedstereotypy eg Carlton 1962 Senkowski Vogel amp Po-zulp 1978)

Endogenous variations external noncontingent eventsand withholding of reinforcement are three sources forthe variability that precedes control by reinforcementBut once control by reinforcement is established a dif-ferent question emerges What behavioral processes areinvolved when variability is maintained as an operantThe sources just discussed generate uncontrolled non-specific and often short-lived variability and cannot ex-plain the precisely controlled operant type

Operant Variability ProcessesThree processes are hypothesized to underlie operant

variability The first involves the use of random environ-

mental events such as the toss of a coin The second in-volves memory for responses such as when an animallearns not to return to previously visited locations Thethird will be referred to as an endogenous stochasticgenerator

Random environmental events Throughout historymany cultures have used random events to influencechoices and decisions such as throws of dice and randomselections of sticks cards bones or organs Today coinflips are used by a referee at the beginning of a game to de-cide which team kicks the ball a computerrsquos random-number generator is employed by scientists to avoid bi-ases in assigning subjects to experimental groups andaleatoric events are sometimes employed in modern artmusic and literature The Dice Man by Rhinehart (1998)provides a fictional example in which the protagonistbored with life writes a number of possible actions onslips of paper and then periodically selects one blindlyand behaves accordingly These examples show that randomexternal events may be used to avoid biases engender un-likely responses and break out of behavioral niches

Memory-based variability Many methods used tostudy operant variability differentially reinforce nonrepe-titions at least over the recent past Therefore one strat-egy that subjects can employ is to cycle through few re-sponses or sequences especially when the variabilitycontingencies are relatively permissive For exampleunder a lag 2 contingency cycling repeatedly throughthree sequences will result in 100 reinforcement Ani-mals and people sometimes do this apparently basingtheir current response on memory for or discriminativecontrol by the just emitted responses One such instancewas shown by Schoenfeld et al (1966) who rewardedrats for IRTs that fell into one of 10 interval classes onlyif the current IRT differed from the interval class of theprevious response The rats came to alternate betweenlong and short IRTs thus behaving systematically to meetthis lenient variability contingency Machado (1993)studied an analogous effect having pigeons peck L and Rkeys under a frequency-dependent variability contin-gency related to Shimprsquos (1967) procedure describedabove When the trial unit consisted of two responses thebirds developed memory-based repetitive sequencessuch as RRLLRRLL and thereby satisfied the variabilitycontingency When the trial unit was three responses inlength the birds did not develop the optimal fixed pat-tern of RRRLRLLLmdashwhich was too complexmdashbut in-stead developed ldquorandom-like behaviorrdquo (Machado1993 p 103) Thus a memory-based strategy was em-ployed when it was possible but when the memory loadbecame too high stochastic responding emerged A sim-ilar pattern was seen when variability of budgerigarsongs was reinforced (Manabe et al 1997) Under lag 1contingencies the birds sang mainly two songs underlag 2 they increased to three songs Only when the lagwas increased to 3 did song diversity increase apprecia-bly in one of the birds In radial-arm mazes as well whencontextual cues are available memory again appears toplay an important role leading to much lower probabil-

OPERANT VARIABILITY 691

ities of repetition than would be expected from a sto-chastic generator (McElroy amp Neuringer 1990)

Note that under lag schedules memory-based strate-gies result in higher reinforcement rates than respondingstochastically does The reason is that a stochastic gen-erator repeats by chance That animals and people oftendo not employ memory-based strategies probably indi-cates the difficulty of remembering long sequences andpossibly other advantages of unpredictable responding

If a memory strategy is the only way to satisfy vari-ability contingencies then it can be used even when thememory requirements are demanding as was shown byNeuringer and Voss (1993) Human subjects were trainedto generate on a computer keyboard a chaotic-like se-quence of responses that matched iterations of the logistic-difference equation

where Rn refers to the nth iteration in a series with R be-tween 00 and 10 and t is a constant between 10 and40 When the t parameter approaches 40 iterations ofthis equation result in very ldquonoisyrdquo sequences As withother chaotic processes however there is an orderlinessunderlying the surface noise In this case if responsesare autocorrelated with Rn drawn as a function of Rn21the resulting function is a parabola In the Neuringer andVoss (1993) study following each response college stu-dents were shown how close their predicted values wereto that from iterations of the logistic-difference model Inone experiment the students made digital predictions inanother they placed a pointer along a horizontal lineand this analogue response was then translated into dig-ital values for computation With training the studentsbecame increasingly adept at responding in chaotic-likefashionmdashthe studentsrsquo predictions matched closely theiterations of the logistic functionmdashand their autocorrela-tions became parabolic The left column in Figure 6 showsperformances by 4 subjects (rows) early in training thatis not well approximated by the logistic-differenceparabola and the right column shows performances at theend of training in which r 2 values were 72 99 98 andapproaching 10 How was this accomplished Sinceeach iteration in the logistic-difference sequence is basedon the prior outputmdashthat is the sequence is generated bya deterministic although nonlinear processmdashit was hy-pothesized that the human subjectsrsquo chaotic-like responseswere also based on memory for prior responses To put itsimply the subjects may have memorized a long series ofpairs in the form ldquoIf the previous response was value Athen the current response must be value Brdquo as Metzger(1994) Ward and West (1994) and Ward (2002) have sug-gested

To test whether subjects in fact relied on memory forprior responses Neuringer and Voss (2002) systemati-cally slowed responding by interposing pausesmdashthat isby requiring an IRT As IRT increased the difference be-tween the subjectsrsquo sequences and the modelrsquos chaotic

Rn n nt R R= -( )- -1 11

Figure 6 Autocorrelations (responses on trial n on the y-axis asa function of n21 on the x-axis) for 4 human subjects at the be-ginning (left column) and end (right column) of training inNeuringer and Voss (1993) Equations for the best-fitting curveand amount of variance accounted for (r2) are shown at the topof each panel To the extent that the circles (responses) approxi-mate the equation y = 40x 2 40x 2 the subjects were respondingin a way that matched the logistic-difference chaotic functionFrom ldquoApproximating Chaotic Behaviorrdquo by A Neuringer andC Voss 1993 Psychological Science 4 p 115 Copyright 1993 bythe American Psychological Society Reprinted with permission

692 NEURINGER

output increased and the underlying parabolic structurewas broken That is when pauses increased subjects wereless and less able to approximate the modelrsquos chaotic dis-tribution (see Figure 7) The important point here is thathighly variable responses which in this case are chaoticin nature can be based on memorial processes

Endogenous stochastic generator The evidence tobe reviewed next indicates that animals and people canrespond stochastically and that this competency is basedneither on chance environmental events nor on memoryfor prior events or responses It is impossible to provethat an endogenous stochastic process underlies operantvariability that would be analogous to proving the nullhypothesis However much evidence is consistent withthe stochastic hypothesis Included are the facts thatmemory limitations do not permit adequate respondingunder many variability contingencies that response dis-tributions match those expected from a stochastic gener-ator that changes in parameter values affect operantvariability as they would a stochastic generator and thatinterference with memory processes leaves stochasti-cally based operant variability unaffected As a wholethe evidence is consistent with an endogenous stochas-tic source of operant variability

Memory limitations One type of evidence is nega-tive in character and is based on the presumed inability ofmemory for prior responses or sequences to satisfy sometypes of variability contingencies Two examples weregiven above (Machado 1993 Manabe et al 1997) andanother was provided by a lag 50 contingency in Pageand Neuringer (1985) In the latter case the sequence in

the current trial (eight responses across L and R keys) wasrequired to differ from those in each of the previous 50 tri-als including at the beginning of a session the last 50trials from the previous session Pigeons responded wellunder these contingencies and in a way similar to that ofa stochastic model It is unlikely that pigeons can remem-ber each of their previous 50 eight-response sequences

Response slowing As was shown above for chaotic-like responding performances are often degraded by de-lays imposed between a stimulus and a response pre-sumably because of limitations in working or short-termmemory (Berryman Cumming amp Nevin 1963 Williams1971) Neuringer (1991) relied on this effect to test whethermemorial processes could explain the generation of vari-able sequences under lag schedules Two groups of ratswere reinforced for L and R leverpresses producing fourresponses per trial One group (Var) was trained under alag 5 contingency and the other (Rep) was trained to asingle sequence of responses LLRR It was assumedthat accurate Rep performance depended on workingmemory After performances had stabilized IRTs weresystematically varied by the imposition of timeouts of05 to 20 sec between consecutive responses in the se-quence During these forced pauses the chamber wasdark and responses were not counted toward completionof the sequence If the animals had used the previous re-sponse(s) as a discriminative cue for the current responsethen performances should have been degraded by theseinterpolated pauses

The results were that for the Rep group as the re-quired pauses increased from 0 to 6 sec percentages ofcorrect sequences were relatively unaffected but at pausesgreater than 6 sec probability of a correct LLRR se-quence fell sharply This pattern of results suggests in-volvement of working memory of prior responses withthe rats able to remember responses within a 6-sec win-dow The Var grouprsquos results were quite different Aspauses increased between 0 and 6 sec the rats were moreand more likely to satisfy the variability contingenciesand with pauses greater than 6 sec percent correct re-mained at a high and constant level Thus the Var grouprsquosresults were opposite to those of the Rep group and op-posite to those predicted from a memory-based strategySimilar results showing that operant variability increasesor is maintained as responding is slowed have been re-ported by Morris (1987) with pigeons and by Baddeley(1966) with random response generation by human sub-jects The results reported above concerning alcoholrsquos ef-fects on variability and repetition were also consistentwith those of Neuringer (1991)

Memory for prior responses may not have controlled theVar grouprsquos performance but why did variability increaseEach of three answers is consistent with a stochastic-generator interpretation Weiss (1964 1965) hypothe-sized that random responding requires that current re-sponses be independent of previous ones memory for orcontrol by prior responses interferes with such indepen-

Figure 7 Approximation to a chaotic sequence (given by delta)as a function of the imposed interresponse times Shown are theaverages of 5 subjects

OPERANT VARIABILITY 693

dence and therefore pauses would be expected to facil-itate variability A second possibility is that at shortIRTs animals tend to repeat responses on the sameoperandum Blough (1966) found this in pigeons and ex-cluded such double pecks from his analyses because theyappeared not to be under the control of reinforcement con-tingencies Morris (1987) also found a tendency for birdsto repeat when no interresponse timeouts were imposedA third hypothesis is that there were two contributors tothe observed variability One was a stochastic processcontrolled by reinforcement and the other elicited vari-ability generated by interposition of pauses The high vari-ability elicited as an effect of slowed responding is ageneral phenomenon supported in many other cases Ac-cording to this interpretation operant variability in the Vargroup was governed by a stochastic-based process operantrepetition in the Rep group was governed by a memory-based process and pauses elicited variability under bothcontingencies Thus Rep performance was interferedwith whereas Var was facilitated Each of these hypothe-ses is consistent with the conclusion that memory for (ordiscriminative control by) prior responses does not con-tribute to and possibly interferes with variable respond-ing when an organism is reinforced for variability andthis in turn is consistent with the stochastic-generatorhypothesis

Stochastic response distributions Other evidence iscorrelational Response distributions are similar to thosefrom a stochastic generator The data from two experi-ments already discussed are shown in Figure 1 In thesecases reinforcement of least frequent IRTs (Blough 1966)and of nonrepetitive choice sequences (Machado 1989)resulted in stochastic-like responding

Neuringer (1986) reported another striking case Highschool and college students generated sequences of 1sand 2s on a computer keyboard with each set of 100 re-sponses constituting one trial A baseline phase lastedfor 60 such trials for a total of 6000 responses duringwhich the students were asked to behave as randomly aspossible as if they were reporting the outcome of a se-ries of coin tosses As in the many previous studies ofthis sort the subjectsrsquo responses differed significantlyfrom those of a stochastic model (Brugger 1997 Wage-naar 1972) During the training phase the subjects re-ceived feedback which enabled them to compare their per-formances with that of a stochastic model first accordingto one statistical test and then according to another untilfollowing each trial feedback was provided from 10 dif-ferent statistics The results were that all of the subjectslearned to approximate the stochastic model accordingto the 10 statistics as well as some (but not all) other sta-tistical tests That is the distributions of the statisticswhich clearly did not match a stochastic model at the be-ginning of training approximated the stochastic modelat the end of it (see Figure 8)

The previous failures of other researchers to obtain ev-idence for random responding had led some to concludethat humans did not have the capacity to behave ran-

domly What would account for the difference betweenthe present positive finding and the many other negativeones Neuringerrsquos (1986) study was the first in which sto-chastic responding was explicitly reinforced in this casewith feedback from 10 statistical tests An analogy mighthelp to explain the difference in procedures and resultsImagine an experimenter walking around a city and ask-ing people if they know what a violin sounds like (analo-gous to asking people whether they know what a series ofcoin tosses or dice throws is like) Most would answeryes If the same individuals were then asked to play theviolin very few if any would succeed The experimentermight conclude after many replications with differentsubjects in different cities that people did not have thecapacity to play the violin The point of course is thatplaying the violin is a skill requiring training and prac-tice and the same may be true for responding randomlyIt is easy to vary but practice and reinforcement may benecessary to approximate a random distribution

Response number Page and Neuringer (1985) testedthe stochastic-generator hypothesis in another way byvarying number of responses per trial with pigeons work-ing under a lag 3 contingency The reasoning was that ifprevious responses served as a discriminative cue forwhat not to domdasha memory-based strategymdashthen perfor-mance should be degraded when the number of re-sponses was increased Eight responses per trial requiresubjects to remember more than four responses The sto-chastic hypothesis predicts the opposite outcome as isdemonstrated by the following example If each trialwere two responses in length with responses determinedby the toss of a coin then the probability that a given trialwould repeat the previous one would be 25 (There arefour possible sequences in the first trial RR RL LRand LL Thus the second trial has a 1-in-4 chance ofmatching the first) If a trial consists of four coin tossesthe probability of a repetition by chance is 0625 or 1 in16 Thus if subjects used a stochastic process to gener-ate Ls and Rs performances should improve with in-creasing responses per trial Results were precisely thosepredicted by the stochastic hypothesis and inconsistentwith a memory strategy Probability of meeting the con-tingencies and therefore probability of reinforcementincreased as responses per trial increased In otherwords eight-response trials were easier for pigeons thanfour-response trials

Response interference Neuringer and Voss (2002)attempted to distinguish between memorial and stochas-tic processes in yet another way They hypothesized thatwhereas chaotic responding was based on memory forprior responses (as was discussed above) responding inNeuringerrsquos (1986) study with feedback from 10 statisti-cal tests indicated involvement of an endogenous sto-chastic generator To show that human subjects could en-gage in both processes at will three college studentswere trained to generate chaotic sequences in one compo-nent of a multiple schedule as in Neuringer and Voss(1993) and to generate stochastic sequences in a sepa-

694 NEURINGER

rate component as in Neuringer (1986) The wordCHAOTICS was on the screen in the first component theword STOCHASTICS was on during the other and thesewords alternated throughout the session After the sub-jects succeeded at each of the component tasks test ses-sions were introduced in which memory for prior re-sponses was subject to interference in both the chaoticsand stochastics components as follows During the

chaotics component four separate chaotic sequenceswere cycled one after the other each cued by differentscreen colors The subjects attempted to approximate achaotic sequence in one color (eg light green) follow-ing which they would have to match a second chaotic se-quence (lavender screen color) and so on the fourchaotic contingencies following one another across thesession Although all four chaotic sequences used the

Figure 8 Performance by one subject at the end of training to respond stochastically in Neuringer(1986) Shown are comparisons on 6 different statistics between the subject (solid lines and Xs) andthe random model (dashed lines and closed circles) At the end of training the subjectrsquos perfor-mance did not differ statistically from that of the random model on these and four additional sta-tistics From ldquoCan people behave lsquorandomlyrsquo The role of feedbackrdquo by A Neuringer 1986 Jour-nal of Experimental Psychology General 115 p 71 Copyright 1986 by the American PsychologicalAssociation Reprinted with permission

OPERANT VARIABILITY 695

same logistic-difference equation that a model did theywere out of phase and independent If chaotic respond-ing is memory based then the four independent se-quences would be expected to interfere with one anotherthat is approximations to the four chaotic models wouldbe lower than when only a single chaotic output was re-quired as in Neuringer and Voss (1993) That was ex-actly the finding Performances by the three subjectswere signif icantly degraded during the four-segmentphase To put it simply chaotic responses interfered withone another

A different pattern of results was seen in the stochasticsportion of the experiment The same interference proce-dure was used with four independent stochastic sequencesrequired in the four different color conditions As inNeuringer (1986) subjects were required to satisfy eightdifferent statistical tests of randomness and to do this in-dependently in each of the four phases (If responses in anytwo or more of the four stochastic phases became corre-lated above a minimal level responses were considered tobe incorrect thereby prohibiting repetition of a single se-quence in two or more of the phases) For one subjectseven of eight statistics were closer to those of a randommodel during this interference phase than they were at theend of the original training with only a single sequenceand for a second subject all eight statistics were closerThe third subject showed closer-to-random approximationunder the four-segment condition on only two of the eightstatistics but most of her changes were also in the directionof a random generator For example if a random generatoremits equal numbers of ldquouprdquo and ldquodownrdquo sequences andthis subject in the baseline phase had emitted only 47ldquouprdquo sequences then during the four-segment tests shecame to emit ldquouprdquo 55 of the time She changed in the di-rection of the random model but overshot the mark Thuswhereas interference degraded chaotic performance sig-nificantly in all three subjects it improved stochastic per-formances in at least two of them These results are con-sistent with the hypothesis that a memory-based processcontrols chaotic variability and that a stochastic processnot dependent on memory controls stochastic variability

Summary of Tripartite TheoryThe following three processes constitute a tripartite

theory of operant variability attention to random events inthe environment (coin toss) memorial processes (chaosradial maze) and stochastic processes (response se-quences and random-number generation with feedback)Random environmental events assure unpredictabilitymdashfor example in making unbiased selections Remember-ing what was done previously will enable one to avoidrepetitions to vary the students called upon in class tovary the locations when looking for a hidden object orto try different approaches when attempting to solve aproblem External aids such as notes to oneself or sci-entific records help to enhance memory capacities

An endogenous stochastic generator provides the thirdand most controversial source of behavioral variability

When an opponent is attempting to predict your behav-ior in a game or at war it may be functional to behavestochastically Similarly operant conditioning of new re-sponses may depend on stochastic variations It is im-possible to prove stochasticity some presently unknowndeterministic system may ultimately explain the resultsHowever the evidence is consistent with the stochastichypothesis and inconsistent with the alternatives For ex-ample memory is not sufficiently powerful to explain alloperant variability memory-interference proceduresleave stochastic responding intact and behaviors corre-late well with those of stochastic models

Both genetics and reinforcement play important rolesin all three processes Reinforcement is always involvedin that learning provides information about when whatwhere and how much to vary Genetic influences areshown in that not all responses are possible and if theyare possible they are not all equiprobable (eg as inNeuringer et al 2001) Stochastic responding may be bi-ased due to experiences and knowledge that is biasesmay be established by reinforcement contingencies or bythe interface between the organismrsquos propensities (or be-havior systems see Timberlake amp Lucas 1989) and theenvironment In the case of random-number generationby human subjects biases may be due to misconceptionsconcerning what ldquorandomrdquo means However some bi-ases can be overcome through feedback and instructions(Lopes amp Oden 1987 Neuringer 1986)

The importance of variations for evolutionary changeis supported by the many sources of genetic variabilityand by the fact that genetic variability is conserved Asimilar argument holds for operant variability That thereare at least three independent sources of operant varia-tions as well as other elicitors such as environmentalnoise and response extinction supports its importancefor normal adaptive functioning

Structure of VariabilityEvery operant ldquoresponserdquo is actually an instance of a

class with each member of the class exerting the sameeffect on the environment (Skinner 1959) When a rat isreinforced for pressing a lever for example the class isdefined by closure of a microswitch and any way of ac-complishing the closure whether with the left paw theright paw or the mouth is as effective in producing re-inforcement as any other If the contingencies of rein-forcement so specify however the response class can beredefined and circumscribed which is another charac-teristic of the operant For example if only right-pawpresses are acceptable members of the reinforced classresponding will tend to occur only with the right paw(Stokes 1995) Of course the capabilities of the organismplay a role for example including ldquotail pressesrdquo in the op-erant class does not mean that the lever would be or couldbe pressed by the ratrsquos tail The structure of the operantclass therefore emerges from the current reinforcementcontingencies and the organismrsquos capability derivedfrom genetics development and prior experiences

696 NEURINGER

Operant variability has structure as well and as for alloperants that structure is due partly to the organism it-self and partly to the reinforcement contingencies Theresearch described throughout this article has providedexamples of the structure of variability generally in theform of sets of four or eight responses across L and Roperanda But the structure of variations can be speci-f ied more precisely For example Mook and her col-leagues (Mook et al 1993 Mook amp Neuringer 1994)reinforced rats for varying four response sequences acrossL and R levers under a lag 1 contingency Unlike in pre-vious studies however the acceptable class was furtherconstrained so that only sequences beginning with two Rresponses (RRLL RRLR RRRL and RRRR) were rein-forced The rats learned to vary their sequences withinthe imposed limitations that is they generally begantheir sequences with two R responses and varied the se-quences within those limitations Similar results wereobtained in the context of a radial-arm maze in whichonly a subset of arms was baited

The precision with which reinforcement contingen-cies can specify class was highlighted in research per-formed by Burns (1997) and Ross and Neuringer (2002)In the latter case college students drew rectangles on acomputer screen with the computerrsquos mouse Each rec-tangle constituted one trial (the screen cleared at the endof each trial) with reinforcement (points) depending onsimultaneous variations along three dimensions of therectangle location on the screen area of the rectangleand shape with the latter defined as the ratio of the rec-tanglersquos height to its width Although the subjects weregiven no instruction other than to ldquogain pointsrdquo theylearned to vary along all three dimensions with levels ofvariability significantly higher than those of yoked controlsubjects who were reinforced independently of variations

A second part of the Ross and Neuringer (2002) studydemonstrated even greater precision of control To gainpoints three new groups of subjects were required to re-peat along one of the dimensions while simultaneouslyvarying along the other two One group was required torepeatedly draw rectangles of approximately the samesize while simultaneously varying location and shapeanother group had to repeat the shape while varying areaand location and the third to repeat location while vary-ing size and shape Verbal instructions were minimal asthey were in the first part of the study Subjects learnedto satisfy these contingencies as well varying two at-tributes while simultaneously repeating one and signif-icant differences were obtained rapidly within 300 trials

These demonstrations raise questions as to how binaryfeedbackmdashreinforcement presented or notmdashcan simul-taneously control independent behavioral dimensionsThe issue is similar to that discussed above concerningsimultaneous reinforcement of variations and selectionsIt is similar also to those issues faced by researchers at-tempting to understand how the nervous system strength-ens particular synapses when many different neurons firesimultaneously and to ldquoallocation of creditrdquo in artificial

learning models For present purposes suffice it to notethe exquisite precision with which reinforcement controlsvariability and repetition

GeneralizationDo animals and people learn to ldquovaryrdquo implying a

generalizable skill or do they learn a specific distributionof specific responses across particular operanda Thequestion of generalization has been asked of many otherbehavioral competencies such as imitation When chil-dren learn to imitate do they learn stimulusndashresponseassociations (if mother moves her right hand the childlearns to move his right hand) or do they learn moregenerally to ldquoimitaterdquo Evidence indicates that reinforce-ment of imitative instances leads to generalized imitation(see eg Baer amp Sherman 1964 Bandura 1986) A sim-ilar question has been raised when pigeons rats or chil-dren learn to choose a stimulus that matches a previouslyshown sample For example if a pigeon is shown a reddisk as a sample it must then choose red when given achoice between that and blue Does the bird learn to matchIn this case the answer appears to depend on the train-ing procedures If trained with few stimuli such as redgreen and blue then the pigeon learns simple associa-tions (if red sample choose red if blue sample chooseblue) but if hundreds of different stimuli (pictures) areemployed during training then generalized ldquomatchingrdquois learned (Wright 1997) In each of these examples thelearning of a general competency or strategy was shownby transfer-of-training tests What was learned with oneset of stimuli or in one situation transferred to new stim-uli and situations We turn now to the question of whetherlearning to vary transfers

After the subjects in Maltzmanrsquos (1960) experimentdescribed above learned to generate remote verbal as-sociates they were asked to devise unusual uses of com-mon objects In comparison with those in the controlgroup these subjects generated significantly more un-usual uses indicating transfer of training to a differenttask However transfer was not observed in the oppositedirection from unusual uses to verbal associates

Holman et al (1977) using a procedure similar to thatof the Goetz and Baer (1973) research described aboverewarded preschool children verbally for novel drawingsand found increased novelty Unrewarded block construc-tions were also monitored and weak evidence was ob-tained for transfer of novel responding from rewardeddrawing to the unrewarded constructions In a second ex-periment preschoolers engaged in four independent taskspainting pen drawing wood block construction and Legoconstruction When the children were reinforced for vary-ing their pen drawings the novelty of paintings increasedalong with the pen drawings indicating transfer from onetask to the other However transfer to the topographicallydifferent block- or Lego-building tasks was not observed

Eisenberger and Armeli (1997) gave monetary rewardsto one group of fifth- and sixth-grade students the vari-ability group for describing unusual uses of common ob-

OPERANT VARIABILITY 697

jects A second group the repeat group received thesame rewards for describing common uses of the sameobjects In a later test all of the children were asked todraw pictures incorporating a circle and the group that hadpreviously been reinforced for varied responses drewmore unusual (or lower frequency) pictures than did therepeat group Thus there was transfer of novelty trainingfrom one domain to another with additional studies in theseries showing similar effects (Eisenberger Armeli ampPretz 1998 Eisenberger Haskins amp Gambleton 1999Eisenberger amp Selbst 1994) However Eisenberger et al(1999) failed to replicate the transfer-of-training effectThe variability and repeat groups responded similarly dur-ing the test phase In this study separate experimenterstrained and tested the subjects thereby avoiding possibleexperimenter-expectancy effects whereas the same ex-perimenter did both training and testing in Eisenbergerrsquosother studies Across the series of experiments there weremany other cases in which transfer was not observed andmore research is necessary to explain why

In Lee et alrsquos (in press) study with autistic individualsdescribed above after variable answers were trained toquestions verbal variability transferred to new experi-menters and new contexts

Arnesenrsquos (2000) results described above are consis-tent with transfer of training in an animal model Rats inthe Var group were trained to interact variably with oneset of objects When later tested in a new situation withnew objects the Var animals showed (in comparisonwith controls) higher levels of exploration manipulationof the novel objects and discovery of hidden reinforcers

Much of the evidence reviewed in this section there-fore indicates transfer of training and supports a generalldquovaryrdquo competency However there remain unresolvedquestions the most important of which is the extent towhich transfer occurs Note also a methodological pointMany of the studies tested whether transfer occurs whenthe contingencies are neutral with respect to the to-be-transferred behavior Holman et al (1977) is a case inpoint Variations were reinforced in one domain draw-ing and the question was whether variations wouldoccur in another domainmdashblock constructionmdashin whichreinforcement did not depend on varying A negativef indingmdashin this case absence of transfermdashindicatesnothing about the ability to generalize but only aboutthe tendency to do so given the absence of a reason (orcontingency) A better test would be to compare experi-mental and control performances when variability in thenew task is in fact functional as in Arnesen (2000)

SummaryBefore variability can be reinforced it must occur for

reasons other than reinforcement Some of the reasonsinclude variability inherent in all behavioral phenomenaldquoaccidentalrdquo environmental events that are unrelated toparticular behaviors and withholding of reinforcementTransfer from other situations may serve a similar functionFor example if an organism can discriminate shaping-

of-new-response periods from others then it may learngenerally to ldquovaryrdquo during shaping episodes After rein-forcement makes initial contact with response variationsthere are at least three behavioral strategies that can beemployed in order to maintain functionally high levelsof variability respond based on external random eventssuch as the toss of a coin avoid repetitions by remem-bering what responses had previously been made and re-spond stochastically Evidence for an endogenous sto-chastic process was seen in the similarity of responsedistributions to those generated by stochastic sourcesand by findings that interfering with memory for priorresponses either improves or leaves unaffected stochas-tic behaviors Stochastic responding as well as othertypes of variability-generating strategies depend uponcontingencies to define the required level of variabilityand the set from which variations emerge

IMPLICATIONS

One implication of reinforced variability is that oper-ant responses may differ in the extent to which their vari-ability is affected by reinforcement contingenciesmdashthatis operants may be ordered along a continuum of sensi-tivity of variability to reinforcement Another is that re-inforced variability causes us to question the generalityof some basic goals of behavioral psychology includingprediction and control A third is that reinforced vari-ability may help us to provide a scientifically useful wayto describe voluntary action In particular the variabilityof all voluntary behaviors can readily be changed mo-ment to moment by discriminative stimuli and conse-quences Implications for everyday life also will be noted

Operant ContinuumAs was indicated above the operant is composed of

instances that is the operant is a class of many differentresponses all functionally equivalent but often topo-graphically distinct Skinner suggested and I hypothe-size that instances within the class emerge stochastically(see Moxley 1997 for a discussion of Skinnerrsquos posi-tion) Although reinforcement contingencies and contextinfluence the individual probabilities and their distribu-tions emission is stochastic nonetheless The particularmember of the operant class generally cannot be predictedwith a high degree of certainty For example a particu-lar rat may be more likely to press the lever with its leftpaw than its right but left- and right-paw responses areemitted probabilistically This class-nature of the operantsuggests one way to conceptualize operant variability interms of size of class and within-classes probability dis-tributions High variability implies large classes equalprobability distributions or both low variability impliesthe opposite

Variability may be modified more readily in some op-erants than in others implying that operant variability isnot an all-or-none phenomenon but rather a continuumwith reinforced variations in variability (RVV) differing

698 NEURINGER

for different operants RVV might best be understood byanalogy to the concepts of speed and acceleration Bothvariability and speed range from low to high RVV isanalogous to acceleration the rate of change in speedHighly sensitive RVV is indicated by the possibility ofreinforcing many different levels of variability from rep-etition to random-like responding and by the fact thatchanges in level occur readily and rapidly

Different operants manifest differences in RVV Forexample variability of chain pulls changed more thanthat of leverpresses when variability was reinforced inrats (Morgan amp Neuringer 1990) Some operant-like re-sponses may show relatively low RVV For example fir-ing rates increase in single hippocampal nerve cellswhen dopaminergic substances are contingent upon theresponse (Stein Xue amp Belluzzi 1994) suggesting thatnerve f iring is controlled by reinforcement To datehowever there is no evidence that variability of nervefiring is similarly controlled Other cases might includereinforcement of nondiscriminable (subconscious) re-sponses such as Hefferline and Pererarsquos (1963) reinforce-ment of subaware thumb twitches and biofeedback ofmuscle and cardiovascular activity In these cases rein-forcement may modify performance but it is not knownwhether variability is also sensitive to reinforcement andif so to what degree Human verbal variability may bethe most readily manipulable of all behaviors the rangeof predictable to unpredictable verbal responses beingimmense with extraordinary sensitivity to context andconsequence RVV may also differ across organisms (per-haps from simple to complex) individuals (perhaps asan index of intelligence or learning ability) and physio-logical and psychological states (WKY vs SHR depres-sion and autism)

RVV may help to distinguish operant responses fromother superficially similar cases of selection by conse-quences That is the ldquooperantrdquo may also best be concep-tualized as a continuum with RVV an important deter-miner of position along the operant continuum Oneexample of selection by consequences but low ldquooperant-nessrdquo is genetic change Variability is essential for evolu-tionary change of course and levels of genetic variabil-ity are sometimes affected by environmental conditions(Moxon 1997) but there are few examples of levels ofvariability being influenced by selection pressures Thatis the source of much genetic variability is thought to berandom not influenced by current pressures and needsThus although selected by consequences genetic changemay be low along the operant continuum Other exam-ples of operant-like phenomena for which variability although important may be relatively insensitive to con-sequencesmdashvariability in these cases is elicited not se-lectedmdashinclude the development of the central nervoussystem and immune system responses (Hull et al 2001)In both of these cases change depends upon selection byconsequences but it is not clear whether variability canalso be selected Most commonly studied operant re-sponses on the other hand show high sensitivity of vari-

ability to reinforcing events But to reiterate the mainpoint of this section operants differ in terms of RVV andtherefore in terms of their ldquooperantnessrdquo from low sensi-tivity (possibly responses of single nerve cells) to extra-ordinarily high sensitivity (verbal behavior) The operantnature of variability as well as that of every other oper-ant dimension is best conceptualized as a continuum

Prediction and ControlSmith (1992) described Skinnerian operant condition-

ing theory as Baconian in nature a primary goal of Skin-nerian behaviorists being to control behavior and hecontrasted this with the more Aristotelian goal of de-scription Skinner and his students (eg Zuriff 1985) in-deed emphasized prediction and control Many of thestudies reviewed above however involved reinforce-ment procedures that increase variability and conse-quently decrease predictability It is not only ignorancetherefore that restricts prediction and control as Skin-ner suggested but reinforcement itself Operant meth-ods and theories when applied to variability lead togreater emphasis on description with control over pre-cise response characteristics possible under some butnot all conditions

In summary an operant is a class of responses underthe control of contingencies of reinforcement The classcan be controlled and predicted given knowledge ofconditioning history Indeed the size of the class andpossibly the distribution of within-classes probabilities orlevel of variability can also be controlled and predictedBut to the extent that instances emerge stochastically aparticular instance cannot be predicted The instant re-sponse can be explained as a member of a particularclass but prediction of the instance will be diff icultsometimes to the extreme This point will be developedin the next section

Voluntary BehaviorI hypothesize that voluntary behaviors are operants for

which RVVs are highly sensitive to influence by conse-quences One way to test this hypothesis would be togather empirical data concerning how voluntary versusinvoluntary actions are classified by human observers Asfar as I know there are no psychophysical data concern-ing this question but the hypothesis implies that highsensitivity of variability to operant contingencies RVVswill be present in all actions perceived to be voluntary

This operant-variability theory of voluntary action isbased on the supposition of two attributes First volun-tary acts must be potentially explainable (given suff i-cient knowledge) on the basis of genetics conditioninghistories current contingencies anticipated conse-quences intentions goals and the like That is volun-tary behaviors are lawful The second attribute is that thevoluntary act must be potentially unpredictable That isvoluntary acts are ldquofreerdquo of control by identifiable stim-uli If behavior is perceived necessarily to result frommdashto be predicted on the basis ofmdashexternal events includ-

OPERANT VARIABILITY 699

ing reinforcement contingencies it will not be perceivedas voluntary ldquoVoluntaryrdquo implies potential indepen-dence from stimulusndashresponse or responsendashreinforcerdetermination According to this theory both explain-ability and unpredictability are necessary attributes ofvoluntary acts

How though can behaviors be simultaneously ex-plainable and unpredictable Reinforcement contingen-cies play two essential roles Reinforcement (togetherwith genes prior experiences and current environment)helps to determine the set of possible behaviors fromwhich a voluntary act emergesmdashthat is the operant classReinforcement (together with the other contributorsspecified above) also helps to determine the level of vari-ability within the class Thus both instances comprisinga class and level of variability within that class can bepredictedmdashagain given sufficient knowledge Howeverthe individual response is at least sometimes (or relatively)unpredictable The reason is that the operant class fromwhich the response stochastically emerges is sometimesextremely large (eg ldquoname a wordrdquo or ldquodo somethingrdquo)and the contingencies sometimes call for equiprobablestochastic emergence (eg ldquofool merdquo) In the latter casea voluntary act may appear to be foolish meaninglessor unwarranted but if it is voluntary it can potentially beexplained (For example in response to the questionldquoWhy did you do that silly actrdquo the answer could beldquoJust to show that you canrsquot predict my behaviorrdquo seeScriven 1965) In these cases prediction of instancesfalls to very low levels indeed and I hypothesize evenan all-knowing observer will be unable to predict thenext response at greater-than-chance levels which aregiven by the size of the operant class and by within-classesprobability distribution Even within small classes (egin the game rockndashscissorsndashpaper in which unpredictabil-ity among the three possible responses is functional) be-havioral stochasticity may lead to unpredictability of theinstant response Note however that although voluntarybehaviors are sometimes unpredictable the opposite isoften the case When contingencies call for highly con-strained particular responses predictions can easily bemade For example if a police officer asks your nameyour response can most likely be predicted Voluntary re-sponses are therefore described as potentially unpre-dictable with variability contingencies engendering dif-ferent levels of unpredictability

According to this theory behaviors low in the sensitivity-of-variability dimension are described as relatively in-voluntary and those that are high in said dimension areperceived as voluntary Low RVV may be shown at anylevel along the variability continuum For example al-ways responding in the same unchanging way will beperceived as involuntary but so will always respondingrandomly (The coin does not voluntarily come up withheads or tails) This point is essential More than 2000years ago Epicurus argued that occasional randomswerves of atoms provided the source of voluntary be-havior in an otherwise determined universe but al-

though this conjecture was prescient it was not quitecorrect ldquoFree-willedrdquo or voluntary action whether it isbased on random swerves or on quantum mechanics isnot explained by variability per se but by functionalchanges in levels of variabilitymdashthat is by rapid andprecise changes in response to environmental demandsResponses can be placed along a voluntaryndashinvoluntarycontinuum isomorphic to the operantndashrespondent con-tinuum with voluntary behavior being functional pre-cisely in the same way as operant behavior is ThusSkinnerrsquos (1974) suggestion that voluntary behavior isoperant behavior was almost correct Only those oper-ants for which variability is highly sensitive to conse-quences will be perceived as voluntary

One additional example might help Operant classes(sets of potential responses) are created ldquoon linerdquo as we goabout our daily activities As we walk from car to officewe tend to walk in a characteristic way take the samepath with tightly constrained operant classes followingone after the other in rapid succession Class sizes aresmall and within-classes probability distributions arehighly peaked Predictions of next responses by a knowl-edgeable observer can readily be made However classsizes can be increased and probability distributions flat-tened we can walk in quite strange ways indeed dancealong a new path sing hand candy to passers-by Strangebehavior indeed but made possible by the ability of op-erant sets to be modified moment to moment not only interms of class definition but in terms of class size lev-els of within-classes variations and so on The same istrue for conversation Moment-to-moment changes inverbal class definition size and predictability of within-classes instances are hallmarks of voluntary behavior

GenerativityRelated to the definition of the operant and to our discus-

sion of voluntary action is that fact that operant behaviormdashleverpressing no less than languagemdashis by its naturegenerative This claim follows from the functional na-ture of the class of responses (all members of the classare effective in producing an outcome) and the stochas-tic emission of instances From these flow never-before-emitted instances patterns and sequences (see Epstein1996) Operants especially those high on the RVV con-tinuum are inherently generative

Self-ExperimentationVariations systematic or otherwise are a necessary part

of scientific work Neuringer (1981) and Roberts and Neu-ringer (1998) described how variations can be brought to bear explicitly on everyday life in the form of self-experiments Individuals vary aspects of their lives suchas the foods they eat social contacts and methods ofmemorizing they keep records concerning the effects ofthese manipulations and form hypotheses concerningcontrolling variables Using self-experimental methodsindividuals can learn how what when where and howmuch to vary Explicitly varying onersquos everyday behav-

700 NEURINGER

iors and thoughts may help to engender what Langer(1989 2000) has referred to as mindful livingmdashthat isnonautomatic voluntary engagement with beneficialeffects in different areas including learning The impor-tant point here is that variability can be voluntarily cho-sen maintained and modified in everyday life and thatexplicit variations may have important consequences

Implications for SocietyConsideration should be given to the importance of re-

inforcement contingencies to support variability in so-cial milieus A basic point made throughout the presentarticle is that merely permitting behavior to vary doesnot necessarily engender or maximize variability (as isshown by Yoke and Prob schedules) Analogously merelypermitting varied social behavior as in a free or laissez-faire society may not suffice to produce the desired di-versity If the results shown for individual organismscan provide useful information for groups (see eg Grottamp Neuringer 1974) then when variations are desiredsocieties might reward individuals for varying not sim-ply permit it

SummaryOperants differ in the extent to which reinforcers in-

fluence levels of variation When variability is highlysensitive to reinforcement the operant may be perceivedto be voluntary Voluntary responses have two attributesThey are functional implying that the class of responsesmdashthe operantmdashcan be predicted given sufficient knowl-edge But instances emerge stochastically implyingsome degree of unpredictability Because reinforcementof variability can generate extremely large classes withstochastic emergence of instances it suggests limitationsof the two basic goals of Skinnerian behavioral theorymdashnamely prediction and control Operant variability also hasimplications for self-experimentation and for society

RELATED AREAS

Operant variability may be important in many otherareas and I will briefly review some of these

Negative reinforcement To my knowledge there havebeen no published studies in which variability is nega-tively reinforced High variability might sometimesavoid or reduce a punisher such as loss of goods ormoney but at other times low variability could be func-tional Effects of positive and negative reinforcement onvariability may have different evolutionary bases onederived from increasing probabilities of locating sourcesof food or mates the other from avoiding confrontationsor predation (Driver amp Humphries 1988) One questionconcerns how quickly variability can be changed by neg-ative as opposed to positive reinforcement There isconsiderable evidence that loss functions differ fromgain functions (Kahneman amp Tversky 1984) possiblyindicating that positive reinforcement-of-variabilityfunctions will differ from negative ones as well

Resistance to change In some of the studies describedabove variability appeared to be more resistant tochange than repetitions were for example alcohol hadlittle effect on variable operant responses but interferedwith repeated ones Doughty and Lattal (2001) tested thegenerality of these f indings using four-response se-quences with pigeons a threshold variability contin-gency in the Var component of a multiple schedule andLRLR in the Rep component Doughty and Lattal chal-lenged stability of responding by prefeeding the birds inone experiment and providing reinforcers independentlyof responding in the other (see Nevin 1992) Operantvariability was again more resistant to change than wasoperant repetition Thus reinforcement of variabilitymight increase the stability of responding (that is its re-sistance to change) a seemingly paradoxical findingHowever Mechner (1992) suggested that this is the casefor complex skills such as piano performance Varyingpractice routines will make the performance relativelyinsensitive to unforeseen changes in the environmentCare must be taken however when variation is comparedwith repetition because they have different measuresAlso the results reported by Doughty and Lattal werebased on the performance of three pigeons and were notconsistent throughout the experiment The hypothesiswarrants further study

Attraction In some bird species complex male songsattract females (for a review see Catchpole amp Slater1995) There is evidence that male song complexity isunder the discriminative stimulus control of a receptive fe-male (Searcy amp Yasukawa 1990) That explicit reinforce-ment may play a role in such song complexity is suggestedby the Manabe et al (1997) study described above An in-teresting question for further research is the extent towhich operant contingencies contribute to the attractioncreated by behavioral variability in these and other species

Discrimination of variability Animals and people canuse the variability of a stimulus array or entropy as a dis-criminative stimulus (Young amp Wasserman 2001) Whatis the relationship between perception of variability onthe one hand and emission of variable responses on theother Do individuals attempt to match their own levelsof variation to those of others in their environments whichwould involve both perception and emission How mightdiscrimination of onersquos own level of variability influenceonersquos ability to modify such levels and do influences thatobscure discriminations therefore affect ability to change(see eg van der Meer van der Weel amp Lee 1995)Hunziker et al (2002) report that although variabilitywas successfully reinforced their human subjects couldnot correctly describe the contingencies of reinforce-ment a finding similar to those of informal observationsin other studies (eg Neuringer Deiss amp Imig 2000)but whether subjects could discriminate variability levelsrather than contingencies was not studied

Attention and awareness When attention is subject tointerference or is withdrawn from the task at hand abilityto vary is degraded (eg Baddeley Emslie Kolodny amp

OPERANT VARIABILITY 701

Duncan 1998 Evans amp Graham 1980 Neuringer amp Voss2002) It is unclear at present how attention influencesthe different processes underlying operant variabilityand in particular memorial and stochastic processesRich areas for additional study are the relationships be-tween attention awareness and consciousness on theone hand and operant variability on the other Voluntaryaction and consciousness are often related (see egLibet 1999)

Concepts Concepts are thought by many to be sets ofinstances (much like operant sets) whose structures aregraded meaning that there are more or less typical in-stances of such concepts as birds (robin is typical ostrichis not) Barsalou (1987) showed that the gradation ofthese cognitive structures is manipulable or as he putsit not stable A number of variables influence structureincluding context individual differences and points ofview Barsalou argued that these structures are createdas the need arisesmdashthat is on line by the thinking indi-vidual The structure of concepts may parallel that of op-erants and in both cases levels of within-structures di-versity or variability may be modified by consequencesOperant variability may be found in the cognitive as wellas in the behavioral domain

Play and humor Highly variable behaviors are ob-served when animals and people especially children en-gage in play and humor (see eg Fagen 1981) The re-sult of play may be acquisition of new responses (seeSutton-Smith 1975) Although not currently needed re-sponses learned during play may provide the source ofvariations for successful shaping of behaviors at anothertime This is quite analogous to the role of conserved ge-netic variants which are passed along to progeny but donot currently serve any identifiable role Play may alsoinvolve learning how to vary operantly

Reinforcement by variability The self-maintainingnature of play suggests that engaging in variable behav-iors may itself be reinforcing Indirect support comesfrom findings that varied stimulation can serve as a re-inforcer and that stimulus repetition becomes aversive(Fiske amp Maddi 1961) Possibly also related are findingsthat varying reinforcers and permitting choice among re-inforcers increase their value (Bowman Piazza FisherHagopian amp Kogan 1997) and that permitting choicesamong varied responses is also preferred to demandingfixed responses (Catania 1980)

Foraging Two issues regarding variability are presentin the foraging literature One has to do with risk prefer-ence versus avoidance (see eg Shettleworth 1998) Asecond has to do with search strategies Most reportedcases of foraging involve systematic search patterns(Stephens amp Krebs 1986) However search patterns aresometimes stochastic or highly variable (see eg Peter-son 1997) One question is whether some species can varybetween these two strategies and the challenge is to iden-tify the pressures leading to and the advantages of each

Animal training Wilkes (1996) has applied operantvariability techniques in developing training proceduresfor pet dogs More generally reinforcement of variabil-

ity may enable the gaining of control over behaviors ofanimals and as has been suggested above may be an im-portant part of the process of shaping

Models of learning Reinforced variations in variabil-ity may be important when human and animal learningis being modeled whether the models are mathematicalin form computer based or machine based (Blumberget al 2002 Brooks 1999 Sutton amp Barto 1998)

CONCLUDING COMMENT

Genes interact with environments to produce diverseanatomical physiological and behavioral phenotypesGenes and environments also provide for another level ofdiversitymdashnamely operant variability There are manysources of operant variability including random environ-mental events memory for past responses and stochasticgeneration each of which ultimately derives from ge-netic and environmental influences To the extent thatoperant variability is based on a stochastic source how-ever the behavioral results cannot be predicted fromknowledge of these influences That is organisms haveevolved to behave unpredictably The positing of sto-chasticity to explain behavior has a history extending atleast as far back as Epicurus with major contributors in-cluding Gustav Fechner (see Heidelberger 1987) EgonBrunswik (see Gigerenzer 1987) Skinner (1959) andSuppes (1984) Knowledge of the stochastic nature ofthe operant and of how it is controlled leads to a worldview that differs from genetic and environmental deter-minism with roots in the works of the scholars just men-tioned At the scientific level this view leads to the ex-perimental analysis of consequence-influenced variabilityAt the philosophical level it leads to explanations of vol-untary action At the personal level it can lead to change

REFERENCES

Amabile T M (1983) The social psychology of creativity New YorkSpringer-Verlag

Antonitis J J (1951) Response variability in the white rat duringconditioning extinction and reconditioning Journal of Experimen-tal Psychology 42 273-281

Archer J amp Birke L (1983) Exploration in animals and humansCambridge Van Nostrand Reinhold

Arnesen E M (2000) Reinforcement of object manipulation in-creases discovery Unpublished undergraduate thesis Reed College

Baddeley A D (1966) The capacity for generating information byrandomization Quarterly Journal of Experimental Psychology 18119-129

Baddeley A [D] Emslie H Kolodny J amp Duncan J (1998)Random generation and the executive control of working memoryQuarterly Journal of Experimental Psychology 51A 819-852

Baer D M amp Sherman J A (1964) Reinforcement control of gen-eralized imitation in young children Journal of Experimental ChildPsychology 1 37-49

Baker M J amp Koegel R L (1999) Transforming children withautismrsquos ritualistic behaviors into positive play interactions with sib-lings Paper presented at the meeting of the California Association ofBehavior Analysis San Francisco

Balsam P D Deich J D Ohyama T amp Stokes P D (1998) Ori-gins of new behavior In W OrsquoDonohue (Ed) Learning and behav-ior therapy (pp 403-420) Boston Allyn amp Bacon

Balsam P D Paterniti A Zechowy K amp Stokes P D (2002)

682 NEURINGER

the variability contingency four times FR 4 The resultswere that as the food reinforcer was approached (ie asprogress was made through the ratio) the subjects wereless and less successful at varying The probability ofmeeting the variability contingency in the first trial afterreinforcement was higher than it was in the second whichin turn was higher than that in the third trial and so onFor comparison a Rep group was reinforced for repeatingsequences once again with reinforcement only after foursuccessful repetitions Probability of repeating increasedwith proximity to reinforcement a finding consistentwith the literature but opposite that for the variabilitygroup Thus approach to reinforcement facilitated oper-ant repetition but interfered with operant variabilitySimilar variability-interfering effects have been docu-mented for within-trials responses The probability that aresponse repeats the just prior response increases within atrial even when sequence variability is reinforced (McElroyamp Neuringer 1990 Neuringer 1991) To return to the ex-ample of a composer rushing to complete her composi-tion she may indeed have increasing difficulty respond-ing creatively under the pressure of a deadline On theother hand the soloist may find it easier to repeat thecorrect performance during practice periods as the con-cert nears How can these results be reconciled with themany findings that variability can be reinforced

An additional result of Cherot et al (1996) provides ananswer The Var animals responded much more variablyoverall than did the Rep animals Although variability inthe Var groups decreased with approach to the reinforceroverall variability was always much higher when it was re-inforced (Var group) than when it was not (Rep group)Reinforcement therefore exerts two influences Overallvariability is elevated sometimes to the highest levelswhen it is reinforced but approach to (and possibly focuson) the reinforcers constrains or lowers variability Sim-ilar reinforcement-produced interference effects had beenreported previously for example when raccoons are re-inforced for dropping a ldquocoinrdquo into a slot the reinforce-ment results in the animalsrsquo holding and rubbing the coinrather than dropping it (Breland amp Breland 1961) Theoverall enhancement of variability was of much greatermagnitude in Cherot et al than the decrease with approachwas although both effects were statistically significant(The notes created by the composer under pressure ofdeadline will be less readily predicted than will thoseplayed by the performer practicing a well-known piece)This study highlights the importance of interactions be-tween operant effects of reinforcement and elicited effectsproduced by anticipation of reinforcement and the resultsmay be relevant to the debate concerning possible detri-mental effects of reinforcement on creative behaviors (seethe discussion below and Neuringer in press)

SummaryReinforcement controls behavioral variability as it

does other operant dimensions Levels of variability areprecisely governed by schedule contingencies and dis-

criminative stimuli Choices to vary or repeat match rel-ative frequencies of reinforcement Extinction increasesvariability slightly whereas the hierarchies of sequenceprobabilities are retained and operant and respondenteffects interact The evidence shows that behavioral vari-ability is an operant controlled by its consequences

GENERALITY AND PARAMETERS

Many different methods measures species and re-sponses have been used in the study of operant variabil-ity and this diversity supports the generality of the phe-nomenon As is the case for repeated operants howeverthere are parametric constraints This section briefly re-views the generality and some of the parameters studiedto date

GeneralityMethods Throughout this section ldquoresponserdquo will

signify a response unit including both an individual re-sponse and a multiresponse sequence Each of the meth-ods to be described increases or maintains variability byreinforcing it

Novel response procedures reinforce responses upontheir first occurrence ldquoeverrdquo (Pryor et al 1969) or upontheir first occurrence within a session (Goetz amp Baer1973) Radial-arm maze procedures also reinforce ini-tial (within a given session) entries into arms of a mazeand result in a distribution of choices across the availableoptions the sequence or pattern of choices often beingquite variable

Lag procedures reinforce a response if it has not oc-curred across a given number of previous trials (Page ampNeuringer 1985) For example under lag 5 the currentsequence is reinforced only if it has not occurred duringany of the previous five trials A variant is the recurrence-time procedure which specifies a minimum number of in-tervening trials before a previously emitted response canagain be reinforced Machado (1989) combined recurrencetime with a percentile-reinforcement contingency to gen-erate high levels of variability (see also Machado 1992)Percentile-reinforcement contingencies base the criterionfor reinforcement on the subjectrsquos own performance over aprevious set of responses (see Galbicka 1994) For exam-ple a sequence will be reinforced only if it has not oc-curred over the previous n trials the value of n continuallychanging so that 50 of the trials are reinforced

Least-frequent response procedures (Blough 1966Schoenfeld Harris amp Farmer 1966 Shimp 1967) rein-force responses that are currently least frequent across amoving window

Threshold procedures reinforce responses whose rela-tive frequencies fall below a given value Because thistype of procedure was used in the research describedabove and in many other more recent studies details willbe provided In Denney and Neuringer (1998) which willserve as an example trials consisted of four responses byrats across L and R levers A running tally was kept of the

OPERANT VARIABILITY 683

number of occurrences of each of the 16 possible se-quences LLLL LLLR LLRL and so on If the relativefrequency of the current sequencemdashthe number of its oc-currences divided by the sum of all 16 sequencesmdashwasbelow a specified threshold value in this case 05 the ratwas rewarded Recently emitted sequences contributedmore to the maintained tally than did those in the distantpast because after each reinforcement all 16 counterswere multiplied by a weighting coefficient in this case95 Therefore the influence of a particular trial decreasedexponentially with successive reinforcements Thresholdprocedures differ from Bloughrsquos (1966) least-frequentprocedure in that more than one sequence (or response)can be reinforced at any given timemdashnamely all those se-quences that have relative frequencies below the thresholdvalue The procedure differs from lag in permitting agiven sequence to be reinforced many times in successionif the associated relative frequency is sufficiently low

The frequency dependence procedure (Machado 19921993) is a variant of threshold in which the probability ofreinforcement is a continuous function of relative re-sponse frequency The more frequent a response is thelower the probability is that it will be reinforced

Statistical feedback procedures provide subjects withcomparisons between their performances and that of amodel Neuringer (1986) showed human subjects feedbackfrom 10 statistical tests thereby enabling them to comparetheir own distributions of responses to that of a randommodel Neuringer and Voss (1993) provided feedbackbased on a chaotic model The goal in each of these caseswas for the subjects to learn to approximate the model andthe results of these experiments will be discussed below

Differential reinforcement of switches versus repetitionsbalance a subjectrsquos tendency to engage in each of these pat-terns Rather than reinforcement for infrequent responsesor random-like distributions reinforcement is providedsometimes for repeating a response and other times forswitching from one response to another In the two-operandum case a switch is L followed by R or R by La repetition is L then L or R then R Bryant and Church(1974) found that stochastic-like responding was gener-ated by rats when switches were reinforced with a 75probability and repetitions with a 25 probability that isthe rats had a natural tendency to repeat at least in theenvironment of this study Machado (1997) reinforcedpigeons for eight pecks across two keys if the sequenceof pecks contained at least one (one group of birds) ortwo (another group) switches Although the birds couldhave satisfied these contingencies by repeating a singlesequencemdashin the ldquoone switchrdquo condition for examplerepetition of LRRRRRR would produce reinforcement on100 of the trialsmdashthey tended to vary their sequencesBarba and Hunziker (2002) showed however that vari-ability was higher when it was directly reinforced thanwhen reinforcement was based on proportions of switchesa finding consistent with that of Machado (1997)

The evidence from each of these methods supports thehypothesis that variability can be reinforced

Measures Variability must be interpreted with refer-ence to a particular measure The reason is that there canbe high variability (and low predictability) at one level ofanalysis but order (and certainty of prediction) at a dif-ferent level For example the sequence 1 1 2 2 3 3 44 5 5 1 1 2 2 has the same variance (a commonlyemployed measure of variability) as 2 3 2 1 5 1 4 41 1 3 2 5 2 but an analysis of conditional proba-bilities in which the unit contains pairs of instances in-dicates that the two sequences have different levels ofvariability In general high variability observed with onemeasure does not rule out the possibility of order ob-served with another and vice versa That is why evalua-tions of random-number generators often employ manydifferent measures and indeed there are hundreds of dif-ferent measures of the randomness of a sequence (Knuth1969) All behavioral studies make simplifying assump-tions and rely on one measure or at the most a few

U-value has been the most commonly employed mea-sure of operant variability (Evans amp Graham 1980Machado 1989 Page amp Neuringer 1985 Stokes 1995)U-value measures the distribution of relative frequen-cies or probabilities of a set of responses For a set of16 possible responses i = 1ndash16 U-value is given by

where RFi = the relative frequency (or probability) ofeach of the 16 responses U-values approach 10 whenrelative frequencies approach equality as would be ex-pected over the long run from a random generator andthey approach 00 when a single instance is repeated

Other measures include the percentage of trials or fre-quency at which variability contingencies are met (Page ampNeuringer 1985) percentages of alternations versus stays(Machado 1992) conditional probabilities of responses(Machado 1992) the number or frequency of novel re-sponses (Goetz amp Baer 1973 Schwartz 1982) the num-ber or frequency of different responses (Machado 1997Schwartz 1982) Markov analyses (Machado 1992) andstatistical tests of distributions of responses (Neuringer1986) All of the measures employed to date indicate thatvariability can be reinforced

Responses and species Generality has further beendemonstrated by the use of many different responses andspecies These include Siamese fighting fish (Betta splen-dens) swimming in quadrants of a tank (Roots 2000)young chicks pecking keys (Neuringer 2002) pigeonspecking keys (Page amp Neuringer 1985) budgerigarssinging songs (Manabe Staddon amp Cleaveland 1997)rats pressing levers (Van Hest van Haaren amp van de Poll1989) porpoises swimming and jumping (Pryor et al1969) rhesus monkeys pressing levers (Sydney Reisbickpersonal communication) children drawing and build-ing with blocks (Holman Goetz amp Baer 1977) andhuman adults and children pressing buttons (Saldana ampNeuringer 1998 Stokes amp Balsam 2001) There havebeen few attempts to compare species or responses but

- ( )[ ] ( )=aring RF RFi ii

log log 2 21

1616

684 NEURINGER

in one study lowest baseline variability was observedwhen rats pulled trapezes intermediate levels were ob-served when they pushed keys and highest levels wereobserved when they pressed levers (Morgan amp Neuringer1990) with variability increasing in all cases when it wasreinforced In another study operant variability wascompared across rats and people with similar effects ob-served in the two species (Neuringer Deiss amp Imig 2000)

ParametersWe turn now to some parameters studied to date Many

other parameters as yet unstudied are no doubt also im-portantmdashfor example motivation and types of reinforcers

Age Older people (see eg Van der Linden Beertenamp Pesenti 1998) and older rats (see eg Lamberty ampGower 1990) respond less variably than their youngercounterparts when variable responding is reinforced or re-quested but in most cases it is not clear to what extent thevariability is elicited as opposed to reinforced Neuringerand Huntley (1992) compared reinforced variability in2-month-old (young) and 10-month-old (mature) Long-Evans rats Under Yoke contingencies the young rats re-sponded significantly more variably (two levers four re-sponses per trial) than the mature rats which constitutesan elicited effect When reinforcement was contingenton variability (lag 4) variability increased significantlyin both groups thereby overcoming the initial differencecorrelated with age Thus explicitly reinforcing vari-ability in older animals caused them to respond no dif-ferently than younger animals did a finding potentiallyrelevant to real-world human aging However when Wells(1999) compared 8-month-old (mature) to 24-month-old(old) rats although the mature rats responded more vari-ably than the old rats did under Yoke contingencies thedifference was not statistically significant Additionalstudies are needed especially of whether reinforcingvariable behaviors in older populations can facilitatelearning and performance

Gender Male and female rats are equally sensitive toreinforcement of variability (Neuringer amp Huntley 1992van Hest et al 1989) However there was a tendency(not statistically significant) for males to respond over-all more variably than females did and because thesestudies did not control for differences in size and weightof subjects or for other aspects of motivation additionalstudies are necessary

Stage of training Stokes and Balsam (2001) reporteddifferences in variability as a function of whether variabil-ity was initially reinforced early or later in training (seealso Gottlieb 1992) Human subjects generated sequencesof 10 L and R responses per trial on a computer keyboardunder lag 0 and lag 25 contingencies Under lag 0 everytrial ended with reinforcement under lag 25 the currentsequence had to differ from each of the previous 25 Whenlag 25 was experienced at the very beginning of traininglevels of variability at the end of training (when lag 0 wasoperative) were lower than they were when lag 25 was ex-perienced after 50 trials of initial lag 0 These results were

interpreted to suggest that there is an ldquooptimal period forsetting variability levelsrdquo (Stokes amp Balsam 2001 p 181)but that general conclusion needs further testing and anumber of questions need to be answered One has to dowith the contribution of reinforcement frequency Underlag 0 every sequence leads to reinforcement but rein-forcement is intermittent under lag 25 Another has to dowith whether learning to vary differs from learning to repeatwith respect to optimal periods (see eg Bouton 1994)

Related to stage of training are effects of prior experi-ence Hunziker Lee Ferreira da Silva and Caramori(2002) showed that for human subjects if Var contingen-cies were experienced just prior to Yoke contingencieslevels of response variation during the Yoke phase werehigher than they would have been if the Var contingencieshad not been previously experienced The time delay be-tween Var and Yoke experiences mattered with a long in-terval (3 months) producing only a small effect Saldanaand Neuringer (1998) also found effects of prior Var train-ing on performances in Yoke (increased variability inYoke) but the opposite did not hold that is training underYoke had relatively little effect on Var Hunziker Caramorida Silva and Barba (1998) reported similar effects withrats Thus there may be an asymmetry Reinforcement forvarying has relatively strong influences but reinforce-ment without regard to variations does not adversely affectlearning to vary These different effects are not surprisinggiven that Yoke contingencies are permissivemdashreinforce-ment is given whether or not responses varymdashwhereascontingencies under Var can be quite demanding How-ever the results motivate a number of additional ques-tions One is whether effects of Var training differ fromthose observed when repetitions are reinforced Another iswhether direct reinforcement of variable behaviors canhelp animals or people to overcome the detrimental effectsof prior noncontingent experiences such as those that pro-duce learned helplessness (Seligman 1975)

Environmental constraints Stokes (1995) arguedalso that variability increases when behavior is con-strained The term ldquoconstraintrdquo is ambiguous because allreinforcement contingencies exert a form of constraintIn one study Stokes (1995) compared different strate-gies of shaping leverpresses Rats in one group were re-inforced initially for pressing the lever with only theright paw (right only) and eventually for any press (withright or left paw or with any other part of the body) Theorder of training was reversed for a second group ldquoanyrdquofirst and then right only Behavior sequences were morevariable in the right-only-first group leading Stokes(1995) to conclude that early constraints lead to highvariability (see also Stokes Mechner amp Balsam 1999)That is an important hypothesis with many implicationsbut controls are needed before the results can be attrib-uted to the contingencies For example ldquoright onlyrdquo andldquoanyrdquo differed in terms of reinforcement distributions aswell as in the precision of the reinforcement operationthe ldquoanyrdquo reinforcers were provided immediately and au-tomatically by the programming equipment whereas in

OPERANT VARIABILITY 685

the right-only condition a human observer had to decidewhether a response had occurred and then present the re-inforcer necessarily leading to longer and more variedlatencies and a less consistent contingency Furthermorethere are many cases in which increasing constraints suchas those due to a decrease in the number of responses pertrial (Page amp Neuringer 1985) or the requirement of ex-actly four responses on each of two keys (Schwartz 1980)results in decreased variability Thus the contributionsof response-definition constraints and environmentallimitations must be explored further

SummaryThe generality of operant variability is shown by the

robustness of the phenomenon across procedures mea-sures responses and species As with other operant phe-nomena parameters influence the effects

FUNCTIONS

All operants including variability are functional Op-erant responses produce reinforcing consequences Be-cause other behavioral competencies depend on it oper-ant variability is functional in a broader sense as wellSome of these functions will now be described

Operant ConditioningThe potential of a reinforcer to change behavior is lim-

ited by the range of available variations (Hull Langmanamp Glenn 2001 Skinner 1981) and thus whatever pro-duces those variations contributes to the conditioning ofnew operants Much work has been done to identifysources of variation (Balsam Deich Ohyama amp Stokes1998 Segal 1972 Staddon amp Simmelhag 1971) butNeuringer (1993) was first to show that the same rein-forcers that strengthen individual responses can also pro-duce the necessary variability and that these two functionsof reinforcement may work simultaneously Implied isthat when an operant response is being conditioned re-inforcement works not only to strengthen an individualresponse but at the same time to engender the variabilityfrom which the response emerges (That is variability isnot solely a baseline phenomenon elicited or noise)Concurrent reinforcement of variations and individualtarget sequences will be the topic of this section

Pigeons in one experiment and rats in another were re-warded for varying among 16 possible response sequences(four responses per trial L and R operanda lag contin-gencies Neuringer 1993) Concurrently one of the 16designated as the target sequence was reinforced when-ever it occurred Thus reinforcements were given bothfor sequence variations and for a single selected targeta reinforced-variation-and-selection (RVS) procedureWhen the target was ldquoeasyrdquo (eg LLLL) it emergedrapidly from the baseline variations Given that the tar-get was always reinforced reinforcers were obtained morefrequently when the subjects did not vary and only re-peated the target When the target was ldquodifficultrdquo (eg

LLLR) it was not learned As in the choice experimentdescribed above (Neuringer 1992) the relative valuesor difficulties of target versus variations governed theprobability that the target would be learned

However even difficult targets could be conditionedunder the RVS procedure as was shown by another ex-periment in the same series (Neuringer 1993) Sequencevariations were again reinforced concurrently with a dif-ficult target RRLR but now the frequency of reinforce-ments for varying was systematically decreased Satis-fying the variability contingency was intermittentlyreinforced according to a VI schedule and over sessionsreinforcement for varying became less and less frequentThis technique resulted in acquisition of even the mostdifficult target sequence whereas a control group rein-forced for the target alone (without the concurrent rein-forcement for variations) never learned

Reinforcers were obtained more frequently in the RVScondition than in the control condition (since the RVSsubjects were rewarded both for varying and for emittingthe target sequence) and the additional reinforcers mighthave generated greater motivation to respond (Killeen1981) Thus motivation not the direct reinforcement ofvariability might have been responsible for the RVS ef-fect To test this possibility Neuringer Deiss and Olson(2000) reinforced three groups of rats for emitting a par-ticularly difficult target sequence LLRLR The experi-mental group was reinforced concurrently for sequencevariability on the average of once per minute (VI 1 min)The target-only control group received no reinforce-ments other than for the LLRLR target A third groupwas yoked to the experimental animalsmdashreinforced onceper minute for any sequence variability not requiredmdashand as for the other two groups whenever the target oc-curred If the additional reinforcement was responsiblefor the facilitated learning under RVS then because theexperimental and yoked groups received equal reinforce-ments they should have learned equally well but thatwas not the result Only the experimental group learnedto emit the LLRLR target The yoked controls respondedat high rates throughout the experiment but they did notvary sufficiently to make contact with the target contin-gency and never learned The target-only control groupreceived insufficient reinforcement and their responsesextinguished Therefore the added reinforcers helped tomaintain motivation as suggested above but reinforce-ment of variations was necessary for learning of a diffi-cult sequence The RVS procedure maintains both highmotivation and high variation and the two together re-sult in the acquisition of difficult-to-learn responses

Grunow and Neuringer (2002 Experiment 2) studiedone possibly important parameter of the RVS proceduremdashnamely the level of variability When a difficult target re-sponse is being trained does it matter whether reinforcedvariability is low intermediate or high As describedabove each of four groups of rats was reinforced for vari-able response sequences the first group for high vari-ability the fourth group for low variability and the other

686 NEURINGER

two for intermediate levels In other ways the procedurewas similar to that of the Neuringer Deiss and Olson(2000) RVS contingencies just described with a singledifficult target sequence always reinforced The resultswere that the higher the level of reinforced baseline vari-ability the more quickly the target was learned Thusvariability levels indeed mattered and if the just-re-ported effect is found to be general it will be most help-ful to reinforce high variability when difficult-to-learnnew responses are being shaped Other parameters re-quiring study are class definitionsmdashwhich responses areincluded in the class of acceptable variationsmdashand num-ber of instances within the class

One additional effect should be noted When varia-tions among a set of responses are reinforced if one vari-ant is selected for extinction (no longer reinforced) thefrequency of that instance decreases Neuringer (1993)showed this for pigeons responding under a lag-variabilityschedule When one of the 16 sequences was no longerreinforced (with no other change to indicate its status) thefrequency of that sequence decreased Thus reinforce-ment of variations provides the baseline to select for (iestrengthen) and select against (ie weaken) individualinstances

These findings have potentially important implicationsFirst as has been indicated the same reinforcers thatstrengthen particular instances can also reinforce varia-tions That a binary eventmdashreinforce or not reinforcemdashsimultaneously affects variations and selected instancesindicates both the power of reinforcement and a concep-tual problem How reinforcers exert such diverse effectsis a question to which we will return

Second shaping procedures may implicitly reinforcebaseline variations An approximation to a goal responseis reinforced for a period of time but eventually rein-forcement is withheld until a response closer to the goalis obtained It is commonly thought that withholding re-inforcement or mini-extinction induces the necessaryvariations for shaping to proceed as has been indicatedabove An alternative suggested by the present studiesis that organisms learn both to vary (within a set) and to re-peat (selected instances) until one instance is always rein-forced That is the process of shaping might involve re-inforcement both of variations (generally done implicitly)and of targeted instances If so a useful contributor tothe shaping process might be explicit reinforcement ofvariations

Third reinforcement of variations could facilitate ac-quisition of other competencies such as cognitive andmotor skills Siegler (1996) has shown in an important se-ries of studies that childrenrsquos acquisition of complex cog-nitive skills is correlated with high variability Anotherexample is that when motor skills are practiced in a vari-able manner high quality of performance results (Ma-noel amp Connolly 1995 1997 Mechner 1992 Schmidtamp Lee 1999) However it is rare that variability in thesedomains is directly reinforced as part of the training oreducational process

Fourth the RVS procedure might be used in therapeu-tic settings to modify maladaptive and self-injurioushabits Therapists sometimes employ noncontingent re-inforcement procedures to weaken undesirable behaviors(Vollmer Iwata Zarcone Smith amp Mazaleski 1993)The reasoning is that many maladaptive behaviors aremaintained by unintended consequences and that non-contingent rewards may compete with those conse-quences thereby lowering the strength of the unwantedhabit For example when a staff personrsquos attention toself-injurious behavior helps to maintain it providing at-tention noncontingently may decrease the strength of theunwanted target behavior However the RVS proceduremight accomplish the same goal more effectively andwithout undesirable side effects (Vollmer RingdahlRoane amp Marcus 1997) Noncontingent rewards some-times result in unwanted ldquosuperstitiousrdquo behaviors andthat would be avoided under RVS Furthermore RVSwould help to provide the set of baseline behaviors nec-essary to establish desirable new responses

Finally RVS procedures may help to gain control overand eventually to weaken other undesirable target behav-iors The target can be incorporated within a set of moredesirable responsesmdasha new operant class established withthe target as one membermdashand variable instances rein-forced Once variable responding is successfully main-tained including the undesirable target response the lattercan be weakened by selectively withholding reinforcement

Summary The conditioning of operant responses de-pends on both baseline variations and selection of suc-cessive approximations to a goal response The same re-inforcers used to select the approximations can reinforcebaseline variations thereby defining the precise set ofappropriate variants and the desired levels of variationSuch concurrent reinforcement of variations and selec-tions may facilitate the acquisition of difficult-to-learnoperants and selective withholding of reinforcementmay weaken undesirable instances

Creativity and Problem SolvingBehaving in a novel unusual or nonrepeating manner is

part of creative activities (Campbell 1960) Novelty andvariation may arise through combinations of previouslylearned behaviors (Epstein 1996) transformations of thesebehaviors (Boden 1994) direct reinforcement of variabil-ity as described above or some combination thereof Ofcourse neither novelty nor variability alone suffices forcreative outputmdashthe throw of the dice or the toss of acoin are not creative nor are haphazard strokes of a paint-brush on a canvasmdashbut without variation creativity islow Whether the variability required for creativity canbe reinforced is relevant to whether creativity itself canbe reinforced The latter question has led to contentiousdebate with some arguing that reinforcement facilitatescreativity (Cameron amp Pierce 1994 Eisenberger ampCameron 1996 Stokes 2001 Winston amp Baker 1985)and others focusing on detrimental effects of reinforce-ment (Amabile 1983 Deci Koestner amp Ryan 1999)

OPERANT VARIABILITY 687

The debate goes well beyond the present discussion butsome of the research described above is relevant Firstreinforcement contingent on variability generates higherlevels of behavioral variability than does reinforcementcontingent simply on responding as has been shown bythe many studies described above in which Var contin-gencies were compared to Yoke contingencies Similarlyprecise reinforcement contingencies matter for creativity(Cameron amp Pierce 1994 Lepper amp Henderlong 2000)If any product rather than only creative ones is rein-forced creativity tends to fall (just as variability decreasesunder yoke conditions) Second approach to or antici-pation of reinforcement interferes with operant variabil-ity (Cherot et al 1996 McElroy amp Neuringer 1990Neuringer 1991) which again might be related to effectson creativity when an individual focuses on the rein-forcer Note however that although variability decreaseswith proximity to the reinforcer overall levels of vari-ability are much higher when variability is reinforced thanwhen it is not The same may be true for creativity aswell anticipation of reward may interfere with creativeoutput to some extent but at the same time high creativ-ity may depend on its reinforcement

Operant variability may be important for problemsolving as well Here too there is disagreement as towhether training is helpful or not Early studies by Maier(1933) indicated that it is and these found support inlater work (eg Lung amp Dominowski 1985) Somestudies showed that those individuals most likely toldquobreak the setrdquo are also most likely to solve a problem(Dover amp Shore 1991) However a number of studiesfound that practice had no influence (Duncan 1961)

Arnesen (2000) explored a related phenomenon witha rat model One group Var was reinforced for variableinteractions with novel objects and another group Yokewas reinforced independently of such interactions EachVar rat received training with 12 different objects oneper session For example in the presence of a soup canthe rat was reinforced the first time it entered the canbut thereafter a different response was required such asrolling it climbing on it and so forth This procedure issimilar to that of Goetz and Baerrsquos (1973) described aboveexcept that objects changed from session to session Ratsin the Yoke control group had identical experiences in anadjoining identical chamber that contained identical ob-jects but now rewards were not contingent on object ma-nipulation After this training the rats were tested in adifferent room with 30 never-before-experienced ob-jects scattered on the floor Hidden in each was a smallpiece of food When compared to their Yoke controls theVar animals were more active during this test exploredmore manipulated objects more and most importantlydiscovered significantly more food pellets Another con-trol group received no preliminary training with objectsand their performance in the test session did not differfrom that of the yoked controls Therefore when ratswere reinforced for variable interactions with objectsthe result was an increased tendency to explore manip-ulate and discover new sources of reinforcement It

would be of considerable interest to determine whethera similar effect occurs with children

Assessment and Treatment of PsychopathologiesAbnormal levels of behavioral variability characterize

some human psychopathologies as has been shown byclinical observations as well as by research with humansubjects and animal models Few studies have attemptedto reinforce variability directly in such problem popula-tions The main question is whether reinforcement canmodify abnormal levels of variability in the direction ofnormalcy and whether such changes might have associ-ated beneficial effects

Autism Stereotyped responding characterizes indi-viduals with autism (Hertzig amp Shapiro 1990) For ex-ample Baron-Cohen (1992) asked children with autismand others in a control group to hide a penny in one handso that the experimenter could not guess which hand heldthe penny Those with autism were more likely thanthose without to generate a simple predictable patternsuch as repeatedly switching back and forth from left toright hands Such repetitive responding can be maladap-tive For example Mullins and Rincover (1985) askedchildren with and without autism to pick one of five cardsFood was occasionally found in a cup behind the cardEach of the five cards was a discriminative stimulus fora different schedule of reinforcement including contin-uous reinforcement (CRF every choice reinforced) FR 2(fixed ratio 2 in which every second choice was rein-forced) FR 4 (every fourth choice reinforced) FR 7 andFR 11 Control participants sampled all five alternativesand quickly learned to choose the most frequently rein-forced CRF card Autistic children sampled only a lim-ited number of cards and therefore often preferred anonoptimal alternative Thus children with autism tendto respond less variably than normal children and thiscan result in failure to adapt

Miller and Neuringer (2000) asked whether direct re-inforcement could influence the variability of individu-als with autism Sequences of responses on two largebuttons were first reinforced probabilistically and inde-pendently of variability (Prob phase) after which rein-forcement was contingent upon satisfying a thresholdvariability contingency (Var phase) The autistic partic-ipantsrsquo variability was significantly lower than that ofmatched controls during both Prob and Var phases butwhen variability was reinforced its levels increased sig-nificantly in both groups This is an important findingbecause it suggests that direct reinforcement of variabil-ity may help to modify some of the behavioral stereo-types that are characteristic of autism Lee McComasand Jawor (in press) offer support for this hypothesisThree individuals with autism were reinforced for variedverbal responses to social questions the responses had tobe appropriate given the context Variability increased intwo of the three individuals In a related procedureBaker and Koegel (1999) reinforced children with autismfor varied social interaction in a game-playing contextAlthough the reinforcement was permission to engage in

688 NEURINGER

the stereotyped behaviors preferred by the child the re-sult was increased social interactions both during theperiod of the experimental manipulation and in 1- and2-month follow-ups These three studies taken togethersuggest that reinforcement of varied behaviors may facil-itate modification of nonfunctional ritualistic and stereo-typed behaviors

Depression Depression provides another example ofmaladaptive consequences of low variability Lapp Mar-inier and Pihl (1982) observed that depressed womenproduced fewer alternative solutions to hypotheticalproblems than did nondepressed women Channon andBaker (1996) found that attempts by moderately de-pressed college students to identify faults in a series ofinterconnected circuits were less varied and less success-ful that were those of nondepressed controls HorneEvans and Orne (1982) found that when depressed pa-tients were asked to generate a random sequence of num-bers their responses were less variable than were thoseof controls These experimental results are consistentwith clinical observations

Can depressed individuals be rewarded for increasingvariation Hopkinson and Neuringer (in press) dividedcollege students into mildly depressed and nondepressedgroups on the basis of the CES-D scale a paper-and-pencil self-evaluative index of depression In the first oftwo phases sequences of responses on a computer key-board were reinforced independently of variability (Prob)following which only highly variable sequences were rein-forced (Var) Under Prob depressed students respondedsignificantly less variably than did the controls in agree-ment with previous findings Under Var variability in-creased significantly in both groups Indeed by the endof the experiment depressed and nondepressed subjectswere responding equally variably (Instructions were alsoeffective in increasing variability in the depressed par-ticipants) Thus even in a simple computer-game envi-ronment mildly depressed subjects manifested lowervariability than controls did but direct reinforcementmodified their behavior to control levels Can reinforce-ment of variability be used in real-world conditions tohelp depressed individuals An affirmative answer comesfrom cognitive behavior modification procedures thatwork to increase alternative solutions to problems (Beck1976) A more general conclusion from these studies ofautistic and mildly depressed individuals is that direct-reinforcement-of-variability procedures may benefitthose who manifest abnormally low levels of variability

Attention Deficit Hyperactivity Disorder (ADHD)ADHD may provide an example of the opposite stateabnormally high levels of variability (Barkley 1990)Wultz Sagvolden Moser and Moser (1990) proposedthat the spontaneously hypertensive rat (SHR) may serveas a good animal model of ADHD the SHR shows manyof the behavioral characteristics of that disorder includ-ing hyperactivity poor learning ability poor ability totolerate delayed reinforcement and poor self-control

The next study to be described asked about operant vari-ability in SHRs

Mook Jeffrey and Neuringer (1993) compared vari-ability of SHRs and of a control strain Wystar Kyoto(WKY) under two conditions Prob and Var The mainresult was that SHRs tended to respond variably whetheror not the contingencies required it consistently with theresults from the human ADHD literature When vari-ability was reinforced the two strains responded at ap-proximately equally high levels of variation These re-sults were confirmed in two other studies (HunzikerSaldana amp Neuringer 1996 Mook amp Neuringer 1994)Thus control rats varied only when the contingencies re-quired variations whereas SHRs varied whether or notthe contingencies required it As might be expected fromthese results when reinforcement depended on repeti-tions the SHRs performed poorly (Mook et al 1993)

Another similarity between SHRs and ADHD humanswas seen when amphetamine a drug that has effects simi-lar to those of Ritalin often prescribed for ADHD was ad-ministered (Mook amp Neuringer 1994) When repetitionswere required for reinforcement amphetamine improvedperformances by the SHRs so that they were as accurate assaline-injected WKYs a finding again consistent withthose in the ADHD literature These studies support SHRsas a model of ADHD and suggest that additional attemptsshould be made to assess whether or not individuals withADHD are less able to change levels of variability thanare controls Although as indicated above there is someevidence in support of this hypothesis (Barkley 1990)when Saldana and Neuringer (1998) directly comparedhuman ADHD and control participantsrsquo operant variabil-ity within the context of a computer game no differenceswere observed Most of those with ADHD were main-tained on Ritalin and other drugs however a fact unfor-tunately not documented in the original article

Drug EffectsAlthough many drugs are known to influence behavioral

variability (Brugger 1997) rarely have attempts beenmade to analyze explain or identify the sources of theeffects As will be discussed below operant variabilitycan result from the memory-based process of remember-ing previous responses and then behaving differently orcan result from responding stochastically a process thatappears to be quite different McElroy and Neuringer(1990) attempted to distinguish the effects of alcohol onmemory from its effects on stochastic processes in astudy in which one group of rats was reinforced for vary-ing four-response sequences under a lag 5 contingency(Var) and another was reinforced for repeating a singlesequence LLRR (Rep) Ethanol injections of 075 and20 gkg affected Var and Rep performances differentlyWhereas the Rep animals were much less able to satisfythe LLRR contingency at both doses the Var rats re-sponded variably whether or not alcohol had been ad-ministered Similar results were obtained by Cohen et al

OPERANT VARIABILITY 689

(1990) with a multiple schedule in which each rat servedas its own control In the same animals and within thesame sessions alcohol degraded Rep performances butleft Var intact Thus alcohol appears to interfere with atask that requires working memory as presumably is thecase for the Rep task but not with one that reinforcesvariability The memory-interfering effect of alcoholmight account for why alcohol also degrades radial-armmaze performance (Devenport 1983)

Alcohol also affects choices to vary or to repeat For ex-ample Gorka (1993) reinforced rats concurrently for re-peating and varying and monitored percentage of choicesas a function of time following injection of ethanol Inthe control sessions when the rats were injected withsaline the probabilities of varying and repeating remainedapproximately constant across the 45-min session witha slight bias in favor of varying After injection with 15gkg of ethanol however there was a significant increasein choices to vary at the beginning of the session andthen a later decrease to below 50 That is alcohol ledinitially to a markedly increased preference for varyingbut later to a preference for repeating (Figure 5) This re-search should be replicated and extended especially be-cause the contingencies ldquopressuredrdquo the animals to re-main at approximately 50 preference but thesuggestion is that alcoholrsquos effects on response variabil-ity may depend in part on the duration of its influence Amore general implication is that drugs may affect re-peated and variable operants differently and that effectsdepend on time following administration

SummaryReinforced variability may contribute to many types

of learning including operant shaping and acquisition ofcognitive and complex motor skills Similarly creativityand problem solving depend on variable responding andin these areas as well explicit reinforcement of variationsmay be functional Abnormal levels of variability aresometimes caused by psychopathologies and drugs andoperant variability may be useful in these areas in twoways first as a way of assessing the variability that is pos-sible and second to modify variability toward more nor-mal levels Studies show that individuals with autism andmild depression vary less than do controls but that ex-plicit reinforcement of variability moves the levels towardsnormalcy Other studies show that drugs such as alcoholmay affect repeated and variable operants differently

EXPLANATIONS

Two questions must be answered in order to explainoperant variability First what are the sources of vari-ability that permit its reinforcement Second how is itmaintained Related questions including how instancesbecome members of a set from which variations emergeand whether operant variability is a generalizable com-petency will also be considered

Origination From Nonoperant SourcesSkinner (1966) reasoned that operant responses orig-

inate from nonoperant sources with reinforcement con-tacting a behavior shaping and strengthening it onlyafter the behavior had occurred for reasons other than re-inforcement To put it simply a behavior must occur be-fore it can be reinforced (Balsam amp Silver 1994) I willconsider three initial sources of variable behavior en-dogenous variability noncontingent environmental eventsand extinction These appear to parallel evolutionaryprocesses and they too will be identified

Endogenous variability All physical phenomenavary and behavior is no exception This fact can providethe opportunity for reinforcement to select particularlevels of variation Examples of spontaneous variabilityinclude spontaneous alternations in mazes (Dember ampRichman 1989) nondifferentially reinforced variationsin operant responses (Guthrie amp Horton 1946) and ex-ploration (Archer amp Birke 1983) Endogenous muta-tions provide a parallel in genetics Base-pair changes inDNA molecules occur spontaneously intron-bearinggenes mix and match segments to encode new proteinsjumping genes introduce new DNA sequences and atthe level of the chromosome random crossings and as-sortment occur as a natural part of reproductive meiosisThese sources of random genetic change occur indepen-dently of external influences

Figure 5 The relative frequency of variations by rats as a func-tion of 5-min blocks of time since the beginning of the sessionThe open circles represent performance after the rats were in-jected with saline indicating a slight tendency to vary more thanrepeat but one that is approximately constant throughout thesession The filled circles represent performance after injectionwith 15 gkg ethanol indicating a much higher likelihood ofvarying soon after the alcohol is injected followed by a decreasein variations to below 50 and then a return to baseline at theend of the session Group averages are shown Data are fromGorka (1993)

690 NEURINGER

Noncontingent events Noncontingent events some-times referred to as chance accidents or luck also affectbehavior One happens upon a particular passage in a bookthat leads to a new thought one happens to sit next to aninteresting woman or man on an airplane which leads tomarriage (Bandura 1982) In science as well as in every-day life accidents or serendipity are an important part ofthe process of discovery (Beveridge 1957) In each of thecases presented above an accidental event leads to new be-haviors that are then strengthened by consequences Thesame is potentially true for variability itself Particular lev-els of variability might occur for adventitious reasons butthen become functionally related to consequences

That organisms are particularly sensitive to such unan-ticipated noncontingent events is shown by two basicprinciples of learning habituation and Pavlovian condi-tioning Organisms attend preferentially to unexpectedevents and stop attending or habituate to repeated onesThe conditioning of new Pavlovian responses dependsaccording to influential theories upon unexpected rein-forcing events (eg Rescorla amp Wagner 1972)

Noncontingent events such as X-rays and other pro-cesses that can act as mutagens play an important role ingenetic variation as well The particular genes affectedand the type of changes are random there is no neces-sary relationship or contingency between the externalevent and the affected gene However sensitivity to mu-tations is not random but has been selected presumablybecause it is functional Thus in both behavior and ge-netics sensitivity to noncontingent events is conserved

Extinction Extinction increases behavioral variabilityat least to some extent as was described above Distance inspace or time from the reinforcer does the same (Cherotet al 1996) A possible parallel at the genetic level in-volves high levels of deprivation at least in simple lifeforms For example when E coli bacteria are deprivedof nutrients unusually high mutation rates are observedThe exact mechanism by which such hypermutations areinduced is not known but they increase the likelihoodthat a population can survive such environmental insults(Moxon 1997) That is deprivation-induced variability isfunctional at the genetic level (The behavioral evidenceregarding long-term deprivation is less clear some stud-ies showing increased variability and others increasedstereotypy eg Carlton 1962 Senkowski Vogel amp Po-zulp 1978)

Endogenous variations external noncontingent eventsand withholding of reinforcement are three sources forthe variability that precedes control by reinforcementBut once control by reinforcement is established a dif-ferent question emerges What behavioral processes areinvolved when variability is maintained as an operantThe sources just discussed generate uncontrolled non-specific and often short-lived variability and cannot ex-plain the precisely controlled operant type

Operant Variability ProcessesThree processes are hypothesized to underlie operant

variability The first involves the use of random environ-

mental events such as the toss of a coin The second in-volves memory for responses such as when an animallearns not to return to previously visited locations Thethird will be referred to as an endogenous stochasticgenerator

Random environmental events Throughout historymany cultures have used random events to influencechoices and decisions such as throws of dice and randomselections of sticks cards bones or organs Today coinflips are used by a referee at the beginning of a game to de-cide which team kicks the ball a computerrsquos random-number generator is employed by scientists to avoid bi-ases in assigning subjects to experimental groups andaleatoric events are sometimes employed in modern artmusic and literature The Dice Man by Rhinehart (1998)provides a fictional example in which the protagonistbored with life writes a number of possible actions onslips of paper and then periodically selects one blindlyand behaves accordingly These examples show that randomexternal events may be used to avoid biases engender un-likely responses and break out of behavioral niches

Memory-based variability Many methods used tostudy operant variability differentially reinforce nonrepe-titions at least over the recent past Therefore one strat-egy that subjects can employ is to cycle through few re-sponses or sequences especially when the variabilitycontingencies are relatively permissive For exampleunder a lag 2 contingency cycling repeatedly throughthree sequences will result in 100 reinforcement Ani-mals and people sometimes do this apparently basingtheir current response on memory for or discriminativecontrol by the just emitted responses One such instancewas shown by Schoenfeld et al (1966) who rewardedrats for IRTs that fell into one of 10 interval classes onlyif the current IRT differed from the interval class of theprevious response The rats came to alternate betweenlong and short IRTs thus behaving systematically to meetthis lenient variability contingency Machado (1993)studied an analogous effect having pigeons peck L and Rkeys under a frequency-dependent variability contin-gency related to Shimprsquos (1967) procedure describedabove When the trial unit consisted of two responses thebirds developed memory-based repetitive sequencessuch as RRLLRRLL and thereby satisfied the variabilitycontingency When the trial unit was three responses inlength the birds did not develop the optimal fixed pat-tern of RRRLRLLLmdashwhich was too complexmdashbut in-stead developed ldquorandom-like behaviorrdquo (Machado1993 p 103) Thus a memory-based strategy was em-ployed when it was possible but when the memory loadbecame too high stochastic responding emerged A sim-ilar pattern was seen when variability of budgerigarsongs was reinforced (Manabe et al 1997) Under lag 1contingencies the birds sang mainly two songs underlag 2 they increased to three songs Only when the lagwas increased to 3 did song diversity increase apprecia-bly in one of the birds In radial-arm mazes as well whencontextual cues are available memory again appears toplay an important role leading to much lower probabil-

OPERANT VARIABILITY 691

ities of repetition than would be expected from a sto-chastic generator (McElroy amp Neuringer 1990)

Note that under lag schedules memory-based strate-gies result in higher reinforcement rates than respondingstochastically does The reason is that a stochastic gen-erator repeats by chance That animals and people oftendo not employ memory-based strategies probably indi-cates the difficulty of remembering long sequences andpossibly other advantages of unpredictable responding

If a memory strategy is the only way to satisfy vari-ability contingencies then it can be used even when thememory requirements are demanding as was shown byNeuringer and Voss (1993) Human subjects were trainedto generate on a computer keyboard a chaotic-like se-quence of responses that matched iterations of the logistic-difference equation

where Rn refers to the nth iteration in a series with R be-tween 00 and 10 and t is a constant between 10 and40 When the t parameter approaches 40 iterations ofthis equation result in very ldquonoisyrdquo sequences As withother chaotic processes however there is an orderlinessunderlying the surface noise In this case if responsesare autocorrelated with Rn drawn as a function of Rn21the resulting function is a parabola In the Neuringer andVoss (1993) study following each response college stu-dents were shown how close their predicted values wereto that from iterations of the logistic-difference model Inone experiment the students made digital predictions inanother they placed a pointer along a horizontal lineand this analogue response was then translated into dig-ital values for computation With training the studentsbecame increasingly adept at responding in chaotic-likefashionmdashthe studentsrsquo predictions matched closely theiterations of the logistic functionmdashand their autocorrela-tions became parabolic The left column in Figure 6 showsperformances by 4 subjects (rows) early in training thatis not well approximated by the logistic-differenceparabola and the right column shows performances at theend of training in which r 2 values were 72 99 98 andapproaching 10 How was this accomplished Sinceeach iteration in the logistic-difference sequence is basedon the prior outputmdashthat is the sequence is generated bya deterministic although nonlinear processmdashit was hy-pothesized that the human subjectsrsquo chaotic-like responseswere also based on memory for prior responses To put itsimply the subjects may have memorized a long series ofpairs in the form ldquoIf the previous response was value Athen the current response must be value Brdquo as Metzger(1994) Ward and West (1994) and Ward (2002) have sug-gested

To test whether subjects in fact relied on memory forprior responses Neuringer and Voss (2002) systemati-cally slowed responding by interposing pausesmdashthat isby requiring an IRT As IRT increased the difference be-tween the subjectsrsquo sequences and the modelrsquos chaotic

Rn n nt R R= -( )- -1 11

Figure 6 Autocorrelations (responses on trial n on the y-axis asa function of n21 on the x-axis) for 4 human subjects at the be-ginning (left column) and end (right column) of training inNeuringer and Voss (1993) Equations for the best-fitting curveand amount of variance accounted for (r2) are shown at the topof each panel To the extent that the circles (responses) approxi-mate the equation y = 40x 2 40x 2 the subjects were respondingin a way that matched the logistic-difference chaotic functionFrom ldquoApproximating Chaotic Behaviorrdquo by A Neuringer andC Voss 1993 Psychological Science 4 p 115 Copyright 1993 bythe American Psychological Society Reprinted with permission

692 NEURINGER

output increased and the underlying parabolic structurewas broken That is when pauses increased subjects wereless and less able to approximate the modelrsquos chaotic dis-tribution (see Figure 7) The important point here is thathighly variable responses which in this case are chaoticin nature can be based on memorial processes

Endogenous stochastic generator The evidence tobe reviewed next indicates that animals and people canrespond stochastically and that this competency is basedneither on chance environmental events nor on memoryfor prior events or responses It is impossible to provethat an endogenous stochastic process underlies operantvariability that would be analogous to proving the nullhypothesis However much evidence is consistent withthe stochastic hypothesis Included are the facts thatmemory limitations do not permit adequate respondingunder many variability contingencies that response dis-tributions match those expected from a stochastic gener-ator that changes in parameter values affect operantvariability as they would a stochastic generator and thatinterference with memory processes leaves stochasti-cally based operant variability unaffected As a wholethe evidence is consistent with an endogenous stochas-tic source of operant variability

Memory limitations One type of evidence is nega-tive in character and is based on the presumed inability ofmemory for prior responses or sequences to satisfy sometypes of variability contingencies Two examples weregiven above (Machado 1993 Manabe et al 1997) andanother was provided by a lag 50 contingency in Pageand Neuringer (1985) In the latter case the sequence in

the current trial (eight responses across L and R keys) wasrequired to differ from those in each of the previous 50 tri-als including at the beginning of a session the last 50trials from the previous session Pigeons responded wellunder these contingencies and in a way similar to that ofa stochastic model It is unlikely that pigeons can remem-ber each of their previous 50 eight-response sequences

Response slowing As was shown above for chaotic-like responding performances are often degraded by de-lays imposed between a stimulus and a response pre-sumably because of limitations in working or short-termmemory (Berryman Cumming amp Nevin 1963 Williams1971) Neuringer (1991) relied on this effect to test whethermemorial processes could explain the generation of vari-able sequences under lag schedules Two groups of ratswere reinforced for L and R leverpresses producing fourresponses per trial One group (Var) was trained under alag 5 contingency and the other (Rep) was trained to asingle sequence of responses LLRR It was assumedthat accurate Rep performance depended on workingmemory After performances had stabilized IRTs weresystematically varied by the imposition of timeouts of05 to 20 sec between consecutive responses in the se-quence During these forced pauses the chamber wasdark and responses were not counted toward completionof the sequence If the animals had used the previous re-sponse(s) as a discriminative cue for the current responsethen performances should have been degraded by theseinterpolated pauses

The results were that for the Rep group as the re-quired pauses increased from 0 to 6 sec percentages ofcorrect sequences were relatively unaffected but at pausesgreater than 6 sec probability of a correct LLRR se-quence fell sharply This pattern of results suggests in-volvement of working memory of prior responses withthe rats able to remember responses within a 6-sec win-dow The Var grouprsquos results were quite different Aspauses increased between 0 and 6 sec the rats were moreand more likely to satisfy the variability contingenciesand with pauses greater than 6 sec percent correct re-mained at a high and constant level Thus the Var grouprsquosresults were opposite to those of the Rep group and op-posite to those predicted from a memory-based strategySimilar results showing that operant variability increasesor is maintained as responding is slowed have been re-ported by Morris (1987) with pigeons and by Baddeley(1966) with random response generation by human sub-jects The results reported above concerning alcoholrsquos ef-fects on variability and repetition were also consistentwith those of Neuringer (1991)

Memory for prior responses may not have controlled theVar grouprsquos performance but why did variability increaseEach of three answers is consistent with a stochastic-generator interpretation Weiss (1964 1965) hypothe-sized that random responding requires that current re-sponses be independent of previous ones memory for orcontrol by prior responses interferes with such indepen-

Figure 7 Approximation to a chaotic sequence (given by delta)as a function of the imposed interresponse times Shown are theaverages of 5 subjects

OPERANT VARIABILITY 693

dence and therefore pauses would be expected to facil-itate variability A second possibility is that at shortIRTs animals tend to repeat responses on the sameoperandum Blough (1966) found this in pigeons and ex-cluded such double pecks from his analyses because theyappeared not to be under the control of reinforcement con-tingencies Morris (1987) also found a tendency for birdsto repeat when no interresponse timeouts were imposedA third hypothesis is that there were two contributors tothe observed variability One was a stochastic processcontrolled by reinforcement and the other elicited vari-ability generated by interposition of pauses The high vari-ability elicited as an effect of slowed responding is ageneral phenomenon supported in many other cases Ac-cording to this interpretation operant variability in the Vargroup was governed by a stochastic-based process operantrepetition in the Rep group was governed by a memory-based process and pauses elicited variability under bothcontingencies Thus Rep performance was interferedwith whereas Var was facilitated Each of these hypothe-ses is consistent with the conclusion that memory for (ordiscriminative control by) prior responses does not con-tribute to and possibly interferes with variable respond-ing when an organism is reinforced for variability andthis in turn is consistent with the stochastic-generatorhypothesis

Stochastic response distributions Other evidence iscorrelational Response distributions are similar to thosefrom a stochastic generator The data from two experi-ments already discussed are shown in Figure 1 In thesecases reinforcement of least frequent IRTs (Blough 1966)and of nonrepetitive choice sequences (Machado 1989)resulted in stochastic-like responding

Neuringer (1986) reported another striking case Highschool and college students generated sequences of 1sand 2s on a computer keyboard with each set of 100 re-sponses constituting one trial A baseline phase lastedfor 60 such trials for a total of 6000 responses duringwhich the students were asked to behave as randomly aspossible as if they were reporting the outcome of a se-ries of coin tosses As in the many previous studies ofthis sort the subjectsrsquo responses differed significantlyfrom those of a stochastic model (Brugger 1997 Wage-naar 1972) During the training phase the subjects re-ceived feedback which enabled them to compare their per-formances with that of a stochastic model first accordingto one statistical test and then according to another untilfollowing each trial feedback was provided from 10 dif-ferent statistics The results were that all of the subjectslearned to approximate the stochastic model accordingto the 10 statistics as well as some (but not all) other sta-tistical tests That is the distributions of the statisticswhich clearly did not match a stochastic model at the be-ginning of training approximated the stochastic modelat the end of it (see Figure 8)

The previous failures of other researchers to obtain ev-idence for random responding had led some to concludethat humans did not have the capacity to behave ran-

domly What would account for the difference betweenthe present positive finding and the many other negativeones Neuringerrsquos (1986) study was the first in which sto-chastic responding was explicitly reinforced in this casewith feedback from 10 statistical tests An analogy mighthelp to explain the difference in procedures and resultsImagine an experimenter walking around a city and ask-ing people if they know what a violin sounds like (analo-gous to asking people whether they know what a series ofcoin tosses or dice throws is like) Most would answeryes If the same individuals were then asked to play theviolin very few if any would succeed The experimentermight conclude after many replications with differentsubjects in different cities that people did not have thecapacity to play the violin The point of course is thatplaying the violin is a skill requiring training and prac-tice and the same may be true for responding randomlyIt is easy to vary but practice and reinforcement may benecessary to approximate a random distribution

Response number Page and Neuringer (1985) testedthe stochastic-generator hypothesis in another way byvarying number of responses per trial with pigeons work-ing under a lag 3 contingency The reasoning was that ifprevious responses served as a discriminative cue forwhat not to domdasha memory-based strategymdashthen perfor-mance should be degraded when the number of re-sponses was increased Eight responses per trial requiresubjects to remember more than four responses The sto-chastic hypothesis predicts the opposite outcome as isdemonstrated by the following example If each trialwere two responses in length with responses determinedby the toss of a coin then the probability that a given trialwould repeat the previous one would be 25 (There arefour possible sequences in the first trial RR RL LRand LL Thus the second trial has a 1-in-4 chance ofmatching the first) If a trial consists of four coin tossesthe probability of a repetition by chance is 0625 or 1 in16 Thus if subjects used a stochastic process to gener-ate Ls and Rs performances should improve with in-creasing responses per trial Results were precisely thosepredicted by the stochastic hypothesis and inconsistentwith a memory strategy Probability of meeting the con-tingencies and therefore probability of reinforcementincreased as responses per trial increased In otherwords eight-response trials were easier for pigeons thanfour-response trials

Response interference Neuringer and Voss (2002)attempted to distinguish between memorial and stochas-tic processes in yet another way They hypothesized thatwhereas chaotic responding was based on memory forprior responses (as was discussed above) responding inNeuringerrsquos (1986) study with feedback from 10 statisti-cal tests indicated involvement of an endogenous sto-chastic generator To show that human subjects could en-gage in both processes at will three college studentswere trained to generate chaotic sequences in one compo-nent of a multiple schedule as in Neuringer and Voss(1993) and to generate stochastic sequences in a sepa-

694 NEURINGER

rate component as in Neuringer (1986) The wordCHAOTICS was on the screen in the first component theword STOCHASTICS was on during the other and thesewords alternated throughout the session After the sub-jects succeeded at each of the component tasks test ses-sions were introduced in which memory for prior re-sponses was subject to interference in both the chaoticsand stochastics components as follows During the

chaotics component four separate chaotic sequenceswere cycled one after the other each cued by differentscreen colors The subjects attempted to approximate achaotic sequence in one color (eg light green) follow-ing which they would have to match a second chaotic se-quence (lavender screen color) and so on the fourchaotic contingencies following one another across thesession Although all four chaotic sequences used the

Figure 8 Performance by one subject at the end of training to respond stochastically in Neuringer(1986) Shown are comparisons on 6 different statistics between the subject (solid lines and Xs) andthe random model (dashed lines and closed circles) At the end of training the subjectrsquos perfor-mance did not differ statistically from that of the random model on these and four additional sta-tistics From ldquoCan people behave lsquorandomlyrsquo The role of feedbackrdquo by A Neuringer 1986 Jour-nal of Experimental Psychology General 115 p 71 Copyright 1986 by the American PsychologicalAssociation Reprinted with permission

OPERANT VARIABILITY 695

same logistic-difference equation that a model did theywere out of phase and independent If chaotic respond-ing is memory based then the four independent se-quences would be expected to interfere with one anotherthat is approximations to the four chaotic models wouldbe lower than when only a single chaotic output was re-quired as in Neuringer and Voss (1993) That was ex-actly the finding Performances by the three subjectswere signif icantly degraded during the four-segmentphase To put it simply chaotic responses interfered withone another

A different pattern of results was seen in the stochasticsportion of the experiment The same interference proce-dure was used with four independent stochastic sequencesrequired in the four different color conditions As inNeuringer (1986) subjects were required to satisfy eightdifferent statistical tests of randomness and to do this in-dependently in each of the four phases (If responses in anytwo or more of the four stochastic phases became corre-lated above a minimal level responses were considered tobe incorrect thereby prohibiting repetition of a single se-quence in two or more of the phases) For one subjectseven of eight statistics were closer to those of a randommodel during this interference phase than they were at theend of the original training with only a single sequenceand for a second subject all eight statistics were closerThe third subject showed closer-to-random approximationunder the four-segment condition on only two of the eightstatistics but most of her changes were also in the directionof a random generator For example if a random generatoremits equal numbers of ldquouprdquo and ldquodownrdquo sequences andthis subject in the baseline phase had emitted only 47ldquouprdquo sequences then during the four-segment tests shecame to emit ldquouprdquo 55 of the time She changed in the di-rection of the random model but overshot the mark Thuswhereas interference degraded chaotic performance sig-nificantly in all three subjects it improved stochastic per-formances in at least two of them These results are con-sistent with the hypothesis that a memory-based processcontrols chaotic variability and that a stochastic processnot dependent on memory controls stochastic variability

Summary of Tripartite TheoryThe following three processes constitute a tripartite

theory of operant variability attention to random events inthe environment (coin toss) memorial processes (chaosradial maze) and stochastic processes (response se-quences and random-number generation with feedback)Random environmental events assure unpredictabilitymdashfor example in making unbiased selections Remember-ing what was done previously will enable one to avoidrepetitions to vary the students called upon in class tovary the locations when looking for a hidden object orto try different approaches when attempting to solve aproblem External aids such as notes to oneself or sci-entific records help to enhance memory capacities

An endogenous stochastic generator provides the thirdand most controversial source of behavioral variability

When an opponent is attempting to predict your behav-ior in a game or at war it may be functional to behavestochastically Similarly operant conditioning of new re-sponses may depend on stochastic variations It is im-possible to prove stochasticity some presently unknowndeterministic system may ultimately explain the resultsHowever the evidence is consistent with the stochastichypothesis and inconsistent with the alternatives For ex-ample memory is not sufficiently powerful to explain alloperant variability memory-interference proceduresleave stochastic responding intact and behaviors corre-late well with those of stochastic models

Both genetics and reinforcement play important rolesin all three processes Reinforcement is always involvedin that learning provides information about when whatwhere and how much to vary Genetic influences areshown in that not all responses are possible and if theyare possible they are not all equiprobable (eg as inNeuringer et al 2001) Stochastic responding may be bi-ased due to experiences and knowledge that is biasesmay be established by reinforcement contingencies or bythe interface between the organismrsquos propensities (or be-havior systems see Timberlake amp Lucas 1989) and theenvironment In the case of random-number generationby human subjects biases may be due to misconceptionsconcerning what ldquorandomrdquo means However some bi-ases can be overcome through feedback and instructions(Lopes amp Oden 1987 Neuringer 1986)

The importance of variations for evolutionary changeis supported by the many sources of genetic variabilityand by the fact that genetic variability is conserved Asimilar argument holds for operant variability That thereare at least three independent sources of operant varia-tions as well as other elicitors such as environmentalnoise and response extinction supports its importancefor normal adaptive functioning

Structure of VariabilityEvery operant ldquoresponserdquo is actually an instance of a

class with each member of the class exerting the sameeffect on the environment (Skinner 1959) When a rat isreinforced for pressing a lever for example the class isdefined by closure of a microswitch and any way of ac-complishing the closure whether with the left paw theright paw or the mouth is as effective in producing re-inforcement as any other If the contingencies of rein-forcement so specify however the response class can beredefined and circumscribed which is another charac-teristic of the operant For example if only right-pawpresses are acceptable members of the reinforced classresponding will tend to occur only with the right paw(Stokes 1995) Of course the capabilities of the organismplay a role for example including ldquotail pressesrdquo in the op-erant class does not mean that the lever would be or couldbe pressed by the ratrsquos tail The structure of the operantclass therefore emerges from the current reinforcementcontingencies and the organismrsquos capability derivedfrom genetics development and prior experiences

696 NEURINGER

Operant variability has structure as well and as for alloperants that structure is due partly to the organism it-self and partly to the reinforcement contingencies Theresearch described throughout this article has providedexamples of the structure of variability generally in theform of sets of four or eight responses across L and Roperanda But the structure of variations can be speci-f ied more precisely For example Mook and her col-leagues (Mook et al 1993 Mook amp Neuringer 1994)reinforced rats for varying four response sequences acrossL and R levers under a lag 1 contingency Unlike in pre-vious studies however the acceptable class was furtherconstrained so that only sequences beginning with two Rresponses (RRLL RRLR RRRL and RRRR) were rein-forced The rats learned to vary their sequences withinthe imposed limitations that is they generally begantheir sequences with two R responses and varied the se-quences within those limitations Similar results wereobtained in the context of a radial-arm maze in whichonly a subset of arms was baited

The precision with which reinforcement contingen-cies can specify class was highlighted in research per-formed by Burns (1997) and Ross and Neuringer (2002)In the latter case college students drew rectangles on acomputer screen with the computerrsquos mouse Each rec-tangle constituted one trial (the screen cleared at the endof each trial) with reinforcement (points) depending onsimultaneous variations along three dimensions of therectangle location on the screen area of the rectangleand shape with the latter defined as the ratio of the rec-tanglersquos height to its width Although the subjects weregiven no instruction other than to ldquogain pointsrdquo theylearned to vary along all three dimensions with levels ofvariability significantly higher than those of yoked controlsubjects who were reinforced independently of variations

A second part of the Ross and Neuringer (2002) studydemonstrated even greater precision of control To gainpoints three new groups of subjects were required to re-peat along one of the dimensions while simultaneouslyvarying along the other two One group was required torepeatedly draw rectangles of approximately the samesize while simultaneously varying location and shapeanother group had to repeat the shape while varying areaand location and the third to repeat location while vary-ing size and shape Verbal instructions were minimal asthey were in the first part of the study Subjects learnedto satisfy these contingencies as well varying two at-tributes while simultaneously repeating one and signif-icant differences were obtained rapidly within 300 trials

These demonstrations raise questions as to how binaryfeedbackmdashreinforcement presented or notmdashcan simul-taneously control independent behavioral dimensionsThe issue is similar to that discussed above concerningsimultaneous reinforcement of variations and selectionsIt is similar also to those issues faced by researchers at-tempting to understand how the nervous system strength-ens particular synapses when many different neurons firesimultaneously and to ldquoallocation of creditrdquo in artificial

learning models For present purposes suffice it to notethe exquisite precision with which reinforcement controlsvariability and repetition

GeneralizationDo animals and people learn to ldquovaryrdquo implying a

generalizable skill or do they learn a specific distributionof specific responses across particular operanda Thequestion of generalization has been asked of many otherbehavioral competencies such as imitation When chil-dren learn to imitate do they learn stimulusndashresponseassociations (if mother moves her right hand the childlearns to move his right hand) or do they learn moregenerally to ldquoimitaterdquo Evidence indicates that reinforce-ment of imitative instances leads to generalized imitation(see eg Baer amp Sherman 1964 Bandura 1986) A sim-ilar question has been raised when pigeons rats or chil-dren learn to choose a stimulus that matches a previouslyshown sample For example if a pigeon is shown a reddisk as a sample it must then choose red when given achoice between that and blue Does the bird learn to matchIn this case the answer appears to depend on the train-ing procedures If trained with few stimuli such as redgreen and blue then the pigeon learns simple associa-tions (if red sample choose red if blue sample chooseblue) but if hundreds of different stimuli (pictures) areemployed during training then generalized ldquomatchingrdquois learned (Wright 1997) In each of these examples thelearning of a general competency or strategy was shownby transfer-of-training tests What was learned with oneset of stimuli or in one situation transferred to new stim-uli and situations We turn now to the question of whetherlearning to vary transfers

After the subjects in Maltzmanrsquos (1960) experimentdescribed above learned to generate remote verbal as-sociates they were asked to devise unusual uses of com-mon objects In comparison with those in the controlgroup these subjects generated significantly more un-usual uses indicating transfer of training to a differenttask However transfer was not observed in the oppositedirection from unusual uses to verbal associates

Holman et al (1977) using a procedure similar to thatof the Goetz and Baer (1973) research described aboverewarded preschool children verbally for novel drawingsand found increased novelty Unrewarded block construc-tions were also monitored and weak evidence was ob-tained for transfer of novel responding from rewardeddrawing to the unrewarded constructions In a second ex-periment preschoolers engaged in four independent taskspainting pen drawing wood block construction and Legoconstruction When the children were reinforced for vary-ing their pen drawings the novelty of paintings increasedalong with the pen drawings indicating transfer from onetask to the other However transfer to the topographicallydifferent block- or Lego-building tasks was not observed

Eisenberger and Armeli (1997) gave monetary rewardsto one group of fifth- and sixth-grade students the vari-ability group for describing unusual uses of common ob-

OPERANT VARIABILITY 697

jects A second group the repeat group received thesame rewards for describing common uses of the sameobjects In a later test all of the children were asked todraw pictures incorporating a circle and the group that hadpreviously been reinforced for varied responses drewmore unusual (or lower frequency) pictures than did therepeat group Thus there was transfer of novelty trainingfrom one domain to another with additional studies in theseries showing similar effects (Eisenberger Armeli ampPretz 1998 Eisenberger Haskins amp Gambleton 1999Eisenberger amp Selbst 1994) However Eisenberger et al(1999) failed to replicate the transfer-of-training effectThe variability and repeat groups responded similarly dur-ing the test phase In this study separate experimenterstrained and tested the subjects thereby avoiding possibleexperimenter-expectancy effects whereas the same ex-perimenter did both training and testing in Eisenbergerrsquosother studies Across the series of experiments there weremany other cases in which transfer was not observed andmore research is necessary to explain why

In Lee et alrsquos (in press) study with autistic individualsdescribed above after variable answers were trained toquestions verbal variability transferred to new experi-menters and new contexts

Arnesenrsquos (2000) results described above are consis-tent with transfer of training in an animal model Rats inthe Var group were trained to interact variably with oneset of objects When later tested in a new situation withnew objects the Var animals showed (in comparisonwith controls) higher levels of exploration manipulationof the novel objects and discovery of hidden reinforcers

Much of the evidence reviewed in this section there-fore indicates transfer of training and supports a generalldquovaryrdquo competency However there remain unresolvedquestions the most important of which is the extent towhich transfer occurs Note also a methodological pointMany of the studies tested whether transfer occurs whenthe contingencies are neutral with respect to the to-be-transferred behavior Holman et al (1977) is a case inpoint Variations were reinforced in one domain draw-ing and the question was whether variations wouldoccur in another domainmdashblock constructionmdashin whichreinforcement did not depend on varying A negativef indingmdashin this case absence of transfermdashindicatesnothing about the ability to generalize but only aboutthe tendency to do so given the absence of a reason (orcontingency) A better test would be to compare experi-mental and control performances when variability in thenew task is in fact functional as in Arnesen (2000)

SummaryBefore variability can be reinforced it must occur for

reasons other than reinforcement Some of the reasonsinclude variability inherent in all behavioral phenomenaldquoaccidentalrdquo environmental events that are unrelated toparticular behaviors and withholding of reinforcementTransfer from other situations may serve a similar functionFor example if an organism can discriminate shaping-

of-new-response periods from others then it may learngenerally to ldquovaryrdquo during shaping episodes After rein-forcement makes initial contact with response variationsthere are at least three behavioral strategies that can beemployed in order to maintain functionally high levelsof variability respond based on external random eventssuch as the toss of a coin avoid repetitions by remem-bering what responses had previously been made and re-spond stochastically Evidence for an endogenous sto-chastic process was seen in the similarity of responsedistributions to those generated by stochastic sourcesand by findings that interfering with memory for priorresponses either improves or leaves unaffected stochas-tic behaviors Stochastic responding as well as othertypes of variability-generating strategies depend uponcontingencies to define the required level of variabilityand the set from which variations emerge

IMPLICATIONS

One implication of reinforced variability is that oper-ant responses may differ in the extent to which their vari-ability is affected by reinforcement contingenciesmdashthatis operants may be ordered along a continuum of sensi-tivity of variability to reinforcement Another is that re-inforced variability causes us to question the generalityof some basic goals of behavioral psychology includingprediction and control A third is that reinforced vari-ability may help us to provide a scientifically useful wayto describe voluntary action In particular the variabilityof all voluntary behaviors can readily be changed mo-ment to moment by discriminative stimuli and conse-quences Implications for everyday life also will be noted

Operant ContinuumAs was indicated above the operant is composed of

instances that is the operant is a class of many differentresponses all functionally equivalent but often topo-graphically distinct Skinner suggested and I hypothe-size that instances within the class emerge stochastically(see Moxley 1997 for a discussion of Skinnerrsquos posi-tion) Although reinforcement contingencies and contextinfluence the individual probabilities and their distribu-tions emission is stochastic nonetheless The particularmember of the operant class generally cannot be predictedwith a high degree of certainty For example a particu-lar rat may be more likely to press the lever with its leftpaw than its right but left- and right-paw responses areemitted probabilistically This class-nature of the operantsuggests one way to conceptualize operant variability interms of size of class and within-classes probability dis-tributions High variability implies large classes equalprobability distributions or both low variability impliesthe opposite

Variability may be modified more readily in some op-erants than in others implying that operant variability isnot an all-or-none phenomenon but rather a continuumwith reinforced variations in variability (RVV) differing

698 NEURINGER

for different operants RVV might best be understood byanalogy to the concepts of speed and acceleration Bothvariability and speed range from low to high RVV isanalogous to acceleration the rate of change in speedHighly sensitive RVV is indicated by the possibility ofreinforcing many different levels of variability from rep-etition to random-like responding and by the fact thatchanges in level occur readily and rapidly

Different operants manifest differences in RVV Forexample variability of chain pulls changed more thanthat of leverpresses when variability was reinforced inrats (Morgan amp Neuringer 1990) Some operant-like re-sponses may show relatively low RVV For example fir-ing rates increase in single hippocampal nerve cellswhen dopaminergic substances are contingent upon theresponse (Stein Xue amp Belluzzi 1994) suggesting thatnerve f iring is controlled by reinforcement To datehowever there is no evidence that variability of nervefiring is similarly controlled Other cases might includereinforcement of nondiscriminable (subconscious) re-sponses such as Hefferline and Pererarsquos (1963) reinforce-ment of subaware thumb twitches and biofeedback ofmuscle and cardiovascular activity In these cases rein-forcement may modify performance but it is not knownwhether variability is also sensitive to reinforcement andif so to what degree Human verbal variability may bethe most readily manipulable of all behaviors the rangeof predictable to unpredictable verbal responses beingimmense with extraordinary sensitivity to context andconsequence RVV may also differ across organisms (per-haps from simple to complex) individuals (perhaps asan index of intelligence or learning ability) and physio-logical and psychological states (WKY vs SHR depres-sion and autism)

RVV may help to distinguish operant responses fromother superficially similar cases of selection by conse-quences That is the ldquooperantrdquo may also best be concep-tualized as a continuum with RVV an important deter-miner of position along the operant continuum Oneexample of selection by consequences but low ldquooperant-nessrdquo is genetic change Variability is essential for evolu-tionary change of course and levels of genetic variabil-ity are sometimes affected by environmental conditions(Moxon 1997) but there are few examples of levels ofvariability being influenced by selection pressures Thatis the source of much genetic variability is thought to berandom not influenced by current pressures and needsThus although selected by consequences genetic changemay be low along the operant continuum Other exam-ples of operant-like phenomena for which variability although important may be relatively insensitive to con-sequencesmdashvariability in these cases is elicited not se-lectedmdashinclude the development of the central nervoussystem and immune system responses (Hull et al 2001)In both of these cases change depends upon selection byconsequences but it is not clear whether variability canalso be selected Most commonly studied operant re-sponses on the other hand show high sensitivity of vari-

ability to reinforcing events But to reiterate the mainpoint of this section operants differ in terms of RVV andtherefore in terms of their ldquooperantnessrdquo from low sensi-tivity (possibly responses of single nerve cells) to extra-ordinarily high sensitivity (verbal behavior) The operantnature of variability as well as that of every other oper-ant dimension is best conceptualized as a continuum

Prediction and ControlSmith (1992) described Skinnerian operant condition-

ing theory as Baconian in nature a primary goal of Skin-nerian behaviorists being to control behavior and hecontrasted this with the more Aristotelian goal of de-scription Skinner and his students (eg Zuriff 1985) in-deed emphasized prediction and control Many of thestudies reviewed above however involved reinforce-ment procedures that increase variability and conse-quently decrease predictability It is not only ignorancetherefore that restricts prediction and control as Skin-ner suggested but reinforcement itself Operant meth-ods and theories when applied to variability lead togreater emphasis on description with control over pre-cise response characteristics possible under some butnot all conditions

In summary an operant is a class of responses underthe control of contingencies of reinforcement The classcan be controlled and predicted given knowledge ofconditioning history Indeed the size of the class andpossibly the distribution of within-classes probabilities orlevel of variability can also be controlled and predictedBut to the extent that instances emerge stochastically aparticular instance cannot be predicted The instant re-sponse can be explained as a member of a particularclass but prediction of the instance will be diff icultsometimes to the extreme This point will be developedin the next section

Voluntary BehaviorI hypothesize that voluntary behaviors are operants for

which RVVs are highly sensitive to influence by conse-quences One way to test this hypothesis would be togather empirical data concerning how voluntary versusinvoluntary actions are classified by human observers Asfar as I know there are no psychophysical data concern-ing this question but the hypothesis implies that highsensitivity of variability to operant contingencies RVVswill be present in all actions perceived to be voluntary

This operant-variability theory of voluntary action isbased on the supposition of two attributes First volun-tary acts must be potentially explainable (given suff i-cient knowledge) on the basis of genetics conditioninghistories current contingencies anticipated conse-quences intentions goals and the like That is volun-tary behaviors are lawful The second attribute is that thevoluntary act must be potentially unpredictable That isvoluntary acts are ldquofreerdquo of control by identifiable stim-uli If behavior is perceived necessarily to result frommdashto be predicted on the basis ofmdashexternal events includ-

OPERANT VARIABILITY 699

ing reinforcement contingencies it will not be perceivedas voluntary ldquoVoluntaryrdquo implies potential indepen-dence from stimulusndashresponse or responsendashreinforcerdetermination According to this theory both explain-ability and unpredictability are necessary attributes ofvoluntary acts

How though can behaviors be simultaneously ex-plainable and unpredictable Reinforcement contingen-cies play two essential roles Reinforcement (togetherwith genes prior experiences and current environment)helps to determine the set of possible behaviors fromwhich a voluntary act emergesmdashthat is the operant classReinforcement (together with the other contributorsspecified above) also helps to determine the level of vari-ability within the class Thus both instances comprisinga class and level of variability within that class can bepredictedmdashagain given sufficient knowledge Howeverthe individual response is at least sometimes (or relatively)unpredictable The reason is that the operant class fromwhich the response stochastically emerges is sometimesextremely large (eg ldquoname a wordrdquo or ldquodo somethingrdquo)and the contingencies sometimes call for equiprobablestochastic emergence (eg ldquofool merdquo) In the latter casea voluntary act may appear to be foolish meaninglessor unwarranted but if it is voluntary it can potentially beexplained (For example in response to the questionldquoWhy did you do that silly actrdquo the answer could beldquoJust to show that you canrsquot predict my behaviorrdquo seeScriven 1965) In these cases prediction of instancesfalls to very low levels indeed and I hypothesize evenan all-knowing observer will be unable to predict thenext response at greater-than-chance levels which aregiven by the size of the operant class and by within-classesprobability distribution Even within small classes (egin the game rockndashscissorsndashpaper in which unpredictabil-ity among the three possible responses is functional) be-havioral stochasticity may lead to unpredictability of theinstant response Note however that although voluntarybehaviors are sometimes unpredictable the opposite isoften the case When contingencies call for highly con-strained particular responses predictions can easily bemade For example if a police officer asks your nameyour response can most likely be predicted Voluntary re-sponses are therefore described as potentially unpre-dictable with variability contingencies engendering dif-ferent levels of unpredictability

According to this theory behaviors low in the sensitivity-of-variability dimension are described as relatively in-voluntary and those that are high in said dimension areperceived as voluntary Low RVV may be shown at anylevel along the variability continuum For example al-ways responding in the same unchanging way will beperceived as involuntary but so will always respondingrandomly (The coin does not voluntarily come up withheads or tails) This point is essential More than 2000years ago Epicurus argued that occasional randomswerves of atoms provided the source of voluntary be-havior in an otherwise determined universe but al-

though this conjecture was prescient it was not quitecorrect ldquoFree-willedrdquo or voluntary action whether it isbased on random swerves or on quantum mechanics isnot explained by variability per se but by functionalchanges in levels of variabilitymdashthat is by rapid andprecise changes in response to environmental demandsResponses can be placed along a voluntaryndashinvoluntarycontinuum isomorphic to the operantndashrespondent con-tinuum with voluntary behavior being functional pre-cisely in the same way as operant behavior is ThusSkinnerrsquos (1974) suggestion that voluntary behavior isoperant behavior was almost correct Only those oper-ants for which variability is highly sensitive to conse-quences will be perceived as voluntary

One additional example might help Operant classes(sets of potential responses) are created ldquoon linerdquo as we goabout our daily activities As we walk from car to officewe tend to walk in a characteristic way take the samepath with tightly constrained operant classes followingone after the other in rapid succession Class sizes aresmall and within-classes probability distributions arehighly peaked Predictions of next responses by a knowl-edgeable observer can readily be made However classsizes can be increased and probability distributions flat-tened we can walk in quite strange ways indeed dancealong a new path sing hand candy to passers-by Strangebehavior indeed but made possible by the ability of op-erant sets to be modified moment to moment not only interms of class definition but in terms of class size lev-els of within-classes variations and so on The same istrue for conversation Moment-to-moment changes inverbal class definition size and predictability of within-classes instances are hallmarks of voluntary behavior

GenerativityRelated to the definition of the operant and to our discus-

sion of voluntary action is that fact that operant behaviormdashleverpressing no less than languagemdashis by its naturegenerative This claim follows from the functional na-ture of the class of responses (all members of the classare effective in producing an outcome) and the stochas-tic emission of instances From these flow never-before-emitted instances patterns and sequences (see Epstein1996) Operants especially those high on the RVV con-tinuum are inherently generative

Self-ExperimentationVariations systematic or otherwise are a necessary part

of scientific work Neuringer (1981) and Roberts and Neu-ringer (1998) described how variations can be brought to bear explicitly on everyday life in the form of self-experiments Individuals vary aspects of their lives suchas the foods they eat social contacts and methods ofmemorizing they keep records concerning the effects ofthese manipulations and form hypotheses concerningcontrolling variables Using self-experimental methodsindividuals can learn how what when where and howmuch to vary Explicitly varying onersquos everyday behav-

700 NEURINGER

iors and thoughts may help to engender what Langer(1989 2000) has referred to as mindful livingmdashthat isnonautomatic voluntary engagement with beneficialeffects in different areas including learning The impor-tant point here is that variability can be voluntarily cho-sen maintained and modified in everyday life and thatexplicit variations may have important consequences

Implications for SocietyConsideration should be given to the importance of re-

inforcement contingencies to support variability in so-cial milieus A basic point made throughout the presentarticle is that merely permitting behavior to vary doesnot necessarily engender or maximize variability (as isshown by Yoke and Prob schedules) Analogously merelypermitting varied social behavior as in a free or laissez-faire society may not suffice to produce the desired di-versity If the results shown for individual organismscan provide useful information for groups (see eg Grottamp Neuringer 1974) then when variations are desiredsocieties might reward individuals for varying not sim-ply permit it

SummaryOperants differ in the extent to which reinforcers in-

fluence levels of variation When variability is highlysensitive to reinforcement the operant may be perceivedto be voluntary Voluntary responses have two attributesThey are functional implying that the class of responsesmdashthe operantmdashcan be predicted given sufficient knowl-edge But instances emerge stochastically implyingsome degree of unpredictability Because reinforcementof variability can generate extremely large classes withstochastic emergence of instances it suggests limitationsof the two basic goals of Skinnerian behavioral theorymdashnamely prediction and control Operant variability also hasimplications for self-experimentation and for society

RELATED AREAS

Operant variability may be important in many otherareas and I will briefly review some of these

Negative reinforcement To my knowledge there havebeen no published studies in which variability is nega-tively reinforced High variability might sometimesavoid or reduce a punisher such as loss of goods ormoney but at other times low variability could be func-tional Effects of positive and negative reinforcement onvariability may have different evolutionary bases onederived from increasing probabilities of locating sourcesof food or mates the other from avoiding confrontationsor predation (Driver amp Humphries 1988) One questionconcerns how quickly variability can be changed by neg-ative as opposed to positive reinforcement There isconsiderable evidence that loss functions differ fromgain functions (Kahneman amp Tversky 1984) possiblyindicating that positive reinforcement-of-variabilityfunctions will differ from negative ones as well

Resistance to change In some of the studies describedabove variability appeared to be more resistant tochange than repetitions were for example alcohol hadlittle effect on variable operant responses but interferedwith repeated ones Doughty and Lattal (2001) tested thegenerality of these f indings using four-response se-quences with pigeons a threshold variability contin-gency in the Var component of a multiple schedule andLRLR in the Rep component Doughty and Lattal chal-lenged stability of responding by prefeeding the birds inone experiment and providing reinforcers independentlyof responding in the other (see Nevin 1992) Operantvariability was again more resistant to change than wasoperant repetition Thus reinforcement of variabilitymight increase the stability of responding (that is its re-sistance to change) a seemingly paradoxical findingHowever Mechner (1992) suggested that this is the casefor complex skills such as piano performance Varyingpractice routines will make the performance relativelyinsensitive to unforeseen changes in the environmentCare must be taken however when variation is comparedwith repetition because they have different measuresAlso the results reported by Doughty and Lattal werebased on the performance of three pigeons and were notconsistent throughout the experiment The hypothesiswarrants further study

Attraction In some bird species complex male songsattract females (for a review see Catchpole amp Slater1995) There is evidence that male song complexity isunder the discriminative stimulus control of a receptive fe-male (Searcy amp Yasukawa 1990) That explicit reinforce-ment may play a role in such song complexity is suggestedby the Manabe et al (1997) study described above An in-teresting question for further research is the extent towhich operant contingencies contribute to the attractioncreated by behavioral variability in these and other species

Discrimination of variability Animals and people canuse the variability of a stimulus array or entropy as a dis-criminative stimulus (Young amp Wasserman 2001) Whatis the relationship between perception of variability onthe one hand and emission of variable responses on theother Do individuals attempt to match their own levelsof variation to those of others in their environments whichwould involve both perception and emission How mightdiscrimination of onersquos own level of variability influenceonersquos ability to modify such levels and do influences thatobscure discriminations therefore affect ability to change(see eg van der Meer van der Weel amp Lee 1995)Hunziker et al (2002) report that although variabilitywas successfully reinforced their human subjects couldnot correctly describe the contingencies of reinforce-ment a finding similar to those of informal observationsin other studies (eg Neuringer Deiss amp Imig 2000)but whether subjects could discriminate variability levelsrather than contingencies was not studied

Attention and awareness When attention is subject tointerference or is withdrawn from the task at hand abilityto vary is degraded (eg Baddeley Emslie Kolodny amp

OPERANT VARIABILITY 701

Duncan 1998 Evans amp Graham 1980 Neuringer amp Voss2002) It is unclear at present how attention influencesthe different processes underlying operant variabilityand in particular memorial and stochastic processesRich areas for additional study are the relationships be-tween attention awareness and consciousness on theone hand and operant variability on the other Voluntaryaction and consciousness are often related (see egLibet 1999)

Concepts Concepts are thought by many to be sets ofinstances (much like operant sets) whose structures aregraded meaning that there are more or less typical in-stances of such concepts as birds (robin is typical ostrichis not) Barsalou (1987) showed that the gradation ofthese cognitive structures is manipulable or as he putsit not stable A number of variables influence structureincluding context individual differences and points ofview Barsalou argued that these structures are createdas the need arisesmdashthat is on line by the thinking indi-vidual The structure of concepts may parallel that of op-erants and in both cases levels of within-structures di-versity or variability may be modified by consequencesOperant variability may be found in the cognitive as wellas in the behavioral domain

Play and humor Highly variable behaviors are ob-served when animals and people especially children en-gage in play and humor (see eg Fagen 1981) The re-sult of play may be acquisition of new responses (seeSutton-Smith 1975) Although not currently needed re-sponses learned during play may provide the source ofvariations for successful shaping of behaviors at anothertime This is quite analogous to the role of conserved ge-netic variants which are passed along to progeny but donot currently serve any identifiable role Play may alsoinvolve learning how to vary operantly

Reinforcement by variability The self-maintainingnature of play suggests that engaging in variable behav-iors may itself be reinforcing Indirect support comesfrom findings that varied stimulation can serve as a re-inforcer and that stimulus repetition becomes aversive(Fiske amp Maddi 1961) Possibly also related are findingsthat varying reinforcers and permitting choice among re-inforcers increase their value (Bowman Piazza FisherHagopian amp Kogan 1997) and that permitting choicesamong varied responses is also preferred to demandingfixed responses (Catania 1980)

Foraging Two issues regarding variability are presentin the foraging literature One has to do with risk prefer-ence versus avoidance (see eg Shettleworth 1998) Asecond has to do with search strategies Most reportedcases of foraging involve systematic search patterns(Stephens amp Krebs 1986) However search patterns aresometimes stochastic or highly variable (see eg Peter-son 1997) One question is whether some species can varybetween these two strategies and the challenge is to iden-tify the pressures leading to and the advantages of each

Animal training Wilkes (1996) has applied operantvariability techniques in developing training proceduresfor pet dogs More generally reinforcement of variabil-

ity may enable the gaining of control over behaviors ofanimals and as has been suggested above may be an im-portant part of the process of shaping

Models of learning Reinforced variations in variabil-ity may be important when human and animal learningis being modeled whether the models are mathematicalin form computer based or machine based (Blumberget al 2002 Brooks 1999 Sutton amp Barto 1998)

CONCLUDING COMMENT

Genes interact with environments to produce diverseanatomical physiological and behavioral phenotypesGenes and environments also provide for another level ofdiversitymdashnamely operant variability There are manysources of operant variability including random environ-mental events memory for past responses and stochasticgeneration each of which ultimately derives from ge-netic and environmental influences To the extent thatoperant variability is based on a stochastic source how-ever the behavioral results cannot be predicted fromknowledge of these influences That is organisms haveevolved to behave unpredictably The positing of sto-chasticity to explain behavior has a history extending atleast as far back as Epicurus with major contributors in-cluding Gustav Fechner (see Heidelberger 1987) EgonBrunswik (see Gigerenzer 1987) Skinner (1959) andSuppes (1984) Knowledge of the stochastic nature ofthe operant and of how it is controlled leads to a worldview that differs from genetic and environmental deter-minism with roots in the works of the scholars just men-tioned At the scientific level this view leads to the ex-perimental analysis of consequence-influenced variabilityAt the philosophical level it leads to explanations of vol-untary action At the personal level it can lead to change

REFERENCES

Amabile T M (1983) The social psychology of creativity New YorkSpringer-Verlag

Antonitis J J (1951) Response variability in the white rat duringconditioning extinction and reconditioning Journal of Experimen-tal Psychology 42 273-281

Archer J amp Birke L (1983) Exploration in animals and humansCambridge Van Nostrand Reinhold

Arnesen E M (2000) Reinforcement of object manipulation in-creases discovery Unpublished undergraduate thesis Reed College

Baddeley A D (1966) The capacity for generating information byrandomization Quarterly Journal of Experimental Psychology 18119-129

Baddeley A [D] Emslie H Kolodny J amp Duncan J (1998)Random generation and the executive control of working memoryQuarterly Journal of Experimental Psychology 51A 819-852

Baer D M amp Sherman J A (1964) Reinforcement control of gen-eralized imitation in young children Journal of Experimental ChildPsychology 1 37-49

Baker M J amp Koegel R L (1999) Transforming children withautismrsquos ritualistic behaviors into positive play interactions with sib-lings Paper presented at the meeting of the California Association ofBehavior Analysis San Francisco

Balsam P D Deich J D Ohyama T amp Stokes P D (1998) Ori-gins of new behavior In W OrsquoDonohue (Ed) Learning and behav-ior therapy (pp 403-420) Boston Allyn amp Bacon

Balsam P D Paterniti A Zechowy K amp Stokes P D (2002)

OPERANT VARIABILITY 683

number of occurrences of each of the 16 possible se-quences LLLL LLLR LLRL and so on If the relativefrequency of the current sequencemdashthe number of its oc-currences divided by the sum of all 16 sequencesmdashwasbelow a specified threshold value in this case 05 the ratwas rewarded Recently emitted sequences contributedmore to the maintained tally than did those in the distantpast because after each reinforcement all 16 counterswere multiplied by a weighting coefficient in this case95 Therefore the influence of a particular trial decreasedexponentially with successive reinforcements Thresholdprocedures differ from Bloughrsquos (1966) least-frequentprocedure in that more than one sequence (or response)can be reinforced at any given timemdashnamely all those se-quences that have relative frequencies below the thresholdvalue The procedure differs from lag in permitting agiven sequence to be reinforced many times in successionif the associated relative frequency is sufficiently low

The frequency dependence procedure (Machado 19921993) is a variant of threshold in which the probability ofreinforcement is a continuous function of relative re-sponse frequency The more frequent a response is thelower the probability is that it will be reinforced

Statistical feedback procedures provide subjects withcomparisons between their performances and that of amodel Neuringer (1986) showed human subjects feedbackfrom 10 statistical tests thereby enabling them to comparetheir own distributions of responses to that of a randommodel Neuringer and Voss (1993) provided feedbackbased on a chaotic model The goal in each of these caseswas for the subjects to learn to approximate the model andthe results of these experiments will be discussed below

Differential reinforcement of switches versus repetitionsbalance a subjectrsquos tendency to engage in each of these pat-terns Rather than reinforcement for infrequent responsesor random-like distributions reinforcement is providedsometimes for repeating a response and other times forswitching from one response to another In the two-operandum case a switch is L followed by R or R by La repetition is L then L or R then R Bryant and Church(1974) found that stochastic-like responding was gener-ated by rats when switches were reinforced with a 75probability and repetitions with a 25 probability that isthe rats had a natural tendency to repeat at least in theenvironment of this study Machado (1997) reinforcedpigeons for eight pecks across two keys if the sequenceof pecks contained at least one (one group of birds) ortwo (another group) switches Although the birds couldhave satisfied these contingencies by repeating a singlesequencemdashin the ldquoone switchrdquo condition for examplerepetition of LRRRRRR would produce reinforcement on100 of the trialsmdashthey tended to vary their sequencesBarba and Hunziker (2002) showed however that vari-ability was higher when it was directly reinforced thanwhen reinforcement was based on proportions of switchesa finding consistent with that of Machado (1997)

The evidence from each of these methods supports thehypothesis that variability can be reinforced

Measures Variability must be interpreted with refer-ence to a particular measure The reason is that there canbe high variability (and low predictability) at one level ofanalysis but order (and certainty of prediction) at a dif-ferent level For example the sequence 1 1 2 2 3 3 44 5 5 1 1 2 2 has the same variance (a commonlyemployed measure of variability) as 2 3 2 1 5 1 4 41 1 3 2 5 2 but an analysis of conditional proba-bilities in which the unit contains pairs of instances in-dicates that the two sequences have different levels ofvariability In general high variability observed with onemeasure does not rule out the possibility of order ob-served with another and vice versa That is why evalua-tions of random-number generators often employ manydifferent measures and indeed there are hundreds of dif-ferent measures of the randomness of a sequence (Knuth1969) All behavioral studies make simplifying assump-tions and rely on one measure or at the most a few

U-value has been the most commonly employed mea-sure of operant variability (Evans amp Graham 1980Machado 1989 Page amp Neuringer 1985 Stokes 1995)U-value measures the distribution of relative frequen-cies or probabilities of a set of responses For a set of16 possible responses i = 1ndash16 U-value is given by

where RFi = the relative frequency (or probability) ofeach of the 16 responses U-values approach 10 whenrelative frequencies approach equality as would be ex-pected over the long run from a random generator andthey approach 00 when a single instance is repeated

Other measures include the percentage of trials or fre-quency at which variability contingencies are met (Page ampNeuringer 1985) percentages of alternations versus stays(Machado 1992) conditional probabilities of responses(Machado 1992) the number or frequency of novel re-sponses (Goetz amp Baer 1973 Schwartz 1982) the num-ber or frequency of different responses (Machado 1997Schwartz 1982) Markov analyses (Machado 1992) andstatistical tests of distributions of responses (Neuringer1986) All of the measures employed to date indicate thatvariability can be reinforced

Responses and species Generality has further beendemonstrated by the use of many different responses andspecies These include Siamese fighting fish (Betta splen-dens) swimming in quadrants of a tank (Roots 2000)young chicks pecking keys (Neuringer 2002) pigeonspecking keys (Page amp Neuringer 1985) budgerigarssinging songs (Manabe Staddon amp Cleaveland 1997)rats pressing levers (Van Hest van Haaren amp van de Poll1989) porpoises swimming and jumping (Pryor et al1969) rhesus monkeys pressing levers (Sydney Reisbickpersonal communication) children drawing and build-ing with blocks (Holman Goetz amp Baer 1977) andhuman adults and children pressing buttons (Saldana ampNeuringer 1998 Stokes amp Balsam 2001) There havebeen few attempts to compare species or responses but

- ( )[ ] ( )=aring RF RFi ii

log log 2 21

1616

684 NEURINGER

in one study lowest baseline variability was observedwhen rats pulled trapezes intermediate levels were ob-served when they pushed keys and highest levels wereobserved when they pressed levers (Morgan amp Neuringer1990) with variability increasing in all cases when it wasreinforced In another study operant variability wascompared across rats and people with similar effects ob-served in the two species (Neuringer Deiss amp Imig 2000)

ParametersWe turn now to some parameters studied to date Many

other parameters as yet unstudied are no doubt also im-portantmdashfor example motivation and types of reinforcers

Age Older people (see eg Van der Linden Beertenamp Pesenti 1998) and older rats (see eg Lamberty ampGower 1990) respond less variably than their youngercounterparts when variable responding is reinforced or re-quested but in most cases it is not clear to what extent thevariability is elicited as opposed to reinforced Neuringerand Huntley (1992) compared reinforced variability in2-month-old (young) and 10-month-old (mature) Long-Evans rats Under Yoke contingencies the young rats re-sponded significantly more variably (two levers four re-sponses per trial) than the mature rats which constitutesan elicited effect When reinforcement was contingenton variability (lag 4) variability increased significantlyin both groups thereby overcoming the initial differencecorrelated with age Thus explicitly reinforcing vari-ability in older animals caused them to respond no dif-ferently than younger animals did a finding potentiallyrelevant to real-world human aging However when Wells(1999) compared 8-month-old (mature) to 24-month-old(old) rats although the mature rats responded more vari-ably than the old rats did under Yoke contingencies thedifference was not statistically significant Additionalstudies are needed especially of whether reinforcingvariable behaviors in older populations can facilitatelearning and performance

Gender Male and female rats are equally sensitive toreinforcement of variability (Neuringer amp Huntley 1992van Hest et al 1989) However there was a tendency(not statistically significant) for males to respond over-all more variably than females did and because thesestudies did not control for differences in size and weightof subjects or for other aspects of motivation additionalstudies are necessary

Stage of training Stokes and Balsam (2001) reporteddifferences in variability as a function of whether variabil-ity was initially reinforced early or later in training (seealso Gottlieb 1992) Human subjects generated sequencesof 10 L and R responses per trial on a computer keyboardunder lag 0 and lag 25 contingencies Under lag 0 everytrial ended with reinforcement under lag 25 the currentsequence had to differ from each of the previous 25 Whenlag 25 was experienced at the very beginning of traininglevels of variability at the end of training (when lag 0 wasoperative) were lower than they were when lag 25 was ex-perienced after 50 trials of initial lag 0 These results were

interpreted to suggest that there is an ldquooptimal period forsetting variability levelsrdquo (Stokes amp Balsam 2001 p 181)but that general conclusion needs further testing and anumber of questions need to be answered One has to dowith the contribution of reinforcement frequency Underlag 0 every sequence leads to reinforcement but rein-forcement is intermittent under lag 25 Another has to dowith whether learning to vary differs from learning to repeatwith respect to optimal periods (see eg Bouton 1994)

Related to stage of training are effects of prior experi-ence Hunziker Lee Ferreira da Silva and Caramori(2002) showed that for human subjects if Var contingen-cies were experienced just prior to Yoke contingencieslevels of response variation during the Yoke phase werehigher than they would have been if the Var contingencieshad not been previously experienced The time delay be-tween Var and Yoke experiences mattered with a long in-terval (3 months) producing only a small effect Saldanaand Neuringer (1998) also found effects of prior Var train-ing on performances in Yoke (increased variability inYoke) but the opposite did not hold that is training underYoke had relatively little effect on Var Hunziker Caramorida Silva and Barba (1998) reported similar effects withrats Thus there may be an asymmetry Reinforcement forvarying has relatively strong influences but reinforce-ment without regard to variations does not adversely affectlearning to vary These different effects are not surprisinggiven that Yoke contingencies are permissivemdashreinforce-ment is given whether or not responses varymdashwhereascontingencies under Var can be quite demanding How-ever the results motivate a number of additional ques-tions One is whether effects of Var training differ fromthose observed when repetitions are reinforced Another iswhether direct reinforcement of variable behaviors canhelp animals or people to overcome the detrimental effectsof prior noncontingent experiences such as those that pro-duce learned helplessness (Seligman 1975)

Environmental constraints Stokes (1995) arguedalso that variability increases when behavior is con-strained The term ldquoconstraintrdquo is ambiguous because allreinforcement contingencies exert a form of constraintIn one study Stokes (1995) compared different strate-gies of shaping leverpresses Rats in one group were re-inforced initially for pressing the lever with only theright paw (right only) and eventually for any press (withright or left paw or with any other part of the body) Theorder of training was reversed for a second group ldquoanyrdquofirst and then right only Behavior sequences were morevariable in the right-only-first group leading Stokes(1995) to conclude that early constraints lead to highvariability (see also Stokes Mechner amp Balsam 1999)That is an important hypothesis with many implicationsbut controls are needed before the results can be attrib-uted to the contingencies For example ldquoright onlyrdquo andldquoanyrdquo differed in terms of reinforcement distributions aswell as in the precision of the reinforcement operationthe ldquoanyrdquo reinforcers were provided immediately and au-tomatically by the programming equipment whereas in

OPERANT VARIABILITY 685

the right-only condition a human observer had to decidewhether a response had occurred and then present the re-inforcer necessarily leading to longer and more variedlatencies and a less consistent contingency Furthermorethere are many cases in which increasing constraints suchas those due to a decrease in the number of responses pertrial (Page amp Neuringer 1985) or the requirement of ex-actly four responses on each of two keys (Schwartz 1980)results in decreased variability Thus the contributionsof response-definition constraints and environmentallimitations must be explored further

SummaryThe generality of operant variability is shown by the

robustness of the phenomenon across procedures mea-sures responses and species As with other operant phe-nomena parameters influence the effects

FUNCTIONS

All operants including variability are functional Op-erant responses produce reinforcing consequences Be-cause other behavioral competencies depend on it oper-ant variability is functional in a broader sense as wellSome of these functions will now be described

Operant ConditioningThe potential of a reinforcer to change behavior is lim-

ited by the range of available variations (Hull Langmanamp Glenn 2001 Skinner 1981) and thus whatever pro-duces those variations contributes to the conditioning ofnew operants Much work has been done to identifysources of variation (Balsam Deich Ohyama amp Stokes1998 Segal 1972 Staddon amp Simmelhag 1971) butNeuringer (1993) was first to show that the same rein-forcers that strengthen individual responses can also pro-duce the necessary variability and that these two functionsof reinforcement may work simultaneously Implied isthat when an operant response is being conditioned re-inforcement works not only to strengthen an individualresponse but at the same time to engender the variabilityfrom which the response emerges (That is variability isnot solely a baseline phenomenon elicited or noise)Concurrent reinforcement of variations and individualtarget sequences will be the topic of this section

Pigeons in one experiment and rats in another were re-warded for varying among 16 possible response sequences(four responses per trial L and R operanda lag contin-gencies Neuringer 1993) Concurrently one of the 16designated as the target sequence was reinforced when-ever it occurred Thus reinforcements were given bothfor sequence variations and for a single selected targeta reinforced-variation-and-selection (RVS) procedureWhen the target was ldquoeasyrdquo (eg LLLL) it emergedrapidly from the baseline variations Given that the tar-get was always reinforced reinforcers were obtained morefrequently when the subjects did not vary and only re-peated the target When the target was ldquodifficultrdquo (eg

LLLR) it was not learned As in the choice experimentdescribed above (Neuringer 1992) the relative valuesor difficulties of target versus variations governed theprobability that the target would be learned

However even difficult targets could be conditionedunder the RVS procedure as was shown by another ex-periment in the same series (Neuringer 1993) Sequencevariations were again reinforced concurrently with a dif-ficult target RRLR but now the frequency of reinforce-ments for varying was systematically decreased Satis-fying the variability contingency was intermittentlyreinforced according to a VI schedule and over sessionsreinforcement for varying became less and less frequentThis technique resulted in acquisition of even the mostdifficult target sequence whereas a control group rein-forced for the target alone (without the concurrent rein-forcement for variations) never learned

Reinforcers were obtained more frequently in the RVScondition than in the control condition (since the RVSsubjects were rewarded both for varying and for emittingthe target sequence) and the additional reinforcers mighthave generated greater motivation to respond (Killeen1981) Thus motivation not the direct reinforcement ofvariability might have been responsible for the RVS ef-fect To test this possibility Neuringer Deiss and Olson(2000) reinforced three groups of rats for emitting a par-ticularly difficult target sequence LLRLR The experi-mental group was reinforced concurrently for sequencevariability on the average of once per minute (VI 1 min)The target-only control group received no reinforce-ments other than for the LLRLR target A third groupwas yoked to the experimental animalsmdashreinforced onceper minute for any sequence variability not requiredmdashand as for the other two groups whenever the target oc-curred If the additional reinforcement was responsiblefor the facilitated learning under RVS then because theexperimental and yoked groups received equal reinforce-ments they should have learned equally well but thatwas not the result Only the experimental group learnedto emit the LLRLR target The yoked controls respondedat high rates throughout the experiment but they did notvary sufficiently to make contact with the target contin-gency and never learned The target-only control groupreceived insufficient reinforcement and their responsesextinguished Therefore the added reinforcers helped tomaintain motivation as suggested above but reinforce-ment of variations was necessary for learning of a diffi-cult sequence The RVS procedure maintains both highmotivation and high variation and the two together re-sult in the acquisition of difficult-to-learn responses

Grunow and Neuringer (2002 Experiment 2) studiedone possibly important parameter of the RVS proceduremdashnamely the level of variability When a difficult target re-sponse is being trained does it matter whether reinforcedvariability is low intermediate or high As describedabove each of four groups of rats was reinforced for vari-able response sequences the first group for high vari-ability the fourth group for low variability and the other

686 NEURINGER

two for intermediate levels In other ways the procedurewas similar to that of the Neuringer Deiss and Olson(2000) RVS contingencies just described with a singledifficult target sequence always reinforced The resultswere that the higher the level of reinforced baseline vari-ability the more quickly the target was learned Thusvariability levels indeed mattered and if the just-re-ported effect is found to be general it will be most help-ful to reinforce high variability when difficult-to-learnnew responses are being shaped Other parameters re-quiring study are class definitionsmdashwhich responses areincluded in the class of acceptable variationsmdashand num-ber of instances within the class

One additional effect should be noted When varia-tions among a set of responses are reinforced if one vari-ant is selected for extinction (no longer reinforced) thefrequency of that instance decreases Neuringer (1993)showed this for pigeons responding under a lag-variabilityschedule When one of the 16 sequences was no longerreinforced (with no other change to indicate its status) thefrequency of that sequence decreased Thus reinforce-ment of variations provides the baseline to select for (iestrengthen) and select against (ie weaken) individualinstances

These findings have potentially important implicationsFirst as has been indicated the same reinforcers thatstrengthen particular instances can also reinforce varia-tions That a binary eventmdashreinforce or not reinforcemdashsimultaneously affects variations and selected instancesindicates both the power of reinforcement and a concep-tual problem How reinforcers exert such diverse effectsis a question to which we will return

Second shaping procedures may implicitly reinforcebaseline variations An approximation to a goal responseis reinforced for a period of time but eventually rein-forcement is withheld until a response closer to the goalis obtained It is commonly thought that withholding re-inforcement or mini-extinction induces the necessaryvariations for shaping to proceed as has been indicatedabove An alternative suggested by the present studiesis that organisms learn both to vary (within a set) and to re-peat (selected instances) until one instance is always rein-forced That is the process of shaping might involve re-inforcement both of variations (generally done implicitly)and of targeted instances If so a useful contributor tothe shaping process might be explicit reinforcement ofvariations

Third reinforcement of variations could facilitate ac-quisition of other competencies such as cognitive andmotor skills Siegler (1996) has shown in an important se-ries of studies that childrenrsquos acquisition of complex cog-nitive skills is correlated with high variability Anotherexample is that when motor skills are practiced in a vari-able manner high quality of performance results (Ma-noel amp Connolly 1995 1997 Mechner 1992 Schmidtamp Lee 1999) However it is rare that variability in thesedomains is directly reinforced as part of the training oreducational process

Fourth the RVS procedure might be used in therapeu-tic settings to modify maladaptive and self-injurioushabits Therapists sometimes employ noncontingent re-inforcement procedures to weaken undesirable behaviors(Vollmer Iwata Zarcone Smith amp Mazaleski 1993)The reasoning is that many maladaptive behaviors aremaintained by unintended consequences and that non-contingent rewards may compete with those conse-quences thereby lowering the strength of the unwantedhabit For example when a staff personrsquos attention toself-injurious behavior helps to maintain it providing at-tention noncontingently may decrease the strength of theunwanted target behavior However the RVS proceduremight accomplish the same goal more effectively andwithout undesirable side effects (Vollmer RingdahlRoane amp Marcus 1997) Noncontingent rewards some-times result in unwanted ldquosuperstitiousrdquo behaviors andthat would be avoided under RVS Furthermore RVSwould help to provide the set of baseline behaviors nec-essary to establish desirable new responses

Finally RVS procedures may help to gain control overand eventually to weaken other undesirable target behav-iors The target can be incorporated within a set of moredesirable responsesmdasha new operant class established withthe target as one membermdashand variable instances rein-forced Once variable responding is successfully main-tained including the undesirable target response the lattercan be weakened by selectively withholding reinforcement

Summary The conditioning of operant responses de-pends on both baseline variations and selection of suc-cessive approximations to a goal response The same re-inforcers used to select the approximations can reinforcebaseline variations thereby defining the precise set ofappropriate variants and the desired levels of variationSuch concurrent reinforcement of variations and selec-tions may facilitate the acquisition of difficult-to-learnoperants and selective withholding of reinforcementmay weaken undesirable instances

Creativity and Problem SolvingBehaving in a novel unusual or nonrepeating manner is

part of creative activities (Campbell 1960) Novelty andvariation may arise through combinations of previouslylearned behaviors (Epstein 1996) transformations of thesebehaviors (Boden 1994) direct reinforcement of variabil-ity as described above or some combination thereof Ofcourse neither novelty nor variability alone suffices forcreative outputmdashthe throw of the dice or the toss of acoin are not creative nor are haphazard strokes of a paint-brush on a canvasmdashbut without variation creativity islow Whether the variability required for creativity canbe reinforced is relevant to whether creativity itself canbe reinforced The latter question has led to contentiousdebate with some arguing that reinforcement facilitatescreativity (Cameron amp Pierce 1994 Eisenberger ampCameron 1996 Stokes 2001 Winston amp Baker 1985)and others focusing on detrimental effects of reinforce-ment (Amabile 1983 Deci Koestner amp Ryan 1999)

OPERANT VARIABILITY 687

The debate goes well beyond the present discussion butsome of the research described above is relevant Firstreinforcement contingent on variability generates higherlevels of behavioral variability than does reinforcementcontingent simply on responding as has been shown bythe many studies described above in which Var contin-gencies were compared to Yoke contingencies Similarlyprecise reinforcement contingencies matter for creativity(Cameron amp Pierce 1994 Lepper amp Henderlong 2000)If any product rather than only creative ones is rein-forced creativity tends to fall (just as variability decreasesunder yoke conditions) Second approach to or antici-pation of reinforcement interferes with operant variabil-ity (Cherot et al 1996 McElroy amp Neuringer 1990Neuringer 1991) which again might be related to effectson creativity when an individual focuses on the rein-forcer Note however that although variability decreaseswith proximity to the reinforcer overall levels of vari-ability are much higher when variability is reinforced thanwhen it is not The same may be true for creativity aswell anticipation of reward may interfere with creativeoutput to some extent but at the same time high creativ-ity may depend on its reinforcement

Operant variability may be important for problemsolving as well Here too there is disagreement as towhether training is helpful or not Early studies by Maier(1933) indicated that it is and these found support inlater work (eg Lung amp Dominowski 1985) Somestudies showed that those individuals most likely toldquobreak the setrdquo are also most likely to solve a problem(Dover amp Shore 1991) However a number of studiesfound that practice had no influence (Duncan 1961)

Arnesen (2000) explored a related phenomenon witha rat model One group Var was reinforced for variableinteractions with novel objects and another group Yokewas reinforced independently of such interactions EachVar rat received training with 12 different objects oneper session For example in the presence of a soup canthe rat was reinforced the first time it entered the canbut thereafter a different response was required such asrolling it climbing on it and so forth This procedure issimilar to that of Goetz and Baerrsquos (1973) described aboveexcept that objects changed from session to session Ratsin the Yoke control group had identical experiences in anadjoining identical chamber that contained identical ob-jects but now rewards were not contingent on object ma-nipulation After this training the rats were tested in adifferent room with 30 never-before-experienced ob-jects scattered on the floor Hidden in each was a smallpiece of food When compared to their Yoke controls theVar animals were more active during this test exploredmore manipulated objects more and most importantlydiscovered significantly more food pellets Another con-trol group received no preliminary training with objectsand their performance in the test session did not differfrom that of the yoked controls Therefore when ratswere reinforced for variable interactions with objectsthe result was an increased tendency to explore manip-ulate and discover new sources of reinforcement It

would be of considerable interest to determine whethera similar effect occurs with children

Assessment and Treatment of PsychopathologiesAbnormal levels of behavioral variability characterize

some human psychopathologies as has been shown byclinical observations as well as by research with humansubjects and animal models Few studies have attemptedto reinforce variability directly in such problem popula-tions The main question is whether reinforcement canmodify abnormal levels of variability in the direction ofnormalcy and whether such changes might have associ-ated beneficial effects

Autism Stereotyped responding characterizes indi-viduals with autism (Hertzig amp Shapiro 1990) For ex-ample Baron-Cohen (1992) asked children with autismand others in a control group to hide a penny in one handso that the experimenter could not guess which hand heldthe penny Those with autism were more likely thanthose without to generate a simple predictable patternsuch as repeatedly switching back and forth from left toright hands Such repetitive responding can be maladap-tive For example Mullins and Rincover (1985) askedchildren with and without autism to pick one of five cardsFood was occasionally found in a cup behind the cardEach of the five cards was a discriminative stimulus fora different schedule of reinforcement including contin-uous reinforcement (CRF every choice reinforced) FR 2(fixed ratio 2 in which every second choice was rein-forced) FR 4 (every fourth choice reinforced) FR 7 andFR 11 Control participants sampled all five alternativesand quickly learned to choose the most frequently rein-forced CRF card Autistic children sampled only a lim-ited number of cards and therefore often preferred anonoptimal alternative Thus children with autism tendto respond less variably than normal children and thiscan result in failure to adapt

Miller and Neuringer (2000) asked whether direct re-inforcement could influence the variability of individu-als with autism Sequences of responses on two largebuttons were first reinforced probabilistically and inde-pendently of variability (Prob phase) after which rein-forcement was contingent upon satisfying a thresholdvariability contingency (Var phase) The autistic partic-ipantsrsquo variability was significantly lower than that ofmatched controls during both Prob and Var phases butwhen variability was reinforced its levels increased sig-nificantly in both groups This is an important findingbecause it suggests that direct reinforcement of variabil-ity may help to modify some of the behavioral stereo-types that are characteristic of autism Lee McComasand Jawor (in press) offer support for this hypothesisThree individuals with autism were reinforced for variedverbal responses to social questions the responses had tobe appropriate given the context Variability increased intwo of the three individuals In a related procedureBaker and Koegel (1999) reinforced children with autismfor varied social interaction in a game-playing contextAlthough the reinforcement was permission to engage in

688 NEURINGER

the stereotyped behaviors preferred by the child the re-sult was increased social interactions both during theperiod of the experimental manipulation and in 1- and2-month follow-ups These three studies taken togethersuggest that reinforcement of varied behaviors may facil-itate modification of nonfunctional ritualistic and stereo-typed behaviors

Depression Depression provides another example ofmaladaptive consequences of low variability Lapp Mar-inier and Pihl (1982) observed that depressed womenproduced fewer alternative solutions to hypotheticalproblems than did nondepressed women Channon andBaker (1996) found that attempts by moderately de-pressed college students to identify faults in a series ofinterconnected circuits were less varied and less success-ful that were those of nondepressed controls HorneEvans and Orne (1982) found that when depressed pa-tients were asked to generate a random sequence of num-bers their responses were less variable than were thoseof controls These experimental results are consistentwith clinical observations

Can depressed individuals be rewarded for increasingvariation Hopkinson and Neuringer (in press) dividedcollege students into mildly depressed and nondepressedgroups on the basis of the CES-D scale a paper-and-pencil self-evaluative index of depression In the first oftwo phases sequences of responses on a computer key-board were reinforced independently of variability (Prob)following which only highly variable sequences were rein-forced (Var) Under Prob depressed students respondedsignificantly less variably than did the controls in agree-ment with previous findings Under Var variability in-creased significantly in both groups Indeed by the endof the experiment depressed and nondepressed subjectswere responding equally variably (Instructions were alsoeffective in increasing variability in the depressed par-ticipants) Thus even in a simple computer-game envi-ronment mildly depressed subjects manifested lowervariability than controls did but direct reinforcementmodified their behavior to control levels Can reinforce-ment of variability be used in real-world conditions tohelp depressed individuals An affirmative answer comesfrom cognitive behavior modification procedures thatwork to increase alternative solutions to problems (Beck1976) A more general conclusion from these studies ofautistic and mildly depressed individuals is that direct-reinforcement-of-variability procedures may benefitthose who manifest abnormally low levels of variability

Attention Deficit Hyperactivity Disorder (ADHD)ADHD may provide an example of the opposite stateabnormally high levels of variability (Barkley 1990)Wultz Sagvolden Moser and Moser (1990) proposedthat the spontaneously hypertensive rat (SHR) may serveas a good animal model of ADHD the SHR shows manyof the behavioral characteristics of that disorder includ-ing hyperactivity poor learning ability poor ability totolerate delayed reinforcement and poor self-control

The next study to be described asked about operant vari-ability in SHRs

Mook Jeffrey and Neuringer (1993) compared vari-ability of SHRs and of a control strain Wystar Kyoto(WKY) under two conditions Prob and Var The mainresult was that SHRs tended to respond variably whetheror not the contingencies required it consistently with theresults from the human ADHD literature When vari-ability was reinforced the two strains responded at ap-proximately equally high levels of variation These re-sults were confirmed in two other studies (HunzikerSaldana amp Neuringer 1996 Mook amp Neuringer 1994)Thus control rats varied only when the contingencies re-quired variations whereas SHRs varied whether or notthe contingencies required it As might be expected fromthese results when reinforcement depended on repeti-tions the SHRs performed poorly (Mook et al 1993)

Another similarity between SHRs and ADHD humanswas seen when amphetamine a drug that has effects simi-lar to those of Ritalin often prescribed for ADHD was ad-ministered (Mook amp Neuringer 1994) When repetitionswere required for reinforcement amphetamine improvedperformances by the SHRs so that they were as accurate assaline-injected WKYs a finding again consistent withthose in the ADHD literature These studies support SHRsas a model of ADHD and suggest that additional attemptsshould be made to assess whether or not individuals withADHD are less able to change levels of variability thanare controls Although as indicated above there is someevidence in support of this hypothesis (Barkley 1990)when Saldana and Neuringer (1998) directly comparedhuman ADHD and control participantsrsquo operant variabil-ity within the context of a computer game no differenceswere observed Most of those with ADHD were main-tained on Ritalin and other drugs however a fact unfor-tunately not documented in the original article

Drug EffectsAlthough many drugs are known to influence behavioral

variability (Brugger 1997) rarely have attempts beenmade to analyze explain or identify the sources of theeffects As will be discussed below operant variabilitycan result from the memory-based process of remember-ing previous responses and then behaving differently orcan result from responding stochastically a process thatappears to be quite different McElroy and Neuringer(1990) attempted to distinguish the effects of alcohol onmemory from its effects on stochastic processes in astudy in which one group of rats was reinforced for vary-ing four-response sequences under a lag 5 contingency(Var) and another was reinforced for repeating a singlesequence LLRR (Rep) Ethanol injections of 075 and20 gkg affected Var and Rep performances differentlyWhereas the Rep animals were much less able to satisfythe LLRR contingency at both doses the Var rats re-sponded variably whether or not alcohol had been ad-ministered Similar results were obtained by Cohen et al

OPERANT VARIABILITY 689

(1990) with a multiple schedule in which each rat servedas its own control In the same animals and within thesame sessions alcohol degraded Rep performances butleft Var intact Thus alcohol appears to interfere with atask that requires working memory as presumably is thecase for the Rep task but not with one that reinforcesvariability The memory-interfering effect of alcoholmight account for why alcohol also degrades radial-armmaze performance (Devenport 1983)

Alcohol also affects choices to vary or to repeat For ex-ample Gorka (1993) reinforced rats concurrently for re-peating and varying and monitored percentage of choicesas a function of time following injection of ethanol Inthe control sessions when the rats were injected withsaline the probabilities of varying and repeating remainedapproximately constant across the 45-min session witha slight bias in favor of varying After injection with 15gkg of ethanol however there was a significant increasein choices to vary at the beginning of the session andthen a later decrease to below 50 That is alcohol ledinitially to a markedly increased preference for varyingbut later to a preference for repeating (Figure 5) This re-search should be replicated and extended especially be-cause the contingencies ldquopressuredrdquo the animals to re-main at approximately 50 preference but thesuggestion is that alcoholrsquos effects on response variabil-ity may depend in part on the duration of its influence Amore general implication is that drugs may affect re-peated and variable operants differently and that effectsdepend on time following administration

SummaryReinforced variability may contribute to many types

of learning including operant shaping and acquisition ofcognitive and complex motor skills Similarly creativityand problem solving depend on variable responding andin these areas as well explicit reinforcement of variationsmay be functional Abnormal levels of variability aresometimes caused by psychopathologies and drugs andoperant variability may be useful in these areas in twoways first as a way of assessing the variability that is pos-sible and second to modify variability toward more nor-mal levels Studies show that individuals with autism andmild depression vary less than do controls but that ex-plicit reinforcement of variability moves the levels towardsnormalcy Other studies show that drugs such as alcoholmay affect repeated and variable operants differently

EXPLANATIONS

Two questions must be answered in order to explainoperant variability First what are the sources of vari-ability that permit its reinforcement Second how is itmaintained Related questions including how instancesbecome members of a set from which variations emergeand whether operant variability is a generalizable com-petency will also be considered

Origination From Nonoperant SourcesSkinner (1966) reasoned that operant responses orig-

inate from nonoperant sources with reinforcement con-tacting a behavior shaping and strengthening it onlyafter the behavior had occurred for reasons other than re-inforcement To put it simply a behavior must occur be-fore it can be reinforced (Balsam amp Silver 1994) I willconsider three initial sources of variable behavior en-dogenous variability noncontingent environmental eventsand extinction These appear to parallel evolutionaryprocesses and they too will be identified

Endogenous variability All physical phenomenavary and behavior is no exception This fact can providethe opportunity for reinforcement to select particularlevels of variation Examples of spontaneous variabilityinclude spontaneous alternations in mazes (Dember ampRichman 1989) nondifferentially reinforced variationsin operant responses (Guthrie amp Horton 1946) and ex-ploration (Archer amp Birke 1983) Endogenous muta-tions provide a parallel in genetics Base-pair changes inDNA molecules occur spontaneously intron-bearinggenes mix and match segments to encode new proteinsjumping genes introduce new DNA sequences and atthe level of the chromosome random crossings and as-sortment occur as a natural part of reproductive meiosisThese sources of random genetic change occur indepen-dently of external influences

Figure 5 The relative frequency of variations by rats as a func-tion of 5-min blocks of time since the beginning of the sessionThe open circles represent performance after the rats were in-jected with saline indicating a slight tendency to vary more thanrepeat but one that is approximately constant throughout thesession The filled circles represent performance after injectionwith 15 gkg ethanol indicating a much higher likelihood ofvarying soon after the alcohol is injected followed by a decreasein variations to below 50 and then a return to baseline at theend of the session Group averages are shown Data are fromGorka (1993)

690 NEURINGER

Noncontingent events Noncontingent events some-times referred to as chance accidents or luck also affectbehavior One happens upon a particular passage in a bookthat leads to a new thought one happens to sit next to aninteresting woman or man on an airplane which leads tomarriage (Bandura 1982) In science as well as in every-day life accidents or serendipity are an important part ofthe process of discovery (Beveridge 1957) In each of thecases presented above an accidental event leads to new be-haviors that are then strengthened by consequences Thesame is potentially true for variability itself Particular lev-els of variability might occur for adventitious reasons butthen become functionally related to consequences

That organisms are particularly sensitive to such unan-ticipated noncontingent events is shown by two basicprinciples of learning habituation and Pavlovian condi-tioning Organisms attend preferentially to unexpectedevents and stop attending or habituate to repeated onesThe conditioning of new Pavlovian responses dependsaccording to influential theories upon unexpected rein-forcing events (eg Rescorla amp Wagner 1972)

Noncontingent events such as X-rays and other pro-cesses that can act as mutagens play an important role ingenetic variation as well The particular genes affectedand the type of changes are random there is no neces-sary relationship or contingency between the externalevent and the affected gene However sensitivity to mu-tations is not random but has been selected presumablybecause it is functional Thus in both behavior and ge-netics sensitivity to noncontingent events is conserved

Extinction Extinction increases behavioral variabilityat least to some extent as was described above Distance inspace or time from the reinforcer does the same (Cherotet al 1996) A possible parallel at the genetic level in-volves high levels of deprivation at least in simple lifeforms For example when E coli bacteria are deprivedof nutrients unusually high mutation rates are observedThe exact mechanism by which such hypermutations areinduced is not known but they increase the likelihoodthat a population can survive such environmental insults(Moxon 1997) That is deprivation-induced variability isfunctional at the genetic level (The behavioral evidenceregarding long-term deprivation is less clear some stud-ies showing increased variability and others increasedstereotypy eg Carlton 1962 Senkowski Vogel amp Po-zulp 1978)

Endogenous variations external noncontingent eventsand withholding of reinforcement are three sources forthe variability that precedes control by reinforcementBut once control by reinforcement is established a dif-ferent question emerges What behavioral processes areinvolved when variability is maintained as an operantThe sources just discussed generate uncontrolled non-specific and often short-lived variability and cannot ex-plain the precisely controlled operant type

Operant Variability ProcessesThree processes are hypothesized to underlie operant

variability The first involves the use of random environ-

mental events such as the toss of a coin The second in-volves memory for responses such as when an animallearns not to return to previously visited locations Thethird will be referred to as an endogenous stochasticgenerator

Random environmental events Throughout historymany cultures have used random events to influencechoices and decisions such as throws of dice and randomselections of sticks cards bones or organs Today coinflips are used by a referee at the beginning of a game to de-cide which team kicks the ball a computerrsquos random-number generator is employed by scientists to avoid bi-ases in assigning subjects to experimental groups andaleatoric events are sometimes employed in modern artmusic and literature The Dice Man by Rhinehart (1998)provides a fictional example in which the protagonistbored with life writes a number of possible actions onslips of paper and then periodically selects one blindlyand behaves accordingly These examples show that randomexternal events may be used to avoid biases engender un-likely responses and break out of behavioral niches

Memory-based variability Many methods used tostudy operant variability differentially reinforce nonrepe-titions at least over the recent past Therefore one strat-egy that subjects can employ is to cycle through few re-sponses or sequences especially when the variabilitycontingencies are relatively permissive For exampleunder a lag 2 contingency cycling repeatedly throughthree sequences will result in 100 reinforcement Ani-mals and people sometimes do this apparently basingtheir current response on memory for or discriminativecontrol by the just emitted responses One such instancewas shown by Schoenfeld et al (1966) who rewardedrats for IRTs that fell into one of 10 interval classes onlyif the current IRT differed from the interval class of theprevious response The rats came to alternate betweenlong and short IRTs thus behaving systematically to meetthis lenient variability contingency Machado (1993)studied an analogous effect having pigeons peck L and Rkeys under a frequency-dependent variability contin-gency related to Shimprsquos (1967) procedure describedabove When the trial unit consisted of two responses thebirds developed memory-based repetitive sequencessuch as RRLLRRLL and thereby satisfied the variabilitycontingency When the trial unit was three responses inlength the birds did not develop the optimal fixed pat-tern of RRRLRLLLmdashwhich was too complexmdashbut in-stead developed ldquorandom-like behaviorrdquo (Machado1993 p 103) Thus a memory-based strategy was em-ployed when it was possible but when the memory loadbecame too high stochastic responding emerged A sim-ilar pattern was seen when variability of budgerigarsongs was reinforced (Manabe et al 1997) Under lag 1contingencies the birds sang mainly two songs underlag 2 they increased to three songs Only when the lagwas increased to 3 did song diversity increase apprecia-bly in one of the birds In radial-arm mazes as well whencontextual cues are available memory again appears toplay an important role leading to much lower probabil-

OPERANT VARIABILITY 691

ities of repetition than would be expected from a sto-chastic generator (McElroy amp Neuringer 1990)

Note that under lag schedules memory-based strate-gies result in higher reinforcement rates than respondingstochastically does The reason is that a stochastic gen-erator repeats by chance That animals and people oftendo not employ memory-based strategies probably indi-cates the difficulty of remembering long sequences andpossibly other advantages of unpredictable responding

If a memory strategy is the only way to satisfy vari-ability contingencies then it can be used even when thememory requirements are demanding as was shown byNeuringer and Voss (1993) Human subjects were trainedto generate on a computer keyboard a chaotic-like se-quence of responses that matched iterations of the logistic-difference equation

where Rn refers to the nth iteration in a series with R be-tween 00 and 10 and t is a constant between 10 and40 When the t parameter approaches 40 iterations ofthis equation result in very ldquonoisyrdquo sequences As withother chaotic processes however there is an orderlinessunderlying the surface noise In this case if responsesare autocorrelated with Rn drawn as a function of Rn21the resulting function is a parabola In the Neuringer andVoss (1993) study following each response college stu-dents were shown how close their predicted values wereto that from iterations of the logistic-difference model Inone experiment the students made digital predictions inanother they placed a pointer along a horizontal lineand this analogue response was then translated into dig-ital values for computation With training the studentsbecame increasingly adept at responding in chaotic-likefashionmdashthe studentsrsquo predictions matched closely theiterations of the logistic functionmdashand their autocorrela-tions became parabolic The left column in Figure 6 showsperformances by 4 subjects (rows) early in training thatis not well approximated by the logistic-differenceparabola and the right column shows performances at theend of training in which r 2 values were 72 99 98 andapproaching 10 How was this accomplished Sinceeach iteration in the logistic-difference sequence is basedon the prior outputmdashthat is the sequence is generated bya deterministic although nonlinear processmdashit was hy-pothesized that the human subjectsrsquo chaotic-like responseswere also based on memory for prior responses To put itsimply the subjects may have memorized a long series ofpairs in the form ldquoIf the previous response was value Athen the current response must be value Brdquo as Metzger(1994) Ward and West (1994) and Ward (2002) have sug-gested

To test whether subjects in fact relied on memory forprior responses Neuringer and Voss (2002) systemati-cally slowed responding by interposing pausesmdashthat isby requiring an IRT As IRT increased the difference be-tween the subjectsrsquo sequences and the modelrsquos chaotic

Rn n nt R R= -( )- -1 11

Figure 6 Autocorrelations (responses on trial n on the y-axis asa function of n21 on the x-axis) for 4 human subjects at the be-ginning (left column) and end (right column) of training inNeuringer and Voss (1993) Equations for the best-fitting curveand amount of variance accounted for (r2) are shown at the topof each panel To the extent that the circles (responses) approxi-mate the equation y = 40x 2 40x 2 the subjects were respondingin a way that matched the logistic-difference chaotic functionFrom ldquoApproximating Chaotic Behaviorrdquo by A Neuringer andC Voss 1993 Psychological Science 4 p 115 Copyright 1993 bythe American Psychological Society Reprinted with permission

692 NEURINGER

output increased and the underlying parabolic structurewas broken That is when pauses increased subjects wereless and less able to approximate the modelrsquos chaotic dis-tribution (see Figure 7) The important point here is thathighly variable responses which in this case are chaoticin nature can be based on memorial processes

Endogenous stochastic generator The evidence tobe reviewed next indicates that animals and people canrespond stochastically and that this competency is basedneither on chance environmental events nor on memoryfor prior events or responses It is impossible to provethat an endogenous stochastic process underlies operantvariability that would be analogous to proving the nullhypothesis However much evidence is consistent withthe stochastic hypothesis Included are the facts thatmemory limitations do not permit adequate respondingunder many variability contingencies that response dis-tributions match those expected from a stochastic gener-ator that changes in parameter values affect operantvariability as they would a stochastic generator and thatinterference with memory processes leaves stochasti-cally based operant variability unaffected As a wholethe evidence is consistent with an endogenous stochas-tic source of operant variability

Memory limitations One type of evidence is nega-tive in character and is based on the presumed inability ofmemory for prior responses or sequences to satisfy sometypes of variability contingencies Two examples weregiven above (Machado 1993 Manabe et al 1997) andanother was provided by a lag 50 contingency in Pageand Neuringer (1985) In the latter case the sequence in

the current trial (eight responses across L and R keys) wasrequired to differ from those in each of the previous 50 tri-als including at the beginning of a session the last 50trials from the previous session Pigeons responded wellunder these contingencies and in a way similar to that ofa stochastic model It is unlikely that pigeons can remem-ber each of their previous 50 eight-response sequences

Response slowing As was shown above for chaotic-like responding performances are often degraded by de-lays imposed between a stimulus and a response pre-sumably because of limitations in working or short-termmemory (Berryman Cumming amp Nevin 1963 Williams1971) Neuringer (1991) relied on this effect to test whethermemorial processes could explain the generation of vari-able sequences under lag schedules Two groups of ratswere reinforced for L and R leverpresses producing fourresponses per trial One group (Var) was trained under alag 5 contingency and the other (Rep) was trained to asingle sequence of responses LLRR It was assumedthat accurate Rep performance depended on workingmemory After performances had stabilized IRTs weresystematically varied by the imposition of timeouts of05 to 20 sec between consecutive responses in the se-quence During these forced pauses the chamber wasdark and responses were not counted toward completionof the sequence If the animals had used the previous re-sponse(s) as a discriminative cue for the current responsethen performances should have been degraded by theseinterpolated pauses

The results were that for the Rep group as the re-quired pauses increased from 0 to 6 sec percentages ofcorrect sequences were relatively unaffected but at pausesgreater than 6 sec probability of a correct LLRR se-quence fell sharply This pattern of results suggests in-volvement of working memory of prior responses withthe rats able to remember responses within a 6-sec win-dow The Var grouprsquos results were quite different Aspauses increased between 0 and 6 sec the rats were moreand more likely to satisfy the variability contingenciesand with pauses greater than 6 sec percent correct re-mained at a high and constant level Thus the Var grouprsquosresults were opposite to those of the Rep group and op-posite to those predicted from a memory-based strategySimilar results showing that operant variability increasesor is maintained as responding is slowed have been re-ported by Morris (1987) with pigeons and by Baddeley(1966) with random response generation by human sub-jects The results reported above concerning alcoholrsquos ef-fects on variability and repetition were also consistentwith those of Neuringer (1991)

Memory for prior responses may not have controlled theVar grouprsquos performance but why did variability increaseEach of three answers is consistent with a stochastic-generator interpretation Weiss (1964 1965) hypothe-sized that random responding requires that current re-sponses be independent of previous ones memory for orcontrol by prior responses interferes with such indepen-

Figure 7 Approximation to a chaotic sequence (given by delta)as a function of the imposed interresponse times Shown are theaverages of 5 subjects

OPERANT VARIABILITY 693

dence and therefore pauses would be expected to facil-itate variability A second possibility is that at shortIRTs animals tend to repeat responses on the sameoperandum Blough (1966) found this in pigeons and ex-cluded such double pecks from his analyses because theyappeared not to be under the control of reinforcement con-tingencies Morris (1987) also found a tendency for birdsto repeat when no interresponse timeouts were imposedA third hypothesis is that there were two contributors tothe observed variability One was a stochastic processcontrolled by reinforcement and the other elicited vari-ability generated by interposition of pauses The high vari-ability elicited as an effect of slowed responding is ageneral phenomenon supported in many other cases Ac-cording to this interpretation operant variability in the Vargroup was governed by a stochastic-based process operantrepetition in the Rep group was governed by a memory-based process and pauses elicited variability under bothcontingencies Thus Rep performance was interferedwith whereas Var was facilitated Each of these hypothe-ses is consistent with the conclusion that memory for (ordiscriminative control by) prior responses does not con-tribute to and possibly interferes with variable respond-ing when an organism is reinforced for variability andthis in turn is consistent with the stochastic-generatorhypothesis

Stochastic response distributions Other evidence iscorrelational Response distributions are similar to thosefrom a stochastic generator The data from two experi-ments already discussed are shown in Figure 1 In thesecases reinforcement of least frequent IRTs (Blough 1966)and of nonrepetitive choice sequences (Machado 1989)resulted in stochastic-like responding

Neuringer (1986) reported another striking case Highschool and college students generated sequences of 1sand 2s on a computer keyboard with each set of 100 re-sponses constituting one trial A baseline phase lastedfor 60 such trials for a total of 6000 responses duringwhich the students were asked to behave as randomly aspossible as if they were reporting the outcome of a se-ries of coin tosses As in the many previous studies ofthis sort the subjectsrsquo responses differed significantlyfrom those of a stochastic model (Brugger 1997 Wage-naar 1972) During the training phase the subjects re-ceived feedback which enabled them to compare their per-formances with that of a stochastic model first accordingto one statistical test and then according to another untilfollowing each trial feedback was provided from 10 dif-ferent statistics The results were that all of the subjectslearned to approximate the stochastic model accordingto the 10 statistics as well as some (but not all) other sta-tistical tests That is the distributions of the statisticswhich clearly did not match a stochastic model at the be-ginning of training approximated the stochastic modelat the end of it (see Figure 8)

The previous failures of other researchers to obtain ev-idence for random responding had led some to concludethat humans did not have the capacity to behave ran-

domly What would account for the difference betweenthe present positive finding and the many other negativeones Neuringerrsquos (1986) study was the first in which sto-chastic responding was explicitly reinforced in this casewith feedback from 10 statistical tests An analogy mighthelp to explain the difference in procedures and resultsImagine an experimenter walking around a city and ask-ing people if they know what a violin sounds like (analo-gous to asking people whether they know what a series ofcoin tosses or dice throws is like) Most would answeryes If the same individuals were then asked to play theviolin very few if any would succeed The experimentermight conclude after many replications with differentsubjects in different cities that people did not have thecapacity to play the violin The point of course is thatplaying the violin is a skill requiring training and prac-tice and the same may be true for responding randomlyIt is easy to vary but practice and reinforcement may benecessary to approximate a random distribution

Response number Page and Neuringer (1985) testedthe stochastic-generator hypothesis in another way byvarying number of responses per trial with pigeons work-ing under a lag 3 contingency The reasoning was that ifprevious responses served as a discriminative cue forwhat not to domdasha memory-based strategymdashthen perfor-mance should be degraded when the number of re-sponses was increased Eight responses per trial requiresubjects to remember more than four responses The sto-chastic hypothesis predicts the opposite outcome as isdemonstrated by the following example If each trialwere two responses in length with responses determinedby the toss of a coin then the probability that a given trialwould repeat the previous one would be 25 (There arefour possible sequences in the first trial RR RL LRand LL Thus the second trial has a 1-in-4 chance ofmatching the first) If a trial consists of four coin tossesthe probability of a repetition by chance is 0625 or 1 in16 Thus if subjects used a stochastic process to gener-ate Ls and Rs performances should improve with in-creasing responses per trial Results were precisely thosepredicted by the stochastic hypothesis and inconsistentwith a memory strategy Probability of meeting the con-tingencies and therefore probability of reinforcementincreased as responses per trial increased In otherwords eight-response trials were easier for pigeons thanfour-response trials

Response interference Neuringer and Voss (2002)attempted to distinguish between memorial and stochas-tic processes in yet another way They hypothesized thatwhereas chaotic responding was based on memory forprior responses (as was discussed above) responding inNeuringerrsquos (1986) study with feedback from 10 statisti-cal tests indicated involvement of an endogenous sto-chastic generator To show that human subjects could en-gage in both processes at will three college studentswere trained to generate chaotic sequences in one compo-nent of a multiple schedule as in Neuringer and Voss(1993) and to generate stochastic sequences in a sepa-

694 NEURINGER

rate component as in Neuringer (1986) The wordCHAOTICS was on the screen in the first component theword STOCHASTICS was on during the other and thesewords alternated throughout the session After the sub-jects succeeded at each of the component tasks test ses-sions were introduced in which memory for prior re-sponses was subject to interference in both the chaoticsand stochastics components as follows During the

chaotics component four separate chaotic sequenceswere cycled one after the other each cued by differentscreen colors The subjects attempted to approximate achaotic sequence in one color (eg light green) follow-ing which they would have to match a second chaotic se-quence (lavender screen color) and so on the fourchaotic contingencies following one another across thesession Although all four chaotic sequences used the

Figure 8 Performance by one subject at the end of training to respond stochastically in Neuringer(1986) Shown are comparisons on 6 different statistics between the subject (solid lines and Xs) andthe random model (dashed lines and closed circles) At the end of training the subjectrsquos perfor-mance did not differ statistically from that of the random model on these and four additional sta-tistics From ldquoCan people behave lsquorandomlyrsquo The role of feedbackrdquo by A Neuringer 1986 Jour-nal of Experimental Psychology General 115 p 71 Copyright 1986 by the American PsychologicalAssociation Reprinted with permission

OPERANT VARIABILITY 695

same logistic-difference equation that a model did theywere out of phase and independent If chaotic respond-ing is memory based then the four independent se-quences would be expected to interfere with one anotherthat is approximations to the four chaotic models wouldbe lower than when only a single chaotic output was re-quired as in Neuringer and Voss (1993) That was ex-actly the finding Performances by the three subjectswere signif icantly degraded during the four-segmentphase To put it simply chaotic responses interfered withone another

A different pattern of results was seen in the stochasticsportion of the experiment The same interference proce-dure was used with four independent stochastic sequencesrequired in the four different color conditions As inNeuringer (1986) subjects were required to satisfy eightdifferent statistical tests of randomness and to do this in-dependently in each of the four phases (If responses in anytwo or more of the four stochastic phases became corre-lated above a minimal level responses were considered tobe incorrect thereby prohibiting repetition of a single se-quence in two or more of the phases) For one subjectseven of eight statistics were closer to those of a randommodel during this interference phase than they were at theend of the original training with only a single sequenceand for a second subject all eight statistics were closerThe third subject showed closer-to-random approximationunder the four-segment condition on only two of the eightstatistics but most of her changes were also in the directionof a random generator For example if a random generatoremits equal numbers of ldquouprdquo and ldquodownrdquo sequences andthis subject in the baseline phase had emitted only 47ldquouprdquo sequences then during the four-segment tests shecame to emit ldquouprdquo 55 of the time She changed in the di-rection of the random model but overshot the mark Thuswhereas interference degraded chaotic performance sig-nificantly in all three subjects it improved stochastic per-formances in at least two of them These results are con-sistent with the hypothesis that a memory-based processcontrols chaotic variability and that a stochastic processnot dependent on memory controls stochastic variability

Summary of Tripartite TheoryThe following three processes constitute a tripartite

theory of operant variability attention to random events inthe environment (coin toss) memorial processes (chaosradial maze) and stochastic processes (response se-quences and random-number generation with feedback)Random environmental events assure unpredictabilitymdashfor example in making unbiased selections Remember-ing what was done previously will enable one to avoidrepetitions to vary the students called upon in class tovary the locations when looking for a hidden object orto try different approaches when attempting to solve aproblem External aids such as notes to oneself or sci-entific records help to enhance memory capacities

An endogenous stochastic generator provides the thirdand most controversial source of behavioral variability

When an opponent is attempting to predict your behav-ior in a game or at war it may be functional to behavestochastically Similarly operant conditioning of new re-sponses may depend on stochastic variations It is im-possible to prove stochasticity some presently unknowndeterministic system may ultimately explain the resultsHowever the evidence is consistent with the stochastichypothesis and inconsistent with the alternatives For ex-ample memory is not sufficiently powerful to explain alloperant variability memory-interference proceduresleave stochastic responding intact and behaviors corre-late well with those of stochastic models

Both genetics and reinforcement play important rolesin all three processes Reinforcement is always involvedin that learning provides information about when whatwhere and how much to vary Genetic influences areshown in that not all responses are possible and if theyare possible they are not all equiprobable (eg as inNeuringer et al 2001) Stochastic responding may be bi-ased due to experiences and knowledge that is biasesmay be established by reinforcement contingencies or bythe interface between the organismrsquos propensities (or be-havior systems see Timberlake amp Lucas 1989) and theenvironment In the case of random-number generationby human subjects biases may be due to misconceptionsconcerning what ldquorandomrdquo means However some bi-ases can be overcome through feedback and instructions(Lopes amp Oden 1987 Neuringer 1986)

The importance of variations for evolutionary changeis supported by the many sources of genetic variabilityand by the fact that genetic variability is conserved Asimilar argument holds for operant variability That thereare at least three independent sources of operant varia-tions as well as other elicitors such as environmentalnoise and response extinction supports its importancefor normal adaptive functioning

Structure of VariabilityEvery operant ldquoresponserdquo is actually an instance of a

class with each member of the class exerting the sameeffect on the environment (Skinner 1959) When a rat isreinforced for pressing a lever for example the class isdefined by closure of a microswitch and any way of ac-complishing the closure whether with the left paw theright paw or the mouth is as effective in producing re-inforcement as any other If the contingencies of rein-forcement so specify however the response class can beredefined and circumscribed which is another charac-teristic of the operant For example if only right-pawpresses are acceptable members of the reinforced classresponding will tend to occur only with the right paw(Stokes 1995) Of course the capabilities of the organismplay a role for example including ldquotail pressesrdquo in the op-erant class does not mean that the lever would be or couldbe pressed by the ratrsquos tail The structure of the operantclass therefore emerges from the current reinforcementcontingencies and the organismrsquos capability derivedfrom genetics development and prior experiences

696 NEURINGER

Operant variability has structure as well and as for alloperants that structure is due partly to the organism it-self and partly to the reinforcement contingencies Theresearch described throughout this article has providedexamples of the structure of variability generally in theform of sets of four or eight responses across L and Roperanda But the structure of variations can be speci-f ied more precisely For example Mook and her col-leagues (Mook et al 1993 Mook amp Neuringer 1994)reinforced rats for varying four response sequences acrossL and R levers under a lag 1 contingency Unlike in pre-vious studies however the acceptable class was furtherconstrained so that only sequences beginning with two Rresponses (RRLL RRLR RRRL and RRRR) were rein-forced The rats learned to vary their sequences withinthe imposed limitations that is they generally begantheir sequences with two R responses and varied the se-quences within those limitations Similar results wereobtained in the context of a radial-arm maze in whichonly a subset of arms was baited

The precision with which reinforcement contingen-cies can specify class was highlighted in research per-formed by Burns (1997) and Ross and Neuringer (2002)In the latter case college students drew rectangles on acomputer screen with the computerrsquos mouse Each rec-tangle constituted one trial (the screen cleared at the endof each trial) with reinforcement (points) depending onsimultaneous variations along three dimensions of therectangle location on the screen area of the rectangleand shape with the latter defined as the ratio of the rec-tanglersquos height to its width Although the subjects weregiven no instruction other than to ldquogain pointsrdquo theylearned to vary along all three dimensions with levels ofvariability significantly higher than those of yoked controlsubjects who were reinforced independently of variations

A second part of the Ross and Neuringer (2002) studydemonstrated even greater precision of control To gainpoints three new groups of subjects were required to re-peat along one of the dimensions while simultaneouslyvarying along the other two One group was required torepeatedly draw rectangles of approximately the samesize while simultaneously varying location and shapeanother group had to repeat the shape while varying areaand location and the third to repeat location while vary-ing size and shape Verbal instructions were minimal asthey were in the first part of the study Subjects learnedto satisfy these contingencies as well varying two at-tributes while simultaneously repeating one and signif-icant differences were obtained rapidly within 300 trials

These demonstrations raise questions as to how binaryfeedbackmdashreinforcement presented or notmdashcan simul-taneously control independent behavioral dimensionsThe issue is similar to that discussed above concerningsimultaneous reinforcement of variations and selectionsIt is similar also to those issues faced by researchers at-tempting to understand how the nervous system strength-ens particular synapses when many different neurons firesimultaneously and to ldquoallocation of creditrdquo in artificial

learning models For present purposes suffice it to notethe exquisite precision with which reinforcement controlsvariability and repetition

GeneralizationDo animals and people learn to ldquovaryrdquo implying a

generalizable skill or do they learn a specific distributionof specific responses across particular operanda Thequestion of generalization has been asked of many otherbehavioral competencies such as imitation When chil-dren learn to imitate do they learn stimulusndashresponseassociations (if mother moves her right hand the childlearns to move his right hand) or do they learn moregenerally to ldquoimitaterdquo Evidence indicates that reinforce-ment of imitative instances leads to generalized imitation(see eg Baer amp Sherman 1964 Bandura 1986) A sim-ilar question has been raised when pigeons rats or chil-dren learn to choose a stimulus that matches a previouslyshown sample For example if a pigeon is shown a reddisk as a sample it must then choose red when given achoice between that and blue Does the bird learn to matchIn this case the answer appears to depend on the train-ing procedures If trained with few stimuli such as redgreen and blue then the pigeon learns simple associa-tions (if red sample choose red if blue sample chooseblue) but if hundreds of different stimuli (pictures) areemployed during training then generalized ldquomatchingrdquois learned (Wright 1997) In each of these examples thelearning of a general competency or strategy was shownby transfer-of-training tests What was learned with oneset of stimuli or in one situation transferred to new stim-uli and situations We turn now to the question of whetherlearning to vary transfers

After the subjects in Maltzmanrsquos (1960) experimentdescribed above learned to generate remote verbal as-sociates they were asked to devise unusual uses of com-mon objects In comparison with those in the controlgroup these subjects generated significantly more un-usual uses indicating transfer of training to a differenttask However transfer was not observed in the oppositedirection from unusual uses to verbal associates

Holman et al (1977) using a procedure similar to thatof the Goetz and Baer (1973) research described aboverewarded preschool children verbally for novel drawingsand found increased novelty Unrewarded block construc-tions were also monitored and weak evidence was ob-tained for transfer of novel responding from rewardeddrawing to the unrewarded constructions In a second ex-periment preschoolers engaged in four independent taskspainting pen drawing wood block construction and Legoconstruction When the children were reinforced for vary-ing their pen drawings the novelty of paintings increasedalong with the pen drawings indicating transfer from onetask to the other However transfer to the topographicallydifferent block- or Lego-building tasks was not observed

Eisenberger and Armeli (1997) gave monetary rewardsto one group of fifth- and sixth-grade students the vari-ability group for describing unusual uses of common ob-

OPERANT VARIABILITY 697

jects A second group the repeat group received thesame rewards for describing common uses of the sameobjects In a later test all of the children were asked todraw pictures incorporating a circle and the group that hadpreviously been reinforced for varied responses drewmore unusual (or lower frequency) pictures than did therepeat group Thus there was transfer of novelty trainingfrom one domain to another with additional studies in theseries showing similar effects (Eisenberger Armeli ampPretz 1998 Eisenberger Haskins amp Gambleton 1999Eisenberger amp Selbst 1994) However Eisenberger et al(1999) failed to replicate the transfer-of-training effectThe variability and repeat groups responded similarly dur-ing the test phase In this study separate experimenterstrained and tested the subjects thereby avoiding possibleexperimenter-expectancy effects whereas the same ex-perimenter did both training and testing in Eisenbergerrsquosother studies Across the series of experiments there weremany other cases in which transfer was not observed andmore research is necessary to explain why

In Lee et alrsquos (in press) study with autistic individualsdescribed above after variable answers were trained toquestions verbal variability transferred to new experi-menters and new contexts

Arnesenrsquos (2000) results described above are consis-tent with transfer of training in an animal model Rats inthe Var group were trained to interact variably with oneset of objects When later tested in a new situation withnew objects the Var animals showed (in comparisonwith controls) higher levels of exploration manipulationof the novel objects and discovery of hidden reinforcers

Much of the evidence reviewed in this section there-fore indicates transfer of training and supports a generalldquovaryrdquo competency However there remain unresolvedquestions the most important of which is the extent towhich transfer occurs Note also a methodological pointMany of the studies tested whether transfer occurs whenthe contingencies are neutral with respect to the to-be-transferred behavior Holman et al (1977) is a case inpoint Variations were reinforced in one domain draw-ing and the question was whether variations wouldoccur in another domainmdashblock constructionmdashin whichreinforcement did not depend on varying A negativef indingmdashin this case absence of transfermdashindicatesnothing about the ability to generalize but only aboutthe tendency to do so given the absence of a reason (orcontingency) A better test would be to compare experi-mental and control performances when variability in thenew task is in fact functional as in Arnesen (2000)

SummaryBefore variability can be reinforced it must occur for

reasons other than reinforcement Some of the reasonsinclude variability inherent in all behavioral phenomenaldquoaccidentalrdquo environmental events that are unrelated toparticular behaviors and withholding of reinforcementTransfer from other situations may serve a similar functionFor example if an organism can discriminate shaping-

of-new-response periods from others then it may learngenerally to ldquovaryrdquo during shaping episodes After rein-forcement makes initial contact with response variationsthere are at least three behavioral strategies that can beemployed in order to maintain functionally high levelsof variability respond based on external random eventssuch as the toss of a coin avoid repetitions by remem-bering what responses had previously been made and re-spond stochastically Evidence for an endogenous sto-chastic process was seen in the similarity of responsedistributions to those generated by stochastic sourcesand by findings that interfering with memory for priorresponses either improves or leaves unaffected stochas-tic behaviors Stochastic responding as well as othertypes of variability-generating strategies depend uponcontingencies to define the required level of variabilityand the set from which variations emerge

IMPLICATIONS

One implication of reinforced variability is that oper-ant responses may differ in the extent to which their vari-ability is affected by reinforcement contingenciesmdashthatis operants may be ordered along a continuum of sensi-tivity of variability to reinforcement Another is that re-inforced variability causes us to question the generalityof some basic goals of behavioral psychology includingprediction and control A third is that reinforced vari-ability may help us to provide a scientifically useful wayto describe voluntary action In particular the variabilityof all voluntary behaviors can readily be changed mo-ment to moment by discriminative stimuli and conse-quences Implications for everyday life also will be noted

Operant ContinuumAs was indicated above the operant is composed of

instances that is the operant is a class of many differentresponses all functionally equivalent but often topo-graphically distinct Skinner suggested and I hypothe-size that instances within the class emerge stochastically(see Moxley 1997 for a discussion of Skinnerrsquos posi-tion) Although reinforcement contingencies and contextinfluence the individual probabilities and their distribu-tions emission is stochastic nonetheless The particularmember of the operant class generally cannot be predictedwith a high degree of certainty For example a particu-lar rat may be more likely to press the lever with its leftpaw than its right but left- and right-paw responses areemitted probabilistically This class-nature of the operantsuggests one way to conceptualize operant variability interms of size of class and within-classes probability dis-tributions High variability implies large classes equalprobability distributions or both low variability impliesthe opposite

Variability may be modified more readily in some op-erants than in others implying that operant variability isnot an all-or-none phenomenon but rather a continuumwith reinforced variations in variability (RVV) differing

698 NEURINGER

for different operants RVV might best be understood byanalogy to the concepts of speed and acceleration Bothvariability and speed range from low to high RVV isanalogous to acceleration the rate of change in speedHighly sensitive RVV is indicated by the possibility ofreinforcing many different levels of variability from rep-etition to random-like responding and by the fact thatchanges in level occur readily and rapidly

Different operants manifest differences in RVV Forexample variability of chain pulls changed more thanthat of leverpresses when variability was reinforced inrats (Morgan amp Neuringer 1990) Some operant-like re-sponses may show relatively low RVV For example fir-ing rates increase in single hippocampal nerve cellswhen dopaminergic substances are contingent upon theresponse (Stein Xue amp Belluzzi 1994) suggesting thatnerve f iring is controlled by reinforcement To datehowever there is no evidence that variability of nervefiring is similarly controlled Other cases might includereinforcement of nondiscriminable (subconscious) re-sponses such as Hefferline and Pererarsquos (1963) reinforce-ment of subaware thumb twitches and biofeedback ofmuscle and cardiovascular activity In these cases rein-forcement may modify performance but it is not knownwhether variability is also sensitive to reinforcement andif so to what degree Human verbal variability may bethe most readily manipulable of all behaviors the rangeof predictable to unpredictable verbal responses beingimmense with extraordinary sensitivity to context andconsequence RVV may also differ across organisms (per-haps from simple to complex) individuals (perhaps asan index of intelligence or learning ability) and physio-logical and psychological states (WKY vs SHR depres-sion and autism)

RVV may help to distinguish operant responses fromother superficially similar cases of selection by conse-quences That is the ldquooperantrdquo may also best be concep-tualized as a continuum with RVV an important deter-miner of position along the operant continuum Oneexample of selection by consequences but low ldquooperant-nessrdquo is genetic change Variability is essential for evolu-tionary change of course and levels of genetic variabil-ity are sometimes affected by environmental conditions(Moxon 1997) but there are few examples of levels ofvariability being influenced by selection pressures Thatis the source of much genetic variability is thought to berandom not influenced by current pressures and needsThus although selected by consequences genetic changemay be low along the operant continuum Other exam-ples of operant-like phenomena for which variability although important may be relatively insensitive to con-sequencesmdashvariability in these cases is elicited not se-lectedmdashinclude the development of the central nervoussystem and immune system responses (Hull et al 2001)In both of these cases change depends upon selection byconsequences but it is not clear whether variability canalso be selected Most commonly studied operant re-sponses on the other hand show high sensitivity of vari-

ability to reinforcing events But to reiterate the mainpoint of this section operants differ in terms of RVV andtherefore in terms of their ldquooperantnessrdquo from low sensi-tivity (possibly responses of single nerve cells) to extra-ordinarily high sensitivity (verbal behavior) The operantnature of variability as well as that of every other oper-ant dimension is best conceptualized as a continuum

Prediction and ControlSmith (1992) described Skinnerian operant condition-

ing theory as Baconian in nature a primary goal of Skin-nerian behaviorists being to control behavior and hecontrasted this with the more Aristotelian goal of de-scription Skinner and his students (eg Zuriff 1985) in-deed emphasized prediction and control Many of thestudies reviewed above however involved reinforce-ment procedures that increase variability and conse-quently decrease predictability It is not only ignorancetherefore that restricts prediction and control as Skin-ner suggested but reinforcement itself Operant meth-ods and theories when applied to variability lead togreater emphasis on description with control over pre-cise response characteristics possible under some butnot all conditions

In summary an operant is a class of responses underthe control of contingencies of reinforcement The classcan be controlled and predicted given knowledge ofconditioning history Indeed the size of the class andpossibly the distribution of within-classes probabilities orlevel of variability can also be controlled and predictedBut to the extent that instances emerge stochastically aparticular instance cannot be predicted The instant re-sponse can be explained as a member of a particularclass but prediction of the instance will be diff icultsometimes to the extreme This point will be developedin the next section

Voluntary BehaviorI hypothesize that voluntary behaviors are operants for

which RVVs are highly sensitive to influence by conse-quences One way to test this hypothesis would be togather empirical data concerning how voluntary versusinvoluntary actions are classified by human observers Asfar as I know there are no psychophysical data concern-ing this question but the hypothesis implies that highsensitivity of variability to operant contingencies RVVswill be present in all actions perceived to be voluntary

This operant-variability theory of voluntary action isbased on the supposition of two attributes First volun-tary acts must be potentially explainable (given suff i-cient knowledge) on the basis of genetics conditioninghistories current contingencies anticipated conse-quences intentions goals and the like That is volun-tary behaviors are lawful The second attribute is that thevoluntary act must be potentially unpredictable That isvoluntary acts are ldquofreerdquo of control by identifiable stim-uli If behavior is perceived necessarily to result frommdashto be predicted on the basis ofmdashexternal events includ-

OPERANT VARIABILITY 699

ing reinforcement contingencies it will not be perceivedas voluntary ldquoVoluntaryrdquo implies potential indepen-dence from stimulusndashresponse or responsendashreinforcerdetermination According to this theory both explain-ability and unpredictability are necessary attributes ofvoluntary acts

How though can behaviors be simultaneously ex-plainable and unpredictable Reinforcement contingen-cies play two essential roles Reinforcement (togetherwith genes prior experiences and current environment)helps to determine the set of possible behaviors fromwhich a voluntary act emergesmdashthat is the operant classReinforcement (together with the other contributorsspecified above) also helps to determine the level of vari-ability within the class Thus both instances comprisinga class and level of variability within that class can bepredictedmdashagain given sufficient knowledge Howeverthe individual response is at least sometimes (or relatively)unpredictable The reason is that the operant class fromwhich the response stochastically emerges is sometimesextremely large (eg ldquoname a wordrdquo or ldquodo somethingrdquo)and the contingencies sometimes call for equiprobablestochastic emergence (eg ldquofool merdquo) In the latter casea voluntary act may appear to be foolish meaninglessor unwarranted but if it is voluntary it can potentially beexplained (For example in response to the questionldquoWhy did you do that silly actrdquo the answer could beldquoJust to show that you canrsquot predict my behaviorrdquo seeScriven 1965) In these cases prediction of instancesfalls to very low levels indeed and I hypothesize evenan all-knowing observer will be unable to predict thenext response at greater-than-chance levels which aregiven by the size of the operant class and by within-classesprobability distribution Even within small classes (egin the game rockndashscissorsndashpaper in which unpredictabil-ity among the three possible responses is functional) be-havioral stochasticity may lead to unpredictability of theinstant response Note however that although voluntarybehaviors are sometimes unpredictable the opposite isoften the case When contingencies call for highly con-strained particular responses predictions can easily bemade For example if a police officer asks your nameyour response can most likely be predicted Voluntary re-sponses are therefore described as potentially unpre-dictable with variability contingencies engendering dif-ferent levels of unpredictability

According to this theory behaviors low in the sensitivity-of-variability dimension are described as relatively in-voluntary and those that are high in said dimension areperceived as voluntary Low RVV may be shown at anylevel along the variability continuum For example al-ways responding in the same unchanging way will beperceived as involuntary but so will always respondingrandomly (The coin does not voluntarily come up withheads or tails) This point is essential More than 2000years ago Epicurus argued that occasional randomswerves of atoms provided the source of voluntary be-havior in an otherwise determined universe but al-

though this conjecture was prescient it was not quitecorrect ldquoFree-willedrdquo or voluntary action whether it isbased on random swerves or on quantum mechanics isnot explained by variability per se but by functionalchanges in levels of variabilitymdashthat is by rapid andprecise changes in response to environmental demandsResponses can be placed along a voluntaryndashinvoluntarycontinuum isomorphic to the operantndashrespondent con-tinuum with voluntary behavior being functional pre-cisely in the same way as operant behavior is ThusSkinnerrsquos (1974) suggestion that voluntary behavior isoperant behavior was almost correct Only those oper-ants for which variability is highly sensitive to conse-quences will be perceived as voluntary

One additional example might help Operant classes(sets of potential responses) are created ldquoon linerdquo as we goabout our daily activities As we walk from car to officewe tend to walk in a characteristic way take the samepath with tightly constrained operant classes followingone after the other in rapid succession Class sizes aresmall and within-classes probability distributions arehighly peaked Predictions of next responses by a knowl-edgeable observer can readily be made However classsizes can be increased and probability distributions flat-tened we can walk in quite strange ways indeed dancealong a new path sing hand candy to passers-by Strangebehavior indeed but made possible by the ability of op-erant sets to be modified moment to moment not only interms of class definition but in terms of class size lev-els of within-classes variations and so on The same istrue for conversation Moment-to-moment changes inverbal class definition size and predictability of within-classes instances are hallmarks of voluntary behavior

GenerativityRelated to the definition of the operant and to our discus-

sion of voluntary action is that fact that operant behaviormdashleverpressing no less than languagemdashis by its naturegenerative This claim follows from the functional na-ture of the class of responses (all members of the classare effective in producing an outcome) and the stochas-tic emission of instances From these flow never-before-emitted instances patterns and sequences (see Epstein1996) Operants especially those high on the RVV con-tinuum are inherently generative

Self-ExperimentationVariations systematic or otherwise are a necessary part

of scientific work Neuringer (1981) and Roberts and Neu-ringer (1998) described how variations can be brought to bear explicitly on everyday life in the form of self-experiments Individuals vary aspects of their lives suchas the foods they eat social contacts and methods ofmemorizing they keep records concerning the effects ofthese manipulations and form hypotheses concerningcontrolling variables Using self-experimental methodsindividuals can learn how what when where and howmuch to vary Explicitly varying onersquos everyday behav-

700 NEURINGER

iors and thoughts may help to engender what Langer(1989 2000) has referred to as mindful livingmdashthat isnonautomatic voluntary engagement with beneficialeffects in different areas including learning The impor-tant point here is that variability can be voluntarily cho-sen maintained and modified in everyday life and thatexplicit variations may have important consequences

Implications for SocietyConsideration should be given to the importance of re-

inforcement contingencies to support variability in so-cial milieus A basic point made throughout the presentarticle is that merely permitting behavior to vary doesnot necessarily engender or maximize variability (as isshown by Yoke and Prob schedules) Analogously merelypermitting varied social behavior as in a free or laissez-faire society may not suffice to produce the desired di-versity If the results shown for individual organismscan provide useful information for groups (see eg Grottamp Neuringer 1974) then when variations are desiredsocieties might reward individuals for varying not sim-ply permit it

SummaryOperants differ in the extent to which reinforcers in-

fluence levels of variation When variability is highlysensitive to reinforcement the operant may be perceivedto be voluntary Voluntary responses have two attributesThey are functional implying that the class of responsesmdashthe operantmdashcan be predicted given sufficient knowl-edge But instances emerge stochastically implyingsome degree of unpredictability Because reinforcementof variability can generate extremely large classes withstochastic emergence of instances it suggests limitationsof the two basic goals of Skinnerian behavioral theorymdashnamely prediction and control Operant variability also hasimplications for self-experimentation and for society

RELATED AREAS

Operant variability may be important in many otherareas and I will briefly review some of these

Negative reinforcement To my knowledge there havebeen no published studies in which variability is nega-tively reinforced High variability might sometimesavoid or reduce a punisher such as loss of goods ormoney but at other times low variability could be func-tional Effects of positive and negative reinforcement onvariability may have different evolutionary bases onederived from increasing probabilities of locating sourcesof food or mates the other from avoiding confrontationsor predation (Driver amp Humphries 1988) One questionconcerns how quickly variability can be changed by neg-ative as opposed to positive reinforcement There isconsiderable evidence that loss functions differ fromgain functions (Kahneman amp Tversky 1984) possiblyindicating that positive reinforcement-of-variabilityfunctions will differ from negative ones as well

Resistance to change In some of the studies describedabove variability appeared to be more resistant tochange than repetitions were for example alcohol hadlittle effect on variable operant responses but interferedwith repeated ones Doughty and Lattal (2001) tested thegenerality of these f indings using four-response se-quences with pigeons a threshold variability contin-gency in the Var component of a multiple schedule andLRLR in the Rep component Doughty and Lattal chal-lenged stability of responding by prefeeding the birds inone experiment and providing reinforcers independentlyof responding in the other (see Nevin 1992) Operantvariability was again more resistant to change than wasoperant repetition Thus reinforcement of variabilitymight increase the stability of responding (that is its re-sistance to change) a seemingly paradoxical findingHowever Mechner (1992) suggested that this is the casefor complex skills such as piano performance Varyingpractice routines will make the performance relativelyinsensitive to unforeseen changes in the environmentCare must be taken however when variation is comparedwith repetition because they have different measuresAlso the results reported by Doughty and Lattal werebased on the performance of three pigeons and were notconsistent throughout the experiment The hypothesiswarrants further study

Attraction In some bird species complex male songsattract females (for a review see Catchpole amp Slater1995) There is evidence that male song complexity isunder the discriminative stimulus control of a receptive fe-male (Searcy amp Yasukawa 1990) That explicit reinforce-ment may play a role in such song complexity is suggestedby the Manabe et al (1997) study described above An in-teresting question for further research is the extent towhich operant contingencies contribute to the attractioncreated by behavioral variability in these and other species

Discrimination of variability Animals and people canuse the variability of a stimulus array or entropy as a dis-criminative stimulus (Young amp Wasserman 2001) Whatis the relationship between perception of variability onthe one hand and emission of variable responses on theother Do individuals attempt to match their own levelsof variation to those of others in their environments whichwould involve both perception and emission How mightdiscrimination of onersquos own level of variability influenceonersquos ability to modify such levels and do influences thatobscure discriminations therefore affect ability to change(see eg van der Meer van der Weel amp Lee 1995)Hunziker et al (2002) report that although variabilitywas successfully reinforced their human subjects couldnot correctly describe the contingencies of reinforce-ment a finding similar to those of informal observationsin other studies (eg Neuringer Deiss amp Imig 2000)but whether subjects could discriminate variability levelsrather than contingencies was not studied

Attention and awareness When attention is subject tointerference or is withdrawn from the task at hand abilityto vary is degraded (eg Baddeley Emslie Kolodny amp

OPERANT VARIABILITY 701

Duncan 1998 Evans amp Graham 1980 Neuringer amp Voss2002) It is unclear at present how attention influencesthe different processes underlying operant variabilityand in particular memorial and stochastic processesRich areas for additional study are the relationships be-tween attention awareness and consciousness on theone hand and operant variability on the other Voluntaryaction and consciousness are often related (see egLibet 1999)

Concepts Concepts are thought by many to be sets ofinstances (much like operant sets) whose structures aregraded meaning that there are more or less typical in-stances of such concepts as birds (robin is typical ostrichis not) Barsalou (1987) showed that the gradation ofthese cognitive structures is manipulable or as he putsit not stable A number of variables influence structureincluding context individual differences and points ofview Barsalou argued that these structures are createdas the need arisesmdashthat is on line by the thinking indi-vidual The structure of concepts may parallel that of op-erants and in both cases levels of within-structures di-versity or variability may be modified by consequencesOperant variability may be found in the cognitive as wellas in the behavioral domain

Play and humor Highly variable behaviors are ob-served when animals and people especially children en-gage in play and humor (see eg Fagen 1981) The re-sult of play may be acquisition of new responses (seeSutton-Smith 1975) Although not currently needed re-sponses learned during play may provide the source ofvariations for successful shaping of behaviors at anothertime This is quite analogous to the role of conserved ge-netic variants which are passed along to progeny but donot currently serve any identifiable role Play may alsoinvolve learning how to vary operantly

Reinforcement by variability The self-maintainingnature of play suggests that engaging in variable behav-iors may itself be reinforcing Indirect support comesfrom findings that varied stimulation can serve as a re-inforcer and that stimulus repetition becomes aversive(Fiske amp Maddi 1961) Possibly also related are findingsthat varying reinforcers and permitting choice among re-inforcers increase their value (Bowman Piazza FisherHagopian amp Kogan 1997) and that permitting choicesamong varied responses is also preferred to demandingfixed responses (Catania 1980)

Foraging Two issues regarding variability are presentin the foraging literature One has to do with risk prefer-ence versus avoidance (see eg Shettleworth 1998) Asecond has to do with search strategies Most reportedcases of foraging involve systematic search patterns(Stephens amp Krebs 1986) However search patterns aresometimes stochastic or highly variable (see eg Peter-son 1997) One question is whether some species can varybetween these two strategies and the challenge is to iden-tify the pressures leading to and the advantages of each

Animal training Wilkes (1996) has applied operantvariability techniques in developing training proceduresfor pet dogs More generally reinforcement of variabil-

ity may enable the gaining of control over behaviors ofanimals and as has been suggested above may be an im-portant part of the process of shaping

Models of learning Reinforced variations in variabil-ity may be important when human and animal learningis being modeled whether the models are mathematicalin form computer based or machine based (Blumberget al 2002 Brooks 1999 Sutton amp Barto 1998)

CONCLUDING COMMENT

Genes interact with environments to produce diverseanatomical physiological and behavioral phenotypesGenes and environments also provide for another level ofdiversitymdashnamely operant variability There are manysources of operant variability including random environ-mental events memory for past responses and stochasticgeneration each of which ultimately derives from ge-netic and environmental influences To the extent thatoperant variability is based on a stochastic source how-ever the behavioral results cannot be predicted fromknowledge of these influences That is organisms haveevolved to behave unpredictably The positing of sto-chasticity to explain behavior has a history extending atleast as far back as Epicurus with major contributors in-cluding Gustav Fechner (see Heidelberger 1987) EgonBrunswik (see Gigerenzer 1987) Skinner (1959) andSuppes (1984) Knowledge of the stochastic nature ofthe operant and of how it is controlled leads to a worldview that differs from genetic and environmental deter-minism with roots in the works of the scholars just men-tioned At the scientific level this view leads to the ex-perimental analysis of consequence-influenced variabilityAt the philosophical level it leads to explanations of vol-untary action At the personal level it can lead to change

REFERENCES

Amabile T M (1983) The social psychology of creativity New YorkSpringer-Verlag

Antonitis J J (1951) Response variability in the white rat duringconditioning extinction and reconditioning Journal of Experimen-tal Psychology 42 273-281

Archer J amp Birke L (1983) Exploration in animals and humansCambridge Van Nostrand Reinhold

Arnesen E M (2000) Reinforcement of object manipulation in-creases discovery Unpublished undergraduate thesis Reed College

Baddeley A D (1966) The capacity for generating information byrandomization Quarterly Journal of Experimental Psychology 18119-129

Baddeley A [D] Emslie H Kolodny J amp Duncan J (1998)Random generation and the executive control of working memoryQuarterly Journal of Experimental Psychology 51A 819-852

Baer D M amp Sherman J A (1964) Reinforcement control of gen-eralized imitation in young children Journal of Experimental ChildPsychology 1 37-49

Baker M J amp Koegel R L (1999) Transforming children withautismrsquos ritualistic behaviors into positive play interactions with sib-lings Paper presented at the meeting of the California Association ofBehavior Analysis San Francisco

Balsam P D Deich J D Ohyama T amp Stokes P D (1998) Ori-gins of new behavior In W OrsquoDonohue (Ed) Learning and behav-ior therapy (pp 403-420) Boston Allyn amp Bacon

Balsam P D Paterniti A Zechowy K amp Stokes P D (2002)

684 NEURINGER

in one study lowest baseline variability was observedwhen rats pulled trapezes intermediate levels were ob-served when they pushed keys and highest levels wereobserved when they pressed levers (Morgan amp Neuringer1990) with variability increasing in all cases when it wasreinforced In another study operant variability wascompared across rats and people with similar effects ob-served in the two species (Neuringer Deiss amp Imig 2000)

ParametersWe turn now to some parameters studied to date Many

other parameters as yet unstudied are no doubt also im-portantmdashfor example motivation and types of reinforcers

Age Older people (see eg Van der Linden Beertenamp Pesenti 1998) and older rats (see eg Lamberty ampGower 1990) respond less variably than their youngercounterparts when variable responding is reinforced or re-quested but in most cases it is not clear to what extent thevariability is elicited as opposed to reinforced Neuringerand Huntley (1992) compared reinforced variability in2-month-old (young) and 10-month-old (mature) Long-Evans rats Under Yoke contingencies the young rats re-sponded significantly more variably (two levers four re-sponses per trial) than the mature rats which constitutesan elicited effect When reinforcement was contingenton variability (lag 4) variability increased significantlyin both groups thereby overcoming the initial differencecorrelated with age Thus explicitly reinforcing vari-ability in older animals caused them to respond no dif-ferently than younger animals did a finding potentiallyrelevant to real-world human aging However when Wells(1999) compared 8-month-old (mature) to 24-month-old(old) rats although the mature rats responded more vari-ably than the old rats did under Yoke contingencies thedifference was not statistically significant Additionalstudies are needed especially of whether reinforcingvariable behaviors in older populations can facilitatelearning and performance

Gender Male and female rats are equally sensitive toreinforcement of variability (Neuringer amp Huntley 1992van Hest et al 1989) However there was a tendency(not statistically significant) for males to respond over-all more variably than females did and because thesestudies did not control for differences in size and weightof subjects or for other aspects of motivation additionalstudies are necessary

Stage of training Stokes and Balsam (2001) reporteddifferences in variability as a function of whether variabil-ity was initially reinforced early or later in training (seealso Gottlieb 1992) Human subjects generated sequencesof 10 L and R responses per trial on a computer keyboardunder lag 0 and lag 25 contingencies Under lag 0 everytrial ended with reinforcement under lag 25 the currentsequence had to differ from each of the previous 25 Whenlag 25 was experienced at the very beginning of traininglevels of variability at the end of training (when lag 0 wasoperative) were lower than they were when lag 25 was ex-perienced after 50 trials of initial lag 0 These results were

interpreted to suggest that there is an ldquooptimal period forsetting variability levelsrdquo (Stokes amp Balsam 2001 p 181)but that general conclusion needs further testing and anumber of questions need to be answered One has to dowith the contribution of reinforcement frequency Underlag 0 every sequence leads to reinforcement but rein-forcement is intermittent under lag 25 Another has to dowith whether learning to vary differs from learning to repeatwith respect to optimal periods (see eg Bouton 1994)

Related to stage of training are effects of prior experi-ence Hunziker Lee Ferreira da Silva and Caramori(2002) showed that for human subjects if Var contingen-cies were experienced just prior to Yoke contingencieslevels of response variation during the Yoke phase werehigher than they would have been if the Var contingencieshad not been previously experienced The time delay be-tween Var and Yoke experiences mattered with a long in-terval (3 months) producing only a small effect Saldanaand Neuringer (1998) also found effects of prior Var train-ing on performances in Yoke (increased variability inYoke) but the opposite did not hold that is training underYoke had relatively little effect on Var Hunziker Caramorida Silva and Barba (1998) reported similar effects withrats Thus there may be an asymmetry Reinforcement forvarying has relatively strong influences but reinforce-ment without regard to variations does not adversely affectlearning to vary These different effects are not surprisinggiven that Yoke contingencies are permissivemdashreinforce-ment is given whether or not responses varymdashwhereascontingencies under Var can be quite demanding How-ever the results motivate a number of additional ques-tions One is whether effects of Var training differ fromthose observed when repetitions are reinforced Another iswhether direct reinforcement of variable behaviors canhelp animals or people to overcome the detrimental effectsof prior noncontingent experiences such as those that pro-duce learned helplessness (Seligman 1975)

Environmental constraints Stokes (1995) arguedalso that variability increases when behavior is con-strained The term ldquoconstraintrdquo is ambiguous because allreinforcement contingencies exert a form of constraintIn one study Stokes (1995) compared different strate-gies of shaping leverpresses Rats in one group were re-inforced initially for pressing the lever with only theright paw (right only) and eventually for any press (withright or left paw or with any other part of the body) Theorder of training was reversed for a second group ldquoanyrdquofirst and then right only Behavior sequences were morevariable in the right-only-first group leading Stokes(1995) to conclude that early constraints lead to highvariability (see also Stokes Mechner amp Balsam 1999)That is an important hypothesis with many implicationsbut controls are needed before the results can be attrib-uted to the contingencies For example ldquoright onlyrdquo andldquoanyrdquo differed in terms of reinforcement distributions aswell as in the precision of the reinforcement operationthe ldquoanyrdquo reinforcers were provided immediately and au-tomatically by the programming equipment whereas in

OPERANT VARIABILITY 685

the right-only condition a human observer had to decidewhether a response had occurred and then present the re-inforcer necessarily leading to longer and more variedlatencies and a less consistent contingency Furthermorethere are many cases in which increasing constraints suchas those due to a decrease in the number of responses pertrial (Page amp Neuringer 1985) or the requirement of ex-actly four responses on each of two keys (Schwartz 1980)results in decreased variability Thus the contributionsof response-definition constraints and environmentallimitations must be explored further

SummaryThe generality of operant variability is shown by the

robustness of the phenomenon across procedures mea-sures responses and species As with other operant phe-nomena parameters influence the effects

FUNCTIONS

All operants including variability are functional Op-erant responses produce reinforcing consequences Be-cause other behavioral competencies depend on it oper-ant variability is functional in a broader sense as wellSome of these functions will now be described

Operant ConditioningThe potential of a reinforcer to change behavior is lim-

ited by the range of available variations (Hull Langmanamp Glenn 2001 Skinner 1981) and thus whatever pro-duces those variations contributes to the conditioning ofnew operants Much work has been done to identifysources of variation (Balsam Deich Ohyama amp Stokes1998 Segal 1972 Staddon amp Simmelhag 1971) butNeuringer (1993) was first to show that the same rein-forcers that strengthen individual responses can also pro-duce the necessary variability and that these two functionsof reinforcement may work simultaneously Implied isthat when an operant response is being conditioned re-inforcement works not only to strengthen an individualresponse but at the same time to engender the variabilityfrom which the response emerges (That is variability isnot solely a baseline phenomenon elicited or noise)Concurrent reinforcement of variations and individualtarget sequences will be the topic of this section

Pigeons in one experiment and rats in another were re-warded for varying among 16 possible response sequences(four responses per trial L and R operanda lag contin-gencies Neuringer 1993) Concurrently one of the 16designated as the target sequence was reinforced when-ever it occurred Thus reinforcements were given bothfor sequence variations and for a single selected targeta reinforced-variation-and-selection (RVS) procedureWhen the target was ldquoeasyrdquo (eg LLLL) it emergedrapidly from the baseline variations Given that the tar-get was always reinforced reinforcers were obtained morefrequently when the subjects did not vary and only re-peated the target When the target was ldquodifficultrdquo (eg

LLLR) it was not learned As in the choice experimentdescribed above (Neuringer 1992) the relative valuesor difficulties of target versus variations governed theprobability that the target would be learned

However even difficult targets could be conditionedunder the RVS procedure as was shown by another ex-periment in the same series (Neuringer 1993) Sequencevariations were again reinforced concurrently with a dif-ficult target RRLR but now the frequency of reinforce-ments for varying was systematically decreased Satis-fying the variability contingency was intermittentlyreinforced according to a VI schedule and over sessionsreinforcement for varying became less and less frequentThis technique resulted in acquisition of even the mostdifficult target sequence whereas a control group rein-forced for the target alone (without the concurrent rein-forcement for variations) never learned

Reinforcers were obtained more frequently in the RVScondition than in the control condition (since the RVSsubjects were rewarded both for varying and for emittingthe target sequence) and the additional reinforcers mighthave generated greater motivation to respond (Killeen1981) Thus motivation not the direct reinforcement ofvariability might have been responsible for the RVS ef-fect To test this possibility Neuringer Deiss and Olson(2000) reinforced three groups of rats for emitting a par-ticularly difficult target sequence LLRLR The experi-mental group was reinforced concurrently for sequencevariability on the average of once per minute (VI 1 min)The target-only control group received no reinforce-ments other than for the LLRLR target A third groupwas yoked to the experimental animalsmdashreinforced onceper minute for any sequence variability not requiredmdashand as for the other two groups whenever the target oc-curred If the additional reinforcement was responsiblefor the facilitated learning under RVS then because theexperimental and yoked groups received equal reinforce-ments they should have learned equally well but thatwas not the result Only the experimental group learnedto emit the LLRLR target The yoked controls respondedat high rates throughout the experiment but they did notvary sufficiently to make contact with the target contin-gency and never learned The target-only control groupreceived insufficient reinforcement and their responsesextinguished Therefore the added reinforcers helped tomaintain motivation as suggested above but reinforce-ment of variations was necessary for learning of a diffi-cult sequence The RVS procedure maintains both highmotivation and high variation and the two together re-sult in the acquisition of difficult-to-learn responses

Grunow and Neuringer (2002 Experiment 2) studiedone possibly important parameter of the RVS proceduremdashnamely the level of variability When a difficult target re-sponse is being trained does it matter whether reinforcedvariability is low intermediate or high As describedabove each of four groups of rats was reinforced for vari-able response sequences the first group for high vari-ability the fourth group for low variability and the other

686 NEURINGER

two for intermediate levels In other ways the procedurewas similar to that of the Neuringer Deiss and Olson(2000) RVS contingencies just described with a singledifficult target sequence always reinforced The resultswere that the higher the level of reinforced baseline vari-ability the more quickly the target was learned Thusvariability levels indeed mattered and if the just-re-ported effect is found to be general it will be most help-ful to reinforce high variability when difficult-to-learnnew responses are being shaped Other parameters re-quiring study are class definitionsmdashwhich responses areincluded in the class of acceptable variationsmdashand num-ber of instances within the class

One additional effect should be noted When varia-tions among a set of responses are reinforced if one vari-ant is selected for extinction (no longer reinforced) thefrequency of that instance decreases Neuringer (1993)showed this for pigeons responding under a lag-variabilityschedule When one of the 16 sequences was no longerreinforced (with no other change to indicate its status) thefrequency of that sequence decreased Thus reinforce-ment of variations provides the baseline to select for (iestrengthen) and select against (ie weaken) individualinstances

These findings have potentially important implicationsFirst as has been indicated the same reinforcers thatstrengthen particular instances can also reinforce varia-tions That a binary eventmdashreinforce or not reinforcemdashsimultaneously affects variations and selected instancesindicates both the power of reinforcement and a concep-tual problem How reinforcers exert such diverse effectsis a question to which we will return

Second shaping procedures may implicitly reinforcebaseline variations An approximation to a goal responseis reinforced for a period of time but eventually rein-forcement is withheld until a response closer to the goalis obtained It is commonly thought that withholding re-inforcement or mini-extinction induces the necessaryvariations for shaping to proceed as has been indicatedabove An alternative suggested by the present studiesis that organisms learn both to vary (within a set) and to re-peat (selected instances) until one instance is always rein-forced That is the process of shaping might involve re-inforcement both of variations (generally done implicitly)and of targeted instances If so a useful contributor tothe shaping process might be explicit reinforcement ofvariations

Third reinforcement of variations could facilitate ac-quisition of other competencies such as cognitive andmotor skills Siegler (1996) has shown in an important se-ries of studies that childrenrsquos acquisition of complex cog-nitive skills is correlated with high variability Anotherexample is that when motor skills are practiced in a vari-able manner high quality of performance results (Ma-noel amp Connolly 1995 1997 Mechner 1992 Schmidtamp Lee 1999) However it is rare that variability in thesedomains is directly reinforced as part of the training oreducational process

Fourth the RVS procedure might be used in therapeu-tic settings to modify maladaptive and self-injurioushabits Therapists sometimes employ noncontingent re-inforcement procedures to weaken undesirable behaviors(Vollmer Iwata Zarcone Smith amp Mazaleski 1993)The reasoning is that many maladaptive behaviors aremaintained by unintended consequences and that non-contingent rewards may compete with those conse-quences thereby lowering the strength of the unwantedhabit For example when a staff personrsquos attention toself-injurious behavior helps to maintain it providing at-tention noncontingently may decrease the strength of theunwanted target behavior However the RVS proceduremight accomplish the same goal more effectively andwithout undesirable side effects (Vollmer RingdahlRoane amp Marcus 1997) Noncontingent rewards some-times result in unwanted ldquosuperstitiousrdquo behaviors andthat would be avoided under RVS Furthermore RVSwould help to provide the set of baseline behaviors nec-essary to establish desirable new responses

Finally RVS procedures may help to gain control overand eventually to weaken other undesirable target behav-iors The target can be incorporated within a set of moredesirable responsesmdasha new operant class established withthe target as one membermdashand variable instances rein-forced Once variable responding is successfully main-tained including the undesirable target response the lattercan be weakened by selectively withholding reinforcement

Summary The conditioning of operant responses de-pends on both baseline variations and selection of suc-cessive approximations to a goal response The same re-inforcers used to select the approximations can reinforcebaseline variations thereby defining the precise set ofappropriate variants and the desired levels of variationSuch concurrent reinforcement of variations and selec-tions may facilitate the acquisition of difficult-to-learnoperants and selective withholding of reinforcementmay weaken undesirable instances

Creativity and Problem SolvingBehaving in a novel unusual or nonrepeating manner is

part of creative activities (Campbell 1960) Novelty andvariation may arise through combinations of previouslylearned behaviors (Epstein 1996) transformations of thesebehaviors (Boden 1994) direct reinforcement of variabil-ity as described above or some combination thereof Ofcourse neither novelty nor variability alone suffices forcreative outputmdashthe throw of the dice or the toss of acoin are not creative nor are haphazard strokes of a paint-brush on a canvasmdashbut without variation creativity islow Whether the variability required for creativity canbe reinforced is relevant to whether creativity itself canbe reinforced The latter question has led to contentiousdebate with some arguing that reinforcement facilitatescreativity (Cameron amp Pierce 1994 Eisenberger ampCameron 1996 Stokes 2001 Winston amp Baker 1985)and others focusing on detrimental effects of reinforce-ment (Amabile 1983 Deci Koestner amp Ryan 1999)

OPERANT VARIABILITY 687

The debate goes well beyond the present discussion butsome of the research described above is relevant Firstreinforcement contingent on variability generates higherlevels of behavioral variability than does reinforcementcontingent simply on responding as has been shown bythe many studies described above in which Var contin-gencies were compared to Yoke contingencies Similarlyprecise reinforcement contingencies matter for creativity(Cameron amp Pierce 1994 Lepper amp Henderlong 2000)If any product rather than only creative ones is rein-forced creativity tends to fall (just as variability decreasesunder yoke conditions) Second approach to or antici-pation of reinforcement interferes with operant variabil-ity (Cherot et al 1996 McElroy amp Neuringer 1990Neuringer 1991) which again might be related to effectson creativity when an individual focuses on the rein-forcer Note however that although variability decreaseswith proximity to the reinforcer overall levels of vari-ability are much higher when variability is reinforced thanwhen it is not The same may be true for creativity aswell anticipation of reward may interfere with creativeoutput to some extent but at the same time high creativ-ity may depend on its reinforcement

Operant variability may be important for problemsolving as well Here too there is disagreement as towhether training is helpful or not Early studies by Maier(1933) indicated that it is and these found support inlater work (eg Lung amp Dominowski 1985) Somestudies showed that those individuals most likely toldquobreak the setrdquo are also most likely to solve a problem(Dover amp Shore 1991) However a number of studiesfound that practice had no influence (Duncan 1961)

Arnesen (2000) explored a related phenomenon witha rat model One group Var was reinforced for variableinteractions with novel objects and another group Yokewas reinforced independently of such interactions EachVar rat received training with 12 different objects oneper session For example in the presence of a soup canthe rat was reinforced the first time it entered the canbut thereafter a different response was required such asrolling it climbing on it and so forth This procedure issimilar to that of Goetz and Baerrsquos (1973) described aboveexcept that objects changed from session to session Ratsin the Yoke control group had identical experiences in anadjoining identical chamber that contained identical ob-jects but now rewards were not contingent on object ma-nipulation After this training the rats were tested in adifferent room with 30 never-before-experienced ob-jects scattered on the floor Hidden in each was a smallpiece of food When compared to their Yoke controls theVar animals were more active during this test exploredmore manipulated objects more and most importantlydiscovered significantly more food pellets Another con-trol group received no preliminary training with objectsand their performance in the test session did not differfrom that of the yoked controls Therefore when ratswere reinforced for variable interactions with objectsthe result was an increased tendency to explore manip-ulate and discover new sources of reinforcement It

would be of considerable interest to determine whethera similar effect occurs with children

Assessment and Treatment of PsychopathologiesAbnormal levels of behavioral variability characterize

some human psychopathologies as has been shown byclinical observations as well as by research with humansubjects and animal models Few studies have attemptedto reinforce variability directly in such problem popula-tions The main question is whether reinforcement canmodify abnormal levels of variability in the direction ofnormalcy and whether such changes might have associ-ated beneficial effects

Autism Stereotyped responding characterizes indi-viduals with autism (Hertzig amp Shapiro 1990) For ex-ample Baron-Cohen (1992) asked children with autismand others in a control group to hide a penny in one handso that the experimenter could not guess which hand heldthe penny Those with autism were more likely thanthose without to generate a simple predictable patternsuch as repeatedly switching back and forth from left toright hands Such repetitive responding can be maladap-tive For example Mullins and Rincover (1985) askedchildren with and without autism to pick one of five cardsFood was occasionally found in a cup behind the cardEach of the five cards was a discriminative stimulus fora different schedule of reinforcement including contin-uous reinforcement (CRF every choice reinforced) FR 2(fixed ratio 2 in which every second choice was rein-forced) FR 4 (every fourth choice reinforced) FR 7 andFR 11 Control participants sampled all five alternativesand quickly learned to choose the most frequently rein-forced CRF card Autistic children sampled only a lim-ited number of cards and therefore often preferred anonoptimal alternative Thus children with autism tendto respond less variably than normal children and thiscan result in failure to adapt

Miller and Neuringer (2000) asked whether direct re-inforcement could influence the variability of individu-als with autism Sequences of responses on two largebuttons were first reinforced probabilistically and inde-pendently of variability (Prob phase) after which rein-forcement was contingent upon satisfying a thresholdvariability contingency (Var phase) The autistic partic-ipantsrsquo variability was significantly lower than that ofmatched controls during both Prob and Var phases butwhen variability was reinforced its levels increased sig-nificantly in both groups This is an important findingbecause it suggests that direct reinforcement of variabil-ity may help to modify some of the behavioral stereo-types that are characteristic of autism Lee McComasand Jawor (in press) offer support for this hypothesisThree individuals with autism were reinforced for variedverbal responses to social questions the responses had tobe appropriate given the context Variability increased intwo of the three individuals In a related procedureBaker and Koegel (1999) reinforced children with autismfor varied social interaction in a game-playing contextAlthough the reinforcement was permission to engage in

688 NEURINGER

the stereotyped behaviors preferred by the child the re-sult was increased social interactions both during theperiod of the experimental manipulation and in 1- and2-month follow-ups These three studies taken togethersuggest that reinforcement of varied behaviors may facil-itate modification of nonfunctional ritualistic and stereo-typed behaviors

Depression Depression provides another example ofmaladaptive consequences of low variability Lapp Mar-inier and Pihl (1982) observed that depressed womenproduced fewer alternative solutions to hypotheticalproblems than did nondepressed women Channon andBaker (1996) found that attempts by moderately de-pressed college students to identify faults in a series ofinterconnected circuits were less varied and less success-ful that were those of nondepressed controls HorneEvans and Orne (1982) found that when depressed pa-tients were asked to generate a random sequence of num-bers their responses were less variable than were thoseof controls These experimental results are consistentwith clinical observations

Can depressed individuals be rewarded for increasingvariation Hopkinson and Neuringer (in press) dividedcollege students into mildly depressed and nondepressedgroups on the basis of the CES-D scale a paper-and-pencil self-evaluative index of depression In the first oftwo phases sequences of responses on a computer key-board were reinforced independently of variability (Prob)following which only highly variable sequences were rein-forced (Var) Under Prob depressed students respondedsignificantly less variably than did the controls in agree-ment with previous findings Under Var variability in-creased significantly in both groups Indeed by the endof the experiment depressed and nondepressed subjectswere responding equally variably (Instructions were alsoeffective in increasing variability in the depressed par-ticipants) Thus even in a simple computer-game envi-ronment mildly depressed subjects manifested lowervariability than controls did but direct reinforcementmodified their behavior to control levels Can reinforce-ment of variability be used in real-world conditions tohelp depressed individuals An affirmative answer comesfrom cognitive behavior modification procedures thatwork to increase alternative solutions to problems (Beck1976) A more general conclusion from these studies ofautistic and mildly depressed individuals is that direct-reinforcement-of-variability procedures may benefitthose who manifest abnormally low levels of variability

Attention Deficit Hyperactivity Disorder (ADHD)ADHD may provide an example of the opposite stateabnormally high levels of variability (Barkley 1990)Wultz Sagvolden Moser and Moser (1990) proposedthat the spontaneously hypertensive rat (SHR) may serveas a good animal model of ADHD the SHR shows manyof the behavioral characteristics of that disorder includ-ing hyperactivity poor learning ability poor ability totolerate delayed reinforcement and poor self-control

The next study to be described asked about operant vari-ability in SHRs

Mook Jeffrey and Neuringer (1993) compared vari-ability of SHRs and of a control strain Wystar Kyoto(WKY) under two conditions Prob and Var The mainresult was that SHRs tended to respond variably whetheror not the contingencies required it consistently with theresults from the human ADHD literature When vari-ability was reinforced the two strains responded at ap-proximately equally high levels of variation These re-sults were confirmed in two other studies (HunzikerSaldana amp Neuringer 1996 Mook amp Neuringer 1994)Thus control rats varied only when the contingencies re-quired variations whereas SHRs varied whether or notthe contingencies required it As might be expected fromthese results when reinforcement depended on repeti-tions the SHRs performed poorly (Mook et al 1993)

Another similarity between SHRs and ADHD humanswas seen when amphetamine a drug that has effects simi-lar to those of Ritalin often prescribed for ADHD was ad-ministered (Mook amp Neuringer 1994) When repetitionswere required for reinforcement amphetamine improvedperformances by the SHRs so that they were as accurate assaline-injected WKYs a finding again consistent withthose in the ADHD literature These studies support SHRsas a model of ADHD and suggest that additional attemptsshould be made to assess whether or not individuals withADHD are less able to change levels of variability thanare controls Although as indicated above there is someevidence in support of this hypothesis (Barkley 1990)when Saldana and Neuringer (1998) directly comparedhuman ADHD and control participantsrsquo operant variabil-ity within the context of a computer game no differenceswere observed Most of those with ADHD were main-tained on Ritalin and other drugs however a fact unfor-tunately not documented in the original article

Drug EffectsAlthough many drugs are known to influence behavioral

variability (Brugger 1997) rarely have attempts beenmade to analyze explain or identify the sources of theeffects As will be discussed below operant variabilitycan result from the memory-based process of remember-ing previous responses and then behaving differently orcan result from responding stochastically a process thatappears to be quite different McElroy and Neuringer(1990) attempted to distinguish the effects of alcohol onmemory from its effects on stochastic processes in astudy in which one group of rats was reinforced for vary-ing four-response sequences under a lag 5 contingency(Var) and another was reinforced for repeating a singlesequence LLRR (Rep) Ethanol injections of 075 and20 gkg affected Var and Rep performances differentlyWhereas the Rep animals were much less able to satisfythe LLRR contingency at both doses the Var rats re-sponded variably whether or not alcohol had been ad-ministered Similar results were obtained by Cohen et al

OPERANT VARIABILITY 689

(1990) with a multiple schedule in which each rat servedas its own control In the same animals and within thesame sessions alcohol degraded Rep performances butleft Var intact Thus alcohol appears to interfere with atask that requires working memory as presumably is thecase for the Rep task but not with one that reinforcesvariability The memory-interfering effect of alcoholmight account for why alcohol also degrades radial-armmaze performance (Devenport 1983)

Alcohol also affects choices to vary or to repeat For ex-ample Gorka (1993) reinforced rats concurrently for re-peating and varying and monitored percentage of choicesas a function of time following injection of ethanol Inthe control sessions when the rats were injected withsaline the probabilities of varying and repeating remainedapproximately constant across the 45-min session witha slight bias in favor of varying After injection with 15gkg of ethanol however there was a significant increasein choices to vary at the beginning of the session andthen a later decrease to below 50 That is alcohol ledinitially to a markedly increased preference for varyingbut later to a preference for repeating (Figure 5) This re-search should be replicated and extended especially be-cause the contingencies ldquopressuredrdquo the animals to re-main at approximately 50 preference but thesuggestion is that alcoholrsquos effects on response variabil-ity may depend in part on the duration of its influence Amore general implication is that drugs may affect re-peated and variable operants differently and that effectsdepend on time following administration

SummaryReinforced variability may contribute to many types

of learning including operant shaping and acquisition ofcognitive and complex motor skills Similarly creativityand problem solving depend on variable responding andin these areas as well explicit reinforcement of variationsmay be functional Abnormal levels of variability aresometimes caused by psychopathologies and drugs andoperant variability may be useful in these areas in twoways first as a way of assessing the variability that is pos-sible and second to modify variability toward more nor-mal levels Studies show that individuals with autism andmild depression vary less than do controls but that ex-plicit reinforcement of variability moves the levels towardsnormalcy Other studies show that drugs such as alcoholmay affect repeated and variable operants differently

EXPLANATIONS

Two questions must be answered in order to explainoperant variability First what are the sources of vari-ability that permit its reinforcement Second how is itmaintained Related questions including how instancesbecome members of a set from which variations emergeand whether operant variability is a generalizable com-petency will also be considered

Origination From Nonoperant SourcesSkinner (1966) reasoned that operant responses orig-

inate from nonoperant sources with reinforcement con-tacting a behavior shaping and strengthening it onlyafter the behavior had occurred for reasons other than re-inforcement To put it simply a behavior must occur be-fore it can be reinforced (Balsam amp Silver 1994) I willconsider three initial sources of variable behavior en-dogenous variability noncontingent environmental eventsand extinction These appear to parallel evolutionaryprocesses and they too will be identified

Endogenous variability All physical phenomenavary and behavior is no exception This fact can providethe opportunity for reinforcement to select particularlevels of variation Examples of spontaneous variabilityinclude spontaneous alternations in mazes (Dember ampRichman 1989) nondifferentially reinforced variationsin operant responses (Guthrie amp Horton 1946) and ex-ploration (Archer amp Birke 1983) Endogenous muta-tions provide a parallel in genetics Base-pair changes inDNA molecules occur spontaneously intron-bearinggenes mix and match segments to encode new proteinsjumping genes introduce new DNA sequences and atthe level of the chromosome random crossings and as-sortment occur as a natural part of reproductive meiosisThese sources of random genetic change occur indepen-dently of external influences

Figure 5 The relative frequency of variations by rats as a func-tion of 5-min blocks of time since the beginning of the sessionThe open circles represent performance after the rats were in-jected with saline indicating a slight tendency to vary more thanrepeat but one that is approximately constant throughout thesession The filled circles represent performance after injectionwith 15 gkg ethanol indicating a much higher likelihood ofvarying soon after the alcohol is injected followed by a decreasein variations to below 50 and then a return to baseline at theend of the session Group averages are shown Data are fromGorka (1993)

690 NEURINGER

Noncontingent events Noncontingent events some-times referred to as chance accidents or luck also affectbehavior One happens upon a particular passage in a bookthat leads to a new thought one happens to sit next to aninteresting woman or man on an airplane which leads tomarriage (Bandura 1982) In science as well as in every-day life accidents or serendipity are an important part ofthe process of discovery (Beveridge 1957) In each of thecases presented above an accidental event leads to new be-haviors that are then strengthened by consequences Thesame is potentially true for variability itself Particular lev-els of variability might occur for adventitious reasons butthen become functionally related to consequences

That organisms are particularly sensitive to such unan-ticipated noncontingent events is shown by two basicprinciples of learning habituation and Pavlovian condi-tioning Organisms attend preferentially to unexpectedevents and stop attending or habituate to repeated onesThe conditioning of new Pavlovian responses dependsaccording to influential theories upon unexpected rein-forcing events (eg Rescorla amp Wagner 1972)

Noncontingent events such as X-rays and other pro-cesses that can act as mutagens play an important role ingenetic variation as well The particular genes affectedand the type of changes are random there is no neces-sary relationship or contingency between the externalevent and the affected gene However sensitivity to mu-tations is not random but has been selected presumablybecause it is functional Thus in both behavior and ge-netics sensitivity to noncontingent events is conserved

Extinction Extinction increases behavioral variabilityat least to some extent as was described above Distance inspace or time from the reinforcer does the same (Cherotet al 1996) A possible parallel at the genetic level in-volves high levels of deprivation at least in simple lifeforms For example when E coli bacteria are deprivedof nutrients unusually high mutation rates are observedThe exact mechanism by which such hypermutations areinduced is not known but they increase the likelihoodthat a population can survive such environmental insults(Moxon 1997) That is deprivation-induced variability isfunctional at the genetic level (The behavioral evidenceregarding long-term deprivation is less clear some stud-ies showing increased variability and others increasedstereotypy eg Carlton 1962 Senkowski Vogel amp Po-zulp 1978)

Endogenous variations external noncontingent eventsand withholding of reinforcement are three sources forthe variability that precedes control by reinforcementBut once control by reinforcement is established a dif-ferent question emerges What behavioral processes areinvolved when variability is maintained as an operantThe sources just discussed generate uncontrolled non-specific and often short-lived variability and cannot ex-plain the precisely controlled operant type

Operant Variability ProcessesThree processes are hypothesized to underlie operant

variability The first involves the use of random environ-

mental events such as the toss of a coin The second in-volves memory for responses such as when an animallearns not to return to previously visited locations Thethird will be referred to as an endogenous stochasticgenerator

Random environmental events Throughout historymany cultures have used random events to influencechoices and decisions such as throws of dice and randomselections of sticks cards bones or organs Today coinflips are used by a referee at the beginning of a game to de-cide which team kicks the ball a computerrsquos random-number generator is employed by scientists to avoid bi-ases in assigning subjects to experimental groups andaleatoric events are sometimes employed in modern artmusic and literature The Dice Man by Rhinehart (1998)provides a fictional example in which the protagonistbored with life writes a number of possible actions onslips of paper and then periodically selects one blindlyand behaves accordingly These examples show that randomexternal events may be used to avoid biases engender un-likely responses and break out of behavioral niches

Memory-based variability Many methods used tostudy operant variability differentially reinforce nonrepe-titions at least over the recent past Therefore one strat-egy that subjects can employ is to cycle through few re-sponses or sequences especially when the variabilitycontingencies are relatively permissive For exampleunder a lag 2 contingency cycling repeatedly throughthree sequences will result in 100 reinforcement Ani-mals and people sometimes do this apparently basingtheir current response on memory for or discriminativecontrol by the just emitted responses One such instancewas shown by Schoenfeld et al (1966) who rewardedrats for IRTs that fell into one of 10 interval classes onlyif the current IRT differed from the interval class of theprevious response The rats came to alternate betweenlong and short IRTs thus behaving systematically to meetthis lenient variability contingency Machado (1993)studied an analogous effect having pigeons peck L and Rkeys under a frequency-dependent variability contin-gency related to Shimprsquos (1967) procedure describedabove When the trial unit consisted of two responses thebirds developed memory-based repetitive sequencessuch as RRLLRRLL and thereby satisfied the variabilitycontingency When the trial unit was three responses inlength the birds did not develop the optimal fixed pat-tern of RRRLRLLLmdashwhich was too complexmdashbut in-stead developed ldquorandom-like behaviorrdquo (Machado1993 p 103) Thus a memory-based strategy was em-ployed when it was possible but when the memory loadbecame too high stochastic responding emerged A sim-ilar pattern was seen when variability of budgerigarsongs was reinforced (Manabe et al 1997) Under lag 1contingencies the birds sang mainly two songs underlag 2 they increased to three songs Only when the lagwas increased to 3 did song diversity increase apprecia-bly in one of the birds In radial-arm mazes as well whencontextual cues are available memory again appears toplay an important role leading to much lower probabil-

OPERANT VARIABILITY 691

ities of repetition than would be expected from a sto-chastic generator (McElroy amp Neuringer 1990)

Note that under lag schedules memory-based strate-gies result in higher reinforcement rates than respondingstochastically does The reason is that a stochastic gen-erator repeats by chance That animals and people oftendo not employ memory-based strategies probably indi-cates the difficulty of remembering long sequences andpossibly other advantages of unpredictable responding

If a memory strategy is the only way to satisfy vari-ability contingencies then it can be used even when thememory requirements are demanding as was shown byNeuringer and Voss (1993) Human subjects were trainedto generate on a computer keyboard a chaotic-like se-quence of responses that matched iterations of the logistic-difference equation

where Rn refers to the nth iteration in a series with R be-tween 00 and 10 and t is a constant between 10 and40 When the t parameter approaches 40 iterations ofthis equation result in very ldquonoisyrdquo sequences As withother chaotic processes however there is an orderlinessunderlying the surface noise In this case if responsesare autocorrelated with Rn drawn as a function of Rn21the resulting function is a parabola In the Neuringer andVoss (1993) study following each response college stu-dents were shown how close their predicted values wereto that from iterations of the logistic-difference model Inone experiment the students made digital predictions inanother they placed a pointer along a horizontal lineand this analogue response was then translated into dig-ital values for computation With training the studentsbecame increasingly adept at responding in chaotic-likefashionmdashthe studentsrsquo predictions matched closely theiterations of the logistic functionmdashand their autocorrela-tions became parabolic The left column in Figure 6 showsperformances by 4 subjects (rows) early in training thatis not well approximated by the logistic-differenceparabola and the right column shows performances at theend of training in which r 2 values were 72 99 98 andapproaching 10 How was this accomplished Sinceeach iteration in the logistic-difference sequence is basedon the prior outputmdashthat is the sequence is generated bya deterministic although nonlinear processmdashit was hy-pothesized that the human subjectsrsquo chaotic-like responseswere also based on memory for prior responses To put itsimply the subjects may have memorized a long series ofpairs in the form ldquoIf the previous response was value Athen the current response must be value Brdquo as Metzger(1994) Ward and West (1994) and Ward (2002) have sug-gested

To test whether subjects in fact relied on memory forprior responses Neuringer and Voss (2002) systemati-cally slowed responding by interposing pausesmdashthat isby requiring an IRT As IRT increased the difference be-tween the subjectsrsquo sequences and the modelrsquos chaotic

Rn n nt R R= -( )- -1 11

Figure 6 Autocorrelations (responses on trial n on the y-axis asa function of n21 on the x-axis) for 4 human subjects at the be-ginning (left column) and end (right column) of training inNeuringer and Voss (1993) Equations for the best-fitting curveand amount of variance accounted for (r2) are shown at the topof each panel To the extent that the circles (responses) approxi-mate the equation y = 40x 2 40x 2 the subjects were respondingin a way that matched the logistic-difference chaotic functionFrom ldquoApproximating Chaotic Behaviorrdquo by A Neuringer andC Voss 1993 Psychological Science 4 p 115 Copyright 1993 bythe American Psychological Society Reprinted with permission

692 NEURINGER

output increased and the underlying parabolic structurewas broken That is when pauses increased subjects wereless and less able to approximate the modelrsquos chaotic dis-tribution (see Figure 7) The important point here is thathighly variable responses which in this case are chaoticin nature can be based on memorial processes

Endogenous stochastic generator The evidence tobe reviewed next indicates that animals and people canrespond stochastically and that this competency is basedneither on chance environmental events nor on memoryfor prior events or responses It is impossible to provethat an endogenous stochastic process underlies operantvariability that would be analogous to proving the nullhypothesis However much evidence is consistent withthe stochastic hypothesis Included are the facts thatmemory limitations do not permit adequate respondingunder many variability contingencies that response dis-tributions match those expected from a stochastic gener-ator that changes in parameter values affect operantvariability as they would a stochastic generator and thatinterference with memory processes leaves stochasti-cally based operant variability unaffected As a wholethe evidence is consistent with an endogenous stochas-tic source of operant variability

Memory limitations One type of evidence is nega-tive in character and is based on the presumed inability ofmemory for prior responses or sequences to satisfy sometypes of variability contingencies Two examples weregiven above (Machado 1993 Manabe et al 1997) andanother was provided by a lag 50 contingency in Pageand Neuringer (1985) In the latter case the sequence in

the current trial (eight responses across L and R keys) wasrequired to differ from those in each of the previous 50 tri-als including at the beginning of a session the last 50trials from the previous session Pigeons responded wellunder these contingencies and in a way similar to that ofa stochastic model It is unlikely that pigeons can remem-ber each of their previous 50 eight-response sequences

Response slowing As was shown above for chaotic-like responding performances are often degraded by de-lays imposed between a stimulus and a response pre-sumably because of limitations in working or short-termmemory (Berryman Cumming amp Nevin 1963 Williams1971) Neuringer (1991) relied on this effect to test whethermemorial processes could explain the generation of vari-able sequences under lag schedules Two groups of ratswere reinforced for L and R leverpresses producing fourresponses per trial One group (Var) was trained under alag 5 contingency and the other (Rep) was trained to asingle sequence of responses LLRR It was assumedthat accurate Rep performance depended on workingmemory After performances had stabilized IRTs weresystematically varied by the imposition of timeouts of05 to 20 sec between consecutive responses in the se-quence During these forced pauses the chamber wasdark and responses were not counted toward completionof the sequence If the animals had used the previous re-sponse(s) as a discriminative cue for the current responsethen performances should have been degraded by theseinterpolated pauses

The results were that for the Rep group as the re-quired pauses increased from 0 to 6 sec percentages ofcorrect sequences were relatively unaffected but at pausesgreater than 6 sec probability of a correct LLRR se-quence fell sharply This pattern of results suggests in-volvement of working memory of prior responses withthe rats able to remember responses within a 6-sec win-dow The Var grouprsquos results were quite different Aspauses increased between 0 and 6 sec the rats were moreand more likely to satisfy the variability contingenciesand with pauses greater than 6 sec percent correct re-mained at a high and constant level Thus the Var grouprsquosresults were opposite to those of the Rep group and op-posite to those predicted from a memory-based strategySimilar results showing that operant variability increasesor is maintained as responding is slowed have been re-ported by Morris (1987) with pigeons and by Baddeley(1966) with random response generation by human sub-jects The results reported above concerning alcoholrsquos ef-fects on variability and repetition were also consistentwith those of Neuringer (1991)

Memory for prior responses may not have controlled theVar grouprsquos performance but why did variability increaseEach of three answers is consistent with a stochastic-generator interpretation Weiss (1964 1965) hypothe-sized that random responding requires that current re-sponses be independent of previous ones memory for orcontrol by prior responses interferes with such indepen-

Figure 7 Approximation to a chaotic sequence (given by delta)as a function of the imposed interresponse times Shown are theaverages of 5 subjects

OPERANT VARIABILITY 693

dence and therefore pauses would be expected to facil-itate variability A second possibility is that at shortIRTs animals tend to repeat responses on the sameoperandum Blough (1966) found this in pigeons and ex-cluded such double pecks from his analyses because theyappeared not to be under the control of reinforcement con-tingencies Morris (1987) also found a tendency for birdsto repeat when no interresponse timeouts were imposedA third hypothesis is that there were two contributors tothe observed variability One was a stochastic processcontrolled by reinforcement and the other elicited vari-ability generated by interposition of pauses The high vari-ability elicited as an effect of slowed responding is ageneral phenomenon supported in many other cases Ac-cording to this interpretation operant variability in the Vargroup was governed by a stochastic-based process operantrepetition in the Rep group was governed by a memory-based process and pauses elicited variability under bothcontingencies Thus Rep performance was interferedwith whereas Var was facilitated Each of these hypothe-ses is consistent with the conclusion that memory for (ordiscriminative control by) prior responses does not con-tribute to and possibly interferes with variable respond-ing when an organism is reinforced for variability andthis in turn is consistent with the stochastic-generatorhypothesis

Stochastic response distributions Other evidence iscorrelational Response distributions are similar to thosefrom a stochastic generator The data from two experi-ments already discussed are shown in Figure 1 In thesecases reinforcement of least frequent IRTs (Blough 1966)and of nonrepetitive choice sequences (Machado 1989)resulted in stochastic-like responding

Neuringer (1986) reported another striking case Highschool and college students generated sequences of 1sand 2s on a computer keyboard with each set of 100 re-sponses constituting one trial A baseline phase lastedfor 60 such trials for a total of 6000 responses duringwhich the students were asked to behave as randomly aspossible as if they were reporting the outcome of a se-ries of coin tosses As in the many previous studies ofthis sort the subjectsrsquo responses differed significantlyfrom those of a stochastic model (Brugger 1997 Wage-naar 1972) During the training phase the subjects re-ceived feedback which enabled them to compare their per-formances with that of a stochastic model first accordingto one statistical test and then according to another untilfollowing each trial feedback was provided from 10 dif-ferent statistics The results were that all of the subjectslearned to approximate the stochastic model accordingto the 10 statistics as well as some (but not all) other sta-tistical tests That is the distributions of the statisticswhich clearly did not match a stochastic model at the be-ginning of training approximated the stochastic modelat the end of it (see Figure 8)

The previous failures of other researchers to obtain ev-idence for random responding had led some to concludethat humans did not have the capacity to behave ran-

domly What would account for the difference betweenthe present positive finding and the many other negativeones Neuringerrsquos (1986) study was the first in which sto-chastic responding was explicitly reinforced in this casewith feedback from 10 statistical tests An analogy mighthelp to explain the difference in procedures and resultsImagine an experimenter walking around a city and ask-ing people if they know what a violin sounds like (analo-gous to asking people whether they know what a series ofcoin tosses or dice throws is like) Most would answeryes If the same individuals were then asked to play theviolin very few if any would succeed The experimentermight conclude after many replications with differentsubjects in different cities that people did not have thecapacity to play the violin The point of course is thatplaying the violin is a skill requiring training and prac-tice and the same may be true for responding randomlyIt is easy to vary but practice and reinforcement may benecessary to approximate a random distribution

Response number Page and Neuringer (1985) testedthe stochastic-generator hypothesis in another way byvarying number of responses per trial with pigeons work-ing under a lag 3 contingency The reasoning was that ifprevious responses served as a discriminative cue forwhat not to domdasha memory-based strategymdashthen perfor-mance should be degraded when the number of re-sponses was increased Eight responses per trial requiresubjects to remember more than four responses The sto-chastic hypothesis predicts the opposite outcome as isdemonstrated by the following example If each trialwere two responses in length with responses determinedby the toss of a coin then the probability that a given trialwould repeat the previous one would be 25 (There arefour possible sequences in the first trial RR RL LRand LL Thus the second trial has a 1-in-4 chance ofmatching the first) If a trial consists of four coin tossesthe probability of a repetition by chance is 0625 or 1 in16 Thus if subjects used a stochastic process to gener-ate Ls and Rs performances should improve with in-creasing responses per trial Results were precisely thosepredicted by the stochastic hypothesis and inconsistentwith a memory strategy Probability of meeting the con-tingencies and therefore probability of reinforcementincreased as responses per trial increased In otherwords eight-response trials were easier for pigeons thanfour-response trials

Response interference Neuringer and Voss (2002)attempted to distinguish between memorial and stochas-tic processes in yet another way They hypothesized thatwhereas chaotic responding was based on memory forprior responses (as was discussed above) responding inNeuringerrsquos (1986) study with feedback from 10 statisti-cal tests indicated involvement of an endogenous sto-chastic generator To show that human subjects could en-gage in both processes at will three college studentswere trained to generate chaotic sequences in one compo-nent of a multiple schedule as in Neuringer and Voss(1993) and to generate stochastic sequences in a sepa-

694 NEURINGER

rate component as in Neuringer (1986) The wordCHAOTICS was on the screen in the first component theword STOCHASTICS was on during the other and thesewords alternated throughout the session After the sub-jects succeeded at each of the component tasks test ses-sions were introduced in which memory for prior re-sponses was subject to interference in both the chaoticsand stochastics components as follows During the

chaotics component four separate chaotic sequenceswere cycled one after the other each cued by differentscreen colors The subjects attempted to approximate achaotic sequence in one color (eg light green) follow-ing which they would have to match a second chaotic se-quence (lavender screen color) and so on the fourchaotic contingencies following one another across thesession Although all four chaotic sequences used the

Figure 8 Performance by one subject at the end of training to respond stochastically in Neuringer(1986) Shown are comparisons on 6 different statistics between the subject (solid lines and Xs) andthe random model (dashed lines and closed circles) At the end of training the subjectrsquos perfor-mance did not differ statistically from that of the random model on these and four additional sta-tistics From ldquoCan people behave lsquorandomlyrsquo The role of feedbackrdquo by A Neuringer 1986 Jour-nal of Experimental Psychology General 115 p 71 Copyright 1986 by the American PsychologicalAssociation Reprinted with permission

OPERANT VARIABILITY 695

same logistic-difference equation that a model did theywere out of phase and independent If chaotic respond-ing is memory based then the four independent se-quences would be expected to interfere with one anotherthat is approximations to the four chaotic models wouldbe lower than when only a single chaotic output was re-quired as in Neuringer and Voss (1993) That was ex-actly the finding Performances by the three subjectswere signif icantly degraded during the four-segmentphase To put it simply chaotic responses interfered withone another

A different pattern of results was seen in the stochasticsportion of the experiment The same interference proce-dure was used with four independent stochastic sequencesrequired in the four different color conditions As inNeuringer (1986) subjects were required to satisfy eightdifferent statistical tests of randomness and to do this in-dependently in each of the four phases (If responses in anytwo or more of the four stochastic phases became corre-lated above a minimal level responses were considered tobe incorrect thereby prohibiting repetition of a single se-quence in two or more of the phases) For one subjectseven of eight statistics were closer to those of a randommodel during this interference phase than they were at theend of the original training with only a single sequenceand for a second subject all eight statistics were closerThe third subject showed closer-to-random approximationunder the four-segment condition on only two of the eightstatistics but most of her changes were also in the directionof a random generator For example if a random generatoremits equal numbers of ldquouprdquo and ldquodownrdquo sequences andthis subject in the baseline phase had emitted only 47ldquouprdquo sequences then during the four-segment tests shecame to emit ldquouprdquo 55 of the time She changed in the di-rection of the random model but overshot the mark Thuswhereas interference degraded chaotic performance sig-nificantly in all three subjects it improved stochastic per-formances in at least two of them These results are con-sistent with the hypothesis that a memory-based processcontrols chaotic variability and that a stochastic processnot dependent on memory controls stochastic variability

Summary of Tripartite TheoryThe following three processes constitute a tripartite

theory of operant variability attention to random events inthe environment (coin toss) memorial processes (chaosradial maze) and stochastic processes (response se-quences and random-number generation with feedback)Random environmental events assure unpredictabilitymdashfor example in making unbiased selections Remember-ing what was done previously will enable one to avoidrepetitions to vary the students called upon in class tovary the locations when looking for a hidden object orto try different approaches when attempting to solve aproblem External aids such as notes to oneself or sci-entific records help to enhance memory capacities

An endogenous stochastic generator provides the thirdand most controversial source of behavioral variability

When an opponent is attempting to predict your behav-ior in a game or at war it may be functional to behavestochastically Similarly operant conditioning of new re-sponses may depend on stochastic variations It is im-possible to prove stochasticity some presently unknowndeterministic system may ultimately explain the resultsHowever the evidence is consistent with the stochastichypothesis and inconsistent with the alternatives For ex-ample memory is not sufficiently powerful to explain alloperant variability memory-interference proceduresleave stochastic responding intact and behaviors corre-late well with those of stochastic models

Both genetics and reinforcement play important rolesin all three processes Reinforcement is always involvedin that learning provides information about when whatwhere and how much to vary Genetic influences areshown in that not all responses are possible and if theyare possible they are not all equiprobable (eg as inNeuringer et al 2001) Stochastic responding may be bi-ased due to experiences and knowledge that is biasesmay be established by reinforcement contingencies or bythe interface between the organismrsquos propensities (or be-havior systems see Timberlake amp Lucas 1989) and theenvironment In the case of random-number generationby human subjects biases may be due to misconceptionsconcerning what ldquorandomrdquo means However some bi-ases can be overcome through feedback and instructions(Lopes amp Oden 1987 Neuringer 1986)

The importance of variations for evolutionary changeis supported by the many sources of genetic variabilityand by the fact that genetic variability is conserved Asimilar argument holds for operant variability That thereare at least three independent sources of operant varia-tions as well as other elicitors such as environmentalnoise and response extinction supports its importancefor normal adaptive functioning

Structure of VariabilityEvery operant ldquoresponserdquo is actually an instance of a

class with each member of the class exerting the sameeffect on the environment (Skinner 1959) When a rat isreinforced for pressing a lever for example the class isdefined by closure of a microswitch and any way of ac-complishing the closure whether with the left paw theright paw or the mouth is as effective in producing re-inforcement as any other If the contingencies of rein-forcement so specify however the response class can beredefined and circumscribed which is another charac-teristic of the operant For example if only right-pawpresses are acceptable members of the reinforced classresponding will tend to occur only with the right paw(Stokes 1995) Of course the capabilities of the organismplay a role for example including ldquotail pressesrdquo in the op-erant class does not mean that the lever would be or couldbe pressed by the ratrsquos tail The structure of the operantclass therefore emerges from the current reinforcementcontingencies and the organismrsquos capability derivedfrom genetics development and prior experiences

696 NEURINGER

Operant variability has structure as well and as for alloperants that structure is due partly to the organism it-self and partly to the reinforcement contingencies Theresearch described throughout this article has providedexamples of the structure of variability generally in theform of sets of four or eight responses across L and Roperanda But the structure of variations can be speci-f ied more precisely For example Mook and her col-leagues (Mook et al 1993 Mook amp Neuringer 1994)reinforced rats for varying four response sequences acrossL and R levers under a lag 1 contingency Unlike in pre-vious studies however the acceptable class was furtherconstrained so that only sequences beginning with two Rresponses (RRLL RRLR RRRL and RRRR) were rein-forced The rats learned to vary their sequences withinthe imposed limitations that is they generally begantheir sequences with two R responses and varied the se-quences within those limitations Similar results wereobtained in the context of a radial-arm maze in whichonly a subset of arms was baited

The precision with which reinforcement contingen-cies can specify class was highlighted in research per-formed by Burns (1997) and Ross and Neuringer (2002)In the latter case college students drew rectangles on acomputer screen with the computerrsquos mouse Each rec-tangle constituted one trial (the screen cleared at the endof each trial) with reinforcement (points) depending onsimultaneous variations along three dimensions of therectangle location on the screen area of the rectangleand shape with the latter defined as the ratio of the rec-tanglersquos height to its width Although the subjects weregiven no instruction other than to ldquogain pointsrdquo theylearned to vary along all three dimensions with levels ofvariability significantly higher than those of yoked controlsubjects who were reinforced independently of variations

A second part of the Ross and Neuringer (2002) studydemonstrated even greater precision of control To gainpoints three new groups of subjects were required to re-peat along one of the dimensions while simultaneouslyvarying along the other two One group was required torepeatedly draw rectangles of approximately the samesize while simultaneously varying location and shapeanother group had to repeat the shape while varying areaand location and the third to repeat location while vary-ing size and shape Verbal instructions were minimal asthey were in the first part of the study Subjects learnedto satisfy these contingencies as well varying two at-tributes while simultaneously repeating one and signif-icant differences were obtained rapidly within 300 trials

These demonstrations raise questions as to how binaryfeedbackmdashreinforcement presented or notmdashcan simul-taneously control independent behavioral dimensionsThe issue is similar to that discussed above concerningsimultaneous reinforcement of variations and selectionsIt is similar also to those issues faced by researchers at-tempting to understand how the nervous system strength-ens particular synapses when many different neurons firesimultaneously and to ldquoallocation of creditrdquo in artificial

learning models For present purposes suffice it to notethe exquisite precision with which reinforcement controlsvariability and repetition

GeneralizationDo animals and people learn to ldquovaryrdquo implying a

generalizable skill or do they learn a specific distributionof specific responses across particular operanda Thequestion of generalization has been asked of many otherbehavioral competencies such as imitation When chil-dren learn to imitate do they learn stimulusndashresponseassociations (if mother moves her right hand the childlearns to move his right hand) or do they learn moregenerally to ldquoimitaterdquo Evidence indicates that reinforce-ment of imitative instances leads to generalized imitation(see eg Baer amp Sherman 1964 Bandura 1986) A sim-ilar question has been raised when pigeons rats or chil-dren learn to choose a stimulus that matches a previouslyshown sample For example if a pigeon is shown a reddisk as a sample it must then choose red when given achoice between that and blue Does the bird learn to matchIn this case the answer appears to depend on the train-ing procedures If trained with few stimuli such as redgreen and blue then the pigeon learns simple associa-tions (if red sample choose red if blue sample chooseblue) but if hundreds of different stimuli (pictures) areemployed during training then generalized ldquomatchingrdquois learned (Wright 1997) In each of these examples thelearning of a general competency or strategy was shownby transfer-of-training tests What was learned with oneset of stimuli or in one situation transferred to new stim-uli and situations We turn now to the question of whetherlearning to vary transfers

After the subjects in Maltzmanrsquos (1960) experimentdescribed above learned to generate remote verbal as-sociates they were asked to devise unusual uses of com-mon objects In comparison with those in the controlgroup these subjects generated significantly more un-usual uses indicating transfer of training to a differenttask However transfer was not observed in the oppositedirection from unusual uses to verbal associates

Holman et al (1977) using a procedure similar to thatof the Goetz and Baer (1973) research described aboverewarded preschool children verbally for novel drawingsand found increased novelty Unrewarded block construc-tions were also monitored and weak evidence was ob-tained for transfer of novel responding from rewardeddrawing to the unrewarded constructions In a second ex-periment preschoolers engaged in four independent taskspainting pen drawing wood block construction and Legoconstruction When the children were reinforced for vary-ing their pen drawings the novelty of paintings increasedalong with the pen drawings indicating transfer from onetask to the other However transfer to the topographicallydifferent block- or Lego-building tasks was not observed

Eisenberger and Armeli (1997) gave monetary rewardsto one group of fifth- and sixth-grade students the vari-ability group for describing unusual uses of common ob-

OPERANT VARIABILITY 697

jects A second group the repeat group received thesame rewards for describing common uses of the sameobjects In a later test all of the children were asked todraw pictures incorporating a circle and the group that hadpreviously been reinforced for varied responses drewmore unusual (or lower frequency) pictures than did therepeat group Thus there was transfer of novelty trainingfrom one domain to another with additional studies in theseries showing similar effects (Eisenberger Armeli ampPretz 1998 Eisenberger Haskins amp Gambleton 1999Eisenberger amp Selbst 1994) However Eisenberger et al(1999) failed to replicate the transfer-of-training effectThe variability and repeat groups responded similarly dur-ing the test phase In this study separate experimenterstrained and tested the subjects thereby avoiding possibleexperimenter-expectancy effects whereas the same ex-perimenter did both training and testing in Eisenbergerrsquosother studies Across the series of experiments there weremany other cases in which transfer was not observed andmore research is necessary to explain why

In Lee et alrsquos (in press) study with autistic individualsdescribed above after variable answers were trained toquestions verbal variability transferred to new experi-menters and new contexts

Arnesenrsquos (2000) results described above are consis-tent with transfer of training in an animal model Rats inthe Var group were trained to interact variably with oneset of objects When later tested in a new situation withnew objects the Var animals showed (in comparisonwith controls) higher levels of exploration manipulationof the novel objects and discovery of hidden reinforcers

Much of the evidence reviewed in this section there-fore indicates transfer of training and supports a generalldquovaryrdquo competency However there remain unresolvedquestions the most important of which is the extent towhich transfer occurs Note also a methodological pointMany of the studies tested whether transfer occurs whenthe contingencies are neutral with respect to the to-be-transferred behavior Holman et al (1977) is a case inpoint Variations were reinforced in one domain draw-ing and the question was whether variations wouldoccur in another domainmdashblock constructionmdashin whichreinforcement did not depend on varying A negativef indingmdashin this case absence of transfermdashindicatesnothing about the ability to generalize but only aboutthe tendency to do so given the absence of a reason (orcontingency) A better test would be to compare experi-mental and control performances when variability in thenew task is in fact functional as in Arnesen (2000)

SummaryBefore variability can be reinforced it must occur for

reasons other than reinforcement Some of the reasonsinclude variability inherent in all behavioral phenomenaldquoaccidentalrdquo environmental events that are unrelated toparticular behaviors and withholding of reinforcementTransfer from other situations may serve a similar functionFor example if an organism can discriminate shaping-

of-new-response periods from others then it may learngenerally to ldquovaryrdquo during shaping episodes After rein-forcement makes initial contact with response variationsthere are at least three behavioral strategies that can beemployed in order to maintain functionally high levelsof variability respond based on external random eventssuch as the toss of a coin avoid repetitions by remem-bering what responses had previously been made and re-spond stochastically Evidence for an endogenous sto-chastic process was seen in the similarity of responsedistributions to those generated by stochastic sourcesand by findings that interfering with memory for priorresponses either improves or leaves unaffected stochas-tic behaviors Stochastic responding as well as othertypes of variability-generating strategies depend uponcontingencies to define the required level of variabilityand the set from which variations emerge

IMPLICATIONS

One implication of reinforced variability is that oper-ant responses may differ in the extent to which their vari-ability is affected by reinforcement contingenciesmdashthatis operants may be ordered along a continuum of sensi-tivity of variability to reinforcement Another is that re-inforced variability causes us to question the generalityof some basic goals of behavioral psychology includingprediction and control A third is that reinforced vari-ability may help us to provide a scientifically useful wayto describe voluntary action In particular the variabilityof all voluntary behaviors can readily be changed mo-ment to moment by discriminative stimuli and conse-quences Implications for everyday life also will be noted

Operant ContinuumAs was indicated above the operant is composed of

instances that is the operant is a class of many differentresponses all functionally equivalent but often topo-graphically distinct Skinner suggested and I hypothe-size that instances within the class emerge stochastically(see Moxley 1997 for a discussion of Skinnerrsquos posi-tion) Although reinforcement contingencies and contextinfluence the individual probabilities and their distribu-tions emission is stochastic nonetheless The particularmember of the operant class generally cannot be predictedwith a high degree of certainty For example a particu-lar rat may be more likely to press the lever with its leftpaw than its right but left- and right-paw responses areemitted probabilistically This class-nature of the operantsuggests one way to conceptualize operant variability interms of size of class and within-classes probability dis-tributions High variability implies large classes equalprobability distributions or both low variability impliesthe opposite

Variability may be modified more readily in some op-erants than in others implying that operant variability isnot an all-or-none phenomenon but rather a continuumwith reinforced variations in variability (RVV) differing

698 NEURINGER

for different operants RVV might best be understood byanalogy to the concepts of speed and acceleration Bothvariability and speed range from low to high RVV isanalogous to acceleration the rate of change in speedHighly sensitive RVV is indicated by the possibility ofreinforcing many different levels of variability from rep-etition to random-like responding and by the fact thatchanges in level occur readily and rapidly

Different operants manifest differences in RVV Forexample variability of chain pulls changed more thanthat of leverpresses when variability was reinforced inrats (Morgan amp Neuringer 1990) Some operant-like re-sponses may show relatively low RVV For example fir-ing rates increase in single hippocampal nerve cellswhen dopaminergic substances are contingent upon theresponse (Stein Xue amp Belluzzi 1994) suggesting thatnerve f iring is controlled by reinforcement To datehowever there is no evidence that variability of nervefiring is similarly controlled Other cases might includereinforcement of nondiscriminable (subconscious) re-sponses such as Hefferline and Pererarsquos (1963) reinforce-ment of subaware thumb twitches and biofeedback ofmuscle and cardiovascular activity In these cases rein-forcement may modify performance but it is not knownwhether variability is also sensitive to reinforcement andif so to what degree Human verbal variability may bethe most readily manipulable of all behaviors the rangeof predictable to unpredictable verbal responses beingimmense with extraordinary sensitivity to context andconsequence RVV may also differ across organisms (per-haps from simple to complex) individuals (perhaps asan index of intelligence or learning ability) and physio-logical and psychological states (WKY vs SHR depres-sion and autism)

RVV may help to distinguish operant responses fromother superficially similar cases of selection by conse-quences That is the ldquooperantrdquo may also best be concep-tualized as a continuum with RVV an important deter-miner of position along the operant continuum Oneexample of selection by consequences but low ldquooperant-nessrdquo is genetic change Variability is essential for evolu-tionary change of course and levels of genetic variabil-ity are sometimes affected by environmental conditions(Moxon 1997) but there are few examples of levels ofvariability being influenced by selection pressures Thatis the source of much genetic variability is thought to berandom not influenced by current pressures and needsThus although selected by consequences genetic changemay be low along the operant continuum Other exam-ples of operant-like phenomena for which variability although important may be relatively insensitive to con-sequencesmdashvariability in these cases is elicited not se-lectedmdashinclude the development of the central nervoussystem and immune system responses (Hull et al 2001)In both of these cases change depends upon selection byconsequences but it is not clear whether variability canalso be selected Most commonly studied operant re-sponses on the other hand show high sensitivity of vari-

ability to reinforcing events But to reiterate the mainpoint of this section operants differ in terms of RVV andtherefore in terms of their ldquooperantnessrdquo from low sensi-tivity (possibly responses of single nerve cells) to extra-ordinarily high sensitivity (verbal behavior) The operantnature of variability as well as that of every other oper-ant dimension is best conceptualized as a continuum

Prediction and ControlSmith (1992) described Skinnerian operant condition-

ing theory as Baconian in nature a primary goal of Skin-nerian behaviorists being to control behavior and hecontrasted this with the more Aristotelian goal of de-scription Skinner and his students (eg Zuriff 1985) in-deed emphasized prediction and control Many of thestudies reviewed above however involved reinforce-ment procedures that increase variability and conse-quently decrease predictability It is not only ignorancetherefore that restricts prediction and control as Skin-ner suggested but reinforcement itself Operant meth-ods and theories when applied to variability lead togreater emphasis on description with control over pre-cise response characteristics possible under some butnot all conditions

In summary an operant is a class of responses underthe control of contingencies of reinforcement The classcan be controlled and predicted given knowledge ofconditioning history Indeed the size of the class andpossibly the distribution of within-classes probabilities orlevel of variability can also be controlled and predictedBut to the extent that instances emerge stochastically aparticular instance cannot be predicted The instant re-sponse can be explained as a member of a particularclass but prediction of the instance will be diff icultsometimes to the extreme This point will be developedin the next section

Voluntary BehaviorI hypothesize that voluntary behaviors are operants for

which RVVs are highly sensitive to influence by conse-quences One way to test this hypothesis would be togather empirical data concerning how voluntary versusinvoluntary actions are classified by human observers Asfar as I know there are no psychophysical data concern-ing this question but the hypothesis implies that highsensitivity of variability to operant contingencies RVVswill be present in all actions perceived to be voluntary

This operant-variability theory of voluntary action isbased on the supposition of two attributes First volun-tary acts must be potentially explainable (given suff i-cient knowledge) on the basis of genetics conditioninghistories current contingencies anticipated conse-quences intentions goals and the like That is volun-tary behaviors are lawful The second attribute is that thevoluntary act must be potentially unpredictable That isvoluntary acts are ldquofreerdquo of control by identifiable stim-uli If behavior is perceived necessarily to result frommdashto be predicted on the basis ofmdashexternal events includ-

OPERANT VARIABILITY 699

ing reinforcement contingencies it will not be perceivedas voluntary ldquoVoluntaryrdquo implies potential indepen-dence from stimulusndashresponse or responsendashreinforcerdetermination According to this theory both explain-ability and unpredictability are necessary attributes ofvoluntary acts

How though can behaviors be simultaneously ex-plainable and unpredictable Reinforcement contingen-cies play two essential roles Reinforcement (togetherwith genes prior experiences and current environment)helps to determine the set of possible behaviors fromwhich a voluntary act emergesmdashthat is the operant classReinforcement (together with the other contributorsspecified above) also helps to determine the level of vari-ability within the class Thus both instances comprisinga class and level of variability within that class can bepredictedmdashagain given sufficient knowledge Howeverthe individual response is at least sometimes (or relatively)unpredictable The reason is that the operant class fromwhich the response stochastically emerges is sometimesextremely large (eg ldquoname a wordrdquo or ldquodo somethingrdquo)and the contingencies sometimes call for equiprobablestochastic emergence (eg ldquofool merdquo) In the latter casea voluntary act may appear to be foolish meaninglessor unwarranted but if it is voluntary it can potentially beexplained (For example in response to the questionldquoWhy did you do that silly actrdquo the answer could beldquoJust to show that you canrsquot predict my behaviorrdquo seeScriven 1965) In these cases prediction of instancesfalls to very low levels indeed and I hypothesize evenan all-knowing observer will be unable to predict thenext response at greater-than-chance levels which aregiven by the size of the operant class and by within-classesprobability distribution Even within small classes (egin the game rockndashscissorsndashpaper in which unpredictabil-ity among the three possible responses is functional) be-havioral stochasticity may lead to unpredictability of theinstant response Note however that although voluntarybehaviors are sometimes unpredictable the opposite isoften the case When contingencies call for highly con-strained particular responses predictions can easily bemade For example if a police officer asks your nameyour response can most likely be predicted Voluntary re-sponses are therefore described as potentially unpre-dictable with variability contingencies engendering dif-ferent levels of unpredictability

According to this theory behaviors low in the sensitivity-of-variability dimension are described as relatively in-voluntary and those that are high in said dimension areperceived as voluntary Low RVV may be shown at anylevel along the variability continuum For example al-ways responding in the same unchanging way will beperceived as involuntary but so will always respondingrandomly (The coin does not voluntarily come up withheads or tails) This point is essential More than 2000years ago Epicurus argued that occasional randomswerves of atoms provided the source of voluntary be-havior in an otherwise determined universe but al-

though this conjecture was prescient it was not quitecorrect ldquoFree-willedrdquo or voluntary action whether it isbased on random swerves or on quantum mechanics isnot explained by variability per se but by functionalchanges in levels of variabilitymdashthat is by rapid andprecise changes in response to environmental demandsResponses can be placed along a voluntaryndashinvoluntarycontinuum isomorphic to the operantndashrespondent con-tinuum with voluntary behavior being functional pre-cisely in the same way as operant behavior is ThusSkinnerrsquos (1974) suggestion that voluntary behavior isoperant behavior was almost correct Only those oper-ants for which variability is highly sensitive to conse-quences will be perceived as voluntary

One additional example might help Operant classes(sets of potential responses) are created ldquoon linerdquo as we goabout our daily activities As we walk from car to officewe tend to walk in a characteristic way take the samepath with tightly constrained operant classes followingone after the other in rapid succession Class sizes aresmall and within-classes probability distributions arehighly peaked Predictions of next responses by a knowl-edgeable observer can readily be made However classsizes can be increased and probability distributions flat-tened we can walk in quite strange ways indeed dancealong a new path sing hand candy to passers-by Strangebehavior indeed but made possible by the ability of op-erant sets to be modified moment to moment not only interms of class definition but in terms of class size lev-els of within-classes variations and so on The same istrue for conversation Moment-to-moment changes inverbal class definition size and predictability of within-classes instances are hallmarks of voluntary behavior

GenerativityRelated to the definition of the operant and to our discus-

sion of voluntary action is that fact that operant behaviormdashleverpressing no less than languagemdashis by its naturegenerative This claim follows from the functional na-ture of the class of responses (all members of the classare effective in producing an outcome) and the stochas-tic emission of instances From these flow never-before-emitted instances patterns and sequences (see Epstein1996) Operants especially those high on the RVV con-tinuum are inherently generative

Self-ExperimentationVariations systematic or otherwise are a necessary part

of scientific work Neuringer (1981) and Roberts and Neu-ringer (1998) described how variations can be brought to bear explicitly on everyday life in the form of self-experiments Individuals vary aspects of their lives suchas the foods they eat social contacts and methods ofmemorizing they keep records concerning the effects ofthese manipulations and form hypotheses concerningcontrolling variables Using self-experimental methodsindividuals can learn how what when where and howmuch to vary Explicitly varying onersquos everyday behav-

700 NEURINGER

iors and thoughts may help to engender what Langer(1989 2000) has referred to as mindful livingmdashthat isnonautomatic voluntary engagement with beneficialeffects in different areas including learning The impor-tant point here is that variability can be voluntarily cho-sen maintained and modified in everyday life and thatexplicit variations may have important consequences

Implications for SocietyConsideration should be given to the importance of re-

inforcement contingencies to support variability in so-cial milieus A basic point made throughout the presentarticle is that merely permitting behavior to vary doesnot necessarily engender or maximize variability (as isshown by Yoke and Prob schedules) Analogously merelypermitting varied social behavior as in a free or laissez-faire society may not suffice to produce the desired di-versity If the results shown for individual organismscan provide useful information for groups (see eg Grottamp Neuringer 1974) then when variations are desiredsocieties might reward individuals for varying not sim-ply permit it

SummaryOperants differ in the extent to which reinforcers in-

fluence levels of variation When variability is highlysensitive to reinforcement the operant may be perceivedto be voluntary Voluntary responses have two attributesThey are functional implying that the class of responsesmdashthe operantmdashcan be predicted given sufficient knowl-edge But instances emerge stochastically implyingsome degree of unpredictability Because reinforcementof variability can generate extremely large classes withstochastic emergence of instances it suggests limitationsof the two basic goals of Skinnerian behavioral theorymdashnamely prediction and control Operant variability also hasimplications for self-experimentation and for society

RELATED AREAS

Operant variability may be important in many otherareas and I will briefly review some of these

Negative reinforcement To my knowledge there havebeen no published studies in which variability is nega-tively reinforced High variability might sometimesavoid or reduce a punisher such as loss of goods ormoney but at other times low variability could be func-tional Effects of positive and negative reinforcement onvariability may have different evolutionary bases onederived from increasing probabilities of locating sourcesof food or mates the other from avoiding confrontationsor predation (Driver amp Humphries 1988) One questionconcerns how quickly variability can be changed by neg-ative as opposed to positive reinforcement There isconsiderable evidence that loss functions differ fromgain functions (Kahneman amp Tversky 1984) possiblyindicating that positive reinforcement-of-variabilityfunctions will differ from negative ones as well

Resistance to change In some of the studies describedabove variability appeared to be more resistant tochange than repetitions were for example alcohol hadlittle effect on variable operant responses but interferedwith repeated ones Doughty and Lattal (2001) tested thegenerality of these f indings using four-response se-quences with pigeons a threshold variability contin-gency in the Var component of a multiple schedule andLRLR in the Rep component Doughty and Lattal chal-lenged stability of responding by prefeeding the birds inone experiment and providing reinforcers independentlyof responding in the other (see Nevin 1992) Operantvariability was again more resistant to change than wasoperant repetition Thus reinforcement of variabilitymight increase the stability of responding (that is its re-sistance to change) a seemingly paradoxical findingHowever Mechner (1992) suggested that this is the casefor complex skills such as piano performance Varyingpractice routines will make the performance relativelyinsensitive to unforeseen changes in the environmentCare must be taken however when variation is comparedwith repetition because they have different measuresAlso the results reported by Doughty and Lattal werebased on the performance of three pigeons and were notconsistent throughout the experiment The hypothesiswarrants further study

Attraction In some bird species complex male songsattract females (for a review see Catchpole amp Slater1995) There is evidence that male song complexity isunder the discriminative stimulus control of a receptive fe-male (Searcy amp Yasukawa 1990) That explicit reinforce-ment may play a role in such song complexity is suggestedby the Manabe et al (1997) study described above An in-teresting question for further research is the extent towhich operant contingencies contribute to the attractioncreated by behavioral variability in these and other species

Discrimination of variability Animals and people canuse the variability of a stimulus array or entropy as a dis-criminative stimulus (Young amp Wasserman 2001) Whatis the relationship between perception of variability onthe one hand and emission of variable responses on theother Do individuals attempt to match their own levelsof variation to those of others in their environments whichwould involve both perception and emission How mightdiscrimination of onersquos own level of variability influenceonersquos ability to modify such levels and do influences thatobscure discriminations therefore affect ability to change(see eg van der Meer van der Weel amp Lee 1995)Hunziker et al (2002) report that although variabilitywas successfully reinforced their human subjects couldnot correctly describe the contingencies of reinforce-ment a finding similar to those of informal observationsin other studies (eg Neuringer Deiss amp Imig 2000)but whether subjects could discriminate variability levelsrather than contingencies was not studied

Attention and awareness When attention is subject tointerference or is withdrawn from the task at hand abilityto vary is degraded (eg Baddeley Emslie Kolodny amp

OPERANT VARIABILITY 701

Duncan 1998 Evans amp Graham 1980 Neuringer amp Voss2002) It is unclear at present how attention influencesthe different processes underlying operant variabilityand in particular memorial and stochastic processesRich areas for additional study are the relationships be-tween attention awareness and consciousness on theone hand and operant variability on the other Voluntaryaction and consciousness are often related (see egLibet 1999)

Concepts Concepts are thought by many to be sets ofinstances (much like operant sets) whose structures aregraded meaning that there are more or less typical in-stances of such concepts as birds (robin is typical ostrichis not) Barsalou (1987) showed that the gradation ofthese cognitive structures is manipulable or as he putsit not stable A number of variables influence structureincluding context individual differences and points ofview Barsalou argued that these structures are createdas the need arisesmdashthat is on line by the thinking indi-vidual The structure of concepts may parallel that of op-erants and in both cases levels of within-structures di-versity or variability may be modified by consequencesOperant variability may be found in the cognitive as wellas in the behavioral domain

Play and humor Highly variable behaviors are ob-served when animals and people especially children en-gage in play and humor (see eg Fagen 1981) The re-sult of play may be acquisition of new responses (seeSutton-Smith 1975) Although not currently needed re-sponses learned during play may provide the source ofvariations for successful shaping of behaviors at anothertime This is quite analogous to the role of conserved ge-netic variants which are passed along to progeny but donot currently serve any identifiable role Play may alsoinvolve learning how to vary operantly

Reinforcement by variability The self-maintainingnature of play suggests that engaging in variable behav-iors may itself be reinforcing Indirect support comesfrom findings that varied stimulation can serve as a re-inforcer and that stimulus repetition becomes aversive(Fiske amp Maddi 1961) Possibly also related are findingsthat varying reinforcers and permitting choice among re-inforcers increase their value (Bowman Piazza FisherHagopian amp Kogan 1997) and that permitting choicesamong varied responses is also preferred to demandingfixed responses (Catania 1980)

Foraging Two issues regarding variability are presentin the foraging literature One has to do with risk prefer-ence versus avoidance (see eg Shettleworth 1998) Asecond has to do with search strategies Most reportedcases of foraging involve systematic search patterns(Stephens amp Krebs 1986) However search patterns aresometimes stochastic or highly variable (see eg Peter-son 1997) One question is whether some species can varybetween these two strategies and the challenge is to iden-tify the pressures leading to and the advantages of each

Animal training Wilkes (1996) has applied operantvariability techniques in developing training proceduresfor pet dogs More generally reinforcement of variabil-

ity may enable the gaining of control over behaviors ofanimals and as has been suggested above may be an im-portant part of the process of shaping

Models of learning Reinforced variations in variabil-ity may be important when human and animal learningis being modeled whether the models are mathematicalin form computer based or machine based (Blumberget al 2002 Brooks 1999 Sutton amp Barto 1998)

CONCLUDING COMMENT

Genes interact with environments to produce diverseanatomical physiological and behavioral phenotypesGenes and environments also provide for another level ofdiversitymdashnamely operant variability There are manysources of operant variability including random environ-mental events memory for past responses and stochasticgeneration each of which ultimately derives from ge-netic and environmental influences To the extent thatoperant variability is based on a stochastic source how-ever the behavioral results cannot be predicted fromknowledge of these influences That is organisms haveevolved to behave unpredictably The positing of sto-chasticity to explain behavior has a history extending atleast as far back as Epicurus with major contributors in-cluding Gustav Fechner (see Heidelberger 1987) EgonBrunswik (see Gigerenzer 1987) Skinner (1959) andSuppes (1984) Knowledge of the stochastic nature ofthe operant and of how it is controlled leads to a worldview that differs from genetic and environmental deter-minism with roots in the works of the scholars just men-tioned At the scientific level this view leads to the ex-perimental analysis of consequence-influenced variabilityAt the philosophical level it leads to explanations of vol-untary action At the personal level it can lead to change

REFERENCES

Amabile T M (1983) The social psychology of creativity New YorkSpringer-Verlag

Antonitis J J (1951) Response variability in the white rat duringconditioning extinction and reconditioning Journal of Experimen-tal Psychology 42 273-281

Archer J amp Birke L (1983) Exploration in animals and humansCambridge Van Nostrand Reinhold

Arnesen E M (2000) Reinforcement of object manipulation in-creases discovery Unpublished undergraduate thesis Reed College

Baddeley A D (1966) The capacity for generating information byrandomization Quarterly Journal of Experimental Psychology 18119-129

Baddeley A [D] Emslie H Kolodny J amp Duncan J (1998)Random generation and the executive control of working memoryQuarterly Journal of Experimental Psychology 51A 819-852

Baer D M amp Sherman J A (1964) Reinforcement control of gen-eralized imitation in young children Journal of Experimental ChildPsychology 1 37-49

Baker M J amp Koegel R L (1999) Transforming children withautismrsquos ritualistic behaviors into positive play interactions with sib-lings Paper presented at the meeting of the California Association ofBehavior Analysis San Francisco

Balsam P D Deich J D Ohyama T amp Stokes P D (1998) Ori-gins of new behavior In W OrsquoDonohue (Ed) Learning and behav-ior therapy (pp 403-420) Boston Allyn amp Bacon

Balsam P D Paterniti A Zechowy K amp Stokes P D (2002)