What Does Exercise-Based AssessmentReally Mean?

FILIP LIEVENSGhent University

My commentary addresses Lance’s (2008)recommendation to reorient assessmentcenter (AC) practice away from dimensionstoward exercise-based assessment. As exer-cise-based assessment is dealt with only ingeneral terms in Lance’s article, I aim todelineate what exercise-based assessmentreally means. Two points are made. First, Iargue that taking dimensions away from ACsdoes not mean that assessee behavior is nolonger determined by latent traits becausebehavior is inherently trait determined. Sec-ond, I elaborate on the practical and researchimplications of exercise-based assessmentbecause these implications are underdevel-oped in Lance.

Exercise-Based Assessment

Theoretical underpinnings. Conceptu-ally, it is crucial to make a distinctionbetween trait expression (on the part of can-didates, employees, etc.) and trait evaluation(on the part of assessors, raters, etc.). In otherfields, examples of this distinction areabounding. For example, in performance

appraisal, one might rate employees on theiraccomplishment of specific objectivesinstead of on dimensions (as is the case inmanagement by objectives). However, atthe same time, it is still acknowledged thatin day-to-day interactions, employees mightdemonstrate behavior that is determined bytheir standing on traits and activated by situ-ational characteristics. By the same token,exercise-based AC practices might go handin hand with acknowledging that candidatebehavior is trait determined. The only differ-ence is that in exercise-based assessment,the traits under consideration (the AC dimen-sions) are no longer rated. Thus, eliminatingdimensions as rating tools from ACs does notimply that candidate behavior is no longertrait determined. In other words, the notionsof assessors rating dimensions in an AC andcandidates demonstrating trait-determinedbehavior are independent from each other.

Let me clarify that ‘‘trait determined’’doesnot imply stability in candidate behavior.This is well reflected in a recent interac-tionist theory such as trait activation theory(Lievens, Chasteen, Day, & Christiansen,2006; Tett & Burnett, 2003). Trait activationtheory focuses on the person–situationinteraction to explain behavior based onresponses to trait-relevant cues found in sit-uations. Trait activation theory starts with thecommon notion that a person’s trait level isexpressed as trait-relevant behavior (at work,in AC exercises, etc.). Apart from the maineffect of situations on work behavior (andvice versa), a key axiom underlying trait

Correspondence concerning this article should beaddressed to Filip Lievens. E-mail: filip.lievens@ugent.be

Address: Department of Personnel Management,Work and Organizational Psychology, Ghent Univer-sity, Henri Dunantlaan 2, 9000 Ghent, Belgium

Filip Lievens, Department of Personnel Manage-ment, Work and Organizational Psychology, GhentUniversity.

Industrial and Organizational Psychology, 1 (2008), 112–115.Copyright ª 2008 Society for Industrial and Organizational Psychology. 1754-9426/08

112

activation theory is that traits will manifest astrait-expressive work behaviors only whentrait-relevant cues are present. It is positedthat specific task features (e.g., a messydesk), social features (e.g., problem with col-leagues), and organizational features (e.g.,team-based organizational culture) are pos-ited to moderate whether and how traits areexpressed in trait-relevant behavior. Forexample, a trait such as autonomy is likelynot to be expressed in routine monotonousjobs (task level), in the presence of a control-ling supervisor (social level), or in a rigidautocratic culture (organizational level),whereas it is likely to be activated in thereverse conditions. All these examples areeasily transferable to AC exercises.

Lance mentions trait activation theory.However, it is treated as another ‘‘fix’’ forsolving ACs. Granted, in prior studies thathave examined trait activation in ACs (e.g.,Lievens et al., 2006), this theory was usedpost hoc for explaining the results obtained.However, relegating trait activation theory toone of the AC fixes does not pay tribute to thistheory because trait activation primarilydeals with candidates demonstrating trait-relevant behavior (trait expression model)instead of with assessors perceiving and rat-ing that behavior in trait-like terms (trait eval-uation model; see Lance, Foster, Gentry, &Thoresen, 2004). Therefore, it has great ben-efits to shed light on strategies for increasingthe amount of candidate behavior to beobserved in exercises. Hence, the remainderdelineates how trait activation theory mightbe used to advance both research and prac-tice of exercise-based assessment.

A research agenda. As Lance noted, mostprior AC research focused on assessors.Given the importance of candidates’ cross-situationally inconsistent performanceacross exercises, I agree that research shouldalso pay attention to assessees. Below, I pro-vide concrete research suggestions aroundthe three general issues (individual charac-teristics, exercise characteristics, and theirinteraction) noted by Lance.

First, we need to better understand whichindividual differences variables affect candi-

date performance across exercises. It mightbe particularly relevant to examine a broadcategory of constructs called social effec-tiveness constructs. According to Ferris,Perrewe, and Douglas (2002), social effec-tiveness is one of a number of ‘‘broad, higher-order, umbrella terms, which groups anumber of moderately related, yet conceptu-ally distinctive, manifestations of socialunderstanding and competence’’ (p. 50).These social effectiveness constructs areknown under various aliases such as socialcompetence, self-monitoring, social skill,and so on. People high on these constructsare typically able to ‘‘read’’ situations betterthan others and flexibly adapt their inter-personal behavior in line with the cuesgathered. In the particular context of selec-tion, the ability to identify the criteria usedin a selection procedure fits in this broadcategory of social effectiveness constructs(Konig, Melchers, Kleinmann, Richter, &Klehe, 2007). For instance, Konig et al.showed that the ability to identify criteriaexplained part of the relationship betweenperformances in selection procedures evenafter controlling for cognitive ability. Suchresearch might provide insight in individ-ual differences variables that affect cross-situational inconsistency.

Second, research should scrutinize exer-cise characteristics. To date, an AC exerciseis largely a black box. We know little abouthow variations in exercise instructions andexercise design might influence perfor-mance. We need to find out which exercisecharacteristics are ‘‘incidentals’’ (i.e., surfaceexercise characteristics that do not determineperformance) and which ones are ‘‘radicals’’(i.e., structural exercise characteristics thatdetermine performance). In a related domain(situational judgment tests), research hasshown that even minor variations in the sit-uations presented to candidates might affectperformance (Lievens & Sackett, 2007).

Third, the interaction between individualdifferences variables and exercise character-istics should be an important focus of futureresearch. In this context, an interactionisttheory as trait activation theory might serveas inspiration. For clarity reasons, trait

Exercise-based assessment 113

activation theory is not used here to increaseconvergent and discriminant validities.Instead, it might help to better understandfactors that affect candidate performancevariations across exercises. For example,trait activation theory might help to identifywhich exercise factors trigger and releasetrait-relevant candidate behavior versuswhich ones impede trait-relevant candidatebehavior (Tett & Burnett, 2003). Clearly,such programmatic research on candidatebehavior and AC exercises is welcome.

The practical side. At a practical level, Iwant to extend Lance’s article by outliningwhat exercise-based assessment mightreally mean in practice. Exercise-basedassessment is more than simulating keytask, social, and organizational demandsand rating general exercise performance.It also means that we build opportunitiesin AC exercises that elicit job-relatedbehavior. In other words, it also includesimportant exercise design issues. Again, Iwant to emphasize that these practicalsuggestions are not meant to fix the ACto obtain ‘‘better’’ convergent and dis-criminant validities. Neither are they pro-vided to rate personality traits in exercises.Conversely, these suggestions are meantto guarantee that trait-relevant and job-relevant behaviors are displayed by candi-dates. Regardless of how that behavior isthen captured by assessors (in task-basedmodels, dimension-based models, etc.),eliciting and observing behavior are keyto effective ACs and development centers.

One way to elicit behavior consists ofpaying attention to exercise instructions. InACs, exercise instructions provide informa-tion and expectations to candidates aboutwhat behavior to show or not to show. Forexample, exercise instructions might bevague (e.g., ‘‘solve the problem’’) or moreconcrete (e.g., ‘‘motivate the problem subor-dinate’’). Similarly, exercise instructionsmight be unidimensional (e.g., reach con-sensus) or multidimensional (e.g., reachconsensus and make the company moreprofitable). Clearly, these are only someexamples of possible variations in exercise

instructions. To date, we know little abouthow such variations might affect the ACbehavior demonstrated and its determinants.

Roleplayer cues are another means foreliciting trait-related behavior. In currentAC practice, roleplayers are typically givena specific list of things to do and to avoid.Roleplayers are also trained to perform real-istically albeit consistently across candi-dates. Although these best practices haveproven their usefulness, a key function oftrained roleplayers consists of evokingdimension-related behavior from candi-dates (Thornton & Mueller-Hanson, 2004).Roleplayer cues that are determined on thebasis of trait activation theory should subtlyelicit assessee behavior because otherwisethe situations might become too strong.

More generally, AC developers shouldcarefully build design characteristics intoexercises that might elicit specific trait-related behavior. In current AC practice,exercises are primarily developed toincrease fidelity and criterion-related valid-ity. I am not proposing that this practiceshould be abandoned. However, strategiesthat activate trait-relevant behavior shouldalso play a role. For example, if organiza-tions want to activate behavior related tothe trait of emotional stability (without rat-ing this trait) that was deemed important onthe basis of a job analysis, they must useexercises that put people in a situation thatmight activate behavior relevant to this trait.An oral presentation with challenging ques-tions might be an obvious strategy. Otherexamples might be the inclusion of strin-gent time limits, sudden obstacles, or infor-mation overload in exercises.

Conclusions

Lance argues that AC researchers have beenasking the ‘‘wrong’’ questions for years. Mycommentary extends Lance’s article by out-lining what might be the ‘‘right’’ questions inthe next years. From a research perspective,my commentary might serve as a spring-board for examining what determines asses-see behavior and performance. At a practicallevel, it might provide practitioners with

114 F. Lievens

clues for better exercise design and for gen-erating more trait-related and job-relatedbehaviors in exercises.

References

Ferris, G. R., Perrewe, P. L., & Douglas, C. (2002). Socialeffectiveness in organizations: Construct validityand research directions. Journal of Leadership andOrganization Studies, 9, 30–55.

Konig, C. J., Melchers, K. G., Kleinmann, M., Richter, G.M., & Klehe, U.-C. (2007). The ability to identifycriteria in nontransparent selection procedures: Evi-dence from an assessment center and a structuredinterview. International Journal of Selection andAssessment, 15, 283–292.

Lance, C. E. (2008). Why assessment centers do notwork the way they are supposed to. Industrial andOrganizational Psychology: Perspectives on Scienceand Practice, 1, 84–97.

Lance, C. E., Foster, M. R., Gentry, W. A., & Thoresen,J. D. (2004). Assessor cognitive processes in anoperational assessment center. Journal of AppliedPsychology, 89, 22–35.

Lievens, F., Chasteen, C. S., Day, E. A., & Christian-sen, N. D. (2006). Large-scale investigation of therole of trait activation theory for understandingassessment center convergent and discriminantvalidity. Journal of Applied Psychology, 91,247–258.

Lievens, F., & Sackett, P. R. (2007). Situationaljudgment tests in high stakes settings: Issuesand strategies with generating alternate forms.Journal of Applied Psychology, 92, 1043–1055.

Tett, R. P., & Burnett, D. D. (2003). A personalitytrait-based interactionist model of job per-formance. Journal of Applied Psychology, 88,500–517.

Thornton, G. C. I., & Mueller-Hanson, R. A. (2004).Developing organizational simulations: A guidefor practitioners and students. Mahwah, NJ:Erlbaum.

Exercise-based assessment 115