feature

0 7 4 0 - 7 4 5 9 / 0 2 / $ 1 7 . 0 0 © 2 0 0 2 I E E E J a n u a r y / F e b r u a r y 2 0 0 2 I E E E S O F T W A R E 6 7

Over the past five years, we have partici-pated in over 25 evaluations of software andsystem architectures in many different appli-cation domains, using first the Software Ar-chitecture Analysis Method (SAAM)2 andlater the Architecture Trade-off AnalysisMethod (ATAM).3 In discussing these meth-ods in the past, we reported almost solely ontheir technical aspects. However, developingand refining these methods exposed us to awide variety of systems, organizations, orga-nizational goals, management styles, and in-dividual personalities. These experienceshave forged our opinions—in particular, weare now convinced of the need to explicitlyteach and manage the nontechnical aspectsof running an architecture review.

We thus decided to record our observa-tions and findings on the management, psy-chology, and sociology of performing archi-tecture evaluations. Getting these factorswrong can doom the best technical effort.

This observation is not particularly an arti-fact of software development; it holds trueof any complex engineering effort:

When Brunel and Robert Stephenson werebuilding railways in the 1830s and 1840s, theywere expected to involve themselves with rais-ing capital, appearing before ParliamentaryCommittees, conciliating influential peoplewho might oppose the necessary bill in Parlia-ment, negotiating with landlords over whoseland the tracks were to be laid, managing hugegangs of labourers, and dealing with subcon-tractors. Railways engineers had to be expertin finance, in politics, in real estate, in labourmanagement, and in procurement. Why shouldwe be surprised if software engineers may needto draw on expertise in mathematics, financialanalysis, business, production quality control,sociology, and law, as well as in each applica-tion area they deal with?4

The point is that these kinds of issues arerelevant for anyone who has a business inter-

Making ArchitectureReviews Work in the Real World

Rick Kazman and Len Bass, Software Engineering Institute, Carnegie Mellon University

Architecturereviews differ fromother technicalreviews because of their closerelationship to asystem’s businessgoals. Consequently,they should beapproacheddifferently, with aneye for nontechnicalissues. Here, theauthors explore the social,psychological, andmanagerial issues offormal architecturereviews.

Asoftware architecture is more than just a technical blueprint of acomplex software-intensive system. In addition to its technicalfunctions, a software architecture has important social, organiza-tional, managerial, and business implications.1 This same obser-

vation holds of architecture reviews. We can’t simply regard them as tech-nical reviews and ignore their other implications.

architecture reviews

est in a system’s success—primarily the archi-tects but also the managers, customers, andarchitecture reviewers. In particular, as archi-tecture reviewers, we continually run into so-cial, psychological, and managerial issuesand must be prepared to deal with them.

Architecture reviewsSince at least 1976,5 reviews have been

recognized as an efficient means for detect-ing defects in code or other artifacts. Whythen, is an architecture review worthy ofspecial consideration?

In one of our engagements, during a dis-cussion of business goals, one stakeholderturned to another and said, “I told youwhen you originally brought this up, and Iwill tell you again—you are a small portionof my market, and I cannot adjust [the topicunder discussion] to satisfy you.”

The difference between an architecturereview and a code review is revealed in thequestion the review is designed to answer.The code review is designed to answer,“Does this code successfully implement itsspecification?” The architecture review isdesigned to answer, “Will the computer sys-tem to be built from this architecture satisfyits business goals?”

Asking whether code meets specificationsassumes the specifications are clear. Often aprecondition for a code inspection is a priorinspection of the specifications. Askingwhether an architecture satisfies businessgoals cannot simply assume clarity of thebusiness goals. A system’s business goals willvary depending on the stakeholders’ perspec-tives and how much time has passed since thesystem was conceived. Time to market, cost,quality, and function priorities can all changebased on events that occur during the initialarchitecture design. If the business goalsaren’t clear, then one (important) portion ofthe architecture review process is to opera-tionalize the business goals.

Who participates in the review?In a code review, only the reviewers (usu-

ally three to five) are present, and they focuson the task at hand. In an architecture re-view, not only are three to five reviewerspresent but also the architect and, possibly,the architecture team. In addition, becausebusiness goals are discussed, a variety ofstakeholders must also participate. At one

review, we had 30 to 40 stakeholders in theroom. We were reviewing a large system forwhich the government had contracted, andmany government agencies and contractorswere developing parts of the hardware andsoftware. As we will later discuss, settingthe business goals and having differentstakeholders involved causes many logisti-cal problems.

Furthermore, in a code review, most ofthe participants are development profes-sionals who are familiar with the reviewprocess. Those who are not can be in-structed to study a description of the codeinspection process prior to the review. Formany stakeholders, the architecture reviewis their first experience with a review thatattempts to match business goals with an ar-chitecture, and many are busy managersand professionals who can’t be expected tostudy a process description. Consequently,we must spend valuable review time ex-plaining the process. Also, having stake-holders who cross organizational bound-aries inherently raises the review’s visibility.Managers from multiple groups interestedin the system’s success (or failure) are awareof the review and interested in its outcome.

What does “success” mean?A client (whose system was behind

schedule and over budget) once complainedth68at his architecture had already been re-viewed multiple times. “I’m tired of beingreviewed by amateurs,” he said. “They sim-ply repeat what they were told and have novalue added.”

Code reviews have as their output a col-lection of discovered defects. There is ampleevidence that discovering defects during acode inspection is more cost-effective thandiscovering the defects after deployment.Furthermore, there is usually no contro-versy about whether a defect has been dis-covered. Someone proposes a sequence ofevents under which incorrect results will oc-cur to convince the inspection team of thedefect. However, the outputs from an archi-tecture review are much more varied—somearchitectural documentation, a set of sce-narios of concern, and list of risks, sensitiv-ity points, and tradeoff points. If the stake-holders do not agree on the form or value ofthe review’s outputs, then success will bedifficult to achieve (or even define).

The differencebetween an

architecturereview and acode review

is revealed inthe question the review isdesigned to

answer.

6 8 I E E E S O F T W A R E J a n u a r y / F e b r u a r y 2 0 0 2

Reviewing the risksATAM-based architecture inspections

have as their major output a collection ofrisks, sensitivity points, and tradeoff points.A risk is an alternative that might causeproblems with respect to meeting a businessgoal. Notice the vagueness in this definition:an alternative that “might” cause problems.Because software systems are expensive toconstruct, it is nearly impossible to gather ev-idence about development paths not taken.

There are four broad categories of out-puts that can emerge from an architecturereview:

� Technical risks, which emerge from aSAAM or an ATAM. For example,“Given the current characterization ofpeak load and the existence of only twoWeb servers, we might not meet the ar-chitecture’s latency goals.” We can miti-gate such risks through more analysis,simulation, or prototyping.

� Information risks, involving areas of thearchitecture lacking information. Thereare times during a review when the re-sponse to a reviewer’s questions is sim-ply, “We haven’t thought about that.”

� Economic risks (cost, benefit, and sched-ule), which are not directly technical innature but are about dollars and deliv-erables. “Can we deliver this functional-ity to our customers by August? Will welose market share if our performance isn’t quite as good as our competitor’s?Can we build this for less than $80 perunit?” Architectural decisions all pro-foundly affect these questions. We canmitigate these risks using architectureanalysis techniques that focus on theseissues (for example, the Cost–BenefitAnalysis Method6).

� Managerial risks, which involve havingan architecture improperly aligned withthe organization’s business goals.7 Suchmisalignment can be risky for an organ-ization—regardless of its product’s tech-nical superiority—and might require acostly realignment process. Examples ofother kinds of managerial risks includemisaligning the architecture’s structureand the development organization’sstructure or depending on supplierswhose reliability is unknown or suspect.Each case requires a kind of realignment

between the architecture and manage-ment or business strategy.

Planned versus unplannedCode reviews are usually included in the

normal development plan as are architecturereviews. However, unplanned architecturereviews also sometimes occur, either becausea stakeholder (typically an architect or man-ager) becomes interested in the possibility ofdoing such a review to validate and improvean existing project or because an upper-levelmanager wants a review to scrutinize a proj-ect that is perceived as being in trouble. Un-planned reviews for troubled projects have acertain amount of inherent tension. Every-one involved understands that the stakes arehigh and the project could be cancelled as aresult of the review. Thus, such reviews oftenresult in finger pointing and blaming but un-fortunately offer little hope of rescuing thearchitecture or project.

Unplanned reviews also require a certainamount of selling. “What are the reviewersreviewing? What will be the outcomes?How much time will it cost us [the client] indollars and days?” These are the kinds ofquestions that the stakeholders ask. Theymust be convinced that an architecture re-view will help and not hinder the project ortheir decision making prior to proceeding.

Review preparationA code inspection meeting typically in-

spects about 300 lines of code. Although notall of a system’s code is necessarily inspected,the decision of which code to inspect is madeoutside of the inspection. Furthermore, a sys-tem’s code should be available prior to the re-view—otherwise, there’s nothing to review.

In an architecture review, because its goalis to decide how well the architecture sup-ports the business goals, reviewers can in-spect any portion of the system. Further-more, because the business goals are oftenambiguous, we need to dedicate a portionof the review to determining which parts ofthe system are involved in meeting thosebusiness goals. This means that the review-ers must be aware of the entire system’s ar-chitecture. Also, the architecture is not al-ways adequately documented—in contrastto code inspections, adequate architecturaldocumentation might not have been pre-pared. For example, one of our reviews was

J a n u a r y / F e b r u a r y 2 0 0 2 I E E E S O F T W A R E 6 9

ATAM-basedarchitectureinspections

have as theirmajor output a collection

of risks,sensitivitypoints, and

tradeoff points.

unplanned in nature, and although we werecommitted to a particular schedule, the ar-chitecture documentation was not forth-coming. Finally, two days before the reviewtook place, we received a vast collection ofclass descriptions said to constitute the “ar-chitecture documentation.” That was thebasis under which we were forced to con-duct the review. As might be expected, thereview did not follow the normal procedureand was somewhat ad hoc in nature.

Managing the review processLet’s apply these observations about ar-

chitecture reviews to motivate a set of con-cerns that a reviewer will face when work-ing through the review process. Differentarchitecture-review techniques have differ-ent processes. For this reason, we restrictourselves to discussing the three broadstages of activity in any review: prework,which involves negotiating and preparingfor the review; work, which involves scruti-nizing the architectural artifacts; and post-work, which is when you report the review’sresults and determine what actions to take.

PreworkIt is easy to pay less attention to the activ-

ities that precede a review, because they don’tinvolve any technical analyses. However, be-cause of an architecture review’s unique na-ture, if these activities are ignored, the re-view’s results might be meaningless. If thecustomer is completely unprepared to be re-viewed, it wastes the time of both the re-viewers and the team under review. Preworkusually involves three main tasks, and al-though it isn’t always exciting, it ensures thatthe later work is exciting and meaningful.

First, prework involves selling the review.Because reviews tend to originate from dif-ferent sources within an organization, it isoften the case that some of the participantsare unhappy with the review. This is partic-ularly the case when the review activity isnot part of the normal software process. Insuch a case, it is often viewed as an intrusionon the stakeholders’ time. So, it is crucial,both before and during the review, to sellmanagement, the architects, and the otherstakeholders on the review’s value—that areviewed system will meet the stakeholder’sgoals better than an unreviewed system.

Second, it is also important to set expec-

tations. All of the stakeholders should un-derstand the method’s capabilities. In a typi-cal ATAM, you meet with a client for justthree days. You need to use this limited timeto evaluate the architecture’s ability toachieve its business goals—you will not dodetailed technical analyses in any dimension.

Finally, you need to decide who should bethere for what stages. Some participants willperceive any review, no matter how crucial,as an intrusion on their precious time. Hence,it is important to identify and forewarn thestakeholders and ensure that each of themknows which steps of the process requiretheir presence, and why. In addition, it is of-ten useful to strictly limit attendance in manyof the steps to the minimal set of stakehold-ers needed, because this typically makes for amore efficient and productive use of time. Asmentioned earlier, there are times when 30 or40 stakeholders will want to attend, perhapsto impress their bosses or sponsors, furthertheir own agendas, or bill for extra time.Whatever the reason, their agendas shouldnot dilute the review’s effectiveness.

As part of the standard set of ATAM ma-terials, we have prepared a presentation de-signed to support these three goals. Thepresentation explains the evaluation’s goalsand process as well as its costs and benefits.This presentation is useful both for thosewho are going to participate in the evalua-tion and those whose concurrence is neededfor the evaluation to proceed.

WorkDuring an architecture review, we not

only perform the steps of the review processbut also act as facilitators. This is where thereviewer’s personal qualities (mentionedearlier) are important. The review teammust have members who are experienced,architecturally savvy, and quick on their feet.

Technical rationaleDuring an architecture review, we should

be able to establish business goals, rely onthe architect, and identify risks.

An effective architecture review processpresents the system’s business goals to thereviewers and stakeholders and identifiesand prioritizes a set of scenarios that repre-sent concrete specifications of these goals.For example, if one business goal is for thesystem to be long-lasting, then modification

The reviewteam must have

members who are

experienced,architecturally

savvy, andquick on

their feet.

7 0 I E E E S O F T W A R E J a n u a r y / F e b r u a r y 2 0 0 2

scenarios become a high priority. Thesehigh-priority scenarios determine the evalu-ation’s focus.

Because the architect identifies how thearchitecture will satisfy such scenarios, heor she is critical to making the architectureunderstandable. Each scenario begins with astimulus—an event arriving at the system, afault occurring, or a change request beinggiven to the development team. The archi-tect walks through the architecture, explain-ing what is affected by the stimulus—for ex-ample, how the system will process an eventor detect a fault, or what components willbe affected by a change. The architect willalso review how a response to the stimuluswill be made—the recovery mechanismsneeded and which modules will changewhen a change request is addressed.

Each scenario reflects specific businessgoals. To ensure that these business goals areadequately met, part of the evaluators’ duediligence is to understand the architect’s ex-planations of how the architecture will re-spond to the stimulus and how any identifiedrisks will be dealt with. Risks are identified asa result of three possible conditions: the ex-planation of how the architecture respondsto the stimulus is not convincing, businessgoals other than the ones reflected in the sce-nario are violated, or fundamental decisionshave not yet been made. Usually, when an ar-chitecture review is held, not all architecturaldecisions have been made. However, this be-comes a risk when too many decisions haveyet to be made or when the indecision threat-ens business goals.

Social behaviorTo best use the stakeholders’ time during

the review, the team needs to

1. control the crowd,2. involve the key stakeholders,3. engage all participants, 4. maintain authority, 5. control the pace, and 6. get concurrence and feedback.

Crowd control is critical. At the start ofthe review meeting, establish how and whenpeople may interact with each other. For ex-ample, it is important to avoid disruptions,so side conversations, cell phones, andpagers should be banned. Establish at the

outset whether people can come and go,and when they absolutely must be present.Latecomers should not expect the proceed-ings to stop so they can be updated. Hold-ing the review meeting away from the homesite of the team being reviewed is an effec-tive way of minimizing interruptions.

Involving key stakeholders is also impor-tant, because any activity that involves theentire group interacting also helps achievebuy-in. As each point important for the de-sign or analysis emerges, record it visibly andverify that the recording is correct. Partici-pants then feel they are helping in theprocess; it isn’t just the reviewers reviewingthe architecture—it’s the entire group discov-ering aspects of the architecture. For exam-ple, in one review, the architecture team (andthe lead architect, in particular) were skepti-cal of the review’s value and hence initiallyparticipated only grudgingly. Throughout thereview, and as insights into the systememerged, we convinced the architect that thisactivity was for everyone’s benefit and thatwe were not there to point fingers but ratherto improve the architecture and hence the re-sulting system. By the end of the review, thearchitect was an enthusiastic participant.

In addition to involving the key stake-holders, it is important to engage all partic-ipants. It is not uncommon, in any kind ofmeeting, to have people who dominate theairwaves or people who are shy about par-ticipating. For example, some people mightbe reluctant to speak frankly in front oftheir bosses or their subordinates. For thesepeople, it is important to provide a forum inwhich they are either alone or only amongpeers. It is also important to provide a mixof free-for-all participation and periodswhere each person has a dedicated “safe”time to speak.

Of course, despite your best efforts, theremight be times when an evaluation gets outof control: people will have side conversa-tions, try to steal the agenda, or resist pro-viding information. The review team needsto know who has ultimate authority if peo-ple are being disruptive. Is it the reviewteam leader, the customer, a specific man-ager, or the architect? The review team fa-cilitator should have a strong (but not dog-matic) personality and should be able tomanage the group dynamics through a com-bination of humor, appeal, and authority.

J a n u a r y / F e b r u a r y 2 0 0 2 I E E E S O F T W A R E 7 1

Because thearchitectidentifies how the

architecturewill satisfy

such scenarios,he or she iscritical tomaking the

architectureunderstandable.

Similarly, the team architecture reviewfacilitator must be able to control the pace.Any meeting with a diverse group of (likelyhighly opinionated) stakeholders will rangeout of control from time to time, as men-tioned. But sometimes these conversationsare revealing—hidden agendas, worries, fu-ture requirements, past problems, and amyriad of other issues come to light. The fa-cilitator must be aware of these digressionsand know both when to squelch them andwhen to let a conversation continue. A por-tion of the time allocated is reserved for thereviewers to prepare their briefing. Thissame time can be used for offline meetingsamong the stakeholders.

This kind of facilitation can be exhaust-ing: assimilating huge amounts of informa-tion, looking for problems, and managingall of the political and personal issues si-multaneously. Thus, we have found that it isuseful to have two facilitators and to switchbetween them periodically to give each amental break.

Finally, be sure to obtain concurrence andfeedback. The activities and information gen-erated in a review are not really directed atthe review team, even though the reviewteam is frequently the focus of the conversa-tion and the source of many of the probingquestions. The review’s outputs are really forthe stakeholders—the review team membersare just there to act as catalysts, experts, andfacilitators. Because of this mismatch be-tween the producers and consumers of the in-formation and the way that the informationis elicited (through the facilitation of the re-view team), extra care must be paid to ensurethat all stakeholders concur with whatever isrecorded. In our reviews, for example, wetypically record information on flip-charts inreal time as well as on a computer—for laterpresentation or out-briefing and for the finalreport. When we put information up on aflip-chart, we have a golden opportunity toensure that the information recorded is cor-rect. Thus, we endeavor to get concurrenceas items are posted and ensure that we keepflip-charts visible around the room so thatthey are always available for consultation,correction, and refinement. Architects arehuman and welcome encouragement andpositive feedback. A review is mostly con-cerned with finding problematic decisions, soit is useful for the reviewers to occasionally

make positive comments about particular ar-chitectural decisions. This helps alleviate thegenerally negative questions and sets a morepositive tone.

PostworkReviews of all kinds are part of a mature

organization’s software process, but thetime and trouble involved are wasted unlessthere is a predefined output and a determi-nation of who will act on the results. Oth-erwise, the review’s outputs will end upburied on someone’s desk and become alow-priority item. Thus, postwork activitiesmust report the outputs and follow up onthe review’s results.

When the review is complete, there mustbe some agreement on how to communi-cate the outputs back to the stakeholders—in particular, to management and the archi-tecture team. In the ATAM, we give slidepresentations to the stakeholders immedi-ately after the review and then, some weekslater, we deliver a formal and more exten-sive written report. Regardless of the re-sult’s form, the review team must knowwho gets told what and when. For exam-ple, can the architects respond to the reviewreport before it goes to management orother stakeholders?

A slide presentation or report from a re-view can have many possible destinies. Inour experience, the review’s outputs (listsof scenarios, architectural styles, risks,nonrisks, sensitivities, and trade offs) vali-date existing project practices, change ex-isting project practices (such as architec-ture design and documentation), argue forand obtain additional funding from man-agement, and plan for future architecturalevolution. In one case, a participant, theday after the review, used the outbrief toconvince management that he needed moreresources for the project—an unsuccessfulappeal prior to the review. It is thus im-portant to consider the report’s goal and toplan accordingly.

The time is ripe for the widespreadadoption of architecture reviews asa standard part of the software engi-

neering lifecycle. Materials to teach andsupport the practice of architecture reviewshave been increasing in both quantity and

When thereview is

complete, theremust be someagreement on

how tocommunicatethe outputsback to the

stakeholders.

7 2 I E E E S O F T W A R E J a n u a r y / F e b r u a r y 2 0 0 2

quality, including a book devoted entirely toarchitecture reviews.3 Many large corpora-tions are now adopting architecture reviewsas part of their standard software engineer-ing development practice, and some areeven including these reviews as part of theircontracting language when dealing withsubcontractors.

In addition, the scope of these reviews isgrowing to include far more than just thetechnical issues. As we have stressed, whendealing with an architecture, business issuesare the driving factors in design. By consid-ering the relations between business goalsand architecture, and considering themearly in the software development (or rede-velopment) process, they might be dealtwith in a way that provides the greatest ben-efit for the system’s many stakeholders.

References1. L. Bass, P. Clements, and R. Kazman, Software Archi-

tecture in Practice, Addison-Wesley, Reading, Mass.,1998.

2. R. Kazman et al., “SAAM: A Method for Analyzing theProperties of Software Architectures,” Proc. 16th Int’lConf. Software Eng., IEEE CS Press, Los Alamitos,Calif., 1994, pp. 81–90.

3. P. Clements, R. Kazman, and M. Klein, Evaluating Soft-ware Architectures: Methods and Case Studies, Addi-son-Wesley, Reading, Mass., 2001.

4. M. Jackson, Software Requirements and Specifications,Addison-Wesley, Reading, Mass., 1995.

5. M.E. Fagan, “Design and Code Inspections to ReduceErrors in Program Development,” IBM Systems J., vol.15, no. 3, 1976, pp. 182–211.

6. R. Kazman, J. Asundi, and M. Klein, “Quantifying theCosts and Benefits of Architectural Decisions,” Proc.23rd Int’l Conf. Software Eng., IEEE CS Press, LosAlamitos, Calif., 2001, pp. 297–306.

7. J. Henderson and N. Venkatraman, “Strategic Align-ment: Leveraging Information Technology for Trans-forming Organizations,” IBM Systems J., vol. 32, no. 1,1993, pp. 4–16.

For more information on this or any other computing topic, please visit ourDigital Library at http://computer.org/publications/dlib.

J a n u a r y / F e b r u a r y 2 0 0 2 I E E E S O F T W A R E 7 3

About the Authors

Rick Kazman is a senior researcher at the Software Engineering Institute of CarnegieMellon University and an adjunct professor at the Universities of Waterloo and Toronto. His pri-mary research interests are software engineering (software architecture, design tools, and soft-ware visualization), human–computer interaction (particularly interaction with 3D environ-ments), and computational linguistics (information retrieval). He received a BA and M.Mathfrom the University of Waterloo, an MA from York University, and a PhD from Carnegie MellonUniversity. His book Software Architecture in Practice (written with Len Bass and Paul Clements,Addison-Wesley, 1998) received Software Development Magazine’s Productivity Award. Hismost recent book is Evaluating Software Architectures: Methods and Case Studies (written with

Paul Clements and Mark Klein, Addison-Wesley, 2001). Contact him at kazman@sei.cmu.edu.

Len Bass is a senior researcher at the Software Engineering Institute of Carnegie MellonUniversity. He has written or edited six books and numerous papers in a wide variety of areasof computer science, including software engineering, human–computer interaction, databases,operating systems, and theory of computation. He received his PhD in computer science fromPurdue University. Contact him at ljb@sei.cmu.edu.

The exploding popularity of mobile Internet access, third-generation wirelesscommunication, and wearable and handheld devices have made pervasivecomputing a reality. New mobile computing architectures, algorithms,environments, support services, hardware, and applications are coming onlinefaster than ever. To help you keep pace, the IEEE Computer Society andCommunications Society are proud to announce IEEE Pervasive Computing.

This new quarterly magazine aims to advance pervasive computing bybringing together its various disciplines, including

• Hardware technologies• Software infrastructure• Real-world sensing and interaction• Human–computer interaction• Security, scalability, and privacy

Led by Editor in Chief M. Satyanarayanan, the founding editorial boardfeatures leading experts from UC Berkeley, Stanford, Sun Microsystems, and Intel.

Don’t miss the premier issue

SUBSCRIBE NOW!

http://computer.org/pervasive

NEW FOR 2002

IEEE Distributed Systems Online, Pervasive Computing’s online resource.

DS Online offers expert-moderated information on topics including mobile & wireless computing, distributed agents,

and operating systems.

computer.org/dsonline

VISIT