software archiecture lecture10

The ATAMA Comprehensive method for Architecture Evaluation

IntroductionIntroduction

• We will introduce the Architecture Tradeoff • We will introduce the Architecture Tradeoff Analysis Method (ATAM), a thorough and comprehensive way to evaluate a software architecture.architecture.

• The ATAM is so named because

▫ it reveals how well an architecture satisfies ▫ it reveals how well an architecture satisfies particular quality goals

▫ it provides insight into how quality goals interact—▫ it provides insight into how quality goals interact—that is, how they trade off.


• Evaluating an architecture for a large system is a complicated undertaking. complicated undertaking. • First, a large system will have a comparably large architecture that will be difficult to understand in a limited amount of time. limited amount of time. • Second, according to the Architecture Business Cycle (ABC), a computer system is intended to support business goals and the evaluation will need to make business goals and the evaluation will need to make connections between those goals and the technical decisions. • Finally, a large system usually has multiple stakeholders • Finally, a large system usually has multiple stakeholders and acquiring their different perspectives in a limited amount of time requires careful management of an evaluation process. • Managing the limited time for an architecture evaluation • Managing the limited time for an architecture evaluation is a central problem.


• The ATAM is designed to • The ATAM is designed to

▫ elicit the business goals for the system as well as for the architecture

▫ those goals and stakeholder participation to focus the attention of the evaluators on the portion of the architecture that is central to the achievement the architecture that is central to the achievement of the goals.

• This chapter will introduce the steps of the • This chapter will introduce the steps of the ATAM and discuss them in light of their intended purpose.

Participants in the ATAMParticipants in the ATAM

• The ATAM requires the participation of three groups• The evaluation team• The ATAM requires the participation of three groups• The evaluation team▫ This group is external to the project.▫ They need to be recognized as competent, unbiased outsiders with no hidden agendas or axes to grind.▫ They need to be recognized as competent, unbiased outsiders with no hidden agendas or axes to grind.

• Project decision makers▫ These people are empowered to speak for the development project or have the authority to mandate ▫ These people are empowered to speak for the development project or have the authority to mandate changes to it

• Architecture stakeholders▫ Their job during an evaluation is to articulate the • Architecture stakeholders▫ Their job during an evaluation is to articulate the specific quality attribute goals that the architecture should meet in order for the system to be considered a success.a success.

ATAM Evaluation Team RolesATAM Evaluation Team Roles

Role Responsibilities Desirable characteristics

Team Leader Sets up the evaluation; coordinates with client, making sure client's needs are met; establishes evaluation contract; forms evaluation team; sees that final report is produced and delivered

Well-organized, with managerial skills; good at interacting with client; able to meet deadlines

report is produced and delivered (although the writing may be delegated)

Evaluation Leader Runs evaluation; facilitates elicitation of scenarios; administers scenario selection/prioritization process; facilitates

Comfortable in front of audience; excellent facilitation skills; good understanding of architectural issues; selection/prioritization process; facilitates

evaluation of scenarios against architecture; facilitates onsite analysis

understanding of architectural issues; practiced in architecture evaluations; able to tell when protracted discussion is leading to a valuable discovery or when it is pointless and should be re-directeddirected



Scenario Scribe Writes scenarios on flipchart or whiteboard during scenario elicitation; captures agreed-on wording of each scenario, halting discussion until exact wording is captured

Good handwriting; stickler about not moving on before an idea (scenario) is captured; can absorb and distill the essence of technical discussionstechnical discussions

Proceedings Scribe Captures proceedings in electronic form on laptop or workstation, raw scenarios, issue(s) that motivate each scenario (often lost in the wording of the scenario itself), and resolution

Good, fast typist; well organized for rapid recall of information; good understanding of architectural issues; able to wording of the scenario itself), and resolution

of each scenario when applied to architecture(s); also generates a printed list of adopted scenarios for handout to all participants

architectural issues; able to assimilate technical issues quickly; unafraid to interrupt the flow of discussion (at opportune times) to test understanding of an issue so that appropriate an issue so that appropriate information is captured

Timekeeper Helps evaluation leader stay on schedule; helps control amount of time devoted to each scenario during the evaluation phase

Willing to interrupt discussion to call time

scenario during the evaluation phase



Process Observer Keeps notes on how evaluation process could be improved or deviated from; usually keeps silent but may make discreet process-based suggestions to the evaluation leader during the evaluation; after evaluation, reports on

Thoughtful observer; knowledgeable in the evaluation process; should have previous experience in the architecture evaluation methodthe evaluation; after evaluation, reports on

how the process went and lessons learned for future improvement; also responsible for reporting experience to architecture evaluation team at large

evaluation method

evaluation team at large

Process Enforcer Helps evaluation leader remember and carry out the steps of the evaluation method

Fluent in the steps of the method, and willing and able to provide discreet guidance to the evaluation leader

Questioner Raise issues of architectural interest that stakeholders may not have thought of

Good architectural insights; good insights into needs of stakeholders; experience with systems in similar domains; unafraid to bring up contentious unafraid to bring up contentious issues and pursue them; familiar with attributes of concern

Outputs of the ATAMOutputs of the ATAM

• An ATAM-based evaluation will produce at least • An ATAM-based evaluation will produce at least the following outputs

▫ A concise presentation of the architecture.▫

▫ Articulation of the business goals.

▫ Quality requirements in terms of a collection of scenarios.scenarios.

▫ Mapping of architectural decisions to quality requirements.requirements.

▫ A set of identified sensitivity and tradeoff points.

▫ A set of risks and nonrisks.▫ A set of risks and nonrisks.

▫ A set of risk themes.

Outputs of the ATAMOutputs of the ATAM

• The outputs are used to build a final written • The outputs are used to build a final written report that recaps the method, summarizes the proceedings, captures the scenarios and their analysis, and catalogs the findings.analysis, and catalogs the findings.• There are intangible results▫ a palpable sense of community on the part of the ▫ a palpable sense of community on the part of the stakeholders▫ open communication channels between the architect and the stakeholdersarchitect and the stakeholders▫ a better overall understanding on the part of all participants of the architecture its strengths and weaknesses▫ its strengths and weaknesses

ATAM Phases and Their CharacteristicsATAM Phases and Their Characteristics

Phase Activity Participants Typical Duration

0 Partnership and preparation

Evaluation team leadership and key project decision makers

Proceeds informally as required, perhaps over a few weeksover a few weeks

1 Evaluation Evaluation team and project decision makers

1 day followed by a hiatus of 2 to 3 weeksweeks

2 Evaluation (continued)

Evaluation team, project decision makers, and stakeholders

2 days

3 Follow-up Evaluation team and evaluation 1 week3 Follow-up Evaluation team and evaluation client

1 week

Phases of the ATAMPhases of the ATAM

• Activities in an ATAM-based evaluation are • Activities in an ATAM-based evaluation are spread out over four phases▫ phase 0, "Partnership and Preparation," the evaluation team leadership and the key project evaluation team leadership and the key project decision makers informally meet to work out the details of the exercise.▫ phase 1, the evaluation team meets with the ▫ phase 1, the evaluation team meets with the project decision makers to begin information gathering and analysis.gathering and analysis.▫ phase 2, the architecture's stakeholders join the proceeding and analysis continues.▫ phase 3 is follow-up in which the evaluation team ▫ phase 3 is follow-up in which the evaluation team produces and delivers a written final report.

Steps of the evaluation phasesSteps of the evaluation phases

• Step 1—Present the ATAM• Step 1—Present the ATAM

• Step 2—Present Business Drivers

• Step 3—Present Architecture• Step 3—Present Architecture

• Step 4—Identify Architectural Approaches

• Step 5—Generate Quality Attribute Utility Tree• Step 5—Generate Quality Attribute Utility Tree

• Step 6—Analyze Architectural Approaches

• Hiatus and Start of Phase 2

• Step 7—Brainstorm and Prioritize Scenarios

• Step 8—Analyze Architectural Approaches

• Step 9—Present Results• Step 9—Present Results

1. Present the ATAM1. Present the ATAM

• The evaluation leader present the ATAM • The evaluation leader present the ATAM

• The leader will describe the ATAM steps in brief and the outputs of the evaluation.and the outputs of the evaluation.

• Explain the process that everyone will be following, to answer questions, and to set the context and expectations for the remainder of context and expectations for the remainder of the activities.

2. Present Business Drivers2. Present Business Drivers

• Let everyone to understand the context for the system and the primary business drivers motivating • Let everyone to understand the context for the system and the primary business drivers motivating its development• A project decision maker (project manager) presents a system overview from a business • A project decision maker (project manager) presents a system overview from a business perspective▫ The system's most important functions▫ Any relevant technical, managerial, economic, or ▫ The system's most important functions▫ Any relevant technical, managerial, economic, or political constraints▫ The business goals and context as they relate to the projectproject▫ The major stakeholders▫ The architectural drivers (that is, the major quality attribute goals that shape the architecture)attribute goals that shape the architecture)

3. Present Architecture3. Present Architecture

• The lead architect (or architecture team) makes a presentation describing the architecture at an appropriate presentation describing the architecture at an appropriate level of detail• Presentation covers technical constraints such as▫ Operating system▫ Operating system▫ Hardware▫ Middleware prescribed for use▫ Other systems with which the system must interact.

• Most important, the architect describes the architectural • Most important, the architect describes the architectural approaches used to meet the requirements.• It should convey the essence of the architecture and not stray into ancillary areas or delve too deeply into the details of just into ancillary areas or delve too deeply into the details of just a few aspects.• During presentation, The evaluation team asks for clarification based on their phase 0 examination of the architecture documentation and their knowledge of the business drivers based on their phase 0 examination of the architecture documentation and their knowledge of the business drivers from the previous step

Example of a template for the Example of a template for the

architecture presentation• Architecture Presentation (~20 slides; 60 minutes)

• Driving architectural requirements, the measurable quantities you associate with these requirements, and any existing standards/models/approaches for meeting these (2–3 slides)Important Architectural Information (4–8 slides)standards/models/approaches for meeting these (2–3 slides)

• Important Architectural Information (4–8 slides)▫ Context diagram—the system within the context in which it will exist. Humans or other systems with which the system will interact.

▫ Module or layer view—the modules (which may be subsystems or layers) that describe the system's decomposition of functionality, along with the

▫ Module or layer view—the modules (which may be subsystems or layers) that describe the system's decomposition of functionality, along with the objects, procedures, functions that populate these, and the relations among them (e.g., procedure call, method invoca-tion, callback, containment).

▫ Component-and-connector view—processes, threads along with the ▫ Component-and-connector view—processes, threads along with the synchronization, data flow, and events that connect them.

▫ Deployment view—CPUs, storage, external devices/sensors along with the networks and communication devices that connect them.Also shown are the processes that execute on the various processors.are the processes that execute on the various processors.

Example of a template for the Example of a template for the

architecture presentation• Architectural approaches, patterns, or tactics employed, including what

quality attributes they address and a description of how the approaches quality attributes they address and a description of how the approaches address those attributes (3–6 slides)

▫ Use of commercial off-the-shelf (COTS) products and how they are chosen/integrated (1–2 slides)chosen/integrated (1–2 slides)

▫ Trace of 1 to 3 of the most important use case scenarios. If possible, include the runtime resources consumed for each scenario (1–3 slides)

▫ Trace of 1 to 3 of the most important change scenarios. If possible, ▫ Trace of 1 to 3 of the most important change scenarios. If possible, describe the change impact (estimated size/difficulty of the change) in terms of the changed modules or interfaces (1–3 slides)

▫ Architectural issues/risks with respect to meeting the driving architectural requirements (2–3 slides)architectural requirements (2–3 slides)

▫ Glossary (1 slide)

4. Identify Architectural Approaches4. Identify Architectural Approaches

• By now, the evaluation team will have a good • By now, the evaluation team will have a good idea of what patterns and approaches the architect used in designing the system.

• In this short step, the evaluation team simply catalogs the patterns and approaches that are evident. evident.

• The list is publicly captured by the scribe for all to see and will serve as the basis for later to see and will serve as the basis for later analysis.

5. Generate Quality Attribute Utility 5. Generate Quality Attribute Utility

Tree• An architecture is either suitable or unsuitable with respect to its ability to deliver particular quality • An architecture is either suitable or unsuitable with respect to its ability to deliver particular quality attributes to the system(s) built from it.• In this step, the quality attribute goals are articulated in detail via a mechanism known as the • In this step, the quality attribute goals are articulated in detail via a mechanism known as the utility tree• the evaluation team works with the project decision makers to identify, prioritize, and refine the system's • the evaluation team works with the project decision makers to identify, prioritize, and refine the system's most important quality attribute goals, which are expressed as scenarios• The utility tree serves to make the requirements • The utility tree serves to make the requirements concrete, forcing the architect and customer representatives to define precisely the relevant quality requirements that they were working to representatives to define precisely the relevant quality requirements that they were working to provide.


Tree• A utility tree begins with utility as the root node.• Second level is set of quality attributes, business drivers • Second level is set of quality attributes, business drivers presentation in step 2 make up the initial set of this second level.• Third level, under each quality attribute are specific • Third level, under each quality attribute are specific quality attribute refinements▫ For example, performance might be decomposed into "data latency" and "transaction throughput”latency" and "transaction throughput”

• Fourth level, Scenarios : ATAM scenarios consist of three parts: ▫ stimulus (what condition arrives at a system, who ▫ stimulus (what condition arrives at a system, who generated it, and what system artifact it stimulates)▫ environment (what is going on at the time)▫ response (system's reaction to the stimulus expressed in a measurable way).measurable way).


Tree

• Some scenarios might express more than one • Some scenarios might express more than one quality attribute and so might appear in more than one place in the tree.

• The evaluation leader should guard against scenarios that try to cover too much diverse territory because they will be difficult to analyze.territory because they will be difficult to analyze.

• A utility tree can easily contain fifty scenarios at its leaves, and there will not be time during the its leaves, and there will not be time during the evaluation meeting to analyze them all. Hence, utility tree generation also includes a utility tree generation also includes a prioritization step.


Tree : prioritization step• By consensus, the decision makers assign a priority • By consensus, the decision makers assign a priority to each scenario. • This prioritization may be on a 0 to 10 scale or use relative rankings such as high, medium, and low.relative rankings such as high, medium, and low.• After that, scenarios are prioritized a second time, by ask the architect to rank each scenario by how difficult he or she believes it will be for the difficult he or she believes it will be for the architecture to satisfy.• Now each scenario has an associated ordered pair: • Now each scenario has an associated ordered pair: (H,H), (H,M), (H,L), and so forth. The scenarios that are the most important and the most difficult will be the ones where precious analysis time will be spent the ones where precious analysis time will be spent and the remainder will be kept as part of the record.

5. Example in Tabular Form of the 5. Example in Tabular Form of the

Utility TreeQuality Attribute Attribute

Refinement ScenariosRefinement Scenarios

Performance Transaction response time

A user updates a patient's account in response to a change-of-address notification while the system is under peak load, and the transaction completes in less than 0.75 second. (H,M)

A user updates a patient's account in response to a change-of-A user updates a patient's account in response to a change-of-address notification while the system is under twice the current peak load, and the transaction completes in less than 4 seconds. (L,M)

Throughput At peak load, the system is able to complete 150 normalized Throughput At peak load, the system is able to complete 150 normalized transactions per second. (M,M)

Generating reports No scenarios suggested.

Usability Proficiency training A new hire with two or more years experience in the business Usability Proficiency training A new hire with two or more years experience in the business becomes proficient in Nightingale's core functions in less than 1 week. (M,L)

A user in a particular context asks for help, and the system provides help for that context. (H,L)provides help for that context. (H,L)

Normal operations A hospital payment officer initiates a payment plan for a patient while interacting with that patient and completes the process without the system introducing delays. (M,M)

6. Analyze Architectural Approaches6. Analyze Architectural Approaches

• In fact, the analysis steps of the ATAM consist of choosing one scenario at a time and seeing how well the • In fact, the analysis steps of the ATAM consist of choosing one scenario at a time and seeing how well the architecture responds to, or achieves, it.• The architect is asked to explain how the architecture • The architect is asked to explain how the architecture supports each scenario.• Questioners probe for the architectural approaches that the architect used to carry out the scenario.the architect used to carry out the scenario.• The team documents the relevant architectural decisions and identifies and catalogs their risks, nonrisks, sensitivity points, and tradeoffs.sensitivity points, and tradeoffs.• The key of analysis is to elicit sufficient architectural information to establish some link between the architectural decisions that have been made and the architectural decisions that have been made and the quality attribute requirements that need to be satisfied.


• For example of analysis• For example of analysis

• The number of simultaneous database clients will affect the number of transactions that a will affect the number of transactions that a database can process per second.

• Thus, the assignment of clients to the server is a sensitivity point with respect to the response as sensitivity point with respect to the response as measured in transactions per second.

• Some assignments will result in unacceptable • Some assignments will result in unacceptable values of this response—these are risks.

Example of architectural approach Example of architectural approach

analysis

Hiatus and Start of Phase 2Hiatus and Start of Phase 2

• At this point, phase 1 is concluded. • The evaluation team retreats to summarize what it has • The evaluation team retreats to summarize what it has learned and interacts informally (usually by phone) with the architect during a hiatus of a week or two. • More scenarios might be analyzed during this period, if • More scenarios might be analyzed during this period, if desired, or questions of clarification can be resolved.• When the project's decision makers are ready to resume and the stakeholders are assembled, phase 2 commences. • This phase is enacted by an expanded list of participants with • This phase is enacted by an expanded list of participants with additional stakeholders attending. ▫ First, step 1 is repeated so that the stakeholders understand the method and the role they are to play. method and the role they are to play. ▫ Then the evaluation leader recaps the results of steps 2 through 6, and shares the current list of risks, nonrisks, sensitivity points, and tradeoff points. ▫ Now the stake holders are up to speed with the evaluation results ▫ Now the stake holders are up to speed with the evaluation results so far, and the remaining three steps can be carried out.

7. Brainstorm and Prioritize Scenarios7. Brainstorm and Prioritize Scenarios

• The evaluation team asks the stakeholders to brainstorm scenarios that are operationally meaningful with respect • The evaluation team asks the stakeholders to brainstorm scenarios that are operationally meaningful with respect to the stakeholders' individual roles. • Stakeholders are free to put them into the brainstorm • Stakeholders are free to put them into the brainstorm pool, which gives them the opportunity to revisit scenarios from step 5 and step 6 that they might feel received too little attention.received too little attention.• Once the scenarios have been collected, they must be prioritized.• Each stakeholder is allocated a number of votes equal to • Each stakeholder is allocated a number of votes equal to 30% of the number of collected scenarios.• Each stakeholder casts his or her votes publicly.• The evaluation leader orders the scenarios by vote total • The evaluation leader orders the scenarios by vote total and looks for a sharp drop-off in the number of votes

7. Brainstorm and Prioritize Scenarios7. Brainstorm and Prioritize Scenarios

• The prioritized list of brainstormed scenarios is • The prioritized list of brainstormed scenarios is compared with those from the utility tree exercise.

• If they agree, it indicates good alignment between what the architect had in mind and what the stakeholders actually wanted. what the stakeholders actually wanted.

• If additional driving scenarios are discovered, this may itself be a risk showing that there was this may itself be a risk showing that there was some disagreement in goals between the stakeholders and the architect.stakeholders and the architect.


• The architect explains how relevant architectural • The architect explains how relevant architectural decisions contribute to realizing each scenario from step 7.

• Ideally this activity will be dominated by the architect's explanation of scenarios in terms of previously discussed architectural approaches.previously discussed architectural approaches.

• In this step the evaluation team performs the same activities as in step 6, mapping the same activities as in step 6, mapping the highest-ranked, newly generated scenarios onto the architectural artifacts uncovered thus far.the architectural artifacts uncovered thus far.

9. Present Results9. Present Results

• Finally, the collected information from the ATAM • Finally, the collected information from the ATAM needs to be summarized and presented once again to stakeholders.

• In this presentation the evaluation leader recapitulates the steps of the ATAM and all the information collected in the steps of the method, information collected in the steps of the method, including the business context, driving requirements, constraints, and architecture.requirements, constraints, and architecture.


• Then the following outputs are presented:• Then the following outputs are presented:

▫ The architectural approaches documented

▫ The set of scenarios and their prioritization from ▫ The set of scenarios and their prioritization from the brainstorming

▫ The utility tree

▫ The risks discovered▫ The risks discovered

▫ The nonrisks documented

▫ The sensitivity points and tradeoff points found▫ The sensitivity points and tradeoff points found


• The evaluation team adds value by grouping • The evaluation team adds value by grouping risks into risk themes, based on some common underlying concern or systemic deficiency.

▫ Ex: a group of risks about inadequate or out-of-date documentation might be grouped into a risk theme stating that documentation is given theme stating that documentation is given insufficient consideration.

• For each risk theme, the evaluation team • For each risk theme, the evaluation team identifies which of the business drivers listed in step 2 are affected.

Using the Limited Time of Evaluation Using the Limited Time of Evaluation

Effectively• We identified limited time as one of the main • We identified limited time as one of the main problems in conducting an architectural evaluation.• ATAM show how can it solves the problem• The business goals are used as motivation for the • The business goals are used as motivation for the collection of scenarios that represent the utility tree.• Scenarios are prioritized, essentially, as a bottom-up check on the top-down scenario generation of the check on the top-down scenario generation of the utility tree.• Only the high-priority and difficult scenarios are • Only the high-priority and difficult scenarios are analyzed. • These are the areas that will yield the most important results.important results.

PHASE 3: Follow-upPHASE 3: Follow-up

• The tangible output of the ATAM is a final report that contains a list of risks, nonrisks, sensitivity points, and tradeoff points.of risks, nonrisks, sensitivity points, and tradeoff points.

• It also contains a catalog of architectural approaches used, the utility tree and brainstormed scenarios, and the record of analysis of each selected scenario.

• Finally, the final report contains the set of risk themes identified by • Finally, the final report contains the set of risk themes identified by the evaluation team and an indication of which business drivers are jeopardized by each one.

• Like the presentation of results, we use a boilerplate template that has many of the standard sections (such as a description of the

• Like the presentation of results, we use a boilerplate template that has many of the standard sections (such as a description of the ATAM) completed and templates for other sections ready to be filled in.

• We also write some of the final report—for instance, the utility tree • We also write some of the final report—for instance, the utility tree and step 6 analysis—during the hiatus between phases 1 and 2.

• Preparation pays off; whereas it used to take about two weeks to produce a final report for an ATAM client, we can now produce a high-quality comprehensive report in about two days.high-quality comprehensive report in about two days.

SummarySummary

• The ATAM is a robust method for evaluating software architectures. architectures. • It works by having project decision makers and stakeholders articulate a precise list of quality attribute requirements (in the form of scenarios) and by requirements (in the form of scenarios) and by illuminating the architectural decisions relevant to carrying out each high-priority scenario.• The decisions can be cast as risks or nonrisks to find any • The decisions can be cast as risks or nonrisks to find any trouble spots in the architecture.• In addition to understanding what the ATAM is, it is also important to understand what it is not.important to understand what it is not.▫ The ATAM is not an evaluation of requirements.▫ The ATAM is not a code evaluation.▫ The ATAM does not include actual system testing.The ATAM is not a precise instrument, but identifies ▫ The ATAM does not include actual system testing.▫ The ATAM is not a precise instrument, but identifies possible areas of risk within the architecture.

software archiecture lecture10

Education

evaluation

evaluation

project decision

analyze architectural

decision makers

quality attribute

evaluation

final report