incremental return on incremental investment: engenio…biglever.com/extras/oopsla06_lsi.pdf ·...
TRANSCRIPT
Incremental Return on Incremental Investment:Engenio’s Transition to Software Product Line PracticeWilliam A. Hetrick
Engenio Storage Group - LSI Logic3718 N. Rock Road
Wichita, KS 67226 USA+1-316-636-8433
Charles W. KruegerBigLever Software
10500 Laurel Hill CoveAustin, TX 78730 USA
+1-512-426-2227
Joseph G. MooreEngenio Storage Group - LSI Logic
3718 N. Rock RoadWichita, KS 67226 USA
+1-316-636-8501
ABSTRACTEngenio made the transition to software product line practice inorder to keep pace with growing business demand for its products.By using an incremental transition strategy, Engenio avoided thetypical upfront adoption barrier – the equivalent developmenteffort of 2 to 3 standalone products – which in their case wasprojected to be 900 to 1350 developer-months. Engeniodiscovered that by making an upfront investment of only 4developer-months, they were able to start a chain reaction inwhich the tactical and strategic incremental returns quicklyoutpaced the incremental investments, making the transition payfor itself.
Categories and Subject DescriptorsD.2.13 [Software Engineering]: Reusable Software – domainengineering, reusable libraries, and reuse models.
General TermsDesign, Economics, Management, Measurement, Theory.
KeywordsSoftware Product Lines, Transition to Software Product LinePractice. Incremental Methods.
1. INTRODUCTIONEngenio faced a daunting challenge. The business demand for itsRAID storage server products was rapidly out pacing their abilityto create, evolve and maintain the firmware for those products[1].Software product line practice, which was just coming into viewat the time, seemed to be the answer. But there was an apparentbarrier. Prevalent in the software product line literature at the timewas the assumption that it requires an upfront investment of 2 to 3products worth of development effort in order to see return onthose investments[2,3]. Based on metrics from two recentdeployments, Engenio’s projected upfront investment was 900 to
1350 developer-months of effort, with a development staff of 100.There was absolutely no slack in Engenio’s contractually-dictatedproduction schedules to divert this amount of upfront effort to re-analyze and re-architect controllers, re-engineer andcomponentize assets, redesign production infrastructure, re-defineprocesses, and re-organize management and development teams.
Rather than attempt to surmount this formidable upfront adoptionbarrier, Engenio chose instead to make an incremental transitioninto product line practice. Not incremental in the sense ofprolonging the time to make the large upfront investments andsimilarly prolonging the time to see returns. But rather,incremental in the sense of making a series of small investments,each of which would immediately yield the order-of-magnitudereturns that are characteristic of software product line practice.
Engenio discovered that with a relatively small upfrontinvestment of 4 developer-months for its first incrementalinvestment, the cumulative returns began to quickly outpace thecumulative investments in terms of time, effort and money. As aresult, the return on investment was almost immediate and theresulting “profits” in time, effort and money from the return oninvestment could be used to fuel the remaining incrementaltransition.
There are, of course, many different facets within a softwaredevelopment organization to consider when making anincremental transition to software product lines. For example,Clements and Northrop characterize 29 key practice areas thatmay be impacted by a software product line approach[2]. Any orall of these might be considered in an incremental transition.Engenio chose to incrementally address, in sequence, those facetsthat represented the primary inefficiencies and bottlenecks in theirdevelopment organization. By eliminating the inefficiencies andbottlenecks in the most critical facet, the next most criticalproduct line problem in the sequence was exposed and targetedfor the next increment.
To date, the sequence of incremental steps in the Engeniotransition to software product line practice can be characterizedby the following four primary stages:
• Transition the infrastructure from conventional configurationmanagement and builds to first class software product linevariation management, configuration management andautomated production
Copyright is held by the author/owner(s).OOPSLA’06 October 22–26, 2006, Portland, Oregon, USA.ACM 1-59593-491-X/06/0010.
• Transition from team organization by products to teamorganization by core assets
• Transition from development processes defined by productreleases to development process defined by feature releases
• Transition from validation and quality assurance for individualproducts to validation and quality assurance for all of thesoftware product line assets
As of the writing of this paper, Engenio is in the fourthincremental stage.
2. BACKGROUNDIn today's customer-driven environment, most companies targetthe needs of their prospective customers by creating a product line– a portfolio of closely related products with variations in featuresand functions – rather than just a single product. For companiesthat utilize standalone or embedded software in their products,this poses a serious problem. Tools and techniques for softwaredevelopment tend to focus on individual products. As a result,developing software for a product line is extremely complex dueto multiple intertwined products, features and productiondeadlines – all aimed at moving target markets. These tacticalsoftware development challenges are big enough to impede therealization of business-critical goals and strategies. Morespecifically, they can greatly hinder a company’s ability to hitmarket windows, provide competitive pricing, maximize productquality, and expand the scale or scope of their product lineportfolio.
A new class of software development methods, tools andtechniques – collectively referred to as software product linedevelopment – is emerging to address this problem, offeringimprovements in development time-to-market, cost, quality, andportfolio scale and scope[4]. What is most interesting is themagnitude of tactical and strategic improvements that arepossible, not measured in percentage points, but more commonlyin factors of two to ten. These improvements are so large that theyimpact the fundamentals of how companies compete[2].
Manufacturers have long used analogous engineering techniquesto create a product line of similar products using a commonfactory that assembles and configures parts designed to be reusedacross the varying products in the product line. For example,automotive manufacturers can now create tens of thousands ofunique variations of one car model using a single pool of carefullyarchitected parts and one factory specifically designed toconfigure and assemble those parts.
The idea of manufacturing software from reusable parts has beenaround for decades, but success has been elusive. Recentadvances in the software product line field have demonstrated thatnarrow and strategic application of these concepts can yield orderof magnitude improvements in time-to-market, quality, portfolioscalability and software engineering cost. The result is often adiscontinuous jump in competitive business advantage, similar tothat seen when manufacturers adopt mass production and masscustomization paradigms.
The characteristic that distinguishes software product lines fromprevious efforts is predictive versus opportunistic software reuse.Rather than put general software components into a library inhopes that opportunities for reuse will arise, software product
lines only call for software artifacts to be created when reuse ispredicted in one or more products in a well defined productline[4].
Conventional methods of portfolio development typically includead hoc combinations of code-level techniques such as:
• configuration management branching or clone-and-own,which often lead to wasteful duplication and divergence ofeffort
• #ifdefs, templates, file naming conventions or runtimeconditionals with configuration files, which often lead tounmanageable complexity
In contrast, software product line development tools, methods andtechniques give first-class engineering status to the issues ofportfolio development, providing constructs that allowdevelopment organizations to take advantage of the similaritiesamong the products, to effectively manage the variations amongthe products. For example, see [5] for specific tools and methodsutilized by Engenio in this case study.
• Feature modeling language, used by software architects tomodel the features that cause variation among the products ina portfolio and to model the feature profile for each of theproducts in the portfolio.
• Variation points, used by developers to encapsulateimplementation-level differences among the products in theportfolio and the logic about how to instantiate a variationpoint based on the feature profile for a product. Variationpoints, which are use in software assets such as requirements,source code, test cases and documentation, enable a singlesoftware asset with variation points to be used for all productsin a portfolio. This consolidation is what allows the portfolioto be developed as though it were a single system.
• Configurator, used by engineers doing a product build, takes afeature profile for a product and instantiates that product bycomposing assets and by instantiating variation points withinthose assets. The configurator enables continuous builds forthe entire portfolio (analogous to continuous builds ofindividual products in agile methods), where any developercan automatically compose and configure all of the assets(requirements, source code, test cases, documentation) for anyproduct at any time, based on the current state of the assetsunder development. For example, a developer’s change tosource code and test cases can immediately and automaticallybe built and tested in all products in the portfolio.
Software product line development approaches provide a shift inperspective, so that development organization can engineer theirentire portfolio as though it were a single system rather than amultitude of products.
3. INITIAL STATE OF ENGENIO’SDEVELOPMENT PRACTICE
Engenio is in the business of providing feature-rich, high-performance, storage servers to major OEM vendors such as IBM,SGI, Cray, StorageTek and Teradata. Each OEM customer wantsto take advantage of Engenio’s core competence in storagetechnology, but each one also wants unique and differentiatedsolutions. Thus, the need for efficient product line engineering iscentral to its business model. Furthermore, Engenio has recently
completed the process of becoming a standalone division withinits parent company, LSI Logic Corporation (Engenio wasformerly known as LSI Logic Storage Systems), so the ability toextend its product line to attract and retain a growing customerbase is key for establishing investor value.
Engenio’s Controller Firmware Development group is the focalpoint for the evolution to a software product line. In recent years,this group has grown include to 180 developers, working at fourdistinct geographic sites. The group currently provides firmwarefor 82 products, with about one million lines of embeddedsoftware going into each product. Approximately 80% of the codeis common among all products.
In the late 1990s, the Engenio Controller Firmware Developmentstaff lived in far simpler world. The firmware was constructed asa single software build, applicable to all deployments. Anyrequired product variability was resolved at runtime, usingdownloadable configuration data stored in non-volatile memory.Evolution of the firmware over time was managed using theconcept of formal releases. These releases were managed in adisciplined manner, with a formal waterfall process guiding theway. Each release would transition through the stages of analysis,design, implementation and test. Once a release was complete andproducts were shipping to customers, work on the next releasecould begin. In retrospect, it is apparent that it was the sequentialnature of the release cycle that kept the firmware developmentworld simple and sane.
Market pressures eventually emerged that greatly disrupted thesimple, sequential release-based perspective that had guided thefirmware team through the nineties. The 21st century broughtincreased success in the marketplace, but this success wasaccompanied by a furor of development demands that could notbe accommodated by sequential development cycles. The tranquilworld of sequential releases was replaced with the nonstop chaosof a half dozen intertwined and overlapping release cycles.
In the midst of the market-driven turbulence, the need for changewithin the engineering group at Engenio was not readily apparent.Instead, the evolution from a single product to a product linestealthily crept into Engenio’s product development world. Theinitial portent of change was the introduction of a new, low-end,hardware platform. While the feature content of the low-endproduct was to be the same and the bulk of the firmware code wasto be shared between the new low-end and existing mid-rangeproducts, there were substantial differences between the twohardware platforms. The introduction of a second, substantiallyindependent, development environment to support the newplatform was the initial solution for managing this variability.Although this move solved the immediate problem of supportingthe variation among the old and new products, it proved to be aless than ideal way to manage the commonality.
During the evolution from one to multiple products, the primarytool utilized by Engenio’s firmware group was a proven, well-integrated, configuration and problem tracking system. Thissystem did an excellent job in the role for which it was intended,but it created the tendency for product variability problems to betreated as configuration management problems. For example,variability to satisfy differing platform requirements was managedby branched versions of the same source files. Similar branchingwas introduced to support the concurrent development of diverse
features. The end result of the evolution from a single product to adiversity of hardware platforms and firmware features wassuccess in the marketplace, but chaos within the code base.Eventually, approximately 34% of the 3300 files in the Engeniosource code base had anywhere from 2 to 16 active branchesunder development. The resultant branching, merging, duplicationand divergence became the dominant effort in the firmwaredevelopment team, leading to the realization that a change indevelopment practice was imperative.
4. DECISION TO TRANSITION TOSOFTWARE PRODUCT LINEAPPROACH
The negative impact of the rapidly increasing file branchesbecame evident from two complimentary viewpoints, one fromthe architects and one from the managers. First, an architect beganto observe code quality degeneration due to excessive filebranching. As new features were implemented, there was astatistical likelihood that 34% of the impacted files would havebranches (recall that 34% of the files in the code base werebranched). The feature implementation on a branched file had tobe duplicated on each branch of that file, which was laborintensive and highly error prone since each branch of a file mightbe slightly different in subtle ways.
Second, from the managers’ viewpoint, defect resolutions werealso becoming increasingly expensive, since defects resulted in alarge percentage of development capacity spent on analysis,correction, and validation on multiple file branches. The changemanagement tool in use modeled this duplication of defectresolution tasks as clone requests. The development managementteam then used this data to track clone metrics – the ratio of clonerequests versus the total number of change requests. Themanagement team considered clone work a waste item in thedevelopment capacity because the duplication of effort prevents adeveloper from working on new revenue-generating features.
Clone metrics were increasing over time, mirroring the codequality issue observed by the architect. The root cause wasattributed to the file branching, which was a result of the growingnumber products under concurrent development, which in turnwas a direct result of Engenio’s success in their OEM business.
To the architects and managers faced with this growing problem,software product lines intuitively resonated as a viable solution,though no one on the team had direct experience with thisemerging field of practice. Before launching into a full-scalechange initiative, they decided to expand their understanding ofbasic product family development principles with consulting andan assessment by an outside organization specializing in softwareproduct line development tools and practices, BigLeverSoftware[5].
During the assessment, a pilot project plan emerged based on avision of the development and production environment after afull-scale transition. The development and productionenvironment would be based a unified set of core assets, residingon the single common trunk in configuration management – nomore branches. The software product line development tool,BigLever Software Gears[5], would be used to manage variationat the source configuration binding time. Product instantiation, orproduction, would be fully automated, driven from decisions in
the Gears feature model and extracted from the core assets. All ofthe common and varying source code would exist as core assetswith variation points under Gears, so no manual applicationengineering would be required.
Over the next few months, the pilot project plan was refined. Twoproducts were selected to become the initial baseline for coreassets in the product line. The pilot was designed to be an initialtransition step in a deployment, if proven successful.
The pilot project was budgeted as a six-week effort, beginning onMay 17, 2004. After consolidating the two product code sets withthe Gears feature model and variation points, the methods andresults of the prototype production line were shared with all of thedevelopment managers, architects, and their superiors on June 8,2004 – three weeks ahead of schedule. With considerablediscussion and analysis, projecting business requirements andscenarios in the development organization, the group concurredthat the product line environment would eliminate the branchingand clone problem, enabling the development staff better satisfyEngenio’s business and engineering requirements. Developmentmanagement agreed to the final deployment plan on June 17,2004, one month after the beginning of the pilot. After hardeningthe pilot prototype and documenting the new configurationmanagement and production procedures, the new softwareproduct line development environment was put into dailyproduction use by the development staff on July 30, 2004. Theentire effort to get to this point was 4 developer-months.
5. ENGENIO’S TRANSITIONEngenio structured its transition to software product line practiceinto four sequential and incremental stages. Each of these fourstages had nested incremental substructure. The objective was tomake small incremental investments in product line practice –small enough not to disrupt ongoing production schedules – in aneffort to immediately reap larger incremental returns on thoseincremental investments.
The four stages were ordered sequentially to address, in order, themost prominent inefficiencies and bottlenecks first.
• Transition the infrastructure and core assets
• Transition the team organization
• Transition the development processes
• Transition the validation and quality assurance
5.1 Transition of the Infrastructure andSource Code Assets
The most critical issue identified to address first was the growingproblem of file branching degrading quality and productivity. To
accomplish this, the development infrastructure was enhanced tosupport a single consolidated collection of core assets rather thanhighly branched files in configuration management. The subtaskswithin this stage were to:
• validate the approach with a pilot project
• create the software product line development and productioninfrastructure
• incrementally extract a core asset base from existing sourcecode branches
• incrementally transition development teams away from branchand clone methods to core asset development using the newinfrastructure
• incrementally refactor the core assets to maximizecommonality and to optimize abstraction in variation points
When the transition started, eight product releases wereconcurrently in various stages of active development. The pilotproject consolidated the branched source code for two of theseproducts, resulting in the initial core asset base. To minimized thepotential disruption of the critical product releases underdevelopment, the pilot and initial deployment strategicallytargeted two products that were early in their release cycle.
The initial code base for the two products was comprised of 3300files, 1000 of which had anywhere from 2 to 16 branches,organized into 212 directories. The core asset base afterconsolidation was comprised of 3103 common files, 51 Gearsvariation point files, 211 common directories, 1 Gears variationpoint directory, and 0 branches.
The transition plan dictated that the remaining product releasescurrently in flight would be completed using existingbranch/merge/clone methods. All subsequent product and featurereleases were to deploy from the core assets of the product line.As illustrated in Figure 1, over the subsequent 9 months,developers rolled off of projects developed under the legacymethods, while the core assets and production line evolved toinitially deploy 23 products.
Early in this stage of the transition, the Gears variation pointswere unceremoniously extracted from the existing branched codebase with no attempt to re-architect or re-engineer the core assetsother than to encapsulate, characterize, and program variationpoints based on a simple Gears feature model. This allowed, forexample, all 51 variation points to be extracted from the 3300files, 1000 of which were branched, in less than 3 days during thepilot. However, over time, the variation points were incrementallyrevisited in order find opportunities for greater commonality andto find better product line abstractions.
Figure 1. Incremental Transition of People and Products
The incremental investment during the first stage of the transitionwas to establish a new software product line development andproduction infrastructure, create a baseline of core assets from theexisting branched source base, and train developers on the newGears-based product line development environment andmethodology. The initial investment to conduct the pilot projectand then to go into live product line development and productionwith those 2 products was 4 developer-months. Subsequenttransition of the remaining 21 products and product teamsrequired less than one half developer-month per product.
The incremental return on investment originated from improvedefficiencies at the tactical developer level. Developers spent lesstime manipulating the configuration management tool, navigatingand making cloned modifications on intricate file-by-file branchstructures. As a result they could spend more time developingnew features.
The dramatic reduction in branch complexity is illustrated inFigure 2 using a file branch factor metric, defined as the numberof file branches normalized by the number of products. Note fromthe graph that branches have been eliminated in the product linecore asset base.
Figure 2. File Branch Factor During the Transition
Our historical metrics demonstrated a strong correlation betweenthe number of branched files and the number of duplicate defectresolutions required. As a result, the reduction in file branches ledto a derivative return on investment – the elimination of clonedefect resolutions.
Finally, at a strategic level, Engenio quickly gained increasedconfidence in their software product line development capabilities
and, as a result, their ability to respond to more aggressively toexpanding business opportunities. As is typical with softwareproduct line transitions, the strategic returns on investmentovershadow the tactical returns.
5.2 Transition of the OrganizationThe new software product line development environment andtransition to core assets facilitated the development of concurrentproduct releases and also helped with the deployment andvalidation of new features across multiple hardware platforms.However, after the first stage of the transition, most of thedevelopment focus still remained product release centric. Thesecond stage in the incremental transition was to start a shift indevelopment focus to core assets by restructuring thedevelopment organization. Developers on product release teamsincrementally transitioned to a set of teams defined by collectionsof core asset components. The collections of core assets weregrouped by affinities and service layers in the firmwarearchitecture.
Each team in the new organizational structure has a core assetmanager and a technical team leader. The core asset manager isresponsible for making sure that their core assets provide the rightcapabilities at the right time on the product release roadmap. Thecore asset technical team lead is responsible for implementingtheir core assets according to the architectural and featurerequirements on the product roadmap. Core asset managers andteam leads also have an internal focus with their team on coreasset stewardship, or maintaining and optimizing the integrity oftheir core assets. Core asset stewardship includes tasks such asrefactoring to optimize commonality and variability, maintainingan appropriate balance for the requirements tradeoffs across theentire product line, and searching for emerging abstractions in thevariation points[6].
Asset team staffing was determined by existing and planneddomain expertise needs of the clustered components and the assetevolution requirements for new features. This was a difficulttransition culturally as developers, who were accustomed toworking on any file in the code base, were now expected torequest changes from the domain experts of other asset teams.
The investment for this increment was the planning to define theasset teams and training to educate team members, team leaders,and development managers in the roles and responsibilities of theasset team roles.
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
Aug-04
Sep-04
Oct-04
Nov-04
Dec-04
Jan-05
Feb-05
Mar-05
Apr-05
May-05
% d
evelo
pers
in
vo
lved
in
SP
L
0
5
10
15
20
25
Aug-04
Sep-04
Oct-04
Nov-04
Dec-04
Jan-05
Feb-05
Mar-05
Apr-05
May-05
# o
f p
rod
uct
s b
uil
t fr
om
pro
du
ct l
ine
0%
5%
10%
15%
20%
25%
30%
35%
40%
1Q2002
2Q2002
3Q2002
4Q2002
1Q2003
2Q2003
3Q2003
4Q2003
1Q2004
2Q2004
3Q2004
4Q2004
1Q2005
2Q2005
Avera
ge %
bra
nch
ed
files
per
pro
du
ct
The return on that investment was better planning and a morecontrolled and efficient asset evolution. This has resulted in betterdesigns and improved defect resolutions due to improvedvisibility into the product family and core asset requirements.Technical experts are making better generalized technicaldecisions for the assets they own, based on a deeper knowledge ofthose components. Having teams with ownership and deepexpertise in a narrow collection of core assets ultimately reducedthe rework in both designs and defect resolutions.
Although it has not been formally characterized yet, there is clearevidence that the feature development capacity within theorganization has been increased by at least 50%. Rather thanscrambling to reallocate overextending developers to meetproduction schedules, managers now have the luxury of choosingwhere to allocate this newly gained development capacity.
5.3 Transition of the Development ProcessesOne of the difficulties in transitioning the development teamfocus to an asset focus was that Engenio’s development processwas release centric. The third stage in the transition was to alignthe development process with product family development. Theprocess needed to include a mapping step from feature and releaserequirements to asset requirements. It needed to increasecommunication effectiveness of product family requirements atthe asset evolution level. The process needed to allow for an assetassembly and feature or product validation function through thedevelopment cycle. Engenio’s process needs extended beyondcompatibility with software product line practices by increasingthe ability to respond to changing customer requirements as wellas addressing the growing geographic distribution of thedevelopment staff.
Engenio made a significant investment to address its processneeds by assembling a task force comprised of its SoftwareEngineering Process Group (SEPG) along with managementrepresentation from other development and quality assuranceroles. The task force held a face-to-face process summit with aclearly defined purpose of creating a software developmentprocess that addressed the needs of the company.
Amazingly enough, the group reached consensus at the summit ona process structure. The process definition was refined andincrementally put into practice. This stage of the transitioncompleted the shift of asset team focus to product familydevelopment, further reducing the development overhead bysatisfying multiple component requirements with coordinateddevelopment efforts and leveraging the domain expertise of assetteam developers.
5.4 Transition of the Validation And QualityAssurance
The fourth incremental transition stage, currently in thedeployment phase, is feature validation and quality assurance.This stage is improving the product engineering capability of theorganization by completing all feature requirements validationiteratively as part of the development work, and shifting theresponsibility of the product certification group to theinteroperability matrix of Engenio’s controller platforms, serversystems, operating systems, network adapters, and networkswitches. This transition is another significant investment,affecting multiple cross functional groups and processes. The
expected return from this investment is to move beyond efficiencyimprovements in development capacity to efficiencyimprovements in product production and release. For example,there are significant opportunities to reduce the time to completefinal product validation, thereby further reducing the time-to-market for new products and new features.
5.5 Future Transition WorkThe ultimate objective is to do software product line developmentat an optimal level of effectiveness and efficiency. After the fourmajor transition stages are complete, we expect to continuallyevolve and search out areas of waste and overhead, though weexpect these to be smaller and more focused optimizations ratherthan major transition efforts.
6. CONCLUSIONS AND LESSONSLEARNED
Engenio’s transition to software product line practicedemonstrated that a large development organization with a largelegacy code base can make the transition to software product linepractice without a large upfront investment and without disruptingongoing production schedules. By investing only 4 developer-months of effort upfront and 12 developer-months overall,Engenio incrementally transitioned 23 products – each comprising1 million lines of code – and 135 developers to a sophisticatedsoftware product line practice. This is two orders of magnitudeless than comparable transitions reported elsewhere.
By staging an incremental transition, small incrementalinvestments very quickly yielded much larger incrementalreturns. These returns – in effort, time and money – could then bepartially or fully reinvested to fuel the next incremental steps inthe transition. Furthermore, the efficiency and effectiveness of thedevelopment organization constantly improved throughout thetransition, demonstrating that development organizations do notneed to take a hit in order to reap the benefits of software productline practice.
Stated a little more strongly, the Engenio transition refutes theconventional wisdom that it takes an upfront transition effortequivalent to developing 2 or 3 standalone products in order toachieve return on investment. Engenio achieved return oninvestment after an incremental investment of only 4 developer-months of effort. In contrast, the conventional wisdom of 2 or 3products predicted the return on investment for Engenio to be 900to 1350 developer-months, or 200 to 300 times greater than thatactually experienced.
The incremental transition approach provided the same strategicbenefits as experienced by waterfall transitions. Following theinitial transition of the infrastructure, organization and processes,the strategic benefits quickly materialized. Over the next 5months, the size of the product line portfolio grew 225% – from23 products to 52 products – expanding at an average rate of anew product every 4 days. At the time of this writing, the productline comprises 82 products.
A big part of this success, we believe, is that Engenio made anexplicit decision to transition to software product line practiceutilizing the emerging knowledge, technology and best practicescoming out of the software product line field. The data and
experiences at Engenio are consistent with others that haveutilized these emerging approaches[7,8].
Another of the lessons learned from this experience is that thetransition in product line practice areas such as infrastructure,core assets, organizational structure, development processes, andQA practices do not have to occur in parallel. Engenioincrementally addressed these in sequential order, choosing tosolve the problems with the largest potential return on investmentfirst. With the sequential approach, after the transition in one areawas complete, the need for transition in the next was more clearlyilluminated. For example, after creating the productioninfrastructure and formal core assets, it was easier to argue whyteams should be organized around core assets and limited tomodifying their core assets rather than be organized aroundproducts and free to modify any source file.
The “people issues” are always the most difficult when impartingchange in an organization[9]. A final lesson learned that we foundto be particularly important was that the incremental transitionmade one aspect of the people issues easier than expected. Byquickly and continually showing benefit, it is much easier to quellthe detractors. In fact, we found that the biggest skeptics anddetractors quickly became the strongest advocates, once theyexperienced the benefits of software product line practice. Byquickly and incrementally showing return on investment,detractors have less time and opportunity to derail the initiative.
7. REFERENCES[1] Engenio Information Technologies, Inc., Milpitas, CA.
http://www.engenio.com
[2] Paul Clements and Linda Northrop. 2001. Software ProductLines: Practice and Patterns, AddisonWesley, Reading, MA.
[3] Davis Weiss and Chi Tau Robert Lai. 1999. SoftwareProduct-line Engineering. Addison-Wesley, Reading, MA.
[4] SoftwareProductLines.com, see Resources section,http://www.SoftwareProductLines.com
[5] BigLever Software, Inc. Austin, TX.http://www.biglever.com
[6] Charles Krueger and Dale Churchett. Eliciting Abstractionsfrom a Software Product Line, in Proceedings of theOOPSLA 2002 PLEES International Workshop on ProductLine Engineering. Seattle, Washington. November 2002,pages 43-48
[7] Ross Buhrdorf, Dale Churchett, Charles Krueger. Salion’sExperience with a Reactive Software Product LineApproach. Proceeding of the 5th International Workshop onProduct Family Engineering. Nov 2003. Siena, Italy.Springer-Verlag LNCS 3014, p 315.
[8] Charles Krueger. Variation Management for SoftwareProduct Lines, Proceeding of the Software Product Lines 2ndInternational Conference, SPLC 2, San Diego, CA, Aug2002, Springer-Verlag LNCS 2379, p 257.
[9] John Kotter. 2002. The Heart of Change, Harvard BusinessSchool Press. Cambridge, MA.