Practical Profiles for Managing Systems Engineering R&D

Download Practical Profiles for Managing Systems Engineering R&D

Post on 24-Sep-2016




1 download

Embed Size (px)



    Practical Profiles for Managing SystemsEngineering R&D

    William W. Agresti, Senior Member, IEEE, and Richard M. Harris, Senior Member, IEEE

    AbstractA new and practical way of profiling R&D projectsis presented. The operational context is the investment of discre-tionary R&D funds by systems engineering firms. In this workprogram positioning profile, R&D projects are identified as one offour types: targeting, reinforcing, enabling, or remodeling, accord-ing to the relationship between the R&D project and the systemsengineering work program. A portfolio map is introduced to pro-vide a visual representation of a set of R&D projects and to showlikely transitions between project types over time. The new profileand portfolio map can support management efforts to character-ize and assess an R&D portfolio for its contribution to ongoingsystems engineering work. In addition to this new project pro-file, other profiling strategies are discussed in the framework ofan R&D balanced scorecard. The multiple profiles are a naturalconsequence of the inherent performance multidimensionality ofthe R&D activity in the wide-ranging discipline and practice ofsystems engineering.

    Index TermsBalanced scorecard (BSC), R&D management,R&D project portfolio, R&D projects.


    THIS PAPER introduces a new and practical way of profil-ing R&D projects in systems engineering. This new pro-file evolved over several years of experience by the authorsand colleagues who were responsible for evaluating proposalsfor R&D investments, making recommendations and decisionson projects to fund, reviewing the progress of ongoing R&Dprojects, and briefing executives on the nature, content, struc-ture, and potential benefits of the R&D portfolio. This last ac-tivity, attempting to characterize collections of R&D projectsalong different dimensions, was especially a motivator to createimproved profiling techniques. Projects were being discussedin terms of dimensions such as risk and expected reward, butthere was a need to be able to discuss the set of projects as theyrelated to each other and to the mission of the organization. Fur-thermore, there was a need to create concise representations forsenior management to help explain how the investments in R&Dprojects may be seen as supporting the current work programand expanding it in the future. This new profile emerged, in atrial-and-error process, from years of professional practice as amanagement aid that was very useful for characterization andevaluation of the R&D portfolio.

    Manuscript received March 6, 2007; revised March 4, 2008, June 2, 2008,and July 18, 2008. First published March 21, 2009; current version publishedApril 17, 2009. Review of this manuscript was arranged by Department EditorJ. K. Pinto.

    W. W. Agresti is with Carey Business School, Johns Hopkins University,Baltimore, MD 21201 USA (e-mail:

    R. M. Harris, retired, Huntingtown, MD 20639 USA (e-mail:

    Digital Object Identifier 10.1109/TEM.2009.2013825

    The specific notion of a profile and the motivation for it shouldbe made clear at the outset. By a profile is meant a representationintended to highlight certain characteristics of the entities beingexamined. Profiles have been shown to be useful in representingportfolios of R&D projects because the collection of projectsand the entire R&D program needs to be understood from manydifferent perspectives and for different audiences and stakehold-ers. For example, an obvious profile of interest would representthe projects according to their relationship to marketable prod-ucts and services. Certain projects may be pursuing technologiesthat could lead to products within, say, a two- to three-year timeframe. Other projects may be investigating foundational sciencethat underlies an organizations products, so these projects re-flect the pursuit of basic science. The path from such projectsto actual products is much longer. A corresponding profile thenwould highlight this dimension, the basic-to-applied nature ofthe projects, to show the relative number of projects and amountof R&D funds plotted along this dimension. Such a portrayalwill facilitate understanding, for example, if the portfolio ap-pears to be more strongly invested in long-term R&D than wasthe firms objective.

    If profiles of R&D portfolios are in common use, why isanother one being proposed? There are three reasons. The profileintroduced here:

    1) is oriented to systems engineering and to organizationsthat provide related products and services;

    2) highlights the relationship of the R&D projects to theongoing professional work program;

    3) has been demonstrated to be useful in the practice of man-aging R&D programs.

    Characterization of R&D portfolios in this contextorientedto the systems engineering domain and highlighting the rela-tionship to the work programposes specific challenges thatare discussed in this paper. A new profile addressing thesechallenges may be a useful contribution to the R&D and sys-tems engineering management communities. In the process ofintroducing the new profile, this paper will review and orga-nize other approaches for characterizing collections of R&Dprojects.

    The remainder of this paper is organized as follows. Thissection concludes with a review of relevant literature and prac-tice and a discussion of the systems engineering R&D envi-ronment. Section II introduces the new work program posi-tioning profile and a visualization of R&D projects, called aportfolio map. Section III reviews other R&D project profil-ing approaches and uses a balanced scorecard (BSC) to or-ganize them for an R&D setting. Section IV discusses theconclusions.

    0018-9391/$25.00 2009 IEEE


    A. Review of the Literature and PracticeAn abundant literature exists in R&D project selection and

    portfolio structuring [1][4]. Often the focus is on developingand refining decision models that can be quite sophisticated,such as those based on real options [5], game theory [6], the-ory of constraints [4], and mathematical programming [7], [8].The purpose in citing the use of these approaches to decisionmaking is not to discuss them here; references are provided tosupport further inquiry. Instead, the purpose is to highlight thestriking difference between the sophistication of such publishedmethods and the relative simplicity of methods that are actuallyused in professional practice. Schmidt and Freeland note that, inthe literature, Project selection has traditionally been formal-ized as a constrained optimization problem, but that ClassicalR&D project-selection models have been virtually ignored byindustry [9]. As an indication that very simple methods are theones being used in practice, a survey by Cooper et al. found that40% of businesses use bubble diagrams, in which the projectsare represented as circles (bubbles) and simply plotted on a 2-Dgrid, based on dimensions such as risk and reward [1].

    More specifically relevant to the current research are the manypublished reports on strategies for profiling, structuring, andcharacterizing portfolios of R&D projects. Both the researchliterature and professional practice reveal many different profilesbeing proposed and used. The abundance of profiles is due to themultidimensionality of the R&D activity. As Osama observes,the problem of performance multidimensionality . . . is mostsevere in research and development (R&D) settings due to theinherent multi-dimensionality of R&Ds output and the long-term and intangible nature of the process itself [10], p. vii].The notion here of multidimensionality is simply that the R&Dactivity cannot be evaluated by a single performance metric.And because there are multiple dimensions and metrics of theR&D performance, it makes sense that it might be helpful to usemultiple profiles to characterize the various dimensions.

    For profiling R&D projects, financial measures are the mostwidely used methods, by 77.3% of businesses [1]. The R&Dprojects strategic fit with corporate priorities is also a verypopular profiling dimension, with 64.8% of companies usingthis approach [1]. In addition to profiles having cross-industryapplicability, profiles that are industry-specific may be particu-larly relevant. For example, in the pharmaceutical and biotechindustries, a meaningful profile of R&D projects is a 2-D mapshowing how each project is rated on efficacy versus safety [11].Within an industry, individual organizations establish profilesthat are important to them. For example, the research divisionof PEMEX, Mexicos oil and gas company, tracks 26 technol-ogy areas. It has identified four of the areas as core technologyarenas, with the remaining classified as maintenance arenas. Itprofiles its projects as being core or maintenance because it hasa policy to allocate 75% of its resources to core arenas [12].

    B. R&D in Systems Engineering FirmsAn immediate challenge for effective R&D in systems engi-

    neering is the breadth of the discipline and associated profes-

    sional practice. Definitions from the literature and the leadingprofessional society in the field illustrate this breadth, definingsystems engineering as:

    1) . . . the design, production, and maintenance of trustwor-thy systems within cost and time constraints [13, p. 10];

    2) an interdisciplinary approach and means to enable therealization of successful systems [14].

    R&D projects in support of systems engineering can reflectthe breadth indicated in these definitions. Within the technologyspace alone, projects can include the entire range of comput-ing, telecommunications, sensor, and information technologyhardware, software, and systems, as well as technologies asso-ciated with application domains of interest to the firm and itscustomers. Attempts to have the R&D program address the un-derlying sciences and potential innovations across such a widespectrum can lead to a portfolio of too many projects that arepoorly funded.

    So as not to focus on the R&D at individual companies,consider the research agenda of the International Council onSystems Engineering (INCOSE) [15]. The following are exam-ples of topics from the INCOSE research agenda, selected toconvey the wide range of R&D possibilities [15].

    1) Develop models of the production, integration, andflow of information within the system developmentprocess.

    2) Search for a grand unification theory for the developmentof man-made systems.

    3) Compare and contrast allocation methods and techniquesfor allocating functions to components in design basedupon performance and cost requirements.

    4) Develop interface management tools that includerequirements allocation, and performance and costtrades.

    5) Develop formal methods for stating requirements and aformal proof-of-correctness method for requirements.

    6) Create a mathematical representation of a systems in-puts, outputs, functions, and components for use in systemspecification and design.

    7) Develop a robust application method of decision theoryfor system trade-offs.

    8) Develop a tool that implements the integrated risk anal-ysis method for product architecture development.

    9) Examine and contrast theories of human systems andhow they create products.

    10) Conduct empirical research of alternate methods for in-stilling a shared vision into a team.

    11) Develop a standard interface that allows SE tools to in-tegrate with legacy and COTS products.

    Systems engineering, as used in this paper, reflects thisbreadth of considerations, which, in turn, further motivatesthe need for organizations to be able to characterize theirR&D projects in multiple ways in an attempt to match thisbreadth. For example, one profiling dimension may show thedistribution of R&D funds across the technology, human,management, quality, cost, and process aspects of systemsengineering.


    In addition to the breadth issue, another R&D challenge insystems engineering firms, when contrasted with R&D in otherorganizations, is that the funds available for the entirety ofdiscretionary investments may be very modest. Firms that donot have margins generated by product lines and instead relysolely on professional services typically have limited funds forR&D. For example, using 2005 data in the IEEE Spectrum R&D100 [16], the top ten companies in R&D expenditures as a per-centage of sales averaged 18.9%, compared to 4.01% for the topten systems integrators.1

    Much of the R&D literature is oriented to operational con-texts different from that of systems engineering. For example,manufacturing companies may rely on their R&D for process in-novations to become more cost-competitive. In pharmaceuticalcompanies, R&D is counted on as the source of new productsand is routinely positioned at the earliest stage of a typicallylengthy product development life cycle. The benefits of the R&Dinvestments are in the value created downstream.

    With systems engineering, the value chain to revenue is notgenerally so clear. While the outcomes of systems engineeringR&D projects may indeed translate into new products or ser-vices, the nature of the projects may mean that the results are dif-fused as innovative techniques or capabilities throughout a workprogram and across a base of customers. The mission of the firmis to deliver the highest quality systems engineering products,systems, and services. The extent to which customers want orexpect highly innovative, state-of-the-art solutions varies widelybecause customers vary and the nature of the systems they needvary as well. Some customers want systems that use more ma-ture and less risky technology. Key to understanding the role ofinnovation is the risk appetite of the customer, the technical andmanagement challenges inherent in the planned systems, theperceived benefit of being positioned as a leading-edge technol-ogy provider, and the comfort level relying upon immature butpromising technologies.

    The preceding discussion sets the context for viewing thispaper as different for its focus on systems engineering R&Dand its bias for practical approaches. It is proposed that therelationship of the R&D program to the work program of a sys-tems engineering firm is worth exploring, with the idea that awell-conceived and managed R&D program may be critical tothe success of the firm. In particular, this paper offers a newand practical way of profiling R&D projects in systems engi-neering firms, highlighting the positioning of the R&D projectsrelative to the systems engineering work program. While the ob-servations here are intended to be applicable to R&D programsgenerally, the specific experience base that led to this profilingmethod is the authors management of in-house R&D, some-times referred to as internal R&D (or IRAD) programs at severalorganizations. In these R&D programs, decisions are made t...