a strategy-based method of assessing information technology investments

A strategy-based method ofassessing information technology

investmentsMarisa Analia Sanchez

Departamento de Ciencias de la Administracion, Universidad Nacional del Sur,Bahıa Blanca, Argentina

Antonio Carlos Gastaud MacadaEscola de Administracao, Universidade Federal do Rio Grande do Sul,

Porto Alegre, Brazil, and

Marcela del Valle SagardoyAPCO Management Consultant, Milan, Italy

Abstract

Purpose – The purpose of the paper is to present a theoretical framework and the preliminary resultsof a research on how to assess information technology (IT) investments so as to deliver maximumbusiness value.

Design/methodology/approach – To see whether IT projects fit strategy, the Strategy Mapprovides a framework for defining the portfolio value and data envelopment analysis (DEA) is used tomeasure the efficiency of project portfolios. Subsequently, an application that illustrates the value ofthe framework is described.

Findings – The authors offer a framework that integrates the Strategy Map and IT project portfoliomanagement (PPM) and suggest that this conceptual framework will allow an organization to enhancethe value of IT investments.

Research limitations/implications – This paper is supported by a case study using secondarydata only.

Practical implications – The suggested method could help CEOs to understand the interactionsbetween projects and strategy and thus supports decision making to prioritize and track ITinvestments. The paper illustrates how the proposed framework is applied. It also provides the basisfor further research.

Originality/value – By explicitly linking IT investment with organizational goals, this approachproduces results that differ from those of previous studies and provides a strategy-based approach to PPM.

Keywords Data envelopment analysis, Information technology investments,Project portfolio management, Strategy maps

Paper type Research paper

1. IntroductionStrategy acts as a fulcrum in the deployment of the firm’s resources in a competitiveenvironment (Harris and Ruefli, 2000) with the aim to generate sustained competitiveadvantage (Bridoux, 2004). The corporate strategy is the input to the creation of

The current issue and full text archive of this journal is available at

www.emeraldinsight.com/1753-8378.htm

Antonio C. Gastaud Macada and Marisa A. Sanchez acknowledge support from Programa deCentros Asociados para el Fortalecimiento de Posgrados CAFP-BA CAPES (Brazil) and SPU(Argentine).

International Journal of ManagingProjects in Business

Vol. 7 No. 1, 2014pp. 43-60

q Emerald Group Publishing Limited1753-8378

DOI 10.1108/IJMPB-12-2012-0073

A strategy-basedmethod

43

initiatives, comprising any number of portfolios of programs and projects, which arethe vehicles for executing the organization’s strategy. According to Young et al. (2012)deficiencies in the way projects are currently selected and managed limit the capabilityto realize strategic goals. A study carried out by Planvieww (Nauyalis and Carlson,2010) shows that managers find it difficult to prioritize resources and to stopdevelopment work on a project. One problem is that few organizations are able toconstantly monitor their projects due to lack of data and information to support suchdecisions. Accordingly, during project portfolio management (PPM), difficulties oftenoccur when attempting to assess information technology (IT) initiatives both duringthe selection process and when tracking developments in progress.

The literature regarding portfolio selection problem and closer to our researchpropose optimization models that allow obtaining efficient portfolio in line with a set ofobjectives pursued by the organization. Some of the main limitations of currentapproaches are derived from the way they calculate business value; and from theirfocus on resource capacity planning as the method to select investments. Very fewproposals indicate how to track portfolio fit to strategy as organizational prioritieschange.

The aim of this paper is to present a theoretical framework and the preliminaryresults of a research on how to assess the strategic value of IT investments. The focusof the research is to assess IT initiatives both during the selection process and whentracking developments in progress.

The rest of this paper is structured as follows: Section 2 provides literaturereview and the research background. Section 3 presents the proposed framework.A demonstrative example illustrating how such a framework could be used isalso presented. Finally, Section 4 discusses the benefits and limitations of ourapproach.

2. Literature reviewIn Kapplan (2005) PPM is defined as a method to govern IT investments throughout theorganization, and manage them so that they provide value. It can be described as theprocess in which ongoing projects are reviewed and updated. Project portfolio selectionis the periodic activity involved in selecting a portfolio of projects, that meets anorganization’s stated objectives without exceeding available resources or violating otherconstraints (Ghasemzahed, 2000). There are many possible methodologies that can beused in selecting a portfolio and evaluating projects. In Archer and Ghasemzadeh (1999)the authors classify project evaluation and selection techniques into three categories:strategic techniques, benefit measurement techniques, and portfolio selection techniques.The first category includes project portfolio matrices where various criteria are shown onone or more displays on two descriptive dimensions (Cooper et al., 1997). The secondcategory consists of techniques that aim to evaluate a project independently of otherprojects and includes net present value, internal rate of return and return on investment.Financial analysis works well in situations where costs and benefits are well defined andcan easily be converted into monetary values, difficulties occur when the subject ofanalysis includes non-tangible benefits. Portfolio selection techniques include scoringmethods, analytic hierarchy process (AHP) based techniques and portfolio matrices.Scoring models use a reduced set of criteria and assign importance weight to eachcriterion. Then each possible project is screened against these criteria and those projects

IJMPB7,1

44

that best fulfill them are selected. The most notorious disadvantage of these techniquesis that they depend on the relevance of the selected criteria and the accuracy of theweight given to them (Pinto, 2010).

In Bible and Bivins (2011) the authors provide a project selection method based on theAHP. The objectives hierarchy in the evaluation model is a representation of thestrategic plan. An alignment matrix shows which candidate projects support whichobjectives from the strategic plan (but not how much or how well). To produce accuratepriorities for the alternatives (candidate projects) with respect to the objectives theysupport, alternatives can be evaluated using pair-wise comparisons, or by using absolutemeasurement scales. Synthesized results of the evaluation provide the relative prioritiesof the candidate projects with respect to achieving a goal. The list of prioritized projectsis used as the input to derive a portfolio using an optimization algorithm (thecombination of projects that provides the greatest benefit for the budget level).

In Pinto and Slevin (1988) and Shenhar et al. (2002) the authors proposemeasurement models based on critical success factors.

Recently, real option theory has applied financial option theory in studies into ITinvestments with the aim of quantifying the value of management flexibility in a worldof uncertainty. Research into real options is mainly concerned with the identification ofvarious options in IT investments, and then framing their valuation, and theinterpretation of the results as pricing problems (Chen et al., 2009). Under the binomialmodel, five parameters are needed to determine the option price. These are the currentstock price, the strike price, the time to expiration, the volatility of the stock price, andthe free-risk interest rate. The volatility of the stock price is a statistical measure of thestock price fluctuation over a specific period of time. Real options approaches aregenerally based on the assumption that financial markets provide valuableinformation sources to assess the market uncertainties. A common solution is tofind a publicly owned firm operating in the same market, which is assumed to besubject to the same market risks (Benaroch and Kauffman, 1999). However, volatility inIT projects depends on a complex interaction of factors such as the company’s internaloperations, and is by changes in process, technology, people, organization and culture.

Norrie (2006) adds the balanced strategic measurement dimension to existing PPMmethods. The purpose of which is to enable consistent relative individual project scoringin relation to both financial and non-financial measures and outcomes appropriate to theorganization’s strategy. To put the idea into practice, Norrie proposes a scoring modelusing a non-linear scale that defines the contribution an individual project makes to themeasures associated with each quadrant of the balanced scorecard (BSC). Since projectcontributions are measured individually in each dimension, the approach does not ensurethat a portfolio fits the whole organizational strategy. Hence this begs some questions,i.e. whether a portfolio is sufficient to satisfy strategic goals, or how much should aportfolio improve to support a strategy?

In Niebecker et al. (2008) the authors extend the work of Norrie (2006) and develop astrategy-based scorecard concept to monitor and control collaborative projects in orderto measure their performance, and to manage risks in automotive industry. Theconcept of the strategy map is applied to a product development project. Limitationsare similar to those described for Norrie (2006).

In Cho and Moon (2006) the authors propose a method for selecting the optimalportfolio of performance improvement projects in a manufacturing system. The phases


45

of the approach include a definition of a prioritized set of performance measures for thesystem of interest; and an assessment of satisfaction of the target values ofperformance measures by a project portfolio (one if satisfied, zero otherwise). The latterprovides information to build a function that depicts the strategic value of a portfolio.Finally, a genetic algorithm is used to select the mix of projects that maximizes theproposed fitness function. The fitness function includes a strategic value term andadds up a penalty value imposed for violating constraints.

Kremmel et al. (2010) present an approach to describe software project portfolioswith a set of multi-objective criteria using the COCOMO II model and introduce amulti-objective evolutionary approach to find the pareto optimal front efficiently. Oneof the main contributions is the consideration of restrictions, strategic alignment,balancing of risks, potential portfolio return, among other goals. The strategicalignment values of the project set are calculated using a scoring model with weightedstrategies. This approach created quantitative strategic alignment values which can beused as another objective to maximize.

In Tohumcu and Karasakal (2010) the authors develop an approach based onanalytic network process (ANP) and data envelopment analysis (DEA) to evaluate theperformance of research and development projects. Interdependency among criteriawas treated using hybrid ANP model consisting of both hierarchy and network.Interval weights are formulated as assurance region constraints in a DEA model,through which the project ranking is obtained.

Gutjahr et al. (2010) develop a multi-objective optimization model for projectportfolio selection taking employee competencies and their evolution into account.

Carazo et al. (2010) propose a multi-objective binary programming model to aid inthe selection and scheduling of project portfolios. One of the main contributions of theapproach is that it allows projects to start at different times depending on anyrequirement which involves timing issues. Due to the complexity of the model theauthors propose to solve it using a meta-heuristic procedure.

2.1 Integration of DEA with the BSCDEA, first proposed by Charnes et al. (1978), is a non-parametric technique used tomeasure the efficiency of decision making units (DMUs). Each DMU is seen as beingengaged in a transformation process, in which, some inputs (resources) are used to tryto produce some outputs (goods or services). DEA uses all the available data toconstruct a best practice empirical frontier to which each inefficient DMU is compared.DEA applications involve a wide range of contexts, such as education, health care,banking, armed forces, auditing, and organization effectiveness, among others. Theinterested reader is refereed to Cook and Seiford (2009) and Cooper and Seiford (2004)for a comprehensive review of DEA models.

Most studies that combine the BSC and DEA are descriptive and classificatory orhave been carried out to test theoretical propositions. In the context of projectevaluation, Eilat et al. (2006, 2008) proposed using a methodology based on DEA andBSC to evaluate R&D projects. In the first paper a methodology is proposed fordeveloping and analyzing the efficiency, effectiveness and balance of a portfolio ofR&D projects that mutually interact. Projects are considered DMUs and the scoresobtained are used to establish the list of candidate projects. Alternative portfolios aregenerated based on resource requirements and availabilities. Because of the nature

IJMPB7,1

46

of the portfolios generation procedure, the approach may not produce the optimalprojects mix from an organizational strategic focus. In the second paper, a method isdeveloped for evaluating R&D projects in different stages of their life cycle. Eilat et al.suggest building a BSC for projects. They also propose the inclusion of weightedrestrictions to balance the importance attached to each perspective by setting lowerand upper bounds to a set of variables that are associated with the perspectives of theBSC. However, by setting these bounds the cause and effect relationships hypothesizedin the BSC literature is ignored.

Similar to Eilat et al. (2006), in Asosheh et al. (2010) the authors propose an approachfor IT project selection using DEA as a non-parametric technique for rankingIT projects. The authors define evaluation criteria that should capture all aspects of ITprojects. These criteria are framed as a BSC. Thus, the efficiency scores are based onpre-defined criteria (similar to scoring methods) but do not resemble efficiency withrespect organizational strategy. Measures are used as single and isolated outputs (forexample, cost reduction and security are used as independent outputs withoutacknowledging of the cause and effect relationship between them). DEA results areharder to interpret because they may not necessarily point the root of problems.

3. The proposed assessment frameworkThe proposed framework can be used in the two stages of PPM. First, it supportsportfolio selection because it can be used to evaluate several portfolios and determinewhich are DEA-efficient. An “efficient” portfolio should be interpreted as supportingstrategic plans. Second, once a portfolio is selected DEA is employed to help monitorthe portfolio value.

The framework considers a medium or long term period of analysis tb 2 te for whichCEOs define goals and targets to be met. At time tb a question arises: given a set ofinversion initiatives belonging to different areas, or implementation alternatives (e.g. owndevelopment vs SAP), which portfolio will deliver more value at the time point te?

Portfolio selection. There are two main pieces of information that should begathered:

(1) for each goal (from the strategy map); the contribution that each portfolio canprovide to it at time point te; and

(2) the one-time portfolio implementation costs (infrastructure, labor, services) andthe future operational (and maintenance) expenditures entailed (the costs thatthe organization will incur after closing a project.

The operational expenditures can be approximated to the total costs during the life ofthe IT or IS (but using a standard timeframe for all portfolios and converting futurevalue of costs to the time tb-value equivalent). The portfolio selection may beformulated as a DEA problem where DMUs represent portfolios; inputs representdevelopment and operational costs; outputs represent the contribution of portfolios toeach goal. In this way, DEA results provide a ranking of portfolios based on businessvalue that takes into account the incurred and future spending.

Project monitoring. Projects are monitored in order to track their development(updating costs and benefits estimates to detect deviations) and re-prioritize them whenstrategic goals (or their target values) change or new initiatives appear. Let td,tb , td , te, be a checking point. At td, a project proposal may arise and we wonder


47

how it compares with the rest of the projects; will it rank high and hence receiveresources previously assigned to other projects? At time td we need to assessmanagement’s capacity to re-prioritize projects and decide whether to cancel, postpone,delay or add projects. The strategy map is in constant update as new goals are set orexisting goals are dismissed as a result of changes in organizational strategy. Hence,some projects may become obsolete. Also, newly defined goals may not be supportedby any project. Both of these situations reveal that some projects are not creatingbusiness value. We aim to capture these scenarios by using DEA. Project monitoringmay also be formulated as a DEA problem where DMUs represent projects; inputsrepresent operational expenditures, switching costs (if the project is abandoned) andother development costs (converted to time td value equivalent); outputs relate thecontribution of projects to each goal.

3.1 Conceptual modelIn order to accurately represent the relationships between the benefits delivered bythe set of projects that make up a portfolio and the strategic goals, a more precisespecification for the Strategic Map (Kaplan and Norton, 2004) and a portfolio wasdeveloped. With this conceptual framework the potential and the realized value of aproject can be connected with the strategic objectives. Thus, the conceptual model is thebasis to structure the information necessary for portfolio selection and for monitoringthe implementation of projects.

Figure 1 shows a schematic diagram of the conceptual model that integrates thestrategy map and project portfolios. The formal conceptual model based on UML(Booch et al., 1998) is included in Appendix. A strategy map is made around a set ofthemes. For each objective, there is a set of metrics. The set of action programs that willenable the targets for all the measures to be achieved are referred to Strategic Initiatives.The action plans that define and provide resources for the strategic initiatives must bealigned around the themes contained in the strategy map. The actions plans are anintegrated bundle of projects. A portfolio is composed of many projects. A project mayadd to a goal; it may be neutral or detracting. Projects provide a set of benefits and goal

Figure 1.Conceptual model

IJMPB7,1

48

achievement depends on the realization of these benefits. Benefits are the link betweenprojects and strategic goals. If a strategic objective is not related with any benefit then itis not supported by any project (Figure A1).

The benefits delivered by projects have a potential and a realized value. Goh andKauffman (2006) defines potential value as the maximum feasible payoff of an ITinvestment under efficient production conditions. The realized value is defined as themeasurable value that can be identified after the implementation ensues. Thisrepresentation of IT value emphasizes the consideration of potential value for an ITinvestment both in ex ante project selection and ex post investment evaluation. For thepurpose of our research the potential value is used during portfolio selection and therealized value is used during project monitoring.

As an illustration consider Figure 2 and Table I which show how IT initiatives combineto create a financial payoff from the strategy based on the templates provided by Kaplanand Norton. One operational project (register customer contacts) and two strategic projects

Figure 2.Partial view of a

strategy mapNotes: Strategic theme: customer satisfaction


49

(database marketing and FAQ development) combine to improve customer valueproposition by diversification based on customer needs and providing rapid response.

3.2 DEA models3.2.1 Portfolio selection. Consider D to be a portfolio database, containing n candidateportfolios. Let P ¼ {Pt, 1 # t # n} be the set of portfolios. Let Pt ¼ {pt

k; 1 # k # nt}denote the projects in portfolio Pt where nt is the number of projects and 1 # t # n.Assume projects in Pt deriving mt expenditures and let Xt ¼ {xt

ij : 1 # i # mt; 1 #j # nt} be the set of all expenditures; and producing st outputs and let Yt ¼ {yt

ij :

1 # i # st; 1 # j # nt} be the set of all outputs (forecasted contribution to goals). The

set of outputs is given by the set of all measures that have a “dependency relationship”

with any project benefit.

Strategy map – theme: customer satisfaction Action plan PortfolioGoals Measure Initiative Project Benefits

FinancialGrow revenues Market valueReduce cost percustomerAttract and retainmore customers

Number of newcustomersPercentage of customersplacing repeated orders

CustomerQuality Number of customers

calls for the samecomplaint

Diversify answersbased on customers’needsInternalPrompt access toinformation

Time required to provideinformation relative tobenchmark

Web siteimprovement

FAQdevelopment

Number FAQreplied by operator

Customer satisfactionwith prompt information

Reduce cost pertransaction

Amount reduced ($) Cost reduction pertransaction

Provide rapidresponse

Time delay to replycomplaints

Relationshipmanagement

Relationshipmanagement

Time to answercomplaints



Improve sales datautilization inforecasting

Percentage of sales dataused in forecasting

Databasemarketing

Databasemarketing

Use of historicalsales data inforecasting

Grow cross-sellingoffers

Cross-sell ratio



Table I.Action plan for thestrategic theme customersatisfaction

IJMPB7,1

50

In portfolio selection, since portfolios in P may have different expenditures, considerP 0 ¼ {P

0

t; 1 # t # n} deriving m 0 ¼ jX 0j expenditures where X 0 ¼ <nt¼1Xt is the set of

all expenditures. In other words, X 0 ¼ {xgh : 1 # g # m0; 1 # h # n} (Figure 3).Furthermore, let Y 0 ¼ <n

t¼1Yt; s0 ¼ jY 0j where Y 0 ¼ {ygh : 1 # g # s0; 1 # h # n} is

the set of all outputs. Note that each portfolio P0

t , 1 # t # n, is the same as Pt

except that it has all the expenditures in X0 (this is necessary to satisfy the DEAassumption that units consume the same type of resources). If a portfolio does notrequire spending Xt then assume Xt assumes a value close to zero.

The model assumes an output orientation and is computed using super-efficiencyslack-based measure (SBM VRS). In most DEA models, the best performers share thefull efficient status denoted by the score unity (1). In practice, multiple DMUs usuallyhave this “efficient” status. The super-efficiency model discriminates between theseefficient DMUs. The formulation of the DEA problem as proposed by Tone (2001) is asfollows. We assume that the data set is positive, i.e. X 0 . 0 and Y 0 . 0. The productionpossibility set is P ¼ {ðx; yÞ · x $ X 0l; y # Y 0l; l $ 0}, where l is a non-negativevector in R n. Consider an expression for describing a certain DMU (x0, y0) as:

x0 ¼ X 0lþ s2 y0 ¼ Y 0lþ sþ

with (l $ 0, s 2 $ 0 and s þ $ 0. The vectors s 2 [ R m 0

and s þ [ R m 0

indicate theinput excess and output shortfall of this expression, respectively, and are called slacks.Let us define a production set P/(x0, y0) spanned by (X0, Y 0) excluding (x0, y0), i.e.:

P=ðx0; y0Þ ¼ ð�x; �yÞj�x $Xn

j¼1;–0

ljxj; �y #Xn

j¼1;–0

ljyj; �y $ 0; l $ 0

( )

The super-efficiency score ðd*0 Þ is evaluated by solving the following program:

d*0 ¼ min d ¼1

1=sPs

r¼1�yr=yro

ð1Þ

Figure 3.Scheme of DEA-based

portfolio selection

ProjectsProject costs and

expenditures

PortfoliosPortfolio costs and

expenditures …

… … …

…

…

… … …

…

…


51

subject to:

�x $Xn

j¼1;–0

ljxj �y #Xn

j¼1;–0

ljyj �x ¼ x0; 0 # �y # y0; l $ 0

Assume n DMUs, each consuming m 0 inputs and producing s0 outputs. In our context,we regard these DMUs as portfolios. Let Yr 0 be the amount of output r generated byunit 0 and lh be the intensity variable for DMU h. The score d obtained from thesolution to this linear programming problem measures the maximum output surplusesthat are achieved by a specific efficient DMU compared with the remaining DMUs.By solving model (1) n times (each time evaluating a different DMU at the objectivefunction) we get the relative efficiency scores for all the DMUs with respect to thestrategy map.

3.2.2 Project monitoring. Once a portfolio has been selected, say Pt ¼ {ptk; 1 #

k # y t} , the focus shifts to project performance. The DEA problem is formulated asfollows. Each project in Pt defines two DMUs: one DMU represents the ongoing projectand input and output data are given by incurred costs and by realized value(implemented projects), or updated costs forecasts and value if project is not closed; theother DMU represents the planned project and input and output data are given byestimated expenditures and planned value. Hence, based on definitions provided inSection 3.2.1 assume that there are 2*nt DMUs. Also, there may be additionalDMUs representing projects that started before selecting portfolio Pt. Ideally, DMUsrepresenting planned projects would define the efficient frontier and would be thereference set for ongoing projects.

Since the purpose of the analysis is to assess contribution to strategic goals, weconsider “current” goals and we gather DEA data as follows: outputs are given bymeasures of current strategic goals, and inputs are given by all expenditures derivedfrom on-going projects. More formally, let M ¼ {mi, 1 # i # s} where mi represents ameasure related with a current goal in the strategy map and s is the number ofmeasures. The set of outputs for DEA is given by M.

There is an issue regarding resource management of an organization. During thetracking process we can identify a project which should be closed. If such a project isclosed, the people involved can be shifted to another project. If those people areinternally employed they may experience a lack of motivation due to the failure toachieve the projected results of their previous activity. On the other hand, if people areexternal, there may be contractual clauses that inhibit the closure of the project. In anycase, we propose to quantify such switching costs and include it as an input variable inthe DEA analysis.

3.3 Illustrative exampleThe example presented in Section 3.1 (Figure 2 and Table I) is used to illustrate DEA.We assume there are five portfolios consisting of a number of projects and that thereare sufficient funds available for any of these portfolios. In this case, the modelcaptures the internal process perspective by including two inputs and five outputs. It isdesirable that the number of DMUs exceeds the number of inputs and outputs severaltimes (Cooper et al., 2006). However, since the focus of this example is on

IJMPB7,1

52

describing how to apply DEA we used a small number of portfolios and projects inorder to keep the explanation as simple as possible.

3.3.1 Portfolio selection. The inputs are “costs” (one-time implementation costs) and“operational expenditures” (labor and services). The outputs used are “time required toprovide information” (the web site improvement is supposed to provide properinformation and then customers do not need to call the operator), “customer satisfactionwith prompt information”; “time delay to reply to complaints”, “percentage of sales dataused in forecasting”, “cross-sell ratio” and “cost reduction per transaction” which measurethe strategic goals as described in Table II. The cost per transaction is computed inmonetary units and is the aggregation of the reduction in labor (less call center operatorsand less time to answer complaints), and lost sales due to lack of market knowledge.

The DEA model was run following the super-efficiency formulation, withoutput-orientation, in order to obtain relative performance scores for the five portfoliosconsidered. Table III includes the data used. As can be seen from Table IV, results may

Super-efficiency SBM BCCDMU Score Rank Score Rank

Portfolio 1 1.64570676 2 1 1Portfolio 2 1.16185251 3 1 4Portfolio 3 2.22131083 1 1 1Portfolio 4 0.44511746 4 0.8259587 5Portfolio 5 1.00 £ 10204 5 1 1

Table IV.Portfolio selection:

DEA efficiency scores

Inputs Outputs

Portfolio CostsOperationalexpenditures

1/timerequired to

provideinformation

Customersatisfactionwith promptinformation

1/timerequired to

answercomplaints

Percentageof sales

data used

Cross-sell

ratio

Costreduction

pertransaction

1 10.00 120.00 0.08 3.00 0.04 10.00 1.00 60.002 150.00 197.00 0.10 8.00 0.10 50.00 5.00 400.003 210.00 132.00 0.08 6.00 0.08 100.00 10.00 720.004 105.00 180.00 0.07 3.00 0.07 20.00 1.00 150.005 10.00 120.00 0.03 1.00 0.04 10.00 1.00 0.00

Table III.Portfolio selection, input

and output data

Portfolio Projects

1 Web site improvement (minor adjustments)2 Web site improvement (major development), relationship management (major

development), database marketing (minor adjustments), payroll system (major perfectivemaintenance)

3 Web site improvement, relationship management, database marketing (major development)4 Web site improvement (minor adjustments), relationship management5 Payroll system (major perfective maintenance)

Table II.IT portfolios


53

be considered an appropriate representation of the portfolios. Those DMUs consideredgood performers (portfolios 1, 2, and 3) include projects that support strategic goals.DMU 4 does not support all goals. DMU 5 includes only a payroll system that is notlinked to any strategic goal. The scores given by the Charnes et al. (1978) (Charnes,Cooper and Rhodes (CCR)) model are included. To summarize, the ranking is consistentwith the value added by each portfolio.

3.3.2 Project monitoring. We now turn into the monitoring phase. Assume thatportfolio 3 is selected. Suppose that there is a new project proposal aimed at supporting aloyalty program. The company offers a loyalty program that rewards frequent buyerswith discounts and promotions. However, the manually-driven loyalty program is slow tocapitalize on sales opportunities and is confusing for store staff and customers. Then aninitiative aimed at supporting the loyalty program arises that will have impact on “timerequired to provide information”, “customer satisfaction with prompt information” and“cost reduction per transaction”. Table V shows the data and Table VI summarize theresults: DMUs that represent planned projects define the efficiency frontier; DMUs thatrepresent the current realization of projects have lower scores; and the “loyalty program”has the highest score. The relationship management project (realized value) has a scoreless than its planned counterpart since the budget was exceeded.

We can also include switching costs as an input variable in the DEA. Suppose that therelationship management and database management projects have high switching costsand the web site improvement has no switching costs. Table VII shows the results:the loyalty program project does not have the highest score; the relationship managementproject is ranked first due to the costs of canceling the project implementation.

4. ConclusionsThe aim of this research was to present a theoretical framework and the preliminaryresults of a study into how to assess IT investments to creating deliverables of maximumbusiness value. The approach focuses on delivering value and results. However, theproposal also links the cost of implementing new projects and the ongoing operationalspending they entail. We assume that the organization defines its strategy using astrategy map. That assumption is not restrictive given the widespread use of this tool inorganizations of all types and sizes. We describe the impact of projects on goals (usingthe strategy map as a guide). Thus, the achievement of objectives can be seen to dependon the realization of the projects’ predicted benefits. Based on this conceptual model weformulate two decision problems to be solved using DEA.

In the portfolio selection problem, portfolios are the decision units, and the efficientones are those that fit the strategy, that is those whose benefits better contribute to therealization of goals. In the project monitoring decision problem, projects are thedecision units and the higher scores are for those that remain aligned to organizationalgoals. Since DEA outputs represent up-to-date goals, any project whose benefit doesnot add up to goals will get a low score. As a result, with this approach the long-termsustainability of IT investments can be evaluated in relation to the long-term strategiesof the CEO.

The proposed approach extends previous research by:. allowing IT value measurement close to the locus of value; and. introducing a systematic and formal approach to rank IT portfolios based on

contribution to the organizational strategy.

IJMPB7,1

54

Inp

uts

Ou

tpu

ts

Pro

ject

sC

osts

Op

erat

ion

alex

pen

dit

ure

s

1/T

ime

req

uir

edto

pro

vid

ein

form

atio

n

Cu

stom

ersa

tisf

acti

onw

ith

pro

mp

tin

form

atio

n

1/T

ime

req

uir

edto

answ

erco

mp

lain

ts

Per

cen

tag

eof

sale

sd

ata

use

d

Cro

ss-

sell

rati

o

Cos

tre

du

ctio

np

ertr

ansa

ctio

n

Web

site

imp

rov

emen

t(P

)10

.00

12.0

00.

086.

000.

0410

.00

1.00

150.

00W

ebsi

teim

pro

vem

ent

(R)

10.0

012

.00

0.08

6.00

0.04

10.0

01.

0015

0.00

Rel

atio

nsh

ipm

anag

emen

t(P

)10

0.00

60.0

00.

031.

0010

0.00

10.0

01.

0090

.00

Rel

atio

nsh

ipm

anag

emen

t(R

)11

0.00

60.0

00.

031.

0010

0.00

10.0

01.

0090

.00

Dat

abas

em

ark

etin

g(P

)10

0.00

60.0

00.

031.

000.

0412

5.00

10.0

048

0.00

Dat

abas

em

ark

etin

g(R

)10

0.00

60.0

00.

031.

000.

0412

5.00

10.0

048

0.00

Loy

alty

pro

gra

m10

.00

50.0

00.

055.

000.

0410

.00

2.00

160.

00

Notes:

P–

pla

nn

ed;

R–

real

ized

Table V.Project monitoring, input

and output data


55

In Davern and Wilkin (2010) the authors address the issue of measuring the value of IT.They explain that one stream of research, which draws on financial accounting andeconomics, employs independently observable measures, such as capital marketreactions, return on assets and changes in market share, to assess the value of IT in anorganization. Another stream of research, which draws on the behavioral sciences,uses more subjective measures such as assimilation, user satisfaction, perceived netbenefit, and perceptions and expectations of quality (Davern and Kauffman, 2000). A keycontribution of our framework is that it facilitates an understanding of the process of ITvalue creation, from the immediately perceived effects of a project right up to financialmeasures. The framework allows the use of both measurements close to the locus ofvalue (using the explicit links between projects and goals) and observable measures.

To our best knowledge, proposals that integrate BSC and DEA, suggest using a BSCto capture aspects of projects and then projects fitting to IT plan is assessed. In order toselect portfolios that contribute to the organizational strategy and to track subsequentalignment we propose to use the organizational strategy map. This is grounded on C £ Olevel need to select and monitor projects that drive the strategic priorities of the firm.

Our work differs from Eilat et al. (2006) in many ways. First, Eilat et al. suggestbuilding a specific BSC for projects. Our approach advocates the use of theorganizational strategy map with the purpose of assessing IT investments fit to strategy.Second, in Eilat et al. (2006) candidate portfolios are generated based on resourcerequirements; hence the projects mix that maximizes business value may never begenerated. A portfolio consisting of projects selected according to a priority order maynot fit the overall organizational strategy. In our work alternative portfolios representreal proposals (with the additional advantage that a reduced set needs to be evaluated).

This proposal differs from Kremmel et al. (2010) because Kremmel et al. offers aportfolio optimized based on multiple goals being alignment one of them; our work

DMU Score

Web site improvement (P) 1Web site improvement (R) 1Relationship management (P) 1.01694222Relationship management (R) 1Database marketing (P) 1Database marketing (R) 1Loyalty program 1.10344828

Table VI.Project monitoring:DEA efficiency scores

DMU Score

Web site improvement (P) 1Web site improvement (R) 1Relationship management (P) 1.17427447Relationship management (R) 1Database marketing (P) 1Database marketing (R) 1Loyalty program 1.10344828

Table VII.Project monitoring:DEA efficiency scores(switching costs)

IJMPB7,1

56

stresses efficiency based on IT investments fit to strategy. Kremmel et al. emphasizeoperational issues (distribution of scarce resources between projects), our work givesanswers to concerns at the strategic level (what is top priority to implementing thestrategy). Budget may not necessarily constrain portfolios. An innovative orentrepreneurial company would surely consider the option of seeking investors todevelop promising projects or cutting edge products.

Finally, some limitations of the approach should be acknowledged. When defininggoals in the strategy maps, targets for their related measures are defined. In theapproach we do not punish projects that exceed targets (this situation may give raise to awaste of resources). A further issue is concerned with DEA efficiency scores. Thesuper-efficient methodology can give “specialized” DMUs an excessively high ranking.Another problem lies with the infeasibility issue, which if it occurs, means that thesuper-efficient technique cannot provide a complete ranking of all DMUs. Some authorsdeveloped super-efficient models that avoid these problems (Adler et al., 2002;Mehrabian et al., 1999).

While the proposed framework provides a good starting point for further research,more empirical research on real cases is needed in order to guide the development ofsupport tools. Particularly, more experience on the application of the approach wouldgive insight about how to deal with uncertainty in the decision process. Manyauthors consider uncertainty as a risk metric. But this approach would provide aportfolio ranking unable to score high promising proposals when they may giveraise to undesirable risks. More research is needed to create a satisfactory solution tothis issue.

References

Adler, N., Friedman, L. and Sinuany-Stern, Z. (2002), “Review of ranking methods in the dataenvelopment analysis context”, European Journal of Operational Research, Vol. 140,pp. 249-265.

Archer, N.P. and Ghasemzadeh, F. (1999), “An integrated framework for project portfolioselection”, International Journal of Project Management, Vol. 17 No. 4, pp. 207-216.

Asosheh, A., Nalchigar, S. and Jamporazmey, M. (2010), “Information technology projectevaluation: an integrated data envelopment analysis and balanced scorecard approach”,Expert Systems with Applications – An International Journal, Vol. 37, pp. 5931-5938.

Benaroch, M. and Kauffman, R.J. (1999), “A case for using real options analysis to evaluateinformation technology investments”, Information Systems Research, Vol. 10 No. 1, pp. 70-86.

Bible, M. and Bivins, S. (2011), Mastering Project Portfolio Management, J. Ross Publishing, Inc.,Fort Lauderdale, FL.

Booch, G., Jacobson, I. and Rumbaugh, J. (1998), The Unified Modeling Language User Guide,Addison-Wesley, Boston, MA.

Bridoux, F. (2004), A Resource-Base Approach to Performance and Competition: An Overview ofThe Connections Between Resources and Competition, Belgium Institut et de Gestion,Universite Catholique de Louvain, Louvain.

Carazo, A., Gomez, T., Molina, J., Hernandez-Dıaz, A., Guerrero, F. and Caballero, R. (2010),“Solving a comprehensive model for multiobjective project portfolio selection”,Computers & Operations Research, Vol. 37 No. 4, pp. 630-639.

Charnes, A., Cooper, W. and Rhodes, E. (1978), “Measuring the efficiency of decision makingunits”, European Journal of Operational Research, Vol. 2, pp. 428-444.


57

Chen, T., Jinlong, Z. and Kin-Keung, L. (2009), “An integrated real options evaluating model forinformation technology projects under multiple risks”, International Journal of ProjectManagement, Vol. 27 No. 8, pp. 776-786.

Cho, K. and Moon, B. (2006), “A method for selecting the optimal portfolio of performanceimprovement projects in a manufactiring system”, International Journal of IndustrialEngineering, Vol. 13 No. 1, pp. 61-70.

Cook, W. and Seiford, L. (2009), “Data envelopment analysis (DEA) – thirty years on”, EuropeanJournal of Operational Research, Vol. 192, pp. 1-17.

Cooper, R., Edgett, S. and Kleinschmidt, E. (1997), “Portfolio management in new products:lessons from the leaders – I”, Research Technology Management, Vol. 40 No. 5, pp. 16-28.

Cooper, W. and Seiford, L. (2004), Handbook on Data Envelopment Analysis, Kluwer, New York, NY.

Cooper, W., Seiford, L. and Tone, K. (2006), Introduction to Data Envelopment Analysis andIts Uses, Springer ScienceþBusiness Media, New York, NY.

Davern, M. and Kauffman, R. (2000), “Discovering value and realizing potential from ITinvestments”, Journal of Management Information Systems, Vol. 16 No. 4, pp. 121-144.

Davern, M. and Wilkin, C. (2010), “Towards and integrated view of IT value measurement”,International Journal of Accounting Information Systems, Vol. 11, pp. 42-60.

Eilat, H., Golany, B. and Shtub, A. (2006), “Constructing and evaluating balanced portfolios ofR&D projects with interactions: a DEA based methodology”, European Journal ofOperational Research, Vol. 172 No. 3, pp. 1018-1039.

Eilat, H., Golany, B. and Shtub, A. (2008), “R&D projects evaluation: an integrated DEA andbalanced scorecard approach”, Omega, Vol. 36 No. 5, pp. 895-912.

Ghasemzahed, F. (2000), “Project portfolio selection through decision support”, Decision SupportSystems, Vol. 29, pp. 73-88.

Goh, K. and Kauffman, R. (2006), Measuring the Potential and Realized Value of IT, WISE, Doha.

Gutjahr, W., Katzensteiner, S., Reiter, P., Stummer, C. and Denk, M. (2010), “Multi-objectivedecision analysis for competence-oriented project portfolio selection”, European Journal ofOperational Research, Vol. 205 No. 3, pp. 670-679.

Harris, I. and Ruefli, T. (2000), “The strategy/structure debate: an examination of the perfomanceimplications”, Journal of Management Studies, Vol. 34 No. 4, pp. 587-603.

Kaplan, R. and Norton, R. (2004), Strategy Maps. Converting Intangible Assets into TangibleOutcomes, Harvard Business School Press, Boston, MA.

Kapplan, J. (2005), Strategic IT Portfolio Management: Governing Enterprise Transformation,PRTM, Lexington, KY.

Kremmel, T., Kubalık, J. and Biffl, S. (2010), “Software project portfolio optimization with adancedmultiobjective evolutionary algorithms”, AppliedSoft Computing, Vol. 11 No. 1, pp. 1416-1426.

Mehrabian, S., Alirezaee, M. and Jahanshahloo, G. (1999), “A complete efficiency ranking ofdecision making units in data envelopment analysis”, Computational Optimization andApplications, Vol. 14, pp. 261-266.

Nauyalis, C. and Carlson, M. (2010), “Portfolio pain points-new study reveals that componiesare suffering from a lack of streamlined product portfolio management processes”, PDMAVisions Magazine, pp. 13-18.

Niebecker, K., Eager, D. and Kubitza, K. (2008), “Inproving cross-company project managementperfomance with collaborative project scorecard”, International Journal of ManagingProjects in Business, Vol. 1 No. 3, pp. 368-386.

IJMPB7,1

58

Norrie, J. (2006), Improving Results of Project Portfolio Management in the Public Sector using aBalanced Strategic Scoring Model, Royal Melbourne Institute of Technology, School ofProperty, Construction and Project Management, Design and Social Context,RMIT University, Melbourne.

Pinto, J. (2010), Project Management: Achieving Competitive Advantage, Prentice-Hall,Englewood Cliffs, NJ.

Pinto, J. and Slevin, D. (1988), “Critical success factors across the project life cycle”, ProjectManagement Journal, Vol. 19 No. 3, pp. 67-75.

Shenhar, A., Tishler, A. and Dvir, D. (2002), “Refining the search for project success factors:a multivariate, typological approach”, R&D Management, Vol. 32 No. 2, pp. 111-126.

Tohumcu, Z. and Karasakal, E. (2010), “R&D project performance evaluation withmultiple and interdependent criteria”, IEEE Transactions on Engineering Management,Vol. 57 No. 4, pp. 620-633.

Tone, K. (2001), “A slacks-based measure of efficiency in data envelopment analysis”, EuropeanJournal of Operation Research, Vol. 130, pp. 498-509.

Young, R., Young, M., Jordan, E. and O’Connor, P. (2012), “Is strategy being implementedthrough projects? Contrary evidence from a leader in new public management”,International Journal of Project Management, Vol. 30 No. 8, pp. 887-900.

Appendix

Figure A1.Conceptual model based

on UML


59

About the authorsMarisa Analia Sanchez received a PhD in computer science from Universidad Nacional del Surin Bahıa Blanca, Argentina. She is currently a Professor of graduate and post-graduate coursesat the Department of Management Sciences at Universidad Nacional del Sur. Her researchinterests include project management and business process modelling. She is also interested insystem dynamics. Marisa Analia Sanchez is the corresponding author and can be contacted at:[email protected]

Antonio Carlos Gastaud Macada is a Professor of information systems in the School ofManagement at Universidade Federal do Rio Grande do Sul (UFRGS), Brazil. He earned his PhDin management information system from UFRGS in, 2001. His main research fields includemanagement information system, information technology portfolio, information governance andsupply chain management.

Marcela del Valle Sagardoy holds a MIB Master in International Business of Ecole des Pontset Chaussees au Paris, France and Belgrano University of Buenos Aires, Argentine. She has adegree in information technology of Catholic University, Argentine. Her degree’s thesis,“Alerte_Actif”, won the First Worldwide Award from TotalErg Spa (1998) as the best innovationmethodology to control risks and prevent environmental catastrophes. She began her career in1997 in Siemens for later moving in Europe where she worked into the Engineering andInformation Technology area for automotive and energy industries. She won the XXIVMarisa Bellisario Prize 2012 as National Woman Testimonial for excellent empowerment andmanagement abilities. She currently works as APCO Management Consultant in Italy.

To purchase reprints of this article please e-mail: [email protected] visit our web site for further details: www.emeraldinsight.com/reprints

IJMPB7,1

60

a strategy-based method of assessing information technology investments

Documents