measuring is success measuring is success: quest for the dependent variable in is research the...

Measuring IS Success

Measuring IS Success: Quest for the Dependent Variable in IS

ResearchThe Journey

Special IT Research Seminar

William H. DeLoneKogod School of Business

February 8, 2008

IT Research SeminarFebruary 10, 2003


IS Success Research Stream

• UCLA Dissertation – Successful use of IS by SMEs; MISQ 1988

• IS Success: Quest for the Dependent Variable; ISR 1992 = “DeLone & McLean Success Model”

• Ten Year Update; JMIS, Spring 2003• Measuring E-Commerce Success, International

Journal of Electronic Commerce, Fall 2004• IS Success Models, Dimensions, Measures, and

Interrelationships, under review European Journal of Information Systems


Research Motivation

• UCLA measurement course (Mason & Swanson) • Peter Keen’s 5 research challenges for MIS (ICIS

1) – What is the dependent variable? How does MIS establish a cumulative tradition?

• Dependent variable for PhD study in SMEs; Use & Impact

• If you can’t measure it, you can’t research it


Quest for MIS Dependent Variable (ISR, 1992) – “IS Success”Purposes• Organize & summarize MIS research related to

defining the dependent variable• Measure progress on defining the dependent

variable• Improve IS research practice • Contribute to “Cumulative Tradition” –

compare apples to apples


Theoretical Underpinnings

Mason’s 1978 article on measuring information outputProduction->Product->Receipt->Influence on Recipient->Influence on System Mason’s work was based on Shannon & Weaver’s 1949 Theory of

Communications book – Levels of communications measurement = technical, semantic, effectiveness

DeLone & McLean Success Categories System Quality->Information Quality-> Use->Satisfaction->Individual Impact-> Organization Impact


Methodolgy

• Literature review

• IS Articles from 1981 to 1988

• Framework/model for organizing success measures

• Empirical measures grouped into six success categories


D & M IS Success Model

UserSatisfaction

Use

IndividualImpact

Organizational

Impact

SystemQuality

Information

Quality

Figure 1 D&M IS Success Model


Results/Conclusions

• A simple and parsimonious framework for organizing IS success measures

• IS Success - multi-dimensional and interdependent construct

• Selection of measures is contingent on objectives and context of study

• Reduce IS Success measures; build on existing measures=> “cumulative tradition” – comparison of results

• Need for organizational impact measures


The Challenge

“ This success model clearly needs further development and validation before it could serve as a basis for the selection of appropriate IS measures.” (p.88)


Ten-Year Update (JMIS 2003)

Purposes• Model Utility• Validate the Model – Causal relationships• Update the Model to recognize the changes in

IS• Assess progress in IS success measurement


Utility of D&M Success Model

• Cited by more than 285 refereed journal and proceedings articles between 1993 and 2002 (according to a recent study in CAIS vol. 20, ISR 1992 article is the most cited article in MIS over the last 15 years; > 400 citations)

• Most articles used the Model “as a drunkard uses a lamppost for support rather than illumination.” Statement was rejected by editor.


Model Validation

• Seddon & Kiew (1994) validated 4 of the proposed associations

• Rai et. al. validated overall model using goodness of fit tests (ISR 2002)

• Fifteen additional empirical studies validated one or more proposed associations


Updated Model

• IS move from production function to production & service function => importance of service quality

• Impacts on whom? Individuals, groups, org., industry, economy => Net Benefits dimension with contextual definition


INFORMATION QUALITY

SYSTEM QUALITY

SERVICE QUALITY

INTENTION TO USE

USE

NET BENEFITS

USER SATISFACTION

Updated Model


Assessment & Conclusions

• D&M IS Success Model supported and validated• More careful attention to multidimensionality of IS

Success• Confusion between Independent & Dependent variables• Operationalization of the model is contextual (Seddon

et. al.)• Progress in parsimonious measure development is slow• System Use is misunderstood and undervalued• Use and satisfaction are not a substitute for Net Benefits

measures (Yuthas & Young, 1998)


Recent Advances in Success Measurement


Application of Model: E-Commerce Success (IJEC 2004)

• Premise: E-commerce does not need a new measurement paradigm but some new measures

• Apply the D&M IS Success Model for measuring E-Commerce Success

• Literature Review and classification of emerging e-commerce effectiveness measures

• Case examples of application of IS Success Model to E-Commerce success measurement


Application of Model: ERP Success

Article by Sedera & Gable (ICIS 2004) – Government & University ERPs

Most comprehensive empirical test of modelFour dimensions of IS Success –

System quality Information quality Individual impact Organizational impact

** 27 Item measures


Validated Measures for IS Success Source: Sedera and Gable (2004)

System Quality - ease of use, ease of learning, user requirements, system features, system accuracy, flexibility, sophistication, integration, and customization

Information Quality – availability, usability, understandability, relevance, format, and conciseness

Individual Impact – learning, awareness/recall, decision effectiveness, and individual productivity

Organizational Impact - organizational costs, staff requirements, cost reduction, overall productivity, improved outcomes/outputs, increased capacity, e-Government, and business process change

What happened to Use and User Satisfaction?


Measuring Systems/Information Usage (Don Marchand)

• Don Marchand, Professor of Strategy & Information Management at IMD – Switzerland

• 20% of value realization is in deployment: 80% of value realization is in information and IT USAGE

• Challenge – Information Usage is difficult to see, measure and manage


Measuring System Usage (Burton-Jones & Straub, ISR 2006)

• Problem – over-simplified measures of use; e.g. duration of use and breadth of use

• Importance of context & purpose • Elements of usage include: systems, user and task• Proposed Dimensions of Systems Usage –

Cognitive Absorption (engagement) + Deep Structure (use of system features that support a specific task)

• Systems Usage empirically related to task performance


Current Research

Measuring IS Success: Models, Dimensions, Measures and Interrelationships under review at European Journal of Information Systems

• Assessing the state of IS Success Measurement (via D & M Model)

• Summarizing empirical literature, 1992 to 2006• Contributions

Summarizes measures used for each dimension of success

Validates significant relationships for 10 of the 15 causal relationships in the Updated Success Model based on empirical studies


Conclusions

• DeLone & McLean IS Success Model remains the most popular, comprehensive framework for guiding the development of the dependent variable in IS research and for comparing results

• More IS researchers are using the D&M Model to inform and guide their measurement of the dependent variable rather than to merely justify their choice of measures

• Information quality and information use are under studied

• Satisfaction is over used as a surrogate for success; therefore much information is lost

• Bias toward ease of data collection threatens rigor and understanding

measuring is success measuring is success: quest for the dependent variable in is research the...

Documents