anderson 1997 decision support systems

8/8/2019 Anderson 1997 Decision Support Systems

1/18

E L S E V I E R Decision Support Systems 19 (1997) 75-92

D0em0"

Microcomputer sof tware evaluat ion: An econometr ic modelE v a n E . A n d e r s o n a , . , Y u - M i n C h e n t ,

a Graduate Business Institute, George Mason University, Enterprise Hall. Mail Stop: 5F5, Fairfax, VA 22030-4444, USAb J.C. Penny Company, Inc., Plano, TX, USA

Received March 1993; revised January 1995; accepted July 1995

Abstrac t

Microcomputer software selection is made difficult by the multiplicity of products, variation in product performance, andthe uncertainties of user needs. This paper presents a methodology for the empirical evaluation of competing softwarepackages. The process proposed identifies the most relevant performance attribute set and, through a simultaneous system ofequations, the relative importance of each attribute in explaining the satisfaction of users. The methodology is illustratedusing sample data derived from user evaluations of five different software types: word processing, spreadsheet, data basemanagement systems, communications, and graphics. The applicability of the methodology and the implications of thefindings are discussed.Keywords: Software evaluation and selection; User evaluations; Econometric modeling

1 . I n t r o d u c t i o n

Prior to the commercial development of the mi-crocomputer, many information needs were unmet oronly partially satisfied through mainframe systems.Organizations soon recognized the advantages ofmicrocomputers and the opportunities that they rep-resented for the improvement o f worker productivity[25,59]. In particular, they could be acquired andoperated at a low per unit cost, required little spaceor support, were flexible in their applications, easy tomaintain, and allowed their users a high degree ofautonomy and control over their computing. Theirgreatest productivity potential was at the lower man-agerial-clerical levels of organizations, where therewere substantial needs to store, process, and report

* Corresponding author.

text and data [41]. However, these users had neitherthe skills nor the time to write computer code inprocedural languages. Hence, the rate of acceptancefor microcomputers was closely tied to the commer-cial development of end user software [40].

End user demands for noncomplex, easy ways toinstruct the microcomputer resulted in the develop-ment of several different types of software: wordprocessing, database management, spreadsheet, andothers. Hundreds of competing software productsemerged with different performance attributes andcapabilities. Unfortunately, the growth in the numberand variety of products, with different performanceattributes, created an uncertain evaluation and selec-tion environment for potential buyers [62]. Theseuncertainties were further complicated by the lack ofunderstanding that users frequently had about theirneeds, and because needs changed with user experi-ence and technological advancements [9].

0167-9236/97/$17.00 1997 Elsevier Science B.V. All rights reserved.PII S0167-9236(96)00042-5


2/18

76 E.E. Anderson, Y.-M. Chen / Decision Support Systems 19 (1997) 75- 92

S im i l a r d i f f i c u l t i e s o f e va lua t ion a nd se l e c t ione x i s t i n c o m p u t e r h a r d w a r e , w h e r e n u m e r o u s a p -p r oa c he s t o a t t r i bu t e e va lua t ion a nd c ho ic e o f t e c h -no logy ha ve be e n p r opose d ( e . g . , [ 2 , 49 , 50 ] ) . The sem e t h o d s , h o w e v e r , l i k e t h o s e t h a t e x i s t f o r s o f t w a r ee va lua t ion , t e nd to c ons i s t o f ve r y ge ne r a l c r i t e r i a ,suc h a s good qua l i t y a nd r e l i a b i l i t y , a s w e l l a sspe c i f i c a pp r oa c he s t o t e s t i ng [ 23 , 44 ] . The y a r e ba se do n t h e t e s t s o f i n d i v i d u a l b u y e r s e v a l u a t i n g c o m p e t -ing p r oduc t s one by one , a nd the i r i n s igh t s i n to t heim p l i c a t ions o f t he se t e s t s .

2 . S o f t w a r e e v a l u a t i o n : E n d u s e r c o m p u t i n gT h e d i f f u s i o n o f m i c r o c o m p u t e r s a n d a p p l i c a t i o n s

s o f t w a r e h a s c o n v e r t e d m a n y i n f o r m a t i o n u s e r s t oe n d u s e r s o f c o m p u t e r s . T h e s e e n d u s e r s d e v e l o p ,m a i n t a i n a n d u s e t h e i r o w n i n f o r m a t i o n s y s t e m s .D e s p i t e t h e i r g r e a te r i n v o l v e m e n t in t h e d e s i g n , d e -v e l o p m e n t a n d u s e o f t h e s e s y s t e m s , e n d u s e r c o m -p u t i n g ( E U C ) o c c u r s w i t h v a r y i n g d e g r e e s o f s u c c e s s[ 1, 4] . S o m e o f the s a m e i s sue s a r i s ing in I S e va lua -t i on r e l a t e d to t he i nd iv idua l , t he i r w or k , t he o r ga n i -z a t ion a nd the na tu r e o f a pp l i c a t ion s a r i s e s in EU Ca s w e l l [ 3 , 11 , 15 , 30 ] . A dd i t i ona l ly , e nd u se r s ha ve ave s t e d in t e r e s t i n t he c on f igu r a t ion o f t he i r w or ks t a t i o n , i n c l u d i n g t h e q u a l i t y a n d p e r f o r m a n c e o fso f tw a r e .

T h i s p a p e r p r e s e n t s a m e t h o d o l o g y d e r i v e d f r o mm u l t i v a r i a t e s t a t i s t i c s a n d e c o n o m e t r i c s f o r a n e m -p i r i c a l l y b a s e d a s s e s s m e n t o f s o f t w a r e . I t is n o tin t e nde d to be a n e m p i r i c a l t e s t o f a t he o r e t i c a lm o d e l o f s o f t w a r e p e r f o r m a n c e o r u s e f u l n e s s . I ta c c e p t s t h e m u l t i p l i c i t y o f u s e r e n v i r o n m e n t s , s o f t -w a r e p r o d u c t s , c o m p u t e r t e c h n o l o g i e s a n d a p p l i c a -t i ons , a nd doe s no t im pose a va lue s t r uc tu r e ons o f t w a r e a t tr i b u te s . I t f o c u s e s o n t h e p r o c e s s o fe m p i r i c a l a s s e s s m e n t , a n d s e e k s t o i d e n t i f y a n d o r g a -n i z e m a n y o f t he m e a s u r e m e n t , m o d e l i n g a n d e s t i m a -t ion i s sue s i nvo lve d .

M or e spe c i f i c a l l y , t he a u tho r s de ve lop a s t e p bys t e p a p p r o a c h f o r c o n s i d e r i n g t h e i s s u e s i n v o l v e d i np e r f o r m a n c e a s s e s s m e n t s , d a t a q u a l i t y a n d r e q u i r e -m e n t s , m o d e l s e l e c t i o n a n d e s t i m a t i o n a n d i n t e rp r e t a -t i on . I t i l l u s t r a t e s t he m e thodo logy f o r f i ve d i f f e r e n ts o f t w a r e t y p e s : w o r d p r o c e s s i n g , s p r e a d s h e e t ,d a t a ba s e m a n a g e m e n t s y s t e m s ( D B M S ) , c o m m u n i c a -

t i o n s a n d g r a p h i c s . I t e m p l o y s d a t a f ro m a s a m p l e o fuse r s . The a u tho r s w i l l p r e se n t t he ke y i s sue s , p r ob -l e m s a nd c ho ic e s t ha t a r e i nvo lve d w i th e a c h s t e p o fthe m e thodo logy , a nd i l l u s t r a t e t he s t e ps t h r ough thea pp l i c a t ion .

W e s s e l i u s a n d V e r v e r s a r g u e t h a t s o f t w a r e q u al i tys h o u l d b e d e f i n e d i n " t e r m s o f a s e t o f p r o d u c tc ha r a c t e r i s t i c s ne c e s sa r y f o r t he p r oduc t t o be su i t e df o r s p e c i f i e d u s e " [ 6 1 ] . T h i s d e f i n i t i o n c o n f o r m sw i t h D O D a n d I S O s t a n d a r d s [ 1 8 , 3 2 ] . E r i k s s o n a n dT~S m ha ve sugge s t e d tha t t he e va lu a t ion o f so f tw a r epa c ka ge s shou ld f oc us on th r e e que s t ions ( [ 22 ] , p .157): H ow w e l l ( e a s i l y , r e l i a b ly , e f f i c i e n t ly ) c a n I u se i t

a s i s? H o w e a s y i s it t o m a i n t a i n ( u n d e r s ta n d , m o d i f y ,

a nd r e t e s t ) ? C a n I s t il l u se i t i f I c ha ng e m y e nv i r on m e n t?

I n t h i s pa pe r , w e de f ine u se r s a t i s f a c t ion ( u t i l i t y )a s a f unc t ion o f pe r f o r m a nc e a t t r i bu t e s , a n a pp r oa c ht h a t i s w e l l a c c e p t e d i n e c o n o m i c m o d e l s a n d i n f o r -m a t ion sys t e m s r e se a r c h [ 33 ,35 , 46 ]. C ons ide r a b le a t -t e n t i o n h a s b e e n d e v o t e d t o t h e f o r m u l a t i o n o f u s e rsa t i s f a c t ion [ 7 ,16 , 36 ,47 , 51 ] , c ons t r uc t c r e a t ion a ndte s t i ng [ 33 , 39 , 45 , 48 , 55 ] , a nd to i t s m e a su r e m e n t[ 20 ,58 ] . The o r e t i c a l l y , i f w e a r e g ive n a budg e t f o rso f tw a r e , po t e n t i a l u se r s w ou ld l i ke t o be a b l e t oa s se s s t he p r oba b le c on t r ibu t ions t o s a t i s f a c t ion o fe a c h p r o d u c t a n d c h o o s e t h e o n e t h a t w o u l d m a x i -m iz e t he i r be ne f i t s . S im i l a r ly , ve ndo r s w ou ld l i ke t ouse t he s a m e in f o r m a t ion to be t t e r unde r s t a nd thene e ds a nd e xpe r i e nc e s o f u se r s , so t ha t t he y c a ni m p r o v e t h e d e s i g n a n d / o r s u p p o r t p r o v i d e d f o rthe i r so f tw a r e .

I n t h e a p p l i c a t i o n o f t h e m e t h o d o l o g y p r e s e n t e dhe r e , u se r s a t i s f a c t ion i s m ode le d a s a f unc t ion o f t hef o l low ing so f tw a r e a t t r i bu t e s : ba s i c f unc t ions , a d -v a n c e d f u n c t i o n s , t ra i n in g t i m e , d o c u m e n t a t i o n , e a s eo f u s e a n d v e n d o r s u p p o r t. E a c h o f t h e s e i s d i s c u s s e db e l o w .T h e u n d e r l y i n g c a p a b i l i t i e s o f s o f t w a r e p a c k a g e sa r e c on ta ine d in t he i r f unc t ions [ 26 ] . F unc t ions de -f ine t he dom a in o f a pp l i c a t ion a nd a r e a t t he c o r e o fp r oduc t d i f f e r e n t i a t i on . D a ta p r o R e se a r c h ( da t asou r c e f o r t h i s s tudy ) , f o r e xa m ple , de f ine s t he ba s i cf unc t ions o f w or d p r oc e s s ing so f tw a r e a s : u se r i n t e r -f a c e , t e x t e n t r y , e d i t i ng f unc t ions , f o r m a t t i ng f unc -t i ons a nd p r in t f unc t ions [ 60 ] . A dd i t i ona l f unc t ions


3/18

E.E. Anderson, Y.-M. Chen / Decision Support Systems 19 (1997) 75-92 77

a r e i nc lude d in a dva nc e d f unc t ions , suc h a s ke ys t r okes to r a ge a nd f i l e c onve r s ion s . E r ik s son a nd T~ Srn no tetha t so f tw a r e c ha r a c t e r i s t i c s a r e no t b ina r y a nd pe r -f o r m in va r y ing de g r e e s [ 22 ] . The y sugge s t t ha tc h a r a c t er i s ti c s s h o u l d b e c o n s i d e r e d p r i m a r y a n d s e c -o n d a r y d e p e n d i n g o n t h e e x p e r i e n c e s a n d n e e d s o fthe u se r [ 22 ] . A s im i l a r c on t inuum i s p r opose d byR a i n e r a n d H a r r i s o n , w h o c r e a t e d a n E U C a c t i v i t ysc a l e t ha t i s ba se d in pa r t on the so f tw a r e f unc t ionsused [45] .

O ne w ou ld e xpe c t t ha t l e s s sk i l l e d , i n f r e que n tu s e r s , w i t h s i m p l e a p p l i c a t i o n s w o u l d b e c o n t e n tw i th ba s i c f unc t ions [ 1 , 16 ] . A s u se r s ga in know le dgea nd sk i l l w i th so f tw a r e pa c ka ge s , t he y e x t e nd thesoph i s t i c a t i on a nd r a nge o f a pp l i c a t ions a nd r e lyinc r e a s ing ly on a dva nc e d f unc t ions [ 30 ] .

P r i o r t o t h e e m e r g e n c e o f E U C , t r a i n i n g w a s o n eo f t he f e w va r i a b l e s i n e m p i r i c a l s t ud i e s t ha t c ons i s -t e n t ly c on t r ibu t e d to u se r s a t i s f a c t ion [ 25 , 43 ] . The see a r l i e r f i n d i n g s h a v e b e e n a f f i r m e d i n E U C e n v i r o n -m e n t s [ 4 , 14 ] . The e f f e c t ive ne s s o f t r a in ing i s i n f lu -e nc e d by the f o r m a t a nd sc he d u le o f t ra in ing , t hem e d i u m e m p l o y e d , t h e a b i l i t y a n d p r e p a r a t i o n o fu s e r s , t h e o r g a n i z a t i o n a l e n v i r o n m e n t a n d m a n a g e -m e n t suppo r t f o r t r a in ing [ 10 , 28 , 54 ] . T r a in ing ha sm ul t i p l e e f f e c t s on u se r s [ 17 ] . I t c a n be u se d tom ot iva t e u se r s , r e m ove unc e r t a in t i e s a nd a l l a y f e a r s ,g a i n a c c e p t a n c e f o r t e c h n o l o g y , a n d t o d e m o n s t r a t et h e o p p o r t u n i ti e s f o r j o b g r o w t h a n d a c h i e v e m e n tth r ough i t s a pp l i c a t ion . A dd i t i ona l ly , i t c on t r ibu t e sd i r e c tl y t o th e a d v a n c e m e n t o f k n o w l e d g e a n d u n d e r -s t a nd ing . T r a in ing , t o be m os t e f f e c t ive , m us t bem a n a g e d b y o r g a n i z a t i o n s t o i n s u re t h e t i m e l i n e s s o fc o n t e n t w i t h u s e r n e e d s a n d d e v e l o p m e n t .

T h e c o n t r i b u ti o n o f d o c u m e n t a t i o n t o u s e r s a t is -f a c t ion i s t oo f r e que n t ly t a ke n f o r g r a n t e d . I t i sw r i t t e n t o m o t iva t e , e xp la in , c l a r i f y a nd t e a c h u se r sa bou t so f tw a r e c a pa b i l i t i e s [ 19 , 21 ] . The m or e e f f e c -t i ve i t i s a s a too l o f c om m unic a t io ns , t he m o r equ ic k ly u se r s w i l l ga in sk i l l i n t he so f tw a r e a nd ga inf r om i t s a pp l i c a t ions . To r kz a de h ha s s t a t e d tha t" . . . t h i s e f f ec t i v en e s s [o f d o c u m e n ta t io n ] i s m o r ec r it ic a l i n a n e n d - u s e r c o m p u t i n g e n v i r o n m e n t w h e r eu s e r s b e c o m e m o r e d e p e n d e n t o n d o c u m e n t a t i o n a n dle s s de pe nde n t on in t e r a c t ion w i th a na ly s t s a nd p r o -g r a m m e r s " [ 5 7 ] .

T h e c a s e f o r s y s t e m s u s a b i l i t y i s m a d e s u c c i n c t l yb y G o u l d a n d L e w i s , " A n y s y s t e m d e s i gn e d f o r

p e o p l e t o u s e s h o u l d b e e a s y t o l e a r n ( a n d r e m e m -b e r ) . . . . a n d b e e a s y a n d p l e a s a n t to u s e " ( [2 7 ], p.300 ) . A c tua l , a s w e l l a s pe r c e ive d , e a se o f u se p l a ya n im por t a n t r o l e i n t he a dop t ion a nd u t i l i z a t i on o fc o m m e r c i a l s o f t w a r e p a c k a g e s [ 5 3 ,5 6 ]. U s e r s a t is f a c-t i o n d e r i v e s f r o m u t i l i z a t i o n a n d a c c o m p l i s h m e n t . I ft h e t i m e a n d h a s s l e o f d e v e l o p i n g u s a b l e a p p l i c a t io n se xc e e ds t he u se r ' s t h r e sho ld , d i s sa t i s f a c t ion r e su l t s .E U C i s i n t e n d e d t o b e p r o d u c t i v i t y e n h a n c i n g . A c -c e p t i n g t h e d i v e r s e s k i l l s / k n o w l e d g e / e x p e r i e n c e o fuse r s , t he f a s t e r u se r s a c c om pl i sh t he i r a pp l i c a t ion ,the g r e a t e r w i l l be t he i r s a t i s f a c t ion [ 34 ] .

The f i na l va r i a b l e i nc lude d in t he da t a s e t o f t h i sp a p e r i s v e n d o r s u p p o r t . S u p p o r t f r e q u e n t l y c o m e sf r o m m a n y d i f f e r e n t s o u r c e s , i n c l u d i n g o r g a n i z a -t i o n a l i n f o r m a t i o n c e n t e r s a n d s u p p o r t g r o u p s[ 4 , 24 , 42 ] . N e ve r the l e s s , ve ndo r s ha ve a s t r ong e c o -nom ic inc e n t ive t o f a c i l i t a t e sk i l l de ve lopm e n t a ndp r o b l e m s o l v i n g f o r u s e r s . T h e y e s t a b l i s h n e w s l e t -t e r s , ho t l i ne s , i n f o r m a l u se r g r oups , a nd sponso ra c t iv i t i e s f o r u se r s . V e ndor suppo r t , b r oa d ly de f ine d ,i n c l u d e s p r o d u c t i n n o v a t i o n a n d t i m e l y u p g r a d e s o ft h e s o f t w a r e p r o d u c t . T h e m o r e v e n d o r s a s s i s t u s e r sto r e a l i z e t he i r po t e n t i a l w i th so f tw a r e p r oduc t s , t hem or e s a t i s f i e d t he y a r e w i th t he m .2.1. Methodology

T h e m e t h o d o l o g y p r e s e n t e d i n t h i s p a p e r h a s t w om a jo r ob j e c t ive s . F i r s t , i t s e e ks t o i de n t i f y a s e t o fp e r f o r m a n c e a t t r ib u t e s t h a t p r o v i d e s t h e b e s t concep-tualization of so f tw a r e s e l e c t ion by u se r s . I t i s r a r etha t t h i s w ou ld r e qu i r e da t a on a l l pos s ib l e a t t r i bu t e s .T h e q u e s t i o n i s , " C a n t h e e s s e n ti a l i n f o r m a t i o n a b o u tthe c on t r ibu t ion o f pe r f o r m a nc e a t t r i bu t e s t o s a t i s f a c -t i o n b e e x t r a c t e d f r o m a s u b s e t o f a t t r i b u t e s ? " S e c -ond , u s ing tha t s e t o f a t t r i bu t e s , t he a u tho r s estimateweights f o r t he a t t r i bu t e s t ha t w ou ld e s t a b l i sh t he i rr e l a t i ve im por t a nc e in t he e va lua t ion p r oc e s s . Tha ti s, " W h a t i s t h e re l a ti v e m a g n i t u d e o f t h e m a r g i n a lc on t r ibu t ion to s a t is f a c t ion o f e a c h pe r f o r m a nc e a t -t r i b u t e ? "

T h e a c c o m p l i s h m e n t o f t h e s e o b j e c t i v e s i s m a d ed i f fi c u l t b y t h e n u m b e r o f f e a t u r es a n d c o m p l e x i t y o fso f tw a r e , t he d ive r s i t y o f u se r sk i l l s , e xpe r i e nc e a nde xpe c ta t i ons , t he va r i e ty o f a pp l i c a t ions , a nd by thef a c t t h a t s o f t w a r e s a t i s f a c t i o n m a y b e a " j o i n t p r o d -u c t " [ 8 ] . H e n c e , a s o u n d m e t h o d o l o g y m u s t b e a b l e


4/18

78 E.E. Anderson, Y.-M. Chen / Decision Support Systems 19 (1997) 75-92

t o pa r t i t i on the da t a , t ha t i s , r e duc e i t s r e dunda nc y ,a n d m e a s u r e o r c o n t r o l t h e i n t e r a c t i o n b e t w e e n s o f t -w a r e t y p e s [ 3 8 ] . T h i s p a p e r h a s u s e d p r i n c i p a l c o m -pone n t s a na ly s i s t o i de n t i f y t he pe r f o r m a nc e a t t r i bu t es e t, a n d a s i m u l t a n e o u s s y s t e m o f e q u a t i o n s , w i t he m b e d d e d r e c u r s iv e e q u a t i o n s f o r e a c h s o f t w a r e t y p e ,to de t e r m ine the r e l a t i ve im por t a nc e o f a t t r i bu t e s t ouse r s .

3 . D a t a : S o u r c e s a n d o r g a n i z a t i o nD a t a f o r s o f t w a r e e v a l u a t i o n c a n b a s i c a l l y b e

o b t a i n e d f r o m t h r e e s o u r c e s : e x p e r i m e n t a l s t u d i e s ,u s e r s u r v e y s a n d e x p e r t o p i n i o n . T h e m e t h o d s p r e -se n te d in t h i s pa pe r ge ne r a l ly do no t pe r t a in t oa s s e s s m e n t s b y e x p e r t s , s i n c e t h e r e r a r e l y i s e n o u g hda ta t o suppo r t s t a t i s t i c a l m ode l ing [ 5 , 49 ] . O ur p r i -m a r y f o c u s i n th i s p a p e r is w i t h t h e m o d e l i n g o f d a t ad e r i v e d f r o m u s e r s u r v e y s . H o w e v e r , i t s h o u l d b en o t e d t h a t t h i s m e t h o d o l o g y , p r o v i d e d t h e r e a r e s u f fi -c i e n t obse r va t ions , i s e qua l ly a pp l i c a b le t o e xpe r i -m e n t a l l y g e n e r a t e d d a t a . I n p a r t i c u l a r , e x p e r i m e n t sw o u l d g e n e r a l l y b e c r e a t e d a r o u n d t a s k s . S o f t w a r ea t t r i bu t e s w ou ld be r e p l a c e d by t a sk c ha r a c t e r i s t i c sa n d u s e r s a t i s f a c t i o n w o u l d b e r e p l a c e d b y s o m em e a s u r e o f e f f i c i e n c y , t i m e t o c o m p l e t i o n , c o r r e c t -n e s s , o r o t h e r m e a s u r e s o f p e r f o r m a n c e .

I t is i m p o r t a n t t o r e c o g n i z e t h a t u s e r s u r v e y s m a ybe r e po r t e d in a n a g g r e g a t e f o r m t o p r o t e c t t h eide n t i t y o f r e sponde n t s o r t o s im p l i f y r e po r t i ng . I n -de e d , t he da t a u se d in ou r a pp l i c a t ion a r e ba se d ont h e a v e r a g e a t t r i bu t e e va lua t ions o f t he u se r s o f e a c hs o f t w a r e p a c k a g e . S i n c e th e n u m b e r o f r e s p o n d e n t s( s a m p l e s i z e ) e v a l u a t i n g e a c h p a c k a g e m a y v a r y , t h eq u a l i ty o f m e a n a t tr i b ut e s c o r e s m a y v a r y . A s i ss h o w n b e l o w , t h i s h a s i m p o r t a n t i m p l i c a t i o n s f o rp a r a m e t e r e s t i m a t i o n .3 . 1 . App l i c a t ion

T h e d a t a e m p l o y e d i n t h i s s t u d y w e r e c o l l e c t e d ,c om pi l e d , a nd r e po r t e d by D a ta p r o [ 60 ] . S e ve r a lh u n d r e d u s e r s o f v a r i o u s s o f t w a r e p r o d u c t s w e r ea s k e d t o e v a l u a t e t h e p r o d u c t ( s ) t h e y o w n . U s e r se va lua t e d e a c h a t t r i bu t e a c c o r d ing to t he i r s a t i s f a c -t i o n u s i n g s c a l e d r e sp o n s e s r a n g i n g b e t w e e n 1 ( p o o r )a nd 10 ( e xc e l l e n t ) f o r t he f o l low ing a t t r i bu t e s : ba s i cf u n c t i o n s , d o c u m e n t a t i o n , a d v a n c e d f u n c t i o n s , y e n -

d o r s u p p o r t , e a s e o f ' u s e , a n d t r a i n i n g t i m e , p l u sove r a l l s a t i s f a c t ion . A ppe nd ix A p r e se n t s t he f o r m a to f t h e q u e st i o n n a ir e . T h e n u m b e r o f p r o d u c t s e v a l u -a t e d b y u s e r s w i t h i n e a c h s o f t w a r e t y p e r a n g e d f r o ma m i n i m u m o f t h i r t e e n f o r s p r e a d s h e e t a n d c o m m u n i -c a t i o n s s o f t w a r e , r e s p e c t i v e l y , t o t w e n t y - t h r e e f o rw o r d p r o c e s s i n g . T h e s a m p l e s i z e s r a n g e d f r o mne a r ly t h r e e hund r e d u se r s f o r g r a ph ic s p r oduc t s t oo v e r s e v e n h u n d r e d f o r w o r d p r o c e s s i n g s o f t w a r e .

T a b l e 1 p r e s e n t s a s u m m a r y o f t h e m e a n s a n ds t a nda r d de v ia t i ons f o r t he va r ious a t t r i bu t e s s tud i e d .The a u tho r s ha ve t r e a t e d the ove r a l l s a t i s f a c t ion va r i -a b l e a s e n d o g e n o u s , w i t h t h e p e r f o r m a n c e a t t r i b u t e sa s e xp la na to r y va r i a b l e s . The da t a a r e c r o s s - se c t iona la n d a g g r e g a t e d f r o m u s e r r e s p o n s e s , i . e . , c o n s i s t o fth e a v e r a g e e v a l u a t i o n s f o r e a c h p r o d u c t w i t h i n t h eva r ious so f tw a r e t ype s , r a the r t ha n ind iv idua l r e -sponse s . I t c a n be s e e n f r om Ta b le I t ha t t he h ighe s tm e a n p e r f o r m a n c e r a t i n g ( 8 . 0 0 ) w a s a c h i e v e d b yD B M S f o r b a s i c f u n c t i o n s ~w h i l e t he l ow e s t ( 6 . 67 )w a s r e c e i v e d b y c o m m u n i c a t i o n s s o f t w a r e f o r i t sv e n d o r s u p p o r t. T h e i m p l i c a t i o n s o f d a t a a g g r e g a t i o nf o r c a usa l m ode l s w i l l be d i s c us se d l a t e r .

4 . D a t a : R e d u n d a n c y a n d p a r t i t i o n i n gI n t h e c o n t e x t o f s o f t w a r e e v a l u a t i o n , o n e m a y

e xpe c t t ha t a t t r i bu t e a s se s sm e n t s w i l l be c o r r e l a t e d ,w he the r t he da t a a r e a gg r e ga t e d o r no t . Tha t i s , t hed a t a e x h i b i t a m u t u a l i n t e r d e p e n d e n c e b e t w e e n s o m eof t he pe r f o r m a nc e a t t r i bu t e s [ 8 ] . I n som e c a se s , t hec o r r e l a t i on pa t t e rn i s t he s a m e f o r a ll so f tw a r e t ype s ,a nd in o the r s i t i s qu i t e d i f f e r e n t . F o r e xa m ple , t heda t a o f t h i s s tudy c ons i s t e n t ly e xh ib i t s a h igh c o r r e l a -t i on b e t w e e n ' e a s e o f u s e ' a n d ' t r a i n i n g ' . I t i s r e a -sona b le t o e xpe c t t ha t t r a in ing w i l l e nha nc e a u se r ' sk n o w l e d g e a n d a p p r e c i a t i o n o f a p r o d u c t , m a k i n g i te a s i e r t o u se , a nd a p r oduc t t ha t i s e a s i e r t o u se w i l li nc r e a se t he be ne f i t s o f t r a in ing .T h e p r e s e n c e o f h i g h c o r r e l a t io n s b e t w e e n v a r i -a b l e s s u g g e s t s t h a t t h e y m a y d e r i v e f r o m t h e s a m esou r c e ( s ) . H igh ly c o r r e l a t e d va r i a b l e s qu i t e l i ke ly dono t r e p r e se n t s e pa r a t e f a c to r s , bu t a r e im pe r f e c t r e p -r e s e n t a t i o n s o f o n e a n o t h e r o r s o m e t h i n g e l s e . F u r -t h e r m o r e , r e g r e s s i o n c o e f f i c i e n t s e s t i m a t e d b y l e a s ts q u a r e s m e t h o d s i n t h e p r e s e n c e o f h i g h l y c o r r e l a t e dv a r i a b l e s m a y b e d i f f ic u l t t o i n t er p r et . A s s u m i n g t h a t


5/18

Te1Smyoueuo

Swe

NmoNmo

tya

posrep

s

WodPon2

7

Seh

1

6

DM

1

5

Cmco

1

4

G~pc

1

2

Eaoyvae

E vae

Bc

DmaoA

V

Eo

Tann

Oa

fuo

fuo

su

u

tm

ssao

M

SDbM

SDM

SDM

SDM

SDM

SDM

SD

77

07

72

1072

1070

1275

0970

1576

08

74

09

70

1269

1374

1269

1178

1475

10

80

10

74

1076

1275

1275

1372

1478

10

75

12

69

1371

1566

1473

1375

1272

13

79

12

71

1573

1769

2172

1473

1574

12

z

aScMicomeSw

DaoRcCpaoNwJ

Mc1

bSD=Saddao

I"--4


6/18

8 0 E . E . A n d e r s o n , Y . - M . C h e n / D e c i s i o n S u p p o r t S y s t e m s 1 9 ( I 9 9 7 ) 7 5 - 9 2

t h e o t h e r a s s u m p t i o n s o f o r d i n a r y l e a s t s q u a r e s a r esa t i s f i e d , t he pa r a m e te r e s t im a te s o f s ing l e e qua t ionm o d e l s w i l l b e u n b i a s e d i n t h e p r e s e n c e o f m u l t i -c o l l i ne a r i t y , bu t s t a nda r d e r r o r s f o r t he r e g r e s s ionpa r a m e te r ( s ) w i l l be i n f l a t e d [ 37 ] . La r ge s t a nda r de r r o r s i n t e r f e r e w i th t he r e se a r c he r s a b i l i t y t o de t e r -m in e the s t a t i s ti c a l s i gn i f i c a nc e o f va r i a b l e s a nd ,he nc e , t o e s t a b l i sh t he i r im por t a nc e in e xp la in ing thev a r i a t i o n o b s e r v e d i n t h e e n d o g e n o u s v a r i a b l e .

B e f o r e f o r m i n g e x p l i c i t e s t im a t i n g e q u a t i o n s , a n a -ly s t shou ld a t t e m p t t o i de n t i f y t he e x t e n t o f r e dun -da nc y p r e se n t i n t he s e t o f a l l pe r f o r m a nc e a t t r i bu t e s ,t ha t i s , t he e xp la na to r y va r i a b l e s o f Ta b le 1 [ 12 ].G i v e n h i g h l y c o r r e l a t e d p e r f o r m a n c e a t t r i b u t e s , " I sthe r e a r e duc e d se t o f a t t r i bu t e s t ha t w ou ld c onc e p tu -a l i z e u s e r s a t i s f a c t i o n w i t h m i c r o c o m p u t e rs o f t w a r e ? " T h i s q u e s t i o n i s a d d r e s s e d h e r e b y f a c -to r ing the s a m p le c o r r e l a t i on m a t r ix u s ing p r inc ipa lc o m p o n e n t a n a l y s i s . W e s e a r c h e d f o r a c o m p o n e n ts t r uc tu r e , a sm a l l e r s e t o f l i ne a r c om bina t ions o f t hea v e r a g e e v a l u a t i o n s , t h a t w a s c a p a b l e o f p r e s e r v i n gm o s t o f t h e i n f o r m a t i o n c o n t a i n e d i n t h e o r i g i n a lda t a .

T h e r e a r e n u m e r o u s a c c e p t a b l e c r i t e r i a f o r d e t e r -m i n i n g t h e n u m b e r o f c o m p o n e n t s t o b e e x t r a c te d . I nt h is s t u d y , t h e a u t h o rs h a v e e m p l o y e d t h e p e r c e n t -a g e o f v a r i a n c e c r i t e ri o n . T h e f a c t o r i n g p r o c e d u r ew a s s t o p p e d w h e n t h e e x t r a c t e d c o m p o n e n t s a c -c oun te d f o r a t l e a s t 95 pe r c e n t o f t he va r i a nc e in t hed a t a ( s e e T a b l e 2 ) . T h e e i g e n v a l u e s o f e a c h c o m p o -n e n t a r e a l s o s h o w n i n T a b l e 2 .

U s i n g t h e p e r c e n t a g e o f v a r i a n c e c r i t e r i o n , t h ea p p r o p r i a t e n u m b e r o f c o m p o n e n t s i s t h r e e f o r a l l

s o f t w a r e t y p e s , e x c e p t w o r d p r o c e s s i n g , w h e r e f o u rc o m p o n e n t s a r e r e q u i r e d t o a c h i e v e a c u m u l a t i v epe r c e n t o f va r i a nc e e qua l t o o r g r e a t e r t ha n 0 . 95 . S oa s t o ga in som e in t e r p r e t ive i n s igh t i n to t he m a pp ingo f p e r f o r m a n c e a t t r i b u t e s t o c o m p o n e n t s , a v a r i m a xr o ta t i on o f t he f a c to r - loa d ing m a t r ix f o r e a c h so f t -w a r e t y p e w a s p e r f o r m e d . T h e v a r i m a x r o t a t i o n i so r thogona l a nd invo lve s a r e d i s t r i bu t ion o f t he va r i -a n c e a c c o u n t e d f o r b y t h e g r o u p o f c o m p o n e n t si d e n t i f i e d u n d e r p r i n c i p a l c o m p o n e n t s , w i t h o u t a n ylos s o f t o ta l va r i a nc e e xp la ine d . A s tudy o f the f a c to rl o a d i n g s f o r s p r e a d s h e e t , D B M S , c o m m u n i c a t i o n s ,a n d g r a p h i c s s o f t w a r e r e v e a ls a c ons i s t e n t c lu s t e r ingo f a t t r ib u t e s a c r o s s s o f t w a r e t y p e s . T h e a s soc i a t i ono f a t t ri bu t e s w i th c om p one n t s , a nd the a u tho r s ' i n t e r -p r e t a t io n o f t h ei r s o u r c e s a n d s u g g e s t e d n a m e s o rl a b e l s a r e: f u n c t i o n / f e a t u r e c o m p o n e n t ( b a s ic a n da d v a n c e d f u n c t i o n s ) , s e r v i c e / s u p p o r t c o m p o n e n t( d o c u m e n t a t i o n a n d v e n d o r s u p p o r t ) , a n d t h e i m p l e -m e n t a t i o n / f r i e n d l i n e s s c o m p o n e n t ( e a s e o f u s e a n dtra in ing) .

I t i s i n t e r e s t i ng to no te t ha t a ne a r i de n t i c a l c om -p o n e n t s t r u c t u r e e m e r g e d f o r w o r d p r o c e s s i n g s o f t -w a r e . T h e d i f f e r e n c e b e i n g t h a t t h e f u n c t i o n / f e a t u r ec o m p o n e n t w a s s p l i t i n t o t w o c o m p o n e n t s , w i t h e a c hh a v i n g h i g h l o a d i n g s f o r b a s i c a n d / o r a d v a n c e df unc t ions . G ive n tha t no a dd i t i ona l i n s igh t s w e r ega ine d in to t he s t r uc tu r e o f so f tw a r e a t t r i bu t e s bye x t r a c t i n g a f o u r t h c o m p o n e n t , t h a t t h e c u m u l a t i v ep r o p o r t i o n o f v a r i a n c e a c c o u n t e d f o r b y t h r e e c o m -p o n e n t s w a s 0 . 9 0 , a n d t h a t t h e a u t h o r s ' s u b s e q u e n ta n a l y s e s a l w a y s f o u n d u s e r s a t i s f a c t i o n m o d e l s w i t hf ou r e xp la na to r y va r i a b l e s ( a t t r i bu t e s ) i n f e r io r t o

T a b l e 2P r i n c ip l e c o m p o n e n t s a n a l y s i sC o m p o n e n t s W o r d P r o c es s i ng S p r ea d s h ee t s D B M S C o m m u n i c a t i o n G r a p h ic s

E i g e n - C u m u l a t i v e E i g e n - C u m u l a t i v e E ig e n - C u m u l a t i v e E i g e n- C u m u l a t i v e E i g e n - C u m u l a t i v ev a l u e s v a r i a n c e v a l u e s v a r i a n c e v a l u e s v a r i a n c e v a l u e s v a r i a n c e v a l u e s v a r i a n c e

( p e r c e n t ) ( p e r c e n t ) ( p e r c e n t ) ( p e r c e n t ) ( p e r c e n t )C o m p o n e n t 1 3 . 2 7 0 . 5 4 4 . 9 3 0 . 8 2 4 . 2 5 0 . 7 1 4 . 9 6 0 . 8 3 4 . 1 6 0 . 6 9C o m p o n e n t 2 1 . 3 9 0 . 7 8 0 . 7 3 0 . 9 4 1 .3 7 0 . 9 4 0 . 6 2 0 . 9 3 1 . 3 0 0 . 9 1C o m p o n e n t 3 0 . 7 4 0 . 9 0 0 . 1 5 0 . 9 7 a 0 .17 0 . 9 7 a 0 .27 0 . 9 8 a 0 .28 0 . 9 6 aC o m p o n e n t 4 0 . 2 9 0 . 9 5 a 0 .1 1 0 . 9 9 0 . 1 4 0 . 9 9 0 . 0 8 0 . 9 9 0 . 2 2 0 . 9 9a T h e m i n i m u m n u m b e r o f c o m p o n e n t s r e q u ir e d t o a c h i e v e a c u m u l a t i v e p r o p o r ti o n o f v a r i a n ce t h a t i s a t l e a s t 0 . 9 5.


7/18

E.E. Anderson, Y.-M. Chen / D ecision Support Systems 19 (1997) 75 -92 8 1

those with three, it was concluded that there wereonly three unique components present in word pro-cessing as well.

5 . M o d e l s p e c i f i c a t i o n a n d d i a g n o s t i c sWhile principal component analysis suggested

three dimensions of end user satisfaction, it does notallow the authors to determine the most importantperformance attribute o f e a c h c o m p o n e n t , nor ther e l a ti v e i m p o r t a n c e o f c o m p o n e n t s . Depending onuser environments and their applications of software,the next methodological step involves forming acausal relationship between user satisfaction andsoftware attributes. If the unit of observation is theindividual user, we may start with the followingcausal model:S i j = a~ B i j + a 2 A i j + a 3 O ij "k- a 4 Vij -b a 5 E i j

+ a 6 T i j + E l , ( l )where Si j= overall satisfaction, Bi i= basic func-tions, Aij = adv anc ed funct ions, Dij =documentation, V~j = vendor support, E u = ease ofuse, ~j = training time and e u = a random errorterm associated with the ith respondent and the jthproduct within a particular software type. a~, a 2,. . . . a 6 are parameters to be estimated. It is assumedthat E(eij) = 0, and that

t r i = kE ( e i j ek j ) = 0 i v~ k .Readers should note that this particular set of

explanatory variables is used in Eq. (1) because theywere the attributes used by Datapro Research tomeasure performance. They are used only to illus-trate the methodology.

A statistical assessment of software based on amodel such as Eq. (1) must consider three criticalquestions: Are the assumptions regarding the error term

justified? Are the explanatory variables independent? Are there unspecified attributes of software or its

usage that are omitted from the model?The authors will next consider each of these

questions, prescribe diagnostic tests and illustrate

their application using the Datapro Research Corpo-ration data.

Before exploring these questions, consider thefact that these data are the m e a n s o f a g g r eg a t es c o r e s given by users for each of the attributes of theith product, i = 1, 2 . . . . , M, and that the meanattribute rating for various products is based ond i f f e r e n t s a m p l e s i z e s . For example, N i was nearlytwo hundred for PerfectWriter, while other wordprocessing products had as few as three or fouruser-respondents. Given these conditions, Eq. (1)should be defined as a model of "average variables".That is

( 2 )where sample averages are noted by a bar "'-".

Now, consider the first question from above. Asis shown in Appendix B, the variance of the errorterm varies inversely with the sample size for eachproduct. Hence, Eq. (2) has a heteroscedastic errorterm, and ordinary least-squares estimates o f parame-ters, while unbiased and consistent, will be ineffi-cient. Additionally, the estimated variances of theparameters will be biased estimates of their truevalues. If the parameters of Eq. (2) are estimated byordinary least-squares (OLS), in the presence ofheteroscedasticity, equal weight is given to observa-tions with large error variances and small error vari-ances. This has serious implications, if left untreated,because it may cause one to conclude that someattributes are important to performance, when theyare not, and vice versa.

There are essentially two approaches to solvingthis problem [31]. The first is to use weighted least-squares. The second is to transform the data suchthat, under the transformation, the assumptions ofOLS are satisfied [13]. The authors have chosen thelatter, as discussed in Appendix B.

We now consider the second question and theindependence of explanatory variables. To form amodel of user software satisfaction, we must knowthe number and identity of explanatory variables(performance attributes) to include in the relationshipfor each product. Principal component analysis hasanswered the first question and given directions for


8/18


f i nd ing a nsw e r s t o t he s e c ond . S pe c i f i c a l l y , t h r e ec o m p o n e n t s w e r e e x t r a c t e d f o r e a c h s o f t w a r e t y p ea n d t h e s a m e t w o p e r f o r m a n c e a t t ri b u t e s w e r e f o u n dt o b e a s s o c i a t e d w i t h e a c h c o m p o n e n t , r e g a r d l es s o fso f tw a r e t ype . Thus , u se r so f tw a r e s a t i s f a c t ion c a nb e " b e s t " m o d e l e d b y t h r e e p e r f o r m a n c e a t t r i b u t e s ,o n e f r o m e a c h c o m p o n e n t . T h e r e f o r e , i t i s n o t n e ce s -sa r y t o c ons ide r a l l pos s ib l e l i ne a r e qua t ions c ons i s t -i n g o f t h r e e e x p l a n a t o r y v a r i a b l e s f r o m a s e t o f s i x ,t ha t i s 120 . U s ing the a t t r i bu t e g r oup ings sugge s t e db y p r i n c i p a l c o m p o n e n t a n a l y s i s , t h e n u m b e r o fe q u a t i o n s t h a t m u s t b e e s t i m a t e d a n d c o m p a r e d f o re a c h so f tw a r e t ype i s e igh t ( 2 2 2 ) . The c r i t e r ionu s e d t o d e t e r m i n e w h i c h e q u a t i o n b e s t m o d e l e d u s e rsa t i s f a c t ion w a s t he m i n i m u m m e a n s q u a r e d e r r o r .

I t i s im por t a n t t o no te t ha t t he u se o f p r inc ipa lc o m p o n e n t a n a l y s i s d o e s n o t i m p l y t h a t t h e r e s u l t i n gth r e e a t t r i bu t e e qua t ions w i l l e xpe r i e nc e no m u l t i -c o l l i ne a r i t y i n t he o r ig ina l va r i a b l e s . M u l t i c o l l i ne a r -i t y m a y s t i l l a r i s e i n e s t im a t ing e qua t ions be c a uset w o o r m o r e o f t h e t h r e e e x p l a n a t o r y v a r i a b l e s a r eh i g h l y c o r r el a t ed . T h o u g h p a r a m e t e r e s t i m a t e s w o u l db e u n b i a s e d , t h e i r v a r i a n c e s w o u l d b e i n f l a t e d a n d ,a s a r e su l t , h igh ly un r e l i a b l e . Th i s c ou ld l e a d e va lua -to r s t o f a l s e c onc lu s ions a bou t w h ic h a t t r i bu t e s be s te xp la in u se r s a t i s f a c t ion . The a u tho r s w i l l p r oposel a t e r a m e t h o d o l o g y i n v o l v i n g e m b e d d e d r e c u r s i v ee qua t ions f o r t hose so f tw a r e t ype s w i th subs t a n t i a lc o r r e l a t i o n b e t w e e n t w o o r m o r e o f t h e t h r e e a t -t r ibutes [29 ,52] .

M o s t s t a t i s t i c a l l y b a s e d e v a l u a t i o n m o d e l s a r es i n g l e e q u a t i o n m o d e l s . T h e y h a v e t h e a d v a n t a g e o fp a u c i t y a n d s i m p l i c i t y o f e s ti m a t i o n . N e v e r t h e l e ss , ac r i t ic a l a r g um e n t i n t he i r j u s t i f i c a t i on ( r e c a l l t he t h i r d

q u e s t io n f r o m a b o v e ) i s t h a t th e i m p o r t a n t s y s t e m a t i cf a c to r s a nd va r i a b l e s c a n a l l be r e p r e se n te d in as ing le e qua t ion m ode l . The a u tho r s be l i e ve tha t t h i sa s s u m p t i o n m u s t b e v a l i d a t e d a s p a r t o f a n y s o u n dm e t h o d o l o g y f o r s o f t w a r e e v a l u a t i o n .

To i l l u s t r a t e t he p l a us ib i l i t y t ha t im por t a n t a t -t r ib u t e s o r f a c t o r s o f s o f t w a r e u s a g e a r e o m i t t e d f r o mEq . ( 2 ) , c ons ide r t he pos s ib i l i t y t ha t u se r - r e sponde n t sm a y o w n m o r e t h a n o n e t y p e o f s o f t w a r e . H e n c e ,t h e ir e v a l u a t i o n s o f p r o d u c t s f r o m o n e s o f t w a r e t y p em a y b e a f f e c t e d b y t h e i r e x p e r i e n c e s w i t h t h e p r o d -u c t s o f o t h e r s o f t w a r e t y p e s . F o r e x a m p l e , m a n yu s e r s o f s p r e a d s h e e t s o f t w a r e w i ll a l s o h a v e c o m p a t i -b l e g r a ph ic s o r da t a ba se so f tw a r e . H e nc e , a p r io r it he i r ove r a l l s a t i s f a c t ion w i th a ny so f tw a r e t ype c a n -n o t b e r e p r e s e n t e d b y a s i n g l e e q u a t i o n , unt i l i t hasb e e n d e t e r m i n e d t h a t t h e i r e v a l u a t i o n s o f s o f t w a r et y p e s a r e u n r e l a t e d . T h e s e p o s s i b l e c o n f o u n d i n g e f -f e c t s sugge s t t ha t som e a t t r i bu t e s o f one so f tw a r et y p e m a y b e f u n c t i o n s o f t h e a t t r i b u t e s o f o t h e rs o f t w a r e t y p e s . T o m o d e l t h e u s e r s a t i s f a c t i o n o fe a c h so f tw a r e t ype a s a s ing l e e qua t ion w ou ld igno r ethe se r e l a t i onsh ip s a nd inc u r pos s ib l e spe c i f i c a t i one r ro r s . H e n c e , u s e r s a t i s f a c ti o n s h o u l d b e m o d e l e d a sa s i m u l t a n e o u s s y s t e m o f ca u s a l r e l a ti o n s h i p s . H o w -e ve r , t he e x t e n t a nd na tu r e o f c r o s s so f tw a r e r e l a t i on -sh ip s a r e unknow n . The r e f o r e w e a t t r i bu t e t he seunspe c i f i e d c a usa l r e l a t i onsh ip s t o t he e r r o r t e r m s o fe a c h s o f t w a r e t y p e . I f t h e a v e r a g e e v a l u a t i o n s o fva r ious pe r f o r m a nc e a t t r i bu t e s a r e r e l a t e d a c r os sso f tw a r e t ype s , t he e r r o r t e r m s w i l l be c o r r e l a t e d ,a nd s im u l t a ne i ty w i l l e x i s t . H e nc e , t he f o l low ingm o d e l o f u s e r s o f t w a r e s a t is f a c ti o n i s p r o p o s e d[31,63]:

Software EquationW o r d P r o c e s s i n g :

W ' S j = o t w W X j , + ~ w W X I 2 + "Y wW 2 ~j3 + W f x j, j = 1 , 2 . . . . . 2 3 , ( 3 )S p r e a d s h e e t

+ s S X : + ,s sx 3 + s r . j = 1 . 2 . . . . . 1 3 , ( 4 )D B M S :

D S j = a a D . g j l + ~ a D X ~ 2 + Y dD )fi3 + D ~ j , j = 1 , 2 . . . . . 1 8 , ( 5 )


9/18

E.E. Anderson, Y.-M. Chen / D ecis ion Supp ort Systems 19 (1997) 75 -92 83C o m m u n i c a t i o n s :

( 6 )G r a p h i c s :

G S j = a g G ) ~ j , + ~ g G J ( j2 + T g G X j 3 + G ~ x , , j = l , 2 . . . . . 1 6, ( 7 )w h e r e " ~ " r e f e rs t o t h e d a t a t r a n s f o r m e d b y f ~ j ; jr e f e r s t o t he j t h p r oduc t , r e spe c t ive ly . I S j , i s t heove r a l l s a t i s f a c t ion o f u se r s f o r so f twa r e type , l ~ {W,S , D , C , G } , l )~ j l . . . . . 1) ~ j3 a r e th r e e o f t he s ixp e r f o r m a n c e a t t r i b u t e s ( e x p l a n a t o r y v a r i a b l e s ) t h a tbe s t a c c oun t f o r t he va r i a t ion o f s a t i s f a c t ion wi thsof t wa re type l , a~ /3~ "y~ a re pa ram eter s to bee s t im a te d a nd l / 2 j a r e t he e r r o r t e r m s .U s i n g v e c t o r - m a t r i x n o ta t i o n , E q s . ( 3 ) - ( 7 ) m a ybe . e xp r e s se d a ss : z a + ( 8 )S e e A p p e n d i x C f o r a c o m p l e t e r e p re s e n t a t i o n o f E q .( 8 ) . T h e v a r i a n c e - c o v a r i a n c e m a t r i x o f / z c a n b es e e n i n A p p e n d i x D .

I f X ( s e e A p p e n d i x D ) i s n o t b l o c k d i a g o n a l , t h a ti s , t he e r r o r t e r m s o f Eqs . ( 3 ) - ( 7 ) a r e c o r r e la t e d , wew o u l d e s t i m a t e th e p a r a m e t e r o f 8 b y8 = ( Z ' ~ - ' Z ) - ' Z ' ~ - ' S . ( 9 )Al t e r na t iv e ly , i f we a r e una b le to r e j e c t the nu l lhy po thes is tha t o- , j = 0 , i and j ~ {W, S, D , C , G}i 4 : j , w e e s t im a te e a c h e qua t ion by O LS . I nde e d , i fn o s i m u l t a n e i t y i s f o u n d , t h e e s t i m a t o r 8 f r o m E q .( 9 ) i s i de n t i c a l t o t he one ob ta ine d by a pp ly ing OLSt o e a c h e q u a t i o n o n e b y o n e .

The t e s t c r i t e r ion f o r r e j e c t ing the nu l l hypo the s i si s Ande r son [ 6 ] .

I r i j [2 . - - > tN_2 ( a ) , ( 1 0 )~ / 1 - - r / j

w h e r e N i s t h e n u m b e r o f o b s e r v a t i o n s , r i j i s thesa m ple c o r r e l a t ion c oe f f i c i e n t , i ~ j , a nd a i s t hep r o b a b i l i t y l e v e l .

Te s t ing f o r c o r r e l a t e d e r r o r s i nvo lve s s e ve r a l s t e ps .I n the f i r s t s t e p , Eqs . ( 3 ) - ( 7 ) we r e e s t im a te d ind iv id -ua l ly u s ing the i r a s soc ia t e d r e c u r s ive e qua t ion . Thespe c i f i c f o r m o f t he r e c u r s ive e qua t ions i s d i sc usse d

b e l o w . S e c o n d , t h e e s t i m a t e d r e s id u a l s f r o m s t e p o n ea r e u se d to t e s t t he nu l l hypo the s i s o f c o r r e l a t e de r r o r s . The r e su l t s o f t he s e c ond s t e p de t e r m inewh e the r a t h i r d s t e p i s r e qu i r e d . I f c o r r e l a t e d e r r o r sa r e f o u n d , t h e e s t i m a t e d v a r i a n c e o f p a r a m e t e r s w i l lbe in f l a t e d , a nd we a r e m or e l i ke ly to m a ke e r r o r s i nhypo the s i s t e s t ing a nd in a s se s s ing the r e l a t ive im -por t a nc e o f va r ious so f twa r e a t t r i bu te s . The r e f o r e ,we wou ld use Eq . ( 9 ) t o e s t im a te 8 in the th i r d s t e p .A l t e r na t ive ly , i f we f a i l t o r e j e c t t he nu l l hypo the s i so f c o r r e l a te d e r r o r s, t h e p a r a m e t e r e s t i m a t e s f r o ms ing le e qua t ion e s t im a t ion a r e e qu iva le n t t o t hose o fEq . ( 9 ) a nd s t e p th r e e i s unne c e s sa r y .

6 . M o d e l r e f i n e m e n t s a n d e s t i m a t i o n

U n d e r s t e p o n e f r o m a b o v e , w e s t i l l m u s t c o n t e n dw i t h w h a t r e m a i n s o f t h e p r o b l e m o f m u l t i c o l l i n e a r -i t y . P r i n c i p a l c o m p o n e n t a n a l y s i s h a s p r o v i d e d apa r t i a l so lu t ion by r e duc ing the a t t r i bu te s e t f r om s ixto th r e e va r i a b le s . Ne ve r the l e s s , som e o f t he poss ib l ec o m b i n a t i o n s o f t h r e e v a r i a b l e s , o n e f r o m e a c h c o m -p o n e n t , m a y b e h i g h l y c o r r e l a t e d . T h e r e m a y b ec a u s a l r el a t io n s h i p s b e t w e e n v a r i o u s c o m b i n a t i o n s o fs ign i f i c a n t a nd in s ign i f i c a n t pe r f o r m a nc e a t t r i bu te sa s soc ia t e d wi th d i f f e r e n t c om pone n t s . Whi l e t hec o m p o n e n t s a r e o r t h o g o n a l , t h e o r i g i n a l v a r i a b l e s( a t t r i bu te s ) a r e no t unc o r r e l a t e d . He nc e , f o r e a c h o fthe th r e e a t t r i bu te m ode l s r e p r e se n te d by Eqs . ( 3 ) -( 7 ) , t he a u tho r s s e a r c he d f o r a l l c a se s whe r e s ign i f i -c a f g z e r o o r de r c o r r e l a t ion c oe f f i c i e n t s e x i s t e d be -t w e e n t w o o r m o r e o f t h e p e r f o r m a n c e a t t r i b u t e sa s soc ia t e d w i th a ny th r e e o f t he d i f f e r e n t c o m p o -ne n t s . Whe r e a t l e a s t two a t t r i bu te s f r om d i f f e r e n tc o m p o n e n t s w e r e c o r r e l a t e d , o n e w a s r e g r e s s e da ga ins t t he o the r s t o de t e r m ine i f a s ign i f i c a n t l i ne a rr e l a t ionsh ip e x i s t e d be twe e n the m . The r e su l t i s a


10/18

84 E.E. Anderson. Y .-M . Chen / Decision Support Systems 19 (1997) 75- 92Table 3User satisfaction models for each softwareSoftware ModelsWord Processing

Spreadsheets

DBMS

Communications

Graphics

~: is the estimate of k in the equation:W g = k W g +

s g )1is the est imate o f k in the equation:

DSj = o~,lDDj 4- ~d DA, + Ya DRE j + D~ ,DRF-.j = DF.j - (k" DD.,)

is the estimate of k in the equation:= +

= + + +c f i u j =1is the estimate of k in the equation:c g = k . c g + % .GS j = o t~GB j + , 8 ~G g + 3 ~GR g + G ~jG R ~ = G ~ -(1 " G ~ )

is the estimate of k in the equation:GEj = k" GBj + GEj.

p r o c e s s o f r e c u r s i v e e s t i m a t i o n . T h a t i s , o n e o f t h epe r f o r m a nc e a t t r i bu t e s , e xp l a na t o r y va r i a b l e s o f t heuse r s a t i s f a c t i on e qua t i on , i s e xpr e s se d a s a n e ndoge -nous v a r i a b l e i n t he r e c u r s i ve e qua t i on . I f the e s t i-m a t e d p a r a m e t e r s o f t h e r e c u r s i v e e q u a t i o n w e r es t a t i s t i ca l ly s igni f i cant , t he per formance a t t r ibute int he use r s a t i s f a c t i on m ode l i s r e p l a c e d by t he r e s i d -ua l o f t he r e c u r s i ve e qua t i on . The r e c u r s i ve e qua -t i ons ha ve t he e f f e c t o f f i l t e r i ng ou t t he c onf ound i nge f f e c ts b e t w e e n e x p l a n a t o r y v a r i a b le s . T h i s i s i m p o r -t a n t be c a use t he r e su l t i ng va r i a b l e s ( a t t r i bu t e s o rt he i r su r r oga t e s ) i n t he use r s a t i s f a c t i on e qua t i onswi l l be m or e ne a r l y o r t hogona l a nd , t he r e f o r e , t hem ul t i c o l l i ne a r i t y i n e a c h e qua t i on wi l l be subs t a n -t i a l l y r e duc e d . Ta b l e 3 p r e se n t s t he f i na l f o r m s f o rr e c u r s i ve a n d use r s a t i s f a c t i on e qua t i ons f o r e a c hso f t wa r e t ype . R e c a l l t ha t t he choice of performanceattributes i nc l ude d i n t he use r s a t i s f a c t i on e qua t i onsw a s b a s e d o n t h e m i n i m u m m e a n s q u a r e d e r r o r( M S E) c r i t e r i on . F o r t he be s t s e t o f a t t r i bu t e s t he r ewe r e f r e que n t l y se ve r a l f e a s i b l e r e c u r s i ve e qua t i ons .Whi l e t he M S E o f a ny use r s a t i s f a c t i on e qua t i on i sc ons t a n t f o r a l l a s soc i a t e d r e c u r s i ve e qua t i ons , t hee s t i m a t e d p a r a m e t e r s m a y v a r y . T h e f i n a l c h o i c e o fr e c u r s iv e e q u a t i o n t o e m p l o y w i t h e a c h o f t h e u s e rsa t i s f a c t i on e qua t i ons wa s ba se d on t he e qua t i on t ha tp r oduc e d t he h i ghe s t l e ve l o f s t a t i s t i c a l s i gn i f i c a nc e ,

Table 4Estimated parameters for each software aSoftware Explanatory variables (attributes) t,

Basic function Documentation Advanced function Vendor support Ease of use Training time R 20.6095 0.4006 0.2148

Word Processing 0.9993(6.54) (3.99) (2.86)0.5485 0.6118 0.4531

Spreadsheets 0.9998(4.76) (4.06) (3.05)

0.4788 0.5639 0.3744DBMS 0.9996(8.56) (10.72) (4.09)

0.8414 0.7362 0.3421Communications 0.9997

(7.80) (5.39) (2.68)0.6501 0.3420 0.4759

Graphics 0.9997(7.08) (3.45) (6.02)

a The model for each software is specified in Table 3; ( ) estimated t statistics.b All parameter estimates are significant for ot = 0.01.


11/18

E.E. Anderson. Y.-M. Chen / Decision Support Systems 19 (1997) 75-92 85

m e a su r e d b y t s t a ti s t i c s, f o r t he e s t im a te d pa r a m e te r so f so f tw a r e a t t r i bu t e s .

E m p l o y i n g t h e e q u a t i o n s o f T a b l e 3 , th e r e s i d u a lso f e a c h u s e r s a t i sf a c t io n e q u a t i o n w e r e e s t i m a t e d a n duse d to t e s t f o r c o r r e l a t e d e r r o r s be tw e e n Eqs . ( 3 ) -( 7 ) . U s ing the t e s t c r i t e r ion e s t a b l i she d by Eq . ( 10 ) ,t he nu l l hypo the s i s w a s a c c e p te d f o r a l l so f tw a r etype s a t c~ = 0 . 01 . The l a r ge s t a bso lu t e v a lue o f a nyc o r r e l a t i on c oe f f i c i e n t f ound ( 0 . 014 ) w a s f o r t hee r r o r t e r m s b e t w e e n D B M S ( E q . (5 ) ) an d g r a p h i c s( Eq . ( 7 ) ) . I n t he a bse nc e o f c o r r e l a t e d e r r o r t e r m s ,w e m a y c onc lude tha t s im u l t a ne i ty i s no t p r e se n t ,a nd a c c e p t t he s ing l e e qua t ion e s t im a te s o f t he pa -r a m e te r s show n in Ta b le 3 . Thus , s t e p t h r e e i su n n e c e s s a r y a n d t h e e s t i m a t e s o f 3 o b t a i n e d f r o ms i n g l e e q u a t i o n e s t i m a t i o n c a n n o t b e i m p r o v e d u p o n .

The pa r a m e te r s e s t im a te s , t he i r s t a nda r d e r r o r s ,a nd R 2 f o r e a c h o f t he u se r s a t i s f a c t ion e qua t ionsshow n in Ta b le 3 a r e p r e se n te d in Ta b le 4 . A sm e a su r e d b y R 2 , t he so f tw a r e pe r f o r m a nc e a t t r i bu t e so f e a c h r e s p e c t i v e e q u a t i o n a c c o u n t s f o r a n u n u s u -a l ly l a r ge p r opo r t i on o f t he va r i a t i on in u se r s a t i s f a c -t i on . S e c ond , a l l t h r e e o f t he va r i a b l e s i n e a c h e qua -t ion a re s ta t i s t ica l ly s ign i f icant fo r o~ = 0 .01 . T hird ,t he r e a r e bo th s im i l a r i t i e s a nd d i f f e r e nc e s i n t her e la t iv e i m p o r t a n c e o f v a r i o u s p e r f o r m a n c e a t tr i b u te sb e t w e e n s o f t w a r e t y p e s . F o r e x a m p l e , t r a i n i n g t i m ea nd ve ndor suppo r t a r e ne ve r s t a t i s t i c a l l y s ign i f i c a n t ,w h i l e e a s e o f u s e a n d d o c u m e n t a t i o n a r e a l w a y ss ign i f i c a n t . S inc e t he se va r i a b l e s a r e a l l m e a su r e d onthe s a m e sc a l e , t he i n t e r p r e t a t i on o f t he i r r e l a t i v em a g n i t u d e s , i n a g ive n e qua t ion , i s a s t hough theva r i a b l e s ha d be e n s t a nda r d i z e d . U s ing tha t f a c t , onec a n f u r the r e va lua t e t he c on t r ibu t ion o f va r ious pe r -f o r m a nc e s a t t r i bu t e s t o u se r s a t i s f a c t ion .

7 . M o d e l a p p l i c a b i l i t y i s s u e s

T h e a u t h o r s h a v e p r o p o s e d a m u l t i - s t e p m e t h o d o l -ogy f o r t he s t a t i s t i c a l e va lua t ion o f so f tw a r e . F r omour a pp l i c a t ion , i t i s e v ide n t t ha t m e thodo log ic a lop t ions a r e i n f lue nc e d a nd , i n som e c a se s , de t e r -m ine d by the c ha r a c t e r i s t i c s o f t he da t a . Though thed a t a u s e d i n t h i s p a p e r w e r e t a k e n f r o m s e c o n d a r ysou r c e s , i t i s r e a sona b le t o c ons ide r t he dom a in o f

a p p l i c a t i o n f o r t h e m e t h o d o l o g y p r e s e n t e d . T h e s p e -c i f i c s t a ti s t i ca l f i nd ing s o f t h i s s tudy , a nd the i r im p l i -c a t ions a r e l e f t t o t he c onc lu s ions .

A s s u m i n g t h a t t h e r e i s a n a d e q u a t e s a m p l e s i z ea n d a c a r e f u l a p p l i c a t i o n o f t h e m e t h o d o l o g y , t h ea u tho r s sugge s t t ha t r e a de r s ( so f tw a r e e va lua to r s )c o n s i d e r t h e f o l l o w i n g q u e s t i o n s i n a s s e s s i n gm e t h o d o l o g i c a l a p p l i c ab i l it y . H o w c o n g r u e n t a r e t h e t e c h n o l o g i e s , u s e r n e e d s ,

a n d a p p l i c a t i o n s r e p r e s e n t e d i n t h e d a ta r e l a t i v et o t h o s e o f t h e e v a l u a t o r ?Thi s que s t ion i s c onc e r ne d w i th t he r e p r e se n ta -

t i ve ne s s o f t he da t a r e l a t i ve t o t he e va lua to r ' s e nv i -r o n m e n t . I f o n e i s u s i n g p r i m a r y d a t a , c o l l ec t e d b ythe e va lua to r , i t a r gue s f o r a c a r e f u l ly c ons t r uc t e ds a m p l i n g d e s i g n t h a t i n s u r e s c o v e r a g e o f i m p o r t a n tva r i a b l e s a nd f a c to r s i n r ough ly the de ns i t i e s ob -s e r v e d i n t h e e v a l u a t o r ' s e n v i r o n m e n t . A l t e r n a ti v e l y ,i f t h e d a ta c o m e s f r o m s e c o n d a r y s o u r c e s , a n s w e r s t oth i s que s t ion m us t be u se d to c ond i t i on the c on f i -de nc e tha t one ha s i n s t a t i s t i c a l a s se s sm e n t s . H o w s t a b l e a r e th e p e r f o r m a n c e a t t r i b u t e s o v e r

t i m e ?C l a s s e s o f t e c h n o l o g i e s te n d t o h a v e " t i m e w i n -

d o w s " o v e r w h ic h t h e y c h an g e s l o w l y a n d h a v er e a sona b ly s t a b l e r e l a t i onsh ip s w i th t he i r e nv i r on -m e n t s . T h e d u r a t i o n o f t h es e t i m e w i n d o w s v a r i e sb e t w e e n t e c h n o l o g i e s a n d o v e r t i m e w i t h in t e c h n o l o -g i e s . S o f tw a r e i s no e xc e p t ion . W hi l e t he r e i s nop r e s c r i p t i o n f o r a s s e s s i n g f u n d a m e n t a l c h a n g e i nso f tw a r e a nd i t s a pp l i c a t ions , u se r s m us t be p r e pa r e dto l im i t t he i r u t i l i z a t i on o f s t a t i s t i c a l f i nd ings w he ni m p o r t a n t c h a n g e h a s o c c u r r ed . H o w b r o a d l y ~ n a r r o w l y d e f in e d a re th e e v a lu a -

t o r ' s d e c i s i o n c r i t er i a f o r s o f t w a r e ?E v a l u a t o r s h a v e d i f f e r e n t n e e d s f o r b r e a d t h / n a r -

r o w n e s s i n t h e d e fi n i ti o n a n d m e a s u r e m e n t o f i m p o r -t a n t va r i a b l e s . V a r i a b l e s f r e que n t ly c a n be pa r t i -t i one d in to a dd i t i ona l d im e ns ions , t he r e by a dd ing tot h e c o s ts o f d a t a c o l l e c t i o n a n d m o d e l i n g . E v a l u a t o r sm us t s t r i ve f o r t he p r ope r ba l a nc e be tw e e n the r e l e -va nc e o f va r i a b l e s , i n t he c on te x t o f t he i r e va lua -t i ons , a nd the vo lum e a nd c os t s o f da t a c o l l e c t ion .

T o w h a t e x t e n t a r e t r a d e o f fs b e t w e e n s o f t w a r ea t t r i b u t e s a c c e p t a b l e i n e v a l u a t i o n s ?M ost s t a t i s t i c a l m ode l s o f so f tw a r e e va lua t ion a r e

c o m p e n s a t o r y ; t h a t i s , p a r a m e t e r e s t i m a t e s a s s i g nw e igh t s t o a t t r i bu t e s suc h tha t a g ive n sc o r e ( va lue )


12/18


o f u s e r s a t i s f a c t i o n m a y d e r i v e f r o m m a n y d i f f e r e n tpr o f i l e s o f a t tr i bu t e s r a ti ngs . He n c e , a h i gh r a t i ng o none a t t r ibu t e , wh i c h i s no t i m por t a n t i n t he e va l u a t o r ' se n v i r o n m e n t , m a y b e u s e d t o o f f s e t a l o w r a t i n g o nanother a t t r ibute tha t i s impor tant .

R e ga r d i ng t he a pp l i c a b i l i t y o f t he spe c i f i c f i nd -i ngs o f t h i s s t udy , t he a u t hor s be l i e ve t ha t t he yr e p r e s e n t a r e a s o n a b l y b r o a d c o v e r a g e o f m i c r o c o m -pu t e r so f t wa r e a nd t he i r u se r s . The a u t hor s be l i e vet ha t t he se f i nd i ngs a r e l i m i t e d , howe ve r , by t he b r oa dde f i n i t i on g i ve n t o t he a t t r i bu t e s . The se da t a a r ea d e q u a t e t o d e m o n s t r a t e t h e m e t h o d o l o g y , b u t t h ea t t r i bu t e s m a y no t be su f f i c i e n t l y we l l f oc use d a ndp r e c i s e t o m e e t t h e n e e d s o f s o m e e v a l u a t o r s .

8 . C o n c l u s i o n s

B u y e r s o f m i c r o c o m p u t e r s o f t w a r e a r e c o n f r o n t e dw i t h a n e v a l u a t i o n / c h o i c e p r o b l e m t h a t i s a n a l o g o u st o t ha t f ound i n c om p ut e r ha r dw a r e , t ha t is , the ywoul d l i ke t o c hoose t he p r oduc t ( s ) t ha t ha s ( ha ve )t he h i ghe s t p r oba b i l i t y o f s a t i s f y i ng t he i r c om put i ngne e ds , g i ve n i t s c os t . I t i s c om pl i c a t e d by t he m ul t i -p l i c i t y o f p r oduc t s , va r i a t i on i n t he pe r f o r m a nc el e ve l s o f p r oduc t s , a nd t he ne e d unc e r t a i n t i e s o fb u y e r s . I n t h i s e n v i r o n m e n t , b u y e r s c a n f r e q u e n t l yb e n e f i t f r o m t h e i n f o r m a t i o n d e r i v e d f r o m t h e e x p e -r i e nc e a nd ob se r va t i ons o f u se r s . In t h i s pa pe r , t hea u t h o r s h a v e p r o p o s e d a m e t h o d o l o g y f o r e v a l u a t i o ntha t i dent i f i es the most re l evant a t t r ibute se t andde t e r m i ne s t he r e l a t i ve we i gh t , o r e m pha s i s , t ha tshou l d be g i ve n t o e a c h a t t r i bu t e ba se d on use re va l ua t i ons .

F r om t he a na l yse s o f t h i s s t udy , s e ve r a l i n t e r e s t -i ng r e su l t s ha ve be e n obse r ve d . C ons i de r f i r s t t hosec onc l us i ons t ha t a pp l y t o a l l so f t wa r e t ype s . F i r s t , i ta ppe a r s t ha t u se r so f t wa r e sa t i s f a c t i on i s a c c oun t e df o r b y t h e f o l l o w i n g t h r e e f a c t o r s : f u n c t i o n / f e a t u r e ,s e r v i c e / s u p p o r t , a n d i m p l e m e n t a t i o n / f r i e n d l i n e s s .S e c o n d , e a s e o f u s e a n d d o c u m e n t a t i o n w e r e c o n s i s -t e n t l y f o u n d t o b e t w o o f t h e t h r e e m o s t i m p o r t a n te xp l a na t i ons f o r so f t wa r e sa t i s f a c t i on . B a s i c f unc -t i ons wa s a l so f ound t o be a ve r y i m por t a n t sou r c e o fuse r s a t i s f a c t i on i n f ou r o f t he f i ve so f t wa r e t ype s .Th i r d , ne i t he r t r a i n i ng t i m e nor ve ndor suppor t o fs o f t w a r e w e r e e v e r i m p o r t a n t e n o u g h a s e x p l a n a t o r yva r i a b l e s t o r e p r e se n t one o f t he t h r e e f a c t o r s .

S e ve r a l i n t e r e s t i ng d i f f e r e nc e s we r e f ound i n t here la t i v e c on t r ibu t ion of a t t r i bu t e s t o use r s a t i s f a c t i onbe t we e n so f t wa r e t ype s . F i r s t , t he r e c u r s i ve e qua -t i ons d i d no t un i f o r m l y i nvo l ve t he r e s i dua l s o f t hesa m e va r i a b l e s . F o r t h r e e so f t wa r e t ype s , wor d p r o -c e s s in g , D B M S , a n d g r a p h i c s , e a s e o f u s e w a s t h ee ndoge nous va r i a b l e i n t he r e c u r s i ve e qua t i on a ndd o c u m e n t a t i o n o r b a s i c f u n c t i o n s w a s t h e e x p l a n a -t o r y va r i a b l e . Th i s r e su l t i s i n t u i t i ve l y a ppe a l i ng ,g i v e n t h e c o n t ri b u ti o n o f g o o d d o c u m e n t a t i o n t o e a s eo f use . S e c ond , s a t i s f a c t i on wi t h ba s i c f unc t i ons i st he m os t i m por t a n t sou r c e o f u se r s a t i s f a c t i on wi t hw o r d p r o c e s s in g , c o m m u n i c a t i o n s , a n d g r a p h i c s s o f t-w a r e , a n d t h e s e c o n d m o s t i m p o r t a n t p e r f o r m a n c ea t t r i bu t e o f sp r e a dshe e t p r oduc t s .

T h i r d , t h e i m p o r t a n c e o f g o o d d o c u m e n t a t i o n i sc onf i r m e d by t he se r e su l t s . F o r f ou r o f t he f i veso f t wa r e t ype s , i t r a nke d a s e i t he r t he f i r s t o r s e c ondm os t i m por t a n t pe r f o r m a nc e a t t r i bu t e . De sp i t e t her e l a t i ve l y poor s t a t i s t i c a l pe r f o r m a nc e o f t he ve ndorsuppor t a nd t r a i n i ng t i m e a t t r i bu t e s , t he a u t hor s dono t be l i e ve t ha t t he se a t t r i bu t e s a r e un i m por t a n t ona n a bso l u t e ba s i s . The y no doub t c on t r i bu t e t o t hesa t i s f a c t i on o f so f t wa r e use r s , bu t do no t a ppe a r t o bep r i o r i t y a t t r i bu t e s . F i na l l y , a s one m i gh t e xpe c t , t hea d v a n c e d s o f t w a r e f u n c t i o n s a r e t h e m o s t i m p o r t a n ts o u r c e o f s a ti s fa c t io n f o r th e u s e r s o f D B M S . D B M Sa r e m o r e l i k e l y t o b e b o u g h t b y t h e m o r e s o p h i s t i -c a t e d u s e r s o f m i c r o c o m p u t e r s a n d t h e i r n e e d s i nd a t a m a n a g e m e n t m a y r e q u i r e t h e o p t i o n s a n d f e a -t u r e s f ound i n a dva nc e d f unc t i ons .

The a u t hor s be l i e ve t ha t t he se r e su l t s ha ve i m pl i -c a t i ons f o r r e se a r c he r s , u se r s a nd ve ndor s . We be -l i e ve t ha t a m or e f unc t i ona l v i e w o f so f t wa r e a s se s s -m e n t a nd e va l ua t i on i s a ppr opr i a t e . S o f t wa r e sa t i s -f a c t i on se e m s t o de r i ve f r om t he f unc t i ona l i t y o fso f t wa r e e m be dde d i n i t s f e a t u r e s , e a se o f u se a nds u p p o r t f o r l e a r n i n g a n d p r o b l e m s o l v i n g t h r o u g hd o c u m e n t a t i o n . T h e a u t h o r s b e l i e v e t h a t m o r e E U Cr e se a r c h shou l d be de vo t e d t o s t ud i e s o f f unc t i ona l -i t y . F o r e xa m pl e , ve r y l i t t l e r e se a r c h e xa m i ne s t hep r ope r t i e s a nd c on t r i bu t i ons t o sa t i s f a c t i on o f doc u-m e n t a t i o n , w h i l e v o l u m e s h a v e b e e n w r i t t e n , w i t hve r y l i t t l e c onc l us i ve e v i de nc e , a bou t e ve r y poss i b l ei n t r i c a c y o f t he i nd i v i dua l , o r ga n i z a t i on a nd e nv i r on -m e n t .

I t w o u l d s e e m t h a t v e n d o r s s h o u ld m a n a g e ,t h r ough de s i gn a nd i nnova t i on , t he f l ow o f s e r v i c e s


13/18

E.E. Anderson, Y.-M. Chen / Decision Support Systems 19 (1997) 75-92 87

f rom so f t wa re . S o f t wa re pa c ka ge s shou l d be de -s i gne d so t ha t u se r s c a n qu i c k l y de ve l op sk i l l s i n i t sba s i c f e a t u re s a nd so t ha t u se r s c a n m ove t o soph i s t i -c a t e d a pp l i c a t i ons wi t h t he suppor t o f a dva nc e d fe a -t u re s a n d g o o d d o c u m e n t a t i o n . T h e n e e d s a n d e x p e c -t a t i ons o f u se r s g row wi t h knowl e dge a nd e xpe r i -e nc e . As t he r a nge o f e xpe c t a t i ons i nc re a se s , ve ndorsmus t ma na ge t he pa c e o f i nnova t i on . S o f t wa re i nno -v a t io n t h a t p r e c e d e s a " d e e p i n g " o f th e m a r k e t m a yno t be p ro f i t a b l e , bu t i f i t i s t oo s l ow, u se r s ma ybe c ome f rus t r a t e d a nd swi t c h t o o t he r p roduc t s .

F i na l l y , we do no t be l i e ve t ha t t r a i n i ng a nd ve n -do r suppor t a re un i mpor t a n t , bu t r a t he r t ha t t he y

h a v e b e e n d e c o u p l e d f r o m t h e p a r t i c u l a r s o f t w a r epa c ka ge bough t a nd i t s ve ndor . Tha t i s , t he re a renume rous , a l t e rna t i ve fo rms o f t r a i n i ng a nd suppor twi t h i n o rga n i z a t i ons , p ro fe s s i ona l soc i e t i e s , a nd e du -ca t iona l ins t i tu t ions .

A c k n o w l e d g e m e n t sThe a u t ho rs wou l d l i ke t o g ra t e fu l l y a c knowl e dge

t h e h e l p f u l c o m m e n t s o f a n o n y m o u s r e v i e w e r s a n dthe gues t edi tor .

A p p en d ix A . Qu es t ion n a ireC i r c le t h e w o r d p r o c e s s i n g p r o g r a m y o u p r i m a r i ly u s e o n y o u r m i c r o co m p u t e r . ( C i rc l e o n e o n l y . )1. M i c r o p r o I n t e r n at i o n al W o r d s t a r , 2 0 0 0 , 2 0 0 0 +2 . P e r fe c t S o f t wa re P e r f e c t W r i t e r3 . S o f t w a r e P u b l i sh i n g C o r p o r a t i o n P F S : W r i t e4 . S o r c i m / I U S E a s y W r i t e r I I5 . Le x i so f t S p e l lb in d er6 . M u l t im a t e I n t e r n at io n a l C o r p o r a t io n M u l t i m a t e7 . P e a c h t r e e P e a c h t e x t 5 0 0 08 . I B M D i s p l a y W r i t e 2 , 39 . I BM Word s Ed i t ion10. IBM Writ ing Ass is tant1 1. L e a d i n g E d g e P r o d u c t s , I n c o rp o r a t e d L e a d i n g E d g e W o r d Proces s or1 2. L i f e tr e e S o f tw a r e , I n c o r p o r a te d V o l k s w r i t e r ( D e l u x e ) Scient i f ic13 . Mi c roso f t Corpo ra t i on M i c r o s o f t W o r d1 4 . N B I , I n c o r p o r a t e d N B I W o r d Proces s in g15 . S a mna Corpora t i on S a m n a W o r d , S a m n a +1 6 . M a r k o f t h e U n i c o r n T h e F i n a l W o r d1 7 . M e t a s o f t C o r p o r a t i o n B e n c h m a r k W o r d P r o c e s s o r18 . S e l e c t In fo rma t i on S ys t e ms , Inc o rpo ra t e d S e l e c t1 9 . X y Q u e s t , I n c o r p o r a t e d X y W r i t e I I2 0 . A p p l e C o m p u t e r , I n c o r p o r at e d M a c w r i t e2 1. I n n o va t iv e S o f tw a r e T h e S m a r t W o r d P r o c e s s o r22 . S a t e l l it e S o f t wa re In t e rna t i ona l W o r d P e r f e c t23 . Of f i c e S o l u t i ons O f f i c e W r i t e r24 . Ot he r


14/18

88 E . E . A n d e r s o n , Y . - M . C h e n / D e c i s i o n S u p p o r t S y s t e m s 1 9 ( 1 9 9 7 ) 7 5 - 9 2

H o w w o u l d y o u r a t e t h i s w o r d p r o c e s s i n g p r o g r a m w i t h r e s p e c t t o :P o o r C i r c l e o n e f o r e a c h E x c e l l e n t

Bas ic func t ions 1 2 3 4 5 6 7 8 9 10D o c um e n ta t io n 1 2 3 4 5 6 7 8 9 10A d va n c e d f unc t ions 1 2 3 4 5 6 7 8 9 10V e ndo r supp o r t 1 2 3 4 5 6 7 8 9 10E a s e o f u s e 1 2 3 4 5 6 7 8 9 1 0Tra in in g t im e 1 2 3 4 5 6 7 8 9 10C os t a nd pe r f o r m a nc e 1 2 3 4 5 6 7 8 9 10Ov era l l sa t i s fac t ion 1 2 3 4 5 6 7 8 9 10

S o u r c e : U s e r R a t i n g s o f M i c r o c o m p u t e r S o f t w a r e , Microcomputer Software, D a t a p r o R e s e a r c h C o r p o r a t i o n ,M a r c h 1986 [ 60 ] .

Appendix BT h e e r r o r t e r m o f E q . ( 2 ) m a y b e e x p r e s s e d a s

ENJi = 1 E i jU jI t s e x p e c t e d v a l u e e q u a l s z e r o , b u t i t s v a r i an c e i s n o t c o n s t an t , V A R ( ~ j ) = ~r2/N . H e nc e , t he va r i a nc e o f ~ : j ,

w o u l d v a r y i n v e r s e l y w i t h N j a n d h e t e r o s c e d a s t ic i t y w o u l d b e p r e s e n t. T h e r e f o r e , t he p a r a m e t e r s e s t im a t e d E q .( 2 ) by o r d ina r y l e a s t - squa r e s , w i thou t a d ju s tm e n t s t o t he da t a , w ou ld l e a d to i ne f f i c i e n t e s t im a te s , i . e . , t hev a r i a n c e s o f t h e e s t i m a t e d p a r a m e t e r s a r e n o t t h e m i n i m u m v a r i a n c e s .

O n e a p p r o a c h t o t h i s p r o b l e m i s t o t r a n s f o r m t h e d a t a s o t h a t th e a s s u m p t i o n o f a c o n s t a n t e r ro r v a r i a n c e i ss a t is f ie d . I f w e p r e m u l t i p l y E q . (2 ) b y ~ - ~ ,

the n


15/18

i

o~

L~

II.

rr

I

i

I

ti

IIf~

I

Ii

I

I

ac

i

II

t

I

t

o.

~--

i

J

u

I

i

It

t

I

i

I

I

i


16/18

90A p p e n d i x D

E t x I~ =

E.E. Anderson , Y . -M. C he n/ Decis ion Support Sys tems 19 (1997) 75-92

- W f z W f ~ ' W f x S f z' W f x D f t 'S f z W f _t' S f x S ~ ' S f z D ~ 'D f z W f z ' D f ~ S ~ ' D ~ D f x 'C f z W f x ' C f z S f z ' C f z D ~ tGpWp' a p Sp' a p Dp'

- o ' w , , l o ' ., ~ I ~ r w , t l O ' w , , l6r+.w ~ .s l Osd l O'~cO ' a w l O d s l ~ r a , t l O ' d c lO ' c w l o - , . ~ I o -c ~ l O - c c lo ' ~ ~ I o ' ~ , ~ o,e t l ~rg . l

wpcp'sp cp'D ~ C ~ 'cp c~'G ~ C ~ '

o-,,g lOrdgl~rc l% e !

Dfz G fx'C p C p '

References[1] A.D. Adams, R. Nelson and P.A. Todd, Perceived Useful-

ness, Ease of Use, and Usage of Information Technology: AReplication, MIS Quarterly 16, No. 2 (June 1992) 227-247.

[2] A.H. Agajanian, A Bibliography on System PerformanceEvaluation, Performance Evaluation Review 5, No. 1 (Feb.1976) 53-64.

[3] D.L. Amoroso, Using End User Characteristics to FacilitateEffective Management of End User Computing, Journal ofEnd User Computing 4, No. 4 (Fall 1992) 5-15 .

[4] D. Amoroso, and P. Cheney, A Report on the State ofEnd-User Computing in Some Larger North American Insur-ance Firm, Journal of Information Management 8, No. 2(Spring 1987) 39-48.

[5] E.E. Anderson, A Heuristic for Software Evaluation andSelection, Software: Practice and Experience 19, No. 8 (Aug.1989) 707-717.

[6] T.W. Anderson, An Introduction to Multivariate StatisticalAnalysis, 2nd edition (Wiley and Sons, New York, 1984).

[7] J.E. Bailey and S. Pearson, Development of a Tool forMeasuring and Analyzing User Satisfaction, ManagementScience 29, No. 5 (1983) 530-545.

[8] B. Beizer, Quality is Not the Goal!, American Programmer,No. 6 (June 1993) 5-11.

[9] H. Bidgoli, DSS Products Evaluation: An Integrated Frame-work, Journal of Systems Management, No. 40 (November,1989) 27-34.[10] R.P. Bostrom, L. Olfman and M. K. Sein, The Importance ofLearning Style in End-User Training, M1S Quarterly 14, No.1 (March, 1990) 101-119.

[11] P.H. Cheney, R.I. Mann and D.L. Amoroso, OrganizationalFactors Affecting the Success of End-Users Computing,Journal of Management Information Systems 3, No. 1(Summer, 1986) 65-80.

[12] D. Coupal and P.N. Robillard, Factor Analysis of Source

Code Metrics, Journal o f Systems and Software 12, No. 3(July 1990) 263-69.

[13] J.S. Cramer, Efficient Grouping, Regression and Correlationin Engel Curve Analysis, Journal of the American StatisticalAssociation 59 (March, 1964) 233-250.

[ 14] T.P. Cronan and D.E. Douglas, End-User Training and Com-puting Effectiveness in Public Agencies, Journal of Manage-ment Information Systems 6, No. 4 (Spring 1990) 21-40.

[15] F.D. Davis, Perceived Usefulness, Perceived Ease of Use,and User Acceptance of Information Technology, MIS Quar-terly 13, No. 3 (September, 1989) 319-340.

[16] F.D. Davis, R.P. Bagozzi and P.R. Warshaw, User Accep-tance of Computer Technology: A Comparison o f Two Theo-retical Models, Management Science 35, No. 8 (August,1989) 982-1003.

[17] S.A. Davis and R.P. Bostrom, Training End Users: AnExperimental Investigation of the Roles of the ComputerInterface and Training Methods, MIS Quarterly 17, No. 1(March 1993) 61-81.

[18] DOD-STD2168: Defense System Software Quality Program,Military Standard of the Department of Defense of theUnited States of America, Draft, April 1987.

[19] W.J. Doll and M.U. Ahmed, Documenting Information Sys-tems for Management: A Key to Maintaining User Satisfac-tion, Information and Management 8, No. 4 (1985) 221-226.

[20] W.J. Doll and G. Torkzadeh, The Measurement of End-UserComputing Satisfaction, M1S Quarterly 12, No. 2 (June1988) 259-276.

[21] W.J. Doll and G. Torkzadeh, The Quality of User Documen-tation, Information and Management 12, No. 2 (1987) 73-78.

[22] I. Eriksson and A. TSrn, A Model For IS Quality, SoftwareEngineering Journal, No. 6 (July 1991) 152-158.

[23] A. Eskenasi, Evaluation of Software Product Quality byMeans of Classification Methods, Journal of Systems andSoftware 10, No. 3 (Oct. 1989) 213-16.

[24] C.R. Franz and D. Robey, Organizational Context, User


17/18

E.E. Anderson, Y.-M. Chen / Decision Support Systems 19 (1997) 75-92 91Involvement, and the Usefulness of Information Systems,Decision Sciences 17, No. 3 (Summer 1986) 329-356,

[25] C.W. Frenzel, Management of Information Technology (Boydand Fraser, Boston, 1992).

[26] N.C. Goodwin, Functionality and Usability, Communicationsof the ACM 30, No. 3 (March 1987) 229-233.

[27] J.D. Gould and C. Lewis, Designing for Usability: KeyPrinciples and What Designers Think, Communications ofthe ACM 28, No. 3 (March 1985) 300-311.

[28] G.I. Green and C.T. Hughes, Effects of DSS Training andCognitive Style or Decision Process, Journal of ManagementInformation Systems 3, No. 2 (Fall 1986) 83-93.

[29] W.H. Greene, Econometric Analysis (Macmillan, New York,1990).

[30] A.W. Harrison and R.K. Rainer, Jr., The Influence of Indi-viduals Difference on Skill in End-User Computing, Journalof Management Information Systems 9, No. I (Summer1992) 93-111.

[31] H. Hwang, Estimation of a Linear Sur Model With UnequalNumber of Observations, Review of Economics and Statis-tics 72, No. 3, 510-515.[32] ISO-8402: Quality Vocabulary, International StandardizationInstitute (1986).

[33] B. Ives, M.H. Olson and J.J. Baroudi, The Measurement ofUser Information Satisfaction, Communications of the ACM26. No. 10 (October 1983) 785-793.

[34] M. Kletke, J.E. Trumbly and D.L. Nelson, Integration ofMicrocomputers into the Organization: A Human AdaptationModel and the Organizational Response, Journal of Micro-computer Systems Management 3, No. 1 (Winter 1991)23-35.

[35] K.E. Knight, A Functional and Structural Measurement ofTechnology, Technological Forecasting and Social Change27, No, 2/3 (May 1985) 107-127.[36] H.C. Lucas, Performance and the Use of an InformationSystem, Management Science 21, No. 8 (April 1975) 908-919.

[37] G.S. Maddala, Econometrics (McGraw-Hill, New York,1988).

[38] R.A. Mata-Toledo and D.A. Gustafson, A Factor Analysis ofSoftware Complexity Measures, Journal of Systems Soft-ware, No. 17 (1992) 267-273.

[39] N.P. Melone, A Theoretical Assessment of the User-Satisfac-tion Construct in Information Systems Research, Manage-ment Science 36, No. 1 (January 1990) 76-91.

[40] H.W. Miller, Quality Software: The Future of InformationTechnology, Journal of Systems Management, No. 40 (Dec.1989) 8-14.[41] E. Mumford and I. Banks, The Computer and the Clerk(Routledge and Kegan Paul, London, 1967).

[42] M.C. Munro, S.L. Huff and G. Moore, Expansion and Con-trol of End-User Computing, Journal of Management Infor-mation Systems 4, No. 3 (Winter 1987-1988) 5-27.

[43] R.R. Nelson and P.H. Cheney, Training End Users: AnExploratory Study, MIS Quarterly 11, No. 4 (December1987) 547-559.

[44] S.I. Nesbit, Evaluating Micro Software, Datamation 30, No.11 (July, 15 1984)74-78.

[45] R.K. Rainer, Jr. and A.W. Harrison, Toward Development ofthe End User Computing Construct in a University Setting,Decision Sciences 24, No. 6 (November/December 1993)1187-1202.

[46] D.W. Rasmussen and T.W. Zuehlke, On the Choice ofFunctional Form for Hedonic Price Function, Applied Eco-nomics 22, No. 4 (April 1990) 431-38.

[47] S. Rivard and S.L. Huff, Factors of Success for End-UserComputing, Communications of the ACM 31, No. 5 (May1988) 552-561.

[48] D. Roby, User Attitudes and Management Information Sys-tem Use, Academy of Management Journal 22, No. 3 (Sep-tember 1979) 527-538.

[49] G.C. Roper-Lowe and J.A. Sharp, The Analytic HierarchyProcess and its Application to an Information TechnologyDecision, Journal of the Operational Research Society 41,No. I (January 1990)49-59.

[50] A. Rushinek and S. Rushinek, Mini/Micro Computer Evalu-ation of System Features: An Empirical Discriminant Modelof Software and Hardware Expendability, Compatibility,Costefficiency, Installation and Delivery, Managerial andDecision Economics 5, No. 3 (June 1984) 150-159.

[51] A. Rushinek and S. Rushinek, What Makes Users Happy?,Communications of the ACM 29, No. 7 (July 1986) 594-598.

[52] L. Sachs, Applied Statistics: A Handbook of Techniques(Springer-Verlag, New York, 1984).

[53] A.H. Segars and V. Grover, Re-Examining Perceived Ease ofUse and Usefulness: A Confirmatory Factor Analysis, MISQuarterly 17, No. 4 (September 1993) 517-525.

[54] M.K. Sein, R.P. Bostrom and L. Olfman, Training End Usersto Compute: Cognitive, Motivational and Social Issues, IN-FOR 25, No. 3 (1987) 236-255.

[55] D.W. Straub, Validating Instruments in MIS Research, MISQuarterly 13, No. 2 (June 1989) 147-170.[56] B. Szajna and R.W. Scamell, The Effects of InformationSystem User Expectations on Their Performance and Percep-tions, MIS Quarterly 17, No. 4 (December 1993) 493-514.

[57] G. Torkzadeh, The Quality of User Documentation: AnInstrument Validation, Journal of Management InformationSystems 5, No. 2 (Fall 1988) 99-108.

[58] G. Torkzadeh and W.J. Doll, Test-Retest Reliability of theEnd-User Computing Satisfaction Instrument, Decision Sci-ence 22, No. 1 (Winter 1991) 26-37 .

[59] J.A. Turner, Computer Mediated Work: The Interplay Be-tween Technology and Structural Jobs, Communications ofthe ACM 27, No. 12 (December 1984) 1210-1217.

[60] User Ratings of Computer System, Microcomputer Software(Datapro Research Corporation, Delran, NJ, 1986).[61] J. Wesselius and F. Ververs, Some Elementary Questions onSoftware Quality Control, Software Engineering Journal, No.5 (November 1990) 319-330.

[62] M.S. Wu, Selecting the Right Software Application Package,Journal of Systems Management 41 (September 1990) 28-32.

[63] A. Zellner, An Efficient Method of Estimating SeeminglyUnrelated Regressions and Tests for Aggregation Bias, Jour-nal of the American Statistical Association 57 (1962) 348-368.


18/18

9 2 E.E. Anderson, Y.-M. Chen / D ecision Suppor t System s 19 (1997) 75 -92E v a n A n d e r s o n i s t h e G M U F o u n d a t i o n P r o f e ss o r a n dD i r e c to r o f T e c h n o l o g y M a n a g e m e n t in t h e G m d u a l e B u s i -n e s s In s t i t u te a t G e o r g e M a s o n U n i v e r s i t y . H e i s t h e D i r e c t o ro f t h e IT - M a n a g e m e n t C o n s o r t i u m a n d a m e m b e r o f t h eI n s t i t u t e f o r C o m p u t a t i o n a l S c i e n c e s a n d l n f o r m a t i c s . H er e c e iv e d h i s P h . D . f r o m C o m e l l U n i v e r s i ty a n d h e l p p r i o rf a c u lt y a n d a d m i n i s t r a ti v e a p p o i n t m e n t s a t T u l a n e U n i v e r -s i t y , V a n d e r b i l t U n i v e r s i t y , a n d t h e U n i v e r s i t y o f T e x a s ,D a l l a s . H e h a s b e e n a V i s i t i n g S c h o l a r a t t h e U n i v e r s i t y o fC h i c a g o a n d a S e n i o r M e m b e r o f St . A n t o n y ' s C o l le g e ,U n i v e r s i t y o f O x f o r d . H i s t e a c h i n g a n d r e s e a r c h i n t e r e s t si n c lu d e e c o n o m i c s o f c o m p u t i n g a n d i n f o rm a t i o n t e c h n o lo -g i e s , m a n a g e m e n t o f I T b u s i n e s s e s , a n d i n t e r n a t io n a l p o l i ti -c a l e c o n o m y . H i s p a p e r s h a v e a p p e a r e d i n Journal o f Bus i -ness (Univers i ty o f Chicago), Opera t ions Research , Manage-ment Sc ience , Accou nt ing Rev iew, Ma nager ia l and Decis ionEconomics, l i E Transactions, M IS Quarterly, Na val Re-search Logis t ics , Journal o f Management In format ion Sys -tems, a n d o t h e r s .

Y u - M i n C h e n i s t h e S e n i o r S t a t i s t i c i a n a t J C P e n n y C o m -p a n y I n c . i n P l a n t , T X . H i s r e s e a r c h i n t e r e s ts in c l u d e t i m e -s e r i e s , f o r e c a s t i n g m e t h o d s , a p p l i e d e c o n o m e t r i c s , c h o i c em o d e l l i n g , a n d n e u r o n e t w o r k s . H e r e c e i v e d h i s P h . D . f r o mP u r d u e U n i v e r s i ty .

anderson 1997 decision support systems

Documents