overview of mcda –general definition –mcdm process –mcda methods evaluation of wba –quality...
Post on 20-Dec-2015
228 views
TRANSCRIPT
• Overview of MCDA– General definition– MCDM process– MCDA methods
• Evaluation of WBA– Quality Attribute Relationships– Aggregation by Choquet Integral– Implementation– Case study and results
Outlines
• Aims to give the decision-maker some tools in order to enable him to advance in solving a decision problem where several – often contradictory-points of view must be taken into account.
What is MCDA?
• Highly structured, disciplined and formal approach to decision making
• evaluating the alternatives in the given set A against the set C of criteria
• Aggregating the individual evaluations to produce global evaluation
• Could be used for selection the best possible alternatives or for ranking the alternatives
What is MCDM?
MCDM Process
Set of Alternatives Set of Criteria
Weights wi /Importance of
Criteria
Overall worth of an alternative Ai
Aggregation Measure
C1, C2,………Cn
A1 x11……..………x1n
A2 x21……..………x2n
.
.
Am xn1……..………xmn
Evaluation of MCDA methods
• Criteria – interdependence, completeness, non-linear preferences
• Weights – transparency of process, type of weights, meaning
• Solution finding procedure – ranking, option
• Project constraints – cost, time
Evaluation of MCDA methods
• Structure of problem solving process – stakeholder participant, tool for learning transparency, actors communication
• Data Situation – Type of data - qualitative or quantitative– Risk/uncertainties – probabilities, thresholds, fuzzy
numbers, sensitive analysis– Data processing amount– Non-substitutability
Evaluation of WBA• Quality of web site is hard to evaluate
– Consists of multiple criteria to be measured• Simple weighted average cannot be used to
summaries the various quality measurements into a single score.
• Inability to account for dependency among the quality criterion.
• Tend to construct independent criteria, or criteria that are supposed to be so– Causing some bias effect in evaluation
Single criteria
• usability aspects(Collins, 1996; Stefani & Xenos, 2001; Hassan & Li, 2005),
• content and structure (Bauer & Scharl, 2000).
• accessibility (Vigo et al., 2007)
WBA Evaluation Approaches
Multi-criteria
• WEBQEM (Olsina et al., 1999)
• EWAM (Schubert & Selz, 1998)
• WebQual (Barnes and Vidgen, 2002)
• WAI (Miranda et al., 2006)
• FQT4Web (Davoli et al., 2005)
WBA Evaluation Approaches
Stated or implied needs
QualityRequirement
Definition
Software Developmen
t
ISO 9126 & other technical info
Quality requirement specification
Metric Selection
Rating level
definition
Assessment criteria definition
Measurement
Rating
Assessment
Products
Measured value
Rated value Result
(acceptable or unacceptable)
Requirement definitionManagerial
requirement
Preparation
Evaluation
ISO/IEC 9126 Evaluation Process
Quality Model
Indicators, scales and preferred values
QU
AL
ITY
CH
AR
AC
TE
RIS
TIC
S
APPLICATIONDOMAIN
e-commercee-learning
e-educatione-government
etc.
SU
B C
HA
RA
CT
ER
IST
ICS
Functionality Reliability Usability Efficiency PortabilityMaintainability
suitabilityaccuracyinteroperabilitysecuritytraceabilityfunctionalitycompliance
maturityfault tolerancerecoverabilityavailabilitydegradabilityreliabilitycompliance
understandabilitylearnabilityoperabilityattractivenessexplicitenesscustomisabilityclarityhelpfulnessuser-friendlinessusability compliance
time behaviourresourceutilisationefficiencycompliance
Analysabilitychangeabilitystabilitytestabilitymanageabilityreusabilitymaintainabilitycompliance
Adaptabilityinstallabilitycoexistencereplaceabilityportabilitycompliance
Quality Attributes for WBA
• Define software product qualities as a hierarchy of factors, criteria and metrics.
• Quality factor represents behavioral characteristics of the system
• Quality criterion is an attribute of a quality factor that is related to software production and design
• Quality metrics is a measure that captures some aspect of a quality criterion.
Factor A
Criteria a1,
weight 4
Criteria a2,
weight 1
Criteria a3,
weight 2
Overall Quality Score
Factor B Factor C
Factor A is split up into three criteria a1, a2, and a3.
Criteria a1 with the weight 4
is considered four times as important as criteria a2 and
twice as important as criteria a3.
Similarly, we can set different weight for each factor to indicate its importance.
Definition of Quality Attributes
Name Description
Functionality
The capability of the Web site to provide functions and properties which meet stated and implied needs when the site is used under specified conditions
Usability
The capability of the Web site to be understood, learned and liked by the user, when used under specified conditions
Reliability
The capability of the Web site to maintain a specified level of performance when used under specified conditions.
Efficiency
The capability of the site to provide appropriate performance, relative to the amount of resource used, under stated conditions
Maintainability The capability of the site to be modified. Modifications may include corrections, improvements or adaptation of the site to changes in environments, and in requirements and functional specifications
Portability
The capability of the site to be transferred from one environment to another
Quality Attributes Relationships
Three types of relationships• Positive, i.e. a good value of one attribute result in a
good value of the other (synergistic goals).– Relationships definitions: If characteristics A is enhanced,
then characteristics B is likely to be enhanced (+)
• Negative, i.e. a good value of one attribute result in a bad value of the other (conflicting goals).– Relationships definitions: If characteristics A is enhanced,
then characteristics B is likely to be degraded (-)
• Independent, i.e. the attributes do not affect each other.– Relationships definitions: If characteristics A is enhanced,
then characteristics B is unlikely to be affected (0)
Interrelationships between quality factors (Perry, 1987)
Relationship Chart (Gillies, 1997)
Techniques to explore the relationships
Ref Attributes Purpose Techniques used
[8, 9]
Correctness, ReliabilityIntegrity, UsabilityEfficiency, MaintainabilityTestability, FlexibilityPortability. ReusabilityInteroperability
To study the relations of different quality goals attribute in developing software
Survey -questionnaire
[10]PerformanceAdaptabilityMaintainability
To address the importance of design decision made during software development
Case Study - Interview
[11]UsabilityTime to marketReliability, UsabilityCorrectness, Portability
To increase the understanding of software quality attributes and their relations
Research Literature and Survey –structured interview
[12] Quality attributes in 3 different perspectives: management, developer and user perspective
To merge different view and discuss the relationships between the quality attributes
Discussion (meeting and offline discussion)
Quality Attributes Relationships for WBA
• method of combining several numerical values into a single one, so that the result of aggregation takes into account in a given manner all the individual values
What is Aggregation?
• use simple weighted average approach• methods are not transparent• assume independency• the choice of summarization method somehow
should depend on the certain parameters– E.g. the kind of importance parameters (weights) and
the type of dependency and interaction
• the definition of the quality factors and their relationships must be clearly specified
Aggregation issues
Common aggregation operators
• Quasi-arithmetic means (arithmetic, geometric, harmonic, etc.)– Not stable under linear transformation and
consider criteria as non interacting• Median
– Typical ordinal operator – defined the middle value of the ordered list
• Weighted minimum and maximum– Possible to increase one of the weights without
having any change in the result• Ordered weighted averaging operators
– Can express vague quantifiers
23
Properties of an aggregation operator
mathematical properties– Properties of extreme values– Idempotence– Continuity– Monotonicity– Commutativity– Decomposability– Stability under the same positive
linear transformation
behavioural properties– express the decisional behavior,
interaction between criteria, interpretability of the parameters and weights on the arguments
Properties of an aggregation operator
Aggregation by fuzzy integral
• Different methods have been developed according to– type of information to be aggregated
and – the properties have to be satisfied.
26
Fuzzy measures and integral
Definition 1: A fuzzy measure on the set X of criteria is a set function : Ƥ (X) [0,1], satisfying the following axioms
i. ()=0, (X)=1.ii. A B X implies (A) (B)
(A) represent the weight of importance of the set of criteria A.
Additive : if (AB) = (A) + (B); A B=Superadditive: if (AB) (A) + (B); A B=Subadditive if (AB) (A) + (B); A B=If a fuzzy measure is additive, then it suffices to define n
coefficients (weights) ({ I}), … ({ n})27
Choquet integral
Definition 2: Let be a fuzzy measure on X. The choquet integral of a function ƒ : (X) [0,1] with respect to is defined by
28
C (f(x1),…. f(xn)):= (f(x(i)) - f(x(i-1))) (A(i) )
ƒ ((0)) = 0
n
i = 1
•Fuzzy integral model does not need to assume independency•Fuzzy integral of ƒ with respect to gives the overall evaluation of an alternative
Importance and interaction of criteria
• Problem of evaluation of student with respect to three subjects: mathematics (M), Physics (P) and literature (L).
• By weighted sum (3 , 3, 2) result:
29
Solved by fuzzy measure and the choquet integral
1. Scientific subjects are more important than literature;
({M}) = ({P}) =0.45; ({L}) = 0.32. M and P are redundant,
({M, P}) = 0.5 < 0.45 + 0.453. Students equally good at scientific subjects and
literature, ({L, M}) = 0.9 > 0.45 + 0.3
({L, P}) = 0.9 > 0.45 + 0.34. ()=0, ({M, P, L})=1
30
Result by applying fuzzy measure:
* The initial ratio of weight (3, 3, 2) is kept unchanged
31
Complexity of the model• Number of coefficients grows exponentially with the
number of criteria to be aggregated. • 3 approaches (to reduce the number of
coefficients)1. Identification based on semantics
– Importance of criteria– Interaction between criteria– Symmetric criteria– Veto effects
2. Identification based learning data– Minimization of squared error– Constraint satisfaction
3. Combining semantics and learning 32
Proposed solution
• Apply 2-additive Choquet integral• provide the information about the
relationships among criteria (redundancy or support among criteria) and the preference among alternatives
• Derive fuzzy measures by constraint satisfaction
Explore relationships
• Techniques to explore how the different attributes are related to each other:– Experience Based Approach – Mathematical Modeling – Statistical Technique (Correlation Analysis)
• measures the strength of a linear relationship among different quality factors
• The main result of a correlation is called the correlation coefficient (r)
Correlation Result
1. Definition of the initial preferences.
2. Convert into Choquet integral form
3. Identify threshold values.
4. If solution exists, calculate the Choquet integral, Shapley value and Interaction indices
Implementation of Choquet Integral
A partial weak order A over A (ranking of the webs), A partial weak order N over N (ranking of the importance of the quality factor),
Quantitative intuitions about the relative importance of some quality factor
A partial weak order P over the set of pairs of quality factor (ranking of interactions),
Intuitions about the type and the magnitude of the interaction between some quality factor,
The behavior of some quality factor as veto or favor, Etc.
Define preference thresholds
P r e f e r e n c e s C h o q u e t I n t e g r a l 'xx A CxuCxuC ))'(())(( R a n k in g o f
a l t e r n a t iv e s '~ xx A CC xuCxuC ))'(())((
ji N Shji )()( R a n k in g o f c r i t e r ia ( w e ig h t s )
ji N~ ShSh ji )()(
klij P IklIijI )()( R a n k in g o f p a i r s o f c r i t e r ia ( in t e r a c t io n s )
klij P~ II klIijI )()(
R a n g e o f in t e r a c t io n s
bijIa )( , 1,1, ba S ig n o f s o m e in t e r a c t io n s
C o m p le m e n t a r y o r R e d u n d a n t
mijI )( o r
mijI )( ; ]1,0[m
Convert into Choquet integral form
• Three preference thresholds C, Sh & I have to be determined before the aggregation take part.
• Range of : 0 to 1• no rule to fix the , we need to compare
the solutions obtain with different value of .
• Once the solution exist, Choquet integral will be calculated
Define preference thresholds
Calculate the Choquet integral
n
i Xjijiijii xxaxaK
1 },{
)(
For 2-additive fuzzy measure, we have for any KX:
w i t h 1ix i f i K , 0ix o t h e r w i s e . W e d e d u c e t h a t ii a f o r
a l l i , a n d ijjiijjiij aaaa .
kKixk
KiK
n
kkiv
||,\
1
0
)(
!
!)!1(
n
kknk
with
Calculate the Shapley value
n = total number of criteria, k = number of elements in a sub-set
Shapley index can be interpreted as a kind of average value of the contribution of element i, individual criteria, alone in all coalitions.
Summation of these Shapley values for a given set of elements would represent the importance of the complete set
Calculate the Interaction Index
kKjixk
KjKiKijK
n
kkijI
||},,\{
2
0
)(
With
)1(21
)!1(
!)!2(
n
k
nn
kknk
The interaction index Iij can be interpreted as a kind of average value of
the added value given by putting i and j together, all coalitions being considered. When Iij is positive (resp. negative), then the interaction is
said to be positive (resp. negative).
Case Study
• Perform on 3 types of WBA– Academic– E-commerce– Museum
• Four quality factor were evaluated– Usability,Functionality, Reliability, Efficiency
• Each has different preference, importance and interaction
Table 1. Academic websites data
Univ Websites Usability (U) Functionality(F) Reliability(R) Efficiency(E)
A1 76.18 61.84 60.40 69.09
A2 51.01 50.39 87.62 52.03
A3 80.08 48.49 90.86 76.11
A4 57.71 38.99 88.70 53.12
A5 71.93 82.04 83.05 85.99
A6 60.94 71.12 63.61 69.47
Result for academic website
Table 2. Relative importances for each quality attribute
U F R E Relative importance
(Weight) 0.3 0.3 0.2 0.2
Table 3. Interaction between quality attribute
U F R
U
F -
R + +
E - - +
T a b l e 4 . I n t e r a c t i o n p r e o r d e r c o n s t r a i n t m a t r i x
U F R
U
F IUFI )(1
R 1)( URII 1)( FRII
E IUEI )(1 IFEI )(1 1)( REII
E a c h r o w c o r r e s p o n d i n g t o a c o n s t r a i n t o f f o r m bijIa )( , 1,1, ba .
Table 5. The Möbius values
LP MV {} 0.000000 0.000000 {U} 0.366704 0.321697 {F} 0.326805 0.285376 {R} 0.000001 0.094367 {E} 0.277765 0.298559 {U, F} -0.226804 -0.100000 {U, R} 0.137865 0.100000 {U, E} -0.100000 -0.100000 {F, R} 0.217664 0.100000 {F, E} -0.100000 -0.100000 {R, E} 0.100000 0.100000
Threshold C= 1, Sh = 0.1, I =0.1,
Table 7. The interaction indices for LP
U F R
U
F -0.2268038
R 0.1378647 0.2176637
E -0.1000000 -0.1000000 0.1000000
Table 8. The interaction indices for MV
U F R U F -0.1000000 R 0.1000000 0.1000000 E -0.1000000 -0.1000000 0.1000000
Table 6. The Shapley values
U F R E LP 0.2722348 0.2722348 0.2277652 0.2277652
MV 0.2716970 0.2353761 0.2443674 0.2485595
Table 9. Overall Scores
Global evaluation obtained by
Univ Websites WA LP MV
A1 67.304 67.73138 67.32480
A2 58.350 51.26041 54.75639
A3 71.965 72.10125 74.05597
A4 57.374 52.36041 55.79382
A5 79.999 81.44097 81.17426
A6 66.234 66.63138 66.32480
LP MV Möbius values Very extreme value
and does not necessarily lead to a unique solution
Lead to unique solution
Shapley value Sometimes incompatible with the initial relative importance
Mostly compatible with the initial relative importance
Interaction indices Satisfy the constraints Satisfy the constraints
Global evaluation Leads to more dispersed values
Closer to the simple weighted arithmetic mean
Summary(1)
Aggregation approaches
Arithmetic Mean (AM)
Weighted Average
(WA)
Logic Scoring
Preference (LSP)
Choquet Integral (CI)
Importance / / / Veto / /
Favour / / Interaction /
Additive / / / / Non-additive / /
Summary(2)
Rank Score Rank Score Rank Score Rank Score Rank Score Rank
A1 3 66.88 3 67.3 3 66.91 3 67.32 3 67.08 3
A2 6 60.26 5 58.35 5 56.55 5 54.76 6 60.16 5
A3 2 73.89 2 71.97 2 69.61 2 74.06 2 73.18 2
A4 5 59.63 6 57.37 6 54.46 6 55.8 5 59.16 6
A5 1 80.75 1 80 1 79.76 1 81.17 1 80.66 1
A6 4 66.29 4 66.23 4 66.05 4 66.32 4 66.09 4
MIN-MAX
STD DEV
Univ Websites
Global evaluation obtained by
AM WA LSPChoquet Integral
(Interaction)Choquet Integral (No Interaction)
59.63-80.75
8.141
57.37-80
8.507
54.46-79.76
9.214
54.76-81.17
10.250
59.16-80.66
8.133
Comparison with other approaches
Conclusion• Aggregation by Choquet integral can be
alternated if there is interaction exist between quality factors.
• The proposed approach can be applied for non-interactive criteria as well. If there is no interaction between the criteria, then the fuzzy measure will be additive measures.
• Results show that the global evaluation obtained is compatible with the weighted average method.
Future works
• The evaluation of WBA which cater the dynamic changes of the quality factors.– Behavior (Preferences,
importance,interaction, etc.) can be change continuously.
• Investigate more than 2 quality attribute interactions