research article an analytic hierarchy model for...

Research ArticleAn Analytic Hierarchy Model for Classification AlgorithmsSelection in Credit Risk Analysis

Gang Kou12 and Wenshuai Wu3

1 School of Business Administration Southwestern University of Finance and Economics Chengdu 611130 China2 Collaborative Innovation Center of Financial Security Southwestern University of Finance and Economics Chengdu 611130 China3 School of Management and Economics University of Electronic Science and Technology of China Chengdu 610054 China

Correspondence should be addressed to Gang Kou kougangyahoocom

Received 23 January 2014 Accepted 16 April 2014 Published 4 May 2014

Academic Editor Fenghua Wen

Copyright copy 2014 G Kou and W Wu This is an open access article distributed under the Creative Commons Attribution Licensewhich permits unrestricted use distribution and reproduction in any medium provided the original work is properly cited

This paper proposes an analytic hierarchy model (AHM) to evaluate classification algorithms for credit risk analysis The proposedAHM consists of three stages data mining stage multicriteria decision making stage and secondary mining stage For verification2 public-domain credit datasets 10 classification algorithms and 10 performance criteria are used to test the proposed AHM in theexperimental study The results demonstrate that the proposed AHM is an efficient tool to select classification algorithms in creditrisk analysis especially when different evaluation algorithms generate conflicting results

1 Introduction

The main objective of credit risk analysis is to classifysamples into good and bad groups [1 2] Many classificationalgorithms have been applied to credit risk analysis such asdecision tree K-nearest neighbor support vector machine(SVM) and neural network [3ndash9] How to select the bestclassification algorithm for a given dataset is an importanttask in credit risk prediction [10ndash12] Wolpert and Macready[13] pointed out in their no free lunch (NFL) theoremthat there exists no single algorithm or model that couldachieve the best performance for a given problem domain[14 15] Thus a list of algorithm rankings is more effectiveand helpful than seeking the optimal performed algorithmfor a particular task Algorithm ranking normally needs toexamine several criteria such as accuracy misclassificationrate and computational timeTherefore it can be modeled asa multicriteria decision making (MCDM) problem [16]

This paper develops an analytic hierarchy model (AHM)to select classification algorithms for credit risk analysis Itconstructs a performance score to measure the performanceof classification algorithms and ranks algorithms using mul-ticriteria decision analysis (MCDA) The proposed AHMconsists of three hierarchy stages data mining (DM) stage

MCDM stage and secondary mining stage An experimentalstudy which selects 10 classic credit risk evaluation classifi-cation algorithms (eg decision trees K-nearest neighborssupport vector machines and neural networks) and 10performance measures is designed to verify the proposedmodel over 2 public-domain credit datasets

The remaining parts of this paper are organized as followsSection 2 briefly reviews related work Section 3 describessome preliminaries Section 4 presents the proposed AHMSection 5 describes experimental datasets and design andpresents the results Section 6 concludes the paper

2 Related Work

Classification algorithm evaluation and selection is an activeresearch area in the fields of data mining and knowledgediscovery (DMKD) machine learning artificial intelligenceand pattern recognition Driven by strong business benefitsmany classification algorithms have been proposed for creditrisk analysis in the past few decades [17ndash22] which canbe summarized into four categories statistical analysis (egdiscriminant analysis and logistic regression) mathematicalprogramming analysis (eg multicriteria convex quadratic

Hindawi Publishing CorporationMathematical Problems in EngineeringVolume 2014 Article ID 297563 7 pageshttpdxdoiorg1011552014297563

2 Mathematical Problems in Engineering

programming) nonparametric statistical analysis (eg recur-sive partitioning goal programming and decision trees)and artificial intelligence modeling (eg support vectormachines neural networks and genetic algorithms)

The advantages of applying classification algorithms forcredit risk analysis include the following It is difficult fortraditionalmethods to handle large size databases while clas-sification algorithms especially artificial intelligence model-ing can be used to quickly predict credit risk even whenthe size of dataset is huge Second classification algorithmsmay provide higher prediction accuracy than traditionalapproaches [23] Third the decision making based on theresults of classification algorithms is objective reducing theinfluence of human biases

However the no free lunch theorem states that noalgorithm can outperform all other algorithms when perfor-mance is amortized over all measures Many studies indicatethat classifiersrsquo performances vary under different datasetsand circumstances [24ndash26] How to provide a comprehensiveassessment of algorithms is an important area Algorithmevaluation and selection normally need to examine multi-criteria Therefore classification algorithm evaluation andselection can be treated as an MCDM problem and MCDMmethods can be applied to systematically choose the appro-priate algorithms [16]

As defined by the International Society on MultipleCriteria Decision Making MCDM is the study of methodsand procedures by which concerns aboutmultiple conflictingcriteria can be formally incorporated into the managementplanning process [27 28] MCDM is concerned with theelucidation of the levels of preference of decision alternativesthrough judgments made over a number of criteria [29 30]MCDMmethods have been developed and applied in evalu-ation and selection of classification algorithms For instanceNakhaeizadeh and Schnabl [31] suggested a multicriteria-based measure to compare classification algorithms Smith-Miles [32] considered the algorithm evaluation and selectionproblem as a learning task and discussed the generalizationof metalearning concepts Peng et al [33] applied MCDMmethods to rank classification algorithms However theseresearch efforts face challenging situations that differentMCDM methods produce conflicting rankings This paperproposes and develops AHM a unified framework based onMCDM and DM to identify robust classification algorithmsespecially when different evaluation algorithms generateconflicting results

3 Preliminaries

31 Performance Measures This paper utilizes the followingten commonly used performance measures [33 35]

(i) Overall accuracy (Acc) accuracy is the percentageof correctly classified instances It is one of the mostwidely used classification performance metrics

Overall accuracy = TN + TPTP + FP + FN + TN

(1)

where TN TP FN and FP stand for true negative truepositive false negative and false positive respectively

(ii) True positive rate (TPR) TPR is the number ofcorrectly classified positive instances or abnormalinstances TPR is also called sensitivity measure

True positive rate = TPTP + FN

(2)

(iii) True negative rate (TNR) TNR is the numberof correctly classified negative instances or normalinstances TNR is also called specificity measure

True negative rate = TNTN + FP

(3)

(iv) Precision this is the number of classified fault-pronemodules that actually are fault-prone modules

Precision = TPTP + FP

(4)

(v) The area under receiver operating characteristic(AUC) receiver operating characteristic stands forreceiver operating characteristic which shows thetradeoff between TP rate and FP rate AUC representsthe accuracy of a classifier The larger the area thebetter the classifier

(vi) 119865-measure it is the harmonic mean of precision andrecall 119865-measure has been widely used in informa-tion retrieval

119865-measure = 2 times Precision times RecallPrecision + Recall

(5)

(vii) Mean absolute error (MAE) thismeasures howmuchthe predictions deviate from the true probability119875(119894 119895) is the estimated probability of 119894 module to beof class 119895 taking values in [0 1]

MAE =sum

119888

119895=1sum

119898

119894=1

1003816100381610038161003816

119891 (119894 119895) minus 119875 (119894 119895)

1003816100381610038161003816

119898 sdot 119888

(6)

(viii) Kappa statistic (Kaps) this is a classifier performancemeasure that estimates the similarity between themembers of an ensemble in multiclassifiers systems

Kaps = 119875 (119860) minus 119875 (119864)1 minus 119875 (119864)

(7)

where 119875(119860) is the accuracy of the classifier and 119875(119864)is the probability that agreement among classifiers isdue to chance

(ix) Training time is the time needed to train a classifica-tion algorithm or ensemble method

(x) Test time is the time needed to test a classificationalgorithm or ensemble method

Algorithm evaluation and selection involves benefit andcost criteria Seven performance measures used in this studyare benefit criteriaThey are accuracy kappa statistic TP rateTN rate precision 119865-measure and AUC The other threeperformance measures (ie MAE training time and testtime) are cost criteria

Mathematical Problems in Engineering 3

Secondarymining stage

DM stage

MCDMstage

Acc Kaps MAE AUC

Target data

SVMNaiumlve Bayesian ANN

TOPSIS VIKOR PROMETHEE II

Result revelation

middot middot middot



Figure 1 The proposed analytic hierarchy model

32 Evaluation Approaches

321 DM Method The DM stage of AHM selects 10 classi-fication algorithms which are commonly used algorithms incredit risk analysis to predict credit risk

The main objective of credit risk analysis is to classifysamples into good and bad groups This paper choosesthe following ten popular classification algorithms for theexperimental study [3 36 37] Bayes network (BNK) [38]naive Bayes (NBS) [39] logistic regression (LRN) [40] J48[41] NBTree [42] IB1 [43 44] IBK [45] SMO [46] RBFNetwork (RBF) [47] and multilayer perceptron (MLP) [48]

322 MCDMMethod Multiple criteria decision making is asubdiscipline of operations research that explicitly considersmultiple criteria in decision making environments Whenevaluating classification algorithms normal multicriterianeed to be examined such as accuracy misclassificationrate and computational timeThus algorithm evaluation andselection can be modeled as an MCDM problem

TheMCDM stage of AHM selects four MCDMmethodsthat is technique for order preference by similarity to idealsolution (TOPSIS) [49] preference ranking organizationmethod for enrichment of evaluations II (PROMETHEE II)[50] VIKOR [51] and grey relational analysis (GRA) [52]to evaluate the classification algorithms based on the 10performance measures described in Section 3

4 The Proposed Model

The proposed AHM is developed to evaluate and selectclassification algorithms for credit risk analysis It is designedto deal with situations when different MCDM methodsproduce conflicting rankings [33 53]The approach combinesMCDM DM knowledge discovery in database (KDD) pro-cess and expert opinions to find out the best classificationalgorithm The proposed AHM consists of three stages

DM stage MCDM stage and secondary mining stage Theframework is presented in Figure 1

In the first stage DM stage 10 commonly used classifica-tion algorithms in credit risk analysis including Bayes net-work (BNK) naive Bayes (NBS) logistic regression (LRN)J48 NBTree IB1 IBK SMO RBF network (RBF) andmultilayer perceptron (MLP) are implemented usingWEKA37 The performance of algorithms is measured by the 10performance measures introduced in Section 31 The DMstage can be extended to other functions such as clusteringanalysis and association rules analysis

The MCDM stage applies four MCDM methods (ieTOPSIS VIKOR PROMETHEE II and gray relational analy-sis) to provide an initial ranking tomeasure the performancesof classification algorithms based on the results of the DMstage as input This stage selects more than one MCDMmethod because the ranking agreed by several MCDMmeth-ods is more credible and convincing than the one generatedby a single MCDM method All these MCDM methods areimplemented using MATLAB 70

In the third stage the secondary mining is presentedto derive a list of algorithm priorities and multicriteriadecision analysis (MCDA) is applied to measure the perfor-mance of classification algorithms Expert consensus withthe importance of each MCDM method is applied to thealgorithm evaluation and selection which can reduce theknowledge gap from different experiments and expertiseof experts especially when different evaluation algorithmsgenerate conflicting results

5 Experiment

51 Datasets The experiment chooses 2 public-domaincredit datasets Australian credit dataset and German creditdataset (Table 1) These 2 datasets are publicly available atthe UCI machine learning repository (httparchiveicsuci)(eduml)


Table 1 The two datasets

Total cases Good cases Bad cases Number of attributesGerman data 1000 700 300 20Australian data 690 307 383 14

Input 2 public-domain credit datasetsOutput Ranking of classification algorithmsStep 1 Prepare target datasets data cleaning data integration and data transformationStep 2 Train and test the selected classification algorithms on randomly sampled

partitions (ie 10-fold cross-validation) using WEKA 37 [34]Step 3 Evaluate classification algorithms using TOPSIS VIKOR PROMETHEE II and

GRA MCDMmethods are all implemented using MATLAB 70 based onperformance measures as input

Step 4 Generate two separate tables of the initial ranking of classification algorithmsprovided by each MCDMmethod

Step 5 Obtain the weights of the selected MCDMmethods with decision-making ofexpert consensus Three invited experts agree on that all MCDMmethods areequally important according to the NFL theorem that is to say the weights ofeach MCDMmethod are 025

Step 6 Recalculate the final rankings of classification algorithms using the MCDAmethod

END

Algorithm 1

TheGerman credit card application dataset contains 1000instances with 20 predictor variables such as age gendermarital status education level employment status credithistory records job account and loan purpose 70 of theinstances are accepted to be credit worthy and 30 arerejected

The Australian dataset concerns consumer credit cardapplications It has 690 instances with 445 examples ofcredit worthy customers and 555 examples for creditunworthy customers It contains 14 attributes where eight arecategorical attributes and six are continuous attributes

52 Experimental Design The experiment is carried outaccording to Algorithm 1

53 Experimental Results The standardized classificationresults of the two datasets are summarized in Tables 2and 3 The best result of each performance measure of thetwo datasets is highlighted in boldface No classificationalgorithm has the best result on all measures

The initial ranking of the classification algorithms of thetwo datasets is generated by TOPSIS VIKOR PROMETHEEII and GRA The results are summarized in Tables 4 and 5respectively Weights of each performance measure used inTOPSIS VIKOR PROMETHEE II and GRA are defined asfollows TP rate and AUC are set to 10 and the other threemeasures are set to 1 the weights are normalized and thesum of all weights equals 1 [33] From Table 4 and Table 5 wecannot identify and find the regular pattern of performancesof classification algorithms with intuition What is more theintuition is not always correct and different people often

have different conclusions Based on these observations thesecondary mining stage is proposed in our developed AHM

The final ranking of classification algorithms is calcu-lated by TOPSIS one of the MCDA methods which isimplemented in the secondary mining stage The weights areobtained by decision making with expert consensus Thatis all algorithms are equally important over all measureshaving their own advantages and weaknesses Three invitedexperts agree on the fact that each MCDMmethod is equallyimportant namely theweight of eachMCDMmethod is 025The final ranking results are presented in Table 6

The ranking of classification algorithms produced by twodatasets is basically the same except Bayes network (BNK)and naive Bayes (NBS) Compared with the initial rankingthe degrees of disagreements of the final ranking are greatlyreduced

6 Conclusion

This paper proposes an AHM which combines DM andMCDM to evaluate classification algorithms in credit riskanalysis To verify the proposed model an experimentis implemented using 2 public-domain credit datasets 10classification algorithms and 10 performance measures Theresults indicate that the proposed AHM is able to identifyrobust classification algorithms for credit risk analysis Theproposed AHM can reduce the degrees of disagreements fordecision optimization especially when different evaluationalgorithms generate conflicting results One future researchdirection is to extend the AHM to other functions such asclustering analysis and association analysis


Table 2 Evaluation results of Australian credit dataset

Australian Acc TPR TNR Precision 119865-measure AUC Kaps MAE Training time Test timeBNK 0852 0798 0896 0860 0828 0913 06986 01702 00125 00009NBS 0772 0586 0922 0857 0696 0896 05244 02253 00055 00014LRN 0862 0866 0859 0831 0848 0932 07224 01906 00508 00005J48 0835 0795 0867 0827 0811 0834 06642 01956 00398 00002NBTree 08333 0779 0877 0836 0806 0885 06603 02195 13584 00008IB1 0794 0775 0809 0765 0770 0792 05839 02058 00005 00473IBK 0794 0775 0809 0765 0770 0792 05839 02067 00003 00164SMO 0885 0925 0799 0787 0850 0862 07116 01449 03744 00008RBF 0830 0752 0893 0849 0798 0895 06528 02463 00683 00009MLP 0825 0818 0830 0794 0806 0899 06460 01807 56102 00014

Table 3 Evaluation results of German credit dataset

German Acc TPR TNR Precision 119865-measure AUC Kaps MAE Training time Test timeBNK 0725 0360 0881 0565 0440 0740 02694 03410 00247 00011NBS 0755 0507 0861 0610 0554 0785 03689 02904 00134 00034LRN 0771 0493 0890 0658 0564 0790 04128 03153 01139 00005J48 0719 0440 0839 0539 0484 0661 02940 03241 01334 00005NBTree 0726 0380 0874 0564 0454 0734 02805 0344 19339 00023IB1 0669 0450 0763 0449 0449 0606 02127 03310 00020 01680IBK 0669 0450 0763 0449 0449 0606 02127 03310 00002 00694SMO 0774 0493 0894 0667 0567 0694 04187 02260 05861 00005RBF 0740 0463 0859 0584 0517 0747 03421 03429 01694 00023MLP 0718 0477 0821 0534 0504 0717 03075 02891 200513 00025

Table 4 Ranking of MCDMmethods of Australian credit dataset

Algorithm TOPSIS PROMETHEE II VIKOR GRAValue Rank Value Rank Value Rank Value Rank

BNK 05807 7 05595 2 02080 2 07910 3NBS 09529 1 minus03056 10 08940 8 06271 6LRN 09332 2 07619 1 00000 1 08038 1J48 06608 5 minus00794 6 06139 7 06411 5NBTree 05986 6 minus00794 5 03673 5 06204 7IB1 04703 10 minus06111 9 10000 10 04628 10IBK 05583 8 minus06111 8 09807 9 04883 9SMO 07944 4 03135 4 03068 4 07987 2RBF 08087 3 minus02659 7 04146 6 06612 4MLP 05511 9 03175 3 02739 3 05669 8

Table 5 Ranking of MCDMmethods of German credit dataset


BNK 05807 7 minus02699 6 08434 8 05921 5NBS 09529 1 07381 2 00091 1 07952 3LRN 09332 2 07778 1 00476 2 08939 2J48 06608 5 minus04127 8 06404 6 05680 7NBTree 05986 6 minus03135 7 07557 7 05728 6IB1 04703 10 minus05635 10 10000 10 04273 10IBK 05583 8 minus05476 9 09863 9 04488 9SMO 07944 4 03412 3 02932 3 09274 1RBF 08087 3 02421 4 03082 4 06564 4MLP 05511 9 00080 5 03268 5 05451 8


Table 6 The final ranking with comparative analysis

Algorithm Australian credit dataset German credit datasetBNK 06471 2 02139 6NBS 03806 8 09380 2LRN 09785 1 09754 1J48 04092 7 02101 7NBTree 04841 6 02099 8IB1 02824 10 00000 10IBK 02979 9 00390 9SMO 06017 3 06945 3RBF 05497 4 06173 4MLP 05163 5 04638 5

Conflict of Interests

The authors declare that there is no conflict of interestsregarding the publication of this paper

Acknowledgments

This research has been partially supported by Grants fromthe National Natural Science Foundation of China (no71222108) the Fundamental Research Funds for the CentralUniversities (no JBK140504) the Research Fund for theDoctoral Program of Higher Education (no 20120185110031)and Program forNewCentury Excellent Talents inUniversity(NCET-10-0293)

References

[1] E I Altman and A Saunders ldquoCredit risk measurementdevelopments over the last 20 yearsrdquo Journal of Banking andFinance vol 21 no 11-12 pp 1721ndash1742 1997

[2] M Crouhy D Galai and R Mark ldquoA comparative analysis ofcurrent credit risk modelsrdquo Journal of Banking and Finance vol24 no 1-2 pp 59ndash117 2000

[3] XWu V Kumar J R Quinlan et al ldquoTop 10 algorithms in dataminingrdquo Knowledge and Information Systems vol 14 no 1 pp1ndash37 2008

[4] A Khashman ldquoA neural network model for credit risk evalua-tionrdquo International Journal of Neural Systems vol 19 no 4 pp285ndash294 2009

[5] T Bellotti and J Crook ldquoSupport vector machines for creditscoring and discovery of significant featuresrdquo Expert Systemswith Applications vol 36 no 2 pp 3302ndash3308 2009

[6] F Wen and X Yang ldquoSkewness of return distribution andcoefficient of risk premiumrdquo Journal of Systems Science andComplexity vol 22 no 3 pp 360ndash371 2009

[7] X Zhou W Jiang Y Shi and Y Tian ldquoCredit risk evalua-tion with kernel-based affine subspace nearest points learningmethodrdquo Expert Systems with Applications vol 38 no 4 pp4272ndash4279 2011

[8] G Kim C Wu S Lim and J Kim ldquoModified matrix splittingmethod for the support vector machine and its application tothe credit classification of companies in Koreardquo Expert Systemswith Applications vol 39 no 10 pp 8824ndash8834 2012

[9] F Wen Z He and X Chen ldquoInvestorsrsquo risk preference charac-teristics and conditional skewnessrdquo Mathematical Problems inEngineering vol 2014 Article ID 814965 14 pages 2014

[10] N Hsieh ldquoHybrid mining approach in the design of creditscoring modelsrdquo Expert Systems with Applications vol 28 no4 pp 655ndash665 2005

[11] L Yu S Wang and K K Lai ldquoCredit risk assessment with amultistage neural network ensemble learning approachrdquo ExpertSystems with Applications vol 34 no 2 pp 1434ndash1444 2008

[12] S Oreski DOreski andGOreski ldquoHybrid systemwith geneticalgorithm and artificial neural networks and its application toretail credit risk assessmentrdquo Expert Systems with Applicationsvol 39 no 16 pp 12605ndash12617 2012

[13] D HWolpert andWGMacready ldquoNo free lunch theorems forsearchrdquo Tech Rep SFI-TR-95-02-010 Santa Fe Institute 1995

[14] G J Koehler ldquoNew directions in genetic algorithm theoryrdquoAnnals of Operations Research vol 75 pp 49ndash68 1997

[15] Y Peng G Kou G Wang W Wu and Y Shi ldquoEnsemble ofsoftware defect predictors an AHP-based evaluation methodrdquoInternational Journal of Information Technology and DecisionMaking vol 10 no 1 pp 187ndash206 2011

[16] L Rokach ldquoEnsemble-based classifiersrdquo Artificial IntelligenceReview vol 33 no 1-2 pp 1ndash39 2010

[17] H Kim S Pang H Je D Kim and S Y Bang ldquoConstructingsupport vector machine ensemblerdquo Pattern Recognition vol 36no 12 pp 2757ndash2767 2003

[18] G Kou Y Peng Y Shi MWise andW Xu ldquoDiscovering creditcardholdersrsquo behavior bymultiple criteria linear programmingrdquoAnnals of Operations Research vol 135 no 1 pp 261ndash274 2005

[19] W Chen and J Shih ldquoA study of Taiwanrsquos issuer credit ratingsystems using support vector machinesrdquo Expert Systems withApplications vol 30 no 3 pp 427ndash435 2006

[20] C Tsai and J Wu ldquoUsing neural network ensembles forbankruptcy prediction and credit scoringrdquo Expert Systems withApplications vol 34 no 4 pp 2639ndash2649 2008

[21] G Nie W Rowe L Zhang Y Tian and Y Shi ldquoCredit cardchurn forecasting by logistic regression and decision treerdquoExpert Systems with Applications vol 38 no 12 pp 15273ndash15285 2011

[22] S H Ha and R Krishnan ldquoPredicting repayment of the creditcard debtrdquo Computers and Operations Research vol 39 no 4pp 765ndash773 2012

[23] B Baesens R Setiono C Mues and J Vanthienen ldquoUsingneural network rule extraction and decision tables for credit-risk evaluationrdquoManagement Science vol 49 no 3 pp 312ndash3292003

[24] B Diri and S Albayrak ldquoVisualization and analysis of classifiersperformance in multi-class medical datardquo Expert Systems withApplications vol 34 no 1 pp 628ndash634 2008

[25] C Ferri J Hernandez-Orallo and R Modroiu ldquoAn experi-mental comparison of performancemeasures for classificationrdquoPattern Recognition Letters vol 30 no 1 pp 27ndash38 2009

[26] S Finlay ldquoMultiple classifier architectures and their applicationto credit risk assessmentrdquo European Journal of OperationalResearch vol 210 no 2 pp 368ndash378 2011

[27] S Opricovic and G Tzeng ldquoCompromise solution by MCDMmethods a comparative analysis of VIKOR and TOPSISrdquoEuropean Journal of Operational Research vol 156 no 2 pp445ndash455 2004

[28] G Kou Y Shi and S Wang ldquoMultiple criteria decision makingand decision support systemsmdashguest editorrsquos introductionrdquoDecision Support Systems vol 51 no 2 pp 247ndash249 2011


[29] M J Beynon ldquoA method of aggregation in DSAHP forgroup decision-making with the non-equivalent importance ofindividuals in the grouprdquo Computers and Operations Researchvol 32 no 7 pp 1881ndash1896 2005

[30] Y Deng F T S Chan Y Wu and D Wang ldquoA new linguis-tic MCDM method based on multiple-criterion data fusionrdquoExpert Systems with Applications vol 38 no 6 pp 6985ndash69932011

[31] G Nakhaeizadeh and A Schnabl ldquoDevelopment of multi-criteria metrics for evaluation of data mining algorithmsrdquo inProceedings of the 3rd International Conference on KnowledgeDiscovery and Data Mining (KDD rsquo97) pp 37ndash42 1997

[32] K A Smith-Miles ldquoCross-disciplinary perspectives on meta-learning for algorithm selectionrdquo ACMComputing Surveys vol4 no 1 pp 6ndash25 2008

[33] Y Peng G Wang G Kou and Y Shi ldquoAn empirical study ofclassification algorithm evaluation for financial risk predictionrdquoApplied Soft Computing Journal vol 11 no 2 pp 2906ndash29152011

[34] M Hall E Frank G Holmes B Pfahringer P Reutemann andI H Witten ldquoThe WEKA data mining software an updaterdquoSIGKDD Explorations vol 11 no 1 pp 10ndash18 2009

[35] G Kou Y Lu Y Peng and Y Shi ldquoEvaluation of classificationalgorithms using MCDM and rank correlationrdquo InternationalJournal of Information Technology and Decision Making vol 11no 1 pp 197ndash225 2012

[36] I H Witten and E Frank Data Mining Practical MachineLearning Tools and Techniques Morgan Kaufmann San Fran-cisco Calif USA 2nd edition 2005

[37] I M Premachandra G S Bhabra and T Sueyoshi ldquoDEA asa tool for bankruptcy assessment a comparative study withlogistic regression techniquerdquo European Journal of OperationalResearch vol 193 no 2 pp 412ndash424 2009

[38] S Weiss and C Kulikowski Computer Systems That LearnClassification and Predication Methods from Statistics NeuralNetsMachine Learning and Expert SystemsMorganKaufmann1991

[39] P Domingos and M Pazzani ldquoOn the optimality of the simpleBayesian classifier under zero-one lossrdquoMachine Learning vol29 no 2-3 pp 103ndash130 1997

[40] S Cessie and J C Houwelingen ldquoRidge estimators in logisticregressionrdquo Applied Statistics vol 41 no 1 pp 191ndash201 1992

[41] R J Quinlan C4 5 Programs for Machine Learning MorganKaufmann Series in Machine Learning Morgan Kaufmann1993

[42] R Kohavi ldquoScaling up the accuracy of Naıve-Bayes classifiera decision-tree hybridrdquo in Proceedings of the 2nd InternationalConference on Knowledge Discovery and Data Mining (KDDrsquo96) pp 202ndash207 AAAI Press 1996

[43] D W Aha A study of instance-based algorithms for supervisedlearning tasks mathematical empirical and psychological eval-uations [PhD dissertation] Department of Information andComputer Science University of California Irvine Calif USA1990

[44] DWAhaDKibler andMKAlbert ldquoInstance-based learningalgorithmsrdquoMachine Learning vol 6 no 1 pp 37ndash66 1991

[45] D Kibler D W Aha and M K Albert ldquoInstance-basedprediction of real-valued attributesrdquoComputational Intelligencevol 5 no 2 pp 51ndash57 1989

[46] J C PlattAdvances inKernelMethods Support VectorMachinesMIT Press Cambridge Mass USA 1998

[47] C M Bishop Neural Networks for Pattern Recognition OxfordUniversity Press 1995

[48] J Park and I W Sandberg ldquoUniversal approximation usingradial basis functions networksrdquoNeural Computation vol 3 no2 pp 246ndash257 1991

[49] C L Hwang and K Yoon Multiple Attribute Decision MakingMethods and Applications Springer Berlin Germany 1981

[50] J Brans and P Vincke ldquoNotemdasha preference ranking organiza-tion method (the PROMETHEE method for multiple criteriadecision-making)rdquoManagement Science vol 31 no 6 pp 647ndash656 1985

[51] S Opricovic Multi-Criteria Optimization of Civil EngineeringSystems Faculty of Civil Engineering Belgrade Serbia 1998

[52] J Deng ldquoControl problems of grey systemsrdquo Systems andControl Letters vol 5 no 2 pp 288ndash294 1982

[53] P Domingos ldquoToward knowledge-rich data miningrdquo DataMining and Knowledge Discovery vol 15 no 1 pp 21ndash28 2007

Submit your manuscripts athttpwwwhindawicom

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

MathematicsJournal of


Mathematical Problems in Engineering

Hindawi Publishing Corporationhttpwwwhindawicom

Differential EquationsInternational Journal of

Volume 2014

Applied MathematicsJournal of


Probability and StatisticsHindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Journal of


Mathematical PhysicsAdvances in

Complex AnalysisJournal of


OptimizationJournal of


CombinatoricsHindawi Publishing Corporationhttpwwwhindawicom Volume 2014

International Journal of


Operations ResearchAdvances in

Journal of


Function Spaces

Abstract and Applied AnalysisHindawi Publishing Corporationhttpwwwhindawicom Volume 2014

International Journal of Mathematics and Mathematical Sciences


The Scientific World JournalHindawi Publishing Corporation httpwwwhindawicom Volume 2014


Algebra

Discrete Dynamics in Nature and Society



Decision SciencesAdvances in

Discrete MathematicsJournal of


Volume 2014 Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Stochastic AnalysisInternational Journal of


programming) nonparametric statistical analysis (eg recur-sive partitioning goal programming and decision trees)and artificial intelligence modeling (eg support vectormachines neural networks and genetic algorithms)

The advantages of applying classification algorithms forcredit risk analysis include the following It is difficult fortraditionalmethods to handle large size databases while clas-sification algorithms especially artificial intelligence model-ing can be used to quickly predict credit risk even whenthe size of dataset is huge Second classification algorithmsmay provide higher prediction accuracy than traditionalapproaches [23] Third the decision making based on theresults of classification algorithms is objective reducing theinfluence of human biases

However the no free lunch theorem states that noalgorithm can outperform all other algorithms when perfor-mance is amortized over all measures Many studies indicatethat classifiersrsquo performances vary under different datasetsand circumstances [24ndash26] How to provide a comprehensiveassessment of algorithms is an important area Algorithmevaluation and selection normally need to examine multi-criteria Therefore classification algorithm evaluation andselection can be treated as an MCDM problem and MCDMmethods can be applied to systematically choose the appro-priate algorithms [16]

As defined by the International Society on MultipleCriteria Decision Making MCDM is the study of methodsand procedures by which concerns aboutmultiple conflictingcriteria can be formally incorporated into the managementplanning process [27 28] MCDM is concerned with theelucidation of the levels of preference of decision alternativesthrough judgments made over a number of criteria [29 30]MCDMmethods have been developed and applied in evalu-ation and selection of classification algorithms For instanceNakhaeizadeh and Schnabl [31] suggested a multicriteria-based measure to compare classification algorithms Smith-Miles [32] considered the algorithm evaluation and selectionproblem as a learning task and discussed the generalizationof metalearning concepts Peng et al [33] applied MCDMmethods to rank classification algorithms However theseresearch efforts face challenging situations that differentMCDM methods produce conflicting rankings This paperproposes and develops AHM a unified framework based onMCDM and DM to identify robust classification algorithmsespecially when different evaluation algorithms generateconflicting results

3 Preliminaries

31 Performance Measures This paper utilizes the followingten commonly used performance measures [33 35]

(i) Overall accuracy (Acc) accuracy is the percentageof correctly classified instances It is one of the mostwidely used classification performance metrics

Overall accuracy = TN + TPTP + FP + FN + TN

(1)

where TN TP FN and FP stand for true negative truepositive false negative and false positive respectively

(ii) True positive rate (TPR) TPR is the number ofcorrectly classified positive instances or abnormalinstances TPR is also called sensitivity measure

True positive rate = TPTP + FN

(2)

(iii) True negative rate (TNR) TNR is the numberof correctly classified negative instances or normalinstances TNR is also called specificity measure

True negative rate = TNTN + FP

(3)

(iv) Precision this is the number of classified fault-pronemodules that actually are fault-prone modules

Precision = TPTP + FP

(4)

(v) The area under receiver operating characteristic(AUC) receiver operating characteristic stands forreceiver operating characteristic which shows thetradeoff between TP rate and FP rate AUC representsthe accuracy of a classifier The larger the area thebetter the classifier

(vi) 119865-measure it is the harmonic mean of precision andrecall 119865-measure has been widely used in informa-tion retrieval

119865-measure = 2 times Precision times RecallPrecision + Recall

(5)

(vii) Mean absolute error (MAE) thismeasures howmuchthe predictions deviate from the true probability119875(119894 119895) is the estimated probability of 119894 module to beof class 119895 taking values in [0 1]

MAE =sum

119888

119895=1sum

119898

119894=1

1003816100381610038161003816

119891 (119894 119895) minus 119875 (119894 119895)

1003816100381610038161003816

119898 sdot 119888

(6)

(viii) Kappa statistic (Kaps) this is a classifier performancemeasure that estimates the similarity between themembers of an ensemble in multiclassifiers systems

Kaps = 119875 (119860) minus 119875 (119864)1 minus 119875 (119864)

(7)

where 119875(119860) is the accuracy of the classifier and 119875(119864)is the probability that agreement among classifiers isdue to chance

(ix) Training time is the time needed to train a classifica-tion algorithm or ensemble method

(x) Test time is the time needed to test a classificationalgorithm or ensemble method

Algorithm evaluation and selection involves benefit andcost criteria Seven performance measures used in this studyare benefit criteriaThey are accuracy kappa statistic TP rateTN rate precision 119865-measure and AUC The other threeperformance measures (ie MAE training time and testtime) are cost criteria



DM stage

MCDMstage

Acc Kaps MAE AUC

Target data



Result revelation
















5 Experiment











END

Algorithm 1









6 Conclusion


















Acknowledgments


References






























































Volume 2014




Journal of











Journal of


Function Spaces






Algebra











DM stage

MCDMstage

Acc Kaps MAE AUC

Target data



Result revelation
















5 Experiment











END

Algorithm 1









6 Conclusion


















Acknowledgments


References






























































Volume 2014




Journal of











Journal of


Function Spaces






Algebra


















END

Algorithm 1









6 Conclusion


















Acknowledgments


References






























































Volume 2014




Journal of











Journal of


Function Spaces






Algebra

























Acknowledgments


References






























































Volume 2014




Journal of











Journal of


Function Spaces






Algebra










































Volume 2014




Journal of











Journal of


Function Spaces






Algebra









research article an analytic hierarchy model for...

Documents