URREF Self-Confidence in Information Fusion Trust Erik Blasch Audun Jsang Jean DezertPaulo C. G. CostaAnne-Laure JousselmeAbstractTheUncertaintyRepresentationandReasoning EvaluationFramework(URREF)includesanontologythat representsconceptsandcriterianeededtoevaluatetheuncertainty management aspects of a fusion system. The URREF ontology defines self-confidenceasameasureoftheinformationcredibilityas evaluated by the sensor itself. The concept of confidence, which is not explicitlydefinedintheontologyatURREF,hasbeenextensively exploredintheliteratureaboutevaluationininformationfusion systems(IFS).Inthispaper,weprovideadiscussiononconfidence as it relates to the evaluation of IFS, and compare it with the existing conceptsintheURREFontology.Ourgoalistwo-fold,sincewe address both the distinctions between confidence and self-confidence, aswellastheimplicationsofthesedifferenceswhenevaluatingthe impact of uncertainty to the decision-making processes supported byt theIFS.Weillustratethediscussionwithanexampleofdecision making that involves signal detection theory, confusion matrix fusion, subjectivelogic,andproportionalconflictredistribution.Weargue thatuncertaintycanbeminimizedthroughconfidence(information evidence)andself-confidence(sourceagent)processing,Theresults here seek to enrich the ongoing discussion at the ISIFs Evaluation of TechniquesforUncertaintyRepresentationWorkingGroup (ETURWG)onself-confidenceandtrustininformationfusion systems design. Keywords: Self-Confidence, Confidence, Trust, Level 5 Fusion, High-Level Information Fusion , PCR5/6, Subjective LogicI.INTRODUCTIONInformationfusionaimstoachieveuncertaintyreduction throughcombininginformation frommultiplecomplementary sources.TheInternationalSocietyofInformationFusion (ISIF)EvaluationofTechniquesofUncertaintyReasoning WorkingGroup(ETURWG)wascharteredtoaddressthe problemofevaluatingfusionsystemsapproachesto representingandreasoningwithuncertainty.Theworking groupdevelopedtheUncertaintyRepresentationand ReasoningEvaluationFramework(URREF)[1].Discussions during2013exploredthenotionsofcredibilityandreliability [2]. One recent issue is the difference between confidence and self-confidenceasrelatedtothedata,source,andprocessing. WhileagreementisnotcompleteamongtheETURWG,this paperseekstoprovideonepossibleapproachtorelatethe mathematical,semantic,andtheoreticalchallengesof confidence analysis. Thekeypositionofthepaperistoanalyzethepractical differencesinevaluatingthetwoconcepts.Morespecifically, self-confidenceismostlyrelevanttoHUMINT,whichmakes itsevaluationaprimarilysubjective;whereasconfidencecan be easily traced to machine data analysis, allowing for the use ofobjectivemetricsinitsevaluation.Thatis,amachinecan processlargeamountsofdatatorepresentthestateofthe world,andtheevaluation of howwell uncertaintyiscaptured intheseprocessescanbetracedtovariousobjectivemetrics. Incontrast,forahumantoassessitsownconfidenceonthe credibilityofhisdatacollectionprocess(i.e.self-confidence),heorshehastomakeajudgmentonlimited choices.Objectiveassessmentisdeterminedfromthe credibilityofthereports,processing,anddecisions.Typical approachesincludeartificialintelligence(AI)methods(e.g., NeuralNetworks),patternrecognition(e.g.,Bayesian, wavelets),andautomatictargetexploitation(i.e.,oversensor, target,andenvironmentoperatingconditions[3]).Subjective analysisisareportqualityopinionthatfactorsinanalysis (e.g.,completeness;accuracy,andveracity),knowledge(e.g., representation,uncertainty,andreasoning),andjudgment (e.g.,intuition,experience,decisionmaking)[4].Intermsof IFSsupportfordecision-making,numerousmethodshave beenexplored,mainlyfromBayesianreasoning,Dempster-ShaferTheory[5],SubjectiveLogic[6],DSmT[7],fuzzy logic and possibility theory; although it also includes research onapproximatingbelieffunctionstosubjectiveprobability measures (BetP [8], DSmP [9]). Figure1providesaframeworkforourdiscussion.Theworld contains some truth T, of which data is provided from different sources (A, B). Source A analysis goes to a machine agent for information fusion processing while source B goes to a human agent.Eitherthemachineorthehumancangeneratebeliefs aboutthestateoftheworld(eitherusingqualitativeor quantitativesemantics).ThecombinationofAandBisa subject of Level 5 Fusion (user refinement) [10, 11]. Figure 1 Information Fusion System assessment of confidence (machine) and self-confidence (sensor or human agent). Originally published as Blasch E., Josang A., Dezert J., Costa P., Jousselme A.-L., URREF Self-Confidence in Information Fusion Trust, in Proc. of Fusion 2014 Int Conf onInformation Fusion, Salamanca, Spain, July 7-10, 2014, and reprinted with permission.Advances and Applications of DSmT for Information Fusion. Collected Works. Volume 4215Ononehand,confidenceistypicallyrelatedtomachine processingsuchassignaldetectiontheorywhereself-confidenceisassociatedwithsensors(andhumans)assessing theirowncapability.Ontheotherhand,themanipulationof thedatarequiresunderstandingofthesource,andself-confidenceisapplicableforthecasesinwhichtheusercan provide a self-assessment on how confident he is on its data. It isimportanttoemphasizethatarenotdealing(atleastnot directly)withinformationveracity:evenifthesensor(e.g.,a humanreportingonaneventorprovidingassessmentona situation)considerstheinformationaspossible,andhetrusts it,itcouldbefalseattheend(e.g.,eveninsummertimewe can have a cloudy day). That is, self-confidence assesses how much the author trust the information, but not necessarily that this information is false or true [1]. For this paper, we take the URREFontologydefinitionofself-confidenceasimpliedin Figure1.Therationaleforthischoiceisthatself-confidence anduncertaintyaretypicallyassociatedwith humanswhereas confidence has been typically used in signal detection. Fusion ofbeliefsultimatelyrelatestostatesoftheworldwitha reportedconfidencethatcanbecomparedtoatruthstate. Debatingontheoverlapsinterminologywouldbewelcomed toclarifythesepositionsfortheETURWGandthe information community as a whole.FromFigure1,wenotetheimportanceofconfidenceas related from a decision to the estimated states. Self-confidence iswithinthehumanagentassessingtheirunderstanding(e.g. experience)thatcanalsobecombinedwiththecomputer agent.Theissueathandforauseriswhetherornotthe machineanalysis(ortheirown)statedecisionrepresents reality.Thenotionofrealitycomesfromthefactthat currently,therearemanytechnicalproductsthatperceivethe world for the user (e.g., video) from which the user must map amentalmodeltoaphysicalmodelofwhatisbeing represented. Some cases are easy such as video of cars moving onaroad[12];however,othersarecomplexsuchascyber networks [13]. The example used through the rest of the paper requiresHigh-LevelInformationFusion(HLIF)oftarget detection from a machine and human [14, 15]. Indesigningcomputer-aideddetectionmachines,itis desirable to provide intelligence amplification (IA) [16] where Qualia motivates subjective analysis as a relation between the humanconsciousness/self-awarenesstoexternalstimuli. Qualia is the internal perception of the subjective aspect of the humansperceptionofthestimuli.Knowingoneselfcanthen beutilizedtounderstand/evaluatetheuseofmeaningfuland relevantdataindecision-making.Themorethatasensor understands its Qualia [17], the better it will be in providing an assessmentofitsself-confidenceinareportoronadecision. Qualiathenencompassesanimportantcomponentto uncertaintyreasoningassociatedwithsubjectivebeliefs,trust, and self-confidence in decision making as a sense of intuition. Not surprisingly, these are natural discussion topics in Level 5 fusion(userrefinement),whichincludesoperator-machine collaboration[18],situationawareness/assessmentdisplays [19],andtrust[20].Inordertoexploreself-confidenceon theseissues,weneedtolookatthepsychologyliteratureon trust as it relates to self-confidence. Fromdataavailableontheweb(e.g.,twitter,documents), intelligentusersneedthecapabilitytorapidlymonitorand analyzeeventinformationovermassiveamountsof unstructuredtextualdata[21].Textfromhumansourcesis subjectedtoopinions,beliefs,andmisperceptions,generating variousformsofself-assumedself-confidence.Incontrast, computer sensed data can be stochastic or deterministic,from whichwehavetocoordinatetheagentinformation.For example,withGaussianobservationsgeneratesstochastic probability analyses (e.g., Kalman Filter). However, structural informationinthesensormodelsandsensitivitiesforagiven statecondition(whichcomefromadeterministicontology) could be used to improve the estimate [22]. This combination of both stochastic and deterministic decisions with uncertainty elementsisusualinmodelingandsystemdeployment,and understandingitskeyaspectsisafertileareaforproducing better decision support from IFS. A related example from the analysis of uncertainty is evidence assessment from opinion makers. Dempster-Shafer theory has beenusedinconnectionwithBayesiananalysisfor decisionmaking[5]. Likewise,Jsang[23]demonstratedhowsubjective analysiswithinDempster-Shafertheorycouldbeusedto determinetheweightofopinions.Ontologiessuchasthe oneusedintheURREFmustbeabletoaccountforthe uncertainty of data and to model it qualitatively, semantically, andquantitatively[24].Metricssuchasqualityofservice (QoS)andqualityorinformation(IQ)areexampleoftools thatcansupportandenhanceamodelingcapabilitybetween ontologiesanduncertaintyanalysis[25].Therestofthis paperincludesSect.IIasanoverviewofself-confidence. Sect.IIIdiscussesthemathematicalanalysis.Sect.IV highlightssubjectivelogicforopinionmaking.Sect.Visan example and Sect. VI provides conclusions. II. URREF NOTIONS OF SELF-CONFIDENCETheETURWGhasexploredmanytopicsasrelatedtoa systemsanalysisofinformationfusion,whichincludes characteristics of uncertainty with many unknowns [26, 27]. In this paper, we categorize the characteristics of uncertainty into fourareas,showninTable1.Assumingthattheflowof informationfirstgoesfromanagenttoevidencebeliefs,and subsequentlytofusionwithknowledgerepresentation,then theseareashelpunderstandtheterminology.Notethatthe definedinformationfusionqualityofservice(QoS) parametersareinblue{timeliness,accuracy,throughput,and confidence}.Thesecouldalsobemeasuresofperformance [28].Formeasuresofeffectiveness[25],oneneedsto understand system robustness (e.g., consistency, completeness, correctness,integrity).Herewefocusontheredtermsas related to self-confidence and confidence.KnowledgerepresentationinIFS[29,30]includesapplying decision-makingsemanticstosupportthestructuringof extracted information. One example is the use of well defined concepts(e.g.confirmed,probable,possible,doubtful,and improbable)tosupportinformationextractionwithnatural languageprocessing(NLP)algorithms.Asrelatedto confidence and self-confidence, there is the notion of integrity. Advances and Applications of DSmT for Information Fusion. Collected Works. Volume 4216Integrityforhumanagentsisassociatedwiththeirsubjective accountability and consistency in making judgments. Integrity for a machine could be objective in the faithful representation and validity on the data [31]. Algorithmperformancefocusontheinformationfusion method.URREFcriteriaforevaluatingitrelatestohowthe uncertaintymodelperformsoperationswithinformation.An exampleofrelatedmetricsistoassessuncertaintyreduction by weighting good data over bad given conflicting data.Evidence:From[2],weexploredtheweightofevidence (WOE)asafunctionofreliability,credibility,relevance,and completeness.InURREF,WOEassesseshowwellan uncertainty representation technique captures the impact of an input affecting the processing and output of the IFS. Source: Self-confidence, while yet to have a clear definition in the engineering literature, is typically associated with trust. A.Trust Closely associated with subjective analysis is trust[32]. Trust includesmanyattributesforman-machinesystemssuchas dependability(machine),competence(user),andapplication [33].Trustisthenrelatedtomachineprocessing(confidence) andhumanassessment(self-confidence).Trustinautomation isakeyattributeassociatedwithmachine-drivensolutions. Humantrustinautomationdeterminesausersrelianceon automation.In[32],theyexploredself-confidencedefinedas theuseranticipatory(orpost)performancewithmachines which impacts with trust in policy application.Measuring trust as related to uncertainty is an open topic [34]. Asafocusofdiscussion,wehaveamachineagentanda humanagentofwhichameasureoftrustcomesfromthe uncertaintyassociatedbetweentheman-machineinteractions. Reliabilitytrustcouldbebetweenhumanagentsofwhich subjectiveprobabilityisuseful[35].Decisiontrustcouldbe between human agents or between a human and a machine and takes into account the risk associated with situation-dependent attitudes,attention,andworkloadofahumanagent.The distinctionbetweenreliabilityanddecisiontrustisimportant as related to self-confidence and confidence. This can be seen in Table 2, which depicts the main aspects for each of the six potential interactions between sensors.. Table 2: Trust Aspects in Sensor InteractionsHumanOthers MachineHuman Self-confidenceReliability TrustworthyMachine Trust Credibility Confidence Human:Individualsmustprovideintrospectionontheirownanalysisandinteractionwithamachine.Herewe distinguishbetweenself-confidenceandtrust.Inthis case,humanagentsmusthaveself-confidencein themselves as well as trust in the machine. Others: With the explosion of the Internet, recent work hasexploredtheuncertaintyofhumansensing,suchas Twitterreportsinsocialnetworks,showinghumansas lesscalibratedandreliableintheirsensing.Wangetal. [36,37]developedanestimationapproachfortruth discoveryinthisdomain.Anotherrecentexample exploredthedecision-makingtrustbetweenhumans interfacingthroughamachine.Theuserinterfacewas shown to have a strong impact on trust, cooperation, and situationawareness[38].Asaninterestingresult, credibilityresultedasthecomputerinteractionafforded completeandincompleteinformationtowards understanding both the machine and the user analysis. Machine:A large body of literature is devoted to networktrust.Examplesincludethehardware,cybernetworks [39],protocolsandpolicies.Giventhelargeamountof cyberattackswrittenbyhackers,itcomesdowntoa trustworthynetworkofconfidentiality,integrity,and availability.Formachine-machineprocessingwithout user-createdmalware,networkengineeringanalysisis mostlyoneofconfidence.Machinetrustisalso important to enterprise systems [40]. Sinceweseektounderstandself-confidenceasaURREF criterion,explorationsincludedhumanprocessingandthe human as a data source as shown in Figure 3.Figure 2 Methods of Trust. Table 1: Characteristics of Uncertainty 3Agent Evidence Algorithm RepresentationSourceInformation Fusion Knowledge ReasoningScalability Knowledge Handling Objectivity Relevance Computational Cost Simplicity Observational SensitivityConclusiveness Adaptability Expressiveness Veracity (truthfulness)Veracity (truth) Traceability (pedigree)PolaritySecure AmbiguityStabilityModalityResilientGenericity Trust Precision ThroughputTenseAccuracy TimelinessReliability CredibilityCorrectness Completeness Self-Confidence Confidence Consistency Integrity 3 This table is presented to the ETURWG in this paper to support ongoing discussions on the categorization of types of uncertaintyAdvances and Applications of DSmT for Information Fusion. Collected Works. Volume 4217In the figure, multiple forms of trust are shown and related to theprocessingsteps.Startingfromtherealworld,datais placedonthenetworkfromwhichamachine(orsensor) processesthedata.Withcontext,adetectionassessmentis madeforsuchdomainsasimage,text,orcyberprocessing. Thatdetectionisthefusedwiththecontextinformation.For example,adetectionofanobjectinanimageisfusedwith contextualroadinformation.Thedetectionconfidenceis assessed and made available to the user with the context. The green line is the human-machine trust boundary as the human can look at the machine results or process the data themselves forgivenalevelofconfidenceandrenderadecision.The dottedlinethenisahumanassessmentofwhetherornotthe information presented represents reality and could be trusted. Note,ifthehumanistheonlysensorsource,thenhe/sheis lookingatdataandmakingadecision.Theirself-confidence could be based on the machine results from which they factor in many types of trust. For example, context, as related to the real world (see Figure 1), provides a validation of the machine (network to algorithm trust as a measure of confidence), while at the same time understanding the situation to determine if the information fusion analysis is providing meaningful and useful informationtowardstheapplicationofinterest.Togethera trusted decision is rendered based on the many factors. Included in Figure 3 are many forms of trust in the analysis all of which can lead to confidence in the decision: TrustProcessingExample NetworkDataputona network Assessment of data timeliness and lost packets MachineSensor transformation Calibrationofcamerasforimage content SoftwareInformation management Gettingthecorrectdatafroma data base (e.g., a priori data)AlgorithmFusion methodTargettrackingandclassification results ModelingState modelsKinematicandtargetrecognition models (e.g., training data)ApplicationSituation of interestAnalysisoverthecorrectarea (e.g. target moving on a road)UserSituation awarenessUseofculturalandbehavior(e.g. assume big cars move on roads).The self-confidence of the user analysis includes working with data,networks,andmachines.TheURREFontologymust accountfortrustoverhuman-machinedecisionsfor confidenceanalysis.TofurtherexplorehowURREFis aligned with trust, we must look at self-confidence. B.Self-Confidence Statisticallyspeaking,themachinedecision-makingaccuracy isbasedonthedataavailable,themodelchosen,andthe estimationuncertaintyassociatedwiththemeasureddata. Giventheaboveanalysis,wecouldstarttoderiveself-confidenceformachinefusionoperationsbasedonthe literature in human self-confidence.Self-confidenceisthesocio-psychologicalconceptrelatedto self-assuredness in one's personal judgment and ability. As an example,researchersareoftencalledtoreviewpapersand aftertheirreviewaskedtogiveaqualityratingoftheirown reviewbasedontheirunderstandingofthesubject,expertise, andexperience.Inanotherexample,apersonmightbeasked toidentifyanobjectinanimagewithacertainrating {unlikely,possible,probable,confirm}fromwhichthenthey coulddetermineself-confidencebasedontheiranswer.Thus, thereisaneedtoassessself-confidenceinrelationto confidence, which is linked to uncertainty measures of trust. C.Accuracy and Precision Selfconfidenceisstronglyrelatedtobothprecisionand accuracy. A source can be self confident in both the precision of its generated data (consistency or variability in its reports - such as reported variance) as well as the accuracy of its reports (the reported bias or the reported distance of the mean value of thegenerateddataformtruevalue).Inotherwords,tomake senseofthetermtheself-confidenceofasource,thedata encapsulateacombinationofprecisionandaccuracy.A distinctionismadebetweenprecisionandaccuracyreported bythemachine(suchastheestimatedmeanandvarianceat theoutputoftheKalmanfilter)andtheactualprecisionand accuracyofthedataemanatingfromthesource.TheURREF ontologycategorizesaccuracy,precision,andself-confidence as types of criteria to evaluate data [1]. Statisticalmethodsofuncertaintyanalysisfrommeasurement systemsincludeaccuracyandprecision,showninFigure4. Theuseofdistancemetrics(accuracy)andprecisionmetrics (standarddeviations)helptoanalyzewhetherthe measurementiscalibratedandrepeatable.Wewoulddesire thesameanalysisforhumansemanticanalysiswithprecise meanings, consistent understanding, and accurate terminology. Figure 3 Uncertainty as a function of accuracy and precision. HumanConfidence-Accuracy:Traditionallyknownasthe confidence-accuracy(CA)relationship,theassumptionisthat asonesconfidenceincreasessodoestheirlevelofaccuracy whichisaffectedbymemory,consistency,andability[41]. Issuesincludeabsoluteversusrelativeassessment,feedback, and performance.Theconfidence-accuracyrelationshipwasshowntobeaby-productoftheconsistency-correctnessrelationship:Itis positivebecausetheanswersthatareconsistentlychosenare generallycorrect,butnegativewhenthewronganswerstend tobefavored.Theoverconfidencebiasstemsfromthe reliability-validitydiscrepancy:Confidencemonitors reliability (or self-consistency), but its accuracy is evaluated in calibrationstudiesagainstcorrectness.Also,theresponse Advances and Applications of DSmT for Information Fusion. Collected Works. Volume 4218speedisafrugalcueforself-consistencyanddependsonthe validity of self-consistency in predicting performance [42]. Koriat[42]explainsthatSensingtasksaredominatedby Thurstonianuncertainty(localrankorderingwithstochastic noise)withinanindividualandexhibitanunder-confidence bias.However,generalknowledgetasksaredominatedby BrunswikianUncertainty(globalprobabilisticmodelfrom limitedsamplesetstoinfergeneralknowledge[43])that supports inter-person ecological relations. Consistency is then the repeatability of the information, which should imply no conflicts in decision-making.We can use the proportional conflict redistribution (PCR6) to get a measure of repeatedconsistencysuchthatfavoredwronganswersare correctedinconfidenceanalysis[44].PCR6ismoregeneral andefficientthanPCR5whencombiningmorethantwo sourcesaltogether.Moreover,PCR6hasbeenproved compatiblewithfrequencyprobabilitieswhenworkingwith binary BBA's, whereas PCR5 and DS are not compatible with frequency probabilities [44]. Self-confidence could be measured with a Receiver Operating Characteristic(ROC)curveasonceadecisioncanbemade, wecanthenassessitsimpactonconfidence.Alowself-confidencewouldleadtochance,andahighself-confidence would remain to the left on the ROC. III. SELF-CONFIDENCESignaldetectiontheoryprovidesameasureofconfidencein decisionmakingthatbyassumingalimitedhypothesissetis actuallyameasureofself-confidence.Oneclassicexampleis WaldsSequentialProbabilityRatioTest(SPRT)[45]. Assumingevidenceissampledat discretetimeintervals,then thehumanorcognitiveagentcomparestheconditional probabilities x(t + t) for two hypothesis H j (j = 1, 2). Using then SPRT, theny(t) = h[x(t)]=LN f 1 [x(t)] f 2 [x(t)](1) If y(t) > 0, then evidence supports H 1, and if y(t) < 0, then H 2 is more likely. As time accumulates for decision making, there is an aggregation of the log likelihood ratios: L(t + t)=L(t)+LN f 1 [x(t + t)] f 2 [x(t + t)](2) where, for a stochastic system L(t) ~ N((t), 2(t)). Eq (2) can be written in Bayesian log odds:LN p(H 1 | D)p(H 2 | D) = t LN f 1 [x(t)]f 2 [x(t)] + LN p(H 1)p(H 2) (3)One then collects information to make a decision such that 2