kbs development life cycle validation uncertainty kbs development life cycle validation uncertainty
TRANSCRIPT
KBS development life cyKBS development life cycle Validation Uncertaintcle Validation Uncertainty y
Dr. R. Weber INFO612Dr. R. Weber INFO612
KBS development life cyKBS development life cycle Validation Uncertaintcle Validation Uncertainty y
Cop
yrig
ht R
osin
a W
eber
ES Development Life CycleES Development Life Cycle
PlanningPlanning Knowledge DefinitionKnowledge Definition Knowledge DesignKnowledge Design Code and CheckoutCode and Checkout Knowledge VerificationKnowledge Verification System EvaluationSystem Evaluation
Cop
yrig
ht R
osin
a W
eber
PlanningPlanning
feasibility assessmentfeasibility assessment resource managementresource management task phasingtask phasing schedulesschedules preliminary functional layout preliminary functional layout
(what)(what) high-level requirements (how)high-level requirements (how)
Cop
yrig
ht R
osin
a W
eber
Knowledge DefinitionKnowledge Definition knowledge source identification and selection
source identification, importance, availability, source identification, importance, availability, selectionselection
knowledge acquisition, analysis, & extraction acquisition strategyacquisition strategy knowledge identification and classificationknowledge identification and classification functional layoutfunctional layout control flowcontrol flow user’s manualuser’s manual requirements specificationrequirements specification knowledge baselineknowledge baseline
Cop
yrig
ht R
osin
a W
eber
Knowledge DesignKnowledge Design knowledge representation
choose the most appropriate knowledge choose the most appropriate knowledge representation formalism and then choose the toolrepresentation formalism and then choose the tool
control structure internal fact structure preliminary user interface initial test plan design structure implementation strategy detailed user interface design specifications and report detailed test plan
Cop
yrig
ht R
osin
a W
eber
Code and CheckoutCode and Checkout
coding testing source listings user’s manual installation/operations guide system description document
Cop
yrig
ht R
osin
a W
eber
Knowledge VerificationKnowledge Verification
formal tests test analysis
incorrect incomplete inconsistent
recommendations
Cop
yrig
ht R
osin
a W
eber
System Evaluation-System Evaluation-ValidationValidation
demonstrate the system serves its purposes efficiently and effectivelycomparing to other systemscomparing to alternate methodscomparing to humans
maintenance
Cop
yrig
ht R
osin
a W
eber
PlanningPlanning Knowledge acquisitionKnowledge acquisition Knowledge engineering: design and Knowledge engineering: design and
implementationimplementation Situation assessmentSituation assessment RetrieveRetrieve ReviseRevise ReviewReview RetainRetain
System EvaluationSystem Evaluation MaintainMaintain
CBR Development Life CycleCBR Development Life Cycle
Cop
yrig
ht R
osin
a W
eberCBR design and CBR design and implementationimplementation Situation assessmentSituation assessment
RetrieveRetrieve ReviseRevise ReviewReview RetainRetain (Validation)(Validation) Maintenance designMaintenance design
Cop
yrig
ht R
osin
a W
eber
casebase
Design decisions in CBR systems (i)
Which are the cases?What is the task?
How will the case base be organized?
How will the cases be represented?Which will be the indexing vocabulary?
What is the task?How will the case base be organized?
Cop
yrig
ht R
osin
a W
eber
How will new cases be input?How to perform retrieval?
Identify featuresInitially match (similarity assessment)
SearchSelect
Retrievalinputproblem initial
solutions
Design decisions in CBR systems (ii)
Cop
yrig
ht R
osin
a W
eber
How to implement reuse?From Select or with a combination?
How to display the proposed solution?
solution
Reuse
proposed
initialsolutions
Design decisions in CBR systems (iii)
Cop
yrig
ht R
osin
a W
eber
Is the proposed solution good?How to determine and find what to adapt?
Where is adaptation knowledge?
solutionReviseproposedconfirmed
solution
case repair
case adaptation
Design decisions in CBR systems (iv)
Cop
yrig
ht R
osin
a W
eber
Is it the type of task that it is worth learning?Index new case before retain.
Retain.
Retain
confirmedsolution
casebase
Design decisions in CBR systems (ii)
Cop
yrig
ht R
osin
a W
eber
Validation refers to establishing the Validation refers to establishing the effectiveness of a system in light of its effectiveness of a system in light of its intended purposesintended purposes
Verification indicates how correct a given Verification indicates how correct a given system can solve its proposed tasks system can solve its proposed tasks (Watson)(Watson)
Retrieval accuracy is indicated by the Retrieval accuracy is indicated by the result given by the system when the result given by the system when the target case is part of the case collection.target case is part of the case collection.
validation & verification (i)validation & verification (i)
Cop
yrig
ht R
osin
a W
eber
validation & verification (ii)validation & verification (ii) Retrieval consistency: the same retrieval Retrieval consistency: the same retrieval
when executed the second time must when executed the second time must retrieve exactly the same cases (e.g., with retrieve exactly the same cases (e.g., with the same similarity if k-NN is used)the same similarity if k-NN is used)
Case Duplication: when two distinct cases Case Duplication: when two distinct cases receive the same value for similarity in receive the same value for similarity in relation to a given target case. relation to a given target case.
When the same value is attributed to When the same value is attributed to different cases the user or the system has to different cases the user or the system has to decide which one to use by evaluating the decide which one to use by evaluating the value for each attribute. The same measure value for each attribute. The same measure of similarity does not mean the cases of similarity does not mean the cases necessarily teach the same lessons.necessarily teach the same lessons.
Cop
yrig
ht R
osin
a W
eber
Case Coverage is checked for the evenly Case Coverage is checked for the evenly distribution of cases when they are manipulated distribution of cases when they are manipulated and not actual experiences that are collected as and not actual experiences that are collected as they happen.they happen.
Efficiency verification : comparison to alternative Efficiency verification : comparison to alternative methods, empirical evaluationsmethods, empirical evaluations
Retrieval time Retrieval time Retrieval sorting Retrieval sorting Case base consistency can be indicated by Case base consistency can be indicated by
retrievals resulting cases with gradual values of retrievals resulting cases with gradual values of similarity. A retrieval that no case has a high value similarity. A retrieval that no case has a high value of similarity or too many cases have the same of similarity or too many cases have the same value suggests inconsistency in the case base value suggests inconsistency in the case base
validation & verification (iii)validation & verification (iii)
Cop
yrig
ht R
osin
a W
eber
MaintenanceMaintenance if the reasoner learns, the maintenance if the reasoner learns, the maintenance
is more elaborateis more elaborate statistics of case usagestatistics of case usage perform validation tests continuouslyperform validation tests continuously special issue on case-based maintenancespecial issue on case-based maintenance Neural networks and other soft Neural networks and other soft
computing methods have been proposedcomputing methods have been proposed methods for distributed case basesmethods for distributed case bases D. B. Leake, B. Smyth, D. C. Wilson, Q. D. B. Leake, B. Smyth, D. C. Wilson, Q.
Yang, “Special issue on maintaining Yang, “Special issue on maintaining case-based reasoning systems,” case-based reasoning systems,” Computational Intelligence, 17(2), Computational Intelligence, 17(2), pp.193-195, 2001. pp.193-195, 2001.
Cop
yrig
ht R
osin
a W
eber
Determine the domain and scope Determine the domain and scope Consider reusing existing ontologies Consider reusing existing ontologies Enumerate important terms in the Enumerate important terms in the
ontology ontology Define the classes and the class Define the classes and the class
hierarchyhierarchy Define the attributes of classes (slots) Define the attributes of classes (slots) Define the facets of the slots Define the facets of the slots Create instances Create instances
from Noy & McGuinness
Ontologies Development Life Ontologies Development Life CycleCycle
Cop
yrig
ht R
osin
a W
eber
Ontologies Development Ontologies Development ProcessProcess
ontology development is an iterative processontology development is an iterative process
determinescope
considerreuse
enumerateterms
defineclasses
considerreuse
enumerateterms
defineclasses
defineproperties
createinstances
defineclasses
defineproperties
defineconstraints
createinstances
defineclasses
considerreuse
defineproperties
defineconstraints
createinstances
from Noy & McGuinness
UncertaintyUncertainty
Cop
yrig
ht R
osin
a W
eber
Uncertainty in InformationUncertainty in Information
Forms of ignorance:Forms of ignorance:incompleteness:incompleteness:
value of a variable is missing value of a variable is missing imprecision:imprecision:
value of a variable is given but not with the value of a variable is given but not with the precision required precision required
ambiguity:ambiguity: more than one possible value of a variablemore than one possible value of a variable
uncertainty:uncertainty: the value of a variable might be wrong the value of a variable might be wrong
Cop
yrig
ht R
osin
a W
eber
Uncertainty in ElementsUncertainty in ElementsFuzziness or vaguenessFuzziness or vagueness
refers to ill-defined bounds to describe an refers to ill-defined bounds to describe an element.element.
RandomnessRandomness refers to the certainty of whether a given refers to the certainty of whether a given
element belongs or not to a well-defined element belongs or not to a well-defined set.set.
ProbabilityProbability quantifies the chance that an event quantifies the chance that an event
might occur or the belief that a might occur or the belief that a proposition is true.proposition is true.
Cop
yrig
ht R
osin
a W
eber
Uncertain ProblemsUncertain Problems
uncertain knowledgeuncertain knowledge uncertain reasoninguncertain reasoning
When trying to solve a problem:When trying to solve a problem:
input --> reasoning --> outputinput --> reasoning --> output
What’s the impact on the output when What’s the impact on the output when uncertainty of the input is not treated?uncertainty of the input is not treated?
Cop
yrig
ht R
osin
a W
eber
Methods for the Treatment of Methods for the Treatment of UncertaintyUncertainty
Cop
yrig
ht R
osin
a W
eber
Fuzzy Set TheoryFuzzy Set Theory
Fuzzy SetsFuzzy Sets Fuzzy LogicFuzzy Logic Extension principleExtension principle
Cop
yrig
ht R
osin
a W
eber
Set of Americancities in the East
Philadelphia
LA
New York q=1q=1
q=0
q=1
q=?
q=0
Fuzzy SetsFuzzy Sets
Wilmington
Set of big American cities
New York
DE
Wilmington
Philadelphia
LA
Cop
yrig
ht R
osin
a W
eber
Fuzzy Sets Fuzzy Sets Degree of membership (Degree of membership (qq))
Extension PrincipleExtension Principle
L.Zadeh, A.Kandel, R. Yager, G.Klir, L.Zadeh, A.Kandel, R. Yager, G.Klir, H. Zimmermann, B. KoskoH. Zimmermann, B. Kosko
Cop
yrig
ht R
osin
a W
eber
AggregationsAggregations
sound image WM
50 50
100 0 50
40 40 40
FI:fuzzy integral
WM: weighted mean
FI
0
40
Cop
yrig
ht R
osin
a W
eber
Theory of ProbabilityTheory of Probability Measures degree of belief Measures degree of belief 80% degree of belief is a fairly strong expectation80% degree of belief is a fairly strong expectation What does x% chance means?What does x% chance means?
Frequentist approachFrequentist approach x% means that x is expected for each 100 times x% means that x is expected for each 100 times Probability summarizes the uncertainty that Probability summarizes the uncertainty that
comes from ignorancecomes from ignorance Must follow statistical principles, for example Must follow statistical principles, for example
additivity and complementarityadditivity and complementarity
Cop
yrig
ht R
osin
a W
eber
Certainty Factors- CFCertainty Factors- CF ignores complementarityignores complementarity BeliefBelief DisbeliefDisbelief Positive CF=> belief > disbeliefPositive CF=> belief > disbelief Negative CF=> disbelief> beliefNegative CF=> disbelief> belief Zero CF-> belief = disbelief and Zero CF-> belief = disbelief and
they both can be zerothey both can be zero
Cop
yrig
ht R
osin
a W
eber
Uncertainty inUncertainty in knowledge-based systems knowledge-based systems
Cop
yrig
ht R
osin
a W
eber
knowledgebase
(frames, uncertain rules)
knowledgebase
(frames, uncertain rules)
explanationexplanation
generalknowledgegeneral
knowledge
userInterface
userInterface
expertproblemexpert
problem
expertsolutionexpert
solution
uncertain inference
engine
uncertain inference
engine
working memory(uncertain
information)
working memory(uncertain
information)
knowledge acquisition
knowledge acquisition
uncertainty in ES (i)uncertainty in ES (i)
Cop
yrig
ht R
osin
a W
eber
uncertainty in ES (ii)uncertainty in ES (ii)Fuzzy rulesFuzzy rulesIf the students are tall then recruit them to If the students are tall then recruit them to
the basketball teamthe basketball teamFuzzy LogicFuzzy LogicFuzzy implications define truth values of Fuzzy implications define truth values of
propositions; generalized modus ponens, propositions; generalized modus ponens, fuzzy modus ponensfuzzy modus ponens
Rules with Certainty FactorsRules with Certainty FactorsIF liquidity is IF liquidity is very high (very high (0.60.6)) AND benefit AND benefit
is is moderatemoderate (0.8) THEN (0.8) THEN ask for ask for discountdiscount
Cop
yrig
ht R
osin
a W
eber
uncertainty in ES (iii)uncertainty in ES (iii)
Fuzzy informationFuzzy informationThe patient is middle-aged, The patient is middle-aged,
exercises regularly, and is slightly exercises regularly, and is slightly overweightoverweight
Fuzzy inference engineFuzzy inference engineGrades of membership in inference Grades of membership in inference
engines using fuzzy implicationsengines using fuzzy implications
Cop
yrig
ht R
osin
a W
eber
Rules can have Certainty FactorsRules can have Certainty Factors
IF IF liquidity is liquidity is very highvery high
AND AND benefit is benefit is moderatemoderate (0.8) (0.8)THEN THEN purchase cashpurchase cash
Cop
yrig
ht R
osin
a W
eber
solution
Retain
FuzzyRetrieval
Revise
Reuse
proposed
inputproblemwith uncertaininformation
confirmedsolution
initialsolutions
casebase
case adaptationwith fuzzyrules
uncertainty in CBR (i)
solution
CBR assumptions:problems recursimilar problems have similar solutions
situation assessment
Cop
yrig
ht R
osin
a W
eber
uncertainty in CBR (ii)uncertainty in CBR (ii)
prediction:prediction: the set of most similar and look for the the set of most similar and look for the
most likely to be the solutionmost likely to be the solution summary or combination of the solutions summary or combination of the solutions
with measures of central tendency or with measures of central tendency or analogousanalogous
Cop
yrig
ht R
osin
a W
eber
uncertainty in CBR (iii)uncertainty in CBR (iii)
• retrievaltarget case candidate case
weightattr 1 a1t a1c w1
attr 2 a2t a2c w2
attr 3 a3t a3c w3
• nearest neighbor algorithms implement synthetic evaluation of similarity
• benefit of fuzzy methods is the relaxation of the additivity axiom
Cop
yrig
ht R
osin
a W
eber
Uncertainty in OntologiesUncertainty in OntologiesIgnorance in classes, objects, facetsIgnorance in classes, objects, facets
incompletenessincompleteness imprecisionimprecision ambiguityambiguity uncertainty uncertainty
Elements and facts (parameters)Elements and facts (parameters) fuzziness or vaguenessfuzziness or vagueness randomnessrandomness probabilityprobabilityUncertain reasoning: methods, functions, Uncertain reasoning: methods, functions,
axiomsaxioms
probabilityprobabilityvs.vs.
possibilitypossibility
Cop
yrig
ht R
osin
a W
eber
Probability Distribution Frequentist approachObserving John eating breakfast 100 times
freq eggs65 235 10 0,3,4,5,…
However, it would still be very easy for him to eat 3 or 0 eggs, so even though these would have very low probability, they would have very high possibility.
Cop
yrig
ht R
osin
a W
eber
Possibility Distribution
How "easy" it is for John to eat eggs for breakfast
PD= 2/0.9 + 3/0.9 + 4/0.75+5/0.6 +6/0.3+ 7/0.05
Cop
yrig
ht R
osin
a W
eber
possibility probability distribution distribution
1 egg 0.8 0.352 eggs 0.9 0.653 eggs 0.9 0.04 eggs 0.75 0.05 eggs 0.6 0.06 eggs 0.3 0.07 eggs 0.05 0.0
Methods based on probability are based on classic binary logic and must meet axioms such as additivityMethods based on Fussy Set Theory have weaker axioms
Cop
yrig
ht R
osin
a W
eber
probability vs. possibilityprobability vs. possibility What’s the difference in a real application?What’s the difference in a real application? Probability is easier to acquire Probability is easier to acquire Ease of happening requires elicitationEase of happening requires elicitation Possibility seems more amenable to specific Possibility seems more amenable to specific
and limited domains where the ‘ease’ will be and limited domains where the ‘ease’ will be availableavailable
More general problems should be more More general problems should be more difficult to validate with possibility, thus difficult to validate with possibility, thus more suited to be dealt with probabilitymore suited to be dealt with probability
Some solutions for some Some solutions for some problemsproblems
Cop
yrig
ht R
osin
a W
eber
What are the problems and solutions?What are the problems and solutions?
IncompletenessIncompleteness: : value of a variable is value of a variable is missingmissingIn CBR, having all values is a requirement;In CBR, having all values is a requirement;
In ES, a rule cannot be triggered;In ES, a rule cannot be triggered;
In NN, an example will not be trained;In NN, an example will not be trained;Solutions:•substitute the missing value for a degree of belief or truth (probability);•create rules that presents a direction for incomplete values based on domain knowledge;•Replace incomplete variables with fuzzy sets;
Cop
yrig
ht R
osin
a W
eber
Imprecision: Imprecision: value of a variable is value of a variable is given but not with the precision given but not with the precision required (e.g., grades)required (e.g., grades)Why does it happen?Why does it happen? originally imprecise (age in yrs)originally imprecise (age in yrs) generated by imprecise methodgenerated by imprecise method need to work at an abstract levelneed to work at an abstract level summarizedsummarized
Solutions:•replace values for intervals;•define degrees of belief/degrees of truth;•reason at an abstract level;•measure the imprecision and convey its effects to the final result;
Cop
yrig
ht R
osin
a W
eber
AmbiguityAmbiguity: : more than one possible more than one possible value of a variable; possible values value of a variable; possible values are knownare knownSolutions: Who can disambiguate? Depends on the source of clarification.• In listening, text understanding, information extraction• degree of belief, truth;• ontologies• choose which is the most likely meaning (with uncertainty)• test the meaning in similar natural language sources (e.g., text, people);
Cop
yrig
ht R
osin
a W
eber
UncertaintyUncertainty: : the value of a variable the value of a variable might be wrong might be wrong
prediction;prediction; during reasoning, partial values;during reasoning, partial values; when you must defuzzify or eliminate the when you must defuzzify or eliminate the
degree of belief;degree of belief;
Solutions:• reduce the uncertainty, where does it come
from?• typicality of a set as an alternative to measures
of central tendency;• degree of belief;• fuzzy implications for possible values,
possibility distribution;