Increasing Trust in Answers from Intelligence Increasing Trust in Answers from Intelligence Applications:Applications:

the Inference Web Approachthe Inference Web Approach

Deborah McGuinnessDeborah McGuinnessCo-Director and Senior Research ScientistCo-Director and Senior Research Scientist

Knowledge Systems LaboratoryKnowledge Systems LaboratoryStanford UniversityStanford University

dlm@ksl.stanford.edudlm@ksl.stanford.eduhttp://www.ksl.stanford.edu/people/dlm http://www.ksl.stanford.edu/people/dlm

Inference Web is joint work with Pinheiro da Silva, Fikes, Chang, Inference Web is joint work with Pinheiro da Silva, Fikes, Chang, Deshwal, Narayanan, Glass, Makarios, Jenkins, Millar, Ding, …Deshwal, Narayanan, Glass, Makarios, Jenkins, Millar, Ding, …

22

Semantic Web LayersSemantic Web LayersOntology Level Ontology Level

Languages (CLASSIC, DAML-ONT, DAML+OIL, OWL, …)Languages (CLASSIC, DAML-ONT, DAML+OIL, OWL, …) Environments (FindUR, Chimaera, OntoBuilder/Server, Environments (FindUR, Chimaera, OntoBuilder/Server,

Sandpiper Tools, …)Sandpiper Tools, …) Standards (NAPLPS, …, W3C’s WebOnt, W3C’s Semantic Web Standards (NAPLPS, …, W3C’s WebOnt, W3C’s Semantic Web

Best Practices, EU/US Joint Committee, OMG ODM, …Best Practices, EU/US Joint Committee, OMG ODM, …Rules Rules

SWRL (previously CLASSIC Rules, explanation environment, SWRL (previously CLASSIC Rules, explanation environment, extensibility issues, contracts, …)extensibility issues, contracts, …)

LogicLogic Description LogicsDescription Logics

ProofProof PML, Inference Web Services and PML, Inference Web Services and InfrastructureInfrastructure

TrustTrust IWTrust, NSF with W3C/MIT IWTrust, NSF with W3C/MIT

http://www.w3.org/2004/Talks/0412-RDF-functions/slide4-0.html

33

Motivation – Trust and UnderstandingMotivation – Trust and UnderstandingIf users (humans and agents) are to use, reuse, and If users (humans and agents) are to use, reuse, and

integrate system answers, they must trust them. integrate system answers, they must trust them. System transparency supports understanding and trust.System transparency supports understanding and trust.Even simple “lookup” systems benefit from providing Even simple “lookup” systems benefit from providing

information about their sources.information about their sources.Systems that manipulate information (with sound Systems that manipulate information (with sound

deduction or potentially unsound heuristics) benefit from deduction or potentially unsound heuristics) benefit from providing information about their manipulations.providing information about their manipulations.

Goal: Provide interoperable infrastructure that supports explanations of sources, assumptions, and answers as an

enabler for trust.

44

Requirements gathered from…Requirements gathered from…DARPA Agent Markup Language (DARPA Agent Markup Language (DAMLDAML))

Enable the next generation of the webEnable the next generation of the webDARPA Personal Assistant that Learns (DARPA Personal Assistant that Learns (PALPAL))

Enable computer systems that can reason, learn, be told what to do, Enable computer systems that can reason, learn, be told what to do, explain what they are doing, reflect on their experience, & respond explain what they are doing, reflect on their experience, & respond robustly to surpriserobustly to surprise

DARPA Rapid Knowledge Formation (DARPA Rapid Knowledge Formation (RKFRKF))Allow distributed teams of subject matter experts to quickly and easily Allow distributed teams of subject matter experts to quickly and easily

build, maintain, and use knowledge bases without need for specialized build, maintain, and use knowledge bases without need for specialized trainingtraining

DARPA High Performance Knowledge Base (DARPA High Performance Knowledge Base (HPKBHPKB))Advance the technology of how computers acquire, represent & Advance the technology of how computers acquire, represent &

manipulate knowledgemanipulate knowledgeARDA Novel Intelligence for Massive Data (ARDA Novel Intelligence for Massive Data (NIMDNIMD))

Avoid strategic surprise by helping analysts be more effective (focus Avoid strategic surprise by helping analysts be more effective (focus attention on critical information and help attention on critical information and help analyze/prune/refine/explain/reuse/…)analyze/prune/refine/explain/reuse/…)

ARDA Advanced Question & Answering for Intelligence (ARDA Advanced Question & Answering for Intelligence (AQUAINTAQUAINT))Advance QA against structured and unstructured infoAdvance QA against structured and unstructured info

Consulting including search, ecommerce, configuration, …Consulting including search, ecommerce, configuration, …

55

RequirementsRequirementsInformation Manipulation Traces Information Manipulation Traces

hybrid, distributed, portable, shareable, hybrid, distributed, portable, shareable, combinable encoding of proof fragments supporting combinable encoding of proof fragments supporting multiple justificationsmultiple justifications

PresentationPresentation multiple display formats supporting browsing, multiple display formats supporting browsing,

visualization, summaries,…visualization, summaries,…AbstractionAbstraction

understandable summariesunderstandable summariesInteractionInteraction

multi-modal mixed initiative options including multi-modal mixed initiative options including natural-language and GUI dialogues, adaptive, natural-language and GUI dialogues, adaptive, context-sensitive interactioncontext-sensitive interaction

TrustTrust source and reasoning provenance, automated trust source and reasoning provenance, automated trust

inferenceinference[McGuinness & Pinheiro da Silva, ISWC 2003, J. [McGuinness & Pinheiro da Silva, ISWC 2003, J.

Journal of Web Semantics 2004]Journal of Web Semantics 2004]

66

Selected HistorySelected History

Historical explanation research motivated by Historical explanation research motivated by explaining theorem provers in practiceexplaining theorem provers in practice

Web version originally aimed at explaining hybrid Web version originally aimed at explaining hybrid (FOL / special purpose) reasoners in a distributed (FOL / special purpose) reasoners in a distributed environment like the web.environment like the web.

User demand drove focus on provenance extensionsUser demand drove focus on provenance extensions Current web environment and programs, such as Current web environment and programs, such as

NIMD, drove connections with extraction enginesNIMD, drove connections with extraction engines

Current view: Any question answering system can be Current view: Any question answering system can be viewed as some kind of information manipulator that viewed as some kind of information manipulator that may benefit from and/or require explanationmay benefit from and/or require explanation

77

Inference Web *Inference Web *Framework for Framework for explainingexplaining question answering tasks by abstracting, storing, question answering tasks by abstracting, storing,

exchanging, combining, annotating, filtering, segmenting, comparing, and exchanging, combining, annotating, filtering, segmenting, comparing, and rendering proofs and proof fragments provided by question answerersrendering proofs and proof fragments provided by question answerers

IW’s Proof Markup Language (PMLIW’s Proof Markup Language (PML)) is an interlingua for proof is an interlingua for proof interchange. It is written in W3C’s recommended Ontology web interchange. It is written in W3C’s recommended Ontology web language (OWL)language (OWL)

IWBaseIWBase is a distributed repository of meta-information related to proofs is a distributed repository of meta-information related to proofs and their explanationsand their explanations

IW RegistrationIW Registration services services provide support for proof generation and provide support for proof generation and checkingchecking

IW BrowserIW Browser provides display capabilities for PML documents containing provides display capabilities for PML documents containing proofs and explanations (possibly from multiple inference engines)proofs and explanations (possibly from multiple inference engines)

IW AbstractorIW Abstractor provides rewriting capabilities enabling more provides rewriting capabilities enabling more understandable presentationsunderstandable presentations

IW ExplainerIW Explainer provides multi-modal dialogue options including provides multi-modal dialogue options including alternative strategies for presenting explanations, visualizations, and alternative strategies for presenting explanations, visualizations, and summariessummaries

*Work with Pinheiro da Silva*Work with Pinheiro da Silva

88

Trust

Explainable System StructureExplainable System Structure

Presentation

Abstraction

PML InferenceML

InferenceRule

Specs

SourceProvenance

Data

InformationManipulation

Data

Proof Interlingua (PML)Source

ProvenanceData

InformationManipulation

Data

Interaction

Explanation

99

Registry InformationRegistry Information

IWBase has core and domain-specific repositories of IWBase has core and domain-specific repositories of meta-data useful for disclosing knowledge meta-data useful for disclosing knowledge provenance and reasoning information such as provenance and reasoning information such as descriptions of descriptions of Question answering systems (Inference Engines, Question answering systems (Inference Engines,

Extractors, …) along with their supported Extractors, …) along with their supported inference rulesinference rules

Information sources such as organizations, Information sources such as organizations, publications and ontologiespublications and ontologies

Representation languages along with their Representation languages along with their axiomsaxioms

1010

1111

1212

Browsing ProofsBrowsing Proofs Enable the visualization of proofs (and abstracted proofs)Enable the visualization of proofs (and abstracted proofs) Proofs can be “extracted” and browsed from both local and Proofs can be “extracted” and browsed from both local and

remote PML node sets and can be combinedremote PML node sets and can be combined Links provide access to proof-related meta-informationLinks provide access to proof-related meta-information

selectselect

1313

Browsing ProofsBrowsing Proofs

1414

ExplainerExplainer

Present Present QueryQuery AnswerAnswer Abstraction of justification (PML information)Abstraction of justification (PML information) Limited meta informationLimited meta information Suggests drill down options (also provides Suggests drill down options (also provides

feedback options)feedback options)

1515

UIMA ExplanationUIMA Explanation

1616

Follow-up : MetadataFollow-up : Metadata

1717

Follow-up: AssumptionsFollow-up: Assumptions

1818

Explaining Extracted Entities Explaining Extracted Entities (Techies)(Techies)

Sentences in English

Sentences in annotated English

Sentences in logical format, i.e., KIF

1919

Further Observations on Explaining Further Observations on Explaining Extracted EntitiesExtracted Entities

Source: fbi_01.txtSource Usage: span from 01 to 78

This extractor decided that Person_fbi-01.txt_46

is a Person and not Occupation

Same conclusion from multiple extractors

conflicting conclusion from one extractor

2020

Search / ConfigurationSearch / Configuration

2121

KSL Wine AgentKSL Wine AgentSemantic Web Integration ExampleSemantic Web Integration Example

Uses emerging web standards to enable smart web applications

Given a meal description •Deborah’s Specialty

Describe matching wines•White, Dry, Full bodied…

Retrieve some specific options from web•Forman Chardonnay from DLM’s cellar, ThreeSteps from wine.com, ….

•

Info: http://www.ksl.stanford.edu/people/dlm/webont/wineAgent/

2222

KSL Wine AgentKSL Wine AgentSemantic Web Integration Semantic Web Integration

TechnologyTechnology OWL

for representing a domain ontology of foods, wines, their properties, and relationships between them

JTP theorem prover for deriving appropriate pairings

DQL/OWL QL for querying a knowledge base

Inference Web for explaining and validating answers (descriptions or instances)

Web Services for interfacing with vendors

Connections to online web agents/information services Utilities for conducting and caching the above transactions

2323

2424

Knowledge Provenance ElicitationKnowledge Provenance Elicitation

A^BDA^IMP

ADA

B

DA

A->(A^B)

DA

A->(A^B) A

A^BMP

A B

A^B^I

A^BDir.Ass.

(CNN,BBC) (BBC,NYT) (CNN)

CNNBBC NYT

Why should I believe this?

XYZ says ‘A^B’ is the answer for my

question.

“has opinion”“has opinion”

“has opinion”

Provenance information may be essential for users to trust answers.Data provenance (aka data lineage) is defined and studied in the database literature. [Buneman et al., ICDT 2001] [Cui and Widom, VLDB 2001]Knowledge provenance extends data provenance by adding data derivation provenance information[Pinheiro da Silva, McGuinness & McCool, Data Eng. Bulletin, 2003]

2525

IWTrust: Trust in ActionIWTrust: Trust in Action

(CNN,FSP) (FSP,NYT) (CNN)

A^BDA^IMP

ADA

B

DA

A->(A^B)

DA

A->(A^B) A

A^BMP

A B

A^B^I

BDA

CNNFSP NYT

Why should I trust the answer?

0

0

++

++

++

+

+ +0

Google-2.0 says ‘A^B’ is the answer

for my question.

? ?

?

Trust can be inferred from a Web of Trust.IWTrust provides infrastructure for building webs of trust.The infrastructure includes a trust component responsible for computing trust values for answers.IWTrust is described in[Zaihrayeu, Pinheiro da Silva & McGuinness, iTrust 2005]

A^B

2626

Explanation Application AreasExplanation Application AreasTheorem provingTheorem proving

First-Order Theorem Provers – Stanford (JTP First-Order Theorem Provers – Stanford (JTP (KIF/OWL/…)); SRI (SNARK); University of Texas, Austin (KIF/OWL/…)); SRI (SNARK); University of Texas, Austin (KM);(KM);

SATisfiability Solvers – University of Trento (JSAT)SATisfiability Solvers – University of Trento (JSAT)Information extraction – IBM (UIMA), Stanford (TAP)Information extraction – IBM (UIMA), Stanford (TAP)Information integration/aggregation – USC ISI Information integration/aggregation – USC ISI

(Prometheus,Mediator -> Fetch); Rutgers , Stanford (TAP)(Prometheus,Mediator -> Fetch); Rutgers , Stanford (TAP)Task processing – SRI International (SPARK)Task processing – SRI International (SPARK)Service composition – Stanford, U. of Toronto, UCSF (SDS)Service composition – Stanford, U. of Toronto, UCSF (SDS)Semantic matching – University of Trento (S-MATCH)Semantic matching – University of Trento (S-MATCH)Debugging ontologies – U of Maryland, College Park Debugging ontologies – U of Maryland, College Park

(SWOOP/Pellet)*(SWOOP/Pellet)*Problem solving – University of FortalezaProblem solving – University of FortalezaTrust Networks – U. of Trento (IWTrust), UMd*Trust Networks – U. of Trento (IWTrust), UMd*

2727

Inference Web ContributionsInference Web Contributions

Trust

Presentation

AbstractionInference

Meta-LanguageInference

RuleSpecs

ProvenanceMeta-data

InformationManipulation

Data

Interaction

Explanation

Proof Markup Language

1. Language for encoding hybrid, distributed proof fragments based on web technologies. Support for both formal and informal proofs (information manipulation traces).

2. Support (registry, language, services) for knowledge provenance

4. Multiple strategies for proof abstraction, presentation and interaction.5. End-to-end trust value computation for answers.

3. Declarative inference rule representation for checking hybrid, distributed proofs.

6. Comprehensive solution for explainable systems

1 2 3

4

2

56

4

4

2828

StatusStatus

Inference Web infrastructure (PML, browser, explainer, Inference Web infrastructure (PML, browser, explainer, registry, toolkit) being used in government programs such registry, toolkit) being used in government programs such as PAL and NIMD, commercial research labs – IBM, as PAL and NIMD, commercial research labs – IBM, Boeing, SRI, Universities – USC, U MD, …Boeing, SRI, Universities – USC, U MD, …

Integration and registration process underway with Integration and registration process underway with extraction communityextraction community

Useful now for helping decide if information is Useful now for helping decide if information is trustworthy, comes from authoritative sources, consistent, trustworthy, comes from authoritative sources, consistent, reliablereliable

Benefits from more meta data and more information Benefits from more meta data and more information population but is useful in an population but is useful in an incremental natureincremental nature

2929

Technical StatusTechnical StatusSome focus areas:Some focus areas:

Follow-up question supportFollow-up question supportTrustTrustContradiction support Contradiction support Abstraction techniquesAbstraction techniquesExtraction extensions Extraction extensions Task-oriented reasoning supportTask-oriented reasoning supportQuery manager explanation supportQuery manager explanation supportToolkit for embeddingToolkit for embedding

Open issues for explanationOpen issues for explanationGranularity of explanationsGranularity of explanationsMeta information filteringMeta information filteringAbstraction techniquesAbstraction techniquesRequests / Suggestions?Requests / Suggestions?

3030

More Info:More Info:

Inference Web: Inference Web: http://iw.stanford.eduhttp://iw.stanford.eduOWL: http://www.w3.org/TR/owl-features/

http://www.w3.org/TR/owl-guide/DAML+OIL: http://www.daml.org/WineAgent: www.ksl.stanford.edu/people/dlm/webont/wineAgent/ Chimaera: http://

www.ksl.stanford.edu/software/chimaera/ OWL-QL/DQL: http://www.ksl.stanford.edu/projects/dql/ UIMA: http://www.research.ibm.com/UIMA/

dlm@ksl.stanford.edu dlm@ksl.stanford.edu

3131

ExtrasExtras

3232

BackgroundBackgroundAT&T Bell Labs AI Principles DeptAT&T Bell Labs AI Principles Dept

Description Logics, CLASSIC, explanation, ontology environmentsDescription Logics, CLASSIC, explanation, ontology environments Semantic Search, FindUR, Collaborative Ontology Building EnvSemantic Search, FindUR, Collaborative Ontology Building Env Apps: Configurators, PROSE/Questar, Data Mining, …Apps: Configurators, PROSE/Questar, Data Mining, …

Stanford Knowledge Systems, Artificial Intelligence LabStanford Knowledge Systems, Artificial Intelligence Lab Ontology Evolution Environments (Diagnostics and Merging) Ontology Evolution Environments (Diagnostics and Merging)

ChimaeraChimaera Explanation and Trust, Inference WebExplanation and Trust, Inference Web Semantic Web Representation and Reasoning Languages, DAML-Semantic Web Representation and Reasoning Languages, DAML-

ONT, DAML+OIL, OWL,ONT, DAML+OIL, OWL, Rules and Services: SWRL, OWL-S, Explainable SDS, KSL Wine Rules and Services: SWRL, OWL-S, Explainable SDS, KSL Wine

AgentAgent`McGuinness Associates`McGuinness Associates

Ontology Environments: Sandpiper, VerticalNet, Cisco…Ontology Environments: Sandpiper, VerticalNet, Cisco… Knowledge Acquisition and Ontology Building – VSTO, GeON, ImEp,Knowledge Acquisition and Ontology Building – VSTO, GeON, ImEp,

…… Applications: GM: Search, etc.; CISCO : meta data org, etc.; Applications: GM: Search, etc.; CISCO : meta data org, etc.; Boards: Network Inference, Sandpiper, Buildfolio, Tumri, KatalytikBoards: Network Inference, Sandpiper, Buildfolio, Tumri, Katalytik

3333

KSL Wine Agent: Semantic Web KSL Wine Agent: Semantic Web Integration Integration (Toy) Example(Toy) Example

Uses emerging web standards to enable “smart” web application

Given a meal description •Deborah’s Specialty, a crab dish, …

Describe matching wines•White, Dry, Full bodied…

Retrieve some specific options from web•Forman Chardonnay from DLM’s cellar, ThreeSteps from wine.com, ….

Explain description or specific suggestion

Info: http://www.ksl.stanford.edu/people/dlm/webont/wineAgent/

3434

KSL Wine AgentKSL Wine AgentSemantic Web Integration Semantic Web Integration

TechnologyTechnology

• OWL: for representing a domain ontology of foods, wines, their properties, and relationships between them• JTP theorem prover: for deriving appropriate pairings• Chimaera: ontology diagnostics and ontology merging• DQL/OWL QL : for querying a knowledge base • Inference Web: for explaining and validating answers (descriptions or instances)• Web Services: for interfacing with vendors• Connections to online web agents/information services• Utilities for conducting and caching the above transactions

3535

Inference Web in KANI ContextInference Web in KANI Context

RelevantKnowledge

Identification(TAP)

The World

Corpus

ExtractedKnowledge

DB WorkingKB

Selection

KnowledgeExtraction

Devil’sAdvocate

HypothesisModeling &

Testing

SemanticSearch

ExplanationGeneration

KeywordSearch

SharedReasoning

AnalysisManagement

BackgroundKB

Ontology

Entities

Models

KnowledgeTransfer

KnowledgeInteraction

KnowledgeBrowsing &Selection

Legend

Data

SystemComponent

SystemService

User InterfaceFeature

InferenceWeb

3636

SummarySummary

Tools are emerging that support Tools are emerging that support understanding informationunderstanding information

Understanding/Evaluating information can help Understanding/Evaluating information can help focus a user’s attention and enable trust, focus a user’s attention and enable trust, reuse, and filtering reuse, and filtering

Semantic Web infrastructure (OWL, Structured Semantic Web infrastructure (OWL, Structured query languages, Semantic Search, Extractors, query languages, Semantic Search, Extractors, Reasoners, Explanation Infrastructure, ….) is Reasoners, Explanation Infrastructure, ….) is ready for use and a growing trend ready for use and a growing trend

3737

Knowledge Provenance Multiple Knowledge Provenance Multiple SourcesSources

Answer

Source

Source

3838

ExtraExtra

3939

Inferences drawn by Information ExtractionInferences drawn by Information Extraction

AER:(Person

“Mr. Ramazi”)

AER:(Org “Select

Gourmet Foods”)

AER:(Person “Abdul

Ramazi”)

BER:(Person “Abdul

Ramazi”)

BER:(Org “Select

Gourmet Foods”)

BER:(Org “SGF”)

ARR:(hasOwner

“Select Gourmet Foods”,

“Mr. Ramazi”)

BRR:(hasOwner

“SGF”,“Abdul Ramazi”)

MCR:(equals “Abdul Ramazi”,

“Mr. Ramazi”, AbdulRamazi)

MCR:(equals “Select Gourmet Foods”,

“SGF”, SelectGourmetFoods)

MCR:(hasOwner SelectGourmetFoods, AbdulRamazi)

Document:CIA Report 117

Document:FBI Report 282

4040

Statistics for relevant domain independent meta-data:

1212Languages Languages

66Derived Rules Derived Rules

1010Method Rules Method Rules

3838Declarative Rules Declarative Rules

5656Axioms Axioms

2929Inference Engines Inference Engines

Infrastructure: Core IWBaseInfrastructure: Core IWBase

select

select

4141

Explaining Answers: GUI ExplainerExplaining Answers: GUI Explainer

Users can exit the explainer providing feedback about their satisfiability with explanation(s)

Users can ask for alternative explanationsor summaries

Select action

4242

Follow-up: PML AbstractionFollow-up: PML Abstraction(Techies only)(Techies only)

4343

BrowsingBrowsing

Present Present QueryQuery AnswerAnswer Alternate formats (KIF, English, Raw, …)Alternate formats (KIF, English, Raw, …) Graph Structure (with lens view)Graph Structure (with lens view) AnnotationsAnnotations

4444

Knowledge Provenance Multiple Knowledge Provenance Multiple SourcesSources

Answer

Source

Source

IWTrust:IWTrust:Improving User Trust inImproving User Trust in

Answers from the Web Answers from the Web

Ilya ZaihrayeuIlya ZaihrayeuITC-IRSTITC-IRST

Paulo Pinheiro da SilvaPaulo Pinheiro da SilvaDeborah L. McGuinnessDeborah L. McGuinness

Stanford UniversityStanford University

4646

Trusting AnswersTrusting Answers It may be challenging for users to establish their degrees It may be challenging for users to establish their degrees

of trust, untrust, mistrust and distrust in a web of trust, untrust, mistrust and distrust in a web application answer if the answer is provided without any application answer if the answer is provided without any kind of kind of justificationjustification

Knowledge ProvenanceKnowledge Provenance (KP) is a description of both (KP) is a description of both the the origins of knowledge and the reasoning process to produce an origins of knowledge and the reasoning process to produce an answeranswer

Users may need KP to establish a degree of trust in the Users may need KP to establish a degree of trust in the answeranswer Which sources were used?Which sources were used? Who are the authors of such sources?Who are the authors of such sources? Which engines were used?Which engines were used? What are the assumptions of the engines? Are the engines’ rules What are the assumptions of the engines? Are the engines’ rules

sound?sound? KP itself may not be enough for trusting the answerKP itself may not be enough for trusting the answer

I may not know anything about one or more sources in the KPI may not know anything about one or more sources in the KP I may have no information about the reliability of one or more of I may have no information about the reliability of one or more of

then engines in the KPthen engines in the KP

4747

Trusting Answers Trusting Answers from the Webfrom the Web The overall process of establishing a degree of trust in The overall process of establishing a degree of trust in

answers from web applications is particularly complex answers from web applications is particularly complex since applications may rely on:since applications may rely on: Hybrid and distributed processing, e.g., web services, the GridHybrid and distributed processing, e.g., web services, the Grid Large number of heterogeneous, distributed information sources, Large number of heterogeneous, distributed information sources,

e.g., the Webe.g., the Web information sources with more variation in their reliability, e.g., information sources with more variation in their reliability, e.g.,

information extraction information extraction Sophisticated information integration methods, e.g., SIMS, Sophisticated information integration methods, e.g., SIMS,

TSIMMISTSIMMIS The definition of trust is a significant part of the processThe definition of trust is a significant part of the process The task of keeping, encoding, sharing and gathering The task of keeping, encoding, sharing and gathering

KP for partial answers towards the generation of the KP for partial answers towards the generation of the KP for answers is another part of the processKP for answers is another part of the process

The use of KP to derive trust values for answers is The use of KP to derive trust values for answers is yet another part of the processyet another part of the process

4848

The Inference WebThe Inference Web

(A1, t11, t12,...)IE1

S2

IE2

PML Documents IWBase

(A2, t21, t22,...)

(An, tn1, tn2,...)

...

Q(U1)

S1

S3...

The Inference Web is an infrastructure supporting The Inference Web is an infrastructure supporting explanations for answers from the webexplanations for answers from the web The Proof Markup Language (PML) is used to encode answer The Proof Markup Language (PML) is used to encode answer

justification, i.e., information manipulation traces, proofsjustification, i.e., information manipulation traces, proofs IWBase is used to annotate PML documents with proof-related data, IWBase is used to annotate PML documents with proof-related data,

i.e., trust values for sources and enginesi.e., trust values for sources and engines User UUser U11 asks question Q asks question Q A question answering system returns the set of answers {AA question answering system returns the set of answers {A11,A,A22,…,A,…,Ann}}

4949

Inference Web and KPInference Web and KP Inference Web is an infrastructure supporting Inference Web is an infrastructure supporting

KP for answers derived by multiple methodsKP for answers derived by multiple methods Information extraction –Information extraction – IBM (UIMA), Stanford (TAP)IBM (UIMA), Stanford (TAP) Information integration –Information integration – USC ISI (Prometheus/Mediator); USC ISI (Prometheus/Mediator);

Rutgers University (Prolog/Datalog)Rutgers University (Prolog/Datalog) Task processing –Task processing – SRI International (SPARK)SRI International (SPARK) Theorem provingTheorem proving

First-Order Theorem Provers –First-Order Theorem Provers –SRI International (SNARK); Stanford SRI International (SNARK); Stanford (JTP); University of Texas, Austin (KM) (JTP); University of Texas, Austin (KM)

SATisfiability Solvers –SATisfiability Solvers – University of Trento (J-SAT)University of Trento (J-SAT) Expert Systems –Expert Systems – University of Fortaleza (JEOPS)University of Fortaleza (JEOPS)

Service composition –Service composition – Stanford, University of Toronto, UCSF Stanford, University of Toronto, UCSF (SDS)(SDS)

Semantic matching –Semantic matching – University of Trento (S-Match)University of Trento (S-Match) Debugging ontologiesDebugging ontologies – – University of Maryland, College Park University of Maryland, College Park

(SWOOP/Pellet)(SWOOP/Pellet) Problem solving –Problem solving – University of Fortaleza (ExpertCop)University of Fortaleza (ExpertCop)

5050

The Inference Web Trust (IWTrust)The Inference Web Trust (IWTrust)

(A1, t11, t12,...)IE1

S2

IE2

IW Trust Framework

PML Documents IWBase

(A2, t21, t22,...)

(An, tn1, tn2,...)

...

Q(U1)

S1

S3...

S4

IW TrustNet

u4

u7 u6

u3

u5u1

t1-5

t5-6

t6-7

t6-3

t1-3

t3-4

t7-S1

t7-IE1

t4-S4

t4-S3

t1-IE2

IWTrust extends the Inference Web to support trust computationIWTrust extends the Inference Web to support trust computation IW TrustNet is a social network of source recommendersIW TrustNet is a social network of source recommenders A trust component implementing an algorithm to compute trust values for A trust component implementing an algorithm to compute trust values for

answersanswers Trust values are used to rank answers and answer justificationsTrust values are used to rank answers and answer justifications User UUser U11 trusts U trusts U33 to a degree t to a degree t1-31-3

5151

http://www.w3.org/2004/Talks/0412-RDF-functions/slide4-0.html

ConclusionsConclusions IWTrust provides infrastructure for building a trust graph from a user IWTrust provides infrastructure for building a trust graph from a user

asking a question to the answers for the questionasking a question to the answers for the question Knowledge Provenance is a key element of the trust graph and a Knowledge Provenance is a key element of the trust graph and a

requirement for trusting answers in generalrequirement for trusting answers in general Inference Web is a Semantic Web solution for Knowledge ProvenanceInference Web is a Semantic Web solution for Knowledge Provenance

iw.stanford.eduiw.stanford.edu Adaptive explanations based on user modeling Adaptive explanations based on user modeling IWBase registration of a large set of software systemsIWBase registration of a large set of software systems

Registration of a comprehensive set of primitive rulesRegistration of a comprehensive set of primitive rules Established library of explanation tacticsEstablished library of explanation tactics

Inference Web is a Inference Web is a solution for the Semantic solution for the Semantic Web proof layerWeb proof layer

IWTrust intends to be a IWTrust intends to be a solution for the Semantic solution for the Semantic Web trust layerWeb trust layer