Week 8

MSE614 – SP 08

Ileana Costea

HW Questions on KA

Due today, Week 8

Assigned last session, Week 7

A few verbal questions

(see Transparency)

HW Question #1What is meta-knowledge?

Answer: K about K, or The system’s K about how it reasons

e.g., • how to use K in specific situations;• how to determine which K is relevant• when the K is insufficient

Meta-knowledge (ctnd.)

• Meta-K allows the system to examine the operation of the descriptive (declarative) and procedural K

• Explanation in an ES can be viewed as Meta-K

In the future Meta-K will allow ES more:– Create the rationale behind individual rules– by reasoning from first

principles– Tailor explanations to fit the audience– Able to change their internal structure through:

– Rule correction– Reorganization of KB– System reconfiguration

HW Question #2Give four reasons why KA is difficult.

• Representation mismatch between the human Expert and the program underdevelopment

• Large number of participants• Transfer via a machine• Difficulty of experts to describe their K

More … next slide

HW Question #2 (Ctnd.)

Give four reasons why KA is difficult.

• Other reasons, see slide.

• Overcoming difficulties:

• Research on KA to reduce mismatch:

• Develop systems able to accept advice

• KA to converse with expert in natural language

• Simplify the syntax of the rules so that an expert can build a system without training

HW Question #3Describe the process of

Protocol Analysis. (Ctnd.)

• The expert is asked to perform a real task.

• Then he/she is asked to verbalize (think aloud) the thought process.

• A record (protocol) of all the details is made.

• The records are transcribed.

HW Question #3Describe the process of

Protocol Analysis. (Ctnd.)Required Web search:• http://www.psy.fsu.edu/faculty/ericsson/

ericsson.proto.thnk.htmlProtocol analysis and Verbal Reports on ThinkingAn updated and extracted version from Ericsson (2002)

• http://www.epistemics.co.uk/Notes/176-0-0.htmProtocol Analysis TechniquesAmong others discusses Repertory Grid Technique

• http://en.wikipedia.org/wiki/Protocol_analysis

Protocol Analysishttp://en.wikipedia.org/wiki/Protocol_analysis

(from Wikipedia, the free encyclopedia)

• Protocol analysis is a psychological research method that elicits verbal reports from research participants. Protocol analysis is used to study thinking in cognitive psychology, cognitive science, and behavior analysis. It has found further application in the design of surveys and interviews, usability testing, and educational psychology.

• Usability testing is a technique used to evaluate a product by testing it on users.

• Cognitive psychology is a school of thought in psychology that examines internal mental processes such as problem solving, memory, and language.

Cognitive science(from Wikipidia)

• the scientific study either of mind or of intelligence.

• an interdisciplinary study drawing from relevant fields including psychology, philosophy, neuroscience, linguistics, anthropology, computer science, and biology.

• term coined by Christopher Longuet-Higgins in his 1973 commentary on the Lighthill report, which concerned the then-current state of Artificial Intelligence research. In the same decade, the journal Cognitive Science and the Cognitive Science Society began.

Protocol Analysis (Ctnd.)

• in AI – KA it is a method of KA

• it is one of the “tracking methods”

• it is the most common method of formal tracking.

(See next slide for “tracking”)

Tracking Methods(Protocol Analysis – Ctnd.)

• Process tracking: a set of techniques that attempt to track the reasoning process of an expert.

• Popular method among cognitive psychologists who:interested in discovering the expert’s “train of thought” while he/she reaches a conclusion

• Tracking methods are: informal and formal

Protocol Analysis (Ctnd.)

• Particularly a set of techniques known as verbal protocol analysis,

• a common method by which KEngineer acquires detailed K from the expert

Protocol Analysis (Ctnd.)

• A protocol = a record or documentation of the expert’s step-by-step info. processing (IP) and decision-making (DM) behavior

• Protocol Analysis Is similar to interviewing but more formal and systematic

• The expert is asked by the KE to perform a real task and verbalize his/her thought process

• Expert is asked to “think aloud” while performing the task or solving a problem under observation

• A recording made while expert “thinks aloud”• Recording describes every aspect of the IP and DM behavior• The recording becomes a record, or protocol, of the expert’s

ongoing behavior

Protocol Analysis (ctnd.)

• Recording is:

– transcribed by the KE for further analysis

( e.g., to deduce the decision process)– Coded by the KE

Protocol Analysis (ctnd.)

• In contrast with interactive interviewing methods, PA involves mainly a one-way communication.

• KE prepares the scenario and plans the process• during the session the Expert does most of the

talking as he/she interacts with the data to solve problem

• KE listens and records the process• Later KE must: analyze, interpret, and structure

the protocol (or record) into Knowledge Representation for a review by the Expert.

Protocol Analysis (ctnd.)

• for Procedure of PA

&

• Advantages and Limitations of PA

see Transparencies

HW Question #4List the major difficulties of KA from

multiple experts.

• Different experts use different methods of problem solving, yet all may be correct.

• It is difficult to reconcile these different methods.

• It is difficult to get all the experts together at the same time and place.

List the major difficulties of KA from multiple experts. (Ctnd.)

The four possible scenarios to deal with multiple experts:

– individual experts– primary and secondary experts– small groups– panels

List the major difficulties of KA from multiple experts. (Ctnd.)

Major methods of dealing with multiple experts:

• Consensus methods Reach consensus by using group dynamics or other methodologies

• Analytical approaches: figure the average estimate of the group

• Selection of an appropriate line of reasoning for each occasion – don’t mix

• Automation of the processautomatic decision by the computer

• Blackboard systemdivide the problem into sub-domains and use one expert for each sub-domain

HW Question #4Define evaluation, validation and

verification of K.

AAAIhttp://www.aaai.org/home.html

• Association for the Advancement of Artificial Intelligence (AAAI)

(formerly the American Association for Artificial Intelligence)

• is a nonprofit scientific society devoted to advancing the scientific understanding of:

• mechanisms underlying thought & intelligent behavior, and

• their embodiment in machines

AI Applications

http://www.aaai.org/AITopics/pmwiki/pmwiki.php/AITopics/SiteMap

a whole list of links of AI applications

http://www.aaai.org/AITopics/pmwiki/pmwiki.php/AITopics/AINews

AI in the News