Expert & Knowledge-Based Systems• One of AI’s greatest areas of success was the

development of large-scale problem solving systems– Originally called expert systems, they would mimic the

problem solving processes of domain experts• Such as doctors performing diagnosis, or engineers performing design,

or wall street analysts selecting stock transactions

• Expert systems were originally developed by hand– And most commonly in some Lisp dialect

• It was discovered that many problems were being solved by chaining through rules (if-then statements) that would operate on a collection of facts and partial conclusions– Called working memory

• These rule-based systems led to the first AI tools or shells– Today, to simplify expert system creation, most people use

these AI shells – you just fill in the knowledge, the problem solving processes are already implemented

Introduction: Dendral• The Dendral system (DENDRitic ALgorithm) was the

first expert system, developed in the 1960s– The idea was, given mass spectrogram data, determine what the

chemical composition was

• The approach: plan-generate-and-test with human feedback– This is a constrained search technique

• Generate a hypothesis: a possible chemical compound• Test the hypothesis: use a series of heuristics and subprograms to

determine if the chemical compound generated is plausible given the data

• If so, show it to the user – the user can steer the direction that DENDRAL takes next by suggesting what chemical elements should be added or removed from the generated hypothesis compound

• DENDRAL repeats until an answer is found acceptable that does not violate the constraints as dictated by the data

DENDRAL’s ProcessesOperation Component

(program)Input Output

Planning •MOLION

•PLANNER

•Mass spectrum

•Planning rules

•Ion constraints

•Other constraints

•Superatoms

Generating •CONGEN

•GENOA

•STEREO

•Constraints •Candidate molecular structures

Testing •PREDICTOR MSPRUNE

•Candidate molecular structures

•Mass spectrometry rules

•Reaction chemistry rules

•Most plausible structures

•Structures consistent with spectrum

•Structures consistent with known reactions

Each of the component programs was written in Lisp

DENDRAL in Action

Pattern Matching• Expert Systems consisted of two major components

– A knowledge base• The knowledge that the system would use to come to conclusions

– In Dendral, this would be chemical elements, compounds, the the mass spectrogram data that could arise from any element

– In Mycin, this would be bacteriological infection knowledge

– An inference engine• How to apply the knowledge

– For Dendral, this was done by generate-and-test, a very weak form of search

– Mycin would perform backward chaining through rules

– More generically though, the inference engine would embody some form of pattern matching

– One of the first systems that clearly demonstrated pattern matching was Eliza

• Eliza was a spoof of an expert system but the code will show us how pattern matching works

Eliza• Created by Joseph

Weizenbaum• This system would act like a

Rogerian psychoanalyst by holding a dialog with a person– the dialog would be consist of

the doctor (Eliza) asking questions, the human responding, and the doctor using the response to ask another question

• Weizenbaum wrote the program so that it would generate an English response/question based on a group of patterns– If the user sentence matched a

pattern, this pattern would be used to generate the next sentence/question

• Sample dialog:– H: Men are all alike– E: In what way?– H: They’re always bugging us about

something or another.– E: Can you be more specific?– H: Well, my boyfriend made me come here.– E: Your boyfriend made you come here?– H: He says I’m depressed much of the time.– E: I am sorry to hear you are depressed.– H: It’s true. I am unhappy.– E: Do you think coming here will help you

not to be unhappy?– H: I need some help, that much seems

certain.– E: What would it mean to you if you got

some help?– H: Perhaps I could learn to get along with

my mother.– E: Tell me more about your family

How Eliza Works• Repeat

– Input a sentence– Find a rule in the Eliza knowledge-base that matches the

pattern• Attempt to perform pattern match (see next slide)• Attempt to perform segment match (see two slides)

– If rule found, select one of the responses randomly (each pattern will have at least one response)

– Fill in any variables– Substitute values (you for I, I for you, me for you, am for are,

etc)– Respond

• Until user quits• Most of the work goes on in the pattern match and

segment match functions

Pattern Match

• A variable is denoted with a ? as in ?x – The variable can represent a symbol such as “I” or “Fred”

• The function works as follows:– Is the input a variable? If so, match the variable– Else, is the input a segment? If so, then call segment-match– Else, break up the input in two parts and recursively try to match both

(defun pat-match (pattern input &optional (bindings no-bindings)) "Match pattern against input in the context of the bindings" (cond ((eq bindings fail) fail) ((variable-p pattern) (match-variable pattern input bindings)) ((eql pattern input) bindings) ((segment-pattern-p pattern) (segment-match pattern input bindings)) ((and (consp pattern) (consp input)) (pat-match (rest pattern) (rest input) (pat-match (first pattern) (first input) bindings))) (t fail)))

Note: fail = nil

Segment Match

• Essentially pattern-match where a variable can be more than a single symbol – For instance “(?* ?x) hate (?* ?y)” can substitute “I” for the first segment

and “anyone from the University of Michigan” for the second segment• Start controls where to start looking over this segment in case part

has already matched

(defun segment-match (pattern input bindings &optional (start 0)) (let ((var (second (first pattern))) (pat (rest pattern))) (if (null pat) (match-variable var input bindings) (let ((pos (position (first pat) input :start start :test #'equal))) (if (null pos) fail (let ((b2 (pat-match pat (subseq input pos) (match-variable var (subseq input 0 pos) bindings)))) (if (eq b2 fail) (segment-match pattern input bindings (+ pos 1)) b2)))))))

Eliza Rules• Here we see an

excerpt from the rules of Eliza

• For instance, if the input were “I want to have a cheeseburger”, the second pattern would match

• Eliza would respond with one of three outputs using “to have a cheeseburger” in place of ?y

• Such as “Why do you want to have a cheeseburger”

(defparameter *eliza-rules* '((((?* ?x) hello (?* ?y)) (How do you do. Please state your problem.)) (((?* ?x) I want (?* ?y)) (What would it mean if you got ?y) (Why do you want ?y) (Suppose you got ?y soon)) (((?* ?x) if (?* ?y)) (Do you really think its likely that ?y) (Do you wish that ?y) (What do you think about ?y) (Really-- if ?y)) (((?* ?x) no (?* ?y)) (Why not?) (You are being a bit negative) (Are you saying "NO" just to be negative?)) (((?* ?x) I was (?* ?y)) (Were you really?) (Perhaps I already knew you were ?y) (Why do you tell me you were ?y now?)) (((?* ?x) I feel (?* ?y)) (Do you often feel ?y ?)) (((?* ?x) I felt (?* ?y)) (What other feelings do you have?))))

Some Miscellaneous Eliza Functions(defun variable-p (x) ;; Is x a variable (a symbol beginning with `?')? (and (symbolp x) (equal (elt (symbol-name x) 0) #\?)))

(defun match-variable (var input bindings) ;; does the given var match input the input? Updates the bindings (let ((binding (get-binding var bindings))) (cond ((not binding) (extend-bindings var input bindings)) ((equal input (binding-val binding)) bindings) (t fail))))

(defun segment-pattern-p (pattern) ;; segment-matching pattern like ((?* var) . pat)? (and (consp pattern) (consp (first pattern)) (symbolp (first (first pattern))) (segment-match-fn (first (first pattern)))))

A Grammar of Patterns• Here, we break down the components of a pattern

matcher into specific grammatical componentspat var match any one expression to a variable

constant or to a constant (see below)segment-pat match against a sequencesingle-pat match against one expression(pat . pat) match the first and the rest of a list

single-pat (?is var predicate) test predicate on one expression(?or pat1 pat2 …) match on any of the patterns(?and pat1 pat2 …) match on every of the expressions(?not pat) match if expression does not match

segment-pat ((?* var) …) match on zero or more expressions((?+ var) …) match on one or more expressions((?? var) …) match zero or one expression((?if expr) …) test if expression is true

var ?chars variables of the form ?nameconstant atom constants are any atoms (symbols, numbers, chars)

Pattern Matching Examples• Here are some examples of applying pat-match as is

– (pat-match ’((?* ?p) need (?* ?x)) ’(the king and queen need a beheading))

• ((?P THE KING AND QUEEN) (?X A BEHEADING))– (pat-match ’((?* ?x) is a (?* ?y)) ’((what he is is a fool))

• ((?X WHAT HE IS) (?Y A FOOL))– (pat-match ’((?* ?x) a b (?* ?x)) ’(1 2 a b a b 1 2 a b))

• (?X 1 2 A B)

• Consider enhancing pat-match to include new arguments ?is, ?or and ?and to apply setf, or and and while doing the pattern matching– (pat-match ’((x = (?is ?n numberp)) ’(x = 34)) ((?n . 34))– (pat-match ’((x = (?is ?n numberp)) ’(x = x)) NIL– (pat-match ’((?x (?or < = > ?y) ’(3 < 4)) ((?Y . 4) (?X . 3))– (pat-match ’(x = (?and (?is ?n numberp) (?is ?n oddp))) ’(x =

3)) ((?N . 3))

MYCIN• Implemented in the early 1970s, Mycin is perhaps the

most recognized and cited expert system– Developed at Stanford, it performs bacteriological

diagnosis – both disease identification and treatment• Tested against doctors, interns, medical teachers, and medical

students, Mycin actually outperformed them all in an experiment of some 80 different cases!

– Primarily formed out of rules that look like this:

(defrule 52 if (site culture is blood) (gram organism is neg) (morphology organism is rod) (burn patient is serious)then .4 (identity organism is pseudomonas))

Premises can be found in “working memory”

Premises have an identifier and a valueOrganism is a sample (tissue, blood) There might be multiple organisms to evaluatePatient is the current patient being diagnosed

.4 represents a certainty factor how plausible is the statement?

MYCIN Problem Solving Structure• Unlike Eliza which merely responded to the latest input, MYCIN

contains a working memory– Working memory stores a number of premises of the form <(condition list),

(conclusion), CF>– These are placed into a hash table for easy lookup based on the conclusion

• Like Eliza, MYCIN also has a list of rules• The process is to

– Identify all rules that can provide the conclusion currently sought (the initial conclusion is called diagnose-and-treat)

– Any rule that can conclude this is added to a list of rules to test– Each of these rules is used to match their premises against working memory– Any that are true are “fired” – that is, their conclusion is an action used to

modify the hash table, either• add a new piece of knowledge to the hash table• find and remove a piece of knowledge which is no longer needed• find and modify a piece of knowledge now that more specific information is

known

Rules and Certainty Factors• The idea behind MYCIN is that there are thousands of

such rules– If the premises allow one rule to be selected, that will modify

working memory which in turn might let another, more specific, rule to be selected

– How certain is the next rule in the chain of logic?

• Certainty factors have to be combined– Imagine the rule (if (premise 1) (premise 2) then .7

(conclusion3))– If premise1 has a CF of .8 and premise2 has a CF of .5, what is

our CF for conclusion3?– We have to first find the CF of the two premises combined

(ANDing them together)– We then have to propagate the CF across the rule

Combining CFs• CF(P1 & P2) = minimum(CF(P1), CF(P2))• CF(P1 ^ P2) = maximum(CF(P1), CF(P2))• CF(R1 R2) = CF(R1) * CF(R2)• Assume R1 concludes C1 and R2 concludes C1, what is our

conclusion of C1 given R1 and R2?• CF(C1) = CF(R1) + CF(R2) – CF(R1) * CF (R2)

– if CF(R1) > 0 and CF(R2) > 0

(defun cf-or (a b) (cond ((and (> a 0) (> b 0)) (+ a b (* -1 a b))) ((and (< a 0) (< b 0)) (+ a b (* a b))) (t (/ (+ a b) (- 1 (min (abs a) (abs b)))))))

MYCIN Code(defun use-rules (parm) ;; Try every rule associated with this parameter (some #'true-p (mapcar #'use-rule (get-rules parm))))

(defun use-rule (rule) ;; apply a rule (put-db 'current-rule rule) ;; If any premise is known false, give up. ;; If every premise can be proved true, then ;; draw conclusions (weighted with the certainty factor). (unless (some #'reject-premise (rule-premises rule)) (let ((cf (satisfy-premises (rule-premises rule) true))) (when (true-p cf) (dolist (conclusion (rule-conclusions rule)) (conclude conclusion (* cf (rule-cf rule)))) cf))))

Code Continued(defun satisfy-premises (premises cf-so-far) ;; see if all premises of a rule are satisfied (if not, try to satisfy them) ;; cf-so-far is an accumulator of certainty factors (cond ((null premises) cf-so-far) ((not (true-p cf-so-far)) false) (t (satisfy-premises (rest premises) (cf-and cf-so-far (eval-condition (first premises)))))))

(defun eval-condition (condition &optional (find-out-p t)) ;; See if this condition is true, optionally using FIND-OUT (multiple-value-bind (parm inst op val) (parse-condition condition) (when find-out-p (find-out parm inst)) ;; Add up all the (val cf) pairs that satisfy the test (loop for pair in (get-vals parm inst) when (funcall op (first pair) val) sum (second pair))))

Even More Code(defun reject-premise (premise) ;; premise is rejected if it is known to be false or can be ;; determined recursively to be false (false-p (eval-condition premise nil)))

(defun conclude (conclusion cf) ;; add this conclusion with CF to our db of knowledge (multiple-value-bind (parm inst op val) (parse-condition conclusion) (update-cf parm inst val cf)))

(defun parse-condition (condition);; conditions are of the form (parm inst op val).;; example: (age patient is 21) (values (first condition) (get-db (second condition)) (third condition) (fourth condition)))

MYCIN’s English Interface

(defun cf->english (cf) (cond ((= cf 1.0) "there is certain evidence") ((> cf .8) "there is strongly suggestive evidence") ((> cf .5) "there is suggestive evidence") ((> cf 0.0) "there is weakly suggestive evidence") ((= cf 0.0) "there is NO evidence either way") ((< cf 0.0) (concatenate 'string (cf->english (- cf)) "AGAINST the conclusion"))))

(defun print-condition (condition stream number) (format stream "~& ~d)~{ ~a~}" number (let ((parm (first condition)) (inst (second condition)) (op (third condition)) (val (fourth condition))) (case val (YES `(the ,inst ,op ,parm)) (NO `(the ,inst ,op not ,parm)) (T `(the ,parm of the ,inst ,op ,val))))))

(defun print-rule (rule &optional (stream t) depth) (declare (ignore depth)) (format stream "~&Rule ~a:~& If" (rule-number rule)) (print-conditions (rule-premises rule) stream) (format stream "~& Then ~a (~a) that" (cf->english (rule-cf rule)) (rule-cf rule)) (print-conditions (rule-conclusions rule) stream))

EMYCIN• MYCIN was developed in Lisp

– it was later determined that MYCIN was performing a task called Heuristic Classification

• EMYCIN (for empty MYCIN or essential MYCIN) was developed– as a pattern matching system that would mimic MYCIN’s problem solving

process without the domain specific rules– to build an expert system, one need only supply a new knowledge base (the

rules) and presto, new expert system• R1: configured VAX computers• Puff: pulmonary disorder diagnosis• GUIDON: tutorial system to teach students how to reason like MYCIN• SACON: structural engineering design and analysis advising

IF: THE MOST CURRENT ACTIVE CONTEXT IS ASSIGNING A POWER SUPPLYAND AN SBI MODULE OF ANY TYPE HAS BEEN PUT IN A CABINETAND THE POSITION IT OCCUPIES IN THE CABINET IS KNOWNAND THERE IS SPACE IN THE CABINET FOR A POWER SUPPLYAND THERE IS NO AVAILABLE POWER SUPPLYAND THE VOLTAGE AND FREQUENCY OF THE COMPONENTS IS KNOWN

THEN: FIND A POWER SUPPLY OF THAT VOLTAGE AND FREQUENCYAND ADD IT TO THE ORDER

R1 samplerule:

Beyond EMYCIN• Once rule-based systems had been introduced, a number

of programming languages were released that allowed quick and easy construction of rule-based systems– Often called Production System Languages because a rule-base

is a type of production system– Most of these languages supported either backward chaining or

forward chaining– OPS5: forward chaining, used to develop many expert systems,

included the ability to encode certainty factors or other forms of uncertainty (such as probabilities)

– Prolog: backward chaining, logic statements only (no mechanisms for uncertainty, no ability to represent NOT)

– SOAR: OPS5 + chunking (a rudimentary learning algorithm)– CLIPS: Written in C++ but looks like Lisp, forward and

backward chaining + salience (how useful a rule might be)– Jess: CLIPS re-written in Java with GUI capabilities

CLIPS Code(defrule advice18 (high mortgage-rate) => (assert (not (buy now))))

(defrule advice19 (and (rising house-prices) (not (high inflation))) => (assert (buy now)))

(defrule advice20 (high inflation) => (assert (high mortgage-rate)) (assert (rising house-prices)))

(defrule diagnose63 (and (parent ?p ?c) (allergy-risk ?p ?d)) => (assert (allergy-risk ?c ?d)))

(defrule diagnose64 (and (parent ?p ?c) (allergy-risk ?c ?d)) => (assert (allergy-risk ?p ?d)))

(defrule prescribe221 (and (infection gram-positive) (tolerable penicillin))) => (assert (indicated penicillin)))

(defrule check95 (not (allergic-to penicillin)) => (assert (tolerable penicillin)))

Critique of Pattern Matching• Advantages:

– Easy to construct (with an EMYCIN-like shell)– Easy to get knowledge in the form of rules

• Disadvantages:– Some knowledge is not necessarily in rule form– Many experts give inconsistent (or hesitate to provide) certainty factors

• The biggest problems though are– Expert systems tend to work well when the number of rules is below

10,000, but once a system has more than 10,000 rules, its efficiency and accuracy begins to deteriorate

– The knowledge is all distributed, finding related rules (to debug the knowledge base) is not easy

– Rules are typically at the same level, but knowledge comes in groupings • Some knowledge is not needed early in the diagnostic process (for instance,

treatment knowledge)• Other knowledge is not needed because the specific category being analyzed

has been ruled out (don’t need to know about Hepatitus if we have ruled out liver disease)

– Meta-knowledge: knowledge that will help you select what knowledge to apply or examine

Puff/Centaur• The Puff expert system performed pulmonary disorder

diagnosis– Implemented using rules and EMCYIN

• Another system, Centaur, used the same knowledge, but in a different way– Rules were grouped together into specialized agents– One agent per diagnostic conclusion– Conclusions were grouped into a hierarchy so that a generic

disease would be higher in the disease taxonomy and its children would be more specific instances

– Rather than thousands of rules, Centaur had dozens of agents (each implemented as an object)

– Each agent contained the knowledge necessary to diagnose that one conclusion

More on Centaur• A portion of Centaur’s

taxonomy is shown to the left

• Diseases were divided into more specific categories all the way down to the most specific such as– Severe Asthma– Mild Bronchitis

• Each object will contain the necessary knowledge– Rules: inference rules (specify how to determine values of clinical

parameters), triggering rules (to select other objects as necessary), fact-residual rules (to account for case data), refinement rules (how to continue down the hierarchy if necessary), summary rules (printout English description)

– Parameters: name, value, degree of certainty, source, classification, justification

– Meta-knowledge for control

Taking It Further• MYCIN demonstrated a task called Heuristic

Classification but confused matters by forcing all knowledge into rule form

• Centaur separated out the classification task from the rules

• Taking this further, we can clearly identify different tasks to perform during a problem solving process– We will call these tasks

• Thus, we might separate out the knowledge that allows us to categorize diseases from the knowledge that allows us to recognize a specific disease from the knowledge that allows us to explain why we believe a specific disease is responsible for the data– These low-level tasks have been dubbed Generic Tasks

Generic Tasks and AI Tools• Classes:

– Classifier (an entire hierarchy)– Classification Specialist (a single node in the hierarchy)– Recognition-Agent (a single set of rules to determine how

likely a given concept/hypothesis matches the data available)– Match-1-Recognition-Agent (an RA with multiple sets of rules)– Discrete-Pattern-Recognition-Agent (an RA with only a single

set of rules to match against)– Abducer (an object that knows how to explain data given

hypotheses that can explain the data)

• A define statement is used to create a new instance– These are macros that create a new class and fill in the various

class slots appropriately, including the code to execute to make the object run

– Code to examine is available on the website

GT ToolsGeneric Task Class

DefinitionSlotSlotSlot…

Compile-class Macro

Fills slots in withproper information

and/or code

Expert Knowledge Defintions

CLOS Objects generated to solve the problem

(define-classification-specialist FuelSystem (display-name= "Fuel System Specialist") (establish-reject= (judge FuelSystemSummary)) (classifier= AutoMechSystem) (super-specialists= AutoMech) (sub-specialists= Delivery Mixture Vacuum AirIntake BadFuel) (creation-date= "5 July 1988") (last-modification-date= "5 July 1988") (author= "John D. McElroy"))

Sample code