automated classification of medical questions using semantic parsing techniques paul e. pancoast, md...
TRANSCRIPT
Automated Classification of Medical Questions Using
Semantic Parsing Techniques
Paul E. Pancoast, MDArthur B. Smith, MSChi-Ren Shyu, PhD
University of Missouri-Columbia
Physicians Have Questionswhen they treat patients
• What is the best treatment for migraines in patients who are diabetic?
• How often should I repeat the TSH for this patient who is on synthroid?
• When should I get an X-ray for this patient with low back pain?
Observational Studies of Physician Information Needs
• Covell – 1985 – Annals of Internal Medicine. Oct 1985;103(4):596-599.
• Osherhoff – 1992– Annals of Internal Medicine. Apr 1 1991;114(7):576-581.
• Gorman – 1994 – Medical Decision Making. Apr-Jun 1995;15(2):113-119.
• Ely – 1999– BMJ. Aug 12 2000;321(7258):429-432.
Common Themes from Observational Studies
• Physicians have questions for 45-65% of all patients they see
• Physicians pursue only about 30% of those questions
• Physicians find answers to 80% of the questions they pursue
Collections of Questions
• Over 10,000 question strings collected – NLM, Ely, Vanderbilt, Duke, FPIN, U of
Washington, Britain, Australia
• No good way to classify the questions• No automated method of finding duplicate
questions
Reasons to Automate Classification
• Organize collections of questions• Improve accuracy of existing classification• Find redundancy (duplicate questions) • Find frequency of occurrence
Research GoalAutomate Classification of Medical Questions• Question Type – based on semantic and
syntactic information (this experiment)• Question Meaning – based on the specific
instantiations of semantic and syntactic information (subsequent experiments)
• Ultimately – to match questions directly with structured medical information
Study Overview
MU
Ely
1101 Specific Questions
Dom
ain
Exper
ts
64 Categories170 Generic Question
Strings
Sem
antic
Gro
upSe
quen
ce
Patte
rns
Automated Classification
Of Specific Questions
Ely Taxonomy
• Generic Category (64 total)– 1111
• Generic Question Strings (GQS)– What is the cause of symptom x?– What is the differential diagnosis of symptom x?– Could symptom x be condition y or be a result of
condition y?– What is the likelihood that symptom x is coming
from condition y?
Methods for this study(overview)
1. Extracted medical concepts from question strings using UMLS MRXNS table
2. Assigned concept unique identifier (CUI) to Semantic Groups
3. Found Semantic Group Sequence (SGS) patterns using Apriori Algorithm (modified)
4. Matched SGS from specific questions to SGS in Ely’s generic question strings to assign the generic category
1. Extracted CUIs from question strings
3 word, 2 word, 1 word window parser matching strings to MRXNS
– [How should I] treat acute pharyngitis?– How [should I treat] acute pharyngitis?– How should [I treat acute] pharyngitis?– How should I [treat acute pharyngitis]?
1. Extracted CUIs from question strings
How should I treat [acute pharyngitis]?– Acute pharyngitis => UMLS semantic type T047
• Disease or Syndrome
How should I [treat] ------------?– Treat (treatment) => UMLS semantic type T061
• Therapeutic or Preventative Procedure
2. Assigned CUIs to Semantic Groups
• Semantic Groups are aggregations of similar semantic types
• 27 Semantic Groups (from UMLS Semantic Network)– T047 is in 017 (PATH-PROC)– T061 is in 027 (THER)
• 39 additional, non-medical Semantic Groups (derived from general thesauri)
3. Found Semantic Group Sequence (SGS) patterns
• Example question:– How should I treat acute pharyngitis– 253 | 250 | 242 | 27 | 17 |
• 253 – How/Why• 250 – Does/Can/Could/Should• 242 – I/You/He/She/We• 27 – treat (treatment)• 17 – acute pharyngitis
• Ran 3000 question strings through the parser and looked for recurrent patterns
3. Found Semantic Group Sequence (SGS) patterns
• Example question:– How should I treat acute pharyngitis
253 250 242 27 17Matching patterns:Semantic types Support / Confidence 253 250 242 27 0.0398 0.5588253 242 27 0.0409 0.5612253 250 27 0.0477 0.5084253 250 242 0.0712 0.7598236 17 0.0613 0.5493253 27 0.0691 0.5077253 242 0.0728 0.5346253 250 0.0938 0.6885
3% occurrence for support50% incidence for confidence
3. Found Semantic Group Sequence (SGS) patterns
Support
the pattern of SGS occurs in at least 3% of all the questions parsed
Confidence
253 250 242 27 occurs 50% of the time when 253 250 242 is found
4. Matched SGS patterns in generic and specific questions
• Generic question:– How should I treat condition y?
Specific questions with some matching SGS patterns
– How do I treat depression?– How do I manage Parkinsonism? – How do I treat acne? – How do I treat conjunctivitis? – How do I treat dementia? – How do I treat STD’s?
Results• 1101 specific questions• 20,710 total words• 867 (2804 instances) did not match in MRXNS
or in MRCON (MRXNW gave too many hits)• The majority of these strings were mapped to
an existing semantic type using ad-hoc stemming techniques
Results• 7183 SGS patterns matched in specific and
generic questions• 204 (18%) specific questions had potential
matches with generic questions (using SGS)• 97 (10%) actual matches between specific
and generic questions (using domain expert)• 67 of these (using SGS) matched the same
category assigned by Dr. Ely
Discussion• 6% of specific question strings mapped to the
generic category assigned by Ely (67/1101)• 33% of those predicted to match by SGS
patterns had matching generic categories• 73% of specific question strings didn’t map to
any generic category• 45% of specific question strings (that did map
to a generic category) mapped to more than one generic category
Discussion• Automatic Classification of Questions
– SGS pattern matching can cluster questions with similar semantic and syntactic information
– These clustered questions often have the same meaning
• Discrepancy in classification – SGS and Ely– Our model needs work– Ely classifications are not semantically-based– Ambiguity in Ely classifications
Why Questions Didn’t Match Categories
Generic Category
Assigned Category
Specific Question
Diagnosis 1111 What is the differential diagnosis of a rash?
Diagnosis 1121 What is the differential diagnosis of a rash?
35 questions are assigned to more than one category
Future Work• Improve accuracy of model
– Refine Semantic Groups – Use relevance feedback and Semantic Group
weighting – Include part-of-speech tagging and syntactic
parsing– Incorporate WordNet for non-medical terms
• Develop an indexing schema that represents the semantic groups and syntactic information as vectors in a high-level feature space model