1

PERSIVAL

a System for Personalized Search and Summarization over Multimedia

Information

PERSIVAL

a System for Personalized Search and Summarization over Multimedia

Information

2

PERSIVAL team membersPERSIVAL team members Medical InformaticsMedical Informatics

James Cimino, Carol Friedman, Steven JohnsonJames Cimino, Carol Friedman, Steven Johnson

Medical School – cardiac anesthesiologyMedical School – cardiac anesthesiology Desmond JordanDesmond Jordan

Computer ScienceComputer Science Steven Feiner, Luis Gravano, Vasileios Hatzivassiloglou, Kathleen McKeownSteven Feiner, Luis Gravano, Vasileios Hatzivassiloglou, Kathleen McKeown

Electrical EngineeringElectrical Engineering Shih-Fu ChangShih-Fu Chang

Center for Research on Information Access, Health Sciences Center for Research on Information Access, Health Sciences LibraryLibrary

Judith Klavans, Pat Molholt, Elizabeth LaRue, David MillmanJudith Klavans, Pat Molholt, Elizabeth LaRue, David Millman

Cognitive ScienceCognitive Science Andre Kushniruk (York), Vimla Patel (Medical Informatics)Andre Kushniruk (York), Vimla Patel (Medical Informatics)

3

StudentsStudents

Computer ScienceComputer Science Eugene AgichteinEugene Agichtein Michel GalleyMichel Galley Noemie ElhadadNoemie Elhadad Panos IpeirotisPanos Ipeirotis

Medical InformaticsMedical Informatics Michael Charney (programmer)Michael Charney (programmer) Eneida MendoncaEneida Mendonca Lyudmila Shagina (programmer)Lyudmila Shagina (programmer)

Electrical EngineeringElectrical Engineering Shahram EbadollahShahram Ebadollah

Min-Yen KanMin-Yen Kan Simon LokSimon Lok Smaranda MuresanSmaranda Muresan

Sergey SigelmanSergey Sigelman (programmer)(programmer)

Yoon –Ho SeolYoon –Ho Seol Di WangDi Wang

4

GoalsGoals

Personalized access to distributed, multimedia Personalized access to distributed, multimedia resourcesresources

information access information access information fusioninformation fusion information understandinginformation understanding

Provision of patient-specific informationProvision of patient-specific information interaction within contextinteraction within context for clinicians, at the point of patient carefor clinicians, at the point of patient care for patients, in terms that can be understoodfor patients, in terms that can be understood online patient record serves as a user modelonline patient record serves as a user model

5

RoundsRounds

Patient-centricPatient-centric Current: Access Current: Access

to clinical datato clinical data Missing: Missing:

Access to Access to literature that literature that fits patient fits patient profileprofile

6

Unique ContributionsUnique Contributions

System focus: querying, search, presentationSystem focus: querying, search, presentation Questions are asked within the context of patient Questions are asked within the context of patient

informationinformation A uniform, personalized view of distributed resources A uniform, personalized view of distributed resources

on the internet through querying and browsingon the internet through querying and browsing Concise, patient specific presentation of relevant Concise, patient specific presentation of relevant

information through summarizationinformation through summarization Access to textual documents linked with access to Access to textual documents linked with access to

multimedia video: library of echocardiogrammultimedia video: library of echocardiogram Dynamic layout of heterogeneous informationDynamic layout of heterogeneous information

7

Where are we now?Where are we now? Prototypes of each system componentPrototypes of each system component Local library of journal articles and consumer health Local library of journal articles and consumer health

sitessites 20 highly ranked journals20 highly ranked journals 30,000 articles30,000 articles

Facilities for distributed online searchFacilities for distributed online search Scenarios for development and testing with three Scenarios for development and testing with three

patientspatients Initial system integrationInitial system integration

Restricted to a limited set of examplesRestricted to a limited set of examples

Formative evaluation of system componentsFormative evaluation of system components

8

Overall Integrated DemoOverall Integrated Demo

What is the prognosis for atrial fibrillation and What is the prognosis for atrial fibrillation and myocardial infarction?myocardial infarction?

Clinician as user Clinician as user On viewing patient discharge summaryOn viewing patient discharge summary Journal articles: controlled clinical trialsJournal articles: controlled clinical trials Re-ranking of search results using patient recordRe-ranking of search results using patient record

What is the treatment for endocarditis? What is the treatment for endocarditis? Patient as user Patient as user On viewing lab resultsOn viewing lab results Consumer health informationConsumer health information

11

User Interface FocusUser Interface Focus

Asking questions within context of patient Asking questions within context of patient recordrecord

Evidence based medicine to suggest Evidence based medicine to suggest questionsquestions

Selection of relevant information from the Selection of relevant information from the patient recordpatient record

Demo of MedleeDemo of Medlee

13

Distributed SearchDistributed Search

Meta-searcher for automated interaction with Meta-searcher for automated interaction with heterogeneous, distributed sourcesheterogeneous, distributed sources

Use of machine learning and query probes to Use of machine learning and query probes to automatically determine topics of distributed automatically determine topics of distributed sourcessources

Information extraction from web pagesInformation extraction from web pages

15

Re-ranking search resultsRe-ranking search results

Re-rank articles which better match the Re-rank articles which better match the patient record -> more relevant articles patient record -> more relevant articles

Use natural language techniques to analyze Use natural language techniques to analyze article and patient recordsarticle and patient records

Articles with many terms and values Articles with many terms and values matching the patient record score highermatching the patient record score higher

17

Presentation FocusPresentation Focus

Multimedia summarizationMultimedia summarization Journal articles, consumer health, videoJournal articles, consumer health, video Highlight retrieved results to help user in finding relevant Highlight retrieved results to help user in finding relevant

informationinformation Personalize summary for patientPersonalize summary for patient Define unknown terminology Define unknown terminology Methods for summarizing and search echocardiogramsMethods for summarizing and search echocardiograms

Dynamic layout and organization of resultsDynamic layout and organization of results Explicitly control level of detailExplicitly control level of detail

18

MilestonesMilestones

Where we said we would be vs. where we are:Where we said we would be vs. where we are: Year 2: Year 2: skeletal end-to-end system prototypeskeletal end-to-end system prototype with with minimal minimal

personalization, interactivitypersonalization, interactivity, and , and limited coverage of structured limited coverage of structured documentsdocuments

Year 3: Year 3: Extend to full prototypeExtend to full prototype, with , with increased personalizationincreased personalization, , interactivityinteractivity, , limited coordination of multimedialimited coordination of multimedia, , full range of full range of structured documentsstructured documents, and , and restricted coverage of consumer documentsrestricted coverage of consumer documents

Use of evidence-based medicine, machine learning to categorize Use of evidence-based medicine, machine learning to categorize sources by topic, provision of definitions, thin-client computing to sources by topic, provision of definitions, thin-client computing to allow PERSIVAL on mobile, hand-held devicesallow PERSIVAL on mobile, hand-held devices

Year 4: Scale prototype with increased robustness, personalization, Year 4: Scale prototype with increased robustness, personalization, coverage to full range of documents and fully integrated multimedia. coverage to full range of documents and fully integrated multimedia. Coordinate with end-to-end evaluationCoordinate with end-to-end evaluation

Year 5: Refine components based on Year 4 evaluation. Transition Year 5: Refine components based on Year 4 evaluation. Transition PERSIVAL to deployment in cooperation with Health Sciences PERSIVAL to deployment in cooperation with Health Sciences LibraryLibrary

19

Plans for next yearPlans for next year

Increase robustnessIncrease robustness Extend question asking to different patient contexts, different Extend question asking to different patient contexts, different

question typesquestion types Allow summarization and re-ranking of online articlesAllow summarization and re-ranking of online articles Extend journal summarization to new genresExtend journal summarization to new genres Extend layout to dynamically incorporate different types of Extend layout to dynamically incorporate different types of

summary inputsummary input

Multimedia integrationMultimedia integration Implement scenarios for integration Implement scenarios for integration Increase interaction with video summary in layout Increase interaction with video summary in layout Enhanced multimedia prototypeEnhanced multimedia prototype