artificial intelligence in medical ... › 2018meeting › handouts › allen_ai_value.pdf · •...
Post on 27-Jun-2020
4 Views
Preview:
TRANSCRIPT
© 2017 | DATA SCIENCE INSTITUTE™: AMERICAN COLLEGE OF RADIOLOGY | ALL RIGHTS RESERVED.
<<November 3, 2018>>
Artificial Intelligence In Medical Imaging: Leveraging The Value of Radiology Professionals
Bibb Allen, MD FACR
Chief Medical Officer
ACR Data Science Institute
© 2017 | DATA SCIENCE INSTITUTE™: AMERICAN COLLEGE OF RADIOLOGY | ALL RIGHTS RESERVED.
Disclosures: Dr. Allen has no relationships with ACCME-defined commercial interests.
Affiliations:
• Chief Medical Officer American College of Radiology Data Science Institute
• Former Board Chair and President ACR
Acknowledgements:
• Keith Dreyer, DO, PhD Chief Science Officer American College of Radiology Data Science Institute and Chair ACR Commission on Informatics
• ACR Data Science Institute team
CONFLICTS
© 2017 | DATA SCIENCE INSTITUTE™: AMERICAN COLLEGE OF RADIOLOGY | ALL RIGHTS RESERVED.
1. Introduction & Background
2. Intro to AI and Imaging
3. The Value of AI in Imaging
4. The Challenges of Using AI in
Image Interpretation
5. Augmented Intelligence for
Radiologists
6. The ACR Data Science Institute
(DSI)
7. Case Study of AI in Image
Interpretation
8. An Example of a DSI Use Case
9. Concluding Slides & Discussion
Agenda
© 2017 | DATA SCIENCE INSTITUTE™: AMERICAN COLLEGE OF RADIOLOGY | ALL RIGHTS RESERVED.
Innovation in healthcare can be lightning fast but often fraught with errors and missed opportunities that cost lives.
X-Ray as a Case Study in Rapid Adoption
© 2017 | DATA SCIENCE INSTITUTE™: AMERICAN COLLEGE OF RADIOLOGY | ALL RIGHTS RESERVED.
Röntgen Tesla
1895
© 2017 | DATA SCIENCE INSTITUTE™: AMERICAN COLLEGE OF RADIOLOGY | ALL RIGHTS RESERVED.
Wilhelm Roentgen discovers X-rays Wuerzburg Germany - 1895
First x-ray image in Ohio 4 months later
© 2017 | DATA SCIENCE INSTITUTE™: AMERICAN COLLEGE OF RADIOLOGY | ALL RIGHTS RESERVED.
1st description of Physicists and physician partnership
in imaging research?
Thanks to Dr. Jeffrey Duerk
Diagnostic Imaging History
4 months later Kenyon College, Ohio judge- nail in hand.
1896
Hand of Roentgen’s wife.*Note wedding ring
1895
© 2017 | DATA SCIENCE INSTITUTE™: AMERICAN COLLEGE OF RADIOLOGY | ALL RIGHTS RESERVED.
No innovation is without risk or unintended consequences!
© 2017 | DATA SCIENCE INSTITUTE™: AMERICAN COLLEGE OF RADIOLOGY | ALL RIGHTS RESERVED.
© 2017 | DATA SCIENCE INSTITUTE™: AMERICAN COLLEGE OF RADIOLOGY | ALL RIGHTS RESERVED.
Improving the health of
populations
Improving the individual
experience of care
Reducing the per capita
costs of care
Improving the work life of those who deliver care
ESTABLISHING A VALUE PROPOSITION HEALTHCARE REFORM AND FOR RADIOLOGY
Berwick DM, Nolan TW, Whittington J. The triple aim: care, health, and cost. Health
affairs. 2008 May;27(3):759-69.
Bodenheimer T, Sinsky C. From triple to quadruple aim: care of the patient requires care of
the provider. The Annals of Family Medicine. 2014 Nov 1;12(6):573-6.
May, 2008
The Quadruple Aim
What Does It Mean For
Radiology?
Why do AI if not to improve quality and value?
© 2017 | DATA SCIENCE INSTITUTE™: AMERICAN COLLEGE OF RADIOLOGY | ALL RIGHTS RESERVED.
3 Key Actions:
Imaging 3.0 is a vision and game plan for providing optimal imaging care.
IMAGING 3.0: VALUE-BASED RADIOLOGY
11
“Our goal is to deliver all the imaging care that is beneficial and necessary and none that is not.”
Culture Change
Portfolio of IT Tools
Alignment of Incentives
© 2017 | DATA SCIENCE INSTITUTE™: AMERICAN COLLEGE OF RADIOLOGY | ALL RIGHTS RESERVED.
3 Key Actions:
Imaging 3.0 is a vision and game plan for providing optimal imaging care.
IMAGING 3.0: VALUE-BASED RADIOLOGY
12
Culture Change
Portfolio of IT Tools
Alignment of Incentives
Clinical Decision Support for Ordering PhysiciansProviding >24 Million examinations per month
Structured ReportingIncorporated in all VR reporting platforms
Artificial Intelligence
RegistriesRadiation Exposure / Patient Outcomes / Quality
Image SharingRSNA / NIH / Vendors
Clinical Decision Support for Image InterpretationIntegrated into >75% of radiologists desktops
IMAGING
PROTOCOL
CLINICAL
CARE
INTERPRET
RADIOLOGY AI DEVELOPMENT CYCLE
Detection, Segmentation, Quantification, ClassificationQA, Workflow, Hanging Protocols, Priors Management
Artifact Reduction, Findings OptimizationDose and Contrast Optimization
Patient Positioning, Exam Protocolling,Priors Management
Exam Clinical Decision SupportPopulation Health Management
INTERPRETATION
IMAGE ACQUISITION
PRE-ACQUISITION
CLINICAL CAREAI
USE CASES
AIUSE CASES
AIUSE CASES
AIUSE CASES
There is currently a limited use of AI in clinical care
Why?
While there is significant research and preliminary applications of AI in healthcare…
HEALTHCARE DATA SCIENCE CHALLENGES
USE CASE
DEFINITION
DATA
ENGINEERING
AI MODEL
CREATION
AI/HUMAN
INTERFACE
REGULATORYBUSINESS MODEL
AI MODEL
VALIDATION
USER ADOPTIONCONTINUOUS LEARNING
CLINICAL
INTEGRATION
© 2017 | DATA SCIENCE INSTITUTE™: AMERICAN COLLEGE OF RADIOLOGY | ALL RIGHTS RESERVED.
Infrastructural Components
Applications and Services
Third-party Applications and Services
Apple Ecosystem
© 2017 | DATA SCIENCE INSTITUTE™: AMERICAN COLLEGE OF RADIOLOGY | ALL RIGHTS RESERVED.
© 2017 | DATA SCIENCE INSTITUTE™: AMERICAN COLLEGE OF RADIOLOGY | ALL RIGHTS RESERVED.
Ecosystem Primary Beneficiary Other Beneficiaries
Apple Apple’s Shareholders Apple Users, Apple Partners and Employees
AI in Medical Imaging Patients Those who serve the patients (Providers, Vendors, Insurers, Regulators, Associations/Societies, Doctors)
© 2017 | DATA SCIENCE INSTITUTE™: AMERICAN COLLEGE OF RADIOLOGY | ALL RIGHTS RESERVED.
© 2017 | DATA SCIENCE INSTITUTE™: AMERICAN COLLEGE OF RADIOLOGY | ALL RIGHTS RESERVED.
The Imaging Lifecycle
Referring PhysicianImaging Modality
Operations
Appropriateness Determination and Patient Scheduling
Imaging Protocol Optimization
CommunicationInterpretation and
Reporting
Data Mining & Business Intelligence
Patient
© 2017 | DATA SCIENCE INSTITUTE™: AMERICAN COLLEGE OF RADIOLOGY | ALL RIGHTS RESERVED.
Validation
Payment Models
Fear of change
Hype that can’t live up to reality
Clinical Integration
Complex AI and medical informatics
The path doesn’t look like an expedition party climbing Mt. Everest.
Regulation
Data Engineering
© 2017 | DATA SCIENCE INSTITUTE™: AMERICAN COLLEGE OF RADIOLOGY | ALL RIGHTS RESERVED.
DATA SCIENCE AND ARTIFICIAL INTELLIGENCE ECOSYSTEM
© 2017 | DATA SCIENCE INSTITUTE™: AMERICAN COLLEGE OF RADIOLOGY | ALL RIGHTS RESERVED.
Infrastructural Components
Applications and Services
Third-party Applications and Services
Apple Ecosystem
© 2017 | DATA SCIENCE INSTITUTE™: AMERICAN COLLEGE OF RADIOLOGY | ALL RIGHTS RESERVED.
© 2017 | DATA SCIENCE INSTITUTE™: AMERICAN COLLEGE OF RADIOLOGY | ALL RIGHTS RESERVED.
Ecosystem Primary Beneficiary Other Beneficiaries
Apple Apple’s Shareholders Apple Users, Apple Partners and Employees
AI in Medical Imaging Patients Those who serve the patients (Providers, Vendors, Insurers, Regulators, Associations/Societies, Doctors)
© 2017 | DATA SCIENCE INSTITUTE™: AMERICAN COLLEGE OF RADIOLOGY | ALL RIGHTS RESERVED.
ACR DSI MISSION
Leverage the value of radiology professionals as AI evolves through the development of appropriate use cases and workflow integration
Protect patients through leadership roles in the regulatory process with government agencies and verification of algorithms
Establish industry relationships by providing credible use cases, help with FDA and other government agencies, and pathways for clinical integration
Educate radiology professionals, other physicians and all stakeholders about AI and the ACR’s role in data science for the good of our patients
AIECOSYSTEM
EDUCATION
http://acrdsi.org/media-library/pdf/Strategic-Plan-Final.pdf
AIECOSYSTEM
© 2017 | DATA SCIENCE INSTITUTE™: AMERICAN COLLEGE OF RADIOLOGY | ALL RIGHTS RESERVED.
Advance data science as core to clinically relevant,
safe and effective radiologic care
• Establish the ACR as a global leader in advancing appropriate data science solutions
• Define, communicate and educate about the benefits of data science for all radiology professionals, patients and the greater community
• Facilitate the development of AI solutions that are free of unintentional bias
• Ensure that data science integrates into all facets of the ACR
• Develop external relationships that support and extend the ACR’s data science goal
• Promote radiology medical education that includes the skills needed to adapt to and implement data science solutions
The ACR Strategic Plan For Data Science Adopted October 2017
© 2017 | DATA SCIENCE INSTITUTE™: AMERICAN COLLEGE OF RADIOLOGY | ALL RIGHTS RESERVED.
Establish the ACR as a global leader in advancing appropriate data science solutions
• The ACR DSI led the development of a collaboration between RSNA, ESR, RANZCR, CAR, and other organizations to develop joint statements and other position papers, as well as shared activities to advance AI for the benefit of radiologists and our patients
• Keith Dreyer presented information about the ACR DSI at the 2018 European Congress of Radiology (ECR)
• Contributed a textbook chapter, “The Role of an AI Ecosystem” for Springer’s Artificial Intelligence in Medical Imaging: Opportunities, Applications and Risks, edited by Erik Ranschaert and Sergey Morozov. Authors include an array of international AI experts
• The ACR DSI has a memorandum of understanding with Medical Image Computing and Computer Assisted Intervention (MICCAI) to provide educational content and support including the use of ACR DSI Use Cases for activities such as AI challenges.
ACR DSI AND THE ACR STRATEGIC PLAN FOR DATA SCIENCE
© 2017 | DATA SCIENCE INSTITUTE™: AMERICAN COLLEGE OF RADIOLOGY | ALL RIGHTS RESERVED.
Define, communicate and educate about the benefits of data science for all radiology professionals, patients and the greater community
• Presentations at multiple national meetings including RSNA, ARRS, SIIM and ACR annual meetings
• Collaboration with RLI for presentations at state chapters and at the ACR-RBMA Practice Leaders Forum (January 2019)
• Collaborations with ACR Quality and Safety
• Presentations at state chapter meeting
• JACR– Data Science and Radiological Science Column
– JACR Special Issue on Artificial Intelligence
– Contribution to JACR Special issue on Health Equity
– JACR Special Issue on Data Science and Quality for July 2019
• AI Journal Advisor to begin Fall 2018
ACR DSI AND THE ACR STRATEGIC PLAN FOR DATA SCIENCE
© 2017 | DATA SCIENCE INSTITUTE™: AMERICAN COLLEGE OF RADIOLOGY | ALL RIGHTS RESERVED.
NIH / NIBIB WORKSHOP ON AI IN MEDICAL IMAGING
© 2017 | DATA SCIENCE INSTITUTE™: AMERICAN COLLEGE OF RADIOLOGY | ALL RIGHTS RESERVED.
NIH / NIBIB WORKSHOP ON AI IN MEDICAL IMAGING
© 2017 | DATA SCIENCE INSTITUTE™: AMERICAN COLLEGE OF RADIOLOGY | ALL RIGHTS RESERVED.
Facilitate the development of AI solutions that are free of unintentional bias
• ACR DSI Senior Scientist Raym Geis is heading up the ACR DSI ethics in AI project with multiple other organizations
• ACR DSI Use Cases and validation process designed to obtain training and validation across a diverse range of practices
• JACR Special Issue on Health Equity
Ensure that data science integrates into all facets of the ACR
• RLI programming as above
• Quality and Safety programming as above
• Economics program “Economics of AI” at SIIM 2018
• Collaboration with Patient and Family Centered Care Commission around issue such as data sharing and other issues regarding ethics of AI
ACR DSI AND THE ACR STRATEGIC PLAN FOR DATA SCIENCE
© 2017 | DATA SCIENCE INSTITUTE™: AMERICAN COLLEGE OF RADIOLOGY | ALL RIGHTS RESERVED.
Promote radiology medical education that includes the skills needed to adapt to and implement data science solutions
• ACR leaders are participating in the RSNA/SIIM informatics curriculum for residents (https://imaging-informatics-course.appspot.com/niic/)
• Kathy Andriole is leading an education effort for ACR members about understanding and integrating AI into clinical practice
ACR DSI AND THE ACR STRATEGIC PLAN FOR DATA SCIENCE
© 2017 | DATA SCIENCE INSTITUTE™: AMERICAN COLLEGE OF RADIOLOGY | ALL RIGHTS RESERVED.
• Lead - ACR to become a global leader in data science– Do the right thing: Advance data science solutions that are appropriate and free of unintentional bias
– Believe in what we do: Integrate data science into all facets of the ACR
• Define - ACR to define the beneficial uses of data science in radiology– Standards: Create standard methodologies to expand beneficial radiological data science throughout healthcare
– Relationships: Develop external relationships to support and extend our data science goals
• Educate - ACR to educate on the use of appropriate data science in radiology – Promote: Radiology medical education that includes the skills needed to adapt and implement data science solutions
– Socialize: With radiology professionals, patients and the greater community on the value of beneficial data science
The ACR Strategic Plan For Data Science - Adopted Oct, 2017
Advance data science as core to clinically relevant, safe and effective radiologic care
Lead, Define, Educate
© 2017 | DATA SCIENCE INSTITUTE™: AMERICAN COLLEGE OF RADIOLOGY | ALL RIGHTS RESERVED.
AIMODELS
DATA
ENGINEERING
AICONCEPTS
AIAPPLICATIONS
AI DEVELOPMENT CYCLE
ACR DATA SCIENCE INSTITUTE
AIMODELS
DATA
ENGINEERING
AICONCEPTS
AIAPPLICATIONS
ACR DATA SCIENCE INSTITUTE
Define standard methods toaggregate and annotate
data for AI model training and testing
Define standard methods to integrate and monitor, AI models in clinical practice
Define standards for Use Cases considering
clinical needs and technical capabilities
Define standardizedmethods for AI model
validation consistent with regulatory processes
ACR TOUCH-AI
ACR ASSESS-AI ACR CERTIFY-AI
ACR DATA-AI
© 2017 | DATA SCIENCE INSTITUTE™: AMERICAN COLLEGE OF RADIOLOGY | ALL RIGHTS RESERVED.
DSI SUPPORT STAFF
Chris Treml
ACR DSI Director of Operations
Current Staffing• Director of Operations• Clinical informatics analyst• 2 data engineers• Data science analyst• Communications specialist
SCALABLE
© 2017 | DATA SCIENCE INSTITUTE™: AMERICAN COLLEGE OF RADIOLOGY | ALL RIGHTS RESERVED.
Structured Data Elements
• Training and testing AI algorithms
• Validating AI algorithms
• Monitoring algorithms in clinical practice
AI Use Case Standard• Authored by experts, used by machines• Converts human language to machine readable language• Open source authoring platform• Trusted partnerships with industry and regulators• Ensure patient safety
MOVING CLINICALLY EFFECTIVE AI USE CASES TO CLINICAL PRACTICE
ACR® TOUCH-AITOUCH-AI Technically Oriented Use Cases for Healthcare AI
ACR ASSESS-AI ACR CERTIFY-AI
ACR DATA-AIACR TOUCH-AI
The Radiology AI EcosystemIdeas To Clinical Practice
WHAT SHOULD DEVELOPERS BUILD?
EXPERT PANELS, PRIORITIZE CLINICAL NEEDS, TECHNICAL SPECS, DATA
PARAMETERS, PUBLIC INPUT
© 2017 | DATA SCIENCE INSTITUTE™: AMERICAN COLLEGE OF RADIOLOGY | ALL RIGHTS RESERVED.
The Radiology AI EcosystemIdeas To Clinical Practice
RadElement.org
ACR TOUCH-AI
ACR ASSESS-AI ACR CERTIFY-AI
ACR DATA-AI
Common Data Elements (CDEs)
STANDARDIZATION AND COMMON DATA ELEMENTS
COLLABORATION, STANDARDIZATION, INTEROPERABLE
© 2017 | DATA SCIENCE INSTITUTE™: AMERICAN COLLEGE OF RADIOLOGY | ALL RIGHTS RESERVED.
ACR ASSESS-AI ACR CERTIFY-AI
ACR DATA-AIACR TOUCH-AI
The Radiology AI EcosystemIdeas To Clinical Practice
ACR DSI DATA SCIENCE SUBSPECIALTY PANELS: ACR DSI USE CASE CREATION
ACR DSI USE CASES: AUTHORED BY EXPERTS – USED BY MACHINES
© 2017 | DATA SCIENCE INSTITUTE™: AMERICAN COLLEGE OF RADIOLOGY | ALL RIGHTS RESERVED.
• Over 40 use cases near completion• Currently undergoing preliminary industry review• Plan to publish October 2018TOUCH-AI
ACR DSI AI USE CASE DIRECTORY
© 2017 | DATA SCIENCE INSTITUTE™: AMERICAN COLLEGE OF RADIOLOGY | ALL RIGHTS RESERVED.
ACR TOUCH-AI
STANDARD SPECIFICATIONS FOR DATA ACCESS
• TOUCH-AI allows multiple institutions to create datasets that developers can use for algorithm training and testing
• The ACR DSI will house a freely available public directory of institutions that have created these datasets
• Using multiple sites provides technical, geographic and patient diversity to prevent unintended bias in algorithm development
• Allows more individuals and institutions to participate in AI development
ACR DSI AI-Data Directory
A Directory of Datasets For AI Training Available To Developers
AI-Data Directory
ACR ASSESS-AI ACR CERTIFY-AI
The Radiology AI EcosystemIdeas To Clinical Practice
ACR DATA-AI
HOW DO WE MAKE IT?RESOURCES FOR AI DEVELOPERS
STANDARDIZATION, DIVERSITY, AVAILABILITY
© 2017 | DATA SCIENCE INSTITUTE™: AMERICAN COLLEGE OF RADIOLOGY | ALL RIGHTS RESERVED.
DATA SCIENCE AND HEALTH EQUITY
• AI algorithms outside of healthcare have been shown to incorporate ethnic, gender and social bias
• The physician community should work with developers and regulators develop pathways to ensure algorithms marketed for widespread clinical practice are safe, effective and free of unintended bias
• Structured use cases with standards for developing datasets for training and testing
• ACR DSI validation and monitoring services, ACR Certify-AI and ACR Assess-AI, incorporate standards to mitigate algorithm bias and promote health equity
• Work with the payer and developer communities to ensure payment models for AI do not limit access to AI tools based on the socioeconomic status of our patients or the resources of their health systems.
March 2019
Health Equity
WHAT ELSE CAN THE DSI TO PROMOTE HEATH EQUITY?
© 2017 | DATA SCIENCE INSTITUTE™: AMERICAN COLLEGE OF RADIOLOGY | ALL RIGHTS RESERVED.
ACR TOUCH-AI
ACR Certify-AI
VALIDATING AI ALGORITHMS
Specifications For Algorithm Validation and Certification
• Centralized performance assessment of AI algorithm according to statistical metrics specified in the TOUCH-AI use case
• Embargoed validation datasets are created at multiple institutions to ensure geographic, technical and patient diversity
• Guidelines for data quality to ensure “ground truth” consistency between sites
• Reports are generated for developers, agencies and customers
ACR ASSESS-AI
ACR DATA-AI
The Radiology AI EcosystemIdeas To Clinical Practice
ACR CERTIFY-AI
HOW DO WE VALIDATE AI ALGORITHMS
FOR MARKETING IN CLINICAL PRACTICE?
EVALUATION METHODOLOGY, HIGHLY DIVERSE DATA, REPRODUCIBLE, HONEST
BROKER, COMPARABLE,
© 2017 | DATA SCIENCE INSTITUTE™: AMERICAN COLLEGE OF RADIOLOGY | ALL RIGHTS RESERVED.
CERTIFY-AI WORKFLOW
GEOGRAPHIC AND TECHNICAL DIVERSITY IN VALIDATION DATASETS
© 2017 | DATA SCIENCE INSTITUTE™: AMERICAN COLLEGE OF RADIOLOGY | ALL RIGHTS RESERVED.
PROCESS FOR ESTABLISHING PERFORMANCE OF AN AI ALGORITHM
Step Description MDDT Tool Pneumothorax Example
1 Define the use case, specifying the trigger, the measure and,
and the clinical context
TOUCH-AI Presence or absence of
Pneumothorax
2 Identify sources of variability in the algorithm’s measurements TOUCH-AI TAI-THOR0000118
3 Determine the performance metrics critical to the specific
clinical role
Certify-AI CIs for sensitivity and
specificity
4 Identify the reference data set for evaluation Certify-AI CAI- THOR00001
5 Define the minimum acceptance criteria for the metrics
identified in step 3
Certify-AI Lower bound for sensitivity is
>0.95 and the lower bound for
specificity is >0.90
6 Test the algorithm’s performance using criteria defined in step 5 Certify-AI Report See example
VALIDATION METRICS DEVELOPED AS PART OF EACH USE CASE
© 2017 | DATA SCIENCE INSTITUTE™: AMERICAN COLLEGE OF RADIOLOGY | ALL RIGHTS RESERVED.
CERTIFY-AI VALIDATION REQUIREMENTS
ACR DSI AI USE CASES SET REQUIREMENTS FOR THE ALGORITHM AND VALIDATION PROCESS
© 2017 | DATA SCIENCE INSTITUTE™: AMERICAN COLLEGE OF RADIOLOGY | ALL RIGHTS RESERVED.
ACCOUNTING FOR SOURCES OF VARIABILITY
ANTICIPATE REAL-WORLD VARIABILITY IN PATIENT POPULATIONS
© 2017 | DATA SCIENCE INSTITUTE™: AMERICAN COLLEGE OF RADIOLOGY | ALL RIGHTS RESERVED.PERFORMANCE CRITERIA AND STATISTICAL METRICS SPECIFIED FOR FOR EACH USE CASE
STATISTICAL SPECIFICATIONS
© 2017 | DATA SCIENCE INSTITUTE™: AMERICAN COLLEGE OF RADIOLOGY | ALL RIGHTS RESERVED.
CERTIFY-AI REPORTS FOR DEVELOPERS AND REGULATORS
© 2017 | DATA SCIENCE INSTITUTE™: AMERICAN COLLEGE OF RADIOLOGY | ALL RIGHTS RESERVED.
ACR TOUCH-AI
MONITORING ALGORITHM PERFORMANCE IN CLINICAL PRACTICE
Specifications For Monitoring In Clinical Practice
• Radiologist input (e.g. agree/disagree) is gathered as the case is being reported
• Specified metadata about the exam such as equipment vendor, slice thickness and exposure are also transmitted to the registry
• Assessment reports include algorithm performance metrics and the exam parameters
• Assessment reports used by the developers for algorithm improvement, continuous learning, and post-market surveillance reporting to the FDA
ACR Assess-AI
ACR CERTIFY-AI
ACR DATA-AI
The Radiology AI EcosystemIdeas To Clinical Practice
ACR ASSESS-AI
HOW DO WE MAKE SURE IT WORKS IN THE
REAL WORLD?
MONITORING, SUCCESS RATE, FAILURE CONDITIONS, SCALABLE
© 2017 | DATA SCIENCE INSTITUTE™: AMERICAN COLLEGE OF RADIOLOGY | ALL RIGHTS RESERVED.
Lung RADS 37 mm nodule
with…..
Rad Report
EHRRegistry
Other
XML Reporting Framework
(CARDS)
Full Initial LungCancer Screening AI
Visualization And Reporting
UI
Cloud / On-prem Modality
PACSTranscription
Detect and Localize
Quantify and Characterize
Classify
Registry
PerformanceAnalytics And
Quality Improvement
Registries
Developers
End Users
Regulators
AlgorithmPerformanceAssessment
AI Output
7 mm
Solid
Lung-RADS 3
AI IN CLINICAL PRACTICE WITH REGISTRY REPORTING FOR MONITORING WITH REAL-WORLD DATA
INTEGRATING AI INTO ROUTINE CLINICAL PRACTICE
© 2017 | DATA SCIENCE INSTITUTE™: AMERICAN COLLEGE OF RADIOLOGY | ALL RIGHTS RESERVED.
ACR TOUCH-AIACR Assess-AI
ACR CERTIFY-AI
ACR DATA-AI
The Radiology AI EcosystemIdeas To Clinical Practice
ACR ASSESS-AI
ACR Certify-AI
ACR DSI AI-Data DirectoryAI-Data Directory
ACR TOUCH-AITOUCH-AI Structured AI Use Cases Define Parameters For
Testing, Training, Validation And Monitoring AI
Diverse Multi-Institution Data For Testing And Training
Embargoed Multi-Institution Data For Algorithm Validation
Monitoring AI Performance In The Wild Supplements Algorithm Validation
MONITORING ALGORITHM PERFORMANCE IN CLINICAL PRACTICE
© 2017 | DATA SCIENCE INSTITUTE™: AMERICAN COLLEGE OF RADIOLOGY | ALL RIGHTS RESERVED.
DSI ACTIVITY TIMELINE
© 2017 | DATA SCIENCE INSTITUTE™: AMERICAN COLLEGE OF RADIOLOGY | ALL RIGHTS RESERVED.
DSI ACTIVITIES (JUN-DEC, 2018)
• ACR DSI Economics Summit (Jun)
• ACR/RSNA Common Data Elements Workshop (Jul)
• ACR/RSNA Standards Workshop (Aug)
• ACR/RSNA/NIBIB NIH AI Workshop (Aug)
• ACR DSI FDA Meeting (Sep)
• MICCAI DSI Conference (Sep)
• AAPM Webinar (Sep)
• Q&S/DSI Joint Conference (Oct)
• RSNA ML Showcase (Nov)
• FDA NEST Pilot of Certify-AI Completion (Dec)
• ACR RLI Practice Leaders Forum (Jan ‘19)
DSI ACTIVITY TIMELINE
© 2017 | DATA SCIENCE INSTITUTE™: AMERICAN COLLEGE OF RADIOLOGY | ALL RIGHTS RESERVED.
RSNA 2018 MACHINE LEARNING SHOWCASE
© 2017 | DATA SCIENCE INSTITUTE™: AMERICAN COLLEGE OF RADIOLOGY | ALL RIGHTS RESERVED.
1. Introduction & Background
2. Intro to AI and Imaging
3. The Value of AI in Imaging
4. The Challenges of Using AI in
Image Interpretation
5. Augmented Intelligence for
Radiologists
6. The ACR Data Science Institute
(DSI)
7. Case Study of AI in Image
Interpretation
8. An Example of a DSI Use Case
9. Concluding Slides & Discussion
Agenda
© 2017 | DATA SCIENCE INSTITUTE™: AMERICAN COLLEGE OF RADIOLOGY | ALL RIGHTS RESERVED.
Radiologists who use AI will replace those who don’t.
© 2017 | DATA SCIENCE INSTITUTE™: AMERICAN COLLEGE OF RADIOLOGY | ALL RIGHTS RESERVED.
1. Introduction & Background
2. Intro to AI and Imaging
3. The Value of AI in Imaging
4. The Challenges of Using AI in
Image Interpretation
5. Augmented Intelligence for
Radiologists
6. The ACR Data Science Institute
(DSI)
7. Case Study of AI in Image
Interpretation
8. An Example of a DSI Use Case
9. Concluding Slides & Discussion
Agenda
© 2017 | DATA SCIENCE INSTITUTE™: AMERICAN COLLEGE OF RADIOLOGY | ALL RIGHTS RESERVED.
Possible Applications of AI in Medical Imaging
Image interpretation
• Quantification of findings
• Quantified comparison between multiple studies
• Multiparametric analysis across multiple modalities
• Volumetric analysis
• Textural analysis
• Automation of Region Of Interest targeting and measuring
© 2017 | DATA SCIENCE INSTITUTE™: AMERICAN COLLEGE OF RADIOLOGY | ALL RIGHTS RESERVED.
Possible Applications of AI in Medical Imaging
Patient care and safety
• Detection and prioritization of potentially critical results
• Radiation dose optimization
• Pre-test probability assessment of patient risk of positive findings and contrast reactions
• Cancer and mammography screening
• Automatic protocoling of studies from EMR data
© 2017 | DATA SCIENCE INSTITUTE™: AMERICAN COLLEGE OF RADIOLOGY | ALL RIGHTS RESERVED.
Possible Applications of AI in Medical Imaging
Practice optimization for productivity and quality• Automated transcription of audio narration
• Automated population of structured reports
• Optimization for case assignment across teams
• Increased accuracy of coding
• Smarter PACS hanging protocols and synchronization protocols
• Communication and tracking of primary and incidental findings
• Decreased patient waiting times
• Quality improvement in scanning
• Prediction and prevention of missed patient appointments
• Preventing imaging machine outages
© 2017 | DATA SCIENCE INSTITUTE™: AMERICAN COLLEGE OF RADIOLOGY | ALL RIGHTS RESERVED.
1999 2015
Improving Diagnosis In Health Care
© 2017 | DATA SCIENCE INSTITUTE™: AMERICAN COLLEGE OF RADIOLOGY | ALL RIGHTS RESERVED.
Radiologist Relevance In Error Reduction
© 2017 | DATA SCIENCE INSTITUTE™: AMERICAN COLLEGE OF RADIOLOGY | ALL RIGHTS RESERVED.
Introducing the team of the Radiology Professional + AI
Receiver Operating Characteristic (ROC) Curves
Specificity
Sen
siti
vity
A Long Term Goal for
Radiology
0
1
1
© 2017 | DATA SCIENCE INSTITUTE™: AMERICAN COLLEGE OF RADIOLOGY | ALL RIGHTS RESERVED.
Why I need AI to help me do better Diagnostic Radiology in Breast Imaging
© 2017 | DATA SCIENCE INSTITUTE™: AMERICAN COLLEGE OF RADIOLOGY | ALL RIGHTS RESERVED.
1. 2. 3.
© 2017 | DATA SCIENCE INSTITUTE™: AMERICAN COLLEGE OF RADIOLOGY | ALL RIGHTS RESERVED.
Why I need AI to help me do better Diagnostic Radiology in Breast Imaging
Tests are not perfect: Mammographic sensitivity decreased from a level of 85.7%–88.8% in patients with almost entirely fatty tissue to 62.2%–68.1% in patients with extremely dense breast tissue.
85.7%–88.8%
62.2%–68.1%
© 2017 | DATA SCIENCE INSTITUTE™: AMERICAN COLLEGE OF RADIOLOGY | ALL RIGHTS RESERVED.
Why I need AI to help me do better Diagnostic Radiology in Breast Imaging
• A good population screening tool needs to be widely available and even in the USA we are not screening every woman who is eligible
• Management of probably benign findings costs money and human capital
• Every year 41,000 American women die of breast cancer
• Guidelines are confusing: give me more time to speak with patients
© 2017 | DATA SCIENCE INSTITUTE™: AMERICAN COLLEGE OF RADIOLOGY | ALL RIGHTS RESERVED.
© 2017 | DATA SCIENCE INSTITUTE™: AMERICAN COLLEGE OF RADIOLOGY | ALL RIGHTS RESERVED.
Disease progressionEarly Late
Diagnostic Imaging, AI & Population Health
PoorGoodOutcome
Det
ecti
on
Symptomatic
Asymptomatic
© 2017 | DATA SCIENCE INSTITUTE™: AMERICAN COLLEGE OF RADIOLOGY | ALL RIGHTS RESERVED.
Disease progressionEarly Late
Symptomatic
Asymptomatic
Det
ecti
on
Pre-symptomatic
Diagnostic Imaging, AI & Population Health
PoorGoodOutcome
© 2017 | DATA SCIENCE INSTITUTE™: AMERICAN COLLEGE OF RADIOLOGY | ALL RIGHTS RESERVED.
Radiologists Making the Most of Data Science and Artificial Intelligence
2 Help sick patients get healthy as soon as possible
1 Prevent illness
3 Stabilize & manage patients with chronic conditions
Doing Better With Less…
Improving the health of populations
Improving the individual experience of care
Reducing the per capita costs of care
Improving the work life of those who deliver care
… Through Imaging.
© 2017 | DATA SCIENCE INSTITUTE™: AMERICAN COLLEGE OF RADIOLOGY | ALL RIGHTS RESERVED.
1. Introduction & Background
2. Intro to AI and Imaging
3. The Value of AI in Imaging
4. The Challenges of Using AI in
Image Interpretation
5. Augmented Intelligence for
Radiologists
6. The ACR Data Science Institute
(DSI)
7. Case Study of AI in Image
Interpretation
8. An Example of a DSI Use Case
9. Concluding Slides & Discussion
Agenda
© 2017 | DATA SCIENCE INSTITUTE™: AMERICAN COLLEGE OF RADIOLOGY | ALL RIGHTS RESERVED.
Another challenge to using AI is that we don’t really understand how AI arrives at a particular conclusion.
© 2017 | DATA SCIENCE INSTITUTE™: AMERICAN COLLEGE OF RADIOLOGY | ALL RIGHTS RESERVED.
Why is the algorithm effective?What’s inside the black box?
What’s in the black box?
Neonatal Intraventricular Hemorrhage
Explicability
© 2017 | DATA SCIENCE INSTITUTE™: AMERICAN COLLEGE OF RADIOLOGY | ALL RIGHTS RESERVED.
What’s in the Black Box?
© 2017 | DATA SCIENCE INSTITUTE™: AMERICAN COLLEGE OF RADIOLOGY | ALL RIGHTS RESERVED.
“…machine prediction is a complement to human
judgment. And cheaper prediction will generate
more demand for decision-making, so there will be
more opportunities to exercise human judgment. ”
© 2017 | DATA SCIENCE INSTITUTE™: AMERICAN COLLEGE OF RADIOLOGY | ALL RIGHTS RESERVED.
• The patient’s prior radiation dose exposure is unknown, which
can this impact a decision of CT vs. MRI?
• Does the patient have to drive 3 hours to get to a more advanced
imaging machine?
• Does the patient have claustrophobia that makes it hard to go in
certain machines?
• The patient is losing her insurance at the end of the month, so a
follow-up exam in the future may not be feasible.
• The patient suffers from multiple, co-morbid conditions so how
sure can we be that any one condition is the cause of the finding?
• How much might we learn from an immediate follow-up study
and what are the cost-benefit factors of how this might impact
decisions about the course of treatment?
Some rewards that computer has a hard time weighing:
© 2017 | DATA SCIENCE INSTITUTE™: AMERICAN COLLEGE OF RADIOLOGY | ALL RIGHTS RESERVED.
© 2017 | DATA SCIENCE INSTITUTE™: AMERICAN COLLEGE OF RADIOLOGY | ALL RIGHTS RESERVED.
Anatomy
(e.g., body part)
Using Representative/Diverse Training Data: Multiple Dimensions of Image Variation
Patient demographics
(e.g., gender, age)
Pathology
(e.g., degree of tear)
Modality
(e.g., X-Ray, MRI, CT, PET, Ultrasound)
© 2017 | DATA SCIENCE INSTITUTE™: AMERICAN COLLEGE OF RADIOLOGY | ALL RIGHTS RESERVED.
Using Representative/Diverse Training Data: Multiple Dimensions of Image Variation
Modality
(e.g., X-Ray, MRI, CT, PET, Ultrasound)
Modality-specific variations
• MRI – For example:• Techniques
(Pulse Sequences, Field of View)• Anatomic planes
(axial, sagittal, coronal)• Equipment variation
(Manufacturer, Product Version and Firmware/Software Version, Field Strengths, Signal-to-Noise Ratio)
• CT – For example:• Exposure parameters• Slice thickness• Number of detectors• Equipment variation
© 2017 | DATA SCIENCE INSTITUTE™: AMERICAN COLLEGE OF RADIOLOGY | ALL RIGHTS RESERVED.
Goal of Image Interpretation
The Triangulation Approach to Radiographic Diagnosis
Step #1 Step #2
Correlation of radiographic findings and Gamut with patients’ clinical and lab findings to arrive at the most likely diagnosis
Step #3
© 2017 | DATA SCIENCE INSTITUTE™: AMERICAN COLLEGE OF RADIOLOGY | ALL RIGHTS RESERVED.
4,600Unique imaging findings
www.gamuts.net contains:
13,000Unique conditions that cause findings
57,000Linkages between findings and conditions
© 2017 | DATA SCIENCE INSTITUTE™: AMERICAN COLLEGE OF RADIOLOGY | ALL RIGHTS RESERVED.
© 2017 | DATA SCIENCE INSTITUTE™: AMERICAN COLLEGE OF RADIOLOGY | ALL RIGHTS RESERVED.
© 2017 | DATA SCIENCE INSTITUTE™: AMERICAN COLLEGE OF RADIOLOGY | ALL RIGHTS RESERVED.
© 2017 | DATA SCIENCE INSTITUTE™: AMERICAN COLLEGE OF RADIOLOGY | ALL RIGHTS RESERVED.
© 2017 | DATA SCIENCE INSTITUTE™: AMERICAN COLLEGE OF RADIOLOGY | ALL RIGHTS RESERVED.
© 2017 | DATA SCIENCE INSTITUTE™: AMERICAN COLLEGE OF RADIOLOGY | ALL RIGHTS RESERVED.
© 2017 | DATA SCIENCE INSTITUTE™: AMERICAN COLLEGE OF RADIOLOGY | ALL RIGHTS RESERVED.
1. Introduction & Background
2. Intro to AI and Imaging
3. The Value of AI in Imaging
4. The Challenges of Using AI in
Image Interpretation
5. Augmented Intelligence for
Radiologists
6. The ACR Data Science Institute
(DSI)
7. Case Study of AI in Image
Interpretation
8. An Example of a DSI Use Case
9. Concluding Slides & Discussion
Agenda
© 2017 | DATA SCIENCE INSTITUTE™: AMERICAN COLLEGE OF RADIOLOGY | ALL RIGHTS RESERVED.
What will it be like when AI is an indispensable tool for radiology professionals?
© 2017 | DATA SCIENCE INSTITUTE™: AMERICAN COLLEGE OF RADIOLOGY | ALL RIGHTS RESERVED.
Case Study for AI Adoption in Imaging
© 2017 | DATA SCIENCE INSTITUTE™: AMERICAN COLLEGE OF RADIOLOGY | ALL RIGHTS RESERVED.
Case Study for AI Adoption in Imaging
• Partnered with company developing algorithms looking for five
findings on chest, abdomen, and pelvis CT scans:
1) coronary calcium scores, 2) pulmonary emphysema, 3) liver
steatosis, 4) spine compression fractures, and 5) bone mineral
density
• Automatically scans images when received by the PACS and notifies
radiologists when they enter the case with a green light/red light
indicator if it identifies something
• Phased testing and adoption to obtain confidence in software and
buy-in from clinicians
• Benefits found with:
• Incidental findings that can be overlooked
• Potentially problematic bone mineral density readings that
are too early stage to be identified by the human eye
© 2017 | DATA SCIENCE INSTITUTE™: AMERICAN COLLEGE OF RADIOLOGY | ALL RIGHTS RESERVED.
Case Study for AI Adoption in Imaging
© 2017 | DATA SCIENCE INSTITUTE™: AMERICAN COLLEGE OF RADIOLOGY | ALL RIGHTS RESERVED.
“It’s like having an extra set of
eyes to help us provide additional
information to referring physicians
for optimal patient care.”
– Dr. Arun Krishnaraj
© 2017 | DATA SCIENCE INSTITUTE™: AMERICAN COLLEGE OF RADIOLOGY | ALL RIGHTS RESERVED.
Deep Learning: A Modern Approach to Early Breast Cancer Detection
Connie Lehman MD PhDProfessor of RadiologyHarvard Medical SchoolDirector of Breast ImagingMassachusetts General Hospital
© 2017 | DATA SCIENCE INSTITUTE™: AMERICAN COLLEGE OF RADIOLOGY | ALL RIGHTS RESERVED.
Modern technology is better but wide variation across radiologists
© 2017 | DATA SCIENCE INSTITUTE™: AMERICAN COLLEGE OF RADIOLOGY | ALL RIGHTS RESERVED.
Technology advances are limited by variable reader performance
© 2017 | DATA SCIENCE INSTITUTE™: AMERICAN COLLEGE OF RADIOLOGY | ALL RIGHTS RESERVED.
2012 2013 2014
© 2017 | DATA SCIENCE INSTITUTE™: AMERICAN COLLEGE OF RADIOLOGY | ALL RIGHTS RESERVED.
© 2017 | DATA SCIENCE INSTITUTE™: AMERICAN COLLEGE OF RADIOLOGY | ALL RIGHTS RESERVED.
- +
+
-
+
+
-- -
--
-
-
+
-
+ or -
Pixel
Pixel
Deep Learning Methods
Pixel
Pixel
+ or -Race
Age
Family
Menopause
Traditional Methods
- +
+
-
+
+
-- -
--
-
-
+
-
High-Risk Benign Breast Lesions: Some Patients Can Avoid Surgery
© 2017 | DATA SCIENCE INSTITUTE™: AMERICAN COLLEGE OF RADIOLOGY | ALL RIGHTS RESERVED.
Bahl et al, Radiology (2017)
100% Excised | 87% Benign Surgery Reduction
Reducing Overtreatment: High Risk Lesions
ML Model
Benign / Malignant
30%
© 2017 | DATA SCIENCE INSTITUTE™: AMERICAN COLLEGE OF RADIOLOGY | ALL RIGHTS RESERVED.
© 2017 | DATA SCIENCE INSTITUTE™: AMERICAN COLLEGE OF RADIOLOGY | ALL RIGHTS RESERVED.
Wide Variation in Radiologists’ Assessment of Mammograms as “Dense”
© 2017 | DATA SCIENCE INSTITUTE™: AMERICAN COLLEGE OF RADIOLOGY | ALL RIGHTS RESERVED.
Density
Deep Learning density assessment to reduce human variation
Connie Lehman MD PhD MGHRegina Barzilay PhD MIT
Bahl et al, Radiology (2017)
© 2017 | DATA SCIENCE INSTITUTE™: AMERICAN COLLEGE OF RADIOLOGY | ALL RIGHTS RESERVED.
1. Introduction & Background
2. Intro to AI and Imaging
3. The Value of AI in Imaging
4. The Challenges of Using AI in
Image Interpretation
5. Augmented Intelligence for
Radiologists
6. The ACR Data Science Institute
(DSI)
7. Case Study of AI in Image
Interpretation
8. An Example of a DSI Use Case
9. Concluding Slides & Discussion
Agenda
© 2017 | DATA SCIENCE INSTITUTE™: AMERICAN COLLEGE OF RADIOLOGY | ALL RIGHTS RESERVED.
How can we make AI an indispensable tool for radiology professionals, referring physicians and patients?
© 2017 | DATA SCIENCE INSTITUTE™: AMERICAN COLLEGE OF RADIOLOGY | ALL RIGHTS RESERVED.
ACR Data Science Institute: Participation
• Industry vendors
• Data scientists
• Physicians
• Informaticists
• Patient advocates
• Healthcare executives
• Regulators and policy makers
• Insurers
• Patients
© 2017 | DATA SCIENCE INSTITUTE™: AMERICAN COLLEGE OF RADIOLOGY | ALL RIGHTS RESERVED.
The ACR: Leading the Way
109
© 2017 | DATA SCIENCE INSTITUTE™: AMERICAN COLLEGE OF RADIOLOGY | ALL RIGHTS RESERVED.
The Tragedy of the Commons
© 2017 | DATA SCIENCE INSTITUTE™: AMERICAN COLLEGE OF RADIOLOGY | ALL RIGHTS RESERVED.
ACR Data Science Institute: Participation
http://www.acrdsi.org/
© 2017 | DATA SCIENCE INSTITUTE™: AMERICAN COLLEGE OF RADIOLOGY | ALL RIGHTS RESERVED.
ACR DSI Mission
Ensure the value of radiologists as AI evolves through the development of appropriate use cases and workflow integration
Protect patients through leadership roles in the regulatory process with government agencies and validation of algorithms
Establish industry relationships by providing credible use cases, help with FDA and other government agencies, and pathways for clinical integration
Educate radiologists, other physicians and all stakeholders about AI and the ACR’s role in data science for the good of our patients
© 2017 | DATA SCIENCE INSTITUTE™: AMERICAN COLLEGE OF RADIOLOGY | ALL RIGHTS RESERVED.
Building AI
MARKETS
IDEAS
NEEDS SOLUTIONS
Assess
Assess
Concept
Concept Create
Create
Produce
Produce
AI DEVELOPMENT
CYCLE
DogsCats
© 2017 | DATA SCIENCE INSTITUTE™: AMERICAN COLLEGE OF RADIOLOGY | ALL RIGHTS RESERVED.
Building AI
APPLICATION ENVIRONMENT(DEPLOYMENT)
RESEARCH ENVIRONMENT(IDEAS)
CLINICAL ENVIRONMENT(NEEDS)
COMPUTE ENVIRONMENT(SOLUTIONS)
Assess
Assess
Concept
Concept Create
Create
Produce
Produce
AI DEVELOPMENT
CYCLE
© 2017 | DATA SCIENCE INSTITUTE™: AMERICAN COLLEGE OF RADIOLOGY | ALL RIGHTS RESERVED.
Clinical Data Science: Considerations
Use cases
Regulatory
Validation
Economics
Standards
Education
Commercialization
Legal
Ethical
Implementation
Content
Compute
© 2017 | DATA SCIENCE INSTITUTE™: AMERICAN COLLEGE OF RADIOLOGY | ALL RIGHTS RESERVED.
Leverage the value of radiology professionals as AI evolves through the development of appropriate use cases and workflow integration
Protect patients through leadership roles in the regulatory process with government agencies and verification of algorithms
Establish industry relationships by providing credible use cases, help with FDA and other government agencies, and pathways for clinical integration
Educate radiology professionals, other physicians and all stakeholders about AI and the ACR’s role in data science for the good of our patients
ACR DSI Mission
EDUCATIONAIECOSYSTEM
© 2017 | DATA SCIENCE INSTITUTE™: AMERICAN COLLEGE OF RADIOLOGY | ALL RIGHTS RESERVED.
ABDOMINAL IMAGING
BREAST IMAGING
CARDIAC IMAGING
EMERGENCY IMAGING
MUSCULOSKELETAL
NEURORADIOLOGY
NUCLEAR MEDICINE
PEDIATRIC IMAGING
THROACIC IMAGING
INTERVENTIONAL
MAGNETICRESONANCE
COMPUTEDTOMOGRAPHY
POSITRONEMISSION
RADIOGRAPHY ANGIOGRAPHY ULTRASOUND FLUOROSCOPY
ANATOMY ANATOMY ANATOMY ANATOMY ANATOMY ANATOMY ANATOMY
FINDINGS
FINDINGS
FINDINGS
FINDINGS
FINDINGS
FINDINGS
FINDINGS
FINDINGS
FINDINGS
FINDINGS
PCL Tear Use Case
© 2017 | DATA SCIENCE INSTITUTE™: AMERICAN COLLEGE OF RADIOLOGY | ALL RIGHTS RESERVED.
Building AI
PRODUCTS
IDEAS
NEEDS SOLUTIONS
Clinical Use
Clinical Use
AI Use Case
AI Use Case
Clinical Data
Clinical Data
AI Models
AI Models
?
No standard AI use cases (annotation, validation, integration and surveillance)
No standard methods for clinical integration of AI
No standard methods for AI model validation
No standard method for AI model training and testing
AI DEVELOPMENT
CYCLE
© 2017 | DATA SCIENCE INSTITUTE™: AMERICAN COLLEGE OF RADIOLOGY | ALL RIGHTS RESERVED.
While there are promising publication and initial applications of AI in healthcare, there is currently a limited use of AI in clinical care.
Possible Reasons Current Impact
1 Clinically effective uses for AI have been poorly defined
2 No standards for clinical integration / care management
3 Large, annotated training sets are difficult to create
4 Currently no successful economic/business models
5 Limitations in current AI/human UX/UI
6 Inconsistent results and explicability between models
7 Healthcare regulatory hurdles are challenging
8 Resulting inference models are too brittle in practice
9 Data science algorithms are limited for healthcare use
10 Poor acceptance of technology in healthcare
Healthcare AI Challenges
© 2017 | DATA SCIENCE INSTITUTE™: AMERICAN COLLEGE OF RADIOLOGY | ALL RIGHTS RESERVED.
Building AI
CLINICAL PRODUCTS
IDEAS
NEEDS ALGORITHMS
Clinical Use
Clinical Use
AI Use Case
AI Use Case
Clinical Data
Clinical Data
AI Models
AI Models
Bring great ideas and clinical needs together
Standardized methods to annotate, or aggregate, data for
AI model training and testing
AI DEVELOPMENT
CYCLE
Mechanisms to integrate and monitor, AI models in clinical practice – using real world experience
* Standardized methods for AI model validation*
© 2017 | DATA SCIENCE INSTITUTE™: AMERICAN COLLEGE OF RADIOLOGY | ALL RIGHTS RESERVED.
• Algorithms useful, safe and effective
• Clinically validated
• Transparency in algorithm output
• Monitored in practice
• Free of unintended bias
• Medicare and insurance coverage issues
Protecting Patients From Unintended Consequences Of AI
!
© 2017 | DATA SCIENCE INSTITUTE™: AMERICAN COLLEGE OF RADIOLOGY | ALL RIGHTS RESERVED.
Applications of AI in Medical Imaging
Image interpretation
• Quantification of findings
• Quantified comparison between multiple studies
• Multiparametric analysis across multiple modalities
• Volumetric analysis
• Textural analysis
• Automation of Region Of Interest targeting and measuring
© 2017 | DATA SCIENCE INSTITUTE™: AMERICAN COLLEGE OF RADIOLOGY | ALL RIGHTS RESERVED.
Applications of AI in Medical Imaging
Patient care and safety
• Detection and prioritization of potentially critical results
• Radiation dose optimization
• Pre-test probability assessment of patient risk of positive findings and contrast reactions
• Cancer and mammography screening
• Automatic protocoling of studies from EMR data
© 2017 | DATA SCIENCE INSTITUTE™: AMERICAN COLLEGE OF RADIOLOGY | ALL RIGHTS RESERVED.
Applications of AI in Medical Imaging
Radiologist and optimization for productivity and quality• Automated transcription of audio narration
• Automated population of structured reports
• Optimization for case assignment across teams
• Smarter PACS hanging protocols and synchronization protocols
• Communication and tracking of primary and incidental findings
• Decreased patient waiting times
• Quality improvement in scanning
• Prediction and prevention of missed patient appointments
© 2017 | DATA SCIENCE INSTITUTE™: AMERICAN COLLEGE OF RADIOLOGY | ALL RIGHTS RESERVED.
ACR DSI Data Science
Panels
ACR DSIUse CaseDirectory(Public)
ACR DSIDataset
AuthoringUtilities(Public)
ACR DSIPerformance
Analytics ServiceNRDR
AI Registry
ACR DSIAlgorithm Validation
Service
AI modelDeployment
Directory(Public)
Testing/TrainingDatasetsDirectory(Public)
ACR Assist-AITM
ClinicalDirectory(Public)
The Radiology AI EcosystemIdeas To Clinical Practice
Radiology’s Value Proposition
• Trusted partnerships with industry and regulators
• Ensure patient safety
• Increase radiology professionals’ value in healthcare
Use Case Development• Use case authoring platform• Human language to machine language
Moving Clinically Effective AI Use Cases To Clinical Practice: Radiology AI Ecosystem
ACR®TOUCH-AI
© 2017 | DATA SCIENCE INSTITUTE™: AMERICAN COLLEGE OF RADIOLOGY | ALL RIGHTS RESERVED.
RadElement.org
AI Data Elements
Common Data Elements
© 2017 | DATA SCIENCE INSTITUTE™: AMERICAN COLLEGE OF RADIOLOGY | ALL RIGHTS RESERVED.
ACR DSI Data Science
Panels
ACR DSIUse CaseDirectory(Public)
ACR DSIDataset
AuthoringUtilities(Public)
ACR DSIPerformance
Analytics ServiceNRDR
AI Registry
ACR DSIAlgorithm Validation
Service
AI modelDeployment
Directory(Public)
Testing/TrainingDatasetsDirectory(Public)
ACR Assist-AITM
ClinicalDirectory(Public)
Making Datasets For AI Training Available To Developers
Standard Specifications For Data Access
ACR DSI Data Access
Directory
Specifications For Data Access
• Standardized definitions and data elements allow multiple institutions to use these standards to create datasets that developers can use for algorithm training and testing.
• Specifications include standardized tools and methods for image annotation.
• Using multiple sites as data sources for these datasets provides technical, geographic and patient diversity to prevent unintended bias in algorithm development.
• Allows more individuals and institutions to participate in AI development.
• The ACR DSI will house a freely available public directory of institutions that have created these datasets around ACR DSI Use Cases to inform the developer community.
© 2017 | DATA SCIENCE INSTITUTE™: AMERICAN COLLEGE OF RADIOLOGY | ALL RIGHTS RESERVED.
ACR DSI Data Science
Panels
ACR DSIUse CaseDirectory(Public)
ACR DSIDataset
AuthoringUtilities(Public)
ACR DSIPerformance
Analytics ServiceNRDR
AI Registry
ACR DSIAlgorithm Validation
Service
AI modelDeployment
Directory(Public)
Testing/TrainingDatasetsDirectory(Public)
ACR Assist-AITM
ClinicalDirectory(Public)
ACR Data Science Institute Certified Algorithms
Validating AI Algorithms
Specialty society certification of AI
algorithms provides an “honest broker”
partnership with radiology, developers
and government regulators
Specifications For Algorithm Validation
• Centralized assessment of algorithm performance will be performed according to the statistical metrics metrics specified in the use case using novel datasets.
• These validation datasets are created at multiple institutions to ensure geographic, technical and patient diversity within the validation dataset.
• Multiple readers and guidelines for data quality to ensure “ground truth” consistency between sites, consistent metrics for measuring performance across sites and standards to protect developers’ intellectual property, ensure patient privacy and diminish bias.
• Reports are generated for developers
© 2017 | DATA SCIENCE INSTITUTE™: AMERICAN COLLEGE OF RADIOLOGY | ALL RIGHTS RESERVED.
ACR DSIPerformance
Analytics ServiceNRDR
AI Registry
ACR DSIAlgorithm Validation
Service
AI modelDeployment
Directory(Public)
ACR Assist-AITM
ClinicalDirectory(Public)
Monitoring Algorithm Performance In Clinical Practice
Specifications For Monitoring In Clinical Practice
• Data elements in each use case specify how the algorithm will be monitored in clinical practice.
• Radiologist input is gathered as the case is being reported, and if the radiologist does not incorporate the algorithm inferences into the report, this change is captured in the background by the reporting software. If the radiologists agrees, changes the output of the agrees with algorithm, this is also noted and transmitted to the registry.
• Specified metadata about the exam such as equipment vendor, slice thickness and exposure are also transmitted to the registry.
• Algorithm assessment reports include algorithm performance metrics and the exam parameters affecting the algorithms’ performance.
• These reports are used by the developers to report to the FDA and for algorithm improvement.
AI Monitoring Program
• Patient safety and FDA surveillance
• Algorithm transparency and radiologist acceptance
• Developer improvements
© 2017 | DATA SCIENCE INSTITUTE™: AMERICAN COLLEGE OF RADIOLOGY | ALL RIGHTS RESERVED.
Monitoring Algorithm Performance In Clinical Practice
AI Algorithm Assessment Reports For Each Developer
And Site
Raw Data Captured In Reporting Software And Transmitted To
An AI Registry
Raw Data From Reports And Modalities
Aggregated In Registry
Working Example of Monitoring Algorithm Performance Using An AI Data Registry
• This example is from a pediatric bone age classification algorithm. The reporting software, PACS or the modality transmits information about the radiologist’s agreement or disagreement with the algorithm along metadata about the examination to the AI data registry.
• The raw data are complied in the registry and reports are aggregated and developer specific reports are generated for developers for use in FDA post-market surveillance reports and to improve the algorithm.
• Site reports are provided to provide AI performance metrics to the clinical practices.
© 2017 | DATA SCIENCE INSTITUTE™: AMERICAN COLLEGE OF RADIOLOGY | ALL RIGHTS RESERVED.
Monitoring Algorithm Performance In Clinical Practice
PostmarketSurveillance
NationalEvaluation
System
“Real World” Data
TIME TO MARKET
Expedited AccessPathway
PremarketReview
Prem
arket Decisio
n
Benefit Risk
INFORMATION FLOW
“Safety Net”
Courtesy Greg Pappas, FDA
© 2017 | DATA SCIENCE INSTITUTE™: AMERICAN COLLEGE OF RADIOLOGY | ALL RIGHTS RESERVED.
Validating AI Algorithms – Regulatory Collaborations
Office of Science And Engineering Labs FDA Center For Devices And Radiological Health
© 2017 | DATA SCIENCE INSTITUTE™: AMERICAN COLLEGE OF RADIOLOGY | ALL RIGHTS RESERVED.
Validating AI Algorithms – Regulatory Update
Academic Partners And Industry Partners
NEST will evaluate program for using real word data to assess AI algorithms
Individual components of the validation process will support applications for MDDT
© 2017 | DATA SCIENCE INSTITUTE™: AMERICAN COLLEGE OF RADIOLOGY | ALL RIGHTS RESERVED.
Integrating AI Into Clinical Workflow
Lung RADS 37 mm nodule
with…..
Rad Report
Lung RADS 3 nodule with…..
Rad ReportRegistry
ReportUI
XML Reporting
Framework
ReportUI
XML Reporting
Framework
Classic Radiologist Decision Support
Hybrid Radiologist Decision Support With AI
AI
Rad Report
Full integrated AI
7 mm
Solid
Lung-RADS 3
Radiologist Input
7 mm
Solid
Lung-RADS 3
Radiologist Input
© 2017 | DATA SCIENCE INSTITUTE™: AMERICAN COLLEGE OF RADIOLOGY | ALL RIGHTS RESERVED.
Integrating AI Into Clinical Workflow: DSI Use Case Implementation
Lung RADS 37 mm nodule
with…..
Rad Report
EHRRegistry
Other
XML Reporting Framework
(CARDS)
Full Initial LungCancer Screening AI
Visualization And Reporting
UI
Cloud / On-prem Modality
PACSTranscription
Detect and Localize
Quantify and Characterize
Classify
Registry
PerformanceAnalytics And
Quality Improvement
Registries
Developers
End Users
Regulators
AlgorithmPerformanceAssessment
AI Output
7 mm
Solid
Lung-RADS 3
© 2017 | DATA SCIENCE INSTITUTE™: AMERICAN COLLEGE OF RADIOLOGY | ALL RIGHTS RESERVED.
The Importance Of Transparency
© 2017 | DATA SCIENCE INSTITUTE™: AMERICAN COLLEGE OF RADIOLOGY | ALL RIGHTS RESERVED.
Use Cases Content Validation Implementation Regulatory Safety
Economics Standards Education Facilitation Legal Ethical
DSI and Healthcare AI IndustryServices to assist industry deliver successful AI solutions to clinical practices• AI Use Case Development (ACR TOUCH-AI)• AI model Certification (ACR CERTIFY-AI)• AI model Integration (ACR ASSIST)• AI model Assessment (ACR DSI ASSESS and ACR AI REGISTRY)
ACR DSI Activities And Relationships: Industry
Moving AI From Concept To Clinical Practice
© 2017 | DATA SCIENCE INSTITUTE™: AMERICAN COLLEGE OF RADIOLOGY | ALL RIGHTS RESERVED.
ACR DSI Use Cases
Highest clinical value
Solvable by artificial
intelligence
USE CASES
Use Case Prioritization
AI
© 2017 | DATA SCIENCE INSTITUTE™: AMERICAN COLLEGE OF RADIOLOGY | ALL RIGHTS RESERVED.
ACR Data Science Institute Use Case Development: Data Science Subspecialty Panels
© 2017 | DATA SCIENCE INSTITUTE™: AMERICAN COLLEGE OF RADIOLOGY | ALL RIGHTS RESERVED.
“The ACR is actively creating use cases for imaging AI and will be working with MICCAI under this memorandum of understanding to leverage this knowledge base in MICCAI’s imaging AI competitions.
ACR will also work with MICCAI to promote learning on a global scale…”
© 2017 | DATA SCIENCE INSTITUTE™: AMERICAN COLLEGE OF RADIOLOGY | ALL RIGHTS RESERVED.
TOUCH-AI: Common Data Elements (CDE) and RadElements
RadElement.org
AI Data Elements
TOUCH-AI
ACR
MonitorData elements for monitoring
in clinicalpractice
ConceptNarratives and
Flowcharts
ValidateData elements and metrics for
validation
BuildData elements
to annotate,train, and test
IntegrateData elements
for clinicalintegration
© 2017 | DATA SCIENCE INSTITUTE™: AMERICAN COLLEGE OF RADIOLOGY | ALL RIGHTS RESERVED.
• Useful
• Safe and effective in clinical practice
• Performance monitored and improvements made based on real world data
• Transparent
• Ensure diversity and preventing unintended bias
Summary Of ACR DSI Objectives
© 2017 | DATA SCIENCE INSTITUTE™: AMERICAN COLLEGE OF RADIOLOGY | ALL RIGHTS RESERVED.
1. Introduction & Background
2. Intro to AI and Imaging
3. The Value of AI in Imaging
4. The Challenges of Using AI in
Image Interpretation
5. Augmented Intelligence for
Radiologists
6. The ACR Data Science Institute
(DSI)
7. Case Study of AI in Image
Interpretation
8. An Example of a DSI Use Case
9. Concluding Slides & Discussion
Agenda
© 2017 | DATA SCIENCE INSTITUTE™: AMERICAN COLLEGE OF RADIOLOGY | ALL RIGHTS RESERVED.
© 2017 | DATA SCIENCE INSTITUTE™: AMERICAN COLLEGE OF RADIOLOGY | ALL RIGHTS RESERVED.
NORMAL GRADE I GRADE II GRADE III GRADE IV
Mortality + + ~20% ~90%
DSI Pediatric Panel
Neonatal Intraventricular Hemorrhage
© 2017 | DATA SCIENCE INSTITUTE™: AMERICAN COLLEGE OF RADIOLOGY | ALL RIGHTS RESERVED.
Is there PVHemorrhage?
(Y/N)
Is there GM hemorrhage?
(Y/N)
Is there IV hemorrhage?
(Y/N)
Is thereHydrocephalus?
(Y/N)
PV Hemorrhage
GM BleedAI Use Case
IV BleedAI Use Case
HydrocephalusAI Use Case
Calculate IVH Grade
IVHGrade(I-IV)
ClinicalManagement
Saliency maps
Saliency maps
ACR Assist (TOUCH-AI) Module: Intraventricular Hemorrhage Reporting
Neonatal Germinal Matrix Hemorrhage Detection System
Header• Purpose - To detect germinal matrix hemorrhages in newborns on imaging.• Description - Long narrative• TOUCH-AI-ID - TAI.5001• Types - Ultrasound (US) Head• Referenced Clinical Algorithms — Papile IVH Grading System• Age – Neonates• Sex – All• Logic – External
Data• Input
• Mandatory - US Head• Optional — Birth weight
• Output• Mandatory - Presence of germinal matrix hemorrhage. (CDE-II) • Optional - Quantification Of GM hemorrhage. (CDE-12)• Optional - Saliency map of GM hemorrhage. (CDE-13)
© 2017 | DATA SCIENCE INSTITUTE™: AMERICAN COLLEGE OF RADIOLOGY | ALL RIGHTS RESERVED.
AI could help the radiologist to detect and/or quantify the following:
Abdominal Imaging• Liver steatosis
Breast Imaging• Malignant breast lesions
Cardiac Imaging• Coronary calcium scores• Risk for aortic aneurysms• Quantify LV/RV stroke volume, ejection
fraction, cardiac output and mass (to speed analysis)
Neuroradiology & Emergency Imaging• Brain bleed locations and assess severity (to
automatically move a case to the top of the radiology group’s worklist)
Musculoskeletal• Spine compression fractures• Bone mineral density
Nuclear Medicine• Alzheimer's disease with beta-amyloid PET/CT
(well before the onset of symptoms)
Pediatric Imaging• Tiny fractures in any bone
Thoracic Imaging• Lung nodules in Chest CTs• Pulmonary emphysema• Risk for pulmonary hypertension
© 2017 | DATA SCIENCE INSTITUTE™: AMERICAN COLLEGE OF RADIOLOGY | ALL RIGHTS RESERVED.
1. Introduction & Background
2. Intro to AI and Imaging
3. The Value of AI in Imaging
4. The Challenges of Using AI in
Image Interpretation
5. Augmented Intelligence for
Radiologists
6. The ACR Data Science Institute
(DSI)
7. Case Study of AI in Image
Interpretation
8. An Example of a DSI Use Case
9. Concluding Slides & Discussion
Agenda
© 2017 | DATA SCIENCE INSTITUTE™: AMERICAN COLLEGE OF RADIOLOGY | ALL RIGHTS RESERVED.
“The future is already here.It’s just not evenly distributed.”
William Gibson
© 2017 | DATA SCIENCE INSTITUTE™: AMERICAN COLLEGE OF RADIOLOGY | ALL RIGHTS RESERVED.
Summary
© 2017 | DATA SCIENCE INSTITUTE™: AMERICAN COLLEGE OF RADIOLOGY | ALL RIGHTS RESERVED.
AI will persistently and pervasively enhance all aspects of radiology
• It’s not about Human vs AI.
• It is about Human augmented by AI vs.
Human working without AI
© 2017 | DATA SCIENCE INSTITUTE™: AMERICAN COLLEGE OF RADIOLOGY | ALL RIGHTS RESERVED.
AI will expand today’s decision-making capabilities
• Earlier and better detection leads to
better treatment options and improved
outcomes
© 2017 | DATA SCIENCE INSTITUTE™: AMERICAN COLLEGE OF RADIOLOGY | ALL RIGHTS RESERVED.
Meaningful AI will improve quality, efficiency and outcomes
• Utilizing all available data to optimize
patient care
© 2017 | DATA SCIENCE INSTITUTE™: AMERICAN COLLEGE OF RADIOLOGY | ALL RIGHTS RESERVED.
My patients (i.e., your friends, family, colleagues and neighbors)thank you for all of your great work!
© 2017 | DATA SCIENCE INSTITUTE™: AMERICAN COLLEGE OF RADIOLOGY | ALL RIGHTS RESERVED.
Discussion and Q&A
Q? A!
155
top related