Artificial Intelligence and the Practice of Pathology
Toby C. Cornish, MD, PhDAssociate Professor of PathologyMedical Director of Informatics
Department of PathologyUniversity of Colorado SOM
DISCLOSURE
In the past 12 months, I have had a significant financial interest or other relationship with the manufacturer(s) of the following product(s) or provider(s) of the following service(s) that will be discussed in my presentation.
• Leica Biosystems, Inc. Pathology Imaging Advisory Board: Consulting Fees
Who am I?
• Anatomic Pathologist (GI Pathology)• Clinical Informaticist• Medical Director of Informatics
– University of Colorado SOM Dept of Pathology• Medical Director of the LIS
– UCHealth
• Flagship: University of Colorado Hospital (Aurora, CO)• 9 additional hospitals in Denver/Fort Collins/Colorado
Springs/Steamboat Springs area• 3 affiliated hospitals in Wyoming and Nebraska• 1+ hospitals under construction (Highlands Ranch)
UCHealth University of Colorado Hospital
UCHealth Broomfield Hospital
UCHealth Memorial Hospital CentralUCHealth Memorial Hospital NorthUCHealth Pikes Peak Hospital
UCHealth Grandview Hospital
UCHealth Longs Peak Hospital
WYOMINGCOLORADO
NEBRASKA
UCHealth Yampa Valley HospitalUCHealth Poudre Valley Hospital
UCHealth Medical Center of the Rockies
Ivinson Memorial Hospital
Cheyenne Regional Medical Center Sidney Regional Medical Center
June 7, 2017
UCHealth Grandview Hospital
AI and Computational Pathology
Artificial intelligence
• Production of intelligent-seeming behavior by machines
• Concept dates back to the early 1950s
• Comprises a broad range of approaches
Hierarchy of artificial intelligence
ArtificialIntelligence
MachineLearning
DeepLearning
Machine learning
• Machine learning allows computers to predict outcomes from data without being explicitly programmed
• Refines a model that predicts outputs using sample inputs (features) and a feedback loop
• Classical machine learning relies heavily on extracting or selecting salient features, which is a combination of art and science (“feature engineering”)
ArtificialIntelligence
MachineLearning
DeepLearning
Deep learning
• The concept dates back to the 1940s• Previously:
– “cybernetics” (1940s-60s)– “connectionism” / “artificial neural networks” (1980s-90s)
• “Deep learning” (c. 2006)• Multiple hidden layers in an artificial neural network• Computationally intensive• Until recently not feasible with large datasets
ArtificialIntelligence
MachineLearning
DeepLearning
Deep learning
• Avoids the need to define specific features in the data as inputs
• Discovers the features from the raw data provided during training
• Hidden layers in the artificial neural network represent increasingly more complex features in the data
• Convolutional Neural Networks (CNNs), a type of deep learning, are commonly used for image analysis
ArtificialIntelligence
MachineLearning
DeepLearning
Machine learning v. deep learning
http://adilmoujahid.com/posts/2016/06/introduction-deep-learning-python-caffe/
A deep learning model example
Goodfellow et al., Deep Learning. 2016
Output (object identity)
3rd hidden layer (object parts)
2nd hidden layer (corners and contours)
1st hidden layer (edges)
Visible layer (input pixels)
Strong v. Weak AI
• Strong AI– AI capable of solving difficult, generalized
problems including the ability to deal with unexpected circumstances
– e.g. perform “universal” histopathologic diagnosis
• Weak AI (Narrow AI)– AI capable of solving specific tasks– e.g. identify breast cancer metastasis in a lymph
node
Ken Jennings, after losing to IBM’s Watson on Jeopardy! in 2011
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
Artificial intelligence Machine learning Deep learning
Watson winsJeopardy
Google Search Trends
Gartner Hype Cycle
Expe
ctat
ions
TimeTechnology Trigger
Peak of Inflated Expectations
Trough of Disillusionment
Slope of Enlightenment
Plateau of Productivity
Hype Cycle for Emerging Technologies (2018)
http
s://w
ww.
gartn
er.c
om/s
mar
terw
ithga
rtner
/top-
trend
s-in
-the-
gartn
er-h
ype-
cycl
e-fo
r-em
ergi
ng-te
chno
logi
es-2
018/
Expe
ctat
ions
Time
Deep Learning
1st “AI Winter” 2nd “AI Winter” 3rd “AI Winter”?
Expert Systems
1987 19931974 1980 ? ?
Early approaches
?
What is an “AI Winter”?
• Extended period of time in which enthusiasm for and investment in AI “dried up”
• Generally attributed to:– Over-promising– Under-delivering– Creation of an economic bubble– Infighting– Inadequate computational hardware
• Two major “winters” are recognized with many smaller events
Beyond Jeopardy: The IBM Watson Legacy
Journal of Clinical Oncology 31, no. 15_suppl (May 2013) 6508-6508.
Background: Electronic decision support is increasingly prevalent in clinical practice. Traditional tools map guidelines into an interactive platform. An alternative method builds on experience-based learning.
Methods: Memorial Sloan-Kettering (MSK), IBM and WellPoint teamed to develop IBM Watson – a cognitive computing system leveraging natural language processing (NLP), machine learning (ML) and massive parallel processing – to help inform clinical decision making. We made a prototype for lung cancers using manufactured and anonymized patient cases. We configured this tool to read medical language and extract specific attributes from each case to identify appropriate treatment options benchmarked against MSK expertise, anonymized patient cases and published evidence. Treatment options reflect consensus guidelines and MSK best practices where guidelines are not granular enough to match treatments to unique patients. Analysis and building accuracy is ongoing and iterative.
Aug. 11, 2018
https://www.statnews.com/wp-content/uploads/2018/09/IBMs-Watson-recommended-unsafe-and-incorrect-cancer-treatments-STAT.pdf
https://www.statnews.com/wp-content/uploads/2018/09/IBMs-Watson-recommended-unsafe-and-incorrect-cancer-treatments-STAT.pdf
https://hackernoon.com/%EF%B8%8F-big-challenge-in-deep-learning-training-data-31a88b97b282
Training data in machine learning
• Large amounts of data and expert annotation are the keys to successful training of machine learning algorithms
• The amount of data needed is typically proportional to the difficulty of the task
Training data for self driving cars
• Professor Amnon Shashua, co-founder and CTO of Mobileye:– “Mobileye has 600 people on staff annotating images and
expects to have 1,000 or more shortly.”
https://www.teslarati.com/mobileye-cto-building-autonomous-driving-systems/
https://aws.amazon.com/blogs/machine-learning/aws-partners-with-mapillary-to-support-the-large-scale-scene-understanding-challenge-at-cvpr-2017/
AI and Computational Pathology
Definition: Computational Pathology• An approach to diagnosis that incorporates multiple sources of
raw data (eg, clinical electronic medical records, laboratory data including “-omics,” and imaging [both radiology and pathology imaging]); extracts biologically and clinically relevant information from these data; uses mathematic models at the molecular, individual, and population levels to generate diagnostic inferences and predictions; and presents this clinically actionable knowledge to customers through dynamic and integrated reports and interfaces, enabling physicians, patients, laboratory personnel, and other health care system stakeholders to make the best possible medical decisions.
Computational Pathology: A Path Ahead. Louis DN, et al. Arch Pathol Lab Med. 2016 Jan;140(1):41-50.
3 5 6 8 11 514
7 12 18 199
269
179
1726
52
88
124141
155 155
119128
137
201
250
309294
406
384
11 1 1 1 2 4 10 9 11
0
50
100
150
200
250
300
350
400
450
1986 1987 1988 1989 1990 1991 1992 1993 1994 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 2015 2016 2017 2018 2019
Pubm
edAr
ticle
s
Pathology and ("Artifical Intelligence" or "Machine Learning" or "Deep Learning") "Computational Pathology"
Bartels PH et al. Anal Quant Cytol Histol. 1988 Aug;10(4):299-306
So which is it: AI or Computational pathology?
• AI applied to pathology is a form of Computational Pathology
• It doesn’t matter what you call it
• I will likely use these terms interchangeably in this talk
How will AI affect the Practice of Pathology?
The future of AI in pathology
• Variety of opinions– Practicing pathologists– AI / Digital Pathology enthusiasts– Data scientists– Informaticists– Vendors
The future of AI in pathology
• Depends on the time scale under consideration– 1 year?– 5 years?– 10 years?– 20 years?– 30 years?– 100 years?
The future of AI in pathology
• Depends on what the underlying technology is– Deep learning?– The next generation of AI?
Impact of AI
Potentialof AI
Weak(Narrow)
Strong(General)
Negative Positive
Skeptics
Dystopians Utopians
Pragmatists
• "artificial intelligence is the future and it always will be"
The pragmatist’s view of AI in Pathology
A pragmatist’s view of AI in Pathology
• Barring disruptive changes in the AI paradigm, such as– A non linear increase in computing power– New methods of building AI
• A General AI that could replace a pathologist is far off, but not impossible
• Narrow AI tools will reach the early market within 3 - 5 years and the mainstream market within 5 – 8 years
A pragmatist’s view of AI in Pathology
• AI will become just another tool we adopt and use alongside our traditional tools
• Consider previous scenarios:– External examination v. dissection– Macroscopic examination v. microscopic examination– Electron microscopy v. light microscopy– IHC v. H&E– Molecular pathology v. histopathology
The fact that immunocytochemistry of human tissues is now a rich and complex field (in terms both of the different techniques available and the large number of different molecular entities detectable in human tissues) has tended to cause a polarisation among pathologists. There are the "monoclonal antibody revolutionaries" and an opposing group of "contras". This polarisationis not, as far as we know, described in the literature but is nevertheless to be found in pathology departments throughout the world.
Mason DY & Gatter KC. J Clin Pathol. 1987, 40(9):1042-54
Mason DY & Gatter KC. J Clin Pathol. 1987, 40(9):1042-54
Skeptic?
Mason DY & Gatter KC. J Clin Pathol. 1987, 40(9):1042-54
Dystopian?
Utopian?
0
5000
10000
15000
20000
25000
30000
35000
EM IHC / ICC Mol Dx AI DP
Pubm
edar
ticle
s
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
EM IHC / ICC Mol Dx AI DP
Perc
enta
ge o
f tot
al a
rticl
es
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
EM IHC / ICC Mol Dx AI DP
Cum
ulat
ive
perc
enta
ge o
f tot
al a
rticl
es
Pathologist benefits
• Use CP as an adjunct to other traditional tests and methods• Derive diagnostic, prognostic and predictive information not
possible before• Increase visibility and importance of pathologists in the healthcare
enterprise• Create new billable tests (where applicable)• “Automate” the boring stuff• “Automate” the time-consuming stuff• “Automate” the hard stuff
Challenges for CP in Digital Pathology
• Accumulation of large well-annotated image datasets and related data
• Creation of robust, generalizable models• Demonstration of value to pathologists, clinicians, administrators• Overcome stakeholder skepticism• Insufficient substrate for implementation (i.e. not enough clinical
deployment of WSI)• Lack of CP expertise amongst pathologists
Creating a “Digital Substrate” for pathology
• The “Digital Substrate” represents the digitization of all pathology slides to permit higher order processes that add value to the practice of pathology
• Value of using digital pathology increases with the percentage of digital pathology adopters (width of the digital substrate)
• Both requires and enables integration of digital pathology functions at a variety of levels
Is it this?
DICOM
• Digital Imaging and Communications in Medicine• DICOM supplement 145 for WSIs (2011)• Still not widely implemented or supported
http://dicom.nema.org/dicom/dicomwsi/Connectathon/index.html
Systems integration (single vendor)
DSR
LIS
ImageAnalysis
Digitalslides
WSIViewer
Image dataAnnotation
IA orderOrders?Results?
Systems integration: 3rd party?
DSR
LIS ImageAnalysis
Digitalslides
No interfaces to 3rd party image Analysis system
WSIViewer Image data
IA order
Clinical slide archiving
Accessioning
Specimen processing
Slide creation
Case distribution
Pathologist review & sign out
Archives
Slide scanning
Contributing Pathologists
Central Lab
Consulting Pathologists
ConsultationNetwork
Secondary diagnosis telepathology
Systems integration: Clinical lab
LISEHR/CPOE MiddlewareProvider
Test QueryOrders
Results
Orders
Results
Labels
Orders
Results
Instrument
Samples
1 4
Test Orders
Results
2
3
6
5
7
891011
LIS
PACS
MiddlewarePathologist
Results
Orders
Results
Orders
Results
1 4
Results
9
10
11
WSIWSI
WSI
AI
Figure 2: Architecture of a computational pathology app and platform. All work is done by the platform, but the app contains all the ‘intelligence’; it defines the algorithm, on which slides to run it, and how to present the results to the pathologist. This upfront programming consists of two steps: 1) select applicable slide types (stain, organ, etc.), and 2) program the app’s tasks on the analytics engine. These tasks are performed automatically in step 3) when a relevant slide is scanned. Finally when a pathologist reviews a relevant case, step 4) displays and allows interaction with the pre-calculated results.
Systems integration
DSR
LIS ImageAnalysis
Digitalslides
IA orders/results
Image, Annotation,
resultsAPIs
WSIViewer
Image dataAnnotation
IA order
No DP
No AI=
Systems integration
PACS(DICOM
)
LIS ImageAnalysis
Digitalslides
IA orders/results
Image, Annotation,
resultsAPIs
WSIViewer
IA order
Image dataAnnotation
Summary: systems integration
• Current interfacing is very lightweight• No standard model exists for robustly interfacing image analysis,
LIS and DSR components– Orders in– Results out– Use of a single image repository (i.e. no duplication of image data)
• Most productive labs rely heavily on FTEs as “human interfaces” or have spent considerably on custom programming
• Current solutions not scalable (think CP)
Summary: systems integration
• Lab information systems will be the biggest impediment to systems integration for DP & CP
• Many are aging poorly and lack a modern UX• Existing data models and APIs are inadequate for many
proposed CP workflows– For example, LISes do not discretely know what type of tissue is in a
block or on a slide
https://ai.googleblog.com/2018/04/an-augmented-reality-microscope.html
Left: Schematic overview of the ARM. A digital camera captures the same field of view (FoV) as the user and passes the image to an attached compute unit capable of running real-time inference of a machine learning model. The results are fed back into a custom AR display which is inline with the ocular lens and projects the model output on the same plane as the slide. Right: A picture of our prototype which has been retrofitted into a typical clinical-grade light microscope.
Example view through the lens of the ARM. These images show examples of the lymph node metastasis model with 4x, 10x, 20x, and 40x microscope objectives.
https://webcast.aacr.org/console/player/38653?sec=1&mediaType=slideVideo&&crd_fl=0&ssmsrq=1524692831210Hipp & Stumpe. Advancing Cancer Diagnostics with Artificial Intelligence. AACR 2018 Annual Meeting. Chicago, IL
Addressing challenges:“technopanic”
“AI Technopanic”
Daniel Castro and Alan McQuinn. 9/10/2015. The Information Technology & Innovation Foundation
Thoughts from another domain…
Kasparov v. Deep Blue
• May 11, 1997• Match result: Deep Blue (W) - Kasparov:
3½ - 2½
http
s://w
ww.
flick
r.com
/pho
tos/
amitr
ajit/
http
s://c
omm
ons.
wik
imed
ia.o
rg/w
iki/F
ile:K
aspa
rov-
29.jp
g
What have they been up to since?
Deep Blue
Computer History Museum (CHM), Mountain View, California, US
https://www.flickr.com/photos/amitrajit/
Garry Kasparov
• In June 1998, Kasparov played the first public game of human-computer collaborative chessagainst Veselin Topalov
• Each used a regular computer with off-the-shelf chess software (Kasparov: Fritz 5, Topalov: ChessBase 7.0)
• The match ended in a 3 - 3 tie• Kasparov called this “advanced chess,” and it has later been
called “centaur chess”• Retired in 2005
“Freestyle” chess
• 2005: the first “freestyle” chess tournament• Teams could consist of any number of humans or computers• Some teams consisted of chess grand masters• The most powerful chess computer at the time was also
entered• The winning team consisted of young, amateur players,
Steven Cramton and Zackary Stephen and their computers
- Garry Kasparov"The Chess Master and the Computer“ The New York Review of Books, 2/11/2010
“weak human + machine + better process was superior to a strong computer alone and, more remarkable, superior to a strong human + machine + inferior process.”
http://www.nybooks.com/articles/2010/02/11/the-chess-master-and-the-computer/
Summary
• Computational pathology is coming with near term applications of narrow (weak) AI to clinical problems
• Abundant, expertly labeled (annotated) training data is the key factor to successful CP solutions
• With some exception, the “gold standard” for labeling pathology images will continue to be the pathologist
• Diligence is required to ensure that our efforts represent ground truth
• Labeling of training data may shift to directly measures of clinical outcome such as response to treatment
Questions? Toby C. Cornish, M.D., Ph.DAssociate Professor of Pathology
University of Colorado School of [email protected]