Generation and Use of Quantitative Pathology Phenotype

Joel Saltz MD,PhDChair Biomedical Informatics, Stony Brook

Associate Director Stony Brook Cancer Center

NCI Early Detection Research Network - Atlanta April 1, 2015

Computational Pathology: High Dimensional Fused-Informatics

• Anatomic/functional characterization at fine and gross level

• Integrate of anatomic/functional characterization, multiple types of “omic” information, outcome

• Predict treatment outcome, select, monitor treatments

• High throughput tissue classification• Computer assisted exploration of new

classification schemes• Integrated analysis and presentation of

observations, features analytical results – human and machine generated

Pathology

Patient Outcome

Radiology

“Omic”Data

Johns Hopkins School of Medicine

Virtual Microscope

1997 Proceedings AMIA Annual Meeting

Information Technology for Cancer Research : NCI U24

Stony Brook: Joel Saltz, Tahsin Kurc, Yi Gao, Allen Tannenbaum, Fusheng Wang, Liangjia Zhu, Ivan Kolesov, Romeil Sandhu, Erich BremerEmory: Adam Marcus, Lee Cooper, Dan Brat, Fadlo Khuri Lee Cooper, Ashish Sharma, Rick Cummings, Roberd BostickOak Ridge National Lab: Scott Klasky, Dave PugmireYale: Michael Krauthammer

Tools to Analyze Morphology and Spatially Mapped Molecular Data

Pathology Analytical Imaging

• Provide rich information about morphological and functional characteristics• Image analysis, feature extraction on multiple scales• Spatially mapped “omics”• Multiple microscopy modalities

Glass Slides Scanning Whole Slide Images Image Analysis

Tumor Heterogeneity

Marusyk 2012

Report: Imaging-Genomics Workshop June 2013

Correlating Imaging Phenotypes with Genomic Signatures: Scientific Opportunities

Clinical Approach and Use• Development of imaging+analysis methods to characterize heterogeneity

• within a tumor at one time point• evolution over time• among different tumor types

• Development of imaging metrics that:• can predict and detect emergence of resistance?• correlates with genomic heterogeneity?• correlates with habitat heterogeneity?• can identify more homogeneous sub-types

Imaging Genomics Workshop NCI June 2013

Direct Study of Relationship Between Image Features vs Clinical Outcome, Response to Treatment, Molecular Information

Lee Cooper,Carlos Moreno

Integrative Morphology/”omics”

Quantitative Feature Analysis in Pathology: Emory In Silico Center for Brain Tumor Research (PI = Dan Brat, PD= Joel Saltz)

NLM/NCI: Integrative Analysis/Digital Pathology R01LM011119, R01LM009239 (Dual PIs Joel Saltz, David Foran)

Marcus Foundation Grant – Ari Kaufman, Joel Saltz

Associations

Gene Expression Correlates of GBM with High Oligo-Astro Ratio

Oligo Related Genes

Myelin Basic ProteinProteolipoproteinHoxD1

Nuclear features mostAssociated with OligoSignature Genes:

Circularity (high)Eccentricity (low)

Pathology Computer Assisted Classification

Gurcan, Shamada, Kong, Saltz

Hiro Shimada, Metin Gurcan, Jun Kong, Lee Cooper Joel Saltz

Neuroblastoma Classification

FH: favorable histology UH: unfavorable histologyCANCER 2003; 98:2274-81

<5 yr

SchwannianDevelopment

≥50%Grossly visible Nodule(s)

absent

present

Microscopic Neuroblastic

foci

absent

present

Ganglioneuroma(Schwannian stroma-dominant)

Maturing subtypeMature subtype

Ganglioneuroblastoma, Intermixed(Schwannian stroma-rich)

FH

FH

Ganglioneuroblastoma, Nodular(composite, Schwannian stroma-rich/stroma-dominant and stroma-poor) UH/FH*

Variant forms*

None to <50%

Neuroblastoma(Schwannian stroma-poor)

Poorly differentiatedsubtype

Undifferentiatedsubtype

Differentiatingsubtype

Any age UH

≥200/5,000 cellsMitotic & karyorrhectic cells

100-200/5,000 cells

<100/5,000 cells

Any age

≥1.5 yr

<1.5 yr

UH

UH

FH

≥200/5,000 cells

100-200/5,000 cells

<100/5,000 cells

Any age UH

≥1.5 yr

<1.5 yr

≥5 yr

UH

FH

UH

FH

Computerized Classification System for Grading Neuroblastoma

• Background Identification

• Image Decomposition (Multi-resolution levels)

• Image Segmentation (EMLDA)

• Feature Construction (2nd order statistics, Tonal Features)

• Feature Extraction (LDA) + Classification (Bayesian)

• Multi-resolution Layer Controller (Confidence Region)

No

YesImage Tile Initialization

I = L Background? Label

Create Image I(L)

Segmentation

Feature Construction

Feature Extraction

Classification

Segmentation

Feature Construction

Feature Extraction

Classifier Training

Down-sampling

Training Tiles

Within ConfidenceRegion ?

I = I -1

I > 1?

Yes

Yes

No

No

TRAINING

TESTING

Large Scale Data Management

Data model capturing multi-faceted information including markups,annotations, algorithm provenance, specimen, etc.Support for complex relationships and spatial query: multi-level

granularities, relationships between markups and annotations, spatialand nested relationshipsHighly optimized spatial query and analysesImplemented in a variety of ways including optimized CPU/GPU,

Hadoop/HDFS and IBM DB2 (Wang, Saltz, Kurc)

NLM/NCI: Integrative Analysis/Digital Pathology R01LM011119, R01LM009239 (Dual PIs Joel Saltz, David Foran)

Spatial Centric – Pathology Imaging “GIS”Point query: human marked point inside a nucleus

.

Window query: return markups contained in a rectangle

Spatial join query: algorithm validation/comparison

Containment query: nuclear featureaggregation in tumor regions

Fusheng Wang

HPC Pipelines (Segmentation and Feature Extraction)

Tony Pan, George Teodoro,Tahsin Kurc and Scott Klasky

Tools to Analyze Morphology and Spatially Mapped Molecular Data1U24CA180924-01

SPECIFIC AIMS

Pathology Image Analysis

• Specific Aim - Develop, deploy, and evaluate robust and scalable methods and analysis pipelines for multi- scale, integrative image analysis.

• Aim 1a Develop methods to segment micro-anatomic objects and extract and classify features from whole slide tissue images.

• Aim 1b Develop analytic methods to carry out spatially mapped molecular tissue characterizations.

• Aim 1c Develop methods to create 3-D reconstructions of multi- scale micro-anatomic features and to characterize changes in morphology over time.

Pipelines, Database, Data modeling, Visualization

• Specific Aim 2: Develop database infrastructure to manage and query image data, image analysis results.

• Specific Aim 3: Develop high performance software that targets clusters, cloud computing, and leadership scale systems.

• Specific Aim 4: Develop visualization middleware for 2D/3D image and feature data and for integrated image and “omic” data.

caMicroscope - A Digital Pathology Integrative Query System – PI Ashish Sharma -CTIIP IIWG

• Interoperate with U24 MongoDB feature database to allow annotation and markup view and generation

• Extensions to support creation and display of annotations (free form pencil tool, polygon tool, rectangle & ellipse)

• Measurement tool & magnifying glass (magnify by 200% a region of interest)

caMicroscope/MongoDB - Multiple Algorithm Comparison

Why we need multiple algorithm comparison

Hierarchical Pipeline (Slicer!)Liangjia ZhuIvan KolesovRomeil SandhuAllen Tannenbaum

Low Power

• Fast GrowCut segmentation• Intensity insufficient: need user

guidance• Boundaries are most time

consuming for user

Medium Power

• Adaptive thresholdingsegmentation

• Allow for global user input (influence parameter settings)

Crypt/Nuclear Segmentation

• Variational active contour• Context is crucial

Convolutional Neural Network Classification

Le Hou, Dimitris Samaras, Tahsin Kurc, Yi Gao, Liz Vanner, James Davis, Joel Saltz

Confocal/Super resolution nuclear morphometry (Slicer!)Ken Shroyer, Yi Gao, Tahsin Kurc, Joel Saltz • Pancreatic Fine Needle

Aspirate• Correlative studies

linking fine needle aspirate cell data, “omic” and Radiology imaging data

• Leverages Marcus foundation virtual biopsy effort

Cells first prepared via Papanicolaou stain – identified as not suspicious

Preliminary Work

Cells first prepared via Papanicolaou stain – identified as suspicious

Cytology shape analysis

volume elongation surface area roundness sp-radius

sp-area ellipse diameter axis 1 ellipse diameter axis2 ellipse diameter axis 3 flatness

3D Imaging Analytics• 3D Vessel Reconstruction with Serial Microscopy Images

• WSIs from serial sections have significant potential to enhance the study of both normal and disease processes

• 3D reconstruction of cellular level objects is a critical step

(Yanhui Liang, Jun Kong, Fusheng Wang, Darren Treanor – Wang NSF CAREER Award)

3D Imaging Analytics: Vessel Association

1

2

2 1

(a)

(b)

(a) Panoramic view of 3D reconstructed vessels; (b) two close-up views ofsegment 1 and 2 indicated in the panoramic view.

Driving Cancer Research, Community Support, Challenges and Engagement

• Specific Aim 5: Drive continuing development of the tools using a suite of cancer driving biomedical problem, and provide collaborative support and training to the cancer research community.

• Aim 5a: Evaluate, demonstrate and drive continuing development of the tools using a suite of cancer driving biomedical problems.

• Aim 5b: Provide support to the cancer research community through: • Engagement in NCI Quantitative Imaging Network• support of community digital Pathology image analysis “grand challenges” – initial

challenge involving analysis of TCGA whole slide image at MICCAI 2014,• partnerships with collaborative efforts described in letters of support and collaboration,

including the National Cancer Imaging Archive, the Mayo Quantitative Imaging Network site, the Colon Cancer Family Registry and the Polyp Prevention Study,

• partnerships with cancer microscopy/Pathology shared resources • development of on line resources and workshops to teach users how to employ U24

tools.

CORRELATIVE FEATURE ANALYSIS

MICCAI 2014 BRAIN TUMORClassification and Segmentation Challenges

TCGA

TCIA

IMAGING CHALLENGE

DIGITAL PATHOLOGY CHALLENGE

Phase 1: TrainingJune 20 - July 31

Phase 2: Leader BoardAug 1 - Aug 29

Phase 3: TestSept 8 - Sept 12

For more information about these challenges and a related workshop on September 14, 2014 at MICCAI in Boston, see: cancerimagingarchive.net

MICCAI: Medical Image Computing and Computer Aided Interventions - MICCAI2014.orgTCGA: The Cancer Genome Atlas - cancergenome.nih.govTCIA: The Cancer Image Archive - cancerimagingarchive.net

Digital Pathology/Brain Tumor Image Segmentation (BRATS)

• Used data currently available through data archive resources of the National Institutes of Health (NIH), namely, the Cancer Genome Atlas (TCGA) and the Cancer Image Archive (TCIA)

• Digital Pathology challenge used digital slides related to patients whose genomics data are available from TCGA. Similarly, BRATS 2014 Challenge used clinical MRI image data, also from the TCGA study subjects.

• Sub-Challenge 1: Classification - Automated classification of LGG and GBM from a collection of 30+ high-resolution digital pathology slides.

• Sub-Challenge 2: Segmentation – Automated segmentation of necrotic and normal brain regions on regions of digital pathology slides from a collection of 20+ GBM cases.

Organizers and Major Contributors

• Daniel J. Brat, Emory University• Larry Clarke, National Cancer Institute• James Davis, Stony Brook Cancer Center• Keyvan Farahani, National Cancer Institute• John Freymann, Leidos Biomedical Res, Inc.• Carl Jaffe, Boston University• Justin Kirby, Leidos Biomedical Res., Inc.• Tahsin Kurc, Stony Brook Cancer Center• Miguel Ossandon, National Cancer Institute• Joel Saltz, Stony Brook Cancer Center• Roberta Seidman, Stony Brook Cancer Center

Example of a “good match”

Example of a problematic match

Thanks!