whispers of speckles (part i: building computational imaging frameworks for acoustic and optical...

1

WHISPERS

OF SPECKLESDEBDOOT SHEET

LAUNCHING THIS MONSOON

Venue: IIT Mandi Date: Thursday, 25 June 2015

Time: 11 am – 12:30 pm

Whispers of SpecklePart I: Building Computational Imaging

Frameworks for Acoustic and Optical Speckle Imaging

Dr. Debdoot SheetAssistant Professor

Department of Electrical EngineeringIndian Institute of Technology Kharagpur

www.facweb.iitkgp.ernet.in/~debdoot/

Whispers of Speckles [Debdoot Sheet] - WMLMIA 3

Inspiration“A wonderful fact to reflect upon, that every human creature is constituted to be that profound secret and mystery to every other.”

- Charles Dickens (A Tale of Two Cities)

“If you want to find the secrets of the universe, think in terms of energy, frequency and vibration.”

- Nikola Tesla

25 June 2015

Motivation


Intuitive and descriptive biology of tissues

Histology, molecular pathology and semi-quantitative evaluation

Joint analysis of structural and molecular attributes and co-located complexity of tissues through multimodal imaging

Learning of uncertainty in tissue energy interaction in acoustic and optical imaging to understand co-located tissue heterogeneity towards in situ Histopathology

D. Sheet (2014), PhD Thesis

25 June 2015

Text books

R. K. Das (2012), PhD Thesis

A. Barui (2011), PhD Thesis


Introduction• Human body consists of organs and

systems made up of different tissues.• Pathological conditions and

abnormalities affect their normal functioning.

• Critical soft tissue abnormalities include– Plaque formation in the blood vascular

system.– Lesions in the breast.– Degeneration of the retina.– Wounds in the skin.

• Traditional practice of Histopathological diagnosis requires invasive Biopsy / Excision for tissue collection

– Not possible in vessels in living Humans– Improper sampling from Breast lesion

affects diagnostic outcome– Not possible in retina in living Humans– Not possible in healing wounds.25 June 2015


ACHIEVING IN SITU HISTOLOGY OF VASCULAR PLAQUES

25 June 2015


• Atherosclerosis– Plaque builds up in arteries– Forms anywhere in the vascular system

• Cardiovascular diseases (CVD) • In vivo Imaging of Plaques

– CT Angiography (CTA)– MR Angiography (MRA)– Intravascular Ultrasound (IVUS)– Intravascular OCT (IV-OCT)– Intravascular Near-Infrared Spectroscopy

(NIR)• Plaque Vulnerability Assessment

– Calcification, fibrosis identification– Lipid pool and Necrosis burden estimation Source: NIH – National Heart,

Lung, and Blood Institute

Blood Vascular System

25 June 2015

• Spectral analysis of received ultrasonic echo signal– Lizzi et al., 1983– Nair et al., 2001– Kawasaki et al., 2002– Virtual Histology (Volcano Corp.)– iMap (BostonScientific)

• Texture analysis of B-mode image/signal– Katouzian et al., 2008, 2010, 2012 (Prog. Hist. / PH)– Esclara et al., 2009– Seabra et al., 2011

• Limitations– Unable to identify heterogeneous tissue composition– Cannot discriminate between dense fibrous tissue

and calcification– Fails to discriminate true necrosis from shadows

Whispers of Speckles [Debdoot Sheet] - WMLMIA

Backdrop

8

White light source350nm 750nm

Power

Stained tissue section

Tissue specific spectrum350nm 750nm

Power

Calcified

Fibrotic

25 June 2015


White light source350nm 750nm

Power

Stained tissue section

Tissue specific spectrum350nm 750nm

Power

Calcified

Fibrotic

: Probing energy (Light)

: Physiological property (Tissue type)

f

1 fInferring tissue type based on color

25 June 2015


Computation Modelling of Tissue Energy Interaction for In situ Histopathology

Computed histology

: Probing energy (Acoustic) : Tissue type (Backscatterer density)

f

1 f

Inferring tissue type based on backscattering response

25 June 2015


Limited Resolution Challenge

11

r1

r2

r3

P. M. Shankar, “A general statistical model for ultrasonic backscattering from tissues”, IEEE Trans. Ultrasonics, Ferroelectrics, Freq. Control., vol. 47, no. 3, pp. 727-736, May 2000.

11 rr f

22 rr f

33 rr f

Ultrasound signal backscattered within a resolution cell

i

i

r

r

fE

E

Signal sensed by the transducer

ir

fEf 1ˆ Estimated functional ensemble of backscatterer density

ˆ Improper estimation of tissue type in inhomogeneous media

25 June 2015


Statistical Physics in Acoustic Imaging

r1

r2

r3

r1

r2

m=0.5

Ω1

Ω2

r

P(r)

m=1.0

Ω1

Ω2

Ω3

r

P(r)

P. M. Shankar, “A general statistical model for ultrasonic backscattering from tissues”, IEEE Trans. Ultrasonics, Ferroelectrics, Freq. Control., vol. 47, no. 3, pp. 727-736, May 2000.

2

12

exp2

,| rm

m

rmmr

m

mm

N

25 June 2015


Statistical physics of ultrasonic backscattering

Lipidic

r

P(r)

Fibrotic

r

P(r)

Calcified

r

P(r)

V. Dumane and P. M. Shankar, “Use of frequency diversity and Nakagami statistics in ultrasonic tissue characterization”, IEEE Trans. Ultrasonics, Ferroelectrics, Freq. Control, vol. 48, no. 4, pp. 1139-1146, Jul. 2001

F. Destrempes, J. Meunier, M. . F. Giroux, G. Soulez, G. Cloutier, “Segmentation in ultrasonic b-mode images of healthy carotid arteries using mixture of Nakagami distributions and stochastic optimization”, IEEE Trans. Med. Imaging, vol. 28, no. 2, pp. 215-229, Feb. 2009.

25 June 2015


32121

221121

,,, 1,

1,,

1,

111

)(,|,|

,|

;),(,),,,,,(||

L

llll

L

llll

L

llll

mrpmrpp

mrpp

ymprfyrp

NN

N

Mathematical intractability, the problem

)()(

)|()|( yP

rp

yrpryp The probabilistic decision making framework

Scales unknown

Correlation among scales unknown

No. components unknown

Prior probab. of each comp. unknown

25 June 2015


Proposed Solution

Statistical physics model of ultrasonic backscattering

Set of signal received by the transducer

Training set of annotated examples to be used for supervised learning

Supervised learner for learning tissue specific statistical physics model

train;,|)(),(

)|,(),|( RR

yHyPrp

yrpryp

Solution through Transfer Learning Framework

25 June 2015


HOW TO DEAL WITH THIS AS A MACHINE LEARNING CHALLENGE?

25 June 2015


Learning?

A computer program is said to learn from experience E with respect to some class of tasks T and performance measure P, if its performance at tasks in T, as measured by P, improves with experience E

-Tom Mitchell

25 June 2015


Demystifying Learning

25 June 2015

Man 1 Man 2 Man 3Man 4

Great Wall logo

Great Wall tower

Kim JungWangDebdoot

Experience (E)

Perfo

rman

ce (P

)

Debdoot, Kim, Jung and Wang are standing near the Great Wall logo and the Great Wall tower is behind them.


How was it Learning?

25 June 2015


Great Wall logo

Great Wall tower

Kim JungWangDebdoot

Salient Segments

Objectify

Detect humans

Recognize inanimate

Describe Scene


Recognize humans


GETTING MACHINES TO LEARN TISSUE – ENERGY INTERACTION FOR IN SITU HISTOLOGY

25 June 2015


IVUS Tissue Characterization

21

BackgroundLipidicFibroticCalcifiedNecrosis

Iterative self-organizing atherosclerotic tissue labeling in intravascular ultrasound images and comparison with virtual histology, IEEE TBME, 59(11), 2012

Hunting for necrosis in the shadows of intravascular ultrasound, CMIG, 38(2), 2014

Joint learning of ultrasonic backscattering statistical physics and signal confidence primal for characterizing atherosclerotic plaques using intravascular ultrasound, Med. Image Anal,18(1), 2014

Nakagami parameter and signal

confidence estimate

Random forest learning

25 June 2015


Ultrasound Signal Confidence• An ultrasonic pulse as well as

backscattered echo travel along the same path through a heterogeneous media.

• They are subjected to the same attenuation.

• Confidence of the received signal is a reflection of fidelity of samples received by the transducer.

• It can be estimated by treating its propagation as a random walk along an ultrasonic scan-line.

• A random walker starting at a point on the scan-line reaches the virtual transducer element placed at the origin of each scan-line.

• This random walk is solved using the electric network equivalent and solving it in the paradigm of graph theory.

25 June 2015

A. Karamalis, W. Wein, T. Klein, N. Navab (2012) Ultrasound confidence maps using random walks, Medical Image Analysis, 16:1101–1112.

Transfer Learning Framework

23

Ultrasound RF data(i) Signal confidence(ii) Speckle statistics

Tissue labels

fLearnt random forest

Learning phase (offline) Tissue labels

Prediction(online)

f

Whispers of Speckles [Debdoot Sheet] - WMLMIA25 June 2015


Random Forests for Learning

25 June 2015

A. Criminisi and J. Shotton, Decision Forests for Computer Vision and Medical Image Analysis, Springer, 2013.


Experiment Design• Data Collection:

– Columbia University, New York City, NY, USA

– Interventional Cardiologist: Dr. Stephane G. Carlier

– Cardiovascular Histopathologist: Dr. Renu Virmani, CV Path Institute, Gaithersburg, USA

– Cases # 13– Tissue Sections # 53– Atlantis, 40 MHz IVUS, Boston Scientific,CA,

USA– Sampling freq: 400 MHz– Sampling geometry: 256 scan lines per

rotation, 2048 samples per scan line

• Learning– Source task: {Ω,m} estimated at 28 scales

+ Ultrasonic Confidence (A. Karamalis, et al. (2012))

– Target task: Random forest 50 decision trees

• Cross validation– 53 fold cross validation– Learn with 52, test on the remaining

25 June 2015


Ultrasonic Histology of Atherosclerotic Plaques

• Characterization based on ultrasonic statistical physics. • Superior machine learning algorithm. • Reliability measure for estimation of tissues.

Probability of Calcified tissues

Probability of Fibrotictissues

Probability of Lipidictissues

Probability of Necrotictissues

CalcifiedLipidicFibroticNecrotic

25 June 2015


Coronary Plaque Characterization and Staging

25 June 2015


SOME (AWESOME SCORES ON) METRICS

25 June 2015


Performance Evaluation

25 June 2015

Inter-observer variability

Intra-observer variability


ANALYZING COMPLEXITY OF MACHINE LEARNING

25 June 2015


Computational Complexity

Training Complexity Testing Complexity

25 June 2015


Roles of Source Knowledge (Features)

Feature 1

Fe

atu

re 2

25 June 2015


Importance of Source Knowledge

Genuer, R., Poggi, J.-M., Tuleau-Malot, C., (2010). Variable selection using random forests. Pat. Recog. Letters. 31(14):2225-2236

25 June 2015


Learning from Approximately Labeled Minimum Samples

25 June 2015

Donomez, P., Lebanon, G., Balasubramanian, K., (2010). Unsupervised supervised learning I: Estimating Classification and Regression Errors without Labels, J. Mach. Learn. Res. 11:1323-1351


Variation of Residual Error in Learning

25 June 2015

Learning example

Test results


END NOTE

25 June 2015


Collaboration and Network

25 June 2015


Take home message• Different types of soft tissues have characteristic

response when interacting with acoustic energy.• Heterogeneous tissues can be identified by

learning of tissue specific energy interaction response using statistical physics models.

• Transfer Learning can be employed for efficiently solving tissue characterization problems modeled as tissue-energy interaction problems.– CPU/GPU handshaking can be used for fast implementation of

such tasks

• Explore possibility of Functional Histopathology In situ

25 June 2015

Whispers of SpecklePart II: Enlightenment from Shallow to

Complex Reasoning with Deep Learning

Dr. Debdoot SheetAssistant Professor

Department of Electrical EngineeringIndian Institute of Technology Kharagpur

www.facweb.iitkgp.ernet.in/~debdoot/


DOES THIS METHOD OF TRANSFER LEARNING APPLY ONLY TO ULTRASONIC IMAGING?

25 June 2015


Skin• Skin forms the general covering of

the body protecting us from external influences.

• Functions– Thermoregulation– Sweat secretion– Tactile, pressure, temperature sensing

• Stratified organization– Epidermis– Papillary dermis– Dermis– Adipose tissue

• Wound – Major pathological injury – Skin is torn, cut, punctured

• Clinical challenge in management– Healing in person specific– Patient specific intervention– In situ investigation of healing is

challenge

25 June 2015


Skin

25 June 2015

• In situ investigation– Optical Coherence Tomography (OCT)

• Cobb et al., (2006)• Barui et al., (2011)

– Optical photography• Cross-sectional information about healing wound

is not available

– NIR imaging• Cross-section histological information not present

• In situ Histology with OCT– G. van Soest et al., (2010) – Cardiovascular

OCT– A. Barui et al., (2011) – Cutaneous wound beds.

• Challenges– Identify co-located tissue heterogeneity– Identify and discriminate Inter-digitated

structures

Tissue Photon Interaction


Incident radiation

Regularreflection Diffuse

reflection

Scattering

Absorption Multispectral optical imageOCT

B. Saleh, Introduction to Subsurface Imaging, Cambridge, 2011.

0.5 mm

0.5 mm

25 June 2015

Optical Coherence Tomography


Low time-coherence light source

Depth scan mirror

Sample

Detector

Source beam

Reference beam

Sample beam

Detector beam

xz

z

OCT Image

Michelson interferometer

25 June 2015

TPI in Swept Source OCT


Source

Ballistic backscattering

Non-ballisticbackscattering

Reference

Detector

A. F. Fercher, et al, Optical coherence tomography — principles and applications, Rep. Prog. Phys. 66 (2003) 239–303

EpitheliumPapillary dermis

DermisAdipose

Speckle intensity

Probability density

25 June 2015

S

S

SS

IIp

exp

1


COMPUTATIONAL OPTICAL COHERENCE HISTOLOGY THROUGH TRANSFER LEARNING

25 June 2015

Framework

25 June 2015 Whispers of Speckles [Debdoot Sheet] - WMLMIA 47

Learn TPI Model

Training Image Ground Truth Labels

Test Image

Learn TPI Model

Characterized tissue

train;,| II, xH


Computational Histology of Skin

• Solution through a transfer learning approach

• Performance benchmark (Accuracy)– Epithelium = 99%– Papilary dermis = 95%– Dermis = 99%– Adipose = 98%

• D. Sheet, et al, “In situ histology of mice skin through transfer learning of tissue energy interaction in optical coherence tomography”, J. Biomed. Optics, 18(9), 2013.

25 June 2015

Multi-scale modeling of

OCT speckles

Trainingimage

set Ground truth



OCT speckles

Test image

Labeled tissue


In situ Histology of Skin

OCT

Histo

Epithelium

EpitheliumPapillary dermis

DermisAdipose tissue

25 June 2015 49

Papillary dermisDermisAdipose tissueAll tissues

In situ histology of mice skin through transfer learning of tissue energy interaction in optical coherence tomography, J. Biomed. Optics, 18(9), 2013


In vitro validation towards In vivo

translation

25 June 2015

Transfer Learning of Tissue Photon Interaction in Optical Coherence Tomography towards In vivo Histology of the Oral Mucosa, Proc. ISBI, 2014.


Computational Histology of Retina

• Transfer learning approach– Retinal OCT tissue labeling

• Performance benchmark (Accuracy)– Anterior coat > 98%– RPE > 92%– Posterior coat > 99%

• SPK Karri and D. Sheet, et al., “Computational Histology of Retina through Transfer Learning of Tissue Photon Interaction in Optical Coherence Tomography”, Proc. Int. Symp. Biomedical Imaging (ISBI), 2014.

25 June 2015


OCT speckles

Trainingimage

set

Ground truth



OCT speckles

Test image

Labeled tissue


DOES SOMETHING LOOK FISHY?

25 June 2015


State of the Art• In situ Histology with OCT

– G. van Soest et al., (2010), G. J. Ughi et al., (2013) – Cardiovascular OCT

– D. Sheet et al., (2013, 2014) – Cutaneous wounds, oral

• Challenges– Heuristic features

• Texture• Intensity statistics

– Heuristic computational models

• Transfer learning of speckle occurrence models

– Incomplete representation dictionary


OCT speckles

Trainingimage

set Ground truth



OCT speckles

Test image

Labeled tissue

25 June 2015


Heuristics in State of Art

25 June 2015


(RE)EXPLORING THE CONCEPTS OF HIERARCHY IN LEARNING

25 June 2015


How was it Learning?

25 June 2015


Great Wall logo

Great Wall tower

Kim JungWangDebdoot

Salient Segments

Objectify

Detect humans

Recognize inanimate

Describe Scene


Recognize humans


Challenges

25 June 2015

Salient Segments

Objectify

Detect humans

Recognize inanimate

Describe Scene

Recognize humans

Salient Segments

Objectify

Detect humans

Recognize inanimate

Describe Scene

Recognize humans

Salient Segments

Objectify

Detect humans

Recognize inanimate

Describe Scene

Recognize humans

Salient Segments

Describe Scene


Challenges

25 June 2015

Salient Segments

Objectify

Recognize inanimate

Describe Scene

Recognize humans

LBP

Wavelets

HoG

Body part recognition

Human appearance

Chromaclustering

Posture realign Silhouette

matchingRecognize

humanDetect

humans


FROM SHALLOW TO COMPLEX REASONING

25 June 2015


Heuristics in State of Art

25 June 2015


The Solution

Deno

ising

Aut

o En

code

r

Deno

ising

Aut

o En

code

r

Logi

stic

Reg.

25 June 2015


Using a Deep Network

25 June 2015


COMPLEX REASONING AND ITS DEEP LEARNING

25 June 2015


Challenges

25 June 2015

Salient Segments

Objectify

Detect humans

Recognize inanimate

Describe Scene

Recognize humans

Salient Segments

Objectify

Detect humans

Recognize inanimate

Describe Scene

Recognize humans

Salient Segments

Objectify

Detect humans

Recognize inanimate

Describe Scene

Recognize humans

Salient Segments

Describe Scene


Challenges

25 June 2015

Salient Segments

Objectify

Recognize inanimate

Describe Scene

Recognize humans

LBP

Wavelets

HoG


Human appearance

Chromaclustering


matchingRecognize

humanDetect

humans


How to tackle this dilemma?

25 June 2015

Great Wall behindGreat Wall logo beside

Debdoot, Kim, Jung, Wang


Multilayer Perceptron (MLP)

25 June 2015

Hidd

en la

yers

Hidd

en la

yers

Hidd

en la

yers


MLP Learning, troubles thereof

25 June 2015

P

T1

T2


MLP Learning troubles, why so?

25 June 2015

P

T1

T2

LBP

Wavelets

HoG


Human appearance

Chromaclustering


matchingRecognize

human?


HOW TO DEEP LEARN?

25 June 2015


Deep Learning, origin and growth• Around 1950 – NN age

– Neural Nets (McCulloch and Pitts, 1943)

– Unsupervised Learn. (Hebb, 1949)– Supervised Learn. (Rosenblatt, 1958)– Associative Memory (Palm, 1980;

Hopfield, 1982)

• 1960– Discovery of visual sensory cells that

respond to Edges (Hubel and Wiesel, 1962)

– Feed Forward Multi Layer Perceptron (FF-MLP) (Ivakhnenko, 1968)

• 1980 – Neocognition– Convolution + WeightReplication +

Subsampling (Fukushima, 1980)– Max Pooling– Back-propagation (Werbos, 1981;

LeCunn, 1985, 1988)

25 June 2015


Deep Learning, origin and growth• 1980-2000 – Search for simple,

low-complexity, problem-solvers– Recurrent Neural Network (RNN)

(Hochreiter and Schmidhuber, 1996)

– Local learning Feed forward NN (Dayan and Hinton, 1996)

– Advanced gradient descent (Schaback and Werner, 1992)

– Sequential Network Construction (Honavar and Uhr, 1988)

– Unsupervised Pre-training (Ritter and Kohonen, 1989)

– Auto-Encoder (Hinton et al., 1989)

– Back Propagating Convolutional Neural Networks (LeCun et al., 1989, 1990a, 1998)

25 June 2015


Deep Learning, origin and growth• 2000 – Era of Deep Learning

– NIPS 2003 Feature Selection Challenge (Neal and Zhang, 2006)

– MNIST digit recognition (LeCun et al., 1989)

– Deep Belief Network (DBN) / Restricted Boltzmann Machines (Hinton et al., 2006)

– Auto Encoders (Bengio, 2009)

• 2006– GPU based CNN (Chellapilla et al.,

2006)

• 2009– GPU DBN (Raina et al., 2009)

• 2011– Max-Pooling CNN on the GPU

(Ciresan et al., 2011)

• 2012– Image Net (Krizhevsky et al., 2012)

25 June 2015


DEEP LEARNING OF COMPLEX REASONING FOR OCT TISSUE CHARACTERIZATION

25 June 2015


Exploring Deep Architecture

25 June 2015


OCT speckles

Trainingimage

set

Ground truth



OCT speckles

Test image

Labeled tissue

Stacked Auto-Encoders,

Logistic Regression

Random Forest

Trainingimage

set

Ground truth

Feature

Representation

http://www.facweb.iitkgp.ernet.in/~debdoot/current.html


Auto Encoder for Deep Learning

25 June 2015


Results in Wounds

(a) OCT image of wound (b) Ground truth (c) In situ histology

Epithelium, Papillary dermis, Dermis, Adipose

Epithelium, Papillary dermis, Dermis, Adipose

25 June 2015


Experiment Design• Data Collection

– School of Medical Science and Technology, Indian Institute of Technology Kharagpur

– 1300 nm (HPBW 100 nm) Swept Source OCT System

• OCS 1300 SS, ThorLabs, NJ, USA

• 8 bit bitmap images

– Histology for ground truth• HE stained

• Samples– Mus musculus (small mice)– 16 healthy skin– 2 wounds on skin

• DNN architecture– Patch size – 36 × 36 px– DAE1 – 400 nodes– DAE2 – 100 nodes– Target – Logistic Reg.

• 5 outputs

– Sparsity – 20%– Mini-batch training

• In situ Histology Performance– Epithelium – 96%– Papillary dermis – 93%– Dermis – 99% – Adipose tissue – 98%

25 June 2015


Learning of Representations

Representation of speckle appearance models learned by DAE1Sparsity of representations learned by

DAE2

25 June 2015


END NOTE

25 June 2015

Messages for Human Learning• Photons interact characteristically with different tissues.

– Stochastic similarity exists in speckle appearance.– Such representations are hard to heuristically encode.

• Deep learning and auto-encoders for computational imaging– Speckle imaging application viz. OCT tissue characterization– Hierarchical learning

• Locally embedded representations.• Sparsity is in learned (auto-encoded) representations.


Queries: Debdoot Sheet ([email protected])

25 June 2015


About Deep Learning“It’s like in quantum physics at the beginning of

the 20th century” Trishul Chilimbi (MSR, DNN, Adam)

“The experimentalists and practitioners were ahead of the theoreticians. They couldn’t explain the results. We appear to be at a similar stage with DNNs. We’re realizing the power and the capabilities, but we still don’t understand the fundamentals of exactly how they work.”

25 June 2015


Take home message

“We’ve humanized the scientist;

now we must scientize the

humanist. We didn’t try to

cover physics... we

uncovered it.”

- Robert Resnick

25 June 2015


Take home message• Challenges

– Architectures• Neural Nets vs. Others

– Implementation• CPU vs. GPU vs. Cloud

– GPU (VLSI) architectures• Hierarchical Temporal

Memory• Potential Causal Connection

• Toolboxes– Theano (Python/SciPy)– Pylearn2– Torch– Caffe– Matlab (Rasmus Berg Palm)

• More information– www.deeplearning.net– Schmidhuber (2014). Deep

Learning in Neural Networks: An Overview (arXiv:1404.7828)

– Bengio (2009). Learning Deep Architectures for AI.

– Deng and Yu (2013). Deep Learning: Methods and Applications.

• Conferences– Int. Conf. Learning

Representations (ICLR)

25 June 2015

http://www.deeplearning.net/

whispers of speckles (part i: building computational imaging frameworks for acoustic and optical...

Education

tissue collection

dense fibrous tissue

f inferring tissue type

phd thesis

white light source

optical speckle imaging

plaque formation

probing energy light