neuroimage analysis for automated brain disease...

Neuroimage Analysis

for Automated Brain Disease Diagnosis

University of North Carolina at Chapel Hill

[email protected]

http://mingxia.web.unc.edu/

07-17-2019

Mingxia Liu

https://www.med.unc.edu/bric/ideagroup


http://bric.unc.edu/


http://www.unc.edu/

http://www.unc.edu/

Healthy Brain Alzheimer’s

Ventricle

Sulcus

Gyrus

Sulcus

Gyrus

Healthy Brain Alzheimer’s

Healthy Brain vs. Alzheimer’s

Background





Background

• Alzheimer’s Disease (AD)

– A progressive disease

Normal

Control

Mild Cognitive

Impairment (MCI)Alzheimer’s

Disease

Bra

inH

ealth

Time

Stable MCI (sMCI)Progressive MCI (pMCI)

• Calling Need

– Developing computer-aided methods for MCI/AD diagnosis





• Structural Magnetic Resonance Imaging (MRI)

• FDG-Positron Emission Tomography (PET)

• Cerebrospinal Fluid (CSF) ‐‐‐ Aβ42, t‐tau and p‐tau

Biomarkers for early diagnosis of AD and MCI

MRI PET CSF

CSF

Brain

Dura

Multi-modal data

Background





M. Liu, etc., Landmark-based Deep Multi-Instance Learning for Brain Disease Diagnosis, Medical Image Analysis, 2018.

M. Liu, etc. Joint Classification and Regression via Deep Multi-task Multi-channel Learning for Alzheimer’s Disease Diagnosis. IEEE

Trans. on Biomedical Engineering, 2018.

fMRIPETsMRI

Neuroimaging

Data

Brain Disease Diagnosis – Typical Pipeline

Image

Preprocessing

Feature

Extraction/Selection

Classifier

Learning

Machine Learning & Deep Learning





Challenges in Computer-aided Disease Diagnosis

• Effective feature representation of neuroimages

• Missing multi-modal data

• Heterogeneous data at different imaging sites





Outline

• Missing Data

• Multi-modal Data Fusion

• Domain Adaptation

Multi-modal Neuroimage

CSFPETsMRI

Single-modal Neuroimage

• Structural MRI (sMRI)

sMRI





Part I. Single-modal Neuroimage Analysis

• Structural MRI based Brain Disease Diagnosis





http://www.unc.edu/

http://www.unc.edu/

Anatomical Landmarks for Structural MRI

• Landmark-based Deep Representation of sMRI

C. Lian, M. Liu, J. Zhang, and D. Shen. IEEE Trans. on Pattern Analysis and Machine Intelligence, 2019.

M. Liu, J. Zhang, C. Lian, and D. Shen. IEEE Trans. on Cybernetics, 2019.

M. Liu, J. Zhang, E. Adeli, and D. Shen. IEEE Trans. on Biomedical Engineering, 2019.

M. Liu, J. Zhang, E. Adeli, and D. Shen. Medical Image Analysis, 2018.

M. Liu, J. Zhang, D. Nie, P.T. Yap, and D. Shen. IEEE Journal of Biomedical and Health Informatics, 2018.

J. Zhang, M. Liu, and D. Shen. IEEE Trans. on Image Processing, 2017.





...

Training Data

Patch-Level Feature Learning

Convolutional Neural

Network 1

Landmark Discovery

Disease

ClassificationTesting Data

Landmark-based Patch Extraction


Network P

...

Image Retrieval

...Deep Feature

Representation 1

Deep Feature

Representation P

Patch-based

Features

MR Image Pre-processing

...

Landmark Detection

Landmark

Definition

...

Patch Extraction

...

Training Data



Network 1

Landmark Discovery

Disease




Network P

...

Image Retrieval

...Deep Feature

Representation 1

Deep Feature

Representation P

Patch-based

Features


...

Landmark Detection

Landmark

Definition

...

...

Training Data



Network 1

Landmark Discovery

Disease




Network P

...

Image Retrieval

...Deep Feature

Representation 1

Deep Feature

Representation P

Patch-based

Features


...

Landmark Detection

Landmark

Definition

...

Landmark Discovery

...

Training Data



Network 1

Landmark Discovery

Disease




Network P

...

Image Retrieval

...Deep Feature

Representation 1

Deep Feature

Representation P

Patch-based

Features


...

Landmark Detection

Landmark

Definition

...

Test MRI

...

Training Data



Network 1

Landmark Discovery

Disease




Network P

...

Image Retrieval

...Deep Feature

Representation 1

Deep Feature

Representation P

Patch-based

Features


...

Landmark Detection

Landmark

Definition

...

Landmark Detection

...

Training Data



Network 1

Landmark Discovery

Disease




Network P

...

Image Retrieval

...Deep Feature

Representation 1

Deep Feature

Representation P

Patch-based

Features


...

Landmark Detection

Landmark

Definition

...

Training MRIs

Pre-processed

MR Images

…

…

CNN

Deep Multi-channel Convolutional

Neural Network (CNN)

DiseaseClassification

Clinical ScorePrediction

* Featured Article of IEEE Journal of Biomedical and Health Informatics, 2018

Anatomical Landmark-based Deep Network





00.01 p-values

1,740 landmarks via group

comparison between AD and NCTop 50 landmarks

Anatomical Landmark-based Deep Network







Concatenated

sMRI

FC8-32 FC8-32FC8-32 …

FC9-32L

Conv1-

32@3×3×3

Conv2-

32@3×3×3

Max-pooling

Conv3-

64@2×2×2

Conv4-

64@2×2×2

Max-pooling

Conv5-

128@2×2×2

Conv6-

128@2×2×2

Max-pooling

Conv1-

32@3×3×3

Conv2-

32@3×3×3

Max-pooling

Conv3-

64@2×2×2

Conv4-

64@2×2×2

Max-pooling

Conv5-

128@2×2×2

Conv6-

128@2×2×2

Max-pooling

FC7-128

…

FC7-128

Conv1-

32@3×3×3

Conv2-

32@3×3×3

Max-pooling

Conv3-

64@2×2×2

Conv4-

64@2×2×2

Max-pooling

Conv5-

128@2×2×2

Conv6-

128@2×2×2

Max-pooling

Patch 1 Patch 2 … Patch L

Landmark-based Deep Network

FC7-128

FC10-8L

FC11-2

Class Label

Soft-max

Global

Image-level

Representation

Local

Patch-level

Representation

Global

Representation?







0.431

Ove

rall

Ac

cu

rac

y

0.404

0.46

7

0.486 0.487

0.518

0.325

Co

rre

lati

on

Co

eff

icie

nt

0.289

0.468

0.492

0.538

0.567

Classification Results for AD vs. sMCI vs. pMCI vs. NC Regression Results for MMSE

Results of Classification and Regression







End-to-end Disease Diagnosis with sMRI

• Hierarchical Fully Convolutional network


M. Liu, J. Zhang, C. Lian, and D. Shen. IEEE Trans. on Cybernetics, 2019.

M. Liu, J. Zhang, E. Adeli, and D. Shen. IEEE Trans. on Biomedical Engineering, 2019.



J. Zhang, M. Liu, and D. Shen. IEEE Trans. on Image Processing, 2017.





15

Hierarchical Network for ROI Identification

• Hierarchical Fully Convolutional Network (H-FCN)

– Automatically and identify disease-related ROIs in the whole sMR image

– Jointly learn multi-scale features and construct a classification model


C. Lian, M. Liu*, L. Wang, and D. Shen. MICCAI 2019.






Input:

sMRI







Input:

sMRI

1) Location

proposals








1) Location

proposals

Input:

sMRI

2) Patch-level sub-

networks (PSN) (shared

weights)


PSN

PSN

Class_P

64 64 64128 12832







PSN

1) Location

proposals

… …

PSN

PSN

PSN

PSN

PSN

PSN

PSN

Input:

sMRI

2) Patch-level sub-


weights)


PSN

Class_P

64 64 64128 12832







…

64

Conv_R Class_R

…

PSN

PSN

PSN

PSN

PSN

PSN

PSN

PSN

2) Patch-level sub-


weights)

3) Region-level

sub-networks

1) Location

proposals

Input:

sMRI


PSN

Class_P

64 64 64128 12832







…

64

Conv_R Class_R

…

64

Conv_R Class_R

…

PSN

PSN

PSN

PSN

PSN

PSN

PSN

PSN

2) Patch-level sub-


weights)

3) Region-level

sub-networks

1) Location

proposals

Input:

sMRI


PSN

Class_P

64 64 64128 12832







…

64

Conv_S

Class_S

64

Conv_R Class_R

64

Conv_R Class_R

… …

Output:

Class label

PSN

PSN

PSN

PSN

PSN

PSN

PSN

PSN

3) Region-level

sub-networks

4) Subject-level

sub-network

1) Location

proposals

Input:

sMRI

2) Patch-level sub-


weights)

Classification (1 × 1 × 1 Conv)

4 × 4 × 4 Conv

1 × 1 × 1 Conv

Channel concatenation

Region-level Conv

2 × 2 × 2 max pooling

3 × 3 × 3 Conv

Spatial concatenation

Subject-level Conv

Potentially pruned sub-networks

Skipped connection Conv: Convolution

PSN

Class_P

64 64 64128 12832




Network Pruning: To remove less informative regions





Identified Voxel-level Discriminative Locations

1

2

3

1 2 3

1 2 3

1

2

3 4

3 4

3 4

1 2

1 2

1 2 3

1 2 3

1 2 3

1 2 3

1

2 3

1 2

1 2

12

12

3

1 2 3

1 2 3

12

3

(a) AD Subject #1 (b) AD Subject #2 (c) AD Subject #3

(d) AD Subject #4 (e) AD Subject #5 (f) AD Subject #6







Identified Patch-level Discriminative Locations

Sagittal View Axial View Coronal View 3D View







Identified Region-level Discriminative Locations

Sagittal View Axial View Coronal View 3D View







0.5

0.6

0.7

0.8

0.9

1

Patch Region Subject

AC

C

0.5

0.6

0.7

0.8

0.9

1

Patch Region Subject

AU

C

H-FCN before network pruning H-FCN after network pruning

Results of AD vs. NC classification obtained by patch-, region-, and subject-level sub-

networks in H-FCN without /with network pruning

Classification Results



1) Global subject-level representation is more useful

2) Network pruning promotes the classification performance





Outline

• Missing Data




CSFPETsMRI



sMRI





Part II. Multi-modal Neuroimage Analysis

• Multi-modality Fusion for Disease Diagnosis

• Imaging Synthesis for Missing Modalities

• Domain Adaptation for Multi-site Data





http://www.unc.edu/

http://www.unc.edu/

M. Liu, J. Zhang, P.-T. Yap, and D. Shen. MICCAI, 2016.

M. Liu, J. Zhang, P.-T. Yap, and D. Shen. Medical Image Analysis, 2017.

M. Liu, Y. Gao, P.-T. Yap, and D. Shen. IEEE Journal of Biomedical and Health Informatics, 2018.

Multi-modality Fusion for Disease Diagnosis

• Hypergraph Learning with Missing Data





30

Multi-View Data Grouping

• Given 3 modalities (i.e., MRI, PET and CSF), 6 views are constructed

according to the availability of different modalities

Multi-Modality Data

CSFPET MRI

Multi-ViewData Grouping

View 1

View 2

View 3

View 4

View 5

View 6

PET

CSF

MRI

Missing Data

* MICCAI Young Scientist Award Nomination, 2016

* MICCAI Travel Award, 2016





31

Multi-View Data Grouping

• Given 3 modalities (i.e., MRI, PET and CSF), 6 views are constructed

according to the availability of different modalities

Multi-Modality Data

CSFPET MRI


View 1

View 2

View 3

View 4

View 5

View 6

PET

CSF

MRI

Missing Data

* MICCAI Young Scientist Award Nomination, 2016

* MICCAI Travel Award, 2016





• Construct multiple hypergraphs, with each hypergraph corresponding to

a specific view

Multi-Modality Data

CSFPET MRI


View 1

View 2

View 3

View 4

View 5

View 6

Sparse

Representation

Sparse

Representation

Sparse Representation based Hypergraph Construction

Sparse

Representation

Sparse

Representation

Sparse

Representation

Sparse

Representation

PET

CSF

MRI

Missing Data




Hypergraph Construction





• A view-aligned hypergraph classification model to capture the coherence

among different views

View-Aligned

Hypergraph Classification

View-Aligned Hypergraph Classification

Multi-Modality Data

CSFPET MRI


View 1

View 2

View 3

View 4

View 5

View 6

Sparse

Representation

Sparse

Representation


Sparse

Representation

Sparse

Representation

Sparse

Representation

Sparse

Representation

PET

CSF

MRI

Missing Data








• Multi-view label fusion strategy (weighted voting)

Multi-View

Label Fusion

Multi-Modality Data

CSFPET MRI


View 1

View 2

View 3

View 4

View 5

View 6

Sparse

Representation

Sparse

Representation


Sparse

Representation

Sparse

Representation

Sparse

Representation

Sparse

Representation

View-Aligned

Hypergraph Classification

PET

CSF

MRI

Missing Data




Multi-view Label Fusion





Label Space

𝐱𝑛1MRI

𝐱𝑛1PET 𝐱𝑛1

CSF

𝑓𝑛1MRI

𝑓𝑛1PET

𝑓𝑛1CSF

𝑓𝑛2MRI

𝑓𝑛2PET

𝐱𝑛2MRI𝐱𝑛2

PET

View-Aligned Constraint

Coherence among views

View-Aligned Regularizer




Missing CSF

MRI

PET

Subject #2

CSF

MRI

PET

Subject #1





36

𝑚𝑖𝑛𝐅, 𝛂, {𝐖𝑚}𝑚=1

𝑀 𝑚=1𝑀 𝛀𝑚 (𝐟𝑚 − 𝐲) 2

2

+𝜆 𝑚=1𝑀 𝐖𝑚

𝐹2

+𝜇 𝑛=1𝑁 𝑚=1

𝑀 𝑝=1𝑀 𝛺𝑛,𝑛

𝑚 𝛺𝑛,𝑛𝑝

𝑓𝑛𝑚 − 𝑓𝑛

𝑝 2

+ 𝑚=1𝑀 𝛼𝑚 2 𝐟𝑚 T 𝐋𝑚𝐟𝑚

𝑠. 𝑡. 𝑚=1𝑀 𝛼𝑚 = 1, ∀𝛼𝑚 ≥ 0;

𝑖=1𝑁𝑒𝑚

𝑊𝑖,𝑖𝑚 = 1, ∀𝑊𝑖,𝑖

𝑚 ≥ 0.

Step 3: View-Aligned Hypergraph Classification

Formulation of view-aligned hypergraph classification (VAHC)

View-aligned Regularizer

Hypergraph Laplacian matrix , and .








Learned Weights for Views

Learned weights for views in four classification tasks

AD vs. NC MCI vs. NC pMCI vs. sMCI pMCI vs. NC

0.0

0.1

0.2

0.3

0.4

0.5

0.6

We

igh

ts

MRI PET CSF PET+MRI

MRI+CSF PET+MRI+CSF




Using all 3 modalities (MRI+PET+CSF) achieves the best results





Experimental Results

pMCI vs. sMCI classification

ACC SEN SPE BAC PPV NPV AUC

50

60

70

80

90

100

Re

su

lts (

%)

(c) pMCI vs. sMCI classification

Zero KNN EM SVD VAHL








Imaging Synthesis for Missing Modalities

• Deep Learning based Automatic PET Synthesis from sMRI

Y. Pan, M. Liu*, C. Lian, Y. Xia, and D. Shen. MICCAI, 2019.

Y. Pan, M. Liu, C. Lian, T. Zhou, Y. Xia, and D. Shen. MICCAI, 2018.

Y. Pan, M. Liu*, C. Lian, L. Yue, S. Xiao, Y. Xia and D. Shen. ISMRM, 2019





Pre-processed

MR and PET

Images

Stage 2:

Brain Disease Classification

Stage 1:

PET Image Synthesis

based on sMRI

Synthesizing Missing PET based on MRI for Brain Disease Diagnosis

Hybrid Cycle-GAN for Missing PET Synthesis

Y. Pan, M. Liu, C. Lian, T. Zhou, Y. Xia, and D. Shen. MICCAI, 2018


• Among 800+ subjects in ADNI, all subjects have sMRI, while

only half of them have FDG-PET scans.

MRI PET





Hybrid cycle-consistent generative adversarial network (HGAN) to impute missing PET scans based on sMRI

Synthetic PET

Synthetic MRI3232

Residual Net

Block (RNB)

Real PET

1 128 64 32 16

𝑫𝑷

112864

3216

𝑫𝑴

1 16 32 32 16 8

𝑮𝑴

⋯

RN

B

RN

B

8 16 32

𝑮𝑷

⋯

RN

B

RN

B

32 16 1

3×3×3 Convolution

7×7×7 Convolution

3×3×3 Deconvolution

4×4×4 Convolution

Addition

Real MRI

6

6

1/01/0

𝐗𝑀

𝐗𝑃

𝐺1 𝐗𝑀

𝐺2 𝐗𝑃








Synthetic PET ImagesP

ET

(RID

:5016)



(a) GAN (b) CGAN (c) VGAN (d) HGAN (Ours) (e) Ground Truth

PE

T(R

ID:

4352)





Synthetic MRI ScansM

RI

(RID

:5016)



(a) GAN (b) CGAN (c) VGAN (d) HGAN (Ours) (e) Ground Truth

MR

I(R

ID:

4352)





PET

(Real + Synthetic)

MRI (Real)

24

2424

24

24

24

24

2424

24

2424

…

3232 32 6464 64

Sub-network 1

DCM DCM DCM

1616 16

Sub-network 2

3232 32 6464 64

DCM DCM DCM

1616 16

Concatenation

32

8K …

Clinical scores at

four time-points

8

32

8

128

32

BL

M06

M24

M12

Fully-connectedDown-sampling Copy 3×3×3 Convolution 2×2×2 Max-poolingChannel concatenation


Landmark-based Deep Network for Brain Disease Classification using MRI and PET (Real+Synthetic)



PET

(Real + Synthetic)





MCI conversion prediction with complete MRI

and complete (after imputation) PET








How to generating classification-oriented

PET/MRI scans for diagnosis?

Feature-consistent GAN for Joint PET Synthesis and Classification

1128643216

𝐷

3×3×3 Conv

7×7×7 Conv

3×3×3 Deconv

4×4×4 Conv

Addition

Difference

32 32

RNB

1163232168

𝐺

RN

B

RN

B ⋯

6

𝐺𝑀

𝐺𝑃

𝐷𝑃

𝐷𝑀 𝔏𝑔

𝔏𝑔

𝐹𝑃

𝐹𝑀 𝔏c

𝔏c

FG

AN

Synthetic

MRI

Real

PET

Synthetic

PET

Real

MRI

Synthetic

PET

Real

PET

1 2

K

1

2

K

⋮1/0

𝑙2

64643216 64

64643216

𝔏c

64

Feature-consistent

Component 𝐹𝑃

Cosin

e K

ern

el

Y. Pan, M. Liu*, C. Lian, Y. Xia, and D. Shen. MICCAI, 2019

Feature-consistent

Component 𝐹𝑃





30

50

70

90

AUC ACC SEN SEP

ROI LMF LDMIL DSNN (Ours)

MCI conversion prediction with complete MRI

and complete (after imputation) PET

Feature-consistent GAN for Joint PET Synthesis and Classification

Y. Pan, M. Liu*, C. Lian, Y. Xia, and D. Shen. MICCAI, 2019

Generating task-oriented PET scans helps boost performance





Domain Adaptation for Multi-site Data

• Low-rank Representation for Multi-site Data Adaptation

M. Wang, D. Zhang, J. Huang, D. Shen, and M. Liu*, and D. Shen. MICCAI, 2018





Low-rank Representation for Domain Adaptation

ABIDE: 17 imaging sites with resting-state fMRI data

Scanners

ScanningParameters

PopulationNoise Level

ImageContrast


*Best Poster Award, MICS, 2019





50

…

P2

P1

PS

P

New Representation

Latent Representation Space

Linear

Representation

Site T

Site S

Site 1

Site 2



Mapping to a

common latent spaceRepresenting source

data using target data

Source Domains

Target Domain






51


Formulation of multi-center low-rank representation (maLRR)

min𝐉,𝐏,𝐏𝑖,𝐙𝑖,𝐄𝑆𝑖 ,𝐄𝑃𝑖 ,𝐅𝑖𝐉 ∗ + 𝑖=1

𝐾 𝐅𝑖 ∗ + 𝛼 𝐄𝑆𝑖 1+ 𝛽 𝐄𝑃𝑖 1

s. t. 𝐏𝑖𝐗𝑆𝑖 = 𝐏𝐗𝑇𝐙𝑖 + 𝐄𝑆𝑖 ,

𝐏𝑖= 𝐏 + 𝐄𝑃𝑖 , 𝑖 = 1,… , 𝐾

𝐏 = 𝐉, 𝐙𝑖 = 𝐅𝑖 , 𝐏𝐏𝑇 = 𝐈1) Mapping to a latent space

2) Representing source data

using target data







52


Performance of different methods in the task of Autism vs. NC classification, with

NYU as the target domain and the other four sites as the source domains

ABIDE: 5 imaging sites with resting-state fMRI data







Outline

• Missing Data




CSFPETsMRI



sMRI





Acknowledge

Dr. Dinggang Shen

Dr. Daoqiang Zhang

Dr. Pew-Thian Yap

Dr. Jun Zhang

Dr. Chunfeng Lian

Dr. Ling Yue

Dr. Jing Zhang

Dr. Aimei Dong

Dr. Bo Wang

Collaborators

Dr. Ehsan Adeli

Dr. Yue Gao

Dr. Biao Jie

Dr. Tao Zhou

Dr. Yong Xia

Visiting Scholars and Students

Mr. Mingliang Wang

Mr. Yongsheng Pan

Mr. Dongren Yao

Mr. Jiashuang Huang





Thanks for Your Attention!

[email protected]

http://mingxia.web.unc.edu/

Mingxia Liu





http://www.unc.edu/

http://www.unc.edu/

neuroimage analysis for automated brain disease...

Documents