computational anatomy modeling of abdominal organs and

Computational Anatomy Modeling of Abdominal Organs and Musculoskeletal Structures

Yoshinobu Sato

Graduate School of Information ScienceNara Institute of Science and Technology (NAIST)

Japan

Symposium on Statistical Shape Models & ApplicationsDelémont, Switzerland

June 11‐13, 2014

Imaging-based Computational Biomedicine Lab

Osaka University

Osaka

Kyoto

Nara

NAISTTokyo

Information ScienceMaterial ScienceBiological Science

NAISTNara Institute of Science

and Technology

Statistical Shape Models (SSMs) & Applications in this talk

Abdominal Organs

Musculoskeletal Structures

Implants & Host Bones

Hierarchical SSM

Conditional SSMMuscles

PLSR prediction‐based conditional SSMs & probabilistic atlas

Non‐conditional Conditional

SSM & statistical distance maps

• Our computational anatomy project: Overview

• Anatomy modeling– Abdominal anatomy– Musculoskeletal anatomy– Whole‐body anatomy

• Therapeutic modeling– Surgeon’s expertise modeling

• Artificial joint surgery (Total Hip Arthroplasty: THA)

Outline

http://www.comp‐anatomy.org/Google search by “computational anatomy”.

Locations of eight core groups

MEXT Grant‐in‐aid for Scientific Research, JapanComputational Anatomy for Computer‐Aided Diagnosis and TherapySep 2009 ‐Mar 2014Fund: $10 millionPrincipal Investigator: Prof. Hidefumi Kobatake(TUAT: Tokyo University of Agriculture & Technology)Eight core groups

Basic theories and technologiesApplication systemsClinical evaluations

The aim was to develop computational anatomy models of the human body (especially in torso), which represent inter‐subject variability of anatomy across a population, and their applications.

One of our goals: Complete understanding of whole‐body CT images


Osaka Univ.(My former affiliation)


Basic theories and technologies (Tokyo, Osaka, Gifu)Application systemsClinical evaluations

Conventional Representation of Human Anatomy

• Book Atlas– Detailed illustrations of

typical anatomy

• 3D Digital Atlas– Detailed segmented 3D data of

a specific subject

Frank H. Netter, Atlas of Human Anatomyhttp://www.voxel‐man.de/

Visible Human data (NIH)

Semi‐automated segmentation

VOXEL‐MAN (Univ. Hamburg)

They are constructed by Manual Drawing or Semi‐automated Segmentation.They only show One Typical Example or One Particular Example.

3D Digital Atlas

Visible Human DataSemi‐automated segmentation

One Particular Anatomy

Reconstructed from Special data with Labor‐intensive efforts


Visible Human DataSemi‐automated segmentation

One Particular Anatomy

Reconstructed from Special data with Labor‐intensive efforts


GoalPatient 3D Data

Fully‐automated segmentation

Patient‐Specific Anatomy(equivalent to Visible Human & VOXEL MAN)

From Clinical data as Routine work

GoalPatient 3D Data


From Clinical data as Routine work


ApproachPatient 3D Data


Reconstructed from Clinical data as Routine work

Computational Anatomy Models Representing Inter‐Patient Variability of Multiple Organs

….….

Atlas (training) datasets

Shape & Location Priors in Bayesian Estimation






Outline

Target Abdominal Organs

• Liver (brown) • Spleen (blue violet)• Kidneys (pink)• Pancreas (yellow)• Gallbladder (green) • Aorta and artery

branches (red) • Inferior vena cava (IVC)

and vein branches (cyan)

• Upper GI tract (cream yellow)

Segmented OrgansToshi Okada, PhD

(Currently, University of Tsukuba)

Masatoshi Hori, MD

Organ segmentation via computational anatomy

Conventional framework

Computational Anatomy (CA) Model

Automated Construction

AutomatedSegmentation

Abdominal CT

Target 3D data Patient

anatomy

[Okada MICCAI 2007][Okada Acad Radiol 2008]

Manually‐traced organ shape data

Training dataLabeled DICOM data

Intensity priorsShape and location priors*

*Inter‐Patient Anatomical Variability of Organ Shape and Location

Inter‐Patient Anatomical Variability of Organ Shape: Conventional Representation

Segmentation ≒ Voxel‐wise MAP (Maximum a Posterior) estimation (Initialization is unnecessary after spatial normalization.)

Probabilistic Atlas (PA)Voxel‐wise probability map of organ existence

in the normalized abdominal space

[Okada et al. MICCAI 2007] [Park et al. TMI 2003]

Inter‐Patient Anatomical Variability of Organ Shape: Conventional Representation

Segmentation ≒ Statistically constrained deformable model fitting ≒ Global MAP estimation (Initialization is needed.)

Statistical Shape Model (SSM) (PCA of 3D shape)Statistical constraints (inter‐patient variability) on shape and location

in the normalized abdominal space

[Okada et al. MICCAI 2007] [Lamecker et al. 2004]

Roles of SSM from the mathematical viewpoint:

• Effective (Dimensionality) Reduction of Solution Space (fewer parameters for representing target shapes)

• Prior Probability Distributions in Bayesian Estimation

e1

eN

e2

)|()()|( MDPMPDMP

n‐dimensional solution space representing all shapes

v2

v1

Reduced mL‐d solution space for possible liver shapes (mL<<n)

Reduced mF‐d solution space for possible femur shapes (mF<<n)

P(M)

v1

e3

Prior LikelihoodPosterior

Conventional Method [Okada MICCAI 2007] +

A Single Organ Segmentation Method

CT image

Spatial standardization

Initial segmentation by PA

SSM refinement

Graph‐cut refinement

Segmentation result

Probabilistic Atlas (PA)

Statistical Shape Model (SSM)

Intensity Model

Liver

[Okada MICCAI 2007]

+ Likelihood

Prior

Conventional Method [Okada MICCAI 2007] +

A Single Organ Segmentation Method

CT image

Spatial normalization


SSM refinement


Segmentation result

Probabilistic Atlas (PA)

Statistical Shape Model (ML‐SSM)

Intensity Model

Right kidney

[Okada MICCAI 2007]

+ Likelihood

Prior

Extension to multi‐organ modeling and segmentation

Organ segmentation via computational anatomyConventional framework

Computational Anatomy (CA) Model



Abdominal CT


anatomy

[Okada MICCAI 2007][Okada Acad Radiol 2008]



Intensity priorsShape and location priors*

LimitationsInter‐relations among organs are not utilized.

• Given predictor organs P, PLSR predicts the target organ shape. The prediction error E(P) is given by

E(P) = S ‐ S’(P) (S is true shape and S’(P) predicted shape.)

PLSR (Partial Least Squares Regression)Prediction‐based Conditional Priors

…

Training data Predictor organs P

PLSR predictor S’(P)

Predicted target shape S’

Training Phase Execution Phase

Predictor

Target


• Given predictor organs P, PLSR predicts the target organ shape. The prediction error E(P) is given by

E(P) = S ‐ S’(P) (S is true shape and S’(P) predicted shape.)• Among all possible combinations of predictor organs, predictor

organs Pminimizing prediction error E(P) are selected for each target organ, which define arcs of OCG (organ correlation graph).

PLSR (Partial Least Squares Regression)Prediction‐based Conditional Priors

…

Training data Predictor organs P

PLSR predictor S’(P)

Predicted target shape S’

Training Phase Execution Phase

Predictor

Target


P(Liver) P(Spleen|Liver)

P(L‐Kidney|Liver,Spleen)

P(Pancreas|Liver,Spleen)


P(Gallbladder|Liver)

P(R‐Kidney|Liver)

Organ correlation graphConditional shape & location prior (SSM & PA) network

Anchor organ

Predictor organ

Predictor organ

Predictor organ

Prediction‐based Statistical AtlasProbabilistic Atlas (PA)

• Prediction error E is modeled as probabilistic atlas (PA) to generate less ambiguous PA. E = S ‐ S’ (S: True shape, S’: Predicted shape, E: Prediction error)


P(R‐Kidney|Liver)

P(Pancreas)

P(R‐Kidney)

P(Gallbladder|Liver )P(Gallbladder)

Prediction‐based (Conditional)Conventional

P(Liver)P(Spleen|Liver)

P(L‐Kidney|Liver)

P(Pancreas|Liver)

P(Gallbladder|Liver)

P(R‐Kidney|Liver)

Organ correlation graph (OCG)Conditional shape & location prior (SSM & PA) network

Anchor organ

Predictor

Predictor

Predictor

Predictor

Predictor


Probabilistic Atlas using Known Liver Shape

Prediction‐based Statistical AtlasProbabilistic Atlas (PA)

• Prediction error E is modeled as probabilistic atlas (PA) to generate less ambiguous PA. E = S ‐ S’ (S: True shape, S’: Predicted shape, E: Prediction error)

Prediction‐based (Conditional)ConventionalPredictor: Liver Predictor: Liver, Spleen, Kidneys

Prediction‐based Statistical AtlasStatistical Shape Model (SSM)

• The prediction error E is also modeled using PCA in prediction‐based SSM to obtain more constrained variability.E = S ‐ S’ (S: True shape, S’: Predicted shape, E: Prediction error)


P(R‐Kidney|Liver)P(R‐Kidney)

P(Gallbladder|Liver )P(Gallbladder)

Prediction‐based (Conditional)ConventionalP(Pancreas)

Prediction‐based Segmentation Method

CT image

Spatial standardization


ML‐SSM refinement


Segmentation result

Prediction‐based PA

Prediction‐based SSM

Intensity Model

Segmentation results of predictor organs

Organ segmentation via computational anatomy

Multi‐organ interrelation modeling

Target‐data specific model

Customized Computational Anatomy (CA) Model



Abdominal CT

AutomatedCustomization


anatomy



Intensity priors

Generic Computational Anatomy (CA) Models

Shape and location priors

Multi‐organ modeling inherent in anatomy

[Okada Abd‐ImgWS 2011]

[Okada EMBC 2012]

Intensity prior modeling (IM)• In abdominal CT segmentation, we have to deal with a variety of contrast enhancement (CE) patterns.

• A new intensity prior model (IM) has to be constructed to deal with a new CE pattern.

Contrast‐enhancedLate arterial phase

Non (blood) contrast but oral contrast

Contrast‐enhancedVenous phase

Contrast‐enhancedEarly arterial phase

Intensity prior modeling (IM)• Supervised intensity modeling (IM) : Conventional

– Intensity prior modeling from “labeled” DICOM dataset• A set of CT images and manual traces on them for each CE

• Unsupervised intensity modeling (IM): Proposed– Intensity prior modeling from “unlabeled” DICOM dataset

• A set of CT images but no traces for each CE pattern

– Target data specific (no training dataset for IM)

Contrast‐enhancedLate arterial phase




Organ segmentation via computational anatomyTowards easily customizable and extendable systems

Target‐data specific model




Abdominal CT


Target 3D data

Patient anatomy

[Okada Abd‐ImgWS 2011]

[Okada EMBC 2012]



Intensity priors




no




Imaging‐condition/Target‐data specific model





Abdominal CT


Target 3D data

Patient anatomy

[Okada MICCAI 2013]


Training data Unlabeled DICOM of specific imaging method/protocol


Intensity priors

Joint segmentation and intensity modeling




Imaging‐condition/Target‐data specific model




Abdominal CT


Target 3D data

Patient anatomy

[Okada MICCAI 2013]


Training data Joint segmentation and intensity modeling


Intensity priors

Cope with Unknown Imaging Condition

Results

Experiments• Upper abdominal CT data at two different hospitals were used.

– Non‐contrast (but artifact due to oral contrast) at NIH: 12 cases– Venous phase at NIH: 25 cases– Early and late arterial phases at Osaka Univ. Hospital

• Old protocol: Slice thickness 2.5 mm: 10 cases for each phase• New protocol: Slice thickness 0.625 mm: 39 cases for each phase

– Totally, CT data of 134 cases (86 patients) with 4 different CE patterns were used.

• 2‐fold cross validation was performed. CT data with the same CE pattern as test data were not involved in any parameter tuning.

• The segmentation methods were fully automated.Contrast‐enhancedLate arterial phase




Case 1 (Osaka, Late arterial phase)

Ground truth Prediction‐based CA(Unsupervised IM)

Conventional CA(Unsupervised IM)

Conventional CA(Supervised IM)

Jaccard Index Liver Spleen R-Kidney L-Kidney Pancreas Gallbladder Aorta IVC

Prediction 0.941 0.980 0.963 0.747 0.543 0.935 0.681

Basic(IC-IM)

0.936 0.985 0.964 0.430 0.591 0.833 0.467

Basic(Supervised IC-IM) 0.940 0.984 0.963 0.578 0.933 0.817 0.438

0.916

• Pancreas, aorta, and IVC were better segmented in the proposed prediction‐based method than our conventional method.

Prediction(Unsupervised IM)

Conventional(Unsupervised IM)

Conventional(Supervised IM)

GI‐tract

Conventional Prediction‐based

Conventional

Prediction‐based

Ground truth

Stomach

Esophagus

Duodenum

[Hirayama, 2013]

Summary of abdominal multi‐organ segmentation

• Multi‐organ modeling and segmentation methods were proposed which effectively utilize the organ interrelations.

• Unsupervised intensity prior modeling combined with prediction‐based CA models can make the method adaptive to different CE patterns.

• Once key organs are segmented, other structures including GI‐tract, vessel branches, and tumors are effectively segmented and anatomically identified.





Outline

Musculoskeletal anatomy

Muscle tissues

17 Muscles

FutoshiYokota, MS

Pelvis & Femur

Nobuhiko Sugano, MD

Masaki Takao, MD

Diseased hip joint

Unaffected hipPrimary

osteoarthritisSecondary

osteoarthritis( Crowe 1)

Secondary osteoarthritis( Crowe 2)

Collapsed hip

[Yokota, MICCAI 2013]

100 CT data of Total Hip Arthroplasty (THA) patients:All patients had healthy hip on one side and diseased the other

1. Globally consistent initial segmentation using hierarchical hip SSM2. Accurate segmentation of joint part using conditional SSMs3. Final refinement by graph cut

Approach of bone segmentation

Hierarchical hip SSM

Conditional femoral head SSM


More AccurateMore RobustSpecificity

>Generality

<

[de Bruijne MICCAI 2006]


Conditional SSM

Given partPelvis and distal femur

Conditionalfemoral head SSM

Standardfemoral head SSM


[de Bruijne MICCAI 2006]

Segmentation by Hierarchical SSM fitting

• Initial rough segmentation of bone regions using simple thresholding where joints part is not separeted.

[Yokota et al. MICCAI 2009]


• Coarse fine fitting of hierarchical SSM is performed.



• Coarse fine fitting of hierarchical SSM is performed.– Initial fitting of combined pelvis and femur SSM

– Subsequent fitting of pelvis & femur SSMs with consistency constraint

– Fitting and edge updating are repeated.


CT image Ground truth Independent SSMs Conditional SSM

Primary osteoarthritis



Collapsed hip

ResultsRed: pelvis Green: femur

Musculoskeletal anatomyPelvis & Femur Muscle tissues 17 Muscles

Different patients

Initialbone & skin segmentation

Hierarchical multi‐atlas label fusionAutomatically segmented patient label images

Atlas datasets

Finalsegmentation

Skin, pelvis & femur

Final stage: 17 muscle segmentation

Intensity images

Label images for label fusion

Target CT image First stage:

Muscle tissue segmentation

Muscle tissue

….

2 datasets

….

38 datasets

5 selected muscles

Second stage: 5 selected muscle segmentation

….

38 datasets

….….

Automatically segmented patient label image

Label images for spatial normalization (cancelation of variability)

[Yokota, CAOS 2012]Best Technical Paper Award

Musculoskeletal segmentationResults

Three‐stage(1.9 mm error)

Two‐stage(3.0 mm error)

Single‐stage(4.1 mm error)

Front views

Back views

[Yokota, CAOS 2012]

Musculoskeletal segmentationResults

Three‐stage(1.9 mm error)

Two‐stage(3.0 mm error)

Single‐stage(4.1 mm error)

Ground truthOriginal CT images

[Yokota, CAOS 2012]





Outline

One of our main goals: Complete understanding of whole‐body CT images



Basic theories and technologiesApplication systemsClinical evaluations

Locations of eight core groups

Tokyo

TUATNagoya

Gifu

Kyushu

Yamaguchi

Tokushima

Osaka

Example of collaborationAbdominal module (Tokyo & Osaka)

Landmark Localization

Abdominal Bounding‐box Localization

Abdominal Multi‐organ Segmentation

, , … ,

Training data

, , … ,

Random forest regression

OsakaTokyo & OsakaTokyo

Prof. Masutani(Univ. of TokyoCurrently, Hiroshima City Univ.)

Musculoskeletal modules

(Gifu, OsakaTokushima)

Lung module (Tokushima)

Vessel modules (Nagoya, Osaka)

Prof. Mori(Nagoya Univ.)

Prof. Fujita(Gifu Univ.)

Prof. Niki(Univ. Tokushima)

Non‐contrast CTFully‐automated Segmentation





Outline

Cup planning of mildly and severely diseased pelvises: Our problem

Mildly diseased case Severely diseased case• The position and size of the acetabular cup should be basically determined

so as to recover the original anatomy of the acetabulum.

Cup planning of mildly and severely diseased pelvises: Our problem

Mildly diseased case Severely diseased case• The position and size of the acetabular cup should be basically determined

so as to recover the original anatomy of the acetabulum. • Although it is not so difficult to predict the original anatomy for mildly

diseased case, it is somewhat difficult for severely diseased acetabulum due to its severe deformation and shift.

Bone‐Implant Statistical Model (1)Prior probability of likely spatial relations

between patient bone and implant

Patient Pelvis Shape Data: D

Cup Plan

Pelvis‐Cup Statistical Model P(Xpelvis, Xcup)

Surgical Plan Database

Automated Planning

Statistical Analysis

Maximize P(Xpelvis, Xcup)P(D|Xpelvis)

Statistical Shape Model (SSM)

Otomaru et al.CAOS 2009

Maximum a Posterior (MAP) Estimation

Bone‐Implant Statistical Model (2)Prior probability of likely spatial relations

between patient bone and implant

Patient Femoral Cavity Shape Data: D

Stem Plan

Femoral Cavity ‐ Stem Statistical Model P(Xfemur, Xstem)

Surgical Plan Database

Automated Planning

Maximize P(Xfemur, Xstem)P(D|Xfemur)

Statistical Distance Map (SDM)penetration 0 gap

Otomaru et al.Med Image Anal2012

Maximum a Posterior (MAP) Estimation

Summary of this talk

• Statistical shape models (SSMs) and other statistical atlas representation incorporating interrelations among multiple organs (structures) are presented.

• Their applications were demonstrated to– Abdominal organs– Musculoskeletal structures– Bone implant surgical planning

• These problems are formulated as MAP estimation based on Bayes theorem, where SSMs are regarded as prior probability distributions.

Sunrise at Yakushi Temple, Nara, Japan

Thank you!

computational anatomy modeling of abdominal organs and

Documents