recent clinical advances and applications for medical...

49
© L. Joskowicz, 2011 Recent clinical advances and applications for medical image segmentation Prof. Leo Joskowicz Lab website: http://www.cs.huji.ac.il/~caslab/site

Upload: lycong

Post on 18-Mar-2018

218 views

Category:

Documents


4 download

TRANSCRIPT

Page 1: Recent clinical advances and applications for medical ...crl.med.harvard.edu/research/MICCAI_Tutorial/MICCAI11-tutorial... · Recent clinical advances and applications for medical

copy L Joskowicz 2011

Recent clinical advances and applications

for medical image segmentation

Prof Leo Joskowicz

Lab website httpwwwcshujiacil~caslabsite

copy L Joskowicz 2011

Key trends in clinical radiology bull Filmlight table Digital imagesscreen -- early 80rsquo

bull 2D X-rays US 25D CT MRI -- mid 80rsquo ndash 90rsquo

bull 25D CT MRI 3D visualization -- mid 00lsquo

bull 3D visualization 3D anatomical modeling -- now

bull 3D modeling Advanced modeling ndash coming soon

copy L Joskowicz 2011

Centerline and lumen

Colon and intestine surface Abdominal CT scan

What is a patient-specific 3D model

Surface model

Polyp

copy L Joskowicz 2011

What is the difference between 3D

visualization and 3D modeling

Visualization Model rendering

You do the interpretation

You fillomit missing info

Computer interprets

Explicit delineation

copy L Joskowicz 2011

Why patient-specific modeling

bull 3D visualization is great -- but it has limitationshellip

minus no explicit delineation

minus no validation ndash what you see is what is there

minus limited measurements

minus limited structures discrimination

bull 3D models allow

minus spatial and volumetric measurements ndash with validation

minus advanced analysis

minus wide variety of uses in the treatment cycle

minus reduction of radiologist time easier learning curve

copy L Joskowicz 2011

Segmentation in commercial systems

Copyright L Joskowicz 2007

copy L Joskowicz 2011

Segmentation in commercial systems

Copyright L Joskowicz 2007

copy L Joskowicz 2011

Copyright L Joskowicz 2007

Segmentation in commercial

systems

copy L Joskowicz 2011

3D models in the patient treatment cycle

Diagnosis Planning

Delivery

CAD mammography

Virtual colonoscopy

Neurosurgery -- trajectory

Orthopaedics -- fixation

Tumor follow-up

Implant location

MODEL Interventional

Radiology

Navigation

Robotics Evaluation

Training

Learning

copy L Joskowicz 2011

Diagnosis computer-aided radiology

stenosis

thrombus aneurism

tumor

volume

with Prof J Sosna

copy L Joskowicz 2011

liver contour

blood vessels

tumors

4-phase CT dataset

kidneycontour

blood vessels

urinary vessels with Dr Y Mintz with Prof J Sosna

Diagnosis computer-aided radiology

copy L Joskowicz 2011

Planning neurosurgery

entry point

with Dr Y Shoshan

copy L Joskowicz 2011

Planning neurosurgery

with Dr Y Shoshan

copy L Joskowicz 2011

Conventional Our method

Planning neurosurgery

with Dr Y Shoshan

copy L Joskowicz 2011

Planning orthopaedics

internal fixation external fixation

Fracture

fixation

alternatives

with Dr E Peleg and Profs M Liebergall R Mosheiff

copy L Joskowicz 2011 with Dr E Peleg and Profs M Liebergall R Mosheiff

Planning orthopaedics

copy L Joskowicz 2011

Planning orthopaedics

with Dr E Peleg and Profs M Liebergall R Mosheiff

copy L Joskowicz 2011

bull EM real-time tracking

bull US and X-ray imaging

add patient-specific

models from CT

Delivery interventional radiology

copy L Joskowicz 2011

Delivery intraoperative image guidance

augmented continuous X-ray fluoroscopy

with Simbionix and Prof J Sosna

copy L Joskowicz 2011

Delivery intraoperative image guidance

with Simbionix and Prof J Sosna

copy L Joskowicz 2011

Evaluation brain tumor follow-up

Dec 15 2009 June 9 2010

Disease

progression T2-weighted T1-weighted

Tumor internal components

solid enhancing cyst

with TA Sourasky and Dana Hospital

copy L Joskowicz 2011

Key issue model creation

Currently

bull mostly manual delineation

bull slice by slice

Desired

bull automaticnearly automatic ndash a few

clicks by physician

bull no technician

bull accurate and reliable

HARD NO METHOD

SUITABLE FOR ALL STRUCTURES

Modeling requires segmentation

Very difficult task

ndash organpathology specific

ndash imagescanning protocol

ndash anatomical variability

ndash intensity values overlap

ndash structures proximity

Identify and delineate anatomical structure contours

1 2

3 4

Modeling requires segmentation

Very difficult task

ndash organpathology specific

ndash imagescanning protocol

ndash anatomical variability

ndash intensity values overlap

ndash structures proximity

Identify and delineate anatomical structure contours

1 2

3 4

Clinical dataset challenges

proximity calcifications stenosis

Segmentation ndash state of the art

bull Hundreds anatomical segmentation methods for a

variety of structures and imaging modalities

bull Families of techniques thresholding region

growing level sets active contours and more

bull Very few if any in routine clinical use

ndash huge gap between prototype and clinical use

ndash time-consuming fragile limited in scope

ndash require technical knowledge

ndash most have limited validation

ndash clinical benefits unproven

bull Segmentation specificity is unavoidable ndash each

anatomical structure and pathology has unique

characteristics and nearby structures

bull Organ structure and pathology-specific

algorithms heart liver long bones spine

bull Very laborious top develop and validate a

segmentation algorithm for each ndash trial and error

process many man-months

No universal segmentation method

Observations (1)

Observations (2) bull Applications have different requirements wrt

ndash Accuracy

ndash Robustness

ndash User interaction

ndash Quality

bull Consider the different requirements of

ndash 3D visualization

ndash Training simulation

ndash FEA simulation

Defining application

requirements ahead of time is

essential

Goals Clinical

bull Automatic or nearly automatic lt 5mins user time

bull Can be used by clinician without a technician

bull Produces robust results

Technical

bull Requires significantly less time than developing from

scratch

bull Takes into account intensity and shape

bull Incorporates shape and intensity priors

MIS validation

bull Algorithms without validation are clinically

worthless

bull Validation is with respect to a clinical task

bull Validation requires a ground truth for comparison

bull Physical anatomical models andor phantoms are

typically not available (except sometimes for bones)

bull Ground truth is usually obtained by manual

identification andor segmentation by a user

bull Experts in most cases radiologists

bull Extrinsic comparison compare vs other methods

Copyright L Joskowicz 2010

Copyright L Joskowicz 2010

MIS validation issues

bull Large inter- and intra- variability across experts and clinical sites

bull May not be representative of population variability

bull Main quantitative parameters

ndash validation set size

ndash number and type of observers

ndash intra and inter-observer manual segmentation variability

ndash surface-based and volume-based error measures

MIS validation anatomical variability

stenosis

looping

narrowing

MIS validation anatomical variability

MIS validation metrics

bull Surface-based error measurements

ndash Mean surface distance

ndash RMS surface distance

ndash Maximum surface distance

bull Volume-based error measurements

ndash Dice coefficient

ndash Volumetric overlap error

Copyright L Joskowicz 2010

MIS validation metrics

Surface-based error measurements

bull Mean surface distance

bull RMS surface distance

bull Maximum surface distance

Copyright L Joskowicz 2010

Reference Result

MIS validation metrics

Volume-based error measurements

bull Volumetric overlap error

bull Dice similarity

Copyright L Joskowicz 2010

Reference Result

Aorta

Int + Ext Carotids

All arteries

Subclavian Arteries Bifurcations

16mm (std=08mm)

13mm (std=10mm)

15mm

std=13mm 17mm

std=09mm

07mm

std=07mm

Surface

RMS

Common

Carotid

MIS validation observers variability

bull Intra-observer variability determine how much

variation there is when a single observer

produced the ground-truth segmentation

repeat 5 times the segmentation

bull Inter-observer variability determine how much

variation there is between multiple observers that

produced the ground-truth segmentation

ask 3 radiologists to do the

segmentation

bull Observer expertise and frequency variability

Copyright L Joskowicz 2010

Study [Weltens 2001]

bull Axial MRI slice

bull Nine independent

observers

bull Repeated delineations

bull Interintra observer

variability 30

Key issue fuzzy tumor boundaries

Inter observer variability

Intra-observer variability

8

50

Carotid Lumen Segmentation and Stenosis

Grading Challenge -- The MIDAS Journal

Copyright L Joskowicz 2010

MIS validation manual annotation

Copyright L Joskowicz 2010

MIS validation ground-truth generation

centerlines

(manual)

contours

(manual)

Partial Volume

Segmentation

Reference

standard

from 3PVS

Copyright L Joskowicz 2010

Measurements characteristics

bull Ground-truth not know

bull Repeatability intra-observer variability

bull Reliability inter observer variability

bull Measure correlation between repeated

measurements

bull Intrinsic uncertainty about the tumor volume

bull Accuracy for clinical significance is unknown

Inaccuracy gt Uncertainty

bull Uncertainty may not be improved

bull Goal improve accuracy to obtain

Inaccuracy vs Uncertainty

Results can be

meaningless

Inaccuracy lt Uncertainty

In summaryhellip

bull Medical image segmentation is on the risehellip

bull Basis of many clinical applications

bull Validation is a MUST

bull Many methods and approaches ndash do not re-invent

the wheel

bull I expect patient-specific 3D models to be in the

clinical mainstream in 3-5 years

Copyright L Joskowicz 2011

Segmentation challenges

bull Rigorous quantification and evaluation of

segmentation algorithms performance

bull Shape priors for pathologies

bull Incorporation of functional information from

diffusionperfusion MRI fMRI PET into

segmentation algorithms

Copyright L Joskowicz 2011

Summary (1)

bull Patient-specific anatomy model creation is

currently a major bottleneck in many clinical

applications

bull Automatic anatomical segmentation is essential for

model creation

bull Current tools are limited in scope and coverage

bull Clinical use requires the elimination of the

technician ndash model generation by the physician

Summary (2) bull Great opportunity for the development and

incorporation of anatomy modeling tools in

commercial platforms

bull Growing need and variety of users for anatomical

models

ndash Training simulators surgery rehersal

ndash Intraoperative guidance

ndash Computer Aided Radiology

bull Service providers -- shifting paradigms

Page 2: Recent clinical advances and applications for medical ...crl.med.harvard.edu/research/MICCAI_Tutorial/MICCAI11-tutorial... · Recent clinical advances and applications for medical

copy L Joskowicz 2011

Key trends in clinical radiology bull Filmlight table Digital imagesscreen -- early 80rsquo

bull 2D X-rays US 25D CT MRI -- mid 80rsquo ndash 90rsquo

bull 25D CT MRI 3D visualization -- mid 00lsquo

bull 3D visualization 3D anatomical modeling -- now

bull 3D modeling Advanced modeling ndash coming soon

copy L Joskowicz 2011

Centerline and lumen

Colon and intestine surface Abdominal CT scan

What is a patient-specific 3D model

Surface model

Polyp

copy L Joskowicz 2011

What is the difference between 3D

visualization and 3D modeling

Visualization Model rendering

You do the interpretation

You fillomit missing info

Computer interprets

Explicit delineation

copy L Joskowicz 2011

Why patient-specific modeling

bull 3D visualization is great -- but it has limitationshellip

minus no explicit delineation

minus no validation ndash what you see is what is there

minus limited measurements

minus limited structures discrimination

bull 3D models allow

minus spatial and volumetric measurements ndash with validation

minus advanced analysis

minus wide variety of uses in the treatment cycle

minus reduction of radiologist time easier learning curve

copy L Joskowicz 2011

Segmentation in commercial systems

Copyright L Joskowicz 2007

copy L Joskowicz 2011

Segmentation in commercial systems

Copyright L Joskowicz 2007

copy L Joskowicz 2011

Copyright L Joskowicz 2007

Segmentation in commercial

systems

copy L Joskowicz 2011

3D models in the patient treatment cycle

Diagnosis Planning

Delivery

CAD mammography

Virtual colonoscopy

Neurosurgery -- trajectory

Orthopaedics -- fixation

Tumor follow-up

Implant location

MODEL Interventional

Radiology

Navigation

Robotics Evaluation

Training

Learning

copy L Joskowicz 2011

Diagnosis computer-aided radiology

stenosis

thrombus aneurism

tumor

volume

with Prof J Sosna

copy L Joskowicz 2011

liver contour

blood vessels

tumors

4-phase CT dataset

kidneycontour

blood vessels

urinary vessels with Dr Y Mintz with Prof J Sosna

Diagnosis computer-aided radiology

copy L Joskowicz 2011

Planning neurosurgery

entry point

with Dr Y Shoshan

copy L Joskowicz 2011

Planning neurosurgery

with Dr Y Shoshan

copy L Joskowicz 2011

Conventional Our method

Planning neurosurgery

with Dr Y Shoshan

copy L Joskowicz 2011

Planning orthopaedics

internal fixation external fixation

Fracture

fixation

alternatives

with Dr E Peleg and Profs M Liebergall R Mosheiff

copy L Joskowicz 2011 with Dr E Peleg and Profs M Liebergall R Mosheiff

Planning orthopaedics

copy L Joskowicz 2011

Planning orthopaedics

with Dr E Peleg and Profs M Liebergall R Mosheiff

copy L Joskowicz 2011

bull EM real-time tracking

bull US and X-ray imaging

add patient-specific

models from CT

Delivery interventional radiology

copy L Joskowicz 2011

Delivery intraoperative image guidance

augmented continuous X-ray fluoroscopy

with Simbionix and Prof J Sosna

copy L Joskowicz 2011

Delivery intraoperative image guidance

with Simbionix and Prof J Sosna

copy L Joskowicz 2011

Evaluation brain tumor follow-up

Dec 15 2009 June 9 2010

Disease

progression T2-weighted T1-weighted

Tumor internal components

solid enhancing cyst

with TA Sourasky and Dana Hospital

copy L Joskowicz 2011

Key issue model creation

Currently

bull mostly manual delineation

bull slice by slice

Desired

bull automaticnearly automatic ndash a few

clicks by physician

bull no technician

bull accurate and reliable

HARD NO METHOD

SUITABLE FOR ALL STRUCTURES

Modeling requires segmentation

Very difficult task

ndash organpathology specific

ndash imagescanning protocol

ndash anatomical variability

ndash intensity values overlap

ndash structures proximity

Identify and delineate anatomical structure contours

1 2

3 4

Modeling requires segmentation

Very difficult task

ndash organpathology specific

ndash imagescanning protocol

ndash anatomical variability

ndash intensity values overlap

ndash structures proximity

Identify and delineate anatomical structure contours

1 2

3 4

Clinical dataset challenges

proximity calcifications stenosis

Segmentation ndash state of the art

bull Hundreds anatomical segmentation methods for a

variety of structures and imaging modalities

bull Families of techniques thresholding region

growing level sets active contours and more

bull Very few if any in routine clinical use

ndash huge gap between prototype and clinical use

ndash time-consuming fragile limited in scope

ndash require technical knowledge

ndash most have limited validation

ndash clinical benefits unproven

bull Segmentation specificity is unavoidable ndash each

anatomical structure and pathology has unique

characteristics and nearby structures

bull Organ structure and pathology-specific

algorithms heart liver long bones spine

bull Very laborious top develop and validate a

segmentation algorithm for each ndash trial and error

process many man-months

No universal segmentation method

Observations (1)

Observations (2) bull Applications have different requirements wrt

ndash Accuracy

ndash Robustness

ndash User interaction

ndash Quality

bull Consider the different requirements of

ndash 3D visualization

ndash Training simulation

ndash FEA simulation

Defining application

requirements ahead of time is

essential

Goals Clinical

bull Automatic or nearly automatic lt 5mins user time

bull Can be used by clinician without a technician

bull Produces robust results

Technical

bull Requires significantly less time than developing from

scratch

bull Takes into account intensity and shape

bull Incorporates shape and intensity priors

MIS validation

bull Algorithms without validation are clinically

worthless

bull Validation is with respect to a clinical task

bull Validation requires a ground truth for comparison

bull Physical anatomical models andor phantoms are

typically not available (except sometimes for bones)

bull Ground truth is usually obtained by manual

identification andor segmentation by a user

bull Experts in most cases radiologists

bull Extrinsic comparison compare vs other methods

Copyright L Joskowicz 2010

Copyright L Joskowicz 2010

MIS validation issues

bull Large inter- and intra- variability across experts and clinical sites

bull May not be representative of population variability

bull Main quantitative parameters

ndash validation set size

ndash number and type of observers

ndash intra and inter-observer manual segmentation variability

ndash surface-based and volume-based error measures

MIS validation anatomical variability

stenosis

looping

narrowing

MIS validation anatomical variability

MIS validation metrics

bull Surface-based error measurements

ndash Mean surface distance

ndash RMS surface distance

ndash Maximum surface distance

bull Volume-based error measurements

ndash Dice coefficient

ndash Volumetric overlap error

Copyright L Joskowicz 2010

MIS validation metrics

Surface-based error measurements

bull Mean surface distance

bull RMS surface distance

bull Maximum surface distance

Copyright L Joskowicz 2010

Reference Result

MIS validation metrics

Volume-based error measurements

bull Volumetric overlap error

bull Dice similarity

Copyright L Joskowicz 2010

Reference Result

Aorta

Int + Ext Carotids

All arteries

Subclavian Arteries Bifurcations

16mm (std=08mm)

13mm (std=10mm)

15mm

std=13mm 17mm

std=09mm

07mm

std=07mm

Surface

RMS

Common

Carotid

MIS validation observers variability

bull Intra-observer variability determine how much

variation there is when a single observer

produced the ground-truth segmentation

repeat 5 times the segmentation

bull Inter-observer variability determine how much

variation there is between multiple observers that

produced the ground-truth segmentation

ask 3 radiologists to do the

segmentation

bull Observer expertise and frequency variability

Copyright L Joskowicz 2010

Study [Weltens 2001]

bull Axial MRI slice

bull Nine independent

observers

bull Repeated delineations

bull Interintra observer

variability 30

Key issue fuzzy tumor boundaries

Inter observer variability

Intra-observer variability

8

50

Carotid Lumen Segmentation and Stenosis

Grading Challenge -- The MIDAS Journal

Copyright L Joskowicz 2010

MIS validation manual annotation

Copyright L Joskowicz 2010

MIS validation ground-truth generation

centerlines

(manual)

contours

(manual)

Partial Volume

Segmentation

Reference

standard

from 3PVS

Copyright L Joskowicz 2010

Measurements characteristics

bull Ground-truth not know

bull Repeatability intra-observer variability

bull Reliability inter observer variability

bull Measure correlation between repeated

measurements

bull Intrinsic uncertainty about the tumor volume

bull Accuracy for clinical significance is unknown

Inaccuracy gt Uncertainty

bull Uncertainty may not be improved

bull Goal improve accuracy to obtain

Inaccuracy vs Uncertainty

Results can be

meaningless

Inaccuracy lt Uncertainty

In summaryhellip

bull Medical image segmentation is on the risehellip

bull Basis of many clinical applications

bull Validation is a MUST

bull Many methods and approaches ndash do not re-invent

the wheel

bull I expect patient-specific 3D models to be in the

clinical mainstream in 3-5 years

Copyright L Joskowicz 2011

Segmentation challenges

bull Rigorous quantification and evaluation of

segmentation algorithms performance

bull Shape priors for pathologies

bull Incorporation of functional information from

diffusionperfusion MRI fMRI PET into

segmentation algorithms

Copyright L Joskowicz 2011

Summary (1)

bull Patient-specific anatomy model creation is

currently a major bottleneck in many clinical

applications

bull Automatic anatomical segmentation is essential for

model creation

bull Current tools are limited in scope and coverage

bull Clinical use requires the elimination of the

technician ndash model generation by the physician

Summary (2) bull Great opportunity for the development and

incorporation of anatomy modeling tools in

commercial platforms

bull Growing need and variety of users for anatomical

models

ndash Training simulators surgery rehersal

ndash Intraoperative guidance

ndash Computer Aided Radiology

bull Service providers -- shifting paradigms

Page 3: Recent clinical advances and applications for medical ...crl.med.harvard.edu/research/MICCAI_Tutorial/MICCAI11-tutorial... · Recent clinical advances and applications for medical

copy L Joskowicz 2011

Centerline and lumen

Colon and intestine surface Abdominal CT scan

What is a patient-specific 3D model

Surface model

Polyp

copy L Joskowicz 2011

What is the difference between 3D

visualization and 3D modeling

Visualization Model rendering

You do the interpretation

You fillomit missing info

Computer interprets

Explicit delineation

copy L Joskowicz 2011

Why patient-specific modeling

bull 3D visualization is great -- but it has limitationshellip

minus no explicit delineation

minus no validation ndash what you see is what is there

minus limited measurements

minus limited structures discrimination

bull 3D models allow

minus spatial and volumetric measurements ndash with validation

minus advanced analysis

minus wide variety of uses in the treatment cycle

minus reduction of radiologist time easier learning curve

copy L Joskowicz 2011

Segmentation in commercial systems

Copyright L Joskowicz 2007

copy L Joskowicz 2011

Segmentation in commercial systems

Copyright L Joskowicz 2007

copy L Joskowicz 2011

Copyright L Joskowicz 2007

Segmentation in commercial

systems

copy L Joskowicz 2011

3D models in the patient treatment cycle

Diagnosis Planning

Delivery

CAD mammography

Virtual colonoscopy

Neurosurgery -- trajectory

Orthopaedics -- fixation

Tumor follow-up

Implant location

MODEL Interventional

Radiology

Navigation

Robotics Evaluation

Training

Learning

copy L Joskowicz 2011

Diagnosis computer-aided radiology

stenosis

thrombus aneurism

tumor

volume

with Prof J Sosna

copy L Joskowicz 2011

liver contour

blood vessels

tumors

4-phase CT dataset

kidneycontour

blood vessels

urinary vessels with Dr Y Mintz with Prof J Sosna

Diagnosis computer-aided radiology

copy L Joskowicz 2011

Planning neurosurgery

entry point

with Dr Y Shoshan

copy L Joskowicz 2011

Planning neurosurgery

with Dr Y Shoshan

copy L Joskowicz 2011

Conventional Our method

Planning neurosurgery

with Dr Y Shoshan

copy L Joskowicz 2011

Planning orthopaedics

internal fixation external fixation

Fracture

fixation

alternatives

with Dr E Peleg and Profs M Liebergall R Mosheiff

copy L Joskowicz 2011 with Dr E Peleg and Profs M Liebergall R Mosheiff

Planning orthopaedics

copy L Joskowicz 2011

Planning orthopaedics

with Dr E Peleg and Profs M Liebergall R Mosheiff

copy L Joskowicz 2011

bull EM real-time tracking

bull US and X-ray imaging

add patient-specific

models from CT

Delivery interventional radiology

copy L Joskowicz 2011

Delivery intraoperative image guidance

augmented continuous X-ray fluoroscopy

with Simbionix and Prof J Sosna

copy L Joskowicz 2011

Delivery intraoperative image guidance

with Simbionix and Prof J Sosna

copy L Joskowicz 2011

Evaluation brain tumor follow-up

Dec 15 2009 June 9 2010

Disease

progression T2-weighted T1-weighted

Tumor internal components

solid enhancing cyst

with TA Sourasky and Dana Hospital

copy L Joskowicz 2011

Key issue model creation

Currently

bull mostly manual delineation

bull slice by slice

Desired

bull automaticnearly automatic ndash a few

clicks by physician

bull no technician

bull accurate and reliable

HARD NO METHOD

SUITABLE FOR ALL STRUCTURES

Modeling requires segmentation

Very difficult task

ndash organpathology specific

ndash imagescanning protocol

ndash anatomical variability

ndash intensity values overlap

ndash structures proximity

Identify and delineate anatomical structure contours

1 2

3 4

Modeling requires segmentation

Very difficult task

ndash organpathology specific

ndash imagescanning protocol

ndash anatomical variability

ndash intensity values overlap

ndash structures proximity

Identify and delineate anatomical structure contours

1 2

3 4

Clinical dataset challenges

proximity calcifications stenosis

Segmentation ndash state of the art

bull Hundreds anatomical segmentation methods for a

variety of structures and imaging modalities

bull Families of techniques thresholding region

growing level sets active contours and more

bull Very few if any in routine clinical use

ndash huge gap between prototype and clinical use

ndash time-consuming fragile limited in scope

ndash require technical knowledge

ndash most have limited validation

ndash clinical benefits unproven

bull Segmentation specificity is unavoidable ndash each

anatomical structure and pathology has unique

characteristics and nearby structures

bull Organ structure and pathology-specific

algorithms heart liver long bones spine

bull Very laborious top develop and validate a

segmentation algorithm for each ndash trial and error

process many man-months

No universal segmentation method

Observations (1)

Observations (2) bull Applications have different requirements wrt

ndash Accuracy

ndash Robustness

ndash User interaction

ndash Quality

bull Consider the different requirements of

ndash 3D visualization

ndash Training simulation

ndash FEA simulation

Defining application

requirements ahead of time is

essential

Goals Clinical

bull Automatic or nearly automatic lt 5mins user time

bull Can be used by clinician without a technician

bull Produces robust results

Technical

bull Requires significantly less time than developing from

scratch

bull Takes into account intensity and shape

bull Incorporates shape and intensity priors

MIS validation

bull Algorithms without validation are clinically

worthless

bull Validation is with respect to a clinical task

bull Validation requires a ground truth for comparison

bull Physical anatomical models andor phantoms are

typically not available (except sometimes for bones)

bull Ground truth is usually obtained by manual

identification andor segmentation by a user

bull Experts in most cases radiologists

bull Extrinsic comparison compare vs other methods

Copyright L Joskowicz 2010

Copyright L Joskowicz 2010

MIS validation issues

bull Large inter- and intra- variability across experts and clinical sites

bull May not be representative of population variability

bull Main quantitative parameters

ndash validation set size

ndash number and type of observers

ndash intra and inter-observer manual segmentation variability

ndash surface-based and volume-based error measures

MIS validation anatomical variability

stenosis

looping

narrowing

MIS validation anatomical variability

MIS validation metrics

bull Surface-based error measurements

ndash Mean surface distance

ndash RMS surface distance

ndash Maximum surface distance

bull Volume-based error measurements

ndash Dice coefficient

ndash Volumetric overlap error

Copyright L Joskowicz 2010

MIS validation metrics

Surface-based error measurements

bull Mean surface distance

bull RMS surface distance

bull Maximum surface distance

Copyright L Joskowicz 2010

Reference Result

MIS validation metrics

Volume-based error measurements

bull Volumetric overlap error

bull Dice similarity

Copyright L Joskowicz 2010

Reference Result

Aorta

Int + Ext Carotids

All arteries

Subclavian Arteries Bifurcations

16mm (std=08mm)

13mm (std=10mm)

15mm

std=13mm 17mm

std=09mm

07mm

std=07mm

Surface

RMS

Common

Carotid

MIS validation observers variability

bull Intra-observer variability determine how much

variation there is when a single observer

produced the ground-truth segmentation

repeat 5 times the segmentation

bull Inter-observer variability determine how much

variation there is between multiple observers that

produced the ground-truth segmentation

ask 3 radiologists to do the

segmentation

bull Observer expertise and frequency variability

Copyright L Joskowicz 2010

Study [Weltens 2001]

bull Axial MRI slice

bull Nine independent

observers

bull Repeated delineations

bull Interintra observer

variability 30

Key issue fuzzy tumor boundaries

Inter observer variability

Intra-observer variability

8

50

Carotid Lumen Segmentation and Stenosis

Grading Challenge -- The MIDAS Journal

Copyright L Joskowicz 2010

MIS validation manual annotation

Copyright L Joskowicz 2010

MIS validation ground-truth generation

centerlines

(manual)

contours

(manual)

Partial Volume

Segmentation

Reference

standard

from 3PVS

Copyright L Joskowicz 2010

Measurements characteristics

bull Ground-truth not know

bull Repeatability intra-observer variability

bull Reliability inter observer variability

bull Measure correlation between repeated

measurements

bull Intrinsic uncertainty about the tumor volume

bull Accuracy for clinical significance is unknown

Inaccuracy gt Uncertainty

bull Uncertainty may not be improved

bull Goal improve accuracy to obtain

Inaccuracy vs Uncertainty

Results can be

meaningless

Inaccuracy lt Uncertainty

In summaryhellip

bull Medical image segmentation is on the risehellip

bull Basis of many clinical applications

bull Validation is a MUST

bull Many methods and approaches ndash do not re-invent

the wheel

bull I expect patient-specific 3D models to be in the

clinical mainstream in 3-5 years

Copyright L Joskowicz 2011

Segmentation challenges

bull Rigorous quantification and evaluation of

segmentation algorithms performance

bull Shape priors for pathologies

bull Incorporation of functional information from

diffusionperfusion MRI fMRI PET into

segmentation algorithms

Copyright L Joskowicz 2011

Summary (1)

bull Patient-specific anatomy model creation is

currently a major bottleneck in many clinical

applications

bull Automatic anatomical segmentation is essential for

model creation

bull Current tools are limited in scope and coverage

bull Clinical use requires the elimination of the

technician ndash model generation by the physician

Summary (2) bull Great opportunity for the development and

incorporation of anatomy modeling tools in

commercial platforms

bull Growing need and variety of users for anatomical

models

ndash Training simulators surgery rehersal

ndash Intraoperative guidance

ndash Computer Aided Radiology

bull Service providers -- shifting paradigms

Page 4: Recent clinical advances and applications for medical ...crl.med.harvard.edu/research/MICCAI_Tutorial/MICCAI11-tutorial... · Recent clinical advances and applications for medical

copy L Joskowicz 2011

What is the difference between 3D

visualization and 3D modeling

Visualization Model rendering

You do the interpretation

You fillomit missing info

Computer interprets

Explicit delineation

copy L Joskowicz 2011

Why patient-specific modeling

bull 3D visualization is great -- but it has limitationshellip

minus no explicit delineation

minus no validation ndash what you see is what is there

minus limited measurements

minus limited structures discrimination

bull 3D models allow

minus spatial and volumetric measurements ndash with validation

minus advanced analysis

minus wide variety of uses in the treatment cycle

minus reduction of radiologist time easier learning curve

copy L Joskowicz 2011

Segmentation in commercial systems

Copyright L Joskowicz 2007

copy L Joskowicz 2011

Segmentation in commercial systems

Copyright L Joskowicz 2007

copy L Joskowicz 2011

Copyright L Joskowicz 2007

Segmentation in commercial

systems

copy L Joskowicz 2011

3D models in the patient treatment cycle

Diagnosis Planning

Delivery

CAD mammography

Virtual colonoscopy

Neurosurgery -- trajectory

Orthopaedics -- fixation

Tumor follow-up

Implant location

MODEL Interventional

Radiology

Navigation

Robotics Evaluation

Training

Learning

copy L Joskowicz 2011

Diagnosis computer-aided radiology

stenosis

thrombus aneurism

tumor

volume

with Prof J Sosna

copy L Joskowicz 2011

liver contour

blood vessels

tumors

4-phase CT dataset

kidneycontour

blood vessels

urinary vessels with Dr Y Mintz with Prof J Sosna

Diagnosis computer-aided radiology

copy L Joskowicz 2011

Planning neurosurgery

entry point

with Dr Y Shoshan

copy L Joskowicz 2011

Planning neurosurgery

with Dr Y Shoshan

copy L Joskowicz 2011

Conventional Our method

Planning neurosurgery

with Dr Y Shoshan

copy L Joskowicz 2011

Planning orthopaedics

internal fixation external fixation

Fracture

fixation

alternatives

with Dr E Peleg and Profs M Liebergall R Mosheiff

copy L Joskowicz 2011 with Dr E Peleg and Profs M Liebergall R Mosheiff

Planning orthopaedics

copy L Joskowicz 2011

Planning orthopaedics

with Dr E Peleg and Profs M Liebergall R Mosheiff

copy L Joskowicz 2011

bull EM real-time tracking

bull US and X-ray imaging

add patient-specific

models from CT

Delivery interventional radiology

copy L Joskowicz 2011

Delivery intraoperative image guidance

augmented continuous X-ray fluoroscopy

with Simbionix and Prof J Sosna

copy L Joskowicz 2011

Delivery intraoperative image guidance

with Simbionix and Prof J Sosna

copy L Joskowicz 2011

Evaluation brain tumor follow-up

Dec 15 2009 June 9 2010

Disease

progression T2-weighted T1-weighted

Tumor internal components

solid enhancing cyst

with TA Sourasky and Dana Hospital

copy L Joskowicz 2011

Key issue model creation

Currently

bull mostly manual delineation

bull slice by slice

Desired

bull automaticnearly automatic ndash a few

clicks by physician

bull no technician

bull accurate and reliable

HARD NO METHOD

SUITABLE FOR ALL STRUCTURES

Modeling requires segmentation

Very difficult task

ndash organpathology specific

ndash imagescanning protocol

ndash anatomical variability

ndash intensity values overlap

ndash structures proximity

Identify and delineate anatomical structure contours

1 2

3 4

Modeling requires segmentation

Very difficult task

ndash organpathology specific

ndash imagescanning protocol

ndash anatomical variability

ndash intensity values overlap

ndash structures proximity

Identify and delineate anatomical structure contours

1 2

3 4

Clinical dataset challenges

proximity calcifications stenosis

Segmentation ndash state of the art

bull Hundreds anatomical segmentation methods for a

variety of structures and imaging modalities

bull Families of techniques thresholding region

growing level sets active contours and more

bull Very few if any in routine clinical use

ndash huge gap between prototype and clinical use

ndash time-consuming fragile limited in scope

ndash require technical knowledge

ndash most have limited validation

ndash clinical benefits unproven

bull Segmentation specificity is unavoidable ndash each

anatomical structure and pathology has unique

characteristics and nearby structures

bull Organ structure and pathology-specific

algorithms heart liver long bones spine

bull Very laborious top develop and validate a

segmentation algorithm for each ndash trial and error

process many man-months

No universal segmentation method

Observations (1)

Observations (2) bull Applications have different requirements wrt

ndash Accuracy

ndash Robustness

ndash User interaction

ndash Quality

bull Consider the different requirements of

ndash 3D visualization

ndash Training simulation

ndash FEA simulation

Defining application

requirements ahead of time is

essential

Goals Clinical

bull Automatic or nearly automatic lt 5mins user time

bull Can be used by clinician without a technician

bull Produces robust results

Technical

bull Requires significantly less time than developing from

scratch

bull Takes into account intensity and shape

bull Incorporates shape and intensity priors

MIS validation

bull Algorithms without validation are clinically

worthless

bull Validation is with respect to a clinical task

bull Validation requires a ground truth for comparison

bull Physical anatomical models andor phantoms are

typically not available (except sometimes for bones)

bull Ground truth is usually obtained by manual

identification andor segmentation by a user

bull Experts in most cases radiologists

bull Extrinsic comparison compare vs other methods

Copyright L Joskowicz 2010

Copyright L Joskowicz 2010

MIS validation issues

bull Large inter- and intra- variability across experts and clinical sites

bull May not be representative of population variability

bull Main quantitative parameters

ndash validation set size

ndash number and type of observers

ndash intra and inter-observer manual segmentation variability

ndash surface-based and volume-based error measures

MIS validation anatomical variability

stenosis

looping

narrowing

MIS validation anatomical variability

MIS validation metrics

bull Surface-based error measurements

ndash Mean surface distance

ndash RMS surface distance

ndash Maximum surface distance

bull Volume-based error measurements

ndash Dice coefficient

ndash Volumetric overlap error

Copyright L Joskowicz 2010

MIS validation metrics

Surface-based error measurements

bull Mean surface distance

bull RMS surface distance

bull Maximum surface distance

Copyright L Joskowicz 2010

Reference Result

MIS validation metrics

Volume-based error measurements

bull Volumetric overlap error

bull Dice similarity

Copyright L Joskowicz 2010

Reference Result

Aorta

Int + Ext Carotids

All arteries

Subclavian Arteries Bifurcations

16mm (std=08mm)

13mm (std=10mm)

15mm

std=13mm 17mm

std=09mm

07mm

std=07mm

Surface

RMS

Common

Carotid

MIS validation observers variability

bull Intra-observer variability determine how much

variation there is when a single observer

produced the ground-truth segmentation

repeat 5 times the segmentation

bull Inter-observer variability determine how much

variation there is between multiple observers that

produced the ground-truth segmentation

ask 3 radiologists to do the

segmentation

bull Observer expertise and frequency variability

Copyright L Joskowicz 2010

Study [Weltens 2001]

bull Axial MRI slice

bull Nine independent

observers

bull Repeated delineations

bull Interintra observer

variability 30

Key issue fuzzy tumor boundaries

Inter observer variability

Intra-observer variability

8

50

Carotid Lumen Segmentation and Stenosis

Grading Challenge -- The MIDAS Journal

Copyright L Joskowicz 2010

MIS validation manual annotation

Copyright L Joskowicz 2010

MIS validation ground-truth generation

centerlines

(manual)

contours

(manual)

Partial Volume

Segmentation

Reference

standard

from 3PVS

Copyright L Joskowicz 2010

Measurements characteristics

bull Ground-truth not know

bull Repeatability intra-observer variability

bull Reliability inter observer variability

bull Measure correlation between repeated

measurements

bull Intrinsic uncertainty about the tumor volume

bull Accuracy for clinical significance is unknown

Inaccuracy gt Uncertainty

bull Uncertainty may not be improved

bull Goal improve accuracy to obtain

Inaccuracy vs Uncertainty

Results can be

meaningless

Inaccuracy lt Uncertainty

In summaryhellip

bull Medical image segmentation is on the risehellip

bull Basis of many clinical applications

bull Validation is a MUST

bull Many methods and approaches ndash do not re-invent

the wheel

bull I expect patient-specific 3D models to be in the

clinical mainstream in 3-5 years

Copyright L Joskowicz 2011

Segmentation challenges

bull Rigorous quantification and evaluation of

segmentation algorithms performance

bull Shape priors for pathologies

bull Incorporation of functional information from

diffusionperfusion MRI fMRI PET into

segmentation algorithms

Copyright L Joskowicz 2011

Summary (1)

bull Patient-specific anatomy model creation is

currently a major bottleneck in many clinical

applications

bull Automatic anatomical segmentation is essential for

model creation

bull Current tools are limited in scope and coverage

bull Clinical use requires the elimination of the

technician ndash model generation by the physician

Summary (2) bull Great opportunity for the development and

incorporation of anatomy modeling tools in

commercial platforms

bull Growing need and variety of users for anatomical

models

ndash Training simulators surgery rehersal

ndash Intraoperative guidance

ndash Computer Aided Radiology

bull Service providers -- shifting paradigms

Page 5: Recent clinical advances and applications for medical ...crl.med.harvard.edu/research/MICCAI_Tutorial/MICCAI11-tutorial... · Recent clinical advances and applications for medical

copy L Joskowicz 2011

Why patient-specific modeling

bull 3D visualization is great -- but it has limitationshellip

minus no explicit delineation

minus no validation ndash what you see is what is there

minus limited measurements

minus limited structures discrimination

bull 3D models allow

minus spatial and volumetric measurements ndash with validation

minus advanced analysis

minus wide variety of uses in the treatment cycle

minus reduction of radiologist time easier learning curve

copy L Joskowicz 2011

Segmentation in commercial systems

Copyright L Joskowicz 2007

copy L Joskowicz 2011

Segmentation in commercial systems

Copyright L Joskowicz 2007

copy L Joskowicz 2011

Copyright L Joskowicz 2007

Segmentation in commercial

systems

copy L Joskowicz 2011

3D models in the patient treatment cycle

Diagnosis Planning

Delivery

CAD mammography

Virtual colonoscopy

Neurosurgery -- trajectory

Orthopaedics -- fixation

Tumor follow-up

Implant location

MODEL Interventional

Radiology

Navigation

Robotics Evaluation

Training

Learning

copy L Joskowicz 2011

Diagnosis computer-aided radiology

stenosis

thrombus aneurism

tumor

volume

with Prof J Sosna

copy L Joskowicz 2011

liver contour

blood vessels

tumors

4-phase CT dataset

kidneycontour

blood vessels

urinary vessels with Dr Y Mintz with Prof J Sosna

Diagnosis computer-aided radiology

copy L Joskowicz 2011

Planning neurosurgery

entry point

with Dr Y Shoshan

copy L Joskowicz 2011

Planning neurosurgery

with Dr Y Shoshan

copy L Joskowicz 2011

Conventional Our method

Planning neurosurgery

with Dr Y Shoshan

copy L Joskowicz 2011

Planning orthopaedics

internal fixation external fixation

Fracture

fixation

alternatives

with Dr E Peleg and Profs M Liebergall R Mosheiff

copy L Joskowicz 2011 with Dr E Peleg and Profs M Liebergall R Mosheiff

Planning orthopaedics

copy L Joskowicz 2011

Planning orthopaedics

with Dr E Peleg and Profs M Liebergall R Mosheiff

copy L Joskowicz 2011

bull EM real-time tracking

bull US and X-ray imaging

add patient-specific

models from CT

Delivery interventional radiology

copy L Joskowicz 2011

Delivery intraoperative image guidance

augmented continuous X-ray fluoroscopy

with Simbionix and Prof J Sosna

copy L Joskowicz 2011

Delivery intraoperative image guidance

with Simbionix and Prof J Sosna

copy L Joskowicz 2011

Evaluation brain tumor follow-up

Dec 15 2009 June 9 2010

Disease

progression T2-weighted T1-weighted

Tumor internal components

solid enhancing cyst

with TA Sourasky and Dana Hospital

copy L Joskowicz 2011

Key issue model creation

Currently

bull mostly manual delineation

bull slice by slice

Desired

bull automaticnearly automatic ndash a few

clicks by physician

bull no technician

bull accurate and reliable

HARD NO METHOD

SUITABLE FOR ALL STRUCTURES

Modeling requires segmentation

Very difficult task

ndash organpathology specific

ndash imagescanning protocol

ndash anatomical variability

ndash intensity values overlap

ndash structures proximity

Identify and delineate anatomical structure contours

1 2

3 4

Modeling requires segmentation

Very difficult task

ndash organpathology specific

ndash imagescanning protocol

ndash anatomical variability

ndash intensity values overlap

ndash structures proximity

Identify and delineate anatomical structure contours

1 2

3 4

Clinical dataset challenges

proximity calcifications stenosis

Segmentation ndash state of the art

bull Hundreds anatomical segmentation methods for a

variety of structures and imaging modalities

bull Families of techniques thresholding region

growing level sets active contours and more

bull Very few if any in routine clinical use

ndash huge gap between prototype and clinical use

ndash time-consuming fragile limited in scope

ndash require technical knowledge

ndash most have limited validation

ndash clinical benefits unproven

bull Segmentation specificity is unavoidable ndash each

anatomical structure and pathology has unique

characteristics and nearby structures

bull Organ structure and pathology-specific

algorithms heart liver long bones spine

bull Very laborious top develop and validate a

segmentation algorithm for each ndash trial and error

process many man-months

No universal segmentation method

Observations (1)

Observations (2) bull Applications have different requirements wrt

ndash Accuracy

ndash Robustness

ndash User interaction

ndash Quality

bull Consider the different requirements of

ndash 3D visualization

ndash Training simulation

ndash FEA simulation

Defining application

requirements ahead of time is

essential

Goals Clinical

bull Automatic or nearly automatic lt 5mins user time

bull Can be used by clinician without a technician

bull Produces robust results

Technical

bull Requires significantly less time than developing from

scratch

bull Takes into account intensity and shape

bull Incorporates shape and intensity priors

MIS validation

bull Algorithms without validation are clinically

worthless

bull Validation is with respect to a clinical task

bull Validation requires a ground truth for comparison

bull Physical anatomical models andor phantoms are

typically not available (except sometimes for bones)

bull Ground truth is usually obtained by manual

identification andor segmentation by a user

bull Experts in most cases radiologists

bull Extrinsic comparison compare vs other methods

Copyright L Joskowicz 2010

Copyright L Joskowicz 2010

MIS validation issues

bull Large inter- and intra- variability across experts and clinical sites

bull May not be representative of population variability

bull Main quantitative parameters

ndash validation set size

ndash number and type of observers

ndash intra and inter-observer manual segmentation variability

ndash surface-based and volume-based error measures

MIS validation anatomical variability

stenosis

looping

narrowing

MIS validation anatomical variability

MIS validation metrics

bull Surface-based error measurements

ndash Mean surface distance

ndash RMS surface distance

ndash Maximum surface distance

bull Volume-based error measurements

ndash Dice coefficient

ndash Volumetric overlap error

Copyright L Joskowicz 2010

MIS validation metrics

Surface-based error measurements

bull Mean surface distance

bull RMS surface distance

bull Maximum surface distance

Copyright L Joskowicz 2010

Reference Result

MIS validation metrics

Volume-based error measurements

bull Volumetric overlap error

bull Dice similarity

Copyright L Joskowicz 2010

Reference Result

Aorta

Int + Ext Carotids

All arteries

Subclavian Arteries Bifurcations

16mm (std=08mm)

13mm (std=10mm)

15mm

std=13mm 17mm

std=09mm

07mm

std=07mm

Surface

RMS

Common

Carotid

MIS validation observers variability

bull Intra-observer variability determine how much

variation there is when a single observer

produced the ground-truth segmentation

repeat 5 times the segmentation

bull Inter-observer variability determine how much

variation there is between multiple observers that

produced the ground-truth segmentation

ask 3 radiologists to do the

segmentation

bull Observer expertise and frequency variability

Copyright L Joskowicz 2010

Study [Weltens 2001]

bull Axial MRI slice

bull Nine independent

observers

bull Repeated delineations

bull Interintra observer

variability 30

Key issue fuzzy tumor boundaries

Inter observer variability

Intra-observer variability

8

50

Carotid Lumen Segmentation and Stenosis

Grading Challenge -- The MIDAS Journal

Copyright L Joskowicz 2010

MIS validation manual annotation

Copyright L Joskowicz 2010

MIS validation ground-truth generation

centerlines

(manual)

contours

(manual)

Partial Volume

Segmentation

Reference

standard

from 3PVS

Copyright L Joskowicz 2010

Measurements characteristics

bull Ground-truth not know

bull Repeatability intra-observer variability

bull Reliability inter observer variability

bull Measure correlation between repeated

measurements

bull Intrinsic uncertainty about the tumor volume

bull Accuracy for clinical significance is unknown

Inaccuracy gt Uncertainty

bull Uncertainty may not be improved

bull Goal improve accuracy to obtain

Inaccuracy vs Uncertainty

Results can be

meaningless

Inaccuracy lt Uncertainty

In summaryhellip

bull Medical image segmentation is on the risehellip

bull Basis of many clinical applications

bull Validation is a MUST

bull Many methods and approaches ndash do not re-invent

the wheel

bull I expect patient-specific 3D models to be in the

clinical mainstream in 3-5 years

Copyright L Joskowicz 2011

Segmentation challenges

bull Rigorous quantification and evaluation of

segmentation algorithms performance

bull Shape priors for pathologies

bull Incorporation of functional information from

diffusionperfusion MRI fMRI PET into

segmentation algorithms

Copyright L Joskowicz 2011

Summary (1)

bull Patient-specific anatomy model creation is

currently a major bottleneck in many clinical

applications

bull Automatic anatomical segmentation is essential for

model creation

bull Current tools are limited in scope and coverage

bull Clinical use requires the elimination of the

technician ndash model generation by the physician

Summary (2) bull Great opportunity for the development and

incorporation of anatomy modeling tools in

commercial platforms

bull Growing need and variety of users for anatomical

models

ndash Training simulators surgery rehersal

ndash Intraoperative guidance

ndash Computer Aided Radiology

bull Service providers -- shifting paradigms

Page 6: Recent clinical advances and applications for medical ...crl.med.harvard.edu/research/MICCAI_Tutorial/MICCAI11-tutorial... · Recent clinical advances and applications for medical

copy L Joskowicz 2011

Segmentation in commercial systems

Copyright L Joskowicz 2007

copy L Joskowicz 2011

Segmentation in commercial systems

Copyright L Joskowicz 2007

copy L Joskowicz 2011

Copyright L Joskowicz 2007

Segmentation in commercial

systems

copy L Joskowicz 2011

3D models in the patient treatment cycle

Diagnosis Planning

Delivery

CAD mammography

Virtual colonoscopy

Neurosurgery -- trajectory

Orthopaedics -- fixation

Tumor follow-up

Implant location

MODEL Interventional

Radiology

Navigation

Robotics Evaluation

Training

Learning

copy L Joskowicz 2011

Diagnosis computer-aided radiology

stenosis

thrombus aneurism

tumor

volume

with Prof J Sosna

copy L Joskowicz 2011

liver contour

blood vessels

tumors

4-phase CT dataset

kidneycontour

blood vessels

urinary vessels with Dr Y Mintz with Prof J Sosna

Diagnosis computer-aided radiology

copy L Joskowicz 2011

Planning neurosurgery

entry point

with Dr Y Shoshan

copy L Joskowicz 2011

Planning neurosurgery

with Dr Y Shoshan

copy L Joskowicz 2011

Conventional Our method

Planning neurosurgery

with Dr Y Shoshan

copy L Joskowicz 2011

Planning orthopaedics

internal fixation external fixation

Fracture

fixation

alternatives

with Dr E Peleg and Profs M Liebergall R Mosheiff

copy L Joskowicz 2011 with Dr E Peleg and Profs M Liebergall R Mosheiff

Planning orthopaedics

copy L Joskowicz 2011

Planning orthopaedics

with Dr E Peleg and Profs M Liebergall R Mosheiff

copy L Joskowicz 2011

bull EM real-time tracking

bull US and X-ray imaging

add patient-specific

models from CT

Delivery interventional radiology

copy L Joskowicz 2011

Delivery intraoperative image guidance

augmented continuous X-ray fluoroscopy

with Simbionix and Prof J Sosna

copy L Joskowicz 2011

Delivery intraoperative image guidance

with Simbionix and Prof J Sosna

copy L Joskowicz 2011

Evaluation brain tumor follow-up

Dec 15 2009 June 9 2010

Disease

progression T2-weighted T1-weighted

Tumor internal components

solid enhancing cyst

with TA Sourasky and Dana Hospital

copy L Joskowicz 2011

Key issue model creation

Currently

bull mostly manual delineation

bull slice by slice

Desired

bull automaticnearly automatic ndash a few

clicks by physician

bull no technician

bull accurate and reliable

HARD NO METHOD

SUITABLE FOR ALL STRUCTURES

Modeling requires segmentation

Very difficult task

ndash organpathology specific

ndash imagescanning protocol

ndash anatomical variability

ndash intensity values overlap

ndash structures proximity

Identify and delineate anatomical structure contours

1 2

3 4

Modeling requires segmentation

Very difficult task

ndash organpathology specific

ndash imagescanning protocol

ndash anatomical variability

ndash intensity values overlap

ndash structures proximity

Identify and delineate anatomical structure contours

1 2

3 4

Clinical dataset challenges

proximity calcifications stenosis

Segmentation ndash state of the art

bull Hundreds anatomical segmentation methods for a

variety of structures and imaging modalities

bull Families of techniques thresholding region

growing level sets active contours and more

bull Very few if any in routine clinical use

ndash huge gap between prototype and clinical use

ndash time-consuming fragile limited in scope

ndash require technical knowledge

ndash most have limited validation

ndash clinical benefits unproven

bull Segmentation specificity is unavoidable ndash each

anatomical structure and pathology has unique

characteristics and nearby structures

bull Organ structure and pathology-specific

algorithms heart liver long bones spine

bull Very laborious top develop and validate a

segmentation algorithm for each ndash trial and error

process many man-months

No universal segmentation method

Observations (1)

Observations (2) bull Applications have different requirements wrt

ndash Accuracy

ndash Robustness

ndash User interaction

ndash Quality

bull Consider the different requirements of

ndash 3D visualization

ndash Training simulation

ndash FEA simulation

Defining application

requirements ahead of time is

essential

Goals Clinical

bull Automatic or nearly automatic lt 5mins user time

bull Can be used by clinician without a technician

bull Produces robust results

Technical

bull Requires significantly less time than developing from

scratch

bull Takes into account intensity and shape

bull Incorporates shape and intensity priors

MIS validation

bull Algorithms without validation are clinically

worthless

bull Validation is with respect to a clinical task

bull Validation requires a ground truth for comparison

bull Physical anatomical models andor phantoms are

typically not available (except sometimes for bones)

bull Ground truth is usually obtained by manual

identification andor segmentation by a user

bull Experts in most cases radiologists

bull Extrinsic comparison compare vs other methods

Copyright L Joskowicz 2010

Copyright L Joskowicz 2010

MIS validation issues

bull Large inter- and intra- variability across experts and clinical sites

bull May not be representative of population variability

bull Main quantitative parameters

ndash validation set size

ndash number and type of observers

ndash intra and inter-observer manual segmentation variability

ndash surface-based and volume-based error measures

MIS validation anatomical variability

stenosis

looping

narrowing

MIS validation anatomical variability

MIS validation metrics

bull Surface-based error measurements

ndash Mean surface distance

ndash RMS surface distance

ndash Maximum surface distance

bull Volume-based error measurements

ndash Dice coefficient

ndash Volumetric overlap error

Copyright L Joskowicz 2010

MIS validation metrics

Surface-based error measurements

bull Mean surface distance

bull RMS surface distance

bull Maximum surface distance

Copyright L Joskowicz 2010

Reference Result

MIS validation metrics

Volume-based error measurements

bull Volumetric overlap error

bull Dice similarity

Copyright L Joskowicz 2010

Reference Result

Aorta

Int + Ext Carotids

All arteries

Subclavian Arteries Bifurcations

16mm (std=08mm)

13mm (std=10mm)

15mm

std=13mm 17mm

std=09mm

07mm

std=07mm

Surface

RMS

Common

Carotid

MIS validation observers variability

bull Intra-observer variability determine how much

variation there is when a single observer

produced the ground-truth segmentation

repeat 5 times the segmentation

bull Inter-observer variability determine how much

variation there is between multiple observers that

produced the ground-truth segmentation

ask 3 radiologists to do the

segmentation

bull Observer expertise and frequency variability

Copyright L Joskowicz 2010

Study [Weltens 2001]

bull Axial MRI slice

bull Nine independent

observers

bull Repeated delineations

bull Interintra observer

variability 30

Key issue fuzzy tumor boundaries

Inter observer variability

Intra-observer variability

8

50

Carotid Lumen Segmentation and Stenosis

Grading Challenge -- The MIDAS Journal

Copyright L Joskowicz 2010

MIS validation manual annotation

Copyright L Joskowicz 2010

MIS validation ground-truth generation

centerlines

(manual)

contours

(manual)

Partial Volume

Segmentation

Reference

standard

from 3PVS

Copyright L Joskowicz 2010

Measurements characteristics

bull Ground-truth not know

bull Repeatability intra-observer variability

bull Reliability inter observer variability

bull Measure correlation between repeated

measurements

bull Intrinsic uncertainty about the tumor volume

bull Accuracy for clinical significance is unknown

Inaccuracy gt Uncertainty

bull Uncertainty may not be improved

bull Goal improve accuracy to obtain

Inaccuracy vs Uncertainty

Results can be

meaningless

Inaccuracy lt Uncertainty

In summaryhellip

bull Medical image segmentation is on the risehellip

bull Basis of many clinical applications

bull Validation is a MUST

bull Many methods and approaches ndash do not re-invent

the wheel

bull I expect patient-specific 3D models to be in the

clinical mainstream in 3-5 years

Copyright L Joskowicz 2011

Segmentation challenges

bull Rigorous quantification and evaluation of

segmentation algorithms performance

bull Shape priors for pathologies

bull Incorporation of functional information from

diffusionperfusion MRI fMRI PET into

segmentation algorithms

Copyright L Joskowicz 2011

Summary (1)

bull Patient-specific anatomy model creation is

currently a major bottleneck in many clinical

applications

bull Automatic anatomical segmentation is essential for

model creation

bull Current tools are limited in scope and coverage

bull Clinical use requires the elimination of the

technician ndash model generation by the physician

Summary (2) bull Great opportunity for the development and

incorporation of anatomy modeling tools in

commercial platforms

bull Growing need and variety of users for anatomical

models

ndash Training simulators surgery rehersal

ndash Intraoperative guidance

ndash Computer Aided Radiology

bull Service providers -- shifting paradigms

Page 7: Recent clinical advances and applications for medical ...crl.med.harvard.edu/research/MICCAI_Tutorial/MICCAI11-tutorial... · Recent clinical advances and applications for medical

copy L Joskowicz 2011

Segmentation in commercial systems

Copyright L Joskowicz 2007

copy L Joskowicz 2011

Copyright L Joskowicz 2007

Segmentation in commercial

systems

copy L Joskowicz 2011

3D models in the patient treatment cycle

Diagnosis Planning

Delivery

CAD mammography

Virtual colonoscopy

Neurosurgery -- trajectory

Orthopaedics -- fixation

Tumor follow-up

Implant location

MODEL Interventional

Radiology

Navigation

Robotics Evaluation

Training

Learning

copy L Joskowicz 2011

Diagnosis computer-aided radiology

stenosis

thrombus aneurism

tumor

volume

with Prof J Sosna

copy L Joskowicz 2011

liver contour

blood vessels

tumors

4-phase CT dataset

kidneycontour

blood vessels

urinary vessels with Dr Y Mintz with Prof J Sosna

Diagnosis computer-aided radiology

copy L Joskowicz 2011

Planning neurosurgery

entry point

with Dr Y Shoshan

copy L Joskowicz 2011

Planning neurosurgery

with Dr Y Shoshan

copy L Joskowicz 2011

Conventional Our method

Planning neurosurgery

with Dr Y Shoshan

copy L Joskowicz 2011

Planning orthopaedics

internal fixation external fixation

Fracture

fixation

alternatives

with Dr E Peleg and Profs M Liebergall R Mosheiff

copy L Joskowicz 2011 with Dr E Peleg and Profs M Liebergall R Mosheiff

Planning orthopaedics

copy L Joskowicz 2011

Planning orthopaedics

with Dr E Peleg and Profs M Liebergall R Mosheiff

copy L Joskowicz 2011

bull EM real-time tracking

bull US and X-ray imaging

add patient-specific

models from CT

Delivery interventional radiology

copy L Joskowicz 2011

Delivery intraoperative image guidance

augmented continuous X-ray fluoroscopy

with Simbionix and Prof J Sosna

copy L Joskowicz 2011

Delivery intraoperative image guidance

with Simbionix and Prof J Sosna

copy L Joskowicz 2011

Evaluation brain tumor follow-up

Dec 15 2009 June 9 2010

Disease

progression T2-weighted T1-weighted

Tumor internal components

solid enhancing cyst

with TA Sourasky and Dana Hospital

copy L Joskowicz 2011

Key issue model creation

Currently

bull mostly manual delineation

bull slice by slice

Desired

bull automaticnearly automatic ndash a few

clicks by physician

bull no technician

bull accurate and reliable

HARD NO METHOD

SUITABLE FOR ALL STRUCTURES

Modeling requires segmentation

Very difficult task

ndash organpathology specific

ndash imagescanning protocol

ndash anatomical variability

ndash intensity values overlap

ndash structures proximity

Identify and delineate anatomical structure contours

1 2

3 4

Modeling requires segmentation

Very difficult task

ndash organpathology specific

ndash imagescanning protocol

ndash anatomical variability

ndash intensity values overlap

ndash structures proximity

Identify and delineate anatomical structure contours

1 2

3 4

Clinical dataset challenges

proximity calcifications stenosis

Segmentation ndash state of the art

bull Hundreds anatomical segmentation methods for a

variety of structures and imaging modalities

bull Families of techniques thresholding region

growing level sets active contours and more

bull Very few if any in routine clinical use

ndash huge gap between prototype and clinical use

ndash time-consuming fragile limited in scope

ndash require technical knowledge

ndash most have limited validation

ndash clinical benefits unproven

bull Segmentation specificity is unavoidable ndash each

anatomical structure and pathology has unique

characteristics and nearby structures

bull Organ structure and pathology-specific

algorithms heart liver long bones spine

bull Very laborious top develop and validate a

segmentation algorithm for each ndash trial and error

process many man-months

No universal segmentation method

Observations (1)

Observations (2) bull Applications have different requirements wrt

ndash Accuracy

ndash Robustness

ndash User interaction

ndash Quality

bull Consider the different requirements of

ndash 3D visualization

ndash Training simulation

ndash FEA simulation

Defining application

requirements ahead of time is

essential

Goals Clinical

bull Automatic or nearly automatic lt 5mins user time

bull Can be used by clinician without a technician

bull Produces robust results

Technical

bull Requires significantly less time than developing from

scratch

bull Takes into account intensity and shape

bull Incorporates shape and intensity priors

MIS validation

bull Algorithms without validation are clinically

worthless

bull Validation is with respect to a clinical task

bull Validation requires a ground truth for comparison

bull Physical anatomical models andor phantoms are

typically not available (except sometimes for bones)

bull Ground truth is usually obtained by manual

identification andor segmentation by a user

bull Experts in most cases radiologists

bull Extrinsic comparison compare vs other methods

Copyright L Joskowicz 2010

Copyright L Joskowicz 2010

MIS validation issues

bull Large inter- and intra- variability across experts and clinical sites

bull May not be representative of population variability

bull Main quantitative parameters

ndash validation set size

ndash number and type of observers

ndash intra and inter-observer manual segmentation variability

ndash surface-based and volume-based error measures

MIS validation anatomical variability

stenosis

looping

narrowing

MIS validation anatomical variability

MIS validation metrics

bull Surface-based error measurements

ndash Mean surface distance

ndash RMS surface distance

ndash Maximum surface distance

bull Volume-based error measurements

ndash Dice coefficient

ndash Volumetric overlap error

Copyright L Joskowicz 2010

MIS validation metrics

Surface-based error measurements

bull Mean surface distance

bull RMS surface distance

bull Maximum surface distance

Copyright L Joskowicz 2010

Reference Result

MIS validation metrics

Volume-based error measurements

bull Volumetric overlap error

bull Dice similarity

Copyright L Joskowicz 2010

Reference Result

Aorta

Int + Ext Carotids

All arteries

Subclavian Arteries Bifurcations

16mm (std=08mm)

13mm (std=10mm)

15mm

std=13mm 17mm

std=09mm

07mm

std=07mm

Surface

RMS

Common

Carotid

MIS validation observers variability

bull Intra-observer variability determine how much

variation there is when a single observer

produced the ground-truth segmentation

repeat 5 times the segmentation

bull Inter-observer variability determine how much

variation there is between multiple observers that

produced the ground-truth segmentation

ask 3 radiologists to do the

segmentation

bull Observer expertise and frequency variability

Copyright L Joskowicz 2010

Study [Weltens 2001]

bull Axial MRI slice

bull Nine independent

observers

bull Repeated delineations

bull Interintra observer

variability 30

Key issue fuzzy tumor boundaries

Inter observer variability

Intra-observer variability

8

50

Carotid Lumen Segmentation and Stenosis

Grading Challenge -- The MIDAS Journal

Copyright L Joskowicz 2010

MIS validation manual annotation

Copyright L Joskowicz 2010

MIS validation ground-truth generation

centerlines

(manual)

contours

(manual)

Partial Volume

Segmentation

Reference

standard

from 3PVS

Copyright L Joskowicz 2010

Measurements characteristics

bull Ground-truth not know

bull Repeatability intra-observer variability

bull Reliability inter observer variability

bull Measure correlation between repeated

measurements

bull Intrinsic uncertainty about the tumor volume

bull Accuracy for clinical significance is unknown

Inaccuracy gt Uncertainty

bull Uncertainty may not be improved

bull Goal improve accuracy to obtain

Inaccuracy vs Uncertainty

Results can be

meaningless

Inaccuracy lt Uncertainty

In summaryhellip

bull Medical image segmentation is on the risehellip

bull Basis of many clinical applications

bull Validation is a MUST

bull Many methods and approaches ndash do not re-invent

the wheel

bull I expect patient-specific 3D models to be in the

clinical mainstream in 3-5 years

Copyright L Joskowicz 2011

Segmentation challenges

bull Rigorous quantification and evaluation of

segmentation algorithms performance

bull Shape priors for pathologies

bull Incorporation of functional information from

diffusionperfusion MRI fMRI PET into

segmentation algorithms

Copyright L Joskowicz 2011

Summary (1)

bull Patient-specific anatomy model creation is

currently a major bottleneck in many clinical

applications

bull Automatic anatomical segmentation is essential for

model creation

bull Current tools are limited in scope and coverage

bull Clinical use requires the elimination of the

technician ndash model generation by the physician

Summary (2) bull Great opportunity for the development and

incorporation of anatomy modeling tools in

commercial platforms

bull Growing need and variety of users for anatomical

models

ndash Training simulators surgery rehersal

ndash Intraoperative guidance

ndash Computer Aided Radiology

bull Service providers -- shifting paradigms

Page 8: Recent clinical advances and applications for medical ...crl.med.harvard.edu/research/MICCAI_Tutorial/MICCAI11-tutorial... · Recent clinical advances and applications for medical

copy L Joskowicz 2011

Copyright L Joskowicz 2007

Segmentation in commercial

systems

copy L Joskowicz 2011

3D models in the patient treatment cycle

Diagnosis Planning

Delivery

CAD mammography

Virtual colonoscopy

Neurosurgery -- trajectory

Orthopaedics -- fixation

Tumor follow-up

Implant location

MODEL Interventional

Radiology

Navigation

Robotics Evaluation

Training

Learning

copy L Joskowicz 2011

Diagnosis computer-aided radiology

stenosis

thrombus aneurism

tumor

volume

with Prof J Sosna

copy L Joskowicz 2011

liver contour

blood vessels

tumors

4-phase CT dataset

kidneycontour

blood vessels

urinary vessels with Dr Y Mintz with Prof J Sosna

Diagnosis computer-aided radiology

copy L Joskowicz 2011

Planning neurosurgery

entry point

with Dr Y Shoshan

copy L Joskowicz 2011

Planning neurosurgery

with Dr Y Shoshan

copy L Joskowicz 2011

Conventional Our method

Planning neurosurgery

with Dr Y Shoshan

copy L Joskowicz 2011

Planning orthopaedics

internal fixation external fixation

Fracture

fixation

alternatives

with Dr E Peleg and Profs M Liebergall R Mosheiff

copy L Joskowicz 2011 with Dr E Peleg and Profs M Liebergall R Mosheiff

Planning orthopaedics

copy L Joskowicz 2011

Planning orthopaedics

with Dr E Peleg and Profs M Liebergall R Mosheiff

copy L Joskowicz 2011

bull EM real-time tracking

bull US and X-ray imaging

add patient-specific

models from CT

Delivery interventional radiology

copy L Joskowicz 2011

Delivery intraoperative image guidance

augmented continuous X-ray fluoroscopy

with Simbionix and Prof J Sosna

copy L Joskowicz 2011

Delivery intraoperative image guidance

with Simbionix and Prof J Sosna

copy L Joskowicz 2011

Evaluation brain tumor follow-up

Dec 15 2009 June 9 2010

Disease

progression T2-weighted T1-weighted

Tumor internal components

solid enhancing cyst

with TA Sourasky and Dana Hospital

copy L Joskowicz 2011

Key issue model creation

Currently

bull mostly manual delineation

bull slice by slice

Desired

bull automaticnearly automatic ndash a few

clicks by physician

bull no technician

bull accurate and reliable

HARD NO METHOD

SUITABLE FOR ALL STRUCTURES

Modeling requires segmentation

Very difficult task

ndash organpathology specific

ndash imagescanning protocol

ndash anatomical variability

ndash intensity values overlap

ndash structures proximity

Identify and delineate anatomical structure contours

1 2

3 4

Modeling requires segmentation

Very difficult task

ndash organpathology specific

ndash imagescanning protocol

ndash anatomical variability

ndash intensity values overlap

ndash structures proximity

Identify and delineate anatomical structure contours

1 2

3 4

Clinical dataset challenges

proximity calcifications stenosis

Segmentation ndash state of the art

bull Hundreds anatomical segmentation methods for a

variety of structures and imaging modalities

bull Families of techniques thresholding region

growing level sets active contours and more

bull Very few if any in routine clinical use

ndash huge gap between prototype and clinical use

ndash time-consuming fragile limited in scope

ndash require technical knowledge

ndash most have limited validation

ndash clinical benefits unproven

bull Segmentation specificity is unavoidable ndash each

anatomical structure and pathology has unique

characteristics and nearby structures

bull Organ structure and pathology-specific

algorithms heart liver long bones spine

bull Very laborious top develop and validate a

segmentation algorithm for each ndash trial and error

process many man-months

No universal segmentation method

Observations (1)

Observations (2) bull Applications have different requirements wrt

ndash Accuracy

ndash Robustness

ndash User interaction

ndash Quality

bull Consider the different requirements of

ndash 3D visualization

ndash Training simulation

ndash FEA simulation

Defining application

requirements ahead of time is

essential

Goals Clinical

bull Automatic or nearly automatic lt 5mins user time

bull Can be used by clinician without a technician

bull Produces robust results

Technical

bull Requires significantly less time than developing from

scratch

bull Takes into account intensity and shape

bull Incorporates shape and intensity priors

MIS validation

bull Algorithms without validation are clinically

worthless

bull Validation is with respect to a clinical task

bull Validation requires a ground truth for comparison

bull Physical anatomical models andor phantoms are

typically not available (except sometimes for bones)

bull Ground truth is usually obtained by manual

identification andor segmentation by a user

bull Experts in most cases radiologists

bull Extrinsic comparison compare vs other methods

Copyright L Joskowicz 2010

Copyright L Joskowicz 2010

MIS validation issues

bull Large inter- and intra- variability across experts and clinical sites

bull May not be representative of population variability

bull Main quantitative parameters

ndash validation set size

ndash number and type of observers

ndash intra and inter-observer manual segmentation variability

ndash surface-based and volume-based error measures

MIS validation anatomical variability

stenosis

looping

narrowing

MIS validation anatomical variability

MIS validation metrics

bull Surface-based error measurements

ndash Mean surface distance

ndash RMS surface distance

ndash Maximum surface distance

bull Volume-based error measurements

ndash Dice coefficient

ndash Volumetric overlap error

Copyright L Joskowicz 2010

MIS validation metrics

Surface-based error measurements

bull Mean surface distance

bull RMS surface distance

bull Maximum surface distance

Copyright L Joskowicz 2010

Reference Result

MIS validation metrics

Volume-based error measurements

bull Volumetric overlap error

bull Dice similarity

Copyright L Joskowicz 2010

Reference Result

Aorta

Int + Ext Carotids

All arteries

Subclavian Arteries Bifurcations

16mm (std=08mm)

13mm (std=10mm)

15mm

std=13mm 17mm

std=09mm

07mm

std=07mm

Surface

RMS

Common

Carotid

MIS validation observers variability

bull Intra-observer variability determine how much

variation there is when a single observer

produced the ground-truth segmentation

repeat 5 times the segmentation

bull Inter-observer variability determine how much

variation there is between multiple observers that

produced the ground-truth segmentation

ask 3 radiologists to do the

segmentation

bull Observer expertise and frequency variability

Copyright L Joskowicz 2010

Study [Weltens 2001]

bull Axial MRI slice

bull Nine independent

observers

bull Repeated delineations

bull Interintra observer

variability 30

Key issue fuzzy tumor boundaries

Inter observer variability

Intra-observer variability

8

50

Carotid Lumen Segmentation and Stenosis

Grading Challenge -- The MIDAS Journal

Copyright L Joskowicz 2010

MIS validation manual annotation

Copyright L Joskowicz 2010

MIS validation ground-truth generation

centerlines

(manual)

contours

(manual)

Partial Volume

Segmentation

Reference

standard

from 3PVS

Copyright L Joskowicz 2010

Measurements characteristics

bull Ground-truth not know

bull Repeatability intra-observer variability

bull Reliability inter observer variability

bull Measure correlation between repeated

measurements

bull Intrinsic uncertainty about the tumor volume

bull Accuracy for clinical significance is unknown

Inaccuracy gt Uncertainty

bull Uncertainty may not be improved

bull Goal improve accuracy to obtain

Inaccuracy vs Uncertainty

Results can be

meaningless

Inaccuracy lt Uncertainty

In summaryhellip

bull Medical image segmentation is on the risehellip

bull Basis of many clinical applications

bull Validation is a MUST

bull Many methods and approaches ndash do not re-invent

the wheel

bull I expect patient-specific 3D models to be in the

clinical mainstream in 3-5 years

Copyright L Joskowicz 2011

Segmentation challenges

bull Rigorous quantification and evaluation of

segmentation algorithms performance

bull Shape priors for pathologies

bull Incorporation of functional information from

diffusionperfusion MRI fMRI PET into

segmentation algorithms

Copyright L Joskowicz 2011

Summary (1)

bull Patient-specific anatomy model creation is

currently a major bottleneck in many clinical

applications

bull Automatic anatomical segmentation is essential for

model creation

bull Current tools are limited in scope and coverage

bull Clinical use requires the elimination of the

technician ndash model generation by the physician

Summary (2) bull Great opportunity for the development and

incorporation of anatomy modeling tools in

commercial platforms

bull Growing need and variety of users for anatomical

models

ndash Training simulators surgery rehersal

ndash Intraoperative guidance

ndash Computer Aided Radiology

bull Service providers -- shifting paradigms

Page 9: Recent clinical advances and applications for medical ...crl.med.harvard.edu/research/MICCAI_Tutorial/MICCAI11-tutorial... · Recent clinical advances and applications for medical

copy L Joskowicz 2011

3D models in the patient treatment cycle

Diagnosis Planning

Delivery

CAD mammography

Virtual colonoscopy

Neurosurgery -- trajectory

Orthopaedics -- fixation

Tumor follow-up

Implant location

MODEL Interventional

Radiology

Navigation

Robotics Evaluation

Training

Learning

copy L Joskowicz 2011

Diagnosis computer-aided radiology

stenosis

thrombus aneurism

tumor

volume

with Prof J Sosna

copy L Joskowicz 2011

liver contour

blood vessels

tumors

4-phase CT dataset

kidneycontour

blood vessels

urinary vessels with Dr Y Mintz with Prof J Sosna

Diagnosis computer-aided radiology

copy L Joskowicz 2011

Planning neurosurgery

entry point

with Dr Y Shoshan

copy L Joskowicz 2011

Planning neurosurgery

with Dr Y Shoshan

copy L Joskowicz 2011

Conventional Our method

Planning neurosurgery

with Dr Y Shoshan

copy L Joskowicz 2011

Planning orthopaedics

internal fixation external fixation

Fracture

fixation

alternatives

with Dr E Peleg and Profs M Liebergall R Mosheiff

copy L Joskowicz 2011 with Dr E Peleg and Profs M Liebergall R Mosheiff

Planning orthopaedics

copy L Joskowicz 2011

Planning orthopaedics

with Dr E Peleg and Profs M Liebergall R Mosheiff

copy L Joskowicz 2011

bull EM real-time tracking

bull US and X-ray imaging

add patient-specific

models from CT

Delivery interventional radiology

copy L Joskowicz 2011

Delivery intraoperative image guidance

augmented continuous X-ray fluoroscopy

with Simbionix and Prof J Sosna

copy L Joskowicz 2011

Delivery intraoperative image guidance

with Simbionix and Prof J Sosna

copy L Joskowicz 2011

Evaluation brain tumor follow-up

Dec 15 2009 June 9 2010

Disease

progression T2-weighted T1-weighted

Tumor internal components

solid enhancing cyst

with TA Sourasky and Dana Hospital

copy L Joskowicz 2011

Key issue model creation

Currently

bull mostly manual delineation

bull slice by slice

Desired

bull automaticnearly automatic ndash a few

clicks by physician

bull no technician

bull accurate and reliable

HARD NO METHOD

SUITABLE FOR ALL STRUCTURES

Modeling requires segmentation

Very difficult task

ndash organpathology specific

ndash imagescanning protocol

ndash anatomical variability

ndash intensity values overlap

ndash structures proximity

Identify and delineate anatomical structure contours

1 2

3 4

Modeling requires segmentation

Very difficult task

ndash organpathology specific

ndash imagescanning protocol

ndash anatomical variability

ndash intensity values overlap

ndash structures proximity

Identify and delineate anatomical structure contours

1 2

3 4

Clinical dataset challenges

proximity calcifications stenosis

Segmentation ndash state of the art

bull Hundreds anatomical segmentation methods for a

variety of structures and imaging modalities

bull Families of techniques thresholding region

growing level sets active contours and more

bull Very few if any in routine clinical use

ndash huge gap between prototype and clinical use

ndash time-consuming fragile limited in scope

ndash require technical knowledge

ndash most have limited validation

ndash clinical benefits unproven

bull Segmentation specificity is unavoidable ndash each

anatomical structure and pathology has unique

characteristics and nearby structures

bull Organ structure and pathology-specific

algorithms heart liver long bones spine

bull Very laborious top develop and validate a

segmentation algorithm for each ndash trial and error

process many man-months

No universal segmentation method

Observations (1)

Observations (2) bull Applications have different requirements wrt

ndash Accuracy

ndash Robustness

ndash User interaction

ndash Quality

bull Consider the different requirements of

ndash 3D visualization

ndash Training simulation

ndash FEA simulation

Defining application

requirements ahead of time is

essential

Goals Clinical

bull Automatic or nearly automatic lt 5mins user time

bull Can be used by clinician without a technician

bull Produces robust results

Technical

bull Requires significantly less time than developing from

scratch

bull Takes into account intensity and shape

bull Incorporates shape and intensity priors

MIS validation

bull Algorithms without validation are clinically

worthless

bull Validation is with respect to a clinical task

bull Validation requires a ground truth for comparison

bull Physical anatomical models andor phantoms are

typically not available (except sometimes for bones)

bull Ground truth is usually obtained by manual

identification andor segmentation by a user

bull Experts in most cases radiologists

bull Extrinsic comparison compare vs other methods

Copyright L Joskowicz 2010

Copyright L Joskowicz 2010

MIS validation issues

bull Large inter- and intra- variability across experts and clinical sites

bull May not be representative of population variability

bull Main quantitative parameters

ndash validation set size

ndash number and type of observers

ndash intra and inter-observer manual segmentation variability

ndash surface-based and volume-based error measures

MIS validation anatomical variability

stenosis

looping

narrowing

MIS validation anatomical variability

MIS validation metrics

bull Surface-based error measurements

ndash Mean surface distance

ndash RMS surface distance

ndash Maximum surface distance

bull Volume-based error measurements

ndash Dice coefficient

ndash Volumetric overlap error

Copyright L Joskowicz 2010

MIS validation metrics

Surface-based error measurements

bull Mean surface distance

bull RMS surface distance

bull Maximum surface distance

Copyright L Joskowicz 2010

Reference Result

MIS validation metrics

Volume-based error measurements

bull Volumetric overlap error

bull Dice similarity

Copyright L Joskowicz 2010

Reference Result

Aorta

Int + Ext Carotids

All arteries

Subclavian Arteries Bifurcations

16mm (std=08mm)

13mm (std=10mm)

15mm

std=13mm 17mm

std=09mm

07mm

std=07mm

Surface

RMS

Common

Carotid

MIS validation observers variability

bull Intra-observer variability determine how much

variation there is when a single observer

produced the ground-truth segmentation

repeat 5 times the segmentation

bull Inter-observer variability determine how much

variation there is between multiple observers that

produced the ground-truth segmentation

ask 3 radiologists to do the

segmentation

bull Observer expertise and frequency variability

Copyright L Joskowicz 2010

Study [Weltens 2001]

bull Axial MRI slice

bull Nine independent

observers

bull Repeated delineations

bull Interintra observer

variability 30

Key issue fuzzy tumor boundaries

Inter observer variability

Intra-observer variability

8

50

Carotid Lumen Segmentation and Stenosis

Grading Challenge -- The MIDAS Journal

Copyright L Joskowicz 2010

MIS validation manual annotation

Copyright L Joskowicz 2010

MIS validation ground-truth generation

centerlines

(manual)

contours

(manual)

Partial Volume

Segmentation

Reference

standard

from 3PVS

Copyright L Joskowicz 2010

Measurements characteristics

bull Ground-truth not know

bull Repeatability intra-observer variability

bull Reliability inter observer variability

bull Measure correlation between repeated

measurements

bull Intrinsic uncertainty about the tumor volume

bull Accuracy for clinical significance is unknown

Inaccuracy gt Uncertainty

bull Uncertainty may not be improved

bull Goal improve accuracy to obtain

Inaccuracy vs Uncertainty

Results can be

meaningless

Inaccuracy lt Uncertainty

In summaryhellip

bull Medical image segmentation is on the risehellip

bull Basis of many clinical applications

bull Validation is a MUST

bull Many methods and approaches ndash do not re-invent

the wheel

bull I expect patient-specific 3D models to be in the

clinical mainstream in 3-5 years

Copyright L Joskowicz 2011

Segmentation challenges

bull Rigorous quantification and evaluation of

segmentation algorithms performance

bull Shape priors for pathologies

bull Incorporation of functional information from

diffusionperfusion MRI fMRI PET into

segmentation algorithms

Copyright L Joskowicz 2011

Summary (1)

bull Patient-specific anatomy model creation is

currently a major bottleneck in many clinical

applications

bull Automatic anatomical segmentation is essential for

model creation

bull Current tools are limited in scope and coverage

bull Clinical use requires the elimination of the

technician ndash model generation by the physician

Summary (2) bull Great opportunity for the development and

incorporation of anatomy modeling tools in

commercial platforms

bull Growing need and variety of users for anatomical

models

ndash Training simulators surgery rehersal

ndash Intraoperative guidance

ndash Computer Aided Radiology

bull Service providers -- shifting paradigms

Page 10: Recent clinical advances and applications for medical ...crl.med.harvard.edu/research/MICCAI_Tutorial/MICCAI11-tutorial... · Recent clinical advances and applications for medical

copy L Joskowicz 2011

Diagnosis computer-aided radiology

stenosis

thrombus aneurism

tumor

volume

with Prof J Sosna

copy L Joskowicz 2011

liver contour

blood vessels

tumors

4-phase CT dataset

kidneycontour

blood vessels

urinary vessels with Dr Y Mintz with Prof J Sosna

Diagnosis computer-aided radiology

copy L Joskowicz 2011

Planning neurosurgery

entry point

with Dr Y Shoshan

copy L Joskowicz 2011

Planning neurosurgery

with Dr Y Shoshan

copy L Joskowicz 2011

Conventional Our method

Planning neurosurgery

with Dr Y Shoshan

copy L Joskowicz 2011

Planning orthopaedics

internal fixation external fixation

Fracture

fixation

alternatives

with Dr E Peleg and Profs M Liebergall R Mosheiff

copy L Joskowicz 2011 with Dr E Peleg and Profs M Liebergall R Mosheiff

Planning orthopaedics

copy L Joskowicz 2011

Planning orthopaedics

with Dr E Peleg and Profs M Liebergall R Mosheiff

copy L Joskowicz 2011

bull EM real-time tracking

bull US and X-ray imaging

add patient-specific

models from CT

Delivery interventional radiology

copy L Joskowicz 2011

Delivery intraoperative image guidance

augmented continuous X-ray fluoroscopy

with Simbionix and Prof J Sosna

copy L Joskowicz 2011

Delivery intraoperative image guidance

with Simbionix and Prof J Sosna

copy L Joskowicz 2011

Evaluation brain tumor follow-up

Dec 15 2009 June 9 2010

Disease

progression T2-weighted T1-weighted

Tumor internal components

solid enhancing cyst

with TA Sourasky and Dana Hospital

copy L Joskowicz 2011

Key issue model creation

Currently

bull mostly manual delineation

bull slice by slice

Desired

bull automaticnearly automatic ndash a few

clicks by physician

bull no technician

bull accurate and reliable

HARD NO METHOD

SUITABLE FOR ALL STRUCTURES

Modeling requires segmentation

Very difficult task

ndash organpathology specific

ndash imagescanning protocol

ndash anatomical variability

ndash intensity values overlap

ndash structures proximity

Identify and delineate anatomical structure contours

1 2

3 4

Modeling requires segmentation

Very difficult task

ndash organpathology specific

ndash imagescanning protocol

ndash anatomical variability

ndash intensity values overlap

ndash structures proximity

Identify and delineate anatomical structure contours

1 2

3 4

Clinical dataset challenges

proximity calcifications stenosis

Segmentation ndash state of the art

bull Hundreds anatomical segmentation methods for a

variety of structures and imaging modalities

bull Families of techniques thresholding region

growing level sets active contours and more

bull Very few if any in routine clinical use

ndash huge gap between prototype and clinical use

ndash time-consuming fragile limited in scope

ndash require technical knowledge

ndash most have limited validation

ndash clinical benefits unproven

bull Segmentation specificity is unavoidable ndash each

anatomical structure and pathology has unique

characteristics and nearby structures

bull Organ structure and pathology-specific

algorithms heart liver long bones spine

bull Very laborious top develop and validate a

segmentation algorithm for each ndash trial and error

process many man-months

No universal segmentation method

Observations (1)

Observations (2) bull Applications have different requirements wrt

ndash Accuracy

ndash Robustness

ndash User interaction

ndash Quality

bull Consider the different requirements of

ndash 3D visualization

ndash Training simulation

ndash FEA simulation

Defining application

requirements ahead of time is

essential

Goals Clinical

bull Automatic or nearly automatic lt 5mins user time

bull Can be used by clinician without a technician

bull Produces robust results

Technical

bull Requires significantly less time than developing from

scratch

bull Takes into account intensity and shape

bull Incorporates shape and intensity priors

MIS validation

bull Algorithms without validation are clinically

worthless

bull Validation is with respect to a clinical task

bull Validation requires a ground truth for comparison

bull Physical anatomical models andor phantoms are

typically not available (except sometimes for bones)

bull Ground truth is usually obtained by manual

identification andor segmentation by a user

bull Experts in most cases radiologists

bull Extrinsic comparison compare vs other methods

Copyright L Joskowicz 2010

Copyright L Joskowicz 2010

MIS validation issues

bull Large inter- and intra- variability across experts and clinical sites

bull May not be representative of population variability

bull Main quantitative parameters

ndash validation set size

ndash number and type of observers

ndash intra and inter-observer manual segmentation variability

ndash surface-based and volume-based error measures

MIS validation anatomical variability

stenosis

looping

narrowing

MIS validation anatomical variability

MIS validation metrics

bull Surface-based error measurements

ndash Mean surface distance

ndash RMS surface distance

ndash Maximum surface distance

bull Volume-based error measurements

ndash Dice coefficient

ndash Volumetric overlap error

Copyright L Joskowicz 2010

MIS validation metrics

Surface-based error measurements

bull Mean surface distance

bull RMS surface distance

bull Maximum surface distance

Copyright L Joskowicz 2010

Reference Result

MIS validation metrics

Volume-based error measurements

bull Volumetric overlap error

bull Dice similarity

Copyright L Joskowicz 2010

Reference Result

Aorta

Int + Ext Carotids

All arteries

Subclavian Arteries Bifurcations

16mm (std=08mm)

13mm (std=10mm)

15mm

std=13mm 17mm

std=09mm

07mm

std=07mm

Surface

RMS

Common

Carotid

MIS validation observers variability

bull Intra-observer variability determine how much

variation there is when a single observer

produced the ground-truth segmentation

repeat 5 times the segmentation

bull Inter-observer variability determine how much

variation there is between multiple observers that

produced the ground-truth segmentation

ask 3 radiologists to do the

segmentation

bull Observer expertise and frequency variability

Copyright L Joskowicz 2010

Study [Weltens 2001]

bull Axial MRI slice

bull Nine independent

observers

bull Repeated delineations

bull Interintra observer

variability 30

Key issue fuzzy tumor boundaries

Inter observer variability

Intra-observer variability

8

50

Carotid Lumen Segmentation and Stenosis

Grading Challenge -- The MIDAS Journal

Copyright L Joskowicz 2010

MIS validation manual annotation

Copyright L Joskowicz 2010

MIS validation ground-truth generation

centerlines

(manual)

contours

(manual)

Partial Volume

Segmentation

Reference

standard

from 3PVS

Copyright L Joskowicz 2010

Measurements characteristics

bull Ground-truth not know

bull Repeatability intra-observer variability

bull Reliability inter observer variability

bull Measure correlation between repeated

measurements

bull Intrinsic uncertainty about the tumor volume

bull Accuracy for clinical significance is unknown

Inaccuracy gt Uncertainty

bull Uncertainty may not be improved

bull Goal improve accuracy to obtain

Inaccuracy vs Uncertainty

Results can be

meaningless

Inaccuracy lt Uncertainty

In summaryhellip

bull Medical image segmentation is on the risehellip

bull Basis of many clinical applications

bull Validation is a MUST

bull Many methods and approaches ndash do not re-invent

the wheel

bull I expect patient-specific 3D models to be in the

clinical mainstream in 3-5 years

Copyright L Joskowicz 2011

Segmentation challenges

bull Rigorous quantification and evaluation of

segmentation algorithms performance

bull Shape priors for pathologies

bull Incorporation of functional information from

diffusionperfusion MRI fMRI PET into

segmentation algorithms

Copyright L Joskowicz 2011

Summary (1)

bull Patient-specific anatomy model creation is

currently a major bottleneck in many clinical

applications

bull Automatic anatomical segmentation is essential for

model creation

bull Current tools are limited in scope and coverage

bull Clinical use requires the elimination of the

technician ndash model generation by the physician

Summary (2) bull Great opportunity for the development and

incorporation of anatomy modeling tools in

commercial platforms

bull Growing need and variety of users for anatomical

models

ndash Training simulators surgery rehersal

ndash Intraoperative guidance

ndash Computer Aided Radiology

bull Service providers -- shifting paradigms

Page 11: Recent clinical advances and applications for medical ...crl.med.harvard.edu/research/MICCAI_Tutorial/MICCAI11-tutorial... · Recent clinical advances and applications for medical

copy L Joskowicz 2011

liver contour

blood vessels

tumors

4-phase CT dataset

kidneycontour

blood vessels

urinary vessels with Dr Y Mintz with Prof J Sosna

Diagnosis computer-aided radiology

copy L Joskowicz 2011

Planning neurosurgery

entry point

with Dr Y Shoshan

copy L Joskowicz 2011

Planning neurosurgery

with Dr Y Shoshan

copy L Joskowicz 2011

Conventional Our method

Planning neurosurgery

with Dr Y Shoshan

copy L Joskowicz 2011

Planning orthopaedics

internal fixation external fixation

Fracture

fixation

alternatives

with Dr E Peleg and Profs M Liebergall R Mosheiff

copy L Joskowicz 2011 with Dr E Peleg and Profs M Liebergall R Mosheiff

Planning orthopaedics

copy L Joskowicz 2011

Planning orthopaedics

with Dr E Peleg and Profs M Liebergall R Mosheiff

copy L Joskowicz 2011

bull EM real-time tracking

bull US and X-ray imaging

add patient-specific

models from CT

Delivery interventional radiology

copy L Joskowicz 2011

Delivery intraoperative image guidance

augmented continuous X-ray fluoroscopy

with Simbionix and Prof J Sosna

copy L Joskowicz 2011

Delivery intraoperative image guidance

with Simbionix and Prof J Sosna

copy L Joskowicz 2011

Evaluation brain tumor follow-up

Dec 15 2009 June 9 2010

Disease

progression T2-weighted T1-weighted

Tumor internal components

solid enhancing cyst

with TA Sourasky and Dana Hospital

copy L Joskowicz 2011

Key issue model creation

Currently

bull mostly manual delineation

bull slice by slice

Desired

bull automaticnearly automatic ndash a few

clicks by physician

bull no technician

bull accurate and reliable

HARD NO METHOD

SUITABLE FOR ALL STRUCTURES

Modeling requires segmentation

Very difficult task

ndash organpathology specific

ndash imagescanning protocol

ndash anatomical variability

ndash intensity values overlap

ndash structures proximity

Identify and delineate anatomical structure contours

1 2

3 4

Modeling requires segmentation

Very difficult task

ndash organpathology specific

ndash imagescanning protocol

ndash anatomical variability

ndash intensity values overlap

ndash structures proximity

Identify and delineate anatomical structure contours

1 2

3 4

Clinical dataset challenges

proximity calcifications stenosis

Segmentation ndash state of the art

bull Hundreds anatomical segmentation methods for a

variety of structures and imaging modalities

bull Families of techniques thresholding region

growing level sets active contours and more

bull Very few if any in routine clinical use

ndash huge gap between prototype and clinical use

ndash time-consuming fragile limited in scope

ndash require technical knowledge

ndash most have limited validation

ndash clinical benefits unproven

bull Segmentation specificity is unavoidable ndash each

anatomical structure and pathology has unique

characteristics and nearby structures

bull Organ structure and pathology-specific

algorithms heart liver long bones spine

bull Very laborious top develop and validate a

segmentation algorithm for each ndash trial and error

process many man-months

No universal segmentation method

Observations (1)

Observations (2) bull Applications have different requirements wrt

ndash Accuracy

ndash Robustness

ndash User interaction

ndash Quality

bull Consider the different requirements of

ndash 3D visualization

ndash Training simulation

ndash FEA simulation

Defining application

requirements ahead of time is

essential

Goals Clinical

bull Automatic or nearly automatic lt 5mins user time

bull Can be used by clinician without a technician

bull Produces robust results

Technical

bull Requires significantly less time than developing from

scratch

bull Takes into account intensity and shape

bull Incorporates shape and intensity priors

MIS validation

bull Algorithms without validation are clinically

worthless

bull Validation is with respect to a clinical task

bull Validation requires a ground truth for comparison

bull Physical anatomical models andor phantoms are

typically not available (except sometimes for bones)

bull Ground truth is usually obtained by manual

identification andor segmentation by a user

bull Experts in most cases radiologists

bull Extrinsic comparison compare vs other methods

Copyright L Joskowicz 2010

Copyright L Joskowicz 2010

MIS validation issues

bull Large inter- and intra- variability across experts and clinical sites

bull May not be representative of population variability

bull Main quantitative parameters

ndash validation set size

ndash number and type of observers

ndash intra and inter-observer manual segmentation variability

ndash surface-based and volume-based error measures

MIS validation anatomical variability

stenosis

looping

narrowing

MIS validation anatomical variability

MIS validation metrics

bull Surface-based error measurements

ndash Mean surface distance

ndash RMS surface distance

ndash Maximum surface distance

bull Volume-based error measurements

ndash Dice coefficient

ndash Volumetric overlap error

Copyright L Joskowicz 2010

MIS validation metrics

Surface-based error measurements

bull Mean surface distance

bull RMS surface distance

bull Maximum surface distance

Copyright L Joskowicz 2010

Reference Result

MIS validation metrics

Volume-based error measurements

bull Volumetric overlap error

bull Dice similarity

Copyright L Joskowicz 2010

Reference Result

Aorta

Int + Ext Carotids

All arteries

Subclavian Arteries Bifurcations

16mm (std=08mm)

13mm (std=10mm)

15mm

std=13mm 17mm

std=09mm

07mm

std=07mm

Surface

RMS

Common

Carotid

MIS validation observers variability

bull Intra-observer variability determine how much

variation there is when a single observer

produced the ground-truth segmentation

repeat 5 times the segmentation

bull Inter-observer variability determine how much

variation there is between multiple observers that

produced the ground-truth segmentation

ask 3 radiologists to do the

segmentation

bull Observer expertise and frequency variability

Copyright L Joskowicz 2010

Study [Weltens 2001]

bull Axial MRI slice

bull Nine independent

observers

bull Repeated delineations

bull Interintra observer

variability 30

Key issue fuzzy tumor boundaries

Inter observer variability

Intra-observer variability

8

50

Carotid Lumen Segmentation and Stenosis

Grading Challenge -- The MIDAS Journal

Copyright L Joskowicz 2010

MIS validation manual annotation

Copyright L Joskowicz 2010

MIS validation ground-truth generation

centerlines

(manual)

contours

(manual)

Partial Volume

Segmentation

Reference

standard

from 3PVS

Copyright L Joskowicz 2010

Measurements characteristics

bull Ground-truth not know

bull Repeatability intra-observer variability

bull Reliability inter observer variability

bull Measure correlation between repeated

measurements

bull Intrinsic uncertainty about the tumor volume

bull Accuracy for clinical significance is unknown

Inaccuracy gt Uncertainty

bull Uncertainty may not be improved

bull Goal improve accuracy to obtain

Inaccuracy vs Uncertainty

Results can be

meaningless

Inaccuracy lt Uncertainty

In summaryhellip

bull Medical image segmentation is on the risehellip

bull Basis of many clinical applications

bull Validation is a MUST

bull Many methods and approaches ndash do not re-invent

the wheel

bull I expect patient-specific 3D models to be in the

clinical mainstream in 3-5 years

Copyright L Joskowicz 2011

Segmentation challenges

bull Rigorous quantification and evaluation of

segmentation algorithms performance

bull Shape priors for pathologies

bull Incorporation of functional information from

diffusionperfusion MRI fMRI PET into

segmentation algorithms

Copyright L Joskowicz 2011

Summary (1)

bull Patient-specific anatomy model creation is

currently a major bottleneck in many clinical

applications

bull Automatic anatomical segmentation is essential for

model creation

bull Current tools are limited in scope and coverage

bull Clinical use requires the elimination of the

technician ndash model generation by the physician

Summary (2) bull Great opportunity for the development and

incorporation of anatomy modeling tools in

commercial platforms

bull Growing need and variety of users for anatomical

models

ndash Training simulators surgery rehersal

ndash Intraoperative guidance

ndash Computer Aided Radiology

bull Service providers -- shifting paradigms

Page 12: Recent clinical advances and applications for medical ...crl.med.harvard.edu/research/MICCAI_Tutorial/MICCAI11-tutorial... · Recent clinical advances and applications for medical

copy L Joskowicz 2011

Planning neurosurgery

entry point

with Dr Y Shoshan

copy L Joskowicz 2011

Planning neurosurgery

with Dr Y Shoshan

copy L Joskowicz 2011

Conventional Our method

Planning neurosurgery

with Dr Y Shoshan

copy L Joskowicz 2011

Planning orthopaedics

internal fixation external fixation

Fracture

fixation

alternatives

with Dr E Peleg and Profs M Liebergall R Mosheiff

copy L Joskowicz 2011 with Dr E Peleg and Profs M Liebergall R Mosheiff

Planning orthopaedics

copy L Joskowicz 2011

Planning orthopaedics

with Dr E Peleg and Profs M Liebergall R Mosheiff

copy L Joskowicz 2011

bull EM real-time tracking

bull US and X-ray imaging

add patient-specific

models from CT

Delivery interventional radiology

copy L Joskowicz 2011

Delivery intraoperative image guidance

augmented continuous X-ray fluoroscopy

with Simbionix and Prof J Sosna

copy L Joskowicz 2011

Delivery intraoperative image guidance

with Simbionix and Prof J Sosna

copy L Joskowicz 2011

Evaluation brain tumor follow-up

Dec 15 2009 June 9 2010

Disease

progression T2-weighted T1-weighted

Tumor internal components

solid enhancing cyst

with TA Sourasky and Dana Hospital

copy L Joskowicz 2011

Key issue model creation

Currently

bull mostly manual delineation

bull slice by slice

Desired

bull automaticnearly automatic ndash a few

clicks by physician

bull no technician

bull accurate and reliable

HARD NO METHOD

SUITABLE FOR ALL STRUCTURES

Modeling requires segmentation

Very difficult task

ndash organpathology specific

ndash imagescanning protocol

ndash anatomical variability

ndash intensity values overlap

ndash structures proximity

Identify and delineate anatomical structure contours

1 2

3 4

Modeling requires segmentation

Very difficult task

ndash organpathology specific

ndash imagescanning protocol

ndash anatomical variability

ndash intensity values overlap

ndash structures proximity

Identify and delineate anatomical structure contours

1 2

3 4

Clinical dataset challenges

proximity calcifications stenosis

Segmentation ndash state of the art

bull Hundreds anatomical segmentation methods for a

variety of structures and imaging modalities

bull Families of techniques thresholding region

growing level sets active contours and more

bull Very few if any in routine clinical use

ndash huge gap between prototype and clinical use

ndash time-consuming fragile limited in scope

ndash require technical knowledge

ndash most have limited validation

ndash clinical benefits unproven

bull Segmentation specificity is unavoidable ndash each

anatomical structure and pathology has unique

characteristics and nearby structures

bull Organ structure and pathology-specific

algorithms heart liver long bones spine

bull Very laborious top develop and validate a

segmentation algorithm for each ndash trial and error

process many man-months

No universal segmentation method

Observations (1)

Observations (2) bull Applications have different requirements wrt

ndash Accuracy

ndash Robustness

ndash User interaction

ndash Quality

bull Consider the different requirements of

ndash 3D visualization

ndash Training simulation

ndash FEA simulation

Defining application

requirements ahead of time is

essential

Goals Clinical

bull Automatic or nearly automatic lt 5mins user time

bull Can be used by clinician without a technician

bull Produces robust results

Technical

bull Requires significantly less time than developing from

scratch

bull Takes into account intensity and shape

bull Incorporates shape and intensity priors

MIS validation

bull Algorithms without validation are clinically

worthless

bull Validation is with respect to a clinical task

bull Validation requires a ground truth for comparison

bull Physical anatomical models andor phantoms are

typically not available (except sometimes for bones)

bull Ground truth is usually obtained by manual

identification andor segmentation by a user

bull Experts in most cases radiologists

bull Extrinsic comparison compare vs other methods

Copyright L Joskowicz 2010

Copyright L Joskowicz 2010

MIS validation issues

bull Large inter- and intra- variability across experts and clinical sites

bull May not be representative of population variability

bull Main quantitative parameters

ndash validation set size

ndash number and type of observers

ndash intra and inter-observer manual segmentation variability

ndash surface-based and volume-based error measures

MIS validation anatomical variability

stenosis

looping

narrowing

MIS validation anatomical variability

MIS validation metrics

bull Surface-based error measurements

ndash Mean surface distance

ndash RMS surface distance

ndash Maximum surface distance

bull Volume-based error measurements

ndash Dice coefficient

ndash Volumetric overlap error

Copyright L Joskowicz 2010

MIS validation metrics

Surface-based error measurements

bull Mean surface distance

bull RMS surface distance

bull Maximum surface distance

Copyright L Joskowicz 2010

Reference Result

MIS validation metrics

Volume-based error measurements

bull Volumetric overlap error

bull Dice similarity

Copyright L Joskowicz 2010

Reference Result

Aorta

Int + Ext Carotids

All arteries

Subclavian Arteries Bifurcations

16mm (std=08mm)

13mm (std=10mm)

15mm

std=13mm 17mm

std=09mm

07mm

std=07mm

Surface

RMS

Common

Carotid

MIS validation observers variability

bull Intra-observer variability determine how much

variation there is when a single observer

produced the ground-truth segmentation

repeat 5 times the segmentation

bull Inter-observer variability determine how much

variation there is between multiple observers that

produced the ground-truth segmentation

ask 3 radiologists to do the

segmentation

bull Observer expertise and frequency variability

Copyright L Joskowicz 2010

Study [Weltens 2001]

bull Axial MRI slice

bull Nine independent

observers

bull Repeated delineations

bull Interintra observer

variability 30

Key issue fuzzy tumor boundaries

Inter observer variability

Intra-observer variability

8

50

Carotid Lumen Segmentation and Stenosis

Grading Challenge -- The MIDAS Journal

Copyright L Joskowicz 2010

MIS validation manual annotation

Copyright L Joskowicz 2010

MIS validation ground-truth generation

centerlines

(manual)

contours

(manual)

Partial Volume

Segmentation

Reference

standard

from 3PVS

Copyright L Joskowicz 2010

Measurements characteristics

bull Ground-truth not know

bull Repeatability intra-observer variability

bull Reliability inter observer variability

bull Measure correlation between repeated

measurements

bull Intrinsic uncertainty about the tumor volume

bull Accuracy for clinical significance is unknown

Inaccuracy gt Uncertainty

bull Uncertainty may not be improved

bull Goal improve accuracy to obtain

Inaccuracy vs Uncertainty

Results can be

meaningless

Inaccuracy lt Uncertainty

In summaryhellip

bull Medical image segmentation is on the risehellip

bull Basis of many clinical applications

bull Validation is a MUST

bull Many methods and approaches ndash do not re-invent

the wheel

bull I expect patient-specific 3D models to be in the

clinical mainstream in 3-5 years

Copyright L Joskowicz 2011

Segmentation challenges

bull Rigorous quantification and evaluation of

segmentation algorithms performance

bull Shape priors for pathologies

bull Incorporation of functional information from

diffusionperfusion MRI fMRI PET into

segmentation algorithms

Copyright L Joskowicz 2011

Summary (1)

bull Patient-specific anatomy model creation is

currently a major bottleneck in many clinical

applications

bull Automatic anatomical segmentation is essential for

model creation

bull Current tools are limited in scope and coverage

bull Clinical use requires the elimination of the

technician ndash model generation by the physician

Summary (2) bull Great opportunity for the development and

incorporation of anatomy modeling tools in

commercial platforms

bull Growing need and variety of users for anatomical

models

ndash Training simulators surgery rehersal

ndash Intraoperative guidance

ndash Computer Aided Radiology

bull Service providers -- shifting paradigms

Page 13: Recent clinical advances and applications for medical ...crl.med.harvard.edu/research/MICCAI_Tutorial/MICCAI11-tutorial... · Recent clinical advances and applications for medical

copy L Joskowicz 2011

Planning neurosurgery

with Dr Y Shoshan

copy L Joskowicz 2011

Conventional Our method

Planning neurosurgery

with Dr Y Shoshan

copy L Joskowicz 2011

Planning orthopaedics

internal fixation external fixation

Fracture

fixation

alternatives

with Dr E Peleg and Profs M Liebergall R Mosheiff

copy L Joskowicz 2011 with Dr E Peleg and Profs M Liebergall R Mosheiff

Planning orthopaedics

copy L Joskowicz 2011

Planning orthopaedics

with Dr E Peleg and Profs M Liebergall R Mosheiff

copy L Joskowicz 2011

bull EM real-time tracking

bull US and X-ray imaging

add patient-specific

models from CT

Delivery interventional radiology

copy L Joskowicz 2011

Delivery intraoperative image guidance

augmented continuous X-ray fluoroscopy

with Simbionix and Prof J Sosna

copy L Joskowicz 2011

Delivery intraoperative image guidance

with Simbionix and Prof J Sosna

copy L Joskowicz 2011

Evaluation brain tumor follow-up

Dec 15 2009 June 9 2010

Disease

progression T2-weighted T1-weighted

Tumor internal components

solid enhancing cyst

with TA Sourasky and Dana Hospital

copy L Joskowicz 2011

Key issue model creation

Currently

bull mostly manual delineation

bull slice by slice

Desired

bull automaticnearly automatic ndash a few

clicks by physician

bull no technician

bull accurate and reliable

HARD NO METHOD

SUITABLE FOR ALL STRUCTURES

Modeling requires segmentation

Very difficult task

ndash organpathology specific

ndash imagescanning protocol

ndash anatomical variability

ndash intensity values overlap

ndash structures proximity

Identify and delineate anatomical structure contours

1 2

3 4

Modeling requires segmentation

Very difficult task

ndash organpathology specific

ndash imagescanning protocol

ndash anatomical variability

ndash intensity values overlap

ndash structures proximity

Identify and delineate anatomical structure contours

1 2

3 4

Clinical dataset challenges

proximity calcifications stenosis

Segmentation ndash state of the art

bull Hundreds anatomical segmentation methods for a

variety of structures and imaging modalities

bull Families of techniques thresholding region

growing level sets active contours and more

bull Very few if any in routine clinical use

ndash huge gap between prototype and clinical use

ndash time-consuming fragile limited in scope

ndash require technical knowledge

ndash most have limited validation

ndash clinical benefits unproven

bull Segmentation specificity is unavoidable ndash each

anatomical structure and pathology has unique

characteristics and nearby structures

bull Organ structure and pathology-specific

algorithms heart liver long bones spine

bull Very laborious top develop and validate a

segmentation algorithm for each ndash trial and error

process many man-months

No universal segmentation method

Observations (1)

Observations (2) bull Applications have different requirements wrt

ndash Accuracy

ndash Robustness

ndash User interaction

ndash Quality

bull Consider the different requirements of

ndash 3D visualization

ndash Training simulation

ndash FEA simulation

Defining application

requirements ahead of time is

essential

Goals Clinical

bull Automatic or nearly automatic lt 5mins user time

bull Can be used by clinician without a technician

bull Produces robust results

Technical

bull Requires significantly less time than developing from

scratch

bull Takes into account intensity and shape

bull Incorporates shape and intensity priors

MIS validation

bull Algorithms without validation are clinically

worthless

bull Validation is with respect to a clinical task

bull Validation requires a ground truth for comparison

bull Physical anatomical models andor phantoms are

typically not available (except sometimes for bones)

bull Ground truth is usually obtained by manual

identification andor segmentation by a user

bull Experts in most cases radiologists

bull Extrinsic comparison compare vs other methods

Copyright L Joskowicz 2010

Copyright L Joskowicz 2010

MIS validation issues

bull Large inter- and intra- variability across experts and clinical sites

bull May not be representative of population variability

bull Main quantitative parameters

ndash validation set size

ndash number and type of observers

ndash intra and inter-observer manual segmentation variability

ndash surface-based and volume-based error measures

MIS validation anatomical variability

stenosis

looping

narrowing

MIS validation anatomical variability

MIS validation metrics

bull Surface-based error measurements

ndash Mean surface distance

ndash RMS surface distance

ndash Maximum surface distance

bull Volume-based error measurements

ndash Dice coefficient

ndash Volumetric overlap error

Copyright L Joskowicz 2010

MIS validation metrics

Surface-based error measurements

bull Mean surface distance

bull RMS surface distance

bull Maximum surface distance

Copyright L Joskowicz 2010

Reference Result

MIS validation metrics

Volume-based error measurements

bull Volumetric overlap error

bull Dice similarity

Copyright L Joskowicz 2010

Reference Result

Aorta

Int + Ext Carotids

All arteries

Subclavian Arteries Bifurcations

16mm (std=08mm)

13mm (std=10mm)

15mm

std=13mm 17mm

std=09mm

07mm

std=07mm

Surface

RMS

Common

Carotid

MIS validation observers variability

bull Intra-observer variability determine how much

variation there is when a single observer

produced the ground-truth segmentation

repeat 5 times the segmentation

bull Inter-observer variability determine how much

variation there is between multiple observers that

produced the ground-truth segmentation

ask 3 radiologists to do the

segmentation

bull Observer expertise and frequency variability

Copyright L Joskowicz 2010

Study [Weltens 2001]

bull Axial MRI slice

bull Nine independent

observers

bull Repeated delineations

bull Interintra observer

variability 30

Key issue fuzzy tumor boundaries

Inter observer variability

Intra-observer variability

8

50

Carotid Lumen Segmentation and Stenosis

Grading Challenge -- The MIDAS Journal

Copyright L Joskowicz 2010

MIS validation manual annotation

Copyright L Joskowicz 2010

MIS validation ground-truth generation

centerlines

(manual)

contours

(manual)

Partial Volume

Segmentation

Reference

standard

from 3PVS

Copyright L Joskowicz 2010

Measurements characteristics

bull Ground-truth not know

bull Repeatability intra-observer variability

bull Reliability inter observer variability

bull Measure correlation between repeated

measurements

bull Intrinsic uncertainty about the tumor volume

bull Accuracy for clinical significance is unknown

Inaccuracy gt Uncertainty

bull Uncertainty may not be improved

bull Goal improve accuracy to obtain

Inaccuracy vs Uncertainty

Results can be

meaningless

Inaccuracy lt Uncertainty

In summaryhellip

bull Medical image segmentation is on the risehellip

bull Basis of many clinical applications

bull Validation is a MUST

bull Many methods and approaches ndash do not re-invent

the wheel

bull I expect patient-specific 3D models to be in the

clinical mainstream in 3-5 years

Copyright L Joskowicz 2011

Segmentation challenges

bull Rigorous quantification and evaluation of

segmentation algorithms performance

bull Shape priors for pathologies

bull Incorporation of functional information from

diffusionperfusion MRI fMRI PET into

segmentation algorithms

Copyright L Joskowicz 2011

Summary (1)

bull Patient-specific anatomy model creation is

currently a major bottleneck in many clinical

applications

bull Automatic anatomical segmentation is essential for

model creation

bull Current tools are limited in scope and coverage

bull Clinical use requires the elimination of the

technician ndash model generation by the physician

Summary (2) bull Great opportunity for the development and

incorporation of anatomy modeling tools in

commercial platforms

bull Growing need and variety of users for anatomical

models

ndash Training simulators surgery rehersal

ndash Intraoperative guidance

ndash Computer Aided Radiology

bull Service providers -- shifting paradigms

Page 14: Recent clinical advances and applications for medical ...crl.med.harvard.edu/research/MICCAI_Tutorial/MICCAI11-tutorial... · Recent clinical advances and applications for medical

copy L Joskowicz 2011

Conventional Our method

Planning neurosurgery

with Dr Y Shoshan

copy L Joskowicz 2011

Planning orthopaedics

internal fixation external fixation

Fracture

fixation

alternatives

with Dr E Peleg and Profs M Liebergall R Mosheiff

copy L Joskowicz 2011 with Dr E Peleg and Profs M Liebergall R Mosheiff

Planning orthopaedics

copy L Joskowicz 2011

Planning orthopaedics

with Dr E Peleg and Profs M Liebergall R Mosheiff

copy L Joskowicz 2011

bull EM real-time tracking

bull US and X-ray imaging

add patient-specific

models from CT

Delivery interventional radiology

copy L Joskowicz 2011

Delivery intraoperative image guidance

augmented continuous X-ray fluoroscopy

with Simbionix and Prof J Sosna

copy L Joskowicz 2011

Delivery intraoperative image guidance

with Simbionix and Prof J Sosna

copy L Joskowicz 2011

Evaluation brain tumor follow-up

Dec 15 2009 June 9 2010

Disease

progression T2-weighted T1-weighted

Tumor internal components

solid enhancing cyst

with TA Sourasky and Dana Hospital

copy L Joskowicz 2011

Key issue model creation

Currently

bull mostly manual delineation

bull slice by slice

Desired

bull automaticnearly automatic ndash a few

clicks by physician

bull no technician

bull accurate and reliable

HARD NO METHOD

SUITABLE FOR ALL STRUCTURES

Modeling requires segmentation

Very difficult task

ndash organpathology specific

ndash imagescanning protocol

ndash anatomical variability

ndash intensity values overlap

ndash structures proximity

Identify and delineate anatomical structure contours

1 2

3 4

Modeling requires segmentation

Very difficult task

ndash organpathology specific

ndash imagescanning protocol

ndash anatomical variability

ndash intensity values overlap

ndash structures proximity

Identify and delineate anatomical structure contours

1 2

3 4

Clinical dataset challenges

proximity calcifications stenosis

Segmentation ndash state of the art

bull Hundreds anatomical segmentation methods for a

variety of structures and imaging modalities

bull Families of techniques thresholding region

growing level sets active contours and more

bull Very few if any in routine clinical use

ndash huge gap between prototype and clinical use

ndash time-consuming fragile limited in scope

ndash require technical knowledge

ndash most have limited validation

ndash clinical benefits unproven

bull Segmentation specificity is unavoidable ndash each

anatomical structure and pathology has unique

characteristics and nearby structures

bull Organ structure and pathology-specific

algorithms heart liver long bones spine

bull Very laborious top develop and validate a

segmentation algorithm for each ndash trial and error

process many man-months

No universal segmentation method

Observations (1)

Observations (2) bull Applications have different requirements wrt

ndash Accuracy

ndash Robustness

ndash User interaction

ndash Quality

bull Consider the different requirements of

ndash 3D visualization

ndash Training simulation

ndash FEA simulation

Defining application

requirements ahead of time is

essential

Goals Clinical

bull Automatic or nearly automatic lt 5mins user time

bull Can be used by clinician without a technician

bull Produces robust results

Technical

bull Requires significantly less time than developing from

scratch

bull Takes into account intensity and shape

bull Incorporates shape and intensity priors

MIS validation

bull Algorithms without validation are clinically

worthless

bull Validation is with respect to a clinical task

bull Validation requires a ground truth for comparison

bull Physical anatomical models andor phantoms are

typically not available (except sometimes for bones)

bull Ground truth is usually obtained by manual

identification andor segmentation by a user

bull Experts in most cases radiologists

bull Extrinsic comparison compare vs other methods

Copyright L Joskowicz 2010

Copyright L Joskowicz 2010

MIS validation issues

bull Large inter- and intra- variability across experts and clinical sites

bull May not be representative of population variability

bull Main quantitative parameters

ndash validation set size

ndash number and type of observers

ndash intra and inter-observer manual segmentation variability

ndash surface-based and volume-based error measures

MIS validation anatomical variability

stenosis

looping

narrowing

MIS validation anatomical variability

MIS validation metrics

bull Surface-based error measurements

ndash Mean surface distance

ndash RMS surface distance

ndash Maximum surface distance

bull Volume-based error measurements

ndash Dice coefficient

ndash Volumetric overlap error

Copyright L Joskowicz 2010

MIS validation metrics

Surface-based error measurements

bull Mean surface distance

bull RMS surface distance

bull Maximum surface distance

Copyright L Joskowicz 2010

Reference Result

MIS validation metrics

Volume-based error measurements

bull Volumetric overlap error

bull Dice similarity

Copyright L Joskowicz 2010

Reference Result

Aorta

Int + Ext Carotids

All arteries

Subclavian Arteries Bifurcations

16mm (std=08mm)

13mm (std=10mm)

15mm

std=13mm 17mm

std=09mm

07mm

std=07mm

Surface

RMS

Common

Carotid

MIS validation observers variability

bull Intra-observer variability determine how much

variation there is when a single observer

produced the ground-truth segmentation

repeat 5 times the segmentation

bull Inter-observer variability determine how much

variation there is between multiple observers that

produced the ground-truth segmentation

ask 3 radiologists to do the

segmentation

bull Observer expertise and frequency variability

Copyright L Joskowicz 2010

Study [Weltens 2001]

bull Axial MRI slice

bull Nine independent

observers

bull Repeated delineations

bull Interintra observer

variability 30

Key issue fuzzy tumor boundaries

Inter observer variability

Intra-observer variability

8

50

Carotid Lumen Segmentation and Stenosis

Grading Challenge -- The MIDAS Journal

Copyright L Joskowicz 2010

MIS validation manual annotation

Copyright L Joskowicz 2010

MIS validation ground-truth generation

centerlines

(manual)

contours

(manual)

Partial Volume

Segmentation

Reference

standard

from 3PVS

Copyright L Joskowicz 2010

Measurements characteristics

bull Ground-truth not know

bull Repeatability intra-observer variability

bull Reliability inter observer variability

bull Measure correlation between repeated

measurements

bull Intrinsic uncertainty about the tumor volume

bull Accuracy for clinical significance is unknown

Inaccuracy gt Uncertainty

bull Uncertainty may not be improved

bull Goal improve accuracy to obtain

Inaccuracy vs Uncertainty

Results can be

meaningless

Inaccuracy lt Uncertainty

In summaryhellip

bull Medical image segmentation is on the risehellip

bull Basis of many clinical applications

bull Validation is a MUST

bull Many methods and approaches ndash do not re-invent

the wheel

bull I expect patient-specific 3D models to be in the

clinical mainstream in 3-5 years

Copyright L Joskowicz 2011

Segmentation challenges

bull Rigorous quantification and evaluation of

segmentation algorithms performance

bull Shape priors for pathologies

bull Incorporation of functional information from

diffusionperfusion MRI fMRI PET into

segmentation algorithms

Copyright L Joskowicz 2011

Summary (1)

bull Patient-specific anatomy model creation is

currently a major bottleneck in many clinical

applications

bull Automatic anatomical segmentation is essential for

model creation

bull Current tools are limited in scope and coverage

bull Clinical use requires the elimination of the

technician ndash model generation by the physician

Summary (2) bull Great opportunity for the development and

incorporation of anatomy modeling tools in

commercial platforms

bull Growing need and variety of users for anatomical

models

ndash Training simulators surgery rehersal

ndash Intraoperative guidance

ndash Computer Aided Radiology

bull Service providers -- shifting paradigms

Page 15: Recent clinical advances and applications for medical ...crl.med.harvard.edu/research/MICCAI_Tutorial/MICCAI11-tutorial... · Recent clinical advances and applications for medical

copy L Joskowicz 2011

Planning orthopaedics

internal fixation external fixation

Fracture

fixation

alternatives

with Dr E Peleg and Profs M Liebergall R Mosheiff

copy L Joskowicz 2011 with Dr E Peleg and Profs M Liebergall R Mosheiff

Planning orthopaedics

copy L Joskowicz 2011

Planning orthopaedics

with Dr E Peleg and Profs M Liebergall R Mosheiff

copy L Joskowicz 2011

bull EM real-time tracking

bull US and X-ray imaging

add patient-specific

models from CT

Delivery interventional radiology

copy L Joskowicz 2011

Delivery intraoperative image guidance

augmented continuous X-ray fluoroscopy

with Simbionix and Prof J Sosna

copy L Joskowicz 2011

Delivery intraoperative image guidance

with Simbionix and Prof J Sosna

copy L Joskowicz 2011

Evaluation brain tumor follow-up

Dec 15 2009 June 9 2010

Disease

progression T2-weighted T1-weighted

Tumor internal components

solid enhancing cyst

with TA Sourasky and Dana Hospital

copy L Joskowicz 2011

Key issue model creation

Currently

bull mostly manual delineation

bull slice by slice

Desired

bull automaticnearly automatic ndash a few

clicks by physician

bull no technician

bull accurate and reliable

HARD NO METHOD

SUITABLE FOR ALL STRUCTURES

Modeling requires segmentation

Very difficult task

ndash organpathology specific

ndash imagescanning protocol

ndash anatomical variability

ndash intensity values overlap

ndash structures proximity

Identify and delineate anatomical structure contours

1 2

3 4

Modeling requires segmentation

Very difficult task

ndash organpathology specific

ndash imagescanning protocol

ndash anatomical variability

ndash intensity values overlap

ndash structures proximity

Identify and delineate anatomical structure contours

1 2

3 4

Clinical dataset challenges

proximity calcifications stenosis

Segmentation ndash state of the art

bull Hundreds anatomical segmentation methods for a

variety of structures and imaging modalities

bull Families of techniques thresholding region

growing level sets active contours and more

bull Very few if any in routine clinical use

ndash huge gap between prototype and clinical use

ndash time-consuming fragile limited in scope

ndash require technical knowledge

ndash most have limited validation

ndash clinical benefits unproven

bull Segmentation specificity is unavoidable ndash each

anatomical structure and pathology has unique

characteristics and nearby structures

bull Organ structure and pathology-specific

algorithms heart liver long bones spine

bull Very laborious top develop and validate a

segmentation algorithm for each ndash trial and error

process many man-months

No universal segmentation method

Observations (1)

Observations (2) bull Applications have different requirements wrt

ndash Accuracy

ndash Robustness

ndash User interaction

ndash Quality

bull Consider the different requirements of

ndash 3D visualization

ndash Training simulation

ndash FEA simulation

Defining application

requirements ahead of time is

essential

Goals Clinical

bull Automatic or nearly automatic lt 5mins user time

bull Can be used by clinician without a technician

bull Produces robust results

Technical

bull Requires significantly less time than developing from

scratch

bull Takes into account intensity and shape

bull Incorporates shape and intensity priors

MIS validation

bull Algorithms without validation are clinically

worthless

bull Validation is with respect to a clinical task

bull Validation requires a ground truth for comparison

bull Physical anatomical models andor phantoms are

typically not available (except sometimes for bones)

bull Ground truth is usually obtained by manual

identification andor segmentation by a user

bull Experts in most cases radiologists

bull Extrinsic comparison compare vs other methods

Copyright L Joskowicz 2010

Copyright L Joskowicz 2010

MIS validation issues

bull Large inter- and intra- variability across experts and clinical sites

bull May not be representative of population variability

bull Main quantitative parameters

ndash validation set size

ndash number and type of observers

ndash intra and inter-observer manual segmentation variability

ndash surface-based and volume-based error measures

MIS validation anatomical variability

stenosis

looping

narrowing

MIS validation anatomical variability

MIS validation metrics

bull Surface-based error measurements

ndash Mean surface distance

ndash RMS surface distance

ndash Maximum surface distance

bull Volume-based error measurements

ndash Dice coefficient

ndash Volumetric overlap error

Copyright L Joskowicz 2010

MIS validation metrics

Surface-based error measurements

bull Mean surface distance

bull RMS surface distance

bull Maximum surface distance

Copyright L Joskowicz 2010

Reference Result

MIS validation metrics

Volume-based error measurements

bull Volumetric overlap error

bull Dice similarity

Copyright L Joskowicz 2010

Reference Result

Aorta

Int + Ext Carotids

All arteries

Subclavian Arteries Bifurcations

16mm (std=08mm)

13mm (std=10mm)

15mm

std=13mm 17mm

std=09mm

07mm

std=07mm

Surface

RMS

Common

Carotid

MIS validation observers variability

bull Intra-observer variability determine how much

variation there is when a single observer

produced the ground-truth segmentation

repeat 5 times the segmentation

bull Inter-observer variability determine how much

variation there is between multiple observers that

produced the ground-truth segmentation

ask 3 radiologists to do the

segmentation

bull Observer expertise and frequency variability

Copyright L Joskowicz 2010

Study [Weltens 2001]

bull Axial MRI slice

bull Nine independent

observers

bull Repeated delineations

bull Interintra observer

variability 30

Key issue fuzzy tumor boundaries

Inter observer variability

Intra-observer variability

8

50

Carotid Lumen Segmentation and Stenosis

Grading Challenge -- The MIDAS Journal

Copyright L Joskowicz 2010

MIS validation manual annotation

Copyright L Joskowicz 2010

MIS validation ground-truth generation

centerlines

(manual)

contours

(manual)

Partial Volume

Segmentation

Reference

standard

from 3PVS

Copyright L Joskowicz 2010

Measurements characteristics

bull Ground-truth not know

bull Repeatability intra-observer variability

bull Reliability inter observer variability

bull Measure correlation between repeated

measurements

bull Intrinsic uncertainty about the tumor volume

bull Accuracy for clinical significance is unknown

Inaccuracy gt Uncertainty

bull Uncertainty may not be improved

bull Goal improve accuracy to obtain

Inaccuracy vs Uncertainty

Results can be

meaningless

Inaccuracy lt Uncertainty

In summaryhellip

bull Medical image segmentation is on the risehellip

bull Basis of many clinical applications

bull Validation is a MUST

bull Many methods and approaches ndash do not re-invent

the wheel

bull I expect patient-specific 3D models to be in the

clinical mainstream in 3-5 years

Copyright L Joskowicz 2011

Segmentation challenges

bull Rigorous quantification and evaluation of

segmentation algorithms performance

bull Shape priors for pathologies

bull Incorporation of functional information from

diffusionperfusion MRI fMRI PET into

segmentation algorithms

Copyright L Joskowicz 2011

Summary (1)

bull Patient-specific anatomy model creation is

currently a major bottleneck in many clinical

applications

bull Automatic anatomical segmentation is essential for

model creation

bull Current tools are limited in scope and coverage

bull Clinical use requires the elimination of the

technician ndash model generation by the physician

Summary (2) bull Great opportunity for the development and

incorporation of anatomy modeling tools in

commercial platforms

bull Growing need and variety of users for anatomical

models

ndash Training simulators surgery rehersal

ndash Intraoperative guidance

ndash Computer Aided Radiology

bull Service providers -- shifting paradigms

Page 16: Recent clinical advances and applications for medical ...crl.med.harvard.edu/research/MICCAI_Tutorial/MICCAI11-tutorial... · Recent clinical advances and applications for medical

copy L Joskowicz 2011 with Dr E Peleg and Profs M Liebergall R Mosheiff

Planning orthopaedics

copy L Joskowicz 2011

Planning orthopaedics

with Dr E Peleg and Profs M Liebergall R Mosheiff

copy L Joskowicz 2011

bull EM real-time tracking

bull US and X-ray imaging

add patient-specific

models from CT

Delivery interventional radiology

copy L Joskowicz 2011

Delivery intraoperative image guidance

augmented continuous X-ray fluoroscopy

with Simbionix and Prof J Sosna

copy L Joskowicz 2011

Delivery intraoperative image guidance

with Simbionix and Prof J Sosna

copy L Joskowicz 2011

Evaluation brain tumor follow-up

Dec 15 2009 June 9 2010

Disease

progression T2-weighted T1-weighted

Tumor internal components

solid enhancing cyst

with TA Sourasky and Dana Hospital

copy L Joskowicz 2011

Key issue model creation

Currently

bull mostly manual delineation

bull slice by slice

Desired

bull automaticnearly automatic ndash a few

clicks by physician

bull no technician

bull accurate and reliable

HARD NO METHOD

SUITABLE FOR ALL STRUCTURES

Modeling requires segmentation

Very difficult task

ndash organpathology specific

ndash imagescanning protocol

ndash anatomical variability

ndash intensity values overlap

ndash structures proximity

Identify and delineate anatomical structure contours

1 2

3 4

Modeling requires segmentation

Very difficult task

ndash organpathology specific

ndash imagescanning protocol

ndash anatomical variability

ndash intensity values overlap

ndash structures proximity

Identify and delineate anatomical structure contours

1 2

3 4

Clinical dataset challenges

proximity calcifications stenosis

Segmentation ndash state of the art

bull Hundreds anatomical segmentation methods for a

variety of structures and imaging modalities

bull Families of techniques thresholding region

growing level sets active contours and more

bull Very few if any in routine clinical use

ndash huge gap between prototype and clinical use

ndash time-consuming fragile limited in scope

ndash require technical knowledge

ndash most have limited validation

ndash clinical benefits unproven

bull Segmentation specificity is unavoidable ndash each

anatomical structure and pathology has unique

characteristics and nearby structures

bull Organ structure and pathology-specific

algorithms heart liver long bones spine

bull Very laborious top develop and validate a

segmentation algorithm for each ndash trial and error

process many man-months

No universal segmentation method

Observations (1)

Observations (2) bull Applications have different requirements wrt

ndash Accuracy

ndash Robustness

ndash User interaction

ndash Quality

bull Consider the different requirements of

ndash 3D visualization

ndash Training simulation

ndash FEA simulation

Defining application

requirements ahead of time is

essential

Goals Clinical

bull Automatic or nearly automatic lt 5mins user time

bull Can be used by clinician without a technician

bull Produces robust results

Technical

bull Requires significantly less time than developing from

scratch

bull Takes into account intensity and shape

bull Incorporates shape and intensity priors

MIS validation

bull Algorithms without validation are clinically

worthless

bull Validation is with respect to a clinical task

bull Validation requires a ground truth for comparison

bull Physical anatomical models andor phantoms are

typically not available (except sometimes for bones)

bull Ground truth is usually obtained by manual

identification andor segmentation by a user

bull Experts in most cases radiologists

bull Extrinsic comparison compare vs other methods

Copyright L Joskowicz 2010

Copyright L Joskowicz 2010

MIS validation issues

bull Large inter- and intra- variability across experts and clinical sites

bull May not be representative of population variability

bull Main quantitative parameters

ndash validation set size

ndash number and type of observers

ndash intra and inter-observer manual segmentation variability

ndash surface-based and volume-based error measures

MIS validation anatomical variability

stenosis

looping

narrowing

MIS validation anatomical variability

MIS validation metrics

bull Surface-based error measurements

ndash Mean surface distance

ndash RMS surface distance

ndash Maximum surface distance

bull Volume-based error measurements

ndash Dice coefficient

ndash Volumetric overlap error

Copyright L Joskowicz 2010

MIS validation metrics

Surface-based error measurements

bull Mean surface distance

bull RMS surface distance

bull Maximum surface distance

Copyright L Joskowicz 2010

Reference Result

MIS validation metrics

Volume-based error measurements

bull Volumetric overlap error

bull Dice similarity

Copyright L Joskowicz 2010

Reference Result

Aorta

Int + Ext Carotids

All arteries

Subclavian Arteries Bifurcations

16mm (std=08mm)

13mm (std=10mm)

15mm

std=13mm 17mm

std=09mm

07mm

std=07mm

Surface

RMS

Common

Carotid

MIS validation observers variability

bull Intra-observer variability determine how much

variation there is when a single observer

produced the ground-truth segmentation

repeat 5 times the segmentation

bull Inter-observer variability determine how much

variation there is between multiple observers that

produced the ground-truth segmentation

ask 3 radiologists to do the

segmentation

bull Observer expertise and frequency variability

Copyright L Joskowicz 2010

Study [Weltens 2001]

bull Axial MRI slice

bull Nine independent

observers

bull Repeated delineations

bull Interintra observer

variability 30

Key issue fuzzy tumor boundaries

Inter observer variability

Intra-observer variability

8

50

Carotid Lumen Segmentation and Stenosis

Grading Challenge -- The MIDAS Journal

Copyright L Joskowicz 2010

MIS validation manual annotation

Copyright L Joskowicz 2010

MIS validation ground-truth generation

centerlines

(manual)

contours

(manual)

Partial Volume

Segmentation

Reference

standard

from 3PVS

Copyright L Joskowicz 2010

Measurements characteristics

bull Ground-truth not know

bull Repeatability intra-observer variability

bull Reliability inter observer variability

bull Measure correlation between repeated

measurements

bull Intrinsic uncertainty about the tumor volume

bull Accuracy for clinical significance is unknown

Inaccuracy gt Uncertainty

bull Uncertainty may not be improved

bull Goal improve accuracy to obtain

Inaccuracy vs Uncertainty

Results can be

meaningless

Inaccuracy lt Uncertainty

In summaryhellip

bull Medical image segmentation is on the risehellip

bull Basis of many clinical applications

bull Validation is a MUST

bull Many methods and approaches ndash do not re-invent

the wheel

bull I expect patient-specific 3D models to be in the

clinical mainstream in 3-5 years

Copyright L Joskowicz 2011

Segmentation challenges

bull Rigorous quantification and evaluation of

segmentation algorithms performance

bull Shape priors for pathologies

bull Incorporation of functional information from

diffusionperfusion MRI fMRI PET into

segmentation algorithms

Copyright L Joskowicz 2011

Summary (1)

bull Patient-specific anatomy model creation is

currently a major bottleneck in many clinical

applications

bull Automatic anatomical segmentation is essential for

model creation

bull Current tools are limited in scope and coverage

bull Clinical use requires the elimination of the

technician ndash model generation by the physician

Summary (2) bull Great opportunity for the development and

incorporation of anatomy modeling tools in

commercial platforms

bull Growing need and variety of users for anatomical

models

ndash Training simulators surgery rehersal

ndash Intraoperative guidance

ndash Computer Aided Radiology

bull Service providers -- shifting paradigms

Page 17: Recent clinical advances and applications for medical ...crl.med.harvard.edu/research/MICCAI_Tutorial/MICCAI11-tutorial... · Recent clinical advances and applications for medical

copy L Joskowicz 2011

Planning orthopaedics

with Dr E Peleg and Profs M Liebergall R Mosheiff

copy L Joskowicz 2011

bull EM real-time tracking

bull US and X-ray imaging

add patient-specific

models from CT

Delivery interventional radiology

copy L Joskowicz 2011

Delivery intraoperative image guidance

augmented continuous X-ray fluoroscopy

with Simbionix and Prof J Sosna

copy L Joskowicz 2011

Delivery intraoperative image guidance

with Simbionix and Prof J Sosna

copy L Joskowicz 2011

Evaluation brain tumor follow-up

Dec 15 2009 June 9 2010

Disease

progression T2-weighted T1-weighted

Tumor internal components

solid enhancing cyst

with TA Sourasky and Dana Hospital

copy L Joskowicz 2011

Key issue model creation

Currently

bull mostly manual delineation

bull slice by slice

Desired

bull automaticnearly automatic ndash a few

clicks by physician

bull no technician

bull accurate and reliable

HARD NO METHOD

SUITABLE FOR ALL STRUCTURES

Modeling requires segmentation

Very difficult task

ndash organpathology specific

ndash imagescanning protocol

ndash anatomical variability

ndash intensity values overlap

ndash structures proximity

Identify and delineate anatomical structure contours

1 2

3 4

Modeling requires segmentation

Very difficult task

ndash organpathology specific

ndash imagescanning protocol

ndash anatomical variability

ndash intensity values overlap

ndash structures proximity

Identify and delineate anatomical structure contours

1 2

3 4

Clinical dataset challenges

proximity calcifications stenosis

Segmentation ndash state of the art

bull Hundreds anatomical segmentation methods for a

variety of structures and imaging modalities

bull Families of techniques thresholding region

growing level sets active contours and more

bull Very few if any in routine clinical use

ndash huge gap between prototype and clinical use

ndash time-consuming fragile limited in scope

ndash require technical knowledge

ndash most have limited validation

ndash clinical benefits unproven

bull Segmentation specificity is unavoidable ndash each

anatomical structure and pathology has unique

characteristics and nearby structures

bull Organ structure and pathology-specific

algorithms heart liver long bones spine

bull Very laborious top develop and validate a

segmentation algorithm for each ndash trial and error

process many man-months

No universal segmentation method

Observations (1)

Observations (2) bull Applications have different requirements wrt

ndash Accuracy

ndash Robustness

ndash User interaction

ndash Quality

bull Consider the different requirements of

ndash 3D visualization

ndash Training simulation

ndash FEA simulation

Defining application

requirements ahead of time is

essential

Goals Clinical

bull Automatic or nearly automatic lt 5mins user time

bull Can be used by clinician without a technician

bull Produces robust results

Technical

bull Requires significantly less time than developing from

scratch

bull Takes into account intensity and shape

bull Incorporates shape and intensity priors

MIS validation

bull Algorithms without validation are clinically

worthless

bull Validation is with respect to a clinical task

bull Validation requires a ground truth for comparison

bull Physical anatomical models andor phantoms are

typically not available (except sometimes for bones)

bull Ground truth is usually obtained by manual

identification andor segmentation by a user

bull Experts in most cases radiologists

bull Extrinsic comparison compare vs other methods

Copyright L Joskowicz 2010

Copyright L Joskowicz 2010

MIS validation issues

bull Large inter- and intra- variability across experts and clinical sites

bull May not be representative of population variability

bull Main quantitative parameters

ndash validation set size

ndash number and type of observers

ndash intra and inter-observer manual segmentation variability

ndash surface-based and volume-based error measures

MIS validation anatomical variability

stenosis

looping

narrowing

MIS validation anatomical variability

MIS validation metrics

bull Surface-based error measurements

ndash Mean surface distance

ndash RMS surface distance

ndash Maximum surface distance

bull Volume-based error measurements

ndash Dice coefficient

ndash Volumetric overlap error

Copyright L Joskowicz 2010

MIS validation metrics

Surface-based error measurements

bull Mean surface distance

bull RMS surface distance

bull Maximum surface distance

Copyright L Joskowicz 2010

Reference Result

MIS validation metrics

Volume-based error measurements

bull Volumetric overlap error

bull Dice similarity

Copyright L Joskowicz 2010

Reference Result

Aorta

Int + Ext Carotids

All arteries

Subclavian Arteries Bifurcations

16mm (std=08mm)

13mm (std=10mm)

15mm

std=13mm 17mm

std=09mm

07mm

std=07mm

Surface

RMS

Common

Carotid

MIS validation observers variability

bull Intra-observer variability determine how much

variation there is when a single observer

produced the ground-truth segmentation

repeat 5 times the segmentation

bull Inter-observer variability determine how much

variation there is between multiple observers that

produced the ground-truth segmentation

ask 3 radiologists to do the

segmentation

bull Observer expertise and frequency variability

Copyright L Joskowicz 2010

Study [Weltens 2001]

bull Axial MRI slice

bull Nine independent

observers

bull Repeated delineations

bull Interintra observer

variability 30

Key issue fuzzy tumor boundaries

Inter observer variability

Intra-observer variability

8

50

Carotid Lumen Segmentation and Stenosis

Grading Challenge -- The MIDAS Journal

Copyright L Joskowicz 2010

MIS validation manual annotation

Copyright L Joskowicz 2010

MIS validation ground-truth generation

centerlines

(manual)

contours

(manual)

Partial Volume

Segmentation

Reference

standard

from 3PVS

Copyright L Joskowicz 2010

Measurements characteristics

bull Ground-truth not know

bull Repeatability intra-observer variability

bull Reliability inter observer variability

bull Measure correlation between repeated

measurements

bull Intrinsic uncertainty about the tumor volume

bull Accuracy for clinical significance is unknown

Inaccuracy gt Uncertainty

bull Uncertainty may not be improved

bull Goal improve accuracy to obtain

Inaccuracy vs Uncertainty

Results can be

meaningless

Inaccuracy lt Uncertainty

In summaryhellip

bull Medical image segmentation is on the risehellip

bull Basis of many clinical applications

bull Validation is a MUST

bull Many methods and approaches ndash do not re-invent

the wheel

bull I expect patient-specific 3D models to be in the

clinical mainstream in 3-5 years

Copyright L Joskowicz 2011

Segmentation challenges

bull Rigorous quantification and evaluation of

segmentation algorithms performance

bull Shape priors for pathologies

bull Incorporation of functional information from

diffusionperfusion MRI fMRI PET into

segmentation algorithms

Copyright L Joskowicz 2011

Summary (1)

bull Patient-specific anatomy model creation is

currently a major bottleneck in many clinical

applications

bull Automatic anatomical segmentation is essential for

model creation

bull Current tools are limited in scope and coverage

bull Clinical use requires the elimination of the

technician ndash model generation by the physician

Summary (2) bull Great opportunity for the development and

incorporation of anatomy modeling tools in

commercial platforms

bull Growing need and variety of users for anatomical

models

ndash Training simulators surgery rehersal

ndash Intraoperative guidance

ndash Computer Aided Radiology

bull Service providers -- shifting paradigms

Page 18: Recent clinical advances and applications for medical ...crl.med.harvard.edu/research/MICCAI_Tutorial/MICCAI11-tutorial... · Recent clinical advances and applications for medical

copy L Joskowicz 2011

bull EM real-time tracking

bull US and X-ray imaging

add patient-specific

models from CT

Delivery interventional radiology

copy L Joskowicz 2011

Delivery intraoperative image guidance

augmented continuous X-ray fluoroscopy

with Simbionix and Prof J Sosna

copy L Joskowicz 2011

Delivery intraoperative image guidance

with Simbionix and Prof J Sosna

copy L Joskowicz 2011

Evaluation brain tumor follow-up

Dec 15 2009 June 9 2010

Disease

progression T2-weighted T1-weighted

Tumor internal components

solid enhancing cyst

with TA Sourasky and Dana Hospital

copy L Joskowicz 2011

Key issue model creation

Currently

bull mostly manual delineation

bull slice by slice

Desired

bull automaticnearly automatic ndash a few

clicks by physician

bull no technician

bull accurate and reliable

HARD NO METHOD

SUITABLE FOR ALL STRUCTURES

Modeling requires segmentation

Very difficult task

ndash organpathology specific

ndash imagescanning protocol

ndash anatomical variability

ndash intensity values overlap

ndash structures proximity

Identify and delineate anatomical structure contours

1 2

3 4

Modeling requires segmentation

Very difficult task

ndash organpathology specific

ndash imagescanning protocol

ndash anatomical variability

ndash intensity values overlap

ndash structures proximity

Identify and delineate anatomical structure contours

1 2

3 4

Clinical dataset challenges

proximity calcifications stenosis

Segmentation ndash state of the art

bull Hundreds anatomical segmentation methods for a

variety of structures and imaging modalities

bull Families of techniques thresholding region

growing level sets active contours and more

bull Very few if any in routine clinical use

ndash huge gap between prototype and clinical use

ndash time-consuming fragile limited in scope

ndash require technical knowledge

ndash most have limited validation

ndash clinical benefits unproven

bull Segmentation specificity is unavoidable ndash each

anatomical structure and pathology has unique

characteristics and nearby structures

bull Organ structure and pathology-specific

algorithms heart liver long bones spine

bull Very laborious top develop and validate a

segmentation algorithm for each ndash trial and error

process many man-months

No universal segmentation method

Observations (1)

Observations (2) bull Applications have different requirements wrt

ndash Accuracy

ndash Robustness

ndash User interaction

ndash Quality

bull Consider the different requirements of

ndash 3D visualization

ndash Training simulation

ndash FEA simulation

Defining application

requirements ahead of time is

essential

Goals Clinical

bull Automatic or nearly automatic lt 5mins user time

bull Can be used by clinician without a technician

bull Produces robust results

Technical

bull Requires significantly less time than developing from

scratch

bull Takes into account intensity and shape

bull Incorporates shape and intensity priors

MIS validation

bull Algorithms without validation are clinically

worthless

bull Validation is with respect to a clinical task

bull Validation requires a ground truth for comparison

bull Physical anatomical models andor phantoms are

typically not available (except sometimes for bones)

bull Ground truth is usually obtained by manual

identification andor segmentation by a user

bull Experts in most cases radiologists

bull Extrinsic comparison compare vs other methods

Copyright L Joskowicz 2010

Copyright L Joskowicz 2010

MIS validation issues

bull Large inter- and intra- variability across experts and clinical sites

bull May not be representative of population variability

bull Main quantitative parameters

ndash validation set size

ndash number and type of observers

ndash intra and inter-observer manual segmentation variability

ndash surface-based and volume-based error measures

MIS validation anatomical variability

stenosis

looping

narrowing

MIS validation anatomical variability

MIS validation metrics

bull Surface-based error measurements

ndash Mean surface distance

ndash RMS surface distance

ndash Maximum surface distance

bull Volume-based error measurements

ndash Dice coefficient

ndash Volumetric overlap error

Copyright L Joskowicz 2010

MIS validation metrics

Surface-based error measurements

bull Mean surface distance

bull RMS surface distance

bull Maximum surface distance

Copyright L Joskowicz 2010

Reference Result

MIS validation metrics

Volume-based error measurements

bull Volumetric overlap error

bull Dice similarity

Copyright L Joskowicz 2010

Reference Result

Aorta

Int + Ext Carotids

All arteries

Subclavian Arteries Bifurcations

16mm (std=08mm)

13mm (std=10mm)

15mm

std=13mm 17mm

std=09mm

07mm

std=07mm

Surface

RMS

Common

Carotid

MIS validation observers variability

bull Intra-observer variability determine how much

variation there is when a single observer

produced the ground-truth segmentation

repeat 5 times the segmentation

bull Inter-observer variability determine how much

variation there is between multiple observers that

produced the ground-truth segmentation

ask 3 radiologists to do the

segmentation

bull Observer expertise and frequency variability

Copyright L Joskowicz 2010

Study [Weltens 2001]

bull Axial MRI slice

bull Nine independent

observers

bull Repeated delineations

bull Interintra observer

variability 30

Key issue fuzzy tumor boundaries

Inter observer variability

Intra-observer variability

8

50

Carotid Lumen Segmentation and Stenosis

Grading Challenge -- The MIDAS Journal

Copyright L Joskowicz 2010

MIS validation manual annotation

Copyright L Joskowicz 2010

MIS validation ground-truth generation

centerlines

(manual)

contours

(manual)

Partial Volume

Segmentation

Reference

standard

from 3PVS

Copyright L Joskowicz 2010

Measurements characteristics

bull Ground-truth not know

bull Repeatability intra-observer variability

bull Reliability inter observer variability

bull Measure correlation between repeated

measurements

bull Intrinsic uncertainty about the tumor volume

bull Accuracy for clinical significance is unknown

Inaccuracy gt Uncertainty

bull Uncertainty may not be improved

bull Goal improve accuracy to obtain

Inaccuracy vs Uncertainty

Results can be

meaningless

Inaccuracy lt Uncertainty

In summaryhellip

bull Medical image segmentation is on the risehellip

bull Basis of many clinical applications

bull Validation is a MUST

bull Many methods and approaches ndash do not re-invent

the wheel

bull I expect patient-specific 3D models to be in the

clinical mainstream in 3-5 years

Copyright L Joskowicz 2011

Segmentation challenges

bull Rigorous quantification and evaluation of

segmentation algorithms performance

bull Shape priors for pathologies

bull Incorporation of functional information from

diffusionperfusion MRI fMRI PET into

segmentation algorithms

Copyright L Joskowicz 2011

Summary (1)

bull Patient-specific anatomy model creation is

currently a major bottleneck in many clinical

applications

bull Automatic anatomical segmentation is essential for

model creation

bull Current tools are limited in scope and coverage

bull Clinical use requires the elimination of the

technician ndash model generation by the physician

Summary (2) bull Great opportunity for the development and

incorporation of anatomy modeling tools in

commercial platforms

bull Growing need and variety of users for anatomical

models

ndash Training simulators surgery rehersal

ndash Intraoperative guidance

ndash Computer Aided Radiology

bull Service providers -- shifting paradigms

Page 19: Recent clinical advances and applications for medical ...crl.med.harvard.edu/research/MICCAI_Tutorial/MICCAI11-tutorial... · Recent clinical advances and applications for medical

copy L Joskowicz 2011

Delivery intraoperative image guidance

augmented continuous X-ray fluoroscopy

with Simbionix and Prof J Sosna

copy L Joskowicz 2011

Delivery intraoperative image guidance

with Simbionix and Prof J Sosna

copy L Joskowicz 2011

Evaluation brain tumor follow-up

Dec 15 2009 June 9 2010

Disease

progression T2-weighted T1-weighted

Tumor internal components

solid enhancing cyst

with TA Sourasky and Dana Hospital

copy L Joskowicz 2011

Key issue model creation

Currently

bull mostly manual delineation

bull slice by slice

Desired

bull automaticnearly automatic ndash a few

clicks by physician

bull no technician

bull accurate and reliable

HARD NO METHOD

SUITABLE FOR ALL STRUCTURES

Modeling requires segmentation

Very difficult task

ndash organpathology specific

ndash imagescanning protocol

ndash anatomical variability

ndash intensity values overlap

ndash structures proximity

Identify and delineate anatomical structure contours

1 2

3 4

Modeling requires segmentation

Very difficult task

ndash organpathology specific

ndash imagescanning protocol

ndash anatomical variability

ndash intensity values overlap

ndash structures proximity

Identify and delineate anatomical structure contours

1 2

3 4

Clinical dataset challenges

proximity calcifications stenosis

Segmentation ndash state of the art

bull Hundreds anatomical segmentation methods for a

variety of structures and imaging modalities

bull Families of techniques thresholding region

growing level sets active contours and more

bull Very few if any in routine clinical use

ndash huge gap between prototype and clinical use

ndash time-consuming fragile limited in scope

ndash require technical knowledge

ndash most have limited validation

ndash clinical benefits unproven

bull Segmentation specificity is unavoidable ndash each

anatomical structure and pathology has unique

characteristics and nearby structures

bull Organ structure and pathology-specific

algorithms heart liver long bones spine

bull Very laborious top develop and validate a

segmentation algorithm for each ndash trial and error

process many man-months

No universal segmentation method

Observations (1)

Observations (2) bull Applications have different requirements wrt

ndash Accuracy

ndash Robustness

ndash User interaction

ndash Quality

bull Consider the different requirements of

ndash 3D visualization

ndash Training simulation

ndash FEA simulation

Defining application

requirements ahead of time is

essential

Goals Clinical

bull Automatic or nearly automatic lt 5mins user time

bull Can be used by clinician without a technician

bull Produces robust results

Technical

bull Requires significantly less time than developing from

scratch

bull Takes into account intensity and shape

bull Incorporates shape and intensity priors

MIS validation

bull Algorithms without validation are clinically

worthless

bull Validation is with respect to a clinical task

bull Validation requires a ground truth for comparison

bull Physical anatomical models andor phantoms are

typically not available (except sometimes for bones)

bull Ground truth is usually obtained by manual

identification andor segmentation by a user

bull Experts in most cases radiologists

bull Extrinsic comparison compare vs other methods

Copyright L Joskowicz 2010

Copyright L Joskowicz 2010

MIS validation issues

bull Large inter- and intra- variability across experts and clinical sites

bull May not be representative of population variability

bull Main quantitative parameters

ndash validation set size

ndash number and type of observers

ndash intra and inter-observer manual segmentation variability

ndash surface-based and volume-based error measures

MIS validation anatomical variability

stenosis

looping

narrowing

MIS validation anatomical variability

MIS validation metrics

bull Surface-based error measurements

ndash Mean surface distance

ndash RMS surface distance

ndash Maximum surface distance

bull Volume-based error measurements

ndash Dice coefficient

ndash Volumetric overlap error

Copyright L Joskowicz 2010

MIS validation metrics

Surface-based error measurements

bull Mean surface distance

bull RMS surface distance

bull Maximum surface distance

Copyright L Joskowicz 2010

Reference Result

MIS validation metrics

Volume-based error measurements

bull Volumetric overlap error

bull Dice similarity

Copyright L Joskowicz 2010

Reference Result

Aorta

Int + Ext Carotids

All arteries

Subclavian Arteries Bifurcations

16mm (std=08mm)

13mm (std=10mm)

15mm

std=13mm 17mm

std=09mm

07mm

std=07mm

Surface

RMS

Common

Carotid

MIS validation observers variability

bull Intra-observer variability determine how much

variation there is when a single observer

produced the ground-truth segmentation

repeat 5 times the segmentation

bull Inter-observer variability determine how much

variation there is between multiple observers that

produced the ground-truth segmentation

ask 3 radiologists to do the

segmentation

bull Observer expertise and frequency variability

Copyright L Joskowicz 2010

Study [Weltens 2001]

bull Axial MRI slice

bull Nine independent

observers

bull Repeated delineations

bull Interintra observer

variability 30

Key issue fuzzy tumor boundaries

Inter observer variability

Intra-observer variability

8

50

Carotid Lumen Segmentation and Stenosis

Grading Challenge -- The MIDAS Journal

Copyright L Joskowicz 2010

MIS validation manual annotation

Copyright L Joskowicz 2010

MIS validation ground-truth generation

centerlines

(manual)

contours

(manual)

Partial Volume

Segmentation

Reference

standard

from 3PVS

Copyright L Joskowicz 2010

Measurements characteristics

bull Ground-truth not know

bull Repeatability intra-observer variability

bull Reliability inter observer variability

bull Measure correlation between repeated

measurements

bull Intrinsic uncertainty about the tumor volume

bull Accuracy for clinical significance is unknown

Inaccuracy gt Uncertainty

bull Uncertainty may not be improved

bull Goal improve accuracy to obtain

Inaccuracy vs Uncertainty

Results can be

meaningless

Inaccuracy lt Uncertainty

In summaryhellip

bull Medical image segmentation is on the risehellip

bull Basis of many clinical applications

bull Validation is a MUST

bull Many methods and approaches ndash do not re-invent

the wheel

bull I expect patient-specific 3D models to be in the

clinical mainstream in 3-5 years

Copyright L Joskowicz 2011

Segmentation challenges

bull Rigorous quantification and evaluation of

segmentation algorithms performance

bull Shape priors for pathologies

bull Incorporation of functional information from

diffusionperfusion MRI fMRI PET into

segmentation algorithms

Copyright L Joskowicz 2011

Summary (1)

bull Patient-specific anatomy model creation is

currently a major bottleneck in many clinical

applications

bull Automatic anatomical segmentation is essential for

model creation

bull Current tools are limited in scope and coverage

bull Clinical use requires the elimination of the

technician ndash model generation by the physician

Summary (2) bull Great opportunity for the development and

incorporation of anatomy modeling tools in

commercial platforms

bull Growing need and variety of users for anatomical

models

ndash Training simulators surgery rehersal

ndash Intraoperative guidance

ndash Computer Aided Radiology

bull Service providers -- shifting paradigms

Page 20: Recent clinical advances and applications for medical ...crl.med.harvard.edu/research/MICCAI_Tutorial/MICCAI11-tutorial... · Recent clinical advances and applications for medical

copy L Joskowicz 2011

Delivery intraoperative image guidance

with Simbionix and Prof J Sosna

copy L Joskowicz 2011

Evaluation brain tumor follow-up

Dec 15 2009 June 9 2010

Disease

progression T2-weighted T1-weighted

Tumor internal components

solid enhancing cyst

with TA Sourasky and Dana Hospital

copy L Joskowicz 2011

Key issue model creation

Currently

bull mostly manual delineation

bull slice by slice

Desired

bull automaticnearly automatic ndash a few

clicks by physician

bull no technician

bull accurate and reliable

HARD NO METHOD

SUITABLE FOR ALL STRUCTURES

Modeling requires segmentation

Very difficult task

ndash organpathology specific

ndash imagescanning protocol

ndash anatomical variability

ndash intensity values overlap

ndash structures proximity

Identify and delineate anatomical structure contours

1 2

3 4

Modeling requires segmentation

Very difficult task

ndash organpathology specific

ndash imagescanning protocol

ndash anatomical variability

ndash intensity values overlap

ndash structures proximity

Identify and delineate anatomical structure contours

1 2

3 4

Clinical dataset challenges

proximity calcifications stenosis

Segmentation ndash state of the art

bull Hundreds anatomical segmentation methods for a

variety of structures and imaging modalities

bull Families of techniques thresholding region

growing level sets active contours and more

bull Very few if any in routine clinical use

ndash huge gap between prototype and clinical use

ndash time-consuming fragile limited in scope

ndash require technical knowledge

ndash most have limited validation

ndash clinical benefits unproven

bull Segmentation specificity is unavoidable ndash each

anatomical structure and pathology has unique

characteristics and nearby structures

bull Organ structure and pathology-specific

algorithms heart liver long bones spine

bull Very laborious top develop and validate a

segmentation algorithm for each ndash trial and error

process many man-months

No universal segmentation method

Observations (1)

Observations (2) bull Applications have different requirements wrt

ndash Accuracy

ndash Robustness

ndash User interaction

ndash Quality

bull Consider the different requirements of

ndash 3D visualization

ndash Training simulation

ndash FEA simulation

Defining application

requirements ahead of time is

essential

Goals Clinical

bull Automatic or nearly automatic lt 5mins user time

bull Can be used by clinician without a technician

bull Produces robust results

Technical

bull Requires significantly less time than developing from

scratch

bull Takes into account intensity and shape

bull Incorporates shape and intensity priors

MIS validation

bull Algorithms without validation are clinically

worthless

bull Validation is with respect to a clinical task

bull Validation requires a ground truth for comparison

bull Physical anatomical models andor phantoms are

typically not available (except sometimes for bones)

bull Ground truth is usually obtained by manual

identification andor segmentation by a user

bull Experts in most cases radiologists

bull Extrinsic comparison compare vs other methods

Copyright L Joskowicz 2010

Copyright L Joskowicz 2010

MIS validation issues

bull Large inter- and intra- variability across experts and clinical sites

bull May not be representative of population variability

bull Main quantitative parameters

ndash validation set size

ndash number and type of observers

ndash intra and inter-observer manual segmentation variability

ndash surface-based and volume-based error measures

MIS validation anatomical variability

stenosis

looping

narrowing

MIS validation anatomical variability

MIS validation metrics

bull Surface-based error measurements

ndash Mean surface distance

ndash RMS surface distance

ndash Maximum surface distance

bull Volume-based error measurements

ndash Dice coefficient

ndash Volumetric overlap error

Copyright L Joskowicz 2010

MIS validation metrics

Surface-based error measurements

bull Mean surface distance

bull RMS surface distance

bull Maximum surface distance

Copyright L Joskowicz 2010

Reference Result

MIS validation metrics

Volume-based error measurements

bull Volumetric overlap error

bull Dice similarity

Copyright L Joskowicz 2010

Reference Result

Aorta

Int + Ext Carotids

All arteries

Subclavian Arteries Bifurcations

16mm (std=08mm)

13mm (std=10mm)

15mm

std=13mm 17mm

std=09mm

07mm

std=07mm

Surface

RMS

Common

Carotid

MIS validation observers variability

bull Intra-observer variability determine how much

variation there is when a single observer

produced the ground-truth segmentation

repeat 5 times the segmentation

bull Inter-observer variability determine how much

variation there is between multiple observers that

produced the ground-truth segmentation

ask 3 radiologists to do the

segmentation

bull Observer expertise and frequency variability

Copyright L Joskowicz 2010

Study [Weltens 2001]

bull Axial MRI slice

bull Nine independent

observers

bull Repeated delineations

bull Interintra observer

variability 30

Key issue fuzzy tumor boundaries

Inter observer variability

Intra-observer variability

8

50

Carotid Lumen Segmentation and Stenosis

Grading Challenge -- The MIDAS Journal

Copyright L Joskowicz 2010

MIS validation manual annotation

Copyright L Joskowicz 2010

MIS validation ground-truth generation

centerlines

(manual)

contours

(manual)

Partial Volume

Segmentation

Reference

standard

from 3PVS

Copyright L Joskowicz 2010

Measurements characteristics

bull Ground-truth not know

bull Repeatability intra-observer variability

bull Reliability inter observer variability

bull Measure correlation between repeated

measurements

bull Intrinsic uncertainty about the tumor volume

bull Accuracy for clinical significance is unknown

Inaccuracy gt Uncertainty

bull Uncertainty may not be improved

bull Goal improve accuracy to obtain

Inaccuracy vs Uncertainty

Results can be

meaningless

Inaccuracy lt Uncertainty

In summaryhellip

bull Medical image segmentation is on the risehellip

bull Basis of many clinical applications

bull Validation is a MUST

bull Many methods and approaches ndash do not re-invent

the wheel

bull I expect patient-specific 3D models to be in the

clinical mainstream in 3-5 years

Copyright L Joskowicz 2011

Segmentation challenges

bull Rigorous quantification and evaluation of

segmentation algorithms performance

bull Shape priors for pathologies

bull Incorporation of functional information from

diffusionperfusion MRI fMRI PET into

segmentation algorithms

Copyright L Joskowicz 2011

Summary (1)

bull Patient-specific anatomy model creation is

currently a major bottleneck in many clinical

applications

bull Automatic anatomical segmentation is essential for

model creation

bull Current tools are limited in scope and coverage

bull Clinical use requires the elimination of the

technician ndash model generation by the physician

Summary (2) bull Great opportunity for the development and

incorporation of anatomy modeling tools in

commercial platforms

bull Growing need and variety of users for anatomical

models

ndash Training simulators surgery rehersal

ndash Intraoperative guidance

ndash Computer Aided Radiology

bull Service providers -- shifting paradigms

Page 21: Recent clinical advances and applications for medical ...crl.med.harvard.edu/research/MICCAI_Tutorial/MICCAI11-tutorial... · Recent clinical advances and applications for medical

copy L Joskowicz 2011

Evaluation brain tumor follow-up

Dec 15 2009 June 9 2010

Disease

progression T2-weighted T1-weighted

Tumor internal components

solid enhancing cyst

with TA Sourasky and Dana Hospital

copy L Joskowicz 2011

Key issue model creation

Currently

bull mostly manual delineation

bull slice by slice

Desired

bull automaticnearly automatic ndash a few

clicks by physician

bull no technician

bull accurate and reliable

HARD NO METHOD

SUITABLE FOR ALL STRUCTURES

Modeling requires segmentation

Very difficult task

ndash organpathology specific

ndash imagescanning protocol

ndash anatomical variability

ndash intensity values overlap

ndash structures proximity

Identify and delineate anatomical structure contours

1 2

3 4

Modeling requires segmentation

Very difficult task

ndash organpathology specific

ndash imagescanning protocol

ndash anatomical variability

ndash intensity values overlap

ndash structures proximity

Identify and delineate anatomical structure contours

1 2

3 4

Clinical dataset challenges

proximity calcifications stenosis

Segmentation ndash state of the art

bull Hundreds anatomical segmentation methods for a

variety of structures and imaging modalities

bull Families of techniques thresholding region

growing level sets active contours and more

bull Very few if any in routine clinical use

ndash huge gap between prototype and clinical use

ndash time-consuming fragile limited in scope

ndash require technical knowledge

ndash most have limited validation

ndash clinical benefits unproven

bull Segmentation specificity is unavoidable ndash each

anatomical structure and pathology has unique

characteristics and nearby structures

bull Organ structure and pathology-specific

algorithms heart liver long bones spine

bull Very laborious top develop and validate a

segmentation algorithm for each ndash trial and error

process many man-months

No universal segmentation method

Observations (1)

Observations (2) bull Applications have different requirements wrt

ndash Accuracy

ndash Robustness

ndash User interaction

ndash Quality

bull Consider the different requirements of

ndash 3D visualization

ndash Training simulation

ndash FEA simulation

Defining application

requirements ahead of time is

essential

Goals Clinical

bull Automatic or nearly automatic lt 5mins user time

bull Can be used by clinician without a technician

bull Produces robust results

Technical

bull Requires significantly less time than developing from

scratch

bull Takes into account intensity and shape

bull Incorporates shape and intensity priors

MIS validation

bull Algorithms without validation are clinically

worthless

bull Validation is with respect to a clinical task

bull Validation requires a ground truth for comparison

bull Physical anatomical models andor phantoms are

typically not available (except sometimes for bones)

bull Ground truth is usually obtained by manual

identification andor segmentation by a user

bull Experts in most cases radiologists

bull Extrinsic comparison compare vs other methods

Copyright L Joskowicz 2010

Copyright L Joskowicz 2010

MIS validation issues

bull Large inter- and intra- variability across experts and clinical sites

bull May not be representative of population variability

bull Main quantitative parameters

ndash validation set size

ndash number and type of observers

ndash intra and inter-observer manual segmentation variability

ndash surface-based and volume-based error measures

MIS validation anatomical variability

stenosis

looping

narrowing

MIS validation anatomical variability

MIS validation metrics

bull Surface-based error measurements

ndash Mean surface distance

ndash RMS surface distance

ndash Maximum surface distance

bull Volume-based error measurements

ndash Dice coefficient

ndash Volumetric overlap error

Copyright L Joskowicz 2010

MIS validation metrics

Surface-based error measurements

bull Mean surface distance

bull RMS surface distance

bull Maximum surface distance

Copyright L Joskowicz 2010

Reference Result

MIS validation metrics

Volume-based error measurements

bull Volumetric overlap error

bull Dice similarity

Copyright L Joskowicz 2010

Reference Result

Aorta

Int + Ext Carotids

All arteries

Subclavian Arteries Bifurcations

16mm (std=08mm)

13mm (std=10mm)

15mm

std=13mm 17mm

std=09mm

07mm

std=07mm

Surface

RMS

Common

Carotid

MIS validation observers variability

bull Intra-observer variability determine how much

variation there is when a single observer

produced the ground-truth segmentation

repeat 5 times the segmentation

bull Inter-observer variability determine how much

variation there is between multiple observers that

produced the ground-truth segmentation

ask 3 radiologists to do the

segmentation

bull Observer expertise and frequency variability

Copyright L Joskowicz 2010

Study [Weltens 2001]

bull Axial MRI slice

bull Nine independent

observers

bull Repeated delineations

bull Interintra observer

variability 30

Key issue fuzzy tumor boundaries

Inter observer variability

Intra-observer variability

8

50

Carotid Lumen Segmentation and Stenosis

Grading Challenge -- The MIDAS Journal

Copyright L Joskowicz 2010

MIS validation manual annotation

Copyright L Joskowicz 2010

MIS validation ground-truth generation

centerlines

(manual)

contours

(manual)

Partial Volume

Segmentation

Reference

standard

from 3PVS

Copyright L Joskowicz 2010

Measurements characteristics

bull Ground-truth not know

bull Repeatability intra-observer variability

bull Reliability inter observer variability

bull Measure correlation between repeated

measurements

bull Intrinsic uncertainty about the tumor volume

bull Accuracy for clinical significance is unknown

Inaccuracy gt Uncertainty

bull Uncertainty may not be improved

bull Goal improve accuracy to obtain

Inaccuracy vs Uncertainty

Results can be

meaningless

Inaccuracy lt Uncertainty

In summaryhellip

bull Medical image segmentation is on the risehellip

bull Basis of many clinical applications

bull Validation is a MUST

bull Many methods and approaches ndash do not re-invent

the wheel

bull I expect patient-specific 3D models to be in the

clinical mainstream in 3-5 years

Copyright L Joskowicz 2011

Segmentation challenges

bull Rigorous quantification and evaluation of

segmentation algorithms performance

bull Shape priors for pathologies

bull Incorporation of functional information from

diffusionperfusion MRI fMRI PET into

segmentation algorithms

Copyright L Joskowicz 2011

Summary (1)

bull Patient-specific anatomy model creation is

currently a major bottleneck in many clinical

applications

bull Automatic anatomical segmentation is essential for

model creation

bull Current tools are limited in scope and coverage

bull Clinical use requires the elimination of the

technician ndash model generation by the physician

Summary (2) bull Great opportunity for the development and

incorporation of anatomy modeling tools in

commercial platforms

bull Growing need and variety of users for anatomical

models

ndash Training simulators surgery rehersal

ndash Intraoperative guidance

ndash Computer Aided Radiology

bull Service providers -- shifting paradigms

Page 22: Recent clinical advances and applications for medical ...crl.med.harvard.edu/research/MICCAI_Tutorial/MICCAI11-tutorial... · Recent clinical advances and applications for medical

copy L Joskowicz 2011

Key issue model creation

Currently

bull mostly manual delineation

bull slice by slice

Desired

bull automaticnearly automatic ndash a few

clicks by physician

bull no technician

bull accurate and reliable

HARD NO METHOD

SUITABLE FOR ALL STRUCTURES

Modeling requires segmentation

Very difficult task

ndash organpathology specific

ndash imagescanning protocol

ndash anatomical variability

ndash intensity values overlap

ndash structures proximity

Identify and delineate anatomical structure contours

1 2

3 4

Modeling requires segmentation

Very difficult task

ndash organpathology specific

ndash imagescanning protocol

ndash anatomical variability

ndash intensity values overlap

ndash structures proximity

Identify and delineate anatomical structure contours

1 2

3 4

Clinical dataset challenges

proximity calcifications stenosis

Segmentation ndash state of the art

bull Hundreds anatomical segmentation methods for a

variety of structures and imaging modalities

bull Families of techniques thresholding region

growing level sets active contours and more

bull Very few if any in routine clinical use

ndash huge gap between prototype and clinical use

ndash time-consuming fragile limited in scope

ndash require technical knowledge

ndash most have limited validation

ndash clinical benefits unproven

bull Segmentation specificity is unavoidable ndash each

anatomical structure and pathology has unique

characteristics and nearby structures

bull Organ structure and pathology-specific

algorithms heart liver long bones spine

bull Very laborious top develop and validate a

segmentation algorithm for each ndash trial and error

process many man-months

No universal segmentation method

Observations (1)

Observations (2) bull Applications have different requirements wrt

ndash Accuracy

ndash Robustness

ndash User interaction

ndash Quality

bull Consider the different requirements of

ndash 3D visualization

ndash Training simulation

ndash FEA simulation

Defining application

requirements ahead of time is

essential

Goals Clinical

bull Automatic or nearly automatic lt 5mins user time

bull Can be used by clinician without a technician

bull Produces robust results

Technical

bull Requires significantly less time than developing from

scratch

bull Takes into account intensity and shape

bull Incorporates shape and intensity priors

MIS validation

bull Algorithms without validation are clinically

worthless

bull Validation is with respect to a clinical task

bull Validation requires a ground truth for comparison

bull Physical anatomical models andor phantoms are

typically not available (except sometimes for bones)

bull Ground truth is usually obtained by manual

identification andor segmentation by a user

bull Experts in most cases radiologists

bull Extrinsic comparison compare vs other methods

Copyright L Joskowicz 2010

Copyright L Joskowicz 2010

MIS validation issues

bull Large inter- and intra- variability across experts and clinical sites

bull May not be representative of population variability

bull Main quantitative parameters

ndash validation set size

ndash number and type of observers

ndash intra and inter-observer manual segmentation variability

ndash surface-based and volume-based error measures

MIS validation anatomical variability

stenosis

looping

narrowing

MIS validation anatomical variability

MIS validation metrics

bull Surface-based error measurements

ndash Mean surface distance

ndash RMS surface distance

ndash Maximum surface distance

bull Volume-based error measurements

ndash Dice coefficient

ndash Volumetric overlap error

Copyright L Joskowicz 2010

MIS validation metrics

Surface-based error measurements

bull Mean surface distance

bull RMS surface distance

bull Maximum surface distance

Copyright L Joskowicz 2010

Reference Result

MIS validation metrics

Volume-based error measurements

bull Volumetric overlap error

bull Dice similarity

Copyright L Joskowicz 2010

Reference Result

Aorta

Int + Ext Carotids

All arteries

Subclavian Arteries Bifurcations

16mm (std=08mm)

13mm (std=10mm)

15mm

std=13mm 17mm

std=09mm

07mm

std=07mm

Surface

RMS

Common

Carotid

MIS validation observers variability

bull Intra-observer variability determine how much

variation there is when a single observer

produced the ground-truth segmentation

repeat 5 times the segmentation

bull Inter-observer variability determine how much

variation there is between multiple observers that

produced the ground-truth segmentation

ask 3 radiologists to do the

segmentation

bull Observer expertise and frequency variability

Copyright L Joskowicz 2010

Study [Weltens 2001]

bull Axial MRI slice

bull Nine independent

observers

bull Repeated delineations

bull Interintra observer

variability 30

Key issue fuzzy tumor boundaries

Inter observer variability

Intra-observer variability

8

50

Carotid Lumen Segmentation and Stenosis

Grading Challenge -- The MIDAS Journal

Copyright L Joskowicz 2010

MIS validation manual annotation

Copyright L Joskowicz 2010

MIS validation ground-truth generation

centerlines

(manual)

contours

(manual)

Partial Volume

Segmentation

Reference

standard

from 3PVS

Copyright L Joskowicz 2010

Measurements characteristics

bull Ground-truth not know

bull Repeatability intra-observer variability

bull Reliability inter observer variability

bull Measure correlation between repeated

measurements

bull Intrinsic uncertainty about the tumor volume

bull Accuracy for clinical significance is unknown

Inaccuracy gt Uncertainty

bull Uncertainty may not be improved

bull Goal improve accuracy to obtain

Inaccuracy vs Uncertainty

Results can be

meaningless

Inaccuracy lt Uncertainty

In summaryhellip

bull Medical image segmentation is on the risehellip

bull Basis of many clinical applications

bull Validation is a MUST

bull Many methods and approaches ndash do not re-invent

the wheel

bull I expect patient-specific 3D models to be in the

clinical mainstream in 3-5 years

Copyright L Joskowicz 2011

Segmentation challenges

bull Rigorous quantification and evaluation of

segmentation algorithms performance

bull Shape priors for pathologies

bull Incorporation of functional information from

diffusionperfusion MRI fMRI PET into

segmentation algorithms

Copyright L Joskowicz 2011

Summary (1)

bull Patient-specific anatomy model creation is

currently a major bottleneck in many clinical

applications

bull Automatic anatomical segmentation is essential for

model creation

bull Current tools are limited in scope and coverage

bull Clinical use requires the elimination of the

technician ndash model generation by the physician

Summary (2) bull Great opportunity for the development and

incorporation of anatomy modeling tools in

commercial platforms

bull Growing need and variety of users for anatomical

models

ndash Training simulators surgery rehersal

ndash Intraoperative guidance

ndash Computer Aided Radiology

bull Service providers -- shifting paradigms

Page 23: Recent clinical advances and applications for medical ...crl.med.harvard.edu/research/MICCAI_Tutorial/MICCAI11-tutorial... · Recent clinical advances and applications for medical

Modeling requires segmentation

Very difficult task

ndash organpathology specific

ndash imagescanning protocol

ndash anatomical variability

ndash intensity values overlap

ndash structures proximity

Identify and delineate anatomical structure contours

1 2

3 4

Modeling requires segmentation

Very difficult task

ndash organpathology specific

ndash imagescanning protocol

ndash anatomical variability

ndash intensity values overlap

ndash structures proximity

Identify and delineate anatomical structure contours

1 2

3 4

Clinical dataset challenges

proximity calcifications stenosis

Segmentation ndash state of the art

bull Hundreds anatomical segmentation methods for a

variety of structures and imaging modalities

bull Families of techniques thresholding region

growing level sets active contours and more

bull Very few if any in routine clinical use

ndash huge gap between prototype and clinical use

ndash time-consuming fragile limited in scope

ndash require technical knowledge

ndash most have limited validation

ndash clinical benefits unproven

bull Segmentation specificity is unavoidable ndash each

anatomical structure and pathology has unique

characteristics and nearby structures

bull Organ structure and pathology-specific

algorithms heart liver long bones spine

bull Very laborious top develop and validate a

segmentation algorithm for each ndash trial and error

process many man-months

No universal segmentation method

Observations (1)

Observations (2) bull Applications have different requirements wrt

ndash Accuracy

ndash Robustness

ndash User interaction

ndash Quality

bull Consider the different requirements of

ndash 3D visualization

ndash Training simulation

ndash FEA simulation

Defining application

requirements ahead of time is

essential

Goals Clinical

bull Automatic or nearly automatic lt 5mins user time

bull Can be used by clinician without a technician

bull Produces robust results

Technical

bull Requires significantly less time than developing from

scratch

bull Takes into account intensity and shape

bull Incorporates shape and intensity priors

MIS validation

bull Algorithms without validation are clinically

worthless

bull Validation is with respect to a clinical task

bull Validation requires a ground truth for comparison

bull Physical anatomical models andor phantoms are

typically not available (except sometimes for bones)

bull Ground truth is usually obtained by manual

identification andor segmentation by a user

bull Experts in most cases radiologists

bull Extrinsic comparison compare vs other methods

Copyright L Joskowicz 2010

Copyright L Joskowicz 2010

MIS validation issues

bull Large inter- and intra- variability across experts and clinical sites

bull May not be representative of population variability

bull Main quantitative parameters

ndash validation set size

ndash number and type of observers

ndash intra and inter-observer manual segmentation variability

ndash surface-based and volume-based error measures

MIS validation anatomical variability

stenosis

looping

narrowing

MIS validation anatomical variability

MIS validation metrics

bull Surface-based error measurements

ndash Mean surface distance

ndash RMS surface distance

ndash Maximum surface distance

bull Volume-based error measurements

ndash Dice coefficient

ndash Volumetric overlap error

Copyright L Joskowicz 2010

MIS validation metrics

Surface-based error measurements

bull Mean surface distance

bull RMS surface distance

bull Maximum surface distance

Copyright L Joskowicz 2010

Reference Result

MIS validation metrics

Volume-based error measurements

bull Volumetric overlap error

bull Dice similarity

Copyright L Joskowicz 2010

Reference Result

Aorta

Int + Ext Carotids

All arteries

Subclavian Arteries Bifurcations

16mm (std=08mm)

13mm (std=10mm)

15mm

std=13mm 17mm

std=09mm

07mm

std=07mm

Surface

RMS

Common

Carotid

MIS validation observers variability

bull Intra-observer variability determine how much

variation there is when a single observer

produced the ground-truth segmentation

repeat 5 times the segmentation

bull Inter-observer variability determine how much

variation there is between multiple observers that

produced the ground-truth segmentation

ask 3 radiologists to do the

segmentation

bull Observer expertise and frequency variability

Copyright L Joskowicz 2010

Study [Weltens 2001]

bull Axial MRI slice

bull Nine independent

observers

bull Repeated delineations

bull Interintra observer

variability 30

Key issue fuzzy tumor boundaries

Inter observer variability

Intra-observer variability

8

50

Carotid Lumen Segmentation and Stenosis

Grading Challenge -- The MIDAS Journal

Copyright L Joskowicz 2010

MIS validation manual annotation

Copyright L Joskowicz 2010

MIS validation ground-truth generation

centerlines

(manual)

contours

(manual)

Partial Volume

Segmentation

Reference

standard

from 3PVS

Copyright L Joskowicz 2010

Measurements characteristics

bull Ground-truth not know

bull Repeatability intra-observer variability

bull Reliability inter observer variability

bull Measure correlation between repeated

measurements

bull Intrinsic uncertainty about the tumor volume

bull Accuracy for clinical significance is unknown

Inaccuracy gt Uncertainty

bull Uncertainty may not be improved

bull Goal improve accuracy to obtain

Inaccuracy vs Uncertainty

Results can be

meaningless

Inaccuracy lt Uncertainty

In summaryhellip

bull Medical image segmentation is on the risehellip

bull Basis of many clinical applications

bull Validation is a MUST

bull Many methods and approaches ndash do not re-invent

the wheel

bull I expect patient-specific 3D models to be in the

clinical mainstream in 3-5 years

Copyright L Joskowicz 2011

Segmentation challenges

bull Rigorous quantification and evaluation of

segmentation algorithms performance

bull Shape priors for pathologies

bull Incorporation of functional information from

diffusionperfusion MRI fMRI PET into

segmentation algorithms

Copyright L Joskowicz 2011

Summary (1)

bull Patient-specific anatomy model creation is

currently a major bottleneck in many clinical

applications

bull Automatic anatomical segmentation is essential for

model creation

bull Current tools are limited in scope and coverage

bull Clinical use requires the elimination of the

technician ndash model generation by the physician

Summary (2) bull Great opportunity for the development and

incorporation of anatomy modeling tools in

commercial platforms

bull Growing need and variety of users for anatomical

models

ndash Training simulators surgery rehersal

ndash Intraoperative guidance

ndash Computer Aided Radiology

bull Service providers -- shifting paradigms

Page 24: Recent clinical advances and applications for medical ...crl.med.harvard.edu/research/MICCAI_Tutorial/MICCAI11-tutorial... · Recent clinical advances and applications for medical

Modeling requires segmentation

Very difficult task

ndash organpathology specific

ndash imagescanning protocol

ndash anatomical variability

ndash intensity values overlap

ndash structures proximity

Identify and delineate anatomical structure contours

1 2

3 4

Clinical dataset challenges

proximity calcifications stenosis

Segmentation ndash state of the art

bull Hundreds anatomical segmentation methods for a

variety of structures and imaging modalities

bull Families of techniques thresholding region

growing level sets active contours and more

bull Very few if any in routine clinical use

ndash huge gap between prototype and clinical use

ndash time-consuming fragile limited in scope

ndash require technical knowledge

ndash most have limited validation

ndash clinical benefits unproven

bull Segmentation specificity is unavoidable ndash each

anatomical structure and pathology has unique

characteristics and nearby structures

bull Organ structure and pathology-specific

algorithms heart liver long bones spine

bull Very laborious top develop and validate a

segmentation algorithm for each ndash trial and error

process many man-months

No universal segmentation method

Observations (1)

Observations (2) bull Applications have different requirements wrt

ndash Accuracy

ndash Robustness

ndash User interaction

ndash Quality

bull Consider the different requirements of

ndash 3D visualization

ndash Training simulation

ndash FEA simulation

Defining application

requirements ahead of time is

essential

Goals Clinical

bull Automatic or nearly automatic lt 5mins user time

bull Can be used by clinician without a technician

bull Produces robust results

Technical

bull Requires significantly less time than developing from

scratch

bull Takes into account intensity and shape

bull Incorporates shape and intensity priors

MIS validation

bull Algorithms without validation are clinically

worthless

bull Validation is with respect to a clinical task

bull Validation requires a ground truth for comparison

bull Physical anatomical models andor phantoms are

typically not available (except sometimes for bones)

bull Ground truth is usually obtained by manual

identification andor segmentation by a user

bull Experts in most cases radiologists

bull Extrinsic comparison compare vs other methods

Copyright L Joskowicz 2010

Copyright L Joskowicz 2010

MIS validation issues

bull Large inter- and intra- variability across experts and clinical sites

bull May not be representative of population variability

bull Main quantitative parameters

ndash validation set size

ndash number and type of observers

ndash intra and inter-observer manual segmentation variability

ndash surface-based and volume-based error measures

MIS validation anatomical variability

stenosis

looping

narrowing

MIS validation anatomical variability

MIS validation metrics

bull Surface-based error measurements

ndash Mean surface distance

ndash RMS surface distance

ndash Maximum surface distance

bull Volume-based error measurements

ndash Dice coefficient

ndash Volumetric overlap error

Copyright L Joskowicz 2010

MIS validation metrics

Surface-based error measurements

bull Mean surface distance

bull RMS surface distance

bull Maximum surface distance

Copyright L Joskowicz 2010

Reference Result

MIS validation metrics

Volume-based error measurements

bull Volumetric overlap error

bull Dice similarity

Copyright L Joskowicz 2010

Reference Result

Aorta

Int + Ext Carotids

All arteries

Subclavian Arteries Bifurcations

16mm (std=08mm)

13mm (std=10mm)

15mm

std=13mm 17mm

std=09mm

07mm

std=07mm

Surface

RMS

Common

Carotid

MIS validation observers variability

bull Intra-observer variability determine how much

variation there is when a single observer

produced the ground-truth segmentation

repeat 5 times the segmentation

bull Inter-observer variability determine how much

variation there is between multiple observers that

produced the ground-truth segmentation

ask 3 radiologists to do the

segmentation

bull Observer expertise and frequency variability

Copyright L Joskowicz 2010

Study [Weltens 2001]

bull Axial MRI slice

bull Nine independent

observers

bull Repeated delineations

bull Interintra observer

variability 30

Key issue fuzzy tumor boundaries

Inter observer variability

Intra-observer variability

8

50

Carotid Lumen Segmentation and Stenosis

Grading Challenge -- The MIDAS Journal

Copyright L Joskowicz 2010

MIS validation manual annotation

Copyright L Joskowicz 2010

MIS validation ground-truth generation

centerlines

(manual)

contours

(manual)

Partial Volume

Segmentation

Reference

standard

from 3PVS

Copyright L Joskowicz 2010

Measurements characteristics

bull Ground-truth not know

bull Repeatability intra-observer variability

bull Reliability inter observer variability

bull Measure correlation between repeated

measurements

bull Intrinsic uncertainty about the tumor volume

bull Accuracy for clinical significance is unknown

Inaccuracy gt Uncertainty

bull Uncertainty may not be improved

bull Goal improve accuracy to obtain

Inaccuracy vs Uncertainty

Results can be

meaningless

Inaccuracy lt Uncertainty

In summaryhellip

bull Medical image segmentation is on the risehellip

bull Basis of many clinical applications

bull Validation is a MUST

bull Many methods and approaches ndash do not re-invent

the wheel

bull I expect patient-specific 3D models to be in the

clinical mainstream in 3-5 years

Copyright L Joskowicz 2011

Segmentation challenges

bull Rigorous quantification and evaluation of

segmentation algorithms performance

bull Shape priors for pathologies

bull Incorporation of functional information from

diffusionperfusion MRI fMRI PET into

segmentation algorithms

Copyright L Joskowicz 2011

Summary (1)

bull Patient-specific anatomy model creation is

currently a major bottleneck in many clinical

applications

bull Automatic anatomical segmentation is essential for

model creation

bull Current tools are limited in scope and coverage

bull Clinical use requires the elimination of the

technician ndash model generation by the physician

Summary (2) bull Great opportunity for the development and

incorporation of anatomy modeling tools in

commercial platforms

bull Growing need and variety of users for anatomical

models

ndash Training simulators surgery rehersal

ndash Intraoperative guidance

ndash Computer Aided Radiology

bull Service providers -- shifting paradigms

Page 25: Recent clinical advances and applications for medical ...crl.med.harvard.edu/research/MICCAI_Tutorial/MICCAI11-tutorial... · Recent clinical advances and applications for medical

Clinical dataset challenges

proximity calcifications stenosis

Segmentation ndash state of the art

bull Hundreds anatomical segmentation methods for a

variety of structures and imaging modalities

bull Families of techniques thresholding region

growing level sets active contours and more

bull Very few if any in routine clinical use

ndash huge gap between prototype and clinical use

ndash time-consuming fragile limited in scope

ndash require technical knowledge

ndash most have limited validation

ndash clinical benefits unproven

bull Segmentation specificity is unavoidable ndash each

anatomical structure and pathology has unique

characteristics and nearby structures

bull Organ structure and pathology-specific

algorithms heart liver long bones spine

bull Very laborious top develop and validate a

segmentation algorithm for each ndash trial and error

process many man-months

No universal segmentation method

Observations (1)

Observations (2) bull Applications have different requirements wrt

ndash Accuracy

ndash Robustness

ndash User interaction

ndash Quality

bull Consider the different requirements of

ndash 3D visualization

ndash Training simulation

ndash FEA simulation

Defining application

requirements ahead of time is

essential

Goals Clinical

bull Automatic or nearly automatic lt 5mins user time

bull Can be used by clinician without a technician

bull Produces robust results

Technical

bull Requires significantly less time than developing from

scratch

bull Takes into account intensity and shape

bull Incorporates shape and intensity priors

MIS validation

bull Algorithms without validation are clinically

worthless

bull Validation is with respect to a clinical task

bull Validation requires a ground truth for comparison

bull Physical anatomical models andor phantoms are

typically not available (except sometimes for bones)

bull Ground truth is usually obtained by manual

identification andor segmentation by a user

bull Experts in most cases radiologists

bull Extrinsic comparison compare vs other methods

Copyright L Joskowicz 2010

Copyright L Joskowicz 2010

MIS validation issues

bull Large inter- and intra- variability across experts and clinical sites

bull May not be representative of population variability

bull Main quantitative parameters

ndash validation set size

ndash number and type of observers

ndash intra and inter-observer manual segmentation variability

ndash surface-based and volume-based error measures

MIS validation anatomical variability

stenosis

looping

narrowing

MIS validation anatomical variability

MIS validation metrics

bull Surface-based error measurements

ndash Mean surface distance

ndash RMS surface distance

ndash Maximum surface distance

bull Volume-based error measurements

ndash Dice coefficient

ndash Volumetric overlap error

Copyright L Joskowicz 2010

MIS validation metrics

Surface-based error measurements

bull Mean surface distance

bull RMS surface distance

bull Maximum surface distance

Copyright L Joskowicz 2010

Reference Result

MIS validation metrics

Volume-based error measurements

bull Volumetric overlap error

bull Dice similarity

Copyright L Joskowicz 2010

Reference Result

Aorta

Int + Ext Carotids

All arteries

Subclavian Arteries Bifurcations

16mm (std=08mm)

13mm (std=10mm)

15mm

std=13mm 17mm

std=09mm

07mm

std=07mm

Surface

RMS

Common

Carotid

MIS validation observers variability

bull Intra-observer variability determine how much

variation there is when a single observer

produced the ground-truth segmentation

repeat 5 times the segmentation

bull Inter-observer variability determine how much

variation there is between multiple observers that

produced the ground-truth segmentation

ask 3 radiologists to do the

segmentation

bull Observer expertise and frequency variability

Copyright L Joskowicz 2010

Study [Weltens 2001]

bull Axial MRI slice

bull Nine independent

observers

bull Repeated delineations

bull Interintra observer

variability 30

Key issue fuzzy tumor boundaries

Inter observer variability

Intra-observer variability

8

50

Carotid Lumen Segmentation and Stenosis

Grading Challenge -- The MIDAS Journal

Copyright L Joskowicz 2010

MIS validation manual annotation

Copyright L Joskowicz 2010

MIS validation ground-truth generation

centerlines

(manual)

contours

(manual)

Partial Volume

Segmentation

Reference

standard

from 3PVS

Copyright L Joskowicz 2010

Measurements characteristics

bull Ground-truth not know

bull Repeatability intra-observer variability

bull Reliability inter observer variability

bull Measure correlation between repeated

measurements

bull Intrinsic uncertainty about the tumor volume

bull Accuracy for clinical significance is unknown

Inaccuracy gt Uncertainty

bull Uncertainty may not be improved

bull Goal improve accuracy to obtain

Inaccuracy vs Uncertainty

Results can be

meaningless

Inaccuracy lt Uncertainty

In summaryhellip

bull Medical image segmentation is on the risehellip

bull Basis of many clinical applications

bull Validation is a MUST

bull Many methods and approaches ndash do not re-invent

the wheel

bull I expect patient-specific 3D models to be in the

clinical mainstream in 3-5 years

Copyright L Joskowicz 2011

Segmentation challenges

bull Rigorous quantification and evaluation of

segmentation algorithms performance

bull Shape priors for pathologies

bull Incorporation of functional information from

diffusionperfusion MRI fMRI PET into

segmentation algorithms

Copyright L Joskowicz 2011

Summary (1)

bull Patient-specific anatomy model creation is

currently a major bottleneck in many clinical

applications

bull Automatic anatomical segmentation is essential for

model creation

bull Current tools are limited in scope and coverage

bull Clinical use requires the elimination of the

technician ndash model generation by the physician

Summary (2) bull Great opportunity for the development and

incorporation of anatomy modeling tools in

commercial platforms

bull Growing need and variety of users for anatomical

models

ndash Training simulators surgery rehersal

ndash Intraoperative guidance

ndash Computer Aided Radiology

bull Service providers -- shifting paradigms

Page 26: Recent clinical advances and applications for medical ...crl.med.harvard.edu/research/MICCAI_Tutorial/MICCAI11-tutorial... · Recent clinical advances and applications for medical

Segmentation ndash state of the art

bull Hundreds anatomical segmentation methods for a

variety of structures and imaging modalities

bull Families of techniques thresholding region

growing level sets active contours and more

bull Very few if any in routine clinical use

ndash huge gap between prototype and clinical use

ndash time-consuming fragile limited in scope

ndash require technical knowledge

ndash most have limited validation

ndash clinical benefits unproven

bull Segmentation specificity is unavoidable ndash each

anatomical structure and pathology has unique

characteristics and nearby structures

bull Organ structure and pathology-specific

algorithms heart liver long bones spine

bull Very laborious top develop and validate a

segmentation algorithm for each ndash trial and error

process many man-months

No universal segmentation method

Observations (1)

Observations (2) bull Applications have different requirements wrt

ndash Accuracy

ndash Robustness

ndash User interaction

ndash Quality

bull Consider the different requirements of

ndash 3D visualization

ndash Training simulation

ndash FEA simulation

Defining application

requirements ahead of time is

essential

Goals Clinical

bull Automatic or nearly automatic lt 5mins user time

bull Can be used by clinician without a technician

bull Produces robust results

Technical

bull Requires significantly less time than developing from

scratch

bull Takes into account intensity and shape

bull Incorporates shape and intensity priors

MIS validation

bull Algorithms without validation are clinically

worthless

bull Validation is with respect to a clinical task

bull Validation requires a ground truth for comparison

bull Physical anatomical models andor phantoms are

typically not available (except sometimes for bones)

bull Ground truth is usually obtained by manual

identification andor segmentation by a user

bull Experts in most cases radiologists

bull Extrinsic comparison compare vs other methods

Copyright L Joskowicz 2010

Copyright L Joskowicz 2010

MIS validation issues

bull Large inter- and intra- variability across experts and clinical sites

bull May not be representative of population variability

bull Main quantitative parameters

ndash validation set size

ndash number and type of observers

ndash intra and inter-observer manual segmentation variability

ndash surface-based and volume-based error measures

MIS validation anatomical variability

stenosis

looping

narrowing

MIS validation anatomical variability

MIS validation metrics

bull Surface-based error measurements

ndash Mean surface distance

ndash RMS surface distance

ndash Maximum surface distance

bull Volume-based error measurements

ndash Dice coefficient

ndash Volumetric overlap error

Copyright L Joskowicz 2010

MIS validation metrics

Surface-based error measurements

bull Mean surface distance

bull RMS surface distance

bull Maximum surface distance

Copyright L Joskowicz 2010

Reference Result

MIS validation metrics

Volume-based error measurements

bull Volumetric overlap error

bull Dice similarity

Copyright L Joskowicz 2010

Reference Result

Aorta

Int + Ext Carotids

All arteries

Subclavian Arteries Bifurcations

16mm (std=08mm)

13mm (std=10mm)

15mm

std=13mm 17mm

std=09mm

07mm

std=07mm

Surface

RMS

Common

Carotid

MIS validation observers variability

bull Intra-observer variability determine how much

variation there is when a single observer

produced the ground-truth segmentation

repeat 5 times the segmentation

bull Inter-observer variability determine how much

variation there is between multiple observers that

produced the ground-truth segmentation

ask 3 radiologists to do the

segmentation

bull Observer expertise and frequency variability

Copyright L Joskowicz 2010

Study [Weltens 2001]

bull Axial MRI slice

bull Nine independent

observers

bull Repeated delineations

bull Interintra observer

variability 30

Key issue fuzzy tumor boundaries

Inter observer variability

Intra-observer variability

8

50

Carotid Lumen Segmentation and Stenosis

Grading Challenge -- The MIDAS Journal

Copyright L Joskowicz 2010

MIS validation manual annotation

Copyright L Joskowicz 2010

MIS validation ground-truth generation

centerlines

(manual)

contours

(manual)

Partial Volume

Segmentation

Reference

standard

from 3PVS

Copyright L Joskowicz 2010

Measurements characteristics

bull Ground-truth not know

bull Repeatability intra-observer variability

bull Reliability inter observer variability

bull Measure correlation between repeated

measurements

bull Intrinsic uncertainty about the tumor volume

bull Accuracy for clinical significance is unknown

Inaccuracy gt Uncertainty

bull Uncertainty may not be improved

bull Goal improve accuracy to obtain

Inaccuracy vs Uncertainty

Results can be

meaningless

Inaccuracy lt Uncertainty

In summaryhellip

bull Medical image segmentation is on the risehellip

bull Basis of many clinical applications

bull Validation is a MUST

bull Many methods and approaches ndash do not re-invent

the wheel

bull I expect patient-specific 3D models to be in the

clinical mainstream in 3-5 years

Copyright L Joskowicz 2011

Segmentation challenges

bull Rigorous quantification and evaluation of

segmentation algorithms performance

bull Shape priors for pathologies

bull Incorporation of functional information from

diffusionperfusion MRI fMRI PET into

segmentation algorithms

Copyright L Joskowicz 2011

Summary (1)

bull Patient-specific anatomy model creation is

currently a major bottleneck in many clinical

applications

bull Automatic anatomical segmentation is essential for

model creation

bull Current tools are limited in scope and coverage

bull Clinical use requires the elimination of the

technician ndash model generation by the physician

Summary (2) bull Great opportunity for the development and

incorporation of anatomy modeling tools in

commercial platforms

bull Growing need and variety of users for anatomical

models

ndash Training simulators surgery rehersal

ndash Intraoperative guidance

ndash Computer Aided Radiology

bull Service providers -- shifting paradigms

Page 27: Recent clinical advances and applications for medical ...crl.med.harvard.edu/research/MICCAI_Tutorial/MICCAI11-tutorial... · Recent clinical advances and applications for medical

bull Segmentation specificity is unavoidable ndash each

anatomical structure and pathology has unique

characteristics and nearby structures

bull Organ structure and pathology-specific

algorithms heart liver long bones spine

bull Very laborious top develop and validate a

segmentation algorithm for each ndash trial and error

process many man-months

No universal segmentation method

Observations (1)

Observations (2) bull Applications have different requirements wrt

ndash Accuracy

ndash Robustness

ndash User interaction

ndash Quality

bull Consider the different requirements of

ndash 3D visualization

ndash Training simulation

ndash FEA simulation

Defining application

requirements ahead of time is

essential

Goals Clinical

bull Automatic or nearly automatic lt 5mins user time

bull Can be used by clinician without a technician

bull Produces robust results

Technical

bull Requires significantly less time than developing from

scratch

bull Takes into account intensity and shape

bull Incorporates shape and intensity priors

MIS validation

bull Algorithms without validation are clinically

worthless

bull Validation is with respect to a clinical task

bull Validation requires a ground truth for comparison

bull Physical anatomical models andor phantoms are

typically not available (except sometimes for bones)

bull Ground truth is usually obtained by manual

identification andor segmentation by a user

bull Experts in most cases radiologists

bull Extrinsic comparison compare vs other methods

Copyright L Joskowicz 2010

Copyright L Joskowicz 2010

MIS validation issues

bull Large inter- and intra- variability across experts and clinical sites

bull May not be representative of population variability

bull Main quantitative parameters

ndash validation set size

ndash number and type of observers

ndash intra and inter-observer manual segmentation variability

ndash surface-based and volume-based error measures

MIS validation anatomical variability

stenosis

looping

narrowing

MIS validation anatomical variability

MIS validation metrics

bull Surface-based error measurements

ndash Mean surface distance

ndash RMS surface distance

ndash Maximum surface distance

bull Volume-based error measurements

ndash Dice coefficient

ndash Volumetric overlap error

Copyright L Joskowicz 2010

MIS validation metrics

Surface-based error measurements

bull Mean surface distance

bull RMS surface distance

bull Maximum surface distance

Copyright L Joskowicz 2010

Reference Result

MIS validation metrics

Volume-based error measurements

bull Volumetric overlap error

bull Dice similarity

Copyright L Joskowicz 2010

Reference Result

Aorta

Int + Ext Carotids

All arteries

Subclavian Arteries Bifurcations

16mm (std=08mm)

13mm (std=10mm)

15mm

std=13mm 17mm

std=09mm

07mm

std=07mm

Surface

RMS

Common

Carotid

MIS validation observers variability

bull Intra-observer variability determine how much

variation there is when a single observer

produced the ground-truth segmentation

repeat 5 times the segmentation

bull Inter-observer variability determine how much

variation there is between multiple observers that

produced the ground-truth segmentation

ask 3 radiologists to do the

segmentation

bull Observer expertise and frequency variability

Copyright L Joskowicz 2010

Study [Weltens 2001]

bull Axial MRI slice

bull Nine independent

observers

bull Repeated delineations

bull Interintra observer

variability 30

Key issue fuzzy tumor boundaries

Inter observer variability

Intra-observer variability

8

50

Carotid Lumen Segmentation and Stenosis

Grading Challenge -- The MIDAS Journal

Copyright L Joskowicz 2010

MIS validation manual annotation

Copyright L Joskowicz 2010

MIS validation ground-truth generation

centerlines

(manual)

contours

(manual)

Partial Volume

Segmentation

Reference

standard

from 3PVS

Copyright L Joskowicz 2010

Measurements characteristics

bull Ground-truth not know

bull Repeatability intra-observer variability

bull Reliability inter observer variability

bull Measure correlation between repeated

measurements

bull Intrinsic uncertainty about the tumor volume

bull Accuracy for clinical significance is unknown

Inaccuracy gt Uncertainty

bull Uncertainty may not be improved

bull Goal improve accuracy to obtain

Inaccuracy vs Uncertainty

Results can be

meaningless

Inaccuracy lt Uncertainty

In summaryhellip

bull Medical image segmentation is on the risehellip

bull Basis of many clinical applications

bull Validation is a MUST

bull Many methods and approaches ndash do not re-invent

the wheel

bull I expect patient-specific 3D models to be in the

clinical mainstream in 3-5 years

Copyright L Joskowicz 2011

Segmentation challenges

bull Rigorous quantification and evaluation of

segmentation algorithms performance

bull Shape priors for pathologies

bull Incorporation of functional information from

diffusionperfusion MRI fMRI PET into

segmentation algorithms

Copyright L Joskowicz 2011

Summary (1)

bull Patient-specific anatomy model creation is

currently a major bottleneck in many clinical

applications

bull Automatic anatomical segmentation is essential for

model creation

bull Current tools are limited in scope and coverage

bull Clinical use requires the elimination of the

technician ndash model generation by the physician

Summary (2) bull Great opportunity for the development and

incorporation of anatomy modeling tools in

commercial platforms

bull Growing need and variety of users for anatomical

models

ndash Training simulators surgery rehersal

ndash Intraoperative guidance

ndash Computer Aided Radiology

bull Service providers -- shifting paradigms

Page 28: Recent clinical advances and applications for medical ...crl.med.harvard.edu/research/MICCAI_Tutorial/MICCAI11-tutorial... · Recent clinical advances and applications for medical

Observations (2) bull Applications have different requirements wrt

ndash Accuracy

ndash Robustness

ndash User interaction

ndash Quality

bull Consider the different requirements of

ndash 3D visualization

ndash Training simulation

ndash FEA simulation

Defining application

requirements ahead of time is

essential

Goals Clinical

bull Automatic or nearly automatic lt 5mins user time

bull Can be used by clinician without a technician

bull Produces robust results

Technical

bull Requires significantly less time than developing from

scratch

bull Takes into account intensity and shape

bull Incorporates shape and intensity priors

MIS validation

bull Algorithms without validation are clinically

worthless

bull Validation is with respect to a clinical task

bull Validation requires a ground truth for comparison

bull Physical anatomical models andor phantoms are

typically not available (except sometimes for bones)

bull Ground truth is usually obtained by manual

identification andor segmentation by a user

bull Experts in most cases radiologists

bull Extrinsic comparison compare vs other methods

Copyright L Joskowicz 2010

Copyright L Joskowicz 2010

MIS validation issues

bull Large inter- and intra- variability across experts and clinical sites

bull May not be representative of population variability

bull Main quantitative parameters

ndash validation set size

ndash number and type of observers

ndash intra and inter-observer manual segmentation variability

ndash surface-based and volume-based error measures

MIS validation anatomical variability

stenosis

looping

narrowing

MIS validation anatomical variability

MIS validation metrics

bull Surface-based error measurements

ndash Mean surface distance

ndash RMS surface distance

ndash Maximum surface distance

bull Volume-based error measurements

ndash Dice coefficient

ndash Volumetric overlap error

Copyright L Joskowicz 2010

MIS validation metrics

Surface-based error measurements

bull Mean surface distance

bull RMS surface distance

bull Maximum surface distance

Copyright L Joskowicz 2010

Reference Result

MIS validation metrics

Volume-based error measurements

bull Volumetric overlap error

bull Dice similarity

Copyright L Joskowicz 2010

Reference Result

Aorta

Int + Ext Carotids

All arteries

Subclavian Arteries Bifurcations

16mm (std=08mm)

13mm (std=10mm)

15mm

std=13mm 17mm

std=09mm

07mm

std=07mm

Surface

RMS

Common

Carotid

MIS validation observers variability

bull Intra-observer variability determine how much

variation there is when a single observer

produced the ground-truth segmentation

repeat 5 times the segmentation

bull Inter-observer variability determine how much

variation there is between multiple observers that

produced the ground-truth segmentation

ask 3 radiologists to do the

segmentation

bull Observer expertise and frequency variability

Copyright L Joskowicz 2010

Study [Weltens 2001]

bull Axial MRI slice

bull Nine independent

observers

bull Repeated delineations

bull Interintra observer

variability 30

Key issue fuzzy tumor boundaries

Inter observer variability

Intra-observer variability

8

50

Carotid Lumen Segmentation and Stenosis

Grading Challenge -- The MIDAS Journal

Copyright L Joskowicz 2010

MIS validation manual annotation

Copyright L Joskowicz 2010

MIS validation ground-truth generation

centerlines

(manual)

contours

(manual)

Partial Volume

Segmentation

Reference

standard

from 3PVS

Copyright L Joskowicz 2010

Measurements characteristics

bull Ground-truth not know

bull Repeatability intra-observer variability

bull Reliability inter observer variability

bull Measure correlation between repeated

measurements

bull Intrinsic uncertainty about the tumor volume

bull Accuracy for clinical significance is unknown

Inaccuracy gt Uncertainty

bull Uncertainty may not be improved

bull Goal improve accuracy to obtain

Inaccuracy vs Uncertainty

Results can be

meaningless

Inaccuracy lt Uncertainty

In summaryhellip

bull Medical image segmentation is on the risehellip

bull Basis of many clinical applications

bull Validation is a MUST

bull Many methods and approaches ndash do not re-invent

the wheel

bull I expect patient-specific 3D models to be in the

clinical mainstream in 3-5 years

Copyright L Joskowicz 2011

Segmentation challenges

bull Rigorous quantification and evaluation of

segmentation algorithms performance

bull Shape priors for pathologies

bull Incorporation of functional information from

diffusionperfusion MRI fMRI PET into

segmentation algorithms

Copyright L Joskowicz 2011

Summary (1)

bull Patient-specific anatomy model creation is

currently a major bottleneck in many clinical

applications

bull Automatic anatomical segmentation is essential for

model creation

bull Current tools are limited in scope and coverage

bull Clinical use requires the elimination of the

technician ndash model generation by the physician

Summary (2) bull Great opportunity for the development and

incorporation of anatomy modeling tools in

commercial platforms

bull Growing need and variety of users for anatomical

models

ndash Training simulators surgery rehersal

ndash Intraoperative guidance

ndash Computer Aided Radiology

bull Service providers -- shifting paradigms

Page 29: Recent clinical advances and applications for medical ...crl.med.harvard.edu/research/MICCAI_Tutorial/MICCAI11-tutorial... · Recent clinical advances and applications for medical

Goals Clinical

bull Automatic or nearly automatic lt 5mins user time

bull Can be used by clinician without a technician

bull Produces robust results

Technical

bull Requires significantly less time than developing from

scratch

bull Takes into account intensity and shape

bull Incorporates shape and intensity priors

MIS validation

bull Algorithms without validation are clinically

worthless

bull Validation is with respect to a clinical task

bull Validation requires a ground truth for comparison

bull Physical anatomical models andor phantoms are

typically not available (except sometimes for bones)

bull Ground truth is usually obtained by manual

identification andor segmentation by a user

bull Experts in most cases radiologists

bull Extrinsic comparison compare vs other methods

Copyright L Joskowicz 2010

Copyright L Joskowicz 2010

MIS validation issues

bull Large inter- and intra- variability across experts and clinical sites

bull May not be representative of population variability

bull Main quantitative parameters

ndash validation set size

ndash number and type of observers

ndash intra and inter-observer manual segmentation variability

ndash surface-based and volume-based error measures

MIS validation anatomical variability

stenosis

looping

narrowing

MIS validation anatomical variability

MIS validation metrics

bull Surface-based error measurements

ndash Mean surface distance

ndash RMS surface distance

ndash Maximum surface distance

bull Volume-based error measurements

ndash Dice coefficient

ndash Volumetric overlap error

Copyright L Joskowicz 2010

MIS validation metrics

Surface-based error measurements

bull Mean surface distance

bull RMS surface distance

bull Maximum surface distance

Copyright L Joskowicz 2010

Reference Result

MIS validation metrics

Volume-based error measurements

bull Volumetric overlap error

bull Dice similarity

Copyright L Joskowicz 2010

Reference Result

Aorta

Int + Ext Carotids

All arteries

Subclavian Arteries Bifurcations

16mm (std=08mm)

13mm (std=10mm)

15mm

std=13mm 17mm

std=09mm

07mm

std=07mm

Surface

RMS

Common

Carotid

MIS validation observers variability

bull Intra-observer variability determine how much

variation there is when a single observer

produced the ground-truth segmentation

repeat 5 times the segmentation

bull Inter-observer variability determine how much

variation there is between multiple observers that

produced the ground-truth segmentation

ask 3 radiologists to do the

segmentation

bull Observer expertise and frequency variability

Copyright L Joskowicz 2010

Study [Weltens 2001]

bull Axial MRI slice

bull Nine independent

observers

bull Repeated delineations

bull Interintra observer

variability 30

Key issue fuzzy tumor boundaries

Inter observer variability

Intra-observer variability

8

50

Carotid Lumen Segmentation and Stenosis

Grading Challenge -- The MIDAS Journal

Copyright L Joskowicz 2010

MIS validation manual annotation

Copyright L Joskowicz 2010

MIS validation ground-truth generation

centerlines

(manual)

contours

(manual)

Partial Volume

Segmentation

Reference

standard

from 3PVS

Copyright L Joskowicz 2010

Measurements characteristics

bull Ground-truth not know

bull Repeatability intra-observer variability

bull Reliability inter observer variability

bull Measure correlation between repeated

measurements

bull Intrinsic uncertainty about the tumor volume

bull Accuracy for clinical significance is unknown

Inaccuracy gt Uncertainty

bull Uncertainty may not be improved

bull Goal improve accuracy to obtain

Inaccuracy vs Uncertainty

Results can be

meaningless

Inaccuracy lt Uncertainty

In summaryhellip

bull Medical image segmentation is on the risehellip

bull Basis of many clinical applications

bull Validation is a MUST

bull Many methods and approaches ndash do not re-invent

the wheel

bull I expect patient-specific 3D models to be in the

clinical mainstream in 3-5 years

Copyright L Joskowicz 2011

Segmentation challenges

bull Rigorous quantification and evaluation of

segmentation algorithms performance

bull Shape priors for pathologies

bull Incorporation of functional information from

diffusionperfusion MRI fMRI PET into

segmentation algorithms

Copyright L Joskowicz 2011

Summary (1)

bull Patient-specific anatomy model creation is

currently a major bottleneck in many clinical

applications

bull Automatic anatomical segmentation is essential for

model creation

bull Current tools are limited in scope and coverage

bull Clinical use requires the elimination of the

technician ndash model generation by the physician

Summary (2) bull Great opportunity for the development and

incorporation of anatomy modeling tools in

commercial platforms

bull Growing need and variety of users for anatomical

models

ndash Training simulators surgery rehersal

ndash Intraoperative guidance

ndash Computer Aided Radiology

bull Service providers -- shifting paradigms

Page 30: Recent clinical advances and applications for medical ...crl.med.harvard.edu/research/MICCAI_Tutorial/MICCAI11-tutorial... · Recent clinical advances and applications for medical

MIS validation

bull Algorithms without validation are clinically

worthless

bull Validation is with respect to a clinical task

bull Validation requires a ground truth for comparison

bull Physical anatomical models andor phantoms are

typically not available (except sometimes for bones)

bull Ground truth is usually obtained by manual

identification andor segmentation by a user

bull Experts in most cases radiologists

bull Extrinsic comparison compare vs other methods

Copyright L Joskowicz 2010

Copyright L Joskowicz 2010

MIS validation issues

bull Large inter- and intra- variability across experts and clinical sites

bull May not be representative of population variability

bull Main quantitative parameters

ndash validation set size

ndash number and type of observers

ndash intra and inter-observer manual segmentation variability

ndash surface-based and volume-based error measures

MIS validation anatomical variability

stenosis

looping

narrowing

MIS validation anatomical variability

MIS validation metrics

bull Surface-based error measurements

ndash Mean surface distance

ndash RMS surface distance

ndash Maximum surface distance

bull Volume-based error measurements

ndash Dice coefficient

ndash Volumetric overlap error

Copyright L Joskowicz 2010

MIS validation metrics

Surface-based error measurements

bull Mean surface distance

bull RMS surface distance

bull Maximum surface distance

Copyright L Joskowicz 2010

Reference Result

MIS validation metrics

Volume-based error measurements

bull Volumetric overlap error

bull Dice similarity

Copyright L Joskowicz 2010

Reference Result

Aorta

Int + Ext Carotids

All arteries

Subclavian Arteries Bifurcations

16mm (std=08mm)

13mm (std=10mm)

15mm

std=13mm 17mm

std=09mm

07mm

std=07mm

Surface

RMS

Common

Carotid

MIS validation observers variability

bull Intra-observer variability determine how much

variation there is when a single observer

produced the ground-truth segmentation

repeat 5 times the segmentation

bull Inter-observer variability determine how much

variation there is between multiple observers that

produced the ground-truth segmentation

ask 3 radiologists to do the

segmentation

bull Observer expertise and frequency variability

Copyright L Joskowicz 2010

Study [Weltens 2001]

bull Axial MRI slice

bull Nine independent

observers

bull Repeated delineations

bull Interintra observer

variability 30

Key issue fuzzy tumor boundaries

Inter observer variability

Intra-observer variability

8

50

Carotid Lumen Segmentation and Stenosis

Grading Challenge -- The MIDAS Journal

Copyright L Joskowicz 2010

MIS validation manual annotation

Copyright L Joskowicz 2010

MIS validation ground-truth generation

centerlines

(manual)

contours

(manual)

Partial Volume

Segmentation

Reference

standard

from 3PVS

Copyright L Joskowicz 2010

Measurements characteristics

bull Ground-truth not know

bull Repeatability intra-observer variability

bull Reliability inter observer variability

bull Measure correlation between repeated

measurements

bull Intrinsic uncertainty about the tumor volume

bull Accuracy for clinical significance is unknown

Inaccuracy gt Uncertainty

bull Uncertainty may not be improved

bull Goal improve accuracy to obtain

Inaccuracy vs Uncertainty

Results can be

meaningless

Inaccuracy lt Uncertainty

In summaryhellip

bull Medical image segmentation is on the risehellip

bull Basis of many clinical applications

bull Validation is a MUST

bull Many methods and approaches ndash do not re-invent

the wheel

bull I expect patient-specific 3D models to be in the

clinical mainstream in 3-5 years

Copyright L Joskowicz 2011

Segmentation challenges

bull Rigorous quantification and evaluation of

segmentation algorithms performance

bull Shape priors for pathologies

bull Incorporation of functional information from

diffusionperfusion MRI fMRI PET into

segmentation algorithms

Copyright L Joskowicz 2011

Summary (1)

bull Patient-specific anatomy model creation is

currently a major bottleneck in many clinical

applications

bull Automatic anatomical segmentation is essential for

model creation

bull Current tools are limited in scope and coverage

bull Clinical use requires the elimination of the

technician ndash model generation by the physician

Summary (2) bull Great opportunity for the development and

incorporation of anatomy modeling tools in

commercial platforms

bull Growing need and variety of users for anatomical

models

ndash Training simulators surgery rehersal

ndash Intraoperative guidance

ndash Computer Aided Radiology

bull Service providers -- shifting paradigms

Page 31: Recent clinical advances and applications for medical ...crl.med.harvard.edu/research/MICCAI_Tutorial/MICCAI11-tutorial... · Recent clinical advances and applications for medical

Copyright L Joskowicz 2010

MIS validation issues

bull Large inter- and intra- variability across experts and clinical sites

bull May not be representative of population variability

bull Main quantitative parameters

ndash validation set size

ndash number and type of observers

ndash intra and inter-observer manual segmentation variability

ndash surface-based and volume-based error measures

MIS validation anatomical variability

stenosis

looping

narrowing

MIS validation anatomical variability

MIS validation metrics

bull Surface-based error measurements

ndash Mean surface distance

ndash RMS surface distance

ndash Maximum surface distance

bull Volume-based error measurements

ndash Dice coefficient

ndash Volumetric overlap error

Copyright L Joskowicz 2010

MIS validation metrics

Surface-based error measurements

bull Mean surface distance

bull RMS surface distance

bull Maximum surface distance

Copyright L Joskowicz 2010

Reference Result

MIS validation metrics

Volume-based error measurements

bull Volumetric overlap error

bull Dice similarity

Copyright L Joskowicz 2010

Reference Result

Aorta

Int + Ext Carotids

All arteries

Subclavian Arteries Bifurcations

16mm (std=08mm)

13mm (std=10mm)

15mm

std=13mm 17mm

std=09mm

07mm

std=07mm

Surface

RMS

Common

Carotid

MIS validation observers variability

bull Intra-observer variability determine how much

variation there is when a single observer

produced the ground-truth segmentation

repeat 5 times the segmentation

bull Inter-observer variability determine how much

variation there is between multiple observers that

produced the ground-truth segmentation

ask 3 radiologists to do the

segmentation

bull Observer expertise and frequency variability

Copyright L Joskowicz 2010

Study [Weltens 2001]

bull Axial MRI slice

bull Nine independent

observers

bull Repeated delineations

bull Interintra observer

variability 30

Key issue fuzzy tumor boundaries

Inter observer variability

Intra-observer variability

8

50

Carotid Lumen Segmentation and Stenosis

Grading Challenge -- The MIDAS Journal

Copyright L Joskowicz 2010

MIS validation manual annotation

Copyright L Joskowicz 2010

MIS validation ground-truth generation

centerlines

(manual)

contours

(manual)

Partial Volume

Segmentation

Reference

standard

from 3PVS

Copyright L Joskowicz 2010

Measurements characteristics

bull Ground-truth not know

bull Repeatability intra-observer variability

bull Reliability inter observer variability

bull Measure correlation between repeated

measurements

bull Intrinsic uncertainty about the tumor volume

bull Accuracy for clinical significance is unknown

Inaccuracy gt Uncertainty

bull Uncertainty may not be improved

bull Goal improve accuracy to obtain

Inaccuracy vs Uncertainty

Results can be

meaningless

Inaccuracy lt Uncertainty

In summaryhellip

bull Medical image segmentation is on the risehellip

bull Basis of many clinical applications

bull Validation is a MUST

bull Many methods and approaches ndash do not re-invent

the wheel

bull I expect patient-specific 3D models to be in the

clinical mainstream in 3-5 years

Copyright L Joskowicz 2011

Segmentation challenges

bull Rigorous quantification and evaluation of

segmentation algorithms performance

bull Shape priors for pathologies

bull Incorporation of functional information from

diffusionperfusion MRI fMRI PET into

segmentation algorithms

Copyright L Joskowicz 2011

Summary (1)

bull Patient-specific anatomy model creation is

currently a major bottleneck in many clinical

applications

bull Automatic anatomical segmentation is essential for

model creation

bull Current tools are limited in scope and coverage

bull Clinical use requires the elimination of the

technician ndash model generation by the physician

Summary (2) bull Great opportunity for the development and

incorporation of anatomy modeling tools in

commercial platforms

bull Growing need and variety of users for anatomical

models

ndash Training simulators surgery rehersal

ndash Intraoperative guidance

ndash Computer Aided Radiology

bull Service providers -- shifting paradigms

Page 32: Recent clinical advances and applications for medical ...crl.med.harvard.edu/research/MICCAI_Tutorial/MICCAI11-tutorial... · Recent clinical advances and applications for medical

MIS validation anatomical variability

stenosis

looping

narrowing

MIS validation anatomical variability

MIS validation metrics

bull Surface-based error measurements

ndash Mean surface distance

ndash RMS surface distance

ndash Maximum surface distance

bull Volume-based error measurements

ndash Dice coefficient

ndash Volumetric overlap error

Copyright L Joskowicz 2010

MIS validation metrics

Surface-based error measurements

bull Mean surface distance

bull RMS surface distance

bull Maximum surface distance

Copyright L Joskowicz 2010

Reference Result

MIS validation metrics

Volume-based error measurements

bull Volumetric overlap error

bull Dice similarity

Copyright L Joskowicz 2010

Reference Result

Aorta

Int + Ext Carotids

All arteries

Subclavian Arteries Bifurcations

16mm (std=08mm)

13mm (std=10mm)

15mm

std=13mm 17mm

std=09mm

07mm

std=07mm

Surface

RMS

Common

Carotid

MIS validation observers variability

bull Intra-observer variability determine how much

variation there is when a single observer

produced the ground-truth segmentation

repeat 5 times the segmentation

bull Inter-observer variability determine how much

variation there is between multiple observers that

produced the ground-truth segmentation

ask 3 radiologists to do the

segmentation

bull Observer expertise and frequency variability

Copyright L Joskowicz 2010

Study [Weltens 2001]

bull Axial MRI slice

bull Nine independent

observers

bull Repeated delineations

bull Interintra observer

variability 30

Key issue fuzzy tumor boundaries

Inter observer variability

Intra-observer variability

8

50

Carotid Lumen Segmentation and Stenosis

Grading Challenge -- The MIDAS Journal

Copyright L Joskowicz 2010

MIS validation manual annotation

Copyright L Joskowicz 2010

MIS validation ground-truth generation

centerlines

(manual)

contours

(manual)

Partial Volume

Segmentation

Reference

standard

from 3PVS

Copyright L Joskowicz 2010

Measurements characteristics

bull Ground-truth not know

bull Repeatability intra-observer variability

bull Reliability inter observer variability

bull Measure correlation between repeated

measurements

bull Intrinsic uncertainty about the tumor volume

bull Accuracy for clinical significance is unknown

Inaccuracy gt Uncertainty

bull Uncertainty may not be improved

bull Goal improve accuracy to obtain

Inaccuracy vs Uncertainty

Results can be

meaningless

Inaccuracy lt Uncertainty

In summaryhellip

bull Medical image segmentation is on the risehellip

bull Basis of many clinical applications

bull Validation is a MUST

bull Many methods and approaches ndash do not re-invent

the wheel

bull I expect patient-specific 3D models to be in the

clinical mainstream in 3-5 years

Copyright L Joskowicz 2011

Segmentation challenges

bull Rigorous quantification and evaluation of

segmentation algorithms performance

bull Shape priors for pathologies

bull Incorporation of functional information from

diffusionperfusion MRI fMRI PET into

segmentation algorithms

Copyright L Joskowicz 2011

Summary (1)

bull Patient-specific anatomy model creation is

currently a major bottleneck in many clinical

applications

bull Automatic anatomical segmentation is essential for

model creation

bull Current tools are limited in scope and coverage

bull Clinical use requires the elimination of the

technician ndash model generation by the physician

Summary (2) bull Great opportunity for the development and

incorporation of anatomy modeling tools in

commercial platforms

bull Growing need and variety of users for anatomical

models

ndash Training simulators surgery rehersal

ndash Intraoperative guidance

ndash Computer Aided Radiology

bull Service providers -- shifting paradigms

Page 33: Recent clinical advances and applications for medical ...crl.med.harvard.edu/research/MICCAI_Tutorial/MICCAI11-tutorial... · Recent clinical advances and applications for medical

stenosis

looping

narrowing

MIS validation anatomical variability

MIS validation metrics

bull Surface-based error measurements

ndash Mean surface distance

ndash RMS surface distance

ndash Maximum surface distance

bull Volume-based error measurements

ndash Dice coefficient

ndash Volumetric overlap error

Copyright L Joskowicz 2010

MIS validation metrics

Surface-based error measurements

bull Mean surface distance

bull RMS surface distance

bull Maximum surface distance

Copyright L Joskowicz 2010

Reference Result

MIS validation metrics

Volume-based error measurements

bull Volumetric overlap error

bull Dice similarity

Copyright L Joskowicz 2010

Reference Result

Aorta

Int + Ext Carotids

All arteries

Subclavian Arteries Bifurcations

16mm (std=08mm)

13mm (std=10mm)

15mm

std=13mm 17mm

std=09mm

07mm

std=07mm

Surface

RMS

Common

Carotid

MIS validation observers variability

bull Intra-observer variability determine how much

variation there is when a single observer

produced the ground-truth segmentation

repeat 5 times the segmentation

bull Inter-observer variability determine how much

variation there is between multiple observers that

produced the ground-truth segmentation

ask 3 radiologists to do the

segmentation

bull Observer expertise and frequency variability

Copyright L Joskowicz 2010

Study [Weltens 2001]

bull Axial MRI slice

bull Nine independent

observers

bull Repeated delineations

bull Interintra observer

variability 30

Key issue fuzzy tumor boundaries

Inter observer variability

Intra-observer variability

8

50

Carotid Lumen Segmentation and Stenosis

Grading Challenge -- The MIDAS Journal

Copyright L Joskowicz 2010

MIS validation manual annotation

Copyright L Joskowicz 2010

MIS validation ground-truth generation

centerlines

(manual)

contours

(manual)

Partial Volume

Segmentation

Reference

standard

from 3PVS

Copyright L Joskowicz 2010

Measurements characteristics

bull Ground-truth not know

bull Repeatability intra-observer variability

bull Reliability inter observer variability

bull Measure correlation between repeated

measurements

bull Intrinsic uncertainty about the tumor volume

bull Accuracy for clinical significance is unknown

Inaccuracy gt Uncertainty

bull Uncertainty may not be improved

bull Goal improve accuracy to obtain

Inaccuracy vs Uncertainty

Results can be

meaningless

Inaccuracy lt Uncertainty

In summaryhellip

bull Medical image segmentation is on the risehellip

bull Basis of many clinical applications

bull Validation is a MUST

bull Many methods and approaches ndash do not re-invent

the wheel

bull I expect patient-specific 3D models to be in the

clinical mainstream in 3-5 years

Copyright L Joskowicz 2011

Segmentation challenges

bull Rigorous quantification and evaluation of

segmentation algorithms performance

bull Shape priors for pathologies

bull Incorporation of functional information from

diffusionperfusion MRI fMRI PET into

segmentation algorithms

Copyright L Joskowicz 2011

Summary (1)

bull Patient-specific anatomy model creation is

currently a major bottleneck in many clinical

applications

bull Automatic anatomical segmentation is essential for

model creation

bull Current tools are limited in scope and coverage

bull Clinical use requires the elimination of the

technician ndash model generation by the physician

Summary (2) bull Great opportunity for the development and

incorporation of anatomy modeling tools in

commercial platforms

bull Growing need and variety of users for anatomical

models

ndash Training simulators surgery rehersal

ndash Intraoperative guidance

ndash Computer Aided Radiology

bull Service providers -- shifting paradigms

Page 34: Recent clinical advances and applications for medical ...crl.med.harvard.edu/research/MICCAI_Tutorial/MICCAI11-tutorial... · Recent clinical advances and applications for medical

MIS validation metrics

bull Surface-based error measurements

ndash Mean surface distance

ndash RMS surface distance

ndash Maximum surface distance

bull Volume-based error measurements

ndash Dice coefficient

ndash Volumetric overlap error

Copyright L Joskowicz 2010

MIS validation metrics

Surface-based error measurements

bull Mean surface distance

bull RMS surface distance

bull Maximum surface distance

Copyright L Joskowicz 2010

Reference Result

MIS validation metrics

Volume-based error measurements

bull Volumetric overlap error

bull Dice similarity

Copyright L Joskowicz 2010

Reference Result

Aorta

Int + Ext Carotids

All arteries

Subclavian Arteries Bifurcations

16mm (std=08mm)

13mm (std=10mm)

15mm

std=13mm 17mm

std=09mm

07mm

std=07mm

Surface

RMS

Common

Carotid

MIS validation observers variability

bull Intra-observer variability determine how much

variation there is when a single observer

produced the ground-truth segmentation

repeat 5 times the segmentation

bull Inter-observer variability determine how much

variation there is between multiple observers that

produced the ground-truth segmentation

ask 3 radiologists to do the

segmentation

bull Observer expertise and frequency variability

Copyright L Joskowicz 2010

Study [Weltens 2001]

bull Axial MRI slice

bull Nine independent

observers

bull Repeated delineations

bull Interintra observer

variability 30

Key issue fuzzy tumor boundaries

Inter observer variability

Intra-observer variability

8

50

Carotid Lumen Segmentation and Stenosis

Grading Challenge -- The MIDAS Journal

Copyright L Joskowicz 2010

MIS validation manual annotation

Copyright L Joskowicz 2010

MIS validation ground-truth generation

centerlines

(manual)

contours

(manual)

Partial Volume

Segmentation

Reference

standard

from 3PVS

Copyright L Joskowicz 2010

Measurements characteristics

bull Ground-truth not know

bull Repeatability intra-observer variability

bull Reliability inter observer variability

bull Measure correlation between repeated

measurements

bull Intrinsic uncertainty about the tumor volume

bull Accuracy for clinical significance is unknown

Inaccuracy gt Uncertainty

bull Uncertainty may not be improved

bull Goal improve accuracy to obtain

Inaccuracy vs Uncertainty

Results can be

meaningless

Inaccuracy lt Uncertainty

In summaryhellip

bull Medical image segmentation is on the risehellip

bull Basis of many clinical applications

bull Validation is a MUST

bull Many methods and approaches ndash do not re-invent

the wheel

bull I expect patient-specific 3D models to be in the

clinical mainstream in 3-5 years

Copyright L Joskowicz 2011

Segmentation challenges

bull Rigorous quantification and evaluation of

segmentation algorithms performance

bull Shape priors for pathologies

bull Incorporation of functional information from

diffusionperfusion MRI fMRI PET into

segmentation algorithms

Copyright L Joskowicz 2011

Summary (1)

bull Patient-specific anatomy model creation is

currently a major bottleneck in many clinical

applications

bull Automatic anatomical segmentation is essential for

model creation

bull Current tools are limited in scope and coverage

bull Clinical use requires the elimination of the

technician ndash model generation by the physician

Summary (2) bull Great opportunity for the development and

incorporation of anatomy modeling tools in

commercial platforms

bull Growing need and variety of users for anatomical

models

ndash Training simulators surgery rehersal

ndash Intraoperative guidance

ndash Computer Aided Radiology

bull Service providers -- shifting paradigms

Page 35: Recent clinical advances and applications for medical ...crl.med.harvard.edu/research/MICCAI_Tutorial/MICCAI11-tutorial... · Recent clinical advances and applications for medical

MIS validation metrics

Surface-based error measurements

bull Mean surface distance

bull RMS surface distance

bull Maximum surface distance

Copyright L Joskowicz 2010

Reference Result

MIS validation metrics

Volume-based error measurements

bull Volumetric overlap error

bull Dice similarity

Copyright L Joskowicz 2010

Reference Result

Aorta

Int + Ext Carotids

All arteries

Subclavian Arteries Bifurcations

16mm (std=08mm)

13mm (std=10mm)

15mm

std=13mm 17mm

std=09mm

07mm

std=07mm

Surface

RMS

Common

Carotid

MIS validation observers variability

bull Intra-observer variability determine how much

variation there is when a single observer

produced the ground-truth segmentation

repeat 5 times the segmentation

bull Inter-observer variability determine how much

variation there is between multiple observers that

produced the ground-truth segmentation

ask 3 radiologists to do the

segmentation

bull Observer expertise and frequency variability

Copyright L Joskowicz 2010

Study [Weltens 2001]

bull Axial MRI slice

bull Nine independent

observers

bull Repeated delineations

bull Interintra observer

variability 30

Key issue fuzzy tumor boundaries

Inter observer variability

Intra-observer variability

8

50

Carotid Lumen Segmentation and Stenosis

Grading Challenge -- The MIDAS Journal

Copyright L Joskowicz 2010

MIS validation manual annotation

Copyright L Joskowicz 2010

MIS validation ground-truth generation

centerlines

(manual)

contours

(manual)

Partial Volume

Segmentation

Reference

standard

from 3PVS

Copyright L Joskowicz 2010

Measurements characteristics

bull Ground-truth not know

bull Repeatability intra-observer variability

bull Reliability inter observer variability

bull Measure correlation between repeated

measurements

bull Intrinsic uncertainty about the tumor volume

bull Accuracy for clinical significance is unknown

Inaccuracy gt Uncertainty

bull Uncertainty may not be improved

bull Goal improve accuracy to obtain

Inaccuracy vs Uncertainty

Results can be

meaningless

Inaccuracy lt Uncertainty

In summaryhellip

bull Medical image segmentation is on the risehellip

bull Basis of many clinical applications

bull Validation is a MUST

bull Many methods and approaches ndash do not re-invent

the wheel

bull I expect patient-specific 3D models to be in the

clinical mainstream in 3-5 years

Copyright L Joskowicz 2011

Segmentation challenges

bull Rigorous quantification and evaluation of

segmentation algorithms performance

bull Shape priors for pathologies

bull Incorporation of functional information from

diffusionperfusion MRI fMRI PET into

segmentation algorithms

Copyright L Joskowicz 2011

Summary (1)

bull Patient-specific anatomy model creation is

currently a major bottleneck in many clinical

applications

bull Automatic anatomical segmentation is essential for

model creation

bull Current tools are limited in scope and coverage

bull Clinical use requires the elimination of the

technician ndash model generation by the physician

Summary (2) bull Great opportunity for the development and

incorporation of anatomy modeling tools in

commercial platforms

bull Growing need and variety of users for anatomical

models

ndash Training simulators surgery rehersal

ndash Intraoperative guidance

ndash Computer Aided Radiology

bull Service providers -- shifting paradigms

Page 36: Recent clinical advances and applications for medical ...crl.med.harvard.edu/research/MICCAI_Tutorial/MICCAI11-tutorial... · Recent clinical advances and applications for medical

MIS validation metrics

Volume-based error measurements

bull Volumetric overlap error

bull Dice similarity

Copyright L Joskowicz 2010

Reference Result

Aorta

Int + Ext Carotids

All arteries

Subclavian Arteries Bifurcations

16mm (std=08mm)

13mm (std=10mm)

15mm

std=13mm 17mm

std=09mm

07mm

std=07mm

Surface

RMS

Common

Carotid

MIS validation observers variability

bull Intra-observer variability determine how much

variation there is when a single observer

produced the ground-truth segmentation

repeat 5 times the segmentation

bull Inter-observer variability determine how much

variation there is between multiple observers that

produced the ground-truth segmentation

ask 3 radiologists to do the

segmentation

bull Observer expertise and frequency variability

Copyright L Joskowicz 2010

Study [Weltens 2001]

bull Axial MRI slice

bull Nine independent

observers

bull Repeated delineations

bull Interintra observer

variability 30

Key issue fuzzy tumor boundaries

Inter observer variability

Intra-observer variability

8

50

Carotid Lumen Segmentation and Stenosis

Grading Challenge -- The MIDAS Journal

Copyright L Joskowicz 2010

MIS validation manual annotation

Copyright L Joskowicz 2010

MIS validation ground-truth generation

centerlines

(manual)

contours

(manual)

Partial Volume

Segmentation

Reference

standard

from 3PVS

Copyright L Joskowicz 2010

Measurements characteristics

bull Ground-truth not know

bull Repeatability intra-observer variability

bull Reliability inter observer variability

bull Measure correlation between repeated

measurements

bull Intrinsic uncertainty about the tumor volume

bull Accuracy for clinical significance is unknown

Inaccuracy gt Uncertainty

bull Uncertainty may not be improved

bull Goal improve accuracy to obtain

Inaccuracy vs Uncertainty

Results can be

meaningless

Inaccuracy lt Uncertainty

In summaryhellip

bull Medical image segmentation is on the risehellip

bull Basis of many clinical applications

bull Validation is a MUST

bull Many methods and approaches ndash do not re-invent

the wheel

bull I expect patient-specific 3D models to be in the

clinical mainstream in 3-5 years

Copyright L Joskowicz 2011

Segmentation challenges

bull Rigorous quantification and evaluation of

segmentation algorithms performance

bull Shape priors for pathologies

bull Incorporation of functional information from

diffusionperfusion MRI fMRI PET into

segmentation algorithms

Copyright L Joskowicz 2011

Summary (1)

bull Patient-specific anatomy model creation is

currently a major bottleneck in many clinical

applications

bull Automatic anatomical segmentation is essential for

model creation

bull Current tools are limited in scope and coverage

bull Clinical use requires the elimination of the

technician ndash model generation by the physician

Summary (2) bull Great opportunity for the development and

incorporation of anatomy modeling tools in

commercial platforms

bull Growing need and variety of users for anatomical

models

ndash Training simulators surgery rehersal

ndash Intraoperative guidance

ndash Computer Aided Radiology

bull Service providers -- shifting paradigms

Page 37: Recent clinical advances and applications for medical ...crl.med.harvard.edu/research/MICCAI_Tutorial/MICCAI11-tutorial... · Recent clinical advances and applications for medical

Aorta

Int + Ext Carotids

All arteries

Subclavian Arteries Bifurcations

16mm (std=08mm)

13mm (std=10mm)

15mm

std=13mm 17mm

std=09mm

07mm

std=07mm

Surface

RMS

Common

Carotid

MIS validation observers variability

bull Intra-observer variability determine how much

variation there is when a single observer

produced the ground-truth segmentation

repeat 5 times the segmentation

bull Inter-observer variability determine how much

variation there is between multiple observers that

produced the ground-truth segmentation

ask 3 radiologists to do the

segmentation

bull Observer expertise and frequency variability

Copyright L Joskowicz 2010

Study [Weltens 2001]

bull Axial MRI slice

bull Nine independent

observers

bull Repeated delineations

bull Interintra observer

variability 30

Key issue fuzzy tumor boundaries

Inter observer variability

Intra-observer variability

8

50

Carotid Lumen Segmentation and Stenosis

Grading Challenge -- The MIDAS Journal

Copyright L Joskowicz 2010

MIS validation manual annotation

Copyright L Joskowicz 2010

MIS validation ground-truth generation

centerlines

(manual)

contours

(manual)

Partial Volume

Segmentation

Reference

standard

from 3PVS

Copyright L Joskowicz 2010

Measurements characteristics

bull Ground-truth not know

bull Repeatability intra-observer variability

bull Reliability inter observer variability

bull Measure correlation between repeated

measurements

bull Intrinsic uncertainty about the tumor volume

bull Accuracy for clinical significance is unknown

Inaccuracy gt Uncertainty

bull Uncertainty may not be improved

bull Goal improve accuracy to obtain

Inaccuracy vs Uncertainty

Results can be

meaningless

Inaccuracy lt Uncertainty

In summaryhellip

bull Medical image segmentation is on the risehellip

bull Basis of many clinical applications

bull Validation is a MUST

bull Many methods and approaches ndash do not re-invent

the wheel

bull I expect patient-specific 3D models to be in the

clinical mainstream in 3-5 years

Copyright L Joskowicz 2011

Segmentation challenges

bull Rigorous quantification and evaluation of

segmentation algorithms performance

bull Shape priors for pathologies

bull Incorporation of functional information from

diffusionperfusion MRI fMRI PET into

segmentation algorithms

Copyright L Joskowicz 2011

Summary (1)

bull Patient-specific anatomy model creation is

currently a major bottleneck in many clinical

applications

bull Automatic anatomical segmentation is essential for

model creation

bull Current tools are limited in scope and coverage

bull Clinical use requires the elimination of the

technician ndash model generation by the physician

Summary (2) bull Great opportunity for the development and

incorporation of anatomy modeling tools in

commercial platforms

bull Growing need and variety of users for anatomical

models

ndash Training simulators surgery rehersal

ndash Intraoperative guidance

ndash Computer Aided Radiology

bull Service providers -- shifting paradigms

Page 38: Recent clinical advances and applications for medical ...crl.med.harvard.edu/research/MICCAI_Tutorial/MICCAI11-tutorial... · Recent clinical advances and applications for medical

MIS validation observers variability

bull Intra-observer variability determine how much

variation there is when a single observer

produced the ground-truth segmentation

repeat 5 times the segmentation

bull Inter-observer variability determine how much

variation there is between multiple observers that

produced the ground-truth segmentation

ask 3 radiologists to do the

segmentation

bull Observer expertise and frequency variability

Copyright L Joskowicz 2010

Study [Weltens 2001]

bull Axial MRI slice

bull Nine independent

observers

bull Repeated delineations

bull Interintra observer

variability 30

Key issue fuzzy tumor boundaries

Inter observer variability

Intra-observer variability

8

50

Carotid Lumen Segmentation and Stenosis

Grading Challenge -- The MIDAS Journal

Copyright L Joskowicz 2010

MIS validation manual annotation

Copyright L Joskowicz 2010

MIS validation ground-truth generation

centerlines

(manual)

contours

(manual)

Partial Volume

Segmentation

Reference

standard

from 3PVS

Copyright L Joskowicz 2010

Measurements characteristics

bull Ground-truth not know

bull Repeatability intra-observer variability

bull Reliability inter observer variability

bull Measure correlation between repeated

measurements

bull Intrinsic uncertainty about the tumor volume

bull Accuracy for clinical significance is unknown

Inaccuracy gt Uncertainty

bull Uncertainty may not be improved

bull Goal improve accuracy to obtain

Inaccuracy vs Uncertainty

Results can be

meaningless

Inaccuracy lt Uncertainty

In summaryhellip

bull Medical image segmentation is on the risehellip

bull Basis of many clinical applications

bull Validation is a MUST

bull Many methods and approaches ndash do not re-invent

the wheel

bull I expect patient-specific 3D models to be in the

clinical mainstream in 3-5 years

Copyright L Joskowicz 2011

Segmentation challenges

bull Rigorous quantification and evaluation of

segmentation algorithms performance

bull Shape priors for pathologies

bull Incorporation of functional information from

diffusionperfusion MRI fMRI PET into

segmentation algorithms

Copyright L Joskowicz 2011

Summary (1)

bull Patient-specific anatomy model creation is

currently a major bottleneck in many clinical

applications

bull Automatic anatomical segmentation is essential for

model creation

bull Current tools are limited in scope and coverage

bull Clinical use requires the elimination of the

technician ndash model generation by the physician

Summary (2) bull Great opportunity for the development and

incorporation of anatomy modeling tools in

commercial platforms

bull Growing need and variety of users for anatomical

models

ndash Training simulators surgery rehersal

ndash Intraoperative guidance

ndash Computer Aided Radiology

bull Service providers -- shifting paradigms

Page 39: Recent clinical advances and applications for medical ...crl.med.harvard.edu/research/MICCAI_Tutorial/MICCAI11-tutorial... · Recent clinical advances and applications for medical

Study [Weltens 2001]

bull Axial MRI slice

bull Nine independent

observers

bull Repeated delineations

bull Interintra observer

variability 30

Key issue fuzzy tumor boundaries

Inter observer variability

Intra-observer variability

8

50

Carotid Lumen Segmentation and Stenosis

Grading Challenge -- The MIDAS Journal

Copyright L Joskowicz 2010

MIS validation manual annotation

Copyright L Joskowicz 2010

MIS validation ground-truth generation

centerlines

(manual)

contours

(manual)

Partial Volume

Segmentation

Reference

standard

from 3PVS

Copyright L Joskowicz 2010

Measurements characteristics

bull Ground-truth not know

bull Repeatability intra-observer variability

bull Reliability inter observer variability

bull Measure correlation between repeated

measurements

bull Intrinsic uncertainty about the tumor volume

bull Accuracy for clinical significance is unknown

Inaccuracy gt Uncertainty

bull Uncertainty may not be improved

bull Goal improve accuracy to obtain

Inaccuracy vs Uncertainty

Results can be

meaningless

Inaccuracy lt Uncertainty

In summaryhellip

bull Medical image segmentation is on the risehellip

bull Basis of many clinical applications

bull Validation is a MUST

bull Many methods and approaches ndash do not re-invent

the wheel

bull I expect patient-specific 3D models to be in the

clinical mainstream in 3-5 years

Copyright L Joskowicz 2011

Segmentation challenges

bull Rigorous quantification and evaluation of

segmentation algorithms performance

bull Shape priors for pathologies

bull Incorporation of functional information from

diffusionperfusion MRI fMRI PET into

segmentation algorithms

Copyright L Joskowicz 2011

Summary (1)

bull Patient-specific anatomy model creation is

currently a major bottleneck in many clinical

applications

bull Automatic anatomical segmentation is essential for

model creation

bull Current tools are limited in scope and coverage

bull Clinical use requires the elimination of the

technician ndash model generation by the physician

Summary (2) bull Great opportunity for the development and

incorporation of anatomy modeling tools in

commercial platforms

bull Growing need and variety of users for anatomical

models

ndash Training simulators surgery rehersal

ndash Intraoperative guidance

ndash Computer Aided Radiology

bull Service providers -- shifting paradigms

Page 40: Recent clinical advances and applications for medical ...crl.med.harvard.edu/research/MICCAI_Tutorial/MICCAI11-tutorial... · Recent clinical advances and applications for medical

Intra-observer variability

8

50

Carotid Lumen Segmentation and Stenosis

Grading Challenge -- The MIDAS Journal

Copyright L Joskowicz 2010

MIS validation manual annotation

Copyright L Joskowicz 2010

MIS validation ground-truth generation

centerlines

(manual)

contours

(manual)

Partial Volume

Segmentation

Reference

standard

from 3PVS

Copyright L Joskowicz 2010

Measurements characteristics

bull Ground-truth not know

bull Repeatability intra-observer variability

bull Reliability inter observer variability

bull Measure correlation between repeated

measurements

bull Intrinsic uncertainty about the tumor volume

bull Accuracy for clinical significance is unknown

Inaccuracy gt Uncertainty

bull Uncertainty may not be improved

bull Goal improve accuracy to obtain

Inaccuracy vs Uncertainty

Results can be

meaningless

Inaccuracy lt Uncertainty

In summaryhellip

bull Medical image segmentation is on the risehellip

bull Basis of many clinical applications

bull Validation is a MUST

bull Many methods and approaches ndash do not re-invent

the wheel

bull I expect patient-specific 3D models to be in the

clinical mainstream in 3-5 years

Copyright L Joskowicz 2011

Segmentation challenges

bull Rigorous quantification and evaluation of

segmentation algorithms performance

bull Shape priors for pathologies

bull Incorporation of functional information from

diffusionperfusion MRI fMRI PET into

segmentation algorithms

Copyright L Joskowicz 2011

Summary (1)

bull Patient-specific anatomy model creation is

currently a major bottleneck in many clinical

applications

bull Automatic anatomical segmentation is essential for

model creation

bull Current tools are limited in scope and coverage

bull Clinical use requires the elimination of the

technician ndash model generation by the physician

Summary (2) bull Great opportunity for the development and

incorporation of anatomy modeling tools in

commercial platforms

bull Growing need and variety of users for anatomical

models

ndash Training simulators surgery rehersal

ndash Intraoperative guidance

ndash Computer Aided Radiology

bull Service providers -- shifting paradigms

Page 41: Recent clinical advances and applications for medical ...crl.med.harvard.edu/research/MICCAI_Tutorial/MICCAI11-tutorial... · Recent clinical advances and applications for medical

Carotid Lumen Segmentation and Stenosis

Grading Challenge -- The MIDAS Journal

Copyright L Joskowicz 2010

MIS validation manual annotation

Copyright L Joskowicz 2010

MIS validation ground-truth generation

centerlines

(manual)

contours

(manual)

Partial Volume

Segmentation

Reference

standard

from 3PVS

Copyright L Joskowicz 2010

Measurements characteristics

bull Ground-truth not know

bull Repeatability intra-observer variability

bull Reliability inter observer variability

bull Measure correlation between repeated

measurements

bull Intrinsic uncertainty about the tumor volume

bull Accuracy for clinical significance is unknown

Inaccuracy gt Uncertainty

bull Uncertainty may not be improved

bull Goal improve accuracy to obtain

Inaccuracy vs Uncertainty

Results can be

meaningless

Inaccuracy lt Uncertainty

In summaryhellip

bull Medical image segmentation is on the risehellip

bull Basis of many clinical applications

bull Validation is a MUST

bull Many methods and approaches ndash do not re-invent

the wheel

bull I expect patient-specific 3D models to be in the

clinical mainstream in 3-5 years

Copyright L Joskowicz 2011

Segmentation challenges

bull Rigorous quantification and evaluation of

segmentation algorithms performance

bull Shape priors for pathologies

bull Incorporation of functional information from

diffusionperfusion MRI fMRI PET into

segmentation algorithms

Copyright L Joskowicz 2011

Summary (1)

bull Patient-specific anatomy model creation is

currently a major bottleneck in many clinical

applications

bull Automatic anatomical segmentation is essential for

model creation

bull Current tools are limited in scope and coverage

bull Clinical use requires the elimination of the

technician ndash model generation by the physician

Summary (2) bull Great opportunity for the development and

incorporation of anatomy modeling tools in

commercial platforms

bull Growing need and variety of users for anatomical

models

ndash Training simulators surgery rehersal

ndash Intraoperative guidance

ndash Computer Aided Radiology

bull Service providers -- shifting paradigms

Page 42: Recent clinical advances and applications for medical ...crl.med.harvard.edu/research/MICCAI_Tutorial/MICCAI11-tutorial... · Recent clinical advances and applications for medical

MIS validation manual annotation

Copyright L Joskowicz 2010

MIS validation ground-truth generation

centerlines

(manual)

contours

(manual)

Partial Volume

Segmentation

Reference

standard

from 3PVS

Copyright L Joskowicz 2010

Measurements characteristics

bull Ground-truth not know

bull Repeatability intra-observer variability

bull Reliability inter observer variability

bull Measure correlation between repeated

measurements

bull Intrinsic uncertainty about the tumor volume

bull Accuracy for clinical significance is unknown

Inaccuracy gt Uncertainty

bull Uncertainty may not be improved

bull Goal improve accuracy to obtain

Inaccuracy vs Uncertainty

Results can be

meaningless

Inaccuracy lt Uncertainty

In summaryhellip

bull Medical image segmentation is on the risehellip

bull Basis of many clinical applications

bull Validation is a MUST

bull Many methods and approaches ndash do not re-invent

the wheel

bull I expect patient-specific 3D models to be in the

clinical mainstream in 3-5 years

Copyright L Joskowicz 2011

Segmentation challenges

bull Rigorous quantification and evaluation of

segmentation algorithms performance

bull Shape priors for pathologies

bull Incorporation of functional information from

diffusionperfusion MRI fMRI PET into

segmentation algorithms

Copyright L Joskowicz 2011

Summary (1)

bull Patient-specific anatomy model creation is

currently a major bottleneck in many clinical

applications

bull Automatic anatomical segmentation is essential for

model creation

bull Current tools are limited in scope and coverage

bull Clinical use requires the elimination of the

technician ndash model generation by the physician

Summary (2) bull Great opportunity for the development and

incorporation of anatomy modeling tools in

commercial platforms

bull Growing need and variety of users for anatomical

models

ndash Training simulators surgery rehersal

ndash Intraoperative guidance

ndash Computer Aided Radiology

bull Service providers -- shifting paradigms

Page 43: Recent clinical advances and applications for medical ...crl.med.harvard.edu/research/MICCAI_Tutorial/MICCAI11-tutorial... · Recent clinical advances and applications for medical

MIS validation ground-truth generation

centerlines

(manual)

contours

(manual)

Partial Volume

Segmentation

Reference

standard

from 3PVS

Copyright L Joskowicz 2010

Measurements characteristics

bull Ground-truth not know

bull Repeatability intra-observer variability

bull Reliability inter observer variability

bull Measure correlation between repeated

measurements

bull Intrinsic uncertainty about the tumor volume

bull Accuracy for clinical significance is unknown

Inaccuracy gt Uncertainty

bull Uncertainty may not be improved

bull Goal improve accuracy to obtain

Inaccuracy vs Uncertainty

Results can be

meaningless

Inaccuracy lt Uncertainty

In summaryhellip

bull Medical image segmentation is on the risehellip

bull Basis of many clinical applications

bull Validation is a MUST

bull Many methods and approaches ndash do not re-invent

the wheel

bull I expect patient-specific 3D models to be in the

clinical mainstream in 3-5 years

Copyright L Joskowicz 2011

Segmentation challenges

bull Rigorous quantification and evaluation of

segmentation algorithms performance

bull Shape priors for pathologies

bull Incorporation of functional information from

diffusionperfusion MRI fMRI PET into

segmentation algorithms

Copyright L Joskowicz 2011

Summary (1)

bull Patient-specific anatomy model creation is

currently a major bottleneck in many clinical

applications

bull Automatic anatomical segmentation is essential for

model creation

bull Current tools are limited in scope and coverage

bull Clinical use requires the elimination of the

technician ndash model generation by the physician

Summary (2) bull Great opportunity for the development and

incorporation of anatomy modeling tools in

commercial platforms

bull Growing need and variety of users for anatomical

models

ndash Training simulators surgery rehersal

ndash Intraoperative guidance

ndash Computer Aided Radiology

bull Service providers -- shifting paradigms

Page 44: Recent clinical advances and applications for medical ...crl.med.harvard.edu/research/MICCAI_Tutorial/MICCAI11-tutorial... · Recent clinical advances and applications for medical

Measurements characteristics

bull Ground-truth not know

bull Repeatability intra-observer variability

bull Reliability inter observer variability

bull Measure correlation between repeated

measurements

bull Intrinsic uncertainty about the tumor volume

bull Accuracy for clinical significance is unknown

Inaccuracy gt Uncertainty

bull Uncertainty may not be improved

bull Goal improve accuracy to obtain

Inaccuracy vs Uncertainty

Results can be

meaningless

Inaccuracy lt Uncertainty

In summaryhellip

bull Medical image segmentation is on the risehellip

bull Basis of many clinical applications

bull Validation is a MUST

bull Many methods and approaches ndash do not re-invent

the wheel

bull I expect patient-specific 3D models to be in the

clinical mainstream in 3-5 years

Copyright L Joskowicz 2011

Segmentation challenges

bull Rigorous quantification and evaluation of

segmentation algorithms performance

bull Shape priors for pathologies

bull Incorporation of functional information from

diffusionperfusion MRI fMRI PET into

segmentation algorithms

Copyright L Joskowicz 2011

Summary (1)

bull Patient-specific anatomy model creation is

currently a major bottleneck in many clinical

applications

bull Automatic anatomical segmentation is essential for

model creation

bull Current tools are limited in scope and coverage

bull Clinical use requires the elimination of the

technician ndash model generation by the physician

Summary (2) bull Great opportunity for the development and

incorporation of anatomy modeling tools in

commercial platforms

bull Growing need and variety of users for anatomical

models

ndash Training simulators surgery rehersal

ndash Intraoperative guidance

ndash Computer Aided Radiology

bull Service providers -- shifting paradigms

Page 45: Recent clinical advances and applications for medical ...crl.med.harvard.edu/research/MICCAI_Tutorial/MICCAI11-tutorial... · Recent clinical advances and applications for medical

bull Intrinsic uncertainty about the tumor volume

bull Accuracy for clinical significance is unknown

Inaccuracy gt Uncertainty

bull Uncertainty may not be improved

bull Goal improve accuracy to obtain

Inaccuracy vs Uncertainty

Results can be

meaningless

Inaccuracy lt Uncertainty

In summaryhellip

bull Medical image segmentation is on the risehellip

bull Basis of many clinical applications

bull Validation is a MUST

bull Many methods and approaches ndash do not re-invent

the wheel

bull I expect patient-specific 3D models to be in the

clinical mainstream in 3-5 years

Copyright L Joskowicz 2011

Segmentation challenges

bull Rigorous quantification and evaluation of

segmentation algorithms performance

bull Shape priors for pathologies

bull Incorporation of functional information from

diffusionperfusion MRI fMRI PET into

segmentation algorithms

Copyright L Joskowicz 2011

Summary (1)

bull Patient-specific anatomy model creation is

currently a major bottleneck in many clinical

applications

bull Automatic anatomical segmentation is essential for

model creation

bull Current tools are limited in scope and coverage

bull Clinical use requires the elimination of the

technician ndash model generation by the physician

Summary (2) bull Great opportunity for the development and

incorporation of anatomy modeling tools in

commercial platforms

bull Growing need and variety of users for anatomical

models

ndash Training simulators surgery rehersal

ndash Intraoperative guidance

ndash Computer Aided Radiology

bull Service providers -- shifting paradigms

Page 46: Recent clinical advances and applications for medical ...crl.med.harvard.edu/research/MICCAI_Tutorial/MICCAI11-tutorial... · Recent clinical advances and applications for medical

In summaryhellip

bull Medical image segmentation is on the risehellip

bull Basis of many clinical applications

bull Validation is a MUST

bull Many methods and approaches ndash do not re-invent

the wheel

bull I expect patient-specific 3D models to be in the

clinical mainstream in 3-5 years

Copyright L Joskowicz 2011

Segmentation challenges

bull Rigorous quantification and evaluation of

segmentation algorithms performance

bull Shape priors for pathologies

bull Incorporation of functional information from

diffusionperfusion MRI fMRI PET into

segmentation algorithms

Copyright L Joskowicz 2011

Summary (1)

bull Patient-specific anatomy model creation is

currently a major bottleneck in many clinical

applications

bull Automatic anatomical segmentation is essential for

model creation

bull Current tools are limited in scope and coverage

bull Clinical use requires the elimination of the

technician ndash model generation by the physician

Summary (2) bull Great opportunity for the development and

incorporation of anatomy modeling tools in

commercial platforms

bull Growing need and variety of users for anatomical

models

ndash Training simulators surgery rehersal

ndash Intraoperative guidance

ndash Computer Aided Radiology

bull Service providers -- shifting paradigms

Page 47: Recent clinical advances and applications for medical ...crl.med.harvard.edu/research/MICCAI_Tutorial/MICCAI11-tutorial... · Recent clinical advances and applications for medical

Segmentation challenges

bull Rigorous quantification and evaluation of

segmentation algorithms performance

bull Shape priors for pathologies

bull Incorporation of functional information from

diffusionperfusion MRI fMRI PET into

segmentation algorithms

Copyright L Joskowicz 2011

Summary (1)

bull Patient-specific anatomy model creation is

currently a major bottleneck in many clinical

applications

bull Automatic anatomical segmentation is essential for

model creation

bull Current tools are limited in scope and coverage

bull Clinical use requires the elimination of the

technician ndash model generation by the physician

Summary (2) bull Great opportunity for the development and

incorporation of anatomy modeling tools in

commercial platforms

bull Growing need and variety of users for anatomical

models

ndash Training simulators surgery rehersal

ndash Intraoperative guidance

ndash Computer Aided Radiology

bull Service providers -- shifting paradigms

Page 48: Recent clinical advances and applications for medical ...crl.med.harvard.edu/research/MICCAI_Tutorial/MICCAI11-tutorial... · Recent clinical advances and applications for medical

Summary (1)

bull Patient-specific anatomy model creation is

currently a major bottleneck in many clinical

applications

bull Automatic anatomical segmentation is essential for

model creation

bull Current tools are limited in scope and coverage

bull Clinical use requires the elimination of the

technician ndash model generation by the physician

Summary (2) bull Great opportunity for the development and

incorporation of anatomy modeling tools in

commercial platforms

bull Growing need and variety of users for anatomical

models

ndash Training simulators surgery rehersal

ndash Intraoperative guidance

ndash Computer Aided Radiology

bull Service providers -- shifting paradigms

Page 49: Recent clinical advances and applications for medical ...crl.med.harvard.edu/research/MICCAI_Tutorial/MICCAI11-tutorial... · Recent clinical advances and applications for medical

Summary (2) bull Great opportunity for the development and

incorporation of anatomy modeling tools in

commercial platforms

bull Growing need and variety of users for anatomical

models

ndash Training simulators surgery rehersal

ndash Intraoperative guidance

ndash Computer Aided Radiology

bull Service providers -- shifting paradigms