© fraunhofer mevis 2015-07-13, heidelberg collaboratory for image processing frank heckel, phd...
TRANSCRIPT
© Fraunhofer MEVIS
2015-07-13, Heidelberg Collaboratory for Image Processing
Frank Heckel, PhD
Software Support for Oncological Therapy Response Assessment
© Fraunhofer MEVIS
2 / 52
FRAUNHOFER MEVIS
Bremen
Additional employees in Berlin, Leipzig, Heidelberg & Nijmegen
Lübeck
© Fraunhofer MEVIS
3 / 52
Largest organization for applied research in Europe Areas of research: life science, communication,
mobility, security, energy, environment 66 institutes, 24.000 employees 2.0 billion EUR research budget,
>70% from industry and public agencies
Fraunhofer-Gesellschaft
Basic Funding
Industry
Public Research
© Fraunhofer MEVIS
4 / 52
67 institutes in Germany
Institutes
Branches, Working Groups, Application Centers
Karlsruhe
DarmstadtWürzburg
Jena
Stuttgart
Duisburg
Oberhausen
Nuthetal
Dortmund
München
Saarbrücken
St. Ingbert
Magdeburg
Halle
Dresden
Leipzig
Ilmenau
Cottbus
Braunschweig
BerlinPotsdam
Teltow
Aachen
Schmallenberg
Sankt Augustin
Erlangen
Nürnberg
Freising
Holzkirchen
Pfinztal
Freiburg
Efringen-Kirchen
RostockItzehoe
Hannover
Bremen
Euskirchen Chemnitz
WertheimKaiserslautern
Paderborn
Schkopau
LübeckBremerhaven
Fraunhofer-Gesellschaft
© Fraunhofer MEVIS
5 / 52
Fraunhofer MEVIS
Non-profit Commercial(~100 employees) (~150 employees)
51%
Institute forMedical Image Computing
Bremen (since 01/2009)
Project Group Image Registration
Lübeck (since 04/2010)
MeVis BreastCare GmbH & Co. KG
Bremen (since 10/2001)
MeVis Medical Solutions AG
Bremen (since 08/2007)
© Fraunhofer MEVIS
6 / 52
Image acquisition and reconstruction Image computing, analysis and visualization Modelling and simulation Application, workflow and usability engineering
Computer assistance for image-based, personalized diagnosis and therapy
Solutions for clinical problems
© Fraunhofer MEVIS
7 / 52
Competences
Diagnosis
Clinical Workflow
Early Detection Diagnostic Planning Procedure MonitoringTherapy
Organs
Liver
Lung
Breast
Brain
Heart/Vessels
Bones/Joints
Pathologies (Tumor, Inflammation, Degeneration, etc.)
MethodsMeVisLab
Validation
Navigation
Risk analysis
Visualization
Quantification
Segmentation
Registration
Modeling/SimulationImaging/Modality
© Fraunhofer MEVIS
8 / 52
Organization Chart
Institute DirectorsProf. Kikinis, Prof. Hahn
Advisory Board
Extended Committee
Steering Committee plus representatives
for:
Software/IT, QA, Employees,
Equal Rights, WTR, PR
Steering Committee
Prof. Kikinis, Prof. Hahn,
T. Forstmann, Prof. Preußer,
Prof. Günther, Prof. Modersitzki, Dr.
Heldmann, Dr. Olesch,Dr. Papenberg, Dr.
Kraß, Dr. Lang, Dr. Prause
AdministrationT. Forstmann
© Fraunhofer MEVIS
9 / 52
Organization of Work
Team-oriented Open-minded Self-organized Flexible Adaptive
© Fraunhofer MEVIS
10 / 52
Certification
Certificate for quality assurance Introduction and application of a quality
management system in compliance with EN ISO 9001 & EN ISO 13485 (medical devices) Since 2005 in Bremen Since 2012 in Lübeck
Scope: Research and development for computer assistance
of medical diagnosis and therapy Development and production of software for
medical products
© Fraunhofer MEVIS
11 / 52
University of Bremen Mathematics (H.-O. Peitgen, until Sep 2012) Medical Image Computing (R. Kikinis, since Jan 2014) MR Imaging & Physics (M. Günther)
Jacobs University Bremen Analysis & Visualization (H. Hahn) Modeling & Simulation (T. Preußer)
University of Lübeck Mathematics & MEVIS Project Group
(B. Fischer †, J. Modersitzki)
University of Nijmegen Computer-Aided Detection & Diagnosis
(N. Karssemeijer, B. van Ginneken)
Links to Universities
© Fraunhofer MEVIS
12 / 52
INNOVATION CENTER COMPUTER ASSISTED SURGERY (ICCAS)
© Fraunhofer MEVIS
13 / 52
Innovation Center Computer Assisted Surgery (ICCAS)
Part of medical faculty Universität Leipzig
Clinical disciplines: ENT-surgery, Heart surgery, Neurosurgery
© Fraunhofer MEVIS
14 / 52
ICCAS Research Areas
STD
MAI – Model-based automation and integration, DPM – Digital patient model, STD - Standardization
© Fraunhofer MEVIS
15 / 52
Research Area: Model-based Automation and Integration
Navigationdata
Model visualisations
Systemmonitoring
Tracked ultrasound probe
Augmented Reality for microscopes
Ultrasound imaging
Information and communication technology in the OR
Head: Prof. Thomas Neumuth
© Fraunhofer MEVIS
16 / 52
Research Area: Model-based Automation and Integration
patient surgeon
SurgicalWorkflow
HMI Imaging Navigation
© Fraunhofer MEVIS
17 / 52
Research Area: Model-based Automation and Integration
Integration into therapeutic process
Ressource monitoringProcess monitoring
Workflow management
Data consolidation and integration
© Fraunhofer MEVIS
18 / 52
Research Area: Digital Patient Models
Head: Dr. Kerstin Denecke
© Fraunhofer MEVIS
19 / 52
Research Area: Standardization
Head: Prof. Heinz Lemke
© Fraunhofer MEVIS
20 / 52
Research Area: Image-guided Interventions
Head (and Insitute Director): Prof. Andreas Melzer
© Fraunhofer MEVIS
21 / 52
ONCOLOGICAL THERAPY RESPONSE ASSESSMENT
© Fraunhofer MEVIS
22 / 52
Overview
Background Semi-Automatic Segmentation Segmentation Editing Partial Volume Correction The Ground Truth Problem Workflow Aspects
© Fraunhofer MEVIS
23 / 52
Background
Cause for 13% of all deaths worldwide Every 2nd man gets cancer every 4th dies
Treatment examples: Surgery Radiotherapy Radiofrequency ablation and …
Chemotherapy Lung nodules, metastases, enlarged lymph
nodes Systemic treatment Severe side effects Different agents
Cancer and Chemotherapy
© Fraunhofer MEVIS
24 / 52
BackgroundCT-Based Follow-Up Examination
Baseline• Find tumors• Identify target lesions• Measure target
lesions • Reporting
1st Follow-Up• Find target lesions• Measure response• Look for new lesions• Reporting
Additional Follow-Ups
3-6
months
© Fraunhofer MEVIS
25 / 52
Background
Change in tumor size is an important criterion RECIST1 1.1: Sum of maximum diameters of target lesions
Relative change
Volume is a more accurate measure Many tumors grow/shrink irregularly in 3D Requires appropriate segmentation Progress/response not defined Not used in clinical routine
Oncological Therapy Response Monitoring
1 RECIST: Response Evaluation Criteria In Solid Tumors
Complete Response
Partial Response
Stable Disease
Progressive Disease
Disappearance < -30% -30% – 20% > +20%
© Fraunhofer MEVIS
26 / 52
BackgroundDiameter vs. Volume
completeresponse
no longer visible
partial response
< -30%
> + 73%
stabledisease
Small change
< -66%
progressivedisease
> +20%
Classification
Diameter
Volume
© Fraunhofer MEVIS
27 / 52
Background
Simulated example: Measured 2% change Reality: 26% change (roughly double volume!)
Robustness of Diameter Measurement
© Fraunhofer MEVIS
28 / 52
The Segmentation Problem
Ultimate Goal: Automatic segmentation for a wide range of objects Reproducible results with no effort for the user Solutions for specific purposes Might fail (low contrast, noise, biological
variability) Unsolved or insufficient for many real-world
problems Alternatives:
Manual segmentation Semi-automatic or interactive tools (Semi-)automatic algorithm followed by manual
correction Drawback: Variability due to different inputs or
judgment
© Fraunhofer MEVIS
29 / 52
Semi-Automatic Segmentation
Familiar user Interaction: draw the (maximum) diameter
Core method: “Smart Opening”1
Region Growing Erosion Dilation Refinement
Specific variation for lung nodules, liver metastases and lymph nodes2
For lymph nodes a spiral-scanning solution has been developed as well3
1 Kuhnigk et al., IEEE TMI, 25(4), 20062 Moltz et al., IEEE Journal of Selected Topics in Signal Processing, 3(1), 20093 Wang et al., SPIE Medical Imaging, 2012
© Fraunhofer MEVIS
30 / 52
Semi-Automatic SegmentationExamples for Challenging Lung Nodules
© Fraunhofer MEVIS
31 / 52
Semi-Automatic Segmentation
Positive examples:
Negative examples:
Examples for Challenging Liver Metastases
© Fraunhofer MEVIS
32 / 52
Semi-Automatic Segmentation
Smart Opening (top) vs. Spiral Scanning (bottom)
Examples for Challenging Lymph Nodes
© Fraunhofer MEVIS
33 / 52
Semi-Automatic Segmentation
Lung: LIDC-Data (674 cases (solid nodules), 4 reference segmentations)
Liver: MDS-Data (371 cases, 1 reference segmentation)
Evaluation
Volume overlap
Hausdorff distance
Computation time
Lung 68,3% 2,46 mm 0,41 s
Liver 62,6% 4,20 mm 0,75 s
© Fraunhofer MEVIS
34 / 52
Semi-Automatic Segmentation
Clinical Evaluation: Amount of Lesions that have not been manually corrected Lung Liver
Evaluation
2009 2010 20120
10
20
30
40
50
60
70
80
90
100
2011 20120
10
20
30
40
50
60
70
80
90
100
© Fraunhofer MEVIS
35 / 52
2008 2010 2010 2010 2011 2012 20120
10
20
30
40
50
60
70
80
90
100
Semi-Automatic Segmentation
Clinical Evaluation: Amount of Lesions that have not been manually corrected Lymph nodes
Evaluation
© Fraunhofer MEVIS
36 / 52
Segmentation Editing
Most existing methods are low-level and unintuitive in 3D High-level correction has not received much attention in research
Segmentation Algorithm
Start
Semi-automatic
AutomaticSegmentation
ResultSatisfying?
Initial Algorithm allows
modification?
SegmentationEditing Algorithmno no
Stop
yes yes
Segmentation Algorithm
InteractiveSegmentation
ResultSatisfying? Stop
yes
no
© Fraunhofer MEVIS
37 / 52
Segmentation EditingSketch-Based Editing in 2D
add
remove
add + remove
replace
© Fraunhofer MEVIS
38 / 52
Segmentation Editing
Image-based method (→ shortest path)
Image-independent method (→ RBF-based 3D object reconstruction)
3D Extrapolation
Heckel et al., Computer Graphics Forum, 32(8), 2013
© Fraunhofer MEVIS
39 / 52
Segmentation Editing
131 representative tumor segmentations in CT (lung nodules, liver metastases, lymph nodes)
5 radiologists with different level of experience
Editing rating score:
Qualitative Evaluation
𝑟 edit=1𝑁
¿
Heckel et al., SPIE Journal of Medical Imaging, 1(3), 2014
© Fraunhofer MEVIS
40 / 52
Segmentation EditingQuantitative Evaluation
Analyze quality over time Editing quality score:
© Fraunhofer MEVIS
41 / 52
Segmentation Editing
Problem: High effort and bad reproducibility of user studies Idea: Replace user by a simulation Benefits:
Objective and reproducible validation Objective comparison Improved regression testing Better parameter tuning
Simulation-Based Evaluation
IntermediateSegmentation
Target Segmentation
Segmentation Editing
Satisfying?
User
Validationno
yes
Stop
Start
Control flow
Data flow
User Input
Previous Inputs
IntermediateSegmentation
Reference Segmentation
Segmentation Editing
Satisfying?
Simulation
Validationno
yes
Stop
Start
Control flow
Data flow
User Input
Previous Inputs
© Fraunhofer MEVIS
42 / 52
Segmentation Editing
Step 1: Find most probably corrected 3D error Step 2: Select slice and view where the error is most probably
corrected Step 3: Generate user-input for sketching Step 4: Apply editing algorithm
Simulation-Based Evaluation
Heckel et al., Scandinavian Conferences on Image Analysis, 2013
© Fraunhofer MEVIS
43 / 52
Segmentation EditingSimulation-Based Evaluation
© Fraunhofer MEVIS
44 / 52
Partial Volume Correction
Smoothing effect caused by limited spatial resolution (of CT) Ill-defined border between tumor and healthy tissue, making
segmentation an ill-defined problem Could cause significant differences in size measurements
The Partial Volume Effect
28.4 ml(-27.5%)
39.2 ml 56.8 ml(+44.9%)
© Fraunhofer MEVIS
45 / 52
Partial Volume Correction
Spatial subdivision into spherical sectors to cover different tissues
Define reference tissue values inside and outside of the object ( and to) per sector
For each sector : compute the weight w of each partial volume voxel
Method
1.0
0.0
0.5
0.75
0.25
𝑤 (𝑉 )=𝑡𝑜 𝑠−𝑣
𝑡𝑜 𝑠− 𝑡𝑖 𝑠
,𝑉∈𝑃 𝑖𝑠∪𝑃𝑜𝑠
𝑉𝑜𝑙𝐿=∑𝑉 ∈𝐿
𝑤 (𝑉 )𝑉𝑜𝑙𝑉71.1 ml
70.8 mlHeckel et al., IEEE TMI, 33(2), 2014
© Fraunhofer MEVIS
46 / 52
Partial Volume CorrectionSoftware Phantom Results
© Fraunhofer MEVIS
47 / 52
Partial Volume CorrectionHardware Phantom Results
© Fraunhofer MEVIS
48 / 52
Partial Volume CorrectionMulti-Reader Data Results
© Fraunhofer MEVIS
49 / 52
The Ground Truth Problem
Expert segmentations differ significantly Variability depends on several aspects
(lesion size, contrast, partial volume effects, interpretation, …) We need to consider n>1 reference segmentations Who are experts? Only clinicians?
There is no „Ground Truth“!
Jan Moltz, PhD Thesis, Jacobs University Bremen, 2013
© Fraunhofer MEVIS
50 / 52
The Ground Truth ProblemWhat is a „good“ segmentation result?
© Fraunhofer MEVIS
51 / 52
Workflow Aspects
CAD Lesion Matching Visualization Reporting
Schwier et al., IJCARS, 6(6), 2011
Schwier et al., CARS 2009 Jan Moltz et al., ISBI, 2009
© Fraunhofer MEVIS
52 / 52
Workflow AspectsPrototyping
© Fraunhofer MEVIS
54 / 52