may 2012 update may 10, 2012 andrew j. buckler, ms principal investigator, qi-bench
DESCRIPTION
May 2012 Update May 10, 2012 Andrew J. Buckler, MS Principal Investigator, QI-Bench. With Funding Support provided by National Institute of Standards and Technology. Resources are needed to address widening gap in imaging capability as practiced vs. capability of modern medicine. 2. 2. 2. - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: May 2012 Update May 10, 2012 Andrew J. Buckler, MS Principal Investigator, QI-Bench](https://reader035.vdocuments.site/reader035/viewer/2022062810/56815c57550346895dca5a2f/html5/thumbnails/1.jpg)
May 2012 UpdateMay 10, 2012
Andrew J. Buckler, MSPrincipal Investigator,
QI-Bench
WITH FUNDING SUPPORT
PROVIDED BY NATIONAL
INSTITUTE OF STANDARDS AND
TECHNOLOGY
![Page 2: May 2012 Update May 10, 2012 Andrew J. Buckler, MS Principal Investigator, QI-Bench](https://reader035.vdocuments.site/reader035/viewer/2022062810/56815c57550346895dca5a2f/html5/thumbnails/2.jpg)
Resources are needed to address widening gap in imaging capability as practiced vs. capability of modern medicine
2222222
![Page 3: May 2012 Update May 10, 2012 Andrew J. Buckler, MS Principal Investigator, QI-Bench](https://reader035.vdocuments.site/reader035/viewer/2022062810/56815c57550346895dca5a2f/html5/thumbnails/3.jpg)
Example: Beyond Anatomy to Palette of Functional Measures
33333333
glucose metabolism
Biologic Target
bone formation
proliferation
hypoxia
amino acid metabolism
angiogenesis
receptor statusapotosis
18F-FDG18F-NaF
18F-FLT18F-
FACBC
18F-FMISO
18F-XXX
DCE-MRI
PET
18F-FES
![Page 4: May 2012 Update May 10, 2012 Andrew J. Buckler, MS Principal Investigator, QI-Bench](https://reader035.vdocuments.site/reader035/viewer/2022062810/56815c57550346895dca5a2f/html5/thumbnails/4.jpg)
QI-Bench is a resource that may be used by single sponsors, defined-entity consortia, or true open science programs
4444444
![Page 5: May 2012 Update May 10, 2012 Andrew J. Buckler, MS Principal Investigator, QI-Bench](https://reader035.vdocuments.site/reader035/viewer/2022062810/56815c57550346895dca5a2f/html5/thumbnails/5.jpg)
Example: COPD
Either:• To assist individual suppliers
in optimizing their offerings• To assist groups like
COPDgene consortia• To enable open
development such as by QIBA
or flexible mix of these.
55555555
![Page 6: May 2012 Update May 10, 2012 Andrew J. Buckler, MS Principal Investigator, QI-Bench](https://reader035.vdocuments.site/reader035/viewer/2022062810/56815c57550346895dca5a2f/html5/thumbnails/6.jpg)
Data warehouse
QI-Bench is composed of building blocks
6666666
Provenance architecture of Iterate
![Page 7: May 2012 Update May 10, 2012 Andrew J. Buckler, MS Principal Investigator, QI-Bench](https://reader035.vdocuments.site/reader035/viewer/2022062810/56815c57550346895dca5a2f/html5/thumbnails/7.jpg)
Example: DCE-MRI using Patient, Synthetic, and Phantom Data• Curate, maintain and serve
reference data sets• Execute batch runs over multi-
parameter synthetic data• Characterize performance
77777777
Data warehouse
Provenance architecture of Iterate
MD Andersen receives
phantoms
Parallel Imaging procedure
GE (new) scanner
Phantoms are comparable
Ship phantom A to UPENN
Ship phantom B to U Chicago
Imaging procedureSiemens MR
scanners
Imaging procedurePhilips (new)
scanner
Ship phantom BTo UC Davis
Image procedureGE (old) scanners
Ship phantom A toDuke
Imaging procedurePhilips (old)
Ship phantom A and B back to MD
Anderson
![Page 8: May 2012 Update May 10, 2012 Andrew J. Buckler, MS Principal Investigator, QI-Bench](https://reader035.vdocuments.site/reader035/viewer/2022062810/56815c57550346895dca5a2f/html5/thumbnails/8.jpg)
Test bed: CT volumetry method challenge (“3A”)
8888888
1. Median Technologies2. Vital Images, Inc.3. Fraunhofer Mevis4. Siemens5. Moffitt Cancer Center6. Toshiba
Pilot
Pivotal
Investigation 1
Train
Test
Pilot
Pivotal
Investigation
Train
Test
Pilot
Pivotal
Investigation
Train
Test
Pilot
Pivotal
Investigation n
Train
Test
Pr im
ar y
Se co n dar y
• Defined set of data• Defined challenge• Defined test set policy
Some of the Participants7. GE Healthcare8. Icon Medical Imaging9. Columbia University10. INTIO, Inc.11. Vital Images, Inc.
![Page 9: May 2012 Update May 10, 2012 Andrew J. Buckler, MS Principal Investigator, QI-Bench](https://reader035.vdocuments.site/reader035/viewer/2022062810/56815c57550346895dca5a2f/html5/thumbnails/9.jpg)
Broader capability: Systematic qualification of CT volumetry
99999999
1. Median Technologies2. Vital Images, Inc.3. Fraunhofer Mevis4. Siemens5. Moffitt Cancer Center6. Toshiba
Some of the Participants7. GE Healthcare8. Icon Medical Imaging9. Columbia University10. INTIO, Inc.11. Vital Images, Inc. PROFILE Authoring and Testing
Inter-analysis technique (algorithm) variability (3A)
Correlation with clinical endpoints and outcomes (3B)
machine view human expert
view
Transformation
Modality Environment
Therapy Decision
Environment
Patient Patient
Transformation Transformation
FeedbackTherapy-Machine Human Observer
0.0
0.2
0.4
0.6
0.8
1.0
% R
each
ing Pa
rtial R
espo
nse
0 42 84 126 168 210 252 294 336 378
Time on Trial (days)
SLDVolumetric
Survival Plot
Intra- and inter-reader variability (1A)
Minimum detectable biological change (1B)
5 readers, 3 reads
each
Expl
ore
figur
es-
of-m
erit
and
QC
proc
edur
es
![Page 10: May 2012 Update May 10, 2012 Andrew J. Buckler, MS Principal Investigator, QI-Bench](https://reader035.vdocuments.site/reader035/viewer/2022062810/56815c57550346895dca5a2f/html5/thumbnails/10.jpg)
Update: Iterate progress• Added small workflows to Iterate
demonstration server.• Stored and retrieved data on server
via workflows.• Distinction between workflow
execution on server and local workstation.
1010
![Page 11: May 2012 Update May 10, 2012 Andrew J. Buckler, MS Principal Investigator, QI-Bench](https://reader035.vdocuments.site/reader035/viewer/2022062810/56815c57550346895dca5a2f/html5/thumbnails/11.jpg)
Update: Iterate coming attractions• Upgraded base software installation.• New site with workflows for local
workstation.• Server workflows that produce
outputs.
1111
![Page 12: May 2012 Update May 10, 2012 Andrew J. Buckler, MS Principal Investigator, QI-Bench](https://reader035.vdocuments.site/reader035/viewer/2022062810/56815c57550346895dca5a2f/html5/thumbnails/12.jpg)
Analyze performs the statistical analyses…Subject Predicate Object
A Is Patient
A isDiagnosedWith DiseaseA
DiseaseA Is NonSmallLCellLunCancer
A hasClinicalObservation
B
B Is TumorShrinkage
C Is Patient
C hasClinicalObservation
B
D hasClinicalObservation
B
Pazopanib Is TyrosoineKinaseInhibitor
A isTreatedWith Pazopanib
A hasOutcome Death
C hasOutcome Survival
Subject Predicate Object
CT images Tumor
Volumetry analyzes CT
<compliant>LongitudinalVolumetry
estimates TumorSizeChange
TumorSizeChange predicts CytotoxicTreatmentResponse
TyrosoineKinaseInhibitor is CytotoxicTreatment
well-controlled Phase II and III efficacy studies
uses CytotoxicTreatmentResponse
CytotoxicTreatment influences NonSmallCellLungCancer
CT images Thorax
Thorax contains NonSmallCellLungCancer
regulatory drug approval dependsOn PrimaryEndpoint
well-controlled Phase II and III efficacy studies
assess PrimaryEndpoint
CT Volumetry is SurrogateEndpoint for CytotoxicTreatment
1
3
2
1212
![Page 13: May 2012 Update May 10, 2012 Andrew J. Buckler, MS Principal Investigator, QI-Bench](https://reader035.vdocuments.site/reader035/viewer/2022062810/56815c57550346895dca5a2f/html5/thumbnails/13.jpg)
…and adds the results to the knowledgebase (using W3C “best practices” for “relation strength”).
Subject Predicate Object
45324 biasMethod <r script used>
45324 bias <summary statistic>
45324 variabilityMethod <r script used>
45324 variability <summary statistic>
9956 <correlation>Method <r script used>
9956 correlation <summary statistic>
9956 <ROC>Method <r script used>
9956 ROC <summary statistic>
98234 Effect of treatment on true endpoint <value>
98234 Effect of treatment on surrogate endpoint <value>
98234 Effect of surrogate on true endpoint <value>
98234 Effect of treatment on true endpoint relative to that on surrogate endpoint
<value>
Subject Predicate Object
CT images Tumor
Volumetry analyzes CT
<compliant>LongitudinalVolumetry
estimates TumorSizeChange
TumorSizeChange predicts CytotoxicTreatmentResponse
TyrosoineKinaseInhibitor is CytotoxicTreatment
well-controlled Phase II and III efficacy studies
uses CytotoxicTreatmentResponse
CytotoxicTreatment influences NonSmallCellLungCancer
CT images Thorax
Thorax contains NonSmallCellLungCancer
regulatory drug approval dependsOn PrimaryEndpoint
well-controlled Phase II and III efficacy studies
assess PrimaryEndpoint
CT Volumetry is SurrogateEndpoint for CytotoxicTreatment
1
3
2
URI=45324
URI=9956
URI=98234
1313
![Page 14: May 2012 Update May 10, 2012 Andrew J. Buckler, MS Principal Investigator, QI-Bench](https://reader035.vdocuments.site/reader035/viewer/2022062810/56815c57550346895dca5a2f/html5/thumbnails/14.jpg)
1414
Demonstration of statistical analysis using R within Iterate• Obtain data• Produce summary plots and statistics• Perform analyses
Provenance architecture of Iterate
![Page 15: May 2012 Update May 10, 2012 Andrew J. Buckler, MS Principal Investigator, QI-Bench](https://reader035.vdocuments.site/reader035/viewer/2022062810/56815c57550346895dca5a2f/html5/thumbnails/15.jpg)
Conceptual Example• Example data based on QIBA 3A layout• Phantom study
– 5 lesions, each read by 3 readers• The study design enables us to answer several questions
– Is there a difference among readers on volume? • If so, which are different?
– Is there a difference among readers on bias? • If so, which are different?
• Appropriate figures flow from the study design and enable us to visually understand the data
151515151515
![Page 16: May 2012 Update May 10, 2012 Andrew J. Buckler, MS Principal Investigator, QI-Bench](https://reader035.vdocuments.site/reader035/viewer/2022062810/56815c57550346895dca5a2f/html5/thumbnails/16.jpg)
DatalesionID readerID trueVolume readVolume
2 lstk 524.7 520.24 lstk 527.4 508.3
10 lstk 431.6 435.239 lstk 280.5 299.663 lstk 439.8 457.8
2 reader1 524.7 560.74 reader1 527.4 504.7
10 reader1 431.6 438.139 reader1 280.5 306.963 reader1 439.8 433.5
2 reader2 524.7 507.34 reader2 527.4 495.1
10 reader2 431.6 396.939 reader2 280.5 258.863 reader2 439.8 421.2
![Page 17: May 2012 Update May 10, 2012 Andrew J. Buckler, MS Principal Investigator, QI-Bench](https://reader035.vdocuments.site/reader035/viewer/2022062810/56815c57550346895dca5a2f/html5/thumbnails/17.jpg)
171717171717
![Page 18: May 2012 Update May 10, 2012 Andrew J. Buckler, MS Principal Investigator, QI-Bench](https://reader035.vdocuments.site/reader035/viewer/2022062810/56815c57550346895dca5a2f/html5/thumbnails/18.jpg)
181818181818
![Page 19: May 2012 Update May 10, 2012 Andrew J. Buckler, MS Principal Investigator, QI-Bench](https://reader035.vdocuments.site/reader035/viewer/2022062810/56815c57550346895dca5a2f/html5/thumbnails/19.jpg)
191919191919
![Page 20: May 2012 Update May 10, 2012 Andrew J. Buckler, MS Principal Investigator, QI-Bench](https://reader035.vdocuments.site/reader035/viewer/2022062810/56815c57550346895dca5a2f/html5/thumbnails/20.jpg)
202020202020
![Page 21: May 2012 Update May 10, 2012 Andrew J. Buckler, MS Principal Investigator, QI-Bench](https://reader035.vdocuments.site/reader035/viewer/2022062810/56815c57550346895dca5a2f/html5/thumbnails/21.jpg)
212121212121
summary1
![Page 22: May 2012 Update May 10, 2012 Andrew J. Buckler, MS Principal Investigator, QI-Bench](https://reader035.vdocuments.site/reader035/viewer/2022062810/56815c57550346895dca5a2f/html5/thumbnails/22.jpg)
222222222222
summary2
![Page 23: May 2012 Update May 10, 2012 Andrew J. Buckler, MS Principal Investigator, QI-Bench](https://reader035.vdocuments.site/reader035/viewer/2022062810/56815c57550346895dca5a2f/html5/thumbnails/23.jpg)
232323232323
tukey1
![Page 24: May 2012 Update May 10, 2012 Andrew J. Buckler, MS Principal Investigator, QI-Bench](https://reader035.vdocuments.site/reader035/viewer/2022062810/56815c57550346895dca5a2f/html5/thumbnails/24.jpg)
242424242424
tukey2
![Page 25: May 2012 Update May 10, 2012 Andrew J. Buckler, MS Principal Investigator, QI-Bench](https://reader035.vdocuments.site/reader035/viewer/2022062810/56815c57550346895dca5a2f/html5/thumbnails/25.jpg)
252525252525
![Page 26: May 2012 Update May 10, 2012 Andrew J. Buckler, MS Principal Investigator, QI-Bench](https://reader035.vdocuments.site/reader035/viewer/2022062810/56815c57550346895dca5a2f/html5/thumbnails/26.jpg)
262626262626
![Page 27: May 2012 Update May 10, 2012 Andrew J. Buckler, MS Principal Investigator, QI-Bench](https://reader035.vdocuments.site/reader035/viewer/2022062810/56815c57550346895dca5a2f/html5/thumbnails/27.jpg)
27272727272727
Up and running now for you to use
![Page 28: May 2012 Update May 10, 2012 Andrew J. Buckler, MS Principal Investigator, QI-Bench](https://reader035.vdocuments.site/reader035/viewer/2022062810/56815c57550346895dca5a2f/html5/thumbnails/28.jpg)
2828
![Page 29: May 2012 Update May 10, 2012 Andrew J. Buckler, MS Principal Investigator, QI-Bench](https://reader035.vdocuments.site/reader035/viewer/2022062810/56815c57550346895dca5a2f/html5/thumbnails/29.jpg)
Value proposition of QI-Bench• Efficiently collect and exploit evidence establishing
standards for optimized quantitative imaging:– Users want confidence in the read-outs– Pharma wants to use them as endpoints– Device/SW companies want to market products that produce them
without huge costs– Public wants to trust the decisions that they contribute to
• By providing a verification framework to develop precompetitive specifications and support test harnesses to curate and utilize reference data
• Doing so as an accessible and open resource facilitates collaboration among diverse stakeholders
2929
![Page 30: May 2012 Update May 10, 2012 Andrew J. Buckler, MS Principal Investigator, QI-Bench](https://reader035.vdocuments.site/reader035/viewer/2022062810/56815c57550346895dca5a2f/html5/thumbnails/30.jpg)
Summary:QI-Bench Contributions• We make it practical to increase the magnitude of data for increased
statistical significance. • We provide practical means to grapple with massive data sets.• We address the problem of efficient use of resources to assess limits of
generalizability. • We make formal specification accessible to diverse groups of experts that are
not skilled or interested in knowledge engineering. • We map both medical as well as technical domain expertise into
representations well suited to emerging capabilities of the semantic web. • We enable a mechanism to assess compliance with standards or
requirements within specific contexts for use.• We take a “toolbox” approach to statistical analysis. • We provide the capability in a manner which is accessible to varying levels of
collaborative models, from individual companies or institutions to larger consortia or public-private partnerships to fully open public access.
3030
![Page 31: May 2012 Update May 10, 2012 Andrew J. Buckler, MS Principal Investigator, QI-Bench](https://reader035.vdocuments.site/reader035/viewer/2022062810/56815c57550346895dca5a2f/html5/thumbnails/31.jpg)
QI-BenchStructure / Acknowledgements• Prime: BBMSC (Andrew Buckler, Gary Wernsing, Mike Sperling, Matt Ouellette)
• Co-Investigators– Kitware (Rick Avila, Patrick Reynolds, Julien Jomier, Mike Grauer)– Stanford (David Paik)
• Financial support as well as technical content: NIST (Mary Brady, Alden Dima, John Lu)
• Collaborators / Colleagues / Idea Contributors– Georgetown (Baris Suzek)– FDA (Nick Petrick, Marios Gavrielides) – UMD (Eliot Siegel, Joe Chen, Ganesh Saiprasad, Yelena Yesha)– Northwestern (Pat Mongkolwat)– UCLA (Grace Kim)– VUmc (Otto Hoekstra)
• Industry– Pharma: Novartis (Stefan Baumann), Merck (Richard Baumgartner)– Device/Software: Definiens, Median, Intio, GE, Siemens, Mevis, Claron Technologies, …
• Coordinating Programs– RSNA QIBA (e.g., Dan Sullivan, Binsheng Zhao)– Under consideration: CTMM TraIT (Andre Dekker, Jeroen Belien)
3131