biomedical engineering in the metroplex: hyperspectral imaging
TRANSCRIPT
Biomedical Engineering in the Metroplex: Hyperspectral Imaging
for Medical Use
3/13/2012 [email protected] 1
Edward H. Livingston, MD, FACS, AGAF
Professor and Chairman-Gastrointestinal and Endocrine Surgery Professor and Chairman-Graduate Program in Biomedical Engineering
University of Texas Southwestern School of Medicine Contributing Editor-JAMA
The Problem
• Surgeons cannot see what they are doing when they operate
• Does this worry you ? (it should)
3/13/2012 2 [email protected]
The Problem
• Gallbladder Disease is one of the most frequent causes for hospitalizations and surgery
• 750,000 hospital admissions annually
• >500,000 Cholecystectomies performed annually
• 0.25-0.5% Bile Duct Injury Result in up to 2,500 injuries per year.
What Keeps Us Out Of Trouble?
• 5+ Years of intense training after medical school
• Inherent skill, experience
• Intuition, touch, being luckier than good.
• Does these characteristics meet an engineers standards for reliability, reproducibility, QC?
3/13/2012 8 [email protected]
We Need Engineering Solutions
• Develop an imaging system that penetrates opaque tissues
• Identify important structures by their unique chemical composition
3/13/2012 10 [email protected]
Solutions
• Infrared imaging-penetrates tissue
• Spectral Imaging-Identify the chemical spectrum for bile
3/13/2012 11 [email protected]
Figure 1. In vivo visible-reflectance hyperspectral imaging system. a, Broadband light
source produces radiation, which is reflected off a subject and spectrally discriminated by
a liquid crystal tunable filter (LCTF), transduced by a CCD detector, and digi...
Zuzak K J et al. Circulation 2001;104:2905-2910 Copyright © American Heart Association
Near Infrared Macroscopic Hyperspectral Imaging
Gallbladder, Cystic Duct and Liver Tissue
650 750 850 950 1050 0.0
0.2
0.4
0.6
0.8
1.0
Wavelength nm A
pp
aren
t Ab
sorb
ance
650 750 850 950 1050 0.0
0.2
0.4
0.6
0.8
1.0
Wavelength nm
Ap
par
ent A
bso
rban
ce
Gallbladder
Liver
650 750 850 950 1050 0.0
0.2
0.4
0.6
0.8
1.0
Wavelength nm
Ap
par
ent A
bso
rban
ce
Cystic Duct
Principle Component Analysis 60 pound swine, post mortem
Zuzak & Livingston
Problems
• The system was too slow
• 1 minute acquisition time
• Liquid crystal filter was limiting
• Too slow for clinical use
• Spectral artifacts from motion
3/13/2012 17 [email protected]
Solution
• Collaboration between UTSW and UTA
• Partner with TI and Elcan
• Develop a DLP based system to generate light of specific wavelengths at discrete time intervals
• Vast system performance improvement
3/13/2012 18 [email protected]
10-04-2011 Case 07 (Schwarz)
Image Frame # 26 of 101 (Wavelength = 480.00)
CBD Arteriole
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
380 480 580 680 780
Ap
par
en
t A
bso
rban
ce (
a.u
.)
Wavelength (nm)
10-4-2011 CBD
Arteriole
0
0.2
0.4
0.6
0.8
1
1.2
1.4
1.6
380 480 580 680 780
Ap
par
en
t A
bso
rban
ce (
a.u
.)
Wavelength (nm)
10-4-2011 CBD
Arteriole
Image Frame # 26 of 101 (Wavelength = 480.00)
10-18-2011 Case 08 (Schwarz)
Procedure notes state middle structure is CBD
0
0.2
0.4
0.6
0.8
1
1.2
1.4
1.6
380 430 480 530 580 630 680 730 780
Ap
par
en
t re
fle
ctan
ce i
nte
nsi
ty (
a.u
.)
Wavelength (nm)
Unfiltered Spectra 07-19-2011 CBD
05-26-2011 CBD
08-09-2011 CBD
08-30-2011 CBD-6
08-30-2011 (2) CBD
09-20-2011 CBD
10-04-2011 CBD
10-18-2011 CBD
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
380 430 480 530 580 630 680 730 780
No
rmal
ize
d a
pp
are
nt
refl
ect
ance
(a.
u.)
Wavelength (nm)
Filtered and Normalized Spectra
07-19-2011 CBD
05-26-2011 CBD
Avg norm spectrum
08-09-2011 CBD
08-30-2011 (1) CBD-6
08-30-2011 (2) CBD
09-20-2011 CBD
10-04-2011 CBD
10-18-2011 CBD
Hemoglobin
• Least Square Solution
• Obtain pure substance spectral standards (saturated and destaurated Hgb)-Vector P
• Projection Matrix
• Project the observed spectra onto the saturated and desaturated Hgb planes to get the amount of Hgb
3/13/2012 [email protected] 25
Oxyhemoglobin
Deoxyhemoblobin
520 540 560 580 600 620 640 Wavelength (nm)
520 540 560 580 600 620 640 0.0 0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
1.0
0.9 0.9 0.9
Ap
par
ent A
bso
rban
ce
Axy(
λ i)=
-lo
g 10(R
xy(
λ i)
/ R
xy(
λ i) ∞
)
Reference Spectral Components:
Oxyhemoglobin and Deoxyhemoglobin
520 0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1 Measured
Superposition
HbO2 Reference
Hb Reference
540 560 580 600 620 640
Wavelength (nm)
Deconvolution Example
520 0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1 Measured
Superposition
HbO2 Reference
Hb Reference
540 560 580 600 620 640
Wavelength (nm)
Deconvolution Example
520 0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1 Measured
Superposition
HbO2 Reference
Hb Reference
540 560 580 600 620 640
Wavelength (nm)
Deconvolution Example
DLP Hyperspectral Imaging System
Focal Plane
Array
Programmable Spectral
Light Source DLP® Inside
Tissue
Sample
COMPUTER
Visualization
Synchronized
Hardware
Control
Image
Processing
Acquire Data Cube Digitized
Hyperspectral Image
Cube
Graphical User Interface (GUI)
DLP Hyperspectral Imaging System
GUI developed by:
Timing Diagram for DLP HSI with 126shot Method
Open Shutter HQ2
Expose CCD
Close Shutter
Save as temp00n.dat
A/D conversion
Command OL 490 to
Illuminate Spectrum n
n = n+1
Start Acquisition
(n = 1)
Is n = 126? no yes
Process with Oxyz
Algorithm in MATLAB
Output bitmap image
tshot = 116.3 ms
tacquisition = 116.3 * 126 = 14 648 ms
tprocessing = 8 432 ms
40 ms
Ttotal = 23 080 ms
Timing Diagram for DLP HSI with 3shot Method
Open Shutter HQ2
Expose CCD
Close Shutter
Save as temp00n.dat
A/D conversion
Command OL 490 to
Illuminate Spectrum n
n = n+1
Start Acquisition
(n = 1)
Is n = 3? no yes
Process with 3shot
Algorithm in MATLAB
Output bitmap image
tshot = 93.3 ms
tacquisition = 93.3 * 3 = 280 ms
tprocessing = 110 ms
1.25 ms
Ttotal = 390 ms
R² = 0.3774
0.6
0.7
0.8
0.9
1
1.1
1.2
0 10 20 30 40 50 60 70
Series1
Poly. (Series1)
Rel
ativ
e %
Hb
O2
No
rmal
ized
to
Co
ntr
ol
Time after Clamping (minutes)
Clamped
Control
Cut
Clamp Off
Iced
Images from Human Partial Case 15 (12-18-09) and Plot of all Cases
Control 8 min after clamping 1 min after clamp off 10
30
50
70
90
Relativ
e Co
ntrib
utio
n o
f Hb
O2
Video Rate Hyperspectral Imaging
Biochemical Visualization of Human Kidney Over Time
Animal Study
DLP Hyperspectral Imaging
• Picture of pig kidney immediately after arterial clamping
• Assuming arterial blood is shunted away from kidney by arterial clamp
Invasive Lycox probe inserted into kidney
Kidney
DLP Hyperspectral Imaging
Kidney
10
20
30
40
50
60
70
80
90
Rel
ativ
e C
ontr
ibuti
on o
f H
bO
2
-Discovery:
The Kidney had two renal arteries.
Only the lower pole artery was clamped,
Hyperspectral Imaging immediately indicated
continued perfusion of the upper Kidney pole.
Hyperspectral Image discovers Pig Kidney with 2 Renal Arteries
DLP Hyperspectral Imaging
Kidney
10
20
30
40
50
60
70
80
90
Rel
ativ
e C
ontr
ibuti
on o
f H
bO
2
Hyperspectral Image of Pig Kidney with Both Renal Arteries Clamped
Re-situating the clamp over both
arteries led to ischemia of both
upper lower poles.
Note:- 3 shot Images showing more oxygenation in intestines as kidney becomes deoxygenated
5 min 32 sec before clamping
4 min 44 seconds before clamping
Clamp on 1 minute after clamping
2 minutes after clamping
3 minutes after
clamping
0
10
20
30
40
50
60
70
80
90
100
-6 -5 -4 -3 -2 -1 0 1 2 3 4
Kidney
Intestine
Time before and after Clamping in Minutes
Clamp On
% H
bO
2
Pig Partial Nephrectomy Trial 2
0
20
40
60
80
100
120
-40 -20 0 20 40 60 80 100
Kidney
Intestines
Clamp On (100%) Clamp Off
Re
lati
ve P
erc
ent
Hb
O2
Time before and after Clamp (Minutes)
Control 27 minutes before clamping
5 minutes after clamping
15 minutes after clamping
34 minutes after clamping
57 minutes after clamping
1 minute after clamp off
14 minutes after clamp off
Pig Partial Nephrectomy Case 1 (Kidney and Intestines)
Control (0 min)
Clamped (15 min)
Clamped & After Icing (11 min)
After Clamping (3 min)
Clamped (21 min)
Clamped (31 min)
After Clamp Off (33 min)
Clamp Off (1 hr 5 min)
Human Partial Nephrectomy Case 3
Partial Nephrectomy
0
10
20
30
40
50
60
70
80
90
100
0 10 20 30 40 50 60 70
Time in Minutes
Rela
tive P
erc
en
t H
bO
2
% HbO2
Clamp On Clamp Off Clamp On & After Icing
In Vivo, Visible Reflectance, Hyperspectral Imager
Coupled to a Zeiss Neurosurgical Microscope
Source
Illumination
Reflection
Of Subject
10
20
30
40
50
60
70
80
90
% H
bO
2
Control After Induced Ischemia
Damaged Tissue Normal Tissue
Brain Surgery Monitoring for
Preventing Post Surgery Stroke
DLP Hyperspectral Microscope
Monitoring Diabetic Retinopathy
Double Knock Out (Apoe-/-, db/db) Mouse Model
B
10
30
50
70
90
10
30
50
70
90
Rel
ati
ve O
xyh
emo
glo
bin
Parental 2: Wildtype Apoe /Hom db
Day: 81 Day: 151
Parental 1: Hom Apoe/ Wildtype db
Day: 79 Day: 142
Double Knock Out Diabetic Mouse:
Hom Apoe/ Hom db
Day: 78 Day: 136
Data Collected Using LCTF
LCTF Hyperspectral Imaging System
For Imaging Human Retina
0
Visible LCTF
Illumination Mirror CCD
Beam Splitter
Magnification
Knob
Light Source
Joystick
Table Top
Common Center
of Rotation
Subject’s
Head rest
Intensity Control
Knob
Eye Piece
10
20
30
40
50
60
70
80
90
Rela
tive
Oxyh
em
og
lob
in
Con
trib
uti
on
Gray-scale Encoded
Hyperspectral Image
530 550 570 590 0
0.2
0.4
0.6
0.8
1.0
Wavelength (nm)
Ap
pare
nt
Ab
sorb
an
ce
530 550 570 590 0
0.2
0.4
0.6
0.8
1.0
Wavelength (nm)
Ap
pare
nt
Ab
sorb
an
ce
Microscopic Hyperspectral Imager:
Human Retinal Imaging of Oxyhemoglobin Contribution %
Hb
O2
Area 1 Area 2 0
10
20
30
40
50
60
70 P < 0.0001
Hyperspectral Fundus Camera
Spectral Light Source
LLG
Fundus Camera
Laptop CCD Camera
10
20
30
40
50
60
Relative HbO2
Need to Improve
Visualization Algorithms
artifacts
In Vivo Hyperspectral Imaging of Human Tissue: Spatial Variation of Percentage of HbO2 and Surface Temperature in Response to Burn
Bright Field
1 inch
Multivariate Least Squares
with respect to Oxyhemoglobin
Ap
pare
nt A
bso
rban
ce
Ax
y(λ i
) =
-lo
g10
(Rx
y(λ i
) /
Rx
y(λ i
) ∞)
500 540 580 620 660
Spectra Under
points and
Wavelength (nm)
Thermal Image
1 inch 1 inch
BURNS
10
30
50
70
90
Relative Contribution of HbO2
10
30
50
70
90
Relative Contribution of H2O
Chemically Encoded
HbO2
Chemically Encoded
H2O Color Photo
BURNS
Post Fasciectomy
10
20
30
40
50
60
70
80
90
Relative Contribution of HbO2
10
20
30
40
50
60
70
80
90
Relative Contribution of H2O
Color Photo Chemically Encoded
Oxyhemoglobin
Chemically Encoded
Water
Patient 002,
Monitoring Post Amputation
Left Lower Limb, Foot, left BKA
Session 1
051707 Session 2
052107
Digital
Image
A B1 A B2
Vis
HbO2
A 62 3.9
B 62±1.2
A 70 4.2
B 70±3.2
10
20
30
40
50
60
70
80
90
A
B1 A
B2
NIR
HbO2
A 76.5 6.5
B 84.5±2.1
A 68.9 1.7
B 67.6±1.1 10
20
30
40
50
60
70
80
90
Per
centa
ge o
f H
bO
2
0
10
20
30
40
50
60
70
80
90
100
Per
centa
ge o
f H
bO
2
0
10
20
30
40
50
60
70
80
90
100
B1
Zuzak & Livingston
Suture Tension
Monitoring Post Amputation Recovery
A 75.83 2.59
B 76.3±0.86
Patient 002,
Amputation, Left AKA
10
20
30
40
50
60
70
80
90
B A
A 56.91 2.91
B 56.2±0.63
10
20
30
40
50
60
70
80
90
Patient 005,
Amputation, Left AKA A
B
A 85.61 3.53
B 81.58±4.38
Patient 006,
Amputation, Left BKA
10
20
30
40
50
60
70
80
90
A B
0
10
20
30
40
50
60
70
80
90
100 P
erce
nta
ge o
f H
2O
0
10
20
30
40
50
60
70
80
90
100
Per
cen
tage
of
H2O
0
10
20
30
40
50
60
70
80
90
100
Per
centa
ge o
f H
2O
Patient 002, 005, 006
Session I
Per
centa
ge o
f H
2O
Per
centa
ge o
f H
2O
Per
centa
ge o
f H
2O
85.83 % HbO2 81.05 % HbO2
Per
centa
ge o
f O
xyhem
ogl
obin
10
20
30
40
50
60
70
80
90
Hyperspectral Imaging
Visible
10
20
30
40
50
60
70
80
90
Per
centa
ge o
f oxy
hem
ogl
obin
NIR
Data collected 10th minute
after removing the tight shoe
71.7893 +/- 3.7673
79.1163 +/- 2.208
57.3525 +/- 5.7005
62.0387 +/- 3.2884
63.8259 +/- 5.3345
69.8058 +/- 2.8421
Diabetic Neuropathy
Other Potential Applications
• Neurological Surgery - Preventing Stroke After Brain Tumors.
• Plastic Surgery - Skin flaps, Wound Management and Burns.
• Surgery - Non-Invasive Laparoscopic Optical Biopsy During.
• Urology – Tumor Removal during Kidney Surgery.
• Ophthalmology – Monitoring Diabetic Retinopathy.
• Clinical Monitoring – Managing Lower Limb Amputations.
The speed and versatility of DLP® technology is making
Hyperspectral Imaging practical for a wide variety of
surgical and clinical applications.
Placenta Control
Prior to ICG infusion
Vascular infusion
Of ICG via cannual
Fluorescence of ICG Vascular Infusion
and
DLP Spectral Excitation
Picture of Placenta
Prior to ICG infusion
The DLP Illuminates tissue with predetermined
spectral illumination exciting the ICG to fluoresce.
Major seed funding provided by
• Texas Instruments
Supplemental funding provided in part by
• Hudson-Pen Endowment
• Smith Endowment
• Department of Energy.
