texture-based computational models of tissue in biomedical images: initial experience with digital...
TRANSCRIPT
TEXTURE-BASED COMPUTATIONAL MODELS OF TISSUE IN BIOMEDICAL IMAGES:
INITIAL EXPERIENCE WITH DIGITAL HISTOPATHOLOGY
Adrien Depeursinge, PhD Journées GdR ISIS “Analyse de Tissu Biologique et Histopathologie Numérique”
Paris, June 23rd 2014
BACKGROUND – RADIOMICS - HISTOPATHOLOMICS
• Personalized medicine aims at enhancing the patient’s quality of life and prognosis
• Tailored treatment and medical decisions based on the molecular composition of diseased tissue
• Current limitations [1]
• Molecular analysis of tissue composition is invasive (biopsy), slow and costly
• Cannot capture molecular heterogeneity
2
[1] Intratumor heterogeneity and branched evolution revealed by multiregion sequencing, Gerlinger et al., N Engl J Med, 366(10):883-92, 2012.
Intr atumor Heterogeneity Revealed by multiregion Sequencing
n engl j med 366;10 nejm.org march 8, 2012 887
tion through loss of SETD2 methyltransferase func-tion driven by three distinct, regionally separated mutations on a background of ubiquitous loss of the other SETD2 allele on chromosome 3p.
Convergent evolution was observed for the X-chromosome–encoded histone H3K4 demeth-ylase KDM5C, harboring disruptive mutations in R1 through R3, R5, and R8 through R9 (missense
and frameshift deletion) and a splice-site mutation in the metastases (Fig. 2B and 2C).
mTOR Functional Intratumor HeterogeneityThe mammalian target of rapamycin (mTOR) ki-nase carried a kinase-domain missense mutation (L2431P) in all primary tumor regions except R4. All tumor regions harboring mTOR (L2431P) had
B Regional Distribution of Mutations
C Phylogenetic Relationships of Tumor Regions D Ploidy Profiling
A Biopsy Sites
R2 R4
DI=1.43
DI=1.81
M2bR9
Tetraploid
R4b
R9 R8R5
R4a
R1R3R2
M1M2b
M2a
VHL
KDM5C (missense and frameshift)mTOR (missense)
SETD2 (missense)KDM5C (splice site)
SETD2 (splice site)
?
SETD2 (frameshift)
PreP
PreM
Normal tissue
PrePPreMR1R2R3R5R8R9R4M1M2aM2b
C2o
rf85
WD
R7SU
PT6H
CD
H19
LAM
A3
DIX
DC
1H
PS5
NRA
PKI
AA
1524
SETD
2PL
CL1
BCL1
1AIF
NA
R1$
DA
MTS
10
C3
KIA
A12
67.
RT4
CD
44A
NKR
D26
TM7S
F4SL
C2A
1D
AC
H2
MM
AB
ZN
F521
HM
G20
AD
NM
T3A
RLF
MA
MLD
1M
AP3
K6H
DA
C6
PHF2
1BFA
M12
9BRP
S8C
IB2
RAB2
7ASL
C2A
12D
USP
12A
DA
MTS
L4N
AP1
L3U
SP51
KDM
5CSB
F1TO
M1
MYH
8W
DR2
4IT
IH5
AKA
P9FB
XO1
LIA
STN
IKSE
TD2
C3o
rf20
MR1
PIA
S3D
IO1
ERC
C5
KLALK
BH8
DA
PK1
DD
X58
SPA
TA21
ZN
F493
NG
EFD
IRA
S3LA
TS2
ITG
B3FL
NA
SATL
1KD
M5C
KDM
5CRB
FOX2
NPH
S1SO
X9C
ENPN
PSM
D7
RIM
BP2
GA
LNT1
1A
BHD
11U
GT2
A1
MTO
RPP
P6R2
ZN
F780
AW
SCD
2C
DKN
1BPP
FIA
1THSS
NA
1C
ASP
2PL
RG1
SETD
2C
CBL
2SE
SN2
MA
GEB
16N
LRP7
IGLO
N5
KLK4
WD
R62
KIA
A03
55C
YP4F
3A
KAP8
ZN
F519
DD
X52
ZC
3H18
TCF1
2N
USA
P172
X4KD
M2B
MRP
L51
C11
orf6
8A
NO
5EI
F4G
2M
SRB2
RALG
DS
EXT1
ZC
3HC
1PT
PRZ
1IN
TS1
CC
R6D
OPE
Y1A
TXN
1W
HSC
1C
LCN
2SS
R3KL
HL1
8SG
OL1
VHL
C2o
rf21
ALS
2CR1
2PL
B1FC
AM
RIF
I16
BCA
S2IL
12RB
2
PrivateUbiquitous Shared primary Shared metastasis
Private
Ubiquitous
Lungmetastases
Chest-wallmetastasis
Perinephricmetastasis
M110 cm
R7 (G4)
R5 (G4)
R9
R3 (G4)
R1 (G3) R2 (G3)
R4 (G1)
R6 (G1)
Hilu
m
R8 (G4)
Primarytumor
Shared primaryShared metastasis
M2b
M2a
Propidium Iodide Staining
No.
of C
ells
The New England Journal of Medicine Downloaded from nejm.org at UNIVERSITE DE GENEVE on June 2, 2014. For personal use only. No other uses without permission.
Copyright © 2012 Massachusetts Medical Society. All rights reserved.
Intr atumor Heterogeneity Revealed by multiregion Sequencing
n engl j med 366;10 nejm.org march 8, 2012 887
tion through loss of SETD2 methyltransferase func-tion driven by three distinct, regionally separated mutations on a background of ubiquitous loss of the other SETD2 allele on chromosome 3p.
Convergent evolution was observed for the X-chromosome–encoded histone H3K4 demeth-ylase KDM5C, harboring disruptive mutations in R1 through R3, R5, and R8 through R9 (missense
and frameshift deletion) and a splice-site mutation in the metastases (Fig. 2B and 2C).
mTOR Functional Intratumor HeterogeneityThe mammalian target of rapamycin (mTOR) ki-nase carried a kinase-domain missense mutation (L2431P) in all primary tumor regions except R4. All tumor regions harboring mTOR (L2431P) had
B Regional Distribution of Mutations
C Phylogenetic Relationships of Tumor Regions D Ploidy Profiling
A Biopsy Sites
R2 R4
DI=1.43
DI=1.81
M2bR9
Tetraploid
R4b
R9 R8R5
R4a
R1R3R2
M1M2b
M2a
VHL
KDM5C (missense and frameshift)mTOR (missense)
SETD2 (missense)KDM5C (splice site)
SETD2 (splice site)
?
SETD2 (frameshift)
PreP
PreM
Normal tissue
PrePPreMR1R2R3R5R8R9R4M1M2aM2b
C2o
rf85
WD
R7SU
PT6H
CD
H19
LAM
A3
DIX
DC
1H
PS5
NRA
PKI
AA
1524
SETD
2PL
CL1
BCL1
1AIF
NA
R1$
DA
MTS
10
C3
KIA
A12
67.
RT4
CD
44A
NKR
D26
TM7S
F4SL
C2A
1D
AC
H2
MM
AB
ZN
F521
HM
G20
AD
NM
T3A
RLF
MA
MLD
1M
AP3
K6H
DA
C6
PHF2
1BFA
M12
9BRP
S8C
IB2
RAB2
7ASL
C2A
12D
USP
12A
DA
MTS
L4N
AP1
L3U
SP51
KDM
5CSB
F1TO
M1
MYH
8W
DR2
4IT
IH5
AKA
P9FB
XO1
LIA
STN
IKSE
TD2
C3o
rf20
MR1
PIA
S3D
IO1
ERC
C5
KLALK
BH8
DA
PK1
DD
X58
SPA
TA21
ZN
F493
NG
EFD
IRA
S3LA
TS2
ITG
B3FL
NA
SATL
1KD
M5C
KDM
5CRB
FOX2
NPH
S1SO
X9C
ENPN
PSM
D7
RIM
BP2
GA
LNT1
1A
BHD
11U
GT2
A1
MTO
RPP
P6R2
ZN
F780
AW
SCD
2C
DKN
1BPP
FIA
1THSS
NA
1C
ASP
2PL
RG1
SETD
2C
CBL
2SE
SN2
MA
GEB
16N
LRP7
IGLO
N5
KLK4
WD
R62
KIA
A03
55C
YP4F
3A
KAP8
ZN
F519
DD
X52
ZC
3H18
TCF1
2N
USA
P172
X4KD
M2B
MRP
L51
C11
orf6
8A
NO
5EI
F4G
2M
SRB2
RALG
DS
EXT1
ZC
3HC
1PT
PRZ
1IN
TS1
CC
R6D
OPE
Y1A
TXN
1W
HSC
1C
LCN
2SS
R3KL
HL1
8SG
OL1
VHL
C2o
rf21
ALS
2CR1
2PL
B1FC
AM
RIF
I16
BCA
S2IL
12RB
2
PrivateUbiquitous Shared primary Shared metastasis
Private
Ubiquitous
Lungmetastases
Chest-wallmetastasis
Perinephricmetastasis
M110 cm
R7 (G4)
R5 (G4)
R9
R3 (G4)
R1 (G3) R2 (G3)
R4 (G1)
R6 (G1)
Hilu
mR8 (G4)
Primarytumor
Shared primaryShared metastasis
M2b
M2a
Propidium Iodide Staining
No.
of C
ells
The New England Journal of Medicine Downloaded from nejm.org at UNIVERSITE DE GENEVE on June 2, 2014. For personal use only. No other uses without permission.
Copyright © 2012 Massachusetts Medical Society. All rights reserved.
Intr atumor Heterogeneity Revealed by multiregion Sequencing
n engl j med 366;10 nejm.org march 8, 2012 887
tion through loss of SETD2 methyltransferase func-tion driven by three distinct, regionally separated mutations on a background of ubiquitous loss of the other SETD2 allele on chromosome 3p.
Convergent evolution was observed for the X-chromosome–encoded histone H3K4 demeth-ylase KDM5C, harboring disruptive mutations in R1 through R3, R5, and R8 through R9 (missense
and frameshift deletion) and a splice-site mutation in the metastases (Fig. 2B and 2C).
mTOR Functional Intratumor HeterogeneityThe mammalian target of rapamycin (mTOR) ki-nase carried a kinase-domain missense mutation (L2431P) in all primary tumor regions except R4. All tumor regions harboring mTOR (L2431P) had
B Regional Distribution of Mutations
C Phylogenetic Relationships of Tumor Regions D Ploidy Profiling
A Biopsy Sites
R2 R4
DI=1.43
DI=1.81
M2bR9
Tetraploid
R4b
R9 R8R5
R4a
R1R3R2
M1M2b
M2a
VHL
KDM5C (missense and frameshift)mTOR (missense)
SETD2 (missense)KDM5C (splice site)
SETD2 (splice site)
?
SETD2 (frameshift)
PreP
PreM
Normal tissue
PrePPreMR1R2R3R5R8R9R4M1M2aM2b
C2o
rf85
WD
R7SU
PT6H
CD
H19
LAM
A3
DIX
DC
1H
PS5
NRA
PKI
AA
1524
SETD
2PL
CL1
BCL1
1AIF
NA
R1$
DA
MTS
10
C3
KIA
A12
67.
RT4
CD
44A
NKR
D26
TM7S
F4SL
C2A
1D
AC
H2
MM
AB
ZN
F521
HM
G20
AD
NM
T3A
RLF
MA
MLD
1M
AP3
K6H
DA
C6
PHF2
1BFA
M12
9BRP
S8C
IB2
RAB2
7ASL
C2A
12D
USP
12A
DA
MTS
L4N
AP1
L3U
SP51
KDM
5CSB
F1TO
M1
MYH
8W
DR2
4IT
IH5
AKA
P9FB
XO1
LIA
STN
IKSE
TD2
C3o
rf20
MR1
PIA
S3D
IO1
ERC
C5
KLALK
BH8
DA
PK1
DD
X58
SPA
TA21
ZN
F493
NG
EFD
IRA
S3LA
TS2
ITG
B3FL
NA
SATL
1KD
M5C
KDM
5CRB
FOX2
NPH
S1SO
X9C
ENPN
PSM
D7
RIM
BP2
GA
LNT1
1A
BHD
11U
GT2
A1
MTO
RPP
P6R2
ZN
F780
AW
SCD
2C
DKN
1BPP
FIA
1THSS
NA
1C
ASP
2PL
RG1
SETD
2C
CBL
2SE
SN2
MA
GEB
16N
LRP7
IGLO
N5
KLK4
WD
R62
KIA
A03
55C
YP4F
3A
KAP8
ZN
F519
DD
X52
ZC
3H18
TCF1
2N
USA
P172
X4KD
M2B
MRP
L51
C11
orf6
8A
NO
5EI
F4G
2M
SRB2
RALG
DS
EXT1
ZC
3HC
1PT
PRZ
1IN
TS1
CC
R6D
OPE
Y1A
TXN
1W
HSC
1C
LCN
2SS
R3KL
HL1
8SG
OL1
VHL
C2o
rf21
ALS
2CR1
2PL
B1FC
AM
RIF
I16
BCA
S2IL
12RB
2
PrivateUbiquitous Shared primary Shared metastasis
Private
Ubiquitous
Lungmetastases
Chest-wallmetastasis
Perinephricmetastasis
M110 cm
R7 (G4)
R5 (G4)
R9
R3 (G4)
R1 (G3) R2 (G3)
R4 (G1)
R6 (G1)H
ilum
R8 (G4)
Primarytumor
Shared primaryShared metastasis
M2b
M2a
Propidium Iodide Staining
No.
of C
ells
The New England Journal of Medicine Downloaded from nejm.org at UNIVERSITE DE GENEVE on June 2, 2014. For personal use only. No other uses without permission.
Copyright © 2012 Massachusetts Medical Society. All rights reserved.
BACKGROUND – RADIOMICS - HISTOPATHOLOMICS
• Huge potential for computerized medical image analysis
• Create imaging biomarkers to predict diagnosis, prognosis, treatment response [2]
3
[2] Imaging and genomics: is there a synergy?, Jaffe et al., Radiology, 264(2):329-31, 2012 [3] Radiomics: the process and the challenges, Kumar et al., Magn Reson Imaging, 30(9):1234-48, 2012 [4] Histopathological image analysis: a review, Gurcan et al., IEEE Reviews in Biomed Eng, 2:147-71, 2009
Radiomics [3] “Histopatholomics” [4]
Reuse existing diagnostic images radiology data digital pathology
Capture tissue heterogeneity
3D neighborhoods(e.g., 512x512x512)
large 2D regions(e.g., 15,000x15,000)
Analytic power beyond naked eyes
complex 3D tissue morphology
exhaustive characterization of 2D tissue structures
Non-invasive x
BACKGROUND – RADIOMICS - HISTOPATHOLOMICS
• Huge potential for computerized medical image analysis
• Create imaging biomarkers to predict diagnosis, prognosis, treatment response
• Local quantitative image feature extraction
• Supervised machine learning
4
malignant, nonrespondermalignant, responder
benign
pre-malignant
undefined
quant. feat. #1
quan
t. fe
at. #
2
Supervised learning, big data
This could include an additional step of studying the spatial relationships between local image properties (e.g., using image graphs)
• Shape, intensity, margin, texture, …
• Shape and margin features often require prior image segmentation
• 2D and 3D texture analysis can quantify micro- and macro- structures in biomedical images [4,6]
IMAGE FEATURES
5
GU
RC
AN
etal
.:H
IST
OPA
TH
OL
OG
ICA
LIM
AG
EA
NA
LYSI
S:A
RE
VIE
W15
5
Fig.
5.R
esul
tsof
the
auto
mat
icse
gmen
tatio
nal
gori
thm
(blu
eco
ntou
rs:l
umen
boun
dary
,bla
ckco
ntou
rs:
inne
rbo
unda
ryof
the
nucl
eiof
the
epith
elia
lce
llssu
rrou
ndin
gth
egl
and)
.Sho
wn
from
left
tori
ghta
reex
ampl
eim
ages
ofbe
nign
epith
eliu
m,i
nter
med
iate
-,an
dhi
gh-g
rade
canc
er.
and
are
2-D
Car
tesi
anco
ordi
nate
sof
.The
evol
utio
nof
isth
ende
scri
bed
bya
leve
l-se
tfor
mul
atio
nad
opte
dfr
om[7
8]
(4.1
)
whe
reth
efu
nctio
nde
fines
the
spee
dof
the
evol
utio
n.T
hecu
rve
evol
utio
nis
driv
enby
the
nucl
eilik
elih
ood
imag
e.T
hein
itial
cont
our
isin
itial
ized
auto
mat
ical
lyus
ing
the
dete
cted
lum
enar
eafr
omth
eca
ndid
ate
glan
dre
gion
s.T
hecu
rve
isev
olve
dou
twar
dfr
omth
ede
tect
edlu
men
regi
ons
inth
eco
mbi
ned
nucl
eilik
elih
ood
imag
eto
avoi
dno
ise
and
allo
wsm
ooth
erev
olut
ion
rela
tive
toth
eor
igin
alim
age.
The
inte
nsiti
esof
the
nucl
eilik
elih
ood
imag
efo
rmth
est
oppi
nggr
adie
nt.T
heal
gori
thm
isru
nun
tilth
edi
ffer
ence
inth
eco
ntou
rsin
two
con-
secu
tive
itera
tions
isbe
low
anem
piri
cally
dete
rmin
edth
resh
old.
Dur
ing
trai
ning
,siz
edi
stri
butio
nssi
mila
rto
thos
eus
edto
calc
u-la
teob
ject
likel
ihoo
dar
ecr
eate
dus
ing
the
final
cont
ours
.The
senu
clea
rbou
ndar
yba
sed
dist
ribu
tions
are
used
tore
mov
ere
gion
sth
atar
eto
ola
rge
tobe
true
glan
ds.F
inal
ly,t
helu
men
and
nu-
clea
rbou
ndar
iese
xtra
cted
from
true
glan
dre
gion
sare
pass
edon
toth
ene
xtst
epfo
rfe
atur
eex
trac
tion.
Sam
ple
resu
ltsfr
omth
eau
tom
ated
glan
dse
gmen
tatio
nal
gori
thm
are
show
nin
Fig.
5.T
helu
men
boun
dari
esar
edi
spla
yed
ina
solid
blue
cont
oura
ndth
ein
teri
ornu
clea
rbo
unda
ries
are
disp
laye
das
dash
edbl
ack
lines
.R
esul
tsof
our
glan
dse
gmen
tatio
nal
gori
thm
are
show
nfo
rsa
mpl
eim
ages
from
beni
gnep
ithel
ium
,in
term
edia
te,
and
high
-gra
depr
osta
teca
ncer
(fro
mle
ftto
righ
t).
B.
Glo
balS
cene
Segm
enta
tion
App
roac
hes
In[2
6],
aun
ified
segm
enta
tion
algo
rith
mfo
rsu
b-ce
l-lu
lar
com
part
men
taliz
atio
nw
aspr
esen
ted.
Qua
ntita
tion
ofbi
omar
kers
atsu
b-ce
llula
rre
solu
tion
requ
ires
segm
enta
tion
ofsu
b-ce
llula
rco
mpa
rtm
ents
such
asnu
clei
,mem
bran
es,a
ndcy
topl
asm
.W
hile
diff
eren
tse
gmen
tatio
nal
gori
thm
sca
nbe
used
for
each
ofth
esu
b-ce
llula
rco
mpa
rtm
ents
,an
alte
rnat
ive
isto
use
the
sam
eal
gori
thm
indi
ffer
entm
odes
.The
algo
rith
min
[26]
capt
ured
ase
tof
brig
htpi
xels
shar
ing
aco
mm
onsh
ape
dist
ribu
tion.
The
algo
rith
mus
eda
set
ofth
ree
feat
ures
,one
isth
eflu
ores
cent
emis
sion
inte
nsity
,and
the
othe
rtw
oar
eba
sed
oncu
rvat
ure
desc
ript
orst
hata
reco
mpu
ted
from
the
eige
nval
ues
ofth
eH
essi
anm
atri
x.Fo
ran
imag
e,,t
heei
genv
alue
s(
)of
the
Hes
sian
mat
rix
enco
deth
ecu
rvat
ure
info
rmat
ion
ofth
eim
age,
and
prov
ide
usef
ulcu
esfo
rde
tect
ing
ridg
e-lik
em
embr
ane
stru
ctur
es,o
rbl
ob-l
ike
nucl
eist
ruct
ures
.How
ever
,th
eei
genv
alue
sar
ede
pend
ent
onim
age
brig
htne
ss.
The
fol-
low
ing
two
curv
atur
e-ba
sed
feat
ures
are
inde
pend
ento
fim
age
brig
htne
ss:
(4.2
)
(4.3
)
and
refe
rred
toas
shap
ein
dex,
and
norm
aliz
ed-c
urva
ture
inde
x,re
spec
tivel
y.T
his
ises
sent
ially
the
sam
eas
defin
ing
the
eige
n-va
lues
ina
pola
rco
ordi
nate
syst
em.
Thi
str
ansf
orm
atio
nal
sore
sults
inbo
unde
dfe
atur
es,
,an
d.
The
estim
atio
npr
oces
sst
arts
with
the
expe
cted
dist
ribu
tions
ofth
esh
ape
inde
xfo
rth
est
ruct
ures
tobe
segm
ente
d.Fo
rex
-am
ple,
forb
righ
tmem
bran
ean
dve
ssel
like
stru
ctur
esth
esh
ape
inde
xis
clos
eto
,bec
ause
the
smal
lere
igen
valu
eis
nega
-tiv
ean
dth
ela
rger
eige
nval
ueap
proa
ches
toze
ro.O
nth
eot
her
hand
,for
the
blob
-lik
enu
clei
stru
ctur
es,t
hesh
ape
inde
xis
clos
eto
,bec
ause
both
eige
nval
ues
are
nega
tive
and
clos
ein
valu
e.Fo
rbot
hst
ruct
ures
,pos
itive
valu
esin
dica
tea
pixe
lbei
ngm
ore
like
aba
ckgr
ound
.The
seco
nstr
aint
sar
eus
edto
com
pute
the
initi
alfo
regr
ound
and
back
grou
ndse
tsfo
rm
embr
ane
and
nucl
eist
ruct
ures
.A
nin
itial
segm
enta
tion
base
don
the
shap
ein
dex
and
the
norm
aliz
ed-c
urva
ture
inde
xse
para
tes
the
imag
epi
xels
into
thre
esu
bset
s:ba
ckgr
ound
,for
egro
und,
and
inde
ter-
min
ate.
The
inde
term
inat
esu
bset
com
pris
esal
lthe
pixe
lsth
atar
eno
tinc
lude
din
the
back
grou
ndor
fore
grou
ndsu
bset
s.Fr
omth
ese
subs
ets,
the
back
grou
ndan
dfo
regr
ound
inte
nsity
dist
ri-
butio
ns,a
sw
ella
sth
ein
tens
itylo
g-lik
elih
ood
func
tions
are
es-
timat
ed.T
heal
gori
thm
keep
site
ratin
gby
usin
gtw
oou
tof
the
thre
efe
atur
esat
atim
eto
estim
ate
the
dist
ribu
tion
ofth
efe
atur
eth
atis
left
out.
Inth
efin
alst
ep,t
hese
log-
likel
ihoo
dfu
nctio
nsar
eco
mbi
ned
tode
term
ine
the
over
all
likel
ihoo
dfu
nctio
n.A
prob
abili
tym
apth
atre
pres
ents
the
prob
abili
tyof
api
xelb
eing
afo
regr
ound
isca
lcul
ated
.C
ytop
lasm
can
bede
tect
edei
ther
byus
ing
asp
ecifi
ccy
-to
plas
mic
mar
ker,
orca
nbe
dete
cted
usin
gco
mpu
tatio
nal
met
hods
usin
gth
efa
ctth
atth
ecy
topl
asm
icar
eas
are
betw
een
nucl
ear
and
mem
bran
ear
eas.
For
mos
tca
ncer
tissu
ety
pes,
itis
very
impo
rtan
tto
diff
eren
tiate
the
epith
elia
ltis
sue
from
the
stro
mal
and
conn
ectiv
etis
sue,
soth
atfo
rIH
Cst
udie
sth
eex
pres
sion
leve
lsof
mos
tm
arke
rsin
the
epith
elia
lre
gion
sca
nbe
quan
tified
.C
ompu
tatio
nal
met
hods
that
use
the
high
conn
ectiv
ityof
mem
bran
em
eshe
sca
nbe
used
todi
ffer
entia
teth
eep
ithel
ial
regi
ons.
For
the
sam
ple
imag
es,
any
conn
ecte
dco
mpo
nent
larg
erth
an80
0pi
xels
isac
cept
edas
apa
rtof
the
epith
elia
lmas
k.T
henu
clei
seti
sth
ense
para
ted
into
epith
elia
lnu
clei
and
stro
mal
nucl
eius
ing
the
epith
elia
lmas
k.
[4] Histopathological image analysis: a review, Gurcan et al., IEEE Reviews in Biomed Eng, 2:147-71, 2009 [5] Quantifying the margin sharpness of lesions on radiological images for content-based image retrieval, Xu et al., Med Phys, 39(9):5405-18, 2012 [6] Three-dimensional solid texture analysis in biomedical imaging: review and opportunities, Depeursinge et al., Med Image Anal, 18(1):176-96, 2014 [7] Prediction of prognosis for prostatic adenocarcinoma by combined histological grading and clinical staging, Gleason et al., J Urol, 111(1):58-64, 1974 [8] HyMaP: A hybrid magnitude-phase approach to unsupervised segmentation of tumor areas in breast cancer histology images, Khan et al., J Pathol Inform, 4, 2013
[4]
5407Xuetal.:Quantifyingthemarginsharpnessoflesions5407
(a)(b)
Xo-4Xo-2XoXo+2Xo+4100
200
300
400
500
600
Sigmoid FitIntensity Values
FIG.2.(a)CroppedCTimagecontainingalungnodulewithanormallineandROIdrawn.(b)Pixelintensityvaluesalongthenormalline(solidline)andfittedsigmoid(dashedline)showingthescaleandwindowparameters.
byradiologists,butcouldalsobegeneratedautomaticallybyaCADsystemorautomatedsegmentationalgorithm.Togener-atethemarginsharpnessfeature,wefirstrepresenttheROIbyapiecewisecubicpolynomialcurve(interpolatedfrom10–20controlpointsdrawnbyaradiologist).WenextdrawnormallinesegmentsoflengthTpixelsacrosstheboundaryofthelesionatfixedintervalsarounditscircumference(thefixedintervalisthelargervalueof1pixelandthelengthoftheboundaryinpixelsdividedby200).Intensityvaluesarethenrecordedalongthesesegmentsusingbilinearinterpolation.Wenextfitasigmoidfunction(Fig.2)tothesevaluesusingarobustweightednonlinearregression.24ForeachlineI,theproblemcanbeformulatedas
argminS,W,xo,Io
!
x
"I(x)−Io−S
1+e−x−xoW
#,
wherexisthedistancealongthenormal,xoistheintersectionoftheboundarypointwiththenormal,I(x)istheintensityalongthenormalatx,andIoistheintensityoffset.
Twoparameters,scale(S)andwindow(W)fromthesig-moidarethenusedtocharacterizeeachlinesegmentI.Thescalemeasuresthedifferenceinintensitiesoutsideandinsidethelesion,andthewindowcharacterizesthemarginblurbymeasuringthetransitionfromthelesiontosurroundingnor-malorgantissueattheboundary.WesettherangeofWtobe
−2*Tto2*TandtherangeofStobe−1000HUto1000HU.Iffittedparametersareoutoftherange,wedonotincludetheresultintothefeaturevector.
II.A.2.Marginsharpnessfeatureimplementation
Themarginsharpnessofeachlesionisrepresentedasafeaturevectorcomposedof[two]30-binhistogramsofthescaleandwindowparametersobtainedfromeachnormal.Weremoveoutliersbydiscardingvaluesbelowthe5thpercentileandabovethe95thpercentileforeachparameterbeforecon-structingthehistogram.Eachhistogramisnormalizedtohaveunitarea.Thefeaturevectoristheconcatenationofthetwohistograms.
Itshouldbenotedthatincreatingthiscomputationalrep-resentationofthelesionboundary,itisassumedthateachnor-mallinesegmentincludesbothlesionsubstanceandnormalorgantissuesurroundingthelesion.However,forlesionsattheedgeofanorgan,aportionofthecircumferenceofthele-sionwillbeadjacenttosomestructureotherthanthenormalsurroundingorgan.Figure3(a)showssuchanexample.Duetothepossiblechangesinintensitybetweentheorgananditssurroundingtissue,thelinesegmentsdrawnontheborderoftheorgandonothaveenoughinformationtocharacterizethemarginofthelesion.Includingthescaleandwindowparam-etersobtainedfromthelinesegmentsfromthisportionofthe
(a)(b)(c)(d)
FIG.3.(a)CroppedlungCTimagedisplayedatstandardlungwindow-levelsetting,withROIoutlined(indarkgray)byradiologist.(b)Automaticallysegmentedlungregion(ingray)showingfillingdefectcausedbyinadequatesegmentationnearthelesionandtheROIoutline(inwhite).(c)Finallungsegmentationmask(theunionoftheautomaticallysegmentedlungmaskandthelesionmask).(d)CroppedlungCTimagewithlinesegments(inwhite)thatarecompletelyinsidethelungmask.
MedicalPhysics,Vol.39,No.9,September2012
[5]
J Pathol Inform 2013, 1:1 http://www.jpathinformatics.org/content/4/1/1
G x y g x y j f xcos ysinfθ σ π θ θ, ( , ) ( , )exp( ( ))= +2 (1)
where gσ(x,y) is a Gaussian kernel with a bandwidth of σ. The parameters f and θ represent frequency and orientation of the 2D Gabor filter, where θ varies between 0 and π in regular intervals, f ∈ F, and F denotes a set of possible frequencies, and is defined as follows.
F i N F iLi
C L( ) . / | ( ) ..= − < <−0 25 2 0 0 250 5 (2)
F i N F iHi
C H( ) . / | . ( ) ..= + < <−0 25 2 0 25 0 50 5 (3)
Where i Ncol= … ( )0 1 82, , , log / , Ncol is the width of the image in terms of the nearest power of 2. We then define the set F of possible frequencies as follows,
F F FL H= ∪ (4)
For an image with 512 columns, for example, a total of 84 Gabor filters can be used (six orientations and 14 frequencies). The hypocellular stromal features are then computed by convolving the Gabor filters Gθ,f(·) with Inorm
β (obtained from step 3 of Algorithm 1), and computing local energy on the results of the convolution.
Hypercellular Stromal FeaturesPhase information could be used as an important cue in modeling the textural properties of a region. Murtaza et al., used local frequency estimates in the Gabor domain over a range of scales and orientations to yield a signature, which was shown to efficiently characterize the texture of a village in satellite images.[16] We chose
the phase spectrum to represent the attributes of the HyperCS regions in a breast histology image, due to the recently established efficacy of the phase in textures exhibiting randomness.
Let vi(x,y) denote the ith Gabor channel for the stain normalized and smoothened version of an input image I(x,y), where i = ,1,2,...,Ng, N N Fg = ×θ | | and Nθ denotes the number of orientations. We can represent it as follows,
v x y v x y j x yi i i( , ) ( , ) exp( ( , ))= φ (5)
where |·| denotes the magnitude operator and φi x y( , ) denotes the local phase. The gradient of the local phase and its magnitude can then be computed as below,
φii
i
i
i
x yv x yv x y
v x yv x y
′′ ′
= −⎡
⎣⎢⎢
⎤
⎦⎥⎥
( , )( , )( , )
( , )( , )
(6)
and
φ φ φi
i ix yddx
ddy
′ = +( , )2 2
(7)
The phase gradient features are computed using (7) for each of the Gabor filter responses, over a window of size N × N(where N=15).
RanPEC SegmentationRanPEC is a fast, unsupervised, and data-independent framework for dimensionality reduction and clustering of high-dimensional data points.[9] The main idea of RanPEC is to project high-dimensional feature vectors onto a relatively small number of orthogonal random vectors belonging to a unit ball and perform ensemble clustering in the reduced-dimensional feature space. By getting an ensemble of projections for each feature vector and then picking a cluster for a pixel by the majority voting selection criterion, ensures stability of the results among different runs. Experimental results in[9] suggest that promising classification accuracy can be achieved by random projections when using fast matrix operations in an unsupervised manner.
RESULTS
All 35 images in the database were hand segmented by two expert pathologists. We generate all experimental results on three criteria: (1) Considering the first pathologist’s markings (P-1) as the ground truth (GT); (2) considering the second pathologist’s markings (P-2) as GT; (3) fusing P-1 and P-2 using the logical OR rule (i.e., a pixel is considered to be tumorous if any one of the two pathologists marked the pixel as tumorous), and considering the fused image as GT. Some of the HPF images contain large tumor regions with small islands of stroma here and there; however, a majority of HPF images contain a fair share of hypo- and hypercellular stroma (approximately 33%, on an average). The average
Figure 2: Overview of the proposed algorithm: HyMaP
Figure 1: A sample H & E–stained breast cancer histology image: (a) Original image, and (b) Overlaid image, with four types of contents shown in different colors. The tumor areas are shown in Red, HypoCS in Purple, and HyperCS in Green. Areas containing background or fat tissue are shown in white with black outline. Note the difference in morphology of the Hypo- and Hypercellular stromal regions
ba
[Downloaded free from http://www.jpathinformatics.org on Tuesday, June 16, 2015, IP: 128.179.146.236]
[7] [8]
COMPUTERIZED TEXTURE ANALYSIS
directionsscale
6
• Image scales and directions are important for visual texture discrimination
• Most approaches are leveraging these two properties
• Explicitly: Gray-level co-occurrence matrices (GLCMs), run-length matrices (RLE), directional filterbanks and wavelets, Fourier, histograms of gradients (HOG), local binary patterns (LBP)
• Implicitly: Convolutional neural networks (CNN), scattering transform, topographic independant component analysis (TICA)
COMPUTERIZED TEXTURE ANALYSIS
7
• Texture invariances: computer vision VS biomedical imaging
Computer vision Biomedical image analysis
scale scale-invariant multi-scale
rotation rotation-invariant rotation-invariant
[4] Histopathological image analysis: a review, Gurcan et al., IEEE Reviews in Biomed Eng, 2:147-71, 2009 [9] A sparse texture representation using local affine regions, Lazebnik et al., IEEE Trans on Pattern Anal and Mach Intel, 27(8):1265-78, 2005
160 IEEE REVIEWS IN BIOMEDICAL ENGINEERING, VOL. 2, 2009
Fig. 10. (a) A digitized histopathology image of Grade 4 CaP and different graph-based representations of tissue architecture via Delaunay Triangulation, VoronoiDiagram, and Minimum Spanning tree.
Fig. 11. Digitized histological image at successively higher scales (magnifica-tions) yields incrementally more discriminatory information in order to detectsuspicious regions.
or resolution. For instance at low or coarse scales color or tex-ture cues are commonly used and at medium scales architec-tural arrangement of individual histological structures (glandsand nuclei) start to become resolvable. It is only at higher res-olutions that morphology of specific histological structures canbe discerned.
In [93], [94], a multiresolution approach has been used for theclassification of high-resolution whole-slide histopathology im-ages. The proposed multiresolution approach mimics the eval-uation of a pathologist such that image analysis starts from thelowest resolution, which corresponds to the lower magnificationlevels in a microscope and uses the higher resolution represen-tations for the regions requiring more detailed information fora classification decision. To achieve this, images were decom-posed into multiresolution representations using the Gaussianpyramid approach [95]. This is followed by color space con-version and feature construction followed by feature extractionand feature selection at each resolution level. Once the classifieris confident enough at a particular resolution level, the systemassigns a classification label (e.g., stroma-rich, stroma-poor orundifferentiated, poorly differentiating, differentiating) to theimage tile. The resulting classification map from all image tilesforms the final classification map. The classification of a whole-slide image is achieved by dividing into smaller image tiles andprocessing each image tile independently in parallel on a clusterof computer nodes.
As an example, refer to Fig. 11, showing a hierarchicalcascaded scheme for detecting suspicious areas on digitizedprostate histopathology slides as presented in [96].
Fig. 12 shows the results of a hierarchical classifier for detec-tion of prostate cancer from digitized histopathology. Fig. 12(a)
Fig. 12. Results from the hierarchical machine learning classifier. (a) Originalimage with the tumor region (ground truth) in black contour, (b) results at scale1, (c) results at scale 2, and (d) results at scale 3. Note that only areas determinedas suspicious at lower scales are considered for further analysis at higher scales.
shows the original image with tumor outlined in black. The nextthree columns show the classifier results at increasing analysisscales. Pixels classified as “nontumor” at a lower magnification(scale) are discarded at the subsequent higher scale, reducingthe number of pixels needed for analysis at higher scales. Ad-ditionally, the presence of more discriminating information athigher scales allows the classifier to better distinguish betweentumor and nontumor pixels.
At lower resolutions of histological imagery, textural analysisis commonly used to capture tissue architecture, i.e., the overallpattern of glands, stroma and organ organization. For each digi-tized histological image several hundred corresponding featurescenes can be generated. Texture feature values are assignedto every pixel in the corresponding image. 3-D statistical, gra-dient, and Gabor filters can be extracted in order to analyzethe scale, orientation, and anisotropic information of the re-gion of interest. Filter operators are applied in order to extractfeatures within local neighborhoods centered at every spatiallocation. At medium resolution, architectural arrangement ofnuclei within each cancer grade can be described via severalgraph-based algorithms. At higher resolutions, nuclei and themargin and boundary appearance of ductal and glandular struc-tures have proved to be of discriminatory importance. Many ofthese features are summarized in Tables I and II.
D. Feature Selection, Dimensionality Reduction,and Manifold Learning
1) Feature Selection: While humans have innate abilities toprocess and understand imagery, they do not tend to excel at
COMPUTERIZED TEXTURE ANALYSIS
7
• Invariances: computer vision versus biomedical imaging
Computer vision Biomedical image analysis
scale scale-invariant multi-scale
rotation rotation-invariant rotation-invariant
[4] Histopathological image analysis: a review, Gurcan et al., IEEE Reviews in Biomed Eng, 2:147-71, 2009
160 IEEE REVIEWS IN BIOMEDICAL ENGINEERING, VOL. 2, 2009
Fig. 10. (a) A digitized histopathology image of Grade 4 CaP and different graph-based representations of tissue architecture via Delaunay Triangulation, VoronoiDiagram, and Minimum Spanning tree.
Fig. 11. Digitized histological image at successively higher scales (magnifica-tions) yields incrementally more discriminatory information in order to detectsuspicious regions.
or resolution. For instance at low or coarse scales color or tex-ture cues are commonly used and at medium scales architec-tural arrangement of individual histological structures (glandsand nuclei) start to become resolvable. It is only at higher res-olutions that morphology of specific histological structures canbe discerned.
In [93], [94], a multiresolution approach has been used for theclassification of high-resolution whole-slide histopathology im-ages. The proposed multiresolution approach mimics the eval-uation of a pathologist such that image analysis starts from thelowest resolution, which corresponds to the lower magnificationlevels in a microscope and uses the higher resolution represen-tations for the regions requiring more detailed information fora classification decision. To achieve this, images were decom-posed into multiresolution representations using the Gaussianpyramid approach [95]. This is followed by color space con-version and feature construction followed by feature extractionand feature selection at each resolution level. Once the classifieris confident enough at a particular resolution level, the systemassigns a classification label (e.g., stroma-rich, stroma-poor orundifferentiated, poorly differentiating, differentiating) to theimage tile. The resulting classification map from all image tilesforms the final classification map. The classification of a whole-slide image is achieved by dividing into smaller image tiles andprocessing each image tile independently in parallel on a clusterof computer nodes.
As an example, refer to Fig. 11, showing a hierarchicalcascaded scheme for detecting suspicious areas on digitizedprostate histopathology slides as presented in [96].
Fig. 12 shows the results of a hierarchical classifier for detec-tion of prostate cancer from digitized histopathology. Fig. 12(a)
Fig. 12. Results from the hierarchical machine learning classifier. (a) Originalimage with the tumor region (ground truth) in black contour, (b) results at scale1, (c) results at scale 2, and (d) results at scale 3. Note that only areas determinedas suspicious at lower scales are considered for further analysis at higher scales.
shows the original image with tumor outlined in black. The nextthree columns show the classifier results at increasing analysisscales. Pixels classified as “nontumor” at a lower magnification(scale) are discarded at the subsequent higher scale, reducingthe number of pixels needed for analysis at higher scales. Ad-ditionally, the presence of more discriminating information athigher scales allows the classifier to better distinguish betweentumor and nontumor pixels.
At lower resolutions of histological imagery, textural analysisis commonly used to capture tissue architecture, i.e., the overallpattern of glands, stroma and organ organization. For each digi-tized histological image several hundred corresponding featurescenes can be generated. Texture feature values are assignedto every pixel in the corresponding image. 3-D statistical, gra-dient, and Gabor filters can be extracted in order to analyzethe scale, orientation, and anisotropic information of the re-gion of interest. Filter operators are applied in order to extractfeatures within local neighborhoods centered at every spatiallocation. At medium resolution, architectural arrangement ofnuclei within each cancer grade can be described via severalgraph-based algorithms. At higher resolutions, nuclei and themargin and boundary appearance of ductal and glandular struc-tures have proved to be of discriminatory importance. Many ofthese features are summarized in Tables I and II.
D. Feature Selection, Dimensionality Reduction,and Manifold Learning
1) Feature Selection: While humans have innate abilities toprocess and understand imagery, they do not tend to excel at
[4][9]
• Averaging texture properties over the entire lesions or images discards tissue heterogeneity [10]
• Exhaustive and non-specific features
• Limited characterization of directions [6]
BIOMEDICAL TEXTURE ANALYSIS: LIMITATIONS (1/4)
8
[6] Three-dimensional solid texture analysis in biomedical imaging: review and opportunities, Depeursinge et al., Med Image Anal, 18(1):176-96, 2014 [10] Quantitative imaging in cancer evolution and ecology, Gatenby et al., Radiology, 269(1):8-15, 2013
feartures are not specific when not learned from data
• A global characterization of directions is not enough [11]
• local organization of image directions:
• independently from their local orientation:
• Rotation-covariance
• (local) grouped steering of the operators:
BIOMEDICAL TEXTURE ANALYSIS: LIMITATIONS (2/4)
image operators: grouped steering:
9
[11] Rotation–covariant texture learning using steerable Riesz wavelets, Depeursinge et al., IEEE Trans Imag Proc, 23(2):898-908, 2014
BIOMEDICAL TEXTURE ANALYSIS: LIMITATIONS (3/4)
GLCM contrast [1 0]
GLC
M c
ontra
st [0
1]
GLCMs (unaligned) Riesz (unaligned) Riesz (aligned)
10
BIOMEDICAL TEXTURE ANALYSIS: LIMITATIONS (4/4)
GLCM contrast [1 0]
GLC
M c
ontra
st [0
1]
GLCMs (unaligned) Riesz (unaligned) Riesz (aligned)
11
OBJECTIVES
12
• Highly adaptive texture-based computational models of biomedical tissue:
Complete coverage of image scales and directions in 2-D and 3-D
Rotation-covariance
Specificity: the models can be trained to characterize specific tissue types
Local characterization of tissue properties
Locate tissue properties in organ anatomy to create digital phenotypes
• Goal: predict survival, function, treatment response and reveal subtypes
be averaged over the folds of the CV and used to build texture models for each location along dROI (see Fig 9).The sum of the weights for all channels from each location will reveal the subregions that are specific to eachtumor subtypes, and most related to patient survival. A K–means clustering of the vectors w will be carried outfor each locations to evaluate the stability of the regional models over the folds of the CV and define homogeneousgroups among patients. A selection of the models based on stability and location importance will be carried out.
Fig. 9 Prototype tissue archi-tecture of a GBM tumor [128].
In a second step, the selected models will be locally steered to maximize theirmagnitude. The energies of the maximal magnitudes will be used to constructa final feature space for (1) predicting the tumor subtypes and (2) performingKaplan–Meier survival analysis. The performance of the proposed approach forpredicting tumor subtypes and patient survival will be compared to (1) unalignedwavelet energies and (2) average wavelet energies over the entire tumor. Whilestarting with a LOPO CV on the TCGA–TCIA dataset, the generalizability of theapproach will be further assessed by training with the TCGA–TCIA and testingwith the SU dataset.
Deliverable 2.1: Predicting tumor subtype and survival from localized of tissueproperties in GBM tumors.Task 2.2: Digital lung tissue atlases of ILD diagnoses (8 months of the PI)In this task, we will use a simple atlas of the lungs to locate texture properties and create prototype diagnosisphenotypes of ILDs. In previous work, we developed a simple 3–D atlas of the lungs with 36 subregionsthat will be used in this task [42]. In a first step, diagnosis–wise digital tissue atlases will be created bylearning 3–D texture models (i.e., the average of w over the folds of the LOPO CV) for each 36 regions ofthe lungs (see Fig. 10). The regions for which the models are most distant11 from all other diagnoses willbe highlighted and compared to previously built models of tissue patterns [25] to create 3–D prototype tissueatlases for each diagnosis. The obtained results will be validated using medical knowledge (e.g., Table 1 of [42]).
Fig. 10 Tissue atlas of the lungs.
For each diagnosis, K–means clustering of the vectors w will be carried outfor each locations to evaluate the stability of the regional models over the foldsof the CV and reveal homogeneous groups among patients (e.g., subtypes ofUIP). The most stable models will be kept for the further characterization oflung tissue types. A hierarchical clustering of the models from all diagnoseswill be carried out to define a radiomics–based hierarchy of all diagnoses, whichwill be compared to medical knowledge [140] (e.g., Fig. 1 of [40]). A largefeature space including the energies of the steered models from each of the 36localizations will be used to predict the diagnoses with uncertainty assessment(using e.g., pairwise coupling [141]). When a minimum amount of trust is notachieved when predicting a given diagnosis, the parent group of ILD diagnosesin the previously built hierarchy will be predicted instead [19].
Deliverable 2.2: Digital tissue atlases of ILD diagnoses and their subtypes.Task 2.3: Digital tissue atlases of ILDs: correlation with PFTs and survival (6 months )Digital tissue atlases will be constructed for poor versus normal/high (1) PFTs and (2) survival. Regions forwhich the models significantly di↵er between poor versus normal/high will be revealed as being of primaryimportance to evaluate pulmonary function. The links between these models and previously built models oftissue patterns will be investigated to define the combination of regions and patterns that are most responsiblefor lung function impairment. The feature space spanned by the energies of the steered models will be used topredict (1) PFT values or (2) survival with a LOPO CV, which can be evaluated using ground truth.
Deliverable 2.3: Estimating pulmonary function from digital tissue atlases of ILD diagnoses in CT images.
5.3 WP3. Imaging genomics
In this WP, we will use the texture models and digital tissue atlases developed in WP2 to learn and locatethe radiological patterns associated with prevailing meta–genes. This will allow for a deeper comprehension ofthe links between the radiological phenotypes and the expression of meta–genes. The molecular heterogeneityof GBM tumors ILD diagnoses will be studied using 3–D patch–based image analysis to enable the localcharacterization of radiogenomic properties.Task 3.1: Texture–based imaging genomics of GBM tumors (6 months of the PI)Based on the hypothesis that every meta–gene is associated with a specific visual pattern inside GBM tumorsin MR T1 images, patch–based analysis of the GBM tumors will be used to reveal local image regions thatcorrelate most with a given driver meta–gene. The gene expression is known to be di↵erent in perilesionaledema, on the tumor margin and in central necrosis [61, 130]. Gene expression patterns are also defining the
11in terms of Euclidean distance
10
8
(a) Original Image (b) Simple Cell Graph
(c) Voronoi Diagram (d) Delaunay Triangulation
(e) ECM-Aware Cell-Graph
Fig. 2 A fractured bone tissue example is shown in 2(a). Note the fracture cells in themiddle of the original image. The simple-cell-graph representation, the Voronoi diagramand the Delaunay triangulation for this sample tissue are depicted in 2(b), 2(c) and 2(d).The corresponding ECM-aware cell-graph is drawn in 2(e). The interactions between fracturecells are drawn with blue and the red cells with red color. Delaunay triangulation representsthe tissue as a single connected component and does not allow crossing of edges. Simple-cell-graphs relaxes these restrictions and allows the tissue to non-planar graph and disconnected.Likewise, ECM-aware cell-graphs do not put such restrictions on the tissue and moreovercan capture the structural organization of different cells in a tissue. Furthermore, Delaunaytriangulations are fixed representations whereas ECM-aware cell-graphs can be adjustedwith different linking thresholds.
8
(a) Original Image (b) Simple Cell Graph
(c) Voronoi Diagram (d) Delaunay Triangulation
(e) ECM-Aware Cell-Graph
Fig. 2 A fractured bone tissue example is shown in 2(a). Note the fracture cells in themiddle of the original image. The simple-cell-graph representation, the Voronoi diagramand the Delaunay triangulation for this sample tissue are depicted in 2(b), 2(c) and 2(d).The corresponding ECM-aware cell-graph is drawn in 2(e). The interactions between fracturecells are drawn with blue and the red cells with red color. Delaunay triangulation representsthe tissue as a single connected component and does not allow crossing of edges. Simple-cell-graphs relaxes these restrictions and allows the tissue to non-planar graph and disconnected.Likewise, ECM-aware cell-graphs do not put such restrictions on the tissue and moreovercan capture the structural organization of different cells in a tissue. Furthermore, Delaunaytriangulations are fixed representations whereas ECM-aware cell-graphs can be adjustedwith different linking thresholds.[12] Automated classification of usual interstitial pneumonia using regional volumetric texture analysis in high-
resolution CT, Depeursinge et al., Invest Radiol, 50(4):261-67, 2015 [13] ECM-Aware Cell-Graph Mining for Bone Tissue Modeling and Classification, Bilgin et al., Data Min Knowl Discov, 20(3):416-38, 2009
[12] [13]
Steerable texture models
Σassociated texture
model
EXAMPLE: PATTERN = WIGGLED CHECKERBOARD
pattern to learn:
30
Steerable texture models: our workhorse ….
13
• Multi-directional, multi-scale and rotation-covariant image analysis is achieved using Riesz wavelets
• The components of the th-order Riesz transform of a 2-D signal are defined in the Fourier domain as [14]:
• Yields allpass filters: only phase (i.e., direction) is kept and is defined by th-order partial derivatives
THE RIESZ TRANSFORM
14
[14] Wavelet Steerability and the Higher-Order Riesz Transform, Unser et al., IEEE Trans on Imag Proc, 19(3):636-52, 2010
• Riesz components
THE RIESZ TRANSFORM
input image
15
• Riesz components
THE RIESZ TRANSFORM
16
• Riesz components
THE RIESZ TRANSFORM
17
• Multi-scale filterbanks are obtained by combining the Riesz transform with isotropic wavelets [15]
• E.g., dyadic ( )
• Systematic coverage of image scales
THE RIESZ TRANSFORM
18
UNSER et al.: STEERABLE PYRAMIDS AND TIGHT WAVELET FRAMES IN 2711
of unity) guarantees that is dense in ; it is
essential for the -completeness of the wavelet decomposition.
We now proceed with the construction of orienta-
tion-free wavelets by projecting the multiresolution Riesz
basis onto some appropriate wavelet subspace
.
B. Isotropic Bandlimited Wavelet FramesThere are a number of constructions in the literature that
fall into this category [13], [27]–[29]. Before reviewing them,
we apply the aforementioned projection strategy to obtain a
straightforward design that is in direct correspondence with
Shannon’s sampling theorem, and that is the starting point for
the specification of Meyer-type wavelets.1) Construction of Isotropic, Shannon-Type Wavelets: By
selecting in (19),
we specify the so-called Shannon multiresolution analysis of
, which consists of a sequence of embedded subspaces
that are bandlimited to
We then define some corresponding wavelet subspaces of radi-ally bandpass functions
(20)
Since is a closed subspace of , we can apply Proposition
3 to its orthogonal sinc basis to obtain the tight wavelet frame
of with
(21)
where is the impulse response of the
ideal radial bandpass filter, whose frequency response is, i.e., the indicator function
corresponding to the spectral support of . Based on the
fact that is dense in , the design procedure
yields a tight wavelet frame of . The energy-preserving
condition that ensures that the latter is true is as follows:
(22)
It is automatically fulfilled, since the sequence of ideal radialbandpass filters constitutes a tilling of the frequency domain ,
as illustrated in Fig. 1. For a complete characterization of these
wavelets, we refer to the work of Papadakis et al. [27].2) Specification of Meyer-Type Wavelets: While the afore-
mentioned construction yields a tight isotropic wavelet frame
of , it has the drawback of producing wavelets with
poor spatial decay (e.g., ), due to the
sharp cutoff in frequency domain. A remarkable observation,
which can be traced back to the early work of Daubechies and
Meyer on frames [37], [38], is that this can corrected via an
appropriate adjustment of the radial bandpass filtering functions, which need not be indicator functions, as long as they
Fig. 1. Tiling of the 2-D frequency domain using radial-bandpass filters. Theshaded area corresponds to the spectral support of the wavelet subspace ;
it is included in the spectral support of (enclosing square).
satisfy (22). This leads to the following extended definition ofthe wavelet subspaces
which is equivalent to (20) if is the impulse response of
the ideal radial bandpass filter. Since can be written as
, there exists a sequence such that
where the wavelet functions are still given by (21). This
indicates that is a frame of , albeit not neces-
sarily a tight one. Yet, if Condition (22) is satisfied, then onerecovers the tight frame property over which is the
union of the wavelet subspaces , . The condition for
the wavelet frame to be isotropic is that the restriction of the fil-tering function over be isotropic, i.e.,
.
The aforementioned functional framework accounts for all
known constructions of isotropic wavelet frames of
which are summarized in Table I. The common feature of the
wavelet profile functions in Table I is that they are compactlysupported within the frequency interval (bandlimited
property) and that they satisfy a rescaled version of the partition
of unity condition
which is the key to ensuring the tight frame property [38]. Two
remarks are in order with respect to Simoncelli’s pioneering
design which is, by far, the solution most widely used in ap-
plications. First, the response of the filter is a warped versionof a bump [cf., Fig. 2(a)] and the furthest away from an ideal
bandpass filter; this simply reflects the fact that a design objec-tive for the steerable pyramid was to approximate the behavior
of a log-Gabor filterbank, which is a well-accepted model ofthe response properties of cortical cells in the mammalian vi-
sual system [39]. Second, Simoncelli and coworkers put a lot of
emphasis on finite-impulse response filter design in their initial
[15] Steerable pyramids and tight wavelet frames in , Unser et al., IEEE Trans on Imag Proc, 20(10):2705-21, 2011
THE RIESZ TRANSFORM
19
• Steerability [16]:
• Example :
• Higher-order steering can be done fully analytical
Complete coverage of the image directions
Steerability enables rotation-covariance[16] The design and use of steerable filters, Adelson et al., IEEE Trans on Pattern Anal and Mach Intel, 13(9):891-906, 1991
• A Riesz filterbank constitutes a dictionary of basic textures:
• Higher-level texture models are built from linear combinations of Riesz components
LEARNING TEXTURE MODELS
20
Σ
EXAMPLE: PATTERN = WIGGLED CHECKERBOARD
pattern to learn:
Σ
EXAMPLE: PATTERN = WIGGLED CHECKERBOARD
pattern to learn:
Σ
EXAMPLE: PATTERN = WIGGLED CHECKERBOARD
pattern to learn:
Σ
EXAMPLE: PATTERN = WIGGLED CHECKERBOARD
pattern to learn:
Σ
EXAMPLE: PATTERN = WIGGLED CHECKERBOARD
pattern to learn:
Σ
EXAMPLE: PATTERN = WIGGLED CHECKERBOARD
pattern to learn:
Σ
EXAMPLE: PATTERN = WIGGLED CHECKERBOARD
pattern to learn:
Σassociated texture
model
EXAMPLE: PATTERN = WIGGLED CHECKERBOARD
pattern to learn:
28
• Support vector machines (SVM) were used to learn (multi-scale )
versus
ONE-VERSUS-ALL SUPERVISED MODEL LEARNING [11]:
texture
all others
[11] Rotation-covariant texture learning using steerable Riesz wavelets, Depeursinge et al., IEEE Trans Imag Proc, 23(2):898-908, 2014
29
• 2-D synthetic textures with noise ( )
Captures specific local properties in terms of image scales and directions
LEARNING TEXTURE MODELS
30
• 2-D tissue from interstitial lung diseases (ILD) in CT [17]
• ( , 32x32 patches)
LEARNING TEXTURE MODELS
healthy emphysema ground glass fibrosis micronodules
3011 patches, 7 patients.
407 patches, 6 patients.
2226 patches, 32 patients.
2962 patches, 37 patients.
5988 patches, 16 patients.
31
[17] Multiscale lung texture signature learning using the Riesz transform, Depeursinge et al., Med Image Comput Comput Assist Interv (MICCAI), 15(3):517-24, 2012
• 3-D synthetic textures ( , based on [18])
• Vertical planes
• 3-D checkerboard
• 3-D wiggled checkerboard
LEARNING TEXTURE MODELS
32
[18] 3D Steerable Wavelets and Monogenic Analysis for Bioimaging, Chenouard et al., IEEE 8th Int Symp on Biomed Imag (ISBI), 2132-5, 2011
• Steering texture models:
STEERABLE TEXTURE MODELS [11]
The expression of the rotated texture model remains a linear combination of the initial Riesz components
[11] Rotation-covariant texture learning using steerable Riesz wavelets, Depeursinge et al., IEEE Trans Imag Proc, 23(2):898-908, 2014
• Steering texture models:
• Rotation-covariant models: local steering of
• Keep the maximum magnitude of the model
• Local quantitative features: energies/abs values of the magnitudes in the patch
STEERABLE TEXTURE MODELS [11]
34
The expression of the rotated texture model remains a linear combination of the initial Riesz components
[11] Rotation-covariant texture learning using steerable Riesz wavelets, Depeursinge et al., IEEE Trans Imag Proc, 23(2):898-908, 2014
Classification of 2-D natural texturesIEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. XX, NO. XX, XX 2013 5
1) canvas001 2) canvas002 3) canvas003 4) canvas005 5) canvas006 6) canvas009 7) canvas011 8) canvas021
9) canvas022 10) canvas023 11) canvas025 12) canvas026 13) canvas031 14) canvas032 15) canvas033 16) canvas035
17) canvas038 18) canvas039 19) tile005 20) tile006 21) carpet002 22) carpet004 23) carpet005 24) carpet009
Fig. 5. 128 128 blocks from the 24 texture classes of the Outex database.
1) canvas 2) cloth 3) cotton 4) grass 5) leather 6) matting 7) paper 8) pigskin
9) raffia 10) rattan 11) reptile 12) sand 13) straw 14) weave 15) wood 16) wool
Fig. 6. 16 Brodatz texture classes of the Contrib TC 00000 test suite.
180 180 images from rotation angles 20
, 70, 90, 120,135
and 150
of the other seven Brodatz images for eachclass. The total number of images in the test set is 672.
G. Experimental setupOVA SVM models using Gaussian kernels as K(x
i
,x
j
) =
exp(
||xixj ||22
2k
) are used both to learn texture signatures andto classify the texture instances in the final feature spaceobtained after k iterations. A number of scales J = 6
was used to cover the whole spectrum of the 128 128
subimages in Outex and J = 3 for covering the spectrum of16 16 subimages in Contrib TC 00000. The angle matrixthat maximizes the response of the texture signature at thesmallest scale
1
(x) (see Eq. (11)) is used to steer Riesztemplates from all scales. The dimensionality of the initialfeature space is J(N + 1). Every texture signature
N
c,K
iscomputed using the texture instances from the training set.The coefficients from all instances are rotated to locally aligneach signature N
c,K
and are concatenated to constitute the finalfeature space. The dimensionality of the final feature space isJ (N + 1) N
c
. OVA SVM models are trained in thisfinal feature space using the training instances. The remainingtest instances obtained are used to evaluate the generalizationperformance. All data processing was performed using MAT-LAB R2012b (8.0.0.783) 64–bit (glnxa64), The MathWorks
Inc., 2012. The computational complexity is dominated by thelocal orientation of N
c
in Eq. 11, which consists of finding theroots of the polynomials defined by the steering matrix A
.It is therefore NP–hard (Non–deterministic Polynomial–timehard), where the order of the polynomials is controlled by theorder of the Riesz transform N .
III. RESULTS
The performance of our approach is demonstrated withthe Outex and the Brodatz databases. The performance oftexture classification is first investigated in Section III-A.The evolution and the convergence of the texture signatures
N
c,k
through iterations k = 1, . . . , 10 is then studied inSection III-B for the Outex TC 00010 test suite.
A. Rotation–covariant texture classification
The rotation–covariant properties of our approach are eval-uated using Outex TC 00010, Outex TC 00012 and Con-trib TC 00000 test suites. The classification performanceof the proposed approach after the initial iteration (k=1)is compared with two other approaches that are based onmultiscale Riesz filterbanks. As a baseline, the classificationperformance using the energy of the coefficients of the initialRiesz templates was evaluated. Since the cardinality of the
APPLICATIONS (1/2)
35
• 24 classes, 180 images/class, 9 rotation angles in
• A SVM classifier is trained with unrotated images only
• 98.4% best acc.
• aligned models ( )
• Literature: 90-99%
• E.g., scattering transform: 98.75% [18]
2-D TEXTURE CLASSIFICATION: OUTEX DATABASE [17]
Errors occur with most stochastic textures
confusion matrix:
IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. XX, NO. XX, XX 2013 6
TABLE IAVERAGE Ac OBTAINED WITH THE LOCAL ORIENTATION MAXIMIZATION OF N
c,1 . THE PERFORMANCE OF EVEN AND ODD ORDERS IS COMPARED.
Outex TC 00010 Outex TC 00012 (P 000) Outex TC 00012 (P 001) Contrib TC 00000Ac for N = 2, 4, 6, 8, 10. 98± 0.7 97.2±0.7 98± 0.4 98.1±4.2Ac for N = 1, 3, 5, 7, 9. 94.4±0.7 93.6±0.6 95.1±0.4 89.9±1.8Ac for N = 1, . . . , 10. 96.2± 2 95.4± 2 96.6±1.6 94± 5.3
0.4
0.5
0.6
0.7
0.8
0.9
1
1 2 3 4 5 6 7 8 9 10
cla
ssifi
catio
n a
ccu
racy
N
aligned
aligned
initial Riesz
Nc,1
R(0,N)
Fig. 7. Classification performance with the Outex TC 00010 test suite. Thetwo rotation covariant approaches are performing much better than usingthe initial Riesz coefficients. The local orientation maximization of N
c,1
outperforms the local orientation of the first Riesz template R(0,N) asproposed in [8]. A maximum Ac of 98.4% is reached with N = 8.
classes are balanced, the classification accuracy A
c
is used asa performance measure of the methods. All performances aresummarized in Table I.
1) Outex TC 00010: The classification performance fororders N = 1, . . . , 10 is shown in Fig. 7. The performanceusing the energy of the coefficients that are maximizing theresponse of the first Riesz template (i.e., R(0,N)) at thesmallest scale was also evaluated as a first rotation–covariantapproach [8]. An average A
c
of 96.2 ± 2% is obtained withN = 1 . . . 10 and the local orientation maximization of
N
c,1
.A maximum A
c
of 98.4% is reached with N = 8.2) Outex TC 00012: The classification performance for
orders N = 1, . . . , 10 is shown in Fig. 8. Average A
c
of95.4 ± 2% (P 000) and 96.6 ± 1.6% (P 001) are obtainedwith N = 1 . . . 10 and the local orientation maximization of
N
c,1
. Maximum A
c
of 97.8% (P 000, N = 10) and 98.4%(P 001, N = 8) are reached.
3) Contrib TC 00000: The classification performance fororders N = 1, . . . , 10 is shown in Fig. 9. An average A
c
of 94 ± 5.3% is obtained with N = 1 . . . 10 and the localorientation maximization of
N
c,1
. A perfect classification ofthe test set (A
c
= 100%) is reached for orders N = 4, 6, 8, 10.
B. Convergence and iterative evolution of N
c,k
The visual appearance and convergence of the texture signa-tures N
c,k
is investigated using the Outex TC 00010 test suite.The assumption is made that after a given number of iterationsK,
N
c,k
will converge to a final template representing theessential “stitch” of the texture class. The evolutions of 8
c,k
forclasses “6) canvas009” and “15) canvas033” are represented in
0.4
0.5
0.6
0.7
0.8
0.9
1
1 2 3 4 5 6 7 8 9 10
cla
ssifi
catio
n a
ccu
racy
N
aligned (P_000)
initial Riesz (P_000)
aligned (P_001)
initial Riesz (P_001)
Nc,1
Nc,1
Fig. 8. Classification performance with the Outex TC 00012 test suitefor P 000 and P 001. Similarly to Outex TC 00010, the local orientationmaximization of N
c,1 performs much better than using the initial Rieszcoefficients for both problems. Maximum Ac of 97.8% (P 000, N = 10)and 98.4% (P 001, N = 8) are reached, which suggests high robustness toillumination changes of the proposed approach.
0.4
0.5
0.6
0.7
0.8
0.9
1
1 2 3 4 5 6 7 8 9 10
cla
ssifi
catio
n a
ccu
racy
N
aligned
initial Riesz
Nc,1
Fig. 9. Classification performance with the Contrib TC 00000 test suite.The importance of the rotation–covariance of the operators is highlightedonce more, where the local orientation maximization of N
c,1 also performsmuch better than using the initial Riesz coefficients. A perfect classification(Ac = 100%) is reached for orders N = 4, 6, 8, 10.
Figures 10 and 11, respectively. The convergence is assessedas the evolution of ||wkwk+1||
||w1|| . The average convergence ofall classes is shown in Fig. 12.
IV. DISCUSSIONS AND CONCLUSIONS
We developed a texture learning framework that leveragesa key property of visual pattern discrimination: the local or-ganizations of scales and directions. Localized comprehensivecharacterizations of scales and directions are enabled usingsteerable Riesz wavelets. Class–wise templates are learnedfrom the data: discriminative combinations of scales and direc-tions are revealed from one–versus–all SVMs. The local ori-entation of the obtained templates or “signatures” is optimizedto maximize their response, which is carried out analyticallywith linear combinations of the initial Riesz templates. The
IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. XX, NO. XX, XX 2013 5
1) canvas001 2) canvas002 3) canvas003 4) canvas005 5) canvas006 6) canvas009 7) canvas011 8) canvas021
9) canvas022 10) canvas023 11) canvas025 12) canvas026 13) canvas031 14) canvas032 15) canvas033 16) canvas035
17) canvas038 18) canvas039 19) tile005 20) tile006 21) carpet002 22) carpet004 23) carpet005 24) carpet009
Fig. 5. 128 128 blocks from the 24 texture classes of the Outex database.
1) canvas 2) cloth 3) cotton 4) grass 5) leather 6) matting 7) paper 8) pigskin
9) raffia 10) rattan 11) reptile 12) sand 13) straw 14) weave 15) wood 16) wool
Fig. 6. 16 Brodatz texture classes of the Contrib TC 00000 test suite.
180 180 images from rotation angles 20
, 70, 90, 120,135
and 150
of the other seven Brodatz images for eachclass. The total number of images in the test set is 672.
G. Experimental setupOVA SVM models using Gaussian kernels as K(x
i
,x
j
) =
exp(
||xixj ||22
2k
) are used both to learn texture signatures andto classify the texture instances in the final feature spaceobtained after k iterations. A number of scales J = 6
was used to cover the whole spectrum of the 128 128
subimages in Outex and J = 3 for covering the spectrum of16 16 subimages in Contrib TC 00000. The angle matrixthat maximizes the response of the texture signature at thesmallest scale
1
(x) (see Eq. (11)) is used to steer Riesztemplates from all scales. The dimensionality of the initialfeature space is J(N + 1). Every texture signature
N
c,K
iscomputed using the texture instances from the training set.The coefficients from all instances are rotated to locally aligneach signature N
c,K
and are concatenated to constitute the finalfeature space. The dimensionality of the final feature space isJ (N + 1) N
c
. OVA SVM models are trained in thisfinal feature space using the training instances. The remainingtest instances obtained are used to evaluate the generalizationperformance. All data processing was performed using MAT-LAB R2012b (8.0.0.783) 64–bit (glnxa64), The MathWorks
Inc., 2012. The computational complexity is dominated by thelocal orientation of N
c
in Eq. 11, which consists of finding theroots of the polynomials defined by the steering matrix A
.It is therefore NP–hard (Non–deterministic Polynomial–timehard), where the order of the polynomials is controlled by theorder of the Riesz transform N .
III. RESULTS
The performance of our approach is demonstrated withthe Outex and the Brodatz databases. The performance oftexture classification is first investigated in Section III-A.The evolution and the convergence of the texture signatures
N
c,k
through iterations k = 1, . . . , 10 is then studied inSection III-B for the Outex TC 00010 test suite.
A. Rotation–covariant texture classification
The rotation–covariant properties of our approach are eval-uated using Outex TC 00010, Outex TC 00012 and Con-trib TC 00000 test suites. The classification performanceof the proposed approach after the initial iteration (k=1)is compared with two other approaches that are based onmultiscale Riesz filterbanks. As a baseline, the classificationperformance using the energy of the coefficients of the initialRiesz templates was evaluated. Since the cardinality of the
[17] Multiresolution gray-scale and rotation invariant texture classification with local binary patterns, Ojala et al., IEEE Trans on Pattern Anal and Mach Intel, 24:7(971-87), 2002 [18] Combined scattering for rotation invariant texture analysis, Sifre et al., European Symposium on Artificial Neural Networks, 2012
Importance of rotation-covariance and learned models
Medulloblastoma tumor classification
APPLICATIONS (2/2)
Medulloblastoma Di↵erentiation Fusing UFL and Riesz Features 5
where m stands for the number of samples, and k is the number of classes, and is the weight decay parameter that penalizes large values for parameters. Thefused representation of an unseen test tissue tile x
(p) 2 R(N+1)Jk+l is classifiedas anaplastic (or non-anaplastic) by calculating a probability:
p(yp = 1|xp;) =exp (1x
(p))P2
l=1 exp (lx(p))
.
A tile belongs to the anaplastic class if p(yp|xp;) > 0.5 and non-anaplasticotherwise.
3 Experimental Results and Discussion
The workflow of the proposed approach is summarized in Fig. 2. As first step, wecompute the UFL features learned by TICA and the supervised features learnedwith Riesz wavelets for each image as described in Sections 2.1 and 2.2. OnceTICA and Riesz wavelets are computed a final step of supervised classification ismade using the combination of the computed features in a concatenated vectoras input for a standard softmax classifier as described in Section 2.3. Parametertuning is presented in Section 3.2.
[2.4] Medulloblastoma Image Cases
Fig. 2. Flowchart for MB feature extraction and classification for both learned repre-sentations: Riesz and TICA, the details of each stage are described in subsections.
3.1 Medulloblastoma Dataset
Our MB database is from St. Jude Childrens Research Hospital in Memphiswhere a neuropathologist manually annotated the cancerous regions of 10 pathol-ogy slides, 5 diagnosed as anaplastic and 5 as non-anaplastic MB. Slides werestained with hematoxylin and eosin (H&E) and digitization was done on anAperio Scanner obtaining WSI with a resolution of 80,00080,000 pixels. Eachimage can have several cancerous regions, which were manually annotated. Fortraining, we randomly extracted a total of 7,500 square regions (750 per case) of200200 pixels of the tumor regions (3,750 anaplastic and 3,750 non-anaplastic).
steerable Riesz texture models
topographic ICA
37
• Medulloblastoma (MB)
• Most common (i.e., 25%) pediatric brain tumor
• Major cause of death in pediatric oncology
• Histological subtypes of MB have different prognosis and treatments
• Anaplastic subtype is the worst!
• “marked nuclear pleomorphism, cell wrapping, high mitotic count, and abundant apoptotic bodies” [19]
MEDULLOBLASTOMA TUMORS [19]
38
[19] Childhood medulloblastoma: novel approaches to the classification of a heterogeneous disease, Ellison et al., Acta Neuropathol, 20(3):305-16, 2010
Pleomorphism (cytology): variability in the size and shape of cells and/or their nuclei
• Proposed approach [20]
• Anaplastic versus non-anaplastic
• Compare/combine steerable Riesz texture models with unsupervised topographic independent component analysis (TICA)
MEDULLOBLASTOMA TUMOR CLASSIFICATION
39
[20] Combining Unsupervised Feature Learning and Riesz Wavelets for Histopathology Image Representation: Application to Identifying Anaplastic Medulloblastoma, Otálora et al., Med Image Comput Comput Assist Interv (MICCAI), 2015
Medulloblastoma Di↵erentiation Fusing UFL and Riesz Features 5
where m stands for the number of samples, and k is the number of classes, and is the weight decay parameter that penalizes large values for parameters. Thefused representation of an unseen test tissue tile x
(p) 2 R(N+1)Jk+l is classifiedas anaplastic (or non-anaplastic) by calculating a probability:
p(yp = 1|xp;) =exp (1x
(p))P2
l=1 exp (lx(p))
.
A tile belongs to the anaplastic class if p(yp|xp;) > 0.5 and non-anaplasticotherwise.
3 Experimental Results and Discussion
The workflow of the proposed approach is summarized in Fig. 2. As first step, wecompute the UFL features learned by TICA and the supervised features learnedwith Riesz wavelets for each image as described in Sections 2.1 and 2.2. OnceTICA and Riesz wavelets are computed a final step of supervised classification ismade using the combination of the computed features in a concatenated vectoras input for a standard softmax classifier as described in Section 2.3. Parametertuning is presented in Section 3.2.
[2.4] Medulloblastoma Image Cases
Fig. 2. Flowchart for MB feature extraction and classification for both learned repre-sentations: Riesz and TICA, the details of each stage are described in subsections.
3.1 Medulloblastoma Dataset
Our MB database is from St. Jude Childrens Research Hospital in Memphiswhere a neuropathologist manually annotated the cancerous regions of 10 pathol-ogy slides, 5 diagnosed as anaplastic and 5 as non-anaplastic MB. Slides werestained with hematoxylin and eosin (H&E) and digitization was done on anAperio Scanner obtaining WSI with a resolution of 80,00080,000 pixels. Eachimage can have several cancerous regions, which were manually annotated. Fortraining, we randomly extracted a total of 7,500 square regions (750 per case) of200200 pixels of the tumor regions (3,750 anaplastic and 3,750 non-anaplastic).
steerable Riesz texture models
topographic ICA
• Unsupervised topographic independent component analysis (TICA) [21]
• Learn while minimizing cost function :
MEDULLOBLASTOMA TUMOR CLASSIFICATION
40[21] Energy correlations and topographic organization, Hyvärinen et al., Natural Image Statistics, 39:249-272, 2009
Medulloblastoma Di↵erentiation Fusing UFL and Riesz Features 3
Neural networks are the dominant approach for representation learning. How-ever there are other representation learning strategies that are able to adaptconventional image descriptors to the needs of a particular image analysis task.In [3], the authors propose a multiscale texture signature learning approach usingrotation-covariant Riesz wavelets, where most relevant combinations of orienta-tions and scales are learned directly from the data. This approach outperformedstate-of-the-art representations based on local binary patterns and grey levelcooccurrence matrices for lung tissue classification [4]. Drawbacks of the datadriven representations approaches are the amount of parameters involved thathave to be manually tuned, which requires more time in model training. Sometexture based representations fail to describe the feature patterns present intraining samples that the data-driven approach is able to find [1].
In this work, we propose a joint framework for classification of MB WSI,where the invariant properties of TICA features and the multiscale rotation-covariant properties of Riesz wavelet features complement each other. We hy-pothesize that this fusion can lead to a better classification performance. Thiswork join e↵orts of [3] and [1] in a simple manner to achieve the best accuracyreported for this histopathology WSI database.
2 Methodological Description
2.1 Topographic Independent Component Analysis
TICA is an unsupervised feature learning model, inspired by findings of the visualcortex behaviour. It groups activations of units in order to discover features thatare rotation and translation invariant [1]. These are appropriate features forhistopathology image characterization since shapes and cell organizations canbe present regardless of the position or orientation of cells. Particularly, TICAorganizes feature detectors in a square matrix for l groups such that adjacentfeature detectors activate in a similar proportion to the same stimulus. To learnsuch groups, we need to optimize the cost function:
JTICA(W) =
2
TX
i=1
WTWx
(i) x
(i)
2
2
+mX
i=1
lX
k=1
»Hk(Wx
(i))2 + (1)
where x
(i) 2 Rm is the i-th sample, T is the number of samples, W 2 Rnm isthe matrix that encodes the features in each row, and H 2 0, 1ln is the binary
topographic organization where H(j)k = 1, if the j-th feature detector, j-th row
of W , belongs to the k-th group, and 0 otherwise. This model sets H fixed whilelearning W. In addition, TICA has two main computational advantages. First,the only parameters to be tuned are the regularization hyperparameter and thesparsity controller . Second, it is an unconstrained optimization problem, whichcan be solved eciently by optimization techniques such as Limited memory-Broyden-Fletcher-Goldfarb-Shanno (L-BFGS).
11111111111111.11.11.111111..1111111111.11...11..11.11..111111111111111111.11siiáiiiisisiiisiii1111111111.iiiiiéiiáiiiiiiiái1111..111111.111111111111iiiéiiiiiiiiii®ii1111111111.11111111.1111.U11.1111iiééiiéiéviii111111111111111111.1111...11iiiiii11éisi11111111111111.11111111111111...1111111111111111.111111.111111..1111.s11.a11
\iriiiiéiiiiiiiiiiiiiio siiióiiiiïiiiiiiiii1=11.e1111MMIIIMS111111111111111MIN
11=1iiiiiiiiiiiiisiiiiiIO
ON1110111011
sRoMM
INIMI
11ErMIIM
MERIIM
Mvs
MMMMMMMMMMMMMMMMMMMiiiMUMMMiMMMiiiiiii iíiíi iiiíiiiii UWE:lMiiMME= M MEEis i iiiiíiiiiMóóóiiiMMMMMlMiMMMMMMMiMMMiMMMMMMMMMiiMMiMMiMMMiMMMiMMMiiMMMMiiiiiiiMiMMiMiMiiM\MiiMiiiiiiiiMMS MMMMiMMMMMMMiiMMMMiiiiiiiiiiiiiiitiiii
Figure 2. Local learned features with di↵erent UFL methods. Left: Sparse Autoencoders. Center: RICA. Right: T-RICAhighlighting some translational (blue), color (red), scale (yellow) and rotational (green) invariances.
2.4 Basal-cell carcinoma detection
In order to compare the proposed method with other state-of-the-art representation schemes, we fixed classifiermodel using the Softmax regression algorithm. Softmax regression is a multinomial classifier that generalizes thelogistic regression binary classifier.12 A softmax classifier was chosen because such model allows to interpret theprediction as a probability, as well as highlighting features related with each class based on learned weights.
3. EXPERIMENTAL EVALUATION
The experimental evaluation was performed over a dataset that comprises 1,407 histopathology images ofskin tissue biopsies at 10X magnification for Basal-cell carcinoma diagnosis (518 cancerous and 889 healthy)which were labeled by an expert pathologist. Experiments were performed on a 5-fold cross validation schemewith stratified sampling, and average of Sensitivity, F-Score(2 precisionrecall
precision+recall
) and Balanced Accuracy (BAC)
( sensitivity+specificity
2
) were reported. To perform result analysis, we compare our proposed method within BOFframework versus traditional DCT, Haar and Sparse Auto-Encoders descriptors for local feature representations.400 features were learned for each UFL Method (AE, RICA and T-RICA) and a 400 visual word BOF dictionarywas used for all experiments.
3.1 Topographic Local Learned Features Analysis
100,000 patches of 8 8 pixels were randomly sampled from training set, and 400 features were learned with atile size of 3 3 (group size in topography). The set of learned features is shown in Fig. 2. T-RICA learns inunsupervised way abstract concepts, like edges and texture patterns, directly from raw pixels. The topographicapproach organizes features in such a way that it discovers not only translational, but also rotational, color andscale invariances. The representation scheme described in section 2.2 exploits such organization to detect thoseinvariances in target image.
Images are described as histograms as explained in Section 2.3. The histogram representation is used as inputto a softmax classifier model. This model gives positive and negative weights to features which are proportionalto the relative importance for generating a classification. So, it is possible to map such learned weights overtopographic organization to highlight the most discriminant features as displayed in Fig. 3.
This organization suggests that T-RICA model captures and organizes relevant patterns of each class. Thisis a very interesting result since the model is totally unsupervised, i.e., the visual patterns were found usingnon-labelled samples.
Classification results are shown in Fig. 4 reporting the average of performance measures. Results show thatUFL representations are competitive with standard DCT descriptors. Though, adding topographic organizationconsiderably improves classification results in 7% with respect to traditional autoencoders, and 6% with respect
Patches were preprocessed using Zero-phase Component Analysis (ZCA) Whitening transformation. See details in13
Proc. of SPIE Vol. 8922 89220M-4
Downloaded From: http://proceedings.spiedigitallibrary.org/ on 06/18/2015 Terms of Use: http://spiedl.org/terms
JTICA(W )
reconstruction penaltyL2 topographic constraint: if the patch belongs to the local group of patchesHk = 1
x
(i)
k
~scale inv. ~rotational inv.
~translational inv.
W
• Dataset:
• 5 anaplastic, 5 non-anaplastic patients
• 750 patches (200x200) / patient
• Leave-2-patients-out cross-validation
• Softmax classifier
• Comparison with state-of-the-art [22]
• Convolutional neural networks (CNN), sparse autoencoders (sAE), bag of features (BOF), Haar, MR8, …
• Predictions over entire-slides:
MEDULLOBLASTOMA TUMOR CLASSIFICATION
41
[22] A comparative evaluation of supervised and unsupervised representation learning approaches for anaplastic medulloblastoma differentiation, Cruz-Roa et al., Proc. SPIE 9287, Int Symp Med Inf Proc Anal, 92870G, 2015
Medulloblastoma Di↵erentiation Fusing UFL and Riesz Features 7
Table 1. MB classification performance (baseline, Riesz, fusion). The measures areaveraged over the 20 test runs with standard deviation where available.
Method Accuracy Sensitivity Specificity
TICA + Riesz[N13 , N
22 , N
21 ] 0.997 ± 0.002 0.995 ± 0.004 1 ± 0
TICA [1] 0.972 ± 0.018 0.977 ± 0.021 0.967 ± 0.031
Riesz [N13 , N
22 , N
21 ] [3] 0.964 ± 0.038 0.999 ± 0.001 0.932 ± 0.07
Riesz [N13 ] [3] 0.958 ± 0.062 0.963 ± 0.05 0.916 ± 0.125
Riesz [N22 ] [3] 0.94 ± 0.02 0.94 ± 0.02 0.3 ± 0.04
2-Layer CNN [1] 0.90 ± 0.1 0.89 ± 0.18 0.9 ± 0.0.3
sAE [1] 0.90 0.87 0.93
BOF + A2NMF (Haar) [2] 0.87 0.86 0.87
Riesz [N21 ] [3] 0.85 ± 0.23 0.9 ± 0.15 0.7 ± 0.47
BOF + K - NN (Haar) [7] 0.80 - -
BOF + K - NN (MR8) [7] 0.62 - -
4 Concluding Remarks
We present a feature fusion between unsupervised feature learning and super-vised Riesz wavelet representation that captures subtle pattern of textures aswell as high level features, allowing to create a more separable feature spacewhere the di↵erentiation of medulloblastoma into anaplastic and non-anaplasticcan be made with high classification accuracy outperforming any other resultpreviously described in the literature. To our knowledge this is the first timethat a feature fusion method is presented between UFL and the Riesz waveletsin the context of histopathology image analysis showing the complementaritybetween these learned features for the challenging task of tumour di↵erentia-tion, we are currently working on extending the method to other patch-basedhistopathology image analysis problems with larger cohorts of patients.
Fig. 3. Predictions over two WSIs, non-anaplastic MB (left) and anaplastic (right).
[22]
[22]
[22]
[22]
[22]
Medulloblastoma Di↵erentiation Fusing UFL and Riesz Features 7
Table 1. MB classification performance (baseline, Riesz, fusion). The measures areaveraged over the 20 test runs with standard deviation where available.
Method Accuracy Sensitivity Specificity
TICA + Riesz[N13 , N
22 , N
21 ] 0.997 ± 0.002 0.995 ± 0.004 1 ± 0
TICA [1] 0.972 ± 0.018 0.977 ± 0.021 0.967 ± 0.031
Riesz [N13 , N
22 , N
21 ] [3] 0.964 ± 0.038 0.999 ± 0.001 0.932 ± 0.07
Riesz [N13 ] [3] 0.958 ± 0.062 0.963 ± 0.05 0.916 ± 0.125
Riesz [N22 ] [3] 0.94 ± 0.02 0.94 ± 0.02 0.3 ± 0.04
2-Layer CNN [1] 0.90 ± 0.1 0.89 ± 0.18 0.9 ± 0.0.3
sAE [1] 0.90 0.87 0.93
BOF + A2NMF (Haar) [2] 0.87 0.86 0.87
Riesz [N21 ] [3] 0.85 ± 0.23 0.9 ± 0.15 0.7 ± 0.47
BOF + K - NN (Haar) [7] 0.80 - -
BOF + K - NN (MR8) [7] 0.62 - -
4 Concluding Remarks
We present a feature fusion between unsupervised feature learning and super-vised Riesz wavelet representation that captures subtle pattern of textures aswell as high level features, allowing to create a more separable feature spacewhere the di↵erentiation of medulloblastoma into anaplastic and non-anaplasticcan be made with high classification accuracy outperforming any other resultpreviously described in the literature. To our knowledge this is the first timethat a feature fusion method is presented between UFL and the Riesz waveletsin the context of histopathology image analysis showing the complementaritybetween these learned features for the challenging task of tumour di↵erentia-tion, we are currently working on extending the method to other patch-basedhistopathology image analysis problems with larger cohorts of patients.
Fig. 3. Predictions over two WSIs, non-anaplastic MB (left) and anaplastic (right).
non-anaplastic anaplastic
TICA: unsupervised Riesz: rotation-covariant information
Small dataset
• Highly adaptive texture-based computational models of biomedical tissue:
Complete/systematic coverage of image scales and directions in 2-D and 3-D
Rotation-covariance
Specificity: the models can be trained to characterize specific tissue types
Local characterization of tissue properties
• Locate tissue properties in organ anatomy to create digital phenotypes
• localization systems, quant. graph analysis
CONCLUSIONS
model learning
Multi-resolution, steerability
patch-based analysis
42
be averaged over the folds of the CV and used to build texture models for each location along dROI (see Fig 9).The sum of the weights for all channels from each location will reveal the subregions that are specific to eachtumor subtypes, and most related to patient survival. A K–means clustering of the vectors w will be carried outfor each locations to evaluate the stability of the regional models over the folds of the CV and define homogeneousgroups among patients. A selection of the models based on stability and location importance will be carried out.
Fig. 9 Prototype tissue archi-tecture of a GBM tumor [128].
In a second step, the selected models will be locally steered to maximize theirmagnitude. The energies of the maximal magnitudes will be used to constructa final feature space for (1) predicting the tumor subtypes and (2) performingKaplan–Meier survival analysis. The performance of the proposed approach forpredicting tumor subtypes and patient survival will be compared to (1) unalignedwavelet energies and (2) average wavelet energies over the entire tumor. Whilestarting with a LOPO CV on the TCGA–TCIA dataset, the generalizability of theapproach will be further assessed by training with the TCGA–TCIA and testingwith the SU dataset.
Deliverable 2.1: Predicting tumor subtype and survival from localized of tissueproperties in GBM tumors.Task 2.2: Digital lung tissue atlases of ILD diagnoses (8 months of the PI)In this task, we will use a simple atlas of the lungs to locate texture properties and create prototype diagnosisphenotypes of ILDs. In previous work, we developed a simple 3–D atlas of the lungs with 36 subregionsthat will be used in this task [42]. In a first step, diagnosis–wise digital tissue atlases will be created bylearning 3–D texture models (i.e., the average of w over the folds of the LOPO CV) for each 36 regions ofthe lungs (see Fig. 10). The regions for which the models are most distant11 from all other diagnoses willbe highlighted and compared to previously built models of tissue patterns [25] to create 3–D prototype tissueatlases for each diagnosis. The obtained results will be validated using medical knowledge (e.g., Table 1 of [42]).
Fig. 10 Tissue atlas of the lungs.
For each diagnosis, K–means clustering of the vectors w will be carried outfor each locations to evaluate the stability of the regional models over the foldsof the CV and reveal homogeneous groups among patients (e.g., subtypes ofUIP). The most stable models will be kept for the further characterization oflung tissue types. A hierarchical clustering of the models from all diagnoseswill be carried out to define a radiomics–based hierarchy of all diagnoses, whichwill be compared to medical knowledge [140] (e.g., Fig. 1 of [40]). A largefeature space including the energies of the steered models from each of the 36localizations will be used to predict the diagnoses with uncertainty assessment(using e.g., pairwise coupling [141]). When a minimum amount of trust is notachieved when predicting a given diagnosis, the parent group of ILD diagnosesin the previously built hierarchy will be predicted instead [19].
Deliverable 2.2: Digital tissue atlases of ILD diagnoses and their subtypes.Task 2.3: Digital tissue atlases of ILDs: correlation with PFTs and survival (6 months )Digital tissue atlases will be constructed for poor versus normal/high (1) PFTs and (2) survival. Regions forwhich the models significantly di↵er between poor versus normal/high will be revealed as being of primaryimportance to evaluate pulmonary function. The links between these models and previously built models oftissue patterns will be investigated to define the combination of regions and patterns that are most responsiblefor lung function impairment. The feature space spanned by the energies of the steered models will be used topredict (1) PFT values or (2) survival with a LOPO CV, which can be evaluated using ground truth.
Deliverable 2.3: Estimating pulmonary function from digital tissue atlases of ILD diagnoses in CT images.
5.3 WP3. Imaging genomics
In this WP, we will use the texture models and digital tissue atlases developed in WP2 to learn and locatethe radiological patterns associated with prevailing meta–genes. This will allow for a deeper comprehension ofthe links between the radiological phenotypes and the expression of meta–genes. The molecular heterogeneityof GBM tumors ILD diagnoses will be studied using 3–D patch–based image analysis to enable the localcharacterization of radiogenomic properties.Task 3.1: Texture–based imaging genomics of GBM tumors (6 months of the PI)Based on the hypothesis that every meta–gene is associated with a specific visual pattern inside GBM tumorsin MR T1 images, patch–based analysis of the GBM tumors will be used to reveal local image regions thatcorrelate most with a given driver meta–gene. The gene expression is known to be di↵erent in perilesionaledema, on the tumor margin and in central necrosis [61, 130]. Gene expression patterns are also defining the
11in terms of Euclidean distance
10
CONCLUSIONS
• Importance of rotation-covariant information to model biomedical textures
• i.e., the local organization of directions
• Rotational invariance is not enough
• Called “roto-translation invariance” in [23]
TextureQbased'biomarkers:'current'limitaGons'
x Assume'homogeneous'texture'properGes'over'the'enGre'lesion'[5]'
'
x NonQspecific'features'x Global'vs'local'characterizaGon'of'image'direcGons'[6]'
Radiology: Volume 269: Number 1—October 2013 n radiology.rsna.org 13
REVIEW: Quantitative Imaging in Cancer Evolution and Ecology Gatenby et al
with the mean signal value. By using just two sequences, a contrast-enhanced T1 sequence and a fluid-attenuated inver-sion-recovery sequence, we can define four habitats: high or low postgadolini-um T1 divided into high or low fluid-at-tenuated inversion recovery. When these voxel habitats are projected into the tu-mor volume, we find they cluster into spatially distinct regions. These habitats can be evaluated both in terms of their relative contributions to the total tumor volume and in terms of their interactions with each other, based on the imaging characteristics at the interfaces between regions. Similar spatially explicit analysis can be performed with CT scans (Fig 5).
Analysis of spatial patterns in cross-sectional images will ultimately re-quire methods that bridge spatial scales from microns to millimeters. One possi-ble method is a general class of numeric tools that is already widely used in ter-restrial and marine ecology research to link species occurrence or abundance with environmental parameters. Species distribution models (48–51) are used to gain ecologic and evolutionary insights and to predict distributions of species or morphs across landscapes, sometimes extrapolating in space and time. They can easily be used to link the environ-mental selection forces in MR imaging-defined habitats to the evolutionary dy-namics of cancer cells.
Summary
Imaging can have an enormous role in the development and implementation of patient-specific therapies in cancer. The achievement of this goal will require new methods that expand and ultimately re-place the current subjective qualitative assessments of tumor characteristics. The need for quantitative imaging has been clearly recognized by the National Cancer Institute and has resulted in for-mation of the Quantitative Imaging Net-work. A critical objective of this imaging consortium is to use objective, repro-ducible, and quantitative feature metrics extracted from clinical images to develop patient-specific imaging-based prog-nostic models and personalized cancer therapies.
rise to local-regional phenotypic adap-tations. Phenotypic alterations can re-sult from epigenetic, genetic, or chro-mosomal rearrangements, and these in turn will affect prognosis and response to therapy. Changes in habitats or the relative abundance of specific ecologic communities over time and in response to therapy may be a valuable metric with which to measure treatment efficacy and emergence of resistant populations.
Emerging Strategies for Tumor Habitat Characterization
A method for converting images to spa-tially explicit tumor habitats is shown in Figure 4. Here, three-dimensional MR imaging data sets from a glioblastoma are segmented. Each voxel in the tumor is defined by a scale that includes its image intensity in different sequences. In this case, the imaging sets are from (a) a contrast-enhanced T1 sequence, (b) a fast spin-echo T2 sequence, and (c) a fluid-attenuated inversion-recov-ery (or FLAIR) sequence. Voxels in each sequence can be defined as high or low based on their value compared
microenvironment can be rewarded by increased proliferation. This evolution-ary dynamic may contribute to distinct differences between the tumor edges and the tumor cores, which frequently can be seen at analysis of cross-sec-tional images (Fig 5).
Interpretation of the subsegmenta-tion of tumors will require computa-tional models to understand and predict the complex nonlinear dynamics that lead to heterogeneous combinations of radiographic features. We have ex-ploited ecologic methods and models to investigate regional variations in cancer environmental and cellular properties that lead to specific imaging character-istics. Conceptually, this approach as-sumes that regional variations in tumors can be viewed as a coalition of distinct ecologic communities or habitats of cells in which the environment is governed, at least to first order, by variations in vascular density and blood flow. The environmental conditions that result from alterations in blood flow, such as hypoxia, acidosis, immune response, growth factors, and glucose, represent evolutionary selection forces that give
Figure 4
Figure 4: Left: Contrast-enhanced T1 image from subject TCGA-02-0034 in The Cancer Genome Atlas–Glioblastoma Multiforme repository of MR volumes of glioblastoma multiforme cases. Right: Spatial distribution of MR imaging–defined habitats within the tumor. The blue region (low T1 postgadolinium, low fluid-attenuated inversion recovery) is particularly notable because it presumably represents a habitat with low blood flow but high cell density, indicating a population presumably adapted to hypoxic acidic conditions.
Radiology: Volume 269: Number 1—October 2013 n radiology.rsna.org 13
REVIEW: Quantitative Imaging in Cancer Evolution and Ecology Gatenby et al
with the mean signal value. By using just two sequences, a contrast-enhanced T1 sequence and a fluid-attenuated inver-sion-recovery sequence, we can define four habitats: high or low postgadolini-um T1 divided into high or low fluid-at-tenuated inversion recovery. When these voxel habitats are projected into the tu-mor volume, we find they cluster into spatially distinct regions. These habitats can be evaluated both in terms of their relative contributions to the total tumor volume and in terms of their interactions with each other, based on the imaging characteristics at the interfaces between regions. Similar spatially explicit analysis can be performed with CT scans (Fig 5).
Analysis of spatial patterns in cross-sectional images will ultimately re-quire methods that bridge spatial scales from microns to millimeters. One possi-ble method is a general class of numeric tools that is already widely used in ter-restrial and marine ecology research to link species occurrence or abundance with environmental parameters. Species distribution models (48–51) are used to gain ecologic and evolutionary insights and to predict distributions of species or morphs across landscapes, sometimes extrapolating in space and time. They can easily be used to link the environ-mental selection forces in MR imaging-defined habitats to the evolutionary dy-namics of cancer cells.
Summary
Imaging can have an enormous role in the development and implementation of patient-specific therapies in cancer. The achievement of this goal will require new methods that expand and ultimately re-place the current subjective qualitative assessments of tumor characteristics. The need for quantitative imaging has been clearly recognized by the National Cancer Institute and has resulted in for-mation of the Quantitative Imaging Net-work. A critical objective of this imaging consortium is to use objective, repro-ducible, and quantitative feature metrics extracted from clinical images to develop patient-specific imaging-based prog-nostic models and personalized cancer therapies.
rise to local-regional phenotypic adap-tations. Phenotypic alterations can re-sult from epigenetic, genetic, or chro-mosomal rearrangements, and these in turn will affect prognosis and response to therapy. Changes in habitats or the relative abundance of specific ecologic communities over time and in response to therapy may be a valuable metric with which to measure treatment efficacy and emergence of resistant populations.
Emerging Strategies for Tumor Habitat Characterization
A method for converting images to spa-tially explicit tumor habitats is shown in Figure 4. Here, three-dimensional MR imaging data sets from a glioblastoma are segmented. Each voxel in the tumor is defined by a scale that includes its image intensity in different sequences. In this case, the imaging sets are from (a) a contrast-enhanced T1 sequence, (b) a fast spin-echo T2 sequence, and (c) a fluid-attenuated inversion-recov-ery (or FLAIR) sequence. Voxels in each sequence can be defined as high or low based on their value compared
microenvironment can be rewarded by increased proliferation. This evolution-ary dynamic may contribute to distinct differences between the tumor edges and the tumor cores, which frequently can be seen at analysis of cross-sec-tional images (Fig 5).
Interpretation of the subsegmenta-tion of tumors will require computa-tional models to understand and predict the complex nonlinear dynamics that lead to heterogeneous combinations of radiographic features. We have ex-ploited ecologic methods and models to investigate regional variations in cancer environmental and cellular properties that lead to specific imaging character-istics. Conceptually, this approach as-sumes that regional variations in tumors can be viewed as a coalition of distinct ecologic communities or habitats of cells in which the environment is governed, at least to first order, by variations in vascular density and blood flow. The environmental conditions that result from alterations in blood flow, such as hypoxia, acidosis, immune response, growth factors, and glucose, represent evolutionary selection forces that give
Figure 4
Figure 4: Left: Contrast-enhanced T1 image from subject TCGA-02-0034 in The Cancer Genome Atlas–Glioblastoma Multiforme repository of MR volumes of glioblastoma multiforme cases. Right: Spatial distribution of MR imaging–defined habitats within the tumor. The blue region (low T1 postgadolinium, low fluid-attenuated inversion recovery) is particularly notable because it presumably represents a habitat with low blood flow but high cell density, indicating a population presumably adapted to hypoxic acidic conditions.
[5]'QuanGtaGve'imaging'in'cancer'evoluGon'and'ecology,'Gatenby'et'al.,'Radiology,'269(1):8Q15,'2013'
5'
global'direcGonal'operators:' local'grouped'steering:'
[6]'RotaGonQcovariant'texture'learning'using'steerable'Riesz'wavelets,'Depeursinge'et'al.,'IEEE'Trans'Imag'Proc.,'23(2):898Q908,'2014.'
global directional operators local grouped steering
43
[11] Rotation-covariant texture learning using steerable Riesz wavelets, Depeursinge et al., IEEE Trans Imag Proc, 23(2):898-908, 2014[23] Rotation, Scaling and Deformation Invariant Scattering for Texture Discrimination, Sifre et al., IEEE Conf Comp Vis and Pat Rec (CVPR), 1233-40, 2013
sentation R(x) of x is invariant to the action of G if it is notmodified by the action of any g ∈ G: R(g.x) = R(x). It iscovariant to G if R(g.x) = g.R(x), where g acts on R(x)by shifting its coefficients. A separable invariant on a groupproduct G = G1G2 combines a first operator R1, which isinvariant to the action of G1 and covariant to the action G2,with a second operator R2 which is invariant to the actionof G2. Indeed for all g1.g2 ∈ G1 G2 and all images x(u):
R2(R1(g1.g2.x)) = R2(g2.R1(x)) = R2(R1(x)) .However, such separable invariants do not capture the jointproperty of the action of G2 relatively to G1, and may loseimportant information. This is why two-dimensional trans-lation invariant representations are not computed by cascad-ing invariants to horizontal and vertical translations. It isalso important for rotations and translations. Let us considerfor example the two texture patches of Figure 1. A separa-ble product of translation and rotation invariant operatorscan represent the relative positions of the vertical patterns,and the relative positions of the horizontal patterns, up toglobal translations. However, it can not represent the po-sitions of horizontal patterns relatively to vertical patterns,because it is not sensitive to a relative shift between thesetwo sets of oriented structures. It loses the relative positionsof different orientations, which is needed to be sensitive tocurvature, crossings and corners. Such a separable invariantthus can not discriminate the two textures of Figure 1.
Figure 1: The left and right textures are not discriminatedby a separable invariant along rotations and translations, butcan be discriminated by a joint roto-translation invariant.
Several authors [6, 7, 8] have proposed to take into ac-count the joint structure of roto-translation operators in im-age processing, particularly to implement diffusion oper-ators. Computing a joint invariant between rotations andtranslations also means taking into account the joint rela-tive positions and orientations of image structures, so thatthe textures of Figure 1 can be discriminated. Section 3introduces a roto-translation scattering operator, which iscomputed by cascading wavelet transforms on the roto-translation group.
Calculating joint invariants on large non-commutativegroups may however become very complex. Keeping a sep-arable product structure is thus desirable as long as it does
not lose too much information. This is the case for scaling.Indeed, local image structures are typically spread acrossscales, with a power law decay. This is the case for con-tours, singularities and most natural textures. As a result ofthis strong correlation across scales, one can use a separa-ble invariant along scales, with little loss of discriminativeinformation.
2.2. Hierarchical Architecture
We now explain how to build an affine invariant repre-sentation, with a hierarchical architecture. We separate vari-abilities of potentially large amplitudes such as translations,rotations and scaling, from smaller amplitude variabilities,but which may belong to much higher dimensional groupssuch as shearing and general diffeomorphisms. These smallamplitude deformations are linearized to remove them withlinear projectors.
Image variabilities typically differ over domains of dif-ferent sizes. Most image representations build localized in-variants over small image patches, for example with SIFTdescriptors [15]. These invariant coefficients are then ag-gregated into more invariant global image descriptors, forexample with bag of words [10] or multiple layers of deepneural network [4, 5]. We follow a similar strategy by firstcomputing invariants over image patches and then aggregat-ing them at the global image scale. This is illustrated by thecomputational architecture of Figure 2.
xroto-trans.
patchscattering
logglobal
space-scaleaveraging
deformat.invariant
linear proj.
Figure 2: An affine invariant scattering is computed by ap-plying a roto-translation scattering on image patches, a log-arithmic non-linearity and a global space-scale averaging.Invariants to small shearing and deformations are computedwith linear projectors optimized by a supervised classifier.
Within image patches, as previously explained, one mustkeep the joint information between positions and orienta-tions. This is done by calculating a scattering invariant onthe joint roto-translation group. Scaling invariance is thenimplemented with a global scale-space averaging betweenpatches, described in Section 4. A logarithmic non-linearityis first applied to invariant scattering coefficients to linearizetheir power law behavior across scales. This is similar to thenormalization strategies used by bag of words [10] and deepneural networks [5].
Because of three dimensional surface curvature in the vi-sual scene, the image patches are also deformed. A scat-tering transform was proved to be stable to deformations[9]. Indeed, it is computed with a cascade of wavelet trans-
[23]
[11]
• Rotation-covariance
• Local orientation of the models is computationally intensive
• 2D:
• 3D:
• Use graphics processor units (GPUs) [24]: 60x speedup
• Explore other steerable wavelet representations [25,26]
LIMITATIONS AND FUTURE WORK
44
G R(2,0,0) G R(0,2,0) G R(0,0,2) G R(1,1,0) G R(1,0,1) G R(0,1,1)
Fig. 1. Second–order Riesz kernels R(n1,n2,n3) convolved with isotropic Gaussian kernels G(x).
Support vector machines (SVM) are then used to classifybetween 9,347 normal and embolic cubic instances of lungparenchyma from 19 patients with APE and 8 control cases.
2. MATERIAL AND METHODS
2.1. Rotation–covariant texture analysis
3D multiscale Riesz filterbanks are used to characterize thetexture of the lung parenchyma in 3D at a given CT energylevel. The N–th order Riesz transform R(N) of a three–dimensional signal f(x) is defined in the Fourier domain as:
¤R(n1,n2,n3)f(!) =
…n1 + n2 + n3
n1!n2!n3!
(j!1)n1(j!2)
n2(j!3)
n3
||!||n1+n2+n3
ˆf(!),
(1)
for all combinations of (n1, n2, n3) with n1 + n2 + n3 = Nand n1,2,3 2 N. Eq. (1) yields
N+22
templates R(n1,n2,n3)
and forms multiscale steerable filterbanks when coupledwith a multi–resolution framework based on isotropic band–limited wavelets (e.g., Simoncelli) [12]. The second–orderRiesz filterbank is depicted in Fig. 1. Rotation–covariance isobtained by locally aligning the Riesz components R(n1,n2,n3)
of all scales based on the local prevailing orientation [13].The steerability property allows synthesizing the responses ofthe templates at any orientation based on a linear combinationof themselves. It therefore does not require additional convo-lutions of oriented filters [12]. Locally prevailing orientationsbased on the three Euler angles parameterizing SO(3) areestimated using a regularized version of the structure tensor,where the latter is computed over a 3D Gaussian window.The variance t of this Gaussian window determines thescale for the local orientation estimation. We use the imple-mentation of the regularized structure tensor by Chenouardet al. [11]. Locally aligned multiscale Riesz features are
denoted as ‹R(N)in the next sections.
2.2. Experimental setup
19 patients with APE and 8 control cases are used to evalu-ate the proposed approach. All 27 cases underwent a DECTscanner1 at the Department of Radiology of the University
1GE Discovery CT750 HD with two X–ray tubes and rapid peak kilovolt-age switching (gemstone spectral imaging, GSI).
Hospitals of Geneva with an inter–slice distance of 1mm, aslice thickness of 1.25mm, and a sub–millimetric resolutionin the axial plane. 11 energy levels are used from 40keV to140keV with a step of 10keV. All five lobes of each patienthave been manually segmented using the OsiriX software2.The perfusion levels of each lobe were quantified using theQanadli index (QI) on a lobe basis [14]. The QI is defined asthe sum of the scores of all arteries as: 0 if no occlusion isvisible, 1 if partially occluded, and 2 if totally obstructed.
DECT data of all energy levels are preprocessed to havean isotropic voxel resolution, which is obtained by dividingsamples along the z axis. All lobes are divided into 32
3
overlapping blocks to constitute a local instance of the lungparenchyma. A block is considered as valid when at least95% of its voxels belong to it. To obtain a sufficient numberof blocks for the middle right lobe, this rule was changed to90% due to its smaller size when compared to the other lobes.A total of 9,347 blocks are obtained for all lobes of all pa-tients. The classes of all blocks from a given lobe are definedas embolism if QI>0. Only blocks from control cases areused to represent the healthy class, as the redistribution of theblood flow in healthy lobes of embolism patients may alterparenchymal texture properties. Since the visual appearanceof the healthy parenchyma may vary from upper and lowerlobes due to gravity effects and anatomy, each lobe type(i.e., left, right, lower, middle, upper) is analyzed separately.Grey–level histograms (GLH) in Hounsfield Units (HU) ofthe extended lung window [1000; 512] HU with Nbins=50bins are used to characterize X–ray attenuations. GLH areproviding essential information to characterize hypo– andhyper–attenuated regions which respectively result fromunder–perfused oligemic areas of the lung or from shunt-ing of blood to irrigated parenchyma. The feature space iscomposed of the concatenation of the HU histogram bins andthe energies of the Riesz coefficients for all 11 DECT energylevels with a total dimensionality of 11 (Nbins +
N+22
).
SVMs with a Gaussian kernel are used to learn from themulti–energy features. Several parameters are exhaustivelyoptimized for each lobe: N , t, the cost C of the errorsof SVMs and the variance K of the associated Gaussiankernel K(xi,xj) = exp(
||xixj ||2122 ) as: N 2 [1; 4],
t = 0.75, 1.5, 3, C 2 [10
0; 10
18] and K 2 [10
0; 10
13].
A leave–one–patient–out cross–validation is used to estimatethe generalization performance of the proposed approach.
2http://www.osirix-viewer.com/, as of 31 October 2012.
IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. XX, NO. XX, XX 2013 10
RN
f
(x) =
0
BBBBBB@
R(0,N)
f
(x)
...R(n,Nn)
f
(x)
...R(N,0)
f
(x)
1
CCCCCCA
| z RNf(x)
=
0
BBBBBB@
A
0,0
. . . A
0,l
. . . A
0,N
......
...A
n,1
. . . A
n,l
. . . A
n,N
......
...A
N,1
. . . A
N,l
. . . A
N,N
1
CCCCCCA
| z A
0
BBBBBB@
R(0,N) f (x)...
R(l,Nl) f (x)...
R(N,0) f (x)
1
CCCCCCA
| z RNf(x)
.
(12)
ˆR(n,Nn)
=
N
n!(N n)!
(j(cos()!
1
+ sin()!
2
))
n
(j( sin()!
1
+ cos()!
2
))
Nn
||!||N
=
1
||!||N
N
n!(N n)!
nX
k1=0
Çn
k
1
å(cos())
k1(j!
1
)
k1(sin())
nk1(j!
2
)
nk1
NnX
k2=0
ÇN n
k
2
å(cos())
k2(j!
1
)
k2(sin())
Nnk2(j!
2
)
Nnk2
=
1
||!||N
N
n!(N n)!
nX
k1=0
NnX
k2=0
(j!
1
)
k1+k2(j!
2
)
Nk1k2
(1)
k2
Çn
k
1
åÇN n
k
2
å(cos())
Nnk2+k1(sin())
nk1+k2
=
nX
k1=0
NnX
k2=0
N
(k
1
+ k
2
)!(N k
1
k
2
)!
(j!
1
)
k1+k2(j!
2
)
Nk1k2
||!||N| z
ˆR(k1+k2,Nk1k2) (k
1
+ k
2
)!(N k
1
k
2
)!
n!(N n)!
(1)
k2n!
k
1
!(n k
1
)!
(N n)!
k
2
!(N n k
2
)!
(cos())
Nnk2+k1(sin())
nk1+k2.
(13)
ˆR(n,Nn)
(!) =
NX
l=0
N
l!(N l)!
(j!
1
)
l
(j!
2
)
Nl
||!||N| z
ˆR(l,Nl)
min(l,n)X
l1=max(0,lN+n)
(1)
ll1
l!(N l)!
l
1
!(n l
1
)!(l l
1
)!
(cos())
Nn+2l1l
(sin())
n2l1+l
| z A
n,l
.
(14)
[32] T. Leung and J. Malik. Representing and recognizing the visualappearance of materials using three–dimensional textons. InternationalJournal of Computer Vision, 43(1):29–44, 2001.
[33] L. Liu, L. Zhao, Y. Long, G. Kuang, and P. Fieguth. Extended localbinary patterns for texture classification. Image and Vision Computing,30(2):86–99, 2012.
[34] D. G. Lowe. Distinctive image features from scale-invariant keypoints.International Journal of Computer Vision, 60(2):91–110, 2004.
[35] S. G. Mallat. A theory for multiresolution signal decomposition: thewavelet representation. IEEE Transactions on Pattern Analysis andMachine Intelligence, 11(7):674–693, 1989.
[36] F. G. Meyer and R. R. Coifman. Brushlets: A tool for directional imageanalysis and image compression. Applied and Computational HarmonicAnalysis, 4(2):147–187, 1997.
[37] T. Ojala, T. Maenpaa, M. Pietikainen, J. Viertola, J. Kyllonen, andS. Huovinen. Outex – new framework for empirical evaluation oftexture analysis algorithms. In 16th International Conference on PatternRecognition, volume 1 of ICPR, pages 701–706, 2002.
[38] T. Ojala, M. Pietikainen, and T. Maenpaa. Gray scale and rotation
invariant texture classification with local binary patterns. In ComputerVision — ECCV 2000, volume 1842 of Lecture Notes in ComputerScience, pages 404–420. Springer Berlin Heidelberg, 2000.
[39] T. Ojala, M. Pietikainen, and T. Maenpaa. Multiresolution gray–scale and rotation invariant texture classification with local binary pat-terns. IEEE Transactions on Pattern Analysis and Machine Intelligence,24(7):971–987, 2002.
[40] P. Perona. Deformable kernels for early vision. IEEE Transactions onPattern Analysis and Machine Intelligence, 17(5):488–499, 1995.
[41] D. D. Y. Po and M. N. Do. Directional multiscale modeling of imagesusing the contourlet transform. IEEE Transactions on Image Processing,15(6):1610–1620, 2006.
[42] R. Porter and N. Canagarajah. Robust rotation–invariant texture classi-fication: wavelet, Gabor filter and GMRF based schemes. IEE Proc. onVision, Image and Signal Processing, 144(3):180–188, 1997.
[43] X. Qian, X.-S. Hua, P. Chen, and L. Ke. PLBP: An effective localbinary patterns texture descriptor with pyramid representation. PatternRecognition, 44(10–11):2502–2515, 2011.
[44] G. Quellec, M. Lamard, G. Cazuguel, B. Cochener, and C. Roux.
[24] GPU-accelerated texture analysis using steerable Riesz wavelets, Vizitiu et al., 11th Int Conf Par Proc and App Math (PPAM), 2015 (submitted)[25] A unifying parametric framework for 2D steerable wavelet transforms, Unser et al., SIAM Jour Imag Sci, 6(1):102-35, 2013 [26] Harmonic Singular Integrals and Steerable Wavelets in , Ward et al., App and Comp Harm Anal, 36(2):183-197, 2014
• Multi-scale
• Influence of surrounding objects: bandlimitedness VS compact support [27]
• Continuous band-limited scale characterization [28]
• Dyadic is not enough!
LIMITATIONS AND FUTURE WORK
45
Optimized Steerable Wavelets for Texture Analysis of Lung Tissue in 3-D CT: Classification of Usual Interstitial Pneumonia
Adrien Depeursinge1,2, Pedram Pad1, Anne S. Chin3, Ann N. Leung3, Daniel L. Rubin3, Henning Müller2, Michael Unser1
1Biomedical Imaging Group, EPFL, Switzerland, 2Institute of Information Systems, HES-SO, Switzerland, 3Department of Radiology, Stanford University, CA, USA.
Contact and more information: [email protected], http://bigwww.epfl.ch/
Background
• An accurate identification of classic usual interstitial pneumonia (UIP) in volumetric CT images can obviate the need for invasive lung biopsies [1]
• Identification of classic UIP:
Proposed scientific contributions
• Automated identification of classic UIPs based on regional multi-scale texture analysis in CT
• Balance between bandlimitedness of the texture operators and influence of surrounding objects
• Extension of maximally localized isotropic wavelet pyramids proposed in [2] to 3-D and for an adjustable radial bandwidth
window, can be seen in aminority of otherwise typical cases ofUIP23 (Fig. 3) and should be distinguished from the non-fibrotic linear and nodular pattern of calcification seen indendriform pulmonary ossification.24 The fibrotic pattern ofUIP is quite often asymmetric but never unilateral (Fig. 6).One of themore challenging features of diagnosingUIPwith
HRCT is determining the presence of honeycombing. A studyby Watadani et al25 showed that substantial interobservervariability exists for identifying honeycombing on HRCT, withkappa values ranging from 0.40-0.58. However, it is not clearwhether this study, which scored only single images, andselectively included atypical cases of honeycombing, is appli-cable to more general assessment of honeycombing. Thedefinition of honeycombing used by the Fleischner Society(clustered cystic airspaces that are usually subpleural withwell-defined walls and often with comparable diameters of3-10 mm but occasionally as large as 2.5 cm)26 should becarefully applied. Honeycombing must be distinguished fromparaseptal emphysema (which is usually associated with largercysts and usually more prominent in the upper lungs and
Figure 2 UIP pattern in a 75-year-old man. (A and B) Axial and coronal CT images show peripheral-predominant, basal-predominant reticular abnormalitywith honeycombing, typical forUIP. The honeycombing ismore evident on the coronalreconstructions. (C and D) Axial and coronal images obtained 3 years later show substantial progression of reticularabnormality and honeycombing.
Figure 3 UIP pattern with ossification. Axial CT image shows typicalfindings of UIP with peripheral-predominant reticular abnormalityand honeycombing. Numerous punctate calcifications are presentwithin the fibrosis.
D.A. Lynch and J.M. Huckleberry14
2 Depeursinge et al.
UIP, however, are found in a substantial proportion of cases ranging from 30%to 50% [2]. In this context, candidate selection for lung biopsy requires a multi-disciplinary consensus of clinicians and radiologists with extensive experience inintersitial lung diseases.
The classic computed tomography (CT) appearances of UIP is characterizedwith basal– and peripheral–predominant reticular abnormality and honeycomb-ing [2] (see Table 1). The fibrotic pattern of UIP is often asymmetric but rarelyunilateral. Importantly, a confident CT diagnosis also requires the absence ofatypical UIP findings. For example, disease that is predominant in the lungapices is atypical for UIP and would suggest other diagnoses such as sarcoidosis,hypersensitive pneumonia, pneumoconiosis or familial pulmonary fibrosis. Pat-terns of fibrosis with anterior predominance are related to respiratory distresssyndrome. Therefore, the accurate identification of classic UIP requires a de-tailed description of the parenchymal alterations and their anatomical locations,which can only be established by experienced thoracic radiologists. The charac-terization of lung tissue types such as honeycombing, reticulation and groundglass requires the subtle appreciation of three–dimensional tissue texture proper-ties (see Fig. 1), for visual inspection has provided little reproducibility [4]. Theimportance of relating these patterns to their anatomical location adds anotherlevel of complexity and is subject to high inter-observer variation.
Table 1. Radiological criteria for UIP [1].
Classic UIP (all required) Inconsistent with UIP (any)
• Peripheral, basal predominance • Upper or mid–lung predominance
• Reticular abnormality • Peribronchovascular predominance
• Honeycombing with or withouttraction bronchiectasis
• Extensive ground glass abnormality(extent > reticular abnormality)
• Absence of features listed asinconsistent with UIP pattern
• Profuse micronodules (bilateral, predominantlyupper lobes)
• Discrete cysts (multiple, bilateral, away fromareas of honeycombing)
• Di↵use mosaic attenuation/air–trapping(bilateral, in three or more lobes)
• Consolidation in bronchopulmonary segment(s)/lobe(s)
normal ground glass reticular honeycombing
Fig. 1. Common parenchymal appearances of UIP in MDCT.
The computerized recognition of lung tissue types in chest CT has been anactive research domain to provide assistance in image interpretation and enhancediagnosis sensitivity and specificity [5]. Whereas most of the studies are based onslice–based 2–D texture analysis, few of them are fully leveraging the wealth ofmodern multiple detector CT (MDCT) protocols using 3–D solid texture anal-
i) tissue type ii) tissue location
peripheral
basal
Experiments Patients • 33 patients: 15 classic and 18 atypical UIP (biopsy proven) • Volumetric CT resampled to 0.6x0.6x0.6mm3 (cubic spline interp.)
3-D rotation-covariant texture analysis of the parenchyma • 3-D Riesz-wavelet frames, defined in the Fourier domain as:
where is the 3-D Fourier transform of
• can be viewed as a multiscale gradient operator when coupled with an isotropic wavelet pyramid
• Yields a steerable filterbank • Rotation-covariance (represented by ) is achieved by
steering the Riesz operator at each voxel to maximize its local energy [3]
Regional lung texture analysis and UIP classification • Average energies of the coefficients computed for each 36
subregions of a basic lung atlas including peripheral and basal
• A generalized linear model estimates the class membership of as , where denotes classic UIP
• 10-fold cross-validation (200 repetitions) to estimate classification performance (AUC and ACC based on and respectively)
References
[1] An Official ATS/ERS/JRS/ALAT Statement: Idiopathic Pulmonary Fibrosis: Evidence-based Guidelines for Diagnosis and Management, G. Raghu et al., Am J Respir Crit Care Med 2011; 183(6):788-824
[2] VOW: Variance Optimal Wavelets for the Steerable Pyramid, P. Pad et al., IEEE ICIP 2014; 2973-2977 [3] 3D Steerable Wavelets and Monogenic Analysis for Bioimaging, N. Chenouard et al., IEEE ISBI 2011;
2132-2135 [4] A Parametric Texture Model Based on Joint Statistics of Complex Wavelet Coefficients, J. Portilla et al.,
Int Jour Comput Vision 2000; 1: 49-70 [5] Nonseparable radial frame multiresolution analysis in multidimensions and isotropic fast wavelet
algorithms, M. Papadakis et al., SPIE Wavelets 2003; 5207: 631-642 [6] Ten Lectures on Wavelets, I. Daubechies, SIAM 1992; 61
Methods
Adjustable maximally localized isotropic wavelet pyramids
• 3-D extension of [2] to be maximally localized over an adjustable radial bandwidth ,
• Minimize , where is the radial profile
• Constraints: • bandwidth limited to • generates tight frames
• Analytical approximation in Fourier:
Results
• The proposed texture operator achieved best AUC and ACC for and outperformed other popular isotropic wavelet pyramids
Conclusions and Perspectives
• New family of 3D isotropic wavelet pyramids with a tunable bandwidth to balance between the bandlimitedness of the texture operators and the influence of surrounding objects
• Importance of rotation-covariance and tuning of the wavelet bandwidth demonstrated for UIP classification
• Future work includes the extension of the texture operators to higher orders of the Riesz transform
0.55
0.6
0.65
0.7
0.75
0.8
0.85
0.9
0.95
1
π/4 3π/8 π/2
Ω
. .
. .
. .
. .
. .
. .
[27] Optimized steerable wavelets for texture analysis of lung tissue in 3-D CT: classification of usual interstitial pneumonia, Depeursinge et al., IEEE Int Symp on Biomed Imag (ISBI), 403-6, 2015[28] Fast detection and refined scale estimation using complex isotropic wavelets, Püspöki et al., IEEE Int Symp on Biomed Imag (ISBI), 512-5, 2015
spatial domain Fourier
(a) Synthetic image containing 3 visual concepts:
1) vertical lines (quadrants I and III),
2) checkerboard (quadrant II),
3) wiggled checkerboard (quadrant IV).
PCA 1
PC
A 2
101
102
103
(b) PCA visualization of 3232 overlapping blocks and clus-
ters from the left image (N = 10, J = 4, K = 3). The tem-
plates
10k corresponding to the respective visual concepts are
dislayed for scale j = 3.
Figure 5: Qualitative evaluation of the visual concepts 10
k found using K–means in the feature space spannedby the energies of the multi–scale Riesz components.
Figure 6: Classification accuracy with the Outex TC 00010 test suite. An optimal number of visual conceptsK = 20 and order N = 4 allowed an accuracy of 97.5%.
• Model learning
• Limited performance for stochastic textures with no clear multi-scale signature
• Reveal visual diversity with unsupervised learning [29,30]
LIMITATIONS AND FUTURE WORK
46
[29] Rotation-covariant visual concept detection using steerable Riesz wavelets and bags of visual words, Depeursinge et al., SPIE Wavelets and Sparsity XV, 8858:885816-885816-11, 2013 [30] Unsupervised texture segmentation using monogenic curvelets and the Potts model, Storath et al., IEEE Int Conf Imag Proc, 4348-52, 2014
(a) Synthetic image containing 3 visual concepts:
1) vertical lines (quadrants I and III),
2) checkerboard (quadrant II),
3) wiggled checkerboard (quadrant IV).
PCA 1
PC
A 2
101
102
103
(b) PCA visualization of 3232 overlapping blocks and clus-
ters from the left image (N = 10, J = 4, K = 3). The tem-
plates
10k corresponding to the respective visual concepts are
dislayed for scale j = 3.
Figure 5: Qualitative evaluation of the visual concepts 10
k found using K–means in the feature space spannedby the energies of the multi–scale Riesz components.
Figure 6: Classification accuracy with the Outex TC 00010 test suite. An optimal number of visual conceptsK = 20 and order N = 4 allowed an accuracy of 97.5%.
IEEETRANSACTIONSONIMAGEPROCESSING,VOL.XX,NO.XX,XX20135
1)canvas0012)canvas0023)canvas0034)canvas0055)canvas0066)canvas0097)canvas0118)canvas021
9)canvas02210)canvas02311)canvas02512)canvas02613)canvas03114)canvas03215)canvas03316)canvas035
17)canvas03818)canvas03919)tile00520)tile00621)carpet00222)carpet00423)carpet00524)carpet009
Fig.5.128128blocksfromthe24textureclassesoftheOutexdatabase.
1)canvas2)cloth3)cotton4)grass5)leather6)matting7)paper8)pigskin
9)raffia10)rattan11)reptile12)sand13)straw14)weave15)wood16)wool
Fig.6.16BrodatztextureclassesoftheContribTC00000testsuite.
180180imagesfromrotationangles20,70,90,120,135and150oftheothersevenBrodatzimagesforeachclass.Thetotalnumberofimagesinthetestsetis672.
G.ExperimentalsetupOVASVMmodelsusingGaussiankernelsasK(x
i
,x
j
)=
exp(
||xix
j||2
22k
)areusedbothtolearntexturesignaturesandtoclassifythetextureinstancesinthefinalfeaturespaceobtainedafterkiterations.AnumberofscalesJ=6
wasusedtocoverthewholespectrumofthe128128
subimagesinOutexandJ=3forcoveringthespectrumof1616subimagesinContribTC00000.Theanglematrixthatmaximizestheresponseofthetexturesignatureatthesmallestscale
1
(x)(seeEq.(11))isusedtosteerRiesztemplatesfromallscales.ThedimensionalityoftheinitialfeaturespaceisJ(N+1).Everytexturesignature
N
c,Kiscomputedusingthetextureinstancesfromthetrainingset.Thecoefficientsfromallinstancesarerotatedtolocallyaligneachsignature
N
c,Kandareconcatenatedtoconstitutethefinalfeaturespace.ThedimensionalityofthefinalfeaturespaceisJ(N+1)N
c.OVASVMmodelsaretrainedinthisfinalfeaturespaceusingthetraininginstances.Theremainingtestinstancesobtainedareusedtoevaluatethegeneralizationperformance.AlldataprocessingwasperformedusingMAT-LABR2012b(8.0.0.783)64–bit(glnxa64),TheMathWorks
Inc.,2012.Thecomputationalcomplexityisdominatedbythelocalorientationof
N
cinEq.11,whichconsistsoffindingtherootsofthepolynomialsdefinedbythesteeringmatrixA
.ItisthereforeNP–hard(Non–deterministicPolynomial–timehard),wheretheorderofthepolynomialsiscontrolledbytheorderoftheRiesztransformN.
III.RESULTS
TheperformanceofourapproachisdemonstratedwiththeOutexandtheBrodatzdatabases.TheperformanceoftextureclassificationisfirstinvestigatedinSectionIII-A.Theevolutionandtheconvergenceofthetexturesignatures
N
c,kthroughiterationsk=1,...,10isthenstudiedinSectionIII-BfortheOutexTC00010testsuite.
A.Rotation–covarianttextureclassification
Therotation–covariantpropertiesofourapproachareeval-uatedusingOutexTC00010,OutexTC00012andCon-tribTC00000testsuites.Theclassificationperformanceoftheproposedapproachaftertheinitialiteration(k=1)iscomparedwithtwootherapproachesthatarebasedonmultiscaleRieszfilterbanks.Asabaseline,theclassificationperformanceusingtheenergyofthecoefficientsoftheinitialRiesztemplateswasevaluated.Sincethecardinalityofthe
• THANKS !
47
Matlab code [email protected]
MICCAI tutorial on Biomedical Texture Analysis: Oct 5th in Münich
https://sites.google.com/site/btamiccai2015/
BIOMEDICAL TISSUE MODELING IN 2D AND 3D
• Interstitial lung diseases in CT • Lung texture classification using locally-oriented Riesz components, Depeursinge A, Foncubierta-
Rodriguez A, Van de Ville D, Müller H, Med Image Comput Comput Assist Interv. (MICCAI) 2011;14(3):231-8.
• Multiscale lung texture signature learning using the Riesz transform, Depeursinge A, Foncubierta-Rodriguez A, Van de Ville D, Müller H, Med Image Comput Comput Assist Interv. (MICCAI) 2012;15(3):517-24.
• Automated classification of usual interstitial pneumonia using regional volumetric texture analysis in high-resolution CT, Depeursinge A, Chin A, Leung A, Terrone D, Bristow M, Rosen G, Rubin D, Invest Radiol.,in press.
• Pulmonary embolism in dual-energy CT • Rotation-covariant texture analysis of 4D dual-energy CT as an indicator of local pulmonary
perfusion, Depeursinge A, Foncubierta-Rodriguez A, Vargas A, Van de Ville D, Platon A, Poletti PA,
48
BIOMEDICAL TISSUE MODELING IN 2D AND 3D
• Liver lesions in CT • Predicting visual semantic descriptive terms from radiological image data: preliminary results with
liver lesions in CT, Depeursinge A, Kurtz C, Beaulieu C, Napel S, Rubin D, IEEE Trans Med Imag. 2014;33(8):1669-76.
• Brain epileptogenic lesions in MRI • Epileptogenic lesion quantification in MRI using contralateral 3D texture comparisons, Jiménez del
Toro OA, Foncubierta-Rodríguez A, Vargas Gómez MI, Müller H, Depeursinge A, Med Image Comput Comput Assist Interv. (MICCAI) 2013;16(2):353-60.
49
EVEN VS ODD ORDERS: 1-D
signal: Heaviside
filter: 1st (dashed) and 2nd
order Gaussian derivatives
convolution: 1st (dashed) versus 2nd
50