nsf medix reu program medical imaging projects @ depaul cdm daniela s. raicu, phd associate...

NSF MedIX REU Program

Medical Imaging Projects@ DePaul CDM

Daniela S. Raicu, PhDAssociate Professor

Email: [email protected] URL: http://facweb.cs.depaul.edu/research/vc/

NSF MedIX REU Program, CDM, DePaul University

Outline

Medical Imaging (Computed Tomography)– Content-based and semantic-based image retrieval

• Projects 1 and 2

– Mappings from low-level image features to semantic concepts

• Projects 3 and 4

– Liver segmentation • Project 5


-

Definition of Content-based Image Retrieval:Content-based image retrieval is a technique for retrieving images on the basis of automatically derived image features such as texture and shape.

Content-based medical image retrieval (CBMS) systems

Applications of Content-based Image Retrieval:• Teaching• Research• Diagnosis• PACS and Electronic Patient Records


Feature Extraction

Similarity Retrieval

Image Features

[D1, D2,…Dn]Image Database

Query Image

Query Results

Feedback Algorithm

User Evaluation

Diagram of a CBIR


An image retrieval system can help when the diagnosis depends strongly on direct visual properties of images in the context of evidence-based medicine or case-based reasoning.

CBIR as a Diagnosis Aid


An image retrieval system will allow students/teachers to browse available data themselves in an easy and straightforward fashion by clicking on “show me similar images”. Advantages:

- stimulate self-learning and a comparison of similar cases- find optimal cases for teaching

Teaching files: • Casimage: http://www.casimage.com• myPACS: http://www.mypacs.net

CBIR as a Teaching Tool


CBIR as a Research Tool

Image retrieval systems can be used:• to complement text-based retrieval methods• for visual knowledge management whereby the images and associated textual data can be analyzed together

• multimedia data mining can be applied to learn the unknown links between visual features and diagnosis or other patient information

• for quality control to find images that might have been misclassified


CBIR as a tool for lookup and reference in CT chest/abdomen

• Case Study: lung nodules retrieval– Lung Imaging Database Resource for Imaging Research

http://imaging.cancer.gov/programsandresources/InformationSystems/LIDC/page7

– 29 cases, 5,756 DICOM images/slices, 1,143 nodule images – 4 radiologists annotated the images using 9 nodule

characteristics: calcification, internal structure, lobulation, malignancy, margin, sphericity, spiculation, subtlety, and texture

• Goals:– Retrieve nodules based on image features:

• Texture, Shape, and Size

– Find the correlations between the image features and the radiologists’ annotations


LIDC Semantic ConceptsCalcification 1. Popcorn

2. Laminated3. Solid4. Non-central5. Central6. Absent

Sphericity 1. Linear2. .3. Ovoid4. .5. Round

Internal structure

1. Soft Tissue2. Fluid3. Fat4. Air

Spiculation 1. Marked2. .3. .4. .5. None

Lobulation 1. Marked2. .3. .4. .5. None

Subtlety 1. Extremely Subtle2. Moderately Subtle3. Fairly Subtle4. Moderately Obvious5. Obvious

Malignancy 1. Highly Unlikely2. Moderately Unlikely3. Indeterminate4. Moderately Suspicious5. Highly Suspicious

Texture 1. Non-Solid2. .3. Part Solid/(Mixed)4. .5. Solid

Margin 1. Poorly Defined2. .3. .4. .5. Sharp


Extracted Image Features

Shape Features Size Features Intensity Features

Texture Features

Circularity Area MinIntensity 11 Haralick features calculated from co-occurrence matrices (Contrast, Correlation, Entropy, Energy, Homogeneity, 3rd Order Moment, Inverse Differential Moment, Variance, Sum Average, Cluster Tendency, Maximum Probability)

Roughness ConvexArea MaxIntensity

Elongation Perimeter MeanIntensity

Compactness ConvexPerimeter SDIntensity

Eccentricity EquivDiameter MinIntensityBG

Solidity MajorAxisLength MaxIntensityBG

Extent MinorAxisLength MeanIntensityBG 24 Gabor features - mean and standard deviation of Gabor filters consistency of four orientations and three scales.

RadialDistanceSD SDIntensityBG

IntensityDifference


Lung nodule representation


Choose a nodule


Choose an image feature& a similarity measure

NSF MedIX REU Program, CDM, DePaul UniversityRetrieved Images


CBIR systems: challenges & REU projects

•Type of features• image features:

- texture features: statistical, structural, model and filter-based

- shape features• textual features (such as physician annotations)

Project 1: Feature reduction for medical image processing- Investigate how many features with respect to the number of unique nodules- Investigate what the most important low-level image features are with respect to the retrieval process - Investigate the uniformity of the features with respect to the same

type of nodules.


CBIR systems: challenges & REU projects (cont.)

•Similarity measures-point-based and distribution based metrics

• Retrieval performance:• precision and recall• clinical evaluation

Project 2: Evaluation of CBIR and SBIR systems• Perform a literature review on the current techniques used to evaluate CBIR

systems both for the general and medical domain• Investigate ways to include radiologists’ feedback in the retrieval process• Investigate ways to evaluate the retrieval process by varying various

numbers of parameters such as number of images retrieved, cutoff value for acceptable precision and recall, and minimum number of radiologists/observers needed to evaluate the system.


Correlations between Image Features and Concepts

0.52, 0.52, 0.52, 0.53, 0.51, 0.51,

0.49

0.48, 0.48, 0.48, 0.47, 0.47, 0.47,

0.46

-0.42, -0.42, 0.34, 0.30

Image FeaturesLobulation

Margin

Spiculation

Sphericity

Malignancy

Texture

Subtlety

InternalStructure

Calcification

Characteristics

Eccentricity, Elongation, Extent, Circularity

Area, ConvexArea, EquivDiameter, MinorAxisLength, ConvexPerimeter, Perimeter, MajorAxisLength

0.65

0.62

0.47


Automatic Mappings Extraction

Step-wise multiple regression analysis was applied to generate prediction models for each characteristic ci based on all image features fk:

kwhere p is the # of used image features, are the regression coefficients, and are the prediction errors per model.

ipk

kkii fcM ,1

0:

i

1

111_ 22

pn

nRRadj

Goodness of fit for the regression model:


Regression Models

Characteristics

Entire dataset(1106 images, 73

nodules)

At least 2 radiologists agreed

At least 3 radiologists agreed

Calcification 0.397 0.578 (884, 41) 0.645 (644, 21)

Internal Structure

0.417 - (855, 40) - (659, 22)

Lobulation 0.282 0.559 (448, 24) 0.877 (137, 6)

Malignancy 0.310 0.641 (489, 23) 0.990 (107, 5)

Margin 0.403 0.376 (519, 28) - (245, 7)

Sphericity 0.239 0.481 (575, 27) 0.682 (207, 9)

Spiculation 0.320 0.563 (621, 29) 0.840 (228, 9)

Subtlety 0.301 0.282 (659, 25) 0.491 (360, 10)

Texture 0.181 0.473 (736, 33) 0.843 (437, 15)


Texture Regression Model


Malignancy Regression Model

Calcification

Lobulation

InternalStructure

Malignancy

Margin

Spiculation

Sphericity

Texture

Subtlety

Characteristics

Regression Coefficients p-value

F-value = 963.560p-value = 0.000

Estimated Malignancy = 5.377275 - 0.02069 Gabormean_45¼_0.5 + 0.003819 MinIntensityBG - 28.5314 Energy - 0.00315 Gabormean_0¼_0.4 + 0.000272 IntesityDifference + 6.317133 InverseVariance + 0.009743 Gabormean_45¼_0.4 - 0.00667 Gabormean_90¼_0.4 - 0.39183 Correlation + 5.16E-06 ClusterTendency - 0.00291 ConvexPerimeter

Adj_R2 = 0.990

(Constant ) 5.377275 1.64E-54Gabormean_45¼_0.5 -0.02069 7.80E-07MinIntensityBG 0.003819 3.30E-82Energy -28.5314 3.31E-12Gabormean_0¼_0.4 -0.00315 5.80E-14IntesityDifference 0.000272 0.003609InverseVariance 6.317133 3.41E-05Gabormean_45¼_0.4 0.009743 0.000259Gabormean_90¼_0.4 -0.00667 5.79E-05Correlation -0.39183 5.67E-05ClusterTendency 5.16E-06 0.000131ConvexPerimeter -0.00291 0.023032


Lobulation Regression Model


Spiculation Regression Model


Image Features – Semantics Mappings: challenges & REU projects

Project 3: Multi-view learning classifier for lung nodule classification

• Investigate which image features are the best for individual semantic characteristics, build classifiers for each one of the individual classifiers, and combine the individual classifies for optimal learning/classification of lung nodules

Project 4: Bridging the semantic gap in lung nodule interpretation

• Investigate ways to clinically evaluate the mappings from low-level image features to semantic characteristics

• Investigate the effect of the imaging acquisition parameters (such as pitch, FOV, and reconstruction kernel) on the proposed mappings


-

Pixel-level Classification: - tissue segmentation - context-sensitive tools for radiology reporting

Liver Segmentation in CT images

Pixel Level Texture Extraction

Pixel Level Classification Organ Segmentation

1 2, , kd d d _tissue label


Liver Segmentation in CT images

Example of Liver Segmentation: (J.D. Furst, R. Susomboon, and D.S. Raicu, "Single Organ Segmentation Filters for Multiple Organ Segmentation", IEEE 2006 International Conference of the Engineering in Medicine and Biology Society (EMBS'06))

Region growing at 70% Region growing at 60% Segmentation Result

Original Image Initial Seed at 90% Split & Merge at 85% Split & Merge at 80%


Liver Segmentation using Automatic Snake a)

b) c) d)

Figure: a) Gradient vector flow segmentation; b) Traditional vector field segmentation; c) and,d) Respective segmentations overlaid on ground truth (white).

a)

Project 5: Automatic selection of initial points for snake-based segmentation


uestions ?

nsf medix reu program medical imaging projects @ depaul cdm daniela s. raicu, phd associate...

Documents

contentbased image retrieval

depaul university cbir

lowlevel image features

derived image features

depaul cdm daniela

cbir slide

depaul university definition

textbased retrieval