an agent based architecture for designing flexible image analysis systems jagath samarabandu dept....

An Agent Based Architecture For Designing Flexible Image Analysis Systems

Jagath Samarabandu

Dept. of Electrical and Computer EngineeringUniversity of Western Ontario

Multi-dimensional Imaging System

Capture Processing Visualization

Storage

Confocal Microscopy

Conventional Microscopy with Deconvolution

Tomography with Reconstruction

MR, CT, PET, SPECT, US

ROI Extraction

Digital Filtering

Segmentation

Analysis

Morphometry

Sectional Views

MIP/Sum Projections

Surface Shading

Volume Rendering

Stereo views

Animation Sequences

Interactive displays

Raw data

Region dataProcessed dataHigh level data

Single frameStereo frameMovie sequences

Acknowledgements Kamal Ranaweera – MESc student who

is doing most of the work Dr. Aaron Fenster for his mentorship CITO Round 3.2 research grant

Outline Why image analysis Image processing basics Image analysis paradigm Analysis system example A flexible framework for image analysis Knowledge organization Current status Outlook

Why Image Analysis Image dimensions are increasing Image sizes and resolutions are

increasing Image quality is getting better Cannot visually assess all the voxels

Image Processing Point Operations

Histogram Equalization Contrast Stretching

Spatial Operations Linear Filtering

Low/High pass filtering Edge Detection

Non-Linear Filtering Median filtering Morphological Filtering

Point OperationsHistogram EqualizationIntensity histogram of the output image has a uniform distribution

Spatial Operations Linear Filtering

g(x,y) = ijh(i,j) u(i+x, j+y) where u(x,y) is the input image intensity at

(x,y) and h(i,j) is the value of convolution kernel at (i,j)

1 1 11 1 11 1 1

-1 0 1-1 0 1-1 0 1

-1 -1 -1 0 0 0 1 1 1

MovingAverage

Pair of kernels for edge detection

Example kernels

Edge Detection Operators Gradient Operators

Finite difference approximations of orthogonal gradients (e.g.. hx and hy for 2D images)

gm (x,y) = gx(x,y)2 + gy(x,y)2

g(x,y) = tan -1 (gy(x,y) / gx(x,y))

-1 0 1 -1 -1 -1-1 0 1 0 0 0-1 0 1 1 1 1

-1 0 1 -1 -1 -1-1 0 1 0 0 0-1 0 1 1 1 1

0 1 1 0-1 0 0 -1

Roberts Prewitt Sobel

Edge Detection Operators Compass Operators

Directional gradients at a given number of directions (h1, h2, h3 .. hn)

gm (x,y) = MAXk {| gk (x,y)|} g(x,y) is k at which the maximum occurred

-1 0 1-1 0 1-1 0 1

0 1 1-1 0 1-1 -1 0

1 1 1 0 0 0-1 -1 -1

1 1 0 1 0 -1 0 -1 -1

1 0 -1 1 0 -1 1 0 -1

0 -1 -1 1 0 -1 1 1 0

-1 -1 -1 0 0 0 1 1 1

-1 -1 0-1 0 1 0 1 1

Kirsh templates for 8 directions

Edge Operators

Roberts Prewitt Sobel

Compass Kirsch Laplace

Spatial Operators Linear Filters

Gaussian smoothing Edge detection operators

Non Linear Filters Median filter Rank order filters morphological filters

Spatial Filters for Smoothing

Gaussian Noise Gaussian (7x7) Median (5x5)

Speckle noise Median (5x5)Gaussian (7x7)

High Level Processing Feature Extraction

Edges and Boundaries Shape and Moments Texture

Segmentation Matching Thresholding Boundary Detection

Classification Statistical Classification Clustering

Canny’s Edge OperatorCombines edge operators and adaptive thresholding with

hysterisys to locate edges.

Image Analysis

PictureInterpretation

Image

Picture DomainCues

Scene DomainCues

Model

InstantiatedModel

Scene Domain

Picture Domain

Signal

Physical

Semantic

Kanade’s paradigm

Bottom Up Cycle Picture domain cues are primitive image

features Scene domain cues are the

interpretation of image features This distinction is somewhat blurred for

3D images

Top Down Cycle Model instantiation or hypothesis

generation Verification and refinement

Data Driven Approach Example

Median Filtering

Difference of Gaussian

Low Pass filter

Boundary Forming

Thresholding

Feature Aggregation

Feature Aggregation

This technique was able to suppress concavities

It also tended to diffuse boundaries of small structures

lyxyxH

L

yxyxl

lyx

lyx

lyx

yxeE

i

iii

iii

i

Ekwii

,| and

Otherwise

if

if

Where

,,max

min

,,

1,

1,,

00

0000

Processing Confocal Images

Single section of the original image

2D median filtering

Processing Confocal Images

Difference of Gaussian filter 3D contours of final segmentation

Detecting DNA Replication Sites Bottom up

Intensity maxima are detected on 2D image slices

Sites cross sections are modeled as circles centered on these maxima

Top Down Site boundaries are then refined according to the

local image features These boundaries are then combined to form 3D

structures using 3D connected component labeling

DNA Replication Sites (MIP)

Segmented Replication Sites

Further Analysis

Further Analysis

Blackboard Model Provides us an elegant architecture

within which both data driven and model driven approaches can be incorporated

A problem is solved in an incremental fashion

Contains a set of knowledge sources or tools

Blackboard Model These tools are designed as

independent modules They interact with each other via a

global database called the blackboard Modules that generate solutions are

separated from those that determine their utility

Blackboard Models can handle Large solution space Noisy and unreliable data Variety of input data and a need to

integrate diverse information The need to use many independent or

semi-independent sources of knowledge to co-operate in forming a solution

Blackboard Models can handle The need to use multiple reasoning

methods (eg. Forward and backward) The need for multiple lines of reasoning The need for an evolutionary solution

Requirements for the Framework

Represent and use knowledge about task domain and processing tools

Evaluate the quality of the image processing algorithms

Transfer information between different tools and knowledge sources

Control the flow of processing Allow easy addition of new tools

Requirements for Processing Tools

Implements the underlying processing algorithm

Contains knowledge about its algorithm Can evaluate its suitability (matching

score) to accomplish a given task Communicates with the framework

Knowledge Organization Domain Expertise

Knowledge about the image Intensity characteristics of the structure of

interest Image Processing Expertise

Knowledge about image processing algorithms (e.g. filtering, detection) and their parameters

Domain Knowledge Initial Model (e.g. circle, line, sphere) Parameter value distribution Expected orientation Boundary definition method (e.g.

intensity +/-, texture, color etc.) Boundary definition strength (strong,

medium, weak) Boundary dispersion (short, medium,

long)

Image Processing Knowledge: Filter Class

Filter type (Gaussian, median, morphological etc.)

Frequency response (low pass/high pass) Linearity (linear/non-linear) Filter parameters (e.g. kernel size,

sigma, structuring-element etc.) Performance against different noise

types (Gaussian, shot noise)

Image Processing Knowledge: Detector Class

Detector name Primitive type

Intensity gradient edge Intensity peak Region

Effective parameter range Confidence against

Strong/medium/weak boundary definition short/medium/long boundary dispersion

Knowledge Representation Information is

encoded using XML Attribute names and

value types are defined for each piece of information

Sub-classing to avoid duplicating information

A 0-1 rating is used to classify behavior when applicable

<filter type=“median” linear=“no” sub-type-of “rank”>

<transfer-function type=“lowpass”/>

<parameter name=“window” min=3 max=100/>

<noise-performance type=“gaussian” rating=0.5/>

<noise-performance type=“shot” rating=0.9/>

</filter>

Design of Tools Each tool is an independent object (agent) Able to emit the XML description of self for

use by the framework (or others) All tools are sub-classed from a base type

that has three primary “methods” Judicious use of OOP features such as

“inheritance” and “polymorphism” helps the design of tools and the framework

Implementing the Tools All tools have the following functionality

Self – Describe itself using XML notation Match – Gives a compatibility score against

an input description SetInput – Defines the input data Process – Performs the processing operation GetOutput – Extract processed data

Example current.SetInput(prev);current.Process();

next.SetInput(current); …

Processing

SequenceGeneratio

n

Processing

Sequence Execution

Tool Library

Task Descriptio

n

Image Descriptio

n

Output

User Control

System Architecture

Framework First implementation was script based Simply arranged the utilities in the

prescribed order Performed the sequence of operations Had limited user interface for arranging

operations Contained two detectors (peak and edge) Mostly used as a test bed to develop the

tool interface

Current Implementation Resembles an expert system Inputs are

Detector type (peak, line) Edge strength and diffusion Structure size Noise level

Creates a sequence of operations Performs the sequence at user’s request

Limitations of the Current System

Linear progression of tools Tools with multiple inputs are not yet

supported Iterative processing methods are not

supported XML tools are not fully mature yet. Not enough tools

Future Work Addressing current limitations Using this system as a knowledge

acquisition tool Research on machine learning Possibilities are endless!

Summary Image analysis helps managing large

amounts of data and extract new information

Agent based framework attempts to use multiple processing tools to improve performance

Incorporating knowledge about analysis tools helps create a user centered system

Flexibility comes from being able to add new agents without modifying the framework

Thank You!

Aruni admiring a Water Lilly