![Page 1: An Agent Based Architecture For Designing Flexible Image Analysis Systems Jagath Samarabandu Dept. of Electrical and Computer Engineering University of](https://reader036.vdocuments.site/reader036/viewer/2022062407/56649de65503460f94adf673/html5/thumbnails/1.jpg)
An Agent Based Architecture For Designing Flexible Image Analysis Systems
Jagath Samarabandu
Dept. of Electrical and Computer EngineeringUniversity of Western Ontario
![Page 2: An Agent Based Architecture For Designing Flexible Image Analysis Systems Jagath Samarabandu Dept. of Electrical and Computer Engineering University of](https://reader036.vdocuments.site/reader036/viewer/2022062407/56649de65503460f94adf673/html5/thumbnails/2.jpg)
Multi-dimensional Imaging System
Capture Processing Visualization
Storage
Confocal Microscopy
Conventional Microscopy with Deconvolution
Tomography with Reconstruction
MR, CT, PET, SPECT, US
ROI Extraction
Digital Filtering
Segmentation
Analysis
Morphometry
Sectional Views
MIP/Sum Projections
Surface Shading
Volume Rendering
Stereo views
Animation Sequences
Interactive displays
Raw data
Region dataProcessed dataHigh level data
Single frameStereo frameMovie sequences
![Page 3: An Agent Based Architecture For Designing Flexible Image Analysis Systems Jagath Samarabandu Dept. of Electrical and Computer Engineering University of](https://reader036.vdocuments.site/reader036/viewer/2022062407/56649de65503460f94adf673/html5/thumbnails/3.jpg)
Acknowledgements Kamal Ranaweera – MESc student who
is doing most of the work Dr. Aaron Fenster for his mentorship CITO Round 3.2 research grant
![Page 4: An Agent Based Architecture For Designing Flexible Image Analysis Systems Jagath Samarabandu Dept. of Electrical and Computer Engineering University of](https://reader036.vdocuments.site/reader036/viewer/2022062407/56649de65503460f94adf673/html5/thumbnails/4.jpg)
Outline Why image analysis Image processing basics Image analysis paradigm Analysis system example A flexible framework for image analysis Knowledge organization Current status Outlook
![Page 5: An Agent Based Architecture For Designing Flexible Image Analysis Systems Jagath Samarabandu Dept. of Electrical and Computer Engineering University of](https://reader036.vdocuments.site/reader036/viewer/2022062407/56649de65503460f94adf673/html5/thumbnails/5.jpg)
Why Image Analysis Image dimensions are increasing Image sizes and resolutions are
increasing Image quality is getting better Cannot visually assess all the voxels
![Page 6: An Agent Based Architecture For Designing Flexible Image Analysis Systems Jagath Samarabandu Dept. of Electrical and Computer Engineering University of](https://reader036.vdocuments.site/reader036/viewer/2022062407/56649de65503460f94adf673/html5/thumbnails/6.jpg)
Image Processing Point Operations
Histogram Equalization Contrast Stretching
Spatial Operations Linear Filtering
Low/High pass filtering Edge Detection
Non-Linear Filtering Median filtering Morphological Filtering
![Page 7: An Agent Based Architecture For Designing Flexible Image Analysis Systems Jagath Samarabandu Dept. of Electrical and Computer Engineering University of](https://reader036.vdocuments.site/reader036/viewer/2022062407/56649de65503460f94adf673/html5/thumbnails/7.jpg)
Point OperationsHistogram EqualizationIntensity histogram of the output image has a uniform distribution
![Page 8: An Agent Based Architecture For Designing Flexible Image Analysis Systems Jagath Samarabandu Dept. of Electrical and Computer Engineering University of](https://reader036.vdocuments.site/reader036/viewer/2022062407/56649de65503460f94adf673/html5/thumbnails/8.jpg)
Spatial Operations Linear Filtering
g(x,y) = ijh(i,j) u(i+x, j+y) where u(x,y) is the input image intensity at
(x,y) and h(i,j) is the value of convolution kernel at (i,j)
1 1 11 1 11 1 1
-1 0 1-1 0 1-1 0 1
-1 -1 -1 0 0 0 1 1 1
MovingAverage
Pair of kernels for edge detection
Example kernels
![Page 9: An Agent Based Architecture For Designing Flexible Image Analysis Systems Jagath Samarabandu Dept. of Electrical and Computer Engineering University of](https://reader036.vdocuments.site/reader036/viewer/2022062407/56649de65503460f94adf673/html5/thumbnails/9.jpg)
Edge Detection Operators Gradient Operators
Finite difference approximations of orthogonal gradients (e.g.. hx and hy for 2D images)
gm (x,y) = gx(x,y)2 + gy(x,y)2
g(x,y) = tan -1 (gy(x,y) / gx(x,y))
-1 0 1 -1 -1 -1-1 0 1 0 0 0-1 0 1 1 1 1
-1 0 1 -1 -1 -1-1 0 1 0 0 0-1 0 1 1 1 1
0 1 1 0-1 0 0 -1
Roberts Prewitt Sobel
![Page 10: An Agent Based Architecture For Designing Flexible Image Analysis Systems Jagath Samarabandu Dept. of Electrical and Computer Engineering University of](https://reader036.vdocuments.site/reader036/viewer/2022062407/56649de65503460f94adf673/html5/thumbnails/10.jpg)
Edge Detection Operators Compass Operators
Directional gradients at a given number of directions (h1, h2, h3 .. hn)
gm (x,y) = MAXk {| gk (x,y)|} g(x,y) is k at which the maximum occurred
-1 0 1-1 0 1-1 0 1
0 1 1-1 0 1-1 -1 0
1 1 1 0 0 0-1 -1 -1
1 1 0 1 0 -1 0 -1 -1
1 0 -1 1 0 -1 1 0 -1
0 -1 -1 1 0 -1 1 1 0
-1 -1 -1 0 0 0 1 1 1
-1 -1 0-1 0 1 0 1 1
Kirsh templates for 8 directions
![Page 11: An Agent Based Architecture For Designing Flexible Image Analysis Systems Jagath Samarabandu Dept. of Electrical and Computer Engineering University of](https://reader036.vdocuments.site/reader036/viewer/2022062407/56649de65503460f94adf673/html5/thumbnails/11.jpg)
Edge Operators
Roberts Prewitt Sobel
Compass Kirsch Laplace
![Page 12: An Agent Based Architecture For Designing Flexible Image Analysis Systems Jagath Samarabandu Dept. of Electrical and Computer Engineering University of](https://reader036.vdocuments.site/reader036/viewer/2022062407/56649de65503460f94adf673/html5/thumbnails/12.jpg)
Spatial Operators Linear Filters
Gaussian smoothing Edge detection operators
Non Linear Filters Median filter Rank order filters morphological filters
![Page 13: An Agent Based Architecture For Designing Flexible Image Analysis Systems Jagath Samarabandu Dept. of Electrical and Computer Engineering University of](https://reader036.vdocuments.site/reader036/viewer/2022062407/56649de65503460f94adf673/html5/thumbnails/13.jpg)
Spatial Filters for Smoothing
Gaussian Noise Gaussian (7x7) Median (5x5)
Speckle noise Median (5x5)Gaussian (7x7)
![Page 14: An Agent Based Architecture For Designing Flexible Image Analysis Systems Jagath Samarabandu Dept. of Electrical and Computer Engineering University of](https://reader036.vdocuments.site/reader036/viewer/2022062407/56649de65503460f94adf673/html5/thumbnails/14.jpg)
High Level Processing Feature Extraction
Edges and Boundaries Shape and Moments Texture
Segmentation Matching Thresholding Boundary Detection
Classification Statistical Classification Clustering
![Page 15: An Agent Based Architecture For Designing Flexible Image Analysis Systems Jagath Samarabandu Dept. of Electrical and Computer Engineering University of](https://reader036.vdocuments.site/reader036/viewer/2022062407/56649de65503460f94adf673/html5/thumbnails/15.jpg)
Canny’s Edge OperatorCombines edge operators and adaptive thresholding with
hysterisys to locate edges.
![Page 16: An Agent Based Architecture For Designing Flexible Image Analysis Systems Jagath Samarabandu Dept. of Electrical and Computer Engineering University of](https://reader036.vdocuments.site/reader036/viewer/2022062407/56649de65503460f94adf673/html5/thumbnails/16.jpg)
Image Analysis
PictureInterpretation
Image
Picture DomainCues
Scene DomainCues
Model
InstantiatedModel
Scene Domain
Picture Domain
Signal
Physical
Semantic
Kanade’s paradigm
![Page 17: An Agent Based Architecture For Designing Flexible Image Analysis Systems Jagath Samarabandu Dept. of Electrical and Computer Engineering University of](https://reader036.vdocuments.site/reader036/viewer/2022062407/56649de65503460f94adf673/html5/thumbnails/17.jpg)
Bottom Up Cycle Picture domain cues are primitive image
features Scene domain cues are the
interpretation of image features This distinction is somewhat blurred for
3D images
![Page 18: An Agent Based Architecture For Designing Flexible Image Analysis Systems Jagath Samarabandu Dept. of Electrical and Computer Engineering University of](https://reader036.vdocuments.site/reader036/viewer/2022062407/56649de65503460f94adf673/html5/thumbnails/18.jpg)
Top Down Cycle Model instantiation or hypothesis
generation Verification and refinement
![Page 19: An Agent Based Architecture For Designing Flexible Image Analysis Systems Jagath Samarabandu Dept. of Electrical and Computer Engineering University of](https://reader036.vdocuments.site/reader036/viewer/2022062407/56649de65503460f94adf673/html5/thumbnails/19.jpg)
Data Driven Approach Example
Median Filtering
Difference of Gaussian
Low Pass filter
Boundary Forming
Thresholding
Feature Aggregation
![Page 20: An Agent Based Architecture For Designing Flexible Image Analysis Systems Jagath Samarabandu Dept. of Electrical and Computer Engineering University of](https://reader036.vdocuments.site/reader036/viewer/2022062407/56649de65503460f94adf673/html5/thumbnails/20.jpg)
Feature Aggregation
This technique was able to suppress concavities
It also tended to diffuse boundaries of small structures
lyxyxH
L
yxyxl
lyx
lyx
lyx
yxeE
i
iii
iii
i
Ekwii
,| and
Otherwise
if
if
Where
,,max
min
,,
1,
1,,
00
0000
![Page 21: An Agent Based Architecture For Designing Flexible Image Analysis Systems Jagath Samarabandu Dept. of Electrical and Computer Engineering University of](https://reader036.vdocuments.site/reader036/viewer/2022062407/56649de65503460f94adf673/html5/thumbnails/21.jpg)
Processing Confocal Images
Single section of the original image
2D median filtering
![Page 22: An Agent Based Architecture For Designing Flexible Image Analysis Systems Jagath Samarabandu Dept. of Electrical and Computer Engineering University of](https://reader036.vdocuments.site/reader036/viewer/2022062407/56649de65503460f94adf673/html5/thumbnails/22.jpg)
Processing Confocal Images
Difference of Gaussian filter 3D contours of final segmentation
![Page 23: An Agent Based Architecture For Designing Flexible Image Analysis Systems Jagath Samarabandu Dept. of Electrical and Computer Engineering University of](https://reader036.vdocuments.site/reader036/viewer/2022062407/56649de65503460f94adf673/html5/thumbnails/23.jpg)
Detecting DNA Replication Sites Bottom up
Intensity maxima are detected on 2D image slices
Sites cross sections are modeled as circles centered on these maxima
Top Down Site boundaries are then refined according to the
local image features These boundaries are then combined to form 3D
structures using 3D connected component labeling
![Page 24: An Agent Based Architecture For Designing Flexible Image Analysis Systems Jagath Samarabandu Dept. of Electrical and Computer Engineering University of](https://reader036.vdocuments.site/reader036/viewer/2022062407/56649de65503460f94adf673/html5/thumbnails/24.jpg)
DNA Replication Sites (MIP)
![Page 25: An Agent Based Architecture For Designing Flexible Image Analysis Systems Jagath Samarabandu Dept. of Electrical and Computer Engineering University of](https://reader036.vdocuments.site/reader036/viewer/2022062407/56649de65503460f94adf673/html5/thumbnails/25.jpg)
Segmented Replication Sites
![Page 26: An Agent Based Architecture For Designing Flexible Image Analysis Systems Jagath Samarabandu Dept. of Electrical and Computer Engineering University of](https://reader036.vdocuments.site/reader036/viewer/2022062407/56649de65503460f94adf673/html5/thumbnails/26.jpg)
Further Analysis
![Page 27: An Agent Based Architecture For Designing Flexible Image Analysis Systems Jagath Samarabandu Dept. of Electrical and Computer Engineering University of](https://reader036.vdocuments.site/reader036/viewer/2022062407/56649de65503460f94adf673/html5/thumbnails/27.jpg)
Further Analysis
![Page 28: An Agent Based Architecture For Designing Flexible Image Analysis Systems Jagath Samarabandu Dept. of Electrical and Computer Engineering University of](https://reader036.vdocuments.site/reader036/viewer/2022062407/56649de65503460f94adf673/html5/thumbnails/28.jpg)
Blackboard Model Provides us an elegant architecture
within which both data driven and model driven approaches can be incorporated
A problem is solved in an incremental fashion
Contains a set of knowledge sources or tools
![Page 29: An Agent Based Architecture For Designing Flexible Image Analysis Systems Jagath Samarabandu Dept. of Electrical and Computer Engineering University of](https://reader036.vdocuments.site/reader036/viewer/2022062407/56649de65503460f94adf673/html5/thumbnails/29.jpg)
Blackboard Model These tools are designed as
independent modules They interact with each other via a
global database called the blackboard Modules that generate solutions are
separated from those that determine their utility
![Page 30: An Agent Based Architecture For Designing Flexible Image Analysis Systems Jagath Samarabandu Dept. of Electrical and Computer Engineering University of](https://reader036.vdocuments.site/reader036/viewer/2022062407/56649de65503460f94adf673/html5/thumbnails/30.jpg)
Blackboard Models can handle Large solution space Noisy and unreliable data Variety of input data and a need to
integrate diverse information The need to use many independent or
semi-independent sources of knowledge to co-operate in forming a solution
![Page 31: An Agent Based Architecture For Designing Flexible Image Analysis Systems Jagath Samarabandu Dept. of Electrical and Computer Engineering University of](https://reader036.vdocuments.site/reader036/viewer/2022062407/56649de65503460f94adf673/html5/thumbnails/31.jpg)
Blackboard Models can handle The need to use multiple reasoning
methods (eg. Forward and backward) The need for multiple lines of reasoning The need for an evolutionary solution
![Page 32: An Agent Based Architecture For Designing Flexible Image Analysis Systems Jagath Samarabandu Dept. of Electrical and Computer Engineering University of](https://reader036.vdocuments.site/reader036/viewer/2022062407/56649de65503460f94adf673/html5/thumbnails/32.jpg)
Requirements for the Framework
Represent and use knowledge about task domain and processing tools
Evaluate the quality of the image processing algorithms
Transfer information between different tools and knowledge sources
Control the flow of processing Allow easy addition of new tools
![Page 33: An Agent Based Architecture For Designing Flexible Image Analysis Systems Jagath Samarabandu Dept. of Electrical and Computer Engineering University of](https://reader036.vdocuments.site/reader036/viewer/2022062407/56649de65503460f94adf673/html5/thumbnails/33.jpg)
Requirements for Processing Tools
Implements the underlying processing algorithm
Contains knowledge about its algorithm Can evaluate its suitability (matching
score) to accomplish a given task Communicates with the framework
![Page 34: An Agent Based Architecture For Designing Flexible Image Analysis Systems Jagath Samarabandu Dept. of Electrical and Computer Engineering University of](https://reader036.vdocuments.site/reader036/viewer/2022062407/56649de65503460f94adf673/html5/thumbnails/34.jpg)
Knowledge Organization Domain Expertise
Knowledge about the image Intensity characteristics of the structure of
interest Image Processing Expertise
Knowledge about image processing algorithms (e.g. filtering, detection) and their parameters
![Page 35: An Agent Based Architecture For Designing Flexible Image Analysis Systems Jagath Samarabandu Dept. of Electrical and Computer Engineering University of](https://reader036.vdocuments.site/reader036/viewer/2022062407/56649de65503460f94adf673/html5/thumbnails/35.jpg)
Domain Knowledge Initial Model (e.g. circle, line, sphere) Parameter value distribution Expected orientation Boundary definition method (e.g.
intensity +/-, texture, color etc.) Boundary definition strength (strong,
medium, weak) Boundary dispersion (short, medium,
long)
![Page 36: An Agent Based Architecture For Designing Flexible Image Analysis Systems Jagath Samarabandu Dept. of Electrical and Computer Engineering University of](https://reader036.vdocuments.site/reader036/viewer/2022062407/56649de65503460f94adf673/html5/thumbnails/36.jpg)
Image Processing Knowledge: Filter Class
Filter type (Gaussian, median, morphological etc.)
Frequency response (low pass/high pass) Linearity (linear/non-linear) Filter parameters (e.g. kernel size,
sigma, structuring-element etc.) Performance against different noise
types (Gaussian, shot noise)
![Page 37: An Agent Based Architecture For Designing Flexible Image Analysis Systems Jagath Samarabandu Dept. of Electrical and Computer Engineering University of](https://reader036.vdocuments.site/reader036/viewer/2022062407/56649de65503460f94adf673/html5/thumbnails/37.jpg)
Image Processing Knowledge: Detector Class
Detector name Primitive type
Intensity gradient edge Intensity peak Region
Effective parameter range Confidence against
Strong/medium/weak boundary definition short/medium/long boundary dispersion
![Page 38: An Agent Based Architecture For Designing Flexible Image Analysis Systems Jagath Samarabandu Dept. of Electrical and Computer Engineering University of](https://reader036.vdocuments.site/reader036/viewer/2022062407/56649de65503460f94adf673/html5/thumbnails/38.jpg)
Knowledge Representation Information is
encoded using XML Attribute names and
value types are defined for each piece of information
Sub-classing to avoid duplicating information
A 0-1 rating is used to classify behavior when applicable
<filter type=“median” linear=“no” sub-type-of “rank”>
<transfer-function type=“lowpass”/>
<parameter name=“window” min=3 max=100/>
<noise-performance type=“gaussian” rating=0.5/>
<noise-performance type=“shot” rating=0.9/>
</filter>
![Page 39: An Agent Based Architecture For Designing Flexible Image Analysis Systems Jagath Samarabandu Dept. of Electrical and Computer Engineering University of](https://reader036.vdocuments.site/reader036/viewer/2022062407/56649de65503460f94adf673/html5/thumbnails/39.jpg)
Design of Tools Each tool is an independent object (agent) Able to emit the XML description of self for
use by the framework (or others) All tools are sub-classed from a base type
that has three primary “methods” Judicious use of OOP features such as
“inheritance” and “polymorphism” helps the design of tools and the framework
![Page 40: An Agent Based Architecture For Designing Flexible Image Analysis Systems Jagath Samarabandu Dept. of Electrical and Computer Engineering University of](https://reader036.vdocuments.site/reader036/viewer/2022062407/56649de65503460f94adf673/html5/thumbnails/40.jpg)
Implementing the Tools All tools have the following functionality
Self – Describe itself using XML notation Match – Gives a compatibility score against
an input description SetInput – Defines the input data Process – Performs the processing operation GetOutput – Extract processed data
Example current.SetInput(prev);current.Process();
next.SetInput(current); …
![Page 41: An Agent Based Architecture For Designing Flexible Image Analysis Systems Jagath Samarabandu Dept. of Electrical and Computer Engineering University of](https://reader036.vdocuments.site/reader036/viewer/2022062407/56649de65503460f94adf673/html5/thumbnails/41.jpg)
Processing
SequenceGeneratio
n
Processing
Sequence Execution
Tool Library
Task Descriptio
n
Image Descriptio
n
Output
User Control
System Architecture
![Page 42: An Agent Based Architecture For Designing Flexible Image Analysis Systems Jagath Samarabandu Dept. of Electrical and Computer Engineering University of](https://reader036.vdocuments.site/reader036/viewer/2022062407/56649de65503460f94adf673/html5/thumbnails/42.jpg)
Framework First implementation was script based Simply arranged the utilities in the
prescribed order Performed the sequence of operations Had limited user interface for arranging
operations Contained two detectors (peak and edge) Mostly used as a test bed to develop the
tool interface
![Page 43: An Agent Based Architecture For Designing Flexible Image Analysis Systems Jagath Samarabandu Dept. of Electrical and Computer Engineering University of](https://reader036.vdocuments.site/reader036/viewer/2022062407/56649de65503460f94adf673/html5/thumbnails/43.jpg)
Current Implementation Resembles an expert system Inputs are
Detector type (peak, line) Edge strength and diffusion Structure size Noise level
Creates a sequence of operations Performs the sequence at user’s request
![Page 44: An Agent Based Architecture For Designing Flexible Image Analysis Systems Jagath Samarabandu Dept. of Electrical and Computer Engineering University of](https://reader036.vdocuments.site/reader036/viewer/2022062407/56649de65503460f94adf673/html5/thumbnails/44.jpg)
Limitations of the Current System
Linear progression of tools Tools with multiple inputs are not yet
supported Iterative processing methods are not
supported XML tools are not fully mature yet. Not enough tools
![Page 45: An Agent Based Architecture For Designing Flexible Image Analysis Systems Jagath Samarabandu Dept. of Electrical and Computer Engineering University of](https://reader036.vdocuments.site/reader036/viewer/2022062407/56649de65503460f94adf673/html5/thumbnails/45.jpg)
Future Work Addressing current limitations Using this system as a knowledge
acquisition tool Research on machine learning Possibilities are endless!
![Page 46: An Agent Based Architecture For Designing Flexible Image Analysis Systems Jagath Samarabandu Dept. of Electrical and Computer Engineering University of](https://reader036.vdocuments.site/reader036/viewer/2022062407/56649de65503460f94adf673/html5/thumbnails/46.jpg)
Summary Image analysis helps managing large
amounts of data and extract new information
Agent based framework attempts to use multiple processing tools to improve performance
Incorporating knowledge about analysis tools helps create a user centered system
Flexibility comes from being able to add new agents without modifying the framework
![Page 47: An Agent Based Architecture For Designing Flexible Image Analysis Systems Jagath Samarabandu Dept. of Electrical and Computer Engineering University of](https://reader036.vdocuments.site/reader036/viewer/2022062407/56649de65503460f94adf673/html5/thumbnails/47.jpg)
Thank You!
Aruni admiring a Water Lilly