Visual Information Visual Information SystemsSystems

Image ContentImage Content

Description of Content Description of Content – image processing– image processing

Primitive image propertiesPrimitive image properties Through image processing techniquesThrough image processing techniques Colour, texture, local shapeColour, texture, local shape The need of combination of these properties The need of combination of these properties

into a consistent set of localised propertiesinto a consistent set of localised properties There can be weighting scheme to balance the There can be weighting scheme to balance the

importance of each type of property. importance of each type of property. Image features Image features

Integration of primitive Integration of primitive propertiesproperties

a separation between color, local a separation between color, local geometry, and texture. geometry, and texture.

an integrated view on color, texture, and an integrated view on color, texture, and local geometry is urgently needed as only local geometry is urgently needed as only an integrated view on local properties can an integrated view on local properties can provide the means to distinguish among provide the means to distinguish among hundreds of thousands different images. hundreds of thousands different images.

Further research is needed in the design of Further research is needed in the design of complete sets of image properties with complete sets of image properties with well-described variant conditions which well-described variant conditions which they are capable of handling. they are capable of handling.

Image featuresImage features

Image featuresImage features Grouping DataGrouping Data, , Global and Accumulating Global and Accumulating

FeaturesFeatures, , Salient Features, SignsSalient Features, Signs, , Shape and Shape and Object FeaturesObject Features, , Description of Structure and Description of Structure and Lay-Out Lay-Out

Also in the description of the image by features, Also in the description of the image by features, it should be kept in mind that for retrieval a total it should be kept in mind that for retrieval a total understanding of the image is rarely needed.understanding of the image is rarely needed.

the deeper one goes into the semantics of the the deeper one goes into the semantics of the pictures, the deeper the understanding of the pictures, the deeper the understanding of the picture will also have to be picture will also have to be 

With segmentationWith segmentation no segmentation no segmentation 

Interpretation And Similarity Interpretation And Similarity MeasureMeasure

Semantic features aim at encoding Semantic features aim at encoding interpretations of the image which may be interpretations of the image which may be relevant to the application.relevant to the application.

feature set can be explained feature set can be explained derives an unilateral interpretation from the derives an unilateral interpretation from the

feature setfeature set compares the feature set with the elements in a compares the feature set with the elements in a

given data set on the basis of a similarity functiongiven data set on the basis of a similarity function In content-based retrieval, it is useful to push In content-based retrieval, it is useful to push

the semantic interpretation of features the semantic interpretation of features derived from the image as far as one can. derived from the image as far as one can.

Similarity MeasurementSimilarity Measurement A different road to assigning a A different road to assigning a

meaning to an observed feature set, meaning to an observed feature set, is to compare a pair of observations is to compare a pair of observations by a similarity function. – a kind of by a similarity function. – a kind of interpretationinterpretation And this is the advantage to have And this is the advantage to have

content-based retrieval.content-based retrieval.

Semantic SimilaritySemantic Similarity knowledge-based type abstraction hierarchiesknowledge-based type abstraction hierarchies concept-space concept-space a linguistic description of texture patch visual a linguistic description of texture patch visual

qualities is given and ordered in a hierarchy of qualities is given and ordered in a hierarchy of perceptual importance on the basis of perceptual importance on the basis of extensive psychological experimentation. extensive psychological experimentation.

A more general concept of similarity is needed A more general concept of similarity is needed for relevance feedback, in which similarity with for relevance feedback, in which similarity with respect to an ensemble of images is required. respect to an ensemble of images is required.

Different Levels of Content-base Different Levels of Content-base Indexing and RetrievalIndexing and Retrieval

syntactical level: mainly deal with colour, shape, syntactical level: mainly deal with colour, shape, texture etc. Some used manual annotation to texture etc. Some used manual annotation to index dataindex data e.g. retrieval system let users to fill forms to provide e.g. retrieval system let users to fill forms to provide

queries, like location, colour etc categories, like the queries, like location, colour etc categories, like the work done in Berkeleywork done in Berkeley

semantic levelsemantic level: : analyse captionsanalyse captions purely used text information and didn’t make use of purely used text information and didn’t make use of

the information inherent in the imagesthe information inherent in the images complicated algorithm applied on small scalecomplicated algorithm applied on small scale

Multi-level indexingMulti-level indexing An advantage of image indexing based An advantage of image indexing based

on multi-level contents rather than on multi-level contents rather than solely on low-level features such as solely on low-level features such as texture and colours, is that it would texture and colours, is that it would readily provide the basic framework readily provide the basic framework required for "semantic interoperability" required for "semantic interoperability" when one tries to search through, not when one tries to search through, not only one, but a federation of image only one, but a federation of image collections from different disciplines.collections from different disciplines.

Learning from FeedbackLearning from Feedback The interacting user brings about many new The interacting user brings about many new

challenges for the response time of the system.challenges for the response time of the system. Content-based image retrieval is only scalable to Content-based image retrieval is only scalable to

large data sets when the database is able to large data sets when the database is able to anticipate what interactive queries will be made.anticipate what interactive queries will be made.

A frequent assumption is that the image set, the A frequent assumption is that the image set, the features, and the similarity function are known in features, and the similarity function are known in advance. In a truly interactive session, the advance. In a truly interactive session, the assumptions are no longer valid. assumptions are no longer valid.

A change from static to dynamic indexing is A change from static to dynamic indexing is required.required. (Arnold 2000) (Arnold 2000)

An integrated issueAn integrated issue It will demand its own view of things It will demand its own view of things

as it is our belief that content-based as it is our belief that content-based retrieval in the end will not be part of retrieval in the end will not be part of the field of computer vision alone. the field of computer vision alone. The man-machine interface, domain The man-machine interface, domain knowledge, and database technology knowledge, and database technology each will have their impact on the each will have their impact on the product.product.

SummarySummary• The heritage of computer vision. The heritage of computer vision. • The influence on computer vision. The influence on computer vision.

• Deal with large data sets. Deal with large data sets. • the absence of a general method for strong the absence of a general method for strong

segmentation. segmentation. • has revitalized interest in color image processing.has revitalized interest in color image processing.• attention for invariance has been revitalized  attention for invariance has been revitalized  

• Similarity and learning. Similarity and learning. • Interaction. Interaction. • The need for databases. The need for databases. • The problem of evaluationThe problem of evaluation. . • The semantic gap and other sources. The semantic gap and other sources.

ColourColour Colour is a visual feature which is immediately Colour is a visual feature which is immediately

perceived when looking at an image.perceived when looking at an image. The recorded colour varies considerably with The recorded colour varies considerably with

the orientation of the surface, the viewpoint of the orientation of the surface, the viewpoint of the camera, the position of the illumination, the camera, the position of the illumination, the spectrum of the illuminant, and the way the spectrum of the illuminant, and the way the light interacts with the object. the light interacts with the object.

The human perception of colour is an intricate The human perception of colour is an intricate topic where many attempts have been made topic where many attempts have been made to capture perceptual similarity. to capture perceptual similarity.

ColourColour Light is energy, specifically electromagnetic Light is energy, specifically electromagnetic

energyenergy – eye (energy detector) – eye (energy detector) The eye can distinguish between some types The eye can distinguish between some types

of electromagnetic energy. Those distinctions of electromagnetic energy. Those distinctions are seen as colours. are seen as colours.

The whiteness of an image area and the The whiteness of an image area and the amount of light hitting the eyeamount of light hitting the eye The actual reflectanceThe actual reflectance The brightness of incident lightThe brightness of incident light The incidence angle: the angle at which the light hits the The incidence angle: the angle at which the light hits the

object (one per light source, and there may be several)object (one per light source, and there may be several) The surface orientation of the area with respect to the The surface orientation of the area with respect to the

viewerviewer

Studies by psychologist and artists have demonstrated that the presence and distribution of colours induce sensations and convey meanings in the observer, according to specific rules - colour images can also be retrieved according to the meaning they convey or to sensations they provoke.

How to measure colour How to measure colour Colour histogram is one of the useful Colour histogram is one of the useful

methods to measure colour content.methods to measure colour content.

TextureTexture What is texture?What is texture? How to detect texture information?How to detect texture information? How to measure it?How to measure it? How to use it?How to use it?

TextureTexture Texture is a phenomenon that is Texture is a phenomenon that is

widespread, easy to recognise and widespread, easy to recognise and hard to define. hard to define.

TextureTexture

Together with colour, texture is a Together with colour, texture is a powerful discriminating feature, powerful discriminating feature, present almost everywhere in nature. present almost everywhere in nature.

Like colours, textures are connected Like colours, textures are connected with psychological effects. In with psychological effects. In particular, they emphasize particular, they emphasize orientations and spatial depth orientations and spatial depth between overlapping object.between overlapping object.

TextureTexture There are three standard problems to do with There are three standard problems to do with

texture:texture: Texture segmentation is the problem of breaking Texture segmentation is the problem of breaking

an image into components within which the an image into components within which the texture is constant. Texture segmentation involves texture is constant. Texture segmentation involves both representing a texture, and determining the both representing a texture, and determining the basis on which segment boundaries are to be basis on which segment boundaries are to be determined. How texture should be represented.determined. How texture should be represented.

texture synthesis seeks to construct large regions texture synthesis seeks to construct large regions of texture from small example images.of texture from small example images.

Shape from texture involves recovering surface Shape from texture involves recovering surface orientation or surface shape from image texture.orientation or surface shape from image texture.

Traditional Definition of TextureTraditional Definition of Texture Texture refers to a spatially repeating pattern on Texture refers to a spatially repeating pattern on

a surface that can be sensed visuallya surface that can be sensed visually In the image, the apparent size, shape, spacing In the image, the apparent size, shape, spacing

etc, of the texture elements (the texels) do etc, of the texture elements (the texels) do indeed varyindeed vary Varying distances of the different texels from the Varying distances of the different texels from the

cameracamera Varying foreshortening of the different texels. Varying foreshortening of the different texels.

texture gradientstexture gradients - systematic change in the - systematic change in the size and shape of the elements making up a size and shape of the elements making up a texturetexture

recover shape from texture

Recover Shape From TextureRecover Shape From Texture

After some mathematical analysis , one can After some mathematical analysis , one can compute expressions for the rate of change of compute expressions for the rate of change of various image texel features, such as area, various image texel features, such as area, foreshortening, and density. These texture foreshortening, and density. These texture gradients are functions of the surface shape as gradients are functions of the surface shape as well as its slant and tilt with respect to the well as its slant and tilt with respect to the viewer.viewer.

To recover shape from texture, one can use two-To recover shape from texture, one can use two-step process: step process: 1) measure the texture gradients 1) measure the texture gradients 2) estimate the surface shape, slant, and tilt that 2) estimate the surface shape, slant, and tilt that

would give rise to the measured texture gradients.would give rise to the measured texture gradients.

RecentRecent Technical Definition of texture:- Texture is a broad term used in pattern recognition to identify image patches (of any size) that are characterized by differences in brightness.

•Techniques to extract meaningful texture descriptors from image are many, based on different models and assumptions. •An effective representation of textures can be based on statistical and structural properties of brightness patterns.

Texture Content MeasurementTexture Content Measurement Textures may be described according to Textures may be described according to

their spatial, frequency or perceptual their spatial, frequency or perceptual properties. Periodicity, coarseness, properties. Periodicity, coarseness, preferred direction, degree of complexity preferred direction, degree of complexity are some of the most perceptually salient are some of the most perceptually salient attributes of a texture. attributes of a texture.

Feature spaces based on these attributes Feature spaces based on these attributes are particularly interesting for image are particularly interesting for image retrieval by texture similarity.retrieval by texture similarity.

Space – based modelsSpace – based models Auto-correlation functionAuto-correlation function A texture can A texture can

be represented taking into account the be represented taking into account the spatial size of grey-level primitives. Fine spatial size of grey-level primitives. Fine textures have a small size of their grey-textures have a small size of their grey-level primitives. Coarse textures a large level primitives. Coarse textures a large size.  size.  

Co – occurrence matrixCo – occurrence matrix A different way A different way of measuring textures is by taking into of measuring textures is by taking into account the spatial arrangement of grey-account the spatial arrangement of grey-level primitives.level primitives.

Calculate TextureCalculate Texture

energy

entropy

contrast

Structural Texture Structural Texture RepresentationsRepresentations

RequireRequire texture primitive - texeltexture primitive - texel placement ruleplacement rule

Ideal for regular - man-made - Ideal for regular - man-made - texturestextures

Statistical DescriptionsStatistical Descriptions Better suited to pseudorandom, Better suited to pseudorandom,

natural texturesnatural textures First Order statisticsFirst Order statistics Second order statisticsSecond order statistics

First Order StatisticsFirst Order Statistics Statistical descriptions of grey level Statistical descriptions of grey level

distributiondistribution Mean grey valueMean grey value Deviation of grey valuesDeviation of grey values Coefficient of variationCoefficient of variation etc.etc.

Can give useful resultsCan give useful results Generally too sensitive to factors other Generally too sensitive to factors other

than identity of surfacethan identity of surface

Second Order StatisticsSecond Order Statistics Measures involving multiple pixelsMeasures involving multiple pixels

Joint difference histogramJoint difference histogram histogram of differences between adjacent histogram of differences between adjacent

pixelspixels Co-Occurrence matricesCo-Occurrence matrices

measure frequency of specific pairs of grey measure frequency of specific pairs of grey valuesvalues

Co-Occurrence MatricesCo-Occurrence Matrices Define a relative separation vectorDefine a relative separation vector

e.g. 3 pixels across, 2 upe.g. 3 pixels across, 2 up Use each pair of pixels separated by the Use each pair of pixels separated by the

vector as matrix indicesvector as matrix indices Increment this matrix elementIncrement this matrix element Shape of matrix characterises the Shape of matrix characterises the

texturetexture Can be characterised by factors derived Can be characterised by factors derived

from it.from it.

Edge FrequencyEdge Frequency Density of microedges is Density of microedges is

characteristic of texturecharacteristic of texture Apply an edge detectorApply an edge detector

Sobel is suitableSobel is suitable Threshold resultThreshold result Compute density of edge elementsCompute density of edge elements

Image featuresImage features