ImplementingImplementing thethe

By: Matthew MarshBy: Matthew Marsh

Supervisors: Prof Shaun Supervisors: Prof Shaun BangayBangay

Mrs Adele LobbMrs Adele Lobb

segmentation technique segmentation technique as a plugin for the GIMPas a plugin for the GIMP

Problem statement:Problem statement: The open source community require a quick, The open source community require a quick,

easy to use segmentation tool with matting easy to use segmentation tool with matting capabilities. capabilities.

Aim:Aim: The GIMP is an open source image editing The GIMP is an open source image editing

package that requires such a tool.package that requires such a tool. The GIMP allows extra tools to be added as The GIMP allows extra tools to be added as

pluginsplugins Implement a quick, easy to use Implement a quick, easy to use

segmentation tool with matting capabilities segmentation tool with matting capabilities as a plugin for the GIMPas a plugin for the GIMP

What is image What is image segmentation?segmentation?

The process of separating an image The process of separating an image into foreground and background into foreground and background parts.parts.

Foreground = object of interestForeground = object of interest Background = the rest of the imageBackground = the rest of the image

Image Segmentation Image Segmentation ContCont

Every pixel has an alpha value, which Every pixel has an alpha value, which sets its opacitysets its opacity

Image segmentation labels all foreground Image segmentation labels all foreground pixels with a full alpha value of 1,pixels with a full alpha value of 1,

Background pixels labelled with a value Background pixels labelled with a value of 0of 0

Can cause jagged edges in finely Can cause jagged edges in finely textured objectstextured objects

Matting smoothes these edgesMatting smoothes these edges

MattingMatting

Labels pixels with alpha values between the Labels pixels with alpha values between the maximum and minimum valuesmaximum and minimum values

Creates edges which vary smoothly from Creates edges which vary smoothly from background to foreground.background to foreground.

The image on the right is known as the alpha The image on the right is known as the alpha mattematte

How computers perform How computers perform segmentationsegmentation

Information encapsulated in the digital image Information encapsulated in the digital image is used to calculate the best segmentation. is used to calculate the best segmentation.

Edge detection and region detection methods Edge detection and region detection methods are most commonly used to segment images. are most commonly used to segment images.

Boundary information is the edge information Boundary information is the edge information in an image which can be gained by noting the in an image which can be gained by noting the difference between the colours of nearby difference between the colours of nearby pixels. A large difference in colour will pixels. A large difference in colour will indicate an edge. indicate an edge.

A region is the area enclosed by an edge, and A region is the area enclosed by an edge, and the colours of neighbouring pixels in a region the colours of neighbouring pixels in a region are similar. are similar.

““GrabCut”GrabCut”

Need to decide on a segmentation Need to decide on a segmentation techniquetechnique

““GrabCut” has been chosenGrabCut” has been chosen Uses both region and boundary Uses both region and boundary

informationinformation Uses new graph cut technique Uses new graph cut technique Includes a method of border mattingIncludes a method of border matting

How “GrabCut” WorksHow “GrabCut” Works

Clue MarkingClue Marking Clues are used to build up Clues are used to build up

region modelsregion models One model for foreground One model for foreground

and one for backgroundand one for background Region models determine Region models determine

weights between pixel and weights between pixel and Source and Sink nodesSource and Sink nodes

Types of region modelsTypes of region models

3 types of region models have been 3 types of region models have been developeddeveloped Greyscale histogramsGreyscale histograms Gaussian Mixture ModelsGaussian Mixture Models Colour histogramsColour histograms

Greyscale HistogramsGreyscale Histograms

Gaussian Mixture ModelsGaussian Mixture ModelsThe 1D Gaussian FunctionThe 1D Gaussian Function

The 2D Gaussian FunctionThe 2D Gaussian Function

3D Gaussians3D Gaussians A Gaussian Mixture A Gaussian Mixture

Model is the weighted Model is the weighted sum of several sum of several Gaussians.Gaussians.

Encapsulate the concept Encapsulate the concept of a colour of a colour

Each Gaussian is Each Gaussian is centred around a centred around a common colour in the common colour in the imageimage

A GMM has several A GMM has several common colourscommon colours

In this plug-in 5 In this plug-in 5 Gaussians in foreground, Gaussians in foreground, 5 in background5 in background

Colour HistogramsColour Histograms

The sum of three histogramsThe sum of three histograms One histogram for each colour channelOne histogram for each colour channel Thus a Red, Green and Blue histogram Thus a Red, Green and Blue histogram

in RGB colour spacein RGB colour space The probability of a pixel lying in the The probability of a pixel lying in the

foreground region model is the sum of foreground region model is the sum of the probabilities of its red value lying the probabilities of its red value lying in the red histogram, its blue in the in the red histogram, its blue in the blue etc.blue etc.

Matting ImplementationMatting Implementation

““GrabCut” adopts an approach of GrabCut” adopts an approach of border mattingborder matting

Assume a smooth alpha profileAssume a smooth alpha profile Defined by the sigmoid functionDefined by the sigmoid function

Border Matting WindowsBorder Matting Windows

Segmentation ResultsSegmentation Results Greyscale histograms proved to Greyscale histograms proved to

work wellwork well Implementation is fairly fastImplementation is fairly fast Results are goodResults are good

GMM’s proved to be slow due to the GMM’s proved to be slow due to the calculation of the initial 5 Gaussians calculation of the initial 5 Gaussians – also the initialization affected the – also the initialization affected the results substantially results substantially

Colour histograms also work well, Colour histograms also work well, and are basically just an extension and are basically just an extension onto greyscale histogramsonto greyscale histograms

Segmentation results Segmentation results contcont

Matting resultsMatting results

Input imageInput image

Matting significantly improved the Matting significantly improved the segmentation results in most cases.segmentation results in most cases.

Image with Image with mattingmatting

Image without Image without mattingmatting

ConclusionConclusion

Have completed my project goal – Have completed my project goal – provide open source community with provide open source community with a segmentation toola segmentation tool

Have implemented the “GrabCut” Have implemented the “GrabCut” paper and tested their claimspaper and tested their claims

The “GrabCut” segmentation The “GrabCut” segmentation technique is definitely usable, and technique is definitely usable, and produces good results.produces good results.