anaglyph tutorial

and(c) 2009-2010 Electronics & Robotics Club, BITS-Pilani, Goa

| Ajusal Sugathan & Utkarsh Sinha

presents(c) 2009-2010 Electronics & Robotics Club, BITS-Pilani, Goa

| Ajusal Sugathan & Utkarsh Sinha

The word robot originally was supposed to mean a slave

It is a machine which performs a variety of tasks, either using manual external

control or intelligent automation

A manually controlled car or a ASIMOV trying to kick a football are all robots


Robotics is a multi disciplinary field of engineering encompassing the vistas of› Mechanical design

› Electronic control

› Artificial Intelligence

It finds it‘s uses in all aspects of our life› automated vacuum cleaner

› Exploring the ‗Red‘ planet

› Setting up a human colony there :D


ROBOTS

CONTROL

AUTONOMOUS

MANUAL

APPLICATIONS

INDUSTRIAL

MEDICAL

INTERFACE

HARDWARE

SOFTWARE

INTERLINKED


Locomotion System

Actuators

Power Supply System

Transmission System

Switches

Sensory Devices For Feedback

Sensor Data Processing Unit


A mobile robot must have a system to make it move. Ob.

This system gives our machine the ability to move forward, backward and take turns

It may also provide for climbing up and down

Or even flying or floating


Each type of locomotion requires different number of degrees of freedom

More degrees of freedom means more the number of actuators you will have to use

Although one actuator can be used to control more than one degree of freedom


Wheeled

Legged

Climbing

Flying

Floating

Snake-Like


The kind of locomotion most frequently used in robotics at the undergrad level

This involves conversion of electrical energy into mechanical energy (mostly using motors)

The issue is to control these motors to give the required speed and torque


We have a simple equation for the constant power delivered to the motor:

› P = ζ X ω

Note that the torque and angular velocity are inversely proportionally to each other

So to increase the speed we have to reduce the torque


The dc motors available have very high speed of rotation which is generally not needed

At high speeds, they lack torque

For reduction in speed and increase in ―pulling capacity‖ we use pulley or gear systems


Differential Drive

Dual Differential Drive

Car-type Drive

Skid-steer Drive

Synchronous Drive


Simplest, easiest to implement and most widely used.

It has a free moving wheel in the front accompanied with a left and right wheel. The two wheels are separately powered

When the wheels move in the same direction the machine moves in that direction.

Turning is achieved by making the wheels oppose each other‘s motion, thus generating a couple


In-place (zero turning radius) rotation is done by turning the drive wheels at the same rate in the opposite direction

Arbitrary motion paths can be implemented by dynamically modifying the angular velocity and/or direction of the drive wheels

Total of two motors are required, both of them are responsible for translation and rotational motion


Simplicity and ease of use makes it the most preferred system by beginners

Independent drives makes it difficult for straight line motion. The differences in motors and frictional profile of the two wheels cause them to move with slight turning effect

The above drawback must be countered with appropriate feedback system. Suitable for human controlled remote robots


Uses synchronous rotation of its wheels to achieve motion & turns

It is made up of a system of 2 motors. One which drive the wheels and the other turns the wheels in a synchronous fashion

The two can be directly mechanically coupled as they always move in the same direction with same speed


The direction of motion is given by black arrow. The alignment of the machine is shown by red arrow


The use of separate motors for translation and wheel turning guarantees straight line motion without the need for dynamic feedback control

This system is somewhat complex in designing but further use is much simpler


Actuators, also known as drives, are mechanisms for getting robots to move.

Most actuators are powered by pneumatics (air pressure), hydraulics (fluid pressure), or motors (electric current).

They are devices which transform an input signal (mainly an electrical signal)) into motion


Widely used because of their

small size and high energy output.

Operating voltage: usually 6,12,24V.

Speed: 1-20,000 rpm..

Power: P = ζ X ω



The stator is the stationary outside part of a motor. The rotor is the inner part which rotates. Red represents a magnet or winding with a north polarization. Green represents a magnet or winding with a south polarization. Opposite, red and green, polarities attract. Commutator contacts are brown and the brushes are dark grey.


Stator is composed of two or more permanent magnet pole pieces.

Rotor composed of windings which are connected to a mechanical commutator.

The opposite polarities of the energized winding and the stator magnet attract and the rotor will rotate until it is aligned with the stator.

Just as the rotor reaches alignment, the brushes move across the commutator contacts and energize the next winding.

A yellow spark shows when the brushes switch to the next winding.


It is an electric motor that can divide a full rotation into a large number of steps. The motor's position can be controlled precisely, without any feedback mechanism. There are three types:

Permanent Magnet Variable Resistance Hybrid type


Stepper motors work in a similar way to dc motors, but where dc motors have 1 electromagnetic coil to produce movement, stepper motors contain many.

Stepper motors are controlled by turning each coil on and off in a sequence.

Every time a new coil is energized, the motor rotates a few degrees, called the step angle.


Full Step Stepper motors have 200 rotor teeth, or 200 full steps per revolution of the motor shaft. Dividing the 200 steps into the 360º's rotation equals a 1.8º full step angle. Achieved by energizing both windings while reversing the current alternately.


Servos operate on the principle of negative feedback, where the control input is compared to the actual position of the mechanical system as measured.

Any difference between the actual and wanted values (an "error signal") is amplified and used to drive the system in the direction necessary to reduce or eliminate the error

Their precision movement makes them ideal for powering legs, controlling rack and pinion steering, to move a sensor around etc.

Suitable power source is needed to run the robots

Mobile robots are most suitably powered by batteries

The weight and energy capacity of the batteries may become the determinative factor of its performance


For a manually controlled robot, you can use batteries or voltage eliminators (convert the normal 220V supply to the required DC voltage 12V , 24V etc.)


Gear

Belt Pulley

Chain Sprocket

Rack and Pinion

Pick Place Mechanisms


Gears are the most common means of transmitting power in mechanical engineering

Gears form vital elements of mechanisms in many machines such as vehicles, metal tooling machine tools, rolling mills, hoisting etc.

In robotics its vital to control actuator speeds and in exercising different degrees of freedom


To achieve torque magnification and speed reduction

They are analogous to transformers in electrical systems

It follows the basic equation:

ω1 x r1 = ω2 x r2

Gears are very useful in transferring motion between different dimension


An arrangement of gears to convert rotational torque to linear motion

Same mechanism used to steer wheels using a steering

In robotics used extensively in clamping systems


It allows for mechanical power, torque, and speed to be transmitted across axes

If the pulleys are of differing diameters, it gives a mechanical advantage

In robotics it can be used in lifting loads or speed reduction

Also it can be used in a differential drive to interconnect wheels


Sprocket is a profiled wheel with teeth that meshes with a chain

It is similar to the system found in bicycles

It can transfer rotary motion between shafts in cases where gears are unsuitable

Can be used over a larger distance

Compared to pulleys has lesser slippage due to firm meshing between the chain and sprocket


For picking and placing many mechanisms can be used:

Hook and pick

Clamp and pick

Slide a sheet below and pick

Many other ways

Lots of Scope for innovation


Image Processing is a tool for analyzing image data in all areas of natural science

It is concerned with extracting data from real-world images

Differences from computer graphics is that computer graphics makes extensive use of primitives like lines, triangles & points. However no such primitives exist in a real world images.


Increasing need to replicate human sensory organs

Eye (Vision) : The most useful and complex sensory organ


Automated visual inspection system Checking of objects for defects visually

Remote Sensing

Satellite Image Processing

Classification (OCR), identification (Handwriting, finger prints) etc.

Detection and Recognition systems (Facial recognition..etc)

Biomedical applications


Camera, Scanner or any other image acquisition device

PC or Workstation or Digital Signal Processor for processing

Software to run on the hardware platform (Matlab, Open CV etc.)

Image representation to process the image (usually matrix) and provide spatial relationship

A particular color space is used to represent the image(c) 2009-2010 Electronics & Robotics Club, BITS-Pilani, Goa | Ajusal Sugathan & Utkarsh Sinha


Image Acquisition Device

(Eg. CCD or CMOS Camera)

Image Processor

(Eg. PC or DSP)

Image Analysis Tool

(Eg. Matlab or Open CV)

Machine Control Of Hardware through serial or parallel interfacing

Using a camera

Analog cameras

Digital cameras› CCD and CMOS cameras

Captures data from a single

light receptor at a time

CCD – Charge Coupled Devices

CMOS – Complementary MOSFET Sensor based


Digital Cameras› CCD Cameras

High quality, low noise images

Genarates analog signal converted using ADC

Consumes high power

› CMOS Cameras

Lesser sensitivity

Poor image quality

Lesser power

Analogue cameras require grabbing card or TV tuner card to interface with a PC



Colored pixels on CCD Chip

Matlab

Open CV


Two types: Vector and Raster

Vector images store curve information

Example: India‘s flag

Three rectangles, one circle and the spokes

We will not deal with vector images at all


Raster images are different

They are made up of several dots


If you think about it, your laptop‘s display is a raster display

Also, vector images are high level abstractions

Vector representations are more complex and used for specific purposes


Raster› Matrix

Vector› Quadtrees

› Chains

› Pyramid

Of the four, matrix is the most general. The other three are used for special purposes. All these representations must provide for spatial relationships


Computers cannot handle continuous images but only arrays of digital numbers

So images are represented as 2-D arrays of points (2-D matrix)(Raster Represenatation)

A point on this 2-D grid (corresponding to the image matrix element) is calledPIXEL (picture element)

It represents the average irradiance over the area of the pixel


Each pixel requires some memory

Color depth : Amount of memory each pixel requires

Examples

› 1-bit

› 8-bit

› 32-bit

› 64-bit


Pixels are tiny little dots of color you see on your screen, and the smallest possible size any image can get

When an image is stored, the image file contains information on every single pixel in that image i.e› Pixel Location› Intensity

The number of pixels used to represent the image digitally is called Resolution

More the number of pixels used, higher the resolution

Higher resolution requires more processing power


MATLAB stands for MATrix LABoratory, a software developed by MathworksInc (www.mathworks.com). MATLAB provides extensive library support for various domains of scientific and engineering computations and simulations


When you click the MATLAB icon (from your desktop or Start>All Programs), you typically see three windows: Command Window, Workspace and Command History. Snapshots of these windows are shown below


This window shows the variables defined by you in current session on MATLAB


Command History stores the list of recently used commands for quick reference


This is where you run your code


In MATLAB, variables are stored as matrices (singular: matrix), which could be either an integer, real numbers or even complex numbers

These matrices bear some resemblance to array data structures (used in computer programming)


Let us start with writing simple instructions on MATLAB command window

To define an integer,

Type a=4 and hit enter

>>a=4

To avoid seeing the variable, add a semicolon after the instruction

>>a=4;


Similarly to define a 2x2 matrix, the instruction in MATLAB is written as

>> b=[ 1 2; 3 4];

If you are familiar with operations on matrix, you can find the determinant or the inverse of the matrix.

>> determin= det(b)

>> d=inv(b)


Images as we have already seen are stored as matrices

So now we try to see this for real on MATLAB

We shall also look into the basic commands provided by MATLAB‘s Image Processing Toolbox


Once you have started MATLAB, type the following in the Command Window

>> im=imread(‗sample.jpg');

This command stores the file image file ‗sample.jpg‘ in a variable called ‗im‘

It takes this file from the Current-Directory specified

Else, entire path of file should be mentioned


You can display the image in another window by using imshow command

>>figure,imshow(im);

This pops up another window (called as figure window), and displays the image ‗im’


The ‗imview‘ command can also be used in order toview the image

imview(im);

Difference is that in this case you can see specific pixel values just by moving the cursor over the image


To know the breadth and height of the image, use the size function,

>>s=size(im);

The size function basically gives the size of any array in MATLAB

Here we get the size of the IMAGE ARRAY


Now that we have our image stored in a variable we can observe and understand the following:

How pixels are stored?

What does the values given by each pixel indicate?

What is Image Resolution?


Have a look at the values stored

Say the first block of 10 x 10

>>im(1:10,1:10);

Or Say view the pixel range 50:150 on both axis

>> figure,imshow(im(50:150,50:150));


1-bit = BLACK or WHITE

8-bit = 28 different shades

24-bit = 224 different shades

64-bit images – High end displays

Used in HDRI, storing extra information per pixel, etc


This is another name for 1-bit images

Each pixel is either White or Black

Technically, this is a black & white image


Another name for 8-bit images

Each pixel can be one of 256 different shades of gray

These images are popularly called Black & White. Though, this is technically wrong.


Again, each pixel gets 8 bits

But each of the 256 values maps to a color in a predefined ―palette‖

If required, you can have different bit depths


We won‘t be dealing with indexed images


8-bits is too less for all the different shades of colors we see

So 24-bits is generally used for color images

Thus each pixel can have one of 224

unique colors


Now, a new problem arises:

How do you manage so many different shades?

Programmers would go nuts

Then came along the idea of color spaces


A color space can be thought of as a way to manage millions of colors

Eliminates memorization, and increases predictability

Common color spaces:

› RGB

› HSV

› YCrCb or YUV

› YIQ


You‘ve probably used this already


Each pixel stores 3 bytes of data

The 24-bits are divided into three 8-bit values

The three are: Red, Green and Blue i.ethe primary colours

Mixing of primary colours in right proportions gives any particular colour

Each pixel has these 3 values


1 byte = 8 bits can store a value between 0-255

We get pixel data in the form RGB values with each varying from 0-255

That is how displays work

So there are 3 grayscale channels


Advantages:

› Intuitive

› Very widely used

Disadvantages:

› Image processing is relatively tough


HSV makes image processing easier

Again, 24 bits = three 8-bit values or 3 channels

The 3 channels are: › Hue

› Saturation (Shade of Colour)

› Value (Intensity)


The Hue is the tint of color used› It represents the colour of the pixel (Eg. Red

Green Yellow etc)

The Saturation is the ―amount‖ of that tint› It represents the intensity of the colour (Eg.

Dark red and light red)

The Value is the ―intensity‖ of that pixel› It represents the intensity of brightness of the

colour


RGB image converted to HSV


RGB

HUE

SATURATION

VALUE

Advantages:

› The color at a pixel depends on a single value

› Illumination independent

Disadvantages:

› Something


Intuitively RGB might seem to be the simpler and better colour space to deal with

Though HSV has its own advantages especially in colour thresholding

As the colour at each pixel depends on a single hue value it is very useful in separating out blobs of specific colours even when there are huge light variations

Thus it is very useful in processing real images taken from camera as there is a large amount of intensity variation in this case

Hence, ideal for robotics applications


Widely used in digital video Has three 8-bit channels:

› Y Component: Gives luminance or intensity

› Cr Component: It is the RED component minus a reference value

› Cb Component: It is the BLUE component minus a reference

value

Hence Cr and Cb components represent the colour called ―Color Difference Components‖


Advantages:

› Used in video processing

› Gives you a 2-D colour space hence helps in closer distinguishing of colours

Disadvantages:


The camera returns images in a certain color space

You might want to convert to different color spaces to process it

Colour space conversions can take place between RGB to any other colourspace and vice versa


Since cameras usually input images in rgb

We would like to convert these images into HSV or YCrCb

Conversions:

› RGB->HSV

› HSV->RGB

› RGB->YCrCb

› YCrCb->RGB


RGB -> HSV



HSV RGB YCrCb

>>h = rgb2hsv(im)

This converts the RGB image to HSV

The new colour space components can be seen using

>> imview(h)

>> imview(h(:,:,1)) ―—HUE—‖

>> imview(h(:,:,2)) ―—Saturation—‖

>> imview(h(:,:,3)) ―—Value—‖(c) 2009-2010 Electronics & Robotics Club, BITS-Pilani, Goa | Ajusal Sugathan & Utkarsh Sinha

>>R = hsv2rgb(im)

This converts the HSV image to RGB


>> imview(R)

>> imview(R(:,:,1)) ―—Red—‖

>> imview(R(:,:,2)) ―—Green—‖

>> imview(R(:,:,3)) ―—Blue—‖


>> Y = rgb2ycbcr(im);

This converts the RGB image to YCbCr


>> imview(Y)

>> imview(Y(:,:,1)) ―—Luminance—‖

>> imview(Y(:,:,2)) ―—Differenced Blue—‖

>> imview(Y(:,:,3)) ―—Differenced Red—‖


>> R = ycbcr2rgb(im);

This converts the YCbCr image to RGB


>> imview(R)

>> imview(R(:,:,1)) ―—Red—‖

>> imview(R(:,:,2)) ―—Green—‖

>> imview(R(:,:,3)) ―—Blue—‖


Formulae for conversion are very complex

But the best thing is, you don‘t need to remember these formulae

Matlab and OpenCV have built-in functions for these transformations :-)


OpenCV is a collection of many functions that help in image processing

You can use OpenCV in C/C++, .netlanguages, Java, Python, etc as well

We will only discuss OpenCV in C/C++


It is blazingly fast

Quite simple to use and learn

Has functions for machine learning, image processing, and GUI creation


Download the latest OpenCV package from

http://sourceforge.net/projects/opencv/

Install the package, and note where you installed it (like C:\Program Files\OpenCV\)




Now, we need to tell Microsoft Visual Studio that we‘ve installed OpenCV

So, we tell it where to find the OpenCV header files

Start Microsoft Visual Studio 2008



1

2


Type these paths into the list

Right now, Visual Studio knows where to find the OpenCV include files and library files

Now we create a new project


Accept all default settings in the project

You‘ll end up with an empty project with a single file (like Mybot.cpp)

Open this file, we‘ll write some code now


Add the following at the top of the code

#include <cv.h>

#include <highgui.h>

This piece of code includes necessary OpenCV functionality


Now, we get to the main() function

int main()

{

The main function is where for program execution begins


Next, we load an image

IplImage* img = cvLoadImage("C:\\hello.jpg");

The IplImage is a data type, like int, char, etc


Comes built-into OpenCV

Any image in OpenCV is stored as an IplImage thingy

It is a ―structure‖


Opens filename and returns it as an IplImage structure

Supported formats:› Windows bitmaps - BMP, DIB› JPEG files - JPEG, JPG, JPE› Portable Network Graphics - PNG› Portable image format - PBM, PGM, PPM› Sun rasters - SR, RAS› TIFF files - TIFF, TIF› OpenEXR HDR images - EXR› JPEG 2000 images - jp2


Now we show this image in a window

cvNamedWindow("myfirstwindow");

cvShowImage("myfirstwindow", img);

This uses some HighGUI functions (comes along with OpenCV)


Creates a window with the caption title

This is a HighGUI function

You can add controls to each window as well (track bars, buttons, etc)


Shows img in the window with caption title

If no such window exists, nothing happens


Finally, we wait for an input, release and exit

cvWaitKey(0);

cvReleaseImage(&img);

return 0;

}


Waits for time milliseconds, and returns whatever key is pressed

If time=0, waits till eternity

Here, we‘ve used it to keep the windows from vanishing immediately


Erases img from the RAM

Get rid of an image as soon as possible. RAM is precious

Note that you send the address of the image (&img) and not just the image (img)


Right now, Visual Studio knows where OpenCV is

But it does not know, whether to use OpenCV or not

We need to tell this explicitly


Got errors?

› Check if the syntax is correct

› Copy all DLL files in *\OpenCV\bin\ into C:\Windows\System32


src is the original image

dst is the destination

code is one of the follow:

› CV_BGR2HSV

› CV_RGB2HSV

› CV_RGB2YCrCb

› CV_HSV2RGB

› CV_<src_space>2<dst_space>


src should be a valid image. Or an error will pop up

dst should be a valid image, i.e. you need a blank image of the same size

code should be valid (check the OpenCV documentation for that)


Allocates memory for an image of size size, with bits bits/pixel and channumber of channels

Used for creating a blank image

Use cvSize(width, height) to specify the size


Example:

› IplImage* blankImg = cvCreateImage(cvSize(640, 480), 8, 3);


Wired

› Motor Driving module

› Interface with PC (Parallel/Serial)

Wireless

› The Motor-driving module

› The Wireless Receiver Circuit

› The Wireless Transmitter Circuit

› Interface with PC (Parallel/Serial)


IC 7805 Voltage Regulator

L293D Motor Driver

MCT2E Opto-Coupler

Parallel Port Male-Connector

RF-RX Connector


It‘s a three terminal linear 5 volt regulator used to supply the board and other peripherals

Prescribed input voltage to this component is about 7-9 Volts


Voltage fluctuations can be controlled by using low pass filter capacitors across output and input

Higher input voltage can be applied if heatsink is provided


Used to control Dc and Stepper Motors Uses a H-Bridge which is an electronic

switching circuit that can reverse direction of current

It‘s a Dual-H bridge Basically used to convert a low voltage input

into a high voltage output to drive the motor or any other component

Eg: Microcontroller Motor DriverMotor

(5 Volts) (12 Volts)


Different Motor Driver ICs› L293D 600mA Current Rating Dual H-bridge (Dc and Stepper Motors)

› L298N 1 Amp Current Rating Dual H-bridge (Dc and Stepper Motors)

› L297-L298 (Coupled) For stepper motor overdriving Dual H-bridge (Dc and Stepper Motors) 2 Ics in parallel

› ULN2003/ULN2803 500mA Current Rating For unipolar stepper motors


Output Current:

› 600 mA

Output Voltage

› Wide Range

› 4.5 V – 36 V



There are many situations where signals and data need to be transferred from one subsystem to another within a piece of electronics

Relays are too bulky as they are electromechanical in nature and at the same time give lesser efficiency

In these cases an electronic component called Optocoupler is used


They are generally used when the 2 subsystems are at largely different voltages

These use a beam of light to transmit the signals or data across an electrical barrier, and achieve excellent isolation


In our circuit, Opto-isolator (MCT2E) is used to ensure electrical isolation between motors and the PC parallel port during wired connection

The Viz-Board has four such chips to isolate the four data lines (pin 2, pin 3, pin 4, pin 5) coming out of the parallel port

Along with the Viz-Board 2 extensions have been provided i.e

› The Rf Transmitter Module

› The Rf Reciever Module


Transmitter

Receiver

Radio frequency modules are used for data transmission wirelessly at a certain frequency

It sends and receives radio waves of a particular frequency and a decoder and encoder IC is provided to encode and decode this information

Wireless transmission takes place at a particular frequency Eg. 315Mhz

Theses modules might be single or dual frequency


Antenna is recommended on both of them - just connect any piece of 23 cm long to the Antenna pin

The kit has a dual frequency RF module with frequencies 315/434 Mhz


The encoder IC encodes the parallel port data and sends it to the RF transmitter module for wireless transmission

They are capable of encoding information which consists of N address bits and (12-N) data bits


The HT12E Encoder IC has 8 address bits and 4 data bits

A DIP-Switch can be used to set or unset the address bits A0-A7



A0-A7—Address Bits

AD8-AD11—Data Bits

The decoder IC decodes the RF transmitter data and sends it to the parallel port for wireless transmission

They are capable of encoding information which consists of N address bits and (12-N) data bits


The HT12D Decoder IC has 8 address bits and 4 data bits

A DIP-Switch can be used to set or unset the address bits A0-A7



A0-A7—Address Bits

D8-D11—Data Bits

Serial Port

Parallel Port

USB


Data is transferred serially i.e packets are sent one after the other through a single port


Data is transferred in parallel through different data pins at the same time

Communication is pretty fast

Found in old printer ports



25th pin : Ground2nd-12th pin : I/O lines

Parallel port is faster than serial

A mass of data can be transmitted at the same time through parallel ports

Though parallel and serial ports are not found these days in laptops

Desktops and old laptops have these ports



Direct Output from parallel port

Output from motor driver

Camera, object and source positions



Image samplingand quantization


Continuous image projected

on an array sensor

Result of image samplingand quantization


Sampling:Digitizing the coordinate values(spatial resolution)

Quantization:Digitizing the amplitude values(intensity levels)


• 1 bit /pixel

• B bits/pixel

–2B gray levels

–1 byte = 8 bits –> 256 levels

–2 possible values

–2 gray levels -> 0 or 1 (binary image)

All this sampling and quantization puts in extra noise on the image!

Noise can be reduced by

› Using hardware

› Using software: filters


Why do we need to enhance images?

Why filter images?


Large amounts of external disturbances in real images

Due to different factors like changing lighting and other real-time effects

To improve quality of a captured image to make it easier to process the image


First step in most IP applications

Used to remove noise in the input image

To remove motion blur from an image

Enhancing the edges of an image to make it appear sharper


Generally used types Of Filtering

› Averaging Filter

› Mean Filter

› Median Filter

› Gaussian Smoothing

› Histogram Equalization


The Averaging filter is used to sharpen the images by taking average over a number of images

It eliminates noise by assuming that different snaps of the same image have different noise patterns


Noise is gaussian in nature i.e follows a gaussian curve

Hence, summing up noises infinite times approaches zero


This is extremely useful for satellites that take intergalactic photographs

The images are extremely faint, and there is more noise than the image itself

Millions of pictures are taken, and averaged to get a clear picture


The Mean is used to soften an image by averaging surrounding pixel values


Center pixel = (22+77+48+150+77+158+0+77+219)/9

The center pixel would be changed from 77 to 92 as that is the mean value of all surrounding pixels

This filter is often used to smooth images prior to processing

It can be used to reduce pixel flicker due to overhead fluorescent lights


This replaces each pixel value by the median of its neighbors, i.e. the value such that 50% of the values in the neighborhood are above, and 50% are below

This can be difficult and costly to implement due to the need for sorting of the values

However, this method is generally very good at preserving edges


Its performance is particularly good for removing short noise

The median is calculated by first sorting all the pixel values from the surrounding neighborhood into numerical order and then replacing the pixel being considered with the middle pixel value

If the neighborhood under consideration contains an even number of pixels, the average of the two middle pixel values is used


Used to `blur' images and remove detail and noise

The effect of Gaussian smoothing is to blur an image

The Gaussian outputs a `weighted average' of each pixel's neighborhood, with the average weighted more towards the value of the central pixels


A Gaussian provides gentler smoothing and preserves edges better than a similarly sized mean filter


Before Blurring

After Blurring

It is very useful in contrast enhancement

Especially to eliminate noise due to changing lighting conditions etc

Transforms the values in an intensity image so that the histogram of the output image approximately matches a specified histogram



Filters and histograms

‗Imfilter‘ function is used for creating different kinds of filters In MATLAB

B = imfilter(A,H,‘option‘) filters the multidimensional array A with the multidimensional filter H

The array A can be a nonsparse numeric array of any class and dimension

The result B has the same size and class as A



Options in imfilter

Convolution is same as correlation except that the h matrix is inverted before applying the filter

h = ones(5,5) / 25;

imsmooth = imfilter(im,h);

Here a mean filter is implemented using the appropriate ‗h‘ matrix

imshow(im), title('Original Image');

figure, imshow(imsmooth), title('Filtered Image')


FSPECIAL is used to create predefined filters

h = FSPECIAL(TYPE);

FSPECIAL returns h as a computational molecule, which is the appropriate form to use with imfilter


The process of adjusting intensity values can be done automatically by the histeq function

>>im = imread('pout.tif');

>>jm = histeq(im);

>>imshow(jm)

>>figure, imhist(jm,64)



Original Image


Histogram Equalized Image

Things aren‘t as simple as they were in Matlab

C/C++ needs a bit of syntax and formalities


We‘ll try doing the following right now

› Gaussian filter

› Median filter

› Bilateral filter

› Simple blur

› Averaging filter


Start Microsoft Visual Studio 2008

I assume you have OpenCV installed


#include <cv.h>



#include <cv.h>


int main()

{


#include <cv.h>


int main()

{

IplImage* img = cvLoadImage(“C:\\noisy.jpg”);


#include <cv.h>


int main()

{

IplImage* img = cvLoadImage(“C:\\noisy.jpg”);

IplImage* imgBlur = cvCreateImage(cvGetSize(img),

8, 3);

IplImage* imgGaussian = cvCreateImage(cvGetSize

(img), 8, 3);

IplImage* imgMedian = cvCreateImage(cvGetSize

(img), 8, 3);

IplImage* imgBilateral = cvCreateImage(cvGetSize

(img), 8, 3);


cvSmooth(img, imgBlur, CV_BLUR, 3, 3);

cvSmooth(img, imgGaussian, CV_GAUSSIAN, 3, 3);

cvSmooth(img, imgMedian, CV_MEDIAN, 3, 3);

cvSmooth(img, imgBilateral, CV_BILATERAL, 3, 3);


cvNamedWindow(“original”);

cvNamedWindow(“blur”);

cvNamedWindow(“gaussian”);

cvNamedWindow(“median”);

cvNamedWindow(“bilateral”);

cvShowImage(“original”, img);

cvShowImage(“blur”, imgBlur);

cvShowImage(“gaussian”, imgGaussian);

cvShowImage(“median”, imgMedian);

cvShowImage(“bilateral”, imgBilateral);

cvWaitKey(0);

return 0;

}


Blur: The plain simple Photoshop blur

Gaussian: The best result (preserved edges and smoothed out noise)

Median: Nothing special

Bilateral: Got rid of some noise, but preserved edges to a greater extend


Your OpenCV installation comes with detailed documentation

*\OpenCV\docs\index.html

Scroll down, and you‘ll see OpenCV Reference Manuals


Try looking up cvSmooth in the CV Reference Manual


Now try looking up cvEqualizeHist


There are no built-in functions for this

So, we‘ll code it ourselves

And this will be a good exercise for getting better at OpenCV


#include <cv.h>



#include <cv.h>


int main()

{


#include <cv.h>


int main()

{

IplImage* imgRed[25];

IplImage* imgGreen[25];

IplImage* imgBlue[25];

Holds the R, G and B channels separately for each of the 25 images


IplImage* imgBlue[25];

for(int i=0;i<25;i++)

{

IplImage* img;

char filename[150];

sprintf(filename, "%d.jpg", (i+1));

img = cvLoadImage(filename);

• Generate the strings ―1.jpg‖, ―2.jpg‖, etc and store them into filename• Load the image filename


img = cvLoadImage(filename);

imgRed[i] = cvCreateImage(cvGetSize(img), 8,

1);

imgGreen[i] = cvCreateImage(cvGetSize(img), 8,

1);

imgBlue[i] = cvCreateImage(cvGetSize(img), 8,

1);

• Allocate memory for each component of image i


imgBlue[i] = cvCreateImage(cvGetSize(img), 8,

1);

cvSplit(img, imgBlue[i], imgGreen[i],

imgRed[i], NULL);

cvReleaseImage(&img);

}

• Split img into constituent channels• Note the order: B G R• Release img


We created 75 grayscale images: 25 for red, 25 for green and 25 for blues

Loaded 25 color images in the loop

Split each image, and stored in an appropriate grayscale image


CvSize imgSize = cvGetSize(imgRed[0]);

IplImage* imgResultRed = cvCreateImage(imgSize, 8,

1);

IplImage* imgResultGreen = cvCreateImage(imgSize,

8, 1);

IplImage* imgResultBlue = cvCreateImage(imgSize,

8, 1);

IplImage* imgResult = cvCreateImage(imgSize, 8,

3);

• This will hold the final, filtered image• It will be a combination of the grayscale channels imgResultRed, imgResultGreen and imgResultBlue


IplImage* imgResult = cvCreateImage(imgSize, 8,

3);

for(int y=0;y<imgSize.height;y++)

{

for(int x=0;x<imgSize.width;x++)

{

• Two loops to take us through the entire image


for(int x=0;x<imgSize.width;x++)

{

int theSumRed=0;

int theSumGreen=0;

int theSumBlue=0;


{

• To figure out the average, we need to find the numerator (the sum) over all 25 images



{

theSumRed+=cvGetReal2D(imgRed[i], y,

x);

theSumGreen+=cvGetReal2D(imgGreen[i],

y, x);

theSumBlue+=cvGetReal2D(imgBlue[i], y,

x);

}

• To figure out the average, we need to find the numerator (the sum) over all 25 images


theSumRed = (float)theSumRed/25.0f;

theSumGreen = (float)theSumGreen/25.0f;

theSumBlue = (float)theSumBlue/25.0f;

cvSetReal2D(imgResultRed, y, x,

theSumRed);

cvSetReal2D(imgResultGreen, y, x,

theSumGreen);

cvSetReal2D(imgResultBlue, y, x,

theSumBlue);

}

}

• Once we have the sum, we divide by 25 and set the appropriate pixels


cvMerge(imgResultBlue, imgResultGreen,

imgResultRed, NULL, imgResult);

cvNamedWindow("averaged");

cvShowImage("averaged", imgResult);

cvWaitKey(0);

return 0;

}

• Merge the three channels, and display the image


cvLoadImage always loads as BGR

cvSplit to get the individual channels

cvMerge to combine individual channels into a color image


IplImage to store any image in OpenCV

cvCreateImage to allocate memory

cvReleaseImage to erase an image from the RAM


cvWaitKey to get a keypress within certain milliseconds

cvNamedWindow to create a window

cvShowImage to show an image in a window


cvGetReal2D to get value at a pixel in grayscale images

cvSetReal2D to set the value at a pixel

CvSize to store an image‘s size


you can always refer to the OpenCV documentation


The process of extracting image components that are useful in representation of image for some particular purpose

Basic morphological operations are:

› Dilation

› Erosion


The operation that grows or thickens objects in a binary image

The specific manner of thickening is controlled by a shape referred to as ―structuring element‖



Structuring Element

Binary Image


Dilated Image

Erosion shrink or thins objects in a binary image

The manner of shrinkage is controlled by the structuring element



Structuring Element

Binary Image


Eroded Image

In practical image processing dilation and erosion are performed in various combinations

An image can undergo a series for diltions and erosion using the same or different structuring element


In practical image processing dilation and erosion are performed in various combinations

An image can undergo a series for diltions and erosion using the same or different structuring element

Two Common Kinds:

› Morphological Opening

› Morphological Closing


It is basically one erosion followed by one dilation by the same structuring element

They are used to smooth object contours, break thin connections and remove thin protrusions



A—ImageB—Structuring element

It is basically one dilation followed by one erosion by the same structuring element

They are used to smooth object contours like opening

But unlike opening they generally join narrow breaks, fill long thin gulfs and fills holes smaller than the structuring element



A—ImageB—Structuring element

Used to generate a structuring element

>>se=strel(shape,parameters)


Dilation in matlab is done using the following command:

>>bw2=imdilate(bw,st)

Bw = Original image

St = Structuring element


Erosion in matlab is done using the following command:

>>bw2=imerode(bw,st)

Bw = Original image



Opening in matlab is done using the following command:

>>bw2=imopen(bw,st)

Bw = Original image



Closing in matlab is done using the following command:

>>bw2=imclose(bw,st)

Bw = Original image



cvErode(src, dst)

cvDilate(src, dst)

Opening & closing: use the appropriate sequence


By default, OpenCV uses the zero structuring element (all are zeros)

You can explicitly specify your structuring element as well

Check the OpenCV Documentation for more information


Computers can manipulate images very efficiently

But, comprehending an image with millions of colors is tough

Solution: Figure out interesting regions, and process them


Each pixel is checked for its value

If it lies within a range, it is marked as ―interesting‖ (or made white)

Otherwise, it‘s made black

Figuring out the range depends on lighting, color, texture, etc


Demo thresholdRGB

Demo thresholdHSV


MATLAB provides a facility to execute multiple command statements with a single command. This is done by writing a .m file

Goto File > New > M-file

For example, the graythresh function can be manually written as a m-file as:


Observe that, comments (in green) can be written after the symbol ‗%‘. A commented statement is not considered for execution

M-files become a very handy utility for writing lengthy programs and can be saved and edited, as and when required

We shall now see, how to define your own functions in MATLAB.


Functions help in writing organized code with minimum repetition of logic

Instead of rewriting the instruction set every time, you can define a function

Syntax:

Create an m-file and the top most statement of the file should be the function header

function [return values] = function-name(arguments)


The inbuilt graythresh function in matlab is used for thresholding of grayscale images

It uses the Otsu‘s Method Of thresholding

A sample thresholding opreation has been shown in the next slide



Image thresholded for the colour blue


The Real thing

Thresholding of a grayscale image can be done in MATLAB using the following commands:

>> level=graythresh(imGRAY);

>> imBW = im2bw(imGRAY,level);

>> imview(imBW);


The graythresh command basically gives an idea as to what exactly the threshold value should be

Graythresh returns a value that lies in the range 0-1

This gives the level of threshold which is obtained by a complex method called the Otsu‘s Method of Thresholding


This level can be converted into pixel value by multiplying by 255

Lets say, level=.4

Then threshold value for the grayscale image is:

0.4 x 255 =102


What this indicates is that for the given image the values below 102 have to be converted to 0 and values from 103-255 to the value 1

Conversion from grayscale to binary image is done using the function:

>>imBW = im2bw(imGRAY,level);


Here level is the threshold level obtained from graythresh function

This function converts pixel intensities between 0 to level to zero intensity (black) and between level+1 to 255 to maximum (white)


In order to threshold an RGB colourimage using the graythresh function, the following have to be done:› Conversion of the RGB image into its 3

grayscale components

› Subtracting each of these components from the other 2 to get the pure colour intensities

› Finding level for each of the grayscale using graythresh

› Thresholding the image using imbw and the level


Commands:

Im=Imread(‗rgb.jpg‘);

R = im(:,:,1); --Red

G = im(:,:,2); --Green

B = im(:,:,3); --Blue


Ronly=R-B-G; --Pure RED

Gonly=G-R-B; --Pure GREEN

Bonly=B-G-R; --Pure BLUE


Levelr=graythresh(Ronly);

Levelg=graythresh(Gonly);

Levelb=graythresh(Bonly);


Rthresh=im2bw (Ronly,levelR);

Gthresh=im2bw(Gonly,levelG);

Bthresh=im2bw(Bonly,levelB);


Using a manually designed thresh_toolfunction to adjust the levels as required

To get a feel of how levels vary


s=size(im); temp=im; thresh=128; for i=1:s(1,1)

for j=1:s(1,2)

if temp(i,j)<thresh temp(i,j)=0;

else temp(i,j)=255;

end end

end imview(temp);


Splitting of HSV image into components

Using the Hue channel and thresholding it for different values

Since the hue value of a single colouris constant it is relatively simple to threshold and gives better accuracy


function [temp] = ht(im,level1,level2)s=size(im); temp=im; for i=1:s(1,1)

for j=1:s(1,2) if (temp(i,j)<level2 & temp(i,j)>level1)

temp(i,j)=1; else

temp(i,j)=0; end

end end imview(temp);


To this function we give the input arguments as the upper and lower bounds of the threshold levels

These levels can be obtained by having a look at the range of hue values for the particular colour


Now that you know the basics


cvThreshold(src, dst, threshold, max, type)

type:

› CV_THRESH_BINARY

› CV_THRESH_BINARY_INV

› And several others (check documentation)


cvInRangeS(src, scalarLower, scalarUpper, dst);

scalarLower = cvScalar(chan1, chan2, chan3, chan4);

scalarUpper = cvScalar(chan1, chan2, chan3, chan4);


Ultra basics: motors, drives, etc

Digital image representation

Color spaces

Inter-conversion of color spaces

Electronics

Filtering

Thresholding

Morphology


After thresholding, we get a binary image

We want useable information like centers, outlines, etc

There geometrical properties can be found using many methods. We‘ll talk about moments and contours only.


Moments are a mathematical concept

∑ ∑intensity*xxorder*yyorder


Consider xorder=0 and yorder=0 for a binary image

So you‘re just summing up pixel values

This means, you‘re calculating the area of the white pixels


Now consider xorder=1 and yorder=0 for a binary image

You sum only those x which are white

So you‘re calculating the numerator of an average


The number of points where the pixel is white is the area of the image

So, dividing this particular moment (xorder=1, yorder=0) by the earlier example (xorder=0, yorder=0) gives the average x

This is the x coordinate of the centroidof the blob


Similarly, for xorder=0 and yorder=1, you‘ll get the y coordinate


The order of a moment = xorder+yorder

So, the area is a zero order moment

The centroid coordinate = a first order moment / the zero order moment


There are entire books written on this topic

You can find complex geometrical properties, like the eccentricity of an ellipse, radius of curvature of objects, etc

Also check for Hu invariants if you‘re interested



Centroid Area etc

These are pixels of an image that are conencted to each other forming separate blobs in an image

They can be seperated out and labelled


>>L = bwlabel(BW,n)

Returns a matrix L, of the same size as BW, containing labels for the connected objects in BW

n can have a value of either 4 or 8, where 4 specifies 4-connected objects and 8 specifies 8-connected objects; if the argument is omitted, it defaults to 8


STATS = regionprops(L,properties)

Measures a set of properties for each labeled region in the label matrix L

The set of elements of L equal to 1 corresponds to region 1; the set of elements of L equal to 2 corresponds to region 2; and so on


'Area'– The actual number of pixels in the region

'Centroid'-- The center of mass of the region. Note that the first element of Centroid is the horizontal coordinate (or x-coordinate) of the center of mass, and the second element is the vertical coordinate (or y-coordinate)

'Orientation' -- Scalar; the angle (in degrees) between the x-axis and the major axis of the ellipse that has the same second-moments as the region. This property is supported only for 2-D input label matrices


BW = imread('text.png');

L = bwlabel(BW);

stats = regionprops(L,'all');


Label into an RGB image for better vizualization

RGB = label2rgb(L)


Binary area open remove small objects

BW2 = bwareaopen(BW,P)

Removes from a binary image all connected components (objects) that have fewer than P pixels, producing another binary image, BW2.


OpenCV supports functions to calculate moments upto order 3

CvMoments *moments = (CvMoments*)malloc(sizeof

(CvMoments));

cvMoments(img, moments, 1);


cvGetSpatialMoment(moments, xorder, yorder)

cvGetCentralMoment(moments, xorder, yorder)

Central = spatial/area


Example


For robotics purposes, moments are fine till have one single object

If we have multiple objects in the same binary image


You can think of contours as an approximation of a binary image


You get polygonal approximation of each connected area


The output you get for the previous binary image is:

› Four ―chains‖ of points

› Each chain can have any number of points

› In our case, each chain has four points



Contour plotting

Contour plot of an image im can be made in MATLAB using the command:

im = imread(‗img.jpg');

imcontour(im,level)

Level=number of equally spaced contour levels

if level is not given it will choose automatically


OpenCV linked lists to store the ―chains‖

We‘ll see some code to find out the squares in the thresholded image you saw


CvSeq* contours;

CvSeq* result;

CvMemStorage *storage = cvCreateMemStorage(0);

• The chains are stored in contours• result is a temporary variable• storage is for temporary memory allocation


cvFindContours(img, storage, &contours, sizeof(CvContour),

CV_RETR_LIST, CV_CHAIN_APPROX_SIMPLE, cvPoint(0,0));

• img is a grayscale thresholded image• storage is for temporary storage• All chains found would be stored in the contours sequence• The rest of the parameters are usually kept at these values• Check the OpenCV documentation for details information about the last four variables


while(contours)

{

result = cvApproxPoly(contours, sizeof(CvContour),

storage, CV_POLY_APPROX_DP,

cvContourPerimeter(contours)*0.02, 0);

• The previous command makes contours point to the first chain• We‘re approximating the contour right now

• After this command, result stores the approximate contour as a polygon (many points)


if(result->total==4)

{

CvPoint *pt[4];


pt[i] = (CvPoint*)cvGetSeqElem(result, i);

}

• We‘re looking for quadrilaterals, so we check if the number of points in this particular polygon is 4

• Then, get extract each point using the command cvGetSeqElem

• Once you have the points, you can actually check the shape of the object as well (by checking angles, lengths, etc)


// Do whatever you want with the 4 points

contours = contours->h_next;

}

• Do whatever you want to do with the four points

• Then, we move onto processing the next contour


MATLAB has an image acquisition toolbox which helps capture images

Now-a-days most of the cameras are available with USB interface

Once you install the driver for the camera, the computer detects the device whenever you connect it


In MATLAB, you can check if the support is available for your camera

MATLAB has built-in adaptors for accessing these devices

An adaptor is a software that MATLAB uses to communicate with an image acquisition device


COMMANDS:

>> imaqhwinfo

>> cam=imaqhwinfo;

>> cam.InstalledAdaptors


To get more information about the device, type

>>dev_info = imaqhwinfo('winvideo',1)

Instead of ‗winvideo‘, if imaqhwinfoshows another adaptor, then type that adaptor name instead of ‘winvideo’.


You can preview the video captured by the image by defining an object and associate it with the device

>>vid=videoinput(‗winvideo‘,1,‗RGB24_320x240‘)



Now to see the video insert the following command:

>> preview(vid)


You should see a window pop-up, that displays what your camera is capturing


The camera may support multiple video formats. To see for yourself all the supported formats, type

>>dev_info = imaqhwinfo('winvideo',1);

>>celldisp(dev_info.SupportedFormats);

Check out for yourself the display of other formats, by replacing `RGB24_320x240` with other formats, in the definition of the object vid


Now to capture an image from the video, define the object vid as described before and use getdata to capture a frame from the video

>>start(vid); % This command initiates capturing of frames and stores the frames in memory

>>im=getdata(vid,1);

>>imview(im);


You can store the captured image as a .jpg or .gif file using imwrite function

>>imwrite(im,'testimage.gif');

The image will be stored in ‗MATLAB71\work‘ folder

Static Processing

Step Processing

Real-Time Processing


Take a single picture of the arena and process it

Find out critical regions and points

Apply some geometry and mathematical calculations

Then blindly follow a specified path


Advantages

› Simplest to implement

› Can be fairly accurate with stepper motors

Disadvantages

› Bot goes blind because only one picdetermines the bot motion

› Accuracy is very low especially with DC motors


Take images of the arena in discrete intervals (say Eg. 10secs/image)

Process the images and find out critical regions and points

Check bot orientation at these intervals and try to correct

Partial feedback mechanism is implemented



Advantages› Quite simple to implement

› Can be very accurate with stepper motors

Disadvantages› Bot goes blind for a particular period of

time

› Accuracy is compromised with DC motors

› Dynamic environmental changes cannot be accounted for

Take images of the arena continuously at a particular frame rate

Process the images and find out critical regions and points

Check bot orientation at every frame and correct

Complete feedback mechanism is implemented



Advantages

› Very accurate for both DC and Stepper motors

› Gives dynamic feedback and accounts for changing environment

› Can give bot orientation at each point of time

Disadvantages

› Requires more processing power

› Requires more memory for taking so many images

Real-time image processing is the best approach for any real application

Dynamic feedback systems give excellent accuracy and precision and hence is the best approach


Image processing is an important tool in many applications

Problem is that one needs to acquire images and pre-process them before doing actual IP

Sometimes it may be required that offline image processing is not possible i.e. one needs to proceed with real-time IP or even more, processing of the video itself


Every time you want to capture an instantaneous image, you have to stop the video, start it again and use the getdata function

To avoid this repetitive actions, the Image Acquisition toolbox provides an option for triggering the video object when required and capture an instantaneous frame


Create an m-file with following sequence of commands:


vid=videoinput('winvideo',1);

triggerconfig(vid,'manual');

set(vid,'FramesPerTrigger',1 );

set(vid,'TriggerRepeat', Inf);

start(vid);


for i=1:5

trigger(vid);

im= getdata(vid,1);

figure,imshow(im);

end

stop(vid);

delete(vid);

clear vid;



In the above code, object im gets overwritten while execution of each of the interations of the for loop

To be able to see all the five images, replace im with im(:,:,:,i)


triggerconfig sets the object to manual triggering, since its default triggering is of type immediate

In immediate triggering, the video is captured as soon as you start the object ‘vid’

The captured frames are stored in memory. Getdata function can be used to access these frames

But in manual triggering, you get the image only when you ‗trigger’ the video


‗FramesPerTrigger‘ decides the number of frames you want to capture each time ‘trigger’ is executed

TriggerRepeat has to be either equal to the number of frames you want to process in your program or it can be set to Inf

If set to any positive integer, you will have to ‘start’ the video capture again after trigger is used for those many number of times


Once you are done with acquiring of frames and have stored the images, you can stop the video capture and clear the stored frames from the memory buffer, using following commands:

>>stop(vid);

>>delete(vid);

>>clear vid;


Getsnapshot function returns one image frame and is independent ofFramesPerTrigger property

So if you want to process your images in real-time, this is all you need:


vid=videoinput(‗winvideo‘,1) triggerconfig(vid,'manual'); set(vid,'FramesPerTrigger',1); set(vid,'TriggerRepeat', Inf); start(vid); while(1) { trigger(vid); im= getdata(vid,1); % write your image processing algorithm here % % you may break this infinite while loop if a certain

condition is met }


Using MATLAB

MATLAB provides support to access serial port (also called as COM port) and parallel port (also called as printer port or LPT port) of a PC

MATLAB has an adaptor to access the parallel port (similar to adaptor for image acquisition)


To access the parallel port in MATLAB, define an object

>> parport= digitalio('parallel','LPT1');

You may obtain the port address using,

>> get(parport,'PortAddress') >> daqhwinfo('parallel'); % To get

data acquisition hardware information


You have to define the pins 2-9 as output pins, by using addline function

>> addline(parport, 0:7, 'out')

Now put the data which you want to output to the parallel port into a matrix; e.g.

>> dataout = logical([1 0 1 0 1 0 1 1]);


Now to output these values, use the putvalue function

>> putvalue(parport,dataout);

Alternatively, you can write the decimal equivalent of the binary data and output it

>> data = 23; >> putvalue(parport,data);


You can connect the pins of the parallel port to the driver IC for the left and right motors of your robot, and control the left, right, forward and backward motion of the vehicle

You will need a H-bridge for driving the motor in both clockwise and anti-clockwise directions



Using MATLAB

Things are a little more involved

There is this library called inpout32

Makes your task really simple

Just follow the instructions that come along, and you‘ll be sending data to your robot!


Or you could use the following code

The idea is:

› Create a virtual file that ―represents‖ the port itself (parallel, serial, etc)

› Keep this file open

› And keep writing to this file

› So, data is automatically send to the desired port


Step 1: Create a global variable named hPort

HANDLE hPort;


Step 2: Create function to create the virtual file, and store its ―handle‖ in hPort

(contd)

bool SerialOpen(LPCWSTR strPort){

// Open the serial port.

hPort = (HANDLE)CreateFile (strPort, // Pointer to the name of the port

GENERIC_READ | GENERIC_WRITE, // Access (read-write) mode

0, // Share mode

NULL, // Pointer to the security attribute

OPEN_EXISTING, // How to open the serial port

0, // Port attributes

(long)NULL); // Handle to port with attribute

// to copy

DCB PortDCB;

DWORD dwError;

// Initialize the DCBlength member.

PortDCB.DCBlength = sizeof (DCB);

// Get the default port setting information.

GetCommState (hPort, &PortDCB);


Step 2: Some more port configuration

(contd)

// Change the DCB structure settings.

PortDCB.BaudRate = 9600; // Current baud

PortDCB.fBinary = TRUE; // Binary mode; no EOF check

PortDCB.fParity = TRUE; // Enable parity checking

PortDCB.fOutxCtsFlow = FALSE; // No CTS output flow control

PortDCB.fOutxDsrFlow = FALSE; // No DSR output flow control

PortDCB.fDtrControl = DTR_CONTROL_ENABLE; // DTR flow control type

PortDCB.fDsrSensitivity = FALSE; // DSR sensitivity

PortDCB.fTXContinueOnXoff = TRUE; // XOFF continues Tx

PortDCB.fOutX = FALSE; // No XON/XOFF out flow control

PortDCB.fInX = FALSE; // No XON/XOFF in flow control

PortDCB.fErrorChar = FALSE; // Disable error replacement

PortDCB.fNull = FALSE; // Disable null stripping

PortDCB.fRtsControl = RTS_CONTROL_ENABLE; // RTS flow control

PortDCB.fAbortOnError = FALSE; // Do not abort reads/writes on error

PortDCB.ByteSize = 8; // Number of bits/byte, 4-8

PortDCB.Parity = NOPARITY; // 0-4=no,odd,even,mark,space

PortDCB.StopBits = ONESTOPBIT; // 0,1,2 = 1, 1.5, 2


Step 2: And finally…

// Configure the port according to the specifications of the DCB

// structure.

if (!SetCommState (hPort, &PortDCB))

{

// Could not configure the serial port.

dwError = GetLastError();

printf("Serial port creation error: %d", dwError);

MessageBox(NULL, L"Unable to configure the serial port", L"Error", MB_OK);

return false;

}

return true;

}


This function works equally well for both serial ports and parallel ports


Step 2: I‘ve assumed you‘re creating a function

This function returns a true when the port is created successfully

If you‘re not, replace the ―return‖ statements with ―printf‖s


Example usage of this function

› hPort = SerialOpen(L”COM8:”); // Serial Port

› hPort = SerialOpen(L”LPT1:”); // Parallel

This is actually how you can access ports in DOS as well… using COM8: and LPT1: instead of C:, D:, etc

The L before the quotes is just syntax for C/C++


Step 3: Next, we‘ll write functions to write to the virtual file we‘ve created

bool SerialWrite(byte theByte)

{

// The port wasn't opened

if(!hPort)

return false;

DWORD dwError, dwNumBytesWritten;

WriteFile (hPort, // Port handle

theByte, // Pointer to the data to write

1, // Number of bytes to write

&dwNumBytesWritten, // Pointer to the number of bytes written

NULL // Must be NULL

);

return true;

}


You pass the byte you want to write as a parameter, and it gets written to the port

Example: SerialWrite(12)


Step 4: And finally, a function to close the port

Example: SerialClose()

bool SerialClose(void)

{

if(!hPort)

return false;

CloseHandle(hPort);

return true;

}


Again, I‘ll emphasize that these functions will work equally well for both parallel ports and serial ports


A critical part of image segmentation

Used for detecting meaningful discontinuities in intensity values

Done by finding first and second order derivatives of the image

Also known as gradient of the image


There are different kinds of edge detectors based on the criteria of derivatives

An edge detector can be sensitive to horizontal or vertical lines or both

In detection we try to find out regions where the derivative or gradient is greater than a specified threshold


General Command in MATLAB

[g t] = edge(im,‘method‘,parameters);

g-gradient, t-threshold value


Different methods of edge detection


A threshold value can be given manually as argument

g=edge(im,‘method‘,t);

t-threshold

Sobel& Canny are the more frequently used edge detectors


Image Acquisition

Image Cropping

Image Filtering

Image Segmentation

Image Thresholding

Finding critical points (Centroids)

Finding bot centroidand orientation

Robot Feedback control

Robot Control(c) 2009-2010 Electronics & Robotics Club, BITS-Pilani, Goa | Ajusal Sugathan & Utkarsh Sinha

Static processing

Step processing

Real-time processing


Manual Cropping:

› I2 = imcrop(I);

Coordinate Cropping:

› I2 = imcrop(I,[x1 y1 x2 y2]);


Mean filter

Median filter

Histogram equalization


Use different kinds of edge detections

› Sobel

› Canny


Thresholding

› In RGB

› In HSV

Separating colors


Finding areas of blobs

Finding centroids of blobs


Using orientation


Control using parallel port


The idea of an orientation tag is to precisely indicate the orientation of the robot

It should happen with the least number of operations


This orientation tag is bad

You can tell the position, but not the direction the robot is facing


This orientation tag is excellent for a single bot


But if you have a team of bots, this won‘t be the best choice

You need to have multiple colors for each bot

So, you have more operations

Slowing down the program


A better choice for a team of bots

The asymmetry helps distinguish between multiple bots, with just two colors


Some more orientation tags


With C/C++ you get two methods to capture images:

› Using OpenCV‘s built in libraries

› Using some 3rd party capturing library


Use OpenCV functions

Use DirectX functions (on Windows)


Images, served in C/C++


OpenCV lets you access cameras through the CvCapture structure

So we create a CvCapture structure


Tries to get access to cam #index

Useful when you have multiple cameras attached to the same machine (like in stereo vision)

Then we check if we were able to get exclusive control of the camera. If not, quit.


This window is for our convenience

Displaying what‘s going on within the program, what decisions are taken, etc


We‘ll create a function to take snapshots

It will use the CvCapture structure to tap into the camera‘s stream

It will return a single frame as an IplImage structure

Scroll down, and add the function


Now, we‘ll display a live stream in the window we had created

Because the getSnapshot() function returns a single frame, we need to take snaps regularly

So it goes into the do…while loop


If you try compiling the program right now, you‘ll get an error

getSnapshot() comes ―after‖ the main function

So the compiler doesn‘t know if it exists

Hence we need the so called ―prototype‖


Once we‘re done, we need to release


The cvCaptureFromCam works only for some supported cameras

DirectX is a much bigger library and supports almost all cameras that exist


First, download the Microsoft Platform SDK for Windows Server 2003 R2

› http://www.microsoft.com/downloads/details.aspx?familyid=E15438AC-60BE-41BD-AA14-7F1E0F19CA0D&displaylang=en


http://www.microsoft.com/downloads/details.aspx?familyid=E15438AC-60BE-41BD-AA14-7F1E0F19CA0D&displaylang=en










Second, download the DirectX SDK

› http://www.microsoft.com/downloads/details.aspx?FamilyID=77960733-06e9-47ba-914a-844575031b81&DisplayLang=en


http://www.microsoft.com/downloads/details.aspx?FamilyID=77960733-06e9-47ba-914a-844575031b81&DisplayLang=en










Third, download the DirectShow SDK

› http://www.microsoft.com/downloads/details.aspx?FamilyID=8af0afa9-1383-44b4-bc8b-7d6315212323&DisplayLang=en


http://www.microsoft.com/downloads/details.aspx?FamilyID=8af0afa9-1383-44b4-bc8b-7d6315212323&DisplayLang=en










Fourth, download the VideoInputlibrary

› http://muonics.net/school/spring05/videoInput/


http://muonics.net/school/spring05/videoInput/

http://muonics.net/school/spring05/videoInput/

These are external libraries. So we need to tell Visual Studio where to find the required external files

So we follow steps similar to the OpenCV ones.

The VideoInput package comes with lots of sample code, so you shouldn‘t have much problem


You can check some sample code at this website as well

› http://opencv.willowgarage.com/wiki/DirectShow


http://opencv.willowgarage.com/wiki/DirectShow

http://opencv.willowgarage.com/wiki/DirectShow

For real time, you need to process captured frames as quickly as possible

So we use a loop of some kind (usually a do…while)

Within the loop, you do the following tasks:


Task 1: Capture an image

Task 2: Pre–processing it

Task 3: Process the image

Task 4: Take a decision

Task 5: Move the bot (with feedback)

Task 6: Go to Task 1


Quite obvious

You need an image

Only then can you process something


Once you have the image, you need to enhance it with pre-processing

Increase contrast, reduce noise, smoothen it out, etc


With the pre-processed image, you figure out the location of each object in the arena

Use moments, contours, or anything else

Thresholding, morphology, etc are helpful here


With the location of each object, you can decide what to do next

Bot: Should I go to the red ball because it‘s the closest? Or should I go to the green ball because it has the maximum number of points


Once decided where to go, you make the robot move

You must include feedback in this step itself (maybe another do…while loop)


You have two options:

› Check if further movement is necessary (have all the balls been potted?) If required, only then go to Task 1

› Blindly go to Task 1 (the decision module will tell the bot when to stop)

Both options are good enough


Task 1: Capturing the image

Task 2: Pre-processing the image

Task 3: Processing the image


Task 4: Taking a decision

Task 6: Go to Task 1


Task 5: Move the bot (with feedback)

We won‘t go into code. Just high level logic on how to go about it


For the simplest feedback mechanism, you use pixels

Everything is measure in terms of pixels: distances, coordinates, etc

Angles are usually represented in radians (cos, sin, etc work well with radians)


First, you need to decide your coordinate system

And you need to stick with it throughout your code

Here‘s a coordinate system I used several times


This is a top-down view of the arena, just like the camera sees it


So all your calculations MUST use this particular coordinate system

In particular, you must make sure all your angle calculations are consistent

And that they cycle through 359-1 degrees perfectly


Possible function you might want to create:

› MoveBotToPosition(x, y)

› TurnBotToAngle(angle)


TurnBotToAngle(angle)

› Turns the bot to angle degrees of the coordinate system

› This would be the very basic feedback function

› Within this function, you have a loop

› This loop keeps running as long as the botisn‘t oriented at angle degrees

› You‘ll take snapshots within this function as well


MoveBotToPosition(x,y)

› Moves the bot until it reaches the coordinates (x,y) in the image

› If required, you can also put a call to TurnBotToAngle (to orient the bot to move)

› Again, there‘s a loop which keeps running until the bot reaches the desired position

› And you‘ll need to take multiple snapshots to check where the bot actually is


You obviously need to set a range

If you say TurnBotToAngle(30), it‘s very unlikely that the bot will orient to exactly 30 degrees

A range, say 28-32 should be good enough


Reminder: All x, y and angle we‘ve talked about are in the image, in terms of pixels

We have no idea how they relate to physical distances and angles

But we‘re sure that they are proportional to physical distances and angles


Though, you CAN calibrate your camera and actually figure out physical distances

For example, 5pixels = 3cm

This is very much possible, but of no use for our purposes!


Use classes and structures as much as possible. And you don‘t need to know OOPs to use them.

They really simplify your work, and even make the code more readable


What we‘ve described requires that the bot stops and then checks if the angle is correct or not, etc

Try working on something which checks the bot‘s angle without stopping the bot


Working in the YUV space (this is what cameras use… so thresholding in YUV itself will eliminate processing time consumed by YUV to RGB conversion)


Kalman filters: If a bot is static, still it‘s position might be calculated as different for different frames.

If you use Kalman filters, you can ―smooth out‖ the position data and get precise positioning and angle data


Can you eliminate thresholdingaltogether?

Yes you can!!!

How, you ask? Figure it out! Its EXTREMELY simple!


You now know enough to participate in image processing based competitions

All this knowledge can even serve as a start point for further studies in image processing

Enjoy!


Utkarsh Sinha

Ajusal Sugathan

Oh, btw, visit http://liquidmetal.in/ for more!


http://liquidmetal.in/

anaglyph tutorial

Documents