1Intelligent Driver Assistance System UsingArtificial Neural Networks

Adarsh V R (B100291EC), Adeeb P A (B100118EC), Ashwin S M (B100422EC),Athul M Das (B100123EC), Dileep R (B100004EC)

S8 ECE A BATCHProject Advisor: Dr. Praveen Sankaran

AbstractRoad accidents are an issue of growing concerntoday. Proper following of traffic rules and signboards can helpreduce the accidents to a good extent. We consider designinga system which can assist in safe driving. A system which cangive prompt messages to the driver in advance about the trafficsigns ahead, pedestrians crossing the road etc. can help reduceaccidents. Natural environments in which traffic signs are presentmakes artificial neural networks a suitable choice for dealing withthe problem. In this paper, we provide a report of our initial workin this direction. A brief explanation of artificial neural networksis provided, followed by problem definition, proposed solution,initial simulations, work schedule and conclusions.

Index Termsmachine learning, neural network, perceptrons,back propagation, road accident, traffic signs.

I. INTRODUCTION

I ncreasing number of road accidents point towards theneed for strict adherence to traffic rules. Traffic signs andsignals act as an assistance for safe driving. Following thesesigns and signals can help reduce the number of accidents to agreat extent. A system which can alert the driver of upcomingtraffic signs and signals appropriately can help him/her drivesafely. Such a system would involve continuous monitoringof the environment in front of the vehicle and detection oftraffic signs and signals. After detecting these, they shouldbe properly recognised and the driver alerted appropriately.Future intelligent vehicles would take decisions about thespeed, trajectory etc. on the basis of the signs detected. Sucha system would automatically limit the speed of the vehicle,warn illegal manoeuvres or indicate the presence of trafficsigns to the driver.

The detection and recognition of the traffic signs presentdifficulties due to the variabilities of the natural environmentin which these are present.

A. Difficulties in traffic sign recognition

1. Lighting conditions are not under our control. It can changewith the time of the day, season, cloudiness etc.2. Partial occlusions, shadows etc. due to other objects.3. Object size depends on distance to the object.4. It is not possible to generate an off-line model for thetraffic sign under consideration due to the numerous degreesof freedom of viewing the traffic sign. If the camera optic

axis is not perpendicular to the sign, it could cause a changein the scale for each axis causing a change in the aspect ratio.5. The physical condition of the traffic sign could change dueto age, accidents etc.

All these indicate that we cannot use a simple algorithmicmethod for traffic sign recognition in a real world scenario.A more efficient way would be to use a technique that cantolerate some differences in the data acquired with respect tothe reference data for a traffic sign.

B. Human Nervous System

A digital computer is algorithm based and makes use ofthe stored program concept proposed by Von Neumann[5].With this, we first obtain a step by step procedure for solvinga problem known as algorithm. This is then converted into aprogram which is then executed sequentially by the computer.This approach is particularly suitable for data manipulation,mathematical computations etc. As such, digital computersare faster than human beings in such numerical computations.Instructions are executed in a modern digital computer at therate of around an instruction per nanosecond.

Humans and other animals have a different mechanism fordata storage and computation. The most important differenceis that we humans can learn from our past experiences. Sowe have the ability to store acquired knowledge and useit for taking decisions. Also human beings have a moredistributed system for computation. It is made up of anumber of processing units known as neurons networkedand interconnected together. All the neurons work in parallelmaking our nervous system a heavily parallel architecture.Knowledge is stored in the strength of the interconnectionsbetween neurons which is adapted and changed with ourexperiences. Human beings are slower than digital computersin performing individual computations. We can completecomputations on the order of around a computation permillisecond.However the heavily parallel architecture of our nervoussystem combined with the fact that we can learn from ourexperience means that we can perform certain tasks far morequickly compared to a digital computer. These tasks includepattern recognition (for example face recognition, object

2recognition), perception etc.

C. Need for Artificial Neural Networks

We can tolerate some variations in the data receivedcompared to that of the original data set that we havelearned. We can also perform these tasks quickly than analgorithm based approach. Machine leaning, a branch ofartificial intelligence is an area that deals with systems whichcan learn from past experiences and take decisions basedon that. There are various machine learning algorithms likedecision tree learning, association rule learning, artificialneural networks, inductive logic programming etc [2]. Amongthese artificial neural network is an idea inspired by thebiological neural networks of human beings. With artificialneural networks, computations are structured in terms ofan interconnected structure of artificial neurons. Knowledgeacquired is stored in the weights of the interconnectionsof the neurons. So artificial neural networks have theability for storing acquired knowledge and use it for takingdecisions. They also have a parallel architecture similar tothe biological neural networks.Hence, they are more suitedfor perceptual tasks than a normal algorithmic based approach.

II. LITERATURE SURVEY

There are different types of Artificial Intelligence (AI)technologies available today like

1. Fuzzy logic2. Artificial Neural Network3. Genetic Algorithms4. Decision tree learningEach of these is motivated by the aspiration to go beyond thecapabilities of normal algorithm based approach. Each has itsown advantages and disadvantages.

A. Artificial Neural Networks

As mentioned earlier, artificial neural network is a typeof machine learning algorithm. Artificial neural networks arebased on the biological nervous system of human beings.Similar to that of a biological neural network, an artificialneural network is made up of a network of processing units.In this case, biological neurons are replaced by artificialneurons. Presence of a number of parallel processing elementsmakes artificial neural networks a heavily parallel architecture.Each interconnection in the artificial neural network has aninterconnection weight which resembles the strength of theinterconnections in a biological neural network. These weightsare adapted with the learning process to suit our applicationat hand.

1) Model of a neuron: A neuron is the basic informationprocessing unit of both the biological and artificial neuralnetwork. A biological neuron consists of a cell body, axon,synapses and dendrites. A neuron receives its inputs from

other neurons through dendrites and gives inputs to otherneurons through synapses. Strength of an interconnection inputdetermines the weight of that input. We try to model ourartificial neuron in a similar way.Following figure gives the model of an artificial neuron

Fig. 1. Model of an artificial neuron

The above model consists of a set of synapses, each char-acterised by a weight of its own, a bias, an adder, anactivation function and finally the output. x1, x2, x3, ..., xmdenote the inputs to the neuron. These inputs are weighted bywk1, wk2, wk3, ..., wkm where k is a number characterising aneuron. These weighed inputs are added together along with abias bk by an adder and fed as input to the activation functionof the neuron. Activation function has basically the job oflimiting the amplitude of the weighted sum computed above.So activation function is sometimes also called squashingfunction. A particular case of activation function is a nor-malised activation function in which the output is limited tothe amplitude range of [1, 1].From the above model, we can describe a neuron mathemati-cally in terms of the following two equations.

uk =

mj=1

wkjxj (1)

yk = (uk + bk) (2)

where yk is the output of the kth neuron. Here bias bk is anexternal parameter of the neuron. We can reformulate (2) as

yk = (vk) (3)

where vk =m

j=0 wkjxj , wk0 = 1 and x0 = bk. vk is alsocalled the induced local field of the neuron.

The commonly used activation function is

Sigmoid function:- Sigmoid function is an s-shaped func-tion[1]. It is probably the most common type of activationfunction under use in artificial networks. It takes on continuousvalues. One eaxmple of a sigmoid function is the logisticfunction defined as

(vk) =1

1 + exp(avk) (4)

Here a is known as the slope parameter. Changing the valueof a changes the slope of the sigmoid function.

3Fig. 2. Sigmoid function for different slopes[1]

Another commonly used sigmoid function is the tansigmoidfunction defined as

(vk) = tanh(vk)

2) Network Architectures: Architecture of a neuralnetwork refers to the structure of the neural network.Network architectures may vary in basic layout as well asin complexity. Choosing a particular network architecturedirectly influences the learning algorithm used for trainingthe neural network and the complexity of the application forwhich the neural network is being used.

In general, there are three differnent classes of networkarchitectures.1. Single layer feedforward networks: A layered neuralnetwork is one in which inputs, neurons and outputs arespread over different layers and are interconnected witheach other. Simplest of the layered neural networks are thesingle layer feedforward networks. These networks have aninput layer of source nodes followed by a layer of outputneurons[1]. Connections flow only in the forward direction,hence the name feedforward network.

Fig. 3. A single layer feedforward network

2. Multilayer feedforward network:- A multi layer feedforwardnetwork is different from a single layer feedforward networkin that it has hidden layer(s) of neurons in between the inputand output layers. Neurons in the hidden layer are calledhidden neurons. These neurons intervene between the inputlayer of source nodes and output in some useful manner.These can hence be more complex than single layer neuralnetworks and can be used for more involved applications.Here again connections flow only in the forward direction,

hence again a feedforward network.

Multilayer feedforward networks are one of the mostwidely used network architectures.

Fig. 4. A multilayer feedforward network

3) Learning: One of the most attractive features of artificialneural network is the ability to learn. We can train our neuralnetwork for a data set. Training has to be optimal to getgood results. The data set used for training is crucial. Networkarchitecture used is critical in choosing the learning paradigm.In supervised training, a set of input output pairs is givento train the neural network. Weights of the neural intercon-nections are adjusted properly to give the optimum results.One of the popular supervised learning paradigms is theback propagation algorithm used with multilayer feed forwardnetworks. In back propagation algorithm, error of the network(difference between desired and actual output) is computedduring each stage of training and the weights are adjusted tominimise the error.Unsupervised learning and reinforcement learning are twoother types of learning paradigms used for training neuralnetworks.

4) Back Propagation Algorithm and Feed FrowardNetwork: The back propagation algorithm is a commonmethod of training artificial neural networks which is usedto learn the weights of a multilayer neural network withfixed architecture. It tries to minimize the sum squared errorbetween networks output and given target value.It is a supervised learning method, and is a generalization ofthe delta rule. It requires a dataset of the desired output formany inputs, making up the training set. It is most useful forfeed-forward networks (networks that have no feedback, orsimply, that have no connections that loop). Back propagationrequires that the activation function used by the artificialneurons (or nodes) be differentiable.Feed forward networks often have one or more hidden layersof sigmoid neurons followed by an output layer of linearneurons. Multiple layers of neurons with non-linear transferfunctions allow the network to learn non-linear and linear

4relationships between input and output vectors.The linear output layer lets the network produce valuesoutside the range -1 to +1. On the other hand, if you want toconstrain the outputs of a network (such as between 0 and 1),then the output layer should use a sigmoid transfer function(such as logsig).To illustrate the back propagation process consider a threelayer neural network with two inputs and one output, whichis shown in the picture below:

Each neuron is composed of two units. First unit adds

Fig. 5. 3-layer neural network

products of weights coefficients and input signals. Thesecond unit realise non-linear function, called neuron transfer(activation) function. Signal e is adder output signal, andy = f(e) is output signal of non-linear element. Signal y isalso output signal of neuron. To teach the neural network

Fig. 6. A neuron

we need training data set. The training data set consists ofinput signals (X1 and X2 ) assigned with correspondingtarget (desired output) Z. The network training is an iterativeprocess. In each iteration weights coefficients of nodes aremodified using new data from training data set. Modificationis calculated using algorithm described below: Each teachingstep starts with forcing both input signals from training set.After this stage we can determine output signals values foreach neuron in each network layer.Let us consider that signals are propagating through inputlayer. Symbols W(xm)n represent weights of connectionsbetween network input Xm and neuron n in input layer.Symbols Yn represents output signal of neuron n.Then,Y1 = f1(W(X1)1X1 +W(X2)1X2)Y2 = f2(W(X1)2X1 +W(X2)2X2)Y3 = f3(W(X1)3X1 +W(X2)3X2)

Propagation of signals through the hidden layer: SymbolsWmn represent weights of connections between outputof neuron m and input of neuron n in the next layer.Y4 = f4(W14Y1 +W24Y2 +W34Y3)Y5 = f5(W15Y1 +W25Y2 +W35Y3)

Propagation of signals through the output layer:Y = f6(W46Y4 +W56Y5)

In the next step the output signal of the network, Y iscompared with the desired output value Z (the target), whichis found in training data set. The difference is called errorsignal of output layer neuron. = Y ZThe idea is to propagate error signal (computed in singleteaching step) back to all neurons.The weights Wmn used to propagate errors back are equalto the ones used for computing the output value. Only thedirection of data flow is changed (signals are propagated fromoutput to inputs one after the other). This technique is usedfor all network layers. If propagated errors come from morethan one neuron, they are added. The illustration is below:4 =W465 =W561 =W144 +W155When the error signal for each neuron is computed, the weightcoefficients of each neuron input node may be modified. Informulas below df(e)de represents derivative of neuron activationfunction.

Fig. 7. Algorithm of Back Propagation

W (X1)1 =W(X1)1 + 1df1(e)de (X1)

W (X2)1 =W(X2)1 + 1df1(e)de (X2)

Similarly other weights coefficients also modified.

III. PROBLEM DEFINITIONThere has been significant increase in road accidents over

the years. Human error is a major contributing factor tothis situation. It is a common case that drivers often neglecttraffic signs due to various distractions. The driver may beseeing the sign but the brain may fail to register this due to avariety of reasons. This can lead to accidents. For example ifa driver fails to observe a Sharp Curve Ahead sign, he maynot decrease the speed of the vehicle and this may lead to anaccident.A machine interface that can analyse the surroundings and

5then raise alarms if necessary can be used as a countermeasure to this. This project deals with the smaller part ofthe problem in which we design a system to monitor thesurroundings for traffic signs and classify them accordingly.

IV. PROPOSED SOLUTION

An algorithmic approach may not be a good strategy sincethe image of the traffic sign may be taken from differentangles and the vehicle may be moving at different speeds.It is difficult to compare the input images with our data setas the images might be different from the images to whichcomparisons are made. Thus we need a way to identify imageseven when they are not exactly similar to the image data setat our hands.

Fig. 8. Traffic Sign Detection Problems: a. Reflections b. Not controlled light-ing c. Shadows d,e. Partial occlusions f. Sign Rotation g. Shape Deformation

Neural networks are a good choice in such cases wherewe need to identify a pattern. Training data can be obtainedby taking images of the traffic signs under various practicalscenarios with the vehicle moving at different speeds and atdifferent angles of viewing w.r.t traffic sign. This can thenbe used to train the neural network. Since the processing iscomplex, we cannot use a single layer perceptron but rather amulti-layer perceptron may have to be used.A video recorder can be used to gather video data which isthen processed so as to separate the target area from the overallinput frame. This is then given as input to the neural networkwhich then classifies the input accordingly.

V. WORK DONE

Our project can be mainly divided into two sections. Thetraffic sign detection part is done using image processing andthe recognition part is done using neural networks.The video issent to the image processing section which does the processingand detects the traffic sign and send the detected traffic signto neural network section which in turn recognises the signand give the alert to the driver.

A. Work Done in seventh Semester

This project involved selecting suitable methods for imageprocessing as well as neural network sections, which requireddetermining many parameters like the number of hiddenneurons, the learning rate and so on in the neural networksection. In image processing section, the best features to beused to detect the traffic signs had to be finalized. All of thesecould only be done using simulations. So, the seventh semesterwas mainly dedicated for building our system in MATLAB,ensuring its working and determining the parameters to beused for building the driver assistance system.

1) Image Processing Section: In this section mainly thetraffic sign detection from video was done in MATLAB.Traffic sign detection was done based on features like colourand shape of traffic signs. Colour is a dominant visual featureand is undoubtedly a key piece of information for drivers toidentify traffic signs. Colour information is widely used intraffic sign detection systems, especially for segmentation oftraffic sign images from the rest of the scene. Shape is anotherimportant parameter that can be used to successfully detecttraffic signs. A technique which can detect various shapes inthe image can help us find out traffic signs present in theimage. Hough Transform, which involves making use of theparametric equation of the shape of interest, was used to findout shapes of interest present in the image. [3]

Fig. 9. Input video used for Traffic Sign Detection based on colour

Fig. 10. Result of Traffic Sign Detection based on colour:Traffic Sign detected

6Initially, detection based on colour was done. As part of thiscolour based recognition, a video was taken as input the andnumber of frames in it were calculated. Here a video with29 frames/second was used. Each frame was then fed to afunction that detected the traffic signs using colour. Initiallythe regions containing red color were determined. The imagewas read into a 3 dimensional matrix that was used to store thecorresponding intensities of Red, Green and Blue. The pixelvalues were then traversed one by one. After that, regions withthe intensity value of the Red component greater than thoseof the corresponding green and blue components were found.Thus what we did in effect was to find the regions that aredominant in red component. Once the starting region of thered colour was found the corresponding matrix indices werestored so that the region could be demarcated later.Using Hough Transform, detection using shape was done. Theparametric equation of circle was used to detect circular trafficsigns from the video. After trying out the detection of traffic

Fig. 11. Traffic sign detection based on shape. The images in clockwisefrom top left position are as follows. 1. Colour image 2. Image converted tograyscale 3. Circular shape detected from edge detected image 4. Traffic signdetected and separated from the image 5. Binary image

signs in separate using colour and shape, both of the methodswere combined together to provide a more efficient detectionmechanism. For this the image was first converted from theRGB space to YCbCr space(Y-luminance, Cb- blue differencechroma component, Cr- red difference chroma component).Here detection of traffic signs with a red boundary was done.Cr component in the YCbCr space was used to detect redcolours in the image more easiy. So after converting the imageto YCbCr space, the image corresponding to Cr componentwas obtained. This image was then converted to a binaryimage using proper thresholding. Edge detection was thencarried out on this image. Hough transform was then appliedto find cicular shapes in the edge detected image. Once thecircular shapes were obtained, percentage of red colour in the

circular regions was computed. If it is above a set threshold,the circular region is detected as a traffic sign. This methodprovided better results. Figure 22 below illustrates the methodwhich was done in MATLAB. It can be noted that the sametraffic sign could not be detected simply based on colour dueto problems in proper edge detection. Converting image toYCbCr space and then performing edge detection solves thisproblem.

Fig. 12. Traffic sign detection based on both colour and shape. Images inclockwise direction from top left are 1.Original image 2.Red difference ofchroma of the image 3. Edge detected image 4. Detected traffic sign 5. Binaryimage of 2

Thus, it was concluded that shape and color features oftraffic sign combined could be used for detection purposes.Also working in a colour space other than RGB colour spacecould improve the detection capability of the system.

2) Neural Network Section: In order to understand howneural network works and in order to familiarize with theneural network toolbox in MATLAB, a simulation was donein MATLAB. Since our project deals with the recognition ofimages, as an initial step, recognition of character imagesusing ANN was done in MATLAB. First, a training dataset was created which contained images of different fontsof the alphabets. Characters A-F was chosen and images of10 different fonts of these six characters were created. (Total60 characters). These images were read from MATLAB andwere reshaped to 100 X 100 pixels. Then these images wereconverted to single dimensional arrays of 10000 size andstored as columns of the input vector. For each letter, differentclasses were defined for different fonts. This is to be used asthe training set. A simple complete multilayer neural networkwas created with a single hidden layer having 15 neurons.Using these letters as the training set, the neural networkwas trained using back propagation algorithm. Random noisewas then added to the each of the letters previously used fortraining. A new neural network was created similar to theearlier one and this new network was trained using the newdata set (the one obtained after adding random noise to the

7first training set). It has more number of train vectors. Nowthe two trained networks were tested with the first training setitself as the test set and percentage of recognition of errorswere plotted in both cases. 15% of the training set was usedas the validation set. Number of hidden layers of the neuralnetwork was then changed and the performance change wasevaluated.It is seen in figure 13 that as the number of neurons in thehidden layer increases, the classification error decreases andfinally becomes zero. It is seen in figure 14 that the percentageof error in recognition is zero for the second neural networkwhereas for the first network, it is non-zero.By doing character recognition using neural networks, a briefidea was obtained about the effect of the parameters likelearning rate, number of hidden layers, number of neuronsin a hidden layer, size of training, testing and validation test.

Fig. 13. Performance Dependence on number of neurons in Hidden Layer

Fig. 14. Percentage Error Recognition using two different data sets

VI. WORK DONE IN EIGHTH SEMESTERA. Use of OpenCV

MATLAB is built on Java, and Java is built upon C. Sowhen you run a MATLAB program, your computer is busy

trying to interpret all that MATLAB code. Then it turns it intoJava, and then finally executes the code. OpenCV, on the otherhand, is basically a library of functions written in C/C++. Itis specific purpose library for image processing and computervision. It covers many function for image/video processing,pattern recognition as well as other well known technique thatare usually used in computer vision. We can directly providemachine language code to the computer to get executed.So ultimately we get more image processing done for yourcomputers processing cycles, and not more interpreting. As aresult of this, programs written in OpenCV run much fasterthan similar programs written in MATLAB. That is OpenCVis faster when it comes to speed of execution. For example,In MATLAB, we would typically get 3-4 frames analysed persecond from a video. Whereas In OpenCV, we would get atleast 30 frames per second, resulting in real-time detection.Also due to the high level nature of MATLAB, it uses a lot ofyour systems resources. A MATLAB code requires over a gigabyte(GB) of RAM to run through video. In comparison, typicalOpenCV programs only require appproximately 70MB ofRAM to run in real-time. So for our system which requires realtime video processing, OpenCV provides a better choice thanMATLAB. Hence the system is implemented in OpenCV.[6]

B. Traffic Sign Detection

Image Processing work done involves detecting traffic signsfrom the captured video signal. It involves detecting theregions of interest using colour and shape of the traffic signsof interest.The captured video signal is processed one frame at a time.As a first step in processing, the frame of interest is convertedfrom RGB colour space to HSV colour space. YCbCr colourspace used for processing during last semester was replacedwith HSV space since it was later learned that HSV spaceprovides better performance under varying lighting and envi-ronmental conditions in which the traffic sign may be captured.At present it was decided to correctly recognise three trafficsigns namely school ahead, no parking and no entry using thesystem designed. It can be noted that all these traffic signs havered coloured boundaries. Hence the processing done involvesidentifying the regions in the frame having red colour. For thisthe frame now in HSV space is thresholded by specifying theproper range of red colour pixels in HSV space. This rangetakes into account the variations produced by various lightingand environmental conditions. The thresholded image will thenhave white in the regions of red colour in the frame and blackcolour in other regions.The thresholded image is then subjected to contour detectionto detect contours in the thresholded image. After detectingcontours, those contours having area less than a minimumspecified area are neglected. This is to reduce the numberof contours to be considered for recognition. After this theremaining contours are filled with white pixels. This is thengiven as input to shape detection process which detects con-tours having triangular or circular shape. These two shapes areconsidered since our traffic signs of interest have only thesetwo shapes. Those regions in the image having this shape are

8then identified from the original input frame and cropped out.These are then reshaped properly and given as input to theArtificial Neural Network for recognition.Following figure shows the results of detection of traffic signusing the above mentioned techniques. Here the figures in the

clockwise order are: 1. Input image, 2. HSV image, 3.Detectedcontours, 4. Thresholded image. The last figure shows thedetected traffic sign.

C. Traffic Sign Recognition

Most of the work done during the previous semester wasbased on MATLAB where we used a neural network to dosome basic tasks like digit recognition. This was done so as toget acquainted with the implementation of a neural networkat the basic level. Some image processing tasks which wererequired as preprocessing steps were also done. During thestart of the current semester the efforts were mainly onimplementing all the things that were done in MATLABwith the help of OpenCV. As such we implemented the digitrecognition using neural network with the help of OpenCV.In order to create the training dataset for the neural network,hundreds of images corresponding to various traffic signswere taken and they were converted into a matrix form insidewhich the corresponding pixel values were stored. The testset was also created like this. These values would then be fedto the neural network to train them. As a next step, a neuralnetwork was created and it was trained using the traffic dataset corresponding to three traffic signs. The number of inputlayers corresponds to the number of attributes of the datawhich is the number of pixels stored per each image. Theoutput layer corresponds to three outputs, the final valuepredicted was taken to be the maximum of the three valuesat the output.Once the neural network was trained, the weightscorresponding to each neuron in the hidden layer werethen stored inside an xml file. This would make it easier toload the neural network parameters faster in the future aswe dont need to train the network again. We then checked

if this was able to differentiate between the three trafficsigns. Initially we received an accuracy of 67.6%. Howeverafter trying with different number of hidden layers, this wasbrought up to around 92.3%. The corresponding number ofneurons in the hidden layer was 125.The results obtained are shown in the figure below. The

testing was done initially with images that were already cutout from the original images. Later on the image processingpart was also combined which would give the required regionof interest as input to the neural network while the neuralnetwork would then recognize the traffic sign. One of themajor problems corresponding to this is that it is that it isquite possible for the input to the system to be somethingsimilar to a traffic sign (a flux board for instance). This mayresult in erroneous output as the neural network will give anoutput corresponding to any of the three values no matterwhat the input is.One way to counter the above mentioned problem is totrain the neural network intentionally with such images (fluxboards and so on). Thus the system can then identify this asa deviation from the normal traffic signs and give output asrequired. However this is still in development stage.

D. Combined System

Traffic sign detection and traffic sign recognition stagescan be combined together to form a complete traffic signrecognition system. Results of the combined system for noentry traffic sign are shown below. Here the no entry traffic

signs have been detected from the frame. The region is croppedout and then given as input to the artificial neural network.The recognised traffic sign result can be seen in the command

9prompt window which in this case correctly recognises thetraffic sign.

E. Challenges

1) Real time processing: One of the most importantchallenges that needs to be addressed is the real timedata processing. The video needs to be continuouslymonitored and further processing needs to be carriedout fast so that the delay is minimal. This is veryimportant as with high speed vehicles there is achance that the traffic signs may become detectedafter some delay. This can render the system useless.It was with this intention that OpenCV was usedin the project. A future scenario where we can usebetter embedded support is also possible because ofthis. For example the software can be embedded intosome platform that needs only minimal processingpower but can do the required job at hand with elegance.

2) Dataset: The collection of possible traffic signs isnumerous. However not all the signs need to be givenequal importance. We need to select only some criticalsigns like School ahead, deep curve ahead; and soon and the system can neglect some signs like thoseinforming things like the location of nearest hotel andso on(This might be added into the system in the futurein order to enhance the intelligence of the system andto make the system work as an assistance for thingsother than driving also). Thus the photos correspondingto some important traffic sign were taken to be used.

3) Improving accuracy: In the image processing partthe major technique by which the region of interestis cropped was by using various attributes like thecolour of the region and also its shape.Howeverthis method has its own disadvantages as well. Forexample it is quite possible that the system mighterroneously detect flex boards that are similar totraffic signs. As already discussed this problem isbeing tackled by using some other training sets forthis kind of data sets so that the system can ignore them.

VII. CONCLUSION

As it can be seen, even though the system was able to detectand recognize a small set of traffic signs, there are still manyproblems that needs to be addressed like erroneous detection,lighting conditions and so on. There is also a chance for adecrease in accuracy with an increase in the number of trafficsign classes.

REFERENCES[1] Simon Haykin , Neural Networks: A comprehensive Foundation, 3rd

ed.Pearson Educaton, 2005.[2] A. de la Escalera, J.M Armingol, M. Mata, Traffic sign recognition and

analysis for intelligent vehicles, 2003.[3] Rafael.C.Gonzalez, Richard.E.Woods, Digital Image Processing, 3rd

ed.Pearson Education.

[4] Xiaoguang HU, Xinyan ZHU, Deren LI,Traffic Sign Detection Based OnBiologically Visual Mechanism, 2012

[5] David A Patterson, John L Hennessy, Computer Organisation and Design-The Hardware Software Interface,Elsevier, 2003

[6] The OpenCV Reference Manual Release 2.4.8.0