multimedia data mining an overview to image processing and machine learning by zaheer ahmad
TRANSCRIPT
8/7/2019 Multimedia Data Mining an Overview to Image Processing and Machine Learning by Zaheer Ahmad
http://slidepdf.com/reader/full/multimedia-data-mining-an-overview-to-image-processing-and-machine-learning 1/68
Multimedia Data Mining:An Overview to Image Processing
and
Machine Learning
Zaheer Ahmad
PhD Scholar
Department of Computer Science University of Peshawar
2/16/2011 1
8/7/2019 Multimedia Data Mining an Overview to Image Processing and Machine Learning by Zaheer Ahmad
http://slidepdf.com/reader/full/multimedia-data-mining-an-overview-to-image-processing-and-machine-learning 2/68
Agenda
• Multimedia Data Mining
• Image Data Mining and Image Processing
•
Machine Learning• Learning Techniques and tools
• Neural Networks and its types
• Training (Learning) of Neural Network
2/16/2011 2
8/7/2019 Multimedia Data Mining an Overview to Image Processing and Machine Learning by Zaheer Ahmad
http://slidepdf.com/reader/full/multimedia-data-mining-an-overview-to-image-processing-and-machine-learning 3/68
Multimedia Data mining
• Multimedia Data Mining is an interdisciplinaryand multidisciplinary field, used tointelligently retrieve and search multimedia
contents.
• A variety of techniques, from machine
learning, statistics, databases, knowledgeacquisition, data visualization, image analysis,high performance computing, and knowledge-based systems are used in MMM
2/16/2011 3
8/7/2019 Multimedia Data Mining an Overview to Image Processing and Machine Learning by Zaheer Ahmad
http://slidepdf.com/reader/full/multimedia-data-mining-an-overview-to-image-processing-and-machine-learning 4/68
2/16/2011 4
8/7/2019 Multimedia Data Mining an Overview to Image Processing and Machine Learning by Zaheer Ahmad
http://slidepdf.com/reader/full/multimedia-data-mining-an-overview-to-image-processing-and-machine-learning 5/68
MACHINE LEARNING
2/16/2011 5
8/7/2019 Multimedia Data Mining an Overview to Image Processing and Machine Learning by Zaheer Ahmad
http://slidepdf.com/reader/full/multimedia-data-mining-an-overview-to-image-processing-and-machine-learning 6/68
Data for MMM
2/16/2011 6
Data a database ?
• No ----- mostly
• Web Image, Audio, Video
• Live Streaming• Geo Sensors data
• But yes….
•
video database• Image or audio database
8/7/2019 Multimedia Data Mining an Overview to Image Processing and Machine Learning by Zaheer Ahmad
http://slidepdf.com/reader/full/multimedia-data-mining-an-overview-to-image-processing-and-machine-learning 7/68
• The word multimedia refers to a combination
of multiple media types together
• Multimedia Data Type
– Any Type of information medium that can be
represented, processed, stored and transmitted
over network in digital form
– Multi-lingual text, numeric, images, videos, audio,graphical, temporal, relational and categorical
data
2/16/2011 7
8/7/2019 Multimedia Data Mining an Overview to Image Processing and Machine Learning by Zaheer Ahmad
http://slidepdf.com/reader/full/multimedia-data-mining-an-overview-to-image-processing-and-machine-learning 8/68
Definition
• MMM is a subfield of data mining that deals
with an extraction of implicit knowledge,
multimedia data relashionships, or other
patterns not explicitly stored in multimedia
databases
– Used for multimedia information system and
retrieval of content based image/audio/video andprovide search and efficient storage organization
2/16/2011 8
8/7/2019 Multimedia Data Mining an Overview to Image Processing and Machine Learning by Zaheer Ahmad
http://slidepdf.com/reader/full/multimedia-data-mining-an-overview-to-image-processing-and-machine-learning 9/68
Media Types
• 0-dimensional data: This type of the data is the regular,
alphanumeric data. A typical example is the text data.
• 1-dimensional data: This type of the data has one dimension
of a space imposed into them. A typical example of this type
of the data is the audio data
• 2-dimensional data: This type of the data has two dimensions
of a space imposed into them. Imagery data and graphics data
are the two common examples of this type of data
• 3-dimensional data: This type of the data has three
dimensions of a space imposed into them. Video data and
animation data are the two common examples of this type of
data
2/16/2011 9
8/7/2019 Multimedia Data Mining an Overview to Image Processing and Machine Learning by Zaheer Ahmad
http://slidepdf.com/reader/full/multimedia-data-mining-an-overview-to-image-processing-and-machine-learning 10/68
Multimeimedia Data
• Spatial Data – Generalize detailed geographic points into clusterd
regions, such as business, residential, industrial, oragricultural areas, according to land usage
•
Image Data – Size, color, shape, texture, orientation, and relative
postions and structure of the contained objects or regionsin the image
• Music data – Summarize its melody: based on the approximate pattern
that repeateldly occure in the segment
– Summarized its type: based on its tone, tempo, or themajor musical insturment played
2/16/2011 10
8/7/2019 Multimedia Data Mining an Overview to Image Processing and Machine Learning by Zaheer Ahmad
http://slidepdf.com/reader/full/multimedia-data-mining-an-overview-to-image-processing-and-machine-learning 11/68
How Multimedia Data Mining System
Works
2/16/2011 11
8/7/2019 Multimedia Data Mining an Overview to Image Processing and Machine Learning by Zaheer Ahmad
http://slidepdf.com/reader/full/multimedia-data-mining-an-overview-to-image-processing-and-machine-learning 12/68
Similarity Search in Multimedia data
• Description based retrieval systems
– Build indices and perform object retrieval based onimage descriptions, such as keywords, captions, sizeand time of creation
– Labor-intensive if performed manually
– Results are typically of poor quality if automated
• Content Based Retrieval Systems
• Support retrieval based on the image content,such as color, histogram, texture, shape, objectsand wavelet transforms
2/16/2011 12
8/7/2019 Multimedia Data Mining an Overview to Image Processing and Machine Learning by Zaheer Ahmad
http://slidepdf.com/reader/full/multimedia-data-mining-an-overview-to-image-processing-and-machine-learning 13/68
Multidimensional Analysis of
Multimedia Data• Multimedia data Cube
– Design and construct similar to that traditional data cubes fromrelational data
– Contain additional dimensions and measures for multimediainformation such as color, texture, and shape
• The database doesn’t store images but their descriptors – Feature Descriptor: a set of vectors for each visual
characteristics• Color Vector: contains the color histogram
• MFC(Most Frequent Color) Vector: Five color centroids
• MFO(Most Frequent Orientation) Vector: Five edge orientationcentroid
– Layout Descriptor: Contains a color layout vector and an edgelayout vector
2/16/2011 13
8/7/2019 Multimedia Data Mining an Overview to Image Processing and Machine Learning by Zaheer Ahmad
http://slidepdf.com/reader/full/multimedia-data-mining-an-overview-to-image-processing-and-machine-learning 14/68
Typical Architecture of MMM
2/16/2011 14
8/7/2019 Multimedia Data Mining an Overview to Image Processing and Machine Learning by Zaheer Ahmad
http://slidepdf.com/reader/full/multimedia-data-mining-an-overview-to-image-processing-and-machine-learning 15/68
Image Data Mining
Image and Machine Learning
2/16/2011 15
8/7/2019 Multimedia Data Mining an Overview to Image Processing and Machine Learning by Zaheer Ahmad
http://slidepdf.com/reader/full/multimedia-data-mining-an-overview-to-image-processing-and-machine-learning 16/68
What is an image?
• An image is a two dimensional
function, f(x,y), where x and y are
spatial coordinates, and the
amplitude of f at any pair of coordinates (x,y) is called the intensity
or grey level of the image at that
point.
8/7/2019 Multimedia Data Mining an Overview to Image Processing and Machine Learning by Zaheer Ahmad
http://slidepdf.com/reader/full/multimedia-data-mining-an-overview-to-image-processing-and-machine-learning 17/68
17
Image Processing Stages
Image Acquisition
Image Processing
Image Segmentation
Image Analysis
Pattern Recognition
Analog to digital conversion
Remove noise,
improve contrast …
Find regions (objects)
in the image
Take measurements of
objects/relationships
Match the description with
similar description of known
objects (models)
8/7/2019 Multimedia Data Mining an Overview to Image Processing and Machine Learning by Zaheer Ahmad
http://slidepdf.com/reader/full/multimedia-data-mining-an-overview-to-image-processing-and-machine-learning 18/68
19
Image Analysis
Input Image
Regions, objectsMeasurements
ImageAnalysis
Measurements:-Size-Position
-Orientation-Spatial relationship-Gray scale or color intensity
8/7/2019 Multimedia Data Mining an Overview to Image Processing and Machine Learning by Zaheer Ahmad
http://slidepdf.com/reader/full/multimedia-data-mining-an-overview-to-image-processing-and-machine-learning 19/68
Image segmentation
The operation of distinguishing important objects from the
background (or from unimportant objects) based on different
feature of the image
Dark objects, bright background
Area B Area A
8/7/2019 Multimedia Data Mining an Overview to Image Processing and Machine Learning by Zaheer Ahmad
http://slidepdf.com/reader/full/multimedia-data-mining-an-overview-to-image-processing-and-machine-learning 20/68
21
Image Segmentation
Input ImageRegions
Objects
Segmentation
-Clasify pixels into groups having similar characteristics
-Two techniques: Region segmentation—Color/smoothness
Edge detection
8/7/2019 Multimedia Data Mining an Overview to Image Processing and Machine Learning by Zaheer Ahmad
http://slidepdf.com/reader/full/multimedia-data-mining-an-overview-to-image-processing-and-machine-learning 21/68
8/7/2019 Multimedia Data Mining an Overview to Image Processing and Machine Learning by Zaheer Ahmad
http://slidepdf.com/reader/full/multimedia-data-mining-an-overview-to-image-processing-and-machine-learning 22/68
Histogram
The data contained in a digitalimage can be displayed as a
histogram which is a plot of the
pixel values ranging from black
to white versus the number of pixels that have that particular
value.
d h h d f
8/7/2019 Multimedia Data Mining an Overview to Image Processing and Machine Learning by Zaheer Ahmad
http://slidepdf.com/reader/full/multimedia-data-mining-an-overview-to-image-processing-and-machine-learning 23/68
Edge through Gradient Information
Edge Location
Edge Direction i
),( ii y x
Neighborhood pixels
Sharpness Change / Contrast change
8/7/2019 Multimedia Data Mining an Overview to Image Processing and Machine Learning by Zaheer Ahmad
http://slidepdf.com/reader/full/multimedia-data-mining-an-overview-to-image-processing-and-machine-learning 24/68
25
Pattern Recognition (PR)
- Measurements- Stuctural
descriptionsClass identifier
PatternRecognition
feature vector
set of information data
8/7/2019 Multimedia Data Mining an Overview to Image Processing and Machine Learning by Zaheer Ahmad
http://slidepdf.com/reader/full/multimedia-data-mining-an-overview-to-image-processing-and-machine-learning 25/68
Content Based Image Retrieval
26
8/7/2019 Multimedia Data Mining an Overview to Image Processing and Machine Learning by Zaheer Ahmad
http://slidepdf.com/reader/full/multimedia-data-mining-an-overview-to-image-processing-and-machine-learning 26/68
27
Fingerprint recognition system
Fingerprintsensor
Fingerprintsensor
Feature Extractor
Feature Extractor
Feature Matcher
ID
Enrollment
Identification
Templatedatabase
8/7/2019 Multimedia Data Mining an Overview to Image Processing and Machine Learning by Zaheer Ahmad
http://slidepdf.com/reader/full/multimedia-data-mining-an-overview-to-image-processing-and-machine-learning 27/68
Machine Learning
A computer program is said to learn from
experience ‘E’ with respect to some class of
tasks ‘T’ and performance measure ‘P’,
If its
performance at tasks in T, as measured by P,
improves with experience E.
Mitchell (1997):
2/16/2011 28
8/7/2019 Multimedia Data Mining an Overview to Image Processing and Machine Learning by Zaheer Ahmad
http://slidepdf.com/reader/full/multimedia-data-mining-an-overview-to-image-processing-and-machine-learning 28/68
Machine Learning
Things learn when they change their behavior in
a way that makes them perform better in the
future.
From Witten and Frank (2000)
2/16/2011 29
8/7/2019 Multimedia Data Mining an Overview to Image Processing and Machine Learning by Zaheer Ahmad
http://slidepdf.com/reader/full/multimedia-data-mining-an-overview-to-image-processing-and-machine-learning 29/68
Machine Learning
• ML is a scientific discipline that is concerned
with the design and development of algorithms
that allow computers to evolve behaviors based
on empirical data, such as from sensor data or
databases.• A major focus of machine learning research is to
automatically learn to recognize complex
patterns and make intelligent decisions based
on data.
2/16/2011 30
8/7/2019 Multimedia Data Mining an Overview to Image Processing and Machine Learning by Zaheer Ahmad
http://slidepdf.com/reader/full/multimedia-data-mining-an-overview-to-image-processing-and-machine-learning 30/68
• the difficulty lies in the fact that the set of all
possible behaviors given all possible inputs is
too large to be covered by the set of observed
examples (training data).
• Hence the learner must generalize from the
given examples, so as to be able to produce a
useful output in new cases
2/16/2011 31
8/7/2019 Multimedia Data Mining an Overview to Image Processing and Machine Learning by Zaheer Ahmad
http://slidepdf.com/reader/full/multimedia-data-mining-an-overview-to-image-processing-and-machine-learning 31/68
Types of Learning
2/16/2011 32
• Supervised Learning
Learning a mapping between an input x and
a desired output y
• Unsupervised LearningUnderstanding the relationships between
data components
• Reinforcement Learning
Learning to act in the environment based on
the delayed rewards
Cl f L i
8/7/2019 Multimedia Data Mining an Overview to Image Processing and Machine Learning by Zaheer Ahmad
http://slidepdf.com/reader/full/multimedia-data-mining-an-overview-to-image-processing-and-machine-learning 32/68
Classes of Learning
Machine learning is not only about classification.
Classification learning: learn to put instances intopre-defined classes-----competitive network:
selects one unit in the output layer (target class)---
(Supervised Learning)
Association learning: learn relationships between theAttributes------ new response becomes associated
with a particular stimulus ---pattern associator:
recalls input patterns based on similarity
Clustering: discover classes of instances that belong
Together------- (Unsupervised)self-organizing map
(SOMs)
2/16/2011 33
8/7/2019 Multimedia Data Mining an Overview to Image Processing and Machine Learning by Zaheer Ahmad
http://slidepdf.com/reader/full/multimedia-data-mining-an-overview-to-image-processing-and-machine-learning 33/68
Learning Tools and Techniques
inShort
2/16/2011 34
8/7/2019 Multimedia Data Mining an Overview to Image Processing and Machine Learning by Zaheer Ahmad
http://slidepdf.com/reader/full/multimedia-data-mining-an-overview-to-image-processing-and-machine-learning 34/68
Learning Rules
• if outlook = sunny and humidity = high then play= no
• if outlook = rainy and windy = true then play = no
• if outlook = overcast then play = yes• if humidity = normal then play = yes
• if none of the above then play = yes
BEST But LABOURUS , HARD TO CODE AND COVERin Large Domains
2/16/2011 35
8/7/2019 Multimedia Data Mining an Overview to Image Processing and Machine Learning by Zaheer Ahmad
http://slidepdf.com/reader/full/multimedia-data-mining-an-overview-to-image-processing-and-machine-learning 35/68
Learning Decision Trees
• Example: XOR (familiar from connectionist
networks).
Nodes represent decisions on attributes, leaves represent classifications.
Some how like Learning Rules
2/16/2011 36
8/7/2019 Multimedia Data Mining an Overview to Image Processing and Machine Learning by Zaheer Ahmad
http://slidepdf.com/reader/full/multimedia-data-mining-an-overview-to-image-processing-and-machine-learning 36/68
Principal component analysis
• PCA is applied as a data reduction or structuredetection method
• combining two correlated variables into onefactor
• PCA defined as an orthogonal lineartransformation that transforms the data to a newcoordinate system such that the greatest varianceby any projection of the data comes to lie on the
first coordinate (called the first principalcomponent), the second greatest variance on thesecond coordinate
2/16/2011 37
8/7/2019 Multimedia Data Mining an Overview to Image Processing and Machine Learning by Zaheer Ahmad
http://slidepdf.com/reader/full/multimedia-data-mining-an-overview-to-image-processing-and-machine-learning 37/68
Support Vector Machine
• Support Vector Machine is a classifier derivedfrom statistical learning theory by VladimirVapnik and his co-workers
• Used for large data set• Good for text classification
• Work as multilayer perceptron
2/16/2011 38
8/7/2019 Multimedia Data Mining an Overview to Image Processing and Machine Learning by Zaheer Ahmad
http://slidepdf.com/reader/full/multimedia-data-mining-an-overview-to-image-processing-and-machine-learning 38/68
Hidden Markov Model
2/16/2011 39
8/7/2019 Multimedia Data Mining an Overview to Image Processing and Machine Learning by Zaheer Ahmad
http://slidepdf.com/reader/full/multimedia-data-mining-an-overview-to-image-processing-and-machine-learning 39/68
Genetic Algorithms
2/16/2011 40
8/7/2019 Multimedia Data Mining an Overview to Image Processing and Machine Learning by Zaheer Ahmad
http://slidepdf.com/reader/full/multimedia-data-mining-an-overview-to-image-processing-and-machine-learning 40/68
Neural Networks
41
8/7/2019 Multimedia Data Mining an Overview to Image Processing and Machine Learning by Zaheer Ahmad
http://slidepdf.com/reader/full/multimedia-data-mining-an-overview-to-image-processing-and-machine-learning 41/68
Inputs Outputs
Connection between cells
NN A Brain-Inspired Model
in
out
42
8/7/2019 Multimedia Data Mining an Overview to Image Processing and Machine Learning by Zaheer Ahmad
http://slidepdf.com/reader/full/multimedia-data-mining-an-overview-to-image-processing-and-machine-learning 42/68
Physical Structure of biological
neuron
2/16/2011 43
• Nerve cells are main processing element in our
central nervous system.
• Humans generally have about 100 billion nerve
cells in the entire nervous system.• Axon and dandroid are signal carrier away and
toward cell body respectively
• Synapse is the point at which the axon of one cell
interconnects with a dendrite of another cell• A basic nerve cell is thought as a black box
8/7/2019 Multimedia Data Mining an Overview to Image Processing and Machine Learning by Zaheer Ahmad
http://slidepdf.com/reader/full/multimedia-data-mining-an-overview-to-image-processing-and-machine-learning 43/68
NN A Brain-Inspired Model
• A neural network acquires knowledge through
learning.
• A neural network's knowledge is stored within
inter-neuron connection strengths known as
synaptic weights.
• The largest modern neural networks
achieve the complexity comparable to anervous system of a fly.
44
8/7/2019 Multimedia Data Mining an Overview to Image Processing and Machine Learning by Zaheer Ahmad
http://slidepdf.com/reader/full/multimedia-data-mining-an-overview-to-image-processing-and-machine-learning 44/68
Historical Background
• 1943 McCulloch and Pitts proposed the firstcomputational models of neuron.
• 1949 Hebb proposed the first learning rule.
• 1958 Rosenblatt’s work in perceptrons.• 1969 Minsky and Papert’s exposed limitation of the
theory.
• 1970s Decade of dormancy for neural networks.
• 1980-90s Neural network return (self-organization,back-propagation algorithms, etc)
45
8/7/2019 Multimedia Data Mining an Overview to Image Processing and Machine Learning by Zaheer Ahmad
http://slidepdf.com/reader/full/multimedia-data-mining-an-overview-to-image-processing-and-machine-learning 45/68
NN Applications
• Process Modeling and Control- Creating a neural network model for a physical
plant then using that model to determine the best control settings for the plant.
• Machine Diagnosis- Detect when a machine has failed so that the system canautomatically shut down the machine when this occurs.
• Target Recognition- Military application which uses video and/or infrared image data todetermine if an enemy target is present.
• Medical Diagnosis- Assisting doctors with their diagnosis by analyzing the reportedsymptoms and/or image data such as MRIs or X-rays.
• Target Marketing- Finding the set of demographics which have the highest responserate for a particular marketing campaign.
• Voice Recogntion- Transcribing spoken words into ASCII text.
• Financial Forecasting( Stock predication) - Using the historical data of a security to
predict the future movement of that security.• Quality Control - Attaching a camera or sensor to the end of a production process to
automatically inspect for defects.
• Intelligent Search - An internet search engine that provides the most relevant contentand banner ads based on the users' past behavior.
• Fraud Detection - Detect fraudulent credit card transactions and automatically decline
the charge. 46
8/7/2019 Multimedia Data Mining an Overview to Image Processing and Machine Learning by Zaheer Ahmad
http://slidepdf.com/reader/full/multimedia-data-mining-an-overview-to-image-processing-and-machine-learning 46/68
How NN Work ( Mathematically)
• Linear and Non Linear Pattern / Classification• Regression / Function Estimation
• Curve Fitting
Why to USE NN
• Parallel Processing
•
Fault tolerance• Self-organization
• Generalization ability
• Continuous adaptivity47
8/7/2019 Multimedia Data Mining an Overview to Image Processing and Machine Learning by Zaheer Ahmad
http://slidepdf.com/reader/full/multimedia-data-mining-an-overview-to-image-processing-and-machine-learning 47/68
48
Artificial Neurons
• Neural networks are made up of nodes which have – Input edges, each with some weight
– Output edges (with weights)
– An activation level (a function of the inputs)
• Weights of edges can be positive or negative and may change
over time (learning)
• The output function is the weighted sum of the activation levels
of inputs
• The activation level is a linear or non-linear transfer function “a”
of the input :
• Some nodes are inputs, some are outputs.
A ifi i l N l N k
8/7/2019 Multimedia Data Mining an Overview to Image Processing and Machine Learning by Zaheer Ahmad
http://slidepdf.com/reader/full/multimedia-data-mining-an-overview-to-image-processing-and-machine-learning 48/68
Artificial Neural Networks
Block Diagram
2/16/2011 49
A ifi i l N l N k
8/7/2019 Multimedia Data Mining an Overview to Image Processing and Machine Learning by Zaheer Ahmad
http://slidepdf.com/reader/full/multimedia-data-mining-an-overview-to-image-processing-and-machine-learning 49/68
Artificial Neural Networks
Process
2/16/2011 50
The Perceptron
8/7/2019 Multimedia Data Mining an Overview to Image Processing and Machine Learning by Zaheer Ahmad
http://slidepdf.com/reader/full/multimedia-data-mining-an-overview-to-image-processing-and-machine-learning 50/68
The Perceptron
51
x1
x2
xn
.
.
.
w1
w2
wn
wn+1
Biasxn+1=-1
a= bias+wi xi
y
1 if a 0y= 0 if a <0{
q=wn+1
•Bias , the extra weight connected to a constant is called the bias of
the element
• It enables to set the threshold equal to zero which help in
calculation•To get an extra dimension for representation This means
that every point in (n + 1)-dimensional weight space can be
associated with a hyperplane in (n + 1)-dimensional extended input
space.
8/7/2019 Multimedia Data Mining an Overview to Image Processing and Machine Learning by Zaheer Ahmad
http://slidepdf.com/reader/full/multimedia-data-mining-an-overview-to-image-processing-and-machine-learning 51/68
Logical Operations
2/16/2011 52
Threshold= 2
Threshold= 2
8/7/2019 Multimedia Data Mining an Overview to Image Processing and Machine Learning by Zaheer Ahmad
http://slidepdf.com/reader/full/multimedia-data-mining-an-overview-to-image-processing-and-machine-learning 52/68
2/16/2011 53
Threshold= 2
The first layer performs the two AND NOT's and the
second layer performs the OR. Both Z neurons and
the Y neuron have a threshold of 2
X1 XOR X2 = (X1 AND NOT X2) OR (X2 AND NOT
X1)
8/7/2019 Multimedia Data Mining an Overview to Image Processing and Machine Learning by Zaheer Ahmad
http://slidepdf.com/reader/full/multimedia-data-mining-an-overview-to-image-processing-and-machine-learning 53/68
Linear Separability Problem
• If two classes of patterns can be separated by a decision boundary,
represented by the linear equation
then they are said to be linearly separable. The simple network can
correctly classify any patterns.
• Decision boundary of linearly separable classes can be determinedeither by some learning procedures or by solving linear equation
systems based on representative patterns of each classes
• If such a decision boundary does not exist, then the two classes are
said to be linearly inseparable.• Linearly inseparable problems cannot be solved by the simple
network , more sophisticated architecture is needed.
01
n
i iiw x b
54
8/7/2019 Multimedia Data Mining an Overview to Image Processing and Machine Learning by Zaheer Ahmad
http://slidepdf.com/reader/full/multimedia-data-mining-an-overview-to-image-processing-and-machine-learning 54/68
• Examples of linearly separable classes
- Logical AND function
patterns (bipolar) decision boundary
x1 x2 y w1 = 1-1 -1 -1 w2 = 1-1 1 -1 b = -11 -1 -1 q = 01 1 1 -1 + x1 + x2 = 0
- Logical OR function
patterns (bipolar) decision boundary
x1 x2 y w1 = 1-1 -1 -1 w2 = 1-1 1 1 b = 11 -1 1 q = 01 1 1 1 + x1 + x2 = 0
x
oo
o
x: class I (y = 1)o: class II (y = -1)
x
xo
x
x: class I (y = 1)o: class II (y = -1)
55
Equation of Line ( Decision Boundary )
8/7/2019 Multimedia Data Mining an Overview to Image Processing and Machine Learning by Zaheer Ahmad
http://slidepdf.com/reader/full/multimedia-data-mining-an-overview-to-image-processing-and-machine-learning 55/68
• Examples of linearly inseparable classes
- Logical XOR (exclusive OR) function
patterns (bipolar) decision boundary
x1 x2 y-1 -1 -1-1 1 11 -1 11 1 -1
o
xo
x
x: class I (y = 1)o: class II (y = -1)
56
8/7/2019 Multimedia Data Mining an Overview to Image Processing and Machine Learning by Zaheer Ahmad
http://slidepdf.com/reader/full/multimedia-data-mining-an-overview-to-image-processing-and-machine-learning 56/68
Multilayer NN
• Neural Net for Nonlinear Classification
• Combination of Perceptron
• Back propagation learning
57
8/7/2019 Multimedia Data Mining an Overview to Image Processing and Machine Learning by Zaheer Ahmad
http://slidepdf.com/reader/full/multimedia-data-mining-an-overview-to-image-processing-and-machine-learning 57/68
What do each of the layers do?
1st layer draws
linear boundaries
2nd layer combines
the boundaries
3rd layer can generate
arbitrarily complex boundaries
Multilayer FFNN
A NN with one or more than one hidden layers
58
B k ti Al ith
8/7/2019 Multimedia Data Mining an Overview to Image Processing and Machine Learning by Zaheer Ahmad
http://slidepdf.com/reader/full/multimedia-data-mining-an-overview-to-image-processing-and-machine-learning 58/68
Back propagation Algorithm• Multiple outputs.
• Forward pass:• Error calculation:
• Backward propagation:
• No guarantee to in getting best possibleweights after correcting.
• Classifies inputs into multiple classes.
• Can be modified to represent any function.
8/7/2019 Multimedia Data Mining an Overview to Image Processing and Machine Learning by Zaheer Ahmad
http://slidepdf.com/reader/full/multimedia-data-mining-an-overview-to-image-processing-and-machine-learning 59/68
NN Training Data
• Training Set: this data set is used to adjust the weights on theneural network.
• Validation Set: this data set is used to minimize overfitting. – not adjusting the weights of the network with this data set,
– just verifying that any increase in accuracy over the training data set
actually yields an increase in accuracy over a data set that has notbeen shown to the network before, or at least the network hasn'ttrained on it (i.e. validation data set).
– If the accuracy over the training data set increases, but the accuracyover then validation data set stays the same or decreases,
– then you're overfitting your neural network and you should stop
training.• Testing Set: this data set is used only for testing the final solution in
order to confirm the actual predictive power of the network.
2/16/2011 60
N d A i i F i
8/7/2019 Multimedia Data Mining an Overview to Image Processing and Machine Learning by Zaheer Ahmad
http://slidepdf.com/reader/full/multimedia-data-mining-an-overview-to-image-processing-and-machine-learning 60/68
Neuron and Activation Functions
2/16/2011 61
A i i F i
8/7/2019 Multimedia Data Mining an Overview to Image Processing and Machine Learning by Zaheer Ahmad
http://slidepdf.com/reader/full/multimedia-data-mining-an-overview-to-image-processing-and-machine-learning 61/68
Activation Functions
2/16/2011 62
These functions can be defined as follows.
Stept(x) = 1 if x >= t, else 0Sign(x) = +1 if x >= 0, else -1
Sigmoid(x) = 1/(1+e-x)
Selection of Nodes for
8/7/2019 Multimedia Data Mining an Overview to Image Processing and Machine Learning by Zaheer Ahmad
http://slidepdf.com/reader/full/multimedia-data-mining-an-overview-to-image-processing-and-machine-learning 62/68
Selection of Nodes for
Neural Network
• Input Nodes----Image/data size
• Output node---output binary
• Middle Layer----o ooo oo….
– Keep middle layer smaller to Generalize and not
memorize
2/16/2011 63
Perceptron Learning Algorithm:
8/7/2019 Multimedia Data Mining an Overview to Image Processing and Machine Learning by Zaheer Ahmad
http://slidepdf.com/reader/full/multimedia-data-mining-an-overview-to-image-processing-and-machine-learning 63/68
64
Perceptron Learning Algorithm:
Initialise weights and threshold.
Set w i (t), (0 <= i <= n), to be the weight i at
time t , and ø to be the threshold value in the
output node. Set w 0
to be -ø, the bias, and x 0
to be always 1.
Set w i ( 0 ) to small random values, thus
initialising the weights and threshold.
Present input and desired outputPresent input x 0, x 1, x 2, ..., x n and desired
output d(t)
Calculate the actual output
y(t) = f h[w 0(t)x 0(t) + w 1(t)x 1(t) + .... + w n(t)x n(t)]
Adapts weights
w i (t+1 ) = w i (t) + ñ[d(t) - y(t)]x i (t) , where 0 <= ñ <= 1 is a positive gain function that controls
the adaption rate.
Steps iii. and iv. are repeated until the iteration
error is less than a user-specified error
threshold or a predetermined number of
iterations have been completed.
Perceptron Learning Algorithm:
start: The weight vector w0 is
generated randomly,set t := 0
test: A vector x 2 P [ N is selected
randomly,
if x 2 P and wt · x > 0 go to test,
if x 2 P and wt · x 0 go to add,
if x 2 N and wt · x < 0 go to test,
if x 2 N and wt · x 0 go to subtract.
add: set wt+1 = wt + x and t := t +
1, goto test
subtract: set wt+1 = wt − x and t :=
t + 1, goto test
Neural Networks
8/7/2019 Multimedia Data Mining an Overview to Image Processing and Machine Learning by Zaheer Ahmad
http://slidepdf.com/reader/full/multimedia-data-mining-an-overview-to-image-processing-and-machine-learning 64/68
65
Neural Networks –
Training
Backpropagation training cycle
8/7/2019 Multimedia Data Mining an Overview to Image Processing and Machine Learning by Zaheer Ahmad
http://slidepdf.com/reader/full/multimedia-data-mining-an-overview-to-image-processing-and-machine-learning 65/68
Urdu OCR Input Data Examplefeeded to FNN
66
8/7/2019 Multimedia Data Mining an Overview to Image Processing and Machine Learning by Zaheer Ahmad
http://slidepdf.com/reader/full/multimedia-data-mining-an-overview-to-image-processing-and-machine-learning 66/68
2/16/2011 67
8/7/2019 Multimedia Data Mining an Overview to Image Processing and Machine Learning by Zaheer Ahmad
http://slidepdf.com/reader/full/multimedia-data-mining-an-overview-to-image-processing-and-machine-learning 67/68
ouY
68
Thank
8/7/2019 Multimedia Data Mining an Overview to Image Processing and Machine Learning by Zaheer Ahmad
http://slidepdf.com/reader/full/multimedia-data-mining-an-overview-to-image-processing-and-machine-learning 68/68
References
• Data Mining and Knowledge Discovery Series, Chapman &Hall/CRC
• Neural Networks a Systematic Approach
• Matlab - development of neural network theory for artificial
life-thesis, matlab and java code
• Digital Image Processing By Gonzalez Using Matlab
• Wikiiiiiiiiiiiiiiiiiiiiiipedia
• And…….