CONTENT-BASED IMAGE
RETRIEVAL FOR AGRICULTURAL
CROPS
Esraa M. Elhariri, MScScientific Research Group n Egypt (SRGE)
www.egyptscience.net
Agenda
Introduction
Problem Definition
Scope of work & Proposed Solutions
Crops Ripeness
The Proposed Ripeness Classification System
Experimental Results
Conclusions & Future Work
Introduction
Product quality is one of the prime factors in
ensuring consistent marketing of crops.
Determining ripeness stages is a very
important issue in produce (fruits and
vegetables) industry for getting high quality.
Optimal harvest dates and prediction of
storage life are still mainly based on
subjective interpretation and practical
experience.
Introduction Cont…
Monitoring and controlling ripeness is a
very important issue in the fruit industry
since the state of ripeness during harvest,
storage and market distribution determines
the quality of the final product measured in
terms of customer satisfaction.
Problem Definition
Crops at greenhouse, storage containers are affected by
changes at temperature and humidity levels.
these changes can cause diseases, lost of nutritional value,
over-ripen and changes at texture, appearance and flavor and
this affect the quality of the agricultural crops.
Monitor the ripeness process of crops is very important issue
at agricultural and industry.
Problem Definition Cont…
So the human labors were required to visit the greenhouse and
check the humidity and temperature levels at specifics times
in according to plant type (e.g. strawberry need the
temperature level and humidity to be checked every 3 hours).
Doing this manually will be time consuming, cost high and
needs a lot of work and efforts.
Scope of work & Proposed
Solutions
This thesis concerns the study and development of different
approaches to help Labors via providing an automated
ripeness assessment system for agricultural crops.
Developing an automatic content based image retrieval
system for agriculture crops, which is tackling the problem
of ripeness assessment.
Having a comparative study among different classification
algorithms for the problem of ripeness classification.
Sweeter
Less Green
Softer
More palatable
Crops Ripeness
Choosing a crop
We choose Bell Pepper and Tomato for our research.
Different Ripeness Stages are described at the following
Charts.
The proposed classification
system: System Architecture
• Resizing images to 250x250 pixels for the followingreason:• Reducing their color index to avoid high
computation.
During this phase, the proposed approach prepares images
for the features extraction phase, The first step is :
The proposed classification
system: Pre-processing Phase
• Remove gray shadow pixels.
• Convert image from RGB to Gray
image.
• Subtract image from background
image and compute a binary
mask.
• Apply the binary mask to original
image.
Background Removal Algorithm
Original Images
Images after background Removal
The second step is removing
image background to get region
of interest by using the following
procedure:
The proposed classification
system: Pre-processing Phase
The third step is : converting each image from
RGB to HSV color space and extract HSV
components for the following reasons:
When applying classification using PCA and
Euclidean distance using both RGB and HSV color
features we got better results using HSV color.
From Survey, HSV color space as it is widely used
in the field of color vision and close to the
categories of human color perception
where R, G and B are colorcomponent of RGB colorspace
The proposed classification
system: Pre-processing Phase
The pre-processing phase can
be summarized as the
following:
• Resize images to 250 X 250
pixels,
• Remove images background,
• Convert images from RGB color
space to HSV color space,
•Extract HSV components.
The proposed classification
system: Pre-processing Phase
Since tomato surface color is the
most important characteristic to
observe ripeness of tomato, this
system uses color features for
classifying ripeness stages.
Two color descriptors which were
used here are color moments and
HSV colored histogram.
The proposed classification
system: Feature Extraction Phase
Standard Equation:
The first descriptor is color
moments: we used mean, standard
deviation, and skewness which have
been proved to be efficient and
effective way for representing color
distribution in any image.
Mean, standard deviation and
skewness for a colored image of
size NxM pixels are defined by the
following equations:
Mean Equation:
Skewness Equation:
The proposed classification
system: Feature Extraction Phase
Colored histogram is the second color descriptor that shows representation of the
distribution of colors in an image. It represents the number of pixels that have
colors in each range of colors.
We used 1D 16x4x4 HSV colored histogram, with 16 level for hue and 4 level
for each of saturation and value.
Then we compute a combination 1D Features vector of the three color moments
and HSV histogram.
The proposed classification
system: Feature Extraction Phase
Principal component analysis is a
statistical common technique, which is
widely used in image recognition and
compression for a dimensionality
reduction, data representation and
features extraction tool as it ensures
better classification.
we use Principal Component
Analysis(PCA) for features vector
Transformation to extract good features
for the classification purpose.
IM1 F1 F2 : : : : : : Fk
IM2 : : : : : : : : :
::: : : : : : : : : :
IMn F1 F2 Fk
IM1 F1 : : Fm
IM2 : : : :
::: : : : :
IMn F1 : : Fm
PCA Algorithm
Whole Features
Extracted Sub-set of Features
The proposed classification
system: Feature Extraction Phase
During this phase, the proposed
approach uses Support Vector
Machine(SVM) as a classifier, this
phase is responsible for classify data
into 5 classes mentioned before.
The proposed classification
system: Classification Phase
The Support Vector Machine (SVM) is a
Machine Learning (ML) algorithm that is
used for classification and regression of
high dimensional datasets with great
results. SVM solves the classification
problem via trying to find an optimal
separating hyperplane between classes.
SVM algorithm seeks to maximize the
margin around a hyperplane that separates
a positive class from a negative class.
The proposed classification
system: Classification Phase
Since SVM is a binary classifier, we have
used two different approaches to use it as
multi-class classifier:
One-against-All
One-against-One
Also we have tested our SVM classifier
using different kernel functions includes:
Linear kernel function,
Polynomial kernel function,
MLP kernel function,
RBF kernel function.
The proposed classification
system: Classification Phase
One against All:
1. Construct N binary SVM.
2. Each SVM separates one class
from the rest classes.
3. Train the ith SVM with all
training samples of the ith class
with positive labels, and
training samples of other classes
with negative labels.
The proposed classification
system: Classification Phase (One-against-All
Approach)
At the begin, we have used One-
against-All multi-class SVM with
different kernel functions and total
of 230 images used for the
classification problem, it was
divided into 175 images as a
training dataset and 55 images as
a testing dataset.
Experimental Results:First Approach using One-against-All Approach
Training dataset is divided into 5 classes representing the different
stages of tomato ripeness.
Experimental Results:Training Dataset
Classification accuracy
using different kernel
function and different
sizes of training
classes:
Experimental Results:Results of First Approach
Experimental Results:Results of First Approach
From the previous figure we
found that classification
accuracy using SVM with
linear kernel function
increased by increasing
number of training images per
class.
Experimental Results: Accuracy of
linear Kernel Function using Different Training Size
Publications List
• Implementation of one-against-all multi-calss SVM for tomato
• Publication of Multiclass SVM Based Classification Approach for Tomato Ripeness
Submission : Jul 2013
Publication :Aug 2013
• Implementation of one-against-all & one-against-one multi-calssSVM
• Publication of Automated Ripeness Assessment System of Tomatoes Using PCA and SVM Techniques(book chapter)
Submission : Nov 2013
Publication: June 2014
• Implementation of one-against-one multi-calss SVM for bell pepper
• Publication of Bell Pepper Ripeness Classification based on Support Vector Machine
Submission : Dec 2013
Presentation: Apr 2014
• Implementation of different machine learning techniques for tomato as an extension of the first paper and the book chapter.
• Publication of Using machine learning techniques for evaluating tomato ripeness
Submission : Feb 2014
Publication: Oct 2014
• Comparative Study of different classification techniques for both crops.
• Publication of Random Forests Based Classification for Crops Ripeness Stages
Submission: Apr 2014
Publication : June 2014
Esraa Elhariri, Nashwa El-Bendary, Mohamed Mostafa M. Fouad, Jan Plato’s,
Aboul Ella Hassanien, Ahmed M. M. Hussein, “Multiclass SVM Based
Classification Approach for Tomato Ripeness.” In Proceedings of the Fourth
International Conference on Innovations in Bio-inspired Computing and
Applications(IBICA 2013), 22 – 24 August, Ostrava, Czech Republic, pp.175-186 ,
2013, Springer International Publishing.
Then, we have used both of
One-against-All & One-against-
One with 10-fold cross-
validation to generalize model
accuracy, using total of 250
images for both of training and
testing.
This figure shows how cross-
validation works.
TrainingTesting
Experimental Results: Second
Approach using both of OAO and OAA with Cross-Validation
One-against-One:
1-train k(k − 1)/2 binary SVMs
(1, 2), (1, 3), . . . , (1, k), (2, 3), (2,
4), . . . , (k − 1, k),
2-Select the one with the largest
vote.
The proposed classification
system: Classification Phase (One-against-One
Approach)
Classification accuracy using different kernel function and total number of
250 image(used for both training and testing using 10 folds cross-
validation) using one against one approach for multi-class SVM
classification:
Experimental Results: One-against-
One(OAO) with 10 fold Cross-Validation Approach
Classification accuracy using different kernel function and total number of
250 image(used for both training and testing using 10 folds cross-
validation) using one against all approach for multi-class SVM
classification:
Experimental Results: One-against-
All(OAA) with 10 fold Cross-Validation Approach
The following figure show a comparison between the classification
accuracy of One-against-All(OAA) multi-class SVM and One-against-
One(OAO) multi-class SVM Approaches:
Experimental Results: Comparison
between of OAO & OAA Approaches
Publications List
• Implementation of one-against-all multi-calss SVM for tomato
• Publication of Multiclass SVM Based Classification Approach for Tomato Ripeness
Submission : Jul 2013
Publication :Aug 2013
• Implementation of one-against-all & one-against-one multi-calssSVM
• Publication of Automated Ripeness Assessment System of Tomatoes Using PCA and SVM Techniques(book chapter)
Submission : Nov 2013
Publication: June 2014
• Implementation of one-against-one multi-calss SVM for bell pepper
• Publication of Bell Pepper Ripeness Classification based on Support Vector Machine
Submission : Dec 2013
Presentation: Apr 2014
• Implementation of different machine learning techniques for tomato as an extension of the first paper and the book chapter.
• Publication of Using machine learning techniques for evaluating tomato ripeness
Submission : Feb 2014
Publication: Oct 2014
• Comparative Study of different classification techniques for both crops.
• Publication of Random Forests Based Classification for Crops Ripeness Stages
Submission: Apr 2014
Publication : June 2014
Esraa Elhariri, Nashwa El-Bendary, Aboul Ella Hassanien, Amr Badr:
"Automated Ripeness Assessment System of Tomatoes Using PCA and SVM
Techniques." Computer Vision and Image Processing in Intelligent Systems and
Multimedia Technologies. IGI Global, 2014. 101-130. Web. 9 Oct. 2014.
doi:10.4018/978-1-4666-6030-4.ch006.
Bell Pepper Ripeness
Assessment
Then we have worked on bell pepper:
we have used One-against-One
multi-class SVM with different
kernel functions and total of 175
images used for the classification
problem, they were used for both
training and testing dataset using
10 fold Cross-validation.
Experimental Results: One-against-One Approach
Training dataset is divided into 5 classes
representing the different stages of Bell Pepper
ripeness.
Experimental Results:Training Dataset
Experimental Results: One-against-
One (OAO) with 10 Cross-validation Approach
Classification accuracy using
different kernel function and
10-folds cross-validation.
Publications List
• Implementation of one-against-all multi-calss SVM for tomato
• Publication of Multiclass SVM Based Classification Approach for Tomato Ripeness
Submission : Jul 2013
Publication :Aug 2013
• Implementation of one-against-all & one-against-one multi-calssSVM
• Publication of Automated Ripeness Assessment System of Tomatoes Using PCA and SVM Techniques(book chapter)
Submission : Nov 2013
Publication: June 2014
• Implementation of one-against-one multi-calss SVM for bell pepper
• Publication of Bell Pepper Ripeness Classification based on Support Vector Machine
Submission : Dec 2013
Presentation: Apr 2014
• Implementation of different machine learning techniques for tomato as an extension of the first paper and the book chapter.
• Publication of Using machine learning techniques for evaluating tomato ripeness
Submission : Feb 2014
Publication: Oct 2014
• Comparative Study of different classification techniques for both crops.
• Publication of Random Forests Based Classification for Crops Ripeness Stages
Submission: Apr 2014
Publication : June 2014
Esraa Elhariri, Nashwa El-Bendary, Ahmed M. M. Hussein, Aboul Ella
Hassanien, and Amr Badr, “Bell Pepper Ripeness Classification based on Support
Vector Machine”, In Proceeding of IEEE ICET, 19 – 21 April, Cairo, Egypt.
The proposed classification
system: System Architecture (Comparison
between SVM and LDA)
Then, we have used both of
One-against-All & One-against-
One, In addition to, LDA with
10-fold cross-validation to
generalize model accuracy,
using total of 250 images for
both of training and testing.
This figure shows how cross-
validation works.
TrainingTesting
Experimental Results: Second Approach
using both of OAO, OAA and LDA with Cross-Validation
Linear Discriminant Analysis
(LDA)
LDA is a commonly used
technique for data classification
and dimensionality reduction.
It’s basic idea is to find a linear
transformation that best
discriminate among classes, then
classification can be performed in
transformed space based on some
metrics(Euclidean distance)
Classification accuracy using different kernel function and total number of 250
image(used for both training and testing using 10 folds cross-validation) using one
against one approach for multi-class SVM classification:
Experimental Results: One-against-
One (OAO) with 10 Cross-validation Approach
Classification accuracy using different kernel function and total number of 250
image(used for both training and testing using 10 folds cross-validation) using one
against all approach for multi-class SVM classification:
Experimental Results: One-against-All
(OAA) with 10 Cross-validation Approach
Classification accuracy using total number of 250 image(used for both training
and testing using 10 folds cross-validation) using LDA classification:
Accuracy= 84 %
Experimental Results LDA with 10
Cross-validation Approach
The following figure show a comparison between the
classification accuracy of One-against-All(OAA) multi-class
SVM and One-against-One(OAO) multi-class SVM and LDA
Approaches:
Experimental Results: Comparison
between of OAO & OAA and LDA Approaches
Publications List
• Implementation of one-against-all multi-calss SVM for tomato
• Publication of Multiclass SVM Based Classification Approach for Tomato Ripeness
Submission : Jul 2013
Publication :Aug 2013
• Implementation of one-against-all & one-against-one multi-calssSVM
• Publication of Automated Ripeness Assessment System of Tomatoes Using PCA and SVM Techniques(book chapter)
Submission : Nov 2013
Publication: June 2014
• Implementation of one-against-one multi-calss SVM for bell pepper
• Publication of Bell Pepper Ripeness Classification based on Support Vector Machine
Submission : Dec 2013
Presentation: Apr 2014
• Implementation of different machine learning techniques fortomato as an extension of the first paper and the book chapter.
• Publication of Using machine learning techniques for evaluatingtomato ripeness
Submission : Feb 2014
Publication: Oct 2014
• Comparative Study of different classification techniques for both crops.
• Publication of Random Forests Based Classification for Crops Ripeness Stages
Submission: Apr 2014
Publication : June 2014
Nashwa El-Bendary, Esraa El Hariri, Aboul Ella Hassanien, Amr Badr, “Using
machine learning techniques for evaluating tomato ripeness”, In International
Journal of Expert Systems with Applications, Volume 42, Issue 4, March 2015,
Pages 1892-1905, Elsevier, ISSN 0957-4174,
http://dx.doi.org/10.1016/j.eswa.2014.09.057.
The proposed classification
system: System Architecture
Random
ForestsOAO-SVM
Experimental Results: One-against-One Approach
we have used One-against-One multi-
class SVM and Random Forests with
different kernel functions and total of 175
and 250 images for bell pepper and
tomato, respectively used for the
classification problem, they were used
for both training and testing dataset.
The proposed classification
system: Classification Phase
The Random Forests (RF) is one of the best
known classification and regression
techniques, which has the ability to classify
large dataset with excellent accuracy.
Random Forests algorithm generates an
ensemble of decision trees. Ensemble
methods main principle is to group weak
learners together to build a strong learner.
The proposed classification
system: Classification Phase
The input is entered at the top and as it traverses
down the tree, the original data is sampled in
random, but with replacement into smaller and
smaller sets.
The class of sample is determined using random
forests trees, which are of an arbitrary number.
Experimental Results: Training Dataset
For Bell Pepper, Training dataset is divided into 5 classes
representing the different stages of Bell Pepper ripeness.
Experimental Results: Training Dataset
For Tomato, Training dataset is divided into 5 classes
representing the different stages of Tomato ripeness.
Experimental Results: One-against-One
(OAO) & RF with 10 Cross-validation Approaches
Bell Pepper Ripeness
Classification accuracy using
Random Forest and SVM
using different kernel function
and 10-folds cross-validation.
Experimental Results: One-against-One
(OAO) & RF with 10 Cross-validation Approaches
Tomato Ripeness Classification
accuracy using Random Forest
and SVM using different kernel
function and 10-folds cross-
validation.
Publications List
• Implementation of one-against-all multi-calss SVM for tomato
• Publication of Multiclass SVM Based Classification Approach for Tomato Ripeness
Submission : Jul 2013
Publication :Aug 2013
• Implementation of one-against-all & one-against-one multi-calssSVM
• Publication of Automated Ripeness Assessment System of Tomatoes Using PCA and SVM Techniques(book chapter)
Submission : Nov 2013
Publication: June 2014
• Implementation of one-against-one multi-calss SVM for bell pepper
• Publication of Bell Pepper Ripeness Classification based on Support Vector Machine
Submission : Dec 2013
Presentation: Apr 2014
• Implementation of different machine learning techniques for tomato as an extension of the first paper and the book chapter.
• Publication of Using machine learning techniques for evaluating tomato ripeness
Submission : Feb 2014
Publication: Oct 2014
• Comparative Study of different classification techniques for both crops.
• Publication of Random Forests Based Classification for Crops Ripeness Stages
Submission: Apr 2014
Publication : June 2014
Esraa Elhariri, Nashwa El-Bendary, Aboul Ella Hassanien, Amr Badr, Ahmed M.
M. Hussein, Václav Snásel, “Random Forests Based Classification for Crops
Ripeness Stages”, In Proceedings of the Fifth International Conference on
Innovations in Bio-Inspired Computing and Applications (IBICA 2014), vol. 303,
23 – 25 June, Ostrava, Czech Republic, pp.205-215, 2014, Springer International
Publishing.
Conclusions & Future Work
In this Thesis, a system for classifying the ripeness stages ofboth tomato and bell pepper has been developed.
For tomato ripeness assessment, the ripeness classificationaccuracy obtained by the OAO multi-class SVM approach isbetter than ripeness classification accuracy obtained by theOAA multi-class SVM and LDA approaches.
For bell pepper ripeness assessment, the ripeness classificationaccuracy obtained by the OAO multi-class SVM approach isbetter than ripeness classification accuracy obtained by theOAA multi-class SVM approach.
Conclusions & Future Work
A number of future research could be achieved via classifyingdifferent objects or crops by involving other features (texture,shape, size, ... etc.) according to the classified objects nature.
Other Machine Learning approaches could be employed inorder to address the advantages and limitations of applyingeach of them.
Another direction of research is to use nondestructive/non-invasive detection technologies of food quality/maturity suchas hyperspectral imaging systems, colorimetric, Near InfraredSpectroscopy, and non-invasive smart sensing technologies
For further questions:Esraa M. El-hariri