Detection of Oranges In Outdoor Conditions

Yüksel Çakır, Mürvet Kırcı, Ece Olcay Güneş

Department of Electronics and Communication Engineering

ITU Faculty of Electrical and Electronics Engineering

Maslak, Istanbul, Turkey

cakiryu@itu.edu.tr, ucerm@itu.edu.tr, ece.gunes@itu.edu.tr

Burak Berk Üstündağ

Department of Computer Engineering

ITU Faculty of Computer and Informatics

Maslak, Istanbul, Turkey

berk@berk.tc

Abstract— In this study, a method is proposed to detect

orange fruits on tree using image processing techniques in order

to develop software for an auto-orange collector robot system.

Fruit detection based on computer vision includes some

problems. The main problem is variable lighting conditions for

the environment in outdoor. The additional problem is the

occlusion of fruits by leaves, branches, and the other fruits. In the

proposed method these problems are also examined.

Keywords—orange detection; image processing; color and

shape based methods

I. INTRODUCTION

In recent years, many researchers have studied on the

design of auto-farming systems (automatic fruit/vegetable

harvesting systems, automatic fertilization systems, automatic

irrigation-spraying systems, and etc.). Auto-farming systems

composed of many different parts and components, so experts

from different fields have to work together for developing

such a system. Furthermore, the availability of advanced

sensors and robotic technology, coupled with the increasing

computational capability of computers has facilitated the

development of the auto-farming systems. In literature

especially studies on automatic fruit/vegetable harvesting

systems are considerably attention [1-7]. For automatic

fruit/vegetable harvesting systems, it is extremely important to

effectively detect the object in outdoor conditions. These

systems have begun to include computer vision systems. In

addition computer vision technology can easily be adapted to

other applications such as on tree yield monitoring, crop

health status monitoring, disease detection, phenological phase

detection and other operations which require vision sensor

output.

Last decades researchers have applied many different

computer vision techniques for the detection of fruit [8-10].

Studies on the detection of different fruits and vegetables such

as apple [10-15], cucumber [16], orange [17-18], pomegranate

[19] and etc. have been reported. There are several problems

on fruit detection in outdoor condition that the previous

researchers did not sufficiently solve, which can be classified

into two groups: lighting and occlusion. Lighting is a

significant problem. In outdoor condition the main

contributing factor to the lighting of the scene is sunlight. The

amount of illumination depends on cloud cover and the solar

angle. This can cause significant differences on scene and

non-uniform illumination on the fruit affect the rate of the

vision based fruit detection. The other problem is the

occlusion of fruits by leaves, branches, and the other fruits,

which minimizes the fruit area visibility and disrupts the shape

of the fruit and this greatly affects the detection of fruits on

tree.

Overcome these problems is very crucial for the success of robotic harvesting. The researches on detection of fruit using images can be handled on two categories: Local and shape analysis. Local analysis depends on intensity and color information on the desired objects. Shape analysis depends on fitting of circles and ellipses. In this study a computer vision based method is proposed for detect visible orange fruits on tree.

II. DETECTION OF ORANGES ON TREE

For detection of oranges on tree, 200 frames color images of orange fruits were taken in outdoor natural lighting conditions in during 2011-2012 from the orange tree orchard in Finike (Mersin, Turkey) (Fig.1). There are 35.000 decare orange tree orchards in this region and every year approximately 200.000 ton orange fruits are harvested. The fruit picking is done by manpower in Turkey. In agriculture, there is a growing need to obtain higher quality products at a lower cost. Development of automatic systems that replace manpower will be speeded up the process and reduced the cost.

Fig. 1. An image of the orange tree from the orchard in Finike (Mersin,

Turkey)

T.R. Ministry of Food, Agriculture and Livestock I.T.U. TARBIL Agroinformatics Research Center (sponsors).

Fig. 2. The block diagram of the proposed method

In the orchard the images were taken with 30 minutes intervals in day time. So, the selected images were taken in different conditions such as the lighting (sunny and cloudy), and the time. This process was done for finding the most proper time intervals and natural lighting conditions for the detection of fruits. Furthermore, images were taken with the camera in different distances (0,5-1,5m). The purpose of this test was to evaluate the performance of the algorithm with respect to occlusion. Moreover, this process was carried out to obtain the optimum camera distance. In existing robotic harvesting systems, the distance from camera to the scene is unknown. However, if the optimum camera distance is obtained, this distance can be provided by another instrument such as an ultrasonic sensor.

The block diagram of the proposed method was given in Fig.2. The steps of the method were explained detailed below.

A. Local Analysis

In general, ripe orange fruit has a color ranging from yellow to orange. Surrounding of orange fruits composed of green leaves, brown branches, and sometimes sky as shown in Fig. 3.

Fig. 3. One of the near camera image which is used in the process (Source

image from Finike, Mersin, Turkey)

Color intensity-Pixels

Fig. 4. Colored histogram diagram of the orange fruit

In the applied method the first step is the segmentation. Segmentation separates the objects (orange fruits) of interest, from the background (leaves, branches, and sky) in the image.

First, the information of the orange color was obtained from the image which includes only an orange fruit by using colored histogram diagram was given in Fig.4. From histogram diagram it can be seen that the orange color includes red color which intensity is between 170 and 255, green color which intensity is between 70 and 220, and the blue color which intensity is smaller than 20. After that, the color intensity values of a near camera RGB image of an orange tree can be examined pixel by pixel to find which pixels are orange color and which pixels are not. If the color intensity values of the pixels are out of the range defined for the orange color, these color intensity values were changed to zeros which correspond to black color. So, the image consists of only orange color shapes and the other parts are black as shown in Fig.5.

The picture obtained in this way includes rounded orange colored shapes which are the images of the oranges and also, depending on the lighting conditions there can be small shapes coming from leaves which color resembles orange color. For removing these unwanted small shapes an adaptive median filter was used.

Fig. 5. The image after segmentation

Source image

(near camera image

of the orange tree)

Segmentation

Filtering

Centroid-based detection

The orange fruits have pale peels and the leaves’ surfaces of the orange trees are bright. In very sunny weather conditions the peels of orange fruits and the surfaces of the leaves seems brighter and reflect the sun light. The color of the leaves converge to the orange color because of the high brightness. The reflection impact in the images causes wrong detection. So, in the proposed method an adaptive threshold value is applied to the images according to the lighting condition. So that the bad influence of the illumination is reduced due to the adaptive threshold. Total illumination level method is used to calculate the adaptive threshold value. Average illumination level (ILave) for each unprocessed orange picture has been calculated by eq.1

Where, pi is a pixel value and M is vector dimension of the picture as M=nxm for a picture matrix nxm. Total illumination level for each picture is found out by Eq.2

(2)

where, bp is a pixel vector that the pixel values are greater than 220, k is dimension of vector bp. This situation is examined on 100 different orange pictures. We can say that ILA value is greater than 85.66, brightness is very high. So, the direction of angular view of camera should be changed. If the total illumination level (TIL) is high from 74.8 and less from 85.66, adaptive threshold method satisfies a good separation between orange and leaves according to brightness.

B. Shape Analysis

The orange fruit is a round shape fruit. For finding round object in the images an algorithm is applied as follows:

After the orange fruit images are segmented by the local analysis, the color image is converted to black and white in order to prepare for boundary tracing (binary image) as shown in Fig.6. It can be expected some of the orange fruit contours have overlapped, because of the fruit occlusion as seen in Fig.3. But the number of this occlusion does not affect the

Fig. 6. Black and white image (binary image)

correctness rate so much. However the occlusion of the leaves and branches affects less than the fruit occlusion in this method. The pixels or regions which do not belong to the objects of interest (any region that was either too small or too large to be an orange) are removed. This was particularly useful for eliminating noise. As shown in Fig.7 regions and parts smaller than oranges were removed. In the other words, if the number of the white (black) regions’ pixels are less than an image dependable threshold value, the colors of these regions were converted to opposite colors (black to white, white to black).

The shape feature of a single orange fruit is approximately circle as shown in Fig. 8. After finding the points of the rectangle surrounding the circle, circle center and the radius can be calculated as follows:

Xdistance = Xmax – Xmin Ydistance = Ymax – Ymin

center point = [(Xmin + Xdistance/2), (Ymin + Ydistance/2)]

radius = (Xdistance + Ydistance) / 2

After these values are calculated the geometric center of estimated orange fruit was positioned depend on shape feature as shown in Fig.9.

Fig. 7. Image after application of filtering and threshold

Fig. 8. Finding shape feature of a circle

(Xmin,Ymax) (Xmax,Ymax)

(Xmin,Ymin) (Xmax,Ymin)

Fig. 9. Positions of the geometric centers

It was obtained that the optimum distance for camera is nearly 50 cm, and suitable time intervals for detection in natural lighting conditions are 9:00 - 10:30 pm and 1:30 - 3:30 am in November in Finike, Mersin region. In these conditions 48 images were used. Two success rates were obtained for the proposed method. The first success rate (SC1) is calculated as follows:

SR1=TNDO/(CVO+BO),

where TNDO is total number of detected oranges, CVO is number of clearly visible oranges, BO is number of blurred oranges. Second success rate (SR2) that important for us is calculated as follows:

SR2=TNDO/CVO,

Average success rates for images test set were obtained as SC1=76,5% and SC2=92,5%. Because of leaves or branches in front of some oranges, detection process is not completely successful. In other words, sometimes one orange is detected as two different objects by the method. The rate of these false detected objects (FDO) was obtained as

FDO=FON//(CVO+BO),

where, FON is the number of the false detected orange. Average rate of false detected object was obtained as 1,87%.

III. CONCLUSION

In this paper a method that depends on color and also shape analysis technique for automatically determine the orange fruits on the tree using near camera images has been presented. In the proposed method variable lighting conditions and occlusion problems have been also examined.

In robotic harvesting the position and the distance of the camera are very important for the fruit detection. The optimum camera distance and suitable time intervals in natural lighting conditions have been also determined.

The aim is to detect the oranges localized near to harvesting robotic arm. So in this work, proposed method has very good performance.

ACKNOWLEDGMENT

This research was founded by TR Ministry of Food, Agriculture and Livestock and ITU TARBIL Agro-informatics Research Center.

REFERENCES

[1] Y. Sarig, “Robotics of Fruit Harvesting: A State-of-the-art Review”, Journal of Agricultural Engineering Research, 54 (4), pp. 265-280, 1993

[2] Y. Edan, “Design of an autonomous agricultural robot”, Applied Intelligence, 5, pp.41-50, 1995

[3] G. Grasso, M. Recce, “Scene analysis for an orange harvesting robot”, Artificial Intelligence Applications, 11, pp. 9-15, 1996

[4] M. Recce, J. Taylor, A.Plebe, G. Tropiano, “Vision and neural control for an orange harvesting robot”, International Workshop on Neural Networks for Identification, Control, Robotics, and Signal/Image Processing, Venice, pp.467-475, 1996

[5] A. Plebe, G. Grasso, “Localization of spherical fruits for robotic harvesting”, Machine Vision and Applications, 13, pp.70-79, 2001

[6] Y.Yanwei, Z. Xiaochao, Z. Huaping, “Apple harvesting robot picking path planning and Simulation”, International Conference on Information Engineering and Computer Science, ICIECS 2009, pp.1-4, 2009

[7] Z. De-An, L. Jidong, J. Wei, Z. Ying, C. Yu, “Design and control of an apple harvesting robot”, Biosystems Engineering, 110(2), pp.112-122, 2011

[8] A. R. Jimenez, R. Ceres, J. L. Pons, “A survey of computer vision methods for locating fruit on trees”, Trans. ASAE, vol.43, no.6, pp.1911-1920, 2000.

[9] K. L. Yam, S. Papadakis “A simple digital imaging method for measuring and analyzing color of food surfaces”, Journal of Food Engineering, No.61, pp.137–142, 2004.

[10] J. Zhao, J. Tow, J. Katupitiya, “On tree fruit receognition using texture properties and color data”, IEEE/RJS International Conference on Intelligence Robots and Systems, pp.263-268, 2005.

[11] D. M. Bulanon, T. Kataoka, S. Zhang, Y. Ota, T. Hiroma, “Optimal thresholding for the automatic recognition of apple fruits”, ASAE Paper, No.01-3133, 2001.

[12] D. M. Bulanon, T. Kataoka, Y. Ota, T. Hiroma, “A color model for recognition of apples by a robotic harvesting systems”, Journal of JSAM, vol. 64, no.5, pp.123-133, 2002.

[13] D. Stanjnko, M. Lakota, M. Hocevar, “Estimation of number and diameter of apple fruits in an orchard during the growing season by thermal imaging”, Computers and Electronics in Agriculture, vol. 42, pp.31-42, 2004.

[14] W. Mao, B. Jia, X. Zhang, X. Hub, “Detection and position method of apple tree image”, Computer and Computing Technologies in Agriculture II, vol. 2, IFIP Advances in Information and Communication Technology, vol. 294, pp. 1039-1048, 2009.

[15] M. Wenhua, J. Baoping, Z. Jicheng, Z. Xiaochao, H. Xiaoan, “Apple location method for the apple harvesting robot”, 2nd International congress on Image and Signal Processing, CISP’09 , pp.1-5, 2009

[16] E.J. Van Henten, B.A.J. Van Tuijl, G.-J. Hoogakker, M.J. Van Der Weerd, J. Hemming, J.G. Kornet, J. Bontsema, “An Autonomous Robot for De-leafing Cucumber Plants grown in a High-wire Cultivation System”, Biosystems Engineering 94 (3), pp. 317–323, 2006

[17] M. W. Hannan, T. F. Burks, D. M. Bulanon, “A machine vision algorithm combining adaptive segmentation and shape analysis for orange fruit detection”, Agricultural Engineering International: The CIGR Ejournal. Manuscript 1281, Vol. XI, December, 2009.

[18] H. N. Patel, R. K. Jain, M. V. Joshi, “Automatic segmentation and yield measurement of fruit using shape analysis”, International Journal of Computer Applications, 45 (7), pp. 19-24, 2012

[19] C. Akın, M. Kırcı, E. O. Güneş, Y. Çakır, “Detection of the Pomegranate Fruits On Tree Using Image Processing”, The first International Conference on Agro-Geoinformatics, China, 2012.