UNIVERSITI PUTRA MALAYSIA

DEVELOPMENT OF A NEURAL NETWORK-BASED CAMERA FOR

TOMATO HARVESTING ROBOTS

AZIZ BIN MAMAT

FK 2008 88

DEVELOPMENT OF A NEURAL NETWORK-BASED CAMERA FOR TOMATO HARVESTING ROBOTS

by

AZIZ BIN MAMAT

Thesis Submitted to the School of Graduate Studies, University Putra Malaysia, in Fulfilment of the requirements for the Degree of Master Of Science

December, 2008

Dedicated to my parents,

Hj. Mamat Bin Hj. Awang and Hjh. Habsah Binti Hj. Awang Kechik

My beloved wife,

Hjh. Sazilawati Binti Abdul Aziz

My beloved daughter and sons,

Nurul Afina Binti Aziz

Zaim Raziqin Bin Aziz

Zaim Wafiuddin Bin Aziz

and

friends

with love.

i

Abstract of thesis presented to the Senate of Universiti Putra Malaysia in fulfilment of the requirement for the degree of Master of Science.

DEVELOPMENT OF A NEURAL NETWORK-BASED CAMERA FOR TOMATO HARVESTING ROBOTS

By

AZIZ BIN MAMAT

December 2008

Chairman : Professor Wan Ishak Wan Ismail, PhD

Faculty : Engineering

Automated tomato harvesting robots were rapidly developed recently. Most of the

designs were more focused on positioning of the end of robotic arm by using various

methods such as combination of the sensor and vision system. This project

concentrated on the artificial intelligent via the Neural Network, in order to provide a

better decision making system for tomato harvesting robot. The objective of this

study was to develop 3 degree of freedom tomato harvesting robotic system complete

with gripper and motion program. The development of software for tomato pattern

identification, determination of the X and Y coordinates from web camera captured

and the determination of the tomato and tomato ripeness using decision making from

Neural Network also become the main objective. The approach is to detect the

desired object using vision system attached to the cylindrical automation system and

perform image analysis. These features will serve as inputs to a neural net, which

will be trained with a set of predetermined ripe tomato. The output is a command for

harvester arm to make the movement for harvesting. The position determination was

done with a conversion of the distance in pixel into a distance in metric unit (mm) of

ii

the tomato image. Whereas the depth of the tomato distance (z direction) was done

by moving the actuator system towards the calculated tomato position until the

object sensor senses the present of the tomato. AWIsoft07 software was developed

to view the harvester vision, display the captured image analysis on the harvester

vision, and display the numerical analysis output and neural network output.

The harvester system with 3 degree of freedoms (3DOF) equips with specially

designed tomato gripper named as AWI2007 Tomato Harvesting Robot was

developed in order to realize the data from the AWISoft07 developed software.

Several calibrations were made to ensure the accuracy of the AWI2007 Tomato

Harvesting Robot. The AWIsoft07 and AWI2007 Tomato Harvesting Robot were

subjected to several harvesting tests under the laboratory environment. The

harvesting result shows the ability of the software and the harvester. Consequently,

AWI2007 Tomato Harvesting Robot with the camera vision was able to recognize

the tomato ripeness intelligently via neural network analysis and moved to the

harvesting position. These situations provided new improvements for tomato

harvesting system compared to the previous findings. Therefore the application of

the neural network based on camera vision was successful perform as artificial

intelligent for tomato harvesting robotic system.

iii

Abstrak tesis dikemukan kepada Senat Universiti Putra Malaysia sebagai memnuhi keperluan untuk ijazah Master Sains

MEMBINA NEURAL NETWORK BERASASKAN KAMERA UNTUK ROBOT PENUAIAN TOMATO

Oleh

AZIZ BIN MAMAT

Disember 2008

Pengerusi : Professor Wan Ishak Wan Ismail, PhD

Fakulti : Kejuruteraan

Robot penuaian tomato yang beroperasi secara outomatik telah dibina secara rancak

kebelakangan ini. Kebanyakan rekabentuknya memfokuskan kepada penentuan

kedudukan pengesan hujung lengan robot dengan menggunakan berbagai kaedah

seperti kombinasi sensor dan sistem penglihatan. Projek ini menumpukan kepada

kepintaran tiruan melalui Jaringan Neural, dalam menyediakan sistem pembuatan

keputusan untuk robot penuaian tomato. Objektif kajian ini adalah membangunkan

robot penuaian tomato dengan tiga darjah kebebasan, lengkap dengan penggengam

dan program pergerakan. Pembangunan perisisan untuk identifikasi bentuk tomato,

menentukan koodinat X dan Y daripada rakaman imej kamera sesawang dan

menentukan tomato serta kemasakan tomato, menggunakan pembuat keputusan

daripada Jaringan Neural yang juga menjadi objektif utama. Pendekatannya ialah

untuk mengesan objek yang perlukan menggunakan sistem penglihatan yang

dipasangkan pada sistem automasi selinder dan menghasilkan analisa imej. Ciri-ciri

ini akan berfungsi sebagai masukan kepada jaringan neural, yang mana akan dilatih

dengan satu set tomato masak yang telah dikenalpasti. Keluarannya ialah arahan

iv

kepada lengan penuai bagi menghasilkan gerakan untuk penuaian. Penentuan

kedudukan dilakukan dengan menukarkan jarak dalam unit pixel kepada jarak dalam

unit metrik (mm) untuk imej tomato. Sebaliknya jarak kedalaman tomato (arah z)

dilakukan dengan menggerakkan lengan penggerak sistem kearah kedudukan tomato

yang telah dikira kedudukannya sehingga sensor objek mengesan kehadiran tomato.

Perisian AWIsoft07 berkeupayaan mempamirkan pandangan penuai, dan

memaparkan analisa imej yang diperolehi daripada pandangan penuai, memaparkan

keluaran analisis berangka dan keluaran jaringan neural.

Sistem penuai slinder dengan tiga darjah kebebasan dengan penggenggam tomato

yang rekabentuk khas dinamakan sebagai AWI2007 Robot Penuaian Tomato telah

dibangunkan demi merealisasikan data daripada perisian AWIsoft07. Beberapa

kalibrasi telah dilakukan untuk memastikan ketepatan AWI2007 Robot Penuaian

Tomato. Perisian AWIsoft07 dan AWI2007 Robot Penuaian Tomato telah melalui

beberapa ujian penuaian di bawah kesekitaran makmal. Keputusan penuaian

menunjukkan kebolehan perisisan dan penuai. Sesungguhnya, AWI2007 Robot

Penuaian Tomato berkeupayaan untuk mengenali kemasakan buah tomato dan

bergerak kekedudukan penuaian secara pintar. Keadaan ini telah menghasilkan

suatu penambahbaikan untuk sistem penuaian tomato berbanding dengan hasil

dapatan sebelumnya. Kesimpulannya aplikasi jaringan neural berasakan penglihatan

kamera telah berjaya dihasilkan sebagai kepintaran tiruan untuk sistem robot

penuaian tomato.

v

ACKNOWLEDGEMENTS

The author would like to thank ALLAH s.w.t the most gracious and merciful for

giving the opportunity and the ability to finish this project.

Special great appreciation and gratitude is expressed to Professor Dr. Ir Wan Ishak

Wan Ismail who keeps excellent supervision, help, continuous encouragement,

guidance and numerous discussions throughout this project. Great appreciation is

also expressed to Associate Professor Dr. Datin Napsiah Ismail and Associate

Professor Dr. Abdul Rahman Ramli for their advice, help valuable remarks and

encouragement all the way throughout this project.

Honestly appreciation is expressed to Politeknik Kota Bharu, Kementerian Pangajian

Tinggi, Department of the Civil Servant and Government of Malaysia for their

support and sponsorship throughout this study.

The deepest appreciation is expressed to Mr Jasman Jaafar, Mr. Khairuddin Abdul

Rahman, Mr. Zakiria Bin Ismail, Mr. Abd. Rahman Abdullah and all the wonderful

people who were involved in this project directly or indirectly for giving the help,

guidance, advice, support and encouragement throughout the course of this study.

Sincere gratitude and appreciation is also addressed to my beloved wife, kids, family

and friends for their full support, help, and encouragement in continuing my study.

vi

I certify that a Thesis Examination Committee has met on 2 December 2008 to conduct the final examination of Aziz bin Mamat on his thesis entitled "Development of a Neural Network-Based Camera for Tomato Harvesting Robots" in accordance with Universities and University Colleges Act 1971 and the Constitution of the Universiti Putra Malaysia [P.U.(A)106] 15 March 1998. The Committee recommends that the student be awarded the Master of Science.

Members of the Thesis Examination Committee were as follows: Ishak Aris, PhD Associate Professor Faculty of Engineering Universiti Putra Malaysia (Chairman) Ir. Desa Ahmad, PhD Professor and Ir Faculty of Engineering Universiti Putra Malaysia (Internal Examiner) Megat Mohamad Hamdan, PhD Professor Faculty of Engineering Universiti Pertahanan Malaysia (Internal Examiner) Mohd Marzuki Mustafa, PhD Professor Faculty of Engineering Universiti Kebangsaan Malaysia (External Examiner)

_________________________ BUJANG KIM HUAT, PhD Professor and Deputy Dean School of Graduate Studies Universiti Putra Malaysia Date: 19 March 2009

vii

This thesis was submitted to the Senate of Universiti Putra Malaysia and has been accepted as fulfilment of the requirement for the degree of Master of Science. The members of Supervisory Committee were as follows: Wan Ishak Wan Ismail, PhD Profesor and Ir Faculty of Engineering Universiti Putra Malaysia (Chairman) Napsiah Ismail, PhD Associate Professor and Datin Faculty of Engineering Universiti Putra Malaysia (Member) Abdul Rahman Ramli, PhD Associate Professor Faculty of Engineering Universiti Putra Malaysia (Member)

_______________________________ HASANAH MOHD GHAZALI, PhD Professor and Dean School of Graduate Studies Universiti Putra Malaysia Date: 9 April 2009

viii

DECLARATION

I declare that the thesis is my original work except for quotations and citations which have been duly acknowledged. I also declare that it has not been previously, and is not concurrently, submitted for any other degree at Universiti Putra Malaysia or at any other institution.

__________________ AZIZ BIN MAMAT Date : 30 January 2009

ix

TABLE OF CONTENTS Page

ABSTRACT ii ABSTRAK iv ACKNOWLEDGEMENTS vi APPROVAL vii DECLARATION ix TABLE OF CONTENTS x LIST OF TABLES xiii LIST OF FIGURES xiv LIST OF ABBREVIATIONS xvi

CHAPTER

1.0 INTRODUCTION 1

1.1 Overview 1 1.2 Problem Statements 3 1.3 Research Scope And Limitations 5 1.4 Objectives 6 1.5 Thesis Layout 6

2.0 LITERATURE REVIEW 8

2.1 Anatomy of the tomato 8 2.2 Malaysian tomato variety 10 2.2.1 Tomato grading and maturity 10 2.3 Color and spectrum 10 2.3.1 Color in outdoor scenes 10 2.4 Camera and robot vision 12 2.5 Neural Network 15 2.6 Harvesting robot in greenhouse 24 2.7 Open Fields harvesting robot 25 2.8 Existing tomato harvesting robot 26

3.0 METHODOLOGY 34

3.1 Preliminary study 34 3.1.1 Image acquisition 34 3.2 Digital Image Processing 37 3.2.1 Image enhancement 37 3.2.2 Color image segmentation 39

x

3.2.3 Mathematical operation on the image and object edge detection

39

3.2.4 Objects indexing 41 3.3 Artificial Neural Network (ANN) 44 3.3.1 Learning mechanism 45 3.4 AWIsoft07 software 47 3.4.1 Overview of the AWIsoft07 software 48 3.4.2 Digital Image Analysis using AWIsoft07

Software 54

3.4.3 Neural Network Analysis using AWIsoft07 software

58

3.4.4 AWIsoft07 software controlling the AWI2007 Tomato Harvesting Robot

59

3.5 AWI2007 Tomato Harvesting Robot Development 61 3.5.1 Design concept 61 3.5.2 Tomato gripper design 61 3.5.3 AWI2007 Tomato Harvesting Robot Design 63 3.5.4 Mobile unit design 64 3.5.5 The X axis direction design mechanism 65 3.5.6 The Y direction design mechanism 66 3.5.7 The Z direction design mechanism 67 3.5.8 The Motor Driver Determination 69 3.5.9 The support system 75 3.6 AWI2007 Tomato Harvesting Robot Calibration 76 3.6.1 Pixel image distance calibration 76 3.6.2 X axis rotation angle calibration 77 3.6.3 Y axis vertical distance calibration 78

4.0 RESULTS AND DISCUSSION 80

4.1 Digital image pixel value calibration 80 4.2 X axis rotation angle calibration 81 4.3 Y axis displacement calibration 83 4.4 Neural Network Analysis 85 4.5 AWIsoft07 Result Display 88 4.6 Robot Motion According To The Tomato

Coordinate 90

4.7 Harvesting Experimental 90 4.8 Problems And Suggestions 92 4.8.1 Effect of the changes of the light intensity

on AWIsoft07 analysis 92

4.8.2 Effect of the changes of the webcam parameter on AWIsoft07 analysis

93

4.8.3 AWI2007 Tomato Harvesting Robot modification.

93

xi

4.8.4 Modification of the communication system 94 4.9 Research Benchmark 95

5.0

CONCLUSIONS AND RECOMMENDATIONS FOR FUTURE RESEARCH 96

5.1 Conclusion 96 5.2 Recommendations for future research 99

REFERENCES 100

APPENDICES BIODATA OF STUDENT LIST OF PUBLICATIONS

xii

LIST OF TABLES

Table Page

2.1 Errors in calculating L*a*b* 12

4.1 RGB average value 86

4.2 Results of Neural Network analysis on five selected colors 87

4.3 Result of the tomato harvesting using AWI2007 Tomato Harvesting Robot.

91

xiii

LIST OF FIGURES Figure Page

2.1 Estimate of L*a*b* values on the basis of RGB measurements.

11

2.2 A sketch of the experimental setup.

13

2.3 A sketch of a laser guidance system.

13

2.4 Configuration of the vision system.

15

2.5 Three-layer back-propagation neural network. 18

2.6 Architecture of the neural network used for the analysis.

22

2.7 Grape harvesting end-effector

25

2.8 Tomato Robot harvester situated in green house

27

2.9 Block diagram of 3-D Vision sensor

28

2.10 End effectors (First prototype).

29

2.11 End effectors (Second prototype).

30

2.12 Motion of suction pad and check valve.

30

2.13 Method of harvesting.

32

3.1 The overall structure flowchart 35

3.2 Tomato color plate sampling 36

3.3 Overall Digital Image Processing Flowchart 38

3.4 Image adding technique

40

3.5 Image subtraction technique

40

3.6 Image threshold

42

3.7 Boundary and Centroid determination.

44

3.8 The determination of the 10 by 10 pixels from tomato image center coordinate.

45

3.9 The flowchart of the neural network sub program 46

3.10 Tomato color plate with three-layer back-propagation neural network

48

xiv

3.11 AWIsoft07 Software

49

3.12 AWIsoft07 software Control Panel:Tomato Calibration Mode menu flowchart

50

3.13 AWIsoft07 software Control Panel:Image Acquisitions menu flowchart

51

3.14 Control Panel Operation

52

3.15 Operation of the MODE B

53

3.16 Image Acquisition control button 54

3.17 AWI2007 Preview window

55

3.18 Image display from the output of the digital image processing

56

3.19 Numerical Output From Digital Image Processing

57

3.20 Neural Network analysis window

58

3.21 Robot motion flowchart sub program 60

3.22 Fingers of the Tomato gripper model 62

3.23 The assembly of the X,Y and Z axis motor 64

3.24 Mobile unit assembly model

65

3.25 The assembly of the X axis motor

66

3.26 The assembly of the Y axis motor

68

3.27 The assembly of the Z axis motor

69

3.28 Portion of the main helical shaft 71

3.29 Force diagram the main helical shaft 71

3.30 Measurement of the webcam distance

77

3.31 Calibration of the rotational angle for X axis

78

3.32 Calibration of the vertical movement on Y axis

79

4.1 Calibration of the pixel distance converted to mm distance

81

4.2 Calibration of the X axis rotation angle to time of X axis motor activation

82

xv

4.3 Calibration of the time of the Y axis motor activated and pointer travel distance measured

84

4.4 The measurement of the pointer travel distance of the Y axis motor activated according to the 10 second time interval

85

4.5 The result from the AWIsoft07 89

4.6 Webcam setting parameter

94

xvi

LIST OF ABBREVIATONS θ The threshold value

α The learning rate.

δk(p) The error gradient at neuron k in the output layer at iteration p.

∂yk (p) The output of neuron k in the output layer at iteration p.

∂xk (p) The net weighted at neuron k in the output layer at iteration p

Δwjk(p) The output layer weight correction at hidden layer j.

2D Two dimensions

3D Three dimensions

ANN Artificial neural network

CCD Couple charge diode

CW Clockwise

CCW Counter clockwise

d The camera distance (mm)

d.c. Direct current

DOF Degree of freedom

ek (p) The error of neuron k in the output layer at iteration p

Ec Calibration error

Er The percentage of error

GUI Graphical User Interface

IGES Improved gray-scale

MSE Mean square of error

MT Metric ton

n The number of neuron input

xvii

NN Neural network

P The distance in pixels

PSD Position-sensitive device

Q The distance in millimeters (mm).

R2 Coefficient of regression

RCCW The counter clockwise X axis angle of rotation in degrees

RCW The clockwise X axis angle of rotation in degrees

RGB Red, Green and Blue color space

TCW The time given to activate the X axis motor in second for clockwise rotation.

TCCW The time given to activate the X axis motor in second for counter clockwise rotation.

TDW The downward time given to activate Y axis motor, in second.

TUW The upward time given to activate Y axis motor, in second.

TV Television

UPM Universiti Putra Malaysia

U.S.A United State of America

USB Universal serial bus

Volt Voltage

wi The weight of input i

wjk(p) The output layer weight.at hidden layer j.

wjk (p + 1) The output layer updating weight at hidden layer j.

xi The value of input i

X The net weighted input to the neuron

yd.k(p) The desired output of neuron k at iteration p.

yj(p) The output of neuron j in the hidden layer p

xviii

xix

yk(p) The actual output of neuron k at iteration p

Y The output of the neuron

YDW The displacement distance in downward direction in millimeter

YUW The displacement distance in upward direction in millimeter

CHAPTER 1

INTRODUCTION

1.1 Overview

Lycopersicum esculentum which is known as tomato was classified under fruity

vegetables. Tomato was originated from South America and widely produced in the

tropical environment. Cameron Highland was the main tomato production area in

Malaysia, and for the low land tomato widely produced in Johor (MOA, 2006).

Consumption of the tomato in Malaysia can be categorized into fresh and process

tomato. The process tomato mainly can be divided into tomato puree, tomato paste

and concentrated tomato. Processed tomato also produced tomato sauce and

ketchup. Ministry of Agriculture in Malaysia reported that in year 2001, Malaysia

imported 8,588.59 metric ton (MT) (RM20,605,256) tomato puree, tomato paste and

concentrated tomato, where as in 2002 (until April) Malaysia imported 2,248.52245

MT(RM5,183,535) (MOA, 2006). Malaysia also exported 12,295 MT fresh tomato

in 2000 and stand 20th place world ranking for tomato exporter. Malaysia exported

7,517MT tomato sauce and ketchup which is 1% of the world market (MOA, 2006).

The decrement in import and increment in export, indicate that the tomato production

in Malaysia showed a positive indicator for expansion.

To produce tomato in a large volume needs a large area of land. The best tomato

cultivation is on the high land area cultivated with the high land variety tomato.

However the highland area is limited, therefore tomato production must be done in

the low land area with the cultivation of the good variety lowland tomato. The

modern production techniques must be practiced with controlled environment.

Under the green house production system, the possibility to apply the automation and

robotic system become more relevant especially when the rising cost of labor and the

difficulties to employ the labor for agricultural work.

Recently many researchers develop the robot in the way of fully autonomous and

move like the human being. The vision system was developed to represent the

human eyes by using capture device such as camera. The capture devices were used,

ranging from an expensive video camera, couple charge diode (CCD) camera, digital

camera, to the low-priced web camera. The different camera will give the different

sensitivity and resolution. The sensitivity of the capture device also depends on the

environment condition, especially to the light intensity. Normally the captured

image especially when using web camera will displayed in the Red, Green and Blue

(RGB) color space. The digital image processing is able to reduce the effect light

intensity to the captured images. The digital image processing also can be applied to

identify the pattern of the object in the image.

In the new era of the robot development, artificial intelligent becomes one of the

most important tools to make robot becomes intelligent. The artificial intelligent is

defined as "the study of the computations that make it possible to perceive, reason,

and act", includes an expert systems, fuzzy logic and artificial neural network

(Winston, 1993).

Traditionally, tomato harvesting was done manually by picking the fruit by hand or

cut by knife or scissor. Traditional harvesting method is normally facing the

2

problems of labors and time consuming. Therefore the paradigm change needed to be

done in this area. Mechanization or automation in the harvesting seems to be a

solution for those problems. The physical characteristics of the tomato plant, such as

soft and brittle stem are constraints to the mechanization application. The fruit itself

is covered by soft layer skin. Consequently, the application of mechanization for

harvesting tomato may damage the tomato stem and bruise the tomato fruit.

Therefore the best way to make the paradigm change is using automation in

harvesting. The advantages of automation are

i) Tomato fruit will avoid from bruise during harvesting.

ii) The tomato plant will not be damaged during harvesting.

iii) The usage of labor will be reduced.

iv) The tomato is possible to be graded during harvesting.

v) The robot is capable of working non stop (due to the power supply life

time).

Further more, there are several disadvantages for automation, such as higher initial

capital needed and good technical knowledge to operate. The cost for maintenance is

also high when using automation.

1.2 Problem Statements

The introduction of the robot in the bioproduction environments not only saves labor

but will help raise productivity. The developments of the agriculture robots were

raise according to the needs. For examples, an autonomous six-legged walking robot

was developed to explore and sample in rugged for outdoor environments

3

(Hoffman, 1991). The three degree of freedom (DOF) agricultural harvester robot

was designed to perform the oil palm harvesting (Bouketir, 1999). The agricultural

harvester robot was equipped with CCD camera for red object identification and an

algorithm was also developed in order to transfer two dimensional image plane to the

base of the three dimensional coordinate frame. However no report was made on the

intelligent robot component. The 'robot eye' system for agriculture robot to predict

actual distance of the target object using videogrammetry technique and triangulation

method was developed toward the intelligent robot (Mohd, 2002).

The evolution of the robot works under the greenhouse environment also grows in

the similar way. The mushroom harvesting robot was developed by Reed et.al.,

(1995) to find the relation between detachment torque and cap diameter and the

relation between torque and twist angle detachment. It was reported that an available

torque of 200Nmm combined with a rotational angle of 180˚ should be necessary.

Van et. al. (2006) reported on the de-leafing cucumber plant using an autonomous

robot. In a number of cases one leaf was reported successfully removed. The vision

system and manipulator control was reported perform relatively well (Van et. al.,

2006; Van et. al., 2002).

The research on the strawberry harvesting robot was reported by Satou et. al, (1996),

and the cherry tomato harvester robot (Fujiura et. al., 1990; Subrata et. al., 1995)

were developed by researches. The robot was guided with machine vision (a 3-D

vision sensor) and the harvesting algorithm. The end-effectors for the tomato

harvesting robot also were developed by Monta et. al. (1998). The end-effectors

with two parallel plate fingers and a suction pad was reported successfully harvest

4