1

Segmentation of Greenhouse Cucumber Plants in Multi-Spectral Imagery

Scott D. Noble*, Dali Li

Department of Chemical and Biological Engineering, University of Saskatchewan, 57 Campus Drive, Saskatoon, S7N 5A9 Canada

*Corresponding author. E-mail: scott.noble@usask.ca

Abstract Four spectral datasets of cucumber plants were collected and examined for spectral features that could be used for segmenting fruit, leaves and vines in spectral images. Two datasets were. Each set was collected with a different instrument or instrument configuration. Two datasets were laboratory based, while the latter two were collected in a production greenhouse. Point-spectra and spectral image datasets were included in each environment. Differences were found between the spectral features observed in point-reflectance data versus spectral-image based data. Lighting quality had a significant impact on segmentation quality. Various relationships of reflectance between bands around 550, 790, 820, 850, 940 and 970 nm showed promise for cucumber image segmentation.

Key words: reflectance, segmentation, water absorption, imaging spectroscopy.

1. Introduction Developing machine vision systems to properly interpret scenes containing plants is one of the on-going challenges in agricultural automation. A robust set of imaging and processing techniques is required to deal with the combination of environmental factors, biological variability, and general lack of situational constraints compared to most industrial machine vision applications. In robotic harvesting and pruning operations one challenge is to properly segment plant parts (fruit, leaves, and vine or stem) from each other. For crops such as cucumbers the similarity of colour between plant parts adds to the difficulty.

In this work the potential for using narrow band (5 nm) spectral imagery in the 450-1000 nm spectral range is investigated. This is done using a combination of point-based spectrometer and imaging spectrometer-based instruments, data collected under ideal conditions, and data collected under production greenhouse conditions.

While somewhat of a simplification, approaches to image segmentation can be broadly characterized as being shape-based (morphological) or spectral (colour) based. Most implementations employ both to a degree at some point in image classification. If the data can be acquired and differences exist, spectral techniques are generally less computationally demanding and should therefore be faster than shape-based techniques.

Studies have been conducted on using spectral reflectance information to improve segmentation of cucumber fruit. These studies have looked at the visible and near infrared (NIR) portions of the spectrum. The NIR wavelengths respond to cellular structure, water content and thickness, unlike the visible spectrum where pigment absorption dominates. Kondo and Endo (1988) measured the reflectance of cucumber fruit, leaves, flowers and stems with 1nm resolution. Fruit had lower reflectance at 550 nm (green) than leaves and stems. Conversely fruit had higher reflectance between 750 and 900 nm than other plant parts. A dual camera system using interference filters at 550 and 850 nm for discriminating cucumber fruit from other plant material was also reported (Kondo & Endo, 1988). The presence of the water absorption band at 970 nm was also noted in fruit and stems, while not being apparent in leaves (Kondo & Endo, 1988). A patent by Kornet and Meuleman (1999) describes the use of this water absorption feature in conjunction with a second measurement at a wavelength between 750 and 925 nm (850 nm being preferred) for detecting water-containing objects such as fruit. An imaging device for acquiring image data was also

2

described, and was similar in function to that described by Kondo & Endo (1988). Van Henten et al. (2002) described the use of this water absorption band method on a greenhouse cucumber harvesting robot. Fruit identification was reported as 95% correct, with a correct harvest rate of 80%.

In a study on using the ratio of reflectance between 900 and 970 nm to estimate water contend, Penuelas et al. (1997) observed a shift in the wavelength at which the local minimum within the water absorption band occurred. The minimum was reported to shift from 970-980 nm to 930-950 nm as the plant water concentration decreased.

While excellent segmentation results have been reported from approaches using ratios between various NIR and/or visible wavelengths, several points remain to be examined. First the spectral data from which these relationships have been developed appear to have been measured under ideal conditions, outside of a working environment, and not using in situ imaging conditions. Reflectance (and reflectance factor) are known to be dependent on measurement geometry (Schaepman-Strub et al., 2006), which has been encountered and in the form of specular reflections from fruit. The measurements have also excluded the impacts of solar illumination filtered through both the atmosphere and the potentially high-humidity of the greenhouse environment. This would effectively reduce the illumination available in the water absorption band. Segmentation of the stems and vines as distinct from either fruit or leaves has also been ignored from the spectral perspective.

The objectives of this work were to verify the spectral relationships between previously observed wavebands using a variety of spectral instruments and conditions, and to develop and test preliminary classification schemes for cucumber fruit, leaves, and vines under laboratory and production greenhouse conditions.

1. Methods Four sets of data were collected and used in evaluating spectral features for segmentation (Table 1). Two sets were collected in 2009 under laboratory conditions. Cucumber plants were grown in pots in a university research greenhouse and transported to the laboratory for measurements. Samples were excised for the diffuse reflectance measurements using a UV-Visible-NIR spectrophotometer (Spectro-2009). Whole plants were imaged using a line-scanning imaging spectrophotometer (Image-2009). This instrument incorporated a uniform light source providing intense, spatially stable, diffuse illumination and frame-by-frame correction of illuminant intensity fluctuations (Noble et al., 2012). The distance from plant to imager was approximately 1.7 m.

Two additional sets of data were collected in a commercial greenhouse. The crop was trellised, with approximately 0.9m between plant rows. All measurements were taken in situ. The 45°/0° reflectance factor of leaves, fruit, and vines was measured using a compact diode-array spectrometer, tungsten-halogen light source, and a sampling device designed to clamp onto thin samples for measurement (Spectro-2012). The clamp was removed for measurements of fruit and vines. Spectral image data (Image-2012) were collected with the same line-scan imaging spectrograph as described by Noble et al. (2012), but without the uniform light source. The imager head was mounted to an upright post on a utility cart, in a side-facing orientation, and moved between rows. The sky was overcast, which reduced the intensity of natural light in the greenhouse, but also made it diffuse. Natural illumination at 970 nm was very weak, presumably due to absorption by atmospheric water absorption and liquid water condensed on the greenhouse cover. Two tungsten-halogen flood lamps were added to supplement the diffuse natural light. These data were subject to significant variation in light distribution. Reflectance correction was done to the average measurement of a 60 cm × 60 cm Spectralon target (Labsphere Inc., North Sutton NH, USA) with a nominal diffuse reflectance of 75%.

Average spectra of fruit, leaf, and vine were calculated using a minimum of three measurements from each dataset. For image data, averages were calculated from manually

3

defined sample regions within an image selected for lighting quality. Spectral indices were defined and stepwise decision-tree classifiers were manually developed based on histograms of the defined indices and observation.

TABLE 1: Summary of datasets and associated parameters

Dataset

Spectro-2009 Image-2009 Spectro-2012 Image-2012

Instrument

Cary 5G UV-Vis-NIR spectrophotometer (Agilent Technologies, Mississauga, Can.)

U. Sask. Imaging Spectrophotometer (Noble et al., 2012)

Ocean Optics Maya 2000 Pro spectrometer (Ocean Optics, Dunedin FL, USA)

U. Sask. Imaging Spectrophotometer (Imager Only)

Measurement Configuration

Point Measurement, Diffuse Reflectance Accessory (0°/d)

Image, Uniform Light Source (d/0°) with incident light correction

Point Measurement, 45°/0°

Image, average reference panel correction

Spectral Range 250-2500 nm 400-1000 nm 350-1100 nm 450-1000 nm

Resolution 1 nm 5.2 nm < 1nm 5 nm, resampled from 5.2 nm

Illumination Deuterium and Tungsten via Monochromator

Tungsten with Metal Halide Arc lamps into integrating sphere diffusor

Tungsten-halogen via fibre-optic line

Sunlight through cloud and glazing with supplemental tungsten/halogen floodlights

Environment Laboratory, low humidity Production Greenhouse, high humidity

Specimen Potted cucumber, untrellised, transported to lab and samples excised for Spectro-2009

measurements

Trellised English-style cucumbers, measured in place

2. Results and Discussion Figure 1 shows average spectra between 500 and 1000 nm for each dataset, arranged by plant part. Spectro-2012 curves for fruit and vine showed evidence of ambient light leaking into the measurement as a result of sample curvature and thickness. Reflectance calibration methods vary between datasets due to data quality limitations, particularly for Spectro-2012. These data should be examined for trends rather than absolute differences in reflectance.

Leaf reflectance in the NIR region was found to differ between imaging and non-imaging datasets. In the non-imaging data, Spectro-2009 and Spectro-2012, leaves did not show a prominent water absorption feature at 970 nm, while fruit and vines did. This is in agreement with published spectra (van Henten et al., 2002; Kondo & Ting, 1998). While these leaves obviously contained water, the argument has been that their thickness is insufficient for water absorption to be observed. However, these same leaves exhibited an absorption feature around 970 nm in both spectral image-based datasets Image-2009 and Image-2012. Thickness alone is clearly an incomplete explanation for this observed difference between the instruments. Two factors are proposed as explanations. First, in the point-spectral measurements, leaves are held normal to the observation direction, minimizing the thickness in the direction of measurement. Second, leaves measured in this way are isolated from the canopy and reflection and re-transmission of light back through the leaf and toward the detector. The additional effect of light being transmitted through the leaf toward the measurement device has been demonstrated to influence apparent reflectance in this band (Lillesaeter, 1982). The image-based measurements do not constrain these factors, so that orientation and canopy influences are observed.

Another such difference was observed in the 820 to 860 nm region, where image-based spectral data exhibited a more negatively-trending slope for cucumber fruit than for leaves or

vines. Pplateau”indicate

Based odifferen

Index =

Indices Table 2step in and 3 uin classClassifie

FIGUR

TABL

Stem-Fru

Fruit

Water-

Mid

A reprethe highClassifieresults fto sub-ounder rilluminafruit or v

Plant spectr” region, a

e the need fo

on the diffece equation

11BBandBBand

+−

=

and their w. From the all three wsed the SF

sifier 1 (seer 3. Classif

E 1: Compa

LE 2: Defini

Index

uit Reflectance

Reflectance (F

-Band Shift (W

d Plateau (MP

sentative sh-quality Imer 1 generfor the Imagoptimal threreal-world tion was advine at a hig

ra are geneand these or further in

erences obsn:

22

BandBand

.

wavelengthsindices of T

was to isolatFRI (separat

parating frufier 2 used

arison of ref

tions of spe

e (SFRI)

FRI)

WBSI)

PI)

ample of clmage-2009 rally exhibitge-2012 da

esholds, butconditions.dequate. Thgher rate tha

rally considdifferences

nvestigation

served, spe

s and thresTable 2, thrte vegetatioting fruit anuit from vinWBSI to se

flectance fa(B),

ectral indice

790, 970 : s

790, 550

940, 97

822, 850

lassificationdataset weted superio

ataset were t is judged t This saidhe SFRI claan did Clas

4

dered to be s between .

ectral indice

sholds usedree classificon from bacd vine fromne) and thegment vine

actor from fo, and vines

es, wavelen

Ban

2009

stem, fruit >0.1

0 : fruit >0.673

0 : vine >0.07

0 : fruit >0.014

n results is ere very goor vine claspoor by co

to be primad, fruit weassifiers 1

ssifier 2 bas

relatively spoint- and

es were def

d for each cation schemckground a

m leaves) ase WBSI to

e, and the M

our sources(C).

gths and th

ds (nm) and T

1228

1

7

4

shown in Food for all ssification mparison. Trily a result

ere generaland 3 had ed on the W

spectrally flad image-ba

fined based

imaging dames were d

and dark regs the first steo segment MPI to segm

s for cucumb

resholds us

Thresholds

790, 965 :

790, 55

935, 96

820, 84

Fig. 2. Classclassifiers in this datThis may beof lower-qu

lly identifiea tendency

WBSI. Class

at through tased spect

d on the no

ataset are sdeveloped. gions. Clasep, followedvines from

ment fruit.

ber fruit (A)

sed in class

2012

stem, fruit >0

50 : fruit >0.54

65 : vine >0.0

45 : fruit >0.0

sification re(Fig. 2, colaset. Classe partially auality data ged correctlyy to classifysifiers 2 and

this “NIR tral data

rmalized

(1)

shown in The first

ssifiers 1 d by FRI

m fruit in

, leaves

sifiers.

.065

4

esults on lumn 1). sification attributed gathered y where y leaf as d 3 also

FIGURdata (c

RE 2: Sampcolumns 2 a

c),

ple classificaand 3). Colo, and 3 (row

ation imageour composiw d). Blue =

5

es of Imageite (row a), fruit, red =

-2009 data Classificatiovegetation,

(column 1)on Scheme, green = vi

, and Imagees 1 (row b)ne.

e-2012 , 2 (row

6

tended to misclassify leaves as vine rather than fruit, which may be the preferred incorrect answer if fruit identification is the priority.

3. Conclusions It was concluded that spectral relationships previously described in the literature that utilize the water absorption band at 970 nm were generally effective for segmenting cucumber fruit from images. The quality of the segmentation is highly dependent on illumination quality, particularly given the absorption of light around 970 nm by atmospheric and greenhouse moisture sources. Artificial illumination is almost certainly required. A spectral feature observed in the image data between 820 and 850 nm was found that also appears effective at segmenting fruit from the image. The water concentration induced spectral shift in the 970-nm absorption band minimum showed some ability to segment vine from the image. These relationships do not appear to be strongly observable in typical spectrophotometer measurements, highlighting the value of in situ imaging spectrometer approaches in developing spectral indices.

4. Acknowledgements The authors would like to acknowledge the assistance of M. Boyko and R. Peters in collecting the 2012 data. Particular thanks are extended Pat and Fred Gittings of Grandora Gardens and their very patient staff. This work was funded in part by the Natural Sciences and Engineering Research Council.

References Kondo, N. & Endo, S. (1988). Calculation of the most suitable wavelength bands for

discrimination between fruits and leaves according to their spectral reflectance. Scientific Reports of the Faculty of Agriculture, Okayama University, 71, 23-9.

Kondo, N. & Ting, K. C. (1998). Robotics for plant production. Artificial Intelligence Review 12, 227-243.

Kornet, J. G. & Meuleman, J. (1999). Werkwijze en inrichting voorhet detecteren van waterrijke objecten [in Dutch] (method and device for detecting water-rich objects). The Netherlands Patent NL1013780, filed 1999, and issued 01.08.2001.

Lillesaeter O. (1982). Spectral reflectance of partly transmitting leaves - laboratory measurements and mathematical modeling. Remote Sensing of Environment. 12(3), 247-254.

Noble, S. D., Brown, R. B. & Crowe, T. G. (2012). Design and evaluation of an imaging spectrophotometer incorporating a uniform light source. Review of Scientific Instruments 83(3), 033112 1-9.

Penuelas, J., Pinol, J., Ogaya, R. & Filella, I. (1997). Estimation of plant water concentration by the reflectance water index WI (R900/R970). International Journal of Remote Sensing 18(13), 2869-2875.

Schaepman-Strub, G., Schaepman, M. E., Painter, T. H., Dangel, S. & Martonchik, J. V. (2006). Reflectance quantities in optical remote sensing-definitions and case studies. Remote Sensing of Environment 103(1), 27-42.

van Henten, E. J, Hemming, J., van Tuijl, B. A. J., Kornet, J. G., Meuleman, J., Bontsema, J. & van Os, E. A. (2002). An autonomous robot for harvesting cucumbers in greenhouses. Autonomous Robots 13(3), 241-258.