Sweet citrus fruit detection in thermal images using

fuzzy image processing

Ingrid Lorena Argote Pedraza John Faber Archila Diaz Renan Moreira Pinto
São Carlos School of Engineering Industrial Design Prof. São Carlos School of Engineering
University of São Paulo Industrial University of Santander University of São Paulo
São Carlos, Brazil Bucaramanga, Colombia São Carlos, Brazil
[email protected] [email protected] [email protected]

Marcelo Becker Mario Luiz Tronco

São Carlos School of Engineering São Carlos School of Engineering
University of São Paulo University of São Paulo
São Carlos, Brazil São Carlos, Brazil
[email protected] [email protected]

Abstract—In agriculture, intelligent systems applications have In this scenario, the improvement of the various tasks involved
generated great advances in automating some processes in the in the pre-harvest; using for example precision agriculture
production chain. To improve the efficiency of those systems is techniques in the harvest and post-harvest activities; it is
proposes a vision algorithm to estimate the amount of fruits
in sweet orange trees. This study proposes a computer vision important to the Brazil citrus producers because it can reduce
system based on the capture of thermal images and fuzzy costs and optimize the entire industry supply chain. Especially
image processing. A bibliographical review has been done to in the state São Paulo, the country’s largest citrus producer.
analyze the state-of-the-art of the different systems used in fruit
recognition, and also the different applications of thermography
in agricultural systems. The algorithm developed for this project Thus, the need to improve the quality of agricultural prod-
uses the intensification operator to contrast-enhanced and the ucts and increase production capacity in agriculture places a
fuzzy divergence for segmentation and Hough transform for huge interest in the research for new technologies that can
fruit identification. It estimates the numbers of fruits in the
tree, a task that is currently manually performed. In order to improve these processes. An example is the harvest of oranges
validate the proposed algorithm a database was created with on farms in São Paulo state. This is a task that has a direct
images of sweet orange acquired in the Maringá Farm. The impact on industry costs, nearly 30% of citrus production
validation process indicated that the variation of the tree branch costs are related to harvesting activities [1]. Thus, the focus
and the fruit temperature is not very high, making it difficult of this work is the development of a vision system that can be
to segment the images using a temperature threshold. Errors
in the segmentation algorithm could mean the increase of false further embedded in a vehicle and perform the identification
positives in the fruit-counting algorithm. Recognition of isolated fruits task. Bulanon et. al [2] developed a survey to assess
fruits with the proposed algorithm presented an overall accuracy changes in orange canopy using thermal imaging for the
of 93.5 percent and grouped fruits accuracy was 80 percent. The fruit detection. Their results suggest that the largest variation
experiments show the need of other image hardware to improve in temperature between the tree and fruit happens between
the recognition of small temperature changes in the image.
Index Terms—agricultural systems, citrus, fuzzy image pro-
16:00 and midnight. Such temperature difference facilitated
cessing, thermal images the fruit recognition using method proposed. The following
year Bulanon et. al [3] proposed the fusion thermal and visible
I. I NTRODUCTION images to detect fruit (more specifically, oranges).

The Brazilian citrus industry is one of the most productive

national agro-industrial sector. Every year are harvested about Another application of thermography in agriculture is the
16 million tons of citrus fruits, which represents about 25% monitoring of water status in the plant. The measurement of
of world production and currently, approximately 85% of the leaf temperature is the physiological characteristic used for the
orange juice that is consumed in the world is produced in development of the proposed method in [4]. The developed
Brazil. All these factors favored for Brazil to become the method showed good correlation between the temperature of
world’s largest producer of orange juice, which represents the kaki tree canopy and plant water status. In the case of
about 5% of the Product Gross Domestic (GDP) agricultural. citrus crop, it failed to show a correlation between tree canopy
temperature and plant water status. Several studies focused on
water stress citrus trees. Among the most relevant, we can
978-1-7281-1614-3/19/$31.00 ©2019 IEEE mention: [5], [6], [7], [8], [9].
 II. M ETHODOLOGY the samples, while the ambient temperature was monitored.
The methodology shown in Fig. 1 was used to develop the We conducted a simple experiment for measuring the reflected
counting fruit algorithm using thermal images. Each step of temperature because were present in the environment and they
this methodology is detailed in the following subsections. could in the measurement as the temperature radiated by the
walls, the lamps and the electronic equipment. It should be
emphasized that the reflected temperature has a considerable
influence in the measurement, when the objects to be measured
have a very low emissivity. Just as with emissivity, reflected
temperature must be measured to have a correct temperature
measurement with the camera, In this experiment, the reflected
temperature was measured to be set the largest number of
parameters that can influence the measurement taken by the
camera.

B. Image Acquisition
This study uses a thermal camera FLIR® A35s with a micro
bolometer detector VOX type uncooled. It has a sensitivity
0.05 °C, fixed focus, resolution of 320 x 256 pixels a spectral
range from 7.5 to 13 nm, an accuracy ±5 °C or ±5% in
reading and a dynamic range of 14 bits. TCP/IP protocol is
used of communication between the camera and acquisition
software. We used TESTO-175 model to monitor the environ-
ment. It is an instrument used to store and perform the reading
of a series of measures. To acquire thermal imaging we used
the camera’s manufacturer software FLIR: FLIR Tools+. This
choice was based on the fact that this software allows the
user to easily configure various parameters, such as emissivity,
reflected temperature, the distance between the camera and
Fig. 1. Fruit identification methodology
the subject, atmospheric temperature and relative humidity.
In addition, the software allows selection of areas of interest
A. Thermal emissivity and reflected temperature in the image and calculates the average temperature in the
The first step of the methodology is orange fruit emissivity selected area.
determination. According to the literature, it is important The image acquisition system was mounted on Mirã
to know the emissivity value of study object to improve (Fig. 2). This mobile robot was developed to be a modular
the accuracy of temperature value measured by the thermal platform for test different sensor configurations and to perform
cameras. The emissivity and reflected temperature are used to basic maneuvers to navigate with ease in the crops. To develop
configure the thermal camera parameters. This setup shall be this study the robotic platform was used to make the sequential
carried out before the images acquisition. capture of images in the farm. The tripod was adapted on top
The proposed experiment for determining thermal emissiv- of the robot.
ity of sweet orange fruit is based on experiments proposed
by [10] and [2]. The method used on this research is based
on a adhesive marker in the fruit surface as object emitter
of reference. For this initial study images were acquired every
two minutes in a closed environment with a controlled temper-
ature of 19 °C. Thermal variations of the acquired images were
analyzed using dedicated software for processing of thermal
images supplied by the manufacturer. Furthermore, a room
temperature sensor enabled a finer control of the variables that
affect the temperature measurement using thermal cameras.To
determine emissivity, the samples were placed on a flat surface
approximately 0.35m from the camera vertically. Over the fruit
sample, an isolating tape tab (3M Scotch 33+) was placed,
whose emissivity is 0.95 [2]. This experimental setup is based
on the emissivity measurement method proposed in [11]. The
images were acquired in periods of two minutes for each of Fig. 2. Platform to images acquisition [12]
 TABLE I
C AMERA PARAMETERS R
RAW =  B
 (3)
Name Type of variable Description  tempR
+F −O
R Integer Constant R
Constant B. ε: Emissivity value of the object
B Float
Range: 1300-1600 tempR : Reflected temperature
Constant F.
F Float
Range: 0.5-2 2) Contrast enhanced with fuzzy intensification operator:
Off-set Constant O. The algorithm used for the segmentation is based on the
O Float
Only positive values. work proposed by [13]. After acquiring the images, Eq. (4),
S Integer Digital Signal 14bits.
proposed by [2], is used to convert the temperature values
measured in degrees Kelvin to gray levels.

!
The image database was collected in field 111 of Maringá T(i,j) − min T(i,j)
I(i,j) = 255

(4)
Farm, located in Gavião Peixoto - SP, the trees were the variety max T(i,j) − min T(i,j)
Ruby (Citrus sinensis (L.) Osbeck) and they were planted in
Where:
November 2010 (the date of the last harvest was 17-26 June
2014). Each orange tree produces on average 2.5 boxes of 40.8 T(i,j) : Temperature value at position i, j
kilograms per year, which is about 600 fruits. 561.600 images I(i,j) : Gray level intensity for the position i, j
were acquired on days 58 and 133 of year, at different times After obtaining the gray-scale image is used the contrast
of day (morning, afternoon and evening). From this database, improvement method using enhancement operator.
50 images of trees with completely visible fruits in the frame Equation. (5) shows the membership function used to con-
were selected. The recorded video used for the experiments struct this operator for a gray-scale image of size (mxn).
had 98 trees: some of them had no fruit, and some others had −Fe
max (g) − g(mn)

been ripped out because of greening disease. The images were µ(mn) = 1 + (5)
obtained in natural lighting conditions, the relative humidity Fd
and temperature parameters were adjusted for each image. Where:
During the experiments, the distance from the camera to the g: Gray level in the image
tree varied between 1 m and 2 m, to cover a larger area of the Fd : Denominational fuzzifier
tree. Fe : Exponential fuzzifier
The fuzzifiers controls the amount of grayness ambiguity.
C. Image Processing
The values of the membership function defined by Eq. (5)
The image processing methodology has four steps: Temper- that are higher than 0.5 are converted to much higher values
ature to gray-scale image conversion, contrast enhanced with and values lower that 0.5 to much lower, which improves the
an fuzzy intensification operator, image segmentation using contrast in the image. The intensifier operator is shown in Eq.
the minimum fuzzy divergence value and fruit detection using (6) and Eq. (7).
Circle Hough Transform. Below, the details of each step are
2
describe. µ0mn = 2 [µmn ] 0 ≤ µmn ≤ 0.5 (6)
1) Temperature to gray-scale image conversion: The con-
version of the temperature values acquired by the thermal
2
camera to gray levels was implemented using MATrix LABo- µ0mn = 1 − 2 [1 − µmn ] 0.5 ≤ µmn ≤ 1 (7)
ratory (MATLAB). The conversion was carried out using Eq.
After applied the intensification operator is used the inverse
(1) provided by FLIR, where known measuring ranges are
function shown in Eq. (8) to transformed the gray values to
employed; the registers related to gain modes are shown in
spatial domain.
Tab. I. The Temperature value is measured in degrees Kelvin.
− 1
h i
0
B gmn = max(g) − Fd (µ0mn ) Fe − 1 (8)
T emperature = R
(1)
Ln (S−O)+F 3) Segmentation using Fuzzy divergence: For the segmen-
tation, it was used the method proposed by [13], whose
The emissivity is another important parameter for correct
membership function is known as Gamma function. This
measurement of temperature. In Eq. (1), emissivity parameter
function is used to find the values of the member function
is not displayed as a parameter for the conversion, but the S
for each pixel in the image, and to facilitate the calculation
parameter can be changed in Eq. (1) by the Robj parameter
of fuzzy divergence for selection of the appropriate level for
in Eq. (2) when the emissivity ε and the reflected temperature
threshold.
tempR are known. The first step to use this technique is choose the regions
S(1 − ε)RAW number that must be segmented. For each of these regions the
Robj = (2) average gray levels is calculated using the Eq. (9).
ε
 Ptx oranges belonged to a commercial orchard of Ruby orange
f =0 f · count(f ) grafted on Swingle rootstock. Six samples of orange fruit were
mx = Ptx (9)
collected from each tree: two in the cup, two in the lower part
f =0 count(f )
and another two from inside. In total, 240 fruits from 40 orange
Where: trees were randomly selected from a 95 hectare orchard. The
mx : Mean of gray levels of the region x first emissivity measurement was performed on the day of
tx : Threshold to the region x sampling and additional measurements were performed over
f : Gray level value the following two days. The obtained emissivity was 0.94 ∓
count(f): Number of occurrences of the gray level f in 5%; with 95% a confidence interval.
the image
b: Maximum gray level in the image. B. Image processing
The gamma function is used to classify the pixels in the The following are the results for each of the processing
image considering the mean values of each region. The general steps. For the development of the proposed algorithm a
equation for the probability density function of the gamma database of 50 was used to validate the proposed algorithm.
distribution is shown in Eq. (10). This group of images was divided into two subgroups: images
taken in the afternoon and images taken at night. The separa-
 γ−1   tion of the images was performed after observing temperature
x−b
β exp − x−b
β changes related to the period of the day, between the fruits and
f (x) = x ≥ b; γ, β > 0 (10) the other structures of the tree. Morning images were discarded
Γ(γ)
because it was not possible to differentiate objects.
γ: Shape parameter Figure 3 shows that oranges are hardly recognizable in the
b: Location parameter image because in different parts of the tree there is a relative
β: Scale parameter thermal equilibrium. Note that higher temperatures are met in
Γ: Gamma function the areas of tree that are exposed to direct sunlight, the white
For the case of region segmented, the parameters are defined areas representing the highest temperature read by the camera
as: b 6= 0, β = 1 and γ = 1. Then, the Eq. (10) take the form correspond to the ground.
shown in Eq. (11).

f (x) = exp(−(x − b)) as Γ(1) = 1 (11)

To perform the region segmentation, a relation between each
pixel value and the regions average gray levels is defined. The
membership function of each pixel depends on its proximity
to a region. The relationship between the pixel and its closest
region is determined by the absolute distance between the pixel
and the the region average gray level.
The fuzzy divergence is used to select the optimum thresh- Fig. 3. Colorful and thermal images of the same tree obtained in the morning
10am to 12pm
old value . Eq. (12) shows the total fuzzy divergence between
A and B. A is the original image MxM and B is an ideally
Figures 4 (a), (b) and (c) show some examples of the images
segmented image MxM.
used to test the contrast-enhanced algorithm. The image (a)
was acquired in the afternoon and images (b) and (c) were
M −1 M −1 
2 − (2 − µA (aij )) · eµA (aij )−1 acquired in the evening. The enhanced-contrast images are
X X 
D(A, B) = shown in Fig. 4 (d), (e) and (f).
−µA (aij ) · e1−µA (aij )
i=0 j=0
(12) Figure 5 shows the result of the contrast-enhanced process,
and the histograms before and after this pre-processing step.
µA (aij ): Membership function of (i,j)th pixel in the The image (a) corresponds to the original image and (b) is
image A its histogram; the image (c) is the image after being pre-
The divergence between the real image and the ideal image processed and (d) its histogram. It can be seen that the
is performed for all possible threshold values, the threshold contrast-enhanced process produced a uniformly distributed
with lowest divergence is chosen as the optimal threshold. histogram in different gray levels, Indeed, a good contrast-
III. R ESULTS enhanced algorithm decreases the difference between the most
and least bright pixels.
A. Emissivity determination Figure 6 (a), (b) and (c) show some examples of the images
For this experimental stage, orange fruits were harvested after applied the contrast-enhanced algorithm and Fig. 6 (d),
to be used as samples. The harvest was carried out in Mar- (e) and (f) show fruit segmentation results. It is important
ingá Farm owned by CITROSUCO company. The harvested to note that some parts of the tree trunk still remain in the
Fig. 4. Contrast-enhanced using the intensification operator. (a), (b) and (c) Fig. 6. Results obtained with the segmentation algorithm. (a), (b) and (c)
show original images, and (d), (e) and (f) show result of the application of show some examples of the images after the contrast-enhanced algorithm and
contrast-enhanced method. (d), (e) and (f) shows fruit segmentation results

Fig. 7. Results obtained with the fruit recognition algorithm.

Confusion matrices were built for data validation and three

metrics were used to validate the result: Accuracy Eq. (13),
positive reliability Eq. (14) and sensitivity Eq. (15) .
The study was divided into two groups: isolated fruits (those
fruits that appear isolated on the images) and the occluded
fruits are those where there is a grouping of various fruits.
TP + TN
Accuracy = (13)
TP + TN + FP + FN
Fig. 5. Enhancement contrast histogram. (a) and (b) show respectively the TP
original image and its histogram. (c) and (d) show enhances contrast image P ositive Reliability = (14)
and its histogram. TP + FP
TP
Sensitivity = (15)
processed image even after it was segmented. This is because TP + FN
after an analysis of temperature variation, it was found that We performed a manual count on the 50 images and the
these parts were in a similar temperature and due to the camera quantities are shown in Tab. II.
sensor sensitivity, it was not possible to obtain a significant
difference between them. TABLE II
To detect and count the fruits, the circular Hough transform T OTAL FRUITS ON THE IMAGES
was applied to the segmented images. The choice of this Real
method of circle detection was based physical orange fruits Isolated fruits 150
characteristics. Fig. 7 shows the results for the sample images. Occluded fruits 275
Total fruit 425
Manual counting was performed to validate the results. It
is important to note that some of the fruits were occluded,
thus the camera cannot detect them. Fruit number count Tab. III and Tab. IV illustrate the confusion matrix of the
corresponds to the number of centroids of circles recognized classification results for each of the groups are presented.
by the Hough transform; this value is stored in a vector and Table. III shows the recognition of isolated fruits. In this case
is then used as the amount of fruit recognized. the overall accuracy was 93.5%, positive reliability was 97.9%,
and the sensitivity was 95.3%. Table. IV shows the recognition R EFERENCES
of isolated fruits. In this case, the overall accuracy was 80%, [1] M. F. Neves, V. G. Trombin, P. Milan, F. F. Lopes, F. Cressoni,
the positive reliability was 87.6%, and sensitivity was 90.2%. and R. Kalaki, “O retrato da citricultura brasileira,” Ribeirão Preto:
CitrusBR, 2010.
[2] D. Bulanon, T. Burks, and V. Alchanatis, “Study on temporal variation
TABLE III in citrus canopy using thermal imaging for citrus fruit detection,”
C ONFUSION MATRIX FOR ISOLATED FRUITS RECOGNITION Biosystems Engineering, vol. 101, no. 2, pp. 161–171, 2008.
[3] ——, “Image fusion of visible and thermal images for fruit detection,”
Current class (expectation) Biosystems Engineering, vol. 103, no. 1, pp. 12–22, 2009.
143 3 [4] M. A. Jiménez-Bello, C. Ballester, J. Castel, and D. Intrigliolo, “De-
Predicted class
True positive False positive velopment and validation of an automatic thermal imaging process for
(Observation)
7 0 assessing plant water status,” Agricultural Water Management, vol. 98,
False negative True negative no. 10, pp. 1497–1504, 2011.
[5] P. J. Zarco-Tejada, V. González-Dugo, and J. A. Berni, “Fluorescence,
temperature and narrow-band indices acquired from a uav platform for
water stress detection using a micro-hyperspectral imager and a thermal
camera,” Remote Sensing of Environment, vol. 117, pp. 322–337, 2012.
TABLE IV
[6] V. Gonzalez-Dugo, P. Zarco-Tejada, E. Nicolás, P. A. Nortes, J. Alarcón,
C ONFUSION MATRIX FOR ISOLATED FRUITS RECOGNITION
D. S. Intrigliolo, and E. Fereres, “Using high resolution uav thermal
imagery to assess the variability in the water status of five fruit tree
Current class (expectation)
species within a commercial orchard,” Precision Agriculture, vol. 14,
248 35
Predicted class no. 6, pp. 660–678, 2013.
True positive False positive
(Observation) [7] I. F. Garcı́a-Tejero, V. H. Durán-Zuazo, J. L. Muriel-Fernández, and J. A.
27 0
Jiménez-Bocanegra, “Linking canopy temperature and trunk diameter
False negative True negative result
fluctuations with other physiological water status tools for water stress
management in citrus orchards,” Functional Plant Biology, vol. 38, no. 2,
pp. 106–117, 2011.
Some of the lost objects make reference to those fruits that [8] C. Ballester, J. Castel, M. A. Jiménez-Bello, J. Castel, and D. Intrigliolo,
are outside the average radius established for recognition. Be- “Thermographic measurement of canopy temperature is a useful tool
for predicting water deficit effects on fruit weight in citrus trees,”
cause in a tree it is possible to find fruits of different flowering, Agricultural water management, vol. 122, pp. 1–6, 2013.
size uniformity is not guaranteed. Some of unexpected data [9] C. Ballester, M. A. Jiménez-Bello, J. R. Castel, and D. S. Intrigliolo,
make reference to those images where tree trunk portion is “Usefulness of thermography for plant water stress detection in citrus
and persimmon trees,” Agricultural and Forest Meteorology, vol. 168,
erroneously recognized as a fruit. pp. 120–129, 2013.
[10] H. J. Hellebrand, H. Beuche, and M. Linke, “Determination of thermal
IV. C ONCLUSIONS emissivity and surface temperature distribution of horticultural prod-
ucts,” in Sixth International Symposium on Fruit, Nut and Vegetable
The night was the best time of day to make the image Production Engineering, Potsdam, Germany, 2001.
acquisition because the temperature difference between the [11] M. Vollmer and K.-P. Möllmann, Infrared thermal imaging: fundamen-
tree of components was higher than at other times of the tals, research and applications. John Wiley & Sons, 2017.
[12] J. F. Archila Diaz, I. L. Argote Pedraza, J. L. Saavedra Guerra,
day, that according to the literature due to the fact that the D. M. B. P. Milori, D. V. Magalhães, and M. Becker, “Mirã rover
fruit contain a larger amount of liquid which take longer to characterization,” in Proceedings, 2015.
reach thermal equilibrium with the rest of the environment. [13] T. Chaira and A. K. Ray, “Segmentation using fuzzy divergence,” Pattern
Recognition Letters, vol. 24, no. 12, pp. 1837–1844, 2003.
The use of pre-processing image and fuzzy techniques show
that it is possible to use simple techniques for fruit recognition
and obtain satisfactory results used the type of technology
of thermal image used for the study. The segmentation step
statistics showed that the algorithm had a 64% satisfactory
results, 20% of images had a segmentation error and 16%
showed other errors type. This possibly happened due to
the temperature difference between the fruit and other parts
of the tree not being large enough to be distinguished as
gray levels. The images obtained in the evening, due to
environmental temperature differences and fruit, are the best
to apply the images proposed processing techniques. Finally,
the fruit counting algorithm has been divided into two groups;
isolated and grouped fruits. For the group of isolates fruits
was obtained an accuracy of 93.5% and 80% grouped fruits
with a database of 50 pictures.

ACKNOWLEDGMENT
The authors express their acknowledgment to CNPq,
CAPES, EMBRAPA, CITROSUCO and EESC-USP for sup-
porting the research project and this article.