sensors

Article
Lettuce Production in Intelligent Greenhouses—3D Imaging
and Computer Vision for Plant Spacing Decisions
Anna Selini Petropoulou * , Bart van Marrewijk , Feije de Zwart , Anne Elings , Monique Bijlaard,
Tim van Daalen, Guido Jansen and Silke Hemming

Business Unit Greenhouse Horticulture, Wageningen University & Research (WUR),

6708 PB Wageningen, The Netherlands
* Correspondence: [email protected]

Abstract: Recent studies indicate that food demand will increase by 35–56% over the period 2010–2050
due to population increase, economic development, and urbanization. Greenhouse systems allow
for the sustainable intensification of food production with demonstrated high crop production per
cultivation area. Breakthroughs in resource-efficient fresh food production merging horticultural and
AI expertise take place with the international competition “Autonomous Greenhouse Challenge”. This
paper describes and analyzes the results of the third edition of this competition. The competition’s
goal is the realization of the highest net profit in fully autonomous lettuce production. Two cultivation
cycles were conducted in six high-tech greenhouse compartments with operational greenhouse
decision-making realized at a distance and individually by algorithms of international participating
teams. Algorithms were developed based on time series sensor data of the greenhouse climate and
crop images. High crop yield and quality, short growing cycles, and low use of resources such as
energy for heating, electricity for artificial light, and CO2 were decisive in realizing the competition’s
goal. The results highlight the importance of plant spacing and the moment of harvest decisions in
promoting high crop growth rates while optimizing greenhouse occupation and resource use. In this
paper, images taken with depth cameras (RealSense) for each greenhouse were used by computer
Citation: Petropoulou, A.S.; van vision algorithms (Deepabv3+ implemented in detectron2 v0.6) in deciding optimum plant spacing
Marrewijk, B.; de Zwart, F.; Elings, A.; and the moment of harvest. The resulting plant height and coverage could be accurately estimated
Bijlaard, M.; van Daalen, T.; Jansen, with an R2 of 0.976, and a mIoU of 98.2, respectively. These two traits were used to develop a light
G.; Hemming, S. Lettuce Production loss and harvest indicator to support remote decision-making. The light loss indicator could be
in Intelligent Greenhouses—3D
used as a decision tool for timely spacing. Several traits were combined for the harvest indicator,
Imaging and Computer Vision for
ultimately resulting in a fresh weight estimation with a mean absolute error of 22 g. The proposed non-
Plant Spacing Decisions. Sensors 2023,
invasively estimated indicators presented in this article are promising traits to be used towards full
23, 2929. https://doi.org/10.3390/
autonomation of a dynamic commercial lettuce growing environment. Computer vision algorithms
s23062929
act as a catalyst in remote and non-invasive sensing of crop parameters, decisive for automated,
Academic Editor: Yiannis objective, standardized, and data-driven decision making. However, spectral indexes describing
Ampatzidis
lettuces growth and larger datasets than the currently accessible are crucial to address existing
Received: 30 January 2023 shortcomings between academic and industrial production systems that have been encountered in
Revised: 20 February 2023 this work.
Accepted: 27 February 2023
Published: 8 March 2023 Keywords: artificial intelligence; computer vision; sensors; lettuce; indoor farming; autonomous
greenhouses; climate control; plant spacing; remote control; data driven growing

Copyright: © 2023 by the authors.

Licensee MDPI, Basel, Switzerland.
1. Introduction
This article is an open access article
distributed under the terms and Recent studies strongly indicate that food demand will increase by 35–56% over
conditions of the Creative Commons the period of 2010–2050 as a result of population increase, economic development, and
Attribution (CC BY) license (https:// urbanization, among other drivers [1]. The expected increase in food demand places
creativecommons.org/licenses/by/ pressure on natural resources and may lead to negative environmental impacts as well
4.0/). as biodiversity losses [2]. Among the possible solutions are the transformation of food

Sensors 2023, 23, 2929. https://doi.org/10.3390/s23062929 https://www.mdpi.com/journal/sensors

Sensors 2023, 23, 2929 2 of 30

production into a green industrial process and the promotion of policies for plant-based
and high-nutrient diets [3].
Greenhouse systems allow sustainable intensification of food production with demon-
strated high crop production per cultivation area [4]. While vegetable production is increas-
ing in area and volume, the number of farms declines, resulting in more vegetable area
and volume per farm and per grower [5]. At the same time, the availability of labor is an
industry-wide challenge as well as the lack of experienced managers and growers in crop
production. Greenhouses are highly dynamic production systems operating through an
integrated set of activities performed by growers [6]. Growers need to consider various
performance indicators such as yield, quality, timing, and sustainability standards and meet
the volatile market demands, and prices in uncertain environmental conditions subject to
weather conditions, for example [7].
Modern horticultural production is highly dependent on up-to-date information on
farm operations. Production processes are already highly automated and controlled [8].
Information systems driven by the rapid developments in cloud computing, the Internet
of Things, Big Data, machine learning, augmented reality, and robotics are changing the
horticulture horizon toward precision horticulture [9–12]. Digital technologies, computa-
tional power, and high-fidelity sensors act as catalysts in the transition toward advanced
and autonomous production systems. Non-invasive, near real-time data and information
with high spatial and temporal resolution create opportunities for advisory or automated
decision software and the design of advanced models, known as digital twins [13]. Moni-
toring and interpretation of the system’s dynamics at coarser and granular levels allow for
location-specific operations to ascertain desired conditions that meet crop demands.
Digital twins are equivalent to real-life objects mimicking the behavior and states
over their lifetime in virtual space [13]. Greenhouse digital twins can be seen as coupled
dynamic climate and crop models representing the actual physical, biological, and inte-
grated technical systems as virtual representations of reality [14]. Digital twins can be used
to simulate the effects of different growing conditions and crop management strategies,
give insights into their effect on performance indicators, and support decision-making [15].
There have been several achievements in the implementation of mechanistic crop and
climate models in horticultural research to facilitate decision making in greenhouse op-
erations [16–18]. Broadly validated dynamic models of the greenhouse climate and crop
include, e.g., KASPRO [19] and INTKAM [20], which have been used for several research
activities. A benchmark experiment in optimizing net profit using AI for the remote con-
trol of cucumber cultivation in 2018 [21] and a follow-up on optimizing the net profit of
tomatoes as a function of yield and quality a year later [22], showed the potential of AI
in controlling and outperforming human decisions by experienced growers. Automated
greenhouse control wasthus demonstrated to be possible; therefore, our next focus was
on the autonomy, robustness, and scalability of such control systems [23]. The goal of the
third edition of the Autonomous Greenhouse Challenge was the full autonomous control
of lettuce cultivation.
Commercial greenhouse production of lettuce (Lactuca sativa L.) is already highly
automated. Lettuce is grown in controlled greenhouse environments including hydroponic,
aquaponic, and vertical growing systems. The systems minimize labor requirements
by using conveyor belts and lifts throughout the growing processes from seedling to
harvesting [24]. Lettuce hydroponic systems include Nutrient Film Techniques (NFT), Deep
Flow Techniques (DFT), as well as Ebb and Flow systems. NFTs are the most widespread
method of recirculating nutrient solution systems [25] and employ a shallow stream of
water with dissolved nutrients flowing over the roots of plants in water-tight gullies, here
referred to as gutters. The nutrient solution is initially stored in a reservoir, pumped out
into the gutters at an angle, and drained to a tank for filtering before re-cycling to the
reservoir for re-use. Gutters are automatically filled with the growing media and lettuce
heads and transported on conveyor belts to the main greenhouse area. When lettuce heads
are fully grown, they are moved toward the harvesting area. At the harvesting area, cutting
Sensors 2023, 23, 2929 3 of 30

machines remove the plants from the gutters and transfer the lettuce heads for packaging
while the gutters are washed, and the process starts again. During the growing period in the
greenhouse, the distance between the gutters and crops on the gutters largely determines
the required amount of greenhouse space and, therefore, resource use. From the perspective
of greenhouse automation, it is important to note that the automated optimization of lettuce
plant spacing is not yet implemented in practice.
Optimal cultivation temperatures for lettuce are relatively low and range from 15.5 ◦ C
to 28 ◦ C during the daytime to 3 ◦ C to 12 ◦ C at night time [26]. The optimal pH ranges
for the nutrient solution from 5.8 to 6.5 and its optimal electrical conductivity (EC) should
be 1.5 mS/cm [27]. A wide variety of crop types can be distinguished among the existing
lettuce cultivars, with crisp head and butterhead commonly grown in the United States and
Western Europe, respectively, whereas Romaine and loose-leaf types are mainly cultivated
in Mediterranean areas [28]. The crop is susceptible to physiological problems including
outer leaf tip burn, inner tip burn, and discoloration of ribs [29]. Growth of lettuces, as with
any crop, is related to incident radiation and CO2 concentration, and due to the relatively
high surface area to volume ratio, has high transpiration rates [29]. A fully autonomous
decision of optimum climate setpoints can contribute to better crop growth and lower
resource use.
Since plant spacing is an important criterion for good vegetative growth on an m2
basis, it is a major aspect of yield maximization. Densely planted lettuces can obstruct
morphological characteristics such as head size, leaf expansion and color, and compact-
ness [30–32]. Wider spacing ensures higher light availability per head and that nutritional
requirements are satisfied; however, this comes at the expense of less efficient utilization of
the growing area and resources used. Optimum plant spacing is a management decision in
hydroponic lettuce cultivation that can potentially be determined using 3D camera images
and other sensor data, together with artificial intelligence algorithms to fully automate the
operational process.
Modern camera systems and innovative artificial intelligence (AI) technologies such
as computer vision allow objective, non-invasive, and continuous data for precision horti-
culture applications [33]. Advances in machine learning for image processing have resulted
in a wide range of research and applications for crop monitoring [34]. Applications of
computer vision can be found in the fields of pests, disease or weed detection [35–37],
fruit and flower detection, counting and fruit ripeness [38,39], crop stress detection [40],
yield estimation, or moment of harvesting [41,42]. Moving cameras or flying drones with
mounted cameras scan plants from various viewpoints, addressing matters of occlusion
and creating 3D representations of the crop [43]. High-resolution imaging in combination
with deep learning techniques is expected to have great potential for precision farming and
remote control operations for purposes of autonomous greenhouses [44].
Traditional computer vision techniques struggle with the challenging greenhouses en-
vironment because of varying environmental conditions. Light conditions are continuously
changing, and occlusion makes it difficult to identify individual plants or plant organs [45].
The development of hand-crafted algorithms was often time-consuming and not reliable
enough. However, recent development in the field of deep learning made it easier to
use vision systems in greenhouses. High classification accuracies of up to 99.7% [46] on
large plant datasets such as the “Oxford-Flowers102” [47] dataset show the power of deep
learning for plant phenotyping. Already in 2017, the first paper appeared on the quality
assessment of lettuce using artificial neural networks [48]. Lettuce was binarily classified
as good” or “reject”. Although the algorithm was not complex as it had only two layers,
it was one of the first publications that showed the possibility of using neural networks
for lettuce classification. The ability of networks to learn plant features from single lettuce
images can be determined by the recently published lettuce dataset [49]. At the moment,
three papers have been published, obtaining high accuracy to estimate fresh weight from
the images with a Root Mean Squared Error (RMSE) up to 25.3 g [50–52].
Sensors 2023, 23, 2929 4 of 30

The above-mentioned examples are only focusing single lettuce images. With the
development of instance segmentation algorithms it is possible to determine the growth
rate of lettuce over time by extracting the leaf area of single lettuce plants, as seen in [53],
their experiments were in a semi-commercial setting without overlapping lettuces. A
more commercial example can be found in [54], in which aerial images were collected and
the number of lettuces was determined including a size estimation into three different
categories. One of the conclusions was that despite the fact that many individual lettuces
can be detected, there is still a gap between object detection and trait measurements [54].
In greenhouses, the environmental conditions are much better for high-quality imaging, re-
ducing the AI and trait measurement gap. Other researchers developed a high-throughput
system for individual plant phenotyping of lettuce [55]. Each lettuce head was placed in
an individual pot; by detecting the pot and by applying semantic segmentation, many
plant traits were calculated including projected area and perimeter. The area and size are
two of the most interesting growth indicators. However, when the leaves became larger
than the pot size, prediction accuracy decreased; as a result, the growth curves were only
accurate for the first weeks. It can be noticed that most experiments were carried out in
semi-commercial conditions. When the leaves were overlapping, either the experiment
stopped, or the extraction of the parameters was removed. Next to that, in each experiment,
the interpretability of the results was difficult. There is still a mismatch between object
detection, determining plant traits, and more importantly, what a grower should do with
the provided information. If AI can extract growth rate, how should a grower use this
information to improve the cultivation? Therefore, more advanced methods are needed
that can extract information in greenhouses and conform to commercial practice while
maintaining interpretability.
This paper describes the results of the third Autonomous Greenhouse Challenge, an
experiment in which the autonomous control of lettuce production has been realized in
six different greenhouse compartments, each controlled by AI algorithms developed by
participating teams. During the experiment, the goal was to decide upon climate and
crop management strategies to optimize the net profit of lettuce production, considering
yield and product prices, resource use, and costs including greenhouse occupation. The
experiment provided valuable public datasets which can be used for future AI training
purposes, and which can be found under the Data Availability Statement. In this paper,
we give an overall analysis of the results obtained by the teams. Next to that, we focus on
the research question, of how computer vision and deep learning algorithms can be used
for automated operational decisions of lettuce greenhouse production, as currently plant
spacing and harvesting are determined on fixed schedules since transplanting. Furthermore,
we examine how better utilization of the occupied growing area, efficient resource use that
meets crop growing demands, and timely planning of harvest events can be supported by
non-invasively estimated indicators such as the proposed light loss and harvest indicator.
Results of other studies focus on answering similar questions on crop trait detection with
computer vision in highly controlled and steady environmental conditions. This research
realizes steps closer to commercial practice by processing smaller datasets of canopy images
under varying environmental conditions.

2. Materials and Methods

This paper describes different steps of the realized research methodology, and an
overview is given in Figure 1. In the preparation phase teams developed their own AI
algorithms based on provided annotated single lettuce images and climate data time
series from a climate and crop simulator [49]. After this preparation phase, two lettuce-
growing experiments were conducted in greenhouses at Wageningen University & Research
(Section 2.1. Greenhouse compartments and equipment Section 2.2. Crop and Section 2.3
Greenhouse climate and crop control). During the first greenhouse experiment, teams
could gain experience in controlling the lettuce growth based on real-time data from
the greenhouse (Section 2.4 Data communication, Section 2.5 Remote sensing, and data
Sensors 2023, 23, x FOR PEER REVIEW 5 of 30

Section 2.3 Greenhouse climate and crop control). During the first greenhouse experiment,
Sensors 2023, 23, 2929 teams could gain experience in controlling the lettuce growth based on real-time data 5 of from
30
the greenhouse (Section 2.4 Data communication, Section 2.5 Remote sensing, and data
collection). New annotated images, of the full crop under the camera’s field of view and
climate time series
collection). data wereimages,
New annotated collected
of [56]. Teams
the full cropcould
underrefine their algorithms
the camera’s before
field of view
the second greenhouse experiment. Another set of annotated images,
and climate time series data were collected [56]. Teams could refine their algorithms full crop canopy
images,
before and climategreenhouse
the second times seriesexperiment.
were collected [56]. After
Another set ofthe experiments,
annotated images,an full
analysis
crop of
climate,
canopy crop,
images,and resource
and climate use
timeswas made
series wereand given in
collected this
[56]. paper
After the (Section 3. Results
experiments, an
Sections
analysis3.1–3.3). Ancrop,
of climate, additional analysis
and resource use ofwas
plant spacing
made decisions
and given was
in this made
paper (Section
(Section 3. 3.
Results Section 3.4) based on different image processing methods (Section 2.6 Image
Results Sections 3.1–3.3). An additional analysis of plant spacing decisions was made
processing
(Section 3. for plant
Results spacing
Section decisions).
3.4) based The results
on different are discussed
image processing and (Section
methods concluded 2.6 in
Sections 4 and 5. for plant spacing decisions). The results are discussed and concluded in
Image processing
Sections 4 and 5.

Preparation Image processing

First greenhouse
(Crop spacing/moment
growing experiment
of harvest)

Provide teams with Annotated full

annotated single lettuce crop-canopy images Results & discussion
images and Climate time series
climate time series
Data analysis (climate,
Refinement of
crop, resource use)
algorithms by teams
Climate & Crop
Simulator
(Kaspro/Intkam) Second remote
cultivation by teams

Annotated full
Teams Develop
crop-canopy images
algorithms
Climate time series

Figure
Figure1.1.Research
Researchmethodology
methodology of the growing
of the growingexperiments
experimentsand
andanalysis
analysisof of results.
results. Data
Data fromfrom
thethe
annotated single lettuce images is found in [49], whereas the annotated full crop data are found in
annotated single lettuce images is found in [49], whereas the annotated full crop data are found
[56].
in [56].

2.1. Greenhouse Compartments and Equipment

2.1. Greenhouse Compartments and Equipment
Each greenhouse compartment at the research facility of Wageningen University &
Each greenhouse
Research in Bleiswijk,compartment
The Netherlands, at the research
had facility
a size of 96 m2of Wageningen
. The University
compartments were &
Research
equippedinwith Bleiswijk,
standard The Netherlands,
actuators had ainsize
also available of 96 m2high-tech
commercial . The compartments
greenhouses wereas
equipped
shown inwith standard
Figure actuators
1. A pipe-rail also available
heating in commercial
on the floor with a peakhigh-tech
capacity ofgreenhouses
120 W/m2 as
shown in Figure
controlled by the1. A pipe-rail
heating heating
temperature on the floor
setpoints, with aroof
continuous peak capacity(ventilation
ventilation of 120 W/m2
controlled
area of 0.3by m the
2
heating
opening pertemperature
2
m greenhouse setpoints, continuous
area, equipped withroof ventilation
anti-thrips (ventilation
netting), two
area
typesof of
0.3inside
m2 opening
moveable per m2 greenhouse
screens (LUXOUS 1547 area,Dequipped
FR energywith
screenanti-thrips
and OBSCURAnetting),
9950two
types
FR Woflight
inside moveable
blocking screens
screen, (LUXOUS
Ludvig 1547white
Svensson), D FR LED
energy screenlights
artificial and OBSCURA
of dimmable 9950
2 /s and efficiency
FR W light blocking screen, Ludvig Svensson), white LED artificial lights of dimmable
intensity controlled in a continuous range between 27 and 270 µmol/m
of 2.4 µmol/J,
intensity controlled(VYPRin a2p, Fluence byrange
continuous Osram), a fogging
between system
27 and (maximum
270 µmol/m capacity
2/s and of
efficiency
330 g/m2 /h), and CO2 supply (maximum capacity 15 g/m2 /h) were available. Plants
of 2.4 µmol/J, (VYPR 2p, Fluence by Osram), a fogging system (maximum capacity of 330
were grown in soil-pressed pots on NFT hydroponic gutters (Hortiplan, Belgium) placed
g/m2/h), and CO2 supply (maximum capacity 15 g/m2/h) were available. Plants were
on an inclination. A recirculating water system was supplying water and nutrients via
grown in soil-pressed pots on NFT hydroponic gutters (Hortiplan, Belgium) placed on an
pressure-compensated narrow tubes injecting water into the gutters.
inclination. A recirculating
The experiment of thewater system was supplying
third Autonomous Greenhouse water and nutrients
Challenge via pressure-
was conducted in
compensated narrow tubes injecting water into the gutters.
the first half of 2022 in six different high-tech Venlo-type greenhouses compartments of
The experiment
Wageningen Universityof the third Autonomous
& Research, in Bleiswijk,Greenhouse Challenge
The Netherlands. was greenhouse
The basic conducted in
the first half ofand
construction 2022 in six different
equipment high-tech
with actuators as Venlo-type greenhouses
well as the standard compartments
sensors and control of
of the greenhouse compartments were identical to the elements which can be found in
commercial greenhouses (see Section 2.2 Crop and Section 2.3 Greenhouse climate and crop
 Wageningen University & Research, in Bleiswijk, The Netherlands. The basic greenhouse
construction and equipment with actuators as well as the standard sensors and control of
Sensors 2023, 23, 2929 the greenhouse compartments were identical to the elements which can be found 6 of in
30
commercial greenhouses (see Section 2.2 Crop and Section 2.3 Greenhouse climate and
crop control). However, the greenhouse compartment size was much smaller than in
commercial
control). practice.
However, theDifferent teams
greenhouse (CVA, DigitalCucumbers,
compartment Koala,
size was much smaller MondayLettuce,
than in commercial
VeggieMight) and a reference were controlling the six compartments.
practice. Different teams (CVA, DigitalCucumbers, Koala, MondayLettuce, VeggieMight)
and a reference were controlling the six compartments.
2.2. Crop
Two cultivation cycles of lettuce cv. “Lugano” (Rijk Zwaan, The Netherlands) were
2.2. Crop
conducted in 6 equalcycles
Two cultivation greenhouse
of lettuce growing compartments.
cv. “Lugano” (Rijk Zwaan, Lettuces were grown were
The Netherlands) in a
hydroponicin NFT
conducted 6 equal system. Seedsgrowing
greenhouse were propagated
compartments. to Lettuces
seedlingswere8 weeks
grown before the
in a hydro-
transplanting
ponic NFT system.date. The
Seedsyoung
wereplants were grown
propagated in cubes
to seedlings of compacted
8 weeks peat.
before the On the days
transplanting
of transplanting
date. (2 February
The young plants and 3 May
were grown 2022,
in cubes of respectively),
compacted peat. theOn
seedlings
the dayswere placed in
of transplant-
the greenhouse
ing (2 February and compartments
3 May 2022,inrespectively),
small holes of theslightly tilted
seedlings weregutters to which
placed water with
in the greenhouse
nutrients was supplied
compartments in small at a certain
holes frequency.
of slightly tilted gutters to which water with nutrients was
The lettuces were grown in 3.2 m plastic gutters, all having 30 plant holes, with an 11
supplied at a certain frequency.
cm heart-to-heart
The lettuces weredistance. Thein
grown gutters
3.2 mwere 10 gutters,
plastic cm wide,all sohaving
the maximum
30 plantplant density
holes, with
was1192cm
an (rounded) plantsdistance.
heart-to-heart per m2 inThe the gutters
initial stage.
were Lettuces
10 cm wide,weresogrown in two rows
the maximum of
plant
such gutters
density was as92 depicted
(rounded) in plants
Figure per m2 in
1. Plant appearance, pests, and
the initial stage. diseases
Lettuces werewere monitored
grown in two
weekly
rows by experts
of such gutterswithout interfering
as depicted in Figure with any appearance,
1. Plant operational pests,
controlanddecisions
diseasesinwere
the
monitored weekly by experts without interfering with any operational
compartments. Irrigation and nutrient recipes were determined by the experienced control decisions
in the compartments.
greenhouse staff of theIrrigation
Bleiswijkand nutrient
Research recipes were determined by the experienced
Center.
greenhouse staff of the Bleiswijk Research Center.
Leafy vegetables are sellable to retail at a particular weight and shape. The lettuce
heads Leafy vegetables
in the are sellable
area of evaluation to retail
(Figure at a particular
2) were classified weight and shape.
at the moment The lettuce
of harvest into
heads in the area of evaluation (Figure 2) were classified at the moment
three categories. Class A were sellable lettuces with a minimum average weight of 250 g, of harvest into
three
Class categories.
B were lettucesClasswithA were sellable
a weight lettuces
between 220with
anda250
minimum average
g, and Class weight
C were of 250 g,
non-sellable
Class B were lettuces with a weight between 220 and 250 g, and Class C were non-sellable
lettuce heads that were underweighted and or showed visible deformations.
lettuce heads that were underweighted and or showed visible deformations. Malformations
Malformations referred to quality aspects related to the shape of the plant and defects of
referred to quality aspects related to the shape of the plant and defects of the leaves (e.g., leaf
the leaves (e.g., leaf discoloration, leaf rotting, and diseases).
discoloration, leaf rotting, and diseases).

(a) (b)
Figure 2.
Figure 2. (a)
(a) Cross-
Cross-andand(b)
(b)top-view
top-viewsections
sectionsofofone
onegreenhouse
greenhouseexperimental
experimentalcompartment
compartmentwith 96
with
m2 ground
2 floor. (a) Compartment with crop and actuators: rail pipe, irrigation system, NFT gutters,
96 m ground floor. (a) Compartment with crop and actuators: rail pipe, irrigation system, NFT
CO2 supply, LED artificial light, and two screens. (b) Arrangement of lettuce gutters. Green boxes
gutters, CO2 supply, LED artificial light, and two screens. (b) Arrangement of lettuce gutters. Green
represent the harvest area for data analysis.
boxes represent the harvest area for data analysis.

2.3. Greenhouse Climate and Crop Control

Strategic and operational climate control was carried out by participating teams of
the third Autonomous Greenhouse Challenge. Strategic decisions include, e.g., the use
Sensors 2023, 23, 2929 7 of 30

(installation) of screens or artificial lighting or the starting density of the crops, operational
decisions included, e.g., the timing and amount of screen or lighting hours or crop spacing
decisions. The AI algorithms of the teams were determining control setpoints of the heating
temperature, CO2 concentration, humidity deficit, lighting intensity, operation of the black-
out, and energy screens, as well as leeward and windward ventilation. The mechanistic
climate and lettuce crop models of WUR (KASPRO and INTKAM, respectively) could be
used by the teams as a training environment for the algorithms before the start of each
cultivation cycle.
Resource use was calculated based on measured data: heating energy (MJ/m2 ) with
a price of 0.0375 EUR/kWh, electricity (kWh/m2 ) with a price of 0.125 EUR/kWh for
the on-peak hours (07:00–23:00), and 0.075 EUR/kWh for the off-peak hours, CO2 use
(kg CO2 /m2 ) with a price of 0.12 EUR/kg.
As in commercial practice, the spacing system allows for several plant densities;
densities could be reduced from the starting density of 92 heads per m2 via a density of 60,
45, 30, 23, and 18, to the lowest density of 15 lettuce heads per m2 . The teams’ algorithms
had to automatically make the spacing decisions.
The following prices per lettuce head were given Class A = 0.50 EUR/head, Class
B = 0.40 EUR/head, and Class C = 0.00 EUR/head. In commercial practice, harvested
lettuce heads are sold per head, but as in reality, the economics of the greenhouse is
eventually expressed in resource usage and production per average m2 of the growing area.
Therefore, the price of the lettuce was multiplied by the average number of heads per m2
of the growing area. The formula to calculate the average lettuce crop density (heads/m2 )
is the following:

D
AverageCropDensity = (1)
∑dD=1 density
1
d

where D is the total number of days since transplanting until harvest and densityd is the
plant density at day d.
Teams had to maximize net profit. Net profit was calculated from income minus
costs. Income was determined by multiplying the yield with the price per class. The
total costs consisted of fixed and variable components associated with the greenhouse
operation. On top of that teams were ‘charged’ for every manual intervention on their
autonomous algorithm (EUR 1 per intervention). This penalty was meant to strongly
discourage such interventions ensuring that the algorithms would work as autonomously
as possible. Fixed costs accounted for the plant material, maintenance, and depreciation
costs of the greenhouse equipment. The variable costs accounted for the resource use
(electricity for artificial lighting, energy for heating, and CO2 injection).

2.4. Data Communication

Data communication between the underlying systems was vital to ensure a stable,
uninterrupted integration and operation. In this experiment, an Azure file share was
made available to ensure enough storage capacity for collected datasets. Azure Virtual
Machines—NCasT4_v3-series (VMs) were used for high-performance computing and
deploying AI workloads, such as real-time inferencing of user requests. The infrastructure
supported the communication between the greenhouse climate computer, control systems,
sensing devices, and the state of actuators, measured indoor and outdoor climate (Figure 3)
(Appendix A Table A1). Numerical time-series data of the realized controls, climate, and
additional sensor sensors can be found under the Data Availability Statement.
Sensors 2023,23,
Sensors2023, 23,2929
x FOR PEER REVIEW 88ofof3030

Figure3.3.Data
Figure Datacommunication
communicationstructure
structureData
Dataflows
flowsfrom
fromindoor
indoorand
andoutdoor
outdoorclimate
climateand
andadditional
additional
sensors to the virtual machines and the online database. Decisions of algorithms of teams are written
sensors to the virtual machines and the online database. Decisions of algorithms of teams are written
from the Virtual Machines to the online database from where another communication protocol
from the Virtual Machines to the online database from where another communication protocol writes
writes the controls to the greenhouse climate computer before their implementation in the actual
the controls tocompartments.
greenhouse the greenhouse Greenhouse
climate computer
staff before their
receives implementation
decisions in the
from online actual greenhouse
database for spacing
and moment of harvest.
compartments. Greenhouse staff receives decisions from online database for spacing and moment
of harvest.
2.5. Remote Sensing and Data Collection
2.5. Remote Sensing and Data Collection
In each greenhouse compartment, standard sensors were made available,
In each greenhouse compartment, standard sensors were made available, comparable
comparable to earlier experiments described in [21,22]. These consist of an outside
to earlier experiments described in [21,22]. These consist of an outside weather station, ob-
weather station, obtained weather forecast, and indoor climate parameters (temperature,
tained weather forecast, and indoor climate parameters (temperature, relative air humidity,
relative air humidity, PAR light, CO2) along with the status of all actuators (heating,
PAR light, CO2 ) along with the status of all actuators (heating, fogging, lighting, screening,
fogging,
CO lighting, screening, CO2) in 5 min intervals. The output of the standard sensors
2 ) in 5 min intervals. The output of the standard sensors was continuously available as
input for the teams’available
was continuously as input for the teams’ algorithms.
algorithms.
In commercial production,
In commercial production, lettuce lettucetraits
traitsare
areseldom
seldomcollected
collectedduring
duringthe thegrowing
growing
cycle and crop performance is evaluated by growers’ visual
cycle and crop performance is evaluated by growers’ visual inspections only. In inspections only. Inthis
this
experiment,RealSense
experiment, RealSenseD415
D415[57][57]cameras
cameraswere
werehung
hung11m mabove
abovethe thegrowing
growingcrop cropininthe
the
area of evaluation. The camera uses stereo vision and stores depth, RGB,
area of evaluation. The camera uses stereo vision and stores depth, RGB, and IR images. All and IR images.
All camera
camera parameters,
parameters, both intrinsic
both intrinsic and extrinsic,
and extrinsic, are provided
are provided with thewith the published
published dataset
under the Data Availability Statement. These parameters could be used totoconvert
dataset under the Supplementary Material. These parameters could be used convertthethe
images to point clouds. Images were taken every 15 min in each compartment
images to point clouds. Images were taken every 15 min in each compartment during the during the
cultivationcycles.
cultivation cycles.
Periodicdestructive
Periodic destructiveharvests
harvestsofofsix
sixplants
plantsper
percompartment
compartmentwere weretaken
takenononthetheday
dayofof
planting and subsequently on a weekly basis Destructive measurements
planting and subsequently on a weekly basis Destructive measurements of plant height, of plant height,
diameter,fresh
diameter, freshweight,
weight,and
anddrydryweight,
weight,andandscores
scoresfor
forleaf
leafdeformation
deformationdue duetotoouter
outerleaf
leaf
tip burn were carried out. The individual lettuce plants were taken from
tip burn were carried out. The individual lettuce plants were taken from the right and left the right and left
sideofofeach
side eachcompartment
compartmentas asshown
shownininFigure
Figure2b.
2b.Next
Nexttotothat,
that,images
imagesofofthe theindividual
individual
plantswere
plants weremade
madeeacheach1515min;
min;an anexample
examplecancanbebefound
foundinin(Figure
(Figure4).4).
Sensors 2023,
Sensors 23,23,
2023, x FOR
2929PEER REVIEW 9 of930
of 30

Figure
Figure4.4.Two
Twoexample
example images
images of RealSense D415
of RealSense D415from
fromthe
theday
dayofofplanting
planting (left)
(left) andand
thethe
dayday
of of
harvest (right). On the day of harvest, plants were sampled from the field of view of the camera
harvest (right). On the day of harvest, plants were sampled from the field of view of the camera and
and
were destructively measured for height, diameter, fresh weight, dry weight, and quality.
were destructively measured for height, diameter, fresh weight, dry weight, and quality.

2.6.
2.6.Image
ImageProcessing
Processingfor for Plant
Plant Spacing Decisions
Spacing Decisions
One
Oneofofthe themain
main research questions isishow
research questions howimages
imagestakentakeninina agreenhouse
greenhouse conform
conform to to
commercial practice and computer vision can be used to determine the optimal spacing
commercial practice and computer vision can be used to determine the optimal spacing
strategy.
strategy.To Todo dothis,
this, the
the images
images of of the
thelettuce
lettucecropcropininthethegreenhouse
greenhouse needneed to to
bebe related
related to to
a relevant crop variable, such as crop growth rate. From the
a relevant crop variable, such as crop growth rate. From the time series of images taken time series of images taken
insidethe
inside thegreenhouses
greenhouses (Figure (Figure 4),
4), the
the coverage
coveragecan canbe becalculated
calculated over
over time.
time.The Thecoverage
coverage
can be defined as the area covered with green leaves relative to the total ground surface
can be defined as the area covered with green leaves relative to the total ground surface
area. However, coverage might not be a good indicator to determine plant growth rate, as
area. However, coverage might not be a good indicator to determine plant growth rate, as
plant growth rate may decline once the leaves touch neighboring plants. It can be assumed
plant growth rate may decline once the leaves touch neighboring plants. It can be assumed
that for very high coverage growth is hampered. So, coverage might not be a suitable
that for
parameter very to high
be usedcoverage
for spacinggrowth is hampered.
decisions. Crop volumeSo, coverage might not
on the contrary might bedescribe
a suitable
parameter to be used for spacing decisions. Crop volume
plant growth rate even if the coverage is close to 100%. Crop volume can be estimated on the contrary might describe
by
plant growth
coverage andrate even
height and if can
the coverage
be used toisdetermine
close to 100%. Crop volume
crop growth rates in can time.beThe estimated
volume by
coverage
over timeand is aheight
relevant andcropcanparameter,
be used to determine
however, crop not
it might growth
directlyrates in time.
assess The volume
if spacing was
over timeout
carried is correctly.
a relevant crop parameter,
Another option might however, it might
be to calculate not
light directly assess
interception (or lightif spacing
loss).
was
Thecarried
different out correctly.
methods Another option
of “coverage” (Section might
2.6.2),be“volume
to calculate light interception
over time” (Section 2.6.3), (or
andlight
loss).
“lightThe lossdifferent
over time” methods
(Section of “coverage”
2.6.4) are explained (Section
in the2.6.2), “volume
following sections over
after time” (Section
describing
2.6.3),
how cropand segmentation
“light loss over time”2.6.1)
(Section (Section
from 2.6.4) are explained
greenhouse images is in the following sections
implemented.
after describing how crop segmentation (Section 2.6.1) from greenhouse
For optimizing spacing decisions, the challenge lies in realizing a fast plant imagesgrowth is
rate on
implemented. one hand and limited use of space on the other hand. Early spacing facilitates fast
growth
For rates thus increasing
optimizing spacingyield over time,
decisions, the late spacinglies
challenge facilitates less occupation
in realizing a fast plant of space
growth
thus decreasing resource use. An optimum spacing decision is therefore necessary.
rate on one hand and limited use of space on the other hand. Early spacing facilitates fast
growth
2.6.1. CropratesSegmentation
thus increasing yield over time, late spacing facilitates less occupation of
space thus decreasing resource use. An optimum spacing decision is therefore necessary.
First, the lettuce crop needs to be segmented from the background. The growth of
each lettuce head over time was identified using instance segmentation. However, this is
2.6.1.
onlyCroppossibleSegmentation
in the first 2 weeks. After that, the lettuce heads start to touch each other,
andFirst,
leaves the lettuce crop
overlapped. Asneeds
visibletoinbeFigure
segmented from the
5 it is almost background.
impossible The growth
to identify which of
each
leaveslettuce
belong head over time
to which lettuce was identified
head. Therefore, using instance
semantic segmentation.
segmentation is used However, this is
in this study.
only possible DeepLabv3+,
Specifically, in the first 2 weeks.which isAfter that, the in
implemented lettuce heads (v0.6)
detectron2 start to touch
[58]. For each other,
training
and leaves 23
purposes overlapped.
images have Asbeen
visible in Figures
annotated, 5it is almost
in which each pixel impossible
was either to annotated
identify which as
leaves belong to which lettuce head. Therefore, semantic segmentation is used in this
background or lettuce, an example is shown in Figure 5. All settings for training were kept
the same
study. as in the original
Specifically, DeepLabv3+,implementation
which isdetectron2.
implemented Only inthedetectron2
number of iterations
(v0.6) [58]. wasFor
set to 2500 and the image size was set to 1024 × 1024. For validation purposes, 12 additional
training purposes 23 images have been annotated, in which each pixel was either
images have been annotated. The evaluation was carried out on the validation dataset
annotated as background or lettuce, an example is shown in Figure 5. All settings for
using the mean Intersection over Union (mIoU) metric Equation (2). In which M denotes
training were kept the same as in the original implementation detectron2. Only the
the mask of the class, respectively.
number of iterations was set to 2500 and the image size was set to 1024 × 1024. For
validation purposes, 12 additional MGTlettuce ∩ images
M Lettuce have been annotated.
MGTbackground ∩ Mbackground The evaluation was
 
MGTlettuce ∪ M Lettuce + M ∪ Mbackground
carried out on the mIoUvalidation
= dataset using the GTbackground
mean Intersection over Union (mIoU) (2)
2
metric Equation (2). In which M denotes the mask of the class, respectively.
Sensors 2023,Sensors
23, x FOR PEER
2023, 23, xREVIEW
FOR PEER REVIEW 10 of 30 10 of

Sensors 2023, 23, 2929 10 of 30

𝑀 ∩𝑀
𝑀 ∩𝑀 𝑀 𝑀 ∩𝑀 ∩𝑀
+ +
𝑀 ∪𝑀
𝑀 ∪𝑀 𝑀 𝑀 ∪𝑀 ∪𝑀 (2) (2
𝑚𝐼𝑜𝑈 = 𝑚𝐼𝑜𝑈 =
2

Figure 5. Example ofExample

Figure 5.of original (left) and annotated image (right). The annotated image has image
two
Figure 5. Example original of original
(left) (left) and image
and annotated annotated image
(right). The(right). The image
annotated annotated
has two has tw
classes in blue thein
classes lettuce class
blue the and pink
lettuce classthe
and background class.
pink the background class.
classes in blue the lettuce class and pink the background class.

2.6.2. Coverage
2.6.2. Coverage
2.6.2. Coverage
Thecoverage
The coverage was
Thewas calculated
coverage by segmenting
segmenting
was calculated
calculated by the images
imagesthe
by segmenting
the with the trained
images
with the trained DeepLabv3+
with the trained DeepLabv3
DeepLabv3+
model. In
model. Inmodel.
these segmented
these segmented images the
In these segmented
images the number
images
number ofnumber
theof pixels classified
pixels classified aslettuce
of pixels as lettucewas
classified was
as divided
lettuce was divide
divided
bythe
by thetotal
total number
bynumber
the totalofnumber
of pixels,see
pixels, seepixels,
of Equation
Equation (3).
see(3).
Equation (3).
# 𝑙𝑒𝑡𝑡𝑢𝑐𝑒 𝑝𝑖𝑥𝑒𝑙𝑠
# 𝑙𝑒𝑡𝑡𝑢𝑐𝑒 𝑝𝑖𝑥𝑒𝑙𝑠
Coverage [%]
𝐶𝑜𝑣𝑒𝑟𝑎𝑔𝑒[% # lettuce
] == #
𝐶𝑜𝑣𝑒𝑟𝑎𝑔𝑒 [%] pixels
= · 100⋅ 100 ⋅ 100 (3)
(3) (
𝑡𝑜𝑡𝑎𝑙 𝑝𝑖𝑥𝑒𝑙𝑠
# total pixels# 𝑡𝑜𝑡𝑎𝑙 𝑝𝑖𝑥𝑒𝑙𝑠

2.6.3.
2.6.3. Volume
Volume over
2.6.3.over Time
VolumeTimeover Time
To
Todetermine
determine the
the lettuce
To determine head
lettucethehead volume
volume
lettuce head over
over time
timefrom
volume images
fromtime
over images taken
fromtaken in
imagesin the
the greenhouse
greenhouse
taken in the greenhous
with conditions
with conditions that conform
that conform
with conditions to
that to practice a “ground
practicetoapractice
conform plane”
“grounda plane”
“ground is needed.
is plane” This
needed.isThis ground
ground
needed. plane
Thisplane
ground plan
was determined
was determined by fitting a
by fitting by
was determined plane
a plane using
fittingusing RANSAC
RANSAC
a plane [59] through
[59] through
using RANSAC the
[59]the non-lettuce
non-lettuce
through pixels
pixelson
the non-lettuceon pixels o
the
the day
day of planting.
ofthe day of RANSAC
planting. RANSACRANSAC
planting. can
can compensate
compensate for
for slight
slight skewness
can compensate skewness
for in
in camera
camerain
slight skewness mounting.
mounting.
camera mountin
This
Thismethod
method assumes
This methodthat
assumes thatthe
thecamera
assumes camera position
that theposition
cameradoes
doesnotnot
position change
does after
change not planting.
after planting.
change The
after Theheight isThe heigh
height
planting.
subsequently
is subsequently calculated by determining
calculatedcalculated
by determiningthe point-to-plane distance
the point-to-plane for each
distance for pixel classified
each forpixeleach pix
is subsequently by determining the point-to-plane distance
as lettuce (Figure 6). The volume was then calculated by multiplying the height by the
classified classified
as lettuce as(Figure
lettuce 6).(Figure
The volume
6). Thewas then calculated
volume by multiplying
was then calculated the heightthe heigh
by multiplying
pixel size in mm and dividing by the density at each moment in time. By dividing by the
by the pixelby size in mm
the pixel and
size individing
mm and by the density
dividing by theatdensity
each moment
at eachin time. By
moment individing
time. By dividin
density, the volume per plant was calculated which was needed to correct for different
by the density,
by the the volume
density, theper plant per
volume wasplant
calculated which was
was calculated needed
which wastoneeded
correcttoforcorrect fo
plant densities. The growth was then determined based on the volume increase.
different plant densities.
different The growth
plant densities. The was then determined
growth based on the
was then determined basedvolume
on theincrease.
volume increas
Height
𝐻𝑒𝑖𝑔ℎ𝑡 [cm]
i [cm ==dist_plane
]𝐻𝑒𝑖𝑔ℎ𝑡 𝑑𝑖𝑠𝑡_𝑝𝑙𝑎𝑛𝑒−− dist
𝑑𝑖𝑠𝑡
[cm] =i 𝑑𝑖𝑠𝑡_𝑝𝑙𝑎𝑛𝑒
i − 𝑑𝑖𝑠𝑡 (4)
(4) (
i ∑ ∑ N(𝐻𝑒𝑖𝑔ℎ𝑡∑ ⋅(𝐻𝑒𝑖𝑔ℎ𝑡
𝑝𝑖𝑥𝑒𝑙𝑠𝑖𝑧𝑒 )2


⋅ 𝑝𝑖𝑥𝑒𝑙𝑠𝑖𝑧𝑒 )
i =0 Height i · pixelsize (5) (
h
[cm3] 3
𝑉𝑜𝑙𝑢𝑚𝑒 𝑜𝑣𝑒𝑟
Volume 𝑡𝑖𝑚𝑒
𝑉𝑜𝑙𝑢𝑚𝑒
over 𝑜𝑣𝑒𝑟
time cm𝑡𝑖𝑚𝑒== [cm3] = ⋅ 1000
· 1000 ⋅ 1000 (5)
𝑑𝑒𝑛𝑠𝑖𝑡𝑦
density 𝑑𝑒𝑛𝑠𝑖𝑡𝑦

Figure 6.6. Side

Figure Side view
Figure
view of point
6. of
Sidepoint
view cloud within
of point
cloud within green
cloud the fitted
within
green the fitted ground
greenground plane
the fitted using
ground
plane using RANSAC
plane [58]
using [58]
RANSAC RANSACto
to [58]
determine the
determine the height.
height.the
determine At the
the bottom
At height. the
At the
bottom plants
plantsat
thebottom atthe
the start
plants
the atdate
start the and
date andabove
start date
above the
andtheplants
above at
the
plants the
at day
plants
the day of the day
at
harvest.
of harvest. harvest.

2.6.4. Light
2.6.4. Light LossLight
2.6.4.
Loss overLoss
over Timeover Time
Time
Foroptimal
For optimal
For use
use of the
optimal
of the usegreenhouse
of thearea,
greenhouse area,
greenhouse it isarea,
important
it is important to evaluate
it evaluate
to is important to the spacing
evaluate
the spacing the spacin
decision.
decision.
In In this
decision.
this research, research,
In this
the light thecalculated
loss was light the
research, lossover
was
light calculated
loss
time. over
was calculated
Simplified time. Simplified
over
we determined time. we
if the Simplified
light w
loss after spacing was larger than the light loss before spacing due to overlapping leaves.
The light loss after spacing was calculated by subtracting 100 minus the current coverage
Sensors 2023, 23, 2929 11 of 30

(Section 2.6.2) (Equation (6)). The light loss before spacing was calculated by determining
the difference between the current coverage and the theoretical coverage. This theoretical
coverage was the projection of previous coverage divided by the previous density and
multiplied by the new density (Equation (7)). This difference indicated how much light
was lost due to overlapping leaves. Now the light loss was calculated (Equation (8)). If for
example, the light loss was negative, it indicated that too many leaves were overlapping,
resulting in a light loss that was larger than the light loss after spacing. On the other hand,
if the light loss was positive, then the spacing decision was too early, because in the new
spacing density, there was more light lost than before.

LightLosscurrent = 100 − coveraget (6)

coveraget−1
LightLossbe f ore spacing = coveraget − densityt (7)
densityt−1
LightLoss = LightLosscurrent − LightLossbe f ore spacing (8)
where t denotes the time when spacing occurred and t − 1 time before spacing.

3. Results
Two experiments were carried out consecutively. The first experiment offered the
teams the possibility to test their algorithms in growing a real crop in a real greenhouse to
bridge the gap between simulation and reality. The second experiment was the eventual
challenge that determined the winner of this third Autonomous Greenhouse Challenge.
In both experiments, the algorithms were optimizing income against costs to achieve
a maximum net profit. The challenge was the outside conditions during the second
experiment (early summer) were very different from those in the first experiment (late
winter). Both data sets are made publicly available and can be used for further development
of intelligent control of lettuce production systems. We show both but focus more on the
second data set which was determining the winner of the competition.
The results of realized climate, resource use, crop yield, and applied plant spacing are
given for the two cultivation cycles in the six greenhouse compartments.

3.1. Climate and Resource Use Analysis

The climate control strategy in the greenhouse largely determines the use of resources.
Figure 7 illustrates the average daily greenhouse air temperature in the different compart-
ments during the late winter and early summer experiments. The realized daily average
temperature ranged between 18 ◦ C and 22.5 ◦ C for the first experiment (winter), whereas
for the second experiment (early summer) the minimum and maximum diurnal tempera-
tures were on average 1 ◦ C higher. Teams decreased their energy consumption for heating
by more than 80%, up to 97%, except for two teams DigitalCucumbers and VeggieMight
(Appendix A Figure A1). For DigitalCucumbers the operation of the heating pipes is
reflected in the high diurnal temperature realized in their compartment augmented by their
low ventilation rates that maintained the highest CO2 concentration (Figure 8), despite
the low CO2 dosage rates. VeggieMight realized lower temperatures, due to the higher
ventilation rates, despite the higher energy used for heating. Appendix A Figure A1 shows
the energy consumption for heating over time in both experiments, an important part of
the resource use.
 Sensors 2023, 23, x FOR PEER REVIEW 12 of 30

Sensors 2023, 23, x FOR PEER REVIEW 12 of 30

Sensors 2023, 23, 2929 12 of 30

Figure 7. Average daily temperature (°C) of all compartments during the first and second
cultivation.
Figure 7. Average daily temperature (◦ C) of all compartments during the first and second cultivation.
Figure 7. Average daily temperature (°C) of all compartments during the first and second
cultivation.

Figure 8. CO2 concentration (ppm) for the different compartments during the first and second
Figure 8. CO2 concentration (ppm) for the different compartments during the first and second
cultivation.
cultivation.
The daily light integral (DLI), mol m−2 plant−1 in the greenhouse compartments is
Figure
the sum8. CO
The 2 concentration (ppm) for the different
of daily light
outside integral
sunlight, (DLI), mol
influenced bymthe compartments
−2 plant −1 in
team’s during
the greenhouse
screen usage the firstAand
compartments
(Appendix second
Figure isA2)
the
cultivation.
sum of outside sunlight, influenced by the team’s screen usage (Appendix A Figure A2)
and topped up with the artificial light for each team (Appendix A Figure A4). Especially the
and topped
longer up with
day length and the artificial
higher light for
intensities eachradiation
of solar team (Appendix resultedAinFigure a higher A4). Especially
cumulative
The daily light integral (DLI), mol m −2 plant−1 in the greenhouse compartments is the
theduring
DLI longerthe day length
second and higher
experiment. Theintensities
realized indoor of solardailyradiation
PAR for each resulted
team in a higher
as the sum
sum of outside
ofcumulative sunlight, influenced by the the team’s
twoscreen usageexperiments
(Appendix is A Figure A2)in
solar DLI and
radiation during the second
artificial experiment.
lightduring The realized
cultivation indoor daily illustrated
PAR for each
and topped up thewith the artificial light for VeggieMight
each team (Appendix A Figure A4). Especially
team 9.asInthe
Figure sum ofexperiment,
second solar radiation
team and artificial lightduring
realized the highestthe cumulative
two cultivation DLI
the longer day length and higher intensities of solar radiation resulted in a higher
experiments is illustrated in Figure 9. In the second experiment, team VeggieMight
(Figure 10) despite the zero hours of their artificial illumination (Appendix A Figure A4)
cumulative DLI during the second experiment. The realized indoor daily PAR for each
realized the highest cumulative DLI (Figure 10) despite the zero hours of their artificial
and the highest total light interception per head of lettuce, as a result of the intelligent
team as the sum blackout
of solar radiation and artificial lightduring the lowtwoplantcultivation
illumination
operation (Appendix A screen
of their Figure (Appendix
A4) and theAhighest Figure total
A3), light
and theirinterception perdensity
head of
experiments
(Table 1). is illustrated
Comparable in Figure
cumulative DLIs 9. for
In all
theteams
second were experiment,
observed in team
the VeggieMight
first experiment.
lettuce, as a result of the intelligent operation of their blackout screen (Appendix A Figure
realized
However, theinhighest
thelow cumulative
first experiment, DLIthe(Figure
light 10) despite
demand the zerowas
of lettuce hoursmainlyof their artificial
A3), and their plant density (Table 1). Comparable cumulative DLIs forcovered
all teamswithwere
illumination
artificial (Appendix
light, as 50% A
to Figure
88% of A4)
the and the
measured highest
light total
originatedlight interception
from LEDs per head ofA
(Appendix
observed in the first experiment. However, in the first experiment, the light demand of
lettuce,
Figure as a result of the intelligent operation of their blackout screen (Appendix Arequired
Figure
lettuceA4).wasDuring
mainlythe secondwith
covered experiment,
artificialcomparable
light, as 50% lightto levels
88% of were the again
measured light
A3), and their
tooriginated low
reach the target plant density (Table 1). Comparable cumulative DLIs for all teams were
from weight, illustrated by
LEDs (Appendix A the colored
Figure A4).circles
During in both
the figures.
second Due to the
experiment,
observed
higher in the
solar first experiment.
radiation, However, in the first experiment, the light Appendix
demand ofA
comparable light levelsless were artificial light was
again required toneeded
reach the in this
target experiment.
weight, illustrated by the
lettuce
Figure was
A5 mainlythe
shows covered
electricitywith artificial light,
consumption over astime
50%into both88% of the measured
experiments of each light
team,
colored circles in both figures. Due to the higher solar radiation, less artificial light was
originated
an important frompartLEDs
of the (Appendix
resource use.A Figure A4). During the second experiment,
needed in this experiment. Appendix A Figure A5 shows the electricity consumption over
comparable light levels were again required to reach the target weight, illustrated by the
time in both experiments of each team, an important part of the resource use.
colored circles in both figures. Due to the higher solar radiation, less artificial light was
needed in this experiment. Appendix A Figure A5 shows the electricity consumption over
time in both experiments of each team, an important part of the resource use.
 Sensors 2023, 23, x FOR PEER REVIEW 13 of 30
Sensors 2023, 23, x FOR PEER REVIEW 13 of 30
Sensors 2023, 23, 2929 13 of 30

Figure 9. Indoor photosynthetic active radiation (PAR) for the different compartments during the
Figure
Figure 9.
9. Indoor
first and Indoor photosynthetic
second photosynthetic active
cultivation. active radiation
radiation (PAR)
(PAR) for
for the
the different compartments during
different compartments during the
the
first
first and
and second
second cultivation.
cultivation.

Figure 10. Cumulative light intercepted by each lettuce head per compartment in the first and second
Figure 10. Cumulative
cultivation. light intercepted
Light interception by each
per lettuce head was lettuce head
calculated byper compartmentofinthe
a multiplication the firstlight
daily and
Figure
second 10. Cumulative
cultivation. light
Light interceptedper
interception bylettuce
each
2 lettuce
head headcalculated
was per compartment
by a in the firstofand
multiplication the
integral (DLI) with the green coverage per m growing area, divided by the head density on each
second
daily cultivation.
light integral Light with
(DLI) interception
the greenper lettuce per
coverage headm 2was calculated
growing area, by a multiplication
divided by the head of the
density
particular day. The circles (o), mark the days at which the2lettuce heads reached the target fresh weight
daily
on250lightparticular
each integral (DLI)
day. withcircles
The the green coverage
(o),data
mark the per
daysmatgrowing
which area,
the divided
lettuce byreached
heads the headthedensity
target
of
on eachgparticular
by linearly interpolating
day. The circles the
(o), mark ofthe
thedays
weekly
at destructive
which the measurements
lettuce heads on randomly
reached the target
fresh weight of 250 g by linearly interpolating the data of the weekly destructive measurements on
fresh weight
selected
randomly of 250
lettuce glettuce
heads.
selected by linearly
heads.interpolating the data of the weekly destructive measurements on
randomly selected lettuce heads.
Table 1. The average density of lettuce heads for the two cultivations as calculated using Equation (1).
All climate strategies applied by the teams resulted in differences in resource use
All climate strategies applied by the teams resulted in differences in resource use
which are summarized in Table 2.
which are summarized
Experiment in Table Koala
Reference 2. CVA
Monday Digital Veggie
Planting Date Lettuce Cucumbers Might
Table 1. The average density of lettuce heads for the two cultivations as calculated using Equation
Table
(1). 1. The average density of lettuce heads for the two cultivations as calculated using Equation
3 February 32.7 34.5 31.9 41.4 37.7 32.9
(1).
3 May 29.0 30.4 29.9 36.7 31.7 28.7
Monday Digital Veggie
Experiment Planting Date Reference Koala CVA Monday Digital Veggie
Experiment Planting Date Reference Koala CVA Lettuce Cucumbers Might
All climate strategies applied by theLettuce
teams resultedCucumbers Might use
in differences in resource
3 February 32.7 34.5 31.9 41.4 37.7 32.9
3 February 32.7
which 34.5 in Table
are summarized 31.92. 41.4 37.7 32.9
3 May 29.0 30.4 29.9 36.7 31.7 28.7
3 May 29.0 30.4 29.9 36.7 31.7 28.7
Table 2. Net profit of different teams in the second experiment consisting of crop income minus
Table 2. Netcosts,
costs (fixed profitheating
of different
costs,teams in the
electricity second
costs, CO2 experiment consisting of
costs, and intervention crop income minus
costs).
costs (fixed costs, heating costs, electricity costs, CO2 costs, and intervention costs).
Veggie Digital Monday
CVA Veggie Digital Koala Monday Reference
CVA Might Cucumbers Koala Lettuce Reference
Might Cucumbers Lettuce
Total income [€/m2] 12.16 10.38 15.84 14.16 11.83 12.12
Total income [€/m2] 12.16 10.38 15.84 14.16 11.83 12.12
Sensors 2023, 23, 2929 14 of 30

Table 2. Net profit of different teams in the second experiment consisting of crop income minus costs
(fixed costs, heating costs, electricity costs, CO2 costs, and intervention costs).

Veggie Digital Monday

CVA Koala Reference
Might Cucumbers Lettuce
Total income [€/m2 ] 12.16 10.38 15.84 14.16 11.83 12.12
Sensors 2023, 23, x FOR PEER REVIEW
Fixed costs [€/m2 ] 7.85 6.41 8.50 7.06 9.64 6.59 14 of 30
Heating Costs
0.01 0.29 0.16 0.04 0.03 0.02
[€/m2 ]
Fixed costs [€/m2] 7.85
Electricity costs 6.41 8.50 7.06 9.64 6.59
2] 0.23 0.00 0.46 0.00 0.45 0.34
Heating Costs [€/m ] 2 [€/m 0.01 0.29 0.16 0.04 0.03 0.02
Electricity costs [€/m ]
2 CO 2 -costs 0.23 2 ]
[€/m 0.00
0.60 0.53 0.46 0.34 0.000.11 0.45
0.18 0.34
0.53
CO2-costs [€/m2] 0.60
Total operational 0.53
8.69 7.24
0.34 9.45
0.117.24 0.18
10.30
0.53
7.48
costs [€/m 2]
Total operational costs [€/m ]
2 8.69 7.24 9.45 7.24 10.30 7.48
Intervention Costs [€/m ]2 Intervention 2.00
Costs 1.00
2.00 1.00 3.00 3.00 1.001.00 2.00
2.00 - -
[€/m2 ]
Net profit [€/m2] 1.47 2.14 3.39 5.93 −0.47 4.64
Net profit [€/m2 ] 1.47 2.14 3.39 5.93 −0.47 4.64
3.2. Crop Yield Analysis
3.2. Crop Yield11
Figure Analysis
shows the initial plant densities of 92 heads/m2 chosen by all teams and the
2 chosen by all teams and the
different
Figuredensities
11 showsrealized over
the initial time.
plant Throughout
densities the cultivation
of 92 heads/m periods, lettuce heads
were spaced
different according
densities realizedtoover
the time.
decisions of each the
Throughout team’s algorithm.
cultivation The lettuce
periods, reason heads
for the
spacing
were was
spaced to balance
according to fast crop growth
the decisions andteam’s
of each minimize greenhouse
algorithm. spacefor
The reason utilization thus
the spacing
resource use and costs for spacing. As the average plant density impacted costs due to
was to balance fast crop growth and minimize greenhouse space utilization thus resource
resource
use usefor
and costs and labor for
spacing. Asspacing events,
the average teams
plant MondayLettuce
density impacted costsand DigitalCucumbers
due to resource use
attempted to exploit the advantage of higher densities and fewer spacing attempted
and labor for spacing events, teams MondayLettuce and DigitalCucumbers interventions,
to
exploit the advantage of higher densities and fewer spacing interventions,
respectively. The average plant densities of the two cultivation cycles for the sixrespectively. The
average plant densities
compartments of the
are shown intwo cultivation
Table 1. cycles for the six compartments are shown in
Table 1.

Figure11.11.
Figure Lettuce
Lettuce density
density (heads
(heads m2m 2) and
) and harvest
harvest dates
dates (o)the
(o) in in different
the different compartments
compartments during
during the
the first and second cultivation
first and second cultivation period. period.

Duringthe
During thefirst
firstexperiment,
experiment,the theteam’s
team’salgorithms
algorithmsseemed
seemedtotohavehavecomputed
computedthe the
harvest time quite accurately
harvest time quite accurately at the target weight of 250 g, as can be seen in Figure 12. The
target weight of 250 g, as can be seen in Figure 12.
crops
The of DigitalCucumbers
crops of DigitalCucumbers andandVeggieMight
VeggieMight grew
grewpoorly,
poorly,they were
they werestill farfar
still offoff
from
fromthe
targeted
the targeted weight
weightthethemoment
momentthat thatthe
the first cultivation was
first cultivation was terminated.
terminated. For Forthethesecond
second
experiment,the
experiment, thealgorithms
algorithmsofofallallparticipants
participantswereweretootoolate
lateininharvesting,
harvesting,the theharvest
harvest
weightwas
weight washigher
higherthan
thanthethetarget
targetweight.
weight.OnlyOnlythe
thereference
referencecompartment,
compartment,was washarvested
harvested
timely.Appendix
timely. Appendix AA Table
Table A3A3 summarizes
summarizes thethe lettuce
lettuce weight
weight at harvest
at harvest andand the number
the number of
of cultivation days per compartment. It also shows the dates at which the ideal target
cultivation days per compartment. It also shows the dates at which the ideal target weight
weight would have been achieved for the different compartments by linearly
interpolating the weekly fresh weight measurement.
Sensors 2023, 23, 2929 15 of 30

Sensors 2023, 23, x FOR PEER REVIEW 15 of 30

would have been achieved for the different compartments by linearly interpolating the
weekly fresh weight measurement.

400

350

300

250

200

150

100

50

0
08-May 15-May 22-May 29-May 05-Jun 12-Jun 19-Jun

Reference Koala CVA

MondayLettuce Digitalcucumbers VeggieMight

Figure 12. Development of fresh weight [g/plant] in the different compartments for the first and
Figurecultivation.
second 12. Development of fresh
The curves wereweight [g/plant]
obtained by linearininterpolation
the differentofcompartments for the sampled
the weekly randomly first and
second
plants andcultivation. The weighed
(destructively) curves were obtained
heads byThe
of lettuce. linear
endinterpolation of the weekly
of the lines represents randomly
the chosen date
ofsampled
harvest plants
and theand (destructively)
fresh weighed
weight at harvest. Theheads
circlesof(o),
lettuce. The end
represent of theonlines
the days represents
which the lettucethe
chosen date of harvest and the fresh weight
heads reached the target fresh weight of 250 g. at harvest. The circles (o), represent the days on which
the lettuce heads reached the target fresh weight of 250 g.
Total lettuce crop yields (Figure 12 and Appendix A Table A3 and the quality assess-
ment Total lettuce
(Appendix crop yields
A Figure (Figure in12a computed
A6) resulted and Appendix A from
income Tablethis
A3cultivation
and the quality
cycle.
assessment (Appendix A Figure A6) resulted in a computed income from this cultivation
In Section 3.3 this income is compared with the costs associated with the second cultivation
cycle.toIn
cycle Section
obtain the3.3 this income
computed is compared
net profit (Table 2).with the costs associated with the second
cultivation cycle to obtain the computed net profit (Table 2).
3.3. Net Profit
3.3. Net
The Profit
combination of climate strategies, resource use, crop yield, and quality realized by
The combination
the teams of climate
resulted in different net strategies, resource
profits. Details use, crop
are shown yield,2.and quality realized
in Table
by the teams resulted in different net profits. Details are shown in Table 2.
3.4. Plant Spacing Analysis
3.4. Plant Spacing
The net profit Analysis
is relying on crop yield, quality, resource use, and greenhouse occupation.
The realized plant
The net profit growth rates, plant
is relying densities,
on crop yield, and realized
quality, final harvest
resource use, anddue to timely
greenhouse
estimation
occupation. The realized plant growth rates, plant densities, and realized final harvest the
of plant weights were shown to be crucial for the net profit. Therefore, due
options of estimation
to timely different detailed
of plantcomputer vision
weights were analyses
shown tocrucial
to be make timely
for thedecisions
net profit.on spacing
Therefore,
decisions are of
the options shown in this
different paper. computer vision analyses to make timely decisions on
detailed
spacing decisions are shown in this paper.
3.4.1. Coverage
3.4.1.The computer-vision-based data analysis of plant growth mainly relies on the segmen-
Coverage
tation of the images of lettuces taken at a defined area over time. An example of such an
The computer-vision-based data analysis of plant growth mainly relies on the
image is shown in Figure 12 (left). There results of being segmented with the DeepLabv3+
segmentation of the images of lettuces taken at a defined area over time. An example of
algorithm as described in the Materials and Methods sectionare shown in Figure 13 (right).
such
The an image
algorithm hadis ashown
mIoU ofin 98.2%
Figureon12the
(left). There dataset,
validation results of being
with 100% segmented
indicating with the
that the
DeepLabv3+ algorithm as described in the Materials and Methods sectionare
segmentation is perfect. Even though the validation dataset was relatively small, the seg- shown in
Figure 13 (right).
mentation procedure Thecan
algorithm had a mIoU
be considered of 98.2% onrobust,
to be sufficiently the validation
since onlydataset, with
a small 100%
fraction
indicating that the segmentation is perfect. Even though the validation
of processed images from the datasets segmentation occurred to be incorrect. An example dataset was
relatively small, the segmentation procedure can be considered to be sufficiently
is given in Figure 13. Although there are white edges visible (right), showing where pixels robust,
sincefalsely
were only aassigned
small fraction of processed
as ‘lettuce’, images
these edges from the datasets
are relatively small. Thesegmentation occurred
algorithm appeared
to be incorrect. An example is given in Figure 13. Although there are white
also to be robust in dealing with different light conditions in the greenhouses as shown edges visible
in
(right),
the showing
correctly wherebottom
processed pixels were
row offalsely
Figureassigned
13. as ‘lettuce’, these edges are relatively
small. The algorithm appeared also to be robust in dealing with different light conditions
in the greenhouses as shown in the correctly processed bottom row of Figure 13.
Sensors 2023, 23, x FOR PEER REVIEW 16 of 30

Sensors2023,
Sensors 2023,23,
23,2929
x FOR PEER REVIEW 16ofof30
16 30

Figure 13. Example of a real (left) and segmented (right) image using DeepLabv3+. From the
segmentation, the coverage [%] was calculated.
Figure13.
Figure 13. Example
Example ofof aa real
real(left)
(left)and
andsegmented
segmented(right)
(right)image
imageusing
usingDeepLabv3+.
DeepLabv3+. From
From the
the
segmentation, the coverage [%] was calculated.
Figure 14
segmentation, theshows
coveragethe[%]
percentage of cultivation area covered by lettuce for each image
was calculated.
over time for each compartment. In the first experiment, less efficient space occupation
Figure14 14 showsthe thepercentage
percentageof of cultivationarea
areacovered
coveredby bylettuce
lettucefor
foreach
each image
and Figure
coverageshows
were observed, due tocultivation
the explorative decision-making of the image
teams.
over time for each compartment. In the first experiment, less efficient space
over time for each compartment. In the first experiment, less efficient space occupation occupation
and
Considering the more strategic decisions during the second experiment, teams targeted
and coverage
coverage were observed,
were observed, due
due to the to the explorative
explorative decision-making
decision-making of the teams.ofConsidering
the teams.
more efficient space occupation. A high coverage percentage was realized in a shorter
Considering
the the more
more strategic strategic
decisions decisions
during during
the second the second
experiment, experiment,
teams targeted teams targeted
more efficient
time. Most teams maintained a coverage above 90%, only teams VeggieMight and
more occupation.
space efficient space occupation.
A high coverageApercentage
high coverage percentage
was realized was realized
in a shorter in a shorter
time. Most teams
Reference seemed to have spaced too early if only coverage is considered for the decision
time. Mosta teams
maintained coveragemaintained
above 90%,a only
coverage
teams above 90%, only
VeggieMight teams VeggieMight
and Reference and
seemed to have
making. However, as explained in the materials and methods section, coverage might not
Reference
spaced tooseemed
early if to have
only spaced too
coverage early if onlyfor
is considered coverage is considered
the decision making.forHowever,
the decision
as
be a good parameter to decide on spacing, since growth might be hampered already by
making. However,
explained as explained
in the materials in the materials
and methods and methods
section, coverage mightsection,
not becoverage might not
a good parameter
too late spacing.
bedecide
to a goodonparameter to decide
spacing, since onmight
growth spacing, since growth
be hampered mightbybetoo
already hampered already by
late spacing.
too late spacing.

Figure 14. Coverage [%] of the greenhouse with lettuce heads, as calculated with the segmentation
Figure 14.The
algorithm Coverage [%] shown
frequently of the greenhouse with
stepwise fall lettuce heads,
of coverage as calculated
is a result of spacingwith the segmentation
actions.
algorithm The frequently shown stepwise fall of coverage is a result of spacing actions.
FigureCrop
3.4.2. 14. Coverage
Volume [%]
overofTime
the greenhouse with lettuce heads, as calculated with the segmentation
algorithm The frequently shown stepwise fall of coverage is a result of spacing actions.
3.4.2.Next
CroptoVolume
coverage,over
cropTime
width, height, and volume are suitable crop traits associated
with growth. To explore the potential of these
andtraits, in this researchcrop
height and volume
3.4.2.Next
CroptoVolume
coverage,
overcrop
Time width, height, volume are suitable traits associated
were determined over time. Since volume is strongly correlated with height,
with growth. To explore the potential of these traits, in this research height and volume a comparison
betweenNextcalculated
to coverage,
and crop
manual width, height, andis volume areFigure
suitable
15.crop traits associated
were determined over time. Sincecrop height
volume shown
is strongly in
correlated with Figure
height, a 15 shows a
comparison
with
strong growth. To
correlation explore
between the
the potential of
calculated these
height traits,
as it in this
follows research
from the height
RealSenseand volume
camera
between calculated and manual crop height is shown in Figure 15. Figure 15 shows a
were
images determined over time. Since volume is strongly correlated with height,
2 a comparison
strong and the manually
correlation between measured height height
the calculated (ground astruth), withfrom
it follows a high
theRRealSense
and slopecamera
close
between
to calculated height
1. Therefore, and manual crop height isthe
shown in Figure 15.were
Figure 15 shows bea
images and thedaily measurements
manually measured heightfrom
(ground RealSense
truth), with camera
a high assumed
R2 and slopetoclose
strong and
correct correlation
were usedbetween the calculated
to calculate the volume.height as it follows from the RealSense camera
to 1. Therefore, daily height measurements from the RealSense camera were assumed to
images and the
In Figure 16,manually measured
the calculated height
height from(ground
images truth),
taken bywiththeaRealSense
high R2 and slope over
camera close
be correct and were used to calculate the volume.
to 1.isTherefore,
time shown fordaily height measurements
all compartments. Reference fromand the RealSense camera
VeggieMight have a were
lowerassumed
predicted to
be correct and were used to calculate the volume.
plant height due to the early spacing decision. DigitalCucumbers has the highest plant
height, elongation occurred when plants were touching each other due to high density. An
interesting phenomenon is that after the last spacing decision (ca. after 5 June) not only
height (Figure 16) but also volume (Figure 17) is reaching a plateau. This means that the
daily fresh weight increase (Figure 12) of the last weeks is not visible in this method of
image analysis.
 20 y = 0.979×x + 0.737
R2 = 0.976
18
MAE = 0.623

Ground truth height [cm]

16

14
Sensors 2023, 23, x FOR PEER REVIEW 12 17 of 30
Sensors 2023, 23, 2929 17 of 30
10

6
22
4
20 y = 0.979×x + 0.737
22
R = 0.976
18
MAE0 = 0.623
0 5 10 15 20

Ground truth height [cm]

16
Predicted height [cm]
14
Figure1215. Correlation of manually measured ground truth height (ground truth height) of lettuces
during10 destructive measurements compared with predicted height (predicted height) from
RealSense
8
camera images of lettuces.
6
In Figure 16, the calculated height from images taken by the RealSense camera over
4
time is shown for all compartments. Reference and VeggieMight have a lower predicted
2
plant height due to the early spacing decision. DigitalCucumbers has the highest plant
0
height, 0elongation 5 occurred 10 when 15 plants were 20 touching each other due to high density.

An interesting phenomenon Predicted height [cm]

is that after the last spacing decision (ca. after 5 June) not only
height
Figure 15.
Figure(Figure
Correlation16)
of but also
manually
15. Correlation volume
measured
of manually (Figure
ground17)
measured is reaching
truth heighttruth
ground aheight
plateau.
(ground truth This means
height)
(ground that the
of lettuces
truth height) of lettuces
during destructive
daily measurements
increase compared
(Figure 12) with
of thepredicted
last weeks height
is (predicted
during destructive measurements compared with predicted height (predicted height) fromof
fresh weight not visible inheight)
this from
method RealSense
RealSense
image camera images of lettuces.
analysis.
camera images of lettuces.
In Figure 16, the calculated height from images taken by the RealSense camera over
time is shown for all compartments. Reference and VeggieMight have a lower predicted
plant height due to the early spacing decision. DigitalCucumbers has the highest plant
height, elongation occurred when plants were touching each other due to high density.
An interesting phenomenon is that after the last spacing decision (ca. after 5 June) not only
Sensors 2023, 23, xheight (Figure
FOR PEER 16) but also volume (Figure 17) is reaching a plateau. This means that the
REVIEW 18 of 30
daily fresh weight increase (Figure 12) of the last weeks is not visible in this method of
image analysis.
explains the lower volume of MondayLettuce with respect to CVA (Figure 17). This
phenomenon is also summarized in Table 3. In this table, the volume is sorted from high
to low. Although the weight of the lettuce at optimal harvest day in each compartment is
approximately 250 g, there are differences in volume per head for similar crop densities.
The Reference for example had a much higher volume than Koala. Both Figures 16 and
17, and Table 3 show differences from the similar weight. From this, it can be concluded
that the volume calculation alone will not be a conclusive trait for the estimation of the
Figure 16. Daily calculated lettuce height from RealSense
cameracamera
imagesimages in all compartments
during during
Figure weight
16. Daily of thelettuce
calculated head height
of lettuce.
from RealSense in all compartments
the the
cultivation start.start.
cultivation

8,000
In Figure 17 the daily estimated plant volume is given using imaging. The circles
7,000
indicate on which day the target harvest weight of 250 g was reached. Large differences
between the compartments were observed. 6,000 In the second experiment, the difference in
volume between CVA and MondayLettuce 5,000 is remarkable. At the optimal harvest day,
CVA has a volume of 6705 cm3/head and 4,000 MondayLettuce 4844 cm /head. A part of this
3

large difference is caused by the plant 3,000 density, which is at the end 15 and 22.5 plants/m2
Figure
for16.CVA
Daily calculated lettuce height from
and MondayLettuce, RealSense camera
respectively. Because images
of in all compartments
the during of
high plant density
2,000
the cultivation start. more leaves were overlapping
MondayLettuce, (coverage was 98.5%, compared to 87.2%
of CVA). Since overlapping leaves do 1,000 not contribute to volume in the image analysis it
In Figure 17 the daily estimated plant volume 0 is given using imaging. The circles
indicate on which day the target harvest weight 08-May of 25015-May
g was22-May
reached. Large differences
29-May 05-Jun 12-Jun 19-Jun

between the compartments were observed. In the second Reference

experiment,
MondayLettuce
Koala
the difference
Digitalcucumbers
CVA
in
VeggieMight
volume between CVA and MondayLettuce is remarkable. At the optimal harvest day,
CVA has a volume
Figure of 6705
17. Daily cm3/head
estimated lettuceand MondayLettuce
volume 4844 cmduring
in all compartments 3/head.the A cultivation
part of thisperiod.
large difference Figure
is caused by the plant density, which is at the end 15
17. Daily estimated lettuce volume in all compartments during the and 22.5 plants/m 2
cultivation period.
for CVA and MondayLettuce,
In Figure 17 the daily respectively.
estimated plant Becausevolumeof theis given
high plant usingdensity
imaging. of The circles
MondayLettuce, more
indicate Table
on 3.leaves
which daywere
Overview theofoverlapping
target harvest
plant traits at(coverage
weight
optimal was 98.5%,
of 250
harvest g was
date compared
(when reached. to Large
87.2%
target weight of differences
250 g per lettuce
of CVA). Since overlapping
between head
the is leaves
reached).
compartments Teams do notsorted
are
were contribute
from high
observed. to
Involume
to
the in the
lowsecond
volume. image analysis
experiment, the it difference in
volume between CVA and MondayLettuce is remarkable. At the optimal harvest day, CVA
Optimal Harvest 3 Density Coverage 3 Volume
Compartmenthas a volume of 6705 cm /head and MondayLettuce Max/head.
4844 cm HeightA[cm]
part of this large
Date [Heads/m 2] [%] [cm
2 3/Plant]
difference is caused by the plant density, which is at the end 15 and 22.5 plants/m for CVA
CVA 7 June 2022 respectively.15Because of the
and MondayLettuce, 87.2 15.7 of MondayLettuce,
high plant density 6705
Reference 8 June 2022 18 96.8 15.3 6379
VeggieMight 4 June 2022 18 90.9 16.5 6090
Koala 9 June 2022 18 98.2 14.4 5581
DigitalCucumbers 7 June 2022 18 86.7 16.8 5356
Sensors 2023, 23, 2929 18 of 30

more leaves were overlapping (coverage was 98.5%, compared to 87.2% of CVA). Since
overlapping leaves do not contribute to volume in the image analysis it explains the lower
volume of MondayLettuce with respect to CVA (Figure 17). This phenomenon is also
summarized in Table 3. In this table, the volume is sorted from high to low. Although
the weight of the lettuce at optimal harvest day in each compartment is approximately
250 g, there are differences in volume per head for similar crop densities. The Reference
for example had a much higher volume than Koala. Both Figures 16 and 17, and Table 3
show differences from the similar weight. From this, it can be concluded that the volume
calculation alone will not be a conclusive trait for the estimation of the weight of the head
of lettuce.

Table 3. Overview of plant traits at optimal harvest date (when target weight of 250 g per lettuce
head is reached). Teams are sorted from high to low volume.

Optimal Harvest Density Coverage Max Height Volume

Compartment
Date [Heads/m2 ] [%] [cm] [cm3 /Plant]
CVA 7 June 2022 15 87.2 15.7 6705
Reference 8 June 2022 18 96.8 15.3 6379
VeggieMight 4 June 2022 18 90.9 16.5 6090
Koala 9 June 2022 18 98.2 14.4 5581
DigitalCucumbers 7 June 2022 18 86.7 16.8 5356
MondayLettuce 9 June 2022 22.5 98.5 16.4 4844

3.4.3. Harvest Indicator over Time

As presented in the previous section teams could have harvested earlier given the
target harvest weight of 250 g per head. From Figure 17 and Table 3, there were differences
in volume for similar harvest moments, indicating that volume might not be an ideal
indicator for determining the ideal moment of harvest. The correlation coefficient of all
calculated traits from the image analysis can be found in Appendix A Table A3. From this
table the area per plant multiplied by the maximum height has the highest correlation
Sensors 2023, 23, x FOR PEER REVIEW 19 of 30
coefficient, higher than volume.
In Figure 18 (left), the fresh weight as a function of area per plant multiplied by
maximum height is visualized. From this figure, it can be concluded that there is still some
depicts
noise, the moment
the Mean thatError
Absolute the harvest
(MAE)indicator of 7840 cm
is 22.98 g/head and 3 is satisfied. In Table 4 the exact
RMSE of 31.2. According to the
second-order equation, the harvest weight is reached at 7840given
dates of the fresh weight criterion and harvest indicator are cm3 .for the different teams.

10,000

8,000

6,000

4,000

2,000

0
M ay ay ay ay ay -J un -Ju
n
-Ju
n
- -M -M -M -M 05 12 19
01 08 15 22 29

Reference Koala CVA

MondayLettuce Digitalcucumbers VeggieMight

Figure 18. Relation of calculated area per plant multiplied by maximum height [cm3 ] and measured
Figure
fresh 18. [g/head]
weight Relation of calculated
(left). areaplant
Area per per plant multiplied
multiplied by maximum
by maximum height
height [cm[cm 3] and as
3 /head]
measured
a harvest
fresh weight
indicator [g/head]
in realizing the(left).
targetArea per plant
weight of 250multiplied
g/head inby allmaximum height(dots)
compartments [cm /head]
3 as a harvest
(right).
indicator in realizing the target weight of 250 g/head in all compartments (dots) (right).

Table 4. Overview of area per plant multiplied by max height criterion at optimal harvest date
(when harvest indicator is 7840 cm3 per lettuce head is reached).
Sensors 2023, 23, 2929 19 of 30

Figure 18 (right) illustrates the area per plant multiplied by the maximum height as
the most representative harvest indicator. The colored circles indicated the moments of
harvest that satisfy the fresh weight criterion 250 g whereas the grey horizontal line depicts
the moment that the harvest indicator of 7840 cm3 is satisfied. In Table 4 the exact dates of
the fresh weight criterion and harvest indicator are given for the different teams.

Table 4. Overview of area per plant multiplied by max height criterion at optimal harvest date (when
harvest indicator is 7840 cm3 per lettuce head is reached).

Harvest Date Harvest Date Satisfying Satisfying the Area

Realized Harvest
Satisfying the FW the Area per Plant × Max per Plant × Max
Compartment Date
Criterion Height Criterion Height Criterion
[dd/mm]
[dd/mm] [dd/mm] [cm3 ]
Reference 9 June 8 June 5 June 79,144
Koala 17 June 9 June 3 June 78,819
CVA 13 June 7 June 3 June 80,717
MondayLettuce 14 June 9 June - -
DigitalCucumbers 15 June 7 June 7 June 83,610
VeggieMight 13 June 4 June 3 June 82,410

3.4.4. Light Loss over Time

In Figure 19, the result of the calculation of light loss over time of the second experi-
ment is shown. This light loss factor can be calculated by comparing the coverage factor
just before and just after a spacing instance. Therefore, the result yields a number of points
rather than a time series. The hypothesis is that spacing is optimal when the light loss
calculation gives a result close to zero at each spacing action. A light loss calculation close
to zero means that the lettuce heads just started touching each other by the time that the
spacing was performed. This allows for minimal greenhouse space occupation, which
saves on resources, whereas quality losses are prevented. Figure 19 shows that especially
VeggieMight and the Reference spaced too early, whereas Koala always spaced very late.
DigitalCucumbers had two good spacing moments at the end of the experiment but was
too late for two others. In the beginning, for both the first and second spacing decisions,
they were the latest team which resulted in large light losses and irreversible damage to the
crop (Appendix A Figure A6). CVA seemed to have the best spacing strategy since most of
their spacing decisions were made with light loss points close to zero. However, even this
team had once a large light loss smaller than −10. Keep in mind that the calculation of light
loss can only be carried out after spacing. It should therefore be treated as an observable
parameter to train decision-making algorithms that base the decision on (a combination of)
covered fraction and average head volume.
After combining Figure 19 with Figure 14, we learn that 98% seems to be a reasonable
coverage strategy for autonomous spacing decisions.
 calculation of light loss can only be carried out after spacing. It should therefore be treated
as an observable parameter to train decision-making algorithms that base the decision on
Sensors 2023, 23, 2929
(a combination of) covered fraction and average head volume. 20 of 30
After combining Figure 19 with Figure 14, we learn that 98% seems to be a reasonable
coverage strategy for autonomous spacing decisions.

Figure 19. Light loss calculation for all compartments for the spacing instances of the second experiment.
Negative values indicate a (too) late spacing and large positive values a too-early spacing.
Figure 19. Light loss calculation for all compartments for the spacing instances of the second
4. Discussion
experiment. Negative values indicate a (too) late spacing and large positive values a too-early
spacing. In the experiment of this study, the strategic and operational scheme of lettuce crop
cultivation was determined by AI algorithms developed by teams participating in the chal-
4. Discussion
lenge. These AI algorithms were based on greenhouse climate and crop sensor information.
InThe
thefinal
experiment of thistarget
optimization study,wasthenetstrategic
profit, and
thusoperational
on the one handscheme sideofalettuce cropgrowth
high crop
cultivation washigh
rate and determined by AIfor
plant quality algorithms
a high incomedevelopedand onby the
teamsotherparticipating in the use
hand low resource
challenge.
for lowThese AISince
costs. algorithms were occupation
greenhouse based on greenhouse climate plant
is essential, optimal and crop sensor
spacing decisions
are important.
information. The final optimization target was net profit, thus on the one hand side a high
crop growth Commercial
rate and high lettuce growing
plant quality is afor
continuous
a high incomeprocess of daily
and on the planting younglow
other hand plantlets
and harvesting the full-grown lettuce heads. Target weight is
resource use for low costs. Since greenhouse occupation is essential, optimal plant spacing realized over a reasonable
timeare
decisions window (6–8 weeks) dependent on the cultivation strategy. Economics were expressed
important.
2
per m of thelettuce
production area, is
therefore the resource use and
Commercial growing a continuous process of selling
2 (Equation (2)).
prices were
daily planting multiplied
young
by the average number of lettuce heads per m
plantlets and harvesting the full-grown lettuce heads. Target weight is realized over a
reasonable Teams had two(6–8
time window cultivation cycles. Theon
weeks) dependent first cycle
the was used
cultivation by the teams
strategy. to test and
Economics
explore their algorithms, the second cycle determined the winner. As this means that the
were expressed per m2 of the production area, therefore the resource use and selling prices
teams must have applied their latest skills and knowledge in this second growing cycle,
were multiplied by the average number of lettuce heads per m2 (Equation (2)).
the discussion is focusing on the early summer results.
Teams had two cultivation cycles. The first cycle was used by the teams to test and
For an efficient greenhouse occupation, and to leverage the effect of the average density
explore their algorithms, the second cycle determined the winner. As this means that the
of lettuce heads on the final profit, some teams maintained high densities (Table 1). At
teams must have applied their latest skills and knowledge in this second growing cycle,
high densities, neighboring plants competed for light (Figure 10). In both experimental
the discussion is focusing on the early summer results.
cycles, 11–15 mols PAR/head was needed to realize the target weight of 250 g per head.
For an efficient greenhouse occupation, and to leverage the effect of the average
However, MondayLettuce used only 9 mol/head in the second experiment. This team
density of lettuce heads on the final profit, some teams maintained high densities (Table
maintained a low cumulative DLI and in combination with the highest density among all
1). At high densities, neighboring plants competed for light (Figure 10). In both
teams in the second cultivation, it yielded the lowest amount of total light interception per
experimental cycles,
plant. Also, 11–15 mols PAR/head
DigitalCucumbers realizedwas needed
a high to realize
density. The the
hightarget
plantweight
densityofresulted
250 in
g per head. However, MondayLettuce used only 9 mol/head in the second
intertwined root systems that made the first spacing difficult and seems to be linked to the experiment.
This team
outermaintained a low cumulative
tip burn (Appendix A FigureDLI A6) and
and in
thecombination with the highest
aversively malformed density plants
and elongated
among(Figure
all teams16). in the second cultivation, it yielded the lowest amount of total light
interceptionTeam per VeggieMight
plant. Also, DigitalCucumbers
realized the highestrealized
cumulative a high density.
indoor The without
PAR, even high plant
applying
densityanyresulted in intertwined root systems that made the first spacing
supplemental lighting (Appendix A Figure A4). This was a result of zero deployment difficult and
hours of the blackout screen and a very limited deployment of the energy screenThe
choice not to use any lighting or any blackout screens saved fixed costs associated with the
equipment and the associated running costs for electricity. However, also this team suffered
from the occurrence of outer tip burn and malformations on the plants, even though they
had the lowest average plant density. The high fraction of class C products resulted in a
Sensors 2023, 23, 2929 21 of 30

low income. Similar to VeggieMight, team Koala, did not use supplemental lighting. This
team was also restrictive with CO2 dosing in the first weeks of the cropping cycle. The
team maintained a high coverage bouncing from 93% to 98.9%. The fact that the algorithm
of this team managed to reduce costs, managed to have a high average head density and
had a high fraction of class A resulted in team Koala being the winner.
The final harvest was too late for all teams (Figure 12). Timely harvest would have
resulted in lower resource use and higher average plant density. The effect of earlier harvest
on net profit cannot be quantified, unfortunately, since the quality of the lettuce heads at
earlier moments in time cannot be predicted from the collected data.
Contrary to commercial greenhouse operations with continuous planting, spacing,
and harvesting, the two growing cycles of this study concerned single batches. The choice
for single batches was required to fit the format of the Autonomous Greenhouse Challenge
aimed at allowing teams to develop and show the potential of autonomous algorithms
growing a crop based on data analyses and vision. As a result, the computed profits,
although valid according to the rules of the Challenge, cannot completely be compared
with commercial practice. Dedicated trials would be needed to reflect deeper on the lettuce
growth responses in continuous commercial cycles. However, such trials were outside the
scope of this research. Nevertheless, results show that greenhouse occupation is essential
and that optimum plant spacing decisions are important.
In fully autonomous cultivation such decisions should be made based on continuous
sensor information. In this study camera images obtained by RealSense cameras in the
greenhouse were used to obtain information on crop growth. DeepLabv3+ was used to
separate the lettuce from the background. The model was only trained with a minimal
amount of data. However, considering the output images in 3.4 and the high mIoU (98.2)
it can be concluded that the segmentation proved suitable to be used as a base for crop
spacing decisions.
The RealSense cameras also provided data on the development of height and volume
over time. We expected that these two traits could be used to describe growth. The
development of volume over time has been related to biomass, as in [60,61]. As the height
and width information was proved to be very accurate (a mIoU of 98.2% for the covering
fraction and an R2 of 0.976 for the height estimation) the lettuce head volume could be
reasonably estimated. However, the computed volume showed to be not suitable to predict
the crop weight. First, this can be explained by the fact that overlapping leaves do not
contribute to coverage or volume. Secondly, in Figure 16 the height over time flattens
during the last 2 weeks, and related to that, in Figure 18 also the volume flattens during the
last days. At the same time, destructive measurements show that the fresh weight grows
especially in these last days. As neither the coverage nor the height and volume indicated
this fresh weight growth, it can be concluded that in the final stage, growth takes place
from the central point of the head, resulting in more compact lettuce heads.
The product of the multiplied area per lettuce head with the maximum height resulted
in the highest correlation coefficient with fresh weight (Appendix A Table A3). Three
papers using the [50–52] dataset had a RMSE up to 25.3. As indicated, we obtained a lower
accuracy, however, we should take into account that the datasets are not fully comparable.
Our dataset is made within the greenhouse, with many plants and resultingly overlapping
leaves. The previous dataset and other research in lettuce growth contained data of single
plants only [53,55]. In our research, the predicted fresh weight was able to determine
non-destructively the moment of harvest for the majority of the teams. The suggested
harvest indicator dates can be closely related to the harvest dates that satisfy the target
weight criterion deducted from the intermediate destructive harvests. For MondayLettuce
no results were derived as the high final density of the team resulted in notably lower
volume for the team (Table 3) that was hampered by the high leaf occlusion.
The light loss indicator proved to be a good and automatically computable parameter
to judge spacing decisions just after the spacing was performed. This hindsight factor is
therefore welcome as an indicator to learn about the suitable covering factor to use as a
Sensors 2023, 23, 2929 22 of 30

threshold for making a spacing step. In Section 3.4.4, based on the light loss indicator, a
covering factor of 98% seemed to be a suitable moment for spacing. The results of teams that
spaced at even higher covering factors correlated with more severe issues with outer leaf
tip burn and malformations and are therefore not advisable. Spacing at lower thresholds
might have given a better quality but would for sure also lead to higher costs per m2 due
to lower average plant densities. Further experience with spacing on a lower threshold
might show that possible higher quality outweighs the additional costs. The Reference and
VeggieMight for example had a light loss indicator that was mostly greater than 10. For
these teams, a later spacing strategy would likely not have had negative consequences.
In the future, other harvest indicators can be explored by deploying spectral indexes
to describe lettuce growth, to address existing shortcomings (overlapping leaves, increased
compactness). Spectral indexes can be successfully linked to the leaf area index in green-
houses [62]. Kizil et al. [63] estimated the yield of lettuce plants using spectral indexes.
Although their solution only worked for single plants it might be an opportunity to explore
further for the purpose of spacing decisions and fresh weight estimation. Also, it is good
to point out that in literature growth from non-destructive measurements is mostly deter-
mined under ‘ceteris paribus’ conditions, meaning that the environment does not change.
In commercial practice and the given dataset, the environment is continually changing due
to different climate, light, and spacing strategies. The latter necessitates the utilization of
larger datasets than those currently accessible. The acquisition of such datasets combined
with the given dataset has the potential to bridge the divide between academic research
and industrial production systems in the future.

5. Conclusions
• In the experiment described here, teams autonomously were able to control greenhouse
lettuce crop production by AI algorithms.
• Autonomous AI algorithms were developed based on greenhouse climate sensor
information in time and on crop images maximizing the net profit of lettuce cultivation.
• Realized crop growth and densities due to timely spacing decisions and realized final
target harvest due to timely estimation of crop weight have shown to have a large
impact on net profit.
• Images from 3D cameras and intelligent computer vision algorithms are helpful to
make timely decisions on plant spacing and final harvest decisions.
• Images of the lettuce crop canopy in the greenhouse have to be related to relevant
crop parameters to predict crop growth. From the images inside the greenhouses over
time, coverage, crop volume, maximum height, and light loss can be calculated to
determine the optimum spacing moment. If the light loss is close to zero, an optimum
spacing moment was reached, in our experiments that were at a coverage of 98%. The
product of area per plant with a maximum height of the plant is a promising indicator
for the moment of harvest given a target weight. Deviations from other destructive
indicators are highly linked to the results of the crop’s architecture as the impact of
leaf occlusion.
• We have shown that computer vision and deep learning algorithms can be used for
automated plant spacing decisions toward the autonomous control of greenhouses.
The provided open-source dataset contributes to another step in the development of
autonomous greenhouses.
• The reality gap between optimum research and commercial production conditions is a
crucial aspect to be considered in computer vision applications. Larger datasets need
to be acquired to bridge the gap.
• Early pest and disease detection, real-time inclusion of the volatile market prices,
robotics in activities of crop handling are among the next steps for higher levels of
automation in horticulture (not part of this research).
Sensors 2023, 23, 2929 23 of 30

Author Contributions: Conceptualization, A.S.P. and S.H.; methodology, S.H., F.d.Z., A.E., B.v.M.
and A.S.P.; software, B.v.M., G.J., T.v.D. and F.d.Z.; validation, F.d.Z., B.v.M. and A.S.P.; formal
analysis, B.v.M. and A.S.P.; investigation, M.B., A.E., F.d.Z., G.J., T.v.D., B.v.M. and A.S.P.; resources,
S.H.; data curation, F.d.Z., B.v.M. and A.S.P.; writing—original draft preparation, A.S.P., B.v.M. and
S.H.; writing—review and editing, F.d.Z., A.E., M.B., G.J. and T.v.D.; visualization, A.S.P. and B.v.M.;
supervision, S.H.; project administration, S.H.; funding acquisition, S.H. All authors have read and
agreed to the published version of the manuscript.
Funding: This organization of the international challenge has been funded by Tencent, with additional
sponsorship by providing materials by Fluence, Gebr. Geers B.V., Sigrow, LetsGrow.com, Ridder,
Hortiplan, Glastuinbouw Nederland, Kas als Energiebron and Gemeente Lansingerland.
Data Availability Statement: The complete dataset of the 3rd Autonomous Greenhouse Challenge:
Time-series data on realized climate with annotated crop lettuce-images is published online at
https://doi.org/10.4121/21960932.v1, (accessed on 2 January 2023).
Acknowledgments: We would like to thank our sponsors and our international jury members.
We also would like to thank the colleagues of our greenhouse facility who carried out part of the
daily crop supervision. Finally, we would like to thank all teams and individual team members
participating in this challenge.
Conflicts of Interest: The authors declare no conflict of interest. The funders had no role in the design
of the study; in the collection, analyses, or interpretation of data; in the writing of the manuscript; or
in the decision to publish the results.

Appendix A

Table A1. Data collected throughout the cultivation cycles for all greenhouse compartments on
outdoor and indoor greenhouse climate, weather forecast, requested and realized operational controls,
weekly destructive plant measurements on which images were annotated, and final harvest plant
data. Data are open access [49,56].

Parameter Unit Intervals Description

Outdoor temperature ◦C 5 min Meteo
Outdoor relative humidity % 5 min Meteo
Global radiation W/m2 5 min Meteo
Wind speed m/s 5 min Meteo
Wind direction - 5 min Meteo
Rain [1 rain–0 dry] 5 min Meteo
Heat emission- pyrgeometer W/m2 5 min Meteo
Measurement

Absolute humidity content 5 min Meteo

Temperature greenhouse ◦C 5 min Indoor climate
Relative humidity greenhouse % 5 min Indoor climate
CO2 concentration greenhouse ppm 5 min Indoor climate
Humidity deficit g/m3 5 min Indoor climate
Leeward vent position % [0–100] 5 min Indoor climate
Windward vent position % [0–100] 5 min Indoor climate
Temperature rail pipe ◦C 5 min Indoor climate
Assimilation lighting (LED) % [0–100] 5 min Indoor climate
Energy screen position % [0–100] 5 min Indoor climate
Blackout screen position % [0–100] 5 min Indoor climate
Sensors 2023, 23, 2929 24 of 30

Table A1. Cont.

Parameter Unit Intervals Description

Cumulative minutes of CO2 dosing minutes 5 min Indoor climate
Heating temperature ◦C 5 min Indoor climate
Outdoor temperature ◦C 5 min Meteo
Outdoor relative humidity % 5 min Meteo
Forecast

Global radiation W/m2 5 min Meteo

Wind speed m/s 5 min Meteo
Degree of cloudiness [1–8] 5 min Meteo
Ventilation temperature ◦C 5 min Indoor climate
Lee side min vent position % [0–100] 5 min Indoor climate
Net pipe minimum ◦C 5 min Indoor climate
Control

Energy screen % [0–100] 5 min Indoor climate

Blackout screen % [0–100] 5 min Indoor climate
CO2 Ppm 5 min Indoor climate
Humidity deficit g/m3 5 min Indoor climate
A class harvest g At harvest >250 g
B class harvest g At harvest 220–250 g
<220 g or visible
C class harvest g At harvest
malformations
92 plants/m2 at
Plant density #/m2 Team dependent
transplanting
Days after
Day of harvest Once Team dependent
transplanting
Weekly sampled plants
Weekly/At
Height cm and at harvest day which
harvest
was team dependent
Crop

Weekly sampled plants

Weekly/At
Diameter cm and at harvest day which
harvest
was team dependent
Weekly sampled plants
Weekly/At
Fresh Weight g and at harvest day which
harvest
was team dependent
Weekly sampled plants
Weekly/At
Dry Weight g and at harvest day which
harvest
was team dependent
Weekly sampled plants
and at harvest day which
Weekly/At
Leaf deformation [1–3] was team dependent.
harvest
Scoring protocol 1–3,
applies to a head of lettuce
End of each Annotated single crop and
RGB, depth images -
cultivation canopy images
Sensors 2023, 23, 2929 25 of 30

Sensors 2023, 23, x FOR PEER REVIEW 25 of 30

Sensors 2023, 23, x FOR PEER REVIEW 25 of

Table A2. Actuators and defaul sensors installed in all the greenhouse compartments during the30
growing cycles and description of the installed equipment.
Table A2. Actuators and defaul sensors installed in all the greenhouse compartments during the
growing cycles and description
Greenhouse of the installed equipment.
Compartments Description
Table A2. Actuators and defaul sensors installed in all the greenhouse compartments during the
growing cycles and description of the
Greenhouse Compartments Railinstalled
pipe equipment. Max capacity 129 W/m2
Description
Greenhouse Rail pipe
Compartments Energy screen Max capacity
LUXOUS129 W/m
Description 1547 2D FR, Ludvig Svensson
Energy screen
Rail pipe BlackoutLUXOUS 1547
screen Max DOBSCURA
FR, Ludvig
capacity 129 W/m Svensson
9950 2FR W, Ludvig Svensson

Blackout screen
Energy screen OBSCURA
LUXOUS 15479950 DFRFR,W,Ludvig
Ludvig
Dimming Svensson
Svensson
27–270 µmol/m2 /s with
Blackout screen DimmingLED 27–270
lightsµmol/m
OBSCURA
2/s with
9950 FR W,efficiency
Ludvig
efficiency 2.4 µmol/J,
Svensson
2.4 µmol/J, VYPR
VYPR 2p, 2p,
Fluence
LED lights
Fluence by Osram by Osram
Dimming 27–270 µmol/m /s with efficiency 2.4 µmol/J, VYPR 2p,
2
LED lights
Fogging 330 g/m 2/h 330 g/m2 /h
Equipment

Fogging Fluence by Osram

CO 2 supply
Fogging CO2 supply Max capacity330 g/m15 2/hg/mcapacity
Max
2/h 15 g/m2 /h
Equipment

Equipment
CO2 supply Length 3.2 m, 30 plantMaxholes, 11 cm
capacity heart-to-heart
15 g/m distance,
2/h30 plant holes, 1110
cmcm
Hydroponic gutters (NFT) Length 3.2 m,
Length 3.2 m,gutters
Hydroponic 30 plant
(NFT)wide,11Hortiplan
holes, cm heart-to-heart
heart-to-heart distance,
distance, 10 cm
10 cm wide,
Hydroponic gutters (NFT) Indoor temperature, relative humidity and CO 2 sensor in ventilated
Hortiplan
wide, Hortiplan
Measuring box measuring
Indoor box placed
temperature, in the humidity
relative middle
Indooroftemperature,
the compartment
and CO2 sensor in above
relative the
ventilated
humidity
Measuring box measuring box placed
Measuring growing
and CO crop
2 sensor in ventilated measuring
box in the middle of the compartment above the
box placed in the middle of the
PAR sensor PAR sensor placed above canopy
growing cropand below LED lights
compartment above the growing crop
RGB,PAR
depth camera
sensor Depth Camera D415—Intel RealSense
PAR sensor placed above canopy and below LED lights
PAR sensor placed above canopy and
PAR sensor
RGB, depth camera Depth Camera D415—Intel RealSense
below LED lights
RGB, depth camera Depth Camera D415—Intel RealSense

Figure A1. Heating energy consumption (MJ/m2) of all compartments during the first and second
Figure A1. Heating energy consumption (MJ/m2 ) of all compartments during the first and second
cultivation.
Figure A1. Heating energy consumption (MJ/m2) of all compartments during the first and second
cultivation.
cultivation.

Figure A2. Energy screen usage in all compartments during the first and second cultivation.
Figure A2. Energy screen usage in all compartments during the first and second cultivation.
Figure A2. Energy screen usage in all compartments during the first and second cultivation.
Sensors 2023, 23, 2929 26 of 30

Table A3. Realized harvest dates and fresh weight at the moment of harvest along with the dates at
which the weight target was realized by interpolating linearly the weekly destructive measured data.

Harvest Date Average FW

Number of Average FW
Realized Satisfying Satisfying
Cultivation at Realized
Compartment Harvest Date the FW the FW
Days Harvest
[dd/mm] Criterion Criterion
[Days] [g/Head]
[dd/mm] [g/Head]
Reference 9 June 38 271.18 8 June 258.10
Koala 17 June 46 402.81 9 June 260.50
CVA 13 June 42 342.06 7 June 265.35
MondayLettuce 14 June 43 294.96 9 June 254.02
DigitalCucumbers 15 June 44 390.85 7 June 260.61
VeggieMight 13 June 43 389.80 4 June 251.91

Table A4. Correlation coefficient between measured and predicted fresh weight using corresponding
parameters derived using RGB and depth image.

Parameters Correlation Coefficient

Coverage percentage 0.5392
Average height [cm] 0.6953
Median height [cm] 0.6946
Max height [cm] 0.7606
Volume [cm3 ] 0.6785
Head density −0.7912
Volume per plant [cm3 /head] 0.8975
Area per plant [cm2 ] 0.8987
Mm per pixel −0.6784
Area per plant divided by volume per plant −0.4801
Volume per plant divided by area per plant 0.6741
Area per plant multiplied by volume per plant 0.9214
Area per plant divided by mm per pixel 0.9048
Area per plant divided by the maximum height 0.8126
Area per plant divided by median height 0.8400
Area per plant divided by average height 0.8360
Area per plant multiplied by the maximum
0.9340
height
Area per plant multiplied by the median height 0.9048
Area per plant multiplied by average height 0.9065
Sensors 2023, 23, x FOR PEER REVIEW 26 of 30

Sensors 2023, 23, x FOR PEER REVIEW 26 of 30

Sensors 2023, 23, x FOR PEER REVIEW 26 of 30

Sensors 2023, 23, 2929 27 of 30

Figure A3. Blackout screen usage in all compartments during the first and second cultivation.
Figure A3. Blackout screen usage in all compartments during the first and second cultivation.
Figure A3. Blackout screen usage in all compartments during the first and second cultivation.
Figure A3. Blackout screen usage in all compartments during the first and second cultivation.

Figure A4. Artificial lighting usage in all compartments during the first and second cultivation.
Figure A4. Artificial lighting usage in all compartments during the first and second cultivation.
Figure A4. Artificial lighting usage in all compartments during the first and second cultivation.
Figure A4. Artificial lighting usage in all compartments during the first and second cultivation.

Figure A5. Electricity consumption (kWh/m2 ) in all compartments during the first and second
Figure A5. Electricity consumption (kWh/m2) in all compartments during the first and second
cultivation.
cultivation.
Figure A5. Electricity consumption (kWh/m2) in all compartments during the first and second
cultivation.
Figure A5. Electricity consumption (kWh/m2) in all compartments during the first and second
cultivation.
 Sensors 2023, 23, x FOR PEER REVIEW 28 of 30
Sensors 2023, 23, 2929 28 of 30

100 Class A
Class B
Class C
80

60

40

20

0
t
CV
A ers ala u ce
en
ce gh
um
b Ko ett fer ie Mi
uc yL Re gg
alc da Ve
gi t M on
Di

Figure A6. Classification of lettuces at the different harvest moments as Class A, Class B, and Class C
Figure A6.
lettuces Classification
for the of lettuces
first (left) and secondat(right)
the different harvest
cultivation. Themoments as Class
classification wasA, Class B,
carried outand Class
using a
C lettuces for the first (left) and second (right) cultivation. The classification was carried out using
standardized protocol that distinguished the harvested products given the satisfaction of the target
a standardized protocol that distinguished the harvested products given the satisfaction of the
fresh weight, the presence, and severity of the outer-leaf burn, and the presence of diseases and or
target fresh weight, the presence, and severity of the outer-leaf burn, and the presence of diseases
malformations that resulted
and or malformations in non-sellable
that resulted products.
in non-sellable products.

References
References
1. van Dijk, M.; Morley, T.; Rau, M.L.; Saghai, Y. A meta-analysis of projected global food demand and population at risk of hunger
1. van Dijk, M.; Morley, T.; Rau, M.L.; Saghai, Y. A meta-analysis of projected global food demand and population at risk of hunger
for the period 2010–2050. Nat. Food 2021, 2, 494–501. [CrossRef]
for the period 2010–2050. Nat. Food 2021, 2, 494–501. https://doi.org/10.1038/s43016-021-00322-9.
2. Foley, J.A.; Ramankutty, N.; Brauman, K.A.; Cassidy, E.S.; Gerber, J.S.; Johnston, M.; Mueller, N.D.; O’Connell, C.; Ray, D.K.; West,
2. Foley, J.A.; Ramankutty, N.; Brauman, K.A.; Cassidy, E.S.; Gerber, J.S.; Johnston, M.; Mueller, N.D.; O’Connell, C.; Ray, D.K.;
P.C.; et al. Solutions for a cultivated planet. Nature 2011, 478, 337–342. [CrossRef]
West, P.C.; et al. Solutions for a cultivated planet. Nature 2011, 478, 337–342. https://doi.org/10.1038/nature10452.
3. Aznar-Sánchez, J.A.; Velasco-Muñoz, J.F.; López-Felices, B.; Román-Sánchez, I.M. An Analysis of Global Research Trends on
3. Aznar-Sánchez, J.A.; Velasco-Muñoz, J.F.; López-Felices, B.; Román-Sánchez, I.M. An Analysis of Global Research Trends on
Greenhouse Technology: Towards a Sustainable Agriculture. Int. J. Environ. Res. Public Health 2020, 17, 664. [CrossRef] [PubMed]
Greenhouse Technology: Towards a Sustainable Agriculture. Int. J. Environ. Res. Public Health 2020, 17, 664.
4. Stanghellini, C. Horticultural production in greenhouses: Efficient use of water. Acta Hortic. 2014, 1034, 25–32. [CrossRef]
https://doi.org/10.3390/ijerph17020664.
5. Vegetables; Yield and Cultivated Area per Kind of Vegetable. 2021. Available online: https://www.cbs.nl/en-gb/figures/detail/
4. Stanghellini, C. Horticultural production in greenhouses: Efficient use of water. Acta Hortic. 2014, 1034, 25–32.
37738ENG (accessed on 5 September 2021).
https://doi.org/10.17660/actahortic.2014.1034.1.
6. Verdouw, C.; Robbemond, R.; Kruize, J.W. Integration of Production Control and Enterprise Management Systems in Horticulture.
5. Vegetables; Yield and Cultivated Area per Kind of Vegetable. 2021. Available online: https://www.cbs.nl/en-gb/figures/de-
In Proceedings of the 7th International Conference on Information and Communication Technologies in Agriculture, Food and
tail/37738ENG (accessed on 5 September 2021).
Environment (HAICTA 2015), Kavala, Greece, 17–20 September 2015; pp. 124–135.
6. Verdouw, C.; Robbemond, R.; Kruize, J.W. Integration of Production Control and Enterprise Management Systems in Horticul-
7. Payne, H.J.; Hemming, S.; van Rens, B.A.; van Henten, E.J.; van Mourik, S. Quantifying the role of weather forecast error on the
ture. In Proceedings of the 7th International Conference on Information and Communication Technologies in Agriculture, Food
uncertainty of greenhouse energy prediction and power market trading. Biosyst. Eng. 2022, 224, 1–5. [CrossRef]
and Environment (HAICTA 2015), Kavala, Greece, 17–20 September 2015; pp. 124–135.
8. Verdouw, C.; Bondt, N.; Schmeitz, H.; Zwinkels, H. Towards a Smarter Greenport: Public-Private Partnership to Boost Digital
7. Payne, H.J.; Hemming, S.; van Rens, B.A.; van Henten, E.J.; van Mourik, S. Quantifying the role of weather forecast error on the
Standardisation and Innovation in the Dutch Horticulture. Int. J. Food Syst. Dyn. 2014, 5, 44–52. [CrossRef]
uncertainty of greenhouse energy prediction and power market trading. Biosyst. Eng. 2022, 224, 1–5.
9. Tzounis, A.; Katsoulas, N.; Bartzanas, T.; Kittas, C. Internet of Things in agriculture, recent advances and future challenges.
8. Verdouw, C.; Bondt, N.; Schmeitz, H.; Zwinkels, H. Towards a Smarter Greenport: Public-Private Partnership to Boost Digital
Biosyst. Eng. 2017, 164, 31–48. [CrossRef]
Standardisation and Innovation in the Dutch Horticulture. Int. J. Food Syst. Dyn. 2014, 5, 44–52.
10. Wolfert, S.; Ge, L.; Verdouw, C.; Bogaardt, M.J. Big data in smart farming—A review. Agric. Syst. 2017, 153, 69–80. [CrossRef]
https://doi.org/10.18461/ijfsd.v5i1.515.
11. Kamilaris, A.; Prenafeta-Boldú, F.X. Deep learning in agriculture: A survey. Comput. Electron. Agric. 2018, 147, 70–90. [CrossRef]
9. Tzounis, A.; Katsoulas, N.; Bartzanas, T.; Kittas, C. Internet of Things in agriculture, recent advances and future challenges.
12. Zhai, Z.; Martínez, J.F.; Beltran, V.; Martínez, N.L. Decision support systems for agriculture 4.0: Survey and challenges. Comput.
Biosyst. Eng. 2017, 164, 31–48.
Electron. Agric. 2020, 170, 105256. [CrossRef]
10. Wolfert, S.; Ge, L.; Verdouw, C.; Bogaardt, M.J. Big data in smart farming—A review. Agric. Syst. 2017, 153, 69–80.
13. Verdouw, C.; Tekinerdogan, B.; Beulens, A.; Wolfert, S. Digital twins in smart farming. Agric. Syst. 2021, 189, 103046. [CrossRef]
11. Kamilaris, A.; Prenafeta-Boldú, F.X. Deep learning in agriculture: A survey. Comput. Electron. Agric. 2018, 147, 70–90.
14. Marshall-Colon, A.; Long, S.P.; Allen, D.K.; Allen, G.; Beard, D.A.; Benes, B.; Von Caemmerer, S.; Christensen, A.; Cox, D.J.; Hart,
https://doi.org/10.1016/j.compag.2018.02.016.
J.C.; et al. Crops In Silico: Generating Virtual Crops Using an Integrative and Multi-scale Modeling Platform. Front. Plant Sci.
12. Zhai, Z.; Martínez, J.F.; Beltran, V.; Martínez, N.L. Decision support systems for agriculture 4.0: Survey and challenges. Comput.
2017, 8, 786. [CrossRef] [PubMed]
Electron. Agric. 2020, 170, 105256.
15. Tzachor, A.; Richards, C.E.; Jeen, S. Transforming agrifood production systems and supply chains with digital twins. Npj Sci. Food
13. Verdouw, C.; Tekinerdogan, B.; Beulens, A.; Wolfert, S. Digital twins in smart farming. Agric. Syst. 2021, 189, 103046.
2022, 6, 1–5. [CrossRef]
https://doi.org/10.1016/j.agsy.2020.103046.
16. Buwalda, F.; Van Henten, E.J.; De Gelder, A.; Bontsema, J.; Hemming, J. Toward an optimal control strategy for sweet pepper
14. Marshall-Colon,
cultivation-1. A.; Long,
A dynamic S.P.;
crop Allen,Acta
model. D.K.; Allen,
Hortic. G.; 718,
2006, Beard, D.A.; Benes,
367–374. B.; Von Caemmerer, S.; Christensen, A.; Cox, D.J.;
[CrossRef]
17. Hart, J.C.; et al. Crops In Silico: Generating Virtual Crops Using an Integrative
Tchamitchian, M.; Henry-Montbroussous, B.; Jeannequin, B.; Lagier, J. Serriste: Climate and Multi-scale
set-pointModeling Platform.
determination Front. Plant
for greenhouse
Sci. 2017, 8, 786. https://doi.org/10.3389/fpls.2017.00786.
tomatoes. Acta Hortic. 1998, 456, 321–328. [CrossRef]
15.
18. Tzachor, A.;
Kolokotsa, D.;Richards,
Saridakis,C.E.; Jeen, S. Transforming
G.; Dalamagkidis, agrifood
K.; Dolianitis, S.; production
Kaliakatsos,systems and supply
I. Development chains
of an with digital
intelligent indoortwins. Npj Sci.
environment
Food 2022, 6, 1–5. https://doi.org/10.1038/s41538-022-00162-2.
and energy management system for greenhouses. Energy Convers. Manag. 2010, 51, 155–168. [CrossRef]
16.
19. Buwalda,
de Zwart, H.F.F.; Van Henten,
Analyzing E.J.; De Gelder,
Energy-Saving A.; Bontsema,
Options J.; Hemming,
in Greenhouse J. Toward
Cultivation Using a an optimal Model.
Simulation control Ph.D.
strategy for sweet
Thesis, pepper
Wageningen
cultivation-1. A dynamic crop model.
University, Wageningen, The Netherlands, 1996. Acta Hortic. 2006, 718, 367–374.
17.
20. Tchamitchian,
Marcelis, M.; Henry-Montbroussous,
L.; Elings, B.; Jeannequin,
A.; De Visser, P.; Heuvelink, B.; Lagier,
E. Simulating growthJ.and
Serriste: climate set-point
development of tomatodetermination for greenhouse
crop. Acta Hortic. 2009, 821,
tomatoes. Acta
101–110. [CrossRef] Hortic. 1998, 456, 321–328. https://doi.org/10.17660/actahortic.1998.456.38.
Sensors 2023, 23, 2929 29 of 30

21. Hemming, S.; de Zwart, F.; Elings, A.; Righini, I.; Petropoulou, A. Remote control of greenhouse vegetable production with
artificial intelligence—Greenhouse climate, irrigation, and crop production. Sensors 2019, 19, 1807. [CrossRef]
22. Hemming, S.; Zwart, F.D.; Elings, A.; Petropoulou, A.; Righini, I. Cherry tomato production in intelligent greenhouses—Sensors
and AI for control of climate, irrigation, crop yield, and quality. Sensors 2020, 20, 6430. [CrossRef]
23. Hamon, R.; Junklewitz, H.; Sanchez, I. Robustness and Explainability of Artificial Intelligence; EUR 30040 EN; Publications Office of
the European Union: Luxembourg, 2020; ISBN 978-92-76-14660-5. [CrossRef]
24. Ciaglia Anne Bennett. High Tech Growing Systems Help Improve Efficiencies and Meet Consumer Demand. 2017. Available online: https:
//gpnmag.com/article/automation-high-tech-growing-systems-help-improve-efficiencies-meet-consumer-demand/ (accessed
on 10 October 2022).
25. Savvas, D.; Passam, H. Hydroponic Production of Vegetables and Ornamentals; Embryo Publications: Athens, Greece, 2002; p. 463.
26. Jones, J.B., Jr. Hydroponics: A Practical Guide for the Soilless Grower; CRC Press: Boca Ratos, FL, USA, 2016.
27. Resh, H.M. Hydroponic Food Production: A Definitive Guidebook for the Advanced Home Gardener and the Commercial Hydroponic
Grower, 7th ed.; CRC Press: Boca Ratos, FL, USA, 2016.
28. van Treuren, R.; van Eekelen, H.D.; Wehrens, R.; de Vos, R.C. Metabolite variation in the lettuce gene pool: Towards healthier
crop varieties and food. Metabolomics 2018, 14, 1–4. [CrossRef]
29. Masarirambi, M.T.; Nxumalo, K.A.; Musi, P.J.; Rugube, L.M. Common physiological disorders of lettuce (Lactuca sativa) found in
Swaziland: A review. Am.-Eurasian J. Agric. Environ. Sci. 2018, 18, 50–56.
30. Martinetti, L.; Ferrante, A.; Penati, M. Influenza della concimazione sulla produzione quanti-qualitativa di ortaggi baby leaf per
la quarta gamma in coltivazione biologica e convenzionale. La Riv. Di Sci. Dell’alimentazione 2009, 38, 23–33.
31. Scuderi, D.; Giuffrida, F.; Noto, G. Effects of salinity and plant density on quality of lettuce grown in floating system for fresh-cut.
Acta Hortic. 2009, 843, 219–226. [CrossRef]
32. Mengistu, F.G.; Tabor, G.; Dagne, Z.; Atinafu, G.; Tewolde, F.T. Effect of planting density on yield and yield components of lettuce
(Lactuca sativa L.) at two agro-ecologies of Ethiopia. Afr. J. Agric. Res. 2021, 17, 549–556.
33. Ojo, M.O.; Zahid, A. Deep Learning in Controlled Environment Agriculture: A Review of Recent Advancements, Challenges and
Prospects. Sensors 2022, 22, 7965. [CrossRef]
34. Javaid, M.; Haleem, A.; Khan, I.H.; Suman, R. Understanding the potential applications of Artificial Intelligence in Agriculture
Sector. Adv. Agrochem 2022. [CrossRef]
35. Mishra, P.; Polder, G.; Vilfan, N. Close Range Spectral Imaging for Disease Detection in Plants Using Autonomous Platforms: A
Review on Recent Studies. Curr. Robot. Rep. 2020, 1, 43–48. [CrossRef]
36. Nieuwenhuizen, A.T.; Kool, J.; Suh, H.K.; Hemming, J. Automated spider mite damage detection on tomato leaves in greenhouses.
Acta Hortic. 2020, 1268, 165–172. [CrossRef]
37. Suh, H.K.; Ijsselmuiden, J.; Hofstee, J.W.; van Henten, E.J. Transfer learning for the classification of sugar beet and volunteer
potato under field conditions. Biosyst. Eng. 2018, 174, 50–65. [CrossRef]
38. Rahnemoonfar, M.; Sheppard, C. Deep Count: Fruit Counting Based on Deep Simulated Learning. Sensors 2017, 17, 905. [CrossRef]
39. Fonteijn, H.; Afonso, M.; Lensink, D.; Mooij, M.; Faber, N.; Vroegop, A.; Polder, G.; Wehrens, R. Automatic Phenotyping of
Tomatoes in Production Greenhouses Using Robotics and Computer Vision: From Theory to Practice. Agronomy 2021, 11, 1599.
[CrossRef]
40. Nishina, H. Development of Speaking Plant Approach Technique for Intelligent Greenhouse. Agric. Agric. Sci. Procedia 2015, 3,
9–13. [CrossRef]
41. Bac, C.W. Improving Obstacle Awareness for Robotic Harvesting of Sweet-Pepper. Ph.D. Thesis, Wageningen University,
Wageningen, The Netherlands, 2015.
42. Barth, R. Vision Principles for Harvest Robotics: Sowing Artificial Intelligence in Agriculture. Ph.D. Thesis, Wageningen
University, Wageningen, The Netherlands, 2018. [CrossRef]
43. Paulus, S. Measuring crops in 3D: Using geometry for plant phenotyping. Plant Methods 2019, 15, 1–13. [CrossRef]
44. Tian, H.; Wang, T.; Liu, Y.; Qiao, X.; Li, Y. Computer vision technology in agricultural automation—A review. Inf. Process. Agric.
2019, 7, 1–19. [CrossRef]
45. Tian, Z.; Ma, W.; Yang, Q.; Duan, F. Application status and challenges of machine vision in plant factory—A review. Inf. Process.
Agric. 2021, 9, 195–211. [CrossRef]
46. Wu, H.; Xiao, B.; Codella, N.; Liu, M.; Dai, X.; Yuan, L.; Zhang, L. Cvt: Introducing convolutions to vision transformers.
In Proceedings of the IEEE/CVF International Conference on Computer Vision, Montreal, BC, Canada, 11–17 October 2021;
pp. 22–31.
47. Nilsback, M.-E.; Zisserman, A. Automated Flower Classification over a Large Number of Classes. In Proceedings of the 2008 Sixth
Indian Conference on Computer Vision, Graphics & Image Processing, Bhubaneswar, India, 16–19 December 2008; pp. 722–729.
[CrossRef]
48. Valenzuela, I.C.; Puno, J.C.V.; Bandala, A.A.; Baldovino, R.G.; de Luna, R.G.; De Ocampo, A.L.; Cuello, J.; Dadios, E.P. Quality
assessment of lettuce using artificial neural network. In Proceedings of the 2017 IEEE 9th International Conference on Humanoid,
Nanotechnology, Information Technology, Communication and Control, Environment and Management (HNICEM), Manila,
Philippines, 1–3 December 2017; pp. 1–5. [CrossRef]
Sensors 2023, 23, 2929 30 of 30

49. Hemming, S.; de Zwart, F.; Elings, A.; Bijlaard, M.; van Marrewijk, B.; Petropoulou, A. 3rd Autonomous Greenhouse Challenge: Online
Challenge Lettuce Images; Dataset: 4TU.ResearchData. 2021. Available online: https://doi.org/10.4121/15023088.v1 (accessed on 2
January 2023).
50. Lin, Z.; Fu, R.; Ren, G.; Zhong, R.; Ying, Y.; Lin, T. Automatic monitoring of lettuce fresh weight by multi-modal fusion based
deep learning. Front. Plant Sci. 2022, 13. [CrossRef] [PubMed]
51. Gang, M.-S.; Kim, H.-J.; Kim, D.-W. Estimation of Greenhouse Lettuce Growth Indices Based on a Two-Stage CNN Using RGB-D
Images. Sensors 2022, 22, 5499. [CrossRef] [PubMed]
52. Zhang, Y.; Li, M.; Li, G.; Li, J.; Zheng, L.; Zhang, M.; Wang, M. Multi-phenotypic parameters extraction and biomass estimation
for lettuce based on point clouds. Measurement 2022, 204, 112094. [CrossRef]
53. Lu, J.-Y.; Chang, C.-L.; Kuo, Y.-F. Monitoring Growth Rate of Lettuce Using Deep Convolutional Neural Networks. In Proceedings
of the 2019 ASABE Annual International Meeting, Boston, MA, USA, 7–10 July 2019. [CrossRef]
54. Bauer, A.; Bostrom, A.G.; Ball, J.; Applegate, C.; Cheng, T.; Laycock, S.; Rojas, S.M.; Kirwan, J.; Zhou, J. Combining computer
vision and deep learning to enable ultra-scale aerial phenotyping and precision agriculture: A case study of lettuce production.
Hortic. Res. 2019, 6, 70. [CrossRef] [PubMed]
55. Du, J.; Lu, X.; Fan, J.; Qin, Y.; Yang, X.; Guo, X. Image-Based High-Throughput Detection and Phenotype Evaluation Method for
Multiple Lettuce Varieties. Front. Plant Sci. 2020, 11, 563386. [CrossRef]
56. Petropoulou, A.; van Marrewijk, B.; Hemming, S.; de Zwart, F.; Elings, A.; Bijlaard, M. 3rd Autonomous Greenhouse Challenge-Real
Challenge Data Climate and Images. Dataset: 4TU.ResearchData. 2023. Available online: https://data.4tu.nl/articles/dataset/3rd_
Autonomous_Greenhouse_Challenge_Online_Challenge_Lettuce_Images/15023088 (accessed on 2 January 2023).
57. Intel ®Depth Camera D415–Intel®RealSenseTM Depth and Tracking Cameras. Available online: https://www.intelrealsense.
com/depth-camera-d415/ (accessed on 14 October 2022).
58. Chen, L.C.; Zhu, Y.; Papandreou, G.; Schroff, F.; Adam, H. Encoder-decoder with atrous separable convolution for semantic image
segmentation. In Proceedings of the European Conference on Computer Vision (ECCV), Munich, Germany, 8–14 September 2018;
pp. 801–818.
59. Fischler, M.A.; Bolles, R.C. Random Sample Consensus: A Paradigm for Model Fitting with Applications to Image Analysis and
Automated Cartography. Commun. ACM 1981, 24, 381–395. [CrossRef]
60. Gray, D.; Steckel, J.R. Hearting and mature head characteristics of lettuce (Lactuca sativa L.) as affected by shading at different
periods during growth. J. Hortic. Sci. 1981, 56, 199–206. [CrossRef]
61. Aikman, D.P.; Benjamin, R. A model for plant and crop growth, allowing for competition for light by the use of potential and
restricted crown zone areas. Ann. Bot. 1994, 73, 185–194. [CrossRef]
62. Sarlikioti, V.; Meinen, E.; Marcelis, L. Crop Reflectance as a tool for the online monitoring of LAI and PAR interception in two
different greenhouse Crops. Biosyst. Eng. 2011, 108, 114–120. [CrossRef]
63. Kizil, Ü.; Genc, L.; Inalpulat, M.; Şapolyo, D.; Mirik, M. Lettuce (Lactuca sativa L.) yield prediction under water stress using
artificial neural network (ANN) model and vegetation indices. Žemdirbystė=Agric. 2012, 99, 409–418.

Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual
author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to
people or property resulting from any ideas, methods, instructions or products referred to in the content.