2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)

October 23-27, 2022, Kyoto, Japan

Towards Autonomous Visual Navigation in Arable Fields

Alireza Ahmadi, Michael Halstead, and Chris McCool

Abstract— Autonomous navigation of a robot in agricultural

fields is essential for every task from crop monitoring to
weed management and fertilizer application. Many current
approaches rely on accurate GPS, however, such technology
2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) | 978-1-6654-7927-1/22/$31.00 ©2022 IEEE | DOI: 10.1109/IROS47612.2022.9981299

is expensive and can be impacted by lack of coverage. As

such, autonomous navigation through sensors that can interpret
their environment (such as cameras) is important to achieve the
goal of autonomy in agriculture. In this paper, we introduce a
purely vision-based navigation scheme that is able to reliably
guide the robot through row-crop fields using computer vision
and signal processing techniques without manual intervention.
Independent of any global localization or mapping, this ap-
proach is able to accurately follow the crop-rows and switch
between the rows, only using onboard cameras. The proposed
navigation scheme can be deployed in a wide range of fields
with different canopy shapes in various growth stages, creating
a crop agnostic navigation approach. This was completed under
various illumination conditions using simulated and real fields
where we achieve an average navigation accuracy of 3.82cm Fig. 1: BonnBot-I following lanes of crops using two sym-
with minimal human intervention (hyper-parameter tuning) on
BonnBot-I. metrically mounted monocular camera in front and back.
Keywords — Robotics and Automation in Agriculture and
Forestry; Agricultural Automation; Vision-Based Navigation.
In our prior work [4] we introduced a GPS independent
technique that was able to traverse a single crop-row and
I. I NTRODUCTION switch between adjacent rows. This system was deployed
on a Husky and tested using a single artificial cropping
Novel agricultural robotic technologies need to ensure they environment (including artificial plants). Furthermore, the
meet the needs of the key stakeholder, farmers. Usually, this prior work was only applicable to navigating along a single
means being cost effective both as a platform and as a labor crop-row and did not take advantage of the overall planting
(or mechanical) replacement technology. However, another structure in the field. For instance, row-crops are planted in
critical element is that the technology can be deployed to a parallel lines and this structure can be exploited to overcome
variety of fields and environmental conditions with minimal issues such as a lack of germination, which can leave gaps
intervention (supervision) from the farmer. In particular, in a crop-row, creating navigation issues for single crop-row
navigating through the field is central to the real-world techniques. Such issues can lead to unrecoverable failure
deployment of agricultural robotics. cases for a single-crop-row navigation system, see Fig. 2.
Hence, automated field navigation for a robot is essential In this paper we introduce a novel crop-agnostic vision
for every task. For autonomous navigation it is common for based navigation scheme as illustrated in Fig. 1. We greatly
platforms to use precise real-time kinematic (RTK) GNSS as extend our prior work [4] and use information from multiple
it is being used in fully controlled and engineered agricultural crop-rows to navigate a weeding robotic platform in five
sites where they heavily rely on structural information [1], different real fields under various weather conditions. The
[2]. But, robotic technologies will never be guaranteed accu- proposed approach takes advantage of planting schemes
rate GPS in every field, first due to its expense and second (standardized distances between rows of crops), ensures
because of the coverage. An example of this is that currently minimum damage to crops and persistent coverage through-
most fields are seeded using traditional methods, not auto- out the field. The proposed approach relies on well-known
seeding geo-referenced systems, which creates a gap between signal processing and computer vision approaches, however,
GPS capabilities and farming requirements [1]. As such, their combination is unique and their robustness has been
utilizing GPS technology in unregistered fields increases the demonstrated by being applied to five crop types on a
risk of damaging crops [3]. In a worst-case scenario GPS robot in the field. We consider this approach as an extra
can fail, therefore, generally applicable crop-row following navigation controller modality for achieving more reliable
techniques are required. traverses in challenging real fields conditions and not denying
the potential of benefiting other technologies like GPS and
All authors are with the University of Bonn, Bonn 53115
Germany. {alireza.ahmadi, michael.halstead, odometry which represents a complete system. To achieve
cmccool}@uni-bonn.de this we make the following novel contributions:

978-1-6654-7927-1/22/$31.00 ©2022 IEEE 6585

Authorized licensed use limited to: LAHORE UNIV OF MANAGEMENT SCIENCES. Downloaded on April 06,2023 at 06:46:45 UTC from IEEE Xplore. Restrictions apply.
 • a robust multi-crop-row detection strategy, capable of
dealing with cluttered and weedy scenes for a variety
of crop types; and
• a multiple crop-row following approach which enables
autonomous guidance in real-fields; and
• a multi-crop-row lane switching strategy which enables
BonnBot-I to switch to a new lane independent of any
Fig. 2: Example images where crops have yet to germinate
global positioning system; and
leaving large gaps in a row, indicated by red boxes.
• deployed and evaluated on a real-sized weed manage-
ment robot in simulation and the field for five crop types deep neural networks (DNN). Also utilising DNNs, [20] pro-
and (with different shapes: straight and curved crop- posed a convolutional neural network (CNN) for strawberry
rows); and crop-row detection with accurate navigation. Lin et al. [21]
• releasing a novel crop-row dataset covering five crop also showcase the potential of CNNs to reliably navigate
types with various growth stages under varying illumi- a tea field by classifying tea rows. These approaches are
nation conditions. often more accurate than their traditional counterparts for
detecting or segmenting specific plants. However, in contrast
II. R ELATED W ORK to traditional approaches, CNNs require a significant amount
of labeled data and more computational resources both for
Autonomous agricultural robots could improve productiv- training and inference, while being less dynamic in nature
ity [5], [6], enable targeted field interventions [7] and facili- and requiring further tuning in different conditions. [22].
tate crop monitoring [8]. For uptake of these platforms they To perform vision based navigation two common ap-
should be deploy-able in a variety of scenarios including dif- proaches exist: proportional-integral-derivative (PID) or vi-
ferent cultivars, crop-row structures, and seeding patterns [9]. sual servoing. Billingsley et al. [12] extracted the row of
A key enabling technology is reliable navigation through the plants using a Hough transform, and used a PID con-
whole field [10]. One potential solution to the navigation troller for navigation through a sugar-beet field. The visual-
problem is to make use of the Global Navigation Satellite servoing [23] technique was also exploited for autonomous
System (GNSS). Such an approach has been used for both car guidance in urban areas [24] by road lane following with
agricultural machinery [11] and robotic platforms [2]. The obstacle avoidance using monocular camera. These methods
downside of this approach is that it relies on an expensive regularize the controlled agents motion within a loop directly
sensor and suffers from limitations such as possible GNSS based on the current visual features.
outages and reliance on geo-referenced auto-seeding. Thus, The technique proposed in this paper draws inspiration
crop-based navigation techniques leveraging the field struc- from our previous work [4] where we are able to both navi-
ture were investigated for autonomous guidance [12], [4] and gate a single crop-row and switch lanes at the end. However,
in-field interventions [13]. this approach was only tested in a single artificially created
In an attempt to use the structure of a field, Barawid et row-crop field, without considering real open-field challenges
al. [14] investigated LiDAR based orchards navigation sys- like (different crop types, illumination variation, appearance
tem a similar strategy was used in a simulated environment of weeds and uneven distribution of plants in rows). We
by [15] for traversing row-crop fields. In addition, Winterhal- propose a real-field applicable method to automatically detect
ter et al. [16] proposed sensor-independent feature represen- the number of crop rows. Multi-crop-row switching is then
tation from different sensor modalities and detects crop rows. enabled by being able to correctly identify new crops as the
They used LiDAR and camera images to extract single lines robot moves across the field. As real-time performance is
in a row-crop field which were spaced equidistantly. While important our methods rely on traditional machine vision
these approaches enable side-applications such as obstacle techniques which we deploy on a real agricultural robot
avoidance, frame drift in self-similar environments can cause (BonnBot-I) to autonomously navigate in real fields with five
issues [17], including crop damage. different crops and three simulated fields. We also made its
To avoid crop damage through GNSS or LiDAR failures, implementation publicly available 1 .
RGB based navigation approaches directly exploit the avail-
able visual information. These techniques can vary signifi- III. AUTONOMOUS NAVIGATION IN ROW-C ROP F IELDS
cantly in terms of cost, algorithm simplicity, and availability.
Classical machine vision approaches detect crop-rows with Our navigation strategy is influenced by the robot that we
juvenile crops [13] or detect crop-rows under challenging use. In this work we use a retrofitted Thorvald platform [25]
illumination conditions [18]. While these approaches are which we refer to as BonnBot-I. Below we provide, in brief,
generally real-time and can, to varying degrees, navigate a the specifications of the BonnBot-I platform and then present
lane they do require hand selected features which reduces the high level guidance strategy of our proposed navigation
the learning abilities of the techniques. scheme.
Kraemer et al. [19] used a deep learning approach to rec- 1 https://github.com/Agricultural-Robotics-
oncile the PSEP features by exploiting the likelihood maps of Bonn/visual-multi-crop-row-navigation

6586

Authorized licensed use limited to: LAHORE UNIV OF MANAGEMENT SCIENCES. Downloaded on April 06,2023 at 06:46:45 UTC from IEEE Xplore. Restrictions apply.
 Fig. 3: In-field navigation strategy, (1) following crop-rows
with front camera, (2) exiting crop-rows using back camera,
(3) switching to the next crop-rows, and (4) following new (a) Lemon-balm (b) Coriander
crop-rows with back camera. Fig. 4: The vegetation segmentation (in green), plant bound-
A. BonnBot-I Platform aries (in bold green) and the resultant plant centers (magenta
dots). In (a) individual plants are easy to see and (b) is a case
BonnBot-I is a prototype crop monitoring and weeding
where crop boundaries have to be estimated.
platform. The platform is a lightweight Thorvald system
which has been adapted for arable farming and phenotyping in Sec. IV-A, integrate information from multiple detected
fields in Europe. The European phenotypic regulation pattern crop-rows to perform visual-servoing based crop-row fol-
(35 cm or 55 cm between crop-rows) leads to either two or lowing in Sec. IV-B, and present a multi-crop-row switching
three crop-rows in each lane with a total width of 1.25m. approach in Sec. IV-C.
For this, the width of the platform was set to 1.4m wheel-
centre-to-wheel-centre with a vertical clearance of 0.86m to A. Multi-Crop-Row Detection
ensure the platform could operate in the field during the early
The first step in a successful vision system capable of
growth cycle of most plants. The length of the robot was set
traversing a field is the assumption that crop-rows are planted
to 1.4m ensuring there is room for all the weeding tools, and
in a parallel fashion. To have a completely crop agnostic
has a maximum speed of 1.5 m/s. BonnBot-I has multiple
navigation system the varying distances between the rows
sensors, of interest for navigation a GNSS (with an IMU)
for the different crops is an important element. Therefore, it
and two RGB-D cameras which are mounted equally distant
is imperative to have a system that can detect the number of
from the center of rotation of the robot symmetrically (in
crop-rows before instantiating the navigation algorithm.
front and back) illustrated in Fig. 1.
We perform crop-row detection by employing a novel slid-
B. In Field Guidance Strategy ing window based approach. This extracts the location of the
A benefit of crop-rows is that they are generally planted prominent crop-rows while being robust to the appearance
in consistent long parallel structures. A downside to this of weeds between them. Our detection approach consists of
parallel structure is that there is no connection between three steps. First, we perform vegetation segmentation fol-
them. Therefore, the platform needs to not only follow the lowed by connected components operations to find individual
crop-row without damaging the crop but also autonomously regions (plants) and their center points. This allows us to
switch between them. To achieve multi-crop-row following remain agnostic to the crop that has been planted. Second, we
we employ a similar high-level algorithm to our previous automatically detect the number of crop-rows by employing
work in [4] for a single crop-row. Fig. 3 outlines our multi- an estimate of the moving-variance which we use to describe
crop-row following strategy. Starting in a corner of the field, the field structure. Finally, we track the detected crop-rows
the platform autonomously follows the current set of crop- by centering a parallelogram on each row while the robot is
rows (a lane) using vision based navigation techniques 1 traversing the lane. We detail each of these steps below.
until the front facing camera detects the end of the current 1) Vegetation Mask and Crop Center Extraction: We
lane. The rear camera then guides the robot to the exit summarize a row by the position of the individual plants
point, end of the lane, actively guiding the robot at all along it. Each plant is represented by its center point. We
times 2 . Using the omni-directional capability, the robot obtain this by first computing the vegetation mask of the
then switches to the next set of crop-rows 3 to be traversed. input RGB image using the excess green index (ExG) [26].
The benefits of the omni-directional platform prevails here To separate foreground and background pixels in the image
as we can directly navigate to the new lane without turning based on ExG we employ Otsu’s method [27] which obviates
around 4 , this also outlines the benefit of symmetrically the need for manual tuning of a threshold. Then, each
mounted sensors at the front and rear. In the next section we connected component in the vegetation mask is converted
describe the vision-based crop-row following and crop-row to an object of interest (plant) with the unique center point
switching algorithms. obtained from the center of mass.
One issue associated with this technique occurs when
IV. V ISION -BASED G UIDANCE IN FARMING F IELDS multiple “plants” are absorbed into a single large region most
We propose a multi-crop-row following approach that can often occurring with bushy plants. A single region represent-
successfully guide a real-sized agricultural robot in row- ing an entire crop-row negatively impacts later stages such as
crop fields of five different crops. In achieving this we have line fitting. To reconcile this we divide contours into smaller
developed a multi-crop-row detection approach described subsections if they exceed a predefined maximum height,

6587

Authorized licensed use limited to: LAHORE UNIV OF MANAGEMENT SCIENCES. Downloaded on April 06,2023 at 06:46:45 UTC from IEEE Xplore. Restrictions apply.
 depicted in Fig. 4. Ultimately, this step allows us to cater for
a larger variety of canopy types.
2) Detecting Individual crop-rows: Fig. 5 illustrates the
crop-row detection algorithm, First, a sliding window Ψ
scans the image from left to right with a stride of S. The size
of the sliding window (w and h) and the stride S are set to
ensure a large overlap between adjacent steps in the scans.
For the n-th sliding window Ψn we compute a line (Ln )
based on the crop centers inside the sliding window using
least-squares method. We then find the intersection point In
of the line Ln with the bottom axis of the image; we only
retain lines that intersect within the image bounds. Each line
is then described by its point of intersection In = [x, y] and
the angle of intersection φn such that Ln = (In , φn ).
We use the estimated crop lines in conjunction with the
moving variance [28] to represent the local structure of the Fig. 5: The sliding window Ψ is applied progressively. The
field. The moving variance of the estimated crop line angles, moving-variance of the estimated line angles are used to
φn , is calculated over a window of size k such that represent the field structure. The peaks (N) and troughs (O)
Pn+k/2 Pn+k/2 from the field structure are used to find the crop-rows and the
2
2 i=n−k/2 (φn − φ̄n ) i=n−k/2 φi center of the between crop-rows respectively. The weighted
σ (φn ) = ; φ̄n = . average of multiple troughs leads to the final trough .
k k
(1)
The moving variance operator yields peaks when there is the multi-crop-row switching algorithm (Sec. IV-C).
discord between the local hypothesised crop lines, this occurs
B. Visual-Servoing Based Crop-Row Following
between the crop-row lines. Troughs occur when there is
consistent agreement regarding the hypothesised crop lines, To guide the robot within the crop-rows, we utilize our pre-
this occurs in the presence of crop-rows. A depiction of this viously developed in-row guidance approach [4] which relies
field structure signal is given in Fig. 5. on the image-based visual servoing (IBVS) controller [23].
The peaks (N) and troughs (O) of the field structure The guiding feature L is the average line of the set of
signal are detected using peak prominence with a constant detected crop-row lines L illustrated in Sec. IV-A computed
threshold. To detect the troughs, the signal is flipped (via from the current camera image. The desired feature L∗ =
negation) and peak prominence is applied with the same [0, H2 , 0] is located at the bottom center of the image I,
threshold. The detection of troughs is more complex as crop- illustrated in Fig. 6. And, the IBVS controls law leads the
H
rows can yield multiple peaks. We resolve this by computing current feature L = [a, , Θ] towards L∗ , where H denotes
the weighted average of the possible solutions in the local 2
the height of the image and a is the deviation from the image
neighbourhood, where the local neighbourhood is defined center of the intersection point I. By continuously regulating
to be adjacent sampling positions with similar standard the robot’s motion based on this controller and observation
deviation values. An example of this is given in Fig. 5 where features explained in Sec. IV-A.3, we ensure the robot stays
the final trough is denoted by . The output of this step is in the middle of the path and follows the desired crop-rows
the set of detected crop-row lines L. to their end. At the end of crop-rows, we switch to the next
3) Crop-Row Tracking: For each detected crop-row lane by using the switching technique described in the next
line Ln we form a parallelogram Pn with a width relative section.
to the canopy width of the crops forming line Ln . All
parallelograms have a minimum constant width of 5% of the C. Multi-Crop-Row Switching
image size to avoid shrinking to zero size width in regions To autonomously navigate over an entire row-crop field
without vegetation or having only tiny plants. For every new a robot must be able to both navigate down lanes and
camera image, we update the position of each line Ln based shift between them. Utilizing an only image-based motion
on the crop centers within the parallelogram Pn at its latest controller in conjunction with other localization techniques
position. If a crop-row line is not detected, the position of (like GPS and wheel odometry) could considerably improve
the previous line (and parallelogram) is maintained for a few the reliability of the system in cases of outage of motion
seconds, this allows our approach to continue guiding the information due to hardware problems and environmental
robot even when there is an uneven crop distribution in the situations. In our previous work [4] the task of changing lanes
crop-rows. The L is then tracked and used to update the was managed successfully, however, it was only designed to
robot’s position to ensure that the crop is always beneath the handle a single crop-row in a lane under a highly engineered
robot, using a visual-servoing controller. Finally, when all of condition. This method was strongly reliant on the seeding
the crop centers are in the top half of the image we consider pattern of the crops-rows and struggled with cases which
that we have reached the end of the crop-row and employ often occur in a real-fields like: uneven seeded crop-rows,

6588

Authorized licensed use limited to: LAHORE UNIV OF MANAGEMENT SCIENCES. Downloaded on April 06,2023 at 06:46:45 UTC from IEEE Xplore. Restrictions apply.
 (a) Sim-Curved (b) Sim-Large-Gaps (c) Sim-Dense-Weed
Fig. 7: Simulated Fields with different plant sizes as weeds
(small) and crops (big), for purpose of visualization soil
background in simulation is removed.
When D(G, G ∗ ) exceeds a threshold τ we assume a new
Fig. 6: The image frame I and L = [I, Θ] denotes the crop-row has been found, τ is a crop type specific constant.
dominant line estimated from the visible crop-rows.
V. E XPERIMENTAL E VALUATIONS
unexpected distances between the rows, and appearance
cluttered and weedy regions. Furthermore, it required sig- We performed three experiments to show the capability
nificant space to perform the switching maneuver and could and robustness of our proposed approaches. These experi-
not reconcile differences between rows and subsequently ments were carried out on both simulated and real phenot-
following the incorrect lane. pying fields. The simulated fields are built in the Gazebo
In this work we propose a multi-crop-row switching tech- environment with either two or three rows in a lane. It is also
nique that detects and counts the rows as it progressively designed with various challenging arrangements, such as:
shifts between them. This takes advantage of the side-ways curved crop-rows (Sim-Curved), crop-rows with large inter-
movement of BonnBot-I allowing easier transitions while crop gaps (Sim-Large-Gaps), and lanes of crops having dense
requiring less space. To detect a new crop-row we exploit weed appearance (Sim-Dense-Weed), which are depicted
SIFT features to ensure we only traverse the desired number in Fig. 7. The real fields represent up to five different crops
of crop-rows to confirm our new lane is in the correct with non-similar canopy shapes and varying numbers of
location without relining on any motion information neither crop-rows per lane. All results outlined in this section are
odometry nor GPS. based on the evaluation data and required limited human in-
We start by considering the robot to have found the end of tervention (apart from minor hyper-parameter tuning) during
the crop-row; stage 3 of the navigation scheme Sec. III-B. run-time and navigation.
The multi-crop-row detection algorithm Sec. IV-A provides A. Experimental Setup
us with the number of crop-rows C that have been traversed.
To find new crop-rows, lane switching, we need to move The experiments of all five crop types were completed on
across C rows and then restart the crop-row following BonnBot-I at Campus Klein-Altendorf of the University of
algorithm. To do this we describe each crop-row by a set Bonn under various illumination and weather conditions such
of SIFT features and follow the algorithm described below. as: relatively wet and very dry grounds, cloudy and sunny
Assuming we are moving left-to-right, we store the fea- day-times with long and short shadow cases (with minimal
tures of the right most parallelogram in the image forming hyper-parameter tuning). To fast track the field experiments
a feature set G. The robot then starts moving to the right- we used a 1:1 scale Gazebo simulation model with a realistic
side with a constant uy = 0.15 m/s velocity. Upon receiving field. From this simulation, we were able to derive our
a new image we detect the crop-rows in a similar manner algorithmic hyper-parameters. On BonnBot-I, the front and
outlined in Sec. IV-A.2 and then only consider the right most back navigation cameras are fixed at a height of 1.0 m and
side of the image. We extract a new set of SIFT features from tilt angle ρ = 65◦ . Both camera resolutions are 1280 × 720
the right most parallelogram in the image forming a feature with a capture rate of 15 f ps. For all experiments, the width
set G ∗ . The new feature set, G ∗ , is potentially a new crop- w of sliding window Ψ was kept constant w = W/10 = 128
row. To determine if G ∗ is a new crop-row we compare it with a height of h = 720 pixels. This window size and
to the stored SIFT features G. If a new crop-row has been S = 13 ensures ' 95% overlap between consecutive sliding
detected, we update the stored features (G=G ∗ ) and continue windows. Also, we empirically set k = 10 in Eq. (1) which
this process until we have moved across C new crop-rows. in simulation provided the best trade-off between sample
To compare the SIFT features, we use a FLANN-matcher consistency and neighbourhood relationship. As the primary
to obtain the best matches Ω between the two sets G and goal of this platform is to perform weeding we set its
G ∗ . This results in the Euclidean distances between G and velocity to be a constant vx∗ = 0.5 m/s. We use differential
G ∗ being stored in Ω. We then take the average of the m velocity control within the crop-rows and omni-directional
matches in Ω which are above a threshold λ. This is used to control for switching between the lanes. Our approach is
provide a distance measure between the two sets of features: implemented using Python and PyCuda ensuring real-time
m
operation (while with CPU only machines performance is
∗ 1 X still acceptable) and runs on a user grade computer (Cincose
D(G, G ) = Ωm (2)
m i=1 DS-1202).

6589

Authorized licensed use limited to: LAHORE UNIV OF MANAGEMENT SCIENCES. Downloaded on April 06,2023 at 06:46:45 UTC from IEEE Xplore. Restrictions apply.
 (a) Sugar-beet (b) Coriander (c) Potato (d) Beans
Fig. 8: Illustrations of four crops (top row) RGB images with marked vegetation index of detected rows and their
corresponding field structure signal (bottom row). The detected peaks (blue triangle) and troughs (orange square) obtained
via their prominence in the signal are also provided. Denote that, the field structure signal only include the values of lines
intersecting with the bottom axes of the image.
B. Multi-Crop-Row Detection the simulated field, 100 images were annotated using data
The first experiment is a qualitative analysis on the ability from BonnBot-I where the camera tilt angle ρ was varied
to detect crop-rows in the field using the technique described from 55◦ to 75◦ . The annotations contain ground-truth of all
in Sec. IV-A. We use ExG with Otsu’s technique [27] to the crop-row lines located underneath the robot and crop
differentiate foreground from background as this obviates the masks belonging to the main crop-rows. To measure the
need for individual hand-tuned plant segmentation thresh- accuracy we compare the predicted lines of each image to the
olds. The goal of this technique is to exploit the dominant ground-truth using two parameters: position and orientation.
crop locations and accurately detect the best location for The position of a line is defined based on its intersection
traversing a lane (i.e. keeping the crop-rows under the plat- with bottom edge of the image, where the distance between
form). Due to weeds growing between the crop-rows this can the prediction and the ground truth is normalized based
be a challenging proposition in real fields. Qualitative results on the width of the image. Fig. 9 outlines the quantitative
for four crops (sugar-beet, coriander, potato, and beans) are performance result of real fields and averaged performance
presented in Fig. 8; we refrained from adding Lemon-balm of simulated fields.
and simulated fields results due to space limitations. The We observe our method is able to estimate crop row
illustrated crops have diverse canopy types (see Fig. 8) and positions with a mean accuracy of 88.1% and standard
are arranged in two standard patterns with two and three deviation of 8% over all types of real crops. Similarly,
crop-rows per lane. In the bottom row of Fig. 8 it can be seen we see that the algorithm is able to correctly estimate the
that our approach to detection crop-row lines through peaks
and troughs works for these chosen crops even with varying
number of crop-rows. This is true even for the challenging
crops such as coriander and beans.
For all crops we were able to consistently detect both the
peak (no crop-row) and trough (crop-row) locations regard-
less of the presence of weeds. This is especially evident in
coriander where even with the small distance within the crop-
row (between the coriander plants) we are still able to detect
the crop-rows. This is an example where dividing single large
regions into sub-regions is essential. Sugar-beet, Fig. 8-a, is
another interesting use case. Visually it is considerably more
difficult to discern the crop locations, however, this technique
was still able to extract crop-rows. Overall, this technique for
crop-row detection successfully located the required troughs
in order to navigate a lane, providing accurate information
required for the other stages of our system. Fig. 9: Performance of multi crop-row detection technique:
To further analyze the robustness of crop-row detection accuracy of detections (a) position (w.r.t the image width)
technique we perform a quantitative evaluate. For each of and (b) orientation w.r.t the acceptable thresholds
the five crops (those listed in Fig. 8 and lemon-balm) and .

6590

Authorized licensed use limited to: LAHORE UNIV OF MANAGEMENT SCIENCES. Downloaded on April 06,2023 at 06:46:45 UTC from IEEE Xplore. Restrictions apply.
 orientation of crop-rows in more than 88.3% of cases when crop-row distance. This minor fluctuation is sufficient to
the acceptance threshold is set to 11 degrees. The crop-row ensure safe navigation without damaging crops. Finally, this
lines of beans and sugar-beet were the hardest to estimate. navigational accuracy was sufficient for the technique to
For sugar-beet, we attribute this to the fact that the crop traverse all the crops in the field without manual intervention.
was at an early growth stage, as seen in Fig. 8, and this
made it more complicated to detect the crop lines. For beans, D. Multi-Crop-Row Switching
we attribute this to its branchy canopy shape, disarranged Our final evaluation is based on the lane switching tech-
seeding pattern, and plant vibration due to the wind in the nique outlined in Sec. IV-C. To evaluate the performance
field. One potential use case where the approach may fail is of this technique we manually annotated randomly selected
when the number of weeds is close to or greater than the positive and negative samples from our three main crop
number of crops in the image (very high weed pressure). types: beans, coriander, and sugar-beet and simulated fields
Furthermore, difficulties may be faced when navigating the as we did not have the switching information for potato and
field once full canopy closure has been achieved and there are lemon-balm due to a technical problem. We store one of the
no visible crop lanes to follow (full vegetation). Nevertheless, positive annotations as our main row and compare it to each
overall we observe our novel crop-row detection method of the other positive and negative rows. Fig. 10 outlines the
could estimate lines of crops in a variety of challenging real- precision-recall curves achieved on each of the main crops.
world conditions for different crop types reliably. We outline the F1 score here which provides a trade-off
between precision and recall. For this simple matching tech-
C. Navigating Along The Crop-Rows nique we are able to achieve promising results across all crop
To analyse the performance of our crop-row navigation types, even sugar-beet which, like outlined Sec. V-C, had a
technique we require accurate ground truth information. number of added complexities. The early germination stage
We collected the ground truth information by manually of the crop added extra complications to crop-row switching
driving the robot down each of the row-crop fields for all as, even visually, the rows appeared similar. However, we
crop types and stored the associated information (e.g. GPS were able to achieve an F1 score of 62.1, which was the
measurements) for later evaluation. Also, all simulated crop lowest performing crop.
rows came with reference lines coordinates from simulation Overall, from these evaluations we were able to empiri-
environment. The associated GPS measurements are then cally set thresholds that favored high precision in order to
used as the “correct” position (accurate to 1cm). Even though remove false positives. From this we were able to provide
manual operation can cause some errors we consider this to a lane switching technique that was robust to the challenges
be an appropriate ground truth to compare to as the crop- of each crop type. In experiments in the field and deployed
rows are not guaranteed to be planted in a straight line. on a robot, it successfully switched 6 lanes of crop, across
The five crops (sugar-beet, coriander, potato, beans, and the three main crop types, without any manual intervention.
lemon-balm) provide a range of challenges such as different Furthermore, we used this technique in simulated crop-row
canopy types, weed densities, and varying growth stages. fields for 20 lane switching cases which outlined an average
Tab. I outlines the performance of our full pipeline, including success rate of 90% percent.
the navigation system, on these crops as well as three A final key analysis of our crop-row switching technique
challenging simulated fields. The most challenging crop for is the distance needed to perform the maneuver. In our
navigation was sugar-beet and we attribute this to two rea- experiments an average of 0.7m was required from the end
sons. First, the crop was at an early growth stage, as seen in of the crop to the location of the camera. This is a marked
Fig. 8, and this made it more complicated to detect the crop improvement over [4] which required more empty space to
lines. Second, not all of the sugar-beet had germinated and perform the switching than the length of the robot itself.
this led to gaps or long “dead space” along the rows, which
the same effect can be seen in Sim-Large-Gaps results too. VI. C ONCLUSION
However, tracking multiple crop-rows allowed our technique In this paper, we presented a novel approach to enable
to still navigate over the entire evaluation area without any autonomous navigation in row-crop fields empowering pre-
manual intervention. This evaluation shows that multiple TABLE I: Lane following performance of BonnBot-I using
crop-row following has considerable benefits over techniques proposed method in real and simulated fields.
that only track a single crop-row.
From a navigational perspective the bean crop had a µ ± σ of µ ± σ of
Crop Length
dist. to crop-rows angular error
large standard deviation between real fields and Sim-Curved
among simulated fields when considering angular error. The Sim-Curved 200 m 9.01 ± 2.63 cm 4.52 ± 3.52 deg
Sim-Large-Gaps 200 m 6.75 ± 3.15 cm 4.76 ± 2.69 deg
weather conditions played a crucial part in this as heavy Sim-Dense-Weed 200 m 7.41 ± 2.86 cm 3.91 ± 1.73 deg
winds consistently changed the location of the leaves of the
Beans 52 m 3.49 ± 2.89 cm 3.73 ± 3.21 deg
crops. This limitation in the navigation technique leads to Potato 37 m 2.18 ± 3.01 cm 4.91 ± 1.63 deg
large angular variations while traversing the lane. Coriander 54 m 2.91 ± 2.38 cm 2.57 ± 1.05 deg
Across the five real crop types the average deviation from Sugar-beet 69 m 8.41 ± 3.79 cm 3.25 ± 1.27 deg
Lemon-balm 40 m 2.12 ± 1.58 cm 3.21 ± 2.83 deg
the ground truth was 3.82cm or approximately 10% of the

6591

Authorized licensed use limited to: LAHORE UNIV OF MANAGEMENT SCIENCES. Downloaded on April 06,2023 at 06:46:45 UTC from IEEE Xplore. Restrictions apply.
 [7] M. Pérez-Ruı́z, D. C. Slaughter, F. A. Fathallah, C. J. Gliever, and
B. J. Miller, “Co-robotic intra-row weed control system,” Biosystems
engineering, vol. 126, pp. 45–55, 2014.
[8] M. Bayati and R. Fotouhi, “A mobile robotic platform for crop
monitoring,” Adv. Robot. Autom, vol. 7, no. 2, 2018.
[9] T. Utstumo, F. Urdal, A. Brevik, J. Dørum, J. Netland, Ø. Overskeid,
T. W. Berge, and J. T. Gravdahl, “Robotic in-row weed control in
vegetables,” Computers and electronics in agriculture, vol. 154, pp.
36–45, 2018.
[10] B. Åstrand and A. J. Baerveldt, “A vision based row-following system
for agricultural field machinery,” Mechatronics, vol. 15, no. 2, pp. 251–
269, 2005.
[11] B. Thuilot, C. Cariou, P. Martinet, and M. Berducat, “Automatic
guidance of a farm tractor relying on a single cp-dgps,” Autonomous
robots, vol. 13, no. 1, pp. 53–71, 2002.
[12] J. Billingsley and M. Schoenfisch, “The successful development of a
vision guidance system for agriculture,” Computers and electronics in
agriculture, vol. 16, no. 2, pp. 147–163, 1997.
[13] B. Åstrand and A.-J. Baerveldt, “A vision based row-following system
for agricultural field machinery,” Mechatronics, vol. 15, no. 2, pp. 251–
Fig. 10: The precision recall plot for switching the platform 269, 2005.
across lanes, includes, beans, coriander, and sugar-beet. [14] O. C. Barawid Jr, A. Mizushima, K. Ishii, and N. Noguchi, “Develop-
ment of an autonomous navigation system using a two-dimensional
laser scanner in an orchard application,” Biosystems Engineering,
cision farming and crop monitoring tasks. This approach ex- vol. 96, no. 2, pp. 139–149, 2007.
ploits the crop-row structure using only the local observation [15] F. B. Malavazi, R. Guyonneau, J.-B. Fasquel, S. Lagrange, and
from the on-board cameras without requiring any global or F. Mercier, “Lidar-only based navigation algorithm for an autonomous
agricultural robot,” Computers and electronics in agriculture, vol. 154,
local position awareness. To achieve this, we have proposed pp. 71–79, 2018.
a novel multi-crop-row detection strategy that can deal with [16] W. Winterhalter, F. V. Fleckenstein, C. Dornhege, and W. Burgard,
cluttered and weedy scenes. We also proposed a novel lane “Crop row detection on tiny plants with the pattern hough transform,”
IEEE Robotics and Automation Letters, vol. 3, no. 4, pp. 3394–3401,
switching strategy which enables BonnBot-I to switch to a 2018.
new lane independent of any global positioning system or [17] M. Bakken, R. J. Moore, and P. From, “End-to-end learning for
human intervention. We evaluated our approach on BonnBot- autonomous crop row-following,” IFAC-PapersOnLine, vol. 52, no. 30,
pp. 102–107, 2019.
I on up to five crop types (with varying canopy shapes) in [18] H. T. Søgaard and H. J. Olsen, “Determination of crop rows by
real field conditions and three challenging simulated fields image analysis without segmentation,” Computers and electronics in
achieving an average navigation accuracy of 3.82cm in real agriculture, vol. 38, no. 2, pp. 141–158, 2003.
[19] F. Kraemer, A. Schaefer, A. Eitel, J. Vertens, and W. Burgard,
fields. Future work could explore alternative approaches to “From plants to landmarks: Time-invariant plant localization that
detecting individual plants (crop/weed semantic segmenta- uses deep pose regression in agricultural fields,” arXiv preprint
tion) and consider how global positioning could augment the arXiv:1709.04751, 2017.
[20] V. R. Ponnambalam, M. Bakken, R. J. Moore, J. Glenn Omholt Gjeves-
robustness of the current system. tad, and P. Johan From, “Autonomous crop row guidance using
adaptive multi-roi in strawberry fields,” Sensors, vol. 20, no. 18, p.
ACKNOWLEDGEMENTS 5249, 2020.
This work was funded by the Deutsche Forschungs- [21] Y.-K. Lin and S.-F. Chen, “Development of navigation system for
tea field machine using semantic segmentation,” IFAC-PapersOnLine,
gemeinschaft (DFG, German Research Foundation) under vol. 52, no. 30, pp. 108–113, 2019.
Germany’s Excellence Strategy - EXC 2070 – 390732324. [22] R. de Silva, G. Cielniak, and J. Gao, “Towards agricultural autonomy:
Furthermore, we appreciate the help of Julius Knechtel and crop row detection under varying field conditions using deep learning,”
arXiv preprint arXiv:2109.08247, 2021.
Gereon Tombrink. [23] B. Espiau, F. Chaumette, and P. Rives, “A new approach to visual
servoing in robotics,” ieee Transactions on Robotics and Automation,
R EFERENCES vol. 8, no. 3, pp. 313–326, 1992.
[24] D. A. de Lima and A. C. Victorino, “A visual servoing approach for
[1] “Garford(2014).robocrop inrow weeder,” https://garford.com/, ac- road lane following with obstacle avoidance,” in 17th International
cessed: 2020-10-22. IEEE Conference on Intelligent Transportation Systems (ITSC). IEEE,
[2] O. Bawden, J. Kulk, R. Russell, C. McCool, A. English, F. Dayoub, 2014, pp. 412–417.
C. Lehnert, and T. Perez, “Robot for weed species plant-specific [25] L. Grimstad and P. J. From, “Thorvald ii-a modular and re-configurable
management,” Journal of Field Robotics, vol. 34, pp. 1179–1199, agricultural robot,” IFAC-PapersOnLine, vol. 50, no. 1, pp. 4588–4593,
2017. 2017.
[3] M. Bakken, V. R. Ponnambalam, R. J. Moore, J. G. O. Gjevestad, and [26] D. Woebbecke, G. Meyer, K. V. Bargen, and D. A. Mortensen, “Color
P. J. From, “Robot-supervised learning of crop row segmentation.” indices for weed identification under various soil, residue, and lighting
[4] A. Ahmadi, L. Nardi, N. Chebrolu, and C. Stachniss, “Visual servoing- conditions,” Trans. of the American Society of Agricultural Engineers
based navigation for monitoring row-crop fields,” in 2020 IEEE (ASAE), vol. 38, no. 1, pp. 259–269, 1995.
International Conference on Robotics and Automation (ICRA). IEEE, [27] C. Reimann, P. Filzmoser, and R. G. Garrett, “Background and
2020, pp. 4920–4926. threshold: critical comparison of methods of determination,” Science
[5] E. J. Van Henten, J. Hemming, B. Van Tuijl, J. Kornet, J. Meuleman, of the total environment, vol. 346, no. 1-3, pp. 1–16, 2005.
J. Bontsema, and E. Van Os, “An autonomous robot for harvesting [28] J. MacGregor and T. Harris, “The exponentially weighted moving
cucumbers in greenhouses,” Autonomous robots, vol. 13, no. 3, pp. variance,” Journal of Quality Technology, vol. 25, no. 2, pp. 106–118,
241–258, 2002. 1993.
[6] S. Bonadies and S. A. Gadsden, “An overview of autonomous crop
row navigation strategies for unmanned ground vehicles,” Engineering
in Agriculture, Environment and Food, vol. 12, no. 1, pp. 24–31, 2019.

6592

Authorized licensed use limited to: LAHORE UNIV OF MANAGEMENT SCIENCES. Downloaded on April 06,2023 at 06:46:45 UTC from IEEE Xplore. Restrictions apply.