A Hybrid Localization Approach For Uav WITH INSTALLED MAP
A Hybrid Localization Approach For Uav WITH INSTALLED MAP
A Hybrid Localization Approach For Uav WITH INSTALLED MAP
Abstract—In this paper, the localization problem of an provide reliable UAV position information, as the state
autonomous unmanned aerial vehicle in case of losing the GPS estimation solution (position, velocity and attitude) drifts in
signal is handled. A vision-based solution approach is proposed time [2]. This would result in errors in determining the
consisting of two phases. In the first phase a hybrid map is accurate position of the UAV during flight mission, which
constructed. Such map consists of a set of reduced features
could have disastrous consequences [3], especially when the
obtained by information-theoretic analysis. This enables faster
UAV localization processing without degenerating the UAV becomes out of sight of the pilot and can not be
accuracy. The features are represented by local descriptors brought back to the homing point manually. Thus, such a
which are additionally tagged with their metric position. The safety-critical UAV system should be able to navigate
second phase localizes the UAV using the map, which is safely, even if the GPS signal is disrupted or totally lost.
performed on two scales. A fast and coarse topological location This problem has been investigated by researchers in the
is identified based on matching features of images taken by the
last few years. One potential solution is based on the
camera with the local descriptors information in the map. This
guides the UAV in fast and safe emergency homing. A second integration of a vision system to support the already existing
precise metric position can be estimated in extension with navigation system (fig. 1). A localization method can be
respect to a previously identified topological location and with utilized to detect some pre-extracted features in the images
the aid of the features’ metric position information. This can and hence to calculate the position of the UAV when the
assist the UAV navigation in case the mission should be signal of the GPS is lost. A Kalman filter can be used to fuse
completed without interruption despite the GPS signal loss.
signals from the inertial sensors with the position sensor
(GPS or the vision system in case the GPS is not available).
I. INTRODUCTION
Vision-based solutions are attractive since video cameras
A.
hybrid feature extraction
Sensor images
Feature Feature
MAP CONSTRUCTION
FINAL
Extraction Localization HYBRID
Data data set including MAP
codewords
features
PHASE
position info.
features +
position info.
Feature
features Information-
Extraction + Theoretic Codebook
INITIAL position
HYBRID Feature Feature
info. feature Compression
MAP Evaluation
clusters
Feature B.
Evaluation map generation
C.
Compression Position
?
PHASE
node
topological metric
features features
Feature entropy-based features
Hybrid UAV
Position Triangulation position info.
Map
hybrid
localization Map Building
D.
Localization
(a) (b)
Figure 3. (a) Map generation concept. (b) Solution structure for a hybrid map building and localization.
applies triangulation for such purpose. Different positions of C. Map Generation Module
the moving vehicle are required for the triangulation In order to generate a compact and more efficient map
execution (fig. 4). For this, the GPS system must be than the previous one, the initial map is processed by an
functioning in the map construction phase to supply such information-theoretic evaluation approach which is
information on the vehicle position. The vehicle different described in [11,17]. Such approach makes use of two
positions are used with the corresponding measured bearings components, information-theoretic feature evaluation and
(i.e. angles between the unknown camera heading direction codebook feature compression, to generate the final map.
and the features which are identified in camera in pixels) to The approach utilizes an Entropy-based evaluating criterion
calculate the position of the features. combined with a clustering technique to provide additional
compression for the evaluated data. It has proven to provide
accuracy combined with computational efficiency for robot
indoor navigation [11] and under severe illumination
conditions [17]. The approach aims at maximizing the
topological node recognition by filtering the most
discriminative information at each node. The filtered set
after evaluation is called entropy-based features as identified
in fig. 3. The feature evaluation component is implemented
in a way that supplies cluster information to a second
Codebook component. This Codebook allows further
compression to be applied for the evaluated features
resulting in a feature format called the codewords (see
Figure 4. Feature localization in map building phase using the figure). This additional compressed version of the features,
vehicle position measured by the GPS and the angles to features
measured by the camera. which contains the non-geometric information part only, is
targeted for use in the topological matching, in order to
B. Initial Map accelerate recognition while still maintaining the accuracy of
Using the hybrid feature extraction, the helicopter can fly recognition and matching.
over the area of interest, collect image data and construct a The idea of the information-theoretic technique is to have
feature map for localization. The area is divided into a set of every topological node initially described with a set of local
discrete regions (nodes) which are related to each other by features, and moreover compress those features to define a
neighborhood interconnectivities. The nodes can be assigned set of local codewords. The compression is done by
automatically by generating them at equidistant distances clustering the features extracted in every node using k-
traveled by the flying UAV. An alternative option is to means clustering [18]. Next, the whole feature pool of the
assign the new node at the region exhibiting a variation in nodes is sampled to determine the true variation
the scene detail which can be monitored by the dynamic (distribution) of the features in the high-dimensional space.
change of the tracked features in the camera view. Afterwards, a conditional-entropy-based criterion is applied
After the UAV executes the mission of collecting data at to evaluate the contribution of the features to uniquely
each node, a hybrid map can be constructed. The hybrid map identify a local codeword. This correspondingly measures
in this case will employ nodes with local features, which the features’ contribution to the identification of a specific
possess both non-geometric and geometric information. The node because the codewords are defined locally. The
information part is represented by the tagged position where ok represents a local codeword and fi represents a true
information of the features in the nodes and will be used variation sample; k=1,…,Ω; Ω is the total number of
together with the non-geometric part to resolve the UAV codewords, and i=1,…,Ψ; Ψ equal to the total number of
metric location. The hybrid structure of the map information feature samples. Finally, the codewords and their
will be used for two different localization modes for the corresponding uncompressed features which exhibit
UAV as will be explained in subsection D. This constructed relatively large entropy values are discarded, while those
map, however, does not specifically contain the most exhibiting low entropy values are preserved in the final map.
informative data. Excessive or redundant data may exist, Previous investigations have indicated that a large number
which increases localization uncertainty and computational of the features can be eliminated (i.e. feature with high-
overhead. Therefore, further processing is suggested to be entropy values), while still maintaining the same localization
applied, in order to extract the possible minimum data that accuracy [11,17].
maximizes the localization efficiency in terms of accuracy Therefore, the final form of the hybrid map will contain
and computational performance. the reduced feature set, where every node preserves features
with their measured gradient and position information, and Rotation transformation
bet. the two frame
Camera coordinate
with an additional compressed gradient information version. coordinate systems Z' Y'
Sytem
R(ω,φ,κ)
X'
D. Hybrid Localization Module Camera
perspective C
The hybrid localization module consists of two center
components: a topological matching component and a ( X C , YC , Z C )T f
feature triangulation component. The former matches the Z Positive focal
plane
current camera view to the map to identify the current
topological node and uses the compressed map data for that, ya xa
a lens axis
while the latter triangulates the identified features to
estimate the position of the UAV in 6-DOF by using the Y
non- compressed map data format. ZC
The UAV localization is processed hierarchically making
use of the divided topological space. At the first level of
hierarchy, the image captured by the camera mounted on the A (Landmark)
ZA ( X A , YA , Z A ) T
UAV is compared to the hybrid map. Features are extracted XA
from the current image and are used in fast matching with XC
the compressed codewords to identify the corresponding YC YA
topological node. After the topological node has been W
identified, the UAV location estimation is executed locally World coordinate
in this region in a second level of hierarchy. The identity of System
the features in the node is resolved by matching the non- X
geometric information part and the position of the UAV is Figure 5. Photogrammetric projective model
estimated by applying a triangulation method for the
features using their metric position data. based on a photogrammetric projective model shown in
The two localization components use different versions of figure 5. The model relates the 3-D Cartesian coordinates of
map data for supplying either a coarse topological location features or landmarks in the real world to their
or a precise metric position estimate for the UAV. In case of corresponding 2-D coordinates projected on the image
flight emergencies signaled by the loss of the GPS signal, plane. The fundamental model used in the majority of
the topological localization component can assist the UAV applications is the Collinearity equations [20]. They relate
in fast and safe emergency homing, since less data is the object point and its corresponding image point to the
involved (codewords). In case that the mission should be perspective camera center. The projection model in the
completed without interruption and exact positioning is figure shows the alignment of the three points, with the
required, the non-compressed data with the metric position image plane set in the positive focal position (i.e. z=-f,
data are used to complete the mission, which still provides where f is the camera focal length). Projections in the x and
good computa-tional performance because of the y directions are proportional to the horizontal and vertical
hierarchical processing. bearings respectively, and are described by the Collinearity
The proposed localization method is robust against equations:
r11 ( X A X C ) r21 (YA YC ) r31 ( Z A Z C )
uncertainties in node recognition and data correspondences.
xa f
r13 ( X A X C ) r23 (YA YC ) r33 ( Z A Z C )
This is because the features used for the map building are (2)
selected based on properties of features discrimination and
r12 ( X A X C ) r22 (YA YC ) r32 ( Z A Z C )
ya f
node distinguishness measured by the introduced evaluation
r13 ( X A X C ) r23 (YA YC ) r33 ( Z A Z C )
criterion (i.e. ambiguities minimized). Moreover, the local (3)
matching confined to a single topological node minimizes
the possibility of mismatches and data correspondence for a feature position vector TW
q A =(XA YA ZA) and a camera
problem when compared to matching features of the whole
position vector located by its perspective center W q c =(XC YC
map. Additional confidence for the data correspondence can
be provided by utilizing features’ metric relationships as ZC)T. rij are elements of the rotation matrix describing the
proposed in [20]. transformation between the camera and the global world
coordinate systems, and which are function of the roll, pitch
V. METRIC LOCALIZATION USING TRIANGULATION and yaw angles of the camera.
r11 r13
R Rx ( ).R y ( ).Rz ( )
r23
Both the hybrid feature extraction and hybrid localization r12
r21
W
modules apply triangulation to estimate the position of C r22 (4)
features and UAV respectively (i.e. feature localization and r31 r32 r33
feature triangulation components). The triangulation is
The problem of either the camera or landmark position accuracy and less computational overhead. The features
estimation will be defined as a nonlinear least squares provide double information that can resolve the topological
(NLLS) problem. Therefore, the solution equations to the location and the metric position separately. The proposed
UAV localization are similar to that of the map-building hybrid structure and hierarchical localization processing
feature localization. offer combined accuracy with computational efficiency of
r xˆ a f i
i i
yˆ a f i
2 2 Mapping: Part II,” IEEE Robotics & Automation Magazine, 13, 2006.
m i i
U V [6] J. Wang, M. Garatt, A. Lambert, J.J. Wang, S. Han and D. Sinclair,
i 1
(5) “Integration of GPS/INS/Vision Sensors to Navigate Unmanned aerial
W W
Vehicles,” The International Archives of the Photogrammetry,
with ( xˆ a , yˆ a ) defining the measured feature position in the Remote Sensing and Spatial Information Sciences, Vol.XXXVII, Part
B1, Beijing, pp. 963-969, 2008.
camera in pixels, and [7] J. Kim, and S. Sukkarieh, “6 DoF Slam aided GNSS/INSS Navigation