July 2021

Evaluating
indoor & outdoor
mobile mapping
accuracy
An assessment of NavVis VLX data quality
compared to a terrestrial laser scanner,
including indoor and outdoor scenarios.

1
Introduction

As the number of mobile scanning systems on the market grows, the need
for an industry standard to evaluate the accuracy of these devices in
indoor and outdoor environments is increasingly apparent. This document
is a practical and reliable method to assess the accuracy of mobile laser
scanning systems in indoor and outdoor environments. It’s part of an
ongoing initiative at NavVis to establish a reliable method for evaluating the
accuracy of mobile laser scanning systems.

Do you need a mobile scanning system? The accuracy assessment criteria described in this
Laser scanning has made it possible to capture many document is therefore aimed at providing laser
environments in full 3D – but using conventional static scanning professionals who have a need for mobile
scanners can be slow and cumbersome. Numerous systems with a reliable means of choosing the right
scanning positions are required to ensure no details tool to deploy – both indoors and outdoors.
are missed, reducing the total area which can be
captured per day. As a result, using a static scanner to Use cases for mobile scanning systems
document buildings and industrial facilities is a highly The following use cases will help you decide if you need
labor-intensive process. a mobile scanning system in your projects:
ç L arge commercial and industrial properties which
In contrast, mobile laser scanning devices are next- would take too much time to capture with static
generation surveying tools with the ability to rapidly scanners.
cover large, complex areas. ç Manufacturing environments, where disruption must
be kept to a minimum.
But while mobile laser scanning solutions promise ç Construction verification purposes, where BIM
faster, easier data capture, in many cases the data (Building Information Modeling) models and point
quality is not comparable. Moreover, there are no clouds are compared in third-party software to have
industry standards to measure the accuracy of mobile complete transparency before construction has
scanning devices. started or during construction.
ç A s-built documentation or models and scan-to-BIM/
This leads to a lack of confidence in mobile solutions CAD.
among laser scanning and AEC professionals who ç Public buildings which are heavily used by people,
might need to deploy these devices. where disruption of ongoing operations must be kept
to a minimum.
In the past, NavVis products were developed especially ç Projects with a tight deadline where scanning must
for indoor operation. The lack of an official industry be completed quickly or only when the area is not in
standard for mobile mapping in this scenario is what use.
we sought to address in our accuracy white paper for
indoor mobile mapping*. With the launch of NavVis
VLX 2nd generation, which includes hardware and
software optimizations for outdoor scanning, we will
now expand the scope of our accuracy evaluation to
include the exterior of buildings.

*Evaluating the accuracy of a wearable mapping system – https://navv.is/accuracy

2
Key difference between mobile and
static scanning systems

When evaluating the accuracy of mobile scanning standard deviation, often 1-sigma, which equals
systems, it is important to consider a key difference 68% confidence, and sometimes 2-sigma, which
in how data is captured. Terrestrial laser scanners equals 95%. A standard deviation of 5 mm with a
(TLS) capture data by scanning at a single position, confidence level of 1 sigma, or 68% means that 68%
while mobile devices continuously capture data at of all measurements must be within a range of 5
multiple positions while being moved through an mm accuracy.
environment.
Meanwhile, the accuracy of a scan using a mobile
Therefore, when we refer to the accuracy of a mapping system is based on a vast number of
TLS, we are talking about the accuracy of discrete discrete measurements taken continuously. This
measurements at a single position. For one scan path of very dense scanning positions is otherwise
position, the specification sheet of a TLS usually known as the mapping trajectory.
refers to certain confidence levels associated with

Just how accurate is the data

captured with NavVis VLX for
indoor and outdoor locations?

When it comes to mobile mapping devices, absolute mobile mapping since

accuracy is dependent on the project geometry, many artificial objects
which means we cannot make a general statement can be recognized by
at this point. Instead, we analyzed the accuracy in the NavVis VLX and
two different scenarios, followed by a mapping and loop closures can be
modeling example: conducted by crossing
the street.
ç T
he first scenario represents a typical indoor
example: an office consisting of small rooms, ç The mapping and
connected through a hallway with some additional modeling example features a complete three-
doors between the rooms, which require only a story residential building, including the interior,
few loop closures to minimize drift. the outer shell of the building and the garden. The
whole property is captured in one data set and
ç T
he second scenario is the outdoor mapping of a demonstrates how fast and simple a potential
facade which is 20.3 m tall and 30.3 m wide. The modeling workflow from scan-to-BIM can be.
selected facade is part of a L-shaped street of
houses with a length of 400 m in total. This type of
environment is an ideal test for outdoor

3
Accuracy benchmarks

NavVis has established a rigorous approach for Global cloud-to-cloud metric based on a TLS
evaluating the relative (local) and absolute (global) The most significant metric for comparing mobile
accuracy of a mobile laser scanner in earlier and static laser scanning accuracy is one that detects
publications*. deviations in a mobile point cloud by comparing it to a
point cloud from a TLS. It is also a benchmark for the
The assessment criteria that were set relied on two absolute accuracy of the system. This assessment is
industry-standard devices as benchmarks for accuracy: conducted by running a full comparison of TLS point
a total station and terrestrial laser scanner. These clouds and mobile point clouds computationally. To
devices were chosen based on the premise that to compute the deviation between the TLS point cloud
assess the accuracy of a mobile scanning system, the and NavVis VLX point cloud, we use points and their
reference data must be of greater accuracy than the normals to obtain an unbiased estimate of the distance
test data. For the purpose of this paper, we will briefly between the two scanned surfaces. To account for
summarize the TLS as a benchmark for accuracy. changed conditions in the environment, such as cars
or pieces of furniture which are present in only one
of the scans, we include deviations up to 50 mm in
the analysis – everything above that we consider as
outliers. In the tests described in this document, we
compared both clouds as a whole.

Because this type of comparison is fully automated, it

can be based on a few million measured points. This
approach therefore provides the most comprehensive
insights into absolute accuracy.

*Evaluating the accuracy of a wearable mapping system – https://navv.is/accuracy

4
Two ways to increase accuracy with a mobile
mapping system
Mobile mapping is inherently prone to accumulate
measurement error along the mapping path (or
mapping trajectory).

There are two ways to reduce this error when using

NavVis devices:
1. Conducting “loop closures”: by returning to a
point where the mapper has been before during
the same scan, the error can be minimized. In In principle, there are three available options:
such cases, NavVis mapping software recognizes 1. No control points: Here, a point cloud in an
overlapping points in the trajectory and uses these arbitrary coordinate system is the result.
to minimize drift error. 2. Control points for (geo-)registration: Here, at least
2. Using “control points”: control points (CPs) three CPs should be used per scan, to allow for
are surveying targets that have already been registration of the point cloud in a local or global
measured with high precision (e.g., using a total cartesian coordinate system.
station). These can be used as fixed points to 3. Control point-based optimization: Here, CPs
minimize drift error with NavVis mapping software. are used to both register the point cloud and
to increase the absolute accuracy. The latter
Both NavVis M6 and NavVis VLX allow for capturing is achieved by doing an additional global
the coordinates of control points during a mapping optimization on the mapping trajectory that takes
session. NavVis M6 can be used with ground control the CPs as constraints into account.
points, while NavVis VLX is compatible with both
ground and wall control points. In the following scenarios, we will touch upon all three
approaches.

5
Scenario 1: Mid-size Office

Point cloud of the office floor test scan (ceiling removed for illustration only)

This scenario represents a typical layout for mid-size optimization of the point cloud geometry and for
offices or residential projects – small rooms aligned registration.
along a corridor and connected with a few additional
doors. The size of this project is around 500 sqm and As ground truth, an external laser scanning provider
was scanned in 20 minutes as one dataset, including was commissioned to scan the office with a highly
capturing six control points and several loop closures. accurate terrestrial laser scanner. For both test
The scan includes panorama images every 1-2 m, this models a) and b) we conducted a full cloud-to-cloud
contributes to the highly realistic coloring of the point comparison using an in-house tool. Below, the
cloud (see the image above). cumulative distribution of the deviations between
the TLS point cloud and the NavVis VLX point cloud is
The data from NavVis VLX was processed twice: shown in two graphs.
a) using the CPs for registration only, without CP-
based optimization, and b) using the CPs for global

Horizontal section through the office environment

6
Cumulative distribution of deviations resulting from the global cloud-to-cloud comparison: a) point cloud without CP-based optimization (left), b) point cloud
with CP-based optimization (right)

Absolute Accuracy a) P
 oint cloud without CP-based b) Point cloud with CP-based
Global cloud-to-cloud optimization optimization

68 % of measurements are below 5.1 mm 4.5 mm

95 % of measurements are below 12.9 mm 12.7 mm

From this evidence, we can state that the absolute accuracy for model b). Therefore, it represents the
accuracy for test model a), which represents the point accuracy that can be achieved without using CPs
cloud without CP-based optimization, is 5.1 mm at at all in this environment. The small difference
68% confidence and 12.9 mm at 95% confidence. For represents the capability of the NavVis SLAM
test model b), which represents the computationally (Simultaneous Localization and Mapping) algorithm
optimized point cloud geometry, the absolute in indoor scenarios, which enables a high-accuracy
accuracy is 4.5 mm at 68% confidence and 12.7 mm at documentation of buildings without the survey of CPs.
95% confidence.
Still, the usage of control points depends on the
Model a) is particularly interesting because it environment being mapped. It cannot be assumed
represents the accuracy that can be achieved when that reality capture of indoor locations without
no control points are used. In this concrete test control points will always have such great accuracy.
model, CPs were used for registration only, and not Furthermore, for the alignment of several data sets
for global optimization of the point cloud. Especially and the (geo-)registration, it may still be necessary to
remarkable is the fact that the absolute accuracy install and measure control points.
for model a) is only 0.6 mm higher than the absolute

7
Scenario 2: Scanning a 20 m high facade

This scenario represents a selected facade that was

part of a mapping of a residential street with houses.
The facade has an extent of 31.3 m (width) by 20.3 m
(height), resulting in 635 sqm facade area in the urban
outdoor environment.

As ground truth, one scan position with high resolution

was acquired with a highly accurate terrestrial laser
scanner. The area between 0 m and a height of 1.55 m
was left out for the analysis in both point clouds to
remove the occlusion created by parked cars in front
of the building. For the test scenario e) we conducted a
full cloud-to-cloud comparison using an in-house tool.
Below, the cumulative distribution of the deviation
between the TLS point cloud and the NavVis VLX point
cloud is shown.
Cumulative distribution of deviations resulting from the global cloud-to-
From this evidence, we can state that for the test cloud comparison.
scenario e) 68% of the points are within 5.6 mm
deviation from the ground truth and 95% of the points
are within a deviation of 12.2 mm from the ground
truth. The example proves that urban environments Absolute Accuracy e) Global cloud-to-
are feasible for fast and accurate data capture using cloud Comparison
NavVis VLX, even if they include narrow streets with
68 % of measurements are below 5.6 mm
high facades.
95 % of measurements are below 12.2 mm

Point cloud of the urban environment scan (front view)

8
Visual impression of the NavVis VLX
point cloud

Point density of the point

cloud of the NavVis VLX,
represented by number of
neighbors in a 50 cm radius

9
For deeper analysis, we cut the described facade into (~ 5 mm distance between two points in the point
three equal parts with respect to height to analyze cloud) within a height of the facade between ~1 m and
if the accuracy changes with increasing distance ~5 m, colored in red. Between the floor (0 m) and ~1 m
between the device and the object. As shown in the and ~5 m and ~10 m facade height, the point distance
frequency graphs below, the results acknowledge our increases to 12 mm in average, as colored in green.
assumption that with increasing distance between The area represented by a mixed colorization of green
scanner and object the measurements get less and blue (10 m to 15 m facade height) shows point
accurate. Still, as shown in the graph on the right, distances of 18 mm. The highest part of the object (15
even for the highest area we can state that 68 % of m to 20 m) still is represented with a point density of
all measurements are below 7.5 mm and 95 % of all 21 mm in average between two points.
measurements are below 16.9 mm.
This analysis leads to the conclusion that facades as
Additionally, we show the point density for the high shown in this example can be modeled in LOD200
facade processed with a resolution of 5 mm. As shown based on NavVis VLX data. A real-world use case
in the figure below, the point density decreases with might be the calculation of the energy efficiency of
increasing distance between the NavVis VLX and the buildings based on the opening proportion of the
area of interest. This results in a high point density facades.

Absolute Accuracy in three heights 1.55 – 7.8 m Absolute Accuracy in three heights 7.8 - 14.05 m

68 % of measurements are below 4.8 mm 68 % of measurements are below 6.8 mm

95 % of measurements are below 9.7 mm 95 % of measurements are below 14.3 mm

Point cloud of the urban environment scan (front view)

10
 Subdivision of the facade for the detailed analysis of three defined heights

Absolute Accuracy in three heights 14.05 - 20.3 m

68 % of measurements are below 7.5 mm

95 % of measurements are below 16.9 mm

11
Mapping and modeling a large family house

This example shows a three-story residential with color information. The NavVis SLAM algorithm is
building including an indoor area of approx. 250 sqm, robust enough to facilitate the connection between
property, and garden with a land area of approx. 400 the three stories of the inside of the house and the
sqm. The project was scanned in a single session in surrounding property outdoors. The point cloud
30 minutes, from the inside to the outside, without enables the creation of BIM models in LOD300 in third-
the capture of control points. Indoors, panorama party software. The panorama images further support
images were captured every 2 m, while the distance a detailed and fast modeling process. Additionally,
increased to 5 m during capture of the outer shell. We the 2D layout creation can be achieved in third-party
demonstrate the simplicity and speed of a scan-to- software based on the dense and sharp point cloud.
BIM workflow from as-built capture to BIM model.

The example shows how quickly and easily a large

family house is comprehensively mapped (including
the garden area), resulting in one data set complete

Rendering of the BIM model

Overlay between point cloud and BIM model

12
Vertical section through the large family house

Horizontal section through the large family house

13
Conclusion
The concrete results from the two scenarios and NavVis VLX is the perfect device for most indoor
one example provide a good basis for laser scanning spaces, even without using control points. In outdoor
professionals to decide if NavVis VLX is suited to projects where no larger loop closures can be
indoor and outdoor projects with specific accuracy achieved, control points are recommended for global
requirements, and to decide if and how many control optimization and additionally might be used for
points measured with a total station should be used. registration of multiple scans.

If the required deliverable is a registered as-built

Key findings survey with an accuracy level up to LOA30, NavVis
VLX together with a total station to measure control
ç For indoor environments comparable to the mid- points is very well suited for most indoor cases and
sized office shown in the first scenario, the usage of outdoor cases where adequate structures or objects
control points is of small importance for accuracy, if are visible. We cannot guarantee an accurate mapping
sufficient loop closures can be conducted during the in forests, caves, tunnels, or other environments where
mapping. manufactured objects within 60 m are missing.
ç The mapping of urban outdoor environments
and buildings facades works well, in use cases Moving forward, we will continue to conduct research
comparable to the second scenario. into other built environments and, in the long term,
ç For a facade of 20.3 m height, we achieved an overall extend the scope of this accuracy evaluation towards
accuracy of 5.6 mm and an accuracy of 7.5 mm in the challenging outdoor scenarios as forests, parks, and
higher regions above 12.5 m at 68% confidence. open areas with fewer artificial structures. Moreover,
we welcome feedback and any suggestions on how
If the required deliverable is as-built CAD or BIM to extend and standardize the evaluation of mobile
documentation for construction or refurbishment, mapping systems.

14
About NavVis
Bridging the gap between the physical and digital
world, NavVis enables service providers and
enterprises to capture and share the built environment
as photorealistic digital twins. Our SLAM-based mobile
mapping systems generate high-quality data with
survey-grade accuracy at speed and scale. And with
our digital factory solutions, users are equipped to
make better operational decisions, boost productivity,
streamline business processes, and improve
profitability. Based in Munich, Germany, with offices
in New York and Shanghai, NavVis has customers
worldwide in the surveying, AEC, and manufacturing
industries.
NavVis has the most accurate mobile mapping systems on the market.
Used together with NavVis IVION, professionals in the laser scanning,
For more information, visit www.navvis.com. surveying and AEC industries have an end-to-end reality capture solution.

Glossary of terms

AEC stands for Architecture, Engineering and Construction, a collective LOD stands for Level of Detail, the depth of the information included in
term for three associated industries. the model.

BIM stands for Building Information Modeling, the methodology of Scan-to-BIM is the process of using laser scanners to capture dimensional
creating a smart 3D model that offers a holistic view of a building to information for an existing building or location, and then using that
enable stakeholder collaboration. It enables a single stakeholder to make information to create a BIM.
decisions at various stages in the building lifecycle, and update the central
model so other stakeholders can always have the latest information. SLAM stands for Simultaneous Localization and Mapping, an algorithm
that fuses data from a mapping system’s onboard sensors to determine
CAD stands for Computer-Aided Design, the use of computers for the its trajectory at the same time as it moves through the environment. This
creation, modification, analysis, or optimization of a design. CAD software calculation is performed a huge number of times every second.
can be used to increase productivity, improve the quality of design,
improve communications through documentation, and to create a shared TLS stands for Terrestrial Laser Scanner, a device which captures data as
database. discrete measurements by scanning at a single position.

CP stands for Control Points, targets on the ground or walls which have
known coordinates. They should be measured with high precision, e.g.
using a total station. Integrating them into a mobile mapping workflow
minimizes the phenomena of drift error.

LOA stands for Level of Accuracy, the acceptable tolerance range for the
building measurements gathered during the capture process – as well as
the tolerance range for how those measurements are represented in the
model.

15
 Version 1.0 | July 2021

US HEADQUARTERS
New York
We enable service providers and
GLOBAL HEADQUARTERS
navvis.com/reality-capture enterprises to capture and share the Munich
linkedin.com/company/navvis built environment as photorealistic CHINA HEADQUARTERS
youtube.com/navvis-tech digital twins. Shanghai

16