Hindawi

Wireless Communications and Mobile Computing

Volume 2018, Article ID 6202854, 12 pages
https://doi.org/10.1155/2018/6202854

Research Article
QoE Evaluation: The TRIANGLE Testbed Approach

Almudena D-az Zayas ,1 Laura Panizo ,1 Janie Baños ,2

Carlos Cárdenas ,2 and Michael Dieudonne 3
1
University of Malaga, Spain
2
DEKRA, Spain
3
Keysight Technologies, Belgium

Correspondence should be addressed to Almudena Dı́az Zayas; adz@uma.es

Received 11 October 2018; Accepted 25 November 2018; Published 18 December 2018

Academic Editor: Giovanni Stea

Copyright © 2018 Almudena Dı́az Zayas et al. This is an open access article distributed under the Creative Commons Attribution
License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly
cited.

This paper presents the TRIANGLE testbed approach to score the Quality of Experience (QoE) of mobile applications, based
on measurements extracted from tests performed on an end-to-end network testbed. The TRIANGLE project approach is a
methodology flexible enough to generalize the computation of the QoE for any mobile application. The process produces a final
TRIANGLE mark, a quality score, which could eventually be used to certify applications.

1. Introduction (app), device, network, services infrastructure, and so on.

Therefore, TRIANGLE brings in a complete end-to-end
The success of 5G (the fifth generation of mobile commu- network testbed and a methodology for the evaluation of the
nications), and to some extent that of 4G, depends on its QoE.
ability to seamlessly deliver applications and services with Consistent with the definition, the majority of the work in
good Quality of Experience (QoE). Along with the user, QoE this area has been concerned with subjective measurements
is important to network operators, product manufacturers of experience. Typically, users rate the perceived quality on
(both hardware and software), and service providers. How- a scale, resulting in the typical MOS (Mean Opinion Score).
ever, there is still no consensus on the definition of QoE, Even in this field, the methodology for subjective assessment
and a number of acronyms and related concepts (e.g., see [1]) is the subject of many studies [3].
add confusion to the subject: QoE (Quality of Experience),
QoS (Quality of Service), QoSD (Quality of Service Deliv- However, there is a clear need to relate QoE scores
ered/achieved by service provider), QoSE (Quality of Service to technical parameters that can be monitored and whose
Experience/Perceived by customer/user), and so forth. This improvement or worsening can be altered through changes in
is a field in continuous evolution, where methodologies and the configurations of the different elements of the end-to-end
algorithms are the subject of study of many organisations and communication channel. The E-model [4], which is based on
standardization bodies such as the ITU-T. modelling the results from a large number of subjective tests
TRIANGLE project has adopted the definition of QoE done in the past on a wide range of transmission parameters,
provided by the ITU-T in Recommendation P.10/G.100 is the best-known example of parametric technique for the
(2006) Amendment 1 “Definition of Quality of Experience computation of QoE. Also, one of the conclusions of the
(QoE)” [2]. Project P-SERQU, conducted by the NGMN (Next Genera-
tion Mobile Networks) [5] and focused on the QoE analysis
“The overall acceptability of an application or of HTTP Adaptive Streaming (HAS), is that it is less complex
service, as perceived subjectively by the end-user” and more accurate to measure and predict QoE based on
In [2], the ITU-T emphasizes that the Quality of Experi- traffic properties than making a one-to-one mapping between
ence includes the complete end-to-end system effects: client generic radio and core network QoS to QoE. The TRIANGLE
2 Wireless Communications and Mobile Computing

project follows also a parametric approach to compute the area [8]. Different types of testbeds can be found in the lit-
QoE. erature, ranging from simulated to emulated mobile/wireless
Conclusions in [5] point out that a large number of testbeds, which are used to obtain subjective or objective QoE
parameters in the model could be cumbersome due to metrics, to extract a QoE model, or to assess the correctness
the difficulty of obtaining the required measurements and of a previously generated QoE model. Many of the testbeds
because it would require significantly more data points reviewed have been developed for a specific research, instead
and radio scenarios to tune the model. The TRIANGLE of for a more general purpose, such as the TRIANGLE
approach has overcome this limitation through the large testbed, which can serve a wide range of users (researchers,
variety of measurements collected, the variety of end-to- app developers, service providers, etc.). In this section, some
end network scenarios designed, and mostly the degree of QoE-related works that rely on testbeds are reviewed.
automation reached, which enables the execution of intensive
The QoE Doctor tool [12] is closely related to the TRI-
test campaigns covering all scenarios.
ANGLE testbed, since its main purpose is the evaluation of
Although there are many proposals to calculate the
quality of experience, in general, they are very much oriented mobile apps QoE in an accurate, systematic, and repeatable
to specific services, for example, voice [6] or video streaming way. However, QoE Doctor is just an Android tool that can
[7, 8]. This paper introduces a methodology to compute the take measurements at different layers, from the app user
QoE of any application, even if the application supports more interface (UI) to the network, and quantify the factors that
than one service. impact the app QoE. It can be used to identify the causes
The QoE, as perceived by the user, depends on many of a degraded QoE, but it is not able to control or monitor
factors: the network conditions, both at the core (CN) and at the mobile network. QoE Doctor uses an UI automation
the radio access (RAN), the terminal, the service servers, and tool to reproduce user behaviour in the terminal (app user
human factors difficult to control. Due to the complexity and flows in TRIANGLE nomenclature) and to measure the user-
the time needed to run experiments or make measurements, perceived latency by detecting changes on the screen. Other
most of the studies limit the evaluation of the QoE to a limited QoE metrics computed by QoE Doctor are the mobile data
set of, or even noncontrolled, network conditions, especially consumption and the network energy consumption of the app
those that affect the radio interface (fading, interference, etc.). by means of an offline analysis of the TCP flows. The authors
TRIANGLE presents a methodology and a framework to have used QoE Doctor to evaluate the QoE of popular apps
compute the QoE, out of technical parameters, weighting the such as YouTube, Facebook, or mobile web browsers. One of
impact of the network conditions based on the actual uses the drawbacks of this approach is that most metrics are based
cases for the specific application. As in ITU recommendation on detecting specific changes on the UI. Thus, the module
G1030 [9] and G1031 [10], the user’s influence factors are in charge of detecting UI changes has to be adapted for each
outside of the scope of the methodology developed in specific app under test.
TRIANGLE. QoE-Lab [13] is a multipurpose testbed that allows the
TRIANGLE has developed an end-to-end cellular net- evaluation of QoE in mobile networks. One of its purposes
work testbed and a set of test cases to automatically test is to evaluate the effect of new network scenarios on services
applications under multiple changing network conditions such as VoIP, video streaming, or web applications. To this
and/or terminals and provide a single quality score. The end, QoE-Lab extends BERLIN [14] testbed framework with
score is computed weighting the results obtained testing support for next generation mobile networks and some new
the different uses cases applicable to the application, for services, such as VoIP and video streaming. The testbed
the different aspects relevant to the user (the domains in allows the study of the effect of network handovers between
TRIANGLE), and under the network scenarios relevant for wireless technologies, dynamic migrations, and virtualized
the application. The framework allows specific QoS-to-QoE resources. Similar to TRIANGLE, the experiments are exe-
translations to be incorporated into the framework based on cuted in a repeatable and controlled environment. However,
the outcome of subjective experiments on new services. in the experiments presented in [13], the user equipment
Note that although the TRIANGLE project also provides were laptops, which usually have better performance and
means to test devices and services, only the process to test more resources than smartphones (battery, memory, and
applications is presented here. CPU). The experiments also evaluated the impact of different
The rest of the paper is organized as follows. Section 2 scenarios on the multimedia streaming services included in
provides an overview of related work. Section 3 presents an the testbed. The main limitations are that it is not possible
overview of the TRIANGLE testbed. Section 4 introduces to evaluate different mobile apps running in different smart-
the TRIANGLE approach. Section 5 describes in detail how phones or relate the QoE with the CPU, battery usage, and so
the quality score is obtained in the TRIANGLE framework. forth.
Section 6 provides an example and the outcome of this De Moor et al. [15] proposed a user-centric methodology
approach applied to the evaluation of a simple App, the for the multidimensional evaluation of QoE in a mobile real-
Exoplayer. Finally, Section 7 summarizes the conclusions. life environment. The methodology relies on a distributed
testbed that monitors the network QoS and context infor-
2. State of the Art mation and integrates the subjective user experience based
on real-life settings. The main component of the proposed
Modelling and evaluating QoE in current and next generation architecture is the Mobile Agent, a component to be installed
of mobile networks is an important and active research in the user device that monitors contextual data (location,
Wireless Communications and Mobile Computing 3

velocity, on-body sensors, etc.) and QoS parameters (CPU, 3. TRIANGLE

memory, signal strength, throughput, etc.) and provides an
interface to collect user experience feedback. A processing The testbed, the test methodology, and the set of test cases
entity receives the (device and network) monitored data and have been developed within the European funded TRIAN-
analyzes the incoming data. The objective of this testbed GLE project. Figure 1 shows the main functional blocks that
infrastructure is to study the effects of different network make up the TRIANGLE testbed architecture.
parameters in the QoE in order to define new estimation To facilitate the use of the TRIANGLE testbed for
models for QoE. different objectives (testing, benchmarking, and certifying),
In [16], the authors evaluated routing protocols BATMAN to remotely access the testbed, and to gather and present
and OLSR to support VoIP and video traffic from a QoS results, a web portal, which offers an intuitive interface,
and QoE perspective. The evaluation took place by running has been implemented. It provides access to the testbed
experiments in two different testbeds. First, experiments hiding unnecessary complexity to App developers. For
were run in the Omnet++ simulator using the InetManet advanced users interested in deeper access to configuration
framework. Second, the same network topology and network parameters of the testbed elements or the test cases, the
scenarios were deployed in the Emulab test bench, a real testbed offers a direct access to the Keysight TAP (Testing
(emulated) testbed, and the same experiments were carried Automation Platform), which is a programmable sequencer
out. Finally, the results of both testbeds (simulated and of actions with plugins that expose the configuration and
real-emulated) were statistically compared in order to find control of the instruments and tools integrated into the
inconsistencies. The experiments in the simulated and emu- testbed.
lated environments showed that BATMAN achieves better In addition to the testbed itself, TRIANGLE has devel-
than OLSR and determined the relation between different oped a test methodology and has implemented a set of
protocol parameters and their performance. These results can test cases, which are made available through the portal. To
be applied to implement network nodes that control in-stack achieve full test case automation, all the testbed components
protocol parameters as a function of the observed traffic. are under the control of the testbed management frame-
In [17], a testbed to automatically extract a QoE model work, which coordinates their configuration and execution,
of encrypted video streaming services was presented. The processes the measurements made in each test case, and
testbed includes a software agent to be installed in the computes QoE scores for the application tested.
user device, which is able to reproduce the user interaction In addition, as part of the testbed management frame-
and collect the end-user application-level measurements; the work, each testbed component is controlled through a TAP
network emulator NetEm, which changes the link conditions driver, which serves as bridge between the TAP engine and
emulating the radio or core network, and a Probe software, the actual component interface. The configuration of the
which processes all the traffic at different levels, computes different elements of the testbed is determined by the test
the TCP/IP metrics, and compares the end-user and network case to run within the set of test cases provided as part
level measurements. This testbed has been used to automat- of TRIANGLE or the customized test cases built by users.
ically construct the model (and validate the model) of the The testbed translates the test cases specific configurations,
video performance of encrypted YouTube traffic over a Wi- settings, and actions into TAP commands that take care of
Fi connection. commanding each testbed component.
More recently, in [18], Solera et al. presented a testbed TRIANGLE test cases specify the measurements that
for evaluating video streaming services in LTE networks. In should be collected to compute the KPI (Key Performance
particular, the QoE of 3D video streaming services over LTE Indicators) of the feature under test. Some measurements
was evaluated. The testbed consists of a streaming server, are obtained directly from measurement instruments but
the NetEm network emulator, and a streaming client. One others require specific probes (either software or hardware)
of the main contributions of the work is the extension of to help extract the specific measurements. Software probes,
NetEm to better model the characteristics of the packet delay running on the same device (UE, LTE User Equipment) that
in bursty services, such as video streaming. Previously to the application under test, include DEKRA Agents and the
running the experiments in the emulation-based testbed, TestelDroid [19] tool from UMA. TRIANGLE also provides
the authors carried out a simulation campaign with an an instrumentation library so that app developers can deliver
LTE simulator to obtain the configuration parameters of measurement outputs, which cannot otherwise be extracted
NetEm for four different network scenarios. These scenarios and must be provided by the application itself. Hardware
combine different positions of the user in the cell and probes include a power analyzer connected to the UE to
different network loads. From the review of these works, it measure power consumption and the radio access emulator
becomes clear that the setup of a simulation or emulation that, among others, provides internal logs about the protocol
framework for wireless or mobile environments requires, in exchange and radio interface low layers metrics.
many cases, a deep understanding of the network scenarios. The radio access (LTE RAN) emulator plays a key role in
TRIANGLE aims to reduce this effort by providing a set of the TRIANGLE testbed. The testbed RAN is provided by an
preconfigured real network scenarios and the computation off-the-shelf E7515A UXM Wireless Test Set from Keysight,
of the MOS in order to allow both researchers and app an emulator that provides state-of-the-art test features. Most
developers to focus on the evaluation of new apps, services, important, the UXM also provides radio channel emulation
and devices. for the downlink radio channel.
4 Wireless Communications and Mobile Computing

Interface and visualization (Portal)

Testbed management
Orcomposutor Measurements and data
Orchestration collections
ETL
framework
DBs
Compositor Executor
DBs
ETL modules

TAP
App
DEKRA WebDriver Android iOS TAP EPC Instrumentat …
Driver Driver Tap Driver Driver Driver ion TAP TAP Driver
Driver

RAN EPC Local application

App App App servers

UE Transport

Figure 1: TRIANGLE testbed architecture.

In order to provide an end-to-end system, the testbed 4. TRIANGLE Approach

integrates a commercial EPC (LTE Evolved Packet Core)
from Polaris Networks, which includes the main elements of The TRIANGLE testbed is an end-to-end framework devoted
a standard 3GPP compliant LTE core network, that is, MME to testing and benchmarking mobile applications, services,
(Mobility Management Entity), SGW (Serving Gateway), and devices. The idea behind the testing approach adopted
PGW (Packet Gateway), HSS (Home Subscriber Server), and in the TRIANGLE testbed is to generalize QoE computation
and provide a programmatic way of computing it. With
PCRF (Policy and Charging Rules Function). In addition,
this approach, the TRIANGLE testbed can accommodate the
this EPC includes the EPDG (Evolved Packet Data Gateway)
computation of the QoE for any application.
and ANDSF (Access Network Discovery and Session Func-
The basic concept in TRIANGLE’s approach to QoE
tion) components for dual connectivity scenarios. The RAN
evaluation is that the quality perceived by the user depends
emulator is connected to the EPC through the standard S1
on many aspects (herein called domains) and that this
interface. The testbed also offers the possibility of integrating
perception depends on its targeted use case. For example,
artificial impairments in the interfaces between the core
battery life is critical for patient monitoring applications but
network and the application servers.
less important in live streaming ones.
The Quamotion WebDriver, another TRIANGLE ele- To define the different 5G uses cases, TRIANGLE based
ment, is able to automate user actions on both iOS and its work in the Next Generation Mobile Network (NGMN)
Android applications whether they are native, hybrid, of Alliance foundational White Paper, which specifies the
fully web-based. This tool is also used to prerecord the app’s expected services and network performance in future 5G
user flows, which are needed to automate the otherwise networks [20]. More precisely, the TRIANGLE project has
manual user actions in the test cases. This completes the full adopted a modular approach, subdividing the so-called
automation operation. “NGMN Use-Cases” into blocks. The name Use Case was
Finally, the testbed also incorporates commercial mobile kept in the TRIANGLE approach for describing the appli-
devices (UEs). The devices are physically connected to the cation, service, or vertical using the network services. The
testbed. In order to preserve the radio conditions configured diversification of services expected in 5G requires a concrete
at the radio access emulator, the RAN emulator is cable con- categorization to have a sharp picture of what the user will be
ducted to the mobile device antenna connector. To accurately expected to interact with. This is essential for understanding
measure the power consumption, the N6705B power analyzer which aspect of the QoE evaluation needs to be addressed.
directly powers the device. Other measurement instruments The final use cases categorization was defined in [11] and
may be added in the future. encompasses both the services normally accessible via mobile
Wireless Communications and Mobile Computing 5

Meas 1
KPI 1 Synthetic MOS 1
Meas ...
Iteration 1
Meas P KPI ... Synthetic MOS ... Aggregation Synthetic MOS Scenario 1

Meas 1
KPI R Synthetic MOS R
Scenario 1 Meas ...
Iteration ...
Meas P

Meas 1
Synthetic MOS
Meas ...
Iteration N
Meas P

Test case
Meas 1 KPI 1 Synthetic MOS 1

Meas ...
Iteration 1
KPI ... Synthetic MOS ... Aggregation Synthetic MOS Scenario K
Meas P

Meas 1 KPI R Synthetic MOS R

Scenario K Meas ...

Iteration ...
Meas P

Meas 1

Meas ...
Iteration N
Meas P

Figure 2: The process to obtain the synthetic-MOS score in a TRIANGLE test case.

Table 1: Uses cases defined in the TRIANGLE project. features questionnaire in the portal), equivalent to the classi-
cal conformance testing ICS (Implementation Conformance
Identifier Use Case Statement), has been developed and is accessible through the
VR Virtual Reality portal. After filling the questionnaire, the applicable test plan,
GA Gaming that is, the test campaign with the list of applicable test cases,
AR Augmented Reality is automatically generated.
CS Content Distribution Streaming Services The sequence of user actions (type, swipe, tap, etc.) a user
LS Live Streaming Services needs to perform in the terminal (UE) to complete a task (e.g.,
SN Social Networking
play a video) is called the “app user flow.” In order to be able
to automatically run a test case, the actual application user
HS High Speed Internet
flow, with the user actions a user would need to perform on
PM Patient Monitoring the phone to complete certain tasks defined in the test case,
ES Emergency Services also has to be provided.
SM Smart Metering Each test case univocally defines the conditions of execu-
SG Smart Grids tion, the sequence of actions the user would perform (i.e., the
CV Connected Vehicles app user flow), the sequence of actions that the elements of
the testbed must perform, the traffic injected, the collection
of measurements to take, and so forth. In order to obtain
phones (UEs) and the ones that can be integrated in, for statistical significance, each test case includes a number
example, gaming consoles, advanced VR gear, car units, or of executions (iterations) under certain network conditions
IoT systems. (herein called scenarios). Out of the various measurements
The TRIANGLE domains group different aspects that made in the different iterations under any specific network
can affect the final QoE perceived by the users. The cur- conditions (scenario), a number of KPIs (Key Performance
rent testbed implementation supports three of the several Indicators) are computed. The KPIs are normalized into a
domains that have been identified: Apps User Experience standard 1-to-5 scale, as typically used in MOS scores, and
(AUE), Apps Energy consumption (AEC), and Applications referred to as synthetic-MOS, a terminology that has been
Device Resources Usage (RES). adopted from previous works [7, 21]. The synthetic-MOS
Table 1 provides the use cases and Table 2 lists the values are aggregated across network scenarios to produce a
domains initially considered in TRIANGLE. number of intermediate synthetic-MOS scores, which finally
To produce data to evaluate the QoE, a series of test are aggregated to obtain a synthetic-MOS score in each test
cases have been designed, developed, and implemented to case (see Figure 2).
be run on the TRIANGLE testbed. Obviously, not all test The process to obtain the final TRIANGLE mark is
cases are applicable to all applications under test, because sequential. First, for each domain, a weighted average of
not all applications need, or are designed, to support all the the synthetic-MOS scores obtained in each test case in
functionalities that can be tested in the testbed. In order the domain is calculated. Next, a weighted average of the
to automatically determine the test cases that are applicable synthetic-MOS values in all the domains of a use case is
to an application under test, a questionnaire (identified as calculated to provide a single synthetic-MOS value per use
6 Wireless Communications and Mobile Computing

Table 2: TRIANGLE domains.

Category Identifier Domain
AUE Apps User experience
AEC Apps Energy consumption
Applications RES Device Resources Usage
REL Reliability
NWR Network Resources
DEC Energy Consumption
DDP Data Performance
Mobile Devices
DRF Radio Performance
Devices DRA User experience with reference apps
IDR Reliability
IoT Devices IDP Data Performance
IEC Energy consumption

Synthetic MOS Test Case

Domain A/Use Case X/01

Synthetic MOS Domain A Use Case X

Synthetic MOS Test Case

Domain A/Use Case X/02

Synthetic MOS Use Case X

Synthetic MOS Test Case
Domain B/Use Case X/01
App Synthetic MOS Domain B Use Case X

Synthetic MOS Test Case

TRIANGLE MARK
Domain B/Use Case X/02

Synthetic MOS Test Case

Synthetic MOS Domain A Use Case Y
Domain A/Use Case Y/01

Synthetic MOS Use Case Y

Synthetic MOS Test Case

Synthetic MOS Domain B Use Case Y
Domain B/Use Case Y/01

Figure 3: The process to obtain the TRIANGLE mark.

case. An application will usually be developed for one specific individual feature, aspect, or behaviour of the application
use case, as those defined in Table 1, but may be designed for under test, as shown in Figure 4.
more than one use case. In the latter case, a further weighted Each test case defines a number of measurements, and
average is made with the synthetic-MOS scores obtained in because the results of the measurements depend on many
each use case supported by the application. These sequential factors, they are not, in general, deterministic, and, thus,
steps produce a single TRIANGLE mark, an overall quality each test case has been designed not to perform just one
score, as shown in Figure 3. single measurement but to run a number of iterations (N)
This approach provides a common framework for testing of the same measurement. Out of those measurements, KPIs
applications, for benchmarking applications, or even for are computed. For example, if the time to load the first
media frame is the measurement taken in one specific test
certifying disparate applications. The overall process for an
case, the average user waiting time KPI can be calculated by
app that implements features of different use cases is depicted
computing the mean of the values across all iterations. In
in Figure 3. general, different use case-domain pairs have a different set of
KPIs. The reader is encouraged to read [11] for further details
5. Details of the TRIANGLE QoE Computation about the terminology used in TRIANGLE.
Recommendation P.10/G.100 Amendment 1 Definition of
For each use case identified (see Table 1) and domain (see Quality of Experience [2] notes that the overall acceptability
Table 2), a number of test cases have been developed within may be influenced by user expectations and context. For
the TRIANGLE project. Each test case intends to test an the definition of the context, technical specifications ITU-T
Wireless Communications and Mobile Computing 7

i) Uses cases ii) Domains iii)Test case

AUE/CS/01
Feature: Non-
interactive
playback
Apps User Experience
•Time to load first

Measurements
media frame

Apps Device Resources •Playback Cut-off

•Video resolution
•Content Stall
Apps Energy Consumption

iv) Context v) Test case execution vi) Synthetic MOS

Network App User Flow

scenarios Measurements KPIs MOS

Figure 4: QoE computation steps.

G1030 “Estimating end-to-end performance in IP networks

for data applications” [9] and ITU-T G1031 “QoE factors in Perform login step (if
web-browsing” [10] have been considered in TRIANGLE. required) and wait for
In particular, ITU-T G1031 [10] identifies the following con- 10 seconds
text influence factors: location (cafeteria, office, and home),
interactivity (high-level interactivity versus low-level inter-
activity), task type (business, entertainment, etc.), and task Start playing a
urgency (urgent versus casual). User’s influence factors are, video of 5
however, outside of the scope of the ITU recommendation. minutes during
In the TRIANGLE project, the context information has 10 seconds
been captured in the networks scenarios defined (Urban -
Internet Cafe Off Peak; Suburban - Shopping Mall Busy
Hours; Urban – Pedestrian; Urban – Office; High speed train
Pause the
– Relay; etc.) and in the test cases specified in [11].
The test cases specify the conditions of the test but reproduction
also a sequence of actions that have to be executed by the
application (app user flows) to test its features. For example,
the test case that tests the “Play and Pause” functionality Resume the
defines the app user flow shown in Figure 5. reproduction after
The transformation of KPIs into QoE scores is the most 2 minutes and
challenging step in the TRIANGLE framework. The execu-
until the end of
tion of the test cases will generate a significant amount of raw
the video
measurements about several aspects of the system. Specific
KPIs can then be extracted through statistical analysis: mean, Figure 5: App user flow used in the “AUE/CS/02 Play and Pause”
deviation, cumulative distribution function (CDF), or ratio. test case.
The KPIs will be individually interpolated in order to pro-
vide a common homogeneous comparison and aggregation
space. The interpolation is based on the application of two function maps a value, v, of a KPI, to v’ (synthetic-MOS) in
functions, named Type I and Type II. By using the proposed the range [1-to-5] by computing the following formula:
two types of interpolations, the vast majority of KPIs can be
translated into normalized MOS-type of metric (synthetic- V − 𝑚𝑖𝑛𝐾𝑃𝐼
MOS), easy to be averaged in order to provide a simple, V󸀠 = (5.0 − 1.0) + 1.0 (1)
𝑚𝑎𝑥𝐾𝑃𝐼 − 𝑚𝑖𝑛𝐾𝑃𝐼
unified evaluation.
This function transforms a KPI to a synthetic-MOS value by
Type I. This function performs a linear interpolation on the applying a simple linear interpolation between the worst and
original data. The variables 𝑚𝑖𝑛𝐾𝑃𝐼 and 𝑚𝑎𝑥𝐾𝑃𝐼 are the worst best expected values from a reference case. If a future input
and best known values of a KPI from a reference case. The case falls outside the data range of the KPI, the new value will
8 Wireless Communications and Mobile Computing

Table 3: AUE/CS/002 test case description.

Identifier AUE/CS/002 (App User Experience/Content Streaming/002)

Title Play and pause
Objective Measure the ability of the AUT to pause and the resume a media file.
Applicability (ICSG ProductType = Application) AND (ICSG UseCases includes CS) AND ICSA CSPause
Initial Conditions AUT in in [AUT STARTED] mode. (Note: Defined in D2.2 [11] Appendix 4)
(1) The Test System commands the AUT to replay the Application User Flow (Application User Flow that
Steps presses first the Play button and later the Pause button).
(2) The Test System measures whether pause operation was successful or not.
Postamble (i) Execute the Postamble sequence (see section 2.6 in D2.2 [11] Appendix 4)
(i) Playback Cut-off: Probability that successfully started stream reproduction is ended by a cause other than
the intentional termination by the user.
Measurements (Raw) (ii) Pause Operation: Whether pause operation is successful or not.
(iii) Time to load first media frame (s) after resuming: The time elapsed since the user clicks resume button
until the media reproduction starts.
(Note: For Exoplayer the RESUME button is the PLAY button)

be set to the extreme value minKPI (if it is worse) or maxKPI Table 4: Measurement points associated with test case AUE/CS/002.
(if it is better).
Measurements Measurement points
Media File Playback - Start
Type II. This function performs a logarithmic interpolation Time to load first media frame
Media File Playback - First Picture
and is inspired on the opinion model recommended by the
ITU-T in [9] for a simple web search task. This function maps Media File Playback - Start
Playback cut-off
a value, v, of a KPI, to v’ (synthetic-MOS) in the range [1-to-5] Media File Playback - End
by computing the following formula: Pause Media File Playback - Pause

5.0 − 1.0
V󸀠 =
ln ((𝑎 ∗ 𝑤𝑜𝑟𝑠𝑡𝐾𝑃𝐼 + 𝑏) /𝑤𝑜𝑟𝑠𝑡𝐾𝑃𝐼 ) (2)
is described in this section. This application only has one use
case: content distribution streaming services (CS).
∙ (ln (V) − ln (𝑎 ∗ 𝑤𝑜𝑟𝑠𝑡𝐾𝑃𝐼 + 𝑎)) + 5 Exoplayer is an application level media player for Android
promoted by Google. It provides an alternative to Android’s
The default values of 𝑎 and 𝑏 correspond to the simple web MediaPlayer API for playing audio and video both locally and
search task case (𝑎 = 0,003 and 𝑏 = 0,12) [9, 22] and the over the Internet. Exoplayer supports features not currently
worst value has been extracted from the ITU-T G1030. If supported by Android’s MediaPlayer API, including DASH
during experimentation a future input case falls outside the and SmoothStreaming adaptive playbacks.
data range of the KPI, the parameters 𝑎 and 𝑏 will be updated The TRIANGLE project has concentrated in testing just
accordingly. Likewise, if through subjective experimentation two of the Exoplayer features: “Noninteractive Playback”
other values are considered better adjustments for specific and “Play and Pause.” These features result in 6 test cases
services, the function can be easily updated. applicable, out of the test cases defined in TRIANGLE. These
Once all KPIs are translated into synthetic-MOS values, are test cases AUE/CS/001 and AUE/CS/002, in the App User
they can be averaged with suitable weights. In the averaging Experience domain, test cases AEC/CS/001 and AEC/CS/002,
process, the first step is to average over the network scenarios in the App Energy Consumption domain, and test cases
considered relevant for the use case, as shown in Figure 2. RES/CS/001 and RES/CS/002, in the Device Resources Usage
This provides the synthetic-MOS output value for the test domain.
case. If there is more than one test case per domain, which is The AUE/CS/002 “Play and Pause” test case description,
generally the case, a weighted average is calculated in order to belonging to the AUE domain, is shown in Table 3. The test
provide one synthetic-MOS value per domain, as depicted in case description specifies the test conditions, the generic app
Figure 3. The final step is to average the synthetic-MOS scores user flow, and the raw measurements, which shall be collected
over all use cases supported by the application (see Figure 3). during the execution of the test.
This provides the final score, that is, the TRIANGLE mark. The TRIANGLE project also offers a library that includes
the measurement points that should be inserted in the
6. A Practical Case: Exoplayer under Test source code of the app for enabling the collection of the
measurements specified. Table 4 shows the measurement
For better understanding, the complete process of obtaining points required to compute the measurements specified in
the TRIANGLE mark for a specific application, the Exoplayer, test case AUE/CS/002.
Wireless Communications and Mobile Computing 9

Table 5: Reference values for interpolation.

Feature Domain KPI Synthetic MOS Calculation KPI min KPI max
Non-Interactive Playback AEC Average power consumption Type I 10 W 0.8 W
Non-Interactive Playback AUE Time to load first media frame Type II KPI worst=20 ms
Non-Interactive Playback AUE Playback cut-off ratio Type I 50% 0
Non-Interactive Playback AUE Video resolution Type I 240p 720p
Non-Interactive Playback RES Average CPU usage Type I 100% 16%
Non-Interactive Playback RES Average memory usage Type I 100% 40%
Play and Pause AEC Average power consumption Type I 10 W 0.8 W
Play and Pause AUE Pause operation success rate Type I 50% 100%
Play and Pause RES Average CPU usage Type I 100% 16%
Play and Pause RES Average memory usage Type I 100% 40%

The time to load first media picture measurement is Video Resolution

obtained subtracting the timestamp of the measurement 1200
point “Media File Playback – Start” from the measurement

Horizontal Resolution
1000
point “Media File Playback – First Picture.”
As specified in [11], all scenarios defined are applicable 800
to the content streaming use case. Therefore, test cases in 600
the three domains currently supported by the testbed are 400
executed in all the scenarios.
200
Once the test campaign has finished, the raw measure-
ment results are processed to obtain the KPIs associated with 0
00:00.000

00:17.280

00:34.560

00:51.840

01:09.120

01:26.400

01:43.680

02:00.960
each test case: average current consumption, average time to
load first media frame, average CPU usage, and so forth. The
processes applied are detailed in Table 5. Based on previous
experiments performed by the authors, the behaviour of the Timestamp
time to load the first media frame KPI resembles the web Figure 6: Video Resolution evolution in the Driving Urban Normal
response time KPI (i.e., the amount of time the user has scenario.
to wait for the service) and thus, as recommended in the
opinion model for web search introduced in [9], a logarithmic
interpolation (type II) has been used for this metric.
The results of the initial process, that is, the KPIs compu- adaptation algorithm is basically throughput-based and some
tation, are translated into synthetics-MOS values. To compute parameters control how often and when switching can occur.
these values, reference benchmarking values for each of the During the testing, the testbed was configured with the
KPIs need to be used according to the normalization and different network scenarios defined in [11]. In these scenarios,
interpolation process described in Section 5. Table 5 shows the network configuration changes dynamically following a
what has been currently used by TRIANGLE for the App random pattern, resulting in different maximum throughput
User Experience domain, which is also used by NGMN as rates. The expected behaviour of the application under test
reference in their precommercial Trials document [23]. is that the video streaming client adapts to the available
For example, for the “time to load first media frame” KPI throughput by decreasing or increasing the resolution of the
shown in Table 5, the type of aggregation applied is averaging received video. Figure 6 depicts how the client effectively
and the interpolation formula used is Type II. adapts to the channel conditions.
To achieve stable results, each test case is executed 10 However, the objective of the testing carried out in the
times (10 iterations) in each network scenario. The synthetic- TRIANGE testbed is not just to verify that the video stream-
MOS value in each domain is calculated by averaging the ing client actually adapts to the available maximum through-
measured synthetic-MOS values in the domain. For example, put but also to check whether this adaptation improves the
synthetic-MOS value is the RES domain obtained by aver- users’ experience quality.
aging the synthetic-MOS value of “average CPU usage” and Table 6 shows a summary of the synthetic-MOS values
“average memory usage” from the two test cases. obtained per scenario in one test case of each domain. The
Although Exoplayer supports several video streaming scores obtained in the RES and AEC domains are always high.
protocols, in this work only DASH [24] (Dynamic Adaptive In the AUE domain, the synthetic MOS associated with the
Streaming over HTTP) has been tested. DASH clients should Video Resolution shows low scores in some of the scenarios
seamlessly adapt to changing network conditions by making because the resolution decreases, reasonable good scores in
decisions on which video segment to download (videos the time to load first media, and high scores in the time to
are encoded at multiple bitrates). The Exoplayer’s default playback cut-off ratio. Overall, it can be concluded that the
10 Wireless Communications and Mobile Computing

Table 6: Synthetic MOS values per test case and scenario for the feature “Noninteractive Playback”.

AUE domain AEC domain RES domain

Test Case
Test Case AUE/CS/001 Test Case RES/CS/001
AEC/CS/001
Time to load Video Average
Playback Average CPU Average
Scenario first media Resolution Power
Cut-off ratio Usage RAM Usage
frame mode Consumption
HighSpeed Direct
2.1 3.1 2.3 4.7 4.3 4.2
Passenger
Suburban Festival 3.8 4.7 3.1 4.8 4.3 4.1
Suburban shopping mall
3.7 3.7 1.3 4.8 4.4 4.1
busy hours
Suburban shopping mall
3.6 3.1 2.3 4.8 4.3 4.1
off-peak
Suburban stadium 3.8 2.9 2.1 4.7 4.4 4.1
Urban Driving Normal 2.6 3.9 2.8 4.7 4.4 4
Urban Driving Traffic
3.4 3.7 1.6 4.8 4.4 4
Jam
Urban Internet Café Busy
3.8 3.7 1.9 4.8 4.4 4
Hours
Urban Internet Café Off
3.8 3.1 2.3 4.8 4.3 4
Peak
Urban Office 3.8 4.7 3.3 4.8 4.5 4.3
Urban Pedestrian 3.9 2.6 2 4.7 4.4 4
3.5 3.6 2.3 4.7 4.4 4.1

DASH implementation of the video streaming client under and enables the execution of extensive and repeatable test
test is able to adapt to the changing conditions of the network, campaigns to obtain meaningful QoE scores. The TRIANGLE
maintaining an acceptable rate of video cut-off, rebuffering project has also defined a methodology, which is based on the
times, and resources usage. transformation and aggregation of KPIs, its transformation
The final score in each domain is obtained by averaging into synthetic-MOS values, and its aggregation over the
the synthetic-MOS values from all the tested network scenar- different domains and use cases.
ios. Figure 7 shows the spider diagram for the three domains
tested. In the User Experience domain, the score obtained is The TRIANGLE approach is a methodology flexible
lower than the other domains, due to the low synthetic-MOS enough to generalize the computation of QoE for any applica-
values obtained for the video resolution. tion/service. The methodology has been validated testing the
The final synthetic MOS for the use case Content Dis- DASH implementation in the Exoplayer App. To confirm the
tribution Streaming is obtained as a weighted average of the suitability of the weights used in the averaging process and the
three domains, representing the overall QoE as perceived by interpolation parameters, as well as to verify the correlation
the user. The final score for the Exoplayer version 1.516 and the of the obtained MOS with that scored by users, the authors
features tested (Noninteractive Playback and Play and Pause) have started experiments with real users and initial results are
is 4.2, which means that the low score obtained in the video encouraging.
resolution is compensated with the high scores in other KPIs. The process described produces a final TRIANGLE mark,
If an application under test has more than one use case,
a single quality score, which could eventually be used to cer-
the next steps in the TRIANGLE mark project approach
would be the aggregation per use case and the aggregation tify applications after achieving a consensus on the different
over all use cases. The final score, the TRIANGLE mark, is an values of the process (weights, limits, etc.) to use.
estimation of the overall QoE as perceived by the user.
In the current TRIANGLE implementation, the weights Data Availability
in all aggregations are the same. Further research is needed
to appropriately define the weights of each domain and each The methodology and results used to support the findings of
use case in the overall score of the applications. this study are included within the article.

7. Conclusions Conflicts of Interest

The main contribution of the TRIANGLE project is the The authors declare that they have no conflicts of inter-
provision of a framework that generalizes QoE computation est.
Wireless Communications and Mobile Computing 11

DER

Device Resource Usage

User Experience Energy Consumption

AUE AEC

Figure 7: Exoplayer synthetic-MOS values per domain.

Acknowledgments [8] S. Baraković and L. Skorin-Kapov, “Survey and Challenges

of QoE Management Issues in Wireless Networks,” Journal of
The TRIANGLE project is funded by the European Union’s Computer Networks and Communications, vol. 2013, Article ID
Horizon 2020 Research and Innovation Programme (Grant 165146, 28 pages, 2013.
Agreement no. 688712). [9] ITU-T, “G.1030: estimating end-to-end performance in IP
networks for data applications,” 2014.
References [10] ITU-T, “G.1031 QoE factors in web-browsing,” 2014.

[1] ETSI, “Human factors: quality of experience (QoE) require- [11] EU H2020 TRIANGLE Project, Deliverable D2.2 Final report
ments for real-time communication services,” Tech. Rep. 102 on the formalization of the certification process, requirements
643, 2010. and use cases, 2017, https://www.triangle-project.eu/project-old/
deliverables/.
[2] ITU-T, “P.10/G.100 (2006) amendment 1 (01/07): new appendix
I - definition of quality of experience (QoE),” 2007. [12] Q. A. Chen, H. Luo, S. Rosen et al., “QoE doctor: diagnosing
[3] F. Kozamernik, V. Steinmann, P. Sunna, and E. Wyckens, mobile app QoE with automated UI control and cross-layer
“SAMVIQ - A new EBU methodology for video quality evalua- analysis,” in Proceedings of the Conference on Internet Mea-
tions in multimedia,” SMPTE Motion Imaging Journal, vol. 114, surement Conference (IMC ’14), pp. 151–164, ACM, Vancouver,
no. 4, pp. 152–160, 2005. Canada, November 2014.
[4] ITU-T, “G.107 : the E-model: a computational model for use in [13] M. A. Mehmood, A. Wundsam, S. Uhlig, D. Levin, N. Sarrar,
transmission planning,” 2015. and A. Feldmann, “QoE-Lab: Towards Evaluating Quality of
[5] J. De Vriendt, D. De Vleeschauwer, and D. C. Robinson, “QoE Experience for Future Internet Conditions,” in Testbeds and
model for video delivered over an LTE network using HTTP Research Infrastructure, Korakis T., Li H., Tran-Gia P., and H.
adaptive streaming,” Bell Labs Technical Journal, vol. 18, no. 4, S. Park, Eds., vol. 90 of TridentCom 2011, Lnicst, pp. 286–301,
pp. 45–62, 2014. Springer, Development of Networks and Communities, Berlin,
[6] S. Jelassi, G. Rubino, H. Melvin, H. Youssef, and G. Pujolle, Germany, 2012.
“Quality of Experience of VoIP Service: A Survey of Assessment [14] D. Levin, A. Wundsam, A. Mehmood, and A. Feldmann,
Approaches and Open Issues,” IEEE Communications Surveys & “Berlin: The Berlin Experimental Router Laboratory for Inno-
Tutorials, vol. 14, no. 2, pp. 491–513, 2012. vative Networking,” in TridentCom 2010. Lnicst, T. Magedanz,
[7] M. Li, C.-L. Yeh, and S.-Y. Lu, “Real-Time QoE Monitoring A. Gavras, N. H. Thanh, and J. S. Chase, Eds., vol. 46 of Lecture
System for Video Streaming Services with Adaptive Media Play- Notes of the Institute for Computer Sciences, Social Informatics
out,” International Journal of Digital Multimedia Broadcasting, and Telecommunications Engineering, pp. 602–604, Springer,
vol. 2018, Article ID 2619438, 11 pages, 2018. Heidelberg, Germany, 2011.
12 Wireless Communications and Mobile Computing

[15] K. De Moor, I. Ketyko, W. Joseph et al., “Proposed framework

for evaluating quality of experience in a mobile, testbed-
oriented living lab setting,” Mobile Networks and Applications,
vol. 15, no. 3, pp. 378–391, 2010.
[16] R. Sanchez-Iborra, M.-D. Cano, J. J. P. C. Rodrigues, and J.
Garcia-Haro, “An Experimental QoE Performance Study for the
Efficient Transmission of High Demanding Traffic over an Ad
Hoc Network Using BATMAN,” Mobile Information Systems,
vol. 2015, Article ID 217106, 14 pages, 2015.
[17] P. Oliver-Balsalobre, M. Toril, S. Luna-Ramı́rez, and R. Garcı́a
Garaluz, “A system testbed for modeling encrypted video-
streaming service performance indicators based on TCP/IP
metrics,” EURASIP Journal on Wireless Communications and
Networking, vol. 2017, no. 1, 2017.
[18] M. Solera, M. Toril, I. Palomo, G. Gomez, and J. Poncela, “A
Testbed for Evaluating Video Streaming Services in LTE,”
Wireless Personal Communications, vol. 98, no. 3, pp. 2753–2773,
2018.
[19] A. Álvarez, A. Dı́az, P. Merino, and F. J. Rivas, “Field mea-
surements of mobile services with Android smartphones,”
in Proceedings of the IEEE Consumer Communications and
Networking Conference (CCNC ’12), pp. 105–109, Las Vegas, Nev,
USA, January 2012.
[20] NGMN Alliance, “NGMN 5G white paper,” 2015, https://www
.ngmn.org/fileadmin/ngmn/content/downloads/Technical/2015/
NGMN 5G White Paper V1 0.pdf.
[21] “Infrastructure and Design for Adaptivity and Flexibility,” in
Mobile Information Systems, Springer, 2006.
[22] J. Nielsen, “Response Times: The Three Important Limits,” in
Usability Engineering, 1993.
[23] NGMN Alliance, “Definition of the testing framework for the
NGMN 5G pre-commercial networks trials,” 2018, https://www
.ngmn.org/fileadmin/ngmn/content/downloads/Technical/2018/
180220 NGMN PreCommTrials Framework definition v1 0.pdf.
[24] 3GPP TS 26.246, “Transparent end-to-end Packet-switched
Streaming Services (PSS); Progressive Download and Dynamic
Adaptive Streaming over HTTP (3GP-DASH),” 2018.
 International Journal of

Rotating Advances in
Machinery Multimedia

The Scientific
Engineering
Journal of
Journal of

Hindawi
World Journal
Hindawi Publishing Corporation Hindawi
Sensors
Hindawi Hindawi
www.hindawi.com Volume 2018 http://www.hindawi.com
www.hindawi.com Volume 2018
2013 www.hindawi.com Volume 2018 www.hindawi.com Volume 2018 www.hindawi.com Volume 2018

Journal of

Control Science
and Engineering

Advances in
Civil Engineering
Hindawi Hindawi
www.hindawi.com Volume 2018 www.hindawi.com Volume 2018

Submit your manuscripts at

www.hindawi.com

Journal of
Journal of Electrical and Computer
Robotics
Hindawi
Engineering
Hindawi
www.hindawi.com Volume 2018 www.hindawi.com Volume 2018

VLSI Design
Advances in
OptoElectronics
International Journal of

International Journal of
Modelling &
Simulation
Aerospace
Hindawi Volume 2018
Navigation and
Observation
Hindawi
www.hindawi.com Volume 2018
in Engineering
Hindawi
www.hindawi.com Volume 2018
Engineering
Hindawi
www.hindawi.com Volume 2018
Hindawi
www.hindawi.com www.hindawi.com Volume 2018

International Journal of
International Journal of Antennas and Active and Passive Advances in
Chemical Engineering Propagation Electronic Components Shock and Vibration Acoustics and Vibration
Hindawi Hindawi Hindawi Hindawi Hindawi
www.hindawi.com Volume 2018 www.hindawi.com Volume 2018 www.hindawi.com Volume 2018 www.hindawi.com Volume 2018 www.hindawi.com Volume 2018