USOO8027264B2

(12) United States Patent (10) Patent No.: US 8,027.264 B2

Erman et al. (45) Date of Patent: Sep. 27, 2011

(54) METHOD AND APPARATUS FOR (58) Field of Classification Search .................. 370/250,
MEASURING SUBJECTIVE ASSESSMENT OF 370/254, 255, 401
DIGITAL VIDEO IMPAIRMENT See application file for complete search history.
(75) Inventors: Bilgehan Erman, Marlboro, NJ (US); (56) References Cited
Vanita K. Katkar, White House Station,
NJ (US); Elissa P. Matthews, Watchung, U.S. PATENT DOCUMENTS
NJ (US) 5,596,364 A * 1/1997 Wolfetal. .................... 348.192
5,825,807 A * 10/1998 Kumar ... ... 375/130
6,741,569 B1* 5/2004 Clark ...... 370.252
(73) Assignee: Alcatel Lucent, Paris (FR) 7,218,928 B2* 5/2007 Parket al. ... ... 455,423
- 2004/0088405 A1* 5/2004 Aggarwal ...... 709,224
(*) Notice: Subject to any disclaimer, the term of this 2004/0210655 A1 * 10, 2004 Martinot et al. TO9,224
patent is extended or adjusted under 35 2005, OO64.820 A1* 3, 2005 Park et al. ...... 455/67.11
U.S.C. 154(b) by 533 days. 2007/0053303 A1* 3/2007 Kryuchkov. ................... 370/250
* cited by examiner
(21) Appl. No.: 11/756,839
Primary Examiner — Hong Cho
(22) Filed: Jun. 1, 2007 (74) Attorney, Agent, or Firm — Wall & Tong, LLP
(65) Prior Publication Data (57) ABSTRACT
US 2008/O298.448A1 Dec. 4, 2008 A method and system for using key performance indicators of
a transport channel to determine key quality indicators asso
(51) Int. Cl. ciated with information transported there through.
H04L 2/26 (2006.01)
(52) U.S. Cl. ........................................ 370/250; 370/255 20 Claims, 4 Drawing Sheets

120
CONTENT 110 130
FROM SOURCE -O-
OTHER TRANSPORT - 116 CONTEN DESTINATION
CONTENT PROCESSOR TRANSPOR
SOURCE --------- 114 DEMUX DECODER

MEMORY - 112 - - - -
NETWORK PRESENTATION
MANAGERMONITOR DEVICE

MPAIRMENT DATA 170

CONSUMERS
SERVICE - 172 IMPAIRMENTASSESSOR 160
PROVIDERS 183
INPUTIOUTPUT 162 1CO
NETWORK - 174 PROCESSOR 164
MANAGERS
MEMORY 66
PROVISIONING
DECISION - 176 NORMINIZATION MODULE NM
DEVICES
WOCALIBRATOR WCC
OTHER -- 178 WQANALYSER WOA
OPERATIONS - 179 SUBJECTIVE INPUT SPM
SUPPORT PREPARATIONMODULE
SYSTEM
 US 8,027,264 B2
1. 2
METHOD AND APPARATUS FOR mission channel impairments to which the content stream
MEASURING SUBJECTIVE ASSESSMENT OF may be subjected without resulting in a presentation artifacts
DIGITAL VIDEO IMPAIRMENT deemed by a user to be objectionable. Within the context of
the present invention, parameters indicative of the quality of
FIELD OF THE INVENTION the transport channel are measured and/or determined. These
channel-related quality parameters are associated with Sub
The present invention relates to communication systems sequent presentation artifacts. Certain quality parameters
and, more specifically, the assessment of information impair have been found by the inventors to be strongly related to the
ments such as video information impairments by correlating type and degree of presentation artifacts.
both objective and subjective viewpoints. 10 A transport channel is deemed to include at least those
portions of a transport network routing, passing or otherwise
BACKGROUND OF THE INVENTION Supporting a content stream of interest. Optionally, the inven
tion is expanded in scope beyond the transport network. That
is, the scope of context of the present invention may exceed
The transmission of video streams over internet protocol the scope of error analysis discussed herein. FIG. 4 depicts a
(IP) networks (referred to as IP-Video) may impair video 15
block diagram illustrating functional elements associated
quality in specific ways. Key considerations for how network with the transport of a content stream from a source to desti
errors impair video image (image quality loss) include the nation. Specifically, an audiovisual source 410 provides a
type of equipment used to decode the video stream and the baseband audiovisual stream to an encoder/multiplexer 420.
subjective perception of the human viewer. The complexity of The resulting encoded/multiplexed stream is coupled from
Video decoding and the Subjective nature of the image quality the encoder/multiplexer 420 to a decoder/demultiplexer 470
assessment make quantitative interpretation of “quality dif via a core network 430, an aggregation network 440, an
ficult. access network 450 and a home network 460. The decoder/
SUMMARY
demultiplexer 470 extracts and decodes the original audiovi
Sual stream for presentation by, for example, an audiovisual
25 monitor 480 which receives the content stream as, for
Various deficiencies of the prior art are addressed by the example, a baseband signal.
present invention of a method and system for using key per The scope of error analysis discussed herein is directed to
formance indicators of a transport channel to determine key error Sources within the core 430, aggregation 440, access
quality indicators associated with information transported 450 and home 460 networks. However, the scope of context
there through. In this manner, an impairment level and/or for the invention further includes the encoder/multiplexer 420
presentation quality level associated with the information is 30
and the decoder/demultiplexer 470 as well as the baseband
determined. signals at the endpoints. Thus, while the discussion of impair
BRIEF DESCRIPTION OF THE DRAWINGS
ment of data will be primarily directed to network-induced
impairments of a content stream, other impairments of the
content stream may also be measured and used to determine
The teachings of the present invention can be readily 35 the impairment level and/or ultimate presentation quality of
understood by considering the following detailed description that stream.
in conjunction with the accompanying drawings, in which: Impairments may be imparted during the encoding pro
FIG. 1 depicts a high-level block diagram of a content cess, the transport process or the decoding process. Such
delivery system adapted according to the present invention; other impairments include encoding artifacts, decoding arti
FIG. 2 depicts a high-level block diagram of key perfor 40 facts, set top box (STB) buffer limitations and/or jitter con
mance indicators (KPIs) as related to key quality indicators ditions, Source or destination storage limitations and the like.
(KQIs); The present invention is adapted to determining the likely
FIG.3 depicts a high-level block diagram depicting inter impairment to a video, audio or audiovisual stream; Stated
relationships of several functional elements according to an more generally, the likely impairment to any information
embodiment of the present invention; and 45
stream or data structure.
FIG. 4 depicts a block diagram illustrating functional ele Within the context of a content stream delivered via an IP
ments associated with the transport of a content stream from network or other packet switched network, artifacts may be
a source to destination. induced by packet jitter, packet loss, and the like. An impor
To facilitate understanding, identical reference numerals tant consideration is the amount of visual and/or aural arti
have been used, where possible, to designate identical ele facts that a customer or user will tolerate prior to finding the
ments that are common to the figures. 50 content stream objectionable. For purposes of this discussion
it will be assumed that the content has been encoded to an
DETAILED DESCRIPTION OF THE INVENTION appropriate or adequate quality level. That is, if the encoded
content is decoded and presented without any transmission
The invention will be primarily described within the con channel impairments being induced, the Subsequent presen
text of a content delivery system adapted according to the 55 tation of content will not be objectionable to a user.
present invention. However, those skilled in the art and Software executing methods according to the present
informed by the teachings herein will realize that the inven invention may be resident within a network management
tion is also applicable to content or information delivery system (NMS), an element management system (EMS), a
systems having differing topologies and/or methodologies remote processing system or other suitable computing device
than those discussed herein. In essence, the Subject invention 60
able to receive and process the quality parameters of the
is applicable to any information delivery system in which transport channel carrying the content stream of interest.
propagation of information through the system may result in Broadly speaking, within the context of the present inven
a degradation of Subsequently presented or utilized informa tion an assessment of transport channel anomalies is made for
tion. the transport channel carrying the content stream of interest.
During transmission of the content from a content source to The assessed transport channel anomalies are associated with
the content destination, presentation artifacts may be induced 65 particular impairments of the content stream, Such impair
within the content stream via transmission channel impair ments resulting in noticeable artifacts or other degradation at
ments. Of particular interest is the amount and type of trans data information presentation/use.
 US 8,027,264 B2
3 4
One embodiment of the invention is a method and appara TABLE 2
tus for determining the impairments likely to be induced in a
Video stream or other stream Such that modifications can be Category Key Performance Indicators (KPI)
made to the content bearing information channel(s) to ensure Network Packet loss
that a Sufficient quality of service level is maintained. In 5 Performance Bursty Jitter
another embodiment of the invention, planned modifications Reordering
for one or more content bearing information channel(s) are Bit-error rate (BER)
Network Decoder buffer size
first evaluated with respect to their impact on the quality of Configuration Decoder type
multiple content streams propagated there through. That is, Transport protocol
the determination or modeling of the effect of transport chan 10 Access line bandwidth
Access technology
nel related impairments to a content stream provides an CBRVBR
opportunity to optimize the quality of the channel based on Content & Service Video frame size
the needs of the particular information stream, the needs of Profile Coding protocol
multiple information streams, the economic model of trans Motion
Video bit-rate
port entities and/or service providers and their respective user 15
communities and so on.
The Subject invention uses key performance indicators A video quality impairment observation threshold is a
(KPIs) associated with network performance, network con threshold level associated with a network error that will likely
figuration and/or content and service profiles to characterize cause a degradation in video quality that either noticeable
a likely impact on a content stream propagated through a (low threshold level) or unacceptable (high threshold level) to
network. For example, quality-related characteristics of the a viewer.
transport channel are determined by monitoring the IP flow of FIG. 1 depicts a high-level block diagram of a content
the transport channel, monitoring quality parameters of the delivery system adapted according to the present invention.
paths from a transport channel, identifying constrained band Specifically, FIG. 1 depicts a content delivery system 100 in
width portions of the transport channel path and, generally 25
which content, such as audiovisual content, video content
speaking, extracting key attributes associated with the trans
port channel carrying the transport stream of interest. and/or related or other data are propagated from a content
source 110 to a content destination 130 via a network 120. A
The impact of the KPIs on a content stream is defined in
terms of key quality indicators (KQIS) to characterize arti presentation device 140. Such as a television, computer moni
facts and/or aberrations of Subsequently presented content. 30 tor, audiovisual system and the like, is used to present content
Thus, the measures of subjective perception are defined by the delivered to the content destination 130. The network 120 is
inventors as key quality indicators (KQI). managed and/or monitored via a network managers/monitor
Thus, the objective key performance indicators captured 150. An impairment assessor 160 performs a video quality
from, for example, the network are used to map Subjective key analysis function to determine a qualitative degradation of a
quality indicators associated with a content stream trans 35 content stream propagated through the network. In this man
ported to an endpoint. The mapping is based upon a pre ner, a determination may be made as to whether adaptations in
established correlation between objective and subjective network path, network provisioning, content stream data
parameters. The correlation is derived empirically or via structure and the like are appropriate to ensure a desired or
human factors based calculations. minimum quality level for delivered and/or presented con
A service performance key quality indicator (KQI) is a 40 tent. Such a determination may be made by, for example,
quality indicator having associated with it a service attribute. impairment data consumers 170.
A table of such key quality indicators and their corresponding Within the context of the present invention, the term
service attributes is provided below as Table 1. “memory” as used herein with respect to the content source,
TABLE 1.
Service Attribute Key Quality Indicator (KQI)
PICTURE QUALITY Image element loss
Measured via gathering of Subjective Image jerkiness or progressive freezing
customer ratings Image blockiness
Non IP-affected Impairment: Color, luminosity
problems ghosting, letterboxing, aspect? Zoom
ratio formatting, horizi vert formatting.
AUDIO QUALITY Noise: static, chirping, clipping, distortion
Measured via gathering of Subjective Volume level fluctuating or inconsistent between
customer ratings channels
Sound quality naturalness
Audio channel (monofstereof quad) problems
Wrong language
SYNCHRONIZATION Audio + Video (lip synch)
milliseconds difference between Video + Program control (e.g. closed captioning)
audio frame and corresponding
video frame or between video frame
and corresponding caption frame.

A key performance indicator (KPI) is a performance indi network, content destination, network manager and/or
cator associated with it a particular category. A table of Such 65 impairment assessor is intended to be broadly construed as
key performance indicators and their course binding catego working memory (e.g., random access memory and the like),
ries is provided below as Table 2. Volatile memory, nonvolatile memory, mass storage device
 US 8,027,264 B2
5 6
memory and so on. The invention may be represented by Each of the content storage, content destination, impair
Software instructions stored in one memory and executed ment assessor and other functions discussed herein may be
within another memory. Buffering errors, jitter errors and the implemented by a special purpose computing device or gen
like associated with different memories used at times to store eral purpose computer, which simply includes one or more
a content stream may be used within the context of projecting processing elements, input/output elements and memory/
the impairment of content stream quality. storage elements.
The content source 110 is depicted as including a memory Impairment assessor 160 comprises input/output (I/O) cir
(data storage) 112, an encoder 114 and a transport processor cuitry 162, a processor 164 and a memory 166. The I/O
116. Generally speaking, the memory 112 stores encoded or circuitry 162 forms an interface between the impairment
unencoded content. The encoder 114 depicted in FIG. 1 10 assessor 160, the network manager/monitor 150, a network
includes a video encoder conforming to the MPEG-2, 120 and/or a content source 110. The processor 164 executes
MPEG-4 or other video encoding protocols. The encoder 114 software instructions retrieved from, for example, the
is used to encode content retrieved from the memory 112 or memory/storage 166 to implement thereby various methods
received in a content stream OC provided by some other and algorithms in accordance with the present invention. The
content source (e.g., a live content stream from a television 15 memory 166 is depicted as storing Software instructions and
studio, satellite feed and the like). The transport processor related data structures Supporting various functional ele
116 is operative to process the encoded content to provide a ments, including a normalization module NM, a VO-calibra
transport encoded content stream TEC for propagation to the tor VOC, a VO-analyzer VQA and a subjective input prepa
content destination via the network 120. ration module SIPM. Other functional elements (not shown)
The network 120 may comprise an Internet protocol (IP) Such as operating systems and the like may also be associated
network, an asynchronous transfer mode (ATM) network or with software instructions and data structures stored within
any other type of data or telecommunications network. More the memory 166.
over, while represented as a single network, the network 120 The normalization module NM. VQ-calibrator VOC, VO
may comprise a plurality of networks including data and/or analyzer VQA and subjective input preparation module SIPM
telecommunications and the like. Since differing networks 25 will be discussed in more detail below with respect to FIG. 3.
often have differing levels of quality, a content stream travers Generally speaking, Software used to execute or instantiate
ing several networks may have a variety of impairments these functional elements may be provided by a single com
imparted thereto. For example, optical networks and electri puting device Such as the impairment assessor 160 depicted in
cal networks and their respective control schemes impair FIG. 1, or multiple computing devices (not shown). Each of
content streams transported there through in different ways 30 the functional elements may be associated with its own com
depending upon network congestion, priority levels, error puting device or multiple computing devices. Moreover, the
correction ability and so on. impairment assessor 160 may be combined with the network
The content destination 130 is depicted as including a manager/monitor 150, provided as multiple distributed func
transport demultiplexer 132 and decoder 134. The transport tions within a hierarchy of management layers, provided as a
demultiplexer (i.e., transport processor) 132 extracts from a 35 local or remote server associated with the network 120 and so
multiplexed transport stream encoded content stream pro O.
vided by the content source 110. The decoder 134 decodes In one embodiment of the invention, impairment data con
this encoded content stream to produce a decoded content Sumers utilize the data provided by the impairment assessor
stream such as a baseband video and/or audio stream Suitable 160 to modify various operational parameters of the network
for use by the presentation device 140. 40 120, the content source 110, the content destination 130 and/
A network manager/monitor 150 may comprise a network or the network manager/monitor 150. Thus, the content deliv
management system (NMS), element management system ery system 100 of FIG. 1 depicts impairment data consumers
(EMS) or other management system. The network manager/ 170 comprising, illustratively, service providers 172, network
monitor 150 interacts with the network 120 to retrieve there managers 174, provisioning decision devices 176, operations
from information indicative of network performance. Such 45 support system and other consumers 178.
information may be derived from network monitoring The VOA calibration data set is created by the calibration
Sources including optical network performance parameters, process. The VOA calibration data set which embodies the
Switch performance parameters, router performance param subjective viewer assessment is created via the calibration
eters, congestion indicators, as well as indicators of bit error process. In one embodiment, the VOA calibration is per
rate, jitter, packet loss, burstiness and the like. Generally 50 formed by, for example, synchronizing real-time network
speaking, the network manager/monitor 150 performs stan measurements 181, network and equipment profiles 182, and
dard network management functions including the receiving content profiles 180 with user “subjective' calibration
of performance measures associated with the network. The samples. This is a calibration phase of operation.
network manager/monitor 150 conveys these performance In one embodiment, the VOA analysis is performed by
measures to the impairment assessor 160 for further process 55 associating real-time network measurements 181, network
ing in accordance with the present invention. and equipment profiles 182, and content profiles 180 with a
In various embodiments of the invention, performance pre-generated user “subjective' calibration data, which
measures of the network 120 are conveyed to the impairment allows real-time evaluation of viewing quality. This associa
assessor 160 via the network manager/monitor 150, directly tion results in a measurement of the video image quality
from the network 120, from the content source 110 and/or 60 without the excessive time, CPU and bandwidth consumption
from the content destination 130. that a full reference schema requires. This is an analysis phase
The impairment assessor 160 performs a video quality of operation.
assessment of the video content stream to be passed through Thus, in one embodiment the VOA is performed by syn
a network 120. While primarily described within the context chronizing real-time network and content profiles with a pre
of a video quality assessment, the impairment assessor 160 65 defined user “subjective” calibration domain, which allows
optionally performs audio quality assessments, data quality real-time evaluation of viewing quality. This synchronization
assessments and other quality assessments. results in a measurement of the video image quality which has
 US 8,027,264 B2
7 8
most of the accuracy of a reference testing schema without the in the art. However, the subject invention utilizes various
excessive time, CPU and bandwidth consumption that a full combinations of KPIs in conjunction with the other VQA
reference schema requires. components to provide a determination of video quality (or
FIG. 2 depicts a high-level block diagram of key perfor other content quality).
mance indicators (KPIs) as related to key quality indicators The subjective input preparation module SIPM records
(KQIs). Within the context of the present invention, a KPI viewer's subjective scores for a series of video reference
may be derived from the categories of content and service samples under varying impaired conditions, and generates a
profiles, network configuration and/or network performance. set of response scores for, illustratively, four Video Quality
FIG. 2 discloses 16 primary key performance indicators dis Indicators (VQIs). These VOIs are determined by a) identi
tributed among the three categories. Additional key perfor 10 fying a set of key quality indicators (KQIS) that provide a
mance indicators are also disclosed in FIG. 2. Further key comprehensive view of digital video quality as it is experi
performance indicators may be identified by those skilled in enced by television viewers and then b) identifying those
the art and informed by the teachings of the present invention. KQIs which are most strongly impacted by network prob
Primary key performance indicators for content and Ser lems. These four VOIs comprise image element loss, image
Vice profiles include, for example, one or more of frame size, 15 jerkiness, image blockiness and overall audio quality. The
number of programs, media types, coding protocol, motion SIPM may be implemented empirically by having viewers
and/or bit rate. Primary key performance indicators for net grade image quality with respect to the VOIs and/or by cal
work configuration include, for example, bandwidth, access culation according to human factor principles. In the case of
technology, decoder buffer size, decoder buffer type, use of an empirically derived data set, in one embodiment multiple
constant bit rate or variable bit rate streams and/or the trans viewers indicate a level of quality (e.g., 0% to 100%) for each
port protocol used. Primary key performance indicators for of a plurality of time slices (e.g., 10 second intervals) as
network performance include, for example, jitter, packet loss, various performance indicators are changed. In this manner,
bit error rate (BER) and/or packet reordering. multiple viewers provide quality assessments of the content
The Subject invention operates to process one or more of streams associated with many KPI combinations.
the key performance indicators from the content and service 25 A Video Quality Analyzer (VQA) according to the inven
profiles 180, network configuration 182 and/or network per tion comprises a method and system for comprehensive deter
formance 181 categories to determine key quality indicators. mination of for example, IPVideo impairment analysis. Gen
As previously noted, key quality indicators include, for erally speaking, the methodology, apparatus and system of
example, image element loss, and jerkiness, blockiness and/ the present invention provide the following capabilities: (1)
or audio impairment. 30 An accurate representation of video quality using a set of
FIG.3 depicts a high-level block diagram depicting inter scores; (2) A network error analysis using distinctive charac
relationships of several functional elements according to an teristics of multiple video quality indicators, assisting net
embodiment of the present invention. In the embodiment of work operators to distinguish between transport and encoding
FIG. 3, the depicted functional elements comprise those impairment VS. impairment from other causes; (3) A determi
described above with respect to the impairment assessor 160 35 nation of the required minimal set of network and video
of FIG.1. The apparatus described herein with respect to FIG. attributes that impact digital video quality, and (4) A cost
3 may be implemented in hardware, software or a combina effective technique to provide high Volume Subjective scoring
tion of hardware and software, such as described elsewhere in of video quality impairments.
this specification. Various embodiments of the above-described invention
The VO-calibrator VOC performs a core reference coordi 40 provide a comprehensive set of metrics that can be used for
nation function between the network and content profiles and both quality measurement and root cause trouble isolation; a
Subjective measurements, creating a VOA calibration data set combinational model that includes unique KQI elements,
for use by the VO-analyzer VOA. The VOC enables synchro based on both theoretical and lab verified results; and a set of
nization of objective measurements of a key performance KQI metrics that are adaptable to differing configurations of
indicators (KPI) set with the impairment scoring of the 45 network core, network access, head end, and customer
human Subjects. In one embodiment, the VOC generates a premise equipment.
calibration data set using subjective VOI references that are Optionally, a separate quality analysis phase is provided in
empirically derived. In another embodiment, the VOC gen which the output from the analysis process is used in a col
erates a calibration data set according to human factor calcu laborative manner with video service providers, network
lations. Such human factor calculations may be based upon, 50 managers, management consultants and/or network probe
for example, the human factor observations associated with vendors to provide, for example, cost effective solutions to IP
the various MPEG standards and their predecessor standards Video quality assurance and customer satisfaction problems.
documents (e.g., Grand Alliance and the like). In one embodiment of the invention, a subjective determi
The VO-analyzer VOA uses a vector of real-time data nation of digital video impairments is provided by executing
values and the VOA calibration data set to determine a rel 55 Subjective tests and calibrating the results, combining this
evant video quality Score set. In one embodiment, a continu calibration with real-time network information, set top box
ous transfer function produces this VQA score vector by (STB) information and content encoding/transport param
interpolating between discrete values sampled in real-time eters. The output result is a set of values for Video Quality
and discrete values in the Video Quality calibration data set. Indicators that provide service quality information about the
The resulting output is displayed and/or passed to monitoring 60 performance of the IP Video Service as it relates to presenta
and management applications. tion quality for content transported there through.
The real-time measurement normalization module NM is The above-described embodiments of the invention may be
based on the development of a set of, illustratively, 16 KPIs implemented within the context of methods, computer read
that have the most significant impact on IP-Video impair able media and computer program processes. Generally
ments (e.g., the primary KPIs discussed herein). Generally 65 speaking, methods according to the invention may be imple
speaking, the impact of any one of the 16 primary KPI ele mented using computing devices having a processor as well
ments on IP-Video image can be determined by those skilled as memory for storing various control programs, other pro
 US 8,027,264 B2
9 10
grams and data. The memory may also store an operating 5. The computer readable medium of claim 4, wherein
system Supporting the programs. The processor cooperates KPIs associated with the content and service profile category
with conventional Support circuitry Such as power Supplies, comprise one or more of frame size, coding protocol, motion
clock circuits, cache memory and the like as well as circuits and coding bit rate.
that assist in executing the Software routines stored in the 6. The computer readable medium of claim 1, wherein said
memory. As such, it is contemplated that some of the steps quality-related characteristics comprise key performance
discussed herein as Software processes may be implemented indicators (KPIs) associated with one or more of a network
within hardware, for example as circuitry that cooperates performance category, a network configuration category and
with the processor to perform various steps. Input/output a content and service profile category.
(I/O) circuitry forms an interface between the various func 10
7. The computer readable medium of claim 6, wherein
tional elements communicating with the device. KPIs associated with the network performance category com
A computing device is contemplated as, illustratively, a prise one or more of packet loss, jitter, packet reordering and
general purpose computer that is programmed to perform bit error rate.
various control functions in accordance with the present
invention, the invention can be implemented in hardware as, 15 8. The computer readable medium of claim 6, wherein
for example, an application specific integrated circuit (ASIC) KPIs associated with the network configuration category
or field programmable gate array (FPGA). As such, the pro comprise one or more of decoder buffer size, decoder buffer
cess steps described herein are intended to be broadly inter type, and transport protocol used.
preted as being equivalently performed by Software, hard 9. The computer readable medium of claim 1, wherein:
ware or a combination thereof. said communications channel comprises an internet proto
The invention may also be implemented as a computer col (IP) network;
program product wherein computer instructions, when pro said content stream comprises a video stream;
cessed by a computer, adapt the operation of the computer said quality-related characteristics comprise key perfor
Such that the methods and/or techniques of the present inven mance indicators (KPIs) associated the performance of
tion are invoked or otherwise provided. Instructions for 25 the IP network; and
invoking the inventive methods may be stored in fixed or said step of estimating comprises mapping said determined
removable media, transmitted via a data stream in a signal KPIs to one or more key quality indicators (KQIs).
bearing medium such as a broadcast medium, and/or stored 10. The computer readable medium of claim 9, wherein:
within a working memory within a computing device operat said KQIS are associated with picture quality attributes,
ing according to the instructions. 30
including one or more of image element loss, image
While the foregoing is directed to various embodiments of jerkiness, image freezing and image blockiness.
the present invention, other and further embodiments of the 11. The computer readable medium of claim 10, wherein:
invention may be devised without departing from the basic said content stream further comprises an audio stream; and
Scope thereof. As such, the appropriate scope of the invention
is to be determined according to the claims, which follow. 35 said KQIS are also associated with audio quality attributes
and audiovisual synchronization attributes.
What is claimed is: 12. The computer readable medium of claim 1, wherein
1. A non-transitory computer readable storage medium said impairment level estimates are adapted for use by an
storing instructions which, when processed by a computer, impairment data consumer.
adapt the operation of the computer to perform a method for 40 13. The computer readable medium of claim 12, wherein
assessing quality of a content stream propagated through a said impairment data consumer comprises one or more of a
communications system, the method comprising: service provider, network manager, provisioning decision
determining a plurality of key-performance indicators device and an operations Support system.
(KPIs) associated with the performance of a communi 14. The computer readable medium of claim 12, wherein
cations channel adapted to carry the content stream; 45 said impairment data enables said impairment data consumer
estimating, based on the determined plurality of KPIs, an to adapt communication channel allocations for one or more
impairment level of one or more quality indicators COntent StreamS.
(KQIs) associated with the presentation quality of the 15. A non-transitory computer readable medium for stor
content stream by executing Subjective tests, calibrating ing computer instructions which, when processed by a com
the results and combining the calibration with deter 50
puter, adapt the operation of the computer to perform a
mined KPIs; and method for analyzing quality of a content stream propagated
determining whether the estimated impairment level of the through a communications system, the method comprising:
one or more KQIs exceeds a threshold level of impair determining a plurality of key-performance indicators
ment.
2. The computer readable medium of claim 1, wherein: 55 (KPIs) associated with the performance of a communi
said communications channel transports said content cations channel adapted to carry the content stream;
stream through a plurality of networks. estimating, based on the determined plurality of KPIs, an
3. The computer readable medium of claim 2, wherein said impairment level of one or more key quality indicators
plurality of networks includes one or more of a core network, (KQIS) associated with presentation quality of the con
an aggregation network, an access network and a home net 60 tent stream by executing Subjective tests, calibrating the
work. results and combining the calibration with determined
4. The computer readable medium of claim 1, wherein: KPIs:
said content stream includes a video stream; and comparing the estimated impairment level of each of the
said quality-related characteristics are dependent on the one or more KQIs to a respective threshold level of
properties of at least one of media type, number of coded 65 impairment to generate impairment data; and
programs, coding protocol, available bandwidth, and transmitting the impairment data toward impairment data
access network transmission technology. COSU.S.
 US 8,027,264 B2
11 12
16. The computer readable medium of claim 15, wherein: 18. The apparatus of claim 17, further comprising:
said communications channel transports said content a video quality calibrator, for generating the calibration
stream through a plurality of networks; and data set in response to subjective assessments of imagery
said quality-related characteristics include at least one of a for each of a plurality KPIs.
packet loss characteristic, a jitter characteristic, a packet 19. The apparatus of claim 17, wherein:
said KPIs are associated with a communications channel
reordering characteristic and a bit-error rate character within an internet protocol (IP) network; and
istic.
said KPIs are associated with a content stream adapted for
17. Apparatus for analyzing quality of a content stream transport via said communications channel.
propagated through a communications system, the apparatus 10 20. An apparatus for analyzing quality of a content stream
comprising: propagated through a communications system, the apparatus
a video quality analyzer (VQA) configured to: comprising:
generate a video quality Score set by analyzing a plural means for determining a plurality of key-performance indi
ity of measured key performance indicators (KPIs), cators (KPIs) associated with the performance of a com
associated with the performance of a communications 15
munications channel adapted to carry the content
channel adapted to carry the content stream, with Stream;
respect to a calibration data set; and means for estimating, based on the determined plurality
estimate, using the plurality of KPIs, an impairment KPIs, an impairment level of one or more quality indi
level of one or more key quality indicators (KQIs) cators (KQIs) associated with presentation quality of the
associated with presentation quality of the content content stream by executing Subjective tests, calibrating
stream by executing Subjective tests, calibrating the the results and combining the calibration with deter
results and combining the calibration with deter mined KPIs; and
mined KPIs; and means for determining whether the estimated impairment
the calibration data set representing quantitative video level of the one or more KQIs exceeds a threshold level
of impairment.
assessments associated with each of the plurality of 25
KPIS.