IEEE/ACM TRANSACTIONS ON NETWORKING, VOL. 14, NO.

3, JUNE 2006 657

Dynamic Inter-SLA Resource Sharing in

Path-Oriented Differentiated Services Networks
Yu Cheng, Member, IEEE, and Weihua Zhuang, Senior Member, IEEE

Abstract—This paper proposes novel resource sharing schemes resource management as long as its SLAs with neighboring
for differentiated services (DiffServ) networks, to achieve both domains are met. At the present time, static inter-domain SLAs
high resource utilization and quality of service (QoS) guarantee. are mainly used, which are negotiated based on an estimation
Service level agreements (SLAs) are negotiated at network bound-
aries and supported by path-oriented resource mapping within the of the average traffic volume from the up-stream domain, i.e.,
network. The recently proposed SLA management scheme based the engineered traffic load. In reality, when the actual traffic
on virtual partitioning (Bouillet et al., 2002) allows overloaded load deviates from the estimation, resources will be utilized
SLAs to exploit the spare capacity of underloaded SLAs for effi- inefficiently. The actual traffic load may be more or less than
cient resource utilization, however, at the the cost of possible SLA the engineered load. We use the terms “overloaded” and “un-
violation of the underloaders. In the bandwidth borrowing scheme
proposed here, the dedicated bandwidth for underloaded SLAs derloaded”, respectively, to indicate the loading status of an
is determined and adaptively adjusted at network boundaries SLA.
according to the actual traffic load and QoS policies; the available DiffServ itself only defines per-hop behaviors (PHBs) at core
spare capacity is then properly distributed to related links for routers to coarsely differentiate QoS, and traffic conditioning
lending to others. On the other hand, the traffic flows admitted schemes at network boundaries to limit the traffic volume
with borrowed bandwidth are tagged and may be preempted
later when the original bandwidth owner needs to claim back flowing into the network. It is widely agreed that the basic
the resources. Through a detailed implementation design and DiffServ architecture should be augmented with intelligent
extensive computer simulation results we show that, by bandwidth traffic engineering functions [5], [6] to facilitate accurate in-
borrowing, both SLA compliance and high resource utilization ternal resource mapping, explicit per-flow admission control
can be achieved in various load conditions, with some side benefits to guarantee QoS [7]–[9], and dynamic resource allocation to
such as call-level service differentiation, small admission overhead,
and convenience for policy-based management. In addition, we handle the traffic load variation.
propose a distributed bandwidth pushing scheme that can dynam- The multiprotocol label switching (MPLS) technique [10],
ically adjust the spare bandwidth distribution over the network. [11] provides an efficient traffic engineering tool for Internet
Combining bandwidth pushing with bandwidth borrowing, the Protocol (IP) networks, by which multiple bandwidth guar-
resource utilization can be further improved. anteed label-switched paths (LSPs) can be explicitly set up
Index Terms—Bandwidth borrowing, DiffServ, inter-SLA re- between each ingress/egress node pair to balance the traffic
source sharing, resource allocation, service level agreement. distribution over the network. With pre-established LSPs and
per-class per-ingress/egress pair SLA resource commitments,
the optimal (from the perspective of network revenue) internal
I. INTRODUCTION resource mapping can be achieved through a properly designed
network planning or dimensioning procedure [12], [13]. In
T HE differentiated services (DiffServ) model [1] has been
proposed as a scalable class-based traffic management
mechanism to ensure Internet quality of service (QoS). In
this paper, we consider a path-oriented DiffServ domain. The
bandwidth broker memorizes the network topology and the
DiffServ networks, resource allocation (mainly bandwidth network resource planning results. Such information is used to
allocation) is based on service level agreements (SLAs) and support explicit per-flow admission control. Each time when
centrally controlled by a bandwidth broker [2]–[4]. Neighboring the bandwidth broker receives a new request forwarded from
administrative domains make long-term bilateral SLAs on the a certain ingress router, it will search for an LSP to admit the
new flow according to the routing algorithm and the stored
allocation of resources to different classes of traffic aggregate
crossing the domain boundaries. Each domain is allowed to network status information. The admission decision will then
freely choose whatever mechanism it deems proper for internal be delivered back to the corresponding ingress router. If ac-
cepted, the flow related information is stored at the ingress
router. It is generally agreed that the edge routers of a DiffServ
Manuscript received November 5, 2003; revised February 17, 2005; approved
by IEEE/ACM TRANSACTIONS ON NETWORKING Editor Z.-L. Zhang. This work
network are capable enough to keep per-flow information [8],
was supported by a research grant from the Bell University Labs at the Univer- [14]. Trimintzios et al. present a resource management archi-
sity of Waterloo. tecture for MPLS DiffServ networks in [13]. They propose
Y. Cheng was with the Department of Electrical and Computer Engineering,
University of Waterloo, Waterloo, ON N2L 3G1, Canada. He is now with the
solutions for operating networks in an optimal fashion through
Department of Electrical and Computer Engineering, University of Toronto, off-line planning and dimensioning, and subsequently through
Toronto, ON M4S 3G4, Canada (e-mail: y.cheng@utoronto.ca). dynamic operations and management functions (“first plan,
W. Zhuang is with the Department of Electrical and Computer Engi- then take care”). This paper focuses on the dynamic resource
neering, University of Waterloo, Waterloo, ON N2L 3G1, Canada (e-mail:
wzhuang@uwaterloo.ca). allocation among SLAs sharing a properly dimensioned Diff-
Digital Object Identifier 10.1109/TNET.2006.876199 Serv domain.
1063-6692/$20.00 © 2006 IEEE
658 IEEE/ACM TRANSACTIONS ON NETWORKING, VOL. 14, NO. 3, JUNE 2006

Dynamic resource allocation can be implemented on different ments during the underloaded periods, and the available spare
time scales. On the largest time scale, dynamic SLA techniques capacity is then properly distributed to related links to be bor-
[3], [15] are proposed to adjust long-term bandwidth allocation rowed by others. On the other hand, traffic flows admitted with
according to the actual traffic load measured over days or weeks. borrowed bandwidth are tagged and may be preempted later
On the smallest time scale, scheduling algorithms [16], [17] and when the original bandwidth owner needs to claim back the
measurement-based admission control [18], [19] are extensively resource.
investigated to utilize the statistical multiplexing gain at the The methodology adopted in the bandwidth borrowing is
packet-level. In the middle ground, the virtual partitioning (VP) that “boundary resource commitment determines link resource
technique [20], [21] has been used in circuit-switched and ATM sharing,” which is consistent with the SLA based management
networks to exploit the statistical multiplexing gain on the time principle; the passive SLA monitoring approach taken in [12]
scale of call duration. The recent work on effective bandwidth in with a predetermined link sharing configuration, however,
a priority queueing system [22] and our study on effective band- addresses a resource mapping problem with the inverse QoS
width in a partitioned buffer [9] extend the linear form connec- analysis approach. In this paper, we show that the newly
tion/call admission control (CAC)1 to DiffServ networks, which proposed approach allows dynamic resource sharing, QoS
makes the call-level QoS (call blocking probability) control pos- guarantee, policy based management to be achieved simultane-
sible and the VP applicable in DiffServ networks for call-level ously with a simple resource management architecture.
dynamic resource allocation [12], [21]. With VP, the free ca- Through a detailed implementation design and exten-
pacity from underloaded classes can be used by the overloaded sive computer simulation results, we demonstrate that with
classes to improve resource utilization, and the trunk reservation bandwidth borrowing, 1) SLA compliance (namely, the QoS
mechanism [23], [24] is used to force the overloaded classes to guarantee) and high resource utilization can always be achieved
back off when an underloaded class needs to claim its allocated with various link resource sharing schemes (for example
share of the capacity. VP, complete sharing (CS),2 or the scheme proposed in this
In [12], Bouillet, Mitra, and Ramakrishnan propose an SLA paper) and in various load conditions; 2) per-hop signalling is
management architecture based on VP at each link for efficient avoided for a small CAC overhead; 3) policy-based resource
resource utilization. The cost of VP is that the QoS of the management [13], [25] can be conveniently supported; and
underloaded SLAs can not be guaranteed. SLA violation for 4) a call-level service differentiation is achieved. Moreover,
underloaders is a serious problem, which could encourage performance of the bandwidth borrowing scheme is further
malicious overloading. Therefore, a penalty payment from strengthened by a bandwidth pushing technique, which can
the service provider to the customer is used in [12] to com- dynamically adjust the distribution of the spare bandwidth over
pensate the possible QoS or SLA violations. However, the the network. As the links (where the bandwidth borrowing
penalty scheme is not a completely satisfying solution from happens) and the resource sharing level on a certain link always
the customers’ perspective. Customers would always prefer to dynamically change with the SLA traffic load variations, an
have guaranteed QoS as well as a fair billing system. To our optimal distribution of spare capacity can result in maximum
best knowledge, currently there is no such resource allocation resource utilization. It is very difficult, if not impossible, to
technique available that can achieve a resource utilization derive an online, centralized optimal distribution algorithm.
close to VP while guaranteeing the QoS of all SLAs involved Therefore, the bandwidth pushing uses a distributed algorithm
in the resource sharing. In this paper, we propose a dynamic to adaptively push the spare bandwidth to the paths where the
inter-SLA resource sharing scheme, also termed as a bandwidth bandwidth borrowing can be successfully executed or where
borrowing scheme, for the above objective. more capacity is required. The efficiency of bandwidth pushing
In the bandwidth borrowing scheme, an SLA is negotiated to further improve resource utilization is also demonstrated via
for each traffic class between each ingress/egress pair. Each computer simulations.
traffic flow is allocated an effective bandwidth which encap- The remainder of this paper is organized as follows. Section II
sulates various packet level issues, such as burstiness and QoS describes the path-oriented DiffServ domain. Section III gives
(delay, jitter, loss) at network elements. The SLA capacity, at details of the SLA that will be provisioned in a DiffServ domain
call level as the maximum number of calls that can be served si- deploying dynamic inter-SLA resource sharing. Section IV
multaneously, is then properly contracted to satisfy customers’ presents the proposed bandwidth borrowing scheme. Section V
call level QoS requirements at an engineered call arrival rate. and Section VI discuss the implementation design, where
The accepted traffic is supported by parallel bandwidth guaran- the proposed data structure and routing/CAC algorithm (with
teed LSPs between each ingress/egress pair. Such path-oriented bandwidth pushing) for bandwidth borrowing are presented,
internal resource mapping is achieved by the network dimen- respectively. Section VII describes some case studies and
sioning module. During operation, if the traffic monitor finds presents computer simulation results. Section VIII gives con-
that an SLA is in the underload status, a protection bandwidth cluding remarks.
smaller than its nominal capacity is calculated according to the 2Two classic schemes for resource sharing are complete sharing (CS), which
QoS policy defined in the SLA. The protection bandwidth is allows all customers to share the available resources indiscriminately, and com-
guaranteed for the underloaders to satisfy their QoS require- plete partitioning (CP), which statically divides the resources among the cus-
tomers. CP can guarantee the resource commitment for each customer, but may
1The term call is often used in the telephone networks and ATM networks. underutilize the resources. On the other hand, CS leads to higher resource uti-
In this paper, we still use this term for convenience. The terms call, connection lization and statistical multiplexing gain, but the traffic from one customer may
and traffic flow are used interchangeably. overwhelm all the others.
CHENG AND ZHUANG: DYNAMIC INTER-SLA RESOURCE SHARING IN PATH-ORIENTED DIFFERENTIATED SERVICES NETWORKS 659

equivalently considered in terms of the acceptable number of

flows. In practice, the identical bandwidth requirement assump-
tion may not be the case, but it can be validated by the fact that
the service class can be defined with a finer granularity consid-
ering both the QoS requirements and the flow bandwidth allo-
cation. The identical bandwidth assumption is also adopted in
[12] to study the SLA management. Effective bandwidths asso-
ciated with different classes are generally different.

III. SERVICE LEVEL AGREEMENT: CALL-LEVEL

DIFFERENTIATION
This section gives details of the SLA, which is based on a
call-level differentiation concept. It is realized that, in practice,
Fig. 1. Path-oriented DiffServ domain. an SLA may be negotiated according to very general QoS re-
quirements and policy rules. For the sake of concreteness, we
consider an SLA to provision the call-level QoS defined as fol-
II. PATH-ORIENTED DIFFSERV DOMAIN lows. The call-level differentiation and bandwidth borrowing
We consider a path-oriented DiffServ domain, as shown concept presented in the SLA definition can be applied to a very
in Fig. 1, where per-class per-ingress/egress pair SLAs are general scenario where a bandwidth requirement can be deter-
negotiated at network boundaries. SLAs for the standard pre- mined based on QoS, traffic load and management policies.
mium service and assured service, supported by the expedited • A nominal capacity is allocated to the SLA according to
forwarding (EF) PHB [26] and the assured forwarding (AF) the engineered call arrival rate, also termed as the specified
PHB [27] respectively, are considered as an illustration. The rate and denoted by , to satisfy the target call blocking
bandwidth broker communicates with all the edge routers in probability (CBP), denoted by . Let be the function
the domain for resource allocation, CAC, and network config- characterizing the statistical relation of CBP versus the call
uration. We assume there exists an off-line routing algorithm arrival rate and SLA capacity. Then is determined by
which sets up several parallel paths for each ingress/egress pair. .
These paths are fixed by MPLS and referred to as virtual paths • During operation, at the ingress router, a call-level traffic
(VPTHs). All traffic traversing an ingress/egress pair is dis- monitor measures the actual call arrival rate for the SLA,
tributed among the VPTHs. VPTHs for different ingress/egress denoted as . Two resource utilization states are defined
pairs may share some common links (VPTH multiplexing). for the SLA, which are lendable state if ,
An MPLS traffic trunk is defined as a logic pipeline within a and unlendable state otherwise. is the CBP requirement
VPTH, which is allocated a certain amount of capacity to serve specified for the underloaded case where the SLA is pos-
a class of traffic. Therefore, a VPTH between an ingress/egress sible to lend out bandwidth, and to provision
pair may include multiple traffic trunks for different service better QoS in the underloaded period.
classes. • In the lendable state, a protection bandwidth is
In the path-oriented environment, boundary SLA resource calculated according to . The capacity of
commitments are mapped to bandwidth allocation at each traffic is reserved for the SLA. The spare bandwidth, ,
trunk by network dimensioning. At each router, the total band- can be exploited by related SLAs, including both lendable
width allocation for a PHB is then derived by summing the band- and unlendable ones, in a complete sharing manner.
width allocation of all the same-class trunks crossing that router. • In the unlendable state, the nominal capacity is guaran-
With feasible bandwidth allocation for each PHB, the specific teed. The SLA may accept overloaded traffic, by borrowing
scheduling algorithm can be designed correspondingly to guar- bandwidth from the lendable SLAs. The traffic flows ac-
antee the resource allocation and packet level QoS requirements. cepted with the borrowed bandwidth are tagged as out pro-
A linear programming procedure can be used for network di- file calls, and the flows accepted with the nominal capacity
mensioning with the objective to maximize the network revenue are considered as in profile calls.
[28], subject to the constraints: 1) the total bandwidth allocated • When the traffic monitor detects that the SLA changes back
to traffic trunks associated with an SLA should not be less than to the unlendable state from the lendable state, the protec-
the SLA resource commitment; and 2) the total bandwidth allo- tion bandwidth is then increased to the nominal capacity
cation at a link does not exceed the physical link capacity. With to claim back resources of the SLA. Some tagged traffic
per-flow CAC, the bandwidth broker puts the newly admitted flows from the borrower trunks may be preempted during
traffic flow into one of the parallel traffic trunks according to the bandwidth claiming.
the routing algorithm. All packets of one traffic flow follow the In the above SLA definition, the possible preemption of the
same VPTH. out profile calls is considered as the QoS differentiation between
In the proposed scheme, call level QoS control is decoupled the in traffic and the out traffic (The in profile calls cannot be pre-
from the packet level QoS control by using the effective band- empted). The counterpart differentiation scheme at the packet
width technique [9], [12], [22]. We assume that each traffic flow level is the AF PHB. The differentiation between in and out
of the same service class has the same effective bandwidth for traffic efficiently utilizes the spare capacity as well as avoids
simplicity. The resource allocation to a service class can be the malicious overloading. The preemption scheme works well
660 IEEE/ACM TRANSACTIONS ON NETWORKING, VOL. 14, NO. 3, JUNE 2006

for the data traffic, where a preempted data transfer may be re- first distributed to each traffic trunk evenly. Let denote the
sumed in the future as a new call. The bandwidth adaption pro- number of trunks, and the protection bandwidth calculated
posed in [29] for multi-layer coded audio/video sources can also for SLA . The protection bandwidth distributed to each
be used to alleviate preemption, where the extracted bandwidth trunk is . The even distribution may not be
from QoS degradation can be returned to the original owner or the best solution, because the traffic loads and resource sharing
used to accept new calls. levels on different routes, and therefore on different links, are
While the SLA definition focuses on resource utilization ef- different. Ideally, the protection bandwidth should be distributed
ficiency without QoS violation, the fairness issue is not fully in such a way that leads to the maximum resource utilization.
considered. As we can see, during operations the unlendable This problem will be addressed in Section VI when we discuss
SLAs can occupy more bandwidth than what they buy; the SLA CAC.
with a higher call arrival rate may grab more spare bandwidth The function for CBP calculation in general depends on
according to CS than those SLAs with a lower rate. The fair- the statistical characteristics of the call arrival process and the
ness issue may be alleviated through a properly designed billing call holding time. The classic call-level modeling in telecommu-
system, where the payment is proportional to the actual resource nication networks is the Poisson call arrival process with a mean
usage. To further extend the bandwidth borrowing scheme pro- arrival rate and an exponentially distributed call holding time
posed in this paper for fair resource allocation is an interesting with mean call duration . The function is then the well
future research topic. known Erlang-B formula. As in the Erlang-B formula does
Note that in the SLA definition, for accuracy the lendable and not have a close-form inverse function, can be determined ac-
unlendable are used to differentiate the SLA load status instead cording to (1) by an iterative search:
of the common terms underloaded and overloaded. The rela-
tionship among the four states is: the overloaded SLA is natu- (2)
rally in the unlendable state; the underloaded SLA may be lend-
able or unlendable depending on how much capacity needs to where is the set of non-negative integers. The value is cal-
be reserved to guarantee the CBP of . In the remainder of culated in terms of the call number. However, in IP networks,
this paper, both pairs of the states are to be used, and the mean- exponentially distributed inter-arrival time or call holding time
ings should be clear from the context. A normally loaded SLA is unlikely the case [30], [31]. It is quite possible that a close-
is in the unlendable state according to the definition. form expression of CBP is not available, and therefore cannot
be determined analytically. In this situation, the relationship
IV. BANDWIDTH BORROWING SCHEME can be obtained by off-line computer simulations or
measurements from the historical traffic data, and recorded in
First, we define some terminologies and symbols. When
a table. In online operation for bandwidth borrowing, corre-
bandwidth borrowing happens, the related unlendable and
sponding to a certain call arrival rate is determined by looking
lendable SLAs are termed as borrower SLAs and lender SLAs,
up the pre-established table. In fact, implementation of the band-
respectively. All traffic trunks of a lendable SLA are termed
width borrowing is independent of the CBP calculation details.
as lender trunks. A traffic trunk belonging to a borrower SLA
In the following discussion, we assume a proper approach avail-
is termed as a borrower trunk only when the trunk runs out of
able to determine the protection bandwidth for an underloaded
its nominal capacity and borrows bandwidth to service traffic
SLA.
flows. A spare route (path) is a route (path) along which a flow
The lent-out spare bandwidth is shared by trunks associated
can be successfully accepted by bandwidth borrowing. Let
with different SLAs that can access it according to the CS
denote a service class, an ingress/egress pair, and a route. In
scheme. As the protection bandwidth is enough to guarantee
the path-oriented DiffServ domain presented in Section II, we
the lendable SLA a CBP of , we can limit a lender trunk’s
use to identify an SLA, a traffic trunk, and
access to the spare bandwidth so that more bandwidth can be
the route set or trunk set of an SLA. The nominal capacity of
exploited by borrowers. At the moment that a lender trunk
SLA is denoted by , and the bandwidth allocated to
successfully captures some spare bandwidth to accept a new
traffic trunk (determined by the network dimensioning)
flow, the flow will be admitted into the network with an access
is denoted by , with .
probability, denoted by . By properly choosing the values
A. Spare Bandwidth: Calculation and Distribution of and , the service provider can control the tradeoff
between statistical multiplexing gain and QoS for lender SLAs.
For an underloaded SLA , the protection bandwidth For example, the configuration of and lead to
is calculated by solving high statistical multiplexing gain according to CS; and
bring lenders a CBP much smaller than ;
(1)
and correspond to allocating all the spare capacity to
With the clear context in this subsection, we omit the subscrip- borrowers.
tion of in related expressions for convenience. If ,
B. Trunk Resource Sharing at a Link
the SLA is determined to be in the lendable state, and the spare
bandwidth can be lent out. Otherwise, the SLA cannot In the bandwidth borrowing, the dynamic resource sharing
lend bandwidth to others. For an unlendable SLA, set is implemented at the trunk level. When a class traffic flow
to guarantee the nominal capacity. For a lendable SLA, the arrives, a trunk between the ingress/egress pair is selected
protection bandwidth, correspondingly the spare bandwidth, is according to a routing algorithm. The traffic trunk first tries to
CHENG AND ZHUANG: DYNAMIC INTER-SLA RESOURCE SHARING IN PATH-ORIENTED DIFFERENTIATED SERVICES NETWORKS 661

admit the traffic flow using its nominal capacity or protection flow in this link, the bandwidth broker steps forward to the
bandwidth depending on the SLA state; if such bandwidth is second hop link. If the trunkshare rejects the flow at any link
used up, the traffic trunk then tries to grab the spare capacity along a VPTH, this VPTH is denied. After all the links along all
from the lender trunks according to the CS scheme; otherwise, VPTHs are checked, we can get a lendable route set. All links
the traffic flow is rejected. Let denote the current bandwidth along a lendable route can lend bandwidth to the borrower
usage of trunk and the protection bandwidth. A route trunk . The leftover bandwidth along the lendable route
or a trunk passing link is represented as or .A is , where the spe-
class- flow with effective bandwidth can be accepted at link cific value of depends on whether BS, VP or CS is used
by exploiting the spare capacity, if for resource sharing. From the lendable route set, the route with
the largest lendable bandwidth (the random selection principle
will be used when multiple such routes exist) will be selected to
(3) hold the new traffic flow. Such an approach is taken to protect
where is the link capacity and guaran- the out profile call from future preemption as much as possible.
tees that the capacity of is dedicated to trunk so In the case that all the routes do not have enough bandwidth to
that bandwidth borrowing happens without SLA violation. The admit the flow, the bandwidth broker rejects the flow. The above
right-hand-side of (3) implies that the spare capacity is con- admission procedure is also applied to an lendable SLA when
tributed by all the lender trunks passing link . For convenience its protection bandwidth is used up, where the admitted flow is
of expression, the algorithm for trunk resource sharing at a link still in profile as long as .
is referred to as the trunkshare algorithm. It is noteworthy that the hop by hop checking of resource
In fact, the borrowing sharing (BS) scheme given in (3), VP availability here is not through signaling, but through looking up
and CS3 are all special cases of the general trunk reservation route table and resource usage information stored in the band-
scheme width broker (to be explained in the next section). Hence, there
is no scalability problem and the CAC overhead time is ex-
(4) pected to be small. The trunkshare checking at each link and
the route selecting procedure are summarized as a sparest
route subroutine process to be used in the CAC procedure,
where the reservation parameter , when , which returns the selected route , or when no lendable route
is for BS, exists.
for VP [12], and 0 for CS; when ,
in all the three cases. Among the three schemes, V. DATA STRUCTURE
BS is normally the most conservative scheme with the largest
trunk reservation4 for QoS guarantee; CS is a greedy scheme A. Data in the Bandwidth Broker
for high resource utilization, while the underloaders may be For bandwidth borrowing, all the information stored in the
overwhelmed by overloaders; VP is a trade-off approach where bandwidth broker is organized into three tables: Route Table,
the overwhelming can be alleviated but not eliminated. With Trunk Status Table, and SLA Status Table.
the call preemption approach as detailed in Section VI-A, the Route Table: The topology and routing information is orga-
dilemma between high resource utilization and QoS guarantee nized into a route table. Assume that each traffic trunk and each
can be successfully solved; VP and CS can also be used in the link is assigned a unique ID in the DiffServ domain. Each row
trunkshare algorithm for higher resource utilization. Any and each column of the route table is indexed with the traffic
aggressive resource usage from the borrowers, which invades trunk ID and the link ID, respectively. Searching along a row,
the protection bandwidth reservations of other SLAs, will then we can find all the links of a traffic trunk. Searching along a
be preempted by the original owners when necessary, so that column, we can get all the trunks crossing a certain link.
the aggressiveness does not lead to SLA violation. Trunk Status Table: The network planning results, current
network resource usages, and bandwidth borrowing information
C. Bandwidth Borrowing Along a Path are organized into the Traffic Trunk Status Table. Each record in
An out profile flow can be accepted only when bandwidth the table is indexed with the traffic trunk ID and has four items:
borrowing via trunkshare is successful at all links along Traffic Trunk Capacity allocated by the network planning,
the selected path. For a new out profile request associated with Traffic Trunk Usage which is updated with call arrival and
an unlendable SLA , the borrowing procedure is as fol- completion, Trunk Protection Bandwidth (For the trunks
lows. The bandwidth broker begins from the first hop link on associated with an unlendable SLA, ; for those
each VPTH . If trunkshare can admit the class with a lendable SLA, is initially set, and
3Note that in bandwidth borrowing, the CS scheme is adopted to exploit the
may be dynamically adjusted during the bandwidth borrowing),
0
spare capacity (C R) on each link. When we compare different link resource
and Trunk Utilization Status (TUS) Flag. The TUS flag indi-
sharing schemes (BS, VP, and CS), CS is then referred to as one approach for cating the current trunk utilization status. Three states are pos-
sharing the whole link capacity. sible, namely, notfull, full, and borrowing. When ,
4It is obvious that examples where max(0; TUS flag is set as notfull; when , flag as
R 0 U ) < max (e ) can be easily constructed. However, in a
full; when , flag as borrowing.
multiclass network with multiple lenders, a separate resource reservation
for each lender is normally more conservative than the single reservation of SLA Status Table: The call-level traffic monitor inside the
max(e ) used in VP. edge routers measures the call arrival rate for each SLA, and
662 IEEE/ACM TRANSACTIONS ON NETWORKING, VOL. 14, NO. 3, JUNE 2006

forwards the measurement results to the bandwidth broker when

the rate changes (according to a predefined threshold, for ex-
ample, increase or decrease by 10%). The bandwidth broker will
then update the SLA status and the protection bandwidth . The
SLA Status Table is just a simple array indexed with the SLA
ID, where each element indicates the status of an SLA, lendable
or unlendable.

B. Data in the Edge Routers

Flow Record Table: To support per-flow CAC, the ingress
routers memorize the per-flow information in the Flow Record
Table. The information is organized and saved according to the
traffic trunks where the traffic flows are placed.
Flow Information Record: A flow information record in-
cludes the Traffic Flow ID, the Effective Bandwidth allocated
to the flow, and the Flow Admission Status (FAS) Flag. The
FAS flag indicates what bandwidth is used to admit the traffic
flow, which is set as nor-admit (normally admitted) if the traffic
flow is admitted into a notfull trunk by the nominal or protection
bandwidth, and set as bor-admit (admitted with borrowed band-
width) if admitted through bandwidth borrowing. A bor-admit
flow is considered as an out profile flow.

VI. ROUTING, CAC, AND BANDWIDTH PUSHING

The flowchart of the proposed routing/CAC algorithm for a
class- traffic flow is given in Fig. 2, which has the following
procedure:
Fig. 2. Routing and admission procedure with bandwidth borrowing and band-
1) The dedicated bandwidth available in the notfull width pushing.
trunks is always used first to accept the traffic flow. The
random trunk selection is adopted for load balancing over
6) If the traffic flow is accepted, the trunk status is then up-
the network. Upon a nor-admit admission, the need of
dated by the subroutine updatetrunkstatus. Specif-
bandwidth claiming is checked. One reason is that the
ically, , and the TUS flag is updated corre-
SLA under consideration may just increase its dedicated
spondingly. Also, the flow information is added to the flow
protection bandwidth due to the increase of traffic load,
record table at the corresponding ingress router.
where the former spare capacity (but not available now)
In the above CAC procedure, several subroutines are called.
may have been borrowed by other SLAs and should be
Among them, the sparestroute subroutine is described in
claimed back if itself needs to use it. The other reason is
Section IV-C. The updatetrunkstatus ( , event) sub-
that VP or CS based trunkshare may lead to aggressive
routine is to update the status of traffic trunk for the speci-
bandwidth usage. The steps to claim back bandwidth are
fied event. The specified event may be admission of a call, com-
summarized in the subroutine claimbackband.
pletion of a call, preemption of a call, or adjustment of the trunk
2) When the dedicated bandwidth is used up, that is, no not-
protection bandwidth. An event leads to the change of or
full trunk is available, the inter-SLA sharing is evoked.
in the specified traffic trunk, and the TUS flag is then updated
The subroutine sparestroute is called to search for the
depending on the current relationship among , and of the
lendable route with the largest leftover band-
trunk. The details of the claimbackband and pushprot-
width. If no lendable route exists, the flow is rejected.
band subroutines are given in the following subsections.
3) For an unlendable SLA, the inter-SLA sharing is a bor-
rowing process. The FAS flag is set as bor-admit. A. Claiming Back Bandwidth
4) For a lendable SLA, the inter-SLA sharing is for better
QoS. Here, the lender trunk is not allowed to go to the bor- In the claimbackband subroutine, if the new request is to
rowing state for a stable resource manage- be admitted in the notfull trunk , the leftover bandwidth
ment scheme. The access probability, , is used to control is calculated as on each link that the
the extent that the lender trunk accesses the spare capacity. trunk passes.5 On any link, a leftover bandwidth less than
The FAS flag for an admitted flow is set as nor-admit. means that the link is an overused link, where borrower trunks
5) Upon each admission via the inter-SLA resource sharing, exist and trunk needs to claim back its lent-out bandwidth
the subroutine pushprotband is called to dynamically by preemption. On each overused link, the associated borrower
adjust the protection bandwidth distribution among related 5Note that the calculation of C 0 U for preemption
lender trunks, so that the spare bandwidth can concentrate checking, other than C 0 t 0 U for admission control,
on the routes where it can be more efficiently utilized. can fully exploit the statistical multiplexing to avoid unnecessary preemption.
CHENG AND ZHUANG: DYNAMIC INTER-SLA RESOURCE SHARING IN PATH-ORIENTED DIFFERENTIATED SERVICES NETWORKS 663

trunk set is searched, from which certain bor-admitted traffic are only slightly overloaded while heavily overloaded with
flows are preempted to release enough bandwidth to hold the the other one. In this case, most of the SLA spare bandwidth
new request. The pseudocode of the subroutine claimback- should be distributed to the heavily overloaded route for higher
band is as follows: resource utilization. Generally, the spare bandwidth distribution
should be properly determined and dynamically adjusted ac-
subroutine name: claimbackband cording to the network status for maximum resource utilization.
Input: ID of trunk (s; r ) holding the new flow It is very difficult, if not impossible, to derive a centralized,
Return: no return value optimal on-line distribution technique. Therefore, we propose
a distributed bandwidth pushing scheme to approximate the
linkset = all the links along route r ;
optimal distribution, which is summarized in the subroutine
for i = 1: size(linkset) %foreachlink pushprotband. The pseudocode of pushprotband is as
leftband = leftover bandwidth on the ith follows:
link;
subroutine name: pushprotband
while leftband< es %preemptionrequired
Input: ID of trunk (s; r) holding the new flow
borrowertrunkset = all the borrower
trunks passing the ith link; Return: no return value
select one borrower trunk, (sb ; rb ), from linkset = all the links along route r ;
the borrowertrunkset with a probability
that is proportional to the borrowed pushedtrunks= 0; %storealreadyprocessedtrunks
bandwidth; for i = 1: size(linkset) % ateachlink
preempt the newest bor-admitted traffic lendertrunkset = all the lender trunks
flow in (sb ; rb ); passing the ith link;
leftband = leftband + es ; for j = 1: size(lendertrunkset)
updatetrunkstatus ((sb ; rb ), flow if (lendertrunkset (j ) 62 pushedtrunks &
preemption); Rlendertrunkset(j ) > Bpush )
(sj ; j ) = SLA that lendertrunkset(j )
end
end belongs to;
from (sj ; j )’s pushable trunk set,
When bandwidth borrowing is implemented based on the randomly choose one trunk, x;
BS trunk sharing scheme, the traffic flow preemption does
not have a severe impact on the long term call-level QoS, Rlendertrunkset(j ) = Rlendertrunkset(j ) 0 Bpush ;
due to three reasons: 1) The preemption happens only during Rx = Rx + Bpush ;
the short period just after a certain lender trunk increases its
protection bandwidth. 2) When the protection bandwidth of a % updating trunk status due to new R
lender trunk increases, the admission rate of new bor-admitted
flows reduces immediately due to the decrease of borrowable updatetrunkstatus (lendertrunkset(j ), new
Rlendertrunkset(j ) );
bandwidth, which can speed up the bandwidth returning. 3) At
each link, unused bandwidth in all trunks is fully exploited to updatetrunkstatus (x, new Rx );
avoid preemption. But with VP or CS trunk sharing scheme, add lendertrunkset(j ) to pushedtrunks;
the aggressive flow admission may result in an unneglectable end
preemption probability to those out profile calls. Preemptions
end
brought by the BS, VP, and CS trunk sharing schemes are
compared in Section VII. end
Call preemption should be considered as a type of cost in
the resource sharing. It may not be a problem for non-realtime In the bandwidth pushing scheme, an adjustment of the
data application, but may be annoying in realtime video/audio protection bandwidth distribution is triggered each time a
applications. Based on the proposed CAC algorithm, a message new traffic flow is put into a full trunk , according to
can be sent to the customer before the actual service regarding the CAC procedure given in Fig. 2. To do the adjustment, in
the SLA load status and flow admission status. The customer can pushprotband the lender trunk set is searched at each link
then determine to continue or try at a later time. With customers’ along route , and the detected lender trunks are called lend-on
awareness of the status, the resource sharing with preemption is trunks for expression convenience. Each lend-on trunk will
expected to be a reasonable service model. then try to push some of the protection bandwidth to its fellow
lender trunks, referred to as push-to trunks, belonging to the
same lender SLA, so that the spare bandwidth can concentrate
B. Dynamic Bandwidth Pushing
on the trunks where bandwidth borrowing is taking place. Let
The even distribution of spare bandwidth (and protection denote the effective bandwidth associated with a lender
bandwidth correspondingly) over lender trunks is not the most SLA. The amount of the protection bandwidth to be pushed
efficient approach. For example, a lender SLA has two parallel is , to reserve more bandwidth
lender trunks, but borrowers associated with one lender trunk for future arrivals possibly with the effective bandwidth
664 IEEE/ACM TRANSACTIONS ON NETWORKING, VOL. 14, NO. 3, JUNE 2006

(which is estimated from the previous arrivals). A push-to trunk

is a pushable trunk only when conditions

(5)

(6)

are satisfied. Condition (5) indicates that the protection band-

width can never exceed the nominal allocation. Condition (6)
means that over-pushing should be avoided so as not to block the
future bandwidth borrowing along the push-to paths. In other
words, the bandwidth pushing should not change the “spare
property” of other paths. Within a lender SLA, all pushable Fig. 3. Network topology, SLAs, and trunk deployment for bandwidth
borrowing.
trunks form a pushable trunk set, and one of them is randomly
picked out to accept the pushed-in protection bandwidth. The
lend-on trunk can push away its protection bandwidth down to TABLE I
zero. SLA QOS REQUIREMENTS AND NETWORK CAPACITY PLANNING RESULTS
It should be noted that the bandwidth pushing is operated in
the whole network in a distributed manner. Each spare route
tries to push the protection bandwidth to other routes but with
different pushing force. With the proposed pushing algorithm,
those routes with a heavier traffic load generate stronger pushing
force and obtain a larger part of the spare capacity in the pushing
competition. At the same time, the over-pushing is limited by
(6) and each spare route has a chance to be the winner of the
pushing competition when the traffic load changes dynamically. capacity (it will be explicitly stated, when the call level capacity
The simulation results in Section VII demonstrate that the band- is used), and efficient bandwidth usage, call/t-unit for call ar-
width borrowing scheme can always be enhanced, independent rival rate and call level throughput, and c-unit/call for effective
of the link resource sharing schemes (BS, VP, or CS), by the bandwidth.
bandwidth pushing technique with a higher resource utilization
A. Operation and Performance of Bandwidth Borrowing
as compared with the no-pushing case.
The network for this simulation study is shown in Fig. 3. We
C. Processing for Call Completion and Preemption use three examples to illustrate various aspects of the proposed
The data processing for call completion is simple. One resource sharing schemes.
operation is to delete the flow information from the flow record Example 1: Resource Utilization Improvement: Five SLAs
table in the corresponding ingress router. Another is to update are negotiated and served with parallel traffic trunks, as shown in
the status of the trunk where the traffic flow resides. When a Fig. 3. SLA QoS requirements, engineered call arrival rates and
traffic flow completes on the trunk , network capacity planning results are given in Table I. The SLA
and TUS flag is updated accordingly by subroutine up- capacity is evenly distributed over the related traffic trunks. A
datetrunkstatus ( , completion). The processing for homogeneous case is considered, where the effective bandwidth
a call preemption is the same as that for a call completion. associated with each SLA equals to 1, and the arrivals in each
SLA have an average call holding time of 1. The link capacity
is 60 for links 1 and 2, and 40 for other links. The BS trunk
VII. SIMULATION RESULTS resource sharing scheme is used for bandwidth borrowing.
This section presents the computer simulation results from The simulation starts at , and ends at 48000. Traffic
some case studies to demonstrate the efficiency of the band- for each SLA starts with the specified call arrival rate, and the
width borrowing, dynamic bandwidth pushing, and QoS guar- call arrival rates for some SLAs are changed at certain moments
antee with preemption. For convenience of calculating the pro- to create the overloaded and underloaded periods. We assume
tection bandwidth, we assume Poisson arrivals for each SLA that the call-level traffic monitor can detect the rate variation
and exponentially distributed call holding times in all simula- timely and accurately, as the operations of the bandwidth bor-
tions. Also, the network dimensioning procedure is simplified rowing are independent of the specific measurement methods
by tailoring the link capacity to exactly hold all the traffic trunks [12], [32]. The actual call arrival rate for each SLA and the
crossing the link. This approach provides us the flexibility to corresponding protection bandwidth for each trunk are given in
configure networks. The units used for related measures are time Table II. and are set for bandwidth borrowing.
unit (t-unit) for time, capacity unit (c-unit) for link/trunk/SLA The measured call blocking probability and call throughput for
CHENG AND ZHUANG: DYNAMIC INTER-SLA RESOURCE SHARING IN PATH-ORIENTED DIFFERENTIATED SERVICES NETWORKS 665

TABLE II
CALL ARRIVAL RATE AND PROTECTION BANDWIDTH FOR EACH SLA

Fig. 4. Performance with bandwidth borrowing. (a) The call blocking proba- Fig. 5. Performance without bandwidth borrowing. (a) The call blocking prob-
bility of each SLA. (b) The call throughput of each SLA. ability of each SLA. (b) The call throughput of each SLA.

each SLA with bandwidth borrowing are presented in Fig. 4(a) 2) During the time period (12000, 36000), SLA-2 and SLA-5
and (b), respectively, and those without bandwidth borrowing become underloaded, where both their protection band-
are presented in Fig. 5(a) and (b), respectively. To estimate the widths are calculated as 23 and first evenly distributed
throughput, each 10000 inter-arrival times are averaged and then between the two parallel trunks as 11 and 12 (fractional
taken the reciprocal to get a throughput sample. To estimate capacity unit not allowed). With bandwidth borrowing,
the CBP, a slide-window algorithm is used. Each 10000 arrivals SLA-1 can utilize the spare capacity from SLA-2 and
compose a slide period, and 10 slide periods compose a window. SLA-5 along trunk-1, but not along trunk-2 and trunk-3,
The number of lost calls in one window is used to get an CBP because there is no spare bandwidth on link-6, link-11,
sample. After one CBP sample is obtained, the window then link-10 and link-5. Therefore, trunk-1 is the borrower
shifts forward by one slide. trunk, trunk-4 and trunk-10 the lend-on trunks, and trunk-5
From Table II and Figs. 4 and 5, we have the following and trunk-11 the push-to trunks. The pushprotband
observations: procedure then pushes protection bandwidth from trunk-4
1) Each SLA starts with the engineered rate and achieves to trunk-5 for SLA-2 and from trunk-10 to trunk-11 for
the target CBP, without inter-SLA resource sharing. After SLA-5, until the protection bandwidth on trunk-5 and
, SLA-1 becomes overloaded. The bandwidth trunk-11 reaches 19, with 1 unit of spare bandwidth re-
borrowing does not happen until when SLA-2 served to maintain their “spare” property. In the steady
and SLA-5 become underloaded. During the time period state, the spare capacity on both trunk-4 and trunk-10 is
of (6000, 12 000), SLA-1 has the CBP of then . The spare capacity of 16 on
, and other SLAs continue with the specified rate link-1 is shared between SLA-1 and SLA-2 according
and achieve the target QoS. to the CS principle, and on link-2 between SLA-1 and
666 IEEE/ACM TRANSACTIONS ON NETWORKING, VOL. 14, NO. 3, JUNE 2006

SLA-5. SLA-1 grabs almost all the spare capacity due to

the large call arrival rate to achieve a CBP approximately
of . The simulation results
match very well with this numerical estimations. The spare
bandwidth of 1 unit on link-9 and link-12 can only be
accessed by SLA-2 and SLA-5, respectively. In addition
to having some spare capacity grabbed from trunk-4 and
trunk-10, both SLA-2 and SLA-5 can achieve a CBP much
smaller than , which are
also demonstrated by the simulation results.
3) After , SLA-5 changes back to the unlend-
able state, and both protection bandwidths of trunk-10
and trunk-11 are then reset to 20 to claim back the
lent-out bandwidth. Since the spare capacity on link-2 is
not available anymore, the bandwidth borrowing along
trunk-1 stops. However, in the simulation, we did not
observe preemptions due to the protection provided by
the statistical multiplexing as explained in Section VI-A.
When the bandwidth borrowing stops, SLA-2 exclu-
sively utilizes the spare capacity, achieves a CBP of
.
4) The throughput simulation result of each SLA is consistent
with the call arrival rate and the achieved CBP. For the
CAC without bandwidth borrowing, each SLA always
exclusively uses its nominal capacity. The difference be-
tween the borrowing and non-borrowing scenarios exists
during (12000, 36000), where the underloaded SLA-2
and SLA-5 in the latter achieve the CBP of near 0, but
the throughput only increases approximately from 14.3
to 14.4 as compared with the former. On the other hand,
the bandwidth borrowing can trade the slightly (almost Fig. 6. Performance of dynamic bandwidth pushing. (a) The call blocking
unnoticeably) degraded QoS of underloaded SLAs for an probability of each SLA. (b) Dynamic adjustment of the protection bandwidth.
obvious throughput increase in the overloaded SLA-1,
approximately from 55.0 to 61.8. During the time period (0, 12000), we have the same observa-
Example 2: Dynamic Bandwidth Pushing: In this example, tions as those in Example 1. During (12000, 24000), bandwidth
we show that the bandwidth pushing can adaptively adjust the borrowing happens between SLA-1, SLA-2 and SLA-5, and
spare bandwidth distribution in the network when traffic load the protection bandwidth distribution is adjusted to
changes. Also, the heterogeneous effective bandwidth alloca- and by bandwidth pushing, inde-
tion and the effect of control are demonstrated. We basically pendent of the initial distribution. Along trunk-5 and trunk-11,
use the same network dimensioning and configurations as those is reserved to keep their “spare” property. Along
used in Example 1, with the following changes: trunk-1, only 14 out of the 15 spare units is accessible to SLA-1
• An SLA-6 between ingress G and egress E is added. SLA-6 due to the discreteness. Therefore, the CBP of SLA-1 can ap-
is supported by trunk 12 passing link-10 and link-5. The proximately goes back to as
engineered call arrival rate for SLA-6 is also 29, with av- shown in the figure. During this period, the lenders SLA-2 and
erage call duration of 1. The nominal capacity of trunk-12 SLA-5 have a CBP with the order of , larger than the order
is 40 calls to guarantee a CBP of 0.01. of in Example 1, due to the spare capacity access control
• Flow effective bandwidth is 2 for SLA-1 and 1 for other with . During the interval (24 000, 36000), the band-
SLAs. Link capacity is adjusted to keep the network well width borrowing stops as SLA-5 claims back its spare capacity.
dimensioned, where , , In this period, although SLA-2 can exclusively access its spare
, and 60 for the other links. capacity, its CBP does not decrease obviously, again due to the
• When SLA-2 becomes underloaded, the initial protection control.
bandwidth is set as . Interesting things happen during (36000, 48000) when
• SLA-5 changes back to the unlendable state at . SLA-6 becomes underloaded. In this period, SLA-2 and SLA-6
SLA-6 goes to underloaded at with , are lenders, and the SLA-1 out profile calls can be accepted
and . along trunk-3. Between SLA-1 and SLA-2, the borrower
• . trunk-3 needs to borrow bandwidth from both lenders trunk-4
The simulation starts at and ends at 48000. The mea- and trunk-5. It is obvious that in this case an even distribution
sured CBP for each SLA is shown in Fig. 6(a). As SLA-4 is not is the best choice of the SLA-2’s spare capacity for maximum
involved in bandwidth borrowing, it is not shown in the figure. resource utilization. In simulation, the bandwidth pushing does
CHENG AND ZHUANG: DYNAMIC INTER-SLA RESOURCE SHARING IN PATH-ORIENTED DIFFERENTIATED SERVICES NETWORKS 667

TABLE III
PERFORMANCE RELATED TO TRUNK RESOURCE SHARING, BANDWIDTH PUSHING AND FLOW PREEMPTION

adjust the distribution to . The dynamic where is the Erlang traffic load to the SLA
adjustment of is plotted in Fig. 6(b), which clearly shows . Calculation in (9) in fact gives a conservative result, be-
the two best values during the two bandwidth borrowing in- cause traffic served within an out profile call before its preemp-
tervals and the adaptive transition between them. Along the tion is not taken into account.
path composed of link (3, 9, 10, 5), link-9 is the bottleneck in In the considered configurations, when VP or CS trunk re-
bandwidth borrowing, so there is enough spare capacity from source sharing scheme is used and flow preemption is not ap-
trunk-12 to serve SLA-3’s traffic. Therefore, SLA-3 achieves plied, the routing/CAC procedure proposed in Section VI is
a very small CBP during this period. Again, the lender SLA-6 slightly adjusted. In such cases, an overwhelmed in profile call
achieves a CBP around due to the control. is rejected instead of preempting out profile calls. The sample
Example 3: Trunk Resource Sharing and Preemption: In this path (call arrival time and call holding time) is identically re-
example, the impact of the link-level trunk resource sharing produced in all the simulations. The simulated number of flows
schemes on resource utilization and efficiency of the preemp- that contribute to the statistical measures is sufficiently large to
tion scheme on QoS guarantee are investigated. In the simu- make the confidence intervals negligibly small.
lation, network dimensioning, effective bandwidth, and From Table III, the following observations can be made:
configurations are the same as those used in Example 1, and we 1) With the CP trunk resource sharing, traffic service in each
here focus on the scenario where SLA works independently. There is no bandwidth pushing
, and , or flow preemption issue in this case. The CBP is directly
are initially set. We consider different configurations that obtained from the Erlang-B formula. Obviously, CP leads
specify which trunk resource sharing scheme (CP, BS, VP or to the worst resource utilization. As we consider a fixed
CS) is used and whether bandwidth pushing or flow preemption load scenario, there is no preemption issue either when BS
is applied. scheme is used;
The CBP is measured for each configuration and compared in 2) For a certain pushing and preemption setting, the VP or CS
Table III. When the flow preemption is involved, the out profile scheme can further improve the EBU as compared with the
call preemption probability (OCPP) is measured as BS scheme. However, when flow preemption is not applied
- in the VP or CS scheme, the aggressive resource usage
(7) in the overloaded SLA-1 leads to QoS violation in all the
- -
other SLAs. The dilemma between high resource utiliza-
and a total call blocking and preemption probability (CBPP) is tion and QoS violation, never effectively solved before,
calculated by can be overcome by the flow preemption scheme. Com-
paring the results in all the paired “X-X-NoPreempt” and
- “X-X-Preempt” settings, we can observe the QoS guar-
(8) antee by preemption for the normally loaded and under-
-
loaded SLAs. Also, the CBPP of SLA-1 in the VP and
As a more straightforward metric to measure the resource uti- CS schemes is obviously smaller than that in the CP and
lization, the total efficient bandwidth usage (EBU) over the net- BS cases due to a higher statistical multiplexing gain, even
work, according to [12], is calculated by with the preemption cost being taken into account. In ad-
(9) dition, “N/A” of OCPP means no bor-admitted call and
therefore no preemption is observed in simulation;
668 IEEE/ACM TRANSACTIONS ON NETWORKING, VOL. 14, NO. 3, JUNE 2006

TABLE IV
SLAS BETWEEN EACH NODE PAIR AND ASSOCIATED SERVICE CLASS

Fig. 7. Network configuration for robustness study.

Underloaded SLAs and overloaded SLAs are created to ob-
3) In most cases, the configuration with the CS scheme serve the operations of the bandwidth borrowing and bandwidth
achieves the highest EBU, as compared with the corre- pushing. For convenience, the SLAs in Table IV are identified
sponding ones with the BS or VP scheme. An excep- as SLA-1 to SLA-32 from left up to right down, row by row.
tion happens between “VP-Pushing-NoPreempt” and The overloaded SLAs are 1, 5, 7, 11, 13, 14, 15, 16, 18, 19, 20,
“CS-Pushing-NoPreempt”. As the exact same sample path 21, 24 and 30, underloaded ones are 2, 3, 8, 10, 12, 17, 22, 23,
is used in each simulation, the above exception is not 25, 26, 27, 28, 31 and 32, and normally loaded are 4, 6, 9 and
due to a statistical error. The exception indicates that the 29. With this setting, all links have both lender trunks and bor-
greedy CS scheme sometimes does not lead to the highest rower trunks. In such an environment, an intuition may be that
resource utilization in a network. the bandwidth pushing is unnecessary as spare capacity can be
4) For any combination of preemption and trunk resource exploited on each link. However, the simulation results demon-
sharing settings, the bandwidth pushing technique can fur- strate that bandwidth pushing can still clearly increase the re-
ther improve the resource utilization, as compared with the source utilization in this case. To show the robustness of the
“NoPushing” case. performance, we run simulations with different trunk sharing
B. Robustness of the Performance schemes and under various load conditions. All the underloaders
are tuned to have , and the corresponding
The above examples comprehensively illustrate the opera- calls (for a ) which is at first evenly distributed
tion and performance of the proposed resource sharing tech- among traffic trunks. Also, is used. Three load condi-
niques. Here, we demonstrate that the bandwidth borrowing and tions are simulated by varying the call arrival rates of overloaded
bandwidth pushing techniques have robust performance of im- SLAs as , , and , respectively, where the net-
proving resource utilization, by simulating a larger scale net- work as a whole is about 22%, 67%, and 112% overloaded as
work supporting tens of SLAs. compared with the engineered traffic load, and referred to as
The network topology, shown in Fig. 7, is the same as that lightly, medially, and heavily loaded, correspondingly.
used in [12], where eight nodes are connected with 10 links. Table V presents the simulated call level throughput (CLT)
In this study, three service classes are considered. The effective and the EBU obtained from (9) when the BS, VP, or CS scheme
bandwidth associated with classes 1, 2, and 3, are 1, 6, and 24, is used as the trunk resource sharing scheme under the three load
respectively. For example, if 1 c-unit corresponds to 16 Kb/s, conditions. The preemption scheme is applied in all cases to en-
the three classes can be used to support voice, medium-rate data, force the SLA compliance. The CLT and EBU of the CP case
and video streaming services, respectively. Correspondingly the are calculated from the Erlang-B formula as the performance
mean duration of calls of the three classes are set as 1, 4, 6.67, benchmark. The sample path is identically reproduced for all
where one t-unit corresponds to 3 minutes. Thus, a video flow the simulations in a given load condition. For each run of sim-
lasts on average for 20 minutes. ulation, a sufficient number of flows are generated to make the
There are 32 SLAs installed in the network and the service confidence intervals negligibly small.
classes associated with each SLA are arbitrarily set and given in With the traffic load changes from lightly loaded to heavily
Table IV. SLAs are negotiated between each pair of the nodes. loaded, the CLT and EBU under CP increase only marginally
For simplicity, only node pairs 1/5 and 4/7 have SLAs negotiated due to the saturation in overloaded SLAs and resource waste in
for all the three classes, and a single SLA is negotiated between underloaded SLAs. In all the load scenarios, BS, VP and CS
other node pairs. Furthermore, an SLA contracts resources for consistently achieve improved resource utilization as compared
traffic in both directions. Each SLA between a node pair is sup- with CP. The bandwidth pushing also has robust performance
ported by parallel traffic trunks. For each SLA, all parallel paths to further increase resource utilization in all the cases. As men-
not longer than 4 hops are searched and form the trunk set. Al- tioned earlier, it is not obvious what factors lead to the good
together, there are 66 traffic trunks to serve the 32 SLAs. Each performance of bandwidth pushing in a basically balanced net-
SLA has an engineered call arrival rate for a target Erlang load work, where lenders and borrowers have a roughly uniform dis-
of 41.5 and a nominal capacity of 54 calls to guarantee a target tribution over the network and bandwidth borrowing happens on
CBP of 0.01. The SLA capacity is evenly distributed to related each link. Simulation results show that the bandwidth pushing
traffic trunks. Each link capacity is tailored to exactly hold all dynamically adjusts the protection/spare bandwidth distribution
the traffic trunks crossing that link to get a well dimensioned on the time scale of call inter-arrival time and efficiently utilizes
network. the call-level statistical multiplexing. The dynamic protection
CHENG AND ZHUANG: DYNAMIC INTER-SLA RESOURCE SHARING IN PATH-ORIENTED DIFFERENTIATED SERVICES NETWORKS 669

TABLE V
ROBUST PERFORMANCE OF BANDWIDTH BORROWING AND BANDWIDTH PUSHING

It should be emphasized that the robust performance of ef-

ficient resource utilization is achieved without any SLA viola-
tion. In operation, the bandwidth borrowing, bandwidth pushing
and preemption may lead to an extra overhead during the CAC.
However, with the proposed data structure, the above procedures
require only addition and table looking-up operations, which are
very convenient for computer execution. Therefore, the impact
of bandwidth borrowing on the CAC overhead time should be
acceptable.
VIII. CONCLUSION
We have proposed a bandwidth borrowing scheme for dy-
namic inter-SLA resource sharing in path-oriented DiffServ net-
Fig. 8. Dynamic bandwidth pushing on call-level time scale. works, where the spare capacity from underloaded SLAs can be
efficiently exploited without SLA violation. The basic method-
bandwidth adjustment for one of the lender traffic trunks asso- ology is “boundary resource commitment determines link re-
ciated with SLA-25, passing link-6 and link-7, is illustrated in source sharing”. In addition to QoS specifications for an engi-
Fig. 8, where the simulated scenario is “Medially loaded-VP- neered traffic load, each SLA explicitly specifies the QoS that
Pushing”. The call-level bandwidth pushing is not fully reflected should be guaranteed in an underloaded period, where a pro-
in Fig. 6(b), due to the single spare path configuration used in tection bandwidth smaller than the nominal capacity is adjusted
the example. dynamically according to the actual traffic arrival rate. The pro-
Under a certain pushing setting and load condition, CS always tection bandwidth is guaranteed for the underloaders, and the
achieves the largest CLT due to its greedy feature, but not always available spare capacity is then properly distributed to related
the largest EBU. This further strengthens the observation (3) links for lending to others.
given in Example-3 in Section VII-A. In fact, in most cases, VP A call-level service differentiation concept is proposed for
achieves the largest EBU. It is not difficult to understand that the resource sharing. Traffic flows admitted with borrowed band-
greedy approach is not the optimal admission policy from the width are tagged as out profile calls, possible to be preempted
bandwidth usage perspective, especially in the heavily loaded later when the original bandwidth owner needs to claim back
heterogeneous environment. For example, assume that at a cer- the resources. The call-level differentiation scheme provides the
tain moment a link has the leftover capacity of 24. Then a new freedom of using any link resource sharing (such as the BS, VP,
class-1 call arrives followed by a class-3 call . or CS) for maximum resource utilization without SLA violation,
According to CS, the class-1 call is admitted and the leftover as any aggressive resource usage from borrowers is then pre-
capacity changes to 23, where the following class-3 call has to empted by the original owners when necessary. Furthermore, a
be rejected, leading to low resource utilization. Our observa- distributed bandwidth pushing scheme is proposed, which fur-
tions regarding the resource sharing policies are consistent with ther exploits the call-level statistical multiplexing by adaptively
Borst and Mitra’s conclusion in [20] that the revenue (propor- adjusting the spare capacity distribution over the network. Ef-
tional to bandwidth usage) generated by VP is extremely close ficiency and robustness of the proposed resource sharing tech-
to the maximum achievable value. Moreover, the improvement niques are demonstrated through extensive simulation studies.
from the bandwidth pushing is also mainly shown in the EBU For future work, we are investigating the implementation of
rather than the CLT. In the design, the pushing procedure intends the bandwidth borrowing scheme using a fully distributed band-
to reserve the spare bandwidth for high-bandwidth flows, which width broker. For inter-SLA resource sharing, messaging be-
can be considered as a form of ungreedy resource sharing. tween the edge routers and the bandwidth broker is required for
670 IEEE/ACM TRANSACTIONS ON NETWORKING, VOL. 14, NO. 3, JUNE 2006

SLA status update and per-flow admission control. If the band- [19] J. Qiu and E. W. Knightly, “Measurement-based admission control
width broker is a physical central controller, it is then prone to with aggregate traffic envelopes,” IEEE/ACM Trans. Netw., vol. 9, no.
2, pp. 199–210, Apr. 2001.
become a congestion point in the network. Therefore, we are [20] S. C. Borst and D. Mitra, “Virtual partitioning for robust resource
trying to achieve the bandwidth broker as a logical central en- sharing: Computational techniques for heterogeneous traffic,” IEEE J.
tity, whose functionalities are physically distributed to all the Sel. Areas Commun., vol. 16, no. 5, pp. 668–678, Jun. 1998.
[21] R. Garg and H. Saran, “Fair bandwidth sharing among virtual networks:
edge routers. In addition, we plan to further study the fairness A capacity resizing approach,” in Proc. IEEE INFOCOM, 2000, vol. 1,
issue involved in the bandwidth borrowing. pp. 255–264.
[22] A. W. Berger and W. Whitt, “Effective bandwidth with priorities,”
ACKNOWLEDGMENT IEEE/ACM Trans. Netw., vol. 6, no. 4, pp. 447–460, Aug. 1998.
[23] R. J. Gibbens and F. P. Kelly, “Network programming methods for loss
The first author would like to thank Prof. A. Leon-Garcia networks,” IEEE J. Sel. Areas Commun., vol. 13, no. 7, pp. 1189–1198,
Sep. 1995.
for stimulating discussions on the computer simulations. The [24] P. B. Key, “Optimal control and trunk reservation in loss networks,”
authors are grateful to the anonymous reviewers for their con- Prob. Eng. Info. Sci., vol. 4, pp. 203–242, 1990.
structive comments and suggestions which helped to improve [25] P. Flegkas, P. Trimintzios, and G. Pavlou, “A policy-based quality of
service management system for IP DiffServ network,” IEEE Network,
the quality of this paper. vol. 16, no. 2, pp. 50–56, Mar.-Apr. 2002.
[26] V. Jacobson, K. Nichols, and K. Poduri, “An expedited forwarding
REFERENCES PHB,” Internet RFC 2598, Jun. 1999.
[1] S. Blake, D. Black, M. Carlson, E. Davies, Z. Wang, and W. Weiss, “An [27] J. Heinanen, F. Baker, W. Weiss, and J. Wroclawski, “Assured for-
warding PHB group,” Internet RFC 2597, Jun. 1999.
architecture for differentiated services,” IETF, RFC 2475, Dec. 1998.
[2] K. Nichols, V. Jacobson, and L. Zhang, “A two-bit differentiated ser- [28] D. Mitra and K. G. Ramakrishnan, “A case study of multiservice, mul-
tipriority traffic engineering design for data networks,” in Proc. IEEE
vices architecture for the Internet,” IETF, RFC 2638, Jul. 1999.
[3] A. Terzis, L. Wang, J. Ogawa, and L. Zhang, “A two-tier resource man- GLOBECOM, 1999, vol. 1B, pp. 1077–1083.
[29] Y. Cheng and W. Zhuang, “Diffserv resource allocation for fast handoff
agement model for the Internet,” in Proc. IEEE GLOBECOM, 1999,
vol. 3, pp. 1779–1791. in wireless mobile internet,” IEEE Commun. Mag., vol. 40, no. 5, pp.
130–136, May 2002.
[4] Z.-L. Zhang, Z. Duan, L. Gao, and Y. T. Hou, “Decoupling QoS control
from core routers: a novel bandwidth broker architecture for scalable [30] V. Paxson and S. Floyed, “Wide area traffic: the failure of Poisson mod-
support of guaranteed services,” in Proc. ACM SIGCOMM, 2000, pp. eling,” IEEE/ACM Trans. Netw., vol. 3, no. 3, pp. 226–244, Jun. 1995.
[31] A. Feldmann, A. Gilbert, W. Willinger, and T. G. Kurtz, “The changing
71–83.
[5] X. Xiao, A. Hannan, B. Bailey, and L. M. Ni, “Traffic engineering with nature of network traffic: scaling phenomena,” Comput. Commun. Rev.,
vol. 28, pp. 5–29, Apr. 1998.
MPLS in the Internet,” IEEE Network, vol. 14, no. 2, pp. 28–33, Mar./
Apr. 2000. [32] S. Jamin, P. B. Danzig, S. J. Shenker, and L. Zhang, “A measure-
ment-based admission control algorithm for integrated service packet
[6] D. Awduche, J. Malcolm, J. Agogbua, M. O’Dell, and J. McManus,
“Requirements for traffic engineering over MPLS,” Internet RFC 2702, networks,” IEEE/ACM Trans. Netw., vol. 5, no. 1, pp. 56–70, Feb. 1997.
Sep. 1999.
[7] F. P. Kelly, P. B. Key, and S. Zachary, “Distributed admission control,” Yu Cheng (S’01–M’04) received the B.E. and M.E.
IEEE J. Sel. Areas Commun., vol. 18, no. 12, pp. 2617–2628, Dec. degrees from Tsinghua University, Beijing, China,
2000. in 1995 and 1998, respectively, and the Ph.D. degree
[8] E. Mykoniati, C. Charalampous, P. Georgatsos, T. Damilatis, D. from the University of Waterloo, Waterloo, ON,
Goderis, P. Trimintzios, G. Pavlou, and D. Griffin, “Admission control Canada, in 2003, all in electrical engineering.
for providing QoS in DiffServ IP networks: the TEQUILA approach,” From September 2003 to August 2004, he was a
IEEE Commun. Mag., vol. 41, no. 1, pp. 38–44, Jan. 2003. Postdoctoral Fellow in the Department of Electrical
[9] Y. Cheng and W. Zhuang, “Effective bandwidth of multiclass Mar- and Computer Engineering at the University of Wa-
kovian traffic sources and admission control with dynamic buffer par- terloo. Since September 2004, he has been a Post-
titioning,” IEEE Trans. Commun., vol. 51, no. 9, pp. 1524–1535, Sep. doctoral Fellow in the Department of Electrical and
2003. Computer Engineering at the University of Toronto,
[10] G. Armitage, “MPLS: the magic behind the myths,” IEEE Commun. Toronto, ON, Canada. His research interests include Internet QoS, traffic engi-
Mag., vol. 38, no. 1, pp. 124–131, Jan. 2000. neering, service and network management, and wireless/wireline interworking.
[11] G. Swallow, “MPLS advantages for traffic engineering,” IEEE Dr. Cheng received a Postdoctoral Fellowship from the Natural Sciences and
Commun. Mag., vol. 37, no. 12, pp. 54–57, Dec. 1999. Engineering Research Council of Canada (NSERC) in 2004.
[12] E. Bouillet, D. Mitra, and K. G. Ramakrishnan, “The structure and
management of service level agreements in networks,” IEEE J. Sel.
Areas Commun., vol. 20, no. 4, pp. 691–699, May 2002.
[13] P. Trimintzios et al., “A management and control architecture for Weihua Zhuang (M’93–SM’01) received the B.Sc.
providing IP differentiated services in MPLS-based networks,” IEEE and M.Sc. degrees from Dalian Maritime University,
Commun. Mag., vol. 39, no. 5, pp. 80–88, May 2001. Liaoning, China, and the Ph.D. degree from the Uni-
[14] S. Wang, D. Xuan, R. Bettati, and W. Zhao, “Providing absolute differ- versity of New Brunswick, Fredericton, NB, Canada,
entiated services for real-time applications in static-priority scheduling all in electrical engineering.
networks,” in Proc. IEEE INFOCOM, 2001, vol. 2, pp. 669–678. Since October 1993, she has been with the De-
[15] C. Dou and F.-C. Ou, “Performance study of bandwidth reallocation partment of Electrical and Computer Engineering,
algorithms for dynamic provisioning in differentiated services net- University of Waterloo, ON, Canada, where she
works,” in Proc. 15th Int. Conf. Information Networking, 2001, pp. is a Professor. She is a coauthor of the textbook
700–705. Wireless Communications and Networking (Prentice
[16] J. Qiu and E. W. Knightly, “Inter-class resource sharing using statis- Hall, 2003). Her current research interests include
tical service envelopes,” in Proc. IEEE INFOCOM, 1999, vol. 3, pp. multimedia wireless communications, wireless networks, and radio positioning.
1404–1411. Dr. Zhuang is a licensed Professional Engineer in the Province of Ontario,
[17] R. R. Boorstyn, A. Burchard, J. Liebeherr, and C. Oottamakorn, “Sta- Canada. She received the Outstanding Performance Award in 2005 from the
tistical service assurances for traffic scheduling algorithms,” IEEE J. University of Waterloo and the Premier’s Research Excellence Award (PREA)
Sel. Areas Commun., vol. 18, no. 12, pp. 2651–2664, Dec. 2000. in 2001 from the Ontario Government. She is an Editor/Associate Editor of
[18] C. Cetinkaya, V. Kanodia, and E. W. Knightly, “Scalable services via IEEE TRANSACTIONS ON WIRELESS COMMUNICATIONS, IEEE TRANSACTIONS
egress admission control,” IEEE Trans. Multimedia, vol. 3, no. 1, pp. ON VEHICULAR TECHNOLOGY, and EURASIP Journal on Wireless Communica-
69–81, Mar. 2001. tions and Networking.