CSCI-1680

Transport Layer III

Congestion Control Strikes Back

Rodrigo Fonseca

Based partly on lecture notes by David Mazières, Phil Levis, John Jannotti, Ion Stoica
 Last Time

• Flow Control
• Congestion Control
 Today

• More TCP Fun!

• Congestion Control Continued
– Quick Review
– RTT Estimation
• TCP Friendliness
– Equation Based Rate Control
• TCP on Lossy Links
• Congestion Control versus Avoidance
– Getting help from the network
• Cheating TCP
 Quick Review

• Flow Control:
– Receiver sets Advertised Window
• Congestion Control
– Two states: Slow Start (SS) and Congestion Avoidance
(CA)
– A window size threshold governs the state transition
• Window <= ssthresh: SS
• Window > ssthresh: Congestion Avoidance
– States differ in how they respond to ACKs
• Slow start: +1 w per RTT (Exponential increase)
• Congestion Avoidance: +1 MSS per RTT (Additive increase)
– On loss event: set ssthresh = w/2, w = 1, slow start
 AIMD

Fair: A = B
AIMD
Flow Rate B

Efficient: A+B = C

Flow Rate A
 States differ in how they respond to acks

• Slow start: double w in one RTT

– There are w/MSS segments (and acks) per RTT
– Increase w per RTT à how much to increase per
ack?
• w / (w/MSS) = MSS
• AIMD: Add 1 MSS per RTT
– MSS/(w/MSS) = MSS2/w per received ACK
 Putting it all together

cwnd

Timeout

Timeout
AIMD
AIMD
ssthresh

Slow Slow Slow Time

Start Start Start
 Fast Recovery and Fast Retransmit

cwnd

AI/MD

Slow Start

Fast retransmit

Time
 TCP Friendliness
• Can other protocols co-exist with TCP?
– E.g., if you want to write a video streaming app using
UDP, how to do congestion control?

10
9
RED
Throughput(Mbps)

8
1 UDP Flow at 10MBps
7
6
31 TCP Flows
5 Sharing a 10MBps link
4
3
2
1
0
1 4 7 10 13 16 19 22 25 28 31
Flow Number
 TCP Friendliness
• Can other protocols co-exist with TCP?
– E.g., if you want to write a video streaming app using
UDP, how to do congestion control?
• Equation-based Congestion Control
– Instead of implementing TCP’s CC, estimate the rate
at which TCP would send. Function of what?
– RTT, MSS, Loss
• Measure RTT, Loss, send at that rate!
 TCP Throughput

• Assume a TCP congestion of window W (segments),

round-trip time of RTT, segment size MSS
– Sending Rate S = W x MSS / RTT (1)
• Drop: W = W/2
– grows by MSS for W/2 RTTs, until another drop at W ≈ W
• Average window then 0.75xS
– From (1), S = 0.75 W MSS / RTT (2)
• Loss rate is 1 in number of packets between losses:
– Loss = 1 / ( 1 + (W/2 + W/2+1 + W/2 + 2 + … + W)
= 1 / (3/8 W2) (3)
 TCP Throughput (cont)

8
– Loss = 8/(3W2) ⇒ W = (4)
3⋅ Loss

– Substituting (4) in (2), S = 0.75 W MSS / RTT ,

€
MSS
Throughput ≈ 1.22 ×
€
RTT⋅ Loss

• Equation-based
€ rate control can be TCP friendly and have better
properties, e.g., small jitter, fast ramp-up…
 What Happens When Link is Lossy?

• Throughput ≈ 1 / sqrt(Loss)
p=0
60

50

40
p = 1%
30

20
p = 10%

10

0
1 26 51 76 101 126 151 176 201 226 251 276 301 326 351 376 401 426 451 476
 What can we do about it?

• Two types of losses: congestion and corruption

• One option: mask corruption losses from TCP
– Retransmissions at the link layer
– E.g. Snoop TCP: intercept duplicate
acknowledgments, retransmit locally, filter them from
the sender
• Another option:
– Tell the sender about the cause for the drop
– Requires modification to the TCP endpoints
 Congestion Avoidance

• TCP creates congestion to then back off

– Queues at bottleneck link are often full: increased delay
– Sawtooth pattern: jitter
• Alternative strategy
– Predict when congestion is about to happen
– Reduce rate early
• Other approaches
– Delay Based: TCP Vegas (not covered)
– Better model of congestion: BBR
– Router-centric: RED, ECN, DECBit, DCTCP
 Another view of Congestion Control
Round Trip Time

Bytes in Flight
Throughput

Bytes in Flight

Tput =
InFlight/
RTTprop
Diagrams based on Cardwell et al., BBR: Congestion Based Congestion Control,”
Communications of the ACM, Vol. 60 No. 2, Pages 58-66.
 Another view of Congestion Control
Round Trip Time

RTTprop

Bytes in Flight
Throughput

BDP Bottleneck BW

Bytes in Flight
 Another view of Congestion Control
Round Trip Time

RTTprop

Bytes in Flight
Throughput

Bottleneck BW

BDP Bytes in Flight BDP+Bottleneck

Queue
 Another view of Congestion Control
Ideal
Round Trip Time
Operating Point

RTTprop Loss-based CC

Bytes in Flight
Throughput

Bottleneck BW

BDP Bytes in Flight BDP+Bottleneck

Queue
 BBR

• Problem: can’t measure both RTTprop and

Bottleneck BW at the same time
• BBR:
– Slow start
– Measure throughput when RTT starts to increase
– Measure RTT when throughput is still increasing
– Pace packets at the BDP
– Probe by sending faster for 1RTT, then slower to
compensate
BBR

From: https://labs.ripe.net/Members/gih/bbr-tcp
 TCP Vegas
TCP Vegas
Idea: source
• Idea: watches
source for some
watches sign that
for sign that router’s
router’squeue
queueisisbuilding
buildingup
andup
congestion will happen—E.g.,
(e.g., sending rate flattens)RTT grows or sending rate flattens.
70
60
50
40
KB

30
20
10
0.5 1.0 1.5 2.0 2.5 3.0 3.5 4.0 4.5 5.0 5.5 6.0 6.5 7.0 7.5 8.0 8.5
Time (seconds)

1100
Sending KBps

900
700
500
300
100
0.5 1.0 1.5 2.0 2.5 3.0 3.5 4.0 4.5 5.0 5.5 6.0 6.5 7.0 7.5 8.0 8.5
Time (seconds)
Queue size in router

10

0.5 1.0 1.5 2.0 2.5 3.0 3.5 4.0 4.5 5.0 5.5 6.0 6.5 7.0 7.5 8.0 8.5
Time (seconds)
 TCP Vegas
• Compare Actual Rate (A) with Expected Rate (E)
6.4 Congestion-Avoidance Mechanisms 491
– If E-A > β, decrease cwnd linearly : A isn’t responding
– If E-A < α, increase cwnd linearly : Room for A to grow
70
60
50
40
KB

30
20
10

0.5 1.0 1.5 2.0 2.5 3.0 3.5 4.0 4.5 5.0 5.5 6.0 6.5 7.0 7.5 8.0
Time (seconds)

240
200
KBps

160
120
80
40

0.5 1.0 1.5 2.0 2.5 3.0 3.5 4.0 4.5 5.0 5.5 6.0 6.5 7.0 7.5 8.0
Time (seconds)
 Vegas

• Shorter router queues

• Lower jitter
• Problem:
– Doesn’t compete well with Reno. Why?
– Reacts earlier, Reno is more aggressive, ends up with
higher bandwidth…
 Help from the network

• What if routers could tell TCP that congestion

is happening?
– Congestion causes queues to grow: rate mismatch
• TCP responds to drops
• Idea: Random Early Drop (RED)
– Rather than wait for queue to become full, drop
packet with some probability that increases with
queue length
– TCP will react by reducing cwnd
– Could also mark instead of dropping: ECN
 RED Details
• Compute average queue length (EWMA)
– Don’t want to reactAvgLen
to very quick fluctuations

Queue length

Instantaneous

Average

Time
• Computing probability P
- TempP = MaxP · (AvgLen MinThreshold)/(MaxThreshold
MinThreshold)
RED Drop Probability
- P = TempP/(1
• Define twocount · TempP)
thresholds: MinThresh, MaxThresh
• Drop• Probability Curve:
Drop probability:
P(drop)

1.0

MaxP
AvgLen
MinThresh MaxThresh

• Improvements to spread drops (see book)

 RED Advantages

• Probability of dropping a packet of a particular

flow is roughly proportional to the share of the
bandwidth that flow is currently getting
• Higher network utilization with low delays
• Average queue length small, but can absorb
bursts
• ECN
– Similar to RED, but router sets bit in the packet
– Must be supported by both ends
– Avoids retransmissions optionally dropped packets
What happens if not everyone cooperates?

• TCP works extremely well when its

assumptions are valid
– All flows correctly implement congestion control
– Losses are due to congestion
 Cheating TCP

• Possible ways to cheat

– Increasing cwnd faster
– Large initial cwnd
– Opening many connections
– Ack Division Attack
 Increasing cwnd Faster

C y
x increases by 2 per RTT
y increases by 1 per RTT

Limit rates:
x = 2y

Figure from Walrand, Berkeley EECS 122, 2003

 Larger Initial Window
x
A B
y
D E
x starts SS with cwnd = 4
y starts SS with cwnd = 1

Figure from Walrand, Berkeley EECS 122, 2003

 Open Many Connections
• Web Browser: has to download k objects for a page
– Open many connections or download sequentially?
x
A B
y
D E
• Assume:
– A opens 10 connections to B
– B opens 1 connection to E
• TCP is fair among connections
– A gets 10 times more bandwidth than B
Figure from Walrand, Berkeley EECS 122, 2003
 Exploiting Implicit Assumptions

• Savage, et al., CCR 1999:

– “TCP Congestion Control with a Misbehaving Receiver”
• Exploits ambiguity in meaning of ACK
– ACKs can specify any byte range for error control
– Congestion control assumes ACKs cover entire sent segments
• What if you send multiple ACKs per segment?
 ACK Division Attack

• 2.1Receiver:
TCP review
“upon receiving a
segment
While with ofNTCP's
a detailed description bytes, divide
error and congestion the
con-
Sender
Data 1
Receiver
trol mechanisms is beyond the scope of this paper, we describe the :1461
bytesof in
rudiments their M groups
behavior and
below to allow thoseacknowledge
unfamiliar with
TCP to understand the vulnerabilities explained later. For simplic-
each
ity, group
we consider TCP withoutseparately”
the Selective Acknowledgment op- RTT ACK 4
87
73
tion (SACK) [MMFR96], although the vulnerabilities we describe ACK 9
• alsoSender
exist when SACKwill grow window M times
is used.
TCP is a connection-oriented, reliable, ordered, byte-stream
ACK 1
461

faster
protocol with explicit flow control. A sending host divides the data
stream into individual segments, each of which is no longer than the
Data 1
Data 2
461:29
21
921:43
• Sender
Could
nection
Maximum Segment Size (SMSS) determined during con-
cause
establishment. Each growth
segment is labeledtowith4GB
explicit in
se- 4 Data 4
381:58
81
41
quence numbers to guarantee ordering and reliability. When a host
RTTs!
receives an in-sequence segment it sends a cumulative acknowl-
Data 5
841:73
01
edgment (ACK) in return, notifying the sender that all of the data
– M = N = 1460
preceding that segment' s sequence number has been received and
can be retired from the sender' s retransmission buffers. If an out-
of-sequence segment is received, then the receiver acknowledges
the next contiguous sequence number that was expected. If out- Figure 1: Sample time line for a ACK division attack. Th
standing data is not acknowledged for a period of time, the sender gins with cwnd=1, which is incremented for each of the three
will timeout and retransmit the unacknowledged segments. received. After one round-trip time, cwnd=4, instead of the ex
TCP uses several algorithms for congestion control, most no- of cwnd=2.
tably slow start and congestion avoidance [Jac88, Ste94, APS99].
Each of these algorithms controls the sending rate by manipulating This attack is demonstrated in Figure 1 with a time
 TCP Daytona!

60000
Sequence number (Bytes)

Sequence number (Bytes)

50000

40000

30000

20000
Data Segments
10000 ACKs
Data Segments (normal)
ACKs (normal)
0
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7
Time (sec)

Figure 4: The TCP Daytona ACK division attack convinces the TCP F
 Defense

• Appropriate Byte Counting

– [RFC3465 (2003), RFC 5681 (2009)]
– In slow start, cwnd += min (N, MSS)
where N is the number of newly acknowledged bytes in
the received ACK
 Cheating TCP and Game Theory
x
A B
y
D E

D à Increases by 1 Increases by 5
A
22, 22 10, 35
(x, y)
Increases by 1
35, 10 15, 15
Increases by 5 Too aggressive
àLosses
Individual incentives: cheating pays àThroughput falls
Social incentives: better off without cheating

Classic PD: resolution depends on accountability 38

 An alternative for reliability

• Erasure coding
– Assume you can detect errors
– Code is designed to tolerate entire missing packets
• Collisions, noise, drops because of bit errors
– Forward error correction
• Examples: Reed-Solomon codes, LT Codes,
Raptor Codes
• Property:
– From K source frames, produce B > K encoded frames
– Receiver can reconstruct source with any K’ frames,
with K’ slightly larger than K
– Some codes can make B as large as needed, on the fly
 LT Codes

• Luby Transform Codes

– Michael Luby, circa 1998
• Encoder: repeat B times
1. Pick a degree d (*)
2. Randomly select d source blocks. Encoded block tn=
XOR or selected blocks

* The degree is picked from a distribution, robust soliton

distribution, that guarantees that the decoding process will succeed
with high probability
 LT Decoder

• Find an encoded block tn with d=1

• Set sn = tn
• For all other blocks tn’ that include sn ,
set tn’=tn’ XOR sn
• Delete sn from all encoding lists
• Finish if
1. You decode all source blocks, or
2. You run out out blocks of degree 1
 Next Time

• Move into the application layer

• DNS, Web, Security, and more…
 Backup slides

• We didn’t cover these in lecture: won’t be in

the exam, but you might be interested J
 More help from the network
• Problem: still vulnerable to malicious flows!
– RED will drop packets from large flows preferentially,
but they don’t have to respond appropriately
• Idea: Multiple Queues (one per flow)
– Serve queues in Round-Robin
– Nagle (1987)
– Good: protects against misbehaving flows
– Disadvantage?
– Flows with larger packets get higher bandwidth
 Solution

• Bit-by-bit round robing

• Can we do this?
– No, packets cannot be preempted!
• We can only approximate it…
 Fair Queueing

• Define a fluid flow system as one where flows

are served bit-by-bit
• Simulate ff, and serve packets in the order in
which they would finish in the ff system
• Each flow will receive exactly its fair share
 Example

Flow 1 1 2 3 4 5 6
(arrival traffic) time

Flow 2
1 2 3 4 5
(arrival traffic)
time

Service 1 2 3 4 5 6
1 2
in fluid flow 3 4 5 time
system

Packet 1 2 1 3 2 3 4 4 5 5 6
system time
 Implementing FQ
• Suppose clock ticks with each bit transmitted
– (RR, among all active flows)
• Pi is the length of the packet
• Si is packet i’s start of transmission time
• Fi is packet i’s end of transmission time
• Fi = Si + Pi
• When does router start transmitting packet i?
– If arrived before Fi-1, Si = Fi-1
– If no current packet for this flow, start when packet
arrives (call this Ai): Si = Ai
• Thus, Fi = max(Fi-1,Ai) + Pi
 Fair Queueing

• Across all flows

– Calculate Fi for each packet that arrives on each flow
– Next packet to transmit is that with the lowest Fi
– Clock rate depends on the number of flows
• Advantages
– Achieves max-min fairness, independent of sources
– Work conserving
• Disadvantages
– Requires non-trivial support from routers
– Requires reliable identification of flows
– Not perfect: can’t preempt packets
 Fair Queueing Example

• 10Mbps link, 1 10Mbps UDP, 31 TCPs

10 2
9 1.8
RED FQ

Throughput(Mbps)
Throughput(Mbps)

8 1.6
7 1.4
6 1.2
5 1
4 0.8
3 0.6
2 0.4
1 0.2
0 0
1 4 7 10 13 16 19 22 25 28 31 1 4 7 10 13 16 19 22 25 28 31
Flow Number Flow Number
 Big Picture

• Fair Queuing doesn’t eliminate congestion:

just manages it
• You need both, ideally:
– End-host congestion control to adapt
– Router congestion control to provide isolation