Decoupling TCP from IP with

Multipath TCP
Olivier Bonaventure
http://inl.info.ucl.ac.be
http://perso.uclouvain.be/olivier.bonaventure

Thanks to Sébastien Barré, Christoph Paasch, Grégory Detal, Mark Handley,

Costin Raiciu, Alan Ford, Micchio Honda, Fabien Duchene and many others
March 2013
 Agenda
• The motivations for Multipath TCP

• The changing Internet

• The Multipath TCP Protocol

• Multipath TCP use cases

 The origins of TCP

Source : http://spectrum.ieee.org/computing/software/the-strange-birth-and-long-life-of-unix
 The Unix pipe model

1234 abbsbbbs
echo wc
The TCP bytestream model
Client Server
ABCDEF...111232

0988989 ... XYZZ

IP:1.2.3.4 IP:4.5.6.7
 Endhosts have evolved

Mobile devices have multiple wireless interfaces

User expectations
What technology provides
3G celltower

IP 1.2.3.4
What technology provides
3G celltower

IP 1.2.3.4

IP 5.6.7.8
 What technology provides
3G celltower

IP 1.2.3.4

IP 5.6.7.8

When IP addresses change TCP connections

have to be re-established !
Datacenters
 Agenda
• The motivations for Multipath TCP

• The changing Internet

• The Multipath TCP Protocol

• Multipath TCP use cases

 The Internet architecture
that we explain to our students
Applicatio
n
Transport
Datalink
Network
Physical
Datalink
Physical

Network
Datalink
Physical
Physical

O. Bonaventure, Computer networking : Principles, Protocols and Practice, open ebook, http://inl.info.ucl.ac.be/cnp3
 A typical "academic" network

Applicatio Applicatio
n n
Transport Transport
Network Network Network
Datalink Datalink Datalink Datalink
Physical Physical Physical Physical
 The end-to-end principle

Applicatio Applicatio
n n
TCP
Transport Transport
Network Network Network
Datalink Datalink Datalink Datalink
Physical Physical Physical Physical
 In reality

– almost as many middleboxes as routers

– various types of middleboxes are deployed
Sherry, Justine, et al. "Making middleboxes someone else's problem: Network processing as a cloud service."
Proceedings of the ACM SIGCOMM 2012 conference. ACM, 2012.
 A middlebox zoo
Web Security VPN Concentrator
Appliance

SSL NAC Appliance

Terminator

ACE XML Cisco IOS Firewall

PIX Firewall
Gateway Right and Left

IP Telephony
Streamer Voice
Router
Gateway

NAT
Content
Engine

http://www.cisco.com/web/about/ac50/ac47/2.html
How to model those middleboxes ?
• In the official architecture, they do not exist
• In reality...

Applicatio Applicatio Applicatio

n n n
TCP
Transport Transport Transport
Network Network Network Network
Datalink Datalink Datalink Datalink
Physical Physical Physical Physical
 TCP segments processed by a router
Ver IHL ToS Total length Ver IHL ToS Total length
Identification Flags Frag. Offset Identification Flags Frag. Offset
IP TTL Protocol Checksum TTL Protocol Checksum
Source IP address Source IP address
Destination IP address Destination IP address
Source port Destination port Source port Destination port
Sequence number Sequence number
Acknowledgment number Acknowledgment number
TCPTHL Reserved Flags Window THL Reserved Flags Window
Checksum Urgent pointer Checksum Urgent pointer
Options Options

Payload Payload
 TCP segments processed by a NAT
Ver IHL ToS Total length Ver IHL ToS Total length
Identification Flags Frag. Offset Identification Flags Frag. Offset
TTL Protocol Checksum TTL Protocol Checksum
Source IP address Source IP address
Destination IP address Destination IP address
Source port Destination port Source port Destination port
Sequence number Sequence number
Acknowledgment number Acknowledgment number
THL Reserved Flags Window THL Reserved Flags Window
Checksum Urgent pointer Checksum Urgent pointer
Options Options

Payload Payload
 TCP segments processed by a NAT (2)

• active mode ftp behind a NAT

220 ProFTPD 1.3.3d Server (BELNET FTPD Server) [193.190.67.15]
ftp_login: user `<null>' pass `<null>' host `ftp.belnet.be'
Name (ftp.belnet.be:obo): anonymous
---> USER anonymous
331 Anonymous login ok, send your complete email address as your password
Password:
---> PASS XXXX
---> PORT 192,168,0,7,195,120
200 PORT command successful
---> LIST
150 Opening ASCII mode data connection for file list
lrw-r--r-- 1 ftp ftp 6 Jun 1 2011 pub -> mirror
226 Transfer complete
 TCP segments processed by an ALG running
on a NAT
Ver IHL ToS Total length Ver IHL ToS Total length
Identification Flags Frag. Offset Identification Flags Frag. Offset
TTL Protocol Checksum TTL Protocol Checksum
Source IP address Source IP address
Destination IP address Destination IP address
Source port Destination port Source port Destination port
Sequence number Sequence number
Acknowledgment number Acknowledgment number
THL Reserved Flags Window THL Reserved Flags Window
Checksum Urgent pointer Checksum Urgent pointer
Options Options

Payload Payload
How transparent is the Internet ?
• 25th September 2010
to 30th April 2011
• 142 access networks
• 24 countries
• Sent specific TCP
segments from client
to a server in Japan

Honda, Michio, et al. "Is it still possible to extend TCP?" Proceedings of the 2011 ACM
SIGCOMM conference on Internet measurement conference. ACM, 2011. © O. Bonaventure, 2011
 End-to-end transparency today
Ver IHL ToS Total length Ver IHL ToS Total length

Identification Flags Frag. Offset Identification Flags Frag. Offset

Protocol Checksum
TTL Middleboxes TTL Protocol Checksum
don't change
the Protocol
Source field,
IP address but Source IP address

many discard
Destination packets with an
IP address Destination IP address

Sourceunknown
port Protocol field
Destination port Source port Destination port

Sequence number Sequence number

Acknowledgment number Acknowledgment number
THL Reserved Flags Window THL Reserved Flags Window

Checksum Urgent pointer Checksum Urgent pointer

Options Options

Payload Payload
 Agenda
• The motivations for Multipath TCP

• The changing Internet

• The Multipath TCP Protocol

• Multipath TCP use cases

 Design objectives
• Multipath TCP is an evolution of TCP

• Design objectives
– Support unmodified applications
– Work over today’s networks
– Works in all networks where regular TCP works
 TCP Connection establishment
• Three-way handshake

SYN,seq=1234,Options

SYN+ACK,ack=1235,seq=5678,Options
ACK,seq=1235,ack=5679
 Data transfer

seq=1234,"abcd"

ACK,ack=1238,win=4

seq=1238,"efgh"

ACK,ack=1242,win=0
 Connection release

seq=1234,"abcd"

RST
 Connection release

seq=1234,"abcd"
FIN, seq=1238
ACK,ack=1239

seq=345,"ijkl"
FIN,seq=349

FIN,ack=350
Identification of a TCP connection
Ver IHL ToS Total length Four tuple
IP
Identification Flags Frag. Offset
TTL Protocol Checksum
– IPsource
Source IP address
– IPdest
Destination IP address
Source port Destination port – Portsource
Sequence number
Acknowledgment number – Portdest
TCP
THL Reserved Flags Window
Checksum Urgent pointer
Options All TCP segments
contain the four
Payload tuple
 The new bytestream model

Client Server
ABCDEF...111232

0988989 ... XYZZ

D C B A
IP:2.3.4.5 IP:6.7.8.9

IP:4.5.6.7
IP:1.2.3.4

38
 The Multipath TCP protocol
• Control plane
– How to manage a Multipath TCP connection that
uses several paths ?
• Data plane
– How to transport data ?

• Congestion control
– How to control congestion over multiple paths ?
A naïve Multipath TCP
SYN+Option
SYN+ACK+Option
ACK
seq=123, "abc"

seq=126, "def"
 A naïve Multipath TCP
In today's Internet ?
SYN+Option
SYN+ACK+Option
ACK
seq=123, "abc"

There is no
corresponding
TCP connection
seq=126, "def"
 Design decision

– A Multipath TCP connection is composed of one of

more regular TCP subflows that are combined

• Each host maintains state that glues the TCP subflows

that compose a Multipath TCP connection together

• Each TCP subflow is sent over a single path and appears

like a regular TCP connection along this path
Multipath TCP and the architecture

Application

socket
Applicatio
n Multipath TCP
Transport
Network TCP1 TCP2 ... TCPn

Datalink
Physical

A. Ford, C. Raiciu, M. Handley, S. Barre, and J. Iyengar, “Architectural guidelines for multipath TCP
development", RFC6182 2011.
 A regular TCP connection
• What is a regular TCP connection ?

– It starts with a three-way handshake

• SYN segments may contain special options

– All data segments are sent in sequence

• There is no gap in the sequence numbers

– It is terminated by using FIN or RST

 Multipath TCP
SYN+Option
SYN+ACK+Option
ACK

SYN+OtherOption
SYN+ACK+OtherOption
ACK
How to combine two TCP subflows ?

SYN+Option
SYN+ACK+Option
ACK

SYN+OtherOption How to link with

SYN+ACK+OtherOption blue subflow ?
ACK
How to link TCP subflows ?
SYN, Portsrc=1234,Portdst=80+Option

SYN+ACK[...]
A NAT could change
ACK addresses and
port numbers

SYN, Portsrc=1235,Portdst=80
+Option[link Portsrc=1234,Portdst=80]
 How to link TCP subflows ?
SYN, Portsrc=1234,Portdst=80
+Option[Token=5678]
SYN+ACK+Option[Token=6543]
ACK
MyToken=5678
YourToken=6543

MyToken=6543
YourToken=5678

SYN, Portsrc=1235,Portdst=80
+Option[Token=6543]
 Subflow agility
• Multipath TCP supports
– addition of subflows
– removal of subflows
 The Multipath TCP protocol
• Control plane
– How to manage a Multipath TCP connection that
uses several paths ?
• Data plane
– How to transport data ?

• Congestion control
– How to control congestion over multiple paths ?
 How to transfer data ?
seq=123,"a"

ack=124
seq=125,"c"
ack=126

seq=124,"b"
ack=125
seq=126,"d"
ack=127
 How to transfer data
in today's Internet ?
seq=123,"a"

ack=124
seq=125,"c"
ack=126
Gap in sequence numbering space
Some DPI will not allow this !

seq=124,"b"
ack=125
 Multipath TCP Data transfer
• Two levels of sequence numbers

ABCDEF

socket socket

Multipath TCP Multipath TCP

Data sequence #
TCP1 TCP1 sequence # TCP1

TCP2 TCP2 sequence # TCP2

 Multipath TCP
Data transfer
Dseq=0,seq=123,"a"

DAck=1,ack=124
DSeq=2, seq=124,"c"
DAck=3, ack=125

DSeq=1, seq=456,"b"
DAck=2,ack=457
 Multipath TCP
How to deal with losses ?
• Data losses over one TCP subflow
– Fast retransmit and timeout as in regular TCP
Dseq=0,seq=123,"a"
DAck=1,ack=12
4 Dseq=0,seq=123,"a"

DAck=1,ack=124
 Multipath TCP
• What happens when a TCP subflow fails ?
Dseq=0,seq=123,"a"

DSeq=1, seq=456,"b"
DSeq=0,ack=457
Dseq=0,seq=457,"a"
DAck=2,ack=458
 Retransmission heuristics
• Heuristics used by current Linux implementation

– Fast retransmit is performed on the same subflow

as the original transmission
– Upon timeout expiration, reevaluate whether the
segment could be retransmitted over another subflow

– Upon loss of a subflow, all the unacknowledged data

are retransmitted on other subflows
 Flow control
• How should the window-based flow control be
performed ?

– Independant windows on each TCP subflow

– A single window that is shared among all TCP

subflows
Independant windows

Dseq=0,seq=123,"a"

DAck=1,ack=124,win=0

DSeq=1, seq=456,"b"
DAck=2,ack=457,win=100
Dseq=2,seq=457,"c"
DAck=3,ack=458,win=100
 Independant windows
possible problem
Dseq=0,seq=123,"a"

DSeq=1, seq=456,"b"
DAck=2,ack=457,win=0

• Impossible to retransmit, window is already full on green subflow

A single window shared by all subflows

Dseq=0,seq=123,"a"

DAck=1,ack=124,win=10

DSeq=1, seq=456,"b"
DAck=2,ack=457,win=10
Dseq=2,seq=457,"c"
DAck=3,ack=458,win=10
 A single window shared by all subflows
Impact of middleboxes

Dseq=0,seq=123,"a"

DAck=1,ack=124,win=10
0

DSeq=1, seq=456,"b"
DAck=2,ack=457,win=100

DAck=2,ack=457,win=5
 Multipath TCP Windows
• Multipath TCP maintains one window per Multipath TCP
connection

– Window is relative to the last acked data (Data Ack)

– Window is shared among all subflows
• It's up to the implementation to decide how the window is shared
– Window is transmitted inside the window field of the regular
TCP header
– If middleboxes change window field,
• use largest window received at MPTCP-level
• use received window over each subflow to cope with the flow control
imposed by the middlebox
 Multipath TCP buffers
MPTCP-level,
send(...) resequencing
possible recv(...)
socket

Multipath TCP
Scheduler

TCP1

TCP2

Transmit queues, Reorder queue,

process only regular processes only
TCP header TCP header
Sending Multipath TCP information
• How to exchange the Multipath TCP specific
information between two hosts ?

• Option 1
– Use TLVs to encode data and control information inside
payload of subflows

• Option 2
– Use TCP options to encode all Multipath TCP
information
Option 1 : Michael Scharf, Thomas-Rolf Banniza , MCTCP: A Multipath Transport Shim Layer, GLOBECOM 2011
 Is it safe to use TCP options ?
• Known option (TS) in Data segments

XD6BHM

Honda, Michio, et al. "Is it still possible to extend TCP?." Proceedings of the 2011 ACM
SIGCOMM conference on Internet measurement conference. ACM, 2011. © O. Bonaventure, 2011
 Is it safe to use TCP options ?
• Unknown option in Data segments

XD6BHM

Honda, Michio, et al. "Is it still possible to extend TCP?." Proceedings of the 2011 ACM
SIGCOMM conference on Internet measurement conference. ACM, 2011. © O. Bonaventure, 2011
 Multipath TCP options
• TCP option format

Kind Length Option-specific data

• Initial design
– One option kind for each purpose
(e.g. Data Sequence number)

• Final design
– A single variable-length Multipath TCP option
 Multipath TCP option
• A single option type
– to minimise the risk of having one option accepted
by middleboxes in SYN segments and rejected in
segments carrying data

Kind LengthSubtype
Subtype specific data
(variable length)
 Data sequence numbers
and TCP segments
• How to transport Data sequence numbers ?
– Same solution as for TCP
• Data sequence number in TCP option is the Data
sequence number of the first byte of the segment
Source port Destination port
Sequence number
Acknowledgment number
THL Reserved Flags Window
Checksum Urgent pointer
Datasequence number

Payload
 Multipath TCP
Data transfer
Dseq=0,seq=123,"a"

DAck=1,ack=124
DSeq=2, seq=124,"c"
DAck=3, ack=125

DSeq=1, seq=456,"b"
DAck=2,ack=457
 TCP sequence number
and middleboxes

Honda, Michio, et al. "Is it still possible to extend TCP?." Proceedings of the 2011 ACM
SIGCOMM conference on Internet measurement conference. ACM, 2011. © O. Bonaventure, 2011
 Which middleboxes change
TCP sequence numbers ?
• Some firewalls change TCP sequence numbers
in SYN segments to ensure randomness
– fix for old windows95 bug

• Transparent proxies terminate TCP

connections
 Other types of
middlebox interference

• Data segments

Data,seq=12,"ab"

Data,seq=14,"cd"
Data,seq=12,"abcd"

Such a middlebox could also be the network adapter of the

server that uses LRO to improve performance.
 Segment coalescing

Honda, Michio, et al. "Is it still possible to extend TCP?." Proceedings of the 2011 ACM
SIGCOMM conference on Internet measurement conference. ACM, 2011. © O. Bonaventure, 2011
 Data sequence numbers
and middleboxes
buffers small
seq=123,Dseq=0, "a" segments
seq=124, DSeq=2,"c" seq=123, DSeq=2, "ac"

copies one option seq=123, DSeq=0, "ac"

in coalesced
segment

seq=456, DSeq=1, "b"

 Data sequence numbers
and middleboxes

seq=123,Dseq=0, "ab"

DSeq=0, seq=123,"a"
Middlebox only DSeq=0, seq=124,"b"
understands
regular TCP
A "middlebox" that both splits and coalesces
TCP segments
 Data sequence numbers
and middleboxes
• How to avoid desynchronisation between the
bytestream and data sequence numbers ?

• Solution
– Multipath TCP option carries mapping between
Data sequence numbers and (difference between
initial and current) subflow sequence numbers
• mapping covers a part of the bytestream (length)
 Multipath TCP
Data transfer
seq=123,DSS[0->123,len=1],"a"

DAck=1,ack=124
seq=124, DSS[2->124,len=1],"c"
DAck=3, ack=125

seq=456, DSS[1->456,len=1],"b"
DAck=2,ack=457
 Data sequence numbers
and middleboxes
seq=123,DSS[0->123,len=1], "a"

seq=124, seq=123,
DSS[2->124, len=1],"c" DSS[0->123, len=1],
"ac"
DAck=2,ack=124
seq=124, DSS[2->124, len=1],"c"

seq=456, DSS[1->456, len=1], "b"

DSeq=0,ack=457
 Multipath TCP and middleboxes
• With the DSS mapping, Multipath TCP can
cope with middleboxes that
– combine segments
– split segments

• Are they the most annoying middleboxes for

Multipath TCP ?

– Unfortunately not
 The worst middlebox
seq=123,
DSS[1->123,len=2], seq=123,
"ab" DSS[1->123, len=2],
"aXXXb"
DAck=3,ack=125
DAck=3,ack=128
seq=125,
DSS[3->125, len=2], seq=128,
"cd" DSS[3->125, len=2],
"cd"

• Is this an academic exercise or reality ?

 The worst middlebox
• Is unfortunately very old...
– Any ALG for a NAT

220 ProFTPD 1.3.3d Server (BELNET FTPD Server) [193.190.67.15]

ftp_login: user `<null>' pass `<null>' host `ftp.belnet.be'
Name (ftp.belnet.be:obo): anonymous
---> USER anonymous
331 Anonymous login ok, send your complete email address as your password
Password:
---> PASS XXXX
---> PORT 192,168,0,7,195,120
200 PORT command successful
---> LIST
150 Opening ASCII mode data connection for file list
lrw-r--r-- 1 ftp ftp 6 Jun 1 2011 pub -> mirror
226 Transfer complete
 Coping with the worst middlebox
• What should Multipath TCP do in the presence of
such a worst middlebox ?
– Do nothing and ignore the middlebox
• but then the bytestream and the application would be broken
and this problem will be difficult to debug by network
administrators

– Detect the presence of the middlebox

• and fallback to regular TCP (i.e. use a single path and nothing
fancy)

Multipath TCP MUST work in all networks where

regular TCP works.
 Detecting the worst middlebox ?
• How can Multipath TCP detect a middlebox
that modifies the bytestream and
inserts/removes bytes ?

– Various solutions were explored

– In the end, Multipath TCP chose to include its own

checksum to detect insertion/deletion of bytes
 The worst middlebox
seq=123, seq=123,
DSS[1->123,len=2,V], DSS[1->123, len=2,Inv],
"ab" "aXXXb"

RST, last DSeq=0 RST, last DSeq=0

seq=456, DSS[1->456, len=2,V], "ab"

DAck=3,ack=458
 Multipath TCP
Data sequence numbers
• What should be the length of the data
sequence numbers ?

– 32 bits
• compact and compatible with TCP
• wrap around problem at highspeed requires PAWS

– 64 bits
• wrap around is not an issue for most transfers today
• takes more space inside each segment
 Multipath TCP
Data sequence numbers
• Data sequence numbers and Data
acknowledgements

– Maintained inside implementation as 64 bits field

– Implementations can, as an optimisation, only

transmit the lower 32 bits of the data sequence
and acknowledgements
 Data Sequence Signal option
A = Data ACK present
a = Data ACK is 8 octets
Cumulative Data ack M = mapping present
m = DSN is 8

Length of mapping, can extend

Computed over data covered by
beyond this segment
entire mapping + pseudo header
 Cost of the DSN checksum

C. Raiciu, et al. “How hard can it be? designing and implementing a deployable multipath TCP,” NSDI'12:
Proceedings of the 9th USENIX conference on Networked Systems Design and Implementation, 2012.
 The Multipath TCP protocol
• Control plane
– How to manage a Multipath TCP connection that
uses several paths ?
• Data plane
– How to transport data ?

• Congestion control
– How to control congestion over multiple paths ?
 TCP congestion control
• A linear rate adaption algorithm

To be fair and efficient, a linear algorithm must use

additive increase and multiplicative decrease
(AIMD)
# Additive Increase Multiplicative Decrease
if congestion :
   rate=rate*betaC    # multiplicative decrease, betaC<1
else
   rate=rate+alphaN    # additive increase, v0>0
 AIMD in TCP
• Congestion control mechanism
– Each host maintains a congestion window (cwnd)
– No congestion
• Congestion avoidance (additive increase)
– increase cwnd by one segment every round-trip-time
– Congestion
• TCP detects congestion by detecting losses
• Mild congestion (fast retransmit – multiplicative decrease)
– cwnd=cwnd/2 and restart congestion avoidance
• Severe congestion (timeout)
– cwnd=1, set slow-start-threshold and restart slow-start
 Evolution of the congestion window

Cwnd Fast retransmit

Fast retransmit

Threshold
Threshold

Time
Slow-start Congestion avoidance
exponential increase of cwnd linear increase of cwnd
 Congestion control for Multipath TCP

• Simple approach
– independant congestion windows

Threshold
Threshold

Threshold
 Independant congestion windows
• Problem

12Mbps
 Coupling the congestion windows
• Principle
– The TCP subflows are not independant and their
congestion windows must be coupled

• EWTCP
– For each ACK on path r, cwinr=cwinr+a/cwinr (in segments)
– For each loss on path r, cwinr=cwinr/2

– Each subflow gets window size proportional to a2

1
– Same throughput as TCP if a=
n
M. Honda, Y. Nishida, L. Eggert, P. Sarolahti, and H. Tokuda. Multipath Congestion Control for Shared Bottleneck. In
Proc. PFLDNeT workshop, May 2009.
 Can we split traffic equally
among all subflows ?

12Mbps

12Mbps 12Mbps

In this scenario, EWTCP would get 3.5 Mbps on the two

hops path and 5 Mbps on the one hop path, less than
the optimum of 12 Mbps for each Multipath TCP connection
D. Wischik, C. Raiciu, A. Greenhalgh, and M. Handley, “Design, implementation and evaluation of congestion
control for multipath TCP,” NSDI'11: Proceedings of the 8th USENIX conference on Networked systems design
and implementation, 2011.
Linked increases congestion control
• Algorithm
– For each loss on path r, cwinr=cwinr/2

– Additive increase
cwndi
max( 2
)
(rtti ) 1
cwinr = cwinr + min( , )
cwndi 2 cwndr
(å )
i rtti

D. Wischik, C. Raiciu, A. Greenhalgh, and M. Handley, “Design, implementation and evaluation of congestion
control for multipath TCP,” NSDI'11: Proceedings of the 8th USENIX conference on Networked systems design
and implementation, 2011.
 Other Multipath-aware
congestion control schemes
R. Khalili, N. Gast, M. Popovic, U. Upadhyay, J.-Y. Le Boudec , MPTCP is not
Pareto-optimal: Performance issues and a possible solution, Proc. ACM
Conext 2012
Y. Cao, X. Mingwei, and X. Fu, “Delay-based Congestion Control for Multipath
TCP,” ICNP2012, 2012.
T. A. Le, C. S. Hong, and E.-N. Huh, “Coordinated TCP Westwood congestion
control for multiple paths over wireless networks,” ICOIN '12: Proceedings of the
The International Conference on Information Network 2012, 2012, pp. 92–96.
T. A. Le, H. Rim, and C. S. Hong, “A Multipath Cubic TCP Congestion Control
with Multipath Fast Recovery over High Bandwidth-Delay Product Networks,”
IEICE Transactions, 2012.
T. Dreibholz, M. Becke, J. Pulinthanath, and E. P. Rathgeb, “Applying TCP-Friendly
Congestion Control to Concurrent Multipath Transfer,” Advanced Information
Networking and Applications (AINA), 2010 24th IEEE International Conference on,
2010, pp. 312–319.
 The Multipath TCP protocol
• Control plane
– How to manage a Multipath TCP connection that
uses several paths ?
• Data plane
– How to transport data ?

• Congestion control
– How to control congestion over multiple paths ?
 The Multipath TCP control plane
• Connection establishment in details

• Closing a Multipath TCP connection

• Address dynamics
 Security threats
• Three main security threats were considered
– flooding attack

– man-in-the middle attack

– hijacking attach Security goal :

Multipath TCP should not be
worse than regular TCP

J. Diez, M. Bagnulo, F. Valera, and I. Vidal, “Security for multipath TCP: a constructive approach,” International
Journal of Internet Protocol Technology, vol. 6, 2011.
Hijacking attack
 Multipath TCP
Connection establishment
• Principle
SYN, MP_CAPABLE

SYN+ACK, MP_CAPABLE
ACK, MP_CAPABLE

seq=123, DSeq=1, "abc"

 Roles of the initial TCP handshake
• Check willingness to open TCP connection
– Propose initial sequence number
– Negotiate Maximum Segment Size
• TCP options
– negotiate Timestamps, SACK, Window scale
• Multipath TCP
– check that server supports Multipath TCP
– propose Token in each direction
– propose initial Data sequence number in each direction
– Exchange keys to authenticate subflows
 How to extend TCP ?
Theory
• TCP options were invented for this purpose
– Exemple SACK

SACK enabled
SYN+SACK Permitted

SYN+ACK SACK Permitted

SACK ?yes
 How to extend TCP ?
practice
• What happens when there are middleboxes
on the path ?

SYN+SACK Permitted SACK no

SYN

SYN+ACK SYN+ACK
SACK no
 TCP options
• In SYN segments

XD6BHM

Honda, Michio, et al. "Is it still possible to extend TCP?." Proceedings of the 2011 ACM
SIGCOMM conference on Internet measurement conference. ACM, 2011. © O. Bonaventure, 2011
 How to extend TCP ?
The worst case
• What happens when there are middleboxes
on the path ?

SYN+SACK Permitted SYN+SACK Permitted

SYN+ACK SACK enabled

SACK no
SYN+ACK SACK Permitted

Client and server do not agree

on TCP extension !!!
Multipath TCP handshake

SYN, MP_CAPABLE[...]

SYN+ACK, MP_CAPABLE[...]
ACK+MPTCPOption

Why an option in
third ACK ?
Multipath TCP option in third ACK
SYN, MP_CAPABLE[...]

SYN+ACK, MP_CAPABLE[...]
SYN+ACK
ACK

No option, disable
Multipath TCP
 Multipath TCP handshake
Token exchange

SYN, MP_CAPABLE[ClientToken=1234]

SYN+ACK, MP_CAPABLE[ServerToken=5678]
ACK, MP_CAPABLE[ClientToken=1234]

Useful if server
wants to send
SYN+ACK without
keeping any state
 Initial Data Sequence number
• Why do we need an initial Data Sequence
number ?

– Setting IDSN to a random value improves security

– Hosts must know IDSN to avoid losing data in

some special cases
Initial Data Sequence number
SYN, MP_CAPABLE

SYN+ACK, MP_CAPABLE
ACK
seq=123,
DSS[456->123,len=2],"ab"
First Data or not ?

SYN...
SYN+ACK...
ACK

seq=789, DSS[458->789, len=2], "cd"

 Initial Data Sequence number
• How to negotiate the IDSN ?
SYN, MP_CAPABLE[DSeq=23456]

SYN+ACK, MP_CAPABLE[DSeq=67890]
ACK
 How to secure Multipath TCP
• Main goal
– Authenticate the establishment of subflows

• Principles
– Each host announces a key during initial
handshake
• keys are exchanged in clear
– When establishing a subflow, use HMAC + key to
authenticate subflow
 Key exchange
SYN, [MyKey="keyABC"]
SYN+ACK, [MyKey="keyDEF"]
ACK[MyKey="keyABC", YourKey="keyDEF"]
MyKey="keyABC" MyKey="keyDEF"
YourKey="keyDEF" YourKey="keyABC"

SYN,[NonceA=123]
SYN+ACK[NonceB=456,
HMAC(123,"keyDEF")]
ACK,[HMAC(456,"keyABC")]
 Putting everything inside the SYN
• How can we place inside SYN segment ?

– Initial Data Sequence Number (64 bits)

– Token (32 bits)

– Authentication Key (the longer the better)

 Constraint on TCP options
Ver IHL ToS Total length • Total length of TCP
Identification Flags Frag. Offset
TTL Protocol Checksum
header : max 64 bytes
Source IP address
Destination IP address – max 44 bytes for TCP
Source port Destination port
options
Sequence number
Acknowledgment number – Options length must be
THL Reserved Flags Window multiple of 4 bytes
Checksum Urgent pointer
Options

Payload
 TCP options in the wild
• MSS option [4 bytes]
– Used only inside SYN segments
• Timestamp option [10 bytes] Only 20 bytes left
– Used in potentially all segments inside SYN !
• Window scale option [3 bytes]
– Used only inside SYN segments
• SACK permitted option [2 bytes]
– Used only inside SYN segments
• Selective Acknowledgements [N bytes]
– Used in data segments
http://www.iana.org/assignments/tcp-parameters/tcp-parameters.xml
 The MP_CAPABLE option
crypto
A: DSN Checksum
required or not B: Extension

Used inside SYN

Only in third ACK
 Initial Data Sequence Numbers
and Tokens
• Computation of initial Data Sequence Number
IDSNA=Lower64(SHA-1(KeyA))

IDSNB=Lower64(SHA-1(KeyB)) There is a small

risk of collision,
different keys
• Computation of token same token

TokenA=Upper32(SHA-1(KeyA))

TokenB=Upper32(SHA-1(KeyB))
 Cost of the Multipath TCP handshake

C. Raiciu, et al. “How hard can it be? designing and implementing a deployable multipath TCP,” NSDI'12:
Proceedings of the 9th USENIX conference on Networked Systems Design and Implementation, 2012.
 The Multipath TCP control plane
• Connection establishment in details

• Closing a Multipath TCP connection

• Address dynamics
Closing a Multipath TCP connection
• How to close a Multipath TCP connection ?
– By closing all subflows ?
RST

RST
Closing a Multipath TCP connection
seq=123, DSS[1->123 ...], "ab"
seq=125, DSS[5->125,DATA_FIN ...], "end"

DAck=9,ack=128
seq=128, FIN

ack=129

seq=456, DSS[3->456], "cd"

DAck=5,ack=458
Closing a Multipath TCP connection
• FAST Close

SYN, [...]
SYN+ACK, [...]
ACK[...]
ACK+FAST_CLOSE
RST
SYN,[...]
SYN+ACK[...]
ACK[..]

RST
 The Multipath TCP control plane
• Connection establishment in details

• Closing a Multipath TCP connection

• Address dynamics
 Multipath TCP
Address dynamics
• How to learn the addresses of a host ?
IP=2.3.4.5

IP=3.4.5.6
IP6=2a00:1450:400c:c05::69

• How to deal with address changes ?

IP=1.2.3.4

IP=4.5.6.7
 Address dynamics
• Basic solution : multihomed server
SYN, [...]

SYN+ACK, [...]
ACK[...]

ADD_ADDR[3.4.5.6] IP=2.3.4.5
IP=3.4.5.6
ADD_ADDR[2a00:1450:400c:c05::69
IP6=2a00:1450:400c:c05::69
]
SYN,[...]
SYN+ACK[...]
ACK[..]
 Address dynamics
• Basic solution : mobile client
SYN, [...]

SYN+ACK, [...]
ACK[...]

ADD_ADDR [4.5.6.7]
IP=2.3.4.5
SYN,[...]
IP=1.2.3.4
SYN+ACK[...]
IP=4.5.6.7
ACK[..]

REMOVE_ADDR[1.2.3.4]
 Address dynamics
in today's Internet

SYN, [...]

SYN+ACK, [...]
ACK[...]

ADD_ADDR [10.0.0.2]
ADD_ADDR [10.0.0.2]
IP=2.3.4.5
IP=1.2.3.4
IP=10.0.0.2 ? SYN [...]
 Address dynamics with NATs
• Solution

– Each address has one identifier

• Subflow is established between id=0 addresses
– Each host maintains a list of <address,id> pairs of
the addresses associated to an MPTCP endpoint
– MPTCP options refer to the address identifier
• ADD_ADDR contains <address,id>
• REMOVE_ADDR contains <id>
 Address dynamics

SYN, [...]

SYN+ACK, [...]
ACK[...]

ADD_ADDR [4.5.6.7,id=1]
IP=2.3.4.5
SYN,[id=1...]
IP=1.2.3.4
SYN+ACK[...]
IP=4.5.6.7
ACK[..]

REMOVE_ADDR[id=0]
 Agenda
• The motivations for Multipath TCP

• The changing Internet

• The Multipath TCP Protocol

• Multipath TCP use cases

– Datacenters
– Smartphones
 Datacenters evolve
• Traditional Topologies are tree-
based
– Poor performance
– Not fault tolerant
…
• Shift towards multipath
topologies: FatTree, BCube,
VL2,
Cisco, EC2

C. Raiciu, et al. “Improving datacenter performance and robustness with multipath TCP,” ACM
SIGCOMM 2011.
Fat Tree Topology [Fares et al., 2008;
Clos, 1953]
K=4
Aggregation
Switches

K Pods with
K Switches
each

Racks of
servers
C. Raiciu, et al. “Improving datacenter performance and robustness with multipath TCP,” ACM
SIGCOMM 2011.
TCP in data centers
 TCP in FAT tree networks
Cost of collissions

C. Raiciu, et al. “Improving datacenter performance and robustness with multipath TCP,” ACM
SIGCOMM 2011.
How to get rid of these collisions ?
• Consider TCP performance as an optimisation
problem
 The Multipath TCP way
ECMP balances
the subflows
over different
paths

Two subflows
differ by their
C. Raiciu, et al. “Improving datacenter performance and robustness with multipath TCP,” ACM
source SIGCOMM
port 2011.
MPTCP better utilizes the FatTree network

C. Raiciu, et al. “Improving datacenter performance and robustness with multipath TCP,” ACM
SIGCOMM 2011.
 Agenda
• The motivations for Multipath TCP

• The changing Internet

• The Multipath TCP Protocol

• Multipath TCP use cases

– Datacenters
– Smartphones
 Motivation
• One device, many IP-enabled interfaces
MPTCP over WiFi/3G

8Mbps, 20ms

2Mbps, 150ms
 TCP over WiFi/3G

C. Raiciu, et al. “How hard can it be? designing and implementing a deployable multipath TCP,” NSDI'12:
Proceedings of the 9th USENIX conference on Networked Systems Design and Implementation, 2012.
 MPTCP over WiFi/3G

C. Raiciu, et al. “How hard can it be? designing and implementing a deployable multipath TCP,” NSDI'12:
Proceedings of the 9th USENIX conference on Networked Systems Design and Implementation, 2012.
MPTCP over WiFi/3G

Multipath
TCP
increases
throughput
 MPTCP over WiFi/3G

What
happened
here?
Understanding the performance issue

Window full !
D C B No new data can be sent on WiFi path

8Mbps, 20ms

A 2Mbps, 150ms Window

Reinject segment on fast path

Halve congestion window on slow subflow
MPTCP over WiFi/3G
 Usage of 3G and WiFI
• How should Multipath TCP use 3G and WiFi ?

– Full mode
• Both wireless networks are used at the same time

– Backup mode
• Prefer WiFi when available, open subflows on 3G and use them as
backup

– Single path mode

• Only one path is used at a time, WiFi preferred over 3G
Multipath TCP : Full mode

SYN...

SYN+ACK...
ACK...

WiFi

3G/LTE
SYN...

SYN+ACK,...

ACK ...
Multipath TCP : Backup mode

SYN...

SYN+ACK...
ACK...

WiFi

3G/LTE
SYN,MP_JOIN[Backup...]
Dynamically
SYN+ACK,...
change backup
ACK ... status of flow
MP_PRIO[B]
 Multipath TCP : Backup mode
• What happens when link fails ?
SYN...

SYN+ACK...
ACK...

WiFi

3G/LTE
SYN,MP_JOIN[Backup...]

SYN+ACK,...
ACK ...
REM_ADDR[id=0]
 Multipath TCP : single-path mode
• Multipath TCP supports break before make
SYN...

SYN+ACK...
ACK...

WiFi

3G/LTE
SYN,MP_JOIN

SYN+ACK,...
ACK ...
REM_ADDR[id=0]
Evaluation scenario
WiFi:
Belgacom ADSL2+
(~8 Mbps, ~30 ms)

3G: Mobistar
(~2 Mbps, ~80ms)
 Recovery after failure

C. Paasch, et al. , “Exploring mobile/WiFi handover with multipath TCP,” presented at the CellNet '12: Proceedings
of the 2012 ACM SIGCOMM workshop on Cellular networks: operations, challenges, and future design, 2012.
 Recovery after failure

C. Paasch, et al. , “Exploring mobile/WiFi handover with multipath TCP,” presented at the CellNet '12: Proceedings
of the 2012 ACM SIGCOMM workshop on Cellular networks: operations, challenges, and future design, 2012.
 Recovery after failure

C. Paasch, et al. , “Exploring mobile/WiFi handover with multipath TCP,” presented at the CellNet '12: Proceedings
of the 2012 ACM SIGCOMM workshop on Cellular networks: operations, challenges, and future design, 2012.
 Conclusion
• Multipath TCP is becoming a reality
– Due to the middleboxes, the protocol is more
complex than initially expected
– RFC has been published
– there is running code !
– Multipath TCP works over today's Internet !
• What's next ?
– More use cases
• IPv4/IPv6, anycast, load balancing, deployment
– Measurements and improvements to the protocol
• Time to revisit 20+ years of heuristics added to TCP
 References
• The Multipath TCP protocol
– http://www.multipath-tcp.org
– http://tools.ietf.org/wg/mptcp/

A. Ford, C. Raiciu, M. Handley, S. Barre, and J. Iyengar, “Architectural guidelines for

multipath TCP development", RFC6182 2011.

A. Ford, C. Raiciu, M. J. Handley, and O. Bonaventure, “TCP Extensions for

Multipath Operation with Multiple Addresses,” RFC6824, 2013

C. Raiciu, C. Paasch, S. Barre, A. Ford, M. Honda, F. Duchene, O. Bonaventure, and

M. Handley, “How hard can it be? designing and implementing a deployable
multipath TCP,” NSDI'12: Proceedings of the 9th USENIX conference on Networked
Systems Design and Implementation, 2012.
 Implementations
• Linux
– http://www.multipath-tcp.org
S. Barre, C. Paasch, and O. Bonaventure, “Multipath tcp: From theory
to practice,” NETWORKING 2011, 2011.
Sébastien Barré. Implementation and assessment of Modern Host-
based Multipath Solutions. PhD thesis. UCL, 2011

• FreeBSD
– http://caia.swin.edu.au/urp/newtcp/mptcp/
• Simulators
– http://nrg.cs.ucl.ac.uk/mptcp/implementation.html
– http://code.google.com/p/mptcp-ns3/
 Middleboxes

M. Honda, Y. Nishida, C. Raiciu, A. Greenhalgh, M. Handley, and H.

Tokuda, “Is it still possible to extend TCP?,” IMC '11: Proceedings
of the 2011 ACM SIGCOMM conference on Internet measurement
conference, 2011.

V. Sekar, N. Egi, S. Ratnasamy, M. K. Reiter, and G. Shi, “Design and

implementation of a consolidated middlebox architecture,” USENIX NSDI,
2012.

J. Sherry, S. Hasan, C. Scott, A. Krishnamurthy, S. Ratnasamy, and V.

Sekar, “Making middleboxes someone else's problem: network
processing as a cloud service,” SIGCOMM '12: Proceedings of the ACM
SIGCOMM 2012 conference on Applications, technologies,
architectures, and protocols for computer communication, 2012.
 Multipath congestion control
– Background
D. Wischik, M. Handley, and M. B. Braun, “The resource pooling principle,”
ACM SIGCOMM Computer …, vol. 38, no. 5, 2008.
F. Kelly and T. Voice. Stability of end-to-end algorithms for joint
routing and rate control. ACM SIGCOMM CCR, 35, 2005.
P. Key, L. Massoulie, and P. D. Towsley, “Path Selection and Multipath
Congestion Control,” INFOCOM 2007. 2007, pp. 143–151.
– Coupled congestion control
C. Raiciu, M. J. Handley, and D. Wischik, “Coupled Congestion Control for
Multipath Transport Protocols,” RFC, vol. 6356, Oct. 2011.
D. Wischik, C. Raiciu, A. Greenhalgh, and M. Handley, “Design,
implementation and evaluation of congestion control for multipath TCP,”
NSDI'11: Proceedings of the 8th USENIX conference on Networked
systems design and implementation, 2011.
 Multipath congestion control
– More
R. Khalili, N. Gast, M. Popovic, U. Upadhyay, J.-Y. Le Boudec , MPTCP is not
Pareto-optimal: Performance issues and a possible solution, Proc. ACM
Conext 2012
Y. Cao, X. Mingwei, and X. Fu, “Delay-based Congestion Control for Multipath
TCP,” ICNP2012, 2012.
T. A. Le, C. S. Hong, and E.-N. Huh, “Coordinated TCP Westwood congestion
control for multiple paths over wireless networks,” ICOIN '12: Proceedings of the
The International Conference on Information Network 2012, 2012, pp. 92–96.
T. A. Le, H. Rim, and C. S. Hong, “A Multipath Cubic TCP Congestion Control
with Multipath Fast Recovery over High Bandwidth-Delay Product Networks,”
IEICE Transactions, 2012.
T. Dreibholz, M. Becke, J. Pulinthanath, and E. P. Rathgeb, “Applying TCP-Friendly
Congestion Control to Concurrent Multipath Transfer,” Advanced Information
Networking and Applications (AINA), 2010 24th IEEE International Conference on,
2010, pp. 312–319.
 Use cases
– Datacenter
C. Raiciu, S. Barre, C. Pluntke, A. Greenhalgh, D. Wischik, and M. J.
Handley, “Improving datacenter performance and robustness with
multipath TCP,” ACM SIGCOMM 2011.
G. Detal, Ch. Paasch, S. van der Linden, P. Mérindol, G. Avoine, O. Bonaventure,
Revisiting Flow-Based Load Balancing: Stateless Path Selection in Data Center
Networks, to appear in Computer Networks
– Mobile
C. Pluntke, L. Eggert, and N. Kiukkonen, “Saving mobile device energy with
multipath TCP,” MobiArch '11: Proceedings of the sixth international
workshop on MobiArch, 2011.
C. Paasch, G. Detal, F. Duchene, C. Raiciu, and O. Bonaventure, “Exploring
mobile/WiFi handover with multipath TCP,” CellNet '12: Proceedings of
the 2012 ACM SIGCOMM workshop on Cellular networks: operations,
challenges, and future design, 2012.