Peering, Transit and IXP

Design
Philip Smith
PacNOG 12
Port Moresby
10th – 14th December 2012

1
The Internet
p  Internet is made up of ISPs of all shapes and
sizes
n  Some have local coverage (access providers)
n  Others can provide regional or per country coverage
n  And others are global in scale
p  These ISPs interconnect their businesses
n  They don’t interconnect with every other ISP (over
43000 distinct autonomous networks) – won’t scale
n  They interconnect according to practical and business
needs
p  Some ISPs provide transit to others
n  They interconnect other ISP networks
2
 Categorising ISPs
Global ISP Global ISP

$
$
$ Global ISP Global ISP
$
$
$
$ Regional ISP Regional ISP
$
Regional ISP Regional ISP
$
$
$ IXP IXP
$ Access ISP Access ISP Access ISP Access ISP
$
$
Access ISP Access ISP
$ 3
Peering and Transit
p  Transit
n  Carrying traffic across a network
n  Usually for a fee
n  Example: Access provider connects to a
regional provider
p  Peering
n  Exchanging routing information and traffic
n  Usually for no fee
n  Sometimes called settlement free peering
n  Example: Regional provider connects to
another regional provider
4
Private Interconnect
p  TwoISPs connect their networks over a
private link
n  Can be peering arrangement
p  No charge for traffic
p  Share cost of the link

n  Can be transit arrangement

ISP 1 ISP 2

5
Public Interconnect
p  SeveralISPs meeting in a common neutral
location and interconnect their networks
n  Usually is a peering arrangement between
their networks
ISP 1 ISP 2

ISP 6 ISP 3
IXP

ISP 5 ISP 4

6
ISP Goals
p  Minimise the cost of operating the business
p  Transit
n  ISP has to pay for circuit (international or domestic)
n  ISP has to pay for data (usually per Mbps)
n  Repeat for each transit provider
n  Significant cost of being a service provider
p  Peering
n  ISP shares circuit cost with peer (private) or runs circuit
to public peering point (one off cost)
n  No need to pay for data
n  Reduces transit data volume, therefore reducing cost

7
Transit – How it works
p  Small access provider provides Internet access
for a city’s population
n  Mixture of dial up, wireless and fixed broadband
n  Possibly some business customers
n  Possibly also some Internet cafes
p  How do their customers get access to the rest of
the Internet?
p  ISP buys access from one, two or more larger
ISPs who already have visibility of the rest of the
Internet
n  This is transit – they pay for the physical connection to
the upstream and for the traffic volume on the link
8
Peering – How it works
p  If two ISPs are of equivalent sizes, they have:
n  Equivalent network infrastructure coverage
n  Equivalent customer size
n  Similar content volumes to be shared with the Internet
n  Potentially similar traffic flows to each other’s networks
p  This makes them good peering partners
p  If they don’t peer
n  They both have to pay an upstream provider for access
to each other’s network/customers/content
n  Upstream benefits from this arrangement, the two ISPs
both have to fund the transit costs

9
The IXP’s role
p  Private
peering makes sense when there
are very few equivalent players
n  Connecting to one other ISP costs X
n  Connecting to two other ISPs costs 2 times X
n  Connecting to three other ISPs costs 3 times X
n  Etc… (where X is half the circuit cost plus a
port cost)
p  The more private peers, the greater the
cost
p  IXP is a more scalable solution to this
problem
10
The IXP’s role
p  Connecting to an IXP
n  ISP costs: one router port, one circuit, and one router to
locate at the IXP
p  Some IXPs charge annual “maintenance fees”
n  The maintenance fee has potential to significantly
influence the cost balance for an ISP
p  Generally connecting to an IXP and peering there
becomes cost effective when there are at least
three other peers
n  The real $ amount varies from region to region, IXP to
IXP

11
Who peers at an IXP?
p  Access Providers
n  Don’t have to pay their regional provider transit fees for
local traffic
n  Keeps latency for local traffic low
n  ‘Unlimited’ bandwidth through the IXP (compared with
costly and limited bandwidth through transit provider)
p  Regional Providers
n  Don’t have to pay their global provider transit for local
and regional traffic
n  Keeps latency for local and regional traffic low
n  ‘Unlimited’ bandwidth through the IXP (compared with
costly and limited bandwidth through global provider)

12
The IXP’s role
p  Global Providers can be located close to IXPs
n  Attracted by the potential transit business available
p  Advantageous for access & regional providers
n  They can peer with other similar providers at the IXP
n  And in the same facility pay for transit to their regional
or global provider
n  (Not across the IXP fabric, but a separate connection)

IXP

Transit

Access
13
Connectivity Decisions
p  Transit
n  Almost every ISP needs transit to reach rest of Internet
n  One provider = no redundancy
n  Two providers: ideal for traffic engineering as well as
redundancy
n  Three providers = better redundancy, traffic engineering
gets harder
n  More then three = diminishing returns, rapidly
escalating costs and complexity
p  Peering
n  Means low (or zero) cost access to another network
n  Private or Public Peering (or both)

14
Transit Goals
1.  Minimise number of transit providers
n  But maintain redundancy
n  2 is ideal, 4 or more is bad
2.  Aggregate capacity to transit providers
n  More aggregated capacity means better value
p  Lower cost per Mbps
n  4x 45Mbps circuits to 4 different ISPs will
almost always cost more than 2x 155Mbps
circuits to 2 different ISPs
p  Yet bandwidth of latter (310Mbps) is greater than
that of former (180Mbps) and is much easier to
operate 15
Peering or Transit?
p  How to choose?
p  Or do both?
p  It comes down to cost of going to an IXP
n  Free peering
n  Paying for transit from an ISP co-located in
same facility, or perhaps close by
p  Or
not going to an IXP and paying for the
cost of transit directly to an upstream
provider
n  There is no right or wrong answer, someone
has to do the arithmetic
16
Private or Public Peering
p  Private peering
n  Scaling issue, with costs, number of providers, and
infrastructure provisioning
p  Public peering
n  Makes sense the more potential peers there are (more is
usually greater than “two”)
p  Which public peering point?
n  Local Internet Exchange Point: great for local traffic and
local peers
n  Regional Internet Exchange Point: great for meeting
peers outside the locality, might be cheaper than paying
transit to reach the same consumer base

17
Local Internet Exchange Point
p  Defined as a public peering point serving
the local Internet industry
p  Local means where it becomes cheaper to
interconnect with other ISPs at a common
location than it is to pay transit to another
ISP to reach the same consumer base
n  Local can mean different things in different
regions!

18
Regional Internet Exchange Point
p  These are also “local” Internet Exchange Points
p  But also attract regional ISPs and ISPs from
outside the locality
n  Regional ISPs peer with each other
n  And show up at several of these Regional IXPs
p  Local ISPs peer with ISPs from outside the
locality
n  They don’t compete in each other’s markets
n  Local ISPs don’t have to pay transit costs
n  ISPs from outside the locality don’t have to pay transit
costs
n  Quite often ISPs of disparate sizes and influences will
happily peer – to defray transit costs
19
Which IXP?
p  How many routes are available?
n  What is traffic to & from these destinations, and by how
much will it reduce cost of transit?
p  What is the cost of co-lo space?
n  If prohibitive or space not available, pointless choosing
this IXP
p  What is the cost of running a circuit to the
location?
n  If prohibitive or competitive with transit costs, pointless
choosing this IXP
p  What is the cost of remote hands/assistance?
n  If no remote hands, doing maintenance is challenging
and potentially costly with a serious outage

20
Internet Exchange Point
p  Solution
n  Every ISP participates in the IXP
n  Cost is minimal – one local circuit covers all domestic
traffic
n  International circuits are used for just international
traffic – and backing up domestic links in case the IXP
fails
p  Result:
n  Local traffic stays local
n  QoS considerations for local traffic is not an issue
n  RTTs are typically sub 10ms
n  Customers enjoy the Internet experience
n  Local Internet economy grows rapidly

21
Exchange Point Design

22
IXP Design
p  Very simple concept:
n  Ethernet switch is the interconnection media
p  IXP is one LAN
n  Each ISP brings a router, connects it to the
ethernet switch provided at the IXP
n  Each ISP peers with other participants at the
IXP using BGP
p  Scaling
this simple concept is the
challenge for the larger IXPs

23
 Layer 2 Exchange
ISP 6 ISP 5 ISP 4

IXP Services:
IXP
Root & TLD DNS, Management
Routing Registry Network
Ethernet Switch
Looking Glass, etc

ISP 1 ISP 3
ISP 2

24
 Layer 2 Exchange
ISP 6 ISP 5 ISP 4

IXP Services:
IXP
Root & TLD DNS, Management
Routing Registry Network
Looking Glass, etc Ethernet Switches

ISP 1 ISP 3
ISP 2

25
Layer 2 Exchange
p  Two switches for redundancy
p  ISPs use dual routers for redundancy or
loadsharing
p  Offer services for the “common good”
n  Internet portals and search engines
n  DNS Root & TLD, NTP servers
n  Routing Registry and Looking Glass

26
Layer 2 Exchange
p  Requires neutral IXP management
n  Usually funded equally by IXP participants
n  24x7 cover, support, value add services

p  Secure and neutral location

27
Layer 2 Exchange
p  Network Security Considerations
n  LAN switch needs to be securely configured
n  Management routers require TACACS+
authentication, vty security
n  IXP services must be behind router(s) with
strong filters

28
“Layer 3 IXP”
p  Layer 3 IXP is marketing concept used by
Transit ISPs
p  Real Internet Exchange Points are only
Layer 2

29
 IXP Design
Considerations

30
Exchange Point Design
p  The IXP Core is an Ethernet switch
n  It must be a managed switch
p  Has
superseded all other types of network
devices for an IXP
n  From the cheapest and smallest managed 12
or 24 port 10/100 switch
n  To the largest switches now handling high
densities of 10GE and 100GE interfaces

31
Exchange Point Design
p  Each ISP participating in the IXP brings a
router to the IXP location
p  Router needs:
n  One Ethernet port to connect to IXP switch
n  One WAN port to connect to the WAN media
leading back to the ISP backbone
n  To be able to run BGP

32
Exchange Point Design
p  IXP
switch located in one equipment rack
dedicated to IXP
n  Also includes other IXP operational equipment
p  Routers from participant ISPs located in
neighbouring/adjacent rack(s)
p  Copper (UTP) connections made for
10Mbps, 100Mbps or 1Gbps connections
p  Fibre used for 1Gbps, 10Gbps, 40Gbps or
100Gbps connections

33
Peering
p  Each participant needs to run BGP
n  They need their own AS number
n  Public ASN, NOT private ASN

p  Eachparticipant configures external BGP

34
Peering (more)
p  Mandatory Multi-Lateral Peering (MMLP)
n  Each participant is forced to peer with every other
participant as part of their IXP membership
n  Has no history of success — the practice is strongly
discouraged
p  Multi-Lateral Peering (MLP)
n  Each participant peers with every other participant
(usually via a Route Server)
p  Bi-Lateral Peering
n  Participants set up peering with each other according to
their own requirements and business relationships
n  This is the most common situation at IXPs today

35
Routing
p  ISP border routers at the IXP must NOT be
configured with a default route or carry the full
Internet routing table
n  Carrying default or full table means that this router and
the ISP network is open to abuse by non-peering IXP
members
n  Correct configuration is only to carry routes offered to
IXP peers on the IXP peering router
p  Note: Some ISPs offer transit across IX fabrics
n  They do so at their own risk – see above

36
Routing (more)
p  ISP
border routers at the IXP should not
be configured to carry the IXP LAN
network within the IGP or iBGP
n  Use next-hop-self BGP concept
p  Don’t
generate ISP prefix aggregates on
IXP peering router
n  If connection from backbone to IXP router goes
down, normal BGP failover will then be
successful

37
Address Space
p  Some IXPs use private addresses for the IX LAN
n  Public address space means IXP network could be leaked
to Internet which may be undesirable
n  Because most ISPs filter RFC1918 address space, this
avoids the problem
p  Some IXPs use public addresses for the IX LAN
n  Address space available from the RIRs
n  IXP terms of participation often forbid the IX LAN to be
carried in the ISP member backbone

38
Charging
p  IXPs should be run at minimal cost to participants
p  Examples:
n  Datacentre hosts IX for free
p  Because ISP participants then use data centre for co-lo
services, and the datacentre benefits long term
n  IX operates cost recovery
p  Each member pays a flat fee towards the cost of the
switch, hosting, power & management
n  Different pricing for different ports
p  One slot may handle 24 10GE ports
p  Or one slot may handle 96 1GE ports
p  96 port 1GE card is tenth price of 24 port 10GE card
p  Relative port cost is passed on to participants
39
Services Offered
p  Services
offered should not compete with
member ISPs (basic IXP)
n  e.g. web hosting at an IXP is a bad idea unless
all members agree to it
p  IXP
operations should make performance
and throughput statistics available to
members
n  Use tools such as MRTG/Cacti to produce IX
throughput graphs for member (or public)
information

40
Services to Offer
p  ccTLD DNS
n  the country IXP could host the country’s top level DNS
n  e.g. “SE.” TLD is hosted at Netnod IXes in Sweden
n  Offer back up of other country ccTLD DNS
p  Root server
n  Anycast instances of I.root-servers.net, F.root-
servers.net etc are present at many IXes
p  Usenet News
n  Usenet News is high volume
n  could save bandwidth to all IXP members

41
Services to Offer
p  Route Collector
n  Route collector shows the reachability
information available at the exchange
p  Looking Glass
n  One way of making the Route Collector routes
available for global view (e.g.
www.traceroute.org)
n  Public or members only access
n  Useful for members to check BGP filters
n  Useful for everyone to check route availability
at the IX
42
Services to Offer
p  Route Server
n  A Route Collector that also sends the prefixes
it has collected to its peers
n  Like a Route Collector, usually a router or Unix
based system running BGP
n  Does not forward packets
n  Useful for scaling eBGP sessions for larger IXPs
n  Participation needs to be optional
p  And will be used by ISPs who have open peering
policies

43
Services to Offer
p  Content Redistribution/Caching
n  For example, Akamised update distribution
service
p  Network Time Protocol
n  Locate a stratum 1 time source (GPS receiver,
atomic clock, etc) at IXP
p  Routing Registry
n  Used to register the routing policy of the IXP
membership

44
What can go wrong?
p  High annual fees
n  Should be cost recovery
p  Charging for traffic between participants
n  Competes with commercial transit services
p  Competing IXPs
n  Too expensive for ISPs to connect to all
p  Too many rules & restrictions
n  Want all network operators to participate
p  Mandatory Multi-Lateral Peering
n  Has no history of success
p  Interconnected IXPs
n  Who pays for the interconnection? 45

p  Etc…
Conclusion
p  IXPs are technically very simple to set up
p  Little more than:
n  An ethernet switch
n  Neutral secure reliable location
n  Consortium of members to operate it

p  Political aspects can be more challenging:

46