FNA Refresh

INSTALLATION GUIDE
May 2019

Copyrights and Trademarks

© 2019 Facebook, Inc. All rights reserved.
 Table of Contents
1. Hardware Inventory ........................................................................................................................................................................................ 3
2. Refresh Types Inventory ............................................................................................................................................................................... 4

2a. New Build Old Decom ................................................................................................................................................................................. 5

2b. Replace Servers .............................................................................................................................................................................................. 6
2c. Switch Swap ..................................................................................................................................................................................................... 7
1) Rack Drain/Un-install ............................................................................................................................................................................. 7

2) Rack the New Switch............................................................................................................................................................................. 7

3) Connect the Network Cables from the Servers to the Switch.......................................................................................... 7
4) Install Optics into Ports ....................................................................................................................................................................... 7
5) Connect the Switch to the Router ................................................................................................................................................. 8
6) Power on the Switch ............................................................................................................................................................................. 8
7) Verify Server Operation ...................................................................................................................................................................... 8
8) Check Optical Light Levels ................................................................................................................................................................. 8
3. Post Installation Checklist ............................................................................................................................................................................ 9
4. Network Configuration .............................................................................................................................................................................. 10
4a. Point to Point ................................................................................................................................................................................................ 11
4b. Border Gateway Protocol (BGP) ......................................................................................................................................................... 12
5. Troubleshooting ............................................................................................................................................................................................ 13
6. Media Storage Removal ............................................................................................................................................................................. 14
7. Communication ............................................................................................................................................................................................. 17

FNA: Refresh Guide | 2

1. Hardware Inventory
Ensure you have received all equipment requirement to conduct your refresh in accordance with the checklist
below. If you are missing any items, please contact Facebook via Network Partner Portal (NPP).
The table below is a reference for a 4-server deployment. Number of items will vary depending on deployment
size.

Quantity Item Dimension

4*** HP ProLiant DL380p Gen9v2 Server or 2RU
HP ProLiant DL380p Gen10 Server
1 Arista DCS-7060CX-32S-R-DC Switch* 1RU
4*** 10G SR/LR Optics N/A
2 100G SR/LR Optics** N/A
4*** DAC Cables 3-meters, 7-meters
4*** Network Cable 3-meters
4*** Rail Kit N/A
8*** Power Cord (C13/C14 AC) (DC Cord not provided) 2-meters

* Please note a switch may or may not be included in your shipment dependent upon your type of refresh
** Shipment of 100G optics is dependent upon FNA traffic levels
*** Number of servers and related items will vary depending on deployment size

FNA Pre-Installation Checklist

Item
n x 10 Gigabit (10Gb or 100gb) available ports on ISP switch or router1
n available SFP+ (or compatible) optics of same type – Long Range (LR) or Short
Range (SR)1
n available fiber patch cables (multi-mode or single mode)1
n available input power connections2
Power connections are divided evenly between two different power sources
Sufficient power (8kW) for each cluster
Sufficient rack space is available3
1
Where n = 2 for every FNA cluster deployed.
2
Where n = 2 times the number of devices (e.g., Cluster 1 is (5) devices, so (10) input power
connections are required).
3
For initial deployment (Cluster 1), 9RU (nine rack unit) is required. For additional capacity
(Cluster 2), 8RU is required.

FNA: Refresh Guide | 3

2. Refresh Types Inventory
There are three different options when it comes to conducting a Facebook network appliance hardware refresh.
Each option has specific requirements as well as a specific set of instructions. At this point, the refresh option has
already been agreed upon by both Facebook and the internet service provider.

Below you will find the refresh options (chapter 2, sections (a) and (b)), their requirements and instructions.
Additionally, a special case scenario of a switch swap is also covered in this chapter (section (c)). Lastly, all
hardware troubleshooting actions are covered in the FNA Operations Troubleshooting Guide location on
Network Partner Portal (NPP).

An overview of each refresh type is listed in Figure 2-1 below.

New Build, Old Decom Handled as a new cluster deployment

Replace servers, Including Switch Server(s) will be replaced while cluster is online, switch
replacement needs to occur while cluster is
offline/drained
Replace servers, Without Switch Servers will be replaced while cluster is online
Switch Replacement While cluster is offline/drained
Figure 2-1

Overall Requirements

1. Installation schedules and downtime must be communicated well ahead of time via NPP and email to all
concerned teams.
2. ISP is responsible for the arrangement of LOA’s for FB managed sites.
3. If setting up new physical space, rack and stack the new cluster(s) prior to draining the old cluster(s).
4. Please wait for the signal from FB to power down the old cluster(s).
5. The MSP will provide all instructions for the ERAD and COD processes.
6. For any issues or questions, please contact Facebook via the Network Partner Portal (NPP).

FNA: Refresh Guide | 4

2a. New Build Old Decom

This Hardware Refresh requires additional space and power at the ISP location. The new cluster will be placed
adjacent to the existing equipment. The legacy cluster will be decommissioned upon successful turn-up of the
newly installed cluster. The new hardware, cluster name, IP allocations, upstream ports from ISP, are all brought
up together in parallel to the legacy cluster it is replacing. Note that the new cluster will have a new name/ID and
new IP addresses, as stated in the diagram below.

Requirements

1. New Rack Space/Power/IPs

2. x days from the day the new rack has been powered up
3. y days from the day new cluster is in production
4. x = 4(best)-7(worst); y = 24 hours
5. For any issues or questions, please contact Facebook via the Network Partner Portal (NPP).

FNA: Refresh Guide | 5

2b. Replace Servers

As the name suggests, we will replace servers in batches and repeat until all the old equipment has been
completely replaced. New and old hardware, reuse IP space, reuse physical space. Replace servers or batch by
referring to old and new server serial mapping provided through NPP notification. The servers will be identified
for each phase of replacements by indication or flashing the LED lights.

Note: During the BGP transition, the cluster continues to serve from the cache without interruption to
your users. Your NOC may see BGP alarms.

Requirements

Once the gear lands, wait for notification from FB with detailed instructions and replace the servers in the first
batch. Look for old server to new server replacement mapping by label/serial numbers.

Please follow this instruction carefully to avoid any chances of duplicate IP addressing scenario:
1. Make sure the DAC cables from old servers are connected the new servers.
2. Wait for the LED signal from FB confirming the ready for next batch.
3. Repeat above until all batches have been replaced.
4. Timeline for each batch 3-4 hours.
5. Smart hands support to uninstall and reinstall servers in batches
6. Locate the serial number on new and old devices. Serial numbers can be located on the top of the server for
all generations, or on the sliding tab on the front of the server for G8 and G10 only. (Figures 2b 1 – 4)

7. The batch number will be identified for each phase of replacements. Once the gear lands, wait for notification
from FB and replace the servers in the first batch/server. Look for old server to new server replacement
mapping by label/serial numbers. Please follow this instruction carefully to avoid any chances of duplicate IP
addressing scenario.

8. For any issues or questions, please contact Facebook via the Network Partner Portal (NPP)

Figure 2b-1 Figure 2b-2

Figure 2b-3 Figure 2b-4

FNA: Refresh Guide | 6

2c. Switch Swap
Before beginning, make sure you have received all items referenced in Chapter 1 of this guide.

1) Rack Drain/Un-install
a) The Facebook deployment engineer will schedule and coordinate the drain of the old cluster.
b) The Facebook deployment engineer will confirm that the drain has been completed and instruct to
progress with the un-installation of the old hardware.
c) Uninstall the old hardware.

2) Rack the New Switch

a) Install the switch into the rack with the airflow as port side exhaust (switch ports must face the rear of the
rack as shown in Figure 2c-1).

Figure 2c-1 Figure 2c-2

3) Connect the Network Cables from the Servers to the Switch

a) Connect the network cable (green sticker facing up) to the Network Interface Controller (NIC) port as
seen in Figure 2c-1 and 2c-2.
b) Connect a server at the highest position in the rack (fna001) to QSFP+ port 32 on the switch. Utilize the
illustration as reference for port connectivity for the remaining servers (four are illustrated in Figure 2c-1)

4) Install Optics into Ports

a) Install QSFP-SFP adapters, optical transceivers and QSFP optics into ports 1, 2, 3 and 4 as shown in Figure
2c-1.
b) For Clusters over four servers, follow the port assignments illustrated in Figure 2c-3.

Figure 2c-3

FNA: Refresh Guide | 7

5) Connect the Switch to the Router
a) Connect the fiber patch cables from the switch to the router as shown in Figure 2c-1.

6) Power on the Switch

a) Connect two power cables to the network switch as shown in Figure 2c-4 (the switch powers on when
connected to power).
b) The switch’s system status LED displays as blinking green while powering up, then steady green for normal
and amber indicates a fan disconnect or malfunction. Port LEDs display green for up, yellow for software
disabled and flashing yellow for failed diagnostics.
c) For any issues or questions, please contact Facebook via the Network Partner Portal (NPP).

Figure 2c-4 Figure 2c-5

7) Verify Server Operation

a) Verify server operation by checking the behavior of the LEDs as illustrated on each server.

Figure 2c-6

8) Check Optical Light Levels

a) Check the optical light levels to ensure the cluster has adequate signal stretch. Light levels need to be
between -2 dB and -7 dB.

FNA: Refresh Guide | 8

3. Post Installation Checklist
Please complete this checklist after you have installed your FNA to verify proper installation.

FNA Post Installation Checklist

Item
All ProLiant DL 380 servers in the FNA node are powered on.
Each power supply of each ProLiant DL380 server indicates an ON, Steady Green
state.
The Life (L) and Status (S) LEDs for each SSD in each ProLiant DL380 server
indicate an ON, Steady Green state.
All accompanying LEDs for the connected fiber uplink SFP+ ports are Green.
The network cables are seated properly to each ProLiant DL380 server.
The Port 2 (P2) Link LED (LNK) on each NIC in the FNA node indicates an ON,
Steady Green state.
Light level readings for all uplinks are between -2dB and -7dB.

If there are any issues during the verification of your FNA installation, please refer to the troubleshooting section
of this guide or contact Facebook via the Network Partner Portal (NPP).

FNA: Refresh Guide | 9

4. Network Configuration
This chapter outlines Point to Point (P2P) and Border Gateway Protocol (BGP) network configuration in sections
(a) and (b). Figure 4-1 below is a high-level FNA system overview. Figure 4-2a and 4-2b below is the basic network
of FNA.

Figure 4-1

Figure 4-2a

FNA: Refresh Guide | 10

 Figure 4-2b

4a. Point to Point

The point-to-point connection is configured on the Link-Aggregation Control Protocol (LACP) interface. Make
sure the following parameters are configured as follows:

1. Set a static route to the allocated subnets via the link-aggregation interface.
2. Set the appropriate maximum transmission unit (MTU) for your router software:
3. For Cisco IOS XR router software, set the MTU to 1514. Cisco IOS SR software includes both the L2 and L3
overhead in the interface MTU command.
4. For all other router software, set the MTU to 1500.

The FNA must have internet connectivity before BGP peering can be established. The BGP peering session does
not need to be routing before the FNA has internet connectivity.
The FNA kit is preconfigured with the IP addresses that were specified in the order. If IP addresses need to be
changed at any time, refer to Section (c) of chapter.

For any issues or questions, please contact Facebook via the Network Partner Portal (NPP).

FNA: Refresh Guide | 11

4b. Border Gateway Protocol (BGP)

1. Use the IPv4 and IPv6 addresses originally indicated in the FNA order form.
2. (Optional) Enable Graceful Restart (GR). Enabling GR is not required, but it is recommended.
3. Enable External BGP (EGBP) Multihop.
4. Use the peering Autonomous System Number (ASN): 63293
5. Configure your router’s BGP settings.
6. FNA’s prefix should be announced to all Facebook peering connections.
7. All servable Facebook netblocks should be announced for network traffic optimization (completed by the
latency-based routing sampling).
Routes advertised to the FNA must be present in the global routing table with the same destination Autonomous
System (AS). FNA applies inbound filtering to reject the following address spaces: default route, bogons, FC 1918
Facebook’s IP, CGNAT, and prefixes smaller than /30 for IPv4 and smaller than /64 for IPv6.

After the BGP settings are saved, the FNA cluster will begin downloading application data. Within about two
business days, BGP peering will be established:

8. The connection will be coming from the 16th IP address in each allocated subnet (192.168.1.16,
2001:db8::10).
9. FNA will not advertise any routes / prefixes.

For any issues or questions, please contact Facebook via the Network Partner Portal (NPP).

FNA: Refresh Guide | 12

5. Troubleshooting
This chapter outlines Point to Point (P2P) and Border Gateway Protocol (BGP) network configuration in sections
(a) and (b). Figure 4-1 below is

Hardware Troubleshooting
For all hardware troubleshooting, please refer to the FNA Operations Troubleshooting Guide available on
NPP.

Connectivity Troubleshooting
If any issues are encountered during the creation of the USB boot disk or IP configuration, see the following
methods for general troubleshooting.

Operating System Boot Issues

1. Take a picture of the screen where the error has occurred.
2. Send this image with brief description of the issue to FNA Operations via email to
[email protected]

Network Connectivity Test

If a network connectivity test is requested after you contact FNA Operations, follow these steps:
1. Ping the default gateway IP address.
2. If the ping was successful (minimal pack loss), ping any known working address on the Internet (for example,
Facebook.com).
3. If the ping was unsuccessful, troubleshoot for network connectivity (per next method).

Network Connectivity Checks

1. Ensure all equipment in the uplink/downlink path is powered on.
2. Ensure that the network cables between each server in the FNA cluster are properly connected to the FNA
cluster switch.
3. Ensure that the SFP+ optical transceivers are properly seated in the switch uplink ports.
4. Ensure that the optical cables in the path between the FNA cluster and the facility router have a good
connection.
5. Ensure that the signal strength (light level) for the FNA cluster is between -2 and -7 dB.

Switch Configuration Issues

1. If the switch has not been configured, press [Control+C] to end the current flow.
2. Take a picture of the output in this log: cat /var/log/nginx/access.log
3. Send the pictures with a brief description of the issues to FNA operations.

For any issues or questions, please contact Facebook via the Network Partner Portal (NPP).

FNA: Refresh Guide | 13

6. Media Storage Removal
This chapter entails a step by step process for the un-installation of the legacy hardware. The primary focus of
these instructions is the proper removal and stowage of all storage media devices for the purposes of data
Eradication (ERAD) and Certificate of Destruction (COD).

1. To remove the hard drive from each server you will need to locate the red button on each server. It is located
on the left side of the server (Figure 6-1). Depress the red button. The hard drive door will open. Remove the
hard drive (Figure 6-2). It is very important to maintain each hard drive with their respective cluster
identification in a secure location prior to ERAD and COD. (Repeat step 1 for each server)

Figure 6-1 Figure 6-2

2. To remove the flash cards from each server, you will need to slide each server out of its respective cabinet. To
slide the server out of the rack, pull down tabs on the bottom of each server, there are two of them, one on
each side of the server (Figure 6-3 & 6-4). You should see a screw with a star shaped pattern when you pull
down on the tabs. Unscrew both screws in the counter clockwise direction. Should now be able to slide the
server out of the rack.

Note: A slow counter-clockwise rotation with a Philips screwdriver will unscrew if you don’t have the correct
star shaped bit.

Figure 6-3 Figure 6-4

3. Slide the server out of the cabinet. The server will catch when it is completely pulled out of the cabinet.
Locate the black tab on the top of server, toward the rear, (Figure 6-5). Lift the black tab and remove the top
cover of the server.

Figure 6-5

FNA: Refresh Guide | 14

4. Once the top cover is removed. Locate the flashcards (there are two) as show in figures below.

Figure 6-6 Figure 6-7

5. To remove the flash card(s), there are two rotating-spring loaded pins that will need to be unlocked. There
are two pins per flash card. These pins are located on the on the left side of each flashcard as shown in Figure
6-8. Lift the tab on the top of the pin and unscrew. The pins only rotate about a quarter turn to loosen.

Figure 6-8

6. Once the pins have been loosened. Pull out the flash card holder (Figure 6-9).

Figure 6-9

With the flashcard holder removed, there is a small screw holding the card in place as shown in Figure 6-10.
The screw has the similar star pattern as the screws that are on the front of each server. Use the same
method to unscrew the flashcards from the holder. Remove the flashcards (Figure 6-11). Maintain flashcards
with their respective server and cluster identification numbers for the purposes of ERAD and COD. (Repeat
steps 3-6 for each server)

Figure 6-10 Figure 6-11

FNA: Refresh Guide | 15

7. Once all media devices are removed, each server can be removed from their respective cabinet and made
ready for shipment. The MSP will provide shipping supplies upon arrival for the shipment of the legacy
equipment.

For any issues or questions, please contact Facebook via the Network Partner Portal (NPP).

FNA: Refresh Guide | 16

7. Communication
This chapter provides communication guidance for any and all questions, issues, troubleshooting and escalation
during the installation and turn up.

1. For all issues and questions at any point during the process, please contact Facebook via the Network
Partner Portal (NPP).
2. For all escalations during the process, please contact Facebook Network Appliance Operations and
Deployment via email at [email protected].

FNA: Refresh Guide | 17