PLX 8796 3U Server Intel Open-Compute-Specification - 190718
PLX 8796 3U Server Intel Open-Compute-Specification - 190718
Rev 1.0
Contributions to this Specification are made under the terms and conditions set forth in Open Compute
Project Contribution License Agreement (“OCP CLA”) (“Contribution License”) by: ZT Group Int’l, Inc,
dba ZT Systems
You can review the signed copies of the applicable Contributor License(s) for this Specification on the
OCP website at http://www.opencompute.org/products/specsanddesign
Usage of this Specification is governed by the terms and conditions set forth in Open Compute Project
Hardware License – Permissive (“OCPHL Permissive”), (“Specification License”).
You can review the applicable Specification License(s) executed by the above referenced contributors to
this Specification on the OCP website at http://www.opencompute.org/participate/legal-documents/
Note: The following clarifications, which distinguish technology licensed in the Contribution License
and/or Specification License from those technologies merely referenced (but not licensed), were accepted
by the Incubation Committee of the OCP:
License .................................................................................................................................. 2
Scope .................................................................................................................................... 8
Overview............................................................................................................................... 9
2.2 Motherboard...................................................................................................................... 10
Table of Figures
Figure 1: Motherboard Block Diagram ....................................................................................................... 12
Figure 3: XPO200 3UN PCIe Bus Connectivity for 5-Slot Active & 4-Slot Active Riser Config ................. 14
Figure 7: 4-Slot Active Riser PCIe Port Mapping and Routing Layers (View from Backside) ................... 17
Figure 12: I2C Block Diagram for 4-Slot Active PCIe Riser ....................................................................... 23
Figure 13: Clock Block Diagram for 4-Slot Active PCIe Riser ................................................................... 24
Figure 14: 5-Slot Active Riser Layout (Top and Backside) ........................................................................ 25
Figure 16: 5-Slot Active Riser PCIe Port Mapping and Routing Layers (View from Backside) ................. 27
Figure 17: 50 pin x8 Sliver Connector to PCIe x8 Edge Connector Cable (Slot #1) ................................. 28
Figure 19: I2C Block Diagram for 5-Slot Active PCIe Riser ....................................................................... 31
Figure 20: Clock Block Diagram for 5-Slot Active PCIe Riser ................................................................... 32
Figure 21: I2C Block Diagram for 13 Single-Wide PCIe Cards ................................................................. 34
Figure 22: 3U Chassis System Fan Position Diagram with Closed Fan Flappers..................................... 36
Figure 25: 3U Chassis Cross Section Showing Middle Riser Baffle (to block airflow) .............................. 38
Figure 31: 3U Chassis/System with Split Top Cover King Slide Rails ....................................................... 43
Figure 33: 3U Chassis/System in 19” Project Olympus Rack (Extended 67%) ......................................... 45
Figure 36: XPO200 3UN 3U Server Power Supply Load Sharing Diagram (Max Supported Power) ....... 49
Figure 38: Split Cable Power Harness for 300W High Power PCIe Card (Example) ................................ 52
Table 8: 50 pin x8 Sliver Connector to PCIe x8 Edge Connector Cable Wire Mapping ............................. 29
Table 10: I2C Bus Mapping for 13 Single-Wide PCIe Cards ...................................................................... 34
This XPO200 3UN 3U Server implementation is compatible with existing Project Olympus building blocks.
Other Project Olympus elements which are utilized in this system are outlined in their respective
specifications. These elements include, but are not limited to, the Power Supply Unit (PSU), Power
Management Distribution Unit (PMDU), Universal Motherboard, Server Rack, and Rack Manager (RM).
Specifications for the overall Project Olympus Rack are posted here…
http://www.opencompute.org/wiki/Server/ProjectOlympus
Overview
This specification defines a Project Olympus based 2 Socket, 3U Server (XPO200 3UN) that is intended
to support multiple single-wide and double-wide PCIe Cards. This server design is based on the Project
Olympus 3U Server Base Specifications and fits within the Project Olympus OCP framework. The OCP
Approved Mount Olympus Motherboard based on the Intel Purley platform can be used in this system
along with various PCIe Riser Options to provide an IO rich platform for up to 6 x Double-wide Full-Length
Cards plus 1 x Single-wide Half-Length Card or up to 12 x Single-wide Full-Length Cards plus 1 x Single-
wide Half-Length Card. Due to the increased power requirements in this XPO200 3UN Server, three of
the Project Olympus 3-phase 1kW Power Supply Silver Boxes will also be used in this system. Each
Silver Box includes 3 internal Power Supply Units (PSUs) providing a total of 9 PSUs in an 8 + 1
redundancy scheme. Each Project Olympus 1kW Power Supply will be stacked vertically in the 3U
Chassis allowing use of the Project Olympus vertical Power Managed Distribution Unit (PMDU).
1. Rack Compatibility
This XPO200 3UN PCIe Expansion Server is compatible with the Project Olympus 19” EIA310-D Rack.
Due to the Server’s weight, it requires custom slide rails, which allow for smoother travel during
Serviceability when sliding in and out of the Rack.
2. System Requirements
The system requires a Mount Olympus Intel Purley generation Motherboard, which can be combined with
several different PCIe Riser Options for system configuration flexibility. It includes a Power Distribution
Board (PDB) to allow load sharing of the 3 Project Olympus 1kW PSUs. These components reside within
a 3U chassis and are compatible with the Project Olympus ecosystem.
2.2 Motherboard
The motherboard used in the XPO200 3UN System is the Mt. Olympus Universal Motherboard (MB),
which is the computational element in the Project Olympus Server. This is a Dual Socket Purley
generation server board defined by Microsoft.
NOTE: The information in this section is for reference only. For the latest detailed Spec information
please refer to the Mt Olympus MB Spec posted here…
http://www.opencompute.org/wiki/Server/ProjectOlympus
• Processor:
o 2 Socket Spread Core Design using Xeon (Skylake-SP/Cascade Lake-SP) Processors
o Supports up to 205W TDP
o Includes 1U Remote Heatsink
• Memory: 24 DDR4 DIMMs, 2 DIMMs per Channel
• PCH: Lewisburg
• PCIe Slots/Connectors:
o PCIe x8 Connector (Slot #1) – CPU0
o PCIe x8 Connector (Slot #2) – CPU0
o PCIe x16 Riser Connector (Slot #3) – CPU0
o PCIe x16 Riser Connector (Slot #4) – CPU1
o PCIe x16 Riser Connector (Slot #5) – CPU1
o PCIe x4 M.2 Connector (M.2 #1) – PCH
o PCIe x4 M.2 Connector (M.2 #2) – PCH
o PCIe x4 M.2 Connector (M.2 #3) – CPU1
o PCIe x4 M.2 Connector (M.2 #4) – CPU1
o PCIe x8 OCuLink Connector – CPU1
• SATA Connectors:
o 4 x SATA 7-pin Connectors (SATA[3:0]) – PCH
o 2 x SATA MiniSAS HD Connectors (SATA[7:4]/PCIe[15:12] & SATA[11:8]/PCIe[19:16]) –
PCH
• BMC: ASPEED AST1250
o All PCIe Slots are connected to BMC I2C Buses for PCIe Card telemetry
o I2C MUXes are used to avoid I2C Address contention
• Security: SPI TPM2.0 Module
• Front IO Ports:
o 1 x BMC Dedicated Management NIC Port
o 2 x USB3.0 Ports
o Power Button (Pre-Production Only)
o Reset Button (Pre-Production Only)
o 1 x Video Port (EMPTY)
o 1 x 10GbE SFP+ Connector (EMPTY)
Figure 3: XPO200 3UN PCIe Bus Connectivity for 5-Slot Active & 4-Slot Active Riser Config
12.68”
2x6 Pwr
74pin
Sliver #2
Backside PEX8796 50pin
Sliver #1
Block Diagram
74pin Sliver
SDA/SCL x8
3.3V (PCA9617) 1.8V
Conn#2
SDA/SCL PCIe G3 Switch Sliver
I2C CLK P8
MUX P0
(PEX8796) Conn#2
(PCA9548) (upper x8)
P4 P12 P20 P16
x8
50pin Sliver Cable
50pin Sliver
x8
PCIe x16 Riser Gold Fingers
Sliver
Conn#1
FRU x16 x16 x16 x16
(AT24C) Conn#1
(lower x8)
Temp
PCIe Switch VRs
PWR_BRK# 3.3V_stby
3.3V VR
VR
TX: L3/BOT
Slot 4 RX: L8 Port20 Port16
TX: L3/BOT
Slot 3 RX: L8 Port12 Port8
PEX8796
TX: L3/BOT
Slot 2 RX: L8 Port4 Port0
TX: TOP
RX: L3/BOT
Slot 1 TX: L3/BOT
RX: L8
TX: TOP
RX: BOT
Figure 7: 4-Slot Active Riser PCIe Port Mapping and Routing Layers (View from Backside)
o Switch Bifurcated into 5 x16 Ports (1 x16 Port to each x16 Slot and 1 x16 Port to Sliver
conns)
o Depending on the PCIe Card requirement, the Switch can be bifurcated into different lane
width configurations. Custom Switch Firmware (FW) is required to change the Switch
bifurcation.
o Switch is located on Backside as required by Chassis Thermals
o PCIe Switch Thermal Design Power Requirements:
▪ 2 x Double-Wide Card Config can use up to 64-lanes and requires 32.5W
▪ 4 x Single-Wide Card Config can use up to 96-lanes and requires 35.8W
▪ Heatsink is required
o Please refer to the PEX8796 Datasheet and User Manual for additional design requirements
• 2 x PCIe x8 Sliver connectors supporting:
o PCIe x16 Peer-to-Peer (P2P) connectivity between two PCIe Switches on two different 4-Slot
Risers (Risers #3 & #5)
o P2P connectivity supports direct transactions between PCIe Switches/PCIe Cards without
using the CPU Root complex to move the data or store in main memory, thus reducing
overall PCIe communication latency.
o 2 different Sliver Connectors/Cables are used from TE Connectivity:
▪ 50 pin Connector/Cable supports lower x8 PCIe lanes
▪ 74 pin Connector/Cable supports upper x8 PCIe lanes
▪ Different Sliver Connector sizes are to ensure proper cabling from Riser to Riser
▪ 85-ohm characteristic impedance
• Each PCIe slot supports up to 75W as defined for a standard PCIe x16 card.
• A 12V Auxiliary connector (2x6 pin cabled from PDB) is required on the Riser to support PCIe Slot
power, PEX8796 VRs, 3.3V VR, and 3.3V_Stby VR.
o The required 12V Current for this Riser is ~30A (assume 35A for conn derating).
o 12V AUX Connector uses 5 x 12V pins: 35A/5 pins = 7A per pin.
o 12V AUX Connector uses 1 x 12V_PSU pin to generate 3.3V_Stby Power.
• Additional 12V power required by high power PCIe cards will cable directly from the PDB to the
cards themselves.
• Different Wire Harnesses can be used depending on the PCIe Card’s 12V Auxiliary Power connector
• 3.3V Slot power is required for all PCIe Cards:
o Per the PCIe Spec, 3.3V requires 3.0A per slot (4 x 3A = 12A), which cannot be pulled from
only three Riser Slot 3.3V Pins.
o 3.3V must be sourced from an on-board VR
• 3.3V_Stby may be required by some PCIe Cards:
o Per the PCIe Spec, 3.3V_Stby requires 375mA per slot (4 x 375ma = 1.5A), which cannot be
pulled from only a single Riser Slot 3.3V_Stby pin.
o A small VR (generated from the 12V_PSU rail) will be added to the design to support the
required current.
P3V3_ST BY P3V3
P3V3_ST BY
NI PCA9548
4.7K 4.7K
4.7K 0xE4
0
0 I2C_0 ISOLATOR Slot4,PCIe x16
I2C 0
I2C_1 ISOLATOR Slot3,PCIe x16
0
I2C_2 ISOLATOR Slot2,PCIe x16
0
I2C_3 ISOLATOR Slot1,PCIe x16
P3V3_ST BY P3V3_ST BY
P3V3 P1V8
4.7K
I2C_4 4.7K 4.7K
4.7K
I2C_5
0
ISOLATOR PCA9617
PEX8796
I2C_6 ISOLATOR
0
P3V3
0xBA
P3V3_ST BY
22 22 22
4.7K
CLK Buffer
4.7K T em p sensor T em p sensor FRU
I2C_7 0xF8(Topside) 0x98(Backside) 0xAC 0xD8
Figure 12: I2C Block Diagram for 4-Slot Active PCIe Riser
Figure 13: Clock Block Diagram for 4-Slot Active PCIe Riser
12.68”
2x6 Pwr
50pin
Sliver #2
Backside PEX8796 50pin
Sliver #1
Block Diagram
MB PCIe x8
SDA/SCL x8 to Edge Conn
Slot#2
SDA/SCL 3.3V (PCA9617)
1.8V Sliver
I2C PCIe G3 Switch
P8 Conn#2 x8
MUX CLK
(PEX8796) (upper x8)
(PCA9548)
P4 P12 P20 P16
x8
50pin Sliver Cable
MB PCIe x8
to Edge Conn
Slot#1
PCIe x16 Riser Gold Fingers
PWR_BRK# 3.3V_stby
3.3V VR
JTAG VR
Figure 16: 5-Slot Active Riser PCIe Port Mapping and Routing Layers (View from Backside)
Figure 17: 50 pin x8 Sliver Connector to PCIe x8 Edge Connector Cable (Slot #1)
Table 8: 50 pin x8 Sliver Connector to PCIe x8 Edge Connector Cable Wire Mapping
• Each PCIe slot supports up to 75W as defined for a standard PCIe x16 card.
• 12V and 3.3V to PCIe Slot #1 is supported from the MB Riser Slot
• A 12V Auxiliary connector (2x6 pin cabled from PDB) is required on the Riser to support PCIe Slots
#2-5 power, PEX8796 VRs, 3.3V VR, and 3.3V_Stby VR.
o The required 12V Current for this Riser is ~30A (assume 35A for conn derating).
o 12V AUX Connector uses 5 x 12V pins: 35A/5 pins = 7A per pin.
o 12V AUX Connector uses 1 x 12V_PSU pin to generate 3.3V_Stby Power.
• Additional 12V power required by high power PCIe cards will cable directly from the PDB to the
cards themselves.
• Different Wire Harnesses can be used depending on the PCIe Card’s 12V Auxiliary Power connector
• Only Slots #2 and #4 can support a High Power, double-wide PCIe Card
• 3.3V Slot power is required for all PCIe Cards:
o Per the PCIe Spec, 3.3V requires 3.0A per slot (5 x 3A = 15A), which cannot be pulled from
only three Riser Slot 3.3V Pins.
o 3.3V must be sourced from an on-board VR
• 3.3V_Stby may be required by some PCIe Cards:
o Per the PCIe Spec, 3.3V_Stby requires 375mA per slot (5 x 375ma = 1.875A), which cannot
be pulled from only a single Riser Slot 3.3V_Stby pin.
o A small VR (generated from the 12V_PSU rail) will be added to the design to support the
required current.
P3V3_ST BY P3V3
P3V3_ST BY
4.7K 4.7K
NI PCA9548 0
ISOLATOR Slot5,PCIe x16
4.7K 0xE4 I2C_0 0
0 I2C_1 ISOLATOR Slot4,PCIe x16
0
I2C I2C_2 ISOLATOR Slot3,PCIe x16
0
I2C_3 ISOLATOR Slot2,PCIe x16
0
I2C_4 ISOLATOR Slot1,PCIe x16
P3V3_ST BY
P3V3 P1V8
4.7K 4.7K
4.7K
I2C_5
0
ISOLATOR PCA9617
PEX8796
I2C_6 ISOLATOR
0
P3V3
0xBA
P3V3_ST BY
22 22 22
4.7K
CLK Buffer
4.7K T em p sensor T em p sensor FRU
I2C_7 0xF8(Topside) 0x98(Backs id e) 0xAC 0xD8
Figure 19: I2C Block Diagram for 5-Slot Active PCIe Riser
Figure 20: Clock Block Diagram for 5-Slot Active PCIe Riser
The response time of the PCIe Card throttling relative to a PWRBRK_N signal assertion must be
approximately 10ms or less. The PCIe Card should throttle down to at least 50% of its max thermal
design power (TDP). The PCIe Card must return to full speed after PWRBRK_N has been de-asserted.
NOTE: This feature is not required for lower power PCIe Cards (< 150W) but is strongly recommended.
P3V3 P1V8_PCH_STBY
0
I2C_2 PCA9617 I2C_3
Riser 2, PCIe x8
0 I2C_4
I2C_3 PCA9617
P3V3_STBY
TCA9548 P3V3_ST BY
P3V3_ST BY P3V3
Physical
PCA9548 Slot # s
0XE0 4.7K 4.7K
NI
0xE4
4.7K 4.7K
0 0 I2C_0
0
ISOLATOR Slot4,PCIe x16 Slot 3.5
I2C_4 I2C
I2C_1
0
ISOLATOR Slot3,PCIe x16 Slot 3.4
0
P3V3_STBY
I2C_2 0
ISOLATOR Slot2,PCIe x16 Slot 3.3
BMC I2C_3 P3V3_ST BY P3V3_ST BY
ISOLATOR
P3V3 P1V8
Slot1,PCIe x16 Slot 3.2
4.7K
I2C_4 4.7K 4.7K
4.7K
4.7K 0 PEX8796
I2C_5 ISOLATOR PCA9617
I2C_5 0 I2C_6 ISOLATOR
0
P3V3
0xBA
P3V3_ST BY
22 22 22
4.7K
CLK Buffer
Riser 3 I2C_7
4.7K T em p sensor
0xF8(Topside)
T em p sensor
0x98(Backside)
FRU
0xAC 0xD8
P3V3_ST BY P3V3
Physical
P3V3_STBY P3V3_ST BY
PCA9548
4.7K 4.7K
Slot # s
NI 0
I2C_5
0
ISOLATOR PCA9617
PEX8796
I2C_6 ISOLATOR
0
P3V3
0xBA
P3V3_ST BY
22 22 22
4.7K
CLK Buffer
4.7K T em p sensor T em p sensor FRU
I2C_7 0xD8
Riser 4 0xF8(Topside) 0x98(Backs id e) 0xAC
P3V3_ST BY P3V3
P3V3_ST BY
Physical
PCA9548
4.7K
NI
0xE4
4.7K 4.7K Slot # s
0
0 I2C_0 ISOLATOR Slot4,PCIe x16 Slot 5.5
Bus Naming: * 0 based I2C
I2C_1
0
0
ISOLATOR Slot3,PCIe x16 Slot 5.4
I2C_2 ISOLATOR Slot2,PCIe x16 Slot 5.3
1-based pin I2C_3
0
P3V3_ST BY P3V3_ST BY
ISOLATOR
P3V3 P1V8
Slot1,PCIe x16 Slot 5.2
4.7K
0-based int I2C_4
0
4.7K
4.7K 4.7K
PEX8796
I2C_5 ISOLATOR PCA9617
I2C_6 ISOLATOR
0
P3V3
0xBA
P3V3_ST BY
22 22 22
4.7K
CLK Buffer
Riser 5 4.7K T em p sensor T em p sensor FRU
I2C_7 0xF8(Topside) 0x98(Backside) 0xAC 0xD8
Valid/Invalid PCIe Riser PCIe I2C Bus I2C Channel I2C Channel
Slot # Physical on First I2C on Second
Slot # MUX I2C MUX
(From BMC
(Silkscreen side, 0-
on Riser) (1 to 5 based) (0-based) (0-based)
bottom
to top)
PCIe Valid 4 3.5 0 4 0
Riser in Valid 3 3.4 0 4 1
Slot 3 Valid 2 3.3 0 4 2
Valid/Invalid PCIe Riser PCIe I2C Bus I2C Channel I2C Channel
Slot # Physical on First I2C on Second
Slot # MUX I2C MUX
(From BMC
(Silkscreen side, 0-
on Riser) (1 to 5 based) (0-based) (0-based)
bottom
to top)
Valid 1 3.2 0 4 3
Invalid N/A 3.1 N/A N/A N/A
Figure 22: 3U Chassis System Fan Position Diagram with Closed Fan Flappers
PDB
Fan Power
Fan
MB 6a/6b
Fan
5a/5b
2U Fan Conn
Fan
3a/3b
Fan
2a/2b
Fan
1a/1b
• Supports isolated PSU Fan Zone with sheet metal and plastic ducting
• For lightly loaded configs where a limited number of PCIe cards are plugged into PCIe Riser 4 (Middle
Riser) a plastic baffle can be used to block airflow through the middle of the Chassis. The baffle
forces more airflow across the high powered PCIe Cards on the outer Riser Slots.
Figure 25: 3U Chassis Cross Section Showing Middle Riser Baffle (to block airflow)
Topside
Outlet sensor
Internal sensor (0x98)
Backside
Topside
Outlet sensor
Internal sensor (0x98)
Backside
5. Mechanical Requirements
5.1 Chassis Requirements:
Due to the weight of the XPO200 3UN Server (90lbs), the standard Project Olympus T-Pin Slide Rails will
not allow the Chassis to pull in and out smoothly. As a result, a custom King Slide Rail Kit is used to
support easier travel during Servicing. However, since the width of the chassis is 17.36” (44.10 cm)
thinner Slide rails were required to fit into the 19” Project Olympus Rack, which limits how far the XPO200
3UN Server can be pulled out of the Rack. The XPO200 3UN Server can only be pulled out about 67%,
with 650mm-travel ball bearing sliding rails. For this reason, the Top cover is split into two sections (Front
and Rear). The Front Top Cover section allows access to most of the System hardware, leaving only the
System Fans, PDB and Power Cables under the Rear Top Cover without access.
Figure 31: 3U Chassis/System with Split Top Cover King Slide Rails
Power VR Subtotal
Function Device Util. Qty.
(W) Eff. (W)
CPU Intel Skylake (SKX) 205W TDP 205 90% 100% 2 455.6
PMAX 348.5 90% 100% 2 774.4
System Memory 32GB DDR4, 2667MHz, 2Rx4 RDIMM 7.54 90% 100% 24 201.1
PCH Lewisburg PCH 6 90% 100% 1 6.7
PCIe Riser Card Riser #3 / #5 4 slots active 4SW 33.9 90% 100% 2 75.3
PCIe Riser Card Riser #4 5 slots active 4SW 33.9 90% 100% 1 37.7
TDP 75 100% 100% 12 900.0
GPU Card Single-Wide Nvidia T4 GPU Card
PMAX 204 100% 100% 12 2448.0
Network 10G Single port SFP+ PCIe 2.0 x8 5GT/s 15 90% 100% 0 0.0
BMC AST1250 1.7 90% 100% 1 1.9
USB USB3.0 2.5 90% 100% 2 5.6
SSD M.2 Samsung PM963 960GB PCIe SSD 7.5 90% 100% 7 58.3
System Fans Delta 6056 GFC0612DSA01XXX-REVX00 32.4 100% 100% 6 194.4
Other Drivers, logic, pull-ups, etc … 10 90% 100% 1 11.1
Total (TDP) 2023
Turbo mode Total (PMAX) 3890
7.48”
Fan PWR
MB PWR
PSU2
Riser4 PWR
4.49”
PCIe Cards (Hi PWR)
Fan Power
Hot Swap
Sense 2x4
(6 x 60mm)
Controller
PSU2 (ADM1172)
2x12
P12V_STBY
0xB0 Riser 3 (4-Slot)
Sense
2x6
PSU3
2x12
0xB0
2x4 GPU/FPGA
or
2x3
Hot Swap
Note: Motherboard has Controller 2x4 GPU/FPGA
(ADM1278) or
its own Hot Swap P3V3_STBY 2x3
Controller & Switch Ckt. VR
P12V_PSU_SW
Riser 4 (2-Slot)
Sense
2x2
SW
P3V3_RISER4
Riser 5 (4-Slot)
Sense
2x6
Riser 4 (5-Slot)
2x4 GPU/FPGA
Alternative to or
2x3
2-Slot Riser Config
2x4 GPU/FPGA
or
2x3
2x4 GPU/FPGA
or
2x3
NOTE: XPO200 3UN uses the 5-Slot Riser Option in Riser Slot 4 and does not use Double-wide PCIe
Cards with 12V Aux power connections. The diagram above is showing optional configurations that can
be supported by the 3U Server.
Since the XPO200 3UN Server supports 3 PSUs with 3 internal PSU Modules, this equals 9 total PSU
Modules per Server. As a result, the Server supports an 8+1 redundancy model, where the system will
ride through any single PSU module failure. Depending on the loading conditions a System may be able
to ride through losing 2 PSU Modules. To maintain 8+1 PSU Module redundancy with margin, the max
supported System Power Budget is approximately 2400W (DC).
Project Olympus PSUs typically support an integrated battery pack (680W max). However, the combined
battery power of 3 x 680W packs does not meet the XPO200 3UN Server max System Power
requirements. Consequently, the Non-Battery PSU will be used in this system and therefore, no Battery
related features will be supported.
Figure 36: XPO200 3UN 3U Server Power Supply Load Sharing Diagram (Max Supported Power)
A 12V current sense chip that can be used for each power delivery branch, except for the MB power
delivery branch, is the INA301:
NOTE: All Fan Power comes directly from the PDB, so the Fan power connectors from the MB are not
used.
• IO Expander: Device that supports exposing additional GPIOs from the PDB to the BMC through I2C.
• FRU: Contains identification information about the PDB.
• Temp Sensor: Supports Temperature Sensor readings that can be used by the BMC to ensure
Thermal Levels remain within an acceptable range.
• 12V Vsense Pins (Reserved for Future Use): Used to support 12V remote sense for more optimal
load sharing between Power Supplies. Not used on current version of the 3U PCIe Expansion Server
or Project Olympus PSUs.
• PMBUS: Supports communication channels from BMC to PSUs and PDB I2C devices
o PSU1_PMBUS has a private connection to BMC and PCH
o PSU2_PMBUS & PSU3_PMBUS share a connection through an I2C MUX to avoid address
contention
• PSU_ALERT_N: PSU output that represents a change in status in any one of the three PSUs.
Although each PSU signal is Open Drain, they include 1kOhm internal pull-ups. These pull-ups in
combination with the 4.7K pull-up on the MB create a strong equivalent pull-up and make it difficult for
the PSU drivers to drive the PSU_ALERT_N signal low enough to meet the Vil_max of the receivers
on the MB. As a result, each PSU_ALERT_N signal has been isolated by its own Open Drain Buffer.
If any PSU asserts its PSU_ALERT_N signal, the BMC should read the status registers of all three
PSUs (through PMBUS) to find out what happened. The PDB PSU_ALERT_N signal feeds both
PSU1_ALERT_N and PSU2_ALERT_N pins to the BMC and on-board Throttle logic:
o PSU1_ALERT_N (thru 1U Conn): Feeds the CPU PROCHOT_N pins for immediate CPU
throttling.
o PSU2_ ALERT_N (thru 2U Conn): Feeds the PCIe Slot PWRBRK_N pins for immediate PCIe
Card throttling.
• PS_ON_N: PSU input used to Power-on/Power-off the PSUs. Each signal is Wire-OR’d together so
that all three PSUs can be Powered-on/Powered-off at the same time.
• BLADE_EN_N: Pass-through signal from Rack Manager (RM) that supports initiating a Power-
On/Power-Off event through assertion/de-assertion of PS_ON_N signal. Only BLADE_EN_N from
PSU1 is tied to PS_ON_N. BLADE_EN_N from PSU2 and PSU3 can be monitored by the BMC
(through I2C IO Expander) as a provision in case separate System actions are required; but are not
currently supported in the 3U PCIe Expansion Server BMC FW.
• BLADE_THROTTLE: Pass-through signal from RM that supports immediate throttling of the System.
Only PSU1 BLADE_THROTTLE is supported/required on the PDB since it ties directly into the
PROCHOT_N and PWRBRK_N inputs of the CPU and GPU cards respectively.
• NODE_ID[5:0]/SLOT_ID[5:0]: Pass-through signal from RM that assigns a unique ID number to each
Server (stored in its BMC). Each ID corresponds to a “U” location in the Rack. Only PSU1 passes the
NODE_ID bits to a given 3U Server/BMC. The other PSU NODE_ID bits are not needed and
subsequently left floating on the PDB.
• PSU_LED[1:0]: PSU output that provides status indications for its corresponding PSU. Since the MB
only supports 2 sets of PSU_LEDs, the PDB supports a separate PSU_LED header/cable, which
houses 3 x Bi-Color LEDs; one for each PSU. See following section for more details on the
PSU_LED Cable.
• BLADE_PRESENT_N: Passthrough signal from BMC to RM that ensures that each Server is present.
In the case of this system, only BLADE_PRESENT_N from PSU1 is used. PSU2 & PSU3
BLADE_PRESENT_N signals are left floating.
PSU2_ALERT_N
GPIOM5
PSU1_ALERT_N PSU1_ALERT_N
GPIOF4 OD PSU_ALERT#
PSU1_PS_ON_N
PS_ON#
PSU1_BLADE_EN_N
BLADE_EN#
1U Conn
MB HSC
PCH ADM1278 (0x22) PSU2_ALERT_N
NI OD PSU_ALERT#
PSU2_PS_ON_N
PSU2_PS_ON_BUF_N PS_ON#
I2C I/O IO00 PSU2_BLADE_EN_N
SMB_GBE PSU3_PS_ON_BUF_N BLADE_EN#
Expander IO01
2U Conn
RST_I2C_MUX_N
SML5 IO03 NC BLADE_THROTTLE
Jumper
PCA9555 HSC_P12V_FAULT_N
(0x40) IO12 NC NODE_ID[5:0]
EN_HSC_IN
CPLD IO02 PSU2_G_LED
PSU2_PS_ON_N PDB_CURRENT_ALERT_N PSU_LED0
PR3A IO13 PSU2_Y_LED
PSU2_BLADE_EN_R_N
IO10 PSU_LED1
SML1 SCL/SDA PSU3_BLADE_EN_R_N NI PSU2_PRESENT_N
IO11 BLADE_PRESENT#
RISER3_OC_CLEAR
Sense Sense Sense Sense
IO04
RISER3_OC_ALERT_N
Temp IO14
FRU RISER4_OC_CLEAR
Sensor IO05 I2C_PSU3_SCL/SDA PSU3
Note: Cabled LEDs mount directly to 0xA6 RISER4_OC_ALERT_N PMBUS
P3V3_STBY 0xF8 IO15 (0xB0)
front panel of Chassis. RISER5_OC_CLEAR
IO06
RISER5_OC_ALERT_N
IO16
FAN_OC_CLEAR
IO07
FAN_OC_ALERT_N
IO17 PSU3_ALERT_N
NI OD PSU_ALERT#
HSC_P12V_CSOUT Comparator PSU3_PS_ON_N
PDB HSC Circuit PS_ON#
PSU LED Cable
PSU3_BLADE_EN_N
ADM1278 (0x26) BLADE_EN#
PSU1_G_LED NC BLADE_THROTTLE
PSU1_Y_LED NC NODE_ID[5:0]
PSU3_G_LED
PSU2_G_LED PSU_LED0
PSU2_Y_LED PSU3_Y_LED
PSU_LED1
PSU3_G_LED NI PSU3_PRESENT_N
PSU3_Y_LED BLADE_PRESENT#
Different 12V Split Cable Power Harnesses will be designed to support delivering power from the PDB to
two high power PCIe Cards per PCIe Riser. Each high power PCIe Card will include one or more of 2x3
or 2x4 12V Auxiliary connectors to support additional power above the base 75W PCIe Slot power.
Below is an example of a 12V Split Cable Power Harness for a high power PCIe Card that supports two
2x4 12V Auxiliary connectors.
Figure 38: Split Cable Power Harness for 300W High Power PCIe Card (Example)
NOTE: Since this system only supports non-LES PSUs, the Battery LED states are not applicable.
The main trigger of a “Quick Response” System Power Capping event in the 3U Server is when a 12V
Over Current (OC) event (Fault or Warning) is indicated in the STATUS_IOUT command of any one of
the three Project Olympus PSUs. In most cases, the Power Capping feature will not be needed, but it is
intended as a precaution to avoid shutting down the System, until the event is over (if temporary), or the
System can be properly serviced (if permanent). If an OC Event should happen, one of the three PSUs
will assert its PSU_ALERT_N pin as a notification to the System to take action. As described in the PDB
Miscellaneous IO Section of this Specification, the PDB combines the PSU_ALERT_N pins of all three
Project Olympus PSUs and cables both the PSU1_ALERT_N (drives CPU_PROCHOT_N pins) and
PSU2_ALERT_N (drives PCIe PWRBRK_N pins) pins on the MB for instant CPU and PCIe Card
throttling. The CPU and PCIe Cards are two of the highest power consumers in the system. So, by
throttling these devices, the overall 12V current should be reduced enough for the System to ride through
the OC Event without shutting down, thus maintaining high availability. While the BMC monitors this
System Power Capping Event, it is not involved in the flow. For System Power Capping to help the 3U
Server ride through an OC Event, the response time of the CPU and PCIe Card throttling must be
approximately 10ms or less. The Power Capping flow through the BMC would not fall within this 10ms
window, therefore the use of the immediate hardware PSU_ALERT_N signal is needed, or the System
would shut down.
There are various causes of an OC Event that occur when System Devices drive up the power based on
peak workloads, failed components, or some combination of these. One such condition that can trigger
an OC Event is when one or more PSU Modules fail, and the remaining PSU modules are expected to
carry the load of the System. The 3U Server supports three PSUs, each with three internal PSU Modules
for a total of 9 x PSU Modules. The 3U Server supports an 8+1 redundancy model, but under lighter
loading conditions could also support a 7+2 or even a 6+3 redundancy model. Whatever the case may
be, any time a PSU module is lost, the current draw on the remaining PSUs will increase. Depending on
the System Loading conditions, or how many PSU Modules fail, the current can increase enough to cross
the PSU OC Limit, thereby asserting the PSU_ALERT_N signal and forcing the System Power to be
reduced.
NOTE: There is no PSU_ALERT_N signal assertion from the Project Olympus PSU when one or more
PSU Modules fail. The PSU_ALERT_N signal is only asserted for OC events. It is not asserted for any
other PSU failure events (over temp, Vin fault, Fan failure, etc…). If one or more PSU Modules fail, a
SEL Event is logged by the BMC FW (during 1 second polling intervals), but there is no System Power
Capping triggered; provided the remaining PSU Modules can handle the load. System Power Capping is
only triggered when the PSU OC limit is tripped, which could potentially be caused by losing one or more
PSU Modules depending on the System loading conditions. This means Power Capping is only used
when the System needs it to protect the PSUs from shutting down in an over load situation.