facebook ocp server hardware - circleb.eu · –ocp mezzanine nic • manufactured by wiwynn....
TRANSCRIPT
Facebook OCP Server Hardware
Facebook Server Hardware
• Facebook uses four main building blocks to create its Infrastructure HW– Compute
– Storage
– Network
– Rack & Power
• Varying the ratios of compute to storage, and the internal configuration of the servers provides Facebook with all its SKUs.
Standard
Systems
I
Web
III
Database
IV
Hadoop
V
Haystack
VI
Feed
VII Cold
Storage
CPU
High
2 x Haswell
(12c)
2.5 GHz
High
2 x Haswell
(12c)
2.5 GHz
High
2 x Haswell
(12c)
2.5 GHz
Low
1 x Avoton
(8c)
2.7 GHz
High
2 x Haswell
(12c)
2.5GHz
High
2 x Haswell
(12c)
2.5 GHz
MemoryLow
32GB
High
256GB
Medium
128GB
Low
32GB
High
256GB
Medium
128GB
DiskLow
500GB
SATA
High IOP
2 x 3.2TB
Flash
128 GB
mSata
High
15 x 4TB (or
6TB)
NL SAS
High
30 x 4TB (or
6 TB)
NL SAS
Medium
2TB SAS
(1.8 TB
FLASH
optional)
Very High
240 X 4TB
16 active
disks
Services Web DatabaseData
Warehouse
(Hadoop)
Object Store
(Photos)
Cache
(Newsfeed)Archival
Type I
• Front end/web service machines.
• Compute servers only
Power Shelf
Compute Empty Compute
Compute Empty Compute
Compute Empty Compute
Compute Empty Compute
Compute Empty Compute
Power Shelf
Compute Empty Compute
Compute Empty Compute
Compute Empty Compute
Compute Empty Compute
Power Shelf
Compute Empty Compute
Compute Empty Compute
Compute Empty Compute
Compute Empty Compute
Switch
Empty
Compute Empty Compute
Compute Empty Compute
Type III
• High performance database applications
• Compute nodes w/ PCI-E Flash Cards
Compute +
FlashEmpty
Compute +
Flash
Power Shelf
Compute +
FlashEmpty
Compute +
Flash
Compute +
FlashEmpty
Compute +
Flash
Power Shelf
Compute +
FlashEmpty
Compute +
Flash
Compute +
FlashEmpty
Compute +
Flash
Compute +
FlashEmpty
Compute +
Flash
Compute +
FlashEmpty
Compute +
Flash
Compute +
FlashEmpty
Compute +
Flash
Compute +
FlashEmpty
Compute +
Flash
Compute +
FlashEmpty
Compute +
Flash
Power Shelf
Compute +
FlashEmpty
Compute +
Flash
Compute +
FlashEmpty
Compute +
Flash
Compute +
FlashEmpty
Compute +
Flash
Switch
Empty
Compute +
FlashEmpty
Compute +
Flash
Compute +
FlashEmpty
Compute +
Flash
Type IV
• Data warehouse storage and analytics (for ex. Hadoop)
• Powerful Compute + Mass Storage
Storage
Power Shelf
Storage
Storage
Storage
Storage
Storage
Compute Compute Compute
Storage
Power Shelf
Compute Compute Compute
Storage
Compute Compute Compute
Storage
Compute Compute Compute
Power Shelf
Compute Compute
Compute Compute Compute
Switch
Empty
Compute
Type V
• Object storage (for ex. Photo and Video storage)
• Cost Efficient Compute + Mass Storage
uServer + Storage
Power Shelf
uServer + Storage
uServer + Storage
uServer + Storage
uServer + Storage
uServer + Storage
uServer + Storage
uServer + Storage
uServer + Storage
uServer + Storage
uServer + Storage
Power Shelf
uServer + Storage
Power Shelf
uServer + Storage
Switch
Empty
uServer + Storage
uServer + Storage
Type VI
• Cache heavy applications (for ex. News Feed)
• Compute w/ large memory
Compute +
High MemEmpty
Compute +
High Mem
Power Shelf
Compute +
High MemEmpty
Compute +
High Mem
Compute +
High MemEmpty
Compute +
High Mem
Power Shelf
Compute +
High MemEmpty
Compute +
High Mem
Compute +
High MemEmpty
Compute +
High Mem
Compute +
High MemEmpty
Compute +
High Mem
Compute +
High MemEmpty
Compute +
High Mem
Compute +
High MemEmpty
Compute +
High Mem
Compute +
High MemEmpty
Compute +
High Mem
Compute +
High MemEmpty
Compute +
High Mem
Power Shelf
Compute +
High MemEmpty
Compute +
High Mem
Compute +
High MemEmpty
Compute +
High Mem
Compute +
High MemEmpty
Compute +
High Mem
Switch
Empty
Compute +
High MemEmpty
Compute +
High Mem
Compute +
High MemEmpty
Compute +
High Mem
Type VII
• Archival back up storage
• Single compute node connected to 240HDDs.
Storage
Storage
Storage
Storage
Storage
Storage
Empty Compute Empty
Storage
Storage
Storage
Storage
Empty Compute Empty
Storage
Storage
Storage
Power Shelf
Switch
Empty
Storage
Storage
Storage
Open Rack
• Wide equipment bay (21”) for maximum space efficiency
– Same external width (24”) as standard 19” rack.
• Shared power supply architecture for better resiliency
• Design optimized for serviceability and efficiency
Open Rack
• 1 OpenU = 48mm height
• Power shelf feeds 12V busbar that connects to servers
– Standardized mechanical interface between servers and rack
Open Compute Server• Dual Socket Xeon E5-2600 V3/V4
(Haswell/Broadwell)
• 16 DIMMs (2 per channel/4 channel per socket/2400MHz)
• Motherboard dimensions 6.5”x20”
• 3 PCI Gen 3 add in slots (x24)
• OCP Mezzanine Card (x8 PCI Gen 3)
• M.2/mSATA/HDD for boot
• ORV1/ORV2 compatible – 2OU 1/3 width
• BMC
• Manufactured by Quanta and WiWynn
Yosemite Compute Platform
• Platform for 4 single socket compute nodes
• ORV1/ORV2 compatible – 2OU 1/3 width
• Multi-host (shared) 40G/50G OCP Mezzanine NIC
• Shared BMC
Mono Lake
• Yosemite compute node card
• Xeon D-1500 SoC
• 4 DIMMs (2 per channel/2 channel/2400MHz/up to 128GB)
• 2 M.2 (NVMe/SATA)
• 210mm x 110mm
• Manufactured by Quanta
Big Sur GPU Server• Full Size Dual Socket Xeon E5-2600
V3/V4 (Haswell/Broadwell) Motherboard
• 8 x16 3 PCI Gen 3 GPU card slots– NVIDIA Tesla, AMD FirePro and Intel Phi
• OCP Mezzanine Card (x8 PCI Gen 3)– Ethernet and Infiniband
• 8x 2.5” SATA HDD/SSD
• 4OU ORV1/ORV2 compatible
• BMC
• Manufactured by Quanta
Open Vault Storage
• JBOD storage attach to Open Compute Server
• 2OU ORV1/ORV2 compatible • 30 HDDs (2 trays x 15 HDDs)• LSI SAS2x28 6Gb expander
per tray• 1 Ext. MiniSAS + 2 Int.
MiniSAS front facing ports per tray
• Manufactured by WiWynn
Microserver Storage
• Light weight server inside Open Vault• Same storage capacity as Open Vault• Microserver
– Intel Avoton C2750– DDR3 SO-DIMM (2 per channel/2
channel/1600 MHz/up to 32GB)– M.2 (NVMe/SATA) and mSATA
• Storage SubSystem– 12G SAS IOC + Expander (SAS 3008
+ SAS3x24)– BMC– OCP Mezzanine NIC
• Manufactured by WiWynn
Lightning (NVMe in Open Vault)
• JBOF storage attach to Open Compute Server over PCIe
• 2OU ORV1/ORV2 compatible
• Up to 60 SSDs (2 trays of 15 x4 PCI Gen 3 slots)
• 4 x4 PCI Gen 3 external upstream ports per tray
• OpenBMC
• Manufactured by WiWynn
Wedge 40
• Broadcom Trident II
• 16 x 40Gb QSFP ports (can be split to 4x10Gb)
• FBOSS
• OpenBMC
• Microserver card used for control plane
• Compatible with 19” and OR
• Manufactured by Accton
Wedge 100• Broadcom Tomahawk (BCM56960)
• 32 x 100G QSFP28 Ports (1x100G, 2x50G, 4x25G, 1x40G and 4x10G)
• FBOSS
• OpenBMC
• RackMon for ORV2
• COM-E CPU module for control plane
• Support DAC and optical transceiver
• Compatible with 19” and OR
• Manufactured by Accton
Open Rack V1
• 2100mm x 600mm x 1067mm
• 950kg weight handling capability
• 3 power zones of 10OU for servers
– Total 45 compute server slots per rack
• 3OU per zone for PSUs
• 3 busbars
• Manufactured by Rittal and Delta
Open Rack V2
• 2210mm x 600mm x 1067mm• 1450kg weight handling
capability• 2 power zones of 16OU for
servers– Total 48 compute server slots per
rack
• 3OU per zone for PSUs• 1 busbar (3 optional)• Manufactured by Rittal and
Delta
ORV1 Power• 3 power zones per rack
• 6 700W PSUs per zone + 1 redundant
• 4200W per shelf, 12.6kW per rack
• 94% efficiency (@50% load)
• 480/277VAC or 400/230VAC primary AC input, 12VDC output
• Optional 48V DC battery back up input
– 90 sec hold up time, 1 Lead Acid battery rack per 6 server racks
• 208/120VAC dual input option also available (no DC input)
• Manufactured by Delta and Power One
ORV2 Power• 2 power zones per rack
• 2 3.3kW PSUs per zone + 1 redundant
• 6600W per shelf, 13.2kW per rack
• 96% efficiency (@50% load)
• 480/277VAC or 400/230VAC primary AC input, 12VDC output
• Optional 52V DC battery back up input
– 90 sec hold up time, Li Ion batteries built into power shelf (no external rack)
• 208/120VAC dual input option also available (no DC input)
• Remote monitoring
• Manufactured by Delta and Power One
Batteries
PSUs