![Page 1: Hyperscale ARM Servers Jon Masters Chief ARM Architect, Red Hat](https://reader031.vdocuments.site/reader031/viewer/2022030323/58a1a1d71a28abb5488b910a/html5/thumbnails/1.jpg)
27th Large Installation System Administration Conference
November 3–8, 2013 • Washington, D.C.
Hyperscale ARM ServersJon Masters
Chief ARM Architect, Red Hat
![Page 2: Hyperscale ARM Servers Jon Masters Chief ARM Architect, Red Hat](https://reader031.vdocuments.site/reader031/viewer/2022030323/58a1a1d71a28abb5488b910a/html5/thumbnails/2.jpg)
27th Large Installation System Administration Conference
November 3–8, 2013 • Washington, D.C.
Raspberry Pi Model B(Supported by Pidora, a Fedora Remix)
![Page 3: Hyperscale ARM Servers Jon Masters Chief ARM Architect, Red Hat](https://reader031.vdocuments.site/reader031/viewer/2022030323/58a1a1d71a28abb5488b910a/html5/thumbnails/3.jpg)
27th Large Installation System Administration Conference
November 3–8, 2013 • Washington, D.C.
World's first bicycle-powered ARM Server(HP Redstone/Calxeda)
![Page 4: Hyperscale ARM Servers Jon Masters Chief ARM Architect, Red Hat](https://reader031.vdocuments.site/reader031/viewer/2022030323/58a1a1d71a28abb5488b910a/html5/thumbnails/4.jpg)
27th Large Installation System Administration Conference
November 3–8, 2013 • Washington, D.C.
HP Moonshot SL1500 Chassis(3 ARM cartridges announced so far)
![Page 5: Hyperscale ARM Servers Jon Masters Chief ARM Architect, Red Hat](https://reader031.vdocuments.site/reader031/viewer/2022030323/58a1a1d71a28abb5488b910a/html5/thumbnails/5.jpg)
27th Large Installation System Administration Conference
November 3–8, 2013 • Washington, D.C.
The old Fedora ARM Project build system
![Page 6: Hyperscale ARM Servers Jon Masters Chief ARM Architect, Red Hat](https://reader031.vdocuments.site/reader031/viewer/2022030323/58a1a1d71a28abb5488b910a/html5/thumbnails/6.jpg)
27th Large Installation System Administration Conference
November 3–8, 2013 • Washington, D.C.
The current Fedora ARM Project build system(Boston Limited Viridis (Calxeda))
![Page 7: Hyperscale ARM Servers Jon Masters Chief ARM Architect, Red Hat](https://reader031.vdocuments.site/reader031/viewer/2022030323/58a1a1d71a28abb5488b910a/html5/thumbnails/7.jpg)
27th Large Installation System Administration Conference
November 3–8, 2013 • Washington, D.C.
Warren East Announcing Linaro Enterprise Group
![Page 8: Hyperscale ARM Servers Jon Masters Chief ARM Architect, Red Hat](https://reader031.vdocuments.site/reader031/viewer/2022030323/58a1a1d71a28abb5488b910a/html5/thumbnails/8.jpg)
27th Large Installation System Administration Conference
November 3–8, 2013 • Washington, D.C.
What a difference a year makes...
32-bit ARM servers now used in production– Running our 32-bit Fedora builds today– Building competence ahead of 64-bit
Hyperscale Computing now a broader term
Red Hat co-founded Linaro Enterprise Group– LEG celebrates one year this week
Fedora AArch64 Bootstrap exercise– Nearly complete Fedora builds
![Page 9: Hyperscale ARM Servers Jon Masters Chief ARM Architect, Red Hat](https://reader031.vdocuments.site/reader031/viewer/2022030323/58a1a1d71a28abb5488b910a/html5/thumbnails/9.jpg)
27th Large Installation System Administration Conference
November 3–8, 2013 • Washington, D.C.
What is Hyperscale Computing?
![Page 10: Hyperscale ARM Servers Jon Masters Chief ARM Architect, Red Hat](https://reader031.vdocuments.site/reader031/viewer/2022030323/58a1a1d71a28abb5488b910a/html5/thumbnails/10.jpg)
27th Large Installation System Administration Conference
November 3–8, 2013 • Washington, D.C.
SoC Commoditization of traditional server– Much higher density (1,000s nodes/rack)– Lower energy profile, reduced PUE
Fabric Interconnectivity to other nodes– Disaggregation (OpenCompute example)
Integrated Management and Offload– Orchestration at very large scale– Value in accelerators offloading from CPU
Failure-in-Place
![Page 11: Hyperscale ARM Servers Jon Masters Chief ARM Architect, Red Hat](https://reader031.vdocuments.site/reader031/viewer/2022030323/58a1a1d71a28abb5488b910a/html5/thumbnails/11.jpg)
27th Large Installation System Administration Conference
November 3–8, 2013 • Washington, D.C.
1,000-10,000 server nodes per rack– Tightly connected, rack-level granularity
Aggregate performance– Targeting “good enough” linear CPU
● PPA tradeoffs favor “sweet spot”– Increasingly data-movement focused
Technical challenges
– (re)provision elastically, and very often– Leverage integrated management, etc.
![Page 12: Hyperscale ARM Servers Jon Masters Chief ARM Architect, Red Hat](https://reader031.vdocuments.site/reader031/viewer/2022030323/58a1a1d71a28abb5488b910a/html5/thumbnails/12.jpg)
27th Large Installation System Administration Conference
November 3–8, 2013 • Washington, D.C.
![Page 13: Hyperscale ARM Servers Jon Masters Chief ARM Architect, Red Hat](https://reader031.vdocuments.site/reader031/viewer/2022030323/58a1a1d71a28abb5488b910a/html5/thumbnails/13.jpg)
27th Large Installation System Administration Conference
November 3–8, 2013 • Washington, D.C.
System-on-Chip revolutionized mobile– Now is revolutionizing the server space– CPU, IO, devices all integrated on-chip– Servers add offload/acceleration engines
Increasingly verticalized/differentiated world
– Future servers have much more variety– Organized, value-added differentiation
Fabric and management integrated on-chip
– IPMI, industry standard interfaces
![Page 14: Hyperscale ARM Servers Jon Masters Chief ARM Architect, Red Hat](https://reader031.vdocuments.site/reader031/viewer/2022030323/58a1a1d71a28abb5488b910a/html5/thumbnails/14.jpg)
27th Large Installation System Administration Conference
November 3–8, 2013 • Washington, D.C.
![Page 15: Hyperscale ARM Servers Jon Masters Chief ARM Architect, Red Hat](https://reader031.vdocuments.site/reader031/viewer/2022030323/58a1a1d71a28abb5488b910a/html5/thumbnails/15.jpg)
27th Large Installation System Administration Conference
November 3–8, 2013 • Washington, D.C.
Fabric powers future datacenters– The next battleground between vendors– Obviate cabling/top-of-rack switches
Disaggregate resources (split depreciation)
– Separate Compute from other resources– Connect network, memory, storage, PCIe– 100GBit+ networking, silicon photonics
NFV and SDN are just workloads
– Separate physical/virt fabric topology
![Page 16: Hyperscale ARM Servers Jon Masters Chief ARM Architect, Red Hat](https://reader031.vdocuments.site/reader031/viewer/2022030323/58a1a1d71a28abb5488b910a/html5/thumbnails/16.jpg)
27th Large Installation System Administration Conference
November 3–8, 2013 • Washington, D.C.
Datacenter management of whole racks– Chassis-level asset/topology tracking– SoC-level KVM/power via IPMI, etc.
Offload/acceleration engines
– Traditional network acceleration– FPGA accelerators, custom IP blocks
Compute is the boring part of the story
![Page 17: Hyperscale ARM Servers Jon Masters Chief ARM Architect, Red Hat](https://reader031.vdocuments.site/reader031/viewer/2022030323/58a1a1d71a28abb5488b910a/html5/thumbnails/17.jpg)
27th Large Installation System Administration Conference
November 3–8, 2013 • Washington, D.C.
![Page 18: Hyperscale ARM Servers Jon Masters Chief ARM Architect, Red Hat](https://reader031.vdocuments.site/reader031/viewer/2022030323/58a1a1d71a28abb5488b910a/html5/thumbnails/18.jpg)
27th Large Installation System Administration Conference
November 3–8, 2013 • Washington, D.C.
The (very) low-hanging fruit– Web Tier– Cloud Computing
New opportunities
– Network Function Virtualization– Software Defined Networking
Existing markets
– Traditional Enterprise computing
![Page 19: Hyperscale ARM Servers Jon Masters Chief ARM Architect, Red Hat](https://reader031.vdocuments.site/reader031/viewer/2022030323/58a1a1d71a28abb5488b910a/html5/thumbnails/19.jpg)
27th Large Installation System Administration Conference
November 3–8, 2013 • Washington, D.C.
The Red Hat ARM Team (RHAT)
![Page 20: Hyperscale ARM Servers Jon Masters Chief ARM Architect, Red Hat](https://reader031.vdocuments.site/reader031/viewer/2022030323/58a1a1d71a28abb5488b910a/html5/thumbnails/20.jpg)
27th Large Installation System Administration Conference
November 3–8, 2013 • Washington, D.C.
Formed three years ago to focus on 64-bit– Small core team focused on architecture
Accelerated development of 32-bit Fedora
– Bootstrapped ARMv7 “hard float” ABI
Lead the bootstrapping of 64-bit Fedora
– cross-compilers, models, and hardware
Co-founded Linaro Enterprise Group
– 2 assignees working on ACPI, validation
![Page 21: Hyperscale ARM Servers Jon Masters Chief ARM Architect, Red Hat](https://reader031.vdocuments.site/reader031/viewer/2022030323/58a1a1d71a28abb5488b910a/html5/thumbnails/21.jpg)
27th Large Installation System Administration Conference
November 3–8, 2013 • Washington, D.C.
32-bit vs. 64-bit ARM
![Page 22: Hyperscale ARM Servers Jon Masters Chief ARM Architect, Red Hat](https://reader031.vdocuments.site/reader031/viewer/2022030323/58a1a1d71a28abb5488b910a/html5/thumbnails/22.jpg)
27th Large Installation System Administration Conference
November 3–8, 2013 • Washington, D.C.
![Page 23: Hyperscale ARM Servers Jon Masters Chief ARM Architect, Red Hat](https://reader031.vdocuments.site/reader031/viewer/2022030323/58a1a1d71a28abb5488b910a/html5/thumbnails/23.jpg)
27th Large Installation System Administration Conference
November 3–8, 2013 • Washington, D.C.
Drop-in alternative driven by demand– Must “just work” for boot/install/etc.– Cannot require users to change workflow
Must be standardized
– “An ARM is an ARM is an ARM”?
Must be manageable (orchestration)
– Leverage OpenStack– On-chip management, etc.
Learn from other architectures
![Page 24: Hyperscale ARM Servers Jon Masters Chief ARM Architect, Red Hat](https://reader031.vdocuments.site/reader031/viewer/2022030323/58a1a1d71a28abb5488b910a/html5/thumbnails/24.jpg)
27th Large Installation System Administration Conference
November 3–8, 2013 • Washington, D.C.
U-Boot is an embedded 32-bit bootloader– Intended for vertical integration scenario
Not a standardized environment– Different magic values per platform, etc.– Many vendor extensions not upstream
DeviceTree used to describe hardware– Bindings evolving over time
![Page 25: Hyperscale ARM Servers Jon Masters Chief ARM Architect, Red Hat](https://reader031.vdocuments.site/reader031/viewer/2022030323/58a1a1d71a28abb5488b910a/html5/thumbnails/25.jpg)
27th Large Installation System Administration Conference
November 3–8, 2013 • Washington, D.C.
Coordinated base platform standards– ABIs, Binary Formats, Linux specific bits
Remove gratuitous differentiation– Base hardware assumptions
● Standardized, compatible interfaces (AHCI/XHCI/UARTS...)
● Enumerable hardware (ACPI, PCIe, etc.)– Boot Architecture
● UEFI abstracts underlying hardware● Network, local, removable media
![Page 26: Hyperscale ARM Servers Jon Masters Chief ARM Architect, Red Hat](https://reader031.vdocuments.site/reader031/viewer/2022030323/58a1a1d71a28abb5488b910a/html5/thumbnails/26.jpg)
27th Large Installation System Administration Conference
November 3–8, 2013 • Washington, D.C.
Red Hat assisted in Fedora ARMv7 bootstrap– Smokescreen “hard float” ABI as cover– Learning exercise anticipating ARMv8
Derived a multi-stage process for bootstrap
– Cross-compilation process at first– Switch to native early (for correctness)– Migrate into containerized build (mock)– Transition to web-based build (koji)
Preparation for Koji currently in progress
![Page 27: Hyperscale ARM Servers Jon Masters Chief ARM Architect, Red Hat](https://reader031.vdocuments.site/reader031/viewer/2022030323/58a1a1d71a28abb5488b910a/html5/thumbnails/27.jpg)
27th Large Installation System Administration Conference
November 3–8, 2013 • Washington, D.C.
Early access to architecture documentation– Initiated long-term work (OpenJDK, etc.)
ARM FAST/Foundation Models– Cross-compiled minimal environment
Booted minimal environment on models– Built 12,000 packages very (very) slowly
Applied Micro X-Gene hardware transition– Can build software in hours, not weeks
![Page 28: Hyperscale ARM Servers Jon Masters Chief ARM Architect, Red Hat](https://reader031.vdocuments.site/reader031/viewer/2022030323/58a1a1d71a28abb5488b910a/html5/thumbnails/28.jpg)
27th Large Installation System Administration Conference
November 3–8, 2013 • Washington, D.C.
Red Hat assisted in initial kernel review– Actively developing new kernel features
Lending resources to critical pieces in Linaro– UEFI kernel shim, ACPI integration, etc.
Porting Red Hat-specific components– Orchestration and management software
Assisted in hardware validation and test– Extensively collaborated w/Applied Micro
![Page 29: Hyperscale ARM Servers Jon Masters Chief ARM Architect, Red Hat](https://reader031.vdocuments.site/reader031/viewer/2022030323/58a1a1d71a28abb5488b910a/html5/thumbnails/29.jpg)
27th Large Installation System Administration Conference
November 3–8, 2013 • Washington, D.C.
Applied Micro X-Gene XC-1 Server Board– 8-core, multi-GHz, custom 64-bit design
Fedora 19 Remix– Running exclusively in 64-bit AArch64– 32-bit AArch32 is not supported
12,000+ software packages ported– Including LAMP stack, GlusterFS
![Page 30: Hyperscale ARM Servers Jon Masters Chief ARM Architect, Red Hat](https://reader031.vdocuments.site/reader031/viewer/2022030323/58a1a1d71a28abb5488b910a/html5/thumbnails/30.jpg)
27th Large Installation System Administration Conference
November 3–8, 2013 • Washington, D.C.
Questions?
https://fedoraproject.org/wiki/Architectures/ARM/#fedora-arm (Freenode IRC)