Arm Neoverse Continues to Claim Territory in Infrastructure

After owning general purpose compute in cell phones and IoT devices, it wasn’t clear what Arm’s next act might be. Seemingly the x86 giants dominated in datacenters and  auguries suggested a bloody war in smaller platforms between Arm and RISC-V. But Arm knew what they were doing all along, growing upwards into infrastructure: cloud compute, wired and wireless communications, out to gateways and even edge devices. At first modestly, recently more aggressively displacing mainstream processors and FPGAs as the front end to AI-centric GPUs, in hyperscaler scale-out and DPUs, in Telcos for gateways and wireless baseband, at the edge in automotive among other applications. To my mind this is an under-appreciated stealth invasion of the fast growing and high value infrastructure underpinning all of our electronic tech.

Consider

Take the Nvidia superchip – the one that propelled the company to a trillion-dollar valuation. This device, called Grace-Hopper, is actually 2 chips. Hopper is the GPU handling all the fancy AI stuff (tensor operations and such) and Grace is a CPU, very tightly coupled to Hopper and designed to handle the bridge to the regular cloud world through I/O and memory management functions. Grace is built on 72 cores of Neoverse V2. As for market value, witness the recent rise in Arm valuation, ascribed to “anything connected to AI”.

Amazon Web Services (AWS) build their power efficient Graviton scale-out servers, most recently their Graviton 4, on Neoverse. Demand for these platforms is high judging by the fact that almost of the AWS EC2 instances are based on Graviton rather than x86 processors. AWS also build their own machine learning platform (Tranium) which Arm cite in their slides, so I’m guessing that has a Neoverse front-end also.

Similarly, Microsoft have announced their Cobalt 100 server chip and their Maia 100 AI chip, both also cited in the Arm Neoverse slides and explicitly said be built on Neoverse. Note that between them AWS and Microsoft Azure own most cloud provider services (CPS) business by a wide margin. Notable also is that other CPS ventures are following similar paths.

Meanwhile, Nvidia have built their Bluefield DPU platform, AWS their Nitro hypervisor system and Microsoft their Azure Boost system, all around Neoverse. Neoverse is appearing all over the place in datacenters. They are already established in Ampere servers and in wireless infrastructure with Nokia, Ericsson and Huawei.

The Engine Driving Neoverse

Arm continues its tempo of new core introductions each year, this year adding V3 in the performance-optimized V-series, N3 in the performance/watt N-series and E3 in the data throughput E-series. Impressively the V2 (previous generation) already benchmarks ahead of X86 processors in SQL database performance, Java and XGBoost (the hot gradient optimizer for machine learning). V3 shows double digit advances over V2 in a range of enterprise-centric benchmarks and a whopping 84% advantage over V2 in AI data analytics. N3 equally shows impressive performance advances over N2 across the board and nearly 200% in AI data analytics.

A recent strategic move builds on Arm’s Compute Subsystem (CSS) cores, announced for the N-series in mid 2023. The previous generation CSS, as Arm describes it, is a customizable compute subsystem, verified, validated, and PPA-optimized by Arm. Think of a multi-core cluster objective for which you don’t just get the Lego pieces (CPU core, coherent interconnect, memory subsystem, etc.) but a complete customizable compute subsystem configured with Neoverse CPU cores, multiple DDR/LPDDR channels, multiple PCIe/CXL PHY/controller. All tuned, verified, validated, and PPA-optimized by Arm to client workloads and a target foundry/process through Arm’s Total Design program.

In this latest announcement, Arm have just announced a new CSS N3 configured to support from 8 to 32 cores per die running at as low as 40W thermal design power for 32 cores. They have also just announced their first V-series CSS, CSS-V3, providing 50% performance boost over CSS-N2. The Microsoft Cobalt chip is built on CSS-N2.

Given fierce competition between CSPs, mobile network operators, and auto OEMS (as an edge example) it seems clear that a diminishing number of product teams will see value in reinventing this CSS wheel themselves. From a CSP perspective, an opportunity to get compute subsystems tuned to their workloads, while saving development cost and time to market, yet preserving differentiated advantage seems like a no-brainer.

What About Software?

Arm is already tied into a well-established ecosystem at all layers of the software stack. For cloud applications it is especially important that Arm be closely tied to cloud-native activities in open-source leadership in languages, tools, Linux distributions, networking, etc., etc. Together with increasingly ubiquitous Arm-based instance, cloud-based software developers see obvious advantages not only to use Arm for development but also to preferentially offer those service on Arm platforms – a virtuous cycle.

To underscore that this strategy is working, Oracle now offer their 19C database platform supporting Arm-based hardware, both in the cloud and on-prem. Similarly, SAP has ported their SAP HANA cloud to AWS Graviton processors. Those are two giant enterprise applications. Given costs and software availability advantages, I am sure more software and SaaS providers will follow.

Very impressive. I don’t know what else to call this other than a stealth invasion into infrastructure. You can read more HERE.