Nvidia Vera: 1.2 TB/s of memory with stable latency, but the socket-to-socket link lags behind

Chips and Cheese analyzes the system architecture of Vera, Nvidia's server chip. Its conclusion is that the chip combines very high per-core performance with exceptional memory bandwidth. Its weak points are core-to-core and cross-socket latency. The full article is long, and this summary covers only what appears in the text.

Vera uses Olympus cores, which have very large out-of-order structures. The aim is high per-core performance in a chip with many cores. The text notes that core design is only part of the equation. Memory access outside the core and the interconnect also weigh heavily on performance. That job falls to the second generation of Nvidia's Scalable Coherency Fabric (SCF), a network linking cores, cache and off-chip I/O. The author believes that, as in Grace, it uses a mesh topology.

The SCF is made up of several types of nodes:

• Coherency Switch Nodes (CSN) host cores and cache. Each CSN hosts a pair of cores, which keeps the node count low. Ampere Altra Max uses a similar approach. • Memory Switch Nodes (MSN) connect the SCF to the LPDDR5X memory controllers. Vera has a 1024-bit LPDDR5X-9600 interface, so the MSNs can move up to 1.228 TB/s. • Bridge Switch Nodes (BSN) host the PCIe controllers and the cross-socket link. Vera has 88 PCIe lanes, with 665.5 GB/s combined to peripherals. • NVLink C2C is the cross-socket link and offers a theoretical 1.8 TB/s.

Memory: Grace used LPDDR5X, but its DRAM performance was mediocre compared with its rivals. Its latency exceeded 200 ns and its bandwidth fell slightly short of DDR5 chips of its time. According to the author, Vera fixes this. Its 1024-bit interface gives it more bandwidth than its competitors, and its latency is now comparable to DDR5 platforms. Vera also adopts SOCAMM2 modules, so memory can be repaired in the field and potentially upgraded. Grace had soldered memory, and a failed memory chip could be a problem.

Vera keeps latency under control as bandwidth load rises. A pointer-chasing thread sees no more than 200 ns, even when other threads push memory to its limit. Other server platforms show a "hockey stick" pattern, with latency soaring near the controllers' limit. The author concludes that this should let Vera better contain the "noisy neighbor" effect. It should also scale better in DRAM-bound workloads, despite having fewer cores.

L3 cache: Vera has a 164 MB L3 shared by all cores. It is non-inclusive of the private L2s, which are 2 MB each. That gives up to 340 MB of effective last-level capacity if no data is duplicated across the L2s. L3 read bandwidth is about 3.35 TB/s in total, or 33.1 GB/s per core. A single uncontested core reaches 88.5 GB/s. The author considers the per-core figure reasonable and probably sufficient, because the L2s reduce demand on the L3.

The text contrasts this with AMD's approach, which prioritizes cache capacity and very high L3 bandwidth. Genoa-X delivers 90 GB/s to a single core and only drops to 85 GB/s with all cores on the die loaded. It splits the chip into eight-core clusters and totals 1248 MB across L2 and L3. That can help with thread-private data or with many parallel jobs. In workloads that share a lot of data, duplication across L3 instances can waste capacity. In addition, a thread on Genoa-X can only allocate into its own 96 MB instance, whereas on Vera a thread could theoretically use the full 164 MB.

Cross-socket link: NVLink C2C enables dual-socket configurations with two NUMA nodes. A cross-node memory access has 309.27 ns of latency, 177.41 ns more than a local one. That figure is only reached when generating a few tens of GB/s on the link. An isolated test measures more than 370 ns. The author suggests the link only switches to a higher-performance state when there is enough demand. Bandwidth is very high for a cross-socket link: just over 587 GB/s for reads and 780 GB/s with read-modify-write patterns. The author says this is more than any other dual-socket configuration they know of. It also exceeds the local memory bandwidth of most servers.

Latency, on the other hand, is worse. Genoa-X has 222.7 ns for cross-socket accesses. On Vera, latency soars under high demand, reaching 780 ns with the read-modify-write pattern at 780 GB/s.

Core-to-core latency: within a socket, the average is 148.5 ns, ranging from 87.5 to 199.3 ns depending on the core pair and the cache line owner. SMT siblings see under 10 ns. Across sockets, latency is around 660 ns, which the author calls "unfortunately high". They recall similar figures from Anandtech on a dual-socket Ampere Altra system. Genoa-X achieves about 209 ns, and Graviton 4 is even better across sockets. Vera's intra-socket latency is consistent, as Nvidia's whitepaper states. It is also consistently high: worse than Grace and, on average, worse than Graviton 4 and Genoa-X.

SPEC CPU2026 performance: with one copy per hardware thread, Vera beats Graviton 5 node for node. It also beats a lower-core-count Zen 5 part configured with a single NUMA node spanning both sockets. At the socket level, Graviton 5 beats Vera. The author cautions that the comparison is not entirely fair, because Graviton 5 cannot expose the whole socket as a single NUMA node. It shows a trade-off between core count and per-core power. SMT narrows the gap, but it is hard to compete with a design that has more than twice as many cores.

Vera retains an edge in bandwidth-hungry workloads. In 749.fotonik3d, which the author assumes is DRAM-bandwidth-bound on both chips, Vera scores almost 20% higher than Graviton 5 despite having far fewer cores. The advantage extends to other SPEC CPU2026 floating-point tests. Graviton 5 clearly dominates in workloads that make good use of cache. The author did not collect performance counter data on the Vera and Graviton 5 runs, and only measured with uProf on a Ryzen 7 9800X3D.

The author's conclusion: Vera is a very strong contender. It offers exceptional per-core performance backed by enormous bandwidth, and it is far more compelling than Grace, which suffered from 1 MB L2s and very high latency on L2 misses. Compared with other Arm, AMD and Intel chips, it trades core count for per-core performance. It will be hard to beat when individual tasks must finish quickly and workloads do not fill all the cores. For the Arm server ecosystem, it adds diversity. Early Arm chips competed on density alone, while Vera pushes the boundary toward per-core and per-thread performance.

Comments on the article raise two points. One is whether SMT actually improves performance on Vera, given its bandwidth. The other is that the total memory width would be 1024 bits on Vera (64 16-bit channels) and 512 bits on Grace. The article does not address either.