The market has long regarded the CUDA software ecosystem as NVIDIA’s deepest moat. But as high-level abstraction frameworks such as Triton gain ground, the hardware-software stack underneath is increasingly hidden from developers. Competitors are chipping away at that software moat, catching up on per-GPU compute and offering open-source alternatives, and some argue CUDA could lose its lead in LLMs as early as this year.
In my view, NVIDIA’s moat has already shifted β from chips and software to platform and system. With NVLink Fabric, In-Network Computing, and the Vera CPU, its outstanding scale-out and scale-up performance has made it the default choice and the engineering standard for ultra-large GPU clusters.
I. The Shifting Moat: From Single-GPU CUDA to Cluster Fabric
1. Alternatives Beyond CUDA
Developers used to write low-level operators directly in CUDA C++, which created powerful ecosystem lock-in. But the AI boom has slashed the cost of leaving an ecosystem behind (see this post). Developers now write against high-level frameworks while the toolchain lowers and retargets operators automatically. Whether the chip underneath speaks CUDA matters less and less to them. That opens a real window for AMD’s MI series β and for the cloud vendors’ homegrown ASICs.
2. From Single-GPU Compute to Cluster Communication
As LLM parameter counts explode, the central challenge of the AI 2.0 era has moved from the TFLOPS ceiling of a single GPU to the bandwidth and latency of communication across multi-GPU clusters and memory-fusion networks.
Running a massive MoE model means tokens constantly being routed all-to-all between expert GPUs, to destinations that are effectively unpredictable. When latency between cluster nodes is high, GPUs end up waiting idle, and long-tail delays pile up.
Whoever builds the faster GPU cluster controls the AI infrastructure.
II. The Extraordinary Performance of Cross-Rack NVLink Fabric
1. A 1:1 Non-Blocking Scaling Ratio at the Hardware Level
What lets NVIDIA defend its premium at the rack level is the remarkable linear scalability of its physical network.
- Blackwell’s NVLink 5 gives a single GPU 1.8 TB/s of bidirectional bandwidth; scaled out to a GB200 NVL72 rack over a non-blocking fat-tree topology, the rack’s aggregate physical bandwidth reaches 130 TB/s β with no loss.
Distributed systems usually pay for consistency with complex communication protocols, and that overhead drags the scaling ratio down β 1 + 1 = 1.5: better than not scaling at all, but short of the sum of two racks running independently. NVLink and NVLink Fabric, by contrast, scale almost losslessly, both horizontally between racks and vertically within a rack. It is nothing short of an engineering marvel.
2. Outpacing the UALink Alliance with Iteration Speed
The UALink (Ultra Accelerator Link) alliance β AMD, Intel, Broadcom, Cisco, Google, Meta, Microsoft, and others β set out to repeat PCIe’s open-standard success, publishing the UALink 1.0 and 2.0 specifications to jointly break NVLink’s monopoly.
But open standards suffer from fragmented software and hardware ecosystems, and from long cycles of multi-vendor horse-trading. NVIDIA, iterating aggressively on a yearly cadence, had already locked up the highest-end compute market with the mature Rubin NVL576 before UALink 2.0 chips shipped in any volume β leaving the open standard with no way to catch up.
III. In-Network Computing: A Compute Network Pushed to the Limit
Even if rivals manage to stitch together a large cluster, their Model FLOPS Utilization (MFU) falls off a cliff in real large-model runs once communication storms hit. NVIDIA’s answer is to make the network itself compute.
1. Arithmetic Evolves Inside the NVSwitch Chip
The latest NVSwitch chips pack in high-density SerDes interfaces β and integrate ALUs (arithmetic logic units) right into the switch, so the switch itself can take on some of the computation.
2. SHARP: Pushing Computation Down into the Network
With the SHARP (Scalable Hierarchical Aggregation and Reduction Protocol) engine, the most critical collective operations in large models β the gradient summation and distribution behind All-Reduce and All-to-All β are computed inside the NVSwitch while the data is in flight, rather than being shipped to a designated GPU for computation.
Data no longer has to travel back across the network to a GPU at all, which slashes both traffic and communication latency and breaks through the traditional bandwidth bottleneck. That kind of optimization β protocol and switching hardware working as one β is something only NVIDIA can deliver today.
IV. NVLink Fusion: Turning Hardware into a Platform
As system architectures evolve, NVIDIA has introduced NVLink Fusion, which lets cloud vendors’ homegrown ASICs or custom CPUs plug directly into the NVIDIA NVLink Fabric through standard interfaces such as UCIe.
It breaks the old image of an exclusionary monopoly β but builds a deeper dependency in its place: cloud vendors are free to use their own chips, provided they adopt NVIDIA’s NVSwitch switches, rack architecture, and NVLink physical fabric. NVIDIA has thus gone from a “chip vendor” to the parent platform of the entire AI data center infrastructure. Hardware itself has become the platform.
V. Conclusion
The thing to watch next is the transition from Blackwell’s copper backplane to Rubin’s cross-rack silicon-photonics switching. If yields for high-precision optical-alignment packaging ramp more slowly than expected, it could temporarily slow NVIDIA’s high-density rack deliveries.
With NVLink Fabric, NVIDIA has drawn the boundaries of the physical machine of the future β and locked them in. This system-level moat β built from materials science, in-network computing, and full-stack coordination β is far harder to reverse-engineer, and far harder for open-source ecosystems to erode, than CUDA software ever was.