In the business history of Silicon Valley, few companies have built a commercial wall worth hundreds of billions of dollars out of a software ecosystem the way NVIDIA has. The wall is called CUDA.
For a long time, the industry held an unbreakable consensus: you buy NVIDIA for its hardware, and you stay for its software.
Yet history is full of irony. As the generative AI wave that NVIDIA itself set off goes deeper, an unexpected boomerang is accelerating back toward it: the more powerful AI becomes, the lower the barrier to rewriting and migrating CUDA code. The AI revolution led by NVIDIA is now weakening its own proudest software moat.
This is a hard-core contest over compilers, middleware, open-source forces, and AI’s own evolution.
1. The Iron Throne: What Makes CUDA NVIDIA’s Moat?
To understand how the wall falls, first understand how it was built.
Before 2006, a GPU (graphics processing unit) was just a gaming graphics card. If scientists wanted to run mathematical matrix operations on a GPU, they had to disguise math formulas as “graphics rendering instructions” and feed them to the card β an extremely painful process to write.
In 2006, NVIDIA launched CUDA (Compute Unified Device Architecture). Its core contribution was letting programmers write code to control GPU parallel computing directly in C/C++.
To promote CUDA, Jensen Huang made a decision that looked insanely loss-making at the time: every graphics card NVIDIA shipped (including thousand-yuan GeForce gaming cards) had to have a CUDA module built in. That decision proved extraordinarily farsighted!
It created a moat worth hundreds of billions for NVIDIA, with two decisive commercial results:
- Absolute monopoly on the talent base: over the past 18 years, countless university students, researchers, and independent developers have been able to learn CUDA at zero cost on their own gaming PCs. When these people graduated into large-model companies or cloud giants, they only knew how to use CUDA.
- Day-0 ecosystem lock-in: nearly all AI open-source papers and innovative large models (such as Transformer, Diffusion, and Sora) shipped their code on GitHub on day one written against CUDA by default.
With this, NVIDIA collected hardware gross margins above 75% β the famous “NVIDIA tax.” Want cheaper hardware? Sorry, you can’t leave CUDA.
The world has suffered under NVIDIA for too long; the old heaven is dead, ROCm shall rise!
2. Homage, Not Copying: How ROCm and CUDA Differ Under the Hood
To break NVIDIA’s monopoly, AMD launched the open-source ROCm (Radeon Open Compute platform) in 2016. Technically, the two share strikingly similar underlying logic, but their approaches to building a software stack run in opposite directions.
1. Hardware control: highly consistent
At the lowest level of chip control, CUDA and ROCm are both built on SIMT (single instruction, multiple threads) architecture. Their core concepts map almost one-to-one in physical hardware:
- NVIDIA’s Thread β AMD’s Work-item
- NVIDIA’s Warp (32 threads) β AMD’s Wavefront (32/64 threads)
- NVIDIA’s Block β AMD’s Work-group
So whether CUDA or ROCm, the mathematical logic for optimizing matrix operations and controlling GPU memory caching is essentially interchangeable.
2. Software ecosystem: closed black box vs. open standards
The real difference lies in compilers and how code is generated:
CUDA’s NVCC compiler: NVIDIA built its own fully closed-source NVCC. It compiles code into a proprietary virtual intermediate language, PTX, which is then translated in real time by a closed-source driver into machine code for a specific graphics card. Its math acceleration libraries (such as cuDNN and TensorRT) have been tuned as black boxes for 18 years, and outsiders cannot peek inside.
ROCm’s LLVM ecosystem (the open highway): AMD did not build a compiler from scratch. Instead, it embraced the industry-standard open-source compiler framework LLVM. AMD developed HIP (Heterogeneous-Compute Interface for Portability) as a bridging layer for code.
AMD’s strategy was clear: use HIP to provide a “one-click translation tool (hipify)” that automatically translates existing CUDA code into HIP code, aiming for “write once, run anywhere.”
Yet for years, ROCm’s market share stayed extremely low. The reason was not hardware specs, but the unbearable cost of migration. Early ROCm was full of compiler bugs and missing documentation, and because it lacked anything like NVIDIA’s user base, companies migrating CUDA to ROCm had to hire extremely expensive system-level engineers to hand-tune and chase down bizarre compiler errors. In an AI race where every second counts, no company wanted to bear that time cost.
Until the generative AI explosion completely changed the underlying rules of the game.
3. The Paradox: AI Is Tearing Down CUDA’s Walls
NVIDIA’s proud AI technology is becoming the most lethal “Trojan horse” against its own software moat. This erosion happens mainly through two paths:
1. Automatic rewriting: generative AI flattens the cost of code migration
Work that once took a top expert team months to complete β rewriting and tuning “CUDA to ROCm/HIP” code β is now being flattened by large language models (such as Claude 3.5 and GPT-4o) to a matter of minutes.
Modern AI coding models have a perfect grasp of low-level abstract syntax trees (ASTs) and GPU memory alignment. AI can easily identify proprietary operators in CUDA code, and it can do more than syntactic replacement: it can automatically rewrite highly optimized, bug-free HIP operators tuned to AMD’s hardware characteristics.
This forms the most ironic commercial loop in tech history:
NVIDIA sells expensive chips β tech giants buy them and train powerful generative AI β the giants use that AI to automatically rewrite and migrate CUDA code to ROCm β the giants switch en masse to cheaper AMD chips β free of NVIDIA
2. Dimensionality reduction: the rise of unified middleware (Triton)
Even more lethal than AI writing code is the architectural evolution of AI infrastructure itself β the rapid rise of middleware represented by OpenAI-led Triton.
In the past, deep learning frameworks (such as PyTorch) needed large amounts of CUDA operators written specifically for NVIDIA at the bottom layer. Triton, released by OpenAI, is a minimalist, Python-based open-source compiler. Its goal is to let ordinary programmers write Python-level simple code, while the Triton compiler automatically handles low-level parallelism and memory management.
Most critically, Triton was designed from the start with backends for both NVIDIA and AMD.
As large-model vendors such as OpenAI and Meta gradually shift their core models from calling CUDA directly to writing through Triton, the underlying hardware starts to become fully transparent and replaceable. The CUDA software wall that NVIDIA so carefully built is being dismantled from within by middleware like Triton.
4. The Counterattack: NVIDIA’s Defense and Response
Faced with its software moat being eroded in reverse by AI and the open-source ecosystem, NVIDIA’s competitive strategy has shifted significantly, pushing its defensive line from “single-card software” toward physical limits that are much harder to breach.
1. From “selling chips” to “selling data center systems”
As the single-card software barrier is gradually flattened, NVIDIA is raising the wall higher on interconnect at hyperscale-cluster scale.
- Large-model training has entered the era of ten thousand, even a hundred thousand cards, where communication latency between chips matters more than single-card compute.
- Through its proprietary NVLink protocol, NVSwitch chips, and the InfiniBand (IB) networking technology gained from acquiring Mellanox, NVIDIA weaves tens of thousands of graphics cards into one extremely efficient “super brain.” This kind of ten-thousand-card physical network topology, and the extreme integration of high-frequency communication hardware and software behind it, is absolutely not something that can be crossed just by having a large model “rewrite a few lines of code.”
2. The terrifying “Moore’s Law” pace suppression (a time-window war)
NVIDIA is adopting a rarely seen high-intensity R&D cadence, pushing hardware iteration to one generation per year (from Hopper to Blackwell, then to the Rubin architecture).
Even if AI can migrate CUDA code to AMD hardware perfectly and at zero cost, if NVIDIA’s new-generation hardware still pulls more than a generation ahead in absolute performance, rational companies will still pay NVIDIA’s hefty premium to seize the critical time-to-market window for getting their models out.
5. Outlook: The Endgame of a Two-Horse Race
As the software barrier dissolves, the second half of AI chip competition is shifting from the old “ecosystem monopoly war” back to the purest physical contest over “hardware price-performance, power efficiency, and large-scale interconnect capability.”
NVIDIA: evolving into a system-level platform and cloud giant
NVIDIA’s short-term position remains hard to shake. Although the single-card CUDA barrier is lowering, with its full-stack data center networking (NVLink) and its terrifying one-generation-per-year iteration speed, it will still firmly hold the very top, most performance-obsessed hyperscale AI training market. NVIDIA’s future role will look more like a “super general contractor for AI infrastructure.”
AMD: a historic breakout in inference and enterprise markets
For AMD, this is the best dividend period in its history. As global large models gradually move from the “training phase” to the “large-scale commercial inference phase,” market demand for compute is shifting from “performance at any cost” to “the ultimate in dollars-per-performance and power efficiency.”
With PyTorch, Triton, and the spread of AI automatic migration tools, ROCm’s software disadvantage is being rapidly leveled. AMD’s Instinct series chips, with their larger memory capacity and high price-performance, will greatly eat into the inference compute market of cloud giants and traditional enterprise private deployments.
Conclusion: AI will not destroy NVIDIA, but AI will liberate the hardware market
This holy fire of AI, lit by NVIDIA’s own hand, will in the near future lift the CUDA shackles from other hardware vendors, returning the entire chip industry to a healthy competitive track centered on physical performance and innovation efficiency. Is that not a beautiful wish?! (Doggy)