Semiwiki on LinkedIn Semiwiki on Facebook Semiwiki on Twitter Register / Log in 
Array
(
    [content] => 
    [params] => Array
        (
            [0] => /forum/threads/just-how-deep-is-nvidias-cuda-moat-really.21703/
        )

    [addOns] => Array
        (
            [DL6/MLTP] => 13
            [Hampel/TimeZoneDebug] => 1000070
            [SV/ChangePostDate] => 2010200
            [SemiWiki/Newsletter] => 1000010
            [SemiWiki/WPMenu] => 1000010
            [SemiWiki/XPressExtend] => 1000010
            [ThemeHouse/XLink] => 1000970
            [ThemeHouse/XPress] => 1010570
            [XF] => 2021770
            [XFI] => 1050270
        )

    [wordpress] => /var/www/html
)

Just how deep is Nvidia's CUDA moat really?

X

XYang2023

Well-known member
Analysis Nvidia is facing its stiffest competition in years with new accelerators from Intel and AMD that challenge its best chips on memory capacity, performance, and price.

However, it's not enough just to build a competitive part: you also have to have software that can harness all those FLOPS – something Nvidia has spent the better part of two decades building with its CUDA runtime.

Nvidia is well established within the developer community. A lot of codebases have been written and optimized for its specific brand of hardware, while competing frameworks for low-level GPU programming are far less mature. This early momentum is often referred to as "the CUDA moat."
But just how deep is this moat in reality?

www.theregister.com

Nvidia's CUDA moat may not be as impenetrable as you think

Not as impenetrable as you might think, but still more than Intel or AMD would like
www.theregister.com www.theregister.com
 
The Register is mildly entertaining and useful for hearing industry rumors and hearsay. For anything technical, not at all. CUDA and the alternatives is a very technical discussion, as is what happens under the covers when you really want to work in PyTorch on various AI processors and their software stacks. That article is mostly :poop:
 
blueone said:
The Register is mildly entertaining and useful for hearing industry rumors and hearsay. For anything technical, not at all. CUDA and the alternatives is a very technical discussion, as is what happens under the covers when you really want to work in PyTorch on various AI processors and their software stacks. That article is mostly :poop:
Click to expand...
I ordered a B580 and currently have both the 4090 and 3090. I plan to compare them based on my use case. If the B580 proves sufficient, I may no longer need the NVIDIA cards for certain tasks. As the article mentioned, many people nowadays tend to focus on working at the PyTorch level.
 
XYang2023 said:
I ordered a B580 and currently have both the 4090 and 3090. I plan to compare them based on my use case. If the B580 proves sufficient, I may no longer need the NVIDIA cards for certain tasks. As the article mentioned, many people nowadays tend to focus on working at the PyTorch level.
Click to expand...
Good for you intel last week allowed use of Intel gpu in pytorch directly like Nvidia hope they continue to mature their SW
 
siliconbruh999 said:
Good for you intel last week allowed use of Intel gpu in pytorch directly like Nvidia hope they continue to mature their SW
Click to expand...
Did you mean this?
www.infoq.com

PyTorch 2.5 Release Includes Support for Intel GPUs

The PyTorch Foundation recently released PyTorch version 2.5, which contains support for Intel GPUs. The release also includes several performance enhancements, such as the FlexAttention API, TorchInductor CPU backend optimizations, and a regional compilation feature which reduces compilation...
www.infoq.com www.infoq.com


Pytorch should update their download options for Intel GPUs:
pytorch.org

PyTorch

pytorch.org pytorch.org

 
Last edited:
XYang2023 said:
Analysis Nvidia is facing its stiffest competition in years with new accelerators from Intel and AMD that challenge its best chips on memory capacity, performance, and price.

However, it's not enough just to build a competitive part: you also have to have software that can harness all those FLOPS – something Nvidia has spent the better part of two decades building with its CUDA runtime.

Nvidia is well established within the developer community. A lot of codebases have been written and optimized for its specific brand of hardware, while competing frameworks for low-level GPU programming are far less mature. This early momentum is often referred to as "the CUDA moat."
But just how deep is this moat in reality?

www.theregister.com

Nvidia's CUDA moat may not be as impenetrable as you think

Not as impenetrable as you might think, but still more than Intel or AMD would like
www.theregister.com www.theregister.com
Click to expand...
NVDA moat is very deep in large scale AI training. It stems primarily from the ecosystem of developer expertise rather than just CUDA itself. While CUDA's performance and usability matter, the true moat lies in the accumulated knowledge of how to handle infrastructure challenges when orchestrating training across massive GPU clusters. This specialized expertise in managing tens of thousands of GPUs simultaneously - dealing with issues like distributed computing, memory management, and system optimization - is much harder for competitors to replicate than the technical aspects of GPU programming interfaces.

That being said, in AI inference, these challenges don't really exist.

If you believe AI load is largely transitioning to inference, then its moat is fading, fast.
 
kevin01 said:
NVDA moat is very deep in large scale AI training. It stems primarily from the ecosystem of developer expertise rather than just CUDA itself. While CUDA's performance and usability matter, the true moat lies in the accumulated knowledge of how to handle infrastructure challenges when orchestrating training across massive GPU clusters. This specialized expertise in managing tens of thousands of GPUs simultaneously - dealing with issues like distributed computing, memory management, and system optimization - is much harder for competitors to replicate than the technical aspects of GPU programming interfaces.

That being said, in AI inference, these challenges don't really exist.

If you believe AI load is largely transitioning to inference, then its moat is fading, fast.
Click to expand...
I think Intel is also competent in that regard. They are likely waiting for Falcon Shores to address training workloads. For inference, Gaudi is sufficient for now. Additionally, for many use cases, people do not train models from scratch; instead, they fine-tune existing models with their own data. In such instances, you don't need an army of GPUs.
 
XYang2023 said:
Did you mean this?
www.infoq.com

PyTorch 2.5 Release Includes Support for Intel GPUs

The PyTorch Foundation recently released PyTorch version 2.5, which contains support for Intel GPUs. The release also includes several performance enhancements, such as the FlexAttention API, TorchInductor CPU backend optimizations, and a regional compilation feature which reduces compilation...
www.infoq.com www.infoq.com


Pytorch should update their download options for Intel GPUs:
pytorch.org

PyTorch

pytorch.org pytorch.org

Click to expand...
Yes it will mature nicely in time for celestial and Falcon shores
 
XYang2023 said:
I ordered a B580 and currently have both the 4090 and 3090. I plan to compare them based on my use case. If the B580 proves sufficient, I may no longer need the NVIDIA cards for certain tasks. As the article mentioned, many people nowadays tend to focus on working at the PyTorch level.
Click to expand...
I would like to hear more about how the B580 works out for you, if you're able to share. I bought one for my niece as a gaming GPU upgrade (arrives later this week), but I'm debating getting one multipurpose uses for myself.
 
Xebec said:
I would like to hear more about how the B580 works out for you, if you're able to share. I bought one for my niece as a gaming GPU upgrade (arrives later this week), but I'm debating getting one multipurpose uses for myself.
Click to expand...
I plan to compare the B580 with my 3090 and 4090 in terms of PyTorch performance.
 
BruceA said:
One got Lisa the other got PSO, BK, Bob and that was that.

Could have been a very different world if after CRB Andy and the BoD had selected Mike Splinter or Pat. It’s hard for a non founder to pivot a goldmine
Click to expand...
Yeah it's even harder to find competent people like Grove or in AMDs case Lisa
 
kevin01 said:
NVDA moat is very deep in large scale AI training. It stems primarily from the ecosystem of developer expertise rather than just CUDA itself. While CUDA's performance and usability matter, the true moat lies in the accumulated knowledge of how to handle infrastructure challenges when orchestrating training across massive GPU clusters. This specialized expertise in managing tens of thousands of GPUs simultaneously - dealing with issues like distributed computing, memory management, and system optimization - is much harder for competitors to replicate than the technical aspects of GPU programming interfaces.

That being said, in AI inference, these challenges don't really exist.
Click to expand...
I think you pegged it rightly - NVIDIA is going to remain tightly wired in on training and the real battle is inference. From what I'm seeing the new HW/SW decision criteria for inference are going to be TCO, tools for building an entire GenAI app / system, plus a minimum level of performance for a particular application. The moat is pretty small when it come to running and benchmarking off-the-shelf GenAI models thanks to PyTorch, etc. But bare models, even ones as good as GPT 4o are pretty useless to enterprises, without an entire application system wrapped around them. Guys like Grok and Cerebras are focused on fastest token rates and super low latency but suffer from high TCO - that's why they have moved to selling hosted services. AMS, Microsoft and Google are looking at their own chips with their own app development environments, and AMD bought a company to help develop application solutions for GenAI. NVIDIA seems to be focused on on creating an Enterprise environment with NIMs/NEM, though they just mentioned they might also offer hosted services.
 
KevinK said:
I think you pegged it rightly - NVIDIA is going to remain tightly wired in on training and the real battle is inference. From what I'm seeing the new HW/SW decision criteria for inference are going to be TCO, tools for building an entire GenAI app / system, plus a minimum level of performance for a particular application. The moat is pretty small when it come to running and benchmarking off-the-shelf GenAI models thanks to PyTorch, etc. But bare models, even ones as good as GPT 4o are pretty useless to enterprises, without an entire application system wrapped around them. Guys like Grok and Cerebras are focused on fastest token rates and super low latency but suffer from high TCO - that's why they have moved to selling hosted services. AMS, Microsoft and Google are looking at their own chips with their own app development environments, and AMD bought a company to help develop application solutions for GenAI. NVIDIA seems to be focused on on creating an Enterprise environment with NIMs/NEM, though they just mentioned they might also offer hosted services.
Click to expand...




 
You must log in or register to reply here.
Back
Top