Google’s 400,000-Chip Monster Tensor Processing Unit Just Destroyed NVIDIA's Future!

[KevinK]

Newbie question: Is the infrastructure cost of AI partly related to inefficient coding languages like Python or PyTorch? Is efficient coding (which we glimpsed but didn't really understand with DeepSeek) something they do in China but not the USA?

My take is that Python and PyTorch both rely on well-optimized underlying application-specific and hardware-optimized packages that are tuned in the underlying native "dataflow assembly code" (CUDA stuff for NVIDIA, ROCM for AMD, etc.). DeepSeek identified some great areas for optimization - model (multi headed latent attention), model partitioning (prefill/decode disaggregation) and system-level communication, and improved them at mostly the low level (not via packages).

DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model

We present DeepSeek-V2, a strong Mixture-of-Experts (MoE) language model characterized by economical training and efficient inference. It comprises 236B total parameters, of which 21B are activated for each token, and supports a context length of 128K tokens. DeepSeek-V2 adopts innovative...

arxiv.org

I don't know about other HW suppliers, but NVIDIA has created new system infrastructure inside Dynamo, plus worked on the three main model serving engines that support PyTorch, SGLang, vLLM, and TensorRT-LLM, to implement these DeepSeek optimizations, and more, in the underlying packages and architecture for inference serving with NVIDIA hardware.

So Python and PyTorch need help be fast and efficient with new models and hardware. DeepSeek mostly optimized at low level, but NVIDIA and associated developers followed up to make the optimizations accessible to all, through mostly open source.

 

[blueone]

Newbie question: Is the infrastructure cost of AI partly related to inefficient coding languages like Python or PyTorch? Is efficient coding (which we glimpsed but didn't really understand with DeepSeek) something they do in China but not the USA?

This is actually not a newbie question, it's very good one.

First, the simple stuff. PyTorch and Python are related only because PyTorch uses Python-similar syntax in its programming interface. The underlying layers of software PyTorch uses to interface with GPUs and TPUs is highly tuned code, from what I can tell usually C or C++. So the notion that using PyTorch implies an inefficient GPU/TPU interface doesn't appear to be accurate.

Regarding my negativity about Python as a programming language... I confess that I'm a high performance software purist, and I don't like programming languages that are interpreted (meaning, they are not compiled into processor-specific assembler code), or use a "virtual machine" (Java is another example) which translates and executes the programming language instructions into run-time subroutines. And Python is also a member of the class of programming languages which are called "memory safe", in which dynamic memory allocation and deallocation is managed by a compiler-defined memory manager, and these memory managers usually use a single-threaded data structure called a "heap" to manage memory segments and assignments. And background "garbage collection" for dynamically-deallocated memory areas to put them back in the heaps. Talk about inefficient! The cloud companies love their memory-safe languages, like Rust (Microsoft), though at least Rust has an "unsafe" mode for high performance programming. Google invented a language called Go, which is also memory-safe, though considered less safe than Rust, but Go does appear to have a "secret" unsafe mode too.

DeepSeek did a lot of optimizations, not all of which are open for examination, but they did include controlling parallelism in PTX for Nvidia, and minimizing data copies and communications inefficiencies, which is common in network drivers. (In the networking world the ultimate in reducing data copies has been available for decades. It's called Remote Direct Memory Access, or RDMA, a capability widely used by Nvidia in Ethernet and InfiniBand.)

The reason most teams outside of China do not go as deep as DeepSeek did is because they focus on the models, training, and inference, and depend on a higher performance hardware to get the response times they need. The US restricted China from buying the best AI processors, so they had to get more creative with classic software tuning efficiency tricks. Not to slight the DeepSeek accomplishments in any way, but I think that's all there is to it. As the saying goes, necessity is the mother of invention.

 

[KevinK]

which we glimpsed but didn't really understand with DeepSeek

If you are really interested in the details - here are some links on how SGLang, vLLM and TensorRT LLM implement some of the DeepSeek optimizations for use by all.

Deploying DeepSeek with PD Disaggregation and Large-Scale Expert Parallelism on 96 H100 GPUs | LMSYS Org

<p>DeepSeek is a popular open-source large language model (LLM) praised for its strong performance. However, its large size and unique architecture, which us...

lmsys.org

Scaling DeepSeek-style MoEs with vLLM and llm-d using Wide EP | Red Hat Developer

Learn how to deploy and scale Mixture of Experts (MoE) models using vLLM's new execution model and llm-d's intelligent Kubernetes-native inference framework

developers.redhat.com

https://www.perplexity.ai/hub/blog/lower-latency-and-higher-throughput-with-multi-node-deepseek-deployment

 

[Xebec]

@benb -
A small add-on to the excellent commentary above -

PyTorch *can* be very inefficient if used improperly, and that can skew some of the perception on PyTorch's efficiency.

Real world example of improper use causing 10X slowdown, (mainly) due to frequent loads from disk instead of RAM: https://stackoverflow.com/questions...h-code-10x-slower-than-the-tensorflow-version

Best practices for running PyTorch efficiently: https://docs.pytorch.org/tutorials/recipes/recipes/tuning_guide.html