Gaudi 3 is not bad.That’s why I question folks proposing lower level standards - I can understand it for HPC problems, but not for GenAI inference. I think the big competitive battle going on right now is going to be about inference cost / power per token at the data center level, for every leading model. The good news is that Llama has been added to MLPerf 5.0. The bad news is that the focus is still on performance, so they aren’t looking at cost/power per token yet.
NVIDIA Blackwell & AMD MI325X Showdown In Latest MLPerf Inference Benchmarks: B200 Shatters Records, Instinct Fights Against Hopper
NVIDIA & AMD have submitted latest MLPerf Inference performance benchmarks of their latest GPUs, including Blackwell B200 & Instinct MI325X.wccftech.com
Tech AMD’s Lisa Su has already vanquished Intel. Now she’s going after Nvidia
- Thread starter Daniel Nenni
- Start date
siliconbruh999
Well-known member
AMD has thrown so much Hardware at the problem it's hilarious just to loose to H100 as for low level you know that part of deepseks success was low level PTX assembly they utilized hardware fully.That’s why I question folks proposing lower level standards - I can understand it for HPC problems, but not for GenAI inference. I think the big competitive battle going on right now is going to be about inference cost / power per token at the data center level, for every leading model. The good news is that Llama has been added to MLPerf 5.0. The bad news is that the focus is still on performance, so they aren’t looking at cost/power per token yet.
NVIDIA Blackwell & AMD MI325X Showdown In Latest MLPerf Inference Benchmarks: B200 Shatters Records, Instinct Fights Against Hopper
NVIDIA & AMD have submitted latest MLPerf Inference performance benchmarks of their latest GPUs, including Blackwell B200 & Instinct MI325X.
Who knows ? Random commissioned benchmarks are a bit meaningless, especially without full disclosure of comparative environments. I’m thinking that MLPerf 5.0 is a far more reliable and trustworthy, transparent comparison. Unfortunately Intel has only done the hard work for Granite Rapids, not Gaudi 3 (yet) for the new LLM Llama benchmarks.
MLCommons Releases New MLPerf Inference v5.0 Benchmark Results - MLCommons
MLCommons' latest MLPerf Inference v5.0 results show Gen AI now the center of attention for performance engineering.
mlcommons.org
Yeah, but most of DeepSeek’s efficiency magic can be duplicated via smarter data center orchestration that does GPU planning, prefill/decode disaggregation, smart KV cache management / routing and communication between GPUs. Guess who has figured out how to make that work (hint 30x tokens/sec improvement on DeepSeek-R1)
Open source, so others can go that route.
siliconbruh999
Well-known member
Intel is the only company that submits ML Perf Result for CPU it feels weird vs ASIC/GPUsWho knows ? Random commissioned benchmarks are a bit meaningless, especially without full disclosure of comparative environments. I’m thinking that MLPerf 5.0 is a far more reliable and trustworthy, transparent comparison. Unfortunately Intel has only done the hard work for Granite Rapids, not Gaudi 3 (yet) for the new LLM Llama benchmarks.
MLCommons Releases New MLPerf Inference v5.0 Benchmark Results - MLCommons
MLCommons' latest MLPerf Inference v5.0 results show Gen AI now the center of attention for performance engineering.
If latency is not important, you can do batch processing with CPUs.
siliconbruh999
Well-known member
Won't CPU be better at latency sensitive task ?
Maybe latency is not the right word. The throughputs of using CPU for inferencing are limited in comparison to accelerators. But it can produce the same outputs given enough time.