Why’s Nvidia such a beast? It’s that CUDA thing.

[siliconbruh999]

Dynamo is open source as well. But more importantly, it offers dynamic reallocation and tuning of resources for max throughout or min token latency for each model inference instance running in an entire data center to optimize operations as models go through different phases (pre fill, token generation).

Cuda is not you can't run it without cuda can you?

 

[KevinK]

AMD runs the open-source precursor to Dynamo, the Triton inference server, using RoCM on the backend.

Triton Inference Server with vLLM on AMD GPUs — ROCm Blogs

This blog provides a how-to guide on setting up a Triton Inference Server with vLLM backend powered by AMD GPUs, showcasing robust performance with several LLMs

rocm.blogs.amd.com

So your starting assumption is likely wrong.