The Double-Edged Sword of AI Processors: Batch Sizes, Token Rates, and the Hardware Hurdles in Large Language Model Processing

Unlike traditional software programming, AI software modeling represents a transformative paradigm shift, reshaping methodologies, redefining execution processes, and driving significant advancements in AI processors requirements.

Software Programming versus AI Modeling: A Fundamental Paradigm Shift

Traditional Software Programming
Traditional software programming is built around crafting explicit instructions (code) to accomplish specific tasks. The programmer establishes the software’s behavior by defining a rigid set of rules, making this approach ideal for deterministic scenarios where predictability and reliability are paramount. As tasks become more complex, the codebase often grows in size and complexity.

When updates or changes are necessary, programmers must manually modify the code—adding, altering, or removing instructions as needed. This process provides precise control over the software but limits its ability to adapt dynamically to unforeseen circumstances without direct intervention from a programmer.

AI Software Modeling
AI software modeling represents a fundamental shift in how to approach problem solving. AI software modeling enables systems to learn patterns from data through iterative training. During training, AI analyzes vast datasets to identify behaviors, then applies this knowledge in the inference phase to perform tasks like translation, financial analysis, medical diagnosis, and industrial optimization.

Using probabilistic reasoning, AI makes predictions and decisions based on probabilities, allowing it to handle uncertainty and adapt. Continuous fine-tuning with new data enhances accuracy and adaptability, making AI a powerful tool for solving complex real-world challenges.

The complexity of AI systems lies not in the amount of written code but in the architecture and scale of the models themselves. Advanced AI models, such as large language models (LLMs), may contain hundreds of billions or even trillions of parameters. These parameters are processed using multidimensional matrix mathematics, with precision or quantization levels ranging from 4-bit integers to 64-bit floating-point calculations. While the core mathematical operations, namely, multiplications and additions (MAC), are rather simple, they are performed millions of times across large datasets with all parameters processed simultaneously during each clock cycle.

Software Programming versus AI Modeling: Implications on Processing Hardware

Central Processing Unit (CPU)
For decades, the dominant architecture used to execute software programs has been the CPU, originally conceptualized by John von Neumann in 1945. The CPU processes software instructions sequentially—executing one line of code after another—limiting its speed to the efficiency of this serial execution. To improve performance, modern CPUs employ multicore and multi-threading architectures. By breaking down the instruction sequence into smaller blocks, these processors distribute tasks across multiple cores and threads, enabling parallel processing. However, even with these advancements, CPUs remain limited in their computational power, lacking the enormous parallelism required to process AI models.

The most advanced CPUs achieve computational power of a few GigaFLOPS and feature memory capacities reaching a few TeraBytes in high-end servers, with memory bandwidths peaking at 500 GigaBytes per second.

AI Accelerators
Overcoming CPU limitations requires a massively parallel computational architecture capable of executing millions of basic MAC operations on vast amounts of data in a single clock cycle.

Today, Graphics Processing Units (GPUs) have become the backbone of AI workloads, thanks to their unparalleled ability to execute massively parallel computations. Unlike CPUs, which are optimized for general-purpose tasks, GPUs prioritize throughput, delivering performance in the range of petaFLOPS—often two orders of magnitude higher than even the most powerful CPUs.

However, this exceptional performance comes with trade-offs, particularly depending on the AI workload: training versus inference. GPUs can experience efficiency bottlenecks when handling large datasets, a limitation that significantly impacts inference but is less critical for training. LLMs like GPT-4, OpenAI’s o1/o3, Llama 3-405B, and DeepSeek-V3/R1 can dramatically reduce GPU efficiency. A GPU with a theoretical peak performance of one petaFLOP may deliver only 50 teraFLOPS when running GPT-4. While this inefficiency is manageable during training, where completion matters more than real-time performance, it becomes a pressing issue for inference, where latency and power efficiency are crucial.

Another major drawback of GPUs is their substantial power consumption, which raises sustainability concerns, especially for inference in large-scale deployments. The energy demands of AI data centers have become a growing challenge, prompting the industry to seek more efficient alternatives.

To overcome these inefficiencies, the industry is rapidly developing specialized AI accelerators, such as application-specific integrated circuits (ASICs). These purpose-built chips offer significant advantages in both computational efficiency and energy consumption, making them a promising alternative for the next generation of AI processing. As AI workloads continue to evolve, the shift toward custom hardware solutions is poised to reshape the landscape of artificial intelligence infrastructure. See Table I.

Attributes	Software Programming	AI Software Modeling
Application Objectives	Deterministic and Targeted Tasks	PredictiveAI and GenerativeAI
Flexibility/Adaptability	Rule-based and Rigid	Data-driven Learning and Evolving
SW Development	Specific Programming Languages	Data Science, ML, SW Engineering
Processing Method	Sequential Processing	Non-linear, Heavily Parallel Processing
Processor Architecture	CPUs	GPUs and Custom ASICs

Table I summarizes the main differences between traditional software programming vis-à-vis AI software modeling.

Source: VSORA

Key and Unique Attributes of AI Accelerators

The massively parallel architecture of AI processors possesses distinct attributes not found in traditional CPUs. Specifically, two key metrics are crucial for the accelerator’s ability to deliver the performance required to process AI workloads, such as LLMs: batch sizes and token throughput. Achieving target levels for these metrics presents engineering challenges.

Batch Sizes and the Impact on Accelerator Efficiency

Batch size refers to the number of independent inputs or queries processed concurrently by the accelerator.

Memory Bandwidth and Capacity Bottlenecks

In general, larger batches improve throughput by better utilizing parallel processing cores. As batch sizes increase, so do memory bandwidth and capacity requirements. Excessively large batches can lead to cache misses and increased memory access latency, thus hindering performance.

Latency Sensitivity

Large batch sizes affect latency because the processor must handle significantly larger datasets simultaneously, increasing computation time. Real-time applications, such as autonomous driving, demand minimal latency, often requiring a batch size of one to ensure immediate response. In safety-critical scenarios, even a slight delay can lead to catastrophic consequences. However, this presents a challenge for accelerators optimized for high throughput, as they are typically designed to process large batches efficiently rather than single-instance workloads.

Continuous Batching Challenges
Continuous batching is a technique where new inputs are dynamically added to a batch as processing progresses, rather than waiting for a full batch to be assembled before execution. This approach reduces latency and improves throughput. It may have an impact on time-to-first token, but provided that the scheduler can handle the execution it achieves higher overall efficiency.

Token Throughput and Its Computational Impact

Token throughput refers to the number of tokens—whether words, sub-words, pixels, or data points—processed per second. It depends on input token sizes and output token rates, requiring high computational efficiency and optimized data movement to prevent bottlenecks.

Token Throughput Requirements
Key to defining token throughput in LLMs is the time to first token output, namely low latency achieved through continuous batching to minimize delays. For traditional LLMs, the output rate must exceed human reading speed, while for agentic AI that relies on direct machine-to-machine communication, maintaining high throughput is critical.

Traditional Transformers vs Incremental Transformers
Most LLMs, such as OpenAI-o1, LLAMA, Falcon, and Mistral, use transformers, which require each token to attend to all previous tokens. This leads to high computational and memory costs. Incremental Transformers offer an alternative by computing tokens sequentially rather than recomputing the full sequence at every step. This approach improves efficiency in streaming inference and real-time applications. However, it requires storing intermediate state data, increasing memory demands and data movement, which impacts throughput, latency, and power consumption.

Further Considerations
Token processing also presents several challenges. Irregular token patterns, such as varying sentence and frame lengths, can disrupt optimized hardware pipelines. Additionally, in autoregressive models, token dependencies can cause stalls in the processing pipeline, reducing the effective utilization of computational resources.

Overcoming Hurdles in Hardware Accelerators
In stark contrast to the CPU that has undergone a remarkable evolutionary journey over the past 70 years, AI accelerators are still in their formative stage, with no established architecture capable of overcoming  all the hurdles in meeting the computational demands of LLMs.

The most critical bottleneck is memory bandwidth, often referred to as the memory wall. Large batches require substantial memory capacity to store input data, intermediate states and activations, while demanding high data transfer bandwidth. Achieving high token throughput depends on fast data transfer between memory and processing units. When memory bandwidth is insufficient, latency increases, and throughput declines. These bottlenecks become a major constraint on computing efficiency, limiting the actual performance to a fraction of the theoretical maximum.

Beyond memory constraints, computational bottlenecks pose another challenge. LLMs rely on highly parallelized matrix operations and attention mechanisms, both of which demand significant computational power. High token throughput further intensifies the need for fast processing performance to maintain smooth data flow.

Data access patterns in large batches introduce additional complexities. Irregular access patterns can lead to frequent cache misses and increased memory access latencies. To sustain high token throughput, efficient data prefetching and reuse strategies are essential to minimize memory overhead and maintain consistent performance.

Addressing these challenges requires innovative memory architectures, optimized dataflow strategies, and specialized hardware designs that balance memory and computational efficiency.

Overcoming the Memory Wall
Advancements in memory technologies, such as high-bandwidth memory (HBM)—particularly HBM3, which offers significantly higher bandwidth than traditional DRAM—help reduce memory access latency. Additionally, larger and more intelligent on-chip caches enhance data locality and minimize reliance on off-chip memory, mitigating one of the most critical bottlenecks in hardware accelerators.

One promising approach involves modeling the entire cache memory hierarchy with a register-like structure that stores data on a single clock cycle rather than requiring tens of clock cycles. This method optimizes memory allocation and deallocation for large batches while sustaining high token output rates, significantly improving overall efficiency.

Enhancing Computational Performance
Specialized hardware accelerators designed for LLM workloads, such as matrix multiplication units and attention engines, can dramatically boost performance. Efficient dataflow architectures that minimize unnecessary data movement and maximize hardware resource utilization further enhance computational efficiency. Mixed-precision computing, which employs lower-precision formats like FP8 where applicable, reduces both memory bandwidth requirements and computational overhead without sacrificing model accuracy. This technique enables faster and more efficient execution of large-scale models.

Optimizing Software Algorithms
Software optimization plays a crucial role in fully leveraging hardware capabilities. Highly optimized kernels tailored to LLM operations can unlock significant performance gains by exploiting hardware-specific features. Gradient checkpointing reduces memory usage by recomputing gradients on demand, while pipeline parallelism allows different model layers to be processed simultaneously, improving throughput.

By integrating these hardware and software optimizations, accelerators can more effectively handle the intensive computational and memory demands of large language models.

About Lauro Rizzatti

Lauro Rizzatti is a business advisor to VSORA, an innovative startup offering silicon IP solutions and silicon chips, and a noted verification consultant and industry expert on hardware emulation.

Also Read:

A Deep Dive into SoC Performance Analysis: Optimizing SoC Design Performance Via Hardware-Assisted Verification Platforms

A Deep Dive into SoC Performance Analysis: What, Why, and How

SystemReady Certified: Ensuring Effortless Out-of-the-Box Arm Processor Deployments

Share this post via:

• 
• 
• 
•