OpenAI’s Jalapeño is a custom accelerator designed specifically for AI inference, particularly the low-latency, multi-chip workloads behind interactive and agentic systems. Its central idea is that useful performance cannot be reduced to peak floating-point operations or memory bandwidth. What matters is how quickly and efficiently the complete request finishes. Accordingly, OpenAI evaluates the chip using end-to-end latency, time between tokens, and tokens per joule across the full latency-throughput trade-off.
The ASIC combines substantial compute and memory capacity: up to 13.4 petaflops of MXFP4 matrix compute, 216 GiB of HBM4, and 15.4 TB/s of memory bandwidth within a 700-watt package. A complete 2,048-chip system reaches 27 exaflops of MXFP4 compute, 432 TiB of memory, and 32 PB/s of aggregate bandwidth. Yet the design argues that raw specifications are not enough. Although aggregate HBM bandwidth theoretically permits thousands of full-model weight reads per second, practical token rates remain far lower because data movement, synchronization, communication, and late-arriving operands leave compute units waiting.
Jalapeño addresses these bottlenecks through a spatial architecture. Each core slice is paired with an HBM slice, creating a fast, high-bandwidth local memory view. Specialized collective networks handle common cross-core communication patterns, while a more flexible network-on-chip supports general traffic. Software places tensors explicitly, performs computation locally, and uses shared paths only when necessary. Distributed control also avoids the costly global fences that can arise when independent cores must synchronize through centralized resources.
This organization reflects the changing demands within a single inference request. Prefill is attention-heavy and largely compute-bound. A speculative draft model is small, operates at extremely low batch sizes, and is highly sensitive to network latency. Verification and decoding combine attention, memory-bandwidth-intensive mixture-of-experts execution, and bursty communication. Rather than assigning these phases to separate accelerators and repeatedly transferring the key-value cache, Jalapeño keeps state local and activates different combinations of compute, memory, and networking resources. Unused blocks can be power-gated. OpenAI’s guiding observation is that “dark” silicon inside one package costs less than idle accelerators that continue consuming baseline HBM, networking, cooling, and rack resources.
At system scale, 128 chips form a local domain, while as many as 2,048 chips are connected through a two-level Clos network. Bandwidth is deliberately tapered: tensor parallelism receives higher bandwidth, expert parallelism requires less, and all traffic benefits from low latency. This makes the interconnect part of the computer rather than an accessory to otherwise independent chips.
On the public InferenceX benchmark, OpenAI reports Jalapeño on the performance-per-watt Pareto frontier for GPT-OSS 120B, DeepSeek R1 670B, and Kimi K2.5 1T. Against the selected GB200 or GB300 configurations, the deck reports 1.5–1.9 times greater peak throughput per kilowatt, 1.7–3.6 times lower end-to-end latency, and 2.1–4.1 times better token interactivity. At the previous system’s time-between-token point, the reported efficiency advantage ranges from 8.6 to 104.3 times. These comparisons use 8,000-token inputs, 1,000-token outputs, four-bit weights, package-TDP normalization, and vendor-optimized software. Importantly, Jalapeño uses single-token prediction while some baselines use multi-token prediction, making the disclosed methodology essential when interpreting the results.
Bottom line: Inference is becoming the dominant operational cost of AI, while agentic applications may invoke models repeatedly before completing one user task. Improvements in requests per second per watt translate into lower energy consumption, greater datacenter capacity, and faster interactive systems. Jalapeño also demonstrates the value of hardware-software-AI co-design: OpenAI moved from initial RTL to tape-out in nine months, used AI to optimize hardware blocks, and generated kernels that reportedly exceeded expert-written implementations. The broader implication is strategic: model developers can now shape silicon around real workloads, shortening the loop between algorithms, compilers, architecture, and deployment—and making frontier intelligence more economical to serve.
Source:
Also Read:
Semicon West: Transforming Tomorrow One Chip at a Time
HBM’s Vertical Ascent: Why Packaging Is Becoming the Heart of AI Memory
Share this post via:
Comparing Advanced Packaging from TSMC, Intel Foundry, and Samsung Foundry