Exploiting agentic automation cost-effectively. Innovation in Verification

Frontier models (Claude, etc.) are very powerful but becoming expensive for high token counts. Methods to exploit a mix of models, including local open-weight models, are attracting attention. Paul Cunningham (GM, Verification at Cadence), Raúl Camposano (Silicon Catalyst, entrepreneur, former Synopsys CTO and lecturer at Stanford, EE292A) and I continue our series on research ideas. As always, feedback welcome.

The Innovation

This month’s pick is Universal Model Routing for Efficient LLM Inference. The authors are from Google. The paper (2025) has not (to my knowledge) been published outside of arXiv, however it already has 92 citations.

Frontier-based agentic methods are already expensive, as noted in a recent quote from the Wall Street Journal:

Companies are coming around to a radical idea: They don’t have to blow their budgets on artificial intelligence. Instead of relying on a single provider, they’re increasingly mixing lower-priced models, including some built in China, alongside products from OpenAI and Anthropic and shopping a la carte for AI. The shift is changing the economics of the industry …

Figuring out best ways to route reasoning between multiple models then becomes a central challenge. This paper provides insight.

Paul’s view

With agentic AI costs rising rapidly, “model routers” that intelligently route a prompt to an LLM which offers a good trade-off between cost and quality are getting a lot of attention. This month’s paper from Google Research tackles the problem of building a model router that can route to a dynamically changing set of LLMs. It’s a heavy read with a lot of formal proofs, but the core insights buried within are quite elegant.

A traditional approach model routing begins with some training set of <prompt, target response> pairs. Each LLM in a fixed pool of available LLMs is profiled on this training set and given an error score for its response vs. the target response. A small fast AI model (e.g. an MLP) is then trained to predict the error of each LLM for a given prompt, and a weighted sum of predicted error and cost is then used to rank LLMs and pick the winner.

Of course, the router must work for any prompt not only prompts in its training set. So the router is actually trained to predict error based on a numeric signature of the prompt (a sequence of numbers called a “feature vector”). This is the world of text embeddings, which strive to generate vectors such that two prompts with similar meanings have vectors with similar directions. In this paper, the authors use the Google Gecko embedding and a simple binary error function (0 if the response is an exact match to the target response, 1 otherwise).

The key idea this paper is to apply the same signature concept to LLMs: predict the response error based on the signature of the prompt and the signature of the LLM. Such a router can route to any LLM, so long as its signature is known.

The authors take all prompt signatures in the training set and cluster them into groups of ~50 prompts, using the angle between their feature vectors as the “distance” measure between them. The signature for an LLM is a vector of numbers, one for each cluster, denoting that LLM’s average response error over all training prompts in that cluster. To support a new LLM, the model router need only run inference once on it across the training set to generate it’s LLM signature. To route a prompt, pick the cluster whose center of gravity is nearest to its signature, and then pick the LLM with the best trade-off between its average error for that cluster (obtainable from its LLM signature), and its cost.

Results look solid, with the author’s solution, UniRoute, almost matching a static trained MLP-based router across all the benchmarks. Also, compared to a basic router that just takes a single consolidated average error for each LLM across the entire training set, UniRoute can route to 3-5x cheaper models on average for the same quality of responses. Awesome paper, well worth wading through all the proofs.

Raúl’s view

One of the simplest ways to reduce the cost of AI inference is to use a cheaper model whenever you can get away with it. An AI router decides which AI model answers which request: cheap models for simple queries, expensive ones only for complex tasks. The router trades off answer quality against inference cost. The idea sounds trivial, but in practice it is difficult.

Why not just build an AI router to pick the right model? You can, and people do, everyone is building AI routers…  An AI router can be trained to predict how well each candidate model will perform on a particular prompt. But models change, new models appear, existing models are upgraded or removed, their relative costs and capabilities vary over time, etc. For conventional routers adding a new model requires collecting new training data and retraining the router. If models are changed continuously, this becomes an increasingly unattractive proposition.

This is the problem addressed by Universal Model Routing for Efficient LLM Inference. Rather than building a router that is tightly coupled to a fixed collection of LLMs, the authors propose UniRoute, a model router that can route among LLMs that were not present during router training.  The paper represents the model by its behavior. A new model is tested on a small set of representative validation prompts to construct a “feature vector” describing where that model succeeds and fails. A new model can be “introduced” to the router without retraining the router itself. The practical implementation is intuitive. Prompts are clustered based on their representations, and each LLM is characterized by its average error on each cluster. When a new query arrives, the system picks the model expected to perform best on this type of problem, accounting for its cost. Adding a new model still has an onboarding cost: it must be evaluated once on a small representative validation set, but the router itself does not need to be retrained.

UniRoute was evaluated on several benchmarks, including settings with more than 30 unseen LLMs, and compared against several routing baselines. It consistently produced competitive quality-cost tradeoffs. The main result is illustrated by the deferral curves shown in figure 3 for EmbedLLM (others in the appendix). They represent the accuracy (Y-axes) for a given cost budget (X-axes) using new models it hasn’t seen. The accuracy/cost of each model is given by an “X”. Using a mixture of models, the router reaches roughly 0.68 accuracy at a cost proxy of about 40 billion parameters, better than the roughly 0.65 accuracy of the best individual model, which has about 70 billion parameters. Routing can do more than reduce cost: by exploiting the different strengths of different models, it can outperform every individual model in the pool.

The paper is clearly written for a machine-learning/AI research audience. The authors formulate the problem formally, with a mathematical treatment of optimal routing and theoretical bounds, the proofs are largely relegated to the appendix. A reader primarily interested in the economics of inference can skip much of the mathematical development and focus on the problem formulation and the experimental results. In fact, the underlying ideas are much easier to understand than the mathematical presentation.

As AI becomes a commodity service, it becomes increasingly important to orchestrate an entire portfolio of models, the frontier models alone are too expensive. That is why we’re reviewing this paper, it has nothing to do with circuit verification, but the problem it addresses is fundamentally an economic one.

Share this post via:

• 
• 
• 
•