A Principled AI Path to Spec-Driven Verification

I have seen a flood of verification announcements around directly reading product specs through LLM methods, and from there directly generating test plans and test suite content to drive verification. Conceptually automating this step makes a lot of sense. Carefully interpreting such specs even today is a largely manual task, requiring painstaking attention to detail in which even experienced and diligent readers can miss or misinterpret critical requirements. As always in any exciting concept, the devil is in the details. I talked with Dave Kelf (CEO at Breker) and Adnan Hamid (President and CTO at Breker) to better understand those details and the Breker approach.

The devil in the specs

We like to idealize specs as one or more finalized documents representing the ultimate description of what is required from the design. Of course that’s not reality, maybe never was but certainly not today where specs change rapidly, even during design, adapting to dynamically evolving market demands and implementation constraints.

A more realistic view according to Adnan is a base spec supplemented by a progression of engineering change notices (ECNs) each reflecting incremental updates to the prior set of ECNs on top of the base spec. Compound that with multiple product variants covering different feature extensions. As a writer I know that even if you can merge a set of changes on top of a base doc, the merge may introduce new problems in readability, even in critical details. Yet in review and updates, reviewers naturally incline to edits considered in a local context rather than trying to evaluate each edit in the context of the whole doc and spec variants on each update.

Worse yet in this fast-paced spec update cycle, clarity of explanation often takes a back seat to urgent partner demands, especially when multiple writers contribute to the spec. In the base doc and especially in ECNs developers/reviewer lean heavily on assumptions that a “reasonable” reader will not need in-depth explanations for basic details. “Basic” in this context is a very subjective judgement, often assuming more in-depth domain expertise as ECNs progress. In the domain we are considering this is specialized know-how. How likely is it that a trained LLM will have captured enough of that know-how? Adnan likes to point to the RISC-V specs as an example illustrating all these challenges, with a rich superstructure of extensions and ECNs which he finds can in places imply multiple possible interpretations that may not have been intended by the spec writers.

Further on the context point, expert DV engineers read the same specs but they also talk to design engineers, architects, apps engineers, marketing, to gather additional input which might affect expected use cases and other constraints. And they tap their own specialized expertise developed over years of building/verifying similar designs. None of this is likely to be captured in the spec or in the LLM model.

Automated spec analysis can be enormously valuable in consolidating ECNs with the base spec to generate a detailed and actionable test plan, but that must then be reviewed by a DV expert and corrected through additional prompts. The process isn’t hands-free but nevertheless could be a big advance in DV productivity.

The Breker strategy

Interestingly Breker test synthesis is built on AI methods created long before we had heard of LLMs or even neural nets. Expert systems and rule-based planning software were popular technologies found in early AI platforms from IBM and others and became the foundation of Breker test synthesis. Using these methods Breker software has already been very effective and in production for many years, auto-generating very sophisticated tests from (human-interpreted) high level system descriptions. Importantly, while the underlying test synthesis methods are AI-based they are much more deterministic than modern LLMs, not prone to hallucination.

Now for the principled part of the strategy. It is widely agreed that all effective applications of AI in EDA must build on a principled base to meet the high accuracy and reliability demands of electronic design. For example production EDA software today builds on principled simulation, principled place and route, principled multiphysics analysis. The same must surely apply to functional test plan definition and test generation for which quality absolutely cannot be compromised. Given this philosophy, Breker are building spec interpretation on top of their test synthesis platform along two branches.

The first branch is their own development around natural language processing (NLP), again not LLM-based. NLP also pre-dates LLMs, using statistical methods rather than the attention-based LLM machinery. Dave and Adnan did not share details, but I feel I can speculate a little without getting too far off track. NLP-based systems are even now popular in domain specific areas. I would guess that a constrained domain, a specialist vocabulary, and a presumed high level of expertise in the source content can greatly simplify learning versus requirements for general purpose LLMs. These characteristics could be an excellent fit in interpreting design behavior specs.

Dave tells me they already have such a system in development which will be able to read a spec and convert it into a set of Breker graphs. From that they can directly run test plan generation and synthesis, with support for debug, coverage, portability to emulation, all the capabilities that the principled base already provides. They can also build on the library of test support functions already in production, for coherency checking, RISC-V compliance and SoC readiness, Arm SocReady, power management and security root of trust checks.  A rich set of capabilities which don’t need to be re-discovered in detail in a general purpose LLM model.

The second branch is partnership with some of the emerging agentic ventures. In general, what I have seen among such ventures is either strong agentic expertise but early in building verification expertise, or strong verification expertise but early in building agentic expertise. A partnership between a strong agentic venture and a strong verification company, seems like a very good idea. Dave and Adnan believe that what they already have in their Breker toolset, combined with what they are learning in their internal development can provide a bridge between these platforms, combining full-blown LLM-based spec interpretation, and the advantages of principled test synthesis.

My takeaway

This is a direction worth following. All the excitement and startup focus in verification is naturally on LLMs but today accuracy levels are not yet where they need to be to ensure production quality test plans and test generation without expert DV guidance. We need to see increased accuracy and we need to gain experience in the level of expert effort required to bring auto-generated tests to signoff quality.

Breker is investing in LLM-based spec-driven verification, but like other established verification providers must meanwhile continue to support production quality tools and advance the capabilities of those tools. Their two-pronged approach to spec interpretation aims to have the best of both worlds – an effective solution for the near term through NLP interpretation and an LLM solution for the longer term.

You can learn more about the principled Breker test synthesis products HERE.