Continuous Integration of UVM Testbenches

In recent years, one of the hot topics in chip design and verification has been continuous integration (CI). Like many innovations in hardware development, it was borrowed from software engineering and the programming world. The concept is simple: all code changes from all developers are merged back into the main development stream frequently, perhaps as often as every few hours. The program is then compiled and rebuilt to check that the changes are compatible, and a regression test suite is run to ensure that no bugs have been introduced. If the changed code contains new features, new tests for these features may be needed in the regression suite. This approach has been around for about 25 years and is adopted more every day.

This is in sharp contrast to the traditional development flow, in which programmers check out parts of the code for days and even weeks before checking it back in. Because multiple parts are evolving in parallel, they often get seriously out of sync and make integration a nightmare. CI finds incompatibilities incrementally as they pop up, making it easier to diagnose and fix any problems. Of course, programmers want to merge code that’s as clean as possible, so using an integrated development environment (IDE) with on-the-fly code checks, quick-fix suggestions, and refactoring capabilities is critical. Other types of code-checking tools such as lint, software static analysis, and security testing should also be run as part of the integration process. This whole flow should be highly automated.

Everything I just said about software is true for hardware as well. Testbenches compliant with the Universal Verification Methodology (UVM) are basically highly complex programs, and even SystemVerilog designs comprise huge amounts of RTL code. Just as with software, integration of design and verification code can be incredibly painful. I’m starting to see more EDA vendors talking about agile development and CI as a way to reduce debug effort and accelerate chip schedules. I turned to Cristian Amitroaie, CEO and co-founder of AMIQ EDA, to understand how all this works. Since AMIQ EDA is the industry leader in IDEs and lint, I figured that they must have at least a few users deploying their tools in CI flows.

Well, it turns out that there’s a whole lot going on in that area. For a start, Cristian reminded me of a press release that they issued about a year ago. It describes how the AMIQ EDA Design and Verification Tools (DVT) Eclipse IDE and Verissimo SystemVerilog Testbench Linter are used in CI flows by leading-edge companies, including Arm. Cristian said that they worked closely with users to develop this capability and to define best practice for CI. They also added new features to Verissimo to integrate tightly with bug tracking systems such as Bugzilla, project management tools, and revision control systems such as Github, Git, CVS, Subversion, and ClearCase.

For example, if the revision control system can identify which engineer changed which part of the code, Verissimo can extract this information and provide it for use in debugging build failures. I was impressed by Cristian’s description of how users can easily compare the latest changed files against those in the most recent successful testbench build and a master reference build. With CI and this level of insight into the changes, different parts of the code don’t drift far apart, and they be quickly brought into alignment when minor deviations occur.

Verissimo detects problems that would break the integration, but it catches them as soon as the code is merged so users don’t have to work backwards from testbench-level compilation errors. It’s interesting that some of the recent conversations Cristian and I have had are directly relevant to this process. The checks available include those for compliance with the latest versions of the SystemVerilog and UVM standards, including new features added and old features deprecated. Verissimo also checks for non-standard constructs supported by some EDA tools but not others, to ensure code portability across vendors. By the way, I’m focusing on the testbench since it’s so complex, but RTL code is a subset of full SystemVerilog so Verissimo and the CI flow work equally well for the design.

Cristian said that users deploying continuous integration run it automatically, in batch mode, so that checks are run as part of the merge/build/test process. When verification engineers wish to examine and fix errors detected by Verissimo, they use the advanced graphical features of DVT Eclipse IDE. Then he mentioned something that I found extremely interesting: AMIQ EDA has been running CI checks on the Github repository of the UVM reference implementation for more than a year. As a reminder, UVM consists of a standard document defining the library API and a reference SystemVerilog implementation of that library. Whenever someone changes the library, the changed code is checked within a few hours. The reports are available here. UVM development is rather quiet right now, but it will likely pick up at some point for the next revision and AMIQ EDA is already set up to contribute to that process.

It seems that this same approach could be used with other SystemVerilog and UVM design and verification IP available from public repositories. I asked Cristian about this, and he hinted that this was a future possibility. I’ll keep my eyes open and alert you all if I see wider propagation of their technology. In the meantime, it seems that AMIQ EDA has provided a solid solution to their users for applying CI to chip design and verification code. I look forward to learning more.

Also Read

What’s New with UVM and UVM Checking?

Why Would Anyone Perform Non-Standard Language Checks?

Does IDE Stand for Integrated Design Environment?