Cocotb for Verification. Innovation in Verification

This time let’s see if we can stir up some lively debate. Cocotb isn’t new but it is an interesting alternative to mainstream testing methodologies. Paul Cunningham (GM, Verification at Cadence), Raúl Camposano (Silicon Catalyst, entrepreneur, former Synopsys CTO and lecturer at Stanford, EE292A) and I continue our series on research ideas. As always, feedback welcome.

The Innovation

This month’s pick is Analyzing AES Verification: A Comparative Study of UVM and Cocotb Approaches. The authors are from the PSG Institute of Technology and Applied Research in Tamilnadu, India. The paper is from the 2024 ICSSEECC conference and has no citations so far.

I must apologize for this month’s selection, not up to our usual standards. That’s on me. Paul subsequently found a better paper on cocotb which we may consider for a later post.

UVM is so dominant in functional verification that to mention any other possibility seems to invite reflexive opposition. But an alternative needn’t be a threat; it can add complementary value. UVM is certainly a fixture in production verification, however there are other verification needs where a flexible platform with a quick learning curve may fit better. This paper looks at relative strengths in UVM and Python-based cocotb in different contexts and finds pluses and minuses for both approaches.

Ultimately, familiarity for quick and easy testing in early RTL development may be the real advantage for cocotb, while still leaving heavy-duty verification to proven UVM flows.

Paul’s view

We wanted to give some airtime in our blog to cocotb (https://www.cocotb.org/). It’s an open-source Python-based alternative to UVM for testbench design. A few of our customers are loyal advocates for it, claiming how much more productive they are coding in Python, and how much easier it is to ramp up new college grads using it. This month’s paper is on comparing UVM with Cocotb to verify an AES crypto block. The paper is light on details – it shows terrible coverage from their UVM bench, with no explanation why, and has no data on the relative runtime performance of UVM vs. cocotb.

Stepping back from the paper and looking at the big picture, a simulation testbench is a software program to generate stimulus and monitor/check design behavior (design outputs, internal registers, internal signals). It can be written in any software programming language. SystemVerilog/UVM and C/C++ are the most common, Python works, Specman-e still has a loyal following, and there’s a growing following for Portable Stimulus (PSS).

The main advantage I can see from SystemVerilog/UVM is not the language itself, but the significant investment commercial EDA vendors have made to natively integrate support for it with our logic simulators. We’ve tuned compiled UVM code for the best possible runtime performance, and we’ve built very sophisticated constraint solvers to pick pseudo-random values with the best possible distributions (an NP-complete Boolean SAT problem). To the extent that a cocotb benchmark might lose to UVM, probably it would come down these reasons and nothing fundamental to cocotb itself.

Raúl’s view

As the title states, this paper compares Universal Verification Methodology (UVM), the SystemVerilog standard verification framework, and Cocotb, a Python verification framework, through just one case study verifying an Advanced Encryption Standard (AES) hardware implementation. Much of this short paper is spent surveying some literature (15 papers), explaining some AES basics and with a very high level UVM and Cocotb introduction. To compare the approaches, the authors built a UVM testbench with drivers, monitors, sequencers, scoreboard, and reference model for AES. For Cocotb, they created Python-based tests using coroutines and Python’s Crypto.Cipher library as a reference model. The key results of the comparison are:

• Coverage: For code coverage, Cocotb 89.55% is slightly higher overall than UVM at 87.49%. For functional coverage, Cocotb 100% of expected cases; UVM covered 47 out of 64 cases. It is unclear why the UVM coverage is so low and how the authors built the testbenches. In a typical design environment this would not be the case.
• Simulation Time is stated as 1000ns for UVM and 10000.5 ns Cocotb, most likely they are referring to the amount of time simulated, it is stated that UVM is superior.
• Build Time: Cocotb required less effort due to Python’s built-in features and simpler synchronization, although no concrete times are given.
• Flexibility, automation: Python (Cocotb) offers richer data types and extensive third-party libraries for automation, computation, and visualization.

For readers who are not familiar with Cocotb, this case study presents a simple overview of its strengths and weaknesses. It is not a real comparison with UVM, but since Python is the “lingua franca” of artificial intelligence, Cocotb may play an increasing role in functional verification.

Also Read:

A Big Step Forward to Limit AI Power Demand

Streamlining Functional Verification for Multi-Die and Chiplet Designs

Chiplets and Cadence at #62DAC