How Good is Your Verification?

The traditional way for analyzing the effectiveness of testing in the software world and in the RTL world is code coverage. Make sure that every line of code is executed. This is a pretty crude measure since even 100% code coverage doesn’t mean that all the condition has really been tested but it is certainly necessary–after all if a line of code is never executed then there is no way to know if it is correct or not.

In the manufacturing test world the criterion is to look at fault coverage. Every signal is considered to be stuck at 0 and 1 and the percentage of these faults that actually propagate to the outputs is calculated. This isn’t perfect since other types of faults can exist (two signals shorted together for example) but again it is a good starting point and there is a good chance that other faults will be detected if the fault coverage is good.

A better way to do things would be to take the design, add a bug, run the test and see if the added bug was detected by the test. Repeat for lots of bugs and you start to get a good idea of how effective your verification is. This gives you an objective metric as to what percentage of the injected bugs are found and, moreover, by looking at which bugs are missed allows the verification tests to be strengthened in the appropriate areas.

Certitude does just this, automatically injecting bugs into the design, running the simulation and then seeing how many of the injected bugs are flagged. For example, it might change “a=b|c” in your original code into “a=b&c”.

This helps with the closure challenge: is it safe to sign off the RTL? Since it is impossible to ever exhaustively test a design, this is always somewhat of a judgement call. But Certitude gives an objective measure of stimulus and checker completeness to support this signoff decision, along with pointers to specific holes to accelerate the closure process by directing incremental efforts to the areas requiring additional attention.

The latest version has been further enhanced by adding new fault types that better represent typical SoC failures. Fault dropping has been expanded to remove redundancy in the results. Results can now be ranked based on the impact of the fault, directing the user to where to analyze first.

Certitudeproduct page