Verification Completion: When is Enough Enough?  Part II

Verification is a complex task that takes the majority of time and effort in chip design. At Veriest, as an ASIC services company, we have the opportunity to work on multiple projects and methodologies, interfacing with different experts.

In this “Verification Talks” series of articles, we aim to leverage this unique position to analyze various approaches to common verification challenges, and to contribute to make the verification process more efficient.

The first part of my discussion with experts in the field is presented in the previous article. It deals with the way in which the criteria for completion are formed and what exactly those criteria are. The experts I talked with were Mirella Negro from STMicroelectronics in Italy, Mike Thompson from the OpenHW Group in the Canada and Elihai Maicas from NVIDIA in Israel.

Tracking the development process

Of course, everyone uses tools from well-known vendors (Synopsys Verdi or Cadence vManager) since unfortunately, as Mike notes, “there is a general lack of open-source tools for generating and collecting usable verification quality metrics.”

On top of this, Elihai uses locally developed tools and scripts to send automated reports in the e-mails. “It’s convenient if all the criteria are formalized to a number that can be collected like coverage percentage. For flows/features, I usually also add coverage item (SVA or just test-pass hit). This way it is visible in all the automatic reports,” he explains.

Mike is a big fan of verification plans and uses their items to track the progress. This is how the process looks like for him: firstly, write and review a detailed verification plan, then assign a qualitative metric to each item in the plan (a cover point, a testcase, an assertion or code coverage) and afterwards develop tests and run regressions until your metrics show that all items in the verification plan are covered.

Mirella, on the other hand, relies a lot on the members of her teams and trusts their assessments and reports related to the planning, as well as to the monitoring of progress. They have crafted a spreadsheet that is simple for engineers to fill out although it has some complex formulas behind it. Each team member reports his own progress using this form. And, as a manager, Mirella uses a business data intelligence tool to analyze all these spreadsheets, which gives her a clear overview of the status. It can give a nice graphical representation of the status of tasks, resources or whatever needed.

Major causes of oversights and problems

So, what can go wrong? Mike believes that “the bulk of the complexity in ASIC verification is simply dealing with the extremely large number of features to cover.  It is all too easy to miss an important detail.  A big source of this can be insufficient information in either a specification or verification plan.  Another significant source of test escapes comes from a lack of time and engineering resources.  There will never be enough time and never be enough people, so it’s important to prioritize the items in your verification plan. Some features simply must work, or the project fails. These should be done as early as possible.”

Elihai says that major pitfalls may occur when not documented well in three situations: specs are changed during the development, new information generated during design review meetings and decision made by architect and designer and not communicated to the verification team.

In Mirella’s view, the major problems are caused by lack of the time which comes from bad planning and “invisible tasks” (sick leave, trainings, holidays, assisting a colleague, meetings…). She overcomes this issue by calculating in the plans that an engineer has less than five working days a week. Also how effectively they work depends on the seniority level. Mirella also adds a risk margin which is usually 10% of the project duration, but it can vary based on the risk analysis.

Finally, how do you cope with the anxiety that comes with “pressing the signoff button”?

According to Elihai, he never finishes all his plans for verification. But he tries to find out what are the things that must be verified, such as important flows or end-to-end data transactions. He regularly maintains a tasks list and constantly prioritizes them with relevant stakeholders (architects, designers, managers). Besides some strict rules, this process also has some “intangible” factor that comes from experience and intuition. And then you just have to trust what you did was enough.

Luckily, from Mike’s experience it usually is: “Create a plan, follow the plan, track progress according to the plan.  When the device is taped out, you’ll know what you’ve verified and how well it’s verified. Any time I have done this, the results have been positive.”

But he still admits:” Having said that, it’s always stressful.”

Takeaways

So, to make a sign-off less stressful, we need to do detailed planning, follow that plan, have well-organized communication between all relevant stakeholders, establish a trust within the team, but also not to be afraid to follow the intuition in some situations.

To view more about Veriest, please visit our web site.

Also Read:

Verification Completion: When is enough enough?  Part I

On Standards and Open-Sourcing. Verification Talks

Agile and DevOps for Hardware. Keynotes at DVCon Europe