Trust, but verify. How to catch peanut butter engineering before it spreads into your system — Part 2: Verification.

This article about verification is part 2 of a two article series. Please see part 1 on validation HERE.

Verification is a field that has emerged as its own discipline. It’s no longer being relegated to an activity led by the design team to which time is allocated as long as it doesn’t get in the way of designing. Chip companies that want to have predictable product release cycles have realized that it is a false choice to pick between designing or verifying. You need to treat both with absolute devotion to be successful in today’s competitive market. And if you’re a system company, you absolutely need to make sure that your custom silicon supplier has top notch verification methodologies and verification engineers deployed to your project so that your system schedule is predictable, and your chip tape out is of a high quality. I have never met engineering executives and sales executives representing a chip supplier that will not claim that; their company has a great track record of on-time tape outs, first pass silicon, total commitment to excellence and top notch methodologies, not a single one has ever said anything different. Yet, truly first pass silicon is rare, and tape out delays are not uncommon, so someone is not telling the truth. That is why system companies are advised to perform a  detailed verification capabilities review during the chip vendor selection reviews, or to perform multiple verification reviews as part of the full silicon management process. System companies can then make an informed decision when choosing a supplier for their custom silicon program. This verification review effort also helps in reducing the amount of mask sets consumed in a project which can be a very costly item as you use process nodes that are closer to the state of the art. Once the system team gets silicon back they will be very difficult to manage to tape out again to fix ECOs that are not acceptable to the system company. This is especially true if the chip supplier is working based on a fix bid quote since paying for additional masks could wipe out their profit margin. This could lead to an impasse between chip supplier and system company.

I hope that from the explanations above the reader can agree that the alternative of hiring a chip company without silicon management processes and experts on your side, writing them checks for large sums of money as milestones are reached and then crossing your fingers hoping the  silicon comes in working condition is not a good plan at all. By the time you get delivery of the first revision of the chips to your system build you have probably already paid the chip design house most of the agreed NRE. So you have little leverage left to get them to fix the chip. Careful drafting of contracts is a must here and you should definitely select competent legal counsel early so you can ensure your legal front is well thought out. The silicon manager is a key resource to help the legal team call out meaningful milestones as triggers of payment and to anticipate the types of issues that can cause an impasse during the program.

Silicon management is not just about technical checks and project management. It’s also about understanding the motivations and incentives of the parties involved in the project and constantly watching for collision courses and blind spots.

Some of the risks to watch out for when seeking to hire a silicon supplier for your custom silicon program are:

• Run-break-fix.

I’ve seen this happen in different situations: 

1. One of them is when you select a mostly analog chip supplier that usually releases small pin count parts, and your custom chip requires them to integrate multiple of those parts into one bigger chip. Add some digital interfaces, control registers and things are very different now compared to what that team usually works on. While this sounds like something simple, when it fails it’s usually because the verification methodologies used by analog designers for small pin count chips may not, and usually don’t scale to higher level integration. To add to the difficulties, analog designers who are used to being top dog in the hierarchy balk at the idea of allowing verification leads to take charge of the top level verification, and instead try to scale up their methodologies and keep control of the project. As irrational as that sounds this happens a lot. Every engineer wants to control their baby. Absent some honest desire by the chip supplier to adopt a verification methodology that scales and can integrate digital and analog, you’re going to have an unpredictable chip release schedule. As a double whammy any delay in design will come at the cost of reduced verification. The ego of many designers simply gets in the way of the success of the program, and that is a very difficult situation to overcome. So best to avoid it altogether and choose a different supplier as soon as you detect that is what is likely to happen.
2. Another situation that leads to a run-break-fix scenario is when the supplier may in theory have a proper verification methodology in place, but they severely understaffed the verification team to reduce the “overhead costs”. This is peanut butter engineering and tends to happen in companies that are too influenced by the traditional designers who don’t even comprehend why we need these fancy verification guys now that look more like a software engineer than a “real chip guy”. They view it like they’ve been releasing chips for X years without them, blah, blah… So you end up with less coverage than you should/could have because the verification engineers simply don’t have enough cycles, which leads to poorly written tests, poor schematic vs model checkers, lack of sufficient automation for the verification suite which leads to poor regression testing. Simply the verification of the chip is sub-par compared to what it could been given the modern tools and techniques available today and it’s your system that will be taking the brunt of the risk.
3. If your project ends up in a run-break-fix loop you could have 2,3,4 or more tape outs as you watch your system development take a huge delay, and once you select a supplier that turns out not to have the proper verification chops you end up in a very bad situation of having to decide between continuing to invest more time and taking on more risk to your schedule with this supplier, or to take the full hit of going with a different supplier late in the game. Some system companies try to solve this by having multiple suppliers developing the same pin to pin compatible chip in parallel, but that will dilute the system company’s engineering team focus on making sure the chip is properly designed to the right specs that support the system.

• Experienced, but done.

It’s not unusual to find, while discussing the requirements of the verification review with a potential chip supplier, and further down the road when discussing the verification that has been run in preparation to tape out, that some of the engineers instead of arguing why some type of verification doesn’t need to be run or improved because it’s covered in some way somewhere else, they will say: “I have X years of experience, and in my experience we just don’t need to do that.” Now don’t get me wrong, any engineer at any experience level can have this attitude, especially the bad ones. But even the good ones can go bad if they don’t watch out. What this engineer is really saying by choosing not to defend his position with an argument, and instead bring up his/her experience is that: “I lost my professional curiosity some time back, and stopped learning, and I am no longer interested in learning. So quit making me uncomfortable by asking me to change the way I do something, and challenging my worldview.” Once an engineer loses their curiosity, they are done as an engineer, and experience is valuable, but it ain’t going to by itself allow you to grow if you stopped being curious. If you as a system company see this type of attitude in a person in lead roles for a custom chip program you need to get out of there, and select another supplier, especially when searching for a company that has good verification methodologies since this is a field that is relatively new and has changed a lot recently compared to other much more mature areas of silicon development.

• Serializers, and false choices.

The traditional way of developing a chip used to be that you first designed it, and then ran the verification before you taped out. Digital chips have usually had the most robust methodologies for design and verification. But as soon as some significant analog content enters the picture the verification really diverges from supplier to supplier, and it seems to me that everyone does their own thing mixing and matching different commercially available tools with overlapping capabilities which are selected to do different jobs in a somewhat arbitrary manner, and mixing it with internally developed scripts and tools. Out of that blend, some concoction of a verification methodology and its results becomes your verification for the tape out. Verification engineers are expected to start developing models and tests in parallel to the design team designing the chip, they will interview the designers to determine functionality and pin out of the blocks they will work on, and with that information they will start putting together a top down behavioral model and test bench testing environment that eventually will intercept the designer’s schematics, and will then be used to do proper schematic vs model checks to speed up some sims, while choosing to leave some blocks at transistor level for others, all this judiciously done to get excellent overall coverage while maintaining reasonable simulation times. The verification engineers are continuously building that verification environment and searching for bugs throughout the chip development, and that is their core job, they are not designers that came off block designs and now are available to run sims and make models. While augmenting the verification team with idle designers could be beneficial, a plan that requires designers to come off their block designs to be able to complete proper verification is a risky one as designers may need to spend more time to finish their blocks than expected and they will prioritize that work over any verification deliverable they may have assigned to them.

• Designers as jacks of all trades – masters of only one.

As you may have noticed in my points above, I am not fond of designers when it comes to them interfering with verification engineering, especially analog designers when it comes to doubling up as verification engineers. Their heart is in design, not in verification, and they usually lack not only the passion but also the skills needed to be an effective verification engineer which include excellent coding skills. It should go without saying that assigning a designer to verify their own design should definitely not be part of the plan if you want to avoid tunnel vision getting in the way of finding bugs before tape out.

• Home brewed, but too cool to show and defend it.

Many companies develop tools, scripts, etc… that they use internally. This is normal, and as long as you can inspect the tools and their inputs, outputs, etc… as part of the tape out phase review this is ok. However, it’s also true that when a commercially available tool that is widely used in the industry is available, but instead the chip supplier uses a home brewed tool they are adding some risk to the chip development because the user base for the commercially available tool is broader and therefore more people are reporting bugs, and there is also an EDA company behind that tool whose business it is to fix it and upgrade it. The home brewed tools may be someone’s pet project, and when that someone moves on from that company, the tool may no longer be updated, and get stale. It’s also especially disruptive if a supplier blocks the system company silicon reviewers from performing an in-depth verification review at tape out because they are trying to protect whatever they think is differentiated IP that this tool contains, and this really handicaps the system company’s ability to check if the tape out is of a high quality or not, and therefore negates the risk mitigation that an independent tape out review provides to the system company.

• Don’t let documentation get in the way of the “real work”.

Chip companies that have poor internal review processes tend to have poor documentation practices. You can spot this easily because as soon as you need to perform an in-depth verification plan review, the plan is poorly written, lacks test details, lacks specifics of what is being tested, etc… It’s basically a document that doesn’t really provide the reader with the full scope of the verification that is planned to be executed. In these cases you usually have one or maybe a few engineers that are directly coding the verification without taking time to review with the broader team their plans and intended scope of coverage. Without the documentation, the internal reviews at that chip supplier will be much less effective, and it will be pretty much impossible for the system company reviewers to check if the plan is good or not. This will also reduce or eliminate the possibility of having the system company engineers, and the system company FW engineers provide their feedback about the chip verification plan, on the types of tests and coverage that they think would be most appropriate considering the system’s use cases.

There are no perfect supplier teams, and there is no perfect verification flow, every team has its strengths and weaknesses, and when selecting the supplier for your custom chip you need to decide if that is the right team for the type of chip you want to develop. Verification can be run forever with all sorts of randomized inputs, analog operating point combinations, etc… But at some point you need to tape out, and some bugs may have been missed which may be easier to find during validation rather than in a simulator. If you did a pretty thorough job you tend to find few (if any) digital bugs since the digital domain has very good tools already to maximize coverage, and most bugs will be found in the analog or RF parts of the chip.

• Trust, but verify.

Custom system silicon when done with the assistance of silicon experts puts the system company in control of its own destiny. It’s important to note that when purchasing catalog parts for your system, unless you perform similar due diligence to what is described above, you’re trusting but not verifying that your components will be of good quality and not likely to cause yield or other issues when you go to production in high volumes.

For more information contact us.

Share this post via:

• 
• 
• 
•