Can you Publicly Benchmark EDA Tools?

There is an interesting discussion on SemiWiki in regards to the age old question aboutbenchmarking EDA tools. I remember benchmark discussions at my first DAC in 1984. It was deemed impossible to do a “fair” public benchmark then and it’s not possible now, just my opinion of course but let me tell you why. Simply stated it is a legal, technical and political mountain that just cannot be climbed.

The first comments in the forum discussion pointed out that current EDA purchase agreements prevent public benchmarks. I don’t remember specific contract clauses when I started in EDA but I do know we signed NDAs (non-disclosure agreements) that begot confidentially. That is definitely the case today as there are specific confidentiality contract clauses and NDAs are a standard part of the EDA product evaluation process so even if it was technically and politically possible there are definitely legal barriers.

The tool reviews on DeepChip were also referenced. The credibility of the reviews are in question since they are generally vendor sponsored with “friendly” customers and ghost written by a PR person. I still see value in this type of tool review assuming there is no financial link and vendors are not hand picking participants. There is recent FTC scrutiny on websites that do product reviews (endorsements). The lawyer I hired to educate us on the legal liabilities of running SemiWiki strongly advised us to clearly document the financial trail related to content mentioning a sponsor or advertiser. As a result we have dedicated landing pages for each SemiWiki subscribing/sponsoring company, the company is clearly identified in the header of the article, and there is a sponsor disclosure in the footer of all pages. The SemiWiki bloggers are also listed with links to their LinkedIn profiles for professional affiliations. It might be overkill but the last thing we need is the FTC on our backs and the credibility of a site with original content such as SemiWiki is critical to our survival, absolutely.

The consensus of the discussion suggests that the technical aspect of benchmarking is the real challenge and something that is just not feasible. Tracking tool versions, replicating hardware environments, guaranteeing the authenticity of the results, and applying the results to different design requirements is a daunting (expensive) task. That is assuming EDA vendors cooperate, which they most certainly will not. I do not see large semiconductor companies supporting this effort either since they view EDA tool usage as a competitive advantage and are very secretive in this regard. Coincidentally, as I write this, I’m at the Si2 conference which started out reviewing their impressive progress over the last 25 years. Unfortunately the hard earned Si2 standards accomplishments pale in comparison to the EDA benchmarking challenge as I see it.

One of the more interesting comments was that we try something like a consumer reports product review which would be a structured and detailed report. Rather than getting friendly customers from vendors, we get this type of feedback directly from users who can answer a standard set of questions directly by posting on a dedicated SemiWiki forum. To post on SemiWiki you need to be a registered member. To register and create a screen name you need to provide a LinkedIn profile which we do check for authenticity. The forums can be moderated by SemiWiki members that have no financial ties to vendors.

Let’s brainstorm here, if we can come up with something modeling consumer reports that the fabless semiconductor ecosystem can benefit from I would be happy to put resources into it. Sound reasonable?

lang: en_US