Metamorphic Test in AMS. Innovation in Verification

We have talked about metamorphic testing before. Here is a clever application to testing an AMS subsystem. Paul Cunningham (GM, Verification at Cadence), Raúl Camposano (Silicon Catalyst, entrepreneur, former Synopsys CTO and lecturer at Stanford, EE292A) and I continue our series on research ideas. As always, feedback welcome.

The Innovation

This month’s pick is System Level Verification of Phase-Locked Loop using Metamorphic Relations and was published in the 2021 DATE conference. The authors are from the University of Bremen and the Johannes Kepler University in Austria. The paper has 5 citations.

A quick recap on metamorphic testing. In some cases it is difficult or impossible to construct a meaningful oracle against which test runs can be compared to find problems. Instead metamorphic testing compares simulation behavior between two or more different tests for which certain properties in simulation results are expected to remain the same (or close to the same). Let’s call these properties “invariants”. This method is especially interesting for AMS testing where oracles inside a circuit are hard to define.

The authors use this approach to test a production PLL – part analog and part digital – by defining invariants that should hold given the structure of that function. Through this testing they were able to find an uncommon but real case in which a production PLL can lock to the wrong frequency.

Paul’s view

Fun paper this month – metamorphic testing of analog circuits. Metamorphic Testing (MT) is testing that doesn’t need a golden reference model for the design being tested. It relies instead on validating that certain relationships hold true between two different executions of a design. A common example given is for testing design that implements the sin(x) function. One “metamorphic relation” (MR) for this function is that sin(x) = sin(180-x) for any value of x. So we can write a test that just runs the design with different values of x and 180-x and check the result is always the same. Simple concept with a lot of published works showing how powerful it can be to catch corner case bugs. Threadmill, which we blogged on last month, can be considered an MT system, since it runs multi-threaded programs many times and checked the behaviors are identical to try and catch concurrency related bugs.

This paper applies MT to a commercial PLL and finds a corner case bug where a small change in the PLL input clock frequency from 1MHz to 1.01MHz causes the phase locking feedback loop in the PLL to breakdown. The paper’s main contribution is a number of clever MRs for PLLs, one of which catches this real silicon bug. This MR states that if the input clock frequency is multiplied by some factor C, then the feedback loop clock frequency inside the PLL must also be multiplied by the same factor C. Another MR states that the locking time should be the same irrespective of the input clock frequency.

Short paper, easy read. A good motivator for us all to look at our DV suites and see if we missed writing assertions/properties that check if certain relationships hold across multiple tests, not only within a single test.

Raúl’s view

“Metamorphic” testing focuses on how a system transforms inputs rather than static input-output pairs. For example, to test a program implementing sin(x), one can use sin(x)=sin(180-x) as a “metamorphic relation”. Instead of checking the expected output for a concrete input, run the program for an input x1 and afterwards for the input x2=180-x1 and check that the program gives the same output in both cases, otherwise there is a bug. Metamorphic testing has been used for software; a major advantage of this technique is that no reference model/value is needed.

This month’s paper discusses the application of metamorphic testing to Analog/Mixed-Signal (AMS) systems, specifically focusing on the verification of Phase-Locked Loops (PLLs). The authors from the University of Bremen and Johannes Kepler University identify 8 metamorphic relations for PLLs, for example: The PLL stays in the locked state if the input frequency is varied inside the lock range, and the Lock Detector signal stays on. They applied these to an industrial PLL (from an industrial partner not named explicitly) coded in SystemC and simulated with COSIDE, and discovered a previously undetected rare but real case where the PLL could lock to the wrong frequency. The bug was related to a dead-zone effect in the Phase Frequency Detector (PFD), which was resolved by adding a delay element.

The paper is succinct and self-contained and is a pleasure to read. It gives a nice introduction to PLLs and metamorphic testing of AMS systems and shows the potential of metamorphic testing for AMS verification and its ability to uncover hard to find bugs.

Also Read:

Compute and Communications Perspectives on Automotive Trends

Bug Hunting in Multi Core Processors. Innovation in Verification

Embracing the Chiplet Journey: The Shift to Chiplet-Based Architectures