PCIe 6.0: Challenges of Achieving 64GT/s with PAM4 in Lossy, HVM Channels

As the premier high-speed communications and system design conference, DesignCon 2023 offered deep insights from various experts on a number of technical topics. In the area of high-speed communications, PCIe has a played a crucial role over the years in supporting increasingly higher communications speed with every new revision. Revision 6.0, the latest revision of this communications interface standard enables system designers to achieve advances in the deployment of AI inference engines and co-processors in data centers. Consequently, PCIe 6.0 was a hot topic at the conference, not just for the 64GT/s speed but also for understanding the engineering challenges to reliably deliver that speed.

PCIe 6.0 poses a demanding set of chip and system design challenges on engineers. To reliably deliver the full benefits of PCIe 6.0, collaboration and cooperation are needed to standardize specifications in the areas of PCIe card, cable, connector assembly, test method, measurement and tools and PCIe PHY and controller IP. An experts panel to discuss these very topics included David Bouse from Tektronix, Rick Eads from Keysight Technologies, Steve Krooswyk from Samtec, Madhumita Sanyal from Synopsys and Timothy Wig from Intel. The panel session was moderated by Pegah Alavi from Keysight Technologies.

Pegah opened the session by highlighting the challenges introduced by multi-level signaling (MLS) when the switch was made from NRZ to PAM4 signaling to support 64GT/s. The adoption of MLS has opened up the path to continue increasing data communications speeds. By mapping more than 1 bit into a transmitted symbol, the required bandwidth/bit is reduced. But MLS introduces a lot of challenges too, which need to be overcome to achieve the speed benefit in a reliably manner.

Under MLS, the signal to noise ratio worsens, negatively impacting the performance of the channel. Consequentially, all aspects of the channel need to be paid close attention to. With that introduction, Pegah set the stage for the panelists to update the audience on their respective areas of focus to deliver a reliable PCIe 6.0 end-user solution. The following is a synthesis of the salient points from the session.

PCIe Card and Cable Form Factor Updates

Rev 6.0 of the PCIe Card Electromechanical (CEM) form factor specification is being finalized in 2023. The Rev 6.0 mechanical updates are completely redefining chassis retention on the North and East vias.

The CEM card physical form factor introduces two new power connectors at 48V to deliver 600W.

A shielded plane/south via approach has been introduced to shield the send signals from the receive signals. Without the shielding plane/south via approach, PCIe 6.0 channels would be completely broken, given known examples of inattentive card layout sabotaging even PCIe 5.0 channels.

Two PCIe cable form factors are being defined. Both these new form factors are distinct from previous PCIe cable solutions. An internal cable form factor is being defined based on the EDSFF-TA-1016 cable system targeting PCIe 5.0 and PCIe 6.0 speeds. An external cable form factor is being defined based on the industry standard CDFP. The Internal PCIe cable form factor has been characterized for a range of connectors and cables from multiple vendors, mounting styles and lengths.

Test Methods and Tools

The PCIe ecosystem is keeping PCIe 7.0 in mind as they define and develop tools and test methods for PCIe 6.0. After all, PCIe 7.0 spec (128 GT/s) is just around the corner as it is expected to arrive in the 2024-2025 time frame. The Tx, Rx and channel compliance requirements are kept in mind as the simulation, test and measurement methods are being developed to validate connectors and cables-connector assemblies. Forward Error Correction (FEC) has been introduced in PCIe 6.0, a first for the PCIe interface standard to accommodate the impact of channel loss.

PCIe v6.0 Retimer

All of the things presented above ensure that the cards, cables, connectors and assemblies are validated to support PCIe 6.0. Depending on the end market and application, a PCIe-based system will be deploying different channel topologies, leveraging the hardware listed above. Consequentially, each channel topology will bring with it, its own characteristic that would impact the channel performance.

The following chart shows four different channel topologies that are commonly found in PCIe-based systems.

From the PCIe PHY perspective, it needs to be able to optimize for all possible channel topologies. Given the Reduced Insertion Loss budget imposed by the PCIe 6.0 specification, how to ensure that the signal from the Root port will reach the destination port without losing fidelity.

The solution is the introduction of a PCIe 6.0 Retimer circuit. PCIe Retimers enable the expansion of PCIe over system boards, backplanes, cables, risers and add-in cards, irrespective of the channel topology that is deployed.  A Retimer is a physical layer and protocol-aware device but software-transparent, and can reside in any place in the channel between PCIe Root-port and End-point.  It fully recovers the data over any channel from the Host PCIe Root-port, extracts clock and re-transmits the clean data over another channel to the PCIe End-point device. The Retimer solution is implemented in the form of customized PHY and light controller logic for the MAC.

Summary

The panelists offered a number of tips and tricks and best practices throughout the session. When DesignCon makes the panelists’ presentation materials available on their website, it would be a good idea to download as reference materials. You may want to reach out to the panelists for more specific detailed information.

Also Read:

Optimization Tradeoffs in Power and Latency for PCIe/CXL in Datacenters

Synopsys Design Space Optimization Hits a Milestone

Webinar: Achieving Consistent RTL Power Accuracy