Samtec Paves the Way to Scalable Architectures at the AI Hardware & Edge AI Summit

AI is exploding everywhere. We’ve all seen the evidence. The same thing is happening with AI conferences. The conference I will discuss here began in 2018 as the AI Hardware Summit. The initial venue was the Computer History Museum in Mountain View, CA. Like most things AI, this conference has grown substantially in a relatively short period of time.  As you will notice, its name has grown, too to encompass a larger mission. At the recent event, there was significant focus on scalability in the deployment of AI systems. Samtec was there to address this challenge head-on. I’ll provide a summary here of the company’s presence at the show and how Samtec paves the way to scalable architectures at the AI Hardware & Edge AI Summit.

Samtec’s Presentation

At the show, Matt Burns, global director of technical marketing at Samtec presented Optimizing Data Routing in Copper & Optical Interconnects in Scalable AI Hardware Architectures. A rather long title, but there is a lot to address here.

Let’s take a look at the some of the topics Matt covered.

AI Agents

Over the past few years, Gen AI has been a driver in the adoption of AI agents. ChatGPT was just a pivot point. Applications such as text-to-chat/audio/image/video are redefining the customer experience in many industries. The next revolution in AI capabilities will be using AI agents to supplement the user’s experience.  The new “co-pilots” we are seeing from companies like Microsoft are good examples of this.  Other examples are actually improving code generation for simplicity and efficiency in real-time for developers.

Enterprise AI

Similarly, Gen AI has been the driving force behind enterprise AI adoption. However, only a fraction of the Fortune 1,000 has really started implementing AI to improve processes internally.  As enterprises discover how to use AI foundation models or application-specific models with their own internal data, AI will then begin to impact the bottom line for innovative companies.  The hyperscalers are leading the charge, but other companies will eventually follow.

Increasing model sizes requires more compute, but . . .

AI models are growing in size and scale. ChatGPT uses GPT-3.5 which has 175 billion parameters. GPT4 is rumored to approach 1 trillion parameters. Other models will soon approach 2 trillion parameters.  Model sizes are growing exponentially annually.  One GPU can’t handle all this.

Literally, hundreds if not thousands of GPUs need to be linked to parallel process the models. So, what’s the problem? AI compute performance is growing ~4.6x per year, but memory bus speeds are growing only ~1.3x per year and interconnect/fabric bus speeds growing only ~1.2x per year.  Those are the bottle necks.  Routing high-speed protocols like HBMx, CXL, PCIe and others over optics is becoming the trend.  Samtec demonstrated its CXL over optics solution at the show. The focus here is to position Samtec FireFly and Halo for some niche AI hardware applications.

Insatiable data center demand, but how are we going to power them?

More GPUs means more power. GPUs and other AI compute engines are approaching 2kW PER CHIP. That’s a lot of power.  System architects need to figure out how to get massive power into a rack and chassis efficiently and in small form factors at scale.

With these challenges as a backdrop, Matt presented the broad class of solutions for both copper and optical interconnect that Samtec offers. What is interesting about this show is that there are exhibits, but the footprint has always been limited to a table-top style of display. This keeps the focus on technology as opposed to fancy booth construction.

Samtec was at the show again this year, demonstrating its wide range of products for AI enablement.

To Learn More

If AI system scalability keeps you awake at night, Samtec can help. You can learn more about this unique company on SemiWiki here. And you can get an overview of Samtec’s AI capabilities here. You can even download a complete Artificial Intelligence/Machine Learning Solutions Guide here. As an aside, the conference is changing its name again. Next year’s event will be called AI Infra Summit. You can learn more about this change here.

And that’s how Samtec paves the way to scalable architectures at the AI Hardware & Edge AI Summit.