Array
(
    [content] => 
    [params] => Array
        (
            [0] => /forum/index.php?threads/highest-number-of-pins.17410/
        )

    [addOns] => Array
        (
            [DL6/MLTP] => 13
            [Hampel/TimeZoneDebug] => 1000070
            [SV/ChangePostDate] => 2010200
            [SemiWiki/Newsletter] => 1000010
            [SemiWiki/WPMenu] => 1000010
            [SemiWiki/XPressExtend] => 1000010
            [ThemeHouse/XLink] => 1000970
            [ThemeHouse/XPress] => 1010570
            [XF] => 2021370
            [XFI] => 1050270
        )

    [wordpress] => /var/www/html
)

Highest number of pins

Wow, 10072 pins in a 92.5mm LGA package. Maybe that's why Intel just cancelled the entire Tofino project... :-(
 
Am presently researching the limits of packaging.
It is not just the package, it is also the limits of the board it sits on. How many layers are practical to route all that to a useful distance.

This is an advantage for CPO where 1 fiber with multiple colors can bring in or take out > 1Tbps, at low energy per bit, too. 64 ports is 128 fibers and less heat.
 
It is not just the package, it is also the limits of the board it sits on. How many layers are practical to route all that to a useful distance.

This is an advantage for CPO where 1 fiber with multiple colors can bring in or take out > 1Tbps, at low energy per bit, too. 64 ports is 128 fibers and less heat.
It's less heat overall for the system but more heat in the router chip package, which is already stupidly hot -- I've been involved in CPO feasibility studies.

Apart from the (solvable) technical problems like fiber attach/connectors, the real problem with CPO is business -- CPO works if you're vertically integrated (e.g. Broadcom) and can provide a switch/router chip with fiber interfaces, great for BCOM but not so good for customers who want to have competing providers or multiple sources for the optics. It would also destroy a large part of the pluggable/component optics industry by swallowing up their business inside the switch/router suppliers... :-(
 
It's less heat overall for the system but more heat in the router chip package, which is already stupidly hot -- I've been involved in CPO feasibility studies.

Apart from the (solvable) technical problems like fiber attach/connectors, the real problem with CPO is business -- CPO works if you're vertically integrated (e.g. Broadcom) and can provide a switch/router chip with fiber interfaces, great for BCOM but not so good for customers who want to have competing providers or multiple sources for the optics. It would also destroy a large part of the pluggable/component optics industry by swallowing up their business inside the switch/router suppliers... :-(
That seems odd. How does a 2pJ/bit IO budget for something like Ayar make your package hotter than copper interconnects running 20x that?

I don't see CPO as coming to ethernet first. The first and best use case is rack-level CXL for disaggregation. The central switch will be a smart (P4 equivalent) CXL 4.0 hub for a rack. Only a fraction of the traffic will be outward facing ethernet.

As for the existing vendors, they have a chasm to cross.
 
Last edited:
That seems odd. How does a 2pJ/bit IO budget for something like Ayar make your package hotter than copper interconnects running 20x that?

I don't see CPO as coming to ethernet first. The first and best use case is rack-level CXL for disaggregation. The central switch will be a smart (P4 equivalent) CXL 4.0 hub for a rack. Only a fraction of the traffic will be outward facing ethernet.

As for the existing vendors, they have a chasm to cross.
There are two uses for CPO in devices like this, one being the short-reach intra-rack or inter-rack (a few m), the other being interconnects across the data centre (up to 2km reach).

As currently planned -- or at least, when I was looking at this a year or two ago -- CPO needs an electrical interface (switch chip to optics) that can drive something like 5cm plus a connector, assuming that the CPO modules are socketed not soldered down (which is a nightmare), and so these interfaces are not the ultra-low-power ones capable of driving a few mm over massively-parallel wires like BOW or UCIe, they need a lot fewer lanes so use more conventional (and higher power) ultra-short-reach SERDES.

The power needed for the external optical interface plus the two short-reach electrical interfaces (one on switch chip, one on optical module) is a bit smaller than using external pluggable optics (e.g. with 20cm reach electrical interfaces), but it's now all inside the single package instead of spread around, so cooling is more difficult.

This all assumes that the optics need too much space to be packed right next to the edge of the switch chip die; if this can be done then the power saving becomes much bigger, but now the problem is getting all the fiber connections in, and they really need to be removable -- you can't handle a big heavy package with a huge number of fiber pigtails dangling off it. Then there's what to do if a fiber or connector is damaged...

The problem is not the basic technology, it's how to build it into something that's usable, reliable and maintainable.
 
As currently planned -- or at least, when I was looking at this a year or two ago -- CPO needs an electrical interface (switch chip to optics) that can drive something like 5cm
Chiplet inside package (that is what "co-packaged means to me).


A little less close:

The opportunity for chiplet CPO inside servers and racks with CXL dwarfs the opportunity for traditional ethernet optics. The competition is copper cabling, not ethernet fiber. A whole new field driven by the cost reduction CXL disaggregation enables.
 
Chiplet inside package (that is what "co-packaged means to me).


A little less close:

The opportunity for chiplet CPO inside servers and racks with CXL dwarfs the opportunity for traditional ethernet optics. The competition is copper cabling, not ethernet fiber. A whole new field driven by the cost reduction CXL disaggregation enables.
The Ranovus solution is exactly what I was describing (and what was used in our feasibility study) -- it needs several cm of in-package tracking not a few mm, and realistically the optics have to be mounted on sockets to allow repairability, which means they need conventional SERDES links not BoW, which means higher power. A possible alternative is so-called "direct-drive" optics (with no retimer in the optics), but this gets more and more difficult as data rates go up due to losses and reflections in the link (via connector) between switch chip and optics, which add up for both ends of the link since the optics are transparent.

It's OK if the entire switch/driver/optics solution comes from a single vendor and is the same at both ends, then there's only one company to point the finger of blame at. If the solution at each end is vertically integrated but different at the two ends of the link, then you get into interoperability certification which customers don't like. If the optics are also pluggable -- and this means standardising the interface, including control and power supplies, which is difficult to say the least -- then there are four parties involved in loss budget of a single link.

So the customer wants both a reliable and predictable system with clear responsibility for delivering and fault-finding (same supplier everywhere), but also supplier competition at both ends of the link and with the CPO -- and these two wished are diametrically opposed. To replace a bad CPO tile the entire shelf has to be removed, the CPO package dismantled and a new tile inserted, which is horrible compared to just plugging in an optical module at the faceplate.

This is why the industry is clinging on to faceplate pluggable optics for as long as possible until they have absolutely no choice, because even if CPO works technically they pose huge business/maintenance problems in reality.

It's why the mass introduction of CPO keeps getting pushed back to the next generation of switch chips, and this has been happening for some time. I'm not just speculating here, I've been involved in extensive discussions about CPO for several years... ;-)
 
Last edited:
The Ranovus solution is exactly what I was describing (and what was used in our feasibility study) -- it needs several cm of in-package tracking not a few mm, and realistically the optics have to be mounted on sockets to allow repairability, which means they need conventional SERDES links not BoW, which means higher power. A possible alternative is so-called "direct-drive" optics (with no retimer in the optics), but this gets more and more difficult as data rates go up due to losses and reflections in the link (via connector) between switch chip and optics, which add up for both ends of the link since the optics are transparent.

It's OK if the entire switch/driver/optics solution comes from a single vendor and is the same at both ends, then there's only one company to point the finger of blame at. If the solution at each end is vertically integrated but different at the two ends of the link, then you get into interoperability certification which customers don't like. If the optics are also pluggable -- and this means standardising the interface, including control and power supplies, which is difficult to say the least -- then there are four parties involved in loss budget of a single link.

So the customer wants both a reliable and predictable system with clear responsibility for delivering and fault-finding (same supplier everywhere), but also supplier competition at both ends of the link and with the CPO -- and these two wished are diametrically opposed. To replace a bad CPO tile the entire shelf has to be removed, the CPO package dismantled and a new tile inserted, which is horrible compared to just plugging in an optical module at the faceplate.

This is why the industry is clinging on to faceplate pluggable optics for as long as possible until they have absolutely no choice, because even if CPO works technically they pose huge business/maintenance problems in reality.

It's why the mass introduction of CPO keeps getting pushed back to the next generation of switch chips, and this has been happening for some time. I'm not just speculating here, I've been involved in extensive discussions about CPO for several years... ;-)

I don't think pluggable optics is going anywhere for reason of simple physics, and that "telecom" gear outnumber dc switches by a giant margin.

Relatively long range transmitters beaming past 100m will melt the package in any significant numbers.
 
I don't think pluggable optics is going anywhere for reason of simple physics, and that "telecom" gear outnumber dc switches by a giant margin.
You can think what you want; the view of many end customers who have to worry about the practicalities is rather different to that of the technology providers like Ranovus who are pushing CPO.

Remember how on-board optics were going to take over the world? CPO has even more of the same problems that killed this off...

It'll happen eventually, but only when there's absolutely no alternative -- and it will completely transform the industry, which will be very painful for a lot of companies who don't have the technology or customers... :-(
 
Much more realistic prospect is short range direct drive optics beaming to an optics-in optics-out longer range transmitter, where much of analog signal conditioning will take place
 
Much more realistic prospect is short range direct drive optics beaming to an optics-in optics-out longer range transmitter, where much of analog signal conditioning will take place
Except I talked to systems companies recently about exactly this approach, and the response was that it was simply too expensive and likely to remain so -- because the longer-range transmitter has to have (electronic) retiming in it (DSP) to avoid the concatenation of losses/eye closure and an even bigger link performance analysis problem...

The problem for all approaches like this -- including CPO or NPO or OBO -- is that copper for the switch-optics link is *extremely* cheap and pluggable optics are *extremely* competitive and easily replaceable, so almost all alternatives founder on the rocks of cost and maintainability.

It takes an *awfully* big technical/power advantage to overcome this. It's also likely that system-level changes will extend the lifetime of pluggable optics, like this:

 
Last edited:
It takes an *awfully* big technical/power advantage to overcome this.

1/3 of terabit class switch chip TDP is about pushing data over few centimetres of PCB to SFPs. Does this qualify?

Do you really need a DSP to do retiming at such short distance? I think non-digital retime should suffice.
 
1/3 of terabit class switch chip TDP is about pushing data over few centimetres of PCB to SFPs. Does this qualify?

Do you really need a DSP to do retiming at such short distance? I think non-digital retime should suffice.
What you describe is usually called "direct drive", it does work over a few cm -- the switch chip packages with CPO are considerably bigger than a chip-only BGA, usually something like 120mm square, but there are issues with integration. Now something called NPO (near-package-optics) is being proposed, which is a conventional LGA package for the switch chip socketed onto another substrate with the optics on, but this all adds to the cost. And SERDES instead of BoW are still needed, which adds to the power consumption.

The problem of having multiple cascaded non-retimed links (electrical-electrical, electrical-optical, optical-optical, optical-electrical, electrical-electrical) remains, as do the maintenance/sourcing issues.

With a rearranged shelf like in the link above, the switch chip is only something like 6cm-7cm from the pluggable optics, which is less than 2x as far as with CPO, and keeps front-panel pluggables...
 
It's OK if the entire switch/driver/optics solution comes from a single vendor and is the same at both ends, then there's only one company to point the finger of blame at. If the solution at each end is vertically integrated but different at the two ends of the link, then you get into interoperability certification which customers don't like. If the optics are also pluggable -- and this means standardising the interface, including control and power supplies, which is difficult to say the least -- then there are four parties involved in loss budget of a single link.

This is why the industry is clinging on to faceplate pluggable optics for as long as possible until they have absolutely no choice, because even if CPO works technically they pose huge business/maintenance problems in reality.
I agree that the single-vendor issues are significant. Only Ayar seems to be able to deliver a chiplet solution, and it works, but deploying it requires a lot of coordination. UCIe may help, and it will also help to have fluid cooling in-package since it eliminates a long pole in package thermal/mechanical design to fit chiplets.

This has nothing to do with faceplate networks. This is replacement of copper and building larger coherent servers with low latencies, efficient power, and disaggregated resources in a rack. The driving forces are not networking companies, they are cloud vendors looking for cheaper, more fungible solutions with denser racks and better efficiency. The network front panel for the rack may be relatively unchanged while the internals do this.
 
I don't think pluggable optics is going anywhere for reason of simple physics, and that "telecom" gear outnumber dc switches by a giant margin.

Relatively long range transmitters beaming past 100m will melt the package in any significant numbers.
A DC currently has around 100 switches per MW and the internal bandwidth can be 100x the external facing capacity. But, the CPO I am interested in is replacing copper links within a rack, just a few meters. There are DC issues with the network at rack and above, but CPO is not relevant to those. Front panel works fine up there until the rack internal revolution is done and the bottlenecks elsewhere will be tackled.

CPO is not about networks. It is about scaling up the modern bus. Network companies just assume it is all about them. It has nothing to do with them.
 
Remember how on-board optics were going to take over the world? CPO has even more of the same problems that killed this off...
The underlying problem with putting optics on the chips is that light wavelengths are huge and sloppy use of a good logic process. Chiplets and 2.5D connection are the happy medium.
 
The underlying problem with putting optics on the chips is that light wavelengths are huge and sloppy use of a good logic process. Chiplets and 2.5D connection are the happy medium.
The optics with CPO *are* on chiplets with 2.5D (high-density on-substrate) connections to the switch chips...
 
Back
Top