IP-over-DWDM’s Fatal Flaw
Last June I wrote a blog about how we might be on the verge, finally, of the IP-over-DWDM era. I say finally, because although this approach was first proposed about 20 years ago (and hyped every few years since) it never caught on. I thought that the imminent introduction of small form factor standard 400GZR colored pluggables would change the facts on the ground sufficiently to allow IP-over-DWDM to have its breakthrough moment.
However, I have changed my mind. I have concluded that while these pluggables could give IP-over-DWDM a small boost, that it is not yet ready for prime time. Let me explain why.
The left-hand diagram illustrates a modern day IP-over-DWDM architecture. Routers connect directly to their nearest neighbors in a massive mesh using colored DWDM interfaces, without relying on a separate optical transport layer. There are no ROADMs. What makes this a modern implementation is that the routers would employ small form factor (e.g. QSFP-sized) 400GZR pluggables to connect with each other. These small pluggables would not consume valuable faceplate space on the router, which had always been a major problem in implementing colored interfaces on routers until now.
Under this architecture, in the event that router A is sending packets to router C (not a neighbor) the physical path would need to transit intermediary routers.
The right-hand diagram shows the prevailing architecture today. Routers use low cost (and equally small form factor) gray interface pluggables to connect to an independent optical layer. This OTN/DWDM layer employs its own colored transponders and muxponders, as well as ROADMs for wavelength routing. Returning to the previous example, router A can use this optical layer to connect directly with router C. Not only does this reduce the number of router interfaces required, but it also improves the latency of the connection.
The table below summarizes the major pros and cons of each architectural approach.
Clearly, pure IP-over-DWDM has the benefit of single layer simplicity. Network engineers only need to concern themselves with router topology and IP traffic, with just a dash of optical skills needed for the point-to-point links between the routers. Longer optical links that cannot be handled with this approach might be contracted to a 3rd-party OTN/DWDM layer provider on an exception basis.
Pure IP-over-DWDM is also less expensive, but this is only moderately so (green arrow trade-off). Gray client interfaces are cheap, and the colored transport interfaces need to exist somewhere, whether in the routers or in a separate OTN/DWDM layer. Moreover, IP-over-DWDM will require more colored interfaces to support high mesh connectivity, and also because this architecture cannot aggregate multiple lower speed P2P connections easily.
The big advantage of a separate OTN/DWDM layer are the many degrees of flexibility it provides the network operator (blue arrow trade-off).
- Muxponders aggregate multiple lower speed client interfaces onto a single wavelength, lowering costs. This can just as easily be multiple 10GE clients onto a 100G wavelength, or multiple 100GE clients onto a 400G or even 600G wavelength.
- This also draws attention to the fact that client (gray) and network (colored line) interfaces evolve independently. Therefore, a separate OTN/DWDM layer can upgrade to higher performance and more efficient transport technologies, without necessitating any client interface changes on the routers.
- Distant routers can be connected directly with each other, bypassing intermediate routers in a hop-by-hop IP-over-DWDM architecture. This delivers a huge benefit of lower latency when connecting to cache or gateway nodes that are hubs for IP traffic.
- Transport for non-IP services, which still exist quite extensively, are part of the fundamental mission of an OTN/DWDM layer. IP-over-DWDM either handles these ‘foreign’ services using clumsy circuit emulation, or as often is the case, abandons them outright saying they are someone else’s problem.
- In the event of a fiber cut, ROADMs can dynamically re-route wavelengths to maintain service.
- At the end of the day, IP and optical are fundamentally different technologies with different skill sets. By having distinct IP and OTN/DWDM layers, different groups of engineers can focus on optimizing and managing networks within their core competencies.
In summary, IP-over-DWDM is superficially appealing. However, when you peel back the covers, its moderate cost advantages come at the expense of a huge loss of flexibility fulfilled by a separate OTN/DWDM/ROADM layer. Yes, 400GZR we will facilitate some increase in router P2P colored connections, but an architectural landslide that does away with the OTN/DWDM/ROADM layer won’t happen just yet.