How 800G Ethernet and Industry Standards are Key for the Future of High-Speed Interconnects
The Ethernet Alliance is heading to the 2023 European Conference on Optical Communication (ECOC), accompanied by a number of its member companies. Among them is global electronic design automation (EDA) leader, Synopsys. In this guest blog, Synopsys Senior Staff Technical Product Manager, Madhumita Sanyal, explores the vital role standardization plays in driving innovation for high-speed interconnects.
Ethernet, the primary data connectivity backbone of the internet, has been instrumental since the 1970s. Every iteration of this standard has mirrored the growing demand for faster Ethernet technologies, underlining its power to transform our interconnected world. As the horizon broadens with newer applications in automotive, smart edge devices, and continuously advancing HPC (High-Performance Computing) applications, Ethernet standards, guided by bodies like the Ethernet Alliance and other consortiums, evolve to cater to this interconnected universe.
What’s fueling the evolution in high-speed interfaces?
The surge in data, growing user base, and the proliferation of devices. The complex nature of today’s data has transformed yesterday’s data centers into high-speed HPC data centers, which now handle intricate computations at unparalleled speeds. The essence of Ethernet is its capability to dictate the data transfer protocols, its impressive flexibility, speed negotiation, backward compatibility, and the flexibility to employ diverse media. It remains the preferred choice for network technology.
The following graph illustrates the increasing demand due to numerous data-centric applications. Given the rising prevalence of the internet of things (IoT), cloud storage, virtual/augmented reality, video streaming, and online collaborative tools, it’s unsurprising that data volumes are predicted to surge dramatically.
Figure 1. Source: IEEE 802.3 Industry Connections Bandwidth Assessment, Part II
Enter 800 Gigabit Ethernet
The growing need for superior networking speed, improved throughput, and reduced error rates within data centers has elevated 800 GbE as the recent industry benchmark. In 2020, the Ethernet Technology Consortium introduced the 800GBASE-R specification for 800G Ethernet, repurposing two sets of existing 400G Ethernet logic from the IEEE 802.3bs standard with some modifications to distribute data across eight 106-Gbps physical lanes. This shift enables a substantially greater volume of data to flow through data centers, and reducing latency. 800 GbE offers double the bandwidth compared to the 400 GbE. In terms of design, the specification introduces a fresh media access control (MAC) and physical coding sublayer (PCS), but also leverages two sets of the current 400 GbE logic to spread the data over eight 106 Gbps lanes. By repurposing the PCS, the forward error correction remains intact, ensuring easy compatibility with current physical layer standards. Because of the increased lane count, data centers won’t need a complete overhaul of their port setup to accommodate 800 GbE, streamlining the adoption process.
Figure 2. Diagram showcasing 800G Implementation
While 800 GbE initially catered to dense switches in expansive data centers, its utility has expanded. But these high data rates introduce challenges, especially as 800 G Ethernet is yet to gain universal standardization. To transition from 10 Gb Ethernet to 400G/800G Ethernet, the most recent Ethernet switch SoCs must be scaled for optimal latency. Given the spike in bandwidth requirements for cloud growth and gigantic data centers, it’s predicted that 800G optical modules and transmissions will become the norm.
Anatomy of a Robust 800G Ethernet Implementation
For successful deployment of advanced Ethernet technologies, a silicon-proven 800G Ethernet solution is vital. This involves careful optimization, from bump-maps to managing potential issues, all while ensuring that the 800G Ethernet tile, from design to placement to silicon remains robust.
Navigating the 200G/400G/800G networking spectrum concurrently unveils a set of intricate challenges in both logical and physical spheres. Achieving minimal latency, power savings, and area efficiency in the face of accelerated clock rates, parallel structures, and advanced signaling is non-trivial. A core requirement is an adept FEC to counterbalance the escalated error rates associated with rapid communication, guaranteeing minimal latency and reduced retransmission events.
Figure 3 offers a representative layout of an 800G chip, characterized by 8 lanes of 100G SerDes, an 800G PCS, MAC, test logic, and an application interface. Having robust testability, debuggability features with an application SW interface is the key to silicon success.
Figure 3 An example of a block diagram for an 800G chip with 8 lanes of 100G SerDes
This 800G Ethernet model is conceived to act as the network interface for next-gen data center ecosystems. To parallel the escalating demands of current CPU, bus, and storage capacities, servers need to support an aggregated 800Gb/s throughput, anchored by a Network Interface Card (NIC) requiring 100G per lane SerDes and cabling technology. Such technology needs to support 100 Gb/s per physical lane via copper twin-ax cable or fiber.
Given the increased bandwidth to endpoints, uplinks from Top-of-Rack (TOR) switches will need to transition from 256 lanes of 56- 100 Gb/s SerDes to 512 lanes of to 100 Gb/s SerDes with 800G ports while ideally maintaining the same per-lane breakout capability and reach. To maintain the overall system performance with many lanes switching simultaneously and to compensate for increasing crosstalk, designers are building a lot of margins into their designs. Carefully optimized bump-maps in the design help to mitigate any IR drop issues, package stacks and route escapes on die and through the package are more important than ever. A 51.2T switch on-die implementation requires both north-south and east-west 800G Ethernet tiles.
These re-usable tiles can be constructed and replicated multiple times in all edges of the die to minimize the beachfront. Package escape, signal integrity and power integrity analysis provide the groundwork for the basic floorplanning of the 800G Ethernet tile. Hardening the 800G Ethernet test chip involves expert knowledge of block partitioning for optimizing beachfront, SerDes, PCS and MAC design. Full visibility into the end-to-end datapaths is needed for retiming, pipelining or optimizing latency versus power and area. Lastly, it involves SoC-level full RTL-to-GDS flow for front-end and back-end integration and close collaboration with EDA tools for signoff.
Figure 4 illustrates the various use-cases of an end-to-end 800G implementation from 51.2T switches, 800G pluggables, extended-reach DACs or active copper cables for in-rack communications to 800G pluggable optics for extended reaches.
Figure 4. End-to-end 800G implementation use cases
Advantages of Engaging in Standards Development
Standards merge competitors and customers, aligning them towards a mutual goal. Actively participating in standard development bestows teams with an edge in design planning, understanding implementation nuances, and calibrating internal strategies with projected timelines. All these even before many have set their course.
While some might argue that standards take considerable time to mature, their longevity and relevance are evident. Dig out an old Ethernet gadget, connect it, and it seamlessly integrates. As SoCs grow more intricate, addressing the requirements of diverse applications generating massive data, the foundation standards will also progress. Engaging early in these developmental stages is paramount. Those lagging risk miss the market’s first mover advantage.
Summary
The ongoing evolution of Ethernet, in particular the implementation of 800G Ethernet and the path to 1.6T speed, needs a robust, silicon-proven 800G Ethernet solution. Designers are looking to focus on their core competencies and looking for proven solutions to help ensure a fast time-to-market. The involvement of bodies like the Ethernet Alliance facilitates the market with seamless integration and interoperability with the complex and wide Ethernet ecosystem.
With the ongoing standardization work being done in 800G (IEEE P802.3df) and 1.6T Ethernet (IEEE P802.3df), Ethernet’s future looks bigger and brighter than ever. To learn more about IEEE 802 Ethernet standards, please visit www.ieee802.org.
The views and opinions expressed are those of the authors and do not necessarily reflect the official policy or position of the Ethernet Alliance.
