TEF 2021: The Road beyond 400 Gigabit Ethernet- Opportunities and Challenges

During the last week of January 2021, the Ethernet Alliance, a global consortium focused on the advancement of Ethernet technology, held its flagship Technology Exploration Forum (TEF) event where I had the opportunity to present insights on the market need for next Ethernet speeds. Attendees also heard from a wide variety of industry voices including system manufacturers, component suppliers, and end-users. Below are some of my key-takeaways from the event:

• Accelerated workloads and memory disaggregation will fuel the need for speeds higher than 400 GE:

One of the main topics discussed during the event is around applications driving the need for speeds beyond 400 GE. We heard from some of the large hyperscalers—such as Google, Microsoft and Facebook—about the explosive growth in bandwidth requirement inside their data centers. Their need for bandwidth is further fueled by the emergence of accelerated workloads such as machine learning (ML) and artificial intelligence (AI).

Source: TEF2021-Facebook: Co-packaged Optics in the Datacenter

Memory disaggregation is another application that is putting tremendous pressure on network performance. And of course numerous other applications and trends are driving the growth in IP traffic: increasing number of connected devices, growth in high resolution 4K video, machine-to-machine connections, mobility, etc.

• Both 800 GE and 1.6 TE will have a place:

With the availability of 100 G SerDes, moving from 400 GE based on 8 lanes of 50 G SerDes to 800 GE based on 8 lanes of 100 G SerDes seems a logical step. However, would doubling the speed from 400 GE to 800 GE provide enough increase to keep up with insatiable bandwidth requirements, especially from the hyperscalers? Or given that the last major speed transitions quadrupled the speed from 100 GE to 400 GE would a direct transition to 1.6 TE make more sense? During my TEF presentation, I provided potential use cases for 800 GE based on end-user requirements and Rakesh Chopra, a Cisco fellow, presented similar use cases.

Source: Dell’Oro Group

We both agree that Ethernet switch speeds will continue to bifurcate, and industry will need both 800 GE and 1.6 TE. It’s a question of timing: 1.6 TE requires 200 G SerDes which is still in development, whereas 800 GE can leverage 100 G SerDes which will ship in large volume over the next five years.

• Power consumption is one of the major constraints for speeds beyond 400 GE:

The industry has done a fairly good job keeping up with the ever increasing bandwidth requirements. Over the last 12 years, the industry delivered 7 switch chip generations, going from 640 Gbps per chip capacity to 25.6 Tbps and soon to 51.2 Tbps. This progression represents about an 80-fold increase in bandwidth capacity in just over a decade. Unfortunately, in the meantime power consumption has also skyrocketed, growing by a factor of more than 20 during the same time period. This increase in power consumption is driven by various components at the switch level (SerDes, Optics, Fans) as well as at the network level. Network topology and the number of tiers in the network have a direct impact on power consumption.

Because data centers have limited power budgets, network speeds can’t increase indefinitely without flattening the curve of power consumption. As a reminder, Cloud SPs do not make money by running networks but rather by running applications on their servers. Therefore, within each data center their goal is to maximize the number of servers deployed, which is dictated by the power budget. In other words, more power consumed by a network means fewer servers can be deployed.

• Co-packaged optics (CPO) are needed to reduce power consumption:

CPO is a new technology that promises to reduce power consumption compared to pluggable optics. However, CPO has several challenges such as serviceability, lack of flexibility to mix and match optics, and lack of standards. Nevertheless, despite these challenges CPO is almost inevitable if the industry is to move to 102.4 Tbps chips. Some hyperscalers that are more constrained by power availability are planning to start deploying CPO even at 51.2 Tbps. Another motivation behind adopting CPO at 51.2 Tbps is the desire to ride the learning curve as early as possible to be ready when CPO becomes the only possible option instead of pluggable optics.

• Future data center networks may require a change in network architecture:

In addition to CPO, speeds beyond 400 GE might also require optimized network architectures with fewer tiers to reduce power consumption. However, to keep the same number of connected servers while reducing the number of network tiers the switch radix has to increase.

Source: TEF2021-Cisco: Impact of Switch Radix

Unfortunately, higher switch radix might also result in complex cabling and lower link utilization. We will keep an eye on how hyperscalers will make tradeoffs between the number of network tiers and the switch radix. I expect hyperscalers to make different choices depending on their network architecture and power budget.

Over the years it has been always exciting to watch how technical challenges in this industry turned into opportunities for new entrants or for the incumbents to disrupt the status quo. With the move to speeds beyond 400 GE it will be equally exciting to see how the market will adapt from an ecosystem perspective and how network topologies and architectures will evolve.

For more details about TEF 2021 you can access the on-demand version here.

About Dell’Oro Group

Founded in 1995 with headquarters in the heart of Silicon Valley, Dell’Oro Group is an independent market research firm that specializes in strategic competitive analysis in the telecommunications, networks, and data center IT markets.  Our firm provides world-class market information with in-depth quantitative data and qualitative analysis to facilitate critical, fact-based business decisions. Visit us at www.delloro.com.

About Dell’Oro Group Research

To effectively make strategic decisions about the future of your firm, you need more than a qualitative discussion – you also need data that accurately shows the direction of market movement. As such, Dell’Oro Group provides detailed quantitative information on revenues, port and/or unit shipments, and average selling prices – in-depth market information to enable you to keep abreast of current market conditions and take advantage of future market trends. Visit us at www.delloro.com/market-research.