Once upon a time there was Ethernet. Every half decade or so, the industry got together and worked out a faster version. Sometimes they didn’t totally agree, but a standard emerged at 10x the speed of the previous version. Throw all that out the window: Faster Ethernet is coming, and it’s going to be weird!
Although it looks like a simple network Channel, Ethernet is anything but. Ever since the development of “Fast Ethernet” (that’s 100 Mb Ethernet to you and me), the IEEE has considered using multi-lane communications in various Ethernet standards. Today’s common 1000BASE-T standard multiplexes 125 Mbps signals, and 10GBASE-CX4 uses four separate 3.125 Gbps lanes.
Faster Than 10 Gb Ethernet in the Datacenter
This multi-lane weirdness is really amped up when we look beyond 10 Gb Ethernet! Essentially, there are two popular lane or channel rates:
- 10.3125 Gbps used alone (as 10 Gb Ethernet) or in 40 Gb or 100 Gb Ethernet (and 1G and NBASE-T, which I’ll get to in a moment). Intel and Cisco seem to like this set of speeds.
- 25.78125 Gbps is also used alone (as 25 Gb Ethernet) or in 50 Gb or 100 Gb Ethernet. This is popular with NIC and switching vendors like QLogic and Avago (via acquisitions of Broadcom and Emulex).
Already, the datacenter server market, looking beyond 10 Gb Ethernet, is facing a choice between the Intel-led 40/100 roadmap or the alternative path of 25/50/100 Gb. Avago, having acquired Broadcom and Emulex, seems to be the lead cheerleader behind the latter, though QLogic and Mellanox are also pushing it. And of course everyone is acting very friendly in public, knowing that customer adoption will ultimately drive which standard is adopted.
So far, 100 Gb Ethernet is touted as an inter-switch link, while 40 Gb or 25/50 Gb Ethernet is the focus for server interconnects. Available PCI Express bandwidth is one reason for this segmentation, since it is very difficult for any server to saturate a 100 Gb link. But PCIe Gen 4 will double per-lane throughput, opening the door to 100 Gb servers sooner rather than later. And the challenge of InfiniBand, already pushing past 50 Gb, means greater interest in 100 Gb Ethernet in high-performance computing.
The question comes down to the two roadmaps for speed between 10 Gb and 100 Gb: Do you hold out for (potentially more expensive) 40 Gb or jump on 25/50 Gb? Are you with Intel or the other chipmakers? What kind of cabling do you have or prefer?
Less Than Ten, More Than One
Many servers and network devices need more than 1 Gb of throughput but don’t want or need 10 Gb and the potential expense of fiber optic connections. That’s where NBASE-T comes in. This new standard would allow 2.5 Gb or 5 Gb Ethernet over existing copper cabling. Although the NIC and switch would need to be changed, NBASE-T has wonderful potential beyond the datacenter to provide more performance with little additional investment.
This is especially true for Wi-Fi access points, which might need more than 1 Gb of backhaul but have to make do with existing cabling. The next generation of Cisco AP’s and campus switches will likely support NBASE-T out of the box and customers will see much improved performance. Another benefit of NBASE-T is Power over Ethernet: Wi-Fi AP’s need PoE but power requires copper, making fiber-based 10 Gb Ethernet use even more difficult.
NBASE-T is a compromise between the hardware vendors, with the Intel side seeming to win out. It’s essentially 10 Gb Ethernet “slowed down” for 2.5 and 5 Gbps rather than a totally new standard. And it can run at 1 Gb too, an intriguing possibility as an alternative to today’s aged 1000BASE-T.
Here’s a great presentation on NBASE-T from Cisco at Tech Field Day
Ideally, future Ethernet chips would automatically detect all these speed combinations and negotiate the best-possible performance for any given connection. Indeed, it’s nice to see the IEEE talking about 2.5 and 5 in the same breath with 10, 25, and 40. But this isn’t a reality today. Buyers have to make a choice and ensure that both ends of the wire support whichever speed they select.
Stephen’s Stance
“One size fits all” doesn’t work for Ethernet, but this proliferation of speed options sounds like trouble without automatic capability negotiation. It’s nice to have options, but the IEEE must remain focused on interoperability and rein in the interests of the various companies proposing next-generation Ethernet technologies.