• Skip to main content
  • Skip to primary sidebar
  • Skip to footer
  • Home
  • About
    • Stephen Foskett
      • My Publications
        • Urban Forms in Suburbia: The Rise of the Edge City
      • Storage Magazine Columns
      • Whitepapers
      • Multimedia
      • Speaking Engagements
    • Services
    • Disclosures
  • Categories
    • Apple
    • Ask a Pack Rat
    • Computer History
    • Deals
    • Enterprise storage
    • Events
    • Personal
    • Photography
    • Terabyte home
    • Virtual Storage
  • Guides
    • The iPhone Exchange ActiveSync Guide
      • The iPhone Exchange ActiveSync Troubleshooting Guide
    • The iPad Exchange ActiveSync Guide
      • iPad Exchange ActiveSync Troubleshooting Guide
    • Toolbox
      • Power Over Ethernet Calculator
      • EMC Symmetrix WWN Calculator
      • EMC Symmetrix TimeFinder DOS Batch File
    • Linux Logical Volume Manager Walkthrough
  • Calendar

Stephen Foskett, Pack Rat

Understanding the accumulation of data

You are here: Home / Everything / Computer History / Musing: Could We Replace Ethernet With PCIe?

Musing: Could We Replace Ethernet With PCIe?

February 9, 2015 By Stephen 3 Comments

Greg “EtherealMind” Ferro recently “mused” that it might be a good idea to replace PCI Express (PCIe) inside servers or rack-scale infrastructure with Ethernet. But this seems to be the exact opposite of the direction the industry is headed. Rather than replacing PCIe with Ethernet, companies like Intel seem set on replacing short-range Ethernet (in rack-scale systems) with PCIe!

oUc4nZF

PCIe vs. Ethernet

Greg points out (rightly) that electrical signals copper traces on motherboards are currently limited to 15.75 Gbps in PCIe 4.0. With 16 lanes, this brings us to 252 Gbps of throughput on the PCIe 4.0 bus. Greg is also correct that current Ethernet switches operating at 25 Gb can handle this kind of throughput across 10 or so connections. QED, right?

Read Greg’s post: Musing: Could We Replace PCIe Bus With Ethernet Switch?

Sorry, Greg! Stuffing an Ethernet switch into a server is exactly the wrong direction for many reasons.

Most pressing is the issue of latency. PCIe latency is measured in hundreds of nanoseconds, while Ethernet interconnects are measured in tens of microseconds. This might not sound like much, but it’s literally two orders of magnitude difference and would be a huge step back in real-world use.

Just because you can push the same amount of data across a link (throughput) doesn’t mean you can do the same tasks. PCIe is like a fleet of shopping carts filling the aisles at your local Costco, while Ethernet is the street of SUVs taking those big boxes of cereal and lightbulbs back home. Although they are theoretically carrying the same payload, Explorers and Caravans just weren’t designed to navigate inside the store!

There are many other issues to consider as well. Ethernet NICs and switches are complex, being designed to handle the vagaries of topology changes, speed differences, and relatively frequent reconfiguration. An in-server Ethernet variant could be stripped down to the basics and integrated into the chipset just like PCIe but this would obviate the external connectivity benefits suggested by Greg. So every device would have to be a full-featured Ethernet endpoint, likely with TCP/IP besides!

Rack-Scale Computing, OPCIe, and SiPh

Greg mentions Intel’s work on rack-scale computing and silicon photonics. Good! But then he suggests running Ethernet over this lovely next-generation interface. Bad!

The intent is to run PCIe over all those integrated silicon/optical interconnects and extend it to rack-scale, rather than ingesting Ethernet. This has a whole raft of benefits, including better real-world performance (thanks to low latency and little protocol overhead) and easier integration, since PCIe is already in use at all points in a rack-scale infrastructure.

IT folks usually express some serious skepticism when I mention PCIe as an externally-exposed interconnect. But then I point out that this entire system is already in use! Apple’s Thunderbolt is simply PCIe over copper DisplayPort cables and long-range optical cables are on sale today. Intel’s silicon photonics (SiPh) optical PCIe (OPCIe) technology has been sampling for over a year now, and Fujitsu has demonstrated a server using these optical interconnects for peripheral interconnection.

Proponents of rack-scale computing seem poised to adopt OPCIe as an interconnect within the rack in the next year or so. This will encroach on the market for current server-to-server and server-to-storage interconnects like Ethernet, Fibre Channel, and InfiniBand. Enterprise products based on OPCIe are being developed as well, though few if any have yet been announced.

You might like to read my Rack Endgame series:

  1. The Fat Middle: Today’s Enterprise Storage Array
  2. Virtualized and Distributed Storage: This Time For Sure!
  3. The Rack Endgame: A New Storage Architecture For the Data Center

Note that Fibre Channel, InfiniBand, RapidIO, and many other technologies besides have attempted to do just what Greg is suggesting: Unify internal and external connectivity with a “master” protocol. But none have succeeded. It seems more logical to standardize on a fast, scalable, low-latency bus like PCIe for short-range communication and a ubiquitous network like Ethernet for longer-range use.

Stephen’s Stance

Rather than pushing Ethernet into the server, the industry is pushing it out of the rack. Soon, racks will function like blade chassis, with high-speed interconnects for internal communication and Ethernet termination points for communication outside the rack. Probably the closest thing to reality in Greg’s vision is the concept of tunneling Ethernet over PCIe and integrating it into server chipsets. This would function something like FCoE, providing a path for a legacy interconnect (Ethernet) right into the heart of the new converged rack.

Note that the title of this piece if farcical, and based on Greg’s title. No, we cannot replace all of Ethernet with PCIe. It’s still the king of the campus and larger-radius networks. But it has no place in the world of PCIe!

You might also want to read these other posts...

  • Introducing Rabbit: I Bought a Cloud!
  • Powering Rabbits: The Mean Well LRS-350-12 Power Supply
  • Tortoise or Hare? Nvidia Jetson TK1
  • Electric Car Over the Internet: My Experience Buying From…
  • How To Connect Everything From Everywhere with ZeroTier

Filed Under: Computer History, Enterprise storage, Virtual Storage Tagged With: Ethernet, Fibre Channel, Fujitsu, Greg Ferro, InfiniBand, Intel, OPCIe, PCIe, rack-scale, RapidIO, Silicon Photonics, SiPh, Thunderbolt

Primary Sidebar

Science is a differential equation. Religion is a boundary condition.

Alan Turing

Subscribe via Email

Subscribe via email and you will receive my latest blog posts in your inbox. No ads or spam, just the same great content you find on my site!
 New posts (daily)
 Where's Stephen? (weekly)

Download My Book


Download my free e-book:
Essential Enterprise Storage Concepts!

Recent Posts

How To Install ZeroTier on TrueNAS 12

February 3, 2022

Scam Alert: Fake DMCA Takedown for Link Insertion

January 24, 2022

How To Connect Everything From Everywhere with ZeroTier

January 14, 2022

Electric Car Over the Internet: My Experience Buying From Vroom

November 28, 2020

Powering Rabbits: The Mean Well LRS-350-12 Power Supply

October 18, 2020

Tortoise or Hare? Nvidia Jetson TK1

September 22, 2020

Running Rabbits: More About My Cloud NUCs

September 21, 2020

Introducing Rabbit: I Bought a Cloud!

September 10, 2020

Remove ROM To Use LSI SAS Cards in HPE Servers

August 23, 2020

Test Your Wi-Fi with iPerf for iOS

July 9, 2020

Symbolic Links

    Featured Posts

    FCoE vs. iSCSI – Making the Choice

    May 20, 2011

    The Four Horsemen of Storage System Performance: The Rule of Spindles

    August 25, 2010

    Virtual Machine Mobility: Of What, and to Where and in What State?

    January 16, 2012

    Why I Am Biased Against FCoE

    October 21, 2011

    GPS Time Rollover Failures Keep Happening (But They’re Almost Done)

    April 6, 2019

    Nimble Storage Rolls Out an All-Flash Array

    February 24, 2016

    Go Get a ProtonMail Account and Protect Your Online Life!

    July 19, 2017

    Free as in Coffee – Thoughts on the State of OpenStack

    May 2, 2016

    Microsoft: Kill the Craptops Before They Destroy Windows!

    January 7, 2013

    It’s Time To Move Beyond Passwords (Especially On Web Sites)

    January 8, 2016

    Footer

    Legalese

    Copyright © 2022 · Log in