I’m trying to figure out just how much data I can move through a modern computer. To grok that I need to understand the bandwidth capabilities of the various types of peripherals one can attach to a modern computer. I therefore run through below what appear to be the main peripheral types and what kind of bandwidth they can maintain. Wikipedia has a great summary of the bandwidths of various technologies. Note however that the bandwidths listed in that article are raw (except, confusingly, when they are not) bandwidth, not data bandwidth, below I try to find numbers that represent actual data bandwidth.
1 PCIe v3
PCIe standards for Peripheral Component Interconnect Express and I’m writing about it’s third generation version. PCIe is meant for attaching physical devices to motherboards and modern CPUs actually put their PCIe controllers right on the chip so in essence PCIe allows one to connect devices right to the CPU.
The PCIe hardware design is based on lanes. Each lane is a single bit fully bi-directional channel. A PCIe interface can be made up of anywhere from 1 to (in theory) 32 lanes. Each group of lanes, called a connection, are point to point and therefore independent from each other. That is, unlike say FireWire, where in theory all devices are chained together, PCIe has a hub and spoke model. This means that different devices can connect at different speeds and not interfere with each other.
Each lane operates at 8 GigaTransfers/Second (GT/S). Connections with multiple lanes simply stripe data across the lanes. An error correcting format is used on PCIe v3 that reduces effective bandwidth by 1.5%. But the actual bandwidth achieved by a PCIe v3 connection depends on what kind of data is being transmitted. If lots of little bits of data are being sent then there will be a lot of command overhead. But if long serial read/writes are being done then efficiency apparently can get up to around 95% of theoretically maximum. So in theory a single lane has an effective bandwidth of 8 GT/s * 1 bit/lane = 8 Gb/s = 1 GB/s. But, again, the lane is fully bi-directional so this is 1 GB/s/lane in each direction.
In practice however things don’t get that fast. For example, I looked up the Z-Drive R4 CloudServ storage solution from oczenterprise.com (it showed up when I looked for insanely fast PCIe ssd drives) and it’s a x16 (meaning 16 lane) connector that can handle about 6 GB/s. In theory it should handle 16 * 1GB/s = 16 GB/s. But in this case one assumes that the underlying flash technology can only go so fast.
2 InfiniBand
InfiniBand is a lot like PCIe, it’s based on bi-directional serial links that can be grouped together, in this case into either groups of x4 or x12. Mostly I saw what are called QSFP or Quad Small Form Factor connectors which are x4 connectors. InfiniBand links can operate at five different speeds:
| Link Speed Name | Effective Bandwidth/Link |
| SDR - Single Data Rate | 2 Gb/s = 0.25 GB/s |
| DDR - Double Data Rate | 4 Gb/s = 0.5 GB/s |
| QDR - Quad Data Rate | 8 Gb/s = 1 GB/s |
| FDR - Fourteen Data Rate | 13.64 Gb/s = 1.705 GB/s |
| EDR - Enhanced Data Rate | 25 Gb/s = 3.125 GB/s |
Note that the quoted bandwidth accounts for encoding overhead, the raw bandwidth is higher.
The ’magic sauce’ for InfiniBand is that it supports switching so you can use it to create a fabric of InfiniBand nodes. PCIe, on the other hand, is strictly point to point. So InfiniBand is really a competitor to Ethernet for setting up local switching fabrics to connect any kind of shared resources such as machines and storage.
I’ve noticed that InfiniBand connectors tend to terminate in PCIe v3 ports. This makes sense since even 4xEDR maxes out at 12.5 GB/s which can be easily handled by a x16 PCIe v3 connector.
3 QPI
QPI stands for QuickPath Interconnect and is an Intel technology that they use primarily these days for connecting processors to each other to create NUMA architectures. That’s why it contains fun features like a snoop protocol to deal with issues like cache coherency (e.g. if two processors have copies of the same data in their cache and one of them writes to the data the other cache entry has to be expired). Much like PCIe or InfiniBand, QPI is lane based using 20 lanes that are fully bi-directional. Due to error correcting overhead only 16 of the 20 lanes actually carries data. QPI is also double data rate, like DDR RAM, so each clock cycle moves 32 bits of data. So the total bandwidth is clock speed * 2 for double rate * 2 bytes/transfer == GT/s * 2 bytes/transfer. So a modern QPI connection runs at a clock rate of 4 GHz which is 8 GT/s which gives a total single directional bandwidth of 16 GB/s. Intel will advertise this bandwidth rate as 32 GB/s but they are accounting for both directions at once. This is the equivalent of arguing that a freeway allows 120 MPH because each side, going in different directions, allow for 60 MPH.
4 SATA
The Serial ATA standard is used for connecting storage and these days typically passes through some kind of adapter that eventually dumps into a PCIe bus. There are three major versions of SATA, 1, 2 and 3. 1 ran at 1.5 Gb/s, 2 ran at 3 Gb/s and 3 ran at 6 Gb/s. SATA, like all the previous and as implied by its name SATA uses a serial bus. Their error encoding is 8b/10b so to get the real theoretical maximum data bandwidth one has to multiply the advertised bandwidth by .8. So 1 - 0.15 GB/s, 2 - 0.3 GB/s and 3 - 0.6 GB/s. In practice HDDs don’t get near the SATA speeds. For example, Tom’s Hardware’s fastest tested HDD when I wrote this article was a Western Digital VelociRaptor that had a maximum measured read rate of 0.2 GB/s which SATA 2 could handle. But that makes sense since SATA 3 was actually introduced to handle SSD drives. According to tests I found on Tom’s Hardware modern SSDs for sequential reads get pretty close to the 0.6 GB/s theoretical maximum speed for SATA 3. This makes sense because PCIe SSDs, as mentioned above, get into the 6 GB/s.
A quick side note is SATA DOM or Disk On Module. This is a form factor where the a flash drive is built into the SATA connector and fits directly into the motherboard. This gets mentioned because typically SATA 3 drives will get connected into a SAS controller. But some motherboards also support SATA DOM which allows connecting a SATA DOM drive directly to the motherboard.
5 SAS
Serial attached SCSI. SAS is intended for more industrial strength applications than SATA. It supports longer cables, complicated multi-device architectures, etc. But it’s speeds and encodings are the same as SATA and in fact SATA and SAS work together to make sure that SATA drives can easily hook into SAS backplanes by keeping timing and such identical. There are announced plans for future versions of SAS that will support 12 Gb/s and 24 Gb/s speeds.
6 DMI/ESI
Direct Media Interface (also sometimes called the Enterprise Southbridge Interface). I’ve had a lot of trouble finding out what exactly this is. I found one random forum posting somewhere on the Intertubes that claimed it’s really just PCIe with a custom protocol on top. In modern systems it seems to be primarily used as a way to connect legacy hardware standards and so is used for lower bandwidth applications.
7 Ethernet
Ethernet and I go way back so my main interest is just in what the modern speeds are. Most machines I run into have Gigabit Ethernet (0.125 GB/s). But I see plenty of 10 Gigabit Ethernet in data centers (1.25 GB/s). I personally haven’t run into 40 Gigabit Ethernet (5 GB/s) or 100 Gigabit Ethernet (12.5 GB/s) but they are standards. My Internet foo might be weak but 40 and 100 Gigabit Ethernet seem to just be used for switching infrastructure so far. At least I couldn’t find any say PCIe 100 Gigabit Ethernet ports for individual machines. There is frame overhead for Ethernet packets but it tends to be in the 2.5% range so in theory 100 Gigabit Ethernet gets around 12.1875 GB/s not 12.5 GB/s. There are also control packets that reduce the number a little further. In the old days when Ethernet systems were connected by hubs actual bandwidth was usually 70%-80% of maximum or even lower due to collisions. However these days I pretty much only see switch and/or router based systems so collisions just aren’t an issue and bandwidth should (famous last words) be more or less at the theoretical maximum.
In QPI text you write
“single directional bandwidth of 16 GB/s. Intel will advertise this bandwidth rate as 32 GB/s but they are accounting for both directions at once. This is the equivalent of arguing that a freeway allows 120 MPH because each side, going in different directions, allow for 60 MPH.”
IMO, this comparison isnt correct, because you compare speed to bandwidth. GB/s would corresponds to cars/s, which is indeed doubled in case of single->bidirectional freeway.
I agree your analogy is internally consistent but is it how most folks would understand Intel when they say “32 GB/s”? When my local Internet Service Provider advertises “1 Gigabit Internet” they mean that both the uplink and downlink can independently run at 1 Gigabit/s. They don’t say “2 Gigabit Internet” and expect me to figure out that they actually mean the aggregate of the two different directions. So I would argue that given the standards of the industry in terms of how bandwidth is advertised I think Intel’s statement of 32 GB/s would mislead most folks.