Computer networks, from congestion control to datacenter fabrics and modern transport

Why this subject matters now

For most of its history, networking was a subject practitioners could treat as infrastructure. TCP was tuned for the wide-area Internet, the wide-area Internet changed slowly, and the interesting engineering happened above the socket. Three shifts ended that comfort. First, the datacenter became the dominant network environment, and it inverted every assumption the classical transport stack was built on, with round-trip times of tens of microseconds instead of tens of milliseconds, loss caused by switch-buffer overflow in microbursts rather than by congested long-haul links, topologies with thousands of equal-cost paths instead of one, and applications (storage disaggregation, RPC fanout, distributed training) that are sensitive to the tail of the latency distribution rather than its mean. A congestion controller that needs a packet loss to learn anything, on a path where a loss costs a retransmission timeout longer than the entire request, is the wrong tool, and the field responded with ECN-based control (DCTCP), rate-based control (BBR, DCQCN), and hardware transport (RDMA over lossless Ethernet), each with its own failure modes that an engineer is now expected to understand.

Second, the transport layer itself became a competitive, evolving artifact again. QUIC moved reliability, congestion control, and encryption into userspace over UDP, was deployed to billions of users by 2017, and was standardized as RFC 9000 in 2021. A third of web traffic now runs on a transport that did not exist fifteen years ago. Third, machine learning made the network a first-order term in the cost of computation. When a training step must allreduce gradients across thousands of accelerators between optimizer updates, the difference between a ring schedule and a naive gather-broadcast, or between an oversubscribed and a full-bisection fabric, shows up directly as idle GPU seconds, and GPU seconds are the most expensive commodity in the building. The arithmetic on this page, bandwidth-delay products, \(2(N-1)/N\) collective costs, bisection counts for a \(k\)-ary fat tree, is the arithmetic that decides whether a cluster design is sound before anyone racks a switch.

Layering and the end-to-end argument

What layering actually buys

The layered model is best understood not as a taxonomy but as a contract structure. Each layer offers a service to the layer above and makes demands of the layer below, and the value of the arrangement is that the parties can change independently. Ethernet became 40 years faster without TCP noticing, and TCP's congestion control was replaced several times without applications noticing. The working set of layers in practice is five. The physical layer moves bits over a medium. The link layer moves frames between directly connected interfaces and handles medium access. The network layer (IP) moves datagrams end to end with no promises, best-effort delivery, possibly reordered, duplicated, or dropped. The transport layer turns that raw datagram service into something applications can use, either a reliable ordered byte stream (TCP) or tagged datagrams (UDP). The application layer is everything above the socket. The thin waist of the hourglass is IP, a single, minimal, universal interconnection layer, with diversity above it and below it. The design bet, made in the 1970s and still paying out, is that the network core should be as dumb as possible and the intelligence should live at the edges.

application   HTTP/3, RPC, NCCL, SSH          (edges only)
transport     TCP, UDP, QUIC                  (edges only)
   ─────────────── the thin waist ───────────────
network       IP: best-effort datagrams       (every router)
link          Ethernet, InfiniBand, WiFi      (per hop)
physical      SerDes, DSP, optics, radio      (per hop)

The end-to-end argument

The classic justification for that bet is the end-to-end argument of Saltzer, Reed, and Clark (1984). Stated carefully, a function (reliability, ordering, deduplication, security) can be completely and correctly implemented only with the knowledge and help of the application standing at the endpoints. Implementing it inside the network can therefore at best be an optimization, never a replacement for the endpoint implementation, and it should be pushed into the network only when the performance win justifies the cost every other user of the network pays for it. The file transfer example from the paper makes it concrete. Suppose every link performs hop-by-hop error detection and retransmission, so packets are never corrupted in flight. The transfer can still fail. The file can be corrupted reading from the source disk, in the sender's buffer, in a router's memory between two perfectly reliable links, or writing to the destination disk. A careful application must therefore verify an end-to-end checksum and be prepared to retry the whole transfer regardless of what the links do, and once it does, the hop-by-hop machinery is redundant for correctness. It may still be worth having, on a link with a high error rate, as a performance optimization, which is exactly the status of link-layer retransmission in WiFi.

The argument is the single most reused piece of reasoning in systems design, and both of its edges matter. It explains why TCP checksums, sequence numbers, and retransmission live in end hosts and why the Internet core keeps no per-connection state. It also explains, run in reverse, most of the datacenter material later on this page. When the performance cost of endpoint-only implementation became intolerable (software TCP burning CPU at 100 Gb/s), the industry moved reliability into NIC hardware (RDMA) and paid the predicted price, a network that now must carry correctness-critical state (lossless flow control) in its core, with failure modes (PFC storms, deadlocks) exactly of the kind the 1984 paper warned about.

Reliable delivery and sliding windows

Stop-and-wait, and why it fails

The primitive problem is to deliver a byte stream, in order, exactly once, over a channel that drops, reorders, duplicates, and delays. Every solution is built from three parts, sequence numbers to name data, acknowledgments to report receipt, and timers to recover from silence. The simplest protocol, stop-and-wait, sends one packet and waits for its ACK before sending the next. Its throughput is bounded by one packet per round trip, \( \text{rate} \le S/\text{RTT} \) for packet size \(S\), which is hopeless on any modern path. A 1500-byte packet per 70 ms transcontinental round trip is 171 kb/s regardless of link speed. The fix is pipelining, keeping a window of \(W\) packets in flight simultaneously, so the rate becomes

Setting the two equal gives the most important number in transport engineering, the bandwidth-delay product. The window needed to fill a path is \(W \cdot S = B \times \text{RTT}\), the number of bytes that fit in the pipe. Everything about window sizing, buffer sizing, and congestion control is a negotiation about who gets to fill how much of the BDP.

The table (computed for this page, with arithmetic verifiable by hand, such as \(100\times 10^9 \,\text{b/s} \times 10^{-5}\,\text{s} / 8 = 125{,}000\) bytes) shows why no single window default can be right. The datacenter path needs 83 packets in flight, the transcontinental path needs 58,333. It also shows why TCP's original 16-bit window field (max 65,535 bytes) became a hard ceiling, and why RFC 7323 window scaling, which left-shifts the advertised window by up to 14 bits for a maximum of 1 GiB, is mandatory on every fast path today.

Sliding-window mechanics

The sender maintains two pointers into the byte stream, the oldest unacknowledged byte and the next byte to send. Their difference is bounded by the window \(W = \min(\text{cwnd}, \text{rwnd})\), the lesser of the congestion window (the sender's estimate of what the network can absorb, the subject of the next section) and the receive window (what the receiver's buffer can absorb). The receiver acknowledges cumulatively, so an ACK for sequence number \(x\) means every byte before \(x\) has arrived. Cumulative ACKs are robust (any ACK loss is repaired by the next ACK) but information-poor. A single hole blocks the ACK number from advancing even as later data arrives. Selective acknowledgment (SACK, RFC 2018) fixes this by letting the receiver report up to three received blocks beyond the hole, so the sender can retransmit exactly what is missing rather than guessing. Go-Back-N, the textbook alternative that retransmits everything from the hole forward, wastes a full window of transmission on a single loss. With the transcontinental window above, one lost packet would trigger 87.5 MB of redundant retransmission.

The Jacobson-Karels retransmission timer

Timers back up everything else, and the retransmission timeout (RTO) must be computed, not configured. Too short, and spurious retransmissions waste bandwidth and confuse loss-based congestion control. Too long, and every genuine loss stalls the connection. The 1988 solution, due to Jacobson with Karels, is two coupled exponentially weighted moving averages over RTT samples \(m\), a smoothed RTT and a smoothed mean deviation,

The deviation term is the insight. An RTO set at a multiple of the mean (the pre-1988 practice, \(2\times\text{SRTT}\)) fires constantly on paths with high RTT variance, which congested paths are by definition. Scaling the margin by measured variance makes the timer adapt to exactly the paths that need slack. The gains \(1/8\) and \(1/4\) are powers of two so the update is two shifts and two adds, a 1988 constraint that survives unchanged in every modern stack. A worked trace was computed for this page with \(\text{RTTVAR}\) initialized to \(m/2\). Feed the estimator a steady 100 ms RTT and the RTO converges downward through 300, 250, 212.5, 184.4, 163.3 ms. A single 300 ms outlier sample then jolts SRTT only to 125 ms but RTTVAR to 61.9 ms, snapping the RTO up to 372.5 ms in one step, after which it decays again through 332.5, 299.0, 270.8 ms as calm returns. The estimator is deliberately asymmetric in effect. One bad sample buys a large safety margin immediately, and confidence is rebuilt slowly. Karn's algorithm supplies the last necessary rule. Never take an RTT sample from a retransmitted segment (the ACK is ambiguous), and double the RTO on each successive timeout (exponential backoff) so a dead path costs exponentially decreasing traffic.

What a round trip costs on a real host

It is worth grounding "RTT" in measured numbers before building theory on it. The benchmarks in this repository were run on this machine, a 52-CPU Xeon Platinum 8480+ host with two H100s, and the caveats recorded with the data apply. It is a shared, multi-tenant, virtualized host, so every syscall and scheduler wakeup pays virtualization overhead, and no second host is reachable, so all socket measurements are loopback. Treat them as this machine's numbers, not the best achievable. A null syscall costs 140.8 ns. A 64-byte TCP ping-pong over loopback, 200,000 iterations, measures a mean RTT of 24,596 ns with p50 24,405 ns, p99 32,153 ns, and p99.9 37,138 ns. Pinning sender and echo threads to distinct cores with TCP_NODELAY brings the p50 to 23,380 ns with a minimum of 13,922 ns. The same ping-pong over a Unix domain socket takes 15,788 ns, cheaper because it skips the TCP/IP stack entirely. So a "zero-distance" RTT through two sockets is roughly 15 to 25 microseconds on this host, hundreds of syscall times. The cost is wakeups and stack traversals, not wires. Real datacenter RTTs of 10 to 50 μs are therefore dominated by end-host software, which is the entire motivation for kernel-bypass transports later on this page.

Throughput has the same syscall-dominated structure. Streaming over loopback TCP on this machine with 64-byte writes achieves 1.07 Gb/s at 477 ns per write call, 1024-byte writes reach 9.55 Gb/s, 64 KiB writes reach 31.77 Gb/s, and 1 MiB writes reach 48.73 Gb/s at 172.1 μs per call. The per-call cost grows far more slowly than the payload, so amortizing the syscall over big writes is worth 45x in throughput. A Unix socket shows the same shape more sharply, 0.51 Gb/s at 64-byte writes and 55.49 Gb/s at 64 KiB. Every high-performance networking design, from writev and sendfile through io_uring to DPDK, is a variation on this one measured fact, that the boundary crossing costs more than the copy.

Congestion control, from collapse to the sawtooth

Why the network needs it at all

Flow control (the receive window) protects the receiver, but nothing so far protects the network. In 1986 the NSFnet backbone demonstrated what happens without that protection. Congestion collapse arrived, with throughput between Lawrence Berkeley Lab and UC Berkeley dropping from 32 kb/s to 40 b/s, a factor of a thousand, as documented in Jacobson's 1988 paper. The mechanism is self-reinforcing. Queues fill, RTTs grow, naive timers expire, senders retransmit data that is still in flight, the duplicates deepen the queues, and eventually the network carries almost nothing but retransmissions of retransmissions. The cure has to be distributed. Each sender must infer, from the only signals it can see (ACK timing, loss, and later ECN marks), how much traffic the path can carry, and the collective result of every sender's private rule must converge to a state that is efficient (the bottleneck stays busy) and fair (competing flows get comparable shares). That is a distributed control problem, and it deserves the control-theoretic treatment the next subsection gives it.

Why AIMD and not the alternatives, the Chiu-Jain analysis

Model two flows sharing one bottleneck of capacity \(C\), with rates \(x\) and \(y\). Once per round trip, each flow receives the same one-bit feedback, whether the sum \(x+y\) is over or under \(C\). A linear control rule sets \(x \leftarrow a_I + b_I x\) on "under" and \(x \leftarrow a_D + b_D x\) on "over". Chiu and Jain (1989) asked which choices of the four constants converge to the fair, efficient point \(x = y = C/2\), and the answer is visible in the geometry of the \((x,y)\) plane. Additive changes (\(a \ne 0\), \(b = 1\)) move the state along the 45-degree line, preserving the difference \(x - y\). Multiplicative changes (\(a = 0\), \(b \ne 1\)) move it along the ray through the origin, preserving the ratio \(x/y\). To converge to fairness, the difference must shrink. Only the decrease step can shrink it while also reducing total load, and it does so only if the decrease is multiplicative, since halving both rates halves their difference while subtracting a constant from both leaves it intact. Symmetrically, the increase must be additive, since multiplying both rates by a constant preserves their ratio and never improves fairness. Additive increase, multiplicative decrease is not one reasonable choice among several. Within linear rules with binary feedback it is the only combination that converges to fair efficiency.

The numeric trace computed for this page makes the convergence concrete. Two AIMD flows with \(\alpha = 1\), \(\beta = 1/2\) start at the unfair allocation \((40, 5)\) on a link of capacity 100. Their difference is 35 and stays exactly 35 through every additive-increase round (both grow to \((41,6), (42,7), \dots\)), then halves to 17.5 at the first multiplicative decrease, then to 8.75, and after ten decrease events sits at 0.547. The difference trace 35, 35, 17.5, 8.75, 8.75, 4.375, 2.1875, 1.09375, 0.546875 is a pure geometric decay clocked by loss events. Jain's fairness index \( J = (x+y)^2 / (2(x^2+y^2)) \) climbs 0.623, 0.855, 0.945, 0.979, 0.992, 0.998, 0.9996, converging to 0.999988. Run the wrong rules on the same start and the geometry's verdict is confirmed. Multiplicative-increase/multiplicative-decrease ends at \((77.3, 9.7)\), the initial 8:1 ratio frozen forever, and additive-increase/additive-decrease ends at \((68, 33)\), the initial difference frozen forever.

   y (flow 2 rate)
   │        efficiency line x+y=C
   │       ╲
   │  fair  ╲          AIMD trajectory:
   │  line   ╲         additive ↗ along 45°  (difference constant)
   │   x=y ╲  ╲        multiplicative ↙ toward origin (ratio constant)
   │        ╲  ╲       net drift → toward the fair point x=y=C/2
   │      ↗ ╲↙ ╲
   │    ↗    ╳  ╲
   │  ↗    ╱  ╲  ╲
   └─────────────────── x (flow 1 rate)

Slow start, congestion avoidance, and Reno

Jacobson's 1988 implementation turned the AIMD principle into the machinery that still structures every TCP. The congestion window cwnd starts at a few segments and must find a path's capacity, which may be anywhere across five orders of magnitude. Additive increase alone would take hours to open 58,000 segments. Slow start solves the search problem. cwnd grows by one segment per ACK, doubling per RTT, an exponential probe that overshoots the capacity by at most 2x and finds it in \(\log_2\) time. The self-clocking observation makes this safe. ACKs return at the rate the bottleneck forwards data, so a sender that transmits only on ACK arrival automatically paces itself to the bottleneck rate. Once cwnd crosses the threshold ssthresh (set to half the window that last overflowed), growth switches to congestion avoidance, the additive regime of one segment per RTT (implemented as \(\text{cwnd} \mathrel{+}= 1/\text{cwnd}\) per ACK). On loss detected by three duplicate ACKs, Reno performs fast retransmit (resend the missing segment immediately, no timeout) and fast recovery, halving cwnd and continuing in avoidance, the multiplicative decrease. Only a retransmission timeout, meaning the ACK clock itself died, resets cwnd to 1 and re-enters slow start. The steady state of a long Reno flow is therefore a sawtooth. Climb linearly from \(W/2\) to \(W\), lose one packet, halve, repeat.

Deriving the \(1/\sqrt{p}\) law from the sawtooth

The sawtooth's geometry fixes the relationship between loss rate and throughput, and it is worth deriving carefully because the result governs everything from CDN tuning to why CUBIC exists. Assume a long-lived flow in steady state with segment size \(S\) (MSS), round-trip time \(R\), and a network that drops exactly one packet each time the window reaches its peak \(W\), and let \(p\) be the resulting per-packet loss probability. One sawtooth cycle runs from window \(W/2\) (just after a halving) to window \(W\) (the next loss). Additive increase adds one segment per RTT, so the climb takes \(W/2\) round trips, and the cycle duration is

The number of packets delivered in one cycle is the area under the sawtooth. The window averages \(\tfrac{1}{2}(W/2 + W) = \tfrac{3}{4}W\) over the cycle, one window's worth per RTT, so

Exactly one packet is lost per cycle, so the loss probability is the reciprocal,

Throughput is packets per cycle over cycle time (equivalently, mean window over RTT),

This is the Mathis-Semke-Mahdavi-Ott equation (1997). Padhye, Firoiu, Towsley, and Kurose (1998) extended it to include delayed ACKs (\(b\) packets per ACK, stretching the climb) and, crucially, retransmission timeouts, giving the full model \( B \approx S \big/ \big( R\sqrt{2bp/3} + T_0 \min(1, 3\sqrt{3bp/8})\, p (1+32p^2) \big) \), which matches real traces even at loss rates above a few percent where timeouts dominate. But the scaling content is already in the square root. Throughput degrades as \(1/\sqrt{p}\), not \(1/p\), because a loss costs only half the window and the window itself shrinks as losses grow, and throughput is inversely proportional to RTT, which means AIMD is systematically unfair to long paths sharing a bottleneck with short ones, a bias that datacenter incast and CDN placement both exploit and suffer from.

A packet-level simulation run for this page checks the formula's accuracy. Each row simulates a Reno sawtooth with random per-packet loss at rate \(p\), MSS 1460 bytes, RTT 50 ms, and compares delivered throughput to the Mathis prediction.

The agreement is within about 1 percent at low loss and 9 percent at \(p = 10^{-2}\), where the neglected timeout term begins to matter. Each decade of loss buys the predicted \(\sqrt{10} \approx 3.16\)x of throughput. The peak-window column confirms the derivation's other prediction, \(W = \sqrt{8/(3p)}\). At \(p = 10^{-4}\) the formula gives 163.3 against the simulated 162.4.

CUBIC and BBR, the two modern answers

CUBIC, decoupling growth from RTT

CUBIC (Ha, Rhee, and Xu, 2008), the Linux default since 2.6.19 and still the majority congestion controller on the Internet, keeps loss as the congestion signal but replaces the one-segment-per-RTT climb with a cubic function of wall-clock time since the last loss. With \(W_{max}\) the window at the last loss event and \(\beta = 0.7\) the multiplicative-decrease factor (gentler than Reno's 0.5), the window \(t\) seconds after the loss is

The constant \(K\) is chosen so that \(W(0) = \beta W_{max}\). Substituting \(t=0\) gives \(W(0) = W_{max} - CK^3 = W_{max} - W_{max}(1-\beta) = \beta W_{max}\), as required. The shape is the point. Growth is fast when far below \(W_{max}\) (the cubic's steep left arm), flattens to a plateau precisely at the old \(W_{max}\) (cautious probing near the level that last caused loss), then accelerates again if no loss occurs (the right arm, probing for newly freed capacity). Because \(W(t)\) depends on elapsed time and not on ACK count, two CUBIC flows with different RTTs sharing a bottleneck grow at the same rate, removing most of Reno's RTT bias. A TCP-friendly region makes CUBIC fall back to Reno-equivalent behavior where Reno would be faster (short RTT, small windows).

The recovery arithmetic from Problem 2 shows the win. After a loss at \(W_{max} = 85{,}616\) segments (the 10 Gb/s, 100 ms operating point), CUBIC returns to \(W_{max}\) at \(t = K = \sqrt[3]{85{,}616 \times 0.3 / 0.4} = \sqrt[3]{64{,}212} = 40.04\) seconds, independent of RTT. Reno needs \(W_{max}/2 = 42{,}808\) round trips, 4,280.8 seconds at 100 ms, so CUBIC recovers 106.9x faster, and the gap widens with bandwidth.

BBR, replacing the signal rather than the schedule

BBR (Cardwell, Cheng, Gunn, Yeganeh, and Jacobson, 2016, from Google) rejects the premise both Reno and CUBIC share, that loss, or any queue-overflow event, is the right congestion signal. On a path with bottleneck bandwidth \(BtlBw\) and minimum round trip \(RTprop\), the optimal operating point, identified by Kleinrock in 1979, keeps exactly one BDP in flight, with delivery rate \(BtlBw\), RTT \(RTprop\), and an empty queue. Loss-based control instead pushes inflight data up to BDP plus the bottleneck buffer, operating at the point where the queue is full. With the deep buffers of edge routers this inflates RTT by seconds (the bufferbloat section next), and with the shallow buffers of datacenter switches it converts every capacity probe into loss. BBR therefore estimates the two path parameters directly, \(BtlBw\) as the windowed maximum of the measured delivery rate (ACKed bytes over time, over roughly ten RTTs) and \(RTprop\) as the windowed minimum RTT (over roughly ten seconds). The two cannot be measured simultaneously, which is the algorithm's central subtlety. Measuring \(BtlBw\) requires filling the pipe, which builds queue and hides \(RTprop\), while measuring \(RTprop\) requires draining the queue, which idles the pipe and hides \(BtlBw\). BBR sequences the measurements. It paces at \(gain \times BtlBw\) with a gain cycle of 1.25 (probe for more bandwidth), then 0.75 (drain the queue the probe built), then six RTTs at 1.0, and every ten seconds enters a brief ProbeRTT phase, cutting inflight to four packets to expose the bare propagation delay. Startup uses gain \(2/\ln 2 \approx 2.885\), the smallest gain that still doubles the delivery rate each RTT, matching slow start's search speed while measuring rather than crashing.

The consequences follow from the model. BBR ignores isolated losses, so it sustains throughput on paths with modest random loss where the \(1/\sqrt{p}\) law throttles Reno and CUBIC (Google reported multi-thousand-fold improvements on lossy long-haul paths when deploying it on B4). It holds queues near empty, so latency under load drops. The costs are equally real. BBRv1 paced at its bandwidth estimate regardless of loss, and when sharing a shallow buffer with loss-based flows it could starve them (its estimate stayed high while Reno-family flows kept halving), and multiple BBR flows could sustain persistent queues on deep-buffered links. BBRv2 and the current v3 add explicit reaction to loss above a threshold and to ECN, trading some of v1's aggression for coexistence, while the pacing-plus-model architecture is unchanged. BBR carries a large fraction of Google's traffic, including YouTube, and ships in mainline Linux as tcp_bbr.c.

Queueing, bufferbloat, and AQM

M/M/1 intuition, derived

Every hop on a path is a queue in front of a transmitter, and the single most useful mental model for one is M/M/1, with Poisson packet arrivals at rate \(\lambda\), exponential service times at rate \(\mu\), one server, and an infinite buffer. Its stationary distribution follows from a one-line balance argument. In steady state, the probability flow from state \(n\) (n packets in system) to \(n+1\) must equal the reverse flow, \(\lambda P_n = \mu P_{n+1}\), so \(P_{n+1} = \rho P_n\) with \(\rho = \lambda/\mu\), giving the geometric distribution \(P_n = (1-\rho)\rho^n\) after normalizing (the sum \(\sum \rho^n = 1/(1-\rho)\) requires \(\rho < 1\)). The mean occupancy is the geometric mean

and Little's law \(L = \lambda W\), which holds for any stable queueing system regardless of distributions, converts occupancy to delay,

The reading matters more than the formula. Total delay is the bare service time \(1/\mu\) multiplied by \(1/(1-\rho)\), and that factor is a wall. Going from 50 percent to 90 percent utilization quintuples delay, and 90 to 99 multiplies it by another ten. Real switch traffic is burstier than Poisson, which makes the wall worse, not better. M/M/1 is the optimistic case, and it already forbids running links hot if latency matters. The concrete table below was computed for this page for a 10 Gb/s link with 1500-byte packets (service rate \(\mu = 10^{10}/(1500\times 8) = 833{,}333\) packets/s, service time 1.2 μs).

Bufferbloat

Queueing theory assumed buffers drain. The deployed Internet broke the assumption. Memory got cheap, vendors sized buffers for worst-case burst absorption, and loss-based TCP does not stop sending until the buffer overflows, so oversized buffers become standing queues, full pipes and full buffers, with every packet paying the whole queue's delay. Gettys and Nichols named the pathology bufferbloat in 2011. The arithmetic is one division, delay = buffer size / drain rate. Computed for this page, 1 MB of buffer at a 10 Mb/s home uplink is 0.8 seconds of queue, 128 kB at classic 1.5 Mb/s DSL is 0.683 seconds, a 12 MB shared switch buffer at 100 Gb/s is 0.96 ms, and a properly BDP-sized buffer for a 100 Gb/s, 10 μs datacenter path is 10 μs. The first two numbers explain a decade of "the internet is slow while uploading". One saturating TCP flow inflates every other flow's RTT, including 20 ms videoconference and DNS packets, to nearly a second. Note what the third number says about datacenters. Even a large shared buffer at 100 Gb/s is only a millisecond, so datacenter bloat is measured against 10 μs expectations, a hundredfold degradation from a fraction of the buffer.

AQM, from RED to CoDel

Active queue management drops or marks packets before the buffer is full, restoring the early feedback that loss-based control needs. RED (Floyd and Jacobson, 1993) maintains an EWMA of queue length and, between thresholds \(min_{th}\) and \(max_{th}\), drops arrivals with probability rising linearly from 0 to \(max_p\). Above \(max_{th}\) it drops everything. RED deployed poorly for a specific reason. Its thresholds are in queue-length units, but the harm is delay, and the same 100-packet queue is 1.2 ms at 1 Gb/s and 120 ms at 10 Mb/s. Tuning had to track link rate and traffic mix, operators got it wrong or turned it off, and twenty years of AQM deployment stalled. CoDel (Nichols and Jacobson, 2012) reframed the problem in the only unit that matters, sojourn time, the measured delay each packet actually experienced in the queue, timestamped on entry and checked on exit. If sojourn time stays above a target (5 ms) for a full interval (100 ms), CoDel drops one packet at the head of the queue and schedules the next drop at \(t + \text{interval}/\sqrt{n}\) for the \(n\)-th consecutive drop, the square-root control law chosen so the drop rate ramps until the sender's \(1/\sqrt{p}\) response brings the queue down. The moment sojourn time falls below target, dropping stops. The design distinguishes good queues (bursts draining within an RTT, which are the network doing its job) from bad queues (standing delay), needs no per-link tuning, and head-drop delivers the congestion signal a full queue-drain time earlier than tail-drop. FQ-CoDel, its flow-queuing extension, hashes flows into separate queues so a bulk upload cannot bloat a videoconference at all. It is the default qdisc in most Linux distributions and the heart of the router firmware that fixed home bufferbloat.

Datacenter fabrics, fat-trees, and ECMP

Why the datacenter abandoned the tree

The traditional enterprise network was a tree, hosts into edge switches, edges into a pair of aggregation boxes, aggregation into a core router, with each level oversubscribed and the top built from the largest, most expensive chassis available. Trees fail at datacenter scale for two independent reasons. The first is capacity. The root carries all cross-tree traffic, so bisection bandwidth is whatever one chassis can switch, and by the mid-2000s a warehouse of commodity servers could generate more traffic than any router that existed. The second is economics. The tree concentrates capacity into low-volume, high-margin big iron instead of high-volume commodity parts. Al-Fares, Loukissas, and Vahdat (2008) showed that a fat-tree built from identical commodity \(k\)-port switches provides full bisection bandwidth at a fraction of the cost, resurrecting Clos's 1953 telephone switching construction, in which any large non-blocking switch can be composed from small crossbars in three stages, trading switch size for path multiplicity.

The \(k\)-ary fat-tree, counted

The construction from \(k\)-port switches uses \(k\) pods, each containing \(k/2\) edge switches and \(k/2\) aggregation switches. Each edge switch spends \(k/2\) ports on hosts and \(k/2\) ports on the pod's aggregation switches. Each aggregation switch spends \(k/2\) ports downward on edges and \(k/2\) upward on core switches. The core layer has \((k/2)^2\) switches, each with one port to every one of the \(k\) pods. The counts follow directly,

Every host-to-host path crosses at most five switches (edge, agg, core, agg, edge), every layer has as much aggregate uplink as downlink capacity, and between any pair of hosts in different pods there are \((k/2)^2\) equal-cost core paths. The worked table is below.

            core        (k/2)² switches, 1 port per pod each
         ┌──┬──┬──┬──┐
         c1 c2 c3 c4          k=4 example: 4 core
        ╱│╲ ╱│╲ ╱│╲ ╱│╲
   ┌────┴────┐   ┌────┴────┐
   │ agg agg │   │ agg agg │   ... k pods, k/2 agg each
   │  │╲ ╱│  │   │  │╲ ╱│  │
   │ edge edge│  │ edge edge│  ... k/2 edge each
   │  ││  ││ │   │  ││  ││ │
   │  hh  hh │   │  hh  hh │   k/2 hosts per edge switch
   └─ pod 1 ─┘   └─ pod 2 ─┘

ECMP and the collision problem

A fat-tree's \((k/2)^2\) equal-cost paths are useless without a mechanism to spread traffic across them. The deployed mechanism, equal-cost multi-path (ECMP), hashes each packet's five-tuple (addresses, ports, protocol) and uses the hash to pick an uplink, so all packets of one flow take one path (preserving intra-flow ordering, which TCP punishes reordering for) while different flows scatter. The weakness is statistical. ECMP balances flow counts, not bytes, and hashing a few large flows onto a few paths is a birthday problem. A simulation computed for this page hashes \(n\) equal-rate flows onto 8 paths and reports the most-loaded path relative to the mean. With 8 flows on 8 paths, the expected maximum is 2.60x the mean (p95 4.0x), so some link typically carries almost triple its share while others sit idle. With 16 flows the figure is 2.11x, with 32 flows 1.76x, with 64 flows 1.52x, and only at 256 flows does the imbalance settle to 1.26x. Wider fabrics make it worse before better, since 32 flows on 32 paths gives 3.53x (p95 5.0x), and even 1,024 flows on 32 paths still shows 1.38x. The practical readings follow. ECMP works well for many small flows and badly for the elephant flows that dominate storage and training traffic. VL2 (Greenberg et al., 2009) built its fabric around Valiant load balancing (randomize each flow through an intermediate switch) for exactly this reason, and modern fabrics add flowlet switching (re-hash on gaps in a flow's packet train), packet spraying with reorder-tolerant transports, or centralized elephant scheduling (Hedera) to recover the stranded capacity. NCCL's own rings, later on this page, sidestep ECMP entirely when they can by using a rail-optimized topology, one NIC per GPU per fabric plane.

The tail at scale

Datacenter latency is judged at the tail because fanout multiplies rare events. If a request touches \(F\) servers in parallel and waits for all, and each server is independently slow with probability \(q\), the request is slow with probability \(1 - (1-q)^F\). Computed for this page, with \(q = 0.01\) (each server slow one time in a hundred), fanout 1 gives 1 percent slow requests, fanout 10 gives 9.6 percent, fanout 100 gives 63.4 percent, and fanout 2000, a realistic web-search leaf count, gives essentially 100 percent. Even at \(q = 10^{-3}\), fanout 2000 yields 86.5 percent. This is why Dean and Barroso's "The Tail at Scale" (2013) is the most operationally consequential paper in the area. At fanout, the p99 of the parts becomes the median of the whole, and every technique in the paper (hedged requests that duplicate a straggling RPC after the p95 mark, tied requests, micro-partitioning) exists to break the multiplication. Transport work like DCTCP feeds the same goal, since the queueing delay eliminated at the switch is bought back directly as tail budget.

DCTCP and ECN, congestion control rebuilt for the datacenter

Inside a datacenter, Reno-family control fails structurally. Commodity switch buffers are shallow and shared. RTTs are tens of microseconds, so the feedback loop is fast, but a retransmission timeout (minimum 1 to 5 ms in most stacks at the time DCTCP was designed, against a 100 μs RTT) is an eternity. The hardest workload is incast, in which a partition-aggregate request fans out to dozens of workers whose responses arrive at one switch port in the same window, overflowing the buffer in microseconds. Losing the last packet of a short response means a timeout that costs 10 to 100x the entire flow's lifetime. The deeper problem is that loss arrives too late. By the time the buffer overflows, the queue, and everyone's latency, is already maximal.

DCTCP (Alizadeh et al., 2010, developed at Microsoft Research with university collaborators) replaces the binary loss signal with a proportional one built from ECN. Switches mark rather than drop. Any packet arriving to a queue deeper than a threshold \(K\) gets its ECN bit set, an instantaneous, single-packet-granularity signal. The receiver echoes marks back precisely (one ACK state machine change from standard ECN), and the sender maintains an EWMA of the marked fraction \(F\) of each window.

The window cut is proportional to the congestion's extent. \(\alpha \to 1\) (every packet marked, serious congestion) recovers Reno's halving, while \(\alpha = 0.1\) (mild congestion) cuts only 5 percent. The sender therefore trims gently and continuously instead of sawing violently, which lets the switch run its queue at the marking threshold \(K\) rather than at buffer capacity. The paper's guideline puts \(K > C\times\text{RTT}/7\) in packets, roughly 20 packets at 1 Gb/s and 65 at 10 Gb/s, a few percent of a shared buffer, so the fabric holds microsecond queues while still absorbing bursts. The EWMA's time constant is worth working through numerically (computed for this page). Starting from \(\alpha = 0\) with every packet suddenly marked, \(\alpha\) reaches only \(1-(1-1/16)^{16} = 0.644\) after 16 round trips and 0.994 after 80, so DCTCP needs tens of RTTs to learn about severe congestion, one reason it coexists poorly with conventional TCP (which must be kept in separate queues) and one cost of the smooth response. At datacenter RTTs, 80 round trips is under 10 ms, so the slowness is acceptable where it is deployed. DCTCP variants are the standard fabric transport at Microsoft, and the marking-threshold idea propagated into Linux (tcp_dctcp.c), into RDMA congestion control as DCQCN, and into the HPCC and Swift line of work that replaced marks with measured delay or in-band telemetry.

RDMA, RoCE, and why lossless fabrics are hard

What RDMA changes

The loopback measurements earlier made the case numerically. A socket round trip costs 15 to 25 μs on this host, nearly all of it software. RDMA (remote direct memory access) removes the software. The NIC implements the transport in hardware, applications post work requests to queue pairs mapped into userspace, and the NIC reads and writes application memory directly on both ends, no syscall, no interrupt, no copy on the data path. One-sided operations (RDMA read, RDMA write) complete without the remote CPU's involvement at all. Round trips drop to 1 to 2 μs and a single NIC saturates 400 Gb/s with a few percent of one core, which is why RDMA in its InfiniBand form has owned HPC since the 1990s and why the hyperscalers brought it onto Ethernet as RoCE (RDMA over Converged Ethernet, whose v2 encapsulates InfiniBand transport headers over UDP/IP so it can route). GPU-direct RDMA extends the path end to end. The NIC DMAs into GPU HBM directly, and NCCL's cross-host path is exactly this.

The lossless bargain and PFC

The historical InfiniBand-derived RoCE NICs implemented go-back-N retransmission in hardware, so a single lost packet forces retransmission of the entire window, the 29,000x penalty from Problem 1's arithmetic. The design assumption was a network that never drops, and Ethernet was made to imitate one with Priority Flow Control (PFC, 802.1Qbb). When a switch ingress queue crosses a threshold, it sends a PAUSE frame upstream for that priority class, and the upstream port stops transmitting that class entirely until resumed. This converts loss into backpressure, and it works, but it purchases losslessness with exactly the coupling the end-to-end argument warns about. Correctness-critical control state now lives in every switch, and pausing is a hop-scoped hammer with no notion of flows.

The failure modes are well documented from production, most thoroughly in Guo et al.'s account of RDMA deployment at Microsoft scale (2016). One is head-of-line blocking. PFC pauses a whole class on a port, so one congested flow stalls every flow sharing the class, and the pause propagates upstream hop by hop (congestion spreading), freezing parts of the fabric far from the hotspot. Another is the pause storm. A malfunctioning NIC that emits continuous PFC frames can freeze its top-of-rack switch, whose queues fill and emit their own pauses, cascading a single sick host into a fabric-wide outage. Guo et al. describe exactly this propagation and the watchdogs (pause-storm detection, PFC timeouts that re-enable dropping) deployed against it. The last is deadlock. PFC dependencies plus a cyclic buffer dependency, which transient routing loops or specific multi-path wirings can create, can produce a cycle of ports each waiting for the next to unpause, a permanent standstill requiring intervention. The engineering response is layered. DCQCN (Zhu et al., 2015) adds ECN-based per-flow rate control so PFC becomes a rarely touched last resort rather than the primary control, careful buffer headroom arithmetic bounds in-flight data per hop, and the current generation moves away from the lossless requirement altogether, with selective-repeat retransmission in NIC hardware (ConnectX-6 and later), Amazon's SRD transport for EFA spraying packets across paths with out-of-order delivery, and the Ultra Ethernet Consortium standardizing a datacenter transport that tolerates loss. The arc is a clean end-to-end-argument case study. Reliability moved into the network for performance, the predicted fragility arrived at scale, and the fix is moving reliability back toward the endpoints with better hardware.

Collective communication for distributed training

The allreduce problem and the ring construction

Data-parallel training gives every one of \(N\) workers a full model replica. Each step, every worker computes gradients on its shard of the batch, and all workers must end the step holding the same summed gradient vector of \(M\) bytes. That operation, elementwise sum then broadcast of the result, is allreduce, and at 70B-parameter scale \(M\) is 140 GB of bf16 gradients per step, so its cost structure is worth deriving exactly. The naive schedule, gather everything to one root and broadcast the sum, moves \((N-1)M\) bytes into one node and \((N-1)M\) out, so the root's links are the bottleneck and cost grows linearly with \(N\). A binary tree improves the latency to \(O(\log N)\) rounds, reduce up then broadcast down, but each round still moves the full \(M\) through every participating link, giving time roughly \(2\log_2 N \cdot M/B\) on link bandwidth \(B\), so the bandwidth term grows with \(\log N\).

The ring algorithm (analyzed as bandwidth-optimal by Patarasuk and Yuan, 2009, with roots in the MPI collective work of Thakur, Rabenseifner, and Gropp) removes the \(N\)-dependence from the bandwidth term almost entirely. Arrange the \(N\) workers in a ring and split the buffer into \(N\) equal chunks of \(M/N\) bytes. Phase one, reduce-scatter, runs \(N-1\) steps. In each step, every worker sends one chunk to its right neighbor and receives one from its left, adding what it receives into its local copy. The chunk indices are staggered so that after \(N-1\) steps each worker holds the complete sum of exactly one chunk. Phase two, allgather, runs \(N-1\) more steps in which the completed chunks circulate, each worker forwarding the newest chunk it received, until everyone holds all \(N\) summed chunks. Each worker sends exactly \(2(N-1)\) chunks of \(M/N\) bytes, so with per-link bandwidth \(B\) and per-step latency \(\alpha\),

Every link in the ring is busy in every step (the schedule is perfectly load-balanced, which is why no algorithm can beat the \(2(N-1)/N \cdot M/B\) bandwidth term for a full allreduce on unidirectional links), and the bandwidth term saturates at \(2M/B\) regardless of \(N\), so a thousand GPUs pay essentially the same bandwidth cost as eight. What grows with \(N\) is the latency term, \(2(N-1)\alpha\), which is why rings are the wrong choice for small messages on large rings, and why NCCL switches to tree algorithms (latency \(O(\log N)\), bandwidth \(2\log_2 N \cdot M/B\)) below a message-size crossover, and to hierarchical schedules (rings inside a node over NVLink, trees or rings across nodes over the NIC) at cluster scale.

NCCL measured on two H100s over NVLink

The numbers below are real NCCL 2.26.2 measurements from this machine, two H100 80GB GPUs connected by an 18-lane NVLink (NV18), PyTorch 2.7.0. The stated caveat applies. Only two GPUs are present, so these measurements cover \(N = 2\) only, and all larger-\(N\) figures on this page are analytical model output, labeled as such. Raw peer-to-peer device-to-device copy bandwidth measures 388.28 GB/s. The allreduce measurements follow.

The table is a textbook \(\alpha\)-\(\beta\) curve. At 1 MiB the operation takes 25.2 μs and achieves 41.68 GB/s, an eighth of what the wire can do, because fixed costs (kernel launch, protocol steps) dominate. A 4-byte allreduce takes 23.62 μs, almost exactly the 1 MiB time, confirming that everything below a megabyte is pure latency, and a bare barrier costs 35.87 μs. By 1 GiB the collective sustains 330.15 GB/s, 85 percent of the measured 388.28 GB/s point-to-point ceiling. Note also that algorithm bandwidth equals bus bandwidth in every row. NCCL's bus-bandwidth metric multiplies algbw by \(2(N-1)/N\), which at \(N = 2\) is exactly 1, a small live confirmation of the ring factor just derived. The neighboring collectives measured on the same pair behave consistently. Allgather reaches 244.86 GB/s and reduce-scatter 239.66 GB/s at 256 MiB (each is one phase of the allreduce, \((N-1)/N\) of the traffic), and alltoall moves 266.79 GB/s at 256 MiB per rank. The practical moral for training code is that gradient bucketing exists because of the left half of this table. A 7B-parameter model whose gradients were allreduced tensor-by-tensor in 100 kB pieces would pay the 25 μs floor thousands of times per step. Fusing into 25 to 100 MB buckets, as PyTorch DDP does by default, moves every transfer into the flat right half of the curve.

Scaling the ring to a training cluster, modeled

Apply the derived cost model at cluster scale (analytical model output, per the \(N=2\) caveat, with \(\alpha = 5\) μs per step and \(B = 50\) GB/s for a 400 Gb/s NIC path). Allreducing \(M = 140\) GB of 70B-model bf16 gradients takes \(T = 2(N-1)\alpha + \frac{2(N-1)}{N} M/B\), which evaluates to 4.90 s at \(N=8\), 5.51 s at \(N=64\), 5.58 s at \(N=256\), and 5.64 s at \(N=4096\). The bandwidth term's \(2(N-1)/N\) factor climbs from 1.75 toward its asymptote 2, and five thousand GPUs pay only 15 percent more than eight. The same model with tree schedules gives 16.8 s at \(N=8\) and 67.2 s at \(N=4096\), the \(\log N\) growth visible. The latency-bandwidth crossover matters at the other end of the size axis. At \(N=256\), the model's fixed cost is \(2 \times 255 \times 5\,\mu s = 2.55\) ms, which is 100 percent of the total for any message under 100 kB, 86.5 percent at 10 MB, and only 6 percent at 1 GB. Small collectives are latency and large ones are bandwidth, and the entire craft of overlap scheduling lives in that split.

QUIC and HTTP/3

QUIC is what a transport looks like when it is designed after the lessons of thirty years of TCP deployment, with the freedom to ignore middleboxes. Its carriers built it (Langley et al., 2017, describes the Google deployment, and the IETF standardized it as RFC 9000 in 2021 with Iyengar and Thomson as editors) on three architectural decisions. First, run over UDP in userspace. TCP's wire format is ossified because millions of middleboxes parse and "normalize" it, so any TCP extension (SACK took a decade, and Fast Open never fully deployed) rolls out at the pace of the slowest firewall. UDP passes through, and QUIC encrypts everything above the UDP header, including its own transport headers, so middleboxes cannot ossify what they cannot read. An update to loss recovery ships in a browser release instead of a kernel upgrade. Second, streams as a first-class transport primitive. A QUIC connection multiplexes many independent ordered byte streams, and a lost packet blocks only the streams whose data it carried. HTTP/2 over TCP had multiplexed streams above a single ordered byte stream, so one lost TCP segment head-of-line-blocked every stream in the connection. QUIC's per-stream ordering removes exactly that, which is most of why HTTP/3 outperforms HTTP/2 on lossy mobile paths. Third, fold the security handshake into the transport handshake. QUIC integrates TLS 1.3, so a new connection completes crypto and transport setup in one round trip, and a resumed connection can send 0-RTT application data in the first flight (with the replay caveat that 0-RTT data must be idempotent, because a network attacker can replay the flight). TCP with TLS 1.3 needs two round trips fresh. On the 70 ms path from Problem 1, QUIC saves 70 to 140 ms of handshake before the first byte of response, which is larger than many entire page budgets.

The engineering details repay attention. Connections are named by connection IDs rather than the five-tuple, so a phone migrating from WiFi to cellular keeps its connections alive, and a load balancer can route by connection ID through NAT rebinding. Loss detection is cleaner than TCP's. Every packet, including retransmissions, gets a fresh monotonic packet number, so the retransmission ambiguity behind Karn's algorithm never arises, and ACK frames carry receive timestamps and up to 32 ranges, giving the sender a far sharper view than SACK's three blocks. Congestion control is pluggable per-connection in userspace (Reno-family, CUBIC, and BBR implementations ship in production stacks). The price is CPU. Kernel TCP enjoys segmentation offload, and encrypting every packet's headers costs more per byte than TLS-over-TCP's bulk path, an actively optimized gap (UDP GSO, crypto batching). As of the mid-2020s, on the order of a third of web traffic is HTTP/3, and the transport research it unblocked (MASQUE proxying, WebTransport, unreliable datagrams in RFC 9221) is a direct dividend of de-ossification.

Software-defined networking, briefly

Classical routers bundle three things per box, forwarding hardware, the distributed protocols that compute routes (OSPF, BGP), and the vendor's configuration surface. SDN's claim (McKeown et al.'s OpenFlow paper, 2008, gave it a concrete interface) is that the second and third belong in a logically centralized controller with a global view, leaving switches as fast, dumb match-action tables. The datacenter is where the claim proved out, because a single operator owns the whole fabric and the topology is regular. Google's B4 WAN ran centralized traffic engineering at near-100 percent link utilization, versus the 30 to 40 percent that decentralized protocols need as headroom, precisely because a controller that sees all demands can pack paths deliberately, and Jupiter (Singh et al., 2015) manages a 1.3 Pb/s fabric the same way. The counterweight is the end-to-end argument's cousin. A centralized controller is a single conceptual point of failure, so production designs are logically centralized but physically replicated, and the failure domain analysis (what happens when the controller partitions from the switches it programs) is the hard part of every deployment. The programmable-dataplane line (P4, Barefoot/Intel Tofino) pushed the idea one level down, letting operators define the match-action pipeline itself. Its datacenter legacy is in-band network telemetry and in-network aggregation experiments (SwitchML, ATP) that sum gradients in the switch, a live research thread for the collective workloads above.

Worked problems

Implementation

Four programs, each small enough to read in one sitting and each demonstrating one load-bearing idea from above. The first measures what this page's loopback numbers measure, RTT percentiles and the syscall-size throughput curve, so the measured tables can be reproduced on any host.

# latency_bench.py: TCP loopback RTT percentiles and write-size sweep.
# Reproduces the methodology behind this page's measured host numbers.
import socket, threading, time, statistics

def echo_server(sock):
    conn, _ = sock.accept()
    conn.setsockopt(socket.IPPROTO_TCP, socket.TCP_NODELAY, 1)
    while True:
        data = conn.recv(65536)
        if not data:
            break
        conn.sendall(data)

def rtt_bench(port=9101, iters=50000, payload=64):
    srv = socket.socket()
    srv.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
    srv.bind(("127.0.0.1", port)); srv.listen(1)
    threading.Thread(target=echo_server, args=(srv,), daemon=True).start()

    cli = socket.create_connection(("127.0.0.1", port))
    cli.setsockopt(socket.IPPROTO_TCP, socket.TCP_NODELAY, 1)  # no Nagle:
    # Nagle would hold small writes waiting for ACKs and serialize the pingpong.
    msg = b"x" * payload
    samples = []
    for _ in range(iters):
        t0 = time.perf_counter_ns()
        cli.sendall(msg)
        got = 0
        while got < payload:            # loopback may still split reads
            got += len(cli.recv(payload - got))
        samples.append(time.perf_counter_ns() - t0)
    samples.sort()
    q = lambda p: samples[int(p * len(samples))]
    print(f"RTT ns: p50={q(.5)} p99={q(.99)} p99.9={q(.999)} "
          f"mean={statistics.mean(samples):.0f}")
    # On the host this page benchmarks (shared, virtualized), 64B pingpong
    # measured p50=24405ns, p99=32153ns, p99.9=37138ns over 200k iters.

def throughput_sweep(port=9102, total=1 << 28):
    srv = socket.socket()
    srv.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
    srv.bind(("127.0.0.1", port)); srv.listen(1)
    def sink(s):
        c, _ = s.accept()
        while c.recv(1 << 20):
            pass
    threading.Thread(target=sink, args=(srv,), daemon=True).start()
    for size in (64, 1024, 65536, 1 << 20):
        cli = socket.create_connection(("127.0.0.1", port))
        buf = b"x" * size
        n = total // size
        t0 = time.perf_counter()
        for _ in range(n):
            cli.sendall(buf)            # one syscall per iteration:
        dt = time.perf_counter() - t0   # the syscall, not the copy, dominates
        print(f"{size:>8}B writes: {total*8/dt/1e9:6.2f} Gb/s "
              f"({dt/n*1e9:8.0f} ns/write)")
        cli.close()
    # Host measured: 64B -> 1.07 Gb/s (477 ns/write); 1 MiB -> 48.73 Gb/s.

if __name__ == "__main__":
    rtt_bench()
    throughput_sweep()

The second is a miniature reliable transport over an unreliable channel, with sliding window, cumulative ACKs, Jacobson-Karels RTO, and Karn's rule, in about eighty lines. Running it with different loss rates reproduces the Go-Back-N-versus-selective-repeat gap and the timer dynamics from the theory sections. The channel is a lossy in-process queue so the whole system is deterministic under a seed.

# minitransport.py: sliding-window reliable delivery over a lossy channel,
# with Jacobson-Karels RTO estimation and Karn's rule.
import random, heapq

class LossyChannel:
    """Delivers (time, packet) events; drops with prob p, delays ~ rtt/2."""
    def __init__(self, loss, one_way, jitter, seed=0):
        self.loss, self.one_way, self.jitter = loss, one_way, jitter
        self.rng = random.Random(seed)
        self.events = []                      # min-heap of (deliver_time, pkt)
    def send(self, now, pkt):
        if self.rng.random() >= self.loss:    # survives with prob 1-p
            dt = self.one_way + self.rng.uniform(0, self.jitter)
            heapq.heappush(self.events, (now + dt, pkt))
    def poll(self, now):
        out = []
        while self.events and self.events[0][0] <= now:
            out.append(heapq.heappop(self.events)[1])
        return out

class Sender:
    def __init__(self, n_packets, window, ch_data, ch_ack):
        self.n, self.W = n_packets, window
        self.data, self.acks = ch_data, ch_ack
        self.base = 0                  # oldest unacked seq
        self.next = 0                  # next seq to send
        self.sent_at = {}              # seq -> (time, retransmitted?)
        self.srtt, self.rttvar, self.rto = None, None, 1.0   # RFC 6298 init
        self.timer = None
        self.retx = 0
    def rtt_sample(self, m):
        if self.srtt is None:
            self.srtt, self.rttvar = m, m / 2
        else:
            self.rttvar = 0.75 * self.rttvar + 0.25 * abs(self.srtt - m)
            self.srtt = 0.875 * self.srtt + 0.125 * m
        self.rto = max(0.2, self.srtt + 4 * self.rttvar)
    def tick(self, now):
        for ack in self.acks.poll(now):        # cumulative: ack = next expected
            if ack > self.base:
                for s in range(self.base, ack):
                    t, retx = self.sent_at.pop(s, (None, True))
                    if t is not None and not retx:   # Karn: never sample
                        self.rtt_sample(now - t)     # a retransmitted seq
                self.base = ack
                self.timer = now + self.rto if self.base < self.next else None
        if self.timer is not None and now >= self.timer:  # timeout:
            self.rto = min(60.0, self.rto * 2)             # exponential backoff
            seq = self.base                                # go-back: resend base
            self.data.send(now, seq)
            self.sent_at[seq] = (now, True)
            self.timer = now + self.rto
            self.retx += 1
        while self.next < min(self.base + self.W, self.n):  # fill the window
            self.data.send(now, self.next)
            self.sent_at[self.next] = (now, False)
            if self.timer is None:
                self.timer = now + self.rto
            self.next += 1
    def done(self):
        return self.base >= self.n

def run(loss=0.02, window=64, n=5000, rtt=0.10):
    data = LossyChannel(loss, rtt / 2, rtt * 0.1, seed=1)
    acks = LossyChannel(loss, rtt / 2, rtt * 0.1, seed=2)
    s = Sender(n, window, data, acks)
    expected, now, dt = 0, 0.0, 0.001
    while not s.done():
        s.tick(now)
        for seq in data.poll(now):             # receiver: cumulative ACK
            if seq == expected:
                expected += 1
            acks.send(now, expected)           # ack = next expected seq
        now += dt
    goodput = n * 1460 * 8 / now / 1e6
    print(f"loss={loss:.3f} W={window}: {now:.1f}s, {goodput:.2f} Mb/s, "
          f"{s.retx} timeouts, final rto={s.rto:.3f}s srtt={s.srtt:.3f}s")

if __name__ == "__main__":
    for loss in (0.0, 0.01, 0.05):
        run(loss=loss)
    # Window 64 over 100ms RTT caps goodput at 64*1460*8/0.1 = 7.5 Mb/s;
    # losses convert to timeouts and the Mathis-style collapse is visible.

The third is the C server pattern underneath every high-connection-count system, a single-threaded epoll event loop, edge-cases handled the way production loops (libuv, nginx) handle them, nonblocking sockets and EAGAIN-driven reads. Compare its structure with the thread-per-connection model. The loop holds one small state record per connection instead of a stack, which is what makes a million concurrent connections a memory problem instead of a scheduler problem.

/* epoll_echo.c: single-threaded event-driven echo server.
 * build: gcc -O2 -o epoll_echo epoll_echo.c    test: nc 127.0.0.1 9200 */
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <unistd.h>
#include <errno.h>
#include <fcntl.h>
#include <netinet/in.h>
#include <netinet/tcp.h>
#include <sys/epoll.h>
#include <sys/socket.h>

#define MAX_EVENTS 1024

static int set_nonblocking(int fd) {
    int fl = fcntl(fd, F_GETFL, 0);
    return fcntl(fd, F_SETFL, fl | O_NONBLOCK);
}

int main(void) {
    int lfd = socket(AF_INET, SOCK_STREAM, 0);
    int one = 1;
    setsockopt(lfd, SOL_SOCKET, SO_REUSEADDR, &one, sizeof one);

    struct sockaddr_in addr = {0};
    addr.sin_family = AF_INET;
    addr.sin_addr.s_addr = htonl(INADDR_LOOPBACK);
    addr.sin_port = htons(9200);
    if (bind(lfd, (struct sockaddr *)&addr, sizeof addr) < 0) {
        perror("bind"); return 1;
    }
    listen(lfd, SOMAXCONN);
    set_nonblocking(lfd);

    int ep = epoll_create1(0);
    struct epoll_event ev = { .events = EPOLLIN, .data.fd = lfd };
    epoll_ctl(ep, EPOLL_CTL_ADD, lfd, &ev);

    struct epoll_event events[MAX_EVENTS];
    char buf[65536];

    for (;;) {
        /* one blocking point for the whole server: the kernel parks us
         * until any registered fd is ready. */
        int n = epoll_wait(ep, events, MAX_EVENTS, -1);
        if (n < 0) { if (errno == EINTR) continue; perror("epoll_wait"); break; }

        for (int i = 0; i < n; i++) {
            int fd = events[i].data.fd;

            if (fd == lfd) {                       /* accept until EAGAIN: */
                for (;;) {                          /* level-triggered would */
                    int cfd = accept(lfd, NULL, NULL); /* re-arm, but drain  */
                    if (cfd < 0) break;             /* anyway for burstiness */
                    set_nonblocking(cfd);
                    setsockopt(cfd, IPPROTO_TCP, TCP_NODELAY,
                               &one, sizeof one);
                    struct epoll_event cev = { .events = EPOLLIN,
                                               .data.fd = cfd };
                    epoll_ctl(ep, EPOLL_CTL_ADD, cfd, &cev);
                }
                continue;
            }
            /* data or hangup on a connection */
            for (;;) {
                ssize_t r = read(fd, buf, sizeof buf);
                if (r > 0) {
                    /* real servers must handle short writes by buffering
                     * and waiting for EPOLLOUT; echo payloads are small
                     * enough that a short write is rare. */
                    ssize_t off = 0;
                    while (off < r) {
                        ssize_t w = write(fd, buf + off, r - off);
                        if (w < 0) break;
                        off += w;
                    }
                } else if (r == 0 ||
                           (r < 0 && errno != EAGAIN && errno != EWOULDBLOCK)) {
                    close(fd);                      /* peer closed or error */
                    break;
                } else {
                    break;                          /* EAGAIN: drained */
                }
            }
        }
    }
    return 0;
}

The fourth reproduces the NCCL allreduce measurements from this page, using torch.distributed with the NCCL backend, one process per GPU, CUDA-event timing around a bus-synchronized loop. Run on this machine's two H100s, this methodology produced the 41.68 to 330.15 GB/s curve in the collectives table.

# allreduce_bench.py: measure NCCL allreduce bandwidth across local GPUs.
# run: torchrun --nproc_per_node=2 allreduce_bench.py
import os, torch
import torch.distributed as dist

def main():
    dist.init_process_group("nccl")             # NCCL: GPU rings over NVLink
    rank = dist.get_rank()
    world = dist.get_world_size()
    torch.cuda.set_device(rank)
    dev = torch.device("cuda", rank)

    for mib in (1, 4, 16, 64, 256, 1024):
        numel = mib * (1 << 20) // 4            # fp32 elements  [numel]
        x = torch.ones(numel, device=dev)       # [numel] on this rank's GPU

        for _ in range(5):                      # warmup: cudagraph/algo setup
            dist.all_reduce(x)
        torch.cuda.synchronize()

        iters = 20
        start = torch.cuda.Event(enable_timing=True)
        stop = torch.cuda.Event(enable_timing=True)
        start.record()
        for _ in range(iters):
            dist.all_reduce(x)                  # sum across all ranks, inplace
        stop.record()
        torch.cuda.synchronize()
        ms = start.elapsed_time(stop) / iters

        bytes_ = numel * 4
        algbw = bytes_ / (ms / 1e3) / 1e9       # data size / time
        busbw = algbw * 2 * (world - 1) / world # ring factor 2(N-1)/N:
        if rank == 0:                           # at N=2 busbw == algbw
            print(f"{mib:5d} MiB  {ms:8.4f} ms  "
                  f"algbw {algbw:7.2f} GB/s  busbw {busbw:7.2f} GB/s")
    # Measured on this host (2x H100 80GB, NV18, NCCL 2.26.2):
    #     1 MiB  0.0252 ms   41.68 GB/s     (latency floor: 4B takes 23.62us)
    #  1024 MiB  3.2522 ms  330.15 GB/s     (85% of 388.28 GB/s p2p ceiling)
    dist.destroy_process_group()

if __name__ == "__main__":
    main()

How it is done in practice

The distance between this page's models and a production fabric is mostly in the places where clean abstractions meet shared hardware. A few load-bearing realities. Congestion control in a real datacenter is an ecosystem, not an algorithm. The storage fleet may run DCTCP variants in dedicated ECN-marked queues, the RDMA fabric runs DCQCN or Swift-style delay-based control below PFC, WAN-facing traffic runs BBR, and the switch QoS configuration that keeps them from starving each other, queue weights, ECN thresholds per queue, PFC headroom per priority, is some of the most carefully reviewed configuration in the company, because a wrong threshold turns into a fleet-wide latency regression. The training-cluster fabric is increasingly its own network, physically separate from the frontend datacenter network. Rail-optimized topologies give each GPU its own NIC and its own fabric plane, so the eight NCCL rings of an 8-GPU node never share a link and ECMP hashing is bypassed by construction. NVIDIA's reference designs and the large public clusters (Meta's 24k-GPU RoCE and InfiniBand clusters described in their Llama 3 infrastructure work) are built this way.

Overlap is the discipline that keeps the network off the critical path. The Problem 5 arithmetic, communication-to-compute ratio 0.34 at the healthy operating point, only helps if the allreduce actually runs concurrently with the backward pass, so every serious training framework buckets gradients (25 MB default in PyTorch DDP) and launches collectives as buckets complete, reverse-topological order, hiding the 5.6 s behind the 16.4 s. Sharded data parallelism (ZeRO, FSDP) changes the collective mix, reduce-scatter plus allgather instead of allreduce, same total bytes but half per phase, and pipeline and tensor parallelism trade collective volume against latency sensitivity. Tensor-parallel allreduces sit on the critical path of every layer's forward pass, which is why they stay inside the NVLink domain where this machine measures 330 GB/s and 24 μs latency, while data-parallel traffic crosses the slower NIC fabric where bandwidth, not latency, is the binding constraint. The numbers compose. A practitioner sizing a cluster works this page's arithmetic, BDP for the transport, \(2(N-1)/N\) for the collectives, bisection for the fabric, \(1-(1-q)^F\) for the tail, before any hardware is ordered, and the measured tables here are the sanity anchors for the model's constants.

The current research frontier

Transport for AI fabrics is the loudest thread. The Ultra Ethernet Consortium (AMD, Arista, Broadcom, Cisco, Meta, Microsoft, and others) is standardizing a transport that drops the lossless requirement, embraces packet spraying and out-of-order delivery, and targets million-endpoint scale. Amazon's SRD (Shalev et al., 2020) demonstrated the approach in production EFA, and NVIDIA's Spectrum-X couples adaptive routing with direct data placement to the same end. In-network aggregation keeps advancing from research (SwitchML from KAUST and Microsoft Research collaborators, ATP from Wisconsin and CMU-adjacent groups) into product (NVIDIA SHARP performs reductions in InfiniBand switches, cutting the allreduce bandwidth term roughly in half by summing in the fabric), though it fights the end-to-end argument on every deployment question, from state in switches to failure semantics to multi-tenancy. Congestion control research has moved delay-first. Google's Swift showed hardware-timestamped delay targets working fleet-wide, HPCC (Alibaba) used in-band telemetry for near-instant convergence, and the BBR line continues through v3 in the IETF ccwg. Learned and formally analyzed control both remain active, from MIT's Remy and the PCC utility-driven line from UIUC and the Hebrew University to the ongoing effort to characterize BBR's fairness envelope analytically (CMU, EPFL, and others).

At the host boundary, the syscall-cost problem this page measured is being attacked from three directions. io_uring batches submission and completion in shared rings inside the kernel, kernel-bypass frameworks are maturing from research (DPDK, and academic lines like Snap at Google, Demikernel from Microsoft Research) into standard deployment, and smartNIC/DPU offload (NVIDIA BlueField, AWS Nitro, Intel IPU) is moving the hypervisor's entire network stack off the host CPU. QUIC research continues on the performance gap with kernel TCP (hardware offload of QUIC crypto, MASQUE-based proxying, multipath QUIC in RFC 9761-adjacent work), and the programmable-dataplane community (P4, Tofino's successors, and eBPF/XDP inside Linux) keeps eroding the line between endpoint and network, which is where the next round of the end-to-end argument will be litigated.

Open source to read

The canonical implementations of nearly everything above are readable. Each entry names the file to open first.

• torvalds/linux: the reference congestion-control implementations. Open net/ipv4/tcp_cubic.c, find cubictcp_update, and match it line by line against the \(W(t) = C(t-K)^3 + W_{max}\) derivation above (the fixed-point cube root is a pleasure). Then open net/ipv4/tcp_bbr.c, whose 100-line header comment is the best compact BBR specification anywhere.
• NVIDIA/nccl: the collectives that produced this page's measured numbers. Open src/device/all_reduce.h for the ring reduce-scatter and allgather phases as templated device code, and src/graph/search.cc to see how NCCL discovers rings and trees through NVLink and NIC topology at init time.
• cloudflare/quiche: a production QUIC in Rust with unusually clean separation. Open quiche/src/recovery/mod.rs to see loss detection and the pluggable congestion controllers (Reno, CUBIC, BBR) behind one trait. The RFC 9002 machinery reads like the RFC.
• microsoft/msquic: the QUIC that ships in Windows and SMB. Open src/core/loss_detection.c, then src/core/congestion_control.c to see how a kernel-grade implementation organizes the same state machines with allocation discipline quiche leaves to Rust.
• h2o/quicly with h2o/picotls: the minimal readable QUIC and TLS 1.3 pair. Open lib/quicly.c and trace one packet from quicly_receive through decryption to stream delivery. It is the shortest path to understanding how QUIC's layers actually nest.
• libuv/libuv: the event loop under Node.js. Open src/unix/linux.c and find the epoll_wait loop. Compare with the C server above to see what production adds, namely timer heaps, deferred close semantics, and the thundering-herd accept dance.
• facebookincubator/gloo: PyTorch's CPU collective backend, where the algorithms are legible without CUDA. Open gloo/allreduce_ring.cc and gloo/allreduce_halving_doubling.cc. The two files are the ring-versus-recursive-halving trade-off as code.
• openucx/ucx: the HPC communication layer over RDMA verbs. Open src/uct/ib/rc/base/rc_ep.c to see what programming an RDMA queue pair actually involves, and how much machinery "the NIC does the transport" hides.
• google/bbr: the BBR development tree, including v3 ahead of mainline. Open the README for deployment status, then net/ipv4/tcp_bbr.c in the v3 branch to diff v3's loss and ECN responses against v1's pure model.

Common misconceptions

"More buffer is always safer." Buffers exist to absorb bursts, not to store standing queues. A buffer that never drains adds its full depth to every packet's latency, and loss-based TCP will fill whatever exists. The arithmetic is one division. At 10 Mb/s, 1 MB of buffer is 0.8 seconds of delay. Sizing is bounded by the BDP the buffer serves, and past that, AQM, not memory, is the tool. The reflex "we saw drops, add buffer" is how a network acquires bufferbloat.

"Throughput is set by the link speed." A transport is limited by \(\min\) of link rate, receive window over RTT, and the congestion regime, and the last two dominate more often than not. A 64 KiB window over 70 ms delivers 7.49 Mb/s over any link, and Reno at \(p = 10^{-6}\), 50 ms RTT delivers 285 Mb/s (simulated on this page's model) no matter how much fiber is underneath. Fast links change the question from "how fast is the wire" to "who is filling the BDP".

"Packet loss means the network is broken." For loss-based congestion control, loss is the protocol working. It is the only feedback signal the design has, produced deliberately by probing until the bottleneck queue overflows. The pathological cases are the extremes, zero loss with rising RTT (bufferbloat, where the signal is being withheld) and random non-congestive loss (wireless corruption), which the \(1/\sqrt{p}\) law punishes as if it were congestion. Whether loss is information or damage depends entirely on which controller is listening.

"ECMP spreads traffic evenly across equal-cost paths." ECMP balances flow placements, not bytes. Hashing 8 equal flows onto 8 paths leaves the busiest path at 2.6x the mean load in expectation (computed above). A few elephant flows can collide on one uplink while others idle. Even load requires many small flows, flowlet or packet-level spraying, or explicit scheduling. Assuming ECMP equals balance is how fabrics end up mysteriously slow at 40 percent average utilization.

"Lossless fabrics remove congestion problems." PFC removes drops, not congestion. It converts overflow into backpressure that spreads hop by hop, blocks innocent flows sharing the paused class, and in the limit deadlocks or storms. The congestion still has to be controlled by something (DCQCN, delay-based control), and the lossless machinery adds its own failure modes on top. "Lossless" describes the packet's fate, not the tail latency's.

"Allreduce cost scales linearly with the number of GPUs." The ring's bandwidth term is \(2(N-1)/N \cdot M/B\), which saturates at \(2M/B\), so 4,096 GPUs pay 15 percent more than 8 in the model above, not 500x. What grows with \(N\) is the latency term and the exposure to stragglers and failures. The linear-scaling intuition comes from the naive gather-broadcast schedule, which no production collective uses.

"QUIC is faster because UDP is faster than TCP." UDP is not faster. It is emptier. QUIC's wins come from what it builds in the empty space, one-round-trip combined handshakes, per-stream ordering that removes cross-stream head-of-line blocking, richer ACKs, unambiguous packet numbering, and deployability of new control algorithms in userspace. Indeed QUIC pays more CPU per byte than kernel TCP with offloads. It wins on round trips and loss behavior, not on datapath cost.

"The p99 only matters for the last 1 percent of users." Under fanout, the tail is the median. At fanout 100 with 1-percent-slow servers, 63.4 percent of requests hit the tail (computed above). Any system that aggregates parallel RPCs inherits its dependencies' p99 as its own typical case, which is why tail budgets, hedging, and DCTCP-class queue control are mainstream engineering rather than perfectionism.

Self-check

References

1. Kurose, J. and Ross, K. Computer Networking: A Top-Down Approach, 8th ed., Pearson, 2021.
2. Peterson, L. and Davie, B. Computer Networks: A Systems Approach, 6th ed., Morgan Kaufmann, 2021. github.com/SystemsApproach/book
3. Saltzer, J., Reed, D., and Clark, D. "End-to-End Arguments in System Design," ACM Transactions on Computer Systems 2(4), 1984. doi:10.1145/357401.357402
4. Jacobson, V. "Congestion Avoidance and Control," SIGCOMM, 1988. doi:10.1145/52324.52356
5. Chiu, D.-M. and Jain, R. "Analysis of the Increase and Decrease Algorithms for Congestion Avoidance in Computer Networks," Computer Networks and ISDN Systems 17(1), 1989. doi:10.1016/0169-7552(89)90019-6
6. Mathis, M., Semke, J., Mahdavi, J., and Ott, T. "The Macroscopic Behavior of the TCP Congestion Avoidance Algorithm," ACM SIGCOMM Computer Communication Review 27(3), 1997. doi:10.1145/263932.264023
7. Padhye, J., Firoiu, V., Towsley, D., and Kurose, J. "Modeling TCP Throughput: A Simple Model and Its Empirical Validation," SIGCOMM, 1998. doi:10.1145/285237.285291
8. Ha, S., Rhee, I., and Xu, L. "CUBIC: A New TCP-Friendly High-Speed TCP Variant," ACM SIGOPS Operating Systems Review 42(5), 2008. doi:10.1145/1400097.1400105
9. Cardwell, N., Cheng, Y., Gunn, C. S., Yeganeh, S. H., and Jacobson, V. "BBR: Congestion-Based Congestion Control," ACM Queue 14(5), 2016. doi:10.1145/3012426.3022184
10. Floyd, S. and Jacobson, V. "Random Early Detection Gateways for Congestion Avoidance," IEEE/ACM Transactions on Networking 1(4), 1993. doi:10.1109/90.251892
11. Nichols, K. and Jacobson, V. "Controlling Queue Delay," ACM Queue 10(5), 2012. doi:10.1145/2208917.2209336
12. Gettys, J. and Nichols, K. "Bufferbloat: Dark Buffers in the Internet," ACM Queue 9(11), 2011. doi:10.1145/2063166.2071893
13. Alizadeh, M., Greenberg, A., Maltz, D., Padhye, J., Patel, P., Prabhakar, B., Sengupta, S., and Sridharan, M. "Data Center TCP (DCTCP)," SIGCOMM, 2010. doi:10.1145/1851182.1851192
14. Clos, C. "A Study of Non-Blocking Switching Networks," Bell System Technical Journal 32(2), 1953.
15. Al-Fares, M., Loukissas, A., and Vahdat, A. "A Scalable, Commodity Data Center Network Architecture," SIGCOMM, 2008. doi:10.1145/1402958.1402967
16. Greenberg, A., Hamilton, J., Jain, N., Kandula, S., Kim, C., Lahiri, P., Maltz, D., Patel, P., and Sengupta, S. "VL2: A Scalable and Flexible Data Center Network," SIGCOMM, 2009. doi:10.1145/1592568.1592576
17. Singh, A. et al. "Jupiter Rising: A Decade of Clos Topologies and Centralized Control in Google's Datacenter Network," SIGCOMM, 2015. doi:10.1145/2785956.2787508
18. Dean, J. and Barroso, L. A. "The Tail at Scale," Communications of the ACM 56(2), 2013. doi:10.1145/2408776.2408794
19. Guo, C., Wu, H., Deng, Z., Soni, G., Ye, J., Padhye, J., and Lipshteyn, M. "RDMA over Commodity Ethernet at Scale," SIGCOMM, 2016. doi:10.1145/2934872.2934908
20. Zhu, Y., Eran, H., Firestone, D., Guo, C., Lipshteyn, M., Liron, Y., Padhye, J., Raindel, S., Yahia, M. H., and Zhang, M. "Congestion Control for Large-Scale RDMA Deployments (DCQCN)," SIGCOMM, 2015. doi:10.1145/2785956.2787484
21. Patarasuk, P. and Yuan, X. "Bandwidth Optimal All-reduce Algorithms for Clusters of Workstations," Journal of Parallel and Distributed Computing 69(2), 2009. doi:10.1016/j.jpdc.2008.09.002
22. Thakur, R., Rabenseifner, R., and Gropp, W. "Optimization of Collective Communication Operations in MPICH," International Journal of High Performance Computing Applications 19(1), 2005. doi:10.1177/1094342005051521
23. Sergeev, A. and Del Balso, M. "Horovod: Fast and Easy Distributed Deep Learning in TensorFlow," 2018. arXiv:1802.05799
24. Langley, A. et al. "The QUIC Transport Protocol: Design and Internet-Scale Deployment," SIGCOMM, 2017. doi:10.1145/3098822.3098842
25. Iyengar, J. and Thomson, M. (eds.) "QUIC: A UDP-Based Multiplexed and Secure Transport," RFC 9000, IETF, 2021. rfc-editor.org/rfc/rfc9000
26. McKeown, N., Anderson, T., Balakrishnan, H., Parulkar, G., Peterson, L., Rexford, J., Shenker, S., and Turner, J. "OpenFlow: Enabling Innovation in Campus Networks," ACM SIGCOMM Computer Communication Review 38(2), 2008. doi:10.1145/1355734.1355746