# hp Onload pool stall: shared-stack lock starvation Recorded 2026-09-18. This supersedes the initial interpretation in `3da38b97` (the Rust flavour plan and hp matrix notes), not the retained benchmark measurements. **No engine, launcher, machine default or published result was changed by these experiments.** ## What future runs should know - A nonblocking `send()` can sleep inside Onload while acquiring its shared stack lock. Wall time alone is not evidence of spinning. - The observed entry path is strongly identified as a transiently empty **nonblocking packet pool** forcing a stack-lock acquisition. The waiter remains stuck even after buffers become available again. - This is not a fixed 600 ms delay, a once-per-process event, or exclusively a first-measured-run event. Capture warmup as well as every measured run. - `EF_BUZZ_USEC=1000` is a **candidate workaround**, not an adopted default. Three diagnostic sittings avoided the long stall, but p99 rose from about 95 to 118 µs. Restoring 100 µs brought the stall back. - Do not infer safety at production loads, on scan, or across all flavours from this hp/rust-pure closed-loop experiment. Validate the intended load, topology, Onload version and flavour before changing a profile. ## Scope and provenance | Item | Value | | --- | --- | | Host | hp, Xeon Gold 6154, isolated cores 8–17 | | Transport | Solarflare DAC, Onload; server sfcB / 10.9.0.2, clients sfcA / 10.9.0.1 | | libhft revision | `3da38b97` | | Installed Onload and matching local source | `dde226d7`, version dated 2026-08-07; source at `/home/yann/onload` | | Shape | rust-pure, closed loop, 32 sessions, one accepting thread, four workers sharing one stack | | Placement | Coordinator 8; workers 9–12; clients 14–17; observers on housekeeping cores 0 and 1 | | Default recipe | 20,000 warmup and 20,000 measured messages/session/run; three runs; `TPUT=0` | | Effective default settings | `EF_POLL_USEC=100000`, `EF_SPIN_USEC=100000`, `EF_BUZZ_USEC=100` | An unchanged-binary control reproduced raw measured round-trip maxima of 685003.718, 130.415 and 128.918 µs in runs 1, 2 and 3. The initial report also observed large pool tails in C++ and .NET Native, but the new instrumented mechanism and tuning tests below were performed on **rust-pure only**. The original build and installed binary had SHA-256 `5b11de7843360fa6794330e7d9f69a43b0569c1a574bc59863dc3be94a11b412`. Both were restored and compared after each temporary diagnostic install. ## Direct observation: a successful send sleeps on the stack lock One send of 212 bytes took **648788.052 µs** and returned 212. Its CPU-accounting window started 6.223 µs before the call and recorded only 29657 µs of thread CPU, 29463 voluntary context switches, and no involuntary switches or page faults. About 95% of the elapsed interval was off-CPU, not continuous spinning. Six own-thread kernel snapshots fall inside that same send interval: TID 3586087, start 589875265855096 ns, end 589875914643148 ns on CLOCK_MONOTONIC; snapshots span 589875266202311–589875521710894 ns. They show: ```text wchan=oo_eplock_lock syscall=16 0x4 0xc0045a44 ... oo_eplock_lock+0xa5/0x170 [onload] oo_eplock_lock_rsop+0x4c/0xb0 [onload] oo_fop_unlocked_ioctl+0xc5/0x280 [onload] __x64_sys_ioctl+0x87/0xc0 do_syscall_64+0x5c/0xe0 entry_SYSCALL_64_after_hwframe+0x76/0x7e ``` The installed library's `__ef_eplock_lock_slow.constprop.0` disassembly also contains ioctl request `0xc0045a44`. Its matching source is `src/lib/transport/ip/eplock_slow.c`: buzz, then `OO_IOC_EPLOCK_LOCK`. `src/lib/efthrm/eplock_resource_manager.c:oo_eplock_lock` wakes a waiter to **compete again**, not to transfer ownership to it; unsuccessful attempts sleep again. ### Duration scales with competing work Warmup remained 20,000/session. These are send durations, not averaged row maxima: | Measured iterations/session | Measured runs | Slow send, µs | CPU window, µs | Pre-send part of CPU window, µs | Voluntary switches | | ---: | ---: | ---: | ---: | ---: | ---: | | 5,000 | 1 | 151377.250 | 7961 | 762.425 | 7473 | | 20,000 | 3 | 648788.052 | 29657 | 6.223 | 29463 | | 40,000 | 1 | 1228233.677 | 56740 | 702.676 | 59857 | An 8× workload increase produced about 8.1× the wait. This strongly supports traffic-dependent lock starvation, not a fixed timeout. It does not identify the particular competing worker winning each acquisition. ## The entry path: packet-pool replenishment After narrowly authorizing `onload_stackdump lots`, a new reproduction captured a 641959.127 µs send on TID 3933305: start 621010561999906 ns, end 621011203959033 ns. The accounting window began 942.831 µs before the send and recorded 31219 µs CPU and 28900 voluntary switches. Kernel snapshots again showed the lock wait. Server stack 2 / PID 3933257 had these time-correlated counters: | Dump | Position | `tcp_send_nonb_pool_empty` | `tcp_send_ni_lock_contends` | `lock_wakes` | Ordinary free buffers | Nonblocking pool nonempty | | --- | --- | ---: | ---: | ---: | ---: | ---: | | 0042 | Before send | 27 | 27 | 0 | 401 | yes | | 0043 | Overlaps send start | 39 | 38 | 538 | 385 | yes | | 0044 | Entire dump inside send | 39 | 38 | 12438 | 392 | yes | | 0045 | Entire dump inside send | 39 | 38 | 23950 | 388 | yes | | 0046 | After send | 44 | 44 | 28900 | 379 | yes | In matching Onload `src/lib/transport/ip/tcp_send.c`, `ci_tcp_sendmsg_no_pkt_buf` increments the pool-empty counter at line 1434, calls `ci_netif_lock` at 1438, and increments the contended-acquisition counter **after acquiring the lock**, at 1443. One allocation-related acquisition remains outstanding throughout the observed send, then completes. The final lock-wakeup count exactly matches the thread's voluntary-switch count. The source retries the lock, **not the nonblocking pool**, once inside `ci_netif_lock`. Thus a transient shortage can become a long wait even after buffers return. This entry-path conclusion is supported by counter timing and source, not a captured userspace backtrace. The kernel wait itself is directly observed. `pkt_wait_spin`, `pkt_wait_primes`, `unlock_slow_pkt_waiter`, `reap_buf_limited`, `refill_rx_limited` and `refill_buf_limited` remained zero. This was not persistent system-wide packet-memory exhaustion: hundreds of ordinary buffers were free, and both wholly-during-send dumps showed `nonb_pool=1`. That field is a boolean, **not a one-buffer count** (`netif_debug.c:548`). `defer_work_limited` and `defer_work_contended_unsafe` also stayed zero, but the latter counts an interrupted/unsafe branch, not every deferred-work lock acquisition. Zero alone does not exclude all deferred-work paths. ## Controlled workaround experiment and reversal The scratch server called `onload_stack_opt_set_int("EF_BUZZ_USEC", 1000)` **before creating its stack**, checked the return value, and read the option back. Stackdump independently confirmed 1000 on the live server stack. Clients and `EF_POLL_USEC` / `EF_SPIN_USEC` were unchanged. This did not depend on caller environment variables surviving sudo or the launcher. | Sitting | Buzz, µs | Iterations/session/run | Largest raw measured RTT, µs | Largest warmup RTT, µs | Maximum sampled `lock_wakes` | Reported p99, µs | | --- | ---: | ---: | ---: | ---: | ---: | ---: | | `counters-1` | 100 | 20,000 | 642096.657 | 135.001 | 28900 | 94.46 | | `buzz1000-1` | 1000 | 20,000 | 293.665 | 133.568 | 0 | 118.39 | | `buzz1000-2` | 1000 | 20,000 | 177.632 | 131.778 | 0 | 118.63 | | `buzz1000-40000` | 1000 | 40,000 | 239.939 | 135.831 | 0 | 118.44 | | `counters-default-return` | 100 | 20,000 | 679498.682 | 671947.177 | 72252 | 95.08 | All five sittings used three measured runs. The tuned sittings contain **7.68 million measured round trips plus 5.76 million warmup round trips**. Empty-pool encounters still occurred (maximum sampled counts 54, 53, 53), but no logged send or retained phase maximum exceeded 1 ms. No sampled lock wakeup occurred. On restoring the archived 100 µs diagnostic build, send waits of 679344.682, 671840.793 and 275501.797 µs returned. They fell in **run 1 measured, run 2 warmup, and run 2 measured**, respectively. Their voluntary-switch counts 30611 + 29627 + 12014 equal the final sampled 72252 lock wakeups. This is a repeatable workaround candidate, not a universal fix. p99 was higher in the tuned tests; aggregate CPU cost was not separately benchmarked. The longer window can still expire. Validate at the intended offered load and duration, across flavours/hosts, before adopting it. Diagnostic rows do not replace published measurements, and historical rows keep their original settings. ## Corrections and ineffective shortcuts - **“First measured run only / once per process” is withdrawn.** Later warmup and later measured phases both stalled. Normal publication discards warmup. - **“Spin tuning excluded” is withdrawn.** `ops/scripts/bench-run` unconditionally exports `EF_POLL_USEC=100000`, and sudo resets caller environment. Checking `/usr/bin/onload` alone did not establish the old experiment's effective value. Moreover, Onload initially sets `buzz_usec=min(spin_usec,100)`: both 100000 and 1000 would produce the same 100 µs lock-buzz default without an explicit override. - Earlier idle-gap, warmup-volume and synchronized-start experiments remain observations under their recipes, not proofs excluding contention or guaranteeing that warmup absorbs it. Several workers can stall; one send blocks the sessions owned by that worker. Reserving a coordinator core did not remove this failure, but CPU placement still matters and remains part of the benchmark contract. - **Not a packet-allocation wait fixed by `EF_TCP_SEND_NONBLOCK_NO_PACKETS_MODE=1`.** That option is checked after the earlier blocking lock acquisition in this path. Increasing total packet memory does not directly address the observed wait either. - **`EF_STACK_PER_THREAD=1` alone does not move accepted sockets.** A structural alternative is worker-owned stacks with accepted sockets moved using `onload_move_fd` before first I/O. That is a separate design/validation task, not something implemented or proved by this workaround test. - **No “production immune” conclusion.** The experiments establish this failure under a specific recipe; they do not establish its absence elsewhere. ## Reproducing and diagnosing a future run 1. Reserve the rig; check for other benchmark processes before installing or running anything. Retain the code/library versions, binary hashes, topology, core placement and effective options. Do not change installed artifacts used by another session. 2. Run an unchanged-binary control with raw sample retention. From the repo root on hp, with the matching benchmark already built and installed: ```bash onload_case_dir=$(mktemp -d /tmp/libhft-onload-pool-XXXXXX) OUT="$onload_case_dir/baseline" ONLY='^rust-pure/onload$' \ LOOP_MODE=closed TPUT=0 BENCH_RAW_SAMPLES=1 BENCH_MACHINE=hp \ bash bench/load/bench.sh --scenario pool ``` 3. Inspect **per-run raw maxima**, not just the row's `max`: the Rust harness reports the mean of per-run maxima, so one 690 ms event can appear as about 230 ms across three runs. Instrument warmup before its samples are cleared. ```bash python3 docs/bench/evidence/onload-pool-hp-20260918/read-maxima.py \ "$onload_case_dir/baseline/raw" ``` This retained reader reports raw `end_to_end` series from HFTBRW1 files; it is not a replacement for delivery, sink or qualification checks. 4. Correlate send start/end CLOCK_MONOTONIC timestamps with thread CPU, context switches, own-thread `/proc/.../stack` and `syscall`, and before/during/after `sudo -n /usr/bin/onload_stackdump lots` snapshots. Select the **server** stack by PID/endpoint, not an assumed stack ID. Record dump start/end times and `EF_BUZZ_USEC`, `EF_SPIN_USEC`, pool/lock counters and buffer availability. 5. Read-only stack access to the root-run benchmark was granted on hp with: ```sudoers yann ALL=(root) NOPASSWD: /usr/bin/onload_stackdump lots, /usr/bin/onload_stackdump stacks ``` Do not assume that permission exists on another host. Run collectors on a verified housekeeping CPU. Here own-thread snapshots used core 0; external dumps used core 1, took about 165 ms each, then waited 100 ms. Dumps are sequential/unlocked snapshots, not atomic state or a userspace call trace. 6. For a tuning comparison, reproduce the **effective server-only** option change and verify it on the live stack. A shell export is not a supported override through the current launcher. Use baseline → candidate → baseline controls, repeated sittings, longer traffic, and both tail and CPU metrics. Do not promote this closed-loop evidence to an open-load recommendation. 7. Keep diagnostic results out of publication inputs. Restore and compare the original installed/build binaries, and verify no benchmark processes or stacks remain. Save evidence durably before temporary output is cleaned. ### Retained diagnostic method The small patches are preserved beside this record, so the method does not rely on a temporary directory: - [Send/CPU/worker-stack and warmup probe](evidence/onload-pool-hp-20260918/send-wait-probe.patch), relative to libhft `3da38b97`. - [Server-only 1000 µs experiment](evidence/onload-pool-hp-20260918/buzz1000-experiment.patch), applied **after** the first patch. It fails unless the Onload extension API is present and the option readback succeeds. - [Per-run raw maximum reader](evidence/onload-pool-hp-20260918/read-maxima.py), used to extract the measured maxima above without averaging across runs. Apply these only in a disposable source copy at the recorded revision, then build `cargo build --offline --release -p rust-pure` from `bench/rust`. They are Linux/x86-64, hp/four-worker diagnostic artifacts, **not shipping changes or a general-purpose profiler**. Review the fixed housekeeping core before use on another machine. Installation still follows the authorized bench workflow; the patches neither install binaries nor change the launcher. The probe reads RUSAGE_THREAD every 256 sends or after a 1 ms checkpoint age, and again after a send lasting at least 1 ms. Its CPU/switch deltas include the reported short pre-send window, not exactly the call. It preserves errno; errno is meaningful as an error only when `send` fails. Own-thread kernel snapshots are capped at six per worker, at least 50 ms apart, so absence of a later snapshot is not proof of no later stall. Slow-send logging is capped at 16 per thread. Phase maxima are retained separately. The first, heavier probe called getrusage around **every** send and failed to reproduce the stall (largest measured RTT 124.644 µs). The unchanged control immediately reproduced 685003.718 µs. Instrumentation can mask this problem; always retain an uninstrumented control. Even the lighter probes alter timing. Original complete local captures, recipes, raw HFTBRW files and scratch sources were retained at `/tmp/libhft-onload-diagnostic-87ItpS` on hp. That path is provenance, **not a durable archive**. This record retains the decisive excerpts, numbers, limitations and probe patches; it does not claim to archive every raw sample or full stack dump (which also contains process environment information).