# FIX benchmark coverage — status by language and engine

Where every engine in the [target roster](TARGET_ENGINE_ROSTER.md) has got to, for both the
microbench (Stage 2 codec) and the full bench (four flows, Stages 3–5), grouped by language so each
libhft flavour sits beside the engines it is compared with. Java and .NET each have two libhft
flavours, so those sections carry two libhft rows.

The plans for moving these cells are [PLAN_MICROBENCH.md](PLAN_MICROBENCH.md) and
[PLAN_FULL_BENCH.md](PLAN_FULL_BENCH.md).

> **Read the provenance section before quoting a number.** The four-flow cells are transcribed from
> [`COVERAGE.csv`](COVERAGE.csv), a campaign record whose evidence tree is not present on every
> developer machine — see [Provenance and the regeneration hazard](#provenance-and-the-regeneration-hazard).
> Every cell is therefore **SOURCE** grade, not **MEASURED** on the machine you are reading this on.

## Status by language

Rendered by `tools/gen-bench-status` from [`engines.tsv`](engines.tsv) (who, and microbench
status) and `COVERAGE.csv` (four-flow cells). An engine with no four-flow evidence shows `not run`
rather than vanishing. Only **S5 pass** is report-grade; `diag`, `preflight` and `smoke` rows are
never a speed claim, and `non-parity` means the engine ran that flow but not in a comparable shape.

<!-- BEGIN gen-bench-status:status -->
<!-- Rendered by tools/gen-bench-status from engines.tsv and COVERAGE.csv. Do not edit by hand: change the data and rerun it. -->

### Summary: what can be compared today, by language

| Language | libhft flavours | Microbench rows (libhft / others) | Others at S5 pass | Publishable full-bench comparison |
|---|---|---|---|---|
| C++ | C++: S5 pass 4/4 | 1 / 6 | none | no: no other engine passes S5 |
| C | — | 0 / 1 | none | — (libhft has no flavour in this language) |
| Java | Java Pure: S5 pass 4/4, Java/JNI: S5 pass 4/4 | 2 / 4 | `crossfix`, `philadelphia`, `philadelphia-fast` | **yes** |
| .NET | .NET Pure: S5 pass 4/4, .NET Native: S5 pass 4/4 | 2 / 2 | none | no: no other engine passes S5 |
| Rust | Rust: S4 diag 4/4 | 1 / 5 | none | no: neither side passes S5 |
| Go | — | 0 / 1 | none | — (libhft has no flavour in this language) |

### C++

| Engine | Microbench | Full bench | nos_er | md | md_nos_er | tick_to_trade | Notes |
|---|---|---|---|---|---|---|---|
| `tcp-ping-pong-cpp` | — | floor | — | — | — | — | network floor: the RTT before any FIX processing |
| **libhft-cpp** (C++) | yes | **S5 pass 4/4** | pass | pass | pass | pass |  |
| `fix8` | yes | S5 diag 4/4 | diag | diag | diag | diag | microbench: D/8/depth-10 W full-message codec; generated benchmark dictionary; 3/3 mutation probes passed; full bench: public codec admission recorded 2026-10-02 |
| `llfix` | partial | 0/4, adapter-ready | blocked | blocked | blocked | blocked | microbench: public encoder only: D/8/depth-10 W; cold field/framing checks passed; no receive parser; full bench: adapter ready; retained session, store/recovery and host evidence missing |
| `nexusfix` | yes | S5 diag 4/4 | diag | diag | diag | diag | pinned to untagged upstream snapshot ffa8b0a; v1.0.0 lacks the public engine headers |
| `openfix` | excluded | 0/4, tooling-blocked | blocked | blocked | blocked | blocked | no licence; Bazel-only; gnu++23 |
| `quickfix_cpp` | yes | S5 diag 4/4 | diag | diag, non-parity | diag, non-parity | diag, non-parity |  |
| `robaho_cpp_fix_engine` | excluded | 0/4, compile-failed | blocked | blocked | blocked | blocked | upstream build fails; licence unknown |
| `fixpp` | yes | codec only | — | — | — | — |  |
| `hffix` | yes | codec only | — | — | — | — |  |
| `robaho_cpp_fix_codec` | excluded | codec only | — | — | — | — | microbench: no licence at 4a95e7c: Stage 2 adapter removed; archived timings withheld; full bench: no tested session endpoint |

### C

| Engine | Microbench | Full bench | nos_er | md | md_nos_er | tick_to_trade | Notes |
|---|---|---|---|---|---|---|---|
| `libtrading` | yes | S5 diag 4/4 | diag | diag, non-parity | diag, non-parity | diag, non-parity | microbench: public numeric-tag parser/unparser; depth-10 W requires the existing 128-field capacity patch; full bench: upstream long dormant |

### Java

| Engine | Microbench | Full bench | nos_er | md | md_nos_er | tick_to_trade | Notes |
|---|---|---|---|---|---|---|---|
| `tcp-ping-pong-java` | — | floor | — | — | — | — | network floor: the RTT before any FIX processing |
| **libhft-java-pure** (Java Pure) | yes | **S5 pass 4/4** | pass | pass | pass | pass |  |
| **libhft-java-jni** (Java/JNI) | yes | **S5 pass 4/4** | pass | pass | pass | pass |  |
| `artio` | yes | S5 diag 4/4 | diag | diag | diag | diag | microbench: source-owned D/8/W dictionary generation; retained host smoke/parity required; full bench: next run pinned at upstream 0.182 |
| `crossfix` | excluded | **S5 pass 4/4** | pass | pass | pass | pass | microbench: its obfuscated buffer API cannot be reset and reparsed fairly; full bench: CrossFIX 1.6; 2.0 is not headline-grade; licensed installation only |
| `falcon` | partial | S3 smoke 1/4 | smoke | unsupported | unsupported | unsupported | microbench: D and 8 only: no V/W/X market-data types; full bench: no market-data messages, so N/A for the four flows |
| `philadelphia` | yes | **S5 pass 4/4** | pass | pass | pass | pass |  |
| `philadelphia-fast` | — | **S5 pass 4/4** | pass | pass | pass | pass | a transport configuration of philadelphia, not a separate codec |
| `quickfixj` | yes | S5 diag 4/4 | diag | diag | diag | diag | holds 750 msg/s; misses the contract rate |

### .NET

| Engine | Microbench | Full bench | nos_er | md | md_nos_er | tick_to_trade | Notes |
|---|---|---|---|---|---|---|---|
| `tcp-ping-pong-net` | — | floor | — | — | — | — | network floor: the RTT before any FIX processing |
| **libhft-dotnet-managed** (.NET Pure) | yes | **S5 pass 4/4** | pass | pass | pass | pass | SCAN mlx5 DAC S5 pass 4/4; HP Solarflare repeat pending |
| **libhft-dotnet-native** (.NET Native) | yes | **S5 pass 4/4** | pass | pass | pass | pass | SCAN mlx5 DAC S5 pass 4/4; HP Solarflare repeat pending |
| `fixantenna_net` | yes | S5 diag 4/4 | diag | diag | diag | diag | microbench: public RawFixUtil/Message codec; source pinned ff38f51; stage-2 parity passed; full bench: SCAN mlx5 DAC S5 diagnostic 4/4: canonical wire, but report rate below 10k/s |
| `quickfix_n` | yes | S5 diag 4/4 | diag | diag | diag | diag | SCAN mlx5 DAC S5 diagnostic 4/4: canonical wire, but report rate below 10k/s |

### Rust

| Engine | Microbench | Full bench | nos_er | md | md_nos_er | tick_to_trade | Notes |
|---|---|---|---|---|---|---|---|
| `tcp-ping-pong-rust` | — | floor | — | — | — | — | closed-loop TCP RTT before FIX processing; bench/rust/ping-pong |
| **libhft-rust-pure** (Rust) | yes | S4 diag 4/4 | S4 diag canon | S4 diag canon | S4 diag canon | S4 diag canon | canonical four-flow two-process localhost diagnostics; physical Stage-5 integration pending |
| **libhft-rust-native** (Rust over the C ABI) | parked | parked | — | — | — | — | parked 2026-09-11 |
| `dfx` | yes | S5 diag 4/4 | diag | diag, non-parity | diag, non-parity | diag, non-parity | derived from QuickFIX/n; market data needs the exact ten-level contract |
| `fixer_rs` | yes | S5 diag 4/4 | diag | diag, non-parity | diag, non-parity | diag, non-parity | a port of QuickFIX/Go; market data needs the exact ten-level contract |
| `nanofix` | yes | S4 diag 4/4 | S4 diag canon | S4 diag canon | S4 diag canon | S4 diag canon | canonical four-flow two-process Stage 4 medium diagnostics pass with verified admin observer overlay; physical Stage 5 pending |
| `truefix` | yes | S4 diag 4/4 | S4 diag canon | S4 diag canon | S4 diag canon | S4 diag canon | canonical four-flow Stage 4 diagnostics with V/W dictionary overlay; mixed-flow rate fails and tick stability warns; Stage 5 pending |
| `ferrumfix` | yes | codec only | — | — | — | — | through the pinned v0.7.0 shim |

### Go

| Engine | Microbench | Full bench | nos_er | md | md_nos_er | tick_to_trade | Notes |
|---|---|---|---|---|---|---|---|
| `quickfix_go` | yes | S5 diag 4/4 | diag | diag | diag | diag | cross-runtime anchor: libhft has no Go flavour |
<!-- END gen-bench-status:status -->

---

## How to read the status

"Bench flavor" is two independent axes: a full-bench cell is a *(stage, scenario)* pair. The
microbench is Stage 2 alone; its "yes" means the Stage-2 launcher runs a fair adapter and it passed a
smoke run. The [2026-09-29 campaign](RECORD_2026-09-29_stage2-microbench.md)
provides host-local timing and parity evidence; inspect each report's build
and parity tables before quoting an engine's numbers. The robaho codec timing
cells are withheld because its pinned source has no licence grant.

### Axis A — stages (increasing rigour)

From [`CAMPAIGN_TEST_PLAN.md`](CAMPAIGN_TEST_PLAN.md).
Each stage is a gate: an engine that fails one does not reach the next.

| Stage | Name | What it establishes |
|---|---|---|
| 0 | Candidate registry | the engine exists, is licensed usably, and is fetchable |
| 1 | Build and upstream test | it compiles and its own test suite passes |
| 2 | Parser / builder microbench | encode and decode cost, no session |
| 3 | Session capability smoke | it can log on and exchange the scenario at all |
| 4 | Realistic workflow benchmark | latency under the canonical scenarios, single machine |
| 5 | Cross-machine Linux benchmark | latency over a real NIC between two hosts |
| 6 | Cross-platform portability | it runs beyond the primary platform |
| 7 | Correctness and conformance | it is *right*, not merely fast |

Stage 5 is the one that produces quotable numbers. Stages 2 and 3 produce signal; stage 4 is
single-machine and therefore loopback-contaminated.

**Stage 5 has two grades, and conflating them is the most common misreading of this matrix:**

- **`S5 pass`** — report-grade. Full sample count, cross-machine, parity-checked.
- **`S5 diag`** — diagnostic only. Ran and produced timings, but sample count or parity is
  insufficient to quote. *Not* a pass.

`non-parity` on a scenario means the engine ran it but not in a shape comparable to the others,
so cross-engine comparison of that cell is invalid.

### Axis B — canonical scenarios

The authoritative contract is `CONTRACT_SCENARIOS.md`. Four scenarios, and only four —
earlier rows (`md_er`, `business`, `nos_md_er`, `nos_md_er_business`) are retired and folded in.

| Scenario | Wire exchange | What it stresses |
|---|---|---|
| `nos_er` | `35=D → 35=8` | order entry round trip |
| `md` | `35=V → 35=W` | market-data request and snapshot |
| `md_nos_er` | `35=V → 35=W → 35=D → 35=8` | mixed session, both flows interleaved |
| `tick_to_trade` | server-side `35=W → 35=D` | the number that actually matters commercially |

**4 scenarios × 8 stages = the full flavor grid.** The tables above report the scenario axis at
the highest stage each engine reached, because that is the only summary that does not overstate.

---

## Name reconciliation

Three name-spaces exist for the same engines. This is why coverage reports disagree.

| CSV matrix | `DECLARED_ENGINES` | Resolution |
|---|---|---|
| `libhft-cpp` | `libhft` | same engine |
| `libhft-java-jni` | `libhft-java-jni` | same engine |
| `libhft-java-pure` | `libhft-java-pure` | same engine |
| `libhft-dotnet-managed` | `libhft-net-managed` / Stage-5 `libhft-net` | managed C# row; legacy Stage-5 executable still emits `libhft-net` |
| `libhft-dotnet-native` | hftnet/provider-smoke | native C++-backed .NET row |
| `quickfixj` | `quickfix_j` | same engine, punctuation differs |

`DECLARED_ENGINES` now includes the full-bench targets, including CrossFIX,
Philadelphia-fast, FIX Antenna, TrueFix and NanoFIX. The codec-only and
network-floor rows still come from the roster rather than this full-engine
list; their absence from it is intentional.

## Provenance and the regeneration hazard

`COVERAGE.csv` is *generated*, by
[`generate_fix_benchmark_engine_coverage.py`](../../bench/cpp/fixbench/tools/generate_fix_benchmark_engine_coverage.py),
from an evidence tree at `C:/Temp/libhft/open-source-fix-engines`.

**That tree is not on every developer machine.** On a box with a partial tree, regenerating turns
most of this matrix to `0/4 / none` — the engines have not regressed, the evidence is simply
absent locally. Two consequences:

- **Do not treat a locally-regenerated matrix as a result.** Check the evidence tree size first.
  A `results/` directory with a handful of entries cannot support a 28-engine matrix.
- **Do not regenerate over `docs/`.** The generator's default `--output` is `COVERAGE.csv`,
  a path that no longer exists. Redirecting it at the live `docs/` copy on a machine with a partial
  tree overwrites a full campaign record with an empty one, and the loss is silent.

The status of every cell above is therefore **SOURCE** — transcribed from the committed campaign
record — not **MEASURED** on the machine you are reading this on.

## Status of this document

Written 2026-08-14 as a hand-transcribed matrix; reorganised by language on 2026-09-29, when the
tables became `tools/gen-bench-status` output and the roster, open work and "what the matrix says"
moved to [TARGET_ENGINE_ROSTER.md](TARGET_ENGINE_ROSTER.md), the two plans and the rendered summary.
The stage and scenario definitions above were extracted from `CAMPAIGN_TEST_PLAN.md` and
`CONTRACT_SCENARIOS.md`; the name reconciliation is still hand-written and is the section most
likely to rot.