Skip to content

Replace mwpf's HiGHS LP backend with a pure-Rust simplex shim (no more cmake) - #461

Merged
ciaranra merged 7 commits into
devfrom
worktree-mwpf-pure-rust-lp
Aug 9, 2026
Merged

Replace mwpf's HiGHS LP backend with a pure-Rust simplex shim (no more cmake)#461
ciaranra merged 7 commits into
devfrom
worktree-mwpf-pure-rust-lp

Conversation

@ciaranra

@ciaranra ciaranra commented Aug 9, 2026

Copy link
Copy Markdown
Member

Summary

The optional mwpf decoder feature was the only reason PECOS builds needed cmake: mwpf's vendored highs wrapper pulls highs-sys, which builds the HiGHS C++ LP solver via cmake. This PR replaces that wrapper with two dependency-free pure-Rust crates, wired in through [patch."https://github.com/yuewuo/mwpf"] — no fork of mwpf, no upstream changes, same pinned revision:

  • crates/pecos-lp — the solver itself, named for what it does: a small dense two-phase tableau simplex with Bland's rule, sized for decoder-relaxation LPs. Owned PECOS primitive, reusable by future work (e.g. LP-relaxation distance bounds).

  • crates/pecos-highs-shim — a thin facade (package name highs, forced by cargo's patch mechanics) exposing exactly the API subset mwpf's float_lp path uses, delegating to pecos-lp.

  • cmake, highs-sys, and the C++ toolchain requirement are gone from the dependency graph (cargo tree -p pecos-mwpf shows only the path-patched shim; Cargo.lock drops cmake and highs-sys).

  • pecos rust check/clippy/test no longer refuse to run without cmake, and pecos python build enables mwpf unconditionally (opt out with PECOS_BUILD_MWPF=0).

  • New crates/pecos-mwpf/tests/differential.rs: an explicitly-invoked (ignored-by-default) oracle harness that decodes a deterministic 800-shot stream on hyperedge-rich DEMs and dumps/compares per-shot observable masks, for comparing LP backends shot-for-shot.

Solver design

Preprocessing (bound shifts, negations, free-variable splits, equality rows as inequality pairs) feeds a two-phase dense tableau simplex. Numerical safeguards, each adversarially motivated:

  • No epsilon-snapping inside tableau arithmetic; elimination runs for every nonzero factor. Zeroing entries without a compensating row operation shifts the row space and a later small pivot amplifies that into order-one silent errors.
  • Ratio test prefers pivots above 1e-7, falling back to smaller ones only when no alternative exists, with exact minimum-ratio / lowest-basic-index (Bland) tie-breaking.
  • A final primal-feasibility audit substitutes the solution back into the original constraints; any violation beyond a scaled 1e-7 tolerance returns SolveError instead of Optimal, so numerical trouble is loud (mwpf panics) rather than a silently corrupted decode.
  • Deterministic by construction: fixed iteration order, no randomization; solving the same model twice is byte-identical.

Verification

  • 800-shot differential against a real-HiGHS build (fixture dumped from this branch's parent with the C++ backend): 797/800 identical observable masks; the 3 divergent shots were investigated by weighing both corrections — differences go in both directions (the shim's correction is strictly lighter on two of the three) and lane-aggregate correction weights agree within 0.3%, i.e. approximate-search steering under JointSingleHair with cluster_node_limit, not solver error. 1611 LP solves exercised, up to 116 columns x 133 rows.
  • Adversarial review (independent reviewer) found a critical tableau-corruption bug in the initial implementation (epsilon-snapping + unfloored pivots: 8/2000 silent wrong-Optimal on mwpf-shaped LPs at the 58-pair scale, worst constraint violation 2.78). Fixed as above; the reviewer's own reproducer now matches HiGHS to 12 significant digits, and re-running the reviewer's census harnesses gives 0 wrong answers in 2000 cases at the failing scale plus 2000 more at mid scale. At double mwpf's observed LP size (100 pairs), 1 case in 500 trips the feasibility audit and fails loudly — documented headroom limit, not silent corruption.
  • Independent QA verification: 5000 random LPs (all bound forms) cross-checked against HiGHS via scipy with zero real mismatches; breakpoint tracing confirmed the shim's simplex is the code mwpf actually reaches at decode time.
  • A second, independent cross-model adversarial review of the final code drove another hardening round: solution values are no longer epsilon-snapped (consumers act on sub-1e-9 magnitudes), the ratio test takes the exact minimum ratio over every above-noise pivot (skipping a binding small-pivot row could change the answer) with safe-pivot preference on exact ties, crossed variable bounds and negative row bounds are no longer silently rewritten by tolerances, and the feasibility audit rejects non-finite activities. Matrix entries at or below 1e-9 are treated as zero, matching HiGHS's small_matrix_value default. Both of the review's concrete counterexamples are now unit tests that assert the correct optimum.
  • 20 shim unit tests, including mutation-verified guards for the implicit-upper-bound row and the epsilon scale (both mutants fail), and the cross-review counterexamples.
  • Decode speed on the differential workload: 0.91s vs 1.67s for the HiGHS lane on the same 800 shots.
  • just lint clean across the repo; cargo clippy -p highs -p pecos-mwpf --all-targets zero warnings; full workspace clippy is exercised by CI's --all-features gate, which previously compiled HiGHS via cmake on every run and now builds the shim instead.

Out of scope (follow-ups)

  • Release CI still installs cmake for wheel builds (python-release.yml); that machinery, the pecos install cmake vendoring path, and docs/user-guide/cmake-setup.md are now vestigial and can be retired in a dedicated PR.
  • The mwpf feature remains opt-in on pecos-rslib for local cargo use; whether to flip it on by default is a separate decision.
  • If future workloads push cluster LPs well past ~200 columns, the dense tableau would want Harris-style ratio testing or periodic refactorization; the feasibility audit turns that horizon into loud errors rather than wrong decodes.

@ciaranra
ciaranra merged commit f8763a4 into dev Aug 9, 2026
63 checks passed
@ciaranra
ciaranra deleted the worktree-mwpf-pure-rust-lp branch August 9, 2026 18:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant