Fix three fabricated-result bugs, close the RCE, and overhaul the documentation

[jeremymanning]

What this is

A sweep of every open issue, plus a documentation overhaul, plus the defects
that turned up while doing both. 52 commits.

The headline is not the issue count. It is that three separate code paths
were returning fabricated answers instead of running your function, all
reachable by default, and none of them detectable by the test suite as it
stood.

The fabricated-result bugs

Where	What you got instead of your answer
create_simple_subprocess_fallback	the literal string "Function execution completed"
_attempt_client_side_gpu_parallelization	traces of random 100×100 matrices
detect_loops range fallback	a tenth of the work

Each was silent — no error, no warning, a plausible-looking value.

1. The subprocess stub. When a function was classed "complex" and flattening
failed, _execute_single substituted a closure whose entire remote body was
result = "Function execution completed". Reproduced:

REAL ANSWER      : 5
complexity_score : 999 | is_complex: True
flatten success  : False
WHAT CLUSTRIX RUNS -> 'Function execution completed'

analyze_function_complexity returned 999 / is_complex=True whenever it could
not read the source — so flattening was attempted precisely when it could not
work, and any function with one nested helper qualified.

2. GPU parallelization. auto_gpu_parallel defaulted to True and triggered
on any host reporting ≥2 GPUs. The path never called the decorated function: it
ran a fixed torch.randn(100, 100) program per GPU and scraped the matrix trace
out of stdout.

3. Loop ranges. detect_loops fell back to range(10) when it could not
evaluate a range, so a loop over range(n) was chunked as ten iterations:

BEFORE:  detect_loops(range(n)) -> range(0, 10)
AFTER:   detect_loops(range(n)) -> None

It reached that value by calling eval() on text sliced out of your source,
under a comment admitting the approach was dangerous. Now a literal-only reader
that cannot execute anything.

All three are deleted rather than patched, along with the machinery that existed
only to serve them: function_flattening.py, dependency_resolution.py and
gpu_utils.py (2,859 lines). Nothing is lost — serialize_function already
round-trips every case flattening was meant to rescue, verified in a fresh
interpreter with the defining module off sys.path.

Security

Remote-to-local code execution. Results were HMAC-verified before
deserialization; error.pkl was not, and no error.pkl.hmac existed anywhere. A
hostile cluster only had to make the job fail. Demonstrated with a payload
whose __reduce__ called os.system; both read sites were wrapped in
except Exception: pass, so a failed exploit was silent. After:

--- unsigned payload, as a hostile cluster would write it ---
  REFUSED: PayloadAuthenticationError: ... produced a payload with no signature
--- signed with a guessed key ---
  REFUSED: PayloadAuthenticationError: ... failed its integrity check
--- empty key recorded on the caller (the old fail-open path) ---
  REFUSED: PayloadAuthenticationError: No result-signing key is recorded ...

exploit ran: False

Also: verification failed open when no key was recorded; cloud results were
never verified at all; SSH host keys were never checked (AutoAddPolicy at 12
sites); the job script had no shlex.quote anywhere, so remote_work_dir,
module loads and environment variables reached the shell unquoted — two of those
sites inside a python -c " string. Saved configs were world-readable with
credentials in plaintext, and ClusterConfig.__repr__ printed them into any
traceback.

Correctness

• Environment replication silently dropped a third of your packages. Any
  freeze line containing @ was skipped; with uv on PATH every conda-built
  package renders as name @ file:///.... Measured: 563 installed, 376 returned.
  Now 559, with the six genuinely unreproducible ones (editable installs, git
  checkouts) refused at submit time by name rather than dropped.
• Loop analysis reported unparallelizable loops as parallelizable — total += i
  came back with zero dependencies. detect_loops_in_function never dedented its
  source, so every method or closure raised IndentationError into a swallowed
  exception and returned [].
• Local auto-parallelization never parallelized: it injected a keyword the
  callee could not accept and swallowed the TypeError. The remote path had the
  same defect.
• Kubernetes reported failed jobs as successful, and decoded results with
  ast.literal_eval on the pod log.
• The by-value serialization walk missed instance attributes, builtin-container
  subclasses, namespace packages and functools.partial, and degraded silently
  to by-reference at its node cap.

The test suite

BEFORE:  83 failed, 1437 passed, 2 errors
AFTER:   1763 passed, 12 skipped, 0 failed

The last 13 failures were test pollution, not product bugs — every one passed
in isolation. Three nested leaks: reset_config restored 8 of a hundred-odd
fields; it restored fields but not the module binding; and a class-local
reset_config fixture shadowed the autouse one. After that file ran, the live
config still carried 20 drifted fields, and the widget reads the live config.

A safety hole worth naming: most files under tests/real_world/ carried no
real_world marker, so the command CLAUDE.md documented as safe —
pytest tests/ -m "not real_world" — executed them, making real SSH and cloud
calls. 26 tests were leaking. The marker is now applied by path.

Documentation

Rewritten, then reviewed adversarially five times until the reviews came back
clean. New: introduction.rst, quickstart.rst, execution_model.rst,
configuration.rst, limitations.rst, troubleshooting.rst.

BEFORE:  36 examples checked in 5 files, 60 sphinx warnings
AFTER:   168 examples checked in 23 files + docstrings, 0 warnings, 0 failures

The checker had a hand-written file list, so the pages written that day were not
checked at all. It now discovers everything Sphinx publishes. Making that work
required fixing three defects in the checker itself: an example calling
sys.exit() killed it at the third of twenty files and exited 0; an example
dialling a fictitious host hung it indefinitely (paramiko retries through
EINTR, so the in-process alarm was ignored — each file now runs in its own
subprocess); and it demanded every third-party import be installed, so an ML
example importing tensorflow counted as broken docs.

The reviews found what the tooling structurally cannot: the checker runs
blocks, it never compares a comment to what the code did. # Loop gets parallelized automatically appeared in README, CLAUDE.md, three API pages and
four notebooks — false in every case, and every block passed. docs/notebooks/
turned out to be a stale duplicate tree that five "Open In Colab" badges pointed
into, so a reader clicking through landed in an un-overhauled copy.

What is still not verified

• PBS and SGE — implemented, share SLURM's environment path, never run
  against a real scheduler.
• Kubernetes — never run against a real cluster.
• AWS, GCP, Azure, Lambda VM backends — no cloud job has been shown to run
  end to end.
• PyPI serves 0.1.1, which predates all of the above fixes. README and the
  install page now say so and give the git install command.

Reproducing

pip install -e ".[dev]"
pytest tests/ -m "not real_world" --ignore=tests/real_world --ignore=tests/integration
python scripts/check_docs_examples.py
python -m sphinx -b html docs/source /tmp/docs

🤖 Generated with Claude Code

https://claude.ai/code/session_012gTBDPK16HUZ3kHQ2QyjuU