Remove every backend that has never been run against real hardware

[jeremymanning]

What this is

Removal of every execution backend that has never been run against real
hardware, and of everything that existed only to serve them. 43 commits,
127 files deleted, 63,437 lines removed.

The supported set is now exactly the four backends that have been
demonstrated end to end:

SUPPORTED_CLUSTER_TYPES = ("local", "ssh", "slurm", "huggingface")

Removed: pbs, sge, kubernetes, aws, gcp, azure, lambda_cloud,
huggingface_spaces. Each has a tracking issue with a file/line inventory,
the introducing commit, and what it would take to bring it back:
#140 PBS, #141 SGE, #142 Kubernetes, #143 AWS, #144 GCP, #145 Azure,
#146 Lambda Cloud.

Nothing here is a claim that the removed code was broken. It is a claim that
nobody has ever seen it work, and that offering it in the cluster-type
dropdown said otherwise.

What a user who was relying on one of these now sees

The old behaviour for a stale clustrix.yml was difflib guessing:
k8s_namespace came back as "did you mean ...?" pointed at an unrelated
field, and cluster_type: pbs was accepted silently and failed much later.

load_config, file containing `cluster_type: pbs`
  ValueError: <path>: cluster_type='pbs' is no longer implemented. It was removed
  in v0.2.0 because it had never been verified against real hardware. Its return
  is tracked in issue #140. Supported types are: local, ssh, slurm, huggingface.

load_config, file containing `k8s_namespace: compute`
  ValueError: <path> contains unknown setting(s): k8s_namespace configured
  Kubernetes, which has been removed (see issue #142)

load_config, file containing a genuine typo `cluster_hostt`
  ValueError: <path> contains unknown setting(s): cluster_hostt
  (did you mean cluster_host?)

The did-you-mean path is deliberately still reachable. A removed setting must
not look like a spelling mistake, and a spelling mistake must not look like a
removed setting.

validate_cluster_type() is now the single check, called from
ClusterConfig.__post_init__, from configure(), from load_config and from
the executor. Previously only load_config checked, so every other route
carried a dead backend all the way to ClusterExecutor.submit_job — which
tested the type after self.connect(). Asking for a backend that no longer
exists cost an SSH round trip to a host that was never going to be used.

configure() also validates before it applies anything. It used to setattr
its way through kwargs and raise partway, so a call that failed had still
changed the live configuration.

Breaking changes

• The cost-monitoring API is gone, with all five exported functions:
  cost_tracking_decorator, get_cost_monitor, start_cost_monitoring,
  generate_cost_report, get_pricing_info (plus ResourceUsage,
  CostEstimate, CostReport). Its dispatcher handled exactly
  lambda/aws/azure/gcp; with those gone it was a public API that could only
  return None.
• configure(auto_install_deps=...) is gone. It installed cloud provider
  dependencies.
• @cluster loses ten parameters: provider, instance_type, region,
  platform, auto_provision, cluster_name, node_count, node_type,
  kubernetes_version, from_scratch.
• Every k8s_*, aws_*, azure_*, gcp_*, lambda_*, cloud_* and
  cost_monitoring field is off ClusterConfig.

Two things worth a second look

hf_hardware, hf_username and hf_sdk sat under a comment reading
"Hugging Face Spaces settings". Two of them are read by hf_jobs.py — the
one verified API backend — as fallbacks for hf_flavor and hf_namespace.
Deleting on the comment's word would have broken it. They are kept and the
comment is corrected. hf_sdk genuinely was Spaces-only (the
gradio/streamlit/static SDK) and nothing reads it, so it went.

The cluster-type list had drifted into three hardcoded copies.
enhanced_notebook_widget.py still offered pbs, sge, kubernetes, aws, azure
and gcp. It now reads SUPPORTED_CLUSTER_TYPES, which is why that constant
exists.

Defects found and fixed along the way

#147 — the pre-push hook could not block a push. run_real_world_tests.py
dropped every runner.run_*_tests() return value and never called sys.exit,
so it exited 0 no matter what. All four categories printed "failed" and the
hook then announced ✅ All real-world tests passed! and allowed the push.
Same class as the flake8 --exit-zero and mypy continue-on-error steps fixed
in #138. Verified by appending a deliberately failing test:

EXIT CODE: 1
❌ Filesystem tests failed (exit 1)
    assert False, "deliberate failure to verify exit-code propagation"
E   AssertionError: deliberate failure to verify exit-code propagation

Before the change that same case exited 0 with an empty message body — the
second half of #147: only stdout was printed, and a pytest collection error
goes to stderr.

The host-key tests were writing into the developer's real
~/.ssh/known_hosts — 83 stale [127.0.0.1]:<ephemeral port> entries had
accumulated, and port reuse against a fresh server key then raised
BadHostKeyException and failed the reject test. The fixture now redirects
~. The 83 entries in the user's own file were left alone.

scripts/setup_validation_credentials.py was unreachable. Every run ended
at ❌ 1Password CLI not available! and told the reader to
brew install --cask 1password-cli — but is_op_available() has returned
False unconditionally since 1Password support was removed in #97. Deleted;
clustrix credentials setup already does the job for real.

tests/test_config.yml was a committed test artifact. A saved profiles
file that load_config has never been able to read, referenced by nothing,
still carrying aws_* / k8s_* / cost_monitoring keys.

Formatting was checked with the wrong black. Seven files were formatted by
black 25.11.0 while the project pins black==26.3.1 in both pyproject.toml
and setup.py. A local black --check passed where CI would have failed.

Also filed, not fixed here

#148 — the real-world SSH tests bypass host-key verification.
set_missing_host_key_policy(paramiko.AutoAddPolicy()) appears at 37 sites
across 29 files under tests/, which clustrix/ssh_security.py and
CLAUDE.md both forbid. clustrix/ itself complies. The consequence is that
the host-key verification path every real user gets is never exercised by a
real SSH connection.

It is not fixed here because the one-line change is deliberately not
behaviour-preserving: configure_host_key_policy defaults to "reject", so
each of those files needs to be run against its real cluster to confirm the
host is trusted and the test still passes. A blind sweep would trade a silent
security gap for a silently broken test suite.

Added after this PR was opened

Reviewing the open issue backlog against the code as it now stands surfaced
several things worth fixing here rather than tracking.

A real correctness bug: the job wait loop had no deadline (#123).
_wait_for_scheduler_result polled under a bare while True, with no timeout
field anywhere in ClusterConfig. A job that never reached a terminal state —
held by the scheduler, behind a queue that never cleared, a node stuck draining
— hung the caller forever, with no diagnostic and no way out but Ctrl-C. This
is on the primary path of both verified scheduler backends.

New job_wait_timeout, default 86400. Deliberately generous: a real HPC queue
wait legitimately runs into hours, so a short default would break correct
usage. None restores the old unbounded wait. On expiry the job is not
cancelled — it may still be queued, and killing someone's allocation because
the client got bored is not this function's call:

TimeoutError: Job job_1 did not finish within 2s (config.job_wait_timeout).
Its last known status was 'running'. The job has NOT been cancelled; its files
are at /scratch/someone/.clustrix/jobs/job_1 on the cluster. Raise
job_wait_timeout, or set it to None to wait indefinitely.

Four tests, no mocks: a real ClusterExecutor running the production loop
against a real subclass of the real SchedulerManager whose job never leaves
the queue.

Documentation examples now run in CI (#124). scripts/check_docs_examples.py
executes 143 blocks, 110 of them for real, and passed — but grep -rn check_docs_examples .github/ returned nothing, so the docs were correct only
for as long as someone remembered to check by hand.

Adding the step exposed a second defect. It first failed with
ValueError: '.../site-packages/clustrix/config.py' is not in the subpath of '/home/runner/work/clustrix/clustrix', because it ran after Test installation's non-editable pip install .. Making the path arithmetic
tolerant would have been worse than the crash — the examples would then have
been checked against a different copy of the code and passed while proving
nothing. The step moved before the install, and the checker now refuses a
foreign install by name.

Coverage has a floor that can fail (#115). grep -rn fail_under returned
zero hits anywhere: the old fail_under = 90 had been removed as
unreproducible, and nothing replaced it, so the project measured 68% and gated
on nothing. Set to 66 — two points of headroom so a version difference across
the matrix cannot turn a green run red, while a regression still does. Verified
it fires rather than decorating: one test file under the same command exits 1
with Required test coverage of 66.0% not reached. Total coverage: 14.40%.

Packaging extras installed SDKs for deleted backends. pip install clustrix[aws] pulled boto3 and the Kubernetes client for code that is not in
the package. The kubernetes, aws, azure, gcp and cloud extras are
gone, along with the cloud SDKs in test and all, and CI's install line
drops the extra with them. Nothing in clustrix/ imports any of them;
scripts/aws/ imports boto3 lazily with its own "not a clustrix dependency"
message. Verified by resolving the new line in a clean 3.11 venv.

real-world-tests.yml handed cloud credentials to jobs that cannot use
them — LAMBDA_CLOUD_API_KEY, GCP_PROJECT_ID, GCP_JSON and two AWS keys
across four steps. Removed; what remains maps exactly to the surviving
backends.

#147 had a second and worse consumer. Every step in real-world-tests.yml
invokes the runner as a bare command, so Actions would fail the step on a
non-zero exit — but the script always exited 0. That workflow is the only one
that runs the real-world suite, so the single thing exercising clustrix against
real clusters had been reporting green unconditionally.

Notebook fixes. All seven surviving tutorials execute locally (the 10
failing cells across slurm_tutorial and ssh_tutorial are DNS on placeholder
hostnames — they need a cluster). Eight :doc:/:ref: roles sat in markdown
cells, so the built HTML literally displayed :ref:`execution-model `` to
readers; two notebooks had no Colab badge; both SSH notebooks claimed the
widget appears on import clustrix, which is false; `ssh_key_automation`
documented the legacy widget rather than the one `%%remote` shows; and it
named a Colab secret that is never read. Colab rendering was verified for the
badged notebooks without signing in — so "renders in Colab", not "runs in
Colab".

Verification

Every check below was run on the final tree, in one cycle, after the last
change:

pytest tests/ -m "not real_world" --ignore=tests/real_world --ignore=tests/integration
    1244 passed, 18 skipped, 0 failed
    Required test coverage of 66.0% reached. Total coverage: 67.81%

black --check clustrix/ tests/ scripts/     224 files unchanged   (black 26.3.1, the pinned version)
flake8 clustrix/ tests/ scripts/            clean
mypy clustrix/                              no issues in 34 source files
python scripts/check_docs_examples.py       143/143 passed (110 executed for real)
python -m sphinx -b html docs/source ... -W  build succeeded, zero warnings

Nothing under tests/real_world/ or tests/integration/ was executed —
tests/integration/ provisions real billable AWS resources. Both were checked
with --collect-only: 158 and 37 collected, no import errors.

All 15 CI checks pass on the branch tip, including both Windows jobs, all
three Ubuntu versions, both macOS versions, and the new documentation-examples
step (confirmed as success, not skipped).

A note on the history

07db2e6 is a deliberate WIP checkpoint whose tree does not import — it
was committed so work would survive a machine suspend, and it is labelled
DO NOT MERGE. The branch tip is clean and passes everything above. Squash on
merge, or the broken commit stays bisectable.

🤖 Generated with Claude Code

https://claude.ai/code/session_012gTBDPK16HUZ3kHQ2QyjuU