test(gateway): run the integration suites under docker compose - #1147
Merged
Conversation
KMS stamps the app_id it verified into the certificate it returns. The local CA branch has the same value available -- the CSR carries the attestation it was derived from -- but dropped it, so every certificate issued through a local CA came back without the extension. A peer that pins app_id therefore rejected all of them. dstack-gateway's cluster mTLS does exactly that, which left clustering working under a KMS key provider and silently broken under every other one. The decode is best effort on purpose: an app whose attestation carries no app-id event decodes to an empty value, and stamping that would make every such peer match every other. Absent stays absent, and the peer rejects it as it did before.
The sync routes are the cluster's write surface: anything that reaches them can insert entries that replicate to every gateway. `verify_gateway_peer` is the only thing in front of them, and no test had ever executed it -- every test reached the body below by turning the check off, and Rocket's local client speaks no TLS, so it can never present the certificate the check wants. Replacing the whole function body with `Ok(())` did not turn the suite red. `handle_sync` and `handle_push` now hold everything the routes do once the caller is known to be a peer, so the tests that are about the gzip framing, the store split, the uuid check and the removed-sender refusal call them directly, and reach that code without going near the check. That frees the check to be tested for what it is: a request through the real route, with no certificate, is refused. One test does present a certificate, over a real mutually authenticated TLS connection, so the accepting half is covered too.
The suites that follow all need the same two things: a gateway container built from the tree, and a guest agent whose quotes verify without TDX hardware. Both were reachable only from the e2e directory. The simulator signs its quotes under trust anchors derived from a seed, and the collateral service reconstructs the matching public roots from that same seed. The two are useless apart and useless with different seeds, so they move into one `attestation/fixture.yml` that a suite includes, rather than being copied per suite where they would drift and leave peer quotes failing to verify for no visible reason. The fixture names no network, so each suite attaches it to its own. `build-gateway-image.sh` builds the static musl binary and wraps it in the alpine runtime image, so two suites cannot end up testing different builds.
`test_suite.sh` ran three gateways as host processes, which is why it was never wired into CI and why it rotted: it wanted host WireGuard, host ports and a host-installed simulator, and nothing checked that any of it still worked. Its 28 tests move into `cluster/tests.sh`, driven from the host against containers. Each test gets its own compose project on a network joined to the long-lived attestation fixture, so a test that leaves a node wedged cannot reach the next one, and the fixture is built once rather than per test. `cluster.sh` goes with it, with nothing harvested. Its one distinct command registered a CVM and printed peer counts; `test_cross_node_data_sync` already registers a CVM and asserts it reaches the other node in both the KvStore and the ProxyState view, and that the two views agree. That is the same ground, asserted instead of printed. Three source comments cited the two deleted scripts for the address shapes a deployment can take. The shapes are unchanged, so they now cite the suites.
`test_certbot.sh` could not be containerised without keeping what made it unrunnable: it needs a real Let's Encrypt staging account and real Cloudflare credentials, so it was never in CI and could only be run by hand, by someone holding both. Its four assertions -- an account is created, an order completes, the certificate lands, a renewal replaces it -- are made more strongly by e2e phases 5 and 6, against Pebble and the mock DNS API, across three nodes rather than one, and on every push.
The suite ran on the host and asked for sudo twice: to create the link the gateway wants at startup, and to `rmmod tls`. It now runs in one container that holds the origin, the probe and the gateway in a single network namespace, which `insecure_localhost_backend` requires -- it resolves an app address to 127.0.0.1, and that has to be the same 127.0.0.1 the origin is on. The suite's ~25 gateway restarts stay inside that container rather than becoming compose lifecycle, so a restart costs what it did before. NET_ADMIN in the container's own namespace replaces the first sudo, and nothing it creates outlives the run. The second is replaced by a seccomp profile that makes `setsockopt(IPPROTO_TCP, TCP_ULP)` return ENOPROTOOPT, which is exactly what `probe_ktls` sees on a kernel built without CONFIG_TLS. `rmmod tls` took the module from the whole host, needed passwordless sudo, and skipped silently whenever anything else on the machine held it -- so the arm that guards against a gated offload returning HTTP 200 with a truncated body mostly did not run. A profile is fixed at container creation, so that arm gets its own container and the main run skips it.
The three e2e gateways built their own simulator and their own copy of the mock collateral service. They now reference the shared fixture's network and volumes, so there is one seed signing quotes and one set of roots derived from it, and the two cannot drift apart. `run-e2e.sh` owns the fixture's lifetime, which is why the gateways can no longer `depends_on` it. With that in place the gateways verify each other's quotes for real: `insecure_allow_external_trust_anchors` lets the anchor come from outside the vendor set, and every check in front of it runs the production path. Two fixes the run needed: `log_*` writes to stderr, so a helper whose stdout is captured no longer returns its log lines to the caller as data. Adding three ZT domains in a loop left two without certificates: the first domain on a fresh deployment creates the global ACME account and holds the shared lock while it does, and a domain added in that window is refused with "retry after it finishes" -- which nothing does. Retried here, as the error asks. The retry belongs in the product, not the harness; the suite is not the right place to fix a race it only reveals.
Only the proxy suite was in CI. The cluster suite and the e2e run were not, which is why the e2e rotted unnoticed for months and why the cluster suite still assumed a host it no longer got. Each workflow builds the gateway image, brings the attestation fixture up, runs its suite and tears everything down in a step that runs on failure too. Paths are scoped so a change to one component does not run all three. The proxy job's timeout goes from 30 to 45 minutes: it now builds the fixture images, which compile Rust from a cold docker cache on every run, and the cargo cache does not reach inside a docker build.
TESTING.md described host processes, manual gateway startup and a certbot suite that no longer exists. It now describes what is there: three suites, what each covers, how to run one, and why they share a single attestation fixture. `.env.example` held Cloudflare credentials for `test_certbot.sh` and has nothing left to configure. The shellcheck exclude list loses the three deleted scripts. The replacements are not exempt; `e2e/run-e2e.sh` keeps its entry, unchanged.
… suite
Every crate that needs a certificate to drive a TLS test built one out of raw
`rcgen`: generate a key, set `IsCa::Ca(BasicConstraints::Unconstrained)`,
self-sign, write three PEM files. The same twenty lines are in the gateway's
sync tests, its HTTPS client tests, and `ra-rpc`'s client-auth tests, each
slightly different, and none of them found `CertRequest::ca_level()` -- which
has done the CA half since it was added.
`wavekv_sync`'s copy shows what that costs. Because it went around
`CertRequest`, it could not use `.app_id()` either, so it hand-encoded the DER
OCTET STRING for the extension:
debug_assert!(TEST_APP_ID.len() < 128);
let mut app_id_der = vec![0x04, TEST_APP_ID.len() as u8];
Three bytes of ASN.1 header, a `debug_assert` pinning the short-form length,
and a comment explaining why -- to set a field the builder already sets.
Added `ra_tls::test_pki` behind an off-by-default `test-pki` feature:
- `TestCa` -- a self-signed CA, `ca_level(0)` since a test CA has no reason to
mint intermediates.
- `TestCert` -- a leaf to mint, self-signed or CA-signed, with `.app_id()`,
alt names and usage flags. `TestCert::localhost()` is the shape local TLS
tests want: valid for `127.0.0.1`, usable at both ends of an mTLS connection.
- `write_mtls_pki` -- CA plus one leaf, written as the `node.crt`/`node.key`/
`ca.crt` trio a client config points at. Returns both, so a test needing a
second peer can sign one under the same CA.
Nothing here needs a TEE: the app_id extension is an ordinary X.509 extension
and a peer check that compares app ids never looks at a quote. Material that
does carry attestation still goes through `generate_ra_cert_with_app_id`.
The feature is off by default and reaches the gateway as a dev-dependency on
the crate its normal dependency already names, so it is enabled for tests and
not for the binary. `cargo tree --no-dev-dependencies` shows `ra-tls` with
`default` only; with dev-dependencies it shows `test-pki`.
Migrated `wavekv_sync`'s three helpers: -61/+18 lines, hand-encoded DER gone.
`https_client`'s copies are left for the branch that already rewrites them.
ra-tls 35 passed (31 + 4 new), dstack-gateway 308 passed, clippy and fmt clean,
and `cargo build -p dstack-gateway` still builds without the feature.
Node state is a bind mount, so `compose down -v` does not remove it, and the gateway writes its certificates 0600 as root, so the host user cannot either. Test names are fixed. Every run after the first therefore resumed each test on the store the same test left behind -- which is the opposite of what `docker-compose.yml` and `tests.sh` say happens. Measured, not reasoned about: with `setup_peers` stubbed so nodes never learn about each other, `test_cross_node_data_sync` passed on a second run and the suite exited 0. Node 2 found its instance already in the snapshot from the run before. With the wipe in place the same mutation fails with `kv2=0`. The same shape reaches `test_partial_cluster_bootstrap`, `test_delayed_bootnode_recovery` and the `keys_after_write > 0` guard in `test_persistence`. The clear-out runs in a throwaway root container, the way the proxy suite already does it, and removes contents only: `run` is what the suite's lock is held on, and replacing the directory would detach that lock exactly the way a deleted lock file does. Teardown is fixed alongside, because it is what leaves the state behind. `compose down` with CURRENT_TEST unset addresses `cluster-suite`, which no test ever uses -- the runner sets CURRENT_TEST before the first one -- so the `down` subcommand tore down nothing, and CI's log collection ran against a `cluster` project that has no containers. Both now ask the daemon which projects came from this compose file, so a killed run's leftovers are found whatever they are called.
…ount `wait_for_instances` and `wait_for_digest_match` ran `timeout_seconds * 10` iterations of "probe, then sleep 0.1", on the assumption that an iteration costs 0.1s. Each probe forks a curl and a python3, so it costs about half again as much: a nominal 3s window measured 4.5s, and the two-probe digest loop is worse. That is not a rounding error. `test_push_fast_path` passes 3 to prove a push arrived before the 5s periodic sync could have done it anyway; the real window was ~4.8s against a 5s interval, so a dead push path was caught by the periodic round and the test still went green. Its own comment describes a bound the code did not implement. The slack cut both ways. `test_node_id_reuse_rejected` asked for 15s and passed only because it was really getting twice that: recovery there needs the rejected node's identity record to arrive in a sync *response* -- its own requests still fail the peer's inbound check -- and then an anti-entropy round to carry the store. Measured three times on an idle machine: 16s, 17s, 18s. With the timer made honest it passed one full run and failed the next, so the constant is raised to 40 with the measurement recorded next to it. `wait_for_debug` counted `sleep 1`s past a `docker compose port` and a curl, and is fixed the same way.
Four places where a failure of the harness read as a pass of the test. `no_peer_was_rejected` is `! grep -q` over a file `dump_log` produces with `|| true`, and `! grep -q` on an empty file is true -- so it passed whenever the log could not be collected at all, which is the failure a negative assertion is most exposed to. It now requires the log to be non-empty first. `wipe_data_keeping_uuid` short-circuits its `rm` if `node_uuid` is not there, and errexit is off inside a test body -- they run as the condition of an `if` -- so the unchecked call let `test_bootstrap_after_data_dir_loss` "recover" a store that was never lost. Both wipes now fail loudly and both call sites check them. `--only` with a name that is not in ALL_TESTS selected nothing, left TESTS_FAILED at 0 and exited 0: a green run of zero tests.
All three suites decide whether to tear the shared attestation fixture down by whether they were the ones who started it. All three read that *after* installing the EXIT trap and next to the `up`, with a multi-minute image build in between -- so any failure in that window ran cleanup with the variable unset, took the `:-0` default, concluded it had started the fixture, and removed one another suite was using. That is the "network dstack-attestation declared as external, but could not be found" the comment there describes. Reading it before the trap is armed closes the window. The `down` subcommands also take the fixture now: it is a project of its own, so a suite-only teardown left it and its globally-named network and volumes on any runner that outlives the job.
`set -e` is on, so `compose run --rm proxy-tests` followed by `MAIN_RC=$?` could only ever record 0 -- the script had already exited -- and the final `[ $MAIN_RC -eq 0 ] && [ $NOTLS_RC -eq 0 ]` was a tautology. The exit status still propagated, but the second arm was skipped on exactly the runs where something was already wrong, and that arm is the only place the kTLS truncation regression is covered. run-e2e.sh in this same series has the correct form. Two things that made a failure undiagnosable go with it: Both arms ran the whole suite against one `WORK=/work`, so the second overwrote every `gw-<label>.log` the first wrote and the artifact for a main-arm failure showed the no-ULP arm's logs. Each arm gets its own subdirectory. CI collected logs with `docker compose logs`, which cannot work here: the suite is driven with `compose run --rm`, so compose ignores `container_name`, names the containers `<project>-<service>-run-<hash>` and deletes them on exit, and the script's trap has already run `compose down -v`. The `container_name` keys were dead config and are removed. What the suite writes into the bind-mounted work directory is root-owned, so it is made host-readable before exiting and the artifact is scoped to the logs -- the work directory also holds each arm's throwaway TLS key. The workflow also modprobes wireguard now. Without it the suite falls back to a dummy link and skips `test_accel_status` entirely, including the only positive proof that splice engaged; the other two workflows already did this.
The three suites share one `dstack-fixture` project and one set of globally
named docker resources -- `dstack-attestation`, its two volumes, the
`dstack-gateway:test` tag. All three trigger on `dstack/gateway/**`, so one
gateway PR launches all three at once, and with `runs-on: ${{ vars.CI_RUNNER }}`
they can land on the same machine and tear down each other's fixture. None of
them declared a concurrency group. They now share one.
`sdk/simulator/**` is added to all three: `Dockerfile.simulator` bakes
`app-compose.json`, `appkeys.json`, `sys-config.json` and `attestation.bin` into
the image every suite depends on, so a change there breaks all three and
triggered none. The proxy workflow gains the simulator and mock-attestation
paths for the same reason -- it runs the gateway with attestation on now, so it
depends on the fixture too.
run-e2e.sh heredocked its own byte-identical `Dockerfile.gateway`, built from it and `rm`ed it afterwards. That defeats what `build-gateway-image.sh` exists for -- "shared so the two suites cannot end up testing different builds" -- and the `rm` was neither conditional nor trapped, so a build that failed under `set -e` left an untracked `e2e/Dockerfile.gateway` behind, which `.gitignore` does not cover. With the suite now clean under shellcheck, it comes off the exclude list in prek.toml; the two SC2317s left are trap handlers shellcheck cannot see.
…cates cert-client had no tests at all, and the app-id stamp is the one production change in this series. Its only coverage was the gateway suites, which are path filtered and heavyweight, and which exercise one consumer of the extension. The derivation moves into a named function so it can be asserted on directly, and carries the reasoning that was in the call site: why reading an unverified attestation is sound here and only here, and which consumers actually broke. (`AppIdValidator` in the sync client and `ensure_from_gateway` read the extension with no fallback; the inbound sync routes fall back to app-info, which is why the break showed on one side of the connection only.) The tests run through the real `CertRequestClient::Local` branch against `sdk/simulator/attestation.bin` -- the same fixture the gateway suites use -- and read the extension back off the leaf the way a peer does. Both mutations they exist to catch were run: passing `None` as before the fix turns the round-trip red, and dropping the empty-app-id filter turns the absence tests red. The empty case needs an attestation that decodes *and* yields nothing, so it clears the app-id event's payload rather than reusing the undecodable one -- which fails earlier, at `.ok()`, and never reaches the filter.
…ence `simulator.toml` pointed at `configs/tee-simulator.json`, a path that does not exist -- and that pointer is the only thing keeping the two copies of the seed from drifting, which is the failure the fixture was consolidated to prevent. TESTING.md's performance section documented a `wg` wrapper and `/bench` endpoints that are not in the tree; nothing under `test-run/proxy/` serves them. It is kept as the record of the one manual run it came from, and says so, rather than reading as a procedure someone could follow. `test_proxy.sh` pointed at `test_suite.sh`, which this series deletes. `dump_log`'s doc block described a `$2` window the function does not take and its own body says it does not need. The proxy README now says what the seccomp profile costs: `defaultAction` is `SCMP_ACT_ALLOW` and `security_opt: seccomp=` replaces Docker's default profile rather than extending it, so the two arms are not syscall-equivalent.
`assert_no_insecure_shortcuts` lists the development trust settings that must never reach a rendered production manifest. The gateway suites now enable `insecure_allow_external_trust_anchors` -- correctly, since verifying real TDX collateral off TDX hardware is not possible -- which puts it in exactly the same class as the `insecure_skip_attestation` already on the list.
`authorize_peer` fell back to the app-info extension when app-id was absent, which made it the only one of the cluster's three identity checks that would accept such a certificate. `AppIdValidator` in the sync client and `ensure_from_gateway` in the RPC handler both read app-id with no fallback, so a certificate the fallback rescued here was still rejected in both other places -- one rule that disagreed with its neighbours rather than a second path worth having. The fallback existed because certificates issued through a local CA carried app-info but not app-id, which is fixed where it is caused now: `local_ca_app_id` in cert-client stamps the app id the way KMS does. Removing the fallback before that fix would have broken clustering under every non-KMS key provider. Deleting the branch left all 308 gateway tests green, so it had no coverage of its own either. `app_info_alone_does_not_authorize_a_peer` pins the rule now; restoring the fallback turns it red. It needs a certificate carrying app info and no app id, which `TestCert` could not mint, so `test_pki` grows an `app_info` builder alongside `app_id`.
`advisory_dns_wait` exists only for the clamp and had two tests; the budget arithmetic in the poll loop -- which the loop's own comment describes as the rule that keeps a short renewal from overrunning into its timeout -- had none, because reaching it needs a resolver and a live challenge. `dns_poll_sleep` splits that arithmetic out the way `handle_sync` and `authorize_peer` were split out elsewhere in this series, and the two halves are now covered together: the boundary at exactly half the renewal budget, the stock defaults on both the gateway and CLI sides, a zero budget, a zero renewal timeout, and the two invariants over a grid rather than at points. Four mutations were run, each caught by three tests: dropping the clamp, sleeping the full backoff regardless of the budget (the overshoot the loop comment describes), changing the share from a half to the whole, and sleeping a millisecond when the budget is spent. The zero case is a real configuration, not a degenerate one -- it is what the gateway e2e suite ran with while Pebble was not validating challenges, and it means the pre-order self-check is skipped outright rather than run once.
242s of unconditional sleep across 42 sites, down to 29s across 19. Two full suite runs went from ~12 minutes to ~6. The dominant shape was `setup_peers`, then a flat 4-20s. Sleeping is both slower and weaker than waiting: a blind settle cannot tell "peering formed in 800ms" from "peering never formed and the assertion below is about to test something else". `wait_for_peers` asserts it instead, keeping each site's original timeout as the ceiling. What that cost, recorded because it is the trap in this kind of change: the first version waited on `peer_addrs` alone, which lands as soon as SetNodeUrl does, while the settle it replaced also covered the sync round that carries the `nodes` records. `test_multi_node_sync` and `test_node_id_reuse_rejected` assert both and failed. The condition has to cover everything the sleep covered, not the one thing whose name matches -- so `wait_for_peers` requires both, and an audit of all 19 sites against what each one asserts turned up three more of the same shape: `test_network_partition` needs the rejoining node's catch-up, and the two `_*_converged` predicates have to include the `kv == ps` equality their callers assert or the wait can exit on an intermediate state the assertion then rejects. Both suite runs after that: 40 passed, 0 failed.
The workspace pins a channel plus rustfmt, clippy, rust-analyzer and three targets that the base image does not carry and this build does not need, so the first cargo invocation makes rustup download and install a complete toolchain. That step sat after `COPY . .`, which is the whole repo -- so any change anywhere redid it, in both fixture images, on every push, in all three gateway workflows. It is ~27s per image and it was the bulk of the cost. Measured on an unrelated one-line change: cargo itself reported `Finished in 2.48s`, inside a 29.6s step. Copying the toolchain file onto a layer of its own keys that work on the pin rather than on the source tree. The cargo registry, git and target directories become BuildKit cache mounts, which survive the `COPY . .` invalidation and are shared by both images, so whichever builds second reuses the first's artifacts. The binary is copied out inside the RUN because a cache mount is not part of the resulting layer. Same machine, same kind of change: unrelated file 1m57s -> 4.2s real dependency 45.6s, recompiling six crates The second number is the check that the first is not a false cache hit.
kvinwang
force-pushed
the
feat/gateway-e2e-compose
branch
from
August 27, 2026 06:13
b11e22c to
cee4fa5
Compare
The hook ran without `-x`, so a `# shellcheck source=` directive was inert and every `source` of a sibling raised SC1091. It is only an info, but shellcheck still exits non-zero on it, so the cluster suite's driver -- which sources the three files next to it -- could not pass. `-P SCRIPTDIR` is the half that matters: `-x` alone resolves the directive against the working directory, and prek runs from the repository root. Verified to add nothing: across the 95 scripts this hook covers, with both flags shellcheck reports no findings at all, where without them it reports those SC1091s and nothing else.
The three suites shared one compose project and one set of globally named
resources -- `dstack-attestation` and its two volumes -- on the reasoning that
the seed signing the quotes and the seed deriving the verifying roots must not
drift apart. They cannot: both come from files in `attestation/`, so separate
instances started from them agree by construction. Sharing the running instance
guaranteed nothing extra and cost two things.
The first is a teardown race. Each suite tears the fixture down if it was the
one that started it, which is right as far as it goes, but the suite that
started it still tears it down when it finishes -- taking the simulator socket
out from under whichever suites joined in the meantime.
The second is that it made the three CI workflows mutually exclusive, and the
attempt to serialise them dropped one instead; see the workflow change that
follows.
`FIXTURE_NS` gives each suite its own project, network and volumes. It is
`${FIXTURE_NS:?}`, not a default, so a caller that forgets it gets an error
naming the variable rather than silently reusing another suite's fixture.
Checked by running the cluster and proxy suites at the same time, which the
shared fixture made impossible: 40 passed / 0 failed and both proxy arms at
exit 0, concurrently.
A group shared across the three gateway workflows does not serialise them, it drops one. A concurrency group holds a single *pending* run, and `cancel-in-progress: false` protects only the run already executing -- so when the third workflow queued behind the first two, the one already waiting was cancelled. On the first push carrying that config, `Gateway cluster tests` was cancelled outright: reported as a cancelled run rather than a failure, and easy to read as noise. Nothing needs serialising now that each suite brings up its own fixture under its own `FIXTURE_NS`. Each workflow gets its own group with the ordinary meaning of `cancel-in-progress`: a new push supersedes the run for the commit it replaced. The fixture log step is fixed alongside -- it addressed `-p dstack-fixture`, which no longer exists.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #1132. Review the last 9 commits.
Problem
Four gateway integration suites existed; one was in CI.
test_suite.sh(28 tests, WaveKV replication and cluster identity) ran threegateways as host processes and needed host WireGuard, host ports and a
host-installed simulator.
test_certbot.shneeded a real Let's Encryptstaging account and real Cloudflare credentials. The docker e2e run was in
compose but not in CI.
test_proxy.shwas in CI, and asked forsudotwice.Nothing checked that any of the three uncovered ones still worked, and they
had stopped:
server refuses, behind a
|| true. 22 of its assertions had been failingfor long enough to read as the environment.
test_suite.shassumed a host it no longer got.All three also turned attestation off to run at all, which meant the cluster
mTLS path — the thing
test_suite.shexists to exercise — was never executedby any of them.
Fix
Three suites, all under docker compose, all driven from the host, none needing
root, none leaving anything behind:
cluster/test_suite.sh+cluster.sh, host processesproxy-e2e/test_proxy.shon the host, with sudoe2e/dns-persist-01Each test in
cluster/gets its own compose project, so a test that leaves anode wedged cannot reach the next one. All three reference one long-lived
attestation fixture — one simulator, one collateral service, one seed — rather
than each building a copy, because the seed that signs quotes and the seed
that derives the verifying roots must not drift apart.
The gateways now verify each other's quotes for real.
insecure_skip_attestationis no longer set anywhere; deleting it is the next PR in the stack.
Two things had to be fixed for that to work:
cert-clientdropped the app_id on locally issued certificates. KMSstamps in the app_id it verified; the local CA had the same value from the
CSR's attestation and threw it away. A peer that pins app_id therefore
rejected every locally issued certificate, so gateway clustering worked
under a KMS key provider and was silently broken under every other one.
verify_gateway_peerhad never been executed by a test. Every testreached the body below it by turning the check off, and Rocket's local
client speaks no TLS so it can never present a certificate. Replacing the
whole function body with
Ok(())did not turn the suite red. The handlersare split from the routes, so those tests call the handler directly and the
check is tested for what it is.
sudo rmmod tlsis replaced by a seccomp profile that makessetsockopt(IPPROTO_TCP, TCP_ULP)return ENOPROTOOPT — exactly whatprobe_ktlssees on a kernel built withoutCONFIG_TLS. The old form tookthe module from the whole host and skipped silently whenever anything else
held it, so the arm guarding against a gated offload returning HTTP 200 with a
truncated body mostly did not run.
test_certbot.shis deleted rather than ported: its four assertions are mademore strongly by e2e phases 5 and 6, against Pebble and the mock DNS API,
across three nodes rather than one, and on every push.
cluster.shis deletedwith nothing harvested — its one distinct command printed peer counts where
test_cross_node_data_syncasserts them.Verification
All three suites, run to completion on a clean host, one instance each:
cluster/e2e/proxy-e2e/cargo test -p dstack-gateway --all-features: 308 passed / 0 failed.cargo fmt --checkclean; all seven shell scripts passbash -nandshellcheck (the three deleted scripts come off the shellcheck exclude list;
the replacements are not exempt).
A suite that reports green without running anything is the failure mode that
matters here, so the 28 ported tests were checked against two mutations of the
code under test, with the expected outcome written down before each run:
setup_peers, so nodes never learn about each other):27 of 28 matched the prediction. The one that did not,
test_partial_cluster_bootstrap, was a real defect in the predicate, nowfixed.
Five tests cannot be reached by either mutation. Their assertions were read
line by line, which is weaker evidence than the other 23 have, and it turned
up three empty assertions — conditions polled in a loop and then never
asserted on. Fixed.
Known gaps
the first holds the global ACME account lock while creating the account, and
a domain added inside that window is refused with "retry after it finishes"
— which nothing does. Worked around with a retry in the harness only. This
belongs in the product; I can open an issue.
actionlintinprek.tomlwould cover them; not added here.