chore(gateway)!: remove insecure_skip_attestation

CHANGELOG.md

@@ -90,6 +90,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 ### Removed
 - sdk: `TlsKeyOptions.path` in the JavaScript SDK. `GetTlsKeyArgs` has no such field and `getTlsKey` never read it, so a caller who set it was silently ignored. Breaking at the type level only, and only for code whose value was already being discarded. `deriveKey`'s `path` is a real, deprecated Tappd-era parameter and stays; the Python, Rust and Go v0 TLS-key options never carried one
 - guest-agent: the `EmitEvent` RPC no longer records anything -- runtime RTMR3 events are system-owned in 0.6.0, so an app can no longer extend the measurement chain. The method itself stays on the unversioned path and always fails with an error naming the removal, rather than being deleted outright: a deleted method answers HTTP 404 `Service not found: EmitEvent`, which tells a 0.5.x caller nothing about why its events stopped being recorded, while the kept stub fails with a message naming the removal and pointing at `report_data`. **Breaking:** any app extending RTMR3 at runtime must stop; bind app data through `report_data` instead, which is what most callers wanted anyway
+- gateway: `core.debug.insecure_skip_attestation`. It turned off both checks that make a gateway cluster a trust boundary: the node stopped asking its guest agent for its own app id, and every WaveKV sync and push, plus `ensure_from_gateway`, stopped checking the peer's. With it set, anything that could reach the sync routes could insert entries that replicated to every gateway in the cluster. It existed only because the integration suites could not run without it; they now run against a guest agent simulator and verify quotes through the production path, so nothing sets it. `docs/security/security-model.md` argues dstack's development switches are acceptable because they are visible in attestation measurements or public contract state -- that argument cannot cover the switch deciding whether attestation happens at all, which is why this one is deleted rather than documented. There is no replacement. **Breaking:** the config struct does not use `deny_unknown_fields`, so a leftover line is ignored rather than rejected, and what follows depends on why it was set. On a TDX host with a guest agent the gateway starts normally and peers that cannot present a verifiable app id stop being accepted. On a host with no guest agent -- the usual reason to have set it -- the gateway no longer starts at all, failing with `Failed to get app info`, because `my_app_id` is what every peer check compares against and a node that cannot learn its own identity must not come up without one. Remove the line and make sure every node can attest
 
 
 ## [0.5.5] - 2025-10-20

docs/security/security-model.md

@@ -329,10 +329,12 @@ The one case dstack does not leave to downstream is a genuinely invalid TCB: `dc
 
 ### Development modes are auditable, not production-safe
 
-dstack keeps several development switches as runtime or on-chain configuration rather than Cargo feature flags. Examples include KMS `attest_rpc_cert = false`, gateway `core.debug.insecure_skip_attestation = true`, KMS `auth_api.type = "dev"`, and KMS contract `gateway_app_id = "any"`. These settings exist for local development and integration tests, not for production deployments.
+dstack keeps several development switches as runtime or on-chain configuration rather than Cargo feature flags. Examples include KMS `attest_rpc_cert = false`, KMS `auth_api.type = "dev"`, and KMS contract `gateway_app_id = "any"`. These settings exist for local development and integration tests, not for production deployments.
 
 This is intentional. Runtime configuration that affects the trust boundary is visible in attestation measurements or public contract state. Cargo feature gates are not automatically more auditable because feature unification can enable a feature through a dependency graph, and the resulting runtime behavior is not represented as a measured deployment setting.
 
+This argument has a limit, and it is worth stating because it is what keeps the list short. A switch qualifies only if the trust decision still happens and is merely recorded as a measured setting. A switch that decides *whether* attestation happens at all does not qualify: there is then no measurement to audit, because the thing that would have produced it was skipped. Gateway `core.debug.insecure_skip_attestation` was such a switch -- it turned off both the peer identity check on WaveKV sync and the gateway's own app id lookup -- and it was removed rather than documented.
+
 Production verifiers should reject deployments that use these development settings. Operators should treat them the same way they treat debug-mode TEE quotes: useful for testing, invalid for production trust.
 
 ### KMS mTLS is route-enforced for sensitive operations

dstack/gateway/docs/cluster-deployment.md

@@ -241,7 +241,6 @@ address = "0.0.0.0"
 
 [core.debug]
 insecure_enable_debug_rpc = true
-insecure_skip_attestation = false
 port = 9015
 address = "0.0.0.0"
 

dstack/gateway/gateway.toml

@@ -60,7 +60,6 @@ insecure_enable_debug_rpc = false
 # arbitrary custom domains, so enabling this lets any DNS zone owner reach the
 # gateway's own loopback on a port of their choosing, bypassing port_policy.
 insecure_localhost_backend = false
-insecure_skip_attestation = false
 address = "127.0.0.1:8012"
 
 [core.wg]

dstack/gateway/src/config.rs

@@ -639,8 +639,6 @@ pub struct DebugConfig {
     /// Enable debug server
     #[serde(default)]
     pub insecure_enable_debug_rpc: bool,
-    #[serde(default)]
-    pub insecure_skip_attestation: bool,
     /// Let the app-address `localhost` resolve to 127.0.0.1, so a hostname can
     /// be routed to a service on the gateway host itself.
     ///

dstack/gateway/src/kv/https_client.rs

@@ -53,14 +53,19 @@ pub trait CertValidator: Debug + Send + Sync + 'static {
     fn validate(&self, cert_der: &[u8]) -> Result<(), String>;
 }
 
-/// TLS configuration for mTLS with optional custom certificate validation
+/// TLS configuration for mTLS with custom certificate validation
 #[derive(Clone)]
 pub struct HttpsClientConfig {
     pub cert_path: String,
     pub key_path: String,
     pub ca_cert_path: String,
-    /// Optional custom certificate validator (checked during TLS handshake)
-    pub cert_validator: Option<Arc<dyn CertValidator>>,
+    /// Custom certificate validator, checked against the peer's certificate
+    /// during the handshake.
+    ///
+    /// Not optional: this is the only thing that distinguishes a gateway in our own
+    /// cluster from any other holder of a certificate the shared CA signed, and CA-path
+    /// validation alone does not make that distinction.
+    pub cert_validator: Arc<dyn CertValidator>,
 }
 
 /// Wrapper that adapts a CertValidator to rustls ServerCertVerifier
@@ -147,10 +152,10 @@ impl ServerCertVerifier for CustomCertVerifier {
 
 type HyperClient = Client<hyper_rustls::HttpsConnector<HttpConnector>, Full<Bytes>>;
 
-/// HTTPS client with mTLS and optional custom certificate validation.
+/// HTTPS client with mTLS and peer identity validation.
 ///
-/// When a `cert_validator` is set in `TlsConfig`, the client runs the validator
-/// during the TLS handshake, before any application data is sent.
+/// The `cert_validator` runs during the TLS handshake, before any application data is
+/// sent, on top of the chain verification against the configured CA.
 #[derive(Clone)]
 pub struct HttpsClient {
     client: HyperClient,
@@ -221,22 +226,17 @@ impl HttpsClient {
             .next()
             .context("no CA certificate found")?;
 
-        // Build rustls config with custom verifier if validator is provided
-        let tls_config_builder = rustls::ClientConfig::builder();
-
-        let tls_config = if let Some(ref validator) = tls.cert_validator {
-            let verifier = CustomCertVerifier::new(validator.clone(), ca_cert)?;
-            tls_config_builder
-                .dangerous()
-                .with_custom_certificate_verifier(Arc::new(verifier))
-        } else {
-            // Standard verification without custom validator
-            let mut root_store = rustls::RootCertStore::empty();
-            root_store.add(ca_cert).context("failed to add CA cert")?;
-            tls_config_builder.with_root_certificates(root_store)
-        }
-        .with_client_auth_cert(certs, key)
-        .context("failed to set client auth cert")?;
+        // `CustomCertVerifier` runs the validator *after* rustls has verified the chain
+        // against `ca_cert`, so this is the CA path plus an identity check, never a
+        // replacement for it. There is deliberately no validator-less branch: it would
+        // accept any certificate the CA signed, which is every gateway in every cluster
+        // that shares the CA.
+        let verifier = CustomCertVerifier::new(tls.cert_validator.clone(), ca_cert)?;
+        let tls_config = rustls::ClientConfig::builder()
+            .dangerous()
+            .with_custom_certificate_verifier(Arc::new(verifier))
+            .with_client_auth_cert(certs, key)
+            .context("failed to set client auth cert")?;
 
         let https = HttpsConnectorBuilder::new()
             .with_tls_config(tls_config)
@@ -433,39 +433,13 @@ mod transport_tests {
     use tokio::net::TcpListener;
     use tokio_rustls::TlsAcceptor;
 
+    /// The app id the HTTP-level tests below run under. They are not about identity,
+    /// but the validator is not optional, so their server and client agree on one.
+    const TEST_APP_ID: &[u8] = b"app-id-of-this-cluster";
+
     /// A CA plus a leaf valid for 127.0.0.1, written where `HttpsClient::new` expects.
     fn tls_material(dir: &std::path::Path) -> (HttpsClientConfig, Vec<u8>, Vec<u8>) {
-        use ra_tls::rcgen::{BasicConstraints, CertificateParams, IsCa, KeyPair};
-
-        let ca_key = KeyPair::generate().expect("ca key");
-        let mut ca_params = CertificateParams::new(vec![]).expect("ca params");
-        ca_params.is_ca = IsCa::Ca(BasicConstraints::Unconstrained);
-        let ca_cert = ca_params.self_signed(&ca_key).expect("ca cert");
-
-        let leaf_key = KeyPair::generate().expect("leaf key");
-        let leaf_params =
-            CertificateParams::new(vec!["127.0.0.1".to_string()]).expect("leaf params");
-        let leaf_cert = leaf_params
-            .signed_by(&leaf_key, &ca_cert, &ca_key)
-            .expect("leaf cert");
-
-        let cert_path = dir.join("node.crt");
-        let key_path = dir.join("node.key");
-        let ca_path = dir.join("ca.crt");
-        std::fs::write(&cert_path, leaf_cert.pem()).expect("write cert");
-        std::fs::write(&key_path, leaf_key.serialize_pem()).expect("write key");
-        std::fs::write(&ca_path, ca_cert.pem()).expect("write ca");
-
-        (
-            HttpsClientConfig {
-                cert_path: cert_path.to_string_lossy().into_owned(),
-                key_path: key_path.to_string_lossy().into_owned(),
-                ca_cert_path: ca_path.to_string_lossy().into_owned(),
-                cert_validator: None,
-            },
-            leaf_cert.der().to_vec(),
-            leaf_key.serialize_der(),
-        )
+        app_id_server_cert(dir, TEST_APP_ID)
     }
 
     /// Serve one fixed response over TLS and return the URL to reach it.
@@ -593,46 +567,32 @@ mod transport_tests {
     }
 
     /// A server certificate carrying an app id, signed by the same test CA.
+    ///
+    /// The client config that comes back validates against that same app id, which is
+    /// what the HTTP-level tests want. `a_peer_from_another_app_cannot_complete_the_handshake`
+    /// overrides `cert_validator` to make the two disagree.
     fn app_id_server_cert(
         dir: &std::path::Path,
         app_id: &[u8],
     ) -> (HttpsClientConfig, Vec<u8>, Vec<u8>) {
-        use ra_tls::cert::CertRequest;
-        use ra_tls::rcgen::{BasicConstraints, CertificateParams, IsCa, KeyPair};
-
-        let ca_key = KeyPair::generate().expect("ca key");
-        let mut ca_params = CertificateParams::new(vec![]).expect("ca params");
-        ca_params.is_ca = IsCa::Ca(BasicConstraints::Unconstrained);
-        let ca_cert = ca_params.self_signed(&ca_key).expect("ca cert");
-
-        let leaf_key = KeyPair::generate().expect("leaf key");
-        let alt_names = vec!["127.0.0.1".to_string()];
-        let leaf_cert = CertRequest::builder()
-            .key(&leaf_key)
-            .subject("peer.test")
-            .alt_names(&alt_names)
-            .app_id(app_id)
-            .usage_server_auth(true)
-            .build()
-            .signed_by(&ca_cert, &ca_key)
-            .expect("leaf cert");
-
-        let cert_path = dir.join("node.crt");
-        let key_path = dir.join("node.key");
-        let ca_path = dir.join("ca.crt");
-        std::fs::write(&cert_path, leaf_cert.pem()).expect("write cert");
-        std::fs::write(&key_path, leaf_key.serialize_pem()).expect("write key");
-        std::fs::write(&ca_path, ca_cert.pem()).expect("write ca");
+        let pki = ra_tls::test_pki::write_mtls_pki(
+            dir,
+            ra_tls::test_pki::TestCert::new("peer.test")
+                .alt_name("127.0.0.1")
+                .app_id(app_id)
+                .server_auth(true),
+        )
+        .expect("write test PKI");
 
         (
             HttpsClientConfig {
-                cert_path: cert_path.to_string_lossy().into_owned(),
-                key_path: key_path.to_string_lossy().into_owned(),
-                ca_cert_path: ca_path.to_string_lossy().into_owned(),
-                cert_validator: None,
+                cert_path: pki.cert_path.to_string_lossy().into_owned(),
+                key_path: pki.key_path.to_string_lossy().into_owned(),
+                ca_cert_path: pki.ca_cert_path.to_string_lossy().into_owned(),
+                cert_validator: Arc::new(AppIdValidator::new(app_id.to_vec())),
             },
-            leaf_cert.der().to_vec(),
-            leaf_key.serialize_der(),
+            pki.leaf.cert_der(),
+            pki.leaf.key_der(),
         )
     }
 
@@ -652,7 +612,7 @@ mod transport_tests {
         {
             let dir = tempfile::tempdir().expect("tempdir");
             let (mut config, cert, key) = app_id_server_cert(dir.path(), &server_app_id);
-            config.cert_validator = Some(Arc::new(AppIdValidator::new(ours.clone())));
+            config.cert_validator = Arc::new(AppIdValidator::new(ours.clone()));
             let url = serve(StatusCode::OK, gzip(b"response"), cert, key).await;
 
             let got = HttpsClient::new(&config)

dstack/gateway/src/main.rs

@@ -166,16 +166,15 @@ async fn main() -> Result<()> {
         set_max_ulimit()?;
     }
 
-    let my_app_id = if config.debug.insecure_skip_attestation {
-        None
-    } else {
-        let dstack_client = dstack_agent().context("Failed to create dstack client")?;
-        let info = dstack_client
-            .info()
-            .await
-            .context("Failed to get app info")?;
-        Some(info.app_id)
-    };
+    // Required, not best-effort: `my_app_id` is what every peer check compares against,
+    // so a gateway that cannot learn its own identity must not start rather than start
+    // without one. A host with no guest agent fails here.
+    let dstack_client = dstack_agent().context("Failed to create dstack client")?;
+    let my_app_id = dstack_client
+        .info()
+        .await
+        .context("Failed to get app info")?
+        .app_id;
     let proxy_config = config.proxy.clone();
     let attestation_verifier = Arc::new(
         AttestationVerifier::load(&config.attestation)

dstack/gateway/src/main_service.rs

@@ -69,7 +69,7 @@ pub struct ProxyInner {
     pub(crate) config: Arc<Config>,
     /// Multi-domain certbot (from KvStore DNS credentials and domain configs)
     pub(crate) certbot: Arc<DistributedCertBot>,
-    my_app_id: Option<Vec<u8>>,
+    my_app_id: Vec<u8>,
     state: Mutex<ProxyState>,
     pub(crate) notify_state_updated: Notify,
     auth_client: AuthClient,
@@ -136,7 +136,12 @@ pub(crate) struct ProxyState {
 /// Options for creating a Proxy instance
 pub struct ProxyOptions {
     pub config: Config,
-    pub my_app_id: Option<Vec<u8>>,
+    /// This gateway's own app id, from the guest agent.
+    ///
+    /// Not optional: it is the only thing `authorize_peer` and `AppIdValidator`
+    /// compare a peer against, so a gateway that does not know it cannot decide
+    /// who belongs in the cluster. `main` fails to start rather than construct one.
+    pub my_app_id: Vec<u8>,
     /// TLS configuration (from Rocket's tls config)
     pub tls_config: TlsConfig,
 }
@@ -372,9 +377,7 @@ impl ProxyInner {
         // Build HttpsClientConfig for mTLS communication
         let https_config = {
             let tls = &tls_config;
-            let cert_validator = my_app_id
-                .clone()
-                .map(|app_id| Arc::new(AppIdValidator::new(app_id)) as _);
+            let cert_validator = Arc::new(AppIdValidator::new(my_app_id.clone()));
             HttpsClientConfig {
                 cert_path: tls.certs.clone(),
                 key_path: tls.key.clone(),
@@ -552,8 +555,8 @@ impl ProxyInner {
         &self.kv_store
     }
 
-    pub(crate) fn my_app_id(&self) -> Option<&[u8]> {
-        self.my_app_id.as_deref()
+    pub(crate) fn my_app_id(&self) -> &[u8] {
+        &self.my_app_id
     }
 }
 
@@ -2668,13 +2671,10 @@ pub struct RpcHandler {
 
 impl RpcHandler {
     fn ensure_from_gateway(&self) -> Result<()> {
-        if self.state.config.debug.insecure_skip_attestation {
-            return Ok(());
-        }
         if self.remote_app_id.is_none() {
             bail!("Client authentication is required");
         }
-        if self.state.my_app_id != self.remote_app_id {
+        if Some(&self.state.my_app_id) != self.remote_app_id.as_ref() {
             bail!("Remote app id is not from dstack-gateway");
         }
         Ok(())

dstack/gateway/src/main_service/tests.rs

@@ -43,7 +43,7 @@ async fn create_test_state_with(tweak: impl FnOnce(&mut Config)) -> TestState {
     tweak(&mut config);
     let options = ProxyOptions {
         config,
-        my_app_id: None,
+        my_app_id: b"test-app-id".to_vec(),
         tls_config: TlsConfig {
             certs: "".to_string(),
             key: "".to_string(),