From f0022099c56aaddd6b33585905dedcbdfac09b95 Mon Sep 17 00:00:00 2001 From: Mark Rousskov Date: Sat, 5 Sep 2026 11:07:58 -0400 Subject: [PATCH] Document AWS GHA runners --- src/SUMMARY.md | 1 + src/infra/docs/ci-runners-aws.md | 142 +++++++++++++++++++++++++++++++ 2 files changed, 143 insertions(+) create mode 100644 src/infra/docs/ci-runners-aws.md diff --git a/src/SUMMARY.md b/src/SUMMARY.md index 46f1a610b..3628f53e1 100644 --- a/src/SUMMARY.md +++ b/src/SUMMARY.md @@ -108,6 +108,7 @@ - [CDN](./infra/docs/cdn.md) - [Crater agents](./infra/docs/crater-agents.md) - [Dev Desktops](./infra/docs/dev-desktop.md) + - [CI Runners on AWS](./infra/docs/ci-runners-aws.md) - [External CI Runners](./infra/docs/external-ci-runners.md) - [GitHub App for dev-desktops](./infra/docs/dev-desktop-github-app.md) - [Domain names and DNS](./infra/docs/dns.md) diff --git a/src/infra/docs/ci-runners-aws.md b/src/infra/docs/ci-runners-aws.md new file mode 100644 index 000000000..7e3d4fd45 --- /dev/null +++ b/src/infra/docs/ci-runners-aws.md @@ -0,0 +1,142 @@ +# CI runners on AWS + +We manage self-hosted GitHub actions runners on AWS compute today. This page +describes the moving pieces we've built at a high level to help someone +navigate where to go if something is broken. + +## Bors + +Bors is responsible for listening for GitHub-sent webhooks and spinning up EC2 +instances on which to run the jobs. It will also spin up instances if it +detects a job is pending. It also handles last-resort termination if an +instance doesn't self-terminate after hitting a systemd timeout (e.g., if the +instance OOMs the wrong thing this may not happen). + +Bors is configured via `rust-bors.toml` in each repository (currently +rust-lang/rust and rust-lang/bors-kindergarten). + +## GitHub configuration + +We have configured two runner groups at the organization level. bors issues a +one time use "JIT" token via its GitHub app associated with these groups, which +is passed into the launched instance. The GitHub-owned runner code uses this +token to authenticate and authorize access to executing jobs. These groups +currently allow executing from a subset of workflows and branches in their +respective repositories, but note that PRs targeting those branches will be +allowed to run (at the GitHub layer). + +Note the settings URLs are not accessible to non-infra-admins today: + +* [rust-lang/rust runner group](https://github.com/organizations/rust-lang/settings/actions/runner-groups/8) + * +* [rust-lang/bors-kindergarten runner group](https://github.com/organizations/rust-lang/settings/actions/runner-groups/11) + * + +## AMIs + +Each repository owns configuring both the instance type and the AMI (disk +image) to run on the instance via `rust-bors.toml`. The AMI is indirectly +specified as an SSM parameter name. Those SSM parameters currently correspond +to AMIs we build ourselves. + +We build AMIs daily from a AWS-cron-job, invoking a [Lambda function](https://github.com/rust-lang/simpleinfra/tree/master/terragrunt/modules/ci-runners/lambda). + +The Lambda function takes as input a JSON blob with a single key (`arch`), +either `aarch64` or `x86_64` today. It performs the following tasks (in this +order): + +1. Downloads a packer release, verifying the hash against a hard-coded checksum +1. Checks if the most recent AMI is more than 2 days old. If yes, we post to Zulip (#t-infra/private > ci-runner-build-notifications). + * This is a simple alerting mechanism that lets us know if the builds stop working. +1. Calls GitHub API (unauthenticated) to get latest actions/runner release version and download URL. +1. Invokes packer to build according to the contained [template], specifying the architecture and GitHub URL. +1. Once build is complete, we persist new AMI ID to an SSM parameter based on + the architecture we built. This will be used immediately on next EC2 runner + start in the relevant repository (ci-staging - bors-kindergarten, ci-prod - + rust-lang/rust). + * Note that there's no validation today that the new AMI "works", and each + account builds AMIs entirely separately. So far we haven't hit issues with + this approach. + * If CI breaks after an AMI release, rolling back the SSM version by + specifying the previous AMI ID (look it up in the EC2 console) should be + an easy hot fix. +1. After a successful build we additionally delete AMIs >14 days old. + +Pre-building AMIs is done so that we: + +* Have fast instance start-up (dependencies are all installed, we start executing within <1 minute after GitHub creates the job) +* Can customize instance configuration, including boot-time configuration + * For example, we tune the mount options for `/` for (theoretically) faster builds +* In theory, easy-ish ability to rollback an update to the base image. In + practice, this still requires infra intervention so it's not perfect. +* Give a convenient hook point to fetch latest runner installation. + * See [docs](https://docs.github.com/en/actions/reference/runners/self-hosted-runners#runner-software-updates-on-self-hosted-runners). + If we don't do this we'd start seeing self-updates and/or breakage roughly + 30 days after a new release, which isn't ideal. + +[template]: https://github.com/rust-lang/simpleinfra/blob/master/terragrunt/modules/ci-runners/lambda/ubuntu.pkr.hcl + +## Security + +### Hostile code within the runner + +We aim to consider runners 'fully untrusted'. We have these protections in place: + +1. EC2 instances only execute one GitHub Actions run/job. They get access via a + JIT token, which GitHub promises can only be used once. Caveat: today there + is only weak enforcement *which* job is executed on the instance. In the + happy path, the JIT token embeds the GHA run + workflow job ID and we + configure CI to execute on that particular label. But there's no enforcement + available in the GitHub control plane to uniquely bind a JIT token to a + particular workflow. There's no real risk from executing on the wrong + instance though. +1. The EC2 instance is launched with no associated IAM role (enforced by IAM + permissions -- bors can't associate a role even if it wanted to). The only + secret present on the instance at startup is the JIT token, but it is used + up by calling GitHub before any code on the instance starts running. +1. Instances currently have zero inbound network access (enforced by EC2 security + group), and outbound access to port 80 and 443 anywhere on the internet. + Restricting outbound access to a limited set of destinations would be ideal, + but is complicated. Today GitHub requires SNI/DNS allowlisting rather than IP + filtering, see [docs](https://docs.github.com/en/actions/reference/runners/self-hosted-runners#communication). + +Note that today, most of these runners end up accessing secrets from GitHub. So +while the AWS environment should generally be safe from compromised code in the +runners, that code can still do bad things. + +### Hostile actor in the account + +We have a weaker goal of defending against access into the runners while +they're running. The ci-prod account has Admin (infra-admins) + ReadOnly +(infra) [access](https://github.com/rust-lang/simpleinfra/blob/00a678c6ea843d87e345a9b896982bc0d1ef9d39/terragrunt/modules/aws-organization/groups.tf#L439) +today. The production hosts intentionally don't have remote access into them +(e.g., ssh) and have a limited install of Ubuntu (minimal base image, installed +just what is needed to run GHA) to limit risk. + +One known threat here is JIT tokens being exposed via EC2 User Data. These are +only useful for a short period of time between instance launch and the usage of +the token to GitHub. Hiding this token from users in the AWS account is +complicated. One possible solution is encrypting under a KMS key with a grant +issued to the EC2 instance's security credentials (which are always available). +This could also be used to prove that the instance has booted from an expected +prebuilt AMI (e.g., via [TPM attestation]). For now we think mitigating this is +not worth the added complexity. + +[TPM attestation]: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/nitrotpm-attestation.html + +## Policy compliance + +Our [external runner policy] is not fully met by these runners. Since we +control the compute, we are on the hook for (minimal) maintenance needed. + +Security compliance is partially met: + +* Access controls: CloudTrail provides a record of any attempts to access the compute. +* Ephemeral builds: fully ephemeral compute, one build/instance. +* Endpoint protection: none currently, but the instances are rebuilt daily from latest Ubuntu AMIs. + +[external runner policy]: https://github.com/rust-lang/infra-team/blob/main/service-catalog/rust-ci/external-runners/README.md#security-requirements-optional + +## Why not CodeBuild? + +We moved away from CodeBuild to speed up (and reduce costs) for CI. See [evaluation issue](https://github.com/rust-lang/simpleinfra/issues/1132) for additional context.