From 0b4c90c9e428970221ed03cef3c08f9982f82409 Mon Sep 17 00:00:00 2001 From: Richard Wall Date: Thu, 10 Sep 2026 11:49:04 +0000 Subject: [PATCH] [VC-59470] Start only the matching e2e suite when its label is added Adding test-e2e, test-ark or test-ngts to an open pull request currently does nothing. The workflow uses `on: pull_request: {}`, which takes the default activity types of opened, synchronize and reopened. There is no `labeled`, so the label sits on the pull request and no run starts. Re-running does not help either, because a re-run replays the original event payload, which had no labels. The only way through is to close and reopen the pull request, or push a commit. The failure mode is silent, so it looks like the job is broken. Move the three e2e jobs into a new e2e.yaml, which subscribes to `labeled` as well as the default types. Each one runs only when github.event.label.name, the single label that was just added, is its own label, so adding test-ngts does not start the ark or GKE suites. Keeping them in tests.yaml would have cost the verify and test results. A skipped job still publishes a check run, and GitHub shows the most recent one per job name, so a label event would have replaced their green results with "skipped". Splitting the file means a label event never creates a check run for verify or test at all. tests.yaml therefore keeps the default activity types and needs no job-level guards. Behaviour on push, on workflow_dispatch and on opened, synchronize and reopened is unchanged, so a suite whose label is already on the pull request still re-runs on every new commit. The e2e jobs never ran on push, so dropping that trigger from e2e.yaml changes nothing. Also document that keep-e2e-cluster must be added before test-e2e. The cleanup step reads the labels from the event payload, frozen when the run started, and test-e2e now starts the run immediately. Co-Authored-By: Claude Signed-off-by: Richard Wall --- .github/workflows/e2e.yaml | 195 +++++++++++++++++++++++++++++++++++ .github/workflows/tests.yaml | 159 +--------------------------- CONTRIBUTING.md | 4 + README.md | 4 +- RELEASE.md | 5 + 5 files changed, 210 insertions(+), 157 deletions(-) create mode 100644 .github/workflows/e2e.yaml diff --git a/.github/workflows/e2e.yaml b/.github/workflows/e2e.yaml new file mode 100644 index 00000000..55d778c5 --- /dev/null +++ b/.github/workflows/e2e.yaml @@ -0,0 +1,195 @@ +name: e2e +# Kept separate from tests.yaml so that adding a label cannot disturb the +# verify and test check runs. A skipped job still publishes a check run, and +# GitHub shows the most recent one per job name, so a label event in a workflow +# containing verify and test would replace their results with "skipped". +on: + # `labeled` is not one of the default activity types, so without it adding + # test-e2e, test-ark or test-ngts to an open pull request starts nothing, and + # re-running does not help because a re-run replays the original, unlabelled + # payload. + # Why?: https://docs.github.com/en/actions/reference/workflows-and-actions/events-that-trigger-workflows#pull_request + # > By default, a workflow only runs when a pull_request event's activity + # > type is opened, synchronize, or reopened. + pull_request: + types: [opened, synchronize, reopened, labeled] + # Lets us run the suites against master, which the label gates below cannot + # do: they read github.event.pull_request.labels, which is empty for a push. + # Needed before tagging a release. + workflow_dispatch: {} +jobs: + ark-test-e2e: + # TEMPORARY: require an explicit label to test disco-agent until the test environment fixes a recurring issue + # where the e2e fails with a 400 error relating to "conflicting tagging values" + # The test is flaky, not broken and re-running eventually makes it pass - but that delays progress on + # other unrelated work. + # Runs when the label is added, and thereafter on every push while it is + # still on the pull request. `github.event.label` names only the label that + # was just added, so adding one e2e label does not start the other suites. + if: >- + github.event_name == 'workflow_dispatch' + || github.event.label.name == 'test-ark' + || (github.event.action != 'labeled' + && contains(github.event.pull_request.labels.*.name, 'test-ark')) + runs-on: ubuntu-latest + timeout-minutes: 30 + steps: + - uses: actions/checkout@8e8c483db84b4bee98b60c0593521ed34d9990e8 # v6.0.1 + # Adding `fetch-depth: 0` makes sure tags are also fetched. We need + # the tags so `git describe` returns a valid version. + # see https://github.com/actions/checkout/issues/701 for extra info about this option + with: { fetch-depth: 0 } + + - uses: ./.github/actions/repo_access + with: + DEPLOY_KEY_READ_VENAFI_CONNECTION_LIB: ${{ secrets.DEPLOY_KEY_READ_VENAFI_CONNECTION_LIB }} + + - id: go-version + run: | + make print-go-version >> "$GITHUB_OUTPUT" + + - uses: actions/setup-go@4dc6199c7b1a012772edbd06daecab0f50c9053c # v6.1.0 + with: + go-version: ${{ steps.go-version.outputs.result }} + + - uses: actions/cache@0057852bfaa89a56745cba8c7296529d2fc39830 # v4.3.0 + with: + path: _bin/downloaded + key: downloaded-${{ runner.os }}-${{ hashFiles('klone.yaml') }}-test-unit + + - run: make -j ark-test-e2e + env: + OCI_BASE: ${{ secrets.ARK_OCI_BASE }} + # These environment variables are required to connect to CyberArk Disco APIs + ARK_DISCOVERY_API: https://platform-discovery.integration-cyberark.cloud/ + ARK_SUBDOMAIN: ${{ secrets.ARK_SUBDOMAIN }} + ARK_USERNAME: ${{ secrets.ARK_USERNAME }} + ARK_SECRET: ${{ secrets.ARK_SECRET }} + + ngts-test-e2e: + # TEMPORARY: require an explicit label to test NGTS until we have a stable test environment + # See `ark-test-e2e`. + if: >- + github.event_name == 'workflow_dispatch' + || github.event.label.name == 'test-ngts' + || (github.event.action != 'labeled' + && contains(github.event.pull_request.labels.*.name, 'test-ngts')) + runs-on: ubuntu-latest + timeout-minutes: 30 + steps: + - uses: actions/checkout@8e8c483db84b4bee98b60c0593521ed34d9990e8 # v6.0.1 + # Adding `fetch-depth: 0` makes sure tags are also fetched. We need + # the tags so `git describe` returns a valid version. + # see https://github.com/actions/checkout/issues/701 for extra info about this option + with: { fetch-depth: 0 } + + - uses: ./.github/actions/repo_access + with: + DEPLOY_KEY_READ_VENAFI_CONNECTION_LIB: ${{ secrets.DEPLOY_KEY_READ_VENAFI_CONNECTION_LIB }} + + - id: go-version + run: | + make print-go-version >> "$GITHUB_OUTPUT" + + - uses: actions/setup-go@4dc6199c7b1a012772edbd06daecab0f50c9053c # v6.1.0 + with: + go-version: ${{ steps.go-version.outputs.result }} + + - uses: actions/cache@0057852bfaa89a56745cba8c7296529d2fc39830 # v4.3.0 + with: + path: _bin/downloaded + key: downloaded-${{ runner.os }}-${{ hashFiles('klone.yaml') }}-test-unit + + - run: make -j ngts-test-e2e + env: + OCI_BASE: ${{ secrets.NGTS_OCI_BASE }} + NGTS_CLIENT_ID: e3c8bde7-5f13-11f1-99f4-5e067e231041 + NGTS_PRIVATE_KEY: ${{ secrets.NGTS_PRIVATE_KEY }} + NGTS_TSG_URL: https://1806660206.ngts.qa.venafi.io + + test-e2e: + # See `ark-test-e2e`. + if: >- + github.event_name == 'workflow_dispatch' + || github.event.label.name == 'test-e2e' + || (github.event.action != 'labeled' + && contains(github.event.pull_request.labels.*.name, 'test-e2e')) + runs-on: ubuntu-latest + # A healthy run takes about 15 minutes. The backstop matters because the job + # holds a GKE cluster for as long as it runs, and the default is 6 hours. + timeout-minutes: 30 + steps: + - uses: actions/checkout@8e8c483db84b4bee98b60c0593521ed34d9990e8 # v6.0.1 + # Adding `fetch-depth: 0` makes sure tags are also fetched. We need + # the tags so `git describe` returns a valid version. + # see https://github.com/actions/checkout/issues/701 for extra info about this option + with: { fetch-depth: 0 } + + - uses: ./.github/actions/repo_access + with: + DEPLOY_KEY_READ_VENAFI_CONNECTION_LIB: ${{ secrets.DEPLOY_KEY_READ_VENAFI_CONNECTION_LIB }} + + - name: Authenticate to Google Cloud + uses: google-github-actions/auth@7c6bc770dae815cd3e89ee6cdf493a5fab2cc093 # v3.0.0 + with: + credentials_json: '${{ secrets.GCP_SA_KEY }}' + + - name: Set up gcloud + uses: google-github-actions/setup-gcloud@aa5489c8933f4cc7a4f7d45035b3b1440c9c10db # v3.0.1 + with: + install_components: "gke-gcloud-auth-plugin" + project_id: machineidentitysecurity-jsci-e + + - name: Configure Docker for Google Artifact Registry + run: gcloud auth configure-docker europe-west1-docker.pkg.dev + + - id: go-version + run: | + make print-go-version >> "$GITHUB_OUTPUT" + + - uses: actions/setup-go@4dc6199c7b1a012772edbd06daecab0f50c9053c # v6.1.0 + with: + go-version: ${{ steps.go-version.outputs.result }} + + - name: Generate timestamp for cluster name + id: timestamp # Give the step an ID to reference its output + run: | + # Generate a timestamp in the format YYMMDD-HHMMSS. + # Extracting from PR name would require sanitization due to GKE cluster naming constraints + TIMESTAMP=$(date +'%y%m%d-%H%M%S') + CLUSTER_NAME="test-secretless-${TIMESTAMP}" + echo "Generated cluster name: ${CLUSTER_NAME}" + echo "cluster_name=${CLUSTER_NAME}" >> $GITHUB_OUTPUT + + - run: | + make helm-plugins + make -j test-e2e-gke + # The VEN_API_KEY_PULL secret is set to my API key (Mladen) for glow.in.the.dark tenant. + env: + VEN_API_KEY: ${{ secrets.VEN_API_KEY_PULL }} + VEN_API_KEY_PULL: ${{ secrets.VEN_API_KEY_PULL }} + OCI_BASE: europe-west1-docker.pkg.dev/machineidentitysecurity-jsci-e/js-agent-ci-repo + VEN_API_HOST: api.venafi.cloud + VEN_ZONE: k8s-agent-CI\Default + VEN_VCP_REGION: us + CLOUDSDK_CORE_PROJECT: machineidentitysecurity-jsci-e + CLOUDSDK_COMPUTE_ZONE: europe-west1-b + CLUSTER_NAME: ${{ steps.timestamp.outputs.cluster_name }} + + - name: Delete GKE Cluster + # 'always()' - Run this step regardless of success or failure. + # '!contains(...)' - AND only run if the list of PR labels DOES NOT contain 'keep-e2e-cluster'. + # NOTE: You will have to delete the test cluster manually when finished with debugging or incur costs. + # + # Add keep-e2e-cluster BEFORE test-e2e. The labels read here are the + # ones in the event payload, frozen when the run started, so a + # keep-e2e-cluster added after that point is not seen and the cluster + # is deleted anyway. Adding test-e2e now starts the run immediately, + # so there is no window to add it afterwards. + if: always() && !contains(github.event.pull_request.labels.*.name, 'keep-e2e-cluster') + run: | + echo "Label 'keep-e2e-cluster' not found. Cleaning up GKE cluster ${{ steps.timestamp.outputs.cluster_name }}" + gcloud container clusters delete ${{ steps.timestamp.outputs.cluster_name }} \ + --project=machineidentitysecurity-jsci-e \ + --zone=europe-west1-b \ + --quiet diff --git a/.github/workflows/tests.yaml b/.github/workflows/tests.yaml index 0981d0cc..42f5e1e3 100644 --- a/.github/workflows/tests.yaml +++ b/.github/workflows/tests.yaml @@ -2,10 +2,10 @@ name: tests on: push: branches: [master] + # Deliberately the default activity types. The label-triggered e2e suites + # live in e2e.yaml, so that adding a label cannot replace the check runs + # these jobs publish. pull_request: {} - # Lets us run the e2e suites against master, which the label gates below - # cannot do: they read github.event.pull_request.labels, which is empty for - # a push. Needed before tagging a release. workflow_dispatch: {} jobs: verify: @@ -78,156 +78,3 @@ jobs: ARK_SUBDOMAIN: ${{ secrets.ARK_SUBDOMAIN }} ARK_USERNAME: ${{ secrets.ARK_USERNAME }} ARK_SECRET: ${{ secrets.ARK_SECRET }} - - ark-test-e2e: - # TEMPORARY: require an explicit label to test disco-agent until the test environment fixes a recurring issue - # where the e2e fails with a 400 error relating to "conflicting tagging values" - # The test is flaky, not broken and re-running eventually makes it pass - but that delays progress on - # other unrelated work. - if: github.event_name == 'workflow_dispatch' || contains(github.event.pull_request.labels.*.name, 'test-ark') - runs-on: ubuntu-latest - timeout-minutes: 30 - steps: - - uses: actions/checkout@8e8c483db84b4bee98b60c0593521ed34d9990e8 # v6.0.1 - # Adding `fetch-depth: 0` makes sure tags are also fetched. We need - # the tags so `git describe` returns a valid version. - # see https://github.com/actions/checkout/issues/701 for extra info about this option - with: { fetch-depth: 0 } - - - uses: ./.github/actions/repo_access - with: - DEPLOY_KEY_READ_VENAFI_CONNECTION_LIB: ${{ secrets.DEPLOY_KEY_READ_VENAFI_CONNECTION_LIB }} - - - id: go-version - run: | - make print-go-version >> "$GITHUB_OUTPUT" - - - uses: actions/setup-go@4dc6199c7b1a012772edbd06daecab0f50c9053c # v6.1.0 - with: - go-version: ${{ steps.go-version.outputs.result }} - - - uses: actions/cache@0057852bfaa89a56745cba8c7296529d2fc39830 # v4.3.0 - with: - path: _bin/downloaded - key: downloaded-${{ runner.os }}-${{ hashFiles('klone.yaml') }}-test-unit - - - run: make -j ark-test-e2e - env: - OCI_BASE: ${{ secrets.ARK_OCI_BASE }} - # These environment variables are required to connect to CyberArk Disco APIs - ARK_DISCOVERY_API: https://platform-discovery.integration-cyberark.cloud/ - ARK_SUBDOMAIN: ${{ secrets.ARK_SUBDOMAIN }} - ARK_USERNAME: ${{ secrets.ARK_USERNAME }} - ARK_SECRET: ${{ secrets.ARK_SECRET }} - - ngts-test-e2e: - # TEMPORARY: require an explicit label to test NGTS until we have a stable test environment - if: github.event_name == 'workflow_dispatch' || contains(github.event.pull_request.labels.*.name, 'test-ngts') - runs-on: ubuntu-latest - timeout-minutes: 30 - steps: - - uses: actions/checkout@8e8c483db84b4bee98b60c0593521ed34d9990e8 # v6.0.1 - # Adding `fetch-depth: 0` makes sure tags are also fetched. We need - # the tags so `git describe` returns a valid version. - # see https://github.com/actions/checkout/issues/701 for extra info about this option - with: { fetch-depth: 0 } - - - uses: ./.github/actions/repo_access - with: - DEPLOY_KEY_READ_VENAFI_CONNECTION_LIB: ${{ secrets.DEPLOY_KEY_READ_VENAFI_CONNECTION_LIB }} - - - id: go-version - run: | - make print-go-version >> "$GITHUB_OUTPUT" - - - uses: actions/setup-go@4dc6199c7b1a012772edbd06daecab0f50c9053c # v6.1.0 - with: - go-version: ${{ steps.go-version.outputs.result }} - - - uses: actions/cache@0057852bfaa89a56745cba8c7296529d2fc39830 # v4.3.0 - with: - path: _bin/downloaded - key: downloaded-${{ runner.os }}-${{ hashFiles('klone.yaml') }}-test-unit - - - run: make -j ngts-test-e2e - env: - OCI_BASE: ${{ secrets.NGTS_OCI_BASE }} - NGTS_CLIENT_ID: e3c8bde7-5f13-11f1-99f4-5e067e231041 - NGTS_PRIVATE_KEY: ${{ secrets.NGTS_PRIVATE_KEY }} - NGTS_TSG_URL: https://1806660206.ngts.qa.venafi.io - - test-e2e: - if: github.event_name == 'workflow_dispatch' || contains(github.event.pull_request.labels.*.name, 'test-e2e') - runs-on: ubuntu-latest - # A healthy run takes about 15 minutes. The backstop matters because the job - # holds a GKE cluster for as long as it runs, and the default is 6 hours. - timeout-minutes: 30 - steps: - - uses: actions/checkout@8e8c483db84b4bee98b60c0593521ed34d9990e8 # v6.0.1 - # Adding `fetch-depth: 0` makes sure tags are also fetched. We need - # the tags so `git describe` returns a valid version. - # see https://github.com/actions/checkout/issues/701 for extra info about this option - with: { fetch-depth: 0 } - - - uses: ./.github/actions/repo_access - with: - DEPLOY_KEY_READ_VENAFI_CONNECTION_LIB: ${{ secrets.DEPLOY_KEY_READ_VENAFI_CONNECTION_LIB }} - - - name: Authenticate to Google Cloud - uses: google-github-actions/auth@7c6bc770dae815cd3e89ee6cdf493a5fab2cc093 # v3.0.0 - with: - credentials_json: '${{ secrets.GCP_SA_KEY }}' - - - name: Set up gcloud - uses: google-github-actions/setup-gcloud@aa5489c8933f4cc7a4f7d45035b3b1440c9c10db # v3.0.1 - with: - install_components: "gke-gcloud-auth-plugin" - project_id: machineidentitysecurity-jsci-e - - - name: Configure Docker for Google Artifact Registry - run: gcloud auth configure-docker europe-west1-docker.pkg.dev - - - id: go-version - run: | - make print-go-version >> "$GITHUB_OUTPUT" - - - uses: actions/setup-go@4dc6199c7b1a012772edbd06daecab0f50c9053c # v6.1.0 - with: - go-version: ${{ steps.go-version.outputs.result }} - - - name: Generate timestamp for cluster name - id: timestamp # Give the step an ID to reference its output - run: | - # Generate a timestamp in the format YYMMDD-HHMMSS. - # Extracting from PR name would require sanitization due to GKE cluster naming constraints - TIMESTAMP=$(date +'%y%m%d-%H%M%S') - CLUSTER_NAME="test-secretless-${TIMESTAMP}" - echo "Generated cluster name: ${CLUSTER_NAME}" - echo "cluster_name=${CLUSTER_NAME}" >> $GITHUB_OUTPUT - - - run: | - make helm-plugins - make -j test-e2e-gke - # The VEN_API_KEY_PULL secret is set to my API key (Mladen) for glow.in.the.dark tenant. - env: - VEN_API_KEY: ${{ secrets.VEN_API_KEY_PULL }} - VEN_API_KEY_PULL: ${{ secrets.VEN_API_KEY_PULL }} - OCI_BASE: europe-west1-docker.pkg.dev/machineidentitysecurity-jsci-e/js-agent-ci-repo - VEN_API_HOST: api.venafi.cloud - VEN_ZONE: k8s-agent-CI\Default - VEN_VCP_REGION: us - CLOUDSDK_CORE_PROJECT: machineidentitysecurity-jsci-e - CLOUDSDK_COMPUTE_ZONE: europe-west1-b - CLUSTER_NAME: ${{ steps.timestamp.outputs.cluster_name }} - - - name: Delete GKE Cluster - # 'always()' - Run this step regardless of success or failure. - # '!contains(...)' - AND only run if the list of PR labels DOES NOT contain 'keep-e2e-cluster'. - # NOTE: You will have to delete the test cluster manually when finished with debugging or incur costs. - if: always() && !contains(github.event.pull_request.labels.*.name, 'keep-e2e-cluster') - run: | - echo "Label 'keep-e2e-cluster' not found. Cleaning up GKE cluster ${{ steps.timestamp.outputs.cluster_name }}" - gcloud container clusters delete ${{ steps.timestamp.outputs.cluster_name }} \ - --project=machineidentitysecurity-jsci-e \ - --zone=europe-west1-b \ - --quiet diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 55c84d2c..02f1f672 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -172,6 +172,10 @@ E2E tests run automatically in CI when you add specific labels to your PR: - Add the `test-e2e` label to trigger GKE-based E2E tests - Add the `keep-e2e-cluster` label if you need to keep the cluster for debugging (remember to delete it manually afterward to avoid costs) +- Add `keep-e2e-cluster` **before** `test-e2e`. Applying `test-e2e` starts the run straight away, and the run only sees the labels that were set at that moment, so adding `keep-e2e-cluster` afterwards will not save the cluster. + +The suites live in [.github/workflows/e2e.yaml](./.github/workflows/e2e.yaml). You can also run them against any branch +without a label, using the **Run workflow** button on the `e2e` workflow. The E2E test script is located at [hack/e2e/test.sh](./hack/e2e/test.sh). diff --git a/README.md b/README.md index 94d3ed0c..6beb3134 100644 --- a/README.md +++ b/README.md @@ -61,6 +61,8 @@ The following metrics are collected: ## End to end testing An end to end test script is available in the [./hack/e2e/test.sh](./hack/e2e/test.sh) directory. It is configured to run in CI -in the tests.yaml GitHub Actions workflow. To run the script you will need to add the `test-e2e` label to the PR. +in the e2e.yaml GitHub Actions workflow. To run the script you will need to add the `test-e2e` label to the PR. The script creates a cluster in GKE and cleanups after itself unless the `keep-e2e-cluster` label is set on the PR. Adding that label will leave the cluster running for further debugging but it will incur costs so manually delete the cluster when done. +Add `keep-e2e-cluster` before `test-e2e`, because `test-e2e` starts the run immediately and the labels are read as they were +at that moment. diff --git a/RELEASE.md b/RELEASE.md index 0f1f5939..e3eb1ed7 100644 --- a/RELEASE.md +++ b/RELEASE.md @@ -42,6 +42,10 @@ The release process is semi-automated. 3. Open the [tests GitHub Actions workflow][tests-workflow] and verify that it succeeds on the master branch. + The end-to-end suites are not part of that workflow and do not run on + master by themselves. Run them with the **Run workflow** button on the + [e2e workflow][e2e-workflow], select `master`, and verify they succeed. + 4. Create a tag for the new release: ```sh @@ -71,6 +75,7 @@ The release process is semi-automated. documentation at . [tests-workflow]: https://github.com/jetstack/jetstack-secure/actions/workflows/tests.yaml?query=branch%3Amaster +[e2e-workflow]: https://github.com/jetstack/jetstack-secure/actions/workflows/e2e.yaml ## Release Artifact Information