Skip to content

test: make the Gateway API conformance report honest and reproducible (backport apache/apisix-ingress-controller#2843) #467

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
AlinsRan wants to merge 4 commits into
base: master
Choose a base branch
from
Open

test: make the Gateway API conformance report honest and reproducible (backport apache/apisix-ingress-controller#2843) #467

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
0f3fcbf
test: make the Gateway API conformance report honest and reproducible…
AlinsRan Aug 19, 2026
81eb78e
test(conformance): adapt the api7ee suite so its run is a real verdict
AlinsRan Aug 20, 2026
851efd4
test(conformance): address review on the conformance workflows and Ma…
AlinsRan Aug 20, 2026
c3f1f8c
test(conformance): fix kind-lb aborting under .SHELLFLAGS -e
AlinsRan Aug 20, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
112 changes: 101 additions & 11 deletions .github/workflows/apisix-conformance-test.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,16 @@ on:
pull_request:
branches:
- master
# The report submitted to the Gateway API conformance reports repository is
# produced here, with image_tag naming the release to test. It cannot key off
# the tag push itself: push-docker.yaml publishes the images on that same
# event, so they do not exist yet when this would start pulling them.
workflow_dispatch:
inputs:
image_tag:
description: "Released tag to test, for example 2.2.0. Run it from that same tag, since the skip list and the manifests come from the checked-out ref. Leave empty to build and test the checked-out commit."
required: false
default: ""

concurrency:
group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}
Expand All @@ -35,13 +45,18 @@ permissions:
jobs:
conformance-test:
env:
CONFORMANCE_TEST_REPORT_OUTPUT: /tmp/apisix-ingress-controller-conformance-report.yaml
# The report name encodes the mode, so each provider writes its own file
# and the two can be submitted side by side. apisix is the controller's
# default provider, the standalone run declares a named mode.
CONFORMANCE_MODE: ${{ matrix.conformance_mode }}
timeout-minutes: 60
strategy:
matrix:
provider_type:
- apisix-standalone
- apisix
include:
- provider_type: apisix-standalone
conformance_mode: apisix-standalone
- provider_type: apisix
conformance_mode: default
runs-on: ubuntu-latest
steps:
- name: Checkout
Expand All @@ -64,7 +79,11 @@ jobs:
username: ${{ secrets.DOCKER_USERNAME }}
password: ${{ secrets.DOCKER_PASSWORD }}

# A release run must exercise the published images for that release,
# otherwise the report would describe a local build while claiming the
# release version. Every other run tests what was just built.
- name: Build images
if: ${{ !inputs.image_tag }}
env:
TAG: dev
ARCH: amd64
Expand All @@ -81,48 +100,119 @@ jobs:

- name: Install And Run Cloud Provider KIND
run: |
go install sigs.k8s.io/cloud-provider-kind@v0.8.0
nohup cloud-provider-kind > /tmp/kind-loadbalancer.log 2>&1 &
make kind-lb

- name: Install Gateway API And CRDs
run: |
make install

- name: Loading Docker Image to Kind Cluster
if: ${{ !inputs.image_tag }}
run: |
make kind-load-adc-image
make kind-load-ingress-image

# A release run pulls the published images instead, and the report names
# the release rather than the commit.
- name: Resolve Released Images
if: ${{ inputs.image_tag }}
shell: bash
env:
# A dispatch input is text someone typed, so it is read from the
# environment rather than expanded into the script, and checked
# against the tag grammar before it is used or written anywhere.
IMAGE_TAG: ${{ inputs.image_tag }}
REF_NAME: ${{ github.ref_name }}
run: |
if ! [[ "${IMAGE_TAG}" =~ ^[A-Za-z0-9_][A-Za-z0-9._-]{0,127}$ ]]; then
echo "::error::image_tag is not a valid image tag: ${IMAGE_TAG}"
exit 1
fi
# The report names image_tag, but the skip list, the supported feature
# list and the manifests come from the ref this runs on, so only a run
# from that same tag is submittable. It stays a warning rather than an
# error because running from another ref is how this path gets
# rehearsed against an already published release, and the workflow a
# tag run executes is the one inside that tag, which cannot be fixed
# afterwards without a new tag.
if [ "${REF_NAME}" != "${IMAGE_TAG}" ]; then
echo "::warning::running from ${REF_NAME} while reporting ${IMAGE_TAG}, this report is a rehearsal and must not be submitted"
fi
printf 'CONFORMANCE_IMAGE_TAG=%s\n' "${IMAGE_TAG}" >> "$GITHUB_ENV"

- name: Run Conformance Test
id: conformance
shell: bash
continue-on-error: true
env:
PROVIDER_TYPE: ${{ matrix.provider_type }}
run: |
make conformance-test

- name: Show Conformance Report
if: always()
shell: bash
run: |
cat ./*-report.yaml || echo "no report was produced"

# The suite runs most tests with t.Parallel, and those report their result
# after the report has been written, so a parallel failure leaves the
# report at Failed: 0 while the suite exits non-zero. The step outcome is
# the only place such a failure shows, which is why the run above is
# continue-on-error and the verdict is reached here instead.
- name: Check Conformance Result
if: always()
shell: bash
env:
STRICT: ${{ inputs.image_tag != '' }}
RUN_OUTCOME: ${{ steps.conformance.outcome }}
run: |
cat ${CONFORMANCE_TEST_REPORT_OUTPUT}
problem=""
if [ "${RUN_OUTCOME}" = "skipped" ]; then
problem="the suite never ran, an earlier step failed"
elif [ "${RUN_OUTCOME}" != "success" ]; then
problem="the suite exited non-zero, see the failures above"
elif ! ls ./*-report.yaml >/dev/null 2>&1; then
problem="no report was produced"
elif grep -qE '^[[:space:]]+result: failure' ./*-report.yaml; then
problem="the report contains a failing profile"
fi
if [ -n "${problem}" ]; then
echo "::error::${problem}"
grep -nE '^[[:space:]]+result:' ./*-report.yaml 2>/dev/null || true
if [ "${STRICT}" = "true" ]; then
exit 1
fi
echo "::warning::not a release run, not failing the job"
fi

- name: Get Logs from apisix-ingress-controller
if: always()
shell: bash
run: |
kubectl logs -n apisix-conformance-test -l app=apisix-ingress-controller

- name: Upload Gateway API Conformance Report
if: always()
uses: actions/upload-artifact@v4
with:
name: apisix-ingress-controller-conformance-report-${{ matrix.provider_type }}.yaml
path: ${{ env.CONFORMANCE_TEST_REPORT_OUTPUT }}
# The artifact keeps the upstream report file name, so a release run
# can be submitted straight from the download without renaming it.
name: conformance-report-${{ matrix.provider_type }}
path: ./*-report.yaml

- name: Format Conformance Test Report
if: ${{ github.event_name == 'pull_request' }}
run: |
echo '# conformance test report - ${{ matrix.provider_type }} mode' > report.md
echo '```yaml' >> report.md
cat ${CONFORMANCE_TEST_REPORT_OUTPUT} >> report.md
# An unmatched glob would make cat fail and take the job with it,
# which would turn a warned-about missing report into a hard failure.
if ls ./*-report.yaml >/dev/null 2>&1; then
cat ./*-report.yaml >> report.md
else
echo 'no report was produced' >> report.md
fi
echo '```' >> report.md

- name: Report Conformance Test Result to PR Comment
Expand Down
52 changes: 43 additions & 9 deletions .github/workflows/conformance-test.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -31,8 +31,6 @@ concurrency:

jobs:
conformance-test:
env:
CONFORMANCE_TEST_REPORT_OUTPUT: /tmp/api7-ingress-controller-conformance-report.yaml
timeout-minutes: 60
runs-on: ubuntu-latest
permissions:
Expand Down Expand Up @@ -82,8 +80,7 @@ jobs:

- name: Install And Run Cloud Provider KIND
run: |
go install sigs.k8s.io/cloud-provider-kind@v0.8.0
nohup cloud-provider-kind > /tmp/kind-loadbalancer.log 2>&1 &
make kind-lb

- name: Install Gateway API And CRDs
run: |
Expand All @@ -98,6 +95,7 @@ jobs:
make download-api7ee3-chart

- name: Run Conformance Test
id: conformance
shell: bash
env:
API7_EE_LICENSE: ${{ secrets.API7_EE_LICENSE }}
Expand All @@ -106,28 +104,64 @@ jobs:
make conformance-test-api7ee

- name: Show Conformance Report
if: always()
shell: bash
run: |
cat ${CONFORMANCE_TEST_REPORT_OUTPUT}
cat ./*-report.yaml || echo "no report was produced"

# The suite runs most tests with t.Parallel, and those report their result
# after the report has been written, so a parallel failure leaves the
# report at Failed: 0 while the suite exits non-zero. The step outcome is
# the only place such a failure shows, which is why the run above is
# continue-on-error and the verdict is reached here instead.
- name: Check Conformance Result
if: always()
shell: bash
env:
RUN_OUTCOME: ${{ steps.conformance.outcome }}
run: |
problem=""
if [ "${RUN_OUTCOME}" = "skipped" ]; then
problem="the suite never ran, an earlier step failed"
elif [ "${RUN_OUTCOME}" != "success" ]; then
problem="the suite exited non-zero, see the failures above"
elif ! ls ./*-report.yaml >/dev/null 2>&1; then
problem="no report was produced"
elif grep -qE '^[[:space:]]+result: failure' ./*-report.yaml; then
problem="the report contains a failing profile"
fi
if [ -n "${problem}" ]; then
echo "::error::${problem}"
grep -nE '^[[:space:]]+result:' ./*-report.yaml 2>/dev/null || true
exit 1
fi

- name: Get Logs from api7-ingress-controller
if: always()
shell: bash
run: |
kubectl logs -n apisix-conformance-test -l app=apisix-ingress-controller

- name: Upload Gateway API Conformance Report
if: ${{ github.event_name == 'push' }}
if: always()
uses: actions/upload-artifact@v4
with:
name: api7-ingress-controller-conformance-report.yaml
path: ${{ env.CONFORMANCE_TEST_REPORT_OUTPUT }}
# The artifact keeps the upstream report file name, so it can be read
# straight from the download without renaming it.
name: conformance-report-api7ee
path: ./*-report.yaml

- name: Format Conformance Test Report
if: ${{ github.event_name == 'pull_request' }}
run: |
echo '# conformance test report' > report.md
echo '```yaml' >> report.md
cat ${CONFORMANCE_TEST_REPORT_OUTPUT} >> report.md
# An unmatched glob would make cat fail and take the job with it.
if ls ./*-report.yaml >/dev/null 2>&1; then
cat ./*-report.yaml >> report.md
else
echo 'no report was produced' >> report.md
fi
echo '```' >> report.md

- name: Report Conformance Test Result to PR Comment
Expand Down
2 changes: 1 addition & 1 deletion .gitignore
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -46,7 +46,7 @@ go.work
dist
.tmp
apisix-ingress-controller
apisix-ingress-controller-conformance-report.yaml
/*-report.yaml

*.mdx
.cursor/
Expand Down
Loading
Loading