From 8ad0fce2b9dc4e64078cc927d28dc9a92df46cce Mon Sep 17 00:00:00 2001 From: Vedant Madane Date: Tue, 18 Aug 2026 22:01:36 +0530 Subject: [PATCH] docs(scorecard): clarify config file usage and --config override Fixes #4464 Signed-off-by: Vedant Madane --- .../testing-operators/scorecard/_index.md | 117 ++++++++++++++++-- 1 file changed, 107 insertions(+), 10 deletions(-) diff --git a/website/content/en/docs/testing-operators/scorecard/_index.md b/website/content/en/docs/testing-operators/scorecard/_index.md index 98305b4dfcf..de809c3c4e4 100644 --- a/website/content/en/docs/testing-operators/scorecard/_index.md +++ b/website/content/en/docs/testing-operators/scorecard/_index.md @@ -52,11 +52,36 @@ for an overview of command invocation. ## Configuration -The scorecard test execution is driven by a configuration file named `config.yaml`, generated by `make bundle`. Note that if run `make bundle` that any -changes you have made to `config.yaml` will be overwritten. To persist -any changes to `config.yaml` you can update the kustomize templates found in the -`config/scorecard` directory. -The configuration file is located at the following location within your bundle directory (`bundle/` by default): +Scorecard decides which tests to run from a Configuration resource +(`kind: Configuration`, `apiVersion: scorecard.operatorframework.io/v1alpha3`). +You can supply that config in either of two ways: + +1. **Bundle config (default)** — a file packaged with the operator bundle at + `tests/scorecard/config.yaml`. This is what `make bundle` generates and what + consumers of a published bundle image get by default. +2. **External config (`--config` / `-c`)** — any path on the machine running + `operator-sdk scorecard`. When set, this file is used **instead of** the + config inside the bundle. Use this when you want one shared test suite for + many bundles, or when test definitions should not live in the release + bundle (for example, internal or CI-only checks). + +### When to use which + +| Goal | Approach | +| ---- | -------- | +| Ship a default test set with your operator | Edit `config/scorecard` kustomize and run `make bundle` so `bundle/tests/scorecard/config.yaml` is updated | +| Run the same custom tests against many bundles | Keep a standalone `mytests.yaml` and pass `--config mytests.yaml` for each bundle | +| Override the bundle's tests for a one-off CI job | Pass `--config` pointing at a CI checkout of your scorecard config | +| Add operator-specific custom test images | Add entries under `stages[].tests` (see [Writing Custom Scorecard Tests](custom-tests/)) | + +The configuration file does **not** have to be inside the bundle. Bundle layout +is only the default lookup path when `--config` is omitted. + +### Bundle config path + +When you do not pass `--config`, scorecard loads the config from the bundle +directory (`bundle/` by default): + ```sh $ tree ./bundle ./bundle @@ -66,9 +91,15 @@ $ tree ./bundle └── config.yaml ``` +That file is produced by `make bundle` from the kustomize bases under +`config/scorecard`. Running `make bundle` regenerates +`bundle/tests/scorecard/config.yaml` and will overwrite manual edits there. +Persist changes in `config/scorecard` (bases and patches), not only in the +generated bundle copy. + ### Config File -The following YAML spec is an example of the scorecard configuration file: +The following YAML is an example scorecard configuration: ```yaml kind: Configuration @@ -106,6 +137,72 @@ follows: | entrypoint | the command and arguments that are invoked in the test image to execute a test | labels | scorecard-defined or custom labels that [select](#selecting-tests) which tests to run +### Adding your own tests to the config + +Each item under `stages[].tests` is one test run. To add a test: + +1. Build (or reuse) a test image that follows the [custom test conventions](#extending-the-scorecard-with-custom-tests). +2. Append a new list entry with `image`, `entrypoint`, and `labels`. +3. Optionally put custom tests in their own stage if they must not run in + parallel with the built-in suite. + +Example fragment that adds one custom test alongside the built-in basic check: + +```yaml +stages: +- parallel: true + tests: + - image: quay.io/operator-framework/scorecard-test:latest + entrypoint: + - scorecard-test + - basic-check-spec + labels: + suite: basic + test: basic-check-spec-test + - image: example.com/my-operator-scorecard:1.0.0 + entrypoint: + - custom-scorecard-tests + - my-operator-check + labels: + suite: custom + test: my-operator-check +``` + +For a full walkthrough of implementing the image, see +[Writing Custom Scorecard Tests](custom-tests/). + +### Using an external config on the command line + +Pass `-c` / `--config` with a path to a Configuration file. Scorecard still +requires a bundle directory or bundle image as the positional argument (the +bundle under test); only the **test list** is taken from `--config`. + +Run one external config against a single on-disk bundle: + +```sh +$ operator-sdk scorecard ./bundle --config mytests.yaml +``` + +Run the same external config against a bundle image: + +```sh +$ operator-sdk scorecard quay.io/example/my-operator-bundle:v1.2.3 --config mytests.yaml +``` + +Reuse one config file across many bundles (for example in CI): + +```sh +$ for b in ./bundles/*/ ; do + operator-sdk scorecard "$b" --config mytests.yaml -o text + done +``` + +Combine `--config` with selectors to run a subset of the external suite: + +```sh +$ operator-sdk scorecard ./bundle --config mytests.yaml --selector=suite=custom +``` + ### Command Args The scorecard command has the following syntax: @@ -114,10 +211,10 @@ $ operator-sdk scorecard [flags] ``` The scorecard requires a positional argument that holds either the -on-disk path to your operator bundle or the name of a bundle image. Note -that the scorecard does not run your operator but merely uses -the scorecard configuration within the bundle contents to know which tests -to execute. +on-disk path to your operator bundle or the name of a bundle image. +Scorecard does not start your operator. It reads the scorecard +configuration (from `--config` or from the bundle) to know which test images +to run, and mounts the bundle contents for those tests at `/bundle`. For further information about the flags see the [CLI documentation][cli-scorecard].