Skip to content
Open
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
117 changes: 107 additions & 10 deletions website/content/en/docs/testing-operators/scorecard/_index.md
Original file line number Diff line number Diff line change
Expand Up @@ -52,11 +52,36 @@ for an overview of command invocation.

## Configuration

The scorecard test execution is driven by a configuration file named `config.yaml`, generated by `make bundle`. Note that if run `make bundle` that any
changes you have made to `config.yaml` will be overwritten. To persist
any changes to `config.yaml` you can update the kustomize templates found in the
`config/scorecard` directory.
The configuration file is located at the following location within your bundle directory (`bundle/` by default):
Scorecard decides which tests to run from a Configuration resource
(`kind: Configuration`, `apiVersion: scorecard.operatorframework.io/v1alpha3`).
You can supply that config in either of two ways:

1. **Bundle config (default)** — a file packaged with the operator bundle at
`tests/scorecard/config.yaml`. This is what `make bundle` generates and what
consumers of a published bundle image get by default.
2. **External config (`--config` / `-c`)** — any path on the machine running
`operator-sdk scorecard`. When set, this file is used **instead of** the
config inside the bundle. Use this when you want one shared test suite for
many bundles, or when test definitions should not live in the release
bundle (for example, internal or CI-only checks).

### When to use which

| Goal | Approach |
| ---- | -------- |
| Ship a default test set with your operator | Edit `config/scorecard` kustomize and run `make bundle` so `bundle/tests/scorecard/config.yaml` is updated |
| Run the same custom tests against many bundles | Keep a standalone `mytests.yaml` and pass `--config mytests.yaml` for each bundle |
| Override the bundle's tests for a one-off CI job | Pass `--config` pointing at a CI checkout of your scorecard config |
| Add operator-specific custom test images | Add entries under `stages[].tests` (see [Writing Custom Scorecard Tests](custom-tests/)) |

The configuration file does **not** have to be inside the bundle. Bundle layout
is only the default lookup path when `--config` is omitted.

### Bundle config path

When you do not pass `--config`, scorecard loads the config from the bundle
directory (`bundle/` by default):

```sh
$ tree ./bundle
./bundle
Expand All @@ -66,9 +91,15 @@ $ tree ./bundle
└── config.yaml
```

That file is produced by `make bundle` from the kustomize bases under
`config/scorecard`. Running `make bundle` regenerates
`bundle/tests/scorecard/config.yaml` and will overwrite manual edits there.
Persist changes in `config/scorecard` (bases and patches), not only in the
generated bundle copy.

### Config File

The following YAML spec is an example of the scorecard configuration file:
The following YAML is an example scorecard configuration:

```yaml
kind: Configuration
Expand Down Expand Up @@ -106,6 +137,72 @@ follows:
| entrypoint | the command and arguments that are invoked in the test image to execute a test
| labels | scorecard-defined or custom labels that [select](#selecting-tests) which tests to run

### Adding your own tests to the config

Each item under `stages[].tests` is one test run. To add a test:

1. Build (or reuse) a test image that follows the [custom test conventions](#extending-the-scorecard-with-custom-tests).
2. Append a new list entry with `image`, `entrypoint`, and `labels`.
3. Optionally put custom tests in their own stage if they must not run in
parallel with the built-in suite.

Example fragment that adds one custom test alongside the built-in basic check:

```yaml
stages:
- parallel: true
tests:
- image: quay.io/operator-framework/scorecard-test:latest
entrypoint:
- scorecard-test
- basic-check-spec
labels:
suite: basic
test: basic-check-spec-test
- image: example.com/my-operator-scorecard:1.0.0
entrypoint:
- custom-scorecard-tests
- my-operator-check
labels:
suite: custom
test: my-operator-check
```

For a full walkthrough of implementing the image, see
[Writing Custom Scorecard Tests](custom-tests/).

### Using an external config on the command line

Pass `-c` / `--config` with a path to a Configuration file. Scorecard still
requires a bundle directory or bundle image as the positional argument (the
bundle under test); only the **test list** is taken from `--config`.

Run one external config against a single on-disk bundle:

```sh
$ operator-sdk scorecard ./bundle --config mytests.yaml
```

Run the same external config against a bundle image:

```sh
$ operator-sdk scorecard quay.io/example/my-operator-bundle:v1.2.3 --config mytests.yaml
```

Reuse one config file across many bundles (for example in CI):

```sh
$ for b in ./bundles/*/ ; do
operator-sdk scorecard "$b" --config mytests.yaml -o text
done
```

Combine `--config` with selectors to run a subset of the external suite:

```sh
$ operator-sdk scorecard ./bundle --config mytests.yaml --selector=suite=custom
```

### Command Args

The scorecard command has the following syntax:
Expand All @@ -114,10 +211,10 @@ $ operator-sdk scorecard <bundle_dir_or_image> [flags]
```

The scorecard requires a positional argument that holds either the
on-disk path to your operator bundle or the name of a bundle image. Note
that the scorecard does not run your operator but merely uses
the scorecard configuration within the bundle contents to know which tests
to execute.
on-disk path to your operator bundle or the name of a bundle image.
Scorecard does not start your operator. It reads the scorecard
configuration (from `--config` or from the bundle) to know which test images
to run, and mounts the bundle contents for those tests at `/bundle`.

For further information about the flags see the [CLI documentation][cli-scorecard].

Expand Down