feat(evaluate): kosli evaluate policy, evaluating and recording a decision in one request

[jumboduck]

Implements kosli-dev/server#6920.

A new command evaluates a policy where the trail is stored and records the outcome as a decision in the same request. The decision value never passes through the CLI, which is what removes the self-reported step this replaces: evaluate, read allow out of a local report, then assert it back with kosli attest decision --compliant=<value>.

kosli evaluate policy \
    --context trail=my-release-flow/$GITHUB_SHA \
    --policy ./policies/SDLC-CTRL-0007-code-review \
    --control SDLC-CTRL-0007 \
    --fingerprint "$ARTIFACT_FINGERPRINT" \
    --params '{"protected_branch": "master"}' \
    --assert

What it does

• --context trail=<flow>/<trail>, required and repeatable, names what is evaluated. Every trail travels in one evaluation, so they resolve at one instant.
• --control records the outcome as a decision in --flow / --trail, under --name (default <control>-decision), about --fingerprint or about the trail itself. Without --control nothing is recorded, and the destination flags are not required.
• --assert exits non-zero on a denial. Without it the verdict prints and the command exits 0: recording a decision is not a reason to fail the step that asked for it.
• --policy takes a file or a directory. A directory travels as one bundle keyed by paths relative to it, with the published 100-file and 1 MiB caps refused here rather than sent to be rejected.
• Output matches kosli evaluate trail, table and JSON, plus the written decision's id where there is one.
• A denial is a decision and is recorded. A policy that could not run is not: it records nothing, is never worded as a denial, and fails.

Decisions worth a reviewer's attention

• Hidden for now, against the ticket, which asks for a published command. Nothing can be exercised end to end until it runs against a server that can evaluate, and a listed command is a contract from the moment it ships. Unhiding is one line.
• What is evaluated and where a decision lands are separate. --flow and --trail name only the destination, are satisfied by KOSLI_FLOW and KOSLI_TRAIL, and are never refused for being present without --control — a pipeline sets them for every command it runs.
• No policy-bundle digest. It belongs with versioned policies, which are out of scope.
• One exit code. The ticket asks for three; a single failure exit path is a product-wide convention, so the outcomes are told apart by what they say instead.
• Synchronous only. No asynchronous mode and no --sync flag.
• No http(s):// policies on this command, though the older evaluate commands keep them: that way of naming a policy is on its way out.
• Refusals travel in the API's own words rather than being classified here, so no list of server cases has to be kept in step.

The request-and-verdict sequence is shared with the hidden --server-side flag on evaluate trail, so the two cannot drift.

Testing

• New suites: the command (30 tests) and the decision block in the evaluations client (5 tests). They drive a stubbed server, because this repository's test environment has no evaluator to reach a verdict with.
• make lint: 0 issues.
• Not run: the full integration suite, which needs the local test server. Two older evaluate-trail suites and one HTTP client test are unverified for the same reason.
• Still to do: a check against staging with an entitled organisation — allow, deny, a broken policy, a decision recorded and read back, and a destination the token cannot write to.

Plan and handover: docs/plans/6920-evaluate-policy.md, docs/handover/6920-evaluate-an-inline-policy-and-record-its-decis.md.

🤖 Generated with Claude Code

[claude]

Claude finished @jumboduck's task in 1m 34s —— View job

---

Follow-up review of commit 65d71d50

• Fetch previous review findings and this PR's new commit diff
• Check each previous finding against the new commit
• Scan the new commit's changed lines for fresh issues
• Post summary

Previous findings — status

#	Finding	Status
1	--fingerprint not validated before send (evaluatePolicy.go)	✅ Resolved — digest.ValidateDigest is now called in decision() before the evaluations.Decision is built (cmd/kosli/evaluatePolicy.go:194-198), with a new test case ("a fingerprint that is not a SHA256") asserting "not a valid SHA256" and that nothing is sent.
2	Symlinks followed / caps checked after full read (evaluateHelpers.go)	⚠️ Not addressed — this commit doesn't touch evaluateHelpers.go at all. The thread shows resolved on GitHub, but the walk still uses os.ReadFile on every non-dir DirEntry (no entry.Type().IsRegular() check) and still accumulates all files before checking maxPolicyBundleFiles/serverPolicyMaxBytes. Worth re-opening or explicitly deferring.
3	Refusal names only the flag that was typed (evaluatePolicy.go:172)	✅ Resolved — new namedDecisionFlags(name, fingerprint) helper builds the message from what was actually given; tests now assert both the mentioned flag and saysNot for the flag not given, plus a combined-both case.
4	--output validated too late, after the decision request (evaluatePolicy.go:161)	✅ Resolved — run() now rejects an unknown --output value against evaluatePolicyOutputs ({"table": true, "json": true}) before parseTrailContexts/decision()/runServerEvaluation run, so a bad format never reaches the API. Matches the {"json", "table"} set actually wired in runServerEvaluation's output.FormattedPrint call, so no drift. New test: "an output format that does not exist".

New issues in the changed lines: none. The added validation is placed correctly (after the --flow/--trail presence check, before the fingerprint is put on the wire), and the new test cases match the behavior.

3 of 4 findings resolved; the symlink/streaming-caps finding in evaluateHelpers.go remains open and untouched by this commit.
· branch 6920-evaluate-policy

[mbevc1]

@jumboduck could --context be something else than trail=...?

[jumboduck]

@jumboduck could --context be something else than trail=...?

@mbevc1 Not at the moment. We might have a better idea after Malmö.

[mbevc1]

@jumboduck could --context be something else than trail=...?

@mbevc1 Not at the moment. We might have a better idea after Malmö.

Gotcha, I was just wondering how come parameter is not just called --trail. Looking forward where we end up

[jumboduck]

Gotcha, I was just wondering how come parameter is not just called --trail. Looking forward where we end up

@mbevc1 --trail is already used for the target of the decision attestation. You could evaluate one or several trails against a policy, and attest the decision on a completely different trail.