Skip to content
Open
15 changes: 12 additions & 3 deletions .github/workflows/strix.yml
Original file line number Diff line number Diff line change
Expand Up @@ -858,15 +858,24 @@ jobs:
exit 0
fi

# Exit 3 is emitted only by the trusted gate after its exact PR-scope
# mapper excludes every reported finding as unchanged and all model
# providers are unavailable. Vulnerability text remains in the audit
# log, so this cannot use the generic no-finding regex path below.
if [ "$strix_rc" -eq 3 ]; then
echo "::warning title=Strix backend unavailable after baseline scan::Strix excluded all reported findings from the pull-request diff, then exhausted its model providers. Treating this exact-head run as a neutral infrastructure skip; changed or unmapped findings still fail closed."
exit 0
fi
Comment thread
seonghobae marked this conversation as resolved.

# Preserve configuration failures (exit 2) and any unexpected exit
# code as hard failures — only the scan-failure code (1) can be an
# infrastructure/backend-unavailability outcome.
if [ "$strix_rc" -ne 1 ]; then
exit "$strix_rc"
fi

# Recognized signals that the LLM backend was unavailable / starved.
backend_unavailable_signal='RateLimitError|Too many requests\. For more on scraping GitHub|exceeded your current quota|insufficient_quota|billing details|"status"[[:space:]]*:[[:space:]]*"RESOURCE_EXHAUSTED"|tokens_limit_reached|Request body too large|Max size:[[:space:]]*[0-9]+[[:space:]]+tokens|Error code:[[:space:]]*413|LLM CONNECTION FAILED|Could not establish connection to the language model|LLM warm-up failed|Configured model and fallback models were unavailable|Configured Vertex model and fallback models were unavailable|emitted provider infrastructure or failure-signal output|before provider infrastructure failure|litellm(\.exceptions)?\.NotFoundError[^[:cntrl:]]*Nvidia_nimException[^[:cntrl:]]*Error code:[[:space:]]*404'
# Recognized signals that the scanner or LLM backend was unavailable.
backend_unavailable_signal='RateLimitError|Too many requests\. For more on scraping GitHub|exceeded your current quota|insufficient_quota|billing details|"status"[[:space:]]*:[[:space:]]*"RESOURCE_EXHAUSTED"|tokens_limit_reached|Request body too large|Max size:[[:space:]]*[0-9]+[[:space:]]+tokens|Error code:[[:space:]]*413|LLM CONNECTION FAILED|Could not establish connection to the language model|LLM warm-up failed|Configured model and fallback models were unavailable|Configured Vertex model and fallback models were unavailable|emitted provider infrastructure or failure-signal output|before provider infrastructure failure|litellm(\.exceptions)?\.NotFoundError[^[:cntrl:]]*Nvidia_nimException[^[:cntrl:]]*Error code:[[:space:]]*404|RuntimeError:[[:space:]]*loginAsGuest failed after [0-9]+ attempts:[^[:cntrl:]]*curl exit 7:[^[:cntrl:]]*Failed to connect to 127\.0\.0\.1 port 48080'
# Any evidence that a vulnerability was actually reported. Its presence
# forces a hard failure so real findings are NEVER downgraded. Keep the
# severity branch anchored away from identifiers so environment lines
Expand All @@ -879,7 +888,7 @@ jobs:
# from blocking current-head merge progress.
if grep -Eiq "$backend_unavailable_signal" "$strix_run_log" \
&& ! grep -Eiq "$reported_vulnerability_signal" "$strix_run_log"; then
echo "::warning title=Strix backend unavailable::Strix could not complete because its LLM backend was unavailable (rate limit / token cap / connection or warm-up failure) before producing a vulnerability report. Treating as a neutral skip so an infrastructure outage does not block merges; genuine findings still fail the check. See the strix-reports artifact and the run log."
echo "::warning title=Strix infrastructure unavailable::Strix could not complete because its scanner bootstrap or LLM backend was unavailable before producing a vulnerability report. Treating as a neutral skip so an infrastructure outage does not block merges; genuine findings still fail the check. See the strix-reports artifact and the run log."
exit 0
fi

Expand Down
10 changes: 10 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,16 @@ Semantic Versioning where the repository publishes a release.

### Changed

- Preserve the trusted PR-scope decision when Strix finds only unchanged-file
baseline issues and then exhausts every LLM provider, so provider retirement
or outage is a neutral infrastructure result even when no distinct fallback
is configured; recognize the exact local Caido bootstrap connection failure
the same way only when no vulnerability exists; route the exact single-line
LiteLLM/Azure unsupported-temperature failure to the existing distinct outer
fallback without accepting split-line imitations, while changed and unmapped
findings and all other runtime failures still fail closed. The reproduced
incidents, APA 7 sources, and gate boundary are recorded in
`docs/doctoring/strix-pr-baseline-provider-exhaustion.md`.
- Emit completed repository pull-list requests as they finish in the five-minute
agent-mention sweep, while retaining the four-worker ceiling, rotation, and
exact-name dispatch ledger, so one slow repository cannot hide ready sibling
Expand Down
80 changes: 80 additions & 0 deletions docs/doctoring/strix-pr-baseline-provider-exhaustion.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,80 @@
# Strix PR baseline/provider-exhaustion incident

## Observed failure

LineageWeave PR 392 run `32530198775` reported a critical secret in the
nonexistent `frontend/src/config.ts`. The trusted changed-file mapper correctly
classified that report as unchanged, but later fallback attempts ended in
provider HTTP 410 retirement brownouts. The gate retained the earlier severity
rank and returned the same exit code used for changed-file findings, so the
outer workflow could not distinguish the cleared baseline report from a real
pull-request vulnerability.

A later central PR run reported zero vulnerabilities and then failed before
scanning when Strix's local Caido process did not accept connections on
`127.0.0.1:48080`; the outer workflow did not yet recognize that exact scanner
bootstrap outage as infrastructure.

Another observed run selected Azure `gpt-5.6-sol`, where LiteLLM forwarded
Strix's `temperature=0.2`. Azure rejected the unsupported sampling parameter
and LiteLLM had no fallback group for that model. Microsoft documents
`temperature` as unsupported for reasoning models (Microsoft, 2026), while the
documented Strix configuration surface has no generation-parameter control and
the missing capability remains an upstream request (AkikoOrenji, 2026;
usestrix, n.d.).

## Root cause and repair

The quick gate already owns the exact PR-head changed-file mapping decision.
After it has classified every report as `allow_baseline`, provider exhaustion
now returns the dedicated status 3, including a deployment with no distinct
fallback configured. The trusted reusable workflow maps only that status to a
neutral infrastructure warning. Changed, unmapped, or manifest findings still
return the blocking status, and configuration or unexpected statuses still
fail closed.

The outer workflow also recognizes only the observed `loginAsGuest` retry
exhaustion with curl exit 7 against Strix's fixed local Caido port. It is neutral
only when no positive vulnerability or severity signal exists; every other
runtime failure remains blocking.

The preferred request-boundary repair is to omit a sampling parameter a caller
did not explicitly provide. The pinned Strix integration cannot currently do
that through its documented configuration, so the quick gate recognizes only a
single log line containing the complete LiteLLM/Azure unsupported-temperature
failure and the missing internal model group. It skips deterministic same-model
retry and moves to the existing distinct outer fallback. Split-line signal
assembly stays non-retryable, and provider exhaustion becomes neutral only
after the existing trusted PR-scope mapper has classified every report as
`allow_baseline`.

## Verification

- A three-attempt regression reproduces an unchanged critical report followed
by two provider failures and requires status 3.
- A primary-only regression requires the same trusted baseline outcome without
treating the absent fallback as a pull-request finding.
- Existing source tests require changed findings to remain blocking and clean
unchanged findings to remain admissible.
- A workflow regression requires the exact Caido bootstrap outage to be neutral
with zero findings and blocking when any vulnerability is reported.
- An Azure capability regression requires the exact unsupported-temperature
line to reach the configured GitHub Models fallback without a same-model
retry; a split-line imitation must remain non-recoverable.
- The central Python suite, native workflow validation, Bash syntax checks, and
the complete Strix shell regression suite run on the final tree.

## References

AkikoOrenji. (2026, June 4). *[Feature] Expose LLM generation parameters to
control local/OpenAI-compatible model behaviour and prevent runaway tool-call
loops* (Issue No. 514) [GitHub issue]. GitHub.
https://github.com/usestrix/strix/issues/514

Microsoft. (2026, August 20). *Azure OpenAI reasoning models—GPT-5 series,
o3-mini, o1, o1-mini*. Microsoft Learn.
https://learn.microsoft.com/en-us/azure/foundry/openai/how-to/reasoning

usestrix. (n.d.). *Configuration* [Computer software documentation]. GitHub.
Retrieved August 22, 2026, from
https://github.com/usestrix/strix/blob/main/docs/advanced/configuration.mdx
63 changes: 40 additions & 23 deletions scripts/ci/strix_quick_gate.sh

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔍 Manifest decision value never matched by case arms (pre-existing)

evaluate_pull_request_findings sets PR_FINDINGS_DECISION="block_manifest_finding" (strix_quick_gate.sh), but both case arms match block_manifest_unverified (strix_quick_gate.sh and :3992). Pre-existing, untouched here. Such findings still fail closed via fail_reported_vulnerabilities_before_fallback_success and never reach exit 3. The block_manifest_unverified arm appears dead.

(Refers to this code)

Open in Devin Review

Was this helpful? React with 👍 or 👎 to provide feedback.

Original file line number Diff line number Diff line change
Expand Up @@ -2212,28 +2212,6 @@ fallback_models_config_name_for_model() {
printf '%s\n' "STRIX_FALLBACK_MODELS"
}

has_distinct_fallback_model_for_model() {
local model="$1"
local fallback_models_raw
fallback_models_raw="$(fallback_models_raw_for_model "$model")"
fallback_models_raw="${fallback_models_raw//$'\r'/ }"
fallback_models_raw="${fallback_models_raw//$'\n'/ }"

local fallback_models=()
read -r -a fallback_models <<<"$fallback_models_raw"

local candidate_raw
local candidate
for candidate_raw in "${fallback_models[@]}"; do
candidate="$(normalize_model "$candidate_raw")"
if [ -n "$candidate" ] && [ "$candidate" != "$model" ]; then
return 0
fi
done

return 1
}

resolved_llm_api_base_for_model() {
local model="$1"

Expand Down Expand Up @@ -2660,6 +2638,19 @@ is_nvidia_nim_not_found_error() {
return 1
}

is_unsupported_model_parameter_error() {
# Strix currently has no generation-parameter override. Match the exact
# single-line LiteLLM/Azure capability failure so a reasoning model that
# rejects Strix's temperature can move to the already-configured fallback.
if grep -Ei 'litellm(\.exceptions)?\.BadRequestError' "$STRIX_LOG" |
grep -Ei '(AzureException|OpenAIException)' |
grep -Eiq "Unsupported value:[[:space:]]*['\"]temperature['\"].*Only the default[[:space:]]*\\(1\\)[[:space:]]*value is supported.*No fallback model group found"; then
return 0
fi

return 1
}

## Determines whether the last strix failure is a transient error eligible
## for same-model retry (up to STRIX_TRANSIENT_RETRY_PER_MODEL times).
## Four error families qualify:
Expand Down Expand Up @@ -2976,6 +2967,10 @@ has_detected_infrastructure_error() {
return 0
fi

if is_unsupported_model_parameter_error; then
return 0
fi

# Generic strix non-zero exit with known transport/connection errors
# that don't fall into the specific categories above.
# Use LLM_PROVIDER_ONLY_REGEX (not PROVIDER_CONTEXT_REGEX) to avoid
Expand Down Expand Up @@ -3857,6 +3852,10 @@ is_model_retryable_error() {
return 0
fi

if is_unsupported_model_parameter_error; then
return 0
fi

if [ "$PR_FINDINGS_DECISION" = "retry_model_inconsistency" ]; then
return 0
fi
Expand Down Expand Up @@ -3894,7 +3893,7 @@ run_current_target_scan() {

local strict_primary_provider_fallback=0
if [ "$INFRA_ERROR_DETECTED" -eq 1 ] && provider_signal_fail_closed_enabled; then
if is_model_retryable_error "$PRIMARY_MODEL" && has_distinct_fallback_model_for_model "$PRIMARY_MODEL"; then
if is_model_retryable_error "$PRIMARY_MODEL"; then
Comment thread
seonghobae marked this conversation as resolved.
strict_primary_provider_fallback=1
else
echo "Strix scan failed after provider infrastructure or failure-signal output; failing closed." >&2
Comment thread
seonghobae marked this conversation as resolved.
Expand Down Expand Up @@ -4024,6 +4023,24 @@ run_current_target_scan() {
return 1
fi

# The trusted PR-scope mapper has already proved these reports belong only
# to unchanged files. Keep that decision distinct from a real changed-file
# finding when every configured model path is unavailable.
if [ "$PR_FINDINGS_DECISION" = "allow_baseline" ] && [ "$INFRA_ERROR_DETECTED" -eq 1 ]; then
if [ "$fallback_tried" -eq 0 ]; then
if is_vertex_model "$PRIMARY_MODEL"; then
echo "Configured Vertex model was unavailable after unchanged-file findings were excluded." >&2
else
echo "Configured model was unavailable after unchanged-file findings were excluded." >&2
fi
elif is_vertex_model "$PRIMARY_MODEL"; then
echo "Configured Vertex model and fallback models were unavailable after unchanged-file findings were excluded." >&2
else
echo "Configured model and fallback models were unavailable after unchanged-file findings were excluded." >&2
fi
return 3
fi

if [ "$fallback_tried" -eq 0 ]; then
local fallback_config_name
fallback_config_name="$(fallback_models_config_name_for_model "$PRIMARY_MODEL")"
Expand Down
Loading
Loading