fix(strix): replace dead GitHub Models fallback with direct OpenAI - #1226
Conversation
GitHub Models entered a platform-wide "scheduled retirement brownout" (confirmed directly against models.github.ai: every request now returns HTTP 410). The universal STRIX_FALLBACK_MODELS fallback wired on 2026-07-13 for all four provider modes (github_models, openai_direct, openrouter, nvidia_nim) ended in `github_models/openai/o3 github_models/openai/gpt-5-chat`, so the fallback chain is now dead on every path -- not one model outage. Observed impact on LineageWeave PR #392: the NVIDIA NIM primary and its NVIDIA fallback both failed, cascaded through the now-dead GitHub Models fallback, and the required check failed outright instead of degrading gracefully. Separately, the nvidia_nim/nvidia/llama-3.3-nemotron-super-49b-v1.5 fallback that ran before the GitHub Models cascade produced a fabricated CRITICAL "hardcoded secret" finding against a frontend/src/config.ts line that does not exist in that form -- Strix's own in-run quality warning already flags this exact model as "not a recommended frontier model... weaker models may miss vulnerabilities or produce lower-quality findings", and this run is a concrete instance of that risk producing a false required-check failure. STRIX_FALLBACK_MODELS now falls back to openai-direct/gpt-5.6-luna (Strix's own top-recommended model) instead of the dead GitHub Models pair, on all four provider-mode branches. The nvidia_nim branch keeps its NVIDIA-hosted fallback as an interim retry before this openai-direct fallback, preserving the existing free/low-cost NVIDIA-first policy -- a separate cost/quality tradeoff this fix does not revisit. STRIX_OPENAI_API_KEY (secrets.STRIX_OPENAI_API_KEY || secrets.OPENAI_API_KEY) is already wired unconditionally, so no new secret is required. Updated the smoke test and the matching assertions in test_strix_quick_gate.sh to the new config; added an assert_file_not_contains guard so this dead dependency cannot silently return. Recorded the migration in PR_GOVERNANCE_AUDIT.md as a new dated entry (the prior 2026-07-13 entry is left as an immutable historical record). Validation: bash -n on both modified shell scripts, actionlint on the workflow, and scripts/ci/strix_required_workflow_smoke.sh (the fast, required-path governance test) all pass. The full test_strix_quick_gate.sh harness times out in this environment before completing even 1 of its 159 cases; PR #1149 (the most recent Strix regression-test change in this file) explicitly documents this harness as "environment-sensitive and not represented as green" and validates with the same bash-syntax/actionlint/governance-test tier used here. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Mk2aU7vk6Pn4pTnzcKT3Ws
|
Warning Review limit reached
Next review available in: 24 minutes Limit details: You’ve used the included review currently available. You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository. How can I continue?Wait for the limit to reset, then comment An organization admin can change what happens after included review limits in Billing. How do review limits work?CodeRabbit enforces per-developer PR review limits within each organization. For paid Pro and Pro+ reviews, CodeRabbit uses a developer's included PR review attempts over the past 7 days to set the current hourly allowance. At typical activity levels, the full plan allowance applies. Higher sustained activity can lower the allowance until earlier attempts leave the 7-day window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (5)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
| STRIX_TRANSIENT_RETRY_PER_MODEL: 2 | ||
| STRIX_TRANSIENT_RETRY_BACKOFF_SECONDS: 60 | ||
| STRIX_FALLBACK_MODELS: ${{ steps.gate.outputs.provider_mode == 'github_models' && 'github_models/openai/o3 github_models/openai/gpt-5-chat' || steps.gate.outputs.provider_mode == 'openai_direct' && 'github_models/openai/o3 github_models/openai/gpt-5-chat' || steps.gate.outputs.provider_mode == 'openrouter' && 'github_models/openai/o3 github_models/openai/gpt-5-chat' || steps.gate.outputs.provider_mode == 'nvidia_nim' && 'nvidia_nim/nvidia/llama-3.3-nemotron-super-49b-v1.5 github_models/openai/o3 github_models/openai/gpt-5-chat' || '' }} | ||
| STRIX_FALLBACK_MODELS: ${{ steps.gate.outputs.provider_mode == 'github_models' && 'openai-direct/gpt-5.6-luna' || steps.gate.outputs.provider_mode == 'openai_direct' && 'openai-direct/gpt-5.6-luna' || steps.gate.outputs.provider_mode == 'openrouter' && 'openai-direct/gpt-5.6-luna' || steps.gate.outputs.provider_mode == 'nvidia_nim' && 'nvidia_nim/nvidia/llama-3.3-nemotron-super-49b-v1.5 openai-direct/gpt-5.6-luna' || '' }} |
There was a problem hiding this comment.
🔴 Direct-OpenAI fallback never routes to a valid provider
Every provider mode's fallback is set to openai-direct/gpt-5.6-luna, but the gate only rewrites the underscore form openai_direct/ (child_model_for_api_base at scripts/ci/strix_quick_gate.sh:2310-2317), so this name reaches the model backend as unknown provider openai-direct and the fallback errors. When the primary model is exhausted the required check fails instead of degrading to direct OpenAI — the outcome this change aims to prevent. The primary-model step converts the hyphen form to underscore at strix.yml, but this fallback value skips that step.
Prompt for agents
The STRIX_FALLBACK_MODELS value at .github/workflows/strix.yml:815 uses the model name openai-direct/gpt-5.6-luna (hyphen) on all provider-mode branches. The Strix gate consumes STRIX_FALLBACK_MODELS directly without the hyphen-to-underscore conversion that the primary-model 'Prepare Strix model input file' step applies (strix.yml:762-763). The gate's child_model_for_api_base (scripts/ci/strix_quick_gate.sh:2310-2317) only strips the openai_direct/ (underscore) prefix down to openai/; it does not recognize openai-direct/ (hyphen), so the fallback name is passed to strix/LiteLLM as an unknown provider and the fallback fails. Change the fallback value to the gate-internal form openai_direct/gpt-5.6-luna on all four branches, and update the assertions that lock in the hyphen form: scripts/ci/strix_required_workflow_smoke.sh:159 and scripts/ci/test_strix_quick_gate.sh:349-351. Confirm consistency with the primary openai_direct model, which the gate already receives as openai_direct/gpt-5.6-luna.
Was this helpful? React with 👍 or 👎 to provide feedback.
| STRIX_TRANSIENT_RETRY_PER_MODEL: 2 | ||
| STRIX_TRANSIENT_RETRY_BACKOFF_SECONDS: 60 | ||
| STRIX_FALLBACK_MODELS: ${{ steps.gate.outputs.provider_mode == 'github_models' && 'github_models/openai/o3 github_models/openai/gpt-5-chat' || steps.gate.outputs.provider_mode == 'openai_direct' && 'github_models/openai/o3 github_models/openai/gpt-5-chat' || steps.gate.outputs.provider_mode == 'openrouter' && 'github_models/openai/o3 github_models/openai/gpt-5-chat' || steps.gate.outputs.provider_mode == 'nvidia_nim' && 'nvidia_nim/nvidia/llama-3.3-nemotron-super-49b-v1.5 github_models/openai/o3 github_models/openai/gpt-5-chat' || '' }} | ||
| STRIX_FALLBACK_MODELS: ${{ steps.gate.outputs.provider_mode == 'github_models' && 'openai-direct/gpt-5.6-luna' || steps.gate.outputs.provider_mode == 'openai_direct' && 'openai-direct/gpt-5.6-luna' || steps.gate.outputs.provider_mode == 'openrouter' && 'openai-direct/gpt-5.6-luna' || steps.gate.outputs.provider_mode == 'nvidia_nim' && 'nvidia_nim/nvidia/llama-3.3-nemotron-super-49b-v1.5 openai-direct/gpt-5.6-luna' || '' }} |
There was a problem hiding this comment.
🔍 openai-direct fallback uses the wrong key/base outside openai_direct mode
In openrouter, nvidia_nim, and github_models modes the new openai-direct fallback would authenticate with the primary provider's key and base, not OpenAI's: child_model_for_api_base reuses LLM_API_KEY/LLM_API_BASE_FILE for non-github, non-vertex models (scripts/ci/strix_quick_gate.sh:2244-2260, 2360-2366), and only STRIX_GITHUB_MODELS_KEY_FILE is provisioned for cross-provider fallback (strix.yml). secrets.STRIX_OPENAI_API_KEY/OPENAI_API_KEY is loaded as LLM_API_KEY only in openai_direct mode (strix.yml). So the graceful-degradation goal holds only for openai_direct primaries; the nvidia_nim branch (the motivating LineageWeave case) would still fail to reach direct OpenAI even after the prefix is corrected.
Was this helpful? React with 👍 or 👎 to provide feedback.
…ck tail exact-head-path-policy caught what the smoke test and test_strix_quick_gate.sh's direct config assertions missed: test_strix_nvidia_nim_not_found_fallback.py::test_workflow_uses_available_free_first_nvidia_plan still asserted the dead "nvidia_nim/nvidia/llama-3.3-nemotron-super-49b-v1.5 github_models/openai/o3 github_models/openai/gpt-5-chat" fallback string from the prior commit. Updated to the new "... openai-direct/gpt-5.6-luna" tail. Ran the full local suite after this fix (isolated .venv, no other changes): 1350 passed, 1 skipped, 16 subtests passed, 0 failures. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Mk2aU7vk6Pn4pTnzcKT3Ws
|
Why `strix` / "Self-test Strix required workflow contract" is expected to stay red on this PR Traced this precisely: the self-test step (`.github/workflows/strix.yml` lines ~416-422) runs `bash "$TRUSTED_STRIX_REQUIRED_SMOKE"` against `trusted-strix-source`, which is checked out at `steps.trusted_source.outputs.ref` — resolved from `job.workflow_sha`, i.e. always the base-branch (`main`) commit the workflow itself was triggered from, by GitHub's own `pull_request_target` trust guarantee. Only two files get materialized from the PR head into the separate `trusted-workspace` for testing (`.github/workflows/strix.yml` and `scripts/ci/pr_review_merge_scheduler.py`, lines ~380-388) — `scripts/ci/strix_required_workflow_smoke.sh` is not one of them. So this self-test always checks this PR's new `strix.yml` against `main`'s current (pre-merge) smoke test — which still asserts the dead `github_models/openai/o3 github_models/openai/gpt-5-chat` fallback string this PR removes. That's not a bug in this PR's smoke-test update; it's the intended self-referential trust boundary (a PR can't rewrite its own verification script and have that rewritten version certify it). Any PR that changes `STRIX_FALLBACK_MODELS`'s exact string content will hit this same red check, by design, until it merges. Everything this PR can self-verify locally passes: `bash -n` on both modified shell scripts, `actionlint` on the workflow, the local run of `scripts/ci/strix_required_workflow_smoke.sh` against this PR's own `strix.yml` (passes), the fixed `test_strix_nvidia_nim_not_found_fallback.py` assertion, and the full local Python suite (1350 passed, 1 skipped, 0 failures). `exact-head-path-policy`'s only failure was that same stale assertion, now fixed in `60af164`. Flagging this here so whoever merges understands the `strix` check's red state is structural, not a sign the fix is wrong — this may need an admin merge/override on this one check rather than waiting for it to go green on its own. |
There was a problem hiding this comment.
📝 Info: GitHub Models fallback credential step is now unused
The Prepare GitHub Models fallback credentials step (.github/workflows/strix.yml:666-687) provisions STRIX_GITHUB_MODELS_KEY_FILE/STRIX_GITHUB_MODELS_API_BASE_FILE, which the gate consumes only for github_models/* fallbacks. This PR removed every github_models/* entry from the fallback list, so the step now provisions credentials nothing reads. Its presence is still pinned by strix_required_workflow_smoke.sh:150 and test_strix_quick_gate.sh:352.
(Refers to this code)
Was this helpful? React with 👍 or 👎 to provide feedback.
There was a problem hiding this comment.
🔍 New fallback path has no functional test coverage
Only string-presence assertions were updated. The single executable fallback scenario (openai-direct-quota-github-models-fallback-success at test_strix_quick_gate.sh) still exercises a github_models/openai/o3 fallback routed to models.github.ai. Nothing runs an openai-direct fallback candidate, so the routing defects go undetected by the harness.
(Refers to this code)
Was this helpful? React with 👍 or 👎 to provide feedback.
opencode-agent
Bot
added
merge: urgent-review
|
Caution Review failedAn error occurred during the review process. Please try again later. Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
1 similar comment
|
Caution Review failedAn error occurred during the review process. Please try again later. Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Picks up #1226 (strix NVIDIA/OpenAI fallback fix), which this branch's own strix self-test was failing against since it predated that merge.
…time (#1223) * fix(scheduler): derive org-queue-sweep rotation tick from wall-clock time #1220 rotated org-queue-sweep's repository walk order by github.run_number to stop the fixed-order budget starvation from #1219, but run_number increments on every trigger of this workflow -- push, pull_request_target, pull_request_review, workflow_run -- not only the */15 sweep schedule, so it could not give the "bounded by repository_count ticks" guarantee the fix documented (review finding on #1220 from Devin, informational rather than blocking, so the original version merged before this correction landed). Derive the rotation tick from wall-clock time instead (ORG_SWEEP_ROTATION_INDEX = $(date -u +%s) / 900), which advances by exactly one every 15 minutes regardless of how many other events fired this workflow in between -- what the repository_count-tick bound actually requires. Left unset in the job's env block in production so the sweep step computes it; tests inject it directly for determinism. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * Merge branch 'main' into fix/org-queue-sweep-rotation-wall-clock Picks up #1226 (strix NVIDIA/OpenAI fallback fix), which this branch's own strix self-test was failing against since it predated that merge. * docs: fully-qualify #1219/#1220 cross-references in CHANGELOG.md CodeRabbit review on #1223: the entry mixed short-form #1220/#1219 with a fully-qualified #1220 reference in the same sentence. Every other cross-reference in this file already uses the fully-qualified owner/repo#N form; align these two. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * fix(scheduler): use a persistent per-execution counter for sweep rotation CodeRabbit's review on #1223 correctly identified that a wall-clock tick alone still cannot give the "bounded by repository_count executions" fairness guarantee: org-queue-sweep is single-flight, non-cancelling, and can run up to 60 minutes. A delayed or backlogged real execution can let more than one 900s window elapse before the next real run, and if that gap happens to be an exact multiple of the repository count the rotation offset repeats -- reintroducing the exact starvation #1220 fixed, for a different reason than the run_number bug #1220 itself had. Replace the wall-clock tick as the primary source with a persistent ORG_SWEEP_ROTATION_COUNTER repository variable on this (.github) repo, incremented by exactly one at the start of every actual sweep execution (PATCH, falling back to POST to create it on first use). This is immune to both prior failure modes: untouched by non-sweep triggers (unlike github.run_number) and advances only when the sweep body actually runs, regardless of how long a slow prior run took (unlike a wall-clock tick alone). The wall-clock tick remains as a degraded fallback if the counter read/write itself is unavailable -- a fairness mechanism must never fail the sweep's much more important review-dispatch/merge work. New tests stub `gh` on PATH to cover the successful-increment, create-on-first-run, and full-fallback paths without a real API call. 60/60 workflow contract tests pass, full suite 1360 passed / 1 skipped (pre-existing, unrelated). actionlint clean. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * fix(scheduler): base-10 counter arithmetic, stale comment, race/token notes Four more Devin findings on #1223's persistent-counter rotation fix: - Forced base-10 parsing ($((10#$counter_current + 1))) matching the same guard already used elsewhere in this file for STALE_OPENCODE_MINUTES. A manually-seeded leading-zero counter value (e.g. "08") passed the ^[0-9]+$ digit check but would have been parsed as an invalid octal literal by unprefixed arithmetic, erroring under set -e. Only reachable by manual variable seeding, not by this workflow's own writes, but cheap and real to fix. - Fixed a stale comment above the walk-order rotation that still said "by the run number" -- a leftover from before #1220/#1223's iterations replaced that source with the persistent counter. - Documented, rather than silently left unaddressed, two harder properties Devin correctly flagged and I'm not redesigning right now: (1) the counter's read-modify-write is not atomic, and a schedule-triggered run racing a manually-dispatched org_sweep=true run (different concurrency groups) could pick the same offset for that one pair of runs -- bounded and self-correcting, not a stuck state, and closing it fully needs either a compare-and-swap primitive the REST Variables API doesn't offer or a broader concurrency-group redesign shared across every trigger type this workflow serves; (2) whether the resolved token actually holds Variables-write scope isn't independently verifiable from inside the workflow -- if it doesn't, every run safely degrades to the wall-clock fallback (logged via ::warning:: each time), which is still strictly better than the pre-#1220 fixed order. New test covers the base-10 fix. 61/61 workflow contract tests, full suite 1376 passed / 1 skipped (pre-existing, unrelated). actionlint clean. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * fix(scheduler): don't let a transient counter-read failure reset the rotation A failed GET of ORG_SWEEP_ROTATION_COUNTER was treated identically to "the variable doesn't exist yet," so a transient read error could PATCH the counter back down to 1 and restart the rotation sequence. Distinguish a successful read (authoritative, even if malformed -> sanitized to 0, safe to PATCH) from a failed read (unknown state -> only a create-on-first-run POST is safe, never a PATCH; falls through to the wall-clock tick otherwise). Devin review finding on #1223. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
1 participant
Summary
models.github.ai: every request now returns HTTP 410github_models_retirement_brownout).STRIX_FALLBACK_MODELSended ingithub_models/openai/o3 github_models/openai/gpt-5-chaton all four provider-mode branches, so that fallback is now dead on every path.Configured model and fallback models were unavailable.).nvidia_nim/nvidia/llama-3.3-nemotron-super-49b-v1.5fallback that did run before the dead-fallback cascade produced a fabricated CRITICAL finding — a "hardcoded secret" report against afrontend/src/config.tsline that doesn't exist in that form on the target branch. Strix's own in-run quality warning already flags this exact model as "not a recommended frontier model... weaker models may miss vulnerabilities or produce lower-quality findings" — this run is a concrete instance of that risk turning into a false required-check block.STRIX_FALLBACK_MODELSnow falls back toopenai-direct/gpt-5.6-luna(Strix's own top-recommended model) instead of the dead GitHub Models pair, on all four branches.nvidia_nimkeeps its NVIDIA-hosted interim fallback before this (unchanged cost/quality tradeoff, out of scope here).STRIX_OPENAI_API_KEY(secrets.STRIX_OPENAI_API_KEY || secrets.OPENAI_API_KEY) is already wired unconditionally — no new secret needed.scripts/ci/strix_required_workflow_smoke.shand the matching assertions inscripts/ci/test_strix_quick_gate.shto the new config, plus anassert_file_not_containsguard so the dead dependency can't silently return.PR_GOVERNANCE_AUDIT.md(the prior 2026-07-13 entry is left as an immutable historical record, per this doc's own convention).Test plan
bash -non both modified shell scriptsactionlint .github/workflows/strix.ymlscripts/ci/strix_required_workflow_smoke.sh(the fast, required-path governance test) — passesscripts/ci/test_strix_quick_gate.sh(full 159-case harness) — times out before completing case 1 in this environment; PR test(ci): expose Strix provider fallback filter case #1149 (the most recent Strix regression-test change to this file) explicitly documents this harness as "environment-sensitive and not represented as green" and validates with the same bash-syntax/actionlint/governance-test tier used here🤖 Generated with Claude Code
https://claude.ai/code/session_01Mk2aU7vk6Pn4pTnzcKT3Ws