fix(report): flag a partial LLM failure as degraded, not only a total one

[AmirF194]

What

report() derives degraded from _llm_runtime_status(), which only set it when every attempted LLM call failed (succeeded == 0). A rate-limited provider that drops one batch (e.g. a 429 on semantic_security_discovery) still has succeeded > 0, so the existing fail-closed floor (CAUTION instead of SAFE on a degraded scan) never triggered. The reported scenario is exactly this: llm_calls_attempted=4, llm_calls_succeeded=3, risk_assessment still SAFE, and the dropped batch happened to be the one analyzer that would have caught the malicious skill in the report.

This widens the condition to succeeded < attempted, so any dropped or throttled batch marks the scan degraded, not just a total failure. The two degraded-scan messages (_llm_degradation_notice, meta["llm_error"]) are updated to say how many of the calls failed instead of assuming all of them did, since that is no longer always true.

Test

Added test_partial_llm_failure_also_floors_recommendation_at_caution, matching the reported 3/4 scenario directly against report(). Renamed the test that pinned the old behavior (test_report_not_degraded_when_some_calls_succeeded -> test_report_degraded_when_some_calls_fail) to assert the corrected one. Full tests/nodes/test_report.py green (63 passed); make lint and make format-check clean.

Scope

This covers request 3 of #303 (surface incompleteness in the verdict). Request 1 (configurable concurrency) shipped in #305; request 2 (retry with backoff) is left to the already-open #29, which this PR does not touch or conflict with (it does not modify _llm_runtime_status).

Refs #303

[rng1995]

The report-level predicate now handles mixed analyzer outcomes, but the core partial-batch path is still recorded as a success upstream, so a multi-batch analyzer can drop work and still produce SAFE. The new predicate also conflates overall coverage with whether the meta-analyzer/provider actually ran. Please address the inline findings so the degraded verdict and metadata are accurate end to end.

[AmirF194]

Pushed a follow-up commit for both findings.

P1 (line 587): llm_call_log records were built with ok=bool(outcome.successful) or not outcome.failures, so a batch that partially failed (one file's batch succeeds, another 429s) still recorded ok=True as long as any batch came back. Changed all four call sites (the three semantic analyzers and meta_analyzer) to ok=not outcome.failures, so any dropped batch marks the record failed. test_partial_batch_failure_records_llm_success in test_semantic_developer_intent.py pinned the old behavior; renamed it to test_partial_batch_failure_records_llm_failure and inverted the assertion. Added test_analyzer_partial_batch_failure_flows_through_to_report_degraded in test_report.py, which drives the real semantic_developer_intent.node() through a mocked one-success/one-timeout batch outcome and feeds its actual llm_call_log output into report(), asserting llm_degraded and the CAUTION floor. That's an end-to-end test, not just a predicate test.

P2 (line 617/626): meta_analysis_applied and llm_available were derived from the aggregate degraded flag, which pools every LLM-backed node together, so a different analyzer's dropped batch could force meta_analysis_applied=False and llm_available=False even when meta_analyzer itself fully succeeded. Both fields now derive from is_llm_available() plus meta_analyzer's own llm_call_log record specifically (a missing record, e.g. no findings to filter, reads as vacuously ok). The coverage loss from other analyzers is unchanged and still surfaces through llm_degraded / llm_calls_attempted / llm_calls_succeeded. Added two tests: one for the 3/4 scenario you described (meta_analyzer ok, one semantic analyzer's batch dropped) asserting meta_analysis_applied/llm_available stay True while llm_degraded stays True; one where meta_analyzer's own record is the failure, asserting both fields correctly go False regardless of the other analyzers.

I did not add a separate coverage field beyond the existing llm_degraded / llm_calls_attempted / llm_calls_succeeded trio; those already report exactly which fraction of calls dropped, so a new field seemed redundant, but happy to add one if you had something more specific in mind.

Verified: ran the new/changed tests against the pre-fix code first to confirm they fail (3 of 4 red, the meta_analyzer-failure case already passed under the old formula since that one case wasn't actually broken), then confirmed green after the fix. tests/nodes/test_report.py: 66 passed. Full suite in a clean python:3.12-slim Docker container: 1947 passed, 13 skipped, 4 xfailed, 0 failed. ruff check and ruff format --check both pass. I did not exercise a real 429 against a live provider; the batch failures are simulated via a mocked arun_batches outcome, same approach the existing test suite uses throughout this file.

[AmirF194]

No rush, just flagging this is still open. Pushed a fix for both review findings on 08-12 (full suite green), but I can't request a re-review myself and the CI run for that commit is still waiting on the approval gate.

[rng1995]

[SkillSpector Review]

Re-review: still requesting changes. Partial-batch failures now correctly reach the degraded verdict, but the metadata contract remains wrong: all([]) marks meta_analysis_applied true when the no-findings path made no meta-analyzer call at all. Require an actual successful meta-analyzer record for that field and add the no-record regression described inline; provider availability should remain separate.

[AmirF194]

You're right, that was a real gap. all([]) is True on an empty list, so my prior fix left meta_analysis_applied True whenever meta_analyzer never made a call at all, which is exactly the no-findings path.

Changed src/skillspector/nodes/report.py: meta_analysis_applied now requires meta_analyzer_records to be non-empty and all of them ok (meta_analyzer_succeeded = bool(meta_analyzer_records) and meta_analyzer_ok), so an empty record list can no longer satisfy it vacuously. llm_available is untouched and still uses the original meta_analyzer_ok (vacuously true on an empty list), so provider availability stays a separate contract from whether meta_analyzer had anything to filter, per your note.

Added test_report_meta_analysis_not_applied_when_no_meta_analyzer_record in tests/nodes/test_report.py: empty llm_call_log, use_llm=True, provider available, asserts meta_analysis_applied is False, llm_available is True, filtering_mode == "heuristic".

Verified in a clean python:3.12-slim Docker container this session: the new test fails on the pre-fix code (meta_analysis_applied came back True) and passes after the fix. Full tests/nodes/test_report.py (67 tests), make lint, and make format-check are all green on the current commit.

[AmirF194]

Rebased onto main, no conflicts in the source (only a test file collided with a new upstream test added at the same location, both kept). Full suite, ruff lint and format-check all clean at the new head.

[rng1995]

[SkillSpector Review]

Re-review approved. The remaining metadata blocker is resolved: meta_analysis_applied now requires at least one successful meta_analyzer record, while provider availability remains independent when no meta-analysis call was needed. The no-record regression covers the prior all([]) gap, and all required checks pass.

[AmirF194]

Thanks for catching both review gaps, the empty-list all([]) one especially would have been an easy miss. Glad the rebase went in clean.