feat(event): score mention confidence with a known-truth Brier rule - #64
Conversation
Perfect forecasts recover Brier 0 and constant 0.5 recovers 0.25. Empty or mismatched streams fail closed. No new migration.
|
Warning Review limit reachedNext included review available in 19 minutes. View limit detailsLimit details: You’ve used the included review currently available. You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository. Review configuration: ⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (8)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
|
Revalidated current head |
# Conflicts: # CHANGELOG.md # DOCUMENTATION.md # docs/TRACEABILITY.md # docs/validation/temporal-event-foundation.md
|
Current-head refresh completed for |
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
|
Fixed current-head validation-ledger duplicates at
Hosted Checks are running for this exact head; no bypass or merge was performed. |
|
Current-head review refresh for :\n\n- The mention-confidence implementation keeps calibration as a known-truth Brier-rule measurement rather than treating a confidence score as scientific authority.\n- , documentation validation, workspace contract, docstring contract, and passed locally.\n- All hosted exact-head checks, including Rust tests, PostgreSQL integration, coverage, security scans, and Strix, pass.\n- No new actionable defect was found at this exact head; no approval is being self-issued. |
|
Current-head review refresh for 22a5955:
|
|
Exact-head review receipt: current-main conflict repair was completed and pushed at |
![opencode-agent[bot]](https://avatars.githubusercontent.com/in/1549082?s=60&v=4){kind=small}
There was a problem hiding this comment.
Pull request overview
OpenCode cannot approve yet because required coverage evidence did not pass.
Review outcome
1. HIGH .github/workflows/opencode-review.yml:1 - Coverage evidence did not prove required test/docstring evidence
-
Problem: The required coverage-evidence job result was
failure, so OpenCode cannot establish approval sufficiency for this head. -
Root cause: Automated approval is only valid when the same-head coverage-evidence job proves supported repository test suites passed and configured docstring gates passed or were advisory, or reports not applicable because no supported source files or package manifests exist. Missing, failed, skipped, unavailable, or unsupported-tooling test evidence is a blocker.
-
Fix: Install or configure the repository test/docstring evidence tooling when source files or package manifests exist, rerun the current-head coverage-evidence job, and approve only after it reports
successwith required evidence or explicit no-source not-applicable evidence. -
Regression test: Keep the approval branch checking
needs.coverage-evidence.result == successbefore posting APPROVE, and publish REQUEST_CHANGES when coverage-evidence blocker states such as cancelled, skipped, failed, unsupported-tooling, or below-100 evidence are present. -
Result: REQUEST_CHANGES
-
Reason: coverage-evidence result was
failure, so required test/docstring evidence was not proven for current headc11adce997d4f3e462f50711211af9f051e747b0. -
Head SHA:
c11adce997d4f3e462f50711211af9f051e747b0 -
Workflow run: 32593778852
-
Workflow attempt: 1
Coverage evidence
Coverage Decision
- Result: FAIL
- Test evidence: not proven passing
- Docstring evidence: not proven passing when configured
- Failure count: 1
Changed-File Evidence Map
flowchart LR
PR["PR changed files"] --> Evidence["OpenCode bounded evidence"]
Evidence --> S1["Changed file (5 files)"]
S1 --> I1["repository behavior"]
I1 --> R1["Review risk: Changed file (5 files)"]
R1 --> V1["required checks"]
Evidence --> S2["Docs (5 files)"]
S2 --> I2["operator or user guidance"]
I2 --> R2["Review risk: Docs (5 files)"]
R2 --> V2["docs review"]
OpenCode Review Overview
Pull request overviewOpenCode cannot approve yet because required coverage evidence did not pass. Review outcome1. HIGH .github/workflows/opencode-review.yml:1 - Coverage evidence did not prove required test/docstring evidence
Coverage evidenceCoverage Decision
Changed-File Evidence Mapflowchart LR
PR["PR changed files"] --> Evidence["OpenCode bounded evidence"]
Evidence --> S1["Changed file (5 files)"]
S1 --> I1["repository behavior"]
I1 --> R1["Review risk: Changed file (5 files)"]
R1 --> V1["required checks"]
Evidence --> S2["Docs (5 files)"]
S2 --> I2["operator or user guidance"]
I2 --> R2["Review risk: Docs (5 files)"]
R2 --> V2["docs review"]
|
Stale coverage-evidence CHANGES_REQUESTED. Exact head c11adce now has coverage-evidence success. Not an approval.
…e-brier # Conflicts: # CHANGELOG.md # DOCUMENTATION.md # docs/TRACEABILITY.md # docs/validation/temporal-event-foundation.md
| pub fn mention_brier_score( | ||
| forecasts: &[EventConfidence], | ||
| outcomes: &[bool], | ||
| ) -> Result<f64, EventError> { | ||
| if forecasts.is_empty() || forecasts.len() != outcomes.len() { | ||
| return Err(EventError::InvalidWirePayload); | ||
| } | ||
| let mut square_sum = 0.0_f64; | ||
| for (forecast, outcome) in forecasts.iter().zip(outcomes) { | ||
| let target = if *outcome { 1.0 } else { 0.0 }; | ||
| let residual = forecast.value() - target; | ||
| square_sum += residual * residual; | ||
| } | ||
| #[allow(clippy::cast_precision_loss)] | ||
| Ok(square_sum / forecasts.len() as f64) | ||
| } |
There was a problem hiding this comment.
📝 Info: Brier score computation is correct and fail-closed
mention_brier_score (confidence.rs) computes the mean of squared residuals against binary truth, matching the standard Brier score. It fails closed on empty or length-mismatched inputs via EventError::InvalidWirePayload, and EventConfidence already guarantees finite values in [0,1], so no NaN/overflow risk exists. The logic verified against the documented test vectors (perfect 0, constant-0.5 = 0.25).
Was this helpful? React with 👍 or 👎 to provide feedback.
| ## Authoritative sources | ||
|
|
||
| Brier, G. W. (1950). Verification of forecasts expressed in terms of probability. *Monthly Weather Review, 78*(1), 1–3. https://doi.org/10.1175/1520-0493(1950)078<0001:VOFEIT>2.0.CO;2 | ||
|
|
||
| Gneiting, T., & Raftery, A. E. (2007). Strictly proper scoring rules, prediction, and estimation. *Journal of the American Statistical Association, 102*(477), 359–378. https://doi.org/10.1198/016214506000001437 | ||
|
|
||
| ## Application | ||
|
|
||
| Brier (1950) defines the mean squared error of a probability forecast. Gneiting and Raftery (2007) treat the Brier score as a strictly proper scoring rule, so a mention that is certain when true and impossible when false is uniquely optimal. TEPP therefore scores mention confidence against known binary outcomes rather than treating a high score as an event instance (Brier, 1950; Gneiting & Raftery, 2007). |
There was a problem hiding this comment.
📝 Info: APA 7 reference placed in separate doctoring note, matching repo pattern
CONTRIBUTING.md states APA 7 references are recorded in docs/research/standards-and-literature.md, and the Brier/Gneiting references are NOT added there. I did not flag this because the established repo convention (e.g. docs/research/adaptive-orchestration-router.md) is per-capability doctoring notes carrying their own "Authoritative sources" section, which mention-confidence-brier.md follows, and AGENTS.md rule 13 only requires citation somewhere in docs/research/. Reviewer may still want to confirm whether the central register should be kept in sync.
Was this helpful? React with 👍 or 👎 to provide feedback.
1 participant
Summary
#37–#63 remain review-blocked (no independent APPROVE). #37 and #45 kept ready. Next no-0008 gap: mention-confidence Brier score in existing
event_core(does not recreate the TDT/CHRONOS #50 slice).(0,1,0,1)forecasts recover Brier00.5recovers0.25with computed residual RMSETest plan
confidence_calibration_contractfailed withE0432cargo test -p event_core --offlinecargo clippy -p event_core --all-targets --offline -- -D warningspython3 scripts/validate_documentation.pyandcheck_docstrings.pyDo not self-approve or merge. Prefer merge #37 then #45 when eligible. Do not allocate 0008.