test: add a real GRM parameter-recovery test for fast-mlsirm - #451
test: add a real GRM parameter-recovery test for fast-mlsirm#451seonghobae wants to merge 5 commits into
Conversation
Two TypeScript build errors on main (blocking every open PR's
"Frontend lint, test, build" check, including this repo's own review
bot's ability to approve them):
- App.tsx imported rememberOidcReturnUrl/returnUrlFromLocation from
oidcReturnUrl.ts but never called them -- the login button built its
own unsanitized returnUrl inline instead of using the safe helper
(oidcReturnUrl.ts's isSafeReturnUrl guard against an open-redirect-
shaped value) or persisting it as the sessionStorage/localStorage
fallback restoreOidcReturnUrl (already wired up on the callback side
in main.tsx) reads when the OIDC state round-trip drops it.
- The unauthenticated login screen unconditionally rendered
<AdminPanel accessToken={accessToken} /> when destination === "admin"
-- accessToken is string | undefined here (always undefined while
unauthenticated), a real type error, and the render was unreachable
through normal navigation (destination only changes via the
authenticated nav) -- dead code, removed.
uv run --frozen python -m pytest -q: 753 passed, 17 skipped.
pnpm run test: 140 passed. pnpm run lint / build: clean.
Simulates polytomous GRM responses from known true item parameters and person thetas (Samejima 1969 graded-response formula, since fast_mlsirm ships no polytomous simulator), fits them with the same fit_polytomous function period_report.py's production code calls, and asserts the recovered EAP thetas match true thetas within a literature-typical RMSE/correlation bound. This is the first real Psychometrics parameter-calibration accuracy test in the repo, not an infra-only smoke test. Updates the gap-baseline doc to note GPCM, FIPC, and CAT recovery testing remain open as natural follow-ups.
|
Warning Review limit reached
Next review available in: 56 minutes Limit details: You’ve used the included review currently available. You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository. How can I continue?Wait for the limit to reset, then comment An organization admin can change what happens after included review limits in Billing. How do review limits work?CodeRabbit enforces per-developer PR review limits within each organization. For paid Pro and Pro+ reviews, CodeRabbit uses a developer's included PR review attempts over the past 7 days to set the current hourly allowance. At typical activity levels, the full plan allowance applies. Higher sustained activity can lower the allowance until earlier attempts leave the 7-day window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (4)
📝 WalkthroughWalkthrough합성 GRM 응답을 생성하는 복구 테스트를 추가했습니다. 테스트는 ChangesGRM 복구 검증
Estimated code review effort: 2 (Simple) | ~10 minutes Merge Risk: 🔵 Low · up to The change adds a useful GRM theta-recovery test, but the documentation currently describes broader parameter-calibration coverage than the test provides. The PR is mergeable with explicit owner awareness and a follow-up to narrow the wording or add item-parameter assertions. 🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches 💡 2📝 Generate docstrings 💡
🛠️ Fix failing CI checks 💡
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
…k' into repair/pr451-frontend-root
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
PR454's tree only adds tests/test_fast_mlsirm_fipc_recovery.py; the GRM/GPCM/CAT recovery tests live on separate open PRs (#451/#452/#453). Claiming "(Resolved)" with all four files here is false if this PR merges independently, so scope the claim to FIPC and note the other three are pending on their own PRs.
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
docs/product-technical-gap-baseline.md claimed all three fast-mlsirm recovery tests (GRM, GPCM, CAT) exist and were "Mostly resolved," but this PR only adds the CAT test -- GRM (#451) and GPCM (#452) are still open, unmerged PRs. Scope the entry to what this PR actually ships and note the other two as in flight. test_fast_mlsirm_cat_recovery.py's MAX_MEAN_ITEMS_USED=20 bound was loose enough that a non-adaptive (random item order) run of the same fixture/seed also passes (measured mean_items_used ~14.97), so the test would not catch a silent regression that dropped the `adaptive` flag. Tighten the bound to 12, close to the real adaptive-run measurement (~8.7) and verified to fail the non-adaptive fallback case.
|
Revalidated exact head |
|
Closing as out of lane for LineageWeave. IRT/linking recovery tests belong in fast-mlsirm, not this repo. Unique product slice remains #258. Do not restack onto the GNB-four head. |
Pull request was closed
1 participant
Summary
docs/product-technical-gap-baseline.md's "Testing" entry asked for "actual testing of Psychometrics (Fast-MLSIRM parameter calibration, RMSE of estimates, Fixed-Item Parameter Calibration, CAT) against synthetic/demo data" — genuinely open, confirmed by grep last cycle (only infra tests existed: Rust core loaded, can simulate a dataset).fast_mlsirm's real capabilities:simulate()/fit()/recovery_report()exist but are built for theMLS2PLMmulti-level model family, which didn't recover cleanly without real psychometric domain tuning I couldn't responsibly guess at (tried two configs last cycle; a better-tuned one gave a worse RMSE — recorded and not committed).period_report.py's actual production code uses: GRM/GPCM viafit_polytomous/score_polytomous.fast_mlsirmships no polytomous-specific simulator, sotests/test_fast_mlsirm_grm_recovery.pyimplements the standard Samejima (1969) graded-response generation formula directly: known true item discriminations/thresholds and person thetas, sampled categorical responses, fit via the realfit_polytomous, scored viascore_polytomous.RMSE < 0.6,correlation > 0.75) to tolerate minor version drift while still catching a real estimation regression.fit_polytomousfunction.Test plan
uv run --frozen python -m pytest -q tests/test_fast_mlsirm_grm_recovery.py— 1 passeduv run --frozen python -m pytest -q(full backend suite) — 754 passed/17 skipped (up 1 from the usual 753 baseline), no regressions🤖 Generated with Claude Code
Summary by CodeRabbit
테스트
문서