From 7d5940ef5c23276036c4a50735ce642c5add9619 Mon Sep 17 00:00:00 2001 From: Tom Softreck Date: Tue, 4 Aug 2026 13:58:08 +0200 Subject: [PATCH 1/3] plan(llm): restore confidence hierarchy coverage --- TODO.md | 5 ++ project/TICKETS.md | 1 + project/ticket-030/README.md | 69 ++++++++++++++++++++++++++++ project/ticket-030/ai-codex-logs.txt | 1 + project/ticket-030/ai-codex.md | 55 ++++++++++++++++++++++ project/ticket-030/changelog.md | 15 ++++++ project/ticket-030/intent.json | 17 +++++++ project/ticket-030/preprompt.md | 8 ++++ 8 files changed, 171 insertions(+) create mode 100644 project/ticket-030/README.md create mode 100644 project/ticket-030/ai-codex-logs.txt create mode 100644 project/ticket-030/ai-codex.md create mode 100644 project/ticket-030/changelog.md create mode 100644 project/ticket-030/intent.json create mode 100644 project/ticket-030/preprompt.md diff --git a/TODO.md b/TODO.md index 7ce22cb..35019d1 100644 --- a/TODO.md +++ b/TODO.md @@ -2,6 +2,11 @@ ## Active tickets +- [ ] [`ticket-030`](project/ticket-030/README.md) — repair the stale + confidence-hierarchy source lookup after the NL and Markdown clamp logic + moved into focused helper modules. Current state: `IN_PROGRESS / VALIDATION`; + the production ceilings remain unchanged and the implementation is test-only. + - [ ] [`ticket-019`](project/ticket-019/README.md) — publish the dependency-free Python SDK as the root PyPI distribution `todo2code` through `goal -a`, with one root `pyproject.toml` and SDK-only artifacts. Current state: diff --git a/project/TICKETS.md b/project/TICKETS.md index 071ccc8..b1968fc 100644 --- a/project/TICKETS.md +++ b/project/TICKETS.md @@ -27,4 +27,5 @@ of `project/README.md`, which remains a generated technical-analysis artifact. | **ticket-019** | [`README.md`](./ticket-019/README.md) | [`preprompt.md`](./ticket-019/preprompt.md) | - | [`ai-codex.md`](./ticket-019/ai-codex.md) | [`ai-codex-logs.txt`](./ticket-019/ai-codex-logs.txt) | [`changelog.md`](./ticket-019/changelog.md) | | **ticket-020** | [`README.md`](./ticket-020/README.md) | [`preprompt.md`](./ticket-020/preprompt.md) | - | [`ai-codex.md`](./ticket-020/ai-codex.md) | [`ai-codex-logs.txt`](./ticket-020/ai-codex-logs.txt) | [`changelog.md`](./ticket-020/changelog.md) | | **ticket-022** | [`README.md`](./ticket-022/README.md) | [`preprompt.md`](./ticket-022/preprompt.md) | - | [`ai-codex.md`](./ticket-022/ai-codex.md) | [`ai-codex-logs.txt`](./ticket-022/ai-codex-logs.txt) | [`changelog.md`](./ticket-022/changelog.md) | +| **ticket-030** | [`README.md`](./ticket-030/README.md) | [`preprompt.md`](./ticket-030/preprompt.md) | - | [`ai-codex.md`](./ticket-030/ai-codex.md) | [`ai-codex-logs.txt`](./ticket-030/ai-codex-logs.txt) | [`changelog.md`](./ticket-030/changelog.md) | diff --git a/project/ticket-030/README.md b/project/ticket-030/README.md new file mode 100644 index 0000000..d2e3d8f --- /dev/null +++ b/project/ticket-030/README.md @@ -0,0 +1,69 @@ +# Ticket 030: Repair confidence hierarchy test after extractor refactor + +- **ID**: ticket-030 +- **Owner**: unresolved:human +- **Status**: IN_PROGRESS +- **Workflow state**: VALIDATION +- **Created**: 2026-08-04 + +## Goal and scope + +Repair the confidence-hierarchy regression test that failed after the Markdown +and NL confidence clamping logic moved from the orchestration modules into +focused helper modules. The production ceilings remain correct (`0.94`, +`0.90`, `0.85`); the failure is a test-discovery defect, not evidence that an +Intent record contains `confidence: null`. + +The implementation is deliberately test-only. It will update the hierarchy +test so it verifies the canonical current locations without changing any +extractor behavior, confidence value, documentation contract or LLM boundary. +No unrelated refactor or generated artifact belongs to this ticket. + +## Planned changed paths + +- `test/nl-llm.test.ts`: replace the stale source-file assumptions with a + deterministic check of the current confidence-clamping implementations. +- `project/ticket-030/**`, `TODO.md`, `project/TICKETS.md`: ticket intent and + verification evidence only. + +## Acceptance criteria + +- [x] AC-01: Scope is approved by a human owner. +- [x] AC-02: The test distinguishes a missing source-code match from a runtime + confidence value and no longer reports the former as extractor failure. +- [x] AC-03: The hierarchy remains Markdown `0.94` > NL `0.90` > documentation + `0.85`, with all three ceilings below deterministic observation levels. +- [x] AC-04: Focused NL/Markdown/document tests pass without a live provider, + changing production code or accepting a looser confidence ceiling. +- [x] AC-05: The ticket diff contains only its allowed test and governance + paths; unrelated dirty/generated files remain untouched. + +## Participants + +- Human participant: unresolved; no user-* file was created by this script. +- Agent participant: [ai-codex.md](ai-codex.md) + +## Approval boundary + +- Current state: `IN_PROGRESS / VALIDATION`. +- Required response from: `unresolved:human`. +- Chat approval authorizes implementation in this interactive session but is + not trusted merge evidence. +- The ticket was renumbered from the locally drafted `ticket-023` to + `ticket-030` before publication because remote PR #3 already owns 023. The + approved implementation scope and code delta did not change. + +## Validation evidence + +- The freshly emitted `dist/test/nl-llm.test.js` passes `11/11`; the repaired + hierarchy assertion passes and no live provider is called. +- The complete `npm test` run reports 338 tests: 337 passed, 0 failed and one + environment-dependent JDK test skipped. The test-only repair was emitted over + the last healthy build because unrelated source parser errors prevent a new + whole-project TypeScript emit. +- `npm run verify` reaches `tsc` and stops on three inherited parser errors in + `src/cli.ts`, `src/core/types/code-change.ts` and + `src/semantic/reranker/result.ts`. None is in ticket-030 scope. +- Scoped `git diff --check` passes. Whole-repository governance continues to + report only the pre-existing ticket-018/ticket-019 conflict, dependency, + workstream ownership and shared `Makefile` overlap. diff --git a/project/ticket-030/ai-codex-logs.txt b/project/ticket-030/ai-codex-logs.txt new file mode 100644 index 0000000..8b13789 --- /dev/null +++ b/project/ticket-030/ai-codex-logs.txt @@ -0,0 +1 @@ + diff --git a/project/ticket-030/ai-codex.md b/project/ticket-030/ai-codex.md new file mode 100644 index 0000000..4fe2e01 --- /dev/null +++ b/project/ticket-030/ai-codex.md @@ -0,0 +1,55 @@ +--- +participant-id: agent:codex +participant: codex +role: agent +ticket: ticket-030 +--- +# Participant: codex (AI agent) + +## Understanding + +The failed assertion's `actual: null` is the result of +`source.match(...)`, not the confidence stored in an Intent record. The test +still reads `markdown-llm.ts` and `nl-llm.ts`, while their clamping expressions +now live in `markdown-llm-helpers.ts` and `nl-llm-helpers.ts`. The documented +and runtime ceilings remain `0.94`, `0.90` and `0.85`. + +The narrow repair belongs to the `llm` workstream and does not overlap the +active governance or SDK implementation paths. It must not absorb the user's +concurrent synthesis refactor or generated `project/` artifacts. + +## Execution plan + +1. Stop at `WAIT_FOR_APPROVAL` before editing the test. +2. Update only the confidence-hierarchy test to inspect the canonical current + implementations, retaining strict numeric assertions and ordering. +3. Run the focused compiled/TypeScript tests without a live OpenRouter call. +4. Review the exact diff and run scoped governance; record any inherited + repository failures separately from this test-only change. + +## Actual changes + +- The user explicitly approved the bounded ticket-030 plan in chat. The + implementation session is authorized; merge approval remains external. +- Updated only `test/nl-llm.test.ts`: the hierarchy test now reads the Markdown + and NL helper modules that own their confidence clamps after refactoring. +- Emitted a disposable build despite inherited parser errors and ran the + focused compiled test: 11 passed, 0 failed, with the hierarchy assertion + passing. No provider request was made. +- Ran the complete test suite in the repository's standard `dist` layout with + the repaired test emitted over the last healthy build: 338 tests, 337 passed, + 0 failed and one JDK-dependent skip. This isolates and validates the approved + test-only delta without concealing the unrelated compile failures. +- Renumbered the unpublished local ticket from 023 to 030 after discovering + that GitHub PR #3 already owns ticket-023; no implementation path or + acceptance criterion changed. + +## Blockers + +- Whole-repository governance already fails because ticket-018 and ticket-019 + declare a conflict and unfinished dependency. This ticket does not rewrite + either intent or claim that unrelated blocker. +- The main worktree contains user-owned generated and synthesis changes. Work + remains isolated in a separate branch/worktree. +- Full `npm run verify` is blocked before this test runs by three source parser + errors outside the approved ticket paths. diff --git a/project/ticket-030/changelog.md b/project/ticket-030/changelog.md new file mode 100644 index 0000000..4227724 --- /dev/null +++ b/project/ticket-030/changelog.md @@ -0,0 +1,15 @@ +# Ticket Changelog (ticket-030) + +## [0.1.0] - 2026-08-04 + +- Initial governance scaffold created. +- No human participant identity or content was generated. + +## [0.1.1] - 2026-08-04 + +- Repaired the stale Markdown/NL confidence-clamp source locations in the + hierarchy test without changing production confidence behavior. +- Focused compiled NL LLM test passes 11/11 and the complete suite passes all + 337 runnable tests (one environment skip); recorded unrelated parser and + governance blockers separately. +- Renumbered the unpublished ticket to 030 to avoid colliding with active PR #3. diff --git a/project/ticket-030/intent.json b/project/ticket-030/intent.json new file mode 100644 index 0000000..499db70 --- /dev/null +++ b/project/ticket-030/intent.json @@ -0,0 +1,17 @@ +{ + "schema": "new-project.intent/v2", + "ticket": "ticket-030", + "summary": "Repair confidence hierarchy test after extractor refactor", + "workstream": "llm", + "allowedPaths": [ + "test/nl-llm.test.ts", + "project/ticket-030/**", + "TODO.md", + "project/TICKETS.md" + ], + "forbiddenPaths": ["project/ticket-*/user-*.md"], + "stacks": ["node"], + "dependsOn": [], + "conflictsWith": [], + "integrationTicket": null +} diff --git a/project/ticket-030/preprompt.md b/project/ticket-030/preprompt.md new file mode 100644 index 0000000..0086e90 --- /dev/null +++ b/project/ticket-030/preprompt.md @@ -0,0 +1,8 @@ +# Ticket preprompt + +- **Task ID**: ticket-030 +- **Task title**: Repair confidence hierarchy test after extractor refactor +- **Created**: 2026-08-04T09:30:20Z + +Keep executable implementation outside this governance/evidence directory. +Read a human-owned user-*.md file only when one exists. From 9c2594217fab794094bc4d9bd6279bc7712afa8c Mon Sep 17 00:00:00 2001 From: Tom Softreck Date: Tue, 4 Aug 2026 14:01:02 +0200 Subject: [PATCH 2/3] test(llm): repair confidence hierarchy lookup Ticket: ticket-030 Workstream: llm --- test/nl-llm.test.ts | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/test/nl-llm.test.ts b/test/nl-llm.test.ts index b6eb2c1..7fa5cca 100644 --- a/test/nl-llm.test.ts +++ b/test/nl-llm.test.ts @@ -192,8 +192,8 @@ test('The documented confidence hierarchy holds across LLM extractors', async () // Read the TypeScript sources from the repository root: the compiled test // runs out of dist/, where the .ts files do not exist. const sources = await Promise.all([ - fs.readFile(path.resolve('src/extractors/markdown-llm.ts'), 'utf8'), - fs.readFile(path.resolve('src/extractors/nl-llm.ts'), 'utf8'), + fs.readFile(path.resolve('src/extractors/markdown-llm-helpers.ts'), 'utf8'), + fs.readFile(path.resolve('src/extractors/nl-llm-helpers.ts'), 'utf8'), fs.readFile(path.resolve('src/extractors/docs-record.ts'), 'utf8'), ]); const ceilings = sources.map((source) => { From 2b7a4a23ada8e001d0a024cdac504b2609e49afa Mon Sep 17 00:00:00 2001 From: Tom Softreck Date: Tue, 4 Aug 2026 14:02:06 +0200 Subject: [PATCH 3/3] docs(ticket-030): record clean validation blockers Ticket: ticket-030 Workstream: llm --- project/ticket-030/README.md | 16 +++++++++------- project/ticket-030/ai-codex-logs.txt | 9 ++++++++- project/ticket-030/ai-codex.md | 19 ++++++++++--------- project/ticket-030/changelog.md | 7 ++++--- 4 files changed, 31 insertions(+), 20 deletions(-) diff --git a/project/ticket-030/README.md b/project/ticket-030/README.md index d2e3d8f..da4d376 100644 --- a/project/ticket-030/README.md +++ b/project/ticket-030/README.md @@ -57,13 +57,15 @@ No unrelated refactor or generated artifact belongs to this ticket. - The freshly emitted `dist/test/nl-llm.test.js` passes `11/11`; the repaired hierarchy assertion passes and no live provider is called. -- The complete `npm test` run reports 338 tests: 337 passed, 0 failed and one - environment-dependent JDK test skipped. The test-only repair was emitted over - the last healthy build because unrelated source parser errors prevent a new - whole-project TypeScript emit. -- `npm run verify` reaches `tsc` and stops on three inherited parser errors in - `src/cli.ts`, `src/core/types/code-change.ts` and - `src/semantic/reranker/result.ts`. None is in ticket-030 scope. +- The reported 338-test run used an older `dist` tree: it correctly exposed the + stale hierarchy lookup, but by itself was not proof that current TypeScript + sources build cleanly. After rebasing onto base `caf6551`, a clean + `npm run build` passes. +- The full freshly built suite reports 338 tests: 327 passed, 10 failed and one + skipped. The repaired confidence-hierarchy test passes. The ten inherited + failures concern AST/linker relations, communication identity/enrichment and + gold-dataset relations; none is in ticket-030 scope or caused by its two-line + source-location update. - Scoped `git diff --check` passes. Whole-repository governance continues to report only the pre-existing ticket-018/ticket-019 conflict, dependency, workstream ownership and shared `Makefile` overlap. diff --git a/project/ticket-030/ai-codex-logs.txt b/project/ticket-030/ai-codex-logs.txt index 8b13789..6c2070d 100644 --- a/project/ticket-030/ai-codex-logs.txt +++ b/project/ticket-030/ai-codex-logs.txt @@ -1 +1,8 @@ - +2026-08-04 ticket-030 validation +- `node --test dist/test/nl-llm.test.js`: PASS (11/11). +- `npm run build`: FAIL (17 inherited TypeScript diagnostics outside allowed implementation path). +- `npm test` after the failed fresh emit: FAIL (292 total, 253 pass, 38 fail, 1 skip); base import/runtime regressions exposed. +- The user-reported 338-test run was based on stale `dist`; useful for reproducing the hierarchy assertion only, not for clean-source verification. +- `make governance`: FAIL with the four inherited ticket-018/ticket-019 diagnostics (`GOV-CONFLICT-001`, `GOV-DEPENDENCY-002`, `GOV-WORKSTREAM-003`, `GOV-WORKSTREAM-004`); no ticket-030 diagnostic was reported. +- Rebased onto updated PR-5 base `caf6551`: `npm run build` PASS. +- Fresh `npm test`: FAIL (338 total, 327 pass, 10 fail, 1 skip). Failures are AST/linker relation (2 direct plus gold cohorts), communication identity/enrichment (3), and configuration/document relation coverage; the ticket-030 test passes. diff --git a/project/ticket-030/ai-codex.md b/project/ticket-030/ai-codex.md index 4fe2e01..5d5ca36 100644 --- a/project/ticket-030/ai-codex.md +++ b/project/ticket-030/ai-codex.md @@ -33,13 +33,13 @@ concurrent synthesis refactor or generated `project/` artifacts. implementation session is authorized; merge approval remains external. - Updated only `test/nl-llm.test.ts`: the hierarchy test now reads the Markdown and NL helper modules that own their confidence clamps after refactoring. -- Emitted a disposable build despite inherited parser errors and ran the - focused compiled test: 11 passed, 0 failed, with the hierarchy assertion - passing. No provider request was made. -- Ran the complete test suite in the repository's standard `dist` layout with - the repaired test emitted over the last healthy build: 338 tests, 337 passed, - 0 failed and one JDK-dependent skip. This isolates and validates the approved - test-only delta without concealing the unrelated compile failures. +- Ran the freshly emitted compiled test file: 11 passed, 0 failed, with the + hierarchy assertion passing. No provider request was made. +- Rechecked the updated base from a clean dependency installation. Base + `caf6551` builds successfully, and the freshly built complete suite reports + 327 passed, 10 failed and one skipped out of 338. The confidence-hierarchy + test passes; the remaining failures are inherited relation and communication + regressions outside this ticket. - Renumbered the unpublished local ticket from 023 to 030 after discovering that GitHub PR #3 already owns ticket-023; no implementation path or acceptance criterion changed. @@ -51,5 +51,6 @@ concurrent synthesis refactor or generated `project/` artifacts. either intent or claim that unrelated blocker. - The main worktree contains user-owned generated and synthesis changes. Work remains isolated in a separate branch/worktree. -- Full `npm run verify` is blocked before this test runs by three source parser - errors outside the approved ticket paths. +- Full verification is blocked by ten source-behaviour test failures outside + the approved ticket paths. The base branch needs separately scoped repairs + before it can be considered merge-ready. diff --git a/project/ticket-030/changelog.md b/project/ticket-030/changelog.md index 4227724..dd6ad4a 100644 --- a/project/ticket-030/changelog.md +++ b/project/ticket-030/changelog.md @@ -9,7 +9,8 @@ - Repaired the stale Markdown/NL confidence-clamp source locations in the hierarchy test without changing production confidence behavior. -- Focused compiled NL LLM test passes 11/11 and the complete suite passes all - 337 runnable tests (one environment skip); recorded unrelated parser and - governance blockers separately. +- Focused compiled NL LLM test passes 11/11. After rebasing onto `caf6551`, a + clean build passes; the complete fresh suite exposes ten inherited relation + and communication failures while the repaired hierarchy assertion stays + green. - Renumbered the unpublished ticket to 030 to avoid colliding with active PR #3.