Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
206 changes: 206 additions & 0 deletions .abcd/development/intents/shipped/itd-3-codebase-mapping.md

Large diffs are not rendered by default.

Large diffs are not rendered by default.

17 changes: 11 additions & 6 deletions .abcd/work/CONTEXT.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,12 +18,17 @@ request, then validate the answer into `findings.jsonl`) and `review`
regression-test drafting layer, `draft-tests` (emit a drafting request
carrying each confirmed finding and its event window, validate the answer
into `tests.jsonl`, render the accepted drafts as a Markdown test plan)
with `review -kind tests` for the accept / edit / reject pass. The oracle is
host-delegated: the CLI never calls a model, holds no keys, and adds no
network dependency; every finding is born `unverified`, every drafted test is
born `proposed`, and each ingest is the sole validation boundary for its own
answer. Next is codebase mapping (itd-3), then Mode B / the
pattern library (itd-4). Command and file contracts in
with `review -kind tests` for the accept / edit / reject pass, and the
codebase-mapping layer, `map` (emit a mapping request carrying each
confirmed anchored finding and the application's repository path, validate
the answer's source references against that repository into `refs.jsonl`,
render an issue draft per mapped finding) with `review -kind refs` for the
accept / reject pass. The oracle is host-delegated: the CLI never calls a
model, holds no keys, and adds no network dependency; every finding is born
`unverified`, every drafted test and every code reference is born
`proposed`, and each ingest is the sole validation boundary for its own
answer. Next is terminal-session anchoring (itd-2609152113364815), then
Mode B / the pattern library (itd-4). Command and file contracts in
[`../../docs/reference/cli.md`](../../docs/reference/cli.md) and
[`../../docs/reference/session-directory.md`](../../docs/reference/session-directory.md).

Expand Down
10 changes: 10 additions & 0 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -160,6 +160,16 @@ jobs:
# sample still has a confirmed finding to draft from.
./testimony draft-tests -session examples/sample-session > /tmp/tests-request.md
grep -q "testimony-testdraft/v1" /tmp/tests-request.md
# The mapping layer over the same merged timeline: render proves the
# bundled refs.jsonl still renders an issue draft for F-001 with its
# accepted reference, and emit proves the sample still has a confirmed
# anchored finding to map and carries the repository path handed to it.
./testimony map -render -session examples/sample-session | tee /tmp/issues.md
grep -q "## F-001" /tmp/issues.md
grep -q "ProfileForm.tsx:46\` (owner) — accepted" /tmp/issues.md
./testimony map -session examples/sample-session -repo internal/coderefs/testdata/repo > /tmp/map-request.md
grep -q "testimony-coderefs/v1" /tmp/map-request.md
grep -q "internal/coderefs/testdata/repo" /tmp/map-request.md
# The counts line above catches events going missing from the merge,
# but not a windowing/attachment regression: report's header counts are
# raw entry counts, computed before the join, so they stay "10 · 10"
Expand Down
9 changes: 6 additions & 3 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -50,14 +50,15 @@ Testimony captures usability evidence, on the record.

## Current state

v0.4.0. A Go CLI (`testimony`, standard library only) whose nine pipeline
v0.4.0. A Go CLI (`testimony`, standard library only) whose ten pipeline
commands are all implemented and dispatched from `internal/cli` behind the
`cmd/testimony` entry point: `record` and `demo`
(capture), `transcribe` and `import` (hand-off of a recording the CLI never
made — a voice recording, or an operator-recorded asciinema terminal session),
`merge`, `report`, the analysis layer `analyze` and `review`, and the
`merge`, `report`, the analysis layer `analyze` and `review`, the
regression-test drafting layer `draft-tests` (with `review -kind tests` for its
human pass) — plus `version` and `help`. The model work is host-delegated —
human pass), and the codebase-mapping layer `map` (with `review -kind refs`
for its human pass) — plus `version` and `help`. The model work is host-delegated —
the CLI never calls a model, holds no keys, and adds no network dependency. The user-facing documentation
is [`docs/README.md`](docs/README.md); the exact command and file contracts are
[`docs/reference/cli.md`](docs/reference/cli.md) and
Expand All @@ -80,6 +81,8 @@ go test -run TestEventsNearWindow ./internal/timeline/ # a single test
./testimony merge -session examples/sample-session # pipeline smoke:
./testimony report -session examples/sample-session # writes timeline.jsonl + report.md
./testimony draft-tests -session examples/sample-session # emit the drafting request (after merge)
./testimony map -render -session examples/sample-session # issue drafts from the bundled refs.jsonl (after merge)
./testimony map -session examples/sample-session -repo internal/coderefs/testdata/repo # emit the mapping request (after merge)
S=$(mktemp -d) && cp examples/sample-session/manifest.json "$S" \
&& ./testimony import -session "$S" -cast internal/cast/testdata/v2.cast # terminal hand-off smoke
sh -n install.sh && bash -n install.sh # installer syntax
Expand Down
30 changes: 30 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,36 @@ break an existing invocation is called out in the entry that records it.

### Added

- `testimony map` maps a **confirmed** finding to the code that owns its
on-screen anchor, and `testimony review -kind refs` records the human accept /
reject pass over each reference. The resolution stays host-delegated, exactly
as for `analyze` and `draft-tests`: `map -session DIR -repo DIR` emits one
self-contained mapping request — a versioned rubric, the repository path, and,
for each confirmed finding whose `ui` carries a selector or route, that
finding's record and its event window — and `map -session DIR -repo DIR
-ingest FILE` is the validation boundary, forcing every reference to
`status: proposed`, refusing any reference to a finding that is not currently
confirmed or carries no anchor, and checking every `path` against the
repository: repo-relative, no `.` or `..` segment, inside the repository even
through a symlinked directory, an existing regular file (a symlink or
directory is refused), with any `line` within the file's length. The CLI
opens the repository read-only for those checks and never writes into it; it
never greps or interprets the source, and a model-asserted `confidence` field
is refused as unknown, so the human decision is the only quality signal. An
answer is bounded to 1000 references before any is checked, and the recorded
path is the checked form. The emitted request is the one artefact that names
an absolute local path, the repository's, so `map` refuses an `-out` inside
the session directory.
References and decisions live in a new `refs.jsonl` session artefact,
append-only in both directions with no edit path: a wrong path is rejected
and a corrected one ingested. `map -session DIR -render` renders one Markdown
issue draft per mapped finding — a title and reproduction steps derived by
the CLI from the finding and its event window, the participant's quote, and
every reference with its current status — so a draft rendered before review
is visibly unreviewed. Nothing is filed anywhere. A session with no confirmed
anchored finding is staged loudly with the counts by status, including how
many confirmed findings carry an anchor, nothing written, exit 1. Terminal
findings carry no anchor and are the subject of their own intent.
- `testimony analyze -ingest` records what answered the analysis request.
`-backend local|cloud` and a free-text `-model NAME` write one provenance
record — `{"kind":"provenance","rubric":…,"backend":…,"model":…,"at":…}` — as
Expand Down
12 changes: 9 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -108,6 +108,7 @@ Each session is one folder of small, inspectable files, created under `~/Testimo
timeline.jsonl # merged, session-relative timeline
findings.jsonl # analysis findings + verdicts
tests.jsonl # regression-test drafts + decisions
refs.jsonl # code references + decisions
report.md # human-readable aligned record
```

Expand All @@ -121,14 +122,19 @@ or whisper.cpp), `import` (an asciinema terminal recording joins the session's
interaction stream on the shared clock), `merge`, `report`, the first-pass
analysis layer — `analyze` (emit an analysis request, then validate the answer
into findings, recording the backend and model you declare answered it) and
`review` (record human verdicts) — and the regression-test
`review` (record human verdicts) — the regression-test
drafting layer, `draft-tests` (turn a confirmed finding into a proposed test
case, then render the accepted ones as a Markdown test plan) with
`review -kind tests` for the accept / edit / reject pass. `record` captures the
`review -kind tests` for the accept / edit / reject pass, and the
codebase-mapping layer, `map` (hand a confirmed finding's selector or route to
your assistant with the application's repository, validate the source
references it returns, and render an issue draft per mapped finding) with
`review -kind refs` for the accept / reject pass. `record` captures the
microphone by default; screen video is opt-in with `-video`. The model work is
host-delegated — the CLI never calls a model, holds no keys, and adds no network
dependency — every finding is *unverified* until you confirm or reject it, and
every drafted test is a *proposal* until you accept it. Run the analysis request
every drafted test and every code reference is a *proposal* until you accept
it. Run the analysis request
against a model on your own machine and no session content leaves it at any
step; `findings.jsonl` and `report.md` carry your declaration that it did
([analyse a session locally](docs/how-to/analyse-locally.md)).
Expand Down
2 changes: 1 addition & 1 deletion docs/README.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Testimony documentation

- **[Tutorials](tutorials/getting-started.md)** — learn by doing: capture, transcribe, and report on your first session in about five minutes.
- **[How-to guides](how-to/)** — recipes for specific tasks: [transcribe a recording](how-to/transcribe-a-recording.md), [record a terminal session](how-to/record-a-terminal-session.md), [instrument your own app](how-to/instrument-your-own-app.md), [analyse a session](how-to/analyse-a-session.md), [analyse a session locally](how-to/analyse-locally.md), [draft regression tests](how-to/draft-regression-tests.md).
- **[How-to guides](how-to/)** — recipes for specific tasks: [transcribe a recording](how-to/transcribe-a-recording.md), [record a terminal session](how-to/record-a-terminal-session.md), [instrument your own app](how-to/instrument-your-own-app.md), [analyse a session](how-to/analyse-a-session.md), [analyse a session locally](how-to/analyse-locally.md), [draft regression tests](how-to/draft-regression-tests.md), [map findings to code](how-to/map-findings-to-code.md).
- **[Reference](reference/)** — exact descriptions of the [command line](reference/cli.md) and the [session directory](reference/session-directory.md).
- **[Explanation](explanation/)** — background and reasoning: [how alignment works](explanation/how-alignment-works.md), [privacy](explanation/privacy.md).
163 changes: 163 additions & 0 deletions docs/how-to/map-findings-to-code.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,163 @@
# Map findings to code

This guide covers the mapping layer: turning a **confirmed** finding into
proposed references to the source files that own its on-screen anchor, deciding
each reference by hand, and rendering an issue draft you can file wherever your
tracker lives. Testimony delegates the model work to an assistant of your
choice: `map` never calls a model, holds no keys, and adds no network
dependency. It emits a request, you run it against the application's
repository, and it validates the answer.

Prerequisites: a session with at least one finding whose current verdict is
`confirmed` and whose `ui` carries a `selector` or a `route`, a merged
`timeline.jsonl`, and the application's repository checked out on this machine.
The step sits downstream of verification on purpose, so
[analyse the session](analyse-a-session.md) first and confirm what you believe.
A session with no such finding is refused, with the finding count by status,
rather than mapped. A terminal-session finding carries no selector or route, so
`map` has nothing to hand over for it.

The anchors resolve best when the application marks its interactive elements
with stable `data-testid` attributes, because a test id is a literal string the
assistant can search the source for, where a route is a router-table entry it
has to read and interpret. [Instrument your own app](instrument-your-own-app.md)
covers the attribute.

The flow is five steps: **emit** the mapping request, **run** it with your
assistant, **ingest** the answer, **review** the references, and **render** the
issue drafts.

## 1. Emit the mapping request

`testimony map` writes a single, self-contained prompt: a versioned rubric, the
session context, the path to the repository, and, for each confirmed finding
with an anchor, that finding's own record plus its event window from the
timeline. Send it to stdout to read it, or to a file to hand off:

```sh
testimony map -session ~/Testimony/sessions/<dir> -repo ~/code/your-app -out request.md
```

`-repo` names the repository root as the assistant will open it, so it is the
one place an absolute local path appears in the request. Keep the request out
of the session directory, which is what you hand to others: `map` refuses an
`-out` inside it, and nothing in the session directory changes.

`-window` sets the event window's half-width in seconds, as it does for
`draft-tests`, and defaults to 10.

## 2. Run it with your assistant of choice

Give the request to an assistant that can read the repository: an agent host
with the checkout open, or a colleague. Ask it to follow the instructions and
return **only** the JSON answer. Save that answer to a file, for example
`refs.json`.

The expected shape is a JSON object with a `refs` array (a bare array is also
accepted):

```json
{"rubric":"testimony-coderefs/v1","refs":[
{"id":"R-001","finding":"F-001","session":"sample-session",
"path":"src/settings/ProfileForm.tsx","line":46,"role":"owner",
"status":"proposed"},
{"id":"R-002","finding":"F-001","session":"sample-session",
"path":"src/settings/saveProfile.ts","line":12,"role":"handler",
"status":"proposed"}
]}
```

One finding may yield several references: the component that renders the
element, the handler behind it, the router entry, a test. The `role` says which,
and `line` is optional. There is no confidence field, and an answer that adds
one is refused: the human decision in step 4 is the only quality signal.

## 3. Ingest the answer

Validate the answer against the reference schema and the repository, and write
`refs.jsonl`:

```sh
testimony map -session ~/Testimony/sessions/<dir> -repo ~/code/your-app -ingest refs.json
```

Ingest is the validation boundary, and it never trusts the model. It rejects,
with a precise message, any reference that names a finding which is not
currently `confirmed` or carries no anchor, claims a session other than this
one, uses a role outside the set, names a path that does not exist as a regular
file under the repository or escapes it, or gives a line past the file's end.
All errors are reported at once and nothing is written until the whole answer is
clean, so you can fix a batch in one pass. Every reference lands
`status: proposed`, whatever the answer claimed.

The repository is opened read-only, to check that each path exists and each line
is in range, and nothing is written into it. The path check is a containment
check: a `..` segment, an absolute path, a symlink, or a path through a
symlinked directory that leaves the repository is refused.

You can also pipe the answer straight in with `-ingest -`:

```sh
your-assistant < request.md | testimony map -session ~/Testimony/sessions/<dir> -repo ~/code/your-app -ingest -
```

Once a decision exists in `refs.jsonl`, ingest refuses to overwrite the file:
the human record is retained, so a re-map needs a fresh session directory or a
removed `refs.jsonl`.

## 4. Review the references

Each reference is a *proposal* until you judge it. `testimony review -kind refs`
walks the proposed references and records your decision:

```sh
testimony review -session ~/Testimony/sessions/<dir> -kind refs -repo ~/code/your-app
```

For each reference it shows its id, path, line and role, the finding it
resolves with that finding's quote, anchor and clock, and, because `-repo` is
given, the source lines around the referenced line, then prompts
`[a]ccept [r]eject [s]kip [q]uit`. Without `-repo` it shows the record alone.

Your decision is *appended* to `refs.jsonl` with today's date. The reference
line is never overwritten, so the record of what the machine proposed and what
you decided both survive. There is no edit: a wrong path is rejected, and a
corrected one arrives through a fresh ingest.

To record a single decision without the interactive walk (handy in scripts):

```sh
testimony review -session ~/Testimony/sessions/<dir> -kind refs -ref R-001 -decision accepted
testimony review -session ~/Testimony/sessions/<dir> -kind refs -ref R-002 -decision rejected
```

Interactive review needs stdin to be a character device (an interactive terminal
is one); when it is a pipe or a redirected regular file (as in CI) it prints a
notice and exits without blocking. A later decision overrides an earlier one, and
both are kept.

## 5. Render the issue drafts

Render one issue draft per mapped finding:

```sh
testimony map -session ~/Testimony/sessions/<dir> -render -out issues.md
```

Each draft carries a title derived from the finding, its severity, anchor and
clock, the participant's own words, the reproduction steps derived from the
event window, and the suspected files: every reference for the finding, with
its role and its current status. A draft rendered before review therefore says
`proposed` beside each file, so a reader can see which paths a person has
vouched for. The title and the steps are derived by the CLI from the records,
not by the model, so the render needs no second round-trip and cannot invent a
step; the steps read mechanically for the same reason.

Nothing is filed anywhere: Testimony never opens an issue and never writes into
the application's repository, so the output defaults to stdout and `-out` puts it
wherever you assemble your tickets. The **record** stays with the session, in
`refs.jsonl`, linked to its finding.

For the exact field rules, see the
[session directory reference](../reference/session-directory.md#refsjsonl); for
every flag, the [command-line reference](../reference/cli.md#testimony-map).
Loading