diff --git a/openspec/changes/qualitative-survey-feedback/.openspec.yaml b/openspec/changes/qualitative-survey-feedback/.openspec.yaml new file mode 100644 index 00000000..d28e909f --- /dev/null +++ b/openspec/changes/qualitative-survey-feedback/.openspec.yaml @@ -0,0 +1,2 @@ +schema: spec-driven +created: 2026-09-17 diff --git a/openspec/changes/qualitative-survey-feedback/design.md b/openspec/changes/qualitative-survey-feedback/design.md new file mode 100644 index 00000000..b66f9ad4 --- /dev/null +++ b/openspec/changes/qualitative-survey-feedback/design.md @@ -0,0 +1,118 @@ +## Context + +See proposal.md for motivation. The constraints that shape the approach: + +- `src/prompts/recipes.ts` is the single render path for every recipe and is imported by Workers without `nodejs_compat`. It must stay free of telemetry, filesystem, and `process` reads; `assert-prompts-graph` in `vite.config.ts` fails the build otherwise. Anything that reads a timestamp or captures an event lives in the command layer. +- `test/prompts.test.ts` asserts that `taskless agent ` matches `getPrompt(topic, { directive: true })` byte for byte, by spawning the built CLI. The test environment sets `DO_NOT_TRACK=1` and `TASKLESS_TELEMETRY_DISABLED=1`. +- `test/recipe-cross-references.test.ts` requires every `src/agent/*.md` to open with `# Topic: `, to name only commands that exist, and to spell no CLI invocation by hand. +- The survey (`01a0b1a0-80fb-0000-5dc1-baa4ec44e619`) is `schedule: once` with five questions. Q3 is `single_choice` with the literal choices `Yes`, `No`, `Unknown`. Its identifier changed when it became single choice; the map below is the current one. +- PostHog's custom-survey contract is three event literals (`survey shown`, `survey dismissed`, `survey sent`) and the `$survey_id` and `$survey_response_` keys. Nothing else is part of that contract. +- The telemetry opt-out is `DO_NOT_TRACK=1` or `TASKLESS_TELEMETRY_DISABLED=1`. `CI` is read elsewhere in the CLI as `CI === "true" || CI === "1"` (`util/interactive.ts`). +- Scratch input files follow `.taskless/.tmp--request.json` with a `--from` flag and a recipe "clean up" step (`rule create`, `rule improve`). Nothing ignores them today. +- Migrations are sequential, keyed numerically, and frozen once shipped. A project at version N runs only migrations above N. + +## Goals / Non-Goals + +**Goals:** + +- One gate, called from both recipe-serving commands, that decides whether to append the invite and owns every side effect of that decision. +- The agent never transcribes a question UUID. The payload it writes uses human keys; the CLI owns the map. +- The `prompts` export, the byte-parity test, and every existing recipe rendering are untouched. +- Test runs never emit a survey event. + +**Non-Goals:** + +- A general feedback channel. `feedback` is reached only through the invite for now; when a general channel exists this surface is refactored into it. +- Retiring `taskless onboard`. Both onboarding paths get the invite in this change; consolidating them is a separate change. +- Anything beyond the four listed topics. The topic set is a constant and widening it is a one-line edit later. +- Server-side survey targeting. The survey's internal targeting flag and `$survey_responded/…` person properties are unused; the CLI is the only scheduler. + +## Decisions + +### The gate lives in the command layer, after `getRecipe()` + +`agent.ts` and `onboard.ts` both call `getRecipe()` and print. A new `src/survey/invite.ts` exports one function that takes the rendered recipe, the topic, and the invocation, and returns the text to print. Inside it: check the gate, and if open, capture `survey shown`, write `next_ask`, and append the rendered invite. The `prompts` module never sees it. + +Alternative considered: a `RecipeOptions.survey` flag in the renderer. Rejected because the renderer cannot read the clock or the config directory without breaking the Workers constraint, and because the parity test would then have to reason about time. + +### Gate condition, stated once + +``` +telemetry enabled + && CI is not "true" or "1" + && topic in { onboard, create-sg-rule, create-vale-rule, create-remote-rule } + && (next_ask is absent, unparseable, or <= now) +``` + +"Telemetry enabled" is the same predicate `getTelemetry()` uses, exposed rather than duplicated. `CI` is checked separately because telemetry is not disabled in CI (`cli_check_completed` from CI is real signal) but a survey in CI has no one to answer it. + +### `next_ask`, epoch milliseconds, keyed by survey id + +Path: `/taskless/surveys//next_ask`, where the survey id is the PostHog UUID. Contents: `Date.now()` at write time plus the interval, as a decimal string. Missing or unparseable reads as "ask now", and the next write repairs it. + +| Event | Writes `next_ask` | +| ------------------ | ----------------- | +| `survey shown` | now + 10 days | +| `survey dismissed` | now + 20 days | +| `survey sent` | now + 20 days | + +Keyed by survey rather than by CLI version so the config directory does not grow a directory per release, and so a CLI upgrade does not reset the cadence: a newer CLI reads the same file and may not ask right away, which is fine. A new survey is a new id and therefore a new ask. Per-release segmentation still works because `cliVersion` rides on every capture. Dismiss and sent both earn the long gap because either is an explicit terminal action; only silence earns the short one. + +Alternative considered: a JSON state file with outcome history. Rejected as more than is needed; a bare epoch is the whole state. + +### `feedback send --from `, `feedback dismiss` + +Naming follows `rule create --from`. `feedback` joins `SUBCOMMAND_NAMES` (the `satisfies` check in `index.ts` enforces registration) and `UNLISTED_COMMANDS` in `agent.ts`, for the same reason `demo` is there: listing it invites an agent to run it unprompted. `agent feedback` still serves `feedback.md`, the way `agent check` serves `check.md` for the `check` subcommand. + +Neither verb takes a topic: the survey events carry PostHog's keys and nothing of ours. Both verbs take `--dir`. Neither runs migrations or requires `.taskless/` to exist: `send` reads whatever path it is given, and neither writes into the project. + +Under a disabled telemetry client both verbs print one line saying nothing was sent and exit zero. An agent should never reach them in that state, because the invite is not served in it, so this is defensive rather than a path the recipe describes. + +### Payload schema: human keys, CLI-owned map + +`src/schemas/feedback.ts`: + +| Key | Type | Question | +| ------------------ | ---------------------------- | --------------------------------------------------------------------------------------------------------- | +| `verbatim` | non-empty string | `5feff6a3-6768-4817-92d7-5ae3975c6baa` What was the user's comments verbatim? | +| `goal` | non-empty string | `561e87f4-a1b7-4855-b728-29d19421f7e7` What was the user trying to accomplish? | +| `completed` | `"Yes" \| "No" \| "Unknown"` | `6ebdfabb-3575-49aa-857c-47b6bbfdebc8` Did the user successfully complete the task in your opinion? | +| `workedWell` | optional string | `2316428e-dc3e-4c96-ae67-a6e8c66d7db5` What steps of the interaction with Taskless worked well? | +| `needsImprovement` | optional string | `67bedbd9-ca70-4c1c-b1a6-6df830a453dd` What steps of the interaction with Taskless could use improvement? | + +`completed` uses PostHog's literal casing so no value mapping exists to drift. The survey id and question map live in one constants module (`src/survey/constants.ts`) and nowhere else. `send` emits `$survey_id` and one `$survey_response_` per answered question, and nothing else survey-specific. Optional questions left blank are omitted rather than sent empty. + +Alternative considered: the agent writes `$survey_response_` keys directly. Rejected because a mangled UUID is a silently missing answer; a mangled human key is a validation error with a message. + +### Survey events are the one exception to the `cli_` prefix + +The three names and the `$survey_*` keys are PostHog's contract, and the CLI adds no survey-specific property of its own. They go through the same `capture()` wrapper as every other event, so `cli`, `cliVersion`, `scaffoldVersion`, `ghOwner`, and the adoption dimensions ride along the way they do on everything else, and `cliVersion` is what makes the per-release segmentation work. Which recipe a response belongs to is not on the event; the `cli_agent` event that served the recipe precedes `survey shown` from the same distinct id, and the `goal` answer names the task in the agent's words. The `analytics` delta records the exception so the prefix rule stays enforceable for everything else. + +### `feedback-invite.md` is a recipe rendered header-less + +The invite text is prose an agent reads, so it belongs under the same Vale rules and the same sprintf substitution (`%(TASKLESS_CLI)s`) as every other recipe. Putting it in `src/agent/` means the cross-reference test requires a `# Topic: feedback-invite` header; the gate renders it with `header: false`, which the renderer already supports and which strips the header block cleanly. It is classified under `INTERNAL_TOPICS`. `agent feedback-invite` will serve it, which is harmless. + +Alternative considered: a string literal in `invite.ts`. Rejected because it would be the only recipe prose that escapes the prose linter, and the `.md` rationale in `recipes.ts` explains at length why that matters. + +### Invite placement: appended after `See Also` + +Pure string append, no parsing of the recipe body. Agents attend to the start and end of a response; the end puts the ask after the task rather than in front of it. + +The invite's own content: one exact sentence to put to the user (`Taskless would like to know how the CLI is doing. Would you be okay sharing a few sentences about your experience? Or just skip it with \`skip\`.`), the rule that a reply of `skip`, silence, or something unrelated to feedback is a dismissal (`feedback dismiss`, then carry on with what the user asked), and that any other reply means fetch `agent feedback` and follow it. Ask once; never repeat the question in the same session. + +### Migration 0007 adds `/.tmp-*`, and 0001 is not edited + +`0007-ignore-scratch-files.ts` calls `addToGitignore(cwd, ["/.tmp-*"])`, anchored with a leading slash for the reason `0001` anchors `/sgconfig.yml`. `addToGitignore` already skips present lines, so the migration is idempotent by construction. A fresh scaffold runs 1 through 7 and ends with the same file as an upgraded one. `feedback send` does not delete the input file; the recipe's clean-up step does, matching `rule create`, and the ignore makes a forgotten file harmless. + +## Risks / Trade-offs + +- [`survey shown` overcounts] The CLI knows it served the invite, not that the agent surfaced it. The funnel will read shown ≫ sent. → Documented in the analytics delta and the constants module; no property is added to pretend otherwise, since the names are PostHog's. +- [Responses are not tagged with the recipe that invited them] → Accepted; PostHog's survey contract has no slot for it, and the preceding `cli_agent` event plus the `goal` answer recover it when it matters. +- [An agent runs `feedback dismiss` after an unrelated reply that was actually feedback] → The recipe defines dismissal narrowly (`skip`, silence, unrelated); misclassification costs one 20-day gap, not data loss. +- [Scaffold version 7 makes a nightly-scaffolded repo "too new" for a release CLI at 6 until the release ships] → Normal cost of a migration, paid five times already; the migration is the bottom of the stack so it lands first. +- [`feedback` is guessable by name] → Not in the index, and an unprompted `feedback send` still needs a valid payload against a schema the agent has not been shown. +- [Test suites that spawn the built CLI with a real `XDG_CONFIG_HOME`] → The gate is telemetry-first and tests disable telemetry, so no test run writes `next_ask` or emits a survey event. Gate tests stub the environment and point `XDG_CONFIG_HOME` at a temp directory. + +## Migration Plan + +Forward-merging stack as in the proposal. Rollback of any slice is a revert; slices 1 through 3 have no runtime effect on their own. Reverting slice 4 stops invites; reverting slice 1 after it has run leaves an extra ignore line, which is inert. diff --git a/openspec/changes/qualitative-survey-feedback/proposal.md b/openspec/changes/qualitative-survey-feedback/proposal.md new file mode 100644 index 00000000..7cf07a5e --- /dev/null +++ b/openspec/changes/qualitative-survey-feedback/proposal.md @@ -0,0 +1,46 @@ +## Why + +Rule authoring and onboarding are where an agent-driven Taskless session most often goes wrong, and the only signal we have about them today is quantitative: `cli_agent` says which recipe was fetched, `cli_run` says whether the command exited zero. Neither says what the person was trying to do, whether they got it, or what the agent stumbled on along the way. A PostHog survey exists for exactly that (`01a0b1a0-80fb-0000-5dc1-baa4ec44e619`), but nothing in the CLI can show it: there is no web surface, and the survey's questions are addressed to the agent as the respondent, relaying the user's words and its own observation of the session. + +## What Changes + +- **A survey invite is appended to four served recipes** (`onboard`, `create-sg-rule`, `create-vale-rule`, `create-remote-rule`) when telemetry is enabled, the run is not in CI, and the survey's `next_ask` timestamp under the XDG config directory has passed. Serving the invite captures `survey shown` and pushes `next_ask` out 10 days. Both serving paths for onboarding (`taskless onboard` and `taskless agent onboard`) carry the invite. +- **A new `feedback` subcommand** with two verbs. `feedback dismiss` captures `survey dismissed` and pushes `next_ask` out 20 days. `feedback send --from ` validates a JSON payload against a Zod schema, maps its human-keyed fields to the survey's question identifiers, captures `survey sent`, and pushes `next_ask` out 20 days. Under a disabled telemetry client both verbs report that nothing was sent and exit zero. `feedback` is deliberately absent from the `agent` topic index: the invite is the only door, pending a general-purpose feedback channel that would refactor this. +- **A new `feedback` recipe** (`taskless agent feedback`) that tells the agent how to fill the payload: the user's words verbatim, what they were trying to accomplish, whether they completed it (`Yes` / `No` / `Unknown`), and optionally what worked and what did not. The schema is embedded through the existing `%(INPUT_SCHEMA)s` mechanism. A companion `feedback-invite` recipe carries the appended text, rendered header-less, so the prose sits under the same Vale rules as every other recipe. +- **The three survey events carry exactly PostHog's keys.** Event names, `$survey_id`, and `$survey_response_` are PostHog's contract and are used as-is, with no survey-specific property of our own; they are the one exception to the `cli_` prefix rule. The standard super-properties ride along the way they do on every capture, which is what lets `cliVersion` split responses by release. +- **Migration `0007` ignores `.taskless/.tmp-*`.** The feedback recipe writes `.taskless/.tmp-feedback.json` the way the rule recipes write `.tmp-rule-request.json`, and a forgotten scratch file should be a stray rather than a commit. Past migrations are not edited; a fresh scaffold runs `0001` through `0007` in sequence and arrives at the same place. + +Nothing here is **BREAKING**. The scaffold version moves 6 → 7, which is a `patch` bump at `0.y.z`: it adds an ignore line, changes no on-disk shape a consumer reads, and is idempotent. + +## Capabilities + +### New Capabilities + +- `cli-feedback-survey`: the survey invite gate and cadence store, the `feedback send` / `feedback dismiss` verbs and their payload contract, the `feedback` and `feedback-invite` recipes, and the three survey events. + +### Modified Capabilities + +- `analytics`: the `cli_` prefix requirement gains its one exception, the three PostHog survey event literals, and states what rides on them. +- `cli`: the `.taskless/.gitignore` requirement enumerates `/.tmp-*` alongside the two entries it already names. +- `cli-taskless-bootstrap`: a new requirement for migration 7, which adds the `/.tmp-*` ignore without editing migration 1. + +## Impact + +- `packages/cli/src/commands/agent.ts` and `commands/onboard.ts`: call a shared gate after `getRecipe()` returns. `src/prompts/recipes.ts` is untouched; the `@taskless/cli/prompts` export never carries an invite. +- `packages/cli/src/commands/feedback.ts` (new), `src/commands/names.ts` (`feedback` joins `SUBCOMMAND_NAMES`), `src/index.ts` registration, and `UNLISTED_COMMANDS` in `agent.ts`. +- `packages/cli/src/schemas/feedback.ts` (new) plus a `TOPIC_INPUT_SCHEMAS` entry; `src/survey/` (new) for the survey constants, question map, gate, and `next_ask` store. +- `packages/cli/src/agent/feedback.md` and `feedback-invite.md` (new); `INTERNAL_TOPICS` in `src/prompts/index.ts`. +- `packages/cli/src/filesystem/migrations/0007-ignore-scratch-files.ts` (new) registered in `migrate.ts`. +- PostHog: three new event names in the project, each carrying PostHog's survey keys and the standard super-properties. No dashboard depends on them yet. +- Tests: the existing `agent`/`prompts` byte-parity suite keeps passing because tests run with telemetry disabled and the gate is telemetry-first; new suites cover the gate, the store, the two verbs, the schema, and the migration. + +## Delivery shape + +**Stacked, merging forward, four PRs.** Nothing triggers a survey until the tip lands, so each slice is safe on `main` by itself: + +1. Migration `0007` and the `cli` / `cli-taskless-bootstrap` deltas. The changeset lives here. +2. The `feedback` subcommand, the payload schema, the survey constants, and the `next_ask` store. Reachable only by someone who types it. +3. The `feedback` and `feedback-invite` recipes. The schema they embed exists from (2). +4. The gate and injection in `agent` and `onboard`, and the `analytics` delta. This is the switch-on, and it archives the change. + +Each PR extends the one changeset on the bottom branch. diff --git a/openspec/changes/qualitative-survey-feedback/specs/analytics/spec.md b/openspec/changes/qualitative-survey-feedback/specs/analytics/spec.md new file mode 100644 index 00000000..74311b1f --- /dev/null +++ b/openspec/changes/qualitative-survey-feedback/specs/analytics/spec.md @@ -0,0 +1,86 @@ +## MODIFIED Requirements + +### Requirement: CLI events use cli\_ prefix + +CLI events SHALL use the `cli_` prefix, with the taxonomy organized as a +`cli_run` denominator plus concrete state-transition events: + +- `cli_run` — exactly one per invocation (see the dedicated requirement). This + replaces every previous `cli_` start event and `cli__completed` + event; the `success`/`durationMs`/`command` signal lives here. +- Concrete state-transition events, each fired at the point the state actually + changes, carrying counts/ids/booleans only (never rule content, prompts, or + matched source): + - `cli_rule_created`, `cli_rule_improved`, `cli_rule_deleted` + - `cli_authenticated`, `cli_logged_out` + - `cli_installed`, `cli_onboarded` + - `cli_check_completed` — error/warning counts and the number of rules the + scan had loaded: `errorCount`, `warningCount`, `findings`, `ruleCount`. + Counts only, never rule content, rule names, or matched source + - `cli_error` — a single failure event with `command` and `code` (a stable + `CLIErrorCode`) +- `cli_agent` — fired when the `agent` command serves a request, with a `topic` + property (the served topic; the exact literal `"(index)"` when invoked with no + topic; the attempted topic for an unknown request). This replaces the previous + `help_index`, `help_`, and `help_unknown` events. + +Commands that carry no concrete state beyond the invocation (e.g. `info`, +`detect`, `update`, `auth status`, `rule verify`, `rule meta`) SHALL rely on +`cli_run` alone and SHALL NOT emit a bespoke event. The previous taxonomy +(`cli_`, `cli__completed`, `help_index`, `help_`, +`help_unknown`) SHALL be removed in this release; there is no dual-emit window. + +The survey events are the one exception to the prefix. `survey shown`, +`survey dismissed`, and `survey sent` are PostHog's own event literals for a +custom survey, and their `$survey_id` and `$survey_response_` +keys are PostHog's property contract; the CLI SHALL emit them under those +exact names and SHALL add no survey-specific property of its own. They go +through the same capture path as every other event, so the standard +properties (`cli`, `cliVersion`, `scaffoldVersion`, `ghOwner`, and the +adoption dimensions) ride on them the way they ride on everything else. No +other event SHALL omit the `cli_` prefix. + +#### Scenario: Rule creation emits a concrete state event plus cli_run + +- **WHEN** a user runs `taskless rule create --from req.json` and a rule is written +- **THEN** PostHog SHALL receive one `cli_run` event with `command: "rule create"` +- **AND** SHALL receive a `cli_rule_created` event +- **AND** SHALL NOT receive `cli_rule_create` or `cli_rule_create_completed` + +#### Scenario: Recipe fetch emits cli_agent with a topic + +- **WHEN** an agent runs `taskless agent create-sg-rule` +- **THEN** PostHog SHALL receive a `cli_agent` event with `topic: "create-sg-rule"` +- **AND** SHALL NOT receive a `help_create_sg_rule` event + +#### Scenario: Fetch with no topic emits cli_agent with the index marker + +- **WHEN** an agent runs `taskless agent` +- **THEN** PostHog SHALL receive a `cli_agent` event with `topic: "(index)"` +- **AND** SHALL NOT receive a `help_index` event + +#### Scenario: A command failure emits cli_error + +- **WHEN** a command fails with a known `CLIErrorCode` +- **THEN** PostHog SHALL receive a `cli_error` event with `command` and `code` + +#### Scenario: Old event names are not emitted + +- **WHEN** any CLI command runs in this release +- **THEN** PostHog SHALL NOT receive any event named `cli__completed`, + `help_index`, `help_`, or `help_unknown` + +#### Scenario: A completed scan reports how many rules were loaded + +- **WHEN** a scan completes +- **THEN** the `cli_check_completed` event SHALL include `ruleCount`, the number + of rules the scan had loaded +- **AND** a scan that loaded no rules SHALL be distinguishable from a scan that + loaded rules and found nothing + +#### Scenario: Survey events keep PostHog's names and carry the standard properties + +- **WHEN** the CLI serves a survey invite, records a dismissal, or sends a response +- **THEN** PostHog SHALL receive `survey shown`, `survey dismissed`, or `survey sent` respectively, under that exact name +- **AND** the event SHALL carry `$survey_id` and the standard properties, and no survey-specific property beyond PostHog's own keys +- **AND** no event named `cli_survey_shown`, `cli_survey_dismissed`, or `cli_survey_sent` SHALL be emitted diff --git a/openspec/changes/qualitative-survey-feedback/specs/cli-feedback-survey/spec.md b/openspec/changes/qualitative-survey-feedback/specs/cli-feedback-survey/spec.md new file mode 100644 index 00000000..622db22c --- /dev/null +++ b/openspec/changes/qualitative-survey-feedback/specs/cli-feedback-survey/spec.md @@ -0,0 +1,153 @@ +## Purpose + +Collects qualitative feedback about rule authoring and onboarding through a PostHog survey the agent answers on the user's behalf: the CLI decides when to invite, the agent gathers the user's words and its own observation of the session, and the CLI validates and sends the response. + +## ADDED Requirements + +### Requirement: A survey invite is appended to surveyed recipes when the gate is open + +When the CLI serves one of the surveyed recipes (`onboard`, `create-sg-rule`, `create-vale-rule`, `create-remote-rule`) through `taskless agent ` or `taskless onboard`, it SHALL append the survey invite to the served text if and only if every gate condition holds: + +- telemetry is enabled (neither `DO_NOT_TRACK=1` nor `TASKLESS_TELEMETRY_DISABLED=1`); +- `CI` is neither `"true"` nor `"1"`; +- the survey's `next_ask` timestamp is absent, unparseable, or not later than now. + +When the gate is open the CLI SHALL capture `survey shown` with `$survey_id`, write `next_ask` to now plus 10 days, and append the invite after the recipe's last section. When any condition fails the served text SHALL be byte-identical to the recipe as rendered without a survey. The `@taskless/cli/prompts` export SHALL never include the invite. + +#### Scenario: Gate open on a surveyed topic + +- **WHEN** telemetry is enabled, `CI` is unset, `next_ask` is absent, and an agent runs `taskless agent create-sg-rule` +- **THEN** stdout SHALL be the `create-sg-rule` recipe followed by the survey invite +- **AND** PostHog SHALL receive `survey shown` with `$survey_id` +- **AND** `next_ask` SHALL be written to a time about 10 days in the future + +#### Scenario: Both onboarding paths carry the invite + +- **WHEN** the gate is open and either `taskless onboard` or `taskless agent onboard` serves the onboarding recipe +- **THEN** both SHALL append the invite and capture `survey shown` with `$survey_id` + +#### Scenario: Within the window + +- **WHEN** `next_ask` is later than now and an agent runs `taskless agent create-vale-rule` +- **THEN** stdout SHALL be the recipe with no invite +- **AND** no survey event SHALL be captured and `next_ask` SHALL be unchanged + +#### Scenario: Telemetry disabled + +- **WHEN** `DO_NOT_TRACK=1` and an agent runs any surveyed topic +- **THEN** stdout SHALL be the recipe with no invite +- **AND** `next_ask` SHALL NOT be read or written + +#### Scenario: Running in CI + +- **WHEN** `CI=true` and telemetry is enabled and an agent runs any surveyed topic +- **THEN** stdout SHALL be the recipe with no invite +- **AND** no survey event SHALL be captured + +#### Scenario: Unsurveyed topic + +- **WHEN** the gate would otherwise be open and an agent runs `taskless agent check` +- **THEN** stdout SHALL be the recipe with no invite +- **AND** `next_ask` SHALL be unchanged + +#### Scenario: Corrupt cadence file is repaired + +- **WHEN** `next_ask` contains text that is not a number and an agent runs a surveyed topic with the other conditions met +- **THEN** the invite SHALL be appended +- **AND** `next_ask` SHALL be overwritten with a valid timestamp + +### Requirement: The cadence store is one epoch timestamp per survey + +The CLI SHALL keep the survey cadence at `/surveys//next_ask`, where the config directory is the same XDG location that holds the anonymous telemetry id and the survey id is the PostHog survey's UUID. The file SHALL contain a single decimal epoch-milliseconds value: the earliest time the next invite may be served. Serving an invite SHALL set it to now plus 10 days; `feedback dismiss` and `feedback send` SHALL each set it to now plus 20 days. Every CLI version SHALL share the file, so upgrading the CLI does not reset the cadence; a different survey id SHALL have its own file. + +#### Scenario: A CLI upgrade keeps the cadence + +- **WHEN** version A served an invite yesterday and the user runs version B for the first time +- **THEN** version B SHALL read the same `next_ask` and SHALL NOT serve the invite + +#### Scenario: A new survey is a new ask + +- **WHEN** the CLI ships with a different survey id than the one whose `next_ask` is on disk +- **THEN** the CLI SHALL find no `next_ask` for the new survey and SHALL serve the invite + +#### Scenario: Dismissal earns the longer gap + +- **WHEN** an invite was served and the agent runs `taskless feedback dismiss` +- **THEN** `next_ask` SHALL be about 20 days in the future, later than the value the invite wrote + +### Requirement: The feedback subcommand records dismissals and sends responses + +The CLI SHALL provide a `feedback` subcommand with two verbs. `feedback dismiss` SHALL capture `survey dismissed` with `$survey_id` and advance `next_ask`. `feedback send --from ` SHALL read a JSON file, validate it against the feedback payload schema, capture `survey sent`, and advance `next_ask`. Both verbs SHALL accept `--dir`. Neither SHALL require `.taskless/` to exist, run migrations, or write into the project. `feedback send` SHALL NOT delete its input file. + +When the telemetry client is disabled, both verbs SHALL print a single line saying nothing was sent and exit zero. + +`feedback` SHALL NOT appear in the `taskless agent` topic index. `taskless agent feedback` SHALL still serve the `feedback` recipe. + +#### Scenario: Dismiss + +- **WHEN** an agent runs `taskless feedback dismiss` +- **THEN** PostHog SHALL receive `survey dismissed` with `$survey_id` +- **AND** the command SHALL exit zero + +#### Scenario: Send a valid payload + +- **WHEN** an agent runs `taskless feedback send --from .taskless/.tmp-feedback.json` with a payload that passes validation +- **THEN** PostHog SHALL receive `survey sent` carrying `$survey_id` and one `$survey_response_` per answered question, and no other survey-specific property +- **AND** the command SHALL exit zero and leave the input file in place + +#### Scenario: Send an invalid payload + +- **WHEN** the payload is missing a required key or `completed` is not one of the allowed values +- **THEN** the command SHALL exit non-zero with `INVALID_INPUT`, naming the failing field +- **AND** no survey event SHALL be captured and `next_ask` SHALL be unchanged + +#### Scenario: Telemetry disabled + +- **WHEN** `DO_NOT_TRACK=1` and an agent runs either verb +- **THEN** the command SHALL print that nothing was sent and exit zero + +#### Scenario: Not in the index + +- **WHEN** an agent runs `taskless agent` +- **THEN** the printed index SHALL NOT list `feedback` + +### Requirement: The feedback payload uses human keys mapped to survey questions by the CLI + +The feedback payload SHALL be a JSON object with these keys: + +| key | required | value | +| ------------------ | -------- | ------------------------------------------------- | +| `verbatim` | yes | the user's own words, non-empty | +| `goal` | yes | what the user was trying to accomplish, non-empty | +| `completed` | yes | exactly `Yes`, `No`, or `Unknown` | +| `workedWell` | no | what worked well | +| `needsImprovement` | no | what could use improvement | + +The CLI SHALL own the map from these keys to the survey's question identifiers; a payload SHALL NOT contain `$survey_*` keys. An omitted optional key SHALL be omitted from the event rather than sent as an empty string. The schema SHALL be embedded in the `feedback` recipe through the recipe input-schema mechanism. + +#### Scenario: Optional answers are omitted, not blanked + +- **WHEN** a payload has no `workedWell` +- **THEN** the `survey sent` event SHALL carry no `$survey_response_` key for that question + +#### Scenario: Completion is one of three literals + +- **WHEN** a payload has `completed: "partially"` +- **THEN** validation SHALL fail naming `completed` + +### Requirement: The feedback and feedback-invite recipes + +The CLI SHALL embed a `feedback` recipe that tells the agent it is the respondent: it records the user's reply verbatim, fills the remaining answers from its own observation of the session, writes the payload to `.taskless/.tmp-feedback.json`, runs `feedback send --from` that path, and deletes the file afterwards. The recipe SHALL embed the payload schema and SHALL NOT ask the agent to put the survey's questions to the user one by one. + +The CLI SHALL embed a `feedback-invite` recipe carrying the text appended to surveyed recipes. Rendered header-less, it SHALL instruct the agent to ask the user exactly once, with the sentence "Taskless would like to know how the CLI is doing. Would you be okay sharing a few sentences about your experience? Or just skip it with `skip`."; to treat a reply of `skip`, silence, or a reply unrelated to feedback as a dismissal and run `feedback dismiss`; and to treat any other reply as feedback and fetch `agent feedback`. Both recipes SHALL follow the recipe conventions: a `# Topic:` header, the CLI named by its rendered invocation, and commands that exist. + +#### Scenario: The appended invite carries no header + +- **WHEN** the gate is open and a surveyed recipe is served +- **THEN** the appended text SHALL NOT contain a second `# Topic:` line +- **AND** SHALL name the rendered CLI invocation for both `feedback dismiss` and `agent feedback` + +#### Scenario: The feedback recipe embeds the schema + +- **WHEN** an agent runs `taskless agent feedback` +- **THEN** stdout SHALL open with `# Topic: feedback` and contain the JSON Schema for the payload diff --git a/openspec/changes/qualitative-survey-feedback/specs/cli-taskless-bootstrap/spec.md b/openspec/changes/qualitative-survey-feedback/specs/cli-taskless-bootstrap/spec.md new file mode 100644 index 00000000..07fe66ff --- /dev/null +++ b/openspec/changes/qualitative-survey-feedback/specs/cli-taskless-bootstrap/spec.md @@ -0,0 +1,23 @@ +## ADDED Requirements + +### Requirement: Migration 7 ignores scratch request files + +Migration `7` SHALL add `/.tmp-*` to `.taskless/.gitignore` through the shared gitignore helper, so that the scratch request files agent recipes write under `.taskless/` (for example `.tmp-rule-request.json`, `.tmp-improve-request.json`, `.tmp-feedback.json`) are never committed. It SHALL NOT modify migration `1` or any other shipped migration; a fresh scaffold reaches the same `.gitignore` by running migrations `1` through `7` in order. + +#### Scenario: An existing scaffold gains the ignore line + +- **WHEN** `taskless.json` records version 6 and the CLI bootstraps `.taskless/` +- **THEN** migration 7 SHALL run +- **AND** `.taskless/.gitignore` SHALL contain a `/.tmp-*` line +- **AND** `taskless.json` SHALL record version 7 + +#### Scenario: A fresh scaffold ends with the same file + +- **WHEN** `.taskless/` does not exist and the CLI bootstraps it +- **THEN** `.taskless/.gitignore` SHALL contain `.env.local.json`, `/sgconfig.yml`, and `/.tmp-*` +- **AND** migration 1's own source SHALL be unchanged from the previous release + +#### Scenario: Migration 7 is idempotent + +- **WHEN** `.taskless/.gitignore` already contains `/.tmp-*` and migration 7 runs +- **THEN** the file SHALL contain exactly one `/.tmp-*` line afterwards diff --git a/openspec/changes/qualitative-survey-feedback/specs/cli/spec.md b/openspec/changes/qualitative-survey-feedback/specs/cli/spec.md new file mode 100644 index 00000000..2ca5e478 --- /dev/null +++ b/openspec/changes/qualitative-survey-feedback/specs/cli/spec.md @@ -0,0 +1,28 @@ +## MODIFIED Requirements + +### Requirement: CLI manages .taskless/.gitignore + +The CLI SHALL proactively create and maintain a `.taskless/.gitignore` file that ignores local-only files. The gitignore SHALL contain entries for `.env.local.json`, `sgconfig.yml`, and `/.tmp-*` (the scratch request files the agent recipes write, such as `.tmp-rule-request.json` and `.tmp-feedback.json`). Any CLI command that writes to `.taskless/` SHALL ensure the `.gitignore` file exists with these entries before writing. + +#### Scenario: .gitignore is created when .taskless/ is first written to + +- **WHEN** the CLI creates any file in `.taskless/` (e.g., during `auth login`, `rule create`, or `check`) +- **AND** `.taskless/.gitignore` does not exist +- **THEN** the CLI SHALL create `.taskless/.gitignore` containing `.env.local.json`, `sgconfig.yml`, and `/.tmp-*` + +#### Scenario: Existing .gitignore is preserved + +- **WHEN** `.taskless/.gitignore` already exists with additional user entries +- **AND** the CLI needs to ensure its entries are present +- **THEN** the CLI SHALL append any missing entries without removing existing content + +#### Scenario: .gitignore entries are idempotent + +- **WHEN** `.taskless/.gitignore` already contains `.env.local.json`, `sgconfig.yml`, and `/.tmp-*` +- **THEN** the CLI SHALL NOT duplicate the entries + +#### Scenario: Scratch files are ignored on an existing scaffold + +- **WHEN** a project scaffolded before `/.tmp-*` was an entry runs any command that bootstraps `.taskless/` +- **THEN** `.taskless/.gitignore` SHALL gain a `/.tmp-*` line +- **AND** a `.taskless/.tmp-feedback.json` left behind by an agent SHALL NOT appear in `git status` diff --git a/openspec/changes/qualitative-survey-feedback/tasks.md b/openspec/changes/qualitative-survey-feedback/tasks.md new file mode 100644 index 00000000..df1f9a39 --- /dev/null +++ b/openspec/changes/qualitative-survey-feedback/tasks.md @@ -0,0 +1,33 @@ +## 1. Slice 1: migration 0007 ignores scratch files (bottom of the stack) + +- [ ] 1.1 Add `packages/cli/src/filesystem/migrations/0007-ignore-scratch-files.ts` calling `addToGitignore(cwd, ["/.tmp-*"])`, with a header comment saying why `0001` is not edited; register it as `"7"` in `migrate.ts`. Verify `git diff` touches no file under `migrations/` other than the new one. +- [ ] 1.2 Add tests: a version-6 scaffold gains `/.tmp-*` and records 7; a fresh scaffold ends with `.env.local.json`, `/sgconfig.yml`, `/.tmp-*`; running 7 twice leaves one line. Verify with `pnpm --filter @taskless/cli test -- migrate`. +- [ ] 1.3 Fix any test that pins the max scaffold version or the exact `.gitignore` contents (`check-gitignore.test.ts`, `migrate-*.test.ts`, `install-state.test.ts`). Verify `pnpm --filter @taskless/cli test` passes. +- [ ] 1.4 Add the single changeset (`patch`, `@taskless/cli`) describing the ignore line, with a body that says the scaffold's own `version` field carries the compatibility signal. Verify `.changeset/*.md` exists and `pnpm changeset status` lists it. +- [ ] 1.5 Run `pnpm typecheck` and `pnpm lint`; open PR 1 against `main`. + +## 2. Slice 2: survey constants, cadence store, and the `feedback` subcommand + +- [ ] 2.1 Add `packages/cli/src/survey/constants.ts` with the survey id, the five `{ key, id, question }` entries (Q3 id `6ebdfabb-3575-49aa-857c-47b6bbfdebc8`), the surveyed-topic list (used only by the gate), and the 10-day / 20-day intervals. Verify a test asserts the five question ids against the values in design.md. +- [ ] 2.2 Add `packages/cli/src/survey/cadence.ts`: read `next_ask` (missing or unparseable → `undefined`), write `next_ask` under `getConfigDirectory()/surveys//`. Verify tests with `XDG_CONFIG_HOME` pointed at a temp directory cover absent, valid, corrupt, and that a different survey id reads its own file. +- [ ] 2.3 Add `packages/cli/src/schemas/feedback.ts` (Zod: `verbatim`, `goal`, `completed` as `"Yes" | "No" | "Unknown"`, optional `workedWell` / `needsImprovement`). Verify tests reject `completed: "partially"` and a missing `verbatim`, each naming the field. +- [ ] 2.4 Export the telemetry "enabled" predicate from `telemetry.ts` rather than duplicating it, and a `TelemetryClient`-typed way for a command to know it holds the no-op client. Verify `telemetry.test.ts` still passes. +- [ ] 2.5 Add `packages/cli/src/commands/feedback.ts` with `dismiss` and `send --from`; `send` maps keys to `$survey_response_`, adds `$survey_id`, and omits blank optionals; both advance `next_ask` by 20 days and print a one-line "nothing sent" under disabled telemetry with exit 0. Verify tests spawn the built CLI with telemetry stubbed and assert the captured payload shape, the `INVALID_INPUT` path, the input file surviving, and no `.taskless/` bootstrap. +- [ ] 2.6 Add `feedback` to `SUBCOMMAND_NAMES`, register it in `index.ts`, add it to `UNLISTED_COMMANDS` in `agent.ts`. Verify `pnpm typecheck` passes and `taskless agent` output does not list `feedback`. +- [ ] 2.7 Extend the changeset on the bottom branch with the new verbs. Run `pnpm typecheck` and `pnpm lint`; open PR 2 against PR 1's branch. + +## 3. Slice 3: the `feedback` and `feedback-invite` recipes + +- [ ] 3.1 Write `packages/cli/src/agent/feedback.md` (`# Topic: feedback … topic v1`): the agent is the respondent, one verbatim answer from the user, the rest from observation, `%(INPUT_SCHEMA)s`, write `.taskless/.tmp-feedback.json`, run `%(TASKLESS_CLI)s feedback send --from …`, clean up. Register `feedback` in `TOPIC_INPUT_SCHEMAS`. Verify `pnpm build && pnpm cli agent feedback` prints the header and the JSON Schema. +- [ ] 3.2 Write `packages/cli/src/agent/feedback-invite.md` (`# Topic: feedback-invite … topic v1`) with the exact ask sentence from design.md, the dismissal rule (`skip`, silence, unrelated → `%(TASKLESS_CLI)s feedback dismiss`), the feedback path (`%(TASKLESS_CLI)s agent feedback`), and ask-once. Verify `pnpm cli agent feedback-invite` renders with no unsubstituted marker. +- [ ] 3.3 Add both topics to `INTERNAL_TOPICS`. Verify `recipe-cross-references.test.ts`, the topic-membership test, and `pnpm cli check` (Vale over the new prose) all pass. +- [ ] 3.4 Extend the changeset. Run `pnpm typecheck` and `pnpm lint`; open PR 3 against PR 2's branch. + +## 4. Slice 4: the gate and injection (tip of the stack) + +- [ ] 4.1 Add `packages/cli/src/survey/invite.ts`: `withSurveyInvite({ recipe, topic, invocation, cwd })` evaluates the gate (telemetry enabled, `CI` not `"true"`/`"1"`, topic surveyed, `next_ask` passed), and when open captures `survey shown` with `$survey_id`, writes `next_ask` + 10 days, and returns recipe + rendered `feedback-invite` (`header: false`). Verify unit tests cover every gate branch in the spec, including the corrupt-file repair. +- [ ] 4.2 Call it from `agent.ts` (after a successful `getRecipe`) and from `onboard.ts` (recipe-serving path only; not `--mark-complete`, not the already-onboarded early return). Verify a test with telemetry stubbed on and `XDG_CONFIG_HOME` in a temp directory shows the invite on `agent create-sg-rule`, `agent onboard`, and `onboard`, and not on `agent check`. +- [ ] 4.3 Confirm the untouched paths: `prompts.test.ts` byte-parity passes as-is, and `getPrompt()` output contains no invite under any option. Verify `pnpm --filter @taskless/cli test` is green. +- [ ] 4.4 Add a comment in `survey/constants.ts` recording that `survey shown` means "served", not "surfaced", and that the funnel reads shown ≫ sent by design. +- [ ] 4.5 Extend the changeset with the invite. Run `pnpm typecheck` and `pnpm lint`; open PR 4 against PR 3's branch. +- [ ] 4.6 After the invite is verified end to end against the real survey (one `survey sent` visible in PostHog under the survey's responses with labelled columns), archive the change on this tip: `pnpm openspec archive qualitative-survey-feedback -y`, checking every prior scenario in the three modified specs survives per the pre-archive procedure in CLAUDE.md.