docs(openspec): propose qualitative survey feedback via PostHog surveys

[thecodedrift]

Stack (root → tip):

• ➡️ docs(openspec): propose qualitative survey feedback via PostHog surveys #341 (you are here)
  • feat(scaffold): migration 0007 ignores .taskless/.tmp-* scratch files #342
    • feat(feedback): add feedback send/dismiss, the survey constants, and the cadence store #343
      • docs(agent): add the feedback and feedback-invite recipes #344
        • feat(agent): append the feedback survey invite to surveyed recipes #345

Opening PR for the qualitative-survey-feedback change: planning artifacts only, no code.

What this proposes

Rule authoring and onboarding are where an agent-driven Taskless session most often goes wrong, and the only signal we have about them is quantitative. A PostHog survey exists whose questions are addressed to the agent as respondent, relaying the user's words and its own read of the session. This change lets the CLI use it:

• Survey invite appended to four served recipes (onboard, create-sg-rule, create-vale-rule, create-remote-rule) when telemetry is on, not in CI, and a per-CLI-version next_ask timestamp has passed. Serving captures survey shown and pushes next_ask out 10 days.
• New feedback subcommand: feedback dismiss --topic (captures survey dismissed, +20 days) and feedback send --from <file> (validates a human-keyed payload, maps to the survey's question ids, captures survey sent, +20 days). Unlisted in the agent index; the invite is the only door.
• feedback and feedback-invite recipes, the second rendered header-less so the appended prose sits under the same Vale rules as every other recipe.
• Migration 0007 ignores .taskless/.tmp-*. 0001 is not edited.

Delivery shape

Stacked, merging forward, four PRs: migration → subcommand + schema + store → recipes → gate + injection (which archives the change). Nothing triggers a survey until the tip lands. The changeset goes on PR 1.

Reviewing the spec deltas

• analytics and cli are MODIFIED requirements copied in full: 6 and 3 existing scenarios retained, one added to each.
• cli-taskless-bootstrap adds a requirement for migration 7.
• cli-feedback-survey is new: 5 requirements, 19 scenarios.

pnpm openspec validate qualitative-survey-feedback --strict passes.

[thecodedrift]

Biggest miss, posthog capture was changed in a significant way

• Only valid event names are below
• Responses contain $survey_id and for "survey sent", $survey_response_QUESTION_UUID

// when the survey recipe is injected into the agent recipe payload
// triggered by select npx @taskless/cli agent ...
posthog.capture("survey shown", {  
  $survey_id: "01a0b1a0-80fb-0000-5dc1-baa4ec44e619",  
});  

// when the user has explicitly dismissed the survey
// triggered by npx @taskless/cli feedback dismiss
posthog.capture("survey dismissed", {  
  $survey_id: "01a0b1a0-80fb-0000-5dc1-baa4ec44e619",
});  

// send the completed responses
// triggered by npx @taskless/cli feedback send --file=./path/to/feedback.json
posthog.capture("survey sent", {  
  $survey_id: "01a0b1a0-80fb-0000-5dc1-baa4ec44e619",  
  "$survey_response_5feff6a3-6768-4817-92d7-5ae3975c6baa": "...", // Q1 (required)  
  "$survey_response_561e87f4-a1b7-4855-b728-29d19421f7e7": "...", // Q2 (required)  
  "$survey_response_48e4a7c8-a63f-4c33-a47f-6c835435a288": "...", // Q3 (required)  
  "$survey_response_2316428e-dc3e-4c96-ae67-a6e8c66d7db5": "...", // Q4 (optional)  
  "$survey_response_67bedbd9-ca70-4c1c-b1a6-6df830a453dd": "...", // Q5 (optional)  
});  

Additionally the --topic constructs seem completely fabricated

[thecodedrift]

Re: @thecodedrift — "Biggest miss, posthog capture was changed in a significant way…"

Addressed in 6d60b6c. The three events now carry exactly what the snippets show: $survey_id on all three, plus $survey_response_<question id> on survey sent. Every topic construct is gone (event property, dismiss --topic, the payload key, the SURVEY_TOPIC render variable), and $survey_questions went with it so the contract in the spec matches the snippets rather than a docs example. Cadence is keyed by survey UUID.

Two things to note against the pasted snippet: Q3's id is 6ebdfabb-3575-49aa-857c-47b6bbfdebc8 in the live survey (it changed when the question became single choice; 48e4a7c8-… no longer exists), and the standard super-properties (cliVersion etc.) still ride on these events because they go through the shared capture wrapper like everything else. The analytics delta says both.

— AI Coding Agent

[thecodedrift]

@claude /review

[claude]

Claude finished @thecodedrift's task in 4m 45s —— View job

---

Review mode: incremental — read 5 prior review thread(s) before reviewing.

All 5 prior threads are resolved and carry an *— AI Coding Agent* reply, so I treated them as already addressed and did not repeat them:

• Cadence-store path bloat (per-CLI-version directories) → now keyed by survey id (<config directory>/surveys/<survey id>/next_ask).
• The "new survey = new ask" scenario gap → added as its own scenario alongside "A CLI upgrade keeps the cadence."
• Fabricated topic property on survey shown / survey dismissed → removed; those events now carry only $survey_id.
• Fabricated topic / $survey_questions on survey sent, and the wrong payload description → corrected to $survey_id + $survey_response_<question id> only, no question re-send.
• Fabricated --topic flag on feedback dismiss / SURVEY_TOPIC render variable → both removed from tasks.md and the spec.

I verified these directly against the current diff (not just the reply text): grepped for 48e4a7c8 (stale Q3 id), survey_questions, and --topic/SURVEY_TOPIC across openspec/changes/qualitative-survey-feedback/ — none remain. The live Q3 id 6ebdfabb-3575-49aa-857c-47b6bbfdebc8 is used consistently in design.md and tasks.md.

I found nothing NEW since the last review. Specifically, I checked for the failure modes this kind of doc-only PR is most likely to hit, and none reproduced:

• Spec-delta correctness (CLAUDE.md's "a MODIFIED block replaces the requirement" rule): compared both MODIFIED requirements against origin/main.
  • analytics: ### Requirement: CLI events use cli\_ prefix — title byte-identical to main; all 6 existing scenarios (Rule creation, Recipe fetch, Fetch with no topic, A command failure, Old event names, Completed scan) are carried in full, plus the new survey scenario. Nothing dropped.
  • cli: ### Requirement: CLI manages .taskless/.gitignore — title byte-identical to main; all 3 existing scenarios (created, preserved, idempotent) retained, plus the new scratch-file scenario.
  • No requirement rename occurred in either delta (titles match exactly), so neither is at risk of the "renamed title → delta discarded" trap.
• Feasibility against real code: cross-checked the plan against the actual source — getConfigDirectory() (packages/cli/src/auth/token.ts) already returns <XDG>/taskless, matching design.md's stated path; header: false / stripHeader (prompts/recipes.ts) really does what the design claims for feedback-invite; TOPIC_INPUT_SCHEMAS and INTERNAL_TOPICS already exist as extension points; onboard.ts always calls ensureTasklessDirectory() before serving its recipe, so the "onboard runs before .taskless/ exists" edge case doesn't arise for the surveyed onboarding path; migration numbering (0001–0006 exist, so 0007 is next) and the /sgconfig.yml-style anchoring precedent both check out against packages/cli/src/filesystem/migrations/.
• Cross-file consistency: survey UUID, the five question ids, the 10-day/20-day intervals, and the "no topic in the payload" contract all agree across proposal.md, design.md, specs/cli-feedback-survey/spec.md, and tasks.md.

No inline comments to post — this pass turned up no new findings to anchor on specific lines.