Problem
Tests are split across two homes and only cover a fraction of the pack, so a publish can go green while most skills are untested.
Tests live inside skills and ship to users. skills/estack-doc-review-viewer/e2etest.mjs and selftest.mjs sit in the skill folder, so npm pack bundles them — 43.3kB of test code downloaded by every user who installs elliot-stack. They are also invisible to pytest, which sets norecursedirs = skills, so nothing in CI runs them.
Coverage is thin. tests/ covers 3 of 23 skills: estack-flight-planner, estack-migrate-claude-session-history, estack-read-agent-history. The other 20 have no automated test at any level.
There is no single entry point. .github/workflows/publish.yml runs pytest -q, then two hand-listed commands for the migrate smoke tests, because those are a Python script and a Node script that pytest does not collect. Adding a test in a new language means editing the workflow, and anything not hand-added is silently skipped.
Nothing runs when a skill did not change. Gate 6 in .agents/skills/manage-e-stack/steps/prep.md runs the suite, but the suite itself only reaches the three covered skills. A regression in an untouched skill — caused by a shared script, a convention change, or a doc migration — surfaces after release.
What to build
One suite at the repo root that runs everything, every publish, regardless of what changed.
- All tests move to
tests/. Nothing test-related stays inside skills/. Move e2etest.mjs and selftest.mjs to tests/estack-doc-review-viewer/ and repoint their imports. Skill folders hold only what a user needs at runtime.
- One command runs the whole suite and returns a single exit code. It must collect Node and Python tests together, so the two hand-listed migrate commands disappear from the workflow.
- Every skill is covered by at least a smoke test — the skill loads, its frontmatter parses, and its scripts run
--help without throwing. That alone would have caught the ENOENT risk found when skills moved to ~/.e-stack/, where tracker-tools.cjs had no mkdir anywhere in it.
- The suite runs unconditionally. No change detection, no skipping unchanged skills.
prep.md gate 6 and publish.yml both call the one command, so local and CI runs cannot drift apart.
Verification
npm pack --dry-run lists no test file under skills/.
- The publish workflow contains exactly one test step.
- Removing a
mkdir from any skill's write path fails the suite.
- Every folder in
skills/ has a corresponding folder in tests/.
Found during the v1.0.68 publish-readiness audit.
Problem
Tests are split across two homes and only cover a fraction of the pack, so a publish can go green while most skills are untested.
Tests live inside skills and ship to users.
skills/estack-doc-review-viewer/e2etest.mjsandselftest.mjssit in the skill folder, sonpm packbundles them — 43.3kB of test code downloaded by every user who installselliot-stack. They are also invisible topytest, which setsnorecursedirs = skills, so nothing in CI runs them.Coverage is thin.
tests/covers 3 of 23 skills:estack-flight-planner,estack-migrate-claude-session-history,estack-read-agent-history. The other 20 have no automated test at any level.There is no single entry point.
.github/workflows/publish.ymlrunspytest -q, then two hand-listed commands for the migrate smoke tests, because those are a Python script and a Node script that pytest does not collect. Adding a test in a new language means editing the workflow, and anything not hand-added is silently skipped.Nothing runs when a skill did not change. Gate 6 in
.agents/skills/manage-e-stack/steps/prep.mdruns the suite, but the suite itself only reaches the three covered skills. A regression in an untouched skill — caused by a shared script, a convention change, or a doc migration — surfaces after release.What to build
One suite at the repo root that runs everything, every publish, regardless of what changed.
tests/. Nothing test-related stays insideskills/. Movee2etest.mjsandselftest.mjstotests/estack-doc-review-viewer/and repoint their imports. Skill folders hold only what a user needs at runtime.--helpwithout throwing. That alone would have caught theENOENTrisk found when skills moved to~/.e-stack/, wheretracker-tools.cjshad nomkdiranywhere in it.prep.mdgate 6 andpublish.ymlboth call the one command, so local and CI runs cannot drift apart.Verification
npm pack --dry-runlists no test file underskills/.mkdirfrom any skill's write path fails the suite.skills/has a corresponding folder intests/.Found during the v1.0.68 publish-readiness audit.