Skip to content

Consolidate all tests into one repo-level suite run on every publish #28

Description

@ElliotDrel

Problem

Tests are split across two homes and only cover a fraction of the pack, so a publish can go green while most skills are untested.

Tests live inside skills and ship to users. skills/estack-doc-review-viewer/e2etest.mjs and selftest.mjs sit in the skill folder, so npm pack bundles them — 43.3kB of test code downloaded by every user who installs elliot-stack. They are also invisible to pytest, which sets norecursedirs = skills, so nothing in CI runs them.

Coverage is thin. tests/ covers 3 of 23 skills: estack-flight-planner, estack-migrate-claude-session-history, estack-read-agent-history. The other 20 have no automated test at any level.

There is no single entry point. .github/workflows/publish.yml runs pytest -q, then two hand-listed commands for the migrate smoke tests, because those are a Python script and a Node script that pytest does not collect. Adding a test in a new language means editing the workflow, and anything not hand-added is silently skipped.

Nothing runs when a skill did not change. Gate 6 in .agents/skills/manage-e-stack/steps/prep.md runs the suite, but the suite itself only reaches the three covered skills. A regression in an untouched skill — caused by a shared script, a convention change, or a doc migration — surfaces after release.

What to build

One suite at the repo root that runs everything, every publish, regardless of what changed.

  • All tests move to tests/. Nothing test-related stays inside skills/. Move e2etest.mjs and selftest.mjs to tests/estack-doc-review-viewer/ and repoint their imports. Skill folders hold only what a user needs at runtime.
  • One command runs the whole suite and returns a single exit code. It must collect Node and Python tests together, so the two hand-listed migrate commands disappear from the workflow.
  • Every skill is covered by at least a smoke test — the skill loads, its frontmatter parses, and its scripts run --help without throwing. That alone would have caught the ENOENT risk found when skills moved to ~/.e-stack/, where tracker-tools.cjs had no mkdir anywhere in it.
  • The suite runs unconditionally. No change detection, no skipping unchanged skills.
  • prep.md gate 6 and publish.yml both call the one command, so local and CI runs cannot drift apart.

Verification

  • npm pack --dry-run lists no test file under skills/.
  • The publish workflow contains exactly one test step.
  • Removing a mkdir from any skill's write path fails the suite.
  • Every folder in skills/ has a corresponding folder in tests/.

Found during the v1.0.68 publish-readiness audit.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions