Skip to content

Add 18 Katalon True Platform testing skills - #2995

Open
huytieu wants to merge 2 commits into
github:mainfrom
huytieu:katalon-true-platform-skills
Open

Add 18 Katalon True Platform testing skills#2995
huytieu wants to merge 2 commits into
github:mainfrom
huytieu:katalon-true-platform-skills

Conversation

@huytieu

@huytieu huytieu commented Sep 9, 2026

Copy link
Copy Markdown

What this adds

The katalon-* skill family: 18 skills covering the testing lifecycle on Katalon True Platform (the test-management platform formerly known as TestOps).

Stage Skills
Plan katalon-test-plan, katalon-test-estimation
Design katalon-create-test-cases, katalon-test-data, katalon-test-review, katalon-exploratory-charter
Execute katalon-execute-test, katalon-playwright-execute, katalon-test-case-to-playwright, katalon-test-case-to-katalon-studio
Report katalon-upload-report, katalon-test-reporting, katalon-analyze-failures
Decide katalon-release-analyze, katalon-test-management, katalon-test-maintenance, katalon-true-platform-testing
Setup katalon-platform-setup

katalon-true-platform-testing is the orchestrator that routes a testing task to the right lane; katalon-platform-setup connects the MCP server and is the entry point for the rest.

How they work

The skills are agent-neutral prose and operate the platform through the Katalon MCP server (https://platform.katalon.io/mcp, streamable HTTP, OAuth 2.1 with dynamic client registration). No local install or vendor CLI is required. Each skill states its capability boundary up front, so an agent that cannot reach the platform says so rather than inventing results.

Source of truth is the public MIT-licensed repo katalon-labs/true-skills, where the skills are maintained and CI-validated; this PR ports them from that repo's .github/skills/ output. All skill names are prefixed katalon- to keep the shared namespace clean, and cross-references between skills were rewritten to match.

Validation

  • npm run skill:validate -> all 436 skills valid
  • node ./eng/update-readme.mjs + node ./eng/generate-marketplace.mjs run; only docs/README.skills.md changed
  • Branched from main, targeting main
  • Skills tested against GitHub Copilot (VS Code and Copilot CLI) as part of the upstream repo's Copilot support work

🤖 Generated with Claude Code

https://claude.ai/code/session_01VjgFMQPShixvLoyCZfwrfL

@huytieu @claude
Add 18 Katalon True Platform testing skills
4fa22a3 
Adds the katalon-* skill family covering the test lifecycle on Katalon
True Platform: planning and estimation, test-case design, execution
(manual, Run-with-AI, Playwright), result upload and reporting,
failure triage, maintenance, and release readiness.

Skills are agent-neutral and operate the platform through the Katalon
MCP server (https://platform.katalon.io/mcp, streamable HTTP, OAuth
2.1). Source of truth is the public MIT-licensed repo
katalon-labs/true-skills; katalon-platform-setup covers connecting the
MCP before the other skills are used.

Validated with npm run skill:validate (436 skills pass) and
docs/README.skills.md regenerated via the build.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VjgFMQPShixvLoyCZfwrfL
@huytieu
huytieu requested a review from aaronpowell as a code owner September 9, 2026 06:38
Copilot AI balanced review requested due to automatic review settings September 9, 2026 06:38
@github-actions github-actions Bot added new-submission PR adds at least one new contribution skills PR touches skills labels Sep 9, 2026
@github-actions

github-actions Bot commented Sep 9, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

🔒 PR Risk Scan Results

Scanned 53 changed file(s).

Severity Count
🔴 High 0
🟠 Medium 15
ℹ️ Info 0
Severity Rule File Line Match
🟠 package-exec-command skills/katalon-platform-setup/SKILL.md 78 npx mcp-remote https://platform.katalon.io/mcp --transport http-first
🟠 package-exec-command skills/katalon-platform-setup/SKILL.md 81 On first run `npx` asks before installing `mcp-remote`; answer it yourself rather than pre-approving. The JSON/TOML config form below passes `-y` because a spawned MCP server has n
🟠 package-exec-command skills/katalon-platform-setup/SKILL.md 105 npx mcp-remote "https://platform.katalon.io/mcp" --transport http-first
🟠 package-exec-command skills/katalon-platform-setup/SKILL.md 130 "command": "npx",
🟠 package-exec-command skills/katalon-playwright-execute/references/playwright-katalon-reporting.md 76 npx playwright test
🟠 package-exec-command skills/katalon-playwright-execute/references/playwright-katalon-reporting.md 77 npx playwright test tests/example.spec.ts
🟠 package-exec-command skills/katalon-playwright-execute/references/playwright-katalon-reporting.md 78 npx playwright test --grep "TC-07"
🟠 package-exec-command skills/katalon-playwright-execute/references/playwright-katalon-reporting.md 79 npx playwright test --project chromium
🟠 package-exec-command skills/katalon-playwright-execute/references/playwright-katalon-reporting.md 88 npx playwright test &lt;target&gt;
🟠 package-exec-command skills/katalon-playwright-execute/references/playwright-katalon-reporting.md 98 npx playwright test &lt;target&gt;
🟠 package-exec-command skills/katalon-test-case-to-playwright/SKILL.md 94 - Run the narrowest available check: TypeScript compile, lint, Playwright list, or a targeted `npx playwright test`.
🟠 package-exec-command skills/katalon-upload-report/SKILL.md 16 - `@​​katalon/playwright-reporter` can generate and upload Playwright report data to Katalon True Platform during `npx playwright test` or through a separate `playwright-reporter upl
🟠 package-exec-command skills/katalon-upload-report/SKILL.md 131 npx playwright test
🟠 package-exec-command skills/katalon-upload-report/SKILL.md 150 3. For Playwright, run `npx playwright test` or the user's package script when execution is requested.
🟠 package-exec-command skills/katalon-upload-report/references/cli-report-upload.md 54 The reporter can upload during `npx playwright test` or upload a generated output directory later through `playwright-reporter upload --output &lt;path&gt; --api-key &lt;key&gt; --project-id &lt;

This is an automated soft-gate report. Findings indicate review targets and do not block merge by themselves.

@github-actions

github-actions Bot commented Sep 9, 2026

Copy link
Copy Markdown
Contributor

🔍 Vally Lint Results

✅ All checks passed

Scope Checked
Skills 18
Agents 0
Total 18
Severity Count
❌ Errors 0
⚠️ Warnings 0
ℹ️ Advisories 0

Summary

Level Finding
ℹ️ ✅ katalon-analyze-failures (2/2 checks passed)
ℹ️ ✓ [spec-compliance] All 1 skill(s) are spec-compliant.
ℹ️ ✓ spec-compliance: All spec checks passed.
ℹ️ ✓ [valid-refs] All file references across 1 skill(s) are valid.
ℹ️ ✓ valid-refs: All file references resolve to existing files within the skill directory.
ℹ️ 1 skill(s) linted, 1 passed
ℹ️ ✅ katalon-create-test-cases (2/2 checks passed)
ℹ️ ✓ [spec-compliance] All 1 skill(s) are spec-compliant.
ℹ️ ✓ spec-compliance: All spec checks passed.
ℹ️ ✓ [valid-refs] All file references across 1 skill(s) are valid.
Full linter output 
### Linting skills/katalon-analyze-failures
✅ katalon-analyze-failures (2/2 checks passed)
    ✓ [spec-compliance] All 1 skill(s) are spec-compliant.
        ✓ spec-compliance: All spec checks passed.
    ✓ [valid-refs] All file references across 1 skill(s) are valid.
        ✓ valid-refs: All file references resolve to existing files within the skill directory.

1 skill(s) linted, 1 passed

### Linting skills/katalon-create-test-cases
✅ katalon-create-test-cases (2/2 checks passed)
    ✓ [spec-compliance] All 1 skill(s) are spec-compliant.
        ✓ spec-compliance: All spec checks passed.
    ✓ [valid-refs] All file references across 1 skill(s) are valid.
        ✓ valid-refs: All file references resolve to existing files within the skill directory.

1 skill(s) linted, 1 passed

### Linting skills/katalon-execute-test
✅ katalon-execute-test (2/2 checks passed)
    ✓ [spec-compliance] All 1 skill(s) are spec-compliant.
        ✓ spec-compliance: All spec checks passed.
    ✓ [valid-refs] All file references across 1 skill(s) are valid.
        ✓ valid-refs: All file references resolve to existing files within the skill directory.

1 skill(s) linted, 1 passed

### Linting skills/katalon-exploratory-charter
✅ katalon-exploratory-charter (2/2 checks passed)
    ✓ [spec-compliance] All 1 skill(s) are spec-compliant.
        ✓ spec-compliance: All spec checks passed.
    ✓ [valid-refs] All file references across 1 skill(s) are valid.
        ✓ valid-refs: All file references resolve to existing files within the skill directory.

1 skill(s) linted, 1 passed

### Linting skills/katalon-platform-setup
✅ katalon-platform-setup (2/2 checks passed)
    ✓ [spec-compliance] All 1 skill(s) are spec-compliant.
        ✓ spec-compliance: All spec checks passed.
    ✓ [valid-refs] All file references across 1 skill(s) are valid.
        ✓ valid-refs: All file references resolve to existing files within the skill directory.

1 skill(s) linted, 1 passed

### Linting skills/katalon-playwright-execute
✅ katalon-playwright-execute (2/2 checks passed)
    ✓ [spec-compliance] All 1 skill(s) are spec-compliant.
        ✓ spec-compliance: All spec checks passed.
    ✓ [valid-refs] All file references across 1 skill(s) are valid.
        ✓ valid-refs: All file references resolve to existing files within the skill directory.

1 skill(s) linted, 1 passed

### Linting skills/katalon-release-analyze
✅ katalon-release-analyze (2/2 checks passed)
    ✓ [spec-compliance] All 1 skill(s) are spec-compliant.
        ✓ spec-compliance: All spec checks passed.
    ✓ [valid-refs] All file references across 1 skill(s) are valid.
        ✓ valid-refs: All file references resolve to existing files within the skill directory.

1 skill(s) linted, 1 passed

### Linting skills/katalon-test-case-to-katalon-studio
✅ katalon-test-case-to-katalon-studio (2/2 checks passed)
    ✓ [spec-compliance] All 1 skill(s) are spec-compliant.
        ✓ spec-compliance: All spec checks passed.
    ✓ [valid-refs] All file references across 1 skill(s) are valid.
        ✓ valid-refs: All file references resolve to existing files within the skill directory.

1 skill(s) linted, 1 passed

### Linting skills/katalon-test-case-to-playwright
✅ katalon-test-case-to-playwright (2/2 checks passed)
    ✓ [spec-compliance] All 1 skill(s) are spec-compliant.
        ✓ spec-compliance: All spec checks passed.
    ✓ [valid-refs] All file references across 1 skill(s) are valid.
        ✓ valid-refs: All file references resolve to existing files within the skill directory.

1 skill(s) linted, 1 passed

### Linting skills/katalon-test-data
✅ katalon-test-data (2/2 checks passed)
    ✓ [spec-compliance] All 1 skill(s) are spec-compliant.
        ✓ spec-compliance: All spec checks passed.
    ✓ [valid-refs] All file references across 1 skill(s) are valid.
        ✓ valid-refs: All file references resolve to existing files within the skill directory.

1 skill(s) linted, 1 passed

### Linting skills/katalon-test-estimation
✅ katalon-test-estimation (2/2 checks passed)
    ✓ [spec-compliance] All 1 skill(s) are spec-compliant.
        ✓ spec-compliance: All spec checks passed.
    ✓ [valid-refs] All file references across 1 skill(s) are valid.
        ✓ valid-refs: All file references resolve to existing files within the skill directory.

1 skill(s) linted, 1 passed

### Linting skills/katalon-test-maintenance
✅ katalon-test-maintenance (2/2 checks passed)
    ✓ [spec-compliance] All 1 skill(s) are spec-compliant.
        ✓ spec-compliance: All spec checks passed.
    ✓ [valid-refs] All file references across 1 skill(s) are valid.
        ✓ valid-refs: All file references resolve to existing files within the skill directory.

1 skill(s) linted, 1 passed

### Linting skills/katalon-test-management
✅ katalon-test-management (2/2 checks passed)
    ✓ [spec-compliance] All 1 skill(s) are spec-compliant.
        ✓ spec-compliance: All spec checks passed.
    ✓ [valid-refs] All file references across 1 skill(s) are valid.
        ✓ valid-refs: All file references resolve to existing files within the skill directory.

1 skill(s) linted, 1 passed

### Linting skills/katalon-test-plan
✅ katalon-test-plan (2/2 checks passed)
    ✓ [spec-compliance] All 1 skill(s) are spec-compliant.
        ✓ spec-compliance: All spec checks passed.
    ✓ [valid-refs] All file references across 1 skill(s) are valid.
        ✓ valid-refs: All file references resolve to existing files within the skill directory.

1 skill(s) linted, 1 passed

### Linting skills/katalon-test-reporting
✅ katalon-test-reporting (2/2 checks passed)
    ✓ [spec-compliance] All 1 skill(s) are spec-compliant.
        ✓ spec-compliance: All spec checks passed.
    ✓ [valid-refs] All file references across 1 skill(s) are valid.
        ✓ valid-refs: All file references resolve to existing files within the skill directory.

1 skill(s) linted, 1 passed

### Linting skills/katalon-test-review
✅ katalon-test-review (2/2 checks passed)
    ✓ [spec-compliance] All 1 skill(s) are spec-compliant.
        ✓ spec-compliance: All spec checks passed.
    ✓ [valid-refs] All file references across 1 skill(s) are valid.
        ✓ valid-refs: All file references resolve to existing files within the skill directory.

1 skill(s) linted, 1 passed

### Linting skills/katalon-true-platform-testing
✅ katalon-true-platform-testing (2/2 checks passed)
    ✓ [spec-compliance] All 1 skill(s) are spec-compliant.
        ✓ spec-compliance: All spec checks passed.
    ✓ [valid-refs] All file references across 1 skill(s) are valid.
        ✓ valid-refs: All file references resolve to existing files within the skill directory.

1 skill(s) linted, 1 passed

### Linting skills/katalon-upload-report
✅ katalon-upload-report (2/2 checks passed)
    ✓ [spec-compliance] All 1 skill(s) are spec-compliant.
        ✓ spec-compliance: All spec checks passed.
    ✓ [valid-refs] All file references across 1 skill(s) are valid.
        ✓ valid-refs: All file references resolve to existing files within the skill directory.

1 skill(s) linted, 1 passed

Copilot AI reviewed Sep 9, 2026
View reviewed changes

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Several skills reference missing assets or stale routes, and the new frontmatter does not follow the required description format.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

Adds 18 Katalon True Platform skills spanning test planning, design, execution, reporting, maintenance, and MCP setup.

Changes:

  • Adds focused Katalon lifecycle skills and supporting references.
  • Adds an orchestrator for routing multi-stage testing workflows.
  • Regenerates the skills catalog.
File summaries
File Description
skills/katalon-analyze-failures/SKILL.md Defines failure triage and defect filing.
skills/katalon-analyze-failures/references/failure-triage.md Documents failure classification.
skills/katalon-create-test-cases/SKILL.md Defines manual test-case creation.
skills/katalon-create-test-cases/references/capability-boundaries.md Lists MCP capabilities.
skills/katalon-create-test-cases/references/istqb-coverage.md Describes coverage techniques.
skills/katalon-create-test-cases/references/manual-test-case-format.md Defines manual-case formatting.
skills/katalon-create-test-cases/references/requirement-analysis.md Provides requirement-analysis guidance.
skills/katalon-execute-test/SKILL.md Defines manual and automated execution.
skills/katalon-execute-test/references/capability-boundaries.md Lists execution boundaries.
skills/katalon-execute-test/references/execution-workflow.md Documents execution sequencing.
skills/katalon-exploratory-charter/SKILL.md Defines exploratory testing sessions.
skills/katalon-exploratory-charter/references/charter-and-session.md Provides charter templates.
skills/katalon-exploratory-charter/references/oracles-and-tours.md Documents exploratory techniques.
skills/katalon-platform-setup/SKILL.md Defines MCP setup and troubleshooting.
skills/katalon-platform-setup/references/capability-boundaries.md Lists platform boundaries.
skills/katalon-playwright-execute/SKILL.md Defines Playwright execution and upload.
skills/katalon-playwright-execute/references/playwright-katalon-reporting.md Documents reporter configuration.
skills/katalon-release-analyze/SKILL.md Defines release-readiness analysis.
skills/katalon-release-analyze/references/capability-boundaries.md Lists analysis boundaries.
skills/katalon-release-analyze/references/release-quality-gates.md Defines readiness gates.
skills/katalon-test-case-to-katalon-studio/SKILL.md Defines Studio test generation.
skills/katalon-test-case-to-playwright/SKILL.md Defines Playwright conversion.
skills/katalon-test-case-to-playwright/references/playwright-typescript.md Provides Playwright patterns.
skills/katalon-test-data/SKILL.md Defines test-data workflows.
skills/katalon-test-data/references/data-design.md Documents data-design strategies.
skills/katalon-test-data/references/seed-and-teardown.md Documents lifecycle-safe test data.
skills/katalon-test-estimation/SKILL.md Defines test-effort estimation.
skills/katalon-test-estimation/references/estimation-model.md Provides the estimation model.
skills/katalon-test-estimation/references/mcp-evidence-map.md Maps estimates to evidence sources.
skills/katalon-test-maintenance/SKILL.md Defines regression maintenance.
skills/katalon-test-maintenance/references/maintenance-loop.md Documents the maintenance loop.
skills/katalon-test-management/SKILL.md Defines asset organization and traceability.
skills/katalon-test-management/references/traceability.md Documents traceability audits.
skills/katalon-test-plan/SKILL.md Defines risk-based test planning.
skills/katalon-test-plan/references/planning-workflow.md Documents planning workflow.
skills/katalon-test-reporting/SKILL.md Defines stakeholder quality reporting.
skills/katalon-test-reporting/references/metric-selection.md Guides metric selection.
skills/katalon-test-reporting/references/snapshot-and-trend.md Defines snapshot-based trends.
skills/katalon-test-review/SKILL.md Defines pre-pipeline test review.
skills/katalon-test-review/references/review-rubric.md Provides the review rubric.
skills/katalon-true-platform-testing/SKILL.md Adds the lifecycle orchestrator.
skills/katalon-true-platform-testing/references/combination-recipes.md Defines multi-skill workflows.
skills/katalon-true-platform-testing/references/execution-workflow.md Documents orchestrated execution.
skills/katalon-true-platform-testing/references/istqb-coverage.md Provides coverage guidance.
skills/katalon-true-platform-testing/references/lifecycle-map.md Maps intents to lifecycle skills.
skills/katalon-true-platform-testing/references/manual-test-case-format.md Defines case structure.
skills/katalon-true-platform-testing/references/mcp-tool-index.md Indexes Katalon MCP tools.
skills/katalon-true-platform-testing/references/prompt-recipes.md Provides invocation examples.
skills/katalon-true-platform-testing/references/requirement-analysis.md Provides analysis guidance.
skills/katalon-true-platform-testing/references/unavailable-capabilities.md Documents unsupported operations.
skills/katalon-upload-report/SKILL.md Defines report upload workflows.
skills/katalon-upload-report/references/capability-boundaries.md Lists upload boundaries.
skills/katalon-upload-report/references/cli-report-upload.md Documents uploader commands.
docs/README.skills.md Registers all 18 skills.
Review details

Suppressed comments (3)

skills/katalon-true-platform-testing/references/lifecycle-map.md:87

  • This route says estimation is not available yet, although katalon-test-estimation is added in the same PR. Requests for duration or staffing will therefore be misrouted to a scope-only plan instead of the owning skill.
    skills/katalon-true-platform-testing/references/lifecycle-map.md:100
  • The routing table still sends Studio authoring requests to the report uploader and states that authoring is unavailable, despite this PR adding katalon-test-case-to-katalon-studio. This prevents the orchestrator from selecting the new conversion skill.
    skills/katalon-true-platform-testing/references/lifecycle-map.md:108
  • Both multi-period reporting intents are still routed to the single-release analyzer as if reporting were unavailable, but katalon-test-reporting is introduced in this PR to own these requests. Route directly to it so the orchestrator uses the new trend/snapshot workflow.
  • Files reviewed: 54/54 changed files
  • Comments generated: 25
  • Review effort level: Balanced

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread skills/katalon-analyze-failures/SKILL.md
@@ -0,0 +1,53 @@
---
name: katalon-analyze-failures
description: Triage Katalon True Platform/TestOps test failures and file defects. Use when you need to investigate failed test results, classify each failure as product defect vs automation defect vs environment/data issue, cluster failures by common signature, find likely root cause from execution data, and optionally create ALM-linked defects for real product bugs. This is failure diagnosis and defect filing; for the overall ship/no-ship release call use katalon-release-analyze, and for repairing the tests themselves use katalon-test-maintenance. Written for the manual tester and the automation tester looking at a red run and needing to know whether the application broke or the test did.
Comment thread skills/katalon-create-test-cases/SKILL.md
@@ -0,0 +1,124 @@
---
name: katalon-create-test-cases
description: Create, update, organize, and link Katalon True Platform/TestOps manual test cases from a synced requirement key such as CEL-6, a read requirement, or free-text product behavior. Use when you need to analyze requirements, design manual test cases using ISTQB techniques as a reference, check existing Katalon coverage, avoid duplicate test cases, import only missing cases, update or link existing cases, or create/reuse a test suite for newly designed cases. For full requirement-to-execution workflows, combine with or defer to katalon-true-platform-testing. Written for the manual tester who has a written requirement in hand and no cases for it yet.
Comment thread skills/katalon-execute-test/SKILL.md
@@ -0,0 +1,90 @@
---
name: katalon-execute-test
description: Execute Katalon True Platform/TestOps tests when the input is an existing test case, manual test case list, test suite, suite collection, execution request, or "run with AI" instruction. Use when you need to create a manual test run, start Run with AI, poll AI session results, schedule automated suites, read execution/test results, or summarize pass/fail/blocked outcomes. For full requirement-to-test-design-to-execution workflows, combine with or defer to katalon-true-platform-testing. Written for the manual tester who has cases and needs a result, by hand or through Run with AI. A coded suite driven from a framework starts at katalon-playwright-execute or katalon-upload-report.
Comment thread skills/katalon-exploratory-charter/SKILL.md
@@ -0,0 +1,177 @@
---
name: katalon-exploratory-charter
description: Write, run, and debrief exploratory testing charters against Katalon True Platform/TestOps when there is no script to follow. Use when you need to turn a vague area into a charter (mission, areas, oracles, timebox), run a timeboxed unscripted session, log what you find as session notes, judge which findings are real defects, and promote what was learned into atomic manual cases plus the folder or suite that holds them. This is unscripted discovery and the debrief after it; when a written requirement is already in hand and the job is designing cases from it, use katalon-create-test-cases instead. Written for the manual tester who has an hour, a feature, and no cases to run against it.
Comment thread skills/katalon-platform-setup/SKILL.md
@@ -0,0 +1,160 @@
---
name: katalon-platform-setup
description: Set up, verify, and troubleshoot Katalon True Platform/TestOps MCP connectivity for any AI coding agent. Use when you need to install or configure the Katalon MCP server, connect to a Katalon platform account, verify available Katalon MCP tools, diagnose missing authentication or repository access, confirm project/repository discovery, or explain what platform operations are and are not exposed through the MCP. One-time cross-cutting setup, run once by whoever wires the agent to the platform, before any lifecycle skill can work.
Comment thread skills/katalon-test-estimation/references/mcp-evidence-map.md
- **No roster or capacity.** No team members, FTE fractions, PTO, or working calendar.
- **No cost model.** No rates, currency, license consumption, or TestCloud minute pricing.
- **No place to store the estimate.** No estimated-effort field on a case, folder, suite, or requirement, and per `unavailable-capabilities.md` no Test Plan, Release, or Build entity to attach one to. Record it in the plan suite's description or the linked Jira/Azure issue, and hand off to `katalon-test-plan`.
- **No cross-cycle aggregate.** Result and execution tools are record-level. Trending across releases belongs to a reporting skill that does not exist yet - state the boundary rather than assembling a trend from single reads and presenting it as one.
Comment thread skills/katalon-test-reporting/references/snapshot-and-trend.md

## The snapshot file

Written on every run, to `.katalon-reporting/<project-slug>.json` in the user's workspace. Plain JSON, appended to, never rewritten. Ask before creating the directory the first time.
Comment thread skills/katalon-true-platform-testing/references/lifecycle-map.md
Comment on lines +83 to +84
| What test data do these cases need | `katalon-create-test-cases`, per-case test data field only | `katalon-test-data` |
| Seed and tear down data for the run | **boundary only.** No skill owns this. State it | `katalon-test-data` |
Comment thread skills/katalon-true-platform-testing/references/prompt-recipes.md
## Non-Claude agent notes

- **Copilot / Cursor / Kiro / Windsurf / Cline / Continue:** invoke the skill by its name or `/skill-name`; the MCP tool calls are identical. The generated adapter for each agent carries the same skill body.
- **Codex / ChatGPT:** the plugin declares the MCP in `.mcp.json`; follow the numbered steps exactly, they are model-agnostic.
Comment thread skills/katalon-upload-report/SKILL.md
-testOpsProjectId="<project_id>" \
--config \
-webui.autoUpdateDrivers=true \
-reportFolder="<run_report_folder>"
@huytieu @claude
Address risk-scan findings: drop npx auto-yes from shell examples
852fa94 
Both High findings were `npx -y` in copy-paste shell commands. Removed
the auto-yes so the reader approves the install themselves, and noted
why the spawned-server JSON/TOML config still needs `-y` (no TTY to
answer the prompt on).

Also unpinned-version: `npm init playwright@latest` -> `npm init
playwright`.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VjgFMQPShixvLoyCZfwrfL
Copilot AI review requested due to automatic review settings September 9, 2026 07:14
huytieu added a commit to katalon-labs/true-skills that referenced this pull request Sep 9, 2026
@huytieu @claude
platform-setup: drop npx auto-yes from shell examples
e6cdd77 
Automated risk scanners flag `npx -y` in copy-paste shell commands as
auto-approved package execution. Removed the flag from the two
`mcp-remote` one-liners so the reader approves the install, and noted
why the spawned-server JSON/TOML config keeps `-y` (no TTY to answer
the prompt on).

Also `npm init playwright@latest` -> `npm init playwright` in the
test-case-to-playwright reference, which trips unpinned-version rules.

Found via the github/awesome-copilot PR risk scan (github/awesome-copilot#2995).

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VjgFMQPShixvLoyCZfwrfL
@huytieu

huytieu commented Sep 9, 2026

Copy link
Copy Markdown
Author

Thanks — risk scan addressed in 852fa94. High is now 0.

Both High findings (autoyes-package-exec) were npx -y in copy-paste shell commands. Dropped the -y so the reader approves the install themselves. The JSON/TOML config block still carries -y in its args array, deliberately: a spawned MCP server has no terminal to answer the install prompt on, so removing it there would hang the handshake rather than make it safer. Added a line saying exactly that, so the difference reads as intentional.

unpinned-version-indicator — npm init playwright@latest → npm init playwright.

The remaining 15 Medium are all package-exec-command, and I've left them on purpose:

  • 10 are npx playwright test, the command Playwright's own docs use to run a suite. These skills exist to drive Playwright, so rewording it would make them wrong.
  • 4 are the mcp-remote proxy invocation and its "command": "npx" config, which is the documented fallback for agents without native remote-MCP support.
  • 1 is the new sentence explaining the -y distinction, which trips the rule by quoting it.

For precedent, npx -y currently appears in acreadiness-policy, acreadiness-assess, acreadiness-generate-instructions, workiq-copilot, transloadit-media-processing, and webmcpify, so I don't think the pattern itself is disqualifying — but happy to change any of the above if you'd rather these skills avoid it entirely.

Same fix landed upstream in katalon-labs/true-skills (e6cdd77) so the source of truth and this port don't drift.

Copilot AI reviewed Sep 9, 2026
View reviewed changes

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Several routes reference obsolete fallbacks, required assets are missing, and documented setup/upload workflows contain broken instructions.

Review details

Suppressed comments (30)

Previously missed (4) — in code that hasn't changed since the last review.

skills/katalon-playwright-execute/references/playwright-katalon-reporting.md:99

  • The preceding installation guidance adds the reporter as a local dev dependency, whose binary is not normally on the interactive shell's PATH. This command can therefore fail with “command not found”; execute the local binary through the package runner.
    skills/katalon-true-platform-testing/references/lifecycle-map.md:87
  • katalon-test-estimation now exists in this PR, so keeping katalon-test-plan as today's route bypasses the skill that owns duration and staffing estimates.

This issue also appears in the following locations of the same file:

  • line 100
  • line 107
    skills/katalon-upload-report/SKILL.md:137
  • This standalone command assumes playwright-reporter is globally available, while the skill only asks users to install @katalon/playwright-reporter locally. Invoke the local binary through npx so the documented upload flow works from a normal project shell.
    skills/katalon-platform-setup/SKILL.md:84
  • The target repository's README has no Katalon per-agent MCP configuration section, so an installed skill cannot obtain the promised file paths from it. Link to actual setup documentation or bundle the per-agent paths with this skill.

skills/katalon-true-platform-testing/references/lifecycle-map.md:84

  • These routes still describe katalon-test-data as a future gap even though that skill is added by this PR. The orchestrator will incorrectly send data-design and seed/teardown requests to the fallback instead of the dedicated skill.
    skills/katalon-test-case-to-katalon-studio/SKILL.md:75
  • The skill requires these two reference guides before authoring, but this directory contains only SKILL.md; both referenced files are absent. An installed skill therefore cannot follow its mandatory file-format and Groovy guidance. Add the referenced assets (and regenerate the skills README) or make this document self-contained.
Read `references/studio-project-anatomy.md` before writing any file. Read `references/groovy-authoring.md` before writing script or custom keyword code.

skills/katalon-true-platform-testing/references/lifecycle-map.md:100

  • The PR adds katalon-test-case-to-katalon-studio, but this router still claims authoring is unavailable and sends the request to an execution/upload skill. Route this intent to the new conversion skill.
    skills/katalon-true-platform-testing/references/lifecycle-map.md:108
  • Both intents are now owned by the newly added katalon-test-reporting skill. Leaving the old fallback routes causes the orchestrator to refuse cross-period reporting that this PR explicitly implements.
    skills/katalon-test-estimation/references/mcp-evidence-map.md:50
  • This boundary is stale because katalon-test-reporting is added in the same PR and explicitly handles cross-cycle trends. It will incorrectly stop estimation users instead of handing the request to the available skill.
- **No cross-cycle aggregate.** Result and execution tools are record-level. Trending across releases belongs to a reporting skill that does not exist yet - state the boundary rather than assembling a trend from single reads and presenting it as one.

skills/katalon-platform-setup/SKILL.md:74

  • No .mcp.json is bundled with this skill or elsewhere in this repository, so this instruction points installers to a nonexistent asset. Use the inline snippet as the canonical example or add the promised file.
   See the bundled `.mcp.json` for the canonical shape.

skills/katalon-true-platform-testing/references/prompt-recipes.md:38

  • No Katalon plugin or .mcp.json is added to this repository, so this note falsely tells Codex/ChatGPT users that MCP is already declared. Direct them through the setup skill instead of assuming an absent plugin configuration.
    skills/katalon-test-reporting/references/snapshot-and-trend.md:16
  • A closed JSON document cannot be appended to without becoming invalid JSON. Since the example stores snapshots inside an array, adding a snapshot must rewrite the document while preserving earlier entries (or the format must change to JSON Lines).
    skills/katalon-analyze-failures/SKILL.md:3
  • The Agent Skills frontmatter convention requires the description value to be wrapped in single quotes.
description: Triage Katalon True Platform/TestOps test failures and file defects. Use when you need to investigate failed test results, classify each failure as product defect vs automation defect vs environment/data issue, cluster failures by common signature, find likely root cause from execution data, and optionally create ALM-linked defects for real product bugs. This is failure diagnosis and defect filing; for the overall ship/no-ship release call use katalon-release-analyze, and for repairing the tests themselves use katalon-test-maintenance. Written for the manual tester and the automation tester looking at a red run and needing to know whether the application broke or the test did.

skills/katalon-create-test-cases/SKILL.md:3

  • The Agent Skills frontmatter convention requires the description value to be wrapped in single quotes.
description: Create, update, organize, and link Katalon True Platform/TestOps manual test cases from a synced requirement key such as CEL-6, a read requirement, or free-text product behavior. Use when you need to analyze requirements, design manual test cases using ISTQB techniques as a reference, check existing Katalon coverage, avoid duplicate test cases, import only missing cases, update or link existing cases, or create/reuse a test suite for newly designed cases. For full requirement-to-execution workflows, combine with or defer to katalon-true-platform-testing. Written for the manual tester who has a written requirement in hand and no cases for it yet.

skills/katalon-execute-test/SKILL.md:3

  • The Agent Skills frontmatter convention requires the description value to be wrapped in single quotes.
description: Execute Katalon True Platform/TestOps tests when the input is an existing test case, manual test case list, test suite, suite collection, execution request, or "run with AI" instruction. Use when you need to create a manual test run, start Run with AI, poll AI session results, schedule automated suites, read execution/test results, or summarize pass/fail/blocked outcomes. For full requirement-to-test-design-to-execution workflows, combine with or defer to katalon-true-platform-testing. Written for the manual tester who has cases and needs a result, by hand or through Run with AI. A coded suite driven from a framework starts at katalon-playwright-execute or katalon-upload-report.

skills/katalon-exploratory-charter/SKILL.md:3

  • The Agent Skills frontmatter convention requires the description value to be wrapped in single quotes.
description: Write, run, and debrief exploratory testing charters against Katalon True Platform/TestOps when there is no script to follow. Use when you need to turn a vague area into a charter (mission, areas, oracles, timebox), run a timeboxed unscripted session, log what you find as session notes, judge which findings are real defects, and promote what was learned into atomic manual cases plus the folder or suite that holds them. This is unscripted discovery and the debrief after it; when a written requirement is already in hand and the job is designing cases from it, use katalon-create-test-cases instead. Written for the manual tester who has an hour, a feature, and no cases to run against it.

skills/katalon-platform-setup/SKILL.md:3

  • The Agent Skills frontmatter convention requires the description value to be wrapped in single quotes.
description: Set up, verify, and troubleshoot Katalon True Platform/TestOps MCP connectivity for any AI coding agent. Use when you need to install or configure the Katalon MCP server, connect to a Katalon platform account, verify available Katalon MCP tools, diagnose missing authentication or repository access, confirm project/repository discovery, or explain what platform operations are and are not exposed through the MCP. One-time cross-cutting setup, run once by whoever wires the agent to the platform, before any lifecycle skill can work.

skills/katalon-playwright-execute/SKILL.md:3

  • The Agent Skills frontmatter convention requires the description value to be wrapped in single quotes.
description: Run Playwright tests or suites and upload the resulting report to Katalon True Platform. Use when you need to execute Playwright scripts, package scripts, spec files, projects, or suites, configure or verify @katalon/playwright-reporter, upload Playwright reports with Katalon CLI/reporter commands, and verify uploaded Katalon Platform runs or reports. Written for the automation tester who wants the suite driven and the results landed in the platform in one step. For a report from a run that has already finished, or from any other framework, use katalon-upload-report.

skills/katalon-release-analyze/SKILL.md:3

  • The Agent Skills frontmatter convention requires the description value to be wrapped in single quotes.
description: Analyze Katalon True Platform/TestOps release readiness from testing quality data. Use when you need to use Katalon MCP metrics and results to assess whether a release, sprint, iteration, version, test plan, suite, or repository is ready to ship; summarize requirement coverage, execution health, defect risk, test stability, configuration coverage, release blockers, quality gaps, and produce a Ready / Ready with risk / Not ready recommendation. Written for the test manager who owns the ship call and the test lead who has to defend it.

skills/katalon-test-case-to-katalon-studio/SKILL.md:3

  • The Agent Skills frontmatter convention requires the description value to be wrapped in single quotes.
description: Convert Katalon True Platform/TestOps manual test cases into Katalon Studio automation inside a local Studio Test Project checkout. Use when you need to author or extend a .tc test case file and its paired Groovy script under Scripts/, keep test case variable GUIDs consistent with the .ts test suite bindings that read them, model page elements as .rs test objects in the Object Repository instead of inline selectors, write WebUI, Mobile, or WS keyword steps in Groovy with Java interop, or wire data files, execution profiles, and GlobalVariable into a generated case. Also use to decide whether a headless katalonc run is possible at all, since Katalon Runtime Engine is a separately licensed paid runtime whose CLI requires -apiKey. Once a Studio run has finished and its report exists, hand off to katalon-upload-report. Written for the automation tester who has to land a manual case inside a Katalon Studio project rather than a code-first framework.

skills/katalon-test-case-to-playwright/SKILL.md:3

  • The Agent Skills frontmatter convention requires the description value to be wrapped in single quotes.
description: Convert Katalon True Platform/TestOps manual test cases, test suites, or requirement-linked cases into Playwright TypeScript automation. Use when you need to fetch/read Katalon Platform test cases and implement Playwright scripts, create or adapt a Playwright framework, apply Page Object Model and fixtures, or translate manual steps into meaningful automated test keywords. Written for the automation tester converting a manual case into code that fits an existing page-object layer.

skills/katalon-test-data/SKILL.md:3

  • The Agent Skills frontmatter convention requires the description value to be wrapped in single quotes.
description: Design, source, seed, and tear down the test data a Katalon True Platform test case or an automated suite runs on. Use when the steps are already settled and the blocker is the values, for example which data classes a case needs, which records must exist before a run, how to keep literals out of the step text and into the Test Data column or a fixture, and how to reset state afterwards so the next run starts clean. Covers choosing between static, generated, and cloned production data, keeping credentials out of test data, and the boundary that the Katalon MCP has no test data, fixture, seeding, or secrets tool of its own. If the cases do not exist yet, start at katalon-create-test-cases. Written for the manual tester filling in a case's Test Data column and precondition, and the automation tester wiring fixtures and teardown for a suite.

skills/katalon-test-estimation/SKILL.md:3

  • The Agent Skills frontmatter convention requires the description value to be wrapped in single quotes.
description: Estimate testing effort, duration, and resourcing for a Katalon True Platform/TestOps cycle. Use when the question is how long testing will take, how many testers it needs, whether the scope fits the sprint window, or what a scope change costs in person-hours. Sizes design, manual execution, automated execution and triage, and rework separately, counts the countable part from platform data (case counts, automation split, historical pass and stability rates, configuration matrix), calibrates the rest against a rate the team supplies, and returns a three-point range with a confidence label instead of a single number. Splits resourcing across the manual and automated lanes and names the assumptions that would move the number most. For what to test and in what order, use katalon-test-plan; for a verdict on a cycle that has already run, use katalon-release-analyze. Written for the test lead sizing a cycle before it starts and the test manager who has to fund it.

skills/katalon-test-maintenance/SKILL.md:3

  • The Agent Skills frontmatter convention requires the description value to be wrapped in single quotes.
    skills/katalon-test-management/SKILL.md:3
  • The Agent Skills frontmatter convention requires the description value to be wrapped in single quotes.
    skills/katalon-test-plan/SKILL.md:3
  • The Agent Skills frontmatter convention requires the description value to be wrapped in single quotes.
    skills/katalon-test-reporting/SKILL.md:3
  • The Agent Skills frontmatter convention requires the description value to be wrapped in single quotes.
    skills/katalon-test-review/SKILL.md:3
  • The Agent Skills frontmatter convention requires the description value to be wrapped in single quotes.
    skills/katalon-true-platform-testing/SKILL.md:3
  • The Agent Skills frontmatter convention requires the description value to be wrapped in single quotes.
    skills/katalon-upload-report/SKILL.md:3
  • The Agent Skills frontmatter convention requires the description value to be wrapped in single quotes.
  • Files reviewed: 54/54 changed files
  • Comments generated: 0 new
  • Review effort level: Balanced

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

Copilot code review Copilot
Copilot review effort, defaults to Balanced
Applies to this pull request for everyone.Learn more about Copilot code review.
Copilot left review comments
@aaronpowell aaronpowell Awaiting requested review from aaronpowell aaronpowell is a code owner

At least 1 approving review is required to merge this pull request.

Assignees

No one assigned

Labels

new-submission PR adds at least one new contribution skills PR touches skills

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@huytieu