Add Agent Playbook section for automated updates - #142
Conversation
Not up to standards ⛔🔴 Issues
|
| Category | Results |
|---|---|
| BestPractice | 2 medium (2 false positives) |
AI Reviewer: first review requested successfully. AI can make mistakes. Always validate suggestions.
TIP This summary will be updated as you push new changes.
There was a problem hiding this comment.
Pull Request Overview
The addition of the Agent Playbook is a significant improvement for automation, and the PR currently meets Codacy quality standards. However, the documentation contains specific point-in-time version examples that will likely mislead automated agents in future update cycles.
There are also gaps in the verification of the playbook's technical accuracy. Specifically, the Makefile targets ('make documentation', etc.) and the file paths for configuration files mentioned in the text have not been cross-referenced with the actual project structure. Addressing these gaps is essential to ensure the playbook is a reliable source of truth for both humans and AI agents.
About this PR
- The playbook references several Makefile targets and configuration file paths (e.g., .circleci/config.yml). Please verify that these targets and paths are correct and functional within the current repository structure to prevent the documentation from providing incorrect instructions to automated agents.
Test suggestions
- Verify Markdown rendering for tables, lists, and formatting in the new Playbook section.
- Verify the accuracy of the documented Makefile targets ('make documentation', 'make configure', 'make build') against the actual project structure.
- Verify that the listed version-encoding files (e.g., .github/dependabot.yml, .circleci/config.yml) exist and match the described paths.
Prompt proposal for missing tests
Consider implementing these tests if applicable:
1. Verify the accuracy of the documented Makefile targets ('make documentation', 'make configure', 'make build') against the actual project structure.
2. Verify that the listed version-encoding files (e.g., .github/dependabot.yml, .circleci/config.yml) exist and match the described paths.
TIP Improve review quality by adding custom instructions
TIP How was this review? Give us feedback
|
|
||
| The `docs/` directory is machine-consumed configuration, not just documentation: | ||
|
|
||
| - `docs/patterns.json` — the list of Sonar VB.NET rules ("patterns") Codacy knows about, their categories/severities/parameters, and which are enabled by default. Generated file, do not hand-edit. **Note:** as of this writing its `"version"` field reads `8.13` while `Analyzer.csproj` pins `SonarAnalyzer.VisualBasic` at `8.15.0.24505` — regeneration has historically lagged behind dependency bumps, so don't assume the two are in sync unless you check. |
There was a problem hiding this comment.
🟡 MEDIUM RISK
Suggestion: Avoid hardcoding specific point-in-time version numbers in the descriptive text. This documentation will quickly become stale and may mislead future automated updates.
| - `docs/patterns.json` — the list of Sonar VB.NET rules ("patterns") Codacy knows about, their categories/severities/parameters, and which are enabled by default. Generated file, do not hand-edit. **Note:** as of this writing its `"version"` field reads `8.13` while `Analyzer.csproj` pins `SonarAnalyzer.VisualBasic` at `8.15.0.24505` — regeneration has historically lagged behind dependency bumps, so don't assume the two are in sync unless you check. | |
| - `docs/patterns.json` — the list of Sonar VB.NET rules ("patterns") Codacy knows about, their categories/severities/parameters, and which are enabled by default. Generated file, do not hand-edit. **Note:** its "version" field often lags behind the `SonarAnalyzer.VisualBasic` version pinned in `Analyzer.csproj` — regeneration has historically lagged behind dependency bumps, so do not assume the two are in sync. |
| ### 3. Step-by-step update procedure | ||
|
|
||
| 1. **Bump the version(s)** in `src/Analyzer/Analyzer.csproj` (and `.github/dependabot.yml` ignore list / `.circleci/config.yml` orbs, if applicable) as scoped by the task. | ||
| 2. **Regenerate the docs.** Requires `xmllint` (`apt-get install libxml2-utils` or `apk add libxml2-utils`), Mono, and `dotnet`, plus network access to download the Sonar plugin JAR: `make documentation`. This runs `update-docs` (downloads `sonar-vbnet-plugin-<version>.jar` from the sonar-dotnet GitHub release matching the bumped version, unzips it, extracts `sonar-vbnet-rules.xml`), then `build-docs` (compiles `src/DocsGenerator`), then runs `DocsGenerator.exe` to regenerate `docs/patterns.json` and `docs/description/*`. Review the diff for new/removed/renamed patterns and stale fixture references. Confirm `docs/patterns.json`'s `"version"` field now matches the bumped version (it can otherwise silently drift, as noted above). |
There was a problem hiding this comment.
⚪ LOW RISK
Suggestion: This section duplicates requirements and commands already documented in the 'Generate Docs' section. Additionally, the playbook depends on docker and gh which are not listed in the initial requirements summary.
Try running the following prompt in your coding agent:
Consolidate the documentation generation instructions by referencing the 'Generate Docs' section for core requirements, and update the 'Agent Playbook' tool requirements to include 'gh' (GitHub CLI) and 'docker' for the end-to-end workflow.
| | `src/Analyzer/Analyzer.csproj` → `Microsoft.CodeAnalysis.VisualBasic.Workspaces` | Roslyn VB workspace APIs the analyzer host depends on | Check [NuGet](https://www.nuget.org/packages/Microsoft.CodeAnalysis.VisualBasic.Workspaces) for compatibility with the target Sonar analyzer version before bumping. | | ||
| | `src/Analyzer/Analyzer.csproj` → `Codacy.Engine.Seed` | Codacy's .NET engine SDK | Check NuGet for newer versions if asked to update it; not tied to Sonar bumps. | | ||
| | `src/Analyzer/Analyzer.csproj` → `ReverseMarkdown`, `Newtonsoft.Json`, `SQLitePCLRaw.core` | Supporting libraries (used by `DocsGenerator`/`Analyzer`) | Handled routinely by Dependabot; bump manually only if asked or if blocking another change. | | ||
| | `.github/dependabot.yml` → `ignore` list | Dependabot is currently told to **skip** several newer `SonarAnalyzer.VisualBasic` and `ReverseMarkdown` releases (e.g. `8.17.0.26580` through `8.21.0.30542`) | If you are deliberately bumping past one of these ignored versions, remove/update the corresponding `ignore` entry so Dependabot doesn't immediately re-flag or conflict; if those versions were ignored due to a known incompatibility, investigate why before un-ignoring. | |
There was a problem hiding this comment.
⚪ LOW RISK
Suggestion: Avoid using specific version strings in the Dependabot examples to prevent the documentation from becoming immediately outdated.
| | `.github/dependabot.yml` → `ignore` list | Dependabot is currently told to **skip** several newer `SonarAnalyzer.VisualBasic` and `ReverseMarkdown` releases (e.g. `8.17.0.26580` through `8.21.0.30542`) | If you are deliberately bumping past one of these ignored versions, remove/update the corresponding `ignore` entry so Dependabot doesn't immediately re-flag or conflict; if those versions were ignored due to a known incompatibility, investigate why before un-ignoring. | | |
| `.github/dependabot.yml` → `ignore` list | Dependabot is currently told to **skip** several `SonarAnalyzer.VisualBasic` and `ReverseMarkdown` releases. | If you are deliberately bumping past an ignored version, update the corresponding `ignore` entry so Dependabot doesn't immediately re-flag or conflict; investigate why versions were ignored before un-ignoring. | |
Summary
Test plan