Rust: rust-analyzer update automation

.gitattributes

@@ -75,6 +75,9 @@
 /ruby/extractor/cargo-bazel-lock.json linguist-generated=true
 /ruby/extractor/cargo-bazel-lock.json -merge
 
+# GitHub Agentic Workflows compiled output
+.github/workflows/*.lock.yml linguist-generated=true
+
 # auto-generated files for the C# build
 /csharp/paket.lock linguist-generated=true
 # needs eol=crlf, as `paket` touches this file and saves it as crlf

.github/workflows/update-rust-analyzer.md

@@ -0,0 +1,131 @@
+---
+name: Update rust-analyzer
+emoji: 🦀
+description: Update the rust-analyzer version used by the Rust extractor and prepare a pull request.
+intent: Keep the Rust extractor on the latest compatible rust-analyzer version with a reviewable, validated pull request.
+on:
+  workflow_dispatch:
+  roles: [admin, maintainer, write]
+permissions:
+  contents: read
+  actions: read
+  pull-requests: read
+  copilot-requests: write
+strict: true
+checkout:
+  fetch-depth: 0
+concurrency:
+  group: update-rust-analyzer
+  cancel-in-progress: false
+timeout-minutes: 180
+tools:
+  github:
+    mode: gh-proxy
+    toolsets: [repos, pull_requests, actions]
+  bash: ["*"]
+  edit: true
+network:
+  allowed:
+    - defaults
+    - github
+    - github-actions
+    - rust
+    - bazel
+    - python
+steps:
+  - name: Configure Git
+    run: |
+      git config user.name "github-actions[bot]"
+      git config user.email "41898282+github-actions[bot]@users.noreply.github.com"
+
+  - name: Install cargo-edit
+    continue-on-error: true
+    run: cargo install cargo-edit@0.13.0 --locked
+
+  - name: Run rust-analyzer updater script
+    run: |
+      mkdir -p /tmp/gh-aw/agent
+      set +e
+      python3 rust/scripts/update_rust_analyzer.py \
+        2>&1 | tee /tmp/gh-aw/agent/rust-analyzer-update.log
+      status=${PIPESTATUS[0]}
+      set -e
+
+      printf '%s\n' "$status" > /tmp/gh-aw/agent/rust-analyzer-update-status.txt
+      if grep -Fxq "No new rust-analyzer version available." \
+        /tmp/gh-aw/agent/rust-analyzer-update.log; then
+        printf '%s\n' "no-update" > /tmp/gh-aw/agent/rust-analyzer-update-result.txt
+      else
+        printf '%s\n' "update" > /tmp/gh-aw/agent/rust-analyzer-update-result.txt
+      fi
+safe-outputs:
+  report-incomplete: {}
+  create-pull-request:
+    title-prefix: "Rust: "
+    branch-prefix: "automation/update-rust-analyzer/"
+    draft: true
+    max-patch-size: 10240
+    max-patch-files: 1000
+    allowed-files:
+      - "Cargo.lock"
+      - "shared/tree-sitter-extractor/Cargo.toml"
+      - "shared/yeast/Cargo.toml"
+      - "shared/yeast-macros/Cargo.toml"
+      - "shared/yeast-schema/Cargo.toml"
+      - "ruby/extractor/Cargo.toml"
+      - "unified/extractor/Cargo.toml"
+      - "unified/swift-syntax-rs/Cargo.toml"
+      - "MODULE.bazel"
+      - "MODULE.bazel.lock"
+      - "rust-toolchain.toml"
+      - "rust/**"
+      - "misc/bazel/3rdparty/**"
+---
+
+# Update rust-analyzer
+
+## Task
+
+The workflow has already run `rust/scripts/update_rust_analyzer.py`. Read:
+
+- `/tmp/gh-aw/agent/rust-analyzer-update.log` for its complete output.
+- `/tmp/gh-aw/agent/rust-analyzer-update-status.txt` for its exit status.
+- `/tmp/gh-aw/agent/rust-analyzer-update-result.txt` for the deterministic result classification.
+
+If the result is `no-update`, call `noop` with the reason
+`No new rust-analyzer version available.` and stop immediately. Do not inspect
+CI, modify files, or create a pull request.
+
+Otherwise, continue the update from the existing working tree and commits:
+
+1. Read `rust/updating-rust-analyzer.md` and all applicable repository
+   instructions.
+2. Review the updater log, exit status, commits, and working tree. Do not rerun
+   the updater script.
+3. Complete as much of the documented update as possible. Fix extractor or
+   code-generation breakage, keep all `ra_ap_` dependency versions aligned,
+   regenerate required files, and add schema upgrade/downgrade scripts, tests,
+   and a change note when the schema changed.
+4. Run the relevant formatting, linting, code generation, build, and tests,
+   including `bazel run //rust:install`. Use `gh` to inspect relevant existing
+   CI configuration and prior failures while diagnosing problems.
+5. Review the complete diff and commits. Do not include secrets. Do not refer
+   to private repositories, internal issues, or internal pull requests in the
+   public pull request.
+6. Use `create_pull_request` exactly once to create a focused draft pull
+   request. Summarize the updater output, changes, and validation. If the
+   update cannot be finished, still create a partial draft pull request when
+   there are useful changes, and clearly list failures, missing work, and the
+   next commands for a maintainer.
+
+The pull request is created after this agent execution, so its newly triggered
+CI cannot be awaited in this run. Compensate with the strongest practical
+local validation and state this limitation accurately in the pull request.
+If the updater did not report `no-update` but no useful patch can be produced,
+call `report_incomplete` with the updater failure and exact blocker.
+
+## Safe Outputs
+
+- Use `create_pull_request` for the update or partial update.
+- Use `noop` only for the exact no-update result.
+- Use `report_incomplete` only when no useful pull request can be created.

rust/README.md

@@ -62,58 +62,4 @@ for code generation to succeed.
 
 ### Updating `rust-analyzer`
 
-Here's a rundown of the typical actions to perform to do a rust-analyzer (and other dependencies) update. A one-time setup consists in
-installing [`cargo-edit`](https://crates.io/crates/cargo-edit) with `cargo install cargo-edit`. On Ubuntu that also requires
-`sudo apt install libssl-dev pkg-config`.
-
-1. From the root of the `codeql` repo checkout, run a Cargo upgrade:
-   ```
-   cargo upgrade --incompatible --pinned
-   ```
-2. Look at a diff of the `Cargo.toml` files: if all `ra_ap_` prefixed dependencies have been updated to the same number, go on to the next step.
-   Otherwise, it means the latest `rust-analyzer` update has not been fully rolled out to all its crates in `crates.io`.
-   _All `ra_ap_` versions must agree!_
-   Downgrade by hand to the minimum one you see, and run a `cargo update` after that to fix the `Cargo.lock` file.
-3. Commit the changes, skipping `pre-commit` hooks if you have them enabled:
-   ```
-   git commit -am 'Cargo: upgrade dependencies' --no-verify
-   ```
-4. Regenerate vendored bazel files (these allow faster builds, particularly on CI where it has to start from scratch each time), commit the changes:
-   ```
-   misc/bazel/3rdparty/update_tree_sitter_extractors_deps.sh
-   git add .
-   git commit -am 'Bazel: regenerate vendored cargo dependencies' --no-verify
-   ```
-   > [!NOTE]
-   > If in step 6 you also bump `rules_rust` or the rust toolchain, those changes invalidate _all_ vendored files (including the
-   > Python ones under `misc/bazel/3rdparty/py_deps`), not just the tree-sitter ones. In that case run the umbrella script
-   > `misc/bazel/3rdparty/update_cargo_deps.sh` instead (it regenerates both `py_deps` and `tree_sitter_extractors_deps`, and runs
-   > `bazel mod tidy`), then commit all the regenerated files.
-5. Run codegen
-   ```
-   bazel run //rust/codegen
-   ```
-   Take note whether `rust/schema/ast.py` was changed. That might need tweaks, new tests and/or downgrade/upgrade scripts down the line.
-6. Try compiling
-   ```
-   bazel run //rust:install
-   ```
-   * if it succeeds: good! You can move on to the next step.
-   * if it fails while compiling rust-analyzer dependencies, you need to update the rust toolchain. Sometimes the error will tell you
-     so explicitly, but it may happen that the error is more obscure. To update the rust toolchain:
-      * you will need to open a PR on the internal repo updating `RUST_VERSION` in `MODULE.bazel`. In general you can have this merged
-        independently of the changes in `codeql`.
-      * in `codeql`, update both `RUST_VERSION` in `MODULE.bazel` _and_ `rust-toolchain.toml` files. You may want to also update the
-        nightly toolchain in `rust/extractor/src/nightly-toolchain/rust-toolchain.toml` to a more recent date while you're at it.
-      * a toolchain and/or `rules_rust` bump invalidates the vendored files, so re-run `misc/bazel/3rdparty/update_cargo_deps.sh`
-       (see the note in step 4) and commit the regenerated files.
-   * if it fails while compiling rust extractor code, you will need to adapt it to the new library version.
-      * for example updating annotations in `annotations.py`, adding / removing generated tests.
-
-   If you had to do any changes, commit them. If you updated the rust toolchain, running `rust/lint.py` might reformat or apply new
-   lints to the code.
-7. Check with CI if everything is in order.
-8. Run DCA with database caching disabled. Iterate on the code if needed.
-9. If in step 5 the schema was updated, add upgrade/downgrade scripts and a change note. This is best done last to reduce the chance of
-merge conflicts (none of the other testing depends on having upgrade and downgrade scripts in place). See
-[Upgrading a language database schema](docs/prepare-db-upgrade.md).
+See [Updating rust-analyzer](updating-rust-analyzer.md).

rust/scripts/update_rust_analyzer.py

@@ -0,0 +1,184 @@
+#!/usr/bin/env python3
+
+import json
+import shlex
+import subprocess
+import sys
+import tomllib
+import urllib.request
+from pathlib import Path
+
+
+REPO_ROOT = Path(__file__).resolve().parents[2]
+RUST_EXTRACTOR_MANIFEST = REPO_ROOT / "rust/extractor/Cargo.toml"
+
+# The files that mention the fixed Rust toolchain version
+TOOLCHAIN_RS = REPO_ROOT / "rust/extractor/src/toolchain.rs"
+INTEGRATION_TEST_CONFTEST = REPO_ROOT / "rust/ql/integration-tests/conftest.py"
+QL_TEST_SETUP = REPO_ROOT / "rust/ql/test/setup.sh"
+
+
+def print_step(number: int, description: str) -> None:
+    print(f"\nStep {number}: {description}", flush=True)
+
+
+def run(*command: str) -> None:
+    print(f"+ {shlex.join(command)}", flush=True)
+    subprocess.run(command, cwd=REPO_ROOT, check=True)
+
+
+def run_codegen() -> None:
+    try:
+        run("bazel", "run", "//rust/codegen")
+    except subprocess.CalledProcessError as error:
+        print(
+            "\nCodegen failed. Carry out the instructions from step 4 in rust/updating-rust-analyzer.md manually.",
+            file=sys.stderr,
+            flush=True,
+        )
+        raise SystemExit(error.returncode) from None
+
+
+def get_rust_analyzer_version() -> str:
+    """Get the minimum version of all ra_ap dependencies from `Cargo.toml`."""
+    dependencies = tomllib.loads(RUST_EXTRACTOR_MANIFEST.read_text())["dependencies"]
+    return min(
+        (
+            version
+            for name, version in dependencies.items()
+            if name.startswith("ra_ap_")
+        ),
+        key=lambda value: tuple(map(int, value.split("."))),
+    )
+
+
+def fetch(url: str) -> bytes:
+    request = urllib.request.Request(
+        url,
+        headers={"User-Agent": "github/codeql rust-analyzer updater"},
+    )
+    with urllib.request.urlopen(request) as response:
+        return response.read()
+
+
+def get_compatible_rust_toolchain(rust_analyzer_version: str) -> str:
+    """Get the latest Rust toolchain that precedes our version of rust-analyzer."""
+    # Get the release date of the rust-analyzer version
+    crate_url = f"https://crates.io/api/v1/crates/ra_ap_syntax/{rust_analyzer_version}"
+    crate = json.loads(fetch(crate_url))
+    rust_analyzer_release = crate["version"]["created_at"].split("T")[0]
+
+    # `manifests.txt` is a list of all toolchains. The one we're interested in looks like
+    # ```
+    # static.rust-lang.org/dist/YYYY-MM-DD/channel-rust-stable.toml
+    # ```
+    # where `YYYY-MM-DD` is the last that is earlier than the rust-analyzer release.
+    rust_manifest = next(
+        manifest
+        for manifest in reversed(
+            fetch("https://static.rust-lang.org/manifests.txt").decode().splitlines()
+        )
+        if manifest.endswith("/channel-rust-stable.toml")
+        and manifest.split("/")[2] <= rust_analyzer_release
+    )
+    manifest = tomllib.loads(fetch(f"https://{rust_manifest}").decode())
+    # The version looks like `version = "0.99.0 (797e8a9bc 2026-08-05)"` - we only want the first part.
+    return manifest["pkg"]["rust"]["version"].split()[0]
+
+
+def update_fixed_rust_toolchain(version: str) -> None:
+    """Change the fixed toolchain in the places where it's hardcoded"""
+    toolchain_rs = TOOLCHAIN_RS.read_text()
+    prefix = 'const FIXED_RUST_TOOLCHAIN: &str = "'
+    old_version = toolchain_rs.split(prefix, 1)[1].split('"', 1)[0]
+    TOOLCHAIN_RS.write_text(
+        toolchain_rs.replace(
+            f'{prefix}{old_version}"',
+            f'{prefix}{version}"',
+        )
+    )
+
+    for test_setup in (INTEGRATION_TEST_CONFTEST, QL_TEST_SETUP):
+        contents = test_setup.read_text()
+        test_setup.write_text(
+            contents.replace(
+                f"rustup toolchain install {old_version} ",
+                f"rustup toolchain install {version} ",
+            )
+        )
+
+
+def align_rust_analyzer_versions() -> None:
+    """Align ra_ap dependencies and update the lockfile when versions differ.
+
+    rust-analyzer crates may be published gradually, so this selects the newest
+    version that is available for every ra_ap dependency.
+    """
+    manifest = RUST_EXTRACTOR_MANIFEST.read_text()
+    dependencies = {
+        name: version
+        for name, version in tomllib.loads(manifest)["dependencies"].items()
+        if name.startswith("ra_ap_")
+    }
+    if len(set(dependencies.values())) == 1:
+        # All the `ra_ap_` dependencies agree
+        return
+
+    version = get_rust_analyzer_version()
+
+    for name, old_version in dependencies.items():
+        manifest = manifest.replace(
+            f'{name} = "{old_version}"',
+            f'{name} = "{version}"',
+        )
+    RUST_EXTRACTOR_MANIFEST.write_text(manifest)
+    run("cargo", "update")
+
+
+def commit_all(title: str) -> None:
+    run("git", "add", "--all")
+    run("git", "commit", "--no-verify", "--allow-empty", "-m", title)
+
+
+def main() -> None:
+    status = subprocess.run(
+        ["git", "status", "--porcelain"],
+        cwd=REPO_ROOT,
+        check=True,
+        capture_output=True,
+        text=True,
+    ).stdout
+    if status:
+        raise RuntimeError("the working tree must be clean")
+
+    print_step(1, "Update dependencies")
+    old_rust_analyzer_version = get_rust_analyzer_version()
+    run("cargo", "upgrade", "--incompatible", "--pinned")
+    new_rust_analyzer_version = get_rust_analyzer_version()
+    if new_rust_analyzer_version == old_rust_analyzer_version:
+        run("git", "restore", ".")
+        print("No new rust-analyzer version available.")
+        return
+
+    align_rust_analyzer_versions()
+    commit_all("Cargo: Upgrade dependencies")
+
+    print_step(2, "Update the fixed Rust toolchain used by the extractor")
+    rust_toolchain = get_compatible_rust_toolchain(new_rust_analyzer_version)
+    update_fixed_rust_toolchain(rust_toolchain)
+    commit_all("Rust: Update fixed toolchain")
+
+    print_step(3, "Regenerate vendored bazel files")
+    run("misc/bazel/3rdparty/update_tree_sitter_extractors_deps.sh")
+    commit_all("Bazel: Regenerate vendored cargo dependencies")
+
+    print_step(4, "Run codegen")
+    run_codegen()
+    commit_all("Rust: Run codegen")
+
+    print_step(5, "Try compiling")
+    run("bazel", "run", "//rust:install")
+
+
+if __name__ == "__main__":
+    main()