From c76ba5940571aa6fdfa36214f61f3f7f1b84eaad Mon Sep 17 00:00:00 2001 From: Owen Mansel-Chan Date: Thu, 17 Sep 2026 11:36:31 +0100 Subject: [PATCH 1/3] Update docs on db upgrade and downgrade --- ...ade.md => prepare-db-upgrade-downgrade.md} | 94 ++++++++++++------- 1 file changed, 62 insertions(+), 32 deletions(-) rename docs/{prepare-db-upgrade.md => prepare-db-upgrade-downgrade.md} (59%) diff --git a/docs/prepare-db-upgrade.md b/docs/prepare-db-upgrade-downgrade.md similarity index 59% rename from docs/prepare-db-upgrade.md rename to docs/prepare-db-upgrade-downgrade.md index 58806ecebebf..59d79e482a4e 100644 --- a/docs/prepare-db-upgrade.md +++ b/docs/prepare-db-upgrade-downgrade.md @@ -13,6 +13,10 @@ This document explains how to write or generate those scripts. 2. Run `misc/scripts/prepare-db-upgrade.sh --lang `. This will generate skeleton upgrade/downgrade scripts in the appropriate directories. 3. Fill in the details in the two `upgrade.properties` files that it generated, and add any required upgrade queries. +The generated directory names are hashes of the old and new `.dbscheme` files. If the +schema changes after generating the scripts, delete the generated directories and run the +script again so that the directory names and schema snapshots use the correct hashes. + It may be helpful to look at some of the existing upgrade/downgrade scripts, to see how they work. ## Details @@ -25,7 +29,9 @@ compatibility: partial some_relation.rel: run some_relation.qlo ``` -The `description` field is a textual description of the aim of the upgrade. +The `description` field is a textual description of the aim of the step. Describe the +operation in its actual direction: for example, a downgrade that removes a newly added +table should say that it removes the table. The `compatibility` field takes one of four values: @@ -37,14 +43,36 @@ The `compatibility` field takes one of four values: * **breaking**: the step is unsafe and will prevent certain queries from working. -The `some_relation.rel` line(s) are the actions required to perform the database upgrade. Do a diff on the new vs old `.dbscheme` file to get an idea of what they have to achieve. Sometimes you won't need any upgrade commands – this happens when the dbscheme has changed in "cosmetic" ways, for example by adding/removing comments or changing union type relationships, but still retains the same on-disk format for all tables; the purpose of the upgrade script is then to document the fact that it's safe to replace the old dbscheme with the new one. +Choose compatibility independently for the upgrade and downgrade, because the two directions +may preserve different amounts of information. + +The `some_relation.rel` line(s) are the actions required to transform the database in the +direction of the step. Upgrade and downgrade directories use the same file name and command +syntax, even though a file in a downgrade directory describes a downgrade. Diff `old.dbscheme` +against the target `.dbscheme` in the generated directory to determine which actions are +needed. + +No action is needed for a relation added by the target schema if it should be empty in the +transformed database. A missing relation is treated as empty, so do not add a `.rel` line just +to create an empty file. If extraction would populate the new relation, however, the upgrade +is not `full`: it will usually be `backwards`, because queries using the new relation may have +degraded results on upgraded databases. + +A relation that exists in `old.dbscheme` but not in the target schema should normally be +deleted explicitly with `relation.rel: delete` so that the transformation does not leave +obsolete data behind. + +Sometimes no commands are needed because the schema changed only cosmetically, for example +by adding or removing comments or changing union type relationships without changing the +on-disk format. The script then documents that it is safe to replace the old schema with the +new one. Ideally, your downgrade script will perfectly revert the changes applied by the upgrade script, such that applying the upgrade and then the downgrade will result in the same database you started with. -Some typical upgrade commands look like this: +Some typical upgrade or downgrade commands look like this: ``` -// Delete a relation that has been replaced in the new scheme +// Delete a relation that does not exist in the target schema obsolete.rel: delete // Create a new version of a table by applying an expression (using a simple @@ -83,7 +111,7 @@ To test the upgrade script, run: codeql test run --search-path= --search-path= ``` -Where `` is an extractor pack containing the old extractor and dbscheme that pre-date your changes, `` is the directory containing the qltests for your language, and `` is the root directory directory of the `github/codeql` clone that contains ``. This will run the tests using an old extractor, and the test databases will all be upgraded in place using your new upgrade script. +Where `` is an extractor pack containing the old extractor and dbscheme that pre-date your changes, `` is the directory containing the qltests for your language, and `` is the root directory of the `github/codeql` clone that contains ``. This will run the tests using an old extractor, and the test databases will all be upgraded in place using your new upgrade script. To test the downgrade script, create an extractor pack that includes your new dbscheme and extractor changes. Then checkout the `main` branch of `codeql` (i.e. a branch that does not include your changes), and run: @@ -107,43 +135,45 @@ You might also choose to test with a real-world database. 5. Verify that your queries produced sensible results. -#### Doing the upgrade manually - -To create the upgrade directory manually, without using `prepare-db-upgrade.sh`: - -1. Get a hash of the old `.dbscheme` file from `main` (i.e. from just before your changes). You can do this by checking out the code prior to your changes and running `git hash-object ql/lib/.dbscheme` - -2. Go back to your branch and create an upgrade directory with that hash as its name, for example: -``` -mkdir ql/lib/upgrades/454f1e15151422355049dc4f1f0486a03baeffef -``` +#### Creating the scripts manually +To create both directions manually, without using `prepare-db-upgrade.sh`: -3. Copy the old `.dbscheme` file to that directory, using the name old.dbscheme. +1. Get the hashes of the old `.dbscheme` from `main` and the new `.dbscheme` from + your branch. For example: -``` -cp ql/lib/.dbscheme ql/lib/upgrades/454f1e15151422355049dc4f1f0486a03baeffef/old.dbscheme -``` + ```sh + old_hash=$(git show main:ql/lib/.dbscheme | git hash-object --stdin) + new_hash=$(git hash-object ql/lib/.dbscheme) + ``` -4. Put a copy of your new `.dbscheme` file in that directory and create an `upgrade.properties` file (as described above). +2. Create the upgrade directory using the old hash and the downgrade directory using the + new hash: -#### Doing the downgrade manually + ```sh + mkdir -p ql/lib/upgrades/$old_hash + mkdir -p downgrades/$new_hash + ``` -The process is similar for downgrade scripts, but there is a reversal in terminology: your **new** dbscheme will now be the one called `old.dbscheme`. +3. Populate the upgrade directory. Here, `old.dbscheme` is the schema from `main`, and + `.dbscheme` is the new target schema: -1. Get a hash of your new `.dbscheme` file, with `git hash-object ql/lib/.dbscheme` + ```sh + git show main:ql/lib/.dbscheme > ql/lib/upgrades/$old_hash/old.dbscheme + cp ql/lib/.dbscheme ql/lib/upgrades/$old_hash/.dbscheme + ``` -2. Create a downgrade directory with that hash as its name, for example: -``` -mkdir downgrades/9fdd1d40fd3c3f8f9db8fabf5a353580d14c663a -``` +4. Populate the downgrade directory in the opposite direction. For a downgrade, the new + schema is called `old.dbscheme`, because it is the schema before the downgrade step: -3. Copy your new `.dbscheme` file to that directory, using the name `old.dbscheme`. -``` -cp ql/lib/.dbscheme ql/lib/upgrades/454f1e15151422355049dc4f1f0486a03baeffef/old.dbscheme -``` + ```sh + cp ql/lib/.dbscheme downgrades/$new_hash/old.dbscheme + git show main:ql/lib/.dbscheme > downgrades/$new_hash/.dbscheme + ``` -4. Put a copy of the `.dbscheme` from `main` in that directory and create an `upgrade.properties` file that performs the downgrade (as described above). +5. Create an `upgrade.properties` file in each directory. The file in the upgrade directory + describes the forward transformation, while the file in the downgrade directory describes + the reverse transformation. ### Debugging your scripts From ee3a225ca671010a1a70e78db704f6acee5384f9 Mon Sep 17 00:00:00 2001 From: Owen Mansel-Chan Date: Thu, 17 Sep 2026 11:51:55 +0100 Subject: [PATCH 2/3] Clarify compatibility for downgrade scripts --- docs/prepare-db-upgrade-downgrade.md | 12 +++++++----- 1 file changed, 7 insertions(+), 5 deletions(-) diff --git a/docs/prepare-db-upgrade-downgrade.md b/docs/prepare-db-upgrade-downgrade.md index 59d79e482a4e..e5d1b9cdcd49 100644 --- a/docs/prepare-db-upgrade-downgrade.md +++ b/docs/prepare-db-upgrade-downgrade.md @@ -33,15 +33,17 @@ The `description` field is a textual description of the aim of the step. Describ operation in its actual direction: for example, a downgrade that removes a newly added table should say that it removes the table. -The `compatibility` field takes one of four values: +The `compatibility` field takes one of four values. In these definitions, the source schema is +`old.dbscheme`, and the target schema is the other `.dbscheme` in the script directory. Thus, +the source is the older schema for an upgrade and the newer schema for a downgrade. - * **full**: results from the upgraded snapshot will be identical to results from a snapshot built with the new version of the toolchain. + * **full**: query results from the transformed database will be identical to results from a database built with the target version of the toolchain. - * **backwards**: the step is safe and preserves the meaning of the old database, but new features may not work correctly on the upgraded snapshot. + * **backwards**: the step is safe and preserves the meaning of the source database, but features provided by the target query and library packs may not work correctly on the transformed database. - * **partial**: the step is safe and preserves the meaning of the old database, but you would get better results if you rebuilt the snapshot with the new version of the toolchain. + * **partial**: the step is safe and preserves the meaning of the source database, but rebuilding the database with the target version of the toolchain would produce better results. - * **breaking**: the step is unsafe and will prevent certain queries from working. + * **breaking**: the step is unsafe and will prevent certain target queries from working. Choose compatibility independently for the upgrade and downgrade, because the two directions may preserve different amounts of information. From 9045b897983155eb64777956490fcb07204c2428 Mon Sep 17 00:00:00 2001 From: Owen Mansel-Chan Date: Thu, 17 Sep 2026 11:52:29 +0100 Subject: [PATCH 3/3] Make paths more consistent --- docs/prepare-db-upgrade-downgrade.md | 28 ++++++++++++++++++---------- 1 file changed, 18 insertions(+), 10 deletions(-) diff --git a/docs/prepare-db-upgrade-downgrade.md b/docs/prepare-db-upgrade-downgrade.md index e5d1b9cdcd49..1ea1dcffd45d 100644 --- a/docs/prepare-db-upgrade-downgrade.md +++ b/docs/prepare-db-upgrade-downgrade.md @@ -139,38 +139,46 @@ You might also choose to test with a real-world database. #### Creating the scripts manually -To create both directions manually, without using `prepare-db-upgrade.sh`: +To create both directions manually, without using `prepare-db-upgrade.sh`, run the following +commands from the repository root. First set `lang` to the language directory and `schema_file` +to the repository-relative path of its `.dbscheme` file. For example, for Go: + + ```sh + lang=go + schema_file=go/ql/lib/go.dbscheme + ``` 1. Get the hashes of the old `.dbscheme` from `main` and the new `.dbscheme` from your branch. For example: ```sh - old_hash=$(git show main:ql/lib/.dbscheme | git hash-object --stdin) - new_hash=$(git hash-object ql/lib/.dbscheme) + old_hash=$(git show "main:$schema_file" | git hash-object --stdin) + new_hash=$(git hash-object "$schema_file") ``` 2. Create the upgrade directory using the old hash and the downgrade directory using the new hash: ```sh - mkdir -p ql/lib/upgrades/$old_hash - mkdir -p downgrades/$new_hash + upgrade_dir="$lang/ql/lib/upgrades/$old_hash" + downgrade_dir="$lang/downgrades/$new_hash" + mkdir -p "$upgrade_dir" "$downgrade_dir" ``` 3. Populate the upgrade directory. Here, `old.dbscheme` is the schema from `main`, and - `.dbscheme` is the new target schema: + the other `.dbscheme` file is the new target schema: ```sh - git show main:ql/lib/.dbscheme > ql/lib/upgrades/$old_hash/old.dbscheme - cp ql/lib/.dbscheme ql/lib/upgrades/$old_hash/.dbscheme + git show "main:$schema_file" > "$upgrade_dir/old.dbscheme" + cp "$schema_file" "$upgrade_dir/$(basename "$schema_file")" ``` 4. Populate the downgrade directory in the opposite direction. For a downgrade, the new schema is called `old.dbscheme`, because it is the schema before the downgrade step: ```sh - cp ql/lib/.dbscheme downgrades/$new_hash/old.dbscheme - git show main:ql/lib/.dbscheme > downgrades/$new_hash/.dbscheme + cp "$schema_file" "$downgrade_dir/old.dbscheme" + git show "main:$schema_file" > "$downgrade_dir/$(basename "$schema_file")" ``` 5. Create an `upgrade.properties` file in each directory. The file in the upgrade directory