Add folder/wiki-metadata catalog operations, one-tag-per-branch releases, and a floating alias tag

.github/workflows/release.yml

@@ -121,3 +121,23 @@ jobs:
           files: dist/*
           generate_release_notes: true
           prerelease: ${{ github.ref_name == 'staging' }}
+
+      # A floating alias tag literally named after the branch (main/staging)
+      # that always points at whatever was just released — unlike the
+      # v{version}[-staging] tag above (deleted/recreated, so it names one
+      # specific release), this one just moves forward every time. Runs only
+      # once the real release above has succeeded.
+      #
+      # Sharing its name with the branch is deliberate: `git` resolves the
+      # ambiguity deterministically (a tag always wins over a same-named
+      # branch — see gitrevisions(7) on ref disambiguation order), so
+      # `pip install ...@main` / `...@staging` always resolves to this tag,
+      # i.e. the latest release — not necessarily the exact branch tip.
+      # Expect (and ignore) a "refname 'main' is ambiguous" warning from
+      # git/pip when that happens. The push uses fully-qualified refs on
+      # both sides specifically to sidestep that same ambiguity for the
+      # push command itself.
+      - name: Move the branch-name alias tag to this release
+        run: |
+          git tag -f "${{ github.ref_name }}"
+          git push origin "refs/tags/${{ github.ref_name }}:refs/tags/${{ github.ref_name }}" --force

README.md

@@ -24,15 +24,23 @@ Two main class domains for the EEA data lakehouse:
 
 These badges are live — each one queries the GitHub API directly and always shows whatever tag
 is *currently* released for that branch, updating on its own every time `main`/`staging` cuts a
-new release (see [Releasing a new version](#releasing-a-new-version) — only one tag exists per
-branch at a time, so pin to whatever the badge shows *now*, not a number copied from here).
+new release (see [Releasing a new version](#releasing-a-new-version)).
+
+`@main`/`@staging` always installs whatever was most recently released for that branch — a
+floating tag sharing the branch's own name, moved forward to the latest release automatically
+each time one is cut, so there's nothing to look up or keep in sync yourself:
 
 ```bash
-# main's latest release (stable) — pin to the tag the "main" badge above shows
-pip install "git+https://github.com/eeadata/EEALakeHouse.python.git@v0.1.6"
+pip install "git+https://github.com/eeadata/EEALakeHouse.python.git@main"       # latest stable
+pip install "git+https://github.com/eeadata/EEALakeHouse.python.git@staging"    # latest early access
+```
 
-# staging's latest release (early access) — pin to the tag the "staging" badge above shows
-pip install "git+https://github.com/eeadata/EEALakeHouse.python.git@v0.1.6-staging"
+To pin to one *specific* release instead (e.g. for a reproducible lockfile), use the exact tag
+the badge above shows — but only one tag is ever kept per branch (see below), so an old pin will
+eventually stop resolving once a newer release replaces it:
+
+```bash
+pip install "git+https://github.com/eeadata/EEALakeHouse.python.git@v0.1.6"
 ```
 
 ## Usage
@@ -123,10 +131,32 @@ exact same arguments.
   count, without fetching any actual rows.
 
 **Wiki & tags** (Dremio's catalog collaboration API — REST-only, no SQL/Flight equivalent):
-- `getwikifrom(path)` / `assignwikito(path, text)` — read, or create/overwrite, the wiki text on
-  a catalog entity.
-- `gettagsfrom(path)` / `assigntagsto(path, tags)` / `deletetags(path, tags)` — read the full tag
-  list; replace it wholesale; or remove just the given tags, leaving the rest untouched.
+- `getwikifrom(path)` / `setwikito(path, text, tags=None)` / `deletewiki(path)` — read,
+  create/overwrite, or clear the wiki text on a catalog entity. `deletewiki` is idempotent — a
+  missing `path`, or one with no wiki at all, is a no-op, not an error (Dremio's collaboration
+  API has no separate delete-wiki endpoint, so this clears the text to empty). `tags`, if given
+  to `setwikito`, is a list of `{"tag_name", "tag_value", "tag_title"}` dicts rendered into a
+  `# Meta Data` section appended to `text` (both a human-readable `title : value` line per tag
+  and the same data as `<meta><tag name=... value=... title=.../>...</meta>`) — Dremio's wiki is
+  plain markdown with no structured-metadata concept of its own, so this is embedded directly in
+  the text.
+- `setmeta2wiki(path, tags=None, overwrite=True)` / `getmetafromwiki(path, tag_name=None,
+  field=None)` — **folders only** (raises `CatalogOperationError` on a table/view — those have
+  Dremio's own tags/labels for this instead). `setmeta2wiki` updates just the `# Meta Data`
+  section of the wiki already at `path`, keeping whatever text comes before it untouched:
+  `overwrite=True` (the default) replaces the whole section with one built fresh from `tags`;
+  `overwrite=False` merges `tags` into whatever tags are already there (parsed back out of the
+  existing `<meta>` block), appended after them, with no deduplication. If `path` has no wiki
+  yet, starts from empty base text rather than raising. `getmetafromwiki` reads it back:
+  without `tag_name` (or if it doesn't match one there), returns every tag as a list; with a
+  matching `tag_name`, returns a single `{"tag_name", ...}` dict instead — both `tag_value` and
+  `tag_title` if `field` isn't given, or just the one `field` (`"tag_value"`/`"tag_title"`) asks
+  for.
+- `gettagsfrom(path)` / `settagsto(path, tags)` / `deletetags(path, tags)` — **tables/views
+  only** (the mirror image of `setmeta2wiki`/`getmetafromwiki` — raises
+  `CatalogOperationError` on a folder, which has no Dremio tags/labels concept of its own). Read
+  the full tag list; replace it wholesale; or remove just the given tags, leaving the rest
+  untouched.
 
 **Folders** (REST-only, idempotent — an already-there/already-gone folder is not an error):
 - `createfolder(path, create_parents=False)` — `create_parents=False` (the default) raises if
@@ -185,23 +215,34 @@ previous release *and* tag first, so **tags aren't permanent** — pin to whatev
 [Install](#install) badge shows *now*, not to an old tag number, since it won't exist once a
 newer release replaces it.
 
+**A separate floating tag literally named `main`/`staging`** always points at that branch's
+latest release — the workflow force-moves it (`git tag -f`, force-push) once the real release
+above succeeds. It deliberately shares its name with the branch: git resolves the ambiguity
+deterministically (a tag always wins over a same-named branch), so `@main`/`@staging` in an
+install command means "latest release," not "current branch tip" — expect (and ignore) a
+"refname is ambiguous" warning from git/pip when that happens.
+
 The workflow also rewrites this README's `pip`/`%pip install ...@vX.Y.Z[-staging]` example lines
 to the version it just released, committing that change back to the branch (`[skip ci]`, so it
 doesn't re-trigger itself) — so the examples above never go stale, without anyone having to
 remember to update them by hand.
 
 ## Install in JupyterLab
 
-Run this in a notebook cell (see the live badges under [Install](#install) for the current
-`main`/`staging` release tags — the lines below are kept in sync with them automatically, see
+Run this in a notebook cell — `@main`/`@staging` always resolves to whatever was most recently
+released for that branch (see [Install](#install) above):
+
+```python
+%pip install "git+https://github.com/eeadata/EEALakeHouse.python.git@main"       # latest stable
+%pip install "git+https://github.com/eeadata/EEALakeHouse.python.git@staging"    # latest early access
+```
+
+To pin to one specific release instead, use the exact tag the live badges under
+[Install](#install) show (kept in sync automatically, see
 [Releasing a new version](#releasing-a-new-version)):
 
 ```python
-# main's latest release (stable) — recommended
 %pip install "git+https://github.com/eeadata/EEALakeHouse.python.git@v0.1.6"
-
-# staging's latest release (early access)
-%pip install "git+https://github.com/eeadata/EEALakeHouse.python.git@v0.1.6-staging"
 ```
 
 Use the `%pip` magic rather than `!pip` — it installs into the kernel the

debugger/debug_run.py

@@ -255,27 +255,86 @@ def run_gettagsfrom() -> None:
     print(f"  tags       {tags}")
 
 
-def run_assignwikito() -> None:
+def run_setwikito() -> None:
     catalog = Catalog(DREMIO_BASE_URL, DREMIO_TOKEN, username=DREMIO_USERNAME)
     if DRY_RUN:
         print("DRY RUN — not setting the wiki. Set DRY_RUN = False to run this for real.")
         print(f"  view      {VIEW_PATH}")
         return
-    wiki_text = "# Bathing water assessments\n\nDebug-set wiki text for testing assignwikito."
+    wiki_text = "# Bathing water assessments\n\nDebug-set wiki text for testing setwikito."
     #wiki_text ="blabla"
-    catalog.assignwikito(VIEW_PATH, wiki_text, idempotency_key=f"{TABLE2VIEW_IDEMPOTENCY_KEY}-set-wiki")
-    print(f"assignwikito  wiki set on {VIEW_PATH}")
+    meta_tags = [
+        {"tag_name": "owner", "tag_value": "bwd-team", "tag_title": "Owner"},
+        {"tag_name": "status", "tag_value": "debug", "tag_title": "Status"},
+    ]
+    catalog.setwikito(
+        VIEW_PATH,
+        wiki_text,
+        tags=meta_tags,
+        idempotency_key=f"{TABLE2VIEW_IDEMPOTENCY_KEY}-set-wiki",
+    )
+    print(f"setwikito  wiki set on {VIEW_PATH} (with {len(meta_tags)} meta tags)")
+
+
+def run_deletewiki() -> None:
+    catalog = Catalog(DREMIO_BASE_URL, DREMIO_TOKEN, username=DREMIO_USERNAME)
+    if DRY_RUN:
+        print("DRY RUN — not deleting the wiki. Set DRY_RUN = False to run this for real.")
+        print(f"  view      {VIEW_PATH}")
+        return
+    catalog.deletewiki(VIEW_PATH, idempotency_key=f"{TABLE2VIEW_IDEMPOTENCY_KEY}-delete-wiki")
+    print(f"deletewiki  wiki cleared on {VIEW_PATH} (if it had one)")
 
 
-def run_assigntagsto() -> None:
+def run_setmeta2wiki() -> None:
+    catalog = Catalog(DREMIO_BASE_URL, DREMIO_TOKEN, username=DREMIO_USERNAME)
+    # Folders only — tables/views (like VIEW_PATH used elsewhere) have
+    # Dremio's own tags/labels for this instead and would raise here.
+    folder_path = "catalog.water_management_resources.bathing_water.bwd.draft.altia_test"
+    if DRY_RUN:
+        print("DRY RUN — not updating wiki metadata. Set DRY_RUN = False to run this for real.")
+        print(f"  folder    {folder_path}")
+        return
+    new_tags = [
+                {"tag_name": "test111", "tag_value": "oskar_value11", "tag_title": "oskar_title11"}
+                ]
+    catalog.setmeta2wiki(
+        folder_path,
+        tags=new_tags,
+        overwrite=True,  # merge with whatever tags are already there
+        idempotency_key=f"{TABLE2VIEW_IDEMPOTENCY_KEY}-write-meta",
+    )
+    print(f"setmeta2wiki  meta updated on {folder_path}")
+
+
+def run_getmetafromwiki() -> None:
+    catalog = Catalog(DREMIO_BASE_URL, DREMIO_TOKEN, username=DREMIO_USERNAME)
+    folder_path = "catalog.water_management_resources.bathing_water.bwd.draft.altia_test"
+    # run_setmeta2wiki() must have run at least once first, so there's a
+    # "# Meta Data" section here to read back.
+    all_tags = catalog.getmetafromwiki(
+        folder_path, idempotency_key=f"{TABLE2VIEW_IDEMPOTENCY_KEY}-read-meta-all"
+    )
+    print(f"getmetafromwiki  all tags on {folder_path}: {all_tags}")
+
+    one_tag = catalog.getmetafromwiki(
+        folder_path,
+        "reviewed_by",
+        "tag_value",
+        idempotency_key=f"{TABLE2VIEW_IDEMPOTENCY_KEY}-read-meta-one",
+    )
+    print(f"getmetafromwiki  reviewed_by's tag_value: {one_tag}")
+
+
+def run_settagsto() -> None:
     catalog = Catalog(DREMIO_BASE_URL, DREMIO_TOKEN, username=DREMIO_USERNAME)
     if DRY_RUN:
         print("DRY RUN — not setting tags. Set DRY_RUN = False to run this for real.")
         print(f"  view      {VIEW_PATH}")
         return
     tags = ["bathing-water", "debug"]
-    catalog.assigntagsto(VIEW_PATH, tags, idempotency_key=f"{TABLE2VIEW_IDEMPOTENCY_KEY}-set-tags")
-    print(f"assigntagsto  tags set on {VIEW_PATH}: {tags}")
+    catalog.settagsto(VIEW_PATH, tags, idempotency_key=f"{TABLE2VIEW_IDEMPOTENCY_KEY}-set-tags")
+    print(f"settagsto  tags set on {VIEW_PATH}: {tags}")
 
 
 def run_deletetags() -> None:
@@ -292,7 +351,7 @@ def run_deletetags() -> None:
 
 def run_createfolder() -> None:
     catalog = Catalog(DREMIO_BASE_URL, DREMIO_TOKEN, username=DREMIO_USERNAME)
-    folder_path = "catalog.water_management_resources.bathing_water.bwd.draft.altia_test.new_folder.level1"
+    folder_path = "catalog.water_management_resources.bathing_water.bwd.draft.altia_test"
     if DRY_RUN:
         print("DRY RUN — not creating a folder. Set DRY_RUN = False to run this for real.")
         print(f"  folder    {folder_path}")
@@ -584,8 +643,11 @@ def create_pat(host: str, access_token: str, username: str, label: str,
     #run_gettablesfrom()
     #run_gettableitemsfrom()
 
-    #run_assignwikito()
-    #run_assigntagsto()
+    #run_setwikito()
+    #run_deletewiki()
+    run_setmeta2wiki()
+    #run_getmetafromwiki()
+    #run_settagsto()
     #run_deletetags()
 
     #run_getwikifrom()
@@ -599,9 +661,7 @@ def create_pat(host: str, access_token: str, username: str, label: str,
 
 
 
-
-
-
+    """
     ap = argparse.ArgumentParser(description=__doc__,
                                  formatter_class=argparse.RawDescriptionHelpFormatter)
     ap.add_argument("--mode", choices=["azcli", "device", "sp"], default="device")
@@ -648,4 +708,4 @@ def create_pat(host: str, access_token: str, username: str, label: str,
         print(pat.get("token") or json.dumps(pat))
 
     print ("OK")
-
+    """    

src/eea_datalakehouse/catalog/__init__.py

@@ -5,10 +5,11 @@
     catalog.draft2version("bwd.draft.bw", "bwd.versions.v1", idempotency_key="bwd-v2025_1")
 
 table2view, draft2version, publishversion, datacopy, datamove, deleteview,
-gettablesfrom, gettableitemsfrom, getwikifrom, gettagsfrom, assignwikito,
-assigntagsto, deletetags, createfolder, deletefolder — over REST (the
-default) or Arrow Flight (opt in via EEA_CATALOG_TRANSPORT=flight in .env),
-with retry-later error handling for a Dremio engine that's still starting up.
+gettablesfrom, gettableitemsfrom, getwikifrom, gettagsfrom, setwikito,
+deletewiki, setmeta2wiki, getmetafromwiki, settagsto, deletetags,
+createfolder, deletefolder — over REST (the default) or Arrow Flight (opt
+in via EEA_CATALOG_TRANSPORT=flight in .env), with retry-later error
+handling for a Dremio engine that's still starting up.
 
 The module-level functions in `operations` (table2view(executor, ...) etc.)
 are what `Catalog`'s methods delegate to — call them directly if you'd
@@ -21,21 +22,24 @@
 from .errors import CatalogOperationError, EngineStartingError
 from .operations import (
     TableInfo,
-    assigntagsto,
-    assignwikito,
     createfolder,
     datacopy,
     datamove,
     deletefolder,
     deletetags,
     deleteview,
+    deletewiki,
     draft2version,
     gettableitemsfrom,
     gettablesfrom,
     gettagsfrom,
     getwikifrom,
     publishversion,
+    getmetafromwiki,
     retry_pending,
+    setmeta2wiki,
+    settagsto,
+    setwikito,
     table2view,
 )
 from .rest import CatalogRestClient
@@ -61,21 +65,24 @@
     "SqlExecutor",
     "SqlResult",
     "TableInfo",
-    "assigntagsto",
-    "assignwikito",
     "createfolder",
     "datacopy",
     "datamove",
     "deletefolder",
     "deletetags",
     "deleteview",
+    "deletewiki",
     "draft2version",
     "gettableitemsfrom",
     "gettablesfrom",
     "gettagsfrom",
     "getwikifrom",
     "publishversion",
+    "getmetafromwiki",
     "resolve_executor",
     "retry_pending",
+    "setmeta2wiki",
+    "settagsto",
+    "setwikito",
     "table2view",
 ]

src/eea_datalakehouse/catalog/client.py

@@ -263,13 +263,47 @@ def getwikifrom(self, path: str, *, idempotency_key: str) -> str:
     def gettagsfrom(self, path: str, *, idempotency_key: str) -> list[str]:
         return operations.gettagsfrom(self._catalog_rest, path, idempotency_key=idempotency_key)
 
-    def assignwikito(self, path: str, text: str, *, idempotency_key: str) -> None:
-        return operations.assignwikito(
-            self._catalog_rest, path, text, idempotency_key=idempotency_key
+    def setwikito(
+        self,
+        path: str,
+        text: str,
+        *,
+        tags: list[dict[str, str]] | None = None,
+        idempotency_key: str,
+    ) -> None:
+        return operations.setwikito(
+            self._catalog_rest, path, text, tags=tags, idempotency_key=idempotency_key
+        )
+
+    def deletewiki(self, path: str, *, idempotency_key: str) -> None:
+        return operations.deletewiki(self._catalog_rest, path, idempotency_key=idempotency_key)
+
+    def setmeta2wiki(
+        self,
+        path: str,
+        *,
+        tags: list[dict[str, str]] | None = None,
+        overwrite: bool = True,
+        idempotency_key: str,
+    ) -> None:
+        return operations.setmeta2wiki(
+            self._catalog_rest, path, tags=tags, overwrite=overwrite, idempotency_key=idempotency_key
+        )
+
+    def getmetafromwiki(
+        self,
+        path: str,
+        tag_name: str | None = None,
+        field: Literal["tag_value", "tag_title"] | None = None,
+        *,
+        idempotency_key: str,
+    ) -> list[dict[str, str]] | dict[str, str]:
+        return operations.getmetafromwiki(
+            self._catalog_rest, path, tag_name, field, idempotency_key=idempotency_key
         )
 
-    def assigntagsto(self, path: str, tags: list[str], *, idempotency_key: str) -> None:
-        return operations.assigntagsto(
+    def settagsto(self, path: str, tags: list[str], *, idempotency_key: str) -> None:
+        return operations.settagsto(
             self._catalog_rest, path, tags, idempotency_key=idempotency_key
         )