databases use races with any other concurrent session on the same profile (shared, unlocked config.yml)

[divs1101-maker]

databases use writes a shared, unlocked config file — races with any other concurrent CLI session on the same profile

Hit while scripting a benchmark that creates a fresh instant database per run and queries it immediately. Sequence, run back-to-back in one shell script:

DBID=$(hotdata databases create --catalog ttfa10mbr1 --name ... -o json | jq -r .id)
hotdata databases use "$DBID"          # prints "Current database set to $DBID" — succeeds
hotdata query "SELECT * FROM ttfa10mbr1.public.trips LIMIT 1"

table 'ttfa10mbr1.public.trips' not found
Tip: 'ttfa10mbr1' isn't in the current database's scope. If it's a catalog, attach it
to query across catalogs: `hotdata databases attach ttfa10mbr1`.

databases use reported success immediately beforehand. Re-reading ~/.hotdata/config.yml right after the failure showed current_databases entries for workspaces this session never touched (other engineers' workspaces on the same shared account/profile — anoop-workspace, roh-workspace, shefeek-workspace, etc. all live under one profiles[].current_databases map, keyed by workspace id). Nothing else explains that: some other concurrent process using the same local CLI profile rewrote the config file between our use call and our query call, and our write either lost the race or got overwritten immediately after.

Root cause

current_databases is a single map in ~/.hotdata/config.yml, one entry per workspace, shared by every CLI invocation running under the same profile (same machine + same login/API key). There's no per-process or per-session isolation, and (going by the symptom) no locking around the read-modify-write when two processes call databases use around the same time — whichever write lands last wins, globally, for every concurrent session on that profile, not just the one that called it last intentionally.

This makes databases use fundamentally unsafe to rely on for any concurrent usage of the CLI under one profile — multiple terminal sessions, CI jobs, or (increasingly likely) multiple agent sessions automating the CLI at once, all on one developer's machine/account.

Workaround (what we did)

Stopped relying on databases use / the active-database pointer entirely for scripted runs. hotdata query and hotdata databases load both resolve unambiguously against -d/--database <id> (query) or --catalog <alias> (load) without needing an "active database" at all — passing those explicitly on every call sidesteps the shared, mutable state completely and was reliable across ~50 scripted runs afterward.

Suggested fix

1. Recommend --database/--catalog for scripts in the CLI docs and skill file, and call out that databases use is a convenience for interactive sessions only, not safe for concurrent/scripted use — today nothing warns about this.
2. An env var override, analogous to the existing HOTDATA_WORKSPACE lock (workspaces use already refuses to run when HOTDATA_WORKSPACE is set) — e.g. HOTDATA_DATABASE, so a script or agent session can pin its own database without touching the shared config file or racing anyone else's databases use.
3. If current_databases is meant to survive across sessions at all, the read-modify-write needs a lock (file lock / atomic rename) so two concurrent databases use calls don't interleave — right now the map itself is a single point of contention for every concurrent CLI user on a profile.

Found while running an automated benchmark (many sequential databases create + databases use + query cycles) on a machine with other concurrent Hotdata CLI sessions active under the same profile. No benchmark numbers were affected — caught and worked around before any timed run — but it's a real footgun for anyone else scripting the CLI.