fix: bootstrap PostgreSQL without a working service manager (#823)

.claude/skills/fix-issue.md

@@ -321,6 +321,7 @@ cases for these three BSON types — they fell to `default: return nil`.
 | An import/export mapping over an entity created with `EXTENDS` maps only its **own** attributes; every inherited field shows unmapped in Studio Pro, and `mx check` reports CE1613 "The selected attribute 'Mod.Child.Attr' no longer exists". An inherited Boolean/DateTime element also gets `DataType=String` | The mapping builder prefixed the entity being mapped unconditionally (`attr = parentEntity + "." + attr`), but a member reference is qualified against the entity that **declares** it — the same rule as entity access rules (#758) and the change-object writer (#451). Separately `resolveAttributeType` scanned only the entity's own attributes and fell through to its `"String"` default | `mdl/executor/cmd_import_mappings.go` and `cmd_export_mappings.go` (both carry the same two lines), `mdl/executor/entity_hierarchy.go` (`ResolveMemberRef`, `ResolveMemberType`) | Route both sites through the generalization walk added for #758: `ResolveMemberRef` returns the declaring-entity reference and `ResolveMemberType` finds the type up the chain, each falling back to the old behaviour when the member cannot be resolved. **Watch for the sibling defect**: the old `resolveAttributeType` matched entities **by name across every domain model**, so a same-named entity in another module could win — resolve the module by name instead. **Generalisable**: when one rule has several call sites, a fix at one of them proves nothing about the others; grep for the *pattern* (`range entity.Attributes`, `parentEntity + "."`) rather than the reported symptom. Repro `mdl-examples/bug-tests/703-mapping-inherited-attributes.mdl`; A/B on the same project shows `Map703.Contract.DocName` (CE1613) become `Map703.DocumentBase.DocName`. Issue #703, umbrella #765 |
 | `alter settings model JavaVersion = 'Java21'` on Mendix 11.12+ produces a project mxbuild refuses to **load**: `mx check` reports `System.ArgumentOutOfRangeException ... (Parameter 'majorVersion is an unsupported value: Java21')` at `JavaVersionExtensions.fromString`. Every check downstream of the settings unit is lost with it | Mendix renamed the property between 11.6 (`JavaVersion` = `"Java21"`) and 11.12 (`JavaMajorVersion` = `"21"`) — and the rename changed the **value format** as well as the key. The #759 fix followed only the key, writing the caller's value through verbatim, so the 11.6 spelling landed in the 11.12 key | `mdl/settingsoverlay/settingsoverlay.go` (`JavaVersionValue`, `SetJavaVersion`) — shared by both engines; the dead third copy in `modelsdk/mpr/serialize_services.go` carried it too | Render the value in the dialect the stored key expects: strip/add the `Java` prefix per key, and pass an unrecognisable value through untouched so a typo surfaces as a Mendix error instead of a mangled setting. **Generalisable**: a renamed property is not only a renamed key — check whether the value encoding moved with it, and cover *both* directions (either spelling in, document's dialect out). Note the sharper failure mode: the original #759 shape was an unknown property, which mxbuild **tolerates**, so only Studio Pro broke; a wrong *value* for a known enum is a hard build failure, which is why this one surfaced as a red nightly rather than a user report. Repro `mdl-examples/bug-tests/759-java-version-value-dialect.mdl`. Issue #759 (follow-up) |
 | On **Mendix 11.13 only**, every microflow using `EXECUTE DATABASE QUERY` fails `mx check` with **CE5277** "Please re-run and save the query to fix the error", once per activity. The queries themselves report nothing — the error lands on the *activities* pointing at them, so it reads like a microflow defect. Both engines. 11.12 and below are clean | 11.13 replaced the integer `QueryType` (1 = custom SQL) on `DatabaseConnector$DatabaseQuery` with a `Type` **string enum** (`Select` / `NonSelect` / `Unknown`), shipping a one-time conversion (`ExternalDatabaseConnectionQueryTypeConversion`) for old documents. mxcli wrote the legacy integer unconditionally, so on 11.13 the new property was simply **absent** — and an absent `Type` reads as Unknown, which is exactly what CE5277 reports | `mdl/dbconnector/querytype.go` (new, shared by both engines), `sdk/mpr/writer_dbconnection.go` + `parser_dbconnection.go`, `mdl/backend/modelsdk/db_write.go` + `integration_read.go`, `model/types.go` (`DatabaseQuery.QueryTypeName`) | Branch on the project's Mendix version (`ProjectVersion().IsAtLeast(11, 13)`) and write **exactly one** spelling. Writing both is not a safe hedge — a property the target's metamodel does not define is the #759 Studio-Pro-won't-open shape. Read side must accept either, or the next ALTER of an 11.13 project writes Unknown straight back. mxcli can't derive the type the way Studio Pro does (running the query and inspecting the result set), so it reads the leading SQL keyword — still better than Mendix's own converter, which marks every migrated query `Select` regardless of statement. **Diagnosis method**: `mx convert -p -s <old-project>` with the NEW mxbuild runs the version's own migration, then diff the BSON — that is what showed `QueryType: 1` → `Type: "Select"` without guessing. **Generalisable**: onboarding a new Mendix minor is not just adding it to the nightly matrix — run the doctype corpus against it first (`MX_BINARY=~/.mxcli/mxbuild/<ver>/modeler/mx go test -tags integration -run TestMxCheck_DoctypeScripts`), because a renamed property surfaces as a red matrix job, not a compile error. Repro `mdl-examples/bug-tests/1113-database-query-type-enum.mdl`. Sibling drift found in the same sweep and deliberately NOT fixed: **CE5278** ("The <db> JDBC driver is missing from the module settings"), a new 11.13 check about the module's Java dependencies, which mxcli has no way to author |
+| `mxcli run --ensure-db` fails to start PostgreSQL when no service manager becomes ready (e.g. Arch): `exec: "pg_ctlcluster": executable file not found in $PATH`, though `initdb`/`pg_ctl`/`psql` are present | `startLocalPostgres` only knew the `service`/`pg_ctlcluster` helpers, and role/database provisioning assumed `sudo -u postgres`, which a user-owned cluster does not need | `cmd/mxcli/docker/ensuredb.go` (`startLocalPostgres`, `startUserCluster`, `resolveSuperuser`) | When no service manager makes PostgreSQL ready, start a user-owned `initdb`/`pg_ctl` cluster under `~/.mxcli/postgres` (idempotent: reuse an initialized data dir, skip a running server). Provision as the cluster's own `postgres` superuser over a direct loopback `psql`, keeping `sudo -u postgres` for system clusters. Tests stub the external tools. Issue #823 |
 
 **Key insight:** `microflows$ListRange` stores offset/limit inside a nested
 `CustomRange` map — must cast `raw["CustomRange"].(map[string]any)` before

.claude/skills/mendix/run-local.md

@@ -50,8 +50,10 @@ association catalog only at startup; behavioural changes are hot-reloaded.
 - A **PostgreSQL** database (defaults: `127.0.0.1:5432`, user `mendix`, db derived
   from the project name; override with `--db-host/--db-name/--db-user/--db-password`).
   - **`--ensure-db`** provisions it for a fresh session: starts local Postgres if the
-    port is down and creates the role + database if missing (local superuser via
-    `sudo -u postgres`). Remote hosts are only checked, not provisioned.
+    port is down and creates the role + database if missing. It uses a service
+    manager, or a user-owned `initdb`/`pg_ctl` cluster under `~/.mxcli/postgres`
+    when no service becomes ready (e.g. Arch) — needing no `postgres` OS account or `sudo`.
+    Remote hosts are only checked, not provisioned.
   - Without `--ensure-db`, create it once and the command errors if it's unreachable:
 
     ```bash

CHANGELOG.md

@@ -6,6 +6,10 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).
 
 ## [Unreleased]
 
+### Fixed
+
+- **`mxcli run --ensure-db` can start PostgreSQL without a working service manager (#823)** — on hosts that ship neither `service` nor Debian's `pg_ctlcluster` (e.g. Arch Linux), `--ensure-db` failed with `exec: "pg_ctlcluster": executable file not found in $PATH` even though the portable `initdb`/`pg_ctl`/`psql` tools were present. `startLocalPostgres` now falls back to a user-owned cluster under `~/.mxcli/postgres` when no service becomes ready. It is idempotent (reuses an initialized data directory, leaves an already-running server alone) and needs neither a `postgres` OS account nor passwordless `sudo`: role/database provisioning connects to the cluster directly as the `postgres` superuser, keeping the existing `sudo -u postgres psql` path for system clusters.
+
 ## [0.16.0] - 2026-07-12
 
 Headline: **Pluggable chart authoring reaches round-trip fidelity**, plus in-place enum-caption editing, named layout placeholders, and a batch of new pre-build `check` heuristics. Charts gain widget-level datasource attributes, the `LINE`/`SCALECOLOR` object-list keywords, and a `DESCRIBE` that reconstructs them as executable MDL; workflows and widget-less pages now describe cleanly; view-entity OQL is validated before build; and several authoring mistakes are caught at `mxcli check` time instead of only by MxBuild.

cmd/mxcli/docker/ensuredb.go

@@ -7,23 +7,29 @@ import (
 	"io"
 	"os"
 	"os/exec"
+	"path/filepath"
 	"regexp"
 	"strings"
 	"time"
 )
 
 // ensuredb.go provisions the local PostgreSQL a standalone runtime needs, so a
 // fresh session comes up testable without a manual createdb (slice 2 of the
-// warm-loop proposal). It is best-effort and devcontainer-shaped: it starts the
-// local Postgres service if the port is down, then ensures the app role and
-// database exist via a superuser (`sudo -u postgres psql`). For a non-local DB
-// host it does nothing but verify reachability — provisioning a remote database
-// is not mxcli's business.
+// warm-loop proposal). It is best-effort: it starts the local Postgres service
+// if the port is down — via a service manager, or a user-owned initdb/pg_ctl
+// cluster when no service becomes ready (e.g. Arch, #823) — then ensures the app
+// role and database exist via a superuser (a direct loopback connection to the
+// user-owned cluster, or `sudo -u postgres psql` for a system cluster). For a
+// non-local DB host it does nothing but verify reachability — provisioning a
+// remote database is not mxcli's business.
 
 // pgIdent is a conservative PostgreSQL identifier (unquoted): a safe database or
 // role name. We refuse anything else rather than quote/escape it into DDL.
 var pgIdent = regexp.MustCompile(`^[a-z_][a-z0-9_]*$`)
 
+// serviceReadyTimeout is a variable so tests can avoid waiting 20 seconds.
+var serviceReadyTimeout = 20 * time.Second
+
 // splitHostPort splits "host:port" into host and port, defaulting the port to
 // 5432 when absent.
 func splitHostPort(hostPort string) (host, port string) {
@@ -77,7 +83,7 @@ func EnsureDatabase(db DBConfig, w io.Writer) error {
 				"start it and create the %q database (user %q)", db.Host, db.Name, db.User)
 		}
 		fmt.Fprintln(w, "  Starting local PostgreSQL...")
-		if err := startLocalPostgres(); err != nil {
+		if err := startLocalPostgres(host, port, w); err != nil {
 			return fmt.Errorf("starting local PostgreSQL: %w", err)
 		}
 		if err := waitPGReady(host, port, 20*time.Second); err != nil {
@@ -86,10 +92,14 @@ func EnsureDatabase(db DBConfig, w io.Writer) error {
 	}
 
 	// Ensure the role and database exist (needs a local superuser).
-	if err := ensureRole(db, w); err != nil {
+	su, err := resolveSuperuser(host, port)
+	if err != nil {
+		return err
+	}
+	if err := ensureRole(su, db, w); err != nil {
 		return err
 	}
-	if err := ensureDatabase(db, w); err != nil {
+	if err := ensureDatabase(su, db, w); err != nil {
 		return err
 	}
 
@@ -111,33 +121,77 @@ func canConnectDB(db DBConfig) bool {
 }
 
 // startLocalPostgres starts the local PostgreSQL service, trying the common
-// service managers in turn. Success is confirmed later by waitPGReady.
-func startLocalPostgres() error {
+// service managers in turn. When none is present — e.g. on Arch — or they do
+// not produce a ready server, it falls back to a user-owned cluster started with
+// the portable initdb/pg_ctl tools (#823).
+func startLocalPostgres(host, port string, w io.Writer) error {
 	attempts := [][]string{
 		{"service", "postgresql", "start"},
 		{"pg_ctlcluster", "--", "start"}, // placeholder; real cluster args vary
 	}
-	var lastErr error
 	for _, a := range attempts {
 		if _, err := exec.LookPath(a[0]); err != nil {
-			lastErr = err
 			continue
 		}
-		cmd := exec.Command(a[0], a[1:]...)
-		if err := cmd.Run(); err == nil {
-			return nil
-		} else {
-			lastErr = err
-		}
 		// `service postgresql start` is the reliable path in the devcontainer; if
 		// it ran (even non-zero) Postgres may still be coming up — let waitPGReady
 		// decide rather than failing here.
+		_ = exec.Command(a[0], a[1:]...).Run()
+		if waitPGReady(host, port, serviceReadyTimeout) == nil {
+			return nil
+		}
+	}
+
+	// No service manager made PostgreSQL ready: start a user-owned cluster with
+	// the portable tools. This needs neither a `postgres` OS account nor sudo.
+	return startUserCluster(host, port, w)
+}
+
+// startUserCluster initializes (once) and starts a PostgreSQL cluster owned by
+// the current user under ~/.mxcli/postgres, listening on host:port. It is safe
+// to run repeatedly: an initialized data directory is reused and an already
+// running server is left alone.
+func startUserCluster(host, port string, w io.Writer) error {
+	home, err := os.UserHomeDir()
+	if err != nil {
+		return fmt.Errorf("determining home directory: %w", err)
+	}
+	stateDir := filepath.Join(home, ".mxcli", "postgres")
+	dataDir := filepath.Join(stateDir, "data")
+	// A separate short socket directory (some systems cap the socket path length).
+	sockDir := filepath.Join(stateDir, "sock")
+	if err := os.MkdirAll(sockDir, 0o700); err != nil {
+		return fmt.Errorf("creating PostgreSQL state directory: %w", err)
+	}
+
+	// Initialize once — PG_VERSION marks a data directory initdb has populated.
+	if _, err := os.Stat(filepath.Join(dataDir, "PG_VERSION")); err != nil {
+		fmt.Fprintln(w, "  Initializing user-owned PostgreSQL cluster...")
+		// -U postgres makes the bootstrap superuser "postgres"; the trust auth
+		// keeps loopback connections password-free for the provisioning below
+		// (the server binds only the local interface).
+		init := exec.Command("initdb", "-D", dataDir, "-U", "postgres",
+			"--auth-local=trust", "--auth-host=trust", "--encoding=UTF8")
+		if out, err := init.CombinedOutput(); err != nil {
+			return fmt.Errorf("initializing PostgreSQL cluster in %s: %w\n%s",
+				dataDir, err, strings.TrimSpace(string(out)))
+		}
+	}
+
+	// Already running? pg_ctl start on a running cluster errors, so skip.
+	if exec.Command("pg_ctl", "-D", dataDir, "status").Run() == nil {
 		return nil
 	}
-	if lastErr != nil {
-		return lastErr
+
+	fmt.Fprintln(w, "  Starting user-owned PostgreSQL cluster...")
+	serverOpts := fmt.Sprintf("-h %s -p %s -k %s", host, port, sockDir)
+	start := exec.Command("pg_ctl", "-D", dataDir, "-o", serverOpts, "-w",
+		"-t", "30", "-l", filepath.Join(stateDir, "server.log"), "start")
+	if out, err := start.CombinedOutput(); err != nil {
+		return fmt.Errorf("starting PostgreSQL cluster in %s: %w\n%s",
+			dataDir, err, strings.TrimSpace(string(out)))
 	}
-	return fmt.Errorf("no known service manager found to start PostgreSQL")
+	return nil
 }
 
 // waitPGReady polls pg_isready until the server accepts connections or timeout.
@@ -169,40 +223,76 @@ func waitTCP(hostPort string, timeout time.Duration) error {
 	return fmt.Errorf("%s did not accept connections within %s", hostPort, timeout)
 }
 
-// superuserPSQL runs a psql command as the postgres superuser (sudo -u postgres).
-func superuserPSQL(args ...string) *exec.Cmd {
-	full := append([]string{"-u", "postgres", "psql", "-v", "ON_ERROR_STOP=1"}, args...)
-	return exec.Command("sudo", full...)
+// superuser is how we reach a PostgreSQL superuser to provision the role and
+// database: a direct loopback `psql -U postgres` against the user-owned cluster,
+// or the original `sudo -u postgres psql` for a system/devcontainer cluster.
+type superuser struct {
+	host, port string
+	sudo       bool
+}
+
+// psql builds a psql command for the superuser. ON_ERROR_STOP makes a failed
+// statement a non-zero exit rather than a silent success.
+func (s superuser) psql(args ...string) *exec.Cmd {
+	if s.sudo {
+		return exec.Command("sudo",
+			append([]string{"-u", "postgres", "psql", "-v", "ON_ERROR_STOP=1"}, args...)...)
+	}
+	// -w never prompts for a password, so probing an unrelated password-protected
+	// system cluster fails fast instead of hanging before we try the sudo path.
+	base := []string{"-v", "ON_ERROR_STOP=1", "-w",
+		"-h", s.host, "-p", s.port, "-U", "postgres", "-d", "postgres"}
+	return exec.Command("psql", append(base, args...)...)
+}
+
+// resolveSuperuser picks a working superuser path: a direct loopback connection
+// (the user-owned initdb cluster, and the only path that works without elevation
+// on Arch) or `sudo -u postgres` for a system cluster.
+func resolveSuperuser(host, port string) (superuser, error) {
+	if isLocalHost(host) {
+		direct := superuser{host: host, port: port}
+		if direct.psql("-tAc", "select 1").Run() == nil {
+			return direct, nil
+		}
+	}
+	if _, err := exec.LookPath("sudo"); err == nil {
+		sudo := superuser{host: host, port: port, sudo: true}
+		if sudo.psql("-tAc", "select 1").Run() == nil {
+			return sudo, nil
+		}
+	}
+	return superuser{}, fmt.Errorf("no local PostgreSQL superuser available to create the " +
+		"role/database (tried a direct 'psql -U postgres' connection and 'sudo -u postgres')")
 }
 
 // ensureRole creates the app login role if it does not already exist.
-func ensureRole(db DBConfig, w io.Writer) error {
-	check := superuserPSQL("-tAc", fmt.Sprintf("select 1 from pg_roles where rolname='%s'", db.User))
+func ensureRole(su superuser, db DBConfig, w io.Writer) error {
+	check := su.psql("-tAc", fmt.Sprintf("select 1 from pg_roles where rolname='%s'", db.User))
 	out, _ := check.Output()
 	if strings.TrimSpace(string(out)) == "1" {
 		return nil
 	}
 	fmt.Fprintf(w, "  Creating role %q...\n", db.User)
 	ddl := fmt.Sprintf("CREATE ROLE %s WITH LOGIN PASSWORD %s CREATEDB", db.User, quoteSQLString(db.Password))
-	cmd := superuserPSQL("-c", ddl)
+	cmd := su.psql("-c", ddl)
 	if out, err := cmd.CombinedOutput(); err != nil {
 		return fmt.Errorf("creating role %q: %w\n%s\n"+
-			"  (need a local postgres superuser via 'sudo -u postgres'; create the role manually if unavailable)",
+			"  (need a local postgres superuser; create the role manually if unavailable)",
 			db.User, err, strings.TrimSpace(string(out)))
 	}
 	return nil
 }
 
 // ensureDatabase creates the app database owned by the app role if it is absent.
-func ensureDatabase(db DBConfig, w io.Writer) error {
-	check := superuserPSQL("-tAc", fmt.Sprintf("select 1 from pg_database where datname='%s'", db.Name))
+func ensureDatabase(su superuser, db DBConfig, w io.Writer) error {
+	check := su.psql("-tAc", fmt.Sprintf("select 1 from pg_database where datname='%s'", db.Name))
 	out, _ := check.Output()
 	if strings.TrimSpace(string(out)) == "1" {
 		return nil
 	}
 	fmt.Fprintf(w, "  Creating database %q owned by %q...\n", db.Name, db.User)
 	ddl := fmt.Sprintf("CREATE DATABASE %s OWNER %s", db.Name, db.User)
-	cmd := superuserPSQL("-c", ddl)
+	cmd := su.psql("-c", ddl)
 	if out, err := cmd.CombinedOutput(); err != nil {
 		return fmt.Errorf("creating database %q: %w\n%s", db.Name, err, strings.TrimSpace(string(out)))
 	}