Sync from ako/mxcli: microflow canvas layout authoring (#884), marketplace cache, dev container fix

.claude/skills/mendix/download-marketplace-content.md

@@ -263,7 +263,7 @@ Administration — installed 4.3.2 (Mendix 11.12.1)
 
 **Tell the user the first one is slow.** Answering needs a reference project —
 a blank app with the published module imported — and `--to` needs two. Measured
-on Administration at 11.12.1: **~50s** the first time, **~13s** afterwards, once
+on Administration at 11.12.1: **~47s** the first time, **~9s** afterwards, once
 `~/.mxcli/marketplace-refs/` holds the blank app and the built references. Run
 `diff` before `update` rather than instead of it: the `update` reuses the base
 reference the `diff` just built, so the pair costs little more than the `diff`.

cmd/mxcli/marketplace/refcache.go

@@ -117,15 +117,19 @@ func safeKey(s string) string {
 
 // defaultRefCacheEntries bounds the finished-reference cache.
 //
-// An entry is a whole Mendix project: 34 MB measured for Administration, the
-// smallest module in a blank app. Updating six modules builds twelve references,
-// so an unbounded cache is ~400 MB of a container that may have 2 GB free — and
-// running out of disk mid-update is a far worse outcome than rebuilding a
-// reference, because `marketplace update` does not roll back.
+// An entry is the model alone (see isModelFile) — 14 MB measured for
+// Administration at 11.12.1, against 34 MB for the whole reference project.
+// Twelve is what a six-module update sweep builds, base and target each, so the
+// default holds a whole sweep at ~170 MB and the `update` that follows a `diff`
+// still hits.
 //
-// The blank-project cache is deliberately NOT bounded: it holds one entry per
-// Mendix version, and it is the one that pays off on every single build.
-const defaultRefCacheEntries = 6
+// It is bounded at all because running out of disk part way through an update is
+// a far worse outcome than rebuilding a reference: `marketplace update` does not
+// roll back, so a failed write leaves the module already dropped.
+//
+// The blank-project cache is deliberately NOT bounded: one entry per Mendix
+// version, and it is the one that pays off on every single build.
+const defaultRefCacheEntries = 12
 
 // refCacheMaxEntries reads the bound, honouring MXCLI_REF_CACHE_MAX. 0 disables
 // pruning for anyone with disk to spare.
@@ -253,12 +257,56 @@ func publishToCache(buildDir, cacheDir string) error {
 	return nil
 }
 
+// isModelFile reports whether a path inside a reference project is part of the
+// MODEL, which is the only thing a cached reference is ever read for.
+//
+// A reference project is a whole blank Mendix app, and most of it is bulk that
+// nothing here touches. Measured on Administration at 11.12.1, a 34 MB entry:
+//
+//	PackageRef.mpr      14 MB   read
+//	widgets/           9.6 MB   never read
+//	themesource/       6.4 MB   never read
+//	theme-cache/       2.1 MB   never read (compiled CSS)
+//	javascriptsource/  1.6 MB   never read
+//
+// Both consumers take the .mpr and nothing beside it: SnapshotModule opens it,
+// and PerformUpdate takes the reference's model from it while taking the
+// module's bundled widgets from the .mpk — deliberately, because the reference
+// project's widgets/ also holds the blank template's copies.
+//
+// THIS IS A CONSTRAINT ON FUTURE CHANGES. A cached reference is model-only, so
+// anything that starts reading a sibling directory of the reference .mpr will
+// see it on a cache miss and not on a hit — a difference that shows up as
+// findings that come and go. Extend this filter in the same commit, or store
+// the whole tree again.
+//
+// mprcontents/ is kept even though `mx module-import` always collapses the
+// reference to MPR v1: the cost is nothing when it is absent, and the failure if
+// that ever changes is an unreadable model rather than a slower run.
+func isModelFile(rel string) bool {
+	if rel == "mprcontents" || strings.HasPrefix(rel, "mprcontents"+string(filepath.Separator)) {
+		return true
+	}
+	return !strings.ContainsRune(rel, filepath.Separator) && strings.HasSuffix(rel, ".mpr")
+}
+
 // copyTree copies a directory tree. Used both to seed a build from the cache and
 // to hand a caller its own copy, so the cached tree is never the one written to.
 //
 // Symlinks are copied as symlinks; a Mendix project has none, and following them
 // would let a crafted package escape the destination.
 func copyTree(src, dst string) error {
+	return copyTreeFiltered(src, dst, nil)
+}
+
+// copyTreeModelOnly copies just the model, for the finished-reference cache.
+// The blank-project cache deliberately does NOT use this: a blank app is the
+// input to `mx module-import`, which reads the whole tree.
+func copyTreeModelOnly(src, dst string) error {
+	return copyTreeFiltered(src, dst, isModelFile)
+}
+
+func copyTreeFiltered(src, dst string, keep func(rel string) bool) error {
 	return filepath.Walk(src, func(path string, info os.FileInfo, err error) error {
 		if err != nil {
 			return err
@@ -274,6 +322,12 @@ func copyTree(src, dst string) error {
 		if rel == completeMarker {
 			return nil
 		}
+		if keep != nil && !keep(rel) {
+			if info.IsDir() {
+				return filepath.SkipDir
+			}
+			return nil
+		}
 		target := filepath.Join(dst, rel)
 
 		switch {

cmd/mxcli/marketplace/refcache_test.go

@@ -273,3 +273,47 @@ func TestPruneDisabledByZero(t *testing.T) {
 		t.Errorf("a malformed bound gave %d, want the default %d", got, defaultRefCacheEntries)
 	}
 }
+
+// TestModelOnlyCopyKeepsWhatIsRead pins the filter that makes a reference entry
+// ~14 MB instead of ~34 MB. The .mpr and mprcontents/ are the model; everything
+// else in a reference project is bulk nothing reads.
+//
+// The negative half matters as much as the positive: if widgets/ started being
+// copied again the cache would quietly double in size, and if the .mpr stopped
+// being copied every hit would serve an unreadable reference.
+func TestModelOnlyCopyKeepsWhatIsRead(t *testing.T) {
+	src, dst := t.TempDir(), filepath.Join(t.TempDir(), "out")
+
+	mustWrite(t, filepath.Join(src, "PackageRef.mpr"), "model")
+	mustWrite(t, filepath.Join(src, "mprcontents", "nested", "unit.mxunit"), "unit")
+	mustWrite(t, filepath.Join(src, "widgets", "big.mpk"), "bulk")
+	mustWrite(t, filepath.Join(src, "themesource", "atlas", "style.scss"), "bulk")
+	mustWrite(t, filepath.Join(src, "theme-cache", "compiled.css"), "bulk")
+	mustWrite(t, filepath.Join(src, "javascriptsource", "a.js"), "bulk")
+
+	if err := copyTreeModelOnly(src, dst); err != nil {
+		t.Fatalf("copyTreeModelOnly: %v", err)
+	}
+
+	if got := readFile(t, filepath.Join(dst, "PackageRef.mpr")); got != "model" {
+		t.Errorf(".mpr = %q, want %q — a served entry would be unreadable", got, "model")
+	}
+	if got := readFile(t, filepath.Join(dst, "mprcontents", "nested", "unit.mxunit")); got != "unit" {
+		t.Errorf("mprcontents unit = %q, want %q", got, "unit")
+	}
+	for _, bulk := range []string{"widgets", "themesource", "theme-cache", "javascriptsource"} {
+		if _, err := os.Stat(filepath.Join(dst, bulk)); !os.IsNotExist(err) {
+			t.Errorf("%s/ was copied into the cache entry; nothing reads it", bulk)
+		}
+	}
+
+	// And the unfiltered copy still takes everything — it is what seeds a blank
+	// project, which mx module-import reads in full.
+	full := filepath.Join(t.TempDir(), "full")
+	if err := copyTree(src, full); err != nil {
+		t.Fatalf("copyTree: %v", err)
+	}
+	if _, err := os.Stat(filepath.Join(full, "widgets", "big.mpk")); err != nil {
+		t.Error("the unfiltered copy dropped widgets/; a blank project needs its whole tree")
+	}
+}

cmd/mxcli/marketplace/scratch.go

@@ -355,7 +355,9 @@ func CacheReference(versionID, mendixVersion, refDir, mpkPath string) {
 	if err != nil {
 		return
 	}
-	if err := copyTree(refDir, filepath.Join(staging, entryProject)); err != nil {
+	// Model only: see isModelFile for what is dropped, why nothing reads it, and
+	// what that constrains.
+	if err := copyTreeModelOnly(refDir, filepath.Join(staging, entryProject)); err != nil {
 		_ = os.RemoveAll(staging)
 		return
 	}

cmd/mxcli/syntax/features_microflow.go

@@ -180,6 +180,31 @@ func init() {
 		SeeAlso: []string{"microflow.error-handling"},
 	})
 
+	Register(SyntaxFeature{
+		Path:    "microflow.layout",
+		Summary: "Canvas layout annotations — @position, @anchor, @curve, @caption, @color",
+		Keywords: []string{
+			"position", "anchor", "curve", "layout", "canvas",
+			"annotation", "caption", "color", "excluded", "bezier",
+		},
+		Syntax: "@position(x, y)                       -- the activity's centre point\n" +
+			"@anchor(from: right, to: left)        -- which SIDE each end of the outgoing flow attaches to\n" +
+			"@curve(from: (40, -90), to: (-40, 90))  -- the flow's bezier control vectors\n" +
+			"@merge(x, y)                          -- the implicit merge that closes a split\n" +
+			"@caption 'text'\n@color Green\n@annotation 'a note'\n@excluded\n\n" +
+			"An unrecognised @name is an error (MDL059): it would parse and do nothing,\n" +
+			"so a typo of @position would silently discard the layout.\n\n" +
+			"Mendix stores no waypoints — a flow's shape is two control vectors, each a\n" +
+			"pixel offset from its end of the line. (0, 0) at both ends is straight.\n" +
+			"@position on a split belongs to the SPLIT, so its end-if join has its own\n" +
+			"annotation. Container Size is still computed, not authorable.",
+		Example: "create microflow MyModule.ACT_Flow ($In: String)\nreturns String as $Out\nbegin\n" +
+			"  @position(200, 100)\n  @anchor(from: bottom, to: top)\n" +
+			"  @curve(from: (40, -90), to: (-40, 90))\n  declare $Tmp String = $In;\n" +
+			"  @position(200, 300)\n  declare $Out String = $Tmp;\n  return $Out;\nend;",
+		SeeAlso: []string{"microflow", "microflow.create"},
+	})
+
 	Register(SyntaxFeature{
 		Path:    "microflow.mapping",
 		Summary: "IMPORT FROM MAPPING / EXPORT TO MAPPING, and the import Range (All/First/Custom)",

cmd/mxcli/tool_templates.go

@@ -352,7 +352,14 @@ RUN apt-get update && apt-get install -y --no-install-recommends \
 
 	return `FROM mcr.microsoft.com/devcontainers/base:bookworm
 
-# Install Adoptium JDK 21 (required by MxBuild), Node.js 22, and utility tools
+# Install Adoptium JDK 21 (required by MxBuild), Node.js 22, and utility tools.
+#
+# The postgresql SERVER (not just -client) is required: the standalone runtime
+# 'mxcli run --local' boots needs a real database, and 'mxcli run --local
+# --ensure-db' provisions it in-container by starting the local service and
+# creating the role + database through a 'sudo -u postgres' superuser. With only
+# postgresql-client installed there is no service to start and no superuser, so
+# --ensure-db fails on a fresh container.
 RUN apt-get update && apt-get install -y --no-install-recommends wget apt-transport-https gpg ca-certificates curl && \
     wget -qO - https://packages.adoptium.net/artifactory/api/gpg/key/public | gpg --dearmor -o /etc/apt/keyrings/adoptium.gpg && \
     echo "deb [signed-by=/etc/apt/keyrings/adoptium.gpg] https://packages.adoptium.net/artifactory/deb bookworm main" > /etc/apt/sources.list.d/adoptium.list && \
@@ -361,6 +368,7 @@ RUN apt-get update && apt-get install -y --no-install-recommends wget apt-transp
     apt-get install -y --no-install-recommends \
        temurin-21-jdk \
        nodejs \
+       postgresql \
        postgresql-client \
        kafkacat \
     && apt-get clean \

cmd/mxcli/tool_templates_test.go

@@ -69,6 +69,23 @@ func TestGenerateDockerfile_Podman(t *testing.T) {
 	}
 }
 
+// TestGenerateDockerfile_PostgresServer guards the database prerequisite of
+// `mxcli run --local`: --ensure-db starts a local PostgreSQL service and creates
+// the role + database via a `sudo -u postgres` superuser, neither of which exists
+// when only postgresql-client is installed. The client alone is easy to mistake
+// for enough, so assert the server package specifically.
+func TestGenerateDockerfile_PostgresServer(t *testing.T) {
+	for _, runtime := range []string{"docker", "podman"} {
+		df := generateDockerfile("MyApp", "App.mpr", runtime)
+		if !strings.Contains(df, "postgresql \\") {
+			t.Errorf("%s Dockerfile must install the postgresql SERVER package, not just postgresql-client — 'run --local --ensure-db' cannot provision a database without it", runtime)
+		}
+		if !strings.Contains(df, "postgresql-client") {
+			t.Errorf("%s Dockerfile should keep postgresql-client (psql is used to probe and provision)", runtime)
+		}
+	}
+}
+
 // TestGenerateDockerfile_PlaywrightArm64 guards the arm64 Playwright provisioning
 // fix: browsers must be installed via @playwright/cli's bundled playwright-core
 // (not a transient "npx playwright"), into a world-readable shared cache, with a

docs-site/src/guides/marketplace.md

@@ -206,31 +206,40 @@ the version being moved to.
 
 Two caches under `~/.mxcli/marketplace-refs/` keep that off the clock:
 
-| Cache | Keyed by | Saves |
-|---|---|---|
-| `blank/` | Mendix version | the `mx create-project` in **every** reference build |
-| `ref/` | published version UUID + Mendix version | the whole reference, on a repeat |
+| Cache | Keyed by | Holds | Saves |
+|---|---|---|---|
+| `blank/` | Mendix version | a whole blank app (~37 MB) | the `mx create-project` in **every** reference build |
+| `ref/` | published version UUID + Mendix version | the model only (~14 MB) | the whole reference, on a repeat |
 
 Measured on Administration (content 23513, 4.3.2 → 4.5.0, Mendix 11.12.1):
 
 ```text
 mxcli marketplace diff 23513 -p app.mpr --to 4.5.0
 
   no cache      66s
-  cold cache    49s     (blank app built once, reused by the second reference)
-  warm          13s
+  cold cache    47s     (blank app built once, reused by the second reference)
+  warm           9s
 ```
 
+A `ref/` entry stores the `.mpr` (and `mprcontents/`) and nothing else. A
+reference project is a whole blank Mendix app, but only its model is ever read —
+`widgets/`, `themesource/`, `theme-cache/` and `javascriptsource/` account for
+20 MB of 34 MB and are never opened, because an update takes the module's
+bundled files from the `.mpk` rather than from the reference. Storing the model
+alone made entries 58% smaller and warm runs faster, since less is copied.
+
 The Mendix version is part of both keys, because a reference built at a
 different version reports Mendix's own conversions as local edits. An entry is
 only used once its completion marker is present and the project's version stamp
 has been re-checked on the way out, so a half-written or mislabelled entry is
 rebuilt rather than trusted.
 
-A reference is about 34 MB, so `ref/` keeps the 6 most recently used entries and
-evicts the rest — running out of disk part way through an update is worse than
-rebuilding one, because `update` does not roll back. `blank/` is not bounded: it
-holds one entry per Mendix version.
+`ref/` keeps the 12 most recently used entries and evicts the rest — twelve is
+what a six-module update sweep builds, so a whole sweep stays cached at ~170 MB.
+It is bounded at all because running out of disk part way through an update is
+worse than rebuilding an entry: `update` does not roll back, so a failed write
+leaves the module already dropped. `blank/` is not bounded — one entry per
+Mendix version.
 
 ```bash
 MXCLI_REF_CACHE_MAX=20 mxcli marketplace diff …   # keep more (0 = keep everything)