MOB-183: mob.deploy verifies release bootfile on Android, not just ERTS

decisions/2026-09-11-android-runtime-check-covers-bootfile.md

@@ -0,0 +1,72 @@
+# mob.deploy Android runtime check verifies release bootfile, not just ERTS
+
+- Date: 2026-09-11
+- Status: accepted
+- Linear: MOB-183
+- GitHub: mob_dev#54
+
+## Context
+
+`mix mob.deploy` (non-native, Android) already had an `ensure_erts_on_device/2`
+guard that ran `ls .../otp/erts-*/bin/erl_child_setup` on the target and
+refused a push when the file was missing. That covered the "no OTP at all"
+case a caller hits when they've never run `mix mob.deploy --native` on that
+device.
+
+It did *not* cover a partial-runtime state, which the reporter of mob_dev#54
+hit on a physical Android 11: the device had `erts-14.2.5/bin/` populated
+(the APK ships those binaries as `.so` in `jniLibs/`), but the release
+directory under `.../otp/releases/*/` was empty. `ls erts-*/bin/erl_child_setup`
+returned the file, so the check passed. `mob.deploy` reported "Deployed
+to 1 device(s) / Apps restarted" and the app crash-dumped at boot with:
+
+    {'cannot get bootfile', .../otp/releases/29/start_clean.boot}
+
+That's the failure family the issue calls out — "green deploy that yields
+a boot-crashing app costs a full diagnosis cycle every time." The check
+was too narrow to see the release-dir-missing subset. Third sighting of
+this class of bug (sim variant tracked as `app-ii3`).
+
+## Decision
+
+Widen the Android runtime probe to verify **both** files in one `ls`
+round-trip:
+
+- `.../otp/erts-*/bin/erl_child_setup` — the ERTS binary (existing check).
+- `.../otp/releases/*/start_clean.boot` — the OTP release the emulator
+  boots from.
+
+Classification is split out to `MobDev.Deployer.classify_android_runtime_ls/3`
+(pure, tested) so the ordering (`run_as_unavailable` → `erts_missing` →
+`bootfile_missing`) is asserted against real fixtures rather than mental
+model. `erts_missing` wins over `bootfile_missing` in the both-missing case:
+provisioning ERTS is what `--native` does first, and pointing a reader at
+the bootfile they don't have yet would send them chasing the wrong file.
+
+The check now recognises three `ls`-missing message variants seen in the
+wild:
+
+- `ls: <glob>: No such file or directory` (Toybox, most Android userland)
+- `<glob>: not found` (older Toybox / minimal shells)
+- `ls: cannot access '<glob>': No such file or directory` (GNU coreutils)
+
+Anchoring on the exact glob rather than substring "No such file" is what
+lets the classifier distinguish which of the two globs is the one that
+failed to expand.
+
+## Consequences
+
+- Non-native `mix mob.deploy` now fails with a targeted message and exit
+  code non-zero when a device is in the release-dir-missing state, instead
+  of a green deploy followed by a boot crash the user has to diagnose from
+  logcat.
+- The check remains best-effort: if `run_adb` itself errors (adb offline,
+  device unauthorised), the deploy proceeds and the current downstream
+  errors take over. Fixing that fallback is a separate concern.
+- iOS is not covered by this change. The sim variant (`app-ii3`) has the
+  same failure family but the runtime path is different (`sim_runtime_dir`
+  under `~/.mob/runtime/ios-sim/<app>/` on the host, not `/data/data/...`
+  on the device). Follow-up ticket.
+- The classifier is public so future adb output shapes can be regressed
+  against without needing a device — pattern already established in this
+  repo for `parse_devices_output/1`, `parse_simctl_json/1`, etc.

lib/mob_dev/deployer.ex

@@ -282,41 +282,104 @@ defmodule MobDev.Deployer do
     end
   end
 
-  # Verify the OTP runtime (erts-X.Y/bin/erl_child_setup) is present on
-  # the device. Without this, the BEAM can't start — symlinks fail with
-  # ENOENT, the app crashes immediately. This typically happens when the
-  # device wasn't connected during a previous `mix mob.deploy --native`.
+  # Verify the OTP runtime (ERTS binary + release bootfile) is present on
+  # the device. Without both, the BEAM can't start:
   #
-  # Returns :ok if ERTS is present, {:error, message} with a helpful hint
-  # if missing.
+  # - Missing `erts-*/bin/erl_child_setup` — symlinks fail with ENOENT, the
+  #   app crashes at BEAM launch.
+  # - Missing `releases/*/start_clean.boot` — the OTP dir has ERTS but no
+  #   release, so `erl_child_setup` starts and the emulator loads, but
+  #   boot-time reports "cannot get bootfile" and the app dies (MOB-183 /
+  #   mob_dev#54). This is the state a device lands in when the APK ships
+  #   ERTS in jniLibs but the release dir was never pushed via
+  #   `mix mob.deploy --native`.
+  #
+  # This typically happens when the device wasn't connected during a
+  # previous `mix mob.deploy --native`. Returns :ok when both are present,
+  # {:error, message} otherwise.
   defp ensure_erts_on_device(serial, pkg) do
-    # The wildcard must be expanded *inside* the run-as sandbox — `run-as`
+    # The wildcards must be expanded *inside* the run-as sandbox — `run-as`
     # itself does not invoke a shell, and the outer adb-shell shell can't
     # see /data/data/<pkg>/, so a literal "erts-*" gets passed to ls if we
-    # don't wrap with `sh -c` here.
-    cmd =
-      "run-as #{pkg} sh -c 'ls /data/data/#{pkg}/files/otp/erts-*/bin/erl_child_setup' 2>&1"
+    # don't wrap with `sh -c` here. `run_adb` already sets
+    # `stderr_to_stdout: true`, so the ls diagnostic lines land in the
+    # `out` we classify either way.
+    erts_glob = "/data/data/#{pkg}/files/otp/erts-*/bin/erl_child_setup"
+    boot_glob = "/data/data/#{pkg}/files/otp/releases/*/start_clean.boot"
+
+    cmd = "run-as #{pkg} sh -c 'ls #{erts_glob} #{boot_glob}'"
+
+    # The MOB-183 case sits on the `{:error, out}` arm: `ls` exits non-zero
+    # when a glob is missing, `run-as` propagates it, `sh -c` propagates
+    # it, and modern adb-shell (shell v2, default since Android 7) forwards
+    # it as the session exit code — so `run_adb` returns `{:error, out}`
+    # with the ls diagnostic lines still in `out`. We MUST classify that
+    # `out` too, or the exact bug this ticket fixes falls through silent.
+    # The classifier's own empty-output branch handles the true transport
+    # failure case (device disappeared between `list_devices` and here).
+    {_status, out} = run_adb(["-s", serial, "shell", cmd])
+
+    case classify_android_runtime_ls(out, erts_glob, boot_glob) do
+      :ok -> :ok
+      {:error, :run_as_unavailable} -> {:error, run_as_unavailable_message(serial, pkg, out)}
+      {:error, :erts_missing} -> {:error, erts_missing_message(serial, pkg)}
+      {:error, :bootfile_missing} -> {:error, bootfile_missing_message(serial, pkg)}
+    end
+  end
 
-    case run_adb(["-s", serial, "shell", cmd]) do
-      {:ok, out} ->
-        cond do
-          String.contains?(out, "run-as:") ->
-            {:error, run_as_unavailable_message(serial, pkg, out)}
+  @doc """
+  Classifies the output of the Android per-app runtime `ls` probe.
+
+  Returns `:ok` when both `erts_glob` and `boot_glob` resolve to a real
+  file. Returns one of the failure tags in preference order:
+
+  - `{:error, :run_as_unavailable}` — the shell couldn't `run-as` the
+    package (release build, non-debuggable APK). Nothing under
+    `/data/data/<pkg>/files/otp/` is reachable to `ls` at all.
+  - `{:error, :erts_missing}` — no `erts-*/bin/erl_child_setup`. The
+    BEAM can't launch; the app crashes at ERTS start.
+  - `{:error, :bootfile_missing}` — ERTS is present but no
+    `releases/*/start_clean.boot`. The emulator starts, then dies with
+    "cannot get bootfile". MOB-183.
+
+  Public so the classifier can be tested against captured adb output
+  without an emulator.
+  """
+  @spec classify_android_runtime_ls(String.t() | nil, String.t(), String.t()) ::
+          :ok | {:error, :run_as_unavailable | :erts_missing | :bootfile_missing}
+  def classify_android_runtime_ls(out, erts_glob, boot_glob) do
+    cond do
+      # No output to classify — a true adb transport failure (device
+      # disappeared between `list_devices` and this probe). Best-effort
+      # pass; downstream push failures still surface the real problem.
+      out in [nil, "", " "] ->
+        :ok
 
-          String.contains?(out, "No such file") or String.contains?(out, "not found") ->
-            {:error, erts_missing_message(serial, pkg)}
+      String.contains?(out, "run-as:") ->
+        {:error, :run_as_unavailable}
 
-          true ->
-            :ok
-        end
+      output_names_missing?(out, erts_glob) ->
+        {:error, :erts_missing}
 
-      _ ->
-        # adb shell failed entirely — let the deploy proceed and fail later
-        # if needed; this check is best-effort.
+      output_names_missing?(out, boot_glob) ->
+        {:error, :bootfile_missing}
+
+      true ->
         :ok
     end
   end
 
+  # A missing glob shows as an `ls: <path>: No such file or directory`
+  # (or "not found") line — the `<path>` in that message is verbatim the
+  # glob we passed since it never expanded. Match on the exact glob so
+  # ERTS-missing vs bootfile-missing don't confuse each other in the
+  # both-missing case.
+  defp output_names_missing?(out, glob) do
+    String.contains?(out, "ls: #{glob}: No such file") or
+      String.contains?(out, "#{glob}: not found") or
+      String.contains?(out, "ls: cannot access '#{glob}'")
+  end
+
   # `run-as` fails silently downstream: every beams/priv/exqlite push shells
   # out to `run-as #{pkg} tar xf ... 2>/dev/null; true`, where the trailing
   # `; true` is there to swallow Toybox tar's benign chown-to-macOS-UID
@@ -361,6 +424,33 @@ defmodule MobDev.Deployer do
     """
   end
 
+  # MOB-183: separate the "release dir missing" case from the "ERTS
+  # missing" case. Both are provisioned by `mix mob.deploy --native` but
+  # a device in the bootfile-missing state has ERTS binaries and would
+  # otherwise pass a check that only looked for `erl_child_setup` — the
+  # bug the reporter hit was a green deploy that yielded an app
+  # crash-dumping at boot with "cannot get bootfile" because
+  # `otp/releases/*/start_clean.boot` was absent.
+  defp bootfile_missing_message(serial, pkg) do
+    """
+    OTP release missing on device #{serial}.
+
+    The ERTS binaries are present under /data/data/#{pkg}/files/otp/erts-*/
+    but the release directory is empty — no /data/data/#{pkg}/files/otp/releases/*/start_clean.boot.
+    The emulator will load but crash-dump at boot with "cannot get bootfile"
+    (see MOB-183 / mob_dev#54).
+
+    This is the state a device lands in when the APK was installed via
+    `adb install` without a matching `mix mob.deploy --native` — or when
+    a previous `--native` pushed ERTS but was interrupted before writing
+    the release tree. Provision the release now:
+
+      mix mob.deploy --native --device #{serial}
+
+    Subsequent `mix mob.deploy` runs (without --native) will work normally.
+    """
+  end
+
   # If the Elixir stdlib on the device was installed by a different Elixir version
   # than the host (e.g. after `asdf` upgrade), regex literals and other stdlib
   # internals will be incompatible. Detect the mismatch and push updated BEAMs.

test/mob_dev/deployer_test.exs

@@ -367,4 +367,102 @@ defmodule MobDev.DeployerTest do
       assert Deployer.__sqlite_nif_target__(lines) == "/data/app/x/lib/arm/libsqlite3_nif.so"
     end
   end
+
+  # MOB-183: non-native mob.deploy used to report success on a device
+  # whose per-app OTP release directory was missing; the app then
+  # boot-crashed with "cannot get bootfile". The old check only verified
+  # ERTS binaries — this classifier verifies both ERTS and the release
+  # bootfile, and distinguishes the failure modes.
+  describe "classify_android_runtime_ls/3" do
+    @erts_glob "/data/data/com.example.demo/files/otp/erts-*/bin/erl_child_setup"
+    @boot_glob "/data/data/com.example.demo/files/otp/releases/*/start_clean.boot"
+
+    test "ok when both erts and bootfile exist" do
+      out = """
+      /data/data/com.example.demo/files/otp/erts-14.2.5/bin/erl_child_setup
+      /data/data/com.example.demo/files/otp/releases/29/start_clean.boot
+      """
+
+      assert Deployer.classify_android_runtime_ls(out, @erts_glob, @boot_glob) == :ok
+    end
+
+    test "run_as failure surfaces before glob checks" do
+      out = "run-as: unknown package: com.example.demo"
+
+      assert Deployer.classify_android_runtime_ls(out, @erts_glob, @boot_glob) ==
+               {:error, :run_as_unavailable}
+    end
+
+    test "erts missing when erts glob has no match" do
+      out = """
+      ls: #{@erts_glob}: No such file or directory
+      ls: #{@boot_glob}: No such file or directory
+      """
+
+      # Both missing prefers the ERTS tag — provisioning ERTS is the
+      # first thing --native does; there is no way to have the bootfile
+      # without ERTS in a real deploy, so reporting bootfile first would
+      # send the reader chasing the wrong file.
+      assert Deployer.classify_android_runtime_ls(out, @erts_glob, @boot_glob) ==
+               {:error, :erts_missing}
+    end
+
+    test "bootfile missing when only the boot glob has no match — MOB-183" do
+      out = """
+      /data/data/com.example.demo/files/otp/erts-14.2.5/bin/erl_child_setup
+      ls: #{@boot_glob}: No such file or directory
+      """
+
+      assert Deployer.classify_android_runtime_ls(out, @erts_glob, @boot_glob) ==
+               {:error, :bootfile_missing}
+    end
+
+    test "recognises the Toybox 'not found' variant of the missing-file message" do
+      # Some Android userland (Toybox `ls`) says "<path>: not found" rather
+      # than "ls: <path>: No such file or directory". The classifier must
+      # match either shape or the check silently passes on that userland.
+      out = "#{@erts_glob}: not found"
+
+      assert Deployer.classify_android_runtime_ls(out, @erts_glob, @boot_glob) ==
+               {:error, :erts_missing}
+    end
+
+    test "recognises the GNU coreutils 'cannot access' variant" do
+      out = "ls: cannot access '#{@boot_glob}': No such file or directory"
+
+      # This message doesn't contain the "ls: <glob>: No such file" prefix
+      # the primary branch looks for, so a naive substring on "No such file"
+      # would misfire between erts and bootfile in the both-missing case.
+      # Anchoring on the specific glob is the point of the classifier.
+      assert Deployer.classify_android_runtime_ls(out, @erts_glob, @boot_glob) ==
+               {:error, :bootfile_missing}
+    end
+
+    # The pre-review revision of the fix fell through to `:ok` whenever
+    # `run_adb` returned `{:error, out}` — which is exactly the arm that
+    # fires on the MOB-183 failure mode, because modern adb-shell (v2,
+    # default since Android 7) forwards the inner shell's non-zero exit as
+    # the session's own exit. `ensure_erts_on_device` now feeds `out` into
+    # the classifier regardless of the adb-shell status, so a real
+    # partial-runtime device reaches this classifier — the pathway must
+    # produce a verdict, not a silent pass. These fixtures are the exact
+    # `ls: ... No such file` strings adb-shell hands back on that arm.
+    test "MOB-183 regression: classifier fires on adb-shell error output" do
+      out = """
+      /data/data/com.example.demo/files/otp/erts-14.2.5/bin/erl_child_setup
+      ls: #{@boot_glob}: No such file or directory
+      """
+
+      assert Deployer.classify_android_runtime_ls(out, @erts_glob, @boot_glob) ==
+               {:error, :bootfile_missing}
+    end
+
+    test "empty output passes through as ok (adb transport failure fallback)" do
+      # A true adb transport failure (device disappeared, offline race) has
+      # no output to classify. Best-effort: let the deploy proceed and let
+      # the actual push failure surface the real problem.
+      assert Deployer.classify_android_runtime_ls("", @erts_glob, @boot_glob) == :ok
+      assert Deployer.classify_android_runtime_ls(nil, @erts_glob, @boot_glob) == :ok
+    end
+  end
 end