turbo: backport native .mpy loading without the emitter (micropython#18596) - #11331
Merged
Conversation
This commit replaces the target architecture definition method, from relying on defines in MicroPython's configuration, to using the host compiler's environment definitions instead. Before these changes the target type was inferred from which kind of native emitter was enabled, since there can be only just one available at all times and it has to be the correct one otherwise generated code would crash the target. However, with the introduction of RV64 there is now a platform without an emitter, and thus RV64 targets would not be able to report their platform via "sys.implementation._mpy". The target is reported only if the interpreter is configured to load external MPY code. Now a series of compile definitions are used to detect the target platform the firmware image is compiled for, making the target definition more accurate and much harder to force the interpreter to report the wrong target architecture due to a misconfiguration. Whilst this is probably not directly affecting user-facing code, some infrastructure components like the testing framework and its associated feature scripts rely on this feature to properly execute test runs. This is also a concern for situations in which loading external binary code is needed but all code generation functions have to be disabled. Signed-off-by: Alessandro Gatti <a.gatti@frob.it> (cherry picked from commit 5ae928a)
This commit lets the interpreter load MPY files containing native code even if the target platform does not have a native emitter, or if native code generation is disabled. Native code loading has been tied to native code generation being enabled as a discriminant to allow said operation. This blocks native code loading on platforms that could benefit from such a thing but they don't (and probably won't) have a native code generation target written for them (ie. AArch64 and RISC-V 64). This also forces a firmware image to have a full native code compiler present even if it doesn't need to generate anything, as native modules already have all the code they will ever need to load. There is a new configuration setting, MICROPY_PERSISTENT_CODE_LOAD_NATIVE, that if enabled it will allow loading native code modules even if code generation (MICROPY_EMIT_<platform> and MICROPY_EMIT_INLINE_<platform>) is explicitly turned off. Signed-off-by: Alessandro Gatti <a.gatti@frob.it> (cherry picked from commit b0f3ecd)
This commit forces a data/instruction cache flush after loading native code blocks even if there's no emitter available. Caches flush upon native code load was still tied to the presence of a native emitter, but this assumption is no longer valid due to previous commits decoupling native code loading from code generation availability. Signed-off-by: Alessandro Gatti <a.gatti@frob.it> (cherry picked from commit 9e9da6c)
This commit makes use of the new native code loading changes to provide a test bed for a target with the native compiler framework available but without no code loading capabilities. mpy-cross already fully disables native code loading, however those changes do not really propagate anywhere outside `py/persistentcode`. Recent changes in the code loading framework actually touch the core code in a few more places, but no default CI target is configured to run with the new native code loading defines switched off. Given that mpy-cross doesn't really need to load code anyway, it is a good target for setting up a configuration that goes a bit deeper when it comes to disabling code loading. Since mpy-cross is built several times during the CI process, it can be repurposed as an early warning system for issues related to these new changes. Normal operation should not be affected in any way. Signed-off-by: Alessandro Gatti <a.gatti@frob.it> (cherry picked from commit 8fb848f)
The cache flush conditions from upstream 9e9da6c compare MPY_FEATURE_ARCH against MP_NATIVE_ARCH_* in preprocessor #if and #elif lines. Those are enum values, which the preprocessor reads as 0, and emitglue.c does not include persistentcode.h, so MPY_FEATURE_ARCH is undefined there as well. Upstream compiles because every identifier is 0 and the expressions are accidentally true; CircuitPython builds with -Werror=undef and stops, on the thumb line for ARM and on the RV32 line for any non-ARM target that falls through to it. Test the compiler defines that persistentcode.h itself now uses to pick the architecture, so a loader-only thumb build flushes the caches and a non-ARM build leaves the block out. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
mikeysklar
changed the title
Member
|
Backporting like this can make updating later tougher. I'd prefer to update to 1.28 (and then 1.29) instead. LLMs should make that easy. |
Collaborator
Author
|
Makes sense. CircuitPython is currently at 1.27 and going to 1.28 is 316 commits. I can open a PR for the 1.28 update and close this if you want me to take that on. I would be using the same squashed wip style @dhalbert used in April. |
Member
|
@mikeysklar Have your clanker give it a shot. Don't spend a bunch of your own time though. |
Collaborator
|
My experience is that an upstream merge requires a lot of manual review of the merges, including both the automatic merges and the merge conflicts. Adding and understanding CIRCUITPY-CHANGE comments helps a lot. But I never tried a merge with LLM assistance. |
Collaborator
Author
|
I was able to boot after the MP 1.28 update on the following boards. Branch
|
mikeysklar
added a commit
to mikeysklar/circuitpython
that referenced
this pull request
Sep 9, 2026
Restores the merge-commit style last used for v1.25.0 in adafruit#10437. v1.26 was skipped and v1.27 landed as a squashed diff port (14c1815), so no MicroPython tag since v1.25.0 has been an ancestor of main and the merge base for this one is v1.25.0 rather than v1.27.0. That is why this merge reports 1104 conflicted files: it replays 1350 upstream commits, 745 of the conflicts being MicroPython ports CircuitPython does not carry. Conflicts are resolved to the tree from the v1.27.0..v1.28.0 diff port, which is restricted to the paths CircuitPython does carry and was tested on hardware. The resulting tree is byte-identical to that branch; only the history differs. Content summary: - 88 files applied without conflict - 26 files hand-merged against CircuitPython deviations - 91 new upstream test and doc files added - py/modstring.c, py/modweakref.c and py/objtemplate.c added and wired into py.mk and py.cmake. All three are inert: their feature gates sit above CircuitPython's CORE_FEATURES ROM level - skipped 23 extmod and lib files CircuitPython does not carry, plus the tests/ports, tests/multi_wlan and tests/net_inet directories Every CIRCUITPY-CHANGE deviation was preserved. One new deviation: upstream moves the tuple and list helper declarations out of py/obj.h, so py/obj.h now includes py/objlist.h and py/objtuple.h at the bottom, keeping the roughly 55 consumers outside py/ working unchanged. Supersedes adafruit#11331. All four commits from micropython#18596 are in v1.28.0, as is upstream's d41b8dc, which fixes the same -Werror=undef breakage that PR fixes by other means. Tested on six ARM boards: Metro RP2040, Metro RP2350, Metro M4 AirLift Lite, Metro M0 Express, Feather nRF52840 Express and Feather STM32F405 Express. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Collaborator
Author
|
Superseded by #11334, which updates MicroPython to v1.28.0 as a merge commit. All four commits from micropython#18596 are in v1.28.0 with their Closing in favour of #11334. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
First of five PRs for turbo, which compiles a module to native
.mpyon the host withmpy-crossso the board needs only the loader, not the on-board emitter. The emitter costs 20 to 50 KB and does not fit on the smaller boards; the loader costs under 3.1 KB. This one is the upstream groundwork and changes no behavior on its own.The rest of the series: the
CIRCUITPY_LOAD_NATIVEbuild flag and five ARM boards, automatic Thumb-2 emitter configuration, the non-ARM pointer bit fix, and Xtensa and RISC-V on ESP32.What
Four commits from micropython#18596, cherry-picked with
-x, upstream authorship kept, plus one fix of ours:5ae928a2b4b0f3ecd96c9e9da6cb0d8fb848f3aaCIRCUITPY-CHANGE9e9da6cb0dcompares enum values inside#if, which fails under-Werror=undef. Test the compiler defines insteadNo behavior change, measured
MICROPY_PERSISTENT_CODE_LOAD_NATIVEkeeps its upstream default ofMICROPY_EMIT_MACHINE_CODE(py/mpconfig.h:431). No CircuitPython override, no build flag, no board changes, so it is off everywhere.Built five boards with and without it on Linux with GCC 14.2.1: Metro M4 AirLift Lite, Metro RP2040, Metro RP2350, Feather nRF52840 Express and Feather STM32F405 Express. Compared with
cmp -l, everyfirmware.binis the same size and byte-identical apart from thegit describestring, 36 differing bytes on the M4 and 48 on the others.mpy-crossbuilds.AI assistance
Claude Code prepared the cherry-picks and the fix commit. I reviewed every hunk against upstream and ran the builds.