Streaming deserialize: add source stream and task

[bjester]

Summary

• Extracts common behavior for streaming source and task classes into new base classes for use with both serialization and deserialization
• Adds new StoreQuerySet methods that apply filtering for selecting records based on whether they have deserialization errors. These can be used in Kolibri later
• Adds method to model registry for querying store records in model dependency order in parity with the method used by serialization
• Creates deserialization stream and task specific classes as foundation for streaming deserialization stage

TODO

• Have tests been written for the new code?
• Has documentation been written/updated?
• New dependencies (if any) added to requirements file

Reviewer guidance

• Do the tests cover the new and refactored code enough?
• Did I properly translate the self-ref-ordering and deserialization error filtering that were added after this PR was opened in draft? (I rebased it and applied

Issues addressed

Closes #317

AI Usage

I used Claude to do the rebase and bring it up-to-date with the upstream changes. It made a rightful mess of it, and so I rewrote some of the changes to better follow the patterns and then had it clean up the mess by updating tests.

[rtibblesbot]

🟡 Waiting for changes

Last updated: 2026-08-25 23:27 UTC

[rtibblesbot]

PR #318 delivers all four of issue #317's items — DeserializeTask, StoreModelSource, the shared MorangoSource base, and tests for both — plus the StoreQueryset error filters and get_store_querysets. No blocking defects.

The substantive comments are about semantics the shared base imports into deserialization: partition-major iteration vs. the model-major ordering _deserialize_from_store guarantees today, a per-task fk_cache that can never hit, a skip_errored default that inverts the legacy one, and a lost __slots__ optimization.

CI: Linting and Migrations pass on acc181f; Python tests were still running. The one completed failure (Handle pull request events at Generate App Token) is infrastructure, unrelated to the diff. Locally tests/sync/stream/, test_registry.py and models/test_core.py pass (106) and makemigrations --check is clean.

No UI files, so Phase 3 / manual QA did not apply.

---

@rtibblesbot's comments are generated by an LLM, and should be evaluated accordingly

How was this generated?

Ran a phased review pipeline over the pull request diff:

• Classified the diff to select review passes (core, frontend, backend) and whether manual QA was required
• Core review pass checked correctness, design, architecture, testing, completeness, and DRY/SRP/Rule-of-Three principles
• Specialized frontend/backend review passes applied framework-specific lenses where those files changed
• For UI changes: manual QA and an accessibility audit against a live dev server, when available
• Checked CI status and linked issue acceptance criteria
• Synthesized one review from those passes and chose the verdict from the findings, CI status, and QA evidence

[rtibblesbot]

suggestion: Partition-major iteration drops the model-dependency ordering deserialization relies on.

This loops prefixes outer, models inner, so emission is prefix1 × (model1..modelN), then prefix2 × (...). The comment at deserialize.py:71-72 claims streaming model by model ensures FK targets come first — that only holds within a single prefix pass.

_deserialize_from_store is the opposite shape: models outer (operations.py:303) with every prefix OR'd into one partition__startswith condition (operations.py:313-319), so dependency order is global across the whole filter. Here, a modelN record in prefix1 whose FK target is a model1 record living only in prefix2 is emitted before its target.

Ordering is irrelevant for serialization, which is presumably why the base landed this way. For deserialization it is the guarantee. Either invert the loops in StoreModelSource (models outer, prefixes OR'd inner, matching the existing code), or drop the comment's claim and document that cross-prefix FK targets are unordered.

[bjester]

There are no cross-partition FKs. The ordering is a naive approach that works with known filters, which prioritize the shortest filters first, which are likely the least specific.

[rtibblesbot]

suggestion: SourceTask declares no __slots__, so it contributes a __dict__ to every subclass and SerializeTask/DeserializeTask's own slots become inert. SerializeTask had working slots before this refactor.

Verified on this class shape:

t = T(); hasattr(t, "__dict__")  # True
t.zzz = 5                        # succeeds

One task instance exists per record streamed, in a pipeline whose stated point is "reducing memory overhead" (core.py:1-6). abc.ABC itself sets __slots__ = (); the same line on SourceTask restores both the memory saving and the typo-catching.

[rtibblesbot]

suggestion: _seen accumulates one entry per record for the source's lifetime, even when nothing can duplicate. With sync_filter is None, prefix_conditions() yields exactly one None, so stream_for_filter runs once and no id can repeat — but a full-store pass still holds every Store id in memory, which is the cost the streaming design exists to avoid.

Skipping the bookkeeping when there is a single prefix condition covers that case. Also note _seen is never reset, so a second stream() call on the same instance yields nothing.

[rtibblesbot]

nitpick: "passes thoughts to stream_for_filter" — "through".

[rtibblesbot]

suggestion: A fresh {} per store record gives the FK cache no reuse. _deserialize_from_store creates it once (operations.py:297) and threads the same dict through every _deserialize_store_model → cached_clean_fields(fk_cache, ...) call (core.py:548). Per-record, it is always empty on first use, so every FK lookup hits the DB.

Issue #317 asks the task to track an FK cache for reuse. A transform could overwrite task.fk_cache with a shared dict later, but this default silently defeats the optimization — having StoreModelSource own one dict and hand the same reference to every task it yields would match the intent.

[rtibblesbot]

suggestion: The _self_ref_order sort is applied to every model, including those with no self-referential FK. get_model_querysets directly above gates its order_by on morango_ordering being non-empty; this one doesn't.

_self_ref_order is NULL on every row for a model without a self-referential FK (operations.py:757-759 nulls it out for exactly those), and it is in no index — Store.Meta.indexes covers partition and (profile, model_name, partition, dirty_bit) (core.py:486-494). Generated SQL:

SELECT ... FROM "morango_store"
WHERE ("model_name" = abc AND "profile" = facilitydata)
ORDER BY "_self_ref_order" ASC NULLS LAST

So for most models it forces a full sort of the result set — once per model, per partition prefix — before the first row can be yielded, working against the .iterator() streaming this exists for. The registry already caches the answer in get_self_referential_fk(model); gating the order_by on it would skip the sort for the majority of models.

[rtibblesbot]

suggestion: This is a verbatim extraction of operations.py:326-336, comment included, but that call site still hand-rolls the annotation. Swapping it for store_models.exclude_has_deserialization_error() is one line and keeps the two from drifting — #317's "do not modify the existing deserialization process" is about behaviour, and this is behaviour-preserving.

[bjester]

The code in operations.py will eventually be removed. There is no need to update it.

[rtibblesbot]

nitpick: NULLIF(deserialization_error, '') yields text, not a boolean. It works because only __isnull is applied, but the annotation is in the SELECT list too, so any caller reading _deserialization_error off a row gets a string from a field declared boolean. Inherited from operations.py:329-333; TextField() would be accurate in both.

[rtibblesbot]

suggestion: StoreModelSourceTestCase mocks the ORM throughout, so nothing here verifies the filters match real rows. Assertions are on call args (qs.filter.call_args_list == [mock.call(dirty_bit=True)], qs.filter.assert_not_called()), which pass for whatever kwargs the source happens to send — spec=StoreQueryset catches a missing method, not a missing field, so a typo'd or renamed field stays green.

test_stream__preserves_registry_queryset_order and test_stream__deduplicates_across_partition_passes genuinely need the registry stubbed; the filter tests don't. GetStoreQuerysetsTestCase in this same PR shows the alternative at about the same length — build a handful of Store rows with varying dirty_bit/deserialization_error/partition and assert which ids stream out.

[rtibblesbot]

praise: Asserting the rows actually returned rather than generated SQL — with the reasoning written down — is the right level for a guarantee deserialization correctness depends on, and it stays valid across SQLite and Postgres.