Skip to content

feat(llc): add normalizeStringForSort and Filter.raw, fix array matching - #181

Draft
xsahil03x wants to merge 20 commits into
mainfrom
feat/chat-migration-gaps
Draft

feat(llc): add normalizeStringForSort and Filter.raw, fix array matching#181
xsahil03x wants to merge 20 commits into
mainfrom
feat/chat-migration-gaps

Conversation

@xsahil03x

@xsahil03x xsahil03x commented Sep 9, 2026

Copy link
Copy Markdown
Member

Three changes to stream_core, all walls the stream_chatstream_core migration hit.

Five more were added and taken back out under review. The history is left intact rather than squashed — Considered and dropped is the more useful half of this PR, and three of those five were things the migration plan had recommended before the evidence came in.

normalizeStringForSort

ComparableField compares strings with String.compareTo, which orders by code unit and pushes lowercase-leading and accented names (jhon, Łukasz, Øystein) past Zara. A locally sorted list then disagrees with the one a query returns, which every product hits the moment it sorts by name.

Folding inside ComparableField was considered and rejected: Filter compares through it too, so $eq, $gt and $lt would become case- and diacritic-insensitive locally while the API stayed exact. This is a utility to apply in a SortField value getter instead. Ported from stream_chat with its tests; preserves Japanese, Thai and Vietnamese-specific runes.

Filter.raw

Filter is sealed, so a downstream package cannot add a variant — a gap here has no workaround. The concrete case: a Stream predefined filter is authored server-side and echoed back to the SDK, and GetStream/chat#15657 withdrew $ne and $nin from the published spec without removing them from the query layer, so an echoed filter can legitimately carry an operator core does not model. raw is the fallback leaf that lets a decoder stay total, the way an unrecognised sort field falls back to a custom one.

matches throws for it. Returning true is right under and and wrong under or/nor, where an opaque node would match every record; returning false inverts the problem. Swift resolves this with a nullable predicate and compactMap, dropping nodes it cannot wire, but bool matches(T) has no third state. Refusing is the only outcome that cannot silently produce a wrong result. ComparableField already throws from this layer.

Array-valued fields match element-wise

Filter.equal and Filter.in_ compared a field's value with deep equality, which is order-sensitive for lists. A field whose getter returns a list therefore never matched in practice: $in asked whether the whole list equalled one of the query values, and $eq asked for the same elements in the same order.

Neither is what a query does. Every array field the API exposes — a channel's members and filter_tags, a user's teams, a message's attachments.type — is matched element-wise, comparing as a set for $eq and intersecting for $in. Not one is order-sensitive, so the old behaviour could not agree with any query the API accepts.

An array passed to $in still compares whole, so a value that is itself an array keeps its meaning; only plain values intersect. The $eq test that asserted order-sensitivity is updated, since no query can produce that answer.

Considered and dropped

$nor. Neither StreamCore (Swift) nor stream-android-core models it, and adding it to Dart alone breaks a parity the three otherwise keep. stream-chat-swift depends on StreamCore and keeps its own 671-line Filter, undeprecated, so no chat SDK uses core's filter today. Chat is removing $nor from its own surface rather than pushing it up.

Filter.empty. Nullability covers it, which is how stream_feeds already works: Filter? on the query, defaulted at the request boundary. It is also not merely unused — it serializes to {}, and queryThreads reads request.Filter != nil rather than a length, so a neutral filter is non-nil there and switches the request onto a wider server query while omitting the field does not.

Sort value equality. It compared field.remote and ignored the comparator, so two sorts naming the same wire field with different value getters compared equal and hashed equal while ordering a list in opposite directions. Unfixable while a field carries a closure: comparing the field itself is identity, and closures have no meaningful equality.

SortDirection.fromJson. The generator never calls it. json_serializable uses a fromJson static for class-typed fields, but decodes enums through $enumDecode against the generated map, taking its fallback from @JsonKey(unknownValue:) at each field site. It would have been reachable only by hand, while generated code disagreed with it.

@JsonEnum(valueField:) on SortDirection. Removed a real duplicate, but once fromJson was gone nothing depended on the two declarations agreeing. sort.dart is untouched by this PR.

A per-field isCollection opt-in. The first shape of the array fix, so the default could stay order-sensitive. It needed a new constructor on FilterField and a flag at every collection declaration, to reach behaviour the runtime type already determines — and no API field turned out to want the order-sensitive reading.

Verification

dart analyze --fatal-infos lib test clean, full suite green at 827 tests, up from 812.

The commit count is high for the size of this — the net diff is one new file, one new filter variant, two matches bodies and their tests. Reviewing the diff rather than the commits is the faster path; the commits are there for the rejections.

🤖 Generated with Claude Code

xsahil03x and others added 4 commits September 9, 2026 22:42
`Filter` modelled `$and` and `$or` but not `$nor`, so a query needing "none of
these" had no way to express it — and with `Filter` sealed, a downstream SDK
cannot add the operator itself.

`NorOperator` joins the existing `LogicalOperator` family, so it serializes and
nests like the other two, and `matches` is the negation of `OrOperator`'s.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`Sort` is `@JsonSerializable(createFactory: false)`, so nothing could read back
the `{field, direction}` shape it writes. Every SDK that persists a sort or
reads one echoed by the API has to re-derive the rule, and it is easy to get
wrong: anything other than `-1` is ascending, so a direction the API did not
write is ascending rather than an error. `SortDirection.fromJson` owns that
rule once. Field resolution stays with the caller, which is the part only it
can do.

`Sort` also had no `==`, so two independently written identical sorts compared
unequal. All its fields are final and its constructors are const, so it is
annotated `@immutable` and compares on the field's remote name, the direction
and the null ordering.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`Filter` is sealed, so a downstream SDK cannot add a variant. Two cases had
nowhere to go as a result.

`Filter.empty` constrains nothing and serializes to `{}`. It is the identity
element of `Filter.and`, for the common case of building a filter
conditionally. `Filter.and([])` is not a substitute: it serializes to
`{"$and": []}`, which the API rejects.

`Filter.raw` carries a query this package does not model, serialized verbatim.
Without it a caller who needs an operator core has not modelled is stuck, since
the sealed hierarchy leaves no way to work around the gap. It cannot be
interpreted, so `matches` returns `true` for it rather than guessing, and its
dartdoc says so.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`ComparableField` compares strings with `String.compareTo`, which orders by
code unit and so pushes lowercase-leading and accented names (`jhon`,
`Łukasz`, `Øystein`) past `Zara`. A list sorted locally then disagrees with the
one a query returns, which every product hits the moment it sorts by name.

Folding inside `ComparableField` was considered and rejected: `Filter` compares
through it too, so `$eq`, `$gt` and `$lt` would become case- and
diacritic-insensitive locally while the API stayed exact. This is a utility to
apply in a `SortField`'s value getter instead, leaving comparison semantics
alone.

Ported from `stream_chat`, along with its tests. Preserves Japanese, Thai and
Vietnamese-specific runes, and mirrors the API's own `NormalizeName`.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@coderabbitai

coderabbitai Bot commented Sep 9, 2026

Copy link
Copy Markdown

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

  • Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@codecov

codecov Bot commented Sep 9, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 97.67442% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 68.17%. Comparing base (2aea141) to head (01982a3).

Files with missing lines Patch % Lines
...kages/stream_core/lib/src/query/filter/filter.dart 93.33% 1 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main     #181      +/-   ##
==========================================
+ Coverage   68.02%   68.17%   +0.14%     
==========================================
  Files         209      210       +1     
  Lines        8554     8597      +43     
==========================================
+ Hits         5819     5861      +42     
- Misses       2735     2736       +1     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Adding it here would have put this package ahead of the other two cores:
neither StreamCore (Swift) nor stream-android-core models `$nor`, and their
operator sets are otherwise identical to this one.

The case for adding it was that the chat SDKs all expose `$nor` and would lose
it on migrating. That does not hold. stream-chat-swift depends on StreamCore
and keeps its own 671-line `Filter`, undeprecated, so no chat SDK uses core's
filter and none is scheduled to. Core has no consumer for the operator.

Widening the core filter family is a cross-core decision to take with all three
owners at once, not a side effect of one product's migration.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@xsahil03x xsahil03x changed the title feat(llc): close the query-DSL gaps the chat migration hit feat(llc): close the sort and filter gaps the chat migration hit Sep 9, 2026
xsahil03x and others added 2 commits September 9, 2026 23:08
`Filter.and([])` was not a substitute for it, but nullability is, and that is
how the one package using this filter already works: `stream_feeds` types its
query filters `Filter?` and defaults at the request boundary, so it never needs
a neutral value.

It is also not merely unused. It serializes to `{}`, and `queryThreads` reads
`request.Filter != nil` rather than a length — so a neutral filter is non-nil
there and switches the request onto a different, wider server query, while
omitting the field does not. Every other endpoint checks the length and cannot
tell the two apart. An API whose only correct use is "never pass this to
threads" is worse than no API.

`Filter.raw` stays: it carries a query this package does not model, which the
sealed hierarchy otherwise leaves no way to express.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The equality was not merely unused, it was wrong. It compared `field.remote`
and ignored the comparator, so two sorts naming the same wire field with
different value getters — a declared field and a custom one for the same key —
compared equal and hashed equal while ordering a list in opposite directions.
As a map key or in a dedupe that silently yields the wrong order.

It cannot be fixed either. Comparing the field itself is identity, since
`SortField` has no `==`, which defeats the purpose; and the field's comparator
is a closure, which has no meaningful equality. There is no correct value
equality for `Sort` while a field carries a closure.

`@immutable` went with it, having been added only to satisfy the lint that
guards `==`/`hashCode`.

Also `@JsonEnum(valueField: 'value')` on `SortDirection`: the wire values were
declared twice, once as `@JsonValue` for the generated map and once as the enum
field that `fromJson` reads. Changing one and not the other would have made
serialization disagree with deserialization silently. The generated map is
unchanged.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@xsahil03x xsahil03x changed the title feat(llc): close the sort and filter gaps the chat migration hit feat(llc): close the gaps the chat migration hit Sep 9, 2026
The generator never calls it. `json_serializable` uses a `fromJson` static for
class-typed fields, but decodes an enum through `$enumDecode` against the
generated value map, taking its fallback from `@JsonKey(unknownValue:)` at each
field site and otherwise throwing.

So the static would be reachable only by hand, while any generated code
disagreed with it: strict where the enum documented itself as lenient. An enum
cannot force the lenient path — only a field can — which makes this the wrong
place for the rule regardless of how right the rule is.

`@JsonEnum(valueField: 'value')` stays. It removes a real duplicate: the wire
values were declared once as `@JsonValue` for the generated map and once as the
enum field, and changing one without the other would have gone unnoticed.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@xsahil03x xsahil03x changed the title feat(llc): close the gaps the chat migration hit feat(llc): add normalizeStringForSort and Filter.raw Sep 9, 2026
xsahil03x and others added 9 commits September 9, 2026 23:20
With `fromJson` gone nothing reads the enum's `value` field for JSON, so the
duplication the `valueField` form removed no longer has anything depending on
it. Unrelated churn in a PR about two additions.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`matches` returned `true`, which is right under `and` and wrong under `or` and
`nor`, where an opaque node would match every record. Reporting no match
inverts the problem. There is no answer that is correct under every
composition, because the node cannot be interpreted at all.

Swift resolves this with a nullable predicate and `compactMap`, dropping nodes
it cannot wire, but `bool matches(T)` has no third state to drop into. So it
throws, which is the only outcome that cannot silently produce a wrong result,
and the message names the query that caused it. `ComparableField` already
throws from this layer for the same reason.

Nothing evaluates filters locally today, so this costs nothing now and turns a
future wrong list into an immediate, precise failure.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Three deviations from STYLE_GUIDE and Effective Dart. The `Filter.raw` summary
opened with an article where every sibling factory in the file uses a bare noun
phrase (`Equality filter matching …`), and its second paragraph was a fragment.
`String.compareTo` was backticked in the normalizer, but this repo reserves
backticks for names that are *not* in scope and `dart doc` resolves the
bracketed form to a link.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The docs had grown into rationale dumps: why refusing to evaluate beats
guessing, why folding is selective. That reasoning belongs in the commit and
the PR, not in what a caller reads on hover.

The normalizer also listed its four folding steps in order and linked the
backend Go source that inspired them. Both are implementation, and the link
points at a repository a consumer of this package cannot open. The link moves
to a `//` comment on the function, where a maintainer will find it and the
generated docs will not carry it.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
It sat between the dartdoc and the declaration, which reads as a mistake, and
it was attached to the wrong thing: the selectivity it explains is `_foldRune`'s
preserved-script branches, not the public entry point. `_foldRune` already
carried a `//` comment describing exactly that.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Trimming the verbosity took the steer with it, leaving `raw` reading as one
option among the operators rather than the escape hatch it is. It now says so
first, and says what a caller gives up: a declared operator names its field
through a `FilterField`, so a typo is a compile error and the filter stays
evaluatable, and `raw` is neither validated nor evaluatable.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
It had grown into one dense paragraph carrying three separate points: when the
escape hatch applies, why a declared operator is better, and what it costs.
Split so each reads on its own — summary, when and what to prefer, then the two
things a caller gives up.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
page_width is 120 and the line is 83; dart format does not reflow comments, so
the narrow wrap was mine to fix rather than the formatter's.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`Filter.equal` and `Filter.in_` compared an array field with deep
equality, which is order-sensitive and so never matched the way a query
does. A field whose getter returns a list now compares as a set for
`$eq`, and intersects for `$in`.

An array passed to `$in` still compares whole, so a value that is itself
an array keeps its meaning.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@xsahil03x xsahil03x changed the title feat(llc): add normalizeStringForSort and Filter.raw feat(llc): add normalizeStringForSort and Filter.raw, fix array matching Sep 9, 2026
xsahil03x and others added 3 commits September 10, 2026 01:40
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant