v2.9.0 - #587
Merged
Merged
v2.9.0#587
Conversation
An `if` whose branch scans a SYM column with a FILE-domain vocabulary translated its cells into runtime ids through the domain's whole- vocabulary LUT. The first request interns every vocabulary entry into the global symbol table (with the dotted split), permanently: on a 100M-row table with a 19.7M-entry column that is about 45 GB for a branch that reads 1.7M distinct values. The branch now carries its own translation table: slots start unresolved and a position is interned the first time the branch reads it, from the raw vocabulary snapshot. The serial scatter path resolves on the fly; the parallel eager fill pre-resolves the positions it will read before dispatching, so workers only read (sym.c frozen-table rule). The ids produced are the ones the LUT would have given. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…ary bytes The distinct-symbol evaluation of a computed group key fed the DAG a SYM column whose ids live in a FILE domain, so every string step of the expression became a permanent cost: substr and str-find intern each distinct result into the global symbol table (with the dotted split, about 1 KB per URL-shaped value), and an `if` that keeps the column itself translates its cells into runtime ids as well. On a 100M-row table with a 19.7M-entry column the key expression left about 65 GB in the symbol table that never came back. When the expression's result over the SYM column is SYM and the column's vocabulary is readable through the raw snapshot, the column is now presented to the same DAG as a STR column built from the vocabulary bytes, two million distinct values at a time. Every intermediate is a chunk-sized STR vector that dies with its chunk; only the final key strings are interned, once per distinct value, giving the ids the SYM evaluation would have produced. Other shapes keep the one-shot SYM evaluation. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Trivial branches through the pooled fill (past the parallel threshold), a computed branch through the selected path, the column on both sides under a row selection, the expression as a group key, and the empty symbol on either side — each against the same expression over the in-memory table. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The existing key expressions over a splayed copy of the table against the in-memory oracle, with the default chunk and with a 100-value chunk so the vocabulary crosses several chunks (RAY_DERIVED_KEY_CHUNK overrides the chunk length for this), a non-symbol key result that keeps the one-shot evaluation, and the null-bearing column. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The pre-resolution pass covered the cells the mask selects, but the SYM fill read both sides of every row before choosing, so workers met unresolved positions and interned them inside the pool. The fill now reads only the chosen side, through a read-only lookup once the sides are prepared. Both side tables are initialised before either result is checked, so the OOM path frees no uninitialised side. The table holds 32-bit ids (the symbol table counts them in 32 bits), half the scratch per vocabulary position. Test: a FILE-domain column on both sides of the `if` without a row selection, through the pooled fill. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…probe ray_table_add_col retains the column itself, so the retain before it left one reference on the distinct-symbol vector after every derived-key evaluation over a file-backed column, and with it the column's domain. The fallback block is re-indented as the else branch it is. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
… routing, composite key compaction - bound replicated dense slabs by the last-level cache (ray_cache_llc_bytes) - keep multi-key arithmetic-over-aggregates on the DAG engine; admit binary aggs into hidden slots - run desc/asc take: N group queries on v2 when a dense plan exists, trim afterwards - parallelize the filtered dense-plan prescan - compact interleaved symbol codes so composite symbol keys pack densely
Radix partitions finalize the emit filter's aggregate into one double per group in parallel, keep a bounded candidate heap each, take the threshold from the union, and emit only the kept superset in first-row order. v2 now reads the emit filter itself; every shape it admits runs there (the ladder keeps only shapes v2 declines). 10M rows, three-key count, desc take 10: 1 core 988 -> 287 ms; 28 cores 154 -> 43 ms (legacy ladder: 47 ms).
Dense task-local, partitioned and shared finishes select the kept groups over the occupied slot list (parallel value fill, bounded candidate heaps, one threshold) before any output column is built. Single-key count with desc take at 28 threads: 6.0 -> 5.4 ms; at one thread 28 -> 17 ms.
…-outs The emit filter no longer rejects a shape from the parallel engine. Routes that cannot select natively return through one wrapper that trims the full result with the shared keep decision; the legacy trim helper and the wide-domain carve-out in exec_group_run are gone.
The scatter, chunk count and compaction were dispatched by element grain over 128 partitions / 77 chunks, which yields one task, so the ordering ran serially. Dispatch by task count and compact out of place into an ng-sized buffer (the input-sized map is freed right after). 10M groups on 28 threads: ordering 63 -> 13.5 ms, whole query 194 -> 143 ms. Also lifts the rfl driver's file cap to 1024.
A bounded group emit no longer excludes the task-local dense strategy: its finish selects the N groups with the smallest first rows (an N-sized heap over the occupied slots) and emits them in first-seen order, exactly the prefix the radix bounded emit produced. 10M rows, 100k-key count take 10: 1 core 106 -> 17 ms, 28 cores 17 -> 5.3 ms.
Adds a --census PATH flag to the test driver that records, per evaluated .rfl line, grouped selects that took the legacy route with their admission reason; the document tallies the corpus (340 hits in five classes) and what v2 needs to close each.
fix(if): translate only the FILE-domain cells an if branch reads
…nstead of an env knob The chunk of the per-distinct-symbol key evaluation is 256 dispatch rounds of morsels (RAY_MORSEL_ELEMS * RAY_DISPATCH_MORSELS * 256, 2M values) rather than a bare power of two; the comment says why. The RAY_DERIVED_KEY_CHUNK environment variable is gone: a DEBUG-only ray_derived_key_chunk_set_for_test replaces it, so release builds carry no knob and no getenv per call. The several-chunks case moves to test_lang.c (lang/select/ derived_key_file_chunks): a 414-value vocabulary through 100-value and 7-value chunks against the in-memory oracle, the seam reset around each evaluation. The rfl test keeps the default-chunk FILE-domain checks. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…e ranges on compaction failure - top-N value view maps nulls/NaN to -INFINITY in both directions, matching sort_nulls_first (first ascending, last descending); the trimmed route applies the same mapping, so every v2 route returns the same set - agg_dense_plan_compact publishes shrunken key ranges only when every remap table exists; an allocation failure restores the raw ranges - drop the unused dense-plan probe and its thread-local cache
… bytes The asc:/desc: with take: path resolved a SYM sort key's two strings through ray_sym_domain_str on every compare. On a FILE domain that materialises an atom per vocabulary entry under the domain lock, so the first such query in a process paid for the whole vocabulary before it compared anything — seconds on a column with millions of distinct values, while the ordering itself takes milliseconds. A FILE-domain key now pins the raw vocabulary snapshot for the duration of the query and compares the entries' bytes in ray_str_cmp order (common prefix, then length); runtime-domain keys keep the borrowed string snapshot. The rows taken are identical. Test: test/rfl/symbol/file_domain_topk.rfl — one key ascending and descending, ties broken by a second key in both directions, the symbol as the second key behind an integer, the empty symbol taking part, a take past the row count, all against the same ordering over the strings and past the parallel threshold; the in-memory column as a control. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
… below the parallel threshold - agg_indexed_run frees the dense plan after the key build (tables leaked on every buffered aggregate over a compacted composite key) - the candidate-union threshold handles a single selection task, whose union holds exactly N values: the worst candidate is the threshold, so small dense top-N queries are selected natively instead of emitting every group; route stats record the kept count and the contract tests pin it - the partition-vs-task-local comparison uses the uncapped replication so the choice no longer moves with the machine's cache size (macOS runner)
The filter is thread-local and stays armed across every grouped select nested in the matched select's from: expression; v2 now requires the slot to hold the operation the filter names (count when unset). Also: a bounded emit that cannot allocate its selection fails instead of emitting the wrong prefix; groups within the limit are emitted in first-seen order too; the compaction prescan bounds its transient bitmap footprint.
…t select The filter is thread-local and stays armed while the arming select's from: evaluates, so a grouped select nested two levels down was filtered by its grandparent's having threshold (pre-existing: count-of-count under where (> n N) returned nothing when the inner counts fell below N). Each arming site now records the target depth (the direct child for the having shape, itself for take/desc and the count-distinct rewrite) and both engines read the filter through ray_group_emit_filter_active(), which returns it only at that depth. The nested-select test now exercises data an erroneous trim would empty.
perf(group): evaluate a derived symbol key in chunks over the vocabulary bytes
perf(sort): bounded-heap ordering compares FILE-domain symbols by raw bytes
perf(group): scale grouping with cores; v2 owns top-N and bounded emit
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What & why
Checklist
dev(notmaster)feat:/fix:/perf:/docs:/ …)makebuilds cleanly (no new warnings)make testpasses; tests added/updated for behaviour changes