Skip to content

sql: read a lag/lead value argument out of the original row - #38852

Draft
djahandarie wants to merge 2 commits into
MaterializeInc:mainfrom
djahandarie:dj/window-lag-lead-arg-from-original-row
Draft

djahandarie wants to merge 2 commits into
MaterializeInc:mainfrom
djahandarie:dj/window-lag-lead-arg-from-original-row

Conversation

@djahandarie

Copy link
Copy Markdown
Contributor

Motivation

Stacked on #38851; review that one first.

The window reduce's per-row value is
row(row(OriginalRow, EncodedArgs), OrderByExprs...). OriginalRow is a record
of every one of the window function's input columns, and it has to be: the
reduce's output reconstructs the input row from it. So a lag/lead whose
value argument is one of those columns stores that column twice in every row
the reduce arranges:

lag[offset=1, default=null, order_by=[#0{x} asc nulls_last]](row(row(row(#0{x}, #1{y}), #0{x}), #0{x}))
                                                                          ^^^^^        ^^^^^

The user-facing case that prompted this is four fused lag calls over one
partition, each on a plain column, so four wide columns were stored twice per
row across a 1.5 billion row input.

Description

LagLeadArgs, the description of the arguments the plan holds rather than
encodes per row, gains a value: Option<usize> naming the OriginalRow field
to read the argument from. It is renamed from ConstantLagLeadArgs because it
no longer describes only constants, and AggregateFunc::LagLead's field is
renamed from constant_args to args to match.

That leaves exactly three shapes for the per-row encoded arguments, and the
description says which one a call uses:

args encoded per row
None the full (value, offset, default) record
Some, value: None the bare value datum
Some, value: Some(k) nothing

With nothing to encode the (OriginalRow, EncodedArgs) record keeps only its
first field, so the plan reads

lag[value=orig_row[0], offset=1, default=null, order_by=[#0{x} asc nulls_last]](row(row(row(#0{x}, #1{y})), #0{x}))

A fused constituent that encodes nothing contributes no field to the fused
argument record, and that record is absent altogether when no constituent
encodes anything. AggregateFunc::encodes_window_args is the one predicate that
answers "does this constituent contribute a field", and the eval, the output
type computation and on_unique all walk the record against funcs through it
rather than zipping the two.

The value field is set in lowering, not in HIR, for two reasons: whether the
argument is an input column is only apparent in its MIR form, and the field
index is a MIR column index. Self::describe_window_args does that, and bounds
the index by the window function's input arity, because lowering an argument can
map further columns onto the input (a subquery in an argument) and those are not
in OriginalRow.

Only lag/lead with both other arguments already described take part. Keeping
the shape count at three is deliberate: a per-row offset would otherwise add a
fourth, (offset, default). The same redundancy exists for first_value,
last_value and window aggregates, whose single argument is also usually a
plain column (window_agg(row(row(row(#0{x}, #1{y}), #0{x})))); extending the
description to them is a follow-up, not in this PR.

Costs

Reading the value back walks OriginalRow as far as the named field, so this
trades a datum walk bounded by the field index, once per row per call, for not
storing the column twice. That is the intended direction for a non-incremental
window function, whose dominant cost is already the per-partition sort, and it
is the direction the reported workload needs.

Tests

  • The ## lag/lead argument encoding section in
    test/sqllogictest/window_funcs.slt now pins one plan per shape: a described
    value, an expression value that stays encoded while the constants are
    described, all three encoded, a fused pair where one call encodes and the
    other does not, and a fused pair where neither does and no argument record is
    emitted.
  • The file's existing lag/lead coverage exercises each shape end to end,
    through both the reduce and the on_unique path.
  • EXPLAIN goldens in test/sqllogictest/explain/ are updated, redacted
    variants included.

The LIR schema snapshot is regenerated in place; LIR_VERSION stays at 1,
which is unshipped.

No user-visible behavior change.

🤖 Generated with Claude Code

djahandarie and others added 2 commits September 15, 2026 05:56
`lag`/`lead` take three arguments, and the window `Reduce` encoded all three
into a record per input row. The `offset` and `default` arguments are usually
plan-time constants, so the arrangement held two constants per row, once for
every row of every partition.

`AggregateFunc::LagLead` now carries the pair in `constant_args` when both are
literals, and the per-row encoded argument is the bare `value` datum instead of
a `(value, offset, default)` record. The evaluation materializes the constant
`default` into the arena once per partition and shares it across the rows.

Hoisting is all-or-nothing per call, because the encoded-argument shape is
either the full record or the bare value. A per-row `offset` or `default` keeps
all three arguments in the record. Fused value window functions decide
per constituent call, so one `lag` in a fused group can hoist while another
does not.

`EXPLAIN` prints the hoisted arguments inside the function's brackets, for
example `lag[offset=1, default=null, order_by=[...]]`, so the plan still fully
describes the call. They are literals and are redacted like any other.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A window function's per-row value carries an `OriginalRow` record holding every
input column, because the reduce's output has to reconstruct the input row. A
`lag`/`lead` whose `value` argument is one of those columns packed a second copy
of it next to the record, once for every row of every partition.

`LagLeadArgs` (the description of the arguments the plan holds rather than
encodes, formerly `ConstantLagLeadArgs`) gains a `value` field naming the
`OriginalRow` field to read the argument from. That leaves three shapes for the
per-row encoded arguments, and the description says which one a call uses: the
full `(value, offset, default)` record, the bare `value` datum, or nothing at
all. With nothing to encode, the `(OriginalRow, EncodedArgs)` record keeps only
its first field, and a fused constituent contributes no field to the fused
argument record, which is itself absent when no constituent encodes anything.

The `value` field is set during lowering rather than in HIR, because whether the
argument is an input column is only apparent in its MIR form, and the field
index is a MIR column index. Lowering also bounds it by the window function's
input arity: lowering an argument can map further columns onto the input, for a
subquery in an argument, and those are not in `OriginalRow`.

Reading the value back walks `OriginalRow` as far as that field, so this trades
a datum walk bounded by the field index for not storing the column twice.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@djahandarie
djahandarie force-pushed the dj/window-lag-lead-arg-from-original-row branch from 0d891c9 to eb25bf4 Compare September 14, 2026 21:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant