sql: read a lag/lead value argument out of the original row - #38852
Draft
djahandarie wants to merge 2 commits into
Draft
djahandarie wants to merge 2 commits into
djahandarie wants to merge 2 commits into
Conversation
This was referenced Sep 14, 2026
`lag`/`lead` take three arguments, and the window `Reduce` encoded all three into a record per input row. The `offset` and `default` arguments are usually plan-time constants, so the arrangement held two constants per row, once for every row of every partition. `AggregateFunc::LagLead` now carries the pair in `constant_args` when both are literals, and the per-row encoded argument is the bare `value` datum instead of a `(value, offset, default)` record. The evaluation materializes the constant `default` into the arena once per partition and shares it across the rows. Hoisting is all-or-nothing per call, because the encoded-argument shape is either the full record or the bare value. A per-row `offset` or `default` keeps all three arguments in the record. Fused value window functions decide per constituent call, so one `lag` in a fused group can hoist while another does not. `EXPLAIN` prints the hoisted arguments inside the function's brackets, for example `lag[offset=1, default=null, order_by=[...]]`, so the plan still fully describes the call. They are literals and are redacted like any other. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A window function's per-row value carries an `OriginalRow` record holding every input column, because the reduce's output has to reconstruct the input row. A `lag`/`lead` whose `value` argument is one of those columns packed a second copy of it next to the record, once for every row of every partition. `LagLeadArgs` (the description of the arguments the plan holds rather than encodes, formerly `ConstantLagLeadArgs`) gains a `value` field naming the `OriginalRow` field to read the argument from. That leaves three shapes for the per-row encoded arguments, and the description says which one a call uses: the full `(value, offset, default)` record, the bare `value` datum, or nothing at all. With nothing to encode, the `(OriginalRow, EncodedArgs)` record keeps only its first field, and a fused constituent contributes no field to the fused argument record, which is itself absent when no constituent encodes anything. The `value` field is set during lowering rather than in HIR, because whether the argument is an input column is only apparent in its MIR form, and the field index is a MIR column index. Lowering also bounds it by the window function's input arity: lowering an argument can map further columns onto the input, for a subquery in an argument, and those are not in `OriginalRow`. Reading the value back walks `OriginalRow` as far as that field, so this trades a datum walk bounded by the field index for not storing the column twice. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
djahandarie
force-pushed
the
dj/window-lag-lead-arg-from-original-row
branch
from
September 14, 2026 21:04
0d891c9 to
eb25bf4
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
Stacked on #38851; review that one first.
The window reduce's per-row value is
row(row(OriginalRow, EncodedArgs), OrderByExprs...).OriginalRowis a recordof every one of the window function's input columns, and it has to be: the
reduce's output reconstructs the input row from it. So a
lag/leadwhosevalueargument is one of those columns stores that column twice in every rowthe reduce arranges:
The user-facing case that prompted this is four fused
lagcalls over onepartition, each on a plain column, so four wide columns were stored twice per
row across a 1.5 billion row input.
Description
LagLeadArgs, the description of the arguments the plan holds rather thanencodes per row, gains a
value: Option<usize>naming theOriginalRowfieldto read the argument from. It is renamed from
ConstantLagLeadArgsbecause itno longer describes only constants, and
AggregateFunc::LagLead's field isrenamed from
constant_argstoargsto match.That leaves exactly three shapes for the per-row encoded arguments, and the
description says which one a call uses:
argsNone(value, offset, default)recordSome,value: NonevaluedatumSome,value: Some(k)With nothing to encode the
(OriginalRow, EncodedArgs)record keeps only itsfirst field, so the plan reads
A fused constituent that encodes nothing contributes no field to the fused
argument record, and that record is absent altogether when no constituent
encodes anything.
AggregateFunc::encodes_window_argsis the one predicate thatanswers "does this constituent contribute a field", and the eval, the output
type computation and
on_uniqueall walk the record againstfuncsthrough itrather than zipping the two.
The
valuefield is set in lowering, not in HIR, for two reasons: whether theargument is an input column is only apparent in its MIR form, and the field
index is a MIR column index.
Self::describe_window_argsdoes that, and boundsthe index by the window function's input arity, because lowering an argument can
map further columns onto the input (a subquery in an argument) and those are not
in
OriginalRow.Only
lag/leadwith both other arguments already described take part. Keepingthe shape count at three is deliberate: a per-row
offsetwould otherwise add afourth,
(offset, default). The same redundancy exists forfirst_value,last_valueand window aggregates, whose single argument is also usually aplain column (
window_agg(row(row(row(#0{x}, #1{y}), #0{x})))); extending thedescription to them is a follow-up, not in this PR.
Costs
Reading the value back walks
OriginalRowas far as the named field, so thistrades a datum walk bounded by the field index, once per row per call, for not
storing the column twice. That is the intended direction for a non-incremental
window function, whose dominant cost is already the per-partition sort, and it
is the direction the reported workload needs.
Tests
## lag/lead argument encodingsection intest/sqllogictest/window_funcs.sltnow pins one plan per shape: a describedvalue, an expressionvaluethat stays encoded while the constants aredescribed, all three encoded, a fused pair where one call encodes and the
other does not, and a fused pair where neither does and no argument record is
emitted.
lag/leadcoverage exercises each shape end to end,through both the reduce and the
on_uniquepath.EXPLAINgoldens intest/sqllogictest/explain/are updated, redactedvariants included.
The LIR schema snapshot is regenerated in place;
LIR_VERSIONstays at 1,which is unshipped.
No user-visible behavior change.
🤖 Generated with Claude Code