Push date bounds that DuckDB widened to TIMESTAMP - #9869
Draft
joseph-isaacs wants to merge 3 commits into
Draft
joseph-isaacs wants to merge 3 commits into
joseph-isaacs wants to merge 3 commits into
Conversation
`date '1993-07-01' + interval '3' month` returns a TIMESTAMP, so DuckDB widens the column side to match and the predicate binds as `CAST(o_orderdate AS TIMESTAMP) < TIMESTAMP '1993-10-01 00:00:00'`. The cast hides the column reference, so `can_push_expression` refused it and the bound stayed in a FILTER above the scan. TPC-H q4, q15 and q20 each pushed only their lower date bound, leaving the scan to emit 77%, 42% and 71% of the table where the full range keeps 3.8%, 3.6% and 14.5%. Fold the cast into the literal instead of evaluating it. The cast is strictly increasing, so the comparison has an exact DATE equivalent: at midnight the operator carries over unchanged, and strictly inside a day both `<` and `<=` admit that day while both `>` and `>=` exclude it. Keeping the predicate as `column <op> literal` is the point -- that form prunes with statistics, which evaluating a cast per batch would not. Deliberately left alone: TIMESTAMP WITH TIME ZONE, whose cast depends on the session timezone; TRY_CAST, which yields NULL where CAST errors; and `=`/`<>` against a bound inside a day, where folding to a constant would drop the null cases. `can_push_cast` is unchanged, so a bare cast still does not push. Only the comparison shape is rewritten. Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk> Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HdDxvVPLxepaFXT3su3tVm
Merging this PR will regress 1 benchmark
|
| Mode | Benchmark | BASE |
HEAD |
Efficiency | |
|---|---|---|---|---|---|
| ❌ | WallTime | filtered_owned_i64_avx512[OneNullInEight] |
23.8 µs | 26.5 µs | -10.16% |
| ⚡ | Simulation | decompress[u64, (4000, 1024)] |
86.9 µs | 71.5 µs | +21.57% |
Tip
Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.
Comparing ji/friendly-edison-kvo62b (68b5a7e) with develop (7ecae9e)
Footnotes
-
218 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩
Benchmarking the fold at sf=1 showed q15 and q20 improving but q4 regressing 1.46x. The cause is not the scan: once the upper bound pushes, `orders` has no filter left above it, so DuckDB's Deliminator collapses the delim join into a plain semi-join. That is the regression `pushdown_complex_filter` already documents for equality filters (duckdb/duckdb#22669); the existing hack just did not cover a `Lt`. Report a folded date bound as not pushed for the same reason. DuckDB then keeps its own copy of the predicate and its plan shape, while the bound still runs inside the scan, which is where the pruning comes from. Measured at sf=1, vortex-file-compressed, median of 10 iterations, 3 interleaved rounds against the same baseline: q4 97.1 -> 88.2 ms (0.91x, was 1.46x before this commit) q15 55.7 -> 42.5 ms (0.76x) q20 106.5 -> 80.2 ms (0.75x) The other 19 queries stay within run-to-run noise and the total moves 0.99x, which matches the plan regeneration touching only these three. Also drop the plan assertions from the e2e test. On a small single-table scan DuckDB converts these bounds into table filters itself without reaching the fold, so those assertions passed whether or not the fold existed. The test now pins semantics against native DuckDB as the oracle, and the checked-in TPC-H plans carry the proof that the bound reaches the scan. Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk> Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HdDxvVPLxepaFXT3su3tVm
joseph-isaacs
added
the
action/bench-sql
github-actions
Bot
removed
the
action/bench-sql
Contributor
Polar Signals Profiling ResultsLatest Run
Powered by Polar Signals Cloud |
Contributor
Benchmarks: PolarSignals Profiling 📖Commits: PR datafusion / vortex-file-compressed / ns (0.997x ➖, 0↑ 0↓)
No file size changes detected. |
Contributor
Benchmarks: TPC-H SF=1 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.001x ➖, 0↑ 0↓)
datafusion / parquet / ns (1.000x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.998x ➖, 1↑ 1↓)
duckdb / parquet / ns (0.999x ➖, 1↑ 0↓)
No file size changes detected. |
Contributor
Benchmarks: FineWeb NVMe 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.012x ➖, 0↑ 0↓)
datafusion / parquet / ns (1.007x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.982x ➖, 1↑ 3↓)
duckdb / parquet / ns (1.011x ➖, 0↑ 0↓)
No file size changes detected. |
Contributor
Benchmarks: Clickbench Sorted on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.006x ➖, 0↑ 0↓)
datafusion / parquet / ns (0.989x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.979x ➖, 1↑ 1↓)
duckdb / parquet / ns (0.993x ➖, 0↑ 0↓)
File Size Changes (100 files changed, -0.0% overall, 47↑ 53↓)
Totals:
|
Contributor
Benchmarks: FineWeb S3 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.986x ➖, 0↑ 0↓)
datafusion / parquet / ns (1.374x ❌, 0↑ 7↓)
duckdb / vortex-file-compressed / ns (1.115x ➖, 0↑ 1↓)
duckdb / parquet / ns (1.113x ➖, 0↑ 1↓)
|
Contributor
Benchmarks: TPC-H SF=10 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.001x ➖, 0↑ 0↓)
datafusion / parquet / ns (1.003x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.986x ➖, 1↑ 0↓)
duckdb / parquet / ns (1.000x ➖, 0↑ 0↓)
No file size changes detected. |
Contributor
Benchmarks: Clickbench on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.006x ➖, 0↑ 0↓)
datafusion / parquet / ns (1.026x ➖, 0↑ 1↓)
duckdb / vortex-file-compressed / ns (0.989x ➖, 2↑ 1↓)
duckdb / parquet / ns (0.986x ➖, 2↑ 0↓)
No file size changes detected. |
Contributor
Benchmarks: TPC-DS SF=1 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.996x ➖, 1↑ 0↓)
datafusion / parquet / ns (0.999x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.991x ➖, 3↑ 1↓)
duckdb / parquet / ns (0.995x ➖, 1↑ 3↓)
No file size changes detected. |
Contributor
Benchmarks: Statistical and Population Genetics 📖Commits: PR How to read Verdict and Engines
duckdb / vortex-file-compressed / ns (1.004x ➖, 0↑ 0↓)
duckdb / parquet / ns (0.998x ➖, 0↑ 0↓)
No file size changes detected. |
Contributor
Benchmarks: TPC-H SF=1 on S3 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.155x ➖, 1↑ 7↓)
datafusion / parquet / ns (1.069x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.154x ➖, 0↑ 7↓)
duckdb / parquet / ns (1.081x ➖, 0↑ 0↓)
|
`slt/duckdb/cast_pushdown.slt` asserted a FILTER remains above the scan for `d < DATE '1993-07-01' + INTERVAL '3' MONTH`. That was the limitation this branch removes, so the assertion inverts. CI caught this and I did not: the earlier runs here filtered the suite to `-- tpch`, which never loaded the file. The full suite is 1 of 8394 tests, and this was it. Add the matching negative case while here: under a non-UTC session timezone a TIMESTAMPTZ bound must stay above the scan, and still return the same rows. That pairs a real exclusion check with the positive one, which the timezone cases in the Rust test could not do on their own. Correct the claim in date_pushdown_test's module docs that a plan assertion on a single-table scan would be vacuous. It is not: the FILTER node is present before this branch and absent after, which is exactly what cast_pushdown.slt keys on. The Rust test keeps the operator and boundary matrix against native DuckDB; the plan assertions live in the slt files, and the docs now say so. Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk> Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HdDxvVPLxepaFXT3su3tVm
joseph-isaacs
marked this pull request as draft
September 15, 2026 08:45
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
TPC-H q4, q15 and q20 each state their date window as
col >= date 'X' and col < date 'X' + interval 'N' unit. Only the lower bound reached the Vortex scan; the upper bound stayed in a DuckDBFILTERabove it, so the scan emitted roughly 77%, 42% and 71% of the table where the full range keeps about 3.8%, 3.6% and 14.5%.The cause is not the interval — DuckDB already constant-folds it.
date + intervalreturns aTIMESTAMP, so DuckDB widens the column side to match and the predicate binds as:That cast hides the column reference.
can_push_castaccepts only integer-to-integer casts (the over-restriction #8570 tracks), socan_push_expressionrefused the comparison and DuckDB kept it above the scan.slt/duckdb/cast_pushdown.slthas asserted thatFILTERsince it was written; this PR inverts it.Measured at sf=1 on
vortex-file-compressed: 0.91x on q4, 0.76x on q15, 0.75x on q20, with the other 19 queries flat. Numbers, method, and why the gains are not larger are below.Changes
1. Fold the cast into the literal (
convert/expr.rs).DATEtoTIMESTAMPis strictly increasing, so the comparison has an exactDATEequivalent:< t,<= tdate(t)<= date(t)> t,>= tdate(t)> date(t)= t,<> tdate(t)Folding rather than evaluating keeps the predicate in
column <op> literalform — the form that can drive statistics pruning where the data supports it, and that costs nothing per batch. Evaluating acastper row would recover the rows but neither of those.Deliberately not folded:
TIMESTAMP WITH TIME ZONE— that cast depends on the session timezone, so it has no fixedDATEequivalent.TRY_CAST— yieldsNULLwhereCASTerrors, which the fold would not reproduce.=/<>against a bound inside a day — unsatisfiable and tautological respectively, but only for non-null rows; folding to a constant would drop the null cases.can_push_castitself is unchanged, so a bareCAST(date AS TIMESTAMP)still does not push. This does not resolve #8570; it removes the reason those three queries needed it.2. Withhold the folded bound from the Deliminator (
table_function.rs). The first commit alone made q4 1.46x slower. Not the scan — the plan shape. Once the upper bound pushes,ordershas no filter left above it, so DuckDB's Deliminator collapses the delim join:That is exactly the regression
pushdown_complex_filteralready documents (duckdb/duckdb#22669); its existing hack reports equality filters as not pushed, and simply did not cover aLt. A folded date bound is now reported the same way. DuckDB keeps its own copy of the predicate and its plan shape; the bound still runs inside the scan, which is where the benefit comes from.try_from_expression_inner's comparison arm moved intotry_from_comparisonto stay under the cognitive-complexity lint.Plan effect
Four checked-in expectations change and no others: the TPC-H plans for
q4,q15andq20, plus thedate + intervalcase inslt/duckdb/cast_pushdown.slt. The full workspace run is 1 changed result in 8394 tests.Benchmarks
duckdb-bench tpch --formats vortex --iterations 10 --threads 4, sf=1, all 22 queries, 3 rounds interleaved baseline/candidate on an idle 4-core box. Median per round, in ms.q15 and q20 have non-overlapping spreads across the three rounds; q4's rounds are 97.1/89.8/102.6 against 96.8/88.2/76.6. No other query's spreads separate — the apparent movement on q5, q8 and q17 is run-to-run noise on this box, consistent with the plan regeneration leaving them untouched.
Why q4 gains 9% and not 20x
The scan emits 20x fewer rows, so it is worth saying where that does and does not turn into time. Isolating the
ordersscan at sf=1, release build, median of 9:count(*)sum(o_orderkey)Two things follow.
The scan is not where the time is. The whole
ordersscan is ~8 ms of a ~97 ms q4. q4's other side —lineitemwithl_commitdate < l_receiptdate, 6M rows — is 14.2 ms on its own and 67.4 ms through the semi-join, and this change does not touch it. So roughly 70% of q4 is untouchable here, and the ~9 ms measured gain is about 1.4 ms of scan plus the join building on 57k rows instead of 1.16M. q15 and q20 do better precisely because there the date bound is onlineitemitself.Statistics pruning contributes nothing on this data.
o_orderdateis scattered in file order: the column spans 2405 days, and all 184 of the 8192-row blocks span more than 2000 of them — including the very first. No zone map can exclude a block, so the bound cannot avoid IO, only downstream work. On date-clustered data the same predicate would prune; TPC-H simply is not that data.Folding also costs nothing relative to the best existing path: a hand-written
o_orderdate < DATE '1993-10-01', which has always pushed, measures 4.5 ms / 6.7 ms against the folded form's 4.5 ms / 6.9 ms.Effect on cardinality estimates
Pushing the second bound also changes what the join-order optimizer sees, so here are the scan estimates against the true post-filter counts, at sf=0.1 where the plans are pinned:
developdeveloperrororders, 150,000)lineitem, 600,572)lineitem, 600,572)q4 and q15 improve sharply; q20 moves from mildly over to meaningfully under. The mechanism is that
cardinality()reports a flat0.2 x total, andRelationStatisticsHelper::ExtractGetStatsthen applies DuckDB's ownDEFAULT_SELECTIVITYof 0.2 once a non-optional table filter is present — so the effective estimate is0.04 x total. That is close to a three-month window out of TPC-H's ~6.5-year span, and too small for a one-year window.Worth knowing for anyone tempted to make
cardinality()selectivity-aware from min/max: DuckDB will not meet you halfway.InspectTableFilterreturns early for anyCONSTANT_COMPARISONthat is notCOMPARE_EQUAL, so range filters never consult statistics at all, and the equality path needsGetDistinctCount(), which is 0 for Vortex exactly as it is for Parquet. Andcardinality()only sees the complex filters inbind_data, not the table filters — for these queries that is the upper bound alone, whose selectivity (0.27 / 0.61 / 0.43) is not representative of the pair. Estimating from it would give 1.44x over / 3.23x over / 1.77x under: better on q20, worse on q4 and q15. The lever that would actually help is a distinct-count statistic, which feeds bothInspectTableFilterand the per-columncolumn_distinct_countthe join-order estimator consumes;Stathas no such variant today.Testing
Plan-level, in
slt/duckdb/cast_pushdown.slt: thedate + intervalbound now asserts<!REGEX>:.*FILTER.*(it reaches the scan), and a new case asserts the opposite for aTIMESTAMPTZbound under a non-UTC session timezone — it must stay above the scan and still return the same rows. The TPC-H plans show the folded($.o_orderdate < 1993-10-01)in the scan's own filter list.Result-level, in
vortex-duckdb/src/e2e_test/date_pushdown_test.rs(new): every bound is evaluated against both a Vortex file and a native DuckDB table built from the same rows, so DuckDB is the oracle — each operator at midnight, strictly inside a day, reversed operand order, andTIMESTAMPTZacross four session timezones. Plus unit tests for the fold's boundary table, including pre-epoch flooring and a day count pasti32.Checks run:
cargo clippy -p vortex-duckdb --all-targets --all-features -- -D warnings,cargo +nightly-2026-09-10 fmt -p vortex-duckdb --check,cargo test -p vortex-duckdb --lib(258 passed), and the fullcargo test -p vortex-sqllogictest --test sqllogictests(76 passed, 5 ignored — ClickBench fixtures absent locally). Not run locally: the rest of the workspace, and the TPC-DS suite.License Check and Audit Check (advisories)is red here and also red ondevelop— a RUSTSEC advisory in therustls/reqwesttree. This PR adds no dependencies and does not touchCargo.lock.🤖 Generated with Claude Code
https://claude.ai/code/session_01HdDxvVPLxepaFXT3su3tVm