duckdb: return projection_ids handling - #9887
Conversation
Merging this PR will degrade performance by 16.97%
|
| Mode | Benchmark | BASE |
HEAD |
Efficiency | |
|---|---|---|---|---|---|
| ❌ | WallTime | arrow_checked_add_u32_neon[16384] |
12.3 µs | 20.4 µs | -39.77% |
| ❌ | Simulation | random_i8[0.5] |
70.6 µs | 94.5 µs | -25.31% |
| ❌ | Simulation | decompress[u64, (4000, 1024)] |
72 µs | 87.5 µs | -17.74% |
| ❌ | WallTime | words_gather_scalar_avx2[65536] |
8.3 µs | 9.4 µs | -11.95% |
| ❌ | WallTime | mul_i32_nonnull_avx512 |
7.2 µs | 8 µs | -10.01% |
| ⚡ | Simulation | allocate_drop_arrow[0] |
516.4 ns | 462.3 ns | +11.72% |
Tip
Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.
Comparing myrrc/duckdb-projection-ids (c2a946f) with develop (313bc4f)
Footnotes
-
218 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩
Polar Signals Profiling ResultsLatest Run
Powered by Polar Signals Cloud |
Benchmarks: PolarSignals Profiling 📖Commits: PR datafusion / vortex-file-compressed / ns (1.002x ➖, 0↑ 0↓)
No file size changes detected. |
Benchmarks: FineWeb NVMe 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.014x ➖, 0↑ 0↓)
datafusion / parquet / ns (0.998x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.071x ➖, 2↑ 3↓)
duckdb / parquet / ns (1.010x ➖, 0↑ 1↓)
No file size changes detected. |
Benchmarks: TPC-H SF=1 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.996x ➖, 0↑ 0↓)
datafusion / parquet / ns (0.992x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.981x ➖, 2↑ 0↓)
duckdb / parquet / ns (0.992x ➖, 0↑ 0↓)
No file size changes detected. |
Benchmarks: Clickbench Sorted on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.995x ➖, 0↑ 0↓)
datafusion / parquet / ns (1.013x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.975x ➖, 1↑ 0↓)
duckdb / parquet / ns (1.041x ➖, 0↑ 1↓)
File Size Changes (100 files changed, +0.0% overall, 42↑ 58↓)
Totals:
|
Benchmarks: FineWeb S3 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.097x ➖, 0↑ 2↓)
datafusion / parquet / ns (1.057x ➖, 0↑ 2↓)
duckdb / vortex-file-compressed / ns (0.948x ➖, 0↑ 0↓)
duckdb / parquet / ns (1.160x ➖, 0↑ 3↓)
|
Benchmarks: Clickbench on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.002x ➖, 0↑ 0↓)
datafusion / parquet / ns (1.002x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.007x ➖, 2↑ 3↓)
duckdb / parquet / ns (1.016x ➖, 0↑ 3↓)
No file size changes detected. |
Benchmarks: TPC-H SF=10 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.995x ➖, 0↑ 0↓)
datafusion / parquet / ns (0.989x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.969x ➖, 3↑ 0↓)
duckdb / parquet / ns (0.994x ➖, 0↑ 0↓)
No file size changes detected. |
Benchmarks: TPC-DS SF=1 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.004x ➖, 0↑ 0↓)
datafusion / parquet / ns (1.003x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.003x ➖, 0↑ 0↓)
duckdb / parquet / ns (1.002x ➖, 3↑ 3↓)
No file size changes detected. |
Benchmarks: Statistical and Population Genetics 📖Commits: PR How to read Verdict and Engines
duckdb / vortex-file-compressed / ns (0.998x ➖, 0↑ 0↓)
duckdb / parquet / ns (1.003x ➖, 0↑ 0↓)
No file size changes detected. |
Benchmarks: TPC-H SF=1 on S3 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.055x ➖, 0↑ 1↓)
datafusion / parquet / ns (0.998x ➖, 1↑ 0↓)
duckdb / vortex-file-compressed / ns (1.023x ➖, 0↑ 0↓)
duckdb / parquet / ns (1.110x ➖, 0↑ 1↓)
|
Pull request was converted to draft
joseph-isaacs
left a comment
There was a problem hiding this comment.
🤞 Can we have some slt tests?
1ef5527 to
ad71152
Compare
I think this is by far most tested part of our code since nearly every test in SLT and in benchmarks references only some columns from underlying tables already, and in most queries we have some columns which are filter-only |
#9383 removed handling of
projection_ids, since MultiFileReader handles them internally.
However, if there's a table filter in get.table_filters which is passed to
Vortex and this filter contains a column that's not part of projection, this
column will stay in column_ids, and Vortex will load and canonicalize it which
is extra work.
Main reason for this change is the removal for the "Eq not pushed" hack
up the stack - if we save some filters in table filters, we don't want
to project these filter-only columns still in column_ids.