duckdb: disable function serialization - #9891
Conversation
Signed-off-by: Mikhail Kot <mikhail@spiraldb.com>
|
This is the same solution as @joseph-isaacs's https://github.com/vortex-data/vortex/pull/9802/changes, albeit with a different comment. |
Merging this PR will degrade performance by 10.06%
|
| Mode | Benchmark | BASE |
HEAD |
Efficiency | |
|---|---|---|---|---|---|
| ❌ | WallTime | arrow_checked_add_u32_neon[16384] |
13.5 µs | 20.4 µs | -33.72% |
| ❌ | Simulation | random_i8[0.5] |
71.3 µs | 94.9 µs | -24.83% |
| ❌ | Simulation | decompress[u64, (4000, 1024)] |
71.5 µs | 86.8 µs | -17.56% |
| ❌ | WallTime | deferred_i64_avx2[PerRowPerRow] |
9.9 µs | 11.5 µs | -13.87% |
| ⚡ | Simulation | random_i16[0.8] |
96.4 µs | 78.1 µs | +23.5% |
| ⚡ | WallTime | arrow_checked_add_u32_avx2[16384] |
21.4 µs | 17.6 µs | +21.2% |
Tip
Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.
Comparing myrrc/duckdb-no-serialize (68fb80d) with develop (b5f43ba)
Footnotes
-
218 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩
Polar Signals Profiling ResultsLatest Run
Powered by Polar Signals Cloud |
Benchmarks: PolarSignals Profiling 📖Commits: PR datafusion / vortex-file-compressed / ns (1.006x ➖, 0↑ 0↓)
No file size changes detected. |
Benchmarks: TPC-H SF=1 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.007x ➖, 0↑ 0↓)
datafusion / parquet / ns (1.009x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.007x ➖, 0↑ 0↓)
duckdb / parquet / ns (0.998x ➖, 0↑ 0↓)
No file size changes detected. |
Benchmarks: FineWeb NVMe 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.013x ➖, 0↑ 0↓)
datafusion / parquet / ns (1.013x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.998x ➖, 2↑ 1↓)
duckdb / parquet / ns (1.016x ➖, 0↑ 0↓)
No file size changes detected. |
Benchmarks: Clickbench Sorted on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.996x ➖, 0↑ 0↓)
datafusion / parquet / ns (1.015x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.995x ➖, 1↑ 0↓)
duckdb / parquet / ns (1.023x ➖, 0↑ 0↓)
File Size Changes (100 files changed, -0.0% overall, 45↑ 55↓)
Totals:
|
Benchmarks: TPC-H SF=10 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.019x ➖, 0↑ 1↓)
datafusion / parquet / ns (1.006x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.024x ➖, 0↑ 1↓)
duckdb / parquet / ns (1.010x ➖, 0↑ 0↓)
No file size changes detected. |
Benchmarks: Clickbench on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.018x ➖, 0↑ 0↓)
datafusion / parquet / ns (1.010x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.016x ➖, 1↑ 2↓)
duckdb / parquet / ns (1.029x ➖, 0↑ 2↓)
No file size changes detected. |
Benchmarks: TPC-DS SF=1 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.005x ➖, 1↑ 2↓)
datafusion / parquet / ns (0.999x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.010x ➖, 2↑ 7↓)
duckdb / parquet / ns (1.012x ➖, 3↑ 3↓)
No file size changes detected. |
Benchmarks: FineWeb S3 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.082x ➖, 2↑ 2↓)
datafusion / parquet / ns (1.144x ➖, 0↑ 2↓)
duckdb / vortex-file-compressed / ns (1.094x ➖, 0↑ 2↓)
duckdb / parquet / ns (1.113x ➖, 0↑ 1↓)
|
Benchmarks: Statistical and Population Genetics 📖Commits: PR How to read Verdict and Engines
duckdb / vortex-file-compressed / ns (1.003x ➖, 0↑ 0↓)
duckdb / parquet / ns (1.015x ➖, 0↑ 0↓)
No file size changes detected. |
Benchmarks: TPC-H SF=1 on S3 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.009x ➖, 1↑ 2↓)
datafusion / parquet / ns (1.076x ➖, 0↑ 1↓)
duckdb / vortex-file-compressed / ns (1.078x ➖, 0↑ 3↓)
duckdb / parquet / ns (1.064x ➖, 0↑ 0↓)
|
Duckdb serializes internal function state only if you don't add
serialize/deserialize callbacks. This is wrong if filters are pushed down.
Tpch q11 plan was incorrect due to that, with 6 different scans replaced by one.
Disable function serialization with a hack (verify_serialization = false).
This is an upstream bug which is reported to duckdb:
duckdb/duckdb#25740