bench: SQL benchmark suite for null-aware (NOT IN) joins

[adriangb]

Which issue does this PR close?

• N/A. This PR adds benchmarks only. It is split out of fix: correlated NOT IN with a non-equality correlation returns wrong results #25339 so that the suite is on main first, and that PR can then be measured against it.

Rationale for this change

A NOT IN subquery becomes a null-aware join. An outer row that finds no match is TRUE only when neither side has a NULL in scope. If a NULL is in scope, the result is UNKNOWN.

This decision is cheap for an uncorrelated NOT IN. For a correlated NOT IN, the correlation predicate stays behind as a join filter. The join must then evaluate that filter for each candidate (build row x probe row) pair, to find which rows the NULLs reach. A non-equality correlation gives no equality key, so there is no scope key to reduce the number of pairs. The cost then grows with the NULL count multiplied by the size of the opposite table.

No benchmark measured this shape, so there was no way to see the cost, or to tell a change from noise. Review on #25339 asked for this benchmark.

These are the measured results for #25339. Each number is the median of 60 iterations, taken as 6 interleaved rounds of 10 iterations on an Apple M4 Pro in release mode. The two sides are the base commit of #25339 and its head commit, each with this suite applied, so the comparison isolates the change in that PR.

Query	Shape	base	#25339
Q01	uncorrelated, non-nullable keys	17.9 ms	17.7 ms	0.99x
Q02	uncorrelated, 1% NULL subquery side	14.9 ms	14.9 ms	1.00x
Q03	uncorrelated, 50% NULL outer side	14.6 ms	14.6 ms	1.00x
Q04	correlated, nullable keys, no NULL present	0.9 ms	0.9 ms	0.96x
Q05	correlated, 1% NULL outer side	0.9 ms	2.9 ms	3.1x
Q06	correlated, 50% NULL outer side	0.9 ms	96.2 ms	109x
Q07	correlated, 50% NULL subquery side	0.8 ms	94.4 ms	112x
Q08	as Q06, with an equality correlation	1.1 ms	15.1 ms	14x

Q01 to Q04 are the comparable rows, and they show no change. The base gives wrong results for Q05 to Q08, which is the bug that #25339 corrects, so those four rows show the cost of correct results, not a regression.

Q06 and Q07 are the rows that the review of #25339 asked about. They also give the baseline to measure any later optimization of that path against. Q08 has the same NULL fraction as Q06 and is 6 times cheaper, which is the value of the equality correlation.

What changes are included in this PR?

A null_aware_join SQL benchmark suite. There are no Rust changes. The runner finds suites in benchmarks/sql_benchmarks/, and the load SQL makes each table from range(), so there is no data generation step.

• Q01 to Q03 are uncorrelated NOT IN at different NULL fractions. Their cost is linear with the table size. They are the regression guard for the plain null-aware path.
• Q04 is the correlated shape with nullable keys that hold no NULL. It separates the baseline cost of the shape from the per-pair filter work.
• Q05 to Q07 are the same correlation at 1% and 50% NULL on each side. This is where that work becomes visible.
• Q08 has the same NULL fraction as Q06, but adds an equality correlation. The candidate pairs then come from a hash lookup. The difference between Q06 and Q08 shows the value of the scope key.

Both table sizes are knobs. NAJ_ROWS (default 10000) sets the size for the correlated queries, whose cost grows with its square. NAJ_LARGE_ROWS (default 1000000) sets the size for the uncorrelated queries.

./bench.sh run null_aware_join

# One query, with more rows for the correlated shape
NAJ_ROWS=20000 ./bench.sh run null_aware_join 6

This PR also adds the suite to bench.sh and documents it in benchmarks/README.md and benchmarks/sql_benchmarks/README.md.

What is the testing strategy for this PR?

This PR adds benchmarks, so it adds no new tests. The existing checked_in_suites_cover_benchmark_directories test in benchmarks/src/sql_benchmark_suite.rs covers suite discovery, and it passes with the new directory. All eight queries were run on this branch. Each one asserts HashJoinExec in its plan through the expect_plan directive.

Each query also runs on main as written. Q08 uses the mark join form on purpose. The plain WHERE ... NOT IN form with an equality correlation does not plan on main, and a query that runs on only one branch cannot compare two branches.

Are there any user-facing changes?

No. This PR changes benchmarks and documentation only. It does not change library code.

🤖 Generated with Claude Code

[codecov-commenter]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 81.93%. Comparing base (6ff484a) to head (b82caf6).
⚠️ Report is 2 commits behind head on main.

Additional details and impacted files

@@           Coverage Diff            @@
##             main   #25386    +/-   ##
========================================
  Coverage   81.93%   81.93%            
========================================
  Files        1136     1136            
  Lines      428859   429152   +293     
  Branches   428859   429152   +293     
========================================
+ Hits       351376   351628   +252     
- Misses      56462    56479    +17     
- Partials    21021    21045    +24     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[kosiew]

@adriangb,

Thanks for adding this benchmark coverage. The new suite does a nice job covering both correlated and uncorrelated NOT IN shapes. I found one small integration issue with the default all benchmark path, but it is non-blocking.

[kosiew]

Small integration nit: all(default) is documented as running all benchmarks, but the all branch does not currently call run_null_aware_join, so the new regression benchmark gets silently skipped in the default aggregate run. Could we add run_null_aware_join here? If the default runtime makes it unsuitable for all, it would be good to explicitly document that instead.

[jayzhan211]

Thanks @adriangb , some non blocking suggestions

[jayzhan211]

expect_plan matches the {:#?} output, and HashJoinExec's Debug impl doesn't print null_aware, so this guard also passes for a plain (non-null-aware) join — Q08 on main is exactly that. Worth pinning, since the whole suite is about that flag:

             .field("mode", &self.mode)
+            .field("null_aware", &self.null_aware)
             .field("metrics", &self.metrics)

then on Q02–Q08:

-expect_plan HashJoinExec
+expect_plan null_aware: true

Fine as a follow-up

[jayzhan211]

On main this suite measures wrong answers for Q05–Q08, so the "base" column in the table is the cost of skipping the work, not a slower/faster comparison. Checked against DuckDB at default sizes:

Query	correct	main
Q05	7460	7450
Q06	5010	5000
Q07	10	0
Q08	5530	10000

parquet_row_filter_skip already uses assert as a correctness canary for the same reason. Please add one per query (in #25339 if they need to stay red on main until it lands), and note in this PR's description that the base numbers for Q05–Q08 come from incorrect results. Example for Q07:

 expect_plan HashJoinExec

+assert I
+SELECT count(*)
+FROM small_outer o
+WHERE o.id_n0 NOT IN (SELECT i.id_n50 FROM small_inner i WHERE i.z < o.z);
+----
+10
+
 run

Fine as a follow-up. We can revisit the result later to figure out whether there's a bug in DataFusion, or whether the expected result legitimately differs from DuckDB.