Skip to content

feat(pwmj): support RightSemi/RightAnti existence joins - #24457

Open
SubhamSinghal wants to merge 3 commits into
apache:mainfrom
SubhamSinghal:pwmj-right-existence-joins
Open

feat(pwmj): support RightSemi/RightAnti existence joins#24457
SubhamSinghal wants to merge 3 commits into
apache:mainfrom
SubhamSinghal:pwmj-right-existence-joins

Conversation

@SubhamSinghal

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Part of #17427 (the PiecewiseMergeJoin epic). This is the second half of its existence-join item: #23870 added LeftSemi/LeftAnti, this adds RightSemi/RightAnti. Only the Mark joins remain unsupported.

Rationale for this change

A range-predicate right semi/anti join — ... RIGHT SEMI JOIN r ON l.v > r.v, or any query that decorrelates to one — currently falls back to NestedLoopJoinExec.

What changes are included in this PR?

New RightExistencePWMJStream (right_existence_join.rs), a 3-state machine: await the buffered reduction, then per streamed batch evaluate the key, compare it against the reduced key with a vectorized cmp kernel, and filter. Plus:

  • exec.rs — dispatch; build_buffered_extreme, which folds every buffered partition on its own SpawnedTask; BufferedExtreme as the shared state; relaxed ordering, distribution and partitioning declarations for these two join types; removal of the dead sort_options flip and the unimplemented!() placeholder that the swap design had left behind.
  • physical_planner.rs — the gate now excludes only the Mark joins.
  • utils.rsis_supported_existence_join widened; new is_supported_right_existence_join (which deliberately excludes RightMark, so a Mark join cannot inherit the relaxed input requirements if that gate is ever loosened); no visited-indices bitmap allocated for these join types.
  • extreme_key made pub(super) so both existence streams share one implementation.

Are these changes tested?

Yes

Benchmark

Microbenchmark, 20K × 20K rows, single inequality, enable_piecewise_merge_join on vs off

Case PWMJ NestedLoopJoin Speedup
RightSemi, all match ~0.023 ms ~77 ms ~3,300×
RightAnti, all match ~0.024 ms ~77 ms ~3,200×
RightSemi, no match ~0.023 ms ~78 ms ~3,300×
RightAnti, no match ~0.024 ms ~78 ms ~3,300×
RightSemi, half match ~0.036 ms ~78 ms ~2,200×
RightAnti, half match ~0.036 ms ~77 ms ~2,100×

Are there any user-facing changes?

No

@SubhamSinghal
SubhamSinghal marked this pull request as ready for review August 18, 2026 12:57
@github-actions github-actions Bot added core Core DataFusion crate sqllogictest SQL Logic Tests (.slt) physical-plan Changes to the physical-plan crate labels Aug 18, 2026
@SubhamSinghal

Copy link
Copy Markdown
Contributor Author

benchmark PR: #24458

@codecov-commenter

codecov-commenter commented Aug 18, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 87.98220% with 81 lines in your changes missing coverage. Please review.
✅ Project coverage is 81.37%. Comparing base (a6e2d3f) to head (0b67be2).

Files with missing lines Patch % Lines
...joins/piecewise_merge_join/right_existence_join.rs 88.08% 20 Missing and 36 partials ⚠️
...ysical-plan/src/joins/piecewise_merge_join/exec.rs 86.97% 6 Missing and 19 partials ⚠️
Additional details and impacted files
@@           Coverage Diff            @@
##             main   #24457    +/-   ##
========================================
  Coverage   81.36%   81.37%            
========================================
  Files        1117     1118     +1     
  Lines      397872   398477   +605     
  Branches   397872   398477   +605     
========================================
+ Hits       323725   324253   +528     
- Misses      55229    55251    +22     
- Partials    18918    18973    +55     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@SubhamSinghal

Copy link
Copy Markdown
Contributor Author

@comphead @kumarUjjawal PR for right semi/anti join. Please help in reviewing this PR. Thanks.

@comphead

Copy link
Copy Markdown
Contributor

Thanks @SubhamSinghal I'll check this today

@comphead

Copy link
Copy Markdown
Contributor

@SubhamSinghal sorry for the delay, last time we got some correctness issues for PWJM and Im planning to merge first #24390 to create a generic join test set and we can check this PR against it

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

core Core DataFusion crate physical-plan Changes to the physical-plan crate sqllogictest SQL Logic Tests (.slt)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants