Skip to content

fix: correlated exists/not exists subqueries hit the count bug for groupless aggregates - #25391

Open
mohammadnaqvi04 wants to merge 7 commits into
apache:mainfrom
mohammadnaqvi04:exists-not-exists-groupless-agg
Open

mohammadnaqvi04 wants to merge 7 commits into
apache:mainfrom
mohammadnaqvi04:exists-not-exists-groupless-agg

Conversation

@mohammadnaqvi04

Copy link
Copy Markdown

Which issue does this PR close?

Rationale for this change

See #24960.

This is a split of #25008, which originally covered EXISTS/NOT EXISTS/IN/NOT IN together. That PR has been split into two smaller PRs between this one for EXISTS/NOT EXISTS and a (to come) follow-up stacked on top of it for IN/NOT IN.

What changes are included in this PR?

Correlated EXISTS/NOT EXISTS subqueries had count-bug compensation available but never turned it on.

  • Turn on count-bug compensation for correlated EXISTS/NOT EXISTS subqueries.
  • When compensation is needed, decorrelate into a LEFT JOIN and substitute the correct value instead of dropping the row.
  • Make unsupported join shapes fail loudly instead of silently falling back to incorrect behavior.

IN/NOT IN compensation is out of scope for this PR and returns an explicit "not implemented" error. This will be covered by the follow-up PR.

What is the testing strategy for this PR?

Added twenty sqllogictest cases in subquery.slt.

Ran all datafusion, datafusion-cli, and datafusion-sqllogictest suites, and ./dev/rust_lint.sh.

Also differentially fuzz-tested against DuckDB with a standalone harness, and did several adversarial review passes with Claude to look for gaps beyond the fuzzer's generated shapes.

Are there any user-facing changes?

None, except in the sense that this fixes a user-visible bug.

AI Usage

AI was used during development of this PR, primarily to help review the test suite and verify test cases against the implemented fix.

@github-actions github-actions Bot added logical-expr Logical plan and expressions optimizer Optimizer rules sqllogictest SQL Logic Tests (.slt) labels Sep 16, 2026
@mohammadnaqvi04
mohammadnaqvi04 force-pushed the exists-not-exists-groupless-agg branch from 9e9e5d9 to d3bb21f Compare September 16, 2026 23:29
Comment on lines +662 to +663
/// Builds the join for a correlated `EXISTS` subquery whose groupless
/// aggregate requires count-bug compensation.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This decorrelates a groupless-aggregate subquery to a LEFT JOIN, grouped by the correlated column, with the unmatched-row NULL replaced by the aggregate's empty-input default (e.g. count(*)0, sum(x) -> NULL).
For

SELECT t1.t1_int FROM t1 WHERE EXISTS (
    SELECT count(*) FROM t2 WHERE t1.t1_int = t2.t2_int
);

this is equivalent to

SELECT t1.t1_int
FROM t1
LEFT JOIN (SELECT t2_int, count(*) AS cnt FROM t2 GROUP BY t2_int) AS sq
  ON t1.t1_int = sq.t2_int;

// Correlated exist subquery, remove the limit(so that correlated expressions can pull up)
(true, false) => Transformed::yes(match limit.get_fetch_type()? {
FetchType::Literal(Some(0)) => {
self.forces_empty_result = true;

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LIMIT 0 on the subquery collapses it to an EmptyRelation before the count-bug join ever sees it, so without this flag, join-compensation has no way of knowing that the subquery is unconditionally empty and defaults every row to matched. For

SELECT t1.t1_int FROM t1 WHERE EXISTS (
    SELECT count(*) FROM t2 WHERE t1.t1_int = t2.t2_int LIMIT 0
);

t1_int should never appear in the result, since LIMIT 0 empties the subquery regardless of whether t2 has a matching row. Setting forces_empty_result is what makes build_join_with_count_bug return false unconditionally instead of falling back to its usual "matched" default.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

logical-expr Logical plan and expressions optimizer Optimizer rules sqllogictest SQL Logic Tests (.slt)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant