fix: correlated exists/not exists subqueries hit the count bug for groupless aggregates - #25391
mohammadnaqvi04 wants to merge 7 commits into
Conversation
9e9e5d9 to
d3bb21f
Compare
| /// Builds the join for a correlated `EXISTS` subquery whose groupless | ||
| /// aggregate requires count-bug compensation. |
There was a problem hiding this comment.
This decorrelates a groupless-aggregate subquery to a LEFT JOIN, grouped by the correlated column, with the unmatched-row NULL replaced by the aggregate's empty-input default (e.g. count(*) → 0, sum(x) -> NULL).
For
SELECT t1.t1_int FROM t1 WHERE EXISTS (
SELECT count(*) FROM t2 WHERE t1.t1_int = t2.t2_int
);this is equivalent to
SELECT t1.t1_int
FROM t1
LEFT JOIN (SELECT t2_int, count(*) AS cnt FROM t2 GROUP BY t2_int) AS sq
ON t1.t1_int = sq.t2_int;| // Correlated exist subquery, remove the limit(so that correlated expressions can pull up) | ||
| (true, false) => Transformed::yes(match limit.get_fetch_type()? { | ||
| FetchType::Literal(Some(0)) => { | ||
| self.forces_empty_result = true; |
There was a problem hiding this comment.
LIMIT 0 on the subquery collapses it to an EmptyRelation before the count-bug join ever sees it, so without this flag, join-compensation has no way of knowing that the subquery is unconditionally empty and defaults every row to matched. For
SELECT t1.t1_int FROM t1 WHERE EXISTS (
SELECT count(*) FROM t2 WHERE t1.t1_int = t2.t2_int LIMIT 0
);t1_int should never appear in the result, since LIMIT 0 empties the subquery regardless of whether t2 has a matching row. Setting forces_empty_result is what makes build_join_with_count_bug return false unconditionally instead of falling back to its usual "matched" default.
Which issue does this PR close?
EXISTS/INsubqueries with groupless aggregates hit the count bug #24960.Rationale for this change
See #24960.
This is a split of #25008, which originally covered
EXISTS/NOT EXISTS/IN/NOT INtogether. That PR has been split into two smaller PRs between this one forEXISTS/NOT EXISTSand a (to come) follow-up stacked on top of it forIN/NOT IN.What changes are included in this PR?
Correlated
EXISTS/NOT EXISTSsubqueries had count-bug compensation available but never turned it on.EXISTS/NOT EXISTSsubqueries.LEFT JOINand substitute the correct value instead of dropping the row.IN/NOT INcompensation is out of scope for this PR and returns an explicit "not implemented" error. This will be covered by the follow-up PR.What is the testing strategy for this PR?
Added twenty sqllogictest cases in
subquery.slt.Ran all
datafusion,datafusion-cli, anddatafusion-sqllogictestsuites, and./dev/rust_lint.sh.Also differentially fuzz-tested against DuckDB with a standalone harness, and did several adversarial review passes with Claude to look for gaps beyond the fuzzer's generated shapes.
Are there any user-facing changes?
None, except in the sense that this fixes a user-visible bug.
AI Usage
AI was used during development of this PR, primarily to help review the test suite and verify test cases against the implemented fix.