perf: bounded distinct count optimization

[HairstonE]

Which issue does this PR close?

• Closes Add bounded distinct count optimization #21051.

Rationale for this change

Distinct accumulates all the values and then counts them, even if the
threshold is met it continues through the rest of the dataset.

Right now count(distinct) doesn't go to the distinct accumulator, DF puts two
aggregates on top of each other (a group by and then a count of the groups).

Measured on a 50M-row memory table (same binary, toggled via
SET datafusion.optimizer.enable_distinct_aggregation_soft_limit):

Query	Optimization on	Optimization off
count(DISTINCT col) > 1, 5 distinct values	0.007–0.012 s	0.16–0.37 s
count(DISTINCT col) > 1, 50M distinct values	0.006–0.013 s	4.0–4.7 s

What changes are included in this PR?

SELECT count(DISTINCT col) > 1 FROM table type queries are when this code
runs. Take the constant value + 2, one extra for the NULL case and another for
clearing the threshold. When we are counting the groups we track the count
until n + 2, if we hit that number we short circuit. This short circuit is
checked after each batch in Partial.

HAVING was left out for simplicity.

Both the existing LIMIT pushdown and the new comparison pattern
do the same thing so they share the flag
(enable_distinct_aggregation_soft_limit).

What is the testing strategy for this PR?

Yes.

• count_distinct_comparison_sets_soft_limit
• count_distinct_escaping_count_prevents_soft_limit
• count_distinct_comparison_capped_matches_uncapped
• existing limited_distinct_aggregation tests
• sqllogictests: aggregate, limit, count, explain

Are there any user-facing changes?

No user facing changes

[codecov-commenter]

Codecov Report

❌ Patch coverage is 78.12500% with 14 lines in your changes missing coverage. Please review.
✅ Project coverage is 81.93%. Comparing base (b0b5471) to head (81b929d).
⚠️ Report is 4 commits behind head on main.

Files with missing lines	Patch %	Lines
...ical-optimizer/src/limited_distinct_aggregation.rs	78.12%	4 Missing and 10 partials ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main   #25390      +/-   ##
==========================================
- Coverage   81.93%   81.93%   -0.01%     
==========================================
  Files        1136     1136              
  Lines      428779   429216     +437     
  Branches   428779   429216     +437     
==========================================
+ Hits       351319   351674     +355     
- Misses      56446    56485      +39     
- Partials    21014    21057      +43     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[2010YOUY01]

Thank you. This is a legitimate optimization, but I'm not sure the pattern is common enough to justify maintaining an additional, less intuitive optimizer rule.

The same goal can already be achieved with a small application-side workaround, using the existing optimization that pushes LIMIT into aggregates:

-- To check whether NDV > 1, run:
SELECT DISTINCT v1 % 100
FROM generate_series(10000000000) AS t1(v1)
LIMIT 10;

-- Then check whether the result contains more than one row.

So I'm leaning toward avoiding this rule. DataFusion is increasingly constrained by implementation complexity, and the optimizer is already one of the trickiest parts to maintain.

I'd reconsider if we can confirm that this is a sufficiently common query pattern to justify the additional complexity.

[HairstonE]

@2010YOUY01 Thanks for the quick review.
I was hoping this addition to the LimitedDistinctAggregation rule would be small enough to keep the complexity low. The only counter I have to the handwritten example would be queries that are generated from dashboards. Even then I don't know how common this shape would be.
Please confirm you would like to avoid this addition to the codebase, and I'll close the PR.
I'd appreciate if you could also point me to an area of the project where contributions would be most valuable right now. Just looking to contribute more to DF.