chore: Deprecate internal-only APIs in AggregateExec by 2010YOUY01 · Pull Request #25257 · apache/datafusion

2010YOUY01 · 2026-09-13T09:07:20Z

Which issue does this PR close?

Closes #.

Rationale for this change

AggregateExec::{with_new_limit_options, with_limit_options, limit_options} set a limit hint on an aggregation. This is part of the public API for AggregateExec

It's nearly impossible to use it correctly (see below code explanation) if we see it as a public ExecutionPlan API, however it requires pub because physical optimizer requires it.

// physical optimizer
if tricky_pre_conditions() {
  agg_exec.with_limit_options(k)
}

This PR does:

Deprecate those API, but it can't get deleted since optimizer requires it. This approach can warn downstreams that this API is unsafe and should not be used directly
Mark #[doc(hidden)]. That is the usual Rust convention for "public for technical reasons, not for external user": the item is dropped from generated docs (docs.rs)

What changes are included in this PR?

What is the testing strategy for this PR?

Are there any user-facing changes?

Yes, deprecations.

2010YOUY01 · 2026-09-13T09:08:16Z

Not sure if there are better approaches for this case, it we have to do this way, we can later doc this pattern in
https://datafusion.apache.org/contributor-guide/api-health.html

github-actions · 2026-09-13T09:13:43Z

Thank you for opening this pull request!

Reviewer note: cargo-semver-checks reported the current version number is not SemVer-compatible with the changes in this pull request (compared against the base branch).

Details

     Cloning apache/main
    Building datafusion v55.0.0 (current)
       Built [  42.429s] (current)
     Parsing datafusion v55.0.0 (current)
      Parsed [   0.024s] (current)
    Building datafusion v55.0.0 (baseline)
       Built [  37.990s] (baseline)
     Parsing datafusion v55.0.0 (baseline)
      Parsed [   0.024s] (baseline)
    Checking datafusion v55.0.0 -> v55.0.0 (no change; assume patch)
     Checked [   0.391s] 223 checks: 223 pass, 31 skip
     Summary no semver update required
    Finished [  82.407s] datafusion
    Building datafusion-physical-optimizer v55.0.0 (current)
       Built [  25.916s] (current)
     Parsing datafusion-physical-optimizer v55.0.0 (current)
      Parsed [   0.015s] (current)
    Building datafusion-physical-optimizer v55.0.0 (baseline)
       Built [  26.308s] (baseline)
     Parsing datafusion-physical-optimizer v55.0.0 (baseline)
      Parsed [   0.015s] (baseline)
    Checking datafusion-physical-optimizer v55.0.0 -> v55.0.0 (no change; assume patch)
     Checked [   0.072s] 223 checks: 223 pass, 31 skip
     Summary no semver update required
    Finished [  53.201s] datafusion-physical-optimizer
    Building datafusion-physical-plan v55.0.0 (current)
       Built [  24.255s] (current)
     Parsing datafusion-physical-plan v55.0.0 (current)
      Parsed [   0.107s] (current)
    Building datafusion-physical-plan v55.0.0 (baseline)
       Built [  24.044s] (baseline)
     Parsing datafusion-physical-plan v55.0.0 (baseline)
      Parsed [   0.106s] (baseline)
    Checking datafusion-physical-plan v55.0.0 -> v55.0.0 (no change; assume patch)
     Checked [   0.464s] 223 checks: 222 pass, 1 fail, 0 warn, 31 skip

--- failure type_method_marked_deprecated: type method #[deprecated] added ---

Description:
A type method is now #[deprecated]. Downstream crates will get a compiler warning when using this method.
        ref: https://doc.rust-lang.org/reference/attributes/diagnostics.html#the-deprecated-attribute
       impl: https://github.com/obi1kenobi/cargo-semver-checks/tree/v0.50.0/src/lints/type_method_marked_deprecated.ron

Failed in:
  method datafusion_physical_plan::aggregates::AggregateExec::with_new_limit_options in /home/runner/work/datafusion/datafusion/datafusion/physical-plan/src/aggregates/mod.rs:946
  method datafusion_physical_plan::aggregates::AggregateExec::with_limit_options in /home/runner/work/datafusion/datafusion/datafusion/physical-plan/src/aggregates/mod.rs:973
  method datafusion_physical_plan::aggregates::AggregateExec::limit_options in /home/runner/work/datafusion/datafusion/datafusion/physical-plan/src/aggregates/mod.rs:986

     Summary semver requires new minor version: 0 major and 1 minor checks failed
    Finished [  50.262s] datafusion-physical-plan
    Building datafusion-proto v55.0.0 (current)
       Built [  34.914s] (current)
     Parsing datafusion-proto v55.0.0 (current)
      Parsed [   0.011s] (current)
    Building datafusion-proto v55.0.0 (baseline)
       Built [  35.213s] (baseline)
     Parsing datafusion-proto v55.0.0 (baseline)
      Parsed [   0.013s] (baseline)
    Checking datafusion-proto v55.0.0 -> v55.0.0 (no change; assume patch)
     Checked [   0.074s] 223 checks: 223 pass, 31 skip
     Summary no semver update required
    Finished [  71.417s] datafusion-proto

codecov-commenter · 2026-09-13T09:33:14Z

Codecov Report

❌ Patch coverage is 91.66667% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 81.88%. Comparing base (681705e) to head (97e4c7a).

Files with missing lines	Patch %	Lines
...hysical-plan/src/aggregates/grouped_hash_stream.rs	0.00%	0 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main   #25257      +/-   ##
==========================================
- Coverage   81.88%   81.88%   -0.01%     
==========================================
  Files        1133     1133              
  Lines      424522   424522              
  Branches   424522   424522              
==========================================
- Hits       347622   347615       -7     
- Misses      56288    56291       +3     
- Partials    20612    20616       +4

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

2010YOUY01 · 2026-09-16T02:04:12Z

    }

    /// Clone this exec, overriding only the limit hint.
+    ///


This file has the only change (deprecation)

Other diff are all mechanical changes like #expect(deprecated) at call sites.

2010YOUY01 · 2026-09-16T02:05:49Z

@adriangb Could you take a look, this API restriction intends to prevent similar bugs like #25349

Building and rewriting an `AggregateExec` is how DataFusion's own physical optimizer rules work, not something external consumers should reach for. Hide that whole surface from the rendered docs, as apache#25257 does for the methods it deprecates: - the deprecated `with_limit_options`, `with_new_limit_options` and `with_new_aggr_exprs` - their replacements, `AggregateExec::builder`, `AggregateExec::to_builder` and `AggregateExecBuilder` itself - the `limit_options` getter `limit_options` is hidden but deliberately not deprecated: it has no replacement, reading the limit of an aggregate is safe, and deprecating it would only force `#[expect(deprecated)]` back into the optimizer rule that copies a limit between aggregates. Doc links from the still-public `AggregateExec::try_new` and `LimitOptions` into the hidden items are now plain code spans, since a link to a hidden item renders as a dead anchor. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PwTc51ca2XHDCyVB7MbJoz

adriangb · 2026-09-16T13:58:34Z

@2010YOUY01 I'm curious what you think of #25376 as an alternative? It's a larger change but it prevents us from hitting these bugs / makes bad states unrepresentable even for internal users of these APIs.

Building and rewriting an `AggregateExec` is how DataFusion's own physical optimizer rules work, not something external consumers should reach for. Hide that whole surface from the rendered docs, as apache#25257 does for the methods it deprecates: - the deprecated `with_limit_options`, `with_new_limit_options` and `with_new_aggr_exprs` - their replacements, `AggregateExec::builder`, `AggregateExec::to_builder` and `AggregateExecBuilder` itself - the `limit_options` getter `limit_options` is hidden but deliberately not deprecated: it has no replacement, reading the limit of an aggregate is safe, and deprecating it would only force `#[expect(deprecated)]` back into the optimizer rule that copies a limit between aggregates. Doc links from the still-public `AggregateExec::try_new` and `LimitOptions` into the hidden items are now plain code spans, since a link to a hidden item renders as a dead anchor. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PwTc51ca2XHDCyVB7MbJoz

`with_limit_options`, `with_new_limit_options` and `with_new_aggr_exprs` each clone the node with one field replaced and leave the caller to know which of the other twelve fields that field feeds. Deprecate them in favour of the builder. Building and rewriting an `AggregateExec` is how DataFusion's own physical optimizer rules work, not a public API, so the whole surface is `#[doc(hidden)]`: the builder, `AggregateExec::builder`, `AggregateExec::to_builder`, the three deprecated methods, and the `limit_options` getter. The getter is not deprecated, because reading a limit is safe and has no replacement. This is what apache#25257 asked for, with a replacement to point callers at and no `#[expect(deprecated)]` left inside DataFusion. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01D7arPq4Frxu8byr17mqVKA

2010YOUY01 · 2026-09-17T06:59:42Z

@2010YOUY01 I'm curious what you think of #25376 as an alternative? It's a larger change but it prevents us from hitting these bugs / makes bad states unrepresentable even for internal users of these APIs.

I thought about the same approach at first, and didn't do it simply because it feels like a lot of work.

So definitely +1, closing this PR for this safer alternative.

`with_limit_options`, `with_new_limit_options` and `with_new_aggr_exprs` each clone the node with one field replaced and leave the caller to know which of the other twelve fields that field feeds. Deprecate them in favour of the builder. Building and rewriting an `AggregateExec` is how DataFusion's own physical optimizer rules work, not a public API, so the whole surface is `#[doc(hidden)]`: the builder, `AggregateExec::builder`, `AggregateExec::to_builder`, the three deprecated methods, and the `limit_options` getter. The getter is not deprecated, because reading a limit is safe and has no replacement. This is what apache#25257 asked for, with a replacement to point callers at and no `#[expect(deprecated)]` left inside DataFusion. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01D7arPq4Frxu8byr17mqVKA

chore: Deprecate public APIs for internal usage in AggregateExec

97e4c7a

github-actions Bot added the auto detected api change Auto detected API change label Sep 13, 2026

2010YOUY01 commented Sep 16, 2026

View reviewed changes

This was referenced Sep 16, 2026

feat: add a validated AggregateExecBuilder for building and rewriting AggregateExec pydantic/datafusion#77

Closed

refactor: build and rewrite AggregateExec through an AggregateExecBuilder #25376

Open

adriangb mentioned this pull request Sep 17, 2026

Reject a limit an AggregateExec cannot execute when the plan is built #25393

Open

2010YOUY01 closed this Sep 17, 2026

Navigation Menu

chore: Deprecate internal-only APIs in AggregateExec - #25257

chore: Deprecate internal-only APIs in AggregateExec#25257
2010YOUY01 wants to merge 1 commit into
apache:mainfrom
2010YOUY01:hide-aggr-limit-api

2010YOUY01 commented Sep 13, 2026 •

edited

Loading

Uh oh!

2010YOUY01 commented Sep 13, 2026

Uh oh!

github-actions Bot commented Sep 13, 2026

Uh oh!

codecov-commenter commented Sep 13, 2026

Uh oh!

2010YOUY01 Sep 16, 2026

Uh oh!

2010YOUY01 commented Sep 16, 2026

Uh oh!

adriangb commented Sep 16, 2026

Uh oh!

2010YOUY01 commented Sep 17, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

2010YOUY01 commented Sep 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Which issue does this PR close?

Rationale for this change

What changes are included in this PR?

What is the testing strategy for this PR?

Are there any user-facing changes?

Uh oh!

2010YOUY01 commented Sep 13, 2026

Uh oh!

github-actions Bot commented Sep 13, 2026

Uh oh!

codecov-commenter commented Sep 13, 2026

Codecov Report

Uh oh!

2010YOUY01 Sep 16, 2026

Choose a reason for hiding this comment

Uh oh!

2010YOUY01 commented Sep 16, 2026

Uh oh!

adriangb commented Sep 16, 2026

Uh oh!

2010YOUY01 commented Sep 17, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

2010YOUY01 commented Sep 13, 2026 •

edited

Loading