Skip to content

[MINOR][SQL] Centralize static and runtime filter eligibility checks - #58752

Open
szehon-ho wants to merge 2 commits into
apache:masterfrom
szehon-ho:codex/block-nondeterministic-filter-pushdown
Open

[MINOR][SQL] Centralize static and runtime filter eligibility checks#58752
szehon-ho wants to merge 2 commits into
apache:masterfrom
szehon-ho:codex/block-nondeterministic-filter-pushdown

Conversation

@szehon-ho

@szehon-ho szehon-ho commented Sep 11, 2026

Copy link
Copy Markdown
Member

What changes were proposed in this pull request?

This PR centralizes filter eligibility checks in PushDownUtils:

  • Static filters are partitioned by determinism once and reused by the SupportsPushDownV2Filters and SupportsPushDownCatalystFilters paths. The shared value is lazy, so the legacy SupportsPushDownFilters path remains unchanged.
  • Runtime filters are screened once before dispatching to the SupportsRuntimeV2Filtering or SupportsRuntimeCatalystFiltering path.

It also updates the runtime-filter method documentation to describe the enforced behavior instead of requiring callers to satisfy a precondition.

Why are the changes needed?

Filter eligibility is independent of the connector interface. Applying each shared check before interface dispatch removes duplicated screening and keeps equivalent connector paths aligned.

Does this PR introduce any user-facing change?

No.

How was this patch tested?

No new tests were added because existing tests cover non-deterministic static and runtime filters for the affected connector paths.

The following focused runtime-filter tests passed:

SPARK_LOCAL_IP=127.0.0.1 build/sbt 'sql/testOnly org.apache.spark.sql.connector.DataSourceV2SQLSuiteV2Filter -- -z "SPARK-58207: non-deterministic scalar subquery filters are not pushed into runtimeFilters"'

SPARK_LOCAL_IP=127.0.0.1 build/sbt 'sql/testOnly org.apache.spark.sql.connector.DataSourceV2CatalystRuntimeFilterSuite -- -z "non-deterministic predicate on fully pushed attributes"'

Also ran dev/lint-scala.

Was this patch authored or co-authored using generative AI tooling?

Generated-by: OpenAI Codex (GPT-5)

@szehon-ho szehon-ho changed the title [MINOR][SQL] Centralize runtime filter eligibility checks [MINOR][SQL] Centralize static and runtime filter eligibility checks Sep 11, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant