Skip to content

[FLINK-40418][python] Support attribute-based column access on DataFrame - #29164

Open
beetle0915 wants to merge 1 commit into
apache:masterfrom
beetle0915:FLINK-40418
Open

beetle0915 wants to merge 1 commit into
apache:masterfrom
beetle0915:FLINK-40418

Conversation

@beetle0915

Copy link
Copy Markdown
Contributor

What is the purpose of the change

Implement FLINK-40418, the attribute-based column access described in FLIP-591. This allows df.name to return a column expression when the name is a valid Python identifier, is not a keyword, and does not conflict with an existing DataFrame attribute.

The implementation follows the approach discussed and accepted on the Jira issue.

Brief change log

  • Add DataFrame.__getattr__, preserving normal Python attribute lookup and reusing the existing Table column expression API.
  • Raise AttributeError for invalid or missing column names; guard against recursive lookup on uninitialized instances. Column lookup resolves the current schema without executing a Flink job.
  • Add unit and execution-level tests, and document attribute access and bracket-access fallbacks.

Verifying this change

This change adds 13 tests covering valid and missing columns, identifier rules, attribute conflicts, transformed schemas, uninitialized instances, schema errors, and execution of a filter/projection using attribute references.

Local validation on Python 3.12 and Java 17:

  • python -m pytest flink-python/pyflink/dataframe/tests -q: 352 passed (10 deprecation warnings).
  • From flink-python: python -m flake8 --config=tox.ini pyflink/dataframe: passed.
  • From flink-python: python -m mypy --config-file tox.ini: passed (83 source files).
  • Full Python HTML documentation build with the repository-pinned Sphinx dependencies, -a -W --keep-going: passed with no warnings.
  • git diff --check: passed.

The local validation does not include the full repository mvn clean verify, the full end-to-end suite, or the other supported Python versions; these checks remain to be covered by CI/review.

Does this pull request potentially affect one of the following parts:

  • Dependencies (does it add or upgrade a dependency): no
  • The public API, i.e., is any changed class annotated with @Public(Evolving): yes
  • The serializers: no
  • The runtime per-record code paths (performance sensitive): no
  • Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: no
  • The S3 file system connector: no

Documentation

  • Does this pull request introduce a new feature? yes
  • If yes, how is the feature documented? Python API reference and method docstring, including examples and fallback syntax for conflicting or invalid names.

Was generative AI tooling used to co-author this PR?
  • Yes (please specify the tool below)

Generated-by: OpenAI Codex (codex-cli 0.154.0-alpha.6.2)

Generated-by: OpenAI Codex (codex-cli 0.154.0-alpha.6.2)
@flinkbot

flinkbot commented Sep 12, 2026

Copy link
Copy Markdown
Collaborator

CI report:

Bot commands The @flinkbot bot supports the following commands:
  • @flinkbot run azure re-run the last Azure build

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants