Skip to content

[python] Support selected-key projection for shared-shredding MAP - #9789

Draft
XiaoHongbo-Hope wants to merge 11 commits into
apache:masterfrom
XiaoHongbo-Hope:codex/map-selected-key-projection
Draft

[python] Support selected-key projection for shared-shredding MAP#9789
XiaoHongbo-Hope wants to merge 11 commits into
apache:masterfrom
XiaoHongbo-Hope:codex/map-selected-key-projection

Conversation

@XiaoHongbo-Hope

@XiaoHongbo-Hope XiaoHongbo-Hope commented Sep 13, 2026

Copy link
Copy Markdown
Contributor

Purpose

Support literal-key projection for top-level MAP<STRING, ...> columns through the existing with_projection() API:

read_builder.with_projection([
    "id",
    "attributes['key.with.dots']",
])

Bracket syntax follows Spark MAP access, keeps dot notation for nested ROW fields, and avoids adding another ReadBuilder method. Exact top-level field names take precedence.

Behavior

  • Shared-shredding files read only required physical columns and overflow.
  • Other layouts and keys not safely encodable in the internal metadata fall back to reading the full MAP.
  • Parquet, ORC, column rename, value type evolution, and data-evolution reads are supported.
  • Full-MAP projection and MAP aggregation retain the existing behavior.
  • Internally, PyPaimon reuses the Java/Spark temporary ROW and field-description protocol; it is not persisted.
  • No new method or table option is introduced.

Tests

  • 115 targeted Python tests pass.
  • Java-written Parquet/ORC fixture tests pass.
  • Python 3.6 syntax and git diff --check pass.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant