Skip to content

Enable nested field statistics with inferred metrics limit - #3810

Open
jkasireddy wants to merge 1 commit into
apache:mainfrom
jkasireddy:fix/2699-nested-metrics
Open

Enable nested field statistics with inferred metrics limit#3810
jkasireddy wants to merge 1 commit into
apache:mainfrom
jkasireddy:fix/2699-nested-metrics

Conversation

@jkasireddy

Copy link
Copy Markdown

Closes #2699

Rationale for this change

PyIceberg currently downgrades nested primitive fields to counts, which prevents lower and upper bound statistics from being collected for nested fields.

This change enables statistics collection for nested primitive fields and adds support for write.metadata.metrics.max-inferred-column-defaults, following the behavior of the Iceberg Java implementation.

When no explicit default metrics mode is configured, inferred metrics are limited to the configured number of fields. Primitive fields at the current struct level are prioritized before descending into nested fields. Explicit per-column metrics configuration continues to take precedence over the inferred limit.

Are these changes tested?

Yes.

Added regression tests covering:

  • statistics collection for nested primitive fields
  • write.metadata.metrics.max-inferred-column-defaults
  • prioritization of top-level primitive fields before nested fields
  • explicit per-column overrides beyond the inferred limit
  • explicit default metrics mode behavior
  • zero and negative inferred-column limits
  • lower and upper bounds for nested struct, list, and map primitive fields

Local test results:

uv run python -m pytest tests/io/test_pyarrow_stats.py -q
24 passed

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Enable stats collection for nested fields and use write.metadata.metrics.max-inferred-column-defaults to control stats growth

1 participant