Skip to content

Support Iceberg decimals with 38-digit precision - #67

Merged
osipovartem merged 1 commit into
embucket-sync-df55.0.0from
decimal-38-fastnum-df55
Sep 2, 2026
Merged

Support Iceberg decimals with 38-digit precision#67
osipovartem merged 1 commit into
embucket-sync-df55.0.0from
decimal-38-fastnum-df55

Conversation

@osipovartem

Copy link
Copy Markdown

Summary

  • replace the 96-bit rust_decimal value used by Iceberg bounds with fastnum::D128
  • support the full Iceberg decimal(38, s) value range in metadata and pruning statistics
  • encode decimal bounds using the Iceberg minimum-length big-endian two's-complement representation
  • keep DataFusion pruning and manifest statistics lossless through Arrow Decimal128

The approach adapts the decimal representation used by Apache iceberg-rust for full 38-digit precision to this fork's Value model.

Why

Snowflake can write 38-digit decimal lower/upper bounds into Iceberg metadata. Decoding those values through rust_decimal panicked because its mantissa is limited to 96 bits, so Rustice could not read otherwise valid Snowflake Iceberg tables.

Verification

  • cargo +1.94.0 test -p iceberg-rust-spec --lib (151 passed, 1015 explicitly ignored)
  • cargo +1.94.0 test -p datafusion_iceberg --lib (50 passed)
  • cargo +stable clippy --all-targets --all-features -- -D warnings
  • cargo +stable fmt --all -- --check
  • regression coverage for positive and negative 38-digit byte round trips and conversion to DataFusion Decimal128

@osipovartem
osipovartem force-pushed the decimal-38-fastnum-df55 branch from bbe112f to 446feb6 Compare September 2, 2026 12:19
@osipovartem
osipovartem merged commit 774a6fe into embucket-sync-df55.0.0 Sep 2, 2026
2 checks passed
@osipovartem
osipovartem deleted the decimal-38-fastnum-df55 branch September 2, 2026 13:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant