Skip to content

[rust] Wire batch statistics into the log batch write and read paths - #4074

Open
leekeiabstraction wants to merge 1 commit into
apache:mainfrom
leekeiabstraction:rust-batch-stats-wiring
Open

[rust] Wire batch statistics into the log batch write and read paths#4074
leekeiabstraction wants to merge 1 commit into
apache:mainfrom
leekeiabstraction:rust-batch-stats-wiring

Conversation

@leekeiabstraction

@leekeiabstraction leekeiabstraction commented Aug 22, 2026

Copy link
Copy Markdown
Contributor

Summary

  • Wires the V1 batch statistics collection ([rust] Add V1 batch statistics collection #4011) into the write and read paths. Closes [rust] Batch statistics read wiring #4009, closes [rust] Batch statistics write wiring #4010.
  • Write: for tables with table.statistics.columns set, the accumulator hands the resolved column mapping to the Arrow builder, which emits a V1 batch with the serialized statistics placed between the header and the records data, mirroring Java's MemoryLogRecordsArrowBuilder. A serialization failure degrades to an empty statistics section, and the batch size estimate reserves room for the statistics.
  • Read: the records data is located per the batch's magic, mirroring Java's DefaultLogRecordBatch.recordsDataOffset, and every decode entry point rejects unknown magic versions before the empty-batch short-circuit. This fixes the Rust reader misparsing V1 batches that Java clients and the server's projection path already produce.

Test Plan

  • A cross-language golden fixture (fluss-common/src/test/resources/encoding/statistics_block.hex, generated and asserted by a new Java test) locks the Rust statistics block byte-for-byte against Java's LogRecordBatchStatisticsWriter across all supported types, nulls, both decimal widths, and both timestamp precisions. CHAR is excluded because Java's collector records no CHAR bounds.
  • New unit tests cover V1 encode/decode round trips from both builder modes, empty and corrupt statistics sections, unsupported magic (including the V2 silent-drop case), and the size reserve. cargo test -p fluss-rs --lib passes 700 tests; clippy and fmt are clean.
  • The end-to-end filter pushdown integration test is left to [rust] Filter pushdown integration test #3845.

🤖 AI-assisted changes - reviewed by human developer

@leekeiabstraction
leekeiabstraction force-pushed the rust-batch-stats-wiring branch 4 times, most recently from fa1d999 to 64a77dd Compare August 22, 2026 22:22
Tables with 'table.statistics.columns' set now produce V1 batches carrying
serialized statistics, matching Java's MemoryLogRecordsArrowBuilder. The
reader now locates the records data per the batch's magic, fixing the
misparsing of V1 batches that Java clients and the server's projection
path already produce.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[rust] Batch statistics write wiring [rust] Batch statistics read wiring

1 participant