Export partition (Antalya 26.6) - allow non matching partition expressions in case we can prove the destination expression does not split the data - #2253
Conversation
arthurpassos
changed the title
arthurpassos
changed the title
arthurpassos
added
antalya
port-antalya
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 8af52375ff
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| for (const auto & part : parts) | ||
| if (exported.contains(part->name)) | ||
| minmax.merge(*part->getMinMaxIndex()); |
There was a problem hiding this comment.
When a replicated Iceberg export runs longer than old_parts_lifetime, merges can make every manifest-listed part outdated and cleanup can then remove them all. This loop accepts only exact part names, so every subsequent commit attempt finds an uninitialized min/max index and retries until the task timeout even though all data files were exported; the retry cannot recover because those part names will not reappear. Persist the representative partition values at scheduling time, or otherwise retain a safe derivation that survives source-part cleanup.
Useful? React with 👍 / 👎.
|
|
||
| The Iceberg manifest files contain statistics about the data. Exporting a merge tree partition is a non ephemeral long running task, in which nodes can be turned off and turned on. This means the stats of individual files need to be persisted somewhere in order to produce the final manifest. This is implemented through sidecars. Each data file exported will contain a "sibling" sidecar file named `<data_file_name>_clickhouse_export_part_sidecar.avro`. ClickHouse does not clean up these files, and they can be safely deleted once the data is comitted. | ||
|
|
||
| #### Source partition key compatibility |
There was a problem hiding this comment.
This newly added documentation heading has no explicit {#kebab-case-anchor} suffix, contrary to the repository requirement for every heading under docs/; add an anchor such as {#source-partition-key-compatibility} so generated navigation and direct links use the required stable target.
AGENTS.md reference: AGENTS.md:L9-L9
Useful? React with 👍 / 👎.
1 participant
Changelog category (leave one):
Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):
Allow export partition through different partition expressions as long as the destination expression does not repartition the data. This is validated at schedule time through two mechanisms:
Documentation entry for user-facing changes
...
CI/CD Options
Exclude tests:
Regression jobs to run: