Skip to content

datalake_fdw: merge-on-read and DML #2020

Description

@MisterRaindrop

Part of #2008. Letters (A, B0–B7, C, D, E) are the PRs listed there; this is E.

Scope

  • Merge-on-read: WrapPositionDeleteFilter implemented on the hidden row-ordinal column; position deletes applied while scanning, equality deletes after.
  • DELETE and UPDATE: rows identified by (file, position) through the scan, written as position delete files; UPDATE is delete plus insert through C's writer; one snapshot per statement.
  • A commit that loses the race is retried or fails with a serialization error, never silently dropped.
  • VACUUM stays the current no-op; compaction and snapshot expiry are a follow-up issue.

Out of scope

Copy-on-write, partition evolution, branches and tags (#1683 §2.3).

Depends on

C, D.

Acceptance

  • Deletes made in Cloudberry are seen by Spark and the reverse.
  • UPDATE/DELETE counts match; a rolled-back statement changes nothing visible.
  • Two concurrent transactions on one table: one commits, the other fails or retries; no lost update.
  • A scan over many small delete files stays within the tracked memory budget.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    datalakecontrib/datalake_fdw and contrib/datalake_agent: Iceberg lake tables

    Type

    No type

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions