Skip to content

datalake_agent: Iceberg operations by metadata location #2013

Description

@MisterRaindrop

Part of #2008. Letters (A, B0–B7, C, D, E) are the PRs listed there; this is B3.

Scope

The kernel behind B1, with no catalog service yet: every operation takes a metadata_location. This is what the Builtin catalog (B4) needs and what every catalog provider (B5–B7) builds on.

  • Through iceberg-java: initial metadata for a new table at a location; load, statistics, the schema changes the AM allows, truncate.
  • Planning: a snapshot's data files, delete files and row-group ranges streamed as fragments; file groups for rewrite.
  • Staging: append and update per statement, producing a new uncommitted metadata file whose location the caller commits (B4 locally, B5–B7 through the catalog).
  • An operation_id claim store so a retried request applies once.
  • Storage from the request: ResolvingFileIO with S3FileIO for s3://.
  • iceberg-java moves from the 1.3 line the port started on to a current 1.x.
  • In-process unit tests per RPC; end-to-end against MinIO alone.

Out of scope

Catalog providers (B5–B7); compaction and snapshot expiry; data-file I/O (the agent never touches data files).

Depends on

B1; verified through B2.

Acceptance

  • Create, load, plan and stage against MinIO alone, driven by the B2 conformance suite.
  • A staged append retried with the same operation_id yields one metadata file.
  • Credentials never appear in agent logs; agent memory is flat over ten thousand plan calls.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    datalakecontrib/datalake_fdw and contrib/datalake_agent: Iceberg lake tables

    Type

    No type

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions