Skip to content

datalake_fdw: Builtin catalog #2014

Description

@MisterRaindrop

Part of #2008. Letters (A, B0–B7, C, D, E) are the PRs listed there; this is B4.

Scope

The type-unset case of #1683 §4: a lake table with no catalog service, only object storage. The first catalog to work end to end, so C and D develop and test against MinIO and the agent alone.

  • src/meta/engine_builtin/: each table's current metadata_location in an extension-owned table on the coordinator; commit is a conditional update under a row lock inside the transaction — atomic with the statement, and across tables.
  • Metadata work delegated to B3 by location; the agent holds no state for these tables.
  • Other engines open the table by path (HadoopCatalog / FileIO).
  • The two commit regimes (builtin: atomic with the transaction; external catalogs: an in-doubt window) written into the engine interface docs.

Out of scope

Sharing a builtin table between two clusters.

Depends on

B2, B3.

Acceptance

  • DDL and the full conformance suite green with only MinIO and the agent running; rollback leaves the HEAD unchanged.
  • Spark opens the table by path and sees Cloudberry's metadata.
  • Two concurrent writers: one commits, the other errors; no lost update.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    datalakecontrib/datalake_fdw and contrib/datalake_agent: Iceberg lake tables

    Type

    No type

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions