Skip to content

datalake_agent: Polaris (REST) catalog #2015

Description

@MisterRaindrop

Part of #2008. Letters (A, B0–B7, C, D, E) are the PRs listed there; this is B5.

Scope

The first external catalog provider.

  • RESTCatalog: OAuth2 client credentials or bearer token, warehouse, vended credentials so a volume can be omitted. polaris accepted as an alias of rest, as in the fdw whitelist.
  • Commit through the catalog's CAS; B3's operation_id store covers the in-doubt window between "committed" and "the client heard".
  • Drop with and without purge; namespaces through CatalogManagementService.
  • End-to-end against Apache Polaris (or the Iceberg REST fixture image) plus MinIO in docker.

Out of scope

Hive, Hadoop. Anything the REST spec leaves optional and Polaris does not serve.

Depends on

B3.

Acceptance

  • Conformance suite green against Polaris; a table Cloudberry creates is visible to Spark in the same catalog, and the reverse.
  • DROP TABLE with and without purge_on_drop behaves as the option says.
  • A commit retried after a simulated timeout yields exactly one snapshot.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    datalakecontrib/datalake_fdw and contrib/datalake_agent: Iceberg lake tables

    Type

    No type

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions