Skip to content

datalake_agent: service framework (Java) #2011

Description

@MisterRaindrop

Part of #2008. Letters (A, B0–B7, C, D, E) are the PRs listed there; this is B1.

Scope

contrib/datalake_agent: JDK 17, Maven, Guice, Netty gRPC. No Iceberg in it yet.

  • Two-phase startup: bind and serve health first, build the injector and flip to SERVING second.
  • Transport: Unix socket by default with configured permissions; TCP only with TLS and a 0600 bearer-token file (current plus rotation-grace token).
  • Interceptors: request context, exception → status with BusinessErrorCode (no stack trace leaves the process), version handshake refusing an old client and naming both versions.
  • HOCON config for process concerns only; everything about a table travels in the request. One request mapper turns the fdw's options into Iceberg property maps.
  • Credential scrubbing in logs; opt-in metrics.
  • Every business RPC answers UNIMPLEMENTED naming the RPC. Catalog management present, off by default.
  • In-process gRPC tests, a Unix-socket test, negative TLS/token tests; CI runs mvn -B verify; RAT headers. The PostgreSQL build does not need Java.

Out of scope

Iceberg operations and catalogs (B3+), the proxy bgworker, installing the jar through make install.

Depends on

B0.

Acceptance

  • Cold start to SERVING under 1.5 s in CI; health answers on the socket.
  • Old client refused naming both versions; TCP without TLS refuses to start; bad token is UNAUTHENTICATED; business RPCs are UNIMPLEMENTED naming the RPC.
  • A fake key in config and request never appears in DEBUG logs (asserted by test).
  • mvn -B verify and RAT green in CI.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    datalakecontrib/datalake_fdw and contrib/datalake_agent: Iceberg lake tables

    Type

    No type

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions