feat: declare and read back a managed table's storage layout (#69)

* feat: declare and read back a managed table's storage layout

Closes #55

add_managed_table() and create_managed_database() take partition_by /
sorted_by; managed_table_layout() reads back what was actually declared.
TablePartitionKey and TableSortKey are re-exported so callers need one import.

BOTH DIRECTIONS, because either alone is not usable. A layout is fixed when the
table is created and there is no alter path -- a table declared without one keeps
that shape until it is recreated and its data rewritten. So declaring is half the
job: a caller has to be able to confirm it took, and to refuse to load when it
cannot. That is why at least one consumer hand-built the HTTP request rather than
using this package: the write side dropped the fields and the read side had
nowhere to report them.

managed_table_layout() raises KeyError for a table that is not declared instead
of returning an empty layout. "Not there" and "declared without a layout" lead a
caller to opposite decisions.

The generated key models are passed through rather than wrapped in a parallel
type system. This is a thin wrapper over a GENERATED client, so the transform
vocabulary and field names stay exactly the API's, and the strict mypy config is
satisfied without Any.

hotdata floor raised to >=0.9.0,<0.10. 0.9.0 is the first release carrying
partition_by / sorted_by on all three models this needs -- the add-table request,
the create-database table declarations, and the table-info response. On an older
hotdata the fields are silently dropped by the model and the table is declared
without a layout, returning success. Verified against the published 0.9.0 wheel.

TESTS ASSERT THE SERIALISED REQUEST, not that the call succeeded, because the
failure mode is silence: a field the model does not know about vanishes at
to_dict() and the API returns 201. Sabotaged all three:

  drop the layout from the request        -> caught
  send empty arrays instead of omitting   -> caught
  collapse missing-table into empty       -> caught

133 -> 138 passed. ruff 2 -> 1 (the auto-fix also cleaned a pre-existing
import-order issue in client.py; the hotdata.uploads comment stayed with its
import, checked). mypy unchanged at 158, all pre-existing.

* fix: refuse layout for a table not being created, and read it back in one request

Five review points, all taken.

create_managed_database now raises ValueError when partition_by or sorted_by
names a table that is not in `tables`. Previously a typo -- {"fils": ...} against
tables=["files"] -- was dropped in silence and the intended table created flat,
which is permanent because a layout is fixed at creation with no alter path. The
asymmetry with `keys` is deliberate: a wrong key is recoverable, since
load_managed_table takes `key=` per call.

managed_table_layout now filters server-side (var_schema=/table=/limit=1) instead
of paging iter_tables until it finds a match, which cost several round trips for
a table sorting late in the listing. Same KeyError contract.

The absent-vs-unpartitioned test shared one response object across both calls, so
its second assertion inspected a consumed result rather than the case named in
the docstring. Each call now gets a fresh response, and the fake answers by table
name so the "listing has files but not missing" scenario is the one exercised.

TableLayout gains to_dict(), matching every other public dataclass here. Not
asdict(): the key lists hold pydantic models, which asdict copies through
untouched, so each key is mapped through its own to_dict(). The reviewer assumed
the omission was deliberate -- it was not.

CONTRACT.md lists managed_table_layout among the methods accepting an
already-resolved ManagedDatabase, which matters because a create-scoped key that
cannot read /databases is exactly the caller needing this read-back.

Re-sabotaged after the rewrite -- dropping the layout, collapsing missing-table
into an empty layout, and removing the unknown-table guard are each caught.
138 -> 139 passed.

* fix: drop TableLayout.to_dict rather than add strict-mypy debt

I added to_dict() for surface consistency and then checked the gates properly:
it added 8 errors under this package s strict mypy settings (dict[str, Any] plus
comprehensions over the pydantic key models), taking the file from the 158 on
main to 166.

The alternative -- hand-building the dict from named fields -- is Any-free but
silently drops any field a later spec adds to TablePartitionKey or TableSortKey,
which is exactly the silent-drop failure this whole feature exists to prevent.

So taking the reviewer s first option: no to_dict, with a comment recording why
the inconsistency is deliberate, so the next reader does not "fix" it. A caller
wanting dicts can map k.to_dict() itself.

Also collapses a nested `with` flagged by ruff.

Back to main s baseline exactly: mypy 158, ruff only the pre-existing long line
in test_request_timeout.py. 139 passed.

Author: anoop-narang

CHANGELOG.md

@@ -7,6 +7,36 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+### Added
+
+- Table storage layout, both directions. `add_managed_table()` and
+  `create_managed_database()` take `partition_by` / `sorted_by`, and
+  `managed_table_layout()` reads back what was actually declared as a
+  `TableLayout`. `TablePartitionKey` and `TableSortKey` are re-exported so
+  callers need one import.
+
+  Both halves matter because a layout is fixed when the table is created and
+  there is no alter path: a table declared without one keeps that shape until it
+  is recreated and its data rewritten. So declaring is not enough — a caller has
+  to be able to confirm it took, and to refuse to load when it cannot.
+
+  `managed_table_layout()` raises `KeyError` for a table that is not declared,
+  rather than returning an empty layout. "Not there" and "declared without a
+  layout" lead to opposite decisions for a caller.
+
+  Until now this package could not express a layout at all, which is why at least
+  one consumer hand-built the HTTP request instead. The generated key models are
+  passed through rather than wrapped, so the transform vocabulary stays exactly
+  the API's.
+
+### Changed
+
+- Require `hotdata>=0.9.0,<0.10`. 0.9.0 is the first release whose models carry
+  `partition_by` / `sorted_by` on the add-table request, the create-database
+  table declarations, and the table-info response. On an older `hotdata` the
+  fields would be silently dropped by the model and the table declared without a
+  layout, returning success — which is the failure this feature exists to end.
+
 ## [0.11.0] - 2026-08-11
 
 ### Changed

CONTRACT.md

@@ -31,6 +31,9 @@ The supported import surface is:
 - `WorkspaceSelection`
 - `ManagedDatabase`
 - `ManagedTable`
+- `TableLayout`
+- `TablePartitionKey`
+- `TableSortKey`
 - `LoadManagedTableResult`
 - `CreateIndexResult`
 - `DEFAULT_SCHEMA`
@@ -56,6 +59,8 @@ Adapters should import from `hotdata_framework` and treat this surface as the st
   adapters should pass `connection_id` when known.
 - `uploads()` returns the uploads API wrapper for parquet staging.
 - `list_managed_databases()` returns all databases via the `/databases` API.
+- `add_managed_table(...)` and `create_managed_database(...)` accept `partition_by` / `sorted_by` to declare a table's storage layout. The layout is fixed when the table is created and cannot be altered afterwards, so omitting it is permanent for that table.
+- `managed_table_layout(database, table, schema=...)` returns the declared layout as `TableLayout`. Empty lists mean no layout was declared — sound only because the table is resolved through a managed database. Raises `KeyError` when the table is not declared, keeping "absent" distinct from "declared without a layout".
 - `resolve_managed_database(name_or_id)` resolves a database by id (direct lookup) or description (list scan). A `403` from `/databases` surfaces as `RuntimeError` (forbidden, not absent), preserving the underlying `ApiException` as `__cause__`.
 - `create_managed_database(description=..., schema=..., tables=..., expires_at=...)` creates a database via the `/databases` API and optionally declares tables up front. Returns a `ManagedDatabase` (id + `default_connection_id`) sufficient to load without a further read.
 - `delete_managed_database(name_or_id)` deletes a database via the `/databases` API.
@@ -64,7 +69,7 @@ Adapters should import from `hotdata_framework` and treat this surface as the st
 - `load_managed_table(database, table, schema=..., upload_id=..., file=...)` publishes parquet data into a declared managed table.
 - `delete_managed_table(database, table, schema=...)` deletes a managed table.
 - `create_index(database, table, schema=..., columns=..., index_type=..., index_name=...)` builds a `"sorted"`, `"bm25"`, or `"vector"` index on a managed table and returns a `CreateIndexResult`. It is the framework-side equivalent of the CLI's `hotdata indexes create`; indexing a table on a plain (non-managed) connection is out of scope. `index_name` defaults to `{table}_{columns}_{index_type}`, matching the CLI's derivation when `--name` is omitted. `index_type` is required rather than defaulting to the API's `"sorted"`. The build runs as a background job; the call polls it to a terminal state and raises `RuntimeError` with the job's `error_message` when it fails, because the submit call reports success regardless. `wait=False` returns as soon as the job is accepted, with `status="pending"` and a `job_id` for the caller to poll. For `index_type="vector"`, omitting `embedding_provider_id` indexes an existing vector column and `metric` (`"l2"`, `"cosine"`, `"dot"`) selects the distance function the index accelerates — a query using a different function silently falls back to a full scan; setting `embedding_provider_id` indexes a source *text* column instead, and the returned `source_column` names the column to pass to `vector_distance`. Argument combinations the server would silently ignore raise `ValueError` before any request is sent.
-- The `database` argument of `list_managed_tables`, `load_managed_table`, `add_managed_table`, `delete_managed_table`, `delete_managed_database`, `create_index`, and `execute_sql` accepts a name/id **or** an already-resolved `ManagedDatabase`. Passing a `ManagedDatabase` skips the name/id read probe, so a create-scoped key that cannot read `/databases` can load into a database it just created.
+- The `database` argument of `list_managed_tables`, `load_managed_table`, `add_managed_table`, `delete_managed_table`, `delete_managed_database`, `create_index`, `managed_table_layout`, and `execute_sql` accepts a name/id **or** an already-resolved `ManagedDatabase`. Passing a `ManagedDatabase` skips the name/id read probe, so a create-scoped key that cannot read `/databases` can load into a database it just created.
 
 ### `QueryResult`
 

hotdata_framework/__init__.py

@@ -2,6 +2,9 @@
 
 from importlib.metadata import PackageNotFoundError, version
 
+from hotdata.models.table_partition_key import TablePartitionKey
+from hotdata.models.table_sort_key import TableSortKey
+
 from hotdata_framework.client import (
     HotdataClient,
     ResultSummary,
@@ -14,6 +17,7 @@
     LoadManagedTableResult,
     ManagedDatabase,
     ManagedTable,
+    TableLayout,
     is_parquet_path,
 )
 from hotdata_framework.env import (
@@ -55,6 +59,9 @@
     "QueryResult",
     "ResultSummary",
     "RunHistoryItem",
+    "TableLayout",
+    "TablePartitionKey",
+    "TableSortKey",
     "WorkspaceSelection",
     "__version__",
     "classify_sdk_error",

hotdata_framework/client.py

@@ -2,7 +2,7 @@
 
 import functools
 import time
-from collections.abc import Iterator
+from collections.abc import Iterator, Sequence
 from dataclasses import asdict, dataclass
 from typing import Any, Literal, get_args
 
@@ -15,9 +15,6 @@
 from hotdata.api.query_api import QueryApi
 from hotdata.api.query_runs_api import QueryRunsApi
 from hotdata.api.results_api import ResultsApi
-# The enriched wrapper (hotdata.uploads), NOT the generated hotdata.api class:
-# it adds the full upload_file orchestration used by upload_parquet.
-from hotdata.uploads import UploadError, UploadsApi
 from hotdata.exceptions import ApiException
 from hotdata.models.add_managed_table_request import AddManagedTableRequest
 from hotdata.models.async_query_response import AsyncQueryResponse
@@ -32,6 +29,12 @@
 from hotdata.models.query_response import QueryResponse
 from hotdata.models.submit_job_response import SubmitJobResponse
 from hotdata.models.table_info import TableInfo
+from hotdata.models.table_partition_key import TablePartitionKey
+from hotdata.models.table_sort_key import TableSortKey
+
+# The enriched wrapper (hotdata.uploads), NOT the generated hotdata.api class:
+# it adds the full upload_file orchestration used by upload_parquet.
+from hotdata.uploads import UploadError, UploadsApi
 from urllib3.exceptions import HTTPError as Urllib3HTTPError
 from urllib3.exceptions import ProtocolError
 
@@ -41,6 +44,7 @@
     LoadManagedTableResult,
     ManagedDatabase,
     ManagedTable,
+    TableLayout,
     api_error_message,
     enum_value,
     is_parquet_path,
@@ -286,18 +290,44 @@ def create_managed_database(
         schema: str = DEFAULT_SCHEMA,
         tables: list[str] | None = None,
         keys: dict[str, list[str]] | None = None,
+        partition_by: dict[str, Sequence[TablePartitionKey]] | None = None,
+        sorted_by: dict[str, Sequence[TableSortKey]] | None = None,
         expires_at: str | None = None,
     ) -> ManagedDatabase:
         """Create a managed database. ``keys`` maps a table to its key columns
-        (enabling delete/update/upsert on it); omitted tables are keyless."""
+        (enabling delete/update/upsert on it); omitted tables are keyless.
+
+        ``partition_by`` and ``sorted_by`` are keyed the same way — table name to
+        that table's keys, in declaration order — so a database can be created
+        with its tables already laid out. Tables absent from the mapping get no
+        layout, and a layout cannot be added afterwards: it is fixed at table
+        creation, so a table created here without one stays that way.
+        """
         keys = keys or {}
+        partition_by = partition_by or {}
+        sorted_by = sorted_by or {}
+        # A layout aimed at a table that is not being created would otherwise be
+        # dropped in silence, and the table it was meant for created flat — which
+        # is permanent, since a layout is fixed at creation with no alter path. A
+        # typo'd `keys` entry costs nothing by comparison: load_managed_table
+        # takes `key=` per call, so it can be corrected later.
+        unknown = (set(partition_by) | set(sorted_by)) - set(tables or ())
+        if unknown:
+            raise ValueError(
+                f"layout given for tables not being created: {', '.join(sorted(unknown))}"
+            )
         schemas = None
         if tables:
             schemas = [
                 DatabaseDefaultSchemaDecl(
                     name=schema,
                     tables=[
-                        DatabaseDefaultTableDecl(name=t, key=list(keys.get(t, [])))
+                        DatabaseDefaultTableDecl(
+                            name=t,
+                            key=list(keys.get(t, [])),
+                            partition_by=list(partition_by.get(t, ())) or None,
+                            sorted_by=list(sorted_by.get(t, ())) or None,
+                        )
                         for t in tables
                     ],
                 )
@@ -417,16 +447,33 @@ def add_managed_table(
         *,
         schema: str = DEFAULT_SCHEMA,
         key: list[str] | None = None,
+        partition_by: Sequence[TablePartitionKey] | None = None,
+        sorted_by: Sequence[TableSortKey] | None = None,
     ) -> ManagedTable:
         """Declare a new table on an existing managed database.
 
         The table is added empty (declared-but-unloaded); populate it with
         :meth:`load_managed_table`. Use this to evolve a managed database's
         schema after creation without recreating it. ``key`` sets the
         row-identity columns for delete/update/upsert; omit for keyless.
+
+        ``partition_by`` and ``sorted_by`` declare the table's storage layout, in
+        the order given. THIS IS THE ONLY CHANCE TO SET IT: a layout is fixed
+        when the table is created and there is no alter path, so a table declared
+        without one keeps that shape until it is recreated and its data rewritten.
+        Confirm what was applied with :meth:`managed_table_layout`.
+
+        The generated key models are passed through rather than wrapped, so the
+        transform vocabulary and field names stay exactly the API's. Both are
+        re-exported from ``hotdata_framework`` so callers need one import.
         """
         db = self._as_managed_database(database)
-        request = AddManagedTableRequest(name=table, key=list(key or []))
+        request = AddManagedTableRequest(
+            name=table,
+            key=list(key or []),
+            partition_by=list(partition_by) if partition_by else None,
+            sorted_by=list(sorted_by) if sorted_by else None,
+        )
         try:
             self._databases_api().add_database_table(db.id, schema, request)
         except ApiException as e:
@@ -439,6 +486,51 @@ def add_managed_table(
             last_sync=None,
         )
 
+    def managed_table_layout(
+        self,
+        database: str | ManagedDatabase,
+        table: str,
+        *,
+        schema: str = DEFAULT_SCHEMA,
+    ) -> TableLayout:
+        """Read back a managed table's declared storage layout.
+
+        The counterpart to the ``partition_by`` / ``sorted_by`` arguments on
+        :meth:`add_managed_table` and :meth:`create_managed_database`. Declaring a
+        layout is only half of it: it is fixed at table creation with no alter
+        path, so a caller that cares whether the layout took has to look, and a
+        caller that cannot confirm it should refuse to load rather than fill a
+        table it can never repair.
+
+        Empty lists here mean no layout was declared. That reading is sound
+        because the table is resolved through a managed database — the same fields
+        on a table discovered from an external connection are empty because its
+        layout belongs to the upstream system, which is not the same claim.
+
+        Raises KeyError when the table is not present on the database, so that
+        "no such table" is distinguishable from "declared without a layout"; the
+        two are very different for a caller deciding whether to load.
+        """
+        db = self._as_managed_database(database)
+        # Filtered server-side rather than paging iter_tables: this answers a
+        # single-table question, and a table sorting late in the listing would
+        # otherwise cost several round trips. include_columns is left off — the
+        # layout lives on the table row, not the columns.
+        resp = self._information_schema().information_schema(
+            connection_id=db.default_connection_id,
+            var_schema=schema,
+            table=table,
+            limit=1,
+        )
+        for info in resp.tables:
+            return TableLayout(
+                schema_name=schema,
+                table_name=table,
+                partition_by=list(info.partition_by or []),
+                sorted_by=list(info.sorted_by or []),
+            )
+        raise KeyError(f"{schema}.{table} is not declared on database {db.id}")
+
     def delete_managed_table(
         self,
         database: str | ManagedDatabase,

hotdata_framework/databases.py

@@ -7,6 +7,8 @@
 from typing import Any
 
 from hotdata.exceptions import ApiException
+from hotdata.models.table_partition_key import TablePartitionKey
+from hotdata.models.table_sort_key import TableSortKey
 
 DEFAULT_SCHEMA = "public"
 
@@ -33,6 +35,47 @@ def to_dict(self) -> dict[str, Any]:
         return asdict(self)
 
 
+@dataclass(frozen=True)
+class TableLayout:
+    """A managed table's declared storage layout, as the server reports it.
+
+    Both lists carry the generated `TablePartitionKey` / `TableSortKey` models,
+    in the order they were declared. A layout is fixed when the table is created
+    and cannot be altered, so reading it back is the only way to confirm what was
+    actually applied — which is why this exists as a first-class return rather
+    than a field on `ManagedTable`, whose other fields describe sync state.
+
+    Empty lists mean no layout was declared. That reading is only safe because
+    this is resolved through a MANAGED database: the same fields on a table
+    discovered from an external connection are empty because its layout belongs
+    to the upstream system, which is "not known from here" rather than
+    "confirmed none".
+    """
+
+    schema_name: str
+    table_name: str
+    partition_by: list[TablePartitionKey]
+    sorted_by: list[TableSortKey]
+
+    # NO to_dict(), unlike every other dataclass here, and deliberately so.
+    # `asdict()` would copy the pydantic key models through untouched rather than
+    # flatten them, so it would not return a plain dict. Mapping each key through
+    # its own `to_dict()` does flatten, but returns `dict[str, Any]` and adds
+    # eight errors under this package's strict mypy settings; hand-building the
+    # dict from named fields avoids that but silently drops any field a later
+    # spec adds to the key models, which is the failure this whole feature exists
+    # to prevent. A caller wanting dicts can map `k.to_dict()` itself and own
+    # that choice.
+
+    @property
+    def is_partitioned(self) -> bool:
+        return bool(self.partition_by)
+
+    @property
+    def is_sorted(self) -> bool:
+        return bool(self.sorted_by)
+
+
 @dataclass(frozen=True)
 class LoadManagedTableResult:
     connection_id: str

pyproject.toml

@@ -37,7 +37,7 @@ dependencies = [
   # uncapped floor turns someone else's release into a break in ours,
   # with no commit of our own to point at. Raise the cap deliberately, after
   # running the suite against the new minor.
-  "hotdata>=0.8.0,<0.9",
+  "hotdata>=0.9.0,<0.10",
   "pandas>=2.0",
   "pyarrow>=14.0",
 ]

tests/test_contract.py

@@ -23,6 +23,9 @@ def test_public_exports_contract():
         "QueryResult",
         "ResultSummary",
         "RunHistoryItem",
+        "TableLayout",
+        "TablePartitionKey",
+        "TableSortKey",
         "WorkspaceSelection",
         "__version__",
         "classify_sdk_error",