diff --git a/CHANGELOG.md b/CHANGELOG.md index 73f9b2d..6c08eff 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -41,7 +41,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 `dbt docs generate`. - Id-first database addressing: pin `database_id` in the profile, or let the first run create a database and print its id. -- Cross-database macros for DataFusion's SQL surface: `dateadd`, `datediff`, +- Cross-database macros for HotSQL: `dateadd`, `datediff`, `convert_timezone`. - Transient API errors (409/429/5xx) retry for ~42s via the shared `hotdata-framework` client; terminal errors fail the node immediately. diff --git a/README.md b/README.md index 024f0f4..8603af7 100644 --- a/README.md +++ b/README.md @@ -6,7 +6,7 @@ Transform data in [Hotdata](https://hotdata.dev) instant databases with [dbt](https://www.getdbt.com) — the companion to [hotdata-dlt-destination](https://github.com/hotdata-dev/hotdata-dlt-destination) for the T in ELT. -Hotdata is a managed analytics engine (Apache DataFusion, Postgres-dialect SQL) with **no DDL surface**: tables are created by loading data, not by `CREATE TABLE`. This adapter embraces that. Every model runs as the Chain pattern, entirely against the API: +Hotdata is a managed analytics engine speaking [HotSQL](https://www.hotdata.dev/docs/sql) (standard SQL with analytics extensions — if you know Postgres, you already know most of it) with **no DDL surface**: tables are created by loading data, not by `CREATE TABLE`. This adapter embraces that. Every model runs as the Chain pattern, entirely against the API: 1. the model's compiled `SELECT` executes **server-side**, 2. the result streams back as Arrow, @@ -28,7 +28,7 @@ No local database engine, no driver, no version matching — pure Python over HT ## Requirements -- Python **3.11+**, dbt-core **1.10+** +- Python **3.11+**, dbt-core **1.10+** (tested through 1.12) - A [Hotdata](https://hotdata.dev) workspace, an API key, and its workspace ID — from your Hotdata dashboard or the [Hotdata CLI](https://github.com/hotdata-dev/sdk-python). ## Install @@ -119,6 +119,10 @@ Tests, `dbt show`, analyses, and source freshness all run as plain SELECTs on th Schema evolution is additive and automatic: a model that starts producing a new column just includes it in the next load — existing data is never touched, and types can widen but never silently shrink. (`on_schema_change` is therefore ignored.) +### SQL dialect + +Write models in [HotSQL](https://www.hotdata.dev/docs/sql). It is Postgres-familiar, so SQL written for Postgres mostly runs unchanged, and the adapter overrides the cross-database macros (`dateadd`, `datediff`, `convert_timezone`) where HotSQL differs. Hotdata's query API also accepts the Postgres, DuckDB, and Snowflake dialects (translated to HotSQL server-side), but this adapter always submits model SQL as native HotSQL. + ## Feature support | Feature | Support | Notes | @@ -129,11 +133,12 @@ Schema evolution is additive and automatic: a model that starts producing a new | Tests (generic + singular) | ✅ | Run server-side; `store_failures` supported | | `dbt docs generate` | ✅ | Catalog from the managed-table API | | Source freshness | ✅ | `loaded_at_field` queries run server-side | +| Cross-database macros | ✅ | `dateadd`, `datediff`, `convert_timezone` implemented for [HotSQL](https://www.hotdata.dev/docs/sql) (`convert_timezone` is DST-aware) | | Hooks (`pre-hook`/`post-hook`, `on-run-*`) | ⚠️ | Run server-side — SELECT-shaped SQL only (no DDL exists) | | Python models | ❌ | | | Model contracts / constraints | ❌ | No DDL; dbt warns they are unenforced | | Grants | ❌ | Ignored with a warning — access is governed by workspace API keys | -| Transactions | ❌ | `begin`/`commit` are no-ops (DataFusion has no transactions) | +| Transactions | ❌ | `begin`/`commit` are no-ops (Hotdata has no transactions) | | Query cancellation | ❌ | An in-flight HTTPS query can't be interrupted client-side | ## Configuration diff --git a/dbt/adapters/hotdata/client.py b/dbt/adapters/hotdata/client.py index 8d88a8a..1a108b4 100644 --- a/dbt/adapters/hotdata/client.py +++ b/dbt/adapters/hotdata/client.py @@ -17,7 +17,7 @@ ``dbtRunner.invoke()`` in the same process gets fresh credentials and resolves fresh — nothing can serve a stale or differently-configured record. -Every SQL statement is executed server-side (Apache DataFusion, Postgres +Every SQL statement is executed server-side (HotSQL, a Postgres-familiar dialect) scoped to the resolved database, and results come back as Arrow. There is no DDL surface: tables are created by declaring them and loading parquet (``replace`` / ``append`` / ``upsert`` / ``delete`` modes). diff --git a/dbt/adapters/hotdata/column.py b/dbt/adapters/hotdata/column.py index 819b5e7..0f64df6 100644 --- a/dbt/adapters/hotdata/column.py +++ b/dbt/adapters/hotdata/column.py @@ -7,7 +7,7 @@ def dtype_from_arrow(arrow_type: pa.DataType) -> str: - """Render an Arrow type as the Postgres-surface name DataFusion presents. + """Render an Arrow type as the SQL type name HotSQL presents. Used when describing relations: the engine returns Arrow schemas, and dbt (docs, `{{ col.data_type }}`, schema tests) expects SQL type names. @@ -43,7 +43,7 @@ def dtype_from_arrow(arrow_type: pa.DataType) -> str: if ( pa.types.is_string(arrow_type) or pa.types.is_large_string(arrow_type) - # DataFusion reads loaded string columns back as Utf8View; the SQL + # The engine reads loaded string columns back as Utf8View; the SQL # name must still be a castable one, never "string_view". or pa.types.is_string_view(arrow_type) ): @@ -57,7 +57,7 @@ def dtype_from_arrow(arrow_type: pa.DataType) -> str: class HotdataColumn(Column): @classmethod def string_type(cls, size: int) -> str: - # The base class renders "character varying(256)", which DataFusion + # The base class renders "character varying(256)", which HotSQL # rejects in casts; strings are unbounded here. return "varchar" diff --git a/dbt/adapters/hotdata/connections.py b/dbt/adapters/hotdata/connections.py index 23c3741..87a865b 100644 --- a/dbt/adapters/hotdata/connections.py +++ b/dbt/adapters/hotdata/connections.py @@ -1,8 +1,8 @@ """dbt connection manager for Hotdata. There is no database driver here: a "connection" is an HTTPS client -(:class:`HotdataDbtClient`), SQL executes server-side on Apache DataFusion -(Postgres dialect) scoped to the run's instant database, and results come +(:class:`HotdataDbtClient`), SQL executes server-side as HotSQL +(Postgres-familiar) scoped to the run's instant database, and results come back as Arrow. Consequences for the dbt contract: * ``begin``/``commit`` are no-ops — the engine has no transactions. diff --git a/dbt/adapters/hotdata/impl.py b/dbt/adapters/hotdata/impl.py index d6d7a64..e37c7ab 100644 --- a/dbt/adapters/hotdata/impl.py +++ b/dbt/adapters/hotdata/impl.py @@ -1,8 +1,8 @@ """Hotdata dbt adapter. Hotdata has no DDL surface — tables are created by declaring them on the -instant database and loading parquet, and queries run server-side (Apache -DataFusion, Postgres dialect) returning Arrow. The adapter therefore keeps +instant database and loading parquet, and queries run server-side (HotSQL, +a Postgres-familiar dialect) returning Arrow. The adapter therefore keeps all metadata and materialization work in Python: * Relation listing, columns, and the docs catalog come from the diff --git a/dbt/include/hotdata/macros/utils.sql b/dbt/include/hotdata/macros/utils.sql index 35688e6..113ca64 100644 --- a/dbt/include/hotdata/macros/utils.sql +++ b/dbt/include/hotdata/macros/utils.sql @@ -1,9 +1,9 @@ {# - Cross-database macro overrides for DataFusion's Postgres surface. + Cross-database macro overrides for HotSQL. dbt's default__ implementations of dateadd/datediff emit Snowflake-style function calls (`dateadd('day', 5, x)`, `datediff('day', a, b)`) that - DataFusion does not have. These rebuild them from constructs it does have: + HotSQL does not have. These rebuild them from constructs it does have: string-built interval casts (dynamic-safe: `x + cast(concat(n, ' days') as interval)`) and integer date subtraction (`date - date` -> whole days). #} @@ -16,7 +16,7 @@ {#- dbt_date dispatch hook (packages need a root-project shim or dispatch config to reach it). NOT the Postgres double-AT TIME ZONE + cast-to-naive - pattern: on DataFusion that final cast re-renders the UTC instant, + pattern: on HotSQL that final cast re-renders the UTC instant, silently losing the shift. Here AT TIME ZONE on a naive timestamp localizes it as the source zone's wall clock (12:00 @ America/New_York -> 12:00-05:00, verified — non-UTC sources convert correctly), and diff --git a/docs/architecture.md b/docs/architecture.md index f0377de..302eeba 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -5,8 +5,8 @@ API-shaped reality. ## The constraint that shapes everything -Hotdata is a managed engine (Apache DataFusion, Postgres-dialect SQL over -HTTPS) with **no DDL surface**: +Hotdata is a managed engine ([HotSQL](https://www.hotdata.dev/docs/sql), +Postgres-familiar SQL over HTTPS) with **no DDL surface**: - Queries: `POST /query` scoped to an instant database (`X-Database-Id`), polled to completion, result fetched as Arrow. SELECT only — no