From ee09017ea27581b53ea0aeff4364a20eb5ae9702 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?St=C3=A9phane=20Manciot?= Date: Wed, 9 Sep 2026 13:13:02 +0200 Subject: [PATCH] feat(HELP-1b): the corpus guard runs parser to doc, and MATERIALIZED VIEW gets help MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Closed Issue #318 Every assertion in HelpCorpusSpec ran doc -> parser, so all nine MATERIALIZED VIEW productions shipped with no help document while the suite stayed green. The new assertion enumerates statements by walking the compiled AST package: `Statement` is sealed, so the compiler guarantees every subtype sits in one package directory and the walk is complete with no allow-list. A scan of production RESULT TYPES was tried first and rejected by review — it cannot see `MultiSearch` (`UNION ALL`), which no production returns directly, and that is the same class of hole the guard exists to close. The result-type scan survives as the failure clue and as a superset assertion over the walk; the set of productions returning a sealed trait is pinned by exact set equality, because every one of them hides its leaves. Eight MATERIALIZED VIEW documents (3 DDL + 5 DQL) with their three _index.json entries — the index update is load-bearing for the shipped jar, not just for the guard. `UNION ALL` is documented in select.json. Fixed here rather than filed: - CreateMaterializedView.sql emitted no space before REFRESH EVERY, so SHOW CREATE MATERIALIZED VIEW returned a statement no parser accepts. Six round-trip rows in ParserSpec, every one carrying a frequency, assert `Parser(stmt.sql) == Right(stmt)`. - create_table.json and create_pipeline.json published `CREATE [OR REPLACE] X [IF NOT EXISTS]`, measured rejected. - documentation/sql/materialized_views.md: the CREATE template, the elided SHOW CREATE output, `DEFAULT CURRENT_TIMESTAMP`, the qualifier semantics, the case-sensitive REFRESH EVERY unit, and the SHOW MATERIALIZED VIEWS gap. The guard cannot see command `syntax` templates; all ten written or edited here were instantiated by hand and run through the real parser. --- .../resources/help/commands/ddl/_index.json | 5 +- .../ddl/create_materialized_view.json | 93 +++++ .../help/commands/ddl/create_pipeline.json | 9 +- .../help/commands/ddl/create_table.json | 7 +- .../commands/ddl/drop_materialized_view.json | 40 +++ .../ddl/refresh_materialized_view.json | 52 +++ .../resources/help/commands/dql/_index.json | 7 +- .../dql/describe_materialized_view.json | 35 ++ .../resources/help/commands/dql/select.json | 17 +- .../dql/show_create_materialized_view.json | 29 ++ .../commands/dql/show_materialized_view.json | 36 ++ .../dql/show_materialized_view_status.json | 30 ++ .../commands/dql/show_materialized_views.json | 38 +++ .../elastic/client/help/HelpCorpusSpec.scala | 317 +++++++++++++++++- documentation/sql/materialized_views.md | 62 +++- .../elastic/sql/query/package.scala | 7 +- .../elastic/sql/parser/ParserSpec.scala | 35 ++ 17 files changed, 794 insertions(+), 25 deletions(-) create mode 100644 core/src/main/resources/help/commands/ddl/create_materialized_view.json create mode 100644 core/src/main/resources/help/commands/ddl/drop_materialized_view.json create mode 100644 core/src/main/resources/help/commands/ddl/refresh_materialized_view.json create mode 100644 core/src/main/resources/help/commands/dql/describe_materialized_view.json create mode 100644 core/src/main/resources/help/commands/dql/show_create_materialized_view.json create mode 100644 core/src/main/resources/help/commands/dql/show_materialized_view.json create mode 100644 core/src/main/resources/help/commands/dql/show_materialized_view_status.json create mode 100644 core/src/main/resources/help/commands/dql/show_materialized_views.json diff --git a/core/src/main/resources/help/commands/ddl/_index.json b/core/src/main/resources/help/commands/ddl/_index.json index 8530ccbc0..5f19dd515 100644 --- a/core/src/main/resources/help/commands/ddl/_index.json +++ b/core/src/main/resources/help/commands/ddl/_index.json @@ -10,5 +10,8 @@ "drop_watcher.json", "create_enrich_policy.json", "drop_enrich_policy.json", - "execute_enrich_policy.json" + "execute_enrich_policy.json", + "create_materialized_view.json", + "drop_materialized_view.json", + "refresh_materialized_view.json" ] diff --git a/core/src/main/resources/help/commands/ddl/create_materialized_view.json b/core/src/main/resources/help/commands/ddl/create_materialized_view.json new file mode 100644 index 000000000..b05850495 --- /dev/null +++ b/core/src/main/resources/help/commands/ddl/create_materialized_view.json @@ -0,0 +1,93 @@ +{ + "name": "CREATE MATERIALIZED VIEW", + "category": "DDL", + "shortDescription": "Create a materialized view maintained by Elasticsearch transforms", + "syntax": [ + "CREATE MATERIALIZED VIEW [IF NOT EXISTS] view_name", + " [REFRESH EVERY n time_unit]", + " [WITH (option = value, ...)]", + " AS select_statement", + "", + "-- Or replace an existing view (IF NOT EXISTS is NOT accepted on this form):", + "CREATE OR REPLACE MATERIALIZED VIEW view_name", + " [REFRESH EVERY n time_unit]", + " [WITH (option = value, ...)]", + " AS select_statement" + ], + "description": "Materialize the result of a SELECT into its own Elasticsearch index, kept up to date by a chain of transforms. A view over a single table generates one transform (source to view); a view with a JOIN also generates changelog transforms, an enrich policy, an ingest pipeline and an enrichment transform. Requires the Materialized Views extension (softclient4es-extensions) on the engine that executes the statement: the core parser accepts the statement, but an engine with no materialized-view extension registered rejects it at execution time with `Unsupported table DDL statement`.", + "clauses": [ + { + "name": "IF NOT EXISTS", + "description": "Do nothing if the view already exists. Accepted only on CREATE MATERIALIZED VIEW - the parser rejects it after CREATE OR REPLACE.", + "optional": true + }, + { + "name": "OR REPLACE", + "description": "Drop the existing view and its artifacts, then recreate them. Mutually exclusive with IF NOT EXISTS.", + "optional": true + }, + { + "name": "REFRESH EVERY", + "description": "How often the transforms look for new data: an integer, whitespace, then MILLISECOND(S), SECOND(S), MINUTE(S), HOUR(S), DAY(S), WEEK(S), MONTH(S) or YEAR(S). Must come before WITH (...). Unlike every other keyword in the dialect the UNIT is case-SENSITIVE and must be upper case, and the whitespace is required: REFRESH EVERY 30 seconds and REFRESH EVERY 30SECONDS are both rejected.", + "optional": true + }, + { + "name": "WITH (...)", + "description": "View options: delay (how long to wait for late-arriving data) and user_latency (acceptable query latency). Must come after REFRESH EVERY.", + "optional": true + }, + { + "name": "AS select_statement", + "description": "The SELECT that defines the view. WHERE, GROUP BY, aggregations and one JOIN are supported.", + "optional": false + } + ], + "examples": [ + { + "title": "Single-table view", + "description": "A view over one table generates exactly one transform - no JOIN is required", + "sql": "CREATE MATERIALIZED VIEW active_orders_mv REFRESH EVERY 30 SECONDS AS SELECT id, amount, status, created_at FROM orders WHERE status = 'active'" + }, + { + "title": "Create only if absent", + "description": "IF NOT EXISTS makes the statement a no-op when the view is already there", + "sql": "CREATE MATERIALIZED VIEW IF NOT EXISTS active_orders_mv AS SELECT id, amount FROM orders WHERE status = 'active'" + }, + { + "title": "Replace a view with a JOIN", + "description": "OR REPLACE drops the existing artifacts first; REFRESH EVERY precedes WITH (...)", + "sql": "CREATE OR REPLACE MATERIALIZED VIEW orders_with_customers_mv REFRESH EVERY 60 SECONDS WITH (delay = '1s', user_latency = '1s') AS SELECT o.id, o.amount, c.name AS customer_name FROM orders AS o JOIN customers AS c ON o.customer_id = c.id WHERE o.status = 'completed'" + }, + { + "title": "Aggregated view", + "description": "A GROUP BY adds a pivot transform to the chain", + "sql": "CREATE MATERIALIZED VIEW orders_by_city_mv AS SELECT c.city, COUNT(*) AS order_count, SUM(o.amount) AS total_amount FROM orders o JOIN customers c ON o.customer_id = c.id GROUP BY c.city" + }, + { + "title": "Quoted name", + "description": "A view name may be written bare, double-quoted or back-quoted", + "sql": "CREATE MATERIALIZED VIEW \"orders_mv\" AS SELECT id, amount FROM orders" + } + ], + "notes": [ + "Requires the Materialized Views extension - an engine without it parses the statement and then rejects it at execution time", + "Clause order is fixed: REFRESH EVERY comes before WITH (...); the reverse order is rejected", + "IF NOT EXISTS and OR REPLACE cannot be combined", + "A view name may be bare, \"double-quoted\" or `back-quoted`. A QUALIFIER behaves differently in the two spellings and the parser does not interpret it: a quoted qualifier is recorded but is NOT part of the name, so \"analytics\".\"orders_mv\" creates orders_mv, while the bare analytics.orders_mv is one name and creates the index analytics.orders_mv", + "With REFRESH EVERY and no explicit delay, the frequency must be at least 2 x (number of transforms) x 10 seconds - 20 seconds for a single-table view, 60 seconds for one JOIN with a WHERE, 80 seconds when computed columns are present", + "A JOIN view creates a watcher to re-run its enrich policies. On a cluster whose LICENCE does not include Watcher the view is still created, with a warning, and REFRESH MATERIALIZED VIEW is the manual equivalent - but a cluster with Watcher explicitly DISABLED (xpack.watcher.enabled: false) fails the CREATE and rolls back" + ], + "limitations": [ + "Only INNER and LEFT JOIN are supported; RIGHT and FULL JOIN are rejected", + "UNNEST is not supported in a materialized view definition" + ], + "seeAlso": [ + "DROP MATERIALIZED VIEW", + "REFRESH MATERIALIZED VIEW", + "SHOW MATERIALIZED VIEWS", + "SHOW CREATE MATERIALIZED VIEW", + "CREATE TABLE" + ], + "minVersion": null, + "aliases": [] +} diff --git a/core/src/main/resources/help/commands/ddl/create_pipeline.json b/core/src/main/resources/help/commands/ddl/create_pipeline.json index 2e8febae8..7dcaea495 100644 --- a/core/src/main/resources/help/commands/ddl/create_pipeline.json +++ b/core/src/main/resources/help/commands/ddl/create_pipeline.json @@ -3,7 +3,7 @@ "category": "DDL", "shortDescription": "Create an ingest pipeline for document processing", "syntax": [ - "CREATE [OR REPLACE] PIPELINE [IF NOT EXISTS] pipeline_name", + "CREATE PIPELINE [IF NOT EXISTS] pipeline_name", "WITH PROCESSORS (", " processor_definition,", " ...", @@ -16,13 +16,16 @@ "REMOVE (field = 'field_name')", "RENAME (field = 'old_name', target_field = 'new_name')", "ENRICH (policy_name = 'policy', field = 'match_field', target_field = 'enriched')", - "... (refer to documentation for full list of processors and syntax)" + "... (refer to documentation for full list of processors and syntax)", + "", + "-- Or replace an existing pipeline (IF NOT EXISTS is not accepted after OR REPLACE):", + "CREATE OR REPLACE PIPELINE pipeline_name WITH PROCESSORS ( ... )" ], "description": "Create an Elasticsearch ingest pipeline that processes documents before indexing. Pipelines can transform, enrich, and validate data.", "clauses": [ { "name": "OR REPLACE", - "description": "Replace existing pipeline if it exists", + "description": "Replace existing pipeline if it exists. Cannot be combined with IF NOT EXISTS - the parser rejects CREATE OR REPLACE PIPELINE IF NOT EXISTS.", "optional": true }, { diff --git a/core/src/main/resources/help/commands/ddl/create_table.json b/core/src/main/resources/help/commands/ddl/create_table.json index 875a0d6b8..ef23b06ef 100644 --- a/core/src/main/resources/help/commands/ddl/create_table.json +++ b/core/src/main/resources/help/commands/ddl/create_table.json @@ -3,7 +3,7 @@ "category": "DDL", "shortDescription": "Create a new table (Elasticsearch index)", "syntax": [ - "CREATE [OR REPLACE] TABLE [IF NOT EXISTS] table_name (", + "CREATE TABLE [IF NOT EXISTS] table_name (", " column_name data_type [NOT NULL] [DEFAULT value] [COMMENT 'text']", " [FIELDS (subfield_name subfield_type [OPTIONS (...)])]", " [OPTIONS (option = value, ...)]", @@ -18,6 +18,9 @@ " [aliases = (alias = value, ...)],", ")]", "", + "-- Or replace an existing table (IF NOT EXISTS is not accepted after OR REPLACE):", + "CREATE OR REPLACE TABLE table_name ( ... )", + "", "-- Or create from SELECT:", "CREATE [OR REPLACE] TABLE table_name AS SELECT ..." ], @@ -25,7 +28,7 @@ "clauses": [ { "name": "OR REPLACE", - "description": "Drop existing table before creating. If the table exists, it will be deleted first.", + "description": "Drop existing table before creating. If the table exists, it will be deleted first. Cannot be combined with IF NOT EXISTS - the parser rejects CREATE OR REPLACE TABLE IF NOT EXISTS.", "optional": true }, { diff --git a/core/src/main/resources/help/commands/ddl/drop_materialized_view.json b/core/src/main/resources/help/commands/ddl/drop_materialized_view.json new file mode 100644 index 000000000..678de3f4c --- /dev/null +++ b/core/src/main/resources/help/commands/ddl/drop_materialized_view.json @@ -0,0 +1,40 @@ +{ + "name": "DROP MATERIALIZED VIEW", + "category": "DDL", + "shortDescription": "Drop a materialized view and every artifact it deployed", + "syntax": [ + "DROP MATERIALIZED VIEW [IF EXISTS] view_name" + ], + "description": "Drop a materialized view together with everything its creation deployed: the transforms, the intermediate indices, the ingest pipelines, the enrich policies, the watcher and the view index itself. Requires the Materialized Views extension (softclient4es-extensions) on the engine that executes the statement: an engine without it rejects the statement at execution time with `Unsupported table DDL statement`.", + "clauses": [ + { + "name": "IF EXISTS", + "description": "Do not raise an error when the view does not exist.", + "optional": true + } + ], + "examples": [ + { + "title": "Drop a view", + "description": "Remove the view and its artifacts", + "sql": "DROP MATERIALIZED VIEW orders_with_customers_mv" + }, + { + "title": "Drop if it exists", + "description": "Safe in a script that may run twice", + "sql": "DROP MATERIALIZED VIEW IF EXISTS orders_with_customers_mv" + } + ], + "notes": [ + "Requires the Materialized Views extension - an engine without it parses the statement and then rejects it at execution time", + "The drop removes the transforms, intermediate indices, ingest pipelines, enrich policies, the watcher and the view index" + ], + "limitations": [], + "seeAlso": [ + "CREATE MATERIALIZED VIEW", + "SHOW MATERIALIZED VIEWS", + "DROP TABLE" + ], + "minVersion": null, + "aliases": [] +} diff --git a/core/src/main/resources/help/commands/ddl/refresh_materialized_view.json b/core/src/main/resources/help/commands/ddl/refresh_materialized_view.json new file mode 100644 index 000000000..69d915aff --- /dev/null +++ b/core/src/main/resources/help/commands/ddl/refresh_materialized_view.json @@ -0,0 +1,52 @@ +{ + "name": "REFRESH MATERIALIZED VIEW", + "category": "DDL", + "shortDescription": "Force a materialized view to pick up new data now", + "syntax": [ + "REFRESH MATERIALIZED VIEW [IF EXISTS] view_name [WITH SCHEDULE NOW]" + ], + "description": "Refresh a materialized view immediately: the changelog indices are refreshed and the enrich policies re-executed - exactly the work the view's watcher performs on a schedule. Use it on a cluster where Watcher is unavailable, or to make a change visible without waiting for the next refresh interval. Requires the Materialized Views extension (softclient4es-extensions) on the engine that executes the statement: an engine without it rejects the statement at execution time with `Unsupported table DDL statement`.", + "clauses": [ + { + "name": "IF EXISTS", + "description": "Do not raise an error when the view does not exist.", + "optional": true + }, + { + "name": "WITH SCHEDULE NOW", + "description": "Also schedule the view's transforms for immediate execution instead of waiting for their next run.", + "optional": true + } + ], + "examples": [ + { + "title": "Refresh a view", + "description": "Re-run the changelogs and enrich policies", + "sql": "REFRESH MATERIALIZED VIEW orders_with_customers_mv" + }, + { + "title": "Refresh and schedule now", + "description": "Also trigger the transforms immediately", + "sql": "REFRESH MATERIALIZED VIEW orders_with_customers_mv WITH SCHEDULE NOW" + }, + { + "title": "Refresh if it exists", + "description": "No error when the view is absent", + "sql": "REFRESH MATERIALIZED VIEW IF EXISTS orders_with_customers_mv" + } + ], + "notes": [ + "Requires the Materialized Views extension - an engine without it parses the statement and then rejects it at execution time", + "This is the documented manual equivalent of the view's watcher, and the supported path on a cluster whose licence does not include Watcher" + ], + "limitations": [ + "A single-table view deploys no enrich policy and no watcher, so it has nothing to re-execute here - it is refreshed by its own transform on its REFRESH EVERY schedule" + ], + "seeAlso": [ + "CREATE MATERIALIZED VIEW", + "SHOW MATERIALIZED VIEW STATUS", + "DROP MATERIALIZED VIEW" + ], + "minVersion": null, + "aliases": [] +} diff --git a/core/src/main/resources/help/commands/dql/_index.json b/core/src/main/resources/help/commands/dql/_index.json index b950d9e0e..170f1d8c8 100644 --- a/core/src/main/resources/help/commands/dql/_index.json +++ b/core/src/main/resources/help/commands/dql/_index.json @@ -14,5 +14,10 @@ "show_enrich_policy.json", "show_cluster_name.json", "show_license.json", - "refresh_license.json" + "refresh_license.json", + "show_materialized_view.json", + "show_materialized_views.json", + "show_materialized_view_status.json", + "show_create_materialized_view.json", + "describe_materialized_view.json" ] diff --git a/core/src/main/resources/help/commands/dql/describe_materialized_view.json b/core/src/main/resources/help/commands/dql/describe_materialized_view.json new file mode 100644 index 000000000..a43d8deff --- /dev/null +++ b/core/src/main/resources/help/commands/dql/describe_materialized_view.json @@ -0,0 +1,35 @@ +{ + "name": "DESCRIBE MATERIALIZED VIEW", + "category": "DQL", + "shortDescription": "Show the columns of a materialized view index", + "syntax": [ + "{DESCRIBE | DESC} MATERIALIZED VIEW view_name" + ], + "description": "Show the schema of the index a materialized view writes to: one row per column, with its type, nullability, default, comment and script. Requires the Materialized Views extension (softclient4es-extensions) on the engine that executes the statement: an engine without it rejects the statement at execution time with `Unsupported table DDL statement`.", + "clauses": [], + "examples": [ + { + "title": "Describe a view", + "description": "List the view's columns and types", + "sql": "DESCRIBE MATERIALIZED VIEW orders_with_customers_mv" + }, + { + "title": "DESC abbreviation", + "description": "DESC is accepted wherever DESCRIBE is", + "sql": "DESC MATERIALIZED VIEW orders_with_customers_mv" + } + ], + "notes": [ + "Requires the Materialized Views extension - an engine without it parses the statement and then rejects it at execution time", + "IF EXISTS is not accepted here - the parser rejects DESCRIBE MATERIALIZED VIEW IF EXISTS view_name", + "Computed columns of the view appear with the script that produces them" + ], + "limitations": [], + "seeAlso": [ + "DESCRIBE TABLE", + "SHOW MATERIALIZED VIEW", + "SHOW CREATE MATERIALIZED VIEW" + ], + "minVersion": null, + "aliases": [] +} diff --git a/core/src/main/resources/help/commands/dql/select.json b/core/src/main/resources/help/commands/dql/select.json index f89821422..492a122e9 100644 --- a/core/src/main/resources/help/commands/dql/select.json +++ b/core/src/main/resources/help/commands/dql/select.json @@ -10,7 +10,11 @@ "[GROUP BY columns]", "[HAVING condition]", "[ORDER BY columns [ASC|DESC]]", - "[LIMIT n [OFFSET m]]" + "[LIMIT n [OFFSET m]]", + "", + "-- Two or more SELECTs may be concatenated. UNION ALL is the ONLY spelling accepted:", + "-- bare UNION (de-duplicating) is rejected.", + "SELECT ... UNION ALL SELECT ..." ], "description": "The SELECT statement retrieves rows from Elasticsearch indices. It supports most standard SQL features including joins, aggregations, and subqueries.", "clauses": [ @@ -51,6 +55,11 @@ "optional": true, "modifiers": ["ASC", "DESC", "NULLS FIRST", "NULLS LAST"] }, + { + "name": "UNION ALL", + "description": "Concatenate the rows of two or more SELECT statements. Only UNION ALL is accepted - bare UNION, which would de-duplicate, is rejected by the parser.", + "optional": true + }, { "name": "LIMIT/OFFSET", "description": "Limit number of returned rows", @@ -81,6 +90,12 @@ "description": "Get the 10 most recent products", "sql": "SELECT name, created_at\nFROM products\nORDER BY created_at DESC\nLIMIT 10", "output": null + }, + { + "title": "Concatenate two result sets", + "description": "UNION ALL keeps every row of both sides; bare UNION is not accepted", + "sql": "SELECT id, amount FROM orders UNION ALL SELECT id, amount FROM archived_orders", + "output": null } ], "notes": [ diff --git a/core/src/main/resources/help/commands/dql/show_create_materialized_view.json b/core/src/main/resources/help/commands/dql/show_create_materialized_view.json new file mode 100644 index 000000000..1e86afc31 --- /dev/null +++ b/core/src/main/resources/help/commands/dql/show_create_materialized_view.json @@ -0,0 +1,29 @@ +{ + "name": "SHOW CREATE MATERIALIZED VIEW", + "category": "DQL", + "shortDescription": "Show the CREATE statement that defines a materialized view", + "syntax": [ + "SHOW CREATE MATERIALIZED VIEW view_name" + ], + "description": "Return the CREATE MATERIALIZED VIEW statement the engine recorded for a view, rendered from the stored definition. The rendered statement is itself accepted by the parser, so it can be replayed against another cluster. Requires the Materialized Views extension (softclient4es-extensions) on the engine that executes the statement: an engine without it rejects the statement at execution time with `Unsupported table DDL statement`.", + "clauses": [], + "examples": [ + { + "title": "Show the definition", + "description": "Recover the statement that created a view", + "sql": "SHOW CREATE MATERIALIZED VIEW orders_with_customers_mv" + } + ], + "notes": [ + "Requires the Materialized Views extension - an engine without it parses the statement and then rejects it at execution time", + "The rendered statement is normalised, not the original text: clauses appear in canonical order and identifiers in canonical spelling" + ], + "limitations": [], + "seeAlso": [ + "SHOW MATERIALIZED VIEW", + "CREATE MATERIALIZED VIEW", + "SHOW CREATE TABLE" + ], + "minVersion": null, + "aliases": [] +} diff --git a/core/src/main/resources/help/commands/dql/show_materialized_view.json b/core/src/main/resources/help/commands/dql/show_materialized_view.json new file mode 100644 index 000000000..8ccc7fd3d --- /dev/null +++ b/core/src/main/resources/help/commands/dql/show_materialized_view.json @@ -0,0 +1,36 @@ +{ + "name": "SHOW MATERIALIZED VIEW", + "category": "DQL", + "shortDescription": "Show the definition and deployed artifacts of one materialized view", + "syntax": [ + "SHOW MATERIALIZED VIEW view_name" + ], + "description": "Show the metadata recorded for one materialized view: its source tables, refresh frequency, delay, the transforms, ingest pipelines and enrich policies it deployed, and whether automatic refresh is available. Requires the Materialized Views extension (softclient4es-extensions) on the engine that executes the statement: an engine without it rejects the statement at execution time with `Unsupported table DDL statement`.", + "clauses": [], + "examples": [ + { + "title": "Show one view", + "description": "Inspect a view's definition and artifacts", + "sql": "SHOW MATERIALIZED VIEW orders_with_customers_mv" + }, + { + "title": "Quoted view name", + "description": "A view name may be double-quoted or back-quoted", + "sql": "SHOW MATERIALIZED VIEW \"orders_with_customers_mv\"" + } + ], + "notes": [ + "Requires the Materialized Views extension - an engine without it parses the statement and then rejects it at execution time", + "IF EXISTS is not accepted here - the parser rejects SHOW MATERIALIZED VIEW IF EXISTS view_name", + "Use SHOW MATERIALIZED VIEWS (plural) to list every view" + ], + "limitations": [], + "seeAlso": [ + "SHOW MATERIALIZED VIEWS", + "SHOW MATERIALIZED VIEW STATUS", + "SHOW CREATE MATERIALIZED VIEW", + "DESCRIBE MATERIALIZED VIEW" + ], + "minVersion": null, + "aliases": [] +} diff --git a/core/src/main/resources/help/commands/dql/show_materialized_view_status.json b/core/src/main/resources/help/commands/dql/show_materialized_view_status.json new file mode 100644 index 000000000..e579525f6 --- /dev/null +++ b/core/src/main/resources/help/commands/dql/show_materialized_view_status.json @@ -0,0 +1,30 @@ +{ + "name": "SHOW MATERIALIZED VIEW STATUS", + "category": "DQL", + "shortDescription": "Show the runtime state of a materialized view's transforms", + "syntax": [ + "SHOW MATERIALIZED VIEW STATUS view_name" + ], + "description": "Show one row per deployed step of a materialized view, with the live state of the transform behind it: transform_id, step_number, step_type, source_table, target_table, latency_seconds, state, documents_indexed, documents_processed, failures, lag, processing_time_ms and last_checkpoint. Requires the Materialized Views extension (softclient4es-extensions) on the engine that executes the statement: an engine without it rejects the statement at execution time with `Unsupported table DDL statement`.", + "clauses": [], + "examples": [ + { + "title": "Show a view's status", + "description": "One row per step that has produced statistics", + "sql": "SHOW MATERIALIZED VIEW STATUS orders_with_customers_mv" + } + ], + "notes": [ + "Requires the Materialized Views extension - an engine without it parses the statement and then rejects it at execution time", + "Only steps whose transform has produced statistics are listed: an empty result means no step has checkpointed yet, most often because the source index has no ingest-timestamp column to synchronise on", + "STATUS is matched before BOTH the plural and the singular forms, so SHOW MATERIALIZED VIEW STATUS x is never read as SHOW MATERIALIZED VIEW on a view named STATUS" + ], + "limitations": [], + "seeAlso": [ + "SHOW MATERIALIZED VIEW", + "REFRESH MATERIALIZED VIEW", + "SHOW WATCHER STATUS" + ], + "minVersion": null, + "aliases": [] +} diff --git a/core/src/main/resources/help/commands/dql/show_materialized_views.json b/core/src/main/resources/help/commands/dql/show_materialized_views.json new file mode 100644 index 000000000..81705eaa6 --- /dev/null +++ b/core/src/main/resources/help/commands/dql/show_materialized_views.json @@ -0,0 +1,38 @@ +{ + "name": "SHOW MATERIALIZED VIEWS", + "category": "DQL", + "shortDescription": "List every materialized view", + "syntax": [ + "SHOW MATERIALIZED VIEWS" + ], + "description": "The plural form of SHOW MATERIALIZED VIEW: list every materialized view. The statement takes no argument. NOT IMPLEMENTED TODAY - see `limitations`: the statement parses, but no engine executes it. Use SHOW MATERIALIZED VIEW view_name for a view you can name.", + "clauses": [], + "examples": [ + { + "title": "List all views", + "description": "The intended form. It parses; see `limitations` - no engine executes it yet", + "sql": "SHOW MATERIALIZED VIEWS" + }, + { + "title": "What works today", + "description": "Name the view you want to inspect", + "sql": "SHOW MATERIALIZED VIEW orders_with_customers_mv" + } + ], + "notes": [ + "Requires the Materialized Views extension - an engine without it parses the statement and then rejects it at execution time", + "See `limitations`: this plural form is not implemented by any engine today", + "The statement takes no argument: SHOW MATERIALIZED VIEWS view_name is rejected. Use the singular SHOW MATERIALIZED VIEW for one view", + "There is no LIKE pattern on this form" + ], + "limitations": [ + "NOT IMPLEMENTED. The statement parses, and the Materialized Views extension claims it (`canHandle` accepts every materialized-view statement), but its `execute` has no branch for the plural form and answers `400 Unsupported statement for Materialized Views extension`. Documented rather than hidden: an example that routes around a gap with the gap recorded nowhere turns an engine defect into invisible documentation degradation. Until it is implemented, name the view: SHOW MATERIALIZED VIEW view_name" + ], + "seeAlso": [ + "SHOW MATERIALIZED VIEW", + "CREATE MATERIALIZED VIEW", + "SHOW TABLES" + ], + "minVersion": null, + "aliases": [] +} diff --git a/core/src/test/scala/app/softnetwork/elastic/client/help/HelpCorpusSpec.scala b/core/src/test/scala/app/softnetwork/elastic/client/help/HelpCorpusSpec.scala index 311f92791..9b825f6b0 100644 --- a/core/src/test/scala/app/softnetwork/elastic/client/help/HelpCorpusSpec.scala +++ b/core/src/test/scala/app/softnetwork/elastic/client/help/HelpCorpusSpec.scala @@ -18,12 +18,15 @@ package app.softnetwork.elastic.client.help import app.softnetwork.elastic.sql.SQLKeywords import app.softnetwork.elastic.sql.parser.Parser +import app.softnetwork.elastic.sql.query.{SearchStatement, Statement} import org.json4s.{DefaultFormats, Formats, JString, JValue} import org.json4s.native.JsonMethods.parse import org.scalatest.flatspec.AnyFlatSpec import org.scalatest.matchers.should.Matchers import java.io.File +import java.lang.reflect.{Modifier, ParameterizedType} +import java.util.Locale import scala.io.Source import scala.util.{Failure, Success, Try} @@ -170,7 +173,7 @@ class HelpCorpusSpec extends AnyFlatSpec with Matchers { */ private def topicOfJson(jdoc: JValue, where: String): String = jdoc \ "name" match { - case JString(n) => n.toUpperCase + case JString(n) => n.toUpperCase(Locale.ROOT) case other => fail(s"help document has no string `name` field: $where ($other)") } @@ -612,6 +615,318 @@ class HelpCorpusSpec extends AnyFlatSpec with Matchers { } } + // --- HELP-1b: the STATEMENT parser -> doc direction ---------------------------------- + // + // The assertion above closes the FUNCTION half of the contract. Its statement half was open, and + // that is how all nine MATERIALIZED VIEW productions shipped with no help document at all - the + // largest hole in the corpus, invisible to every doc -> parser assertion in this file. + // + // AD-2 (story HELP-1b) - productions are enumerated by their AST RESULT TYPE, through reflection + // over `Parser`'s own zero-argument `PackratParser[T]` members. There is no runtime registry of + // productions, so the only alternative is a hand-written list of the ~46 method names - an + // allow-list, which is exactly the artefact that let MATERIALIZED VIEW drift, and which stops + // guarding the day production 47 lands. Deriving from the result type also collapses the five + // `createOrReplaceX` / `createX` pairs for free: they share one AST type and therefore need one + // document, which is a consequence of the rule rather than a rule of its own. + + /** The two raw parser types a production can carry. Matched by NAME: `PackratParser` and `Parser` + * are inner classes of the `PackratParsers` / `Parsers` traits, so neither has a stable + * `classOf[...]` spelling. Both are matched because "a production is a `PackratParser`" is a + * convention of this file, not a rule the compiler enforces. + */ + private val ParserRawTypeNames: Set[String] = Set( + "scala.util.parsing.combinator.PackratParsers$PackratParser", + "scala.util.parsing.combinator.Parsers$Parser" + ) + + /** (production name, AST result type) for every `PackratParser[T <: Statement]` on `Parser`. */ + private def statementProductions: Seq[(String, Class[_])] = + Parser.getClass.getMethods.toSeq + .filterNot(m => m.isSynthetic || m.isBridge) + .filter(_.getParameterCount == 0) + .flatMap { m => + m.getGenericReturnType match { + case pt: ParameterizedType if ParserRawTypeNames.contains(pt.getRawType.getTypeName) => + pt.getActualTypeArguments.headOption.collect { + case c: Class[_] if classOf[Statement].isAssignableFrom(c) => m.getName -> c + } + case _ => None + } + } + .sortBy(_._1) + + /** The productions whose result type is one of the SEALED TRAITS, so they name no statement a + * user can type. Today: the four dispatchers plus `searchStatement`. + */ + private def abstractStatementProductions: Seq[(String, Class[_])] = + statementProductions.filter { case (_, c) => + c.isInterface || Modifier.isAbstract(c.getModifiers) + } + + private def concreteStatementProductions: Seq[(String, Class[_])] = + statementProductions.filterNot { case (_, c) => + c.isInterface || Modifier.isAbstract(c.getModifiers) + } + + /** 🔴 The production scan alone is INCOMPLETE, and the gap is not hypothetical. + * + * `searchStatement` is `rep1sep(single, union)` (`Parser.scala:99-102`), so `SELECT ... UNION + * ALL SELECT ...` yields `MultiSearch` - a concrete, user-typeable statement produced by NO + * `def` of its own, whose production returns the abstract `SearchStatement`. A scan of + * production RESULT TYPES cannot see it, and the first version of this guard shipped green with + * `UNION ALL` undocumented. Whenever a production returns an abstract type, its leaves hide + * behind it. + * + * `Statement` is `sealed`, so the compiler guarantees every subtype is declared in the same file + * and therefore compiled into the same package directory. Listing that one directory is the + * complete enumeration, and it needs no list of names. + * + * A non-`file:` classpath entry FAILS rather than skips: this spec is the deliverable, and a + * silent skip here would restore exactly the hole it exists to close. + */ + private def astStatementTypes: Seq[Class[_]] = { + val pkg = classOf[Statement].getName.split('.').init.mkString("/") + val loader = classOf[Statement].getClassLoader + val url = Option(loader.getResource(pkg)).getOrElse( + fail(s"the AST package `$pkg` is not on the test classpath") + ) + if (url.getProtocol != "file") + fail( + s"the AST package `$pkg` resolves to a ${url.getProtocol} URL ($url). This walk lists the " + + "compiled classes of a SEALED hierarchy and needs a directory; teach it to read the " + + "archive rather than letting the statement guard go vacuous." + ) + val dir = new File(url.toURI) + entries(dir) + .filter(f => f.isFile && f.getName.endsWith(".class")) + .map(f => s"${pkg.replace('/', '.')}.${f.getName.stripSuffix(".class")}") + // The `Option[Class[_]]` needs its type written out: on the 2.12 leg the existential defeats + // `flatMap`'s inference (`no type parameters for method flatMap ... forSome { type ?0 }`). + .flatMap { n => + val loaded: Option[Class[_]] = Try(Class.forName(n, false, loader)).toOption + loaded.toSeq + } + .filter(c => classOf[Statement].isAssignableFrom(c)) + .filterNot(c => c.isInterface || Modifier.isAbstract(c.getModifiers)) + .distinct + } + + /** The production that yields each concrete statement type, for the failure clue. A type with no + * production of its own (`MultiSearch`) is produced inside a combinator. + */ + private def producersOf(c: Class[_]): String = + concreteStatementProductions.collect { case (n, t) if t == c => n }.sorted match { + case Nil => "(no production of its own - built inside a combinator)" + case ns => ns.mkString(", ") + } + + /** `app.softnetwork.elastic.sql.query.package$ShowMaterializedViews$` -> `ShowMaterializedViews`. + * The AST lives in a package object, so every class name carries a `package$` prefix, and a + * Scala `case object` adds a trailing `$`. `getSimpleName` is deliberately NOT used: it is + * specified in terms of the SOURCE name and has historically thrown `InternalError` on + * Scala-shaped nested names. + */ + private def astSimpleName(c: Class[_]): String = { + val last = c.getName.split('.').last.stripSuffix("$") + val i = last.lastIndexOf('$') + if (i >= 0) last.substring(i + 1) else last + } + + /** `ShowMaterializedViewStatus` -> `SHOW MATERIALIZED VIEW STATUS`. + * + * The second alternative handles an acronym followed by a word (`ShowDDLStatement` -> `SHOW DDL + * STATEMENT`); without it the phrase would read `SHOW DDLSTATEMENT` and demand a document under + * that name. `Locale.ROOT` because a Turkish default locale uppercases `i` to `\u0130`, which + * would never equal the ASCII `name` field of any document. + */ + private def camelToStatementPhrase(n: String): String = + n.split("(?<=[a-z0-9])(?=[A-Z])|(?<=[A-Z])(?=[A-Z][a-z])") + .mkString(" ") + .toUpperCase(Locale.ROOT) + + /** Every concrete `SearchStatement` IS the SELECT statement - `SingleSearch`, `MultiSearch` + * (`UNION ALL`) and `SelectStatement` are its three leaves. This is a STRUCTURAL rule, not an + * exception list: a fourth leaf added tomorrow resolves to `SELECT` with no edit here, which is + * what keeps `MultiSearch`-shaped holes closed. + */ + private def isSearchStatement(c: Class[_]): Boolean = + classOf[SearchStatement].isAssignableFrom(c) + + /** AD-3 - the ONE AST type whose class name is not the statement phrase a user types and which + * the structural rule above does not reach. `FromlessSelect` (issue #251's `SELECT 1`) is + * deliberately NOT a `SearchStatement` - it is its own `DqlStatement` - so it needs the entry. + * + * What the two assertions below actually enforce, stated exactly: a key must name a live + * statement type, and its CamelCase phrase must not already be documented. That stops a stale + * key and stops a reroute of a still-documented statement. It does NOT stop an author who + * deletes a document and adds an override for it in the same edit - that is a source change + * saying what it does, and review is what catches it. + */ + private val StatementTopicOverrides: Map[String, String] = Map( + "FromlessSelect" -> "SELECT" + ) + + private def topicOfAst(c: Class[_]): String = { + val n = astSimpleName(c) + if (isSearchStatement(c)) "SELECT" + else StatementTopicOverrides.getOrElse(n, camelToStatementPhrase(n)) + } + + "The statement grammar" should "expose every production's AST type to reflection" in { + // AD-2 rests entirely on the generic `Signature` attribute surviving compilation. If it ever + // stopped being emitted, `getGenericReturnType` would hand back the ERASED `PackratParser` and + // the assertion below would enumerate NOTHING while staying green - the worst possible failure + // mode for a guard. Assert the mechanism itself, over every production, not just the ones that + // happen to return a statement. + val erased = Parser.getClass.getMethods.toSeq + .filterNot(m => m.isSynthetic || m.isBridge) + .filter(m => m.getParameterCount == 0 && ParserRawTypeNames.contains(m.getReturnType.getName)) + withClue("no production returns a parser type - reflection reached the wrong class\n") { + erased should not be empty + } + val unparameterised = erased.filterNot(_.getGenericReturnType.isInstanceOf[ParameterizedType]) + withClue( + "these productions expose no generic type argument, so the parser -> doc assertion cannot " + + s"see them: ${unparameterised.map(_.getName).sorted.mkString(", ")}\n" + ) { + unparameterised shouldBe empty + } + // The set dropped as abstract is asserted EXACTLY, in both directions. An inclusion test would + // let a new abstract-typed production join silently - and every abstract production hides its + // concrete leaves from a result-type scan, which is precisely how `MultiSearch` (`UNION ALL`) + // slipped past `searchStatement`. A new entry here must be accompanied by a check that the + // package walk below still reaches its leaves. + val expectedAbstract = + Set("statement", "dqlStatement", "ddlStatement", "dmlStatement", "searchStatement") + withClue( + "the set of productions returning a SEALED TRAIT has changed. Every one of them hides its " + + "concrete leaves from a result-type scan; confirm the package walk reaches them, then " + + s"update this set: ${abstractStatementProductions.map(_._1).sorted.mkString(", ")}\n" + ) { + abstractStatementProductions.map(_._1).toSet shouldBe expectedAbstract + } + withClue("no concrete statement production survived - the assertion below would be vacuous\n") { + concreteStatementProductions should not be empty + } + // The package walk must be a strict SUPERSET of the production scan: it is the enumeration the + // guard actually uses, and this is what proves it did not lose anything the productions name. + val fromProductions = concreteStatementProductions.map(_._2).toSet + withClue( + "the AST package walk missed statement types the productions return: " + + s"${fromProductions.diff(astStatementTypes.toSet).map(astSimpleName).mkString(", ")}\n" + ) { + fromProductions.diff(astStatementTypes.toSet) shouldBe empty + } + } + + it should "keep the statement-topic override map honest" in { + withClue( + "the help SOURCE tree was not found; the override honesty check resolves against documents " + + s"and cannot run on the classpath view alone. Working directory: ${new File(".").getAbsolutePath}\n" + ) { + sourceRoot("commands") should not be empty + } + val produced = astStatementTypes.map(astSimpleName).toSet + withClue( + "these override keys name no statement type - a stale entry can only hide a missing " + + s"document: ${StatementTopicOverrides.keySet.diff(produced).mkString(", ")}\n" + ) { + StatementTopicOverrides.keySet.diff(produced) shouldBe empty + } + // An override may only exist where the DERIVATIONAL rule has nothing to resolve. Without this, + // adding `"CreateTable" -> "SELECT"` would be a legal entry (the key IS a real type) that + // silently redirects CREATE TABLE onto `select.json` and absorbs the deletion of + // `create_table.json`. + val documented = sourceDocsUnder("commands").map(_._2).toSet + val reroutes = + StatementTopicOverrides.keySet.filter(k => documented.contains(camelToStatementPhrase(k))) + withClue( + "these override keys already resolve through the CamelCase rule, so the override can only " + + s"redirect a documented statement onto another document: ${reroutes.mkString(", ")}\n" + ) { + reroutes shouldBe empty + } + // The structural SELECT rule must stay a rule about SEARCH statements, not a synonym for + // "everything": if `SearchStatement` ever acquired a non-SELECT leaf this would need revisiting. + withClue( + "every statement type now claims to be a SELECT - the structural rule has collapsed\n" + ) { + astStatementTypes.filterNot(isSearchStatement) should not be empty + } + } + + "Every statement the parser accepts" should "have a command help document" in { + // AD-4 - the SOURCE view alone, for the reason recorded on the union comment at the top of this + // file: here documents are the RESOLUTION TARGET, so a stale `target/` copy would REMOVE a + // failure. `docsUnder` would make this assertion weaker the fuller the classpath view is. + withClue( + "the help SOURCE tree was not found; a parser -> doc assertion cannot run against the " + + s"classpath view alone (AD-2b). Working directory: ${new File(".").getAbsolutePath}\n" + ) { + sourceRoot("commands") should not be empty + } + withClue("the AST package walk found no statement type - this assertion would be vacuous\n") { + astStatementTypes should not be empty + } + val documented = sourceDocsUnder("commands").map(_._2).toSet + val undocumented = astStatementTypes + .map(c => (topicOfAst(c), s"${astSimpleName(c)} [${producersOf(c)}]")) + .filterNot { case (topic, _) => documented.contains(topic) } + .groupBy(_._1) + .map { case (topic, ts) => s" $topic <- ${ts.map(_._2).sorted.mkString(", ")}" } + .toSeq + .sorted + withClue( + "the parser accepts these statements and the corpus documents none of them - `HELP ` " + + "returns nothing for a statement the engine runs. Add a document under " + + "`help/commands/{ddl,dml,dql}/` AND its `_index.json` entry (`loadResourceDirectory` reads " + + "only what the index names, so the index update is load-bearing for the SHIPPED JAR), with " + + "`category` matching the directory:\n" + undocumented.mkString("\n") + "\n" + ) { + undocumented shouldBe empty + } + } + + "Every command help document" should "declare the category of the directory it lives in" in { + // `parseCategory` (`HelpJsonLoader:196-203`) `toUpperCase`s the field and falls through to + // `HelpCategory.Functions` for ANYTHING it does not recognise - so a typo, or a document filed + // in the wrong directory, mis-files the topic in the REPL listing with no error anywhere. The + // statement guard above cannot see it either: `documented` is a union across ddl/dml/dql. + val wrong = (for { + root <- sourceRoot("commands").toSeq + cat <- dirsOf(root) + (rel, doc) <- docsOf(cat).map(d => (s"commands/${cat.getName}/${d.getName}", d)) + declared = stringAt(json(doc), "category", rel) + if declared.toUpperCase(Locale.ROOT) != cat.getName.toUpperCase(Locale.ROOT) + } yield s" $rel declares `$declared` but lives in `${cat.getName}`").distinct + withClue( + "these command documents declare a category that is not their directory; an unrecognised " + + "category silently becomes `HelpCategory.Functions`:\n" + wrong.mkString("\n") + "\n" + ) { + wrong shouldBe empty + } + } + + it should "publish at least one example" in { + // A three-field stub satisfies the statement guard, the `loadAll()` containment assertion and + // the example probe (which iterates an EMPTY array happily). Requiring one example is what + // makes "documented" mean something: the example probe then has to run the parser over it. + val exampleless = (for { + root <- sourceRoot("commands").toSeq + cat <- dirsOf(root) + (rel, doc) <- docsOf(cat).map(d => (s"commands/${cat.getName}/${d.getName}", d)) + if (json(doc) \ "examples").children.isEmpty + } yield s" $rel").distinct + withClue( + "these command documents publish no example, so nothing about them is ever run through the " + + "parser - `HELP ` shows syntax nobody has executed:\n" + exampleless.mkString( + "\n" + ) + "\n" + ) { + exampleless shouldBe empty + } + } + "Every seeAlso pointer" should "name a documented topic" in { // `HelpDatabase.getHelp` can only resolve a topic that has a document (the `aliases` map is // built empty today), so a `seeAlso` naming an accepted ALIAS - `TO_DATE`, `TO_TIMESTAMP` - diff --git a/documentation/sql/materialized_views.md b/documentation/sql/materialized_views.md index 10540c0a4..37a73d14f 100644 --- a/documentation/sql/materialized_views.md +++ b/documentation/sql/materialized_views.md @@ -71,17 +71,39 @@ Rollback is automatic on deployment failure. ### Syntax ```sql -CREATE [OR REPLACE] MATERIALIZED VIEW [IF NOT EXISTS] view_name +CREATE MATERIALIZED VIEW [IF NOT EXISTS] view_name [REFRESH EVERY interval time_unit] [WITH (option = value [, ...])] AS select_statement ``` +`OR REPLACE` is a **separate form**, and it does **not** accept `IF NOT EXISTS` — the parser rejects +`CREATE OR REPLACE MATERIALIZED VIEW IF NOT EXISTS ...`: + +```sql +CREATE OR REPLACE MATERIALIZED VIEW view_name +[REFRESH EVERY interval time_unit] +[WITH (option = value [, ...])] +AS select_statement +``` + +The clause order is fixed: `REFRESH EVERY` comes **before** `WITH (...)`. The reverse order is +rejected. + +`view_name` may be written bare (`orders_mv`), double-quoted (`"orders_mv"`) or back-quoted +(`` `orders_mv` ``) — all three name the same view. + +> ⚠️ A **qualifier** is not interpreted, and the two spellings are **not** equivalent. The parser +> records a *quoted* qualifier without making it part of the name, so +> `CREATE MATERIALIZED VIEW "analytics"."orders_mv"` creates the view **`orders_mv`**; a *bare* +> dotted name is a single legal index name, so `CREATE MATERIALIZED VIEW analytics.orders_mv` +> creates the view **`analytics.orders_mv`**. The two statements create two different indices. + | Component | Required | Description | |--------------------|----------|----------------------------------------------------------------| | `view_name` | Yes | Unique name for the materialized view | -| `OR REPLACE` | No | Replace existing view (drops and recreates artifacts) | -| `IF NOT EXISTS` | No | Skip creation if view already exists | +| `OR REPLACE` | No | Replace existing view (drops and recreates artifacts). Cannot be combined with `IF NOT EXISTS` | +| `IF NOT EXISTS` | No | Skip creation if view already exists. Accepted only on `CREATE MATERIALIZED VIEW`, never after `OR REPLACE` | | `REFRESH EVERY` | No | Automatic refresh interval (default: engine-defined) | | `WITH (...)` | No | Additional options (see below) | | `AS select` | Yes | The SELECT query defining the view | @@ -98,6 +120,10 @@ REFRESH EVERY 1 HOUR **Supported time units:** `MILLISECOND(S)`, `SECOND(S)`, `MINUTE(S)`, `HOUR(S)`, `DAY(S)`, `WEEK(S)`, `MONTH(S)`, `YEAR(S)` +> ⚠️ Unlike every other keyword in the dialect, the **unit is case-sensitive** and must be upper +> case, and the whitespace between the number and the unit is required: `REFRESH EVERY 30 seconds` +> and `REFRESH EVERY 30SECONDS` are both rejected, while `refresh every 30 SECONDS` is accepted. + ### Options | Option | Type | Description | Example | @@ -328,16 +354,13 @@ SHOW CREATE MATERIALIZED VIEW orders_with_customers_mv; Returns: ```sql -CREATE OR REPLACE MATERIALIZED VIEW orders_with_customers_mv -REFRESH EVERY 8 SECONDS -WITH (delay = '1s', user_latency = '1s') -AS -SELECT o.id, o.amount, c.name AS customer_name, c.email, ... -FROM orders AS o -JOIN customers AS c ON o.customer_id = c.id -WHERE o.status = 'completed' +CREATE OR REPLACE MATERIALIZED VIEW orders_with_customers_mv REFRESH EVERY 8 SECONDS WITH (delay = '1s', user_latency = '1s') AS SELECT o.id, o.amount, c.name AS customer_name, c.email FROM orders AS o JOIN customers AS c ON o.customer_id = c.id WHERE o.status = 'completed' ``` +The statement is rendered from the stored definition, not replayed verbatim: clauses come back in +canonical order and on one line. It is itself accepted by the parser, so it can be run against +another cluster as-is. + --- ## SHOW MATERIALIZED VIEW STATUS @@ -377,7 +400,10 @@ SHOW MATERIALIZED VIEW STATUS orders_with_customers_mv; SHOW MATERIALIZED VIEWS; ``` -Returns a list of all materialized views registered in the system. +> ⚠️ **Not implemented today.** The statement parses, and the Materialized Views extension claims it, +> but the extension has no branch for the plural form and answers +> `400 Unsupported statement for Materialized Views extension`. Until that is implemented, inspect a +> view you can name with [`SHOW MATERIALIZED VIEW `](#show-materialized-view). --- @@ -398,7 +424,7 @@ CREATE TABLE IF NOT EXISTS orders ( quantity INT, price DOUBLE ), - createdAt TIMESTAMP DEFAULT CURRENT_TIMESTAMP, + createdAt TIMESTAMP DEFAULT _ingest.timestamp, PRIMARY KEY (id) ); @@ -579,8 +605,14 @@ Note that **transforms** (which power the continuous data sync), **enrich polici ### Syntax Summary ```sql --- Create -CREATE [OR REPLACE] MATERIALIZED VIEW [IF NOT EXISTS] name +-- Create (IF NOT EXISTS and OR REPLACE are mutually exclusive, and REFRESH EVERY precedes WITH) +CREATE MATERIALIZED VIEW [IF NOT EXISTS] name + [REFRESH EVERY n {MILLISECONDS|SECONDS|MINUTES|HOURS|DAYS|WEEKS|MONTHS|YEARS}] + [WITH (delay = 'interval', user_latency = 'interval')] + AS SELECT ... + +-- Replace +CREATE OR REPLACE MATERIALIZED VIEW name [REFRESH EVERY n {MILLISECONDS|SECONDS|MINUTES|HOURS|DAYS|WEEKS|MONTHS|YEARS}] [WITH (delay = 'interval', user_latency = 'interval')] AS SELECT ... diff --git a/sql/src/main/scala/app/softnetwork/elastic/sql/query/package.scala b/sql/src/main/scala/app/softnetwork/elastic/sql/query/package.scala index 24f672636..cb1789cbd 100644 --- a/sql/src/main/scala/app/softnetwork/elastic/sql/query/package.scala +++ b/sql/src/main/scala/app/softnetwork/elastic/sql/query/package.scala @@ -1208,8 +1208,13 @@ package object query { ) extends MaterializedViewStatement with DdlStatement { override def sql: String = { + // The leading space belongs HERE, not to `Frequency.sql`: `TransformConfig` renders the same + // value on a line of its own and supplies its own indentation. Without it the render read + // `... MATERIALIZED VIEW mvREFRESH EVERY 8 SECONDS ...`, which re-parses as a view literally + // named `mvREFRESH` and then fails - so `SHOW CREATE MATERIALIZED VIEW` emitted a statement + // no parser accepts whenever the view carried a `REFRESH EVERY` clause (story HELP-1b). val frequencySql = frequency match { - case Some(freq) => freq.sql + case Some(freq) => s" ${freq.sql}" case None => "" } val optionsSql = if (options.nonEmpty) { diff --git a/sql/src/test/scala/app/softnetwork/elastic/sql/parser/ParserSpec.scala b/sql/src/test/scala/app/softnetwork/elastic/sql/parser/ParserSpec.scala index 7f5f3f06c..4b90f1b5b 100644 --- a/sql/src/test/scala/app/softnetwork/elastic/sql/parser/ParserSpec.scala +++ b/sql/src/test/scala/app/softnetwork/elastic/sql/parser/ParserSpec.scala @@ -2223,6 +2223,41 @@ class ParserSpec extends AnyFlatSpec with Matchers { } } + // Story HELP-1b. `CreateMaterializedView.sql` rendered the frequency with NO separating space + // (`... VIEW orders_mvREFRESH EVERY 8 SECONDS ...`), so the statement `SHOW CREATE MATERIALIZED + // VIEW` hands back re-parsed as a view literally named `orders_mvREFRESH` and then failed. The + // pre-existing MV round-trip row in `QuotedTableRoundTripSpec` carries no `REFRESH EVERY` clause, + // which is exactly why nothing saw it. The `WITH`-only row is the control that shows the + // neighbouring clause was always spaced correctly. + // + // Measured while falsifying this block: restoring the defect reddens exactly the three rows whose + // view name is BARE, and the QUOTED row stays green - a closing quote terminates the identifier, + // so `"sch"."mv"REFRESH` still tokenises. A regression suite for a missing separator must + // therefore carry a BARE-name row; a quoted-only matrix would have certified the bug. Both the + // no-frequency row and the quoted row are controls, not falsifiers. + private val materializedViewFixedPoints = Seq( + "CREATE MATERIALIZED VIEW mv AS SELECT id FROM orders", + "CREATE MATERIALIZED VIEW mv REFRESH EVERY 30 SECONDS AS SELECT id FROM orders", + "CREATE MATERIALIZED VIEW IF NOT EXISTS mv REFRESH EVERY 1 MINUTE AS SELECT id FROM orders", + "CREATE MATERIALIZED VIEW mv WITH (delay = '1s') AS SELECT id FROM orders", + "CREATE OR REPLACE MATERIALIZED VIEW mv REFRESH EVERY 60 SECONDS WITH (delay = '1s') AS SELECT id FROM orders", + """CREATE MATERIALIZED VIEW "sch"."mv" REFRESH EVERY 2 HOURS AS SELECT id FROM orders""" + ) + + materializedViewFixedPoints.foreach { sql => + it should s"render a materialized view the parser accepts back: [$sql]" in { + val parsed = Parser(sql) + withClue(s"[$sql] ") { parsed.isRight shouldBe true } + val stmt = parsed.toOption.get + // Equality against the ORIGINAL statement, never `isRight`. On the corrupt render the two + // happen to coincide (`mvREFRESH EVERY 30 SECONDS AS ...` fails at the `AS` keyword, because + // `REFRESH EVERY` has been eaten into the name) - but they need not: a render that loses a + // qualifier or a clause re-parses perfectly well into a DIFFERENT statement, which is the + // failure mode `QuotedTableRoundTripSpec` was written for. + Parser(stmt.sql) shouldBe Right(stmt) + } + } + behavior of "Parser DDL with Pipeline Statements" it should "parse CREATE OR REPLACE PIPELINE" in {