Skip to content

sqlite: replace the ANTLR parser with meyer - #4535

Open
kyleconroy wants to merge 6 commits into
mainfrom
claude/sqlite-parser-meyer-migration-c3g043
Open

sqlite: replace the ANTLR parser with meyer#4535
kyleconroy wants to merge 6 commits into
mainfrom
claude/sqlite-parser-meyer-migration-c3g043

Conversation

@kyleconroy

@kyleconroy kyleconroy commented Jul 31, 2026

Copy link
Copy Markdown
Collaborator

meyer is a hand-written recursive descent parser for SQLite's dialect, ported from src/parse.y and src/tokenize.c and checked against SQLite itself. Moving to it deletes 1.2 MB of generated ANTLR code and the antlr4-go direct dependency, and replaces a grammar that only approximated SQLite with one that accepts what SQLite accepts.

This is step 8 of meyer's PLAN.md. Rebased on #4537, which does the sqlc.arg() half of the problem properly — see below.

What changed

  • parse.go calls meyer and slices statement extents out of the node spans. A parse failure becomes a sqlerr.Error positioned from meyer's byte offset, so syntax errors now report a line and column instead of a bare message.
  • convert.go rewritten to map meyer's tree onto sql/ast.
  • reserved.go defers to meyer's keyword table, which the hand-maintained list was missing MATERIALIZED and RETURNING from.
  • internal/engine/sqlite/parser/ deleted. antlr4-go/antlr/v4 stays in go.mod as an indirect dependency of cel-go.

Pinned to meyer v0.1.1, which adds parser.Options. UpdateDeleteLimit compiles in the grammar SQLITE_ENABLE_UPDATE_DELETE_LIMIT does — the pinned build defines neither it nor SQLITE_UDL_CAPABLE_PARSER, so ORDER BY and LIMIT on UPDATE and DELETE are a syntax error by default. The option picks between two forms of SQLite's own rules rather than inventing a dialect, and meyer cannot tell which one is wanted from the SQL alone, so the caller says. sqlc has always accepted them, so this engine asks for them. sqlc's AST has somewhere to put the LIMIT and nowhere to put the ORDER BY, which is what the ANTLR converter did with them too.

Behavior the old grammar got wrong

  • col TEXT NULL was read as a column whose type was TEXTNULL, because ANTLR's type_name swallowed the constraint. The column generated as interface{} rather than a nullable string. One golden file changes.
  • GROUP BY was dropped whenever the query also had a WHERE clause, from an off-by-one over the ANTLR expression list. It only worked on queries with no WHERE.
  • Comparison operators other than = were all recorded as = (there was a // TODO: add actual comparison), and IS, ISNULL, NOTNULL and LIKE reached the AST as = as well.
  • ORDER BY, DISTINCT, WITH RECURSIVE, ON CONFLICT, OVER and FILTER were parsed and then discarded, so ast.Format dropped them from the SQL it rebuilds — which the expander relies on.
  • Quoted identifiers other than "..." kept their quotes, and ""-escaped quotes inside one produced the empty string. Float literals became the integer 0.

Two changes outside the engine follow from these:

  • internal/compiler/output_columns.go recognizes SQLite's implicit rowid / _rowid_ / oid. GROUP BY reaching the compiler for the first time means its column references are validated, and GROUP BY transactions.rowid in testdata/table_function would otherwise fail.
  • lang.IsComparisonOperator learned the keyword-spelled comparisons (IS, IS NOT, LIKE, ISNULL, ...) so SELECT x IS NULL still types as bool now that the operator carries its own name. These are written upper case, which is how SQLite's parser records them; PostgreSQL and MySQL use distinct node types or their own operator spellings, so the strings do not overlap.

One gap in the preprocessing pass

SQLite has no schema-qualified function call, so sqlc.arg() is not SQL meyer will parse. #4537 handles that above the engines and SQLite is one of the preprocessed dialects, so nothing is needed here — with one exception.

An invalid sqlc.* call is deliberately copied through for the engine to parse, so that the preprocessor's message can be reported against the statement the user wrote. That assumes the engine can parse it, which holds for PostgreSQL and MySQL but not for SQLite: sqlc.argh(x) is a syntax error there, and it aborted the parse before any of the recorded diagnostics were reported. It surfaced from parseCatalog rather than parseQueries, since a schema file and a query file are frequently the same file.

The compiler now reports the sqlc syntax errors recorded for a file, if there are any, in place of the parse failure they produced. testdata/sqlc_arg_invalid/sqlite goes back to the five messages the other engines give.

Worth flagging

TRUE and FALSE are now column references rather than integer constants, because that is what SQLite's grammar does with them — they are resolved at name-resolution time, not tokenized as literals. Nothing in the test suite exercises it.

Testing

gofmt, go vet ./... and go test --tags=examples ./... all clean, against native PostgreSQL and MySQL: endtoend (TestReplay, TestFormat, TestExamples, TestExamplesVet, TestJsonSchema), the expander, and the sqlite examples. The endtoend suite is the acceptance gate; the sqlite engine's unit tests were removed.

claude added 5 commits July 31, 2026 17:41
meyer is a hand-written recursive descent parser for SQLite's dialect,
ported from src/parse.y and src/tokenize.c and checked against SQLite
itself. Moving to it deletes 1.2 MB of generated ANTLR code and the
antlr4-go dependency, and replaces a grammar that only approximated
SQLite with one that accepts what SQLite accepts.

parse.go calls meyer and slices statement extents out of the node spans;
convert.go maps meyer's tree onto sqlc's AST; reserved.go now defers to
meyer's keyword table, which the hand-maintained list was missing
MATERIALIZED and RETURNING from.

SQLite has no schema-qualified function call, so sqlc.arg() and its
siblings are not SQL meyer will parse. The parser folds "sqlc.name(" to
"sqlc_name(" over the token stream -- one byte for another, so every
offset still points into the original source, and string literals and
comments are untouched -- and the converter puts the schema back.

Behavior the old grammar got wrong and this corrects:

- "col TEXT NULL" was read as a column of type "TEXTNULL", so the column
  generated as interface{} rather than a nullable string. This changes
  one golden file.
- GROUP BY was dropped whenever the query also had a WHERE clause, from
  an off-by-one over the ANTLR expression list. Now that GROUP BY reaches
  the compiler, its column references are validated, which SQLite's
  implicit rowid would have failed; the compiler now recognizes rowid,
  _rowid_ and oid for SQLite.
- Comparison operators other than "=" were all recorded as "=", and IS,
  ISNULL, NOTNULL and LIKE reached the AST as "=" too. They now carry
  their own spelling, and lang.IsComparisonOperator learned the
  keyword-spelled comparisons so they still type as bool.
- ORDER BY, DISTINCT, WITH RECURSIVE, ON CONFLICT, OVER and FILTER were
  parsed and then discarded, so ast.Format dropped them from the SQL it
  rebuilds. All are carried through now.
- Quoted identifiers other than "..." kept their quotes; float literals
  became the integer 0.

One capability is lost: the pinned SQLite build meyer targets defines
neither SQLITE_ENABLE_UPDATE_DELETE_LIMIT nor SQLITE_UDL_CAPABLE_PARSER,
so UPDATE and DELETE no longer accept ORDER BY or LIMIT. No test covered
it.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RkrYmzwTktB2CA5Bvqg6z5
The end-to-end tests are the acceptance gate for the parser.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RkrYmzwTktB2CA5Bvqg6z5
The end-to-end tests cover the catalog the schema builds. Removing the
test leaves newTestCatalog with no callers, so it goes too.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RkrYmzwTktB2CA5Bvqg6z5
The end-to-end tests exercise the analyzer against a real database.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RkrYmzwTktB2CA5Bvqg6z5
#4537 rewrites sqlc's syntax to native placeholders above the engines, and
SQLite is one of the preprocessed dialects, so sqlc.arg() and @name never
reach the parser. The fold this branch carried is dead.

One gap follows from the pass's rule that a statement whose sqlc syntax
did not validate is copied through for the engine to parse: that assumes
the engine can parse it. SQLite cannot, so an invalid sqlc.arg() surfaced
as `near "(": syntax error` and the preprocessor's own diagnostics were
never reported -- from parseCatalog, since a schema file and a query file
are often the same file. On a parse failure the compiler now reports the
sqlc syntax errors recorded for that file, if any, in place of the failure
they produced.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RkrYmzwTktB2CA5Bvqg6z5
@kyleconroy
kyleconroy force-pushed the claude/sqlite-parser-meyer-migration-c3g043 branch from 1ee7004 to e95491e Compare July 31, 2026 18:00
v0.1.1 adds parser.Options, whose UpdateDeleteLimit field compiles in the
grammar SQLITE_ENABLE_UPDATE_DELETE_LIMIT does. The option selects between
two forms of SQLite's own rules, so meyer cannot tell which one a caller
wants from the SQL alone; sqlc has always accepted ORDER BY and LIMIT on
UPDATE and DELETE, so it asks for them.

UpdateStmt and DeleteStmt grow OrderBy and Limit fields to match. sqlc's
AST has somewhere to put the LIMIT and nowhere to put the ORDER BY, which
is what the ANTLR converter did with them too.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RkrYmzwTktB2CA5Bvqg6z5
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants