sqlite: replace the ANTLR parser with meyer - #4535
Open
kyleconroy wants to merge 6 commits into
Open
Conversation
meyer is a hand-written recursive descent parser for SQLite's dialect,
ported from src/parse.y and src/tokenize.c and checked against SQLite
itself. Moving to it deletes 1.2 MB of generated ANTLR code and the
antlr4-go dependency, and replaces a grammar that only approximated
SQLite with one that accepts what SQLite accepts.
parse.go calls meyer and slices statement extents out of the node spans;
convert.go maps meyer's tree onto sqlc's AST; reserved.go now defers to
meyer's keyword table, which the hand-maintained list was missing
MATERIALIZED and RETURNING from.
SQLite has no schema-qualified function call, so sqlc.arg() and its
siblings are not SQL meyer will parse. The parser folds "sqlc.name(" to
"sqlc_name(" over the token stream -- one byte for another, so every
offset still points into the original source, and string literals and
comments are untouched -- and the converter puts the schema back.
Behavior the old grammar got wrong and this corrects:
- "col TEXT NULL" was read as a column of type "TEXTNULL", so the column
generated as interface{} rather than a nullable string. This changes
one golden file.
- GROUP BY was dropped whenever the query also had a WHERE clause, from
an off-by-one over the ANTLR expression list. Now that GROUP BY reaches
the compiler, its column references are validated, which SQLite's
implicit rowid would have failed; the compiler now recognizes rowid,
_rowid_ and oid for SQLite.
- Comparison operators other than "=" were all recorded as "=", and IS,
ISNULL, NOTNULL and LIKE reached the AST as "=" too. They now carry
their own spelling, and lang.IsComparisonOperator learned the
keyword-spelled comparisons so they still type as bool.
- ORDER BY, DISTINCT, WITH RECURSIVE, ON CONFLICT, OVER and FILTER were
parsed and then discarded, so ast.Format dropped them from the SQL it
rebuilds. All are carried through now.
- Quoted identifiers other than "..." kept their quotes; float literals
became the integer 0.
One capability is lost: the pinned SQLite build meyer targets defines
neither SQLITE_ENABLE_UPDATE_DELETE_LIMIT nor SQLITE_UDL_CAPABLE_PARSER,
so UPDATE and DELETE no longer accept ORDER BY or LIMIT. No test covered
it.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RkrYmzwTktB2CA5Bvqg6z5
The end-to-end tests are the acceptance gate for the parser. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RkrYmzwTktB2CA5Bvqg6z5
The end-to-end tests cover the catalog the schema builds. Removing the test leaves newTestCatalog with no callers, so it goes too. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RkrYmzwTktB2CA5Bvqg6z5
The end-to-end tests exercise the analyzer against a real database. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RkrYmzwTktB2CA5Bvqg6z5
#4537 rewrites sqlc's syntax to native placeholders above the engines, and SQLite is one of the preprocessed dialects, so sqlc.arg() and @name never reach the parser. The fold this branch carried is dead. One gap follows from the pass's rule that a statement whose sqlc syntax did not validate is copied through for the engine to parse: that assumes the engine can parse it. SQLite cannot, so an invalid sqlc.arg() surfaced as `near "(": syntax error` and the preprocessor's own diagnostics were never reported -- from parseCatalog, since a schema file and a query file are often the same file. On a parse failure the compiler now reports the sqlc syntax errors recorded for that file, if any, in place of the failure they produced. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RkrYmzwTktB2CA5Bvqg6z5
kyleconroy
force-pushed
the
claude/sqlite-parser-meyer-migration-c3g043
branch
from
July 31, 2026 18:00
1ee7004 to
e95491e
Compare
v0.1.1 adds parser.Options, whose UpdateDeleteLimit field compiles in the grammar SQLITE_ENABLE_UPDATE_DELETE_LIMIT does. The option selects between two forms of SQLite's own rules, so meyer cannot tell which one a caller wants from the SQL alone; sqlc has always accepted ORDER BY and LIMIT on UPDATE and DELETE, so it asks for them. UpdateStmt and DeleteStmt grow OrderBy and Limit fields to match. sqlc's AST has somewhere to put the LIMIT and nowhere to put the ORDER BY, which is what the ANTLR converter did with them too. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RkrYmzwTktB2CA5Bvqg6z5
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
meyer is a hand-written recursive descent parser for SQLite's dialect, ported from
src/parse.yandsrc/tokenize.cand checked against SQLite itself. Moving to it deletes 1.2 MB of generated ANTLR code and theantlr4-godirect dependency, and replaces a grammar that only approximated SQLite with one that accepts what SQLite accepts.This is step 8 of meyer's PLAN.md. Rebased on #4537, which does the
sqlc.arg()half of the problem properly — see below.What changed
parse.gocalls meyer and slices statement extents out of the node spans. A parse failure becomes asqlerr.Errorpositioned from meyer's byte offset, so syntax errors now report a line and column instead of a bare message.convert.gorewritten to map meyer's tree ontosql/ast.reserved.godefers to meyer's keyword table, which the hand-maintained list was missingMATERIALIZEDandRETURNINGfrom.internal/engine/sqlite/parser/deleted.antlr4-go/antlr/v4stays ingo.modas an indirect dependency of cel-go.Pinned to meyer v0.1.1, which adds
parser.Options.UpdateDeleteLimitcompiles in the grammarSQLITE_ENABLE_UPDATE_DELETE_LIMITdoes — the pinned build defines neither it norSQLITE_UDL_CAPABLE_PARSER, soORDER BYandLIMITonUPDATEandDELETEare a syntax error by default. The option picks between two forms of SQLite's own rules rather than inventing a dialect, and meyer cannot tell which one is wanted from the SQL alone, so the caller says. sqlc has always accepted them, so this engine asks for them. sqlc's AST has somewhere to put theLIMITand nowhere to put theORDER BY, which is what the ANTLR converter did with them too.Behavior the old grammar got wrong
col TEXT NULLwas read as a column whose type wasTEXTNULL, because ANTLR'stype_nameswallowed the constraint. The column generated asinterface{}rather than a nullable string. One golden file changes.GROUP BYwas dropped whenever the query also had aWHEREclause, from an off-by-one over the ANTLR expression list. It only worked on queries with noWHERE.=were all recorded as=(there was a// TODO: add actual comparison), andIS,ISNULL,NOTNULLandLIKEreached the AST as=as well.ORDER BY,DISTINCT,WITH RECURSIVE,ON CONFLICT,OVERandFILTERwere parsed and then discarded, soast.Formatdropped them from the SQL it rebuilds — which the expander relies on."..."kept their quotes, and""-escaped quotes inside one produced the empty string. Float literals became the integer0.Two changes outside the engine follow from these:
internal/compiler/output_columns.gorecognizes SQLite's implicitrowid/_rowid_/oid.GROUP BYreaching the compiler for the first time means its column references are validated, andGROUP BY transactions.rowidintestdata/table_functionwould otherwise fail.lang.IsComparisonOperatorlearned the keyword-spelled comparisons (IS,IS NOT,LIKE,ISNULL, ...) soSELECT x IS NULLstill types asboolnow that the operator carries its own name. These are written upper case, which is how SQLite's parser records them; PostgreSQL and MySQL use distinct node types or their own operator spellings, so the strings do not overlap.One gap in the preprocessing pass
SQLite has no schema-qualified function call, so
sqlc.arg()is not SQL meyer will parse. #4537 handles that above the engines and SQLite is one of the preprocessed dialects, so nothing is needed here — with one exception.An invalid
sqlc.*call is deliberately copied through for the engine to parse, so that the preprocessor's message can be reported against the statement the user wrote. That assumes the engine can parse it, which holds for PostgreSQL and MySQL but not for SQLite:sqlc.argh(x)is a syntax error there, and it aborted the parse before any of the recorded diagnostics were reported. It surfaced fromparseCatalograther thanparseQueries, since a schema file and a query file are frequently the same file.The compiler now reports the sqlc syntax errors recorded for a file, if there are any, in place of the parse failure they produced.
testdata/sqlc_arg_invalid/sqlitegoes back to the five messages the other engines give.Worth flagging
TRUEandFALSEare now column references rather than integer constants, because that is what SQLite's grammar does with them — they are resolved at name-resolution time, not tokenized as literals. Nothing in the test suite exercises it.Testing
gofmt,go vet ./...andgo test --tags=examples ./...all clean, against native PostgreSQL and MySQL: endtoend (TestReplay,TestFormat,TestExamples,TestExamplesVet,TestJsonSchema), the expander, and the sqlite examples. The endtoend suite is the acceptance gate; the sqlite engine's unit tests were removed.