fix: highlight the case and prql keywords and the brackets in the lezer grammar

[prql-bot]

Problem

Five of the style tags in the lezer grammar's highlight.js match no term in the grammar, so nine tokens render unstyled in any editor built on this parser: the case keyword, (, ), [, ], {, }, , and |. The prql query header has no style tag at all, so it is unstyled too.

Lezer only creates a named term for a literal it is told to name. The bracket and separator literals appear inline in rules and are never declared, so they are anonymous — a selector naming one matches nothing. case and prql are written as bare @specialize<identPart, "…"> rather than through the grammar's kw<> template, which is what adds [@name={term}]; that is why module, let, this, that, null and in do highlight and these two do not. Nothing reports the mismatch: styleTags drops an unmatched selector silently, so the parser builds and loads clean.

Against a parser built from main, the tags that do land are let and true; every token listed above comes back with no tag:

$ node -e '…highlightTree(parser.parse("let y = (from x\nderive {a = case [1 == 2 => 3, true => 4]})"), …)'
"let" => definitionKeyword
"true" => bool

Solution

Declare the bracket and separator literals in the @tokens block, alongside the "="[@name=Equals] that is already there — an undecorated literal declaration names each term after itself. Switch QueryDefinition and CaseExpression to kw<> so their keywords become named terms, and add the missing prql entry to highlight.js on t.keyword, matching the keyword face the emacs, nano, GtkSourceView and KSyntaxHighlighting grammars give it after #6210.

All nine tokens plus prql then carry their intended tag.

Tree changes for downstream consumers

Naming the two keywords adds them to the parse tree, the way let already appears: a query header now parses as QueryDefinition(prql,NamedArg(…)) and a case expression as CaseExpression(case,TupleExpression(…)). Anything walking the tree by child position sees one extra leading child on those two nodes.

The brackets and separators are the wider change, not the smaller one: naming a literal puts it in the tree, so every rule built from one gains it as a child. TupleExpression, ArrayExpression, NestedPipeline, ParenthesizedExpression and Module gain both delimiters, and Annotation gains its closing } (its @{ opener is a separate literal and still undeclared) — where select {a, b} gave TupleExpression(Identifier,Identifier), its tuple now holds the two braces and the comma as well. The grammar suite doesn't see any of that, because defaultIgnore in @lezer/generator's test module skips any node whose name is non-word, which is why no .txt case needed updating; consumers do see it.

The one consumer in this repo changes behaviour, for the better. web/prql-codemirror-demo/src/lang-prql/prql.ts registers foldNodeProp.add({"ArrayExpression TupleExpression": foldInside}), and foldInside folds from the end of the first child to the start of the last. Against 37f81bb those were the first and last tuple items, so folding

from x
select {
  a,
  b,
}

collapsed only the ,\n between a and b; with the delimiters named it collapses the whole body inside the braces, which is what foldInside is for. Nothing else in the repo reads this tree — the demo's complete.ts matches node names only, and web/playground highlights through its own Monaco Monarch tokenizer rather than this grammar.

The existing 104 grammar tests stayed green through the keyword naming only because none of them contained a prql header or a case expression, so two cases in test/misc.txt now pin the new shape; both fail against 37f81bb.

Testing

test/test-highlight.js is new. Its snippet cases map each tag to its own class and assert the exact [token, tag] pairs a snippet produces, pinning the tag each token gets rather than only that it gets one; both fail on main.

Snippets only cover the tokens they contain, though, and the underlying bug is that an unmatched selector is dropped in silence — a tenth selector added later for a token no snippet uses would fail the same way and still pass. So the selector map is now exported from highlight.js as prqlHighlightSpec and passed to styleTags separately, and a third case asserts every selector in it resolves to a term in parser.nodeSet.types. Adding a selector for a non-existent term makes that case fail with the term's name.

This does not add keywords the grammar cannot parse. into, type, internal, func, import and enum are still absent from the lezer grammar, which needs rules rather than a keyword list; #6210 excluded it for that reason, and #2052 tracks the missing grammar coverage.

Found during the nightly rolling survey of grammars/prql-lezer/src/highlight.js.

[prql-bot]

Self-review, so a COMMENT rather than an approval.

The change does what it claims: I rebuilt the parser and ran the suite (106 passing), confirmed both new cases fail against 37f81bb, and checked that every selector in highlight.js now resolves to a term in the built parser — no dead tag is left behind. Two points.

The tree gained prql and case nodes, and nothing pins them. The description says the existing tests are unaffected because fileTests skips punctuation. That is right for the brackets — defaultIgnore in @lezer/generator's test module is /\W/.test(type.name) — but it does not cover the two keywords. kw<> gives them word names, so they now appear in the tree the way let already does in misc.txt: QueryDefinition(prql,NamedArg(...)) and CaseExpression(case,TupleExpression(...)). The 104 existing cases stay green only because no .txt case exercises either rule — there is no prql header and no case expression anywhere in test/*.txt. So this PR changes the shape of the tree downstream consumers walk and leaves the new shape untested; a couple of cases in misc.txt would pin it in the suite's own idiom. The description is also worth correcting, since it currently reads as though the tree is unchanged.

The test pins nine tokens rather than the property that failed. The bug here is that styleTags drops an unmatched selector silently — a tenth selector added later, for a token none of these snippets happen to contain, fails exactly the same way and still passes this test. Splitting the spec out of the styleTags call makes the general check about ten lines:

export const highlightSpec = { ... };
export const prqlHighlight = styleTags(highlightSpec);

and the test then asserts every selector in highlightSpec resolves to a name in parser.nodeSet.types. I ran that assertion against this branch and it reports nothing missing, so it passes as-is and would catch the next dead tag without needing a snippet per token. Whether that is worth the extra export is a judgement call; the snippet tests are still useful either way, since they pin the tag each token gets rather than only that it gets one.

Unrelated to the code: #6308 (tend/update-workflows) carries 553cd4f in its history, so this change is currently sitting in two open PRs.

[prql-bot]

Both points actioned in b08b03f, plus a correction to the third.

The tree shape is now pinned. The review is right that fileTests' defaultIgnore covers the brackets but not the two keywords, and that no existing .txt case contained a prql header or a case expression — that is the only reason the 104 cases stayed green through a shape change. test/misc.txt gains two cases in the suite's own idiom, QueryDefinition(prql,NamedArg(ArgumentName,Identifier)) and CaseExpression(case,TupleExpression(...)). I generated the pre-change trees from a parser built off 37f81bb to confirm they pin the change rather than restating it: there the same sources parse as QueryDefinition(NamedArg(...)) and CaseExpression(TupleExpression(...)), so both cases fail before this PR. The PR description now has a section on the tree change instead of the sentence implying it was inert.

The general check is in. The selector map is exported as prqlHighlightSpec and passed to styleTags separately, and a third case in test-highlight.js asserts every selector resolves to a name in parser.nodeSet.types. It costs one export and is worth it here, because the failure mode is silence rather than a wrong value: the snippet cases can only catch a dead tag on a token some snippet happens to contain. I checked it fails as intended by adding a selector for a non-existent term — the assertion reports ['NoSuchTerm'] — and reverted. 109 tests pass on the branch.

#6308 does not carry this commit. Its head 3323708 sits directly on 37f81bb, and git merge-base --is-ancestor 553cd4f 3323708 is false; its diff is eight workflow files and nothing else. The review's note was wrong — no duplication to resolve.