Skip to content

fix: [AI-9011] bump sqlglot[c] pin to 30.11.0 for the resolver recursion fix - #117

Merged
Sourabhchrs93 merged 1 commit into
mainfrom
fix/sqlglot-resolver-recursion-bump
Sep 16, 2026
Merged

Sourabhchrs93 merged 1 commit into
mainfrom
fix/sqlglot-resolver-recursion-bump

Conversation

@Sourabhchrs93

Copy link
Copy Markdown
Contributor

Bumps the sqlglot[c] pin from 30.7.0 to 30.11.0 to pick up an upstream fix for unbounded recursion in the optimizer's column Resolver.

Why

sqlglot 30.7.0 — the version we pin today — carries a regression (upstream #7732) where qualifying a LATERAL FLATTEN of an unqualified column whose base table is absent from the supplied schema recurses forever:

get_source_columns → _get_unnest_column_type → get_table
  → _get_table_name_from_sources → _get_all_source_columns → get_source_columns → ∞

Each pass constructs a fresh Resolver, so the per-instance caches never break the cycle and there is no visited guard.

On a pure-Python build this raises a catchable RecursionError. On the compiled build we pin here, the recursion runs in mypyc-native frames that don't honour Python's recursion limit — so it overflows the C stack into an uncatchable SIGSEGV. No try/except can recover it; the process simply dies.

Impact

In altimate-backend, the dbt column-lineage subprocess runs this exact path. Six sunrun models hit the trigger and segfaulted the worker nightly, which then wedged the Celery task and cost the dbt worker one of its five pool slots every night since 2026-07-31 (AI-9011).

The fix

Upstream PR #7737 fixed it; first release containing it is 30.11.0.

Keeping the [c] extra. Dropping it would also stop the crash (a RecursionError is catchable), but the compiled build is ~2.7x faster on optimize (1.8 ms vs 4.9 ms/call measured on real dbt SQL) and the speedup it was added for in #101 still applies. This bump gets the fix without giving that up.

Verification

  • All six previously-crashing models: rc=139 (SIGSEGV) on 30.7.0 → no crash on 30.11.0. Four now optimize cleanly; two raise a normal catchable OptimizeError from validate_qualify_columns, which is the documented behaviour for partial schemas.
  • Full suite on 30.11.0: 157 passed.
  • SqlCheck inspects optimizer rules via inspect.getfullargspec(rule).args, which chore: bump sqlglot pin to sqlglot[c]==30.7.0 #101 called out as the compat risk — still fine on 30.11.0, covered by the passing suite.

Chose 30.11.0 rather than latest (30.18.0) as the minimal delta containing the fix; 30.18.0 was also verified clean if we'd rather jump further.

Note for whoever picks this up: altimate-dags pins sqlglot too and #101 deliberately aligned all consumers on one version — worth bumping it in step. altimate-backend can only move off 30.7.0 once this is released, since the pin here is exact.

🤖 Generated with Claude Code

…rsion fix

`sqlglot` 30.7.0 carries a regression (upstream issue #7732) where the column
`Resolver` recurses without bound when qualifying a `LATERAL FLATTEN` of an
unqualified column whose base table is absent from the supplied schema. The
cycle is `get_source_columns` -> `_get_unnest_column_type` -> `get_table` ->
`_get_table_name_from_sources` -> `_get_all_source_columns` and back; each pass
builds a fresh `Resolver`, so the per-instance caches never break it.

On the pure-Python build that surfaces as a catchable `RecursionError`. On the
compiled build we pin here, the recursion runs in mypyc-native frames that do
not honour Python's recursion limit, so it overflows the C stack into an
uncatchable `SIGSEGV` that kills the process outright.

Downstream this killed the lineage subprocess in `altimate-backend` on six
sunrun models nightly, which in turn wedged the dbt worker and cost it a pool
slot per night.

Fixed upstream by sqlglot PR #7737, first released in 30.11.0. Keeping the `[c]`
extra: the compiled build is ~2.7x faster on `optimize` and the speedup it was
added for still applies.

Verified: all six previously-crashing models stop segfaulting on 30.11.0, and
the full suite passes (157 passed).

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@kilo-code-bot

kilo-code-bot Bot commented Sep 16, 2026

Copy link
Copy Markdown

Code Review Summary

Status: No Issues Found | Recommendation: Merge

Files Reviewed (1 files)
  • setup.py

Reviewed by gpt-sol-latest · Input: 0 · Output: 0 · Cached: 0

@Sourabhchrs93
Sourabhchrs93 merged commit 919ea17 into main Sep 16, 2026
33 checks passed
@Sourabhchrs93
Sourabhchrs93 deleted the fix/sqlglot-resolver-recursion-bump branch September 16, 2026 09:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants