Skip to content

perf(core): bound sqlite-vec reconciliation work - #1582

Closed
nogueirathiago wants to merge 1 commit into
basicmachines-co:mainfrom
nogueirathiago:fix/sqlite-vec-reconciliation-performance
Closed

nogueirathiago wants to merge 1 commit into
basicmachines-co:mainfrom
nogueirathiago:fix/sqlite-vec-reconciliation-performance

Conversation

@nogueirathiago

Copy link
Copy Markdown

Summary

  • replace the correlated sqlite-vec stale-row DELETE with an indexed manifest JOIN
  • delete resolved orphan and stale rowids in bounded 500-row batches within one transaction
  • add a regression test that interrupts quadratic SQLite work while covering multi-batch cleanup

Why

The previous correlated DELETE can repeatedly scan the sqlite-vec virtual table for every manifest row. On large projects this leaves reindex CPU-bound in reconciliation even after embedding work has completed.

The new flow resolves rowids from the indexed relational manifest first, then performs direct virtual-table rowid deletes. Project-scoped stale cleanup and global ownerless-orphan cleanup retain their existing behavior.

Validation

  • focused sqlite-vec repository tests: 28 passed
  • full SQLite unit suite: 7,681 passed, 48 skipped
  • SQLite integration run: 547 passed, 15 skipped, 35 deselected; 59 Redis-backed setup errors because the local Docker daemon was unavailable
  • Ruff lint and format checks passed
  • full ty check across src, tests, and test-int passed
  • an equivalent local patch completed reconciliation for 210,946 ready chunks in about 46 seconds with zero errors; the previous query remained CPU-bound beyond 23 minutes

Environment limitation

The Postgres and real-Redis Docker suites could not run because the local Docker daemon was not available. The change is isolated to the SQLite sqlite-vec adapter.

Signed-off-by: Thiago Henrique Nogueira <thnogueira+SEDUCSP@apoioprodesp.sp.gov.br>
@CLAassistant

Copy link
Copy Markdown

CLA assistant check
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.


Thiago Henrique Nogueira seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account.
You have signed the CLA already but the status is still pending? Let us recheck it.

@nogueirathiago

Copy link
Copy Markdown
Author

Closing because this change is intended to remain in the personal fork, not be proposed upstream.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants