Fix job list performance and SQLite pagination - #1374
Open
bgentry wants to merge 2 commits into
Open
Conversation
bgentry
force-pushed
the
bg/job-list-finalized-index
branch
7 times, most recently
from
September 10, 2026 01:14
e8e6290 to
05cde7f
Compare
Listing one finalized state by time can scan and sort the entire job table. PostgreSQL cannot use the partial finalized-time index without an explicit non-null predicate, and singleton ANY prevents it from using the index's time ordering. Build state equality with the list API's existing condition mechanism when custom SQL cannot depend on the existing array argument. Add `finalized_at IS NOT NULL` only for one known finalized state ordered by finalized time. Keep the shared builder and drivers unchanged. Preserve custom SQL, multi-state filters, and deletion queries. Build cursor predicates locally so conversion preserves the caller's conditions. Cover finalized states, both ordering directions, tied timestamps, PostgreSQL pagination, and combined filters across supported drivers. Check custom OR expressions, contradictory conditions, argument binding, and unchanged delete-many query generation.
bgentry
force-pushed
the
bg/job-list-finalized-index
branch
from
September 10, 2026 01:19
05cde7f to
e50368a
Compare
SQLite stores job timestamps as formatted text, while JobList passes cursor values directly to database/sql. Different encodings can skip or repeat jobs when a page boundary shares a timestamp. Format `time.Time` and `*time.Time` list arguments with SQLite's existing timestamp helpers. Copy the argument map so conversion leaves reusable parameters intact, including custom conditions and nullable values. Enable finalized-job pagination coverage for SQLite, libSQL, and Turso. Cover scheduled-job pagination, millisecond precision, time zones, nullable arguments, and repeated use of driver parameters.
bgentry
changed the title
bgentry
marked this pull request as ready for review
September 10, 2026 01:28
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Listing completed jobs can scan the entire job table even though an index exists for this query. Earlier work explicitly aimed to use that index (#304). The
finalized_at IS NULL/IS NOT NULLconditions may have been lost along the way, but we haven't found a removal commit.When filtering by one state, use
state = ...instead ofstate = ANY(...). For completed, cancelled, or discarded jobs sorted by finalized time, also addfinalized_at IS NOT NULL. Both changes are needed to make the measured query fast. River UI benefits without any changes to its calls. This optimization skips queries with custom SQL, multiple states, and job deletion.A separate commit fixes SQLite pagination skipping or repeating jobs. Format cursor timestamps the same way as stored timestamps so comparisons work correctly.
Listing 100 completed jobs on a local PostgreSQL 17.11 database with 888,000 jobs:
The fixed query uses the existing index and reads 103 rows instead of scanning 888,000. These are individual measurements, not averages.