Skip to content

feat(api): add search, filtering, and sorting to the v2 list endpoints - #6189

Merged
TheodoreSpeaks merged 2 commits into
improvement/v2-endpointsfrom
feat/v2-list-search-filter
Aug 2, 2026
Merged

feat(api): add search, filtering, and sorting to the v2 list endpoints#6189
TheodoreSpeaks merged 2 commits into
improvement/v2-endpointsfrom
feat/v2-list-search-filter

Conversation

@TheodoreSpeaks

Copy link
Copy Markdown
Collaborator

Summary

  • One convention for every v2 list, documented on lib/api/contracts/v2/shared.ts: search (case-insensitive substring on the resource's single natural name field), sortBy + sortOrder (per-resource enum, never a free string), and enumerated resource-specific filters. Reuses the sortBy/sortOrder pair v2 logs and v2 knowledge-documents already ship — no third dialect alongside the Logs filters or the Tables predicate grammar.
  • Applied to files, folders, tables, workflows, knowledge bases, credentials, MCP servers, skills, and custom tools. Logs and audit logs untouched.
  • Everything is pushed into SQL. GET /api/v2/files used to read the whole scope and sort/slice it in JS; it now goes through a new queryWorkspaceFiles that filters, orders, and bounds the page in one query. Workflows' keyset now follows sortBy.
  • Cursors are stamped with the sort they were minted under — replaying one under a different sort (or a malformed/short cursor) is a 400 instead of silently duplicated or skipped rows.
  • Added folderId to the files, tables, and knowledge lists; credentials had no ORDER BY at all and now has a deterministic default.
  • OpenAPI specs updated in step for every list operation touched.

Notes

  • position names a resource's stored manual arrangement (the sort_order column on workflows/folders), spelled differently from the sortOrder param on purpose.
  • Skills is the one partial pushdown: built-in skills live in code, not the DB, so their filter and merge stay in memory. It's a fixed, tiny array, documented in the function and in the spec.
  • New v2 list queries extend their own schemas rather than the shared v1 ones — v1 doesn't implement these params, and advertising a param a route ignores is worse than not having it.
  • Added mapWith to @sim/testing's mock sql fragment (a real gap that made getKnowledgeBases throw under test). That surfaced one existing knowledge test that was green only because of the throw; corrected its assertion to what it actually claims.

Type of Change

  • New feature

Testing

  • bun run check:api-validation:strict, bun run check:openapi (7 specs, 89 operations), bun run type-check, bun run lint:check — all pass
  • Full audit suite (boundaries, utils, zustand-v5, react-query, client-boundary, bare-icons, icon-paths, realtime-prune, skills, agent-stream-docs) — all pass
  • 3608 tests pass across the touched areas. New suites: the shared SQL helper against real drizzle (asserting rendered SQL and bound params), the files query, the workflows route, knowledge, and a table-driven list-convention.test.ts pinning each resource's search column and ordering
  • Per endpoint: search narrows the query, is case-insensitive, escapes %/_; an unknown sort field 400s before reaching SQL; pagination terminates with a filter applied; a filter plus a cursor returns a consistent page

Checklist

  • Code follows project style guidelines
  • Self-reviewed my changes
  • Tests added/updated and passing
  • No new warnings introduced
  • I confirm that I have read and agree to the terms outlined in the Contributor License Agreement (CLA)

One convention across every v2 list, documented on lib/api/contracts/v2/shared.ts:
`search` (case-insensitive substring on the resource's natural name field),
`sortBy` + `sortOrder` (per-resource enum, never a free string), and enumerated
resource-specific filters. Reuses the sortBy/sortOrder pair v2 logs and v2
knowledge-documents already ship rather than inventing a third dialect
alongside the Logs filters and the Tables predicate grammar.

Every filter and sort is pushed into SQL. GET /api/v2/files previously read the
whole scope and sorted/sliced it in JS; it now goes through a new
queryWorkspaceFiles that filters, orders, and bounds the page in one query.

Cursors are stamped with the sort they were minted under, so replaying one
under a different sort is a 400 instead of silently duplicated or skipped rows.
@vercel

vercel Bot commented Aug 2, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment
Project Deployment Actions Updated (UTC)
docs Skipped Skipped Aug 2, 2026 4:14am

Request Review

@cursor

cursor Bot commented Aug 2, 2026

Copy link
Copy Markdown

PR Summary

Medium Risk
Large API surface change with new query paths and keyset pagination; risk is mainly inconsistent list behavior or cursor bugs on files/workflows, not auth or data writes.

Overview
Adds a shared v2 list convention (search, sortBy/sortOrder, enumerated filters) documented in shared.ts, with Zod contracts and OpenAPI updates across files, folders, tables, workflows, knowledge bases, credentials, MCP servers, skills, and custom tools.

Query layer: New lib/api/list-query builds parameterized ILIKE search, keyset ORDER BY, and cursor resume. List routes and domain queries forward validated params into SQL instead of filtering or sorting in memory. GET /api/v2/files switches from listWorkspaceFiles + in-route slicing to queryWorkspaceFiles for filtered, sorted, bounded pages. GET /api/v2/workflows replaces a fixed (sortOrder, createdAt, id) keyset with sort-aware keysets via the same helpers.

Cursors: encodeSortedCursor / decodeSortedCursor stamp the active sort on paginated lists; mismatched or malformed cursors return 400 instead of duplicating or skipping rows.

Other: folderId on files, tables, and knowledge lists; deterministic ORDER BY for credentials. Skills merge built-in templates in memory when sorting/searching (documented exception). Tests cover SQL binding, per-resource search columns, files query, workflows route, and list-convention table.

Reviewed by Cursor Bugbot for commit 0888ee0. Bugbot is set up for automated code reviews on this repo. Configure here.

@greptile-apps

greptile-apps Bot commented Aug 2, 2026

Copy link
Copy Markdown
Contributor

Greptile Summary

The PR adds a consistent search, filtering, and sorting convention to v2 list endpoints and moves file filtering, ordering, and keyset pagination into SQL.

  • Adds resource-specific validated query contracts and corresponding OpenAPI parameters.
  • Introduces reusable SQL search, ordering, and typed cursor-key helpers.
  • Updates files and workflows to use sort-stamped keyset cursors with malformed-key handling.
  • Extends tests across routes, query helpers, and database-backed list behavior.

Confidence Score: 5/5

The PR appears safe to merge.

No blocking failure remains; the previously reported malformed-cursor path now rejects invalid key values before SQL and maps the rejection to a 400 response.

Important Files Changed

Filename Overview
apps/sim/lib/api/list-query.ts Adds reusable escaped search predicates, deterministic ordering, and typed keyset cursor codecs; malformed key values are rejected before SQL.
apps/sim/app/api/v2/lib/response.ts Adds sort-stamped cursor envelope encoding and decoding while delegating value validation to the keyset codecs.
apps/sim/app/api/v2/files/route.ts Replaces in-memory file pagination with validated, SQL-backed filtering, sorting, and keyset pagination.
apps/sim/lib/uploads/contexts/workspace/workspace-file-manager.ts Implements bounded workspace-file queries and classifies invalid cursor keys as validation errors.
apps/sim/app/api/v2/workflows/route.ts Makes workflow keyset pagination follow the selected sort and returns a client error for incompatible cursor keys.
apps/sim/lib/api/contracts/v2/shared.ts Defines the shared v2 list search and sort conventions used by resource-specific schemas.

Sequence Diagram

sequenceDiagram
  participant Client
  participant Route as V2 List Route
  participant Codec as Cursor Codec
  participant Query as SQL Query Layer
  participant DB as PostgreSQL
  Client->>Route: GET with search/filter/sort/cursor
  Route->>Route: Validate query parameters
  Route->>Codec: Decode sort-stamped cursor
  Codec-->>Route: Typed keys or invalid
  alt Invalid envelope or keys
    Route-->>Client: 400 BAD_REQUEST
  else Valid request
    Route->>Query: Filters, ordering, limit, cursor keys
    Query->>DB: Bounded keyset query
    DB-->>Query: Page and next-row marker
    Query-->>Route: Items and next keys
    Route->>Codec: Encode next cursor
    Route-->>Client: Items and nextCursor
  end
Loading

Reviews (2): Last reviewed commit: "fix(api): validate v2 cursor key values ..." | Re-trigger Greptile

Comment thread apps/sim/app/api/v2/lib/response.ts
Comment thread apps/sim/lib/uploads/contexts/workspace/workspace-file-manager.ts Outdated
…precision

Two review findings, fixed at the root by making a keyset key own its cursor
codec instead of hand-writing a decoder per sort.

Cursor key values are caller-controlled, and matching the sort stamp and key
count was not enough: an unparseable timestamp or a non-numeric size reached
the query as an Invalid Date or NaN and surfaced as a 500. Each key now type-
checks its own value and rejects a cursor it cannot hold, which both routes
render as the documented 400.

Timestamp keys now order and compare on date_trunc('milliseconds', col).
Postgres keeps microseconds and defaultNow() populates them, but a cursor value
round-trips through a millisecond-only JS Date — comparing the raw column
against the truncated value re-admitted the page's own last row, duplicating it
and stalling pagination outright at a page size of one. Reachable today via
workspace_files.updated_at, which insertFileMetadata leaves to defaultNow().
@TheodoreSpeaks

Copy link
Copy Markdown
Collaborator Author

@greptile

@TheodoreSpeaks

Copy link
Copy Markdown
Collaborator Author

@cursor review

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

Reviewed by Cursor Bugbot for commit 0888ee0. Configure here.

@TheodoreSpeaks
TheodoreSpeaks merged commit 2d8350e into improvement/v2-endpoints Aug 2, 2026
5 checks passed
@waleedlatif1
waleedlatif1 deleted the feat/v2-list-search-filter branch August 2, 2026 05:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant