Add to the queue, and let a person's submission cut in line

[ralyodio]

/submit was not queueing. Below 100 entries every URL was resolved inline — an outbound fetch each, up to eleven sequential candidates at a 15s timeout, then the feed insert, the items and the topics — and the route only answered early when something had been queued, which below 100 nothing ever was.

Measured against production before this change:

pasted URLs	before	after
1	0.4–2.7s warm, 19.6s cold	0.18s, still redirects to the blog
8 (all already indexed)	65.5s, queued: 0, nothing inserted	0.108s
~100	~13 min, cut off at maxDuration = 300	instant
101–2,000	could not finish	instant
>2,000	instant (uploader's staged path)	unchanged

Response time was non-monotonic: 2,001 URLs were instant while 101 timed out.

What was actually slow

Neither the database nor the crawler. Prod reads and autocommit writes measured ~90–100ms throughout, and resolveFeed is 47ms–2.5s per URL. Three separate things:

1. A feed already in the directory still paid a full network resolve. submitOne called resolveFeed before q.feedByUrl. That was the whole 65 seconds — every one of those fetches was of a document we already held.
2. Nothing under 100 entries was queued at all. INLINE_LIMIT was 100, so the tail was empty, so queued was 0, so the early 303 never fired.
3. importFeeds reads the entire directory first. select feed_url, slug … limit 5000 offset ? is 84 pages at 416k feeds, and offset paging is O(offset) — late pages measured 16.5s each. I let it run 550s without finishing, inside a route capped at 300s.

The other half: the queue never reached you

Queueing instantly is no use if the queue never gets there. dueFeeds orders by next_fetch_at asc, a new feed is stamped now, and ~307,000 bulk-uploaded feeds were already overdue — so a blog submitted today sorted behind every one of them.

Submissions of ≤100 entries (EXPRESS_MAX — the line between a person and an export) now go in an express lane read before the backlog. It is bounded at both ends:

• It cannot starve the backlog. At most half a tick, so a flood of submissions can never stop the directory draining.
• It cannot be camped in. The lane is priority > 0 and last_fetched_at is null, and the crawler writes last_fetched_at on success and on failure — so a feed leaves after exactly one attempt, with nothing to clear afterwards.

The partial index means the express query reads an index that is normally empty and never larger than one afternoon's submissions.

Also

• One URL keeps its inline resolve, but bounded (SUBMIT_INLINE_WAIT_MS, 8s): past that the submitter gets the status page and the same promise finishes in the background, so there is no queued duplicate racing the insert.
• The submit_feed MCP tool gets the same treatment and the same EXPRESS_MAX bound — it accepts up to 200 URLs, so that check does real work.

Verification

1,105 tests pass (12 new). Verified end to end against a local instance on real SQLite: 8 URLs 65s → 0.108s, 150 URLs 0.030s, a resubmitted feed 0.033s with no network at all, one URL still lands on /high-signal in 0.182s, and the next tick ordering is 5 express then 5 backlog exactly as designed.

🤖 Generated with Claude Code