Redis write queue halves throughput and stalls the import drain; disabled in production

[ralyodio]

REDIS_URL was set on both Railway services on 2026-08-19 to switch on the write queue from #132, which had been merged but never actually wired (the variable had never existed, so connect() had always taken the serializeWrites fallback).

It made throughput worse and stalled imports outright. REDIS_URL has been removed from both services, so production is back on the in-process path. The code is still merged and correct as written — this issue exists so nobody re-enables it without knowing what happens.

Measured, same database, same hour

	with Redis	after removing it
items ingested / 10 min	4,132	8,132
feeds crawled / 10 min	582	1,003
import drain	stalled — 65,474 entries, unmoved for >1h	draining again
poller crawl tick	ms=823314, ms=500966	back to normal

The poller logged crawl log write failed: The operation was aborted due to timeout continuously while it was on.

Why — it is not Redis being slow

Redis latency is irrelevant here. The cause is that the queue serialises writes that used to run in parallel, and the resulting ceiling is below what the workload needs.

• createWriteWorker runs at concurrency: 1, deliberately and correctly — SQLite permits one writer.
• Every job is one client.batch(statements, 'write'), i.e. one remote Turso transaction, which client.js documents at ~370ms.
• So global write throughput is capped at roughly 1 / 370ms ≈ 2.7 transactions per second, cluster-wide, for every process combined.

Before the queue, that work was not serialised that way: serializeWrites is per-process (web and poller each had their own), and the crawl path runs under TURSO_CRAWL_AUTOCOMMIT=1, which issues single-statement execute() writes. Neither wrapper overrides execute, so those went straight to the database in parallel. Turning on Redis funnelled every batch() write in the cluster through a single 2.7/s consumer.

Two consequences worth calling out:

1. Head-of-line blocking. BullMQ is FIFO with one consumer, so the import drain's large periodic batch queues behind every small crawl batch. That is why it did not merely slow down, it stopped.
2. No folding to compensate. serializeWrites can fold several waiting callers into one transaction, but TURSO_WRITE_GROUP_STATEMENTS=1 in production disables that (default is 1; the comment notes a 5-crawl group exceeded the 30s request deadline when canaried). So neither path folds — the Redis path just adds a global serialisation point the in-process path never had across processes.

Also seen

MaxListenersExceededWarning: 11 closing listeners added to [Queue] on the poller — connect() builds a fresh Queue + QueueEvents pair on every call, so connections accumulate. Worth memoising the client regardless of what happens to this issue.

What would need to change before re-enabling

• Batch multiple jobs per transaction in the worker (fold at the consumer, since folding at the producer is what serializeWrites does and it is switched off).
• Or separate queues per class of write so a bulk drain cannot block interactive writes.
• Or raise the ceiling some other way — the 2.7/s figure is the thing to beat, and it should be measured before flipping the variable, not after.

Repro is cheap: set REDIS_URL on the poller, watch import_entries stop moving.