Skip to content

fix(redis-worker): stop fair queue leaking concurrency slots #4540

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
matt-aitken wants to merge 2 commits into
base: main
Choose a base branch
from
+177 −29
Open

fix(redis-worker): stop fair queue leaking concurrency slots #4540

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
4df6cda
fix(redis-worker): stop fair queue leaking concurrency slots
matt-aitken Aug 8, 2026
35568a3
fix(redis-worker): keep metadata-derived concurrency groups on the re…
matt-aitken Aug 8, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions .changeset/fair-queue-concurrency-slot-leak.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
---
"@trigger.dev/redis-worker": patch
---

Fair queue consumers no longer leak concurrency slots. A slot is now always released when a message completes or is put back on the queue, even when its in-flight record has already gone. Leaked slots were never reclaimed, so enough of them would permanently stall every queue belonging to that tenant.
52 changes: 24 additions & 28 deletions packages/redis-worker/src/fair-queue/index.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1245,22 +1245,20 @@ export class FairQueue<TPayloadSchema extends z.ZodTypeAny = z.ZodUnknown> {
}
}

const descriptor: QueueDescriptor = storedMessage
? (this.queueDescriptorCache.get(queueId) ?? {
id: queueId,
tenantId: storedMessage.tenantId,
metadata: storedMessage.metadata ?? {},
})
: { id: queueId, tenantId: this.keys.extractTenantId(queueId), metadata: {} };

// Complete in visibility manager
await this.visibilityManager.complete(messageId, queueId);
const descriptor: QueueDescriptor = this.queueDescriptorCache.get(queueId) ?? {
id: queueId,
tenantId: storedMessage?.tenantId ?? this.keys.extractTenantId(queueId),
metadata: storedMessage?.metadata ?? {},
};
Comment on lines +1248 to +1252

@devin-ai-integration devin-ai-integration Bot Aug 8, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔍 Reserve and release build the queue descriptor from different sources

Reservation builds the descriptor as queueDescriptorCache.get(queueId) ?? { tenantId, metadata: {} } (packages/redis-worker/src/fair-queue/index.ts:1091-1095) — it never falls back to the stored message metadata — while completeMessage/releaseMessage now fall back to storedMessage.metadata. If the cache misses at claim time but hits (or resolves differently) at completion time, the SADD and SREM target different Redis sets for metadata-derived concurrency groups. Aligning both paths on a single descriptor source (or recording the reserved group keys with the reservation) would remove this class of asymmetry entirely.

Open in Devin Review

Was this helpful? React with 👍 or 👎 to provide feedback.

Comment on lines +1248 to +1252

@coderabbitai coderabbitai Bot Aug 8, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🩺 Stability & Availability | 🟠 Major | 🏗️ Heavy lift

Preserve the descriptor that reserved each message slot.

queueDescriptorCache is keyed only by queueId and enqueue() overwrites it. If message A reserves organization:org-1, then a later enqueue to the same queue sets organization:org-2, completing or releasing A after its in-flight data is deleted calls release() for org-2. The org-1 member remains in Redis and can block that concurrency group.

Store the reservation descriptor by messageId when reserve() succeeds. Use that immutable descriptor for every release path. Remove it after release or reclaim. Add a regression test that changes queue metadata between reservation and completion.

Based on learnings, ConcurrencyManager.release() derives group IDs from descriptor metadata, so a queue-level fallback is safe only when its metadata cannot change.

Also applies to: 1301-1305

Source: Learnings


// Release concurrency
if (this.concurrencyManager && storedMessage) {
if (this.concurrencyManager) {
await this.concurrencyManager.release(descriptor, messageId);
}
Comment thread
matt-aitken marked this conversation as resolved.
Comment thread
coderabbitai[bot] marked this conversation as resolved.

// Complete in visibility manager
await this.visibilityManager.complete(messageId, queueId);

// Update both old and new indexes, clean up caches if queue is empty
const { queueEmpty } = await this.#updateAllIndexesAfterDequeue(queueId, descriptor.tenantId);
if (queueEmpty) {
Expand Down Expand Up @@ -1300,13 +1298,16 @@ export class FairQueue<TPayloadSchema extends z.ZodTypeAny = z.ZodUnknown> {
}
}

const descriptor: QueueDescriptor = storedMessage
? (this.queueDescriptorCache.get(queueId) ?? {
id: queueId,
tenantId: storedMessage.tenantId,
metadata: storedMessage.metadata ?? {},
})
: { id: queueId, tenantId: this.keys.extractTenantId(queueId), metadata: {} };
const descriptor: QueueDescriptor = this.queueDescriptorCache.get(queueId) ?? {
id: queueId,
tenantId: storedMessage?.tenantId ?? this.keys.extractTenantId(queueId),
metadata: storedMessage?.metadata ?? {},
};

// Release concurrency
if (this.concurrencyManager) {
await this.concurrencyManager.release(descriptor, messageId);
}
Comment thread
matt-aitken marked this conversation as resolved.

// Release back to queue (visibility manager updates dispatch indexes atomically)
// Dispatch shard is tenant-based, not queue-based
Expand All @@ -1324,11 +1325,6 @@ export class FairQueue<TPayloadSchema extends z.ZodTypeAny = z.ZodUnknown> {
Date.now() // Put at back of queue
);

// Release concurrency
if (this.concurrencyManager && storedMessage) {
await this.concurrencyManager.release(descriptor, messageId);
}

this.logger.debug("Message released", {
messageId,
queueId,
Expand Down Expand Up @@ -1411,6 +1407,11 @@ export class FairQueue<TPayloadSchema extends z.ZodTypeAny = z.ZodUnknown> {
attempt: storedMessage.attempt + 1,
};

// Release concurrency
if (this.concurrencyManager) {
await this.concurrencyManager.release(descriptor, storedMessage.id);
}

// Release with delay, passing the updated message data so the Lua script
// atomically writes the incremented attempt count when re-queuing.
const tenantQueueIndexKey = this.keys.tenantQueueIndexKey(descriptor.tenantId);
Expand All @@ -1427,11 +1428,6 @@ export class FairQueue<TPayloadSchema extends z.ZodTypeAny = z.ZodUnknown> {
JSON.stringify(updatedMessage)
);

// Release concurrency
if (this.concurrencyManager) {
await this.concurrencyManager.release(descriptor, storedMessage.id);
}

this.telemetry.recordRetry();

this.logger.debug("Message scheduled for retry", {
Expand Down
149 changes: 148 additions & 1 deletion packages/redis-worker/src/fair-queue/tests/fairQueue.test.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ import {
WorkerQueueManager,
} from "../index.js";
import type { FairQueueKeyProducer, FairQueueOptions } from "../types.js";
import type { RedisOptions } from "@internal/redis";
import { createRedisClient, type RedisOptions } from "@internal/redis";

// Define a common payload schema for tests
const TestPayloadSchema = z.object({ value: z.string() });
Expand Down Expand Up @@ -1370,4 +1370,151 @@ describe("FairQueue", () => {
}
);
});

describe("concurrency slot release", () => {
redisTest(
"should release the concurrency slot when the in-flight record is already gone",
{ timeout: 15000 },
async ({ redisOptions }) => {
const processed: string[] = [];
keys = new DefaultFairQueueKeyProducer({ prefix: "test" });

const scheduler = new DRRScheduler({
redis: redisOptions,
keys,
quantum: 10,
maxDeficit: 100,
});

const queue = new TestFairQueueHelper(redisOptions, keys, {
scheduler,
payloadSchema: TestPayloadSchema,
shardCount: 1,
consumerCount: 1,
consumerIntervalMs: 20,
visibilityTimeoutMs: 60000,
concurrencyGroups: [
{
name: "tenant",
extractGroupId: (q) => q.tenantId,
getLimit: async () => 1,
defaultLimit: 1,
},
],
startConsumers: false,
});

const redis = createRedisClient(redisOptions);

try {
queue.onMessage(async (ctx) => {
if (ctx.message.payload.value === "msg-0") {
await redis.hdel(keys.inflightDataKey(0), ctx.message.id);
}
processed.push(ctx.message.payload.value);
await ctx.complete();
});

for (let i = 0; i < 2; i++) {
await queue.enqueue({
queueId: "tenant:t1:queue:q1",
tenantId: "t1",
payload: { value: `msg-${i}` },
});
}

queue.start();

await vi.waitFor(
() => {
expect(processed).toHaveLength(2);
},
{ timeout: 10000 }
);

const held = await redis.scard(keys.concurrencyKey("tenant", "t1"));
expect(held).toBe(0);
} finally {
await redis.quit();
await queue.close();
}
}
);

redisTest(
"should release metadata-derived concurrency groups when the in-flight record is gone",
{ timeout: 15000 },
async ({ redisOptions }) => {
const processed: string[] = [];
keys = new DefaultFairQueueKeyProducer({ prefix: "test" });

const scheduler = new DRRScheduler({
redis: redisOptions,
keys,
quantum: 10,
maxDeficit: 100,
});

const queue = new TestFairQueueHelper(redisOptions, keys, {
scheduler,
payloadSchema: TestPayloadSchema,
shardCount: 1,
consumerCount: 1,
consumerIntervalMs: 20,
visibilityTimeoutMs: 60000,
concurrencyGroups: [
{
name: "tenant",
extractGroupId: (q) => q.tenantId,
getLimit: async () => 5,
defaultLimit: 5,
},
{
name: "organization",
extractGroupId: (q) => (q.metadata.orgId as string) ?? "default",
getLimit: async () => 1,
defaultLimit: 1,
},
],
startConsumers: false,
});

const redis = createRedisClient(redisOptions);

try {
queue.onMessage(async (ctx) => {
if (ctx.message.payload.value === "msg-0") {
await redis.hdel(keys.inflightDataKey(0), ctx.message.id);
}
processed.push(ctx.message.payload.value);
await ctx.complete();
});

for (let i = 0; i < 2; i++) {
await queue.enqueue({
queueId: "tenant:t1:queue:q1",
tenantId: "t1",
metadata: { orgId: "org-1" },
payload: { value: `msg-${i}` },
});
}

queue.start();

await vi.waitFor(
() => {
expect(processed).toHaveLength(2);
},
{ timeout: 10000 }
);

expect(await redis.scard(keys.concurrencyKey("organization", "org-1"))).toBe(0);
expect(await redis.scard(keys.concurrencyKey("organization", "default"))).toBe(0);
} finally {
await redis.quit();
await queue.close();
}
}
);
});
});
Loading