Skip to content

fix(queue): ISS-013 keep DLQ reconciliation retrying - #680

Merged
sbalabanov merged 1 commit into
mainfrom
sergeyb/fix-iss-013-unbounded-dlq-retry
Sep 17, 2026
Merged

sbalabanov merged 1 commit into
mainfrom
sergeyb/fix-iss-013-unbounded-dlq-retry

Conversation

@sbalabanov

@sbalabanov sbalabanov commented Sep 4, 2026

Copy link
Copy Markdown
Contributor

Summary

Intent:

  • Keep final-DLQ reconciliation durable during dependency outages longer than any finite attempt budget.
  • Preserve finite retry budgets for primary subscriptions.

Changes:

  • Define MaxAttempts zero as unlimited in both direct Nack and visibility-expiry poll paths.
  • Configure the shared DLQ subscription for unlimited retries with second-level dead-lettering disabled.
  • Verify the orchestrator pipeline and Runway wiring inherit the shared behavior and update the operational docs.

Reproduction:

  • A signal or storage dependency remains unavailable for more than 1000 DLQ reconciliation attempts.
  • Previously the finite cap was exhausted; because the reconciliation subscription had its own DLQ disabled, MySQL acknowledged the row and advanced past it, losing the reconciliation message.
  • The row now remains retryable until reconciliation succeeds or an operator removes it.

Generated by the 🪄 pr-create skill in devexp-agent-marketplace

Test Plan

  • ./tool/bazel test //service/runway/server:all
  • make fmt
  • make lint
  • make check-tidy
  • make check-gazelle

Revert Plan

Revert commit 08152f6.

AI Verification

Validated at 08152f6 on Sep 17 22:03 UTC · 13 files analyzed · 2s

Validator Status Issues
arc-lint not_applicable 0
android-coverage not_applicable 0
ios-lint not_applicable 0
uber-one not_applicable 0
fix-disclosure not_applicable 0
merge-conflict not_applicable 0
app-validation not_applicable 0
go-thrift-lint not_applicable 0
java-coverage not_applicable 0
ios-test not_applicable 0
web-typecheck not_applicable 0
web-coverage not_applicable 0
diff-template not_applicable 0
arc-unit not_applicable 0
go-proto-lint not_applicable 0
java-lint not_applicable 0
web-repocheck not_applicable 0
go-gazelle not_applicable 0
android-lint not_applicable 0
web-lint not_applicable 0
web-unit not_applicable 0
artifacts completed 0
custom not_applicable 0
ureview completed 0
go-lint failed 0
go-coverage failed 0

0 issues detected

Skipped validators: claude · EngWiki

Prior runs

Run at eeb6402 on Sep 17 18:05 UTC · 13 files · 2s · 0 issues detected

Issues

@CLAassistant

Copy link
Copy Markdown

CLA assistant check
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
You have signed the CLA already but the status is still pending? Let us recheck it.

// A zero budget is not "dead-letter immediately" — it is unconfigured,
// and the poll loop still governs.
{name: "unset budget never dead-letters here", attempt: 9},
{name: "unlimited budget keeps retrying beyond former cap", attempt: 1001},

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

no need for this regression test

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removed the unnecessary regression test in ee84d5f. The remaining table cases cover retry-budget behavior without testing the former implementation cap.

[addressed by agent]

@sbalabanov
sbalabanov force-pushed the sergeyb/fix-iss-013-unbounded-dlq-retry branch from 68fa422 to ee84d5f Compare September 8, 2026 22:56
@sbalabanov
sbalabanov marked this pull request as ready for review September 8, 2026 22:56
@sbalabanov
sbalabanov requested review from a team and behinddwalls as code owners September 8, 2026 22:56
@sbalabanov
sbalabanov force-pushed the sergeyb/fix-iss-013-unbounded-dlq-retry branch from ee84d5f to 9ec8c42 Compare September 17, 2026 21:44
Summary:
Intent:
- Keep final-DLQ reconciliation durable during dependency outages longer than any finite attempt budget.
- Preserve finite retry budgets for primary subscriptions.

Changes:
- Define MaxAttempts zero as unlimited in both direct Nack and visibility-expiry poll paths.
- Configure the shared DLQ subscription for unlimited retries with second-level dead-lettering disabled.
- Verify the orchestrator pipeline and Runway wiring inherit the shared behavior and update the operational docs.

Reproduction:
- A signal or storage dependency remains unavailable for more than 1000 DLQ reconciliation attempts.
- Previously the finite cap was exhausted; because the reconciliation subscription had its own DLQ disabled, MySQL acknowledged the row and advanced past it, losing the reconciliation message.
- The row now remains retryable until reconciliation succeeds or an operator removes it.

---

<sub>Generated by the 🪄 [pr-create](https://sg.uberinternal.com/code.uber.internal/uber-code/devexp-agent-marketplace/-/blob/claude-code/plugins/dev/uber-dev/skills/pr-create/SKILL.md) skill in devexp-agent-marketplace</sub>
@sbalabanov
sbalabanov force-pushed the sergeyb/fix-iss-013-unbounded-dlq-retry branch from 9ec8c42 to 08152f6 Compare September 17, 2026 22:03
@sbalabanov
sbalabanov merged commit c941a7c into main Sep 17, 2026
26 of 29 checks passed
@behinddwalls
behinddwalls deleted the sergeyb/fix-iss-013-unbounded-dlq-retry branch September 17, 2026 22:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants