Skip to content

fix(stovepipe): terminate failed record work - #726

Draft
mnoah1 wants to merge 4 commits into
mainfrom
mnoah1/stovepipe-promotion-dlq-reconciliation
Draft

mnoah1 wants to merge 4 commits into
mainfrom
mnoah1/stovepipe-promotion-dlq-reconciliation

Conversation

@mnoah1

@mnoah1 mnoah1 commented Sep 18, 2026

Copy link
Copy Markdown
Contributor

Summary

Intent:

  • Ensure record-stage work that exhausts the primary retry budget reaches an observable stopping point instead of retrying indefinitely on the DLQ.
  • Preserve whatever durable effects already succeeded without trying to reconstruct or complete the original workflow.
  • Keep fail-closed behavior when no validation fact was recorded. Persistent promotion permission failures are one example of this broader case.

Changes:

  • Add a standalone record DLQ controller that reloads the durable Request and appends an idempotent record_failed history event.
  • Retain recognized failure context as bounded event metadata; promotion is recorded as record_stage=promotion rather than as a separate event type.
  • Acknowledge malformed or unresolvable poison after logging and counting it; retry only Request reads and history persistence.
  • Do not create facts, advance bookmarks, call source control, or publish hooks from the record DLQ.

Generated by the 🪄 pr-create skill in devexp-agent-marketplace

Test Plan

  • go test ./stovepipe/entity ./stovepipe/core/requestlog ./stovepipe/controller/record ./service/stovepipe/server
  • ./tool/bazel test //stovepipe/entity:go_default_test //stovepipe/core/requestlog:go_default_test //stovepipe/controller/record:go_default_test //service/stovepipe/server:go_default_test --test_output=errors
  • make lint
  • make check-tidy
  • make check-gazelle

Issues

Summary:
Intent:
- Prevent permanent promotion failures from occupying the record DLQ indefinitely.
- Preserve successful validation state while making abandoned promotion visible in request history.

Changes:
- Add record-specific DLQ handling that does not repeat known promotion failures.
- Retain an idempotent promotion_failed history event before acknowledging abandoned promotion work.
- Keep non-promotion reconciliation under the existing DLQ retry policy.

---

<sub>Generated by the 🪄 [pr-create](https://sg.uberinternal.com/code.uber.internal/uber-code/devexp-agent-marketplace/-/blob/claude-code/plugins/dev/uber-dev/skills/pr-create/SKILL.md) skill in devexp-agent-marketplace</sub>
@mnoah1 mnoah1 changed the title fix(stovepipe): bound record promotion retries fix(stovepipe): terminate failed record work Sep 18, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant