Skip to content

fix: record why an async publish failed, and restore the retry #1529 disabled - #2171

Open
netomi wants to merge 2 commits into
mainfrom
fix/publish-failure-visibility
Open

fix: record why an async publish failed, and restore the retry #1529 disabled#2171
netomi wants to merge 2 commits into
mainfrom
fix/publish-failure-visibility

Conversation

@netomi

@netomi netomi commented Sep 5, 2026

Copy link
Copy Markdown
Contributor

Follow-up to #2170, covering the second half of #1450: a version whose publish fails just stays inactive.

Async publishing is a deliberate design decision — storing, signing and checksumming a package are slow and have no business holding the upload request open. The gap is that when that work fails, nothing writes down why.

Nothing recorded the reason

A failed attempt left the row at active = false and that was the whole of it. The only account was Spring's SimpleAsyncUncaughtExceptionHandler line, which names PublishExtensionVersionHandler.publishAsync and not the extension it was working on — so an operator finding an inactive version had to correlate it against a stack trace by hand, which is exactly what the reporter did.

V1_75 adds extension_version.publish_error, set when an attempt fails and cleared when a version is activated, so it always describes the latest attempt rather than accumulating history.

Two deliberate choices in it:

  • Type and message only, no cause chain. A chain carries file system paths, host names and connection strings out of the server and into a column that admin tooling reads back, and none of it tells an operator more than the log already has — which is where the stack trace stays.
  • Its own transaction (REQUIRES_NEW), because the transaction the failure happened in is on its way to being rolled back and would take the record with it.

The user-facing side of the same gap

UserAPI.enrichWithReviewStatus is driven entirely by scan records. On an instance that does not run scanning — the reporter's setup — scanResult is null, so an inactive version reported "Your extension is being reviewed" indefinitely, for a version nothing was reviewing or ever would.

A recorded failure now outranks every scan state there. rejected is the closest of the three statuses that vocabulary has (the version will not become live without someone intervening) where under_review promises attention nothing is giving it. The recorded reason itself stays out of that response — it names server internals and there is nothing in it a publisher could act on — so they are pointed at the operator, who has the column.

The Error walked past the catch

publishAsync caught Exception in order to call markScanAsErrored. OutOfMemoryError is an Error, so the very failure that prompted #1450 slipped past it — even on an instance running scanning, nothing was recorded. Now it catches Throwable, records the reason, names the version in the log, and marks the scan where there is one.

The retry has not run since #1529

doPublish carries @Retryable, and the comment above its file-resource cleanup — "Delete file resources in case publishAsync is retried" — exists to make an attempt repeatable. Neither has done anything for a while.

Before #1529 the method was:

@Async
@Retryable
public void publishAsync(TempFile extensionFile, ExtensionService extensionService) {

Public, called from ExtensionService through the proxy, so both annotations applied. #1529 extracted the body into a private doPublish and moved @Retryable onto it — a method reached only by self-invocation, where no proxy can ever apply it. Nothing failed; the retry just quietly stopped happening. (@EnableResilientMethods is on RegistryApplication, and the @Retryable on the public createExtensionVersion does still work, which is what makes this easy to miss.)

Moved back onto the public methods, with one deliberate change: includes = Exception.class, so an Error fails on the first attempt instead of being retried three more times into a JVM that has just said it has no room — the exact scenario in #1450. That also matches what was retried before the move: spring-retry's SimpleRetryPolicy defaults to Exception, not Throwable.

Tests

PublishExtensionVersionHandlerRetryTest builds the handler behind real async and retry advice, rather than constructing it directly the way the existing handler test does — the behaviour under test is the advice, so a bare object cannot show it. A synchronous executor makes @Async run inline so nothing has to wait.

  • recordsWhyThePublishDidNotFinish — the reason reaches the version.
  • recordsTheFailureTypeWhenItCarriesNoMessage — the type alone, not a bare null appended to it.
  • retriesAFailedPublish — a storage failure is attempted four times (one plus three retries).
  • doesNotRetryAnError — an OutOfMemoryError is attempted once.
  • recordsAnErrorAgainstTheScan — the scan is marked errored with the OutOfMemoryError named.

Plus, on the service: the reason is written onto the managed row, a purged row is ignored rather than turning a failed publish into a second failure, and activation clears an earlier failure.

I checked the retry test actually bites rather than trusting a green run: with @Retryable back on the private doPublish, it fails with one attempt instead of four. The migration was applied against a real Postgres 16 to confirm the column lands nullable with no default — which also means import-db-dump.sh correctly treats it as needing no backfill.

Full server suite green (1175 tests), formatter clean.

Not in here

Surfacing the recorded reason through the admin API, so an operator can read it without reaching for psql. The column is the prerequisite; where it gets displayed is a separate decision.

🤖 Generated with Claude Code

netomi and others added 2 commits September 5, 2026 15:41
Two things behind #1450's second complaint - that a publish which fails
leaves the version inactive while the CLI reports success.

An OutOfMemoryError is an Error, and publishAsync caught Exception, so the
failure that prompted the report walked straight past the one handler that
would have recorded it: even an instance with scanning enabled marked
nothing, and the only trace was Spring's SimpleAsyncUncaughtExceptionHandler
line, which names the method rather than the extension it was publishing.
Catch Throwable, log which version it was and that the version stays
inactive, mark the scan errored where there is one, and rethrow.

The retry has not run since #1529. That PR extracted the body of the public
@async @retryable publishAsync into a private doPublish and moved @retryable
with it - onto a method reached by self-invocation, where no proxy can apply
it. The annotation, and the comment above the file-resource cleanup that
exists to make an attempt repeatable, have been describing something that
does not happen. Put it back on the public methods where the advice can see
it, with includes = Exception.class so that an Error fails on the first
attempt rather than being retried into a JVM that has just run out of room -
which also matches what spring-retry retried before the move.

The new test builds the handler behind real async and retry advice, because
that is where this behaviour lives: with @retryable back on the private
method, retriesAFailedPublish sees one attempt instead of four and fails.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Async publishing is a deliberate design decision - storing, signing and
checksumming a package are slow and have no business holding the upload
request open. What was missing is that when that work fails, nothing writes
down why. The row keeps active = false, and an operator looking at it has
only a stack trace in the log naming publishAsync rather than the version it
was working on, to be correlated by hand (#1450).

Add extension_version.publish_error, set when an attempt fails and cleared
when a version is activated, so it always describes the latest attempt. It
holds the failure's type and message and nothing deeper: a cause chain
carries paths, host names and connection strings into a column that tooling
reads back, and the log keeps the full account anyway.

Recorded in its own transaction, because the one the failure happened in is
on its way to being rolled back and would take the record with it.

The user-facing side of the same gap: enrichWithReviewStatus is driven
entirely by scan records, so on an instance that does not run scanning an
inactive version reported "Your extension is being reviewed" indefinitely -
for a version nothing was reviewing, or ever would. A recorded failure now
outranks every scan state there. The reason itself is not part of that
response; the publisher gets told to contact the operator, who has the
column.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@netomi netomi changed the title fix: record a failed async publish, and restore the retry #1529 disabled fix: record why an async publish failed, and restore the retry #1529 disabled Sep 5, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant