fix(etl-uvicorn): declare a rate limit retryable and normalize plugin_error.error_reason

[paulkarayan]

What & why

Problem: A consumer that trusts this library's retryable flag is told a provider rate limit is permanent, which is the one failure whose entire meaning is "try again later". plugin_error_of hard-coded retryable = false AND error_type = "configuration" for the whole legacy UserError family; RateLimitError carries status_code = 429 and inherited both, so a throttled request serialized as configuration / invalid_input / user / retryable = false -- terminal, and typed as caller misconfiguration rather than a dependency failure. The taxonomy this envelope implements says the opposite on both counts: a 429 from the caller's own provider is user-audience and retryable, and error.type, audience and retryable answer three independent questions, so retryability is orthogonal to audience rather than derivable from it. Separately, plugin_error.error_reason was assigned straight from the raised error's failure_category, so any plugin that set one put a SCREAMING_SNAKE token (AUTH_PERMISSION_DENIED) on a field the same taxonomy specifies as lower_snake_case. These are two distinct vocabularies and the platform already treats them as such elsewhere -- a sibling module maps each SCREAMING_SNAKE category onto its own distinct lower_snake reason -- and this library collapsed them into one. Both defects are this library mis-declaring the error contract it publishes to every plugin that imports it.

Change: RateLimitError now serializes error_type = "dependency", audience = "user", retryable = true and an error_reason defaulting to rate_limited; every other member of the family keeps configuration / invalid_input / user / retryable = false, because terminal is the safe default and the transient condition is the one that has to declare itself. error_reason is now normalized to a lower_snake_case token rather than copied verbatim, and the top-level failure_category response field still carries the plugin's original string unchanged.

Known gap left alone

For any error that is not a UserError, plugin_error_of returns None and emits no envelope at all, so audience is absent rather than merely wrong, and a plugin has no way to declare a non-user audience. That is deliberately untouched here. The repository's own test pins the current behaviour, and changing it would add a new plugin-facing contract surface -- a design decision about what a plugin may declare, not a bug fix. Calling it out so it is acknowledged rather than discovered.

Risk / rollback

Low. A pure mapping function with no state, no migration, and no configuration. Back it out by reverting the PR, or by pinning consumers to 0.1.0.

How it was verified

Ran the library's own suite (make check-ruff clean on the repo's pinned ruff; full pytest run green, six new tests added and no existing test modified or deleted). Wrote a probe that serializes the whole unstructured_ingest.error family through plugin_error_of and prints every field a consumer reads off the wire, ran it against the base revision and against this branch, and diffed the two -- output below. The consumer enumeration above was checked against each repository's origin/main, not a local checkout. Not verified: no deployed environment was exercised, because this package is the wrapper library a plugin imports rather than a deployed service, so a real local run of the wrapper plus its suite is the closest real environment. The retryability flip in particular cannot be proven end to end from a consumer's behaviour today, since nothing reads the field yet.

Proof

Repro (on the base revision), same probe both times:

error                   status  error_type      error_reason              audience    retryable
UserError               401     configuration   invalid_input             user        False
UserAuthError           401     configuration   invalid_input             user        False
RateLimitError          429     configuration   invalid_input             user        False
QuotaError              401     configuration   invalid_input             user        False
ProviderError           500     (no envelope emitted)
CategorizedUserError    401     configuration   AUTH_PERMISSION_DENIED    user        False

A 429 reached the wire terminal and typed as a misconfiguration, and a plugin-supplied AUTH_PERMISSION_DENIED landed verbatim on a lower_snake_case field.

Failing test before the fix:

$ pytest test/api/test_api.py -q -k "rate_limit or error_reason or non_rate_limited or failure_category"
>       assert plugin_error["retryable"] is True
E       assert False is True
>       assert body["plugin_error"]["error_reason"] == "auth_permission_denied"
E       AssertionError: assert 'AUTH_PERMISSION_DENIED' == 'auth_permission_denied'
>       assert body["plugin_error"]["error_reason"] == "auth_permission_denied"
E       AssertionError: assert 'Auth Permission-Denied ' == 'auth_permission_denied'
>       assert client.post("/invoke").json()["plugin_error"]["error_reason"] == "invalid_input"
E       AssertionError: assert '///' == 'invalid_input'
FAILED test/api/test_api.py::test_rate_limit_error_reaches_the_wire_as_retryable
FAILED test/api/test_api.py::test_precheck_rate_limit_error_is_retryable
FAILED test/api/test_api.py::test_error_reason_is_lower_snake_case_on_the_wire
FAILED test/api/test_api.py::test_error_reason_normalizes_a_non_snake_failure_category
FAILED test/api/test_api.py::test_unusable_failure_category_falls_back_to_the_default_reason
5 failed, 5 passed, 58 deselected

After the fix, same probe and same command. Only the two rows the change targets moved:

BEFORE (base)                                    AFTER (this branch)
UserError      configuration invalid_input F     UserError      configuration invalid_input          F
UserAuthError  configuration invalid_input F     UserAuthError  configuration invalid_input          F
RateLimitError configuration invalid_input F     RateLimitError dependency    rate_limited           T
QuotaError     configuration invalid_input F     QuotaError     configuration invalid_input          F
ProviderError  (no envelope emitted)             ProviderError  (no envelope emitted)
Categorized... configuration AUTH_PERMISSION...  Categorized... configuration auth_permission_denied F

The four rows that should not move are byte-identical, and ProviderError still emits no envelope -- the gap described above, reported and deliberately not fixed here.

Why normalize rather than lowercase: failure_category is arbitrary plugin-supplied text validated only as a string, so .lower() on "Auth Permission-Denied " yields a space rather than a snake_case token. Punctuation and whitespace runs collapse to single underscores, and a category with no usable characters ("///") falls back to the class default instead of putting garbage on the wire. Both cases are pinned by tests.

Dependencies / merge order

none

[paulkarayan]

Repro-first proof

Bugfix — plugin_error_of: stop marking a provider 429 terminal, and stop putting the SCREAMING_SNAKE failure_category vocabulary in the lower_snake_case error.reason field

Ticket: FIE-376

Reproduced the broken state

• Environment: local
• How: This package is the FastAPI wrapper library plugins import, not a deployed service, so the
  closest real environment is a real local run of the wrapper plus its own suite (no SND, DI or
  mirror was touched -- explicitly out of scope for this task).

Probe: serialize the legacy unstructured_ingest.error family through the wrapper's own
plugin_error_of and print every field a consumer reads off the wire.
Script: /private/tmp/claude-502/-Users-pk-orca-workspaces-field-engineering-biogen-ai-gateway/ce349394-98ea-4542-b122-1701082429b1/scratchpad/repro.py
Run: PYTHONPATH=. uv run python
Suite: PYTHONPATH=. uv run pytest -q -W ignore::DeprecationWarning

• Observed: RateLimitError (HTTP 429) reached the wire as error_type=configuration, error_reason=invalid_input,
  retryable=False -- terminal, and typed as a misconfiguration rather than a dependency failure.
  A UserError subclass carrying failure_category="AUTH_PERMISSION_DENIED" put that SCREAMING_SNAKE
  token straight into error_reason, which ERRORS.md specifies as lower_snake_case.
• Evidence: $ PYTHONPATH=. uv run python .../repro.py # on origin/main (d119d8e) error status error_type error_reason audience retryable UserError 401 configuration invalid_input user False UserAuthError 401 configuration invalid_input user False RateLimitError 429 configuration invalid_input user False QuotaError 401 configuration invalid_input user False ProviderError 500 (no envelope emitted) CategorizedUserError 401 configuration AUTH_PERMISSION_DENIED user False

Failing test (red)

• test/api/test_api.py::test_rate_limit_error_reaches_the_wire_as_retryable (unit) — committed
• transcribed by the author (not captured by a runner)

$ PYTHONPATH=. uv run pytest test/api/test_api.py -q -W ignore::DeprecationWarning \
    -k "rate_limit or error_reason or non_rate_limited or failure_category"
>       assert plugin_error["retryable"] is True
E       assert False is True
>       assert plugin_error["retryable"] is True
E       assert False is True
>       assert body["plugin_error"]["error_reason"] == "auth_permission_denied"
E       AssertionError: assert 'AUTH_PERMISSION_DENIED' == 'auth_permission_denied'
>       assert body["plugin_error"]["error_reason"] == "auth_permission_denied"
E       AssertionError: assert 'Auth Permission-Denied ' == 'auth_permission_denied'
>       assert client.post("/invoke").json()["plugin_error"]["error_reason"] == "invalid_input"
E       AssertionError: assert '///' == 'invalid_input'
FAILED test/api/test_api.py::test_rate_limit_error_reaches_the_wire_as_retryable
FAILED test/api/test_api.py::test_precheck_rate_limit_error_is_retryable
FAILED test/api/test_api.py::test_error_reason_is_lower_snake_case_on_the_wire
FAILED test/api/test_api.py::test_error_reason_normalizes_a_non_snake_failure_category
FAILED test/api/test_api.py::test_unusable_failure_category_falls_back_to_the_default_reason
5 failed, 5 passed, 58 deselected, 1 warning in 0.63s

Fix

• plugin_error_of declares RateLimitError as dependency / rate_limited / user / retryable=true,
  mirroring utic_plugin_base.errors.RateLimitError (platform-libs #889) field for field; every other
  UserError member keeps configuration / invalid_input / user / retryable=false, because terminal is
  the safe default and the transient condition declares itself. _as_error_reason normalizes a
  plugin-supplied failure_category to a lower_snake_case token for error_reason only; the top-level
  failure_category field still carries the original verbatim.
• Files: unstructured_platform_plugins/etl_uvicorn/api_generator.py

Proof it's resolved

• Test green: yes
• Environment: local
• Evidence: ``Same probe, same command, after the fix. Only the two rows the fix targets moved; the four that
  should not move did not, and ProviderError is still emitting no envelope at all (the audience hole,
  reported but deliberately NOT fixed here -- see [risk]).

BEFORE (origin/main d119d8e) AFTER (this branch)
UserError configuration invalid_input False UserError configuration invalid_input False
UserAuthError configuration invalid_input False UserAuthError configuration invalid_input False
RateLimitError configuration invalid_input False RateLimitError dependency rate_limited True
QuotaError configuration invalid_input False QuotaError configuration invalid_input False
ProviderError (no envelope emitted) ProviderError (no envelope emitted)
Categorized... configuration AUTH_PERMISSION_DENIED Categorized... configuration auth_permission_denied False

Suite, before and after:
before: 188 passed, 558 warnings in 3.17s
after: 194 passed, 3 warnings in 1.28s (6 new tests, no existing test changed)
Lint: make check-ruff -> All checks passed! (both). ruff format --check on the two changed
files is clean; the repo's 5 pre-existing format-drift files are unchanged and were already
drifting on origin/main.``

---

Auto-generated from this branch's .proof.toml (repro-first proof gate). Advisory.

[cubic-dev-ai]

1 issue found across 4 files

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="unstructured_platform_plugins/etl_uvicorn/api_generator.py">

<violation number="1" location="unstructured_platform_plugins/etl_uvicorn/api_generator.py:129">
P2: When a plugin supplies a `str` subclass whose `lower()` raises, `_as_error_reason` escapes while building the sanitized error response and produces a raw 500. Invoke the base `str.lower` before normalization.</violation>
</file>

<sub>Shadow auto-approve: would not auto-approve because issues were found.

Re-trigger cubic</sub>

[cubic-dev-ai]

P2: When a plugin supplies a str subclass whose lower() raises, _as_error_reason escapes while building the sanitized error response and produces a raw 500. Invoke the base str.lower before normalization.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At unstructured_platform_plugins/etl_uvicorn/api_generator.py, line 129:

<comment>When a plugin supplies a `str` subclass whose `lower()` raises, `_as_error_reason` escapes while building the sanitized error response and produces a raw 500. Invoke the base `str.lower` before normalization.</comment>

<file context>
@@ -111,19 +112,52 @@ def failure_category_of(error: BaseException) -> Optional[str]:
+    """
+    if category is None:
+        return None
+    reason = re.sub(r"[^a-z0-9]+", "_", category.lower()).strip("_")
+    return reason or None
+
</file context>

Suggested change

	reason = re.sub(r"[^a-z0-9]+", "_", category.lower()).strip("_")
	reason = re.sub(r"[^a-z0-9]+", "_", str.lower(category)).strip("_")