feat(examples): add durable seller production wiring

.github/workflows/ci.yml

@@ -130,6 +130,7 @@ jobs:
                  tests/conformance/decisioning/test_pg_buyer_agent_registry.py \
                  tests/conformance/decisioning/test_pg_idempotency_backend.py \
                  tests/conformance/decisioning/test_pg_task_webhook_outbox.py \
+                 tests/conformance/decisioning/test_pg_reference_workflow_queue.py \
                  -v
 
   conventional-commits:
@@ -472,7 +473,7 @@ jobs:
       - name: Install dependencies
         run: |
           python -m pip install --upgrade pip
-          pip install -e ".[dev]"
+          pip install -e ".[dev,pg]"
           # Example-local deps: the v3 reference seller imports
           # sqlalchemy + asyncpg + httpx-respx but those aren't in the
           # SDK's [dev] extras. Install them inline rather than adding

README.md

@@ -11,7 +11,7 @@ Official Python SDK for the **Ad Context Protocol (AdCP)**. Build and connect to
 This README serves both sides of an AdCP integration. Jump to what you're doing:
 
 - **Connect as a buyer** → [Quick Start: Test Helpers](#quick-start-test-helpers) and [Quick Start: Distributed Operations](#quick-start-distributed-operations). Entry point: `from adcp import ADCPClient, AgentConfig`; start with the `client.simple.*` API.
-- **Build a seller / agent** → [Building an AdCP Agent](#building-an-adcp-agent). Entry point: `from adcp.server import ADCPHandler, serve`.
+- **Build a seller / agent** → [Building an AdCP Agent](#building-an-adcp-agent). Entry point: `from adcp.server import ADCPHandler, serve`; use the [production seller path](docs/production-seller.md) when adding tenants, durable tasks, and webhooks.
 - **Understand the type system & imports** → [Type Safety](#type-safety) (import surface, partial modules, cold-start note).
 - **Test against reference agents** → [Quick Start: Test Helpers](#quick-start-test-helpers) and [Test Helpers](#test-helpers). Entry point: `from adcp.testing import test_agent, creative_agent`.
 
@@ -354,6 +354,7 @@ forward traffic degrades gracefully rather than failing.
 - **[API Reference](https://adcontextprotocol.github.io/adcp-client-python/)** - Complete API documentation with type signatures and examples
 - **[Protocol Spec](https://github.com/adcontextprotocol/adcp)** - Ad Context Protocol specification
 - **[Handler authoring](docs/handler-authoring.md)** - Building an AdCP-compliant agent on `adcp.server`
+- **[Production seller path](docs/production-seller.md)** - Choose the server abstraction and wire durable multi-tenant tasks, idempotency, and webhook delivery
 - **[Migrating from SDK 6 to 7](https://github.com/adcontextprotocol/adcp-client-python/blob/main/MIGRATION_v6_to_v7.md)** - Breaking API, security, concurrency, and webhook changes
 - **[Migrating from SDK 7 to 8](https://github.com/adcontextprotocol/adcp-client-python/blob/main/MIGRATION_v7_to_v8.md)** - Secure webhook defaults and telemetry changes
 - **[Migrating from AdCP 3.1 to 3.2 beta](MIGRATION_ADCP_3.1_TO_3.2.md)** - Compact lifecycle adoption and old/new compatibility matrix
@@ -1033,8 +1034,10 @@ from adcp.server import ADCPHandler, IdempotencyStore, MemoryBackend, serve
 from adcp.server.responses import capabilities_response
 
 idempotency = IdempotencyStore(
-    backend=MemoryBackend(),  # PgBackend with transactional commit is a follow-up
+    backend=MemoryBackend(),  # Use PgBackend for a durable, multi-worker cache
     ttl_seconds=86400,        # 24h, spec-recommended floor
+    # In production, also set raise_on_persist_error=True and independently
+    # deduplicate the business effect using the same buyer key.
 )
 
 class MySeller(ADCPHandler):
@@ -1063,9 +1066,14 @@ serve(MySeller(), name="my-seller")
 - On cache hit with different hash: raises `IdempotencyConflictError`, which the framework surfaces as `IDEMPOTENCY_CONFLICT` on both MCP (`is_error=true` + text) and A2A (failed task with `adcp_error` DataPart)
 - On cache miss: runs your handler, then commits the response
 
-**Backends:** `MemoryBackend` ships now (tests, single-process agents). `PgBackend` is scaffolded — it raises `NotImplementedError` with a pointer to the follow-up issue. For production use across multiple workers, implement your own `IdempotencyBackend` subclass against Redis, Postgres, etc.
+**Backends:** use `MemoryBackend` for tests and single-process agents. `PgBackend` provides a durable PostgreSQL replay cache for production deployments with multiple workers; it requires a separate advisory-lock pool and the `pg` extra.
 
-**Atomicity caveat:** `MemoryBackend` commits the cache entry AFTER your handler returns, so a crash between `handler success` and `cache commit` causes the retry to re-execute. `PgBackend` (follow-up) will commit the cache row in the same transaction as your business writes. Read the module docstring at `adcp.server.idempotency` before shipping this against a production database.
+**Atomicity caveat:** both backends commit the cache entry after your handler returns. `PgBackend` is durable and coordinates concurrent workers, but its cache transaction is not atomic with unrelated business writes. Put a uniqueness constraint on the business effect using the buyer's idempotency key so a crash between the side effect and cache commit cannot duplicate the effect. Read the `PgBackend` docstring before shipping it.
+
+With `raise_on_persist_error=True`, a failed cache write becomes retryable
+`SERVICE_UNAVAILABLE`, but the handler has already completed and its outcome
+may be unknown. A retry is safe only when the downstream business effect is
+independently deduplicated using the same buyer key.
 
 **How caller identity gets populated.** The middleware scopes its cache by `(caller_identity, idempotency_key)` — same key from two buyers must hit different cache slots, and a buyer's retry must replay only against its own prior call. `caller_identity` comes from `ToolContext`, which the transport layer builds per request:
 

docs/production-seller.md

@@ -0,0 +1,250 @@
+# Production seller path
+
+This is the shortest route from a working AdCP seller to a durable,
+multi-tenant deployment. The runnable implementation is
+[`examples/v3_reference_seller`](../examples/v3_reference_seller/README.md);
+this page explains which SDK layer to choose and who owns each lifecycle
+transition.
+
+## Choose the server abstraction
+
+| Start with | Use it when | Move up when |
+|---|---|---|
+| `adcp_server()` decorators | You need a small stateless agent and prefer functions | You need custom inheritance or reusable handler behavior |
+| `ADCPHandler` | You want direct control of AdCP request and response handlers | You need account resolution, specialism validation, upstream routing, or framework-managed tasks |
+| `DecisioningPlatform` | You are building an operational seller with accounts, capabilities, async work, and one upstream | Keep it; add a `PlatformRouter` when routing differs by tenant or platform |
+| `PlatformRouter` | One process serves multiple tenants or decisioning backends | This is the top-level composition layer |
+
+The production wiring reference uses `DecisioningPlatform`. Its modules deliberately
+separate transport wiring, tenant identity, business translation, persistence,
+and webhook delivery so adopters can replace one boundary at a time.
+
+## Runtime ownership
+
+```text
+buyer request
+    │
+    ▼
+auth → tenant router → buyer registry → idempotency lock
+    │
+    ▼
+DecisioningPlatform method → upstream ad server
+    │
+    ├─ terminal result ───────────────► return inline and cache result
+    │
+    └─ TaskHandoff / WorkflowHandoff ─► persist submitted task
+                                             │
+                          complete/fail ─────┤ one PostgreSQL transaction
+                                             ▼
+                                      task + outbox row
+                                             │
+                                      separate worker
+                                             ▼
+                                   signed webhook retry
+```
+
+The web process owns request validation and the atomic task/outbox commit. The
+worker owns network delivery and retry. Both processes use the same PostgreSQL
+database, 32-byte encryption key, signing key, and advertised retry horizon.
+They construct separate pools and `WebhookSender` instances.
+
+The bundled mock upstream completes its approval path inline, so it does not
+pretend that a process-local poll loop is restart-safe. A production adapter
+whose upstream approval can outlive the request must return
+`WorkflowHandoff`, persist the framework-issued task id in its own durable
+queue, and have that queue's consumer call `registry.complete()` or
+`registry.fail()`. The PostgreSQL registry makes task state durable; it does
+not make arbitrary in-process work durable. The reference includes a leased
+PostgreSQL queue and restart-recovery test; adopters still provide the
+business-specific approval handler.
+
+The reference's `IdempotencyStore` wrapping is intentionally paired with its
+inline terminal responses. Do not put the same method-level wrapper around a
+method that returns a raw `TaskHandoff` or `WorkflowHandoff`: the wrapper runs
+before framework task issuance and therefore cannot cache the projected
+`{status: "submitted", task_id}` envelope. A durable workflow adapter must not
+advertise method-level idempotency for that method unless an external durable
+request-to-task mapping can reuse the prior task id. The reference queue's
+uniqueness constraint is only on the framework-issued `task_id`, not the
+buyer's idempotency key. A web-process
+crash after enqueue commits but before the `submitted` response reaches the
+buyer can therefore cause a retried request to issue a second task and queue
+row. Fully closing that window requires SDK support for looking up or reusing
+a workflow task id by buyer idempotency key.
+
+For a mixed adapter, make the split explicit with
+`method_level_idempotency_methods`: include only methods that return terminal
+responses from the method-level cache, and implement the workflow method's
+deduplication in the durable queue. The default reference includes
+`create_media_buy` because its mock path is inline.
+
+## Task transitions
+
+| Handler outcome | Work owner | Persistence owner | What the buyer does next |
+|---|---|---|---|
+| Return a result | Request handler | Idempotency backend caches the terminal response | Consume the inline result |
+| Raise `AdcpError` | Request handler | Framework projects the structured error; no task is created | Follow its recovery guidance |
+| `ctx.handoff_to_task(fn)` | SDK runs `fn` in the web process | `PgTaskRegistry` records submitted, progress, and terminal state | Poll `tasks/get` or await a webhook |
+| `ctx.handoff_to_workflow(enqueue)` | The adopter's queue/worker/HITL system | The enqueue callback stores the task id; that system later calls `registry.complete()` or `registry.fail()` | Poll `tasks/get` or await a webhook |
+| Input-required response | The business workflow that needs clarification | That workflow must retain the continuation state and context id | Resume the same context with the requested input |
+
+Use `TaskHandoff` only for bounded in-process work. A human review, Airflow
+DAG, or long-running queue consumer belongs in `WorkflowHandoff`; otherwise a
+web-process restart can strand the work even though the task row survived.
+
+For push-configured tasks, `PgTaskRegistry.complete()` and `.fail()` write the
+terminal task and encrypted outbox envelope atomically. The worker leases the
+outbox row, sends the exact stored body, and retries it with a stable
+idempotency key. Polling and callbacks therefore observe the same terminal
+artifact.
+
+## Durable WorkflowHandoff example
+
+[`workflow_queue.py`](../examples/v3_reference_seller/src/workflow_queue.py)
+is a PostgreSQL-backed adopter queue with expiring leases. The enqueue callback
+stores the framework task id before `WorkflowHandoff` returns `submitted`; a
+replacement worker reclaims an expired lease after a crash. Handler failures
+retry with capped exponential backoff; after the configured attempt limit the
+registry task fails and the queue row moves to `dead_lettered`. Jobs without a
+matching account-scoped registry task dead-letter immediately.
+
+```python
+queue = task_wiring.workflow_queue
+
+async def create_media_buy(self, req, ctx):
+    upstream_order = await create_upstream_order(req)
+    payload = {
+        "upstream_order_id": upstream_order["id"],
+        "downstream_idempotency_key": req.idempotency_key,
+    }
+
+    async def enqueue(task_ctx):
+        await queue.enqueue_from_handoff(
+            task_ctx,
+            account_id=ctx.account.id,
+            workflow_type="manual_media_buy_approval",
+            payload=payload,
+        )
+
+    return ctx.handoff_to_workflow(enqueue)
+
+async def handle_approval(job):
+    # Any external write here must deduplicate on the stored buyer key.
+    return await approve_and_build_result(job.payload)
+
+# In the separately supervised worker entrypoint (includes SIGTERM handling):
+await run_with_signals(workflow_handler=handle_approval)
+```
+
+The queue completes the original `PgTaskRegistry` record only after the
+handler returns. A crash after an external side effect but before queue
+acknowledgement causes deliberate re-execution after lease expiry, so the
+business effect must be independently idempotent. The PostgreSQL conformance
+test kills the first logical worker after claim, creates fresh queue/registry
+objects, and verifies that the replacement completes the same task id.
+Queue payloads are ordinary JSONB: store only minimal continuation state and
+never copy push-notification credentials or other secrets into them.
+
+## Run the reference deployment
+
+Install the PostgreSQL extra and start the development database:
+
+```bash
+pip install -e '.[dev,pg]'
+cd examples/v3_reference_seller
+docker compose up -d postgres
+```
+
+Generate distinct webhook-signing and outbox-encryption keys. Keep both in a
+secret manager in production; the environment variables below are for the
+local runnable path.
+
+```bash
+adcp-keygen --alg ed25519 --purpose webhook-signing \
+  --kid reference-webhook-key \
+  --out /tmp/adcp-reference-webhook-signing.pem
+
+export ADCP_TASK_DATABASE_URL=postgresql://postgres@localhost/adcp
+export ADCP_TASK_WEBHOOK_ENCRYPTION_KEY="$(openssl rand -base64 32)"
+export ADCP_WEBHOOK_SIGNING_KEY_PATH=/tmp/adcp-reference-webhook-signing.pem
+export ADCP_WEBHOOK_SIGNING_KEY_ID=reference-webhook-key
+export ADCP_WEBHOOK_SIGNING_ALG=ed25519
+export ADCP_TASK_WEBHOOK_RETRY_HORIZON_SECONDS=86400
+```
+
+For a local smoke test, start the mock upstream and web process as background
+jobs, then leave the worker in the foreground. The exported configuration is
+shared by all three processes:
+
+```bash
+npx -y -p @adcp/client@latest \
+  adcp mock-server sales-guaranteed --port 4503 --api-key test-key &
+
+DATABASE_URL=postgresql+asyncpg://postgres@localhost/adcp python -m seed
+
+ADCP_ENV=development \
+DATABASE_URL=postgresql+asyncpg://postgres@localhost/adcp \
+MOCK_AD_SERVER_URL=http://127.0.0.1:4503 \
+MOCK_AD_SERVER_API_KEY=test-key \
+python -m src.app &
+
+python -m src.worker
+```
+
+That block is a loopback smoke test using public fixture credentials; never
+expose it or reuse its token. A real production entrypoint must replace the
+seeded bearer map with OAuth or RFC 9421 verification, use managed TLS
+PostgreSQL and a non-mock upstream, and inject database credentials and signing
+material from a secret manager. Set `ADCP_ENV=production` only for that real
+configuration; it makes the durable bundle mandatory at boot.
+
+In production, supervise those long-lived commands separately and inject the
+same secret-manager values into both the web and worker processes. The example
+validates the complete durable configuration before binding the HTTP listener.
+A partial key, database, encryption, or retry configuration fails with the
+missing field names. The retry horizon is projected into capabilities and must
+match the outbox value.
+
+`DurableTaskWiring.startup()` calls `create_schema()` for a convenient local
+bootstrap. The workflow example performs the one additive upgrade shown here,
+but these runtime DDL calls are not a general schema migration system and do
+not detect or safely evolve an arbitrarily mismatched table.
+For production, copy the SDK-owned SQL files (`decisioning_tasks.sql` and
+`task_webhook_outbox.sql`), the `PgBackend.create_schema()` DDL, and the
+reference workflow-queue DDL into reviewed, versioned migrations and apply
+them before either process starts. Runtime bootstrap can remain a safety net,
+but migrations own schema evolution and rollback.
+
+The worker installs `SIGTERM` and `SIGINT` handlers, cancels its polling loops,
+awaits their cleanup, and then closes the sender and PostgreSQL pools. This is
+the shutdown path used by ordinary container and process supervisors.
+
+## Production checklist
+
+- Replace bearer fixture authentication with your OAuth or RFC 9421 verifier.
+- Use managed PostgreSQL with TLS and credential authentication; never deploy
+  the example Compose file or seed data.
+- Publish the public webhook JWK and keep request-signing and webhook-signing
+  keys distinct.
+- Run at least one separately supervised outbox worker and alert on expired or
+  quarantined rows.
+- Route human or long-running approval work through `WorkflowHandoff` and a
+  durable queue; reserve `TaskHandoff` for bounded work that may safely fail on
+  web-process restart.
+- Put a uniqueness constraint on business effects keyed by the buyer's
+  idempotency key. The SDK cache cannot make a separate upstream transaction
+  atomic.
+- Schedule `PgBackend.delete_expired()` (or equivalent SQL/pg_cron cleanup) so
+  expired idempotency rows do not accumulate.
+- Apply URL challenge and SSRF validation before accepting durable callback
+  destinations.
+- Run the in-process tests and the media-buy seller storyboard before deploy.
+
+`DurableTaskWiring` remains example-owned so its configuration surface can be
+validated by adopters first. Once the registry, queue, signing, migration, and
+shutdown contracts stabilize together, it is a candidate for an SDK-supported
+production builder rather than copyable scaffolding.
+
+For constructor details and multi-tenant sender resolution, continue with
+[`handler-authoring.md`](handler-authoring.md#webhooks). For tenant scoping
+invariants, see [`multi-tenant-contract.md`](multi-tenant-contract.md).