Skip to content

live runner: metered session dies after ~600 tickets because ticket params are never rotated #62

Description

@rickstaa

Summary

A metered live-runner session dies after ~600 signed tickets because the SDK reuses the ticket params from the original 402 challenge for the session's whole life. The orchestrator's recipient caps distinct sender nonces per recipientRand at 600 (pm/recipient.go:26, maxSenderNonces), so once that many tickets have been signed against one params set, every further payment is rejected:

HTTP 400 ... body='invalid ticket senderNonce: too many values sender=0x491F...E0DB nonce=611'

The SDK retries the same doomed nonce sequence, the orchestrator releases the session a few seconds later, and the stream ends.

How fast this happens is set by the orchestrator's ticketEV, not by anything the client controls: tickets per payment is fee / ticketEV. Measured against two orchestrators serving livepeer-example/realtime-transcription at the same price (73562861230 wei/s):

orch faceValue winProb ticketEV tickets/payment 600 reached after
0xdc28F2… 1.196e15 8.361e-04 1e12 wei 1.0 ~600 payments (~30 min)
0x9727b4… 1.196e15 8.361e-06 1e10 wei 30.4 ~20 payments (~60 s)

Both configurations are legal. Payment cadence is irrelevant: total tickets is spend / ticketEV however you batch them.

Root cause

src/livepeer_gateway/remote_signer.py (same on main and rs/live-runner-session-payments):

  • send_payment() POSTs to the session payment URL and discards the response (_post_empty, line 273). go-livepeer returns PaymentResult{Info: oInfo} on every successful payment (server/ai_http.go:569), and that OrchestratorInfo carries fresh TicketParams with a new seed — i.e. a new recipientRand that would reset the nonce map.
  • _payment_request() always sends "orchestrator": self._challenge.payment_params, the params from the initial 402, for the entire session.
  • run_payments() treats any 4xx except 408/429 as fatal, so the nonce error stops funding instead of refreshing.

Refresh is wired up, but only for one trigger: the signer returns HTTP 480 when it knows params expired, raising SignerRefreshRequiredPOST /refresh-payment. The 600-nonce cap is a recipient-side limit the signer cannot see, so it never fires. That is why the 1e12 ticketEV orchestrator survives: its params expire (40 blocks) and get refreshed before 600 tickets accumulate. It is timing, not correctness — a long enough session there fails identically.

Fix

  1. Preferred: consume PaymentResult.Info.TicketParams from each successful payment and use it for the next cycle. Matches what the orchestrator already sends and is safe at any ticketEV.
  2. Minimum: treat invalid ticket senderNonce like ticketparams expired — call /refresh-payment and retry. go-livepeer's own gateway does exactly this (server/ai_process.go:1556, isInvalidTicketSenderNonce).

Reproduction

24h WebSocket soak of livepeer-example/realtime-transcription, one held session streaming 16 kHz PCM, signer at a $0.50/hour cap:

  • against the 1e10 ticketEV orchestrator: 200 sessions in 5.2h, lifetimes median 87s / min 86s / max 88s, first payment failure at a constant +67s, 1,182 payment failures, 197 forced reconnects. Deterministic, not flaky.
  • against the 1e12 ticketEV orchestrator: 1 session, unbroken 5.17h, 4 TicketParams expired failures, all self-recovered.

The socket itself never faulted in either lane: 0 stalls, 0 transcript gaps, 0 socket errors across 9.9h of streaming. The only failure mode is payments.

Harness: live-runner-test/realtime_transcription_soak.py.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions