Isolate long-polls and file transfers on separate HTTP/2 pools by gautam-rl · Pull Request #825 · runloopai/api-client-python

gautam-rl · 2026-07-27T21:26:06Z

Problem

Under concurrent load, the Python SDK can stall short API calls (for example devboxes.create) while many long-lived requests share one HTTP/2 connection.

httpx/httpcore multiplexes streams on a single H2 connection and can block when that connection’s stream capacity is exhausted instead of opening another connection. Long-polls such as wait_for_status and large upload_file / download_file bodies are the usual way to fill that capacity.

Change

Route traffic onto three separate sharded HTTP/2 pools:

Pool	Used for	Default shards
API	Short RPCs (`create`, `execute`, list/get, …)	8
Background	Paths ending in `/wait_for_status`	16
Transfer	`/upload_file` and `/download_file`	2

Each shard is roughly one H2 connection. Requests are assigned with per-client round-robin from a random start offset, so busy clients spread across shards and many short-lived clients do not all land on shard 0.

Pooling is encapsulated in _SharedTransport.Registry / _SharedAsyncTransport.Registry, with per-client wrappers and a single _get_client_for_path(path) entry point.

Runloop(
    ...,
    api_pool_shards=8,
    background_pool_shards=16,
    transfer_pool_shards=2,
)

Passing a custom http_client disables this isolation; that client owns the full transport stack.

Shared transports remain process-wide so multiple SDK instances can reuse connections. Round-robin counters stay local to each SDK client.

Notes

Retries may use a different shard than the original attempt. That is intentional; idempotency is handled at the API layer, not by connection affinity.
Within a single shard, very high concurrency can still queue on httpcore’s per-connection stream limit until a long-poll completes. Raising the corresponding *_pool_shards increases capacity.

Test plan

Unit coverage for path classification, round-robin, random start offsets, and shared transports
TLS + ALPN integration tests: blocked waits / stalled uploads do not delay create on the API pool
Concurrent sync bulkhead initialization
Resources-tree invariant so new wait/upload/download paths stay classified

…s on HTTP/1.1 httpcore blocks on a per-connection stream semaphore once slots fill instead of opening another connection. Patch is_available + non-blocking acquire so the pool spreads load. Route upload_file/download_file through a dedicated HTTP/1.1 pool so bulk bodies cannot starve latency-sensitive RPCs on the shared H2 session. Co-authored-by: Cursor <cursoragent@cursor.com>

reflex-loop · 2026-07-27T21:26:14Z

✅ Reflex agent status: Completed

The agent completed its work.

Servers running in the agent's sandbox

Server	Port	URL
Body delay H2 server (HTTPS+ALPN h2 measuring headers→body delay)	`8765`	Open →

This PR was created by Reflex.

View the agent run →

_{This comment updates in place as the agent works.}

Drop the httpcore monkeypatch. Route long-polls (/wait_for_status) and upload/download onto separate shared H2 bulkhead pools with configurable shards (default 2), hashed by resource id, so control-plane RPCs keep their own connection without HTTP/1.1 explosion. Co-authored-by: Cursor <cursoragent@cursor.com>

Co-authored-by: Cursor <cursoragent@cursor.com>

Add TLS+ALPN integration tests that saturate waits / stall upload bodies and assert create lands on a different connection and stays fast. Guard sync bulkhead client creation with a lock and enumerate long-lived resource paths so new endpoints cannot silently join the API pool. Co-authored-by: Cursor <cursoragent@cursor.com>

Co-authored-by: Cursor <cursoragent@cursor.com>

Replace CRC32 resource affinity with per-client counters so concurrent waits/uploads spread across shards instead of pinning a busy resource to one connection. Co-authored-by: Cursor <cursoragent@cursor.com>

Avoid every new SDK client landing first wait/upload on global shard 0 when transports are process-shared. Co-authored-by: Cursor <cursoragent@cursor.com>

jrvb-rl · 2026-07-28T17:36:15Z

+        :param background_pool_shards: H2 shards for long-polls (round-robin), defaults to 2
+        :type background_pool_shards: int, optional
+        :param transfer_pool_shards: H2 shards for upload/download (round-robin), defaults to 2
+        :type transfer_pool_shards: int, optional


Hmm... It sounds like the python http stack doesn't auto-increase the number of connections? I'm surprised there isn't some way to enable that. (I think we should still have the separate pools -- it would just be nice if the shards-per-pool could scale with the request load...)

I'm also curious about the underlying problem here. HTTP/2 is supposed to share connections among many streams and break large messages on one stream into data frames to interleave them with smaller messages on other streams. I wonder if we could achieve the same thing by setting the stream config when we know we have large messages in either direction. (eg, lower the priority, set the interleave bit, potentially reduce the window size)

I believe the max streams-per-connection is set by the server, so that will influence things here as well. Do we know what Jetty & the NLB are doing here? (I imagine there might be some interplay b/w these as well?)

Regarding the pool sizes, 2 seems pretty small for the hanging-poll type messages. Clients might reasonably want to have 1000s of those at once. For the file upload/download pool, 2 also seems small, since it is often easier to saturate the link with a few more connections? (Then again, we could keep it small to start as a protection mechanism against accidentally spammy workloads...)

Long poll doesn't wait forever, it's capped to 25-30s. It's not ideal but this prevents the connection from getting starved out
The other option is to just scrap http2 but that's also expensive for these types of long poll endpoints

Ugh yeah - I just read up on python httpx - it really is just a single connect per server! That's a shockingly broken design...

Given that, I think it would be good to add a pool for the short-lived connections as well. It looks like the Jetty default is 128 streams/connection, so what about these pool sizes?
fast-pool: 5-10
long-polls: 10-20
data-xfer: 2

Updated defaults to API 8 / background 16 / transfer 2, and the short-RPC (API) pool is sharded as well now.

jrvb-rl · 2026-07-28T17:47:43Z

 class _SharedTransport(httpx.BaseTransport):
    """Refcounted wrapper: delegates to a real transport, closes it when refcount hits 0."""

    def __init__(self, transport: httpx.BaseTransport) -> None:


It might be cleaner to add the HTTP/2 connection pooling logic to the _SharedTransport class, so then we can just re-use that directly for the different request categories

Good call — pooling now lives on _SharedTransport.Registry / _SharedAsyncTransport.Registry, with per-client _SyncClientPool / _AsyncClientPool wrappers. Routing is a single _get_client_for_path(path).

Also took your size guidance: defaults are now API 8, background 16, transfer 2, and the API pool is sharded too (not a single control-plane connection anymore).

jrvb-rl · 2026-07-28T17:55:58Z

+    def _next_background_client(self) -> httpx.Client:
+        # Select under the lock; ensure afterward so _ensure_* can take the same lock.
+        with self._bulkhead_lock:
+            shard = self._background_next % self._background_pool_shards
+            self._background_next += 1
+        return self._ensure_background_client(shard)


we've some duplication here that would be cleaner if we pushed it into the _SharedTransport class. I think that would simplify the locking and acquire logic as well, and reduce the chance of errors later on.

Done — acquire/refcount/shard map are in the Registry; client wrappers no longer duplicate that locking.

jrvb-rl · 2026-07-28T17:59:59Z

+        if _is_background_path(path):
+            return self._next_background_client()
+        if _is_transfer_path(path):
+            return self._next_transfer_client()


how about just:

client = get_client_for_path(path)

and then we push all of the pool management down in to the wrapper library?

Switched to _get_client_for_path(path) as the single entry point; pool selection lives under that.

jrvb-rl

I like this direction - it definitely seems necessary given httpx's single-connection-per-server behavior for HTTP/2.

I think we can simplify this by encapsulating the pooling logic in the _SharedTransport class that we already have. See embedded comments below..

Move shard registries onto SharedTransport, route via get_client_for_path, and shard the API pool too (defaults: API 8 / background 16 / transfer 2). Co-authored-by: Cursor <cursoragent@cursor.com>

jrvb-rl · 2026-07-28T21:58:29Z

+                self._shards.clear()
+                return values
+
+        def shard_ids(self) -> set[int]:


Let's note that this is used only for testing

Done — take_all / shard_ids (and async clear) are documented as test-only.

jrvb-rl

Looks good - thanks for the tweaks!

Co-authored-by: Cursor <cursoragent@cursor.com>

gautam-rl changed the title ~~fix: H2 stream overflow → new conns; isolate file transfers on HTTP/1.1~~ fix: isolate waits and transfers on sharded H2 pools Jul 27, 2026

Reflex and others added 6 commits July 27, 2026 23:11

chore: fix ruff import sorting for pool bulkhead changes

6783ef6

Co-authored-by: Cursor <cursoragent@cursor.com>

chore: apply ruff format to bulkhead pool changes

bb09391

Co-authored-by: Cursor <cursoragent@cursor.com>

chore: fix pyright on H2 bulkhead integration test

8bdcd53

Co-authored-by: Cursor <cursoragent@cursor.com>

fix: round-robin H2 bulkhead shards per SDK client

b6451ee

Replace CRC32 resource affinity with per-client counters so concurrent waits/uploads spread across shards instead of pinning a busy resource to one connection. Co-authored-by: Cursor <cursoragent@cursor.com>

fix: randomize per-client H2 shard start offsets

5900470

Avoid every new SDK client landing first wait/upload on global shard 0 when transports are process-shared. Co-authored-by: Cursor <cursoragent@cursor.com>

gautam-rl changed the title ~~fix: isolate waits and transfers on sharded H2 pools~~ Isolate long-polls and file transfers on separate HTTP/2 pools Jul 28, 2026

gautam-rl requested review from james-rl and jrvb-rl July 28, 2026 00:33

jrvb-rl reviewed Jul 28, 2026

View reviewed changes

refactor: encapsulate sharded H2 pools; raise defaults

e6ef85a

Move shard registries onto SharedTransport, route via get_client_for_path, and shard the API pool too (defaults: API 8 / background 16 / transfer 2). Co-authored-by: Cursor <cursoragent@cursor.com>

gautam-rl requested a review from jrvb-rl July 28, 2026 21:43

jrvb-rl reviewed Jul 28, 2026

View reviewed changes

jrvb-rl approved these changes Jul 28, 2026

View reviewed changes

Reflex and others added 2 commits July 28, 2026 22:04

chore: mark Registry test helpers; fix pyright on pool cleanup

e7042d7

Co-authored-by: Cursor <cursoragent@cursor.com>

chore: ruff format _base_client.py

a0f69b3

Co-authored-by: Cursor <cursoragent@cursor.com>

gautam-rl merged commit d316188 into main Jul 28, 2026
7 checks passed

gautam-rl deleted the gautam/http2-stream-overflow-conns branch July 28, 2026 22:16

gautam-rl mentioned this pull request Jul 29, 2026

release: 1.27.0 #823

Merged

Uh oh!

Conversation

gautam-rl commented Jul 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Problem

Change

Notes

Test plan

Uh oh!

reflex-loop Bot commented Jul 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jrvb-rl left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jrvb-rl left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

gautam-rl commented Jul 27, 2026 •

edited

Loading

reflex-loop Bot commented Jul 27, 2026 •

edited

Loading