Add response body flow control

[mkurz]

Summary

• Add a thread-safe ResponseBodyControl callback after final response headers.
• Support suspending, resuming, and cancelling HTTP/1.1 and HTTP/2 response bodies without coupling AHC to a streaming API.
• Pause the network read timeout while reads are intentionally suspended while leaving the request timeout active.
• Keep suspended HTTP/2 streams independent and cover transport backpressure, cancellation, timeouts, and connection reuse.

Motivation

AHC 3 removed StreamedAsyncHandler and its Reactive Streams integration in pull request #1843. That removal avoids coupling AHC to a particular streaming library, but AsyncHandler by itself has no way to stop transport reads while a downstream consumer has no demand.

Play WS is the driving consumer for this change. Play WS needs transport backpressure to preserve its existing Pekko Streams and Reactive Streams response APIs while upgrading to AHC 3. A working adapter on a currently local Play WS development branch turns this control into a single-subscriber Reactive Streams publisher and has been tested against this AHC branch.

This pull request adds only the transport primitive. Streaming-library policy and dependencies remain in Play WS, so AHC does not regain a dependency on Reactive Streams or JDK Flow.

Semantics

• AsyncHandler.onResponseBodyStart runs after final headers and before body parts, including for a response with no body.
• Calls to the supplied control are thread-safe, idempotent, and ignored after response completion.
• suspend() stops requesting new transport data, although body parts already read may still be delivered.
• resume() permits transport reads again.
• Returning State.ABORT is the synchronous callback-time way to stop processing. A handler can retain the control and call cancel() when an asynchronous decision is made after the callback returns.
• Cancelling an HTTP/1.1 body closes its connection when bytes may remain unread; cancellation after terminal content has been received can reuse a keep-alive connection. Cancelling an HTTP/2 body closes only its stream.
• Fully consumed responses retain the existing HTTP/1.1 pooling and HTTP/2 parent-connection reuse behavior.
• Suspension pauses only the network read timeout. The request timeout remains active; if the request timeout is disabled, an application that never resumes or cancels can retain the exchange and its transport resources indefinitely.

HTTP/2 flow control

Connections with no actively suspended response retain Netty's normal connection-level receive-window accounting. The default 65,535-byte connection window therefore continues to cap unconsumed flow-controlled DATA across all streams for users that never call suspend().

When the first response on a connection is suspended, AHC returns connection-level credit that has already accumulated and continues returning that shared credit as DATA arrives. This prevents the suspended stream from exhausting the connection window and starving sibling streams. Per-stream credit remains consumption-driven at all times, so each suspended stream is still bounded by its own receive window. Multiple simultaneous suspensions are counted, and normal connection accounting resumes after the last one ends. Credit already returned and data already queued cannot be revoked, and the controller tracks credit returned early so later application consumption does not return it twice.

During an active suspension, aggregate buffering can still scale with the number of concurrent streams; the relevant controls are http2InitialWindowSize, http2MaxConcurrentStreams, and the connection limits. The defaults do not impose a hard client-side aggregate bound during that interval: the initial per-stream window is 16 MiB and http2MaxConcurrentStreams = -1 leaves concurrency server-controlled. A rough upper-bound estimate is connection count times effective concurrent streams times the initial window, excluding network and decoder overhead. Applications requiring a finite policy must configure these values together. A hard aggregate byte budget is a separate design and is outside this pull request.

Netty's connection auto-refill state is private and fixed when DefaultHttp2LocalFlowController is constructed, so it cannot be enabled only for the lifetime of a suspension through composition or subclassing. The package-private SuspensionAwareHttp2LocalFlowController therefore adapts DefaultHttp2LocalFlowController from Netty 4.2.17.Final, preserving its normal behavior while making connection refill suspension-scoped.

This adaptation has a maintenance cost: AHC owns the copied flow-control logic and every Netty upgrade must compare it with the corresponding upstream implementation for correctness and security fixes. The exact source version and that obligation are recorded in the class Javadoc. An upstream Netty API that permits connection auto-refill to be changed at runtime would provide the exit path and allow AHC to remove the adaptation; no such API exists in Netty 4.2.17.Final.

The auto-refill mode requires a custom Http2Connection. Netty's builder treats server() and connection() as mutually exclusive, so ClientHttp2FrameCodecBuilder supplies the connection through the protected builder API and overrides isServer() to retain client mode.

The explicit gracefulShutdownTimeoutMillis(0) is not a new shutdown policy. Http2FrameCodecBuilder.forClient() selects zero through its package-private client constructor; the subclass must use the protected no-argument constructor, so it sets zero explicitly to preserve the existing client-factory behavior.

Scope and commit structure

The API/lifecycle work and the HTTP/2 independence work are kept as separate logical commits, but they belong in one pull request. Without the HTTP/2 work, a suspended response can consume the shared connection window and block unrelated sibling streams, so the public control would not have correct multiplexed behavior. Follow-up review fixes are also split into focused commits covering exchange ownership, terminal HTTP/1.1 cancellation, bodyless responses, interim responses, indefinite-suspension diagnostics, and suspension-scoped HTTP/2 refill.

History checked

• Issue #544 originally identified the lack of AsyncHandler backpressure, and pull request #963 addressed it by adding Reactive Streams support.
• Issues #1233 and #1721 document the interaction between downstream demand and read timeouts in the former streamed handler.
• Pull request #1843 removed StreamedAsyncHandler for AHC 3.
• Discussion #1925 asks how to migrate streamed consumers to AHC 3; the maintainer response declines restoring Reactive Streams because other libraries provide that policy and maintaining it adds overhead.
• Focused GitHub issue, pull-request, discussion, and local history searches found no existing AHC 3 proposal for a streaming-library-neutral suspend/resume/cancel response-body control.

Compatibility

• AsyncHandler.onResponseBodyStart is a new Java default method, so existing handler implementations remain source- and binary-compatible and retain their previous behavior unless they override it.
• ResponseBodyControl is a new public interface.
• NettyResponseBodyControl is public only to support AHC's cross-package transport integration and is marked @ApiStatus.Internal; consumers should depend on ResponseBodyControl instead.
• HTTP/1.1 informational responses from 102 through 199 are now treated as interim and no longer reach onStatusReceived or onHeadersReceived; this matches the existing HTTP/2 behavior and prevents a 103 Early Hints response from completing the exchange before the final response. Existing 100 Continue handling and 101 protocol switching are preserved.
• Cancelling HTTP/1.1 processing from a terminal trailer or body callback now reuses a keep-alive connection because LastHttpContent has already been received; it previously closed that fully read connection.
• The full JDK 11 verification, including Revapi, passes.

AI disclosure

OpenAI Codex on behalf of Matthias Kurz. The commits include Co-Authored-By: OpenAI Codex <codex@openai.com> per AGENTS.md.

Test plan

• On the first commit alone, the new HTTP/2 sibling-stream test reproduced shared connection-window starvation: a sibling response did not progress while the first response remained suspended.
• The focused ResponseBodyControlTest, Http2ResponseBodyControlTest, TimeoutTimerTaskTest, SuspensionAwareHttp2LocalFlowControllerTest, and Continue100InterceptorTest suites pass 29 tests: 12 HTTP/1.1 control tests, 6 HTTP/2 integration tests, 4 timeout-task tests, 5 flow-controller accounting tests, and 2 HTTP/1.1 Continue tests.
• The flow-controller tests cover unchanged connection accounting without suspension, connection-only refill during suspension, DATA received before the suspension callback, overlapping suspensions, and avoiding double credit after normal accounting resumes.
• JAVA_HOME=<jdk-11> ./mvnw clean verify: BUILD SUCCESS for the full reactor, including tests, Javadocs, coverage, and Revapi.
• The locally published AHC snapshot passes the full Play WS Scala 2.13 and Scala 3.3.8 test matrices on Java 17 and Java 21: 183 integration tests and 76 unit tests pass in each combination, with 2 expected pending tests.
• Play WS code validation, documentation, Scala 2 and Scala 3 MiMa checks, and dependency-tree verification pass against the locally published AHC branch.

[hyperxpro]

This turns the connection level bound off for every H2 connection, not only the ones that use the new control. With the shipped defaults (http2InitialWindowSize 16 MiB, http2MaxConcurrentStreams -1) nothing aggregate is left, so the ceiling becomes the per stream window times whatever concurrency the server picks. Before this a slow consumer was capped at 64 KiB per connection.

Can we auto refill only while a control on that connection is actually suspended, or keep an aggregate byte budget? I would rather not change the default for users who never touch ResponseBodyControl. The README points at http2MaxConcurrentStreams as the mitigation, but that is off by default too.

[mkurz]

Agreed. I chose suspension-scoped connection auto-refill and implemented it in 7bf9011.

Connections with no suspended response now retain Netty's normal 65,535-byte shared connection-window bound. When the first response on a connection is suspended, AHC returns connection credit that had already accumulated and continues returning connection-level credit while leaving per-stream credit consumption-driven. Multiple suspensions are counted, and normal connection accounting resumes after the last one ends. Credit already returned and data already queued cannot be revoked, so the remaining aggregate-buffering trade-off exists only during an active suspension and is documented in the API, README, and PR text.

The implementation also tracks connection credit returned early so application consumption after resume cannot return it twice. Tests cover the unchanged no-suspension path, DATA received before the suspension callback, overlapping suspensions, no double credit after resume, and the sibling-stream integration case.

This required adapting Netty's default local flow controller because its connection auto-refill state is private and fixed at construction; composition or subclassing cannot switch it for the lifetime of a suspension. That is a real maintenance cost, so 08be4ab5f records that the package-private implementation is forked from Netty 4.2.17.Final and that every Netty upgrade must compare it with upstream for correctness and security fixes. A runtime-switchable Netty API is the exit path that would let AHC delete the adaptation. I did not add a hard aggregate byte budget in this PR, and I have not claimed that an upstream Netty issue already exists.

[hyperxpro]

This runs before the two branches below that keep the exchange alive on purpose, the IOException filter one and recoverOnReadOrWriteException. In both we do not close the channel, but the control is already discarded, so the handler is left holding one where suspend(), resume() and cancel() are silent no ops. detach(false) also leaves autoRead off on a channel that survives. Should we move it down to the paths that really tear the channel down?

[mkurz]

Agreed. Fixed in 631bdb2.

The control is no longer discarded at the start of exceptionCaught. It is discarded only on paths that actually close the channel. Replay now completes the old control before the old response is drained, restoring reads, while a genuinely reused exchange receives a replacement control. The ownership also moved from the channel to NettyResponseFuture, so it follows the exchange and stale controls cannot affect a later exchange on the same channel.

The regression test suspends the first response, triggers an IOExceptionFilter replay, verifies that the old channel has autoRead restored for draining, verifies that a replacement control is delivered, and confirms that calls through the stale control do not affect the replayed response.

[hyperxpro]

Nit: this does an attribute lookup on every read complete for every channel, including each H2 stream child, and in the normal case readIfNeeded then no ops anyway. suspend0() always sets autoRead to false, so checking isAutoRead() first is equivalent and keeps the lookup off the hot path.

[mkurz]

Agreed. Fixed in 631bdb2.

channelReadComplete now checks channel.config().isAutoRead() first and returns immediately in the normal case. It only looks up the exchange/control when auto-read is already disabled, so the extra lookup is off the hot path for ordinary HTTP/1.1 channels and HTTP/2 stream children.

[hyperxpro]

When the HEADERS frame ends the stream we still create a control and call this, and finishUpdate runs a few lines down. A handler that suspends here gets no data and no error, and the resume() its consumer sends later hits !active and returns silently. Both new tests suspend unconditionally in onResponseBodyStart, so the path is exercised but the ignored suspend is never asserted. Either skip the callback when there is no body, or state on ResponseBodyControl that suspending in that case does nothing.

[mkurz]

I kept the callback for bodyless responses so a handler receives one consistent response-body lifecycle callback after final headers, and documented the exact limitation in dfd26a6: if the final headers also end the response, suspend() cannot defer completion. The control becomes inactive when the callback returns and later calls are no-ops.

There are now explicit HTTP/1.1 and HTTP/2 tests that suspend in this callback, verify that completion is not deferred, and verify that later control calls have no effect.

[hyperxpro]

finishUpdate calls complete() as its first statement, and if cancel() ran inline we already detached, so this never does anything. Same for the copy in HttpHandler.abortAfterStartingResponseBody. I would drop both and keep finishUpdate as the single completion point.

[mkurz]

Agreed. Fixed in 31434bb: both callback-abort copies of complete() are gone, and finishUpdate is the single control-completion point.

[hyperxpro]

Keying this on the channel is fragile. HttpHandler.handleRead returns early when applyIoExceptionFiltersAndReplayRequest replays and nothing detaches the control, so the channel enters drainChannelAndOffer still suspended with autoRead off. LastHttpContent never arrives, the drain never finishes, and the channel, its pool slot and the future are gone for good. If such a channel is reused later this throw fails a request that has nothing wrong with it. Keying the control to the exchange and replacing instead of throwing would remove both.

[mkurz]

Agreed. Fixed in 631bdb2.

The current control is now held atomically by NettyResponseFuture, not in a channel attribute. Creating a control replaces and deactivates the previous one instead of throwing, and stale controls become harmless no-ops. Replay explicitly completes the old control before draining its channel, restoring reads, and the next response installs a new control for the same exchange.

The replay regression test covers the suspended-channel drain, replacement control, and stale suspend()/cancel() calls without letting those calls affect the replayed response.

[hyperxpro]

ResponseBodyControl says calls made after completion have no effect, but this throws RejectedExecutionException at the caller once the event loop group is shutting down, which is exactly when a consumer calls cancel(). Catch it and treat it as the documented no op.

[mkurz]

Agreed. Fixed in 631bdb2.

NettyResponseBodyControl.execute now catches RejectedExecutionException; once the channel event loop is shutting down there is no remaining transport for the control to affect, so this follows the documented post-completion no-op behavior. A regression test closes the client and then calls suspend(), resume(), and cancel() through the retained control, verifying that none throws.

[hyperxpro]

With the request timeout disabled this re-arms forever and never reaches clean(), so a handler that suspends and then loses its consumer pins the future, the channel and the AsyncHandler for the life of the process. The description mentions the risk but there is no bound in the code. Can we cap the total suspended time, or at least log once when it runs long?

[mkurz]

I took the logging option and implemented it in ea01b27.

When the read-timeout interval expires while the response is still suspended and the request timeout is disabled, AHC now logs one WARN for that exchange explaining that transport resources remain retained until the application resumes or cancels it. It logs only once even though the read timeout continues to re-arm. I did not impose a new suspension deadline because that would turn an intentionally disabled request timeout into a different cancellation policy.

The test verifies the warning level, the request-timeout-disabled wording, and one-shot behavior.

[hyperxpro]

cancelTimeouts() is called from finishUpdate on the event loop for every request, so this monitor now sits on the I/O path. The timer thread holds the same monitor inside startReadTimeout across nettyTimer.newTimeout(), which blocks on first use and contends on the wheel's queue. On an H2 connection that parks the loop for every other stream on it. Re-checking cancelled after scheduling, or a CAS on the readTimeout field, closes the same race without a shared lock.

[mkurz]

Agreed. After rebasing onto current main, the synchronized implementation is gone. The upstream timeout changes now use an AtomicBoolean for one-shot cancellation, schedule without a shared monitor, and call cancelIfRaced after recording the scheduled handle. That closes the arm/cancel race without putting a monitor on the event-loop completion path. The suspension warning in ea01b27 builds on that lock-free implementation.

[hyperxpro]

Nit: this is the same bootstrap as Http2StreamingBodyFlowControlTest and five other H2 tests. Worth pulling into a helper next to TestUtils at some point so the next pkitesting or ALPN change is one edit instead of eight.

[mkurz]

Agreed that the repeated HTTP/2 bootstrap is worth consolidating. I kept that test-infrastructure refactor out of this PR because doing it properly would touch the existing HTTP/2 test classes as well and would add unrelated churn around the flow-control change. I will leave this as a focused follow-up rather than extracting a helper used only by the new test here.