New RPCConnectionManager: Single unified manager, no ringbuffer, OpenSSL-owned sockets - #8117
Draft
Eddy Ashton (eddyashton) wants to merge 78 commits into
Draft
New RPCConnectionManager: Single unified manager, no ringbuffer, OpenSSL-owned sockets#8117Eddy Ashton (eddyashton) wants to merge 78 commits into
RPCConnectionManager: Single unified manager, no ringbuffer, OpenSSL-owned sockets#8117Eddy Ashton (eddyashton) wants to merge 78 commits into
Conversation
…ransport cert-deferred listening + ALPN + outbound client, RPCConnectionManager (AbstractRPCSessions). Not yet wired into enclave/run.cpp.
…wire RPCConnectionManager into enclave.h/run.cpp, delete RPCSessions/rpc_connections/tls_session. Full build green, 53/53 unit tests pass.
…:Cert::use) and request client cert on inbound for caller auth; add peer-cert capture test. Full build green, unit tests pass.
…xing localhost/[::1] interfaces (cpp, cpp_cose_only, common_ipv6 e2e). Add localhost/IPv6 binding tests.
… large response queued just before close_socket() is not truncated (fixes cpp/cpp_cose_only receipt 'server disconnected'). Add truncation test.
…e (ERR_clear_error) before each SSL op so a stale error from one connection cannot poison SSL_get_error for another (root cause of cpp/cpp_cose_only 'server disconnected'). Add persistent-connection + peer-cert tests.
… handler; branch udp interfaces to listen_udp. Clearly marked QUIC extension points (substrate for OpenSSL >=3.5 native QUIC). e2e_logging udp echo passes; full suite green.
…ned quic_session.h/src/quic, udp.h + udp/msg_types.h + UDPImpl vestiges in run.cpp, dead RPC ringbuffer message enums (keep tcp::ConnID). Drop old-implementation comments. Fix build after cert.h use->configure_ssl rename + commit-callback include.
…Config (so the enclave-side RPC manager receives it); track per-connection last_active in OpenSSLServer and sweep idle connections off the epoll timeout. Plumbed manager->bridge->server. idletimeout e2e passes.
Notify the enclave work beacon when transport tasks enter the JobBoard, while preserving direct worker handoff and coalescing redundant wakeups. Keep bounded task drains moving immediately when a backlog remains.\n\nRefs #8117\n\nCo-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…code - Modify CMakeLists.txt for better configuration. - Refactor rpc_tls_client.h for improved clarity and functionality. - Enhance openssl_server_test.cpp with additional test cases.
RPC task workers can query consensus state concurrently with Raft message processing. Publish a coherent query snapshot without taking the Raft lock from KV-backed endpoints, avoiding both data races and KV/Raft lock inversion.\n\nCo-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> # Conflicts: # CMakeLists.txt # src/tls/test/main.cpp
Amaury Chamayou (achamayou)
marked this pull request as ready for review
August 19, 2026 10:12
These changes were added while chasing TSAN failures exposed during the RPC connection manager work, but they are broader Raft synchronization changes and should be handled separately rather than bundled into this PR. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Amaury Chamayou <amchamay@microsoft.com>
This was referenced Aug 26, 2026
Eddy Ashton (eddyashton)
marked this pull request as draft
August 27, 2026 09:39
Refresh the long-lived RPC connection manager work against current main, preserving the upstream AFT concurrency fixes and adapting the transport/task changes to current APIs. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Include the latest partition-test race fix after main advanced during final verification. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Restore the legacy behavior of presenting configured intermediate certificates, add regression coverage, and update the TLS protocol golden report for intentionally disabled session resumption. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Eddy Ashton (eddyashton)
added a commit
that referenced
this pull request
Sep 7, 2026
Prioritise the recovery lifecycle over the startup snapshot when selecting locally downloadable ledger history. Previous-service recovery input may be read-only, so every recovery-start node must begin at the current service creation transaction. Add deterministic snapshot and ledger-only recovery/readability coverage, including explicit lifecycle watermark assertions for recovery nodes, snapshot joiners, and genesis or snapshot-less joiners. Follow up on #8266 and the residual failure exposed by #8117 ASAN. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR replaces the old split RPC networking path with an OpenSSL-native connection layer. Previously, socket handling lived in host-side libuv code while TLS and protocol sessions were driven through enclave-side ringbuffer messages and memory BIOs. With CCF now running as a single process, that split is no longer useful, so RPC sockets, TLS, protocol session creation, and per-interface policy now live behind a single RPC connection manager.
TLS now terminates at the connection layer. Protocol sessions receive plaintext and write responses through a
SessionWriter, so HTTP, HTTP/2, and custom protocols no longer own TLS state directly. This removes the RPC ringbuffer message path, the memory-BIO TLS session layer, and the old libuv RPC connection containers.The new transport uses non-blocking sockets bound directly to OpenSSL, and splits the work in two. The existing host libuv loop owns the server's own state - accepting connections,
uv_poll_tregistration, theSSL_CTX, idle connection cleanup viauv_timer_t, and closing file descriptors - and performs noSSLoperations itself. EverySSLoperation for a connection instead runs on that connection's ownOrderedTasksqueue, keeping handshakes and bulk encryption off the loop thread. The loop only schedules a pass over a connection, driven either by file descriptor readiness or by a cross-thread request (a queued write, a close, or a certificate update) marshalled throughuv_async_t. A connection is serviced by at most one pass at a time. Per-interface behavior such as certificates, session caps, metrics, HTTP parser settings, and custom protocol dispatch is centralized inRPCConnectionManager.Session caps are applied when a connection is accepted, before any TLS state exists, rather than when its first request arrives.
max_open_sessions_hardis documented as a bound on connections, and counting at first-request time missed any client that completed the TCP and TLS handshakes and then went silent while still holding a file descriptor and TLS state.Node outbound requests are outside this transport and use libcurl.
UDP remains as a small datagram transport driven by
uv_poll_t. The temporary QUIC/UDP echo behavior is stateless, so it consumes no session and no interface capacity, while custom UDP protocols are routed to per-peer sessions. Native QUIC is still future work and requires OpenSSL 3.5 or later.Structural Breakdown
OpenSSLServeris the low-level inbound connection transport. It owns the listening and accepted socket file descriptors,SSLobjects,uv_poll_thandles, read/write buffers, handshake state, graceful-close state, certificate reload requests, and idle-timeout sweeping.OpenSSLSessionManagerbridges transport connections toccf::Session. It lazily creates sessions for inbound connections, forwards plaintext bytes into sessions, implementsSessionWriter, and reports connection closure back to the owner.RPCConnectionManageris the higher-level RPC owner. It replaces the old RPC session container and owns one transport bridge per TCP interface, plus UDP interface state. It applies per-interface admission and caps, certificates, parser settings, application protocol selection, session metrics, custom protocol routing, and UDP peer demultiplexing.SessionWriter,Session, andPlaintextSessionform the new session boundary. Sessions no longer encrypt or decrypt; they parse plaintext and emit plaintext responses to their writer. HTTP/1 and HTTP/2 sessions now use this boundary.CustomProtocolSubsystemInterfacenow creates sessions from(ConnID, SessionWriter&)rather than a TLS context. This matches the new layering: custom protocols see plaintext and write through the transport-neutral writer.DatagramServeris the UDP socket transport. It uses auv_poll_thandle on the existing libuv loop.RPCConnectionManagerechoes datagrams directly for the temporary QUIC behavior, and maps UDP peers to sessions for custom datagram protocols.Startup wiring moved accordingly. The enclave creates and owns the RPC manager, binds RPC interfaces, resolves actual bound addresses including ephemeral ports, and reports those addresses back through the enclave entry point so the host can write the RPC addresses file.
The removed files are the old RPC transport stack:
RPCSessions,TLSSession, host RPC connections, legacy UDP plumbing, the old QUIC session, and the TCP/UDP ringbuffer message types that were specific to the split RPC path. Ledger, consensus, and node-to-node uses of ringbuffer and libuv are not part of this change.