Skip to content

Isolate node and clients on separate CPUs for the locust perf test - #8161

Closed
Amaury Chamayou (achamayou) wants to merge 22 commits into
mainfrom
achamayou-cpu-isolation-for-perf
Closed

Isolate node and clients on separate CPUs for the locust perf test#8161
Amaury Chamayou (achamayou) wants to merge 22 commits into
mainfrom
achamayou-cpu-isolation-for-perf

Conversation

@achamayou

@achamayou Amaury Chamayou (achamayou) commented Aug 14, 2026 •
edited
Loading

Copy link
Copy Markdown
Member

Summary

Perf tests run the CCF node and their load generator on the same machine, so both compete for the same CPUs. What gets measured is then partly a property of the load generator: a client that burns CPU while waiting takes it away from the node.

This pins the node and Locust clients to disjoint CPU sets. The node now gets exactly 3 logical CPUs, matching its two worker threads plus host thread.

How

  • A taskset prefix is applied as each process is spawned. The mask is inherited by every thread and child process, including Locust workers.
  • The split starts from sched_getaffinity(0), so outer restrictions such as a container cpuset or taskset around the runner are respected.
  • SMT sibling groups remain isolated between node and clients. If an exact 3-CPU node mask only partially fills a sibling group, the unmatched sibling is left unassigned rather than given to clients.
  • If SMT topology cannot be read reliably, or the machine is too small to split usefully, the benchmark warns and runs unisolated.

The local 16-vCPU simulation and the VMSS CI host both expose 8 physical cores with adjacent SMT pairs. Their layout is:

CPU isolation: node -> CPUs 13-15 (3 of 16), clients -> CPUs 0-11 (12 of 16), unassigned -> CPUs 12 (1 of 16)

Changes

Four files: tests/infra/cpu_isolation.py, the node hook in tests/infra/remote.py, the Locust client hook in tests/infra/locust_benchmark.py, and util-linux in scripts/setup-ci.sh to provide taskset.

No CHANGELOG.md entry: test infrastructure only.

Measurements

Local: 16-vCPU simulation

Local 20-vCPU/10-core host restricted with taskset -c 0-15, producing the same 16-vCPU/8-core SMT topology as VMSS CI. Basic Blocking Locust used the CI settings: 320 users, spawn rate 320, 10 Locust processes, 20 seconds, and 2ms signatures. Five isolated and five unisolated runs were interleaved. Individual request latencies were captured after dropping the first 3 seconds of each measurement window.

metric unisolated isolated, node=3 change
throughput 12,527 tx/s 13,571 tx/s +8.3%
latency CV 23.3% 23.3% -0.1%
p50 22.06ms 20.42ms -7.4%
p99 40.61ms 37.18ms -8.4%
p99.9 71.36ms 61.96ms -13.2%

The run ranges overlap substantially. The median paired throughput change was +4.0%, with individual pairs ranging from -6.6% to +30.2%, so this is directional rather than clean separation.

CI pool: two VMSS attempts

Workflow run 33183403235 ran twice on gha-vmss-d16av6-ci. Basic Blocking completed in both attempts. The baseline is the latest 30 unisolated main runs on the same pool.

signature interval main throughput node=3 throughput change main p50/p99 node=3 p50/p99
2ms 31,879 tx/s 33,040 tx/s +3.6% 8.97/14.2ms 8/14ms
20ms 15,161 tx/s 14,906 tx/s -1.7% 19/23.5ms 19/22ms
100ms 3,052 tx/s 3,016 tx/s -1.2% 99/110ms 99/130ms

These differences are small. The 2ms results sit within the recent main throughput range, while the 20ms and 100ms throughput reductions are about 1-2%.

Blocking issue: Basic JS fails with 3 node CPUs

The full perf suite fails reproducibly in Basic JS Locust:

environment node CPUs client CPUs failed requests
local 16-vCPU simulation shared shared 0 / 98,421
local 16-vCPU simulation 4 12 0 / 95,733
local 16-vCPU simulation 3 12 7 / 93,950
VMSS CI, attempt 1 3 12 18 / 135,331
VMSS CI, attempt 2 3 12 20 / 138,107

The local 3-CPU and 4-CPU controls give Locust the same 12 CPUs. Only the extra node SMT sibling changes, so the failure is caused by reducing the node from 4 CPUs to 3 rather than by starving the clients.

Conclusion

Do not merge this as configured. Three logical CPUs are enough for Basic Blocking, where isolation is broadly neutral and may improve the fastest-signature case, but they are not enough for the Basic JS workload: it returns HTTP 500s locally and in both CI attempts. Keep the PR draft until node capacity is workload-aware or restored to at least 4 CPUs.

@achamayou Amaury Chamayou (achamayou) mentioned this pull request Aug 14, 2026
Merged
eddyashton
Eddy Ashton (eddyashton) reviewed Aug 14, 2026
View reviewed changes
Comment thread CMakeLists.txt Outdated
eddyashton
Eddy Ashton (eddyashton) reviewed Aug 14, 2026
View reviewed changes
Comment thread CMakeLists.txt Outdated
@achamayou
Amaury Chamayou (achamayou) force-pushed the achamayou-cpu-isolation-for-perf branch 3 times, most recently from 6a803b7 to 82c3896 Compare August 14, 2026 16:44
@achamayou
Amaury Chamayou (achamayou) force-pushed the achamayou-cpu-isolation-for-perf branch from 82c3896 to 35256d4 Compare August 14, 2026 16:45
@achamayou
Add Basic Blocking Locust perf test
8cee892 
Adds pi_basic_blocking_locust, which measures the same blocking-write
workload as pi_basic_blocking but drives it with locust rather than
piccolo, so the number of concurrent clients can be varied.

The load is defined in tests/infra/basicperf_locustfile.py and uses
FastHttpUser, since HttpUser cannot drive enough requests per second to
saturate the service. tests/basicperf_locust.py owns the network, runs
locust against it, and converts locust statistics into bencher metrics
(throughput, latency, memory).

The key space helper shared with basicperf.py moves to
tests/infra/key_space.py, since basicperf.py can only be imported from
tests/infra.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 2da7bd10-ceb0-41a4-b6c4-d574595d91c9
@achamayou
Normalise new test files to LF line endings
0504b1a 
The three files added by this branch were committed with CRLF, unlike
every other Python file under tests/. No functional change.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 2da7bd10-ceb0-41a4-b6c4-d574595d91c9
@achamayou
Measure for a fixed window after users have spawned
1123d10 
Locust --run-time starts counting when locust starts, so it includes the
ramp, and --reset-stats discards the statistics gathered during the ramp
without extending the deadline. The measurement window was therefore
shorter than requested, and shrank as the spawn rate was lowered, until
it disappeared entirely.

This mattered because varying the client count is the point of this
test: at --users 128 --spawn-rate 4 the run ended mid-ramp and reported
127 tx/s instead of the ~1250 tx/s that 128 users actually sustain, and
did so without failing.

Start the shutdown timer from locust spawning_complete instead, so the
window is the same length whatever the spawn rate is, and rename
--run-time-s to --measure-time-s to describe what it now does.
--run-time is kept as a backstop against a run which never finishes
spawning.

Also fail, rather than report, when a run ends without having spawned
all users, or when the window measured is shorter than the one asked
for. Both produce plausible looking figures that do not describe steady
state.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 2da7bd10-ceb0-41a4-b6c4-d574595d91c9
@achamayou
Bind the locust master to a free port
cd4d789 
Locust workers reach the master on port 5557 by default, so a second
locust run anywhere on the same machine fails to bind. Pick a free port
per run instead, via the existing infra.net helper.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 2da7bd10-ceb0-41a4-b6c4-d574595d91c9
@achamayou
Fail with a diagnosis when locust reports N/A
13b2506 
Locust writes N/A rather than a number in the statistics CSV when it has
too few samples to compute a percentile. float() then raised a bare
ValueError, after the network had already been stopped, losing the run
with no indication of what had gone wrong.

Read the numeric columns through a helper which reports the column and
value, and says that the run did not gather enough data. Found by a run
which produced almost no samples.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 2da7bd10-ceb0-41a4-b6c4-d574595d91c9
@achamayou
Snapshot as rarely as the piccolo perf tests do
8ecbead 
add_piccolo_test passes --snapshot-tx-interval 10000 for every piccolo
perf test. add_e2e_test does not pass it at all, and e2e_args defaults
it to 10, which is sensible for functional tests but not for a
benchmark.

This test was therefore writing and fsyncing a ~213KB snapshot every 10
transactions for the whole run, which measures the disk rather than the
service, and makes the figure incomparable with Basic Blocking.

Found by the vegeta comparison work, which hit the same defect.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 2da7bd10-ceb0-41a4-b6c4-d574595d91c9
@achamayou
Measure at 5ms, 100ms and 1s signature intervals
35e9aae 
Blocking writes return once their transaction commits, and commit cannot
outpace the signature interval, so a single interval only measures one
regime. Sweeping three separates them: at 1s and 100ms the workload is
latency-bound and throughput is simply the client count divided by the
interval, while at 5ms the node becomes the limit and the benchmark
measures capacity instead.

Each interval gets its own network, since the interval is fixed in the
node configuration at startup. consensus_update_timeout_ms moves with
it, as in commit_latency.py, because commit cannot be observed faster
than the primary sends updates.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 2da7bd10-ceb0-41a4-b6c4-d574595d91c9
@achamayou
Plot benchmarks which main has no history for
afb4ad2 
A benchmark added by a branch has no main runs to build an EWMA baseline
from, so render_chart skipped it entirely. That made a new benchmark
invisible on the very pull request which adds it, which is when it is
most worth seeing.

Plot such a benchmark against the branch's own earliest run instead, so
its movement across the branch's runs is visible, and mark it as new. It
carries no standard deviation band and is never coloured as an
improvement or a regression, because there is nothing on main to compare
it against.

Borrowing a related benchmark's baseline was considered and rejected: an
axis normalised against something which measures a different thing shows
a difference which is not a change in CCF, and because the chart scale
follows the largest axis, one such axis compresses every other benchmark
into illegibility.

Also truncate long axis labels in the middle rather than at the end.
Benchmarks measured at several settings differ only in their suffix, so
truncating the end left the 100ms and 1000ms axes indistinguishable.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 2da7bd10-ceb0-41a4-b6c4-d574595d91c9
@achamayou
Drive 320 clients from 10 locust processes
ab07e8d 
The blocking workload is latency-bound at the longer signature intervals,
so throughput there is set by the client count: 320 clients raises the
100ms point from ~1260 to ~3100 tx/s and the 1s point from ~130 to ~314,
both within a few percent of clients divided by interval.

Ten sending processes rather than four keeps locust from becoming the
limit while driving that many clients, since each process drives all of
its users from a single thread.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 2da7bd10-ceb0-41a4-b6c4-d574595d91c9
@achamayou
Measure the fastest signatures at 2ms rather than 5ms
6df04a0 
The shortest interval is the point at which the node, rather than the
signature timer, becomes the limit, so it is the one which measures
capacity. Shortening it to 2ms pushes further past the latency-bound
regime. The node ticks every 1ms in these tests, so a 2ms interval is
representable.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 2da7bd10-ceb0-41a4-b6c4-d574595d91c9
@achamayou
Normalize new benchmarks against the branch's first run
46167f1 
The previous version reported a benchmark absent from main by its value
alone, marked (new), with no percentage. That read differently from every
other axis, and the marker made the label long enough to be truncated,
which is exactly what it should not have been for benchmarks whose names
differ only in a suffix.

Treat such a benchmark like any other axis instead, using this branch's
earliest run as its reference in place of the main EWMA baseline, so it
is normalized, labelled and coloured identically. The chart description
records that the reference differs.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 2da7bd10-ceb0-41a4-b6c4-d574595d91c9
@achamayou
Elide the middles of words to fit long axis labels
6dd8606 
Truncating the middle of the whole label cut out several words at once.
Elide word by word from the left instead, keeping each word's first and
last letter and replacing the middle with a single ellipsis character, so
a label degrades gradually and the last word, which is what distinguishes
one setting of a benchmark from another, stays readable longest.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 2da7bd10-ceb0-41a4-b6c4-d574595d91c9
@achamayou
Give new benchmarks a band like every other axis
829f0b8 
A benchmark with no main history had its standard deviation hardcoded to
zero, so all four band curves collapsed to a single point at the baseline
while every other axis carried a spread. That left a visible pinch in the
band and, because nothing fell inside a zero-width noise threshold, any
movement between branch runs was coloured as an improvement or a
regression.

Measure its spread the same way as for a benchmark with main history,
across the runs available, which for such a benchmark are the branch's
own. The radial zoom already covered these axes, since their values were
always part of the data the scale is fitted to.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 2da7bd10-ceb0-41a4-b6c4-d574595d91c9
Add opt-in CPU isolation for the locust perf test
500605e 
Pin the CCF node and the locust clients to disjoint sets of CPUs, so that the
Basic Blocking Locust benchmark measures the node rather than how the node and
the load generator happened to divide the machine between them.

The mechanism is a taskset prefix, which applies the mask before exec so it is
inherited by every thread and grandchild. That needs no privileges and no
cgroup access, so it works inside the unprivileged CI container.

Isolation is off by default and enabled only for pi_basic_blocking_locust, so
other tests are unaffected. Where it cannot be applied the test runs
unisolated with a warning, and the mode is recorded in bencher.json.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@achamayou
Amaury Chamayou (achamayou) force-pushed the achamayou-cpu-isolation-for-perf branch from 35256d4 to 500605e Compare August 18, 2026 16:07
Base automatically changed from locust-blocking-perf to main August 19, 2026 10:10
@achamayou
Merge main into CPU isolation branch
35ff105 
Keep main's 2ms, 20ms, and 100ms benchmark intervals while applying CPU isolation, and restore the standalone isolation test to bucket_c.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@achamayou
Merge remote-tracking branch 'origin/main' into achamayou-update-pr-8161
bf4e54b
@achamayou
Update CI test bucket inventory
372aaab 
Register cpu_isolation_test in the checked-in bucket_c snapshot so the CI inventory matches CMake.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot-Session: b591ca67-881e-413e-b387-e4f091201815
@achamayou
Preserve CI bucket snapshot ending
6a28fdc 
Keep the generated snapshot's final blank line so the byte-for-byte CI comparison remains stable.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot-Session: b591ca67-881e-413e-b387-e4f091201815
@achamayou
Merge remote-tracking branch 'origin/main' into achamayou-cpu-isolati…
6317d12 
…on-for-perf

# Conflicts:
#	tests/basicperf_locust.py
@achamayou
Trim CPU isolation to the Locust perf path
a64d450 
Keep only what isolates the node from the Locust clients: the taskset
prefix applied as each process is spawned, and the opt-in flags on the
Locust benchmarks.

Drop the unit test and its ctest registration, the piccolo client
pinning in remote_client.py, the bencher metadata recording, and the
CCF_CPU_ISOLATION environment overrides along with the disable() and
current() helpers that only existed to support them.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@achamayou
Always isolate, and log the layout up front
faf7c51 
Isolation was opt-in, enabled for one test by a CMake flag. We want it
everywhere, so drop the mode and the flags and split the CPUs
unconditionally at the top of every Locust benchmark run.

That call is now the first thing the benchmark does, so the layout is
reported before any other output:

  CPU isolation: node -> CPUs 16-19 (4 of 20), clients -> CPUs 0-15 (16 of 20)

A machine too small to split still warns and runs unisolated rather
than failing.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@achamayou
Reserve exactly three CPUs for Locust nodes
96db711 
The 16-vCPU CI hosts expose pairs of SMT siblings. Give the node exactly three logical CPUs and leave the unmatched sibling unassigned, preserving isolation without rounding the mask up to four. Refuse isolation when sibling topology is unavailable rather than risk sharing a physical core with clients.\n\nCo-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@eddyashton Eddy Ashton (eddyashton) eddyashton left review comments
Copilot code review Copilot
Copilot will automatically review once the pull request is marked ready for review

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@achamayou @eddyashton