Skip to content

fix(ctmf): pin CPU dispatch away from the crashing AVX-512 kernel - #83

Merged
StuartCameronCode merged 1 commit into
mainfrom
fix/ctmf-avx512-crash
Aug 26, 2026
Merged

fix(ctmf): pin CPU dispatch away from the crashing AVX-512 kernel#83
StuartCameronCode merged 1 commit into
mainfrom
fix/ctmf-avx512-crash

Conversation

@StuartCameronCode

Copy link
Copy Markdown
Owner

Fixes the nightly Windows failure. ctmf.CTMF r5's AVX-512 kernel for 8-bit input (ctmfHelper_avx512<uint8_t, 16>) crashes the process — a 0xC0000005 access violation, so vspipe dies having printed nothing at all.

That makes the symptom actively misleading. The encode surfaces as the encoder ffmpeg failing to read the empty Y4M pipe:

[in#0] Header too large.
[in#0] Error opening input: Invalid argument
ffmpeg exited with exit code -22

and a preview surfaces as a bare exit code 1 whose log ends after the routine API3 plugin warnings. Nothing anywhere names CTMF, and "Header too large" reads like a muxer or template bug.

Why it appeared with no diff

The nightly went red on 2026-08-24 against a tree unchanged since 08-20, with deps unchanged since 08-15. The runner hardware was the variable — x264's capability line in the same logs:

nightly cpu capabilities result
08-21 → 08-23 ... BMI2 AVX2 pass
08-24, 08-25 ... BMI2 AVX2 **AVX512** fail

Scope

Measured against the bundle, not assumed:

8-bit ≥10-bit
radius=2 OK OK
every other radius crash OK

Radius 2 escapes because it has its own filterRadius2_* kernel; ≥10-bit escapes because it uses the uint16_t helpers. opt=1 (C), 2 (SSE2) and 3 (AVX2) are bit-identical to each other and none of them crash. CTMF r5 (2020) is the newest upstream release, so there is no fixed build to take instead.

The fix

script_generator::ctmf_opt queries the CPU and emits 3 where AVX2 exists, else 2; both templates now pass it as opt=.

The query is the load-bearing part. The plugin does not verify that the CPU supports the level it is handedctmfCreate carries only a 0..4 range check, so opt=3 on a pre-AVX2 machine installs the AVX2 kernels and crashes exactly as opt=4 does here. A hardcoded 3 would have traded one crash for another. opt=0 is the plugin's own auto-detect, which is precisely what selects the broken kernel.

It lives in the worker rather than in the script because it is a property of the machine, not of the clip — unlike the depth scalings, no preceding pass can change the answer. Non-x86 builds compile the dispatch out and ignore the value.

Verification

  • Negative control: removing the pin reproduces the CI error byte for byte (Header too large, exit -22), confirming the local repro is the nightly failure rather than a lookalike.
  • The failing nightly case (integration_new_passes_test → CTMF end-to-end) passes on an AVX-512 CPU.
  • All 20 depth × radius combinations (8/10/12/14/16-bit × radius 1/2/3/7) run clean with the pin.
  • Rust suite green (155 filter integration tests + 204/196/1); Flutter push gate 816 pass.

Deliberately not changed

cas, grain.Add, tcanny, dfttest, warp.AWarpSharp2 and eedi3m expose the same opt parameter and the same instrset_detect() dispatch. All six were swept at opt=0 against opt=3 on an AVX-512 CPU at 8- and 16-bit and are clean. Pinning them pre-emptively would cost throughput and mask a real regression later.

Tests

test_152 asserts both generated scripts carry the pin and never opt=0/opt=4; test_153 asserts the value is one the CPU can actually run — the first test alone would pass against a level no CPU present can execute, which is the failure mode that matters. Dart twin extends the existing CTMF case in integration_filter_parameters_test.dart.

Note when reading the CI result: runner assignment is not controlled, so a Windows job that lands on a non-AVX-512 runner would have passed before this change too. Check the x264 capability line in the log to know which kind of runner it got.

🤖 Generated with Claude Code

…kernel

CTMF r5's AVX-512 kernel for 8-bit input crashes the process outright — a
0xC0000005 access violation, so vspipe dies having printed nothing. The
encode then surfaces as the *encoder* ffmpeg failing on an empty Y4M pipe
("Header too large", exit -22) and the preview as a bare "exit code 1"
whose log ends after the routine API3 warnings. Nothing names CTMF.

It bites every radius except 2 (radius 2 has its own filterRadius2_*
kernel) and only at 8 bits (>=10-bit uses the uint16_t helpers). opt=1,
2 and 3 are bit-identical to each other and none of them crash. CTMF r5
(2020) is the newest upstream release, so there is nothing to upgrade to.

This is why the nightly Windows job went red on 2026-08-24 against a tree
unchanged since 08-20: x264's capability line in the same logs reads
"... AVX2" on the three passing nights and "... AVX2 AVX512" on the
failing ones. The runner hardware was the variable.

The plugin does NOT verify that the CPU supports the level it is handed —
opt=3 on a pre-AVX2 machine crashes exactly as opt=4 does here, and
ctmfCreate carries only a 0..4 range check. So the value cannot be a
constant: script_generator::ctmf_opt queries the CPU and emits 3 where
AVX2 exists, else 2 (SSE2 is part of the x86-64 baseline). It lives in
the worker rather than the script because it is a property of the
machine, not of the clip. Non-x86 builds ignore the argument.

Verified on an AVX-512 CPU: the failing nightly case now passes, and
removing the pin reproduces the CI error byte for byte. All 20
depth x radius combinations pass with the pin. The other six bundled
plugins exposing the same opt dispatch were swept and are clean.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@StuartCameronCode
StuartCameronCode merged commit 0d61996 into main Aug 26, 2026
11 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant