fix(ctmf): pin CPU dispatch away from the crashing AVX-512 kernel - #83
Merged
Conversation
…kernel
CTMF r5's AVX-512 kernel for 8-bit input crashes the process outright — a
0xC0000005 access violation, so vspipe dies having printed nothing. The
encode then surfaces as the *encoder* ffmpeg failing on an empty Y4M pipe
("Header too large", exit -22) and the preview as a bare "exit code 1"
whose log ends after the routine API3 warnings. Nothing names CTMF.
It bites every radius except 2 (radius 2 has its own filterRadius2_*
kernel) and only at 8 bits (>=10-bit uses the uint16_t helpers). opt=1,
2 and 3 are bit-identical to each other and none of them crash. CTMF r5
(2020) is the newest upstream release, so there is nothing to upgrade to.
This is why the nightly Windows job went red on 2026-08-24 against a tree
unchanged since 08-20: x264's capability line in the same logs reads
"... AVX2" on the three passing nights and "... AVX2 AVX512" on the
failing ones. The runner hardware was the variable.
The plugin does NOT verify that the CPU supports the level it is handed —
opt=3 on a pre-AVX2 machine crashes exactly as opt=4 does here, and
ctmfCreate carries only a 0..4 range check. So the value cannot be a
constant: script_generator::ctmf_opt queries the CPU and emits 3 where
AVX2 exists, else 2 (SSE2 is part of the x86-64 baseline). It lives in
the worker rather than the script because it is a property of the
machine, not of the clip. Non-x86 builds ignore the argument.
Verified on an AVX-512 CPU: the failing nightly case now passes, and
removing the pin reproduces the CI error byte for byte. All 20
depth x radius combinations pass with the pin. The other six bundled
plugins exposing the same opt dispatch were swept and are clean.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes the nightly Windows failure.
ctmf.CTMFr5's AVX-512 kernel for 8-bit input (ctmfHelper_avx512<uint8_t, 16>) crashes the process — a 0xC0000005 access violation, so vspipe dies having printed nothing at all.That makes the symptom actively misleading. The encode surfaces as the encoder ffmpeg failing to read the empty Y4M pipe:
and a preview surfaces as a bare
exit code 1whose log ends after the routine API3 plugin warnings. Nothing anywhere names CTMF, and "Header too large" reads like a muxer or template bug.Why it appeared with no diff
The nightly went red on 2026-08-24 against a tree unchanged since 08-20, with deps unchanged since 08-15. The runner hardware was the variable — x264's capability line in the same logs:
... BMI2 AVX2... BMI2 AVX2 **AVX512**Scope
Measured against the bundle, not assumed:
radius=2Radius 2 escapes because it has its own
filterRadius2_*kernel; ≥10-bit escapes because it uses theuint16_thelpers.opt=1(C),2(SSE2) and3(AVX2) are bit-identical to each other and none of them crash. CTMF r5 (2020) is the newest upstream release, so there is no fixed build to take instead.The fix
script_generator::ctmf_optqueries the CPU and emits 3 where AVX2 exists, else 2; both templates now pass it asopt=.The query is the load-bearing part. The plugin does not verify that the CPU supports the level it is handed —
ctmfCreatecarries only a0..4range check, soopt=3on a pre-AVX2 machine installs the AVX2 kernels and crashes exactly asopt=4does here. A hardcoded3would have traded one crash for another.opt=0is the plugin's own auto-detect, which is precisely what selects the broken kernel.It lives in the worker rather than in the script because it is a property of the machine, not of the clip — unlike the depth scalings, no preceding pass can change the answer. Non-x86 builds compile the dispatch out and ignore the value.
Verification
Header too large, exit-22), confirming the local repro is the nightly failure rather than a lookalike.integration_new_passes_test→ CTMF end-to-end) passes on an AVX-512 CPU.Deliberately not changed
cas,grain.Add,tcanny,dfttest,warp.AWarpSharp2andeedi3mexpose the sameoptparameter and the sameinstrset_detect()dispatch. All six were swept atopt=0againstopt=3on an AVX-512 CPU at 8- and 16-bit and are clean. Pinning them pre-emptively would cost throughput and mask a real regression later.Tests
test_152asserts both generated scripts carry the pin and neveropt=0/opt=4;test_153asserts the value is one the CPU can actually run — the first test alone would pass against a level no CPU present can execute, which is the failure mode that matters. Dart twin extends the existing CTMF case inintegration_filter_parameters_test.dart.🤖 Generated with Claude Code