Conversation
Open
41 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem and change
On MetaX Flash Attention wheels that append
return_max_logittomha_varlen_fwd, InfiniCore builds a caller with the wrong C++ signature and fails to import with an undefined symbol. This change inspects the extension that will actually be linked and selects the matching declaration and call, passingfalseonly when the argument exists.FLASH_ATTN_2_CUDA_SOoverride. Other entry points retain their existing version handling; NVIDIA and other device adapters are unchanged.InfiniLM-v0.2.9c.Technical benefit and measured motivation
Direct before/after: affected-wheel import fails -> import, Prefill, Decode and model inference succeed. This restores access to the existing MetaX Flash Attention backend. There is no latency ratio against an import failure.
The following archived experiment quantifies the practical value of that backend. Both arms use the same ABI-compatible prototype library, changing only
attn_backendfrompaged-attntoflash-attn. These are backend-selection benefits, not kernel acceleration implemented by this ABI patch, and not a performance rerun of the final generalized resolver.Conditions: C500 50% compute / 32,000 MiB slice, six CPU cores; MACA3.5.3.20, driver3.8.30, torch2.8.0+metax3.5.3.9, flash-attn2.6.3+metax3.5.3.9torch2.8. BF16 model/KV, TP1/PP1, dense, eager, page256/pool32, chunking and prefix reuse off. 2048 input / 16 output tokens, greedy with EOS ignored for timing; two measurements after warmup per model/backend. TTFT is their mean; ITL is the mean of request medians. All eight measured outputs match HF eager and paired-backend controls. Full-library hashes match within each comparison.
Finite single-request output rates including Prefill were 21.93 -> 87.92 and 9.99 -> 39.46 token/s; these are not sustained serving throughput. Only two samples on a shared slice and two same-family models were measured. Gains come from the vendor attention implementation; InfiniLM still owns scheduling and KV-cache management.
Detailed conditions, attribution and reproduction · per-request measurements and source/binary hashes.
Validation
The result JSON, screenshot and detailed experiment report are preserved at archive revision
9ecc781f; the final diff contains only the ABI change, build resolver, fixture test and short configuration guide. Hardware measurements were not repeated for this cleanup.xmake lua tests/xmake/test_metax_flash_abi.lua: three accepted signatures and three negative cases passed; also checked against the installed C500 extension usingFLASH_ATTN_2_CUDA_SO.matrix.typeto the install command, so this is not evidence of distinct debug and release builds. Validation branch has the same implementation asf02ec327, with only an added workflow trigger; it is not a MetaX hardware CI result. Ruff passed.git diff --checkpassed. The scope cleanup only removes result attachments and trims the guide; the ABI implementation and fixtures are unchanged. The portable fixture was rerun successfully.Screenshot of the audited saved-results report (not a new GPU run):
Limits and remaining review gates
Older signatures have fixture coverage only; execution against older installed wheels remains untested. The MetaX hardware build reused unchanged objects; the successful clean CPU CI does not establish a clean accelerator build. Backend timings preceded the generalized signature resolver and are presented as motivation; no new-kernel speedup is claimed. No Ascend/Moore or unrelated device-support claim is added.
The upstream workflow reports
action_requiredwith zero jobs started and needs repository-side approval. Fork CI success does not replace upstream required checks.