Add support for Qwen3.8 27B model CPU + GPU - #150
Open
orionpapadakis wants to merge 47 commits into
Open
Conversation
Qwen3.8-27B declares general.architecture=qwen35: a hybrid stack where only every fourth layer is attention and the remaining 48 of 64 are Gated Delta Net linear attention, plus one MTP/NextN block. Records the inventory (metadata, tensor table, both layer geometries, the quantization mix), walks the extension points, and justifies the four OperationKind values the mixer needs -- L2_NORM, CAUSAL_CONV_1D, DELTA_RULE_UPDATE, GATED_NORM -- against the alternatives, including the family-shaped composite the porting skill warns about. Also records why the GPU path is blocked on memory rather than kernels: materializing this file's Q4_0 weights as Q8_0 needs ~28GB.
Qwen3.8-27B-Q4_0.gguf is Q4_0 throughout except the first eight layers' ffn_down, which the quantizer emitted as Q4_1 -- so the file could not be loaded at all without it. Q4_1 is Q4_0's affine sibling: 32 values to a block, unsigned nibbles, and a per-block minimum alongside the scale, reconstructing d*q + m. The vectorized dot product distributes over that form, carrying the running products and the running activations and scaling each by its own block parameter rather than reconstructing weights. Format-decoded like Q4_0: no target materializes it, and the GPU path maps it to Q8_0 at load, which ForwardPlanFactory now states by name. Also clarifies in the port proposal that the architecture is qwen35 while the model is Qwen3.8 -- the GGUF's declared architecture is what names the family, and llama.cpp's LLM_ARCH_QWEN35 covers 3.5/3.6/3.8.
Adds the qwen35 architecture: 64 trunk layers of which only every fourth attends, the other 48 mixing with a Gated Delta Net recurrence, plus the MTP block the file carries but the trunk does not execute. Four operations enter the shared vocabulary, because the arithmetic is not this family's -- every recurrent architecture needs it: L2_NORM unit-length scaling, which RMS_NORM is not CAUSAL_CONV_1D depthwise causal convolution over a retained window DELTA_RULE_UPDATE decay, correct, accumulate, read back GATED_NORM rms_norm(x, w) * silu(gate) RoPE gains a partial form: this model states rope.dimension_count = 64 against a 256-wide head, so three quarters of every head is unrotated. Partial rotation is a parameter of RoPE, not a second scheme. Recognition needed no case: the file declares general.architecture = qwen35 and nothing else claims that name, so a provider and a service line were the whole of it. The recurrent state is session state, not KV storage -- fixed size, unpageable, and with no position mask, which is why State gains resetSequenceState(): a delta-net matrix that survives a reset silently conditions the new sequence on the old one. Key/value caches are allocated only for the layers that attend, sparing 48 unused ones. No accelerator claims the architecture. Both the loader and the model refuse a device by name rather than falling back to the host path, which would report GPU throughput for CPU work. Verified on Qwen3.8-27B-Q4_0.gguf: 'What is the capital of France? Answer in one word.' answers Paris.
The delta rule has 48 value heads against 16 key heads. The reference
repeats the key heads with ggml_repeat_4d, which *tiles* -- cycling
0,1,...,15,0,1,... -- and the fused kernel states the same mapping
directly as iv1 % nek1. Dividing instead blocks them 0,0,0,1,1,1,...,
which pairs every value head with the wrong key.
The symptom is the one this defect class always has: short answers stay
correct ('capital of France' still answered Paris) while long output
decays into a repetition loop, because the error compounds through the
recurrence rather than failing outright.
The equivalence test passed through all of this, because its reference
carried the same misreading -- exactly the trap the porting skill names:
an invalid fixture fails both paths identically and looks like
agreement. Both sides are corrected, and the reference now says in a
comment which ordering is right and why, so the next reader cannot
re-derive the wrong one from the divisibility alone.
l2Norm also now floors the divisor at eps rather than adding eps under
the root, which is what ggml_l2_norm does.
Adds the checks that were missing:
- Q4_1FloatTensorTest, against hand-encoded blocks, pinning the
unsigned nibble and the affine reconstruction
- Qwen35ConfigurationTest, over Qwen3.8-27B's real metadata, pinning
the derived delta-net widths and that the head width is stated
rather than derived
- the fixture's SHA-256 in GoldenFixture
Verified: 'Write three sentences about the Roman Empire' now produces a
coherent reasoning block and three correct sentences, and stops.
Qwen3.8-27B carries a NextN block past its 64 trunk layers: a complete attention decoder block whose input is not the previous layer's output but the pair (the trunk's hidden state at position p, the token chosen for p+1), each through its own norm, concatenated and projected back to dim. It predicts the token at p+2. Neither nextn.embed_tokens nor nextn.shared_head_head is present in this file, so both fall back to the trunk's, as llama.cpp does when they are absent. Generation through the head is behind -Dllama.qwen35.speculative, off by default, and the reason is stated rather than hedged: an accepted draft only saves work on a backend that verifies several positions in one forward pass, and this host path verifies them one at a time. What the default-off path does deliver is a measurement -- -Dllama.qwen35.speculative.stats reports how often the head agreed with the trunk, which is the only way to tell a correctly fed draft head from a badly fed one: the trunk's token is what gets emitted either way, so no amount of fluent output would show the difference. The head gets its own logits buffer and its own residual stream. Sharing the trunk's would let a draft overwrite the prediction the loop is about to commit, and would move a stream that a question about the future must leave where it is. Qwen35MtpTest asserts both, plus the block itself against an independently written reference -- the concatenation order and which hidden state is consumed being the two things that change nothing about any shape when they are wrong.
Adding four OperationKind values and a DataType left five gates
failing, each of them correctly:
- Operation is sealed and every kind must have exactly one type, so
L2Norm, CausalConv1d, DeltaRuleUpdate and GatedNorm now exist as
descriptions, not only as host implementations. DeltaRuleUpdate
takes keyHeads rather than a ratio on purpose: a ratio invites the
division that produces the wrong head pairing.
- CausalConv1d and DeltaRuleUpdate declare their state as an output as
well as an input. A description that did not would let a backend
advance a window twice, or not at all.
- OperationSupport lists Q4_1 among the representations the host reads
weights in, and lists the four recurrent kinds as unsupported on the
GPU by name rather than leaving them absent.
- The dependency allowlists gain Qwen35 (TornadoVMMasterPlan in the
generateTokensGPU signature, like every other model), Q4_1FloatTensor
(LlamaApp and GGMLType, like Q4_0FloatTensor) and ModelType$12.
549 tests pass; the shaded jar still carries all seven service files,
with the two new providers in them.
verification.md records what was verified and, at more length, the two
defects that produced fluent output -- the tiled key-head mapping whose
test reference shared its misreading, and why the MTP head's acceptance
rate is measured rather than assumed.
These two files were untracked in the working tree before this branch and were swept in by a git add -A. They are not part of this port; this restores them to untracked so they stay the user's own working files.
standalone-inference.yml asserts a resolved backend and a real execution_path on every row; qwen35 has neither, and the smallest release of it is far larger than any fixture that matrix carries. Saying so is the point -- an omission with no stated cause reads as an oversight, and the porting checklist asks for the rows.
…fits Qwen 3.5 abandoned the JSON body Qwen 3 put inside <tool_call> for nested pseudo-XML, one element per argument: <tool_call><function=get_weather><parameter=location>Boston</parameter> </function></tool_call> Reusing Qwen3ChatFormat meant prompting the model for a format it was not trained on and parsing it for one it does not emit -- wrong in both directions, and invisible, because a model that emits calls we cannot read looks exactly like a model that chose not to call anything. Qwen35ToolCalls does the translation at the boundary; nothing above it sees anything but a name and a JSON object. The format erases types -- the template writes strings verbatim and everything else through tojson, so 'Boston' and '42' occupy the same position -- so they are recovered by shape on the way back. That is a heuristic and the class says so: a tool whose string argument is literally '42' round-trips as a number. Quoting everything instead would break every numeric and boolean parameter, which is far commoner. Separately: a caller who asks for no context length now gets 8192 rather than the 262144 this family declares. Sixteen of its layers attend with a 1024-wide key and value, so the declared maximum is 34 GB of host arrays allocated eagerly at session construction -- the public API's ModelOptions.defaults() died on it with an OutOfMemoryError before generating a token, where the CLI survived only because it passes its own default. Scoped to this loader, and the comment says why: the facade's 'the model's own maximum' default is optimistic for every long-context family, but this is the first model where it cannot run at all, and changing that rule for everyone is not this port's call.
The chat template puts consecutive tool results in one user turn; ConversationEncoder calls encodeToolResultTurn once per result, so they become several. No difference for a single result. Merging them means a batched entry point on the shared encoder, which changes every family -- so it is recorded rather than worked around here.
examples.ToolCalling through the public API: the model emitted this
family's pseudo-XML, the call parsed as get_weather({"city":"Athens"}),
the assistant turn replayed, and the final answer used the tool's data.
The one check a unit test cannot stand in for. A model prompted for the
wrong tool-call format answers in prose, which is indistinguishable from
one that decided a tool was unnecessary -- so the parser being correct
says nothing about whether the model is being asked correctly.
Q4_0 was materialized as Q8_0 at load, taking 4.5 bits per weight to
8.5 and roughly doubling what a model occupies on the device. That is
the same problem retaining Q4_K solved for Devstral, and this is the
same solution for the representation that is far commoner.
Measured on one file, switching only -Dllama.q4_0.retain, so the
comparison isolates the representation rather than the quantization:
retained as Q4_0 1570 MiB 172.4 tok/s
materialized Q8_0 2060 MiB 136.0 tok/s
Faster as well as smaller: single-token decode is bandwidth-bound, so
fewer bytes per weight is fewer bytes read per token.
- Q4_0TornadoTensor wraps the file's bytes shallowly, as its Q4_K and
Q8_0 siblings do; nothing is converted at load.
- TransformerComputeKernelsQ4_0 decodes inside the dot product, shaped
like the Q4_K kernels so a layer differs only in which method
reference it names.
- LlamaQ4_0FFNLayers extends the Q8_0 layer and replaces four tasks.
Its feed-forward block applies the RMS norm as its own task rather
than folding it into the gate/up projection -- one more task per
layer, one fewer kernel to keep correct, which is the trade the Q4_K
path already makes.
- The loader retains Q4_0 only when *every* per-layer weight is Q4_0.
All or nothing, because these layers have no per-tensor dispatch: a
retained tensor in a plan whose kernels read Q8_0 blocks would read
18-byte blocks as 34-byte ones, and produce fluent, wrong text.
- The decision follows the per-layer weights, not the output
projection. A Q4_0 file leaves token_embd as Q6_K, so the output
weight alone would say Q8_0 and select the wrong plan.
Q4_0 has single-token kernels only. TornadoPlanRegistry now refuses the
prefill and batch modes for it by name instead of failing on a cast into
an interface the components do not implement -- a general fix, since any
provider may support a mode for one representation and not another.
Q4_0DecodeTest holds the device decode against the host tensor on random
bytes and against the specification on hand-built ones. Agreement alone
would not be enough: the two could agree and both be a different format.
Verified on CUDA with execution_combination llama/Q4_0/STANDARD, and the
Q8_0 and F16 paths are unchanged.
The earlier note said a qwen35 GPU path was blocked by needing ~28GB against 24GB of VRAM. With Q4_0 retained that becomes roughly 17GB -- this file is mostly Q4_0 -- so memory is no longer the objection it was. The delta-net kernels are, and the retention would still have to be wired into that family's loader. Also records why the Q4_0 measurement is an A/B on one file rather than a comparison between a Q4_0 and a Q8_0 model: the easier measurement would have measured the quantization as well as the residency.
The four kernels the qwen35 mixer needs -- depthwise causal convolution with its rolling window, per-head L2 norm, the gated delta rule, and the gated norm -- plus the small one that turns the raw alpha/beta projections into a decay and a write strength. Two decisions shape the file. Every kernel body is a static method taking an explicit lane index, and the kernel is a two-line wrapper passing context.globalIdx. A body written directly against KernelContext cannot be called on the host, so it can only be exercised by running a model on a device, where an indexing mistake surfaces as slightly wrong text rather than as a failure. Lifting the arithmetic out is what lets the parity test run every lane on the host against CpuOperations. TornadoVM inlines it, so it costs nothing at run time. The delta rule needs no barrier and no cross-lane reduction, which is not obvious for what looks like a matrix-vector product per head. Give a lane one value column of a head's state and every quantity it needs is its own: the decayed column, the prediction for that column, the correction, the rank-one update, the readout. The existing state layout then makes it coalesced as well as correct -- per lane the access is strided, but across the lanes of a head it is contiguous, which is what a GPU is paid for. It is the same layout the host uses, deliberately: two layouts would mean the parity test compares a transpose against a transpose and proves nothing about either. Four of six comparisons are bit-exact, including the delta rule's state and the convolution's window. The other two are equal to float rounding because the host evaluates sqrt, log and the logistic in double; that is stated in the test rather than smoothed over with a blanket tolerance. Establishes the arithmetic and the addressing only. That these compile and run on a device, and that a layer graph binds them correctly, are separate gates and are not met.
The three things that separate a qwen35 attention layer from Qwen3's,
none of them a different algorithm:
- the query projection is twice as wide as the query, carrying an
interleaved output gate, split here into contiguous halves so that
the per-head norm, the rotation and attention can all address
head * headDim as they already do;
- the rotary width is 64 of a 256-wide head, so three quarters of
every head passes through unrotated -- a parameter of the rotation,
not a second scheme;
- the attention result is scaled by the logistic of its gate. A
logistic, not a SiLU: reusing the SwiGLU kernel would multiply in an
extra factor of the gate and stay plausible.
The key/value append is its own kernel rather than fused into the
rotation, which is what Llama and Qwen3 do. They can fuse because they
rotate the whole head, so the lanes that rotate are the lanes that must
be written; here the rotation covers a quarter of each head and the
append covers all of it.
The rotation reads the host's precomputed frequency tables rather than
recomputing them with pow and cos, which is what makes it bit-identical
to the host rather than merely close.
Two of the six checks assert a property directly instead of against the
host, because both guard a wrong reading that is also well-formed: the
whole-buffer query/gate split, and a paged append with the wrong stride
landing inside another layer's slice. A comparison in which both sides
made the same assumption would pass.
Also corrects a claim in the class comment. fusedQKRmsNorm is expected
to serve a 256-wide head unchanged, but it reduces through local memory
and cannot run on the host, so nothing has exercised it at that width.
Saying it is reused would have been stating an untested assumption as a
result.
…recurrent Three things about this family's device memory are unlike any other's, and each of them is a size question rather than a correctness one -- which is why they are asserted rather than left to read correctly. Key/value storage covers a quarter of the blocks. Only the attending layers write to it, so they address it by a dense index and the store is sized by them; sizing it by the block count would cost nearly four times as much, which is gigabytes at any useful context. State gains a fillKvFields overload taking that count -- the kernels need no change, since a layer is already just a stride multiplier to them. The recurrent layers hold state that is neither cache nor scratch. The convolution windows and delta-net matrices persist across tokens and are updated in place, so they live in one array per kind addressed by a per-layer offset; 48 device buffers per kind would be 48 transfers to arrange and keep resident. The delta-net and convolution kernels take that offset now, and the parity tests cover a non-zero one -- a lane that dropped it, or applied it to one of its two passes and not the other, would read one layer's state while writing another's and look fine for a while. That state must start at zero on whichever path is running, and a reset must clear both representations. A key/value cache needs neither: attention reads no further than the current position. A recurrence has no such mask. Whether the device arrays are allocated is decided by the use.tornadovm property, which is the facade's default and not the same question as which backend a session resolved. That is adequate only because no plan provider claims this architecture, so nothing can disagree with it; the comment says what has to replace it and when. Zeroing moved behind TornadoWorkspaces rather than calling init on the arrays from the state: naming a TornadoVM type outside the backend is Rule 1, whose allowlist is empty and stays empty. Also records that the memory preflight is materialization-only. It refuses Qwen3.8-27B at 27760 MiB against a 14 GB budget, which is right today and becomes wrong as soon as qwen35 retains Q4_0 -- the retained figure is roughly 17 GB.
…ined Qwen35TornadoWeights and the loader that fills it. Like the host weights, it implements Weights directly rather than extending TornadoWeights, whose fields assume every layer has a query, key and value projection -- here three in four have none. Retention matters more for this model than for any other. Qwen3.8-27B is 16GB and almost entirely Q4_0; materialized as Q8_0 it costs about 28GB, which does not fit on a 24GB device, where retained it is about 17GB, which does. The rest of the file -- ssm_out in Q5_K, eight ffn_down in Q4_1, output in Q6_K -- has no kernel and is still materialized, so the model is mixed and each weight is read by the kernel matching its type. Retention is all-or-nothing over the projections, not per tensor: the graphs dispatch per tensor, but one layer's q/k/v are read by a single fused kernel and must agree. Mixing block layouts inside one kernel reads 18-byte blocks as 34-byte ones and produces fluent, wrong text. The memory preflight predicted every quantized weight at its Q8_0 size, which was already wrong for Llama's Q4_0 files and would have refused qwen35 at 28GB once it retains. It now asks the plan provider which representations that family reads as they lie -- not a new declaration, just supportedDataTypes, which already answers exactly this. Measured on Llama-3.2-1B-Instruct-Q4_0: per-layer weights 547MB retained against 1034MB materialized, the 34/18 block ratio. The preflight decides per tensor where a loader may decide per model, so a file mixing Q4_0 with another quantization in its layers would be predicted smaller than it loads. No quantizer produces one. The direction is the tolerable one: this prediction refuses loads and a refusal cannot be overruled, so an over-estimate blocks a configuration that would have run where an under-estimate reaches the backend's own error. None of this makes qwen35 runnable on a device. Nothing consumes these weights yet, and the preflight's retention does not apply to a family with no provider -- so the model is still predicted at 27760 MiB, which is the right answer while nothing can build it a plan.
Completes the set of quantized representations the engine can hold on a device in the file's own layout. Q4_0, Q4_K, Q6_K and Q8_0 already had wrappers and kernels; Q4_1 and Q5_K did not, and Qwen3.8-27B needs both -- eight Q4_1 ffn_down tensors and forty-eight Q5_K ssm_out. Q4_1 is 32 weights in 20 bytes, d * q + m with an unsigned nibble. Q5_K is Q4_K with a fifth bit held in a separate 32-byte plane, indexed by position within the pair's half and by which nibble the element came from -- not by the element's index in the super-block. Both kernel files are shaped like their siblings, so a layer differs only in which method reference it names, and each decode lives in a package-private helper the parity test can call on the host. QuantizedDeviceDecodeParityTest covers all six against their CPU tensors on adversarial blocks: each block's scale fields walk the half-precision corners -- both zeros, both ends of the subnormal range, unity of both signs, both extremes of the normal range -- and payloads walk all-zero, all-ones and both alternations, with every fourth block random. Bit equality, not a tolerance. It also asserts its own sensitivity. Four deliberate faults -- the wrong nibble half, Q4_0's recentring applied to Q4_1, Q5_K's fifth bit from the wrong bit, K-quant scales read without the straddled high bits -- must each be seen to disagree with the host. A parity test that could not detect them would prove less than it appears to. Writing it found a fixture bug worth recording rather than a kernel one: Q6_K's scale is at offset 208, after ql, qh and sixteen signed sub-block scales, so injecting halves at offset 0 corrupted ql and left the scale random. Random scales reach Inf and NaN, which TransformerComputeKernelsQ6_K states it does not handle because a quantized block scale is neither -- a constraint of well-formed GGUF, not an oversight, and the alternative is two branches in the innermost loop of every Q6_K weight. The offsets are now passed per format and the divergence is recorded in the test instead of asserted away. Also corrects the port proposal's architecture model: quantized storage is retained on every backend, decoding during compute is an implementation property rather than a CPU-only one, materialization to Q8_0 is not the normal accelerator fallback, support is declared per operation and dtype, fusion stays backend-owned, mixed quantization between tensors is legal, and every operand fused into one kernel must have a layout combination that kernel explicitly supports.
DataType conflated three questions, and the conflation is what made the engine believe quantized weights were CPU-only: 1. must arithmetic decode a block to read a value 2. can a backend store the representation 3. does a given operation have a kernel for it Only the first is a property of a representation, and it is true on every backend -- a device kernel decodes inside its dot product exactly as a host one does. The second belongs to the backend's storage vocabulary; the third to OperationSupport, per operation and per target. 'The GPU cannot do Q5_K' was never a fact about Q5_K. So isFormatDecoded is gone rather than renamed, and materializedFallback becomes narrowedFallback with one case left: BF16 to F16, which is a real loss of mantissa bits for want of BF16 arithmetic rather than a capability gap dressed up as a representation. It no longer answers Q8_0 for anything, which is what used to double a 4-bit model on a device. The Q8_0 promotion still exists for families on the older loading path, but it now lives in DataTypeMapping as legacyDevicePromotion, named and documented as a property of that path rather than of the type. Families with native kernels use ModelLoader.loadTornadoTensorNative, which keeps every representation as the file gave it and refuses one the device cannot store instead of promoting it. OperationSupport now declares what is actually true: matrix-vector and vocabulary projection read all six quantizations on the GPU; matrix-matrix has tensor-core kernels for F16 and Q8_0 only; the device embedding gather covers F16 and Q8_0, other representations being gathered on the host. FusedOperandSupport is the safety net the mixed model needs. Mixed quantization between tensors is legal, but a fused kernel that decodes its operands with one block layout must reject a mixture -- giving it a Q5_K operand where it expects Q4_0 reads 176-byte super-blocks as ten Q4_0 blocks and produces fluent, wrong text. Refused at plan construction with the operands and their representations named. Five tests asserted the retired premise and are replaced by two that state the new one. One further test was removed by an over-eager edit and is restored, reworded.
A model is not one representation. Qwen3.8-27B holds Q4_0 projections and token embeddings, eight Q4_1 ffn_down, forty-eight Q5_K ssm_out, a Q6_K vocabulary projection, a Q8_0 MTP projection and F32 norms, SSM parameters and convolution kernels. A prediction derived from one model-wide dtype is wrong for every tensor that is not that dtype, and here the error runs to twelve gigabytes. DeviceRetention is asked per tensor, by name as well as representation. The name matters because support can differ by role: a family may have a matrix-vector kernel for a representation and no vocabulary-projection kernel for it, and only the name distinguishes them. Most policies will not need it; it is there so the ones that do are expressible rather than approximated. Measured on the fixture: retained per-layer 13.306 GiB global 1.637 GiB total 14.944 GiB converted per-layer 24.590 GiB global 2.516 GiB total 27.107 GiB The retained figure is the file's own weight bytes, arrived at independently from the GGUF tensor inventory rather than recorded from a run. Qwen35WeightFootprintTest validates each representation separately: retaining exactly one dtype at a time isolates its contribution, and the saving must be a whole number of that format's blocks times the difference from Q8_0's block size. A predictor with one format's block size wrong would pass a whole-model check and fail this. Q8_0 and F32 are asserted unchanged either way.
QuantizedDeviceDecodeParityTest settles the arithmetic on the host. It
cannot settle whether a kernel compiles, or whether its row addressing
survives a launch, and both have bitten this backend before.
NativeQuantizedKernelAccelTest compiles and executes Q4_0, Q4_1, Q4_K,
Q5_K and Q6_K matrix-vector kernels on CUDA and compares each against
its CPU tensor over the same bytes, on a rectangular shape (512x37) that
would expose a row/column transposition.
Four passed immediately. Q5_K did not, and finding out why took seven
formulations:
element-strided loop, variable shift deopt scaffolding the CUDA
backend cannot declare --
identifier 'context' is
undefined, identifier 'slots'
is undefined
scale/min extracted to a helper same
mask + conditional move assertion inside
CUDALIRGenerator.emitIntegerTestMove
mask + if branch deopt scaffolding
index-derived branch decomposition deopt scaffolding
shuffle reduction, no local memory deopt scaffolding -- so 'slots'
was the deopt frame, never the
local array
sub-block-wise nested loop deopt scaffolding
The decode compiled and ran correctly outside a loop throughout, which
is what narrowed it: the problem was ByteArray.getHalfFloat inlined into
a loop. TransformerComputeKernelsQ6_K already documents that call as one
TornadoVM's sketcher chokes on, and assembles its half from two byte
loads instead. Q5_K now does the same and compiles.
Two of those attempts are kept because they are better code regardless:
the scale/min helper, and walking whole 32-element sub-blocks so a
sub-block's scale, minimum, nibble plane and fifth-bit position are
computed once rather than thirty-two times.
decode() uses the same byte-assembled half as the kernel. It had been
left on getHalfFloat, which would have meant the parity test covering
arithmetic the device never runs.
The host parity tests run every lane on the CPU. That settles the arithmetic and says nothing about whether TornadoVM can compile the kernel -- a distinction that is not academic, since Q5_K's matvec passed its host parity test and then failed to compile in seven formulations. These kernels use nested loops, a retained state array written in place, and TornadoMath transcendentals, none of which the host tests exercise as device code. All five groups compile and run on CUDA and match the host operations: causal convolution, and the window it advances the delta rule, and the 48 x 128 x 128 state it updates in place gated norm and per-head L2 norm decay and write strength, through exp/log/logistic attention: query/gate split, partial RoPE, output gate Dimensions are the 27B's own where it matters -- a 128-wide delta-net head, 48 value heads against 16 key heads, a 256-wide attention head with a 64-wide rotary.
…sor dtype Two shared paths still assumed a model-wide representation, which is wrong for every mixed file and blocks a Qwen3.5 device plan. The embedding lookup staged and converted by the model-wide quantization string. Qwen3.5 reports Q8_0 model-wide while holding its token embeddings as retained Q4_0, so 18-byte blocks would have been staged and read as 34-byte ones -- a plausible activation and wrong output. Activation now dispatches on the embedding tensor's own dataType, which is what TornadoForwardPass already did for staging; this is the other half. convertQ4_0toFP32 is the kernel it needed, assembling its half from two byte loads rather than getHalfFloat, which TornadoVM's sketcher rejects inside a kernel. The vocabulary projection was hard-wired to the Q8_0 kernel. It now selects by what the output projection actually holds -- Qwen3.5's is Q6_K where its layers are Q4_0 -- across all six quantizations, and refuses an unsupported one by name rather than converting it, which would double what it occupies and hide a missing kernel behind a memory cost. Qwen35TornadoWeights now extends TornadoWeights, reversing an earlier choice. Implementing Weights directly kept the base class's per-layer arrays honest, but Activation, AbstractLogitsTaskGraph and TornadoForwardPass are all written against TornadoWeights, and staying outside it meant changing each of them for one family. Fitting the shape costs a documented convention -- per-layer arrays indexed by absolute block, null where the block is of the other kind, exactly as the host weights already do -- and the class says so. 587 tests pass; Llama Q4_0/Q8_0 and Qwen3-0.6B unchanged on CUDA.
…ssion supportedDataTypes answers whether a provider can build a plan for a model whose weights report one representation. A mixed model reports one and holds several, so the memory preflight cannot read that set as what each tensor occupies on the device without under- or over-predicting every tensor whose representation is not the model's. nativeTensorTypes is that second declaration, defaulting to the first so a family whose model is uniform states it once. The preflight reads it. The qwen35 loader now retains every tensor in the file's own representation rather than materializing all but Q4_0 as Q8_0, and reports the projections' shared representation as the model-wide one, failing by name when they disagree instead of picking a representative tensor.
A delta-net mixer splits its convolved projection into q | k | v of unequal widths and scales the queries before the recurrence. splitQKV assumes a key and a value of equal width, which is true of attention and false here: on the 27B the widths are 2048 | 2048 | 6144, and an equal-halves split takes the value slice from inside the keys. Both new kernels are stated per operation rather than per family, and both carry a lane helper so their addressing is checked on the host as well as on the device. scaleInPlace moves out of Gemma4Kernels for the same reason: the arithmetic belongs to the operation, and a second copy named after another family is how one kernel becomes two. Adds device coverage for the F32 matrix-vector kernel at an ssm_alpha shape and for the fused Q4_0 gate/up SwiGLU, neither of which had been launched.
A graph per trunk layer: 20 tasks for a recurrent block, 17 for an attention block, each following the host branch operation for operation. Both kinds run the same dense SwiGLU feed-forward and the same two normalizations, and only the mixer differs. Every task that reads a weight is bound to a kernel chosen from that tensor's own representation at construction, so the compiler sees fixed block addressing rather than a dtype switch inside a K-loop. The one task reading two weights states that they must agree and refuses a mixture by name. A block whose weights do not match the kind the metadata declares fails naming the layer, the role and what is missing, rather than dereferencing null inside graph construction. Key/value storage is addressed by a dense index because the store is sized by the layers that attend; the convolution window and delta-net state are per-layer slices of one array each, uploaded once and updated in place. The topology test builds the smallest model with both layer kinds and pins the graph count, the task counts, the mixer selection, the absence of the draft head, and that changing one tensor's representation changes exactly one task's dispatch.
Adds the plan components, the provider and its service entry, and points the model's GPU generation at the shared single-token loop. The provider declares STANDARD only, and Q4_0 admission — the representation this family's trunk projections share — while declaring separately the eight representations its tasks decode per tensor, which is what the memory preflight needs to predict 14.944 GiB rather than 27. A quantized row must be a whole number of blocks, and the dispatch now says so by name: every block-decoding kernel addresses a row by its block offset, so a partial row would silently read the next row's blocks. The synthetic parity test runs a whole forward pass on the device against the same one on the host, over the same weight bytes, on a model with both layer kinds and the same mixture of representations as the real file. Two positions, because the first says nothing about whether the recurrence carried.
A session's device arrays were gated on use.tornadovm, which the CLI sets and nothing else does. A caller that loaded device weights without it got a state whose device arrays were all null, and the failure arrived from inside TornadoVM as "null object passed into streamIn()" in the activation graph rather than anywhere that named the cause. The model decides now, from whether its weights are device weights, handed to the state for one construction. Attention uses the single-workgroup online-softmax kernel rather than the split-KV one: that kernel fixes its query staging at 128 floats per head and this family's head is 256 wide, which reads past the array and faults with an illegal address that surfaces as an unrelated allocation failure. The load message reports the representation the weights were actually built in, rather than predicting one from the output tensor's type before loading. Adds the real-fixture parity gate. Qwen3.8-27B against the CPU reference on CUDA, teacher-forced over 63 rows of 248320 logits: no elementwise violations against the tightest bounds in the suite, worst ratio 0.074, argmax agreement 63/63, cosine 1.000000.
…mily holds The memory model sized the key/value cache by the layer count, which is right for a stack that attends throughout and four times too large for one that attends in a layer of four. It also had no term for state that is neither cache nor scratch: a recurrent layer keeps its history in a fixed-size convolution window and delta-net matrix, updated in place and independent of the context length — 149.6 MiB on Qwen3.8-27B, previously counted nowhere. Both are asked of the configuration, which is where the answer differs by family, and both default to what an attention-only stack would say. Measured on the real fixture by bisecting the device budget: 15455 MiB succeeds and 15450 MiB fails, against a prediction of 15453.6 MiB. Adds two gates: the retained representation of every role at runtime, and the public route on the device through generate, reset, close and use-after-close, asserting the resolved execution path so a silent host fallback cannot pass.
The measured baseline on the real fixture — 15.0 tok/s against 0.77 on the CPU, where the time goes per kernel, and the memory prediction against the measured minimum budget — plus the two shape facts the port established: a 256-wide head does not fit the split-KV kernel's fixed local arrays, and a quantized row must be a whole number of blocks. Also records an optimization that was tried and rejected. Walking 32-element groups in the Q6_K projection hoists an fp16 scale out of the inner loop and is slower, because the element-strided loop it replaced had consecutive lanes reading consecutive bytes. Coalescing beat the arithmetic saving, and reverting reproduced the original timing to within 0.05%.
The split-KV attention's combine pass went with the split-KV kernel; the count and the topology table still named it.
What a recurrence costs during ingestion, why a chunk cannot reorder tokens inside a recurrent layer, and where the scan lives: inside the kernel, one lane per channel or per value column, walking the chunk in order.
PREFILL_DECODE for this family is the single-token layer graphs with the logits graph skipped for prompt positions: the recurrence it advances is the same device buffer decode continues from, and the only thing that differs is the graph layer 0 consumes its activation from. The layer builder takes that name; nothing else about the computation changes. Sequential prefill staged the token embedding by the model-wide weight type and knew only F16 and Q8_0, so a retained Q4_0 embedding was a hard failure before it was a wrong one. It dispatches on the embedding tensor now, as the decode step already did. The prefill loop produced one row more than STANDARD, for both this family and Qwen3 — measured on Qwen3-0.6B as 63 rows against 64. The two decode loops disagree about what a prompt costs: one charges the positions it feeds, the other charges promptTokens.size() and stops. The rule now travels with the caller instead of being guessed, so a prompt yields the same number of tokens however it was scheduled, and no other family's count moves. Real fixture, teacher-forced against the CPU at the single-token bounds: 63 rows, no elementwise violations, argmax 63/63.
…n order A chunk of prompt tokens per invocation, with its own layer graphs. Most of a layer batches by adding a row index; the convolution and the delta rule do not, because token t reads what token t-1 wrote inside the layer. Those two scan the chunk inside the kernel — one lane per convolution channel, one per delta-net value column — so the sequential dependency stays inside a lane and needs no barrier and no ordering between lanes. That is what makes the chunk width unobservable: the arithmetic per token is the same expression in the same order the single-token kernel performs, and the width only decides how many iterations a lane runs. The scan tests hold both against the single-token kernels exactly, at one, two, three, seven and eight rows, comparing the retained state as well as the outputs. Attention batches as a causal chunk: each row rotates at its own position, appends its key and value exactly once, and reads no further than itself. Its local arrays are sized from the head width rather than fixed at 128, so a 256-wide head fits. The decode graphs consume the weights and the recurrent state the prefill graphs uploaded rather than binding their own — a plan would otherwise hold the model twice, which is the duplication batched prefill was measured to cost before. Verified on the synthetic model at real head geometry: widths 1, 2, 7, 8 and wider than the prompt against the CPU, the batched widths against each other, prompts on and either side of a chunk boundary, and each mixer alone.
A layout says how many families of per-layer graphs a mode builds, and the memory model multiplied the weights by that. It is the right multiplier only for a family whose graphs each upload their own copy; qwen35's batched decode graphs consume what the prefill graphs uploaded, so the model is on the device once. At 14.944 GiB of weights the difference decides whether the 27B is refused on a device it runs on — and this prediction refuses loads, so an over-estimate is not the safe direction here. Also counts the chunk-wide scratch this family allocates beyond the generic batch staging: a projection twice a query's width, a convolved q|k|v of unequal parts, and the delta-net's own inputs. Real fixture, batched prefill against the CPU at the single-token bounds, at widths 2, 7, 32 and 64: 63 rows each, no elementwise violations, argmax 63/63, and the same numbers to the last digit at every width — the chunk is a scheduling unit and is not observable in the result.
The batched projections read the weight matrix once per row, which is exactly what running the rows separately reads — measured as no prompt-evaluation gain at all over STANDARD. A quantized projection is memory-bound, so what a chunk is worth is weight reuse. The Q4_0 projections now cover eight rows per workgroup and the fused gate/up four, decoding each block once for the tile. Per row the arithmetic is unchanged — same lane-strided order over the same values — so the tiled and untiled kernels agree element for element and the chunk width stays unobservable. Measured on Qwen3.8-27B, 381 prompt tokens, paired and repeated: prompt evaluation 11.97 -> 26.0 tok/s (2.17x), prefill 31.8 s -> 14.7 s, decode unchanged at 10.2 against 10.6. The projections alone gave 1.69x; the fused gate/up is the rest. The grid for a tiled task is keyed per layer, not per task name: ffn_down is Q4_1 on the first eight blocks and Q4_0 on the rest, so the same name is tiled in one layer and not in another.
Parity in all three modes at the single-token bounds, the same numbers at every chunk width, prompt evaluation at 2.17x with where that came from, and predicted against measured device budgets.
Qwen3 generated one token more when the prompt was ingested as its own phase — 64 logits rows against the reference's 63 — because its decode loop charges promptTokens.size() and the prefill loop charged the positions it fed. The mismatch is the one execution.md records as unfixed; the rule now travels with the caller, so this family asks for the same budget its own decode loop uses. Qwen3-0.6B F16 batched prefill passes CPU/GPU parity outright as a result: it was failing on the row count alone. Q8_0 still violates the bounds by 0.29% of elements at 11.6x tolerance, which is the separate and documented FP16 tensor-core accumulation gap, untouched here. Llama is unaffected — its decode loop is the one that charges positions.
Weight reuse is what makes a chunk worth scheduling; a grid keyed on a task name breaks when one name maps to two representations; a prefill budget is the caller's rule; and a recurrence has to be scanned inside the kernel.
The benchmark's model columns described the run loosely enough that a matched comparison against llama.cpp's llama-bench could not be made from them: `quant` came from `Configuration.quantization()`, which names the *activation* class (a Q4_0 file reports "Q8_0"), `params` was a file-size estimate that assumed a single weight type, and nothing recorded the execution mode the plan was built in. Read the architecture, the file's `general.file_type` and the exact parameter count from the GGUF header instead, take the mode off the plan object rather than off the request, and print both. `--expect arch/quant/mode` turns that into an assertion, and a failed model now leaves a non-zero exit status so a sweep script cannot record a missing row as a passing one. Also reports the median alongside the mean, since a five-repetition comparison is quoted on the median.
mikepapadim
marked this pull request as ready for review
September 9, 2026 09:28
mikepapadim
self-requested a review
September 9, 2026 09:29
Contributor
There was a problem hiding this comment.
🔵 Needs a closer look
It introduces a large new model family plus substantial backend/memory/kernels changes, and at least one correctness issue was found in newly added device-side Q4_0 scale decoding.
Pull request overview
Adds end-to-end support for the qwen35 architecture (Qwen 3.5/3.6/3.8 hybrid attention + delta-net stack), including CPU execution and TornadoVM GPU execution with native (non-materialized) retention of multiple quantized tensor layouts per role.
Changes:
- Introduces the
qwen35model family (provider/loader/type wiring), tokenizer + chat/tool-call format, and TornadoVM plan provider/components for all three execution modes. - Enables native GPU retention/dispatch for additional quantized layouts (notably Q4_0/Q4_1/Q5_K/Q6_K) and updates memory preflight/planning to account for per-tensor retention, sparse KV (qwen35), and recurrent state.
- Adds broad correctness coverage (unit + kernel + synthetic + golden + lifecycle tests), including parity and footprint assertions for mixed-quantization fixtures.
File summaries
| File | Description |
|---|---|
| src/test/java/org/beehive/gpullama3/tensor/Q4_1FloatTensorTest.java | Adds host Q4_1 decode + dot-product parity tests. |
| src/test/java/org/beehive/gpullama3/runtime/tensor/DataTypeTest.java | Updates DataType expectations around narrowing behavior. |
| src/test/java/org/beehive/gpullama3/program/op/OperationVocabularyTest.java | Extends operation round-trip vocabulary to new ops. |
| src/test/java/org/beehive/gpullama3/program/op/OperationSupportTest.java | Revises GPU support assertions for quantized layouts per op. |
| src/test/java/org/beehive/gpullama3/model/loader/Qwen35ConfigurationTest.java | Validates derived qwen35 geometry against real metadata. |
| src/test/java/org/beehive/gpullama3/model/format/Qwen35ToolCallsTest.java | Tests qwen35 tool-call wire format parse/render. |
| src/test/java/org/beehive/gpullama3/inference/state/Qwen35StateTest.java | Tests qwen35 state allocation/reset semantics on host vs device. |
| src/test/java/org/beehive/gpullama3/golden/RetainedWeightFootprintTest.java | Asserts retained-vs-converted footprint prediction for Q4_0 fixture. |
| src/test/java/org/beehive/gpullama3/golden/Qwen35WeightFootprintTest.java | Validates per-dtype footprint accounting on mixed-quant qwen35 fixture. |
| src/test/java/org/beehive/gpullama3/golden/Qwen35PrefillDecodeParityAccelTest.java | Golden parity for sequential prefill/decode on accelerator. |
| src/test/java/org/beehive/gpullama3/golden/Qwen35NativeRepresentationAccelTest.java | Asserts actual device tensor layouts after load by role. |
| src/test/java/org/beehive/gpullama3/golden/Qwen35CpuGpuParityAccelTest.java | Golden CPU↔GPU logits parity for qwen35 mixed-quant fixture. |
| src/test/java/org/beehive/gpullama3/golden/Qwen35BatchedPrefillWidth7ParityAccelTest.java | Golden parity for batched prefill at width 7. |
| src/test/java/org/beehive/gpullama3/golden/Qwen35BatchedPrefillWidth64ParityAccelTest.java | Golden parity for batched prefill at width 64. |
| src/test/java/org/beehive/gpullama3/golden/Qwen35BatchedPrefillWidth2ParityAccelTest.java | Golden parity for batched prefill at width 2. |
| src/test/java/org/beehive/gpullama3/golden/Qwen35BatchedPrefillParityAccelTest.java | Golden parity for batched prefill at default width. |
| src/test/java/org/beehive/gpullama3/golden/GoldenFixture.java | Adds new fixtures and metadata docs for qwen35 and retained Q4_0 llama. |
| src/test/java/org/beehive/gpullama3/golden/GoldenCapture.java | Makes prefill phase selection explicit in capture harness. |
| src/test/java/org/beehive/gpullama3/golden/CpuGpuParity.java | Adds sequential prefill/decode parity path and improves reporting. |
| src/test/java/org/beehive/gpullama3/format/DataTypeMappingTest.java | Reframes tests around “converting device path” vs native retention. |
| src/test/java/org/beehive/gpullama3/backend/tornado/Qwen35SyntheticParityAccelTest.java | Synthetic host↔device forward-pass parity on generated weights/bytes. |
| src/test/java/org/beehive/gpullama3/backend/tornado/plan/TornadoPlanRegistryTest.java | Adds qwen35 registration + native per-tensor retention assertions. |
| src/test/java/org/beehive/gpullama3/backend/tornado/kernels/SharedComputeKernelParityTest.java | Unit-parity tests for shared kernels needed by recurrence/mixers. |
| src/test/java/org/beehive/gpullama3/backend/tornado/kernels/Q4_0DecodeTest.java | Device-vs-host Q4_0 decode parity and spec-layout assertions. |
| src/test/java/org/beehive/gpullama3/arch/Allowlists.java | Updates architectural allowlists for new types/providers/tensors. |
| src/test/java/org/beehive/gpullama3/api/Qwen35LifecycleAccelTest.java | Public API lifecycle test for qwen35 on accelerator (STANDARD). |
| src/test/java/org/beehive/gpullama3/api/Qwen35BatchedLifecycleAccelTest.java | Public API lifecycle test for qwen35 on accelerator (BATCH_PREFILL_DECODE). |
| src/main/resources/META-INF/services/org.beehive.gpullama3.model.provider.ModelProvider | Registers qwen35 model provider via ServiceLoader. |
| src/main/resources/META-INF/services/org.beehive.gpullama3.backend.tornado.plan.TornadoPlanProvider | Registers qwen35 Tornado plan provider. |
| src/main/resources/META-INF/services/org.beehive.gpullama3.backend.cpu.CpuForwardProvider | Registers qwen35 CPU forward provider. |
| src/main/java/org/beehive/gpullama3/tokenizer/Qwen3Tokenizer.java | Makes split-pattern configurable for derived tokenizers. |
| src/main/java/org/beehive/gpullama3/tokenizer/Qwen35Tokenizer.java | Adds qwen35 pre-tokenizer split regex variant. |
| src/main/java/org/beehive/gpullama3/runtime/memory/DeviceRetention.java | Introduces per-tensor device retention policy abstraction. |
| src/main/java/org/beehive/gpullama3/program/op/OperationSupport.java | Expands support matrices for Q4_1 + new ops and clarifies GPU sets. |
| src/main/java/org/beehive/gpullama3/program/op/OperationKind.java | Adds kinds for L2 norm, causal conv, delta rule, gated norm. |
| src/main/java/org/beehive/gpullama3/program/op/Operation.java | Extends sealed operation set with new op records. |
| src/main/java/org/beehive/gpullama3/program/op/L2Norm.java | Adds op description for per-group L2 normalization. |
| src/main/java/org/beehive/gpullama3/program/op/GatedNorm.java | Adds fused RMSNorm + SiLU gate op description. |
| src/main/java/org/beehive/gpullama3/program/op/DeltaRuleUpdate.java | Adds delta-rule recurrence op description with state update semantics. |
| src/main/java/org/beehive/gpullama3/program/op/CausalConv1d.java | Adds depthwise causal convolution op description with window state. |
| src/main/java/org/beehive/gpullama3/model/qwen35/Qwen35.java | Introduces qwen35 model implementation and GPU generation behavior. |
| src/main/java/org/beehive/gpullama3/model/qwen3/Qwen3.java | Aligns prefill/decode GPU generation with shared prefill loop. |
| src/main/java/org/beehive/gpullama3/model/provider/Qwen35Provider.java | Adds provider for general.architecture=qwen35 files. |
| src/main/java/org/beehive/gpullama3/model/ModelType.java | Adds QWEN_3_5 type and loader routing. |
| src/main/java/org/beehive/gpullama3/model/loader/LlamaModelLoader.java | Adds retained-Q4_0 path selection and loading helpers. |
| src/main/java/org/beehive/gpullama3/model/loader/AbstractModelLoader.java | Improves logging to reflect actual loaded weight dtype. |
| src/main/java/org/beehive/gpullama3/model/format/Qwen35ChatFormat.java | Adds qwen35 tool-call encoding/extraction semantics. |
| src/main/java/org/beehive/gpullama3/model/Configuration.java | Adds default hooks for sparse KV layers, weight binding families, recurrent state, and batch workspace. |
| src/main/java/org/beehive/gpullama3/inference/weights/tornado/Qwen35TornadoWeights.java | Introduces device weights container for qwen35 mixed tensors. |
| src/main/java/org/beehive/gpullama3/inference/state/State.java | Adds prefill batch width capture, sparse-KV sizing overload, and reset hook. |
| src/main/java/org/beehive/gpullama3/format/DataTypeMapping.java | Adds Q4_1 mapping and documents legacy converting device promotion. |
| src/main/java/org/beehive/gpullama3/backend/tornado/workspace/TornadoWorkspaces.java | Adds helpers for qwen35 recurrent buffers (allocate/zero). |
| src/main/java/org/beehive/gpullama3/backend/tornado/workspace/TornadoWorkspace.java | Adds qwen35 recurrent + batched workspace buffers to workspace. |
| src/main/java/org/beehive/gpullama3/backend/tornado/TornadoPrefillPass.java | Stages embeddings by embedding tensor dtype (supports retained Q4_0). |
| src/main/java/org/beehive/gpullama3/backend/tornado/TornadoForwardPass.java | Stages embeddings by embedding tensor dtype (supports retained Q4_0). |
| src/main/java/org/beehive/gpullama3/backend/tornado/TornadoBatchPrefillPass.java | Adds retained-Q4_0 embedding decode for batched prefill and decode staging. |
| src/main/java/org/beehive/gpullama3/backend/tornado/tensor/Q5_KTornadoTensor.java | Adds retained Q5_K tensor wrapper. |
| src/main/java/org/beehive/gpullama3/backend/tornado/tensor/Q4_1TornadoTensor.java | Adds retained Q4_1 tensor wrapper. |
| src/main/java/org/beehive/gpullama3/backend/tornado/tensor/Q4_0TornadoTensor.java | Adds retained Q4_0 tensor wrapper. |
| src/main/java/org/beehive/gpullama3/backend/tornado/plan/TornadoPlanRegistry.java | Adds native per-tensor retention query and better unsupported-mode errors. |
| src/main/java/org/beehive/gpullama3/backend/tornado/plan/TornadoPlanProvider.java | Separates admission dtypes from per-tensor native tensor types. |
| src/main/java/org/beehive/gpullama3/backend/tornado/plan/Qwen35PlanProvider.java | Adds qwen35 plan provider with mixed per-tensor native types. |
| src/main/java/org/beehive/gpullama3/backend/tornado/plan/LlamaPlanProvider.java | Extends llama provider to include retained Q4_0 plan selection. |
| src/main/java/org/beehive/gpullama3/backend/tornado/plan/FusedOperandSupport.java | Adds enforcement for fused kernel operand dtype uniformity. |
| src/main/java/org/beehive/gpullama3/backend/tornado/plan/ForwardPlanFactory.java | Allows Q4_0 plan quantization while disallowing non-materialized plan dtypes. |
| src/main/java/org/beehive/gpullama3/backend/tornado/plan/components/Qwen35PlanComponents.java | Implements qwen35 plan components for all three execution modes. |
| src/main/java/org/beehive/gpullama3/backend/tornado/plan/components/q4_0/LlamaQ4_0PlanComponents.java | Adds llama single-token plan components for retained Q4_0 layers. |
| src/main/java/org/beehive/gpullama3/backend/tornado/memory/TornadoMemoryModel.java | Updates memory prediction for sparse KV, recurrent state, and weight-binding families. |
| src/main/java/org/beehive/gpullama3/backend/tornado/layers/type/q8_0/LogitsQ8_0Layer.java | Dispatches vocab projection kernel by tensor dtype (supports multiple quants). |
| src/main/java/org/beehive/gpullama3/backend/tornado/layers/type/q8_0/LlamaQ8_0FFNLayers.java | Makes attention configuration reusable for Q4_0 sibling. |
| src/main/java/org/beehive/gpullama3/backend/tornado/layers/type/q8_0/Gemma4Q8_0FFNLayers.java | Reuses shared scale kernel for embedding scaling. |
| src/main/java/org/beehive/gpullama3/backend/tornado/layers/type/fp16/Gemma4FP16FFNLayers.java | Reuses shared scale kernel for embedding scaling. |
| src/main/java/org/beehive/gpullama3/backend/tornado/layers/Qwen35FFNLayersBatchDecode.java | Adds batch-decode layers that consume caches/state produced by batch prefill. |
| src/main/java/org/beehive/gpullama3/backend/tornado/layers/Qwen35BatchDecodeActivation.java | Adds batch-decode activation graph relaying caches + recurrent state and converting embeddings. |
| src/main/java/org/beehive/gpullama3/backend/tornado/layers/Activation.java | Stages Q4_0 embeddings into FP32 activation via Q4_0 conversion kernel. |
| src/main/java/org/beehive/gpullama3/backend/tornado/kernels/TransformerComputeKernels.java | Adds shared kernels (scale/split) and Q4_0→FP32 conversion. |
| src/main/java/org/beehive/gpullama3/backend/tornado/kernels/TransformerBatchPrefillKernels.java | Adds F32 batched mat-vec kernel for SSM projections. |
| src/main/java/org/beehive/gpullama3/backend/tornado/kernels/Gemma4Kernels.java | Removes duplicate scale kernel in favor of shared compute kernel. |
| src/main/java/org/beehive/gpullama3/backend/cpu/Qwen35CpuForwardProvider.java | Registers CPU forward-pass provider for qwen35. |
| src/main/java/org/beehive/gpullama3/api/MemoryPreflight.java | Makes preflight consult per-architecture native device tensor types. |
| src/main/java/org/beehive/gpullama3/api/LegacySessionRuntime.java | Ensures session reset clears any recurrent state. |
| docs/architecture/models-and-backends.md | Documents qwen35 family, retention semantics, and updated capability matrix. |
| docs/architecture/execution.md | Updates execution-mode accuracy notes and documents qwen35 batch semantics. |
| docs/architecture/architecture-diagram-codex.dot | Adds updated architecture diagram source. |
| .claude/skills/port-model-to-gpullama/SKILL.md | Extends porting skill guidance with new lessons learned (qwen35/batching/retention). |
Review details
- Files reviewed: 121/122 changed files
- Comments generated: 1
- Review effort level: Lite
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Comment on lines
+198
to
+213
| int mantissa = h & 0x3FF; | ||
| int exponent = (h >>> 10) & 0x1F; | ||
| float magnitude; | ||
| if (exponent == 0) { | ||
| magnitude = mantissa * 5.9604645E-8f; | ||
| } else { | ||
| float scaled = 1.0f + mantissa * 9.765625E-4f; | ||
| int shift = exponent - 15; | ||
| float power = 1.0f; | ||
| if (shift > 0) { | ||
| power = (float) (1 << shift); | ||
| } else if (shift < 0) { | ||
| power = 1.0f / (float) (1 << (-shift)); | ||
| } | ||
| magnitude = scaled * power; | ||
| } |
Both engines over the same five llama-bench cases, the same model file and checksum, full offload, five measured repetitions after a warm-up, on the same device — and a batch-width sweep on each side rather than one chosen GPULlama configuration against llama.cpp's default. The result is that prompt processing is 0.015x llama.cpp's at each side's best width (22.96 against 1511 t/s at pp381) and generation is 0.31x (13.20 against 42.71). The width sweep says where that comes from: GPULlama saturates at a batch of 16, while llama.cpp keeps scaling to 512 because above `MMVQ_MAX_BATCH_SIZE` (8) it stops running a matrix-vector kernel and runs int8-MMA MMQ instead. llama.cpp's *matrix-vector* configuration, ub 8, is still 8x GPULlama's best — so the gap opens before tensor cores are involved. Raw rows for every case are kept alongside the record; the directory is force-tracked the way the existing baseline directory is. `llama-tornado` printed its resolved-backend line on stdout, which made `--bench -o json` output unparseable. It goes to stderr now.
Nsight Systems over the baseline's pp381 b32 case. The GPU is busy 89.3% of the profile and launch overhead is 0.3% of kernel time, so this is kernel-bound and the ~1000 launches per chunk are not the thing to fix. Quantized projections are 89.4% of prompt processing. The Gated Delta Net scan is 1.6% and the convolution scan 0.05% — the recurrence is not what limits batched prefill today, which reorders the plan: warp-cooperative delta-net execution is worth at most 1.6% here and should be judged on decode instead. Two different limits among the projections. `ssm_out` (Q5_K) and the early `ffn_down` (Q4_1) are untiled and decode each weight 32 times per chunk of 32. The tiled Q4_0 kernels are limited by activation traffic instead: a workgroup reads one output row of weights and ROW_TILE whole rows of activations, and both the tiled and untiled kernels run at the same ~2.3 TB/s aggregate rate, differing only in bytes asked for. Nsight Compute counters are unavailable on this host (ERR_NVGPUCTRPERM), so every bandwidth figure is bytes-moved over measured kernel time rather than a hardware counter, and that is said in the record.
The prefill profile put 25.6% of prompt processing in `ssm_out` (Q5_K) and 3.3% in the early `ffn_down` (Q4_1). Both were the untiled batched matrix-vector kernel — one workgroup per (prompt row, output row) — so a chunk of 32 read and decoded every weight 32 times, which is exactly what running the rows separately reads. They now cover eight prompt rows per workgroup and decode each weight once for the tile. Q4_1 is Q4_0's tiled kernel with Q4_1's decode, and it pays: pp381 b32 goes 22.92 -> 24.89 t/s. Q5_K needed a different lane mapping, and the first version was a 20% regression. The untiled K-quant kernel gives each lane a whole 32-element sub-block, so a lane reads 32 contiguous activations no other lane in the warp reads. That is affordable against one activation row and is eight times the uncoalesced traffic against a tile of eight — decoding each weight a quarter as often bought less than the lost coalescing cost. Assigning a warp to a sub-block and a lane to one element of it makes the tile's activation reads contiguous across the warp, at the cost of all 32 lanes recomputing a three-byte sub-block header. With that, pp381 b32 reaches 28.38 t/s. Together: 22.92 -> 28.38 t/s, +23.8%, with tg unchanged. The tile width now travels with the kernel that was selected rather than being read off Q4_0's constant for every tiled task, so a tile size can only be changed in one place. The untiled batch forms of both representations are deleted rather than kept as unreachable siblings; the new accel test holds the tiled kernels against the host tensor over the same bytes, across widths on both sides of the tile boundary, and checks that rows past the active count are left untouched. Verified: the new kernel test, and all 27 qwen35 accel tests including teacher-forced 27B parity in all three execution modes at widths 2, 7, 32 and 64, at unchanged bounds.
orionpapadakis
force-pushed
the
feat/qwen3-8
branch
from
September 9, 2026 11:31
cb37592 to
4144244
Compare
Tiling the prompt-row axis answers "how many times is a weight decoded". It leaves the larger number alone. A workgroup covering one output row of `ffn_down` reads 8,704 bytes of weights and 557 KB of activations, and the profile bore that out: after the Q4_1 and Q5_K tiles landed, the tiled and untiled kernels were still running at the same ~2.3 TB/s aggregate rate and differed only in bytes asked for. So a workgroup now covers a tile of output rows as well, staging the activation tile once and reusing it across them. Weight traffic is unchanged; activation traffic per output row is divided by the tile. Measured at pp381, chunk of 32, each figure a median of five: projections, output tile 1 -> 2 -> 4 -> 8 28.4 -> 35.9 -> 38.9 -> 38.8 fused gate/up, rows 4 cols 1 -> 4x2 -> 8x2 38.9 -> 46.4 -> 48.6 fused gate/up at 4x4 and 8x4 41.4 and 44.4 projections at a row tile of 16 48.3 Settled at four output rows for the projections and two for the fused gate/up, whose row tile also moves from four to eight. Past those the accumulators and the wider shared-memory reduction cost what the saved traffic buys, and the curve turns over rather than flattening. pp64 25.11 -> 55.81 (2.22x) pp381 22.92 -> 48.40 (2.11x) pp512 22.22 -> 45.31 (2.04x) pp1024 20.61 -> 37.25 (1.81x) Generation is untouched, as it should be — these are the batched kernels. One trap, worth the comment it now carries: a private array in generated device code is uninitialized stack rather than Java's zero-filled allocation. The first version left the accumulators unzeroed and produced NaN logits — loud, at least, rather than slightly wrong numbers. Verified: the kernel test now covers Q4_0's output tiling directly against the host tensor, including a partly filled tile and untouched inactive rows, and all 27 qwen35 accel tests pass, teacher-forced 27B parity included, at unchanged bounds.
The output-row tile landed on Q4_0 only. The re-profile put `ssm_out` (Q5_K) back at 11.3% and the early `ffn_down` (Q4_1) at 4.9% — not because they got slower but because everything around them got faster, and their per-instance times had barely moved: 5.43 -> 3.96 ms for Q4_1 against 3.39 -> 1.38 ms for the Q4_0 kernel that also tiles the output axis. Row tiling alone cuts weight decoding, which was never the limit. Both now cover two output rows per workgroup. Q5_K keeps the warp-per- sub-block mapping that made its row tile pay at all, and each of the two output rows recomputes its own sub-block header. pp381 b32 48.40 -> 52.28 t/s Two output rows, not four: four measured 49.71. Also restores the Q4_0 case of the kernel test, which the previous commit claimed and did not contain — a global `spotless:apply` followed by reverting the files it touched took the file back to its committed state, and the new test method with it. The Q4_0 output tiling was covered when it was measured and is covered again now; the claim was true of the code and false of that commit's diff. Verified: all three kernel-parity cases and the 27 qwen35 accel tests, teacher-forced 27B parity included.
…nadoVM findings llama.cpp does not dequantize inside its matmul: above MMVQ_MAX_BATCH_SIZE it quantizes the activations to Q8_1 and runs an integer dot product. Since the profile says activation traffic is what limits our tiled kernels, that looked like the thing to copy. It was built twice and kept neither time. Per-block Q8_1 with a warp-per-block integer dot was a 21% regression (52.28 -> 41.46 t/s): a lane owning a 32-element block reads 32 contiguous bytes no other lane reads, and every lane re-reads the block scales for all eight tiled rows. Per-row byte activations, keeping the float kernel's lane striding and therefore its coalescing, gained 1.8% (52.11 -> 53.06) — a real 4x cut in the dominant read, which says that read was already being served from cache. Neither pays for a per-row scale buffer, a quantization pass after every norm, and eight-bit activations through 64 layers. The useful conclusion is the negative one: the limit is no longer activation bytes, so the next thing to try is not a smaller activation. Two TornadoVM findings came out of the attempt and are written up with reproducers under docs/architecture/tornadovm-issues, for upstream: - a private array inside a kernel is not zero-initialized, against Java's guarantee. A 256-thread kernel that allocates `new float[8]`, adds 1.0f to each slot and sums returns NaN instead of 8.0. This cost a debugging round here: the accumulators of a batched projection produced NaN logits for the 27B model. - a kernel helper over 600 Graal nodes is refused with TornadoInliningException rather than falling back to a call, so a legitimately large kernel body has to be hand-duplicated. This repository now does that in three places.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.