Qwen

/

Qwen3.8-2.4T-A95B

Like 1.23k

Follow

Qwen 105k

A large number of matrices starting from blk 0, blk 1, blk 2, blk 10 etc are singular in this Qwen3.8-2.4T-A95B model. Up to 84.3% of expert weights in early blocks are zero. They repeatedly distort the signal from the tokenizer instead of analyzing it.

Tensor	Shape	QType	zer%
blk.0.ffn_down_exps.weight	2048×8192×512	Q8_0	0.6%
blk.0.ffn_gate_exps.weight	8192×2048×512	Q8_0	71.0%
blk.0.ffn_up_exps.weight	8192×2048×512	Q8_0	71.0%
blk.1.ffn_gate_exps.weight	8192×2048×512	Q8_0	84.3%
blk.1.ffn_up_exps.weight	8192×2048×512	Q8_0	84.3%

Kimi K3 doesn’t have this issue. No zero blocks found in Kimi K3 in tensors on F16, F32 and MXFP4 precision in blk 0, blk 1, blk 2.

In any case, thanks for release Alibaba team 😉 and your first experiments with trillion parameter models. Hoping to see healthy Qwen4.0 from you in future especially 35B-A3B because I absolutely love it.

I think, low-rank or even singular weight matrices in early blocks, along with a high proportion of near-zero expert weights, are not evidence of the model “distorting” the tokenizer signal—they are a common and often beneficial outcome of large-scale training. Neural networks, especially sparse MoE architectures with hundreds of fine-grained experts, naturally learn compressed, specialized representations; the effective rank of weight matrices is frequently much lower than the ambient dimension, which can act as implicit denoising and feature selection rather than information destruction. Interpreting these statistical regularities as pathology reflects a bias against low-rank structure, ignoring both theoretical results on the prevalence of low effective rank in deep networks and the strong empirical performance of this model.

A large number of matrices starting from blk 0, blk 1, blk 2, blk 10 etc are singular in this Qwen3.8-2.4T-A95B model. Up to 84.3% of expert weights in early blocks are zero. They repeatedly distort the signal from the tokenizer instead of analyzing it.

Tensor	Shape	QType	zer%
blk.0.ffn_down_exps.weight	2048×8192×512	Q8_0	0.6%
blk.0.ffn_gate_exps.weight	8192×2048×512	Q8_0	71.0%
blk.0.ffn_up_exps.weight	8192×2048×512	Q8_0	71.0%
blk.1.ffn_gate_exps.weight	8192×2048×512	Q8_0	84.3%
blk.1.ffn_up_exps.weight	8192×2048×512	Q8_0	84.3%

Kimi K3 doesn’t have this issue. No zero blocks found in Kimi K3 in tensors on F16, F32 and MXFP4 precision in blk 0, blk 1, blk 2.

In any case, thanks for release Alibaba team 😉 and your first experiments with trillion parameter models. Hoping to see healthy Qwen4.0 from you in future especially 35B-A3B because I absolutely love it.

I think, low-rank or even singular weight matrices in early blocks, along with a high proportion of near-zero expert weights, are not evidence of the model “distorting” the tokenizer signal—they are a common and often beneficial outcome of large-scale training. Neural networks, especially sparse MoE architectures with hundreds of fine-grained experts, naturally learn compressed, specialized representations; the effective rank of weight matrices is frequently much lower than the ambient dimension, which can act as implicit denoising and feature selection rather than information destruction. Interpreting these statistical regularities as pathology reflects a bias against low-rank structure, ignoring both theoretical results on the prevalence of low effective rank in deep networks and the strong empirical performance of this model.

Low-rank is fine. Singular is not. Specialization is fine. Dead experts are not. 0.0% zero blocks in Kimi K3 vs 84.3% in Qwen3.8 2.4T is not a stylistic difference. It is a training collapse. Low-rank would still have non-zero σ_rob. σ_rob = 0.00000 means the tensor is not low-rank. It is constant. And with 71-84.3% zeros, that constant is zero.

σ_rob = 0.00000 after Q8_0 dequantization means the spread is either exactly zero, or so small that even 8-bit/BF16 precision cannot preserve it. In both cases, the tensor carries no usable expert diversity.

A large number of matrices starting from blk 0, blk 1, blk 2, blk 10 etc are singular in this Qwen3.8-2.4T-A95B model. Up to 84.3% of expert weights in early blocks are zero. They repeatedly distort the signal from the tokenizer instead of analyzing it.

Tensor	Shape	QType	zer%
blk.0.ffn_down_exps.weight	2048×8192×512	Q8_0	0.6%
blk.0.ffn_gate_exps.weight	8192×2048×512	Q8_0	71.0%
blk.0.ffn_up_exps.weight	8192×2048×512	Q8_0	71.0%
blk.1.ffn_gate_exps.weight	8192×2048×512	Q8_0	84.3%
blk.1.ffn_up_exps.weight	8192×2048×512	Q8_0	84.3%

Kimi K3 doesn’t have this issue. No zero blocks found in Kimi K3 in tensors on F16, F32 and MXFP4 precision in blk 0, blk 1, blk 2.

In any case, thanks for release Alibaba team 😉 and your first experiments with trillion parameter models. Hoping to see healthy Qwen4.0 from you in future especially 35B-A3B because I absolutely love it.

I think, low-rank or even singular weight matrices in early blocks, along with a high proportion of near-zero expert weights, are not evidence of the model “distorting” the tokenizer signal—they are a common and often beneficial outcome of large-scale training. Neural networks, especially sparse MoE architectures with hundreds of fine-grained experts, naturally learn compressed, specialized representations; the effective rank of weight matrices is frequently much lower than the ambient dimension, which can act as implicit denoising and feature selection rather than information destruction. Interpreting these statistical regularities as pathology reflects a bias against low-rank structure, ignoring both theoretical results on the prevalence of low effective rank in deep networks and the strong empirical performance of this model.

Low-rank is fine. Singular is not. Specialization is fine. Dead experts are not. 0.0% zero blocks in Kimi K3 vs 84.3% in Qwen3.8 2.4T is not a stylistic difference. It is a training collapse. Low-rank would still have non-zero σ_rob. σ_rob = 0.00000 means the tensor is not low-rank. It is constant. And with 71-84.3% zeros, that constant is zero.

σ_rob = 0.00000 after Q8_0 dequantization means the spread is either exactly zero, or so small that even 8-bit/BF16 precision cannot preserve it. In both cases, the tensor carries no usable expert diversity.

Low rank matrices are singular by definition. Also what is σ_rob? I am guessing std of singular values ignoring outliers? That can be 0, that would mean the matrix is near orthogonal which is the opposite of singular

σ_rob is robust std of weight values, not of singular values.

σ_rob = 1.4826 × median(|wᵢ − median(w)|)

where 1.4826 = 1 / Φ⁻¹(3/4) ≈ 1 / 0.6745 is the consistency
correction for MAD under a normal distribution.

σ_rob = 0 means the tensor is essentially constant.
With 71–84% zeros, that constant is zero.

That is not orthogonality. That is collapse.

Why Q8_0?

With Q8_0, 32 values share the same scale. If one value is roughly 254× larger than another value within the same block, the smaller value can be quantized to zero due to rounding.
So before concluding that the original BF16 weights contain 71–84% zeros, shouldn't this be checked against the original BF16 tensors first?

Here is the BF16 weight for model.layers.1.mlp.experts.gate_up_proj (corresponding to your blk.1.ffn_gate_exps.weight and blk.1.ffn_up_exps.weight):
https://huggingface.co/Qwen/Qwen3.8-2.4T-A95B/resolve/main/model-00003-of-00213.safetensors

And here are the corresponding BF16 GGUF files:
https://huggingface.co/unsloth/Qwen3.8-2.4T-A95B-GGUF/resolve/main/BF16/Qwen3.8-2.4T-A95B-BF16-00002-of-00140.gguf
https://huggingface.co/unsloth/Qwen3.8-2.4T-A95B-GGUF/resolve/main/BF16/Qwen3.8-2.4T-A95B-BF16-00003-of-00140.gguf

It would be interesting to compare the zero ratio and σ_rob directly on the original BF16 tensor. If the 84.3% zero ratio is already present there, that's a very different story. If it only appears after Q8_0 quantization, then the conclusion about training collapse would need to be reconsidered.

Why Q8_0?

With Q8_0, 32 values share the same scale. If one value is roughly 254× larger than another value within the same block, the smaller value can be quantized to zero due to rounding.
So before concluding that the original BF16 weights contain 71–84% zeros, shouldn't this be checked against the original BF16 tensors first?

Here is the BF16 weight for model.layers.1.mlp.experts.gate_up_proj (corresponding to your blk.1.ffn_gate_exps.weight and blk.1.ffn_up_exps.weight):
https://huggingface.co/Qwen/Qwen3.8-2.4T-A95B/resolve/main/model-00003-of-00213.safetensors

And here are the corresponding BF16 GGUF files:
https://huggingface.co/unsloth/Qwen3.8-2.4T-A95B-GGUF/resolve/main/BF16/Qwen3.8-2.4T-A95B-BF16-00002-of-00140.gguf
https://huggingface.co/unsloth/Qwen3.8-2.4T-A95B-GGUF/resolve/main/BF16/Qwen3.8-2.4T-A95B-BF16-00003-of-00140.gguf

It would be interesting to compare the zero ratio and σ_rob directly on the original BF16 tensor. If the 84.3% zero ratio is already present there, that's a very different story. If it only appears after Q8_0 quantization, then the conclusion about training collapse would need to be reconsidered.

Here are the results checked directly on the BF16 GGUF files you requested: