HermiHg/Qwen3.8-27B-DFlash2-Q2_K_S-MIX-GGUF

Qwen3.8-27B-DFlash2 — Q2_K_S-MIX draft (half the reference size)

A mixed-precision 2–3-bit quantization of the DFlash 2 draft model for Qwen/Qwen3.8-27B, built to be 50% the size of the reference Q4_K_M checkpoint while retaining ~98% of its throughput.

Q2_K_S-MIX is a mixed-precision quant that compresses the large feed-forward blocks hard while keeping the small, high-impact tensors (the path selector and feature projection) more precise, so it lands at half the reference size with only a small acceptance loss.

Measured performance (vs the reference Q4_K_M)

All draft models served with llama-server (DFlash 2, PR #27342) against the Qwen3.8-27B target on a 24 GB NVIDIA GPU, one fixed conversational prompt with medium reasoning effort, temperature 1.0, concurrency 1, 5 replicate runs. n_max is the draft block length (speculative tokens drafted per verification step).

n_max	Metric	Q4_K_M (1,090 MiB)	Q3_K_M (874 MiB)	Q2_K (673 MiB)	Q2_K_S-MIX (535 MiB)	Ratio (vs Q4)
3	acceptance	0.539	0.543	0.525	0.524	0.97
4	acceptance	0.466	0.459	0.441	0.435	0.93
5	acceptance	0.403	0.403	0.388	0.373	0.93
3	draft len	2.62	2.63	2.57	2.57	0.98
4	draft len	2.86	2.83	2.76	2.74	0.96
5	draft len	3.01	3.01	2.93	2.86	0.95
3	tok/s	95.2	96.6	94.9	96.2	1.01
4	tok/s	99.2	98.3	96.1	95.6	0.96
5	tok/s	104.2	104.8	102.4	101.9	0.98
—	Size	1,090 MiB (4.76 bpw)	874 MiB (3.81 bpw)	673 MiB (2.93 bpw)	535 MiB (2.33 bpw)	0.49

Q2_K_S-MIX is the smallest draft (535 MiB) and sits on the size–throughput frontier: it posts the lowest acceptance of the four but converts it into within-a-few-percent throughput at every n_max. Non-imatrix Q2_K quants were strictly worse than Q2_K_S-MIX — larger, with lower acceptance and throughput — so they are omitted here.

The 2.33-bit draft model accepts slightly fewer tokens per step than the 4.76-bit reference (e.g. 0.524 vs 0.539 acceptance, 2.57 vs 2.62 draft len at n_max=3), which shows up as a small throughput gap (within a few percent at each n_max). Because DFlash 2 is lossless, this costs speed, not quality — for the same prompt the output is accepted by the same target at the same quality; the smaller drafter just needs marginally more verification steps.

If you have the VRAM to spare, I highly recommmend the Q3_K_M imatrix quant as an option — it posts higher acceptance at every n_max at the the cost of just a bit of context size when memory-constrained, however vision is broken on that release until you run the patcher script in this repo to add the required metadata.

Throughput vs draft size, by block length n_max:

How it was built (changes vs the reference)

Built clean from the upstream BF16 drafter (incoai/Qwen3.8-27B-DFlash2 GGUF) with llama-quantize on a build with DFlash 2 support (PR #27342). No dequant-from-quant: the source is the full-precision checkpoint. It is a q2_k_s base with per-tensor --tensor-type overrides, and the low-bit I-quants are quantized with a real activation-calibrated importance matrix (imatrix) — captured from the draft's own decode activations — rather than a flat one, so each super-block is weighted by the magnitudes it actually sees:

Component	Tensors	Quant
Feed-forward (SwiGLU gate/up/down)	~69% of params	iq2_xxs
Token-path selector (hidden / predecessor / successor)	~7%	hidden q5_k; pred/succ iq2_s
Feature projection fc	5120 × 25600	q3_k
Two-tap dynamic-conv projections (attn + ffn)	2 × 5 blocks	iq2_xxs
Attention (q / output ; k / v)	per block	iq2_s ; iq3_s
Layer norms + conv bases	32 tensors	f32 (held, not quantized)

The feed-forward block is 69% of the parameters, so it carries the size savings; the selector and fc are kept higher-precision because they drive which tokens the draft model proposes (acceptance), and the norms/conv-bases stay full precision.

Required build: DFlash 2 support

This drafter needs a llama.cpp build with DFlash 2 support (merged into main on 2026.08.27).

Saving memory elsewhere: multimodal projector (mmproj)

If you are also looking to save memory on the vision side, try my Qwen3.8-27B-mmproj-Q5_K-MIX projector checkpoint: 37% of the size of the upstream BF16 mmproj (331 MiB vs 888 MiB) at a measured accuracy cost within the evaluation's noise (see its model card for the full numbers).

Usage

Install llama.cpp with DFlash 2 support, then serve with this checkpoint as the draft:

llama-server \
  -hf <your-target-repo>/Qwen3.8-27B-GGUF:<target-file> \
  -hfd HermiHg/Qwen3.8-27B-DFlash2-Q2_K_S-MIX-GGUF:Q2_K_S \
  --spec-type draft-dflash \
  --spec-draft-n-max 3

---

Qwen3.8-27B-DFlash2-GGUF

Blog | GitHub

This repository contains GGUF conversions of incoai/Qwen3.8-27B-DFlash2, the DFlash 2 draft model for Qwen/Qwen3.8-27B. It is not a standalone language model: it runs inside a speculative decoding server and drafts tokens for the target model to verify. The checkpoints are also mirrored at z-lab/Qwen3.8-27B-DFlash2-GGUF.

DFlash 2 is a block-diffusion drafter for speculative decoding. It predicts a whole block of tokens in a single pass and keeps the top candidates at every position. A lightweight selector then traces one coherent path through them. Two-tap dynamic convolutions in the backbone keep the draft from decaying toward the end of the block. Decoding is lossless: greedy output matches the target model exactly, and sampling preserves its distribution.

File	Size
Qwen3.8-27B-DFlash2-Q4_K_M.gguf	1.1 GB
Qwen3.8-27B-DFlash2-Q8_0.gguf	2.0 GB
Qwen3.8-27B-DFlash2-BF16.gguf	3.8 GB

Quick Start

Build llama.cpp with DFlash 2 support (PR #27342):

git clone https://github.com/ggml-org/llama.cpp.git
cd llama.cpp
git fetch origin pull/27342/head:pr-27342
git switch pr-27342

# NVIDIA CUDA
cmake -B build -DCMAKE_BUILD_TYPE=Release -DGGML_CUDA=ON
cmake --build build -j

# Apple Silicon
cmake -B build -DCMAKE_BUILD_TYPE=Release -DGGML_METAL=ON
cmake --build build -j

Then serve:

./build/bin/llama-server \
  -hf ggml-org/Qwen3.8-27B-GGUF:Q4_K_M \
  -hfd incoai/Qwen3.8-27B-DFlash2-GGUF:Q4_K_M \
  --spec-type draft-dflash \
  --spec-draft-n-max 7

See the blog post for other engines and more details.

Evaluation

• Target: ggml-org/Qwen3.8-27B-GGUF, Q4_K_M
• Sampling: Qwen3.8's officially recommended parameters (temperature 1.0, top-p 0.95, top-k 20), with xhigh reasoning effort
• Maximum new tokens: 2048
• Prompts: the first eight GSM8K test examples

Acceptance Length

Acceptance length is the per-request mean of completion tokens divided by verification steps. Higher is better.

Draft GGUF	Acceptance Length
BF16	5.28
Q8_0	5.13
Q4_K_M	5.39

Full evaluations of the base checkpoint are on the main model card.

Citation

If you find DFlash 2 useful, please cite:

@misc{inco2026dflash2,
  title  = {{DFlash 2: Keep Drafting Parallel}},
  author = {{Inco AI}},
  year   = {2026},
  month  = {August},
  url    = {https://inco.ai/blog/dflash2/}
}

Please also cite the original DFlash paper:

@inproceedings{chen2026dflash,
  title     = {{DFlash: Block Diffusion for Flash Speculative Decoding}},
  author    = {Chen, Jian and Liang, Yesheng and Liu, Zhijian},
  booktitle = {International Conference on Machine Learning (ICML)},
  year      = {2026}
}