RedHatAI/Qwen3-8B-speculator.dflash

This is a DFlash speculator model for Qwen/Qwen3-8B, trained without reasoning (enable_thinking=False).

Training Details

This model was trained using the Speculators library on a mix of Magpie-Align/Magpie-Llama-3.1-Pro-300K-Filtered, the train_sft split of HuggingFaceH4/ultrachat_200k, and nvidia/Nemotron-Personas data. Responses were regenerated by Qwen3-8B (without reasoning). Training uses D-PACE loss weighting and the Muon optimizer.

Commands

Using the Speculators library and the helper scripts provided in the repo.

Prepare data

python scripts/prepare_data.py \
  --model Qwen/Qwen3-8B \
  --data ./regenerated_data.jsonl \
  --output ./output \
  --seq-length 65536

Launch vLLM

CUDA_VISIBLE_DEVICES=0,1 vllm_venv/bin/python scripts/launch_vllm.py \
  Qwen/Qwen3-8B \
  --target-layer-ids 2 10 18 26 34 \
  -- --port 8000 \
  --gpu-memory-utilization 0.9 \
  --disable-uvicorn-access-log \
  --tensor-parallel-size 2

Launch training

CUDA_VISIBLE_DEVICES=2,3 torchrun \
  --standalone \
  --nproc_per_node 2 \
  scripts/train.py \
  --verifier-name-or-path Qwen/Qwen3-8B \
  --speculator-type dflash \
  --block-size 16 \
  --num-layers 5 \
  --data-path ./output \
  --vllm-endpoint http://localhost:8000/v1 \
  --save-path ./output/checkpoints \
  --epochs 20 \
  --lr 6e-4 \
  --total-seq-len 65536 \
  --max-anchors 1536 \
  --target-layer-ids 2 10 18 26 34 \
  --sliding-window 2048 \
  --mask-token-id 151669 \
  --on-missing generate \
  --on-generate delete \
  --loss-fn ce \
  --per-position-loss-weight dpace \
  --dpace-alpha 0.5 \
  --optimizer muon \
  --muon-lr 0.003 \
  --muon-momentum 0.95 \
  --muon-weight-decay 0.1 \
  --muon-ns-steps 5 \
  --muon-adjust-lr-fn match_rms_adamw \
  --weight-decay 0.01 \
  --scheduler-type cosine \
  --prefetch-factor 2 \
  --num-workers 8 \
  --log-freq 100 \
  --seed 42

Model Specifications


Base Model	Qwen/Qwen3-8B
Chat Template	Qwen/Qwen3-8B (use `/chat/completions` endpoint)
Reasoning	Disabled (`enable_thinking=False`)
Format	Safetensors
License	Apache 2.0
Validation Hardware	4× Nvidia H100

Deployment

pip install vllm

vllm serve Qwen/Qwen3-8B \
    --tensor-parallel-size 1 \
    --max-model-len 16384 \
    --speculative-config '{
        "model": "RedHatAI/Qwen3-8B-speculator.dflash",
        "num_speculative_tokens": 15,
        "method": "dflash"
    }'

Evaluations

Per-position token acceptance rates on the RedHatAI speculative decoding benchmark (native 15 draft tokens, enable_thinking=False, max_tokens=512, temperature=0):

Dataset	Pos 1	Pos 2	Pos 3	Pos 4	Pos 5	Pos 6	Pos 7	Avg Length
HumanEval	87.6%	70.5%	55.3%	45.3%	37.6%	32.0%	27.6%	5.883
math_reasoning	91.7%	77.4%	66.3%	56.6%	48.7%	42.2%	35.8%	6.805
qa	67.9%	40.7%	24.0%	14.8%	9.3%	6.3%	4.4%	2.798
question	73.2%	47.0%	29.8%	20.3%	14.1%	10.6%	8.3%	3.341
rag	73.7%	49.6%	33.8%	23.5%	16.7%	12.5%	9.6%	3.564
summarization	70.6%	41.0%	24.1%	15.1%	9.3%	6.1%	3.6%	2.750
tool_call	76.1%	51.7%	35.3%	26.1%	18.4%	14.5%	11.4%	3.701
translation	79.2%	51.8%	26.3%	10.9%	5.0%	2.4%	1.2%	2.783
writing	73.3%	46.8%	29.8%	20.3%	14.5%	10.9%	8.2%	3.317