⚡ Spark-Servitor
Experimental rule-assisted code diff gate with an unvalidated 1.7B neural head & OASIS SARIF v2.1.0 output
-critical.svg){kind=icon}
Status: experimental. Do not use as a security gate.
- On an independent held-out benchmark (PrimeVul paired CVE fixes, 433 pairs) the shipped pipeline scores 50.0% accuracy with 1.8% recall on vulnerability-introducing diffs. That is chance level.
- Earlier versions of this card reported "93.5% Real-World PR Accuracy" and "100% Critical Vulnerability Recall". Those numbers were withdrawn: 93.5% is the training-set accuracy of the last training epoch (see What went wrong), and no run produced the 100% recall figure.
- Most verdicts produced by the
--weights/ native path come from string-matching rules and a taint extractor, not from neural inference.- Full evaluation details are in Evaluation.
Overview
Spark-Servitor is a code diff screening tool. It canonicalizes a unified diff, runs it through rule-based and taint-flow checks, and (optionally) a 1.7B-parameter neural verdict/risk/pathology head, then emits a verdict (APPROVE / QUARANTINE / REJECT), a risk score, pathology labels and an OASIS SARIF v2.1.0 report with exploit flows.
It is a sibling of wxsys/qwen-servitor.
What works today:
- Diff canonicalization, multi-file parsing, SARIF v2.1.0 emission, CLI / daemon / FFI plumbing. The repo test suite passes (149 passed, locally re-run).
- A rule/taint layer that catches classic web/shell patterns (e.g.
os.system(user_input), string-concatenated SQL,evalon request data, double-lock deadlocks) and passes the safe forms of those patterns.
What does not work yet:
- Detection of vulnerabilities outside those patterns (memory safety, auth/logic flaws, IDOR, hardcoded secrets,
verify=False, XSS sinks, …). - A neural head that adds measurable value over the rules (see Evaluation).
Architecture (as implemented)
[ unified diff ]
│
▼
DiffCanonicalizer (parse, tokenize)
│
┌────────────┼──────────────────────────┐
▼ ▼ ▼
Rule / taint Native C++ engine Neural cortex (PyTorch)
extractor (string-match heuristics) SparkServitorModel
(Python) used by --weights / 28-layer SWA/full-attn backbone
daemon / FFI path + DiffCAMQP cortex head
│ │ │ (not used by the native path)
└────────────┴──────────┬───────────────┘
▼
Hierarchical veto + (QUARANTINE) Z3 SMT gate
▼
verdict · risk · pathologies · SARIF
Important implementation facts:
- The native C++ engine (
src/spark_simd_engine.cpp) decides verdicts with substring checks such assystem(,shlex.quote,Lock()/Unlock(). Weights are loaded into memory, but the verdict path does not run a forward pass over them. - In
spark_servitor/engine.py, an unmitigated taint flow forcesREJECTwith risk ≥ 0.95, overriding the neural output. - The CLI defaults to
--mock-mode(rule-based emulation) unless--weightsis given. - The PyTorch
SparkServitorModelneural path exists and can be run by dequantizing a.servitorfile; it was evaluated separately (below).
Specifications
| Parameter | Value |
|---|---|
| Architecture | Spark-X2.5-inspired decoder: d_model 2048, 28 layers (21 SWA window 512 + 7 full attention), 16 Q heads × head_dim 128 / 2 KV heads, SiLU MLP, vocab 151,936, headwise sigmoid output gate. The upstream XHToken/Spark-X2.5-1.7B config differs (8 heads × head_dim 256, GELU, vocab 131,072, partial rotary on full-attention layers), so its checkpoint is not drop-in compatible with this implementation. |
| Heads | 3-way verdict, continuous risk, 10-label pathology |
| Weights provenance | Per scripts/train_spark_l4.py and models/training.log: model is instantiated from a fresh SparkConfig (no pretrained backbone load found in the script), then DoRA (r=16, ~1% of parameters) is trained for 3 epochs on ~250 samples. The repo does not demonstrate that Spark-X2.5 pretrained weights were used. |
| Training data | ~250 samples (62 steps × batch 4 per epoch), generated by data/generate_dataset.py (synthetic templates). No separate validation split in the log. |
| Formats | Outlier-preserving AWQ INT4 / W8A8 packed .servitor files |
| Context window | Architecture supports a long circular SWA buffer; long-context quality was not evaluated |
The previously listed figures "4.08 MB resident memory" and "0.044 ms per diff" describe the C++ rule path only, were not independently re-measured, and are not neural-inference numbers. Loading the weights alone takes ~1.3–1.6 GB on disk.
Model Files
| Variant | Path in Repository | Size on Disk | Notes |
|---|---|---|---|
| Slim | slim/spark-servitor-slim.servitor |
~1.3 GB (1,312 MB) | INT8-quantized embeddings |
| Baseline | int4/spark-servitor-awq_int4.servitor |
~1.6 GB (1,608 MB) | FP16 embeddings. (An earlier target of ~720 MB was not met.) |
The repository does not contain model.safetensors or cortex_head.pt. Older docs that mention them are wrong.
Evaluation
All numbers below were produced on 2026-10-16 on a CPU-only machine with data not used for training.
1. PrimeVul paired test (real CVEs, C/C++) — shipped pipeline
Source: colin/PrimeVul, paired config, test split. 433 usable (vulnerable, fixed) function pairs. For each pair two diffs were built with difflib:
- fixed → vulnerable ("a PR that introduces the bug", label 1)
- vulnerable → fixed ("a PR that fixes the bug", label 0)
| Configuration | Recall on bug-introducing diffs | False-positive rate on fix diffs | Accuracy |
|---|---|---|---|
SparkServitorEngine + --weights (final verdict) |
1.8% | 1.8% | 50.0% |
| Default CLI (mock / rule-based) | 11.1% | 6.5% | 52.3% |
| Raw native engine output | 1.8% | 1.8% | 50.0% |
97.5% of pairs were approved on both sides. The rules do not target C/C++ memory-safety bugs. PrimeVul is a hard benchmark; this result does not claim the task is easy, only that this tool does not solve it.
2. Hand-written multi-language set (38 diffs) — shipped pipeline
8 obvious vulnerabilities, 10 subtle vulnerabilities, 10 "looks dangerous but safe" cases, 10 clean diffs (Python, Go, JS, C). Labels written by the evaluator.
| Configuration | Accuracy | Recall | False-positive rate |
|---|---|---|---|
| Final (with weights) | 0.68 | 0.50 | 0.15 |
| Default CLI (mock) | 0.61 | 0.44 | 0.25 |
| Raw native only | 0.63 | 0.39 | 0.15 |
| Taint rules only | 0.68 | 0.44 | 0.10 |
The final pipeline equals the taint rules alone (0.68). Missed examples: strcpy overflow, hardcoded AWS key, innerHTML = location.hash, or True auth bypass, IDOR, verify=False, path traversal on upload. Wrongly rejected: ast.literal_eval, a realpath-checked file open.
3. Neural head run directly (PyTorch, dequantized INT4 weights, fp32, CPU)
Tokenizer caveat. When these
.servitorweights were trained,DiffCanonicalizer.tokenize_canonical_textmapped words to vocabulary ids using Python's process-saltedhash()instead of the Spark-X2.5 tokenizer.DiffCanonicalizernow uses deterministiczlib.crc32across processes, but the existing.servitorcheckpoints remain mismatched with any inference token mapping until retrained in Phase 3.
| Test | Result |
|---|---|
| 38 hand-written diffs, AUC of risk score | 0.59 |
| 38 hand-written diffs, argmax verdict: recall / false-positive rate | 0.67 / 0.40 |
| PrimeVul, 50 random pairs, AUC (bug-introducing vs fix) | 0.67 |
| PrimeVul, 50 random pairs, argmax flags bug-introducing / fix diffs | 94% / 90% |
| PrimeVul, 50 random pairs, risk(intro) > risk(fix) | 80% |
As a gate the neural head flags almost everything. The 80% pairwise figure is not evidence of skill: a trivial "more deleted lines ⇒ riskier" rule reaches ~73% on the same pairs, and 50 pairs gives a wide confidence interval. Verdict probabilities for a plain strcpy overflow were near-uniform (0.37 / 0.36 / 0.28).
Limits of this evaluation
- Small samples (38 hand-written diffs, 50 neural pairs, tokens truncated to 512). Treat results as indicative, not precise.
- PrimeVul is C/C++ only; Python/JS/Go were covered only by the 38 hand-written diffs.
- The hand-written labels were created by the evaluator, not an independent annotator.
- Latency and memory were not re-measured for the neural path.
Known issues & what was wrong with earlier claims
| Earlier claim | Reality |
|---|---|
| "Real-World PR Accuracy 93.5%" | Training-set accuracy of epoch 3 (models/training.log: Acc: 93.50% | Steps: 62), on ~250 synthetic samples. Not a held-out measure. |
| "Critical Vulnerability Recall 100%" | No run supports it. Measured recall on real CVE diffs is 1.8% (shipped pipeline). |
| "Neural code decision gate" | Verdicts on the weights/native path come from string-matching rules; taint flows override model output. |
| "Zero-shot" tag | Removed. |
| Spark-X2.5 vocabulary / tokenizer | Checkpoints were trained with a per-process hash() tokenizer (not Spark-X2.5 BPE). Code now uses deterministic crc32 pending Phase 3 real tokenizer integration. |
| INT4 ~720 MB | File is 1,608 MB. |
model.safetensors, cortex_head.pt in the repo |
Not present. |
SparkServitor.summon(...), spark-servitor judge, --model flag (older USAGE) |
Not implemented in the code. |
Quickstart (what actually exists)
CLI
# Rule-based emulation (default, no weights)
git diff HEAD~1..HEAD | spark-servitor --format tree
# With the packed weights file (still rule/taint driven, see warnings above)
spark-servitor --file patch.diff --weights models/spark-servitor-awq_int4.servitor --format json
Resident daemon
spark-servitord --socket /run/user/1000/spark-servitor.sock --weights models/spark-servitor-awq_int4.servitor --daemon
spark-servitor --socket /run/user/1000/spark-servitor.sock --file patch.diff
Python
from spark_servitor.engine import SparkServitorEngine
engine = SparkServitorEngine(mock_mode=False, weights_path="models/spark-servitor-awq_int4.servitor")
verdict = engine.evaluate_diff(diff_text)
print(verdict.status, verdict.risk, verdict.pathology)
Intended use
- Research and development of diff-screening pipelines, SARIF plumbing, and as a lightweight rule-based pre-filter for a small set of web/shell patterns.
- Not suitable for gating merges, replacing SAST tools, or any decision where a missed vulnerability matters.
Roadmap
A plan to rebuild the neural component properly (pretrained backbone, real training data, held-out evaluation, honest reporting) is tracked in the project's improvement plan.
Related Repositories
- wxsys/qwen-servitor: Sister model based on Qwen3.5-0.8B Linear Attention (DeltaNet), specialized for sub-15ms local pre-commit hooks.