docs/GETTING_STARTED.md

Getting started with DaisyChain-Infer

This is a local project. There is no hosted version — you clone it and run it on your own machines.

Added 2026-10-22

The rest of this file is the previous getting-started note. This list is what changed.

• Context length still defaults to 64. A prompt longer than 32 tokens on a multi-device ring uses even chunked prefill. If this device can hold the whole model, the ring is not used.
• New log lines: lap estimate for the whole plan, queued prompts, overlap prefill, draft span. A disagreeing probe is left out of the plan.
• New failures: "a peer left the ring" (one replay), "REFUSED activation" for a non-finite hop, "refusing the run" when the lap estimate exceeds 120 seconds, "stays out of the plan: kernel probe".

Added 2026-10-23

• A device that owns two or more layers shows flat shard, 2 layers live on its slice line. The weights are one buffer on that device. They are not fetched from a peer. A device with one layer still says resident.
• Output can be free or JSON. JSON masks the completion. The seed above temperature 0 is tied to the token position. A prompt typed while a turn is running is tokenized before that turn ends.
• DeepGEMM is not used. The multiply stays the verified INT8 GEMM, one scale per row and one per column.

Run the server

npm install
npm start            # http://localhost:8788

Open it in two tabs to try the ring on one machine, or on two devices to try it for real.

HTTPS matters. WebGPU and cross-device WebRTC need a secure context: localhost or HTTPS. Plain http://192.168.x.x from another device will not get WebGPU and may not connect at all. For real multi-device runs on your own network, put it behind a tunnel (cloudflared, ngrok) or a local TLS reverse proxy.

Pick a model

Type any Hugging Face repo id — owner/name — and press Load.

Works with .safetensors repos whose architecture is:

• Llama-style: Llama, Mistral, Qwen2/2.5, SmolLM, TinyLlama
• GPT-2-style: GPT-2 and close relatives

Good ones to start with:

Repo	Size	Notes
HuggingFaceTB/SmolLM-135M	513 MB f32	30 layers — splits nicely
openai-community/gpt2	523 MB f32	the GPT-2 path
Qwen/Qwen2.5-0.5B	942 MB f32	GQA 14/2, and it ships QKV biases

Loading reads only config.json, tokenizer.json and the weight headers — a few tens of KB, whatever the model's size. No weights move until you press Generate, and then each device fetches only its own layers.

Weights are held as f32, so budget ~4 bytes per parameter across the group. The ring plan shows exactly how many MB each device will download and hold.

An unsupported architecture or tokenizer is refused with a message naming what it found. That is deliberate: approximating one would produce fluent, confident, wrong text.

Gated or private models

You are asked for a Hugging Face token once, and only when a request actually fails for want of one. Create a read token at huggingface.co/settings/tokens.

The token is held in memory for that tab and nowhere else — not localStorage, not sessionStorage, not a cookie, not the URL, never logged, and never sent to another device. Each device is prompted for its own, because each device downloads its own layers. Reloading the tab forgets it, and Forget token clears it immediately.

For a gated model, accept its licence on the model page first — a token alone will not get past a licence you have not accepted.

Rooms

Devices on one network group automatically (by public IP, Snapdrop-style). To include devices on other networks, everyone opens:

https://<host>/?room=MY-SECRET-CODE

The first person in is the host and approves each device individually.

Run it

1. Wait until the devices see each other in the peer list.
2. On one device, load the model. That device becomes the head: it holds the embedding table, embeds your prompt, and turns the returning hidden state back into words.
3. Check the ring plan — who holds which layers, and how much each will download.
4. Type a prompt and press Generate. Each device fetches its layers (this takes a moment the first time), reports ready, and the ring starts.

Setting	Default	What it does
Tokens	60	how many to generate
Context length	64	(added 2026-10-22) full-window size; the main cost per token on that path. A prompt longer than 32 tokens on a multi-device ring uses even chunked prefill instead. If this device can hold the whole model, the ring is not used
Temperature ÷100	0	0 = greedy, deterministic and therefore checkable
Seed	1234	drives sampling above temperature 0, so runs stay reproducible
Verify	off	afterwards, re-run with every layer here and compare

Every device shows the token stream as it arrives — including ones holding a few layers and no vocabulary.

Reading the numbers

• tokens/sec — the whole ring's rate. Expect it to fall as you add stages: each one adds a round trip per token. More stages buy capacity, not speed.
• ring hops — stages × tokens.
• kernel probe (in the log) — (added 2026-10-22) the same number on every device that is allowed into the plan. A peer reporting a different one is left out.
• lap estimate — (added 2026-10-22) milliseconds per token for the plan that will actually run, each stage at its own rate. Above 120 seconds the run is refused before the download.
• queued — (added 2026-10-22) a prompt typed while a turn is generating. It starts when the ring is free.
• overlap prefill — (added 2026-10-22) chunks of about 16, two in flight, per-stage KV. Not the full-window GEMM. Verify repeats this path.
• draft span — (added 2026-10-22) a repeated suffix scored in one lap on the full-window path.
• my slice — which layers this device holds and how much memory they take. (added 2026-10-23) Two or more layers are kept in one flat shard, with at most two unpacked during a forward. That does not shrink the download.
• Output — (added 2026-10-23) free or JSON. JSON applies after the hop, to the top 40 tokens of the completion.

Verify

Tick Verify before generating. Afterwards the head downloads the whole model and re-runs the identical prompt locally, then compares token ids:

VERIFIED: the distributed run and the single-device run produced identical token ids.

In a pipeline nothing recomputes anything, so this is the only check that can show a distributed answer is right rather than merely self-consistent. It needs a model one device can hold, which is why it is optional.

Without a browser:

npm test

When it goes wrong

• (added 2026-10-22) "a peer left the ring" — the head replans and replays the prompt once. If the replay also fails, press Generate again.
• (added 2026-10-22) "REFUSED activation" — a non-finite hop. It is not forwarded.
• (added 2026-10-22) "refusing the run: estimated … ms per token" — the lap budget of 120 seconds was exceeded. Add a faster device or lower the context length.
• "unsupported architecture" — the model is not Llama-style or GPT-2-style.
• "tokenizer type … is not supported" — the repo uses SentencePiece or WordPiece; only byte-level BPE is implemented.
• 401 / 403 — the model needs a token, or the token lacks access. For gated repos, accept the licence on the model page first.
• "the server ignored the byte range" — the whole file came back instead of a slice; refused rather than accepted, since that defeats the point.
• "REFUSED activation … belongs to model X" — a stage still holds a slice of a previous model. Reload that device.
• (added 2026-10-22) "stays out of the plan: kernel probe" — that device's arithmetic differs. It is not given layers.
• out of memory — the stage's slice does not fit. Add devices, or pick a smaller model; the plan shows the per-device cost before you start.

More

• ARCHITECTURE.md — the ring, the plan, the wire protocol.
• The parent projects: DaisyChain-Train · DaisyChain-Web