Chapter 44

Octo — model shell

github.com/openfluke/welvet/apps/octo✅ runs

Why it exists

A model is only useful with a shell around it: pull weights from Hugging Face, convert them to a Welvet .entity, then chat, serve, or benchmark. Octo is that shell, kept in its own module so the engine never depends on an app.

What it is

Subcommands cover the whole loop: hub download/ensure repos, convert pack to a single .entity, interactive run/serve/chat, plus image and speech menus. Its bench harness sweeps every quant format across a CPU Plan 9 SIMD fused profile and a WebGPU fused profile.

Bench: SmolLM2-135M-Instruct across quant formats

template smol2-135m-fuse · linux/amd64 · NVIDIA GTX 1650 SUPER · 3m17s · 80 runs (76 ok, 4 skipped: "none" has no fused kernel). Throughput is total tokens/s on the first prompt; entity is on-disk size.

Quant	GPU-fused tok/s	SIMD-fused tok/s	Entity MB
Q8_0	108.2	33.2	252.5
Q6_K	73.5	4.9	216.4
Q5_K	71.7	6.3	208.4
Q4_K	73.4	7.8	192.3
Q4_0	112.1	36.8	188.3
IQ4_XS	105.5	7.6	204.4
IQ4_NL	84.1	8.8	188.3
TernaryPacked	106.9	18.9	172.3
BinaryPacked	88.3	21.0	140.2

21 quant formats run end to end; the table shows a representative slice. Full report: apps/octo/dist/octo-run-v0.76.txt.

Honest low-bit noteHigh-bit formats (Q8, Q6, Q5, Q4, IQ4) stay coherent. Sub-2-bit formats (IQ1/IQ2, Ternary, Binary) are fast and tiny but degrade into gibberish. The bench keeps them in to show the speed, size, and quality trade honestly rather than hiding it.

Sample replies

Q8_0  / gpu   I'm a helpful assistant. I'm here to help you with your inquiries.
Q4_0  / gpu   I'm ready to help. What can I help you with?
Q6_K  / gpu   2 + 2 is 4.
Q6_K  / simd  I'm a helpful assistant.
IQ2_XXS/gpu   ,))hidAPAAPAAPAAPAAPA...   (sub-2-bit degrades)

Go example

examples/44-octo/main.go

Run:cd welvet/examples/44-octo && source ../env.sh && go run .

package main

import "fmt"

func main() {
	fmt.Println("Octo is its own module — the engine never imports it.")
	fmt.Println("  cd apps/octo && go run .")
	fmt.Println("  download -> convert -> .entity -> chat / serve / bench")
}

Output

Run python3 _gen_welvet_book.py --run to capture output.

Validate (harness)

cd apps/octo && go run .