Chapter 5

quant — 20 pack formats

github.com/openfluke/welvet/quant✅

Why it exists

Inference and storage need classic Q-packs, k-quants, IQ, Ternary/Binary, and Affine without a separate QAT training mode. Format is storage truth.

What it is

Pack / Unpack / MatVec / MatVecT for FormatNone…AffinePacked. SIMD inflate caches (EnsureFloatCache, EnsureQ4SIMDCache) exist for correct-but-not-peak paths.

Families: classic Q8/Q4/Q5 · K-quants · IQ · TernaryPacked/BinaryPacked · AffinePacked.

Honesty: fused SIMD for Q4/Q8/BitNet; k/IQ/Affine SIMD often inflate-once F32Cache+DotTile (🚧).

Go example

examples/05-quant/main.go

Run:cd welvet/examples/05-quant && source ../env.sh && go run .

package main

import (
	"fmt"

	"github.com/openfluke/welvet/quant"
)

func main() {
	w := []float32{0.1, -0.2, 0.3, 0.4, -0.5, 0.6, 0.05, -0.15}
	b, err := quant.Pack(quant.FormatQ4_0, w, 2, 4)
	if err != nil {
		panic(err)
	}
	y := make([]float32, 2)
	if err := quant.MatVec(b, []float32{1, 1, 1, 1}, y); err != nil {
		panic(err)
	}
	fmt.Println("rows", b.Rows, "cols", b.Cols, "y", y)
}

Output

exit 0 · last run via go run .

rows 2 cols 4 y [0.6857143 -1.4901161e-08]