Chapter 10

fusedgpu — decoder on device

github.com/openfluke/welvet/fusedgpu✅

Why it exists

Token-by-token host round-trips kill decode. A fused engine keeps weights and scratch resident for Q4_0 and BinaryG128 hybrid paths.

What it is

Engine from Spec (AppendTokens/Reset/Close); HybridEngine (PrefillSample/DecodeSample/DecodeChunk). Specs come from model/transformer export.

Go example

examples/10-fusedgpu/main.go

Run:cd welvet/examples/10-fusedgpu && source ../env.sh && go run .

package main

import (
	"fmt"

	"github.com/openfluke/welvet/fusedgpu"
	"github.com/openfluke/welvet/model/transformer"
)

func main() {
	m, err := transformer.LoadEntity("model.entity")
	if err != nil {
		fmt.Println("need an ENTITY file:", err)
		return
	}
	spec, err := m.ExportFusedGPUSpec()
	if err != nil {
		panic(err)
	}
	eng, err := fusedgpu.NewFromSpec(spec)
	if err != nil {
		panic(err)
	}
	defer eng.Close()
	logits, err := eng.AppendTokens([]uint32{1, 2, 3})
	fmt.Println(len(logits), err)
}

Output

exit 0 · last run via go run .

need an ENTITY file: open model.entity: no such file or directory