10. fusedgpu — decoder on device

Part: II · Foundation
Package: github.com/openfluke/welvet/fusedgpu
Status: ok — ✅

When

You need the fusedgpu foundation package.

Where

import "github.com/openfluke/welvet/fusedgpu"

cd 10-fusedgpu && source ../env.sh && go run .

Why

Token-by-token host round-trips kill decode. A fused engine keeps weights and scratch resident for Q4_0 and BinaryG128 hybrid paths.

What

Engine from Spec (AppendTokens/Reset/Close); HybridEngine (PrefillSample/DecodeSample/DecodeChunk). Specs come from model/transformer export.

Sample output (captured)

need an ENTITY file: open model.entity: no such file or directory

Live capture from go run ./cmd/runall on local ../../welvet (exit 0).

Source

Copied from the Welvet feature book examples (openfluke.github.io/welvet/examples/10-fusedgpu).

package main

import (
	"fmt"

	"github.com/openfluke/welvet/fusedgpu"
	"github.com/openfluke/welvet/model/transformer"
)

func main() {
	m, err := transformer.LoadEntity("model.entity")
	if err != nil {
		fmt.Println("need an ENTITY file:", err)
		return
	}
	spec, err := m.ExportFusedGPUSpec()
	if err != nil {
		panic(err)
	}
	eng, err := fusedgpu.NewFromSpec(spec)
	if err != nil {
		panic(err)
	}
	defer eng.Close()
	logits, err := eng.AppendTokens([]uint32{1, 2, 3})
	fmt.Println(len(logits), err)
}