14. layers/rmsnorm

Part: III · Layers
Package: github.com/openfluke/welvet/layers/rmsnorm
Status: ok — ✅

When

Building or training a net that needs the layers/rmsnorm Op (also usable as a Parallel cam).

Where

import "github.com/openfluke/welvet/layers/rmsnorm"

cd 14-rmsnorm && source ../env.sh && go run .

Why

Llama-style blocks normalize by RMS, not mean+var. Needs native fwd/bwd and WebGPU shaders.

What

Per-token RMS + γ on weights.Store; WebGPU fwd+bwd; SIMD DotTile stats + host scale.

Sample output (captured)

[0 0 0 0] <nil>

Live capture from go run ./cmd/runall on local ../../welvet (exit 0).

Source

Copied from the Welvet feature book examples (openfluke.github.io/welvet/examples/14-rmsnorm).

package main

import (
	"fmt"

	"github.com/openfluke/welvet/core"
	"github.com/openfluke/welvet/layers/rmsnorm"
	"github.com/openfluke/welvet/quant"
)

func main() {
	gamma := []float32{1, 1, 1, 1}
	l, err := rmsnorm.NewConfigured(rmsnorm.Config{Dim: 4}, core.DTypeFloat32, quant.FormatNone, gamma)
	if err != nil {
		panic(err)
	}
	x := core.NewTensor[float32](1, 4)
	_, y, err := rmsnorm.Forward(l, x)
	fmt.Println(y.Data, err)
}