← OpenFluke

III · Layers

Chapter 27

layers/parallel — MoE combine

github.com/openfluke/welvet/layers/parallel🚧

Why it exists

Mixture-of-experts and multi-path cells need concat/add/avg/filter combines over branches.

What it is

Dense branches today; filter gate mode for soft MoE. Smoke+census. Heterogeneous residual graft still open.

Go example

examples/27-parallel/main.go

Run:cd welvet/examples/27-parallel && source ../env.sh && go run .
package main

import (
	"fmt"

	"github.com/openfluke/welvet/core"
	"github.com/openfluke/welvet/layers/parallel"
)

func main() {
	l, err := parallel.New(parallel.Config{
		Dim: 16, Branches: 2, OutFeat: 16, Combine: parallel.CombineConcat,
	})
	if err != nil {
		panic(err)
	}
	x := core.NewTensor[float32](1, 16)
	_, y, err := parallel.Forward(l, x)
	fmt.Println(y.Shape, err)
}

Output

exit 0 · last run via go run .

[1 32] <nil>