THE OPEN NOTEBOOK

A book for the curious.

Name: The Welvet feature book
Author: Samuel Watson

70 chapters inside the Welvet AI engine. Start with why it exists, then follow the ideas into layers, learning, and the systems that connect them.

New to the work? Start with Welvet in plain language, then read What Welvet is. Code examples and reported results below are preserved from the source edition. Read the complete book on one page →

Explore:Everything I · Orientation II · Foundation III · Layers IV · Runtime V · Systems VI · Model IO VII · Apps VIII · Stubs IX · Validate

18 entries · Clear filters

III · Layers

layers/dense — MatVec microkernel

Most FLOPs are W@x. One Dense stack owns FormatNone×34 and all quants × three backends so every composite proj shares one correctness surface — including native in-dtype SGD.

III · Layers

layers/mha — attention

Transformers need multi-head attention with masks, RoPE/ALiBi, GQA/MQA, and cross-attn — without forking MatVec for every projection.

III · Layers

layers/swiglu — gated FFN

Modern decoder FFNs are SiLU(gate)⊙up → down. Projections must share Dense’s quant/backend matrix.

III · Layers

layers/rmsnorm

Llama-style blocks normalize by RMS, not mean+var. Needs native fwd/bwd and WebGPU shaders.

III · Layers

layers/layernorm

Classic mean+var normalization with γ/β — still required for many HF architectures.

III · Layers

layers/embedding

Token IDs must gather rows from a table — not a Dense MatVec — with scatter grads on backward.

III · Layers

layers/softmax

Classification heads and attention need stable softmax variants, including sparse/Gumbel/Entmax for research paths.

III · Layers

layers/sequential

Some cells need an ordered Dense chain without burning grid hops.

III · Layers

layers/residual

Skip connections stabilize deep stacks: y = F(x) + x with correct skip grads.

III · Layers

layers/cnn1 · cnn2 · cnn3

Conv nets must sit on the same dtype×quant×backend matrix as Dense. im2col → Dense GEMV is the intentional first cut.

III · Layers

layers/rnn · lstm

Sequence models before transformers still need vanilla RNN and LSTM with BPTT on the shared MatVec stack.

III · Layers

layers/seqmix — mixer contract

Attention, SSM, linear attn, and conv mixers must not be accidental forks of mha. Naming the contract keeps packages honest.

III · Layers

layers/gdn — gated delta net

Linear attention / decode-first mixers (Gated DeltaNet) need a first-class package under KindLinearAttn.

III · Layers

layers/mamba — selective SSM

SSM mixers (KindSSM) are not MHA clones — they need their own selective-scan path.

III · Layers

layers/convt1 · convt2 · convt3

Generators and U-Nets need transposed convolution as a peer of CNN, on the same Dense proj surface.

III · Layers

layers/kmeans

Soft clustering as a differentiable layer lets topology experiments sit inside the same train loop.

III · Layers

layers/parallel — MoE + cameral

Mixture-of-experts and multi-path cells need concat/add/avg/filter combines. Cameral graphs need sibling hemispheres that share input, merge outputs, and optionally train under distinct mode

III · Layers

layers/metacognition

Observed layers can apply heuristic stability rules (gate/scale/reset) without dtype morph/QAT.