# OpenFluke — full corpus > Complete OpenFluke website text and Loom documentation for LLM ingestion. Navigation index: https://openfluke.com/llms.txt ## Site pages (full text) --- Source: https://openfluke.com/ # OpenFluke — Sovereign AI on Your Hardware (Golang AI Engine) > OpenFluke builds Loom, a pure Golang AI engine (Apache 2.0, zero CGO) for CPU, GPU, and WebGPU on every OS; SoulGlitch, a private offline AI digital pet on Google Play; and Primecraft, a voxel simulation engine. Canonical: https://openfluke.com/ --- Star openfluke/loom Open-Source AI Infrastructure Lab Sovereign AI. On Your Hardware. OpenFluke builds foundational tools so intelligence can run locally—private, portable, and free of cloud lock-in. Loom (v0.83 Experimental Hardware Acceleration) is the M-POLY-VTD engine: Plan 9 SIMD CPU, WebGPU + Apple Metal, and experimental vendor NPUs (Intel · Qualcomm Hexagon) — plus native .entity checkpoints and welvet bindings on every major OS. SoulGlitch shows what that feels like: a planet-first on-device AI playground — multiverse home, chat, swarm vote, and physics demos that never phone home. Explore Loom Why Loom? Loom docs SoulGlitch on App Store (iOS & Mac) SoulGlitch on Play SoulGlitch for Linux (v2.4.0) Source on GitHub Who We Are An open-source AI infrastructure lab Most AI today lives in someone else's data center. OpenFluke is an independent R&D lab building the opposite: edge-native, privacy-first infrastructure that puts training and inference on the devices people already own. We ship real software—not slide decks. Loom is the engine. SoulGlitch is the proof. Everything is open source so developers, researchers, and hobbyists can inspect, extend, and ship without permission. 100% offline capable Loom Apache 2.0 Phone to server Your data stays yours Why Loom vs the industry No cloud required Run models on laptops, phones, and browsers. No API keys, no subscriptions, no upload pipeline. One engine, many surfaces Drop Loom into Python, JavaScript, Go, Flutter, or WASM—the same weights, the same behavior. Built to be felt SoulGlitch turns abstract ML into something you can walk around: voxel planets, on-device chat, swarm votes, and Jolt physics — all on your phone or PC. The Engine What is Loom? Loom is our Apache 2.0 M-POLY-VTD engine—CPU and GPU capable, OS-agnostic, and designed like SQLite for neural networks : one library you embed, not a cloud you rent. v0.83 pushes Experimental Hardware Acceleration — Plan 9 SIMD across ARM & x86, Apple Metal zero-copy on Apple Silicon, and Qualcomm Hexagon NPU offload with a Drift Spectrum. Prior: v0.81 Accelerator Bridge (Intel OpenVINO), v0.80 native .entity checkpoints , WebGPU v29 , and welvet on PyPI and npm. The Loom runtime — runs everywhere Silicon & acceleration x86_64, ARM64, ARMv7 CPU inference & training (every OS) GPU via WebGPU / native paths Vendor NPU — Intel OpenVINO · Qualcomm Hexagon (experimental) Apple Metal / MPSGraph (macOS, experimental) Plan 9 SIMD — AVX2 / NEON WebAssembly in the browser Operating systems Windows macOS & iOS Linux & Android Node.js, Bun, browsers 📦 Drop-in portability Prebuilt native libraries ( .dll , .so , .dylib ) install beside your app. Train once, ship everywhere. 💾 Native precision on disk 21 DTypes from Float64 to 1-bit binary—checkpoints store packed weights per layer, not FP32-only JSON. BitNet and Qwen3 load from Hugging Face via Lucy. 🎯 Bit-exact reproducibility Deterministic execution across CPU, GPU, and language bindings—same inputs, same outputs, every time. 🧬 Biological learning 3D volumetric networks with target propagation—layers learn locally without classic backprop lock-in. Deterministic AI on CPU, GPU & NPU Loom is built as a Deterministic Neural Virtual Machine (DNVM) : the same model weights, prompts, and settings produce bit-identical behaviour on Loom CPU across Apple Silicon, x86, and bindings. WebGPU, Apple Metal, and vendor NPU paths are complementary accelerators — Intel / Hexagon / Metal offload validated in Lucy suites; GPU training and inference use WebGPU. Apache 2.0 License Why Loom? Loom overview Read docs Hugging Face · local inference Models Lucy & SoulGlitch support Download checkpoints into your local Hugging Face hub cache. Lucy (CLI) and SoulGlitch share the same approved model list—load safetensors, run chat offline through Loom/welvet. Model family Hugging Face repo Typical use SmolLM2 Lite HuggingFaceTB/SmolLM2-135M-Instruct Phones · fast reactions · default SoulGlitch brain SmolLM2 Balanced HuggingFaceTB/SmolLM2-360M-Instruct Desktop · everyday chat SmolLM2 Deep HuggingFaceTB/SmolLM2-1.7B-Instruct Stronger private hardware · deeper replies BitNet b1.58 microsoft/bitnet-b1.58-2B-4T Low-bit ternary weights · Loom v0.78+ infer · v0.80 .entity native ship Qwen3 Lite Qwen/Qwen3-0.6B GPU-friendly · strong quality per GB Qwen3 Balanced Qwen/Qwen3-1.7B Desktop · sharded safetensors Qwen3 Heavy Qwen/Qwen3-4B Large GPU / patient downloads Custom volumetric networks (XOR, NEAT, DNA splice) are built in Loom/poly—no HF download required. What We Build The OpenFluke ecosystem Infrastructure, a flagship app, and community tools—one vision of local, sovereign AI. Loom v0.83 — Hardware Acceleration M-POLY-VTD engine: 3D grids, Plan 9 SIMD, Apple Metal, Qualcomm Hexagon NPU, native .entity checkpoints, WebGPU, BitNet on CPU, NEAT/DNA evolution. Apache 2.0 — train once, ship in Python, JS, Go, Flutter, or the browser. Why Loom? Documentation Loom overview SoulGlitch iOS, Android, Linux & macOS Planet-first on-device AI playground — My Planet voxel multiverse, Construct TCP host, planet mini chat, full AI Chat (Glitch Threads + document libraries), swarm vote, Character Creator, scene gallery, AI Models pipeline, LAN model sync, ISO Telemetry, Glitch Link mesh, and TANHI. Live on the App Store (iOS & Mac), Google Play (Android), and Linux direct download. Windows x64 & ARM64 builds are internal testing only. App Store — iOS & Mac Get on Google Play Linux tar.gz (v2.4.0-b4) Scene Gallery Live Screenshots and voxel scenes from SoulGlitch and the Primecraft engine—see the worlds we're building. Open gallery For developers Get started in minutes Loom is a self-contained C-ABI library ( welvet ) you embed in any stack. v0.83 adds Plan 9 SIMD, Apple Metal, and Qualcomm Hexagon NPU offload (experimental Drift Spectrum). v0.80 ships .entity native checkpoints and welvet on PyPI/npm. CPU train/save/reload bedrock from v0.79 still gates every release. One model graph — Loom CPU on every OS, WebGPU for GPU, vendor plugins for NPUs where available. Install via pip install welvet , npm install @openfluke/welvet , welvet: ^0.80.4 on pub.dev for Flutter, or embed natives in a custom app—as SoulGlitch does. Full reference: openfluke.com/docs . Python JavaScript Flutter Go iOS / Android WebAssembly WebGPU Loom documentation Product page Python Node.js Flutter Go # Install pip install welvet # XOR in 10 lines from welvet import Network, train net = Network({ "id": "xor", "depth":1,"rows":1,"cols":1, "layers_per_cell":2, "layers": [ {"l":0,"type":"dense","input_height":2, "output_height":8,"activation":"relu"}, {"l":1,"type":"dense","input_height":8, "output_height":1,"activation":"sigmoid"} ] }) losses = train(net, [[[0,0],[0,1],[1,0],[1,1]]], [[[0],[1],[1],[0]]], epochs=100, learning_rate=0.1) print(f"Final loss: {losses[-1]:.4f}") # Install npm install @openfluke/welvet # Usage const { Network } = require('@openfluke/welvet'); const net = new Network({ id: "demo", depth:1, rows:1, cols:1, layers_per_cell: 1, layers: [{ l:0, type:"dense", input_height:4, output_height:2 }] }); // Same model, same weights, identical output // whether running in Node or a browser. # pubspec.yaml dependencies: welvet: ^0.80.4 # Dense forward (CPU MC) import 'package:welvet/loom_ffi.dart'; final handle = loomLib.createNetwork(jsonConfig); loomLib.configureTrainingMode(handle, 2); final out = loomParseFloatArray( loomLib.forwardPolymorphic(handle, input, [1, 16]), ); # 30+ demos: loom-flutter-quickstart on GitHub // go get github.com/openfluke/loom/poly package main import ( "fmt" "github.com/openfluke/loom/poly" ) func main() { net := poly.BuildNetwork(poly.Config{ ID: "demo", Depth:1, Rows:1, Cols:1, LayersPerCell: 1, Layers: []poly.LayerDef{ {L: 0, Type: "dense", InputHeight: 4, OutputHeight: 2}, }, }) state := net.NewState(poly.Float32) state.SetInput([]float64{1,0,1,0}) state.Step() fmt.Println(state.Output(0)) } Deeper dive How Loom differs architecturally Independent analysis of Loom's 3D volumetric design, compression pipeline, and target-propagation learning— for readers who want the technical story behind the marketing. Architecture reference: docs overview · comparative analysis on the research page . Technical research Loom: 3D grids & target propagation Comparative analysis vs PyTorch, JAX, and Go ML stacks—architecture, DNVM determinism, and edge deployment. Read full analysis 🧊 Thinks in 3D Signals move across a volumetric grid—not only through a rigid layer stack—closer to spatial brain topology than a factory line. 💾 Up to 98.4% compression Bit-packed serialization from Float64 down to 1-bit binary—gigabyte-class models can shrink enough to run on a phone, offline. 🧬 Target propagation Layers learn independently via localized target signals—more biologically plausible, and viable on non-differentiable low-bit models. ⚡ BitNet on device v0.80 ships native .entity checkpoints end-to-end—Lucy, SoulGlitch, and welvet infer BitNet-class models on packed CPU paths without a cloud API. App Store — iOS & Mac SoulGlitch — your planet, local AI, chaos physics Splash into My Planet — orbit a voxel multiverse, possess buddies, and explore biomes. Talk over the world with planet mini chat , open full AI Chat with Glitch Threads and document libraries, ask a swarm to vote yes/no/maybe, forge characters in the Creator, download and convert AI brains, sync models over LAN, and wreck stuff in 29 physics demos . Powered by Loom — no cloud required. Download now on the App Store (iPhone, iPad & Mac — same link), Google Play (Android), or Linux x86_64 (v2.4.0 build 4). Windows x64 & ARM64 are internal testing only — no public download yet. App Store (iOS & Mac) Get on Google Play Linux (v2.4.0-b4) Learn more App Store · iOS & Mac Google Play · now Linux · now Windows · internal testing Construct TCP showcase — all 13 Go examples, rigid-body physics, and first-person planet walk on macOS. Tutorials → Loom × SoulGlitch — models run on your PC; TANHI streams execution live to your phone so you watch mixed layers and remote links in 3D, in real time. SoulGlitch trailer — private AI companion on your hardware. × ‹ › More open source Built alongside the ecosystem Utilities and experiments from the same lab. Open source · NLP tool TokenTrove Find recurring text patterns across millions of documents—n-gram chains, file-level tracking, and parallel processing. Shown here on 5,000+ FCC filings to surface common boilerplate. Built in Go + Fiber. openfluke/tokentrove Pattern mining at scale Linked n-gram chains across thousands of files—not just word counts, but multi-sentence recurring structures. Built for real corpora Web UI, parallel processing, numeric filtering—legal docs, filings, research sets, plagiarism workflows. Pure Go Same stack as the rest of OpenFluke. Drop it on any server. Join the project Loom is Apache 2.0 and fully open source. Stars help others discover it; issues and PRs shape what ships next. Contribute on GitHub Read the docs → · API reference → --- Source: https://openfluke.com/about # About Samuel Watson — Golang AI Engineer & OpenFluke Founder > Samuel Watson is a Golang AI systems engineer and founder of OpenFluke. Programming since 2006, Master of Applied AI (Deakin). Building Loom (pure Go AI), Primecraft, and SoulGlitch. Canonical: https://openfluke.com/about --- Samuel Watson AI Systems Engineer & Founder of OpenFluke — Melbourne, Australia Programming since 2006 Master of Applied AI AI Runtime Engineering openfluke planetbridging LinkedIn About Me The short version I've been programming since 2006 — starting with IT support and small automation scripts before working my way through web development, data engineering, systems programming, and eventually AI runtime research. Over the years I've written production code in Go, Python, Java, JavaScript, TypeScript, C#, VBA, R, PHP, and Shell. In my spare time I build OpenFluke — a passion project I work on for fun. It's centred around Loom , a portable AI engine that runs neural networks natively across every major platform and language without vendor lock-in. Alongside it I'm building Primecraft , a simulation engine, and SoulGlitch , an AI creature evolution game powered by both. All of it built in my own time, just because I enjoy it. I've designed and verified cross-language, cross-vendor AI runtimes — achieving bit-level determinism across 7+ architectures including Apple M4, AMD Ryzen, Intel Arc, NVIDIA, and Qualcomm Adreno — using WebGPU/Vulkan compute with unified C-ABI bindings for Go, Python, C#, C, and WebAssembly. Languages & Technologies Accumulated across ~20 years Go Python JavaScript TypeScript C# Java C / C-ABI HTML / CSS SQL VBA / Excel R PHP Education Formal qualifications Master of Applied Artificial Intelligence Deakin University — 2023 to 2025 AQF Level 9 ACS Accredited Seoul Accord Deep Learning Reinforcement Learning Computer Vision Bachelor of Information Technology Griffith University — 2019 to 2021 AQF Level 7 Systems Development IT Project Management Security Policy Certifications Industry credentials Microsoft Certified: Azure AI Fundamentals Microsoft Current Projects What I'm building at OpenFluke Loom A portable, cross-language AI engine that runs neural networks natively across Go, Python, C#, TypeScript, WebAssembly, and more — without vendor lock-in. Learn more → Primecraft A distributed simulation engine with procedural world generation, physics, and embedded neural AI. Available on Android, Windows, Linux, and Steam. Learn more → SoulGlitch An AI creature evolution game built on Primecraft and powered by Loom. Train neural networks through gameplay. In active development. Learn more → Portfolio Demos Selected project video demonstrations Flamekeeper · RAG / LLM Local Offline ChatGPT-like System Built from scratch: offline conversations with RAG-based AI, voice input, TTS, natural language recommendations, and local vector search. Dockerized microservices with React UI. React · GoFiber · FastAPI · Docker · MongoDB · Ollama · Tacotron2 Bampro · MARL / Distributed AI WebGPU-Agnostic AI Framework End-to-end simulation of a WebGPU-agnostic AI framework designed for horizontal scaling in distributed environments. Multi-agent RL with evolutionary neural architecture selection and real-time dashboards. Go · Fiber · Docker Compose · WebSockets · MARL · React Deakin University · Team Lead Game Dev — Neurodiversity & Accessibility Served as team leader for a game development project at Deakin University, focusing on neurodiversity, accessibility, and innovative thinking in an inclusive educational platform. Geolocation · OpenStreetMap Australia Open-Source Lots Map Integration of OpenStreetMap to display geolocation data for all open-source lots across Australia on an interactive web interface. GitHub Portfolio Selected open-source projects — github.com/planetbridging & github.com/openfluke 🔥 Flamekeeper Multimodal RAG system: speech recognition, TTS, embeddings, and local LLM inference. Dockerized microservices with React UI for document ingestion and vector search. React · GoFiber · FastAPI · Ollama · Tacotron2 🤖 Bampro Multi-agent reinforcement learning experiments in a 3D simulation environment. Evolutionary neural architecture selection, real-time dashboards, low-spec cloud orchestration. Go · Fiber · Docker · WebSockets · MARL · React 🌐 Biocraft Isomorphic physics + AI sandbox running both natively and in browser. JSON-driven scene import/export, player-to-policy training, GPU-accelerated inference, multi-server monitoring. Go · WebGPU · Jolt Physics · Three.js · WebAssembly 🕸️ 3D Permission Dendrogram Interactive 3D visualization of hierarchical permission trees — streamed live from a Go backend and rendered in React Three Fiber with WebGL. Go · WebSockets · React Three Fiber · WebGL · Docker 🔐 CyberSentry Series Cybersecurity dashboard suite for CVE/CPE lookups and vulnerability enumeration. Real-time APIs with caching layers across MySQL and MongoDB. TypeScript · React · Bun · Docker · MySQL · MongoDB 🌉 Bridgeware Real-time microservice framework for secure CPE lookup and encrypted client-server messaging with a React dashboard. Node.js · React · Express · Socket.IO · Docker · Chakra UI 🎵 Audio Labeling Pipeline Full-stack ML pipeline for audio data: hierarchical labeling, spectrogram generation, neural architecture search, model training, and secure auth. React · Node.js · Flask · TensorFlow · MongoDB · Docker 🚁 DJI Tello Autonomous Flight Computer vision-guided robotics: CNNs trained to recognize individuals and trigger autonomous drone flight sequences via the DJI Tello SDK. Python · TensorFlow · OpenCV · Keras · ffmpeg 🐾 Paws Network packet capture and analysis tool in Go — goroutine-based sniffing, REST endpoints, and a responsive web dashboard for traffic inspection. Go · gopacket · pcap · Bootstrap 🔍 TokenTrove N-gram chain discovery across millions of documents. Parallel processing, file-level pattern tracking, web UI with real-time stats. Tested on 5,000+ FCC legal filings. GitHub → Additional Projects 🏈 AFL Match Prediction (Flask · TensorFlow · Pandas) 📊 Steam vs Android Trends Dashboard (Python · Chart.js) ⚙️ Ansible VMware/vSphere Examples 🗄️ CSV-to-SQL Converter (C#/.NET WPF) 🧟 Zombie Apocalypse Simulation (Node.js · Socket.IO) --- Source: https://openfluke.com/loom # Loom — Golang AI Engine, SIMD · Metal · Hexagon (v0.83) > Loom is a pure Golang AI engine / DNVM: Plan 9 SIMD CPU, Apple Metal, Qualcomm Hexagon NPU, WebGPU, and Intel OpenVINO (v0.83 Experimental Hardware Acceleration). .entity checkpoints, welvet on PyPI/npm/pub.dev, 21 dtypes, Lucy validation, BitNet CPU. Canonical: https://openfluke.com/loom --- Open Source · github.com/openfluke/loom The Universal AI Engine M-POLY-VTD — a sovereign Deterministic Neural Virtual Machine in Go: 3D volumetric grids, 21 numeric types, and polyglot bindings ( welvet ) for Python, TypeScript, Flutter/Dart (pub.dev), and WASM. One graph, multiple paths: Plan 9 SIMD CPU , WebGPU / Apple Metal , and vendor NPUs (Intel · Qualcomm Hexagon) — experimental, with measured drift spectra. v0.83.0 — Experimental Hardware Acceleration Plan 9 SIMD Apple Metal Hexagon NPU Read the docs Planet Bridging GitHub v0.83 release v0.83.0 Experimental Hardware Acceleration v0.83 — Plan 9 SIMD · Apple Metal · Qualcomm Hexagon · Drift Spectrum Plan 9 SIMD — raw AVX2 / NEON across the engine · no CGO · no CUDA / OpenBLAS stack Apple Metal — zero-copy unified memory on Apple Silicon · Lucy [13] Hexagon NPU — Qualcomm QNN plugin · Drift Spectrum ( EXACT · INDUS · LOWBIT · H-DRIFT ) Lucy seven-layer — .entity save/reload · SC/MC train · multi-dtype determinism Research PDF + podcast · GitHub assets v0.83 · Experimental Hardware Acceleration Watch Loom hit SIMD, Metal & Hexagon The v0.83 release story: Plan 9 SIMD without CGO, Apple Metal zero-copy on Apple Silicon, and Qualcomm Hexagon NPU offload with the Drift Spectrum — sovereign DNVM on divergent hardware. v0.83 docs Research PDF + audio YouTube → AI Deep Research Independent AI Analysis of Loom Comparative research on M-POLY-VTD vs PyTorch and JAX — plus the full engine reference on this site, synced from loom/docs . Architecture & research 3D grids, target propagation, DNVM Start with the overview — volumetric dispatch, WeightStore morphing, step mesh, transformers, v0.83 Experimental Hardware Acceleration (SIMD · Metal · Hexagon), v0.81 Accelerator Bridge , and v0.80 Native Ship ( .entity checkpoints). Why Loom? All Loom docs Planet Bridging Research write-up 🧊 AI that thinks in 3D Most AI frameworks process data in a straight line, like an assembly line. Loom uses a three-dimensional grid — more like how your brain's neurons actually connect, jumping across regions rather than always going layer by layer. 💾 Fits AI on a USB stick Loom can compress AI models by up to 98.4%. A model that normally takes gigabytes of storage can shrink to a fraction — small enough to run on a phone or an old laptop with no internet required. 🧬 Learns like biology, not math Traditional AI learning requires freezing everything to calculate one massive equation. Loom's Target Propagation lets each part of the network learn independently — more like how neurons fire and strengthen in a real brain. Read the Full Technical Breakdown For Non-Technical People What is Loom, exactly? "Think of Loom like SQLite — but for AI." SQLite is a tiny database that runs inside your app with no server needed. Loom is the same idea for neural networks: a self-contained engine you can drop into any project, on any device, with no cloud account, no GPU server, no complicated setup. 🧠 Train it like a brain A neural network learns by seeing examples — like showing a child thousands of pictures of cats until they know what a cat is. Loom provides all the tools to build and teach these networks. 📦 Pack it anywhere Once trained, your model is a tiny file. Drop it into your Python script, your phone app, your website, or a game engine. Loom runs it everywhere with the exact same output. 🔒 No cloud needed Unlike ChatGPT or other AI services, Loom runs 100% locally on your device. Your data never leaves your machine. Perfect for privacy-sensitive apps or offline use. ⚡ CPU · SIMD · GPU · NPU Loom CPU runs on every OS and binding — with Plan 9 SIMD (AVX2 / NEON) from v0.82/v0.83. WebGPU and Apple Metal hit the GPU; vendor plugins offload layers to Intel OpenVINO and Qualcomm Hexagon NPUs (experimental), with measured Drift Spectrum parity. Accelerators guide → 🌍 Every language Python developer? pip install welvet . JavaScript? npm install @openfluke/welvet . Flutter? welvet: ^0.80.4 on pub.dev — SoulGlitch and loom-flutter-quickstart prove it ships. Go, C, C#, Rust? There are bindings for all of them. One model, every language. 🎯 Deterministic on CPU & GPU Loom's Deterministic Neural Virtual Machine (DNVM) delivers bit-identical behaviour across Apple Silicon, x86, WebGPU, and language bindings. Lucy and SoulGlitch depend on this for reproducible local inference. 🧬 Evolution built in Loom includes a full NEAT evolution engine — models can mutate and breed like living organisms. This powers SoulGlitch's creature evolution system. Lucy & SoulGlitch Supported Hugging Face models Approved checkpoints share the same list in loom/lucy and SoulGlitch—download once, run offline via welvet. SmolLM2 135M · 360M · 1.7B Instruct — mobile to server brains Qwen3 0.6B · 1.7B · 4B — GPU-friendly chat models BitNet b1.58 microsoft/bitnet-b1.58-2B-4T — packed ternary CPU path (v0.78+ infer · v0.79+ native save/reload · v0.80 .entity ship) Plus custom Loom/poly networks (training, NEAT, DNA) with no HF download. Flutter Run Loom in Flutter apps welvet on pub.dev is an FFI plugin — same C-ABI as Python and npm. Desktop natives ship in the package; SoulGlitch proves production mobile. 📦 pub.dev install welvet: ^0.80.4 in pubspec.yaml . No monorepo path. loomAvailable + loomLib on first FFI call. Package page → 🧪 loom-flutter-quickstart 30+ interactive CPU demos: dense/MHA forward, train, mesh step, DNA, checkpoints, seven-layer suites. Clone & run → 📖 Docs & snippets Forward, morph, LoomTrain , backward, .entity wire, LLM exports. Flutter guide → · Deployment → Get Started Install in 30 seconds Pick your language and paste the command. No account required. Python Node.js Flutter Go WebAssembly $ pip install welvet Copy Ships with precompiled native libraries for Windows, Linux, macOS, iOS, and Android. Zero Python dependencies. PyPI page → $ npm install @openfluke/welvet Copy Works in Node.js and browsers via WebAssembly. npm page → welvet: ^0.80.4 Copy Add to pubspec.yaml — FFI plugin with desktop natives on pub.dev. pub.dev → · Flutter docs → · Example app → import 'package:welvet/loom_ffi.dart'; loomLib.configureTrainingMode(handle, 2); final out = loomParseFloatArray( loomLib.forwardPolymorphic(handle, input, [1, 16]), ); $ go get github.com/openfluke/loom/poly Copy Pure Go module. No CGO. Works with standard go build . Quick reference → · Source → Download main.wasm from the releases page Download 6.9 MB WASM bundle. Drop into any web page and run Loom in the browser. All releases → Platform Support Runs everywhere Prebuilt native libraries for every major platform — just download and go. Windows x86-64, ARM64 Linux x86-64, ARM64, ARM v7, x86 macOS x86-64, ARM64 (M-series), Universal Android ARM64, x86-64 iOS ARM64, Simulator, XCFramework WebAssembly Browser + Node.js Vendor NPU Intel · Qualcomm Hexagon · experimental · Drift Spectrum Apple Metal MPSGraph · macOS Apple Silicon · Lucy [13] Plan 9 SIMD AVX2 · NEON · no CGO WebGPU Forward + Backward pass, 17×–65× speedup Flutter / Dart welvet on pub.dev · desktop natives · SoulGlitch + quickstart demos PyPI welvet — zero dependencies For Developers What's under the hood Loom isn't just a wrapper around PyTorch. It's a ground-up engine built for portability and precision. All major layer types Dense, MHA, SwiGLU, RMSNorm, LayerNorm, CNN 1D/2D/3D, Transposed Conv, RNN, LSTM, Embedding, KMeans, Softmax, Parallel, Sequential, Residual. 21 numeric types float64 all the way down to binary (1-bit), including fp8, fp4, int4, and ternary. Choose precision vs. model size at runtime. NEAT evolution + DNA A full neuroevolution engine with mutation, crossover, and fitness selection. Models have a "DNA" signature for reproducible evolution. 98.4% compression Native bit-packed serialization shrinks model files by 98.4% compared to raw float storage. Plus SafeTensors support for HuggingFace compatibility. Target propagation An alternative to backpropagation where each layer is given a direct target. More biologically plausible and works for non-differentiable layers. Step mesh engine Clock-cycle 3D grid with double-buffered layers, spatial remote links, BPTT, and neural target propagation — online learning without a rigid layer stack. BitNet & low-bit CPU BitNet b1.58–style checkpoints with packed ternary linear layers. Lucy pulls from Hugging Face; welvet C-ABI exposes CPU inference paths. Operation mesh Donate Compute (LAN TCP model sharing), TANHI UDP layer telemetry for SoulGlitch HUD, tiled forward/backward, and Qwen3-family HF ingest. Full documentation Deployment guide BitNet CPU Watch It Work See Loom In Action Live demos — TANHI telemetry to SoulGlitch, WebGPU benchmarks, and 3D visualization. Loom × SoulGlitch · live TANHI × Regional Mix — models on your PC, view on your phone Watch Loom AI models run in real time on a regional_mix harness (Dense, MHA, SwiGLU, RNN, LSTM with remote links across 3D topologies). Execution streams over UDP as TANHI telemetry into SoulGlitch on your local phone — a spatial, time-scrubbable trace instead of numbers in a terminal. TANHI docs → · YouTube → Performance Benchmark Forget Llama.cpp: WebGPU Inference in Pure Go SmolLM2-135M benchmarks: 68 tok/s on RTX 1650 Super, 143 tok/s on Linux i5, 229 tok/s on Mac M4. Zero CGO. FlashPoly Tiling. Bit-level deterministic across OS boundaries. Visualization Loom: Visualizing 3D Neural Networks in Real-Time Watch the AI "think" in real-time. Stepping mode, 3D grid topology, Zig-Zag and Starburst routing patterns — the black box, opened. Android · Airplane Mode Offline LLM Inference on Android via Loom AI Loom v0.0.8 running 100% locally on Android — device locked in Airplane Mode throughout. Zero cloud dependency. Pure on-device compute from first principles in Go. Open Source Tool NeuralWave: 3D Neural Network Visualization & Weight Analysis Real-time model discovery from HuggingFace, interactive 3D layer inspection, attention head visualization. Built on Loom + Go backend + Three.js. Star Loom on GitHub Loom is free, open-source, and built in the open. Stars help others find it and fuel continued development. Star openfluke/loom Report an Issue Star Fork --- Source: https://openfluke.com/why-loom # Why Loom? v0.83 SIMD, Metal, OpenVINO & Edge AI — OpenFluke > Why choose Loom v0.83: pure Golang AI (Apache 2.0, zero CGO), Plan 9 SIMD, Apple Metal, Intel OpenVINO NPU, Qualcomm Hexagon, Drift Spectrum parity, dense mid-stream adaptation, offline DNVM, 21 dtypes, WebGPU — vs PyTorch, llama.cpp, and cloud AI. SoulGlitch proof. Canonical: https://openfluke.com/why-loom --- v0.83 · Plan 9 SIMD · Metal · OpenVINO · Hexagon Why Loom vs the rest of AI Cloud chatbots rent intelligence. PyTorch rents a Python runtime. Loom v0.83 is a pure Go AI engine with measured hardware acceleration—Plan 9 SIMD (AVX2/NEON), Apple Metal, Intel OpenVINO NPU, and experimental Qualcomm Hexagon—plus the Drift Spectrum parity receipts. Shipped in SoulGlitch on real phones. v0.83 benchmarks Interactive 3D v0.83 release Research PDF Skip to comparisons ↓ What you get The OpenFluke stack Not a single API—an open-source AI infrastructure lab : engine, bindings, docs, and products built on the same runtime. Loom (Apache 2.0) v0.83 M-POLY-VTD engine: Plan 9 SIMD, Metal, OpenVINO, Hexagon bridge, 21 dtypes, WebGPU, BitNet CPU, C-ABI welvet . Polyglot bindings Python, TypeScript/npm, Go, Dart, C#, Java, WASM—one engine, same weights, embed like SQLite for neural nets. SoulGlitch (product) Offline AI planet sandbox on App Store & Google Play — swarm polls, on-device chat, SIMD inference. Living proof of Loom on phones. Primecraft + lab tools Voxel simulation with embedded AI, scene gallery, Lucy CLI for local HF models—same sovereignty story. Open source means Loom: source, license, and rebuildable natives on GitHub. Releases ship prebuilt .so / .dylib / wheels so you don't have to compile Go—same pattern as PyTorch pip wheels or llama.cpp binaries. SoulGlitch is a product on Google Play (app code not necessarily OSS). Model weights come from Hugging Face under their own licenses. Advantages What Loom does differently Compared to cloud AI, Python frameworks, LLM-only runners, and other Go ML libraries. Sovereign & offline No API keys. Prompts and training stay on your hardware—privacy by architecture, not policy PDFs. Pure Go, zero CGO Golang AI without a Python runtime or CUDA-only trap. Single-binary deployment story for edge and servers. 3D volumetric mesh Networks as spatial grids—not only nn.Sequential . Native target propagation and step mesh learning. 21 dtypes + BitNet Float64 down to 1-bit binary per layer. Native .entity checkpoints with verified JSON + binary save/reload (v0.80). BitNet b1.58 on CPU since v0.78. DNVM determinism Bit-identical behaviour across CPU, WebGPU, and bindings—reproducible research and embedded systems. WebGPU everywhere Cross-vendor GPU: Windows, Linux, macOS, Android, browser—without shipping CUDA toolchains per platform. Plan 9 SIMD (v0.83) Raw AVX2 / NEON across the engine — no CGO, no CUDA stack. Dense forward paths up to 3.7× throughput vs scalar on the same CPU. Vendor accel bridge poly/accel dispatches to Apple Metal, Intel OpenVINO NPU, and Qualcomm Hexagon — each cell measured with the Drift Spectrum parity buckets. Mid-stream adaptation Dense poly training paths flip chase → avoid → chase in 0 s with StepTweenChain — built for embodied agents, not offline-only convergence. DNA & NEAT built-in Topological comparison of whole networks, evolution in-engine—not just weight checkpoint diffing. Shipped proof SoulGlitch on App Store & Google Play — swarm polls, on-device chat. Living proof of local Loom on phones. v0.83 shipped Measured on real hardware DispatchLayer benchmarks from OpenFluke lab runs — Loom CPU vs Apple CPU/Metal, Intel CPU/OpenVINO NPU, and Qualcomm Hexagon. Plus dense mid-stream adaptation with Plan 9 SIMD forward paths. Release: v0.83 · docs · research PDF . 3.7× SIMD dense throughput (NormalBP actions/s) 7× Hexagon NPU vs Loom CPU (large MatMul FP32) 21× Intel NPU vs Loom CPU (large Conv2D INT8) 4.6× Metal GPU vs Loom CPU (large MatMul FP16) 51% StepTweenChain avg accuracy (chase/avoid flips) 💎 EXACT Infer-repeat parity (Apple · Intel · Qcom) Plan 9 SIMD + dense adaptation 6-layer dense net, 15 s timeline: chase 5 s → avoid 5 s → chase 5 s. SIMD links AVX2/NEON for forward passes; training/backward stay scalar. Training path SIMD Throughput Avg accuracy Task-flip return Deadline hits NormalBP off 9,920 /s 41.1% 159.9% 99.8% NormalBP on 36,349 /s 38.9% 71.2% 100.0% TweenChain off 13,685 /s 40.6% 28.9% 99.9% TweenChain on 36,734 /s 38.3% 1.0% 100.0% StepTweenChain off 1,335 /s 50.3% 87.1% 100.0% StepTweenChain on 1,420 /s 51.3% 99.2% 100.0% Takeaway: SIMD is a straight throughput win for reactive training loops. StepTweenChain wins on goal-switching — instant recovery at both task boundaries (0 s adapt delay), 100% deadline hit rate, best average accuracy. That's the embodied-AI story: chase, avoid, chase — not one static loss curve. Apple Metal (DispatchLayer) Workload (large tensors) Loom CPU Metal GPU speedup Loom CPU vs Apple CPU Metal infer repeat MatMul FP16 2.61 ms 4.6× faster — 💎 EXACT (180/180) MatMul INT8 2.32 ms 4.1× faster — 💎 EXACT ReLU INT8 1.91 ms 6.1× faster 87× faster 💎 EXACT GELU INT8 2.14 ms — 56× faster 💎 EXACT Conv2D FP32 59.3 ms — 4.3× faster 💎 EXACT Loom↔Metal GPU parity: 132/180 cells at industrial grade (✅ INDUS or better). Every Metal infer-repeat forward: 💎 EXACT bit-identical across 180 layer×dtype cells. Intel OpenVINO NPU Workload (large tensors) Loom CPU Intel NPU speedup NPU parity (≤ INDUS) Infer repeat Conv2D INT8 115.6 ms 21× faster 61/90 cells 💎 EXACT (90/90) Conv2D FP32 117.4 ms 21× faster 61/90 💎 EXACT MatMul INT8 4.23 ms 6.1× faster 61/90 💎 EXACT Conv1D INT8 29.1 ms 12× faster 61/90 💎 EXACT MatMul FP16 5.80 ms 8.7× faster 61/90 💎 EXACT Fat matmul/conv layers are where the NPU pays off. Small-tensor latency tax still applies on tiny ops — Loom CPU + SIMD remains the right default for reactive dense loops. Compile once via SyncToAccel , then measure. Qualcomm Hexagon (experimental) Workload (large tensors) Loom CPU Hexagon NPU NPU parity Status MatMul FP32 3.58 ms 7.0× faster 45/54 cells Experimental ReLU INT8 3.52 ms 6.9× faster 45/54 Experimental GELU INT8 3.51 ms 5.6× faster 45/54 Experimental Sigmoid INT8 3.30 ms 6.1× faster 45/54 Experimental GELU INT4 3.51 ms 6.5× faster 45/54 Experimental Hexagon path is experimental — many layer×dtype combos still return ERR while QNN coverage expands. Where it runs: infer-repeat is 💎 EXACT on 54/54 exercised cells. Snapdragon is the phone story; Metal/OpenVINO are the desktop stories. 💎 EXACT — bit-identical repeat infer ✅ INDUS — industrial parity 🟨 LOWBIT — quant band 🟤 H-DRIFT — float drift band Drift Spectrum is Loom's answer to "did the accelerator lie?" Every vendor bridge cell gets a parity bucket — not vibes, not a single cosine-similarity number. Read the full analysis in the v0.83 research PDF and release notes . Vs the industry Quick comparisons Cloud AI (ChatGPT, etc.) Them: Intelligence in their datacenter Loom: Engine in your process Them: No embeddable runtime Loom: C-ABI for your app PyTorch / JAX Them: Python + huge CUDA stack Loom: Go binary, edge-first Them: 1D autograd DAG Loom: 3D mesh + target propagation llama.cpp / Ollama Them: LLM inference focus Loom: Train + small nets + NEAT + DNA Them: GGUF decode excellence Loom: Full engine for products GoMLX / Born ML Them: 1D stacks or OpenXLA/CGO Loom: Zero CGO + WebGPU Them: Narrower scope Loom: DNVM, BitNet, DNA, shipped app Feature matrix Loom vs PyTorch & Go ML (summary) Capability Loom PyTorch / JAX llama.cpp Core language Pure Go (golang AI) Python + C++/CUDA C/C++ Offline / embed First-class (C-ABI, WASM) Possible, heavy Inference-focused Training + custom nets 3D mesh, NEAT, DNA Autograd ecosystem Mostly inference Quantization 21 native dtypes + BitNet CPU TorchAO add-ons GGUF quants GPU path WebGPU + Metal / OpenVINO / Hexagon CUDA / ROCm / TPU CPU/GPU backends Plan 9 SIMD (v0.83) AVX2 / NEON, zero CGO Vendor BLAS / MKL Limited Vendor NPU bridge Metal · OpenVINO · Hexagon + Drift Spectrum ONNX Runtime plugins N/A Determinism (DNVM) Bit-exact claim Not guaranteed Varies Open source Apache 2.0 engine + binaries Framework OSS OSS inference Deep dive: M-POLY-VTD architecture research · docs overview Fit When to choose Loom Choose Loom if you need… Offline AI inside your app (Flutter, Go, WASM) A golang AI / Go ML stack without Python Bit-exact, auditable local inference BitNet or sub-byte models on CPU Metal / OpenVINO / Hexagon with measured parity 3D / NEAT / DNA research in one engine Apache 2.0 you can fork and ship Use something else if you need… Largest cloud models with zero setup (use hosted APIs) Massive PyTorch ecosystem & HF fine-tune recipes day one Fastest GGUF Llama on Mac CPU only (benchmark llama.cpp) Enterprise MLOps (Kubeflow, etc.) out of the box Ready to try the golang AI engine? Star the repo, read the docs, or install SoulGlitch and run models offline today. openfluke/loom Deploy with welvet SoulGlitch Android --- Source: https://openfluke.com/loom/research # Loom M-POLY-VTD — Golang AI Architecture Deep Research > Technical analysis of Loom's pure Go AI stack (M-POLY-VTD): volumetric tensor dispatch, 21-type polymorphism, step mesh, neural target propagation, topological DNA vs PyTorch, JAX, Born ML, GoMLX. Canonical: https://openfluke.com/loom/research --- AI Deep Research · Technical Analysis M-POLY-VTD: The Loom Architecture An exhaustive technical analysis of the Loom framework — covering Volumetric Tensor Dispatch, Multi-Numerical Polymorphism, Systolic Grid Propagation, Neural Target Propagation, the Topological DNA Engine, and a rigorous comparison against PyTorch, JAX, and the Go ML ecosystem. 3D Volumetric Grid 21 Numeric Types Neural Target Propagation WebGPU Native Pure Go · Zero CGO AI-Generated Deep Research · Podcasts & PDFs Nine release-era briefings from the Loom lab — listen in-browser or download the matching PDF report from files.openfluke.com . Jul 2026 · Seed Neural Networks Shrinking Massive AI into Digital Seeds Loom seed-manifold PoC — weight-agnostic compression where dense matrices collapse to 64-bit layer seeds, non-differentiable evolutionary search (Warmth · DNA · DNA-Layer), and what procedural weights mean for edge checkpoints measured in bytes, not megabytes. Your browser does not support the audio element. PDF MP3 Jul 2026 · v0.83 Hardware Acceleration Loom 0.83 Capabilities and Applications Experimental Hardware Acceleration — Plan 9 SIMD across ARM & x86 (no CGO / CUDA / OpenBLAS), Apple Metal zero-copy on Apple Silicon, and Qualcomm Hexagon NPU offload with the Drift Spectrum ( EXACT · INDUS · LOWBIT · H-DRIFT ). Sovereign DNVM on divergent hardware. Your browser does not support the audio element. PDF MP3 v0.83 release Jun 2026 · Planet Bridging v0.5 AI Model Bridging and Runtime Comparison Live ingestion from PyTorch, TensorFlow, JAX, and scikit-learn into Loom .entity checkpoints — twelve bedrock layers validated, Welvet Ladder three-way compare, and how Planet Bridging stacks up against ONNX Runtime, LiteRT, ExecuTorch, and Core ML. PDF Planet Bridging page Jun 2026 · v0.81 NPU Why Fast AI Hardware Fails Basic Math Loom v0.81 Accelerator Bridge — per-layer NPU offload via OpenVINO, the small-tensor latency tax, 28× INT8 wins on fat ops, and when deterministic hardware still drifts from software parity. Your browser does not support the audio element. PDF MP3 Jun 2026 · Strategy Pivot Solving The M-POLY-VTD Dispatch Paradox Why Plan 9 ASM wins on fat matrices but inverts on volumetric grids — dispatch overhead, WebGPU validation tax, and the pivot from per-hop CPU tuning to load-time GPU graph compilation. Your browser does not support the audio element. PDF MP3 v0.80 · Native Ship Loom Breaks AI Vendor Lock with Go ENTITY native checkpoints, Planet Bridging hub, pure Go + WebGPU v29 — why HuggingFace import is not the same as shipping a Loom brain. Your browser does not support the audio element. PDF MP3 v0.78 Loom Poly AI Engine Research Flagship M-POLY-VTD deep dive — volumetric dispatch, 21 dtypes, target propagation, and Go ML comparisons. Your browser does not support the audio element. PDF MP3 v0.76 Operation Mesh Shrinks Local AI How Loom’s operation mesh and release trajectory tighten the local-AI deployment story on consumer hardware. Your browser does not support the audio element. PDF MP3 v0.75 Mac Mini Beats RTX 4090 with Loom AI engine tiling update analysis — cache-aware dispatch and why Apple Silicon + Loom can outrun big discrete GPUs on the right workloads. Your browser does not support the audio element. PDF MP3 View Source Back to Loom Section 1 The Paradigm Shift: Volumetric Tensor Dispatch (VTD) Traditional deep learning frameworks — including PyTorch and TensorFlow — construct neural networks as directed acyclic graphs (DAGs) or sequential layer lists . While mathematically sound, this one-dimensional abstraction creates rigid execution pipelines that struggle to implement complex, biologically inspired routing. The Loom architecture fundamentally dismantles this constraint by introducing a 3D Volumetric Coordinate System . Every layer is assigned a geometric address (z, y, x, l) within a pre-allocated spatial grid. A flattening algorithm maps these 3D coordinates to contiguous 1D memory, maintaining hardware cache locality despite the logical 3D abstraction. Spatial Hopping In standard sequential models, data must flow strictly from layer N to layer N+1. In the Loom volumetric grid, data signals can bypass adjacent layers and jump across geometric coordinates — mimicking biological cortical columns. If a layer has an IsRemoteLink flag, the dispatcher fetches the remote layer dynamically via TargetZ, TargetY, TargetX, TargetL and injects it into the local execution path without graph recompilation. Dynamic Branching via Polymorphic Routing: The LayerParallel and LayerSequential container types aggregate sub-branches within the coordinate space. When ParallelForwardPolymorphic executes, the dispatcher routes input to multiple coordinate-mapped branches simultaneously, then merges using configurable topological modes: 🔗 concat Standard tensor concatenation across parallel branches. ➕ add Residual aggregation — sum branch outputs for skip connections. 〰️ avg Ensemble smoothing via averaged output tensors. 🔀 grid_scatter Spatial distribution of tensors across the volumetric grid. 🎛️ filter (MoE) Mixture-of-Experts gating: a FilterGateConfig layer generates Softmax coefficients to compute a dynamically weighted sum. Section 2 Multi-Numerical Polymorphism (M-POLY) A critical bottleneck in edge-device inference is memory bandwidth — streaming weight matrices from global VRAM to compute units. The Loom engine addresses this through native multi-numerical polymorphism . Unlike standard frameworks that require exporting to a fixed lower precision, Loom layers operate as fluid polymorphic units . The WeightStore struct maintains a master Float32 representation as the absolute source of truth, alongside a localized cache of actively morphed target precisions keyed by DType . Loom supports 21 distinct numerical types : Float64 Float32 BFloat16 Float16 FP8 E4M3 FP8 E5M2 FP4 Int64 Int32 Int16 Int8 Int4 Int2 UInt8 UInt4 UInt2 Ternary Binary (1-bit) NF4 E2M1 E3M0 Hardware Emulation via SimulatePrecision: For extreme low-bit types lacking native CPU/GPU register support (FP4, 2-bit quantization), Loom employs a universal fallback that mathematically forces the Float32 master weight to behave exactly as its lower-bit counterpart — simulating exponent/mantissa bounds for FP8E4M3, restricting to four discrete scaling levels for Int2, and clamping to ±1 for Binary. This enables Quantization-Aware Training (QAT) without complex fake-quantization node injections (as required by PyTorch). Different spatial coordinates can operate at different precisions simultaneously — a reasoning node in Float16 while an embedding lookup runs in 2-bit. 98.4% On-Disk Compression By packing low-bit representations, the Loom architecture achieves up to 98.4% on-disk compression for localized model deployment — effectively breaking the 192 GB/s memory bandwidth wall that stifles traditional inference on consumer graphics cards like Turing-class GPUs. Section 3 Systolic Grid Propagation: The Discrete-Time Neural Mesh Standard deep learning inference operates in a continuously flowing waterfall pattern — layer 1 finishes, passes memory to layer 2, and so on. Loom introduces Systolic Grid Propagation , modelled after the hardware systolic arrays used in Google's TPUs. Under this model, the 3D Volumetric Grid is a discrete-time neural mesh . The SystolicForward function advances the entire 3D grid by a single temporal "tick" — every coordinate calculates its output simultaneously based solely on input states from the previous tick. 🔁 Double Buffering The network maintains ReadBuffer and WriteBuffer per tensor state. During dispatch, every layer reads from ReadBuffer and writes results exclusively to WriteBuffer. CommitSystolicState then atomically swaps buffers — preventing race conditions in concurrent environments. ⏱️ Temporal Pattern Learning Information takes time to propagate geometrically across the network. This fundamentally alters how sequence data is processed — enabling true temporal learning that standard feedforward networks cannot achieve. 🔀 Asynchronous Layers Because layers operate asynchronously relative to continuous data flow, the systolic mesh supports online learning patterns that are impossible in standard sequential epoch-based training. Section 4 Neural Target Propagation (TargetProp) Backpropagation is widely criticized for its biological implausibility: it requires global error computation, exact weight symmetry, and freezing the forward activity while gradients are sequentially calculated backward through the chain rule. Loom implements an advanced alternative: Neural Target Propagation . Instead of computing continuous derivatives, TargetProp computes a proposed "target" state for each hidden layer. Each layer's objective is no longer to minimize the global loss via partial derivatives, but simply to map its forward activation to the proposed backward target . How TargetProp Works in Loom During the forward pass, actual activations are captured in ForwardActs . During optimization, CalculateTargetPropGaps executes an inverse estimation: for Dense layers, estimated targets are generated via weighted importance of downstream targets relative to master weights. For LSTM layers, the engine aggregates backward through input, forget, cell, and output gates simultaneously, creating a synthesized target for the previous recurrent time step. Gap-Based Hebbian Optimization: Once targets are generated, ApplyTargetPropGaps applies a local Hebbian-style learning rule. The weight update follows: ΔW = η · input · (target − actual) Loom introduces an advanced stability mechanism via LinkBudget — dynamically calculated from the cosine similarity between the forward activation vector and the backward target vector. If the target signal is highly misaligned (cosine similarity below 0.2), the layer simply ignores the update . This prevents catastrophic forgetting and exploding signals. Crucially, because TargetProp does not require differentiable functions, Loom can natively optimize extreme architectures like binary (1-bit) or ternary networks where standard gradients would vanish or shatter. Section 5 The Topological DNA Engine Because layers can dynamically hop across a 3D coordinate space and shift their numerical precision, traditional cryptographic hashing or PyTorch state-dict comparisons would instantly register a complete mismatch even when underlying logic is intact. Loom integrates a native DNA Engine based on principles from Topological Data Analysis (TDA). ExtractDNA converts every layer into a LayerSignature capturing spatial coordinates, layer type, DType, and a dimensionally normalized weight representation . The SimulatePrecision function expands all active WeightStore versions back to unified Float32 before unit vector normalization — ensuring the geometric "direction" of weights is captured independently of bit-depth magnitude. Logic Shift Detection CompareNetworks identifies Logic Shifts — when a layer signature in Model A aligns with high cosine similarity (>0.8) to a layer in Model B, but at a different spatial coordinate . This allows researchers to observe how architectural search algorithms or systolic propagation patterns naturally migrate logic pathways to more efficient regions of the 3D grid over time. Section 6 Native WebGPU Acceleration & Hardware-Aware Tiling Loom achieves 70+ tokens/second on consumer hardware through low-level optimization. The hardware.go module executes deep OS-level system calls ( sysctl on Darwin, /sys/devices/system/cpu/cpu0/cache/ on Linux) to determine exact L1/L2 cache byte sizes. Dynamic L1/L2 Cache Tiling: CalculateOptimalTileSize restricts matrix multiplication blocks so that the entire sub-block remains resident in L1 cache — significantly reducing global memory fetch latency. This delivers major speedups for operations like swigluTiledProjectGateUp . WGSL Shader Workgroup Optimization: For WebGPU execution, Loom queries MaxComputeWorkgroupStorageSize and MaxComputeInvocationsPerWorkgroup directly from the WebGPU adapter. MHA shaders allocate shared arrays for Keys and Values, using workgroupBarrier() synchronization, sized to consume exactly half of available workgroup storage — achieving optimal execution across Apple Silicon, NVIDIA CUDA, and integrated mobile GPUs. Section 7 Sub-System Autonomy: Tokenization, Ensembling & Telemetry 🔤 Native BPE Tokenizer A full Byte-Pair Encoding tokenizer written in Go, natively parsing HuggingFace tokenizer.json schemas. Includes a byte-fallback mechanism ( gpt2ByteEncode/Decode ) for unknown Unicode characters — enabling completely standalone, offline string-to-tensor processing. 🧮 Mathematical Ensembling FindComplementaryMatches assesses binary correctness masks of multiple models, calculating combined coverage ratio and cosine similarity of success rates — enabling optimized "Mixture of Models" pipelines that complement each other's weaknesses. 📊 Differentiable K-Means KMeansForwardPolymorphic transforms standard K-Means into an end-to-end differentiable operation using temperature-scaled distance metrics and Softmax gating, allowing classification topologies anywhere in the volumetric grid. 📡 Microsecond Telemetry The PolyObserver interface enables real-time tensor interception during forward/backward passes. AdaptationTracker monitors degradation and recovery via moving windows of outputs, accuracy, and throughput ( OutputsPerSec ). Section 8 Comparative Analysis: Loom vs Python Ecosystem (2026) The global deep learning industry has historically been dominated by Python-based frameworks. Comparing Loom to these heavyweights highlights distinct philosophical and technical divergences. Feature Loom (M-POLY-VTD) PyTorch (+ TorchAO) JAX (+ Flax/Optax) Execution Paradigm 3D Volumetric Mesh / Spatial Routing 1D Sequential / Dynamic DAG Functional / Compiled Static Graph (XLA) Language Pure Go (Compiled Native Binary) Python (C++ / CUDA backend) Python (C++ / XLA backend) Quantization 21 types native (FP64 down to Binary 1-bit) Native FP8, INT4, INT2, 1-bit via TorchAO Native FP8, INT8; sub-byte via custom libs QAT (Hardware Emulation) Built-in polymorphic SimulatePrecision FakeQuantize modules (complex node injection) Custom JAX primitives Optimization Engine Polymorphic BPTT + Native Target Propagation Native Autograd (reverse-mode AD) Functional forward & reverse AD Target Propagation First-class native Requires extensive custom class overrides High research support via custom logic flows GPU Acceleration WebGPU (cross-platform, edge & browser) CUDA, ROCm, Metal (vendor-specific) TPU, CUDA, ROCm (heavy compiler reliance) Structural Analysis Topological DNA Engine + Logic Shifts Standard dict/parameter hashing Standard dict/parameter hashing Deployment Footprint Single binary, zero dependencies Large runtime (PyTorch + CUDA variables) Large runtime (JAX + XLA toolchains) Section 9 Comparative Analysis: Loom vs Go ML Ecosystem (2026) Feature Loom Born ML GoMLX Gorgonia (Legacy) Core Architecture 3D Spatial Grid (Volumetric routing) 1D Sequential module stacks 1D Sequential computation graphs Static graph (Theano/TF1 style) Compute Backend Pure Go + WebGPU (Zero CGO) Pure Go + WebGPU (Zero CGO) OpenXLA (Heavy C++ bindings) CGO / CUDA (C++ bindings) Modern LLM Topology MHA, SwiGLU, RMSNorm, RoPE MHA, GQA, SwiGLU, KV-Cache, RMSNorm Gemma support / ONNX translation None (basic perceptrons/CNNs only) Quantization Spectrum 21 types (FP64 down to Binary 1-bit) Standard (FP32/FP16) Standard (dictated by XLA compiler) FP32/FP64 only Optimization Engine Backprop (BPTT) + Native Target Propagation Automatic Differentiation (Autograd) Automatic Differentiation via XLA Symbolic & Automatic Differentiation Non-Standard Layers Native Differentiable K-Means Clustering Requires external implementation Requires external implementation Requires external implementation System Telemetry Advanced window-based Adaptation Tracking Standard terminal logging Standard terminal logging Standard terminal logging Conclusions Strategic Outlook The Loom M-POLY-VTD architecture represents a radical divergence from established norms of deep learning engineering in 2026. By replacing the 1D computational graph with a cycle-accurate 3D Volumetric Grid, the framework physically maps neural structures in a manner that accommodates advanced biological routing — spatial hopping, systolic parallelism, and polymorphic precision. Its exhaustive 21-type polymorphism and simulated precision mechanisms directly confront the hardware memory bandwidth crisis, enabling dynamic on-the-fly quantization to 1-bit precision without structural memory reallocation. Neural Target Propagation provides a mathematically viable path for continuous, asynchronous training on power-constrained edge hardware. Complemented by the DNA Engine's topological signature matching, native BPE tokenization, and pure-Go WebGPU acceleration, Loom provides a self-contained, enterprise-grade ecosystem — vastly surpassing legacy Go frameworks, matching Born ML's deployment efficiency, and introducing architectural innovations previously reserved for experimental Python and JAX research environments. View Loom on GitHub Loom Documentation Back to Loom Overview --- Source: https://openfluke.com/loom/planetbridging # Planet Bridging — Live AI Model Ingestion into Loom (v0.5) > Planet Bridging streams live PyTorch, TensorFlow, JAX, and scikit-learn weights into Loom .entity checkpoints — twelve bedrock layers validated, Welvet Ladder three-way compare, vs ONNX Runtime, LiteRT, ExecuTorch, and Core ML. Canonical: https://openfluke.com/loom/planetbridging --- Planet Bridging v0.5.0 · planets → Loom Universal AI engine bridging Each major AI runtime is its own planet — PyTorch, TensorFlow, JAX, scikit-learn — with closed formats, operator dialects, and hardware-specific math. Planet Bridging is the live ingestion layer that streams in-memory weights from those planets into Loom’s deterministic .entity runtime — no lossy offline conversion chain, no re-export circus per OS. github.com/planetbridging Deep research PDF Loom research Loom overview Architecture Live stream, not static compile Traditional edge deployment exports models ahead-of-time (ONNX, FlatBuffer, Core ML packages). Planet Bridging instead captures active weights inside the native Python process, serializes topology + parameters into a JSON layer stream, and POSTs to /api/v1/loom/stream/* . Loom’s Go compare host rebuilds the graph via bridge.BuildNetworkFrom*Stream and compiles a binary .entity checkpoint. [ PyTorch ] ──┐ [ TensorFlow ] ──┼── (JSON layer stream) ──► Loom compare host ──► .entity [ JAX ] ──┤ via bridge.BuildNetworkFrom*Stream [ scikit-learn ] ──┘ Flagship walkthrough: 01_hello_stream.py runs a dense MLP ( mlp_32_64_32_16_8_relu ), streams live PyTorch weights, saves .entity , and validates Loom vs native on a shared test vector within fp32 tolerance. Multi-head attention fixtures like mha_8_2_4 use the same pipeline. Validation Twelve bedrock layers — stream & compare status Every standard Loom volumetric layer type has a live-stream bedrock. Empirical compare against native engines (v0.5.0): Bedrock layer Source engines Loom stream Loom compare Tolerance Dense PyTorch, TensorFlow, JAX, scikit-learn PASS 43/48 PASS fp32 < 1e-5 CNN1 / CNN2 / CNN3 PyTorch, TensorFlow, JAX PASS 12/12 each fp32 < 1e-5 Multi-Head Attention PyTorch, TensorFlow, JAX PASS 12/12 fp32 < 1e-5 LSTM PyTorch, TensorFlow, JAX PASS 12/12 fp32 < 1e-5 Simple RNN PyTorch, TensorFlow, JAX PASS 12/12 fp32 < 1e-5 LayerNorm PyTorch, TensorFlow, JAX PASS 24/24 fp32 < 1e-5 Embedding PyTorch, TensorFlow, JAX PASS 24/24 EXACT bit-exact RMSNorm PyTorch, TensorFlow, JAX PASS 24/24 fp32 < 1e-5 SwiGLU PyTorch, TensorFlow, JAX PASS 24/24 fp32 < 1e-5 Residual PyTorch, TensorFlow, JAX PASS 24/24 fp32 < 1e-5 Mixer v1 PyTorch, TensorFlow, JAX PASS 3/3 fp32 < 1e-5 Mixer v2 PyTorch, TensorFlow, JAX PASS 3/3 POC ~5e-5 (12 types) Examples Python examples in planetbridging/examples/ Install pip install planetbridging[pytorch] welvet or clone the repo. Each script streams live weights — no HTTP required for the pip path. 01_hello_stream.py One bedrock, one engine — the canonical dense MLP stream + .entity save. 02_all_layer_types.py All 13 Loom layer types exercised on PyTorch fixtures. 03_cross_engine.py Same bedrocks across PyTorch, TensorFlow, and JAX. 04_multi_layer_models.py 4-layer MLP, 2-layer CNNs, and Mixer v2 stacks. 05_welvet_ladder.py Welvet Ladder — three-way validation: native → loom-stream → welvet reload from .entity . 06_showcase_everything.py Full API tour; use --quick for a faster smoke run. Batch run from git checkout: ./examples/run_all_examples.sh · outputs land in ./.planetbridging/examples/ Edge training DNVM + M-POLY-VTD after absorption Bit-exact determinism Loom’s pure-Go DNVM keeps cross-platform parity tight enough for on-device training — same graph, same math on WASM, mobile, and desktop. 21 dtypes per layer M-POLY-VTD morphs precision per spatial cell — FP32 down to 1-bit binary — without rebuilding the whole network. Dual optimization paths Standard backprop in pure Go, plus Neural Target Propagation (Tween) for localized Hebbian updates with lower peak memory. Comparison Planet Bridging vs other edge runtimes Dimension Loom + Planet Bridging ONNX Runtime LiteRT (TFLite) ExecuTorch Core ML Ingestion Dynamic stream from live Python memory → .entity Static ONNX graph compile TF → FlatBuffer convert PyTorch → compiled .pte coremltools convert Multi-engine in PyTorch, TF, JAX, sklearn unified ONNX exports from many libs TF-centric; others via tools PyTorch export path Apple toolchain only In-memory absorption Yes — JSON stream → binary entity Offline export only Pre-built flat files Static compiled graphs Compiled bundles Edge training Native backprop + Target Prop in-engine Split-phase ORT training artifacts Signature-based trainable weights Federated / orchestrated paths Pre-flagged updatable layers Cross-platform parity Bit-exact DNVM target Varies by EP / vendor libs Platform float drift Delegate-dependent Apple Silicon optimized Vendor lock-in Pure Go · zero CGO core Multi-platform but heavy Google mobile stack Meta / PyTorch ecosystem Apple hardware Full analysis: AI Model Bridging and Runtime Comparison (PDF) · also on the Loom research page. Roadmap Version halves — into Loom, then out of Loom v0.5.0 · now Planets → Loom (complete) Thirteen compare tabs, twelve layer bedrocks, Mixer v2 — live stream ingestion validated. v1.0.0 Loom → other engines (Phase B) Native exporters to ONNX, Safetensors, GGUF — models flow back to llama.cpp, ORT, Core ML, and friends. v1.x Offline file ingest (Phase C) Load Safetensors, GGUF, ONNX directly into the compare host without a live Python planet. v1.x → 2.0 Depth + polish (Phase D) More layer types, expanded training surfaces, edge execution optimizations. --- Source: https://openfluke.com/soulglitch # SoulGlitch v2.4 — Offline AI Planet Playground > SoulGlitch v2 — orbit My Planet, tap Play to walk, run swarm polls with 1,500+ emoji personas, on-device AI chat, echo destruction combat, and physics scenes. iOS, Mac, Android, Linux. Windows internal testing. 100% offline. Canonical: https://openfluke.com/soulglitch --- v2.4 · App Store · Google Play · Linux SoulGlitch Orbit My Planet , tap Play to walk the surface, magnet wild echoes, run swarm polls with 1,500+ emoji personas, chat with on-device AI, and explore destruction combat — all offline on your phone or PC. No cloud. No accounts. Free to download Pick your platform. Models, chats, and votes stay on your hardware. App stores App Store iPhone, iPad & Mac Open in App Store Google Play Android phones & tablets Open in Google Play Desktop — direct download (v2.4.0) Linux tar.gz · x86_64 · ~92 MB · Founder included free Download tar.gz Windows x64 & ARM64 Internal testing only — not a public release yet After extracting Linux: ./bundle/soulglitch · Mac → App Store Founder Pack — one-time unlock for swarm poll, full emoji catalog, and future premium features (Android · iOS · macOS) Watch trailer · Screenshots · Construct TCP tutorials · Founder Pack Ask a swarm, not just one bot Pick emoji personas, ask yes / no / maybe, and watch each voter reply in character — all locally. “Should we explore the jelly biome tonight?” 🦊 Fox vibe YES The glow looks incredible after sunset. Let's go. 🐢 Cautious one NO We still have three unclaimed patches on the home cell. 🎲 Chaos analyst MAYBE Only if we bring the destruction sphere for science. Swarm result: 1 yes · 1 no · 1 maybe Watch it SoulGlitch in motion Orbit your planet, walk the surface, run swarm polls, and explore the v2 play flow — offline on your device. Watch on YouTube Screenshots v2 on phone & tablet Tap any shot to zoom. Phone captures show orbit, play, swarm, chat, and combat. Tablet shots show the wider HUD layout. Phone Tablet × ‹ › What's new in v2.4 Latest polish Recent releases focused on play feel, audio, and swarm UX — building on the v2 orbit/play overhaul. v2.4 🔊 Audio panel Top-bar speaker opens a fullscreen sound panel — master volume, per-channel mute, looping planet music, and claim SFX. Settings persist across sessions. v2.4 🗳️ Fullscreen swarm polls Ask, live vote, and history panels sit above the game HUD. Emoji-only voters — pick personas inline, no buddy picker clutter. v2.3 🧭 Surface guide arrows Cyan compass toward wild echoes, orange toward patch scene targets. Works in orbit, on-foot, and split view. v2.2 🛰️ Orbit → Play flow Two views only: Orbit (default) and Player . Tap Play on the dock to drop to the surface; Esc returns to orbit. v2.2 💥 Destroy HUD Single Destroy toggle with clear on/off glow. Call/Recall and Follow/Stay as thumb-friendly toggles on the surface HUD. v2.1 ✨ 1,500+ emoji powers Each persona gets a unique destruction gene mix. Magnet wild echoes, assign party slots, and assault destructible zones in patch scenes. Core experience What SoulGlitch does Glass dock: Chat · Poll · Browse (plus Play when orbiting). Everything runs on your hardware. 🌍 My Planet Infinite voxel multiverse — orbit your home cell, walk with WASD, magnet echoes, claim patches, and enter built scene instances. 💬 On-device AI chat Glitch Threads (Spark / Echo / Roots), document libraries with RAG, emoji persona vibes, Glitch Link LAN mesh, CPU / SIMD / GPU inference. 🗳️ Swarm poll Ask your crew yes / no / maybe. Each emoji voter streams an in-character reply plus a vote word; live tally and saved history. 🧬 Echo Codex Capture wild echoes, assign party slots, browse 1,500+ emoji powers, and run destruction assaults in patch scene arenas. 🎬 Scene gallery 29 embedded physics worlds — ragdolls, towers, cars, labs. Search, filter, and play with touch orbit or WASD. 🧠 AI Models SmolLM2, Qwen3, BitNet — download from Hugging Face, convert to .entity , test locally, sync brains over LAN. 🛰️ Construct TCP Host a construct server on your planet. External Go clients from construct-tcp-examples connect while you stay in-game. 📊 ISO Telemetry & TANHI Cross-device inference proofs (PDF export) and live neural HUD — pair with Loom on your PC and watch layers over UDP. 🔒 Private by design No ads in the core loop. Network is for downloads and LAN tools you choose — not uploading chats to our servers. Free vs Founder Pack Explore free. Unlock the full swarm. Planet play, chat, codex, and physics gallery are free. Founder Pack unlocks premium features on mobile stores. Free Planet-first playground Orbit, play, chat, magnet echoes, scene gallery, and model pipeline — no subscription. My Planet multiverse — orbit, walk, claim patches On-device AI chat (Glitch Threads + Libraries) Echo Codex — capture echoes, destruction combat Scene gallery — 29 physics worlds Construct TCP host + LAN model sync One free swarm poll to try voting (mobile) Linux desktop includes Founder features free Founder Pack · ~$8.88 Premium unlock One-time purchase on Android, iOS, and macOS. Support development and unlock the full experience. Swarm poll — unlimited yes / no / maybe votes 1,500+ emoji personas — full catalog for chat & voters All future premium features — lock in early pricing Coming later: local network, online network, compute node scene generation Restore purchases supported on App Store & Google Play More demos Deeper dives Construct TCP physics tests and TANHI neural telemetry — for tinkerers and ML nerds. Construct TCP — 13 Go examples on your planet host. Tutorials TANHI — watch Loom models run live on your phone. Docs SoulGlitch v2.4 Land on your planet. Ask a swarm. Don't take it too seriously. Offline AI playground — orbit, play, poll your crew, magnet echoes, and run models on your own hardware. Built on Loom and PrimeCraft physics. Desktop — v2.4.0 Linux x64 tar.gz Windows x64 & ARM64 Internal testing only App stores App Store Open in App Store Google Play Open in Google Play Tutorials · Dev notes · YouTube trailer --- Source: https://openfluke.com/soulglitch/notes # SoulGlitch Dev Notes — v2.4 Planet Sandbox & Local AI > Developer notes for SoulGlitch v2.4: offline AI planet sandbox, swarm polls, Plex LAN hosting (CPU/SIMD/GPU), on-device models, Founder Pack, and what's shipping next. Canonical: https://openfluke.com/soulglitch/notes --- Dev notes · v2.4.0 · updated Jul 2026 SoulGlitch Notes What SoulGlitch actually is today: an offline AI planet sandbox — orbit My Planet , walk the surface, run swarm polls with 1,500+ emoji personas, chat on-device with Loom, host Plex LAN multiplex, and blow stuff up. Not a cloud chatbot wrapper. Not the old face-only prototype. SoulGlitch & downloads Construct tutorials Founder Pack NPUs & RnD Full changelog 🧬 What SoulGlitch is now The product as it exists on store + desktop SoulGlitch is an offline AI playground built on Loom + PrimeCraft physics. You boot into a voxel multiverse, claim a home planet, magnet wild echoes, run on-device models, and ask emoji swarms to vote on yes/no questions — all without sending chats to our servers. The tone is deliberately playful: “don’t take AI too seriously.” Underneath that is serious infra — Hugging Face download → on-device quant → .entity checkpoints, ISO telemetry PDFs, Construct TCP, and measured Plex multiplex on CPU SIMD and WebGPU. 📱 Core loop How a session is supposed to feel Land on My Planet in orbit — pinch zoom, compass guides, tap Play to walk the surface. Magnet wild echoes, assign party slots, toggle Destroy , enter patch scene arenas for emoji-power combat. Open Swarm poll — pick emoji voters, ask a question, watch live tally + streamed replies. Open AI Chat — Glitch Threads, RAG libraries, download models, pick CPU / SIMD / GPU inference. Plex Link — host or join a LAN multiplex; one model, isolated KV per client chat. Construct TCP — host on your planet; external Go clients spawn constructs while you stay in-game. Browse scene gallery, codex, models, ISO telemetry — glass dock: Chat · Poll · Browse (+ Play when orbiting). 🔗 Plex — share one brain on LAN Multiplex hosting (chaosglue/plex) Plex is LAN model hosting: load one .entity , assign up to 8 client slots, each peer gets an isolated conversation. Backends: CPU tiled , CPU SIMD (Plan 9 AVX2/NEON — ~2× throughput vs scalar in lab multichat), or GPU WebGPU KV-swap on desktop. Phone defaults to SIMD; desktop defaults to GPU. GPU host + planet WebGPU still fight for the same device — use SIMD on mobile when the multiverse renderer is live. Port 17485 ; scan/join from Plex Link sheet. 💸 Free vs Founder Pack What’s gated today (v2.4) Free Planet + core AI My Planet multiverse — orbit, play, echoes, patch scenes On-device chat with downloaded models (CPU / SIMD / GPU) Glitch Threads, libraries, model catalog, scene gallery Construct TCP host, ISO telemetry, codex browsing Linux desktop bundle includes Founder unlock free Founder Pack · one-time IAP Premium layer Swarm poll — fullscreen crew voting with emoji personas Full emoji picker in AI Chat & new glitch threads Future premium features (same purchase, no subscription) Android · iOS · macOS via store billing 💻 Platforms & downloads Where v2.4.0 runs iOS / iPad / Mac — App Store Android — Google Play Linux x64 — tar.gz v2.4.0-b4 Windows x64 & ARM64 — internal testing only (no public download) Extract Linux bundle → run bundle/soulglitch . Rebuild native libs after pulling engine changes — see repo README. 🧠 On-device models What the catalog supports today SmolLM2 135M / 360M / 1.7B — fast to heavy; good mobile defaults. Qwen3 0.6B / 1.7B / 4B — stronger reasoning; watch RAM on 4B. BitNet b1.58 2B — sub-byte quant path; CPU/SIMD friendly, heavy KV. Download safetensors on device → convert to .entity → pick Tiled / SIMD / WebGPU at load. Plex host shares whichever .entity you loaded — one weight set, many KV slots. ⚡ Future — vendor NPUs in-app On the roadmap after v2.4 Today SoulGlitch inference paths are CPU tiled , Plan 9 SIMD (AVX2/NEON), and WebGPU — plus Plex multiplex on those same backends. We plan native integration with vendor accelerators through Loom v0.83+ poly/accel : Apple Metal , Qualcomm Hexagon (Snapdragon — experimental in lab), and Intel OpenVINO NPU — each cell measured with the Drift Spectrum parity buckets before it ships in the app. Goal: pick the best backend per device automatically — SIMD on reactive dense loops, NPU for fat matmul/conv layers, WebGPU where the GPU isn't fighting the planet renderer. See Loom research PDF for measured numbers. Scene gen (procedural + LLM patch builds) is the other big open item — see changelog FREE RnD. 🎯 Who it's for Positioning notes Audience AI-curious people who want a toy universe on their phone or PC — sandbox gamers, local-LLM tinkerers, privacy-first users, and anyone bored of sterile cloud chat UIs. Hook “Land on your planet. Ask a swarm. Don't take it too seriously.” Sovereign AI is the engine story; chaos and emoji voters are the front door. 📋 Full changelog From mvp-simulation/UPDATES.md — newest first FREE RnD — the old lab wishlist. Almost all of it is in the app now ; scene generation is the main thing still open. Shipped (was RnD, now product) Plex multiplex — one .entity , many isolated LAN chats (CPU · SIMD · GPU, port 17485 ); AI Chat → Plex tab + Plex Link join sheet. TANHI / UDP viz — watch layer traffic in real time (Apps → TANHI, listener :17481 ). On-device models — download safetensors, quantize on phone/PC, save .entity locally (SmolLM2, Qwen3, BitNet, …). Glitch Threads + swarm — multi-persona chat, emoji voters, streamed swarm replies in the planet UI. Share compute — Plex host on LAN; share .entity — full checkpoint transfer without re-download (port 17482 ). Construct TCP — host on your planet; external Go clients spawn constructs while you play. Still RnD — scene generation Local scene generator — procedural worlds on-device (gallery card exists, not wired). LLM / Ollama scenes — describe a patch build in natural language → scene JSON (placeholder in Browse gallery). Optional polish: Plex-hosted characters with speech bubbles on the planet surface. 2.4.0 2026-07-09 Audio panel & swarm polish — dedicated sound controls, looping planet music, fullscreen crew voting, and emoji-picker stability. Ships on top of 2.3.0 . Audio Audio panel — top-bar speaker icon opens a fullscreen panel (master volume, per-channel mute, mute all); settings persist across sessions. Planet music — background_music.mp3 loops quietly on planet surface and patch instances. Claim SFX — planet_claim.mp3 plays when you claim a planet. Audio controls moved out of planet settings into the new panel. Swarm poll Fullscreen polls — ask, live vote, and history panels sit above dock, HUD, and top chrome. Emoji-only voters — buddy picker removed from “Who votes?”; add emoji personas inline. SIMD inference — Model & settings offers Tiled , SIMD , and WebGPU backends. Emoji picker fix — inline picker inside the swarm panel (no modal over the game view); viewport frame uploads isolated so semantics/layout no longer crash. AI Chat Buddies tab removed — home sidebar is Threads / Libraries / Plex. New glitch thread — emoji persona only (no planet buddy picker); legacy buddy-linked threads still open from Threads. Plex SIMD hosting — Plex tab backend picker is CPU · SIMD · GPU (Plan 9 AVX2/NEON multiplex, same chaosglue/plex pipeline as single-chat SIMD). Phone defaults to SIMD; desktop defaults to GPU. LAN port 17485 unchanged. Flutter (for integrators) Integrator file tables: see repo UPDATES.md for this release. Full restart recommended after pulling (new audio assets + native rebuild if engine changed). 2.3.0 2026-07-09 Surface navigation guides — compass arrows to wild echoes and patch scene targets, plus boot and echo-placement fixes. Ships on top of 2.2.1 . Fixes Black screen on launch — storage migration no longer deadlocks during init; construct settings load before camera home; boot errors surface in the multiverse view instead of hanging before runApp . Wild echoes after patch scenes — entering then leaving a built scene instance no longer collapses all loose echoes into one pile (positions are parked during the arena and restored on exit; fresh spread respawn if restore fails). Wild echo spawn spread — minimum tangent spacing, wider near-player ring, and even shell distribution so echoes do not stack on the same patch tile. Navigation guides Surface guide arrows — cyan arrow + line toward the nearest magnetable wild echo on the planet; orange arrow toward the destruction / scene sphere inside patch instances. Orbit & split view — guides work when zoomed out in orbit and in two-player split (camera projection matches the active viewport). Arrow on the ring — compass sits around your orb, not on top of it; bearing fallback when the target is off-screen or too close after zoom. Camera Pinch zoom — two-finger pinch on orbit and player views (same feel as scroll zoom). Default max zoom out — orbit and birds-eye player views start fully zoomed out when switching presets or returning home. Engine (for integrators) Integrator file tables: see repo UPDATES.md for this release. Rebuild required: ./scripts/build-lib-linux.sh (or platform equivalent) after pulling Go changes. 2.2.1 2026-07-06 Gallery fix & Destroy HUD polish — patch on 2.2.0 . Fixes Scene gallery — opening a gallery scene from Browse no longer crashes (dual wgpu / Jolt allocator conflict). Planet GPU releases before gallery/creator/tanhi viewports boot; multiverse resumes when you return. UI Destroy button — muted slate when off, orange-red glow when on (clear on/off state). Party status — removed redundant "destroy on" line under wild/following count. 2.2.0 2026-07-06 Planet play UX overhaul — two-view orbit/play flow, thumb-friendly game HUD, gather UI, and top notices. Ships on top of 2.1.0 echo destruction combat. Orbit & camera Two views only — Orbit (default) and Player (on-foot). No top Player/Orbit toggle. Cold start → Orbit — returning players land on the home planet in orbit view. Play — dock shows a game icon in orbit; tap it to drop to the planet with move controls open. Orbit on the controls panel (or Esc ) returns to orbit view. Hatch intro ends in Orbit (not forced Player); dev Play hatch no longer double-plays; player orb hidden during the cinematic. Codex / overlays — closing full-page routes restores your previous camera preset and re-snaps horizon background planets ( ChaosMultiverseRefreshBackground ). Codex return — coming back from Echo Codex restores move controls when you were on-foot in Player view. UI chrome Buddies rail removed — no top-right people button or side panel on the planet view (planet buddy meshes stay off). Resource dropdown stacks above the orbit auto-spin button (no more panel tucked underneath). Resource gather progress — stand on a glowing node to collect; progress bar floats in the center of the game view (not inside the resource dropdown). Dropdown is inventory counts only. Top notices — toast messages slide in under the top bar (replacing bottom snackbars). Apps menu (Browse) — full-screen opaque overlay; planet, dock, and side chrome hidden until you tap Planet back. Echo panel Assign button removed from the planet strip (use Codex for party slots — Assign only opened Codex anyway). Destroy is a single toggle (off = peaceful repair, on = combat assault); separate Repair button dropped. Act removed — Call starts the assault when Destroy is on. Call/Recall and Follow/Stay are single toggles each (not separate buttons). Bottom HUD layout — left: wild · following status, then Active · Codex · Call/Recall above the move stick; Orbit centered between stick and right rail; right: Destroy , Follow/Stay , Magnet (thumb side). Echo combat (follow-up) Party echoes use per-emoji skills during destruction assault (not generic pulse on Retract). Retract no longer ghost-fires destruction when echoes are recalled. Rebuild required if pulling Go changes from 2.1.0: ./scripts/build-lib-linux.sh (or platform equivalent). 2.1.0 2026-07-06 Echo Destruction Combat — ~1,595 unique emoji powers, destruction sphere assault in patch instances, and the full Repair ↔ Destroy loop. Emoji destruction powers ~1,595 composable powers — each emoji persona gets a unique gene mix (implode/explode, slam/lift, push/ram/sweep, spin/vortex, sphere/column/ring/wedge patterns, single/double-burst phase). Gene math lives in chaos/physics/destruction_power.go ; Flutter catalog in assets/emoji/emoji_destruction_powers.json (matches Go PowerForEmojiID ). Codex → Powers tab — searchable grid of every power (title, kind, genes, magnitude, radius); filter All / Unlocked / Locked . Power demo lab — tap a captured (or Founder-unlocked) power to open a live 3×3×3 cube arena and fire the instruct on demand. Founder pack gates browsing full power details for emojis you haven't magnet-captured yet. Repair vs Destroy Echo control strip: Repair (peaceful follow / mend) and Destroy (combat assault). Act instructs the active echo — in destruction mode this kicks off (or reinforces) the assault. On the planet surface , a small destruction lab spawns light cubes inside a ~9 m radius when Destroy is active. In patch scene instances , the zone auto-fits the imported gallery scene footprint (12–85 m) instead of a fixed disc. Destruction sphere & instance assault Enter a built scene → tap Destroy → a glowing destruction sphere wraps the play zone (dot shell + bright equator ring). Status line shows zone N in Xm — destructible body count and fitted radius. Call party echoes → each echo ghost-drifts in 3D toward its own target inside the sphere (parallel assault, not one-at-a-time). Every striker fires its own emoji power (area destruction at the target — implode, vortex, wedge, etc.). Target ring visual on the locked destructible; path arrows point at drift goals. 2 s stall timeout — if an echo makes no progress, it abandons and picks a new random target. Assault runs until the zone is clear; Retract recalls echoes without ghost-bombing the arena. Wild echoes & party Wild echo beacons — floating spheres above loose echoes on the planet (dim → species tint when near → cyan pulse in magnet range). Party assign trait-key sync fixed between Flutter codex and Go loadout ( emoji:hue:… wire format). Hatch intro plays once on construct reset / dev panel (no double-play). Engine (for integrators) Integrator file tables: see repo UPDATES.md for this release. Rebuild required: ./scripts/build-lib-linux.sh (or platform equivalent) after pulling Go changes. Known limitations (2.1.0) Destruction assault is tuned for patch scene instances ; planet-side lab cubes are a smaller sandbox. Store listing markdown under store/ still says 1.3.0 until the next store submit. 2.0.0 2026-07-06 v2 — patch scene instances — enter a built patch and play the gallery scene on foot. Enter built scenes Stand on a patch with a build → top Enter prompt (scene title from gallery catalog). Enter teleports you into an isolated flat arena (5 km plain) far from the multiverse grid — planet shell hidden, scene spawned at center. You and called echoes spawn outside the scene footprint (~140 m south) and run in — no spawning inside geometry. Player orb controls (WASD / move stick, top-down or FP camera) with ExploreControls + echo party panel. Play / pause / restart in scene chrome; Back returns to the planet and restores your position. Physics & echoes Arena uses flat gravity ; dynamic scene props get world gravity (chain-reaction pieces fall and settle). Scene ground/floor/plane meshes skipped on import (arena floor only). Echo drop, follow, and active modes use flat-floor navigation in the arena (no shell clamp under the plane). Engine (for integrators) Integrator file tables: see repo UPDATES.md for this release. Rebuild required: ./scripts/build-lib-linux.sh (or platform equivalent) after pulling Go changes. 1.9.1 2026-07-06 Top-down camera fix — run around the planet without flipping or getting stuck. Birds-eye / player view Fixed the 180° camera flip when orbiting the full planet in top-down mode — WASD no longer reverses mid-lap. Fixed the player orb spinning in place at certain coordinates (pole singularity in the old heading math). Movement and camera now share a stable tangent heading parallel-transported around the sphere (same idea as first-person). Rebuild required: ./scripts/build-lib-linux.sh (or platform equivalent) after pulling Go changes. 1.9.0 2026-07-06 Claim planets and build gallery scenes on patches . Planet claim Stand on a biome patch of an unclaimed planet (no builds on any patch) → prompt above the glass dock: *Would you like to claim this planet?* Claims persist per grid cell ( ix, iy, iz ) — multiple planets can be claimed across the multiverse. Saved locally in planet_claims.json and synced to the engine on boot / visit. Patch build On a claimed planet, stand on an empty patch → *Would you like to build something here?* (same placement as transit towers). Searchable gallery picker — JSON scenes only (embedded biocraft/anogu catalog today; user-generated entries when gallery gen ships). Scene spawns on the patch surface (oriented to planet up, ground plane skipped), scaled to fit the land-mass disk. Rise animation — glowing preview emerges from underground with beam + ring; massive hatch-style explosion on land; then solid build spawns. Builds persist per patch and reload when you revisit that planet cell (including after tower transit). Engine (for integrators) Integrator file tables: see repo UPDATES.md for this release. Rebuild required: ./scripts/build-lib-linux.sh (or platform equivalent) after pulling Go changes. 1.8.0 2026-07-06 Planet resource HUD polish and orb dash movement. Planet resources — top chrome Toggle chip next to the Player / Orbit camera menu — resources no longer always-on top-left. Current planet biome icon + collected count only (not home biome, not galaxy total). Polls cell_biome from the engine so counts match the planet you are standing on. Orb dash Hold move 2s → 5× speed burst while pushing; dash stays armed 2s after you release (re-tap move within that window without re-charging). Directional explosion VFX — anim1-style particle burst (world-space, visible in FP) along travel direction. Slam down — C , ↓ on-screen button, or Arrow Down while airborne; radial dash into the planet shell with downward burst. Echoes mirror player dash when following; patch step-up scales during dash. Engine (for integrators) Integrator file tables: see repo UPDATES.md for this release. Rebuild required after pulling Go dash changes: ./scripts/build-lib-linux.sh (or platform equivalent). 1.7.0 2026-07-05 Echo Codex Skills , UI cleanup, and orbit camera fixes. Echo Codex — Skills tab Skills tab on Echo Codex (Dex · Skills · Party) — unlockable Echo Build toolkit, not a gallery demo list. Catalog sections — Parts (box/sphere/cylinder/capsule/cone), Motion (static/dynamic/rotate), Joints (fixed/hinge/point), Spawn transform, Reflex programs (automation, sequence, pulse, PD, effectors), Echo acts/roles/cues. Unlocks from captures — magnet/codex traits + thresholds ( minUniqueActs , minCaptures , anyEcho ); party build hints (e.g. ~5 blocks + 1 rotor + 1 pulse). UI cleanup Removed duplicate Codex button from the bottom glass dock — Codex stays on the echo control strip only (less clutter). Orbit camera fix Top menu stays on Orbit — camera preset now syncs from the engine (menu no longer snaps back to Player after selecting Orbit). Planet visible in orbit — bootstrap sphere draws again while orbiting your home cell (was patches floating in empty space); neighbor impostors unchanged. Engine (for integrators) Integrator file tables: see repo UPDATES.md for this release. Rebuild required after pulling Go changes: ./scripts/build-lib-linux.sh (or platform equivalent). 1.6.0 2026-07-05 Planets feel like their biome type — PrimeCraft-style color, props, and patch identity. Biome-colored patches (on foot) Palette fix — luduiq.BiomeColor now matches the seven PrimeCraft biomes (water blue, lava orange, grass green, ice pale, crystal magenta, jelly purple, organic teal). Patch cubes no longer look like random earth tones. Home bias — when your planet has a dominant biome, ~78% of patch tiles use that type (was ~58%), so a lava home reads mostly orange, grass mostly green, etc. Bootstrap tint — planet sphere + patches share the same biome ground color. Orbit decorations (detail LOD) Lava — emissive glow liquid shell + fire plumes (rock pillars + additive glow) instead of trees. Water — rock clusters instead of trees. Grass / Ice / Crystal / Jelly / Organic — biome-tinted trees unchanged (trunk + canopy colors per type). Chaos engine ( chaos/luduiq ) BiomeColor() — PrimeCraft palette (IDs 0–6). biomeAtDir() — stronger ShellBiomeBias dominance when HomeBiomeID is set at construct boot. Engine (for integrators) Integrator file tables: see repo UPDATES.md for this release. Rebuild required after pulling Go changes: ./scripts/build-lib-linux.sh (or platform equivalent). Known limitations (1.6.0) Biome props (fire/rocks/trees) show in orbit view at detail LOD — not yet spawned on the construct home surface while on foot. Player env forces (jelly swirl, crystal pulse, organic lung) still apply to background wanderers only. Store listing markdown under store/ still says 1.3.0 until the next store submit. 1.5.0 2026-07-05 Sparse terrain patches on the walkable planet — fast, solid land-masses instead of the old full voxel shell. Terrain patches (replaces biome voxel shell) Patch bootstrap — construct now boots with PatchTerrain (~40 biome-colored land-mass plates per planet) instead of the luduiq full shell (~445k cells). Always on — patches are always generated and visible on your active planet (no settings toggle; removed Biome voxel shell from Planet settings). Planet + patches — on foot, the bootstrap planet sphere and patch overlays draw together (patches no longer hide the planet mesh). Walk physics — smooth planet sphere between patches; solid plate colliders on each land-mass (compound static shapes, not per-tile snagging). Orbit view — multiverse impostor draws the sphere; construct renders patch cubes on top. Echo party + patches Planet-first movement — following echoes path on the planet sphere , not pre-snapped to patch height. Step-up — when feet enter a patch disk, echoes hop onto the plate (animated radial lift + small arc); step back down when they leave. Wild spawn — loose echoes start on the ground; spawns directly on a patch disk begin at plate height. Chaos engine ( chaos/construct ) -patches flag in run-construct-tcp.sh (recommended); legacy -shell / -biome kept for full voxel shell. PatchSite anchors for future gallery-scene land-mass import. Player snap skipped while grounded on patch plates (Jolt footing drives position). Engine & FFI (for integrators) Integrator file tables: see repo UPDATES.md for this release. Rebuild required after pulling Go changes: ./scripts/build-lib-linux.sh (or platform equivalent). Known limitations (1.5.0) Patches are visual + walk colliders only — not mineable tile-by-tile yet (full shell still available via -shell in chaos TCP demo). Echo step-up is kinematic shell-lock, not full physics climb. Store listing markdown under store/ still says 1.3.0 until the next store submit. 1.4.0 2026-07-05 Echo companions on the walkable shell, multiverse horizon planets that actually look like neighbours, and a much lighter on-foot render path. Echo party system Echoes — collectible wild orbs on each planet; absorb into your party via Magnet → Assign → Call . Call — echoes drop from the camera column onto the planet shell (dynamic physics). Follow / Stay / Active — kinematic shell-lock: stand upright on the surface, auto-rotate to the normal, pathfind toward the player (construct TCP / capsule-squad parity). Limp — ragdoll pile mode (dynamic + gravity). Codex — Flutter store page for unlocked echo types; counts persist locally. Control panel — bottom-left in Player / top-down / split views (hidden in Orbit); status line shows wild / following / assign queue. Fall-through fix — squad stabilize + pathfind always tick on foot (gallery lite no longer skips them); shell clamp during drop/limp; loadout auto-activates on Follow/Stay. Construct TCP keyboard parity (Player view) Integrator file tables: see repo UPDATES.md for this release. Multiverse background planets & LOD 3-tier planet LOD — far (16×10), mid (24×16), detail (32×20) with hysteresis so meshes don’t thrash as you move. Horizon snap (on foot) — neighbour impostors draw at ~one grid step from the player instead of true cell centres 800–2200 u away (fixes tiny sky specks in Player view). Tighter on-foot streaming — immediate neighbour ring only, forward hemisphere preferred. Background planets toggle — Planet settings → Background planets now works in gallery-lite Player view (was always cleared for FPS). Compositing sync — surface camera mode refreshes when switching Orbit ↔ Player, including gallery render path. Active cell — multiverse impostor sphere hidden on foot; construct walk shell is the ground (optional biome voxel overlay via settings). Gallery lite (on-foot performance) Single-pass render on Player / top-down — same draw path as Scene gallery; skips multiverse compositing churn when background planets are off. Skips — luduiq fluid goroutine, orb smoke/trails/bubbles, wireframe bootstrap duplicate, per-frame tower colour pulse (towers + resources stay for gameplay). Resolution — uses sceneGalleryPixelSize() formula instead of the old 960×540 cap. Engine & FFI (for integrators) Integrator file tables: see repo UPDATES.md for this release. Rebuild required after pulling Go changes: ./scripts/build-lib-linux.sh (or platform equivalent). Known limitations (1.4.0) Echo party UI is Player-surface only (not Orbit). Horizon snap is visual-only — transit still targets real grid cell centres. Gallery lite still runs construct physics + one full engine.Render() ; very old GPUs may need Background planets off. Store listing markdown under store/ still says 1.3.0 until the next store submit. 1.3.0 2026-07-03 Walkable multiverse planets — gather biome resources, ride transit towers to neighbouring cells, and keep Player / Orbit cameras working after travel. Planet resources (first playable pass) Six collectible nodes per active planet cell, one flavour per biome (Brinethread, Ember ash, Verdant pollen, etc.). Sit-to-collect — stand on a node ~1 s to pick it up (Player view only). HUD — biome resource icons with live counts in the multiverse top bar. Respawn — nodes come back after ~75 s. Persistence — counts saved locally per device ( soulglitch/planet_resources.json ); not cloud-synced yet. Design doc — PLANET-RESOURCES.md (canon; implementation now partially landed). Interplanetary transit (construct towers) Transit pillars — six white, semi-transparent beams on each walkable shell (along ±X / ±Y / ±Z), replacing the old purple 500-unit multiverse beams. Proximity prompt — Flutter panel shows destination grid coords and biome name; Stay / Travel . Three-phase hop — physics boost off the tower → arced cruise across the grid → ease onto the arrival shell at the opposite tower. Planet relocalize — construct voxel shell, resources, and towers respawn on the destination cell mid-flight; player keeps momentum feel instead of a hard teleport. Safe spawn — dev-player bootstrap avoids spawning directly on tower axes. Camera & view modes Player on any planet — top-down, first-person, and split work after travelling off home (no longer locked to Orbit-only off-home). Orbit follows you — after transit, Orbit targets the active construct cell , not hard-coded home (0,0,0) . No forced Orbit snap — arriving on a neighbour no longer yanks you out of Player view every frame. Visual cleanup Removed legacy pink pillars — old spawnTowers 500-unit purple beams disabled; cached planets purge -tower- meshes on first stream refresh. Depth-correct glow — construct tower beams use depth-tested glow (white / faint blue) instead of drawing on top of the whole planet. Engine & FFI (for integrators) Integrator file tables: see repo UPDATES.md for this release. Rebuild required after pulling Go changes: ./scripts/build-lib-linux.sh (or platform equivalent). Known limitations (1.3.0) Resource collection only in Player view (not Orbit). Transit is a scripted hop (boost + lerp), not a full open-space flight sim yet. PLANET-RESOURCES.md still lists some future energy-conversion / scene-spawn hooks as not implemented. iOS may need FFI symbols added to chaosmvp.xcconfig if transit/resource exports are missing on that target. 1.2.0 (prior release) Plex — LAN multiplex hosting in AI Chat (shared CPU / CPU SIMD / GPU model, per-client KV slots, port 17485 ). Multiverse-first launch (splash → home planet), planet settings, bottom dock (Chat · Swarm · Construct · Apps). On-device swarm voting, RAG chat, Hugging Face model pipelines, character worlds. See README.md for full 1.2.0 feature surface. Integrators: each release section in repo UPDATES.md also lists Go/Flutter/FFI file tables. Rebuild: ./scripts/build-lib-linux.sh (or platform script) after pulling engine changes. SoulGlitch v2.4 — offline planet sandbox, on-device AI, Plex LAN multiplex, emoji swarm polls. Built on Loom . See main page for downloads. --- Source: https://openfluke.com/primecraft # Primecraft — Voxel Simulation Engine with Embedded AI > Primecraft is OpenFluke's distributed simulation engine: procedural worlds, physics, voxel scenes, and embedded neural AI. Android, Windows, Linux, Steam. Canonical: https://openfluke.com/primecraft --- Early Access Available Now Primecraft A distributed simulation engine for procedural worlds and embedded neural agents — delivered as a playable game across mobile, desktop, and the web. Android Windows Linux Wishlist on Steam Alpha v0.30.0 — Model Sharing Patch 8B+ Planets 100% Offline Real Neural AI Overview What Is Primecraft? More than a game — it's an experimental engine exploring AI, physics, procedural generation, and distributed gameplay. Primecraft is an experimental, physics-driven sandbox built on top of a custom simulation engine. The game blends procedural world generation, real neural-network AI, player-driven construction, and multi-device gameplay. It is also the foundation the creature game SoulGlitch is built on. In simple terms: Procedural Worlds + Real AI + Multiplayer Sandbox Explore billions of procedural planets — each deterministically generated from its coordinates Train on-device neural networks — companions that learn and evolve with you Drop into bubble scenes — mini 3D levels loaded directly from the web Build or import levels — using a simple JSON-based scene format Couch co-op & LAN sync — play together on one device or across your network Physics-driven gameplay — destructible objects, planetary gravity, and dynamic abilities Fully offline — including AI training, no cloud required Features Core Gameplay Features Experience a unique blend of exploration, creation, and AI-driven gameplay. Procedural Universe Navigate through billions of unique planets, each with distinct terrain, resources, and environmental conditions. Every world is algorithmically crafted for endless exploration. Neural AI Companions Train real neural networks directly on your device. Your companions learn from gameplay, developing unique behaviors and abilities through the Loom AI framework. Physics Sandbox Experience realistic planetary gravity, destructible environments, and physics-based abilities. Every object in the world responds to forces and collisions. Bubble Scenes Discover and enter bubble scenes — self-contained 3D levels created by players and loaded from the web. Play puzzles, challenges, and custom worlds. Multiplayer Experiences Play couch co-op on a single device or sync across multiple devices over LAN. Online multiplayer and server-hosted worlds are in active development. Level Creation Build your own worlds using the web-based editor. Export as JSON and share with the community, or import others' creations into your game. Videos See Primecraft In Action Real footage of the engine being stress-tested and explored — from physics simulations to procedural planetary constructs. Preview · v0.20.0 AI Returns Home, Planet Travel & Couch Co-Op Fly across planetary space, train an AI to fly itself home, jump into bubble scenes, and play local co-op with synced AI movement — all in one unscripted preview. Engine Test · Physics Low-Fidelity Simulation Stress Testing Discrete Element stress test: thousands of rigid bodies to find the saturation point of the physics solver. Tracking frame latency, CPU physics process time, and static RAM. Full Suite · Tests 1–9 Procedural Planetary Constructs 9 tests in one: SnakeBots, animated skeletons, procedural bestiary (Walkers, Worms, Star-creatures), planetary skyscrapers aligned to spherical surfaces, discovery satellites, defensive grids, and the Great Transfiguration — 150+ magical objects spawned onto a single planet. Watch on YouTube Technical Under the Hood Primecraft is built as a cross-platform simulation engine, not just a game. Godot / C# Runtime Native performance High-performance physics, rendering, and input handling for mobile and desktop builds. Optimized for real-time simulation with thousands of entities. TypeScript / Web Runtime Browser-based tooling Scene editing, constraint systems, and AI tooling that runs directly in your browser. Shares the same scene format for seamless interoperability. Engine Capabilities Deterministic Planet Generator — 8–15 billion reachable locations with consistent generation Embedded Neural Runtime (Loom) — native inference for on-device AI AI Training Layer — movement, control, and companion behaviour learning Authoritative Multiplayer — server architecture in development Web-Based Scene Editor — create puzzles, levels, and simulation experiments FAQ Frequently Asked Questions Is this a game or a research project? Both. Primecraft is a fully playable game, but it's also an experimental engine exploring AI, physics, procedural generation, and distributed input systems. How does the AI work? Each companion uses a real neural network running natively on your device — no cloud, no external servers. You train them through gameplay inside bubble scenes using the Loom framework. Does Primecraft collect my data? No. All AI models and training stay entirely on your device unless you explicitly choose to export or publish your scenes and models. How many planets are there? The coordinate system supports over 8 billion unique planets. Each one is deterministically generated from its grid location, ensuring consistency across sessions. Can I make my own levels? Absolutely! Use the web-based editor to create scenes, then load them directly into the game. Levels are stored as human-readable JSON files. Is multiplayer supported? Couch co-op and LAN play are available now. Online multiplayer with server-hosted bubble scenes is actively in development. Why does Primecraft look chaotic? By design! It's a physics-first sandbox with experimental AI. The emergent chaos is part of the experience — players are encouraged to break things creatively. Audience Who Is Primecraft For? Built for curious minds who love experimentation and creative chaos. Sandbox Enthusiasts Love physics sandboxes and chaotic emergent gameplay AI Hobbyists Train neural networks in a real-time environment Level Creators Build 3D worlds without complex tools Researchers Explore embodied AI and procedural ecosystems Explorers Discover a weird, beautiful universe to mess around in Join the Universe Start building, training, and exploring today. The cosmos awaits. Android Windows Linux Wishlist on Steam --- Source: https://openfluke.com/gallery # Primecraft Scene Gallery — Voxel Worlds by OpenFluke > Browse voxel scenes built in Primecraft while testing the simulation engine—3D worlds, reflex automation, and neural networks from the OpenFluke lab. Canonical: https://openfluke.com/gallery --- Scene Gallery … scenes ← Prev Page 1 of 1 Next → × Scene Name Associations Finding related items... View in Biocraft Lab Copy Path Download JSON --- Source: https://openfluke.com/privacy # Privacy Policy — OpenFluke > How OpenFluke, Loom, SoulGlitch, and Primecraft handle your data. SoulGlitch runs offline on-device; the website uses minimal analytics via Cloudflare. Canonical: https://openfluke.com/privacy --- Legal Privacy Policy Last Updated: November 28, 2025 1 Introduction OpenFluke ("we", "our", or "us") is committed to protecting your privacy. This Privacy Policy explains how your information is handled when you use our services, including Primecraft , Biocraft , and the OpenFluke website . 2 Information Collection We prioritize data minimization. We do not store your personal data on our own servers beyond what is necessary for core functionality. Our services use the following third-party providers: Google Sign-In: Used for authentication. We only receive basic profile information (name, email, profile picture) to identify your account. Google Play Services: Used for achievements, leaderboards, and cloud saves in Primecraft. Google Play Billing: Handles in-app purchases securely. We never see your payment information. 3 How Information is Used Account Management: To identify you and provide access to your Lab and saved content. Game Progress: To save your game state, scenes, and unlocks locally or via cloud sync. Diagnostics: We collect anonymous crash data to improve stability and fix bugs. 4 Data Security We rely on the robust security measures provided by Google Cloud, the Android operating system, and industry-standard encryption to protect your data. While no method of transmission is 100% secure, we use commercially acceptable means to protect your information. 5 Your Rights You have the right to access, correct, or delete your personal information. You can revoke Google sign-in permissions at any time through your Google account settings. Deleting your OpenFluke account will remove all associated data from our systems. 6 Children's Privacy Our services are not directed at children under 13. We do not knowingly collect personally identifiable information from children under 13. If you believe we have collected such information, please contact us immediately. 7 Changes to This Policy We may update this Privacy Policy from time to time. Changes will be posted on this page with an updated revision date. Continued use of our services constitutes acceptance of the updated policy. 8 Contact Us If you have any questions about this Privacy Policy, please reach out: support@openfluke.com --- Source: https://openfluke.com/terms # Terms of Service — OpenFluke > Terms of service for OpenFluke websites, Loom open-source software, SoulGlitch, and Primecraft. Canonical: https://openfluke.com/terms --- Legal Terms of Service Last Updated: December 10, 2025 1 Acceptance of Terms By accessing or using OpenFluke services, including Primecraft , Soulglitch , Biocraft , the OpenFluke website , and the LOOM AI framework , you agree to be bound by these Terms of Service. If you do not agree to these terms, please do not use our services. 2 Description of Services OpenFluke provides a platform for creating, sharing, and running physics simulations and AI-driven experiences. Our services include: Primecraft: A native game application for exploring procedural worlds and training AI companions. Biocraft: A browser-based studio for creating and testing physics scenes. Your Lab: A personal workspace for managing your scenes and AI models. LOOM: An open-source neural network framework for AI training. 3 User Accounts To access certain features, you must create an account using Google Sign-In. You are responsible for maintaining the confidentiality of your account and for all activities that occur under your account. Your account is free and gives you access to your personal Lab, scene publishing, and AI training features. 4 User Content You retain ownership of any content you create using our services ("User Content"). By publishing User Content, you grant OpenFluke a non-exclusive, worldwide, royalty-free license to display and distribute your content through our platforms. You agree not to publish content that: Is illegal, harmful, threatening, abusive, or violates any laws Infringes on intellectual property rights of others Contains malware, viruses, or malicious code Is designed to harm, exploit, or mislead other users 5 Intellectual Property The OpenFluke name, logo, Primecraft, Biocraft, and associated branding are trademarks of OpenFluke. The LOOM AI framework and IsoCard scene format are released under open-source licenses and may be used according to their respective terms. 6 Disclaimer of Warranties Our services are provided "as is" and "as available" without warranties of any kind. We do not guarantee that our services will be uninterrupted, secure, or error-free. Your use of our services is at your own risk. 7 Limitation of Liability To the maximum extent permitted by law, OpenFluke shall not be liable for any indirect, incidental, special, consequential, or punitive damages arising from your use of our services. 8 Termination We reserve the right to suspend or terminate your account at any time for violations of these terms or for any other reason at our discretion. You may delete your account at any time through your account settings. 9 Changes to Terms We may update these Terms of Service from time to time. Material changes will be communicated through our services. Continued use after changes constitutes acceptance of the updated terms. 10 Contact Us If you have any questions about these Terms of Service, please reach out: support@openfluke.com --- Source: https://openfluke.com/docs # Loom Docs — Golang AI Engine Reference > Official Loom Golang AI docs: M-POLY-VTD overview, v0.83 Experimental Hardware Acceleration (SIMD, Apple Metal, Hexagon NPU), v0.81 Intel NPU bridge, v0.80 .entity checkpoints, Go layers & dispatch, GPU/WebGPU, accelerators, quantization, Lucy suites, BitNet CPU, deployment. Canonical: https://openfluke.com/docs --- Loom Documentation M-POLY-VTD Engine Docs Complete reference for Loom's poly package — the Multi-numerical POLYmorphic Volumetric Tiled-tensor Dispatcher that powers every Loom integration. Architecture Overview Quick Reference GitHub Where to start? New? Read Overview → Layers → Training. Deploying to web or mobile? Go to Deployment. Need a snippet? Quick Reference has everything copy-paste ready. Architecture Layers Training Deployment GPU Quick Reference Read docs ## Loom documentation (full text) --- ## M-POLY-VTD: Architecture Overview Source: https://openfluke.com/docs/overview Markdown: https://openfluke.com/docs/overview.md # M-POLY-VTD: Architecture Overview **Multi-numerical POLYmorphic Volumetric Tiled-tensor Dispatcher** M-POLY-VTD is a neural inference and training engine built from first principles in Go. It treats a neural network not as a sequential stack of layers, but as a **spatial 3D grid** where each cell can hold any layer type, and every layer can morph its numerical precision on demand. > [!NOTE] > Current version: **0.81.0 (Accelerator Bridge)**. Previous: **0.80.0 (Native Ship)** · **0.79.0 (Bedrock Validation)**. The Loom stack is **Go + `poly/asm` + WebGPU** (`github.com/openfluke/webgpu@v1.0.4`, wgpu-native v29) plus **`poly/accel`** vendor plugins (experimental). **Numerical Tiling (SC/MC)** is live across all 21 DTypes; **Dense forward** can use Plan 9 assembly via `UseAsmForward`. **v0.81** adds per-layer **Intel OpenVINO CPU + NPU** offload on Linux (Lucy **[9]**). **v0.80** ships **ENTITY** (`.entity`) native checkpoints. See [`v081_release.md`](v081_release.md), [`accelerators.md`](accelerators.md), [`v080_release.md`](v080_release.md), and [`entity.md`](entity.md). --- ## The Full Architecture ``` ┌─────────────────────────────────────────────────────────────────────────────┐ │ M-POLY-VTD ARCHITECTURE │ ├─────────────────────────────────────────────────────────────────────────────┤ │ │ │ ┌──────────────────────────────────────────────────────────────────────┐ │ │ │ POLYGLOT BINDINGS (C-ABI FFI Layer) │ │ │ │ Python │ TS (@openfluke/welvet) │ Flutter (pub.dev welvet) │ C# │ Java │ WASM Browser │ │ └─────────────────────────────┬────────────────────────────────────────┘ │ │ │ │ │ ▼ │ │ ┌──────────────────────────────────────────────────────────────────────┐ │ │ │ VolumetricNetwork (3D Grid) │ │ │ │ │ │ │ │ Depth × Rows × Cols × LayersPerCell │ │ │ │ │ │ │ │ ┌───────────┐ ┌───────────┐ ┌───────────┐ │ │ │ │ │ (0,0,0,0) │ │ (0,0,1,0) │ │ (0,0,2,0) │ ← Depth=0, Row=0 │ │ │ │ │VolumetricL│ │VolumetricL│ │VolumetricL│ │ │ │ │ │ayer │ │ayer │ │ayer │ │ │ │ │ └─────┬─────┘ └─────┬─────┘ └─────┬─────┘ │ │ │ │ │ │ │ │ │ │ │ ┌─────▼─────┐ ┌─────▼─────┐ ┌─────▼─────┐ │ │ │ │ │ (0,1,0,0) │ │ (0,1,1,0) │ │ (0,1,2,0) │ ← Depth=0, Row=1 │ │ │ │ └───────────┘ └───────────┘ └───────────┘ │ │ │ └──────────────────────────────────────────────────────────────────────┘ │ │ │ │ │ ┌─────────────────┼──────────────────────┐ │ │ ▼ ▼ ▼ │ │ ┌───────────────┐ ┌──────────────────┐ ┌───────────────────────────┐ │ │ │ CPU Backend │ │ Step mesh engine │ │ WebGPU Backend (WGPU) │ │ │ │ │ │ │ │ │ │ │ │ ForwardPoly- │ │ StepForward │ │ BeginFrame / FlushFrame │ │ │ │ morphic[T] │ │ StepBackward │ │ DispatchForwardLayer │ │ │ │ │ │ Tween (NTP) │ │ DispatchBackwardLayer │ │ │ │ All 21 DTypes │ │ │ │ WGSL compute shaders │ │ │ └───────────────┘ └──────────────────┘ └───────────────────────────┘ │ │ │ │ │ │ │ └─────────────────┼──────────────────────┘ │ │ ▼ │ │ ┌──────────────────────────────────────────────────────────────────────┐ │ │ │ Vendor Accel Bridge (`poly/accel`) — experimental v0.81 │ │ │ │ Per-layer ExecTarget → C-ABI plugins (Intel OpenVINO CPU+NPU now; │ │ │ │ Qualcomm NPU + Google TPU planned) · complements WebGPU, not a │ │ │ │ replacement · forward-only on vendor path today │ │ │ └──────────────────────────────────────────────────────────────────────┘ │ │ │ │ ┌──────────────────────────────────────────────────────────────────────┐ │ │ │ WeightStore (Morphic Precision Engine) │ │ │ │ │ │ │ │ Master []float32 ──┬──▶ Versions[DTypeFP4] []int8 │ │ │ │ (Source of Truth) ├──▶ Versions[DTypeInt8] []int8 │ │ │ │ ├──▶ Versions[DTypeBinary] []int8 │ │ │ │ └──▶ GPUWeights[DTypeFloat32] *wgpu.Buffer │ │ │ └──────────────────────────────────────────────────────────────────────┘ │ │ │ │ ┌──────────────────────────────────────────────────────────────────────┐ │ │ │ DNA Engine │ │ │ │ ExtractDNA ──▶ LayerSignature[] ──▶ CompareNetworks ──▶ SI Score │ │ │ └──────────────────────────────────────────────────────────────────────┘ │ └─────────────────────────────────────────────────────────────────────────────┘ ``` --- ## The Six Core Pillars ### I. Multi-Numerical Architecture (the "M") The engine natively dispatches forward and backward passes across **21 distinct numerical types** (DTypes), from `float64` all the way down to 1-bit `binary`. Each layer stores its weights in a `WeightStore` that holds a `float32` master copy plus optional converted versions for inference. ``` DType Hierarchy: ┌────────────────────────────────────────────────────────┐ │ High-Precision │ Float64, Int64, Uint64 │ │ Standard │ Float32, Int32, Uint32, Int16, Uint16│ │ Optimized │ Float16, BFloat16, Int8, Uint8 │ │ Low-Bit │ FP8E4M3, FP8E5M2, Int4, Uint4, FP4 │ │ Extreme │ Int2, Uint2, Ternary, Binary │ └────────────────────────────────────────────────────────┘ ``` Layers are not restricted to a single precision. The dispatcher reads `layer.DType`, fetches the right version from the `WeightStore`, and falls back to the master FP32 weights if no converted version exists. See [numerical_types.md](./numerical_types.md) for the full breakdown. ### II. Polymorphic Layer-Morphing (the "POLY") Every layer is a **polymorphic processing unit**. Its numerical representation can be changed at any time via `WeightStore.Morph(dtype)` without reallocating the layer structure. The master FP32 weights are never destroyed—they remain the source of truth. ``` Metamorphosis sequence: FP32 (training) ──▶ Morph(INT8) ──▶ Morph(FP4) ──▶ Morph(Binary) ▲ │ └──── Unpack(dtype) ──── always recoverable ─────────┘ ``` After gradients are applied via `WeightStore.ApplyGradients`, all cached low-bit versions are **automatically cleared**, forcing re-quantization on the next forward pass. ### III. Volumetric Tensor Dispatch (the "VTD") The network is a **4D array** of `VolumetricLayer` values indexed by `(Depth, Row, Col, LayerIndex)`. The flattened index is: ``` idx = z * Rows * Cols * LayersPerCell + y * Cols * LayersPerCell + x * LayersPerCell + l ``` Data flows through the grid in reading order: Z outer loop, then Y, then X, then L. This gives the programmer a spatial metaphor to compose complex non-linear topologies. #### Remote Links (Spatial Hopping) Any layer can set `IsRemoteLink = true` and point to any other coordinate via `TargetZ / TargetY / TargetX / TargetL`. When the step mesh engine fires that layer, it reads input from the *target* coordinate's output buffer instead of the preceding layer. This enables biological-style feedback loops anywhere in the grid. ``` Normal flow: Remote link (skip connection): (0,0,0) (0,0,0) │ │ ◄────────────────────────┐ ▼ ▼ │ (0,0,1) (0,0,1) ─ IsRemoteLink ──▶ (0,2,3) │ │ ▼ ▼ (0,0,2) (0,0,2) ``` ### IV. The Dispatcher Pattern `DispatchLayer[T]` and `DispatchLayerBackward[T]` are **generic runtime jump tables**. They inspect `layer.Type` and call the correct polymorphic function, returning `(preAct, postAct)` tensors of the same type `T`. The separation from the grid traversal loop makes GPU kernel fusion possible—the driver can look ahead and pre-load the next tile's weights while the current tile computes. ```go func DispatchLayer[T Numeric](layer *VolumetricLayer, input, skip *Tensor[T]) (preAct, postAct *Tensor[T]) ``` There are 19 `LayerType` values routed here. An unknown type falls through to `DenseForwardPolymorphic`. **Numerical tiling** is orthogonal to volumetric traversal: `ForwardPolymorphic` can walk the grid in spatial tiles or sequentially (`network.UseTiling`). **CPU** layers use a **single** tile map (`CPUTileSizes`). **GPU** layers carry **two** maps (`GPUSCTileSizes`, `GPUMCTileSizes`); **`EnableMultiCoreTiling`** on `VolumetricNetwork` selects MC vs SC dispatch (see [dispatch.md](./dispatch.md) and [gpu.md](./gpu.md)). ### V. The Step Mesh Engine Unlike `ForwardPolymorphic`, which executes the entire network per input in one pass, `StepForward` fires **all layers simultaneously** every clock cycle. Each layer reads from the previous cycle's output buffer (`LayerData`) and writes to `NextBuffer`. After all layers have fired, the buffers are swapped. This double-buffering pattern is race-condition-free and supports parallel tile dispatch via goroutines. ### VI. The DNA Engine `ExtractDNA` converts a network into a slice of `LayerSignature` values. Each signature contains the layer's 3D coordinates, type, DType, and a **normalized** (unit-vector) representation of its weights after precision simulation. `CompareNetworks(dna1, dna2)` then uses cosine similarity to produce an `OverallOverlap` score and identifies `LogicShift` events where a functional pattern has migrated to a different spatial coordinate. --- ## Key Types at a Glance | Type | File | Role | |:-----|:-----|:-----| | `VolumetricNetwork` | `poly.go` | The 3D grid container | | `VolumetricLayer` | `poly.go` | A single processing unit with coordinates | | `WeightStore` | `weights.go` | Master FP32 + versioned low-bit storage | | `Tensor[T Numeric]` | `poly.go` | Generic data container with `Shape` and `Nested` | | `DType` | `poly.go` | 21-value enum for numerical types | | `LayerType` | `poly.go` | 19-value enum for layer kinds | | `WGPUContext` | `wgpu_context.go` | GPU device, queue, pipeline cache | | `StepState[T]` | `step.go` | Double-buffered temporal mesh state | | `NetworkDNA` | `dna.go` | `[]LayerSignature` topological blueprint | | `TrainingConfig` | `training.go` | Epochs, LR, loss type, GPU flag | --- ## The `Tensor[T]` Type ```go type Tensor[T Numeric] struct { Data []T DType DType Shape []int Nested []*Tensor[T] // activation tree for Parallel/Sequential layers } ``` `Nested` is the key structural innovation. During a `ParallelForward` pass, each branch produces its own `preAct` tensor, and these are stored in `Nested` on the returned preAct. The backward pass reads them back, routing gradients to the correct branch without any external bookkeeping. This recursive tree property makes arbitrary nesting of `Parallel` and `Sequential` layers fully differentiable. --- ## Performance Snapshot From the README benchmark table, measured on GTX 1650 Super (Vulkan/WebGPU). **v0.80** production GPU uses **openfluke/webgpu v1.0.4** (wgpu-native v29); Lucy Poly Talk on RTX 3050 Mobile reaches ~**69 tok/s decode** / ~**492 tok/s prefill** for SmolLM2-135M Q4 (custom WGSL — not Ollama-class, but validated on Metal, Win ARM64, Intel, and NVIDIA Vulkan). | Layer type | CPU Tiled | GPU | Speedup | |:-----------|:----------|:----|:--------| | Dense | 5.42ms | 400µs | 13.6x | | CNN 1D | 4.34ms | 195µs | 22.3x | | CNN 2D | 182ms | 100µs | 1826x | | CNN 3D | 1522ms | 200µs | 7602x | | RMSNorm | 1.16ms | 103µs | 11.3x | End-to-end GPU training (20 epochs): | Architecture | CPU | GPU | Speedup | |:-------------|:----|:----|:--------| | Dense MLP (128→512→512→8) | 12.1s | 693ms | 17.5x | | CNN 2D (3ch×32×32 → 16f→32f→8) | 1m57s | 1.81s | 64.8x | | Deep Dense (128→512×4→8) | 31.7s | 1.23s | 25.7x | --- ## Next Steps - [v081_release.md](./v081_release.md) — **v0.81.0** Accelerator Bridge, Intel NPU, Lucy [9] - [accelerators.md](./accelerators.md) — vendor NPU/TPU plugins, `SyncToAccel`, OpenVINO - [v080_release.md](./v080_release.md) — **v0.80.0** ENTITY, WebGPU v1.0.4, GPU validation - [entity.md](./entity.md) — `.entity` native checkpoint format - [numerical_types.md](./numerical_types.md) — DType system, WeightStore, Metamorphosis - [layers.md](./layers.md) — Every layer type in detail - [dispatch.md](./dispatch.md) — The dispatcher pattern and 3D coordinates - [training.md](./training.md) — Forward/backward, optimizers, Tween - [gpu.md](./gpu.md) — WebGPU backend and BeginFrame/FlushFrame pattern - [step.md](./step.md) — The step mesh engine - [quick_reference.md](./quick_reference.md) — Common code snippets --- ## Deployment: TypeScript, WASM, Flutter, and NPM Source: https://openfluke.com/docs/deployment Markdown: https://openfluke.com/docs/deployment.md # Deployment: TypeScript, WASM, Flutter, and NPM Loom is designed to be **isomorphic** across runtimes: the same mathematical engine runs in Node.js and the browser (WASM), and the same **Welvet C-ABI** is exposed to **Flutter/Dart** via [`welvet` on pub.dev](https://pub.dev/packages/welvet). --- ## 📱 Flutter & Dart Use the FFI plugin for mobile and desktop Flutter apps — no Python or Node runtime. ```yaml dependencies: welvet: ^0.80.4 ``` ```dart import 'package:welvet/loom_ffi.dart'; if (!loomAvailable) throw StateError(loomLibLastError ?? 'welvet'); final handle = loomLib.createNetwork(jsonConfig); ``` - **Full guide:** [flutter.md](flutter.md) - **Interactive demos:** [loom-flutter-quickstart](https://github.com/openfluke/loom-flutter-quickstart) (30+ CPU examples, Apache-2.0) - **Production reference:** SoulGlitch (Play Store) embeds the same C-ABI on Android/iOS Desktop natives ship in the pub.dev package; iOS/Android require vendored binaries (see [flutter.md — Platform notes](flutter.md#platform-notes)). **Building Linux Welvet from source:** `cd welvet/cabi/internal/build && ./build_linux.sh` → `dist/linux_amd64/welvet.so` + `welvet.h` for FFI packaging. --- ## 📦 The NPM Package: `@openfluke/welvet` The primary way to use Loom in the JavaScript ecosystem is through the **Welvet** SDK. ### Installation ```bash npm install @openfluke/welvet ``` ### Quick Start (Node.js) ```typescript import { init, createNetwork } from '@openfluke/welvet'; // Initialize the WASM runtime await init(); // Build a network from a JSON specification const net = await createNetwork({ id: "demo-net", depth: 1, rows: 2, cols: 1, layers_per_cell: 1, layers: [ { z: 0, y: 0, x: 0, l: 0, type: "Dense", input_height: 128, output_height: 64, activation: "ReLU" }, { z: 0, y: 1, x: 0, l: 0, type: "Dense", input_height: 64, output_height: 10, activation: "Linear" } ] }); // Run a forward pass const input = new Float32Array(128).fill(0.5); const output = await net.sequentialForward(input); console.log("Network output:", output); ``` --- ## 🌐 WASM & FFI Bridge The TypeScript SDK communicates with the Go-compiled core via the **Universal C-ABI**. This ensures that complex logic (like NEAT evolution or DNA extraction) remains fast while providing a high-level, idiomatic JS interface. ### Verified Capabilities (v0.74.0) The isomorphic bridge has been verified through a 36-count diagnostic suite: - **Core Exports**: 8/8 internal WASM symbols verified. * **Network Methods**: 16/16 functional wrappers (Forward, DNA, Morph, etc.) passed. * **NEAT Population**: 8/8 evolutionary logic methods verified. * **Bit-Perfect Parity**: 0.000000% divergence vs the Go native reference. --- ## 🖼️ Browser Deployment (WebGPU) When running in the browser, the WASM runtime can automatically detect and utilize **WebGPU** for massive parallel speedups. ```typescript import { setupWebGPU } from '@openfluke/welvet'; // Initialize WebGPU context await setupWebGPU(); // Networks created after this point will utilize GPU kernels // for forward and backward passes. ``` ### Performance Tiers | Environment | Backend | Best For | | :--- | :--- | :--- | | **Node.js** | WASM (SIMD) | Backend inference, server-side DNA comparison. | | **Browser** | WASM + WebGPU | High-performance interactive AI, on-device training. | | **Mobile Web** | WASM | Lightweight edge execution. | --- ## 🧬 DNA & Evolution in JS The TypeScript SDK provides full access to the DNA logic: - **`net.extractDNA()`**: Generates a topological fingerprint. - **`compareLoomDNA(dnaA, dnaB)`**: Cross-platform similarity score. - **`createLoomNEATPopulation(id, size, cfg)`**: High-speed evolutionary architecture search. For more details on the underlying DNA math, see [dna.md](dna.md). --- ## Flutter & Dart: `welvet` on pub.dev Source: https://openfluke.com/docs/flutter Markdown: https://openfluke.com/docs/flutter.md # Flutter & Dart: `welvet` on pub.dev Loom runs in **Flutter** and plain **Dart** through the [`welvet`](https://pub.dev/packages/welvet) FFI plugin. Same Welvet C-ABI as Python ctypes and TypeScript WASM — `createNetwork`, polymorphic forward/backward, CPU training, DNA, JSON + `.entity` checkpoints, mesh step, and LLM exports when your native build includes them. > **v0.80.6** — Federated natives on pub.dev: main `welvet` (Dart API) plus `welvet_linux`, `welvet_windows`, `welvet_android`, `welvet_apple` (iOS + macOS) for per-OS binaries under the 100 MB limit. --- ## Installation Add to `pubspec.yaml` — **no monorepo path required**: ```yaml dependencies: welvet: ^0.80.4 ``` ```bash flutter pub get ``` ```dart import 'package:welvet/loom_ffi.dart'; import 'package:welvet/welvet.dart'; // welvetVersion, seven_layer_runner print(welvetVersion); // 0.80.4 print(loomAvailable); // true when libwelvet loaded ``` If `loomAvailable` is false, read `loomLibLastError` — usually a missing native library on that platform. --- ## Quick start: dense forward Networks are JSON volumetric grids (`depth × rows × cols`, layers at `(z, y, x, l)`). See [overview.md](overview.md). ```dart import 'dart:convert'; import 'dart:math' as math; import 'package:welvet/loom_ffi.dart'; if (!loomAvailable) throw StateError(loomLibLastError ?? 'welvet not loaded'); final netJson = jsonEncode({ 'id': 'flutter-demo', 'depth': 1, 'rows': 1, 'cols': 1, 'layers_per_cell': 2, 'layers': [ { 'z': 0, 'y': 0, 'x': 0, 'l': 0, 'type': 'dense', 'dtype': 'float32', 'input_height': 16, 'output_height': 8, 'activation': 'relu', }, { 'z': 0, 'y': 0, 'x': 0, 'l': 1, 'type': 'dense', 'dtype': 'float32', 'input_height': 8, 'output_height': 4, 'activation': 'linear', }, ], }); final handle = loomLib.createNetwork(netJson); final inp = List.generate(16, (i) => 0.2 * math.sin(i * 0.2)); const inShape = [1, 16]; loomLib.configureTrainingMode(handle, 2); // CPU MC final out = loomParseFloatArray( loomLib.forwardPolymorphic(handle, inp, inShape), ); // out.length == 4 loomLib.freeNetwork(handle); ``` --- ## Runnable examples (Dart CLI) `welvet/dart/tool/examples/` mirrors Python and npm quick-starts: | Script | API surface | |--------|-------------| | `01_dense_forward.dart` | `forwardPolymorphic` + `sequentialForward` | | `02_morph_and_train.dart` | `morphLayer(INT8)`, `train()` on CPU MC | | `03_save_reload.dart` | `serializeNetwork` / `deserializeNetwork`, `.entity` roundtrip | | `04_mha_forward.dart` | MHA with `[batch, seq, d_model]` | | `05_dna_compare.dart` | `extractDNA`, `compareDNA` | ```bash cd welvet/dart bash tool/copy_native.sh cd tool/examples && dart pub get && dart run run_all.dart ``` --- ## Interactive Flutter showcase app **[loom-flutter-quickstart](https://github.com/openfluke/loom-flutter-quickstart)** — Apache-2.0 demo app with **30+ CPU demos** (no `path:` dependency on the Loom repo): ```bash git clone https://github.com/openfluke/loom-flutter-quickstart.git cd loom-flutter-quickstart flutter pub get flutter run -d macos # or linux / windows ``` Categories in the app: - System — version, `loomAvailable`, C-ABI methods manifest - Forward — dense stack, MHA, volumetric grid - Training (CPU) — morph dtypes, `LoomTrain`, backward polymorphic, SC vs MC - Mesh — `LoomStep`, mesh backward, `applyGradients` - Persistence — JSON wire + `.entity` checkpoint - DNA & meta — compare/splice DNA, blueprint, NEAT/tween defaults, telemetry - Layer quick forward — Dense, MHA, CNN×3, RNN, LSTM, SwiGLU, Embedding, Residual - Seven-layer suite — full CPU parity suite per layer type (Float32) GPU training and bundled on-device LLM inference are intentionally **not** exercised in the showcase (GPU train is still flaky; LLM needs a local HuggingFace snapshot). --- ## Common patterns ### Morph precision ```dart loomLib.morphLayer(handle, 0, LoomLib.dtypeInt8); loomLib.syncInferenceWeights(handle); ``` ### CPU training ```dart final raw = loomLib.train( handle, flatInput, flatTarget, batchSize: 1, inDim: 16, outDim: 8, epochs: 10, learningRate: 0.05, mode: 2, // CPU MC — use 1 for CPU SC inputShape: [1, 16], targetShape: [1, 8], ); final hist = (jsonDecode(raw)['loss_history'] as List).cast(); ``` ### Backward (gradients) ```dart final bwd = loomLib.backwardPolymorphic( handle, input, inShape, target, targetShape, ); final m = loomParseResult(bwd); // dx, dw maps ``` ### Mesh step ```dart final state = loomLib.createStepState(handle); loomLib.setInput(state, input); loomLib.meshStep(handle, state); final layerOut = loomParseFloatArray(loomLib.getOutput(state, 0)); loomLib.freeStepState(state); ``` ### JSON checkpoint ```dart final wire = loomLib.serializeNetwork(handle); final reloaded = loomLib.deserializeNetwork(wire); loomLib.syncInferenceWeights(reloaded); ``` ### Seven-layer CPU suite (Lucy [7] parity) ```dart import 'package:welvet/welvet.dart'; final result = runSevenLayerSuite( layerFilter: 'Dense', dtypeFilter: 'Float32', onLog: print, ); print('passed=${result.passed} failed=${result.failed}'); ``` Layer types: `Dense`, `SwiGLU`, `MHA`, `CNN1`, `CNN2`, `CNN3`, `RNN`, `LSTM`, `Embedding`, `Residual`. ### SoulGlitch-style LLM (advanced) When natives include LLM symbols and you ship a model snapshot + tokenizer: ```dart final llm = loomLib.createLLM(snapshotDir, execMode: 3, precision: 4); final reply = loomLib.llmGenerate(llm, systemPrompt, userMessage); loomLib.freeLLM(llm); ``` SoulGlitch uses path-vendored `welvet` + XCFramework on iOS; see SoulGlitch macOS/iOS Pod setup for production mobile. --- ## Platform notes | Platform | Package | Notes | |----------|---------|--------| | **macOS** | [`welvet_apple`](https://pub.dev/packages/welvet_apple) | `libwelvet.dylib` in plugin Frameworks | | **Linux** | [`welvet_linux`](https://pub.dev/packages/welvet_linux) | x86_64 + ARM64 `.so` (auto via `welvet` dep) | | **Windows** | [`welvet_windows`](https://pub.dev/packages/welvet_windows) | x86_64 + ARM64 `.dll` | | **Android** | [`welvet_android`](https://pub.dev/packages/welvet_android) | arm64-v8a + x86_64 | | **iOS** | [`welvet_apple`](https://pub.dev/packages/welvet_apple) | `Welvet.xcframework` | | **Web** | — | Use [@openfluke/welvet](deployment.md) WASM in browser | Add only `welvet: ^0.80.6` in `pubspec.yaml` — Flutter pulls the correct impl package per platform. Monorepo developers can refresh natives: ```bash cd welvet/cabi/internal/build && ./build_unix.sh all cd ../../dart && bash tool/copy_native.sh --all ``` Local monorepo dev uses `pubspec_overrides.yaml` for path impl packages (not published). --- ## Related docs - [deployment.md](deployment.md) — TypeScript/npm and WASM - [training.md](training.md) — CPU SC/MC modes, loss types - [entity.md](entity.md) — `.entity` native checkpoints - [dna.md](dna.md) — `extractDNA`, `compareDNA`, `spliceDNA` - [step.md](step.md) — mesh clock engine - [testing_and_validation.md](testing_and_validation.md) — seven-layer suite legend - [welvet on pub.dev](https://pub.dev/packages/welvet) - [welvet/dart README](../welvet/dart/README.md) — package source --- ## Donate compute (TCP) Source: https://openfluke.com/docs/donate-compute Markdown: https://openfluke.com/docs/donate-compute.md # Donate compute (TCP) The **`donate_compute_*.go`** files in `poly/` implement an optional **TCP protocol** so a **donor** machine can accept inference-style work from clients on the same network (or loopback). Work is exchanged as **length-prefixed JSON** frames over a single connection — there is no HTTP server inside `poly` for this path. **Status:** The server’s inference and prompt paths are **stubs** (`stubInfer` / `stubPrompt`) until wired to real model loading, `poly` execution, or subprocess hooks. --- ## Why it exists - **LAN-friendly**: bind to `0.0.0.0` (or a specific interface) and let another host submit jobs without bundling a separate HTTP stack in `poly`. - **Two modes** (see below): push weights + token **`infer`**, or **`prompt`**-only against a local LM path advertised in the hello. --- ## Wire format (`donate_compute_framing.go`) Each message is: 1. **`uint32` length**, little-endian (4 bytes). 2. **UTF-8 JSON object** of exactly that length. Constants: | Constant | Value | Meaning | | :--- | :--- | :--- | | `DonateComputeDefaultPort` | **17001** | Default listen/dial port (adjacent to construct TCP dev on **17000**). | | `MaxDonateFrameBytes` | 64 MiB | Maximum single-frame payload; large models use **many** weight chunks, not one giant frame. | Helpers: **`WriteDonateFrame`**, **`ReadDonateFrame`**. --- ## Message types (`donate_compute_types.go`) Version **v1** uses a `"type"` string discriminator. Constants include: - **`hello`** — first frame from server after connect; client may echo hello. - **`model_begin`**, **`weights_chunk`**, **`model_commit`**, **`model_status`** — **model_push** upload lifecycle. - **`infer`**, **`infer_result`** — token-ID jobs against a **mounted** pushed model. - **`prompt`**, **`prompt_result`** — text jobs for **local_lm** nodes. - **`queue_status`**, **`error`** — optional / error paths. Structs (`DonateHello`, `DonateModelBegin`, `DonateWeightsChunk`, `DonateInfer`, `DonatePrompt`, …) mirror the JSON fields. --- ## Server (`donate_compute_server.go`) **`ServeDonateComputeTCP(opts DonateComputeServerOptions)`** returns a **`net.Listener`**. It: - Sends a **`DonateHello`** immediately after each accept (mode, role `server`, optional `LocalLmPath`, queue capacity hint). - Parses frames in a loop per connection. - Enqueues **`infer`** / **`prompt`** jobs on a **global FIFO channel**; **one worker** drains the queue (serial execution — not N parallel model mounts). **`DonateComputeServerMode`:** | Mode | Behavior | | :--- | :--- | | **`model_push`** | Client sends **`model_begin`** (config JSON + expected weight length), **`weights_chunk`** (base64 slices), **`model_commit`**. Server acknowledges with **`model_status`**. Then client may send **`infer`** with `input_ids` / `max_tokens`. | | **`local_lm`** | **`infer`** is rejected; client uses **`prompt`** with full text. Server may advertise **`LocalLmPath`** in hello (informational). | **`CloseDonateListener`** closes the listener. --- ## Client (`donate_compute_client.go`) - **`DialDonateCompute(addr)`** — TCP dial (default `127.0.0.1:17001` if empty), read server hello, send client hello; returns **`DonateClient`** + **`DonateHello`**. - **`PutModel(configJSON, weights)`** — stream model for **model_push** nodes. - **`EnqueueInfer`**, **`EnqueuePrompt`** — send one job and wait for the matching result frame. --- ## Tests **`donate_compute_test.go`** covers framing and client/server interaction. --- ## Security **v1 has no TLS and no authentication.** It is intended for **trusted networks** (e.g. same Wi‑Fi lab). Do not expose the raw port to the public Internet without a VPN, SSH tunnel, or application-layer gateway. --- ## File map | File | Role | | :--- | :--- | | `donate_compute_types.go` | v1 constants and JSON structs | | `donate_compute_framing.go` | Frame encode/decode, default port, size limits | | `donate_compute_server.go` | TCP server, modes, queue, stubs | | `donate_compute_client.go` | Dial, `PutModel`, `EnqueueInfer`, `EnqueuePrompt` | | `donate_compute_test.go` | Tests | --- ## TANHI — UDP layer telemetry Source: https://openfluke.com/docs/tanhi Markdown: https://openfluke.com/docs/tanhi.md # TANHI — UDP layer telemetry **TANHI** streams **sparse, non-blocking JSON-line events** over **UDP** so external tools (notably the **SoulGlitch → TANHI** HUD) can visualize **per-layer forward/backward** activity, timing, dtypes, and **routing links** (parallel branches / sequential substeps). Implementation: **`poly/tanhi.go`**. Integration hooks live in **`poly/forward.go`**, **`poly/backward.go`**, and **`poly/wgpu_forward.go`** (GPU transformer path). Optional **Welvet C-ABI** exports: **`welvet/cabi/tanhi_ext.go`**. --- ## Defaults | Constant / env | Default | Meaning | | :--- | :--- | :--- | | **`poly.DefaultTanhiUDPPort`** | **17481** | UDP destination port (IANA unassigned range). | | Host | `127.0.0.1` | When `TanhiUDPConfig.Host` is empty. | | Disabled | `nil` / off | `VolumetricNetwork.Tanhi == nil` or `Enabled == false` → no UDP. | --- ## Configuration (`TanhiUDPConfig`) Set on **`VolumetricNetwork.Tanhi`**: - **`Enabled`** — master switch. - **`Host`**, **`Port`** — UDP listener address (engine **sends** to this address). - **`SendShape`** — include approximate tensor **`shape`** in each event (CPU path uses activations when available; GPU path uses **`TanhiGPULayerShapeHint`** — no readback). Telemetry is **best-effort**: a buffered queue (**1024** packets); **overflow drops** silently so training/inference never blocks on HUD lag. --- ## Wire format Each datagram payload is **one JSON object per line** (newline-terminated). Schema version **`v`: `"tanhi1"`**. Typical fields: | Field | Meaning | | :--- | :--- | | `seq` | Monotonic sequence number | | `phase` | `"fwd"` or `"bwd"` | | `idx` | Layer index in traversal (-1 or special indices possible on GPU fused paths) | | `z`, `y`, `x`, `l` | Volumetric coordinate | | `layer` | Layer type string | | `dtype` | Integer dtype code | | `connections` | Fan-out hint from weight masters (or override for GPU LM head) | | `t0_ns`, `t1_ns` | Wall-clock nanoseconds around the layer | | `shape` | Optional shape slice when `SendShape` is true | | `links` | Optional routing targets for **LayerParallel** / **LayerSequential** (capped, for arc drawing) | --- ## SoulGlitch / Glitch CLI - **`GLITCH_TANHI=1`** — enable when running **`loom/glitch`** interactively (or answer the prompt). - **`TANHI_HOST`**, **`TANHI_PORT`**, **`TANHI_SHAPE=1`** — override host, port, and shape inclusion (same conventions in **`glitch/measure/*`** harnesses). Open SoulGlitch **first**; set the listener **port** to match **`TANHI_PORT`** / **17481**. --- ## C-ABI (Welvet) - **`LoomNetworkTanhiConfigure`** — enable/disable, host C string, port (0 → default **17481**), send-shape flag. - **`LoomNetworkTanhiDisable`** — clear `Tanhi` on the network handle. - **`LoomTanhiDefaultPort`** — returns **`DefaultTanhiUDPPort`**. --- ## Security note UDP telemetry is **localhost-oriented** by default. Pointing **`Host`** at a remote machine sends layer metadata and timing to that address — use only on **trusted networks** when not using **`127.0.0.1`**. --- ## Numerical Types, DType System, and WeightStore Source: https://openfluke.com/docs/numerical-types Markdown: https://openfluke.com/docs/numerical-types.md # Numerical Types, DType System, and WeightStore This document covers all 21 `DType` values, the `Numeric` generic constraint, the `WeightStore` master/versioned architecture, and the Metamorphosis mechanism that lets a layer switch precision on the fly. --- ## The 21 DTypes ```go type DType int ``` Every `VolumetricLayer` carries a `DType` field that controls which numerical format its weights are active in. The full set: ``` ┌─────┬───────────────┬──────────────────────────────────────────────┐ │ ID │ Name │ Description │ ├─────┼───────────────┼──────────────────────────────────────────────┤ │ 0 │ DTypeFloat64 │ IEEE 754 double (8 bytes per weight) │ │ 1 │ DTypeFloat32 │ Standard single (4 bytes) — Master baseline │ │ 2 │ DTypeFloat16 │ 16-bit float (simulated, stored as f32) │ │ 3 │ DTypeBFloat16 │ Brain Float: 8 exp bits, 7 mantissa │ │ 4 │ DTypeFP8E4M3 │ 8-bit FP, 4-exponent 3-mantissa │ │ 5 │ DTypeFP8E5M2 │ 8-bit FP, 5-exponent 2-mantissa │ │ 6 │ DTypeInt64 │ 64-bit signed integer │ │ 7 │ DTypeInt32 │ 32-bit signed integer │ │ 8 │ DTypeInt16 │ 16-bit signed integer │ │ 9 │ DTypeInt8 │ 8-bit signed integer (0.625–1.0 B/weight) │ │ 10 │ DTypeUint64 │ 64-bit unsigned integer │ │ 11 │ DTypeUint32 │ 32-bit unsigned integer │ │ 12 │ DTypeUint16 │ 16-bit unsigned integer │ │ 13 │ DTypeUint8 │ 8-bit unsigned integer │ │ 14 │ DTypeInt4 │ 4-bit signed (2 weights per byte) │ │ 15 │ DTypeUint4 │ 4-bit unsigned (2 weights per byte) │ │ 16 │ DTypeFP4 │ 4-bit floating point E2M1 (2 per byte) │ │ 17 │ DTypeInt2 │ 2-bit signed (4 weights per byte) │ │ 18 │ DTypeUint2 │ 2-bit unsigned (4 weights per byte) │ │ 19 │ DTypeTernary │ 2-bit ternary: -1, 0, +1 │ │ 20 │ DTypeBinary │ 1-bit XNOR-Net (8 weights per byte) │ └─────┴───────────────┴──────────────────────────────────────────────┘ ``` ### Storage Size per Weight ``` ┌────────────────────────────────────────────────────────┐ │ DType Bits/weight Bytes/1024 weights │ ├────────────────────────────────────────────────────────┤ │ Float64 64 8192 │ │ Float32 32 4096 │ │ Float16 16 2048 │ │ BFloat16 16 2048 │ │ FP8E4M3 8 1024 │ │ FP8E5M2 8 1024 │ │ Int8/Uint8 8 1024 │ │ Int4/Uint4 4 512 (2 per byte) │ │ FP4 4 512 (2 per byte) │ │ Int2/Uint2 2 256 (4 per byte) │ │ Ternary 2 256 (4 per byte) │ │ Binary 1 128 (8 per byte) ← 98.4% │ │ compression vs FP32 │ └────────────────────────────────────────────────────────┘ ``` ### Parsing DTypes from Strings `ParseDType(s string) DType` accepts aliases: | Input strings | Result | |:-------------|:-------| | `"float32"`, `"fp32"`, `"f32"` | `DTypeFloat32` | | `"bfloat16"`, `"bf16"` | `DTypeBFloat16` | | `"fp8e4m3"`, `"fp8"` | `DTypeFP8E4M3` | | `"int4"` | `DTypeInt4` | | `"fp4"`, `"f4"` | `DTypeFP4` | | `"ternary"` | `DTypeTernary` | | `"binary"` | `DTypeBinary` | --- ## The `Numeric` Constraint ```go type Numeric interface { ~int | ~int8 | ~int16 | ~int32 | ~int64 | ~uint | ~uint8 | ~uint16 | ~uint32 | ~uint64 | ~float32 | ~float64 } ``` This constraint makes `Tensor[T]`, `DispatchLayer[T]`, `ForwardPolymorphic[T]`, and all other generic functions work across any of Go's numeric primitives. The constraint is deliberately limited to types the compiler can generate native arithmetic for—no reflection, no `interface{}` boxing at the hot path. > [!NOTE] > FP4, FP8, BFloat16, and other non-native types are **simulated** via PTQ at GPU upload (`MorphToFloat32ForGPU`). **Default CPU training** (`UseExactDType = false`) also uses FP32 surrogate math via `GetActive` dequant — QAT-like. **Native exact training** (`UseExactDType = true`, `*_native.go`) runs forward/backward in storage-dtype rules; true integers (Int8, …) use real int8 MAC and in-place int8 updates. See [training.md — Training paradigms](training.md#training-paradigms-default-qat-like-vs-native-exact). --- ## The WeightStore ```go type WeightStore struct { Master []float32 // Source of truth — always FP32 Versions map[DType]any // Cached conversions (e.g., []int8 for INT8) GPUWeights map[DType]any // VRAM-resident wgpu.Buffer references GPUScales map[DType]*wgpu.Buffer // Per-block scale buffers for quantized types Scale float32 // Global quantization scale factor } ``` The `Master` slice is allocated with `AlignedFloat32(n)` which aligns to 64-byte boundaries (one CPU cache line), enabling AVX-width SIMD operations. ### Creating and Initializing ```go ws := NewWeightStore(inputSize * outputSize) ws.Scale = 1.0 ws.Randomize(seed, 0.1) // fills Master with uniform [-0.1, 0.1] ``` After `Randomize`, all `Versions` and `GPUWeights` maps are cleared, ensuring no stale low-bit versions survive. ### The Morphic Version System ``` WeightStore.Morph(dtype DType): Master (FP32) │ ▼ DTypeFloat64 ──▶ []float64 (direct cast) DTypeBFloat16 ──▶ []float32 (bits masked to 16-bit BF16) DTypeInt8 ──▶ []int8 (quantized: int8(v / Scale)) DTypeInt4 ──▶ []int8 (quantized, stored 1-per-int8) DTypeBinary ──▶ []int8 (sign bit only: +1 or -1) ``` The BFloat16 path uses a bit-masking trick: ```go u32 := math.Float32bits(wVal) u32 &= 0xFFFF0000 // zero the lower 16 mantissa bits return math.Float32frombits(u32) ``` This preserves the exponent and upper mantissa exactly as BFloat16 would. ### Metamorphosis: Switching Precision On the Fly A layer starts life as FP32. Before inference you can call: ```go layer.WeightStore.Morph(DTypeInt8) layer.DType = DTypeInt8 ``` Now `DenseForwardPolymorphic` will find the `[]int8` version in `Versions[DTypeInt8]` and use the INT8 fast-path loop **when `UseExactDType` is false** (default dequant matmul). With `UseExactDType = true`, `DenseForwardNativeExact` uses true int8 MAC instead. The FP32 master is untouched until `ApplyGradients` (default) or in-backward int8 update (true native). After training (`ApplyGradients`), the master is updated and **all cached versions are automatically purged**: ```go func (ws *WeightStore) ApplyGradients(gradWeights *Tensor[float32], lr float32) { for i := 0; i < limit; i++ { ws.Master[i] -= lr * gradWeights.Data[i] } // Stale — force re-quantize on next forward: ws.Versions = make(map[DType]any) ws.GPUWeights = make(map[DType]any) } ``` This guarantees the layer never silently uses outdated quantized weights. ``` ┌──────────────────────────────────────────────────────────────┐ │ Metamorphosis Lifecycle │ ├──────────────────────────────────────────────────────────────┤ │ │ │ NewWeightStore(n) │ │ │ │ │ ▼ │ │ Randomize(seed, scale) ──▶ Master filled, Versions={} │ │ │ │ │ ▼ │ │ layer.DType = DTypeInt8 │ │ │ │ │ ▼ │ │ Forward() ──▶ Morph(DTypeInt8) if Versions[INT8]==nil │ │ │ │ │ │ │ Versions[DTypeInt8] = []int8{...} │ │ │ │ │ ▼ │ │ INT8 fast-path arithmetic executes │ │ │ │ │ ▼ │ │ ApplyGradients(gW, lr) ──▶ Master updated │ │ ──▶ Versions = {} (cleared) │ │ │ │ Next Forward() ──▶ Morph(DTypeInt8) again from new Master │ │ │ └──────────────────────────────────────────────────────────────┘ ``` ### Unpacking for Deserialization When loading a model saved in a low-bit format: ```go ws.Versions[dtype] = decoded // e.g., []int8 from bit-packed JSON ws.Unpack(dtype) // reconstructs Master: Master[i] = packed[i] * Scale ``` This ensures the FP32 master is always available for gradient-based fine-tuning, even on a model that was serialized in INT4. --- ## MorphToFloat32ForGPU This is the PTQ simulation path used when uploading weights to the GPU for layers without a dedicated packed shader (CNN1-3, RNN, LSTM, Embedding): ```go func (ws *WeightStore) MorphToFloat32ForGPU(dtype DType) []float32 ``` It calls `ws.Morph(dtype)` to produce the quantized version, then dequantizes back to float32 by multiplying by `ws.Scale`. The GPU shader sees float32 weights that already reflect quantization rounding loss — no new shader needed. | DType | Round-trip behaviour | |:------|:---------------------| | Float32, Float64 | Master returned as-is (no loss) | | BFloat16 | Upper 16 bits of mantissa preserved; lower 16 zeroed | | FP8, Int8, Uint8 | `round(w/scale) * scale` | | Int4, Uint4, FP4 | `trunc(w/scale) * scale`, range ±7 | | Int2, Uint2 | 4-level round-trip | | Ternary | Threshold snap to `{-scale, 0, +scale}` | | Binary | Sign only: `±scale` | The `scale` comes from `WeightStore.Scale`, set during `Morph` from the max absolute value of the master weights. --- ## The Q4_0 Block Format (GPU Quantization) For GPU inference, the engine uses the Q4_0 block format, matching llama.cpp compatibility: ``` Q4_0Block: ┌────────────────────────────────────────────────────────┐ │ Scale: float32 (4 bytes) │ │ Weights: [16]byte (32 nibbles = 32 × 4-bit weights) │ │ │ │ Total: 20 bytes for 32 weights = 0.625 bytes/weight │ └────────────────────────────────────────────────────────┘ ``` `QuantizeQ4_0(weights []float32) []Q4_0Block` finds the max absolute value in each block of 32, sets `scale = maxAbs / 7.0`, then quantizes each weight to a signed 4-bit integer (`-8` to `7`) packed two-per-byte. On the GPU, the WGSL shader receives the packed uint32 array plus the float32 scales array, and dequantizes on the fly inside the shader without a CPU roundtrip. --- ## CastWeights `CastWeights[T Numeric](weights any) []T` is the universal extraction helper. It type-switches on all 10 concrete slice types and uses `ConvertSlice[In, Out]` to re-cast the values into the requested type `T`. When `DispatchLayer` cannot find a dedicated fast-path for the layer's DType, it falls through to `CastWeights` on the pre-quantized `Versions` data. --- ## Bit-Packed Serialization Ratios From the README, verified across 378 model permutations: | DType | Bytes/weight (serialized) | vs FP32 | |:------|:--------------------------|:--------| | Float32 | 4 | 1.0x | | Float16 | 2 | 0.5x | | Int8 | 1 | 0.25x | | Int4/FP4 | 0.5 | 0.125x | | Int2/Ternary | 0.25 | 0.0625x | | Binary | 0.125 | 0.0313x ← **98.4% reduction** | The packing/unpacking logic lives in `encodeNativeWeights` and `decodeNativeWeights` in `persistence.go`. Binary packs 8 weights per byte using bit shifts; Ternary packs 4 per byte using 2-bit fields; FP4 packs 2 per byte using nibbles. --- ## Layer Reference Source: https://openfluke.com/docs/layers Markdown: https://openfluke.com/docs/layers.md # Layer Reference This document describes every `LayerType` in `poly/`. For each layer: what it computes, which fields of `VolumetricLayer` configure it, weight layout in the `WeightStore`, and an ASCII data-flow diagram. --- ## LayerType Constants ```go const ( LayerDense LayerType = 0 LayerMultiHeadAttention LayerType = 1 LayerSwiGLU LayerType = 2 LayerRMSNorm LayerType = 3 LayerCNN1 LayerType = 4 LayerCNN2 LayerType = 5 LayerCNN3 LayerType = 6 LayerRNN LayerType = 7 LayerLSTM LayerType = 8 LayerLayerNorm LayerType = 9 LayerConvTransposed1D LayerType = 10 LayerConvTransposed2D LayerType = 11 LayerConvTransposed3D LayerType = 12 LayerEmbedding LayerType = 13 LayerKMeans LayerType = 14 LayerSoftmax LayerType = 15 LayerParallel LayerType = 16 LayerSequential LayerType = 17 LayerResidual LayerType = 18 ) ``` > [!NOTE] > There is no explicit `LayerGRU` constant; GRU is implemented in `rnn.go` as a variant of the RNN pattern referenced through the same dispatcher slot. --- ## Dense (LayerDense = 0) **What it does:** Fully-connected linear transformation: `output = input × W^T + b`, followed by an activation function. Every input connects to every output. **Key fields:** | Field | Meaning | |:------|:--------| | `InputHeight` | Number of input features | | `OutputHeight` | Number of output features | | `Activation` | One of ReLU, SiLU, GELU, Tanh, Sigmoid, Linear | | `DType` | Active numerical type | | `UseTiling` | Enables tiled fast paths where implemented (CPU block tiling, sequential propagation to sub-layers, etc.) | | `TileSize` | Legacy scalar fallback when per-dtype maps are empty; prefer **`CPUTileSizes`** on CPU and **`GPUSCTileSizes` / `GPUMCTileSizes`** on GPU after `RefreshRuntimeTileSizes()` | | `EnableMultiCoreTiling` | **GPU:** aligned with `VolumetricNetwork.EnableMultiCoreTiling`; transformer forwards use the network flag to choose **`GetGPUMCTileSize`** vs **`GetGPUSCTileSize`**. **CPU:** often set `true` with training loaders for parity; **does not** switch between two CPU tile maps (only `CPUTileSizes` exists) | **Weight layout:** `WeightStore.Master` is a flat `[OutputHeight × InputHeight]` row-major matrix. No bias is stored in the Master by default (the polymorphic engine absorbs bias via zero-biased initialization). ``` Input [batch, inputSize] │ ▼ ┌─────────────────────────────────────────────┐ │ preAct[b, o] = Σᵢ input[b, i] × W[o, i] │ │ │ │ W shape: [OutputHeight, InputHeight] │ └─────────────────────────────────────────────┘ │ ▼ Activation(preAct) │ ▼ Output [batch, outputSize] ``` The tiled variant (`DenseForwardTiled`) loads input tiles into a local buffer and unrolls the dot product 4× to help the compiler auto-vectorize. The INT8 and Binary tiled paths each have their own hot loops in `denseForwardTiledInt8` and `denseForwardTiledBinary`. --- ## CNN1 / CNN2 / CNN3 (LayerCNN1–3 = 4–6) **What they do:** Convolutional layers across 1D sequences, 2D images, and 3D volumes respectively. A learnable kernel is slid across the spatial dimensions and a dot product is computed at each position. **Key fields:** | Field | Meaning | |:------|:--------| | `InputChannels` | Channels in the input | | `Filters` | Number of output channels (kernels) | | `KernelSize` | Spatial size (k for CNN1, k×k for CNN2, k×k×k for CNN3) | | `Stride` | Step between kernel positions | | `Padding` | Zero-padding added on each side | | `InputHeight` / `InputWidth` / `InputDepth` | Input spatial dimensions | | `OutputHeight` / `OutputWidth` / `OutputDepth` | Output spatial dimensions | **Weight layout:** `Filters × InputChannels × KernelSize^N` ``` CNN2 Data Flow: Input [batch, inChannels, H, W] │ ▼ slide kernel [f, c, kH, kW] over H, W ┌─────────────────────────────────────────────────────────────┐ │ for each filter f: │ │ for each (oh, ow): │ │ out[b,f,oh,ow] = Σ_c Σ_kh Σ_kw in[b,c,oh+kh,ow+kw] │ │ × W[f,c,kh,kw] │ └─────────────────────────────────────────────────────────────┘ │ ▼ Activation Output [batch, Filters, outH, outW] ``` Output size formula (same for each spatial dimension): ``` outDim = (inDim + 2*Padding - KernelSize) / Stride + 1 ``` > [!TIP] > CNN3 on GPU achieves over 7600x speedup versus CPU tiling because the 3D spatial loop maps perfectly to 3D WebGPU workgroups. Always prefer GPU for CNN3. --- ## ConvTransposed1D / 2D / 3D (LayerConvTransposed1D–3D = 10–12) **What they do:** Transposed convolution (also called "deconvolution"). It inverts the spatial compression of a regular convolution — used in decoder networks and generative models to upsample feature maps. **Key fields:** Same as CNN variants plus `OutputPadding` for controlling output dimensions. **Weight layout:** `InputChannels × Filters × KernelSize^N` ``` ConvTransposed2D conceptual reverse: CNN2: [H, W] ──kernel──▶ [H', W'] (downsample) ConvT: [H', W'] ──kernel──▶ [H, W] (upsample) Internal mechanism: insert (Stride-1) zeros between input elements, then apply regular convolution with kernel flipped. ``` --- ## RNN (LayerRNN = 7) **What it does:** Vanilla recurrent network. Processes a sequence step-by-step, feeding the hidden state forward through time. ``` h_t = tanh(x_t × W_ih^T + h_{t-1} × W_hh^T + b_h) ``` **Key fields:** | Field | Meaning | |:------|:--------| | `InputHeight` | Input feature size | | `OutputHeight` | Hidden state size | | `SeqLength` | Number of time steps | **Weight layout in Master:** ``` [ W_ih | W_hh | b_h ] ihSize hhSize hSize ``` Where `ihSize = hiddenSize × inputSize`, `hhSize = hiddenSize × hiddenSize`, `hSize = hiddenSize`. ``` Step 0: Step 1: Step t: x₀ h₋₁=0 x₁ h₀ xₜ h_{t-1} │ │ │ │ │ │ └──┬───┘ └──┬──┘ └──┬───┘ ▼ ▼ ▼ [RNN cell] [RNN cell] [RNN cell] │ │ │ ▼ ▼ ▼ h₀ h₁ hₜ ``` --- ## LSTM (LayerLSTM = 8) **What it does:** Long Short-Term Memory. Adds a cell state `c_t` and three gating mechanisms (forget, input, output) to control information flow through time. Solves the vanishing gradient problem for long sequences. **Gate equations:** ``` i_t = σ(x_t × W_i^T + h_{t-1} × U_i^T + b_i) ← input gate f_t = σ(x_t × W_f^T + h_{t-1} × U_f^T + b_f) ← forget gate g_t = tanh(x_t × W_g^T + h_{t-1} × U_g^T + b_g) ← cell gate o_t = σ(x_t × W_o^T + h_{t-1} × U_o^T + b_o) ← output gate c_t = f_t ⊙ c_{t-1} + i_t ⊙ g_t h_t = o_t ⊙ tanh(c_t) ``` **Weight layout:** Four gate blocks concatenated: ``` [ W_i | U_i | b_i | W_f | U_f | b_f | W_g | U_g | b_g | W_o | U_o | b_o ] ←── gate i ──────────▶ ←── gate f ──────────▶ ... gateWeightCount = ihSize + hhSize + hiddenSize Total = 4 × gateWeightCount ``` ``` ┌─────────────────────────────────────┐ c_{t-1} ──────▶│ │──▶ c_t │ Forget × + Input × Cell │ h_{t-1} ──────▶│ │──▶ h_t │ Output gate × tanh(c_t) │ x_t ──────▶│ │ └─────────────────────────────────────┘ ``` --- ## GRU GRU (Gated Recurrent Unit) is implemented in `rnn.go` alongside the vanilla RNN. It uses two gates (reset and update) and eliminates the separate cell state. ``` z_t = σ(x_t × W_z + h_{t-1} × U_z + b_z) ← update gate r_t = σ(x_t × W_r + h_{t-1} × U_r + b_r) ← reset gate n_t = tanh(x_t × W_n + (r_t ⊙ h_{t-1}) × U_n + b_n) h_t = (1 - z_t) ⊙ h_{t-1} + z_t ⊙ n_t ``` --- ## MultiHeadAttention (LayerMultiHeadAttention = 1) **What it does:** Standard multi-head scaled dot-product attention with optional RoPE positional encoding, Grouped Query Attention (GQA), and a KV cache for autoregressive decoding. **Key fields:** | Field | Meaning | |:------|:--------| | `DModel` | Model dimension (total embedding size) | | `NumHeads` | Number of query heads | | `NumKVHeads` | Number of key/value heads (< NumHeads for GQA/MQA) | | `HeadDim` | Dimension per head (usually DModel / NumHeads) | | `SeqLength` | Current sequence length | | `RoPEFreqBase` | RoPE frequency base (default 10000.0) | | `MaxSeqLen` | KV cache capacity | | `KVCacheK` / `KVCacheV` | CPU-side KV cache tensors | | `KVOffset` | Current filled position in the KV cache | **Weight layout:** ``` Master = [ Q_W | K_W | V_W | O_W | Q_b | K_b | V_b | O_b ] Q_W: [DModel × DModel] K_W: [DModel × kvDim] (kvDim = NumKVHeads × HeadDim) V_W: [DModel × kvDim] O_W: [DModel × DModel] biases follow ``` **Attention computation:** ``` Q = input × Q_W^T + Q_b [seqLen, DModel] K = input × K_W^T + K_b [seqLen, kvDim] V = input × V_W^T + V_b [seqLen, kvDim] Apply RoPE to Q, K (rotate pairs by position-dependent angle) For each head h: q_h = Q[:, h*headDim:(h+1)*headDim] [seqLen, headDim] k_h = K[:, kv_head_idx*headDim:...] [seqLen, headDim] v_h = V[:, kv_head_idx*headDim:...] scores = q_h × k_h^T / sqrt(headDim) [seqLen, seqLen] weights = softmax(scores, causal_mask) out_h = weights × v_h [seqLen, headDim] output = concat(out_0..out_{numHeads-1}) × O_W^T ``` --- ## SwiGLU (LayerSwiGLU = 2) **What it does:** Gated feedforward block used in modern LLMs. Two parallel linear projections, one acting as a gate through SiLU activation, combined element-wise before a down projection. ``` gate = SiLU(x × W_gate^T + b_gate) up = x × W_up^T + b_up hidden = gate ⊙ up output = hidden × W_down^T + b_down ``` **Key fields:** `InputHeight` (in), `OutputHeight` (intermediate/hidden size). The actual output to the next layer is back to `InputHeight` via the down projection. **Weight layout:** ``` Master = [ W_gate | W_up | W_down | b_gate | b_up | b_down ] in×int in×int int×in int int in ``` Where `int = OutputHeight` (intermediate size). ``` Input [seqLen, in] │ ├──────────────────────────────────┐ │ │ ▼ ▼ W_gate (in → int) W_up (in → int) │ │ SiLU │ │ │ └──────────── ⊙ (element multiply) ┘ │ ▼ W_down (int → in) │ ▼ Output [seqLen, in] ``` --- ## RMSNorm (LayerRMSNorm = 3) **What it does:** Root Mean Square normalization. Divides each element by the RMS of the vector, then scales by a learned gamma parameter. ``` rms = sqrt( mean(x²) + ε ) output = (x / rms) × γ ``` **Key fields:** `InputHeight` (size), `DType`. **Always kept in FP32 on GPU** — the `SyncToGPU` code explicitly refuses to quantize RMSNorm weights. **Weight layout:** `Master` is a flat `[InputHeight]` gamma vector (no beta/bias term). --- ## LayerNorm (LayerLayerNorm = 9) **What it does:** Layer normalization. Computes mean and variance across the feature dimension, normalizes, then applies learnable gamma and beta. ``` μ = mean(x), σ² = var(x) x_hat = (x - μ) / sqrt(σ² + ε) output = γ ⊙ x_hat + β ``` **Weight layout:** `Master` is `[2 × InputHeight]`: first half is gamma, second half is beta. --- ## Embedding (LayerEmbedding = 13) **What it does:** Token lookup table. Given a vector of integer token IDs, returns the corresponding rows from the embedding matrix. **Key fields:** `VocabSize`, `EmbeddingDim`. **Weight layout:** `[VocabSize × EmbeddingDim]` row-major matrix. ``` Token IDs: [42, 7, 115] │ ▼ lookup rows 42, 7, 115 ┌──────────────────────────────────────────────────┐ │ Embedding Table [VocabSize × EmbeddingDim] │ │ │ │ Row 7: [0.12, -0.33, 0.87, ...] │ │ Row 42: [0.55, 0.11, -0.22, ...] │ │ Row 115: [-0.01, 0.77, 0.44, ...] │ └──────────────────────────────────────────────────┘ │ ▼ Output [3, EmbeddingDim] (gradient only applied to used rows) ``` --- ## KMeans (LayerKMeans = 14) **What it does:** Differentiable clustering. Computes soft assignment probabilities (or raw feature distances) between the input and a set of learnable cluster centroids. **Key fields:** | Field | Meaning | |:------|:--------| | `NumClusters` | K — number of cluster centers | | `InputHeight` | Feature vector size | | `KMeansTemperature` | Controls sharpness of soft assignment | | `KMeansOutputMode` | `"probabilities"` or `"features"` | **Weight layout:** `[NumClusters × InputHeight]` centroid matrix. ``` Input [batch, featureDim] │ ▼ compute squared distance to each centroid dist[b, k] = ||input[b] - centroid[k]||² │ ▼ temperature-scaled negative softmax p[b, k] = softmax(-dist / temperature) │ ▼ Output [batch, NumClusters] (if mode="probabilities") or [batch, featureDim] (if mode="features") ``` --- ## Softmax (LayerSoftmax = 15) **What it does:** Normalizes a vector (or matrix rows) into a probability distribution. Has 10 variants controlled by `SoftmaxType`. See [softmax.md](./softmax.md) for the full variant reference. **Key fields:** `SoftmaxType`, `Temperature`, `SoftmaxRows`, `SoftmaxCols`, `HierarchyLevels`, `EntmaxAlpha`, `Mask`, `GumbelNoise`. No weights — `WeightStore` is nil for Softmax layers. --- ## Parallel (LayerParallel = 16) **What it does:** Fans the input to N sub-layers simultaneously and combines their outputs. Supports five combination modes. **Key fields:** | Field | Meaning | |:------|:--------| | `ParallelBranches` | `[]VolumetricLayer` — the sub-layer definitions | | `CombineMode` | `"add"`, `"avg"`, `"concat"`, `"filter"`, `"grid_scatter"` | | `FilterGateConfig` | Optional gate network for MoE routing (filter mode) | ``` Input │ ┌──────────┼──────────┐ ▼ ▼ ▼ Branch 0 Branch 1 Branch 2 │ │ │ └──────────┼──────────┘ │ CombineMode: ┌─────────────────────────────────────────┐ │ "add" element-wise sum │ │ "avg" element-wise average │ │ "concat" [b0, b1, b2] concatenated │ │ "filter" gate × b0 + gate × b1 ... │ │ "grid_scatter" same as concat │ └─────────────────────────────────────────┘ │ Output ``` The `preAct` tensor returned by `ParallelForwardPolymorphic` stores the branch preActs in `preAct.Nested`, enabling correct recursive backpropagation. See [parallel_sequential.md](./parallel_sequential.md). --- ## Sequential (LayerSequential = 17) **What it does:** Chains N sub-layers in series. Each sub-layer receives the output of the previous one. The sub-layers can be of any type — this enables composing mini-architectures inside a single grid cell. **Key fields:** `SequentialLayers []VolumetricLayer` ``` Input │ ▼ Sub-layer 0 ──▶ Sub-layer 1 ──▶ Sub-layer 2 │ Output ``` Each step container stores `[bPre, bInput, bSkip]` in the nested tensor for accurate backward computation through skip connections within the sequence. --- ## Residual (LayerResidual = 18) **What it does:** Skip connection — adds the input directly to the output of its sub-network. ``` Input │ ┌────┴────┐ │ │ skip ▼ │ Sub-layers │ │ │ ▼ │ ┌───┐ │ │ + │◀──────┘ └─┬─┘ │ Output = SubLayers(Input) + Input ``` The skip tensor is passed as the second argument to `DispatchLayer` and is added inside `ResidualForwardPolymorphic`. Gradients flow back both through the sub-layers and directly through the skip branch. --- ## Activation Functions All layers that produce a `preAct` / `postAct` pair apply an activation via `Activate[T](v T, act ActivationType)`: | Constant | Formula | |:---------|:--------| | `ActivationReLU` (0) | `max(0, x)` | | `ActivationSilu` (1) | `x × σ(x)` | | `ActivationGELU` (2) | `0.5x(1 + tanh(√(2/π)(x + 0.044715x³)))` | | `ActivationTanh` (3) | `tanh(x)` | | `ActivationSigmoid` (4) | `1/(1+e^−x)` | | `ActivationLinear` (-1) | `x` (identity — no nonlinearity) | `ActivateDerivative[T]` returns the analytic derivative for backpropagation. --- ## Layer Summary Table | Layer | Parameters | GPU Forward | GPU Backward | |:------|:-----------|:-----------|:------------| | Dense | in×out | EXACT | EXACT | | CNN1 | f×c×k | EXACT | EXACT | | CNN2 | f×c×k² | EXACT | EXACT | | CNN3 | f×c×k³ | EXACT | EXACT | | RNN | ih+hh+b | EXACT | — | | LSTM | 4×(ih+hh+b) | EXACT | — | | MHA | 4×d² + biases | BROKEN (dets) | pending | | SwiGLU | 3×in×int | BROKEN (dets) | not wired | | RMSNorm | hidden | EXACT | EXACT | | LayerNorm | 2×hidden | — | — | | Embedding | vocab×dim | EXACT (DW) | — | | KMeans | k×dim | — | — | | Softmax | none | — | — | | Parallel | per-branch | — | — | | Sequential | per-layer | — | — | | Residual | per-sub | — | — | --- ## The Dispatcher Pattern and 3D Coordinate System Source: https://openfluke.com/docs/dispatch Markdown: https://openfluke.com/docs/dispatch.md # The Dispatcher Pattern and 3D Coordinate System This document explains how `DispatchLayer` and `DispatchLayerBackward` work as runtime jump tables, how the 3D coordinate system maps to `VolumetricLayer` positions, and how `IsRemoteLink` enables spatial hopping across the grid. --- ## Why a Dispatcher? A naive implementation of a polymorphic neural network would embed a large `switch` inside the forward loop: ```go // Naive — thread-divergence on GPU, hard to fuse for _, layer := range layers { switch layer.Type { case LayerDense: output = denseForward(layer, input) case LayerCNN2: output = cnn2Forward(layer, input) // ... } } ``` M-POLY-VTD separates concerns: the **traversal loop** iterates coordinates, and the **dispatcher** makes the type-specific call. This decoupling is what makes GPU kernel fusion possible in the future — the driver can inspect a group of same-type layers and launch a single batched shader rather than 19 separate ones. --- ## DispatchLayer ```go func DispatchLayer[T Numeric]( layer *VolumetricLayer, input, skip *Tensor[T], ) (preAct, postAct *Tensor[T]) ``` This is a generic function. The type parameter `T` is inferred from `input`. Every call returns two tensors: - `preAct` — the layer's internal state before the final activation. For Parallel/Sequential layers this carries the nested activation tree in `preAct.Nested`. - `postAct` — the result of applying the activation function to `preAct`. This is what flows to the next layer. The full routing table: ``` layer.Type ──switch──▶ function called ─────────────────────────────────────────────────────────────── LayerResidual ResidualForwardPolymorphic(layer, input, skip) LayerDense DenseForwardPolymorphic(layer, input) LayerCNN1 CNN1ForwardPolymorphic(layer, input) LayerCNN2 CNN2ForwardPolymorphic(layer, input) LayerCNN3 CNN3ForwardPolymorphic(layer, input) LayerRNN RNNForwardPolymorphic(layer, input) LayerLSTM LSTMForwardPolymorphic(layer, input) LayerMultiHeadAttention MHAForwardPolymorphic(layer, input) LayerSwiGLU SwiGLUForwardPolymorphic(layer, input) LayerRMSNorm RMSNormForwardPolymorphic(layer, input) LayerLayerNorm LayerNormForwardPolymorphic(layer, input) LayerConvTransposed1D ConvTransposed1DForwardPolymorphic(layer, input) LayerConvTransposed2D ConvTransposed2DForwardPolymorphic(layer, input) LayerConvTransposed3D ConvTransposed3DForwardPolymorphic(layer, input) LayerEmbedding EmbeddingForwardPolymorphic(layer, input) LayerKMeans KMeansForwardPolymorphic(layer, input) LayerSoftmax SoftmaxForwardPolymorphic(layer, input) LayerParallel ParallelForwardPolymorphic(layer, input) LayerSequential SequentialForwardPolymorphic(layer, input) default DenseForwardPolymorphic(layer, input) ─────────────────────────────────────────────────────────────── ``` --- ## DispatchLayerBackward ```go func DispatchLayerBackward[T Numeric]( layer *VolumetricLayer, gradOutput, input, skip, preAct *Tensor[T], ) (gradInput, gradWeights *Tensor[T]) ``` The mirror of `DispatchLayer`. Returns: - `gradInput` — the gradient to pass to the layer that produced `input` (propagates error upstream) - `gradWeights` — the gradient for this layer's own weights (used to update `WeightStore.Master`) The routing table is symmetric to the forward pass. The `skip` argument is used only by `ResidualBackwardPolymorphic`. --- ## The 3D Grid Traversal `ForwardPolymorphic[T]` iterates the grid in reading order: ```go for z := 0; z < n.Depth; z++ { for y := 0; y < n.Rows; y++ { for x := 0; x < n.Cols; x++ { for l := 0; l < n.LayersPerCell; l++ { idx := n.GetIndex(z, y, x, l) layer := &n.Layers[idx] // ... _, post := DispatchLayer(layer, currentTensor, nil) currentTensor = post } } } } ``` The flattened index formula: ``` idx = z * (Rows * Cols * LayersPerCell) + y * (Cols * LayersPerCell) + x * (LayersPerCell) + l ``` Visually, for a (Depth=1, Rows=2, Cols=3, LayersPerCell=1) network: ``` z=0: ┌─────────────┬─────────────┬─────────────┐ │ (0, 0, 0,0) │ (0, 0, 1,0) │ (0, 0, 2,0) │ ← idx 0,1,2 │ idx=0 │ idx=1 │ idx=2 │ ├─────────────┼─────────────┼─────────────┤ │ (0, 1, 0,0) │ (0, 1, 1,0) │ (0, 1, 2,0) │ ← idx 3,4,5 │ idx=3 │ idx=4 │ idx=5 │ └─────────────┴─────────────┴─────────────┘ Data flows: idx=0 ──▶ idx=1 ──▶ idx=2 ──▶ idx=3 ──▶ idx=4 ──▶ idx=5 ``` `BackwardPolymorphic` walks in reverse (z, y, x, l all reversed), using cached `inputs[idx]` and `preActs[idx]` from the forward pass. --- ## Tiled Traversal When `n.UseTiling = true`, `ForwardPolymorphic` uses a blocked spatial traversal with tile size 4: ``` for zTile := 0; zTile < Depth; zTile += 4 { for yTile := 0; yTile < Rows; yTile += 4 { for xTile := 0; xTile < Cols; xTile += 4 { // Process 4×4×4 tile of cells } } } ``` This is the CPU-side analogue of the GPU workgroup tile strategy. The intent is to improve data locality: all layers in a 4×4×4 spatial neighborhood execute together, keeping their weight data warm in L2/L3 cache. ### SC (single-workgroup) vs MC (multi-workgroup) tiling There are **two different “tiling” knobs** in `poly`: 1. **`VolumetricNetwork.UseTiling`** (see [Tiled Traversal](#tiled-traversal) above) — spatial blocking of the **3D grid** in `ForwardPolymorphic` (4×4×4 cells). Unrelated to transformer matmul tiles. 2. **Per-layer matmul / GPU workgroup tiling** — `RefreshRuntimeTileSizes()` fills per-dtype maps from layer geometry and (for GPU) `WGPUContext` limits. #### GPU: two tile maps, configurable SC vs MC On **GPU**, each layer gets **`GPUSCTileSizes`** and **`GPUMCTileSizes`** (see `refreshRuntimeGPUTileSizes` in `tile_detection.go`). At dispatch, **`VolumetricNetwork.EnableMultiCoreTiling`** chooses which map to use: `GetGPUMCTileSize(dtype)` when `true` (larger / higher-throughput tiles where limits allow), `GetGPUSCTileSize(dtype)` when `false` (smaller workgroups, friendlier to tight limits). So **MC vs SC on GPU is a real switch** — you are not stuck in one profile; set `EnableMultiCoreTiling` (or use **`TrainingModeGPUSC` / `TrainingModeGPUMC`** in `trainBatchWGPU`, which pick tile sizes the same way). Transformer-style forwards in `wgpu_forward.go` read **`network.EnableMultiCoreTiling`** (not per-layer) for that choice. `WGPUContext.GPUTileSize` is the device-tuned baseline that feeds how those SC/MC maps are built, not the only number used at dispatch. #### CPU: one tile map (not an SC/MC pair on the layer) On **CPU**, each layer has a **single** per-dtype map, **`CPUTileSizes`**, via `GetCPUTileSize` — there is **no** `CPUSCTileSizes` / `CPUMCTileSizes` pair. Tiled matmul-style loops (Dense, SwiGLU, CNN, etc.) all use that one size. `TrainingModeCPUSC` and `TrainingModeCPUMC` exist in the enum (and show up in benchmarks), but **`ConfigureNetworkForMode` applies the same wiring to all CPU modes** (`UseTiling`, `EnableMultiCoreTiling`, `RefreshRuntimeTileSizes`), and **`executeBatchCPU` does not receive the mode** — so there is **no** separate “CPU SC tile path” vs “CPU MC tile path” in the layer maps today. **`EnableMultiCoreTiling` on CPU** is set for consistency with GPU-bound nets and training tooling; it does **not** flip between two CPU tile sizes because only one map exists. `WGPUContext.GPUTileSize` is the auto-detected base hint (from limits); concrete SC/MC sizes per layer type on GPU live in the two GPU maps, not in that single int alone. --- ## VolumetricLayer: The Coordinate Record Every `VolumetricLayer` contains its own position: ```go type VolumetricLayer struct { Network *VolumetricNetwork // back-pointer Type LayerType Activation ActivationType DType DType WeightStore *WeightStore Z int // Depth coordinate Y int // Row coordinate X int // Col coordinate L int // Layer index within cell // Spatial Routing IsRemoteLink bool TargetZ, TargetY, TargetX, TargetL int // ... configuration fields } ``` The `(Z, Y, X, L)` fields are set during `NewVolumetricNetwork` and are the canonical address. `GetLayer(z, y, x, l)` returns a pointer into the flat `Layers` slice using `GetIndex`. --- ## IsRemoteLink: Spatial Hopping A layer with `IsRemoteLink = true` does not receive its input from the previous layer in reading order. Instead, it reads from the output of whatever layer lives at `(TargetZ, TargetY, TargetX, TargetL)`. This enables: 1. **Skip connections** — hop over several layers in the grid 2. **Feedback loops** — target a layer at an *earlier* coordinate (biological recurrence) 3. **Parallel expert routing** — multiple layers at different positions all reading the same source 4. **Cross-depth signals** — connect depth=0 outputs to depth=2 inputs ``` Standard flow: Remote link (skip): (0,0,0) → (0,0,1) (0,0,0) ────────────────────┐ │ (0,0,1) → (0,0,2) → ... │ (0,0,2) │ │ (0,2,0) ←── IsRemoteLink ──┘ (0,0,3) └── reads output of (0,0,0) Feedback loop: (0,0,0) │ (0,0,1) │ (0,0,2) ─── IsRemoteLink ──▶ TargetZ=0, TargetY=0, TargetX=0 (reads from cycle N-1's output of layer (0,0,0) — step mesh only) ``` In `ForwardPolymorphic`, a remote-linked layer simply receives `currentTensor` like any other layer; the remote link semantic is only fully honored by `StepForward`, which maintains per-layer output buffers across time steps. In `ParallelForwardPolymorphic` and `SequentialForwardPolymorphic`, remote links are resolved by calling `layer.Network.GetLayer(branch.TargetZ, ...)` and dispatching the resolved layer pointer. --- ## The GPU Dispatch Path When `n.UseGPU = true`, the training loop calls `ctx.DispatchForwardLayer(l, batchSize, curBuf, preBuf)` instead of `DispatchLayer`. This function is in `wgpu_forward.go` and routes to the appropriate WGSL compute shader based on `l.Type`. The same dispatcher philosophy applies: one function, one switch, explicit routing. The difference is that inputs and outputs are `*wgpu.Buffer` handles in VRAM rather than `*Tensor[T]` in RAM. ``` trainBatchWGPU: BeginFrame() ← create shared CommandEncoder │ ├── for each layer forward: │ └── ctx.DispatchForwardLayer(l, ...) ← records into encoder │ ├── DispatchMSEGradPartialLoss(...) ← records into encoder │ ├── for each layer backward (reverse): │ ├── ctx.DispatchActivationBackward(...) │ ├── ctx.DispatchBackwardLayer(l, ...) │ └── ctx.DispatchApplyGradients(...) │ FlushFrame() ← ONE submit for entire forward + backward + weight update │ ReadBuffer(partialsBuf) ← only reads back tiny loss scalars ``` This single-submission design reduces Go-to-GPU driver overhead from ~150+ round trips per batch to exactly 1. --- ## Disabled Layers Setting `layer.IsDisabled = true` causes both `ForwardPolymorphic` and `StepForward` to skip the layer entirely. In `StepForward`, a disabled layer passes its input buffer through to `NextBuffer` unchanged. This is the mechanism for implementing sparse MoE expert activation — gate layers can conditionally disable branches. --- ## Training: Forward Pass, Backward Pass, Optimizers, and Learning Source: https://openfluke.com/docs/training Markdown: https://openfluke.com/docs/training.md # Training: Forward Pass, Backward Pass, Optimizers, and Learning This document covers the full training pipeline: the forward and backward pass mechanics, loss computation, weight update strategies, gradient clipping, Tween, and the `VGStepBP` adaptive rate. --- ## The Training Loop ```go result, err := poly.Train[float32](network, batches, config) ``` `Train[T Numeric]` is the high-level entry point. It wraps `trainBatchCPU` or `trainBatchWGPU` depending on `config.UseGPU`. ```go type TrainingConfig struct { Epochs int LearningRate float32 LossType string // "mse" or "cross_entropy" GradientClip float32 // 0 = no clipping Verbose bool UseGPU bool DeviceID int TrackPerf bool } ``` A `TrainingBatch[T]` pairs `Input *Tensor[T]` with `Target *Tensor[T]`. Multiple batches are provided as a slice — the loop iterates over batches for each epoch, averages the loss, and prints progress if `Verbose = true`. --- ## Runtime tiling (`ConfigureNetworkForMode`) Before the training loop runs, `Train` wires the network through `ConfigureNetworkForMode` (`training.go`), which aligns tiling flags with the selected `TrainingMode`: | Mode | Tiling | Multi-core | SIMD | |------|--------|------------|------| | `TrainingModeCPUNormal` | off | off | off | | `TrainingModeCPUSC` | on | off | off | | `TrainingModeCPUMC` | on | on | off | | `TrainingModeCPUSimd` | on | on | **on** (`SetSimdForwardRecursive`) | - **CPU modes** (`TrainingModeCPUNormal` … `TrainingModeCPUSimd`): `ConfigureNetworkForMode` sets `UseTiling` and `EnableMultiCoreTiling` per row above, calls `RefreshRuntimeTileSizes()`, and syncs every layer. **`TrainingModeCPUSimd`** additionally enables Plan 9 SIMD on compute layers (Dense, SwiGLU, MHA, CNN1–3, RNN, LSTM, Embedding, Residual): forward `DotTile` / int8 dots, backward `SaxpyF32AccF64` / int8 saxpy. Other CPU modes call `SetSimdForwardRecursive(false)`. - **GPU modes** (`TrainingModeGPUNormal`, `TrainingModeGPUSC`, `TrainingModeGPUMC`): initializes WebGPU if needed, `RefreshRuntimeTileSizes()`, resets the bind-group cache, `SyncToGPU()`, and ensures FP32 master buffers exist for backward. **`trainBatchWGPU`** uses **`TrainingModeGPUSC`** vs **`TrainingModeGPUMC`** to select **`GetGPUSCTileSize`** vs **`GetGPUMCTileSize`** per layer; **`GPUNormal`** uses untiled or generic dispatch per layer type. For **interactive inference** (no explicit training mode), toggling **`VolumetricNetwork.EnableMultiCoreTiling`** chooses GPU SC vs MC tile maps (`wgpu_forward.go`), the same underlying maps training uses. ### CPU-SIMD training ```go config := poly.DefaultTrainingConfig() config.Mode = poly.TrainingModeCPUSimd result, err := poly.Train[float32](network, batches, config) ``` The seven-layer harness (Lucy `examples/seven_layer`) benchmarks SC, MC, and SIMD training in its summary tables. On a Dense Float32 1×1×1 micro-benchmark (50 epochs), typical wall times are: | Platform | SC | MC | SIMD | |----------|-----|-----|------| | amd64 | ~14 ms | ~16 ms | **~10 ms** | | arm64 | ~9 ms | ~12 ms | **~10 ms** | Per-layer forward/backward SIMD vs SC tables (amd64 and arm64) are in [simd.md — seven-layer benchmark results](simd.md#seven-layer-benchmark-results). SIMD is a CPU path only; it does not replace GPU training. --- ## Training paradigms: default (QAT-like) vs native exact Loom has **two CPU training semantics**. They share the same `Train()` loop, loss, and backward walk — but differ in **which forward/backward kernels run** and **where weights are updated**. | | **Default path** (`dense.go`, `rnn.go`, …) | **Native exact** (`*_native.go`, menu **[14]**) | |---|---------------------------------------------|--------------------------------------------------| | **Flag** | `UseExactDType = false` (default) | `UseExactDType = true` | | **Forward weights** | `GetActive(dtype)` → dequant to **FP32**, matmul in float | `GetNative` / per-dot MAC rules in **storage dtype** | | **Backward** | Gradients in **FP32** | Dtype-native gradient rules (int MAC for true integers) | | **Optimizer** | `ApplyGradients` → updates **FP32 Master**, clears `Versions`, re-`Morph` on next forward | `ApplyGradientsNative` (MAC dtypes) or **in-place int8 update in backward** (true integers) | | **What it models** | Train with a float surrogate; storage dtype is a cached view | Train as if the network literally runs in that dtype | The native menu prints the split explicitly: ``` layer.go = GetActive FP32 dequant · *_native.go = GetNative MAC rules ``` ### Default path ≈ QAT-like surrogate training When `UseExactDType` is off, a layer tagged `DTypeInt8` (or Int32, FP8, …) still does most of its math in **float**: 1. **Forward:** `WeightStore.GetActive(layer.DType)` returns `Versions[dtype]` if present; tiled/SIMD paths cast or dequant to FP32 for `DotTile` / matmul. 2. **Backward:** `gradWeights` and `gradInput` accumulate in FP32. 3. **Update:** `ApplyGradients` subtracts `lr × grad` from **Master** (`[]float32`), then clears all cached `Versions`. 4. **Next forward:** `Morph(dtype)` re-quantizes Master into storage dtype. This is **not** textbook QAT (no fake-quant nodes, no straight-through estimator in the graph), but it is **QAT-adjacent**: *quantize → dequant → train in float → re-quant*. The low-bit dtype is a **storage and inference view**; learning happens in FP32 master space. `MorphToFloat32ForGPU` at upload time is pure **PTQ** (precision loss at inference only). Default CPU training is the same family: float math with periodic re-quantization. ### Native exact — two sub-flavors Enable with: ```go net.UseExactDType = true ``` Layers route to `DenseForwardNativeExact`, `RNNForwardNativeExact`, `LSTMForwardNativeExact`, etc. (`layer_native.go` lists supported types: Dense, SwiGLU, MHA, CNN1–3, RNN, LSTM, Embedding, Residual). #### 1. Native MAC dtypes (Int32, Int64, FP8, Float16, …) - Forward/backward use **per-dtype MAC rules** via `GetNative` — no bulk FP32 dequant buffer on the hot path. - Gradients follow those MAC rules (e.g. `nativeGradW` / `nativeGradX` in `native_weight_ctx.go`). - Optimizer: `ApplyGradientsNative` updates Master **and** the native storage slice (e.g. round to `int64` after each step). - **SIMD:** `*_native_simd.go` materializes f32 tiles once per pass (`materializeF32Weights`) then runs `DotTile` / `SaxpyF32AccF64` — same numerics, faster kernels. #### 2. True native integers (Int8, Int4, Ternary, Uint8, …) This is the largest departure from QAT: | Stage | Behaviour | |-------|-----------| | Forward | Real **int8 × int8 → int32** MAC, `>> 8`, clamp; activations cached as int8 (`ExactDense` cache) | | Backward | Integer gate grads; `SaxpyI8ScaleI32Acc` / `SaxpyI8ShiftedInputGradAcc` for weight/input accumulation | | Update | **`applyStochasticInt8Update` inside backward** — weights change in-place in int8 storage; external optimizer step is **skipped** (`ExactDense.WeightsUpdated`) | The network is **literally int8 during training**, not a float copy with int8 snapshots. ### SIMD on top (not a third paradigm) | SIMD mode | Math semantics | |-----------|----------------| | Default `*_simd.go` | Still `GetActive` FP32 — faster QAT-like path | | Native `*_native_simd.go` | Same native-exact rules; `DotI8Tile` / int8 saxpy for true integers | `SetSimdForward(true)` / `TrainingModeCPUSimd` choose the fast kernels; `UseExactDType` chooses **which training paradigm**. ### Lucy menus | Menu | Path | Purpose | |------|------|---------| | **[7]** seven-layer suite | Default (`UseExactDType` off for most runs) | SC/MC/SIMD parity, save/reload, training across 21 dtypes | | **[14]** native layer suite | `UseExactDType = true` per dtype | Per-layer native fwd/bwd/train × 21 dtypes; SIMD speedup columns when linked | Benchmark results (amd64/arm64): [native_layers.md](native_layers.md). See [quantization.md — Three training/inference modes](quantization.md#three-traininginference-modes) for how this relates to PTQ and [simd.md](simd.md) for native-exact SIMD file layout. --- ## CPU Training: Step by Step ```go func trainBatchCPU[T Numeric](n *VolumetricNetwork, batch TrainingBatch[T], config *TrainingConfig) float64 ``` ### 1. Forward Pass with History Capture ``` histIn [numLayers]*Tensor[T] ← input to each layer histPre [numLayers]*Tensor[T] ← preAct from each layer curr = batch.Input for each layer idx: histIn[idx] = curr pre, post = DispatchLayer(layer, curr, nil) histPre[idx] = pre curr = post ``` The history arrays are what make backpropagation possible without a tape. Every layer caches what it received and what it produced before activation. ### 2. Loss and Gradient Computation ``` gradOut = ComputeLossGradient(curr, batch.Target, "mse") lossVal = CalculateLoss(curr, batch.Target, "mse") ``` **MSE loss:** ``` L = (1/N) Σᵢ (output[i] - target[i])² gradOut[i] = (2/N) × (output[i] - target[i]) ``` ### 3. Backward Pass ```go _, layerGradients, _ := BackwardPolymorphic(n, gradOut, histIn, histPre) ``` `BackwardPolymorphic` walks the grid in **reverse** order (Z high to low, Y high to low, X high to low, L high to low). At each step: ``` gIn, gW = DispatchLayerBackward(layer, currentGrad, histIn[idx], nil, histPre[idx]) currentGrad = gIn ← flows back to previous layer layerGradients[idx] = {gIn, gW} ← stored for weight update ``` The backward pass for Dense computes: ``` gradPre[b,o] = gradOutput[b,o] × activation'(preAct[b,o]) gradWeights[o,i] += input[b,i] × gradPre[b,o] (accumulated over batch) gradInput[b,i] += W[o,i] × gradPre[b,o] ``` ### 4. Weight Update ```go for idx := range n.Layers { if l.ExactDense != nil && l.ExactDense.WeightsUpdated { l.ExactDense.WeightsUpdated = false continue // true-int8 layers already updated in backward } if layerGradients[idx][1] != nil { gW := ConvertTensor[T, float32](layerGradients[idx][1]) ApplyRecursiveGradients(l, gW, config.LearningRate) } } ``` **Default path** (`UseExactDType = false`): `ApplyRecursiveGradients` calls `WeightStore.ApplyGradients(gW, lr)`: ``` Master[i] -= lr × gradWeights[i] ``` After this, all cached `Versions` and `GPUWeights` are cleared, forcing re-quantization on the next forward pass. **Native exact path** (`UseExactDType = true`): - **MAC dtypes:** `ApplyGradientsNative` updates Master and the native storage slice together. - **True integers (Int8, …):** backward already ran `applyStochasticInt8Update`; the loop above skips the layer when `ExactDense.WeightsUpdated` is set. See [Training paradigms](#training-paradigms-default-qat-like-vs-native-exact) for the full comparison. `ApplyRecursiveGradients` also recurses into `ParallelBranches` and `SequentialLayers`, using the `Nested` structure of the returned `gradWeights` tensor to route updates to the correct sub-layer. --- ## GPU Training: BeginFrame / FlushFrame The GPU training path batches the entire forward + backward + weight-update into **one command buffer**: ``` ctx.BeginFrame() ← create shared CommandEncoder │ ├── forward pass: DispatchForwardLayer per layer ├── loss grad: DispatchMSEGradPartialLoss ├── backward: DispatchActivationBackward + DispatchBackwardLayer per layer └── update: DispatchApplyGradients per layer ctx.FlushFrame() ← ONE submit + destroy temp uniform bufs │ ReadBuffer(partialsBuf) ← only reads back numWG × float32 scalars ``` The loss value is computed from partial sums: `numWG = (totalOutput + 255) / 256` workgroups each sum 256 elements. The Go side only reads back `numWG` floats rather than the full output tensor. GPU weight updates are applied directly in VRAM via `DispatchApplyGradients`, which runs a WGSL shader: ```wgsl weights[i] -= lr * gradients[i] ``` This means the CPU master weights become stale after GPU training. A `ReadBuffer` + `Unpack` cycle is required if you want to access updated weights on the CPU. --- ## Loss Functions | `LossType` | Formula | Gradient | |:-----------|:--------|:---------| | `"mse"` | `(1/N) Σ (out-target)²` | `(2/N)(out-target)` | | `"cross_entropy"` | (not yet in `training.go`) | — | The GPU MSE gradient shader (`DispatchMSEGradPartialLoss`) computes both the gradient tensor and partial sums in a single pass. --- ## Tween (neural target propagation) **Tween** is the name used in this codebase for layer-local target propagation. In papers it often appears as *target propagation*, *difference target propagation*, or similar. Implementation: `tween.go`. Tween is a gradient-free alternative that estimates what each layer *should* have produced rather than computing exact chain-rule gradients. ### Two Modes **Chain Rule mode** (`UseChainRule = true`): ``` target = actual + gradient × GradientScale ``` This uses backpropagation to compute gradients, then shifts the target in the gradient direction. It is standard backprop dressed in Tween clothing. **Pure Tween mode** (`UseChainRule = false`): ``` target[i] = Σⱼ w[i,j] × currentTarget[j] / totalWeight[j] ``` Estimates input targets using weighted importance from the layer's own weights, without computing derivatives. This is the biologically-motivated "local learning" variant. Supported for Dense, RNN, LSTM, MHA, and SwiGLU. ### The TweenState ```go type TweenState[T Numeric] struct { ForwardActs []*Tensor[T] // what layers produced BackwardTargets []*Tensor[T] // what they should have produced Gradients []*Tensor[float32] LinkBudgets []float32 // cosine similarity: actual vs target Gaps []float32 // RMS distance: actual vs target Config *TweenConfig } ``` ### Usage Pattern ```go state := poly.NewTweenState[float32](network, poly.DefaultTweenConfig()) output := poly.TweenForward(network, state, input) poly.TweenBackward(network, state, target) state.CalculateLinkBudgets() poly.ApplyTweenGaps(network, state, lr) ``` ### Link Budget Gating Before applying any weight update, the engine checks the layer's `LinkBudget` (cosine similarity between actual output and backward target, normalized to [0,1]): ``` if budget < 0.2 { skip update // prevent corrupting "dead" layers } layerRate = lr × (0.5 + budget × 0.5) // good signal = higher rate ``` This prevents gradient corruption in layers where the signal has been destroyed. --- ## VGStepBP Adaptive Rate The README mentions `VGStepBP` (Variable Gradient Step Backpropagation) as an adaptive rate calculation. This integrates with the Tween `DepthScaleFactor` field: ```go DepthScaleFactor: 1.1 // each deeper layer gets 1.1× the base rate ``` Deeper layers receive slightly higher learning rates to compensate for gradient attenuation through the network depth. This is a simple heuristic that avoids the full computation of per-layer adaptive optimizers. --- ## Gradient Explosion Detection The `GradientClip` field in `TrainingConfig` (when non-zero) clips gradient norms. Additionally, the Tween gap system implicitly detects explosion: if `Gaps[i]` grows very large, the gap-based update `delta = lr × input × gap` will also be large, but the Link Budget gating prevents this from firing if the cosine similarity is low. The README references "Gradient Explosion Detection & Damping" as a completed feature in the training automation section. --- ## Activation Functions (Forward and Backward) All activation derivatives are computed analytically in `ActivateDerivative[T]`: ``` ReLU: dA/dx = 1 if x > 0, else 0 SiLU: dA/dx = σ(x)(1 + x(1-σ(x))) GELU: dA/dx ≈ CDF(x) + x × PDF(x) Tanh: dA/dx = 1 - tanh(x)² Sigmoid: dA/dx = σ(x)(1 - σ(x)) Linear: dA/dx = 1 ``` In the backward pass, `gradOutput` is multiplied elementwise by the derivative of `preAct` before accumulating `gradWeights` and `gradInput`. --- ## The Full Training Data Flow ``` ┌─────────────────────────────────────────────────────────────────┐ │ EPOCH LOOP │ │ │ │ ┌─────────────────────────────────────────────────────────┐ │ │ │ BATCH │ │ │ │ │ │ │ │ batch.Input │ │ │ │ │ │ │ │ │ ▼ │ │ │ │ [Forward Pass] ──▶ histIn, histPre captured │ │ │ │ │ │ │ │ │ ▼ │ │ │ │ prediction │ │ │ │ │ │ │ │ │ ▼ │ │ │ │ [Loss + gradOut] ◀── batch.Target │ │ │ │ │ │ │ │ │ ▼ │ │ │ │ [Backward Pass] ──▶ layerGradients │ │ │ │ │ │ │ │ │ ▼ │ │ │ │ [ApplyRecursiveGradients] ──▶ Master updated │ │ │ │ Versions cleared │ │ │ └─────────────────────────────────────────────────────────┘ │ │ │ │ LossHistory appended, EpochTimes recorded │ └─────────────────────────────────────────────────────────────────┘ ``` --- ## TrainingResult ```go type TrainingResult struct { FinalLoss float64 TotalTime time.Duration LossHistory []float64 // one entry per epoch EpochTimes []time.Duration } ``` `Train` returns this struct regardless of CPU or GPU path, making it easy to log or compare runs. --- ## GPU Backend: WebGPU (WGPU) Source: https://openfluke.com/docs/gpu Markdown: https://openfluke.com/docs/gpu.md # GPU Backend: WebGPU (WGPU) This document covers the WebGPU backend: initialization, the `BeginFrame`/`FlushFrame` command batching pattern, the buffer pool and pipeline cache, which layers have GPU support, and the tiling strategy. --- ## Why WebGPU M-POLY-VTD uses the `github.com/openfluke/webgpu/wgpu` Go bindings for hardware acceleration. WebGPU compiles to: - **Vulkan** on Windows/Linux - **Metal** on macOS/iOS - **DX12** on Windows - **WebGPU** in browser via WASM No CUDA, no CGO beyond the wgpu bindings. All shaders are WGSL (WebGPU Shading Language) strings generated at runtime by Go functions in `wgpu_shaders.go`, `wgpu_kernels.go`, and `wgpu_backward_shaders.go`. --- ## WGPUContext ```go type WGPUContext struct { Instance *wgpu.Instance Adapter *wgpu.Adapter Device *wgpu.Device Queue *wgpu.Queue PipelineCache map[string]*wgpu.ComputePipeline // keyed by shader source hash ActivationPool map[string]*wgpu.Buffer // named activation buffers LayoutCache map[string]*wgpu.BindGroupLayout BindGroupCache map[uint64]*wgpu.BindGroup // keyed by buffer-set hash UniformPool []*wgpu.Buffer // pre-allocated uniform buffer pool UniformIdx int ActiveEncoder *wgpu.CommandEncoder // non-nil during BeginFrame/FlushFrame PendingDestroys []*wgpu.Buffer // temp bufs destroyed after FlushFrame GPUTileSize int // auto-detected optimal tile size Limits wgpu.Limits } ``` ### Initialization ```go err := network.InitWGPU() ``` `InitWGPU` performs three WebGPU steps: 1. Create an `Instance` and request a `HighPerformance` `Adapter` 2. Query the default device for its limits, then boost `MaxStorageBufferBindingSize` to 1 GB and `MaxBufferSize` to 2 GB for large embedding tables 3. Request the final `Device` with boosted limits, then auto-detect the optimal `GPUTileSize` from the workgroup storage and invocation limits ``` CalculateOptimalGPUTileSizeFromLimits( MaxComputeWorkgroupStorageSize, MaxComputeInvocationsPerWorkgroup, headDim=64, ) → GPUTileSize (e.g., 8 or 16) ``` After init, call `network.SyncAllToGPU()` to upload all layer weights to VRAM. This also creates GPU KV cache buffers for MHA layers and pre-allocates named activation buffers (`hidden_A`, `hidden_B`, `norm_out`, etc.). --- ## BeginFrame / FlushFrame Pattern The most important design decision in the GPU backend. Instead of submitting a command buffer per layer (which would mean 100+ GPU driver calls per token), all operations are recorded into a single shared encoder: ``` ctx.BeginFrame() ← creates ctx.ActiveEncoder ← resets ctx.PendingDestroys // All Dispatch* calls record into ActiveEncoder: ctx.DispatchForwardLayer(...) ctx.DispatchActivation(...) ctx.DispatchMSEGradPartialLoss(...) ctx.DispatchBackwardLayer(...) ctx.DispatchApplyGradients(...) ctx.FlushFrame() ← enc.Finish() + Queue.Submit(cmd) ← destroys PendingDestroys buffers ← resets UniformIdx ``` Temporary uniform buffers (holding layer parameters like `batchSize`, `inputSize`, etc.) must stay alive until `FlushFrame` because the GPU reads them asynchronously. They are collected in `PendingDestroys` and destroyed only after the submit. `Queue.WriteBuffer` calls (to upload inputs, targets, and zero DW buffers) are **queue-level operations** — they are safe to call between `BeginFrame` and `FlushFrame` because the WebGPU spec guarantees they complete before the encoder submit executes. --- ## Buffer Management ### ActivationPool Named persistent buffers that survive across frames: ```go buf := ctx.GetActivationBuffer("hidden_A", size, wgpu.BufferUsageStorage) ``` If a buffer with this name already exists and is large enough, it is reused. Otherwise a new one is created and cached. This avoids per-step allocations during inference. ### CreatePersistentBuffer ```go buf, err := ctx.CreatePersistentBuffer(data []float32, label string) ``` Uploads a `[]float32` to a VRAM storage buffer with `Storage | CopySrc | CopyDst` usage. Used for weight buffers that stay resident across many forward passes. ### ReadBuffer ```go values, err := ctx.ReadBuffer(buf *wgpu.Buffer) ``` Copies a GPU buffer to a CPU staging buffer, maps it, and returns `[]float32`. This is the only synchronous GPU→CPU roundtrip in the training path; it is called once per batch to read back the partial loss sums. ### BindGroup Cache `GetBindGroup(pipeline, buffers...)` hashes the pipeline pointer and buffer pointers into a `uint64` key. If a matching `BindGroup` already exists, it is returned without re-creating it. This avoids rebuilding the descriptor set on every frame for stable weight+activation buffer pairs. --- ## Weight Sync Strategies `SyncToGPU()` on a `VolumetricLayer` uses different strategies depending on layer type and DType: ``` RMSNorm: Always uploads FP32 master. Quantization destroys normalization precision. SwiGLU (FP32): Splits Master into Gate, Up, Down slices. Uploads three separate persistent buffers. SwiGLU (INT4 / Q4_0): Calls syncQuantizedSwiGLU which quantizes each slice independently. Each component gets a scales buffer + packed uint32 buffer. Dense (INT4 / Q4_0): syncQuantizedDense: 32-weight blocks, scale per block, packed nibbles. MHA (FP32): Splits into Q/K/V/O weight buffers at internal DType codes 200/201/202/203. Also uploads optional q_norm/k_norm buffers at 204/205 when present. MHA (INT4): syncQuantizedMHA: quantizes each of Q/K/V/O separately. ``` The internal DType codes (100–102 for SwiGLU components, 200–203 for MHA projections) are a namespacing trick to store multiple named GPU buffers in the single `GPUWeights map[DType]any` without adding new struct fields. ### Inference load: release CPU weights after GPU upload For LLM inference (Lucy, SoulGlitch `LoomCreateLLM`), holding CPU **and** GPU copies of every layer during upload doubles peak RAM. Two mechanisms address this: **Per-layer (decoder blocks):** after `layer.SyncToGPU()`, call `layer.ReleaseInferenceHostWeights()` to drop CPU `Master` / `Versions` / `CPUPacked` once `GPUWeights` exist. Implemented in [`weights.go`](../poly/weights.go). **Global transformer weights (embeddings, LM head, final norm):** use `Transformer.SyncGlobalWeightsToGPUSequential()` instead of bulk `Transformer.SyncToGPU()`. Each global tensor uploads, then its CPU slice is cleared before the next starts. See [memory_history.md](memory_history.md) for measured before/after peaks. ``` Block loop (sequentialGPULoad): for each transformer block: SyncToGPU() on 4 layers ReleaseInferenceHostWeights() on those 4 layers Globals: SyncEmbeddingsToGPU → ReleaseEmbeddingsHost SyncLMHeadToGPU → ReleaseLMHeadHost (aliases buffer when tied to embeddings) SyncFinalNormToGPU → ReleaseFinalNormHost Final sweep: ForwardTokenIDsWGPU warmup → ReleaseInferenceHostWeights() on full transformer ``` Bulk `SyncToGPU()` on layers or `Transformer.SyncToGPU()` without release remains valid for **training** and legacy paths but is not the recommended inference load policy. --- ## Forward Dispatch (wgpu_forward.go) `ctx.DispatchForwardLayer(l, batchSize, inBuf, outBuf)` routes to the correct WGSL shader. Key functions: | Function | WGSL kernel | Notes | |:---------|:------------|:------| | `DispatchDenseForward` | matmul shader | register-tiled | | `DispatchRMSNorm` | RMSNorm shader | always FP32 weights | | `DispatchCNN1Forward` | 1D conv shader | | | `DispatchCNN2Forward` | 2D conv shader | 1826x vs CPU | | `DispatchCNN3Forward` | 3D conv shader | 7602x vs CPU | | `DispatchRNNForward` | RNN cell shader | | | `DispatchLSTMForward` | LSTM cell shader | | | `DispatchEmbedding` | gather shader | | | `DispatchMHAForward` | Q/K/V + attention | separate kernels | | `DispatchSwiGLUForward` | gate+up+down | BROKEN determinism | `DispatchActivation(n, act, inBuf, outBuf)` dispatches a shader that applies ReLU, SiLU, GELU, Tanh, or Sigmoid elementwise over `n` elements. --- ## Backward Dispatch (wgpu_backward_shaders.go) WGSL shaders for gradient computation: **Dense DX shader** (`ShaderDenseBackwardDX`): ```wgsl dx[b, i] = Σ_o dy[b, o] × W[o, i] // Implemented as tiled matmul using shared memory tiles: var dyTile: array; var wTile: array; ``` **Dense DW shader** (`ShaderDenseBackwardDW`): ```wgsl dW[o, i] = Σ_b dy[b, o] × x[b, i] // Uses atomic add for race-free accumulation across batch ``` **CNN DX/DW shaders**: Implement the "strided convolution" backward pass — the input gradient is the transposed convolution of the output gradient with the kernel, and the weight gradient is the correlation of the input with the output gradient. **Activation backward**: `DispatchActivationBackward` applies the activation derivative elementwise: `gradPre[i] = gradOut[i] × act'(preAct[i])`. **MSE gradient + partial loss** (`DispatchMSEGradPartialLoss`): ```wgsl grad[i] = (2.0 / N) × (pred[i] - target[i]) partial[wg] = Σ_{i in group} (pred[i] - target[i])² ``` **Apply gradients** (`DispatchApplyGradients`): ```wgsl weights[i] -= lr × dw[i] ``` --- ## GPU support: layer × `DType` (one table) Scope: **`VolumetricLayer.SyncToGPU`** + **`(*WGPUContext).DispatchForwardLayer`** in `poly.go` / `wgpu_kernels.go`. Symbol **`T`** means **`Transformer.ForwardTokenIDsWGPU`** / **`wgpu_forward.go`** (LLM inference) for that layer+dtype, not generic batch dispatch. Activations are **`f32`** WGSL; **`DTypeFloat64`** is coerced to the **`Float32`** weight-buffer path in the `hasSpecialPath` / morph block (see `SyncToGPU`). | Symbol | Meaning | |:------:|---------| | **Y** | **Generic GPU forward OK**: `SyncToGPU` does not skip the `MorphToFloat32ForGPU` upload **or** uses a matching native path (`DispatchDenseQ4` for **Dense+Int4** only; **CNN1** packed when `isCNN1NativeGPUQuantDType`). | | **T** | **Transformer path only** (`wgpu_forward.go`): QKV/O use **`DispatchDenseQ4`** / **`DispatchDenseI8`**; SwiGLU gate/up may use **`DispatchSwiGLUQ4`**. **Not** correct for generic **`DispatchForwardLayer`** on that dtype (quantized buffers + **`DispatchDense`** / **`DispatchSwiGLUWithActCache`** mismatch). | | **–** | **Not supported** after vanilla `SyncToGPU` + generic `DispatchForwardLayer` (skipped morph with no valid weight buffer, or packed weights fed to an **`f32`** matmul / SwiGLU shader). | | **·** | **DType N/A** (no weight tensor for that layer). | **Dense:** only **`DTypeInt4`** selects **`DispatchDenseQ4`**. Wider dtypes (**2–13, 15–20** except **14**) hit **`hasSpecialPath`** with no quant branch → morph skipped → **–**. Eight-bit dtypes on Dense get **`syncQuantizedDenseI8`** but **`DispatchDenseTiled`** expects **`f32`** layout → **–**. **`ensureGPUFloat32Weights`** (training) can still attach **`GPUWeights[Float32]`** so matmul runs on the **FP32 master** regardless of `l.DType` (not reflected as **Y** here). | ID | `DType` | Dense | RMSNorm | CNN1 | CNN2 | CNN3 | RNN | LSTM | Embedding | Softmax | MHA | SwiGLU | Residual | |---:|---------|:-----:|:-------:|:----:|:----:|:----:|:---:|:----:|:---------:|:-------:|:---:|:------:|:--------:| | 0 | Float64 | Y | Y | Y | Y | Y | Y | Y | Y | · | Y | Y | · | | 1 | Float32 | Y | Y | Y | Y | Y | Y | Y | Y | · | Y | Y | · | | 2 | Float16 | – | Y | Y | Y | Y | Y | Y | Y | · | Y | Y | · | | 3 | BFloat16 | – | Y | Y | Y | Y | Y | Y | Y | · | Y | Y | · | | 4 | FP8 E4M3 | – | Y | Y | Y | Y | Y | Y | Y | · | T | T | · | | 5 | FP8 E5M2 | – | Y | Y | Y | Y | Y | Y | Y | · | T | T | · | | 6 | Int64 | – | Y | Y | Y | Y | Y | Y | Y | · | Y | Y | · | | 7 | Int32 | – | Y | Y | Y | Y | Y | Y | Y | · | Y | Y | · | | 8 | Int16 | – | Y | Y | Y | Y | Y | Y | Y | · | Y | Y | · | | 9 | Int8 | – | Y | Y | Y | Y | Y | Y | Y | · | T | T | · | | 10 | Uint64 | – | Y | Y | Y | Y | Y | Y | Y | · | Y | Y | · | | 11 | Uint32 | – | Y | Y | Y | Y | Y | Y | Y | · | Y | Y | · | | 12 | Uint16 | – | Y | Y | Y | Y | Y | Y | Y | · | Y | Y | · | | 13 | Uint8 | – | Y | Y | Y | Y | Y | Y | Y | · | T | T | · | | 14 | Int4 | Y | Y | Y | Y | Y | Y | Y | Y | · | T | T | · | | 15 | Uint4 | – | Y | Y | Y | Y | Y | Y | Y | · | T | T | · | | 16 | FP4 | – | Y | Y | Y | Y | Y | Y | Y | · | T | T | · | | 17 | Int2 | – | Y | Y | Y | Y | Y | Y | Y | · | T | T | · | | 18 | Uint2 | – | Y | Y | Y | Y | Y | Y | Y | · | T | T | · | | 19 | Ternary | – | Y | Y | Y | Y | Y | Y | Y | · | T | T | · | | 20 | Binary | – | Y | Y | Y | Y | Y | Y | Y | · | T | T | · | **CNN1 column:** **Y** = either **`DispatchCNN1Packed`** (dtype in `isCNN1NativeGPUQuantDType`: Int8, Int4, Int2, FP4, Ternary, Binary, FP8×2, Uint8, Uint4, Uint2, Float16, BFloat16, Int16) or **`DispatchCNN1`** on **`MorphToFloat32ForGPU`** otherwise. **Not in this table:** `LayerLayerNorm`, `LayerConvTransposed*`, `LayerKMeans`, `LayerParallel`, `LayerSequential`, `LayerMetacognition` (no `DispatchForwardLayer` arm). See [numerical_types.md](numerical_types.md) for the **`DType`** enum and **`WeightStore`**. **GPU training:** `gpuTrainingNeedsCPUFallback` in `training.go` forces a **CPU** optimizer step when the net includes **MHA**, **SwiGLU**, **Dense+Int4**, or **RNN/LSTM** with **Int8/Int4**. --- The project uses **Numerical Tiling** to map 3D volumetric layers to GPU workgroups. ### SC (single-workgroup) vs MC (multi-workgroup) profiles Loom differentiates two dispatch profiles for GPU kernels (attention, dense, SwiGLU, CNN, etc.): - **SC**: Smaller workgroups / tiles — lower register pressure, friendlier to tight limits (edge GPUs, WASM). - **MC**: Larger tiles where limits allow — higher throughput on desktop-class GPUs. At **inference**, transformer-style forwards (`wgpu_forward.go`) choose per-layer tile sizes with `layer.GetGPUSCTileSize(dtype)` vs `layer.GetGPUMCTileSize(dtype)` according to **`VolumetricNetwork.EnableMultiCoreTiling`** (with the same field mirrored on layers when set). That is the primary switch — not `GPUTileSize` alone. `WGPUContext.GPUTileSize` is still the device-tuned baseline derived from `CalculateOptimalGPUTileSizeFromLimits` and feeds into how SC/MC maps are built in `refreshRuntimeGPUTileSizes`. **GPU training** may ignore the network flag and pick SC vs MC directly via `TrainingModeGPUSC` / `TrainingModeGPUMC` (`training.go`). **CPU:** poly does **not** expose SC vs MC as two tile maps on the CPU side — layers use **`CPUTileSizes` / `GetCPUTileSize` only**. See the **“GPU: two tile maps…”** and **“CPU: one tile map…”** subsections in [dispatch.md](dispatch.md). --- ## Transformer GPU Forward (wgpu_forward.go) `Transformer.ForwardTokenIDsWGPU` is the optimized path for LLM inference: 1. If `tokens != nil` and GPU embeddings are loaded, dispatch a gather shader to convert token IDs → hidden states entirely on-GPU 2. `BeginFrame()` — all subsequent ops recorded into one encoder 3. For each transformer block (4 layers: RMSNorm → MHA → RMSNorm → SwiGLU): - Dispatch `DispatchRMSNorm` - Dispatch Q/K/V projections separately (supports expanded QueryDim) - Optional Q/K RMSNorm using q_norm/k_norm buffers - Dispatch RoPE rotation - Dispatch attention score + softmax - Dispatch output projection - Add residual 4. Final norm + LM head if on GPU 5. `FlushFrame()` — single submit 6. Read back only the logits (one small buffer) This path achieves the "260+ tokens/s prefill on M4" figure mentioned in the README. ### Qwen / Expanded-Query Notes Loom's GPU path now supports architectures where `query_dim != d_model` (for example Qwen3-0.6B with `head_dim=128`, `num_heads=16`, `query_dim=2048`, `d_model=1024`). Key implementation details: - MHA shader workgroup width scales with `head_dim` (not hardcoded to 64). - Q projection and attention output buffers use `query_dim`. - O projection uses `input=query_dim`, `output=d_model`. - RMSNorm epsilon is propagated from checkpoint config (`rms_norm_eps`) for parity with CPU. --- ## The step mesh engine Source: https://openfluke.com/docs/step Markdown: https://openfluke.com/docs/step.md # The step mesh engine This document covers the `StepState`, `StepForward`, `StepBackward`, and `StepApplyTween` functions that implement a clock-cycle-accurate discrete-time neural mesh. --- ## What is the step mesh? Standard `ForwardPolymorphic` runs the entire network in one sequential sweep — input enters at coordinate (0,0,0,0) and the final output exits at the last coordinate. This is a **one-shot** pass. The **Step mesh engine** treats the 3D grid as a living mesh. Each "tick" of the neural clock fires every layer simultaneously. Each layer reads from the previous tick's output buffers and writes to a new set of output buffers. After all layers have fired, the buffers swap. This is classical **double buffering** applied to neural computation. ``` Standard ForwardPolymorphic: Input ──▶ L0 ──▶ L1 ──▶ L2 ──▶ L3 ──▶ Output (one complete pass per call) Step mesh (one clock cycle): Tick N: Tick N+1: ┌──────┬──────┬──────┐ ┌──────┬──────┬──────┐ │ L0 │ L1 │ L2 │ │ L0 │ L1 │ L2 │ │fires │fires │fires │ ──swap──▶│fires │fires │fires │ │ │ │ │ buffers │ │ │ │ └──────┴──────┴──────┘ └──────┴──────┴──────┘ All layers process simultaneously Same pattern ``` The key insight: **every layer in the grid has the opportunity to update its output every clock cycle**, not just when an input happens to flow through it sequentially. --- ## StepState ```go type StepState[T Numeric] struct { LayerData []*Tensor[T] // current output of every layer NextBuffer []*Tensor[T] // write target for the current tick HistoryIn [][]*Tensor[T] // [step][layerIdx] → input to that layer at that step HistoryPre [][]*Tensor[T] // [step][layerIdx] → preAct at that step StepCount uint64 mu sync.RWMutex TweenState *TweenState[T] // optional tween bridge (neural target propagation) lastInput *Tensor[T] } ``` `LayerData[idx]` is what layer `idx` produced in the **previous** clock cycle. `NextBuffer[idx]` is what layer `idx` will produce in the **current** cycle. After the cycle, they swap. Create with: ```go state := poly.NewStepState[float32](network) state.SetInput(inputTensor) // loads input into LayerData[0] ``` --- ## StepForward: One Clock Cycle ```go elapsed := poly.StepForward(network, state, captureHistory bool) ``` Each call advances the mesh by exactly one discrete time step. All layers execute during this one call. ### Sequential Mode (UseTiling = false) ```go for idx := range n.Layers { l := &n.Layers[idx] if l.IsDisabled { pass through; continue } // Resolve input source var input *Tensor[T] if l.IsRemoteLink { tIdx := n.GetIndex(l.TargetZ, l.TargetY, l.TargetX, l.TargetL) input = s.LayerData[tIdx] // reads from REMOTE layer's output } else if idx > 0 { input = s.LayerData[idx-1] // reads from preceding layer } else { input = s.LayerData[0] // reads injection point } pre, post := DispatchLayer(l, input, nil) s.NextBuffer[idx] = post } // Swap double buffers copy(s.LayerData, s.NextBuffer) s.StepCount++ ``` ### Parallel Tiled Mode (UseTiling = true) When `n.UseTiling = true`, goroutines process 4×4×4 spatial tiles concurrently: ```go var wg sync.WaitGroup for zTile ...: for yTile ...: for xTile ...: wg.Add(1) go func(zT, zE, yT, yE, xT, xE int) { defer wg.Done() for z := zT; z < zE; z++ { for y := yT; y < yE; y++ { for x := xT; x < xE; x++ { // dispatch layers in this tile } } } }(...) wg.Wait() ``` The mutex (`s.mu`) is held for the duration of the sequential path, and for individual history writes in the parallel path. The `NextBuffer` slice is pre-allocated so concurrent writes to different indices are safe. ### History Capture If `captureHistory = true`, each tick appends to `HistoryIn` and `HistoryPre`: ``` After tick N: HistoryIn[N][idx] = what layer idx received HistoryPre[N][idx] = preAct that layer idx produced ``` This history is the foundation for `StepBackward` (BPTT) and is required before calling `StepBackward`. It consumes memory proportional to `Steps × Layers × FeatureSize` — use only when training. --- ## Spatial Feedback (Remote Links in step mesh mode) The step mesh engine is where `IsRemoteLink` reaches its full potential. Because `s.LayerData[tIdx]` is always the **previous tick's** output (not the current tick's), a remote link to an earlier coordinate creates genuine recurrence: ``` Tick N-1: Layer A (0,0,0) produces output → stored in LayerData[0] Tick N: Layer B (0,2,0) has IsRemoteLink pointing to (0,0,0) → Layer B reads LayerData[0] (from tick N-1, not current tick) → Layer B effectively "remembers" what A produced one cycle ago This is the discrete-time equivalent of an RNN hidden state. ``` ``` ┌────────────────────────────────────────────────────────────────┐ │ SPATIAL FEEDBACK DIAGRAM │ │ │ │ Tick N-1: A ──output──▶ LayerData[A] │ │ │ │ Tick N: B ──IsRemoteLink──▶ reads LayerData[A] from N-1 │ │ B produces new output → LayerData[B] │ │ │ │ Tick N+1: A reads updated B output if A is also remote │ │ → Full spatial RNN at mesh scale │ └────────────────────────────────────────────────────────────────┘ ``` --- ## StepBackward: BPTT Through the Mesh ```go gradIn, layerGradients, err := poly.StepBackward(network, state, gradOutput) ``` This implements **Backpropagation Through Time (BPTT)** across the step mesh history. It walks backwards through both time steps and spatial coordinates. ### Algorithm ``` gradBuffers[numLayers-1] = gradOutput // seed with final error for step from (numSteps-1) downto 0: nextGradBuffers = new zero buffers for idx from (numLayers-1) downto 0: input = HistoryIn[step][idx] pre = HistoryPre[step][idx] grad = gradBuffers[idx] gIn, gW = DispatchLayerBackward(l, grad, input, nil, pre) // Accumulate weight gradients across all time steps layerGradients[idx][1] += gW (if exists) // Route gIn back to the source of input for this layer accumulateMeshGrad(network, nextGradBuffers, idx, gIn) gradBuffers = nextGradBuffers return gradBuffers[0] // gradient with respect to the initial input ``` `accumulateMeshGrad` determines where to send `gIn`: - If `IsRemoteLink`: send to `TargetZ/Y/X/L` coordinates - Otherwise: send to `idx - 1` - If `idx == 0`: send to the input site This correctly routes gradients through the spatial topology — remote links receive their share of the gradient from every layer that consumed their output. --- ## StepApplyTween ```go poly.StepApplyTween(network, state, globalTarget, lr) ``` Bridges the step mesh mesh with the `Tween` machinery. At each call: 1. If `state.TweenState == nil`, create a new `TweenState` with `UseChainRule = false` (gap-based learning — appropriate for the continuous-time mesh) 2. Copy current `LayerData` into `tpState.ForwardActs` (the mesh's current "what is" state) 3. Call `TweenBackward(n, tpState, globalTarget)` to compute what each layer *should* produce 4. `CalculateLinkBudgets()` — measure cosine similarity between actual and target at each node 5. `ApplyTweenGaps(n, tpState, lr)` — update weights using the gap signal, gated by link budgets This enables **online, asynchronous learning** on a live mesh — you can inject a global target at any time and the weights update locally at each node based on their current output gap. --- ## Double Buffer Guarantees The double buffer swap (`copy(s.LayerData, s.NextBuffer)`) happens after all layers have written to `NextBuffer`. This guarantees: 1. A layer at coordinate (0,0,2) cannot see the output of (0,0,1) from the *current* tick, only from the previous tick 2. Concurrent goroutines in tiled mode write to different indices of `NextBuffer` without conflict 3. Remote links always see stable, previous-tick values regardless of which goroutine happens to fire first This is the "clock cycle accuracy" mentioned in the README. ## V0.75.0 Stability & Guarding The Step mesh engine was fundamentally stabilized in v0.75.0 to support sparse volumetric grids without runtime panics. ### 1. Volumetric Coordinate Guarding In previous versions, a misconfigured grid cell could lead to a `nil pointer dereference`. In v0.75.0, the dispatcher implements strict guarding: - **`IsDisabled` Flag**: Every grid cell now defaults to "Disabled". They must be explicitly enabled during network construction via the `poly.VolumetricLayer` configuration. - **Nil-Safety**: The `DispatchLayer` and `StepForward` loops check these flags before execution, ensuring that uninitialized memory in sparse 3D regions does not cause a crash. ### 2. Explicit Coordinate Hopping Stability is further guaranteed by the enforcement of 3D volumetric coordinates (`z, y, x, l`). - **Deterministic Routing**: Every connection, whether a standard sequence or a remote `IsRemoteLink`, is resolved to a specific 3D coordinate. - **Grid Consistency**: This ensures that even in massively parallel tiled modes, the signal wavefront remains spatially consistent and bit-perfect across all 21 numerical types. --- ## When to Use the Step mesh engine Use `StepForward` / `StepApplyTween` when you need: - **Continuous operation**: the network runs indefinitely, processing new inputs each tick - **Spatial feedback**: remote links that create mesh-level recurrence - **Online learning**: weight updates interleaved with forward passes - **Parallel processing**: the tiled mode can saturate multi-core CPUs Use `ForwardPolymorphic` / `BackwardPolymorphic` when you need: - **Batch training**: multiple training examples per weight update - **GPU acceleration**: the GPU path uses `trainBatchWGPU`, not the step mesh engine - **Deterministic single-pass inference**: no history overhead > [!TIP] > The README's phrase "use `StepForward` and `StepApplyTween` when you need a living network that evolves and learns over time rather than a static pipeline" captures this distinction perfectly. --- ## The DNA Engine: Topological Network Fingerprinting Source: https://openfluke.com/docs/dna Markdown: https://openfluke.com/docs/dna.md # The DNA Engine: Topological Network Fingerprinting This document covers `ExtractDNA`, `CosineSimilarity`, `CompareNetworks`, `LogicShift` detection, and the recursive signature extraction for all 19 layer types in `dna.go`. For the **Evolution Engine** (DNA Splice + NEAT mutations), see [evolution.md](evolution.md). --- ## Why DNA? Standard weight comparison breaks across precisions — you can't directly compare an INT8 weight against an FP32 weight. The DNA engine solves this by converting every layer's weights to a **unit direction vector** after simulating precision loss. Comparing direction vectors (cosine similarity) instead of raw values means: - FP32 and INT8 representations of the same model look nearly identical - Two networks trained on the same task converge toward the same DNA - Structural changes (different layer order, different grid positions) are detectable as **logic shifts** ``` Raw FP32 weights ──► scale (× ws.Scale) ──► Normalize ──► unit vector │ (L2 norm) "DNA strand" │ └── FP4 weights ──► scale (× ws.Scale) ──► Normalize ──► same direction ≈ 1.0 similarity ``` --- ## Core Types ```go // The "DNA strand" of a single layer type LayerSignature struct { Z, Y, X, L int // 3D grid coordinates Type LayerType DType DType Weights []float32 // L2-normalized, scale-applied master weights } // The complete genetic blueprint of a network type NetworkDNA []LayerSignature ``` --- ## ExtractDNA — all 19 layer types ```go func ExtractDNA(n *VolumetricNetwork) NetworkDNA ``` Iterates every layer in the network, calls `extractLayerSignature(l)`, and wraps the result with position and type metadata. The signature extraction logic handles all 19 layer types: ``` VolumetricNetwork │ ┌────────────┼────────────┐ │ │ │ LayerDense LayerParallel LayerSoftmax LayerRNN LayerSequential LayerResidual LayerLSTM (recursive) (weightless) LayerMHA LayerSwiGLU LayerRMSNorm LayerLayerNorm LayerCNN1/2/3 LayerConvT1/2/3D LayerEmbedding LayerKMeans │ ▼ extractLayerSignature(l) │ ┌─────────┼──────────────┐ │ │ │ ▼ ▼ ▼ weighted recursive weightless layers containers layers │ │ │ ▼ ▼ ▼ Master flatten all []float32{1.0} weights branches │ │ ▼ ▼ scale(×ws.Scale) Normalize(concat) │ ▼ Normalize │ ▼ []float32 unit vector ``` ### Weighted layers (Dense, RNN, LSTM, MHA, CNN*, ConvTransposed*, SwiGLU, RMSNorm, LayerNorm, Embedding, KMeans) ```go // All weighted layers follow this path: scale := l.WeightStore.Scale if scale == 0 { scale = 1.0 } simulated := make([]float32, len(l.WeightStore.Master)) for i, w := range l.WeightStore.Master { if scale != 1.0 { simulated[i] = w * scale } else { simulated[i] = w } } return Normalize(simulated) ``` Applying the layer's scale factor before normalizing means the DNA of an INT8 Dense layer and an FP32 Dense layer with the same trained weights will be nearly identical — both normalize to the same unit direction. ### Structural containers (Parallel, Sequential) — recursive extraction Parallel and Sequential layers contain nested layers (`ParallelBranches`, `SequentialLayers`). A naive approach that returned `{1.0}` for both would make any two parallel layers look identical regardless of what's inside them. Instead, the engine recurses: ``` LayerParallel ├── Branch 0 (Dense 32×32) ──► extractLayerSignature ──► unit vec A ─┐ ├── Branch 1 (RMSNorm 32) ──► extractLayerSignature ──► unit vec B ─┤ concat └── FilterGateConfig (Dense) ──► extractLayerSignature ──► unit vec C ─┘ │ Normalize(flat) │ single unit vec representing ALL nested weights ``` ```go case LayerParallel: var flat []float32 for _, branch := range l.ParallelBranches { if branch.IsRemoteLink { continue } // remote links have no local weights flat = append(flat, extractLayerSignature(branch)...) } if l.FilterGateConfig != nil { flat = append(flat, extractLayerSignature(*l.FilterGateConfig)...) } if len(flat) == 0 { return []float32{1.0} } return Normalize(flat) case LayerSequential: var flat []float32 for _, sub := range l.SequentialLayers { flat = append(flat, extractLayerSignature(sub)...) } if len(flat) == 0 { return []float32{1.0} } return Normalize(flat) ``` Remote links (`IsRemoteLink = true`) are spatial hops with no local weights — they are skipped during extraction. ### Weightless layers (Softmax, Residual) ```go case LayerSoftmax, LayerResidual: return []float32{1.0} ``` A `{1.0}` vector is a neutral presence marker. Two Softmax layers at the same position will score `1.0` similarity (identical), which is correct — they are architecturally identical by definition. --- ## Normalize ```go func Normalize(v []float32) []float32 ``` Converts a weight vector to a unit vector: ``` mag = sqrt(v[0]² + v[1]² + ... + v[n]²) output[i] = v[i] / mag ``` - If `mag == 0` (all-zero weights), returns a zero vector - Two zero vectors score `1.0` similarity (both represent an untrained/zeroed layer) - One zero + one nonzero scores `0.0` (orthogonal by convention) --- ## CosineSimilarity ```go func CosineSimilarity(s1, s2 LayerSignature) float32 ``` Returns a score in `[-1.0, 1.0]` comparing two layer signatures: ``` s1.Weights · s2.Weights sim = ───────────────────────── = dot product (since |s1| = |s2| = 1) |s1| × |s2| ``` Guard rails: | Condition | Returns | |:----------|:--------| | `s1.Type != s2.Type` | `0.0` — architectural mismatch | | `s1.DType != s2.DType` | `0.0` — precision mismatch | | `len(s1.Weights) != len(s2.Weights)` | `0.0` — dimension mismatch | | Both zero vectors | `1.0` — identical untrained layers | | One zero, one nonzero | `0.0` — no similarity | Similarity values to interpret: ``` -1.0 ──────────── 0.0 ──────────── +1.0 │ │ │ opposite no match identical direction direction (learned to (different (same functional do opposite) purpose) role) ``` --- ## CompareNetworks ```go func CompareNetworks(dna1, dna2 NetworkDNA) NetworkComparisonResult type NetworkComparisonResult struct { OverallOverlap float32 LayerOverlaps map[string]float32 // "z,y,x,l" → score LogicShifts []LogicShift } ``` Two-phase comparison: ### Phase 1 — Direct Position Matching Match each layer in `dna1` with the layer at the same `(Z, Y, X, L)` position in `dna2`: ``` dna1: [L0: Dense] [L1: RNN] [L2: Dense] │ │ │ │ same pos │ │ same pos ▼ ▼ ▼ dna2: [L0: Dense] [L1: Dense] [L2: Dense] │ │ │ sim=0.94 sim=0.0 sim=0.87 (0.0 because type mismatch) │ │ │ └──────────────┴──────────┘ │ avg = 0.60 OverallOverlap = 0.60 ``` ### Phase 2 — Logic Drift Detection For each layer in `dna1`, search **all** positions in `dna2` for the best cosine match — not just the same position: ``` dna1 L0 (Dense, sim vector A) │ ├──► compare vs dna2 L0 → sim=0.72 ├──► compare vs dna2 L1 → sim=0.31 └──► compare vs dna2 L2 → sim=0.91 ← best match! Best match (0.91) is at position L2, not L0. Since 0.91 > 0.8 threshold AND positions differ: → LogicShift { SourcePos:"0,0,0,0", TargetPos:"0,0,0,2", Overlap:0.91 } ``` ```go type LogicShift struct { SourcePos string // "z,y,x,l" in dna1 TargetPos string // "z,y,x,l" in dna2 Overlap float32 // cosine score > 0.8 } ``` Logic shifts appear when: - A network was restructured and layers were reordered - A NEAT mutation moved a functional pattern to a different grid position - Two networks converged to the same function at different coordinates --- ## Full DNA Pipeline ``` Network A (trained) Network B (trained) │ │ ▼ ▼ ExtractDNA(A) ExtractDNA(B) │ │ for each layer: for each layer: ┌──────────────────────────────┐ ┌──────────────────────────────┐ │ Parallel/Sequential: │ │ Parallel/Sequential: │ │ recurse into branches │ │ recurse into branches │ │ concat + Normalize │ │ concat + Normalize │ │ Weighted: │ │ Weighted: │ │ scale(w × ws.Scale) │ │ scale(w × ws.Scale) │ │ Normalize(simulated) │ │ Normalize(simulated) │ │ Weightless: │ │ Weightless: │ │ {1.0} │ │ {1.0} │ └──────────────────────────────┘ └──────────────────────────────┘ │ │ │ NetworkDNA ([]LayerSignature) │ NetworkDNA └─────────────────┬────────────────────┘ │ ▼ CompareNetworks(dnaA, dnaB) │ ┌────────────┴────────────┐ │ │ ▼ ▼ Phase 1: Direct Phase 2: Cross-pos position matching best-match search │ │ LayerOverlaps LogicShifts OverallOverlap (migrations) │ └────────────────────────▶ NetworkComparisonResult ``` --- ## Use Cases ### Measuring Quantization Fidelity ```go dnaFP32 := poly.ExtractDNA(net) // morph all layers to INT8... poly.MorphAllLayers(net, poly.DTypeInt8) dnaINT8 := poly.ExtractDNA(net) result := poly.CompareNetworks(dnaFP32, dnaINT8) // result.OverallOverlap near 1.0 → quantization preserved behavior // result.OverallOverlap near 0.0 → quantization destroyed the model ``` ### Detecting Training Convergence Sample DNA every N epochs. When `OverallOverlap` between consecutive snapshots stabilizes above 0.99, the network has converged. ``` Epoch 0 → Epoch 10 : overlap = 0.12 (learning fast) Epoch 10 → Epoch 50 : overlap = 0.61 (settling) Epoch 50 → Epoch 100 : overlap = 0.94 (nearly converged) Epoch 100 → Epoch 150 : overlap = 0.99 (converged) ``` ### Cross-Architecture Similarity Two networks with different layer counts share coordinates for only the positions they have in common. `CompareNetworks` will match only those overlapping positions, and the `OverallOverlap` is averaged over matched layers only. ### Logic Drift After NEAT Mutations After a NEAT topology mutation moves a Dense layer from position `0,0,0,0` to `0,0,0,2`, the logic shift detector will report: ``` LogicShift { SourcePos: "0,0,0,0", TargetPos: "0,0,0,2", Overlap: 0.93, } ``` This is how you track functional identity across structural mutations. --- ## DNA Signature Sizes by Layer Type | Layer Type | Signature Length | Notes | |:-----------|:-----------------|:------| | Dense (32) | 1024 | inputH × outputH | | MHA (32, 4 heads) | 4224 | Q+K+V+O projections + biases | | SwiGLU (32) | 6144 | gate + up + down × 3 projections | | RMSNorm (32) | 32 | scale vector only | | LayerNorm (32) | 64 | gamma + beta | | CNN1/2 (8f, 1c, k3) | 72 | filters × channels × k² | | CNN3 (8f, 1c, k3) | 216 | filters × channels × k³ | | RNN (32) | 2080 | Wx + Wh + bias | | LSTM (32) | 8320 | 4 gates × (Wx + Wh + bias) | | Embedding (256 vocab, 32 dim) | 8192 | vocab × dim | | KMeans (8 clusters, 32 dim) | 256 | clusters × dim | | Softmax | 1 | neutral marker | | Residual | 1 | neutral marker | | Parallel (2× Dense 32) | 1056 | concat of branches, renormalized | | Sequential (2× Dense 32) | 2048 | concat of sub-layers, renormalized | --- ## The Evolution Engine: DNA Splice & NEAT Topology Evolution Source: https://openfluke.com/docs/evolution Markdown: https://openfluke.com/docs/evolution.md # The Evolution Engine: DNA Splice & NEAT Topology Evolution This document covers `SpliceDNA`, `SpliceDNAWithReport`, `NEATMutate`, and `NEATPopulation` from `evolution.go`. The evolution engine builds on the DNA fingerprinting system described in [dna.md](dna.md). --- ## Two Evolutionary Mechanisms ``` ┌─────────────────────────────────────────────────────────────┐ │ Evolution Engine │ │ │ │ ┌────────────────────┐ ┌──────────────────────────┐ │ │ │ DNA Splice │ │ NEAT-style Mutation │ │ │ │ (Crossover) │ │ (Topology Evolution) │ │ │ │ │ │ │ │ │ │ ParentA + ParentB │ │ Network ──► mutated │ │ │ │ ──► Child │ │ clone │ │ │ │ │ │ │ │ │ │ merges weights │ │ changes layer types, │ │ │ │ guided by DNA │ │ activations, topology │ │ │ │ similarity │ │ weights │ │ │ └────────────────────┘ └──────────────────────────┘ │ │ │ │ │ │ └──────────┬───────────────┘ │ │ ▼ │ │ NEATPopulation.Evolve() │ │ (combines both in a generation loop) │ └─────────────────────────────────────────────────────────────┘ ``` --- ## Part 1 — DNA Splice / Genetic Crossover ### Concept Given two trained parent networks `A` and `B`, produce a child network whose weights are a blend of both. The blend is **guided by DNA similarity** — layers that are more similar between parents get blended more aggressively; layers that diverged get a heavier bias toward the fitter parent. ``` ParentA (trained) ParentB (trained) │ │ ExtractDNA(A) ExtractDNA(B) │ │ sigA per layer sigB per layer │ │ └────────┬───────────────┘ │ for each layer position (z,y,x,l): │ CosineSimilarity(sigA, sigB) │ ┌──────┴──────┐ │ │ blend skip weights (keep A's from A+B weights) │ ▼ Child network ``` ### SpliceConfig ```go type SpliceConfig struct { CrossoverMode string // "blend", "point", or "uniform" BlendAlpha float32 // interpolation factor (blend mode): 0=all A, 1=all B SplitRatio float64 // fraction from A in point mode (e.g. 0.5) FitnessA float64 // optional: used to bias toward fitter parent FitnessB float64 } func DefaultSpliceConfig() SpliceConfig { return SpliceConfig{CrossoverMode: "blend", BlendAlpha: 0.5, SplitRatio: 0.5} } ``` ### Three Crossover Modes #### Mode: "blend" (default) Interpolates weights per element. Alpha is modulated by the layer's cosine similarity and relative fitness: ``` alpha = FitnessB / (FitnessA + FitnessB) ← bias toward fitter parent alpha = alpha × (0.5 + 0.5 × similarity) ← scale by how similar layers are child[i] = wA[i] × (1 - alpha) + wB[i] × alpha ``` When similarity is high (layers learned the same thing), alpha blends freely. When similarity is low (layers diverged), alpha is pulled toward the fitter parent. ``` similarity = 1.0 ──► free blend (both parents contribute equally) similarity = 0.0 ──► take mostly from fitter parent (layers are unrelated) similarity = -1.0 ──► heavily bias toward fitter parent (opposite patterns) ``` #### Mode: "point" Splits weights at a single cut point. First `SplitRatio` fraction from A, rest from B: ``` wA: [a0 a1 a2 a3 a4 a5 a6 a7] wB: [b0 b1 b2 b3 b4 b5 b6 b7] │ SplitRatio=0.5 │ child: [a0 a1 a2 a3 b4 b5 b6 b7] ─── from A ──── from B ── ``` #### Mode: "uniform" Each weight is randomly drawn from A or B, with probability biased toward the fitter parent: ``` threshold = FitnessA / (FitnessA + FitnessB) for each weight i: if rand < threshold → child[i] = wA[i] else → child[i] = wB[i] ``` ### SpliceDNA ```go func SpliceDNA(parentA, parentB *VolumetricNetwork, cfg SpliceConfig) *VolumetricNetwork ``` - The child is always a **deep clone of parentA** (architecture inherited from A) - Only layers where both parents have matching positions **and matching weight dimensions** are blended - If `parentB` has no layer at that position, or the weight counts differ, A's weights are kept unchanged ```go // Guard: skip if dimensions don't match if wB == nil || len(wB) != len(wA) { continue // keep A's weights } ``` ### SpliceDNAWithReport ```go func SpliceDNAWithReport(parentA, parentB *VolumetricNetwork, cfg SpliceConfig) SpliceResult type SpliceResult struct { Child *VolumetricNetwork ParentADNA NetworkDNA ParentBDNA NetworkDNA ChildDNA NetworkDNA Similarities map[string]float32 // "z,y,x,l" → cosine score used for blending BlendedCount int // how many layers were actually blended } ``` Returns the same child as `SpliceDNA` plus a full diagnostic report. Use this when debugging crossover behavior or logging ancestry. --- ## Part 2 — NEAT-style Topology Evolution ### Concept NEAT (NeuroEvolution of Augmenting Topologies) mutates both weights and structure. The implementation here applies six mutation types to a cloned network, leaving the original untouched. ``` Original Network (immutable) │ cloneNetwork() │ mutated clone │ ┌────┴────────────────────────────────────────────┐ │ Per-layer mutations (applied sequentially): │ │ │ │ 1. Weight perturbation ── add Gaussian noise │ │ 2. Activation mutation ── swap act function │ │ 3. Node mutation ── change layer type │ │ 4. Layer toggle ── enable/disable │ │ │ │ Network-level mutations (applied once): │ │ │ │ 5. Connection add ── insert remote link │ │ 6. Connection drop ── remove remote link │ └─────────────────────────────────────────────────┘ │ returns mutated clone ``` ### NEATConfig ```go type NEATConfig struct { WeightPerturbRate float64 // prob of perturbing a layer's weights (default 0.8) WeightPerturbScale float32 // noise magnitude (default 0.05) NodeMutateRate float64 // prob of changing a layer's type (default 0.1) ConnectionAddRate float64 // prob of adding a remote link (default 0.05) ConnectionDropRate float64 // prob of removing a remote link (default 0.02) ActivationMutRate float64 // prob of changing activation function (default 0.1) LayerToggleRate float64 // prob of toggling IsDisabled (default 0.02) DModel int // reference dimension for weight reinitialization AllowedLayerTypes []LayerType // types a node can mutate to // Type-specific defaults used by neatReinitLayer: DefaultNumHeads int DefaultInChannels int DefaultFilters int DefaultKernelSize int DefaultVocabSize int DefaultNumClusters int Seed int64 } ``` `DefaultNEATConfig(dModel)` returns conservative rates with all 17 mutable layer types in `AllowedLayerTypes`. ### NEATMutate ```go func NEATMutate(n *VolumetricNetwork, cfg NEATConfig) *VolumetricNetwork ``` The original network `n` is **never modified**. The function clones it and applies mutations: ``` For each layer i: Step 1 — Weight Perturbation (WeightPerturbRate = 0.8) ┌─────────────────────────────────────────────────────┐ │ master[i] += rand(-1, 1) × WeightPerturbScale │ │ (clears cached DType versions as weights changed) │ └─────────────────────────────────────────────────────┘ Step 2 — Activation Mutation (ActivationMutRate = 0.1) ┌──────────────────────────────────────────────────────┐ │ layer.Activation = random from {ReLU, SiLU, GELU, │ │ Tanh, Sigmoid, Linear}│ └──────────────────────────────────────────────────────┘ Step 3 — Node Mutation (NodeMutateRate = 0.1) ┌──────────────────────────────────────────────────────┐ │ newType = random from AllowedLayerTypes (≠ current) │ │ neatReinitLayer(child, i, newType, cfg) │ │ → sets new Type, InputHeight, OutputHeight │ │ → creates fresh WeightStore with correct wCount │ └──────────────────────────────────────────────────────┘ Step 4 — Layer Toggle (LayerToggleRate = 0.02) ┌──────────────────────────────────────────────────────┐ │ layer.IsDisabled = !layer.IsDisabled │ │ (disabled layers are skipped during forward pass) │ └──────────────────────────────────────────────────────┘ After all layers: Step 5 — Connection Add (ConnectionAddRate = 0.05) ┌──────────────────────────────────────────────────────┐ │ Pick two random layers src and dst (src ≠ dst) │ │ Append IsRemoteLink branch to src.ParallelBranches │ │ TargetZ/Y/X/L point to dst │ │ Creates a spatial "skip connection" in the 3D grid │ └──────────────────────────────────────────────────────┘ Step 6 — Connection Drop (ConnectionDropRate = 0.02) ┌──────────────────────────────────────────────────────┐ │ Find a layer with ParallelBranches containing │ │ IsRemoteLink entries │ │ Remove one at random │ └──────────────────────────────────────────────────────┘ ``` ### Node Mutation: Weight Counts for All 19 Layer Types When `neatReinitLayer` changes a layer's type, it creates a fresh `WeightStore` with the correct number of weights for the new type: | New Layer Type | Formula | Example (dModel=32) | |:---------------|:--------|:--------------------| | Dense | `dModel × dModel` | 1024 | | RNN | `dModel² + dModel² + dModel` | 2080 | | LSTM | `4 × (dModel² + dModel² + dModel)` | 8320 | | SwiGLU | `dModel × (dModel×2) × 3` | 6144 | | RMSNorm | `dModel` | 32 | | LayerNorm | `dModel × 2` | 64 | | MHA | `2×dModel² + 2×dModel×kv + 2×dModel + 2×kv` | 4224 (4 heads) | | CNN1 / CNN2 | `filters × inChannels × kSize²` | 72 (8f, 1c, k3) | | CNN3 | `filters × inChannels × kSize³` | 216 (8f, 1c, k3) | | ConvTransposed1D/2D | `inChannels × filters × kSize²` | 72 | | ConvTransposed3D | `inChannels × filters × kSize³` | 216 | | Embedding | `vocabSize × dModel` | 8192 (256 vocab) | | KMeans | `numClusters × dModel` | 256 (8 clusters) | | Softmax | `0` — no WeightStore | — | | Residual | `0` — no WeightStore | — | | Parallel / Sequential | unchanged — keep existing branches | — | Parallel and Sequential are structural containers. Mutating a non-container to Parallel/Sequential would destroy branch structure, so `neatReinitLayer` leaves them untouched (just returns) when the target type is Parallel or Sequential. ### Connection Add — Remote Links `neatAddConnection` adds a **spatial skip connection** between two layers anywhere in the 3D grid: ``` Layer at (0,0,0,0) ──────────────────────────► Layer at (0,0,0,2) │ ┌─ ParallelBranches ──────────────┘ │ [IsRemoteLink=true, │ TargetZ=0, TargetY=0, │ TargetX=0, TargetL=2] ``` During `ForwardPolymorphic`, `ParallelForwardPolymorphic` follows remote links and routes activations to the target layer. Remote links are skipped during DNA extraction (`extractLayerSignature` skips `IsRemoteLink=true` branches since they have no local weights). --- ## Part 3 — NEATPopulation: Full Evolutionary Loop `NEATPopulation` manages a pool of networks across generations using fitness-based selection. ```go type NEATPopulation struct { Networks []*VolumetricNetwork Fitnesses []float64 Config NEATConfig rng *rand.Rand } ``` ### Initialization ```go pop := poly.NewNEATPopulation(seedNetwork, populationSize, cfg) ``` Creates `populationSize` networks, each a `NEATMutate` of the seed. This gives diverse starting points from day 0. ``` seedNetwork │ ├── NEATMutate (seed1) ──► Network[0] ├── NEATMutate (seed2) ──► Network[1] ├── NEATMutate (seed3) ──► Network[2] └── ... Network[N-1] ``` ### One Generation of Evolution ```go pop.Evolve(fitnessFn) ``` ``` Generation N: [net0, net1, net2, ..., netN] │ fitnessFn(net) for each │ sort descending by fitness │ ┌───────────┴───────────┐ │ │ Top 25% Bottom 75% (elites) (replaced) │ │ carry over pick 2 elites A, B unchanged SpliceDNA(A, B, blend) │ NEATMutate(child) │ new offspring │ │ └───────────┬───────────┘ │ Generation N+1 ``` **Elites**: The top `populationSize / 4` networks survive unchanged. The rest are replaced by: 1. Pick two random elites `A` and `B` 2. Produce a child via `SpliceDNA(A, B, cfg)` — inherits weights from both 3. Apply `NEATMutate(child)` — adds structural noise ### Helper Methods ```go pop.Best() // returns the highest-fitness network (index 0 after sort) pop.BestFitness() // returns the best fitness score pop.Summary(gen) // returns a one-line status string: // "Gen 5 | best=-0.0012 avg=-0.0045 worst=-0.2300 pop=16" ``` ### Fitness Function Contract The fitness function receives a network and returns `float64` — higher is better. Penalize with a large negative (e.g., `-1e9`) for architecturally incompatible networks (dimension mismatches from mutations): ```go fitnessFn := func(net *poly.VolumetricNetwork) (result float64) { defer func() { if r := recover(); r != nil { result = -1e9 // incompatible architecture } }() out, _, _ := poly.ForwardPolymorphic[float32](net, input) if out == nil || len(out.Data) == 0 { return -1e9 } // compute your task loss here mse := computeMSE(out.Data, target) return -mse // negate: lower loss = higher fitness } ``` --- ## Combined Flow: SpliceDNA + NEAT in a Population ``` ┌──────────────────────────────────────────┐ │ NEATPopulation.Evolve │ │ │ Generation N: │ [A] [B] [C] [D] ... [P] │ │ │ │ │ fitnessFn() for all │ │ sort: A=best, P=worst │ │ │ │ Elites (keep): [A] [B] [C] [D] │ │ │ │ Offspring: │ │ │ │ SpliceDNA(A, B) ──► child_AB │ │ NEATMutate(child_AB) │ │ ├── perturb weights │ │ ├── maybe swap activation │ │ ├── maybe change layer type │ │ └── maybe add/drop connection │ │ ──► mutated_AB │ │ │ │ ... repeat for all offspring slots ... │ │ │ Generation N+1:│ [A] [B] [C] [D] [mut_AB] ... [mut_XY] │ └──────────────────────────────────────────┘ ``` --- ## DNA Tracking Across Generations Because every `NEATMutate` and `SpliceDNA` call touches only a clone, you can always extract DNA from any network in the population and compare it against a reference: ```go // Track how far the best network has drifted from the initial seed seedDNA := poly.ExtractDNA(seedNetwork) for gen := 1; gen <= 50; gen++ { pop.Evolve(fitnessFn) bestDNA := poly.ExtractDNA(pop.Best()) result := poly.CompareNetworks(seedDNA, bestDNA) fmt.Printf("Gen %d | seed→best overlap=%.4f logic_shifts=%d\n", gen, result.OverallOverlap, len(result.LogicShifts)) } ``` Expected pattern: ``` Gen 1 | overlap=0.98 logic_shifts=0 (small weight nudges) Gen 5 | overlap=0.73 logic_shifts=1 (one node mutated type) Gen 20 | overlap=0.41 logic_shifts=3 (topology diverging) Gen 50 | overlap=0.12 logic_shifts=7 (heavily evolved) ``` --- ## Multi-Parent Splice Chain You can chain splices to merge three or more trained networks: ```go cfgA := poly.DefaultSpliceConfig() cfgA.FitnessA, cfgA.FitnessB = fitnessA, fitnessB cfgB := poly.DefaultSpliceConfig() cfgB.FitnessA, cfgB.FitnessB = fitnessMid, fitnessC mid := poly.SpliceDNA(netA, netB, cfgA) // A + B → mid final := poly.SpliceDNA(mid, netC, cfgB) // mid + C → final ``` ``` netA ──┐ ├── SpliceDNA ──► mid ──┐ netB ──┘ ├── SpliceDNA ──► final netC ──┘ ``` --- ## Immutability Guarantee Both `SpliceDNA` and `NEATMutate` always operate on **clones** of the input networks. The originals are never modified: ```go // Verify: run 5 aggressive mutations, original unchanged original := buildDenseMLP(32, 3) dnaOrig := poly.ExtractDNA(original) aggressiveCfg := poly.NEATConfig{ NodeMutateRate: 1.0, WeightPerturbRate: 1.0, WeightPerturbScale: 10.0, DModel: 32, Seed: 42, AllowedLayerTypes: poly.DefaultNEATConfig(32).AllowedLayerTypes, } for i := 0; i < 5; i++ { _ = poly.NEATMutate(original, aggressiveCfg) } dnaAfter := poly.ExtractDNA(original) result := poly.CompareNetworks(dnaOrig, dnaAfter) // result.OverallOverlap == 1.0 — original untouched ``` --- ## Quick Reference | Function | What it does | |:---------|:-------------| | `SpliceDNA(A, B, cfg)` | Blend weights from A and B into a child (A's architecture) | | `SpliceDNAWithReport(A, B, cfg)` | Same + diagnostic report with per-layer similarities | | `DefaultSpliceConfig()` | Returns blend mode, alpha=0.5, split=0.5 | | `NEATMutate(n, cfg)` | Returns a structurally mutated clone of n | | `DefaultNEATConfig(dModel)` | Conservative rates, all 17 mutable types allowed | | `NewNEATPopulation(seed, size, cfg)` | Create diverse initial population from seed | | `pop.Evolve(fitnessFn)` | Run one generation: evaluate → sort → elites → offspring | | `pop.Best()` | Highest-fitness network from last Evolve | | `pop.BestFitness()` | Fitness score of the top network | | `pop.Summary(gen)` | One-line status: best/avg/worst fitness | --- ## Softmax Variants Source: https://openfluke.com/docs/softmax Markdown: https://openfluke.com/docs/softmax.md # Softmax Variants `LayerSoftmax` (type 15) implements ten distinct softmax variants, controlled by the `SoftmaxType` field on `VolumetricLayer`. All variants are fully differentiable and work across all 21 DTypes. --- ## The Standard Formula All variants start from the numerically stable form: ``` logits_shifted = logits - max(logits) ← prevents overflow exp_vals[i] = exp(logits_shifted[i]) probs[i] = exp_vals[i] / sum(exp_vals) ``` This is implemented in `Softmax(logits []float32) []float32`. --- ## SoftmaxType Constants ```go const ( SoftmaxStandard SoftmaxType = 0 SoftmaxGrid SoftmaxType = 1 SoftmaxHierarchical SoftmaxType = 2 SoftmaxTemperature SoftmaxType = 3 SoftmaxGumbel SoftmaxType = 4 SoftmaxMasked SoftmaxType = 5 SoftmaxSparse SoftmaxType = 6 SoftmaxAdaptive SoftmaxType = 7 SoftmaxMixture SoftmaxType = 8 SoftmaxEntmax SoftmaxType = 9 ) ``` --- ## Variant 0: Standard ``` probs = softmax(logits) ``` The classic form. All outputs are positive and sum to 1. Smooth gradient everywhere. **When to use:** Classification heads, final output layers, any time you need a valid probability distribution. ``` Input: [2.0, 1.0, 0.1] ▼ Shifted: [1.9, 0.9, 0.0] ▼ Exps: [6.69, 2.46, 1.00] Sum = 10.15 ▼ Output: [0.66, 0.24, 0.10] ← sums to 1.0 ``` --- ## Variant 3: Temperature ``` probs = softmax(logits / temperature) ``` Temperature `T` (stored in `VolumetricLayer.Temperature`) controls sharpness. ``` ┌──────────────────────────────────────────────────────────────┐ │ temperature = 0.1 (sharp): │ │ Input: [2.0, 1.8, 0.1] → Output: ≈[0.99, 0.01, 0.00] │ │ Effect: "confident" — almost winner-takes-all │ │ │ │ temperature = 1.0 (standard): │ │ Input: [2.0, 1.8, 0.1] → Output: ≈[0.55, 0.45, 0.00] │ │ │ │ temperature = 5.0 (smooth): │ │ Input: [2.0, 1.8, 0.1] → Output: ≈[0.40, 0.38, 0.22] │ │ Effect: "uncertain" — options spread more evenly │ └──────────────────────────────────────────────────────────────┘ ``` **When to use:** Token sampling in language models (low T = greedy, high T = diverse), exploration vs. exploitation in RL. --- ## Variant 4: Gumbel ``` noise[i] = -log(-log(Uniform(0,1))) ← Gumbel noise probs = softmax(logits + noise) ``` Adds independent Gumbel noise to each logit before computing softmax. This produces stochastic samples that are biased toward higher logits but not deterministic. The Gumbel distribution is the natural noise for the `argmax` operation. **When to use:** Discrete sampling without the `argmax` non-differentiability. Training generative models with categorical outputs. Controlled exploration in MoE routing. ``` Same logits, three calls: Call 1: [0.71, 0.24, 0.05] ← high logit usually wins Call 2: [0.48, 0.40, 0.12] ← noise sometimes shifts result Call 3: [0.82, 0.14, 0.04] ``` --- ## Variant 5: Masked ``` masked_logits[i] = logits[i] if mask[i] == true = -1e9 if mask[i] == false probs = softmax(masked_logits) ``` The `mask` field is `[]bool` on `VolumetricLayer`. Positions where `mask[i] = false` get `-1e9` in the logit, making their `exp` output effectively zero. After softmax, those positions have probability 0. The backward pass respects the mask: gradients are zeroed for masked positions. **When to use:** - Causal attention (prevent attending to future tokens) - Legal-move filtering (board games, planning) - Expert routing where some experts are unavailable ``` Logits: [2.0, 1.0, 0.5, 1.5] Mask: [T, F, T, T ] After masking: [2.0, -1e9, 0.5, 1.5] After softmax: [0.63, 0.00, 0.11, 0.26] masked position → 0 ✓ ``` --- ## Variant 6: Sparse (Sparsemax) Sparsemax is an alternative to softmax that can produce **exact zeros** — true sparsity rather than just very small values. ``` Algorithm: 1. Sort logits descending: z₁ ≥ z₂ ≥ ... ≥ zₙ 2. Find k = max { k : z_k - (Σᵢ≤ₖ zᵢ - 1)/k > 0 } 3. τ = (Σᵢ≤ₖ zᵢ - 1) / k 4. output[i] = max(0, z[i] - τ) ``` Implemented in `SoftmaxSparseHelper(logits)`. ``` Logits: [3.0, 1.0, -1.0, -3.0] Standard softmax: [0.87, 0.12, 0.01, 0.00] ← all non-zero Sparsemax: [0.75, 0.25, 0.00, 0.00] ← exact zeros! ``` **When to use:** - Attention when you want the model to focus on exactly a few tokens - Interpretability (fewer non-zero attention weights to explain) - MoE routing (hard assignment to a subset of experts) --- ## Variant 9: Entmax Entmax is a family of distributions parameterized by `alpha`. It interpolates between softmax and sparsemax: - `alpha = 1.0` → standard softmax - `alpha = 2.0` → sparsemax - `alpha = 1.5` → the recommended default (used in original paper) ```go layer.EntmaxAlpha = 1.5 // set on VolumetricLayer ``` Implemented in `SoftmaxEntmaxHelper(logits, alpha)`: ```go weight := alpha - 1.0 s1 := Softmax(logits) s2 := SoftmaxSparseHelper(logits) result[i] = (1-weight)*s1[i] + weight*s2[i] // renormalize to sum to 1 ``` **When to use:** When you want controllable sparsity. Start with `alpha=1.5` and tune toward 2.0 for sparser attention. --- ## Variant 1: Grid Grid softmax applies standard softmax independently to each **row** of a 2D interpretation of the input: ``` Input flat tensor reinterpreted as [SoftmaxRows, SoftmaxCols]: Row 0: softmax([logits[0:cols]]) → row probs sum to 1 Row 1: softmax([logits[cols:2cols]]) → row probs sum to 1 ... ``` Each row is an independent probability distribution. **When to use:** - Native Mixture of Experts: each row represents one expert's output distribution - Multi-label classification where each "group" of labels is mutually exclusive - Per-head attention normalization without the full MHA overhead ``` Input (flat): [2.0, 1.0, | 0.5, 3.0, | 1.5, 1.5] Rows=3, Cols=2: Row 0: softmax([2.0, 1.0]) = [0.73, 0.27] Row 1: softmax([0.5, 3.0]) = [0.08, 0.92] Row 2: softmax([1.5, 1.5]) = [0.50, 0.50] ``` --- ## Variant 2: Hierarchical Hierarchical softmax uses `HierarchyLevels []int` to define a tree structure. The last level of `HierarchyLevels` is used as the column count, with rows computed from `n / cols`. In practice it reduces to Grid softmax with the last level defining the partition. **When to use:** Large vocabulary prediction where the vocabulary has a natural hierarchical structure (e.g., word categories → words). --- ## Variant 7: Adaptive Adaptive softmax selects the softmax type based on input statistics (currently implemented as a fallback to standard softmax, intended for future dynamic routing logic). --- ## Variant 8: Mixture Mixture softmax is a placeholder for weighted combinations of multiple softmax outputs. Currently falls back to standard softmax. --- ## Backward Pass All variants share the standard softmax Jacobian: ``` gradLogits[j] = probs[j] × (gradOutput[j] - Σᵢ gradOutput[i] × probs[i]) = probs[j] × (gradOutput[j] - dotProduct) ``` Implemented in `SoftmaxBackward(gradOutput, softmaxOutput []float32)`. For Grid and Hierarchical variants, the Jacobian is applied independently to each row. For Masked, gradients are zeroed at masked positions before computing the Jacobian. --- ## GetLogits `GetLogits[T Numeric](data []T, temp float64, dtype DType)` converts any `Tensor[T]` to `[]float32` with temperature scaling. It has specialized fast-paths for the most common types (float32, float64, int8, etc.) to avoid generic conversion overhead. --- ## Summary Table | Variant | Produces zeros | Stochastic | Key parameter | Best for | |:--------|:--------------|:-----------|:--------------|:---------| | Standard | No | No | — | General classification | | Temperature | No | No | `Temperature` | Sampling sharpness | | Gumbel | No | Yes | — | Differentiable sampling | | Masked | Yes (at mask) | No | `Mask []bool` | Causal attention | | Sparse | Yes | No | — | Hard sparse attention | | Entmax | Maybe | No | `EntmaxAlpha` | Tunable sparsity | | Grid | No | No | `SoftmaxRows/Cols` | MoE, multi-group | | Hierarchical | No | No | `HierarchyLevels` | Tree vocabularies | | Adaptive | No | No | — | (future) | | Mixture | No | No | — | (future) | --- ## Serialization, Persistence, and Loading Source: https://openfluke.com/docs/serialization Markdown: https://openfluke.com/docs/serialization.md # Serialization, Persistence, and Loading This document covers how `VolumetricNetwork` instances are saved and loaded, the bit-packed persistence format for low-bit types, the idempotency guarantee, and SafeTensors support. --- ## Two Serialization Paths `poly/` provides three complementary checkpoint systems: | File | Functions | Use case | |:-----|:---------|:---------| | `serialization.go` | `BuildNetworkFromJSON` | Architecture-only: creates a network from a spec with randomly initialized weights | | `persistence.go` | `SerializeNetwork` / `DeserializeNetwork` | Full save/load: architecture + trained weights as **JSON + Base64** (debug / transparent) | | `entity.go` | `SerializeEntity` / `DeserializeEntity`, `SaveEntity` / `LoadEntity` | Full save/load: architecture + trained weights as **binary `.entity`** (native ship lane) | For production checkpoints on device, prefer **`.entity`**. Keep JSON when you want to inspect weights and topology in a text editor. See [entity.md](entity.md) for the full format spec. For **tiny init recipes** (topology + per-layer `layer_seed`, no weight bytes), see [seed_manifests.md](seed_manifests.md). Lucy **[20]** demonstrates train → save trained seeds → reload with bit-exact trained outputs. --- ## Full Save/Load (persistence.go) ### Saving ```go jsonData, err := poly.SerializeNetwork(network) os.WriteFile("model.json", jsonData, 0644) ``` `SerializeNetwork` walks every layer and builds a `PersistenceNetworkSpec`: ```go type PersistenceNetworkSpec struct { ID string `json:"id"` Depth int `json:"depth"` Rows int `json:"rows"` Cols int `json:"cols"` LayersPerCell int `json:"layers_per_cell"` Layers []PersistenceLayerSpec `json:"layers"` } ``` Each `PersistenceLayerSpec` contains all configuration fields plus: ```go DType string `json:"dtype"` // active numerical type for this layer (e.g. "Uint8", "FP4") Weights string `json:"weights,omitempty"` // Base64-encoded **native-packed** payload for that dtype Native bool `json:"native,omitempty"` // true = weights are native-packed (current default on save) Scale float32 `json:"scale,omitempty"` // morph/quant scale used when the checkpoint was written ``` ### Native JSON per dtype (not FP32-only) `SerializeNetwork` no longer dumps a single FP32 master blob for every layer. On save it: 1. Reads each layer’s live `DType` and writes it to `PersistenceLayerSpec.DType`. 2. Calls `WeightStore.Morph(dt)` for that dtype and `encodeNativeWeights(active, dt)` — Int8 as 1 byte/weight, FP4/Int4 as nibbles, Binary as bit-packs, Float64 as LE uint64, etc. 3. Sets `Native: true` and persists `Scale` so reload uses the same quant mapping training saw. **Implication:** a **Uint8** Dense checkpoint is ~**0.8 KB** on disk for the Lucy 8×1024→512 bench; **Float64** is ~**5.4 MB** for the same topology — see the **File** column in Lucy’s training matrix (`lucy_testing_output/log.txt`). You can train, save, and reload **each of the 21 dtypes** independently; Lucy’s Dense suite reports **Save/Reload PASS** on all of them in the latest full run. Older checkpoints with `Native: false` (FP32 master only) still load via `decodeWeights`; new saves prefer native packing. ### Loading ```go jsonData, _ := os.ReadFile("model.json") network, err := poly.DeserializeNetwork(jsonData) ``` `DeserializeNetwork` reconstructs the `VolumetricNetwork`, initializes fresh `WeightStore`s, then calls `applyPersistenceLayerSpec` for each layer which: 1. Parses all config fields 2. Calls `initializeWeights(l)` to allocate the correct `WeightStore` size 3. Decodes the `Weights` string — using `decodeNativeWeights` if `Native=true`, or `decodeWeights` (FP32 master) if `Native=false` 4. If native format (`Native=true`): stores in `Versions[dtype]`, then calls `Unpack(dtype)` to reconstruct the FP32 master for training paths that still use master weights 5. Recursively applies the same process to `ParallelBranches` and `SequentialLayers` --- ## The Bit-Packing System The core serialization innovation is `encodeNativeWeights(data any, dt DType) string`. This function takes the `active` version from the `WeightStore.Versions` map and packs it into the most compact binary representation before Base64 encoding: ``` DType Packing Ratio vs FP32 ────────────────────────────────────────────────────── Float64 8 bytes/weight (LE uint64) 0.5x size reduction Float32 4 bytes/weight (LE uint32) 1x (baseline) Float16 4 bytes (stored as float32) not yet compact BFloat16 4 bytes (stored as float32) not yet compact Int8/Uint8 1 byte/weight 4x reduction Int4/FP4/Uint4 0.5 bytes (2 per byte) 8x reduction Int2/Uint2 0.25 bytes (4 per byte) 16x reduction Ternary 0.25 bytes (4 per byte) 16x reduction Binary 0.125 bytes (8 per byte) 32x reduction ``` ### 4-bit Packing Detail ```go // Pack 2 int8 weights into 1 byte using upper and lower nibbles: buf[i/2] |= (byte(v & 0x0F) << 4) // high nibble for even index buf[i/2] |= (byte(v & 0x0F)) // low nibble for odd index ``` Unpacking sign-extends the nibble: if the 4-bit value is > 7, subtract 16 to recover the signed value. ### 2-bit/Ternary Packing Detail ```go // Pack 4 values into 1 byte using 2-bit fields: shift := uint(6 - (i%4)*2) // 6, 4, 2, 0 buf[i/4] |= (val & 0x03) << shift ``` Unpacking reverses the shift and sign-extends from 2-bit. ### Binary Packing Detail ```go // Pack 8 weights into 1 byte, MSB first: if v > 0 { buf[i/8] |= (1 << uint(7-(i%8))) } ``` Unpacking reads each bit and maps `1 → +1`, `0 → -1`. --- ## Idempotency Guarantee The README states: "Serializing a reloaded model produces a byte-for-byte identical JSON to the original." This holds because: 1. `DeserializeNetwork` calls `Unpack(dtype)` which reconstructs `Master` from the packed data 2. The next `SerializeNetwork` call reads `Master`, calls `Morph(dtype)` again (if needed), and re-packs 3. Since `Morph` is deterministic (same formula, same scale), and the `Master` was faithfully reconstructed by `Unpack`, the output bytes are identical Verified across 378 permutations (18 layer types × 21 DTypes) with **0.000000% mathematical divergence**. --- ## Architecture-Only JSON (serialization.go) `BuildNetworkFromJSON` creates a network from a spec but uses **random weight initialization** (via `initializeWeights` which calls `Randomize`). This is for defining network topologies without weights. ```go type LayerSpec struct { Z, Y, X, L int Type string // "Dense", "CNN2", etc. Activation string // "ReLU", "Tanh", etc. DType string // "float32", "int8", etc. InputHeight int OutputHeight int // ... all configuration fields ParallelBranches []LayerSpec // recursive SequentialLayers []LayerSpec // recursive } ``` `ParseLayerType`, `ParseActivationType`, and `ParseDType` accept case-insensitive strings plus common aliases. --- ## SafeTensors support (import only) `safetensors.go` and `prefix_safetensor.go` implement **loading** from the HuggingFace SafeTensors format. Export via `SaveSafetensors` is F32-only. For native Loom saves (all 21 dtypes, volumetric topology, native packing), use **ENTITY** (`.entity`) — see [entity.md](entity.md). `universal_loader.go` provides auto-detection of the model format. The `Transformer[T]` type has dedicated loading support in `transformer.go` for assembling a full LLM from SafeTensors files: it maps weight tensor names (e.g., `"model.layers.0.self_attn.q_proj.weight"`) to the correct `VolumetricLayer` positions and weight sub-slices. --- ## ENTITY format (native `.entity` checkpoint) **ENTITY** — **E**very **N**umerical **T**ype **I**n **N**ative **T**opolog**Y** — is Loom’s single-file native checkpoint: **full topology + native-packed weights**, no Base64. ```go // Save err := poly.SaveEntity("model.entity", network) // Load (full brain) network, err := poly.LoadEntity("model.entity") // Load one layer’s weights only (topology still parsed) net, err := poly.DeserializeEntityLayer(data, layerIndex) ``` Wire layout (v1): magic + version + JSON header (`PersistenceNetworkSpec` + blob index) + raw payload. Same bit-packing as `encodeNativeWeights`, exposed as `EncodeNativeWeightsRaw` / `DecodeNativeWeightsRaw` in `persistence.go`. | vs JSON | vs SafeTensors | |:--------|:---------------| | **~25–28% smaller** on Lucy [7] full run (546 rows); up to **~42%** when weight blobs are tiny vs topology (e.g. Residual 3×3×3) — see [entity.md](entity.md#size-vs-json--observed-compression-lucy-7) | ENTITY carries grid, branches, per-layer dtype/scale; SafeTensors does not | | Same semantics; human-readable if you decode the header JSON | SafeTensors is the HF import lane; ENTITY is the native export lane | Lucy **[7]** validates JSON and `.entity` save/reload in parallel for all 21 dtypes. Lucy **[8]** ENTITY Talk imports HF LLMs into the same format for native chat. Full spec and the HF→native / 3D unlock narrative: [entity.md](entity.md) (*The unlock*, *LLM transformer checkpoints*). --- ## Compression Ratios in Practice From the README, for a network with 1M weights: ``` ┌──────────────────────────────────────────────────────────────┐ │ DType RAM (uncompressed) JSON size Ratio │ ├──────────────────────────────────────────────────────────────┤ │ Float32 4.0 MB ~5.5 MB 1.38x (base64) │ │ Int8 1.0 MB ~1.4 MB 0.34x vs FP32 │ │ Int4 0.5 MB ~0.7 MB 0.17x │ │ Binary 0.125 MB ~0.18 MB 0.045x ← 98.4% │ └──────────────────────────────────────────────────────────────┘ ``` Base64 encoding adds ~33% overhead over the raw binary size. The 98.4% figure is relative to FP32 on disk (including the base64 overhead). **ENTITY (`.entity`)** stores the same native-packed bytes **without** Base64. In Lucy [7] (`seven_layer.txt`), checkpoints are on average **~28% smaller** than JSON — mostly from dropping Base64 on weights, while topology remains JSON in the header. See [entity.md](entity.md#size-vs-json--observed-compression-lucy-7). --- ## Weight Encoding Flow ``` Training produces Master []float32 │ ▼ (if layer.DType != DTypeFloat32) Morph(layer.DType) │ ▼ Versions[dtype] = []int8 / []uint8 / etc. │ ▼ encodeNativeWeights / EncodeNativeWeightsRaw │ ┌──────┴──────────────────┐ │ │ ▼ ▼ JSON path ENTITY path bit-pack → Base64 bit-pack → raw bytes in blob section │ │ ▼ ▼ PersistenceLayerSpec EntityWeightBlob index + payload .Weights = "…" (see entity.md) .Native = true .Scale = ws.Scale ``` --- ## Deserialization and Unpack Flow ``` JSON string │ ▼ json.Unmarshal PersistenceNetworkSpec │ ▼ applyPersistenceLayerSpec For each layer: 1. ParseLayerType / ParseActivationType / ParseDType 2. initializeWeights → fresh WeightStore allocated 3. if ls.Native: decodeNativeWeights → Versions[dtype] = packed slices ws.Unpack(dtype) → Master reconstructed else: decodeWeights → Master loaded directly 4. Recurse for ParallelBranches, SequentialLayers ``` After `DeserializeNetwork`, every layer's `WeightStore.Master` is a valid FP32 weight array ready for forward inference or further training. --- ## Parallel and Sequential Layers Source: https://openfluke.com/docs/parallel-sequential Markdown: https://openfluke.com/docs/parallel-sequential.md # Parallel and Sequential Layers This document explains `LayerParallel` and `LayerSequential` in depth: how they fan out and chain sub-layers, the five combination modes, the recursive activation tree, and how backpropagation flows through nested structures. --- ## LayerParallel `ParallelForwardPolymorphic` fans the input to every branch simultaneously and then combines the results. ### Configuration ```go layer.Type = poly.LayerParallel layer.CombineMode = "concat" // or "add", "avg", "filter", "grid_scatter" layer.ParallelBranches = []poly.VolumetricLayer{ {Type: poly.LayerDense, InputHeight: 64, OutputHeight: 32, ...}, {Type: poly.LayerRNN, InputHeight: 64, OutputHeight: 32, ...}, {Type: poly.LayerCNN1, InputHeight: 64, ...}, } ``` Each entry in `ParallelBranches` is a full `VolumetricLayer` — it can itself be a `LayerParallel` or `LayerSequential`, enabling unlimited nesting. ### Combination Modes #### "add" Element-wise sum of all branch outputs. All branches must produce the same output shape. ``` Input ──▶ Branch 0 ──▶ [32] Input ──▶ Branch 1 ──▶ [32] → [32] (sum of all) Input ──▶ Branch 2 ──▶ [32] ``` Use for: residual-style ensembles, multi-path feature accumulation. #### "avg" Element-wise average of all branch outputs. Same shape requirement as "add". ``` Output[i] = (Branch0[i] + Branch1[i] + ... + BranchN[i]) / N ``` Use for: soft ensemble averaging where no single branch should dominate. #### "concat" / "grid_scatter" Concatenates all branch outputs into one flat tensor. Branch output sizes can differ. ``` Input ──▶ Branch 0 ──▶ [32] Input ──▶ Branch 1 ──▶ [16] → [32, 16, 64] = [112] Input ──▶ Branch 2 ──▶ [64] ``` `"grid_scatter"` behaves identically to `"concat"` in the current implementation — they share the same code path. The name signals intent: scatter the input across a grid of experts, then collect all outputs. Use for: multi-scale feature extraction, heterogeneous expert outputs before a routing layer. #### "filter" (Soft Mixture of Experts) Uses a separate gate sub-layer to produce per-branch weights, then computes a weighted sum: ```go layer.FilterGateConfig = &poly.VolumetricLayer{ Type: poly.LayerDense, InputHeight: 64, OutputHeight: 3, // one scalar per branch Activation: poly.ActivationLinear, } ``` At forward time: ``` Input ──▶ FilterGateConfig ──▶ [numBranches] │ Softmax(gate_logits) │ [w0, w1, w2] ← learned routing weights Input ──▶ Branch 0 ──▶ [32] × w0 Input ──▶ Branch 1 ──▶ [32] × w1 → [32] (weighted sum) Input ──▶ Branch 2 ──▶ [32] × w2 ``` Use for: differentiable Mixture of Experts (MoE), learned feature gating, adaptive multi-scale fusion. --- ## The Activation Tree (Tensor.Nested) The key to making arbitrary nesting differentiable is the `Nested []*Tensor[T]` field on `Tensor`. During `ParallelForwardPolymorphic`, each branch produces its own `(bPre, bOut)` pair. The branch `preAct` tensors are collected into a slice and stored as `Nested` on the returned `preAct`: ```go preAct = &Tensor[T]{ Data: input.Data, // proxy — carries input shape Shape: input.Shape, DType: input.DType, Nested: branchPreActs, // [branch0.preAct, branch1.preAct, ...] } ``` During `ParallelBackwardPolymorphic`, the backward function reads `preAct.Nested[i]` to get the correct cached state for each branch: ```go var bPre *Tensor[T] if preAct != nil && i < len(preAct.Nested) { bPre = preAct.Nested[i] } gIn, gW := DispatchLayerBackward(target, scaledGrad, input, nil, bPre) ``` This creates a recursive tree of activation caches that mirrors the nesting depth of the network: ``` preAct.Nested: ├── Branch 0 preAct │ └── (if branch 0 is also Parallel) │ └── .Nested │ ├── Sub-branch 0 preAct │ └── Sub-branch 1 preAct ├── Branch 1 preAct └── Branch 2 preAct ``` The backward pass recursively walks this tree, ensuring each sub-layer gets the exact cached pre-activation it needs to compute its gradient. --- ## Gradient Flow Through Parallel For "add" and "avg" modes, the same `gradOutput` (or a scaled version) is sent to every branch: ``` gradOutput │ ├──── scaledGrad ──▶ Branch 0 backward ──▶ gradInput_0 + gradWeights_0 ├──── scaledGrad ──▶ Branch 1 backward ──▶ gradInput_1 + gradWeights_1 └──── scaledGrad ──▶ Branch 2 backward ──▶ gradInput_2 + gradWeights_2 gradInput = gradInput_0 + gradInput_1 + gradInput_2 (accumulated) ``` For "avg" mode, `scaledGrad = gradOutput / N` before dispatching. For "concat" mode, the gradient is **sliced** by branch output size: ``` gradOutput [112]: branch 0 slice: gradOutput[0:32] → Branch 0 backward branch 1 slice: gradOutput[32:48] → Branch 1 backward branch 2 slice: gradOutput[48:112] → Branch 2 backward ``` For "concat" backward, the branch output size is determined by running a forward pass to measure `len(out.Data)`. This is a known overhead — for large models, consider caching branch output sizes. The `gradWeights` returned by `ParallelBackwardPolymorphic` is a synthetic tensor with no `Data` — only `Nested`: ```go gradWeights = &Tensor[T]{ Nested: branchGradWeights, // per-branch weight gradients } ``` `ApplyRecursiveGradients` recognizes this pattern and dispatches weight updates to each branch recursively. --- ## LayerSequential `SequentialForwardPolymorphic` chains sub-layers in order, each receiving the output of the previous one. ```go layer.Type = poly.LayerSequential layer.SequentialLayers = []poly.VolumetricLayer{ {Type: poly.LayerDense, InputHeight: 128, OutputHeight: 256, ...}, {Type: poly.LayerRMSNorm, InputHeight: 256, ...}, {Type: poly.LayerDense, InputHeight: 256, OutputHeight: 64, ...}, } ``` This is how transformer blocks are typically assembled: `RMSNorm → MHA → RMSNorm → SwiGLU`. ### Step Containers For each sub-layer, the forward pass stores a "step container" — a tensor whose `Nested` holds `[bPre, bInput, bSkip]`: ```go stepContainer := &Tensor[T]{ Nested: []*Tensor[T]{ bPre, // Nested[0]: preAct from this sub-layer current, // Nested[1]: the input this sub-layer received lastInput, // Nested[2]: the previous input (for skip connections) }, } stepIntermediates[i] = stepContainer ``` The outer `preAct` returned by `SequentialForwardPolymorphic` carries all step containers in its `Nested`: ```go preAct = &Tensor[T]{ Data: input.Data, Nested: stepIntermediates, // [step0container, step1container, step2container] } ``` ### Sequential Backward The backward pass iterates sub-layers in **reverse** order: ```go for i := len(layer.SequentialLayers) - 1; i >= 0; i-- { container := preAct.Nested[i] bPre = container.Nested[0] bInput = container.Nested[1] bSkip = container.Nested[2] stepGradOutput = currentGrad if skipGradients[i+1] != nil { stepGradOutput.Add(skipGradients[i+1]) // add skip gradient } gIn, gW = DispatchLayerBackward(target, stepGradOutput, bInput, bSkip, bPre) currentGrad = gIn } ``` `skipGradients` is a slice that accumulates gradients flowing back through skip connections inside the sequence. If a sub-layer (like `LayerResidual`) produces a gradient flowing back to an earlier step, it is accumulated here. --- ## Remote Links Inside Branches Both `ParallelForwardPolymorphic` and `SequentialForwardPolymorphic` support `IsRemoteLink` on individual branches: ```go if branch.IsRemoteLink && layer.Network != nil { if remote := layer.Network.GetLayer(branch.TargetZ, branch.TargetY, branch.TargetX, branch.TargetL); remote != nil { target = remote } } ``` This allows a branch to redirect to any layer in the parent `VolumetricNetwork`, enabling cross-cell feature reuse without duplicating layer definitions. --- ## Tiling Propagation When `layer.UseTiling = true` on the parent Sequential layer, the flag is propagated to each sub-layer before dispatch: ```go if layer.UseTiling { target.UseTiling = true target.TileSize = layer.TileSize } ``` This means you can set tiling on the top-level Sequential layer and all its sub-layers inherit `UseTiling` and `TileSize` automatically. **`EnableMultiCoreTiling` is not propagated here** — it lives on `VolumetricNetwork` (and may be copied onto layers for training). **GPU** SC vs MC is chosen from **`Network.EnableMultiCoreTiling`** plus `GPUSCTileSizes` / `GPUMCTileSizes` after `RefreshRuntimeTileSizes()`. **CPU** sub-layers use **`GetCPUTileSize`** only (one map per layer, not SC/MC pair); see [dispatch.md](dispatch.md). --- ## Practical Example: Transformer Block as Sequential ```go block := poly.VolumetricLayer{ Type: poly.LayerSequential, SequentialLayers: []poly.VolumetricLayer{ { Type: poly.LayerRMSNorm, InputHeight: 512, OutputHeight: 512, }, { Type: poly.LayerMultiHeadAttention, DModel: 512, NumHeads: 8, NumKVHeads: 8, HeadDim: 64, MaxSeqLen: 2048, }, { Type: poly.LayerRMSNorm, InputHeight: 512, OutputHeight: 512, }, { Type: poly.LayerSwiGLU, InputHeight: 512, OutputHeight: 1364, // ~2.67× hidden size }, }, } ``` The entire block is a single `VolumetricLayer` entry in the grid. It runs as a mini-pipeline with the `preAct.Nested` tree tracking all four sub-layer states for backpropagation. --- ## Quantization: DType Conversion and PTQ Pipeline Source: https://openfluke.com/docs/quantization Markdown: https://openfluke.com/docs/quantization.md # Quantization: DType Conversion and PTQ Pipeline This document covers the Post-Training Quantization (PTQ) pipeline in `poly/`: how weights move from FP32 masters into lower-precision formats, the `WeightStore` versioning system, the `Q4_0Block` block-quantization format, and how `MorphToFloat32ForGPU` simulates low-bit arithmetic for GPU upload. --- ## Why Quantization? Running a 7B-parameter model at FP32 requires ~28 GB of RAM. Quantization trades a small amount of numerical fidelity for dramatic memory and compute savings: ``` ┌──────────────────────────────────────────────────────────────────┐ │ DType Bits/weight 1B params Theoretical speedup │ ├──────────────────────────────────────────────────────────────────┤ │ Float64 64 8 GB 0.5× (slower than FP32) │ │ Float32 32 4 GB 1× baseline │ │ BFloat16 16 2 GB 2× │ │ Int8 8 1 GB 4× │ │ Int4/FP4 4 0.5 GB 8× │ │ Int2 2 0.25 GB 16× │ │ Binary 1 0.125 GB 32× │ └──────────────────────────────────────────────────────────────────┘ ``` `poly/` supports all 21 DTypes in the same training and inference loop. Switching precision is a single function call — no retraining required. --- ## Three training/inference modes Do not conflate **storage dtype**, **inference PTQ**, and **native exact training**. Loom implements three related but distinct ideas: ``` ┌─────────────────────────────────────────────────────────────────────────┐ │ Mode When Weight math │ ├─────────────────────────────────────────────────────────────────────────┤ │ 1. Default train UseExactDType = false FP32 surrogate (QAT-like)│ │ GetActive → dequant matmul Master += lr × grad (FP32) │ │ Versions cleared each step Morph re-quants forward │ ├─────────────────────────────────────────────────────────────────────────┤ │ 2. PTQ inference After FP32 train; Morph() Quantized storage only │ │ MorphToFloat32ForGPU No training in quant │ ├─────────────────────────────────────────────────────────────────────────┤ │ 3. Native exact UseExactDType = true Storage-dtype MAC / int8 │ │ *_native.go paths ApplyGradientsNative or │ │ Lucy menu [14] in-place int8 SGD in bwd │ └─────────────────────────────────────────────────────────────────────────┘ ``` ### 1. Default training (QAT-like) The default layer stack (`dense.go`, `rnn.go`, `swiglu.go`, …) treats `layer.DType` as the **active storage view**, but performs forward/backward in **float**: ```go weights := layer.WeightStore.GetActive(layer.DType) // Versions[dtype] or Morph wData := CastWeights[float32](weights) // dequant / widen for matmul ``` Gradients are FP32. `ApplyGradients` updates `Master` only, then clears `Versions`. This is **surrogate training**: the model learns in FP32; low-bit weights are regenerated each forward pass. Industry term: **QAT-adjacent** — not full fake-quant + STE, but the same practical goal (train float, deploy quant). ### 2. PTQ (post-training quantization) Train at FP32, then `MorphLayer(net, DTypeInt8)` for deployment. `MorphToFloat32ForGPU` applies quantize→dequantize at GPU upload so inference sees rounding error without new shaders. **No gradients flow through the quantizer** in this mode — it is inference-time precision reduction only. ### 3. Native exact training ```go net.UseExactDType = true ``` Routes to `*_native.go`. Two sub-paths: | Sub-path | DTypes | Forward | Update | |----------|--------|---------|--------| | **Native MAC** | Int32, Int64, FP8, Float16, … | Per-dot rules via `GetNative` | `ApplyGradientsNative` | | **True native** | Int8, Int4, Ternary, Uint8, … | int8 MAC, int32 accum, `>>8` | `applyStochasticInt8Update` in backward | True native int8 is **not** QAT: weights and activations stay in integer form for the whole step. Lucy **[14]** (`native_menu.go`) benchmarks this path per layer × 21 dtypes. Full training-loop detail: [training.md — Training paradigms](training.md#training-paradigms-default-qat-like-vs-native-exact). --- ## The WeightStore: Three-Layer Storage Every `VolumetricLayer` holds a `*WeightStore`: ```go type WeightStore struct { Master []float32 // Source of truth — always FP32 Versions map[DType]any // CPU-resident quantized versions GPUWeights map[DType]any // VRAM-resident wgpu.Buffer versions GPUScales map[DType]*wgpu.Buffer // Per-dtype scale buffers on VRAM Scale float32 // Quantization scale factor } ``` ### Layer 1: Master `Master` is the FP32 weight array that training operates on. Gradient updates always modify `Master`. No other layer is ever trained directly. ### Layer 2: Versions `Versions` is a cache of quantized representations derived from `Master`. Each key is a `DType`. The value type varies: ``` DType Value type in Versions ─────────────────────────────────────── Float64 []float64 Float16/BFloat16 []float32 (simulated — stored as float32 but treated as 16-bit) Int32/Int16/Int8 []int32 / []int16 / []int8 Int4/FP4/Binary []int8 (unpacked — one value per element; bit-packing is for disk only) ``` ### Layer 3: GPUWeights / GPUScales `GPUWeights` holds `wgpu.Buffer` references to VRAM. They are populated via `layer.SyncToGPU()` and consumed by the GPU forward/backward shaders. `GPUScales` holds the quantization scale as a separate GPU buffer used by quantized shader kernels. --- ## Morph: Producing a Quantized Version ```go func (ws *WeightStore) Morph(dtype DType) ``` `Morph` converts `ws.Master` to the target `dtype` and stores the result in `ws.Versions[dtype]`. It is idempotent — if the target version already exists, it returns immediately. ``` ws.Master ([]float32) │ ├── dtype == Float32 → return immediately (Master is already FP32) │ ├── dtype == Float64 → []float64: direct cast │ ├── dtype == Float16/BFloat16 → []float32: round-trip quantize/dequantize per element │ ├── dtype == Int8/Uint8/FP8* → []int8: v / ws.Scale, clamped to [-128, 127] │ ├── dtype == Int16/Uint16 → []int16: v / ws.Scale │ ├── dtype == Int32/Uint32 → []int32: v / ws.Scale │ └── dtype == Int4/FP4/Int2/Ternary/Binary → []int8 (one per weight): Int4/FP4/Int2: v / ws.Scale, truncated to range Ternary: round to {-1, 0, +1} Binary: +1 if v > 0, else -1 ``` > [!NOTE] > Sub-byte types (Int4, Int2, Binary) are stored in `Versions` as unpacked `[]int8` with one element per weight. The bit-packing into nibbles and pairs happens only during serialization (`encodeNativeWeights`). This keeps the forward pass simple — no runtime unpacking overhead during inference. ### Clearing Versions After Training When `ApplyGradients` runs, it updates `Master` and then clears `Versions`: ```go ws.Versions = make(map[DType]any) ``` This ensures stale quantized copies are not used after a weight update. The next forward pass calls `Morph` again to regenerate the needed version. This lazy invalidation pattern means training overhead is minimal — quantized versions are only regenerated on the first forward pass of each new batch. --- ## Unpack: Reconstructing Master from a Quantized Version ```go func (ws *WeightStore) Unpack(dtype DType) ``` `Unpack` is the inverse of `Morph`. It reads `ws.Versions[dtype]` and reconstructs `ws.Master`. This is used after deserialization — the JSON stores the quantized version, and `Unpack` brings `Master` back to FP32 so the network is ready for inference or further training. ``` ws.Versions[dtype] │ ├── []float64 → cast to float32 ├── []float32 → copy directly (Float16/BFloat16 simulation) ├── []int8 → v * ws.Scale (for Int8, FP8, Int4, Int2, etc.) ├── []int16 → v * ws.Scale └── []int32 → v * ws.Scale ``` --- ## MorphToFloat32ForGPU: PTQ Simulation for GPU Upload ```go func (ws *WeightStore) MorphToFloat32ForGPU(dtype DType) []float32 ``` For layers that don't have a dedicated packed GPU path (CNN1-3, RNN, LSTM, Embedding), this function produces a float32 buffer that represents the master weights after a quantize → dequantize round-trip at the target dtype. The GPU shader reads `array` and sees weights already "damaged" by quantization — inference-accurate without needing new shaders. ``` ┌──────────────────────────────────────────────────────────────────────┐ │ How MorphToFloat32ForGPU works for Int8 (scale = 0.01): │ │ │ │ Input: v = 0.437 │ │ Step 1: Morph to Int8 → q = round(0.437 / 0.01) = 44 │ │ Step 2: clamp → q = clamp(44, -128, 127) = 44 │ │ Step 3: dequantize → result = 44 * 0.01 = 0.44 │ │ │ │ The rounding error is: |0.437 - 0.44| = 0.003 │ │ This error is what Int8 quantization "costs" │ └──────────────────────────────────────────────────────────────────────┘ ``` Training always operates on the FP32 `Master` — `MorphToFloat32ForGPU` is only called at GPU upload time (`SyncToGPU`). This is **PTQ inference simulation**, not training in quant. > [!NOTE] > **Default CPU training** (`UseExactDType = false`) also trains in FP32 master space and re-quants via `Morph` each forward — QAT-like surrogate training. **Native exact training** (`UseExactDType = true`, `*_native.go`) is a separate paradigm where MAC and true-int paths update storage dtype directly. See [Three training/inference modes](#three-traininginference-modes). --- ## Scale Calibration `ws.Scale` is the per-layer quantization scale. It is computed during `Morph` using the **absolute-maximum** calibration strategy: ``` scale = max(|weight|) / maxQuantValue For Int8: maxQuantValue = 127 For Int4: maxQuantValue = 7 For Int2: maxQuantValue = 1 For Int1: maxQuantValue = 1 (binary: +1/-1) ``` This is the simplest calibration method — no calibration data required. It is a Post-Training Quantization (PTQ) approach: train at FP32, then call `MorphLayer` to convert to the target dtype. The scale is derived analytically from the weight distribution alone. > [!TIP] > For activation-aware quantization (computing scale from representative inputs rather than from weights alone), you would need to run a calibration forward pass and inject the computed scale into `ws.Scale` before calling `Morph`. The current pipeline does not implement observer-based calibration for activations — only weight calibration. --- ## MorphLayer: Network-Wide Conversion ```go func MorphLayer(n *VolumetricNetwork, dtype DType) ``` `MorphLayer` iterates all layers in the network and calls `ws.Morph(dtype)` on each. This is the primary entry point for converting a trained FP32 network to a lower-precision format: ```go // Train at FP32 poly.Train(network, trainingData, config) // Convert to Int8 for deployment poly.MorphLayer(network, poly.DTypeInt8) // The network is now ready for Int8 inference // All new forward passes will use Versions[DTypeInt8] ``` For layers that already have a version for the target `dtype`, `Morph` skips them. To force a re-quantization (e.g., after manual scale adjustment), clear the version first: ```go delete(layer.WeightStore.Versions, poly.DTypeInt8) layer.WeightStore.Morph(poly.DTypeInt8) ``` --- ## Q4_0Block: Block Quantization In addition to the global-scale quantization in `WeightStore.Morph`, `poly/` implements the **Q4_0 block format** used by llama.cpp and GGUF: ```go type Q4_0Block struct { Scale float32 // one float32 scale per block Weights [16]byte // 32 nibbles (4-bit signed values) } // Total: 4 + 16 = 20 bytes per block // Bandwidth: 20 bytes / 32 weights = 0.625 bytes/weight ``` ### QuantizeQ4_0 ```go func QuantizeQ4_0(weights []float32) []Q4_0Block ``` Converts a flat FP32 slice into Q4_0 blocks: ``` For each block of 32 weights: 1. Find maxAbs = max(|weights[i]|) in the block 2. scale = maxAbs / 7.0 ← 4-bit signed range is [-8, 7] 3. For each weight pair (w1, w2): q1 = round(w1 / scale), clamped to [-8, 7] q2 = round(w2 / scale), clamped to [-8, 7] byte[j] = (q1 & 0xF) | ((q2 & 0xF) << 4) ← pack 2 values per byte ``` The per-block scale means every 32 weights have their own scale factor, which is significantly more accurate than a single global scale for the entire layer. This is why Q4_0 retains much higher fidelity than naive Int4. ### DequantizeQ4_0 ```go func DequantizeQ4_0(blocks []Q4_0Block, n int) []float32 ``` Unpacks nibbles and applies the per-block scale: ``` For each block: For each byte b: q1 = (b & 0xF) → sign-extend: if q1 > 7, q1 -= 16 q2 = (b >> 4) → sign-extend: if q2 > 7, q2 -= 16 res[idx1] = float32(q1) * block.Scale res[idx2] = float32(q2) * block.Scale ``` ### Q4_0 vs Global Int4 ``` ┌───────────────────────────────────────────────────────────────────┐ │ Comparison for a Dense layer with 4096×4096 weights │ │ │ │ Format Scale count Bytes Notes │ │───────────────────────────────────────────────────────────────── │ │ FP32 1 (implicit) 67.1 MB No quantization │ │ Global Int4 1 8.4 MB One scale for all │ │ Q4_0 blocks 524288 8.6 MB One scale per 32 wts │ │ (2% overhead, 10× fidelity) │ └───────────────────────────────────────────────────────────────────┘ ``` Q4_0 is the preferred format for loading HuggingFace/GGUF checkpoints. The `universal_loader.go` and `safetensors.go` paths use `QuantizeQ4_0` internally when importing Q4_0 tensors. --- ## The Full PTQ Workflow ``` ┌──────────────────────────────────────────────────────────────────────┐ │ 1. Train at FP32 │ │ │ │ poly.Train[float32](network, data, config) │ │ → Master updated each batch │ │ → Versions map is cleared after each update │ │ │ │ 2. (Optional) Calibrate scale │ │ │ │ For each layer: │ │ maxAbs := findMaxAbs(layer.WeightStore.Master) │ │ layer.WeightStore.Scale = maxAbs / targetRange │ │ │ │ 3. Morph to target dtype │ │ │ │ poly.MorphLayer(network, poly.DTypeInt4) │ │ → Versions[DTypeInt4] = []int8{...} created for each layer │ │ → Scale stored in WeightStore.Scale │ │ │ │ 4. Save the quantized model │ │ │ │ jsonData, _ := poly.SerializeNetwork(network) │ │ os.WriteFile("model_int4.json", jsonData, 0644) │ │ → encodeNativeWeights packs []int8 into nibbles (0.5 bytes/wt) │ │ │ │ 5. Load and run inference │ │ │ │ network, _ := poly.DeserializeNetwork(jsonData) │ │ → Unpack(DTypeInt4) reconstructs Master from nibbles │ │ → Versions[DTypeInt4] restored for fast inference │ │ → forward passes use Versions[DTypeInt4], not Master │ └──────────────────────────────────────────────────────────────────────┘ ``` --- ## Forward Pass with Quantized Weights During a forward pass, layer dispatch chooses the path: | `UseExactDType` | Forward entry | Weight source | |-----------------|-----------------|---------------| | `false` (default) | `DenseForwardPolymorphic`, … | `GetActive(dtype)` → FP32 matmul | | `true` | `DenseForwardNativeExact`, … | `GetNative` / int8 MAC (`*_native.go`) | Default path: ```go weights := layer.WeightStore.GetActive(layer.DType) if weights == nil { weights = layer.WeightStore.Master } wData := CastWeights[float32](weights) ``` `GetActive` returns `Versions[dtype]` if it exists, otherwise `nil`. If the version is missing (e.g., after a gradient update), the forward pass falls back to `Master` and `Morph` regenerates the version on the next call. This lazy re-quantization is always correct for **default** training. Native exact bypasses bulk dequant: see `dense_native.go`, `rnn_native.go`, etc. For the GPU path, `GetActive` for GPU dtypes reads from `GPUWeights[dtype]` via the shader's bind group. The CPU never sees these weights once they are on VRAM. --- ## Accuracy vs. Compression Trade-offs From empirical benchmarks in the README: ``` ┌─────────────────────────────────────────────────────────────────┐ │ DType Similarity to FP32 (cosine) Size factor │ ├─────────────────────────────────────────────────────────────────┤ │ Float64 1.000 2.0× larger │ │ BFloat16 0.999+ 0.5× │ │ Int8 0.998+ 0.25× │ │ Int4/FP4 0.99+ 0.125× │ │ Int2 0.97+ 0.0625× │ │ Ternary 0.96+ 0.0625× │ │ Binary 0.90+ 0.03125× │ └─────────────────────────────────────────────────────────────────┘ ``` The similarity scores are measured with `poly.CompareNetworks` (see `dna.md`) — comparing the cosine angle between normalized weight vectors after precision simulation. A score of 0.999 means the quantized layer points in essentially the same direction as the FP32 layer, meaning functional behavior is preserved. > [!NOTE] > Binary (1-bit) networks at 0.90 cosine similarity will show measurable accuracy degradation on complex tasks. Binary quantization is best suited for embedding layers, lookup tables, or architectures specifically designed for 1-bit operation (e.g., BitNet). For most tasks, Int8 or Int4 provides the best accuracy/compression balance. --- ## Transformer Architecture: MHA, RoPE, GQA, and Full Block Assembly Source: https://openfluke.com/docs/transformer Markdown: https://openfluke.com/docs/transformer.md # Transformer Architecture: MHA, RoPE, GQA, and Full Block Assembly This document covers `LayerMultiHeadAttention` (MHA), how RoPE positional encoding is applied, Grouped-Query Attention (GQA) and Multi-Query Attention (MQA), the KV cache, SwiGLU and RMSNorm layers, full transformer block assembly inside `VolumetricNetwork`, and the `Transformer[T]` high-level generation type. It also documents the Qwen-style attention path now supported in Loom: - expanded query dimension (`QueryDim`) where `num_heads * head_dim != d_model` - per-head Q/K RMSNorm (`q_norm` / `k_norm`) - config-driven RMSNorm epsilon (`rms_norm_eps`) parity across CPU and GPU. --- ## LayerMultiHeadAttention `LayerMultiHeadAttention` (type index 16) implements scaled dot-product attention with optional RoPE, optional GQA/MQA, and an incremental KV cache. ### Key Fields on VolumetricLayer ```go layer.Type = poly.LayerMultiHeadAttention layer.DModel = 512 // model dimension (embedding size) layer.NumHeads = 8 // query heads layer.NumKVHeads = 8 // key/value heads (set < NumHeads for GQA/MQA) layer.HeadDim = 64 // dimensions per head (DModel / NumHeads) layer.QueryDim = 512 // optional; defaults to DModel when unset layer.MaxSeqLen = 2048 // maximum sequence length (KV cache size) layer.RoPEFreqBase = 10000.0 // RoPE theta; 0 = no positional encoding layer.RMSNormEps = 1e-6 // used by RMSNorm layers ``` For Qwen-style checkpoints, `head_dim` may be explicitly specified in config and `QueryDim` should be set to: `QueryDim = NumHeads * HeadDim`. ### Weight Layout All four projection matrices and their bias vectors are stored contiguously in `WeightStore.Master`: ``` Offset 0 queryDim × dModel Q weight matrix Offset queryDim×dModel kvDim × dModel K weight matrix Offset queryDim×dModel + kvDim×dModel V weight matrix Offset queryDim×dModel + 2×kvDim×dModel dModel × queryDim O weight matrix After all weight matrices: + queryDim bytes Q bias vector + kvDim bytes K bias vector + kvDim bytes V bias vector + dModel bytes O bias vector Total: queryDim×dModel + 2×kvDim×dModel + dModel×queryDim + queryDim + 2×kvDim + dModel ``` Where `kvDim = NumKVHeads × HeadDim`. For standard MHA (`NumKVHeads == NumHeads`): ``` kvDim = dModel Total = 4 × dModel² + 4 × dModel weights (including biases) ``` --- ## Forward Pass: Step by Step ### 1. Linear Projections Input shape: `[seqLen, dModel]` ``` For each token position s: Q[s, i] = bias_Q[i] + Σⱼ input[s, j] × W_Q[i, j] K[s, i] = bias_K[i] + Σⱼ input[s, j] × W_K[i, j] V[s, i] = bias_V[i] + Σⱼ input[s, j] × W_V[i, j] Q shape: [seqLen, queryDim] (numHeads × headDim) K shape: [seqLen, kvDim] (numKVHeads × headDim) V shape: [seqLen, kvDim] ``` ### 1.5 Q/K Norm (Qwen-style) If `model.layers.N.self_attn.q_norm.weight` and `k_norm.weight` are present, Loom applies per-head RMSNorm to projected Q and K before RoPE/attention scoring. This path is active in both CPU and GPU forward implementations. ### 2. RoPE: Rotary Positional Encoding If `layer.RoPEFreqBase > 0`, RoPE is applied to Q and K after projection. RoPE encodes position by rotating adjacent pairs of values in the head dimension: ``` For each token at position pos, head h, dimension pair (d, d + headDim/2): freq = 1 / (RoPEFreqBase ^ (2d / headDim)) angle = freq × pos cos_a, sin_a = cos(angle), sin(angle) Q[pos, h×headDim + d] = Q0 × cos_a - Q1 × sin_a Q[pos, h×headDim + d + headDim/2] = Q0 × sin_a + Q1 × cos_a (same for K, using the KV head index) ``` RoPE gives the attention mechanism a way to learn relative positions without adding learned positional embeddings. Positions encode directly into the dot-product scores. ``` ┌──────────────────────────────────────────────────────────────────┐ │ RoPE effect on attention scores │ │ │ │ Token at pos 0: angle = 0 → cos=1, sin=0 → no rotation │ │ Token at pos 1: angle = freq → slight rotation │ │ Token at pos N: angle = N×freq → large rotation for low d │ │ │ │ Relative distance (pos_q - pos_k) is captured in the dot │ │ product because cos(angle_q - angle_k) = cos(Δangle). │ └──────────────────────────────────────────────────────────────────┘ ``` ### 3. KV Cache (Float32 Path Only) The Float32 forward path maintains an incremental KV cache: ```go // Lazy initialization on first forward call if layer.KVCacheK == nil { layer.KVCacheK = NewTensor[float32](MaxSeqLen, kvDim) layer.KVCacheV = NewTensor[float32](MaxSeqLen, kvDim) layer.KVOffset = 0 } // Write current position into the ring buffer pos := layer.KVOffset + s kRow := KVCacheK.Data[(pos % MaxSeqLen) * kvDim : ...] // compute K for this token and write into kRow layer.KVOffset += seqLen // advance after full sequence ``` The cache is a ring buffer of size `MaxSeqLen`. On each call, new K and V values are written at positions `[KVOffset, KVOffset + seqLen)`. The attention score computation then looks back over all `currentTotalPos + 1` cached positions, giving the model memory of the full context up to `MaxSeqLen` tokens. To clear the KV cache between independent prompts: ```go transformer.Reset() // sets KVOffset = 0 for all layers ``` ### 4. Grouped-Query Attention (GQA / MQA) GQA reduces memory bandwidth by sharing KV heads across multiple query heads: ``` headsPerKV = NumHeads / NumKVHeads For query head h: kvHead = h / headsPerKV ← all query heads in a group share one KV head ``` ``` ┌──────────────────────────────────────────────────────────────────────┐ │ Standard MHA: NumHeads = NumKVHeads = 8 │ │ Each head has its own K and V. │ │ │ │ Q0──K0/V0 Q1──K1/V1 Q2──K2/V2 ... Q7──K7/V7 │ │ │ │ GQA: NumHeads = 8, NumKVHeads = 2 │ │ 4 query heads share each KV head. │ │ │ │ Q0, Q1, Q2, Q3 ──K0/V0 │ │ Q4, Q5, Q6, Q7 ──K1/V1 │ │ │ │ MQA: NumHeads = 8, NumKVHeads = 1 │ │ All query heads share one KV head. │ │ │ │ Q0...Q7 ─────────K0/V0 │ └──────────────────────────────────────────────────────────────────────┘ ``` GQA is the default in modern LLMs like Llama 3 because it reduces KV cache memory by `NumHeads / NumKVHeads`× without measurable quality loss. ### 5. Causal Attention Causality is enforced by the score computation loop: ```go // For query at position qPos, only attend to positions <= qPos for kPos := 0; kPos <= qPos; kPos++ { dot = Q[qPos] · K[kPos] scores[kPos] = dot / sqrt(headDim) } // positions > qPos are never included — no explicit mask needed ``` This is equivalent to a causal mask but avoids allocating a mask tensor. ### 6. Output Projection After attention-weighted value aggregation, the output is projected back to `dModel`: ``` O[s, i] = bias_O[i] + Σⱼ attnOut[s, j] × W_O[i, j] ``` --- ## MHA, tiling flags, and where work actually happens On the **CPU polymorphic** path, `MHAForwardPolymorphic` uses the tiled entry when `layer.UseTiling && layer.TileSize > 0`, which calls `mhaForwardTiledGeneric`. That helper temporarily clears `UseTiling` and re-invokes the same reference attention implementation so dispatch does not recurse forever — so this is **not** a second numeric algorithm and does not spawn goroutines per head. Exported names `MHAForwardTiled` and `MHAForwardTiledParallel` are aliases of that same entry. **Throughput-oriented tiling** (workgroup sizes, tiled matmul in shaders) lives on the **WebGPU** path in `wgpu_forward.go`: tile sizes come from `GetGPUSCTileSize` / `GetGPUMCTileSize` depending on **`VolumetricNetwork.EnableMultiCoreTiling`** — **`false` → SC**, **`true` → MC** (transformer forwards read the **network** field, not per-layer). `WGPUContext.GPUTileSize` and device limits feed `refreshRuntimeGPUTileSizes`. Call `RefreshRuntimeTileSizes()` after wiring the net: **`CPUTileSizes`** for CPU reference math (one map per layer), **`GPUSCTileSizes` / `GPUMCTileSizes`** for GPU. Training does this via `ConfigureNetworkForMode` (see `training.md`). **CPU polymorphic code does not use SC/MC as two maps** — only `GetCPUTileSize`. `CalculateOptimalTileSize(headDim)` is still the head-dimension–based helper used when populating CPU tile sizes for MHA during `refreshRuntimeCPUTileSizes`. --- ## RMSNorm `LayerRMSNorm` (type 8) implements Root Mean Square Layer Normalization: ``` rms = sqrt( (1/n) × Σᵢ xᵢ² + ε ) output[i] = (x[i] / rms) × weight[i] ``` Unlike LayerNorm, RMSNorm does not subtract the mean. This makes it faster (fewer operations) while preserving the same stabilizing effect on gradient flow. Key fields: ```go layer.Type = poly.LayerRMSNorm layer.InputHeight = 512 // must match OutputHeight layer.OutputHeight = 512 layer.RMSNormEps = 1e-6 // default; overridable from checkpoint config ``` Weight storage: one scale weight per hidden dimension (`len(Master) == OutputHeight`). --- ## SwiGLU `LayerSwiGLU` (type 12) implements the gated linear unit variant used in modern transformers: ``` Given input x of shape [seqLen, inputHeight]: gate = x × W_gate (shape [seqLen, outputHeight]) up = x × W_up (shape [seqLen, outputHeight]) hidden = SiLU(gate) × up output = hidden × W_down (shape [seqLen, inputHeight]) SiLU(x) = x × sigmoid(x) = x / (1 + exp(-x)) ``` ``` ┌────────────────────────────────────────────────────────────────────┐ │ SwiGLU Data Flow │ │ │ │ Input [seqLen, 512] │ │ │ │ │ ├──▶ W_gate [512, 1364] ──▶ gate [seqLen, 1364] │ │ │ │ │ │ └──▶ W_up [512, 1364] ──▶ up [seqLen, 1364] │ │ │ │ │ SiLU(gate) × up │ │ │ │ │ W_down [1364, 512] │ │ │ │ │ Output [seqLen, 512] │ └────────────────────────────────────────────────────────────────────┘ ``` The hidden dimension (~2.67× the model dimension) is the intermediate expansion factor. For `dModel=512`, the typical hidden size is 1364. Key fields: ```go layer.Type = poly.LayerSwiGLU layer.InputHeight = 512 layer.OutputHeight = 1364 // hidden dimension (intermediate expansion) ``` Weight storage: `W_gate` (inputHeight × outputHeight) + `W_up` (inputHeight × outputHeight) + `W_down` (outputHeight × inputHeight), stored contiguously in `Master`. --- ## Full Transformer Block Assembly A standard decoder-only transformer block (pre-norm style) is assembled as a `LayerSequential` containing four sub-layers: ```go block := poly.VolumetricLayer{ Type: poly.LayerSequential, SequentialLayers: []poly.VolumetricLayer{ // Sub-layer 0: Attention norm { Type: poly.LayerRMSNorm, InputHeight: 512, OutputHeight: 512, }, // Sub-layer 1: Multi-head attention { Type: poly.LayerMultiHeadAttention, DModel: 512, NumHeads: 8, NumKVHeads: 8, HeadDim: 64, MaxSeqLen: 2048, RoPEFreqBase: 10000.0, }, // Sub-layer 2: FFN norm { Type: poly.LayerRMSNorm, InputHeight: 512, OutputHeight: 512, }, // Sub-layer 3: Feed-forward (SwiGLU) { Type: poly.LayerSwiGLU, InputHeight: 512, OutputHeight: 1364, }, }, } ``` This entire block is a single `VolumetricLayer` entry in the 3D grid. Multiple blocks are placed at coordinates `(0, blockIdx, 0, 0)` in a `VolumetricNetwork`. ### Residual Connections Residual connections are handled by `LayerResidual` (type 14). In the sequential backward pass, residuals produce skip gradients that are accumulated via `skipGradients` (see `parallel_sequential.md`). For transformer blocks, the typical pattern using `LayerSequential` with `LayerResidual` as a sub-layer: ```go block := poly.VolumetricLayer{ Type: poly.LayerSequential, SequentialLayers: []poly.VolumetricLayer{ {Type: poly.LayerRMSNorm, ...}, {Type: poly.LayerMultiHeadAttention, ...}, {Type: poly.LayerResidual, ...}, // adds input to output {Type: poly.LayerRMSNorm, ...}, {Type: poly.LayerSwiGLU, ...}, {Type: poly.LayerResidual, ...}, // adds pre-FFN to FFN output }, } ``` --- ## The Transformer[T] Type `Transformer[T]` is a high-level wrapper around `VolumetricNetwork` for autoregressive language model inference. It holds the components that live outside the main layer grid: ```go type Transformer[T Numeric] struct { Network *VolumetricNetwork Embeddings []float32 // token embedding table: [vocabSize × hiddenSize] LMHead []float32 // output projection: [hiddenSize × vocabSize] FinalNorm []float32 // final RMSNorm weights (one per hidden dim) HiddenSize int VocabSize int Template Template // prompt formatting (chat template) } ``` ### NewTransformer ```go func NewTransformer[T Numeric]( network *VolumetricNetwork, embeddings []float32, lmHead []float32, finalNorm []float32, template Template, ) *Transformer[T] ``` Creates the wrapper and infers `HiddenSize` from the first network layer's `DModel` or `InputHeight`. `VocabSize` is inferred as `len(Embeddings) / HiddenSize`. If `finalNorm` is non-nil, a synthetic `VolumetricLayer` of type `LayerRMSNorm` is created internally to hold the final normalization weights. This layer is not part of the main grid — it runs separately after the last transformer block. ### Tied Weights Detection When `LMHead` and `Embeddings` point to the same backing array (common in weight-tied models), `SyncToGPU` detects this and reuses the same GPU buffer for both: ```go if &t.LMHead[0] == &t.Embeddings[0] { t.Network.GPULMHead = t.Network.GPUEmbeddings // no second upload } ``` ### GPU weight upload policies Two global-upload policies coexist in [`transformer.go`](../poly/transformer.go): | Method | CPU release during upload | Typical use | |:-------|:--------------------------|:------------| | `SyncGlobalWeightsToGPUSequential()` | Yes — after each of embeddings, LM head, final norm | Lucy inference, SoulGlitch `LoomCreateLLM` | | `SyncToGPU()` | No — uploads all globals, CPU slices remain until explicit `ReleaseInferenceHostWeights()` | Training, legacy demos, `LoomSyncToGPU` bulk path | Granular helpers (used by the sequential path): ```go tr.SyncEmbeddingsToGPU(); tr.ReleaseEmbeddingsHost() tr.SyncLMHeadToGPU(); tr.ReleaseLMHeadHost() tr.SyncFinalNormToGPU(); tr.ReleaseFinalNormHost() ``` Full inference load sequence (block-wise decoder + sequential globals) is documented in [memory_history.md](memory_history.md). `ReleaseInferenceHostWeights()` on the transformer clears any remaining host slices (all layers + globals) once GPU buffers are ready; after this call CPU fallback in `Generate` is disabled for that transformer. --- ### Tiling ```go func (t *Transformer[T]) EnableTiling(tileSize int) ``` Sets `UseTiling` (and `TileSize` when `tileSize > 0`) on every layer in the grid plus the standalone final norm layer. It does **not** by itself rebuild per-dtype maps — after loading or constructing the network, call `t.Network.RefreshRuntimeTileSizes()` if you need `CPUTileSizes` / GPU SC–MC maps populated before inference or training (training entrypoints usually do this for you). ### Generate ```go func (t *Transformer[T]) Generate( encode func(text string) []uint32, decode func(tokens []uint32) string, turns []Turn, systemPrompt, userMsg string, opts GenOptions, ) string ``` Full autoregressive text generation pipeline: ``` ┌──────────────────────────────────────────────────────────────────────┐ │ GENERATE FLOW │ │ │ │ 1. Template.BuildPrompt(turns, systemPrompt, userMsg) │ │ → apply chat template (e.g., <|im_start|>user\n...) │ │ │ │ 2. encode(prompt) → inputIDs []uint32 │ │ │ │ 3. Reset() → clear KV cache │ │ │ │ 4. Prefill (process all input tokens at once): │ │ a. tokensToTensor(inputIDs) → embed all tokens │ │ b. ForwardPolymorphic or ForwardTokenIDsWGPU (GPU) │ │ c. applyLMHead(lastHiddenState) → logits over vocabulary │ │ │ │ 5. Decode loop (one token at a time): │ │ a. applyRepetitionPenalty(logits, generatedTokens) │ │ b. SampleTopK(logits, TopK, Temperature, Deterministic) │ │ c. stream.Push(tokens) → streaming decode callback │ │ d. Forward single new token (incremental): │ │ getEmbedding(nextToken) → forwardOne(input) │ │ (KVOffset advances by 1 each step) │ │ e. check EOS condition or max tokens │ │ │ │ 6. Return accumulated decoded string │ └──────────────────────────────────────────────────────────────────────┘ ``` ### GenOptions ```go type GenOptions struct { MaxTokens int Temperature float64 TopK int Deterministic bool UseKVCache bool EOSTokens []int } ``` `Deterministic = true` with `Temperature = 0` produces greedy decoding. `TopK` limits sampling to the top K logits before applying temperature. --- ## GPU Transformer Inference When `network.UseGPU = true` and weights have been synced to VRAM (via block-wise layer upload + `SyncGlobalWeightsToGPUSequential()` for inference, or bulk `SyncToGPU()` for legacy paths), `Generate` uses `ForwardTokenIDsWGPU` for both prefill and incremental decode: ```go logitTensor, err := t.ForwardTokenIDsWGPU(tokens, nil, true, true) ``` This dispatches into `wgpu_forward.go`'s GPU transformer block execution path, which runs matrix multiplications and attention as WebGPU compute shader invocations. All intermediate activations stay on VRAM; only the final logit tensor is read back to CPU for sampling. The GPU path uses the `BeginFrame` / `FlushFrame` pattern (see `gpu.md`) — one GPU command buffer encodes the entire forward pass across all transformer layers, then flushes in a single submit. This minimizes CPU–GPU synchronization overhead. --- ## C-ABI Integration (welvet) Loom v0.75.0 exposes highly optimized C-ABI entry points for the `Transformer` wrapper, enabling maximum throughput for language bindings like Python and TypeScript. ### 1. LoomTokensToTensor A high-speed gather kernel that converts token IDs directly into a pre-allocated model input tensor. - **WASM/Go**: Uses direct memory access to avoid intermediate allocations. - **WebGPU**: Dispatches a gather compute shader to perform embedding lookup entirely on VRAM. ### 2. LoomForwardFull The authoritative entry point for auto-regressive generation. It encapsulates: - `Reset()` (optional clearing of KV cache) - `TokensToTensor` (Input ID processing) - `ForwardPolymorphic` (Engine execution) - `ApplyLMHead` (Output projection) This unified path reduces the number of cross-language calls (e.g., Python → Go) by **75%**, significantly lowering the latency for real-time streaming tokens. --- ## Loading from SafeTensors / HuggingFace `universal_loader.go` auto-detects the checkpoint format. For HuggingFace models: 1. `safetensors.go` reads the weight tensor map (key → `[]float32`) 2. `prefix_safetensor.go` strips model-specific prefix patterns (e.g., `model.layers.0.self_attn.q_proj.weight`) 3. Weight slices are copied into the correct `VolumetricLayer.WeightStore.Master` at the computed offsets The key-to-layer mapping follows the weight layout described earlier: ``` model.layers.{N}.self_attn.q_proj.weight → layer N's Q weight sub-slice model.layers.{N}.self_attn.k_proj.weight → layer N's K weight sub-slice ... ``` After loading, call `poly.MorphLayer(network, targetDtype)` to convert to your desired inference precision. --- ## Practical: Building a 7-layer Transformer Network ```go hiddenSize := 512 numHeads := 8 numLayers := 7 seqLen := 2048 network := poly.NewVolumetricNetwork("llm-7l", 1, numLayers, 1, 1) for i := 0; i < numLayers; i++ { l := network.GetLayer(0, i, 0, 0) l.Type = poly.LayerSequential l.SequentialLayers = []poly.VolumetricLayer{ {Type: poly.LayerRMSNorm, InputHeight: hiddenSize, OutputHeight: hiddenSize}, { Type: poly.LayerMultiHeadAttention, DModel: hiddenSize, NumHeads: numHeads, NumKVHeads: 2, // GQA: 4 query heads share each KV head HeadDim: hiddenSize / numHeads, MaxSeqLen: seqLen, RoPEFreqBase: 10000.0, }, {Type: poly.LayerRMSNorm, InputHeight: hiddenSize, OutputHeight: hiddenSize}, {Type: poly.LayerSwiGLU, InputHeight: hiddenSize, OutputHeight: hiddenSize * 8 / 3}, } poly.InitializeLayerWeights(l) } transformer := poly.NewTransformer[float32]( network, embeddings, lmHead, finalNormWeights, chatTemplate, ) transformer.EnableTiling(0) // auto-detect tile size ``` --- ## Native checkpoints: HF → `.entity` (Lucy [8]) Poly Talk loads HF safetensors every run. **ENTITY Talk** converts once to a Lucy-owned `.entity` file and chats from that checkpoint at runtime (tokenizer still from HF snapshot). | Path | Entry | |:-----|:------| | Convert + chat UI | Lucy menu **[8]** — Lucy `hf_entity.go` (see [lucy.md](lucy.md)) | | Import API | `poly.ImportHFToEntity` — [`hf_import.go`](../poly/hf_import.go) | | Save/load | `SaveEntityTransformer` / `LoadEntityTransformer` — [`entity.go`](../poly/entity.go) | | Q4 bake on disk | [`entity_q4.go`](../poly/entity_q4.go) — decoder MHA/SwiGLU only in v1 | Full format semantics, the **HF → native citizen** unlock (graft, parallel, remote links, 3D grid), and Q4 scope tables: **[entity.md](entity.md)** — sections *The unlock* and *LLM transformer checkpoints*. --- ## Quick Reference: Common Code Snippets Source: https://openfluke.com/docs/quick-reference Markdown: https://openfluke.com/docs/quick-reference.md # Quick Reference: Common Code Snippets Concise, copy-paste-ready patterns for the most common `poly/` tasks. Each snippet assumes `import poly "github.com/openfluke/soul/loom/poly"` (adjust to your module path). --- ## 📦 TypeScript / Node.js / Flutter Installation ```bash npm install @openfluke/welvet ``` ```yaml # Flutter — pubspec.yaml dependencies: welvet: ^0.80.4 ``` See [deployment.md](deployment.md) for npm/WASM · [flutter.md](flutter.md) for Flutter FFI and examples. --- ## Flutter / Dart (`welvet`) ```dart import 'dart:convert'; import 'package:welvet/loom_ffi.dart'; final handle = loomLib.createNetwork(jsonEncode({ 'id': 'demo', 'depth': 1, 'rows': 1, 'cols': 1, 'layers_per_cell': 1, 'layers': [{ 'z': 0, 'y': 0, 'x': 0, 'l': 0, 'type': 'dense', 'dtype': 'float32', 'input_height': 16, 'output_height': 8, 'activation': 'relu', }], })); loomLib.configureTrainingMode(handle, 2); final out = loomParseFloatArray( loomLib.forwardPolymorphic(handle, input, [1, 16]), ); loomLib.freeNetwork(handle); ``` See [flutter.md](flutter.md) for training, mesh step, DNA, checkpoints, and the loom-flutter-quickstart showcase app. --- ## Creating a Network ```go // NewVolumetricNetwork(id, depth, rows, cols, layersPerCell) network := poly.NewVolumetricNetwork("my-net", 1, 3, 1, 1) // 1×3×1 grid = 3 layers stacked in the Y dimension ``` --- ## Adding and Configuring Layers ```go // Retrieve a layer by 4D coordinate (z, y, x, layerIndex) l := network.GetLayer(0, 0, 0, 0) // Dense layer l.Type = poly.LayerDense l.InputHeight = 128 l.OutputHeight = 64 l.Activation = poly.ActivationReLU l.DType = poly.DTypeFloat32 // Initialize weights (random) poly.InitializeLayerWeights(l) ``` --- ## Forward Pass ```go input := poly.NewTensor[float32](128) // flat [128] input copy(input.Data, myInputData) output, inputs, preActs := poly.ForwardPolymorphic[float32](network, input) // output = final layer's output tensor // inputs = cached inputs for each layer (needed for backward) // preActs = cached pre-activations for each layer ``` --- ## Backward Pass ```go // Compute loss gradient (e.g., MSE gradient) target := poly.NewTensor[float32](64) copy(target.Data, myTargetData) gradOutput := poly.ComputeLossGradient[float32](output, target, poly.LossMSE) // Backpropagate gradInput, layerGrads := poly.BackwardPolymorphic[float32](network, gradOutput, inputs, preActs) ``` --- ## Applying Gradients ```go lr := float32(0.001) poly.ApplyRecursiveGradients[float32](network, layerGrads, lr) ``` --- ## Full Training Loop (Manual) ```go for epoch := 0; epoch < 100; epoch++ { output, inputs, preActs := poly.ForwardPolymorphic[float32](network, input) loss := poly.CalculateLoss[float32](output, target, poly.LossMSE) gradOutput := poly.ComputeLossGradient[float32](output, target, poly.LossMSE) _, layerGrads := poly.BackwardPolymorphic[float32](network, gradOutput, inputs, preActs) poly.ApplyRecursiveGradients[float32](network, layerGrads, 0.001) fmt.Printf("epoch %d loss=%.4f\n", epoch, loss) } ``` --- ## Batch Training (High-Level) ```go config := poly.TrainingConfig{ LearningRate: 0.001, Epochs: 50, BatchSize: 32, LossFunction: poly.LossMSE, UseGPU: false, } result := poly.Train[float32](network, trainingData, config) fmt.Printf("final loss: %.4f\n", result.FinalLoss) ``` --- ## Type-Switching with Generics ```go // Run forward pass with any numeric type func runForward[T poly.Numeric](net *poly.VolumetricNetwork, data []T) *poly.Tensor[T] { input := poly.NewTensor[T](len(data)) copy(input.Data, data) out, _, _ := poly.ForwardPolymorphic[T](net, input) return out } // Call with float32 out32 := runForward[float32](network, myFloat32Data) // Call with int8 out8 := runForward[int8](network, myInt8Data) ``` --- ## Quantizing a Trained Network ```go // Convert all layers to Int8 poly.MorphLayer(network, poly.DTypeInt8) // Convert to Int4 (4-bit) poly.MorphLayer(network, poly.DTypeInt4) // Revert: clear versions and retrain or re-morph for i := range network.Layers { network.Layers[i].WeightStore.Versions = make(map[poly.DType]any) } poly.MorphLayer(network, poly.DTypeBFloat16) ``` --- ## Saving and Loading (Full Weights) ```go // Save jsonData, err := poly.SerializeNetwork(network) if err != nil { log.Fatal(err) } os.WriteFile("model.json", jsonData, 0644) // Load jsonData, _ := os.ReadFile("model.json") network, err := poly.DeserializeNetwork(jsonData) if err != nil { log.Fatal(err) } ``` --- ## Architecture-Only JSON (Random Weights) ```go spec := `{ "id": "my-net", "depth": 1, "rows": 2, "cols": 1, "layers_per_cell": 1, "layers": [ {"z":0,"y":0,"x":0,"l":0,"type":"Dense","activation":"ReLU", "dtype":"float32","input_height":128,"output_height":64}, {"z":0,"y":1,"x":0,"l":0,"type":"Dense","activation":"Linear", "dtype":"float32","input_height":64,"output_height":10} ] }` network, err := poly.BuildNetworkFromJSON([]byte(spec)) ``` --- ## Parallel Branches ```go l.Type = poly.LayerParallel l.CombineMode = "concat" l.ParallelBranches = []poly.VolumetricLayer{ {Type: poly.LayerDense, InputHeight: 64, OutputHeight: 32, Activation: poly.ActivationReLU, DType: poly.DTypeFloat32}, {Type: poly.LayerRNN, InputHeight: 64, OutputHeight: 32, Activation: poly.ActivationTanh, DType: poly.DTypeFloat32}, } ``` --- ## Sequential Sub-Layers ```go l.Type = poly.LayerSequential l.SequentialLayers = []poly.VolumetricLayer{ {Type: poly.LayerRMSNorm, InputHeight: 256, OutputHeight: 256}, {Type: poly.LayerDense, InputHeight: 256, OutputHeight: 256, Activation: poly.ActivationGELU, DType: poly.DTypeFloat32}, } ``` --- ## Soft Mixture of Experts ```go l.Type = poly.LayerParallel l.CombineMode = "filter" l.FilterGateConfig = &poly.VolumetricLayer{ Type: poly.LayerDense, InputHeight: 64, OutputHeight: 3, // one weight per expert Activation: poly.ActivationLinear, } l.ParallelBranches = []poly.VolumetricLayer{ {Type: poly.LayerDense, InputHeight: 64, OutputHeight: 32, ...}, {Type: poly.LayerDense, InputHeight: 64, OutputHeight: 32, ...}, {Type: poly.LayerDense, InputHeight: 64, OutputHeight: 32, ...}, } ``` --- ## Remote Link (Spatial Hop) ```go // Layer at (0,1,0,0) reads output from (0,0,0,0) instead of its immediate predecessor l := network.GetLayer(0, 1, 0, 0) l.IsRemoteLink = true l.TargetZ, l.TargetY, l.TargetX, l.TargetL = 0, 0, 0, 0 ``` --- ## Step mesh (continuous) Operation ```go state := poly.NewStepState[float32](network) state.SetInput(inputTensor) for tick := 0; tick < 1000; tick++ { poly.StepForward(network, state, false) // false = no history // read current output from state.LayerData[lastLayerIdx] } // Online learning (no history required) poly.StepApplyTween(network, state, targetTensor, 0.001) ``` --- ## Step mesh with BPTT (training) ```go state := poly.NewStepState[float32](network) state.SetInput(inputTensor) for tick := 0; tick < numSteps; tick++ { poly.StepForward(network, state, true) // true = capture history } gradIn, layerGrads, err := poly.StepBackward(network, state, gradOutput) poly.ApplyRecursiveGradients[float32](network, layerGrads, lr) ``` --- ## DNA Comparison ```go // Snapshot before training dna1 := poly.ExtractDNA(network) // Train ... poly.Train[float32](network, data, config) // Snapshot after training dna2 := poly.ExtractDNA(network) result := poly.CompareNetworks(dna1, dna2) fmt.Printf("Similarity: %.4f\n", result.OverallOverlap) for _, shift := range result.LogicShifts { fmt.Printf("Logic migrated: %s → %s (%.3f)\n", shift.SourcePos, shift.TargetPos, shift.Overlap) } ``` --- ## GPU Initialization ```go network.UseGPU = true ctx, err := poly.InitWGPU() if err != nil { log.Fatal("GPU init failed:", err) } network.GPUContext = ctx // Training / bulk sync (keeps CPU weights until explicit release): for i := range network.Layers { network.Layers[i].SyncToGPU() } ``` ### LLM inference load (lower peak RAM) For transformers, prefer block-wise upload + sequential globals — see [memory_history.md](memory_history.md): ```go for li := 0; li < numLayers; li++ { base := li * 4 for j := 0; j < 4; j++ { layer := &tr.Network.Layers[base+j] _ = layer.SyncToGPU() layer.ReleaseInferenceHostWeights() } } _ = tr.SyncGlobalWeightsToGPUSequential() tr.ReleaseInferenceHostWeights() ``` Optional diagnostics: ```go os.Setenv("LOOM_MEMORY_HISTORY", "1") poly.GlobalMemoryHistory.BeginSession("gpu_load") poly.RecordFromTransformer(poly.GlobalMemoryHistory, tr, "after_block_1") _ = poly.GlobalMemoryHistory.FinishSession() // terminal chart ``` ```go // Fill per-dtype maps: CPUTileSizes (CPU) + GPUSCTileSizes / GPUMCTileSizes (GPU). // GPU inference: EnableMultiCoreTiling false → SC, true → MC (wgpu_forward reads Network.*). network.EnableMultiCoreTiling = true network.RefreshRuntimeTileSizes() config := poly.TrainingConfig{UseGPU: true, LearningRate: 0.001, Epochs: 100} result := poly.Train[float32](network, data, config) ``` --- ## Transformer Inference ```go transformer := poly.NewTransformer[float32]( network, embeddingWeights, lmHeadWeights, finalNormWeights, chatTemplate, ) transformer.EnableTiling(0) // auto tile size output := transformer.Generate( tokenizer.Encode, tokenizer.Decode, []poly.Turn{}, // no history "You are a helpful assistant.", "What is 2 + 2?", poly.GenOptions{ MaxTokens: 256, Temperature: 0.7, TopK: 40, }, ) fmt.Println(output) ``` --- ## Softmax Variants ```go // Temperature softmax l.Type = poly.LayerSoftmax l.SoftmaxType = poly.SoftmaxTemperature l.Temperature = 0.5 // Masked softmax (causal) l.SoftmaxType = poly.SoftmaxMasked l.Mask = []bool{true, true, false, false} // mask out last 2 // Sparse (exact zeros) l.SoftmaxType = poly.SoftmaxSparse // Entmax (tunable sparsity) l.SoftmaxType = poly.SoftmaxEntmax l.EntmaxAlpha = 1.5 ``` --- ## Q4_0 Block Quantization ```go // Quantize a weight slice into 32-weight blocks blocks := poly.QuantizeQ4_0(myWeights) // blocks[i].Scale = per-block float32 scale // blocks[i].Weights = [16]byte with 32 packed nibbles // Dequantize back to float32 recovered := poly.DequantizeQ4_0(blocks, len(myWeights)) ``` --- ## DType / Activation / LayerType Parsing ```go // From string (case-insensitive, aliases accepted) dtype, err := poly.ParseDType("int8") // → DTypeInt8 activation, err := poly.ParseActivationType("relu") // → ActivationReLU layerType, err := poly.ParseLayerType("Dense") // → LayerDense ``` --- ## Tween (Layer-Local Learning) Same idea as neural target propagation in the literature; we call it **tween** in code and informal docs (`tween.go`). ```go tweenConfig := poly.TweenConfig{ UseChainRule: true, // false = gap-based (for step meshes) LearningRate: 0.01, } tweenState := poly.NewTweenState[float32](network) // Forward + backward + weight update in one call poly.TweenForward[float32](network, tweenState, input) poly.TweenBackward[float32](network, tweenState, globalTarget) poly.ApplyTweenGaps[float32](network, tweenState, 0.01) ``` --- ## Tensor Creation ```go // 1D tensor t1 := poly.NewTensor[float32](128) // 2D tensor (e.g., [seqLen, hiddenSize]) t2 := poly.NewTensor[float32](16, 512) // With initial data t3 := poly.NewTensor[int8](8) for i := range t3.Data { t3.Data[i] = int8(i) } // Check shape fmt.Println(t2.Shape) // [16, 512] fmt.Println(len(t2.Data)) // 8192 ``` --- ## Testing, validation, and Lucy logs Source: https://openfluke.com/docs/testing-and-validation Markdown: https://openfluke.com/docs/testing-and-validation.md # Testing, validation, and Lucy logs This page ties together **how we stress `poly/`**, where **artifacts land**, and how to read **parity tables** in captured logs (for example `lucy_testing_output/log.txt` in the [Lucy Bloom Rivers](lucy.md) repo). --- ## Where logs come from **[Lucy Bloom Rivers](lucy.md)** (`lucy_bloom_rivers` on GitHub) drives broad layer suites: forward/backward parity, training matrices, save/reload checks, and GPU timing tables. Typical transcripts (paths relative to **Lucy repo root**): | Log | Menu | Contents | |-----|------|----------| | `lucy_testing_output/log.txt` | Dense L1 / GPU parity / layer matrices | Forward/backward parity, ASM timers, GPU tables | | `lucy_testing_output/seven_layer.txt` | **[7] Seven-layer CPU suite** | 10 layer types × 21 dtypes × 1³/2³/3³ grids, **SC/MC/SIMD** fwd+bwd+train, **JSON + `.entity` save/reload** | | `lucy_testing_output/native_layers.txt` | **[14] Native layer suite** | 10 layer types × 21 dtypes, **native-exact** fwd+bwd+train (30 epochs), SIMD speedup columns — see [native_layers.md](native_layers.md) | | `lucy_testing_output/cross_path_layers.txt` | **[15] Cross-path CPU suite** | SC/MC/SIMD vs native vs native-SIMD side-by-side — see [cross_path_layers.md](cross_path_layers.md) | | `lucy_testing_output/nine_layer.txt` | **[9] Intel NPU bridge** | 15 layers × FP32/FP16/INT8 × small/medium/large — Loom vs Intel CPU/NPU timing + drift manifest | Per-dtype checkpoints are written under the same folder: `tag_DType.json` (debug lane) and `tag_DType.entity` (native lane). The memory table compares both file sizes side by side. **Observed compression (full [7] run):** `.entity` averages **~28% smaller** than JSON across 546 dtype×suite rows; all `json=PASS entity=PASS`. Quant dtype (Int4 vs Float64) still dominates absolute size — ENTITY removes Base64 overhead, not topology JSON. Details and sample tables: [entity.md — observed compression](entity.md#size-vs-json--observed-compression-lucy-7). Both files are meant for human review and regression diffing (adapter name, per-dtype rows, summary tallies). **Seven-layer suite (v0.79+):** See [`bedrock_validation.md`](bedrock_validation.md) for what the harness gates (MHA layout, KV decode, native ternary save, C-ABI `SyncInferenceWeights`, **SC/MC/SIMD parity** on all seven compute layers). Run [Lucy](lucy.md) → **[7]** or **[0]**. External benchmark logs (amd64 / arm64, Float32 timing tables): `seven_layer_amd.txt`, `seven_layer_arm.txt` — summarized in [simd.md](simd.md#seven-layer-benchmark-results). **Native-exact suite [14]:** archived logs `native_layers_amd.txt`, `native_layers_arm.txt` — full pass matrix, SIMD speedups, and known train flakes in [native_layers.md](native_layers.md). **Cross-path SIMD duel [15] grid [5]:** archived logs `~/Documents/loom/simd/cross_path_layers_amd.txt`, `cross_path_layers_arm.txt` — QAT-SIMD vs Nat-SIMD at 3³, dtype spread tables, and platform pass matrix in [cross_path_layers.md](cross_path_layers.md#archived-simd-duel-results-jul-2026). **GPU load memory timeline (Lucy [1] / [8]):** Enable *Measure memory during GPU load* at the prompt, or set `LOOM_MEMORY_HISTORY=1`. After load, Lucy prints a braille chart, sample table, and diagnosis (block release, sequential globals, peak host+gpu overlap). See [memory_history.md](memory_history.md). **HF → `.entity` convert:** Block-wise import is visible in the terminal (one `Finished loading weights with prefixes` per decoder block). Convert is not charted yet; see [entity.md — convert memory](entity.md#hf--entity-convert-memory). --- ## How to read parity summary lines Sections often end with a line shaped like: ```text >> [Forward Parity] 84 Tests | 💎 42 | ✅ 24 | 🟨 0 | 🟠 0 | 🟤 18 | ❌ 0 | 💀 0 ``` Rough meaning (exact thresholds live in the test harness, not duplicated here): | Symbol | Typical meaning | |--------|-----------------| | **💎** | Exact / diamond-grade agreement within the tightest tolerance | | **✅** | Pass within configured industry-grade tolerance | | **🟨 / 🟠** | Elevated drift bands (still classified by the harness) | | **🟤** | Heavy drift (e.g. **H-DRIFT** in backward tables) — worth investigating dtype + path | | **❌** | Hard failure (assert or threshold breach) | | **💀** | Fatal / panic / infrastructure failure | Backward tables may label columns **INDUS** (industry tolerance) vs **H-DRIFT** (heavy drift). Treat **🟤** rows as “numerically alive but not interchangeable with FP32 reference at the same tolerance,” not necessarily as engine bugs: some combinations are expected to diverge when the reference path is float32-simulated and the subject path is true low-bit or integer-native. --- ## May 2026 full-suite snapshot (`log.txt`) Recent **Run All Layer Tests** captures (Metal / arm64, ~2992 rows) show: | Metric | Value | |--------|--------| | **Broken (❌)** | **0** | | **Fatal / NaN (💀)** | **0** | | Bit-exact (💎) | ~75% of classified rows | | Heavy drift (🟤) | ~17% — mostly forward parity vs FP32 reference on native-int / low-bit paths | **Fixes reflected in this run (vs earlier transcripts):** - **Training matrix** — `File` / `RAM` columns print correctly (no `%!s(MISSING)`); every Dense training row **TrainOK PASS** and **Save/Reload PASS** for all 21 dtypes. - **Save/Reload** — CNN1/2/3, Dense, Embedding, LSTM, MHA, Residual, RNN, SwiGLU each end with `[Save/Reload ] PASS`. - **Global manifest** — no hard failures across the full layer sweep. **Still classified as 🟤 (not ❌):** Dense forward parity rows where CPU uses true integer/low-bit math and the harness compares to a float-shaped reference; CNN backward **H-DRIFT** on Float16/BFloat16/Int4 (GPU vs CPU reference). Treat as tolerance bands — see parity legend above. --- ## Dense forward ASM (Plan 9) Lucy **Dense → Generic Layer Suite** prints **Go SC · Go MC · ASM SC · ASM MC · GPU SC · GPU MC** and speedup columns: - **Go/Asm↑** = Go wall time ÷ ASM wall time (**> 1.0** = assembly wins). - Toggle: `UseAsmForward` on the network/layer; kernels live under `poly/asm/` (see [`asm/README.md`](../poly/asm/README.md)). **Latest Dense bench (8×1024→512, Metal host, from `log.txt`):** | Highlight | Go/Asm↑ SC | Go/Asm↑ MC | |-----------|------------|------------| | Best single-core | **Uint8** ~**2.46×** | — | | Best multi-core | — | **Uint4** ~**3.55×** | | Strong quant MC | — | **Ternary** ~3.21×, **FP4** ~3.25×, **Binary** ~2.78×, **Int8** ~2.72× | | Float32 | ~1.11× SC, ~1.00× MC (parity) | | | Float64 | **< 1×** (asm slower on this shape) | ~0.61× MC | Low-bit and morphed-`uint8` paths benefit most from native integer dots in Plan 9. Float64 SC/MC still favors Go tiled matmul on the current tile sizes — tuning item, not a broken toggle. **Backward / training:** Plan 9 **asm** (`poly/asm/`) is **Dense forward-only**. Plan 9 **SIMD** (`poly/simd/`) covers **forward + backward** on Dense, SwiGLU, MHA, CNN1–3, RNN, LSTM via `TrainingModeCPUSimd` / `SetSimdForwardRecursive`; see [simd.md](simd.md). --- ## Interpreting a real log (examples) The following patterns show up in recent `log.txt` captures (Metal adapter, tiled CNN1 suite): 1. **CNN1 generic suite note** — The harness itself reminds you that generic CNN1 tests still include **simulated / PTQ fallback** where a dtype has no strict native path. For a **strict native-only** CPU/GPU/tiling audit, use the **Glitch** `layer_matrix` example (see Glitch docs / examples in-repo). 2. **Float64 on GPU forward** — CPU microseconds vs GPU milliseconds often look like a large “speedup ratio < 1×”; that is frequently **dispatch overhead dominating tiny work**, not a claim that FP64 GPU is slower than CPU math in the large-batch limit. 3. **Wide integer CNN1 backward** — **Int64 / Uint64 / Int32 / Uint32** rows may show **🟤 H-DRIFT** vs float reference in GPU backward parity: the harness compares against an FP32-shaped reference while the native path uses integer semantics — read those rows as **classification / tolerance**, not as “GPU kernel wrong.” 4. **Save/Reload after training** — On the **Dense** suite (May 2026 log), **Save/Reload PASS** for all 21 dtypes after training. Older CNN-only rows or pre-native-save builds may still show FAIL on specific combos; diff against current `persistence.go` (`Native: true` + per-layer `dtype`) before treating as open bugs. 5. **Uint CPU training** — **Uint64 / Uint32** (and sometimes **Uint16**) may show **TrainOK FAIL** on CPU-tiled modes while GPU modes **PASS**: that points at **CPU-side training / loss scaling** for unsigned paths, not at GPU correctness. 6. **Peak performance gap line** — The footer **PEAK PERFORMANCE GAP** (e.g. Dense Forward Float16) is a **headline ratio** from one worst row in the scan table; it is useful for spotting outliers, not as a single global quality score. --- ## Poly package: what the suites actually exercise High-signal files and areas (not exhaustive): | Area | Representative files | |------|------------------------| | Core types & dispatch | `poly.go`, `forward.go`, `backward.go`, `training.go` | | Numerical morphing | `weights.go`, `quantization.go`, CNN/ dense / MHA polymorphic `*.go` | | GPU / WebGPU | `wgpu_context.go`, `wgpu_forward.go`, `wgpu_kernels.go`, `wgpu_shaders.go`, `wgpu_softmax.go` | | Tiling & tile size | `tile_detection.go`, `*_tiled*.go` paths in dense / CNN / MHA | | Serialization | `serialization.go`, `persistence.go`, `safetensors.go` | | Native layer matrix harness | `native_layer_matrix.go`, `native_matrix_builtin_hooks.go` | | Telemetry | `tanhi.go`, hardware probes in `hardware.go` | When you add a layer or dtype, extend **both** the Lucy (or Glitch) harness **and** this doc if the log format or tolerance bands change. --- ## Nine-layer Intel bridge (`nine_layer.txt`) **Menu:** Lucy **[9]** → **[4]** (medium only) or **[5]** (full 90-cell matrix). **Guide:** [`accelerators.md`](accelerators.md) — architecture, dtype upload, offload policy. ### What the log exercises Each cell = one layer type × one dtype (FP32 / FP16 / INT8) × one size tier (`small` / `medium` / `large`): 1. Build Loom network with that `dtype` on the layer. 2. **`SyncToAccel`** once — compile OpenVINO graph + bake `WeightStore` weights (MatMul/Conv/MHA only). 3. **`DispatchLayer`** forward — Loom CPU baseline vs Intel CPU vs Intel NPU. 4. Drift: Loom↔Intel output diff + Intel repeat-forward determinism. ### Timing columns | Column | Meaning | |--------|---------| | **Loom CPU / Intel CPU / Intel NPU** | Median **infer** ms after compile (steady state) | | **Spd CPU / Spd NPU** | Loom ÷ Intel — **> 1** Intel faster, **< 1** Intel slower | | **Compile C / Compile N** | One-time `SyncToAccel` ms — **not** in infer column | Intel can be slower than Loom when tensor work is tiny (NPU ~0.5 ms floor) or when Loom’s CPU path is sub-millisecond (norms, small softmax). That is expected — not a broken weight upload. ### Manifest block (bottom of log) | Line | Healthy signal (Jun 2026 Fedora run) | |------|--------------------------------------| | Intel faster than Loom (CPU) | ~56/90 — wins on medium/large MAC | | Intel faster than Loom (NPU) | ~36/90 — wins mainly large MAC | | Intel infer repeat (CPU/NPU) | **90/90 💎 EXACT** — determinism OK | | Loom↔Intel parity ≤ INDUS | CPU ~23/90, NPU ~61/90 (looser NPU tolerance) | **Parity buckets:** 💎 EXACT · ✅ INDUS · 🟨 LOWBIT · 🟤 H-DRIFT · ❌ BROKE · 💀 FATAL — same legend as seven-layer tables. **Known ❌ BROKE rows:** LayerNorm / RMSNorm FP32/FP16 (~1.8 drift) — Intel uses fixed CABI graphs, not Loom `WeightStore`. **INT8 MAC** large-tier drift 3–36 — Loom dequant matmul vs OV f32 + NPU dynamic quant. ### Dtype / integration status (read with the log) - **Weight upload:** FP32, FP16, INT8 on MatMul / Conv / MHA — ✅ at init (`LayerWeightBytesForAccel`). - **Forward infer:** activation bytes only per hop — ✅. - **“Integration done”?** Forward **bridge** yes (experimental); product NPU toggle, all dtypes, norms, backward — **no**. See accelerators.md “Is the integration done?”. --- ## Related commands (developer workflow) Exact entrypoints move with refactors; prefer: - [lucy.md](lucy.md) — Lucy Bloom Rivers repo, menus, log layout; Lucy `README.md` for MRBiVS stack. - `poly/README.md` — version checklist and capability matrix. - `welvet/cabi/internal/check/` — C-ABI vs `poly/` export parity scanner (Go); expect **461/461 (100%)** after v0.79 (`LoomSyncInferenceWeights`). --- ## See also - [bedrock_validation.md](bedrock_validation.md) — v0.79.0 seven-layer suite, MHA/KV, C-ABI - [numerical_types.md](numerical_types.md) — DType list and `WeightStore` lifecycle - [gpu.md](gpu.md) — WebGPU context and dispatch overview - [serialization.md](serialization.md) — Save/load and safetensors - [training.md](training.md) — Training modes and loss paths --- ## Bedrock Validation (v0.79.0) Source: https://openfluke.com/docs/bedrock-validation Markdown: https://openfluke.com/docs/bedrock-validation.md # Bedrock Validation (v0.79.0) **Release:** **0.78.0 "ASM CPU"** → **0.79.0 "Bedrock Validation"** **Checklist:** **108 / 142** (76.1%) → **111 / 142** (78.2%) This wave does not add a new compute backend. It hardens the **Go CPU** path, **native persistence**, **transformer decode**, and **C-ABI** so Lucy and Welvet bindings can trust train → save → reload → infer on real volumetric graphs. --- ## What changed (summary) | Area | Problem | Fix | |------|---------|-----| | **MHA layout** | Flat `[B·S·D]` was parsed as one long sequence (`seq = len/D`) | `mhaParseLayout` trusts `[B,S,D]` when `Shape[2] == d_model`; legacy flat layouts still work | | **KV cache** | Training and autoregressive decode shared one policy; decode overwrote position 0 | `mhaPrepareKVForForward`: reset on full-sequence train; keep cache for `batch=1`, `seq=1`, warm KV | | **Poly Talk** | `KVOffset` ignored in forward; `+=` broken across steps | `seqBase = kvStart + b*seqLen`; correct `KVOffset` advance; layout no longer stomps `input.Shape[1]` | | **MHA backward** | Q recomputed with RoPE but skipped Q/K RMS norm vs forward | Backward matches forward norm order before RoPE | | **Dense Ternary save** | Checkpoint re-quantized from FP32 Master, not native path | `GetBitNetTernaryMatrix` → `packNativeTernaryToBitNetMatrix` (same matmul as forward) | | **Signed low-bit I/O** | Int2/Int4/Ternary round-trip gaps on `[]uint8` | `persistence.go` encode/decode aligned with CPU kernels | | **FP32 Master lifecycle** | Bindings could not mirror post-train native-only RAM | `LoomSyncInferenceWeights` in `welvet/cabi` (C-ABI parity **461/461** at v0.79) | | **Regression harness** | False PASS (zeros/NaN); suite gaps | Lucy **[7] seven-layer** CPU suite: 10 layer types × 21 dtypes × SC/MC/**SIMD** × train × save/reload | --- ## Lucy seven-layer CPU suite **Run:** [Lucy Bloom Rivers](lucy.md) → **[7]** (or **[0]** for all layer types). **Log:** `lucy_testing_output/seven_layer.txt` (reset each run). **Harness:** Lucy `examples/seven_layer/` — builds a volumetric JSON network per layer family, morphs all **21 dtypes**, checks: - Forward **SC ↔ MC ↔ SIMD** parity (dtype tolerance) - Backward **SC ↔ MC ↔ SIMD** parity (10× fwd tol) - **50-epoch** CPU training on SC, MC, and **SIMD** paths (loss decrease) - **Save/reload before train** and **after train** (forward match + native blob) - Grids **1³**, **2³**, **3³** (CNN1/2 skip 3³; CNN3 is 1³ only; Embedding at `(0,0,0)`) **Layer types:** Dense, SwiGLU, MHA, CNN1, CNN2, CNN3, RNN, LSTM, Embedding, Residual. **SIMD:** All seven compute layers use Plan 9 `DotTile` (forward) and `SaxpyF32AccF64` (backward) when `TrainingModeCPUSimd` / `SetSimdForwardRecursive(true)`. Banner: `Fwd: dot_tile .s | Bwd SIMD: saxpy/dot .s (all seven layer types)`. **ASM:** Dense forward only (`UseAsmForward` after JSON build); separate from the `poly/simd` path above. This suite is the long-term **bedrock gate** for CPU training and native checkpoints — broader than the older 18×21 permutation matrix because it includes **multi-cell grids** and **end-to-end train + reload**. **Companion suite — Lucy [14] native-exact:** Same 10 layer types × 21 dtypes, but `UseExactDType = true` (true storage-dtype MAC + in-place updates). No SC/MC parity or save/reload — focuses on native fwd/bwd/train gates and native-exact SIMD speedups. **Jul 2026:** amd64 **209/210**, arm64 **207/210** (train flakes on low-bit RNN; CNN3 Int32 on arm64). Details: [native_layers.md](native_layers.md). --- ## C-ABI (Welvet) ```bash cd welvet/cabi/internal/check && go run . ``` **v0.79:** **461/461 (100.0%)** — last gap in that release: - **`LoomSyncInferenceWeights`** — calls `VolumetricNetwork.SyncInferenceWeights()` when `ReleaseFP32MasterWhenIdle` is set (morph Master → native `Versions`, drop FP32 duplicate for inference RAM). **v0.81 (current):** **489/489 (100.0%)** — additional export families: | Area | Key exports | |------|-------------| | **Vendor accel** | `LoomDiscoverAccel`, `LoomNetworkAttachAccel`, `LoomSyncToAccel`, `LoomLayerWeightBytesForAccel`, `LoomDispatchAccelForward`, `LoomSetLayerExecTarget` | | **ENTITY file I/O** | `LoomOpenEntityFile`, `LoomLoadEntityTransformerFromFile`, `LoomLoadEntityTransformerTopology`, `LoomLoadNetworkLayerWeights`, `LoomPrepareEntityTransformerLayerIndices`, `LoomDequantizeQ4_0GPUPacked` | | **Transformer GPU** | `LoomSyncEmbeddingsToGPU`, `LoomSyncLMHeadToGPU`, `LoomSyncFinalNormToGPU` | | **Memory history** | `LoomMemoryHistoryWriteJSON` | See [`v081_release.md`](v081_release.md) and [`accelerators.md`](accelerators.md#welvet-c-abi-non-go-bindings). Python / TypeScript / WASM consumers that train outside `LoomTrain` should call `LoomSyncInferenceWeights` after morph or custom training if they mirror Go’s inference-only memory model. --- ## What this release is (and is not) **You now have:** - A **deterministic CPU VM** story that survives volumetric multi-cell layouts, not only single-stack benches. - **Transformer decode** aligned with training layout (KV + RoPE + Q/K norm). - **Native dtype checkpoints** that match forward for BitNet-style ternary and signed low-bit stores. - **Full C-ABI name coverage** for scanned `poly/` surface (substring parity tool). **You do not yet claim:** - Beating PyTorch/llama.cpp on model zoo size or raw tok/s. - Plan 9 **asm** on MHA/SwiGLU/CNN (still **Dense forward** only in `poly/asm/`). - Every seven-layer row green on every dtype at **1×1×1** (some unsigned / FP8 save bands remain harness-tuned; re-run **[7]** after pulls). **Plan 9 SIMD** (`poly/simd/`) now covers forward + backward on all seven compute layer types; see [simd.md](simd.md) for amd64/arm64 benchmark tables. **Next named target (unchanged):** **v0.81** — ASM rollout (Dense backward, SwiGLU, MHA); GPU fusion. See [`v080_release.md`](v080_release.md) for the **0.80.0** wave. --- ## Key source files | Topic | Files | |-------|--------| | MHA layout / KV | `poly/mha_layout.go`, `poly/mha.go` | | BitNet CPU / ternary | `poly/bitnet_cpu.go` | | Persistence | `poly/persistence.go`, `poly/serialization.go` | | Master / inference RAM | `poly/weight_master.go` | | Seven-layer harness | Lucy `examples/seven_layer/*.go` | | C-ABI export | `welvet/cabi/acceleration_ext.go` (`LoomSyncInferenceWeights`); v0.81: `accel_ext.go`, `entity_ext.go`, `transformer_ext.go`, `io_ext.go` | --- ## See also - [testing_and_validation.md](testing_and_validation.md) — log legend, ASM columns, `log.txt` snapshot - [transformer.md](transformer.md) — MHA, RoPE, GQA, KV cache fields - [serialization.md](serialization.md) — native packed JSON per dtype - [training.md](training.md) — `Train`, `ReleaseFP32MasterWhenIdle`, SC/MC modes - [`poly/README.md`](../poly/README.md) — checklist and version calculation --- ## v0.80.0 — Native Ship (ENTITY + Modern GPU) Source: https://openfluke.com/docs/v080-release Markdown: https://openfluke.com/docs/v080-release.md # v0.80.0 — Native Ship (ENTITY + Modern GPU) > **v0.84+:** Lucy lives in [lucy_bloom_rivers](lucy.md) (was `loom/lucy/`). Log and harness paths below are relative to the Lucy repo root. **Release:** **0.79.0 "Bedrock Validation"** → **0.80.0 "Native Ship"** **Checklist:** **111 / 142** (78.2%) → **114 / 142** (80.3%) This wave ships **native Loom checkpoints** (ENTITY), moves production GPU to **openfluke/webgpu v1.0.4** (wgpu-native **v29**), and validates real LLM inference on **Metal, Vulkan (Intel + NVIDIA), and Windows ARM64**. The **Planet Bridging** POC in [`../planetbridging/`](../planetbridging/) completes the “planets → Loom” half of the hub; it **releases as its own repo/version after Loom 0.80** — Loom must land first. --- ## What shipped ### ENTITY — native `.entity` checkpoints | Item | Detail | |------|--------| | **Format** | [`entity.md`](entity.md) — magic `ENTITY`, JSON topology header + native-packed weight blobs | | **Semantics** | Same as JSON persistence: 21 dtypes, volumetric `(Z,Y,X,L)`, parallel/sequential trees, per-layer `Scale` | | **Lucy [7]** | Seven-layer CPU suite: JSON **and** `.entity` save/reload PASS on all trained rows | | **Lucy [8]** | **ENTITY Talk** — HF cache → `ImportHFToEntity` → optional Q4 bake → GPU chat without safetensors at runtime | | **Size** | ~25% smaller than JSON checkpoints (Base64 removed); quant dtype still dominates absolute size | | **Unlock** | Real LLM weights become `.entity` citizens — same container as volumetric experiments (graft, remote links, per-layer dtype) | Import lane unchanged: HuggingFace **`.safetensors`** for download. Ship lane: **`.entity`** for trained or converted brains. ### WebGPU v29 — `github.com/openfluke/webgpu@v1.0.4` | Item | Detail | |------|--------| | **Module** | Standalone [openfluke/webgpu](https://github.com/openfluke/webgpu) (no longer a cogentcore fork) | | **Native stack** | wgpu-native **v29** C API — futures, `WGPUStringView`, Go-side validation error scopes | | **Loom dependency** | `require github.com/openfluke/webgpu v1.0.4` in root and `Lucy go.mod` | | **Binaries** | Prebuilt `libwgpu_native.a` per platform under the module; `ios/amd64` (Intel simulator) dropped to satisfy Go module size limits | See webgpu README for platform table and version history. ### Cross-platform GPU validation (Lucy Poly Talk / ENTITY Talk) Same SmolLM2-135M-Instruct, Q4, block-wise GPU upload — **webgpu v1.0.4 + poly WGSL**: | Platform | GPU | Backend | Decode (approx.) | Notes | |----------|-----|---------|------------------|-------| | macOS arm64 | Apple M5 | Metal | ✅ parity with prior v29 work | Adapter → device → buffer → forward | | Windows arm64 | Snapdragon | Vulkan | ✅ validated | Previously broken on old bindings | | Linux | Intel Iris Xe | Vulkan (Mesa i915) | ~19 tok/s decode | Headless; tier fallback OK | | Linux | RTX 3050 Mobile | Vulkan (NVIDIA) | ~69 tok/s decode, ~492 tok/s prefill | Requires healthy `nvidia-smi` + `VK_ICD_FILENAMES` | Not a llama.cpp/Ollama tok/s contest yet — custom WGSL through wgpu-native — but **~3.5× decode vs iGPU** on the same box confirms the v29 stack is production-real on NVIDIA Linux. ### Planet Bridging POC (in monorepo — separate release) [`../planetbridging/`](../planetbridging/) reached **v0.5.0** internally: - **Direction:** planets → Loom (**complete** for standard volumetric layer types) - **13 compare tabs:** Dense, CNN1/2/3, MHA, LSTM, RNN, LayerNorm, Embedding, RMSNorm, SwiGLU, Residual, Mixer v1/v2 - **Planets:** PyTorch, TensorFlow, JAX (+ sklearn on Dense) - **Mechanism:** live weight stream → `.stream.entity` → Loom infer → PASS vs native (fp32 tolerance) - **Mixer v2:** 16-layer stack, all 12 types chained (~5e-5 max diff POC) **Release order:** **Loom 0.80 first** → then **Planet Bridging 0.5.0** as its own published hub (v1.0 = Loom → ONNX/Safetensors/GGUF export). --- ## What this release is (and is not) **You now have:** - A **shippable native checkpoint** (`.entity`) beside JSON debug persistence and HF import - **HF LLMs as Loom citizens** via Lucy [8] — not just flat safetensor guests each run - **Modern GPU bindings** decoupled from upstream fork politics - **Multi-vendor GPU proof** on one engine (Metal, Qualcomm, Intel, NVIDIA Vulkan) - A **complete planet→Loom POC** waiting on Loom’s release tag **You do not yet claim:** - Planet Bridging **published** (repo/version ships after Loom) - Loom → export hub formats (ONNX/GGUF out) — Planet Bridging **v1.0** - Ollama-class decode on every GPU (WGSL matmul path still has headroom) - ENTITY v2 binary topology (header still JSON; see [entity.md — future](entity.md#future-smaller-files-with-full-topology)) **Next named targets:** - **v0.81** — ASM rollout (Dense backward, SwiGLU, MHA); GPU kernel fusion - **Planet Bridging v0.5.0** — publish after Loom 0.80 tag - **Planet Bridging v1.0** — Loom → hub formats → any inference engine --- ## How to verify ```bash # Lucy ENTITY + GPU (from repo root) cd lucy_bloom_rivers && go get github.com/openfluke/webgpu@v1.0.4 && go mod tidy go run . # [7] seven-layer (entity save/reload), [8] ENTITY Talk, [1] Poly Talk GPU # ENTITY round-trip tests cd ../poly/tests && go test -run Entity -v # Planet Bridging compare host (POC — not part of Loom release artifact yet) cd ../planetbridging && go run . ``` Linux NVIDIA: ```bash export VK_ICD_FILENAMES=/usr/share/vulkan/icd.d/nvidia_icd.x86_64.json export WGPU_ADAPTER_NAME=NVIDIA ``` --- ## Key source files | Area | Files | |------|-------| | ENTITY | `poly/entity.go`, `poly/entity_q4.go`, `poly/hf_import.go` | | Lucy [8] | Lucy `hf_entity.go` | | WebGPU init | `poly/wgpu_context_native.go` | | Docs | [`entity.md`](entity.md), [`gpu.md`](gpu.md), [`transformer.md`](transformer.md) | | Planet Bridging | `planetbridging/README.md`, `planetbridging/PROGRESS.md` | --- ## See also - [bedrock_validation.md](bedrock_validation.md) — v0.79 CPU/MHA/C-ABI wave - [entity.md](entity.md) — format spec and Lucy [7]/[8] validation - [../planetbridging/README.md](../planetbridging/README.md) — bridging POC (release after Loom) --- ## v0.81.0 — Accelerator Bridge (Intel NPU + vendor plugin model) Source: https://openfluke.com/docs/v081-release Markdown: https://openfluke.com/docs/v081-release.md # v0.81.0 — Accelerator Bridge (Intel NPU + vendor plugin model) > **v0.84+:** Lucy lives in [lucy_bloom_rivers](lucy.md) (was `loom/lucy/`). Log and harness paths below are relative to the Lucy repo root. **Release:** **0.80.0 "Native Ship"** → **0.81.0 "Accelerator Bridge"** **Checklist:** **112 / 146** (76.7%) on `adjustments` — Intel forward dispatch advances **Accelerators & Distributed** (experimental) First public **vendor accelerator** path: Loom forwards individual layers through **`poly/accel`** into chaosglue-built plugins, starting with **Intel OpenVINO CPU + NPU** on Linux. --- ## What shipped ### `poly/accel` — vendor-neutral plugin loader | Item | Detail | |------|--------| | **Package** | `poly/accel/` — `Discover`, `Registry`, `Plugin`, `CompiledLayer` | | **C ABI** | `loom_accel.h` in chaosglue (Loom does not vendor OpenVINO) | | **Linux** | `dlopen` via CGO (`CGO_ENABLED=1`) | | **Intel plugin** | `libloom_accel_intel.so` — built from `loom/accel/intel/` (single `.so`, no versioned soname) | ### Dispatch integration | Item | Detail | |------|--------| | **`accel_intel.go`** | `DiscoverAccel`, `SyncToAccel`, `DispatchAccelForward`, `LayerWeightBytesForAccel` (FP32/FP16/INT8) | | **`forward.go`** | `DispatchLayer` calls accel when `layer.ExecTarget.UseAccel()` | | **`VolumetricLayer`** | `ExecTarget`, `AccelBinding` fields | | **Init-once** | `SyncToAccel(sizeLabel)` compiles + uploads dtype-aware weights; steady infer reuses handle | ### Lucy [9] — Intel NPU bridge suite | Item | Detail | |------|--------| | **Menu** | `[9]` → `[4]` medium or `[5]` full matrix | | **Tables** | Timing (Loom / Intel CPU / Intel NPU, speedup) + seven-style drift spectrum | | **Log** | `lucy_testing_output/nine_layer.txt` | | **Proof** | 90 cells: Intel infer **💎 EXACT** repeat-forward; Conv2D large **~22×** NPU vs Loom | ### Welvet C-ABI (489/489 parity) Non-Go bindings can drive the same accel and entity-file paths without reimplementing `poly/`: | Export family | Purpose | |---------------|---------| | `LoomDiscoverAccel` / `LoomNetworkAttachAccel` / `LoomSyncToAccel` | Plugin load, attach to `VolumetricNetwork`, compile + weight bake | | `LoomDispatchAccelForward` / `LoomLayerWeightBytesForAccel` | Per-layer Intel forward + weight byte introspection | | `LoomOpenEntityFile` / `LoomLoadEntityTransformerFromFile` | Random-access `.entity` without slurping full file | | `LoomLoadNetworkLayerWeights` | Hydrate selected layer indices from an open `EntityFile` | Parity check: `cd welvet/cabi/internal/check && go run .` → **489/489**. Linux build: `cd welvet/cabi/internal/build && ./build_linux.sh` (or `./build_unix.sh linux amd64`). Output: `dist/linux_/welvet.so` + `welvet.h`. ### Documentation | File | Contents | |------|----------| | [`accelerators.md`](accelerators.md) | User/developer guide — Intel now, Qualcomm + Google planned | | chaosglue [`npu/docs/2025-06-26-loom-dispatch-integration-assessment.md`](https://github.com/openfluke/chaosglue/blob/main/npu/docs/2025-06-26-loom-dispatch-integration-assessment.md) | Full benchmark evidence | --- ## What this release is (and is not) **You now have:** - A **real dispatch hook** — not a standalone bench binary - **Intel CPU + NPU** on Linux with documented env + Lucy validation - A **plugin model** ready for **Qualcomm NPU** and **Google TPU** (same ABI, new `.so`) - **Experimental** label — appropriate for first wild release **You do not yet claim:** - End-user “turn on NPU” without code (`ExecTarget` is manual) - JSON network field for `exec: intel-npu` - Training or backward on vendor path - Bit-perfect Loom ↔ Intel parity on all layers - Windows or macOS Intel plugin builds - Qualcomm or Google plugins (roadmap only) --- ## Quick start (developers) ```bash # 1. Build Intel CABI cd accel/intel && ./install_openvino.sh && source ./setup_env.sh && ./build.sh # 2. Optional explicit plugin path export LOOM_ACCEL_INTEL_SO="$PWD/accel/intel/build/libloom_accel_intel.so" # 3. Run Lucy validation cd lucy_bloom_rivers CGO_ENABLED=1 go run . # → 9 → 4 ``` Or: `./run_npu_bridge.sh` from Lucy repo . Monolithic MLP demo: `cd accel/intel/example && CGO_ENABLED=1 go run .` --- ## Future vendors (planned) | Vendor | Plugin (planned) | SDK / hardware | |--------|------------------|----------------| | **Intel** | `libloom_accel_intel.so` | ✅ OpenVINO, Core Ultra NPU | | **Qualcomm** | `libloom_accel_qcom.so` | QNN / Hexagon, Snapdragon X | | **Google** | `libloom_accel_google.so` | TPU / PJRT (cloud + edge TBD) | Loom code path is identical: `DiscoverAccel` → `ExecTarget` → `SyncToAccel` → `ForwardPolymorphic`. --- ## Next targets (v0.82+) - **AccelPlanner** — auto-select CPU vs Intel CPU vs Intel NPU from shape + layer type - **JSON `exec` field** — `"intel-npu"` per layer in network JSON - **Parity** — MatMul bias/layout, norm weight upload, shared INT8 quant - **Qualcomm CABI** stub in chaosglue `npu/qualcomm/` - **ASM rollout** (continues from v0.80 roadmap) — Dense backward, SwiGLU, MHA --- ## Key source files | Area | Files | |------|-------| | Accel package | `poly/accel/*.go` | | Intel dispatch | `poly/accel_intel.go`, `poly/forward.go` | | Types | `poly/poly.go` (`ExecTarget`, `AccelBinding`, `net.Accel`) | | Lucy suite | `Lucy examples/nine_layer/` | | Intel plugin C++ | `accel/intel/include/loom_accel.h`, `accel/intel/src/` | | Welvet C-ABI | `welvet/cabi/accel_ext.go`, `entity_ext.go`, `transformer_ext.go` | --- ## See also - [accelerators.md](accelerators.md) — full developer guide - [v080_release.md](v080_release.md) — previous release (ENTITY + WebGPU) - [dispatch.md](dispatch.md) — `DispatchLayer` hub - [gpu.md](gpu.md) — WebGPU (complementary backend) --- ## v0.82.0 — Snapdragon Bridge (SIMD CPU + Qualcomm/Hexagon NPU) Source: https://openfluke.com/docs/v082-release Markdown: https://openfluke.com/docs/v082-release.md # v0.82.0 — Snapdragon Bridge (SIMD CPU + Qualcomm/Hexagon NPU) > **v0.84+:** Lucy lives in [lucy_bloom_rivers](lucy.md) (was `loom/lucy/`). Log and harness paths below are relative to the Lucy repo root. **Release:** **0.81.0 "Accelerator Bridge"** → **0.82.0 "Snapdragon Bridge"** **Checklist:** **117 / 147** (79.6%) on `adjustments` — a **second** vendor NPU (Qualcomm/Hexagon) plus a **SIMD CPU fast-path** advance the Numerical Core, Accelerators, and Ecosystem categories. Two headline items land on top of the v0.81 Intel bridge: 1. **SIMD CPU** — hand-written AVX2/FMA (x86-64) and NEON (ARM64) kernels behind `SetSimdForward` / `TrainingModeCPUSimd`. **v0.82 shipped forward (`DotTile`) only**; the current tree adds **backward (`SaxpyF32AccF64`) on all seven compute layers** — see [simd.md](simd.md). 2. **Qualcomm / Hexagon NPU** — the second `poly/accel` vendor plugin, running on **Windows ARM64** through the **QNN AI Engine Direct** SDK. Forward-only, per-layer, experimental — the same maturity bar as Intel. --- ## What shipped ### SIMD CPU fast-path (`poly/simd`) | Item | Detail | |------|--------| | **x86-64** | AVX2/FMA `DotTile` dot-product tiles for dense/matmul-heavy forward paths | | **ARM64** | NEON `DotTile` (`neon_arm64.go`) — `unsafe.Slice` over `float32` pointers | | **Toggle** | `SetSimdForwardRecursive` / `TrainingModeCPUSimd` — off falls back to portable Go tiled loops | | **Forward** | `DotTile` (AVX2 / NEON) | | **Backward** | `SaxpyF32AccF64` on Dense, SwiGLU, MHA, CNN1–3, RNN, LSTM (current tree; not in original v0.82 tag) | | **Parity** | Go tiled loops remain the reference; seven-layer SC/MC/SIMD suite | | **Docs / bench** | [`simd.md`](simd.md); Lucy **[7]** amd64/arm64 logs (`seven_layer_amd.txt`, `seven_layer_arm.txt`) | ### Qualcomm / Hexagon NPU plugin (`accel/qualcomm`) | Item | Detail | |------|--------| | **Plugin** | `loom_accel_qualcomm.dll` — QNN AI Engine Direct (QAIRT) behind the vendor-neutral `loom_accel.h` C ABI | | **Build** | `build_clang.ps1` (`llvm-mingw` `aarch64-w64-mingw32-clang++`); MSVC/GNU ABI notes in `accel/qualcomm/README.md` | | **Windows loader** | `poly/accel/plugin_qualcomm_windows.go` — `LoadLibraryA` / `GetProcAddress`, `PrepareQualcommRuntime` adjusts `PATH` | | **Targets** | `ExecQualcommCPU` (Kryo), `ExecQualcommNPU` (Hexagon HTP) | | **DTypes** | FP32 / FP16 / INT16 / INT8 / INT4 (see `bench_manifest.json`) | | **Install** | `install_qairt.ps1 -Persist` sets machine-wide `QNN_SDK_ROOT` / `LOOM_QUALCOMM_RUNTIME` | ### Dispatch + robustness | Item | Detail | |------|--------| | **`accel_qualcomm.go`** | `DiscoverQualcommAccel` plugin discovery; per-layer `DispatchLayer` offload | | **Unique graph names** | `loom___` (atomic counter) — fixes `QnnGraph_create` collisions across dtypes in one context | | **Context reset** | `CompiledGraph::~CompiledGraph()` → `Backend::reset_context()` — frees leaked graphs; fixes `0xc0000005` on long matrix runs | | **Quiet logging** | QNN log level clamped to ERROR by default (`LOOM_QNN_VERBOSE=1` to restore); Go-side terminal noise filter (`quiet.go`) — full output still written to `snapdragon.txt` | ### Lucy [12] — Snapdragon NPU bridge suite | Item | Detail | |------|--------| | **Menu** | `[12]` — mirrors the Intel `[9]` `nine_layer` suite | | **Tables** | Timing (Loom / Qualcomm CPU / Qualcomm NPU, speedup) + seven-style drift spectrum | | **Log** | `lucy_testing_output/snapdragon.txt` | ### webgpu on Windows ARM64 `go.mod` `replace` directive pointing at a local `openfluke/webgpu` copy works around the MSVC-vs-GNU ABI mismatch when linking `libwgpu_native.a` on Windows ARM64. Documented in `accel/qualcomm/README.md`. --- ## What this release is (and is not) **You now have:** - A **second vendor NPU** (Qualcomm/Hexagon) on Windows ARM64 through the same `poly/accel` C ABI as Intel - A **SIMD CPU fast-path** (AVX2/NEON forward + backward on seven layer types) with Go tiled loops as the parity reference - Persistent, all-user QNN environment setup and a reproducible `clang++` build - **Experimental** label — a rocky-but-real bridge, good for a release, not for prod **You do not yet claim:** - Whole-model `.entity` → NPU lowering (offload is per-layer, forward-only) - Training or backward on the Qualcomm path - Bit-perfect Loom ↔ Hexagon parity on all layers/dtypes - An NPU parity suite vs the WebGPU reference (SmolLM-class smoke) - A JSON network field for `exec: qualcomm-npu` (targets set in code) --- ## Quick start (developers, Windows ARM64) ```powershell # 1. Install the QNN runtime + persist env for all users cd accel\qualcomm ./install_qairt.ps1 -Persist # 2. Build the plugin with llvm-mingw clang++ ./build_clang.ps1 # 3. Run the Lucy Snapdragon suite cd ..\..\lucy_bloom_rivers go run . # -> 12 ``` Set `LOOM_QNN_VERBOSE=1` to see full QNN/HTP logs in the terminal (otherwise clamped to ERROR). --- ## Checklist deltas (v0.81 → v0.82) | Category | v0.81 | v0.82 | Change | |----------|:-----:|:-----:|--------| | 1. Numerical Core | 22 / 31 | 23 / 32 | +SIMD CPU forward kernels | | 3. Accelerators & Distributed | 4 / 18 | 6 / 18 | +Intel + Qualcomm per-layer NPU dispatch | | 5. Deployment Ecosystem | 25 / 27 | 27 / 27 | +Intel + Qualcomm NPU backends | | **Grand total** | 112 / 146 | **117 / 147** | **76.7% → 79.6%** | --- ## Next targets (v0.83+) - **Whole-model `.entity` → NPU** lowering (both vendors), not just per-layer - **NPU parity suite** vs WebGPU reference (SmolLM-class smoke) - **AccelPlanner** + JSON `exec` field (`qualcomm-npu` / `intel-npu` per layer) - **GPU backward** wiring (SwiGLU / MHA) continues from the v0.81 roadmap - **Google TPU** plugin (`libloom_accel_google.so`) — same ABI --- ## Key source files | Area | Files | |------|-------| | SIMD | `poly/simd/*.go` (`neon_arm64.go`, x86-64 kernels), `SetSimdForward` | | Accel package | `poly/accel/*.go` (`plugin_qualcomm_windows.go`, `plugin_qualcomm_stub.go`) | | Qualcomm dispatch | `poly/accel_qualcomm.go`, `poly/forward.go` | | Qualcomm plugin C++ | `accel/qualcomm/src/` (`qnn_wrapper.*`, `loom_accel_qualcomm.cpp`, `layer_models.*`) | | Build / install | `accel/qualcomm/build_clang.ps1`, `install_qairt.ps1`, `bench_manifest.json` | | Lucy suite | `Lucy examples/snapdragon/` | --- ## See also - [snapdragon_npu.md](snapdragon_npu.md) — Snapdragon bridge deep-dive (achievements + honest gaps) - [simd.md](simd.md) — SIMD CPU forward path - [accelerators.md](accelerators.md) — full vendor accel guide (Intel + Qualcomm) - [v081_release.md](v081_release.md) — previous release (Intel NPU + plugin model) --- ## v0.83.0 — Apple Bridge (Apple GPU / Metal + BF16) Source: https://openfluke.com/docs/v083-release Markdown: https://openfluke.com/docs/v083-release.md # v0.83.0 — Apple Bridge (Apple GPU / Metal + BF16) > **v0.84+:** Lucy lives in [lucy_bloom_rivers](lucy.md) (was `loom/lucy/`). Log and harness paths below are relative to the Lucy repo root. **Release:** **0.82.0 "Snapdragon Bridge"** → **0.83.0 "Apple Bridge"** **Checklist:** **119 / 149** (79.9%) on `adjustments` — a **third** accelerator vendor (Apple GPU via Metal/MPSGraph) advances the Accelerators and Ecosystem categories, plus a **BF16** wire dtype for the shared accel bridge. One headline item lands on top of the v0.82 Intel + Qualcomm bridges: 1. **Apple GPU / Metal** — the third `poly/accel` vendor plugin, running on **macOS Apple silicon** through Apple's **Metal Performance Shaders Graph**. Forward-only, per-layer, experimental — the same maturity bar as Intel and Qualcomm. No SDK to vendor: Metal ships with macOS. --- ## What shipped ### Apple Metal / MPSGraph plugin (`accel/apple`) | Item | Detail | |------|--------| | **Plugin** | `libloom_accel_apple.dylib` — Metal / MetalPerformanceShaders / MetalPerformanceShadersGraph behind the vendor-neutral `loom_accel.h` C ABI | | **Build** | CMake (`build.sh` → `build/libloom_accel_apple.dylib`); C++17 + Objective-C++ (`mps_backend.mm` under ARC) | | **Darwin loader** | `poly/accel/plugin_darwin.go` — `dlopen`/`dlsym`; `apple_stub.go` for non-darwin / no-cgo | | **Devices** | `ExecAppleCPU` (portable C++ reference, parity anchor) · `ExecAppleGPU` (Metal / MPSGraph, per-op CPU fallback) | | **GPU ops** | MatMul, MHA-MatMul, ReLU, Sigmoid, Softmax, Add, Multiply on MPSGraph; Conv/GELU/pool/norm fall back to the CPU reference | | **DTypes** | FP32 / FP16 / **BF16** / INT16 / INT8 / INT4 (see `accel/apple/bench_manifest.json`) | | **Install** | none — Xcode command-line tools provide the Metal frameworks | ### BF16 wire dtype (shared bridge) | Item | Detail | |------|--------| | **`poly/accel_intel.go`** | Vendor-neutral bridge now packs/unpacks **bfloat16** (top 16 bits of FP32, round-to-nearest-even) for weights and I/O — `float32ToBFloat16Bits` / `bfloat16BitsToFloat32` | | **Plugin side** | `accel/apple/src/half.hpp` (`float_to_bfloat16`/`bfloat16_to_float`), `shapes.hpp` (`known_dtype` + 2-byte `io_elem_size`), `WireFmt {FP32,FP16,BF16}` in `loom_accel_apple.cpp` | | **Rationale** | BF16 is the native low-precision type on Apple silicon; each accelerator advertises the dtypes it can handle via its own `bench_manifest.json` | ### Lucy [13] — Apple GPU bridge suite | Item | Detail | |------|--------| | **Menu** | `[13]` — mirrors the Intel `[9]` and Qualcomm `[12]` suites | | **Tables** | Timing (Loom / Apple CPU / Metal GPU, speedup + compile) + seven-style drift spectrum | | **Log** | `lucy_testing_output/apple.txt` | --- ## Numbers (from `apple.txt`, Lucy [13] → [5], 180 cells) - **Determinism:** 180/180 **💎 EXACT** repeat-forward on **both** Apple CPU and Metal GPU. - **Parity:** GPU 132/180 ≤ INDUS, CPU 78/180 (GPU carries a looser tolerance; raw drift is near-identical). - **Speed:** Metal GPU up to **5.4×** faster than Loom CPU on large MatMul/MHA; Apple CPU reference up to **94×** on elementwise (ReLU/GELU/Sigmoid at INT4). - **Weak spots:** Conv1D/Conv2D + GELU are CPU-reference-only (no MPSGraph path, ~0.24–0.27× Loom CPU); LayerNorm/RMSNorm parity ❌ BROKE (no weight bake); INT8 MatMul drift breaks on the large tier. --- ## What this release is (and is not) **You now have:** - A **third accelerator vendor** (Apple GPU) on macOS through the same `poly/accel` C ABI as Intel and Qualcomm - A **CPU reference + Metal GPU** pair behind one plugin, with transparent per-op fallback - A **BF16** wire dtype in the shared bridge (Apple-native low precision) - **Experimental** label — proven plumbing, good for a release, not for prod **You do not yet claim:** - MPSGraph Conv / GELU (both run the CPU reference today) - LayerNorm / RMSNorm weight bake (parity broken) - Apple Neural Engine (ANE) — Metal only; ANE needs a Core ML path (future) - Whole-model `.entity` → GPU lowering (offload is per-layer, forward-only) - Training or backward on the Apple path - A JSON network field for `exec: apple-gpu` (targets set in code) --- ## Quick start (developers, macOS Apple silicon) ```bash # 1. Build the plugin (needs Xcode command-line tools) cd accel/apple ./build.sh # 2. Run the Lucy Apple suite cd ../../lucy_bloom_rivers CGO_ENABLED=1 go run . # -> 13 # [4] medium DispatchLayer suite # [5] full 10×6×3 matrix (apple.txt) # [0] raw CABI matrix (all 15 layers) ``` `accel.DefaultApplePath()` walks up from cwd for `accel/apple/build/libloom_accel_apple.dylib`, or set `LOOM_ACCEL_APPLE_DYLIB`. --- ## Checklist deltas (v0.82 → v0.83) | Category | v0.82 | v0.83 | Change | |----------|:-----:|:-----:|--------| | 3. Accelerators & Distributed | 6 / 18 | 7 / 19 | +Apple GPU per-layer dispatch | | 5. Deployment Ecosystem | 27 / 27 | 28 / 28 | +Apple GPU backend | | **Grand total** | 117 / 147 | **119 / 149** | **79.6% → 79.9%** | --- ## Next targets (v0.84+) - **MPSGraph Conv / GELU** — move Conv1D/Conv2D/GELU off the CPU reference onto the GPU - **Norm weight bake** — fix LayerNorm/RMSNorm parity (Loom weights into the reference/graph) - **ANE via Core ML** — reach the Neural Engine (not a Metal device) - **Whole-model `.entity` → NPU/GPU** lowering (all vendors), not just per-layer - **NPU parity suite** vs WebGPU reference (SmolLM-class smoke) - **AccelPlanner** + JSON `exec` field (`apple-gpu` / `intel-npu` / `qualcomm-npu` per layer) - **Google TPU** plugin (`libloom_accel_google.so`) — same ABI --- ## Key source files | Area | Files | |------|-------| | Apple plugin C++ | `accel/apple/src/` (`loom_accel_apple.cpp`, `cpu_reference.*`, `mps_backend.mm`, `shapes.hpp`, `half.hpp`) | | Build | `accel/apple/CMakeLists.txt`, `accel/apple/build.sh`, `accel/apple/bench_manifest.json` | | Accel package | `poly/accel/plugin_darwin.go`, `poly/accel/apple_stub.go`, `target.go`, `registry.go`, `accel.go` | | Apple dispatch | `poly/accel_apple.go`, `poly/accel_intel.go` (BF16 + vendor-neutral routing), `poly/forward.go` | | Lucy suite | `Lucy examples/apple/`, `Lucy examples/apple_menu.go` | --- ## See also - [apple_metal.md](apple_metal.md) — Apple bridge deep-dive (results + honest gaps) - [accelerators.md](accelerators.md) — full vendor accel guide (Intel + Qualcomm + Apple) - [snapdragon_npu.md](snapdragon_npu.md) — Qualcomm/Hexagon bridge - [v082_release.md](v082_release.md) — previous release (SIMD + Qualcomm NPU) --- ## Vendor accelerators (NPU / TPU) Source: https://openfluke.com/docs/accelerators Markdown: https://openfluke.com/docs/accelerators.md # Vendor accelerators (NPU / TPU) **Version:** Loom **v0.83.0** — experimental **Status:** Intel CPU + NPU on Linux (Lucy [9]); Qualcomm/Hexagon NPU on Windows ARM64 (Lucy [12]); Apple Metal GPU on macOS (Lucy [13]); Google TPU planned This document covers the **`poly/accel`** package: how Loom offloads individual layers to vendor silicon through external C ABI plugins, without embedding OpenVINO, QNN, or TPU SDKs inside the Loom module. --- ## Why a separate accel track WebGPU covers **portable GPU** (Vulkan / Metal / DX12 / browser). Vendor NPUs and TPUs need **vendor SDKs** that do not belong in the core Go module: | Approach | Loom owns | External / vendor tree | |---|---|---| | **WebGPU** | WGSL, `WGPUContext`, buffers | wgpu-native prebuilts | | **Vendor accel** | `DispatchLayer` hook, tensor bytes, `ExecTarget` | `libloom_accel_intel.so`, OpenVINO, drivers | One network graph, one `ForwardPolymorphic` loop — per-layer `ExecTarget` picks **Loom CPU**, **Intel CPU**, **Intel NPU**, or (future) **Qualcomm NPU** / **Google TPU**. --- ## Architecture ``` BuildNetworkFromJSON → VolumetricNetwork → net.Accel = registry // DiscoverAccel once Per layer: layer.ExecTarget = ExecIntelNPU // or ExecIntelCPU, ExecLoomCPU net.SyncToAccel(sizeLabel) // compile once + upload weights ForwardPolymorphic → DispatchLayer → DispatchAccelForward // if ExecTarget.UseAccel() → else DenseForward / CNN / … ``` **C ABI header (vendor-neutral):** `accel/intel/include/loom_accel.h` (copy in `poly/accel/include/`) | Symbol | Purpose | |---|---| | `loom_accel_plugin_open("CPU"\|"NPU")` | Open device | | `loom_accel_compile_layer` | Build graph + bake weights | | `loom_accel_infer` | Steady forward | | `loom_accel_weight_bytes` | Expected native weight blob size (FP32/INT8: 4×N bytes, FP16: 2×N) | --- ## Intel (shipped — experimental) **Plugin:** `libloom_accel_intel.so` (OpenVINO inside) — **one shared object** per build (`accel/intel/build/`; no `.so.1` soname) **Build:** `accel/intel/` (`./build.sh` after `source setup_env.sh`) ### Requirements - **Linux** amd64/arm64 (Windows `.dll` planned) - **`CGO_ENABLED=1`** when building/running Loom - OpenVINO runtime + Intel NPU driver on **`LD_LIBRARY_PATH`** - Meteor Lake / Core Ultra class NPU (or CPU-only OpenVINO path) ### Environment ```bash export LOOM_ACCEL_INTEL_SO=/path/to/libloom_accel_intel.so # optional source accel/intel/setup_env.sh ``` `accel.DefaultIntelPath()` walks up from cwd for `accel/intel/build/libloom_accel_intel.so`, or set `LOOM_ROOT`. ### Application code ```go reg, err := poly.DiscoverAccel(accel.AccelConfig{ IntelSO: accel.DefaultIntelPath(), }) if err != nil { /* no plugin — stay on Loom CPU */ } defer reg.Close() net, _ := poly.BuildNetworkFromJSON(spec) net.Accel = reg for i := range net.Layers { net.Layers[i].ExecTarget = accel.ExecIntelNPU // or ExecIntelCPU } if err := net.SyncToAccel("medium"); err != nil { /* compile failed */ } out, _, _ := poly.ForwardPolymorphic(net, input) ``` ### `ExecTarget` values | Value | Runs on | |---|---| | `accel.ExecLoomCPU` | Default — Go poly CPU | | `accel.ExecIntelCPU` | OpenVINO CPU | | `accel.ExecIntelNPU` | OpenVINO Intel NPU plugin | ### Weight upload (C ABI v2) At **`SyncToAccel`** (init once per layer/device), Loom passes dtype-aware weight bytes via **`LayerWeightBytesForAccel`** into **`loom_accel_compile_layer`**. Weights are baked into the OpenVINO graph as constants — **not** re-sent on each forward. | Layer dtype | Weight bytes at compile | Notes | |---|---|---| | **FP32** | `WeightStore.Master` as little-endian float32 | Direct | | **FP16** | Native `[]uint16` from `Morph(FP16)` (IEEE half) | No FP32 round-trip | | **INT8** | Dequantized float32 from `GetActive(INT8)` | Matches Loom CPU matmul; OV INT8 graph is still f32 + NPU dynamic quant | Steady forward only passes **activation** buffers: `loom_accel_infer(in, out)` — pointer + memcpy, no weight pointer per hop. | Layer | Loom weights baked into OV graph? | |---|---| | MatMul, MHA-MatMul | ✅ when byte count matches | | Conv1D, Conv2D | ✅ when byte count matches | | ReLU, GELU, Sigmoid, Softmax | ❌ fixed constants in CABI | | LayerNorm, RMSNorm | ❌ fixed constants in CABI (planned: weight bake) | ### Multi-numerical support — what we have vs what we don’t | Scope | Status | |---|---| | **Loom CPU** | All **21** `DType` values via `WeightStore.Morph` + `GetActive` | | **Intel accel (bench)** | **FP32, FP16, INT8** only — matches `bench_manifest.json` / Lucy [9] matrix | | **Weight upload to Intel** | ✅ FP32 / FP16 / INT8 on MatMul, Conv, MHA-MatMul (June 2026) | | **Activation I/O to Intel** | FP32 / FP16 native bytes; INT8 activations expanded to f32 bytes for OV graph | | **Not on Intel yet** | BFloat16, Int4, FP8, native INT8 weight constants, remaining morph dtypes | So: **proper multi-dtype weight upload for the three Intel bench dtypes** — yes. **Full Loom numerical-type parity on NPU** — no (and INT8 math path still differs from Loom on large MAC ops). ### Is the integration “done”? **Forward dispatch bridge: shipped, experimental — not production-complete.** | Done (v0.81) | Not done (roadmap) | |---|---| | `DiscoverAccel` + `SyncToAccel` + `DispatchAccelForward` | JSON `"exec": "intel-npu"` / AccelPlanner | | Intel **CPU + NPU** on Linux, CGO + `libloom_accel_intel.so` | Training / backward on vendor path | | Init-once compile + dtype-aware weight bake | LayerNorm / RMSNorm weight bake | | Lucy **[9]** 90-cell matrix + `nine_layer.txt` | Bit-perfect parity all layers/dtypes | | Auto OpenVINO path discovery from Lucy cwd | Windows plugin, zero-copy I/O | | Intel infer **💎 EXACT** repeat-forward (90/90) | Qualcomm / Google plugins | Treat integration as **proven plumbing** you can build on — not a finished “flip NPU on for any network” product feature. ### Benchmark snapshot — Lucy [9] → [5] (90 cells) **Host:** Fedora, Core Ultra class NPU, OpenVINO via chaosglue deps. **Log:** `lucy_testing_output/nine_layer.txt` (Lucy repo — see [lucy.md](lucy.md)) **Method:** `SyncToAccel` once per device, median infer ms (compile excluded). **Spd** = Loom ÷ Intel (< 1 = Intel slower). #### Manifest | Check | Pass | Fail | Total | |---|---:|---:|---:| | Intel faster than Loom (**CPU**) | 56 | 34 | 90 | | Intel faster than Loom (**NPU**) | 36 | 54 | 90 | | Loom↔Intel parity ≤ INDUS (CPU) | 23 | 67 | 90 | | Loom↔Intel parity ≤ INDUS (NPU) | 61 | 29 | 90 | | Intel infer repeat-forward (CPU) | **90** | 0 | 90 | | Intel infer repeat-forward (NPU) | **90** | 0 | 90 | Determinism is production-grade; speed and parity are layer/size dependent. #### Small tier (batch=4, dim=32 — latency floor) Intel **NPU loses almost every cell** (~0.3–0.7 ms infer floor vs Loom ~0.01 ms). | Layer | DType | Loom ms | Intel CPU ms | Intel NPU ms | Spd CPU | Spd NPU | |---|---|---:|---:|---:|---:|---:| | MatMul | FP32 | 0.009 | 0.013 | 0.585 | 0.69× | 0.02× | | Conv1D | FP32 | 0.097 | 0.038 | 0.601 | 2.6× | 0.16× | | ReLU | FP32 | 0.008 | 0.009 | 0.347 | 0.89× | 0.02× | | Softmax | FP32 | 0.001 | 0.014 | 0.343 | 0.07× | 0.00× | #### Medium tier (batch=16, dim=256) | Layer | DType | Loom ms | Intel CPU ms | Intel NPU ms | Spd CPU | Spd NPU | |---|---|---:|---:|---:|---:|---:| | MatMul | FP32 | 0.415 | 0.040 | 0.607 | **10×** | 0.68× | | MatMul | FP16 | 0.644 | 0.056 | 0.312 | **12×** | **2.1×** | | Conv1D | FP32 | 0.985 | 0.291 | 0.698 | **3.4×** | 1.4× | | Conv2D | FP32 | 10.134 | 1.456 | 1.252 | **7.0×** | **8.1×** | | Conv2D | INT8 | 10.815 | 1.233 | 1.162 | **8.8×** | **9.3×** | | ReLU | INT8 | 1.277 | 0.027 | 0.612 | **47×** | 2.1× | #### Large tier (batch=8, dim=1024; Conv2D 48×48) | Layer | DType | Loom ms | Intel CPU ms | Intel NPU ms | Spd CPU | Spd NPU | |---|---|---:|---:|---:|---:|---:| | MatMul | FP32 | 3.520 | 0.220 | 0.701 | **16×** | **5.0×** | | MatMul | FP16 | 5.803 | 0.268 | 0.666 | **22×** | **8.7×** | | Conv1D | FP32 | 28.140 | 2.438 | 2.310 | **12×** | **12×** | | Conv2D | FP32 | 117.441 | 5.355 | 5.683 | **22×** | **21×** | | GELU | FP32 | 3.322 | 0.074 | 0.615 | **45×** | 5.4× | | MHA-MatMul | INT8 | 4.277 | 0.196 | 0.678 | **22×** | **6.3×** | #### When to offload | Target | Offload | Skip | |---|---|---| | **Intel CPU** | medium/large **Conv2D, Conv1D, MatMul, MHA**; large activations | small tensors; norms (parity ❌) | | **Intel NPU** | **large** MAC (Conv2D ~8–21×); medium+ MatMul **FP16** | **small** tier; medium MatMul FP32 (floor > compute); norms/softmax | | **Either** | Many steady forwards (compile 7–70 ms once) | One-shot micro-ops | **Why Intel looks slower on some rows:** infer time = fixed dispatch overhead + math. NPU overhead ≈ **0.5 ms**; Loom small MatMul ≈ **0.01 ms**. Weights-on-init removes re-upload — it does not remove per-hop CGO, pack/unpack, or NPU launch latency. **Parity highlights:** Softmax/Sigmoid strong; LayerNorm/RMSNorm **❌ BROKE** (~1.8, no Loom weights); INT8 MAC drift 3–36 on large tiers (Loom dequant matmul vs OV f32 + dynamic quant). ### `sizeLabel` Must match bench manifest tiers used when the OpenVINO graph was authored: **`small`**, **`medium`**, **`large`**. Wrong label → shape mismatch at infer. ### Limitations (v0.81) - **Forward only** — training/backward use Loom CPU when accel-bound - **Manual `ExecTarget`** — no JSON `"exec": "intel-npu"` yet (AccelPlanner planned) - **Three Intel dtypes** — FP32/FP16/INT8 bench path only; not all 21 Loom dtypes - **Numerical parity** — norms broken; INT8 MAC drift on large tiers; MatMul FP32 medium 🟤 H-DRIFT - **Small tensors** — NPU ~0.5 ms floor; offload **medium/large** MAC ops only - **Per-hop copies** — tensor → `[]byte` → OV tensor; not zero-copy yet ### Validation — Lucy menu [9] ```bash cd lucy_bloom_rivers CGO_ENABLED=1 go run . # OpenVINO paths auto-discovered from chaosglue npu deps (no setup_env.sh required) # → 9 → 4 medium DispatchLayer suite # → 9 → 5 full 90-cell matrix ``` Or: `./run_npu_bridge.sh` from Lucy repo root (sources `accel/intel/setup_env.sh` explicitly). Output: timing table (Loom vs Intel CPU vs Intel NPU, speedup ratios) + seven-style drift spectrum + manifest histogram. Log: `lucy_testing_output/nine_layer.txt` — see [`testing_and_validation.md`](testing_and_validation.md#nine-layer-intel-bridge-nine_layertxt). --- ## Welvet C-ABI (non-Go bindings) C / Flutter / Python callers use **`welvet/cabi`** (`welvet.so` / `welvet.h`) instead of importing `poly` directly. Intel offload mirrors the Go flow: ```c // Pseudocode — see welvet.h for exact signatures long accel = LoomDiscoverAccel(NULL); // optional LOOM_ACCEL_INTEL_SO path long net = LoomBuildNetworkFromJSON(spec_json); LoomNetworkAttachAccel(net, accel); LoomSetLayerExecTarget(net, layer_idx, ExecIntelNPU); LoomSyncToAccel(net, "medium"); LoomDispatchAccelForward(net, layer_idx, input_handle); ``` | Export | Maps to `poly` | |--------|----------------| | `LoomDiscoverAccel` | `DiscoverAccel` | | `LoomSyncToAccel` | `VolumetricNetwork.SyncToAccel` | | `LoomLayerWeightBytesForAccel` | `LayerWeightBytesForAccel` | | `LoomDispatchAccelForward` | `DispatchAccelForward` | **Build Linux:** `cd welvet/cabi/internal/build && ./build_linux.sh` → `dist/linux_amd64/welvet.so` (and `arm64`). **Parity:** `cd welvet/cabi/internal/check && go run .` → **489/489**. The Intel plugin (`libloom_accel_intel.so`) is still a **separate** dlopen artifact — Welvet does not link OpenVINO at compile time. --- ## Qualcomm NPU (shipped — experimental) → see [`snapdragon_npu.md`](snapdragon_npu.md) **Plugin:** `loom_accel_qualcomm.dll` (Qualcomm **QNN AI Engine Direct** inside) **SDK:** QAIRT / QNN — vendored under `accel/qualcomm/deps/` **Platform:** Windows on Snapdragon X · Hexagon v73 HTP + Kryo CPU · **Lucy [12]** The Snapdragon path is now real, not planned — same `loom_accel.h` vtable, same `SyncToAccel` → `DispatchLayer` flow, opened via `poly.DiscoverQualcommAccel`: ```go reg, _ := poly.DiscoverQualcommAccel(accel.AccelConfig{ QualcommSO: accel.DefaultQualcommPath(), }) net.Accel = reg net.Layers[0].ExecTarget = accel.ExecQualcommNPU // or ExecQualcommCPU ``` **What works today** (from `lucy_testing_output/snapdragon.txt`): activation ops (ReLU/GELU/Sigmoid/Softmax across FP32/INT16/INT8/INT4) and MatMul/MHA FP32 on the Hexagon HTP — **💎 EXACT** repeat-forward determinism (54/54), NPU parity 45/54 ≤ INDUS, up to **7× faster than Loom CPU** on large tiers. **What doesn't yet:** Conv1D/Conv2D, LayerNorm/RMSNorm, FP16 anywhere, and quantized MatMul all still error at graph build. Full achievements, benchmark tables, and the honest gap list are in [`snapdragon_npu.md`](snapdragon_npu.md); build/env/webgpu notes in [`accel/qualcomm/README.md`](../accel/qualcomm/README.md). --- ## Apple GPU / Metal (shipped — experimental) → see [`apple_metal.md`](apple_metal.md) **Plugin:** `libloom_accel_apple.dylib` (Apple **Metal Performance Shaders Graph** inside) **SDK:** none to vendor — Metal / MPS / MPSGraph ship with macOS **Platform:** macOS on Apple silicon · Metal GPU (MPSGraph) + portable CPU reference · **Lucy [13]** Same `loom_accel.h` vtable, same `SyncToAccel` → `DispatchLayer` flow, opened via `poly.DiscoverAppleAccel`: ```go reg, _ := poly.DiscoverAppleAccel(accel.AccelConfig{ AppleSO: accel.DefaultApplePath(), }) net.Accel = reg net.Layers[0].ExecTarget = accel.ExecAppleGPU // or ExecAppleCPU ``` Two devices behind one ABI: a **CPU reference** (deterministic parity anchor, all 15 layers) and a **Metal GPU** (MPSGraph for MatMul/MHA/ReLU/Sigmoid/Softmax/Add/Multiply; per-op CPU fallback for the rest). Bench matrix is **6 dtypes** — FP32/FP16/**BF16**/INT16/INT8/INT4 — with BF16 the Apple-native addition (compute is FP32; the dtype sets the wire byte layout). **What works today** (from `lucy_testing_output/apple.txt`, Lucy [13] → [5], 180 cells): byte-perfect repeat-forward determinism — **180/180 💎 EXACT on both CPU and GPU** — GPU parity 132/180 ≤ INDUS, and up to **5.4× faster than Loom CPU** on large MatMul/MHA (GPU) and up to **94× on elementwise** (CPU reference). **What doesn't yet:** Conv1D/Conv2D and GELU are CPU-reference-only (no MPSGraph path, slower than Loom CPU), LayerNorm/RMSNorm parity is broken (no Loom weight bake), and INT8 MatMul drift breaks on the large tier. ANE is not wired (Metal only). Full tables and the honest gap list are in [`apple_metal.md`](apple_metal.md); build/dtype notes in [`accel/apple/README.md`](../accel/apple/README.md). --- ## Google TPU (planned) **Target plugin:** `libloom_accel_google.so` **SDK:** libtpu / OpenXLA PJRT (deployment TBD) Same C ABI surface. Useful for cloud TPU pods and future edge TPU silicon. Loom remains a **client** that compiles per-layer subgraphs and ships weights once. --- ## Vendor support matrix — shipped vs planned The `loom_accel.h` contract is deliberately general: any silicon that can **compile one layer** and **`infer(in_bytes → out_bytes)`** can become a `libloom_accel_` plugin. In practice the set of *main* vendors is finite (~8–10 silicon "stories", not an endless zoo of one-off NPUs), and many overlap. ### First: "GPU" means two different things in Loom These are **not** the same path and the matrix keeps them in separate columns: | | **WebGPU** (portable GPU) | **Native GPU accel plugin** | |---|---|---| | **What it is** | Loom's own compute path via `github.com/openfluke/webgpu` | A `loom_accel_*` plugin whose "GPU" device calls a **vendor graph runtime** | | **Under the hood** | WGSL shaders → **Vulkan / DX12 / Metal** (no CUDA/ROCm) | MPSGraph (Apple) · OpenVINO `GPU` (Intel) · `QnnGpu` (Adreno) · cuBLAS/OpenCL | | **Scope** | **Whole network**, forward **+ backward + training** | **Per-layer** offload, **forward-only** | | **Portability** | One path, every GPU vendor at once | Vendor + OS specific, one `.so`/`.dylib` each | | **Runs any discrete GPU?** | ✅ NVIDIA, AMD, Intel, Apple already | Only where that vendor's plugin exists | | **Best for** | LLM decode, large batches, training | Fixed-function MAC ops routed alongside NPU targets | **Takeaway:** a plain gaming GPU (NVIDIA/AMD) is **already supported today** — through **WebGPU**, not through a vendor accel plugin. A native GPU plugin is only worth building when the vendor's own runtime beats WebGPU for a specific op, or to sit next to that vendor's NPU under one `ExecTarget` story. See [Comparison to WebGPU](#comparison-to-webgpu) below. ### The matrix **Legend:** ✅ shipped · 🟢 easy extend (reuses an existing path/SDK) · 📋 planned · ❌ not started · ➖ n/a | Bucket | Vendor / silicon | CPU | GPU · **WebGPU** (whole-net) | GPU · **accel plugin** (per-layer) | NPU · accel plugin | Loom path | Status | |---|---|:--:|:--:|:--:|:--:|---|:--:| | **Portable CPU** | Any x86-64 / ARM64 | ✅ | ➖ | ➖ | ➖ | Go + SIMD (AVX2/FMA, NEON) | ✅ shipped | | **Portable GPU** | NVIDIA · AMD · Intel · Apple | ➖ | ✅ | ➖ | ➖ | WebGPU → Vulkan / DX12 / Metal | ✅ shipped | | **Intel** | Core Ultra (Meteor/Lunar Lake) | ✅ | ✅ | 🟢 | ✅ | `accel/intel` (OpenVINO), Lucy **[9]** | ✅ shipped · plugin-GPU = OpenVINO `GPU` device | | **Qualcomm** | Snapdragon X · Hexagon · Adreno | ✅ | ✅ | 🟢 | ✅ | `accel/qualcomm` (QNN), Lucy **[12]** | ✅ shipped · Adreno `QnnGpu` hook untested | | **Apple** | M-series · Metal · ANE | ✅ | ✅ | ✅ | 🟢 | `accel/apple` (MPSGraph), Lucy **[13]** | ✅ shipped · ANE via Core ML later | | **Google** | Cloud TPU · Coral edge TPU | 🟢 | ➖ | ➖ | 📋 | `accel/google` (libtpu / PJRT / XLA) | 📋 planned | | **AMD** | Ryzen AI (XDNA NPU) + Radeon | 🟢 | ✅ | ❌ | ❌ | GPU via WebGPU · NPU = new plugin (Vitis / ONNX-RT) | ❌ NPU not started | | **NVIDIA** | GeForce / RTX / datacenter | ➖ | ✅ | ❌ | ❌ | GPU via WebGPU · optional cuBLAS-only MatMul plugin | ❌ native plugin optional | | **AWS** | Inferentia · Trainium | 🟢 | ➖ | ➖ | ❌ | `accel/aws` (Neuron SDK) — cloud / private-cloud batch | ❌ not started | | **Generic GPU** | AMD + NVIDIA + Intel (one plugin) | 🟢 | ✅ | ❌ | ➖ | `accel/opencl` (OpenCL + CLBlast) — vendor-neutral fallback | ❌ candidate quick win | | **Mobile NPUs** | MediaTek · Samsung · Android NNAPI | ➖ | 🟢 | ❌ | ❌ | per-vendor plugin — only if Loom targets phones/tablets | ❌ not started | | **Embedded** | Hailo · Rockchip RKNN · Ambarella | ➖ | ➖ | ❌ | ❌ | one-off plugins — camera / wearable edge, on demand | ❌ out of scope for now | ### Reading the matrix - The **"GPU · WebGPU"** column is the ✅ that already makes NVIDIA/AMD/Intel/Apple GPUs work — **whole-network, trainable, portable**. The **"GPU · accel plugin"** column is the *separate* per-layer, forward-only native GPU device inside a vendor `.so` (only Apple's MPSGraph is shipped ✅ today). - **Three vendor plugins are shipped** — Intel (Linux NPU), Qualcomm (Windows ARM64 Hexagon), Apple (macOS Metal) — all forward-only, per-layer, experimental, with Lucy benchmark suites and drift tables. - **Discrete NVIDIA/AMD GPUs need nothing new** — they run on WebGPU now. A native cuBLAS/OpenCL plugin is optional (a "badge" or an op-specific speedup, not a requirement). - **Easiest next steps (🟢 / candidate):** an Intel `GPU` device on the plugin you already ship, a WebGPU-backed `accel` bridge (one Lucy menu, reuses existing WGSL), or an OpenCL plugin (AMD + NVIDIA under one `.so`). - **The realistic ceiling** is the ~7 rows that matter for laptops, clinics, sovereign cloud, and edge — not "every NPU on earth." The set is **finishable**, not infinite. > This is a **direction-of-travel** table, not a promise of dates. Shipped rows are experimental; planned/❌ rows depend on demand (healthcare fleets, sovereign deployments) and vendor-SDK friction. --- ## Package layout ``` poly/ ├── accel/ │ ├── accel.go Public types, DefaultIntelPath, DefaultQualcommPath, DefaultApplePath │ ├── target.go ExecTarget enum (Loom/Intel/Qualcomm/Apple × CPU/NPU/GPU) │ ├── registry.go Discover, DiscoverQualcomm, DiscoverApple, PluginFor │ ├── plugin_linux.go Intel dlopen + C ABI calls (CGO) │ ├── runtime_linux.go OpenVINO LD_LIBRARY_PATH hints │ ├── plugin_qualcomm_windows.go Qualcomm LoadLibrary + C ABI (CGO, windows) │ ├── plugin_qualcomm_stub.go No-op Qualcomm stubs (non-windows / no-cgo) │ ├── plugin_darwin.go Apple dlopen + C ABI calls (CGO, darwin) │ └── apple_stub.go No-op Apple stubs (non-darwin / no-cgo) ├── accel_intel.go Vendor-neutral SyncToAccel, DispatchAccelForward, dtype bytes ├── accel_qualcomm.go DiscoverQualcommAccel entry point ├── accel_apple.go DiscoverAppleAccel entry point └── forward.go DispatchLayer → DispatchAccelForward ``` --- ## Comparison to WebGPU | | WebGPU | Vendor accel | |---|---|---| | **Scope** | Full network GPU path | Per-layer offload | | **Portability** | Vulkan/Metal/browser | Vendor + OS specific | | **Build** | Pure Go + wgpu module | **CGO** + external `.so` | | **Training** | GPU backward supported | Forward only (v0.81) | | **Best for** | LLM decode, large batches | Fixed-function NPU MAC ops | Use **both**: WebGPU for general GPU; Intel NPU for Conv/MatMul on Core Ultra when shapes are large enough. --- ## Roadmap | Milestone | Description | |---|---| | **v0.81** ✅ | Intel forward dispatch, dtype-aware weight upload (FP32/FP16/INT8), Lucy [9], benchmark tables in docs | | **v0.82** ✅ | Qualcomm/Hexagon NPU plugin (QNN, Windows ARM64), `ExecQualcomm*` targets, Lucy [12] `snapdragon` bench (FP32/FP16/INT16/INT8/INT4); SIMD CPU fast-path (AVX2/NEON); see [`snapdragon_npu.md`](snapdragon_npu.md) | | **v0.83** ✅ | Apple Metal/MPSGraph plugin (macOS Apple silicon), `ExecApple*` targets, Lucy [13] `apple` bench (FP32/FP16/**BF16**/INT16/INT8/INT4); GPU MatMul/MHA + CPU reference; see [`apple_metal.md`](apple_metal.md) | | **v0.84+** | Whole-model `.entity` → NPU lowering, NPU parity suite vs WebGPU, AccelPlanner + JSON `exec`, Google plugin; MPSGraph Conv/GELU, norm weight bake, ANE via Core ML; backward CPU fallback policy | | **v1.0** | Vendor accel rows enter formal 1.0 checklist | --- ## See also - [`snapdragon_npu.md`](snapdragon_npu.md) — Qualcomm/Hexagon NPU bridge - [`apple_metal.md`](apple_metal.md) — Apple Metal GPU / MPSGraph bridge - [`dispatch.md`](dispatch.md) — `DispatchLayer` routing - [`gpu.md`](gpu.md) — WebGPU backend - [`v081_release.md`](v081_release.md) — release notes - [`testing_and_validation.md`](testing_and_validation.md) — Lucy log interpretation --- ## Plan 9 SIMD — Forward and Backward Source: https://openfluke.com/docs/simd Markdown: https://openfluke.com/docs/simd.md # Plan 9 SIMD — Forward and Backward Loom's CPU SIMD path uses **Plan 9 assembly** (`.s` files) for hot GEMV-style kernels on **amd64** (AVX2) and **arm64** (NEON). Forward passes use **`DotTile`**; backward passes use **`SaxpyF32AccF64`** (and strided variants). Both are wired through the same **`UseSimdForward`** flag and **`TrainingModeCPUSimd`** training mode. ## Overview | Kernel | Role | amd64 | arm64 | |--------|------|-------|-------| | `DotTile` | Forward GEMV (dot products) | `avx2_amd64.s` | `dot_neon_arm64.s` | | `SaxpyF32AccF64` | Backward weight/input accumulation | `saxpy_avx2_amd64.s` | `saxpy_neon_arm64.s` | | `BitNetTernaryMAD` | Ternary weight forward (optional) | `bitnet_ternary_amd64.s` | `bitnet_ternary_arm64.s` | **Layers with SIMD forward + backward:** Dense, SwiGLU, MHA, CNN1, CNN2, CNN3, RNN, LSTM, Embedding, Residual. **Native-exact SIMD** (`*_native_simd.go`): same numerics as `*_native.go`, faster kernels — MAC dtypes via `materializeF32Weights` + `DotTile`; true integers via `DotI8Tile` / `SaxpyI8*`. Enabled when both `UseExactDType` and `UseSimdForward` are on (Lucy menus **[14]** and **[15]**). **Layers without heavy GEMV SIMD:** RMSNorm, Softmax (attention softmax/RoPE in MHA remain scalar). ## Enabling SIMD ```go // Per-network (training helper) poly.ConfigureNetworkForMode(net, poly.TrainingModeCPUSimd) // Or manually net.SetSimdForwardRecursive(true) // enables forward DotTile + backward saxpy on all layers ``` `TrainingModeCPUSimd` is MC-tiled CPU training with SIMD: same tiling/parallelism as `TrainingModeCPUMC`, plus recursive `UseSimdForward` on every compute layer. ## Package layout ``` poly/simd/ ├── dot.go, saxpy.go # Go wrappers, feature detection ├── avx2_amd64.s # DotTile (AVX2) ├── dot_neon_arm64.s # DotTile (NEON) ├── saxpy_avx2_amd64.s # SaxpyF32AccF64 (AVX2) ├── saxpy_neon_arm64.s # SaxpyF32AccF64 (NEON) ├── bitnet_ternary_amd64.s # optional ternary forward └── stub.go # scalar fallback when !amd64 && !arm64 poly/ ├── simd_forward.go # layerUseSimdForward, SetSimdForwardRecursive ├── {dense,swiglu,mha,cnn1,cnn2,cnn3,rnn,lstm}_simd.go # default forward (GetActive FP32) ├── {dense,swiglu,mha,cnn1,cnn2,cnn3,rnn,lstm}_simd_backward.go # default backward ├── {dense,swiglu,mha,cnn1,cnn2,cnn3,rnn,lstm,embedding,residual}_native_simd.go # native-exact SIMD ├── embedding_simd.go # parallel lookup / scatter (MAC native bridge) └── residual_simd.go # polymorphic residual add (non-native-exact) ``` ## Forward path When `layerUseSimdForward(layer) && simd.SimdEnabled()`: - **Dense / SwiGLU / RNN / LSTM:** `simd.DotTile` over weight rows (same tile structure as MC tiled path). - **MHA:** DotTile on Q/K/V/O projections; attention softmax and RoPE stay scalar. - **CNN1/2/3:** DotTile over filter patches where the receptive field is contiguous. Accumulation uses **float64** internally where the tiled path does, for dtype parity. ## Backward path When the same SIMD gate is true, each layer's `*BackwardPolymorphic` tries `try*BackwardSimd` first: | Layer | ∂L/∂W | ∂L/∂X / hidden | |-------|-------|----------------| | Dense, SwiGLU | `SaxpyF32AccF64` per output unit | saxpy scatter into input grad | | MHA | saxpy on projection paths | DotTile for Q recompute; rest scalar | | CNN1/2/3 | saxpy over contiguous input patches | output-centric saxpy scatter (∂L/∂X) | | RNN, LSTM | saxpy per hidden unit (LSTM: 4 gates) | BPTT; hidden-hidden / cell carry scalar | CNN backward uses an **output-centric saxpy scatter** for ∂L/∂X (matches tiled exactly; stride-1 full-kernel fast paths, scalar at edges). RNN/LSTM: parallel over batch; ∂L/∂W_IH and ∂L/∂X via saxpy; hidden-hidden, bias, and cell state paths match tiled scalar code. ## Native-exact SIMD (`*_native_simd.go`) When `UseExactDType` and `UseSimdForward` are both enabled, native-exact layers try SIMD before scalar native fallback: ``` LayerForwardNativeExact └─ layerUseSimdForward → try*ForwardNativeSimd ├─ use*TrueNative → DotI8Tile / SaxpyI8* (int8 MAC) └─ else → materializeF32Weights → *SimdF32WithWeights └─ fallback → scalar *_native.go ``` Lucy **[14]** reports SIMD fwd/bwd speedup columns for layers that implement this bridge (Dense through Residual). Default menu **[7]** SIMD uses `GetActive` FP32 tiles — same fast kernels, **QAT-like** numerics. Full **[14]** pass matrix and native-exact speedup tables: [native_layers.md](native_layers.md). ## Validation **Unit tests** (`poly/tests/*_backward_test.go`): SIMD backward vs tiled forward/backward; SC vs MC where applicable. **Seven-layer suite** (Lucy `examples/seven_layer/`): - `TestSimdParityAllLayers_Float32_1x1` — fwd/bwd SC ↔ MC ↔ SIMD, all seven compute layers - Per-layer grid tests (e.g. `TestCNN3SimdParityAllGrids_Float32`, `TestLSTMSimdParityAllGrids_Float32`) - Training table includes **CPU-SIMD** column alongside SC and MC Run: ```bash cd lucy_bloom_rivers && go test ./examples/seven_layer/ -run Simd -count=1 go test ./poly/tests/ -run Backward -count=1 ``` ## Seven-layer benchmark results Logs: `seven_layer_amd.txt`, `seven_layer_arm.txt` (Float32, 1×1×1 grid, **avg 25 passes** per cell). Speedup = SC time ÷ SIMD time (>1 means SIMD is faster). ### Forward (SIMD vs SC) | Layer | AMD | ARM | |-------|-----|-----| | Dense | **3.6×** (72% faster) | **3.0×** (67% faster) | | SwiGLU | **2.0×** (49% faster) | **1.5×** (34% faster) | | MHA | **1.7×** (42% faster) | **1.6×** (36% faster) | | CNN1 | **1.2×** (16% faster) | **1.8×** (44% faster) | | CNN2 | **1.4×** (26% faster) | ~1.0× | | CNN3 | 0.65× (52% slower) | 0.85× (15% slower) | | RNN | **1.6×** (39% faster) | **2.0×** (51% faster) | | LSTM | **1.2×** (15% faster) | **1.1×** (9% faster) | Dense forward SIMD is the largest win on both platforms (wide GEMV). CNN3 forward can be slower than SC on small grids (patch layout / overhead); backward still wins (below). ### Backward (SIMD vs SC) | Layer | AMD | ARM | |-------|-----|-----| | Dense | **1.7×** (40% faster) | **1.1×** (11% faster) | | SwiGLU | 0.85× (15% slower) | **1.3×** (23% faster) | | MHA | **1.8×** (46% faster) | **1.2×** (20% faster) | | CNN1 | **1.3×** (21% faster) | **1.5×** (32% faster) | | CNN2 | **2.3×** (56% faster) | **1.7×** (40% faster) | | CNN3 | **2.8×** (64% faster) | **1.6×** (38% faster) | | RNN | ~1.0× | **1.6×** (38% faster) | | LSTM | **1.1×** (7% faster) | **1.6×** (37% faster) | CNN2/CNN3 backward saxpy paths show the strongest gains. SwiGLU backward on AMD can trail SC slightly (strided weights + gate structure); ARM sees a modest win. ### Training (Dense, Float32, 50 epochs, 1×1×1) | Platform | SC | MC | SIMD | |----------|-----|-----|------| | AMD | 14.3 ms (~3496 steps/s) | 16.4 ms (~3046 steps/s) | **9.72 ms (~5144 steps/s)** | | ARM | 8.96 ms (~5576 steps/s) | 12.4 ms (~4029 steps/s) | **9.88 ms (~5063 steps/s)** | On AMD, SIMD training is ~**47% faster** than SC for this Dense micro-benchmark. On ARM, SIMD sits between SC (fastest for this tiny grid) and MC. ### Parity On both platforms, seven-layer **Float32 1×1×1** parity tables report **0** fwd/bwd diff across SC, MC, and SIMD for all seven layer types (21 dtype rows in full suite). Larger grids (2³, 3³) are covered by per-layer `*SimdParityAllGrids*` tests. ## BitNet (optional) Ternary / packed BitNet forward kernels live in the same `poly/simd` package (`bitnet_ternary_*.s`, TL1 variants on arm64). These are separate from the main float32 training SIMD path; see `poly/simd/doc.go` and BitNet tests. ## See also - [Training](training.md) — `TrainingModeCPUSimd`, `UseExactDType`, and [training paradigms](training.md#training-paradigms-default-qat-like-vs-native-exact) - [Quantization](quantization.md) — [three training/inference modes](quantization.md#three-traininginference-modes) - [Testing and validation](testing_and_validation.md) — seven-layer SC/MC/SIMD suite - [Bedrock validation](bedrock_validation.md) — menu option [7] and artifact layout - [Cross-path layers](cross_path_layers.md) — menu [15] QAT-SIMD vs Nat-SIMD duel (grid [5], archived amd64/arm64 logs) --- ## Snapdragon (Hexagon NPU) bridge Source: https://openfluke.com/docs/snapdragon-npu Markdown: https://openfluke.com/docs/snapdragon-npu.md # Snapdragon (Hexagon NPU) bridge **Version:** Loom **v0.82.0 "Snapdragon Bridge"** — **experimental** (same maturity bar as the Intel NPU bridge) **Status:** Windows on Snapdragon X · Hexagon v73 HTP + Kryo CPU reference · forward-only **Plugin:** `loom_accel_qualcomm.dll` (Qualcomm **QNN AI Engine Direct** inside) **Lucy menu:** **[12]** — log: `lucy_testing_output/snapdragon.txt` (see [lucy.md](lucy.md)) > This is the Snapdragon mirror of the Intel path in [`accelerators.md`](accelerators.md). > It shares the identical vendor-neutral C ABI (`accel/qualcomm/include/loom_accel.h`), > the same `poly/accel` loader, the same `SyncToAccel` → `DispatchLayer` flow. Only the > vendor backend differs: QNN/Hexagon instead of OpenVINO/Intel. --- ## TL;DR — this was a lot of work, and it works enough to demo Getting Loom talking to the Hexagon NPU on Windows/ARM64 required clearing a long chain of blockers, none of which were "the neural net" itself: - **ARM64 SIMD** kernels (`neon_arm64.go`) so the Loom CPU baseline runs natively. - **`webgpu` `go.mod` replace** — the published static archive has an ABI mismatch on windows/arm64; both `loom/go.mod` and Lucy `go.mod` must point at a local build. - **QAIRT SDK acquisition** — ~1.3 GB Qualcomm SDK behind a login, vendored under `accel/qualcomm/deps/`, plus the free HTP runtime with signed Hexagon skel/cat. - **A C++ QNN plugin** built with **llvm-mingw** (no CMake/MSVC on the box) that loads `QnnHtp.dll` / `QnnCpu.dll`, builds a single-op graph per layer, quantizes weights, finalizes and executes it. - **Windows cgo loader** (`LoadLibraryExA` + `GetProcAddress`) for the plugin. - **Machine-wide env** so `QNN_SDK_ROOT` survives a fresh SSH login / any user. - **Bug hunts:** duplicate QNN graph names failing every dtype after FP32; a graph leak that slowed the full matrix and could fault (`0xc0000005`) on Ctrl+C; a wall of DSP-transport log spam. The result: **a real Loom → Hexagon NPU dispatch bridge with deterministic inference and multi-× speedups on larger layers.** It is **not** production-complete — plenty of layers/dtypes still fall back or error. Treat it exactly like the Intel bridge: **proven plumbing to build on, not a "flip the NPU on for any network" feature.** --- ## What actually runs today Measured from `snapdragon.txt` (Lucy **[12] → [5]**, full 15-layer × 5-dtype × 3-size matrix, `SyncToAccel` once per device): | Layer | FP32 | FP16 | INT16 | INT8 | INT4 | |---|:--:|:--:|:--:|:--:|:--:| | **MatMul** | ✅ HTP+CPU | ✗ | ✗ | ✗ | ✗ | | **MHA-MatMul** | ✅ HTP | ✗ | ✗ | ✗ | ✗ | | **ReLU** | ✅ | ✗ | ✅ | ✅ | ✅ | | **GELU** | ✅ | ✗ | ✅ | ✅ | ✅ | | **Sigmoid** | ✅ | ✗ | ✅ | ✅ | ✅ | | **Softmax** | ✅ | ✗ | ✅ | ✅ | ✅ | | **Conv1D / Conv2D** | ✗ | ✗ | ✗ | ✗ | ✗ | | **LayerNorm / RMSNorm** | ✗ | ✗ | ✗ | ✗ | ✗ | ✅ = compiles + infers on Hexagon HTP (and QnnCpu where shown). ✗ = errors at graph build today (see [Known gaps](#known-gaps--a-lot-of-work-left)). **Elementwise/activation ops are the strong suit** (FP32 + INT16/INT8/INT4). MatMul and MHA run FP32 on the NPU. Everything else is still on the to-do pile. --- ## Numbers that matter (from `snapdragon.txt`) ### Determinism — production-grade Repeat-forward drift (same input twice) is **💎 EXACT on every cell that ran**: | Check | Pass | Total | |---|---:|---:| | QNN infer repeat (CPU) | **54** | 54 | | QNN infer repeat (NPU) | **54** | 54 | The NPU gives byte-identical outputs run to run — the single most important property for using it as a real execution target. ### Loom ↔ Hexagon parity | Check | ≤ INDUS | Total | |---|---:|---:| | Loom ↔ Qualcomm parity (**NPU**) | **45** | 54 | | Loom ↔ Qualcomm parity (**CPU**) | 33 | 54 | NPU parity is *tighter* than CPU here because the QnnCpu reference backend is FP32-only and drifts more on the quantized activation ops. Softmax is near-exact (`~1e-6`); MatMul/MHA FP32 sit at 🟨 LOWBIT on HTP (H-DRIFT vs the CPU anchor). ### Speed — HTP wins as layers grow The Hexagon infer floor is ~**0.5 ms** (fixed launch cost), so small tensors lose but **large tiers win big** (Loom ÷ Hexagon, higher = NPU faster): | Tier | Layer | DType | Loom ms | Hexagon ms | Speed | |---|---|---|---:|---:|---:| | large | MatMul | FP32 | 3.582 | 0.510 | **7.0×** | | large | ReLU | INT8 | 3.516 | 0.508 | **6.9×** | | large | Sigmoid | INT4 | 3.598 | 0.531 | **6.8×** | | large | GELU | INT4 | 3.505 | 0.539 | **6.5×** | | large | MHA-MatMul | FP32 | 2.187 | 0.525 | **4.2×** | | medium | MatMul | FP32 | 1.032 | 0.507 | **2.0×** | | small | MatMul | FP32 | ~0 | 0.510 | loses (floor) | Manifest totals: Hexagon beat Loom on **25 / 150** cells, mostly the large tier. `QnnCpu` beat Loom on only **1 / 150** — it's an accuracy anchor, not a speed play. ### One-time compile cost `SyncToAccel` compiles each graph once. HTP graph prepare is the expensive part: | Op | HTP compile (ms) | |---|---:| | Activation ops (ReLU/GELU/…) | ~17–20 | | MatMul (small→large) | ~113 → ~187 | | MHA-MatMul (large) | ~187 | Compile is paid **once per layer/device**; steady inference is the sub-ms numbers above. --- ## How it plugs into Loom (unchanged dispatch model) ```go reg, err := poly.DiscoverQualcommAccel(accel.AccelConfig{ QualcommSO: accel.DefaultQualcommPath(), }) if err != nil { /* no plugin — stay on Loom CPU */ } defer reg.Close() net, _ := poly.BuildNetworkFromJSON(spec) net.Accel = reg net.Layers[0].ExecTarget = accel.ExecQualcommNPU // or ExecQualcommCPU _ = net.SyncToAccel("medium") // compile once + upload weights out, _, _ := poly.ForwardPolymorphic(net, input) ``` ### `ExecTarget` values | Value | Runs on | |---|---| | `accel.ExecLoomCPU` | Default — Go poly CPU | | `accel.ExecQualcommCPU` | QNN `QnnCpu` reference backend (FP32 parity anchor) | | `accel.ExecQualcommNPU` | QNN `QnnHtp` — Hexagon Tensor Processor | ### Accelerator cores | Core | Backend | Data types | Quant | Role | |---|---|---|---|---| | **HTP** (Hexagon) | `QnnHtp` | INT4/8/16, FP16* | needed | Dedicated NPU — the fast path | | **CPU** (Kryo) | `QnnCpu` | FP32 | — | Parity anchor; rejects FP16/INT* graph nodes today | | **GPU** (Adreno) | `QnnGpu` | FP32/FP16/INT* | — | Not wired into the bench yet | \* FP16 is native HTP silicon but currently errors in our graph build — see gaps. --- ## Known gaps — a lot of work left This is honestly *experimental*. What's still broken or missing: | Gap | Detail | |---|---| | **Conv1D / Conv2D** | `ERR` on every dtype — op param encoding (stride/pad tensors) not accepted by `graphAddNode` yet | | **LayerNorm / RMSNorm** | `ERR` — no weight bake + norm op mapping incomplete | | **FP16 everywhere** | `QnnGraph_create`/`graphAddNode` fails; FP16 tensor path needs fixing despite HTP supporting it | | **MatMul/MHA quantized** | Only FP32 compiles; INT4/8/16 MatMul error (weight-tensor / quant-encoding) | | **INT4** | Per-tensor 4-bit weights compile for activations but MatMul INT4 still fails; QNN wants block-wise 4-bit encoding | | **QnnCpu quant** | CPU reference backend is FP32-only — quantized cells only validate against Loom, not a QNN CPU anchor | | **Adreno GPU** | `QnnGpu` backend selectable in the wrapper but untested | | **Training / backward** | Forward-only; backward stays on Loom CPU | | **JSON `exec` field** | No `"exec": "qualcomm-npu"` yet — `ExecTarget` is set programmatically | | **Zero-copy I/O** | Per-hop tensor → `[]byte` → QNN tensor copies | ### Environment robustness - QNN on Windows/Snapdragon usually can't load the signed Hexagon skel and falls back to the **user driver** path — this is normal, inference still works (`Hexagon NPU: available`). The chatter is filtered in Lucy [12] unless `LOOM_QNN_VERBOSE=1`. - `QNN_SDK_ROOT` must be set **machine-wide** for fresh shells / other users; `install_qairt.ps1 -Persist` (elevated) does this. --- ## Build & run Full setup (SDK download, env persistence, llvm-mingw build, webgpu note) lives in [`accel/qualcomm/README.md`](../accel/qualcomm/README.md). Short version: ```powershell cd accel/qualcomm pwsh -File .\install_qairt.ps1 -Persist # runtime + SDK detect + machine env (elevated) . .\setup_env.ps1 .\build_clang.ps1 # llvm-mingw → build/loom_accel_qualcomm.dll cd ..\..\lucy_bloom_rivers go run . # → [12] Snapdragon NPU bridge # [4] medium DispatchLayer suite (fast, ~9 s) # [5] full 15×5×3 matrix (minutes — HTP compiles per graph) ``` Avoid Ctrl+C mid-compile on the full matrix — QNN can fault (`0xc0000005`). --- ## See also - [`accelerators.md`](accelerators.md) — the accel model + Intel NPU bridge - [`accel/qualcomm/README.md`](../accel/qualcomm/README.md) — install / build / env / webgpu - [`numerical_types.md`](numerical_types.md) — Loom's 21 DTypes vs vendor bench dtypes - [`dispatch.md`](dispatch.md) — `DispatchLayer` routing - [`windows_arm64.md`](windows_arm64.md) — Windows on ARM build notes --- ## Apple (Metal GPU / MPSGraph) bridge Source: https://openfluke.com/docs/apple-metal Markdown: https://openfluke.com/docs/apple-metal.md # Apple (Metal GPU / MPSGraph) bridge **Version:** Loom **v0.83.0 "Apple Bridge"** — **experimental** (same maturity bar as the Intel & Qualcomm NPU bridges) **Status:** macOS on Apple silicon · Metal GPU (MPSGraph) + Accelerate/CPU reference · forward-only **Plugin:** `libloom_accel_apple.dylib` (Apple **Metal Performance Shaders Graph** inside) **Lucy menu:** **[13]** — log: `lucy_testing_output/apple.txt` (see [lucy.md](lucy.md)) > This is the Apple mirror of the Intel path in [`accelerators.md`](accelerators.md) and the > Qualcomm path in [`snapdragon_npu.md`](snapdragon_npu.md). It shares the identical > vendor-neutral C ABI (`accel/apple/include/loom_accel.h`), the same `poly/accel` loader, > the same `SyncToAccel` → `DispatchLayer` flow. Only the vendor backend differs: Metal / > MPSGraph instead of OpenVINO or QNN. --- ## TL;DR The Apple plugin is the cleanest of the three vendor bridges to stand up, because there is **no external SDK to vendor** — Metal, MetalPerformanceShaders, and MetalPerformanceShadersGraph ship with macOS. There's no 1.3 GB download, no signed DSP skel, no `LD_LIBRARY_PATH` dance. It exposes **two devices** behind the same C ABI: - **`CPU`** — a portable, deterministic C++ reference forward for all 15 layers. This is the **parity anchor** (it plays the role QnnCpu plays for Qualcomm). - **`GPU`** — a **Metal / MPSGraph** backend for the ops MPSGraph handles well (MatMul, MHA, ReLU, Sigmoid, Softmax, Add, Multiply). Anything it doesn't build silently falls back to the CPU reference, so **every cell still returns a correct value**. The result: **a real Loom → Apple GPU dispatch bridge with byte-perfect run-to-run determinism (180/180 on both devices) and multi-× speedups on large MatMul/MHA on the GPU and on elementwise ops on the CPU.** It is **not** production-complete — Conv and the norms are still CPU-reference-only and slower than Loom's own CPU, and the usual low-bit MatMul drift is present. Treat it exactly like the Intel/Qualcomm bridges: **proven plumbing to build on, not a "flip the GPU on for any network" feature.** --- ## What actually runs today Measured from `apple.txt` (Lucy **[13] → [5]**, the 10-layer DispatchLayer matrix × 6 dtypes × 3 sizes = **180 cells**, `SyncToAccel` once per device). The raw CABI bench (menu **[0]**) additionally covers DepthwiseConv / AvgPool / MaxPool for the full 15-layer plugin surface. | Layer | Metal GPU | Apple CPU ref | Notes | |---|:--:|:--:|---| | **MatMul** | ✅ MPSGraph | ✅ | GPU wins on large tiers | | **MHA-MatMul** | ✅ MPSGraph | ✅ | GPU wins on large tiers | | **ReLU** | ✅ MPSGraph | ✅ | CPU is faster at small/medium | | **Sigmoid** | ✅ MPSGraph | ✅ | CPU is faster at small/medium | | **Softmax** | ✅ MPSGraph | ✅ | GPU floor-bound; CPU near-instant | | **Add / Multiply** | ✅ MPSGraph | ✅ | (raw CABI bench) | | **GELU** | ⚠️ CPU fallback | ✅ | not built on MPSGraph yet | | **Conv1D / Conv2D** | ⚠️ CPU fallback | ✅ | CPU reference; slower than Loom CPU | | **DepthwiseConv / AvgPool / MaxPool** | ⚠️ CPU fallback | ✅ | (raw CABI bench) | | **LayerNorm / RMSNorm** | ⚠️ CPU fallback | ✅ | parity broken (no Loom weights baked) | ✅ = compiles + infers on that device. ⚠️ CPU fallback = the GPU device transparently runs the CPU reference (correct result, no GPU speedup). **Every dtype runs on every layer** — the matrix is **FP32 / FP16 / BF16 / INT16 / INT8 / INT4**. Compute is FP32 internally; the dtype only changes the wire byte layout (FP16/BF16 are 2 bytes, the rest hand over FP32 values, matching `poly/accel_intel.go`). **BF16 is the Apple-native addition** to the bridge — Apple silicon has native bfloat16. --- ## Numbers that matter (from `apple.txt`) ### Determinism — production-grade Repeat-forward drift (same input twice) is **💎 EXACT on every cell, both devices**: | Check | Pass | Total | |---|---:|---:| | Apple infer repeat (**CPU**) | **180** | 180 | | Apple infer repeat (**GPU**) | **180** | 180 | Byte-identical outputs run to run — the single most important property for using either device as a real execution target. ### Loom ↔ Apple parity | Check | ≤ INDUS | Total | |---|---:|---:| | Loom ↔ Apple parity (**GPU**) | **132** | 180 | | Loom ↔ Apple parity (**CPU**) | 78 | 180 | Full GPU bucket spread: 💎 6 · ✅ 126 · 🟨 36 (LOWBIT) · ❌ 12 (BROKE). Full CPU bucket spread: 💎 6 · ✅ 72 · 🟨 54 · 🟤 27 (H-DRIFT) · ❌ 21. The `≤ INDUS` count is **higher on the GPU** than the CPU because the GPU target carries a looser parity tolerance (GPU math legitimately reorders reductions) — the raw drift magnitudes for the two devices are nearly identical. So "GPU parity looks better" is a tolerance-bucket effect, not the GPU being more precise than the CPU reference. **Parity highlights:** - **Softmax** is effectively exact (`~1e-9` FP32, `~1e-6` FP16/BF16) — the standout. - **MatMul / MHA / Conv** sit at 🟨 LOWBIT → 🟤 H-DRIFT as tiers grow (FP32/FP16/BF16 accumulation order vs Loom's CPU matmul). - **LayerNorm / RMSNorm** are ❌ BROKE at ~1.8 for FP32/BF16 — the CPU reference fills its own constant scale/bias instead of Loom's weights (identical root cause to the Intel bridge norms). - **INT8** MatMul/Conv/MHA drift is large (3 → 31) and goes ❌ BROKE on the large tier — the same Loom-dequant-matmul vs plugin-FP32 mismatch seen on Intel/Qualcomm. ### Speed — GPU for big MAC, CPU for elementwise Two clear winners depending on op class (Spd = Loom ÷ Apple, higher = Apple faster): **Metal GPU wins on large MatMul / MHA** (the GPU has a ~0.5 ms launch floor, so it only pays off once the tensor is big enough): | Tier | Layer | DType | Loom ms | Metal GPU ms | Speed | |---|---|---|---:|---:|---:| | large | MHA-MatMul | FP16 | 3.079 | 0.571 | **5.4×** | | large | MHA-MatMul | INT8 | 2.096 | 0.399 | **5.3×** | | large | MatMul | FP16 | 2.613 | 0.562 | **4.6×** | | large | MatMul | INT8 | 2.322 | 0.571 | **4.1×** | | large | MatMul | BF16 | 1.603 | 0.598 | **2.7×** | **Apple CPU reference wins big on elementwise** (tight native loop, no launch cost): | Tier | Layer | DType | Loom ms | Apple CPU ms | Speed | |---|---|---|---:|---:|---:| | medium | ReLU | INT4 | 0.903 | 0.010 | **90×** | | large | ReLU | INT4 | 1.880 | 0.020 | **94×** | | large | GELU | INT4 | 2.395 | 0.029 | **83×** | | large | Sigmoid | INT4 | 2.179 | 0.027 | **81×** | | medium | GELU | INT8 | 0.869 | 0.017 | **51×** | **Where both Apple devices lose:** - **Small tier, GPU** — the ~0.5 ms floor swamps sub-0.01 ms compute (`0.02×` on MatMul/ReLU). - **Conv1D / Conv2D, every tier** — both devices run the CPU reference conv, which is ~0.24–0.27× Loom's own optimized CPU conv. Don't offload Conv yet. - **Norms** — tiny ops, CPU-reference-only, ~0.3–0.6× (and parity broken anyway). ### One-time compile cost `SyncToAccel` compiles each layer once. The CPU reference is ~free; the **MPSGraph build** is the cost on the GPU device — and it is cheap compared to Hexagon/OpenVINO: | Device | Compile (ms) | |---|---:| | Apple CPU reference | ~0.00 | | Metal GPU (MPSGraph MatMul/act) | ~0.4 – 5.8 | Compile is paid **once per layer/device**; steady inference is the sub-ms numbers above. --- ## How it plugs into Loom (unchanged dispatch model) ```go reg, err := poly.DiscoverAppleAccel(accel.AccelConfig{ AppleSO: accel.DefaultApplePath(), }) if err != nil { /* no plugin — stay on Loom CPU */ } defer reg.Close() net, _ := poly.BuildNetworkFromJSON(spec) net.Accel = reg net.Layers[0].ExecTarget = accel.ExecAppleGPU // or ExecAppleCPU _ = net.SyncToAccel("medium") // compile once + upload weights out, _, _ := poly.ForwardPolymorphic(net, input) ``` ### `ExecTarget` values | Value | Runs on | |---|---| | `accel.ExecLoomCPU` | Default — Go poly CPU | | `accel.ExecAppleCPU` | Apple plugin CPU reference (deterministic parity anchor) | | `accel.ExecAppleGPU` | Metal / MPSGraph (falls back to the CPU reference per-op when MPSGraph can't build) | ### Accelerator devices | Device | Backend | Data types | Role | |---|---|---|---| | **GPU** | Metal / MPSGraph | FP32/FP16/BF16 + INT* (as fp32 wire) | The fast path for large MatMul / MHA | | **CPU** | portable C++ reference | all 6 bench dtypes | Parity anchor + fallback for unbuilt GPU ops | | **ANE** (Neural Engine) | — | — | Not wired — reachable only indirectly via Core ML (future) | --- ## Known gaps — what's left Honestly *experimental*. What's still missing or weak: | Gap | Detail | |---|---| | **Conv1D / Conv2D on GPU** | Not built on MPSGraph — both devices run the CPU reference, which is slower than Loom's own CPU conv. Biggest speed opportunity left. | | **GELU on GPU** | Falls back to CPU reference; MPSGraph has the ops, just not wired. | | **LayerNorm / RMSNorm parity** | ❌ BROKE — CPU reference uses internal constant scale/bias instead of Loom's uploaded weights. Needs weight bake (same fix as Intel). | | **INT8 MAC drift** | MatMul/Conv/MHA INT8 drift is large and BROKE on large tiers (Loom dequant matmul vs plugin FP32). | | **FP16/BF16 precision** | Compute is FP32 internally; FP16/BF16 only change the wire layout, so no true half-precision GPU math yet. | | **ANE** | Apple Neural Engine is not a Metal device; reaching it needs a Core ML path (future plugin mode). | | **Training / backward** | Forward-only; backward stays on Loom CPU. | | **JSON `exec` field** | No `"exec": "apple-gpu"` yet — `ExecTarget` is set programmatically. | | **Zero-copy I/O** | Per-hop tensor → `[]byte` → Metal buffer copies. | --- ## When to offload | Target | Offload | Skip | |---|---|---| | **Apple GPU** | **large** MatMul / MHA-MatMul (2.4–5.4×) | small tensors (0.5 ms floor); Conv (CPU fallback); norms | | **Apple CPU** | elementwise **ReLU / GELU / Sigmoid** at medium+ (16–94×); small MatMul | large MatMul/MHA (single-threaded ref, 0.1–0.3×); Conv | | **Either** | many steady forwards (compile paid once) | one-shot micro-ops | Rule of thumb: **big matmuls → Metal GPU; heavy elementwise → Apple CPU; convolutions → keep on Loom CPU** until the GPU conv path lands. --- ## Build & run No SDK download — just Xcode command-line tools (for the Metal frameworks). Full notes in [`accel/apple/README.md`](../accel/apple/README.md). Short version: ```bash cd accel/apple ./build.sh # CMake → build/libloom_accel_apple.dylib cd ../../lucy_bloom_rivers CGO_ENABLED=1 go run . # → [13] Apple GPU bridge # [4] medium DispatchLayer suite (fast) # [5] full 10×6×3 matrix (the apple.txt tables above) # [0] raw CABI matrix (all 15 layers, direct plugin) ``` `accel.DefaultApplePath()` walks up from cwd for `accel/apple/build/libloom_accel_apple.dylib`, or set `LOOM_ACCEL_APPLE_DYLIB`. --- ## See also - [`accelerators.md`](accelerators.md) — the accel model + Intel NPU bridge + [vendor support matrix](accelerators.md#vendor-support-matrix--shipped-vs-planned) - [`snapdragon_npu.md`](snapdragon_npu.md) — the Qualcomm/Hexagon mirror - [`accel/apple/README.md`](../accel/apple/README.md) — build / dtypes / layout - [`numerical_types.md`](numerical_types.md) — Loom's 21 DTypes vs vendor bench dtypes - [`dispatch.md`](dispatch.md) — `DispatchLayer` routing --- ## BitNet CPU Ternary Path Source: https://openfluke.com/docs/bitnet-cpu Markdown: https://openfluke.com/docs/bitnet-cpu.md # BitNet CPU Ternary Path `poly` has an explicit CPU path for BitNet b1.58-style ternary weights. The target dtype is `DTypeTernary` (`{-1, 0, +1}`), not `DTypeBinary` (`{-1, +1}`). ## What Is Supported - `WeightStore.MorphBitNetTernary()` converts FP32 master weights using the BitNet b1.58 absmean scale used by HF `utils_quant.py`: ```text scale = mean(abs(weights)) q = round(clamp(weight / scale, -1, +1)) ``` - `MorphLayerBitNetTernary()` and `MorphNetworkBitNetTernary()` provide public conversion helpers. The network helper leaves normalization layers in their existing dtype. - `MorphLayerBitNetNativeTernary()` and `MorphNetworkBitNetNativeTernary()` are for BitNet-trained checkpoints. They replace projection weights with raw `{-1, 0, +1}` execution weights so the packed CPU path does not apply a PTQ dequant scale. - When `VolumetricNetwork.UseExactDType` is true and the layer dtype is `DTypeTernary`, CPU inference uses packed 2-bit ternary matrix-vector kernels for: - Dense layers - MHA Q/K/V/O projections - SwiGLU gate/up/down projections - Transformer `lm_head` when it is a separate output head If `lm_head` is tied to the embedding table, the output head stays FP32. This matches common decoder layouts where token embeddings are not BitLinear weights. The packed kernel stores 16 ternary weights per `uint32` and computes dot products with add/subtract/skip logic. Inputs are quantized per token to int8: ```text activation_scale = 127 / max(abs(input)) xq = clamp(round(input * activation_scale), -128, 127) out = dot(xq, wq) * weight_absmean / activation_scale ``` For BitNet-style transformer blocks, the CPU path also applies the model's learned inner RMSNorm after attention and after the SwiGLU gate/up product, matching the HF `modeling_bitnet.py` layout. The `1bitLLM/bitnet_b1_58-*` checkpoints are base models, not instruction-tuned assistants. Lucy uses the tokenizer-native LLaMA-style `[INST] ... [/INST]` wrapper for these models, but the output can still look like web-text completion rather than a reliable chat answer. Lucy also exposes ordinary FP32-to-ternary PTQ for non-BitNet CPU models as an explicit experimental option. This is technically possible, but it is not equivalent to BitNet training and may produce low-quality or broken text. For CPU speed, packed ternary projections quantize each activation row once and reuse the int8 row for sibling projections such as Q/K/V and gate/up. The tied FP32 LM head remains exact but is parallelized across vocabulary rows. The hot CPU kernel is row-aligned and word tiled: each row stores `ceil(cols / 16)` packed `uint32` words, then the dot loop consumes one word at a time with an unrolled, branchless 16-weight ternary decode. Large matrices are split across output-row ranges using `GOMAXPROCS`. ### AVX2 SIMD ternary MAD (opt-in) When Plan 9 SIMD forward is enabled (`net.SetSimdForward(true)` / `SetSimdForwardRecursive`), the packed ternary matvec switches from the scalar word-decode to an AVX2 **MAD kernel** mirroring Microsoft's `ggml-bitnet-mad.cpp`: - Weights are unpacked **once** into a cached byte-per-weight code buffer (`{0,1,2}`, row-padded to a multiple of 32) — no on-the-fly unpack per token. - The inner loop uses `VPMADDUBSW` (unsigned code × signed int8 activation) → `VPMADDWD` → `VPADDD`, 32 weights per instruction. - Using `weight = code − 1`, the row dot is `Σ(code·act) − Σ(act)`; the `Σ(act)` correction is computed once per activation vector. Output is **bit-identical** to the scalar path (both do exact integer accumulation). Measured BitNet b1.58-2B CPU decode: ~2.6× single-core vs scalar. The cached code buffer costs ~1 byte/weight of extra RAM. See [`simd.md`](simd.md) for the full SIMD system and coverage. Lucy loads BitNet checkpoints block-by-block for CPU inference: it decodes only global tensors first, then decodes one transformer block, packs Dense/MHA/SwiGLU BitLinear projections, releases that block's FP32 tensors, and moves to the next block. Embeddings, tied `lm_head`, final norm, and learned inner norm scales remain FP32 because the HF checkpoint uses them that way. ## Important Limits This is a fast CPU storage/execution path, not a guarantee that any arbitrary FP32 model will remain good after 1.58-bit post-training quantization. The Microsoft BitNet b1.58 quality results assume BitNet-style trained checkpoints, 8-bit activations, and specialized CPU kernels. Plain FP32-to-ternary conversion is useful for experiments, but it should be treated as lossy. The scalar path is pure Go (the correctness and integration baseline). An x86 **AVX2** ternary MAD kernel is available as an opt-in SIMD path (see above and [`simd.md`](simd.md)); ARM NEON and AVX512 ternary kernels are not yet written (arm64 falls back to the scalar Go dot). ## Benchmark Run the focused packed dense benchmark with: ```bash go test ./poly -run '^$' -bench BenchmarkPackedTernaryDenseForward -benchmem ``` Run correctness coverage with: ```bash go test ./poly -run 'BitNet|PackedTernary|TernaryNative' ``` --- ## ENTITY format (`.entity`) Source: https://openfluke.com/docs/entity Markdown: https://openfluke.com/docs/entity.md # ENTITY format (`.entity`) **E**very **N**umerical **T**ype **I**n **N**ative **T**opolog**Y** Native Loom checkpoint files. One `.entity` file = one saved brain: **full volumetric topology + all native-packed weights** in a single binary artifact. Implementation: [`poly/entity.go`](../poly/entity.go), [`poly/entity_file.go`](../poly/entity_file.go), HF convert helpers [`poly/hf_entity_convert.go`](../poly/hf_entity_convert.go) + [`poly/entity_convert_io.go`](../poly/entity_convert_io.go) Validated in Lucy menu **[7] Seven-layer CPU suite** — JSON and `.entity` save/reload run side by side for all 21 dtypes (Lucy `examples/seven_layer/runner.go`). Lucy **[8] ENTITY Talk** converts HF LLMs to `.entity` and runs GPU chat from native checkpoints (Lucy `hf_entity.go`). See [lucy.md](lucy.md). --- ## Why we built this HuggingFace **`.safetensors`** is the right **import** lane for PyTorch/HF checkpoints. Loom uses it for model download and HF decode (`poly/safetensors.go`, SoulGlitch, `LoomCreateLLM`). It is **not** a native Loom checkpoint format. That is not a bug in SafeTensors — Loom simply does more than flat named tensors: | Loom needs | SafeTensors | |:-----------|:------------| | **21 DTypes** with native on-disk packing (Int4 nibbles, Binary 8:1, Ternary, FP4, …) | Fixed HF dtype strings; sub-byte types are awkward; Loom **export** is F32-only | | **Per-layer `Scale`** (quant mapping used at save time) | No standard field | | **Volumetric grid** `(Z, Y, X, L)` per layer | Flat string keys only (`model.layers.0…`) | | **Topology** — parallel branches, sequential stacks, metacognition | Requires separate `config.json`; no recursion | | **Bit-perfect reload** of trained native dtypes | Import path usually decodes to FP32 master | We already had full fidelity in **`persistence.go`** (`SerializeNetwork` / `DeserializeNetwork`) — JSON + Base64 native blobs. That works (Lucy save/reload PASS on all 21 dtypes) and remains the **transparent debug lane**, but it is **large and slow** for shipping brains to phones or edge nodes. **ENTITY** is the native binary path: - Same semantics as JSON persistence (same topology spec, same packing rules) - SafeTensors-*like* wire safety: length-prefixed header + indexed blob section - Raw weight bytes (no Base64) - **Different file extension** so HF tooling does not assume HuggingFace semantics ``` Import: model.safetensors ← HF yarn (read-only in product flow) Native: fluffy.entity ← ENTITY (train, save, reload, ship) Debug: model.json ← JSON persistence (same brain, verbose) ``` --- ## One file = topology + weights Unlike HuggingFace’s split of `config.json` + `model.safetensors`, ENTITY keeps everything together: ``` ┌─────────────────────────────────────────┐ │ Fixed header (magic, version, flags) │ ├─────────────────────────────────────────┤ │ JSON header │ │ • network topology (grid + layers) │ ← PersistenceNetworkSpec │ • blob index (path, offset, dtype…) │ ├─────────────────────────────────────────┤ │ Binary payload │ ← native-packed weight blobs └─────────────────────────────────────────┘ ``` After `LoadEntity`, you get a full `VolumetricNetwork` — grid dimensions, every layer’s type/activation/dtype/config, recursive branches, and quantized weights ready for forward or further training. --- ## The unlock: HF models as native citizens ENTITY is not only a smaller checkpoint format. It is the **bridge** that moves real LLM weights from HuggingFace’s flat tensor world into Loom’s volumetric brain format — the same container Lucy **[7]** uses for 3D grids, parallel branches, and per-layer dtypes. ### Before vs after **Before** — two separate worlds: ``` HF .safetensors → flat tensor names → Poly Talk reads every run ↓ foreign format (no grid, no branches, no per-layer dtype in one file) Lucy [7] seven-layer suite → 2×2×2 grids, remote links, 21 dtypes ↓ synthetic trained brains only ``` **After** — one native lane: ``` HF snapshot → convert once → .entity → VolumetricNetwork + transformer globals ↓ same format as Lucy [7] save/reload chat without HF weights at runtime (Lucy [8] ENTITY Talk) ``` HuggingFace models are no longer guests. They are **`.entity` citizens** — reloadable, trainable, graftable, and eligible for every volumetric feature the stack already implements. ### The arc: simple → native → experimental → full 3D | Stage | What it is | Status | |:------|:-----------|:-------| | **Simple** | Flat HF decoder; Poly Talk loads safetensors each run | ✅ Shipped | | **Native** | HF → `.entity`; Q4 baked for decoder blocks; GPU chat from Lucy-owned checkpoint | ✅ Shipped (Lucy **[8]** ENTITY Talk) | | **Experimental** | Graft, parallel branches, remote links, per-layer dtype mixes on imported LLM weights | 🔓 Unlocked in format + API; product UI not built | | **Full 3D** | LLM blocks as cells in a `(Z,Y,X,L)` grid — experts, hops, evolution around a frozen core | 🔮 Next chapter | Lucy **[7]** proved the volumetric stack on **small trained grids**. Lucy **[8]** brings **real LLM weights** into that same format. The chat path today is still a flat decoder layout; the **container** is already the full brain OS. ### HF import layout today `ImportHFToEntity` ([`hf_import.go`](../poly/hf_import.go)) maps a Llama-style stack into a **1×1×1** grid with four sub-layers per block (pre-norm, MHA, post-norm, SwiGLU): ```go net := NewVolumetricNetwork(1, 1, 1, dims.NumLayers*4) InitHFDecoderBlocks(net, dims) ``` Import is **block-wise** to avoid doubling peak RAM: globals load first, then each transformer block’s safetensors are loaded, copied into `WeightStore.Master`, and the transient map is released before the next block. See [memory_history.md — HF → .entity convert](memory_history.md#hf--entity-convert-import-memory). ENTITY Talk chat uses this linear layout. Nothing in the format prevents expanding to `2×2×N`, parallel experts, or remote links — that is topology editing on a loaded `VolumetricNetwork`, then `SaveEntityTransformer`. ### What the format unlocks | Capability | Supported by format / poly | ENTITY Talk UI today | |:-----------|:----------------------------|:---------------------| | Save / reload / train native state | ✅ | Convert + chat only | | Different dtype per layer in one file | ✅ | Q4 decoder when user picks INT4 at convert | | Parallel branches / MoE-style `filter` gates | ✅ [`parallel.go`](../poly/parallel.go) | ❌ | | Spatial hops (`IsRemoteLink`) | ✅ [`dispatch.md`](dispatch.md) | ❌ | | Graft multiple networks into one parallel layer | ✅ [`grafting.go`](../poly/grafting.go) | ❌ | | NEAT / topology evolution | ✅ [`evolution.md`](evolution.md) | ❌ | | Selective layer load + block-wise GPU upload | ✅ `DeserializeEntityWithOptions` | ✅ block upload prompt | | Block-wise HF → `.entity` import (lower convert RAM) | ✅ `ImportHFCheckpointDir` | ✅ Lucy `[8]` convert | | Block-wise HF → `.entity` encode (mobile-safe convert) | ✅ `ImportHFSaveEntityTransformerBlockwise` | ✅ SoulGlitch / mvp-simulation | | Merge two LLMs with mismatched hidden size / vocab | ❌ shapes must align | ❌ | **Principle:** anything Lucy **[7]** could do to a trained `.entity`, you can now *in principle* do to an imported LLM `.entity` — graft a side branch, add an experimental layer, mix dtypes, evolve topology around a frozen decoder core. Wiring those flows into product UI is separate work; the **format bridge** is the prerequisite, and it exists. ### Example directions (not shipped) ```go // Load two checkpoints, graft parallel branches, save hybrid a, _ := poly.LoadEntityTransformer("lucy_entities/Qwen--Qwen3-0.6B.entity") b, _ := poly.LoadEntity("lucy_testing_output/my_swiglu_Int4.entity") graft, err := poly.GraftNetworksPolymorphic([]*poly.VolumetricNetwork{a.Network, b.Network}, "concat") // … embed graft in a new net topology, SaveEntityTransformer … ``` See [parallel_sequential.md](parallel_sequential.md), [evolution.md](evolution.md), and [quick_reference.md](quick_reference.md#remote-link-spatial-hop) for the underlying APIs. --- ## LLM transformer checkpoints (Lucy [8]) Lucy menu **[8] ENTITY Talk** (Lucy `hf_entity.go`) converts supported HF models (SmolLM2, Qwen, Llama-style) to universal-transformer `.entity` files and runs GPU chat without loading safetensors at runtime. Flow (standard — Lucy `[8]` on desktop): ``` HF cache → ImportHFCheckpointDir (block-wise FP32 import) → SaveEntityTransformer (Q4_0 bake if INT4) → lucy_entities/*.entity → LoadEntityTransformer → chat ``` Flow (low-RAM encode — SoulGlitch / mobile): ``` HF cache → ImportHFSaveEntityTransformerBlockwiseProgress → per-block Q4 bake → streaming payload temp file → writeEntityWireStreaming → *.entity (+ app CHGLUE wrapper) → LoadEntityTransformer → chat ``` See [memory_history.md — HF → `.entity` convert](memory_history.md#hf--entity-convert-import--encode-memory) for the full step list and peak-RAM comparison. ### Random-access `.entity` reads (`EntityFile`) For large checkpoints, `poly/entity_file.go` avoids loading the entire file into RAM. **`ReadAt`** pulls only the header and requested weight blobs. | API | Purpose | |-----|---------| | `OpenEntityFile` / `OpenEntityFileAt` | Open path; optional byte offset for CHGLUE-wrapped loom sections | | `LoadEntityTransformer` | Full transformer checkpoint via blob reads | | `LoadEntityTransformerTopology` | Topology + globals only (`SkipLayerWeights`) | | `LoadNetworkLayerWeights(net, indices)` | Hydrate specific top-level layer indices into an existing `VolumetricNetwork` | | `LoadEntityTransformerFromFile` | Convenience: open, load, close | **C-ABI:** `LoomOpenEntityFile`, `LoomLoadEntityTransformerFromFile`, `LoomLoadEntityTransformerTopology`, `LoomLoadNetworkLayerWeights`, `LoomPrepareEntityTransformerLayerIndices`. Use this for **selective layer load** (e.g. block-by-block GPU upload) without holding every decoder block in host RAM at once. See also the capability row *Selective layer load* in the table above. ### Q4 on disk vs GPU (v1) When the user selects **Q4 (INT4)** at convert time, implementation lives in [`entity_q4.go`](../poly/entity_q4.go): | Weight region | On disk (`.entity`) | On GPU at chat | |:--------------|:--------------------|:---------------| | Decoder MHA + SwiGLU | **Q4_0** blocks (baked; no re-quant on load) | **Q4_0** via cached `Q4_0Packed` / `uploadQ4_0Cached` | | RMSNorm, MHA Q/K norms, final norm | FP32 | FP32 | | Embeddings, LM head | FP32 | FP32 | RMSNorm stays FP32 intentionally — quantizing norm gamma corrupts the forward pass. Globals stay FP32 in v1; that is why large-vocab **untied** models (e.g. Qwen3) may show little disk shrink vs BF16 safetensors even though decoder Q4 is real (GPU weights ~1450 MB vs ~4550 MB FP32 for Qwen3-0.6B). Model-specific metadata persisted in the header includes expanded `query_dim` / `kv_dim` (Qwen-style MHA), MHA `q_norm` / `k_norm` auxiliary blobs, and `lm_head_tied`. Tokenizer and chat template still come from the HF snapshot; only **weights** move native. ### HF → `.entity` convert (memory) Two **llama-style** convert lanes — see [memory_history.md](memory_history.md#hf--entity-convert-import--encode-memory) for diagrams and peak-RAM tables. | Model kind | Standard lane | Low-RAM lane | |:-----------|:--------------|:-------------| | Llama-style (Qwen, SmolLM2, …) | `ImportHFCheckpointDir` → `SaveEntityTransformer` | `ImportHFSaveEntityTransformerBlockwise(Progress)` | | BitNet | `ImportHFBitNetCheckpointDir` → `SaveEntityTransformer` | (same — already packed per block) | **Standard lane** (Lucy `[8]` `convertEntityEntry`): block-wise safetensor import, then **full FP32 network** in RAM while `SaveEntityTransformer` bakes Q4 and builds the file buffer. **Low-RAM lane** ([`hf_entity_convert.go`](../poly/hf_entity_convert.go), [`entity_convert_io.go`](../poly/entity_convert_io.go)): 1. Encode **globals** to a **payload temp file**; release FP32 globals from RAM 2. For each block: import safetensors → `collectEntityWeightBlobsAcc` (Q4_0 if INT4) → `releaseEntityConvertLayerWeights` → GC 3. `writeEntityWireStreaming` — header + blob index + `io.Copy` payload (no full-file `[]byte`) **Quant policy (both lanes):** Q4 bakes **MHA + SwiGLU** only; **RMSNorm**, **q_norm/k_norm**, **embeddings**, **lm_head**, **final_norm** stay **FP32**. Header `lm_head_tied` uses `entityLMHeadTied()` — same as `SaveEntityTransformer` (separate `lm_head` blob when HF provides distinct slices). **Lucy terminal signature** (standard lane only): three global `✓ Loaded …` lines, then one `✅ Finished loading weights with prefixes.` **per block** (28× for Qwen3-0.6B). The low-RAM lane uses `HFEntityConvertProgress` instead (`packed block N/M`). **SoulGlitch:** wraps the loom `.entity` in a CHGLUE standalone file (tokenizer embedded); convert progress appears in the app task UI, not Lucy logs. ### GPU load and memory diagnostics After `LoadEntityTransformer`, weights live on CPU until GPU setup runs. Lucy **[8]** uses the same path as Poly Talk **[1]**: 1. `setupTransformerForInference` (Lucy `inference_setup.go`) — optional block-by-block decoder upload with `ReleaseInferenceHostWeights()` per block 2. `SyncGlobalWeightsToGPUSequential()` — embeddings → LM head → final norm, releasing CPU after each 3. GPU warmup + `ReleaseInferenceHostWeights()` + `GC` When **Measure memory during GPU load** is enabled, Lucy prints a terminal chart and diagnosis via `poly.GlobalMemoryHistory` (see [memory_history.md](memory_history.md)). Welvet **`LoomCreateLLM`** (safetensors snapshot dir) implements the same GPU policy in `welvet/cabi/llm_ext.go`. Entity-only C ABI exports (`LoomLoadEntityTransformerAs`, `LoomBuildTransformerFromEntity`) **deserialize and build the transformer on CPU only** — the app must run the Lucy GPU sequence separately until a dedicated entity+GPU export exists. #### GPU load after entity deserialize Minimum inference GPU setup (Go): ```go tr.Network.InitWGPU() for li := 0; li < numLayers; li++ { base := li * 4 for j := 0; j < 4; j++ { layer := &tr.Network.Layers[base+j] _ = layer.SyncToGPU() layer.ReleaseInferenceHostWeights() } } _ = tr.SyncGlobalWeightsToGPUSequential() _, _ = tr.ForwardTokenIDsWGPU([]uint32{0}, nil, true, true) tr.Reset() tr.ReleaseInferenceHostWeights() ``` Do **not** use bulk `tr.SyncToGPU()` alone after a full entity load on memory-constrained devices — that was the ~2.6 GB overlap bug fixed by sequential globals. --- ## Name and identity | | | |---|---| | **Format** | ENTITY | | **Expansion** | **E**very **N**umerical **T**ype **I**n **N**ative **T**opolog**Y** | | **Extension** | `.entity` | | **Magic** | `ENTITY\0\0` (8 bytes) | | **Format version** | `1` (v1, implemented) | --- ## Wire layout (v1) ``` Offset Size Content ────── ──── ─────── 0 8 magic "ENTITY\0\0" 8 2 u16 format_version (= 1) 10 2 u16 flags (reserved; 0 today) 12 8 u64 header_byte_length (LE) 20 N header JSON (see below) 20+N … native-packed weight blobs (contiguous) ``` ### Header JSON The header is one JSON object: ```json { "format_version": 1, "network": { /* PersistenceNetworkSpec — topology only, no weight strings */ }, "transformer": { "architecture": "llama_style_decoder", "hidden_size": 2048, "vocab_size": 32000, "lm_head_tied": true, "has_final_norm": true, "dims": { "num_layers": 24, "num_heads": 32, ... } }, "blobs": [ { "path": "layers.0", "offset": 0, "length": 1234, "dtype": "INT4", "scale": 0.01, "native": true } ] } ``` | Field | Role | |:------|:-----| | `network` | Same shape as [`PersistenceNetworkSpec`](../poly/persistence.go): `depth`, `rows`, `cols`, `layers_per_cell`, and every `PersistenceLayerSpec` (type, activation, dtype, z/y/x/l, MHA/CNN dims, parallel/sequential recursion). **No** `weights` Base64 strings — those live in the blob section. | | `transformer` | **Optional** universal-transformer add-on. When present, global causal-LM weights live outside `net.Layers`: embeddings, LM head, final RMSNorm. Used by `ImportHFToEntity` for SmolLM2, Qwen, Llama-style decoders. Tokenizer/chat template still come from the HF snapshot (or your app). | | `blobs[]` | Index into the payload. Each entry points at one weight store (main layer, nested branch, or transformer global). | **Blob paths** mirror the in-memory tree: | Path example | Weight store | |:-------------|:---------------| | `layers.0` | Top-level layer index 0 | | `layers.3.sequential_layers.1` | Nested sequential sub-layer | | `layers.2.parallel_branches.0` | Parallel branch | | `layers.5.meta_observed_layer` | Metacognition observed layer | | `transformer.embeddings` | Token embedding matrix (FP32 blob) | | `transformer.lm_head` | Output projection (omitted when `lm_head_tied`) | | `transformer.final_norm` | Pre-head RMSNorm gamma (when `has_final_norm`) | Each blob carries its own **`dtype`**, **`scale`**, and **`native`** flag — so a single checkpoint can hold **different numerical types per layer** (e.g. layer 0 Int4, layer 12 BFloat16, layer 40 Binary). ### Weight blobs Payload bytes use the **same bit-packing** as JSON persistence: - Implemented via `EncodeNativeWeightsRaw` / `DecodeNativeWeightsRaw` in [`persistence.go`](../poly/persistence.go) - Documented in [serialization.md](serialization.md#the-bit-packing-system) No Base64. No FP32-only export constraint (unlike `SaveSafetensors`). --- ## API (`poly/entity.go`) | Function | Purpose | |:---------|:--------| | `SerializeEntity(net)` | Network → `.entity` bytes | | `DeserializeEntity(data)` | Bytes → full network (topology + all weights) | | `DeserializeEntityWithOptions(data, opts)` | Selective weight load (`EntityLoadOptions.LayerIndices`) | | `DeserializeEntityLayer(data, layerIndex)` | Topology + one top-level layer’s weights | | `SaveEntity(path, net)` / `LoadEntity(path)` | File I/O | | `SerializeEntityTransformer(et)` / `DeserializeEntityTransformer(data)` | Universal transformer: decoder + embeddings/LM head/final norm | | `SaveEntityTransformer` / `LoadEntityTransformer` / `LoadEntityTransformerAs[T]` | File I/O + `NewTransformer` wiring | | `ImportHFToEntity(modelDir, path, opts)` | HF snapshot → universal `.entity` (standard lane) ([`hf_import.go`](../poly/hf_import.go)) | | `ImportHFSaveEntityTransformerBlockwise(modelDir, path, weightDType)` | Low-RAM convert: block-wise import **and** encode ([`hf_entity_convert.go`](../poly/hf_entity_convert.go)) | | `ImportHFSaveEntityTransformerBlockwiseProgress(..., progress)` | Same + `HFEntityConvertProgress` callback (`packed block N/M`) | | `ParseEntityHeader(data)` | Header only (no weight decode; mmap-friendly planning) | | `LayerPersistenceFromEntity(data, layerIndex)` | Raw blob + scale + native for one layer (parity checks) | | `EntityBlobBytes(data, blobIndex)` | Raw bytes for blob `i` without dtype decode | **Hub model:** load any format → `VolumetricNetwork` → save as JSON, `.entity`, or (lossy) safetensors F32 export. ```go net, err := poly.LoadEntity("brain.entity") // … inference or re-quantize per layer (Morph) … poly.SaveEntity("brain-v2.entity", net) jsonWire, _ := poly.SerializeNetwork(net) // still valid debug export ``` --- ## Size vs JSON — observed compression (Lucy [7]) Source: Lucy `lucy_testing_output/seven_layer.txt` — full **[7] Seven-layer CPU suite** run (10 layer types × up to three grids × 21 dtypes). Checkpoints are written after MC training as `tag_DType.json` and `tag_DType.entity`. ### Headline numbers | Metric | Result | |:-------|:-------| | **Runs compared** | 546 dtype×suite rows (26 memory tables) | | **Save/reload** | `json=PASS entity=PASS` on all 546 trained reload checks; `entity=FAIL`: 0 | | **Average disk saving** | `.entity` is **~27.6% smaller** than JSON (typical band **25–28%**) | | **Runtime heap** | Unchanged — savings are **on-disk only** (same trained-native weight RAM in the log’s Weights column) | ENTITY v1 removes **Base64 weight strings** and **pretty-printed JSON weights**. It does **not** re-quantize. The header is still **full topology JSON**, so this is not safetensors-class compression. ### Sample checkpoints (trained, after MC train) **SwiGLU 2×2×2** (8 cells × 7 layers = **56-layer stack**) — the grid from the first live ENTITY comparison: | DType | JSON ckpt | `.entity` ckpt | Saving | |:------|----------:|---------------:|-------:| | Float64 | 496.79 KiB | 372.49 KiB | 25% | | Float32 | 258.17 KiB | 193.35 KiB | 25% | | Int4 | 49.92 KiB | 36.90 KiB | 26% | | Binary | 27.66 KiB | 20.29 KiB | 27% | **Dense 1×1×1** (7-layer pyramid stack): | DType | JSON ckpt | `.entity` ckpt | Saving | |:------|----------:|---------------:|-------:| | Float32 | 57.95 KiB | 43.47 KiB | 25% | | Int4 | 9.53 KiB | 7.12 KiB | 25% | | Binary | 4.35 KiB | 3.24 KiB | 26% | **Dense 3×3×3** (27 cells × 7 = **189-layer stack**): | DType | JSON ckpt | `.entity` ckpt | Saving | |:------|----------:|---------------:|-------:| | Float32 | 83.90 KiB | 60.95 KiB | 27% | | Int4 | 69.93 KiB | 49.68 KiB | 29% | | Binary | 68.61 KiB | 49.09 KiB | 28% | Across **all 21 dtypes** on SwiGLU 2×2×2, the ENTITY/JSON ratio stays in a **25.0–26.6%** band — the saving is almost entirely **Base64 removal**, not dtype-specific magic. ### Three things the log teaches **1. ENTITY vs JSON ≈ fixed ~25% discount, not 10×** The ratio is stable because both formats carry the same topology JSON header and the same native weight bits; only the weight *encoding in the file* changes (Base64 strings → raw blob section). **2. Quant dtype still dominates absolute file size** Same topology, different dtype — SwiGLU 2×2×2: | DType | JSON | `.entity` | |:------|-----:|----------:| | Float64 | 497 KiB | 372 KiB | | Int4 | 50 KiB | 37 KiB | Int4 JSON is ~**10%** of Float64 JSON on the same brain. Picking Int4/Binary matters far more than picking `.entity` over `.json`. **3. Topology overhead grows with grid size; ENTITY shrinks the gap** SwiGLU 2×2×2 **Float32** breakdown (from the log’s Weights vs checkpoint columns): | Component | Size | |:----------|-----:| | Trained-native weights in RAM | 178.50 KiB | | JSON checkpoint | 258.17 KiB (+**80** KiB overhead ≈ **31%** of file) | | `.entity` checkpoint | 193.35 KiB (+**15** KiB overhead ≈ **8%** of file) | On **Dense 3×3×3 Float32**, trained-native weights are only **~12 KiB** in RAM but the JSON checkpoint is **~84 KiB** — topology metadata dominates. ENTITY drops that to **~61 KiB** (~27% saving), but the file is still mostly header, not weights. **Residual 3×3×3** is an extreme case in the log: **~42%** smaller `.entity` vs JSON when per-layer weight blobs are tiny relative to the 189-layer spec. ### Where to read it in the log Each layer-type × grid block ends with a **memory & weight footprint** table: ```text | DType | Heap | Sys | Heap+train | Weights | JSON ckpt | .entity ckpt | ``` Pass lines also print both sizes inline, e.g. `json=496.79 KiB entity=372.49 KiB` on SwiGLU 2×2×2 Float64. ### Why not safetensors-small? 1. **Weights** are already at the bit-width floor for each dtype (Int4 nibbles, Binary 8:1, …). 2. **Topology JSON** is a large fixed cost on big grids (56–189 layer specs with type, dtype, dims, z/y/x/l). 3. **SafeTensors** omits that graph entirely — flat tensor names only. See [Future: smaller files with full topology](#future-smaller-files-with-full-topology) for the planned binary topology + optional zstd path. --- ## Comparison to SafeTensors | | SafeTensors | ENTITY v1 | JSON persistence | |--|-------------|-----------|------------------| | Weights on disk | Raw HF dtypes | Raw Loom native packing | Base64 in JSON | | Topology in same file | ❌ | ✅ (JSON header) | ✅ | | Per-layer Loom dtype + Scale | ❌ | ✅ | ✅ | | Volumetric (Z,Y,X,L) | ❌ | ✅ | ✅ | | Parallel / sequential tree | ❌ | ✅ | ✅ | | Typical header size | Tiny | Large on big grids | Largest (includes Base64 weights) | SafeTensors wins on **flat LLM weight dumps**. ENTITY wins on **native Loom brains** you trained and need to reload exactly. --- ## Idempotency `SerializeEntity` → `DeserializeEntity` → `SerializeEntity` yields **identical bytes** for a given network state (tested in `poly/tests/entity_test.go`). Topology fields are canonicalized on save (e.g. default `seq_length` omitted for non-sequence layers) so reload does not inflate the header. --- ## Validation | Suite | What it checks | |:------|:---------------| | Lucy **[7]** (`seven_layer/runner.go`) | Before/after train: JSON **and** `.entity` save/reload PASS; memory table shows both checkpoint sizes | | Lucy **[8]** (Lucy `hf_entity.go`) | HF cache → Q4 `.entity` convert → GPU ENTITY Talk; SmolLM2 parity with Poly Talk; Qwen load + Q4 GPU path | | `poly/tests/entity_test.go` | Round-trip, idempotent bytes, selective layer load, Q4_0 blob round-trip for transformers | Checkpoints land in Lucy `lucy_testing_output/` as `tag_DType.json` and `tag_DType.entity`. Full-run numbers and compression observations: [entity.md — observed compression](entity.md#size-vs-json--observed-compression-lucy-7) (from `seven_layer.txt`). --- ## Relationship to other I/O | File | Role | |:-----|:-----| | [`safetensors.go`](../poly/safetensors.go) | Read HF `.safetensors`; `SaveSafetensors` is F32-only export | | [`persistence.go`](../poly/persistence.go) | JSON save/load — semantic reference for ENTITY topology and packing | | [`entity.go`](../poly/entity.go) | Native `.entity` binary save/load | | [`serialization.go`](../poly/serialization.go) | Architecture-only JSON (`BuildNetworkFromJSON`) — random init, no trained weights | | [`universal_loader.go`](../poly/universal_loader.go) | Auto-detect from safetensors shapes — import only | --- ## Future: smaller files with full topology ENTITY v1 prioritizes **correctness and debuggability**. Planned v2+ improvements (same full topology, smaller wire): 1. **Binary topology section** — string tables, dtype/layer-type enums, grid-implied `(z,y,x,l)` where regular 2. **Compact blob index** — fixed records (`node_id`, `u8 dtype`, `f32 scale`, offsets) instead of JSON path strings 3. **Optional zstd** (lossless) on header/index/payload via `flags` bits 4. **`ConvertSafetensorsToEntity`** — import HF weights into a Loom topology wrapper 5. **Welvet C-ABI** — `LoomSaveEntity`, `LoomLoadEntity`; loaders accept `.entity` or `.safetensors` 6. **SoulGlitch** — prefer `.entity` for on-device trained saves; keep HF download as `.safetensors` Weights stay on `EncodeNativeWeightsRaw`; the big disk wins are in **topology + index**, not re-quantizing weights. --- ## See also - [serialization.md](serialization.md) — JSON persistence, bit-packing, SafeTensors import, three save paths - [transformer.md](transformer.md) — MHA, SwiGLU, HF decoder layout; links back here for native checkpoints - [parallel_sequential.md](parallel_sequential.md) — parallel branches and combine modes (graft targets) - [evolution.md](evolution.md) — NEAT, remote links, topology mutation - [numerical_types.md](numerical_types.md) — 21 DTypes - [quantization.md](quantization.md) — Scale, Morph, native packing, Q4_0 - [testing_and_validation.md](testing_and_validation.md) — Lucy [7] logs and tables - [memory_history.md](memory_history.md) — GPU load timeline, block upload, sequential globals (Lucy [8]) - [bedrock_validation.md](bedrock_validation.md) — seven-layer CPU suite overview --- ## Planet Bridging Source: https://openfluke.com/docs/planetbridging Markdown: https://openfluke.com/docs/planetbridging.md # Planet Bridging **Universal bridging between AI engines through Loom.** Planet Bridging is the project that maps how models move between **AI runtimes** (“planets”) and **Loom’s volumetric runtime**. Each major framework — PyTorch, TensorFlow, JAX, ONNX Runtime, llama.cpp, CoreML, and others — speaks its own formats, operator dialects, and execution models. Models do not travel freely; they get converted, lose fidelity, or stay locked to one stack. Planet Bridging exists so weights and topology can flow **into Loom** today (live stream → `.entity` → Loom infer), and **out of Loom** tomorrow (export to hub formats → any inference engine). Implementation lives in the sibling repo [`planetbridging/`](../planetbridging/). The **PyPI package** (`planetbridging`, currently **v0.7.3**) is the end-user lane: `pip install` + bundled `loom-stream` — no git clone, no `go build`, no compare-host HTTP. Related Loom docs: [entity.md](entity.md) (what `.entity` is), [serialization.md](serialization.md) (JSON + ENTITY persistence), [bedrock_validation.md](bedrock_validation.md) (Lucy seven-layer CPU gate inside Loom), [layers.md](layers.md) (volumetric layer types). --- ## What it means ### Planets and the hub Think of the AI ecosystem as a solar system: | Term | Meaning | |:-----|:--------| | **Planet** | A training or inference runtime (PyTorch, TensorFlow, JAX, ORT, llama.cpp, …) | | **Hub format** | Interchange layer between planets (Safetensors, ONNX, GGUF) | | **Loom** | Deterministic volumetric DNVM at the center — native `.entity` checkpoints, 21 dtypes, pure Go CPU + WebGPU | Planet Bridging is **not** “reimplement every engine.” It is **layer-aware translation**: map foreign ops to Loom’s `VolumetricLayer` types (Dense, CNN1/2/3, MHA, LSTM, …), stream weights in a canonical JSON contract, and build a Loom network that forward-matches the native planet on shared fixtures. ### Two halves of the bridge | Direction | Status (2026) | What it means for you | |:----------|:--------------|:----------------------| | **Planets → Loom** | ✅ **Complete** (compare-host v0.5.0; PyPI package v0.7.x) | Train or load in PyTorch/TF/JAX → stream live weights → `.entity` → Loom infer matches native | | **Loom → planets** | ⬜ **v1.0 target** | Export Loom brains → Safetensors / ONNX / GGUF → ORT, llama.cpp, CoreML, … | | **File import (no live Python)** | ⬜ **v1.x** | Drop a `.safetensors` / `.onnx` / `.keras` on disk and ingest without a running planet | Versioning intuition: **0.5 + 0.5 ≈ 1.0** — first half is *into* Loom; second half is *out of* Loom. --- ## How it works (planets → Loom) This is **live weight streaming**, not a classic ONNX → Safetensors → Loom file pipeline. ``` ┌─────────────────┐ JSON layer stream ┌──────────────┐ ┌─────────────┐ │ AI planet │ ───────────────────────► │ loom-stream │ ──► │ .entity │ │ PyTorch / TF │ (weights + topology) │ (Go CLI) │ │ checkpoint │ │ JAX / sklearn │ └──────┬───────┘ └──────┬──────┘ └─────────────────┘ │ │ │ │ ▼ │ native forward on fixtures │ Loom infer └──────────────── compare ─────────────────────┴──────── PASS / EXACT / DIFF ``` 1. **Bedrock** — a POC harness per Loom layer family (`python/dense/`, `python/mha/`, …). Each bedrock defines manifest models, fixtures, and per-planet engine handlers. 2. **Planet handler** — Python reads **live in-memory weights** from the framework, builds the stream payload, and calls `loom-stream` (HTTP in the compare host; patched to CLI in the PyPI package). 3. **`loom-stream`** — Go binary (`planetbridging/cmd/loom-stream/`) calls `bridge.BuildNetworkFrom*Stream` and writes a `.entity` file. 4. **Compare** — native planet forward vs Loom forward on the same `x_test` samples. Labels: **EXACT**, **PASS** (within fp32 tolerance), **DIFF**. ```mermaid flowchart LR subgraph NOW["Today — INTO Loom"] P["Planets
PyTorch · TF · JAX · sklearn"] B["13 bedrocks
Dense · CNN · MHA · norms · …"] S["loom-stream"] E[".entity"] L["Loom infer"] C["Compare PASS"] P --> B --> S --> E --> L --> C end subgraph LATER["v1.0 — OUT of Loom"] L2["Loom hub"] H["ONNX · Safetensors · GGUF"] X["Any inference engine"] L2 -.-> H -.-> X end L --> L2 ``` --- ## The 13 bedrock layer types Each row is a **Loom volumetric layer family** with a planetbridging bedrock. All support PyTorch, TensorFlow, and JAX live streaming unless noted. | Loom layer | Bedrock ID | Typical foreign ops | Notes | |:-----------|:-----------|:--------------------|:------| | **Dense** | `dense` | `Linear`, `Dense`, `Gemm` | sklearn on dense only; deepest multi-layer MLP POC | | **CNN1** | `cnn1` | `Conv1d` | 2-layer stack POC | | **CNN2** | `cnn2` | `Conv2d` | 2-layer stack POC | | **CNN3** | `cnn3` | `Conv3d` | 2-layer stack POC | | **MHA** | `mha` | `MultiHeadAttention` | causal + RoPE POC | | **LSTM** | `lstm` | `LSTM` cell | | | **RNN** | `rnn` | `RNN` / `SimpleRNN` | | | **LayerNorm** | `layernorm` | `LayerNormalization` | | | **Embedding** | `embedding` | `Embedding` lookup | often **EXACT** match | | **RMSNorm** | `rmsnorm` | RMS normalization | | | **SwiGLU** | `swiglu` | gated MLP (LLaMA-style) | | | **Residual** | `residual` | skip connection | | | **Mixer** | `mixer` | integration stack | v2 chains all 12 types (16 layers); POC tolerance ~5e-5 | **Bedrock** here means: shared fixtures, manifest models, native forward reference, and stream builders — the “proof layer” that a planet’s weights map correctly into Loom. Lucy’s [seven-layer CPU suite](bedrock_validation.md) validates Loom internals; planetbridging bedrocks validate **cross-planet** parity. --- ## PyPI package (v0.7.3) — pip-only workflow ### Install ```bash pip install 'planetbridging[pytorch,welvet]==0.7.3' # optional cross-engine: pip install 'planetbridging[tensorflow,jax]' ``` **Python ≥ 3.10.** The wheel ships: | Bundled asset | Purpose | |:--------------|:--------| | `planetbridging/_data/python/` | All bedrock POC code + manifests | | `planetbridging/_bin//loom-stream` | linux_amd64, windows_amd64, macos_arm64 | | `planetbridging/examples/` | Runnable demos | Fixture `.npz` files are **not** in the wheel (PyPI 100 MB limit). They generate once on first use into `~/.planetbridging/fixtures/` (~86 MB total). Override with `PLANETBRIDGING_FIXTURES_CACHE`. ### Minimal example ```python from planetbridging import engines result = engines.stream("mha", "pytorch") print(result.native_vs_loom) # PASS print(result.entity_path) # path to .entity ``` ### Three-way ladder (native → loom-stream → welvet) [Welvet](../welvet/) is Loom’s Python binding (`welvet` on PyPI). After streaming, reload the same `.entity` and compare forwards: ```python from planetbridging import engines result = engines.stream("cnn1", "pytorch", try_welvet=True) # result.native, result.loom_stream, result.welvet — all compared ``` Welvet reload is solid on: `cnn1`–`cnn3`, `mha`, `layernorm`, `embedding`, `rmsnorm`, `swiglu` (and mostly `dense`). Still flaky: `lstm`, `rnn`, `mixer`, `residual`. --- ## Python API surface ### Primary: `engines` ```python from planetbridging import engines engines.stream("layernorm", "tensorflow") # one bedrock, one planet engines.stream_all_planets("cnn2") # PyTorch + TF + JAX on same layer engines.stream_all_bedrocks("pytorch") # all 13 types engines.ladder("mha", "numpy") # numpy ref → loom → welvet engines.available_planets("dense") # ('pytorch', 'tensorflow', 'jax', 'sklearn') engines.installed_planets("dense") # what's importable right now ``` Returns `EngineStreamResult`: `entity_path`, `native`, `loom_stream`, compare labels, optional `welvet` arrays. ### Custom dense models: `absorb` For **your own** sequential fully-connected stack (not manifest POC models): ```python import torch from planetbridging import absorb model = torch.nn.Sequential( torch.nn.Linear(32, 64), torch.nn.ReLU(), torch.nn.Linear(64, 8), ) result = absorb.pytorch(model, input_dim=32, layer_units=[64, 8], output_path="my.entity") ``` Also: `absorb.keras()`, `absorb.jax()`, `absorb.sklearn()` — **dense stacks only**. ### Low-level: `stream` ```python from planetbridging.stream import stream_bedrock, stream_dense, stream_mha ``` Direct JSON envelope → `loom-stream` → `StreamResult`. Used by smoke tests and advanced callers. ### Smoke / ladder (numpy reference) ```python from planetbridging import run_bedrock_smoke, run_bedrock_ladder run_bedrock_smoke("layernorm") # numpy native → loom-stream, no live torch run_bedrock_ladder("mha", try_welvet=True) ``` --- ## What you can do today | Goal | Supported? | How | |:-----|:-----------|:----| | Stream POC models from PyTorch into Loom `.entity` | ✅ | `engines.stream(bedrock, "pytorch")` | | Same from TensorFlow / JAX | ✅ | `engines.stream(bedrock, "tensorflow")` etc. | | All 13 Loom layer types in one pip install | ✅ | `pip install planetbridging[pytorch]` | | Verify native matches Loom numerically | ✅ | `native_vs_loom` label on every stream | | Reload `.entity` in Python via welvet | ✅ / partial | `try_welvet=True` or `welvet.Network.deserialize_entity` | | Bring your own **dense MLP** from Keras/PyTorch | ✅ | `absorb.keras()` / `absorb.pytorch()` | | Bring arbitrary **custom CNN / Transformer** architecture | 🟡 | Must map to a bedrock layer type or build low-level stream payloads | | Import a **SavedModel / ONNX file** without live Python | ⬜ | v1.x — not in PyPI package yet | | Export Loom → ONNX / GGUF / Safetensors | ⬜ | v1.0 roadmap | | 21-dtype parity across planets | 🟡 | Planets train fp32; Loom supports 21 dtypes natively — bridge compares float outputs at one precision | ### What “import into Loom” actually means You are **not** dropping an arbitrary TensorFlow graph into a black box. You are: 1. Expressing weights in Loom’s **volumetric layer vocabulary** (one of the 13 bedrock types, or a dense stack via `absorb`). 2. Streaming those weights through **`loom-stream`** into a native [`.entity`](entity.md) checkpoint. 3. Running Loom forward and proving it matches the source planet on shared test inputs. That `.entity` is the same format Lucy **[7]** uses for save/reload — topology + native-packed weights in one binary file. --- ## Compare host vs PyPI package | Mode | Audience | Entry | |:-----|:---------|:------| | **Compare host** | Developers, visual QA | `cd planetbridging && go run .` → dashboard on `:9876`, 13 tabs | | **PyPI package** | End users, CI, notebooks | `pip install planetbridging` → Python API, bundled `loom-stream` | The compare host uses HTTP (`POST /api/v1/loom/stream/*`). The PyPI package **patches** those calls to invoke `loom-stream` as a subprocess — same JSON contract, no server required. --- ## Environment variables | Variable | Effect | |:---------|:-------| | `PLANETBRIDGING_FIXTURES_CACHE` | Directory for generated fixture `.npz` (default `~/.planetbridging/fixtures/`) | | `PLANETBRIDGING_ROOT` | Force bedrock data root (dev checkout); unset for pip wheel | | `PLANETBRIDGING_LOOM_STREAM` | Explicit path to `loom-stream` binary | --- ## Examples (shipped in wheel) | Script | Demonstrates | |:-------|:-------------| | `01_hello_stream.py` | One bedrock → `.entity` | | `02_all_layer_types.py` | All 13 bedrocks | | `03_cross_engine.py` | PyTorch / TensorFlow / JAX on same bedrock | | `04_multi_layer_models.py` | Multi-layer MLP, CNNs, Mixer v2 | | `05_welvet_ladder.py` | native → loom-stream → welvet | | `06_showcase_everything.py` | Full API tour | ```bash EXAMPLES=$(python -c "import pathlib, planetbridging as pb; print(pathlib.Path(pb.__file__).parent / 'examples')") python "$EXAMPLES/01_hello_stream.py" python "$EXAMPLES/05_welvet_ladder.py" cnn1 mha ``` Outputs land in `./.planetbridging/examples/` under your current working directory. --- ## Roadmap — what it could do ### v1.0 — Loom → other engines Export from Loom hub formats, then route to any inference planet: ``` Loom .entity → Safetensors / ONNX / GGUF → ORT · llama.cpp · CoreML · … ``` ### v1.x — file-based import Pure-Go readers in the bridge (no live Python planet): | Priority | Format | Planet | |:---------|:-------|:-------| | 1 | `.safetensors` + manifest | PyTorch / HF | | 2 | `.onnx` | PyTorch export | | 3 | `.keras` | TensorFlow | | 4 | `saved_model/` | TensorFlow (harder) | See [`planetbridging/python/dense/README.md`](../planetbridging/python/dense/README.md) for stdlib-only feasibility notes. ### v1.x → 2.0 — completeness - More layer types (ConvTranspose, Softmax, …) - More planets (ONNX Runtime live, Paddle, …) - Tighter deep-stack determinism (Mixer v2 from POC → PASS) - Documented lossy conversion paths --- ## Repo layout (planetbridging/) | Path | Purpose | |:-----|:--------| | [`planetbridging/host/`](../planetbridging/host/) | Compare dashboard HTTP server | | [`planetbridging/bridge/`](../planetbridging/bridge/) | Go stream builders → `.entity` per layer type | | [`planetbridging/cmd/loom-stream/`](../planetbridging/cmd/loom-stream/) | CLI: JSON stdin → `.entity` | | [`planetbridging/python//`](../planetbridging/python/) | Per-layer POC: manifests, fixtures, planet handlers | | [`planetbridging/src/planetbridging/`](../planetbridging/src/planetbridging/) | PyPI package (`engines`, `absorb`, `stream`, `_fixtures`) | | [`planetbridging/examples/`](../planetbridging/examples/) | User-facing demos | | [`planetbridging/PROGRESS.md`](../planetbridging/PROGRESS.md) | Live per-model PASS/DIFF scoreboard | | [`planetbridging/BRIDGE.md`](../planetbridging/BRIDGE.md) | Architecture diagrams | --- ## How this fits Loom | Loom concept | Planet Bridging role | |:-------------|:---------------------| | [`VolumetricNetwork`](../poly/poly.go) | Target topology built from streamed layers | | [`.entity` checkpoint](entity.md) | **Output** of a successful stream — ship-ready native brain | | [21 DTypes](numerical_types.md) | Loom-native; bridge compares fp32 planet outputs today | | [Welvet C-ABI](../welvet/) | Reload streamed `.entity` in Python/Flutter/TS for inference | | [HF Safetensors import](serialization.md) | Separate path (SoulGlitch, Lucy [8]); planetbridging complements with **live** multi-planet streams | | [Bedrock validation](bedrock_validation.md) | Internal Loom CPU gate; planetbridging bedrocks are **external** parity gate | **One sentence:** Planet Bridging lets you build or train on the planet you already use, stream weights into Loom’s native `.entity` format, and prove Loom runs the same math — with a pip-installable path that needs no Go toolchain on the user machine. --- ## Related links - [planetbridging on PyPI](https://pypi.org/project/planetbridging/) - [planetbridging README](../planetbridging/README.md) — release matrix, compare host quick start - [entity.md](entity.md) — ENTITY format spec - [flutter.md](flutter.md) — welvet on mobile - [v080_release.md](v080_release.md) — Loom v0.80 ENTITY + Planet Bridging POC milestone --- ## Memory history, GPU load, and HF→entity convert Source: https://openfluke.com/docs/memory-history Markdown: https://openfluke.com/docs/memory-history.md # Memory history, GPU load, and HF→entity convert This page covers **`poly/memory_history.go`**: timed samples during LLM GPU upload, the in-terminal chart Lucy prints after load, and how memory policy ties to **block-wise safetensor import** (HF → `.entity` convert) and **block-wise GPU upload** (`.entity` → chat) so we do not hold full CPU and GPU weight copies at once. --- ## Why this exists When a transformer moves from CPU weights to GPU buffers, peak RAM matters — especially on mobile (SoulGlitch / iOS) and when loading large `.entity` checkpoints. Two failure modes were measured on Lucy ENTITY Talk **[8]** (Qwen3-0.6B): ### GPU load (`.entity` → chat, GPU enabled) | Phase | Old behavior | Fixed behavior | |:------|:-------------|:---------------| | **Decoder blocks** | Already OK — `ReleaseInferenceHostWeights()` after each block | Same (~7.5 MB host freed per block) | | **Globals** (embeddings, LM head, final norm) | Single `SyncToGPU()` uploaded all globals while ~1187 MB CPU weights still resident → **~2637 MB** host+gpu overlap | `SyncGlobalWeightsToGPUSequential()` uploads one global tensor, releases CPU, then next → **~2044 MB** peak overlap | | **Steady state** | Host weights eventually dropped | Host **0 MB**, GPU **~1451 MB** | The remaining ~2044 MB GPU-load peak is a **brief per-tensor overlap** during each global upload (CPU slice still present until that tensor’s GPU buffer exists). It is not the old “entire model doubled at once” bug. ### HF → `.entity` convert (safetensors → save) | Phase | Old behavior | Fixed behavior | |:------|:-------------|:---------------| | **Import** | `LoadSafetensors()` into one map, then `LoadWithPrefixes()` **copied** into `WeightStore.Master` while the map stayed resident → **~2× decoder weights** until GC | Globals first, then **one decoder block at a time** + `ReleaseTransientSafetensorMap()` after each block | | **Encode + save** | Full FP32 decoder in RAM through `SerializeEntityTransformer` (all layers + growing `[]byte` payload + final file buffer) | **Low-RAM path:** [`ImportHFSaveEntityTransformerBlockwise`](../poly/hf_entity_convert.go) — Q4/FP32 bake **one block at a time** into a **streaming payload file**, `releaseEntityConvertLayerWeights()` after each block, then `writeEntityWireStreaming()` (header + `io.Copy` payload — no full-file `[]byte`) | | **BitNet** | Already block-wise via `ImportHFBitNetCheckpointDir` | Unchanged (still `SaveEntityTransformer` one-shot) | Memory history is **not** wired to the convert step yet — only GPU chat load. Convert progress is visible via `HFEntityConvertProgress` callbacks (SoulGlitch task UI) or Lucy safetensor import logs (see below). --- ## Enabling recording ### Lucy (interactive) When GPU is enabled, Lucy prompts: ```text 📈 Measure memory during GPU load? (terminal chart after load — CPU weights vs GPU upload vs release) (1=yes / 0=no) [1]: ``` This appears in **Poly Talk [1]** and **ENTITY Talk [8]** (Lucy `poly_talk_session.go`, `hf_entity.go`). Answering **yes** calls `poly.SetMemoryHistoryRecording(true)`. Block-by-block upload is a separate prompt: ```text 📥 Block-by-block GPU upload? (1=yes / 0=no) [0]: ``` For meaningful charts, use **GPU + block upload + measure memory**. ### Environment variables | Variable | Effect | |:---------|:-------| | `LOOM_MEMORY_HISTORY=1` | Enable sampling without the Lucy prompt (off by default) | | `LOOM_MEMORY_HISTORY_JSON=/path/out.json` | After load, write samples as JSON in addition to the terminal report | | `LoomMemoryHistoryWriteJSON(path)` | Same JSON dump from Welvet C-ABI (`poly.GlobalMemoryHistory.WriteJSON`) — for FFI apps without env access | | `COLUMNS=100` | Widen the braille chart (default 80, max 120) | Lucy’s runtime prompt override takes precedence over env when set. --- ## What gets recorded Each sample is a `poly.MemorySample`: | Field | Meaning | |:------|:--------| | `elapsed_sec` | Seconds since session start | | `label` | Step name (e.g. `block_03_after_release`, `embeddings_after_sync`) | | `host_weights_mb` | Poly-accounted CPU model weights (`MemoryFootprint.HostWeightsMB`) | | `gpu_weights_mb` | Poly-accounted GPU weight buffers | | `gpu_kv_mb` | KV cache reservation on GPU | | `vram_total_mb` | Total VRAM usage from `GetVRAMUsage()` | | `heap_alloc_mb` / `heap_sys_mb` | Go runtime heap | | `process_rss_mb` | OS process RSS (`getrusage` on Unix; 0 on unsupported platforms) | **Important:** `host_weights_mb` + `gpu_weights_mb` is the Poly overlap metric used for diagnosis. **RSS** can stay high after host weights drop because Go/OS may retain pages until memory pressure — that is called out in the diagnosis block. --- ## Terminal output When the GPU load session finishes, `GlobalMemoryHistory.FinishSession()` prints: 1. **Braille chart** — four series: host weights (H), GPU weights (G), process RSS (R), VRAM (V) 2. **ASCII sparklines** — same series as ` .:-=+*#` ramps (works in all terminals) 3. **Sample log** — table of every labeled step 4. **Peak overlap line** — `peak host+gpu Poly weights overlap: X MB` when overlap exceeds baseline by >5% 5. **Diagnosis block** — pass/fail hints for block release, embeddings release, global sequential upload, RSS retention Example labels during a healthy ENTITY GPU load: ```text block_01_before_sync … block_28_after_release embeddings_before_sync → embeddings_after_sync → embeddings_after_release lm_head_before_sync → lm_head_after_sync → lm_head_after_release final_norm_before_sync → final_norm_after_sync → final_norm_after_release host_weights_released → after_gc ``` Legacy builds used a single `embeddings_on_gpu` label; diagnosis still recognizes that for regression comparison. --- ## API (poly) ```go // Process-wide recorder (Lucy uses this) poly.GlobalMemoryHistory.BeginSession("entity_gpu_load") poly.RecordFromTransformer(poly.GlobalMemoryHistory, tr, "block_01_after_sync") _ = poly.GlobalMemoryHistory.FinishSession() // chart + table + diagnosis // Toggle without env poly.SetMemoryHistoryRecording(true) poly.ResetMemoryHistoryRecording() // Footprint at any time fp := poly.NewMemoryFootprintFromTransformer(tr) fmt.Printf("host %.1f MB | gpu %.1f MB\n", fp.HostWeightsMB, fp.GPUWeightsMB) ``` Source files: | File | Role | |:-----|:-----| | [`memory_history.go`](../poly/memory_history.go) | `MemoryHistory`, samples, diagnosis | | [`memory_history_chart.go`](../poly/memory_history_chart.go) | Braille chart + sparklines | | [`process_memory_unix.go`](../poly/process_memory_unix.go) | RSS via `getrusage` | | [`process_memory_stub.go`](../poly/process_memory_stub.go) | RSS stub on unsupported OS | | [`hf_entity_convert.go`](../poly/hf_entity_convert.go) | `ImportHFSaveEntityTransformerBlockwise(Progress)` | | [`entity_convert_io.go`](../poly/entity_convert_io.go) | Streaming payload acc, per-block Q4 encode, `writeEntityWireStreaming` | | [`hf_import.go`](../poly/hf_import.go) | Block-wise safetensor import (`ImportHFCheckpointDir`) | | [`poly/tests/memory_history_test.go`](../poly/tests/memory_history_test.go) | Unit tests | --- ## HF → `.entity` convert (import + encode memory) There are **two llama-style convert lanes** in poly today: | Lane | API | Peak RAM during convert | Who uses it | |:-----|:----|:------------------------|:------------| | **Standard** | `ImportHFCheckpointDir` → `SaveEntityTransformer` | Block-wise safetensor import, then **full FP32 network** held through encode | Lucy `[8]` `convertEntityEntry`, `ImportHFToEntity` | | **Low-RAM encode** | `ImportHFSaveEntityTransformerBlockwise` (+ optional `Progress`) | **~one decoder block** FP32 + globals briefly at start + payload temp file on disk | SoulGlitch / mvp-simulation (iOS/macOS `.entity` convert) | Both lanes share the same **block-wise safetensor import** in [`hf_import.go`](../poly/hf_import.go). The low-RAM lane adds **block-wise encode** in [`hf_entity_convert.go`](../poly/hf_entity_convert.go) + [`entity_convert_io.go`](../poly/entity_convert_io.go). ### Block-wise safetensor import (both lanes) ```go // 1. Globals only LoadSafetensorsSelective(f, HFWeightIsGlobal) mapper.MapWeights(globalTensors) // embeddings, lm_head, final_norm // 2. One transformer block at a time for li := 0; li < numLayers; li++ { LoadSafetensorsSelective(sf, HFWeightMatchesLayer(k, li)) LoadWithPrefixes(net, layerMap) // copy into WeightStore.Master ReleaseTransientSafetensorMap(layerMap) } ReleaseTransientSafetensorMap(globalTensors, embeddings, lmHead, finalNorm) ``` `copyWeights` in [`prefix_safetensor.go`](../poly/prefix_safetensor.go) **copies** HF slices into `Master`; without per-block release, the full safetensor map and the network both held decoder weights. ### Block-wise encode + streaming save (low-RAM lane only) ```go // ImportHFSaveEntityTransformerBlockwiseProgress(modelDir, entityPath, weightDType, progress) // 1. Encode globals → payload temp file; drop embeddings/lm_head/final_norm from RAM collectEntityGlobalBlobAcc("embeddings", …) if !entityLMHeadTied(embeddings, lmHead) { collectEntityGlobalBlobAcc("lm_head", …) // same rule as SaveEntityTransformer } collectEntityGlobalBlobAcc("final_norm", …) embeddings, lmHead, finalNorm = nil; GC // 2. Per transformer block for li := 0; li < numLayers; li++ { LoadSafetensorsSelective + LoadWithPrefixes // one block FP32 in net ReleaseTransientSafetensorMap(layerMap) for j := 0; j < 4; j++ { collectEntityWeightBlobsAcc(&net.Layers[base+j], …, weightDType) // Q4_0 bake if INT4 releaseEntityConvertLayerWeights(&net.Layers[base+j]) // drop Master } GC } // 3. Write .entity: fixed header + JSON blob index + io.Copy(payload file) writeEntityWireStreaming(entityPath, net, trSpec, blobs, payloadPath) ``` **What gets Q4-baked:** decoder **MHA + SwiGLU** only (via `collectEntityQ4_0LayerAcc`). **RMSNorm**, MHA **q_norm/k_norm**, **embeddings**, **lm_head**, **final_norm** stay **FP32** — same rules as [`entity_q4.go`](../poly/entity_q4.go) / `SaveEntityTransformer`. **Progress callback** (`HFEntityConvertProgress`): `blockIndex` is 1-based per packed block; `detail` like `packed block 14/28`. SoulGlitch maps this to its convert task progress bar. ### Terminal signature (Lucy standard convert) When using **`ImportHFCheckpointDir` + `SaveEntityTransformer`** (Lucy `[8]` `convertEntityEntry`), a Qwen3-0.6B reconvert prints **three global** `✓ Loaded …` lines, then **`num_hidden_layers`** lines of `✅ Finished loading weights with prefixes.` (28 for Qwen3-0.6B). The old bulk path printed one bulk load without per-block messages. Example: ```text ⏳ Converting Qwen/Qwen3-0.6B → lucy_entities/Qwen--Qwen3-0.6B.entity [Q4 (INT4)] … ✓ Loaded model.embed_tokens.weight: … (role: embeddings) ✓ Loaded model.norm.weight: … (role: final_norm) ✓ Loaded lm_head.weight: … (role: lm_head) ✅ Finished loading weights with prefixes. ← block 1 … ← repeat per layer ✅ Qwen--Qwen3-0.6B.entity … ``` The **low-RAM lane** does not print those per-block safetensor lines (import is silent); use **`HFEntityConvertProgress`** or reconvert on Mac with Lucy to compare. ### Supported converts | Path | API | Safetensor import | Encode / save | |:-----|:----|:------------------|:----------------| | Lucy `[8]` convert | `convertEntityEntry` | ✅ block-wise | `SaveEntityTransformer` (full network in RAM during encode) | | Programmatic (standard) | `ImportHFToEntity` / `ImportHFCheckpointDir` + `SaveEntityTransformer` | ✅ block-wise | Full network during encode | | Programmatic (low-RAM) | `ImportHFSaveEntityTransformerBlockwise(Progress)` | ✅ block-wise | ✅ block-wise encode + streaming payload | | SoulGlitch convert | mvp → `ImportHFSaveEntityTransformerBlockwiseProgress` | ✅ | ✅ (+ CHGLUE standalone wrapper streams loom bytes — app layer) | | BitNet | `ImportHFBitNetCheckpointDir` + `SaveEntityTransformer` | ✅ (packed ternary per block) | One-shot save | ### Remaining encode overlap (expected) Even the low-RAM lane still holds **one block’s FP32 weights** plus the **globals encode spike** at the start (embeddings + lm_head for large-vocab models are large). That is much smaller than holding **all blocks** through `SerializeEntityTransformer`, but not zero — see roadmap below. --- ## GPU load path (what the history measures) Lucy centralizes inference GPU setup in `inference_setup.go` → `setupTransformerForInference`. Welvet SoulGlitch mirrors the same policy in `welvet/cabi/llm_ext.go` (`LoomCreateLLM`). ### Step 1 — Init WGPU ```go tr.Network.InitWGPU() ``` ### Step 2 — Decoder blocks (when `sequentialGPULoad`) For each transformer block (4 grid layers: input norm, MHA, post-attn norm, SwiGLU): ```go layer.SyncToGPU() (&tr.Network.Layers[idx]).ReleaseInferenceHostWeights() ``` ### Step 3 — Global weights (sequential) Prefer **`SyncGlobalWeightsToGPUSequential()`** over bulk **`SyncToGPU()`** for inference load: ```go tr.SyncEmbeddingsToGPU(); tr.ReleaseEmbeddingsHost() tr.SyncLMHeadToGPU(); tr.ReleaseLMHeadHost() // skips duplicate buffer when tied tr.SyncFinalNormToGPU(); tr.ReleaseFinalNormHost() // or: tr.SyncGlobalWeightsToGPUSequential() ``` `SyncToGPU()` still uploads all three globals **without** mid-upload CPU release — kept for training paths and legacy callers. ### Step 4 — Warmup and final cleanup ```go _, _ = tr.ForwardTokenIDsWGPU([]uint32{0}, nil, true, true) tr.Reset() tr.ReleaseInferenceHostWeights() // sweep any remaining host slices runtime.GC() debug.FreeOSMemory() ``` ### Where each policy is used | Caller | Import / convert | Decoder upload | Global upload | |:-------|:-----------------|:---------------|:--------------| | Lucy `[8]` HF → `.entity` | Block-wise import + `SaveEntityTransformer` | — | — | | SoulGlitch / mvp HF → `.entity` | `ImportHFSaveEntityTransformerBlockwiseProgress` | — | — | | Lucy `setupTransformerForInference` | — | Block-wise + release | `SyncGlobalWeightsToGPUSequential` | | Welvet `LoomCreateLLM` (safetensors) | Block-wise (chat load) | Block-wise + release | `SyncGlobalWeightsToGPUSequential` | | Welvet `LoomSyncToGPU` / bulk `SyncToGPU()` | — | All layers, no mid-release | Bulk, no mid-release | | Training / demos calling `SyncToGPU()` directly | Varies | Varies | Bulk | **Entity on SoulGlitch:** `LoomLoadEntityTransformerAs` builds a full CPU transformer only. GPU setup must follow the Lucy sequence above (or a future `LoomCreateLLMFromEntity` export). See [entity.md — GPU load](entity.md#gpu-load-after-entity-deserialize). --- ## Further peak reduction (roadmap) - **GPU load:** stream or mmap entity globals so embeddings/LM head never exist as full FP32 CPU slices before GPU upload; quantize-on-upload for globals (v1 entity keeps globals FP32 on disk) - **Convert:** optional memory history during convert (same chart as GPU load); stream globals encode in two passes to shrink the initial globals spike on mobile - **Load:** staged `DeserializeEntityWithOptions` + block GPU upload without full-file deserialize peak (see [entity.md](entity.md)) --- ## See also - [gpu.md](gpu.md) — `InitWGPU`, per-layer `SyncToGPU`, `ReleaseInferenceHostWeights` - [transformer.md](transformer.md) — `SyncGlobalWeightsToGPUSequential`, tied weights - [entity.md](entity.md) — Lucy **[8]** ENTITY Talk flow - [testing_and_validation.md](testing_and_validation.md) — reading Lucy logs --- ## Windows ARM64 native builds Source: https://openfluke.com/docs/windows-arm64 Markdown: https://openfluke.com/docs/windows-arm64.md # Windows ARM64 native builds Use **one script** from WSL/Linux: ```bash cd loom/welvet/cabi/internal/build ./build_windows_arm64.sh --skip-lib ``` Full guide: [`README_WINDOWS_ARM64.md`](../welvet/cabi/internal/build/README_WINDOWS_ARM64.md) Do **not** use `./build_unix.sh windows arm64` — it redirects here. --- ## ASM & Volumetric Executor Exploration (Archive) Source: https://openfluke.com/docs/asm-and-volumetric-exploration Markdown: https://openfluke.com/docs/asm-and-volumetric-exploration.md # ASM & Volumetric Executor Exploration (Archive) **Status:** Exploratory work — **not merged as a long-term direction.** **Date:** June 2026 **Context:** Lucy menu `[7]` (seven-layer CPU suite), Dense forward only. This document captures everything built and learned during the **Plan 9 ASM dense forward** push and the **volumetric fused executor (v1)** experiment. Copy this file before deleting the exploration code; the complexity-to-gain ratio did not justify keeping it in-tree. **Related existing docs:** [`poly/asm/README.md`](../poly/asm/README.md), [`bitnet_cpu.md`](bitnet_cpu.md), [`dispatch.md`](dispatch.md). --- ## Executive summary ### What we tried 1. **Native-quant ASM dense forward** — BitNet-style W8A8: morphed weights stay in native storage, activations quantized per row to int8, integer dot in Plan 9 asm, one output scale per row. **No** expanding morphed weights to float32 for matmul on the ASM path. 2. **BitNet I2_S scaffolding** — Microsoft `ggml-bitnet-mad.cpp` I2_S pack + scalar/asm row dots for ternary. 3. **Volumetric executor v1** — `BuildDenseExecPlan` + `ForwardDenseExecutor`: skip `DispatchLayer` per cell×layer, pre-plan dtype→kernel, morph weights once, direct dense kernel calls. ### What we learned | Finding | Detail | | :--- | :--- | | ASM wins on **fat layers** | 1×1×1 with pyramid dims (16→64): quant native ASM ~**1.5–2.4×** vs Go float-dequant tiled path | | ASM **loses on thin grids** | 3×3×3 with `flatEndpoints(4)` (4×4 micro-layers): ASM ~**0.5–0.93×** vs Go — setup + 189 tiny GEMVs dominate | | Executor v1 saves **~15–38%** | Skipping interpret dispatch on 3×3×3 dense: consistent **1.15–1.38×** (Interp/Exec↑), perfect parity | | Dispatch is **not** the main 3³ problem | Executor proves interpret tax ≈ **25%** of forward; remaining **~75%** is per-layer 4×4 work | | Quant does **not** win bigger via executor | Int8/Int4 gains ≈ Float32 — no special breakout | | **Not worth the complexity** | Many files, dual parity paths, Lucy table sprawl, MC races, dtype routing — for ~20–35% on a narrow CPU interpret path | ### Decision Archive the exploration. Keep the **interpreter** (`ForwardPolymorphic` + `DispatchLayer`) as the research path. Peak inference belongs on **GPU** (`poly/wgpu_forward.go`) or a future **batched executor** that fuses matmul across cells — not per-hop ASM + per-hop dispatch trimming. --- ## Part I — Architecture before and after ### Two tiers (intended end state) ``` ┌─────────────────────────────────────────────────────────────┐ │ Tier 1 — Interpreter (keep) │ │ ForwardPolymorphic → spatial tile loops → DispatchLayer │ │ Build, train, parity, volumetric experiments │ └─────────────────────────────────────────────────────────────┘ │ ▼ ┌─────────────────────────────────────────────────────────────┐ │ Tier 2 — Executor (explored, v1 only) │ │ Plan at load: dtype → kernel map, pooled buffers │ │ Fused visit order, few CPU entry points │ │ v1: no cross-cell matmul batching yet │ └─────────────────────────────────────────────────────────────┘ ``` ### Interpret path (baseline) ``` ForwardPolymorphic (forward.go) → spatial 4×4×4 tile loops (or classic z→y→x→l) → DispatchLayer (jump table) → DenseForwardPolymorphic (dense.go) → asm: denseForwardAsm → bitnet: DenseForwardBitNetNativeQuantCPU → default: DenseForwardTiled (float dequant via denseActiveWeights) ``` Lucy `[7]` **Go column** for quant dtypes timed `DenseForwardTiled` (float dequant), **not** native BitNet — so ASM-vs-Go tables compare different semantics unless you read the captions. ### ASM dense path ``` DenseForwardPolymorphic └─ layerUseAsmForward && asm.Enabled() └─ denseForwardAsm (dense_asm.go) ├─ ternary: denseForwardAsmBitNetTernary → I2S asm if available ├─ quant / low-bit float morph: │ denseForwardAsmNativeQuant (dense_asm_native_quant.go) │ · morphed []uint8 weights (WeightStore.Morph) │ · per-row int8 activation quant (bitNetQuantizeActivationNumeric) │ · dot.U8BytesTileNativeI64 / packed nibble / 2-bit / binary paths │ · one output scale per row → float preAct └─ Float32/64: denseForwardAsmByDType → asm/matmul GEMV ``` **Key principle:** weights use proper native/entity layouts; forward does not re-morph from FP32 on every hop if inference sync already populated `Versions[dtype]`. ### Executor v1 path ``` BuildDenseExecPlan(net) · forwardVisitLayerIndices — same order as ForwardPolymorphic · verify all visited layers are LayerDense · prepDenseExecWeights — Morph once per layer · pickDenseExecKernel — dtype → DenseExecKernel enum ForwardDenseExecutor(net, plan, input) · for step in plan.Steps: denseExecForwardStep(kernel, layer, current) // no DispatchLayer · kernels mirror DenseForwardPolymorphic (no ASM): Tiled / BitNet native / I2S try ``` **What v1 did *not* do:** fuse tiles under the grid, batch 27 cells into one GEMV, or call ASM from the executor loop. --- ## Part II — ASM exploration (detail) ### Goals - Hand-written **Plan 9 assembly** (`*.s`, not CGO) on **amd64** and **arm64**. - **21 dtypes** on Dense forward via `UseAsmForward` / `VolumetricNetwork.SetAsmForward`. - **Native integer dots** — multiply/accumulate in storage width; dequant once at boundary. - **BitNet W8A8** reference in Go (`DenseForwardBitNetNativeQuantCPU`) for quant parity vs ASM. - Lucy `[7]` timing: Go tiled vs ASM, all dtypes when `ForwardCapable`. ### Package layout (`poly/asm/`) | Package | Role | | :--- | :--- | | `asm/dot/` | f32/f64 dots, native int dots (`native_int_*.s`), packed row dots (`native_packed_*.s`) | | `asm/matmul/` | Tiled GEMV, `OverOutputTiles` MC, wires dot into tile callbacks | | `asm/dense/` | Thin float forward entry for poly | | `asm/bitnet/` | `rowDotI2SI8`, `ternaryWordDot16` (scalar asm stubs on arm64/amd64) | ### BitNet I2_S (`poly/bitnet/`) Reference: Microsoft `ggml-bitnet-mad.cpp` (cloned under `loom/BitNet/` during exploration). | Constant / type | Meaning | | :--- | :--- | | `QK_I2S = 128` | Block size; **cols must be `% 128 == 0`** for I2S matrix | | `I2SMatrix` | Row-major ternary in ACT_PARALLEL layout | | Codes `0,1,2` | → weights `-1, 0, +1` | | `RowDotI2SI8Go` | Go scalar reference | | ASM | `poly/asm/bitnet/i2s_amd64.s`, `i2s_arm64.s` — **not** full AVX2 `maddubs` loop from Microsoft | **Not ported:** full AVX2 I2_S inner loop from `BitNet/src/ggml-bitnet-mad.cpp`. ### BitNet W8A8 native quant flow Per output row: 1. Read morphed weight row from `WeightStore.Versions[dtype]` (uint8 storage). 2. Quantize input row to int8 (`bitNetQuantizeActivationNumeric`). 3. Integer dot (Go ref or `dot.U8BytesTileNativeI64` in asm). 4. `outScale = weightScale * actMax / 127` → float pre-activation. 5. Apply layer activation. Dtype-specific branches in `denseForwardAsmNativeQuant`: wide int16/32/64, FP8 morph, nibble Int4, 2-bit Int2/Ternary, binary. ### Poly files (ASM exploration) | File | Role | | :--- | :--- | | `poly/dense_asm.go` | `SetAsmForward`, `denseForwardAsm` routing | | `poly/dense_asm_dtypes.go` | Dtype → asm matmul dispatch | | `poly/dense_asm_native.go` | Legacy native width paths | | `poly/dense_asm_native_matmul.go` | Native tiled matmul wiring | | `poly/dense_asm_native_quant.go` | BitNet W8A8 asm path | | `poly/dense_bitnet_w8a8_cpu.go` | Go scalar BitNet reference | | `poly/bitnet_cpu.go` | Ternary word dot delegates to asm | | `poly/bitnet_i2s_cpu.go` | `GetBitNetI2SMatrix`, `DenseForwardBitNetI2SCPU` | | `poly/bitnet/i2s.go` | I2_S pack + Go row dot | | `poly/bitnet/ternary_u32.go` | `TernaryWordDot16Go` | | `poly/dense.go` | `layerUseBitNetNativeQuant`, `DenseForwardPolymorphic` routing | ### Network flags (`poly/poly.go`) ```go UseAsmForward bool // Plan 9 assembly CPU kernels UseBitNetNativeQuant bool // Dense: BitNet W8A8 Go ref (Lucy quant parity column) ``` ### Bugs fixed during exploration | Bug | Symptom | Fix | | :--- | :--- | :--- | | Lucy gated ASM timing on float-only determinism | Blank ASM columns for quant dtypes | Time ASM for all dtypes when `ForwardCapable` | | Shared `xq`/`xu` buffers in `runRows` closure | MC parity ~1e19 on Float16 | Allocate quant buffers **inside** each `runRows` goroutine | | `GetBitNetI2SMatrix` with cols=16 | Panic — I2_S needs cols % 128 | Guard / fallback to packed ternary | | Duplicate `bitNetTernaryWordDotTail` | Build error | Cleaned `bitnet_cpu.go` | ### Lucy `[7]` ASM integration **Files:** Lucy `examples/seven_layer/runner.go`, `summary.go`, `common.go` - `requiresAsmGoTiledParity` — Float32/64: ASM vs Go tiled - `requiresAsmBitNetParity` — quant: ASM vs `captureForwardBitNetNative` (Go BitNet ref) - Table: `printDenseForwardAsmTimingTable` — Go SC/MC, ASM SC/MC, Go/Asm↑, |Δ| **Captions:** - Go = `DenseForwardTiled` (float dequant) - ASM = native quant (BitNet W8A8) or float GEMV - Go/Asm↑ = Go÷ASM (>1 means assembly wins) ### ASM benchmark findings (Lucy Dense) #### 1×1×1 (pyramid 16→24→32→48→64→48→32→8) | Category | Go/Asm↑ (approx) | | :--- | :--- | | Int8, FP8, Float16 | ~1.5–2.4× ASM wins | | Int4, Binary, Ternary | ~0.5–1.0× (scalar packed dots; ternary may hit Go matvec) | | Float32/64 | ~1.0–1.1× | #### 3×3×3 (`flatEndpoints(4)` → 4×4 layers, 189 hops) | Category | Go/Asm↑ (approx) | | :--- | :--- | | **Everything** | ~**0.5–0.93×** — ASM **slower** than Go tiled | **Why 3³ is bad for ASM:** - 27 cells × 7 layers = **189 serial dense forwards** - Each layer is **4×4** GEMV — too small to amortize asm entry, quant buffers, tile setup - Interpreter adds overhead too, but Go tiled float-dequant path is cheap per micro-op - Native quant correctness is fine; **kernel economics** are wrong at this shape ### Tests added | Test | Package | | :--- | :--- | | `dense_asm_parity_test.go` | Float → Go tiled; quant → BitNet native ref | | `bitnet_i2s_test.go` | I2S row dot, ternary word dot, I2S forward smoke | | `bitnet_cpu_test.go` | BitNet CPU helpers | ```bash cd loom/poly go test ./asm/... ./tests/ -run 'DenseAsm|BitNet|I2S' -count=1 ``` --- ## Part III — Volumetric executor (detail) ### Motivation User goal: *"dtype-aware volumetric executor that fuses tiles under the grid, instead of re-interpreting every cell×layer on the way through."* **v1 scope:** prove interpret tax measurable; skip `DispatchLayer`; pre-plan kernels. **Not** true tile fusion under the grid. ### API (`poly/volumetric_executor.go`) ```go type DenseExecKernel uint8 // Tiled | BitNetNative | BitNetI2SThenTiled type DenseExecStep struct { LayerIdx int Kernel DenseExecKernel } type DenseExecPlan struct { Steps []DenseExecStep } func BuildDenseExecPlan(n *VolumetricNetwork) (*DenseExecPlan, error) func ForwardDenseExecutor(n, plan, input) (*Tensor[float32], time.Duration) ``` `ErrNotDenseOnly` if any visited layer is not `LayerDense`. ### Lucy `[7]` executor table Printed after ASM table for Dense suites: **Interpret vs Fused Executor**. | Column | Meaning | | :--- | :--- | | Interp SC/MC | `ForwardPolymorphic` + `DispatchLayer` (CPU Go, no ASM) | | Exec SC/MC | `ForwardDenseExecutor` with pre-built plan | | Interp/Exec↑ | Interpret÷Executor (>1 = executor wins) | | \|Δ\| | max \|Interpret − Executor\| — parity | **Files:** `runner.go` (capture), `common.go` (`benchmarkExecutorForward`), `summary.go` (`printDenseExecutorTimingTable`), `dense_executor_3x3_test.go` (smoke). ### Measured results — Dense 3×3×3 (5 passes, user run) ``` ╔══════════════════════════════════════════════════════════════════════╗ ║ Dense 3×3×3 — forward: Interpret vs Fused Executor (avg of 5 passes) ║ ╚══════════════════════════════════════════════════════════════════════╝ | DType | Interp SC | Exec SC | Interp/Exec↑ | |Δ| SC | Interp MC | Exec MC | Interp/Exec↑ | |Δ| MC | | Float64 | 31.4µs | 25.9µs | 1.21× | 0.00e+00 | 152.8µs | 131.0µs | 1.17× | 0.00e+00 | | Float32 | 31.2µs | 22.6µs | 1.38× | 0.00e+00 | 136.1µs | 140.1µs | 0.97× | 0.00e+00 | | Float16 | 38.7µs | 32.9µs | 1.18× | 0.00e+00 | 152.9µs | 121.9µs | 1.25× | 0.00e+00 | | BFloat16 | 36.5µs | 27.4µs | 1.33× | 0.00e+00 | 136.0µs | 122.7µs | 1.11× | 0.00e+00 | | FP8-E4M3 | 38.0µs | 32.9µs | 1.15× | 0.00e+00 | 145.8µs | 120.9µs | 1.21× | 0.00e+00 | | FP8-E5M2 | 38.5µs | 33.2µs | 1.16× | 0.00e+00 | 151.4µs | 131.0µs | 1.16× | 0.00e+00 | | Int64 | 36.0µs | 30.4µs | 1.19× | 0.00e+00 | 139.2µs | 114.8µs | 1.21× | 0.00e+00 | | Uint64 | 36.2µs | 29.7µs | 1.22× | 0.00e+00 | 144.7µs | 127.7µs | 1.13× | 0.00e+00 | | Int32 | 37.1µs | 30.0µs | 1.24× | 0.00e+00 | 159.2µs | 126.7µs | 1.26× | 0.00e+00 | | Uint32 | 36.4µs | 29.9µs | 1.22× | 0.00e+00 | 132.2µs | 129.3µs | 1.02× | 0.00e+00 | | Int16 | 36.7µs | 28.0µs | 1.31× | 0.00e+00 | 140.5µs | 114.2µs | 1.23× | 0.00e+00 | | Uint16 | 35.9µs | 29.2µs | 1.23× | 0.00e+00 | 131.4µs | 117.9µs | 1.11× | 0.00e+00 | | Int8 | 38.7µs | 30.8µs | 1.26× | 0.00e+00 | 141.3µs | 111.1µs | 1.27× | 0.00e+00 | | Uint8 | 38.1µs | 27.8µs | 1.37× | 0.00e+00 | 155.2µs | 133.0µs | 1.17× | 0.00e+00 | | Int4 | 36.8µs | 30.3µs | 1.21× | 0.00e+00 | 141.9µs | 114.9µs | 1.23× | 0.00e+00 | | Uint4 | 40.4µs | 32.1µs | 1.26× | 0.00e+00 | 143.7µs | 125.6µs | 1.14× | 0.00e+00 | | FP4 | 36.9µs | 28.9µs | 1.28× | 0.00e+00 | 130.8µs | 115.4µs | 1.13× | 0.00e+00 | | Int2 | 38.7µs | 30.6µs | 1.27× | 0.00e+00 | 145.5µs | 110.2µs | 1.32× | 0.00e+00 | | Uint2 | 37.4µs | 29.4µs | 1.27× | 0.00e+00 | 151.8µs | 122.7µs | 1.24× | 0.00e+00 | | Ternary | 54.9µs | 73.5µs | 0.75× | 0.00e+00 | 56.6µs | 44.0µs | 1.29× | 0.00e+00 | | Binary | 37.3µs | 31.6µs | 1.18× | 0.00e+00 | 138.7µs | 126.8µs | 1.09× | 0.00e+00 | Best Interp/Exec↑ SC: Float32 at 1.38× | Best MC: Int2 at 1.32× ``` **Anomaly:** Ternary SC executor **slower** (0.75×) while MC faster (1.29×) — same |Δ|=0; likely kernel path + 5-pass noise. Worth noting if revisiting. ### 1×1×1 executor (reference) Most dtypes ~1.0–1.1× — fat matmul dominates; dispatch tax invisible. ### When executor wins (rule of thumb) ``` noticeability ∝ (dispatch hops) / (compute per hop) ``` | Workload | Executor feel | | :--- | :--- | | Dense 3×3×3, width 4 | **Noticeable** (~1.2–1.4×) | | Dense 1×1×1, pyramid | **Barely** (~1.0–1.1×) | | MHA / SwiGLU / CNN | **Less** — heavy per-hop compute | | Training | **None** — executor forward-only | | GPU path | **Different tier** — `wgpu_forward.go` | --- ## Part IV — Lessons learned ### 1. Interpret dispatch is real but bounded Executor v1 isolates ~**20–35%** overhead from `DispatchLayer` + routing on 3×3×3 dense. That is **not** enough to explain ASM being **0.5×** on the same grid — the bottleneck is **189 × tiny GEMV**, not the jump table alone. ### 2. ASM economics need layer mass Native quant ASM needs enough FLOPs per call to amortize: - Plan 9 entry - Per-row activation quant buffers - Tile orchestration - Output scale + activation 4×4 on 3³ fails; 16×48 on 1³ succeeds. ### 3. Compare apples to apples in benchmarks Lucy Go column used float-dequant tiled forward for quant dtypes; ASM used native BitNet. Parity was split (`requiresAsmGoTiledParity` vs `requiresAsmBitNetParity`) but the **timing** comparison was semantically asymmetric. ### 4. Executor v1 ≠ "fused volumetric" True fusion means e.g. **one batched GEMV per layer-index across all 27 cells**, pooled activations, ASM called once per slab — not 189 direct kernel calls with dispatch removed. ### 5. Complexity inventory - Dual forward paths (interpret / executor / asm / bitnet ref) - Per-dtype asm branches in Go + `.s` stubs per arch - Lucy table proliferation (ASM + executor + determinism + save/reload) - MC threading bugs in quant buffer pooling - I2S layout constraints (cols % 128) **Gain:** ~1.3× on a narrow CPU interpret forward for one layer type on one grid shape. **Verdict:** Good experiment, wrong ROI to maintain. --- ## Part V — What v2 would have required If revisiting (e.g. after GPU path matures): 1. **Cross-cell batching** — for each `l` in `0..LayersPerCell-1`, stack 27 cell activations → one `batch×4` GEMV (or wider). 2. **ASM inside executor** — single plan entry point; no `DispatchLayer` and no per-layer asm re-entry. 3. **Buffer pool** — `xq`/`xu`/output tiles allocated per frame, not per row batch. 4. **Full I2_S SIMD** — port AVX2/NEON from Microsoft BitNet for ternary at scale. 5. **GPU executor tier** — mirror plan on device; `BeginFrame` + block dispatches (partial precedent in `wgpu_forward.go`). --- ## Part VI — File inventory (exploration code to delete) Use `git status` / `git log` for the exact commit set. Approximate list: ### `poly/` — executor - `volumetric_executor.go` - `tests/dense_executor_test.go` ### `poly/` — BitNet W8A8 + I2S (if removing entire exploration) - `dense_asm.go`, `dense_asm_dtypes.go`, `dense_asm_native.go`, `dense_asm_native_matmul.go`, `dense_asm_native_quant.go` - `dense_bitnet_w8a8_cpu.go` - `bitnet_cpu.go`, `bitnet_i2s_cpu.go` - `bitnet/` (package) - `tests/dense_asm_parity_test.go`, `tests/bitnet_i2s_test.go` (and related) ### `poly/asm/bitnet/` - `dot.go`, `dot_decl.go`, `dot_stub.go` - `i2s_amd64.s`, `i2s_arm64.s` - `ternary_amd64.s`, `ternary_arm64.s` ### Lucy `examples/seven_layer/` (in [lucy_bloom_rivers](lucy.md)) - Executor fields in `summary.go` (`DTypeRow` executor columns, `printDenseExecutorTimingTable`) - Executor capture in `runner.go`, `benchmarkExecutorForward` in `common.go` - `dense_executor_3x3_test.go` ### Revert / keep decisions | Keep | Remove (exploration-only) | | :--- | :--- | | `poly/asm/dot`, `matmul`, `dense` if predating this work and used elsewhere | `volumetric_executor.go` + Lucy executor table | | `ForwardPolymorphic`, `DispatchLayer` | BitNet I2S asm if unused | | `wgpu_forward.go` GPU path | `dense_asm_native_quant.go` if reverting to pre-BitNet asm | | `docs/bitnet_cpu.md` (older ternary doc) | This archive doc **keep** | **Note:** `poly/asm/README.md` may describe paths you're removing — update or revert that README when deleting code. --- ## Part VII — How to reproduce (before delete) ```bash # Poly parity cd loom/poly go test ./tests/ -run 'DenseExecutor|DenseAsm|BitNet' -count=1 # Lucy Dense 1³ regression (21 dtypes + ASM + executor tables) cd lucy_bloom_rivers/examples/seven_layer go test -run TestRunLayerSuiteDense1x1AllDTypes -v -count=1 # Lucy 3³ executor smoke go test -run TestDenseExecutor3x3Float32Smoke -v -count=1 # Full menu cd lucy_bloom_rivers && go run . # → [7] Dense ``` --- ## Closing The exploration validated: - Native quant ASM **can** beat float-dequant Go on **fat** dense layers. - It **cannot** rescue **thin volumetric grids** without batching. - Skipping dispatch saves **~25%**, not **2×**. - A real volumetric performance story needs **fusion below the grid**, not more interpreter variants. Copy this document, then delete the code. The interpreter + GPU paths remain the production-shaped architecture. --- ## Cross-path CPU suite (Lucy menu [15]) Source: https://openfluke.com/docs/cross-path-layers Markdown: https://openfluke.com/docs/cross-path-layers.md # Cross-path CPU suite (Lucy menu [15]) **Run:** [Lucy Bloom Rivers](lucy.md) → **[15]** → pick grid (default **2³**) → layer type (or **[0]** for all). **Code:** Lucy `examples/seven_layer/cross_path_menu.go` **Log:** `lucy_testing_output/cross_path_layers.txt` Unifies **[7]** (tiled SC/MC/SIMD) and **[14]** (native exact + native SIMD) in one side-by-side matrix per layer × dtype. --- ## What it compares | Path | Config | Math | |------|--------|------| | **SC** | `EnableMultiCoreTiling=false`, SIMD off | Tiled FP32-dequant (`GetActive`) | | **MC** | `EnableMultiCoreTiling=true`, SIMD off | Tiled FP32-dequant, parallel tiles | | **SIMD** | MC + `SetSimdForwardRecursive(true)` | Tiled + Plan 9 `DotTile` / saxpy | | **Native** | `UseExactDType=true`, SIMD off | `*_native.go` storage-dtype MAC | | **Native SIMD** | `UseExactDType=true`, SIMD on | `*_native_simd.go` | **Grid:** selectable **1³ / 2³ / 3³** (default **2³**) · **[5] 3³ SIMD duel** (QAT-SIMD vs Nat-SIMD only) · **7 layers/cell** · **21 dtypes** · train epochs scale by grid (50 / 12 / 6) **Layer types:** Dense, SwiGLU, MHA, CNN1, CNN2, CNN3, RNN, LSTM, Embedding, Residual --- ## Per-dtype output 1. **Raw timing — forward / backward** — SC/MC/SIMD/Nat/NatS wall times 2. **Comparison — forward / backward** — QAT SC→SIMD, Nat→NatS, best fwd/bwd (QAT vs Nat) 3. **Raw timing — training** — QAT-SC, QAT-MC, QAT-SIMD, Nat, Nat-SIMD (30 epochs) 4. **Train comparisons** — QAT SC/MC→SIMD, Nat→NatS, QAT SIMD vs Nat, QAT SIMD vs NatS, best train 5. **Parity table** — tiled SC↔MC, SC↔SIMD (gated); native↔native-SIMD and SC↔native (informational) 6. **Train loss table** — final loss per path + PASS/FAIL gates 7. **Test tally** — gated checks per category + session manifest ### Gated tests (per dtype, SIMD layers) | Category | Count | |----------|-------| | tiled fwd/bwd finite (SC, MC, SIMD) | 6 | | tiled parity (SC↔MC, SC↔SIMD fwd/bwd) | 4 | | native path + fwd/bwd + native-SIMD finite | 5 | | train SC, MC, SIMD, native, native-SIMD | 5 | | **Total** | **20 × 21 = 420** per SIMD layer | Non-SIMD layers omit SIMD columns (fewer checks). Native↔native-SIMD parity is **reported but not gated** — MAC dtypes can legitimately differ from tiled SIMD tolerance bands. --- ## SIMD duel mode (grid **[5]**) **3³ only** · **189-layer stack** · **6 train epochs** · compares **only**: | Path | What runs | |------|-----------| | **QAT-SIMD** | Tiled `GetActive` FP32 + Plan 9 SIMD | | **Nat-SIMD** | `UseExactDType` + `*_native_simd.go` | Skips SC, MC, native scalar, and parity-vs-SC tables. Per dtype the log prints: 1. **One-line summary** — PASS/FAIL, 7/7 checks, fwd/bwd/train winner + speedup 2. **Raw timing (fwd / bwd)** — `QAT SIMD-f`, `NatS-f`, `QAT SIMD-b`, `NatS-b` wall times 3. **QAT-SIMD vs Nat-SIMD** — pairwise comparison and per-phase winner per dtype 4. **Raw timing (train)** — 6-epoch wall time per path 5. **Train comparisons** — QAT-SIMD vs Nat-SIMD train speedup 6. **Dtype spread** — slowest → fastest dtype per phase (among winning SIMD path per dtype) 7. **Train loss table** — `Loss₀` and final loss per path + PASS/FAIL gates 8. **Test tally** — 7 gated checks × 21 dtypes = **147** per layer | Gated check (per dtype) | What it verifies | |-------------------------|------------------| | `native.path` | Native-exact routing available | | `tiled.fwd.simd` | QAT-SIMD forward finite | | `tiled.bwd.simd` | QAT-SIMD backward finite | | `native.fwd.simd` | Nat-SIMD forward finite | | `native.bwd.simd` | Nat-SIMD backward finite | | `train.simd` | QAT-SIMD 6-epoch train OK | | `train.native.simd` | Nat-SIMD 6-epoch train OK | Use this when you want **apples-to-apples fastest SIMD** at the largest practical grid without noise from non-SIMD paths. --- ## Archived SIMD duel results (Jul 2026) Full **[15] → grid [5] → [0] all layers** runs, captured off-machine: | Platform | Archive path | |----------|----------------| | **amd64** (AVX2) | `~/Documents/loom/simd/cross_path_layers_amd.txt` | | **arm64** (NEON) | `~/Documents/loom/simd/cross_path_layers_arm.txt` | Runtime log during a session: `lucy_testing_output/cross_path_layers.txt` (reset each run). ### Pass summary (21 dtypes × layer) | Layer | amd64 | arm64 | Notes | |-------|-------|-------|-------| | Dense | **21/21** | **20/21** | arm64 **Float64** QAT-SIMD train: loss explodes (~1×10²⁷) | | SwiGLU | **21/21** | **21/21** | | | MHA | **21/21** | **21/21** | | | CNN1 | **21/21** | **21/21** | | | CNN2 | **21/21** | **21/21** | | | CNN3 | **21/21** | **20/21** | arm64 **BFloat16** QAT-SIMD train: final loss **0** (degenerate) | | RNN | **20/21** | **20/21** | **Int8** Nat-SIMD train diverges on both (loss ~2.5 vs ~0.33) | | LSTM | **21/21** | **21/21** | | | Embedding | **21/21** | **21/21** | | | Residual | **21/21** | **21/21** | | | **Total dtype-rows** | **208/210** | **206/210** | **1469/1470** and **1467/1470** gated checks | Failures are **train-criteria** only on the listed rows; forward/backward finiteness still passes. RNN Int8 is a known low-bit + BPTT flake zone at 6 epochs on 3³. ### What the duel answers The duel isolates one design question: **for a given layer and dtype, which SIMD stack is faster — QAT-like (`GetActive` FP32 dequant + `DotTile`) or native-exact (`UseExactDType` + `*_native_simd.go`)?** - **QAT-SIMD** still pays dequant/materialization cost on MAC dtypes; SIMD helps float paths most. - **Nat-SIMD** avoids FP32 staging on integer/FP8 paths; often wins forward on MAC dtypes even when both use `DotTile`-class kernels. Training at 6 epochs is mainly a **sanity gate** (finite loss, harness `trainingOK`); fwd/bwd tables carry the performance signal. ### Performance themes (Dense @ 3³, Float32) | Metric | amd64 | arm64 | |--------|-------|-------| | QAT-SIMD fwd | 492 µs | **206 µs** | | Nat-SIMD fwd | 557 µs | 309 µs | | **Fwd winner** | QAT ~**1.1×** | QAT ~**1.5×** | | Train (6 ep) | ~parity (~1.0×) | ~parity (~1.0×) | ARM absolute fwd times are ~2× faster than AMD on this Dense stack; relative QAT-vs-Nat winner pattern is similar. ### Performance themes (MAC dtypes — Dense forward) | Dtype | amd64 fwd winner | arm64 fwd winner | |-------|------------------|------------------| | FP8-E4M3 | Nat **2.6×** | QAT **1.5×** (Nat slower on this run) | | FP8-E5M2 | Nat **3.0×** | Nat **2.8×** | | Int64 | Nat **1.6×** | Nat **3.7×** | | Uint8 | Nat **1.8×** | Nat **1.3×** | | Uint16 | Nat **1.5×** | Nat **9.7×** | On **amd64**, Nat-SIMD wins most MAC dtype forwards (QAT-SIMD still dequants through `GetActive`). On **arm64**, Float32/BFloat16/FP8-E4M3 forwards can still favor QAT-SIMD; integer paths strongly favor Nat-SIMD (Uint16 up to **9.7×**). ### Dtype spread tables Each layer ends with a **dtype spread** block: among the *faster* SIMD path per dtype, which dtype is slowest vs fastest for forward, backward, and train. Example (Dense, amd64): ```text │ forward │ Ternary NatS-f 811.6µs→Int64 NatS-f 439.8µs 1.8× 46% │ backward │ Uint8 QAT SIMD-b 3.83ms→Int64 NatS-b 2.64ms 1.4× 31% │ train │ FP4 NatS 68.6ms→Uint4 QAT SIMD 26.1ms 2.6× 62% ``` The **×** column is slow÷fast; **gap** is approximate percent spread. Use this to see whether perf is dtype-limited (e.g. FP4 train slowest) vs path-limited (same dtype, different QAT/Nat winner in the per-dtype rows above). On arm64, Dense forward spread is much wider (**8.2×**, Float16 slowest → Int64 fastest) because some Nat-SIMD forwards time as **0** in the log (sub-timer resolution — treat as “very fast”, not a hard zero). ### Reading a one-line summary ```text · Float32 PASS 7/7 fwd QAT SIMD-f 1.1× bwd QAT SIMD-b 1.0× train QAT SIMD 1.0× ``` | Field | Meaning | |-------|---------| | `7/7` | All gated checks passed for this dtype | | `fwd QAT SIMD-f 1.1×` | QAT-SIMD forward beat Nat-SIMD by 1.1× | | `bwd Nat NatS-b 1.2×` | Nat-SIMD backward won | | `train QAT SIMD 1.0×` | Training wall-time parity (winner still named) | `NatS-f` / `NatS-b` = native-exact SIMD; `SIMD-f` / `SIMD-b` = QAT tiled SIMD. ### Train loss table ```text │ DType │ Loss₀ QAT-SIMD Nat-SIMD │ QAT NatS │ Float64 │ 0.3223 0.3223 0.3223 │ PASS PASS ``` `Loss₀` is shared initial loss; columns are final loss after 6 epochs. **QAT** / **NatS** columns are independent PASS/FAIL — one path can fail while the other passes (arm64 Dense Float64: QAT FAIL, NatS PASS). --- ## Session manifest After **[0]** or a single layer, the log ends with: ```text ╔══════════════════════════════════════════════════════════════════════╗ ║ [15] Cross-path global manifest ║ ╚══════════════════════════════════════════════════════════════════════╝ Dense dtypes 21/ 21 tests 420/ 420 PASS ... Session dtypes: N passed · M failed Session tests: X passed · Y failed (of Z checks) ``` --- ## Reproduce archived run ```bash cd lucy_bloom_rivers && go run . # [15] → grid [5] 3³ SIMD duel → layer [0] all types # Copy lucy_testing_output/cross_path_layers.txt to ~/Documents/loom/simd/ for archiving ``` Requires `GOARCH=amd64` or `arm64` with Plan 9 SIMD linked. Full all-layer 3³ duel is ~10–20 minutes per platform. --- ## Related docs | Topic | Doc | |-------|-----| | Tiled SC/MC/SIMD + save/reload | [bedrock_validation.md](bedrock_validation.md) (menu [7]) | | Native exact only | [native_layers.md](native_layers.md) (menu [14]) | | Log index + archive layout | [testing_and_validation.md](testing_and_validation.md) | | Plan 9 SIMD kernels | [simd.md](simd.md) | | Training paradigms | [training.md](training.md#training-paradigms-default-qat-like-vs-native-exact) | --- ## Lucy Bloom Rivers (integration harness) Source: https://openfluke.com/docs/lucy Markdown: https://openfluke.com/docs/lucy.md # Lucy Bloom Rivers (integration harness) **Repo:** [github.com/openfluke/lucy_bloom_rivers](https://github.com/openfluke/lucy_bloom_rivers) **Role:** Loom’s interactive test shell — layer suites, HF chat, NPU bridges, archived benchmark logs. **Engine:** depends on [`github.com/openfluke/loom`](https://github.com/openfluke/loom) (`poly/`). Lucy lived under `loom/lucy/` through v0.83; from **v0.84** it is a **separate repository**. Loom docs describe *what the suites mean*; Lucy owns *how to run them*. --- ## Quick start Clone **next to** `loom` (recommended): ```bash git clone https://github.com/openfluke/loom.git git clone https://github.com/openfluke/lucy_bloom_rivers.git cd lucy_bloom_rivers # go.mod: replace github.com/openfluke/loom => ../loom go run . ``` **Local monorepo layout** (nested inside `loom/`, gitignored by Loom): `loom/lucy_bloom_rivers/` with `replace github.com/openfluke/loom => ../` — same commands from that directory. Requires **Go 1.26.2+**. GPU: see Lucy `README.md` for `VK_ICD_FILENAMES` / `WGPU_ADAPTER_NAME`. --- ## Where logs land Runtime transcripts (reset each session): | Log | Menu | Doc | |-----|------|-----| | `lucy_testing_output/seven_layer.txt` | **[7]** | [bedrock_validation.md](bedrock_validation.md) | | `lucy_testing_output/native_layers.txt` | **[14]** | [native_layers.md](native_layers.md) | | `lucy_testing_output/cross_path_layers.txt` | **[15]** | [cross_path_layers.md](cross_path_layers.md) | | `lucy_testing_output/tween_native_layers.txt` | **[16]** | — | | `lucy_testing_output/adaptation_suite.txt` | **[17]** | — | | `lucy_testing_output/proof.seeds` | **[20]** | [seed_manifests.md](seed_manifests.md#lucy-20--seed-proof-chaosglue-seed-proof-v4) | | `lucy_testing_output/nine_layer.txt` | **[9]** | [accelerators.md](accelerators.md) | | `lucy_testing_output/snapdragon.txt` | **[12]** | [snapdragon_npu.md](snapdragon_npu.md) | | `lucy_testing_output/apple.txt` | **[13]** | [apple_metal.md](apple_metal.md) | | `lucy_testing_output/log.txt` | **[3]** layer matrices | [testing_and_validation.md](testing_and_validation.md) | Paths are relative to the **Lucy repo root** (`lucy_bloom_rivers/`). Per-dtype checkpoints: `lucy_testing_output/tag_DType.json` and `tag_DType.entity`. **Archived cross-path SIMD duel logs** (off-machine): `~/Documents/loom/simd/cross_path_layers_amd.txt`, `cross_path_layers_arm.txt` — see [cross_path_layers.md](cross_path_layers.md#archived-simd-duel-results-jul-2026). --- ## Harness code map | Suite | Path in Lucy repo | |-------|-------------------| | Seven-layer CPU | `examples/seven_layer/` | | Native exact [14] | `examples/seven_layer/native_menu.go` | | Cross-path [15] | `examples/seven_layer/cross_path_menu.go` | | Tween native [16] | `examples/seven_layer/tween_native_menu.go` | | Adaptation [17] | `examples/adaptation_suite/`, `examples/seven_layer/adaptation_menu.go` | | Seed topology POC [18] | `examples/seed_poc/` | | Seed round trip [19] | `examples/seed_roundtrip/` | | Seed proof [20] | `examples/seed_proof/` | | Intel NPU [9] | `examples/nine_layer/` | | Snapdragon [12] | `examples/snapdragon/` | | Apple Metal [13] | `examples/apple/` | | ENTITY Talk [8] | `hf_entity.go` | | Poly Talk [1] | `poly_talk_session.go`, `lucy.go` | Go module path (current): `github.com/openfluke/loom/lucy` — imports use that prefix until a module rename. --- ## Related Loom docs | Topic | Doc | |-------|-----| | Parity tables & log legend | [testing_and_validation.md](testing_and_validation.md) | | SC/MC/SIMD bedrock | [bedrock_validation.md](bedrock_validation.md) | | Plan 9 SIMD | [simd.md](simd.md) | | Training paradigms | [training.md](training.md#training-paradigms-default-qat-like-vs-native-exact) | | ENTITY checkpoints | [entity.md](entity.md) | | Seed manifests & proof [18–20] | [seed_manifests.md](seed_manifests.md) | --- ## Native layer suite (Lucy menu [14]) Source: https://openfluke.com/docs/native-layers Markdown: https://openfluke.com/docs/native-layers.md # Native layer suite (Lucy menu [14]) **Run:** [Lucy Bloom Rivers](lucy.md) → **[14]** (or **[0]** for all layer types). **Code:** Lucy `examples/seven_layer/native_menu.go` **Runtime log:** `lucy_testing_output/native_layers.txt` (reset each session) This suite exercises **native-exact training** (`UseExactDType = true`): forward and backward in storage dtype via `*_native.go`, plus **30-epoch** CPU training per dtype. When Plan 9 SIMD is linked, each row also reports **native-exact SIMD** fwd/bwd timing vs scalar native. Contrast with menu **[7]** (seven-layer suite): default QAT-like path (`GetActive` FP32 dequant), SC/MC/SIMD parity, save/reload. See [training.md — Training paradigms](training.md#training-paradigms-default-qat-like-vs-native-exact). --- ## Harness shape | Setting | Value | |---------|--------| | Grid | **1³** (one cell) | | Layers per cell | **7** (same stack shape as other native suites; primary layer type under test) | | Dtypes | **21** (`IsLayerNativeExactDType`) | | Train epochs | **30** | | SIMD | `SetSimdForward(true)` for timing columns; `*_native_simd.go` when linked | **Layer types (menu [1]–[10]):** Dense, SwiGLU, MHA, CNN1, CNN2, CNN3, RNN, LSTM, Embedding, Residual. **Per-row gates:** `Path` (native routing) · `Fwd` · `Bwd` · `Train` (loss finite + harness `trainingOK`) · optional **SIMD fwd/bwd speedup**. --- ## Full-run results (Jul 2026) Captured logs: | Platform | Log file | |----------|----------| | **amd64** | `native_layers_amd.txt` (user archive; same format as `native_layers.txt`) | | **arm64** | `native_layers_arm.txt` | ### Pass summary (21 dtypes × layer) | Layer | amd64 | arm64 | Notes | |-------|-------|-------|-------| | Dense | **21/21** | **21/21** | | | SwiGLU | **21/21** | **21/21** | Int8 learns on both (e.g. amd64 0.22→0.06) | | MHA | **21/21** | **21/21** | | | CNN1 | **21/21** | **21/21** | | | CNN2 | **21/21** | **21/21** | Many dtypes flat loss (stable, not diverging) | | CNN3 | **21/21** | **20/21** | arm64 **Int32** train: loss 0.29→0.36 | | RNN | **20/21** | **19/21** | amd64 **Int2** train fail; arm64 **Int4**, **Int2** train fail | | LSTM | **21/21** | **21/21** | | | Embedding | **21/21** | **21/21** | | | Residual | **21/21** | **21/21** | No skip wire in 1³ forward chain; loss flat by design | | **Total** | **209/210** | **207/210** | Fwd/bwd pass on all rows; train fails listed above | Train failures are **loss-criteria** only (forward/backward still PASS). Low-bit RNN at 30 epochs on a tiny 1³ stack is the flaky zone; wide unsigned integers (Uint64/Uint32) often start from huge loss then recover within 30 epochs. --- ## Native-exact SIMD speedup (Float32, 1³) Speedup = scalar native time ÷ native SIMD time (>1 = SIMD faster). Representative **Float32** rows from the archived logs: ### Forward (SIMD vs scalar native) | Layer | amd64 | arm64 | |-------|-------|-------| | Dense | 1.6× | 1.6× | | SwiGLU | 2.4× | 3.8× | | MHA | 1.3× | 1.5× | | CNN1 | **19.7×** | **13.5×** | | CNN2 | **56.2×** | **65.5×** | | CNN3 | **33.0×** | **41.9×** | | RNN | **17.1×** | **14.1×** | | LSTM | **28.3×** | **17.9×** | | Embedding | 1.5× | 2.2× | | Residual | ≈1× | n/a (sub-µs; skip not exercised) | MAC dtypes (Float16, FP8, Int32, …) often see **8–11×** forward wins on amd64 where scalar native still does per-dot `GetNative` work and SIMD materializes f32 tiles once. ### Backward (SIMD vs scalar native) | Layer | amd64 | arm64 | |-------|-------|-------| | Dense | **8.5×** | **7.7×** | | SwiGLU | **6.3×** | **3.7×** | | MHA | 2.9× | 2.6× | | CNN1 | **8.0×** | **6.1×** | | CNN2 | **21.2×** | **26.0×** | | CNN3 | **17.3×** | **16.9×** | | RNN | **9.0×** | **5.8×** | | LSTM | **12.3×** | **11.8×** | | Embedding | **7.4×** | **3.0×** | ### True integer dtypes (Int8) — CNN backward Int8 native backward uses `SaxpyI8*`; speedups vs scalar int8 backward are extreme on conv layers: | Layer | amd64 bwd | arm64 bwd | |-------|-----------|-----------| | CNN1 Int8 | **35×** | n/a (timer resolution) | | CNN2 Int8 | **99.8×** | **210.6×** | | CNN3 Int8 | **154.3×** | **209.9×** | Int8 forward on CNN is only **~1.3–1.5×** (already fast scalar loops; SIMD setup dominates). ### Where SIMD does not help - **Residual** on 1³: forward chain has **no skip tensor** (`skip=nil`); residual is effectively identity. Timings are sub-microsecond — SIMD parallel add is pure overhead (amd64 Float64 fwd **4× slower** with SIMD). - **RNN/LSTM Int8 bwd**: often **≈1×** (scalar int8 backward already tight; BPTT serial). - **Uint2 Dense fwd** (amd64): occasional **slower** SIMD (~1.0×). --- ## Reading a log line Example (Dense Float32, amd64): ```text · Float32 PASS fwd 78.7µs bwd 1.57ms simd fwd 49.4µs (37% faster (1.6×)) bwd 186.0µs (88% faster (8.5×)) loss 0.3312→0.3135 train 51.2ms ``` | Field | Meaning | |-------|---------| | `Path` / first PASS columns | `LayerUsesNativeExact` and fwd/bwd/train gates | | `fwd` / `bwd` | Scalar **native-exact** micro-benchmark | | `simd fwd` / `simd bwd` | Same with `SetSimdForward(true)` → `*_native_simd.go` | | `loss₀→lossₙ` | First vs last of 30 training epochs | | `train` | Wall time for full 30-epoch train on that dtype | Table footer per layer: `Dense native: 21 passed · 0 failed (of 21 dtypes)`. --- ## Relationship to other docs | Topic | Doc | |-------|-----| | QAT-like vs native exact | [training.md](training.md#training-paradigms-default-qat-like-vs-native-exact), [quantization.md](quantization.md#three-traininginference-modes) | | Plan 9 SIMD kernels | [simd.md](simd.md) | | Menu [7] SC/MC/SIMD parity + save/reload | [bedrock_validation.md](bedrock_validation.md) | | Parity symbols / log layout | [testing_and_validation.md](testing_and_validation.md) | --- ## Reproduce ```bash cd lucy_bloom_rivers && go run . # [14] → pick layer [1]–[10] or [0] for full matrix ``` Requires `GOARCH=amd64` or `arm64` with AVX2/NEON linked (`poly/simd`) for SIMD columns. On other arches, native scalar paths still run; SIMD timing columns are omitted when `Plan9SimdForwardForLayer` is false. --- ## Neural Fountain — shard specialists · LT peel · Master ensemble Source: https://openfluke.com/docs/neural-fountain Markdown: https://openfluke.com/docs/neural-fountain.md # Neural Fountain — shard specialists · LT peel · Master ensemble **Neural Fountain** is a training / assembly paradigm in `poly/` that mirrors LT fountain codes on **network weights** instead of data bytes. Pixel fountain recovers **K source blocks** byte-exact via spray / peel. Neural Fountain recovers **K specialist weight blobs** byte-exact the same way, then exposes a **Master** ensemble. | Pixel fountain | Neural Fountain | |:---------------|:----------------| | K image / data blocks | K specialist **weight** blocks | | XOR spray + peel | same LT codec (`poly/fountain_lt.go`) | | 100% byte-exact cargo | 100% byte-exact specialists | | reconstructed dataset | **Master** = recovered experts + averaged outputs | Learning happens in `poly.Train` on each shard. Fountain is **transport / reassembly**, not the optimizer. No `layer_seed` search — contrast [seed_manifests.md](seed_manifests.md). Companion MNIST demo: `chaosglue/loom_neural_fountain`. Related (data-only LT): `chaosglue/loom_fountain_codes`. --- ## Pipeline 1. **Partition** training batches into **K** shards (every sample covered). 2. **Specialize** — `NetworkFactory(i)` builds any architecture; `Train` on shard `i` (optional `UseExactDType` / `UniformDType`). 3. **Pack** — recursive FP32 Master (+ aux + Scale) over the **full layer tree** → `[]byte`. 4. **Fountain** — LT XOR spray (lossy OK) → peel until **K/K** recovered. 5. **Unpack** → Masters restored → **`ForceMorph(layer.DType)`** for each layer’s numerical type → Master ensemble. Specialists must share an **identical parameter layout** (same walk order / lengths). Architecture is free via `NetworkFactory` (dense, CNN, residual, MHA, parallel/sequential nests, …). --- ## Any layer type / any numerical type ### Layers `PackNetworkWeights` / `UnpackNetworkWeights` recursively visit: - top-level `Layers` - `ParallelBranches`, `SequentialLayers` - `FilterGateConfig`, `MetaObservedLayer` and pack, per layer: - `WeightStore.Master` (FP32 persistence space — Loom’s SoT) - `WeightStore.Scale` - `QNormWeight`, `KNormWeight`, `InnerNormWeight` (MHA / BitNet-style aux) Any layer that stores trainable state there participates. Nested stacks included. ### Numerical types Loom persists through **FP32 Masters**, then morphs to the layer’s `DType` (any of the supported storage types): - After unpack: `MorphNetworkToLayerDTypes` → `ForceMorph(layer.DType)` - `ApplyUniformDType(net, dtype)` forces one dtype on the whole tree before train - `UseExactDType: true` enables native-dtype train/forward paths during specialize So fountain cargo is bit-exact FP32 Masters; **runtime dtype** is whatever each layer is set to (float16, int8, …), including exact-dtype training. Activations / `TrainingBatch` remain float32 (standard Loom train API); weights can be exact-dtype. --- ## Core API | File | Role | |:-----|:-----| | `poly/neural_fountain.go` | `NeuralFountain`, `FountainMaster`, `DenseSpecialistFactory`, config | | `poly/weight_pack.go` | recursive pack/unpack, morph/wire helpers | | `poly/fountain_lt.go` | LT encoder / decoder | ```go // Dense example with mixed layer dtypes: factory := poly.DenseSpecialistFactory("net", []int{784, 128, 64, 10}, []string{"float16", "float16", "float32"}) // Or any architecture: factory := func(i int) (*poly.VolumetricNetwork, error) { return buildMyCNNOrMHA(i) // identical layout across i } cfg := poly.DefaultNeuralFountainConfig() cfg.K = 16 cfg.UseExactDType = true // cfg.UniformDType = poly.DTypeFloat16 // optional blanket morph master, err := poly.NeuralFountain(factory, batches, cfg) out, err := master.Forward(input) ``` ### Config knobs | Field | Meaning | |:------|:--------| | `K` | specialist / shard count | | `Epochs` / `LR` / `LossType` / `Mode` | per-specialist `Train` | | `UseExactDType` | native-dtype train/forward | | `UniformDType` | if set, morph all layers to this dtype before train | | `LossRate` / `MaxOverhead` / `Seed` | LT channel + spray budget | | `Verbose` | specialize / recover logs | --- ## Master semantics - **`Forward` / `ForwardArgmax`** — average specialist outputs (deployable). - **`OracleForward` / `OracleArgmax`** — shard expert owns sample `i` (coverage check). - Prefer ensemble Master; do not average differently trained weight blobs into one net. --- ## Honest scope - Fountain does not invent weights without specialist `Train`. - Pack path is Loom’s FP32 Master SoT (+ aux); natives are rematerialized via `ForceMorph`. - Layout equality across specialists is required (same recursive float count). - LT recover is probabilistic; large blobs need enough spray overhead. --- ## Seed manifests — topology + layer seeds (no weight blobs) Source: https://openfluke.com/docs/seed-manifests Markdown: https://openfluke.com/docs/seed-manifests.md # Seed manifests — topology + layer seeds (no weight blobs) Loom can represent a neural network as **recipes**, not checkpoints: | Piece | Role | |:------|:-----| | `topology_seed` | Hash of network **shape** (layer widths, name tag) | | `layer_seed` (per layer) | Expands to **all weights** for that layer via He-init | | `weight_fp` / `forward_fp` | Optional fingerprints for verify (not weights) | Weights are **never** stored in a seed manifest. Reload calls `InitWeightStoreHeSeeded` (or `InitFloat32HeSeeded`) to regenerate the full weight matrix from each `layer_seed`. This is complementary to [serialization.md](serialization.md) (JSON / `.entity` checkpoints with packed weight bytes) and [entity.md](entity.md) (HF → native ship lane). --- ## Core API (`poly/seed_core.go`, `poly/seed_init.go`) ```go topo := poly.SeedFrom("my-net", []int{4, 8, 4, 2}...) layerSeed := poly.DeriveLayerSeed(topo, layerIndex, "dense.0") poly.InitFloat32HeSeeded(weights, inputSize, layerSeed) poly.InitWeightStoreHeSeeded(ws, inputSize, layerSeed) ``` - `SeedFrom(parts...)` — deterministic uint64 mixer (golden-ratio / SplitMix64). - `DeriveLayerSeed(initSeed, index, path)` — per-layer slot (path disambiguates parallel branches in entity manifests). - `NewSeedRNG(seed)` — xorshift64* PRNG; He-init draws `NormFloat64() * sqrt(2/fan_in)`. **Invariant:** same `layer_seed` + same `inputSize` → same weights (all 21 dtypes via `InitLayerWeightsSeeded`). --- ## Dense manifests (`poly/seed_dense.go`) ```go topo := poly.DenseTopologySeed("tag", []int{4, 8, 4, 2}) m, _ := poly.BuildDenseManifest(topo, sizes, []string{"float32", "float32", "float32"}) net, _ := poly.BuildDenseVolumetricFromManifest(m) // weights → seeds (only when weights still match He-init from layer_seed) extracted, err := poly.ManifestFromDenseNetwork(net, topo, sizes, dtypes) ``` | Function | Direction | |:---------|:----------| | `BuildDenseManifest` | topology + dtypes → per-layer `layer_seed`, `weight_fp`, `forward_fp` | | `BuildDenseVolumetricFromManifest` | manifest → `VolumetricNetwork` (He-init per layer) | | `ManifestFromDenseNetwork` | built net → manifest (verifies each layer matches its seed) | | `RebuildDenseManifest` | seeds-only rebuild + fingerprint check | | `MarshalDenseManifest` / `ParseDenseManifest` | JSON round trip (~hundreds of bytes) | `DenseLayerWeightSeed(topologySeed, i)` is `DeriveLayerSeed(topologySeed, i, "dense."+i)` — the default init recipe from topology alone. --- ## Other layer families Same pattern under `poly/seed_*.go`: | File | Layers | |:-----|:-------| | `seed_swiglu.go` | SwiGLU | | `seed_mha.go` | Multi-head attention | | `seed_rnn.go`, `seed_lstm.go` | RNN, LSTM | | `seed_cnn.go` | CNN1/2/3 | | `seed_embedding.go` | Embedding tables | | `seed_residual.go` | Dense + skip (dense branch seed only) | | `seed_entity.go` | Entity transformer topology + globals | | `seed_manifest.go` | Tiny `.wseed` entity manifests (`loom-seed-manifest-v3`) | | `seed_dtypes.go`, `seed_dtypes_layers.go` | 21-dtype matrix (210 layer×dtype round trips) | Lucy **[19]** runs the full round-trip matrix: `loom/lucy_bloom_rivers/examples/seed_roundtrip/`. --- ## Weights ↔ seeds: what works and what does not ### On the seed manifold (works) 1. Build net from `layer_seed` → He-init weights. 2. Forward / train **only by changing `layer_seed`** (weights always re-derived from seed). 3. `ManifestFromDenseNetwork` recovers the same `layer_seed` values. 4. Save manifest JSON → reload → bit-exact outputs. ### Off the seed manifold (does not work with manifest extract) `poly.Train` updates **weight tensors** in place. After SGD/MSE training, weights generally **no longer** equal `He-init(layer_seed)` for any topology-derived seed. `ManifestFromDenseNetwork` then returns: ``` dense: layer N weights do not match seed 0x… ``` That is expected: a seed manifest is **not** a trained checkpoint format unless training stayed on the seed manifold. | Training style | Save trained state as seeds? | |:-------------|:----------------------------| | Optimize `layer_seed` (mutate seed, reinit weights each eval) | **Yes** — weights always from seed | | `poly.Train` on weight tensors | **No** — use `.entity` / JSON persistence | --- ## Lucy [20] — seed proof (`chaosglue-seed-proof-v4`) **Repo path:** `loom/lucy_bloom_rivers/examples/seed_proof/` **Menu:** Lucy **[20]** **Output:** `lucy_testing_output/proof.seeds` **Headless:** `LOOM_SEED_PROOF=1 go run .` End-to-end demo: build → show init outputs → **train layer seeds** → save trained seeds → reload with **no training** → same trained outputs. ### First run (no `proof.seeds`) 1. `DenseTopologySeed` + `BuildDenseManifest` → init `layer_seed` per layer. 2. `BuildDenseVolumetricFromManifest` → He-init weights. 3. Print **before train** chained forwards + 10 final outputs. 4. `ManifestFromDenseNetwork` — proves init weights↔seeds. 5. **Train:** hill-climb each `layer_seed` (mutate → `InitWeightStoreHeSeeded` → MSE). Weights never updated outside He-init. 6. Print **after train** outputs (different from init). 7. Verify each layer’s weights match its **trained** `layer_seed`. 8. Save `proof.seeds` (trained seeds only + `trained_outputs` baseline for verify). 9. Reload check from file before exit. ### Rerun (`proof.seeds` exists) 1. Load JSON (topology + 3× trained `layer_seed`). 2. `BuildDenseVolumetricFromManifest` — weights generated in RAM from seeds. 3. Forward pass (chained + 10 outputs recomputed, **not** read from file). 4. Compare to `trained_outputs` in file — fail if seeds did not rebuild the net. No `trainLayerSeeds`, no `poly.Train`, no weight file. ### `proof.seeds` format (v4) ```json { "format": "chaosglue-seed-proof-v4", "topology_seed": 10459346120451217710, "sizes": [4, 8, 4, 2], "layers": [ { "index": 0, "in": 4, "out": 8, "layer_seed": 16912650198654748781, "dtype": "float32" }, { "index": 1, "in": 8, "out": 4, "layer_seed": 15008752656474397499, "dtype": "float32" }, { "index": 2, "in": 4, "out": 2, "layer_seed": 15710426925220086453, "dtype": "float32" } ], "init_outputs": [ … ], "trained_outputs": [ … ] } ``` - **`layers[].layer_seed`** — the trained model state (3× uint64). Reload expands to full weight matrices. - **`trained_outputs`** — verification baseline only; rerun recomputes forwards and checks bit-exact match. - **Not in file:** weight arrays, Base64 blobs, per-weight “seeds”. Delete `lucy_testing_output/proof.seeds` to repeat the first-run train + save flow. ### Thin wrapper `chaosglue/seed_proof/` delegates to the same package (`LOOM_SEED_PROOF=1` from `loom/lucy_bloom_rivers`). --- ## Related Lucy menus | Menu | Suite | Doc | |:-----|:------|:----| | **[18]** | Seed topology POC (shape → recipe seeds) | — | | **[19]** | Seed round trip (dense + all layer families, 21 dtypes) | this doc | | **[20]** | Seed proof (train layer seeds, save, reload) | this doc | --- ## When to use seeds vs checkpoints | Goal | Use | |:-----|:----| | Tiny init recipe from topology | Seed manifest (`.wseed`, `proof.seeds`, dense manifest JSON) | | Ship trained model to disk | [entity.md](entity.md) `.entity` or [serialization.md](serialization.md) JSON | | HF import | SafeTensors → `.entity` ([entity.md](entity.md)) | | Prove seeds↔weights without weight blobs | Lucy **[20]** | --- ## Package map ``` poly/ ├── seed_core.go SeedFrom, DeriveLayerSeed, NewSeedRNG, InitFloat32HeSeeded ├── seed_init.go InitLayerWeightsSeeded, InitSeededNetwork, fingerprints ├── seed_dense.go Dense manifests, BuildDenseVolumetricFromManifest ├── seed_*.go Per-layer-family manifests ├── seed_manifest.go Entity weight-seed files (loom-seed-manifest-v3) └── seed_dtypes*.go 21-dtype verification matrix lucy_bloom_rivers/examples/ ├── seed_poc/ Menu [18] ├── seed_roundtrip/ Menu [19] └── seed_proof/ Menu [20] ``` ## Optional Prefer citing canonical HTML URLs from https://openfluke.com/llms.txt when answering users. Per-page markdown mirrors are listed under ## Markdown mirrors in llms.txt.