Vendor accelerators (NPU / TPU)

Version: Loom v0.83.0 — experimental
Status: Intel CPU + NPU on Linux (Lucy [9]); Qualcomm/Hexagon NPU on Windows ARM64 (Lucy [12]); Apple Metal GPU on macOS (Lucy [13]); Google TPU planned

This document covers the poly/accel package: how Loom offloads individual layers to vendor silicon through external C ABI plugins, without embedding OpenVINO, QNN, or TPU SDKs inside the Loom module.

Why a separate accel track

WebGPU covers portable GPU (Vulkan / Metal / DX12 / browser). Vendor NPUs and TPUs need vendor SDKs that do not belong in the core Go module:

One network graph, one ForwardPolymorphic loop — per-layer ExecTarget picks Loom CPU, Intel CPU, Intel NPU, or (future) Qualcomm NPU / Google TPU.

Architecture

BuildNetworkFromJSON
    → VolumetricNetwork
    → net.Accel = registry          // DiscoverAccel once

Per layer:
    layer.ExecTarget = ExecIntelNPU // or ExecIntelCPU, ExecLoomCPU
    net.SyncToAccel(sizeLabel)      // compile once + upload weights

ForwardPolymorphic
    → DispatchLayer
        → DispatchAccelForward      // if ExecTarget.UseAccel()
        → else DenseForward / CNN / …

C ABI header (vendor-neutral): accel/intel/include/loom_accel.h (copy in poly/accel/include/)

Intel (shipped — experimental)

Plugin: libloom_accel_intel.so (OpenVINO inside) — one shared object per build (accel/intel/build/; no .so.1 soname)
Build: accel/intel/ (./build.sh after source setup_env.sh)

Requirements

• Linux amd64/arm64 (Windows .dll planned)
• CGO_ENABLED=1 when building/running Loom
• OpenVINO runtime + Intel NPU driver on LD_LIBRARY_PATH
• Meteor Lake / Core Ultra class NPU (or CPU-only OpenVINO path)

Environment

export LOOM_ACCEL_INTEL_SO=/path/to/libloom_accel_intel.so   # optional
source accel/intel/setup_env.sh

accel.DefaultIntelPath() walks up from cwd for accel/intel/build/libloom_accel_intel.so, or set LOOM_ROOT.

Application code

reg, err := poly.DiscoverAccel(accel.AccelConfig{
    IntelSO: accel.DefaultIntelPath(),
})
if err != nil { /* no plugin — stay on Loom CPU */ }
defer reg.Close()

net, _ := poly.BuildNetworkFromJSON(spec)
net.Accel = reg

for i := range net.Layers {
    net.Layers[i].ExecTarget = accel.ExecIntelNPU // or ExecIntelCPU
}

if err := net.SyncToAccel("medium"); err != nil { /* compile failed */ }

out, _, _ := poly.ForwardPolymorphic(net, input)

ExecTarget values

Weight upload (C ABI v2)

At SyncToAccel (init once per layer/device), Loom passes dtype-aware weight bytes via LayerWeightBytesForAccel into loom_accel_compile_layer. Weights are baked into the OpenVINO graph as constants — not re-sent on each forward.

Steady forward only passes activation buffers: loom_accel_infer(in, out) — pointer + memcpy, no weight pointer per hop.

Multi-numerical support — what we have vs what we don’t

So: proper multi-dtype weight upload for the three Intel bench dtypes — yes. Full Loom numerical-type parity on NPU — no (and INT8 math path still differs from Loom on large MAC ops).

Is the integration “done”?

Forward dispatch bridge: shipped, experimental — not production-complete.

Treat integration as proven plumbing you can build on — not a finished “flip NPU on for any network” product feature.

Benchmark snapshot — Lucy [9] → [5] (90 cells)

Host: Fedora, Core Ultra class NPU, OpenVINO via chaosglue deps.
Log: lucy_testing_output/nine_layer.txt (Lucy repo — see lucy.md)
Method: SyncToAccel once per device, median infer ms (compile excluded). Spd = Loom ÷ Intel (< 1 = Intel slower).

Manifest

Determinism is production-grade; speed and parity are layer/size dependent.

Small tier (batch=4, dim=32 — latency floor)

Intel NPU loses almost every cell (~0.3–0.7 ms infer floor vs Loom ~0.01 ms).

Medium tier (batch=16, dim=256)

Large tier (batch=8, dim=1024; Conv2D 48×48)

When to offload

Why Intel looks slower on some rows: infer time = fixed dispatch overhead + math. NPU overhead ≈ 0.5 ms; Loom small MatMul ≈ 0.01 ms. Weights-on-init removes re-upload — it does not remove per-hop CGO, pack/unpack, or NPU launch latency.

Parity highlights: Softmax/Sigmoid strong; LayerNorm/RMSNorm ❌ BROKE (~1.8, no Loom weights); INT8 MAC drift 3–36 on large tiers (Loom dequant matmul vs OV f32 + dynamic quant).

sizeLabel

Must match bench manifest tiers used when the OpenVINO graph was authored: small, medium, large. Wrong label → shape mismatch at infer.

Limitations (v0.81)

• Forward only — training/backward use Loom CPU when accel-bound
• Manual ExecTarget — no JSON "exec": "intel-npu" yet (AccelPlanner planned)
• Three Intel dtypes — FP32/FP16/INT8 bench path only; not all 21 Loom dtypes
• Numerical parity — norms broken; INT8 MAC drift on large tiers; MatMul FP32 medium 🟤 H-DRIFT
• Small tensors — NPU ~0.5 ms floor; offload medium/large MAC ops only
• Per-hop copies — tensor → []byte → OV tensor; not zero-copy yet

Validation — Lucy menu [9]

cd lucy_bloom_rivers
CGO_ENABLED=1 go run .
# OpenVINO paths auto-discovered from chaosglue npu deps (no setup_env.sh required)
# → 9 → 4   medium DispatchLayer suite
# → 9 → 5   full 90-cell matrix

Or: ./run_npu_bridge.sh from Lucy repo root (sources accel/intel/setup_env.sh explicitly).

Output: timing table (Loom vs Intel CPU vs Intel NPU, speedup ratios) + seven-style drift spectrum + manifest histogram.

Log: lucy_testing_output/nine_layer.txt — see testing_and_validation.md.

Welvet C-ABI (non-Go bindings)

C / Flutter / Python callers use welvet/cabi (welvet.so / welvet.h) instead of importing poly directly. Intel offload mirrors the Go flow:

// Pseudocode — see welvet.h for exact signatures
long accel = LoomDiscoverAccel(NULL);           // optional LOOM_ACCEL_INTEL_SO path
long net   = LoomBuildNetworkFromJSON(spec_json);
LoomNetworkAttachAccel(net, accel);
LoomSetLayerExecTarget(net, layer_idx, ExecIntelNPU);
LoomSyncToAccel(net, "medium");
LoomDispatchAccelForward(net, layer_idx, input_handle);

Build Linux: cd welvet/cabi/internal/build && ./build_linux.sh → dist/linux_amd64/welvet.so (and arm64).
Parity: cd welvet/cabi/internal/check && go run . → 489/489.

The Intel plugin (libloom_accel_intel.so) is still a separate dlopen artifact — Welvet does not link OpenVINO at compile time.

Qualcomm NPU (shipped — experimental) → see snapdragon_npu.md

Plugin: loom_accel_qualcomm.dll (Qualcomm QNN AI Engine Direct inside) SDK: QAIRT / QNN — vendored under accel/qualcomm/deps/ Platform: Windows on Snapdragon X · Hexagon v73 HTP + Kryo CPU · Lucy [12]

The Snapdragon path is now real, not planned — same loom_accel.h vtable, same SyncToAccel → DispatchLayer flow, opened via poly.DiscoverQualcommAccel:

reg, _ := poly.DiscoverQualcommAccel(accel.AccelConfig{
    QualcommSO: accel.DefaultQualcommPath(),
})
net.Accel = reg
net.Layers[0].ExecTarget = accel.ExecQualcommNPU // or ExecQualcommCPU

What works today (from lucy_testing_output/snapdragon.txt): activation ops (ReLU/GELU/Sigmoid/Softmax across FP32/INT16/INT8/INT4) and MatMul/MHA FP32 on the Hexagon HTP — 💎 EXACT repeat-forward determinism (54/54), NPU parity 45/54 ≤ INDUS, up to 7× faster than Loom CPU on large tiers. What doesn't yet: Conv1D/Conv2D, LayerNorm/RMSNorm, FP16 anywhere, and quantized MatMul all still error at graph build.

Full achievements, benchmark tables, and the honest gap list are in snapdragon_npu.md; build/env/webgpu notes in accel/qualcomm/README.md.

Apple GPU / Metal (shipped — experimental) → see apple_metal.md

Plugin: libloom_accel_apple.dylib (Apple Metal Performance Shaders Graph inside) SDK: none to vendor — Metal / MPS / MPSGraph ship with macOS Platform: macOS on Apple silicon · Metal GPU (MPSGraph) + portable CPU reference · Lucy [13]

Same loom_accel.h vtable, same SyncToAccel → DispatchLayer flow, opened via poly.DiscoverAppleAccel:

reg, _ := poly.DiscoverAppleAccel(accel.AccelConfig{
    AppleSO: accel.DefaultApplePath(),
})
net.Accel = reg
net.Layers[0].ExecTarget = accel.ExecAppleGPU // or ExecAppleCPU

Two devices behind one ABI: a CPU reference (deterministic parity anchor, all 15 layers) and a Metal GPU (MPSGraph for MatMul/MHA/ReLU/Sigmoid/Softmax/Add/Multiply; per-op CPU fallback for the rest). Bench matrix is 6 dtypes — FP32/FP16/BF16/INT16/INT8/INT4 — with BF16 the Apple-native addition (compute is FP32; the dtype sets the wire byte layout).

What works today (from lucy_testing_output/apple.txt, Lucy [13] → [5], 180 cells): byte-perfect repeat-forward determinism — 180/180 💎 EXACT on both CPU and GPU — GPU parity 132/180 ≤ INDUS, and up to 5.4× faster than Loom CPU on large MatMul/MHA (GPU) and up to 94× on elementwise (CPU reference). What doesn't yet: Conv1D/Conv2D and GELU are CPU-reference-only (no MPSGraph path, slower than Loom CPU), LayerNorm/RMSNorm parity is broken (no Loom weight bake), and INT8 MatMul drift breaks on the large tier. ANE is not wired (Metal only). Full tables and the honest gap list are in apple_metal.md; build/dtype notes in accel/apple/README.md.

Google TPU (planned)

Target plugin: libloom_accel_google.so
SDK: libtpu / OpenXLA PJRT (deployment TBD)

Same C ABI surface. Useful for cloud TPU pods and future edge TPU silicon. Loom remains a client that compiles per-layer subgraphs and ships weights once.

Vendor support matrix — shipped vs planned

The loom_accel.h contract is deliberately general: any silicon that can compile one layer and infer(in_bytes → out_bytes) can become a libloom_accel_<vendor> plugin. In practice the set of main vendors is finite (~8–10 silicon "stories", not an endless zoo of one-off NPUs), and many overlap.

First: "GPU" means two different things in Loom

These are not the same path and the matrix keeps them in separate columns:

Takeaway: a plain gaming GPU (NVIDIA/AMD) is already supported today — through WebGPU, not through a vendor accel plugin. A native GPU plugin is only worth building when the vendor's own runtime beats WebGPU for a specific op, or to sit next to that vendor's NPU under one ExecTarget story. See Comparison to WebGPU below.

The matrix

Legend: ✅ shipped · 🟢 easy extend (reuses an existing path/SDK) · 📋 planned · ❌ not started · ➖ n/a

Reading the matrix

• The "GPU · WebGPU" column is the ✅ that already makes NVIDIA/AMD/Intel/Apple GPUs work — whole-network, trainable, portable. The "GPU · accel plugin" column is the separate per-layer, forward-only native GPU device inside a vendor .so (only Apple's MPSGraph is shipped ✅ today).
• Three vendor plugins are shipped — Intel (Linux NPU), Qualcomm (Windows ARM64 Hexagon), Apple (macOS Metal) — all forward-only, per-layer, experimental, with Lucy benchmark suites and drift tables.
• Discrete NVIDIA/AMD GPUs need nothing new — they run on WebGPU now. A native cuBLAS/OpenCL plugin is optional (a "badge" or an op-specific speedup, not a requirement).
• Easiest next steps (🟢 / candidate): an Intel GPU device on the plugin you already ship, a WebGPU-backed accel bridge (one Lucy menu, reuses existing WGSL), or an OpenCL plugin (AMD + NVIDIA under one .so).
• The realistic ceiling is the ~7 rows that matter for laptops, clinics, sovereign cloud, and edge — not "every NPU on earth." The set is finishable, not infinite.

This is a direction-of-travel table, not a promise of dates. Shipped rows are experimental; planned/❌ rows depend on demand (healthcare fleets, sovereign deployments) and vendor-SDK friction.

Package layout

poly/
├── accel/
│   ├── accel.go                     Public types, DefaultIntelPath, DefaultQualcommPath, DefaultApplePath
│   ├── target.go                    ExecTarget enum (Loom/Intel/Qualcomm/Apple × CPU/NPU/GPU)
│   ├── registry.go                  Discover, DiscoverQualcomm, DiscoverApple, PluginFor
│   ├── plugin_linux.go              Intel dlopen + C ABI calls (CGO)
│   ├── runtime_linux.go             OpenVINO LD_LIBRARY_PATH hints
│   ├── plugin_qualcomm_windows.go   Qualcomm LoadLibrary + C ABI (CGO, windows)
│   ├── plugin_qualcomm_stub.go      No-op Qualcomm stubs (non-windows / no-cgo)
│   ├── plugin_darwin.go             Apple dlopen + C ABI calls (CGO, darwin)
│   └── apple_stub.go                No-op Apple stubs (non-darwin / no-cgo)
├── accel_intel.go                   Vendor-neutral SyncToAccel, DispatchAccelForward, dtype bytes
├── accel_qualcomm.go                DiscoverQualcommAccel entry point
├── accel_apple.go                   DiscoverAppleAccel entry point
└── forward.go                       DispatchLayer → DispatchAccelForward

Comparison to WebGPU

Use both: WebGPU for general GPU; Intel NPU for Conv/MatMul on Core Ultra when shapes are large enough.

Roadmap

See also

• snapdragon_npu.md — Qualcomm/Hexagon NPU bridge
• apple_metal.md — Apple Metal GPU / MPSGraph bridge
• dispatch.md — DispatchLayer routing
• gpu.md — WebGPU backend
• v081_release.md — release notes
• testing_and_validation.md — Lucy log interpretation