Chapter 40

model/tokenizer

github.com/openfluke/welvet/model/tokenizer✅

Why it exists

Generate needs encode/decode of HF tokenizer.json without pulling Python.

What it is

LoadTokenizer, Encode/Decode, LoadForEntity.

Go example

examples/40-tokenizer/main.go

Run:cd welvet/examples/40-tokenizer && source ../env.sh && go run .

package main

import (
	"fmt"

	"github.com/openfluke/welvet/model/tokenizer"
)

func main() {
	tok, err := tokenizer.LoadTokenizer("tokenizer.json")
	if err != nil {
		fmt.Println(err)
		return
	}
	ids := tok.Encode("hello welvet", true)
	fmt.Println(ids, tok.Decode(ids, true))
}

Output

exit 0 · last run via go run .

failed to read tokenizer file: open tokenizer.json: no such file or directory