A quantized transformer inference engine hand-written in WGSL compute shaders. No compiler stack and no generated kernels: every GEMM, norm, and attention pass is authored, measured, and verified against a reference implementation.
What it will demonstrate
Packed 4-bit weight GEMM with dequantize-on-load, benchmarked against fp16
RoPE and RMSNorm as fused WGSL passes
Causal attention with a persistent KV cache, visualized as it fills
Top-p sampling with a live token stream
A per-kernel latency dashboard driven by GPU timestamp queries
Roadmap
Milestone 1: fp16 forward pass of a GPT-2 class model, correctness ledger vs fp32
Milestone 2: causal attention with a persistent KV cache
Milestone 3: 4-bit quantized kernels and packed weight loading
Milestone 4: streaming UI with per-kernel instrumentation
The honest benchmark
Target: memory-bound decode within striking distance of llama.cpp, on every GPU vendor, from a single URL
Milestone 1 numbers will be published here, unedited
Questions or collaboration: contact.
Progress notes land on the main page as milestones ship.