hexagon: add an experimental Qualcomm FP8 DiT and VAE path - #1970

Motivation

We are building high-performance DiT inference on Qualcomm devices, covering both Adreno GPUs and Hexagon NPUs, for Local Dream.

Our Adreno work is already being contributed to llama.cpp: #22755, #26085, and #26331 are merged; Fused QKNorm-RoPE #28446 is currently open.

This PR provides the integrated Hexagon NPU path for stable-diffusion.cpp.

Implementation

The v79+ path includes standard F8_E4M3 safetensor loading; HMX 32x32 set_tensor repacking; FP8 W8A16 GEMM; non-causal HMX FA; fused Q/K RMSNorm-RoPE; segmented-K Klein linear2; FP16 VAE activations; direct implicit-GEMM convolution; fused GroupNorm-affine-SiLU; and fused upscale-convolution. Qwen3-4B uses the existing generic Hexagon Q4 kernels.

The FP8 GEMM uses the v79 activation.hf / weight.f8 HMX path with FP32 accumulation. HMX cvt.hf = acc(2) applies the architectural 1/256 conversion; the HVX output epilogue compensates by 256 and folds the model's FP32 weight scale and any graph scalar scale into the same store pass.

Temporary integration

The FP8 plumbing required for full llama.cpp integration has not yet been accepted into llama.cpp's embedded ggml, for reasons that remain unclear to us. This PR temporarily uses happyyzy/ggml, branch work/qualcomm-hexagon-optimizations, commit 8121decc678bfb9f02a0f7190bce143bfb1599fa.

The generic pieces are being upstreamed into llama.cpp incrementally. This is a temporary integration vehicle for Local Dream until that work is complete.

End-to-end results

Device: Snapdragon 8 Elite / SM8750 / HTP v79. TE, DiT, and VAE all run on HTP. Prompt is a lovely cat, Euler, CFG 1, seed 42. ``E2Eisgenerate_image completed`, including live TE, all sampling steps, and VAE decode. None of these runs uses tiled VAE.

During the long 1536 runs, Z-Image iterations 5-8 rose to 36.16-38.37 s/it and the final Klein iteration rose to 25.24 s/it. The E2E numbers above include that thermal throttling.

This puts phone-class Snapdragon inference close to mainstream NVIDIA RTX GPU latency for these distilled pipelines.

Commands

export LD_LIBRARY_PATH="$PWD" ADSP_LIBRARY_PATH="$PWD"
LLM=/data/local/tmp/sd_bench_clean/qwen_3_4b-Q4_0.gguf
ZM="$PWD/z-image-turbo_fp8_scaled_e4m3fn_KJ.safetensors"
KM="$PWD/flux-2-klein-4b-fp8.safetensors"
ZV=/data/local/tmp/sd_bench_clean/ae.safetensors
KV=/data/local/tmp/validate6_model_pack_20260412_aux/flux2-vae.safetensors
run() { ./sd-cli --log-level verbose --diffusion-model "$1" --llm "$LLM" --vae "$2" --backend diffusion=HTP0,te=HTP0,vae=HTP0 --fa --vae-conv-direct -t "$5" -p "a lovely cat" --cfg-scale 1 --steps "$4" --sampling-method euler -W "$3" -H "$3" --seed 42 -o "$6"; }
run "$ZM" "$ZV" 1024 8 4 zimage_1024_s8.png
run "$KM" "$KV" 1024 4 8 klein_1024_s4.png
run "$ZM" "$ZV" 1536 8 8 zimage_1536_s8.png
run "$KM" "$KV" 1536 4 8 klein_1536_s4.png

Images

Z-Image 1024, 8 steps

Klein 1024, 4 steps

Z-Image 1536, 8 steps

Klein 1536, 4 steps

FP8 vs Upstream Q4_0/Q8_0

Compared with upstream Hexagon using standard Q4_0/Q8_0 GGUF weights, FP8 is 8.87x faster for Z-Image and 9.30x faster for FLUX.2/Klein at 1K, while also delivering visibly better image quality.

• Resolution: 1024x1024
• Sampling steps: 8
• Prompt:

雨夜的未来上海外滩，镜头前是一辆旧式有轨电车穿过积水街道，街边霓虹牌同时写着“欢迎光临”“火锅”“Open 24 Hours”，远处玻璃摩天楼与石库门老建筑并列，空中漂浮无人机广告屏，屏幕上有清晰汉字“春风
得意”，画面里有穿风衣的人群、红色雨伞、湿漉漉的柏油路反射青蓝与橙红灯光，构图复杂、层次深、电影感、超细节

Q4_0 + Q8_0

F8_E4M3