Summary

M-GPU-MOE-1.4 NaN root cause pinpointed to layer 6 moe_ffn_out. Operator-dispatched run of the M80 heavy harness (PR #1524) on Blackwell GB10 (gx10) against cached 18 GB Qwen3-Coder-30B-A3B-Instruct GGUF completed cleanly in 23.18s and produced the expected bisection signal.

This discharges 2 of the 4 still-open falsifiers in the M-MOE-SUB cascade and narrows the M-GPU-MOE-1.4 fix scope to a single-layer surface.

Per-layer cos-sim summary

Full output: evidence/m-gpu-moe-1-4-bisection-gx10-2026-05-06/m80-bisection.txt (853 lines)

layer | moe_router (cos / verdict)        | moe_ffn_out (cos / verdict)
------|-----------------------------------|-----------------------------------
L00   | 1.000000 MATCH                    | 0.999987 MATCH
L01   | 0.999998 MATCH                    | 0.999904 MATCH
L02   | 0.999998 MATCH                    | 0.999953 MATCH
L03   | 0.999975 MATCH                    | 0.999876 MATCH
L04   | 0.999994 MATCH                    | 0.999861 MATCH
L05   | 0.999993 MATCH                    | 0.999896 MATCH
L06   | 0.999986 MATCH                    |             NanGpu  ← FIRST NaN
L07   | 0.818903 DIVERGE                  |             NanGpu  ← NaN poisons router
L08–L47 | DIVERGE                         |             NanGpu  ← downstream poisoning

Harness decision tree firing:

If first_NaN_GPU(moe_ffn_out) > 0 and earlier layers MATCH: bug is layer-N specific (rare).

Status promotions

Contract status: ACTIVE_ALGORITHM_LEVEL → ACTIVE.

Architectural portability finding

This ran on sm_120 (Blackwell GB10). Original M-GPU-MOE-1.3 NaN bug (PR #1493) was characterized on sm_89 (Ada RTX 4090). Both architectures produce NaN at the same layer → the bug is algorithmic / numerical, NOT kernel codegen. trueno#200 Blackwell PTX JIT pre-warming did NOT block the dispatch — Q4K/Q6K matvec compiled and ran on sm_120 first-shot.

This is a stronger signal than expected. We initially assumed gx10 might fail to reproduce or hit JIT issues; instead it gave us a clean arch-portable bisection result that means a single fix at the bisected stage discharges both arch-specific manifestations.

Bug surface narrowed for M-GPU-MOE-1.4 fix

• crates/aprender-serve/src/gguf/cuda/moe_ffn_forward_layer_cuda.rs (per-layer GPU helper)
• crates/aprender-serve/src/gguf/cuda/expert_swiglu_cuda.rs (per-expert SwiGLU)
• CudaExecutor::q4k_matvec / q6k_gemv (custom PTX)

Hypotheses for layer-6-specific NaN (priority order):

1. Numerical overflow in expert SwiGLU at L6 — layer-6 intermediate activations have distribution causing silu(gate) * up to overflow accumulator
2. Expert weight distribution at L6 — layer-6 experts have weights that combined with CPU-traced L5 output produce large activations
3. Q4K dequant accumulator at L6 — a specific Q4K block at layer 6 has a scale value causing overflow during dequant + matmul fusion

Verification

$ pv validate contracts/trace-moe-gpu-sub-stages-v1.yaml
0 error(s), 0 warning(s)
Contract is valid.

$ ssh gx10 "cd ~/src/aprender && cargo test -p aprender-serve --features cuda --release \
    --test qwen3_moe_gpu_per_stage_diff falsify_moe_sub_002_cpu_gpu_traced_per_stage_diff \
    -- --include-ignored --nocapture"
test falsify_moe_sub_002_cpu_gpu_traced_per_stage_diff ... ok
test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 6 filtered out; finished in 23.18s

Production hot paths

Byte-unchanged. YAML + evidence-only — additive-purity invariant pinned in v1.1.0 still holds.

Test plan

• pv validate 0/0
• LIVE harness ran on gx10 (Blackwell GB10) producing first NaN_GPU at L6 moe_ffn_out
• Evidence captured in evidence/m-gpu-moe-1-4-bisection-gx10-2026-05-06/
• No production code touched

🤖 Generated with Claude Code