Tags: kvcache-ai/ktransformers
Tags
feat(GLM): add native GLM-5.3-flash support (#2173) * feat: support SwiGLU limit for block FP8 experts * perf(fp8): add native layerwise batch transport * feat(fp8): support layerwise transport for TP1/2/4/8 * release: prepare GLM-5.3-flash support * docs(glm): refine GLM-5.3-flash documentation * docs(glm): simplify release announcement
[feat](kt-sft): Activation reuse & Int8 kernel refactor & native bloc… …k-FP8 LoRA SFT (#2141) * feat(sft): support distributed activation reuse policies * feat(sft): add frozen-base INT8 LoRA training * fix(sft): make INT8 expert LoRA rank-zero authoritative * fix(sft): preserve DeepSeek router LoRA routing * feat(sft): enable persistent INT8 LoRA training * perf(sft): accelerate INT8 VNNI with oneDNN BRGEMM * perf(int8): fuse oneDNN compensation into backward repack * [feat]: support BF16 expert LoRA training * [fix]: honor forwarded activation policy in SFT workers * feat(sft): add native block-FP8 routed expert LoRA * feat(sft): expose explicit expert placeholder ownership * fix(sft): publish fused adapter artifacts atomically * feat(sft): own artifact and adapter lifecycle contracts * fix(sft): harden artifact and rank-local contracts * fix(sft): auto-adapt owner before adapter restore * style(sft): keep lifecycle comments concise * fix(sft): require fused adapter manifests * test(sft): use spawn for distributed workers * fix(sft): preserve runtime checkpoint metadata * fix(sft): validate wrapped runtime configuration * fix(sft): preserve expert format provenance * fix(sft): own routed experts during device dispatch * test(sft): lock explicit quantization conflict * fix(cpu): make shared memory buffers lifetime-safe * release: prepare v0.7.0
[feat](kt-kernel): end-to-end full-parameter and LoRA SFT(#2094) * Add full FT development snapshot * [fix](kt-kernel): fix Full FT TP base weight gradients * [fix]: bug fix of 2d81e86 * [fix](kt-kernel): fix AMX BF16 full-weight gradients * [docs](kt-kernel): document Full FT fork changes and debug history * [docs](kt-kernel): align fork remote terminology * [chore](kt-kernel): keep agent notes local * [fix](kt-kernel): configure SFT OpenMP threads * [perf](kt-kernel): optimize AMX Full-FT weight gradients Coarsen base-weight gradient work from individual output tiles to fixed-intermediate strips so each task reuses packed panels across the hidden dimension. Keep aligned thread-local BF16 panels across tasks and retain FP32 AMX accumulator tiles for the full K reduction. Gate and up run separate K passes while sharing the packed input panel. On the matched Qwen3-30B-A3B 1-GPU test, stable Full-FT backward drops from 9.281s to 6.792s (-26.82%), step time drops from 19.252s to 16.140s, and TPS rises from 212.76 to 253.78. The LoRA-only backward control changes by -2.74%. Validated with clang-format, the Release AMX/CUDA extension build, TP1/TP2 reference gradients across boundary token counts, and the 15-step Full-then-LoRA performance run. * [feat](kt-kernel): add staged SFT profiling * [fix](kt-kernel): reuse inference BF16 kernel for SFT * [perf](kt-kernel): add fine-grained Full-FT profiling * [perf](kt-kernel): batch BF16 Full-FT weight gradients Use one expert-aggregated tile driver for AVX512-BF16 and AMX base-weight gradients, and pack updated full-precision weights directly into TP BufferB layouts without temporary partitions. Add worker-local profiling and focused dWeight/strided-repack coverage. * [test](kt-kernel): benchmark BF16 dWeight AMX driver * [fix](kt-kernel): label dWeight store as worker CPU time * [docs](sft): record BF16 Full-FT performance * [docs](sft): remove Qwen3 Full-FT performance report * [perf](kt-kernel): reduce BF16 Full-FT checkpoint overhead Retain the first CPU MoE forward state across non-reentrant checkpoint recomputation, write BF16 activations directly into the backward cache, and reduce dWeight packing and gradient-clear traffic. Extend staged profiling and cover checkpoint reuse plus AMX/AVX dWeight paths. * [perf](kt-kernel): make SFT optimizer gradients authoritative Bind Full-FT and LoRA Parameter.grad directly to the KT-managed BF16 gradient buffers, avoiding PyTorch duplicate accumulation. Accumulate microbatch gradients in C++, lazily clear expert buffers between optimizer windows, preserve rank-0 distributed ownership, and add lifecycle and AMX dWeight coverage. * [perf](kt-kernel): avoid eager Full-FT gradient zeroing Allocate authoritative Full-FT gradient buffers with torch.empty. The C++ state machine performs the mandatory full clear before first use, avoiding redundant Python-side first touch. * perf(sft): enable checkpoint forward reuse for LoRA * fix(sft): serialize checkpoint recompute with async repack * fix(sft): persist authoritative full weights * fix(sft): normalize legacy distributed gradients * [fix](sft): train gated shared experts * fix(sft): preserve expert placeholders across state dict loads * ci: publish ktransformers sdist in release workflow * fix(sft): preserve router autograd for LoRA training Determine routing graph tracking from the router's trainable parameters instead of the Full-FT mode flag, so PEFT LoRA adapters on MoE gates receive routing-weight gradients. Fail fast when a trainable router returns detached weights and cover frozen, TopK, checkpoint-reuse, and two-step optimizer behavior. --------- Co-authored-by: illu <[email protected]>
Release/0.6.2.post3: carry kt-kernel SwiGLU clamp companion missing f… …rom post2
0.6.2.post2: submodule refactor and update tutorial (#1993) - sglang submodule -> 43ed1ec77: V4-Flash hybrid SWA chunked-prefill hang fix (#44) + DSV4 plugin registry refactor (#47) - pin sglang-kt==0.6.2.post2 - tutorial: switch V4-Flash launch example from 8x RTX 5090 to single-card (decode 20+ tok/s); flip Ada Lovelace SM_89 row to validated; update Hardware Requirements GPU line accordingly
[release]: 0.6.2.post1 V4-Flash MXFP4 full-GPU prefill fallback now works: - Previously crashed all TP schedulers with StopIteration/AttributeError whenever --kt-gpu-prefill-token-threshold was low enough to actually fire (path was hardcoded for FP8/INT4 layouts). - Now detects MXFP4, re-runs the V4 swizzle on the 256-expert gpu_layer, caches the load across prefill chunks. - Measured on 8x RTX 5090 (threshold=1024, chunked=1024): 16k input -> 2011 tok/s, 65k -> 2798, 262k -> 2154 prefill TPS.
PreviousNext