What're your thoughts on the TG speed decrease? Obviously it's pretty negligible overall, but it's curious to see..

Will these efforts also help with ARM repacking?

Hi @bartowski1182 , we were trying to check on the text generation if the recent introduction of restrict keyword in vec_dot_q4_K_q8_K is causing this difference and tried to replicate the restrict keyword usage in the repacking implementation, but currently we do see this difference.

Regarding ARM Repacking, the repacking of Q4_K has been designed similar to Q4_0_8_8. So there's a possibility of this being extended to ARM, although not sure on finer details. Thanks

Nice work!

I don't have an easy access to a AVX2 machine, but will try to do some testing soon. I am curious how the performance looks like for different threads and model sizes (e.g. 1B, 3B).

Update : A couple of updates were made in the PR

1. The quantize_q8_K_4x8 was vectorized and updated and that resulted in further gains with prompt processing. The status of text generation remains consistent with the previous update
2. Another minor update is usage of roundf replaced with nearest int to be consistent with the non interleaved version

GCC Linux :

Q4_K_M Model :

Q4_K_S Model :

GCC Version = 12.3

The models were quantized and tested from meta-llama2 7B model - https://huggingface.co/meta-llama/Llama-2-7b

The PR was tested in AMD Raphael 7600X which supports the following flags by default :

CPU : SSE3 = 1 | SSSE3 = 1 | AVX = 1 | AVX2 = 1 | F16C = 1 | FMA = 1 | BMI2 = 1 | AVX512 = 1 | AVX512_VBMI = 1 | AVX512_VNNI = 1 | AVX512_BF16 = 1 | LLAMAFILE = 1 | OPENMP = 1 | AARCH64_REPACK = 1 |

The perplexity results are tabulated as follows post changes:

@Srihari-mcw Hi, could you please help me check this error? ： #12528

Do I understand correctly that this PR repackages Q4_K tensors to some other memory layout on the fly during model loading from existing GGUF files? I noticed that after updating llama.cpp to the current master DeepSeek R1 Q4_K_S model loading times (with the model file already cached) increased from:

llama_perf_context_print: load time = 25738.14 ms

to:

llama_perf_context_print: load time = 500558.83 ms

and only a single core is active when loading the model. It this how it should work?

Yes, loading time is increased due to the runtime repacking of the weights.

Hi, I have one question, why doesn't this patch modify GGML_TYPE_Q8_0 to PARAM_TYPE in forward_mul_mat_id as like in forward_mul_mat? Won't this cause any problem for gemv of Q4_K & Q8_K ? @Srihari-mcw @ggerganov

Hi @Nor7th , I guess this is addressed in #12544 . The models we tested while developing this particular PR did not go through the particular code path that you pointed out and I guess hence this was missed (Guess MoE models go through the code path you had pointed). Sorry, @Yangxiaoz missed your message and glad the fix is already there

Thanks @ggerganov

Hi @Nor7th , I guess this is addressed in #12544 . The models we tested while developing this particular PR did not go through the particular code path that you pointed out and I guess hence this was missed (Guess MoE models go through the code path you had pointed). Sorry, @Yangxiaoz missed your message and glad the fix is already there

Thanks @ggerganov

Thanks! Good to know it's already solved.

Yes #12544 should have fixed that.

Aside from the extreme slowdown in model loading times due to repacking, this has also slowed down token generation for me, at least on my Dual Xeon 4216 system.

Before this change, I could load the Cohere Command A model in Q4_K_M almost instantly (excluding the dry run), and token generation was at approximately 1.92 tokens per second. After this change, loading the model takes ages (20+ minutes), and token generation is around 1.63 tokens per second (15% slower).

System:
Dual Xeon 4216

Command:
./llama-cli -m ~/c4ai-command-a-03-2025-Q4_K_M-00001-of-00002.gguf --numa distribute --threads 64 -c 32768 --temp 0.5

Command A, with repacking/interleaving:
llama_perf_context_print:        load time = 1221419.61 ms
llama_perf_context_print: prompt eval time =    5232.90 ms /    17 tokens (  307.82 ms per token,     3.25 tokens per second)
llama_perf_context_print:        eval time =   33647.99 ms /    55 runs   (  611.78 ms per token,     1.63 tokens per second)

Command A, without repacking/interleaving:
llama_perf_context_print:        load time =  109457.37 ms
llama_perf_context_print: prompt eval time =    4787.59 ms /    17 tokens (  281.62 ms per token,     3.55 tokens per second)
llama_perf_context_print:        eval time =   32230.27 ms /    62 runs   (  519.84 ms per token,     1.92 tokens per second)

Maybe we need a command line option to disable repacking for those that have the compile flag but doing like it?

I just tried out Phi 4 Q4_K_M with the same command line arguments I used for Command A, and had a similar slowdown. Model loading time with repacking on my machine for Phi 4 is at least reasonable-ish (around a minute), but that's still much worse than near-instant loading without the repacking. This commit slowed down token generation from 9.53 T/s to 8.81 T/s, a 7.5% slowdown.

System:
Dual Xeon 4216

Command:
./llama-cli -m ~/phi-4-abliterated.Q4_K_M.gguf --numa distribute --threads 64 -c 32768 --temp 0.5

Phi 4, with repacking/interleaving:
llama_perf_context_print:        load time =   76962.44 ms
llama_perf_context_print: prompt eval time =     822.04 ms /    22 tokens (   37.37 ms per token,    26.76 tokens per second)
llama_perf_context_print:        eval time =    6582.56 ms /    58 runs   (  113.49 ms per token,     8.81 tokens per second)

Phi 4, without repacking/interleaving:
llama_perf_context_print:        load time =   17113.48 ms
llama_perf_context_print: prompt eval time =     862.61 ms /    22 tokens (   39.21 ms per token,    25.50 tokens per second)
llama_perf_context_print:        eval time =    6401.67 ms /    61 runs   (  104.95 ms per token,     9.53 tokens per second)~~~

I did another experiment, only using one socket of my NUMA system. In single socket (non-NUMA) usage, this patch speeds up both token generation and prompt processing significantly, at the expense of slow initial load times due to repacking.

System:
Dual Xeon 4216 (but only using one socket to simulate non-NUMA)

Command:
./llama-cli -m ~/phi-4-abliterated.Q4_K_M.gguf --numa isolate --threads 32 -c 32768 --temp 0.5

Phi 4, with repacking/interleaving:
llama_perf_context_print:        load time =   77142.05 ms
llama_perf_context_print: prompt eval time =    1171.54 ms /    22 tokens (   53.25 ms per token,    18.78 tokens per second)
llama_perf_context_print:        eval time =    7431.09 ms /    62 runs   (  119.86 ms per token,     8.34 tokens per second)

Phi 4, without repacking/interleaving:
llama_perf_context_print:        load time =   19604.91 ms
llama_perf_context_print: prompt eval time =    1412.99 ms /    22 tokens (   64.23 ms per token,    15.57 tokens per second)
llama_perf_context_print:        eval time =    5974.69 ms /    46 runs   (  129.88 ms per token,     7.70 tokens per second)

I think an option (or maybe a default) to disable block interleaving and the associated repacking would be good to have to speed up initial loading times. This patch does speed up both prompt processing and token generation for me in single socket non-NUMA usage.

A separate issue is that this patch doesn't seem to properly handle NUMA systems, with minimal to no improvement in prompt processing, and substantially slower token generation. For Command A, both prompt processing and token generation slowed down with this patch when using NUMA.

The repacking process takes ages for large models, and only seems to use around 16% of a single CPU core according to top. I'm guessing that the NUMA performance issues are because of the repacked weights not being redistributed across CPU sockets. It takes my system 20 minutes to repack the 62 GB of quantized weights for Command A, which seems absurdly slow.

You should be able to disable repacking with -ot .=CPU.

You should be able to disable repacking with -ot .=CPU.

Thanks, this brings it back to the performance before this commit or at least close to it. Nonetheless, making the repacking faster and NUMA friendly would be the ideal solution.

Hi, is there any evaluation data for pp and tg in other sizes? Is there any acceleration effect of the method when the pp is small, like 16, 32?

Hi @longaaalong , though we have llama-bench data with the default configuration (pp512) currently with llama-cli we have earlier observed good performance gains with relatively smaller prompt sizes

For eg,
./llama-cli -m "../../../test_models/ggml-model-Q4_K_M.gguf" -p "Once upon a time, the idea of a 20-year mortgage was a" -n 100 -s 46732 (input of 20 tokens)

We observed a gain from ~46 t/s to ~71 t/s for prompt processing

Thanks