It's not very clear to me how to handle SWA with a unified cache where there may be multiple sequences, and it is not always obvious what tokens can be dropped from the cache. However I think it is definitely worth it for the single user case, which after all is the main use case of llama.cpp.

However, while working on this, I realized that enabling this option would prevent context caching, which IMO is a pretty big deal. So I am wondering if I am missing something.

Yes this is what I was thinking about for months now. There is no better solution than to disable context caching in this case.

An alternative solution is to allow user to choose one of the 2: either a proper SWA cache (good for memory) or allocate full (good for reusing cache)

So I am having some doubts if this is really worth supporting.

I'm feeling 50/50 here. One of the biggest use case would be to process large and diverse set of documents locally. In this case, user may never reuse the cache because each new request is a new document

It's not very clear to me how to handle SWA with a unified cache where there may be multiple sequences, and it is not always obvious what tokens can be dropped from the cache. However I think it is definitely worth it for the single user case, which after all is the main use case of llama.cpp.

The way I am approaching it is to have the "KV cells" information maintained separately for the non-SWA and SWA layers. This way, upon each KV cache commit (see #12799), we can do a pass over the SWA cells and automatically remove those that have position pos < pos_max(seq_id) - n_swa. Note that such tokens are only pruned from the SWA cells, while they remain in the non-SWA cells. When constructing the KQ mask for the graph, we use the non-SWA cells to construct the kq_mask and the SWA cells to construct the kq_mask_swa.

The rest of the logic is the same - it just operates on both set of cells. For example, find_slot searches in both the non-SWA and SWA cells.

My experience with the Gemma models in the context of Elo HeLLM has been that they required a disproportionate amount of computational resources to run benchmarks. The reason is that I was able to fit comparatively fewer parallel slots on 1 or 2 GPUs and my throughput was lower as a consequence. At least for my use case I value low memory usage for the context more than I value prompt caching because I have O(10000) short prompts and I'm bottlenecked mostly by generation throughput.

Continuing thinking about the logic for when to discard tokens from the cache, it's indeed tricky and not very clear how to do. For example, when doing speculative decoding, we can submit a draft batch with D tokens to the target model. If we apply the pruning logic from my previous comment strictly, then this would cause to "forget" D-1 of the oldest tokens in the SWA layers, which depending if the draft gets rejected would be problematic. This makes me think that we should probably have some "extra room" in the SWA cache - for example n_swa + 2*n_batch. And the prune logic should be something like: pos < pos_max(seq_id) - n_swa - n_batch.

It's not very clear to me how to handle SWA with a unified cache where there may be multiple sequences, and it is not always obvious what tokens can be dropped from the cache. However I think it is definitely worth it for the single user case, which after all is the main use case of llama.cpp.

I second slaren's opinion. As far as I know, vllm also doesn't support iSWA while hf transformers and ollama does. vllm is geared toward multi-user server use case. I suppose that's why they don't support it.

Ideally, it should be implemented as a switch to let user choose which one to use. By default, iSWA should be on for llama-cli but off for llama-server.

This makes me think that we should probably have some "extra room" in the SWA cache - for example n_swa + 2*n_batch. And the prune logic should be something like: pos < pos_max(seq_id) - n_swa - n_batch.

Yes I was thinking about this too, I think it can be a bit complicated to manage this case, but totally possible.

We can let user specify how many tokens are allocated in the sliding layers. For example, given n_swa=512, if llama_context is created with n_ctx=4096 and n_ctx_swa=1024, this will allow user to rollback until n_past - (1024 - 512)

We can further let n_ctx_swa = n_ctx * scale by default to make it transparent to end-user, with scale=0.5 by default for example. If scale=-1 then n_ctx_swa=n_swa

And finally, we may need to add an API to return the furthest n_past that user can rollback to, maybe something like llama_kv_self_get_minimum_pos ?

I'd +1 the ability to allow the user to switch.

Some use-cases benefit greatly from the prefix caching (example: on Metal systems with 48GB of RAM/VRAM, where pp is much slower than non-Metal pp and we have plenty of VRAM anyway) so allowing the user to choose would be optimal.

It's not very clear to me how to handle SWA with a unified cache where there may be multiple sequences, and it is not always obvious what tokens can be dropped from the cache. However I think it is definitely worth it for the single user case, which after all is the main use case of llama.cpp.

Is llama.cpp single user mode the most used case because that’s what the user base prefer or is it like that because the server performance goes down a lot with more than 3 users ? (#10860 )

We are really thankful of all the work you main contributors do on this project, but please do not fall in this « self-fulfilling prophecy » trap.

I personally use llama.cpp for server use (with multiple users).
I wonder if we could do something hybrid between iSWA and what is currently done.
I wonder if partial kV cache offload could work, with iSWA on the accelerator, and slower cache on RAM.

According to the Gemma3 paper, interleaved Sliding Window Attention reduces KV Cache memory usage by 1/5, so it would be much easier to run as right now KV Cache size is much heavier than comparable models.

If the drawback is the absence of prompt caching, then indeed it would make sense to give the option to the user and let them decide on a per use case basis. I think for cases where you use RAG/Vector DB it would prove to be very useful as prompt caching does not work when beginning of the context changes anyway. I would personally agree with Johannes here, faster token generation thanks to SWA would be more useful for me as well since I'm using vector DB.

So for the use cases short prompts/RAG it would make a lot of sense. For simple chat use cases without any RAG, prompt caching would probably make it faster overall compared to SWA and no prompt cache. Overall, I think having the option would be a great addition to llama.cpp.

If it helps, Ollama implemented iSWA support for Gemma 3, since the project is pretty similar to llama.cpp, perhaps it's useful to get a rough idea on how to implement it (although Ollama is a different coding language): https://github.com/ollama/ollama/blob/2fec73eef6e9482f606f185ebb2ae4f75ad1a37c/model/models/gemma3/model_text.go#L190

I've been thinking, does Ollama support prompt caching? Since Gemma 3 SWA is supported in Ollama, how did they handle it?

Some people recently mentioned concerns with this PR - I think caching is quite important for a subset of users who don't have GPUs and run purely CPU only.

They are fine spending initial minutes or more ingesting a large initial prompts which they then reuse for many future turns - generation speed itself is usable, but the inability to cache would be crippling for such users.

Both the old cache (i.e. more memory usage, but with advanced caching supported) and the new cache (less memory with just last-prefix caching) will be supported. Still figuring the implementation details - will likely be supported via a flag or a parameter.

Thanks for all the feedback in this discussion. This branch should be ready for testing - I've listed some important use cases that need to be exercised. If something does not work, please let me know - at the moment I've done very little testing, so there could be some issues remaining.

I will soon write up a detailed summary of the changes and the approach taken. And after that will add some comments to the code and open the PR for review.

Regarding the parameter for controlling the size of the SWA cache - for now I haven't introduced it because some initial tests show that Gemma 3 remains coherent even when it "forgets" the local SWA cache - likely thanks to the data in the non-SWA cache. So I am thinking about giving this approach a try because it keeps the UX simple (i.e. we won't have to add new parameter and handle the use cases where context editing is not possible). If we determine that this breaks some important use cases, we can add the parameter - the libllama change is simple and the behavior would basically fallback to what currently happens on master.

To people who have the bandwidth to test models, FYI Cohere 2 arch includes R7B which is much smaller than Command-A

for now I haven't introduced it because some initial tests show that Gemma 3 remains coherent even when it "forgets" the local SWA cache

Does this mean in the current implementation the model isn't executed correctly?

FWIW, Gemma 3 worked better for me on main with Q8 cache quantization than on this branch + unquantized kv cache.

@andportnoy It's evaluated correctly, as long as you don't use context shift, cache reuse or branching from old states. Do you do any of that in your tests? Can you provide a repro?

Edit: Also don't change 2 things at the same time when testing. Use the same KV cache type, so we can rule out differences that are not relevant to the changes in this branch.

No issue so far with lastest commit, though I'm not seeing as much of a sepeedup as I was expecting.

There will be additional speed-up after we support SWA cache with the smaller size n_swa + n_ubatch instead of n_swa + n_batch. But this needs a larger rework that I will do in a follow-up PR to avoid making this PR even more complicated.

Does the current implementation detect cases where the context has been lost? For example, starting a sequence from an older position, or deleting the last tokens of a sequence. Is that what you mean when you mention "context reuse"?

Does the current implementation detect cases where the context has been lost? For example, starting a sequence from an older position, or deleting the last tokens of a sequence. Is that what you mean when you mention "context reuse"?

Currently, we do not detect these and we silently proceed with partial SWA cache which is not mathematically correct. So branching from an older position currently is allowed, but technically it is not exact. Context/cache reuse is when you enable the --cache-reuse N flag that allows to move chunks in the cache to new positions - also allowed.

Basically everything that works on master is also allowed with SWA and we don't handle the cases where the context has "slid away".

From server logic PoV, we can force a recompute in such cases in order to make the computation exact, at the cost of extra re-processing.

From the libllama PoV, we can add a check that the batch positions are compatible with the contents of the cache and if they are not, we can return some error or at least print a warning.

Open to suggestions.

I think this is likely to lead to quality issues and at least should be detected and reported. Maybe llama_decode could return an error in these cases, applications like the server then would have the option of reprocessing the prompt.

We can make llama_decode still perform the computation but return a new code 3 in such cases. So this can either be ignored or handled by the serverapplication. Does this sound OK?

I guess that depends on how useful you consider the evaluation with a partial context. I think that it is very unlikely that applications or users would consider this acceptable, and would always choose to recompute, so I don't think that proceeding with the computation regardless would be desirable, since it would only waste time.

I guess that depends on how useful you consider the evaluation with a partial context. I think that it is very unlikely that applications or users would consider this acceptable, and would always choose to recompute, so I don't think that proceeding with the computation regardless would be desirable, since it would only waste time.

I agree with @slaren here. I believe an error should be returned. Then, in server, the lost context could be recomputed.

From what I can gather, people spend much more time simply continuing the output rather than going back in time.

If the prompt processing is fast enough, most wouldn't care anyway. It's not as if we lost the whole context anyway.

llama_decode() returning an error is actually not a good solution, because then we wouldn't know which was the specific sequence that triggered the error when the batch contains multiple sequences. So we need some way to determine at the beginning of the request whether it will have the necessary cache data and if we don't have it, then we simply process the entire prompt.

However, the existing API is not enough to do that. One way that I can think of is to introduce a new:

// Returns the smallest position present in the KV cache for the specified sequence
LLAMA_API llama_pos llama_kv_self_seq_pos_min(
            struct llama_context * ctx,
                    llama_seq_id   seq_id);

This way, upon preparing the prompt of the new task and determining which prefix is already available, we can check if the minimum position available in the cache is equal to 0. If is is not, then we cannot perform any sort of cache/prefix reuse, so we submit the full prompt.

I'm wondering if there is another way that does not involve adding this extra API call?

Another option could be to store the token ids/embeddings in the KV cache, and do the recomputation automatically if necessary. For token ids that wouldn't be a problem since the extra memory would be insignificant, for embeddings the overhead would be higher, but probably still not very important.

Yes, I'm definitely looking towards extending the KV cache to maintain the token/embeddings information and use this to simplify the user logic for tracking what is currently inside the cache. Planning some follow-up PRs in that direction.

Pushed tentative proposal:

• Print warnings when we detect partial SWA contexts
• Utilize the llama_kv_self_seq_pos_min() to detect partial SWA contexts and force recompute in the server logic

Now the computation should always be exact when using llama-server. If an application fails to do the check for partial SWA context, they will get a warning spam in the logs.

The idea is this to be a temporary solution until we are able to automatically reprocess what is necessary.

At the moment it seems that there is no way to disable SWA cache. It might be good now to add an option to disable SWA in llama_context_params, and leave it disabled by default to avoid breaking other applications until automatic re-computation is implemented.

I added bool llama_context_params::swa_full; - when enabled (default value) the SWA cache will be created with the full context size, just like we do on master. In this case, the iSWA KV cache will not prune the old tokens on commit and this way we will have all the necessary data always precomputed.

When the flag is false, we create the small SWA cache. Even now, third-party apps have a way to fully work with this, similar to how the llama-server is implemented. For now, they need to be careful about when the cache data has been lost. One way to do it is like this:

The llama-bench always uses small SWA cache (i.e. swa_full = false).

In common we set the swa_full = false by default, but can be changed with --swa-full CLI arg.