Hey @irthomasthomas that should work, are you sure you have the latest version installed?

python3 -c "import llama_cpp; print(llama_cpp.__version__)"

If you run that the version should be at least 0.2.38 for speculative decoding support.

Thanks @abetlen, I forgot I was running multiple python environments... Anyway, after updating, I'm now getting a new problem: ValueError: Attempt to split tensors that exceed maximum supported devices. Current LLAMA_MAX_DEVICES=1

Is that related to this? ggml-org/llama.cpp#5240

Here is the command I ran:
python3 -m llama_cpp.server --model /run/media/thomas/828ca7e5-381d-4149-8fc5-6d6aa26b90f2/Models/deepseek/deepseek-coder-33B-instruct-GGUF/deepseek-coder-33b-instruct.Q4_K_M.gguf --n_gpu_layers 56 --tensor_split 64 36 --offload_kqv false --n_ctx 8000 --n_batch 56 --chat_format chatml

I opened a new issue as its unrelated to this one. #1166