@jharonfe I haven't tested it personally but according to the linked discussion it's automatically detected by llama_load_model_from_file which llama-cpp-python uses.

One caveat is that this probably doesn't work with .from_pretrained yet because that method looks for a single file to pull via the huggingface_hub library. I think adding an option like additional_files there would be good, I'll look into it.

One thing that would help is a link to a small model that's been split and uploaded, preferrably <7b.

I tested this using the latest version, 0.2.61, and the model appears to load correctly. Thanks for the feedback on this.

I just hit this issue today. I had tried using a wild card to specify all of the files, but then had it complain about seeing multiple files. An option like additional_files would be a nice quality of life change.

Prototype: 0e67a83
Will test and open PR if it works.

PR opened: #1457

i would appreciate this feature! currently, i do something very hacky like below and it works (once all files are downloaded), but it's pretty clunky

from llama_cpp import Llama

for i in range(1, 6+1):  # assuming the shards are split into 6
    try:
        llm = Llama.from_pretrained(
            repo_id="MaziyarPanahi/Llama-3-70B-Instruct-DPO-v0.4-GGUF",
            filename=f"*IQ4_XS-{i:05d}*",
            verbose=True,
            n_gpu_layers=-1, 
        )
    except:
        pass

i would appreciate this feature! currently, i do something very hacky like below and it works (once all files are downloaded), but it's pretty clunky

Well, my PR has been sitting there for a month now. You might try editing the code additions in yourself to get rid of the jank.