I was about to submit a PR, great I checked 😄.

Already uploaded the model to the hub.
https://huggingface.co/mlx-community/c4ai-command-r-plus-4bit

@Blaizzy Thank you! How much RAM does it require to run 4bit q?

Needs about 65GB to generate with 4-bit. But the generation is slow right now, trying to debug the performance issue.

@Blaizzy Thank you! How much RAM does it require to run 4bit q?

@DenisSergeevitch, as @awni said 👆🏽.

I can't run it myself, I use a M1 Air 16GB :)

Thank you, I will wait for i_q1 then

Btw to get this to run reasonably fast on an M2 Ultra you need to set the wired GPU memory lower limit appropriately. Something like:

sudo sysctl iogpu.wired_lwm_mb=100000

I am running the 4-bit version of Command-R-Plus and I consistently see the GPU usage dropping during generation and performance becoming abysmal.

My machine is the M2 Ultra 192GB and,

ProductName: macOS
ProductVersion: 14.3
BuildVersion: 23D56

@awni 👆🏽

@jeanromainroy did you set the memory limits? You could try making it larger:

sudo sysctl iogpu.wired_lwm_mb=150000

Even after setting this,

sudo sysctl iogpu.wired_lwm_mb=150000

I still see the GPU usage dropping before the completion ends.

Do you mind to open an issue and include the command, versions of MLX / MLX LM, OS etc?