I'm using Kernel-memory with LLamaSharp. Despite having a RTX 3080 and the latest CUDA drivers installed, CUDA is not used.
Not sure if this is a bug or I'm missing something, so here's a question instead:
The LlamaSharp.csproj contains
<PackageReference Include="LLamaSharp.Backend.Cpu"/>
<PackageReference Include="LLamaSharp.Backend.Cuda12"/>
I found out that if both Cpu and Cuda12 back-ends are referenced, only the CPU is being used even if the CUDA DLL is loaded.
Interestingly, the logs do say that the CUDA back-end is loaded, but no Cuda is used.
[LLamaSharp Native] [Info] Detected OS Platform: WINDOWS
[LLamaSharp Native] [Info] Detected cuda major version 12.
[LLamaSharp Native] [Info] ./runtimes/win-x64/native/cuda12/libllama.dll is selected and loaded successfully.
llama_model_loader: loaded meta data with 19 key-value pairs and 291 tensors from D:\Source\km\Models\llama-2-7b-chat.Q5_K_M.gguf (version GGUF V2)
...
If I remove the reference to LLamaSharp.Backend.Cpu, then the CUDA back-end will start to be used. The logs show:
[LLamaSharp Native] [Info] Detected OS Platform: WINDOWS
[LLamaSharp Native] [Info] Detected cuda major version 12.
[LLamaSharp Native] [Info] ./runtimes/win-x64/native/cuda12/libllama.dll is selected and loaded successfully.
ggml_init_cublas: GGML_CUDA_FORCE_MMQ: no
ggml_init_cublas: CUDA_USE_TENSOR_CORES: yes
ggml_init_cublas: found 1 CUDA devices:
Device 0: NVIDIA GeForce RTX 3080, compute capability 8.6
llama_model_loader: loaded meta data with 19 key-value pairs and 291 tensors from D:\Source\km\Models\llama-2-7b-chat.Q5_K_M.gguf (version GGUF V2)
...
I've reported this to the kernel memory project, but was advised to report this here.
I'm using Kernel-memory with LLamaSharp. Despite having a RTX 3080 and the latest CUDA drivers installed, CUDA is not used.
Not sure if this is a bug or I'm missing something, so here's a question instead:
The LlamaSharp.csproj contains
I found out that if both Cpu and Cuda12 back-ends are referenced, only the CPU is being used even if the CUDA DLL is loaded.
Interestingly, the logs do say that the CUDA back-end is loaded, but no Cuda is used.
If I remove the reference to LLamaSharp.Backend.Cpu, then the CUDA back-end will start to be used. The logs show:
I've reported this to the kernel memory project, but was advised to report this here.