Greetings, Pls. post link bellow. Thanks.

Link to my post: #2043

I just stoped trying to use GPU with llama-cpp-python. Seems some other people have made it work for their env. im not sure how they made it work or if their soultion would fix the issue on my end.

6.14.0-27-generic #27~24.04.1-Ubuntu

same error

(tf) oba@mail:/llama-cpp/llama-cpp-python$ export VERBOSE=1
export CMAKE_ARGS="-DGGML_CUDA=ON -DLLAVA_BUILD=OFF -DCMAKE_CUDA_ARCHITECTURES=all -DGGML_CUDA_FORCE_MMQ=OFF -DGGML_AVX2=OFF -DGGML_FMA=OFF -DGGML_F16C=OFF"
(tf) oba@mail:/llama-cpp/llama-cpp-python$ pip install -e . --verbose
Using pip 25.1 from /home/oba/miniconda3/envs/tf/lib/python3.10/site-packages/pip (python 3.10)
Obtaining file:///home/oba/llama-cpp/llama-cpp-python
Running command pip subprocess to install build dependencies
Using pip 25.1 from /home/oba/miniconda3/envs/tf/lib/python3.10/site-packages/pip (python 3.10)
Collecting scikit-build-core>=0.9.2 (from scikit-build-core[pyproject]>=0.9.2)
Obtaining dependency information for scikit-build-core>=0.9.2 from https://files.pythonhosted.org/packages/45/23/0ffa0df7550ca0535f6e03b9a9ab2bf0495ac62e15fd322544c98321a10c/scikit_build_core-0.11.5-py3-none-any.whl.metadata
Using cached scikit_build_core-0.11.5-py3-none-any.whl.metadata (18 kB)
Collecting exceptiongroup>=1.0 (from scikit-build-core>=0.9.2->scikit-build-core[pyproject]>=0.9.2)
Obtaining dependency information for exceptiongroup>=1.0 from https://files.pythonhosted.org/packages/36/f4/c6e662dade71f56cd2f3735141b265c3c79293c109549c1e6933b0651ffc/exceptiongroup-1.3.0-py3-none-any.whl.metadata
Using cached exceptiongroup-1.3.0-py3-none-any.whl.metadata (6.7 kB)
Collecting packaging>=23.2 (from scikit-build-core>=0.9.2->scikit-build-core[pyproject]>=0.9.2)
Obtaining dependency information for packaging>=23.2 from https://files.pythonhosted.org/packages/20/12/38679034af332785aac8774540895e234f4d07f7545804097de4b666afd8/packaging-25.0-py3-none-any.whl.metadata
Using cached packaging-25.0-py3-none-any.whl.metadata (3.3 kB)
Collecting pathspec>=0.10.1 (from scikit-build-core>=0.9.2->scikit-build-core[pyproject]>=0.9.2)
Obtaining dependency information for pathspec>=0.10.1 from https://files.pythonhosted.org/packages/cc/20/ff623b09d963f88bfde16306a54e12ee5ea43e9b597108672ff3a408aad6/pathspec-0.12.1-py3-none-any.whl.metadata
Using cached pathspec-0.12.1-py3-none-any.whl.metadata (21 kB)
Collecting tomli>=1.2.2 (from scikit-build-core>=0.9.2->scikit-build-core[pyproject]>=0.9.2)
Obtaining dependency information for tomli>=1.2.2 from https://files.pythonhosted.org/packages/6e/c2/61d3e0f47e2b74ef40a68b9e6ad5984f6241a942f7cd3bbfbdbd03861ea9/tomli-2.2.1-py3-none-any.whl.metadata
Using cached tomli-2.2.1-py3-none-any.whl.metadata (10 kB)
Collecting typing-extensions>=4.6.0 (from exceptiongroup>=1.0->scikit-build-core>=0.9.2->scikit-build-core[pyproject]>=0.9.2)
Obtaining dependency information for typing-extensions>=4.6.0 from https://files.pythonhosted.org/packages/b5/00/d631e67a838026495268c2f6884f3711a15a9a2a96cd244fdaea53b823fb/typing_extensions-4.14.1-py3-none-any.whl.metadata
Using cached typing_extensions-4.14.1-py3-none-any.whl.metadata (3.0 kB)
Using cached scikit_build_core-0.11.5-py3-none-any.whl (185 kB)
Using cached exceptiongroup-1.3.0-py3-none-any.whl (16 kB)
Using cached packaging-25.0-py3-none-any.whl (66 kB)
Using cached pathspec-0.12.1-py3-none-any.whl (31 kB)
Using cached tomli-2.2.1-py3-none-any.whl (14 kB)
Using cached typing_extensions-4.14.1-py3-none-any.whl (43 kB)
Installing collected packages: typing-extensions, tomli, pathspec, packaging, exceptiongroup, scikit-build-core

Successfully installed exceptiongroup-1.3.0 packaging-25.0 pathspec-0.12.1 scikit-build-core-0.11.5 tomli-2.2.1 typing-extensions-4.14.1
Installing build dependencies ... done
Running command Checking if build backend supports build_editable
Checking if build backend supports build_editable ... done
Running command Getting requirements to build editable
Getting requirements to build editable ... done
Running command Preparing editable metadata (pyproject.toml)
*** scikit-build-core 0.11.5 using CMake 3.28.3 (metadata_editable)
Preparing editable metadata (pyproject.toml) ... done
Requirement already satisfied: typing-extensions>=4.5.0 in /home/oba/miniconda3/envs/tf/lib/python3.10/site-packages (from llama_cpp_python==0.3.14) (4.14.1)
Requirement already satisfied: numpy>=1.20.0 in /home/oba/miniconda3/envs/tf/lib/python3.10/site-packages (from llama_cpp_python==0.3.14) (1.26.4)
Collecting diskcache>=5.6.1 (from llama_cpp_python==0.3.14)
Obtaining dependency information for diskcache>=5.6.1 from https://files.pythonhosted.org/packages/3f/27/4570e78fc0bf5ea0ca45eb1de3818a23787af9b390c0b0a0033a1b8236f9/diskcache-5.6.3-py3-none-any.whl.metadata
Using cached diskcache-5.6.3-py3-none-any.whl.metadata (20 kB)
Requirement already satisfied: jinja2>=2.11.3 in /home/oba/miniconda3/envs/tf/lib/python3.10/site-packages (from llama_cpp_python==0.3.14) (3.1.6)
Requirement already satisfied: MarkupSafe>=2.0 in /home/oba/miniconda3/envs/tf/lib/python3.10/site-packages (from jinja2>=2.11.3->llama_cpp_python==0.3.14) (2.1.1)
Using cached diskcache-5.6.3-py3-none-any.whl (45 kB)
Building wheels for collected packages: llama_cpp_python
Running command Building editable for llama_cpp_python (pyproject.toml)
*** scikit-build-core 0.11.5 using CMake 3.28.3 (editable)
*** Configuring CMake...
loading initial cache file /tmp/tmpcf8naay4/build/CMakeInit.txt
Re-run cmake no build system arguments
-- The C compiler identification is GNU 13.3.0
-- The CXX compiler identification is GNU 13.3.0
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /usr/bin/gcc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /usr/bin/g++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Found Git: /home/oba/miniconda3/envs/tf/bin/git (found version "2.45.2")
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Success
-- Found Threads: TRUE
-- Warning: ccache not found - consider installing it for faster compilation or disable this warning with GGML_CCACHE=OFF
-- CMAKE_SYSTEM_PROCESSOR: x86_64
-- GGML_SYSTEM_ARCH: x86
-- Including CPU backend
-- Found OpenMP_C: -fopenmp (found version "4.5")
-- Found OpenMP_CXX: -fopenmp (found version "4.5")
-- Found OpenMP: TRUE (found version "4.5")
-- x86 detected
-- Adding CPU backend variant ggml-cpu: -march=native
-- Found CUDAToolkit: /usr/include (found version "12.0.140")
-- CUDA Toolkit found
-- Using CUDA architectures: all
CMake Error at /usr/share/cmake-3.28/Modules/CMakeDetermineCompilerId.cmake:780 (message):
Compiling the CUDA compiler identification source file
"CMakeCUDACompilerId.cu" failed.

Compiler: /usr/bin/nvcc

Build flags:

Id flags: --keep;--keep-dir;tmp -v



The output was:

1

#$ NVCC_PREPEND_FLAGS="
-ccbin=/home/oba/miniconda3/bin/x86_64-conda-linux-gnu-c++"

#$ _NVVM_BRANCH_=nvvm

#$ _SPACE_=

#$ _CUDART_=cudart

#$ _HERE_=/usr/lib/nvidia-cuda-toolkit/bin

#$ _THERE_=/usr/lib/nvidia-cuda-toolkit/bin

#$ _TARGET_SIZE_=

#$ _TARGET_DIR_=

#$ _TARGET_SIZE_=64

#$ NVVMIR_LIBRARY_DIR=/usr/lib/nvidia-cuda-toolkit/libdevice

#$
PATH=/usr/lib/nvidia-cuda-toolkit/bin:/tmp/pip-build-env-fwu3_8_s/overlay/bin:/tmp/pip-build-env-fwu3_8_s/normal/bin:/home/oba/miniconda3/envs/tf/bin:/home/oba/miniconda3/condabin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin


#$ LIBRARIES= -L/usr/lib/x86_64-linux-gnu/stubs -L/usr/lib/x86_64-linux-gnu

#$ rm tmp/a_dlink.reg.c

#$ "/home/oba/miniconda3/bin"/x86_64-conda-linux-gnu-c++
-D__CUDA_ARCH_LIST__=520 -E -x c++ -D__CUDACC__ -D__NVCC__
-D__CUDACC_VER_MAJOR__=12 -D__CUDACC_VER_MINOR__=0
-D__CUDACC_VER_BUILD__=140 -D__CUDA_API_VER_MAJOR__=12
-D__CUDA_API_VER_MINOR__=0 -D__NVCC_DIAG_PRAGMA_SUPPORT__=1 -include
"cuda_runtime.h" -m64 "CMakeCUDACompilerId.cu" -o
"tmp/CMakeCUDACompilerId.cpp4.ii"

<command-line>: fatal error: cuda_runtime.h: No such file or directory

compilation terminated.

# --error 0x1 --

Call Stack (most recent call first):
/usr/share/cmake-3.28/Modules/CMakeDetermineCompilerId.cmake:8 (CMAKE_DETERMINE_COMPILER_ID_BUILD)
/usr/share/cmake-3.28/Modules/CMakeDetermineCompilerId.cmake:53 (__determine_compiler_id_test)
/usr/share/cmake-3.28/Modules/CMakeDetermineCUDACompiler.cmake:135 (CMAKE_DETERMINE_COMPILER_ID)
vendor/llama.cpp/ggml/src/ggml-cuda/CMakeLists.txt:43 (enable_language)

-- Configuring incomplete, errors occurred!

*** CMake configuration failed
error: subprocess-exited-with-error

× Building editable for llama_cpp_python (pyproject.toml) did not run successfully.
│ exit code: 1
╰─> See above for output.

note: This error originates from a subprocess, and is likely not a problem with pip.
full command: /home/oba/miniconda3/envs/tf/bin/python3.10 /home/oba/miniconda3/envs/tf/lib/python3.10/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py build_editable /tmp/tmp_ghueols
cwd: /home/oba/llama-cpp/llama-cpp-python
Building editable for llama_cpp_python (pyproject.toml) ... error
ERROR: Failed building editable for llama_cpp_python
Failed to build llama_cpp_python
ERROR: Failed to build installable wheels for some pyproject.toml based projects (llama_cpp_python)