currently, and for quite some time actually, we have been in a weird situation where no python install is possible without using flags, at least with standard setups such as uv. here's an example:

uv venv --python 3.9
Using CPython 3.9.6 interpreter at: /Applications/Xcode.app/Contents/Developer/usr/bin/python3
Creating virtual environment at: .venv
Activate with: source .venv/bin/activate

source .venv/bin/activate

uv pip install -r requirements.txt
  × No solution found when resolving dependencies:
  ╰─▶ Because only the following versions of numpy are available:
          numpy<1.26.4
          numpy>1.27.dev0
      and you require numpy>=1.26.4,<1.27.dev0, we can conclude that your requirements are unsatisfiable.

      hint: `numpy` was requested with a pre-release marker (e.g., numpy>=1.26.4,<1.27.dev0), but
      pre-releases weren't enabled (try: `--prerelease=allow`)

      hint: `numpy` was found on https://download.pytorch.org/whl/cpu, but not at the requested version
      (numpy>=1.26.4,<1.27.dev0). A compatible version may be available on a subsequent index (e.g.,
      https://download.pytorch.org/whl/cpu). By default, uv will only consider versions that are published
      on the first index that contains a given package, to avoid dependency confusion attacks. If all
      indexes are equally trusted, use `--index-strategy unsafe-best-match` to consider all versions from
      all indexes, regardless of the order in which they were defined.
      
uv pip install --index-strategy unsafe-best-match -r requirements.txt
  × No solution found when resolving dependencies:
  ╰─▶ Because the current Python version (3.9.6) does not satisfy Python>=3.10 and matplotlib>=3.10.0
      depends on Python>=3.10, we can conclude that matplotlib>=3.10.0 cannot be used.
      And because only the following versions of matplotlib are available:
          matplotlib<=3.10.0
          matplotlib==3.10.1
          matplotlib==3.10.3
      and you require matplotlib>=3.10.0, we can conclude that your requirements are unsatisfiable.

      hint: `matplotlib` was requested with a pre-release marker (e.g., all of:
          matplotlib>3.10.0,<3.10.1
          matplotlib>3.10.1,<3.10.3
          matplotlib>3.10.3,<3.11.dev0
      ), but pre-releases weren't enabled (try: `--prerelease=allow`)

The numpy thing was an issue at least straight through torch 2.5 (torch 2.5.1 switched to numpy 2.x). Torch never published their own numpy 1.26.4, switching at 1.26.3 to 2.0, and uv stops with the first valid provider and its already on torch, so uv gives up when it finds the provider doesn't have the version it needs.

this same install works starting at 3.10, (numpy aside). But starting at python 3.13, we get a problem with torch:

uv pip install --index-strategy unsafe-best-match -r requirements.txt
  × No solution found when resolving dependencies:
  ╰─▶ Because only the following versions of torch are available:
          torch<=2.2.1
          torch==2.2.1+cpu
          torch==2.2.2
          torch==2.2.2+cpu
          torch>2.3.dev0
      and torch>=2.2.1,<=2.2.2+cpu has no wheels with a matching Python ABI tag (e.g., `cp313`), we can
      conclude that torch>=2.2.1,<=2.2.2+cpu cannot be used.
      And because you require torch>=2.2.1,<2.3.dev0, we can conclude that your requirements are
      unsatisfiable.

      hint: `torch` was requested with a pre-release marker (e.g., all of:
          torch>2.2.1,<2.2.1+cpu
          torch>2.2.1+cpu,<2.2.2
          torch>2.2.2,<2.2.2+cpu
          torch>2.2.2+cpu,<2.3.dev0
      ), but pre-releases weren't enabled (try: `--prerelease=allow`)

      hint: You require CPython 3.13 (`cp313`), but we only found wheels for `torch` (v2.2.2+cpu) with the
      following Python ABI tags: `cp38`, `cp39`, `cp310`, `cp311`, `cp312`

BTW, PyTorch (>= 2.3) officially dropped support for Python 3.9 as of June 22, 2024, following the NEP 29 deprecation schedule. So it really makes sense to bump this project up to 3.10.

Can we just update Pytorch?

I see 3 files that use Pytorch in the project.

• ./convert_hf_to_gguf.py
• ./convert_lora_to_gguf.py
• ./tools/mtmd/legacy-models/minicpmv-convert-image-encoder-to-gguf.py

I asked gemini to help look for issues. It notes:

Okay, I have analyzed the PyTorch usage in the three provided scripts: ./convert_hf_to_gguf.py, ./convert_lora_to_gguf.py, and ./tools/mtmd/legacy-models/minicpmv-convert-image-encoder-to-gguf.py.

Here's the verification for each file regarding compatibility with PyTorch 2.3 or greater:

1. ./convert_hf_to_gguf.py

• PyTorch Imports:
  • import torch
  • from torch import Tensor (for type hinting)
• Key PyTorch Usages & Verification:
  • torch.load(str(self.dir_model / part_name), map_location="cpu", mmap=True, weights_only=True):
    • This is the primary method for loading PyTorch model parts if they are not in safetensors format.
    • Crucially, it explicitly sets weights_only=True. This is the recommended security practice and aligns with PyTorch's direction. It means that even when PyTorch 2.6+ changes the default of weights_only to True, this script's behavior will remain consistent and safe.
    • Compatibility: This usage is perfectly compatible with PyTorch 2.3+ and future versions.
  • Tensor Attributes and Methods:
    • .dtype (e.g., first_tensor.dtype == torch.float16, data_torch.dtype)
    • .to(torch.float32) (and other type conversions like to torch.float16, torch.bfloat16)
    • .numpy()
    • .shape
    • .reshape(...), torch.cat(...), torch.stack(...), torch.permute(...), torch.transpose(...), torch.div_(...) (in-place), torch.view(...) (used in various model-specific modify_tensors methods).
    • torch.arange(...), torch.tensor(...), torch.exp(...), torch.equal(...), torch.norm(...), torch.from_numpy(...) (via torch.frombuffer in LazyTorchTensor), torch.split(...).
    • Compatibility: All these are fundamental and stable PyTorch tensor operations and attributes. No breaking changes affecting these are anticipated in PyTorch 2.3+.
  • LazyTorchTensor Class:
    • This custom class uses torch.empty for meta tensor creation and torch.frombuffer for creating tensors from remote data.
    • It implements __torch_function__ to wrap PyTorch functions, which is a stable PyTorch feature.
    • The dtypes used (torch.float64, torch.float32, torch.bfloat16, torch.float16, torch.int64, etc.) are all standard.
    • Compatibility: The PyTorch interactions within this class are stable.
  • torch.inference_mode():
    • Used as a context manager.
    • Compatibility: Standard and stable.
• Conclusion for ./convert_hf_to_gguf.py:
  • This script is compatible with PyTorch 2.3 or greater. The explicit use of weights_only=True in torch.load is key. All other PyTorch operations are standard and stable.

2. ./convert_lora_to_gguf.py

• PyTorch Imports:
  • import torch
  • from torch import Tensor (for type hinting)
  • It also imports LazyTorchTensor and ModelBase from ./convert_hf_to_gguf.py, inheriting their PyTorch usage patterns.
• Key PyTorch Usages & Verification:
  • torch.load(input_model, map_location="cpu", weights_only=True):
    • Used as a fallback if adapter_model.safetensors is not found, loading adapter_model.bin.
    • It explicitly sets weights_only=True.
    • Compatibility: Same as in the first script, this is perfectly compatible and future-proof regarding weights_only defaults.
  • LoraTorchTensor Class:
    • This custom class wraps LoRA A and B tensors.
    • Its methods (__getitem__, reshape, view, permute, transpose, to, __torch_function__ for torch.stack, torch.cat) perform operations on the underlying PyTorch tensors using standard PyTorch tensor methods. For example, self._lora_A.reshape(...).
    • Uses lora_a.T (tensor transpose property).
    • Compatibility: All internal operations rely on stable PyTorch tensor functionalities.
  • Inheritance from ModelBase:
    • The LoraModel class inherits from ModelBase (defined in convert_hf_to_gguf.py). It reuses the tensor processing logic, which, as established above, is compatible.
    • Uses LazyTorchTensor.from_eager(tensor).
  • torch.inference_mode():
    • Used as a context manager.
    • Compatibility: Standard and stable.
• Conclusion for ./convert_lora_to_gguf.py:
  • This script is compatible with PyTorch 2.3 or greater. It benefits from the safe torch.load usage and the compatible base class from convert_hf_to_gguf.py.

3. ./tools/mtmd/legacy-models/minicpmv-convert-image-encoder-to-gguf.py

• PyTorch Imports:
  • import torch
  • import torch.nn.functional as F
  • from torch import nn
• Key PyTorch Usages & Verification:
  • Model Definitions (e.g., SiglipVisionEmbeddings, SiglipAttention, Idefics2VisionTransformer):
    • These classes inherit from torch.nn.Module or PreTrainedModel (which itself is an nn.Module).
    • They use standard PyTorch layers: nn.Conv2d, nn.Embedding, nn.Linear, nn.LayerNorm, nn.ModuleList.
    • Standard activation functions are used (via ACT2FN or directly like F.pad).
    • Weight initialization functions (nn.init.normal_, nn.init.zeros_, _trunc_normal_, variance_scaling_) use basic tensor operations like tensor.uniform_, tensor.erfinv_(), tensor.mul_(), tensor.add_(), tensor.clamp_(), tensor.normal_().
    • Compatibility: These are core, stable components of torch.nn and associated tensor manipulations. tensor.erfinv_ is a standard operation.
  • torch.load(os.path.join(dir_model, "minicpmv.clip")):
    • This call loads the weights for the Idefics2VisionTransformer (or SiglipVisionTransformer).
    • Crucially, weights_only is NOT specified.
    • In PyTorch 2.3, 2.4, and 2.5, the default for weights_only is False. This means the script will load the file as is.
    • The loaded object is then passed to model.load_state_dict(...). This function expects a dictionary mapping tensor names to torch.Tensor objects (a state dictionary).
    • If minicpmv.clip is a pickled file containing only a state dictionary (which is highly likely given its use with load_state_dict), then:
      • Loading with weights_only=False (default in 2.3-2.5) works fine.
      • Loading with weights_only=True would also work fine and would be safer.
      • When PyTorch 2.6+ changes the default to weights_only=True, this line of code will continue to work correctly if minicpmv.clip is indeed just a state dictionary of tensors.
    • Compatibility: This usage is compatible with PyTorch 2.3, 2.4, and 2.5. While not an immediate breaking issue for these versions, it's recommended to add weights_only=True to this call if minicpmv.clip is confirmed to be a safe state dictionary, for future-proofing and best practice.
  • Tensor Manipulations:
    • state_dict = model.state_dict()
    • data.squeeze().numpy()
    • v.chunk(3, dim=0)[0], v.transpose(-1, -2).contiguous(), v.unsqueeze(0) (in _replace_name_resampler and _replace_name).
    • torch.from_numpy(...) (used implicitly when torch.from_numpy(get_2d_sincos_pos_embed(...)) is called).
    • Compatibility: All these are standard and stable PyTorch operations.
  • torch.no_grad(): Used as a context manager.
    • Compatibility: Standard and stable.
• Conclusion for ./tools/mtmd/legacy-models/minicpmv-convert-image-encoder-to-gguf.py:
  • This script is compatible with PyTorch 2.3 or greater.
  • The torch.load call without an explicit weights_only parameter is not a breaking issue for PyTorch 2.3-2.5 for loading state dictionaries. It will also likely work with PyTorch 2.6+ if the file is a standard state dictionary. However, for best practice and to eliminate any ambiguity, adding weights_only=True to this specific torch.load call is advisable, assuming minicpmv.clip contains only a state dictionary.

Overall Conclusion:

All three scripts (./convert_hf_to_gguf.py, ./convert_lora_to_gguf.py, and ./tools/mtmd/legacy-models/minicpmv-convert-image-encoder-to-gguf.py) are compatible with upgrading PyTorch to 2.3 or greater.

• The critical torch.load calls in the first two scripts explicitly use weights_only=True, which is ideal.
• The torch.load call in the third script does not specify weights_only, but given its context (loading a state dictionary), this is not a breaking issue for PyTorch 2.3-2.5 and is likely to remain compatible with PyTorch 2.6+ for this specific use case. Adding weights_only=True to it is a good recommendation for robustness.
• All other PyTorch API usages identified are core, stable functionalities that are not subject to breaking changes that would affect these scripts in PyTorch 2.3+.

So, I created a script to check general availability of torch at python version: https://gist.github.com/robbiemu/c224d6daf2f05d918ac8cf759db33288

uv run main.py --toml $GH/llama.cpp/pyproject.toml --requirements $GH/llama.cpp/requirements.txt --targets cpu,cuda
Building dynamic compatibility grid — this may take a few seconds…
Stable variants discovered: cpu, cu100, cu101, cu102, cu110, cu111, cu113, cu115, cu116, cu117, cu118, cu121, cu124, cu126, cu128, cu92, rocm3, rocm4, rocm5, rocm6, xpu
Nightly variants discovered: cpu, cu118, cu126, cu128, rocm6, xpu
Python \ Torch | 2.2 | 2.3 | 2.4 | 2.5 | 2.6 | 2.7 | 2.8
---------------+-----+-----+-----+-----+-----+-----+----
3.9 | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | !
3.10 | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | !
3.11 | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | !
3.12 | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | !
3.13 | X | X | X | ✓ | ✓ | ✓ | !

To support all python versions from 3.10 forward, we will need to upgrade torch to 2.5.

Taken together, we should upgrade to python >=3.10, torch 2.5.1, and numpy >2.0.

So, I wrote a small script to crawl the repo for "import numpy" and "from numpy" and collect all the numpy members used in the files, and asked gemini to check them for safety in upgrading to numpy 2.x, and verified our usage was compatible:

Now, let's verify these against potential breaking changes, primarily focusing on the NumPy 1.x to NumPy 2.0 transition, as this is the most significant recent shift. You're aiming for numpy>=2.0.0 to go with torch>=2.5.0 and Python 3.10+.

Verification of NumPy Members for NumPy 2.0 Compatibility:

1. Core Array Object and Creation:

   • ndarray, array, asarray, empty, ones, zeros, arange, frombuffer, fromfile, fromiter: These are fundamental and remain. Their core behavior is stable.
   • memmap: Stable.
   • Status: ✅ Safe.

2. Data Types (dtypes):

   • dtype (the object itself and as a parameter): Stable.
   • float16, float32, float64: Stable. These are concrete types.
   • int8, int16, int32, int64: Stable.
   • uint8, uint16, uint32, uint64: Stable.
   • bool_: Stable (np.bool_ is the canonical boolean type).
   • complex64: Stable.
   • generic (likely np.generic for type checking): Stable.
   • signedinteger (likely np.signedinteger for type checking): Stable.
   • DTypeLike, NDArray (from numpy.typing): These are for type hinting and are designed to work with NumPy 2.0.
   • Potential Issue (Minor & Unlikely Here): The main breaking change in NumPy 2.0 regarding types was the removal of aliases like np.int, np.float, np.bool. Your list shows specific types (np.int32, np.float64, etc.) or np.bool_, which are the correct, non-deprecated forms. So, this common pitfall is avoided.
   • Status: ✅ Safe.

3. Array Manipulations:

   • concatenate, stack, split, array_split, hsplit: Stable.
   • reshape (implied, as ndarray objects have this method, though not explicitly listed as np.reshape): Stable.
   • moveaxis, take_along_axis: Stable.
   • pad: Stable.
   • packbits, unpackbits: Stable.
   • clip: Stable.
   • triu: Stable.
   • Status: ✅ Safe.

4. Mathematical Operations & Functions:

   • cos, sin, exp, sqrt, floor, trunc, sign: Stable.
   • sum, prod, argmax, count_nonzero: Stable.
   • dot, einsum: Stable.
   • linalg (e.g., np.linalg.norm): The np.linalg submodule is stable.
   • all, allclose, array_equal: Stable.
   • fmax, fmin: Stable.
   • nan (np.nan): Stable.
   • fft (likely np.fft submodule): Stable.
   • hanning (likely np.hanning for window functions): Stable.
   • Status: ✅ Safe.

5. Utilities & Submodules:

   • random (e.g., np.random.random): The np.random module and its functions are stable.
   • lib (e.g., np.lib.stride_tricks): This submodule contains utilities. While some very internal parts could change, common uses are generally stable. np.lib.pad is now np.pad. Your list shows pad, suggesting the direct np.pad is likely used, which is good.
   • errstate: Stable (context manager for floating point errors).
   • set_printoptions: Stable.
   • meshgrid: Stable.
   • where: Stable.
   • Status: ✅ Safe.

Overall Assessment for NumPy >= 2.0.0:

Based on the list of members your script identified:

• The vast majority of these NumPy features are fundamental and have stable APIs that are fully compatible with NumPy 2.0.
• The most common breaking change in NumPy 2.0 (removal of np.int, np.float aliases) does not appear to be an issue here, as the list shows specific dtypes (e.g., np.float32, np.int64).
• The type promotion rules in NumPy 2.0 are more consistent. This is generally a good thing and unlikely to break code unless it relied on very specific, possibly ambiguous, old promotion behaviors. For the types of operations typically done in these scripts (data conversion, basic math on well-defined types), this is usually not a problem.

Conclusion:

Given the specific NumPy members used by the llama.cpp Python scripts, an upgrade to numpy>=2.0.0 is highly likely to be safe and successful.

The identified usages are predominantly core NumPy functionalities that have remained stable or had very clear migration paths that an active project would typically handle. The benefits of moving to NumPy 2.0 (better Python 3.13+ support, API consistency, potential performance improvements) outweigh the minimal risks for this particular set of functions.

This confirms the earlier recommendation to set your NumPy requirement to numpy>=2.0.0 when using torch>=2.5.0 and targeting Python 3.10+.

After updating the requirements (for torch 2.5.1 and torchvision) and pyproject.toml, and installing like:

rm -rf .venv

uv venv --python 3.10
Using CPython 3.10.17
Creating virtual environment at: .venv
Activate with: source .venv/bin/activate

uv pip install --index-url https://pypi.org/simple --index-strategy unsafe-best-match --extra-index-url https://download.pytorch.org/whl/ -r requirements.txt
Resolved 73 packages in 780ms
Prepared 2 packages in 1.50s
Installed 73 packages in 578ms
...
 + numpy==1.26.4
...

and we can verify that numpy v2 would always be acceptable:

pipdeptree --reverse --packages numpy --warn silence

numpy==1.26.4
├── contourpy==1.3.2 [requires: numpy>=1.23]
│   └── matplotlib==3.10.3 [requires: contourpy>=1.0.1]
│       └── seaborn==0.13.2 [requires: matplotlib>=3.4,!=3.6.1]
├── pandas==2.2.3 [requires: numpy>=1.22.4]
│   └── seaborn==0.13.2 [requires: pandas>=1.2]
├── seaborn==0.13.2 [requires: numpy>=1.20,!=1.24.0]
├── gguf==0.16.3 [requires: numpy>=1.17]
├── matplotlib==3.10.3 [requires: numpy>=1.23]
│   └── seaborn==0.13.2 [requires: matplotlib>=3.4,!=3.6.1]
├── torchvision==0.20.1 [requires: numpy]
└── transformers==4.46.3 [requires: numpy>=1.17]

and torch 2.5.1 was compiled on numpy 2.x when numpy 2.1 was current, per this nvidia thread, so >=2.1 makes sense.

This finally uncovers another dependency issue stemming from Pytorch's own published version of requests (they have literally only one):

uv pip install -r requirements/requirements-all.txt
  × No solution found when resolving dependencies:
  ╰─▶ Because only requests==2.28.1 is available and you require requests>=2.32.3, we can conclude that
      your requirements are unsatisfiable.

      hint: `requests` was found on https://download.pytorch.org/whl/cpu, but not at the requested
      version (requests>=2.32.3). A compatible version may be available on a subsequent index (e.g.,
      https://download.pytorch.org/whl/cpu). By default, uv will only consider versions that are published
      on the first index that contains a given package, to avoid dependency confusion attacks. If all
      indexes are equally trusted, use `--index-strategy unsafe-best-match` to consider all versions from
      all indexes, regardless of the order in which they were defined.

and this is easy, because they are both minor versions is a very stable library, and the version numbers are actually close. we can reasonably set requests>=2.28.1. with that, we get at the easy install you'd expect:

uv pip install -r requirements/requirements-all.txt
Resolved 84 packages in 949ms
Prepared 17 packages in 2.45s
Installed 84 packages in 509ms
...

And if we want the latest, we can use uv pip install --upgrade -r requirements/requirements-all.txt --resolution=highest --index-strategy unsafe-best-match