The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Confirmed it works with optimum-cli after the fix. Can now export it like below:

optimum-cli export onnx --model vidore/colpali-v1.3-hf ./onnx_output --task feature-extraction --variant vision

The dtype of fp16 is not working in optimum-cli for now but it seems to be the problem with transformers, but works with fp32 so I just keep it as it is. Will probably fix in transformers in the future.

@IlyasMoutawwakil

Okay I took a deeper look and it seems to me that doing it this way with variants will require users to create two onnx models to solve the retrieval task (which is what colpali is for), the problem is that those two models share most of the same weights.
I think a better way to do it is to follow the example in optimum-intel where for vlms we define a couple "behaviors" that are exported separately and used together during inference.
So instead of having to load two VLMs, one that takes pixel values and another that doesn't, we can divide the forward call defined here into "components"/"behaviors" that run depending on the input.
@Balladie tell me if what I said makes sense 😄

@echarlaix what do you think ? I think we should have at least one clean VLM implementation (currently we have none) so that contributors can use it as a base recipe.

Okay I took a deeper look and it seems to me that doing it this way with variants will require users to create two onnx models to solve the retrieval task (which is what colpali is for), the problem is that those two models share most of the same weights. I think a better way to do it is to follow the example in optimum-intel where for vlms we define a couple "behaviors" that are exported separately and used together during inference. So instead of having to load two VLMs, one that takes pixel values and another that doesn't, we can divide the forward call defined here into "components"/"behaviors" that run depending on the input. @Balladie tell me if what I said makes sense 😄

@echarlaix what do you think ? I think we should have at least one clean VLM implementation (currently we have none) so that contributors can use it as a base recipe.

You've got the point and that's what I exactly thought and looking for (but I did not have much time to work on it atm), so I greatly appreciate that you pointed out the necessity for baseline VLM configs. I did not know there are some vlms already in optimum-intel, and I think it looks fine for the base for now.
So basically what I understand is that we can divide into parts (ex. vision tower, text embeds, lm decoder) for each behavior and export them into three onnx models like this which has compatible formats with transformers.js (that's also what I'd like to have). Would love to see optimum naturally supports that (which has been done quite manually as far as I know), and also it would also be great if optimum-cli supports that as an option for the users (ex. --behavior vision).

I think your team would have better understanding of how it should be implemented but if it's helpful to make a simple starting point at least in this model I can definitely work on it. Kindly request your suggestion on how we can move forward for that!

@IlyasMoutawwakil

Yeah you're right, Also I just noticed that in the notebook created by @kstavro, simply passing zeros as pixel values is enough to getting text embedding from the same "vision" variant. I think having two variants is okay ! Maybe we can even create an inference class ORTModelForRetrieval that enables a user to load a "vision" model and use it the same way as in https://huggingface.co/docs/transformers/en/model_doc/colpali

Hey @Balladie @IlyasMoutawwakil, I just saw the tag from the other issue, thanks for taking care of this.

The main thing to take from my notebook when exporting the model was that in order to export a single model instead of two, the ‘ColPaliProcessor’ needs to be changed to also return dummy pixel_values for text as well (maybe with an argument like return_dummy_pixel_values or so). And that is actually enough for everything to work, as this makes the expected input for the ONNX model uniform.

I think the dummy values don’t actually play any role when processing text, as there are no image tokens as input for the model (image tokens are only concatenatied for image and images are embedded only when model image tokens exist). I first tried to manually concatenate the dummy pixel values to the text tokens for the text cases (while properly setting the attention mask to 0 for the image tokens) and that had the exact same results. But with the additional computational cost of 1024 unnecessary image tokens, so not worth it. Then I realized that what I was doing was a modified ColPaliProcessor in its essence.

The trade off to such an approach is that the ColPaliProcessor is part of the transformers library, not optimum. Where as with the export of two models everything remains more self contained, but with double the memory consumption. That is why I asked in the other issue about any design preferences.

I hope this helps somehow, thank you for integrating this model to the library :)

@IlyasMoutawwakil have manually tested that separating into three parts works (codes and onnx models here)
it's not a full reflection of the concept of "behavior" but the flow would be aligned so I just share it with current form, hope it helps

@kstavro no need to modify the processor logic, dummy pixel values could be created rather in the inference class ORTModelForRetrieval for example if not passed.
We do something similar for LLMs where we export them with past key values and create dummy ones in the first forward pass.

Hi @Balladie @IlyasMoutawwakil, I am just back from my vacation and I noticed that the PR is still open. Do you require any assistance from my side for the implementation? If desired and/or preferred, I could find some time on the weekend to assist with the PR.

Thanks again for taking care of this so far :)