Thanks to visit codestin.com
Credit goes to github.com

Skip to content

support multi modal inputs #1617

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 1 commit into
base: main
Choose a base branch
from
Open

support multi modal inputs #1617

wants to merge 1 commit into from
+135 −23

Conversation

learner0810
Copy link
Contributor

@learner0810 learner0810 commented Sep 19, 2025 •
edited
Loading

What type of PR is this?

/kind feature

What this PR does / why we need it:

support multi modal inputs

curl ${IP}:${PORT}/v1/chat/completions \
        -H "Content-Type: application/json" \
        -d '{
            "model": "HuggingFaceTB/SmolVLM-256M-Instruct",
            "messages": [
                {"role": "system", "content": "You are a helpful assistant."},
                {"role": "user", "content": "Who won the world series in 2020?"}
            ]
        }'
{"id":"chatcmpl-4224e04933fc430a9011d62161ec7ab9","object":"chat.completion","created":1758270447,"model":"HuggingFaceTB/SmolVLM-256M-Instruct","choices":[{"index":0,"message":{"role":"assistant","reasoning_content":null,"content":" Annariah Roberts did just that.","tool_calls":[]},"logprobs":null,"finish_reason":"stop","stop_reason":null}],"usage":{"prompt_tokens":30,"total_tokens":38,"completion_tokens":8,"prompt_tokens_details":null},"prompt_logprobs":null}

curl ${IP}:${PORT}/v1/chat/completions  \
     -H "Content-Type: application/json"  \
     -d '{
                "model": "HuggingFaceTB/SmolVLM-256M-Instruct",
                "messages": [
                        {
                                "role": "user",
                                "content": [
                                        {
                                                "type": "text",
                                                "text": "Describe this image in one sentence."
                                        },
                                        {
                                                "type": "image_url",
                                                "image_url": {
                                                        "url": "https://cdn2.thecatapi.com/images/dui.jpg"
                                                }
                                        }
                                ]
                        }
                ]
        }'
{"id":"chatcmpl-a7815a0088d6435b8c0d6c2a6b2624ce","object":"chat.completion","created":1758270428,"model":"HuggingFaceTB/SmolVLM-256M-Instruct","choices":[{"index":0,"message":{"role":"assistant","reasoning_content":null,"content":" Turning out toilet behavior of a white dog.","tool_calls":[]},"logprobs":null,"finish_reason":"stop","stop_reason":null}],"usage":{"prompt_tokens":877,"total_tokens":887,"completion_tokens":10,"prompt_tokens_details":null},"prompt_logprobs":null}

Which issue(s) this PR fixes:

Fixes #

Does this PR introduce a user-facing change?:

@k8s-ci-robot k8s-ci-robot added kind/feature Categorizes issue or PR as related to a new feature. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. labels Sep 19, 2025
@k8s-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: learner0810
Once this PR has been reviewed and has the lgtm label, please assign kfswain for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@netlify Netlify
Copy link

netlify bot commented Sep 19, 2025 •
edited
Loading

✅ Deploy Preview for gateway-api-inference-extension ready!

Name Link
🔨 Latest commit 354a79d
🔍 Latest deploy log https://app.netlify.com/projects/gateway-api-inference-extension/deploys/68ddd44545f1aa0007c3165b
😎 Deploy Preview https://deploy-preview-1617--gateway-api-inference-extension.netlify.app
📱 Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify project configuration.

@k8s-ci-robot k8s-ci-robot added the needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. label Sep 19, 2025
@k8s-ci-robot
Copy link
Contributor

Hi @learner0810. Thanks for your PR.

I'm waiting for a kubernetes-sigs member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@k8s-ci-robot k8s-ci-robot added the size/M Denotes a PR that changes 30-99 lines, ignoring generated files. label Sep 19, 2025
@learner0810 learner0810 changed the title support multi modal inputs WIP: support multi modal inputs Sep 19, 2025
@k8s-ci-robot k8s-ci-robot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Sep 19, 2025
@learner0810
Copy link
Contributor Author

learner0810 commented Sep 19, 2025 •
edited
Loading

@nirrozenbaum

Hello, sorry to bother you, but I have a question to ask. If we change Content in types.Message to any, the test case for message with non-string content fails. While using a custom type or generics could resolve this, both approaches require significant code changes. Which solution would you prefer?

image

@kfswain
Copy link
Collaborator

kfswain commented Sep 19, 2025

@nirrozenbaum

Hello, sorry to bother you, but I have a question to ask. If we change Content in types.Message to any, the test case for message with non-string content fails. While using a custom type or generics could resolve this, both approaches require significant code changes. Which solution would you prefer?

image

Generics usually result in pretty clean code, happy to see a solution with those implemented

@learner0810
Copy link
Contributor Author

@nirrozenbaum
Hello, sorry to bother you, but I have a question to ask. If we change Content in types.Message to any, the test case for message with non-string content fails. While using a custom type or generics could resolve this, both approaches require significant code changes. Which solution would you prefer?
image

Generics usually result in pretty clean code, happy to see a solution with those implemented

If we were using generics, we would need two structs to parse the 'chat/completions request body. If we define two structs, then generics feels unnecessary. Please correct me if my understanding is wrong

image

@learner0810 learner0810 force-pushed the support-multi-modal-inputs branch from c1934b9 to 9ee1c1c Compare September 22, 2025 05:37
@k8s-ci-robot k8s-ci-robot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. and removed size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels Sep 22, 2025
@learner0810 learner0810 force-pushed the support-multi-modal-inputs branch from 9ee1c1c to 6420740 Compare September 22, 2025 05:38
@learner0810 learner0810 changed the title WIP: support multi modal inputs support multi modal inputs Sep 23, 2025
@k8s-ci-robot k8s-ci-robot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Sep 23, 2025
@danehans
Copy link
Contributor

danehans commented Oct 1, 2025

@learner0810 we already have this issue that relates to this PR. Since @zhengkezhou1 submitted a PR for this issue and has some overlap with your PR, can you work together to resolve the overlap? @zhengkezhou1 maybe your PR simply adds the e2e test case based on this PRs type changes. Let me know what you think?

@learner0810
Copy link
Contributor Author

@learner0810 we already have this issue that relates to this PR. Since @zhengkezhou1 submitted a PR for this issue and has some overlap with your PR, can you work together to resolve the overlap? @zhengkezhou1 maybe your PR simply adds the e2e test case based on this PRs type changes. Let me know what you think?

I'm very happy to work with @zhengkezhou1 to resolve the overlap. I'll modify the type to follow the llm-d-sim types.

@learner0810 learner0810 force-pushed the support-multi-modal-inputs branch from 6420740 to 5fd9357 Compare October 2, 2025 01:23
@learner0810 learner0810 force-pushed the support-multi-modal-inputs branch from 5fd9357 to 354a79d Compare October 2, 2025 01:24
@learner0810
Copy link
Contributor Author

learner0810 commented Oct 2, 2025 •
edited
Loading

@learner0810 we already have this issue that relates to this PR. Since @zhengkezhou1 submitted a PR for this issue and has some overlap with your PR, can you work together to resolve the overlap? @zhengkezhou1 maybe your PR simply adds the e2e test case based on this PRs type changes. Let me know what you think?

I'm very happy to work with @zhengkezhou1 to resolve the overlap. I'll modify the type to follow the llm-d-sim types.

PTAL, I've already modified my PR. After merging my PR, @zhengkezhou1 reverted the code changes to the types, and then his e2e test case passed. cc @danehans

@zhengkezhou1
Copy link
Contributor

yes, it's good to change Content type here.

@zhengkezhou1 zhengkezhou1 mentioned this pull request Oct 2, 2025
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@danehans danehans Awaiting requested review from danehans

@nirrozenbaum nirrozenbaum Awaiting requested review from nirrozenbaum

Assignees
No one assigned
Labels
cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. kind/feature Categorizes issue or PR as related to a new feature. needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. size/L Denotes a PR that changes 100-499 lines, ignoring generated files.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@learner0810 @k8s-ci-robot @kfswain @danehans @zhengkezhou1