Thanks to visit codestin.com
Credit goes to github.com

Skip to content

Running two instances of skynet against the one instance of skynet with vllm running with a model loaded #164

@ConalMullan

Description

@ConalMullan

I have been trying to find a way to run two instances of skynet which query the same vllm instance accross a docker network.
The idea is that one instance runs vllm, the other doesnt run vllm but querys vllm running on the other instance.

I can setup the docker instances and docker network so that one docker instance can see the instance with vllm running on port 8003 accross the network but I cant get the fastapi endpoints like /summaries to initiate on the instance that vllm is not running on.

Is there some way that I can acheive this?

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions