OpenArc is an inference engine for Intel devices. Serve LLMs, VLMs, Whisper, Kokoro-TTS, Embedding and Reranker models over OpenAI compatible endpoints, powered by OpenVINO on your device. Local, private, open source AI.

OpenArc 2.0 arrives with more endpoints, better UX, pipeline paralell, NPU support and much more!

Drawing on ideas from llama.cpp, vLLM, transformers, OpenVINO Model Server, Ray, Lemonade, and other projects cited below, OpenArc has been a way for me to learn about inference engines by trying to build one myself.

Along the way a Discord community has formed around this project, which was unexpected! If you are interested in using Intel devices for AI and machine learning, feel free to stop by.

Thanks to everyone on Discord for their continued support!

• Features
• Quickstart
  • Linux
  • Windows
• OpenArc CLI
  • openarc add
  • openarc list
  • openarc serve
  • openarc load
  • openarc status
  • openarc bench
  • openarc tool
• Model Sources
  • LLMs
  • VLMs
  • Whisper
  • Kokoro
  • Embedding
  • Reranker
• Converting Models to OpenVINO IR
• Learning Resources
• Acknowledgments

OpenArc 2.0 arrives with more endpoints, better UX, pipeline paralell, NPU support and much more!

• Multi GPU Pipeline Paralell
• CPU offload/Hybrid device
• NPU device support
• OpenAI compatible endpoints
  • /v1/models
  • /v1/completions: llm only
  • /v1/chat/completions
  • /v1/audio/transcriptions: whisper only
  • /v1/audio/speech: kokoro only
  • /v1/embeddings: qwen3-embedding #33 by @mwrothbe
  • /v1/rerank: qwen3-reranker #39 by @mwrothbe
• jinja templating with AutoTokenizers
• OpenAI Compatible tool calls with streaming and paralell
  • tool call parser currently reads "name", "argument"
• Fully async multi engine, multi task architecture
• Model concurrency: load and infer multiple models at once
• Automatic unload on inference failure
• llama-bench style benchmarking for llm w/automatic database sqlite
• metrics on every request
  • ttft
  • prefill_throughput
  • decode_throughput
  • decode_duration
  • tpot
  • load time
  • stream mode
• More OpenVINO [examples](examples)
• OpenVINO implementation of hexgrad/Kokoro-82M

uv sync

source .venv/bin/activate

uv pip install "optimum-intel[openvino] @ git+https://github.com/huggingface/optimum-intel"

uv pip install --pre -U openvino-genai --extra-index-url https://storage.openvinotoolkit.org/simple/wheels/nightly

	export OPENARC_API_KEY=<api-key>

openarc --help

uv sync

.venv\Scripts\activate

uv pip install "optimum-intel[openvino] @ git+https://github.com/huggingface/optimum-intel"

uv pip install --pre -U openvino-genai --extra-index-url https://storage.openvinotoolkit.org/simple/wheels/nightly

setx OPENARC_API_KEY openarc-api-key

openarc --help

This section documents the CLI commands available to you.

All commands have aliases but are written here for clarity.

openarc add --model-name <model-name> --model-path <path/to/model> --engine <engine> --model-type <model-type> --device <target-device>

openarc add --model-name <model-name> --model-path <path/to/model> --engine <engine> --model-type <model-type> --device <target-device> --vlm-type <vlm-type>

openarc add --model-name <model-name> --model-path <path/to/whisper> --engine ovgenai --model-type whisper --device <target-device> 

openarc add --model-name <model-name> --model-path <path/to/kokoro> --engine openvino --model-type kokoro --device CPU 

openarc add --model-name <model-name> --model-path <path/to/model> --engine ovgenai --model-type llm --device <HETERO:GPU.0,GPU.1> --runtime-config {"MODEL_DISTRIBUTION_POLICY": "PIPELINE_PARALLEL"}

openarc add --model-name <model-name> --model-path <path/to/model> --engine ovgenai --model-type llm --device CPU --runtime-config {"MODEL_DISTRIBUTION_POLICY": "TENSOR_PARALLEL"}

openarc add --model-name <model-name> -model-path <path/to/model> --engine ovgenai --model-type llm --device <HETERO:GPU.0,CPU> --runtime-config {"MODEL_DISTRIBUTION_POLICY": "PIPELINE_PARALLEL"}

openarc list

openarc list --remove --model-name <model-name>

openarc serve start # defauls to 0.0.0.0:8000

openarc serve start --host --openarc-port

openarc serve start --load-models model1 model2

openarc load <model-name>

openarc load <model-name1> <model-name2> <model-name3>

openarc status

openarc bench <model-name> --p <512> --n <128> --r <5>

openarc tool device-props

openarc tool device-detect

There are a few sources of preconverted models which can be used with OpenArc;

OpenVINO on HuggingFace

My HuggingFace repo

LLMs optimized for NPU

Optimum-Intel provides a hands on primer you can use to build some intuition about quantization and post training optimization using OpenVINO.

Intel provides a suite of tools you can use to apply different post training optimization techniques developed over at Neural Network Compression Framwork.

• Use the Optimum-CLI conversion tool to learn how you can convert models to OpenVINO IR from other formats.

• Visit Supported Architectures to see what models can be converted to OpenVINO using the tools described in this section.

• If you use the CLI tool and get an error about an unsupported architecture or "missing export config" follow the link, open an issue reference the model card and the maintainers will get back to you.

Demos help illustrate what you can do with OpenArc and are meant to be extended. I will continue adding to these, but for now they are a good start.

talk_to_llm.py sets up a "chain" between whisper, an LLM, and kokoro. Talk with any LLM you can run on your PC from the command line. Accumulates context and does not filter reasoning (very interesting).

whisper_button.py use spacebar to record audio with whisper and see the transcription right in the terminal. NPU users should probably start here.

Learn more about how to leverage your Intel devices for Machine Learning:

Install OpenVINO

openvino_notebooks

OpenVINO Python API

OpenVINO GenAI Python API

Inference with Optimum-Intel

Optimum-Intel

NPU Devices

vllm with IPEX

Mutli GPU Pipeline Paralell with OpenVINO Model Server

Transformers Auto Classes

OpenArc stands on the shoulders of many other projects:

Optimum-Intel

OpenVINO

OpenVINO GenAI

llama.cpp

vLLM

Transformers

FastAPI

click

rich-click

@article{zhou2024survey,
  title={A Survey on Efficient Inference for Large Language Models},
  author={Zhou, Zixuan and Ning, Xuefei and Hong, Ke and Fu, Tianyu and Xu, Jiaming and Li, Shiyao and Lou, Yuming and Wang, Luning and Yuan, Zhihang and Li, Xiuhong and Yan, Shengen and Dai, Guohao and Zhang, Xiao-Ping and Dong, Yuhan and Wang, Yu},
  journal={arXiv preprint arXiv:2404.14294},
  year={2024}
}

Thanks for your work!!