os-chat allows you to chat with your computer, retrieving all kind of information from log files, CLI commands and an osquery interface.

1. Install osquery
2. Install PDM

Setup the project with

pdm install

Setup OpenAI API key:

export OPENAI_API_KEY=sk-***

pdm run start

The goal of the benchmark is to measure the ability of different LLMs to retriev the relevant information from the available interfaces.

Currently local LLMs are not able to use the interface correctly and therefore retriev no information.

To run the benchmark use the following pdm script:

pdm run benchmark

Note: Currently the benchmark targets are hardcoded for one specific test system and need to be adapted to run on any other system.