ZNOB is a multimodal benchmark measuring frontier LLMs' capabilities in passing Ukrainian national exams.
conda create -n znob python=3.9 -y # up to 3.13
conda activate znob
git clone https://github.com/Inc44/ZNOB.git
cd ZNOB
pip install -r requirements.txtSet environment variable:
setx /M OPENROUTER_API_KEY your_api_keyFor Linux/macOS:
echo 'export OPENROUTER_API_KEY="your_api_key"' >> ~/.bashrc # or ~/.zshrcOr create a .env file or modify /etc/environment:
OPENROUTER_API_KEY=your_api_key
Check by restarting the terminal and using:
echo %OPENROUTER_API_KEY%For Linux/macOS:
echo $OPENROUTER_API_KEYpython -m znob.cli -d your_zno_dataset -u your_zno_sourcepython -m znob.cli -d your_zno_dataset --model google/gemini-2.5-flashpython -m znob.cli -d your_zno_dataset -r responses,combined_responses,summary # or questions or all| Argument | Description |
|---|---|
-u, --url <url> |
Dataset source. |
-d, --dataset <path> |
Dataset to test. |
-m, --model <name> |
AI model to test. |
-r, --reset <name> |
Reset outputs. |
--no-text |
Send only image, no text. |
--no-image |
Send only text, no image. |
--necessary-image-only |
Send image only if necessary. |
LLMs have made significant progress since November 2024, so I decided to measure their progress and also verify the claims of the Benchmarking Multimodal Models for Ukrainian Language Understanding Across Academic and Cultural Domains research paper.
Not yet found.
Not yet known.
Not yet planned.