Search results

Feature Request: Adaptive Item Selection for Expensive Generative Evaluations

Status: Open.

#34 In allenai/olmes;

· ada-ggf25 opened on Dec 18, 2025

[BUG] alpaca_eval_v2:bpb::tulu yields incorrect, inflated BPB scores due to num_chars defaulting to 1

Status: Open.

#33 In allenai/olmes;

· DigtalFinanceAndWorld opened on Dec 17, 2025

Unexpected keyword argument 'prompt_token_ids' for Gemma 3 Eval

Status: Open.

#32 In allenai/olmes;

· DavidUlloa6310 opened on Dec 11, 2025

Multi-worker evaluation fails: CUDA multiprocessing error

Status: Open.

#31 In allenai/olmes;

· zizi0123 opened on Dec 6, 2025

Reproduction of Olmo3

Status: Open.

#30 In allenai/olmes;

· Sawyer117 opened on Dec 5, 2025

GGUF models

Status: Open.

#28 In allenai/olmes;

· arielyako opened on Nov 29, 2025

Reproduction of Tulu-3-8B-SFT on MMLU

Status: Open.

#27 In allenai/olmes;

· danielm1405 opened on Nov 28, 2025

Argument list too long

Status: Open.

#26 In allenai/olmes;

· asparius opened on Nov 26, 2025

Missing datasets

Status: Open.

#25 In allenai/olmes;

· shaltielshmid opened on Nov 20, 2025

What's the recommended way to test local models?

Status: Open.

#21 In allenai/olmes;

· hugoabonizio opened on Jul 22, 2025

potential breaking API changes from alpacal_eval?

Status: Open.

#13 In allenai/olmes;

· mickelliu opened on Jan 31, 2025

Reproducing MATH numbers from OLMo-2 release blog

Status: Open.

#11 In allenai/olmes;

· Neelectric opened on Dec 26, 2024