This dataset is a benchmark that consists of building blocks for 13 IMO problems, and also the complete formal proofs for 20 IMO problems. The topics cover a variety of concepts ranging from divisibility to finite sets and functions. All proof steps are written in Lean 4.

All files compile with no error in Lean v4.17.0.

This dataset is also available on Hugging Face at https://huggingface.co/datasets/roozbeh-yz/IMO-Steps.

The purpose of the dataset is to expose current theorem provers' ability in solving IMO problems and highlight their strengths and weaknesses.

Paper is available at arxiv.org/abs/2411.18872. It is also published at TMLR: https://openreview.net/forum?id=CrKMqRAhBo.

The dataset is released under MIT license. Contributions are welcome through pull requests.

IMO problems formalized in this paper are as follows:

This repository provides a Python script designed to convert .lean files into .jsonl format, making it easier for you to test and analyze your models.

Our tool extracts information such as headers, formal and informal statements from .lean files. The extracted data is saved as separate entries in a .jsonl file in order to integrate with tools like Deepseek-Prover-V1.5 for comprehensive LLM evaluation.

Import the conversion function and call it with the path to your .lean file. Here's a quick example:

from llm_evaluation.utils.lean_to_jsonl import lean_to_jsonl
lean_to_jsonl('path/to/leanfile.lean', save_path='test.jsonl')

This script will extract lemmas from your file and store them as individual entities in the resulting .jsonl file, ensuring that your data is ready to be evaluated by your LLM evaluation tools.

We’re currently working towards improving this repository. In the near future, you can expect a more comprehensive evaluation suite to enhance your workflow and test different LLMs with our dataset. Stay tuned for updates!

The figure demonstrates that the SOTA LLMs tend to write shorter proofs, which is a potential challenge for lengthier proof procedures that require long-term planning.

The dataset, motivations and additional results are available here

@article{
yousefzadeh2025a,
title={A Lean Dataset for International Math Olympiad: Small Steps towards Writing Math Proofs for Hard Problems},
author={Roozbeh Yousefzadeh and Xuenan Cao and Azim Ospanov},
journal={Transactions on Machine Learning Research},
year={2025},
url={https://openreview.net/forum?id=CrKMqRAhBo}
}

The Lemmas Dataset is explicitly written for the latest available Lean4 version at the moment - v4.17.0. Any subsequent release can impact the released lemmas and some proofs might need future revision.