Thanks to visit codestin.com
Credit goes to github.com

Skip to content

Comments

feat: add MMMLU eval benchmark.#596

Merged
terrykong merged 1 commit intoNVIDIA-NeMo:mainfrom
xxman-google:xx/mmmlu
Jul 9, 2025
Merged

feat: add MMMLU eval benchmark.#596
terrykong merged 1 commit intoNVIDIA-NeMo:mainfrom
xxman-google:xx/mmmlu

Conversation

@xxman-google
Copy link
Contributor

What does this PR do ?

Adds MMMLU eval benchmark.

Issues

N/A

Usage

  • Run eval on MMMLU (Chinese)
uv run examples/run_eval.py \
--config examples/configs/evals/mmlu.yaml \
data.dataset_name="mmlu_ZH-CN" \
cluster.gpus_per_node=8

Before your PR is "Ready for review"

Pre checks:

  • Make sure you read and followed Contributor guidelines
  • Did you write any new necessary tests?
  • Did you run the unit tests and functional tests locally? Visit our Testing Guide for how to run tests
  • Did you add or update any necessary documentation? Visit our Document Development Guide for how to write, build and test the docs.

Additional Information

*N/A

@github-actions github-actions bot added the documentation Improvements or additions to documentation label Jul 3, 2025
@terrykong terrykong requested a review from yuki-97 July 3, 2025 04:04
Copy link
Contributor

@yuki-97 yuki-97 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @xxman-google , overall LGTM! Left some comments.

@xxman-google
Copy link
Contributor Author

@YUki-666 comments addressed. PTAL again.

@yuki-97
Copy link
Contributor

yuki-97 commented Jul 4, 2025

Thanks @xxman-google ! Approved.

@parthchadha @terrykong Can you take a review and enqueue it if it looks good to you?

@terrykong terrykong added this pull request to the merge queue Jul 4, 2025
@github-merge-queue github-merge-queue bot removed this pull request from the merge queue due to failed status checks Jul 4, 2025
@terrykong terrykong added this pull request to the merge queue Jul 8, 2025
Merged via the queue into NVIDIA-NeMo:main with commit 668fe48 Jul 9, 2025
13 of 14 checks passed
@xxman-google xxman-google deleted the xx/mmmlu branch July 11, 2025 16:36
jialei777 pushed a commit to jialei777/nemo-rl that referenced this pull request Jul 23, 2025
KiddoZhu pushed a commit that referenced this pull request Jul 28, 2025
FannYYW pushed a commit to xxman-google/NeMo-RL that referenced this pull request Aug 5, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation external

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants