Add multi-node verl SLURM job #1798
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
oelachqar
reviewed
Jul 8, 2025
| @@ -0,0 +1,5 @@ | |||
| working_dir: ./ | |||
There was a problem hiding this comment.
Is this file needed? How would a user change the conda environment etc ?
There was a problem hiding this comment.
It was recommended in the verl guide. They could update the value in this file, but given the pattern we use in our job configs (copying our working directory to the slurm cluster and installing that in the oumi conda env), hardcoding it makes sense IMO
There was a problem hiding this comment.
- How about the environment "oumi" ?
- I think this would not work with pip installed oumi right ?
There was a problem hiding this comment.
+1, how would this work for users who are pip installing oumi from pypi?
taenin
reviewed
Jul 9, 2025
| # - Run step 1 of verl quickstart: https://verl.readthedocs.io/en/latest/start/quickstart.html | ||
| # | ||
| # Usage: | ||
| # oumi launch up -c configs/examples/misc/grpo_verl_gsm8k_slurm_job.yaml --cluster $OUMI_SLURM_CONNECTIONS --user wizeng |
There was a problem hiding this comment.
Suggested change
| # oumi launch up -c configs/examples/misc/grpo_verl_gsm8k_slurm_job.yaml --cluster $OUMI_SLURM_CONNECTIONS --user wizeng | |
| # oumi launch up -c configs/examples/misc/grpo_verl_gsm8k_slurm_job.yaml --cluster $OUMI_SLURM_CONNECTIONS --user $USER |
| @@ -0,0 +1,5 @@ | |||
| working_dir: ./ | |||
There was a problem hiding this comment.
+1, how would this work for users who are pip installing oumi from pypi?
oelachqar
approved these changes
Jul 10, 2025
oelachqar
reviewed
Jul 10, 2025
penfever
pushed a commit
that referenced
this pull request
Aug 27, 2025
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
configs/examples/misc/slurm_ray_init.sh, which sets up a Ray cluster on Slurm nodes. Reference: https://docs.ray.io/en/latest/cluster/vms/user-guides/community/slurm.html.configs/examples/misc/grpo_verl_gsm8k_slurm_job.yamlfor running verl on Slurm, using an Oumi job configRelated issues
Towards OPE-1338
Before submitting