Tidy up FLAMINGO job scripts #27
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This reduces startup time when processing a small number of halos in a large simulation.
This modifies the paths to be more consistent with previous runs and makes the compression scripts work in the same way as the group membership and halo properties scripts. Not implemented for the other boxes yet.
This submits array jobs to * compute halo membership * compute halo properties * compress membership files * compress halo properties with dependencies to ensure they run in the right order.
|
For L0100N0180 this now includes a script that submits a set of array jobs with dependencies to compute group membership, run SOAP, and then compress the output for a specified range of snapshots. |
|
Can this be merged now? |
|
Does it work for you @jemme07 ? |
|
It would be good to hear that this works for someone other than me before we merge it. Some of the time limits probably need to be increased too. |
|
@jemme07 ? (I have been using my own older scripts....) |
|
@jemme07 any updates on this? |
|
@jemme07 any updates? |
I have been using my own old scripts as well, but I can test it on one snap. |
|
Any news? |
|
I'm not sure anyone else is using this version of the scripts, but they work for me. |
|
Let's merge this for the record then. |
MatthieuSchaller
approved these changes
Aug 17, 2023
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Here I've tried to set up the FLAMINGO batch scripts so that they will work for Virgo users on Cosma without modifications. Scratch files are written to /snap8/scratch/dp004/${USER}/FLAMINGO/ and compressed output goes to /cosma8/data/dp004/${USER}/FLAMINGO/.
I've also removed the separate hydro and dmo versions of the halo properties scripts to reduce duplication. SOAP is now run with the
--dmoflag if the simulation name starts withDMO_.I haven't done anything with the compression scripts yet.
(also fixes #23 and includes small test scripts from #15)
This now also adds a top level script that can submit jobs:
This will add dependencies between jobs where necessary.