Failed job submission should not break the build/launch/update #504
Description
Example, this error below was causing the entire build/launch/update to fail for the project.
launch_MORGAN_DWI_20250402-110114.log:
2025-04-02 11:01:15,069 - INFO - launcher - Launching job: MORGAN_DWI-x-159650-x-159650-x-slant_gpu_v1-x-e8276773-7574-4b1b-99ad-da7aeab0cdf3
2025-04-02 11:01:15,079 - ERROR - cluster - sbatch: error: QOSGrpGRES
sbatch: error: Batch job submission failed: Job violates accounting/QOS policy (job submit limit, user's size and/or time limits)