@mergify rebase

rebase

✅ Branch has been successfully rebased

CUDA job failure here is because of #3111 which is not related to python 3.12

E2E (NVIDIA L40S x4) workflow launched on this PR: View run

e2e workflow succeeded on this PR: View run, congrats!

@RobotSail so the NVIDIA L40S x4 job passed. What else is missing / broken that we could validate?

@booxter, probably worth rebasing to use prebuilt vLLM wheel and pass the failed test.

@booxter This test didn't test the codepath of training with use_dolomite: true, which is the part that blocked us from migrating last time.

@RobotSail thanks for this. Would that be enough to trigger it?

diff --git a/src/instructlab/defaults.py b/src/instructlab/defaults.py
index 67f4695e..7afb9eda 100644
--- a/src/instructlab/defaults.py
+++ b/src/instructlab/defaults.py
@@ -123,7 +123,7 @@ class _InstructlabDefaults:
         "lora_alpha": 32,
         "lora_dropout": 0.1,
         "lora_target_modules": ["q_proj", "k_proj", "v_proj", "o_proj"],
-        "use_dolomite": False,
+        "use_dolomite": True,
     }
     SUPPORTED_CONTENT_FORMATS = ["json"]

Yeah that should work

@booxter If you are already running on 4xL40s then make sure to disable lora to do a full fine tune

@RobotSail probably a stupid question but! :) how do I do this in CI? Isn't it already disabled for 4xL40s job?

This pull request has been automatically marked as stale because it has not had activity within 60 days. It will be automatically closed if no further activity occurs within 30 days.

This pull request has merge conflicts that must be resolved before it can be merged. @booxter please rebase it. https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

We can revive when there's a request and staffing to make this happen.

FYI: functional failure in the latest run is because of infra issue in Quay returning 502. Unrelated.

E2E (NVIDIA L40S x4) workflow launched on this PR: View run

e2e workflow failed on this PR: View run, please investigate.

E2E (NVIDIA L40S x4) workflow launched on this PR: View run

e2e workflow succeeded on this PR: View run, congrats!

This pull request has merge conflicts that must be resolved before it can be merged. @booxter please rebase it. https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

E2E (NVIDIA L40S x4) - Python 3.12 workflow launched on this PR: View run

e2e workflow failed on this PR: View run, please investigate.

Sigh. I hit the bug with symbols because the workflow files from the Cortney's branch for #3161 didn't include it. I will have to wait for #3161 to land before retrying.

E2E (NVIDIA L40S x4) - Python 3.12 workflow launched on this PR: View run

E2E (NVIDIA L40S x4) - Python 3.12 workflow launched on this PR: View run