Thanks to visit codestin.com
Credit goes to github.com

Skip to content

feat: Add fallback logic to the small, medium, and large E2E tests to select a new AZ when AWS has insufficient capacity #3161

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
mergify merged 1 commit into from
Apr 28, 2025
Merged

feat: Add fallback logic to the small, medium, and large E2E tests to select a new AZ when AWS has insufficient capacity #3161

mergify merged 1 commit into from
Apr 28, 2025

Conversation

@courtneypacheco
Copy link
Contributor

@courtneypacheco courtneypacheco commented Feb 18, 2025

Issue resolved by this Pull Request:
Resolves #3160

Checklist:

  • Commit Message Formatting: Commit titles and messages follow guidelines in the
    conventional commits.
  • Changelog updated with breaking and/or notable changes for the next minor release.
  • Documentation has been updated, if necessary.
  • Unit tests have been added, if necessary.
  • Functional tests have been added, if necessary.
  • E2E Workflow tests have been added, if necessary.

Overview

The large E2E test has been failing due to "insufficient instance capacity" in AWS. In other words, whenever we manually kick off our large E2E job or the large E2E job gets kicked off at its regularly scheduled interval, AWS almost always returns an error letting us know that there aren't enough machines (VMs) available to run that job in AWS. Thus, the large E2E job fails to run:

Screenshot 2025-02-17 at 7 25 33 PM

See linked issue for more details.

Proposed Solution

This PR takes the fallback logic I added in #2975 and tweaks it so that it can also be used in our small, medium, and large E2E jobs.

@mergify mergify bot added CI/CD Affects CI/CD configuration ci-failure PR has at least one CI failure labels Feb 18, 2025
@courtneypacheco courtneypacheco force-pushed the add-fallback-logic-to-small-med-and-large-e2e-tests branch from 8bec4e9 to c7a6acd Compare February 18, 2025 00:35
@mergify mergify bot added ci-failure PR has at least one CI failure and removed ci-failure PR has at least one CI failure labels Feb 18, 2025
@github-actions
Copy link

github-actions bot commented Apr 19, 2025

This pull request has been automatically marked as stale because it has not had activity within 60 days. It will be automatically closed if no further activity occurs within 30 days.

@github-actions github-actions bot added the stale label Apr 19, 2025
@courtneypacheco courtneypacheco force-pushed the add-fallback-logic-to-small-med-and-large-e2e-tests branch from c7a6acd to d37f039 Compare April 28, 2025 21:03
@mergify mergify bot removed the ci-failure PR has at least one CI failure label Apr 28, 2025
@courtneypacheco courtneypacheco force-pushed the add-fallback-logic-to-small-med-and-large-e2e-tests branch from d37f039 to 75b7b8d Compare April 28, 2025 21:17
@courtneypacheco courtneypacheco marked this pull request as ready for review April 28, 2025 21:18
@mergify mergify bot added the ci-failure PR has at least one CI failure label Apr 28, 2025
@courtneypacheco courtneypacheco force-pushed the add-fallback-logic-to-small-med-and-large-e2e-tests branch from 75b7b8d to f98f92b Compare April 28, 2025 21:21
@mergify mergify bot removed the ci-failure PR has at least one CI failure label Apr 28, 2025
@github-actions
Copy link

github-actions bot commented Apr 28, 2025

E2E (NVIDIA L40S x4) workflow launched on this PR: View run

@github-actions
Copy link

github-actions bot commented Apr 28, 2025

e2e workflow failed on this PR: View run, please investigate.

@mergify mergify bot added the ci-failure PR has at least one CI failure label Apr 28, 2025
@booxter
Copy link
Contributor

booxter commented Apr 28, 2025

The linked job failed when running tests:

+ ilab model train --strategy=lab-skills-only --phased-phase2-data= --phased-phase2-num-epochs=1 --skip-user-confirm --phased-base-dir=/home/tmp/tmp.L2P9KCOOvo/.local/share/instructlab/skills-only
+ tee /home/tmp/tmp.L2P9KCOOvo/skills_only_training.log
Usage: ilab model train [OPTIONS]
Try 'ilab model train --help' for help.

Error: Invalid value for '--phased-phase2-data': File '' is a directory.
+ grep 'Best final checkpoint: ' /home/tmp/tmp.L2P9KCOOvo/skills_only_training.log
+ grep -o '/[^ ]*'
+ rm -rf /home/tmp/tmp.L2P9KCOOvo

This is irrelevant to the patch here because all it does is it affecs the start and stop steps of CI jobs, not the test itself.

booxter
booxter approved these changes Apr 28, 2025
View reviewed changes
Copy link
Contributor

@booxter booxter left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This approach is validated in other jobs before; both in this repo and in training (maybe more). I checked that instance types are the same, so no change there. And we have at least several different workflow runs collected that prove that the retry logic worked and that the test phase was started successfully (even if failed due to unrelated reasons later). I think we should get this in asap not to waste time of engineers and CI resources (trying to respin jobs over and over - and failing).

@booxter booxter requested a review from a team April 28, 2025 22:43
@mergify mergify bot added the one-approval PR has one approval from a maintainer label Apr 28, 2025
@booxter booxter mentioned this pull request Apr 28, 2025
Closed
6 tasks
@mergify mergify bot merged commit 4ca72aa into main Apr 28, 2025
28 of 30 checks passed
@mergify mergify bot removed the one-approval PR has one approval from a maintainer label Apr 28, 2025
@mergify mergify bot deleted the add-fallback-logic-to-small-med-and-large-e2e-tests branch April 28, 2025 23:10
@courtneypacheco
Copy link
Contributor Author

courtneypacheco commented Apr 30, 2025

@mergify backport release-v0.26

@mergify
Copy link
Contributor

mergify bot commented Apr 30, 2025 •
edited
Loading

backport release-v0.26

✅ Backports have been created

Details

@mergify mergify bot mentioned this pull request Apr 30, 2025
Closed
6 tasks
@booxter booxter mentioned this pull request Apr 30, 2025
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@RobotSail RobotSail RobotSail approved these changes

+1 more reviewer

@booxter booxter booxter approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

CI/CD Affects CI/CD configuration ci-failure PR has at least one CI failure

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

feat: Add fallback logic to select new AZ when AWS has insufficient capacity -- for small, medium, and large E2E tests

4 participants

@courtneypacheco @booxter @RobotSail