Thanks to visit codestin.com
Credit goes to github.com

Skip to content

Disable broken and outdated CI #39467

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 16 commits into from
Mar 9, 2025
Merged

Conversation

tobiasdiez
Copy link
Contributor

@tobiasdiez tobiasdiez commented Feb 6, 2025

Many of the CI runs after a new release are failing, for months now, see eg https://github.com/sagemath/sage/actions/runs/12979684199/job/36218126145. Some of these failures are genuine (eg a certain package cannot be built on a certain system) and some others are due to constraints of the build system (eg running out of free space). Since there is very little point in senselessly burning energy, all runs that were failing for the last releases are disabled. Once the underlying issues are fixed, they can be easily be re-enabled.

Moreover, the "minimal" runs where only a couple of systems packages are installed and most are build using sage are removed, keeping only the "standard" where all available system packages (corresponding to standard sage packages) are installed.

New test run: https://github.com/tobiasdiez/sage/actions/runs/13199372232/job/36847711005

📝 Checklist

  • The title is concise and informative.
  • The description explains in detail what this PR is about.
  • I have linked a relevant issue or discussion.
  • I have created tests covering the changes.
  • I have updated the documentation and checked the documentation preview.

⌛ Dependencies

Copy link

github-actions bot commented Feb 6, 2025

Documentation preview for this PR (built with commit 1364c41; changes) is ready! 🎉
This preview will update shortly after each push to this PR.

@user202729
Copy link
Contributor

There's a risk of we just forget to reenable it. Is there a good way to do this dynamically (i.e. check which tests the last release fail and skip these)?

@tobiasdiez
Copy link
Contributor Author

There's a risk of we just forget to reenable it.

I'll open an issue where we can track which systems are failing, so we don't forget it. The issue tracker might actually be a more visible place in the first place as most people will not check the CI runs.

Is there a good way to do this dynamically (i.e. check which tests the last release fail and skip these)?

Not that I know.

@tobiasdiez tobiasdiez marked this pull request as ready for review February 7, 2025 12:04
@tobiasdiez tobiasdiez requested review from dimpase and kwankyu February 7, 2025 12:04
@dimpase
Copy link
Member

dimpase commented Feb 7, 2025

Great, thanks for taking care of this.

@dimpase

This comment was marked as off-topic.

@kwankyu
Copy link
Collaborator

kwankyu commented Feb 8, 2025

Removing things in CI may break other things elsewhere. Especially the developer guide. In particular, https://doc-release--sagemath.netlify.app/html/en/developer/portability_testing. Did you check?

The docker images created by CI linux are used by developers. The sage binder repo (https://github.com/sagemath/sage-binder-env) is using them (in particular the "minimal" image). The interactive sage doc relies on them. This PR will break the doc.

Changes affecting developers widely should be discussed or at least posted on sage-devel. This PR falls in the category.

all runs that were failing for the last releases are disabled. Once the underlying issues are fixed, they can be easily be re-enabled.

We do not remove code for temporary change. Do not invent a new protocol that code is removed to be disabled.

Improving CI is good, but the present PR is also destructive. I object to this PR.

@kwankyu kwankyu added disputed PR is waiting for community vote, see https://groups.google.com/g/sage-devel/c/IgBYUJl33SQ and removed s: positive review labels Feb 8, 2025
@dimpase
Copy link
Member

dimpase commented Feb 8, 2025

@kwankyu What exactly is broken by this PR?
Please point it out. This PR is removing CI bits which have been broken for a long time. There is nothing wrong with this.

@dimpase

This comment was marked as abuse.

@dimpase dimpase added s: needs info and removed disputed PR is waiting for community vote, see https://groups.google.com/g/sage-devel/c/IgBYUJl33SQ labels Feb 8, 2025
@user202729
Copy link
Contributor

Technically you're right in that this merely remove part of CI that has been broken by some other change made months ago (thus doesn't "in addition" break anything), and reverting this is as simple as a git revert, but I think there is a small disadvantage in deleting the code entirely (compared to e.g. just comment it out): people unfamiliar with the change, coming across the file, would have no indication that something was temporarily disabled.

I don't personally use the minimal configurations and have no strong opinion on whether they're actually useful to developers. Of course having more configurations lead to additional maintenance burden and spent computing power (such as the maintenance cost needed now to make the minimal configuration build again), the cost may not be justified by the usefulness to developers/users.

Anyway, since the intention of this PR is to temporarily disable the run until the bug is fixed, can't we just comment it out?

I mean, we're all volunteers, if anyone volunteers to fix the minimal configurations then of course that will supersede this pull request and everyone are happy [1]… but if nobody does, the feature unfortunately will remain broken.

[1] maybe except those who prefer the minimal configurations to be deleted anyway, but I think the usual workflow is deleting a feature requires 1-year deprecation period and/or announcement of some sort.

@dimpase

This comment was marked as off-topic.

@kwankyu
Copy link
Collaborator

kwankyu commented Feb 8, 2025

@kwankyu What exactly is broken by this PR? Please point it out. This PR is removing CI bits which have been broken for a long time. There is nothing wrong with this.

Did you see the section of the developer guide that I pointed out?

The removed chunks of ci-linux are responsible to push various docker images. One of the image, ubuntu-jammy-standard, is used in the CI check workflows for PRs. This PR will collapse the whole CI checks.

Other images, ubuntu-jammy-minimal-with-targets and ubuntu-jammy-minimal-with-system-packages, are used in sage-binder-env. This PR will collapse it and the interactive sage doc that is using it.

@user202729
Copy link
Contributor

user202729 commented Feb 8, 2025

Can we not escalate this, thanks.

Anyway, I think a good compromise is

  • just merge this
  • open an issue to track which systems it fails on (i.e. equivalent to saying "it is a known bug that the build fails on system X")
  • if someone comes forward with a pull request to fix it later we should incorporate it (the burden is on them to show it works well across multiple operating systems, and we may add a big red warning that tool X or package Y is better installed with operating system installer)

Do you think this is reasonable? (anyway I'm not a maintainer and would be happy with anything)

p/s. @kwankyu , I don't really completely understand what's going on, but if the particular CI has been failing for months, how come the incremental things etc. still work (this is test-new right?) Or does this PR also remove working runs?

@tobiasdiez
Copy link
Contributor Author

Follow-up is at #39603 for anyone interested in fixing the most serious issues.

@kwankyu
Copy link
Collaborator

kwankyu commented Feb 27, 2025

No one would object to reducing complexity of the build system. The build system changes first, and then CI will follow. Not the other way around.

you cannot CI with build targets/configs which are not there any more. So it can't be done sequentially.

If you makes changes to targets/configs in Makefile.in or config.ac, then you need to adjust CI reflecting the changes in the build system. If you do both at the same time, that is nice.

But if you make changes in the build system without adjusting CI, and thus CI is broken temporarily until someone fixes CI, then that is not ideal. But that is really not a big deal.

That's not a problem, just populate your pip cache with all the packages not in upstream/ that Sage needs, and then you don't need an internet connection.

OK.

vbraun pushed a commit to vbraun/sage that referenced this pull request Mar 1, 2025
sagemathgh-39467: Disable broken and outdated CI
    
<!-- ^ Please provide a concise and informative title. -->
<!-- ^ Don't put issue numbers in the title, do this in the PR
description below. -->
<!-- ^ For example, instead of "Fixes sagemath#12345" use "Introduce new method
to calculate 1 + 2". -->
<!-- v Describe your changes below in detail. -->
<!-- v Why is this change required? What problem does it solve? -->
<!-- v If this PR resolves an open issue, please link to it here. For
example, "Fixes sagemath#12345". -->

Many of the CI runs after a new release are failing, for months now, see
eg https://github.com/sagemath/sage/actions/runs/12979684199/job/3621812
6145. Some of these failures are genuine (eg a certain package cannot be
built on a certain system) and some others are due to constraints of the
build system (eg running out of free space). Since there is very little
point in senselessly burning energy, all runs that were failing for the
last releases are disabled. Once the underlying issues are fixed, they
can be easily be re-enabled.

Moreover, the "minimal" runs where only a couple of systems packages are
installed and most are build using sage are removed, keeping only the
"standard" where all available system packages (corresponding to
standard sage packages) are installed.

New test run: https://github.com/tobiasdiez/sage/actions/runs/1319937223
2/job/36847711005

### 📝 Checklist

<!-- Put an `x` in all the boxes that apply. -->

- [ ] The title is concise and informative.
- [ ] The description explains in detail what this PR is about.
- [ ] I have linked a relevant issue or discussion.
- [ ] I have created tests covering the changes.
- [ ] I have updated the documentation and checked the documentation
preview.

### ⌛ Dependencies

- sagemath#39251 (to resolve merge conflicts)

<!-- List all open PRs that this PR logically depends on. For example,
-->
<!-- - sagemath#12345: short description why this is a dependency -->
<!-- - sagemath#34567: ... -->
    
URL: sagemath#39467
Reported by: Tobias Diez
Reviewer(s): Dima Pasechnik, Kwankyu Lee, Tobias Diez, user202729
vbraun pushed a commit to vbraun/sage that referenced this pull request Mar 1, 2025
sagemathgh-39467: Disable broken and outdated CI
    
<!-- ^ Please provide a concise and informative title. -->
<!-- ^ Don't put issue numbers in the title, do this in the PR
description below. -->
<!-- ^ For example, instead of "Fixes sagemath#12345" use "Introduce new method
to calculate 1 + 2". -->
<!-- v Describe your changes below in detail. -->
<!-- v Why is this change required? What problem does it solve? -->
<!-- v If this PR resolves an open issue, please link to it here. For
example, "Fixes sagemath#12345". -->

Many of the CI runs after a new release are failing, for months now, see
eg https://github.com/sagemath/sage/actions/runs/12979684199/job/3621812
6145. Some of these failures are genuine (eg a certain package cannot be
built on a certain system) and some others are due to constraints of the
build system (eg running out of free space). Since there is very little
point in senselessly burning energy, all runs that were failing for the
last releases are disabled. Once the underlying issues are fixed, they
can be easily be re-enabled.

Moreover, the "minimal" runs where only a couple of systems packages are
installed and most are build using sage are removed, keeping only the
"standard" where all available system packages (corresponding to
standard sage packages) are installed.

New test run: https://github.com/tobiasdiez/sage/actions/runs/1319937223
2/job/36847711005

### 📝 Checklist

<!-- Put an `x` in all the boxes that apply. -->

- [ ] The title is concise and informative.
- [ ] The description explains in detail what this PR is about.
- [ ] I have linked a relevant issue or discussion.
- [ ] I have created tests covering the changes.
- [ ] I have updated the documentation and checked the documentation
preview.

### ⌛ Dependencies

- sagemath#39251 (to resolve merge conflicts)

<!-- List all open PRs that this PR logically depends on. For example,
-->
<!-- - sagemath#12345: short description why this is a dependency -->
<!-- - sagemath#34567: ... -->
    
URL: sagemath#39467
Reported by: Tobias Diez
Reviewer(s): Dima Pasechnik, Kwankyu Lee, Tobias Diez, user202729
vbraun pushed a commit to vbraun/sage that referenced this pull request Mar 2, 2025
sagemathgh-39467: Disable broken and outdated CI
    
<!-- ^ Please provide a concise and informative title. -->
<!-- ^ Don't put issue numbers in the title, do this in the PR
description below. -->
<!-- ^ For example, instead of "Fixes sagemath#12345" use "Introduce new method
to calculate 1 + 2". -->
<!-- v Describe your changes below in detail. -->
<!-- v Why is this change required? What problem does it solve? -->
<!-- v If this PR resolves an open issue, please link to it here. For
example, "Fixes sagemath#12345". -->

Many of the CI runs after a new release are failing, for months now, see
eg https://github.com/sagemath/sage/actions/runs/12979684199/job/3621812
6145. Some of these failures are genuine (eg a certain package cannot be
built on a certain system) and some others are due to constraints of the
build system (eg running out of free space). Since there is very little
point in senselessly burning energy, all runs that were failing for the
last releases are disabled. Once the underlying issues are fixed, they
can be easily be re-enabled.

Moreover, the "minimal" runs where only a couple of systems packages are
installed and most are build using sage are removed, keeping only the
"standard" where all available system packages (corresponding to
standard sage packages) are installed.

New test run: https://github.com/tobiasdiez/sage/actions/runs/1319937223
2/job/36847711005

### 📝 Checklist

<!-- Put an `x` in all the boxes that apply. -->

- [ ] The title is concise and informative.
- [ ] The description explains in detail what this PR is about.
- [ ] I have linked a relevant issue or discussion.
- [ ] I have created tests covering the changes.
- [ ] I have updated the documentation and checked the documentation
preview.

### ⌛ Dependencies

- sagemath#39251 (to resolve merge conflicts)

<!-- List all open PRs that this PR logically depends on. For example,
-->
<!-- - sagemath#12345: short description why this is a dependency -->
<!-- - sagemath#34567: ... -->
    
URL: sagemath#39467
Reported by: Tobias Diez
Reviewer(s): Dima Pasechnik, Kwankyu Lee, Tobias Diez, user202729
vbraun pushed a commit to vbraun/sage that referenced this pull request Mar 3, 2025
sagemathgh-39467: Disable broken and outdated CI
    
<!-- ^ Please provide a concise and informative title. -->
<!-- ^ Don't put issue numbers in the title, do this in the PR
description below. -->
<!-- ^ For example, instead of "Fixes sagemath#12345" use "Introduce new method
to calculate 1 + 2". -->
<!-- v Describe your changes below in detail. -->
<!-- v Why is this change required? What problem does it solve? -->
<!-- v If this PR resolves an open issue, please link to it here. For
example, "Fixes sagemath#12345". -->

Many of the CI runs after a new release are failing, for months now, see
eg https://github.com/sagemath/sage/actions/runs/12979684199/job/3621812
6145. Some of these failures are genuine (eg a certain package cannot be
built on a certain system) and some others are due to constraints of the
build system (eg running out of free space). Since there is very little
point in senselessly burning energy, all runs that were failing for the
last releases are disabled. Once the underlying issues are fixed, they
can be easily be re-enabled.

Moreover, the "minimal" runs where only a couple of systems packages are
installed and most are build using sage are removed, keeping only the
"standard" where all available system packages (corresponding to
standard sage packages) are installed.

New test run: https://github.com/tobiasdiez/sage/actions/runs/1319937223
2/job/36847711005

### 📝 Checklist

<!-- Put an `x` in all the boxes that apply. -->

- [ ] The title is concise and informative.
- [ ] The description explains in detail what this PR is about.
- [ ] I have linked a relevant issue or discussion.
- [ ] I have created tests covering the changes.
- [ ] I have updated the documentation and checked the documentation
preview.

### ⌛ Dependencies

- sagemath#39251 (to resolve merge conflicts)

<!-- List all open PRs that this PR logically depends on. For example,
-->
<!-- - sagemath#12345: short description why this is a dependency -->
<!-- - sagemath#34567: ... -->
    
URL: sagemath#39467
Reported by: Tobias Diez
Reviewer(s): Dima Pasechnik, Kwankyu Lee, Tobias Diez, user202729
@user202729 user202729 removed s: positive review disputed PR is waiting for community vote, see https://groups.google.com/g/sage-devel/c/IgBYUJl33SQ labels Mar 4, 2025
@user202729
Copy link
Contributor

Wait… this issue wasn't addressed yet.

retrofit-worktree is to perform "incremental build" from the previous stage.

The change here makes ""incremental build" broken, and thus also the whole point of the multi-stage docker builds.

Won't this make test-new take much longer than it would have? (evidently the test-new on this pull request takes 79 minutes instead of the normal 15ish)

@user202729
Copy link
Contributor

user202729 commented Mar 6, 2025

evidently the test-new on this pull request takes 79 minutes instead of the normal 15ish

@tobiasdiez can you take a quick look at this? I don't think you want this to miss 10.6.beta9.

@tobiasdiez
Copy link
Contributor Author

Wait… this issue wasn't addressed yet.

retrofit-worktree is to perform "incremental build" from the previous stage.
The change here makes ""incremental build" broken, and thus also the whole point of the multi-stage docker builds.

Won't this make test-new take much longer than it would have? (evidently the test-new on this pull request takes 79 minutes instead of the normal 15ish)

Thanks for bringing this up again. I've indeed missed this comment.

The reason why I changed it was that it was failing in the ci-linux as soon as git is installed in the image. For now, I've simply restored the retrofit script to be the same as on develop. Fixing it can be done in a follow-up PR. Moreover, #39641 makes test-new more reliable/faster.

Resetting it to positive review as the last commit was a trivial restore with the hope that this new version, and not the old one, makes it in the new release.

@tobiasdiez tobiasdiez added s: positive review disputed PR is waiting for community vote, see https://groups.google.com/g/sage-devel/c/IgBYUJl33SQ and removed s: needs review labels Mar 7, 2025
@user202729
Copy link
Contributor

user202729 commented Mar 7, 2025

Hope so. (also hopefully whatever process that is used to generate a new docker image every release that test-new is based on isn't broken, though I have no idea which part of the code deal with this.)

Wait a minute, then it isn't really fixed yet? But well since #39641 is there it's probably acceptable to leave it temporarily broken… (except it doesn't seem to quite work yet)

Or is it? Looks like it's fast again at least, I don't understand what

it was failing in the ci-linux as soon as git is installed in the image

mean here.

@tobiasdiez
Copy link
Contributor Author

Hope so. (also hopefully whatever process that is used to generate a new docker image every release that test-new is based on isn't broken, though I have no idea which part of the code deal with this.)

Wait a minute, then it isn't really fixed yet? But well since #39641 is there it's probably acceptable to leave it temporarily broken… (except it doesn't seem to quite work yet)

Or is it? Looks like it's fast again at least, I don't understand what

it was failing in the ci-linux as soon as git is installed in the image

mean here.

test-new is fixed now again with restoring the retrofit. But this may have broken some of the ci-linux runs (see eg https://github.com/tobiasdiez/sage/actions/runs/13179835737/job/36787605470#step:11:7244), but perhaps its only the maximum and not standard that has this error.

vbraun pushed a commit to vbraun/sage that referenced this pull request Mar 9, 2025
sagemathgh-39467: Disable broken and outdated CI
    
<!-- ^ Please provide a concise and informative title. -->
<!-- ^ Don't put issue numbers in the title, do this in the PR
description below. -->
<!-- ^ For example, instead of "Fixes sagemath#12345" use "Introduce new method
to calculate 1 + 2". -->
<!-- v Describe your changes below in detail. -->
<!-- v Why is this change required? What problem does it solve? -->
<!-- v If this PR resolves an open issue, please link to it here. For
example, "Fixes sagemath#12345". -->

Many of the CI runs after a new release are failing, for months now, see
eg https://github.com/sagemath/sage/actions/runs/12979684199/job/3621812
6145. Some of these failures are genuine (eg a certain package cannot be
built on a certain system) and some others are due to constraints of the
build system (eg running out of free space). Since there is very little
point in senselessly burning energy, all runs that were failing for the
last releases are disabled. Once the underlying issues are fixed, they
can be easily be re-enabled.

Moreover, the "minimal" runs where only a couple of systems packages are
installed and most are build using sage are removed, keeping only the
"standard" where all available system packages (corresponding to
standard sage packages) are installed.

New test run: https://github.com/tobiasdiez/sage/actions/runs/1319937223
2/job/36847711005

### 📝 Checklist

<!-- Put an `x` in all the boxes that apply. -->

- [ ] The title is concise and informative.
- [ ] The description explains in detail what this PR is about.
- [ ] I have linked a relevant issue or discussion.
- [ ] I have created tests covering the changes.
- [ ] I have updated the documentation and checked the documentation
preview.

### ⌛ Dependencies

- sagemath#39251 (to resolve merge conflicts)

<!-- List all open PRs that this PR logically depends on. For example,
-->
<!-- - sagemath#12345: short description why this is a dependency -->
<!-- - sagemath#34567: ... -->
    
URL: sagemath#39467
Reported by: Tobias Diez
Reviewer(s): Dima Pasechnik, Kwankyu Lee, Tobias Diez, user202729
@vbraun vbraun merged commit a12e957 into sagemath:develop Mar 9, 2025
18 of 22 checks passed
@tobiasdiez tobiasdiez deleted the disable-broken-ci branch March 9, 2025 18:00
@kwankyu kwankyu mentioned this pull request Mar 17, 2025
5 tasks
@user202729
Copy link
Contributor

user202729 commented Mar 21, 2025

hopefully whatever process that is used to generate a new docker image every release that test-new is based on isn't broken

test-new starts to take ≈ 75 minutes instead of ≈ 15 as it was before… (e.g. https://github.com/sagemath/sage/actions/runs/13957203177/job/39071095253 or https://github.com/sagemath/sage/actions/runs/13967188537/job/39100200709 ) I don't know why but this might be the cause. (Could also be unrelated cause)

At least it's temporary ( #39641 )

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
disputed PR is waiting for community vote, see https://groups.google.com/g/sage-devel/c/IgBYUJl33SQ
Projects
None yet
Development

Successfully merging this pull request may close these issues.

9 participants