• path [default=/Lib/${input.lang}]: A variable to filter the paths where the CI should run or not

Note this is more complicated to do correctly that just /Lib/${input.lang} - that wouldn't be useful even for PRs as we want to know if a PR to fix e.g. a Python-specific issue breaks any other target languages before merging it. So anything touching generic library, testsuite or source files needs to trigger a full set of jobs, but we certainly can save running jobs when the only files touched are target-language-specific such as Lib/python, Examples/python, Examples/test-suite/python and Source/Modules/python.cxx.

• ignore-path: ignore the arguments in path and run the workflow. This is to be used on master branch

If we've got accurate filtering of which jobs to run (which would be nice), I don't really see a good enough reason to avoid using it on master too. We're not going to have long gaps without a full set of jobs because anything which touches a generic files needs to trigger jobs for all languages.

Hooking up on.workflow_dispatch such that a full set of jobs can be manually triggered from the web UI might be handy sometimes - e.g. I can imagine wsfulton might want to be able to verify CI is definitely all green before making a release.

So anything touching generic library, testsuite or source files needs to trigger a full set of jobs, but we certainly can save running jobs when the only files touched are target-language-specific such as Lib/python, Examples/python, Examples/test-suite/python and Source/Modules/python.cxx.

That's precisely the design that I was planning to add. My plan is to use the design here and a json/yaml file configuration to define all the languages, variants and paths.

If we've got accurate filtering of which jobs to run (which would be nice), I don't really see a good enough reason to avoid using it on master too. We're not going to have long gaps without a full set of jobs because anything which touches a generic files needs to trigger jobs for all languages.

My reasoning for that is to have a workflow that could do a sanity check (e.g. there are changes in python-dev, or changes in the compiler, etc.). Pushes to master should be fine in terms of frequency, but this could be on a weekly schedule as well.

Hooking up on.workflow_dispatch such that a full set of jobs can be manually triggered from the web UI might be handy sometimes

👍

My reasoning for that is to have a workflow that could do a sanity check (e.g. there are changes in python-dev, or changes in the compiler, etc.). Pushes to master should be fine in terms of frequency, but this could be on a weekly schedule as well.

In the case of python-dev that comes from the Ubuntu package repos which means it'll only change for targetted fixes for security or other high severity bugs, or else when we move to a new Ubuntu release but for the latter case the CI config file changes (from saying e.g. ubuntu-20.04 to ubuntu-22.04) and that changing should trigger a full set of jobs anyway.

There are some dependencies that get installed from sources with more liberal update policies than Ubuntu, but I think the benefits of reducing the number of CI jobs for many pushes to master outweighs the potential downside of not seeing the effects of a breaking change to a target language dependency until we next happen to trigger a build for it (that's actually true already, it's just that such triggering would happen a bit less often with these changes).

By python-dev I was referring to python-3.qw-dev from setup-python github action. And similarly for the other languages, it should be designed to allow for running the latest development versions so that breaking changes can be caught early in the development stage instead of after release

@ojwb @erezgeva @wsfulton gentle poke to request some attention on this PR

Hi @LecrisUT ,

I like your work and I think the pull request is a good one.

From my side there are 2 blocking:

1. The job title on the web are too long and contain words like 'test' 'language' and 'run'. Theses worlds provide no information. The words appear on the start of the title make it harder to read.
   Please use the original titles!
   See my run: https://github.com/erezgeva/swig/actions/runs/6357334511

2. There are jobs that fails. They are probably new tests.
   I do not oppose adding extra tests for a "full" testing.
   But the default tests must pass!

1. The job title on the web are too long and contain words like 'test' 'language' and 'run'. Theses worlds provide no information. The words appear on the start of the title make it harder to read.
   **Please use the original titles!**
   See my run: https://github.com/erezgeva/swig/actions/runs/6357334511

Agree. After some experimentation I found where the naming comes from and how to simplify, but I cannot get rid of the folder-like structure though, i.e. the CI/.../.... So, how should we:

• distinguish between lang and toolchain workflows? I think the best would be with emoji like 🛠️ , but what can we use for lang? 🗨️ ?
• are the current versions still too long? Can you be more specific on how you would want them simplified?

2. There are jobs that fails. They are probably new tests.
   I do not oppose adding extra tests for a "full" testing.
   **But the default tests must pass!**

Pay attention to the CI/pass. There it shows if the normal tests fail or not. The additional tests are for experimental to highlight and keep track of how things are failing there and what needs to be done. What I can do is to mask the failing tests when pushed to main, that way the checks appear green in the commit history. I strongly recommend to keep them and show the failures in the PR.

1. The job title on the web are too long and contain words like 'test' 'language' and 'run'. Theses worlds provide no information. The words appear on the start of the title make it harder to read.
   **Please use the original titles!**
   See my run: https://github.com/erezgeva/swig/actions/runs/6357334511

Agree. After some experimentation I found where the naming comes from and how to simplify, but I cannot get rid of the folder-like structure though, i.e. the CI/.../.... So, how should we:

I like the folder like view.

• distinguish between lang and toolchain workflows? I think the best would be with emoji like 🛠️ , but what can we use for lang? 🗨️ ?

The folder make it easier to read for example:

TCL lang >
C++ 11
C++ 17

You can put the 'lang' and 'tool' after as most developers know TCL is a language and 'GCC' a compiler (tool)
i.e.
GCC tool
TCL lang

As for emoji, I like the idea. You're choosing is better than my ideas.

• are the current versions still too long? Can you be more specific on how you would want them simplified?

2. There are jobs that fails. They are probably new tests.
   I do not oppose adding extra tests for a "full" testing.
   **But the default tests must pass!**

Pay attention to the CI/pass. There it shows if the normal tests fail or not. The additional tests are for experimental to highlight and keep track of how things are failing there and what needs to be done. What I can do is to mask the failing tests when pushed to main, that way the checks appear green in the commit history. I strongly recommend to keep them and show the failures in the PR.

There is no difference.
If the PR fails, it will not be merged into 'main'.
Developers can not differ the 2 kinds.

What we can do is mark these failing tests with "continue-on-error: true" flag.
Once we add the features and publish experimental languages, we can revert the "continue-on-error" flag and make these tests mandatory.

Usually, as the tests take so long, the maintainers prefer not to run them and save the time.
Today there is only one test of that sort: "SWIGLANG: mzscheme"

If you improve the action: use artefacts and copy the tools from one job to another, than we can save time and add more tests :-)

You can put the 'lang' and 'tool' after as most developers know TCL is a language and 'GCC' a compiler (tool)

As for emoji, I like the idea. You're choosing is better than my ideas.

Thanks for the feedback, I'll try out some simplifications along those lines.

What we can do is mark these failing tests with "continue-on-error: true" flag.
Once we add the features and publish experimental languages, we can revert the "continue-on-error" flag and make these tests mandatory.

That is already what is done there (experimental == continue-on-error)

There is no difference.
If the PR fails, it will not be merged into 'main'.
Developers can not differ the 2 kinds.

Indeed, that is why I was trying to get the attention of the maintainers to discuss this change. Basically on my projects I set up a required workflow that points to Pass job so that the red jobs are ignored. Here are my notes on how to mask the results, and you can find it in action here (also in the PRs there).

I can disable the experimental jobs, but do you have a suggestion on where to track that these need to be re-enabled?

You can put the 'lang' and 'tool' after as most developers know TCL is a language and 'GCC' a compiler (tool)
As for emoji, I like the idea. You're choosing is better than my ideas.

Thanks for the feedback, I'll try out some simplifications along those lines.

What we can do is mark these failing tests with "continue-on-error: true" flag.
Once we add the features and publish experimental languages, we can revert the "continue-on-error" flag and make these tests mandatory.

That is already what is done there (experimental == continue-on-error)

There is no difference.
If the PR fails, it will not be merged into 'main'.
Developers can not differ the 2 kinds.

Indeed, that is why I was trying to get the attention of the maintainers to discuss this change. Basically on my projects I set up a required workflow that points to Pass job so that the red jobs are ignored. Here are my notes on how to mask the results, and you can find it in action here (also in the PRs there).

I can disable the experimental jobs, but do you have a suggestion on where to track that these need to be re-enabled?

Seems you look for a project decision.
I am merely a Developer happy to contribute and help.
You should talk to @ojwb about this issue.