A cron job or something like then one you proposed would make the trick. But if a change is mandatory the maintainers need somehow to be notified early, no? But I suppose that is for internal use and to detect a potential impact of a change to the downstream libs.

• astroML

Frankly the downstream packages should be doing this integration testing by
testing various use cases and checking with scikit-learn master. We should
be encouraging downstream packages to include such integration tests (e.g.
various usages of their estimators with grid search).

Completely agree that it's our task downstream to test against sklearn master, personally it's on my list for a while (with a disturbingly old WIP PR), I just didn't get to the point to set up the build with the nightly build rather than pip install (which takes for ages).

I'd be happy to see, in the first instance, a script that runs the test suites of these downstream packages. We can look separately at how we best deploy that on CI

Frankly the downstream packages should be doing this integration testing by
testing various use cases and checking with scikit-learn master.

I think this is reasonable.

I just didn't get to the point to set up the build with the nightly build rather than pip install (which takes for ages).

@bsipocz This is something that we do in imbalanced-learn. The install is quick.

imbalanced-learn not necessarily as popular as others but with high use of sklearn's API

I would suggest the packages from the scikit-learn-contrib in general.

I have started a integration testing library to start addressing this issue: https://github.com/thomasjpfan/scikit-learn-integration-testing

Added to the 1.0 milestone so that we do not forget about it.

Makes sense. I am thinking of moving the CI to github actions so I can integrate it into github discussions or issues to ping the maintainers of the upstream projects when there is an test failure.

Added to the 1.0 milestone so that we do not forget about it.

Seems like we did forget about it @ogrisel 😁 . With some of the changes coming in the next release, and the one after, this would be really nice to have though.

Moving to 1.1 for now.

I think that we can address the following when we have a more granular check_estimator to be sure that we are testing the right things for these third-party libraries. xref: #16241

very unlikely to be part of 1.1, moving to 1.2

I think https://github.com/scientific-python/reverse-dependency-testing from @martinfleis is a good candidate to test downstream dependencies. What do you think?

@jjerphan it may work for a subset of packages out of the box. Packages organised like imbalanced-learn will work since they have tests included inside the package itself. But for example scikit-matter will not as it does not ship tests. For the latter, we shall find another way of getting the tests within the action but I haven't looked into that yet.

As a workaround, is there any way that the action could git clone the repo (assuming it's provided and is not a huge pile of hack to get the project links from PyPI)? Or we, as a community, should just strongly recommend including the test suite in the packaging, too?

As a workaround, is there any way that the action could git clone the repo (assuming it's provided and is not a huge pile of hack to get the project links from PyPI)?

That is not complicated, just not implemented at the moment.

Or we, as a community, should just strongly recommend including the test suite in the packaging, too?

I don't think so. @henryiii was advising against that (though I personally do see a certain value in that) while suggesting that tests should be in the sdist. We should then fetch the sdist from PyPI and run the tests. Though in practice, I am not sure how common that practice is. I bet that it is currently more likely that tests are shipped with the package than that they're in the sdist and not in the package.

Though in practice, I am not sure how common that practice is.

Some packages that have a top-level "tests" that is not in the wheel:

• pip
• build
• requests
• urllib3
• pytest
• wheel
• packaging

Also most (or all) of Scikit-HEP. Every package I'm involved with uses this structure.

I feel the ones listed above are more generic, and part of the basic infrastructure than the end-user libraries we try to tackle with downstream testing. E.g. occasionally, we found it rather useful to be able to ask the users to run astropy.test() to decipher what really goes wrong for them. Same for all (at least the ones I'm involved with) the packages in that sub-ecosystem.

Similarly, here for downstream testing it could be useful to be super easily just run the tests.

I'm not against <package>.test(), but that doesn't need to be your entire test suite - it could be a simple smoke check. A good full test suite, for example, should have extra dependencies (pytest, for example), often has some data (which should not be in every single wheel or in production systems!), and should run with all warnings as errors, which doesn't affect a user, but is critical for both you and package managers (which is what the full test suite is for). If you ship the whole test suite as .test(), then you'll tend to a) want it to work without test-only dependencies, b) provide data, and c) only fail if there's a real problem, not a deprecation warning or something similar. Also d) run quickly. Providing a "I'm working" smoke check solves all those issues without causing you to be tempted toward bad designs in your main test suite. You could even implement the smoke check in unittest.

Also, pytest's config doesn't ship in the wheel.

PS: Those "simple" packages are in the top 10 or so most downloaded packages or from the PyPA, by the way.

Oh, we fully agree on most of the things. Also, I didn't mean "simple" to say they are not important, those are indeed the most important infrastructure in the whole ecosystem. However, I feel this issue about downstream dependencies are targeting less well-engineered or maintained projects to see the effect of deprecation and changes introduced in core projects, which in practice means testing not those most relied-on libraries but those further down the chain, so the design choices of the core infrastructure libraries have little relevance.

We had some IRL discussions with @jeremiedbb regarding some expectations.

I am not sure that getting the latest release from PyPI is actually enough. I would expect to use the main branch to actually be able to report issues to the downstream software. The issue that we could detect might be potentially solved either on the downstream package or the parent package: if the issue is solved in the parent package then using the latest release "might" be enough; if the issue is in the downstream package then the changes will happen in main and using the latest release is not enough (or you will get a broken CI until the release of the downstream package).

If we want to get the main branch then the nightly builds could be a good candidate (until we can test it in someway). I don't see it feasible to rely on cloning a GitHub repository and build a downstream project. Each project might have a different build stack (pure Python package, Cython package, C++ package relying on CMake) that would make it super challenging.