Could you please post the full pytest output?

The CLI is still very much a mess with lots of global state, and the tests which have been written in the past did a lot of monkey-patching while trying to reset state afterwards, but none of them ever did this properly.

I've recently rewritten the CLI's logging/console-print setup tests with that in mind. However, I can't reproduce the log level issue. The tests should always run in the same order, determined by this:

• https://github.com/streamlink/streamlink/blob/6.7.0/tests/conftest.py#L16-L26
• https://github.com/streamlink/streamlink/blob/6.7.0/tests/conftest.py#L39-L52
• https://docs.pytest.org/en/7.1.x/reference/reference.html#pytest.hookspec.pytest_collection_modifyitems

Also, this diff should be better, as it doesn't rely on a static value (even though it's the same value as set in streamlink.logger):

diff --git a/tests/cli/test_main_logging.py b/tests/cli/test_main_logging.py
index 3416fd69..f7024787 100644
--- a/tests/cli/test_main_logging.py
+++ b/tests/cli/test_main_logging.py
@@ -33,10 +33,13 @@ def _setup(monkeypatch: pytest.MonkeyPatch, session: Streamlink):
     monkeypatch.setattr("streamlink_cli.main.setup_signals", Mock())
     monkeypatch.setattr("streamlink_cli.argparser.find_default_player", Mock())
 
+    level = streamlink_cli.main.logger.root.level
+
     try:
         yield
     finally:
         streamlink_cli.main.logger.root.handlers.clear()
+        streamlink_cli.main.logger.root.setLevel(level)
         streamlink_cli.main.args = None  # type: ignore[assignment]
         streamlink_cli.main.console = None  # type: ignore[assignment]

Here are the results:
tests_result.txt

The folder in /workspaces/build-area/streamlink-6.7.0/.pybuild/cpython3_3.11/build is created by Debian's packaging scripts, but tests/conftest.py is there.

The uploaded file is 404... Please post a GH gist instead.

Ok, uploaded here: https://gist.github.com/amurzeau/a787142847cdd962c80b96492ae2026e

I think I've found the issue, this is because of item.nodeid which in Debian setup is something like this: .pybuild/cpython3_3.11/build/tests/webbrowser/test_webbrowser.py::TestLaunch::test_terminate_on_nursery_cancellation

So nothing will match _TEST_PRIORITIES items.

I get no error with python3.11 -m pytest --rootdir . tests (with rootdir)
I will check if there is something to do about that on Debian packaging side.

I think I've found the issue, this is because of item.nodeid which in Debian setup is something like this:
I will check if there is something to do about that on Debian packaging side.

https://docs.pytest.org/en/latest/reference/customize.html#initialization-determining-rootdir-and-configfile

There's nothing that can be done here to fix this. We can't set the rootdir option in pyproject.toml. It's something that must be fixed in the Debian testing scripts. You'll probably need to set the CWD to what the rootdir was previously pointing to and then run pytest from there without the rootdir option.

Yes, this is related with how Debian handle this.
It does this:

• Build/Copy files to be included in the Debian package in a temporary directory: <streamink_root>/.pybuild/cpython3_3.11/build/
• Run pytest from there, I guess to ensure there is no missing stuff from the package

I can fix the issue by adding this to the pytest command line: --rootdir=. -o cache_dir=$(CURDIR)/.pytest_cache
It's kind of hacky, but given:

• tests nodeid are relative to the rootdir
• a temporary .pytest_cache is created in the rootdir
  if I want to keep most the of Debian packaging scripts (ie. keep the .pybuild temporary build directory), I guess I need to do that.

An alternative would be to run tests directly on the root streamlink source directory instead, but I'm not sure this is really better from a Debian package perspective.

I'm closing this PR given the issue is caused by packaging scripts, let me know if you me to still reopen it.

The logger level should still be reset, so I'd appreciate if you could reopen and update the PR with the diff I have posted. Thanks.

Thanks!