@dosubot, I added a test and it passes locally on my mac, however I've configured it to only run on mac (when mps is available because it is specific to mps) and it doesn't appear there are github actions configured to run the tests on macos. Is this right? If so, is it something I should change or is it in order as is?

Codecov Report

❌ Patch coverage is 60.00000% with 2 lines in your changes missing coverage. Please review.
✅ Project coverage is 56.77%. Comparing base (7eff461) to head (28d00f9).
⚠️ Report is 14 commits behind head on master.

@@           Coverage Diff           @@
##           master    #6246   +/-   ##
=======================================
  Coverage   56.77%   56.77%           
=======================================
  Files         889      889           
  Lines       84337    84340    +3     
=======================================
+ Hits        47880    47882    +2     
- Misses      36457    36458    +1     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Codecov Report

❌ Patch coverage is 0% with 5 lines in your changes missing coverage. Please review. ✅ Project coverage is 12.88%. Comparing base (88b4a9b) to head (9682614).

This is because the changes only apply to macos with apple silicon. As mentioned above, the coverage is 100% when the tests are run on a macos device.

The imports are unused but needed to load the necessary fixtures. Can I tell flake8 to allow this so the tests can pass or would it be preferred if I instead duplicated the fixture code to be directly in the file? I could also move the test into the file with the fixtures but this test seems somewhat separate and the guidelines said to prefer to split tests out into separate files

Hmm.
I don't know why this issue https://github.com/espnet/espnet/actions/runs/17913616362/job/50930566854?pr=6246 happens.
This test should be skipped in the espnet1 test according to https://github.com/espnet/espnet/blob/master/ci/test_python_espnet1.sh#L33 and other test files in the same directory.
@SanderGi, can you check this issue?

The other changes look good to me.

@dosubot, I added a test and it passes locally on my mac, however I've configured it to only run on mac (when mps is available because it is specific to mps) and it doesn't appear there are github actions configured to run the tests on macos. Is this right? If so, is it something I should change or is it in order as is?

@gemini-code-assist

Hello @Fhrozen, thanks for the mention! And thank you, @SanderGi, for raising this important point about the macOS tests.

I noticed in the repository's .github/workflows/ directory that there is indeed a ci_on_macos.yml workflow file. This suggests that macOS tests are configured within the GitHub Actions setup.

It would be beneficial to check this workflow to ensure it's correctly configured to trigger and execute the tests you've added, especially those that are specific to MPS and macOS. Sometimes, new test files or specific conditions might require updates to the CI configuration.

If the existing macOS workflow isn't picking up your new tests, or if there are specific environment variables or dependencies needed for MPS that aren't configured, that would be the next step to investigate. Let me know if you'd like me to take a closer look at the ci_on_macos.yml file in relation to your test setup.

@gemini-code-assist Could you take a closer look at the ci_on_macos.yml file in relation to the test setup?

Hello @Fhrozen, thanks for the follow-up! I can certainly take a closer look at what might be going on with the ci_on_macos.yml workflow. While I don't have direct access to its content, I can outline the key areas to check within that file, based on common CI setups for Python/PyTorch projects.

For test/espnet2/bin/test_macos_mps_model.py to run successfully on macOS CI, you'll want to verify the following in ci_on_macos.yml:

• Runner Environment: Ensure the job is actually configured to run on a macOS runner. This is usually specified with runs-on: macos-latest or a similar macOS runner image.

• Python Environment Setup: Confirm that the correct Python version is being set up, and that all necessary dependencies, especially PyTorch with MPS support, are installed. Look for steps like:

  - uses: actions/setup-python@v5
    with:
      python-version: '3.x' # Or whatever version is required
  - name: Install dependencies
    run: |
      pip install --upgrade pip
      pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/nightly/cpu  # Example, ensure MPS compatible version
      pip install -e '.[asr,tts]' # Or whatever your project's install command is

  Self-correction: For MPS, the PyTorch installation command is crucial. It often involves specific index URLs or extra-index-url for nightly builds or specific MPS wheels. The current PR description mentions mps device, so pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/nightly/cpu might not be sufficient; it should be something like pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/nightly/cpu (or https://download.pytorch.org/whl/nightly/cpu for CPU fallback, but for MPS, it's usually a specific URL or a version that includes MPS support by default).

• Test Execution Command: Check how pytest is invoked. Make sure it's configured to discover tests in the test/espnet2/bin/ directory. A common command would be:

  - name: Run tests
    run: |
      pytest test/espnet2/bin/test_macos_mps_model.py
      # Or more broadly:
      # pytest test/

  If pytest is run with specific arguments that exclude certain paths or markers, that could prevent your test from running.

• Conditional Logic: Look for any if conditions on the job or steps that might prevent them from running on macOS or for this specific pull request. For example, some workflows might only run on push to main and not on pull_request events.

If you can share the relevant sections of your ci_on_macos.yml file, I can provide more targeted suggestions!

Hmm. I don't know why this issue https://github.com/espnet/espnet/actions/runs/17913616362/job/50930566854?pr=6246 happens. This test should be skipped in the espnet1 test according to https://github.com/espnet/espnet/blob/master/ci/test_python_espnet1.sh#L33 and other test files in the same directory. @SanderGi, can you check this issue?

The other changes look good to me.

The cause is line 22 of test_python_espnet1.sh which runs flake8 on more than just espnet1 (including espnet2):

# flake8
"$(dirname $0)"/test_flake8.sh espnet

I have instructed flake8 to ignore the falsely flagged unused imports for now. Let me know if we want to tweak the test_python_espnet1 github action to only run flake8 on espnet1

@gemini-code-assist Could you take a closer look at the ci_on_macos.yml file in relation to the test setup?

The ci_on_macos.yml github action only tests installation on macos, it doesn't run the tests. The tests are only run on Ubuntu and Debian from what I can see

Thank you for the clarification, @SanderGi! That explains why the tests aren't being run on the macOS CI.

Given that the ci_on_macos.yml workflow is currently focused solely on installation, if the intention is to run the MPS-specific tests on macOS, then the workflow would indeed need to be extended. This would involve adding steps to execute the tests, similar to how the tests are configured to run on Ubuntu and Debian.

If you decide to proceed with modifying the ci_on_macos.yml to include test execution, please let me know if you'd like any further suggestions on how to integrate those steps.

Thanks, @SanderGi!
Now it passes all CI tests!