Codestin Search App

Masao-Someki · 2025-11-17T13:15:50Z

What did you change?

Why did you make this change?

To implement ESPnet-3.
See #6133 for more details

Is your PR small enough?

No, but all changes are confirmed by reviewers, and this is a merge PR based on discussion with them.

Additional Context

for more information, see https://pre-commit.ci

…3/install_only

for more information, see https://pre-commit.ci

…3_upstream

for more information, see https://pre-commit.ci

[espnet-3] Merge ESPnet master branch into espnet3 branch

… into espnet3/install_only

Espnet3/install only

…anizer

- Due to the review comment for `multiple_iterator` flag, I recognized that the exception with the flag is not properly tested. I've added a new test case for this fix.

[espnet3-1] Add Data Organizer

for more information, see https://pre-commit.ci

…into espnet3/load_config

for more information, see https://pre-commit.ci

…into espnet3/load_config

…3_upstream

- I mistakenly removed important line when fixing the flake8..

for more information, see https://pre-commit.ci

sw005320

Can you remove the egs directory?
- check whether some scripts are used in egs2 by using a grep for symlink
- move them in espnet2
- config files must not be used, and you can safely remove them
Why did you change espnet.utils --> espnet2.legacy.utils?
- espnet.utils.cli_utils seems to be frequently used, and we can move this to espnet2.utils
ditto espnet.nets --> espnet2.legacy.nets
- to remove espnet1?
- espnet2.legacy.nets.pytorch_backend --> espnet2.legacy.nets (we don't need pytorch_backend

.github/workflows/ci_on_macos.yml

.github/workflows/ci_on_ubuntu.yml

sw005320 · 2025-11-21T12:13:34Z

.github/workflows/ci_on_ubuntu.yml

+      # - name: test espnet3 integration
+      #   run: ./ci/test_integration_espnet3.sh
+      # - uses: codecov/codecov-action@v2
+      #   with:
+      #     flags: test_integration_espnet3


Why is this disabled?

Sorry, currently we don't have them. I will activate this in recipe-related PR!

.github/workflows/ci_on_ubuntu.yml

espnet2/legacy/utils/cli_utils.py

espnet2/tts/fastspeech2/fastspeech2.py

sw005320 · 2025-11-21T12:52:11Z

test/espnet2/legacy/test_batch_beam_search.py

Why does this file have such a large change?

I refactored the beam search test to use espnet2 models from using espnet-1 model. Models are built via ASRTask.build_model from espnet2, and the encoder/LM setup follows the espnet2 APIs.

tools/setup.py

sw005320 · 2025-11-21T12:57:41Z

tools/setup_python.sh

@@ -42,4 +42,4 @@ export PATH=${PYTHON_DIR}:\${PATH}
 EOF

 . ./activate_python.sh
-python3 -m pip install -U pip wheel
+python3 -m pip install -U pip==25.2 wheel


Do we need this? pip==25.2

sw005320 · 2025-11-21T13:47:12Z

Can you remove the egs directory?

check whether some scripts are used in egs2 by using a grep for symlink

move them in espnet2

config files must not be used, and you can safely remove them

Why did you change espnet.utils --> espnet2.legacy.utils?

espnet.utils.cli_utils seems to be frequently used, and we can move this to espnet2.utils

ditto espnet.nets --> espnet2.legacy.nets

to remove espnet1?

espnet2.legacy.nets.pytorch_backend --> espnet2.legacy.nets (we don't need pytorch_backend

Some of them (e.g., remove egs) can be done later PR

Masao-Someki · 2025-12-10T23:22:18Z

I double-checked all removed espnet functions using a full static analysis pipeline.
It identifies every espnet2 -> espnet dependency, recursively resolves internal calls inside espnet, and verifies that all required symbols exist in espnet2/legacy.

Except for one known aliasing case (TransformerLoss), everything was fully consistent.
All CI tests pass as well, so we can be confident that the removed functions were truly unused and safe to delete.

All the tools and output resources are uploaded in the following gist:
https://gist.github.com/Masao-Someki/1955ee8930cd5dc63da5fcab31c68454

Masao-Someki · 2025-12-12T01:37:50Z

The pip version change (adding ==25.2) comes from a CI bug we hit when building warp-transducer in editable mode with pip 25.3. With pip 25.3, warp-transducer/pytorch_binding/setup.py is executed in a temporary environment where torch is not installed. Because setup.py imports torch at module import time, this results in a ModuleNotFoundError.

With pip 25.2, pip falls back to the legacy editable path (= setup.py develop) or otherwise runs setup.py without strict build isolation, so it sees the main environment where torch is already available and the build succeeds.

It seems the real reason is that the --no-build-isolation flag inside the installation script is not taking effect under pip 25.3. As a result, warp-transducer’s setup.py is executed inside pip’s isolated build environment, where torch is unavailable.

I am not sure why under the current master branch it works, and in espnet3 branch it doesn't. Pinning the pip version to 25.2 is the simplest workaround for now, but if needed I would investigate the exact behavior of pip 25.3 in more detail...

for more information, see https://pre-commit.ci

Fhrozen · 2025-12-14T01:24:49Z

@Masao-Someki, Please list the TODO items (or add TODO(Nelson)) that will need attention after merging this PR.
like:

Remove constraint at pip==25.2
or similar. I will test and fix these issues later.

Masao-Someki · 2025-12-14T07:23:08Z

@Fhrozen
Thank you, here are the todos.

Remove egs folder
directory structure update
- espnet2.legacy.utils -> espnet2.utils
- espnet2.legacy.nets.pytorch_backend -> espnet2.legacy.nets
Find a better workaround for pip version
Add integration test (mainly in the recipe PR)

sw005320 · 2025-12-15T12:37:45Z

Done!

Masao-Someki and others added 30 commits June 30, 2025 10:51

Removed chainer from test_utils

e906c7c

Removed chainer from check_install

a0a11dd

[pre-commit.ci] auto fixes from pre-commit.com hooks

7745f3d

for more information, see https://pre-commit.ci

Merge branch 'master' of https://github.com/espnet/espnet into espnet…

b26f1e9

…3/install_only

[pre-commit.ci] auto fixes from pre-commit.com hooks

b840b9f

for more information, see https://pre-commit.ci

Merge branch 'master' of https://github.com/espnet/espnet into espnet…

f4387c5

…3_upstream

[pre-commit.ci] auto fixes from pre-commit.com hooks

0e3ea64

for more information, see https://pre-commit.ci

Merge pull request #6192 from Masao-Someki/espnet3_upstream

f649813

[espnet-3] Merge ESPnet master branch into espnet3 branch

Merge branch 'espnet3' into espnet3/install_only

d62ba21

Fixed copilot's review

ed9098f

Merge branch 'espnet3/install_only' of github.com:Masao-Someki/espnet…

8e018bc

… into espnet3/install_only

Merge pull request #6139 from Masao-Someki/espnet3/install_only

e56773e

Espnet3/install only

Merge remote-tracking branch 'upstream/espnet3' into espnet3/data_org…

d314ef5

…anizer

Fixed format issue in workflow file

7090176

Fixed shell-check CI

7217501

Fixed copilot's review

f074ae3

- Due to the review comment for `multiple_iterator` flag, I recognized that the exception with the flag is not properly tested. I've added a new test case for this fix.

Merge pull request #6167 from Masao-Someki/espnet3/data_organizer

385e034

[espnet3-1] Add Data Organizer

Merge branch 'espnet3' into espnet3/load_config

86b8c98

Applied copilot's review

ac3834b

Worksround for shellcheck v0.11.0

e75eb60

[pre-commit.ci] auto fixes from pre-commit.com hooks

d1897f7

for more information, see https://pre-commit.ci

Fixed flake8

ec429ed

Merge branch 'espnet3/load_config' of github.com:Masao-Someki/espnet …

2f731a0

…into espnet3/load_config

[pre-commit.ci] auto fixes from pre-commit.com hooks

04a9c94

for more information, see https://pre-commit.ci

Fixed flake8

2747ed1

Applied black

41717fe

Merge branch 'espnet3/load_config' of github.com:Masao-Someki/espnet …

b9bec37

…into espnet3/load_config

Merge branch 'master' of https://github.com/espnet/espnet into espnet…

d36536f

…3_upstream

Fixed CI

128531a

- I mistakenly removed important line when fixing the flake8..

Pin setuptools-scm to <9.0 to avoid CI breakage from recent updates

dc5a35e

Masao-Someki and others added 10 commits November 17, 2025 19:02

Fixed CI

5dc1943

Merge branch 'espnet3' of github.com:espnet/espnet into espnet3

27c2566

[pre-commit.ci] auto fixes from pre-commit.com hooks

bd930b9

for more information, see https://pre-commit.ci

Fixed flake8 docstring

da52856

[pre-commit.ci] auto fixes from pre-commit.com hooks

237a9a8

for more information, see https://pre-commit.ci

Fixed CI

d23e2d6

Merge branch 'espnet3' of github.com:espnet/espnet into espnet3

beb5272

[pre-commit.ci] auto fixes from pre-commit.com hooks

6cdf95e

for more information, see https://pre-commit.ci

Added shinx markdown tables as the dependency for homepage.

fe36fc6

Merge branch 'espnet3' of github.com:espnet/espnet into espnet3

9477f48

Fhrozen requested a review from sw005320 November 21, 2025 10:26

sw005320 reviewed Nov 21, 2025

View reviewed changes

Fhrozen and others added 5 commits November 25, 2025 20:51

Merge branch 'master' into espnet3

5fc6bfe

Fixed CI tool versions

9f2061a

Fixed espnetez to espnet3

f8f9044

Fixed espnet -> espnet2 replacement

b5f2582

Changed dependency group names from task-<taskname> into <taskname>

f97af27

Masao-Someki and others added 2 commits December 11, 2025 22:11

Update espnet -> espnet2 or espnet3

035dd31

[pre-commit.ci] auto fixes from pre-commit.com hooks

09b6278

for more information, see https://pre-commit.ci

Fhrozen requested a review from sw005320 December 14, 2025 07:24

Merge branch 'master' into espnet3

6fa81d7

sw005320 merged commit c61e211 into master Dec 15, 2025
66 of 74 checks passed

sw005320 deleted the espnet3 branch December 15, 2025 12:37

Fhrozen added this to the v.202601 milestone Jan 5, 2026

Conversation

Masao-Someki commented Nov 17, 2025

What did you change?

Why did you make this change?

Is your PR small enough?

Additional Context

Uh oh!

sw005320 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

sw005320 Nov 21, 2025

Choose a reason for hiding this comment

Uh oh!

Masao-Someki Nov 21, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

sw005320 Nov 21, 2025

Choose a reason for hiding this comment

Uh oh!

Masao-Someki Dec 12, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

sw005320 Nov 21, 2025

Choose a reason for hiding this comment

Uh oh!

sw005320 commented Nov 21, 2025

Uh oh!

Masao-Someki commented Dec 10, 2025

Uh oh!

Masao-Someki commented Dec 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Fhrozen commented Dec 14, 2025

Uh oh!

Masao-Someki commented Dec 14, 2025

Uh oh!

Uh oh!

sw005320 commented Dec 15, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Masao-Someki commented Dec 12, 2025 •

edited

Loading