@jctian98 @Jungjee @wyh2000 Hi guys, may I request your review for this PR? Since it has a few framework-level design choices, it would be difficult to change in the later stage.

Thanks @ftshijt for your cool work! LGTM in general.

Two comments from me (maybe close to questions):

1. Having a gan_codec tasks sounds like there would be other *_codec tasks upcoming. How many tasks do you expect? Is is GAN vs. GAN-irrelevant?
2. Could we have a folder where we could see all sorts of modules used for the codec tasks? (gan_codec and others) I initially suggested espnet2/codec/layer but after reviewing everything maybe we could put all fundamental modules (that could be re-usable) to espnet2/codec/shared? The idea comes from my assumption that some modules could be used across different *_codec tasks. But maybe this is not the optimal selection.

For 1, I'm following the setting to tts and svs. Using the name has the intention to state the task is trained with gan_trainer instead of the existing normal trainer used in many other tasks. I think we will only support the gan_codec task for codec training for now.

For 2, yeah, I tried to put some shared modules to shared folder.

Some comments after discussion with @ftshijt Please let me know if I'm wrong.
(1) Since DAC / soundstream / encoded share the very similar structure like SEANet, quantizer, and overall encoder-decoder-discriminator design, it might be better to have a unified file (current soundstream.py) for them three rather than have the espnet2/gan_codec/{soundstream, encoded, DAC} folders. Instead, we can use if/else to specify the distinctions among these models.
(2) we can also make the loss classified and more configurable. Specifically, the losses can be categorized (my current thinking) adversarial loss / reconstruction loss / loss that specify the similarity between generator decoder and discriminator.

Some comments after discussion with @ftshijt Please let me know if I'm wrong. (1) Since DAC / soundstream / encoded share the very similar structure like SEANet, quantizer, and overall encoder-decoder-discriminator design, it might be better to have a unified file (current soundstream.py) for them three rather than have the espnet2/gan_codec/{soundstream, encoded, DAC} folders. Instead, we can use if/else to specify the distinctions among these models. (2) we can also make the loss classified and more configurable. Specifically, the losses can be categorized (my current thinking) adversarial loss / reconstruction loss / loss that specify the similarity between generator decoder and discriminator.

My concern is that for DAC, it actually modifies some loss and quantizer details from Soundstream. It might be clearer for DAC to have a separate dac.py file.

Consider the following PR (decoding/evaluation), I will merge the current one first. Let's keep discussing better code organization.