Obtain hidden representations of Encoder model #2866
|
Hi, is it possible to obtain the hidden representations of an Encoder model? This was also asked in #606, but apparently unanswered. It seems that it is impossible to do so in speechbrain? I would be open to modifying the code for my personal needs, but having looked at the code I'm not sure how I could let it output the hidden representations as well. |
Replies: 1 comment 3 replies
|
Hi, it is possible. What do you want to extract and where exactly. Some code change may be needed, but it should be light. |
|
Hi, thank you for your comment! I am doing research on speaker verification models, and am training a probe on representations from different speaker verification models. Besides training a probe on the final speaker embedding from e.g. speechbrain/spkrec-ecapa-voxceleb, I would like to also train probes on the hidden representations of each layer of the ECAPA-TDNN model (these will of course be frame-based, but I can do pooling to get an utterance-level representation). |
|
You would have to modify the forward of this class: to return the things that you are interested in, as a list or something like that. |
|
Thank you for the encouragement, it was indeed quite doable. In case anyone else is looking to do the same thing, I made a public fork: https://github.com/607GitHub/speechbrain. I did not create a pull request for speechbrain, besides other reasons because I have only implemented this functionality for the three models that I'm working with. |
You would have to modify the forward of this class:
speechbrain/speechbrain/lobes/models/ECAPA_TDNN.py
Line 400 in 309130b