Hi @alanakbik,
if I understand this PR correctly, this is to set a different LR for pretrained weights and randomly initialized weights on all DefaultClassifier's?

I suppose this could be extended even further to for example SequenceTagger, by filtering by embedding e.g:

embedding_parameters = [param for name, param in self.model.named_parameters() if "embedding" in name]
model_parameters = [param for name, param in self.model.named_parameters() if "embedding" not in name]

Such that we could train a Transformer-Bert model with a higher LT for the CRF part

Yes, it is for training the non-pretrained (i.e. randomly initialized) parts with a higher LR. Since the decoder is always randomly initialized, it is handled here. Extending this to the LSTM-CRF of the SequenceTagger would be great, but some embeddings (like CharacterEmbeddings) are randomly initialized, while others are not. So I think it's not easy to come up with a good heuristic to identify those parts.

Edit: I'll merge this now for experimentation, but any ideas to improve are welcome!