shouldn't the solution be adding a custom double backward, rather than slowing down backward with autograd ops?

Sure , I ll try that and update once done.

@kshitij12345 this needs a rebase now

@weiyangfb sure will do that.

Wouldn't it still be better to have a fused index_add_ + mul for the backward than implementing a specific double backward? I'd think that it is probably a bit less code and more similar to what we do for other ops.

@ssnl I have tried and here is my opinion from what I understand

In the derivatives.yaml,

- name: embedding(Tensor weight, Tensor indices, int64_t padding_idx, bool scale_grad_by_freq, bool sparse)
  weight: embedding_backward(grad, indices, weight.size(0), padding_idx, scale_grad_by_freq, sparse)

- name: _embedding_bag(Tensor weight, Tensor indices, Tensor offsets, bool scale_grad_by_freq, int64_t mode, bool sparse)
  weight: _embedding_bag_backward(grad, indices, offsets, result1, result2, result3, weight.size(0), scale_grad_by_freq, mode, sparse)

We'll need to pass in weight as an argument to embedding_backward for the embedding_double_backward to update gradient on weight, but that will also require minor changes in embedding_bag_sparse_backward as it calls embedding_backward.

Also embedding_backward calls embedding_dense_backward and embedding_sparse_backward. Thus each will need its own version of double backward.

So I believe that as @t-vi suggested, we should opt for fused index_add_ + mul.
Please let me know what you think.
Thank You.

@t-vi @kshitij12345 I'm fine if you want to implement fused index_add_ + mul and also write a backward for that. But that would be considerably more work than just writing a custom double backward for this.

Oh , will take a look and update you on it.

I installed pytorch from two source codes:

Commit 6dcaa47 from your repo (kshitij12345/pytorch:master)
Commit ae1a972 from original pytorch:master (Which is the last commit your repo has from the pytorch:master)
So, there shouldn't be any problem with the single backward call for both. But I get the mentioned error from the former one while the latter one was fine.

It seems that your commit causes this error (btw, I'm not sure). I'll double check and will have a look and let you know if I found anything.

In that case can you please share a minimal code snippet that produces the given error, so even I can check.
Thank You.

I installed pytorch from two source codes:

Commit 6dcaa47 from your repo (kshitij12345/pytorch:master)
Commit ae1a972 from original pytorch:master (Which is the last commit your repo has from the pytorch:master)
So, there shouldn't be any problem with the single backward call for both. But I get the mentioned error from the former one while the latter one was fine.

It seems that your commit causes this error (btw, I'm not sure). I'll double check and will have a look and let you know if I found anything.

In that case can you please share a minimal code snippet that produces the given error, so even I can check.
Thank You.

I am trying to write a minimal code to reproduce the given error.
Meanwhile, you can simply reproduce the error by training a simple model with OpenNMT. If you train the very example in their README, you will get the same error (there is no need for double gradients there).
As far as I understand, the issue is with the (single) backward of the embedding table in both encoder and decoder.
https://github.com/OpenNMT/OpenNMT-py

I installed pytorch from two source codes:

Commit 6dcaa47 from your repo (kshitij12345/pytorch:master)
Commit ae1a972 from original pytorch:master (Which is the last commit your repo has from the pytorch:master)
So, there shouldn't be any problem with the single backward call for both. But I get the mentioned error from the former one while the latter one was fine.

It seems that your commit causes this error (btw, I'm not sure). I'll double check and will have a look and let you know if I found anything.

In that case can you please share a minimal code snippet that produces the given error, so even I can check.
Thank You.

I am trying to write a minimal code to reproduce the given error.
Meanwhile, you can simply reproduce the error by training a simple model with OpenNMT. If you train the very example in their README, you will get the same error (there is no need for double gradients there).
As far as I understand, the issue is with the (single) backward of the embedding table in both encoder and decoder.
https://github.com/OpenNMT/OpenNMT-py

Here is the minimal code to replicate the error. Sorry, but you need OpenNMT to import Embeddings.

import torch

from onmt.modules.embeddings import Embeddings

class Test(torch.nn.Module):

    def __init__(self):
        super(Test, self).__init__()
        self.dense = torch.nn.Linear(100, 1)
        self.oembd = Embeddings(word_vec_size=100,
                   position_encoding=False,
                   dropout=0.3,
                   word_padding_idx=1,
                   word_vocab_size=1000)

    def forward(self, inp):
        inp = self.oembd(inp)
        return self.dense(inp)

test = Test()
# test.cuda()
inp = torch.tensor([1, 1, 2, 1, 1])
inp=inp.unsqueeze(0).unsqueeze(-1)
out = test(inp)
raw_loss = out.mean(dim=1)

loss_grad = torch.autograd.grad(outputs=raw_loss,
                                inputs=list(test.parameters(recurse=False)),
                                retain_graph=True, create_graph=True, only_inputs=True)

If you run this code with your version of PyTorch (I mean commit 6dcaa47) you'll see the error while the original version (ae1a972) doesn't generate error.

@pooryapzm Indeed problem was in my part of code. Sorry. Have fixed it. Please let me know if it works for you as well. Have checked the following code:

import torch
torch.manual_seed(6)

from onmt.modules.embeddings import Embeddings

class Test(torch.nn.Module):

    def __init__(self):
        super(Test, self).__init__()
        self.dense = torch.nn.Linear(100, 1)
        self.oembd = Embeddings(word_vec_size=100,
                   position_encoding=False,
                   dropout=0.3,feat_merge=None,
                   word_padding_idx=1,
                   word_vocab_size=1000)

    def forward(self, inp):
        inp = self.oembd(inp)
        
        return self.dense(inp[0])

test = Test()
test.cuda()
inp = torch.tensor([1, 1, 2, 1, 1, 2],device='cuda')
inp=inp.unsqueeze(0).unsqueeze(-1)
out = test(inp)
raw_loss = out.mean(dim=1)
loss_grad = torch.autograd.grad(outputs=raw_loss,
                                inputs=list(test.parameters()),
                                retain_graph=True, create_graph=True, only_inputs=True)

norm = sum([param.norm()**2 for param in loss_grad])

loss = raw_loss + norm

loss.backward()
print("Succesful")

@kshitij12345 Awesome. It works with my code as well.
Thanks very much.

I'm afraid I have to own up to the fact that there the indexAdd approach isn't good after all.
There is a pretty strict drawback: dense embedding's backward is currently deteministic (on in particular on cuda) and it would become non-deterministic when switched to indexAdd.
As such, I have come to the conclusion that the best way to fix this is to implement a double backward explicitly.
@kshitij12345 I'm sorry about putting you on a wrong track here.

@t-vi Oh, no worries , I ll try the double backwards approach. If I have any doubts or get stuck will ask for your help. Thank You Again for guiding me.

However I am confused about what you meant in regards to non-deterministic. Cause if iyou mean the non-determinism in time, from the benchmark it is visible that the running time for embedding with index_add_ has less standard deviation, be it on CUDA or CPU. Just curious to understand and know more.

index_add_ does not produce deterministic results, as the order of addition is unspecified (see #12217 for a tiny bit more context).

@colesbury , please have a look.

That looks simple enough for me to be a bit embarrassed to have suggested the weighted index_add_ first.
You're the better expert for embeddings than me, but:

• Would having a second derivative also make sense for sparse?
• We are certain that embeddings are always vectors, right? If not it might make sense to pass the weight sizes to the double backward instead deriving the shape from the indices there. (But I don't know if that would be good or not.)

That looks simple enough for me to be a bit embarrassed to have suggested the weighted index_add_ first.
You're the better expert for embeddings than me, but:

* Would having a second derivative also make sense for sparse?

* We are certain that embeddings are always vectors, right? If not it might make sense to pass the weight sizes to the double backward instead deriving the shape from the indices there. (But I don't know if that would be good or not.)

Even I am embarrased, for my first attempt of contributing here is taking so long with all these mistakes (also with the fact that in excitement I started work in master branch and sent PR through master )

As for second point, in the little experience I have, embeddings have always been vectors in all the courses that I have taken and literature that I have seen. (Would be interesting to see if that is not always the case).

@colesbury @ssnl please review.

@colesbury , please review

@pytorchbot rebase this please

There's nothing to do! This branch is already up to date with master (1240327).

(To learn more about this bot, see Bot commands.)