Codestin Search App

t-vi · 2019-04-16T21:50:25Z

Fixes: #19253

Fixing pow(Tensor, float) is straightforward.
The breakage for pow(float, Tensor) is a bit more subtle to trigger, and fixing needs torch.log (math.log didn't work) from the newly merged #19115 (Thanks @ngimel for pointing out this has landed.)

Fixes: pytorch#19253

t-vi · 2019-04-16T23:30:58Z

So one of the problems seems to be from Scalar<->float magic in symbolic_script, which seems to break when Scalar is an int. Unfortunately, the only thing I could think of is a gross hack adding torch._float to make a float from int/float IValues passed into a scalar.

eellison · 2019-04-17T00:01:54Z

                  exponent: float):
            def backward(grad_output):
-                grad_self = torch.where(torch.tensor(exponent == 0.0), torch.zeros_like(self), grad_output * exponent * torch.pow(self, exponent - 1))
+                if torch._float(exponent) == 0.0:


What's preventing float(exponent) instead?

So what I think is going on: When creating the pow_0 operator for Tensor self, Scalar exponent, symbolic_script replaces Scalar by float. The float(exponent) cast gets eliminated because the JIT "knows" it is a float. What then happens is that unpacking Scalar IValues that the JIT thinks must be float but that are, in fact, int fails.

In a way it comes down to

// 2. to make sure the input of any graph node does not contain scalar type // in its argument, all scalar arg should already be passed with float // value since scalar/int aren't differentiable either way.

not being the complete picture because (as here) the scalar might not be the thing we want to differentiate for, but a parameter (in the mathematical sense as opposed to the variable) of the function we want to differentiate.

So I'm changing the patch to do the following:
If Scalar -> float conversion happened, I change back the input type of the graph to Scalar, and insert a conversion (prim::Float) as the first thing.
It'll be troublesome for use when we get operations that actually rely on the difference between float and int in Scalar ops, but currently we don't as far as I know.

I think this is a clean-up of the Scalar->float conversion as it ensures that the graph inputs actually match the schema. @ailzhang does that seem reasonable?

So it turns out that re-Scalarizing breaks something (the double backward?). 😓

cc: @wanchaol added the scalar to float conversion.

Also, the scalar to float conversion only happens on the second pass, when allow_conversions is true, if there is an op defined for Scalar than that will be matched and there won't be a conversion.

@t-vi Actually as @eellison pointed out offline, we actually can get rid of the scalar -> float conversion entirely.
For example this works for me.

def pow_0(self, exponent: number):

Note that we don't expose number in torchscript but we CAN compile it! :D With this we can easily get rid of current symbolic_variable.h and c10::ReplaceAll(schema_str, "Scalar", "float");.
Huge thanks to @eellison who pointed this out!
Let us know if this fixes your problem :D

Ha. That works (I think)! And I've been poking around much too long. Awesome.
Thanks so much @ailzhang and @eellison !

So my understanding is that with this, one would only have definitions that match the operator schemas. Could we actually check that? For now I left a fallback in but converted those replacements that were hit by a test_jit.py run.

…ert to float

ailzhang · 2019-04-17T20:46:49Z


    auto sym_script_it = schema_to_graphs.find(schema_str);

+    if (sym_script_it == schema_to_graphs.end()) {


Yea this if should be dropped before merging.

So I removed it after double checking by means of calling sig(schema_string) and seeing whether it throws here:

pytorch/torch/csrc/jit/symbolic_script.cpp

Lines 1351 to 1353 in 3e0b46b

auto schema_string = overloadedSchemaString(actual_schema);

schema_to_graphs[schema_string] = std::move(pair);

ailzhang

LGTM! Let me know when you are done with changing and want to merge it.

t-vi · 2019-04-18T20:51:10Z

I think its good to merge. Am 18. April 2019 20:27:33 MESZ schrieb Ailing <[email protected]>:

…

ailzhang approved this pull request. LGTM! Let me know when you are done with changing and want to merge it. -- You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub: #19324 (review)

facebook-github-bot

@ailzhang has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.

facebook-github-bot · 2019-04-19T01:11:00Z

@ailzhang merged this pull request in b9291f5.

Summary: Fixes: pytorch#19253 Fixing pow(Tensor, float) is straightforward. The breakage for pow(float, Tensor) is a bit more subtle to trigger, and fixing needs `torch.log` (`math.log` didn't work) from the newly merged pytorch#19115 (Thanks ngimel for pointing out this has landed.) Pull Request resolved: pytorch#19324 Differential Revision: D15003531 Pulled By: ailzhang fbshipit-source-id: 8b22138fa27a43806b82886fb3a7b557bbb5a865

t-vi added 3 commits April 16, 2019 23:18

pow: Don't convert scalar exponent to CPU tensor in backward

0d9e069

Fixes: pytorch#19253

Merge branch 'master' into pow_scalar_exponent_autodiff

a4b28c3

take out prints etc

2a30ea8

facebook-github-bot added the oncall: jit Add this issue/PR to JIT oncall triage queue label Apr 16, 2019

t-vi added 2 commits April 17, 2019 00:27

python 2

9171ddd

fix IValue confusion from float<->Scalar when the Scalar is an integer

2fe4128

eellison reviewed Apr 16, 2019

View reviewed changes

Comment thread torch/csrc/jit/register_prim_ops.cpp Outdated

eellison reviewed Apr 17, 2019

View reviewed changes

for Scalar->float converted schemata, get back Scalar inputs and conv…

9a8663a

…ert to float

t-vi changed the title ~~pow scalar exponent / base autodiff, fusion~~ [WIP] pow scalar exponent / base autodiff, fusion Apr 17, 2019

use number in symbolic_script

dd1e18b

ailzhang reviewed Apr 17, 2019

View reviewed changes

remove fallback

d7c9577

ailzhang approved these changes Apr 18, 2019

View reviewed changes

ailzhang changed the title ~~[WIP] pow scalar exponent / base autodiff, fusion~~ pow scalar exponent / base autodiff, fusion Apr 18, 2019

facebook-github-bot reviewed Apr 18, 2019

View reviewed changes

facebook-github-bot closed this in b9291f5 Apr 19, 2019

facebook-github-bot added the merged label Apr 19, 2019

ezyang added the open source label Jun 24, 2019


		auto sym_script_it = schema_to_graphs.find(schema_str);

		if (sym_script_it == schema_to_graphs.end()) {

	auto schema_string = overloadedSchemaString(actual_schema);

	schema_to_graphs[schema_string] = std::move(pair);

Conversation

t-vi commented Apr 16, 2019

Uh oh!

t-vi commented Apr 16, 2019

Uh oh!

Uh oh!

eellison Apr 17, 2019

Choose a reason for hiding this comment

Uh oh!

t-vi Apr 17, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

t-vi Apr 17, 2019

Choose a reason for hiding this comment

Uh oh!

ailzhang Apr 17, 2019

Choose a reason for hiding this comment

Uh oh!

eellison Apr 17, 2019

Choose a reason for hiding this comment

Uh oh!

ailzhang Apr 17, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

t-vi Apr 17, 2019

Choose a reason for hiding this comment

Uh oh!

ailzhang Apr 17, 2019

Choose a reason for hiding this comment

Uh oh!

t-vi Apr 17, 2019

Choose a reason for hiding this comment

Uh oh!

ailzhang left a comment

Choose a reason for hiding this comment

Uh oh!

t-vi commented Apr 18, 2019 via email

Uh oh!

facebook-github-bot left a comment

Choose a reason for hiding this comment

Uh oh!

facebook-github-bot commented Apr 19, 2019

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

t-vi Apr 17, 2019 •

edited

Loading

ailzhang Apr 17, 2019 •

edited

Loading