Codestin Search App

IvanYashchuk · 2021-07-25T09:29:13Z

This PR adds forward mode differentiation for torch.linalg.cholesky, torch.linalg.cholesky_ex, and transpose functions.
Complex tests for Cholesky fail because for some reason the gradcheck sends matrices full of zeros to cholesky_jvp function.

cc @ezyang @albanD @zou3519 @gqchen @pearu @nikitaved @soulitzer @lezcano @Varal7 @jianyuh @mruberry @heitorschueroff @walterddr @IvanYashchuk @xwang233

facebook-github-bot · 2021-07-25T09:29:20Z

🔗 Helpful links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/62159

💊 CI failures summary and remediations

As of commit 14dcdec (more details on the Dr. CI page):

💚 💚 Looks good so far! There are no failures yet. 💚 💚

This comment was automatically generated by Dr. CI (expand for details).

Follow this link to opt-out of these comments for your Pull Requests.

Please report bugs/suggestions to the (internal) Dr. CI Users group.

Click here to manually regenerate this comment.

lezcano

Just left a comment. The rest of the code LGTM. Nice use of triangular_solve to compute the inverse :)

lezcano · 2021-07-26T14:38:39Z

-           decorators=[skipCUDAIfNoMagmaAndNoCusolver, skipCUDAIfRocm, skipCPUIfNoLapack]),
+           decorators=[skipCUDAIfNoMagmaAndNoCusolver, skipCUDAIfRocm, skipCPUIfNoLapack],
+           skips=(
+               # Gradcheck for complex generates invalid inputs for this function


Might this be because it does not create Hermitian inputs? It's not clear whether we can assume that the inputs are Hermitian or not. See #62163 (comment)

Ho yes, I encountered that one but forgot to fix it.
We should update

pytorch/torch/autograd/gradcheck.py

Line 925 in b73d759

return apply_to_c_inps(fn, lambda x: x + 0 * 1j), apply_to_c_inps(fn, lambda x: x * 1j)

to make it keep the original imag/real respectively instead of setting them to 0s.

albanD

LGTM!
Thanks

albanD · 2021-07-26T17:57:21Z

-           decorators=[skipCUDAIfNoMagmaAndNoCusolver, skipCUDAIfRocm, skipCPUIfNoLapack]),
+           decorators=[skipCUDAIfNoMagmaAndNoCusolver, skipCUDAIfRocm, skipCPUIfNoLapack],
+           skips=(
+               # Gradcheck for complex generates invalid inputs for this function


Ho yes, I encountered that one but forgot to fix it.
We should update

pytorch/torch/autograd/gradcheck.py

Line 925 in b73d759

return apply_to_c_inps(fn, lambda x: x + 0 * 1j), apply_to_c_inps(fn, lambda x: x * 1j)

to make it keep the original imag/real respectively instead of setting them to 0s.

albanD · 2021-07-26T17:58:44Z


 - name: transpose.int(Tensor(a) self, int dim0, int dim1) -> Tensor(a)
  self: grad.transpose(dim0, dim1)
+  result: auto_linear


note that this will conflict with #59993
Not a big deal as the rebase will be simple, but we should not land them at the same time.

pytorch-probot · 2021-09-06T10:29:48Z

CI Flow Status

⚛️ CI Flow

Ruleset - Version: v1
Ruleset - File: https://github.com/IvanYashchuk/pytorch/blob/14dcdecbd232848798202d46c1e5e2f42c526496/.github/generated-ciflow-ruleset.json
PR ciflow labels: ciflow/default

Workflows	Labels (bold enabled)	Status
Triggered Workflows
linux-bionic-py3.6-clang9	`ciflow/all`, `ciflow/cpu`, `ciflow/default`, `ciflow/linux`, `ciflow/noarch`, `ciflow/xla`	✅ triggered
linux-bionic-py3.8-gcc9-coverage	`ciflow/all`, `ciflow/coverage`, `ciflow/cpu`, `ciflow/default`, `ciflow/linux`	✅ triggered
linux-xenial-cuda11.3-py3.6-gcc7	`ciflow/all`, `ciflow/cuda`, `ciflow/default`, `ciflow/linux`	✅ triggered
linux-xenial-py3.6-gcc5.4	`ciflow/all`, `ciflow/cpu`, `ciflow/default`, `ciflow/linux`	✅ triggered
linux-xenial-py3.6-gcc7-bazel-test	`ciflow/all`, `ciflow/bazel`, `ciflow/cpu`, `ciflow/default`, `ciflow/linux`	✅ triggered
win-vs2019-cpu-py3	`ciflow/all`, `ciflow/cpu`, `ciflow/default`, `ciflow/win`	✅ triggered
win-vs2019-cuda10.1-py3	`ciflow/all`, `ciflow/cuda`, `ciflow/default`, `ciflow/win`	✅ triggered
Skipped Workflows
libtorch-linux-xenial-cuda10.2-py3.6-gcc7	`ciflow/all`, `ciflow/cuda`, `ciflow/libtorch`, `ciflow/linux`	🚫 skipped
libtorch-linux-xenial-cuda11.3-py3.6-gcc7	`ciflow/all`, `ciflow/cuda`, `ciflow/libtorch`, `ciflow/linux`	🚫 skipped
linux-bionic-cuda10.2-py3.9-gcc7	`ciflow/all`, `ciflow/cuda`, `ciflow/linux`, `ciflow/slow`	🚫 skipped
linux-xenial-cuda10.2-py3.6-gcc7	`ciflow/all`, `ciflow/cuda`, `ciflow/linux`, `ciflow/slow`	🚫 skipped
periodic-libtorch-linux-xenial-cuda11.1-py3.6-gcc7	`ciflow/all`, `ciflow/cuda`, `ciflow/libtorch`, `ciflow/linux`, `ciflow/scheduled`	🚫 skipped
periodic-linux-xenial-cuda11.1-py3.6-gcc7	`ciflow/all`, `ciflow/cuda`, `ciflow/linux`, `ciflow/scheduled`	🚫 skipped
periodic-win-vs2019-cuda11.1-py3	`ciflow/all`, `ciflow/cuda`, `ciflow/scheduled`, `ciflow/win`	🚫 skipped
win-vs2019-cuda11.3-py3	`ciflow/all`, `ciflow/cuda`, `ciflow/win`	🚫 skipped

You can add a comment to the PR and tag @pytorchbot with the following commands:

# ciflow rerun, "ciflow/default" will always be added automatically
@pytorchbot ciflow rerun

# ciflow rerun with additional labels "-l <ciflow/label_name>", which is equivalent to adding these labels manually and trigger the rerun
@pytorchbot ciflow rerun -l ciflow/scheduled -l ciflow/slow

For more information, please take a look at the CI Flow Wiki.

codecov · 2021-09-06T13:52:50Z

Codecov Report

Merging #62159 (14dcdec) into master (544c8e6) will increase coverage by 0.08%.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##           master   #62159      +/-   ##
==========================================
+ Coverage   66.63%   66.72%   +0.08%     
==========================================
  Files         707      707              
  Lines       92338    92338              
==========================================
+ Hits        61534    61612      +78     
+ Misses      30804    30726      -78

IvanYashchuk · 2021-09-06T19:23:24Z

torch.linalg.cholesky now has the upper argument. I updated the code and OpInfo's sample inputs to test that.
@albanD, could you please take another look and merge if it's good to go?

albanD

LGTM

facebook-github-bot · 2021-09-07T14:48:29Z

@albanD has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.

lezcano · 2021-09-08T15:22:29Z

A small note: We should have a look at the performance of solve_triangular once #63568 is merged (or even solve_triangular_out). If it's not much worse than matmul, we should do two solves rather than inverse + 2 matmuls.

facebook-github-bot · 2021-09-08T16:45:57Z

@albanD merged this pull request in dd8f6ac.

…ose (pytorch#62159) Summary: This PR adds forward mode differentiation for `torch.linalg.cholesky`, `torch.linalg.cholesky_ex`, and `transpose` functions. Complex tests for Cholesky fail because for some reason the gradcheck sends matrices full of zeros to `cholesky_jvp` function. cc ezyang albanD zou3519 gqchen pearu nikitaved soulitzer Lezcano Varal7 jianyuh mruberry heitorschueroff walterddr IvanYashchuk xwang233 Pull Request resolved: pytorch#62159 Reviewed By: mrshenli Differential Revision: D30776829 Pulled By: albanD fbshipit-source-id: 32e5539ed6423eed8c18cce16271330ab0ea8d5e

IvanYashchuk added module: autograd Related to torch.autograd, and the autograd engine in general module: linear algebra Issues related to specialized linear algebra operations in PyTorch; includes matrix multiply matmul labels Jul 25, 2021

IvanYashchuk requested a review from albanD July 25, 2021 09:29

IvanYashchuk requested a review from soulitzer as a code owner July 25, 2021 09:29

facebook-github-bot added the cla signed label Jul 25, 2021

Add cholesky forward mode differentiation

72b08ba

IvanYashchuk force-pushed the cholesky-jvp branch from b1b9ef6 to 72b08ba Compare July 25, 2021 09:31

pytorchbot added the open source label Jul 25, 2021

Use const Tensor&

4344fa5

jbschlosser added the triaged This issue has been looked at a team member, and triaged and prioritized into an appropriate module label Jul 26, 2021

lezcano approved these changes Jul 26, 2021

View reviewed changes

albanD reviewed Jul 26, 2021

View reviewed changes

Merge remote-tracking branch 'upstream/master' into cholesky-jvp

9478095

IvanYashchuk mentioned this pull request Sep 6, 2021

Forward mode AD for linear algebra functions #64545

Closed

14 tasks

IvanYashchuk added 2 commits September 6, 2021 04:18

Merge remote-tracking branch 'upstream/master' into cholesky-jvp

0e17d8b

Add upper kwarg support to forward ad rule

14dcdec

pytorch-probot Bot added the ciflow/default label Sep 6, 2021

pytorch-probot Bot assigned pytorchbot and unassigned pytorchbot Sep 6, 2021

albanD approved these changes Sep 7, 2021

View reviewed changes

facebook-github-bot closed this in dd8f6ac Sep 8, 2021

facebook-github-bot added the Merged label Sep 8, 2021

Conversation

IvanYashchuk commented Jul 25, 2021 • edited by pytorch-probot Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

facebook-github-bot commented Jul 25, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔗 Helpful links

💊 CI failures summary and remediations

Uh oh!

lezcano left a comment

Choose a reason for hiding this comment

Uh oh!

lezcano Jul 26, 2021

Choose a reason for hiding this comment

Uh oh!

albanD Jul 26, 2021

Choose a reason for hiding this comment

Uh oh!

albanD left a comment

Choose a reason for hiding this comment

Uh oh!

albanD Jul 26, 2021

Choose a reason for hiding this comment

Uh oh!

albanD Jul 26, 2021

Choose a reason for hiding this comment

Uh oh!

pytorch-probot Bot commented Sep 6, 2021

⚛️ CI Flow

Uh oh!

codecov Bot commented Sep 6, 2021

Codecov Report

Uh oh!

IvanYashchuk commented Sep 6, 2021

Uh oh!

albanD left a comment

Choose a reason for hiding this comment

Uh oh!

facebook-github-bot commented Sep 7, 2021

Uh oh!

lezcano commented Sep 8, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

facebook-github-bot commented Sep 8, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

IvanYashchuk commented Jul 25, 2021 •

edited by pytorch-probot Bot

Loading

facebook-github-bot commented Jul 25, 2021 •

edited

Loading

lezcano commented Sep 8, 2021 •

edited

Loading