Adding MPS support for 3D convolutions #99246

mattiaspaul · 2023-04-15T21:34:57Z

Fixes #77818

this pull request enables 3D convolutions (forward/backward) for MPS (Apple Silicon) within the same Convolution.mm file as conv2d.
does not support channel_last (since pytorch doesn't implement channel_last for 3D tensors)
does not support conv3d_transpose and treats depth-separable convolutions not as normal case (there are no MPS kernels available for either of those so far)
requires MacOS >=13.2 (Ventura), I'm not sure how to add this specific case to MPSGraphVenturaOps.h
@kulinseth @albanD could you please check whether this is implemented as intended by you and whether we would require additional tests in test_mps.py?

pytorch-bot · 2023-04-15T21:34:59Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/99246

📄 Preview Python docs built from this PR
📄 Preview C++ docs built from this PR
❓ Need help or want to give feedback on the CI? Visit the bot commands wiki or our office hours

Note: Links to docs will display an error until the docs builds have been completed.

❌ 2 New Failures

As of commit 5ec4aa5:

NEW FAILURES - The following jobs have failed:

This comment was automatically generated by Dr. CI and updates every 15 minutes.

linux-foundation-easycla · 2023-04-15T21:35:00Z

The committers listed above are authorized under a signed CLA.

✅ login: mattiaspaul (06362f1, 8b8284e, 5ec4aa5)

mattiaspaul · 2023-04-21T18:16:02Z

@kulinseth gentle reminder to respond to the review request or to help me re-assign it to someone else

Berzeg · 2023-04-22T10:28:03Z

aten/src/ATen/native/mps/operations/Convolution.mm

+                                                                descriptor:conv2dDescriptor_
+                                                                      name:nil];
+                }
+            }


In line 319 below you want to update the biasPlaceholder to support 3d convolutions. I think something like this should work:

if (is3DConv) { biasPlaceholder = native_mps::Placeholder(cachedGraph->biasTensor_, (bias_opt.value()).view({1, bias_shape[0], 1, 1, 1})); } else { biasPlaceholder = native_mps::Placeholder(cachedGraph->biasTensor_, (bias_opt.value()).view({1, bias_shape[0], 1, 1})); }

@Berzeg Thanks a lot for your feedback. Yes you are absolutely right, the bias shape was wrong for 3D and is now fixed

Berzeg · 2023-04-22T10:33:29Z

aten/src/ATen/native/mps/operations/Convolution.mm

+                                                                  name:nil];
+
+            } else {
+                if (bias_defined) {


This conditional statement and its block, which contains the biasTensor assignment should be moved outside the conditional block that contains it. Otherwise, biasTensor will always be nil for 3d convolutions.

You can move it right below where you set MPSGraphTensor* biasTensor = nil;

Thanks! I moved the block accordingly so that it is called regardless of isConv3d or isDepthwiseConv. I amended my tests to include biases and it now works as expected. I created a set of tests at https://gist.github.com/mattiaspaul/b63cd65c9afa4290b316d9297e19ca03 (maybe some could be added to the official test_mps.py at some point)

mattiaspaul · 2023-04-25T20:12:05Z

needed to merge the intermediate changes to Convolution.mm and can hopefully reopen the corrected pull request

kulinseth · 2023-04-22T04:05:53Z

aten/src/ATen/native/mps/MPSGraphVenturaOps.h

+#if !defined(__MAC_13_0) && \
+    (!defined(MAC_OS_X_VERSION_13_0) || (MAC_OS_X_VERSION_MIN_REQUIRED < MAC_OS_X_VERSION_13_0))
+
+@compatibility_alias MPSGraphConvolution3DOpDescriptor unsupported_MPSGraphConvolution3DOpDescriptor;


Can you move this down to Line 41

@mattiaspaul , can you please move this down to Line 41

I tried moving it, but if it appears under the MPSGraph interface the build fails. I found the suggestion for compatibility_alias at nshipster to solve the challenge that the build is done on MacOS12 where the Descriptor is undefined and runs on MacOS13.2 where it becomes available, but am happy to learn about cleaner alternatives.

kulinseth · 2023-04-22T04:06:33Z

aten/src/ATen/native/mps/operations/Convolution.mm

+#if !defined(__MAC_13_0) && \
+    (!defined(MAC_OS_X_VERSION_13_0) || (MAC_OS_X_VERSION_MIN_REQUIRED < MAC_OS_X_VERSION_13_0))
+
+@implementation MPSGraphConvolution3DOpDescriptor


I think this can be included in the MPSGraphVenturaOps.h file.

Thanks for the suggestion I tried this but got "duplicate symbol" errors from Upsample.mm and Unique.mm for MPSGraphConvolution3DOpDescriptor so left it in the .mm file for now

kulinseth · 2023-04-26T06:48:52Z

aten/src/ATen/native/mps/MPSGraphVenturaOps.h

+                                           forwardConvolutionDescriptor:(MPSGraphConvolution3DOpDescriptor * _Nonnull) forwardConvolutionDescriptor
+                                                                   name:(NSString * _Nullable) name;
+
+- (MPSGraphTensor * _Nonnull) convolution3DWeightsGradientWithIncomingGradientTensor:(MPSGraphTensor * _Nonnull) incomingGradient


These need to be defined outside the macro.

Thanks a lot @kulinseth this was exactly the problem that caused the build failure. It is now fixed and passes the basic tests.

kulinseth · 2023-04-26T06:52:06Z

@mattiaspaul , thanks for the PR. I provided few comments. There is also a build failure:

2023-04-25T21:06:21.0540990Z /Users/ec2-user/runner/_work/pytorch/pytorch/aten/src/ATen/native/mps/operations/Convolution.mm:576:66: error: use of undeclared identifier 'MPSGraphConvolution3DOpDescriptor'
2023-04-25T21:06:21.0542000Z         MPSGraphConvolution3DOpDescriptor* conv3dDescriptor_ = [[MPSGraphConvolution3DOpDescriptor new] autorelease];
2023-04-25T21:06:21.0542640Z                                                                  ^
2023-04-25T21:06:21.0544640Z /Users/ec2-user/runner/_work/pytorch/pytorch/aten/src/ATen/native/mps/operations/Convolution.mm:624:23: error: instance method '-convolution3DWeightsGradientWithIncomingGradientTensor:sourceTensor:outputShape:forwardConvolutionDescriptor:name:' not found (return type defaults to 'id'); did you mean '-convolution2DWeightsGradientWithIncomingGradientTensor:sourceTensor:outputShape:forwardConvolutionDescriptor:name:'? [-Werror,-Wobjc-method-access]
2023-04-25T21:06:21.0546470Z             [mpsGraph convolution3DWeightsGradientWithIncomingGradientTensor:gradOutputTensorTranspose
2023-04-25T21:06:21.0547080Z                       ^

This issue is related to the declaration of Conv3D methods to be outside the macro.

Berzeg

Looks good, works well!

mattiaspaul · 2023-05-17T17:57:32Z

@kulinseth could you please check the revised version again that I posted three weeks ago?

vcasellesb · 2023-05-30T14:47:14Z

Hey mattias,

Thank you very much for your work. I wanted to train an nnUNet network for image segmentation, but I need to be able to do conv_transpose3d using mps. Do you have any news on that front?

Thank you very much again.

Best regards,
Vicent

kulinseth

Now looks good.

amitayas · 2023-06-02T07:37:21Z

@pytorchbot merge

pytorchmergebot · 2023-06-02T07:41:09Z

Merge started

Your change will be merged once all checks pass (ETA 0-4 Hours).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging

Check the merge workflow status
here

pytorchmergebot · 2023-06-02T07:41:10Z

Merge failed

Reason: This PR is too stale; the last push date was more than 3 days ago. Please rebase and try again. You can rebase and merge by leaving the following comment on this PR:
@pytorchbot merge -r
Or just rebase by leaving @pytorchbot rebase comment

Details for Dev Infra team

Raised by workflow job

pytorch-bot · 2023-06-02T07:42:31Z

You don't have permissions to rebase this PR since you are a first time contributor. If you think this is a mistake, please contact PyTorch Dev Infra.

pytorchmergebot · 2023-06-02T07:45:09Z

Merge failed

Reason: Comment with id 1573298960 not found

Details for Dev Infra team

Raised by workflow job

pytorch-bot · 2023-06-08T21:58:06Z

You are not authorized to force merges to this repository. Please use the regular @pytorchmergebot merge command instead

kulinseth · 2023-06-09T22:46:22Z

MacOS 12 is still failing with build:

F/Library/Developer/CommandLineTools/SDKs/MacOSX13.sdk/System/Library/Frameworks  -mfpu=neon -D__NEON__ -DTH_HAVE_THREAD -Wall -Wextra -Wno-unused-parameter -Wno-unused-function -Wno-unused-result -Wno-missing-field-initializers -Wno-unknown-pragmas -Wno-type-limits -Wno-array-bounds -Wno-strict-overflow -Wno-strict-aliasing -Wno-missing-braces -Wno-range-loop-analysis -fvisibility=hidden -O2 -std=gnu++17 -MD -MT caffe2/CMakeFiles/torch_cpu.dir/__/aten/src/ATen/native/mps/operations/Inverse.mm.o -MF caffe2/CMakeFiles/torch_cpu.dir/__/aten/src/ATen/native/mps/operations/Inverse.mm.o.d -o caffe2/CMakeFiles/torch_cpu.dir/__/aten/src/ATen/native/mps/operations/Inverse.mm.o -c /Users/ec2-user/runner/_work/pytorch/pytorch/aten/src/ATen/native/mps/operations/Inverse.mm
In file included from /Users/ec2-user/runner/_work/pytorch/pytorch/aten/src/ATen/native/mps/operations/Inverse.mm:2:
/Users/ec2-user/runner/_work/pytorch/pytorch/aten/src/ATen/native/mps/MPSGraphVenturaOps.h:58:52: error: expected a type
                                       descriptor:(MPSGraphConvolution3DOpDescriptor * _Nonnull) descriptor
                                                   ^
/Users/ec2-user/runner/_work/pytorch/pytorch/aten/src/ATen/native/mps/MPSGraphVenturaOps.h:64:74: error: expected a type
                                           forwardConvolutionDescriptor:(MPSGraphConvolution3DOpDescriptor * _Nonnull) forwardConvolutionDescriptor
                                                                         ^
/Users/ec2-user/runner/_work/pytorch/pytorch/aten/src/ATen/native/mps/MPSGraphVenturaOps.h:70:74: error: expected a type
                                           forwardConvolutionDescriptor:(MPSGraphConvolution3DOpDescriptor * _Nonnull) forwardConvolutionDescriptor
                                                                         ^
3 errors generated.

kulinseth · 2023-06-20T17:03:57Z

@pytorchbot rebase

pytorchmergebot · 2023-06-20T17:06:15Z

@pytorchbot started a rebase job onto refs/remotes/origin/viable/strict. Check the current status here

pytorchmergebot · 2023-06-20T17:06:25Z

Successfully rebased master onto refs/remotes/origin/viable/strict, please pull locally before adding more changes (for example, via git checkout master && git pull --rebase)

github-actions · 2023-08-19T20:33:40Z

Looks like this PR hasn't been updated in a while so we're going to go ahead and mark this as Stale.
Feel free to remove the Stale label if you feel this was a mistake.
If you are unable to remove the Stale label please contact a maintainer in order to do so.
If you want the bot to never mark this PR stale again, add the no-stale label.
Stale pull requests will automatically be closed after 30 days of inactivity.

kevinjohncutler · 2023-08-30T06:26:06Z

Any news on this? Seems like a lot of work went into it and it might be close.

kulinseth · 2023-09-06T06:02:57Z

@mattiaspaul , can you please rebase and fix the conflicts ? We don't have MacOS12 support in CI anymore and shouldn't hit previous issues.

cubehead · 2023-10-03T11:07:05Z

aten/src/ATen/native/mps/operations/Convolution.mm

+
+namespace at::native {
+//Create 3D convolution descriptor
+void fill_conv3d_desc(MPSGraphConvolution3DOpDescriptor* descriptor_,


you need to add the 'static' keyword on this function.

mattiaspaul · 2023-11-20T23:20:09Z

Hi @LucasSte glad to see you reopened the topic at #114183. I lost track of the progress of this so missed to do the rebase. But I’m happy to help if there’s anything new to resolve.

LucasSte · 2023-11-20T23:25:40Z

Hi @LucasSte glad to see you reopened the topic at #114183. I lost track of the progress of this so missed to do the rebase. But I’m happy to help if there’s anything new to resolve.

Hey @mattiaspaul, would you like to continue on this yourself? I think the tests for MacOS 12 need some fixes. If not, I'll have a look at them later.

@mattiaspaul

Fixes #77818 I saw that PR #99246 was approved, but no one fixed the rebase conflicts, so I am bringing this up again to be merged. I am leveraging @mattiaspaul work. Quoting the description here: > * this pull request enables 3D convolutions (forward/backward) for MPS (Apple Silicon) within the same Convolution.mm file as conv2d. > * does not support channel_last (since pytorch doesn't implement channel_last for 3D tensors) > * does not support conv3d_transpose and treats depth-separable convolutions not as normal case (there are no MPS kernels available for either of those so far) > * requires MacOS >=13.2 (Ventura) Please, let me know if there are any other changes needed and I'll be happy to implement them. Pull Request resolved: #114183 Approved by: https://github.com/malfet

@mattiaspaul

Fixes pytorch#77818 I saw that PR pytorch#99246 was approved, but no one fixed the rebase conflicts, so I am bringing this up again to be merged. I am leveraging @mattiaspaul work. Quoting the description here: > * this pull request enables 3D convolutions (forward/backward) for MPS (Apple Silicon) within the same Convolution.mm file as conv2d. > * does not support channel_last (since pytorch doesn't implement channel_last for 3D tensors) > * does not support conv3d_transpose and treats depth-separable convolutions not as normal case (there are no MPS kernels available for either of those so far) > * requires MacOS >=13.2 (Ventura) Please, let me know if there are any other changes needed and I'll be happy to implement them. Pull Request resolved: pytorch#114183 Approved by: https://github.com/malfet

@mattiaspaul

Fixes pytorch#77818 I saw that PR pytorch#99246 was approved, but no one fixed the rebase conflicts, so I am bringing this up again to be merged. I am leveraging @mattiaspaul work. Quoting the description here: > * this pull request enables 3D convolutions (forward/backward) for MPS (Apple Silicon) within the same Convolution.mm file as conv2d. > * does not support channel_last (since pytorch doesn't implement channel_last for 3D tensors) > * does not support conv3d_transpose and treats depth-separable convolutions not as normal case (there are no MPS kernels available for either of those so far) > * requires MacOS >=13.2 (Ventura) Please, let me know if there are any other changes needed and I'll be happy to implement them. Pull Request resolved: pytorch#114183 Approved by: https://github.com/malfet

mattiaspaul requested a review from kulinseth as a code owner April 15, 2023 21:34

pytorch-bot bot added ciflow/mps Run MPS tests (subset of trunk) release notes: mps Release notes category labels Apr 15, 2023

pytorchbot added the open source label Apr 15, 2023

Berzeg suggested changes Apr 22, 2023

View reviewed changes

mikaylagawarecki added the triaged This issue has been looked at a team member, and triaged and prioritized into an appropriate module label Apr 25, 2023

mattiaspaul requested a review from Berzeg April 25, 2023 19:21

mattiaspaul closed this Apr 25, 2023

mattiaspaul force-pushed the master branch from deaae9a to 865d30a Compare April 25, 2023 20:04

mattiaspaul reopened this Apr 25, 2023

kulinseth reviewed Apr 26, 2023

View reviewed changes

mattiaspaul requested a review from kulinseth April 27, 2023 13:28

Berzeg approved these changes May 4, 2023

View reviewed changes

Berzeg mentioned this pull request May 5, 2023

Add support for MaxPool3D on the MPS backend #100674

Open

kulinseth approved these changes May 30, 2023

View reviewed changes

pytorch-bot bot added the ciflow/trunk Trigger trunk jobs on your pull request label Jun 2, 2023

pytorchmergebot added the merging label Jun 2, 2023

pytorchmergebot removed the merging label Jun 2, 2023

pytorchmergebot added the merging label Jun 2, 2023

pytorchmergebot removed the merging label Jun 2, 2023

mattiaspaul added 3 commits June 20, 2023 17:06

Adding MPS support for 3D convolutions (on MacOS13) fixes bias/current

06362f1

Adding MPS support for 3D convolutions (on MacOS13) fixes bias/current

8b8284e

Adding MPS support for 3D convolutions (on MacOS13) fixes bias/2nd try

5ec4aa5

pytorchmergebot force-pushed the master branch from 4f35848 to 5ec4aa5 Compare June 20, 2023 17:06

Aviral-A mentioned this pull request Jun 25, 2023

Add M1/ M2 support to text-to-video pipeline huggingface/diffusers#2785

Closed

github-actions bot added the Stale label Aug 19, 2023

cubehead reviewed Oct 3, 2023

View reviewed changes

github-actions bot closed this Nov 2, 2023

LucasSte mentioned this pull request Nov 20, 2023

[MPS] Add Conv3D support for MPS #114183

Closed

Adding MPS support for 3D convolutions #99246

Adding MPS support for 3D convolutions #99246

Conversation

mattiaspaul commented Apr 15, 2023

pytorch-bot bot commented Apr 15, 2023 • edited Loading

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/99246

❌ 2 New Failures

linux-foundation-easycla bot commented Apr 15, 2023 • edited Loading

mattiaspaul commented Apr 21, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Berzeg Apr 22, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mattiaspaul commented Apr 25, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kulinseth commented Apr 26, 2023

Berzeg left a comment

Choose a reason for hiding this comment

mattiaspaul commented May 17, 2023

vcasellesb commented May 30, 2023

kulinseth left a comment

Choose a reason for hiding this comment

amitayas commented Jun 2, 2023

pytorchmergebot commented Jun 2, 2023

Merge started

pytorchmergebot commented Jun 2, 2023

Merge failed

pytorch-bot bot commented Jun 2, 2023

pytorchmergebot commented Jun 2, 2023

Merge failed

pytorch-bot bot commented Jun 8, 2023

kulinseth commented Jun 9, 2023

kulinseth commented Jun 20, 2023

pytorchmergebot commented Jun 20, 2023

pytorchmergebot commented Jun 20, 2023

github-actions bot commented Aug 19, 2023

kevinjohncutler commented Aug 30, 2023

kulinseth commented Sep 6, 2023

Choose a reason for hiding this comment

mattiaspaul commented Nov 20, 2023

LucasSte commented Nov 20, 2023 • edited Loading

pytorch-bot bot commented Apr 15, 2023 •

edited

Loading

linux-foundation-easycla bot commented Apr 15, 2023 •

edited

Loading

Berzeg Apr 22, 2023 •

edited

Loading

LucasSte commented Nov 20, 2023 •

edited

Loading