Thanks to visit codestin.com
Credit goes to github.com

Skip to content

dnn: add DepthToSpace and SpaceToDepth #25779

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 13 commits into from
Jun 21, 2024
Merged

dnn: add DepthToSpace and SpaceToDepth #25779

merged 13 commits into from
Jun 21, 2024
+663 −95

Conversation

@fengyuentau
Copy link
Member

@fengyuentau fengyuentau commented Jun 18, 2024 •
edited
Loading

We are working on updating WeChat QRCode module. One of the new models is a fully convolutional model and hence it should be able to run with different input shapes. However, it has an operator DepthToSpace, which is parsed as a subgraph of Reshape -> Permute -> Reshape with a fixed shape getting during parsing. The subgraph itself is not a problem, but the true problem is the subgraph with a fixed input and output shape regardless input changes. This does not allow the model to run with different input shapes.

Solution is to add a dedicated layer for DepthtoSpace and SpaceToDepth.

Backend support:

  • CPU
  • CUDA
  • OpenCL
  • OpenVINO
  • CANN
  • TIMVX
  • Vulkan (missing fundamental tools, like permutation and reshape)

Pull Request Readiness Checklist

See details at https://github.com/opencv/opencv/wiki/How_to_contribute#making-a-good-pull-request

  • I agree to contribute to the project under Apache 2 License.
  • To the best of my knowledge, the proposed patch is not based on a code under GPL or another license that is incompatible with OpenCV
  • The PR is proposed to the proper branch
  • There is a reference to the original bug report and related work
  • There is accuracy test, performance test and test data in opencv_extra repository, if applicable
    Patch to opencv_extra has the same branch name.
  • The feature is well documented and sample code can be built with the project CMake
force_builders=Linux OpenCL,Win64 OpenCL

@asmorkalov asmorkalov added category: dnn (onnx) ONNX suport issues in DNN module feature labels Jun 18, 2024
@asmorkalov asmorkalov added this to the 4.11.0 milestone Jun 18, 2024
@fengyuentau

This comment was marked as resolved.

Sign in to view
@fengyuentau fengyuentau marked this pull request as ready for review June 20, 2024 10:08
@fengyuentau fengyuentau added category:dnn_timvx TIM-VX related issues in DNN module category:dnn_cann CANN backend related issues in DNN module labels Jun 20, 2024
@fengyuentau fengyuentau changed the title dnn: add dedicated DepthToSpace and SpaceToDepth layers dnn: add a dedicated DepthSpaceLayer for DepthToSpace and SpaceToDepth Jun 20, 2024
dkurt
dkurt reviewed Jun 20, 2024
View reviewed changes
dkurt
dkurt reviewed Jun 20, 2024
View reviewed changes
@fengyuentau fengyuentau changed the title dnn: add a dedicated DepthSpaceLayer for DepthToSpace and SpaceToDepth dnn: add DepthToSpace and SpaceToDepth Jun 20, 2024
zihaomu
zihaomu approved these changes Jun 21, 2024
View reviewed changes
Copy link
Member

@zihaomu zihaomu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! 👍

@dkurt
Copy link
Member

dkurt commented Jun 21, 2024

Failed CUDA tests:

[  FAILED  ] Test_ONNX_conformance.Layer_Test/test_depthtospace_crd_mode_CUDA_CUDA, where GetParam() = (test_depthtospace_crd_mode, CUDA/CUDA)
[  FAILED  ] Test_ONNX_conformance.Layer_Test/test_depthtospace_crd_mode_CUDA_CUDA_FP16, where GetParam() = (test_depthtospace_crd_mode, CUDA/CUDA_FP16)
[  FAILED  ] Test_ONNX_conformance.Layer_Test/test_depthtospace_crd_mode_example_CUDA_CUDA, where GetParam() = (test_depthtospace_crd_mode_example, CUDA/CUDA)
[  FAILED  ] Test_ONNX_conformance.Layer_Test/test_depthtospace_crd_mode_example_CUDA_CUDA_FP16, where GetParam() = (test_depthtospace_crd_mode_example, CUDA/CUDA_FP16)
[  FAILED  ] Test_ONNX_conformance.Layer_Test/test_depthtospace_dcr_mode_CUDA_CUDA, where GetParam() = (test_depthtospace_dcr_mode, CUDA/CUDA)
[  FAILED  ] Test_ONNX_conformance.Layer_Test/test_depthtospace_dcr_mode_CUDA_CUDA_FP16, where GetParam() = (test_depthtospace_dcr_mode, CUDA/CUDA_FP16)
[  FAILED  ] Test_ONNX_conformance.Layer_Test/test_depthtospace_example_CUDA_CUDA, where GetParam() = (test_depthtospace_example, CUDA/CUDA)
[  FAILED  ] Test_ONNX_conformance.Layer_Test/test_depthtospace_example_CUDA_CUDA_FP16, where GetParam() = (test_depthtospace_example, CUDA/CUDA_FP16)
[  FAILED  ] Test_ONNX_conformance.Layer_Test/test_spacetodepth_CUDA_CUDA, where GetParam() = (test_spacetodepth, CUDA/CUDA)
[  FAILED  ] Test_ONNX_conformance.Layer_Test/test_spacetodepth_CUDA_CUDA_FP16, where GetParam() = (test_spacetodepth, CUDA/CUDA_FP16)
[  FAILED  ] Test_ONNX_conformance.Layer_Test/test_spacetodepth_example_CUDA_CUDA, where GetParam() = (test_spacetodepth_example, CUDA/CUDA)
[  FAILED  ] Test_ONNX_conformance.Layer_Test/test_spacetodepth_example_CUDA_CUDA_FP16, where GetParam() = (test_spacetodepth_example, CUDA/CUDA_FP16)

@fengyuentau
Copy link
Member Author

fengyuentau commented Jun 21, 2024

Failed CUDA tests:

[  FAILED  ] Test_ONNX_conformance.Layer_Test/test_depthtospace_crd_mode_CUDA_CUDA, where GetParam() = (test_depthtospace_crd_mode, CUDA/CUDA)
[  FAILED  ] Test_ONNX_conformance.Layer_Test/test_depthtospace_crd_mode_CUDA_CUDA_FP16, where GetParam() = (test_depthtospace_crd_mode, CUDA/CUDA_FP16)
[  FAILED  ] Test_ONNX_conformance.Layer_Test/test_depthtospace_crd_mode_example_CUDA_CUDA, where GetParam() = (test_depthtospace_crd_mode_example, CUDA/CUDA)
[  FAILED  ] Test_ONNX_conformance.Layer_Test/test_depthtospace_crd_mode_example_CUDA_CUDA_FP16, where GetParam() = (test_depthtospace_crd_mode_example, CUDA/CUDA_FP16)
[  FAILED  ] Test_ONNX_conformance.Layer_Test/test_depthtospace_dcr_mode_CUDA_CUDA, where GetParam() = (test_depthtospace_dcr_mode, CUDA/CUDA)
[  FAILED  ] Test_ONNX_conformance.Layer_Test/test_depthtospace_dcr_mode_CUDA_CUDA_FP16, where GetParam() = (test_depthtospace_dcr_mode, CUDA/CUDA_FP16)
[  FAILED  ] Test_ONNX_conformance.Layer_Test/test_depthtospace_example_CUDA_CUDA, where GetParam() = (test_depthtospace_example, CUDA/CUDA)
[  FAILED  ] Test_ONNX_conformance.Layer_Test/test_depthtospace_example_CUDA_CUDA_FP16, where GetParam() = (test_depthtospace_example, CUDA/CUDA_FP16)
[  FAILED  ] Test_ONNX_conformance.Layer_Test/test_spacetodepth_CUDA_CUDA, where GetParam() = (test_spacetodepth, CUDA/CUDA)
[  FAILED  ] Test_ONNX_conformance.Layer_Test/test_spacetodepth_CUDA_CUDA_FP16, where GetParam() = (test_spacetodepth, CUDA/CUDA_FP16)
[  FAILED  ] Test_ONNX_conformance.Layer_Test/test_spacetodepth_example_CUDA_CUDA, where GetParam() = (test_spacetodepth_example, CUDA/CUDA)
[  FAILED  ] Test_ONNX_conformance.Layer_Test/test_spacetodepth_example_CUDA_CUDA_FP16, where GetParam() = (test_spacetodepth_example, CUDA/CUDA_FP16)

Thank you for listing the error on CUDA backend. Hopefully the latest commit can fix this problem.

@asmorkalov asmorkalov merged commit 3f13ce7 into opencv:4.x Jun 21, 2024
@fengyuentau fengyuentau deleted the dnn/fix_onnx_depthtospace branch June 21, 2024 16:32
@asmorkalov asmorkalov mentioned this pull request Jul 1, 2024
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@asmorkalov asmorkalov asmorkalov left review comments

@zihaomu zihaomu zihaomu approved these changes

@dkurt dkurt dkurt approved these changes

Assignees

No one assigned

Labels

category:dnn_cann CANN backend related issues in DNN module category: dnn (onnx) ONNX suport issues in DNN module category:dnn_timvx TIM-VX related issues in DNN module feature

Projects

None yet

Milestone

4.11.0

Development

Successfully merging this pull request may close these issues.

4 participants

@fengyuentau @dkurt @asmorkalov @zihaomu