dnn: add shared fastNorm kernel for mvn, instance norm and layer norm #24409

fengyuentau · 2023-10-16T11:37:15Z

Relates #24378 (comment)

TODO:

add fastNorm
refactor layer norm with fastNorm
refactor mvn with fastNorm
add onnx mvn in importer (in a new PR?)
refactor instance norm with fastNorm (in another PR dnn onnx: add instance norm layer #24378, need to merge this one first though)

Pull Request Readiness Checklist

See details at https://github.com/opencv/opencv/wiki/How_to_contribute#making-a-good-pull-request

I agree to contribute to the project under Apache 2 License.
To the best of my knowledge, the proposed patch is not based on a code under GPL or another license that is incompatible with OpenCV
The PR is proposed to the proper branch
There is a reference to the original bug report and related work
There is accuracy test, performance test and test data in opencv_extra repository, if applicable
Patch to opencv_extra has the same branch name.
The feature is well documented and sample code can be built with the project CMake

fengyuentau · 2023-10-17T03:47:34Z

It seems there are a few caffe models using mvn layer. I cannot find one in opencv_extra.

There is one layer test though: testdata/dnn/layers/layer_mvn.prototxt.

dkurt · 2023-10-17T07:22:58Z

@fengyuentau, AlexNet maybe?

fengyuentau · 2023-10-17T07:26:50Z

@fengyuentau, AlexNet maybe?

Seems like a no, see below:

dkurt · 2023-10-17T08:02:11Z

@fengyuentau, sorry, it's LRN.

There are tests which use MVN:

[  FAILED  ] Test_Caffe_layers.MVN/0, where GetParam() = OCV/CPU
[  FAILED  ] Test_Caffe_layers.BatchNorm/0, where GetParam() = OCV/CPU
[  FAILED  ] Test_ONNX_layers.InstanceNorm/0, where GetParam() = OCV/CPU
[  FAILED  ] Test_TensorFlow_layers.batch_norm_10/0, where GetParam() = OCV/CPU
[  FAILED  ] Test_TensorFlow_layers.batch_norm_11/0, where GetParam() = OCV/CPU
[  FAILED  ] Test_TensorFlow_layers.batch_norm_13/0, where GetParam() = OCV/CPU
[  FAILED  ] Test_Torch_layers.run_batch_norm/0, where GetParam() = OCV/CPU
[  FAILED  ] Test_Torch_nets.FastNeuralStyle_accuracy/0, where GetParam() = OCV/CPU

fengyuentau · 2023-10-17T08:06:59Z

@dkurt fuse_batch_norm seems like turning mvn into batch norm right? It somehow confuses me.

dkurt · 2023-10-17T09:33:38Z

@fengyuentau, exactly. This is for channel-wise scale/bias.

fengyuentau · 2023-10-17T10:15:47Z

@fengyuentau, exactly. This is for channel-wise scale/bias.

Thank you for the clarification. This PR is all set for review. Most of the tests are green on my mac m1 except for something wrong with Test_Torch_nets.FastNeuralStyle_accuracy.

fengyuentau · 2023-10-17T12:23:54Z

Should we just tune the threshold?

ubuntu arm64:

[ RUN      ] Test_Torch_nets.FastNeuralStyle_accuracy/0, where GetParam() = OCV/CPU
/home/ci/opencv/modules/dnn/test/test_common.impl.hpp:79: Failure
Expected: (normInf) <= (lInf), actual: 1.14948 vs 1.11
  |ref| = 255
[  FAILED  ] Test_Torch_nets.FastNeuralStyle_accuracy/0, where GetParam() = OCV/CPU (279 ms)

ubuntu x64:

[ RUN      ] Test_Torch_nets.FastNeuralStyle_accuracy/0, where GetParam() = OCV/CPU
/home/ci/opencv/modules/dnn/test/test_common.impl.hpp:79: Failure
Expected: (normInf) <= (lInf), actual: 1.15276 vs 1.11
  |ref| = 255
[  FAILED  ] Test_Torch_nets.FastNeuralStyle_accuracy/0, where GetParam() = OCV/CPU (335 ms)

windows10 x64 vulkan:

[ RUN      ] Test_Torch_nets.FastNeuralStyle_accuracy/0, where GetParam() = VKCOM/VULKAN
C:\GHA-OCV-2\_work\opencv\opencv\opencv\modules\dnn\test\test_common.impl.hpp(79): error: Expected: (normInf) <= (lInf), actual: 1.1487 vs 1.11
  |ref| = 255
[  FAILED  ] Test_Torch_nets.FastNeuralStyle_accuracy/0, where GetParam() = VKCOM/VULKAN (686 ms)
[ RUN      ] Test_Torch_nets.FastNeuralStyle_accuracy/1, where GetParam() = OCV/CPU
C:\GHA-OCV-2\_work\opencv\opencv\opencv\modules\dnn\test\test_common.impl.hpp(79): error: Expected: (normInf) <= (lInf), actual: 1.15276 vs 1.11
  |ref| = 255
[  FAILED  ] Test_Torch_nets.FastNeuralStyle_accuracy/1, where GetParam() = OCV/CPU (572 ms)

macos arm64:

[ RUN      ] Test_Torch_nets.FastNeuralStyle_accuracy/0, where GetParam() = OCV/CPU
/Users/opencv-cn/GHA-OCV-1/_work/opencv/opencv/opencv/modules/dnn/test/test_common.impl.hpp:79: Failure
Expected: (normInf) <= (lInf), actual: 1.15292 vs 1.11
  |ref| = 255
[  FAILED  ] Test_Torch_nets.FastNeuralStyle_accuracy/0, where GetParam() = OCV/CPU (350 ms)
[ RUN      ] Test_Torch_nets.FastNeuralStyle_accuracy/1, where GetParam() = OCV/CPU_FP16
/Users/opencv-cn/GHA-OCV-1/_work/opencv/opencv/opencv/modules/dnn/test/test_common.impl.hpp:76: Failure
Expected: (normL1) <= (l1), actual: 0.633552 vs 0.62
  |ref| = 255
[  FAILED  ] Test_Torch_nets.FastNeuralStyle_accuracy/1, where GetParam() = OCV/CPU_FP16 (336 ms)

modules/dnn/src/layers/cpu_kernels/fast_norm.cpp

modules/dnn/src/layers/layer_norm.cpp

modules/dnn/src/layers/cpu_kernels/fast_norm.cpp

modules/dnn/include/opencv2/dnn/all_layers.hpp

dkurt · 2023-11-01T07:36:13Z

modules/dnn/include/opencv2/dnn/all_layers.hpp

    class CV_EXPORTS LayerNormLayer : public Layer
    {
    public:
+        bool hasBias; // Deprecated, preserve for compatibility


CV_DEPRECATED_EXTERNAL bool hasBias

dnn: add shared fastNorm kernel for mvn, instance norm and layer norm opencv#24409 Relates opencv#24378 (comment) TODO: - [x] add fastNorm - [x] refactor layer norm with fastNorm - [x] refactor mvn with fastNorm - [ ] add onnx mvn in importer (in a new PR?) - [ ] refactor instance norm with fastNorm (in another PR opencv#24378, need to merge this one first though) ### Pull Request Readiness Checklist See details at https://github.com/opencv/opencv/wiki/How_to_contribute#making-a-good-pull-request - [x] I agree to contribute to the project under Apache 2 License. - [x] To the best of my knowledge, the proposed patch is not based on a code under GPL or another license that is incompatible with OpenCV - [x] The PR is proposed to the proper branch - [x] There is a reference to the original bug report and related work - [x] There is accuracy test, performance test and test data in opencv_extra repository, if applicable Patch to opencv_extra has the same branch name. - [x] The feature is well documented and sample code can be built with the project CMake

fengyuentau added 2 commits October 16, 2023 19:07

extract fastNorm kernel

7a54375

fix bugs in fastNorm; refactor layer norm with fastNorm

47d9f32

fengyuentau added optimization category: dnn labels Oct 16, 2023

fengyuentau added this to the 4.9.0 milestone Oct 16, 2023

fengyuentau requested a review from dkurt October 16, 2023 11:37

fengyuentau assigned vpisarev Oct 16, 2023

fengyuentau added 2 commits October 17, 2023 18:06

add kernels; use the kernel without bias for layer norm

797565a

refactor mvn

38015db

fengyuentau marked this pull request as ready for review October 17, 2023 10:12

fengyuentau added 2 commits October 18, 2023 10:52

tune threshold

e526114

remove extra whitespaces

d77e11a

dkurt reviewed Oct 18, 2023

View reviewed changes

modules/dnn/src/layers/cpu_kernels/fast_norm.cpp Show resolved Hide resolved

dkurt reviewed Oct 18, 2023

View reviewed changes

modules/dnn/src/layers/layer_norm.cpp Outdated Show resolved Hide resolved

dkurt reviewed Oct 18, 2023

View reviewed changes

modules/dnn/src/layers/cpu_kernels/fast_norm.cpp Show resolved Hide resolved

fengyuentau added 2 commits October 19, 2023 15:13

remove unused code

5f59a58

add fastNormChannel; assume axis are normalized in kernels

bd88d81

dkurt mentioned this pull request Oct 30, 2023

dnn onnx: add instance norm layer #24378

Merged

12 tasks

dkurt reviewed Oct 30, 2023

View reviewed changes

modules/dnn/include/opencv2/dnn/all_layers.hpp Show resolved Hide resolved

dkurt approved these changes Oct 30, 2023

View reviewed changes

preserve hasBias for compatibility

8cb0542

dkurt reviewed Nov 1, 2023

View reviewed changes

use CV_DEPRECATED_EXTERNAL for hasBias

6c869a6

fengyuentau mentioned this pull request Nov 1, 2023

dnn cann backend: add hardswish, layernorm and instasnce norm for cann and bug fix #24462

Merged

9 tasks

asmorkalov assigned dkurt and unassigned vpisarev Nov 1, 2023

asmorkalov merged commit c91af16 into opencv:4.x Nov 1, 2023

asmorkalov mentioned this pull request Nov 3, 2023

(5.x) Merge 4.x #24486

Merged

dkurt mentioned this pull request Nov 30, 2023

dnn onnx: add group norm layer #24610

Merged

12 tasks

fengyuentau deleted the norm_kernel branch June 28, 2024 09:57

Uh oh!

dnn: add shared fastNorm kernel for mvn, instance norm and layer norm #24409

dnn: add shared fastNorm kernel for mvn, instance norm and layer norm #24409

Uh oh!

Conversation

fengyuentau commented Oct 16, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Pull Request Readiness Checklist

Uh oh!

fengyuentau commented Oct 17, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dkurt commented Oct 17, 2023

Uh oh!

fengyuentau commented Oct 17, 2023

Uh oh!

dkurt commented Oct 17, 2023

Uh oh!

fengyuentau commented Oct 17, 2023

Uh oh!

dkurt commented Oct 17, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

fengyuentau commented Oct 17, 2023

Uh oh!

fengyuentau commented Oct 17, 2023

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

dkurt Nov 1, 2023

Choose a reason for hiding this comment

Uh oh!

fengyuentau Nov 1, 2023

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

fengyuentau commented Oct 16, 2023 •

edited

Loading

fengyuentau commented Oct 17, 2023 •

edited

Loading

dkurt commented Oct 17, 2023 •

edited

Loading