Thanks to visit codestin.com
Credit goes to github.com

Skip to content

Conversation

fengyuentau
Copy link
Member

@fengyuentau fengyuentau commented Jul 14, 2022

Fixes #22221.

In this case, the pack operator stacks three 4-dimensional tensors into a 5-dimensional tensor. However, the data layout of the output of the pack operator is still recognized as 4-dimensional (DATA_LAYOUT_NHWC) in tf_importer, which will lead to some problems like #22221 . Changing the data layout to DATA_LAYOUT_UNKNOWN fixes the problem.

For testing, I doubt whether we should put the model in opencv_extra since it is ~432MB.

Pull Request Readiness Checklist

See details at https://github.com/opencv/opencv/wiki/How_to_contribute#making-a-good-pull-request

  • I agree to contribute to the project under Apache 2 License.
  • To the best of my knowledge, the proposed patch is not based on a code under GPL or another license that is incompatible with OpenCV
  • The PR is proposed to the proper branch
  • There is a reference to the original bug report and related work
  • There is accuracy test, performance test and test data in opencv_extra repository, if applicable
    Patch to opencv_extra has the same branch name.
  • The feature is well documented and sample code can be built with the project CMake

@zihaomu
Copy link
Member

zihaomu commented Jul 15, 2022

Hi @fengyuentau, you can use generate_tf_models.py or generate_tf2_models.py to export specific pb model.

Copy link
Member

@zihaomu zihaomu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for your contribution!

if (dim != 0)
CV_Error(Error::StsNotImplemented, "Unsupported mode of pack operation.");

data_layouts[name] = DATA_LAYOUT_UNKNOWN;
Copy link
Member

@zihaomu zihaomu Jul 15, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In tf_importer, some layers are sensitive to the input's data layout (like Transpose). And these layers may error if they are after the Pack layer.
Maybe we can judge the output data layout by the input data layout.
For example, the input is 4 dim and NHWC data layout, which means the output should be the 5 dims.
But I'm not sure if this is the right way to do it.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can consider cases that have defined data layout in tf_importer, for example stacking hwc can have nhwc data layout output. The problem is we cannot take every single data layout into account since theoratically there is countless.

I wonder in which case pack is followed by transpose though.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we keep it unknown for the output data layout of pack since there are only 4 data layouts defined:

enum DataLayout
{
    DATA_LAYOUT_NHWC,
    DATA_LAYOUT_NCHW,
    DATA_LAYOUT_NDHWC,
    DATA_LAYOUT_UNKNOWN,
    DATA_LAYOUT_PLANAR  // 2-dimensional outputs (matmul, flatten, reshape to 2d)
};

These data layouts do not satisfy our need if we want to describe the output data layout of pack unless more are added.

Copy link
Member

@zihaomu zihaomu Jul 15, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A complete test example would be helpful.
From stack API, we can find that the axis is [-(R+1), R+1).
The following code generates all possible cases of tf.stack.

for d in range(1, 7):
    for axis in range(-d - 1, d + 1):
        inp0 = tf.placeholder(tf.float32, list(range(1, d + 1)), 'input0')
        constant = tf.random_normal(list(range(1, d + 1)))
        pack_model = tf.stack([inp0, constant])
        print(pack_model)
        # save(inp0, pack_model, f'pack_dim{d}_{axis}', optimize=False)

Maybe in tf.stackcase, we should use two placeholders as model input. I'm not sure the save function in generate_tf_models.py can do this.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Testing a single Pack operator does not make any sense in this pull request. This pull request is changing the output data layout, and with or without this change, such tests will always pass.

Copy link
Member

@zihaomu zihaomu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! 👍

@opencv-pushbot opencv-pushbot merged commit 1d13402 into opencv:4.x Jul 16, 2022
@fengyuentau fengyuentau deleted the fix_for_22221 branch July 18, 2022 03:07
@alalek alalek mentioned this pull request Aug 21, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

human_parsing example from dnn Assertion failed (int)_numAxes == inputs[0].size()

5 participants