-
-
Notifications
You must be signed in to change notification settings - Fork 56.3k
Changing data_layout for pack in tf_importer #22249
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
Hi @fengyuentau, you can use generate_tf_models.py or generate_tf2_models.py to export specific pb model. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for your contribution!
if (dim != 0) | ||
CV_Error(Error::StsNotImplemented, "Unsupported mode of pack operation."); | ||
|
||
data_layouts[name] = DATA_LAYOUT_UNKNOWN; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
In tf_importer
, some layers are sensitive to the input's data layout (like Transpose
). And these layers may error if they are after the Pack
layer.
Maybe we can judge the output data layout by the input data layout.
For example, the input is 4 dim and NHWC data layout, which means the output should be the 5 dims.
But I'm not sure if this is the right way to do it.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We can consider cases that have defined data layout in tf_importer
, for example stacking hwc can have nhwc data layout output. The problem is we cannot take every single data layout into account since theoratically there is countless.
I wonder in which case pack is followed by transpose though.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think we keep it unknown for the output data layout of pack since there are only 4 data layouts defined:
enum DataLayout
{
DATA_LAYOUT_NHWC,
DATA_LAYOUT_NCHW,
DATA_LAYOUT_NDHWC,
DATA_LAYOUT_UNKNOWN,
DATA_LAYOUT_PLANAR // 2-dimensional outputs (matmul, flatten, reshape to 2d)
};
These data layouts do not satisfy our need if we want to describe the output data layout of pack unless more are added.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
A complete test example would be helpful.
From stack API, we can find that the axis is [-(R+1), R+1)
.
The following code generates all possible cases of tf.stack
.
for d in range(1, 7):
for axis in range(-d - 1, d + 1):
inp0 = tf.placeholder(tf.float32, list(range(1, d + 1)), 'input0')
constant = tf.random_normal(list(range(1, d + 1)))
pack_model = tf.stack([inp0, constant])
print(pack_model)
# save(inp0, pack_model, f'pack_dim{d}_{axis}', optimize=False)
Maybe in tf.stack
case, we should use two placeholders as model input. I'm not sure the save
function in generate_tf_models.py can do this.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Testing a single Pack
operator does not make any sense in this pull request. This pull request is changing the output data layout, and with or without this change, such tests will always pass.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM! 👍
Fixes #22221.
In this case, the
pack
operator stacks three 4-dimensional tensors into a 5-dimensional tensor. However, the data layout of the output of thepack
operator is still recognized as 4-dimensional (DATA_LAYOUT_NHWC
) in tf_importer, which will lead to some problems like #22221 . Changing the data layout toDATA_LAYOUT_UNKNOWN
fixes the problem.For testing, I doubt whether we should put the model in opencv_extra since it is ~432MB.
Pull Request Readiness Checklist
See details at https://github.com/opencv/opencv/wiki/How_to_contribute#making-a-good-pull-request
Patch to opencv_extra has the same branch name.