Can one of the admins verify this patch?

When input tensor has zero elements, the reference BN forward implementation in the test script gives NaN for mean/variance, however I return zeros.
NaN seems to make more sense in terms of math, though it will then require some special treatment in the actual training. If there is no objections I'll switch to NaNs.

@zhangyaobit Could you comment on this: #15264 (comment) please?

@tensorflow-jenkins test this please

I haven't switched from returning zeros to returning NaNs, so the test is failing. I'll do that later

Let's wait for @zhangyaobit 's comment on this first then.

@ppwwyyxx, could you comment on the use case of an empty input (e.g. [0, 64, 64, 3])? Should we require the input is non-empty?

In object detection we may run CNN on patches (object candidates) cropped from an image with predicted boxes. When the image has no candidates we'll get zero patches. In training we can filter out these data but still can't avoid it in testing. A workaround is to use tf.cond but I hope the op can support it by itself.

In fact, I just found that the existing CPU (eigen) implementation of fused_batch_norm can work with empty input and returns NaNs for mean/variance.

Thanks! This sounds good. Could you make the behavior of GPU implementation consistent with Eigen (return NaNs)? Please let me know once the PR is ready for review.

@zhangyaobit The changes have been made. Could you review it when you have a time? Thanks!

@tensorflow-jenkins test this please

The implementation of FillFunctor, SetZeroFunctor, etc, are split in two bazel targets: :fill_functor and :constant_op. However :constant_op depends on a lot of stuff: it depends on :transpose_functor which depends on conv2d (I saw a TODO for this by @yzhwang). But conv2d_grad_filter needs to use SetZeroFunctor which ends up being a cyclic reference.

-    deps = ARRAY_DEPS,
+    deps = [
+        "//tensorflow/core:array_grad",
+        "//tensorflow/core:array_ops_op_lib",
+        "//tensorflow/core:framework",
+        "//tensorflow/core:lib",
+        "//third_party/eigen3",
+        ":bounds_check",
+        ":fill_functor",
+        ":ops_util",
+    ],

Cherry-picking the dependencies seems to make this PR build, but doesn't sound like an ideal solution. In general I guess functors should not depend on ops, but here fill_functor is actually in :constant_op, and :transpose_functor depends on :conv_ops.

Jenkins, test this please.

Thanks @drpngx for help! However I haven't yet fixed the cyclic dependency error mentioned above. I
think a proper fix would be one of the following:

1. Move GPU implementation of fill_functor to target :fill_functor.
2. Don't let :transpose_functor depend on :conv_ops.

The first one seems to be within my reach. I can give it a try but not sure if there is any reason why this is not done before.

@yifeif for some reason, this is stuck with kokoro-run. There are others PRs as well.

Ah if a PR has ran Kokoro tests before, it will need the force-run tag :).

Oh, makes sense, of course.

@ppwwyyxx there are some build breakages on GPU, could you check?

The build was failing because of the bazel dependency problem, which should've been fixed now after I moved implementations to :fill_functor target.

Jenkins, test this please.

/CC @gunan ran out of devmapper space on the Jenkins build.

devmapper: Thin Pool has 968455 free data blocks which is less than minimum required 983040 free data blocks. Create more free space in thin pool or use dm.min_free_space option to change behavior
ERROR: docker build failed. Dockerfile is at /var/lib/jenkins/workspace/tensorflow-pull-requests-cpu-python3/tensorflow/tools/ci_build/Dockerfile.cpu

Jenkins, test this please.

Why is 'XLA' build failing without details?

OK, looks like @yifeif might have fixed it. We just ran out of space on the machine.

@ppwwyyxx looks like an internal infra failure. I kicked it off again.

Merged. Woohoo!