Codestin Search App

ppwwyyxx · 2017-08-25T03:25:21Z

tensorflow-jenkins · 2017-08-25T03:25:22Z

Can one of the admins verify this patch?

mention-bot · 2017-08-25T03:25:23Z

@ppwwyyxx, thanks for your PR! By analyzing the history of the files in this pull request, we identified @zhangyaobit, @tensorflower-gardener and @keveman to be potential reviewers.

zhangyaobit

Thanks for the nice contribution, Yuxin!

zhangyaobit · 2017-08-29T20:38:24Z

+                  const Tensor& mean_input, const Tensor& variance_input,
+                  T epsilon, Tensor* x_backprop_output,
+                  Tensor* scale_backprop_output, Tensor* offset_backprop_output,
+                  typename TTypes<T>::Vec scratch1, typename TTypes<T>::Vec scratch2) {


Are these two scratch allocation needed? Could you follow the Eigen implementation of FusedBatchNorm, where no temp allocation is needed (you can still use something like "Eigen::Tensor<T, 1, Eigen::RowMajor> mean(depth)" though).

Seems like Eigen::Tensor<T, 1, Eigen::RowMajor> scratch1(depth) only allocate memory on CPUs? In a GPU kernel, using this ends up with CUDA_ERROR_ILLEGAL_ADDRESS. Using the OpKernelContext seems like the standard device-agnostic way to allocate memory.

Thanks, sounds good to me.

zhangyaobit · 2017-08-29T20:39:51Z

+// Functor used by FusedBatchNormGradOp to do the computations when is_training=False.
+// Both CPU and GPU will use this functor.
+template <typename Device, typename T>
+struct FusedBatchNormFreezeGrad {


Could you move this function to fused_batch_norm_op.cc?

If I understand the build process correctly, this functor needs to be instantiated in both fused_batch_norm_op.cc and fused_batch_norm_op.cu.cc, to be compiled to two kernels by nvcc and gcc respectively. Therefore it has to be in a header file to be included by both .cc and .cu.cc. This seems like what's been done for other kernels as well (e.g. reverse_op).

Sounds good.

zhangyaobit · 2017-08-29T20:47:52Z

  grad_y = op.inputs[0]
  x = op.inputs[1]
  scale = op.inputs[2]
+  pop_mean = op.inputs[3]


I think here you will need the input 3 and 4 of op FusedBatchNorm instead of FusedBatchNormGrad. Could you forward the pop mean and pop variance to the output 3 and 4 of FusedBatchNorm in the C++ code? This way you can also unify the two branches in the "_FusedBatchNormGrad(op, *grad)".

pop_mean and pop_var are inputs to FusedBatchNormGrad as well, what's the reason to not use them directly like this?

Note that input 3 and 4 are reserve_space_1 and reserve_space_2, which are not pop mean and pop var, but
reserve_space_1: A 1D Tensor for the computed batch mean, to be reused
in the gradient computation.
reserve_space_2: A 1D Tensor for the computed batch variance (inverted variance
in the cuDNN case), to be used in the gradient computation.

REGISTER_OP("FusedBatchNormGrad")
.Input("y_backprop: T")
.Input("x: T")
.Input("scale: T")
.Input("reserve_space_1: T")
.Input("reserve_space_2: T")
.Output("x_backprop: T")
.Output("scale_backprop: T")
.Output("offset_backprop: T")
.Output("reserve_space_3: T")
.Output("reserve_space_4: T")
.Attr("T: {float}")
.Attr("epsilon: float = 0.0001")
.Attr("data_format: string = 'NHWC'")
.Attr("is_training: bool = true")
......

Ok, you did the forwarding in _FusedBatchNormGrad, instead of on the C++ side. I think that is fine too. If you go with this way, could you update the comments of reserve_space_1 and reserve_space_2 saying they are pop mean and variance when is_training is False?

Comments were updated.

zhangyaobit · 2017-08-30T01:37:28Z

+                  T epsilon, Tensor* x_backprop_output,
+                  Tensor* scale_backprop_output, Tensor* offset_backprop_output,
+                  typename TTypes<T>::Vec scratch1, typename TTypes<T>::Vec scratch2) {
+    typename TTypes<T, 4>::ConstTensor out_backprop(y_backprop_input.tensor<T, 4>());


Rename to y_backprop?

zhangyaobit · 2017-08-30T01:38:31Z

+
+    // db = out_backprop
+    // dg = out_backprop * ((x - m) * rsqrt(v + epsilon))
+    // dx = out_backprop * (gamma * rsqrt(v + epsilon))


Rename all names to be consistent of what is used in the program?

zhangyaobit · 2017-08-30T01:40:51Z

+                                               .eval()
+                                               .reshape(one_by_depth)
+                                               .broadcast(rest_by_one));
+    scale_backprop.device(d) = scratch2 * scratch1;


Are what implemented above equivalent to python implementation below?
grad_offset = reduce_sum(grad_y)
grad_scale = reduce_sum(grad_y*(x-pop_mean)*var_rsqrt)
grad_x = grad_y * scale * var_rsqrt

That looks equivalent to me

zhangyaobit · 2017-08-30T17:45:17Z

 x: A 4D Tensor for input data.
 scale: A 1D Tensor for scaling factor, to scale the normalized x.
-reserve_space_1: A 1D Tensor for the computed batch mean, to be reused
+reserve_space_1: A 1D Tensor for the computed batch mean when is_training is True,


How about something like this?

A 1D Tensor for the computed batch mean when is_training is True, to be reused in the gradient computation; or the population mean when is_training is False, to be used in the second-order gradient computation.

And the same for reserve_space_2.

When is_training is False, pop_mean/pop_variance is needed for first-order gradient computation as well.

Ah, ok, then "to be used in the first-order and second-order gradient computation."

zhangyaobit · 2017-08-30T19:24:25Z

-        epsilon_, x_backprop, scale_backprop, offset_backprop, tensor_format_);
+    if (is_training_) {
+      functor::FusedBatchNormGrad<Device, T>()(
+          context, y_backprop, x, scale, saved_mean, saved_maybe_inv_var,


This is a bit confusing. Rename to saved_mean_or_pop_mean and saved_maybe_inv_var_or_pop_var?

Did some rename (on one existing kernel as well) to improve clarity.

zhangyaobit

Nice. Thanks!

zhangyaobit · 2017-08-30T22:02:31Z

Let's wait a bit to see if zheng-xq has any comments (This PR may affect lots of users, let's be extra careful :) ). Thanks!

ppwwyyxx · 2017-09-11T19:35:21Z

Any updates?

zhangyaobit · 2017-09-12T03:20:51Z

Thanks for the patience, Yuxin! zheng-xq will respond soon.

drpngx · 2017-09-17T18:29:17Z

/CC @zheng-xq with the @ sign to trigger notification.

drpngx · 2017-09-17T18:29:35Z

Jenkins, test this please.

zhangyaobit · 2017-09-19T21:26:58Z

+// Functor used by FusedBatchNormGradOp to do the computations when is_training=False.
+// Both CPU and GPU will use this functor.
+template <typename Device, typename T>
+struct FusedBatchNormFreezeGrad {


Should this be inside functor namespace? Note this test failure: tensorflow/core/kernels/fused_batch_norm_op.cc:645:7: error: 'FusedBatchNormFreezeGrad' is not a member of 'tensorflow::functor'

at https://ci.tensorflow.org/job/tensorflow-pull-requests-cpu-python3/6452/console

Looks like this header is only included when GPU is enabled. I'll fix it.

Ah this is more complicated than I thought. Looks like I need to change BUILD file somehow.

…rflow#10857)

…inv var, etc

zhangyaobit · 2017-09-20T01:05:50Z

Jenkins, test this please.

drpngx · 2017-09-20T02:33:34Z

Jenkins, test this please.

zhangyaobit · 2017-09-21T23:49:43Z

Please merge this change. Thanks!

googlebot added the cla: yes label Aug 25, 2017

jhseu assigned zhangyaobit Aug 27, 2017

zhangyaobit assigned zheng-xq Aug 29, 2017

zhangyaobit reviewed Aug 29, 2017

View reviewed changes

zhangyaobit reviewed Aug 30, 2017

View reviewed changes

yifeif requested a review from zheng-xq September 8, 2017 00:10

yifeif added the awaiting review Pull request awaiting review label Sep 8, 2017

zhangyaobit reviewed Sep 19, 2017

View reviewed changes

ppwwyyxx added 5 commits September 19, 2017 14:45

Add kernels for FusedBatchNormGrad when is_training=False. (fix tenso…

b475519

…rflow#10857)

remove unnecessary headers

333fd69

Rename some variables

22791ea

Update comments about reserve_space of FusedBatchNormGrad

5e6bf5c

Comments and variable rename to avoid confusing of batch var/pop var/…

8144029

…inv var, etc

ppwwyyxx force-pushed the master branch from a90952c to 8144029 Compare September 19, 2017 21:46

fix build for CPU

9757156

(Hopefully) fix build for android.

f33ea38

zhangyaobit removed the request for review from zheng-xq September 21, 2017 23:46

zhangyaobit unassigned zheng-xq Sep 21, 2017

zhangyaobit approved these changes Sep 21, 2017

View reviewed changes

caisq merged commit 5ee3804 into tensorflow:master Sep 22, 2017

ozabluda mentioned this pull request Dec 6, 2017

acgan: Add batch normalization to the Generator, etc keras-team/keras#8616

Merged

csnemes2 mentioned this pull request Jan 5, 2018

Faster R-CNN training in TensorFlow < 1.4 ? tensorpack/tensorpack#578

Closed

Conversation

ppwwyyxx commented Aug 25, 2017

Uh oh!

tensorflow-jenkins commented Aug 25, 2017

Uh oh!

mention-bot commented Aug 25, 2017

Uh oh!

zhangyaobit left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ppwwyyxx Aug 29, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ppwwyyxx Aug 29, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

zhangyaobit Aug 29, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

zhangyaobit Aug 30, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

zhangyaobit left a comment

Choose a reason for hiding this comment

Uh oh!

zhangyaobit commented Aug 30, 2017

Uh oh!

ppwwyyxx commented Sep 11, 2017

Uh oh!

zhangyaobit commented Sep 12, 2017

Uh oh!

drpngx commented Sep 17, 2017

Uh oh!

drpngx commented Sep 17, 2017

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

zhangyaobit commented Sep 20, 2017

Uh oh!

drpngx commented Sep 20, 2017

ppwwyyxx Aug 29, 2017 •

edited

Loading

ppwwyyxx Aug 29, 2017 •

edited

Loading

zhangyaobit Aug 29, 2017 •

edited

Loading

zhangyaobit Aug 30, 2017 •

edited

Loading