Codestin Search App

mrTsjolder · 2018-11-15T17:23:50Z

From this discussion it appears that the linear layer should be initialised according to (LeCun et al., 1998). However, LeCun et al. promote a variance of 1/n, rather than a uniform distribution with bounds 1/n! Also biases are generally expected to be initialised with zeros.

ezyang · 2018-11-15T20:28:17Z

@soumith I'm not sure we can change the default initialization of linear, this might be too BC-breaking. Thoughts?

ezyang · 2018-12-06T05:46:06Z

I'm going to say that we are not going to change this. If you want to submit a doc patch suggesting the alternate initialization method, I'd be very happy to accept it.

ssnl · 2018-12-06T13:08:34Z

I’d say we should initialize bias to zeros. It is very standard and reasonable. We did a similar patch for BN recently.

ezyang · 2018-12-06T17:28:24Z

OK, reopening.

fmassa

I have mixed feelings about changing the initialization. This is a backward-incompatible change and people's code might suffer performance change, which will be hard to catch.

From [this discussion](https://discuss.pytorch.org/t/whats-the-default-initialization-methods-for-layers/3157/17) it appears that the linear layer should be initialised according to (LeCun et al., 1998). However, LeCun et al. promote a variance of `1/n`, rather than a uniform distribution with bounds `1/n`! Also biases are generally initialised to be zero.

kostmo · 2019-12-20T00:28:37Z

💊 CircleCI build failures summary and remediations

As of commit f76098a:

1/3 broken upstream at merge base 348d421 (see grid view)
You may want to rebase on the viable/strict branch (expand for instructions)

If your commit is newer than viable/strict, you can try basing on an older, stable commit:
```
git fetch viable/strict
git rebase --onto viable/strict $(git merge-base origin/master HEAD)
```
If your commit is older than viable/strict:
```
git fetch viable/strict
git rebase viable/strict
```
Check out the recency history of this "viable master" tracking branch.
2/3 failures introduced in this PR

Detailed failure analysis

One may explore the probable reasons each build failed interactively on the Dr. CI website.

🕵️ 2 new failures recognized by patterns

The following build failures don't appear to be due to upstream breakage:

caffe2_onnx_py2_gcc5_ubuntu16_04_test (1/2)

Step: "Test" (full log | pattern match details)

Dec 20 00:30:15 test/onnx/test_operators.py::TestOperators::test_linear FAILED [ 2%]

Dec 20 00:30:15 test/onnx/test_operators.py::TestOperators::test_gather_opset11 PASSED   [  2%] 
Dec 20 00:30:15 test/onnx/test_operators.py::TestOperators::test_ge PASSED               [  2%] 
Dec 20 00:30:15 test/onnx/test_operators.py::TestOperators::test_gelu PASSED             [  2%] 
Dec 20 00:30:15 test/onnx/test_operators.py::TestOperators::test_gt PASSED               [  2%] 
Dec 20 00:30:15 test/onnx/test_operators.py::TestOperators::test_hardtanh PASSED         [  2%] 
Dec 20 00:30:15 test/onnx/test_operators.py::TestOperators::test_implicit_expand PASSED  [  2%] 
Dec 20 00:30:15 test/onnx/test_operators.py::TestOperators::test_index PASSED            [  2%] 
Dec 20 00:30:15 test/onnx/test_operators.py::TestOperators::test_isnan PASSED            [  2%] 
Dec 20 00:30:15 test/onnx/test_operators.py::TestOperators::test_layer_norm_aten PASSED  [  2%] 
Dec 20 00:30:15 test/onnx/test_operators.py::TestOperators::test_le PASSED               [  2%] 
Dec 20 00:30:15 test/onnx/test_operators.py::TestOperators::test_linear FAILED           [  2%] 
Dec 20 00:30:15 test/onnx/test_operators.py::TestOperators::test_log_sigmoid PASSED      [  2%] 
Dec 20 00:30:15 test/onnx/test_operators.py::TestOperators::test_logsoftmax PASSED       [  2%] 
Dec 20 00:30:15 test/onnx/test_operators.py::TestOperators::test_lt PASSED               [  2%] 
Dec 20 00:30:15 test/onnx/test_operators.py::TestOperators::test_master_opset PASSED     [  2%] 
Dec 20 00:30:15 test/onnx/test_operators.py::TestOperators::test_max PASSED              [  2%] 
Dec 20 00:30:16 test/onnx/test_operators.py::TestOperators::test_maxpool PASSED          [  2%] 
Dec 20 00:30:16 test/onnx/test_operators.py::TestOperators::test_maxpool_dilations PASSED [  2%] 
Dec 20 00:30:16 test/onnx/test_operators.py::TestOperators::test_maxpool_indices PASSED  [  2%] 
Dec 20 00:30:16 test/onnx/test_operators.py::TestOperators::test_mean PASSED             [  3%] 
Dec 20 00:30:16 test/onnx/test_operators.py::TestOperators::test_mean_dtype PASSED       [  3%]

caffe2_onnx_py3_6_clang7_ubuntu16_04_test (2/2)

Step: "Test" (full log | pattern match details)

Dec 20 00:39:08 test/onnx/test_operators.py::TestOperators::test_linear FAILED [ 2%]

Dec 20 00:39:07 test/onnx/test_operators.py::TestOperators::test_gather_opset11 PASSED   [  2%] 
Dec 20 00:39:07 test/onnx/test_operators.py::TestOperators::test_ge PASSED               [  2%] 
Dec 20 00:39:07 test/onnx/test_operators.py::TestOperators::test_gelu PASSED             [  2%] 
Dec 20 00:39:08 test/onnx/test_operators.py::TestOperators::test_gt PASSED               [  2%] 
Dec 20 00:39:08 test/onnx/test_operators.py::TestOperators::test_hardtanh PASSED         [  2%] 
Dec 20 00:39:08 test/onnx/test_operators.py::TestOperators::test_implicit_expand PASSED  [  2%] 
Dec 20 00:39:08 test/onnx/test_operators.py::TestOperators::test_index PASSED            [  2%] 
Dec 20 00:39:08 test/onnx/test_operators.py::TestOperators::test_isnan PASSED            [  2%] 
Dec 20 00:39:08 test/onnx/test_operators.py::TestOperators::test_layer_norm_aten PASSED  [  2%] 
Dec 20 00:39:08 test/onnx/test_operators.py::TestOperators::test_le PASSED               [  2%] 
Dec 20 00:39:08 test/onnx/test_operators.py::TestOperators::test_linear FAILED           [  2%] 
Dec 20 00:39:08 test/onnx/test_operators.py::TestOperators::test_log_sigmoid PASSED      [  2%] 
Dec 20 00:39:08 test/onnx/test_operators.py::TestOperators::test_logsoftmax PASSED       [  2%] 
Dec 20 00:39:08 test/onnx/test_operators.py::TestOperators::test_lt PASSED               [  2%] 
Dec 20 00:39:08 test/onnx/test_operators.py::TestOperators::test_master_opset PASSED     [  2%] 
Dec 20 00:39:08 test/onnx/test_operators.py::TestOperators::test_max PASSED              [  2%] 
Dec 20 00:39:08 test/onnx/test_operators.py::TestOperators::test_maxpool PASSED          [  2%] 
Dec 20 00:39:08 test/onnx/test_operators.py::TestOperators::test_maxpool_dilations PASSED [  2%] 
Dec 20 00:39:08 test/onnx/test_operators.py::TestOperators::test_maxpool_indices PASSED  [  2%] 
Dec 20 00:39:08 test/onnx/test_operators.py::TestOperators::test_mean PASSED             [  3%] 
Dec 20 00:39:08 test/onnx/test_operators.py::TestOperators::test_mean_dtype PASSED       [  3%]

🚧 1 upstream failure recognized by patterns:

These builds matched patterns, but were probably caused by upstream breakages:

pytorch_linux_xenial_py3_clang5_asan_test

This comment was automatically generated by Dr. CI (expand for details).

Follow this link to opt-out of these comments for your Pull Requests.

Please report bugs/suggestions on the GitHub issue tracker.

This comment has been revised 2 times.

pytorchbot · 2022-04-12T01:34:56Z

Looks like this PR hasn't been updated in a while so we're going to go ahead and mark this as Stale.
Feel free to remove the Stale label if you feel this was a mistake.
If you are unable to remove the Stale label please contact a maintainer in order to do so.
Stale pull requests will automatically be closed 30 days after being marked Stale

github-actions · 2022-06-11T02:15:06Z

Looks like this PR hasn't been updated in a while so we're going to go ahead and mark this as Stale.
Feel free to remove the Stale label if you feel this was a mistake.
If you are unable to remove the Stale label please contact a maintainer in order to do so.
If you want the bot to never mark this PR stale again, add the no-stale label.
Stale pull requests will automatically be closed after 30 days of inactivity.

ezyang added the module: bc-breaking Related to a BC-breaking change label Nov 15, 2018

ezyang closed this Dec 6, 2018

ezyang reopened this Dec 6, 2018

zou3519 added the triage review label Dec 11, 2018

fmassa reviewed Jan 29, 2019

View reviewed changes

Comment thread torch/nn/modules/linear.py Outdated

Comment thread torch/nn/modules/linear.py Outdated

ezyang added the open source label Jun 5, 2019

mrTsjolder added 3 commits December 20, 2019 00:12

use kaiming uniform over custom uniform

8ef71f9

use zero initialiser directly

f76098a

mrTsjolder force-pushed the patch-1 branch from 3258734 to f76098a Compare December 19, 2019 23:13

mrTsjolder requested a review from apaszke as a code owner December 19, 2019 23:13

facebook-github-bot added the cla signed label Mar 10, 2021

pytorchbot added Stale and removed Stale labels Apr 12, 2022

github-actions Bot added the Stale label Jun 11, 2022

github-actions Bot closed this Jul 11, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

use correct initialisation in linear module#14034

use correct initialisation in linear module#14034
mrTsjolder wants to merge 3 commits into
pytorch:masterfrom
mrTsjolder:patch-1

mrTsjolder commented Nov 15, 2018

Uh oh!

ezyang commented Nov 15, 2018

Uh oh!

ezyang commented Dec 6, 2018

Uh oh!

ssnl commented Dec 6, 2018

Uh oh!

ezyang commented Dec 6, 2018

Uh oh!

fmassa left a comment

Uh oh!

Uh oh!

Uh oh!

kostmo commented Dec 20, 2019 •

edited

Loading

Uh oh!

pytorchbot commented Apr 12, 2022

Uh oh!

github-actions Bot commented Jun 11, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants

Uh oh!

Conversation

mrTsjolder commented Nov 15, 2018

Uh oh!

ezyang commented Nov 15, 2018

Uh oh!

ezyang commented Dec 6, 2018

Uh oh!

ssnl commented Dec 6, 2018

Uh oh!

ezyang commented Dec 6, 2018

Uh oh!

fmassa left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

kostmo commented Dec 20, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

💊 CircleCI build failures summary and remediations

Detailed failure analysis

🕵️ 2 new failures recognized by patterns

caffe2_onnx_py2_gcc5_ubuntu16_04_test (1/2)

caffe2_onnx_py3_6_clang7_ubuntu16_04_test (2/2)

🚧 1 upstream failure recognized by patterns:

Uh oh!

pytorchbot commented Apr 12, 2022

Uh oh!

github-actions Bot commented Jun 11, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants

kostmo commented Dec 20, 2019 •

edited

Loading