add Tensor class #400

edgarriba · 2016-11-12T17:37:07Z

Add the Tensor structure:

data hold by std::vector<> with 64bytes alignment
three different data accessors: t.ptr<float_t>(), t.at<float_t>() and t[idx]
basic reshape() and resize() routines
basic toDevice() and fromDevice() routines
implement element-wise add, sub, mul, div

beru · 2016-11-12T19:03:25Z

tiny_dnn/core/framework/tensor.h

+        }
+
+        U* value = &host_data_->at(shape_[1] * shape_[2] *
+                ( shape_[3] * batch + depth ) + height + width);


Is it really OK to calculate data position in this way?

if you unroll the formula you will see that corresponds to access data with following shape: NxWxHxD

beru · 2016-11-12T19:04:31Z

tiny_dnn/core/framework/tensor.h

+    void init_data(const cnn_size_t batch,  const cnn_size_t width,
+                   const cnn_size_t height, const cnn_size_t depth) {
+        host_data_ = data_ptr(new std::vector<U>(
+            batch * width * height * depth, float_t(0.0)));


Everyone loves bacon and C-style casts.

what do you mean here? probably this method could be public and implemented a resize routine

beru · 2016-11-12T19:05:51Z

tiny_dnn/core/framework/tensor.h

+                    const cnn_size_t height, const cnn_size_t depth) {
+       shape_[0] = batch;  shape_[1] = width;
+       shape_[2] = height; shape_[3] = depth;
+    }


[0] : batch
[1] : width
[2] : height
[3] : depth
If so, what about using enum?

how would you do it?

beru · 2016-11-12T19:14:22Z

@edgarriba
I checked the Tensor class.
I'm not sure why Tensor class has b,w,h,d template parameters since they should be dynamic and not compile-time fixed-size. Can you please tell me the reason?
What about adding getter methods to return batch, width, height, depth?

beru · 2016-11-13T13:54:24Z

tiny_dnn/core/framework/tensor.h

+    // Checked version (throw exceptions for out-of-range error)
+    template<typename T>
+    T& at(const cnn_size_t batch,
+          const cnn_size_t width,


I think width parameter should be renamed to x.

beru · 2016-11-13T13:54:52Z

tiny_dnn/core/framework/tensor.h

+    template<typename T>
+    T& at(const cnn_size_t batch,
+          const cnn_size_t width,
+          const cnn_size_t height,


I think height argument should be renamed to y.

beru · 2016-11-13T13:55:27Z

tiny_dnn/core/framework/tensor.h

+
+    template<typename T>
+    const T& at(const cnn_size_t batch,
+                const cnn_size_t width,


In my opinion, width parameter should be renamed to x.

beru · 2016-11-13T13:56:20Z

tiny_dnn/core/framework/tensor.h

+    template<typename T>
+    const T& at(const cnn_size_t batch,
+                const cnn_size_t width,
+                const cnn_size_t height,


From my perspective, height parameter should be renamed to y in here.

beru · 2016-11-13T13:57:36Z

tiny_dnn/core/framework/tensor.h

+    T* ptr(const cnn_size_t batch,
+           const cnn_size_t width,
+           const cnn_size_t height,
+           const cnn_size_t depth) {


I propose to rename argument names width to x and height to y in here.

beru · 2016-11-13T13:59:06Z

tiny_dnn/core/framework/tensor.h

+    template<typename T>
+    const T* ptr(const cnn_size_t batch,
+                 const cnn_size_t width,
+                 const cnn_size_t height,


Please rename width to x and height to y.

beru · 2016-11-13T14:01:27Z

tiny_dnn/core/framework/tensor.h

+    // zero-overhead version (same performance to raw pointer access.
+    // have an assertion for out-of-range error)
+    template<typename T>
+    T& operator[] (cnn_size_t index) {


Please add assert statement to check index parameter.

beru · 2016-11-13T14:01:43Z

tiny_dnn/core/framework/tensor.h

+    }
+
+    template<typename T>
+    const T& operator[] (cnn_size_t index) const {


Please add assert statement to check index parameter.

beru · 2016-11-13T14:03:46Z

tiny_dnn/core/framework/tensor.h

+    T* access_data(const cnn_size_t batch,
+                   const cnn_size_t width,
+                   const cnn_size_t height,
+                   const cnn_size_t depth) const {


Renaming width to x and height to y would be good.

beru · 2016-11-13T14:09:51Z

tiny_dnn/core/framework/tensor.h

+                   const cnn_size_t height,
+                   const cnn_size_t depth) const {
+        if (batch  > shape_[0] || width > shape_[1] ||
+            height > shape_[2] || depth > shape_[3]) {


Instead of using shape_[magic_number_index], what about using getter methods such as batch(), width(), height(), depth()?

if (batch >= batch() || x >= width() || y >= height() || depth >= depth() ) { nn_error("Fool! You have just ensured the doom of this world."); }

batch or N? width or w or x? ...

edgarriba · 2016-11-13T22:04:20Z

@beru
regarding template parameters you are right, I was just playing with templates :-)

Besides, what do you think about the data container ? E.g. in Caffe2 they use void* to store it maybe to have a more general case? https://github.com/caffe2/caffe2/blob/master/caffe2/core/tensor.h#L538

beru · 2016-11-14T22:39:16Z

tiny_dnn/core/framework/tensor.h

+                   const cnn_size_t d2,
+                   const cnn_size_t d3) const {
+        if (d0 > shape_[0] || d1 > shape_[1] ||
+            d2 > shape_[2] || d3 > shape_[3]) {


I think comparison operators should be >= instead of >.

beru · 2016-11-14T22:48:27Z

tiny_dnn/core/framework/tensor.h

+    void resize(const U value = 0) {
+        if (!host_data_) {
+            host_data_ = std::unique_ptr<std::vector<U> >(
+                new std::vector<U>(size(), value));


Probablly, you can use std::make_unique in here.

@beru make_unque is not supported until c++14
http://stackoverflow.com/questions/17902405/how-to-implement-make-unique-function-in-c11

Workaround solution: 32e589c

beru · 2016-11-14T22:54:45Z

tiny_dnn/core/framework/tensor.h

+    }
+
+    // Returns the tensor shape
+    std::vector<cnn_size_t> shape() const { return shape_; }


I guess const std::vector<cnn_size_t>& shape() const { return shape_; } would be better.
But I'm wondering type of shape_ needs to be std::vector container..
An alternative type is std::array<cnn_size_t, 4>.

if we return here a reference it means that a user could "reshape" the tensor just modifying this vector but we also need to resize and copy data in case that we want a real reshape. Do we want this feature now?

If you return const reference, caller cannot modify the returned vector without using const_cast.
Besides, if you return vector container instance by value, another heap allocation and copy occurs inside (without return value optimization).

beru · 2016-11-14T22:57:36Z

tiny_dnn/core/framework/tensor.h

+    T& at(const cnn_size_t d0,
+          const cnn_size_t d1,
+          const cnn_size_t d2,
+          const cnn_size_t d3) {


Are const type qualifiers for arguments really absolutely necessary?

@beru not really needed since we are passing arguments by value. However, for sanity and readability/maintenance I think it's a good practice to mark non-mutable arguments.

Randl · 2016-11-16T07:24:05Z

tiny_dnn/core/framework/tensor.h

+    }
+
+    // Move constructor
+    Tensor(Tensor<U>&& other) = default;


VS2013 compiler doesn't support defaulted move constructors. https://stackoverflow.com/questions/24573963/move-constructor-invalid-type-for-defaulted-constructor-vs-2013

Randl · 2016-11-16T08:58:16Z

tiny_dnn/core/framework/tensor.h

+     *
+     *  Data will be hold by a std::vector with 64bytes alignment.
+     */
+    explicit Tensor(const cnn_size_t d0,


You should use size_t here. cc @nyanp

ok, but before that: are we going to support tensors serialization? (could be cool this feature :D) If so, I think that cnn_size_t a.k.a std::uint32_t should be okay. Otherwise, I'll move to standard size_t.

Randl · 2016-11-16T15:22:20Z

Looks like VS somewhere tries to copy unique_ptr:

C:\Program Files (x86)\Microsoft Visual Studio 12.0\VC\include\xstddef(316): error C2280: 'std::unique_ptr<std::vector<tiny_dnn::float_t,tiny_dnn::aligned_allocator<tiny_dnn::float_t,0x040>>,std::default_delete<_Ty>>::unique_ptr(const std::unique_ptr<_Ty,std::default_delete<_Ty>> &)' : attempting to reference a deleted function [C:\projects\build\test\tiny_dnn_test.vcxproj]          with          [              _Ty=std::vector<tiny_dnn::float_t,tiny_dnn::aligned_allocator<tiny_dnn::float_t,0x040>>          ] (C:\projects\tiny-dnn\test\test.cpp)          C:\Program Files (x86)\Microsoft Visual Studio 12.0\VC\include\memory(1486) : see declaration of 'std::unique_ptr<std::vector<tiny_dnn::float_t,tiny_dnn::aligned_allocator<tiny_dnn::float_t,0x040>>,std::default_delete<_Ty>>::unique_ptr'          with          [              _Ty=std::vector<tiny_dnn::float_t,tiny_dnn::aligned_allocator<tiny_dnn::float_t,0x040>>          ]          This diagnostic occurred in the compiler generated function 'tiny_dnn::Tensor<tiny_dnn::float_t>::Tensor(const tiny_dnn::Tensor<tiny_dnn::float_t> &)'

Randl · 2016-11-16T15:23:52Z

tiny_dnn/core/framework/tensor.h

+        for_i(true, res.size(), [&](size_t i) {
+            const U tmp = src[i];
+            res[i] = tmp == U(0.0) ? std::numeric_limits<U>::quiet_NaN() :
+                this->operator[](i) / (tmp + std::numeric_limits<U>::min());


Now you don't need addition

Looks like VS somewhere tries to copy unique_ptr:

maybe because the destructor is missing ? I don't have a MS environment to test :(

Now you don't need addition

thx!

Adding a destructor might not suffice. If you add a destructor, copy constructor and copy assignment are still generated (for backward compatibility with pre-C++11).

See here: http://stackoverflow.com/questions/3734247/what-are-all-the-member-functions-created-by-compiler-for-a-class-does-that-hap

In your case, if unique_ptr is a member, when a copy (constructor|assignment) is generated, it will invoke the copy (constructor|assignment) from unique_ptr, resulting in an error (since unique_optr is uncopiable).

The only way is to implement your custom copy constructor and copy assignment, or delete them if you want to make your class uncopiable.

Take a look at https://katyscode.wordpress.com/2012/10/04/c11-using-stdunique_ptr-as-a-class-member-initialization-move-semantics-and-custom-deleters/

Randl · 2016-11-16T17:13:23Z

on first glance the problem is that VS tries to copy unique pointer in copy constructor

Randl · 2016-11-16T19:15:27Z

Custom copy constructor seems necessary anyway. @edgarriba @bhack @nyanp

bhack · 2016-11-16T19:38:44Z

@Randl see https://accu.org/index.php/journals/1896

Randl · 2016-11-16T19:42:22Z

@bhack Correct me if I'm wrong, but enforcing rule of zero together with unique_ptr results in non-copyable class, and that's not what we want

bhack · 2016-11-16T19:44:11Z

Yes exactly: "By using a unique_ptr we make our class non-copyable/assignable and identical in behavior to the previous examples."
And we need to avoid this here.

Randl · 2016-11-16T19:45:43Z

You can still make a custom copy constructor which will work the unique_ptr. That's what I suggested

bhack · 2016-11-16T19:52:06Z

Yes was a documentation enforcing this. And also http://stackoverflow.com/questions/29124592/copy-constructor-to-transfer-ownership-of-a-unique-ptr

bhack · 2016-11-17T00:07:31Z

/off topic/ @keryell Any news from the United Nations conference on "Bringing About HPC Open-Standards World Peace" ;)

bhack · 2016-11-17T01:28:46Z

/cc @gfursin We are designing Tensor here before integrating GPU and then go to CK. If you have any design advertise you are welcome.

edgarriba · 2016-11-20T08:45:46Z

I close this. You can find the valid pull request in #41

bhack · 2016-11-20T11:35:11Z

The new tensor PR is #411

beru · 2016-11-23T05:06:32Z

bhack wrote:
@beru As you see R3 in the description section point to P0367 implemented at https://github.com/keryell/accessors

@bhack Thank you for specifying the proposal for C++ standard.
Even if it can survive under a fluid situation and will be adopted into the standard, it will may take a few years so I guess we need to build or find different one.

keryell · 2016-11-23T14:25:59Z

@beru: array_ref was well received. Probably for C++20, but anyway there is a special department in the C++ committee dedicated to bike-sheding, so it will be probably slightly different... :-)

@bhack : stop asking off-topic questions all over GitHub threads :-) But yes, everybody survived to this SC16 panel...

bhack · 2016-11-23T14:38:18Z

@keryell We are living in a too covered tecnological roadmaps era that create fragmentation. HPC standard domain it is only part of this trend. Probably after C++20 all this discussions will be so useless like discussing an if statement. :)

edgarriba added 5 commits November 12, 2016 17:18

add Tensor class for CPU mode

abedd3b

move data pointer to unique_ptr and fix data accessors

91bd4fa

transfer tensor data to device

72faaab

rename members, add fromDevice() and accessors methods

c159c48

convert to templated class

342c88c

beru reviewed Nov 12, 2016

View reviewed changes

beru reviewed Nov 13, 2016

View reviewed changes

bhack mentioned this pull request Nov 13, 2016

Tensor class design #388

Closed

bhack mentioned this pull request Nov 13, 2016

Typo fix #404

Merged

edgarriba force-pushed the tensor branch 2 times, most recently from acffaa0 to 146ae6f Compare November 14, 2016 16:36

refactor constructor and minor changes

2c96d60

edgarriba force-pushed the tensor branch from 146ae6f to 2c96d60 Compare November 14, 2016 16:38

beru reviewed Nov 14, 2016

View reviewed changes

Randl reviewed Nov 16, 2016

View reviewed changes

edgarriba added 6 commits November 16, 2016 12:36

disable move constructor for VS2013 compilers

2064868

fix typo errors in tests

a346686

fix tensor operations tests to avoid fp related problems

42fc7e0

add more tests to handle tensor shape mismatch

842c005

fix zero division issue

3c7d048

fix sqrt error for negative numbers

1d66faf

Randl reviewed Nov 16, 2016

View reviewed changes

edgarriba added 3 commits November 16, 2016 17:03

add destructor,copy ctors, and code cleaning

f9f6c57

remove small value sum in div

7898bdd

fix tests precision

40429b6

workaround solution for unique_ptr in move/copy ctors

1536028

pansk mentioned this pull request Nov 17, 2016

Tensor (reprise) #411

Merged

edgarriba closed this Nov 20, 2016

add Tensor class #400

add Tensor class #400

Uh oh!

Conversation

edgarriba commented Nov 12, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

beru commented Nov 12, 2016

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

beru Nov 13, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

beru Nov 13, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

beru Nov 13, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

edgarriba commented Nov 13, 2016

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Randl commented Nov 16, 2016

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

edgarriba commented Nov 12, 2016 •

edited

Loading

beru Nov 13, 2016 •

edited

Loading

beru Nov 13, 2016 •

edited

Loading

beru Nov 13, 2016 •

edited

Loading

bhack Nov 16, 2016 •

edited

Loading