[ublas] Supporting DNNs with Tensors/Multidimensional Arrays

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

[ublas] Supporting DNNs with Tensors/Multidimensional Arrays

Boost - Dev mailing list
Gsoc <https://summerofcode.withgoogle.com/organizations/4507228564881408/>
2018 just ended one week ago and we had many successefully completed student
projects <https://github.com/BoostGSoC18>.

I was responsible for adding tensor support to Boost.uBLAS for primarily
supporting multilinear algebra operations in the field of numerics. The
wiki description along with the implementation can be found here
<https://github.com/BoostGSoC18/tensor>.

Similar to Boost.multi_array
<https://www.boost.org/doc/libs/1_68_0/libs/multi_array/doc/index.html> the
runtime-reshapable tensor data structure is parametrized in terms of number
of dimensions (rank/order), dimension extents, data type, layout (first-
and last-order) and storage type. The first two are runtime-variable. I am
also about to add subtensor (view/handle of a tensor) along with
multidimensional iterators for convenient algorithm implementation.

It is yet not as flexible as GSL's multi_span
<https://github.com/Microsoft/GSL/blob/master/include/gsl/multi_span>: does
not yet support static rank and dimensions. However, basic generic tensor
operations (contraction/transposition/reshaping/...), including a nice
syntax for Einstein's summation convention with placeholders, using C++17
features are provided. The operations are evaluated using expression
templates (not smart yet).

Similar to the tensor
<https://eigen.tuxfamily.org/dox/unsupported/group__CXX11__Tensor__Module.html>
framework of Eigen, that is used by tensor flow
<https://github.com/tensorflow/tensorflow>, the tensor data structure in
Boost.uBlas could be taken for implementing deep neural networks or
higher-order statistics I think. I am not sure if the C++ community would
appreciate if Boost has some form of basic operations for building *deep
neural networks* (DNNs). I would like to ask

1. if it make sense for boost to support basic operations for DNNs?
2. what are the obligatory, necessary basic operations for creating DNN
building blocks?
3. if there are any additional data structure parameters that needs to be
added for (efficiently) supporting DNNs?

Best,
Cem

_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Reply | Threaded
Open this post in threaded view
|

Supporting DNNs with Tensors/Multidimensional Arrays

Boost - Dev mailing list
Gsoc <https://summerofcode.withgoogle.com/organizations/4507228564881408/>
2018 just ended one week ago and we had many successefully completed student
projects <https://github.com/BoostGSoC18>.

I was responsible for adding tensor support to Boost.uBLAS for primarily
supporting multilinear algebra operations in the field of numerics. The
wiki description along with the implementation can be found here
<https://github.com/BoostGSoC18/tensor>.

Similar to Boost.multi_array
<https://www.boost.org/doc/libs/1_68_0/libs/multi_array/doc/index.html> the
runtime-reshapable tensor data structure is parametrized in terms of number
of dimensions (rank/order), dimension extents, data type, layout (first-
and last-order) and storage type. The first two are runtime-variable. I am
also about to add subtensor (view/handle of a tensor) along with
multidimensional iterators for convenient algorithm implementation.

It is yet not as flexible as GSL's multi_span
<https://github.com/Microsoft/GSL/blob/master/include/gsl/multi_span>: does
not yet support static rank and dimensions. However, basic generic tensor
operations (contraction/transposition/reshaping/...), including a nice
syntax for Einstein's summation convention with placeholders, using C++17
features are provided. The operations are evaluated using expression
templates (not smart yet).

Similar to the tensor
<https://eigen.tuxfamily.org/dox/unsupported/group__CXX11__Tensor__Module.html>
framework of Eigen, that is used by tensor flow
<https://github.com/tensorflow/tensorflow>, the tensor data structure in
Boost.uBlas could be taken for implementing deep neural networks or
higher-order statistics I think. I am not sure if the C++ community would
appreciate if Boost has some form of basic operations for building *deep
neural networks* (DNNs). I would like to ask

1. if it make sense for boost to support basic operations for DNNs?
2. what are the obligatory, necessary basic operations for creating DNN
building blocks?
3. if there are any additional data structure parameters that needs to be
added for (efficiently) supporting DNNs?


Cem

_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Reply | Threaded
Open this post in threaded view
|

Re: Supporting DNNs with Tensors/Multidimensional Arrays

Boost - Dev mailing list
In reply to this post by Boost - Dev mailing list
On 08/29/18 15:47, Cem Bassoy via Boost wrote:

> 2. what are the obligatory, necessary basic operations for creating DNN
> building blocks?

You may want to investigate Automatic Differentiation, which is a
building block that extends to many use cases besides DNN.

A good starting point is this talk by Conal Elliott:

   https://www.youtube.com/watch?v=ne99laPUxN4

_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Reply | Threaded
Open this post in threaded view
|

Re: Supporting DNNs with Tensors/Multidimensional Arrays

Boost - Dev mailing list
Am Do., 30. Aug. 2018 um 20:17 Uhr schrieb Bjorn Reese via Boost <
[hidden email]>:

> On 08/29/18 15:47, Cem Bassoy via Boost wrote:
>
> > 2. what are the obligatory, necessary basic operations for creating DNN
> > building blocks?
>
> You may want to investigate Automatic Differentiation, which is a
> building block that extends to many use cases besides DNN.
>
>
I am not really sure if. I have skimmed his paper
https://arxiv.org/pdf/1804.00746.pdf. I need more concrete things, I guess
before reasoning about DNNs. I thought to implement some Eigen::Tensor
Operations such as Broadcast(), Convolution(), etc.

However, I need some input from the boost community in what direction we
want to go and if we want to enhance the support for multidimensional
arrays, tensors, etc. We are now behind Eigen::Tensor and some other
libraries!

Cheers
C

_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Reply | Threaded
Open this post in threaded view
|

Re: [ublas] Supporting DNNs with Tensors/Multidimensional Arrays

Boost - Dev mailing list
In reply to this post by Boost - Dev mailing list
>
>
> 1. if it make sense for boost to support basic operations for DNNs?
> 2. what are the obligatory, necessary basic operations for creating DNN
> building blocks?
> 3. if there are any additional data structure parameters that needs to be
> added for (efficiently) supporting DNNs?
>

DNN are certainly interesting models in machine learning, but they
represent a very small part of what we can do and what really works in real
life.
Tensors can be applied to many more situations. On top of it, tensors can
be applied to many other field of science.

What would be interesting is to start thinking more generically and not
focus too much on just DNN, especially if we want to have machine learning
in ublas (which was another successful GSOC this year by the way).

I'm glad to see we have tensors, I'm happy to see we have basics stats and
ML.
Let's go to the next step and have some more numerical techniques related
to linear algebra in ublas.

Open discussion now .....


David

_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Reply | Threaded
Open this post in threaded view
|

Re: [ublas] Supporting DNNs with Tensors/Multidimensional Arrays

Boost - Dev mailing list
I totally agree. DNNs are just one part. Lets put in more numerical
algorithms using tensors, matrices and vectors.  However, IMHO I think that
the community needs better documentation, interfaces, (possibly
implementation) of basic tensor, matrix and vector operations. I am now
also changing the interface of the tensor data structure for being able to
be an alias for matrix and vector.

Am Sa., 1. Sep. 2018 um 11:54 Uhr schrieb David Bellot <
[hidden email]>:

>
>> 1. if it make sense for boost to support basic operations for DNNs?
>> 2. what are the obligatory, necessary basic operations for creating DNN
>> building blocks?
>> 3. if there are any additional data structure parameters that needs to be
>> added for (efficiently) supporting DNNs?
>>
>
> DNN are certainly interesting models in machine learning, but they
> represent a very small part of what we can do and what really works in real
> life.
> Tensors can be applied to many more situations. On top of it, tensors can
> be applied to many other field of science.
>
> What would be interesting is to start thinking more generically and not
> focus too much on just DNN, especially if we want to have machine learning
> in ublas (which was another successful GSOC this year by the way).
>
> I'm glad to see we have tensors, I'm happy to see we have basics stats and
> ML.
> Let's go to the next step and have some more numerical techniques related
> to linear algebra in ublas.
>
> Open discussion now .....
>
>
> David
>

_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost