Cholesky space #1142

SaitejaUtpala · 2021-09-07T05:12:27Z

This PR implements following things

vector space of lower triangular matrices
- geometry
- tests
cholesky space : manifold of lower triangular matrices with positive diagonal elements (as open submanifold of lower triangular matrices)
- geometry
- tests
Riemannian metric on choleksy space
- metric
- tests

nguigs · 2021-09-07T08:49:58Z

Hi Saiteja!
Great input! In fact the space of lower triangular matrices with positive diag (LT+) is a Lie group, and the vector space of LT matrices is its Lie algebra, so it should be implemented with the additional structure.
What metrics did you have in mind for LT+ ?
It can be embedded with left-invariant metrics, and there is one that coincides with the affine-invariant one on SPD. I have code that checks this.

SaitejaUtpala · 2021-09-07T10:54:13Z

Hi Saiteja!
Great input! In fact the space of lower triangular matrices with positive diag (LT+) is a Lie group, and the vector space of LT matrices is its Lie algebra, so it should be implemented with the additional structure.
What metrics did you have in mind for LT+ ?
It can be embedded with left-invariant metrics, and there is one that coincides with the affine-invariant one on SPD. I have code that checks this.

Hey @nguigs . yup LT+ is a lie group. The metric is constructed by imposing different kinds of inner products on strictly lower part and diagonal elements (this metric doesn't have name). this is actually bi-invariant metric

Then this metric is push forwarded to SPD space, this is called Log Cholesky metric . Log cholesky metric has lots of nice properties and is kind of best of two worlds (Log euclidean, Affine Invariant ). It has closed form expression for Frechet mean and easily computable expressions for parallel trasnport, R-log , R-exp maps . So, I don't think this is equivalent to Affine Invariant Metric?

here is the paper : https://arxiv.org/abs/1908.09326

nguigs · 2021-09-07T13:25:09Z

Then this metric is push forwarded to SPD space, this is called Log Cholesky metric . Log cholesky metric has lots of nice properties and is kind of best of two worlds (Log euclidean, Affine Invariant ). It has closed form expression for Frechet mean and easily computable expressions for parallel trasnport, R-log , R-exp maps . So, I don't think this is equivalent to Affine Invariant Metric?

Only the left-invariant metric with 2 on the diagonal and 1 on other coefficients is equivalent to AI metric when pushed to SPD. The others are different. This is good to test the implementation

ninamiolane · 2021-09-10T19:29:33Z

Wooohoo, thanks for this! As @nguigs mentioned, we should have Cholesky inherits from LieGroup too.

On a minor note, maybe check your docstrings, there are periods missing at the end of descriptions here and there, e.g.:

    @classmethod
    def differential_gram(cls, tanget_vec, base_point):
        """Compute gram matrix of rows  
        Gram_matrix is mapping from point to point.point^{T}. 
        This is diffeomorphism between cholesky space and spd manifold

and at other places.

Otherwise this is awesome, can't wait for the tests.

nguigs

Good start!
I think we should discuss the naming though. Firstly, I'm not sure the naming Cholesky space is justified and well known enough, maybe we should stick to something more informative (PositiveLowerTriangular?).
This is even more the case for the metrics. I think the one you are implementing is called Log-Cholesky, and we should be precise because we can implement other ones such as Euclidean-Cholesky, LogEuclidean-Cholesky, and invariant metrics.
@ythanwerdas what do you think?

nguigs · 2021-09-14T09:07:42Z

geomstats/geometry/matrices.py

-    @classmethod
-    def is_diagonal(cls, mat, atol=gs.atol):
-        """Check if a matrix is square and diagonal.
-
-        Parameters
-        ----------
-        mat : array-like, shape=[..., n, n]
-            Matrix.
-        atol : float
-            Absolute tolerance.
-            Optional, default: backend atol.
-
-        Returns
-        -------
-        is_diagonal : array-like, shape=[...,]
-            Boolean evaluating if the matrix is square and diagonal.
-        """
-        is_square = cls.is_square(mat)
-        if not gs.all(is_square):
-            return False
-        diagonal_mat = from_vector_to_diagonal_matrix(cls.diagonal(mat))
-        is_diagonal = gs.all(
-            gs.isclose(mat, diagonal_mat, atol=atol), axis=(-2, -1))
-        return is_diagonal


Why does this appear in this PR?

nguigs · 2021-09-14T09:10:00Z

geomstats/geometry/cholesky.py

+        vec_diag = gs.exp(gs.diagonal(point, axis1 = -2, axis2 = -1))
+        diag = gs.vec_to_diag(vec_diag)
+        strictly_lower_triangular = Matrices.to_lower_triangular(point)
+        projection = diag + strictly_lower_triangular
+        return projection


Ideally, the projection should minimize the Frobenius norm. This is tricky as we have an open set here, but one property that must be verified for a projection function p is p(p(x))=p(x) for all x, i.e. p(x)=x if x is already in the manifold.
This is not the case here, so maybe only do the projection if point does not already belong to LT+, or use epsilon to floor negative values of the diagonal.

@nguigs nice catch! I forgot we should we have p(p(x))=x. I am using abs to make sure this happens

nguigs · 2021-09-14T09:14:43Z

geomstats/geometry/cholesky.py

+        ip_diagonal = gs.einsum("...ii,...ii ,...i->...", 
+            tangent_vec_a, tangent_vec_b, inv_sqrt_diagonal)


I think our torch and tf einsum don't work with 3 inputs, but you can use tangent_vec_b * inv_sqrt_diagonal or two lines instead.

nguigs · 2021-09-14T09:36:55Z

Maybe the Riemannian metric on SPD could be a PullBack metric @ninamiolane ?

ythanwerdas · 2021-09-14T09:56:01Z

Dear all, let me give you an overview of Riemannian metrics on SPD matrices based on the Cholesky decomposition to help you decide the names.

First of all, Saiteja and Nicolas don't speak about the same Lie group structure. For Nicolas, the internal law is the matrix multiplication. For Saiteja, following Zhenhua Lin, the Lie group structure is a direct product of Lie groups: additive on the strict lower triangular matrix and multiplicative on the diagonal. By the way, it is even a vector space structure. Otherwise said, there is a diffeomorphism from LT^+(n) to the vector space LT(n) which is identity on the off-diagonal part and logarithm on the diagonal part.

I am preparing a paper on left-invariant metrics on the multiplicative Lie group LT^+(n), I will update this comment with the preprint as soon as it is available. I call Lie-Cholesky metrics the pullbacks of these metrics onto SPD matrices.

Zhenhua Lin calls log-Cholesky the metric Saiteja is implementing in this PR. Some people use directly the Euclidean metric on LT^+(n) and the pullback onto SPD matrices has been called "Cholesky metric" (see chapter 3 of Riemannian Geometric Statistics in Medical Image Analysis). It could be called Euclidean-Cholesky to avoid confusions. Another vector space parameterization is given by the matrix logarithm which sends diffeomorphically LT^+(n) onto LT(n) (see this paper). This is also geodesically complete of course. The pullback onto SPD matrices could be called log-Euclidean-Cholesky to avoid confusions again.

I don't have opinion on the naming of the space, Cholesky or PositiveLowerTriangular have different advantages.

One last comment if I may, from a modelling point of view, I wouldn't say that log-Cholesky is better than log-Euclidean and affine-invariant since it is not invariant under orthogonal transformations and not even under permutations. It means that changing the order of the axes (representing your signals for example if your SPD matrices are covariance matrices) would change the statistical analysis. This property depends on the application but in many cases, we would like the analysis to be independent under orthogonal transformations (at least).

ninamiolane · 2021-09-14T16:24:40Z

Yes, this metric should inherit from the PullbackMetric, while still overwriting most methods: if we can use closed forms instead of automatic differentiation the results will be better.

I agree with having LowerTriangularMatrices and LowerTriangularPositiveMatrices, even if it is long: it is more self-explanatory and I am indeed not sure that everybody knows the term CholeskySpace.

Thanks @ythanwerdas for the inputs on the classification of metrics for SPD matrices 🙌

ninamiolane

Very nice, I think we will be able to merge this great PR today 🎉 High-level comments:

I have important comments/questions on the structure of the algebraic tools in the backends, see inlines.
Some tests do not include tests on minibatch, aka _vectorization tests
DeepSource is failing (there's not much to do, though)
Can you make a pass on all the docstrings?
- Some docstrings are not right (I believed you copy pasted something but then didn't edit it)
- Some words have uppercase in the docstrings:
  - cholesky -> Cholesky
  - spd -> SPD
- Some "." are missing here and there.

Thank you very much for this!

geomstats/_backend/autograd/__init__.py

geomstats/_backend/numpy/__init__.py

geomstats/_backend/pytorch/linalg.py

geomstats/_backend/tensorflow/__init__.py

geomstats/geometry/lower_triangular_matrices.py

tests/tests_geomstats/test_lower_triangular_matrices.py

tests/tests_geomstats/test_matrices.py

tests/tests_geomstats/test_positive_lower_triangular_matrices.py

ninamiolane · 2021-12-10T19:37:16Z

geomstats/geometry/positive_lower_triangular_matrices.py

+
+    @classmethod
+    def inner_product(cls, tangent_vec_a, tangent_vec_b, base_point):
+        """Compute the inner product using only strictly lower triangular elements.


Adapt the docstring

geomstats/geometry/positive_lower_triangular_matrices.py

ninamiolane

Almost there! A few cosmetic comments to address (e.g. see my previous code review comments on the docstrings' cleanup).

Will be merged very soon! 🎉

geomstats/_backend/pytorch/__init__.py

geomstats/_backend/pytorch/linalg.py

ninamiolane · 2021-12-12T14:09:34Z

geomstats/_backend/tensorflow/__init__.py

    return tf.tile(x_reshape, multiples)


+def vec_to_diag(vec):


Put on top of the file as, see:

geomstats/geomstats/_backend/tensorflow/__init__.py

Line 74 in 1b2c2cc

geomstats/geometry/matrices.py

geomstats/geometry/positive_lower_triangular_matrices.py

ninamiolane · 2021-12-12T14:13:39Z

geomstats/geometry/positive_lower_triangular_matrices.py

+
+    @classmethod
+    def inner_product(cls, tangent_vec_a, tangent_vec_b, base_point):
+        """Compute the inner product using only strictly lower triangular elements.


(the copy-pasting has had a problem)

ninamiolane · 2021-12-12T14:14:41Z

tests/tests_geomstats/test_positive_lower_triangular_matrices.py

@@ -0,0 +1,291 @@
+"""Unit tests for Positive lower  triangular matrices"""


and double space between lower and triangular

tests/tests_geomstats/test_positive_lower_triangular_matrices.py

ninamiolane

Moving to "request changes" to clean up docstrings and address other minor comments. Excellent work, looking forward to merge.

SaitejaUtpala · 2021-12-16T20:43:27Z

@ninamiolane ready to merge. deep source is complaining about test files. that is fine though.

ninamiolane

@SaitejaUtpala there are still some comments to be addressed + can you look at how this integrates with the PR #1232 merged two days ago? Someone needed triu, and we didn't want to wait until the Cholesky PR is merged thus it got merged. Thank you very much!

geomstats/_backend/tensorflow/__init__.py

ninamiolane · 2021-12-17T07:25:57Z

geomstats/geometry/lower_triangular_matrices.py

+        return gs.tril_to_vec(mat)
+
+    def projection(self, point):
+        """Make a square matrix lower triangular.


@SaitejaUtpala

ninamiolane · 2021-12-17T07:26:05Z

geomstats/geometry/matrices.py

+        Returns
+        -------
+        is_triu : array-like, shape=[...,]
+            Boolean evaluating if the matrix is upper triangular


@SaitejaUtpala

ninamiolane · 2021-12-17T07:26:28Z

geomstats/geometry/positive_lower_triangular_matrices.py

+
+    @staticmethod
+    def gram(point):
+        """Compute gram matrix of rows


@SaitejaUtpala

ninamiolane

Fantastic, thank you very much!

SaitejaUtpala added 3 commits September 7, 2021 10:22

first commit

99129e9

add aux methods

6140e6a

clean

e1bb569

SaitejaUtpala marked this pull request as draft September 7, 2021 05:14

SaitejaUtpala added 3 commits September 7, 2021 10:58

projection methods

c5ac1df

to from vecs

980335c

tf tril_to_vec

6755c6f

SaitejaUtpala added 2 commits September 7, 2021 15:59

add cholesky factor

6372c49

cleaning

c5d2417

testing

995b485

SaitejaUtpala added 7 commits September 9, 2021 12:25

cholesky metric

878e476

metric

7e308c6

R-exp and R-log

cbf57b0

differential

ed89a0a

gram and its differential

1ea9273

comments

4e1345b

inv differential

2782f2a

SaitejaUtpala added 3 commits September 12, 2021 19:19

some tests

981ee1b

tests

7c6cabb

tests for lwt

9d5136a

nguigs reviewed Sep 14, 2021

View reviewed changes

SaitejaUtpala added 2 commits September 25, 2021 21:08

squared_dist for cholesky

ed8da9c

resolve conflicts

8393bcf

SaitejaUtpala added 6 commits December 10, 2021 21:45

typo fix

e0501dc

fix typo again

d18d8b8

make it work for tf

448490d

cumprod fix

a88bcaf

reshape

91906ea

fix deepsource

1b2c2cc

ninamiolane reviewed Dec 10, 2021

View reviewed changes

SaitejaUtpala added 2 commits December 11, 2021 01:18

chol

5c02cc9

some changes

5c1be75

ninamiolane reviewed Dec 12, 2021

View reviewed changes

ninamiolane requested changes Dec 12, 2021

View reviewed changes

nguigs mentioned this pull request Dec 13, 2021

Add 3D heisenberg group in vector representation and unit tests #1213

Merged

SaitejaUtpala added 12 commits December 16, 2021 23:07

resolve conflicts

79183c1

add dots

2acf22b

address pr review comments-1

bb0d188

fix

8aff1a1

change to tests

dbb49ba

cholesky

72924eb

checking

f1883f0

cleaning

6007984

testing

175f107

testing

1411e5d

test even more

812c9aa

add

3ae2113

final

81b02c2

ninamiolane requested changes Dec 17, 2021

View reviewed changes

realy final

d7f4584

ninamiolane approved these changes Dec 17, 2021

View reviewed changes

ninamiolane merged commit 7649949 into geomstats:master Dec 17, 2021

		ip_diagonal = gs.einsum("...ii,...ii ,...i->...",
		tangent_vec_a, tangent_vec_b, inv_sqrt_diagonal)

		@@ -0,0 +1,291 @@
		"""Unit tests for Positive lower triangular matrices"""

Cholesky space #1142

Cholesky space #1142

Uh oh!

Conversation

SaitejaUtpala commented Sep 7, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

nguigs commented Sep 7, 2021

Uh oh!

SaitejaUtpala commented Sep 7, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

nguigs commented Sep 7, 2021

Uh oh!

ninamiolane commented Sep 10, 2021

Uh oh!

nguigs left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

nguigs Sep 14, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

SaitejaUtpala Sep 26, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

nguigs commented Sep 14, 2021

Uh oh!

ythanwerdas commented Sep 14, 2021

Uh oh!

ninamiolane commented Sep 14, 2021

Uh oh!

ninamiolane left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

ninamiolane left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

ninamiolane left a comment

Choose a reason for hiding this comment

Uh oh!

SaitejaUtpala commented Dec 16, 2021

Uh oh!

ninamiolane left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

SaitejaUtpala commented Sep 7, 2021 •

edited

Loading

SaitejaUtpala commented Sep 7, 2021 •

edited

Loading

nguigs Sep 14, 2021 •

edited

Loading

SaitejaUtpala Sep 26, 2021 •

edited

Loading