Minimum not attained in graphical lasso #30768

Bastien-mva · 2025-02-04T14:27:21Z

Bastien-mva
Feb 4, 2025

Hello,

I am trying to use the graphical lasso in torch, as my backend uses GPU.

I am comparing my implementation with the one of scikit-learn.covariance.graphical_lasso, but I find that my implementation gets a better minimum, and that the output are quite different concerning the support of the precision matrix.

Here is my implementation:

import torch
from sklearn.covariance import graphical_lasso as GL

def loss(S, Theta, alpha=0.1):
    return -torch.logdet(Theta) + torch.trace(S @ Theta) + alpha * torch.norm(Theta, p=1)

def graphical_lasso_torch(S, alpha=0.1, max_iter=100, tol=1e-4):
    n = S.shape[0]
    C = torch.eye(n, requires_grad=True) 
    optimizer = torch.optim.Rprop([C], lr=0.01, step_sizes = (1e-10, 50))

    for _ in range(max_iter):
        optimizer.zero_grad()
        Theta = C@(C.T)
        loss_value = loss(S, Theta, alpha=alpha)
        loss_value.backward()
        optimizer.step()
        if loss_value.item() < tol:
            break
    return Theta.detach()




torch.manual_seed(0)
n = 5
p = 10
S = torch.rand(n, p)
S = ([email protected]) / 2
S += torch.eye(n)

alpha = 1

Theta_torch = graphical_lasso_torch(S, alpha=alpha)
_, Theta_sk = GL(S.numpy(), alpha = alpha)
print('loss torch:', loss(S, Theta_torch, alpha=alpha).item())
print('loss scikit-learn:', loss(S, torch.tensor(Theta_sk), alpha=alpha).item())

wich outputs on my machine:

loss torch: 11.097545623779297
loss scikit-learn: 11.462298393249512

Results are even more strange with greater values of alpha, for example 7. There is no warning when calling GL so that it seems it has converged, and using much more iteration (max_iter = 3000 with 0 tolerance) does not help.

Note that I use Theta = [email protected] to make sure the output matrix is symmetric.

Is there something I am missing here ?

Kxrma47 · 2026-08-31T21:58:36Z

Kxrma47
Aug 31, 2026

The two functions are not minimizing the same objective. Scikit-learn's graphical lasso applies the L1 penalty to the off-diagonal entries of the precision matrix; the diagonal is not penalized. Your torch.norm(Theta, p=1) penalizes every entry, including the diagonal, so comparing those two loss values does not test scikit-learn's optimum.

Use an explicit off-diagonal penalty:

off_diag = Theta - torch.diag(torch.diagonal(Theta))
sign, logabsdet = torch.linalg.slogdet(Theta)
objective = -logabsdet + torch.trace(S @ Theta) + alpha * off_diag.abs().sum()

Then evaluate both solutions with that same function.

There are two more caveats in the Torch solver. Parameterizing Theta = C @ C.T turns a convex optimization in Theta into a non-convex optimization in C, so Rprop has no guarantee of finding the graphical-lasso optimum. It also does not create exact zeros reliably, whereas the coordinate-descent formulation is designed around the nonsmooth L1 term. Finally, the returned Theta was computed before the last optimizer.step(), so it is one update stale.

For large alpha, scikit-learn's solution should become diagonal because only off-diagonal terms are shrunk. Once the objective is matched, compare the dual gap reported by scikit-learn and the KKT conditions rather than only the raw objective from the Rprop iterate.

0 replies

Navigation Menu

Uh oh!

Uh oh!

Minimum not attained in graphical lasso #30768

Uh oh!

{{title}}

Uh oh!

Replies: 1 comment

Uh oh!

{{title}}

Uh oh!

Select a reply

Uh oh!

Uh oh!

Uh oh!

Minimum not attained in graphical lasso #30768

Uh oh!

Bastien-mva Feb 4, 2025

Replies: 1 comment

Uh oh!

Kxrma47 Aug 31, 2026

Bastien-mva
Feb 4, 2025

Kxrma47
Aug 31, 2026