This is nice! Thank you!

It appears to me that the benchmarks show that this only becomes relevant for very large data. So we need to be mindful to not break backward compatibility for all the small and medium-size datasets that people use (which we do by introducing the tiny difference). Don't you think that in the light of this, it would be better to leave the default as is (densifying) and have an option sparse_pca or something similar?

It appears to me that the benchmarks show that this only becomes relevant for very large data.

Hm, even for my example it is 77.14 MiB vs 893.92 MiB, so 10 times difference. This seems large to me, no?

Hm, even for my example it is 77.14 MiB vs 893.92 MiB, so 10 times difference. This seems large to me, no?

Yes, it's definitely large and it's awesome that you solved this problem! I just meant that it's not hitting people's computational resources limits: your example is 60K x 2K, so quite big already, if you densify you need 800MB, which is easily available even on a laptop. That's what I meant.

What do you think?

Yeah, this seems important only for large datasets in that sense. So, i will add sparse_pca option with False by default.

Not sure what kind of test to add for this...

As discussed, @Koncopd will try to integrate this into scikit-learn itself and not into Scanpy. 😄

Similar pull request exists already in sklearn.
scikit-learn/scikit-learn#12841
Will watch.

Shall we replace sparse_pca with something else like implicit_centering or so?

sparse_pca might give rise to confusion since there is indeed a PCA technique called sparse PCA.

Hm, it was decided to suspend this pr earlier.
There is an analogous pr in scikit-learn, but i'm not sure it will got forward.
I'm not sure what to do with this pr...

Ah, then we should leave it open and unchanged until there’s a decision on the sklearn PR. Sorry I missed the decision.

What's the current state here?

This just came out
https://genomebiology.biomedcentral.com/articles/10.1186/s13059-019-1900-3
favoring sklearn's PCA implementation while stating that it cannot yet handle sparse matrices. Is that still true?

Last time i checked it was true. Need to check again, but i bet it is still the same.

That's still the case (at least for randomized PCA @Koncopd linked above), though it looks like there may be a another path forward using other solvers: scikit-learn/scikit-learn#12794. Still needs an implementation though.

You could just add a sparse argument to pca. If True, just call this function instead of scikit-learn's PCA:

def sparse_pca(X,npcs,mu = None):
    # X -- scipy sparse data matrix
    # npcs -- number of principal components
    # mu -- precomputed feature means. if None, calculates them from X.

    # compute mean of data features
    if mu is None: 
        mu = X.mean(0).A.flatten()[None,:]

    # dot product operator for the means
    mmat = mdot = mu.dot 
    # dot product operator for the transposed means
    mhmat = mhdot = mu.T.dot 
    # dot product operator for the data
    Xmat = Xdot = X.dot 
    # dot product operator for the transposed data
    XHmat = XHdot = X.T.conj().dot 
    # dot product operator for a vector of ones
    ones = np.ones(X.shape[0])[None,:].dot 

    # modify the matrix/vector dot products to subtract the means
    def matvec(x): 
        return Xdot(x) - mdot(x)
    def matmat(x): 
        return Xmat(x) - mmat(x)
    def rmatvec(x): 
        return XHdot(x) - mhdot(ones(x))
    def rmatmat(x): 
        return XHmat(x) - mhmat(ones(x))
    
    # construct the LinearOperator
    XL = sp.sparse.linalg.LinearOperator(matvec = matvec, dtype = X.dtype,
                                         matmat = matmat,
                                         shape = X.shape,
                                        rmatvec = rmatvec, rmatmat = rmatmat)
     
    u,s,v = sp.sparse.linalg.svds(XL,solver='arpack',k=npcs)
    
    # i like my eigenvalues sorted in decreasing order
    idx = np.argsort(-s)
    S = np.diag(s[idx])
    # principal components
    pcs = u[:,idx].dot(S) 
    # equivalent to PCA.components_ in sklearn 
    components_ = v[idx,:] 
    return pcs,components_

This only works for the arpack solver. It's a bit slower than PCA on dense matrices (since arpack is slower than randomized), but it's super memory efficient.