Is it OK to make it multithreaded by default?

We haven't made a decision yet :/

How do I provide non-regression tests?

Tests related to this are in test_pairwise.py. We already test some cases, but you can add a test to check that dense and sparse return the same result, on float32 and float64, with or without sorted indices. You should name it test_manhattan_distances. To write it you can get inspiration from other tests.

It would be nice if you could also provide a small benchmark vs master (in both single and multi threaded modes)

General comment: if you had not noticed already, I am not quite up to speed with this development process. I read around and try to do the right thing, but please do not hesitate to set me on the right path.
For instance, is it OK to push often new versions of the PR?
Should I hit the Resolve Conversation only when I have pushed a version that I think addresses the issue? Or is it up to who raised the issue to accept the resolution?

@jeremiedbb, below is a preliminary benchmark/regression test.
I timed three cases, each with 10 repeats.
What spurred me to work also on the dense matrix case is that in v21.2 it appears to run single-threaded and I did not like that,
Anyway, on my laptop (4 vCPU) there is not much of an advantage (multi-threading overhead? slow memory subsystem?). But I often work on HPC systems, with 24 to 32 cores and fast memory subsystems, and it seems to make a difference there.

Random sparse matrices, 10 repeats, shapes: (1000, 3000) (1500, 3000)
sklearn version: 0.21.2
real	0m14.808s
user	0m14.780s
sys	0m0.248s
Random sparse matrices, 10 repeats, shapes: (1000, 3000) (1500, 3000)
sklearn version: 0.22.dev0
real	0m3.283s
user	0m8.001s
sys	0m0.211s
np.all(np.isclose(.,.)) True
Max abs diff: 4.263256414560601e-14


Random sparse matrices, after train_test_split(), 10 repeats, shapes: (800, 3000) (1200, 3000)
sklearn version: 0.21.2
X.has_sorted_indices: 0
Y.has_sorted_indices: 0
real	0m9.819s
user	0m9.804s
sys	0m0.237s
Random sparse matrices, after train_test_split(), 10 repeats, shapes: (800, 3000) (1200, 3000)
sklearn version: 0.22.dev0
X.has_sorted_indices: 1
Y.has_sorted_indices: 1
real	0m2.321s
user	0m5.325s
sys	0m0.230s
np.all(np.isclose(.,.)) True
Max abs diff: 4.263256414560601e-14


Random dense matrices, 10 repeats, shapes: (1000, 1000) (1500, 1000)
sklearn version: 0.21.2
real	0m10.974s
user	0m10.971s
sys	0m0.213s
Random dense matrices, 10 repeats, shapes: (1000, 1000) (1500, 1000)
sklearn version: 0.22.dev0
real	0m7.572s
user	0m24.814s
sys	0m0.235s
np.all(np.isclose(.,.)) True
Max abs diff: 1.8189894035458565e-12

For instance, is it OK to push often new versions of the PR?

I have no problem with that.

Should I hit the Resolve Conversation only when I have pushed a version that I think addresses the issue? Or is it up to who raised the issue to accept the resolution?

If you made changes that impact the lines a conversation is about, this conversation will be marked as outdated. You don't need to mark it as resolved. And it's better to let the person who started the conversation to confirm that it has been addressed correctly :)

Regarding the benchmarks
The sparse case is very convincing. The dense case less. I'll take a look tomorrow to see what's going on with the dense case.

About the dense case.

It's a bit subtle to get the best performance. The reduction is not vectorized by gcc. The way to vectorize it is to manually unroll the loop. Here's how to do it:

def _dense_manhattan(floating[:, :] X, floating[:, :] Y, floating[:, :] out):
    cdef:
        floating s = 0.0
        int n_samples_x = X.shape[0]
        int n_samples_y = Y.shape[0]
        int n_features = X.shape[1]
        np.npy_intp i, j

    for i in prange(n_samples_x, nogil=True):
        for j in range(n_samples_y):
            out[i, j] = _manhattan_1d(&X[i, 0], &Y[j, 0], n_features)

cdef floating _manhattan_1d(floating *x, floating *y, int n_features) nogil:
    cdef:
        int i
        int n = n_features // 4
        int rem = n_features % 4
        floating result = 0

    for i in range(n):
        result += (fabs(x[0] - y[0])
                  +fabs(x[1] - y[1])
                  +fabs(x[2] - y[2])
                  +fabs(x[3] - y[3]))
        x += 4; y += 4

    for i in range(rem):
        result += fabs(x[i] - y[i])

    return result

With this you can equal (even slightly outperform) scipy. And scalability is good.

However
there's already pairwise_distances, which supports metric='manhattan' and n_jobs, so you can already have parallel manhattan distances. Isn't it enough for you ?

@jeremiedbb, thanks for looking into this and for pointing out the vectorization opportunities, of which I was not aware.

Regarding the motivation for this PR, I was actually trying to improve the performance of the calculation of the laplacian kernel (in metrics/pairwise.py)), which uses the manhattan_distances() function:

def laplacian_kernel(X, Y=None, gamma=None):
    X, Y = check_pairwise_arrays(X, Y)
    if gamma is None:
        gamma = 1.0 / X.shape[1]

    K = -gamma * manhattan_distances(X, Y)
    np.exp(K, K)  # exponentiate K in-place
    return K

Ok makes sense. In that case, what would be even faster is to re-implement the whole kernel computation.

because this

K = -gamma * manhattan_distances(X, Y)
np.exp(K, K)

is sub-optimal. It involves 2 extra loop over the distance matrix.

Re-using the above code, it would be more efficient to do something like this:

from libc.math cimport exp

def _dense_laplacian_kernel(floating[:, :] X, floating[:, :] Y, floating[:, :] out, int gamma):
    cdef:
        int n_samples_x = X.shape[0]
        int n_samples_y = Y.shape[0]
        int n_features = X.shape[1]
        np.npy_intp i, j
        floating tmp

    for i in prange(n_samples_x, nogil=True):
        for j in range(n_samples_y):
            tmp =  _dense_manhattan_1d(&X[i, 0], &Y[j, 0], n_features)
            out[i, j] = exp(-gamma * tmp)

Anyway, I think these considerations can be moved to another PR. It will be easier to review if you stick to speed up sparse manhattan for this PR to actually fix the original issue.

OK, I will reinstate the existing implementation of manhattan_distances() for the dense matrix case
(By the way, the title of the PR is now inaccurate. Can one change it?) and perhaps create another PR with your suggestions?

Regarding the sparse case, there is one detail that I am not sure about.
As noted, the proposed _sparse_manhattan() assumes that the CSR matrices in input have sorted indices.
To ensure that, I sort them in-place prior to calling _sparse_manhattan().
I chose to do it in-place because I prefer to avoid copies whenever possible. But it is debatable if it is OK to modify something that is passed as an argument. What is your view?

I created PR #15049 to cover just the sparse matrix case.
I suppose this PR can be closed and perhaps any further discussion on the dense case should go into a new PR.

closing in favor of #15049 (already merged). Thanks @ptocca for your contribution !