Nice work @TomDLT and thanks for the summary in #10463.

The key caveat is that it requires all required neighborhoods to be computed in advance. This may be a memory burden still for something like DBSCAN (although our current implementation is already memory heavy).
It would be really good if we could illustrate plugging in an approximate nearest neighbors estimator, but it would likely include adding a pyann or similar dependency, unless we attempt to write a very short LSH just to prove the point.

The API with a transformer looks quite nice, and caching the NN can certainly help for repeated estimators fits with different parameters. However, as far as ANN are concerned I don't find it immediately obvious that pre-computing a sparse distance matrix with ANN, and e.g. using it in DBSCAN has an equivalent performance to directly using the ANN in the estimator.

For instance, I would be curious to know the results of the following benchmark. Take some low dimensional data where we know that kd_tree will perform well (e.g. n_features=2). Use it to pre-compute the sparse distance matrix and fit DBSCAN. Compare the performance to using kd_tree directly in the DBSCAN. This would allow estimating the overhead of this approach, without necessarily having to add an ANN implementation.

I'm not sure what you're trying to show about ANN with that, @rth. Currently DBSCAN calculates the nearest neighbors of all X once and computes the clustering from that. There is some overhead when passing in sparse D rather than just passing in features X: unpacking, thresholding on radius and repacking.

This is not how it was originally implemented, mind you, and was optimised for speed rather than memory (see e.g. #5275). Perhaps with a *NeighborsTransformer, we could go back to that original low memory implementation, but get most of the speed benefits as long as the user provides the precomputed sparse matrix.

One disadvantage of this design as opposed to #8999 is that there the host algorithm can pass in a radius or a n_neighbors, which would save time relative to precomputing and then filtering/checking the precomputed input. We also do not currently require that the precomputed input is in sorted order, so there is a O(n_train_samples log n_train_samples) cost per query in order to sort it. We could make some constraints on precomputed sparse matrix input corresponding to *neighbors_graph(None, mode='distance') output (or check it and raise a warning but accept it) if we'd rather: it needs to be in CSR with each row's data increasing, and at train time the diagonal needs to be implicit, not stored zeros.

One advantage of this design as opposed to #8999 is that it may allow you to precompute more neighbors than you need, then estimate downstream with various parameters (e.g. DBSCAN.eps); thus it works well with caching pipelines. Of course you could do that with #8999 by implementing your own precomputed neighborhood wrapper.

Regardless of all this, I think it would be good to give an example of, or at least describe, the use of this for exploiting approximate nearest neighbor approaches. Writing up a simple (but not super-efficient) gaussian projection hash, or a minhash, might not be excessive.

Thanks for the feedback.

additional TODO:

• Separate into RadiusNeighborsTransformer and KNeighborsTransformer
• Consider adding a constraint on graph sorting order
• Adapt as much of [MRG] Modifies T-SNE for sparse matrix #10206 as possible, in particular unpacking with different n_neighbors/radius
• Add small speed benchmark to quantify unpacking overhead
• Add example with ANN

I suggest we do something like:

def _check_precomputed(D, n_neighbors=None):
    if not issparse(D):
        return check_array(D, ensure_min_features=n_neighbors)
    if X.format not in ('csr', 'csc', 'coo', 'lil'):
        raise TypeError('Sparse matrix in {!r} format is not supported due to its handling of explicit zeros'.format(D.format))
    copied = X.format != 'csr'
    D = D.tocsr()
    if n_neighbors is not None and np.diff(D.indptr).min() < n_neighbors:
        raise ValueError('Provide at least {} precomputed sparse neighbors. Got {}'.format(n_neighbors, np.diff(D.indptr).min()))
    # check sort
    out_of_order = D.data[:-1] > D.data[1:]
    if out_of_order.sum() != out_of_order.take(D.indptr[1:-1], mode='clip').sum():
        warnings.warn('Precomputed sparse input was not sorted by data.', EfficiencyWarning)
        if not copied:
            D = D.copy()
        for start, stop in zip(D.indptr, D.indptr[1:]):
            order = np.argsort(D.data[start:stop], kind='mergesort')
            D.data[start:stop] = D.data[order]
            D.indices[start:stop] = D.indices[order]
    return D

It's a pity the sorting check takes O(nnz) memory and time

Everything is red (and that might be my fault)

We kind of hide this implementation detail in our KNeighborsTransformer, but a custom *NeighborsTransformer would need to know when to do this. (Which is only shown in the example)

I think this PR is ready to merge.

Awesome !

it is unclear when to set n_neighbors = self.n_neighbors + 1 in the implementation of a custom *NeighborsTransformer

• This happens only for KNeighborsTransformer, since RadiusNeighborsTransformer does not need a n_neighbors parameter.
• This is not really a strong API specification, since passing too many neighbors will always work (the following estimators will just filter the extra neighbors). Passing too few neighbors will raise an error.
• The distinction mode == 'distance' is only for internal compatibility of n_neighbors definitions. A custom estimator does not need to do this distinction.

I added a bit more details in the documentation, tell me if you think something else should be added.

To maximise compatiblity with all estimators, a safe choice is to always include one extra neighbor in a custom nearest neighbors estimator, since unnecessary neighbors will be filtered by following estimators.

I like this. As a user, I would always include one more just to keep things simple . (Or a few more to have the flexibility to explore parameters downstream.)

Thank you @TomDLT !

Now it opens a lot of opportunity to try to precompute neighbors with state of the art approximate NN implementations such as annoy or https://github.com/kakao/n2.

oh boy ! big congrats @TomDLT for never giving up on this one ! 🍻

Awesome !
Thanks a lot for all the comments and reviews !