At this phase, we've got an 8x speedup over the current implementation. From here, we have three tasks.

Conceptually, we want the final implementation to look like this:

def crand_plus(*data, local_function=moran_i):
    cardinalities, weight_pointers, *etc = setup_crand()
    with multiprocessing.Pool() as P:
        larger, random_locals = P.map(_do_one_observation, stuff_needed_for_that_iteration)
    return larger, random_locals

defining the inner loop & lifting it into a numba method.

We need to lift up the body of neighbors_perm_plus (e.g. l1655) into its own thing. This will entail solving a few issues:

• indexing into the flat weights array? *is the start position for i given by numpy.cumsum(cardinalities)[i]?
• defining the signature of _do_one_observation. It'll probably need
  • its weights or a pointer & into the weights flat array
  • the cardinality of i
  • the permutation matrix
  • data for z
  • the index of the observation

parallelizing the computation

If (and only if) there's gains to be made by parallelizing the simulations, we want to explore using joblib, which should be available on all platforms.

defining local functions

Ideally, we'd like to implement the permutation logic once and then use it in all permutation-inference driven things. If we do that, we need to define something like a numba-fied local statistic for anything using the permutation inference engine. For example, a Moran version might be:

@njit(fastmath=True)
def local_statistic(i, z, permutations, cardinality, weights_i):
    mask = numpy.ones_like(z)
    n = z.shape[0]
    scaling = (n - 1) / (z * z).sum()
    mask[i] = False
    z_no_i = z[mask].reshape(z.shape)
    flat_random_z = z_no_i[permutations[:, :cardinality]]
    z_no_i_lag = flat_random_z.reshape(-1,cardinality) @ weights_i
    rlisa = z[i] * z_no_i_lag.sum(axis=1) * scaling
    return rlisa

This will enable us to pass arbitrary JIT-ted functions to the random permutation engine.

defining the inner loop & lifting it into a numba method.

We need to lift up the body of neighbors_perm_plus (e.g. l1655) into its own thing. This will entail solving a few issues:

lifting is pending at ljwolf#4

The only failures now are the known failures for the chi-squared test in join counts (#123). This should be ready to merge 🥳

The only failures now are the known failures for the chi-squared test in join counts (#123). This should be ready to merge partying_face

I think once #124 is merged, this will also pass if rerun.