Note that you can probably get 90% of the performance you want with numba:

import numba
lookup_table = np.array([bin(x).count("1") for x in range(256)], np.uint8)
@numba.guvectorize([(numba.uint8[:], numba.int64[:])], '(n)->()', nopython=True)
def sum_bits(x, res):
    res[0] = 0
    for xi in x:
        res[0] += lookup_table[xi]

In [41]: a = np.array([0b1100, 0b11011011], np.uint8)

In [42]: sum_bits(a)
Out[42]: 8

In [43]: %timeit sum_bits(a)
1.83 µs ± 80.6 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)

In [44]: def totalbits_bytearr(arr):
    ...:     return np.sum(bytebitcounts[arr])
    ...:

In [45]: bytebitcounts=lookup_table

In [46]: %timeit totalbits_bytearr(a)
12.7 µs ± 1.66 µs per loop (mean ± std. dev. of 7 runs, 100000 loops each)

Thank you very much for that insight and very impressive results. So numpy is not vectorizing things as nicely as was thought then? I had assumed it was... Also surprised to see the numba can take Python code with initialization and figure out how to correctly vectorize a for loop an outer operation of addition and an inner operation of a table lookup. That is impressive. So apparently this report, is mainly to add a new packedbits data type, which has a bunch of numba code for all the ufuncs. But the indexing/accessing/setting is extra bit twiddling.

I suppose bitwise & and | are already implemented using numba or could they be improved too due to their generic data type inspection? Update: 5 times slower using guvectorize than the native &, |.

Also surprised to see the numba can take Python code with initialization and figure out how to correctly vectorize

I think you have the wrong model of how numba works, like I used to. I think the things you need to know about the numba are that:

• it translates python into unoptimized LLVM IR
• it treats any closures (eg the lookup table) as compile-time constants
• LLVM does all the heavy lifting regarding optimization, the same as is used by clang

which has a bunch of numba code for all the ufuncs.

Numpy cannot take a dependency on numba, it would make everything far too cyclic.

I suppose bitwise & and | are already implemented using numba

They're implemented using native loops in C. The problem with np.sum(bytebitcounts[arr]) is it uses two loops, an intermediate array, and no compile-time knowledge of the lookup table.

Certainly did not know about the table lookup optimization. So sounds like it could be wrong to think that we are using the SSE2/AVX instructions in the silicon for that sort of batched parallelism unless LLVM has gotten to that level. These type of operations could even be thrown over to the GPU if dealing with large enough data sets or done with multi-threading, etc.

I am a bit sorry about it. But I will close this. I am not against helpers for speeding up or handling bit arrays better, but I do not think they fit on the current agenda/roadmap (or design) of numpy, since it would require things like bit-sized strides or similar.
This does not mean that we cannot do it, but I think it needs some careful design proposals outside of the issue tracker.