Does it work without numpy?

import sys
for i in range(1, sys.maxunicode + 1):
    chr(i).encode('utf-32-le').decode('utf-32-le')

      1 for i in range(1, sys.maxunicode + 1):
----> 2     chr(i).encode('utf-32-le').decode('utf-32-le')
      3

UnicodeEncodeError: 'utf-32-le' codec can't encode character '\ud800' in position 0:
    surrogates not allowed

But note that the array problem doesn't start until '\ud82d', so some surrogates clearly are allowed.

(it would also be handy if Numpy reported the problematic string in the error message)

Are there open issues that point to this being a problem users have encountered? There are many dark corners in using characters/strings in NumPy arrays since there will always be an impedance mismatch between arrays and unicode. Perhaps we should leave them alone.

If the impedence mismatch is so bad that users can't rely on being able to index their arrays, surely we should be deprecating unicode (and perhaps all string) arrays altogther? object arrays of strs are generally easier to reason about, at least.

I haven't seen issues related to it, but I would expect this to arise in practice for users who are processing text files on Windows (utf-16), and that seems more and more likely with the Python 2 transition now happening even in large and slow institutions. String arrays, numpy, and memory-mapped columnar text files are otherwise a great fit!

ISTR reading recent discussions of Python and unicode surrogates somewhere. My impression was that things were in flux.

For reference this succeeds:

import sys
for i in range(1, sys.maxunicode + 1):
    chr(i).encode('utf-32-le', errors='surrogatepass').decode('utf-32-le', errors='surrogatepass')

I think using errors='surrogatepass' when accessing elements of unicode arrays would resolve this issue 😄

Solution here is to stop using our PyUCS2Buffer_AsUCS4 workaround and just use the Python 3.3 API. Will put out a patch in the next few days.

Awesome! Thanks @eric-wieser and all reviewers ❤️