the beginning of a UTF-8 encoded bytestream containing the string data. We do

not append a trailing null character to the byte stream, so users attempting to

pass the ``buf`` field to an API expecting a C string must must create a copy

with a trailing null. This choice also means that unlike the fixed-width strings

in NumPy, ``StringDType`` array entries can contain arbitrary embedded or

trailing null characters.

that the string buffers for nearby array elements are usually nearby on the

heap. We do not guarantee that neighboring array elements are contiguous on the

heap to support missing data and allow mutation of array entries, see below for

more discussion on how these topics affect the memory layout.

In addition to making a typedef for ``npy_static_string`` public, we also plan

to add an interface for allocating, copying, and freeing strings with this

layout via the arena allocator to the public numpy C API to ease downstream

integration.

In the future we may decide to exploit the small string optimization [6]_ to

encode strings smaller than the size of the ``npy_static_string`` struct

entry. If this is implemented, we will reserve the most significant byte in the

``len`` for flags, including a flag to indicate the array element is stored in

the array buffer. For arrays consisting entirely of small strings this will

bypass the need to do any sidecar heap allocations. This should be relatively

straightforward to add but has not been completed yet to focus on other aspects

of the proposal. While reserving a whole byte for flags may be unnecessary, we

will still have 12 bits of space in the ``len`` field, which is much more than

is likely to be necessary to store the length of a single array element in

real-world use, and having 256 possibilities for flags gives us flexibility for

the future.

Besides the string data itself, each array element requires 16 bytes of storage

for the ``npy_static_string`` instance in the array buffer. In principle we

could use a 32 bit integer to store the ``len`` field, saving 4 bytes per array

element, but if we only use a single bit for the small string optimization

that will still leave us with an uncomfortably small 7 bits of space in the

``len`` field. In addition, making use of the small string optimization will

somewhat offset the memory cost of a 64 bit ``len`` field, since many real-world

use-cases employ small strings.

Missing Data

By default, zeroed out entries in the array buffer represent empty

strings. However, if the DType instance was created with an ``na_object`` field,

zeroed-out entries represent missing data. By making this choice, a zero-filled

newly allocated buffer returned by ``calloc`` does not need any additional

post-processing to produce an empty array. This choice also means casts between

different missing data representations are views.

Whether or not the ``na_object`` is set, empty strings are not stored in the

sidecar buffer since they require no additional storage besides the entry in the

array buffer itself. This means that adjacent entries in the sidecar buffer are

not necessarily adjacent entries in the array buffer.