BUG: truthiness of strings is arbitrary, context-dependent, and inconsistent #9875

eric-wieser · 2017-10-17T06:22:25Z

The following table shows the extents of the problem

x : str
sc = np.str_(x)
arr = np.array(x, np.str_)

`x`	`bool(x)`	`bool(sc)`	`bool(arr[()])`	`bool(arr)`	`sc.astype(bool)`
`''`	`False`	`False`	`False`	`False`	`ValueError`
`'\0'`	`True`	`True`	`False`	`False`	`ValueError`
`' '`	`True`	`True`	`True`	`False`	`ValueError`
`' \0'`	`True`	`True`	`True`	`False`	`ValueError`
`'0'`	`True`	`True`	`True`	`True`	`False`
`'1'`	`True`	`True`	`True`	`True`	`True`
`'\0 '`	`True`	`True`	`True`	`True`	`ValueError`
`' \0 '`	`True`	`True`	`True`	`True`	`ValueError`

def call_or_exc(f):
    try:
        return f()
    except Exception as e:
        return e

def truthiness(x):
    x_arr = np.array(x)
    x_sc = np.unicode_(x)
    x_sc2 = x_arr[()]
    return bool(x), bool(x_sc), bool(x_sc2), bool(x_arr), call_or_exc(lambda: x_sc.astype(bool))

def fmt(x):
    if isinstance(x, str): return repr(x).replace(r'\x00', r'\0')
    if isinstance(x, Exception): return type(x).__name__
    return repr(x)

print(' | '.join("`{}`".format(x) for x in ['x', 'bool(x)', 'bool(sc)', 'bool(arr[()])', 'bool(arr)', 'sc.astype(bool)']))
print(' | '.join(['--'] * 6))
for val in ['', '\0', ' ', ' \0', '0', '1', '\0 ', ' \0 ']:
   print(' | '.join("`{}`".format(fmt(x)) for x in (val,)+truthiness(val)))

Generated with...

The differences come down to:

bool(arr[()]) - indexing string arrays causes trailing nulls to be truncated. This is unavoidable.
bool(arr) - has special handling to treat spaces as falsy. This seems unpythonic, since that's not how str.__bool__ behaves - are we stuck with it
arr.astype(bool) - treated as arr.astype(int).astype(bool), which seems bizarre

The text was updated successfully, but these errors were encountered:

eric-wieser · 2017-10-17T06:53:28Z

Continues from #5967

mattip · 2019-04-29T23:17:51Z

Should we move forward with this? PR #9877 requests discussion take place here, but the issue does not seem to have garnered much attention.

Edit: PR number

LevN0 · 2020-04-20T15:10:07Z

Looking at that PR, what is the future path for doing the conversation it cites if some version of it gets approved.

Previously this was interpreted as string.astype(int).astype(bool), which
interpeted '0' as False.

The behavior is now consistent with count_nonzero and nonzero, treating only
the empty string as False.

So is the suggestion for future code going from an array of strings containing 0/1 to bool to typecast twice?

seberg · 2020-04-20T15:26:33Z

@LevN0 it is the only consistent way and it is identical to python: bool("0") vs. bool(int("0")). So yes, what to do right now is a bit trickier, unless we do a quick 1.18.4 release just for this. @charris ping just in case you did not see gh-16023 yet.

Now, there are some exceptions to this and I am not sure what they currently do. That is textreaders should behave differently, since they parse True/False. But I guess those must not have been using this anyway.

eric-wieser · 2020-04-20T16:58:06Z

Can someone compute the diff between the table here and the one from 1.18.3?

seberg · 2020-04-20T17:11:46Z

Well, I used master, but I doubt anything else changed this. Only the last column changed, and that of course changed quite a bit:

`x`	`bool(x)`	`bool(sc)`	`bool(arr[()])`	`bool(arr)`	`sc.astype(bool)`
`''`	`False`	`False`	`False`	`False`	`False`
`'\0'`	`True`	`True`	`False`	`False`	`False`
`' '`	`True`	`True`	`True`	`False`	`True`
`' \0'`	`True`	`True`	`True`	`False`	`True`
`'0'`	`True`	`True`	`True`	`True`	`True`
`'1'`	`True`	`True`	`True`	`True`	`True`
`'\0 '`	`True`	`True`	`True`	`True`	`True`
`' \0 '`	`True`	`True`	`True`	`True`	`True`

The NULL padding getting lost is the problem of the scalar though, numpy unicode though, not the bool cast itself.

seberg · 2020-04-20T17:37:12Z

As to the bool(sc) logic, it is mirrored by len(sc) since the NULL stripping is not consistent there.

seberg · 2020-04-24T20:16:18Z

Jeesh, I did not remember the crazyness of ignoring all whitespaces for the nonzero implementation. I am not even sure we can put a warning into that easily, since it might be used all over without the GIL...
That is:

np.nonzero(np.array([" ", "\n", "\t", ""]))
# Should only consider the last "" as Falsy, but considers them all False.

See also gh-9877

TomAugspurger · 2020-05-01T16:26:39Z

Just to confirm, any changes here would eventually apply to where and argwhere? So the output of

In [12]: np.argwhere(["a", " ", "c"])
Out[12]:
array([[0],
       [2]])

might change to count ' ' as true in the future?

seberg · 2020-05-01T17:20:21Z

Yeah, I think we should/have to do that... I am not quite sure yet how to do the transition(or lets say how nasty it will be). For current strings. I suppose arr == arr.dtype.type() is a way to opt in to the future behaviour (for most dtypes anyway, not for object).

What would this mean for pandas (and dask)? The current (white-space ignoring) way seems so strange to me from a modern Python perspective...

TomAugspurger · 2020-05-01T17:33:17Z

The current (white-space ignoring) way seems so strange to me from a modern Python perspective

Agreed.

I only discovered this issue because of a test in dask. We currently implement argwhere via array.astype(bool), which does treat ' ' as True, so our results differ from NumPy. I'm probably OK with that difference.

This shouldn't really affect pandas users, since we convert incoming fix-width string dtype to object-dtype.

seberg · 2020-05-01T17:38:04Z

Right, the only real thing I would be worried about if we manage to add reasonable warnings in our code paths and you all suddenly have to try and work around that.

TomAugspurger · 2020-05-01T17:41:06Z

Ah, no, I don't think so (at least not for where) since we're using astype(bool) and masking, Dask users shouldn't see any warnings.

seberg · 2020-12-18T21:39:55Z

Since Chuck gave a ping on the related PR. Do we think we could make a decision to move forward with this?

I doubt it is much in the way of DTypes, just one more of those annoying things to look out for. The reason is that all this does is that nonzero can't be implemented in terms of .astype(bool) and .astype(bool) is hopeless inconsistent for strings. That is all very annoying, but I am not sure it will float up somewhere as more than a simple special case.

Of course, this is still a major bug and trap, so I would still be happy to form a plan about just fixing it!

seberg · 2021-01-27T19:22:51Z

We had discussed this, and we somewhat agreed that this is desireable, and should have FutureWarnings (or VisibleDeprecationWarning.

A specific issue may be np.loadtext and genfromtxt since that might be a more common use case for using 0 and 1 as boolean, so that may need some special consideration.

I hope to look at this, hopefully, FutureWarnings (and opting in to future behaviour) will be reasonably possible.

eric-wieser added 00 - Bug component: numpy._core labels Oct 17, 2017

eric-wieser mentioned this issue Oct 17, 2017

BUG: count_non_zero behaves differently with axis argument #9873

Open

This was referenced Oct 17, 2017

BUG: Make bool(str_scalar) and str_scalar.astype(bool) consistent #9848

Closed

WIP: Make str → bool conversion consistent #9877

Closed

BUG: bool(scalar) is not the same as scalar.astype(bool) #9847

Closed

jakirkham mentioned this issue Apr 1, 2018

nonzero on arrays of characters with whitespace #9462

Closed

This was referenced Oct 23, 2018

Bug in masked array boolean comparison in Numpy 1.15.3 #12248

Closed

Should be able to convert string arrays to bool_ #12252

Closed

seberg mentioned this issue Sep 3, 2019

Converting char to bool fails when it encounters false #14406

Closed

masonproffitt mentioned this issue Oct 3, 2019

Error from astype() on StringArray and inconsistencies with zeros_like() scikit-hep/awkward-0.x#199

Open

eric-wieser mentioned this issue Apr 20, 2020

Change in bool type coersion? #16023

Closed

TomAugspurger mentioned this issue May 1, 2020

Upstream compat dask/dask#6159

Merged

This was referenced Jun 16, 2020

TST: Add new tests for array coercion #16571

Merged

TRACKING: Review and possibly address changes regard new dtypes #16624

Open

seberg added the triage review Issue/PR to be discussed at the next triage meeting label Dec 18, 2020

seberg self-assigned this Jan 27, 2021

seberg added triaged Issue/PR that was discussed in a triage meeting and removed triage review Issue/PR to be discussed at the next triage meeting labels Jan 27, 2021

seberg mentioned this issue Nov 10, 2021

Value of zero in array with dtype C char evaluates to 'True' #13973

Closed

seberg mentioned this issue Jun 8, 2023

API: Change string to bool conversions to be consistent with Python #23898

Merged

charris closed this as completed in #23898 Jun 9, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

BUG: truthiness of strings is arbitrary, context-dependent, and inconsistent #9875

BUG: truthiness of strings is arbitrary, context-dependent, and inconsistent #9875

eric-wieser commented Oct 17, 2017 •

edited

Loading

eric-wieser commented Oct 17, 2017

Uh oh!

mattip commented Apr 29, 2019 •

edited

Loading

Uh oh!

LevN0 commented Apr 20, 2020 •

edited

Loading

Uh oh!

seberg commented Apr 20, 2020

Uh oh!

eric-wieser commented Apr 20, 2020

Uh oh!

seberg commented Apr 20, 2020 •

edited

Loading

Uh oh!

seberg commented Apr 20, 2020

Uh oh!

seberg commented Apr 24, 2020

Uh oh!

TomAugspurger commented May 1, 2020

Uh oh!

seberg commented May 1, 2020 •

edited

Loading

Uh oh!

TomAugspurger commented May 1, 2020

Uh oh!

seberg commented May 1, 2020

Uh oh!

TomAugspurger commented May 1, 2020

Uh oh!

seberg commented Dec 18, 2020

Uh oh!

seberg commented Jan 27, 2021

Uh oh!

Uh oh!

BUG: truthiness of strings is arbitrary, context-dependent, and inconsistent #9875

BUG: truthiness of strings is arbitrary, context-dependent, and inconsistent #9875

Comments

eric-wieser commented Oct 17, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

eric-wieser commented Oct 17, 2017

Uh oh!

mattip commented Apr 29, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

LevN0 commented Apr 20, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

seberg commented Apr 20, 2020

Uh oh!

eric-wieser commented Apr 20, 2020

Uh oh!

seberg commented Apr 20, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

seberg commented Apr 20, 2020

Uh oh!

seberg commented Apr 24, 2020

Uh oh!

TomAugspurger commented May 1, 2020

Uh oh!

seberg commented May 1, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

TomAugspurger commented May 1, 2020

Uh oh!

seberg commented May 1, 2020

Uh oh!

TomAugspurger commented May 1, 2020

Uh oh!

seberg commented Dec 18, 2020

Uh oh!

seberg commented Jan 27, 2021

Uh oh!

eric-wieser commented Oct 17, 2017 •

edited

Loading

mattip commented Apr 29, 2019 •

edited

Loading

LevN0 commented Apr 20, 2020 •

edited

Loading

seberg commented Apr 20, 2020 •

edited

Loading

seberg commented May 1, 2020 •

edited

Loading