imshow in 2.1: color resolution depends on outliers? #10567

trettberg · 2018-02-22T15:32:23Z

Since 2.1 the effective color resolution of imshow(x, vmin=-1, vmax=1) seems to depend on large outliers in x.

Is this intended behavior?

Thanks for clarification!

import numpy as np
import matplotlib as mpl
import matplotlib.pyplot as plt

x = np.linspace(-1, 1, 500 )
x = np.ones([20, 1]) * x[np.newaxis, :]

x1 = x.copy()
x1[0, 0] = 1e16

x2 = x.copy()
x2[0, 0] = 1e17

_, axes = plt.subplots(nrows=3)
axes[0].imshow(x, vmin=-1, vmax=1)
axes[1].imshow(x1, vmin=-1, vmax=1)
axes[2].imshow(x2, vmin=-1, vmax=1)

axes[0].set_title(mpl.__version__)
plt.show()

Actual outcome

Expected outcome

Matplotlib version

Operating system: Linux
Matplotlib version: 2.1.0 (expected behavior in 2.0.2)
Matplotlib backend: Qt5Agg
Python version: 3.6.4
Other libraries: numpy 1.13.3
via conda

The text was updated successfully, but these errors were encountered:

dstansby · 2018-02-22T17:03:34Z

Hmm, I'm not sure about whether this is intended or not, it bisects to 12c27f3 so maybe @tacaswell will know more?

WeatherGod · 2018-02-22T17:16:02Z

Comment in that new code: ``` # This may introduce numeric instabilities in very broadly # scaled data ``` Looks like we might have found that situation. The data is getting scaled for the purposes of the AGG renderer (somewhat independent of the norm that was specified). Unfortunately, due to some fun effects with floating point numbers, that normalization gets very digitized when the range is very large. I would say that this is unintentional, but I am not sure how we can fix it, though.

…

On Thu, Feb 22, 2018 at 12:03 PM, David Stansby ***@***.***> wrote: Hmm, I'm not sure about whether this is intended or not, it bisects to 12c27f3 <12c27f3> so maybe @tacaswell <https://github.com/tacaswell> will know more? — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#10567 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AARy-OVeKeKqxTPE6_6m-ZIB0IJ_JJ3Uks5tXZ3ogaJpZM4SPgUA> .

jklymak · 2018-02-22T17:29:15Z

We messed around w/ all of this for 2.1.2. #10133 W/o looking carefully, does that fix this?

jklymak · 2018-02-22T17:32:40Z

Pretty sure it does - it was a round off error that caused this, but its now been fixed for large floats.

dstansby · 2018-02-22T19:57:23Z

It didn't seem fixed when I ran the above test on the master branch.

jklymak · 2018-02-22T21:14:58Z

agreed. The image is stored as float64. You lose resolution at 2^53=9e15. I’m not sure how this worked in 2.1, but that’s the problem here.

WeatherGod · 2018-02-22T21:18:34Z

the rescaling is relatively recent, iirc, as part of the work to better handle over/under/invalid.

…

On Thu, Feb 22, 2018 at 4:15 PM, Jody Klymak ***@***.***> wrote: agreed. The image is stored as float64. You lose resolution at 2^53=9e15. I’m not sure how this worked in 2.1, but that’s the problem here. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#10567 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AARy-Chvzb8jOCgsCtG6TG-vonUm2Oreks5tXdjUgaJpZM4SPgUA> .

jklymak · 2018-02-22T21:23:01Z

I tried to set the dtype to float 128 fro big numbers, but I was told there is no such thing...

tacaswell · 2018-02-23T00:35:57Z

Image interpolation is the bug that just keeps giving!

The chain of bugs and fixes here goes:

while doing 2.0 we noticed that we were interpolating after doing norm + rgb mapping -> leads to colors not in the color map in the final image so we moved the interpolation to be done on the normed, but not color mapped data.
this introduced imshow interpolation uses masked values #8012, so we better masked around 'invalid' pixels when color mapping [MRG+1] Fix imshow masked interpolation #8024
as a consequence of this over/under/invlaid pixels effectively poisoned everything in their in their kernel window (Image interpolation wrong for pixel values exceeding vmax #8631) so we moved interpolation to the raw data (Fix image interpolation #8966)
we now have numerics issues for data with large dynamic range (this bug + imshow doesn't properly display some images #10072 which was fixed by FIX: Image scaling for large dynamic range ints #10133)

jklymak · 2018-02-23T00:42:49Z

It’s the same bug, it’s just that #10133 only fixed up to 10^16 or so.

jklymak · 2018-02-23T02:36:35Z

Is there some reason to not apply the norm (but not RGB mapping) first? If I do that, it seems to work for this case. It fails a bunch of the image tests meant to catch the other bugs, but I'm not clear why my version is worse than the test version...

Test

Moving the norm step up (pre interp)

tacaswell · 2018-02-23T03:12:23Z

The ringing / saturation in the lower left of the test is correct. If you normalize first you are effectively clipping the impact of the outliers.

jklymak · 2018-02-23T03:57:39Z

Well, we could do something like pre-clip the data to some suitably large number around the vmin/vmax, but not 17 orders of magnitude. That should preserve both behaviours

Or folks could mask their invalid data instead of passing in huge (I assume) invalid values.

hagenw · 2018-02-23T09:15:14Z

I would not assume that huge data points are automatically invalid. In our application we have values between -1 and 1 in most of the image, but it can include regions with values approaching Inf.

I would find it strange to mask some of the data before plotting them. I'm also not aware that you have to do something like this in Matlab or gnuplot.

jklymak · 2018-02-26T21:34:41Z

See #10613 for a fix....

tacaswell · 2018-03-03T21:24:13Z

closed by #10613

trettberg mentioned this issue Feb 22, 2018

Monofrequent dipole source example plot sfstoolbox/sfs-python#50

Closed

tacaswell added this to the v2.2.0 milestone Feb 22, 2018

tacaswell assigned jklymak Feb 26, 2018

jklymak mentioned this issue Feb 26, 2018

FIX: resolution of imshow for floats and 2-D greyscale PIL images #10613

Merged

2 tasks

hagenw mentioned this issue Feb 27, 2018

Clip soundfield into float64 range for plotting sfstoolbox/sfs-python#57

Merged

tacaswell closed this as completed Mar 3, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

imshow in 2.1: color resolution depends on outliers? #10567

imshow in 2.1: color resolution depends on outliers? #10567

trettberg commented Feb 22, 2018

dstansby commented Feb 22, 2018

Uh oh!

WeatherGod commented Feb 22, 2018 via email

Uh oh!

jklymak commented Feb 22, 2018

Uh oh!

jklymak commented Feb 22, 2018

Uh oh!

dstansby commented Feb 22, 2018

Uh oh!

jklymak commented Feb 22, 2018

Uh oh!

WeatherGod commented Feb 22, 2018 via email

Uh oh!

jklymak commented Feb 22, 2018

Uh oh!

tacaswell commented Feb 23, 2018

Uh oh!

jklymak commented Feb 23, 2018

Uh oh!

jklymak commented Feb 23, 2018

Uh oh!

tacaswell commented Feb 23, 2018

Uh oh!

jklymak commented Feb 23, 2018

Uh oh!

hagenw commented Feb 23, 2018

Uh oh!

jklymak commented Feb 26, 2018

Uh oh!

tacaswell commented Mar 3, 2018

Uh oh!

Uh oh!

imshow in 2.1: color resolution depends on outliers? #10567

imshow in 2.1: color resolution depends on outliers? #10567

Comments

trettberg commented Feb 22, 2018

dstansby commented Feb 22, 2018

Uh oh!

WeatherGod commented Feb 22, 2018 via email

Uh oh!

jklymak commented Feb 22, 2018

Uh oh!

jklymak commented Feb 22, 2018

Uh oh!

dstansby commented Feb 22, 2018

Uh oh!

jklymak commented Feb 22, 2018

Uh oh!

WeatherGod commented Feb 22, 2018 via email

Uh oh!

jklymak commented Feb 22, 2018

Uh oh!

tacaswell commented Feb 23, 2018

Uh oh!

jklymak commented Feb 23, 2018

Uh oh!

jklymak commented Feb 23, 2018

Test

Moving the norm step up (pre interp)

Uh oh!

tacaswell commented Feb 23, 2018

Uh oh!

jklymak commented Feb 23, 2018

Uh oh!

hagenw commented Feb 23, 2018

Uh oh!

jklymak commented Feb 26, 2018

Uh oh!

tacaswell commented Mar 3, 2018

Uh oh!