Thanks to visit codestin.com
Credit goes to github.com

Skip to content

imshow doesn't normalize the color range in RGB images #9391

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
michalkahle opened this issue Oct 13, 2017 · 26 comments
Closed

imshow doesn't normalize the color range in RGB images #9391

michalkahle opened this issue Oct 13, 2017 · 26 comments
Milestone

Comments

@michalkahle
Copy link

imshow doesn't normalize the color range in RGB images

Bug summary
When single channel float image is passed to imshow it gets normalized to range [0,1] before display. This does not happen for RGB images.

Code for reproduction

img = np.arange(0, 0.6, .1).reshape(6,1)
plt.subplot(141)
plt.title('grayscale')
plt.imshow(img)
plt.colorbar()

img = np.arange(0, 0.6, .1).repeat(3).reshape(6,1,3)
plt.subplot(142)
plt.title('RGB')
plt.imshow(img)
plt.colorbar()

img = np.arange(0, 1.6, .2).reshape(8,1)
plt.subplot(143)
plt.title('grayscale')
plt.imshow(img)
plt.colorbar()

img = np.arange(0, 1.6, .2).repeat(3).reshape(8,1,3)
plt.subplot(144)
plt.title('RGB')
plt.imshow(img)
plt.colorbar()

Actual outcome
mpl_color_norm
Please also note how the colorbars are misleading.

Expected outcome
I'd expect the RGB images to look the same as grayscale.

Real life example
This is how I encountered this issue. What is the value in all three channels at [3,18]? It is actually around 1.2!
download

Matplotlib version: 2.0.2

Discussion
The same issue: #5382 , on SO, and another SO.

In #5382 @tacaswell explains that and RGB image "goes through a different code path which do not get passed through the normalize/colormap framework". Colormapping of course makes no sense for color images but I believe that normalization should be consistently applied to RGB images.

I got used to matplotlib normalization of single channel float images and the need of using vmin and vmax to avoid this normalization and intuitively expected RGB images to be treated consistently. As I noticed subsequently the docstrigs say that "the value for each component of MxNx3 and MxNx4 float arrays should be in the range 0.0 to 1.0." But I believe that to simply overflow without raising error or warning is not correct.

@jklymak
Copy link
Member

jklymak commented Oct 13, 2017

RGB means RGB. I don't think it should be normalized.

I think a case could be made that it shouldn't do a modulo past 1.0 or below 0.

@anntzer
Copy link
Contributor

anntzer commented Oct 13, 2017

I think it's either error out ("Errors should never pass silently."), or clipping (because sometimes you may end up with values just a bit below 0 or a bit above 1 due to fp imprecision, so "practicality beats purity.").

See also #8854.

@WeatherGod
Copy link
Member

WeatherGod commented Oct 13, 2017 via email

@anntzer
Copy link
Contributor

anntzer commented Oct 13, 2017

that is definitely an issue too... :)

@tacaswell tacaswell added this to the 2.2 (next feature release) milestone Oct 13, 2017
@tacaswell tacaswell added the Release critical For bugs that make the library unusable (segfaults, incorrect plots, etc) and major regressions. label Oct 13, 2017
@tacaswell
Copy link
Member

I am confused why it wraps...

Agree the colorbar should refuse to let it's self be created for an RGB(A) image.

@anntzer
Copy link
Contributor

anntzer commented Oct 13, 2017

It wraps because ultimately everything is cast to uint8 (essentially (uint8_t)(val * 0xff)).

I also don't think it's release critical, the wraparound is also present in 1.5.0 and probably since ever.

Upon further thought I think I prefer erroring on such inputs, possibly with a message that points to np.clip.

@michalkahle
Copy link
Author

It might have been unfortunate to label my data "RGB image". It would be better to call it three channel float images. uint8 RGB images should NOT be normalized.
I think the viewer should do the most useful thing by default. As a user I often just want to quickly see what is in my data without much fuss. With arrays of floats my data can take the whole range. The clipping or modulo destroy my data. If I get an error about wrong range I'd be forced to normalize the data myself in which case I'd wonder why the viewer does not do this for me as it does with single channel image. If I wanted to be specific I'd use the norm, vmin and vmax parameters just as I do with single channel image. Simple. Consistent.

@jklymak
Copy link
Member

jklymak commented Oct 16, 2017

What three-channel data do you look at that isn’t meant to be an RGB image and why do you look at it as an image?

@michalkahle
Copy link
Author

@jklymak these are three repeated absorbance measurements in 384 well plate. I just wanted to quickly see the pattern on the plate as well as variability in the repeated measurements. I agree this is not typical use of 3 channel images.

@jklymak
Copy link
Member

jklymak commented Oct 17, 2017

Cool. OTOH I’m sure you can appreciate that it is wrong to automatically normalize an arbitrary RGB image. If the whole image is dark, it should stay dark unless the user wants to explicitly lighten it.

@michalkahle
Copy link
Author

I absolutely agree with that in case of integer images. But in case of say 16 bit per channel RGB images we want to show the whole range (not clip or wrap around). In case of float images we cannot show the whole range (the image would be almost always gray). Forcing the user to normalize the image to a range [0, 1] is a solution but I feel too restrictive one. Float images would be mostly used for image processing. While these are mostly single channel images, color images can be used too with tools like adapt_rgb. Please also note in the adapt_rgb examples they normalize rescale_intensity(1 - sobel_each(image)). I wondered why the 1 was needed. If you omit it the image gets normalized to [-1, 1] and the colors get wrapped around by imshow.

@WeatherGod
Copy link
Member

WeatherGod commented Oct 18, 2017 via email

@jklymak
Copy link
Member

jklymak commented Oct 18, 2017

imshow accepts floats between 0. and 1 or uint8. I agree that it shouldn't round floats to uint8 unless it has to for rendering. I agree it probably shouldn't wrap, and clipping between 0 and 1 makes sense to me, or returning an error. I think it should definitely not normalize arbitrary data between 0 and 1.

It'd probably be easy to add support for 16-bit integers given that a lot of images are now 16 bits. But I guess there is a bit of an issue telling between a dark 16-bit image and an 8-bit image if the user doesn't specifically cast to the right-size uint.

@Zac-HD
Copy link
Contributor

Zac-HD commented Jan 10, 2018

tacaswell, I've recently been working on imshow in Xarray, where we concluded that the proper place for scaling and clipping fixes was upstream (ie here). My plan:

  • Support norm or vmin and vmax arguments with RGB[A] images. Among other things, this makes it quite trivial to support high-bit-depth images - just pass vmin=0, vmax=2**12 for 12-bit images, for example. Rejected by consensus on pull request.
  • If data are outside the valid range after scaling - that is, [0 .. 255] for integer types and [0 .. 1] for floating types - log a warning and then clip to the valid range before casting, to clip instead of modulo. Without this any pixel could represent a much higher or lower value than it seems.
  • Open a new issue for "colorbar should refuse to let it's self be created for an RGB(A) image" - it's a logically distinct change and I don't have the time to do both right now.

What do people think? If this seems reasonable I'll write a draft tomorrow 😄

@anntzer
Copy link
Contributor

anntzer commented Jan 10, 2018

Untagged from release critical as neither the wrapping and the inappropriate colorbar'ing are "new" bugs, and RGB normalization is another can of worms (see discussion followup in #10220).

@anntzer anntzer removed the Release critical For bugs that make the library unusable (segfaults, incorrect plots, etc) and major regressions. label Jan 10, 2018
@michalkahle
Copy link
Author

@Zac-HD I like the proposal. Even without normalization the out-of-range warning and clipping instead of modulo would still be nice improvement.

@Zac-HD
Copy link
Contributor

Zac-HD commented Jan 11, 2018

That's what I'm doing in the pull request!

(and normalisation will happen downstream in Xarray, which will cover most of my use at least 😉 )

@QuLogic QuLogic modified the milestones: needs sorting, v2.2.0 Feb 12, 2018
@matplotlib matplotlib deleted a comment from biswajitcsecu Sep 12, 2018
@denis-bz
Copy link

denis-bz commented Jul 4, 2021

In 3.4.2, @michalkahle's code (with subplot 143 = 142 as uint8) gives red colorbars. Is that expected ?

5jul-matplotlib-issue-9391

@jklymak
Copy link
Member

jklymak commented Jul 4, 2021

Yes colorbars do not do anything sensible for rgba data.

@shcrela
Copy link

shcrela commented Aug 29, 2022

I realize it's an old (and unfortunately closed) issue, but I stumbled upon this discussion when searching for solution to my problem.
Here are my two cents:
I am trying to use plt.imshow to represent simultaneously the concentrations (values from 0 to 1) of three different components as one 2D "RGB" image (note the quotes). For completeness, the sum of the three concentrations is equal to 1 for each individual pixel.
If I am to simply put each of the concentration maps in its' proper channel and plot the image as RGB, I will obtain a passable result. But there still remains the problem of the non-linearity of human eye perception. If I'm not mistaken, the common remedy would be the gamma-correction. This works well for single-channel images using the norm argument of imshow (like mpl.colors.PowerNorm(gamma=1/2.2) for example), but, alas, matplotlib doesn't seem to allow the normalization for multichannel images.

Some contributors (specifically @jklymak ) seem to consider that RGB images are not to be messed with. Why? What is your target audience? I can't think of a better way for representing multiple channels in one plot, other than this "hacking" of the RGB format. If someone does, I'd be more than happy to hear it.
Or, if RGB(A) is in fact considered sacred, could we add a flag to signal that it's a multichannel array and not an rgb?

There is, of course, a possibility to re-normalize my data, as some suggest - but I don't want to change my data, the values have a precise meaning as they are, I just want to change the visual perception of my data by the human observer.

I hope this little message will make some of the decision-makers reconsider adding the possibility to normalize the data regardless of the input data type (multichannel or not).

All in all, thanks a lot for all the work you invested so far and for all your continuous efforts to make matplotlib even better!

I'll add here some of the relevant issues and comments:
#18627
#10220 (comment)
#9391 (comment)
#10220 (comment)

@jklymak
Copy link
Member

jklymak commented Aug 29, 2022

Matplotlib shows the RGBA image as-is. There is nothing in Matplotlib that prevents the user from modifying a copy of their RGBA signal using gamma correction or whatever they want, and plotting that. Other packages, like scikit-image, provide many image manipulation tools.

@tacaswell
Copy link
Member

Currently imshow can do colormapping on images of scalar data or handle RGB(A) data exactly as handed in. I completely agree with Jody here that the best course of action is to write your self a function that does what ever normalization you want and then do

ax.imshow(my_to_rgb_fuction(my_data))

We did have a GSOC project to add bi-variate color maps, which ale less than you want, but in the same direction: #8738 . I think this is still an interesting extension (mostly related to generating the 2D colorbar or wheel). If you @shcrela would like to pick this up and get it going again we would be happy to work with you.

@shcrela
Copy link

shcrela commented Aug 30, 2022

Thank you for your comments @tacaswell, I am well aware of the possiblity to modify my data (or the copy of my data) and then plot it. But in my honest opinion, if the problem is not in the data itself, but purely in the visual representation of that data - that should be matplotlib's concern.
Other than that, if I modify the data, hovering over a pixel in an interactive backend would read non-sensical values (perhaps there is a workaround for this?)
Other minor nuisances would be explaining and documenting what is now this modified data you're plotting - can become mind boggling, how to inverse-transform it after .get_array(), or not to forget to transform it before set_data().

Multi-variate colormap could be a solution, but it seems to me much more complicated compared to the existing norm tool.

I'm writing all this just to make you aware that there are some people out there who might need this fonctionality.

In the short run, as @mkcor mentionned, Plotly seems to provide the solution.

Cheers!

@michalkahle
Copy link
Author

Slightly off-topic I wanted to thank the matplotlib maintainers for their hard work and especially for them to still being responsive and explaining their point of view. Even though they not always do what we want them to. 😉

@tacaswell
Copy link
Member

I'm writing all this just to make you aware that there are some people out there who might need this fonctionality.

We are well aware, I have personally relied on multi-variate color maps in my research in the past and had a GSOC student work on exactly this problem.

Multi-variate colormap could be a solution, but it seems to me much more complicated compared to the existing norm tool.

What you are asking for is a multi-variate color mapping where the channels are fully separable at the norm stage and then straight mapping the data channels to color channels.

Even though they not always do what we want them to.

Come join us! A vast majority of the development effort on Matplotlib is volunteer. As I said above, the multi-variate colormap work needs a champion to pick it back up, sort through the outstanding API concerns, and get it in. I do not have the bandwidth to lead that work, but would be very happy to support someone to do so.

@denis-bz
Copy link

@shcrela, "Plotly seems to provide the solution":
could you post a link, or links to other examples of your use cases ?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

9 participants