@kdpenner Please supply the code used to generate the before/after example.

https://gist.github.com/kdpenner/5dd18a42d68ea144fcb5aa7e37a1d849

Looking at this has revealed a bug in master that also affects the PR, so my immediate priority is to get that fixed. My first reaction to this PR, though, is that it illustrates a way of using colorbar that was not originally intended, but that looks like a reasonable use case. I will probably have some questions about the implementation.

OK---it's not like I'm going anywhere soon.

GMT generates colorbars like these for this colormap.

I'm sorry to have let this languish. I simply forgot about it.

For this to be discoverable, and hence useful for anyone but the author, I think it will need an advanced tutorial or addition to an existing tutorial. I think the use case that it is adding will not be common, but will be useful enough to justify inclusion.

Some rearrangement of _process_values will be needed, possibly factoring the new use case out into a helper; as it stands, the new use case is adding an up-front big block of (rarely used) code plus double conditionals that push it over the readability cliff.

The code modification should be straightforward; what will take the most work to complete this PR will be adding tests and documentation, including the whatsnew and tutorial addition. It would be nice to see a simple example inspired by a real application, demonstrating why and how one would use this sort of colorbar and colormap.

Pinging @jklymak also, since you have done a lot of colorbar work.

I can't easily review this without any sort of description of what it is meant to do, preferably with code. If its fixing the off-by-one bug that boundaries seem to have, that seems useful.

Sorry, I didn't see the gist. I've simply added it above...

Another question: do we need to preserve the old behavior? Is anyone using it? If so, we need to make the new behavior opt-in. I think that would require a keyword argument; I don't think an rcParam would be desirable here.

I guess I don't really understand what boundaries and values are meant to do for colorbars. I don't understand why we want the colorbar to care about those things explicitly, rather than doing wha I think the original poster is try to do through the norm and/or colormap. I don't understand how the data would be plotted in a way to correspond to these colorbars.

I was wondering the same thing. After staring at it and playing with it for a couple hours, I couldn't see how to get the same effect with a norm and cmap. The key is that in this PR, the boundaries are used to divide the colorbar into sections, with a linear mapping in each section. Notice that in the gist, there is one norm and one cmap, and the only thing that is changed is the boundaries, which delimit the overall range shown as well as the linear segments. Getting the 6 different versions without this PR, using norm and cmap, would require multiple norms and maps, I think.

Problem: notice that in the gist, the colormap is generated with N=5000 entries in the LUT! It turns out that the algorithm is sensitive to this, and using a more normal value like 500 results in very different output for the colorbar. This seems like a weakness in the approach. It is putting all of the responsibility for resolving small fractions of a large range on the colormap LUT instead of delegating that scaling to a norm.

Boundaries and values: the basic idea is that a colormap is always a sequence of blocks of color--as in a pcolormesh with a single column--so we need the boundaries of those blocks and the values, or colors, to be assigned to each. In the most common case for image-like plots, the norm and cmap have enough information to calculate the boundaries and values. But there are other cases where we might want to control one or both of them directly, hence the additional logic in _process_values.

Right, but i think what this user really wants is for the colormap to have those constant regions. Just making the colorbar have them won't help when plotting. For bathymetry they may like a nonlinear norm as well. This seems more appropriate for an example mixing norms and colormap creation. Trying to do this in colorbars doesn't help.

The norm and cmap are used normally in plotting. I presume this is inspired by something like topographic mapping; there is a "shallow water" color for 0-100, then a gradient, with one or more thin ranges of contrasting color to act as pseudo-contours. The point of the PR is to make a colorbar that can show this mixture of region types in a comprehensible way.

@kdpenner, can you point to applications? You mentioned GMT: is there an online example like this with GMT?

In https://matplotlib.org/tutorials/colors/colorbar_only.html#sphx-glr-tutorials-colors-colorbar-only-py we give examples in which boundaries are specified in the colorbar call. They appear to work with this PR, as before.

Right, but there they use a BoundaryNorm which makes sense with bounds. ( I guess the only thing that confuses me is why the colorbar call needs any extra arguments at all in that case. )

But again, it this case, this PR is making a fancy colorbar with a mix of boundaries and linear ramps, but they are not making the norm that can actually allow that colorbar to be applied to data. After that, yes I agree that the colorbar should follow the Norm.

I'd love to see us move to a model where the colorbar just does everything automatically from the Norm, plus or minus a few extra arguments, but we aren't quite there yet.

I'm here---no need to assume what I want. I'm busy today and will respond tomorrow.

Great! An example with a pcolor plot would help us understand.

In the interim: the gist makes both a norm and a cmap that goes with it; using them in pcolormesh or imshow gives the expected result. The norm is linear; all the structure is in the colormap. This is the opposite of the way we mostly think about colormaps--we emphasize those that are visually linear, and tell people to put the structure they want (e.g., log scaling) in the norm.

I want the colorbar to match the colormap if I pass boundaries. With the old behavior, if boundaries is passed, even a dead simple viridis colormap is shown incorrectly, because colorbar shows one color between 2 boundaries. (My example is perhaps unnecessarily complicated, but it does show that the PR handles whatever colormap you throw at it.) A few responses:

I think it will need an advanced tutorial or addition to an existing tutorial.

I agree and am happy to write one.

Another question: do we need to preserve the old behavior? Is anyone using it? If so, we need to make the new behavior opt-in. I think that would require a keyword argument; I don't think an rcParam would be desirable here.

I argue for making values obsolete.

Passing boundaries gives a deprecation warning in mpl 3.3. If y'all have decided to remove the functionality, this PR isn't worth pursuing---but I do use boundaries quite a bit, because...

this is inspired by something like topographic mapping

Problem: notice that in the gist, the colormap is generated with N=5000 entries in the LUT! It turns out that the algorithm is sensitive to this, and using a more normal value like 500 results in very different output for the colorbar. This seems like a weakness in the approach. It is putting all of the responsibility for resolving small fractions of a large range on the colormap LUT instead of delegating that scaling to a norm.

Yes; for the colorbar to accurately represent the colormap, the number of entries in the LUT increases as the complexity of the colormap increases.

This is the opposite of the way we mostly think about colormaps--we emphasize those that are visually linear, and tell people to put the structure they want (e.g., log scaling) in the norm.

Sure, but see my point at the beginning---I want colorbar to match the colormap if I pass boundaries.

... sure, but what I don't understand is how you are making the colormap.

@jklymak I think that code is at the top of this PR.

Hm. I don't understand. I'm using LinearSegmentedColormap.

A LinearSegmentedColormap interpolates between the colors linearly. It won't make a range of data a single solid color as you do for 0-50, 50-100 and 100-200 in the above plot. The only way I think this is possible is if you make two of the colors the same in the colormap, which I do not see that you are doing, or you make the norm round some of the colors to a certain value. So I do not understand how the provided colormap and norm are able to produce a figure like you have shown. I don't doubt that you achieved it, but a minimal working example would certainly clear up my confusion.

Gotcha. Interpolation returns a constant if there's no variation. From the red channel:

(norm(-200.), 182./255., 14./255.),
(norm(-100.), 14./255., 125./255.),
(norm(-50.), 125./255., 234./255.)

which means between -200 and -100 the red channel is a constant 14/255, etc.

Ok sorry I get it now. Thanks.

So the desire here is that the colormap and norm are normal but the scale on the colorbar is displayed non-linearly. If I were doing this I'd make a new scale rather than trying to use the colorbar boundaries machinery which I don't think was meant for this. We have a func scale that may be helpful?

I'll look into scale. What is boundaries meant for?

Boundaries and values were both meant to provide full control over the contents of a colorbar as an ordered sequence of colored intervals, potentially of varying lengths. The original idea was that there would be only one color between a pair of boundaries, hence the present behavior. Your PR is extending the meaning to a colorbar as a sequence of intervals within which there may be a single color or a sequence of colored blocks.

In your Gulf of Mexico example, why does 0-50 m range on the map appear white? It does not match the colorbar, as far as I can see.
Edited: I think it is an optical illusion--it looks white because of its surroundings, but it actually matches the pale color on the colorbar.

So first, you can specify the tick marks, but in that case the 50 and the 100 would be very close together and the other values their normal linear spread apart. I assume you do not want that?

So, if i understand correctly, you would like a way for "values" to be linearly distributed across the colorbar such that each "value" is placed equidistance along the colorbar? I guess my understanding is that this is not what boundaries and values was really meant to do, but I'm not saying it could not be used for that meaning.

However, we've been trying to move colorbar axes to being the same as regular axes. So the way to do this on a regular axes would be to define a scale, perhaps using FuncScale (https://matplotlib.org/3.1.0/api/scale_api.html#matplotlib.scale.FuncScale), or if we think this is useful enough, with a new scale. I'd prefer to see that, rather than a new overload using boundaries and values. But, perhaps there is a subtlety I'm missing. You shouldn't trust too well someone who can't read a LinearSegmentedColormap clearly.

This basically does what you want, but for some reason the extends don't work. Note I simplified your code a fair bit.

import numpy as np
import matplotlib.colors as colors
import matplotlib.pyplot as plt
import matplotlib.cm as cm

norm = colors.Normalize(vmin = -8000, vmax = 0)
cdict = {'red': [(norm(-8000.), 0., 0.),
                 (norm(-6000.), 0., 0.),
                 (norm(-4000.), 0., 0.),
                 (norm(-2010.), 69./255., 1.),
                 #(norm(-2000.), 69./255., 69./255.),
                 (norm(-1990.), 1., 69./255.),
                 (norm(-200.), 182./255., 14./255.),
                 (norm(-100.), 14./255., 125./255.),
                 (norm(-50.), 125./255., 234./255.),
                 (norm(0.), 234./255., 0.)],
         'green': [(norm(-8000.), 0., 0.),
                   (norm(-6000.), 8./255., 8./255.),
                   (norm(-4000.), 94./255., 94./255.),
                   (norm(-2010.), 188./255., 0.),
                   #(norm(-2000.), 188./255., 188./255.),
                   (norm(-1990.), 0., 188./255.),
                   (norm(-200.), 244./255., 147./255.),
                   (norm(-100.), 147./255., 174./255.),
                   (norm(-50.), 174./255., 254./255.),
                   (norm(0.), 254./255., 1.)],
         'blue': [ (norm(-8000.), 0., 0.),
                   (norm(-6000.), 40./255., 40./255.),
                   (norm(-4000.), 140./255., 140./255.),
                   (norm(-2010.), 187./255., 0.),
                   #(norm(-2000.), 187./255., 187./255.),
                   (norm(-1990.), 0., 187./255.),
                   (norm(-200.), 185./255., 229./255.),
                   (norm(-100.), 229./255., 237./255.),
                   (norm(-50.), 237./255., 1.),
                   (norm(0.), 1., 0.)]}

cmap = colors.LinearSegmentedColormap('blah',
                                      segmentdata = cdict,
                                      N=2560)
cmap.set_over('r')
cmap.set_under('k')

fig, ax = plt.subplots( figsize = (4, 7))

boundaries = [[-8000, -6000, -4000, -2000, -1000, -200, -100, -50, 0.],
              [-8000, -6000, -4000, -200, -100, -50, 0.],
              [-8000, -4000, -2000, -200, -100, -50, 0.],
              [-8000, -6000, -2000, -200, -100, -50, 0.],
              [-8000, -6000, -4000, -2000, -100, 0, 10.],
              [-6000, -2000, -200, -100, 0.]]


cb = fig.colorbar(cm.ScalarMappable(norm = norm, cmap = cmap),
                  cax = ax, extend = 'both')

i = 2

def forward(x):
    N = len(boundaries[i])
    x = np.interp(x, np.array(boundaries[i]), np.linspace(0, 1, N))
    return x

def inverse(x):
    N = len(boundaries[i])
    x = np.interp(x, np.linspace(0, 1, N) , boundaries[i])
    return x

cb.ax.set_yscale('function', functions=(forward, inverse))
cb.set_ticks(boundaries[i])

plt.savefig('t.png', bbox_inches = 'tight', dpi = 100)
plt.show()
plt.close()

Note I didn't plot all the permutations because the forward/inverse functions are persistent, and you end up with the last one...

So to me, the problem is that the extends are not working for some reason.

I spent some time with this last night, and its not super trivial to fix the way we make colorbars, but I actually think that we should fix it.

Currently we only support linear and logarithmic scales on colorbars. As we see above, if there are no extend triangles (extends) then other arbitrary scales work just fine. The pcolormesh that makes the colorbar is defined in data space so attaching whatever scale we want is easy.

However, the extends are also defined on the same pcolormesh, and in data space, as triangles that go from vmax to vmax+0.05*(vmax-vmin). However the extends are really meant to be triangles in axes space, not data space, so applying an arbitrary scale past vmin/vmax causes them to be distorted. In the case above, I didn't provide an extrapolation past boundary[0]/boundary[-1], so the extends were mapped to vmin/vmax, and had no size.

I think the solution is to still make the pcolormesh in data space on an axes that goes from vmin to vmax., so arbitrary scales can be applied, but to make the extends single-colour patches in axes space. This will actually greatly simplify a good amount of code in colorbar. The only fudge will be that the actual axes will have to shrink or expand a bit, depending on the presence of extends.

I think this change would be a bit of a re-architecture, but I think the cost-benefit would be worth it to make the scale properties of colorbars more flexible. As we move to making norms and scales much more in sync, it makes sense for the colorbar to follow the norm's scale. I suspect the cost, however, is that almost all colorbars with extends will change slightly. I'm skeptical that we can make the extend-patches look exactly the same as an extra pcolormesh cell. So we will have a bunch of images regenerated.

OK, that is all orthogonal to the current PR. The current PR seeks to override boundaries to essentially create a new scale that has equal-spaced boundaries and a linear ramp in data space between those boundaries. This could be thought of as a short-cut to creating a scale as I did in the comment above. Ignoring the implementation, I'm not 100% against this idea, but I'm not sure if we should support it versus just asking the user to create a scale. I think a linear-piecewise scale is something that could easily be added to scale.py. Implementation wise, I'm against manually making a scale as this PR does, versus using the scale machinery, because that is orthogonal to the direction we are trying to push colorbars. OTOH, if making colorbars more flexible WRT scales fails in the v3.4 timeframe, perhaps this idea should be re-visited. Its certainly very reasonable in the current paradigm.

I think the solution is to still make the pcolormesh in data space on an axes that goes from vmin to vmax., so arbitrary scales can be applied, but to make the extends single-colour patches in axes space.

With the additional complication that 2 more parameters, the data values at which the extend triangles start, are needed.

I quibble with the triangles being single-color patches, but having them show the structure of the colormap is probably a refinement for later.

Would using an Arrow resolve the data vs. axes space complication?

The triangles are definitely single color. They are the over/under colors.

Yeah, single-color patches make sense for many, but not all, colormaps, so I have this quibble with current behavior, too.

I mean, there are only one each over/under color. What else would you expect the triangles to mean?

In my gist, I use extend but haven't defined over and under colors---I've defined a full colormap. If I cut the colorbar at -4000 and didn't know better, I'd naively expect the triangle to show the gradient in the colormap.

Have y'all decided if the refactor makes my PR unnecessary?

I'm actively working on the refactor, but am currently against more overloading of the already complicated API. I really don't think this should be declared at the Norm level, and hence the refactor to make that more straight forward.

OK. I'm happy to write a tutorial or example of the scale functionality.

I believe this refactor is #18900.

Works as expected:

Now I think the documentation should be updated to include notes about: 1) using FuncScale to control ticks and tick spacing in this context; 2) the difference between a scale and boundaries + values; and 3) the forward and inverse functions of a scale.

Re: 2): I'm not sure why values exists. To take my example, why would someone want to map the color shown between 0 and -50 to the color value between -50 and -100?

Re: 3): there's no discussion of what the forward and inverse functions do, what arguments they accept, and what they return. I had to reconstruct that the forward function takes in a data value and outputs a normalized position and that the inverse function takes in a normalized position and outputs a data value.

Edit: disregard my point about 2

The colorbar overhaul will be in 3.5 (I hope) and then we can come back to this after it gets merged, again, hopefully for 3.5

@kdpenner, if you wanted to take a stab at this given that #20054 (originally #18900) is in, that would be great.

@jklymak Your PR looks good and obviates the need for mine; I'll close this one. However, I still think some work on the documentation is needed. I also like @efiring 's suggestion of a tutorial, but y'all are better plugged in to the uniqueness of my use case and thus the usefulness of a tutorial.