I agree this behavior is... less bad 😃

ping @efiring because I remember he had an opinion about this (though I don't rememebr what it was exactly)....

Only fails one image comparison in test_axes.py and I'd argue it now does the "right" thing except for the auto-scaling of the x and y limits:

Old:

New:

Fails the tests with datenumbers. That'll be a project for another day or evening... See todo above

Fails some image tests and a couple of x/y limits tests

It looks like this PR may have gotten lost along the way, and I think it would be an excellent addition to matplotlib. I have also been writing wrapper functions to help out with data before passing it into pcolormesh and was about to contribute something, but then found this old PR out there which looks great!

In particular, often the data I'm working with from models are given at the center of a cell and frequently do not include the edge data of the grid cells, so this would simplify a lot of the workflow when making meshed plots.

One aspect that hasn't been brought up here is the missing cyclic point in longitude that you get currently when just quick plotting a meshgrid on a map. Here is a quick polar plot demonstrating how this new method would help on a regular angular grid (by eliminating the blank space found at the wraparound).

I don't understand the doc build failure. It says there are two "Notes" section in pocolor, but I didn't add a "Notes" section....

ValueError: The section Notes appears twice in the docstring of <function Axes.pcolor at 0x7fc6d690c670> in /home/circleci/project/lib/matplotlib/__init__.py.

^ that error message arose because of an indentation error. 🙄

@greglucas Thanks for the comments. First, I'm still open to changing the name from dropdata Lets get some more opinions. I guess psychologically, I want this to sound like something you don't want to do, even if the default is to do it when necessary.

do you want to worry about whether someone uses dropdata=False without any X/Y input?

Data doesn't get dropped when there are no x/y vectors given, so I'm not inclined to change that behaviour. Again, I'm not looking at this as a way to shift the cells for the user, I'm looking at this as a simple way to give the user what they meant by giving us N values for x when Z is MxN rather than just arbitrarily throwing out data.

I went and looked at MATLAB's documentation for pcolor and saw that they state X/Y/Z are supposed to be the same size (and then they drop the last row/column), so I'm not even sure if they allow you to make Z be a different shape at all (have not tested this myself).

My thoughts in favor of preserving all data are that I'm usually plotting some function evaluated at grid cells, which turns into z = f(x, y). All x/y/z have the same exact shape and I can scatter plot those values. Now, I want to plot a surface and fill the space between the contours, so I would prefer to center the x/y in the middle of the cell where the function was evaluated.

Here is an example demonstrating a few of the different use cases.

import numpy as np
import matplotlib as mpl
import matplotlib.pyplot as plt
cmap = mpl.cm.get_cmap('viridis')
norm = mpl.colors.Normalize(0, 1)

x, y = np.arange(4), np.arange(4)
xx, yy = np.meshgrid(x, y)
z = (xx**2 + yy**2)/18
xedges, yedges = np.meshgrid(np.arange(5) - 0.5, np.arange(5) - 0.5)


def add_scatter(ax):
    ax.scatter(xx, yy, c=z, s=100, cmap=cmap, norm=norm, edgecolor='k')
    ax.set_xlim(-1, 4)
    ax.set_ylim(-1, 4)
    ax.set_aspect('equal')


fig, axarr = plt.subplots(1, 4, figsize=(8, 3), constrained_layout=True)

ax = axarr[0]
ax.set_title('pcolormesh(Z)')
ax.pcolormesh(z, cmap=cmap, norm=norm, edgecolor='w')
ax.quiver(xx, yy, np.ones(xx.shape), np.ones(yy.shape), scale=10, color='w', linewidth=2)
add_scatter(ax)

ax = axarr[1]
ax.set_title('pcolormesh(X, Y, Z)')
ax.pcolormesh(xx, yy, z, cmap=cmap, norm=norm, edgecolor='w')
ax.quiver(xx, yy, np.ones(xx.shape), np.ones(yy.shape), scale=10, color='w', linewidth=2)
add_scatter(ax)

ax = axarr[2]
ax.set_title('pcolormesh(X, Y, Z)\ndropdata=False')
ax.pcolormesh(xedges, yedges, z, cmap=cmap, norm=norm, edgecolor='w')
add_scatter(ax)

ax = axarr[3]
ax.set_title('imshow')
ax.imshow(z, origin='lower', cmap=cmap, norm=norm)
add_scatter(ax)

plt.show()

@greglucas Well obseved that dropping data and the edge/midpoint semantics are to seprate issues. BTW nice illustrative plots!

It is an important point whether X, Y denote edges or center positions.

Current state

X, Y is always edges.

For N patches, we need N+1 edges along a direction.

• If edges are not given, we can easily create them at positions range(N+1). --> Ok.
• If X, Y have size N+1, we can just use them. --> Ok.
• If X, Y have size N, we're missing the last edge.
  Current solution: Silently drop last value. --> May be problematic as some data is not visualized.

Possible changes

Alternatives for the last case:
a) Try to extrapolate an additional value (#9629 (comment)).
--> Could be constant or linearly extrapolated from the previous deltas. But might be too clever and give unexpected results in more complicated cases.
b) Just make it expicit, so that you need pcolor(X, Y, C, dropdata=True) to get the current behavor - possibly with warning exception (see c), d)) if not given.
c) Warn about dropped data.
d) Fail with an exception.
--> Might actually be reasonable ("In the face of ambiguity, remove the temptation to guess.").

If we go with cases b) or c), we need some alternative method/parameter, that actually supports X,Y,C of the same size because C = f(X, Y) is a common usecase. If we do not want b) for that, center positions would come into play.

Center positions is a whole new feature for pcolor. Some things to consider:

• Should be controlled by a new kwarg, proposal: xy_type.
• It's a separate issue from dropping data; and only related in the aspect, that it is an alternative way of plotting same-size X, Y, C, which may allow us to be more restrictive on the dropping.
• Should be implemented for pcolor and pcolormesh. Also check the relation to pcolorfast.
• For info: This is a bit similar to plt.step(where=...) https://matplotlib.org/devdocs/gallery/lines_bars_and_markers/step_demo.html. But I don't think we can immitate/learn from that. Color patches are too different from a height profile.

We have two ways pcolormesh can be called for Z of dimensions MxN:

1.x has length N+1, y M+1
2. x has length N, y M.

The first case is unambiguous. x and y specify cell edges, and Z sets the color between the cell edges.

The second case is formally ambiguous, but what do we think users actually mean when they pass this to pcolormesh? I'd argue they expect what dropdata=True gives us here 99.99% of the time, which is that x, y specify where the data was collected and they want to color a cell around that point with the color given by Z. Other meanings seems obscure enough to me that we should not complicate things by giving them an API. The only reason to provide a dropdata=True toggle is for back compatibility.

As for @greglucas other point about pcolormesh(Z), that seems an orthogonal issue to me; if the user doesn't specify x, y they will get some undefined default behaviour, and the one given is as good as any other.

Naively, I would expect pcolormesh(Z) and imshow(Z) to produce the same image (albeit with a different return-type). Right now, panels 1 and 4 are off by a half-step. In the image code there was the arbitrary decision to make the extent be shifted by 0.5.

It would be nice to be consistent in my opinion... If there is going to be breaking changes I think this would be a welcome one to tidy up as well.

Ok but again that’s really orthogonal to this PR.

Let's disregard backward compatibility for the moment. What would be a clean design given Z is MxN?

Option 1) only x ~ N+1, y ~ M+1 is accepted --> x, y define edges

Option 2) only x ~ N, y ~ M is accepted --> x, y define centers

One may be more intuitive than the other but both are technically clean APIs, with different semantics.

However I don't think, one should mix these implicitly. Deciding if values define edges or centers based on their shape is too much magic. If we want both behaviors in a single function, the clean way out is to let the user specify the meaning via an extra xy_type (name up for discussion) kwarg explicitly.

Part of the problem here is that the only perfectly general and unambiguous way to do a pcolor-type plot is to specify the edges. With an irregular grid and/or nonlinear scale, there is more than one way to generate a reasonable set of edges.

In a clean API:

1. Data would not be dropped, ever. If the dimensions don't match appropriately, raise a ValueError or TypeError.
2. The basic routine (let's just keep the pcolormesh name for now) would require edges, which could be specified via 1-D or 2-D arrays.
3. One or more routines would be supplied to generate edges from 1-D or 2-D arrays of centers.
4. An additional convenience routine would combine the edge generation with pcolormesh. This function might include the gouraud shading option, or that might also be a separate function.

I think that cramming too much functionality into a single function is counterproductive. It can be done, and maybe it is what we will end up with, but I think using two functions would lead to simpler code (both at the user level and at our level) and less confusion.

I kind of feel like we are letting perfect get in the way of fixing something that is terrible. Right now the way we deal with x ~ N, y ~ M in the worse way possible by dropping data, whereas I feel pretty strongly that most people who do this expect the data to be centred on x and y. I'm not convinced that making people call a helper function, or having yet another pcolormesh is the right way forward.

The xy_type kwarg is basically the same as dropdata, unless you think this should have a meaning if x ~ N+1.

I guess we could deprecate x ~ N. I guarantee we will get lots of feedback if we do that!

I'm very sympathetic with this PR, and with the "good" versus "perfect" argument; the data-dropping behavior is an unfortunate inheritance from Matlab, and certainly there are many cases where data are specified on centers. When the user actually wants the data-dropping behavior, the user could do it explicitly with simple indexing. I'm a little uncomfortable with the dropdata boolean kwarg and rcParam. How about setting out a strategy for getting rid of it:

1. Let it take None (use the default) or the string values, "allow", "warn", "error".
2. Start with "allow" in the rcParams, but urge users to manually set the default to "error", noting that it will eventually go away.
3. With successive releases, change the rcParams default to "warn" and then "error".
4. Then remove it.

Maybe that's too much fuss, but it would be nice to have a strategy for ending up with a cleaner API instead of just settling for another kwarg forever.

Use of a different function for center-specification was implicit in my comment above, so it doesn't address the desire to have one do-everything function that figures out what to do based partly on the dimensions of the inputs. The PR is certainly trying to be friendly, handling edge versus center independently for each dimension. That's not terribly magical; it's pretty easy to understand and explain. (The docstring modifications in this PR don't quite go far enough in describing the new situation, but there is no point in doing more work on them until we are sure of the API.)

We already have a "shading" kwarg, which actually affects the meaning of the X, Y locations: with gouraud shading, (X, Y) specify the corner colors, and the plot does not extend beyond the X, Y points. This is similar to X, Y specifying centers (as in imshow for uniform grids), the differences being (1) that the colors are blocks rather than gradients, and (2) the domain is expanded slightly so that for uniform grids the blocks are all the same size. This second difference is not essential; it would be possible to handle center-specification, with X, Y, C all of the same dimensions, using half-cells on the edges so that the domain would be the same as for gouraud shading. The advantage is that this interpolates but does not extrapolate. The disadvantage is that most often the user probably will want more image-like behavior.

Would it make sense to supersede "shading" with (perhaps) "mode", which would take values of

• "legacy" (or "dropdata"?)
• "gouraud"
• "auto" which would have the dropdata=False behavior of this PR
• "edge" which would enforce edge specification
• "center" enforcing center specification

Or maybe just use the first 3 of these. And maybe leave the name as "shading" so as to minimize the changes. The advantage of keeping "edge" and "center" is code readability: it would make clear to the reader what sort of data input is required. If they are used, though, the kwarg name "shading" becomes problematic.

TL;DR I am 👎 on putting in logic to try and guess the cell edges, I think it opens up a can of worms we don't want to get into (as exemplified by the suggestion from @pharshalp for detecting log spaced values).

I am more amenable to changing the default behavior with no X or Y to match imshow (center the cells on the integers rather than anchor a corner on the integers), but that is an orthogonal discussion (and in that case the user probably should be using imshow anyway).

I am 👍 on including the centers -> edges code as a public helper in cbook, probably in both a linear and log sampled flavors.

I swear this started as a short comment and then I went and wrote an essay...

Right now the way we deal with x ~ N, y ~ M in the worse way possible by dropping data

I disagree that the current behavior is the worst possible solution, I think incorrectly inferring the edges would be worse behavior.

"dropping" the last row / column is consistent and correct in the sense that the patch defined by X[n:n+1], Y[n:n+1] is colored by the value in C[n]. The docstring of pcolormesh has this diagram:

(X[i+1, j], Y[i+1, j])          (X[i+1, j+1], Y[i+1, j+1])
                      +--------+
                      | C[i,j] |
                      +--------+
    (X[i, j], Y[i, j])          (X[i, j+1], Y[i, j+1]),

If you take that as the invanint of pcolor*, then the behavior of dropping the last row / column of the color input is justifiable nicety to save your users from typing C[:-1, :-1] (or in the case of MATLAB C(1:end-1, 1:end-1)).

Thinking about how the shading works, this actually becomes even more defendable. In pcolormesh, if the 'flat' shading is thought of as the 0-order interpolation and the input giving use the value of C at every vertex and we use that value to color the cell the vertex is the lower-left (using the conventions of that diagram) corner of. We are then we are not dropping data, we are just getting data from the user that indentaly does not participate in the interpolation. If you use the gouraud shading then the color within the patch depends on the values of all 4 vertices. Looking at it from this point of view, accepting the X/Y and C as off-by-one shapes is the convenience as it does not require the user to provide "junk" data that we are never going to use.

I can think of stories where the "natural" representation in the users code is either with matching or off-by-one sized arrays.

As @greglucas pointed out this kwarg is completely changing the meaning of the inputs. That needs to be called out much more clearly.

@greglucas : those are amazing examples, no matter how this PR ends up, I am in favor of those ending up in the documentation (as either an example or a tutorial) someplace.

Maybe that's too much fuss, but it would be nice to have a strategy for ending up with a cleaner API instead of just settling for another kwarg forever.

This is one of the core reasons I want to push the API to be many top-level functions rather than as Axes methods as being able to experiment with various APIs as stand-alone packages / have several variants co-exist cleanly. I prefer many simple functions over a smaller number of complex functions (which is consistent with the library PoV, from the application PoV fewer more complex functions may be better because (I am told) people find it easier to remember).

As @efiring points out, once you have a non-uniform grid then you need to specify the edges of the cells, not just one point in them.

If you have uniform square grid of data (for example, from a pixelated detector), then to uniquely place the cells in data space you only need to know the bounding box of the whole image, the size of the input, and then the index of the cell in question (eliding the issues of up / down sampling here, lets stick with "nearest" lots of up-sampling for now). You could also specify and offset within the cell of where "the value is", but that is degenerate with translating the bounding box. Internally, we always think of the value as being on the corner (this ends up being more complicated than one expects, see https://matplotlib.org/3.1.1/tutorials/intermediate/imshow_extent.html). When the user does not specify the extent, we pick one that puts the center of the cells at the integers (which is the behavior you want when working with pixel detectors and images in general).

On the other hand, once you allow non-uniform spacing, many bets are off (we are still assuming a rectangular grid). We are still assuming that the data is contiguous so we can just specify the vertices and then construct the patches based on the neighbors, but this does lead to the off-by-one issue (which is a pretty common thing, see histograms).

To go one assumption less (that the data is contiguous) then you have to go to PatchCollection directly and you need 4 scalars per cell and putting them in a rectangular array is no longer helpful (to Matplotlib, user side might still be useful).

As @WeatherGod points out, if the user is naturally thinking in centers or edges depends on the user and the field. I think having helpers that do the conversion for the common cases is a good idea.

If I am reading the edge computing code correctly, I would not call that linear interpolation as it is just re-using the width of the second to last cell for the width of the last cell. Is there prior art to reference on this? I would not be surprised if this function already existed in scipy or skimage.

This is an interesting partner to the discussion @anntzer was leading about making step be more forgiving about N / N+1 length inputs.

the behavior of dropping the last row / column of the color input is justifiable nicety to save your users from typing C[:-1, :-1]

Nicety, or footgun, depending on how you look at it. I personally think that silently dropping data is literally one of the worse things we can do as a library (insert rant about how each data point may be years in the life of a grad student). On the other hand my personal solution is to use my own private wrappers to imshow() and friends to avoid that... (so I already moved past these discussions :-))

This is an interesting partner to the discussion @anntzer was leading about making step be more forgiving about N / N+1 length inputs.

In a sense supporting N/N+1 in step plots is similar, right now step plots "drop" the first or last point (but not completely because there's still a vertical segment leading to it) and supporting N/N+1 would avoid that.

Somewhat unrelatedly, actually in the axes-aligned case I guess an improvement would be to return a NonUniformImage, which is likely much faster to render than even a QuadMesh?

Thanks for the great comments. A couple of small ones for you to chew on:

If your data Z is collected on your grid points, and we want to think about this as interpolation between those points , this algorithm implements (linear) nearest-neighbor between the provided points, which is usually the lowest-order interpolation. The current behaviour implements nearest-to-the-left, which is just weird.

I agree with everything said above about how specifying the edges is the "unambiguous" thing to do. And I also agree that just dropping the data is unambiguous. But I've not seen any argument that this is what the naive user expects, and I think its a terrible thing to do silently to someone's data set when they are not expecting it.

If the way forward is deprecating mismatched x, y, Z, and providing helpers, I'd be fine with that. But I think the first user request would be to wrap that in a kwarg to pcolormesh, so...

See Axes.pcolorfast.

Ah yes, sorry I forgot that. Anyways that discussion point is orthogonal to the main issue at hand here.

The current behaviour implements nearest-to-the-left, which is just weird.

That is if you are saying the point at C[n,m] is associated with the (X[n,m], Y[n, m]). Instead it is associated with 4 points in the X/Y data as an "intercalated" data set. Something like

x x x x
 o o o
x x x x
 o o o
x x x x

Where x is the location data and y is the value data. Although we can't express it in strided arrays, it make be best to think of the value data as being at half-indexes.

But I've not seen any argument that this is what the naive user expects, and I think its a terrible thing to do silently to someone's data set when they are not expecting it.

The naive user also would not expect that a kwarg would completely change the semantics of the input and makes the function a fundamentally different function.

To be fair, we already have shades of that with the shading kwarg, but there you get to pick your poison of "nearest-up-and-left" interpolation or semantic changing kwarg.

This also suggests that the core of the problem here is a documentation one not a functionality one.

But I think the first user request would be to wrap that in a kwarg to pcolormesh, so...

If that is a case then this needs to be very clearly documented in the docstring that setting a kwarg will radically change the behavior of the function.

Let's start with just a warning and see how much feedback we get. The warning should be something like: "We think you did this by accident, if you meant to do this please let us know why".

I think we should also change the first line of the docstrings to make clearer these functions take edges not centers as input.

That is if you are saying the point at C[n,m] is associated with the (X[n,m], Y[n, m]). Instead it is associated with 4 points in the X/Y data as an "intercalated" data set.

@tacaswell, just to be clear I'm not simply "saying" this: its how almost every data set I've ever seen is organized: the data is collected at X[n, m] and Y[n, m] so those are the values stored. I think you'd be hard pressed to find a lot of data sets that are organized differently, unless they come from numerical models. If you want specific examples where X and Y have the same dimensions as C:

• https://www.esrl.noaa.gov/psd/data/gridded/data.ncep.reanalysis.surface.html
• https://podaac.jpl.nasa.gov/dataset/REYNOLDS_NCDC_L4_MONTHLY_V4
  but there are infinitely more. People expect to be able to load those data sets and do pcolormesh(x, y, Z) without going through a manual regrinding step.

If you want prior art, xarray has an infer_intervals that defaults to True (unless X and Y are matrices and we are dealing with a projection, in which case they don't do it). The logic is here: https://github.com/pydata/xarray/blob/db36c5c0cdee2f5313a81fdeca8a8ae5491d1c8f/xarray/plot/plot.py#L959

Seaborn's heatmap only works the way that this PR works.

GMT assumes the data is centered on the grid for grdimage.

Let's start with just a warning and see how much feedback we get. The warning should be something like: "We think you did this by accident, if you meant to do this please let us know why".

We can do that, (and add the helper), but many people are going to say "I didn't do this by accident, why doesn't it work?". I really think this is a super-common use case.

I agree completely with @jklymak--in physical oceanography and meteorology, gridded data products and numerical model output is usually provided with the center positions as the grid. Sometimes there are "boundary" grids, but the ones I have seen are (N,2), not N+1. Being able to handle potentially non-uniform center grids easily and efficiently would be a real convenience, and an improvement over the present quick and dirty workaround of dropping a row and a column and ignoring the systematic half-grid position error. Which is probably what all of us do when we are in a hurry, which is most of the time.
The question is therefore how to provide this major user convenience with a clean, simple, unambiguous, preferably explicit API. Hence my general suggestions: either use a separate function name for the center case, or use a kwarg design that is clear and readable. Then it all comes down to the problem of naming, which is hard. But maybe not impossible. And there is also the horrible backward-compatibility, slow deprecation mess to deal with. Putting the center-based version in its own function requires coming up with a name (I don't have a good suggestion so far), but apart from that it is likely the cleanest approach, with the simplest compatibility and deprecation pathway.

Incidentally, the NonUniformImage class, which we have had since early days but which is probably rarely used, already handles this centered-grid case. I modified it to make the PcolorImage class which is used in pcolorfast to speed up the common case of a nonuniform but rectilinear set of boundaries.

I'd be happy to put the example in the docs. I am going to wait until the discussion settles on the implementation specifics (kwarg, new function, etc...).

My personal opinion is that, yes this is a highly desired option/default. Another common 'gotcha' here that this would fix is when I do animations and call coll.set_array(Z) to update the data in the meshgrid. But, I didn't realize that the data in the stored array is not the same size as the initial data I passed in... Again, that is another orthogonal issue to the main code here though (updating the set_array machinery to have more helpful checks/warnings to users).

My preferences would be to:

• Move this 'calculate edges from centers' code to the cbook.
• Add a new kwarg to pcolor* that would call that cbook helper.
• Add a warning to the current pcolor* if the sizes of X/Y aren't N+1 and suggest calculating edges themselves or using the new kwarg.
• Make the kwarg default in mpl 4+ to prevent dropping any data.

There are currently too many functions to remember what they all do, in my opinion, to warrant yet another one. I often forget that pcolorfast, pcolorimage even exist, so adding another function would just add something else that I forget when working fast.

@tacaswell should this be discussed at a weekly call, are you onboard for the solutions listed above, or should this just get the warning and we can see how much pushback there is?

Please see #16258 for a new version of this.