Ensure streamplot Euler step is always called when going out of bounds. #11947

tomflannaghan · 2018-08-27T09:07:32Z

PR Summary

When the integrator goes out of bounds of the grid, it can perform a step using Euler integration to make the trajectory stop right at the edge of the grid, rather than stopping before reaching the edge. The current implementation relies on an IndexError being thrown to perform an Euler step, and there are several cases in which this won't happen.

It won't currently check the end point, but only intermediate points
It doesn't always raise an IndexError if the trajectory leaves the left or bottom as negative indices are valid.

Here I add checks explicitly to the integrator rather than relying on IndexErrors to fix both cases. This results in neater figures without ragged edges.

I also spotted that we were using dmap.grid.nx for the width of the grid, which should be dmap.grid.nx - 1 (and similarly for ny) as the grid goes from 0 to nx-1. This also means that the rightmost and uppermost edges of the mask now correspond to the rightmost and uppermost edges of the data, rather than sitting outside the grid (and therefore being unused).

This change changes all test results for streamplot as streamlines go right to the edge of the plot now. For example,

Before:

After:

PR Checklist

Has Pytest style unit tests
Code is Flake 8 compliant
New features are documented, with examples if plot related
Documentation is sphinx and numpydoc compliant
Added an entry to doc/users/next_whats_new/ if major new feature (follow instructions in README.rst there)
Documented in doc/api/api_changes.rst if API changed in a backward-incompatible way

jklymak · 2018-08-27T15:33:03Z

I seem to remember someone already fixed something similar. Can you make sure this is still a problem in master? Thanks!

tacaswell · 2018-08-27T15:41:36Z

lib/matplotlib/streamplot.py

@@ -473,6 +473,10 @@ def integrate(x0, y0):
    return integrate


+class OutOfBounds(Exception):


might make sense to inherit from IndexError here and still catch IndexError below (so in the case where f does still raise it (not sure if that is possible) everything still works?

Yep good point. I think the code shouldn't raise IndexError any more as we are checking bounds prior to interpolating, but I guess a bit safer to do as you suggest so I have pushed this to the PR now. Thanks!

tacaswell · 2018-08-27T15:43:26Z

lib/matplotlib/streamplot.py

+            else:
+                raise OutOfBounds
+
+        except OutOfBounds:


If all of the raising logic is here it might be clearer to just use if and else statements.

The raising logic is all here directly within the try block, including line 529 as well (so can't simply be put in the else). I can rewrite it using a flag variable like so. What do you think? Happy to update if you prefer it.

try: out_of_bounds = False if dmap.grid.within_grid(xi, yi): xf_traj.append(xi) yf_traj.append(yi) k1x, k1y = f(xi, yi) if dmap.grid.within_grid(xi + ds * k1x, yi + ds * k1y): k2x, k2y = f(xi + ds * k1x, yi + ds * k1y) else: out_of_bounds = True else: out_of_bounds = True if out_of_bounds: # Out of the domain during this step. # Take an Euler step to the boundary to improve neatness # unless the trajectory is currently empty. if xf_traj: ds, xf_traj, yf_traj = _euler_step(xf_traj, yf_traj, dmap, f) stotal += ds break except TerminateTrajectory: break

Another simple approach could be to move the bounds check and raise OutOfbounds into f. Would simplify the logic in the interpolator.

I've pushed a change to the PR moving the raise OutOfBounds into f. I tried quite a few things and I think this is the clearest.

tacaswell · 2018-08-27T16:00:57Z

Thanks for your work on this! Left a few comments in-line about the exception handling logic.

I am a bit concerned about the - 1 change, that seems to be a pretty big change to the integrator?

tomflannaghan · 2018-08-27T17:06:23Z

@tacaswell thanks a lot for the feedback. I'll reply inline to the code comments. The - 1 change is most obvious at low densities as it makes a row and a column of the mask redundant. These plots show density=3/30 which gives a mask size of 3x3. Without the - 1, the last row and column are off the grid so it appears only to be 2x2.

Before

After

@jklymak I've checked and the issue is still in master. The image in the PR that shows the problem is from the master tests here.

tacaswell · 2018-08-27T17:51:46Z

so the - 1 change is mostly about deciding if we need a stream line is a given location, not the computation of stream line once we decide we need one?

tomflannaghan · 2018-08-27T18:12:27Z

Yes, the main difference is that it fixes the transforms between mask (which is what decides where to make streamlines) and grid. This is what drew my attention to the - 1 issue.

The change does have a very small effect on the integration too though, as it rescales the error and speed by nx / (nx - 1) so they are in correct axes coordinates.

The change makes no difference to the direction of trajectories as u and v are already transformed correctly and interpolation isn't affected.

jklymak · 2018-09-01T04:57:33Z

OK, but i'm not clear that it shouldn't be mask.nx/grid.nx with no -1 on both of them. I think I agree that they should be the same, however... I haven't given it much thought, but a quick skim of grid2mask would seem to bear that out.

tomflannaghan · 2018-09-01T10:28:49Z

Another way to explain the -1 change in grid2mask is the following. The grid and mask are both ny by nx arrays, so the index [grid.ny-1, grid.nx-1] is the upper right corner of the data. This should map to the upper right corner of the mask which is at [mask.ny-1, mask.nx-1]. For grid2mask to do this correctly, it must map index xg in the grid to xm in the mask by

xm = xg * (mask.nx - 1) / (grid.nx - 1)

If you substitute in the example xg = grid.nx - 1, this gives xm = mask.nx - 1.

The current code, which does xm = xg * (mask.nx - 1) / grid.nx gets this slightly wrong. It would map the top right of the grid to [(mask.ny - 1) * (grid.ny - 1) / grid.ny, (mask.nx- 1) * (grid.nx - 1) / grid.nx]. In practice this means that the top-most and right-most columns of mask map to locations outside the grid.

Please let me know if anything doesn't make sense. Very happy to clarify further.

jklymak · 2018-09-01T18:13:56Z

No that’s clear. I just would usually expect that streamlines and data points would be offset from each other by half a grid cell

tomflannaghan · 2018-09-03T20:29:02Z

That's not the case here - they are on different resolution grids but not offset. The images I added above with the mask size of 3 show that both with and without the change, the streamlines start at the corners of the grid. The difference is that with the change, the whole of the mask is used.

jklymak · 2018-10-26T21:08:54Z

@tomflannaghan Sorry if this fell off the radar. It needs a rebase, and I'll probably need to spend a bit of time reviewing it. Feel free to ping me.

I do find this whole code a bit overengineered - I think it could be made a lot simpler. But lets at least decide if your change is the right one.

tomflannaghan · 2018-10-28T11:17:23Z

@jklymak Thanks for getting back to this. I've rebased.

I agree that simplifying things would be great. Since submitting this PR I've also had a look at issue #12025 and I have got a fix that also simplifies the overall implementation somewhat (mainly as it removes the need for the grid coordinates and grid transforms).

Do you think it would be better for me to submit a new PR including both my fix for #12025 and this change, or shall we keep the two separate and continue with this PR?

jklymak · 2019-02-26T00:25:53Z

@tomflannaghan sorry this fell off the radar again!

This has way too many commits 😉 so needs a proper rebase with just the changes you made. I expect this change is a good first step and then we can work on more general improvements.

Again, please do ping about these things (within reason). PRs get burried.

This makes figures neater. Also fix places that were using dmap.grid.nx for the width of the grid, which should be dmap.grid.nx - 1.

Moves the raising to the gradient steps simplifying the integrator code a bit. Also makes OutOfBounds inherit from IndexError as suggested by tacaswell.

tacaswell · 2019-08-28T01:15:39Z

It looked like what happened here was a rebase and then merging the original code back. I took the liberty of dropping the merge commit, rebasing onto current master, regenerating the tests, and force pushing. I hope that is ok with you @tomflannaghan !

tacaswell · 2019-08-28T03:41:47Z

Can someone with a mac build this locally and see how the images are failing?

dopplershift · 2019-08-31T06:32:59Z

Looks like truly minor differences.

Baseline:

Output:

Diff:

I can't even see the difference in the diff images. I had to open them up in numpy to see the differences:

Just 2 pixels, only the red channel for some reason.

We are seeing 1bit differences on 2 pixel for this test between linux/win vs mac.

tacaswell · 2019-09-01T21:05:19Z

We have seen similar off-by-one-bit differences between the platforms before

In [1]: 1 / 256                                                                                                                                               
Out[1]: 0.00390625

In [2]: 0.69411767 - 0.69803923                                                                                                                               
Out[2]: -0.003921559999999991

In [3]:

so something somewhere is rounding slightly differently between platforms. I pushed a commit upping the tolerance on the test, I don't think it is worth getting into the numerics of why this is happening.

tacaswell · 2019-09-03T19:54:38Z

Please squash merge this.

dopplershift · 2019-09-03T20:13:51Z

As per our call today, here's what it looks like when you do a quiver plot on top with:

import matplotlib.pyplot as plt
import numpy as np

def velocity_field():
    Y, X = np.mgrid[-3:3:100j, -3:3:100j]
    U = -1 - X**2 + Y
    V = 1 + X - Y**2
    return X, Y, U, V

X, Y, U, V = velocity_field()
plt.streamplot(X, Y, U, V, color=U, density=0.6, linewidth=2,
               cmap=plt.cm.autumn)
slices = [slice(None, None, 5), slice(None, None, 5)]
plt.quiver(X[slices], Y[slices], U[slices], V[slices], pivot='mid')
plt.colorbar()
plt.show()

Old:

New:

The arrows look tangential to the streamlines everywhere, so I'm not seeing any lingering issues in the streamplot calculation, at least as far as offsets are concerned. I'll merge once CI passes.

tacaswell added this to the v3.1 milestone Aug 27, 2018

tacaswell reviewed Aug 27, 2018

View reviewed changes

jklymak modified the milestones: v3.1.0, v3.2.0 Feb 26, 2019

tomflannaghan added 3 commits August 27, 2019 21:13

FIX: Ensure Euler step is always called when going out of bounds.

e6926e6

This makes figures neater. Also fix places that were using dmap.grid.nx for the width of the grid, which should be dmap.grid.nx - 1.

Remove unused code

f924361

Improve the OutOfBounds logic in the integration.

e00e512

Moves the raising to the gradient steps simplifying the integrator code a bit. Also makes OutOfBounds inherit from IndexError as suggested by tacaswell.

tacaswell force-pushed the streamplot_grid_bounds branch from b459fdd to e00e512 Compare August 28, 2019 01:14

TST: push up the tolerance on mac

91c6718

We are seeing 1bit differences on 2 pixel for this test between linux/win vs mac.

tacaswell approved these changes Sep 1, 2019

View reviewed changes

STY: add whitespace

4f1c54e

efiring approved these changes Sep 3, 2019

View reviewed changes

dopplershift merged commit a368170 into matplotlib:master Sep 3, 2019

QuLogic mentioned this pull request Dec 18, 2019

Prepare for Matplotlib 3.2 and 3.3 SciTools/cartopy#1425

Merged

		@@ -473,6 +473,10 @@ def integrate(x0, y0):
		return integrate


		class OutOfBounds(Exception):

Uh oh!

Ensure streamplot Euler step is always called when going out of bounds. #11947

Ensure streamplot Euler step is always called when going out of bounds. #11947

Uh oh!

Conversation

tomflannaghan commented Aug 27, 2018

PR Summary

PR Checklist

Uh oh!

jklymak commented Aug 27, 2018

Uh oh!

tacaswell Aug 27, 2018

Choose a reason for hiding this comment

Uh oh!

tomflannaghan Sep 3, 2018

Choose a reason for hiding this comment

Uh oh!

tacaswell Aug 27, 2018

Choose a reason for hiding this comment

Uh oh!

tomflannaghan Aug 27, 2018

Choose a reason for hiding this comment

Uh oh!

tomflannaghan Aug 27, 2018

Choose a reason for hiding this comment

Uh oh!

tomflannaghan Sep 3, 2018

Choose a reason for hiding this comment

Uh oh!

tacaswell commented Aug 27, 2018

Uh oh!

tomflannaghan commented Aug 27, 2018

Uh oh!

tacaswell commented Aug 27, 2018

Uh oh!

tomflannaghan commented Aug 27, 2018

Uh oh!

jklymak commented Sep 1, 2018

Uh oh!

tomflannaghan commented Sep 1, 2018

Uh oh!

jklymak commented Sep 1, 2018

Uh oh!

tomflannaghan commented Sep 3, 2018

Uh oh!

jklymak commented Oct 26, 2018

Uh oh!

tomflannaghan commented Oct 28, 2018

Uh oh!

jklymak commented Feb 26, 2019

Uh oh!

tacaswell commented Aug 28, 2019

Uh oh!

tacaswell commented Aug 28, 2019

Uh oh!

dopplershift commented Aug 31, 2019

Uh oh!

tacaswell commented Sep 1, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

tacaswell commented Sep 3, 2019

Uh oh!

dopplershift commented Sep 3, 2019

Uh oh!

Uh oh!

tacaswell commented Sep 1, 2019 •

edited

Loading