Fixed several accuracy bugs with image resampling #30184

ayshih · 2025-06-18T04:58:01Z

PR summary

This PR fixes several accuracy bugs with image resampling:

The filtering weight at one extreme is not being calculated correctly. In the case of linear interpolation, this makes the one weight that should be zero be instead nonzero. This then results in a pixels of constant values manifesting a small blip after linear interpolation if the center of an input pixel exactly coincides with the center of a output pixel (within the agg backend, its pixels may not be exactly aligned with the true pixels).
The filtering weights are not aligned correctly in their array as expected by subsequent code. For a weight array of N values, the "pivot" needs to be at N / 2 - 1, not at N / 2. This results in a bias where the linearly interpolated values are calculated slightly to the right of where they should be calculated.
Specific to nonaffine transforms, the values in the lookup table used to store the calculation are being truncated when they should be rounded. This results in a bias (smaller than the above) to the right.

Below are illustrative plots and the generating script for linear resampling of a 2-pixel array. There are fundamental limitations of the accuracy due to the subpixel approach of the agg backend, but the motivation here is for the average behavior to be accurate (i.e., eliminate bias).

Before this PR

After this PR

Script

import matplotlib.pyplot as plt
import numpy as np
from matplotlib.transforms import Affine2D, Transform
from matplotlib.image import resample, BILINEAR

in_data = np.array([[0.1, 0.9]])
in_shape = in_data.shape
in_edges = np.arange(in_shape[1] + 1)

out_shape = (1, 20)
out_edges = np.arange(out_shape[1] + 1)

ideal_data = np.array([[0.1, 0.1, 0.1, 0.1, 0.1, 0.14, 0.22, 0.3, 0.38, 0.46,
                        0.54, 0.62, 0.7, 0.78, 0.86, 0.9, 0.9, 0.9, 0.9, 0.9]])

# Create a simple affine transform for scaling the input array
affine = Affine2D().scale(sx=out_shape[1] / in_shape[1], sy=1)

# Create a nonaffine version of the same transform by compositing with a nonaffine identity transform
class NonAffineIdentityTransform(Transform):
    input_dims = 2
    output_dims = 2

    def inverted(self):
        return self
nonaffine = NonAffineIdentityTransform() + affine

affine_data = np.empty(out_shape)
nonaffine_data = np.empty(out_shape)
resample(in_data, affine_data, affine, interpolation=BILINEAR)
resample(in_data, nonaffine_data, nonaffine, interpolation=BILINEAR)

fig, axs = plt.subplots(3, 1, figsize=(4.8, 6.4), layout="constrained")

axs[0].stairs(in_data[0, :], in_edges)
axs[0].grid(ls='dotted')
axs[0].set_xticks(in_edges)
axs[0].set_title('Original data')

axs[1].stairs(affine_data[0, :], out_edges, label='affine')
axs[1].stairs(nonaffine_data[0, :], out_edges, ls='dashed', label='nonaffine')
axs[1].grid(ls='dotted')
axs[1].set_xticks(out_edges)
axs[1].legend()
axs[1].set_title('Interpolated data')

axs[2].stairs(affine_data[0, :] - ideal_data[0, :], out_edges, label='affine')
axs[2].stairs(nonaffine_data[0, :] - ideal_data[0, :], out_edges, ls='dashed', label='nonaffine')
axs[2].grid(ls='dotted')
axs[2].set_xticks(out_edges)
axs[2].set_ylim(-0.006, 0.006)
axs[2].legend()
axs[2].set_title('Interpolated data minus ideal result')

plt.show()

PR checklist

[N/A] "closes #0000" is in the body of the PR description to link the related issue
new and changed code is tested
[N/A] Plotting related features are demonstrated in an example
[N/A] New Features and API Changes are noted with a directive and release note
[N/A] Documentation complies with general and docstring guidelines

ayshih · 2025-06-18T05:00:16Z

extern/agg24-svn/include/agg_image_filters.h

            }
-            unsigned end = (diameter() << image_subpixel_shift) - 1;
-            m_weight_array[0] = m_weight_array[end];


This line is one of the bugs. 0 and end are different distances from pivot, so the corresponding weights typically should not be the same.

Please put the new version under a macro guard, as was done in #28122, so that we keep a track of what's the original Agg and what we changed on top of it.

Makes sense, done

It turns out that I can't add a preprocessor definition in _image_resample.h a la #28122. There is buggy code in agg_image_filters.cpp that needs to be skipped, but since it is a CPP file rather than a H file, it gets compiled separately with no awareness of _image_resample.h. I have put the preprocessor definition (MPL_FIX_IMAGE_FILTER_LUT_BUGS) here at the top of agg_image_filters.h, which is lower level than would be nice, but at least still makes it clear what is custom code.

ayshih · 2025-06-18T12:53:16Z

The filtering weight at one extreme is not being calculated correctly. In the case of linear interpolation, this makes the one weight that should be zero be instead nonzero. This then results in a pixels of constant values manifesting a small blip after linear interpolation if the center of an input pixel exactly coincides with the center of a output pixel (within the agg backend, its pixels may not be exactly aligned with the true pixels).

You can see such a blip in the example above, but it's easy to overlook. Here's a starker example: an array of two pixels with the same value (0.1) is resampled to an array of ten pixels. Before this PR, two of the pixels in the output array have values slightly greater than 0.1.

>>> import numpy as np
>>> from matplotlib.transforms import Affine2D
>>> from matplotlib.image import resample, BILINEAR
>>>
>>> in_data = np.array([[0.1, 0.1]])
>>> in_shape = in_data.shape
>>>
>>> out_shape = (1, 10)
>>>
>>> # Create a simple affine transform for scaling the input array
>>> affine = Affine2D().scale(sx=out_shape[1] / in_shape[1], sy=1)
>>>
>>> affine_data = np.empty(out_shape)
>>> resample(in_data, affine_data, affine, interpolation=BILINEAR)
>>>
>>> print(affine_data)
[[0.1        0.10001221 0.1        0.1        0.1        0.1
  0.10001221 0.1        0.1        0.1       ]]

After this PR, the output is the expected:

>>> print(affine_data)
[[0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1]]

jklymak · 2025-06-18T15:29:27Z

This killed the log-scale images?

Also this changed a bunch of image tests, but I can't see any difference by eye. Is it worth just relaxing the tolerance on those tests a bit and at the same time adding new tests for these fixes?

ayshih · 2025-06-18T15:36:10Z

This killed the log-scale images?

Indeed, I need to track down why.

Also this changed a bunch of image tests, but I can't see any difference by eye. Is it worth just relaxing the tolerance on those tests a bit and at the same time adding new tests for these fixes?

Yes, the other 17 image tests are mostly subtle differences, but merely relaxing the tolerance doesn't make sense to me. The updated baseline images do represent what the output should be.

jklymak · 2025-06-18T15:39:02Z

Fair enough - however, it would be good if there were some tests devised where it showed a visual effect of these inaccuracies.

ayshih · 2025-06-18T20:44:32Z

extern/agg24-svn/src/agg_image_filters.cpp

@@ -88,6 +88,7 @@ namespace agg
            }
        }

+#ifndef MPL_FIX_IMAGE_FILTER_LUT_BUGS


The following code block is removed because it not only has the same bugs as the code in agg_image_filters.h, but also it shouldn't even exist because it modifies the weight array after it has been normalized, thus potentially destroying the normalization.

ayshih · 2025-06-19T21:00:33Z

I fixed the issue with log_scale_image, so this PR is now ready for review. The final count is 65 baseline images that need to be updated. Maybe a third of those images have (subtle) changes in the rendering of content, e.g.:

Before this PR:

After this PR:

Difference:

On the other hand, most of the updated images have solely very slight changes in the appearance of text. As @jklymak suggested, it might make more sense in those cases to simply increase the tolerance instead for the comparison instead of updating the baseline image, so let me know if I should do that instead.

github-actions bot added the topic: images label Jun 18, 2025

ayshih commented Jun 18, 2025

View reviewed changes

ayshih force-pushed the more_resample_bugs branch from d2fa1e5 to 7eb18b2 Compare June 18, 2025 13:54

github-actions bot added the topic: mpl_toolkit label Jun 18, 2025

ayshih force-pushed the more_resample_bugs branch from 7eb18b2 to d9f8bb5 Compare June 18, 2025 14:35

github-actions bot removed the topic: mpl_toolkit label Jun 18, 2025

ayshih force-pushed the more_resample_bugs branch from 7eb1cea to 5cbb7bb Compare June 18, 2025 15:31

ayshih force-pushed the more_resample_bugs branch from 5cbb7bb to c967025 Compare June 18, 2025 17:38

ayshih added 3 commits June 18, 2025 15:59

Fix bugs in the calculation of the weight array

23656e8

Fix bug with using the distortion table for nonaffine transformations

a0902fe

Update values in a test

717a221

ayshih force-pushed the more_resample_bugs branch 2 times, most recently from e198af6 to 717a221 Compare June 18, 2025 20:39

ayshih commented Jun 18, 2025

View reviewed changes

Update baseline images

1de3e2e

github-actions bot added topic: mpl_toolkit topic: mplot3d and removed topic: mpl_toolkit labels Jun 19, 2025

ayshih force-pushed the more_resample_bugs branch 6 times, most recently from b7bcb7a to ef8d5b2 Compare June 19, 2025 18:24

Update more baseline images

90bd95f

ayshih force-pushed the more_resample_bugs branch from 43b912c to 90bd95f Compare June 19, 2025 19:20

ayshih marked this pull request as ready for review June 19, 2025 20:47

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

Fixed several accuracy bugs with image resampling #30184

Fixed several accuracy bugs with image resampling #30184

ayshih commented Jun 18, 2025 •

edited

Loading

Uh oh!

ayshih Jun 18, 2025 •

edited

Loading

Uh oh!

anntzer Jun 18, 2025

Uh oh!

ayshih Jun 18, 2025

Uh oh!

ayshih Jun 18, 2025 •

edited

Loading

Uh oh!

ayshih commented Jun 18, 2025 •

edited

Loading

Uh oh!

jklymak commented Jun 18, 2025

Uh oh!

ayshih commented Jun 18, 2025

Uh oh!

jklymak commented Jun 18, 2025

Uh oh!

ayshih Jun 18, 2025

Uh oh!

ayshih commented Jun 19, 2025

Uh oh!

Uh oh!

Uh oh!

Fixed several accuracy bugs with image resampling #30184

Are you sure you want to change the base?

Fixed several accuracy bugs with image resampling #30184

Conversation

ayshih commented Jun 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

PR summary

Before this PR

After this PR

Script

PR checklist

Uh oh!

ayshih Jun 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

anntzer Jun 18, 2025

Choose a reason for hiding this comment

Uh oh!

ayshih Jun 18, 2025

Choose a reason for hiding this comment

Uh oh!

ayshih Jun 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ayshih commented Jun 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jklymak commented Jun 18, 2025

Uh oh!

ayshih commented Jun 18, 2025

Uh oh!

jklymak commented Jun 18, 2025

Uh oh!

ayshih Jun 18, 2025

Choose a reason for hiding this comment

Uh oh!

ayshih commented Jun 19, 2025

Uh oh!

Uh oh!

ayshih commented Jun 18, 2025 •

edited

Loading

ayshih Jun 18, 2025 •

edited

Loading

ayshih Jun 18, 2025 •

edited

Loading

ayshih commented Jun 18, 2025 •

edited

Loading