Fixed several accuracy bugs with image resampling by ayshih · Pull Request #30184 · matplotlib/matplotlib

ayshih · 2025-06-18T04:58:01Z

PR summary

This PR fixes several accuracy bugs with image resampling:

The filtering weight at one extreme is not being calculated correctly. In the case of linear interpolation, this makes the one weight that should be zero be instead nonzero. This then results in a pixels of constant values manifesting a small blip after linear interpolation if the center of an input pixel exactly coincides with the center of a output pixel (within the agg backend, its pixels may not be exactly aligned with the true pixels).
The filtering weights are not aligned correctly in their array as expected by subsequent code. For a weight array of N values, the "pivot" needs to be at N / 2 - 1, not at N / 2. This results in a bias where the linearly interpolated values are calculated slightly to the right of where they should be calculated.
Specific to nonaffine transforms, the values in the lookup table used to store the calculation are being truncated when they should be rounded. This results in a bias (smaller than the above) to the right.
[subsequently added, with updated images therein] Specific to affine transforms, the end point for interpolation is not specified correctly, which can slightly degrade the accuracy of the sampling locations.

Below are illustrative plots and the generating script for linear resampling of a 2-pixel array. There are fundamental limitations of the accuracy due to the subpixel approach of the agg backend, but the motivation here is for the average behavior to be accurate (i.e., eliminate bias).

Before this PR

After this PR

Script

import matplotlib.pyplot as plt
import numpy as np
from matplotlib.transforms import Affine2D, Transform
from matplotlib.image import resample, BILINEAR

in_data = np.array([[0.1, 0.9]])
in_shape = in_data.shape
in_edges = np.arange(in_shape[1] + 1)

out_shape = (1, 20)
out_edges = np.arange(out_shape[1] + 1)

ideal_data = np.array([[0.1, 0.1, 0.1, 0.1, 0.1, 0.14, 0.22, 0.3, 0.38, 0.46,
                        0.54, 0.62, 0.7, 0.78, 0.86, 0.9, 0.9, 0.9, 0.9, 0.9]])

# Create a simple affine transform for scaling the input array
affine = Affine2D().scale(sx=out_shape[1] / in_shape[1], sy=1)

# Create a nonaffine version of the same transform by compositing with a nonaffine identity transform
class NonAffineIdentityTransform(Transform):
    input_dims = 2
    output_dims = 2

    def inverted(self):
        return self
nonaffine = NonAffineIdentityTransform() + affine

affine_data = np.empty(out_shape)
nonaffine_data = np.empty(out_shape)
resample(in_data, affine_data, affine, interpolation=BILINEAR)
resample(in_data, nonaffine_data, nonaffine, interpolation=BILINEAR)

fig, axs = plt.subplots(3, 1, figsize=(4.8, 6.4), layout="constrained")

axs[0].stairs(in_data[0, :], in_edges)
axs[0].grid(ls='dotted')
axs[0].set_xticks(in_edges)
axs[0].set_title('Original data')

axs[1].stairs(affine_data[0, :], out_edges, label='affine')
axs[1].stairs(nonaffine_data[0, :], out_edges, ls='dashed', label='nonaffine')
axs[1].grid(ls='dotted')
axs[1].set_xticks(out_edges)
axs[1].legend()
axs[1].set_title('Interpolated data')

axs[2].stairs(affine_data[0, :] - ideal_data[0, :], out_edges, label='affine')
axs[2].stairs(nonaffine_data[0, :] - ideal_data[0, :], out_edges, ls='dashed', label='nonaffine')
axs[2].grid(ls='dotted')
axs[2].set_xticks(out_edges)
axs[2].set_ylim(-0.006, 0.006)
axs[2].legend()
axs[2].set_title('Interpolated data minus ideal result')

plt.show()

PR checklist

[N/A] "closes #0000" is in the body of the PR description to link the related issue
new and changed code is tested
[N/A] Plotting related features are demonstrated in an example
[N/A] New Features and API Changes are noted with a directive and release note
[N/A] Documentation complies with general and docstring guidelines

ayshih · 2025-06-18T05:00:16Z

extern/agg24-svn/include/agg_image_filters.h

+                if(i <= pivot) m_weight_array[pivot - i] = value;
            }
-            unsigned end = (diameter() << image_subpixel_shift) - 1;
-            m_weight_array[0] = m_weight_array[end];


This line is one of the bugs. 0 and end are different distances from pivot, so the corresponding weights typically should not be the same.

Please put the new version under a macro guard, as was done in #28122, so that we keep a track of what's the original Agg and what we changed on top of it.

Makes sense, done

It turns out that I can't add a preprocessor definition in _image_resample.h a la #28122. There is buggy code in agg_image_filters.cpp that needs to be skipped, but since it is a CPP file rather than a H file, it gets compiled separately with no awareness of _image_resample.h. I have put the preprocessor definition (MPL_FIX_IMAGE_FILTER_LUT_BUGS) here at the top of agg_image_filters.h, which is lower level than would be nice, but at least still makes it clear what is custom code.

You could also put the define in the corresponding meson.build if you prefer.

Ah, that's a nicer approach. Done!

ayshih · 2025-06-18T12:53:16Z

The filtering weight at one extreme is not being calculated correctly. In the case of linear interpolation, this makes the one weight that should be zero be instead nonzero. This then results in a pixels of constant values manifesting a small blip after linear interpolation if the center of an input pixel exactly coincides with the center of a output pixel (within the agg backend, its pixels may not be exactly aligned with the true pixels).

You can see such a blip in the example above, but it's easy to overlook. Here's a starker example: an array of two pixels with the same value (0.1) is resampled to an array of ten pixels. Before this PR, two of the pixels in the output array have values slightly greater than 0.1.

>>> import numpy as np
>>> from matplotlib.transforms import Affine2D
>>> from matplotlib.image import resample, BILINEAR
>>>
>>> in_data = np.array([[0.1, 0.1]])
>>> in_shape = in_data.shape
>>>
>>> out_shape = (1, 10)
>>>
>>> # Create a simple affine transform for scaling the input array
>>> affine = Affine2D().scale(sx=out_shape[1] / in_shape[1], sy=1)
>>>
>>> affine_data = np.empty(out_shape)
>>> resample(in_data, affine_data, affine, interpolation=BILINEAR)
>>>
>>> print(affine_data)
[[0.1        0.10001221 0.1        0.1        0.1        0.1
  0.10001221 0.1        0.1        0.1       ]]

After this PR, the output is the expected:

>>> print(affine_data)
[[0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1]]

jklymak · 2025-06-18T15:29:27Z

This killed the log-scale images?

Also this changed a bunch of image tests, but I can't see any difference by eye. Is it worth just relaxing the tolerance on those tests a bit and at the same time adding new tests for these fixes?

ayshih · 2025-06-18T15:36:10Z

This killed the log-scale images?

Indeed, I need to track down why.

Also this changed a bunch of image tests, but I can't see any difference by eye. Is it worth just relaxing the tolerance on those tests a bit and at the same time adding new tests for these fixes?

Yes, the other 17 image tests are mostly subtle differences, but merely relaxing the tolerance doesn't make sense to me. The updated baseline images do represent what the output should be.

jklymak · 2025-06-18T15:39:02Z

Fair enough - however, it would be good if there were some tests devised where it showed a visual effect of these inaccuracies.

ayshih · 2025-06-18T20:44:32Z

extern/agg24-svn/src/agg_image_filters.cpp

            }
        }

+#ifndef MPL_FIX_IMAGE_FILTER_LUT_BUGS


The following code block is removed because it not only has the same bugs as the code in agg_image_filters.h, but also it shouldn't even exist because it modifies the weight array after it has been normalized, thus potentially destroying the normalization.

ayshih · 2025-06-19T21:00:33Z

I fixed the issue with log_scale_image, so this PR is now ready for review. The final count is 65 baseline images that need to be updated. Maybe a third of those images have (subtle) changes in the rendering of content, e.g.:

Before this PR:

After this PR:

Difference:

On the other hand, most of the updated images have solely very slight changes in the appearance of text. As @jklymak suggested, it might make more sense in those cases to simply increase the tolerance instead for the comparison instead of updating the baseline image, so let me know if I should do that instead.

QuLogic · 2025-06-26T00:04:04Z

Are you sure you have the right set of images? lib/matplotlib/tests/baseline_images/test_axes/formatter_ticker_004.png for example doesn't have any images to be resampled.

QuLogic · 2025-06-26T00:54:06Z

OK, I see that RendererAgg::draw_text_image uses image_filter_spline36 when rotating text, so it only affects tests with rotated text. So for any tests that are purely rotated text, we should just increase the tolerance and make a note of them, then revert the tolerance changes in #30161 when regenerating the images. Or re-target this PR to the text-overhaul branch without test image changes (but then they'd all be regenerated much later.)

anntzer · 2025-12-09T09:20:24Z

I likely won't be able to review properly the changes since Oct. 8 before the holiday break. Sorry about that, this looks like great work.

greglucas

This looks good to me.

It was decided on the call to merge this into main and then update the text overhaul branch to remove the added tolerances here.

We should merge #30824 first because they both update the bivar cmap image.

ayshih · 2025-12-18T20:53:26Z

Thanks for reviewing this!

We should merge #30824 first because they both update the bivar cmap image.

I agree that #30824 should be merged first, and then 71c7f4e in this PR will be removed as no longer applicable (because SegmentedBivarColormap no longer calls the Agg resampler)

…rrect

ayshih · 2026-01-08T16:54:50Z

I've updated this PR now that #30824 has been merged

ayshih · 2026-01-15T20:39:46Z

Whew, thanks!

github-actions bot added the topic: images label Jun 18, 2025

ayshih commented Jun 18, 2025

View reviewed changes

ayshih force-pushed the more_resample_bugs branch from d2fa1e5 to 7eb18b2 Compare June 18, 2025 13:54

github-actions bot added the topic: mpl_toolkit label Jun 18, 2025

ayshih force-pushed the more_resample_bugs branch from 7eb18b2 to d9f8bb5 Compare June 18, 2025 14:35

github-actions bot removed the topic: mpl_toolkit label Jun 18, 2025

ayshih force-pushed the more_resample_bugs branch from 7eb1cea to 5cbb7bb Compare June 18, 2025 15:31

ayshih force-pushed the more_resample_bugs branch 3 times, most recently from e198af6 to 717a221 Compare June 18, 2025 20:39

ayshih commented Jun 18, 2025

View reviewed changes

github-actions bot added topic: mpl_toolkit topic: mplot3d and removed topic: mpl_toolkit labels Jun 19, 2025

ayshih force-pushed the more_resample_bugs branch 7 times, most recently from 43b912c to 90bd95f Compare June 19, 2025 19:20

ayshih marked this pull request as ready for review June 19, 2025 20:47

ayshih force-pushed the more_resample_bugs branch from 90bd95f to 9fdca21 Compare July 10, 2025 13:51

ayshih force-pushed the more_resample_bugs branch from 35b0fc8 to 48e28b8 Compare December 8, 2025 13:45

ayshih mentioned this pull request Dec 8, 2025

Fixed bilinear interpolation for SegmentedBivarColormap #30824

Merged

1 task

greglucas approved these changes Dec 18, 2025

View reviewed changes

timhoffm mentioned this pull request Dec 19, 2025

Colormaps are not matching with reference pngs #20459

Open

github-actions bot added the status: needs rebase label Jan 8, 2026

ayshih added 10 commits January 8, 2026 10:46

Fix bugs in the calculation of the weight array

71bb56f

Fix bug with using the distortion table for nonaffine transformations

ca634d5

Move macro-guard definition to meson.build

e2f1a5f

Fix bug where the interpolation endpoint was one step further than co…

632e4e3

…rrect

Adjust test to use the downsampled image as reference

072f1a4

Loosen tolerances for text-only image differences

7b59187

Refactor bug fixes in the calculation of the weight array

131648a

Support differing subpixel alignment for nonaffine distortion table

acbedc6

Avoid affine code path when scaling will make subpixels visible

86100aa

Update baseline images

c19b295

ayshih force-pushed the more_resample_bugs branch from 48e28b8 to c19b295 Compare January 8, 2026 15:48

github-actions bot removed topic: color/color & colormaps status: needs rebase labels Jan 8, 2026

QuLogic added this to the v3.11.0 milestone Jan 15, 2026

QuLogic merged commit cf2b13c into matplotlib:main Jan 15, 2026
38 of 41 checks passed

This was referenced Jan 15, 2026

Misalignment imshow vs. grid lines #1441

Closed

imshow pixel boundaries wrong when zoomed in #14143

Closed

ayshih mentioned this pull request Jan 21, 2026

TST: image comparison checks fail against matplotlib-dev astropy/astropy#19201

Closed

pllim mentioned this pull request Jan 21, 2026

TST: mpldev CircleCI job has 35 failures (Image files did not match) astropy/astropy#19170

Closed

ayshih mentioned this pull request Jan 23, 2026

Fixed inaccurate image placement and even more resampling bugs #31021

Merged

3 tasks

ayshih deleted the more_resample_bugs branch March 2, 2026 14:02

Uh oh!

Conversation

ayshih commented Jun 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

PR summary

Before this PR

After this PR

Script

PR checklist

Uh oh!

ayshih Jun 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

anntzer Jun 18, 2025

Choose a reason for hiding this comment

Uh oh!

ayshih Jun 18, 2025

Choose a reason for hiding this comment

Uh oh!

ayshih Jun 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

QuLogic Sep 10, 2025

Choose a reason for hiding this comment

Uh oh!

ayshih Sep 11, 2025

Choose a reason for hiding this comment

Uh oh!

ayshih commented Jun 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jklymak commented Jun 18, 2025

Uh oh!

ayshih commented Jun 18, 2025

Uh oh!

jklymak commented Jun 18, 2025

Uh oh!

ayshih Jun 18, 2025

Choose a reason for hiding this comment

Uh oh!

ayshih commented Jun 19, 2025

Uh oh!

QuLogic commented Jun 26, 2025

Uh oh!

QuLogic commented Jun 26, 2025

Uh oh!

anntzer commented Dec 9, 2025

Uh oh!

greglucas left a comment

Choose a reason for hiding this comment

Uh oh!

ayshih commented Dec 18, 2025

Uh oh!

ayshih commented Jan 8, 2026

Uh oh!

Uh oh!

ayshih commented Jan 15, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

ayshih commented Jun 18, 2025 •

edited

Loading

ayshih Jun 18, 2025 •

edited

Loading

ayshih Jun 18, 2025 •

edited

Loading

ayshih commented Jun 18, 2025 •

edited

Loading