gd_interpolation.c uses a gdFixed type for some calculations. Currently this is typedef'd as long which is either 32bit or 64 bit on the supported platforms. That leads to the following issue on x64 Windows, for example (written in PHP for simplicity):

$im = imagecreate(1 << 23, 1);
imagefilledrectangle($im, 0, 0, 1 << 23, 1, imagecolorallocate($im, 255, 0, 0));
$im2 = imagescale($im, 1, 1, IMG_NEAREST_NEIGHBOUR);

ext\gd\libgd\gd_interpolation.c:1125:23: runtime error: 2.14748e+09 is outside the range of representable values of type 'long'
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior ext\gd\libgd\gd_interpolation.c:1125:23

That line corresponds to

So we attempt to cast (1 << 23) * 256 to long, what doesn't fit in a signed 32bit integer.

It gets worse if we look at gd_mulfx():

So gd_mulfx(gd_itofx(256), gd_itofx(256)) already causes a signed integer overflow (UB) if long is 32bit.

The obvious fix would be to typedef int64_t gdFixed (and change the casts in the macros accordingly) at least for LLP64, but I seriously doubt the usefulness of gdFixed in general. I mean, having only 8 binary decimals is very limited (note that we sometimes calculate trigonometric functions and store the results as gdFixed, and I doubt that the gdFixed arithmetic is much faster than double or float arithmetic on most contempary machines (I'm ignoring embedded devices, obviously); it's not unlikely to be slower.

In my opinion, we should either drop gdFixed altogether, or at least alternatively implement it as float/double (maybe manually configurable, or possibly auto-configured depending on the machine).