If 'x' is pressed during zoom and the user zoom, if the x spread is too short (ie abs(x - lastx) < 5) the zoom is not done.

Problem lies in function release_zoom at https://github.com/matplotlib/matplotlib/blob/master/lib/matplotlib/backend_bases.py#L3045

if abs(x - lastx) < 5 or abs(y - lasty) < 5:

where the test could be replaced by

if (abs(x - lastx) < 5 and (self._zoom_mode != "y")) or 
   (abs(y - lasty) < 5 and (self._zoom_mode != "x")):

(but maybe the "x" and "y" should be swapped ...)