The first culprit occurs in _AxesBase.add_artist

    def add_artist(self, a):
        ....
        a.axes = self
        self.artists.append(a)
        self._set_artist_props(a)
        a.set_clip_path(self.patch) # <-- oops
        a._remove_method = lambda h: self.artists.remove(h)
        self.stale = True
        return a

or, if calling ax.text directly, at

    def text(...):
        ...
                t.set_clip_path(self.patch)
        self._add_text(t)
        return t

i.e., when adding the text object to the axes, the clip path is simply dropped, and replaced by the axes clip path. (I think we should just not set the clip path if it is already set.)

But even delaying the setting of the clip path until after the artist is added to the axes does not seem to help. I suspect there are some issues with Artist.set_clip_path, which is handling a bunch of different cases, but am not certain of that.

As commented above

diff --git i/lib/matplotlib/axes/_base.py w/lib/matplotlib/axes/_base.py
index aec405415..4206956d7 100644
--- i/lib/matplotlib/axes/_base.py
+++ w/lib/matplotlib/axes/_base.py
@@ -1898,7 +1898,8 @@ class _AxesBase(martist.Artist):
         self.artists.append(a)
         a._remove_method = self.artists.remove
         self._set_artist_props(a)
-        a.set_clip_path(self.patch)
+        if a.get_clip_path() is None:
+            a.set_clip_path(self.patch)
         self.stale = True
         return a

is a first necessary step to fix this (this is consistent with e.g. add_line(); there's a few other places (e.g. add_table(), text(), etc.) where we call set_clip_path without checking a previously existing clip path and should likewise be fixed.
This is sufficient to make things work with mplcairo, but not with agg.

This issue has been marked "inactive" because it has been 365 days since the last comment. If this issue is still present in recent Matplotlib releases, or the feature request is still wanted, please leave a comment and this label will be removed. If there are no updates in another 30 days, this issue will be automatically closed, but you are free to re-open or create a new issue if needed. We value issue reports, and this procedure is meant to help us resurface and prioritize issues that have not been addressed yet, not make them disappear. Thanks for your help!

#8270 (comment) has been integrated, but as noted this is not enough for agg.