Doc: tell 3.11 upgraders to remove the pre-shaping workaround - #32263

Thank you for opening your first PR into Matplotlib!

If you have not heard from us in a week or so, please leave a new comment below and that should bring it to our attention. Most of our reviewers are volunteers and sometimes things fall through the cracks. We also ask that you please finish addressing any review comments on this PR and wait for it to be merged (or closed) before opening a new one, as it can be a valuable learning experience to go through the review process.

You can also join us on discourse chat for real-time discussion.

For details on testing, writing docs, and our review process, please see the developer guide.
Please let us know if (and how) you use AI, it will help us give you better feedback on your PR.

We strive to be a welcoming and open project. Please follow our Code of Conduct.

Answering the AI question in the template, since it was asked directly.

The finding is mine. I hit it building something in Arabic that looked wrong to me, then measured it against a Pillow/Raqm render of the same string in the same font, and wrote the reproducer in #32262, which runs on my own machine.

I used an AI assistant to help draft and tighten the wording of this section, and to check that the module docstring still parsed as RST before I pushed. The technical claims I checked myself, including reading extern/meson.build to confirm libraqm is a required dependency rather than assuming it, since that is what makes this unconditional here and not in Pillow.

Happy to rewrite any of the prose if it does not sound like the rest of the page.

To verify there is no generic way to determine if an Arabic, Hebrew, or Devanagari string is pre-ordered from logical to display order? How big of a problem/common is it that applications have pre-reversed strings kicking around?

Per https://www.w3.org/International/questions/qa-bidi-unicode-controls.en.html#basedirection if the user wraps pre-reversed text in the LRO/PDF unicode points the text will render correctly in all versions of mpl (because <3.11 we ignore the bidirectional details all together and >= 3.11 we (via libraqm) respect them correctly).

import matplotlib
import matplotlib.pyplot as plt

# editors / GH will correctly render RTL so use \N to be clear
# hello = "שלום"
hello = "\N{HEBREW LETTER SHIN}\N{HEBREW LETTER LAMED}\N{HEBREW LETTER VAV}\N{HEBREW LETTER FINAL MEM}"
hello_r = hello[::-1]
hello_fixed = "\N{LRO}" + hello_r + "\N{PDF}"

fig, ax = plt.subplots()
ax.set_title(f'{matplotlib.__version__=}')

ax.text(0, .7, "mpl direct")
ax.text(.5, .7, hello, fontsize=30)

ax.text(0, .5, "pre-flipped")
ax.text(.5, .5, hello_r, fontsize=30)

ax.text(0, .25, "pre-flipped + LRO/PDF")
ax.text(.5, .25, hello_fixed, fontsize=30)

plt.show()
fig.savefig(f'/tmp/bidi_mpl{matplotlib.__version__.replace('.', '-')}.png')

I think this works correctly with Arabic as well but I do not know enough Arabic to be 100% confident it is correct.

from bidi import get_display
import arabic_reshaper
import matplotlib
import matplotlib.pyplot as plt

# The Emirates 
word = "الإمارات"


word_pre =get_display(arabic_reshaper.reshape(word))
word_pre_fixed = "\N{LRO}" + word_pre + "\N{PDF}"

fig, ax = plt.subplots()
ax.set_title(f'{matplotlib.__version__=}')

ax.text(0, .9, "mpl direct")
ax.text(.5, .9, word, fontsize=30)
ax.text(0, .7, "pre-flipped")
ax.text(.5, .7, word[::-1], fontsize=30)
ax.text(0, .5, "pre-shaped")
ax.text(.5, .5, word_pre, fontsize=30)
ax.text(0, .25, "pre-shaped + LRO/PDF")
ax.text(.5, .25, word_pre_fixed, fontsize=30)

plt.show()
fig.savefig(f'/tmp/bidi_mpl{matplotlib.__version__.replace('.', '-')}.png')

I think the path forward here is to document:

• if you can pin to > 3.11, then in just works
• if you can version-gate behavior then put in two branches so you can drop pre-processing the text before passing to mpl when you can pin your minimum to 3.11
• if you have to support both sides and do not control the direct mpl calls (e.g. you are passing text into a thrid-party library that is calling mpl on your behalf) then pre-process and wrape the text in LRO/PDF.

Thanks, all three of these are more useful than what I filed.

On LRO/PDF with Arabic. You said you were not confident, so I checked it. It works, with one caveat I think is worth writing down.

I rendered LRO + get_display(reshape(w)) + PDF against the plain logical string on 3.11 and compared the rasters directly. Mean absolute pixel difference, both crops normalised to 72px height, so 0.0 means the two images are identical:

The reading is correct in all nine, including with a lam-alef ligature and with Latin digits mixed in, so your recommendation holds. But it is not always the same rendering. The wrapped string draws the font's static Presentation Forms glyphs instead of the font's own shaping, so wherever those are not metrically identical to the shaped output you get looser joins and a wider line. With Geeza Pro that is visible at a glance, not just in the numbers.

So for the third branch I would document it as "renders correctly on every version" rather than "renders identically". For someone who does not control the matplotlib calls that still seems like clearly the right trade.

Is there a generic way to tell a string is pre-ordered? Partly, and the split is the interesting part.

• The reshaping half leaves a fingerprint. arabic_reshaper output lands in Arabic Presentation Forms-B (U+FE70 to U+FEFF), which essentially never appears in text a person typed. Over 1,856 generated samples, arabic_reshaper 3.0.0 emits 122 distinct codepoints from that block and never U+FEFF, which matters because U+FEFF sits inside the block and is the byte order mark, so a naive range check flags every file that has one.
• The reordering half does not. Your hello[::-1] case has the same codepoints in a different order. Nothing distinguishes it from correctly stored text without knowing the language, so a general detector is not possible there.
• Presentation Forms-A looks like a second signal and is not usable as one. The reshaper does emit from it (U+FDF2, and U+FBE8/U+FBE9 for the Uighur alef), and ﷺ ﷻ ﷽ are used deliberately in ordinary Arabic, so flagging that block flags correct text.

I wrapped the Forms-B check in a small linter and ran it across 3,883 mixed files here: 31 flagged spans, all true positives.

How common is it? A floor, counting only public code that GitHub indexes: 1,172 files match get_display(arabic_reshaper.reshape, and 4,048 match arabic_reshaper.reshape. arabic-reshaper had 5.31M downloads in the last month. Not all of that reaches matplotlib, but the recipe is the accepted answer almost everywhere the question is asked, which is what makes 3.11 a quiet change rather than a loud one.

@rcomer thank you for editing those StackOverflow answers. That reaches more people than this PR will, and I should have thought to do it myself.

@story645 that makes sense, and it is a better home than where I put it. I will rewrite this as an admonition on the 3.11 change note covering @tacaswell's three branches, and drop the section I added to the fonts page unless you would still like something on RTL there.