I have some (no-guarantee) plans to fix that, but that'll likely wait after the deprecation in #18378 elapses (so 3.6 timeframe) as that'll likely require large amounts of surgery and maintaining backcompat on all the internals would be a lot of work...

But that is how TeX works, right? \bf and \it cannot be combined in regular TeX. https://tex.stackexchange.com/questions/41681/correct-way-to-bold-italicize-text

The textbf / textit commands on the other hand can be nested. The point is to allow bold-italic which currently isn't possible in mathtext.

Actually, looking at this again, we don't supply textbf/textit at all, but mathbf/mathit, which cannot be nested. The correct solution seems to be to provide \boldmath and/or \bm. See https://tex.stackexchange.com/questions/3238/bm-package-versus-boldsymbol for a discussion of the differences, but I'd say that these are minor enough that we can just ignore them and pretend the commands are the same.

Just doing some preliminary checkings for this... First, let's see what currently works. Because of the way things are currently implemented (font specs get stored as State.font and rm/bf/it get additionally stored as State.font_class), the only combos that can possibly work are one of rm/bf/it together with one of {none,tt,cal,sf,default,bb,frak,scr}. Let's look at all of them (in uppercase, as e.g. cal doesn't cover lowercase at all):

We see that the only multicombos that currently work are it+bb, bf+bb, bf+frak, and, for stix only, bf+sf, it+sf. (A quick check with usetex=True and \usepackage{amssymb} (for mathfrak) indicates that none of these stack properly in TeX, which always follows the inner (last) font macro).

Looking at https://ctan.gutenberg.eu.org/fonts/stix/doc/stix.pdf, reasonable additional combos we may want to support are bfit, bfscr, bfsfit, bfbbit, and expand the support for bfsf, sfit for other fonts. https://ctan.gutenberg.eu.org/macros/unicodetex/latex/unicode-math/unicode-math.pdf further mentions bfcal, bfscr, bffrak, although I don't known if we actually have any fonts with such glyphs.

I think the API here that maintains reasonable levels of compatibility with TeX is to actually stop making \mathit{\mathbb{...}} and friends stack properly (reverting to TeX's standard behavior -- fortunately, 1) I guess these usages are rare and 2) this is something on which we can reasonably emit deprecation warnings) and introduce the \mathbbit, \mathbfbb, \mathbffrak, \mathbfit, etc. (using the same names as stix and unicode-math) macros that select the corresponding fonts. (Then we may get away without \bm/\boldmath, but these should be easy enough to add on top of that as well.)

The implementation, for reference:

from matplotlib import pyplot as plt

plt.rcParams["text.latex.preamble"] = r"\usepackage{amssymb}"
fig = plt.figure(figsize=(15, 8))
sfs = fig.subfigures(2, 3)
fontsets = ["dejavusans", "dejavuserif", "cm", "stixsans", "stix"]
styles = ["rm", "it", "bf"]
families = ["", "tt", "cal", "sf", "default", "bb", "frak", "scr"]

for sf, fs in zip(sfs.ravel(), fontsets):
    sf.suptitle(fs, size=16)
    for x, st in enumerate(styles):
        for y, fm in enumerate(families[::-1]):
            # Upper-case text as some fonts, e.g. cal, don't cover lowercase.
            s = ("$\math%s{\math%s{%s+%s}}$" % (st, fm, st.upper(), fm.upper())
                 if fm else
                 "$\math%s{%s}$" % (st, st.upper()))
            sf.text(
                (x+.5)/len(styles), (y+.5)/(len(families)+1), s,
                math_fontfamily=fs, size=16, ha="center")

plt.show()

Here is a related issue that probably can be solved simultaneously (and is another twist of the same topic): #22056

Hi, getting bold italics on matplotlib would be interesting, if no one is working on this, I would like to take this issue.