Someone also asked the same question on Discourse:
https://discourse.matplotlib.org/t/how-to-make-mathdefault-the-default/20120?u=qzhang234

It is possible to set a single font for the entire figure, but multiple rcParams have to be set (as far as I know—if that's not the case, hopefully somebody will correct me). As a temporary workaround to avoid having to type mathdefault over and over again, if it is specifically Times New Roman that is needed, consider using STIX, which is designed to blend with Times New Roman.

import matplotlib.pyplot as plt
plt.rc('mathtext', fontset='stix')
plt.rc('font', family='STIXGeneral')
plt.text(.1, .5, 'Normal Text\n$y=e^x$\n$\mathbf{A}x=b$', size=20)
plt.show()

Any other font can be used as well.

import matplotlib.pyplot as plt
plt.rc('mathtext', fontset='custom', it='Segoe Script:italic', bf='Segoe Script:bold', rm='Segoe Script')
plt.rc('font', family='Segoe Script')
plt.text(.1, .5, 'Normal Text\n$y=e^x$\n$\mathbf{A}x=b$', size=20)
plt.show()

(It may be necessary to delete the font cache if you are using a recently-installed font.)

I believe that the solution linked on discourse (setting mathtext.fontset = "custom", mathtext.rm = ..., mathtext.sf = ... as desired) is effectively the correct one, and using STIX as suggested by @tfpf should work too. Closing, but feel free to ping for reopen if you think more should be done.