Unicode with usetex=True and pgf backend #5234

itoijala · 2015-10-13T08:56:58Z

The documentation says that registering the backend should work as well (or better) than use('pgf').
This breaks when using unicode because the TeX strings are handled by the default matplotlib texmanager instead of the pgf backend.

If there is no way to disable the standard texmanager when rendering using the pgf backend, the documentation should be changed to make this problem clear.

import matplotlib as mpl
#mpl.use("pgf")   # works
import matplotlib.pyplot as plt
from matplotlib.backends.backend_pgf import FigureCanvasPgf  # does not work
mpl.backend_bases.register_backend('pdf', FigureCanvasPgf)   #
mpl.rcParams.update({
    'font.family': 'serif',
    'text.usetex': True,
    'pgf.rcfonts': False,
    'pgf.preamble': r'\usepackage{unicode-math}',
})
plt.plot([1, 2], [3, 4])
plt.xlabel(r'$α$')
plt.savefig('foo.pdf')

Traceback (most recent call last):
  File "baz.py", line 19, in <module>
    plt.savefig('baz.pdf')
  File "/usr/lib/python3.5/site-packages/matplotlib/pyplot.py", line 577, in savefig
    res = fig.savefig(*args, **kwargs)
  File "/usr/lib/python3.5/site-packages/matplotlib/figure.py", line 1476, in savefig
    self.canvas.print_figure(*args, **kwargs)
  File "/usr/lib/python3.5/site-packages/matplotlib/backends/backend_qt5agg.py", line 162, in print_figure
    self.draw()
  File "/usr/lib/python3.5/site-packages/matplotlib/backends/backend_qt5agg.py", line 148, in draw
    FigureCanvasAgg.draw(self)
  File "/usr/lib/python3.5/site-packages/matplotlib/backends/backend_agg.py", line 469, in draw
    self.figure.draw(self.renderer)
  File "/usr/lib/python3.5/site-packages/matplotlib/artist.py", line 59, in draw_wrapper
    draw(artist, renderer, *args, **kwargs)
  File "/usr/lib/python3.5/site-packages/matplotlib/figure.py", line 1085, in draw
    func(*args)
  File "/usr/lib/python3.5/site-packages/matplotlib/artist.py", line 59, in draw_wrapper
    draw(artist, renderer, *args, **kwargs)
  File "/usr/lib/python3.5/site-packages/matplotlib/axes/_base.py", line 2110, in draw
    a.draw(renderer)
  File "/usr/lib/python3.5/site-packages/matplotlib/artist.py", line 59, in draw_wrapper
    draw(artist, renderer, *args, **kwargs)
  File "/usr/lib/python3.5/site-packages/matplotlib/axis.py", line 1128, in draw
    self.label.draw(renderer)
  File "/usr/lib/python3.5/site-packages/matplotlib/artist.py", line 59, in draw_wrapper
    draw(artist, renderer, *args, **kwargs)
  File "/usr/lib/python3.5/site-packages/matplotlib/text.py", line 595, in draw
    bbox, info, descent = self._get_layout(renderer)
  File "/usr/lib/python3.5/site-packages/matplotlib/text.py", line 320, in _get_layout
    ismath=ismath)
  File "/usr/lib/python3.5/site-packages/matplotlib/backends/backend_agg.py", line 223, in get_text_width_height_descent
    renderer=self)
  File "/usr/lib/python3.5/site-packages/matplotlib/texmanager.py", line 670, in get_text_width_height_descent
    dvifile = self.make_dvi(tex, fontsize)
  File "/usr/lib/python3.5/site-packages/matplotlib/texmanager.py", line 394, in make_dvi
    texfile = self.make_tex(tex, fontsize)
  File "/usr/lib/python3.5/site-packages/matplotlib/texmanager.py", line 309, in make_tex
    fh.write(s.encode('ascii'))
UnicodeEncodeError: 'ascii' codec can't encode character '\u03b1' in position 298: ordinal not in range(128)

The text was updated successfully, but these errors were encountered:

mdboom · 2015-10-13T12:27:09Z

Cc: @pwuertz

pwuertz · 2015-10-13T16:07:44Z

I am pretty sure that it did work at some point, but it's true that usetex true + Unicode cannot be handled by other backends that are using pdflatex. So maybe it should be made clear that this combination of parameters is only possible with xe or lualatex.

What puzzles me is the error you are seeing. Your savefig is calling into the qt5 backend instead of pgf. I will try to find out why.

QuLogic · 2015-10-13T17:44:18Z

usetex true + Unicode cannot be handled by other backends that are using pdflatex

Why's that? pdflatex mostly will work with UTF-8 if you set the input encoding, though it somewhat depends on what version of everything is installed.

pwuertz · 2015-10-13T18:56:01Z

AFAIK pdflatex has no support for unicode fonts and generally doesn't know how to handle non-ascii characters. By including inputenc and a few other packages you can add some hardcoded character-to-glyph mappings so you can get away with common language specific characters. But this isn't exactly what you would call "unicode support".

pwuertz · 2015-10-13T22:10:00Z

Ok, this is a bit confusing, and I'm not sure how to wrap this up in the documentation.
The savefig command actually works, although you see an exception later. The reason for this is that the Qt5 / Agg backend is (and stays) the default backend and after saving the figure it additionally tries to draw its own canvas once. This then fails when the agg backend uses pdflatex to process your unicode label.
One could argue that the additional draw() call after print_figure() isn't required, so if you are not using the GUI the GUI shouldn't complain about the usetex configuration. In case you are using the GUI and backend_pgf for PDF output you'd have to change the rc params to something compatible to qt5 / agg though. This usecase is actually my motivation for preferring the register_backend solution - being able to interact with the figure and to save PGF->PDF figures within the same run.

itoijala · 2015-10-14T06:56:37Z

So basically using the register_backend approach I have to make sure that the labels are compatible with pdflatex. For me that means I will have to use the mpl.use approach. Otherwise using lualatex doesn't have any advantages.

pwuertz · 2015-10-14T11:53:40Z

Not really, the problem is the usetex flag. It isn't really required for backend pgf since all text is processed by tex anyways, so you could just omit it. The GUI will try to process your labels without pdflatex and probably won't be able to interpret more complex latex macros, which looks weird but it won't crash on it. Unfortunately matplotlib internally misuses(?) the usetex flag for some other decisions as well, but if you don't care you can just set usetex=False.

itoijala · 2015-10-14T16:33:29Z

I am using usetex to get the tick labels in the math font.
I guess that's not so important.

pwuertz · 2015-10-14T20:50:08Z

Ah yes, this is one of the things I meant by "other decisions". According to the documentation enabling "usetex" means that the non-latex backends should delegate text rendering to the latex manager and include the result, instead of using the internal matplotlib latex formatter. Unfortunately this flag is also being used for changing the style of the tick labels (putting them in math mode) instead of using a dedicated switch.

By the way, I just realized that there is a setting for enabling unicode input encoding in pdflatex/latexmanager. Maybe this is of interest to you?
http://matplotlib.org/users/usetex.html#usetex-with-unicode

Anyway, if you are not planning on using the GUI and a non-GUI-savefig within the same workflow just go with use("pgf") if this turns out to be more trouble than its worth.

ArchangeGabriel · 2016-06-26T19:07:20Z

I’m not sure whether this is what I’m facing, because your guys talk about pdflatex while I definitively use xelatex for pgf rendering. But still, needing both pdf export and interactive plots, I’m trying the “registering” method, and when saving a pdf:

---------------------------------------------------------------------------
UnicodeEncodeError                        Traceback (most recent call last)
<ipython-input-7-7a82c592d709> in <module>()
     39 #ax.set_title("Mode $m = 1$ ($\mathcal{M}_1 = \infty$)")
     40 
---> 41 plt.savefig("out.pdf",transparent=True,dpi=300,bbox_inches='tight',pad_inches=0.05)
     42 plt.close("all")

/usr/lib/python3.5/site-packages/matplotlib/pyplot.py in savefig(*args, **kwargs)
    686 def savefig(*args, **kwargs):
    687     fig = gcf()
--> 688     res = fig.savefig(*args, **kwargs)
    689     fig.canvas.draw_idle()   # need this if 'transparent=True' to reset colors
    690     return res

/usr/lib/python3.5/site-packages/matplotlib/figure.py in savefig(self, *args, **kwargs)
   1563             self.set_frameon(frameon)
   1564 
-> 1565         self.canvas.print_figure(*args, **kwargs)
   1566 
   1567         if frameon:

/usr/lib/python3.5/site-packages/matplotlib/backends/backend_qt5agg.py in print_figure(self, *args, **kwargs)
    195     def print_figure(self, *args, **kwargs):
    196         FigureCanvasAgg.print_figure(self, *args, **kwargs)
--> 197         self.draw()
    198 
    199 

/usr/lib/python3.5/site-packages/matplotlib/backends/backend_qt5agg.py in draw(self)
    156         # The Agg draw is done here; delaying causes problems with code that
    157         # uses the result of the draw() to update plot elements.
--> 158         FigureCanvasAgg.draw(self)
    159         self.update()
    160 

/usr/lib/python3.5/site-packages/matplotlib/backends/backend_agg.py in draw(self)
    472 
    473         try:
--> 474             self.figure.draw(self.renderer)
    475         finally:
    476             RendererAgg.lock.release()

/usr/lib/python3.5/site-packages/matplotlib/artist.py in draw_wrapper(artist, renderer, *args, **kwargs)
     59     def draw_wrapper(artist, renderer, *args, **kwargs):
     60         before(artist, renderer)
---> 61         draw(artist, renderer, *args, **kwargs)
     62         after(artist, renderer)
     63 

/usr/lib/python3.5/site-packages/matplotlib/figure.py in draw(self, renderer)
   1157         dsu.sort(key=itemgetter(0))
   1158         for zorder, a, func, args in dsu:
-> 1159             func(*args)
   1160 
   1161         renderer.close_group('figure')

/usr/lib/python3.5/site-packages/matplotlib/artist.py in draw_wrapper(artist, renderer, *args, **kwargs)
     59     def draw_wrapper(artist, renderer, *args, **kwargs):
     60         before(artist, renderer)
---> 61         draw(artist, renderer, *args, **kwargs)
     62         after(artist, renderer)
     63 

/usr/lib/python3.5/site-packages/matplotlib/axes/_base.py in draw(self, renderer, inframe)
   2322 
   2323         for zorder, a in dsu:
-> 2324             a.draw(renderer)
   2325 
   2326         renderer.close_group('axes')

/usr/lib/python3.5/site-packages/matplotlib/artist.py in draw_wrapper(artist, renderer, *args, **kwargs)
     59     def draw_wrapper(artist, renderer, *args, **kwargs):
     60         before(artist, renderer)
---> 61         draw(artist, renderer, *args, **kwargs)
     62         after(artist, renderer)
     63 

/usr/lib/python3.5/site-packages/matplotlib/axis.py in draw(self, renderer, *args, **kwargs)
   1118         self._update_label_position(ticklabelBoxes, ticklabelBoxes2)
   1119 
-> 1120         self.label.draw(renderer)
   1121 
   1122         self._update_offset_text_position(ticklabelBoxes, ticklabelBoxes2)

/usr/lib/python3.5/site-packages/matplotlib/artist.py in draw_wrapper(artist, renderer, *args, **kwargs)
     59     def draw_wrapper(artist, renderer, *args, **kwargs):
     60         before(artist, renderer)
---> 61         draw(artist, renderer, *args, **kwargs)
     62         after(artist, renderer)
     63 

/usr/lib/python3.5/site-packages/matplotlib/text.py in draw(self, renderer)
    747 
    748         with _wrap_text(self) as textobj:
--> 749             bbox, info, descent = textobj._get_layout(renderer)
    750             trans = textobj.get_transform()
    751 

/usr/lib/python3.5/site-packages/matplotlib/text.py in _get_layout(self, renderer)
    359                 w, h, d = renderer.get_text_width_height_descent(clean_line,
    360                                                         self._fontproperties,
--> 361                                                         ismath=ismath)
    362             else:
    363                 w, h, d = 0, 0, 0

/usr/lib/python3.5/site-packages/matplotlib/backends/backend_agg.py in get_text_width_height_descent(self, s, prop, ismath)
    227             fontsize = prop.get_size_in_points()
    228             w, h, d = texmanager.get_text_width_height_descent(s, fontsize,
--> 229                                                                renderer=self)
    230             return w, h, d
    231 

/usr/lib/python3.5/site-packages/matplotlib/texmanager.py in get_text_width_height_descent(self, tex, fontsize, renderer)
    673         else:
    674             # use dviread. It sometimes returns a wrong descent.
--> 675             dvifile = self.make_dvi(tex, fontsize)
    676             dvi = dviread.Dvi(dvifile, 72 * dpi_fraction)
    677             try:

/usr/lib/python3.5/site-packages/matplotlib/texmanager.py in make_dvi(self, tex, fontsize)
    397 
    398         if DEBUG or not os.path.exists(dvifile):
--> 399             texfile = self.make_tex(tex, fontsize)
    400             outfile = basefile + '.output'
    401             command = self._get_shell_cmd(

/usr/lib/python3.5/site-packages/matplotlib/texmanager.py in make_tex(self, tex, fontsize)
    312             else:
    313                 try:
--> 314                     fh.write(s.encode('ascii'))
    315                 except UnicodeEncodeError as err:
    316                     mpl.verbose.report("You are using unicode and latex, but "

UnicodeEncodeError: 'ascii' codec can't encode character '\xe9' in position 307: ordinal not in range(128)

ArchangeGabriel · 2016-06-26T19:10:59Z

Oh, indeed the figure is saved (which is the most important point to me), and I also rely on ticks to be in math mode. ;)

ArchangeGabriel · 2016-06-26T19:14:19Z

And finally, I was indeed able to work around this by setting text.latex.unicode = True and also adding text.latex.preamble config with the relevant packages.

ArchangeGabriel · 2017-02-09T16:54:50Z

Just for the record, this is obviously still an issue with 2.0.

github-actions · 2023-03-17T01:57:05Z

This issue has been marked "inactive" because it has been 365 days since the last comment. If this issue is still present in recent Matplotlib releases, or the feature request is still wanted, please leave a comment and this label will be removed. If there are no updates in another 30 days, this issue will be automatically closed, but you are free to re-open or create a new issue if needed. We value issue reports, and this procedure is meant to help us resurface and prioritize issues that have not been addressed yet, not make them disappear. Thanks for your help!

QuLogic · 2023-04-18T01:50:52Z

The traceback is still happening, but it's a little different now:

Traceback (most recent call last):
  File "mpl-tests/issue5234.py", line 16, in <module>
    plt.savefig('foo.pdf')
  File "lib/matplotlib/pyplot.py", line 1082, in savefig
    res = fig.savefig(*args, **kwargs)  # type: ignore[func-returns-value]
  File "lib/matplotlib/figure.py", line 3362, in savefig
    self.canvas.print_figure(fname, **kwargs)
  File "lib/matplotlib/backends/backend_qtagg.py", line 76, in print_figure
    self.draw()
  File "lib/matplotlib/backends/backend_agg.py", line 401, in draw
    self.figure.draw(self.renderer)
  File "lib/matplotlib/artist.py", line 95, in draw_wrapper
    result = draw(artist, renderer, *args, **kwargs)
  File "lib/matplotlib/artist.py", line 72, in draw_wrapper
    return draw(artist, renderer)
  File "lib/matplotlib/figure.py", line 3126, in draw
    mimage._draw_list_compositing_images(
  File "lib/matplotlib/image.py", line 131, in _draw_list_compositing_images
    a.draw(renderer)
  File "lib/matplotlib/artist.py", line 72, in draw_wrapper
    return draw(artist, renderer)
  File "lib/matplotlib/axes/_base.py", line 3061, in draw
    mimage._draw_list_compositing_images(
  File "lib/matplotlib/image.py", line 131, in _draw_list_compositing_images
    a.draw(renderer)
  File "lib/matplotlib/artist.py", line 72, in draw_wrapper
    return draw(artist, renderer)
  File "lib/matplotlib/axis.py", line 1368, in draw
    self.label.draw(renderer)
  File "lib/matplotlib/artist.py", line 72, in draw_wrapper
    return draw(artist, renderer)
  File "lib/matplotlib/text.py", line 716, in draw
    bbox, info, descent = self._get_layout(renderer)
  File "lib/matplotlib/text.py", line 350, in _get_layout
    w, h, d = _get_text_metrics_with_cache(
  File "lib/matplotlib/text.py", line 69, in _get_text_metrics_with_cache
    return _get_text_metrics_with_cache_impl(
  File "lib/matplotlib/text.py", line 77, in _get_text_metrics_with_cache_impl
    return renderer_ref().get_text_width_height_descent(text, fontprop, ismath)
  File "lib/matplotlib/backends/backend_agg.py", line 226, in get_text_width_height_descent
    return super().get_text_width_height_descent(s, prop, ismath)
  File "lib/matplotlib/backend_bases.py", line 647, in get_text_width_height_descent
    return self.get_texmanager().get_text_width_height_descent(
  File "lib/matplotlib/texmanager.py", line 359, in get_text_width_height_descent
    dvifile = cls.make_dvi(tex, fontsize)
  File "lib/matplotlib/texmanager.py", line 291, in make_dvi
    cls._run_checked_subprocess(
  File "lib/matplotlib/texmanager.py", line 254, in _run_checked_subprocess
    raise RuntimeError(
RuntimeError: latex was not able to process the following string:
b'$\\u03b1$'

Here is the full command invocation and its output:

latex -interaction=nonstopmode --halt-on-error --output-directory=tmppnau5tte 9ea043e51f5cc6de5d5eeeb5b2484a5c.tex

This is pdfTeX, Version 3.141592653-2.6-1.40.22 (TeX Live 2021) (preloaded format=latex)
 restricted \write18 enabled.
entering extended mode
(./9ea043e51f5cc6de5d5eeeb5b2484a5c.tex
LaTeX2e <2020-10-01> patch level 4
L3 programming layer <2021-05-07>
(/usr/share/texlive/texmf-dist/tex/latex/base/article.cls
Document Class: article 2020/04/10 v1.4m Standard LaTeX document class
(/usr/share/texlive/texmf-dist/tex/latex/base/size10.clo))
(/usr/share/texlive/texmf-dist/tex/latex/type1cm/type1cm.sty)
(/usr/share/texlive/texmf-dist/tex/latex/cm-super/type1ec.sty
(/usr/share/texlive/texmf-dist/tex/latex/base/t1cmr.fd))
(/usr/share/texlive/texmf-dist/tex/latex/base/inputenc.sty)
(/usr/share/texlive/texmf-dist/tex/latex/geometry/geometry.sty
(/usr/share/texlive/texmf-dist/tex/latex/graphics/keyval.sty)
(/usr/share/texlive/texmf-dist/tex/generic/iftex/ifvtex.sty
(/usr/share/texlive/texmf-dist/tex/generic/iftex/iftex.sty)))
(/usr/share/texlive/texmf-dist/tex/latex/underscore/underscore.sty)
(/usr/share/texlive/texmf-dist/tex/latex/base/textcomp.sty)
(/usr/share/texlive/texmf-dist/tex/latex/l3backend/l3backend-dvips.def)
No file 9ea043e51f5cc6de5d5eeeb5b2484a5c.aux.
*geometry* driver: auto-detecting
*geometry* detected driver: dvips

! Package inputenc Error: Unicode character α (U+03B1)
(inputenc)                not set up for use with LaTeX.

See the inputenc package documentation for explanation.
Type  H <return>  for immediate help.
 ...                                              
                                                  
l.29 {\rmfamily $α
                   $}%
No pages of output.
Transcript written on tmppnau5tte/9ea043e51f5cc6de5d5eeeb5b2484a5c.log.

This one is because pgf.preamble doesn't get applied to the latex drawing in Qt.

But we go through a lot of trouble to use the right canvas when saving, so it's weird that Qt is drawing here. That's because the QtAgg backend overrides print_figure to call draw after. This also occurs in the Wx backend, but not GTK or Tk.

AFAICT, `print_figure` is only called in `savefig`, and thus has no relation to backend drawing. We went through a lot of trouble to make `savefig` self-contained, so if this really had any effect, it would be a bug in it instead. But other backends don't do this override and seem to work fine, so these seem superfluous as well. After this change, just opening a window in Qt and using the save button did not seem to break anything, nor did running `savefig` with an interactive window open. I did not test Wx, but given the callers of `print_figure`, I don't see how it could be different. Fixes matplotlib#5234

tacaswell added the backend: pgf label Oct 13, 2015

tacaswell added this to the next bug fix release (2.0.1) milestone Oct 13, 2015

QuLogic modified the milestones: 2.0.1 (next bug fix release), 2.0.2 (next bug fix release) May 3, 2017

tacaswell modified the milestones: 2.1.1 (next bug fix release), 2.2 (next feature release) Oct 9, 2017

jkseppan added the topic: text/usetex label Jan 19, 2018

github-actions bot added the status: inactive Marked by the “Stale” Github Action label Mar 17, 2023

github-actions bot closed this as not planned Won't fix, can't repro, duplicate, stale Apr 17, 2023

QuLogic reopened this Apr 18, 2023

github-actions bot removed the status: inactive Marked by the “Stale” Github Action label Apr 18, 2023

QuLogic modified the milestones: future releases, v3.8.0 Apr 18, 2023

QuLogic mentioned this issue Apr 18, 2023

Remove print_figure overrides in backend subclasses #25713

Merged

tacaswell closed this as completed in #25713 Apr 19, 2023

QuLogic mentioned this issue Jul 15, 2023

qt: Mark canvas for re-draw after savefig #26309

Merged

2 tasks

anntzer mentioned this issue Apr 16, 2024

[ENH]: Add a QtPgfPng backend? #28086

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

Unicode with usetex=True and pgf backend #5234

Unicode with usetex=True and pgf backend #5234

itoijala commented Oct 13, 2015

mdboom commented Oct 13, 2015

Uh oh!

pwuertz commented Oct 13, 2015

Uh oh!

QuLogic commented Oct 13, 2015

Uh oh!

pwuertz commented Oct 13, 2015

Uh oh!

pwuertz commented Oct 13, 2015

Uh oh!

itoijala commented Oct 14, 2015

Uh oh!

pwuertz commented Oct 14, 2015

Uh oh!

itoijala commented Oct 14, 2015

Uh oh!

pwuertz commented Oct 14, 2015

Uh oh!

ArchangeGabriel commented Jun 26, 2016

Uh oh!

ArchangeGabriel commented Jun 26, 2016

Uh oh!

ArchangeGabriel commented Jun 26, 2016

Uh oh!

ArchangeGabriel commented Feb 9, 2017

Uh oh!

github-actions bot commented Mar 17, 2023

Uh oh!

QuLogic commented Apr 18, 2023

Uh oh!

Uh oh!

Unicode with usetex=True and pgf backend #5234

Unicode with usetex=True and pgf backend #5234

Comments

itoijala commented Oct 13, 2015

mdboom commented Oct 13, 2015

Uh oh!

pwuertz commented Oct 13, 2015

Uh oh!

QuLogic commented Oct 13, 2015

Uh oh!

pwuertz commented Oct 13, 2015

Uh oh!

pwuertz commented Oct 13, 2015

Uh oh!

itoijala commented Oct 14, 2015

Uh oh!

pwuertz commented Oct 14, 2015

Uh oh!

itoijala commented Oct 14, 2015

Uh oh!

pwuertz commented Oct 14, 2015

Uh oh!

ArchangeGabriel commented Jun 26, 2016

Uh oh!

ArchangeGabriel commented Jun 26, 2016

Uh oh!

ArchangeGabriel commented Jun 26, 2016

Uh oh!

ArchangeGabriel commented Feb 9, 2017

Uh oh!

github-actions bot commented Mar 17, 2023

Uh oh!

QuLogic commented Apr 18, 2023

Uh oh!