More fixes for doc building with python 3 #1383

dmcdougall · 2012-10-13T17:19:17Z

In #1361, @cjmayo pointed out some extra fixes that needed to be implemented to make doc building under python 3 work properly. These have been modified to not break backward compatibility with doc building in python 2.

dmcdougall · 2012-10-13T17:33:25Z

@cjmayo Let me know if this works with python 3. I have no reason to believe it shouldn't.

Edit: Er, I mean let me know if it doesn't work with python 3.

mdboom · 2012-10-13T22:52:54Z

+1. This works for me. Thanks for taking care of this -- building the docs under Python 3 is something that fell through the cracks during the Python 3 port.

dmcdougall · 2012-10-13T23:11:07Z

Thanks goes to @cjmayo for providing the original patches.

cjmayo · 2012-10-14T12:31:46Z

df93eca and 19b1a05 work with Python 3 for me and I can get the docs to build (I can't formally verify but I have lots of html, images and pdfs and all the links I have tried in the html work). But it is a bit more complicated.

I can only get this to work with Python 3.3.0 and not with Python 3.2.3 (specifically Gentoo Linux python-3.2.3). 3.2 fails with:

sphinx 1.1.3-r3
reading sources... [ 80%] examples/pylab_examples/unicode_demo   
Sphinx error:
'ascii' codec can't decode byte 0xc3 in position 134: ordinal not
Building HTML failed.

and I haven't figured that out.

My recipe:

Dependencies needed for Python 3.3 compatibility:

docutils 0.9.1 plus patch combining:
http://repo.or.cz/w/docutils.git/patch/12e5aa7b94e0a0ec37e754527e06707005d28d1c
http://repo.or.cz/w/docutils.git/patch/46a3a9f46ebda4cb49e73d2edf5cdb5851d98a9f
sphinx tip (6bb9dfbdf66e)

Matplotlib library patches (most of these probably break Python 2 use so need work):

I have to revert the manual reference count patches to ft2font.cpp (for Python 2.7.3 also), and it seems others maybe have the same problem in Issue matplotlib.tests.test_mathtext.mathfont_cm_23_test.test makes python debug to crash #1309. But for yet others it fixed it stix_fonts_demo.py fails with bad refcount #1054???
To cope with the seeming 2to3 import exception bug http://bugs.python.org/issue16214

--- lib/matplotlib/sphinxext/plot_directive.py.orig
+++ lib/matplotlib/sphinxext/plot_directive.py
@@ -125,8 +125,7 @@
 """
 from __future__ import print_function

-import sys, os, glob, shutil, imp, warnings, cStringIO, re, textwrap, \
-       traceback, exceptions
+import sys, os, glob, shutil, imp, warnings, cStringIO, re, textwrap, traceback

 from docutils.parsers.rst import directives
 from docutils import nodes

hash functions in Python 3 take bytes not strings:

--- lib/matplotlib/sphinxext/mathmpl.py.orig
+++ lib/matplotlib/sphinxext/mathmpl.py
@@ -65,7 +65,7 @@
 def latex2html(node, source):
     inline = isinstance(node.parent, nodes.TextElement)
     latex = node['latex']
-    name = 'math-%s' % md5(latex).hexdigest()[-10:]
+    name = 'math-%s' % md5(latex.encode()).hexdigest()[-10:]

     destdir = os.path.join(setup.app.builder.outdir, '_images', 'mathmpl')
     if not os.path.exists(destdir):

Matplotlib documentation build system patches:

make.py, not sure what the final proposal was from Make make.py build docs under python 3 #1361 I just do:

--- doc/make.py.orig
+++ doc/make.py
@@ -5,108 +5,9 @@
 import glob
 import os
 import shutil
+from shutil import ignore_patterns, copytree
 import sys

-### Begin compatibility block for pre-v2.6: ###
-#
-# ignore_patterns and copytree funtions are copies of what is included
-# in shutil.copytree of python v2.6 and later.
-#
....

df93eca
19b1a05
Simple 2to3 on gen_gallery.py:

--- doc/sphinxext/gen_gallery.py.orig
+++ doc/sphinxext/gen_gallery.py
@@ -121,18 +121,18 @@

     gallery_path = os.path.join(app.builder.srcdir, '_templates', 'gallery.html')
     if os.path.exists(gallery_path):
-        fh = file(gallery_path, 'r')
+        fh = open(gallery_path, 'r')
         regenerate = fh.read() != content
         fh.close()
     else:
         regenerate = True
     if regenerate:
-        fh = file(gallery_path, 'w')
+        fh = open(gallery_path, 'w')
         fh.write(content)
         fh.close()

     for key in app.builder.status_iterator(
-        thumbnails.iterkeys(), "generating thumbnails... ",
+        iter(thumbnails.keys()), "generating thumbnails... ",
         length=len(thumbnails)):
         image.thumbnail(key, thumbnails[key], 0.3)

mdboom · 2012-10-15T16:09:34Z

Sorry for the noise. I was wrong about this "working for me". I was inadvertently using the Python 2.x installation of Sphinx. I'm working through some remaining Python 3 issues and will have a PR against this soon.

dmcdougall · 2012-10-15T19:55:27Z

@cjmayo Have those docutils patches been merged upstream? We might not be able to get this in for 1.2 if we have to wait for upstream changes.

cjmayo · 2012-10-15T20:16:27Z

Yes they are direct from upstream:
http://docutils.sourceforge.net/docs/dev/repository.html#id5
It's just the git interface is a bit easier to browse.

But they are only needed for Python 3.3. All the other the other matplotlib fixes will be needed for Python 3.2 too and there is something else needed for 3.2 for that 'ascii' codec error.

dmcdougall · 2012-10-15T21:04:47Z

Having major headaches with python2/3 unicode differences. Bleh. I'm trying to use the utilities from six.py but it claims six is not a module...

dmcdougall · 2012-10-16T09:01:17Z

@cjmayo I have an idea about that ascii error on python 3.2. I'm pretty sure in python 3.2 things like u'asd' are invalid syntax, but in python 3.3 u'asd' is allowed.

dmcdougall · 2012-10-16T09:01:59Z

I'm moving the commits from #1361 into this branch.

dmcdougall · 2012-10-16T09:05:46Z

@mdboom Are you saying running python3.2 make.py is not sufficient, and that I need a separate python3 build of sphinx installed?

mdboom · 2012-10-16T14:08:36Z

@dmcdougall: Yes -- you'll need to install sphinx in Python 3 (using pip-3.2 install sphinx or similar, or from a package), but then you also need to make sure that the Python 3 version of sphinx-build is the default on the path, probably by manually overriding the PATH environment variable.

dmcdougall · 2012-10-16T15:09:37Z

$ which sphinx-build
/opt/local/Library/Frameworks/Python.framework/Versions/3.2/bin/sphinx-build
$ python --version
Python 3.2.3
$ python make.py
Running Sphinx v1.1.3
Initializing GitHub plugin
loading pickled environment... done
animation, api, axes_grid, event_handling, misc, mplot3d, old_animation, pylab_examples, tests, units, user_interfaces, widgets, 
building [html]: targets for 566 source files that are out of date
updating environment: 0 added, 53 changed, 0 removed
reading sources... [  5%] api/artist_api                                                                                          
Exception occurred:
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.2/lib/python3.2/site-packages/sphinx/ext/inheritance_diagram.py", line 146, in _class_info
    builtins = list(vars(__builtin__).values())
NameError: global name '__builtin__' is not defined
The full traceback has been saved in /var/folders/rs/pqvv7l192xs426r_016501f80000gn/T/sphinx-err-v0lp1g.log, if you want to report the issue to the developers.

mdboom · 2012-10-16T15:13:23Z

@dmcdougall: You're farther along than I ever got at this point. That traceback is too promising -- it suggests that the inheritance_diagram hasn't yet been updated for Python 3 (I guess). I wonder if disabling that plugin in our conf.py let's it go further, or whether there are just other problems hiding behind that one?

dmcdougall · 2012-10-16T15:29:35Z

@mdboom I'm using all of the changes in this PR. I added some commits earlier today. I also merged the ones from #1361 to keep everything together.

MacPorts doesn't seem to sport a py31/py32 version of sphinx. I have no experience with sphinx. My guess is that support for python 3 is minimal. I'm not sure where to go from here.

dmcdougall · 2012-10-16T16:55:01Z

@mdboom Commenting out the inheritance diagram in conf.py it mostly works. I get the following error, but the doc build still completes!

<autodoc>:0: ERROR: Unknown interpreted text role "function".
/Users/damon/git/github/matplotlib/doc/devel/documenting_mpl.rst:384: ERROR: Unknown directive type "inheritance-diagram".

.. inheritance-diagram:: matplotlib.patches matplotlib.lines matplotlib.text
   :parts: 2
/Users/damon/git/github/matplotlib/doc/devel/transformations.rst:5: ERROR: Unknown directive type "inheritance-diagram".

.. inheritance-diagram:: matplotlib.transforms matplotlib.path
   :parts: 1
/Users/damon/git/github/matplotlib/doc/examples/pylab_examples/multipage_pdf.rst:8: WARNING: Exception occurred in plotting multipage_pdf
 from /Users/damon/git/github/matplotlib/doc/mpl_examples/pylab_examples/multipage_pdf.py:
Traceback (most recent call last):
  File "/Users/damon/python/lib/matplotlib/sphinxext/plot_directive.py", line 515, in run_code
    exec(code, ns)
  File "<string>", line 36
    d['Author'] = u'Jouni K. Sepp\xe4nen'

dmcdougall · 2012-10-16T16:55:29Z

One of them is a unicode issue, because u'asd' is not valid syntax in py3.2.

mdboom · 2012-10-16T18:40:59Z

It's great that it completes. That unicode issue is probably fixable -- something along the lines of

'Jouni K. Sepp\xe4nen'.decode('latin-1')

If that's the only problem, we're in pretty good shape.

dmcdougall · 2012-10-16T19:20:02Z

I'm also getting this: loading pickled environment... failed: unsupported pickle protocol: 3.

Edit: but the build still completes.

dmcdougall · 2012-10-16T19:29:48Z

Er, nevermind. Turns out the pickling thing gets done to save time on subsequent runs? Since I keep switching between python2 and python3 the pickled environment gets corrupted but it gets regenerated correctly. Disregard my previous comment.

cjmayo · 2012-10-16T19:39:48Z

I can get past the ascii error with Python 3.2 with an updated version of the gen_gallery.py patch above:

--- doc/sphinxext/gen_gallery.py.orig
+++ doc/sphinxext/gen_gallery.py
@@ -20,7 +20,7 @@
 {%% endblock %%}
 """

-import os, glob, re, sys, warnings
+import io, os, glob, re, sys, warnings
 import matplotlib.image as image

 multiimage = re.compile('(.*?)(_\d\d){1,2}')
@@ -121,18 +121,18 @@

     gallery_path = os.path.join(app.builder.srcdir, '_templates', 'gallery.html')
     if os.path.exists(gallery_path):
-        fh = file(gallery_path, 'r')
+        fh = open(gallery_path, 'r')
         regenerate = fh.read() != content
         fh.close()
     else:
         regenerate = True
     if regenerate:
-        fh = file(gallery_path, 'w')
+        fh = io.open(gallery_path, 'w', encoding='utf8')
         fh.write(content)
         fh.close()

     for key in app.builder.status_iterator(
-        thumbnails.iterkeys(), "generating thumbnails... ",
+        iter(thumbnails.keys()), "generating thumbnails... ",
         length=len(thumbnails)):
         image.thumbnail(key, thumbnails[key], 0.3)

And a new patch to plot_directive.py:

--- lib/matplotlib/sphinxext/plot_directive.py.orig
+++ lib/matplotlib/sphinxext/plot_directive.py
@@ -125,6 +125,7 @@
 """
 from __future__ import print_function

+import io
 import sys, os, glob, shutil, imp, warnings, cStringIO, re, textwrap, \
        traceback, exceptions

@@ -667,7 +668,7 @@
         else:
             function_name = None

-        with open(source_file_name, 'r') as fd:
+        with io.open(source_file_name, 'r', encoding='utf8') as fd:
             code = fd.read()
         output_base = os.path.basename(source_file_name)
     else:
@@ -809,7 +810,7 @@

     # copy script (if necessary)
     target_name = os.path.join(dest_dir, output_base + source_ext)
-    with open(target_name, 'w') as f:
+    with io.open(target_name, 'w', encoding='utf8') as f:
         if source_file_name == rst_file:
             code_escaped = unescape_doctest(code)
         else:

This now gets me through the html generation and also png creation. Fails now in using graphviz.

Couple of comments on the recent comments.

On the 'Jouni K. Sepp\xe4nen' would from __future__ import unicode_literals work as used in examples/pylab_examples/unicode_demo.py?

Have you tested:

-    name = 'math-%s' % md5(latex).hexdigest()[-10:]
+    name = 'math-%s' % md5(latex.encode()).hexdigest()[-10:]

on Python 2?

dmcdougall · 2012-10-16T19:49:14Z

@cjmayo Cheers for the patch updates. For some reason I don't have a problem with the current ones, but I'll add your updates in since they work for you.

On the 'Jouni K. Sepp\xe4nen' would from __future__ import unicode_literals work as used in examples/pylab_examples/unicode_demo.py?

Probably. I didn't know this existed. I have never worked with python 3, so this whole PR/frustration is a combination of me learning and getting headaches.

Have you tested:

-    name = 'math-%s' % md5(latex).hexdigest()[-10:]
+    name = 'math-%s' % md5(latex.encode()).hexdigest()[-10:]

on Python 2?

Nope. All the testing I did was the generate the documentation under python2, and the build succeeds. I have not checked the output.

dmcdougall · 2012-10-16T19:50:09Z

I am getting a new error building the docs under python 3:

Running Sphinx v1.1.3

Exception occurred:
  File "/Users/damon/python/lib/matplotlib/dates.py", line 122, in <module>
    from dateutil.rrule import rrule, MO, TU, WE, TH, FR, SA, SU, YEARLY, \
  File "/Users/damon/python/lib/dateutil/rrule.py", line 55
    raise ValueError, "Can't create weekday with n == 0"
                    ^
SyntaxError: invalid syntax

I have no idea why this is happening. 2to3 should have fixed this. Argh! I think it's time for me to call it a day and look at this tomorrow.

cjmayo · 2012-10-17T06:39:15Z

With the last two patches I can now get to the end of the doc building with Python 3.2.3, but I have to use sphinx tip (6bb9dfbdf66e) - sphinx 1.1.3 does not work for me.

dmcdougall · 2012-10-17T10:12:03Z

@cjmayo Progress! Interesting that you need current Sphinx HEAD. What errors do you get with sphinx 1.1.3?

We have two options on proceeding depending on your response to my above question:

We may be able to patch our sphinxext files to work with sphinx v1.1.3.
If 1) is not possible, then this pull request will fester until the problem is solved upstream.

mdboom · 2012-10-17T13:10:01Z

At the end of the day, the most important thing is to be able to produce the docs under some version of Python (and Python 2.x is currently working). Having them build under 3.x is of course very desirable, but if that can't be done with a stable version of Sphinx, than there's not much point to putting this is our release either. IMHO, of course.

dmcdougall · 2012-10-17T13:15:09Z

@mdboom Agreed; there are more important v1.2.x bugs currently milestoned. Furthermore, I think if we are waiting in either case the v1.2.x milestone is unnecessary.

I will keep this PR open and remove the milestone. If anybody disagrees then adding it back on is fine.

ariddell · 2012-12-07T16:01:06Z

Even with these changes, I'm encountering an import error related to this bug: RuntimeError: dictionary changed size during iteration from colors.py #1401

pelson · 2012-12-10T11:20:39Z

Thanks @ariddell - the error you are describing is a matplotlib issue relating to python3.3 to try and keep the issues separate, this build process fix is for the documentation building with a currently supported python (lets say python3.2 for sake of argument). So whilst I think what you highlight is important, it should be addressed as a completely independent issue (#1401).

Cheers,

pelson · 2012-12-10T11:27:48Z

@dmcdougall - do you remember the status of this? Is it worth re-visiting and getting this PR merged?

ariddell · 2012-12-10T12:31:31Z

I'm getting the error with 3.2.3.

I've setup a clean virtualenv, pip install'd only numpy and git://github.com/dmcdougall/matplotlib.git@py3_docs_2#egg=matplotlib and I get this:

Python 3.2.3 (default, Oct 19 2012, 19:53:16) 
[GCC 4.7.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import matplotlib.sphinxext.plot_directive
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/abr/.virtualenvs/matplotlib-py3.2/src/matplotlib/lib/matplotlib/__init__.py", line 151, in <module>
    from matplotlib.rcsetup import (defaultParams,
  File "/home/abr/.virtualenvs/matplotlib-py3.2/src/matplotlib/lib/matplotlib/rcsetup.py", line 20, in <module>
    from matplotlib.colors import is_color_like
  File "/home/abr/.virtualenvs/matplotlib-py3.2/src/matplotlib/lib/matplotlib/colors.py", line 205, in <module>
    for k, v in cnames.items():
RuntimeError: dictionary changed size during iteration

Python path is: ['', '/home/abr/.virtualenvs/matplotlib-py3.2/lib/python3.2/site-packages/distribute-0.6.24-py3.2.egg', '/home/abr/.virtualenvs/matplotlib-py3.2/lib/python3.2/site-packages/pip-1.1-py3.2.egg', '/home/abr/.virtualenvs/matplotlib-py3.2/lib/python3.2', '/home/abr/.virtualenvs/matplotlib-py3.2/lib/python3.2/plat-linux2', '/home/abr/.virtualenvs/matplotlib-py3.2/lib/python3.2/lib-dynload', '/usr/lib/python3.2', '/usr/lib/python3.2/plat-linux2', '/home/abr/.virtualenvs/matplotlib-py3.2/lib/python3.2/site-packages']

Hope this helps. Thanks for all your work on this.

pelson · 2012-12-10T13:01:36Z

I'm getting the error with 3.2.3.

Ah... Ok - thanks for pointing that out. In that case I think it is very pertinent for this PR.

Cheers,

pelson · 2013-03-29T23:45:57Z

@dmcdougall - I'd be happy to merge this. Is there more work to be done before I do, or shall I go ahead and press the green button?

dmcdougall · 2013-04-01T19:00:36Z

@pelson It appears that @cjmayo is currently operational with these changes, and that @ariddell is having issues with that dictionary error. That error, however, appears to have been resolved in the comments in #1401. So, I think we should merge this. If there are further dictionary errors, we can resolve those separately.

Cheers @cjmayo for the help.

More fixes for doc building with python 3

dmcdougall added 2 commits October 13, 2012 18:07

Make conf.py work with py3

df93eca

Make gen_rst behave with py3

19b1a05

dmcdougall mentioned this pull request Oct 13, 2012

Make make.py build docs under python 3 #1361

Closed

dmcdougall added 5 commits October 16, 2012 09:20

Make plot_directive py3 friendly

818cf19

Make make.py build docs under python 3

91319b1

Make call to raise py3 compatible

222e742

Fix py2/3 unicode differences

196636f

Simple 2to3 on gen_gallery.py

600fd2d

Make md5 take byte as argument, not string

5e36bb7

mdboom mentioned this pull request Dec 6, 2012

CXX/Python2/cxx_extensions.cxx:1320: Assertion `ob_refcnt == 0' #1532

Merged

pelson added a commit that referenced this pull request Apr 2, 2013

Merge pull request #1383 from dmcdougall/py3_docs_2

1d6abc1

More fixes for doc building with python 3

pelson merged commit 1d6abc1 into matplotlib:v1.2.x Apr 2, 2013

dmcdougall mentioned this pull request Apr 2, 2013

Docs don't build with Python3 (make.py except) #1356

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

More fixes for doc building with python 3 #1383

More fixes for doc building with python 3 #1383

dmcdougall commented Oct 13, 2012

dmcdougall commented Oct 13, 2012

mdboom commented Oct 13, 2012

dmcdougall commented Oct 13, 2012

cjmayo commented Oct 14, 2012

mdboom commented Oct 15, 2012

dmcdougall commented Oct 15, 2012

cjmayo commented Oct 15, 2012

dmcdougall commented Oct 15, 2012

dmcdougall commented Oct 16, 2012

dmcdougall commented Oct 16, 2012

dmcdougall commented Oct 16, 2012

mdboom commented Oct 16, 2012

dmcdougall commented Oct 16, 2012

mdboom commented Oct 16, 2012

dmcdougall commented Oct 16, 2012

dmcdougall commented Oct 16, 2012

dmcdougall commented Oct 16, 2012

mdboom commented Oct 16, 2012

dmcdougall commented Oct 16, 2012

dmcdougall commented Oct 16, 2012

cjmayo commented Oct 16, 2012

dmcdougall commented Oct 16, 2012

dmcdougall commented Oct 16, 2012

cjmayo commented Oct 17, 2012

dmcdougall commented Oct 17, 2012

mdboom commented Oct 17, 2012

dmcdougall commented Oct 17, 2012

ariddell commented Dec 7, 2012

pelson commented Dec 10, 2012

pelson commented Dec 10, 2012

ariddell commented Dec 10, 2012

pelson commented Dec 10, 2012

pelson commented Mar 29, 2013

dmcdougall commented Apr 1, 2013

More fixes for doc building with python 3 #1383

More fixes for doc building with python 3 #1383

Conversation

dmcdougall commented Oct 13, 2012

dmcdougall commented Oct 13, 2012

mdboom commented Oct 13, 2012

dmcdougall commented Oct 13, 2012

cjmayo commented Oct 14, 2012

mdboom commented Oct 15, 2012

dmcdougall commented Oct 15, 2012

cjmayo commented Oct 15, 2012

dmcdougall commented Oct 15, 2012

dmcdougall commented Oct 16, 2012

dmcdougall commented Oct 16, 2012

dmcdougall commented Oct 16, 2012

mdboom commented Oct 16, 2012

dmcdougall commented Oct 16, 2012

mdboom commented Oct 16, 2012

dmcdougall commented Oct 16, 2012

dmcdougall commented Oct 16, 2012

dmcdougall commented Oct 16, 2012

mdboom commented Oct 16, 2012

dmcdougall commented Oct 16, 2012

dmcdougall commented Oct 16, 2012

cjmayo commented Oct 16, 2012

dmcdougall commented Oct 16, 2012

dmcdougall commented Oct 16, 2012

cjmayo commented Oct 17, 2012

dmcdougall commented Oct 17, 2012

mdboom commented Oct 17, 2012

dmcdougall commented Oct 17, 2012

ariddell commented Dec 7, 2012

pelson commented Dec 10, 2012

pelson commented Dec 10, 2012

ariddell commented Dec 10, 2012

pelson commented Dec 10, 2012

pelson commented Mar 29, 2013

dmcdougall commented Apr 1, 2013