Thanks to visit codestin.com
Credit goes to github.com

Skip to content

Commit 573f974

Browse files
larsonerTitan-C
larsoner
authored and
Titan-C
committed
API: Refactor image scraping (#313)
* API: Refactor image scraping * FIX: Py2 * FIX: One for old pytest * FIX: Doc * FIX: Fix for old mpl * FIX: Order * FIX: Broader API * FIX: Bad example * API: Refoctar into submodule * FIX: Fix test
1 parent 1d6015c commit 573f974

File tree

11 files changed

+769
-325
lines changed

11 files changed

+769
-325
lines changed

‎doc/advanced_configuration.rst

Lines changed: 114 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -26,7 +26,9 @@ file:
2626
- ``line_numbers`` (:ref:`adding_line_numbers`)
2727
- ``download_all_examples`` (:ref:`disable_all_scripts_download`)
2828
- ``plot_gallery`` (:ref:`without_execution`)
29-
- ``find_mayavi_figures`` (:ref:`find_mayavi`)
29+
- ``image_scrapers`` (and the deprecated ``find_mayavi_figures``)
30+
(:ref:`image_scrapers`)
31+
- ``reset_modules`` (:ref:`reset_modules`)
3032
- ``abort_on_example_error`` (:ref:`abort_on_first`)
3133
- ``expected_failing_examples`` (:ref:`dont_fail_exit`)
3234
- ``min_reported_time`` (:ref:`min_reported_time`)
@@ -520,20 +522,124 @@ The highest precedence is always given to the `-D` flag of the
520522
``sphinx-build`` command.
521523

522524

523-
.. _find_mayavi:
525+
.. _image_scrapers:
526+
527+
Image scrapers
528+
==============
529+
530+
By default, at the end of each code block Sphinx-gallery will detect new
531+
:mod:`matplotlib.pyplot` figures and turn them into images. This behavior is
532+
equivalent to the default of::
533+
534+
sphinx_gallery_conf = {
535+
...
536+
'image_scrapers': ('matplotlib',),
537+
}
538+
539+
Built-in support is also provided for finding :mod:`Mayavi <mayavi.mlab>`
540+
figures. To enable this feature, you can do::
541+
542+
sphinx_gallery_conf = {
543+
...
544+
'image_scrapers': ('matplotlib', 'mayavi'),
545+
}
546+
547+
.. note:: The parameter ``find_mayavi_figures`` which can also be used to
548+
extract Mayavi figures is **deprecated** in version 0.2+,
549+
and will be removed in a future release.
550+
551+
Custom scrapers
552+
^^^^^^^^^^^^^^^
553+
554+
.. note:: The API for custom scrapers is currently experimental.
555+
556+
You can also add your own custom function (or callable class instance)
557+
to this list. See :func:`sphinx_gallery.scrapers.matplotlib_scraper` for
558+
a description of the interface. Here is pseudocode for what this function
559+
and :func:`sphinx_gallery.scrapers.mayavi_scraper` do under the hood, which
560+
uses :func:`sphinx_gallery.scrapers.figure_rst` to create standardized RST::
561+
562+
def mod_scraper(block, block_vars, gallery_conf)
563+
import mymod
564+
image_names = list()
565+
image_path_iterator = block_vars['image_path_iterator']
566+
for fig, image_path in zip(mymod.get_figures(), image_path_iterator):
567+
fig.save_png(image_path)
568+
image_names.append(image_path)
569+
mymod.close('all')
570+
return figure_rst(image_names, gallery_conf['src_dir'])
571+
572+
573+
For example, a naive class to scrape any new PNG outputs in the
574+
current directory could do, e.g.::
575+
576+
import glob
577+
import shutil
578+
from sphinx_gallery.gen_rst import figure_rst
524579

525-
Finding Mayavi figures
526-
======================
527-
By default, Sphinx-gallery will only look for :mod:`matplotlib.pyplot` figures
528-
when building. However, extracting figures generated by :mod:`mayavi.mlab` is
529-
also supported. To enable this feature, you can do::
580+
581+
class PNGScraper(object):
582+
def __init__(self):
583+
self.seen = set()
584+
585+
def __call__(self, block, block_vars, gallery_conf):
586+
pngs = sorted(glob.glob(os.path.join(os.getcwd(), '*.png'))
587+
image_names = list()
588+
image_path_iterator = block_vars['image_path_iterator']
589+
for png in pngs:
590+
if png not in seen:
591+
seen |= set(png)
592+
image_names.append(image_path_iterator.next())
593+
shutil.copyfile(png, image_names[-1])
594+
return figure_rst(image_names, gallery_conf['src_dir'])
595+
596+
597+
sphinx_gallery_conf = {
598+
...
599+
'image_scrapers': ('matplotlib', PNGScraper()),
600+
}
601+
602+
If you have an idea for a scraper that is likely to be useful to others
603+
(e.g., a fully functional image-file-detector, or one for another visualization
604+
libarry), feel free to contribute it on GitHub!
605+
606+
607+
.. _reset_modules:
608+
609+
Resetting modules
610+
=================
611+
612+
By default, at the end of each file ``matplotlib`` is reset using
613+
:func:`matplotlib.pyplot.rcdefaults` and ``seaborn`` is reset by trying
614+
to unload the module from ``sys.modules``. This is equivalent to the
615+
entries::
616+
617+
sphinx_gallery_conf = {
618+
...
619+
'reset_modules': ('matplotlib', 'seaborn'),
620+
}
621+
622+
You can also add your own custom function to this tuple.
623+
The function takes two variables, the ``gallery_conf`` dict and the
624+
``fname``, which on the first call is the directory name being processed, and
625+
then the names of all files within that directory.
626+
For example, to reset matplotlib to always use the ``ggplot`` style, you could do::
627+
628+
629+
def reset_mpl(gallery_conf, fname):
630+
from matplotlib import style
631+
style.use('ggplot')
530632

531633

532634
sphinx_gallery_conf = {
533635
...
534-
'find_mayavi_figures': True,
636+
'reset_modules': (reset_mpl,),
535637
}
536638

639+
.. note:: Using resetters such as ``reset_mpl`` that deviate from
640+
standard behaviors that users will experience when manually running
641+
examples themselves is discouraged due to the inconsistency
642+
that results between the rendered examples and local outputs.
537643

538644
Dealing with failing Gallery example scripts
539645
============================================
@@ -660,5 +766,4 @@ The ``min_reported_time`` configuration can be set to a number of seconds. The
660766
duration of scripts that ran faster than that amount will not be logged nor
661767
embedded in the html output.
662768

663-
664769
.. _regular expressions: https://docs.python.org/2/library/re.html

‎doc/conf.py

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -316,14 +316,14 @@ def setup(app):
316316
# Run the mayavi examples and find the mayavi figures if mayavi is
317317
# installed
318318
from mayavi import mlab
319-
find_mayavi_figures = True
319+
image_scrapers = ('matplotlib', 'mayavi')
320320
examples_dirs.append('../mayavi_examples')
321321
gallery_dirs.append('auto_mayavi_examples')
322322
# Do not pop up any mayavi windows while running the
323323
# examples. These are very annoying since they steal the focus.
324324
mlab.options.offscreen = True
325325
except Exception: # can raise all sorts of errors
326-
find_mayavi_figures = False
326+
image_scrapers = ('matplotlib',)
327327

328328

329329
sphinx_gallery_conf = {
@@ -334,11 +334,11 @@ def setup(app):
334334
},
335335
'examples_dirs': examples_dirs,
336336
'gallery_dirs': gallery_dirs,
337+
'image_scrapers': image_scrapers,
337338
'subsection_order': ExplicitOrder(['../examples/sin_func',
338339
'../examples/no_output',
339340
'../tutorials/seaborn']),
340341
'within_subsection_order': NumberOfCodeLinesSortKey,
341-
'find_mayavi_figures': find_mayavi_figures,
342342
'expected_failing_examples': ['../examples/no_output/plot_raise.py',
343343
'../examples/no_output/plot_syntaxerror.py'],
344344
'binder': {'org': 'sphinx-gallery',

‎doc/reference.rst

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -22,6 +22,7 @@ every module.
2222
gen_gallery
2323
backreferences
2424
gen_rst
25+
scrapers
2526
py_source_parser
2627
docs_resolv
2728
notebook

‎sphinx_gallery/gen_gallery.py

Lines changed: 80 additions & 21 deletions
Original file line numberDiff line numberDiff line change
@@ -19,17 +19,24 @@
1919
from sphinx.util.console import red
2020
from . import sphinx_compatibility, glr_path_static, __version__ as _sg_version
2121
from .gen_rst import generate_dir_rst, SPHX_GLR_SIG
22+
from .scrapers import _scraper_dict, _reset_dict
2223
from .docs_resolv import embed_code_links
2324
from .downloads import generate_zipfiles
2425
from .sorting import NumberOfCodeLinesSortKey
25-
from .binder import copy_binder_files, check_binder_conf
26+
from .binder import copy_binder_files
2627

2728
try:
2829
FileNotFoundError
2930
except NameError:
3031
# Python2
3132
FileNotFoundError = IOError
3233

34+
try:
35+
basestring
36+
except NameError:
37+
basestring = str
38+
unicode = str
39+
3340
DEFAULT_GALLERY_CONF = {
3441
'filename_pattern': re.escape(os.sep) + 'plot',
3542
'ignore_pattern': '__init__\.py',
@@ -53,6 +60,8 @@
5360
'thumbnail_size': (400, 280), # Default CSS does 0.4 scaling (160, 112)
5461
'min_reported_time': 0,
5562
'binder': {},
63+
'image_scrapers': ('matplotlib',),
64+
'reset_modules': ('matplotlib', 'seaborn'),
5665
}
5766

5867
logger = sphinx_compatibility.getLogger('sphinx-gallery')
@@ -89,13 +98,31 @@ def parse_config(app):
8998
plot_gallery = eval(app.builder.config.plot_gallery)
9099
except TypeError:
91100
plot_gallery = bool(app.builder.config.plot_gallery)
101+
src_dir = app.builder.srcdir
102+
abort_on_example_error = app.builder.config.abort_on_example_error
103+
gallery_conf = _complete_gallery_conf(
104+
app.config.sphinx_gallery_conf, src_dir, plot_gallery,
105+
abort_on_example_error)
106+
# this assures I can call the config in other places
107+
app.config.sphinx_gallery_conf = gallery_conf
108+
app.config.html_static_path.append(glr_path_static())
109+
return gallery_conf
92110

111+
112+
def _complete_gallery_conf(sphinx_gallery_conf, src_dir, plot_gallery,
113+
abort_on_example_error):
93114
gallery_conf = copy.deepcopy(DEFAULT_GALLERY_CONF)
94-
gallery_conf.update(app.config.sphinx_gallery_conf)
115+
gallery_conf.update(sphinx_gallery_conf)
116+
if sphinx_gallery_conf.get('find_mayavi_figures', False):
117+
logger.warning(
118+
"Deprecated image scraping variable `find_mayavi_figures`\n"
119+
"detected, use `image_scrapers` instead as:\n\n"
120+
" image_scrapers=('matplotlib', 'mayavi')",
121+
type=DeprecationWarning)
122+
gallery_conf['image_scrapers'] += ('mayavi',)
95123
gallery_conf.update(plot_gallery=plot_gallery)
96-
gallery_conf.update(
97-
abort_on_example_error=app.builder.config.abort_on_example_error)
98-
gallery_conf['src_dir'] = app.builder.srcdir
124+
gallery_conf.update(abort_on_example_error=abort_on_example_error)
125+
gallery_conf['src_dir'] = src_dir
99126

100127
if gallery_conf.get("mod_example_dir", False):
101128
backreferences_warning = """\n========
@@ -105,15 +132,43 @@ def parse_config(app):
105132
version of Sphinx-Gallery. For more details, see the backreferences
106133
documentation:
107134
108-
https://sphinx-gallery.readthedocs.io/en/latest/advanced_configuration.html#references-to-examples"""
135+
https://sphinx-gallery.readthedocs.io/en/latest/advanced_configuration.html#references-to-examples""" # noqa: E501
109136
gallery_conf['backreferences_dir'] = gallery_conf['mod_example_dir']
110137
logger.warning(
111138
backreferences_warning,
112139
type=DeprecationWarning)
113140

114-
# this assures I can call the config in other places
115-
app.config.sphinx_gallery_conf = gallery_conf
116-
app.config.html_static_path.append(glr_path_static())
141+
# deal with scrapers
142+
scrapers = gallery_conf['image_scrapers']
143+
if not isinstance(scrapers, (tuple, list)):
144+
scrapers = [scrapers]
145+
scrapers = list(scrapers)
146+
for si, scraper in enumerate(scrapers):
147+
if isinstance(scraper, basestring):
148+
if scraper not in _scraper_dict:
149+
raise ValueError('Unknown image scraper named %r' % (scraper,))
150+
scrapers[si] = _scraper_dict[scraper]
151+
elif not callable(scraper):
152+
raise ValueError('Scraper %r was not callable' % (scraper,))
153+
gallery_conf['image_scrapers'] = tuple(scrapers)
154+
del scrapers
155+
156+
# deal with resetters
157+
resetters = gallery_conf['reset_modules']
158+
if not isinstance(resetters, (tuple, list)):
159+
resetters = [resetters]
160+
resetters = list(resetters)
161+
for ri, resetter in enumerate(resetters):
162+
if isinstance(resetter, basestring):
163+
if resetter not in _reset_dict:
164+
raise ValueError('Unknown module resetter named %r'
165+
% (resetter,))
166+
resetters[ri] = _reset_dict[resetter]
167+
elif not callable(resetter):
168+
raise ValueError('Module resetter %r was not callable'
169+
% (resetter,))
170+
gallery_conf['reset_modules'] = tuple(resetters)
171+
del resetters
117172

118173
return gallery_conf
119174

@@ -137,7 +192,8 @@ def get_subsections(srcdir, examples_dir, sortkey):
137192
138193
"""
139194
subfolders = [subfolder for subfolder in os.listdir(examples_dir)
140-
if os.path.exists(os.path.join(examples_dir, subfolder, 'README.txt'))]
195+
if os.path.exists(os.path.join(
196+
examples_dir, subfolder, 'README.txt'))]
141197
base_examples_dir_path = os.path.relpath(examples_dir, srcdir)
142198
subfolders_with_path = [os.path.join(base_examples_dir_path, item)
143199
for item in subfolders]
@@ -214,11 +270,14 @@ def generate_gallery_rst(app):
214270
# :orphan: to suppress "not included in TOCTREE" sphinx warnings
215271
fhindex.write(":orphan:\n\n" + this_fhindex)
216272

217-
for subsection in get_subsections(app.builder.srcdir, examples_dir, gallery_conf['subsection_order']):
273+
for subsection in get_subsections(
274+
app.builder.srcdir, examples_dir,
275+
gallery_conf['subsection_order']):
218276
src_dir = os.path.join(examples_dir, subsection)
219277
target_dir = os.path.join(gallery_dir, subsection)
220-
this_fhindex, this_computation_times = generate_dir_rst(src_dir, target_dir, gallery_conf,
221-
seen_backrefs)
278+
this_fhindex, this_computation_times = \
279+
generate_dir_rst(src_dir, target_dir, gallery_conf,
280+
seen_backrefs)
222281
fhindex.write(this_fhindex)
223282
computation_times += this_computation_times
224283

@@ -258,9 +317,9 @@ def touch_empty_backreferences(app, what, name, obj, options, lines):
258317

259318

260319
def summarize_failing_examples(app, exception):
261-
"""Collects the list of falling examples during build and prints them with the traceback
320+
"""Collects the list of falling examples and prints them with a traceback.
262321
263-
Raises ValueError if there where failing examples
322+
Raises ValueError if there where failing examples.
264323
"""
265324
if exception is not None:
266325
return
@@ -271,9 +330,9 @@ def summarize_failing_examples(app, exception):
271330

272331
gallery_conf = app.config.sphinx_gallery_conf
273332
failing_examples = set(gallery_conf['failing_examples'].keys())
274-
expected_failing_examples = set([os.path.normpath(os.path.join(app.srcdir, path))
275-
for path in
276-
gallery_conf['expected_failing_examples']])
333+
expected_failing_examples = set(
334+
os.path.normpath(os.path.join(app.srcdir, path))
335+
for path in gallery_conf['expected_failing_examples'])
277336

278337
examples_expected_to_fail = failing_examples.intersection(
279338
expected_failing_examples)
@@ -297,9 +356,9 @@ def summarize_failing_examples(app, exception):
297356
failing_examples)
298357
# filter from examples actually run
299358
filename_pattern = gallery_conf.get('filename_pattern')
300-
examples_not_expected_to_pass = [src_file
301-
for src_file in examples_not_expected_to_pass
302-
if re.search(filename_pattern, src_file)]
359+
examples_not_expected_to_pass = [
360+
src_file for src_file in examples_not_expected_to_pass
361+
if re.search(filename_pattern, src_file)]
303362
if examples_not_expected_to_pass:
304363
fail_msgs.append(red("Examples expected to fail, but not failing:\n") +
305364
"Please remove these examples from\n" +

0 commit comments

Comments
 (0)