[ENH] Add custom thumbnails for failing examples #1313

tsbinns · 2024-05-24T16:50:33Z

Problem

Belated follow-up of #1220.

In short, currently expected failing examples use the "BROKEN" stamp as their thumbnail, which cannot be overwitten by including e.g. # sphinx_gallery_thumbnail_path = '_static/my_preferred_thumbnail.png' in the example file.

Setting a different thumbnail could be useful if there is only one part of your example which is designed to fail, but you don't want it to appear as if the entire example is about code failure from the thumbnail.

Possible Solution

This is just one idea I had. I am very open to suggestions if someone thinks there is a better solution.

In the save_thumbnail() function of gen_rst.py, the path to the thumbnails is set for the examples.

Currently, in the case where it is an expected failing example, the sphinx_gallery_thumbnail_path specified in the file will be carried over and the proper path to the thumbnail will be generated, however this will be ignored and the path to the "BROKEN" stamp file will be set.

Summary of the changes:

adds a flag ignore_thumbnail that is default False

sphinx-gallery/sphinx_gallery/gen_rst.py

Line 372 in fc1d423

ignore_thumbnail = False # only used for failing examples with no given thumbnail
if no thumbnail is specified and it is a failing example, ignore_thumbnail is set to True

sphinx-gallery/sphinx_gallery/gen_rst.py

Lines 380 to 381 in fc1d423

if src_file in gallery_conf["failing_examples"]:

ignore_thumbnail = True

when the thumbnail image is finally set...

if it is a failing example and ignore_thumbnail==True (i.e. no thumbnail was specified), the "BROKEN" stamp is used

sphinx-gallery/sphinx_gallery/gen_rst.py

Lines 397 to 398 in fc1d423

    
           if src_file in gallery_conf["failing_examples"] and ignore_thumbnail: 
        
               img = os.path.join(glr_path_static(), "broken_example.png")

otherwise even if it is a failing example, if ignore_thumbnail==False (i.e. a custom thumbnail was specified), this thumbnail will be used in place of the "BROKEN" stamp

sphinx-gallery/sphinx_gallery/gen_rst.py

Lines 399 to 400 in fc1d423

elif os.path.exists(thumbnail_image_path):

img = thumbnail_image_path

In the case that a thumbnail is specified for a failing example and the image does not exist, the default "no_image.png" is used, like it would for a non-failing example.

No changes have been introduced for non-failing examples.

Not sure if this is worthy of a new example, but I included one since it helps here to demonstrate the behaviour. I have no problem if you want this removed before merging.

sphinx-gallery/examples/no_output/plot_raise_thumbnail.py

Lines 1 to 28 in fc1d423

    
           """ 
        
           Example that fails to execute (with a specified thumbnail) 
        
           ========================================================== 
        
           By default, examples with code blocks that raise an error will have the broken 
        
           image stamp as their gallery thumbnail. However, this may not be desired, e.g. 
        
           if only part of the example is expected to fail and it should not look like the 
        
           entire example fails. 
        
           In these cases, the `sphinx_gallery_thumbnail_path` variable can be set to 
        
           specify a desired thumbnail. 
        
           """ 
        
           # Code source: Thomas S. Binns 
        
           # License: BSD 3 clause 
        
           # sphinx_gallery_line_numbers = True 
        
           # sphinx_gallery_thumbnail_path = 'auto_examples/no_output/images/sphx_glr_plot_raise_001.png'  # noqa 
        
           import numpy as np 
        
           import matplotlib.pyplot as plt 
        
           plt.pcolormesh(np.random.randn(100, 100)) 
        
           # %% 
        
           # This block will raise an AssertionError 
        
           assert False

Unit tests

The big thing this is missing is unit tests. I am not super familiar with the inner workings of sphinx-gallery so I do not know where to start with this.

I noticed in test_gen_rst.py the following notes, so it seems like there is not an existing test which I could use as a template for checking the new behaviour (but I may have missed something). Would anyone have some ideas for this?

sphinx-gallery/sphinx_gallery/tests/test_gen_rst.py

Lines 1225 to 1227 in fc1d423

    
           # TODO: test that broken thumbnail does appear when needed 
        
           # TODO: test that examples are executed after a no-plot and produce 
        
           #       the correct image in the thumbnail

larsoner · 2024-05-24T16:55:14Z

It would be best to keep backward compat with existing behavior. So in other words, if nobody changes anything, the code should behave as before.

So one option would be to make any new behavior as opt-in. For example if you add:

# sphinx_gallery_failing_thumbnail = false

to your example, even if the example fails sphinx-gallery could use the standard, non-failing thumbnail processing path. So if you have matplotlib outputs or set sphinx_gallery_thumbnail_path or don't produce any output, it will use your matplotlib figure, thumbnail path, or no_image.png like in a normal example, respectively.

tsbinns · 2024-05-24T17:33:48Z

Makes sense, then the change is also super simple:

sphinx-gallery/sphinx_gallery/gen_rst.py

Lines 394 to 397 in f066eb5

    
           if src_file in gallery_conf["failing_examples"] and file_conf.get( 
        
               "failing_thumbnail", True 
        
           ): 
        
               img = os.path.join(glr_path_static(), "broken_example.png")

And in the example:

sphinx-gallery/examples/no_output/plot_raise_thumbnail.py

Lines 1 to 19 in f066eb5

    
           """ 
        
           Example that fails to execute (with a specified thumbnail) 
        
           ========================================================== 
        
           By default, examples with code blocks that raise an error will have the broken 
        
           image stamp as their gallery thumbnail. However, this may not be desired, e.g. 
        
           if only part of the example is expected to fail and it should not look like the 
        
           entire example fails. 
        
           In these cases, the `sphinx_gallery_failing_thumbnail` variable can be set to 
        
           ``False``, which will change the thumbnail selection to the default behaviour 
        
           as for non-failing examples. 
        
           """ 
        
           # Code source: Thomas S. Binns 
        
           # License: BSD 3 clause 
        
           # sphinx_gallery_line_numbers = True 
        
           # sphinx_gallery_failing_thumbnail = False

If people are happy with this, should the new variable be listed in the "Configurations inside examples" section of the User guide?

larsoner · 2024-05-24T17:56:24Z

Yeah I think that makes sense!

tsbinns · 2024-05-27T09:11:16Z

Yeah I think that makes sense!

Sorry, just to clarify, is this a thumbs up just for the new solution or also for adding to the user guide?

larsoner · 2024-05-28T17:35:20Z

To both -- any new (inline or otherwise) configuration option should be documented in our docs 👍

tsbinns · 2024-05-29T17:39:55Z

Updated the name of the new example to better reflect what it now demonstrates.

Also added an entry into the user guide explaining the new option.

larsoner

Rendered HTML and docs look good, thanks @tsbinns !

larsoner

Argh actually I just realized that there is no modified test. One option would be to modify an example in sphinx_gallery/tests/tinybuild to have this new line and then assert that "broken" does not show up in the rendered HTML for example

lucyleeow

Nice addition, LGTM, just needs the test as Eric suggested.

doc/configuration.rst

Co-authored-by: Lucy Liu <[email protected]>

larsoner · 2024-06-18T16:45:53Z

@tsbinns don't forget about the test when you get a chance!

tsbinns · 2024-06-18T16:54:07Z

@larsoner Sorry, have it on my to-do list but haven't gotten around to it yet. Will try to sort soon!

for more information, see https://pre-commit.ci

lucyleeow · 2024-07-12T03:23:39Z

Fixed a merge conflict - hopefully correctly!

tsbinns · 2024-07-13T18:05:52Z

Sorry again for the delay. Have now added two examples to tinybuild that mirror what's in the actual docs:

an expected failing example where default failing behaviour is used (i.e. the "BROKEN" stamp)
an expected failing example where default thumbnail behaviour is used (i.e. the plot from the example).

The thumbnails are tested here:

sphinx-gallery/sphinx_gallery/tests/test_full.py

Lines 333 to 371 in 7c33f22

    
           def test_thumbnail_expected_failing_examples(sphinx_app, tmpdir): 
        
               """Test thumbnail behaviour for expected failing examples.""" 
        
               import numpy as np 
        
               # Get the "BROKEN" stamp for the default failing example thumbnail 
        
               stamp_fname = op.join( 
        
                   sphinx_app.srcdir, "_static_nonstandard", "broken_example.png" 
        
               ) 
        
               stamp_fname_scaled = str(tmpdir.join("new.png")) 
        
               scale_image( 
        
                   stamp_fname, 
        
                   stamp_fname_scaled, 
        
                   *sphinx_app.config.sphinx_gallery_conf["thumbnail_size"], 
        
               ) 
        
               Image = _get_image() 
        
               broken_stamp = np.asarray(Image.open(stamp_fname_scaled)) 
        
               assert broken_stamp.shape[2] in (3, 4)  # optipng can strip the alpha channel 
        
               # Get thumbnail from example with failing example thumbnail behaviour 
        
               # (i.e. thumbnail should be "BROKEN" stamp) 
        
               thumb_fname = op.join( 
        
                   sphinx_app.outdir, "_images", "sphx_glr_plot_failing_example_thumb.png" 
        
               ) 
        
               thumbnail = np.asarray(Image.open(thumb_fname)) 
        
               assert broken_stamp.shape[:2] == thumbnail.shape[:2] 
        
               corr = np.corrcoef(broken_stamp[..., :3].ravel(), thumbnail[..., :3].ravel())[0, 1] 
        
               assert corr > 0.99  # i.e. thumbnail and "BROKEN" stamp are identical 
        
               # Get thumbnail from example with default thumbnail behaviour 
        
               # (i.e. thumbnail should be the plot from the example, not the "BROKEN" stamp) 
        
               thumb_fname = op.join( 
        
                   sphinx_app.outdir, 
        
                   "_images", 
        
                   "sphx_glr_plot_failing_example_thumbnail_thumb.png", 
        
               ) 
        
               thumbnail = np.asarray(Image.open(thumb_fname)) 
        
               assert broken_stamp.shape[:2] == thumbnail.shape[:2] 
        
               corr = np.corrcoef(broken_stamp[..., :3].ravel(), thumbnail[..., :3].ravel())[0, 1] 
        
               assert corr < 0.99  # i.e. thumbnail and "BROKEN" stamp are not identical

I can't seem to properly run tinybuild on my Windows machine, so I'm not sure if there are some errors for other things I didn't correctly adjust, but the new test passes at least.

Also added broken_example.png to tinybuild/doc/_static_nonstandard, if you think that's an appropriate place for it.

sphinx_gallery/tests/test_full.py

tsbinns · 2024-07-13T18:09:00Z

sphinx_gallery/tests/test_gen_rst.py

@@ -1239,6 +1239,5 @@ def test_newlines(log_collector_wrap):
    assert tee.newlines == tee.output.newlines


-# TODO: test that broken thumbnail does appear when needed


Since the new test also looks at the default behaviour for expected failing examples, can remove this TODO from elsewhere.

…sbinns/sphinx-gallery into add_thumbnails_broken_examples

tsbinns · 2024-07-16T13:02:35Z

sphinx_gallery/tests/test_full.py

-    assert len(suite) == 3
-    assert suite[0].attrib["classname"] == "plot_numpy_matplotlib"
-    assert suite[0][0].tag == "failure", suite[0].attrib["classname"]
-    assert suite[0][0].attrib["message"].startswith("RuntimeError: Forcing")
-    assert suite[1].attrib["classname"] == "plot_scraper_broken"
-    assert suite[1][0].tag == "skipped", suite[1].attrib["classname"]
-    assert suite[2].attrib["classname"] == "plot_future_imports_broken"
-    assert suite[2][0].tag == "failure", suite[2].attrib["classname"]
-    assert suite[2][0].attrib["message"] == "Passed even though it was marked to fail"
+    skips_and_fails = [
+        {
+            "classname": "plot_failing_example",
+            "tag": "skipped",
+            "message": None,
+        },
+        {
+            "classname": "plot_failing_example_thumbnail",
+            "tag": "skipped",
+            "message": None,
+        },
+        {
+            "classname": "plot_numpy_matplotlib",
+            "tag": "failure",
+            "message": "RuntimeError: Forcing",
+        },
+        {
+            "classname": "plot_scraper_broken",
+            "tag": "skipped",
+            "message": None,
+        },
+        {
+            "classname": "plot_future_imports_broken",
+            "tag": "failure",
+            "message": "Passed even though it was marked to fail",
+        },
+    ]
+    assert len(suite) == len(skips_and_fails)
+    for this_suite, this_example in zip(suite, skips_and_fails):
+        assert this_suite.attrib["classname"] == this_example["classname"]
+        assert this_suite[0].tag == this_example["tag"], this_suite.attrib["classname"]
+        if this_example["message"] is not None:
+            assert this_suite[0].attrib["message"].startswith(this_example["message"])


Needed to add new items for the 2 new examples. Thought perhaps this was easier to read.

tsbinns · 2024-07-16T13:03:41Z

That should now be all the tests working.

larsoner · 2024-07-16T14:53:32Z

Looks good marking for merge-when-green, thanks in advance @tsbinns !

… to version 0.17.0 v0.17.0 ------- Support for Python 3.8 and Sphinx 4 dropped in this release. Requirement is now Python >= 3.9 and Sphinx >= 5. **Implemented enhancements:** - Introduction tooltip corresponds to the first paragraph `#1344 <https://github.com/sphinx-gallery/sphinx-gallery/pull/1344>`__ (`fgmacedo <https://github.com/fgmacedo>`__) - FIX Jupyterlite in CircleCI artifact `#1336 <https://github.com/sphinx-gallery/sphinx-gallery/pull/1336>`__ (`lesteve <https://github.com/lesteve>`__) - MNT: Rename README.rst to GALLERY_HEADER.rst `#1321 <https://github.com/sphinx-gallery/sphinx-gallery/pull/1321>`__ (`timhoffm <https://github.com/timhoffm>`__) - [ENH] Add custom thumbnails for failing examples `#1313 <https://github.com/sphinx-gallery/sphinx-gallery/pull/1313>`__ (`tsbinns <https://github.com/tsbinns>`__) - ENH integrate download/launcher links into ``pydata-sphinx-theme`` secondary sidebar `#1312 <https://github.com/sphinx-gallery/sphinx-gallery/pull/1312>`__ (`Charlie-XIAO <https://github.com/Charlie-XIAO>`__) - add option for zip downloads `#1299 <https://github.com/sphinx-gallery/sphinx-gallery/pull/1299>`__ (`jamiecook <https://github.com/jamiecook>`__) - Allow setting animation format from gallery config `#1243 <https://github.com/sphinx-gallery/sphinx-gallery/pull/1243>`__ (`QuLogic <https://github.com/QuLogic>`__) (NEWS truncated at 15 lines)

Add custom thumbnails for failing examples

fc1d423

Add failing_thumbnail conf_file variable

f066eb5

tsbinns added 2 commits May 29, 2024 19:37

Update example name

e97d574

Add user guide entry

069ffa6

Merge branch 'master' into add_thumbnails_broken_examples

01f6501

larsoner added the enhancement label May 30, 2024

larsoner approved these changes May 30, 2024

View reviewed changes

larsoner enabled auto-merge (squash) May 30, 2024 13:31

larsoner requested changes May 30, 2024

View reviewed changes

larsoner disabled auto-merge May 30, 2024 13:32

lucyleeow reviewed May 31, 2024

View reviewed changes

doc/configuration.rst Outdated Show resolved Hide resolved

Update user guide from review

a003b57

Co-authored-by: Lucy Liu <[email protected]>

Merge branch 'master' into add_thumbnails_broken_examples

804df96

lucyleeow mentioned this pull request Jul 10, 2024

ENH: Parallel gallery generation #877

Merged

lucyleeow and others added 2 commits July 12, 2024 13:11

merge main

0918ad5

[pre-commit.ci] auto fixes from pre-commit.com hooks

5a04568

for more information, see https://pre-commit.ci

tsbinns added 2 commits July 13, 2024 19:52

Add failing example thumbnail test

8c73700

Merge branch 'master' into add_thumbnails_broken_examples

7c33f22

tsbinns commented Jul 13, 2024

View reviewed changes

sphinx_gallery/tests/test_full.py Show resolved Hide resolved

tsbinns commented Jul 13, 2024

View reviewed changes

sphinx_gallery/tests/test_full.py Outdated Show resolved Hide resolved

tsbinns commented Jul 13, 2024

View reviewed changes

tsbinns added 4 commits July 15, 2024 12:28

Fix test_junit

49a9ef1

Merge branch 'master' into add_thumbnails_broken_examples

c6c8ede

Fix remaining failing tests

507f9d6

Merge branch 'add_thumbnails_broken_examples' of https://github.com/t…

7d3d2a4

…sbinns/sphinx-gallery into add_thumbnails_broken_examples

tsbinns commented Jul 16, 2024

View reviewed changes

Update sphinx_gallery/tests/test_full.py

9039dab

larsoner approved these changes Jul 16, 2024

View reviewed changes

larsoner enabled auto-merge (squash) July 16, 2024 14:53

larsoner merged commit d2c8e47 into sphinx-gallery:master Jul 16, 2024
18 checks passed

tsbinns deleted the add_thumbnails_broken_examples branch July 16, 2024 15:00

	if src_file in gallery_conf["failing_examples"]:
	ignore_thumbnail = True

	if src_file in gallery_conf["failing_examples"] and ignore_thumbnail:
	img = os.path.join(glr_path_static(), "broken_example.png")

	elif os.path.exists(thumbnail_image_path):
	img = thumbnail_image_path

	"""
	Example that fails to execute (with a specified thumbnail)
	==========================================================

	By default, examples with code blocks that raise an error will have the broken
	image stamp as their gallery thumbnail. However, this may not be desired, e.g.
	if only part of the example is expected to fail and it should not look like the
	entire example fails.

	In these cases, the `sphinx_gallery_thumbnail_path` variable can be set to
	specify a desired thumbnail.
	"""

	# Code source: Thomas S. Binns
	# License: BSD 3 clause
	# sphinx_gallery_line_numbers = True

	# sphinx_gallery_thumbnail_path = 'auto_examples/no_output/images/sphx_glr_plot_raise_001.png' # noqa

	import numpy as np
	import matplotlib.pyplot as plt

	plt.pcolormesh(np.random.randn(100, 100))

	# %%
	# This block will raise an AssertionError

	assert False

	# TODO: test that broken thumbnail does appear when needed
	# TODO: test that examples are executed after a no-plot and produce
	# the correct image in the thumbnail

		@@ -1239,6 +1239,5 @@ def test_newlines(log_collector_wrap):
		assert tee.newlines == tee.output.newlines


		# TODO: test that broken thumbnail does appear when needed

[ENH] Add custom thumbnails for failing examples #1313

[ENH] Add custom thumbnails for failing examples #1313

Uh oh!

Conversation

tsbinns commented May 24, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Problem

Possible Solution

Unit tests

Uh oh!

larsoner commented May 24, 2024

Uh oh!

tsbinns commented May 24, 2024

Uh oh!

larsoner commented May 24, 2024

Uh oh!

tsbinns commented May 27, 2024

Uh oh!

larsoner commented May 28, 2024

Uh oh!

tsbinns commented May 29, 2024

Uh oh!

larsoner left a comment

Choose a reason for hiding this comment

Uh oh!

larsoner left a comment

Choose a reason for hiding this comment

Uh oh!

lucyleeow left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

larsoner commented Jun 18, 2024

Uh oh!

tsbinns commented Jun 18, 2024

Uh oh!

lucyleeow commented Jul 12, 2024

Uh oh!

tsbinns commented Jul 13, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

tsbinns Jul 13, 2024

Choose a reason for hiding this comment

Uh oh!

tsbinns Jul 16, 2024

Choose a reason for hiding this comment

Uh oh!

tsbinns commented Jul 16, 2024

Uh oh!

larsoner commented Jul 16, 2024

Uh oh!

Uh oh!

Uh oh!

tsbinns commented May 24, 2024 •

edited

Loading

tsbinns commented Jul 13, 2024 •

edited

Loading