I had a conversation with @lesteve and @choldgraf at scipy about moving this functionality into sphinx-gallery and they werereceptive to moving this upstream.

I think we should hold off merging this if there is a chance of the flexibility landing in sphinx-gallery in the near term.

I'm not too optimistic on what "near term" really means, but sure, if sphinx-gallery provided this functionality by itself it would of course be advantageous. If anything, this PR can also be seen as a kind of requirement statement for what would need to be implemented by sphinx-gallery.

@ImportanceOfBeingErnest could you explain a bit more about not being able to produce valid Binder links for matplotlib? Binder expects the notebooks to be in the repository, but code could be from anywhere else (e.g. you can put arbitrary shell commands like git clone or pip install -e into a postBuild file if you need to pull code from another online repo)

re: sphinx-gallery, @tacaswell did you open an issue in the repo about what kind of use-case sphinx-gallery would need to meet? I am a bit doubtful they'd be open to pyodide as a first-class citizen, only because that project looks to be in a super-alpha state and seems redundant with Binder functionality. However it might make sense to let people customize their own links at the bottom of examples, provided that they don't add to much complexity to sphinx gallery.

However it might make sense to let people customize their own links at the bottom of examples, provided that they don't add to much complexity to sphinx gallery.

That was my understanding of what @tacaswell wanted to do. For including that in sphinx-gallery: I would agree that keeping additional complexity as low as possible is key. Adding a test for the feature in sphinx-gallery will be needed too.

@choldgraf Matplotlib example notebooks are hosted on https://matplotlib.org which is not a github repo. However, "binder" seems to require a github repo to get the notebooks from. This looks like an unecessary constraint to me. But does that mean that we cannot use binder now?
As alternative, the current development version is hosted in https://github.com/matplotlib/devdocs/ and the notebooks are in tree/gh-pages/_downloads, so one could decide to provide links to this version (although they might be out of sync from time to time) but using those parameters in the binder dictionary resulted in the notebook correctly loaded, but not execute correctly.

What sphinx-gallery should support is the ability to add custom rst to the examples (probably at various points in the document). For the case discussed here, it would be enough to be able to provide some function, probably through the sphinx_gallery_conf dictionary like def customrst(fname), which returns restructured text that is added at some point in the document (in this case the footer).
More generally, one would also probably like to have this mechanism for the complete document, e.g. to be able to provide a function def customrst(fname, parsedcode, parsedtext) in order to be able to add other stuff to the examples as well.

I would agree that sphinx-gallery should not (need to) support directly links to any specific service (like webassembly, pyodide in this case); this needs to be in the responsibility of the user.

@ImportanceOfBeingErnest some thoughts below

Matplotlib example notebooks are hosted on https://matplotlib.org which is not a github repo. However, "binder" seems to require a github repo to get the notebooks from.

Binder does require code to be hosted in a git repo of some sort, though I believe Matplotlib's website code does exist in a GitHub repository, no? Many sites use gh-pages or readthedocs to host their website online. The final page is a "regular" website but it's backed by a regular repository.

For example, the sphinx-gallery website is hosted at a repo "sphinx-gallery.github.io". The configuration for the sphinx-gallery binder links is in the sphinx-gallery repo here:

https://github.com/sphinx-gallery/sphinx-gallery/blob/master/doc/conf.py#L344

relevant sections:

    'binder': {'org': 'sphinx-gallery',
               'repo': 'sphinx-gallery.github.io',  # Whatever repo you know will serve the website
               'url': 'https://mybinder.org',  # The BinderHub URL to use as a base
               'branch': 'master',
               'dependencies': './binder/requirements.txt',
               'notebooks_dir': 'notebooks',  # Jupyter notebooks will be copied here and binder links generated to point to them
               }

so a similar pattern should work for devdocs, e.g. something like:

    'binder': {'org': 'matplotlib',
               'repo': 'devdocs', 
               'url': 'https://mybinder.org',  
               'branch': 'master',
               'dependencies': '../rel/path/to/requirements'
               'notebooks_dir': 'notebooks',  
               }

re: the sphinx-gallery stuff, I'd recommend opening up something in the sphinx-gallery repo so the community can discuss. Supporting something like footer would be useful I think, again depending on the amount of complexity it introduces.

So the webpage is hosted on https://github.com/matplotlib/matplotlib.github.com.
The notebooks are available in the _download folder:
https://github.com/matplotlib/matplotlib.github.com/tree/master/_downloads

I hence used this configuration

    'binder': {'org': 'matplotlib', 
           'repo': 'github.com/matplotlib/matplotlib.github.com',
           'url': 'https://mybinder.org', 
           'branch': 'master',
           'notebooks_dir': '_downloads',
           'dependencies': 'sphinxext/binder_requirement/requirements.txt',
           'use_jupyter_lab': True,

This will produce a link like

https://mybinder.org/v2/gh/matplotlib/github.com/matplotlib/matplotlib.github.com/master?urlpath=lab/tree/_downloads/gallery/pyplots/align_ylabels.ipynb

which is not valid.

I found that the following link actually lets the notebook start

https://mybinder.org/v2/gh/matplotlib/matplotlib.github.com/master?filepath=_downloads%2Falign_labels_demo.ipynb

however, once the notebook appears, it says that the kernel cannot be started or has died. There also seems to be no way to generate this link through the 'binder' dictionary in sphinx-gallery - so it might as well be wrong.

A couple quick thoughts:

• The notebooks_dir keyword actually overrides where sphinx puts the notebooks. It allows them to be copied in the same hierarchy that they have in the examples/ folder, whereas sphinx puts things in _downloads in a totally flat fashion. I wonder if this is why the links are breaking. Can you try adding a different directory? (sphinx gallery will generate it on the fly)
• At https://github.com/matplotlib/matplotlib.github.com I don't see any of the binder configuration files in there. Sphinx-gallery should generate a folder called binder/ that contains the files you specify in dependencies keyword. This folder needs to be in the root of the repository that's serving the documentation so that Binder can find it.

(if you share a URL to your fork where you're doing this work, I can try to get the config working. I'll open up a PR in sphinx-gallery to clarify some of the points you bring up as well, thanks for taking the time to mention where the confusion is)

I see. So there is just a big conceptional gap between how things are done in matplotlib and how binder assumes things to be. Matplotlib docs are not genereated within the matplotlib/matplotlib github repo (this one here, where we are talking) but they are generated "somewhere" and put to a different repo, matplotlib/matplotlib.github.com. Including links would hence mean that this later step of copying is performed before any link can potentially work. And the copying step would need to include copying of the notebooks that would be generated in a (then different) structure, as well as the binder configuration. Since I do not have access to the complete workflow and it cannot be tested, I have to stop here I think.

Leaving this as it is, because it will still work for adding links to pyodide - in case that is desired.

The documentation build and deploy process is documented at https://matplotlib.org/devel/release_guide.html#build-and-deploy-documentation

https://github.com/matplotlib/matplotlib.github.com is what hosts the live documentation, I would prefer we not debug this process against that. From a serving point of view not sure what would have to be done to get github to serve from a fork of matplotlib.github.com, but it should be easy to get binder to work off of a fork.

Given the two-layer deployment of the docs (into the top level and into the version specific folders) would it make sense to push the notebooks to a new repo (maybe in a new org matplotlib-binder with matplotlib-vX.X.X repos) that will be a stable public API over time?

Not sure if it makes sense to host the notebook at yet another place. I mean all notebooks are there, namely on https://matplotlib.org/_downloads/. I find it more realistic that binder might at some point change its interface to allow for simple links like
http://binderserver.xyz/?link=https://server1.xyz/link/to/notebook.ipynb&req=https:/server2.abc/link/to/requirements.txt

I'll close this as it was just a POC...