Description

Earlier this month, @lucyleeow and I met over a call to discuss the topic of shared functionality that resides in both Sphinx-Gallery and jupyterlite-sphinx, through the former's JupyterLite integration. The JupyterLite integration was established in #977 and #1312, intended for use downstream in scikit-learn/scikit-learn#25887 and other contemporary projects such as scikit-image (scikit-image/scikit-image#6974, scikit-image/scikit-image#7644).

Based on our discussion and through this issue, I would like to propose moving the notebook_modification_function, and other JupyterLite-related functionality to jupyterlite-sphinx—which is already an optional dependency of Sphinx-Gallery—so that Sphinx-Gallery can use it more to provide extra features on top of it that are more relevant to its use case. In contrast jupyterlite-sphinx can continue to provide the building blocks for integrating JupyterLite with Sphinx-based documentation websites.

My rationale behind this is based on the fact that we currently provide some basic functionality ourselves to customise the mini-notebooks of sorts for docstring-based examples, provided by the TryExamples directive (https://jupyterlite-sphinx.readthedocs.io/en/stable/directives/try_examples.html#global-configuration) – we have a try_examples_global_warning_text configuration option that helps add a “"JupyterLite warning” or other custom Markdown messages at the top. However, it is limited in scope, as it only adds a warning; however, we could add a more generic function that Sphinx-Gallery can get from jupyterlite-sphinx, and then jupyterlite-sphinx can override that per directive to customise the contents of the notebooks.

This would also resolve the problem of circular dependencies among the API, as jupyterlite-sphinx wouldn’t need to import anything from Sphinx-Gallery the other way around.

Plan of action

This is a summarised plan of action, from the points above:

• First, move the notebook_modification_function() API away from Sphinx-Gallery to a private API under jupyterlite-sphinx
• Expose this API through jupyterlite-sphinx in the form of jupyterlite_sphinx.notebook_modification_function (or similar)

Then, we have the following development plans:

on the jupyterlite-sphinx side

• Override the jupyterlite_sphinx.notebook_modification_function API based on the type of content
  • For the docstring-based examples, provide a try_examples_notebook_modification_function configuration option, and
  • For long-form notebooks through the NotebookLite and JupyterLite directives, provide notebooklite_notebook_modification_function and jupyterlite_notebook_modification_function configuration options
• Collect the string from the currently existing try_examples_global_warning_text if it has been added by a user, and prepend it to the changes going to be performed by the try_examples_notebook_modification_function

on the Sphinx-Gallery side

• Sphinx-Gallery can remove its code when integrated in jupyterlite-sphinx and added in a release, and use said API from jupyterlite-sphinx to provide its current notebook_modification_function option, so that there are no breaking changes for users. Or, we could even tinker with some of these things, as the JupyterLite integration is marked as experimental.
• everything else should remain the same, more or less.

(I don't know how best to avoid breaking changes in this scenario)

The reason for prepending _try_examples_, notebooklite_, and jupyterlite_ before _notebook_modification_function is also because we don't want to capture the namespace that has already been provided to the users by Sphinx-Gallery.

What this will help with

• Less code for Sphinx-Gallery to maintain, and
• Less redundancy in jupyterlite-sphinx's code, and expanded configuration options for users for the features provided by it that are not possible to do without generic-ness

For example, try_examples_notebook_modification_function being generic would make it possible to add virtually any form of content before the TryExamples docstrings, ranging from a %pip install <...> command to ensure that the version of the project is up-to-date (similar to sympy/live#26 + a WASM build + docs workflow), or to various styles of Markdown content, such as tables, etc., or even raw HTML or images if they want.

If done right, Sphinx-Gallery retains its JupyterLite functionality as is, with as minimal breaking changes as possible.

Happy to hear everyone's thoughts! Please let me know if anything I wrote among these seems unclear.

cc: @lesteve @lucyleeow @steppi

Additional links

Please read through these meeting notes for additional context: https://hackmd.io/@fjkQkmoESkOT_lCcSYoGnQ/SJtr0rQwyg (view/comment access only for now, please let me know if you want to have write access!)