Change pyapi links to use a pyapi: prefix

Kobzol · Kobzol · commit b4ae949d6834 · 2025-07-03T17:04:13.000+02:00
diff --git a/docs/python/client.md b/docs/python/client.md
@@ -1,6 +1,6 @@
 # Client
 To submit [jobs](../jobs/jobs.md) using the Python API, you first need to create
-a [`Client`](hyperqueue.client.Client) that connects to a running HyperQueue cluster. You have two
+a [`Client`](pyapi:hyperqueue.client.Client) that connects to a running HyperQueue cluster. You have two
 options of deploying the cluster. Once you have an instance of a `Client`, you can use it to
 [submit](submit.md) a job.
 
@@ -21,14 +21,14 @@ the node that executes the Python code, you can simply create an instance of a `
 passing any parameters.
 
 ## Using a local cluster
-You can use the [`LocalCluster`](hyperqueue.cluster.LocalCluster)
+You can use the [`LocalCluster`](pyapi:hyperqueue.cluster.LocalCluster)
 class to spawn a HyperQueue server and a set of workers directly on your local machine.
 This functionality is primarily intended for local prototyping and debugging, but it can also be
 used for actual computations for simple use-cases that do not require a distributed deployment of
 HyperQueue.
 
 When you create the cluster, it will initially only start the HyperQueue server. To connect workers
-to it, use the [`start_worker`](hyperqueue.cluster.LocalCluster#f_start_worker) method.
+to it, use the [`start_worker`](pyapi:hyperqueue.cluster.LocalCluster#f_start_worker) method.
 
 ```python
 from hyperqueue import LocalCluster
diff --git a/docs/python/dependencies.md b/docs/python/dependencies.md
@@ -13,7 +13,7 @@ workflows.
 
 ## Defining dependencies
 To define a dependency between tasks, you will first need to store the
-[`Task`](hyperqueue.task.task.Task) instances that you get when you create a [task](submit.md#tasks).
+[`Task`](pyapi:hyperqueue.task.task.Task) instances that you get when you create a [task](submit.md#tasks).
 You can then use the `deps` parameter when creating a new task and pass an existing task instance
 to define a dependency:
 
diff --git a/docs/python/submit.md b/docs/python/submit.md
@@ -1,12 +1,12 @@
 # Submitting jobs
 You can use the Python API to submit [jobs](../jobs/jobs.md) (directed acyclic graphs of tasks)
-through a [`Client`](hyperqueue.client.Client). In addition to the functionality offered by the
+through a [`Client`](pyapi:hyperqueue.client.Client). In addition to the functionality offered by the
 HyperQueue CLI, you can use the Python API to add [dependencies](dependencies.md) between jobs,
 configure each task [individually](#parametrizing-tasks) and create tasks out of
 [Python functions](#python-functions).
 
 ## Job
-To build a job, you first have to create an instance of the [`Job`](hyperqueue.job.Job) class.
+To build a job, you first have to create an instance of the [`Job`](pyapi:hyperqueue.job.Job) class.
 
 ```python
 from hyperqueue import Job
@@ -23,19 +23,19 @@ To create complex workflows, you can also specify [dependencies](dependencies.md
 
 ### External programs
 To create a task that will execute an external program, you can use the
-[`program`](hyperqueue.job.Job#f_program) method of a `Job`:
+[`program`](pyapi:hyperqueue.job.Job#f_program) method of a `Job`:
 
 ```python
 job.program(["/bin/my-program", "foo", "bar", "--arg", "42"])
 ```
 
 You can pass the program arguments or various other [parameters](#parametrizing-tasks) to the task.
-The `program` method will return a [`Task`](hyperqueue.task.task.Task) object that represents the
+The `program` method will return a [`Task`](pyapi:hyperqueue.task.task.Task) object that represents the
 created task. This object can be used further e.g. for defining [dependencies](dependencies.md).
 
 ### Python functions
 If you want to execute a Python function as a task, you can use the
-[`function`](hyperqueue.job.Job#f_function) method of a `Job`:
+[`function`](pyapi:hyperqueue.job.Job#f_function) method of a `Job`:
 
 ```python
 def preprocess_data(fast, path):
@@ -57,7 +57,7 @@ Python tasks can be useful to perform e.g. various data preprocessing and organi
 co-locate the logic of Python tasks together with the code that defines the submitted workflow (job),
 without the need to write an additional external script.
 
-Same as with the `program` method, `function` will return a [`Task`](hyperqueue.task.task.Task)
+Same as with the `program` method, `function` will return a [`Task`](pyapi:hyperqueue.task.task.Task)
 that can used to define [dependencies](dependencies.md).
 
 !!! note
@@ -71,7 +71,7 @@ a worker - this means that it needs to have access to the correct Python version
 that contains the `hyperqueue` package!
 
 To make sure that the function will be executed in the correct Python environment, you can use
-[`PythonEnv`](hyperqueue.task.function.PythonEnv) and its `prologue` argument. It lets you specify
+[`PythonEnv`](pyapi:hyperqueue.task.function.PythonEnv) and its `prologue` argument. It lets you specify
 a (shell) command that will be executed before the Python interpreter that executes your
 function is spawned.
 
@@ -96,20 +96,20 @@ output paths, environment variables or HyperQueue specific parameters like
 the CLI, where you can only use a single set of parameters for all tasks of a job, with the Python
 API you can specify these parameters individually for each task.
 
-You can find more details in the documentation of the [`program`](hyperqueue.job.Job#f_program) or
-[`function`](hyperqueue.job.Job#f_function) methods.
+You can find more details in the documentation of the [`program`](pyapi:hyperqueue.job.Job#f_program) or
+[`function`](pyapi:hyperqueue.job.Job#f_function) methods.
 
 ## Submitting a job
 Once you have added some tasks to the job, you can submit it using the `Client`'s
-[`submit`](hyperqueue.client.Client#f_submit) method:
+[`submit`](pyapi:hyperqueue.client.Client#f_submit) method:
 
 ```python
 client = Client()
 submitted = client.submit(job)
 ```
 
 To wait until the job has finished executing, use the
-[`wait_for_jobs`](hyperqueue.client.Client#f_wait_for_jobs) method:
+[`wait_for_jobs`](pyapi:hyperqueue.client.Client#f_wait_for_jobs) method:
 
 ```python
 client.wait_for_jobs([submitted])
diff --git a/scripts/docs/postprocess_nedoc.py b/scripts/docs/postprocess_nedoc.py
@@ -21,10 +21,11 @@
 import nedoc.config
 from nedoc import core
 
+# Regex to match PyAPI links like [Foo](pyapi:hyperqueue.cluster.LocalCluster)
 # The second group is used to avoid adding the hash fragment to the link lookup key
-LINK_REGEX = re.compile(r"\[`.*?`]\(((?!#)[^#]*?)(#.*?)?\)")
+PYAPI_LINK_REGEX = re.compile(r"\[([^\]]+)\]\(pyapi:([^#]*?)(#.*?)?\)")
 
-PYTHON_API_DIR = "python/apidoc"
+PYTHON_API_DIR = "python/apidoc/"
 
 site_url = None
 url_map = {}
@@ -81,25 +82,21 @@ def on_files(files, config, **kwargs):
         logging.warning(f"WARNING: {map_file} file is missing")
 
 
-def on_page_markdown(src: str, page, config, *args, **kwargs):
-    global api_links
-
-    # TODO: use Markdown parser
-    lines = []
-    for line_index, line in enumerate(src.splitlines(keepends=False)):
-        # Iterate from the end to make replacing substrings easier
-        for match in reversed(list(LINK_REGEX.finditer(line))):
-            link_key = match.group(1)
-            link_url = url_map.get(link_key)
-            if link_url:
-                api_links += 1
-                url = f"{site_url}/{link_url}"
-                start, end = match.span(1)
-                line = line[:start] + url + line[end:]
-            else:
-                raise Exception(f"Link key {link_key} not found in {page.file.src_path} on line {line_index}")
-        lines.append(line)
-    return "\n".join(lines)
+def on_page_markdown(markdown: str, page, config, *args, **kwargs):
+    def replace_cli_link(match):
+        global api_links
+        link_text = match.group(1)
+        pyapi_key = match.group(2)
+        hash = match.group(3) or ""
+        link_url = url_map.get(pyapi_key)
+        if link_url:
+            api_links += 1
+            pyapi_url = f"{site_url}{link_url}"
+            return f"[{link_text}]({pyapi_url}{hash})"
+        else:
+            raise Exception(f"PyAPI link key {pyapi_key} not found in {page.file.src_path}")
+
+    return PYAPI_LINK_REGEX.sub(replace_cli_link, markdown)
 
 
 def on_post_build(config, **kwargs):
