Add server capabilities.

• New exported module Text.Pandoc.Server [API change].
• The pandoc executable now starts up a web server when renamed or symlinked as pandoc-server, and functions as a CGI program when renamed or symlinked as pandoc-server.cgi. See the man page for pandoc-server for full documentation.

Text.Pandoc.App.Opts: Redo FromJSON for Opt so that optional values can be omitted (in which case the values from defaultOptions are used).

Org reader: treat “abstract” block as metadata (Albert Krewinkel, #8204). A block of type “abstract” is assumed to define the document’s abstract. It is transferred from the main text to the metadata.

Org template: add abstract from metadata as block of type “abstract” (#8204).

HTML writer: use flex property for column widths (Albert Krewinkel, #8232).

LaTeX writer:

• Add label to tables that have an identifier (Albert Krewinkel, #8219). Tables with an identifier are marked with a \label. A caption is always included in this case, even if the caption is empty.
• Use \textquotesingle for straight quotes in text.
• Fix widths of multicolumn cells (#8218).

LaTeX template: fix behavior of colorlinks variable (Albert Krewinkel, #8226). Fixes a regression in 2.19 that required the boxlinks variable to be set in addition to the usual link coloring variables. Otherwise links were never colored in LaTeX PDF output.

Text.Pandoc.Highlighting: Export lookupHighlightingStyle [API change]. Previously this lived in an unexported module Text.Pandoc.App.CommandLineOptions, under the name lookupHighlightStyle.

Text.Pandoc.App:

• Remove unneeded MonadIO constraints in readSources.
• Factor out convertWithOpts' from convertWithOpts. This runs in any PandocMonad, MonadIO, MonadMask instance. So far it is not exported, but it might find a use later.

Support --strip-comments in commonmark/gfm (#8222). This change makes the commonmark reader sensitive to readerStripComments.

Lua: add function pandoc.utils.citeproc (Albert Krewinkel). The function runs the citeproc processor on a Pandoc document. Exposing this functionality to Lua allows to make citation processing part of a filter or writer, simplifies the creation of multiple bibliographies, and enables the use of varying citation styles in different parts of a document.

Refactor linux/make_artifacts.sh.

Update INSTALL.md installation from source instructions.

Use base64 package instead of base64-bytestring. It is supposed to be faster and more standards-compliant.

trypandoc improvements:

• Add dropdown with canned examples.
• Add citeproc support.
• Support csv, bibliographic and binary formats.
• Add load from file.
• Add permalink. Don’t always reload page.
• Use vanilla JS and CSS + the new pandoc-server.cgi.

Allow haddock-library-1.11.0.

Convert tool/extract-changes.hs to a Lua filter.

Add --embed-resources flag (Elliot Bobrow, #7331). This can be used to embed resources without implying --standalone. Deprecate --self-contained in favor of --embed-resources --standalone.

Allow environment variable interpolation in highlight-style and pdf-engine fields in defaults files (#8061; Jaehwang Jung, #8073).

Allow placing custom readers and writers in user data directory (Albert Krewinkel, #8112) (readers and writers subdirectories).

Add tsv (tab separated values) as an input format (#7974). [API change]: Text.Pandoc.Readers.CSV now exports readTSV. Internal change: In Text.Pandoc.CSV, CSVOptions has changed so that csvQuote takes a Maybe value.

Add tex_math_dollars to gfm default extensions (reflecting gfm’s new support for math).

RST, Org, Markdown readers: support rowspans and colspans in grid tables (#8202, Albert Krewinkel). Note: the writers does not yet support these more complex grid table features, so these complex grid tables will not round-trip.

HTML, LaTeX, and MediaWiki readers: use formatCode (#8162, #8129, Elliot Bobrow). This moves formatting from inside inline code elements to the outside, since pandoc’s Code element only takes string content.

Markdown reader:

• Don’t parse inline notes with blank lines inside (#8028).
• Allow attributes in special spans (e.g. smallcaps, underline) (Albert krewinkel, #4102). These spans are parsed as SmallCaps or Underline elements, but any attributes are included in a wrapping Span.

HTML reader:

• Allow sublists that are not marked as items (Albert Krewinkel, #8150). This is technically invalid HTML, but it can be found in the wild and browsers handle it.

Org reader (Albert Krewinkel):

• Recognize absolute paths on Windows (Albert Krewinkel, #8201).
• Recognize {webp,jxl} files as images (YI).
• Allow attrs for Org tables (Albert Krewinkel, #8049). Tables with attributes are no longer wrapped in Div elements; attributes are added directly to the table element.
• Support line selection in INCLUDE directives (Brian Leung, #8060).
• Fix Post / Pre mixup when setting emphasis chars (Amir Dekel, #8134).

LaTeX reader:

• Support \includesvg (#8027).
• Unescape characters in \lstinline inside \passthrough (#8179).
• Improve mathEnvWith (#8122). When converting e.g. an align environment to an aligned environment inside a Math element, we need to include a newline before the \end{aligned}, since the previous line might end in a comment.
• Fix treatment of extensions for \input in LaTeX reader (#8092). Previously we required a .tex extension, but TeX allows any extension for \input (as opposed to \include).

RTF reader:

• support \nosupersub (#8170).

TikiWiki reader:

• Support underlined text

DocBook reader:

• Improved reading <xref> elements (Frerich Raabe, #8065).

JATS reader:

• Strip ref- prefix from ref id in xref (#8007).
• Support edition in references (#8087).

RIS reader:

• Make parser more forgiving (#8034). Allow blank lines after entries. Allow entries with no space after the -, provided they just have a newline, e.g. DB -\n.
• Get right order of names (#8055).

MediaWiki reader:

• Allow HTML comment after row start (#8110).

DokuWiki reader:

• The tex_math_dollars extension is now supported for dokuwiki (but off by default) (#8178).
• Content inside <latex>...</latex> is parsed as raw LaTeX inline, and inside <LATEX>..</LATEX> as raw LaTeX block (#8178).
• The behavior of <php>...</php> is changed, so that instead of producing a code block, it produces raw HTML with <?php ... ?>.

LaTeX writer:

• Improve grouping with autocites (#8088).
• Extend list of book documentclasses (Wentau Han, #8053).
• Fix width of multicolumn cells (Albert Krewinkel, #8090). Cells spanning multiple columns must be given an explicit width, calculated from the table properties.
• Beamer: allow containsverbatim as alternative to fragile (#8080).

HTML writer:

• Add ‘footnotes’ identifier to footnotes section (#8043).
• Fix bug with --number-offset. This formerly caused section divs to be produced, even when --section-divs was not specified (#8097).
• Use CSS flexboxes for columns (Albert Krewinkel). This allows an arbitrary number of columns, while the previous approach assumed exactly two columns.
• Allow “spanlike” classes to be combined (see #8194). Previously classes like “underline” and “marked” had to be the first class in a span in order for the span to be interpreted as a “ul” or “mark” element. This commit allows these special classes to be “stacked,” e.g. [test]{.mark .underline}; in addition, the special classes are no longer required to come first in the list of classes.
• Avoid doubled style attribute when height and width are added to style because of an image, but the image already has a style attribute (#8047).
• Do not include the deprecated doc-endnote role (#8030). doc-endnote was deprecated in DPUB-ARIA 1.1.
• Remove extra soft break for tasklist (black-desk, #8142). Browser will display the extra newline character between checkbox and text as a space, which make tasklist items cannot be aligned.

EPUB writer:

• Allow choice of math method for v3 (#8164). Previously we always used MathML for math in EPUB3, because the spec includes MathML. But this is not widely supported by readers, so it seems better to allow users to choose their math method as they can with EPUB2 or HTML. NOTE: Existing workflows that produce EPUBv3 documents including math will be affected by this change. You must add --mathml to your command line if you want to continue producing MathML.

RST writer:

• Fix missing spaces with nested inlines (#8182).
• Always escape literal backslash (#8178).

Ms writer:

• Add comment in preamble stating generator.
• Fix roff ms syntax highlighting definitions (#8175, thanks to Branden Robinson).

ConTeXt writer:

• Support complex table structures (Albert Krewinkel, #8116). The following table feature are now supported in ConTeXt:

  • colspans,
  • rowspans,
  • multiple bodies,
  • row headers, and
  • multi-row table head and foot.

  The wrapping placetable environment is also given a reference option with the table identifier, enabling referencing of the table from within the document.

• Unify link handling (Albert Krewinkel, #8096). Autolinks, i.e. links with content that’s the same as the linked URL, are now marked with the \url command. All other links, both internal and external, are created with the \goto command, leading to shorter, slightly more idiomatic code. As before, autolinks can still be styled via \setupurl, other links via \setupinteraction.

• Use “sectionlevel” environment for headings (Albert Krewinkel, #5539). The document hierarchy is now conveyed using the \startsectionlevel/\stopsectionlevel by default. This makes it easy to include pandoc-generated snippets in documents at arbitrary levels. The more semantic environments “chapter”, “section”, “subsection”, etc. are used if the --top-level-division command line parameter is set to a non-default value.

Docx writer:

• Add w:lang to rPr for Span and Div with lang attribute, so that Word can know that “Apfel” is not a spelling error (#8026).
• Prevent crashing when handling invalid tables (Albert Krewinkel, #8102). Tables with different numbers of cells per row would sometimes crash pandoc. This fix prevents this by cutting off overlong rows.

ICML writer:

• Support custom-style attribute on Table (#8079).

AsciiDoc writer:

• Fix commas in link text (#8070). Commas in link text trigger interpretation of attributes. To block this, we replace them with numeric entities.
• Fix underline. We were rendering it as +++text+++; this is now changed to [.underline]#text#. See comment at #8070 (comment).

FB2 writer:

• Fix handling of non-section Divs (#8123).

Markdown writer:

• Disable soft wrapping when hard_line_breaks enabled (#8035). We were already doing this for markdown; this commit does the same thing for markua and commonmark and gfm.
• Avoid excessive indentation on bullet lists for commonmark, markua, gfm. They are now nested by 2 spaces instead of 4 (#8011).

Text.Pandoc.Class:

• Add new function findFileWithDataFallback [API Change] (Albert Krewinkel).
• fillMediaBag: Keep attributes of original image on Span (Albert Krewinkel, #8099). Images that cannot be fetched are replaced with a Span that contains the image’s description. The span now also retains all original image attributes and inherits all attributes of the image. Furthermore, the classes image and placeholder are added, and path and title are store in attributes original-image-src and original-image-title, respectively.

Text.Pandoc.Shared:

• makeSections: don’t make a section for a div with class “fragments” (#8098).
• Ensure that Nulls are ignored by makeSection and in segmenting slides (#8155).
• Add formatCode function to Text.Pandoc.Shared [API change] (Elliot Bobrow, #8129).
• taskListItemToAscii: handle asciidoctor’s characters (#8011). Asciidoctor uses different unicode characters for task lists; we should recognize them too and be able to convert them to ascii task lists in formats like gfm.
• Deprecate deLink and mark for later removal.

Text.Pandoc.Writers.Shared:

• toTableOfContents: Don’t replace links with empty spans in TOC (#8020).

Text.Pandoc.Readers.Metadata:

• Ensure that metadata values w/o trailing newlines are parsed as inlines, ...

New input formats: endnotexml (EndNote XML bibliography), ris (RIS bibliography).

A RIS bibliography file may now be used with --citeproc.

Citeproc: Allow a formatted bibliography to be placed in metadata fields via a Div with class refs (#7969, #526). Thus, one can include a metadata field, say refs, whose content is an empty div with id refs, and the formatted bibliography will be put into this metadata field. It may then be interpolated into a template using the variable refs.

Ensure that you don’t get PDF output to terminal. -t pdf now behaves like -t docx and gives an error unless the output is redirected.

--version now prints hslua version (#7929) and Lua version (#7997, Albert Krewinkel).

Change --metadata-file parsing so that, when the input format is not markdown or a markdown variant, pandoc’s markdown is used (#6832, #7926). When the input format is a markdown variant, the same format is used. Reason for the change: it doesn’t make sense to run the markdown parser with a set of extensions designed for a non-markdown format, and this dramatically limits what people can do in metadata files.

Trim whitespace from math in --webtex (#7892). This fixes problems with –webtex and markdown output, when display math starts or ends with a newline.

New exported module Text.Pandoc.Readers.EndNote, exporting readEndNoteXML, readEndNoteXMLCitation, and readEndNoteXMLReferences. [API change]

--self-contained: issue warning rather than failing with an error if a resource can’t be found (#7904).

New exported module, Text.Pandoc.Readers.RIS, exporting readRIS (#7894).

LaTeX reader:

• Handle subequations as inline math environment (#7883).
• Rudimentary support for vbox (#7939).
• Support \today (#7905).
• Handle \label and \ref for footnotes (#7930).
• Allow inline groups starting with \bgroup (#7953).
• Use custom TokStream that keeps track of whether macros are expanded. This allows us to improve performance a bit by avoiding unnecessary runs of the macro expansion code (e.g. from 24 ms to 20 ms on our standard benchmark).
• Further optimizations for inline parsing.
• Better handling of \usepackage. If the package is local but causes parse errors, parse everything up to the error and skip the rest. Issue a CouldNotParseIncludeFile warning indicating that parsing failed at that point.
• Text.Pandoc.Readers.LaTeX.Parsing: Monoid and Semigroup instances for TokStream.

HTML reader:

• Give warnings and emit empty note when parsing <a epub:type="noteref"> and the identifier doesn’t correspond to anything in the note table (#7884). Previously we just silently skipped these cases.
• Fix parsing of epub footnotes (#7884).

DocBook reader:

• Handle complete set of entities as specified at https://www.w3.org/2003/entities/2007doc/byalpha.html (#7938).
• Handle abstract in info section (#7747).
• Improve info parsing.
• Simplify metadata parsing code (#7747). Handle abstract as block-level content. Report skipped info elements with --verbose.
• Handle address and coyright in metadata (#7747).

DokuWiki reader:

• Add DokuWiki table alignment (#5202, damon-sava-stanley).

RST reader:

• Fix treatment of headerless simple tables (#7902).
• Wrap math in Span to preserve attributes (#7998, Albert Krewinkel). Math elements with a name, classes, or other fields are wrapped in a Span with these attributes.

JATS reader:

• Improve handling of fn-group elements (#6348, Albert Krewinkel). Footnotes in <fn-group> elements are collected and re-inserted into the document as proper footnotes in the place where they are referenced.
• Handle pub-date (#8000).
• Support PMID, DOI, issue in citations (#7995).
• Improve refs parsing. Handle issn and isbn; use simpler form for issued date.
• Strip ‘ref-’ from ref id in constructing CSL id. This allows better round-tripping, because the JATS writer adds the ref- prefix to the citation id to get the ref element’s id.

Org reader:

• Allow “:” in property drawer keys (Lucas V. R). Any non-space character is allowed as property drawer key, including “:” itself (so it is not really a delimiter). The real delimiter is a space character, so in a drawer like

  :PROPERTIES:
  ::k:ey:: value
  :END:
  

  “:k:ey:” is a key with value “value”.

• Allow comments above property drawer.

• More flexible LaTeX environments (Lucas V. R).

• Handle #+bibliography: as metadata so that it can work with --citeproc.

• Parse #+print_bibliography: as Div with id refs.

• Allow multiple #+bibliography:.

Markdown reader:

• Allow one-column pipe tables with pipe on right (#7919).
• Remove restriction on identifiers, so they no longer need to begin with a letter (#7920).

Docx reader:

• Enable citations extension for docx reader (#7840). When enabled, Zotero, Mendeley, and EndNote citations embedded in a docx are parsed as native pandoc citations. (When disabled, the generated citation text and bibliography are passed through as regular text.) The bibliography generated by the plugin is suppressed. Instead, bibliographic data embedded in citation items is added to the references metadata field so that it can be used with --citeproc.

Docbook writer:

• Interpret links without contents as cross-references (#7360, Jan Tojnar). Links without text contents are converted to <xref> elements. DocBook processors will generate appropriate cross-reference text when presented with an xref element.

Docx writer:

• Single numbering ID for examples (#7895, mjfs). This change ensures that example list items all belong to a single number sequence, so that if items are added or deleted in a word processor, the other items will renumber automatically.
• Add bookmark with table id to table (#7989, Nikolai Korobeinikov, #7285). This allows tables with ids to be linked to.

Ipynb writer:

• Handle metadata better (#7928). Previously we used the markdown writer to render metadata. This had some undesirable consequences (e.g. en dash expanded to -- when smart enabled), so now we use the plain writer.

LaTeX writer:

• Avoid extra space before \CSLRightInline (#7932).
• Add scrreport to chaptersClasses (#6168, ivardb).
• Support page,trim,clip attributes on images (#7181).
• Add () after booktabs rules (#8001). These commands take optional arguments with () and [], which can lead to problems if the content of the table cell begins with these characters.

RST writer:

• Support all standard metadata (“bibliographic”) fields.

HTML writer: performance improvements.

Org writer:

• Stop indenting property drawers, quote blocks (#3245, Albert Krewinkel). This follows the current default org-mode behavior.

Markdown writer:

• Move table-related code into submodule (Albert Krewinkel).
• Don’t produce redundant header identifier when the gfm_auto_identifiers extension is set (#7941).
• Update escaping rules for \. We now escape \ only if raw_tex is enabled or it is followed by a non-alphanumeric.

JATS writer:

• Encode author “others” as <etal/> (Albert Krewinkel). Citeproc adopted the BibTeX convention to use the author name “others” when there are additional authors that are not named. JATS uses the <etal> element for this.
• Avoid doubled ref-list element (#7990). Previously when generating JATS with the element_citations extension enabled, the references were put in a doubly-nested ref-list element (<ref-list><ref-list>...).
• Keep edition info in element citations (#7993, Albert Krewinkel).
• Fix handling of CSL variable ‘page’ (not ‘pages’ as we had before). It should go to ‘lpage’ and ‘rpage’, not ‘page-range’.

EPUB writer: refactor for clarity (#7991, Jonathan Dönszelmann, Ola Wolska, Ivar de Bruin, Jaap de Jong).

Custom writer (Albert Krewinkel):

• Support new-style Writer function (Albert Krewinkel). See the documentation for custom writers for details.
• Produce stacktrace if Writer function fails

Text.Pandoc.Logging: add CouldNotParseIncludeFile constructor for LogMessage [API change].

Text.Pandoc.Shared:

• Put id attributes on TOC entries (#7907, damon-sava-stanley). Naming scheme of id is “toc-” + id of linked to header/section. Effects HTML, Markdown, Powerpoint, and RTF.
• Define ordNub as alias for nubOrd from containers package (#7963, Albert Krewinkel).
• Export ensureValidXmlIdentifiers. This function changes identifiers that don’t start with letters, and internal links to these identifiers, making them compatible with XML standards. The change is simple: we add id_ to the front. There is potential for duplication if there are already id_... identifiers defined, but this seems rare enough not to worry too much about.

Ensure that valid XML identifiers are used in Docbook, EPUB, FB2, HTML4, S5, Slidy, Slideous, ICML, ODT, TEI writers. Thus, if you convert [anchor]{#1} and [link to](#1), id_1 will be used instead of 1 for the identifier.

Lua (Albert Krewinkel).

• Add module pandoc.layout to format and layout text.
• Move custom writer code into Lua hierarchy.
• Use pandoc-lua-marshal 0.1.5.
• Allow any type of callable object as argument to List functions filter, map, and find_if. These previously required the argument to be of...

Fix regression in 2.17.1 which caused problems finding default files in the default user data directory. (Reverts the item “logic bug in fullDefaultsPath”, which was misguided.)

Sample custom writer: use single quotes for strings (#7487, Albert Krewinkel).

Support pagedjs-cli as pdf engine (#7838, Albert Krewinkel). PagedJS is a polyfill and supports the Paged Media standards by the W3C. https://www.pagedjs.org/

CommonMark reader: fix source position after YAML metadata (#7863).

LaTeX reader:

• Remove retokenizing in rawLaTeXParser.

• Ensure that \raggedright doesn’t gobble an argument (#7757).

• Improve descItem. For some reason we were skipping arbitrary blocks before \item. This is now changed to “skip whitespace and comments.”

• Improve handling of \newif. Adding a pair of braces around the second argument of \def prevents LaTeX from an emergency stop on input like the following (#6096).

  \newif\ifepub
  \epubtrue
  \ifepub
  hi
  \fi
  

Docx reader: Parse both Zotero citation and bibliography as FieldInfo (#7840).

LaTeX writer:

• Allow arbitrary frameoptions to be passed to a beamer frame, using the frameoptions attribute (#7869).
• Add s and squeeze to recognized beamer frameoptions (#7869).

Markdown writer: handle explicit column widths with pipe tables (#7847). If a table has explicit column width information and the content extends beyond the --columns width, we need to adjust the widths of the pipe separators to encode this width information.

Docx writer: Separate tables even with RawBlocks between (#7224, Michael Hoffmann). Adjacent docx tables need to be separated by an empty paragraph. If there’s a RawBlock between tables which renders to nothing, be sure to still insert the empty paragraph so that they will not collapse together.

Man writer: use custom font V for inline code (#7506). The V font is defined conditionally, so that it renders like CB in output formats that support that, and like B in those that don’t (e.g. the terminal). Aliases also defined for VI, VB, VBI.

Asciidoc writer: Support checklists in asciidoctor writer (#7832, Nikolai Korobeinikov, ricnorr). The checklist syntax (similar to task_list in markdown) seems to be an asciidoctor-only addition.

HTML writer:

• Avoid duplicate “style” attributes on table cells (#7871).
• Don’t break lines inside code elements. With the new (default) line wrapping of HTML, in conjunction with the default CSS which includes code { whitespace: pre-wrap; }, spurious line breaks could be introduced into inline code (#7858).

Custom writer: preserve order of element attributes (#7489, Albert Krewinkel). Attribute key-value pairs are marshaled as AttributeList, i.e., as a userdata type that behaves both like a list and a map. This allows to preserve the order of key-value pairs.

Switch to hslua-2.1 (Albert Krewinkel). This allows for some code simplification and improves stability.

Don’t read files outside of user data directory (Even Brenden). If a file path does not exist relative to the working directory, and it does exist relative to the user data directory, but outside of of the user data directory, do not read it. This applies to readDataFile and readMetadataFile in PandocMonad and, by extension, any module that uses these by passing them relative paths.

Text.Pandoc.Class.makeCanonical: Correctly handle consecutive “..”s at the beginning of a path (Even Brenden). Prior to this commit, ../../file would evaluate to file, when it should be unchanged.

Search for metadata files in $DATADIR/metadata (#7851, Even Brenden). If files specified with --metadata-file are not found in the working directory, look in $DATADIR/metadata (#5876).

Text.Pandoc.Class: export readMetadataFile [API change] (#5876).

Text.Pandoc.Error: export new PandocCouldNotFindMetadataFileError constructor for PandocError [API change] (#5876).

Avoid putting a frame around speaker notes in beamer (#7857). If speaker notes (a Div with class ‘notes’) occur right after a section heading, but above slide level, the resulting \note{..} caommand should not be wrapped in a frame, as that will cause a spurious blank slide.

CSS in HTML template: adjust #TOC and h1 on mobile (#7835, Mauro Bieg).

Text.Pandoc.Readers.LaTeX.Parsing: don’t export totoks. Make the first param of tokenize a SourcePos instead of SourceName, and use it instead of totoks.

Text.Pandoc.Shared: Modify stringify so it ignores [Citation] inside Cite (#7855). Otherwise we’ll sometimes get two copies of things, one from the citationPrefix or citationSuffix and another from the embedded fallback text. When there is no fallback text, we’ll get no content. However, it really isn’t an alternative to just rely on the result of running query on the embedded Citations; this will result in a jumble of text rather than anything structured.

Omit --enable-doc in the cabal haddock invocation in tools/build-and-upload-api-docs.sh.

Text.Pandoc.App.Opt: fix logic bug in fullDefaultsPath. Previously we would (also) search the default user data directory for a defaults file, even if a different user data directory was specified using --data-dir. This was a mistake; if --data-dir is used, the default user data directory should not be searched.

Text.Pandoc.Shared: defaultUserDataDir behavior change (#7842). If the XDG data directory is not defined (e.g. because it’s not supported in the OS or HOME isn’t defined), we return the empty string instead of raising an exception.

Update command tests to distinguish stderr and test exit status.

MANUAL: add that speaker notes can be used with beamer (#7856).

Update build-and-upload-api-docs.sh.

Document --trace option. Document no-check-certificate in defaults files. Document ‘sandbox’ option for defaults files. (#7873).

Fix pattern syntax in sample readability custom reader.

doc/custom-readers.lua: add example for “readable HTML.”

Fix message in man page about where code can be found.

manfilter.lua: remove extra indent in table cells with code blocks.

Fix lua-filters documentation for table column widths (#7864).

epub.doc: Update links to KindleGen (#7846, Benson Muite, Mauro Bieg). KindleGen has been deprecated and we need to link to archived versions.

Use tables in defaults files documentation, so each default option is paired with the corresponding command-line option (Carsten Allefeld).

Use skylighting 0.12.2.

Add pandoc-lua-marshal to Nix shell (#7849, Even Brenden).

Require pandoc-lua-marshal 0.1.3.1 (#7831, Albert Krewinkel). Fixes a problem with List.includes and List.find that caused a Lua stackoverflow and subsequent program crash.

HTML template: load header-includes before math (#7833, Kolen Cheung). MathJax expect the config comes before loading the MathJax script. This change of order allows one to config MathJax via header-includes, which loads before the MathJax script. Cf. #2750.

When reading defaults file, stop at a line .... This line signals the end of a YAML document. This restores the behavior we got with HsYaml. yaml complains about content past this line. See #4627 (comment)

Text.Pandoc.Citeproc: allow notes-after-punctuation to work with numerical styles that use superscripts (e.g. american-medical-association.csl), as well as with note styles. The default setting of notes-after-punctuation is true for note styles and false otherwise. This restores a behavior of pandoc-citeproc that wasn’t properly carried over to Citeproc (#7826, cf. jgm/pandoc-citeproc#384).

Use commonmark-pandoc 0.2.1.2 (#7769).

Add FAQ on images in ipynb containers (#7749, Kolen Cheung).

Support markua as an output format (#1871, Tim Wisotzki and Saumel Lemmenmeier). Markua is a markdown variant used by Leanpub.

Add text wrapping for HTML output (#7764). Previously the HTML writer was exceptional in not being sensitive to the --wrap option. With this change --wrap now works for HTML. The default (as with other formats) is automatic wrapping. Note that the contents of script, textarea, and pre tags are always laid out with the flush combinator, so that unwanted spaces won’t be introduced if these occur in an indented context in a template.

Don’t read sources until in/out format are verified (#7797).

Issue error with --list-extensions for invalid formats (#7797).

Make --citeproc recognize .yml as well as .yaml extensions as YAML bibliography files (#7707, Jörn Krenzer).

Use latest version of KaTeX with --katex.

Fix parsing of footnotes in --metadata-file (#7813). Previously non-inline footnotes were not being parsed.

ODT reader:

• Parse list-header as a list item (Tuong Nguyen Manh).

Commonmark reader:

• Put sourcepos attribute on header, not enclosing div with -f commonmark+sourcepos (#7769).

Markdown reader:

• Don’t allow ^ at beginning of link or image label (#7723). This is reserved for footnotes. Fixes regression from 0a93acf.
• Fix parsing of “bare locators” after author-in-text citations. Previously @item [p. 12; @item2] was incorrectly parsed as three citations rather than two. This is now fixed by ensuring that prefix doesn’t gobble any semicolons.
• Revert changes to inlinesInBalancedBrackets (commit fa83246), which caused regressions.
• Improve detection of pipe table line widths (#7713). Fixed calculation of maximum column widths in pipe tables. It is now based on the length of the markdown line, rather than a “stringified” version of the parsed line. This should be more predictable for users. In addition, we take into account double-wide characters such as emojis.

Custom (Lua) readers:

• First argument is now a list of sources instead of the concatenated text (Albert Krewinkel). The list structure can easily be converted to a string by applying tostring, but it is also possible to access the elements (each with a text and name). A small example is added to the custom reader documentation, showcasing its use in a reader that creates a syntax-highlighted code block for each source code file passed as input. Existing readers will still work through a fallback mechanism, issuing a deprecation notice.

Org reader:

• Parse official org-cite citations (#7329). We also support the older org-ref style as a fallback. We no longer support the “markdown style” or “Berkeley style” citations.
• Support alphabetical (fancy) lists (Lucas Viana). When the fancy_lists extension is enabled, alphabetical list markers are allowed, mimicking the behaviour of Org Mode when org-list-allow-alphabetical is enabled.
• Support counter cookies in lists (Lucas Viana). Such cookies are used to override the item counter in ordered lists. In org it is possible to set the counter at any list item, but since Pandoc AST does not support this, we restrict the usage to setting an offset for the entire ordered list, by using the cookie in the first list item.
• Allow trailing spaces after key/value pairs in directives (Albert Krewinkel). Ensures that spaces at the end of attribute directives like #+ATTR_HTML: :width 100% (note the trailing spaces) are accepted.

LaTeX reader:

• Omit visible content for \label{...}. Previously we included the text of the label in square brackets, but this is undesirable in many cases. See discussion in #813 (comment).
• Improve references (#813). Resolve references to theorem environments. Remove the Span caused by “label” in figure, table, and theorem environments; this had an id that duplicated the environments’ id.
• Fix semantics of \ref. We were including the ams environment type in addition to the number. This is proper behavior for \cref but not for \ref. To support \cref we need to store the environment label separately.
• Add babel mappings for Guajati (gu) and Oriya (or) (#7815).
• Fix typo panjabi -> punjabi in babel mappings (#7814).

HTML reader:

• Parse attributes on links and images (#6970).

Docx reader:

• Handle multiple pic elements inside a drawing (#7786).
• Change elemToParPart to return [ParPart] instead of ParPart. Also remove NullParPartconstructor, as it is no longer needed. This will allow us to handle elements that contain multiple ParParts, e.g. w:drawing elements with multiple pic:pic.

DocBook reader:

• Collapse internal spaces in literal and other similar tags (#7821), as the standard docbook toolchain does.
• Be sensitive to spacing=“compact” in lists (#7799). When spacing="compact" is set, Para elements are turned into Plain, so we get a “tight” list.

Markdown writer:

• Add new exported function writeMarkua from Text.Pandoc.Writers.Markdown [API change] (#1871, Tim Wisotzki and Saumel Lemmenmeier).
• Fix indentation issue in footnotes (#7801).
• Avoid extra space before citation suffix if it already starts with a space.
• Ensure semicolon between the locator and the next citation when an author-in-text citation has a locator and following citations.
• Improve escaping for # (#7726).

Custom (Lua) writers:

• Allow variables to be set via second return value of Doc (#6731, Albert Krewinkel). New templates variables can be added by giving variable-value pairs as a second return value of the global function Doc. Example:

  function Doc (body, meta, vars)
    vars.date = vars.date or os.date '%B %e, %Y'
    return body, vars
  end
  

• Provide global PANDOC_WRITER_OPTIONS (#6731, Albert Krewinkel).

• Assign default Pandoc object to global PANDOC_DOCUMENT (Albert Krewinkel). The default Pandoc object is now non-strict, i.e., only the parts of the document that are accessed will be marshaled to Lua. A special type is no longer necessary. This change also makes it possible to use the global variable with library functions such as pandoc.utils.references, or to inspect the document contents with walk().

LaTeX writer:

• Fix typo panjabi -> punjabi in babel mappings (#7814).

MediaWiki writer:

• Remove redundant display text for wiki links (Jesse Hathaway).

Docx writer:

• Handle bullets correctly in lists by not reusing numIds (#7689, Michael Hoffmann). This fixes a bug in which a Div in a list item would receive bullets on its contained paragraphs.

Org writer:

• Fix list items starting with a code block or other non-paragraph content (#7810).
• Avoid blank lines after tight sublists (#7810).
• Fix extra blank line inserted after empty list item (#7810).
• Don’t add blank line before lists (#7810).
• Support starting number cookies (Lucas Viana). This is necessary for lists that start at a number other than 1.
• Support the new org-cite syntax (#7329).

Haddock writer:

• Avoid blank lines after tight sublists (#7810).

Ipynb writer:

• Ensure deterministic order of keys.
• Handle cell output with raw block of markdown (#7563, Kolen Cheung). Write RawBlock of markdown in code-cell output. This is designed to fit the behavior of #7561, which makes the ipynb reader parse code-cell output with mime “text/markdown” to a RawBlock of markdown. This commit makes the ipynb writer writes this RawBlock of markdown back inside a code-cell output with the same mime, preserving this information in round-trip.
• In choosing between multiple output options, always favor those marked with the output format over images (Kolen Cheung). Previously, both fmt == f case and Image have a rank of 1.

Ipynb reader & writer: properly handle cell “id” (#7728). This is passed through if it exists (in Nb4); otherwise the writer will add a random one so that all cells have an “id”.

Ms writer:

• Properly encode strings for PDF contents (#7731).

JATS writer:

• Keep quotes in element-citations (Albert Krewinkel). Fixed a bug that lead to quote characters being lost in element-citations.

RTF writer:

• Properly handle images in data URIs (#7771).

Commonmark writer:

• Allow ‘)’ delimiters on ordered lists.

RST writer:

• Avoid extra blank line after empty list item (#7810).

HTML writer:

• Make line breaks more consistent. With --wrap=none, we now output line breaks between block-level elements. Previously they were omitted entirely, so the whole document was on one line, unless there were literal line breaks in pre sections. This makes the HTML writer’s behavior more consistent with that of other writers. Also, regardless of wrap settings, put newline after <dd> and after block-level elements in the footnotes section. And add a line break between an img tag and the associated figcaption.
• reveal.js: Make sure images with r-stretch are not in p tags. They must be direct children of the section. There was previously code to make this work with the older class name stretch, but the name has changed in reveal.js.
• reveal.js: don’t add r-fit-text class to section. It must go on the header only.

AsciiDoc writer:

• Improve detection of intraword emphasis (#7803).

OpenDocument writer:

• Fix vertical alignment bug with...

Add interface for custom readers written in Lua (#7669). Users can now do -f myreader.lua and pandoc will treat the script myreader.lua as a custom reader, which parses an input string to a pandoc AST, using the pandoc module defined for Lua filters. A sample custom reader can be found in data/creole.lua. Also see documentation in doc/custom-readers.md.

New module Text.Pandoc.Readers.Custom, exporting readCustom [API change].

Allow plain to be used in raw attribute syntax.

Accept empty --metadata-file (#7675). This was a regression from 2.15 behavior.

Markdown reader: Improve inlinesInBalancedBrackets. This is just a small improvement in terms of performance, but it’s simpler and more direct code. Also, we avoid parsing interparagraph spaces in balanced brackets, as the original did.

BibTeX reader: Properly handle commented lines in BibTeX/BibLaTeX (#7668).

RST reader: handle class attribute for for custom roles (#7699, willj-dev). Previously the class attribute was ignored, and the name of the role used as the class.

DocBook reader:

• Add <titleabbr> support (Rowan Rodrik van der Molen).
• Support for <indexterm> (#7607, Rowan Rodrik van der Molen).

LaTeX reader:

• Add rudimentary support for \autoref (#7693).
• Add ‘uri’ class when parsing \url, for consistency with treatment of autolinks in other formats (#7672).

JATS reader: Capture alt-text in figures (#7703, Aner Lucero).

MediaWiki writer: use HTML spans for anchors when header has id (#7697). We need to generate a span when the header’s ID doesn’t match the one MediaWiki would generate automatically. Note that MediaWiki’s generation scheme is different from pandoc’s (it uses uppercase letters, and _ instead of -, for example). This means that in going from markdown to mediawiki, we’ll now get spans before almost every heading, unless explicit identifiers are used that correspond to the ones MediaWiki auto-generates. This is uglier output but it’s necessary for internal links to work properly.

Markdown writer: don’t create autolinks when this loses information (#7692). Previously we sometimes lost attributes when rendering links as autolinks.

Text.Pandoc.Readers.Metadata: allow multiple YAML documents when parsing YAML for yamlBsToRefs. Some people use --- as the end delimiter in YAML bibliography files, which causes the yaml library to emit an error unless we explicitly allow multiple YAML documents (and just consider the first).

JATS writer:

• Ensure figures are wrapped with <p> in list items (Albert Krewinkel). This prevents the generation of invalid output.
• Add URL to element citation entries (Albert Krewinkel). The URL of a reference, if present, is added in tag <uri> to element-citation entries.

HTML writer: Don’t create invalid data- attribute for empty attribute key (#7546).

LaTeX writer:

• Babel mappings: use ancientgreek for grc.
• With -t latex-smart, don’t generate \ldots from ellipsis (#7674). Instead just use unicode ellipsis.

JATS template: fix equal-contrib attribute (Albert Krewinkel). The standard requires the value to be either yes or no, but is was set to true for authors who contributed equally.

reveal.js template: Add disableLayout variable (Christophe Dervieux).

Text.Pandoc.Error: sort errors in handleError by exit code (Albert Krewinkel).

Text.Pandoc.Writers.Shared: Improve toLegacyTable (#7683, Christian Despres).

Lua subsystem:

• Include lpeg module (#7649, Albert Krewinkel). Compiles the lpeg library (Parsing Expression Grammars For Lua) into the program. Package maintainers may choose to rely on package dependencies to make lpeg available, in which case they can compile the with the constraint lpeg +rely-on-shared-lpeg-library. lpeg and re are always made available in global variables, without the need for a require.

• Set lpeg and re as globals; allow shared lib access via require. The lpeg and re modules are loaded into globals of the respective name, but they are not necessarily registered as loaded packages. This ensures that

  • the built-in library versions are preferred when setting the globals,
  • a shared library is used if pandoc has been compiled without lpeg, and
  • the require mechanism can be used to load the shared library if available, falling back to the internal version if possible and necessary.

• Fix argument order in constructor pandoc.Cite (Albert Krewinkel). This restores the old behavior; argument order had been switched accidentally in pandoc 2.15.

• Add Pushable instance for ReaderOptions (Albert Krewinkel).

• Allow to pass custom reader options to pandoc.read as an optional third argument (#7656, Albert Krewinkel). The object can either be a table or a ReaderOptions value like PANDOC_READER_OPTIONS. Creating new ReaderOptions objects is possible through the new constructor pandoc.ReaderOptions.

• Display Pandoc values using their native Haskell representation (Albert Krewinkel).

• Require latest hslua (2.0.1) (#7661, #7657, Albert Krewinkel). This fixes issues with

  • misleading error messages when a required function parameter is omitted;
  • absent properties still being listed in the output of pairs; and
  • alias accessing leading to errors instead of returning nil, e.g. with (pandoc.Str '').identifier.

• Add missing space in “package not found” message (#7658, Albert Krewinkel).

Update build files (#7696, Fabián Heredia Montiel). Drop old windows 32-bit constraints. Update cabal tested-with field to correspond to ci.yml matrix

Remove unneeded package dependencies from benchmark target.

Require ghc >= 8.6, base >= 4.12. This allows us to get rid of the old custom prelude and some crufty cpp. But the primary reason for this is that conduit has bumped its base lower bound to 4.12, making it impossible for us to support lower base versions.

Require Cabal 2.4. Use wildcards to ensure that all pptx tests are included (#7677).

Update bash_completion.tpl (S.P.H.).

Add data/creole.lua as sample custom reader.

Add doc/custom-readers.md and doc/custom-writers.md.

doc/lua-filters.md: add section on global modules, including lpeg (Albert Krewinkel).

MANUAL.txt: update table of exit codes and corresponding errors (Albert Krewinkel).

Use latest texmath.

Docx reader: don’t let first line indents trigger block quotes (#7655). This fixes a regression introduced in pandoc 2.15.

Docx writer: use getTimestamp for modification times in reference.docx (#7654). This ensures that when SOURCE_DATE_EPOCH is set, the modification times of files taken from the reference.docx will be set deterministically, allowing for reproducible builds.

Lua subsystem (Albert Krewinkel):

• Load module pandoc.path on startup (#7524). Previously the module always had to be loaded via require 'pandoc.path'.
• Fix typo in SoftBreak constructor.
• Re-add content property to Strikeout elements. Fixes a regression introduced in 2.15.
• Be more forgiving when retrieving the Image caption property. Fixes a regression introduced in 2.15.
• Display Attr values using their native Haskell representation.
• Allow omitting the 2nd parameter in pandoc.Code constructor. Fixes a regression introduced in 2.15 which required users to always specify an Attr value when constructing a Code element.
• Allow to compare, show Citation values. Comparisons of Citation values are performed in Haskell; values are equal if they represent the same Haskell value. Converting a Citation value to a string now yields its native Haskell string representation.
• Restore List behavior of MetaList (#7650). Fixes a regression introduced in 2.16 which had MetaList elements lose the pandoc.List properties.
• Restore content property on Header elements.
• Ensure Block elements have all expected properties.
• Ensure Inline elements have all expected properties.

Allow tasty-bench 0.3.x.

Switch back from HsYAML to yaml for parsing YAML metadata (#6084). HsYAML is around 20 times slower in parsing large YAML bibliographies. In addition, HsYAML is not being actively maintained. This sets us back in our attempts to free ourselves from C dependencies (#4535). But I don’t see a good alternative until a faster pure Haskell parser is available. Notes:

• We’ve removed the FromYAML instances for all types that had them, since this is a HsYAML-specific typeclass [API change]. (The yaml package just uses From/ToJSON instead of having a dedicated From/ToYAML class.)
• Unlike HsYAML (in the configuration we were using), yaml parses ‘Y’, ‘N’, ‘Yes’, ‘No’, ‘On’, ‘Off’ as boolean values. Users may need to quote these when they are meant to be interpreted as strings. Similarly, ‘null’ is parsed as a YAML null value (and will be treated as an empty string by pandoc rather than the string ‘null’). Quoting it will force it to be interpreted as a string.
• Some tests had to be adjusted accordingly.
• Pandoc now behaves in a more useful way when the YAML metadata contains escaping errors: instead of just failng silently and falling back to some other interpretation of the section, it raises a YAML parsing error.

Markdown writer: Ensure that special values are quoted in YAML metadata. These include “Y”, “yes”, “on”, and “off”, which are now (with yaml library) considered boolean values, as well as “null”.

Change JSON encodings of some types.

• For LineEnding use lowercase constructors, e.g. crlf, native.
• For HTMLSlideVariant use lowercase constructors.
• For ReaderOptions use e.g. default-image-extension instead of readerDefaultImageExtension for field names.
• For Extension, use e.g. tex_math_dollars instead of Ext_tex_math_dollars as constructor.
• For Extensions, use an array of Extensions, instead of an object wrapping the tag Extensions and an integer. (The integer representation is not supposed to be part of the public API.)
• For Opt, use field names like tab-stop instead of optTabStop.

Docx writer:

• Add IDs to native_numbering test (Tristan Stenner).
• Move “:” out of the caption bookmark (Tristan Stenner). This is needed so that native references to the figure are included as “As seen in Figure X, it is…” instead of “As seen in [Figure: X, it is…”

Lua (Albert Krewinkel, except as noted):

• Use hslua module abstraction where possible.

• Fix placement of tests for Block elements in pandoc module tests

• Increase strictness when getting attribute keys

• Re-add t and tag property to Attr values. Removal of these properties from Attr values was a regression.

• Fix pandoc.utils.stringify regression. The pandoc.utils.stringify function returned empty strings when called with a string argument.

• Fix a copy/paste bug in Lua marshalling code (John MacFarlane, #7639). This caused links to be changed to figures when Lua filters changed link properties.

• Re-add content property to Link elements (#7647). This was a regression introduced in version 2.15.

• Generate constants in module pandoc programmatically.

• Marshal SimpleTable, ListAttributes, Citation, and Block values as userdata objects. Properties of Block values are marshalled lazily, which generally improves performance considerably. Script users may also notice the following differences:

  • Block element properties can no longer be accessed by numerical indexing of the .c field. The .c property now serves as an alias for .content, so some filter that used this undocumented method for property access may continue to work, while others will need to be updated and use proper property names.
  • The marshalled Block elements now have a show method, and a __tostring metamethod. Both return the Haskell string representation of the element.
  • Block values now have the Lua type userdata instead of table.

Add a short guide to pandoc’s sources (Albert Krewinkel).

Fix epub files in epub reader tests, so that they are valid according to epubcheck (#7586).

Allow time 1.13.

Require latest skylighting (0.12.1).

Fix build on GHC 9.2 (Joseph C. Sible).

Fix trypandoc so it builds with aeson > 2.