appjsonify¹ is a handy PDF-to-JSON conversion tool for academic papers implemented in Python. appjsonify allows you to obtain a structured JSON file that can be easily used for various downstream tasks such as paper recommendation, information extraction, and information retrieval from papers.

• Linux or macOS (Not tested on Windows)
• Python 3.10 or later
• pdfplumber
• registrable
• tqdm
• pillow
• pdf2image
• torch
• detectron2

  Please manually install it based on the instructions.

If your environment does not have poppler, please install it. This is necessary to obtain PDF images using pdf2image. For more details, refer to Prerequisites.

pip install appjsonify
python -m pip install 'git+https://github.com/facebookresearch/detectron2.git'

git clone https://github.com/hitachi-nlp/appjsonify.git
python -m pip install --editable .
python -m pip install 'git+https://github.com/facebookresearch/detectron2.git'

appjsonify offers two options to structure your paper PDF file into a JSON file.

1. Use the existing templates
   Suitable if a paper adopts the AAAI, ACL, ICML, ICLR, NeurIPS, IEEE, ACM, or Springer styles. See Templates for more details.
2. Configure pipelines and parameters by yourself
   If a paper does not adopts the above formats, you need to specify the processing pipeline and its parameters. Please refer to Build your own pipeline for further information.

appjsonify provides two types of the templates for each of the following paper types: AAAI, ACL, ICML, ICLR, NeurIPS, IEEE, ACM, and Springer. One is more accurate but slower due to the use of machine learning based models, the other is less accurate but faster due to its rule based approach.

appjsonify /path/to/pdf/dir/or/path /path/to/output/dir --paper_type AAAI

If your environment has a GPU(s), it is better to also specify --detectron_device_mode cuda to speed up the process.

appjsonify /path/to/pdf/dir/or/path /path/to/output/dir --paper_type AAAI2

appjsonify /path/to/pdf/dir/or/path /path/to/output/dir --paper_type ACL

appjsonify /path/to/pdf/dir/or/path /path/to/output/dir --paper_type ACL2

appjsonify /path/to/pdf/dir/or/path /path/to/output/dir --paper_type ICML

appjsonify /path/to/pdf/dir/or/path /path/to/output/dir --paper_type ICML2

appjsonify /path/to/pdf/dir/or/path /path/to/output/dir --paper_type ICLR

appjsonify /path/to/pdf/dir/or/path /path/to/output/dir --paper_type ICLR2

appjsonify /path/to/pdf/dir/or/path /path/to/output/dir --paper_type NeurIPS

appjsonify /path/to/pdf/dir/or/path /path/to/output/dir --paper_type NeurIPS2

Currently only tested with IEEE BigData papers.

appjsonify /path/to/pdf/dir/or/path /path/to/output/dir --paper_type IEEE

appjsonify /path/to/pdf/dir/or/path /path/to/output/dir --paper_type IEEE2

Currently only tested with TALLIP papers.

appjsonify /path/to/pdf/dir/or/path /path/to/output/dir --paper_type ACM

appjsonify /path/to/pdf/dir/or/path /path/to/output/dir --paper_type ACM2

appjsonify /path/to/pdf/dir/or/path /path/to/output/dir --paper_type Springer

appjsonify /path/to/pdf/dir/or/path /path/to/output/dir --paper_type Springer2

• --verbose: If you want to check the intermediate processing results, please set this flag. The log files will be saved under output_dir. Optionally, you can use the following four flags to add the corresponding information.
  • --show_pos: Bounding box information.
  • --show_font: Font name and size information.
  • --show_style: Style information (e.g., section, body, abstract, etc.)
  • --show_meta: Supplementary information (e.g., information on objects and footnotes.)
  • --insert_page_break: Insert breaks between pages.
• --save_image: If you are using a more accurate but slower version of templates or load_objects_with_ml, appjsonify can save detected table and figure images if this flag is set. In addition to this, please also specify the output directory path as --output_image_dir.

appjsonify also allows users to build their own academic paper PDF-to-JSON processing pipeline. For more details, please refer to Available Modules and Document Handling in appjsonify.

Users can add their own modules to appjsonify for more flexible document processing. To add modules, appjsonify must be installed in editable mode. See Customize appjsonify for more details and feel free to make a PR if you wish to add your module to this repository and package!

Contributions are more than welcome! Feel free to raise an issue and/or make a PR. Possible future work is as follows:

• Better documentation
• More paper templates
• More robust references extraction
• Powerful mathematical equation support
• Robust algorithm description detection
• Multilingual support
• Add more test scripts

If you use appjsonify in your work, please cite the following.

@article{yamaguchi2023appjsonify,
      title={appjsonify: An Academic Paper PDF-to-JSON Conversion Toolkit}, 
      author={Atsuki Yamaguchi and Terufumi Morishita},
      year={2023},
      eprint={2310.01206},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}

© 2023 Atsuki Yamaguchi and Terufumi Morishita (Hitachi, Ltd.)

This work is licensed under the MIT license unless specified.

appjsonify uses the follwoing publicly available works.

• pdfplumber by Jeremy Singer-Vine (MIT License).
• registrable by epwalsh (Apache License 2.0).
• tqdm (MIT License, Mozilla Public License 2.0 (MPL 2.0)).
• pillow by Jeffrey A. Clark (Historical Permission Notice and Disclaimer License).
• pdf2image by Edouard Belval (MIT License).
• torch (BSD-style license).
• Detectron2 by Facebook AI Research (Apache License 2.0) in detectron2_demo.
• DocBank pretrained model by Microsoft Research Asia (Apache License 2.0) in docbank.py.
• TableBank pretrained model by Microsoft Research Asia (Apache License 2.0) in tablebank.py.
• PubLayNet pretrained model by hpanwar08 (Apache License 2.0) in publaynet.py.