KML: Knowledge Module Learning

PKR-QA: A Benchmark for Procedural Knowledge Reasoning

---

Overview

Knowledge Module Learning (KML) is a neurosymbolic framework that learns structured knowledge modules from relational data and performs procedural reasoning over multi-step tasks.

PKR-QA is the first benchmark for Procedural Knowledge Reasoning, combining instructional videos (COIN dataset), knowledge graphs, step predictions, and structured question-answer pairs.

This repository contains:

• KML source code
• PKR-QA dataset (JSON format)
• ProcedureVRL task/step predictions
• Knowledge graph files
• Training and evaluation scripts

---

Table of Contents

• Overview
• Dataset
• Dataset Structure
• COIN Dataset
• ProcedureVRL Predictions
• Installation
• Citations
• License

---

Dataset

Dataset is available when you clone this repo. Download and extract the PKR-QA dataset using:

tar -I zstd -xf pkr-qa.tar.zst

PKR-QA includes:

• A knowledge graph (cointrain_kgv2.json)

• Step and task predictions from ProcedureVRL

• QA splits for training, validation, and testing

• Small sample splits for fast prototyping

Dataset Structure

dataset/
├── cointrain_kgv2.json                     # Knowledge graph (KG)
├── QA_25Oct24_testing_pred.json            # ProcedureVRL predictions (test)
├── QA_25Oct24_validation_pred.json         # ProcedureVRL predictions (val)
└── s4_QADataset_12Feb2025/
    ├── testing.json    
    ├── train/
    │   └── training_small_100.json
    └── val/
        └── validation_small_50.json

COIN Dataset

Videos used in PKR-QA are from the COIN dataset: https://coin-dataset.github.io/

@INPROCEEDINGS{
    title={COIN: A Large-scale Dataset for Comprehensive Instructional Video Analysis},
    author={Yansong Tang, Dajun Ding, Yongming Rao, Yu Zheng, Danyang Zhang, Lili Zhao, Jiwen Lu, Jie Zhou},
    booktitle={IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},
    year={2019}
}

ProcedureVRL Predictions

PKR-QA uses ProcedureVRL for task and step predictions: https://github.com/facebookresearch/ProcedureVRL

@inproceedings{zhong2023learning,
  title={Learning Procedure-aware Video Representation from Instructional Videos and Their Narrations},
  author={Zhong, Yiwu and Yu, Licheng and Bai, Yang and Li, Shangwen and Yan, Xueting and Li, Yin},
  booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  pages={14825--14835},
  year={2023}
}

Installation

• Clone the repo
• Run setup.sh
• Need pytorch 2.8 or later.

Train and testing

python KML_Main.py -s

Citations

@article{nguyen2025neuro,
  title={Neuro Symbolic Knowledge Reasoning for Procedural Video Question Answering},
  author={Nguyen, Thanh-Son and Yang, Hong and Neoh, Tzeh Yuan and Zhang, Hao and Keat, Ee Yeo and Fernando, Basura},
  journal={arXiv preprint arXiv:2503.14957},
  year={2025}
}

@inproceedings{nguyen2025aaai,
  title={PKR-QA: A Benchmark for Procedural Knowledge Reasoning with Knowledge Module Learning},
  author={Nguyen, Thanh-Son and Yang, Hong and Neoh, Tzeh Yuan and Zhang, Hao and Keat, Ee Yeo and Fernando, Basura},
  booktitle={AAAI},
  year={2026}
}

Acknowledgments

This research/project is supported by the National Research Foundation, Singapore, under its NRF Fellowship (Award# NRF-NRFF14-2022-0001) and by funding allocation to Basura Fernando by the A*STAR under its SERC Central Research Fund (CRF), as well as its Centre for Frontier AI Research.