Do language models understand time?🧐 In the kitchen arena🧑‍🍳, where burritos are rolled🌯, rice waits patiently🍚, and sauce steals the spotlight, LLMs try their best to keep up. Captions flow like a recipe—precise and tempting—but can they truly tell the difference between prepping, cooking, and eating? After all, in cooking, timing isn’t just everything—it’s the secret sauce!🥳🥳🥳

👋👋👋 A collection of papers and resources related to Large Language Models in video domain🎞️.

📌 More details please refer to our paper.

🛠️ Please let us know if you find out a mistake or have any suggestions by e-mail: [email protected]

If you find our work useful for your research, please cite the following paper:

@inproceedings{10.1145/3701716.3717744,
author = {Ding, Xi and Wang, Lei},
title = {Do Language Models Understand Time?},
year = {2025},
isbn = {9798400713316},
publisher = {Association for Computing Machinery},
address = {New York, NY, USA},
url = {https://doi.org/10.1145/3701716.3717744},
doi = {10.1145/3701716.3717744},
pages = {1855–1868},
numpages = {14},
keywords = {interaction, language language models, temporal, videos},
location = {Sydney NSW, Australia},
series = {WWW '25}
}

• [10/02/2025] 🎁 The GitHub repository for our paper has been released.
• [27/01/2025] 🎈 Our paper has been accepted as an oral presentation at the Companion Proceedings of The Web Conference 2025 (WWW 2025)

• Video-LLM
  • 🔥🔥🔥 Do Language Models Understand Time?
  • 📑 Citation
  • 🚀 News
  • 📸 Models with Image Encoder
    • Normalizer-Free ResNet
    • CLIP ViT
    • EVA-CLIP ViT
    • BLIP-2 ViT
    • SigLIP
    • Oryx ViT
  • 🎥 Models with Video Encoder
    • Traditional (e.g., I3D, SlowFast)
    • TimeSformer
    • VideoSwin
    • UMT
    • LanguageBind
    • VideoMAE V2
    • InternVL
    • InternVideo/InternVideo2
  • 💻 Datasets
    • Action Recognition
    • Video QA
    • Video Captioning
    • Video Retrieval
    • Anomaly Detection
    • Multimodal Video Tasks
  • ❤️‍🔥❤️‍🔥❤️‍🔥 Contribution

Performance comparison of visual encoders. (Left): Image classification accuracy for various image encoders pretrained and fine-tuned on the ImageNet-1K dataset. (Right): Action recognition accuracy for different video encoders pretrained and fine-tuned on the Kinetics-400 and Something-Something V2 datasets.

✨ The tables below present summaries of the latest multimodal video-LLMs with image encoders and their interaction and fusion mechanisms.

✨ The tables below present summaries of the latest multimodal video-LLMs with video encoders and their interaction and fusion mechanisms.

The distributions of interaction/fusion mechanisms and data modalities in 66 closely related video-LLMs from January 2024 to December 2024. (Left): Fusion mechanisms are classified into five categories: Cross-attention (e.g., crossattention modules, gated cross-attention), Projection layers (e.g., linear projection, MLP projection), Q-Former-based methods (e.g., Q-Former aligns multi-modal embeddings, Trainable Q-Former), Motion/Temporal-Specific mechanisms (e.g., temporal samplers, scene-level sequential alignment), and Other Methods (e.g., Tracklet-centric, Perceiver Resampler, MQ-Former). (Right): The distribution of data modalities used in these video-LLMs, with text modalities appearing across all models. Note that a model may use multiple fusion methods and/or data modalities.

✨The tables below provide a comprehensive overview of video datasets across various tasks.

Left: Performance (accuracy) comparison of recent videoLLMs on the Video-MME benchmark. Right: Performance comparison of recent video-LLMs on video QA benchmarks. Models using pretrained video encoders (e.g., Video-LLaVA and VideoChat2) are marked with squares, while models using pretrained image encoders are represented by circles.

Performance comparison of recent video-LLMs on (a) video retrieval and (b) video captioning benchmarks.

We warmly invite everyone to contribute to this repository and help enhance its quality and scope. Feel free to submit pull requests to add new methods, datasets or other useful resources, as well as to correct any errors you discover. To ensure consistency, please format your pull requests using our tables' structures. We greatly appreciate your valuable contributions and support!