*This model was published in HF papers on 2024-02-20 and contributed to Hugging Face Transformers on 2026-06-19.*

<div style="float: right;">

<div class="flex flex-wrap space-x-1">

<img alt="FlashAttention" src="https://img.shields.io/badge/%E2%9A%A1%EF%B8%8E%20FlashAttention-eae0c8?style=flat">

</div>

</div>

The VideoPrism model was proposed in the paper [VideoPrism: A Foundational Visual Encoder for Video Understanding](https://huggingface.co/papers/2402.13217) by Google DeepMind ([blog post](https://research.google/blog/videoprism-a-foundational-visual-encoder-for-video-understanding/)).

VideoPrism is a general-purpose video encoder that tackles diverse video understanding tasks with a single frozen model. The model is pretrained on a large-scale heterogeneous corpus containing 36M high-quality video-caption pairs and 582M video clips with noisy parallel text (e.g., ASR transcripts). The pretraining approach improves upon masked autoencoding through global-local distillation of semantic video embeddings and a token shuffling scheme, enabling the model to focus primarily on the video modality while leveraging text associated with videos. VideoPrism achieves state-of-the-art performance on 31 out of 33 video understanding benchmarks across four broad task groups, from web video question answering to computer vision for science.

<div class="flex justify-center">

<img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/model_doc/VideoPrism_Overview.jpeg" alt="drawing" width="600"/>

</div>

You can find all original VideoPrism checkpoints under the [VideoPrism](https://huggingface.co/collections/google/videoprism) collection.

Notes:

- VideoPrism uses a factorized spatio-temporal encoder architecture, processing videos through separate spatial and temporal transformers.

- The model supports video-text contrastive learning through `VideoPrismClipModel`, which combines a video encoder and a text encoder. `VideoPrismConfig` must be used with this model.

- For video classification tasks, use `VideoPrismForVideoClassification` which adds a classification head on top of the video encoder. `VideoPrismVisionConfig` must be used with this model.

- The vision encoder can be used standalone via `VideoPrismVisionModel` for extracting video features. `VideoPrismVisionConfig` must be used with this model.

- The default input resolution is 288x288 pixels with 16 frames per video clip for the base models and 8 frames for the large models. Set interpolate_pos_encoding=True to use the models with custom resolution and frames per clip.

This model was contributed by [MHRDYN7](https://github.com/MHRDYN7) and reviewed by [vasqu](https://github.com/vasqu) & [zucchini-nlp](https://github.com/zucchini-nlp).

The original code can be found [here](https://github.com/google-deepmind/videoprism).

The snippet below shows how to load the VideoPrismVisionModel for feature extraction using the `AutoModel` class.

import torch

from transformers import AutoModel, AutoVideoProcessor

processor = AutoVideoProcessor.from_pretrained("google/videoprism-base-f16r288", revision="refs/pr/4")

model = AutoModel.from_pretrained(

"google/videoprism-base-f16r288",

revision="refs/pr/4",

device_map="auto",

# use "flash_attention_2" for faster inference on supported hardware

# attn_implementation="flash_attention_2"

)

video_url = "https://huggingface.co/datasets/nateraw/kinetics-mini/resolve/main/val/archery/-Qz25rXdMjE_000014_000024.mp4"

processed_video_inputs = processor(videos=[video_url], return_metadata=True, do_sample_frames=True)

video_metadata = processed_video_inputs["video_metadata"]

video_inputs = processed_video_inputs["pixel_values_videos"].to(model.device)

outputs = model(video_inputs)

encoder_outputs = outputs.last_hidden_state

[[autodoc]] VideoPrismVisionConfig

[[autodoc]] VideoPrismTextConfig

[[autodoc]] VideoPrismConfig

[[autodoc]] VideoPrismTokenizer

[[autodoc]] VideoPrismProcessor

[[autodoc]] VideoPrismVisionModel

- forward

[[autodoc]] VideoPrismVideoModel

- forward

[[autodoc]] VideoPrismTextModel

- forward

[[autodoc]] VideoPrismClipModel

- forward

[[autodoc]] VideoPrismForVideoClassification

- forward

class VideoPrismConverter(T5Converter):

def post_processor(self):

return None

"VideoPrismTokenizer": VideoPrismConverter,

from .videoprism import *

("videoprism", "VideoPrismConfig"),

("videoprism_text_model", "VideoPrismTextConfig"),

("videoprism_vision_model", "VideoPrismVisionConfig"),

("videoprism_text_model", "videoprism"),

("videoprism_vision_model", "videoprism"),

("videoprism", "VideoPrismProcessor"),

("videoprism", "VideoPrismClipModel"),

("videoprism_vision_model", "VideoPrismVisionModel"),

("videoprism_vision_model", "VideoPrismVisionModel"),

("videoprism_vision_model", "VideoPrismForVideoClassification"),

("videoprism", "VideoPrismClipModel"),

("videoprism", "VideoPrismTokenizer" if is_tokenizers_available() else None),

("videoprism", "LlavaOnevisionVideoProcessor"),

from typing import TYPE_CHECKING

from ...utils import _LazyModule

from ...utils.import_utils import define_import_structure

if TYPE_CHECKING:

from .configuration_videoprism import *

from .modeling_videoprism import *

from .processing_videoprism import *

from .tokenization_videoprism import *

else:

import sys

_file = globals()["__file__"]

sys.modules[__name__] = _LazyModule(__name__, _file, define_import_structure(_file), module_spec=__spec__)