• Overview
• Use Case / Problem Description
• Software Components
• Target Audience
• Repository Structure Overview
• Documentation
• Prerequisites
• Hardware Requirements
• Quickstart Guide
• Known CVEs
• License

This repository is what powers the build experience, showcasing video search and summarization agent with NVIDIA NIM microservices.

Insightful, accurate, and interactive video analytics AI agents enable a range of industries to make better decisions faster. These AI agents are given tasks through natural language and can perform complex operations like video summarization and visual question-answering, unlocking entirely new application possibilities. The NVIDIA AI Blueprint makes it easy to get started building and customizing video analytics AI agents for video search and summarization — all powered by generative AI, vision language models (VLMs) like Cosmos Nemotron VLMs, large language models (LLMs) like Llama Nemotron LLMs, NVIDIA NeMo Retriever, and NVIDIA NIM.

The NVIDIA AI Blueprint for Video Search and Summarization addresses the challenge of efficiently analyzing and summarizing large volumes of video data. This can be used to create vision AI agents, that can be applied to a multitude of use cases such as monitoring smart spaces, warehouse automation, and SOP validation. This is important where quick and accurate video analysis can lead to better decision-making and enhanced operational efficiency.

1. NIM microservices: Here are models used in this blueprint:

   • VILA
   • meta / llama-3.1-70b-instruct
   • llama-3_2-nv-embedqa-1b-v2
   • llama-3_2-nv-rerankqa-1b-v2

2. Ingestion Pipeline:

   The process involves decoding video segments (chunks) generated by the stream handler, selecting frames, and using a vision-language model (VLM) along with a caption prompt to generate detailed captions for each chunk. A computer vision pipeline enhances video analysis by providing detailed metadata on objects. In parallel, the audio is extracted and a transcription is generated. These dense captions, along with audio transcripts and CV metadata are then indexed into vector and graph databases for use in the Context-Aware Retrieval-Augmented Generation workflow.

3. CA-RAG module:

   The Context-Aware Retrieval-Augmented Generation (CA-RAG) module leverages both Vector RAG and Graph-RAG as primary sources for video understanding. This module is utilized in key features such as summarization, Q&A, and sending alerts. During the Q&A workflow, the CA-RAG module extracts relevant context from the vector database and graph database to enhance temporal reasoning, anomaly detection, multi-hop reasoning, and scalability. This approach offers deeper contextual understanding and efficient management of extensive video data. Additionally, the context manager effectively maintains its working context by efficiently using both short-term memory, such as chat history, and long-term memory resources like vector and graph databases, as needed.

This blueprint is designed for ease of setup with extensive configuration options, requiring technical expertise. It is intended for:

1. Video Analysts and IT Engineers: Professionals focused on analyzing video data and ensuring efficient processing and summarization. The blueprint offers 1-click deployment steps, easy-to-manage configurations, and plug-and-play models, making it accessible for early developers.

2. GenAI Developers / Machine Learning Engineers: Experts who need to customize the blueprint for specific use cases. This includes modifying the RAG pipeline for unique datasets and fine-tuning LLMs as needed. For advanced users, the blueprint provides detailed configuration options and custom deployment possibilities, enabling extensive customization and optimization.

• deploy/: Contains scripts for docker compose and helm chart deployment, along with notebook for Brev launchable deployment.
• src/: Source code for the video search and summarization agent.

For detailed instructions and additional information about this blueprint, please refer to the official documentation.

• NVIDIA AI Enterprise developer licence required to local host NVIDIA NIM.
• API catalog keys:
  • NVIDIA API catalog or NGC (steps to generate key)

The platform requirement can vary depending on the configuration and deployment topology used for VSS and dependencies like VLM, LLM, etc. For a list of validated GPU topologies and what configuration to use, see the supported platforms.

Ideal for: Quickly getting started with your own videos without worrying about hardware and software requirements.

Follow the steps from the documentation and notebook in deploy directory to complete all pre-requisites and deploy the blueprint using Brev Launchable in an 8xL040s Crusoe instance.

• deploy/1_Deploy_VSS_docker_Crusoe.ipynb: This notebook is tailored specifically for the Crusoe CSP which uses Ephemeral storage.

Ideal for: Development phase where you need to run VSS locally, test different models, and experiment with various deployment configurations. This method offers greater flexibility for debugging each component.

For custom VSS deployments through Docker Compose, multiple samples are provided to show different combinations of remote and local model deployments. The /deploy/docker directory contains a README with all the details. Link to README

• Ubuntu 22.04
• NVIDIA driver 535.161.08 (Recommended minimum version)
• CUDA 12.2+ (CUDA driver installed with NVIDIA driver)
• NVIDIA Container Toolkit 1.13.5+
• Docker 27.5.1+
• Docker Compose 2.32.4

Ideal for: Production deployments that need to integrate with other systems. Helm offers advantages such as easy upgrades, rollbacks, and management of complex deployments.

The /deploy/helm/ directory contains a nvidia-blueprint-vss-2.3.0.tgz file which can be used to spin up VSS. Refer to the documentation here for detailed instructions.

• Ubuntu 22.04
• NVIDIA driver 535.183.06 (Recommended minimum version). NVIDIA driver 570.86.15 (for H200)
• CUDA 12.2+ (CUDA driver installed with NVIDIA driver)
• Kubernetes v1.31.2
• NVIDIA GPU Operator v23.9 (Recommended minimum version)
• Helm v3.x

VSS Engine 2.3.0 Container has the following known CVEs:

VSS Engine 2.3.0 Source Code has the following known CVEs:

The software and materials in this repository are governed by the NVIDIA Software License Agreement and the Product-Specific Terms for NVIDIA AI Products; except for models which are governed by the NVIDIA Community Model License.

ADDITIONAL INFORMATION: Llama 3.1 Community License Agreement for Llama-3.1-70b-instruct; Llama 3.2 Community License Agreement for NVIDIA Retrieval QA Llama 3.2 1B Embedding v2 and NVIDIA Retrieval QA Llama 3.2 1B Reranking v2, Apache, Version 2.0 for https://github.com/google-research/big_vision/blob/main/LICENSE and Apache License, Version 2.0 for https://github.com/01-ai/Yi/blob/main/LICENSE. Built with Llama.