SAM3-Edge

Real-time video segmentation on NVIDIA Jetson using SAM3 + TensorRT + DeepStream.

Overview

SAM3-Edge brings Meta's Segment Anything Model 3 (SAM3) to edge devices, specifically optimized for NVIDIA Jetson AGX Orin. It provides:

• TensorRT Acceleration: ViT encoder and mask decoder exported to TensorRT FP16 engines
• DeepStream Integration: GStreamer pipeline with nvinfer plugin for hardware-accelerated video processing
• FastAPI Server: REST API for inference on images and video streams
• CUDA Streams: Async pipelined inference to minimize latency
• Text Prompts: Segment objects by description ("red car", "person") using SAM3's native VETextEncoder
• SQLite Storage: Persistent detection storage for video processing

Architecture

sam3-edge/
├── sam3/                     # SAM3 model (local submodule)
│   └── sam3/model/trt_export.py  # TensorRT export wrapper
├── sam3_deepstream/          # Main package
│   ├── api/                  # FastAPI REST server
│   ├── export/               # TensorRT export utilities
│   ├── inference/            # Runtime inference engines
│   └── deepstream/           # DeepStream pipeline configs
├── install_jetson.sh         # One-shot Jetson installation script
└── pyproject.toml            # UV workspace config

Dependencies:

• sam3/ - SAM3 model with TensorRT export support (local directory)

Quick Start

Prerequisites

• NVIDIA Jetson AGX Orin with JetPack 6.x (R36.4+)
• Docker with NVIDIA Container Toolkit
• PyTorch with CUDA from NVIDIA's Jetson wheel server

Installation (One Command)

# Clone with Git LFS (required for TensorRT engines)
git lfs install
git clone https://github.com/oldhero5/sam3-edge.git
cd sam3-edge
git lfs pull

# Place SAM3 checkpoint in expected location
mkdir -p sam3/checkpoints
cp /path/to/sam3.pt sam3/checkpoints/

# Run full installation (builds Docker, exports TensorRT engines)
./install_jetson.sh

The installation script:

1. Installs system dependencies and UV package manager
2. Builds the Docker image with all dependencies
3. Exports TensorRT engines inside Docker (ensures TRT version compatibility)
4. Prepares database directories

Run API Server

cd sam3_deepstream
docker compose -f docker-compose.jetson.yml up -d

# Check health
curl http://localhost:8000/health

API Usage

# Health check
curl http://localhost:8000/health

# Text-based segmentation (returns JSON with RLE masks, saves PNG to outputs/)
curl -X POST http://localhost:8000/api/v1/segment \
  -F "[email protected]" \
  -F "text_prompt=car"

# Point-based segmentation
curl -X POST http://localhost:8000/segment \
  -F "[email protected]" \
  -F "points=0.5,0.5,1"

Configuration

Environment variables:

Variable	Default	Description
SAM3_USE_TRT	0	Enable TensorRT acceleration
SAM3_ENGINE_DIR	./engines	Directory containing TRT engines
SAM3_OUTPUT_DIR	./outputs	Directory for saved results
SAM3_USE_PE_BACKBONE	0	Enable Perception Encoder backbone
SAM3_CHECKPOINT	-	Path to SAM3 checkpoint
API_HOST	0.0.0.0	API server host
API_PORT	8000	API server port

Performance

On Jetson AGX Orin (64GB):

Component	Precision	Latency
ViT Encoder	FP16	~35ms
Mask Decoder	FP16	~8ms
End-to-end	FP16	~50ms

With keyframe optimization (reusing embeddings):

• Keyframe: ~50ms
• Non-keyframe: ~8ms (decoder only)

Development

# Run tests (Docker with GPU)
make test

# Build Docker image
make build

# Export TensorRT engines
make export

Jetson Installation

For complete Jetson AGX Orin setup, use the installation script:

# Full installation (system deps, Docker, TRT export, database)
./install_jetson.sh

# With explicit checkpoint path
./install_jetson.sh --checkpoint /path/to/sam3_checkpoint.pt

# Docker image only (no TRT export)
./install_jetson.sh --docker-only

# Skip Docker rebuild (use existing image)
./install_jetson.sh --skip-docker

# Health check
./install_jetson.sh --health-check

TensorRT Version Compatibility

TensorRT engines are platform-specific and must be built with the same TensorRT version used at runtime. The installation script handles this automatically by:

1. Building the Docker image first
2. Running TensorRT export inside the Docker container
3. Mounting the output directory so engines are accessible on host

This ensures the engines work correctly regardless of host vs container TensorRT version differences.

Installation Options

Option	Description
--checkpoint <path>	Export TensorRT engines from checkpoint
--docker-only	Only build Docker image
--skip-docker	Skip Docker image build
--init-db	Initialize database only
--health-check	Run system health checks
--precision <fp16|fp32|int8>	TensorRT precision (default: fp16)
--push	Push image to DockerHub
--repo <user/repo>	DockerHub repository name

DockerHub

Pulling Pre-built Images

# Pull Jetson image
docker pull sam3edge/sam3-edge:jetson-latest

# Run with docker-compose
cd sam3_deepstream
docker compose -f docker-compose.jetson.yml up -d

Publishing to DockerHub

# Build and push to your DockerHub account
./install_jetson.sh --push --repo yourusername/sam3-edge

# Or manually:
docker login
docker tag sam3-edge:jetson yourusername/sam3-edge:jetson-latest
docker push yourusername/sam3-edge:jetson-latest

Federation (Future)

SAM3-Edge is designed for future multi-device federation where many Jetson devices process video at the edge and sync detections to a central server.

Current design supports:

• device_id tracking in all database tables
• Export endpoint for syncing detections

Future features (planned):

• Automatic sync to central aggregation server
• Distributed query across device network

License

MIT License - see LICENSE file.

Acknowledgments

• Meta AI SAM3 - Original SAM3 implementation
• NVIDIA DeepStream - Video analytics SDK