Disaggregated serving system for Large Language Models (LLMs).

Unify Efficient Fine-Tuning of 100+ LLMs

LLM inference in C/C++

High-speed Large Language Model Serving on PCs with Consumer-grade GPUs

TensorRT-LLM provides users with an easy-to-use Python API to define Large Language Models (LLMs) and build TensorRT engines that contain state-of-the-art optimizations to perform inference efficie…

A high-throughput and memory-efficient inference and serving engine for LLMs

NVIDIA® TensorRT™, an SDK for high-performance deep learning inference, includes a deep learning inference optimizer and runtime that delivers low latency and high throughput for inference applicat…

Running large language models on a single GPU for throughput-oriented scenarios.

This is the implementation repository of our SOSP'23 paper: Ditto: An Elastic and Adaptive Memory-Disaggregated Caching System.

CLI tool for spawning and running containers according to the OCI specification

Memkind is an easy-to-use, general-purpose allocator which helps to fully utilize various kinds of memory available in the system, including DRAM, NVDIMM, and HBM

ZenFS is a storage backend for RocksDB that enables support for ZNS SSDs and SMR HDDs.

Curve is a high-performance, lightweight-operation, cloud-native open source distributed storage system. Curve can be applied to: 1) mainstream cloud-native infrastructure platforms OpenStack and K…

OpenDAL: Access data freely, painlessly, and efficiently

To record some notes when I read the rocksdb source code