Transformer related optimization, including BERT, GPT

"Everyday life is like programming, I guess. If you love something you can put beauty into it." ― Donald E. Knuth

Machine Learning Engineering Open Book

🔥 Comprehensive survey on Context Engineering: from prompt engineering to production-grade AI systems. hundreds of papers, frameworks, and implementation guides for LLMs and AI agents.

The Book of Statistical Proofs

😎 Awesome list of Infrastructure-from-Code

A concise but complete full-attention transformer with a set of promising experimental features from various papers

Vector (and Scalar) Quantization, in Pytorch

H-Net: Hierarchical Network with Dynamic Chunking

Foundation Architecture for (M)LLMs

An implementation of local windowed attention for language modeling

🚀 Efficient implementations of state-of-the-art linear attention models

Large Context Attention

Torch implementation of ResNet from http://arxiv.org/abs/1512.03385 and training scripts

Useful resources on data quality for machine learning and artificial intelligence.

1 Line of code data quality profiling & exploratory data analysis for Pandas and Spark DataFrames.

A collection of research papers on low-precision training methods

Quantized Neural Networks - networks trained for inference at arbitrary low precision.

This is a plugin which lets EC2 developers use libfabric as network provider while running NCCL applications.

Ring attention implementation with flash attention

official code for "Large Language Models as Optimizers"

TextGrad: Automatic ''Differentiation'' via Text -- using large language models to backpropagate textual gradients. Published in Nature.

Triton-based Symmetric Memory operators and examples

Write a fast kernel and run it on Discord. See how you compare against the best!

A curated list of awesome resources combining Transformers with Neural Architecture Search