awesome-llm-resources

The following table highlights seminal papers in the development of large language models (LLMs) and key transformer-based NLP research, listed chronologically. These works introduced foundational architectures, techniques, or insights that have significantly impacted real-world LLM capabilities and applications.

Date	Title	Link	By Whom	Key Contributions
1991	Learning to Control Fast-Weight Memories: An Alternative to Dynamic Recurrent Neural Networks	Paper	Jürgen Schmidhuber	Anticipated the transformer’s attention mechanism decades early by proposing a “fast weight” program where one network learns to rapidly modify another’s weights. This scheme is mathematically equivalent to the key-query-value self-attention in modern Transformers (Understanding Large Language Models), providing a historical blueprint for later attention-based models.
1997	Long Short-Term Memory	Paper	Sepp Hochreiter & Jürgen Schmidhuber	Introduced the LSTM neural network architecture with gating mechanisms to overcome vanishing gradients. LSTMs enabled learning long-range dependencies in sequence data and became a workhorse for NLP tasks (like speech recognition and translation) prior to transformers. This fundamental advance in recurrent networks laid the groundwork for handling long sequences in later models.
2014	Sequence to Sequence Learning with Neural Networks	Paper	Ilya Sutskever et al.	Pioneered the encoder–decoder architecture for neural sequence transduction. Using LSTM networks, the authors showed end-to-end translation of sentences was possible. This seq2seq framework introduced the idea of encoding an input sequence into a vector and decoding it to an output sequence, which became a core template for many NLP tasks and set the stage for neural machine translation (and later transformer-based seq2seq models).
2014	Neural Machine Translation by Jointly Learning to Align and Translate	Paper	Dzmitry Bahdanau et al.	Introduced the attention mechanism for sequence models (Understanding Large Language Models). This paper added an “alignment” model to focus a translation RNN on relevant parts of the input sentence (soft attention), greatly improving translation of long sentences. The concept of attention – dynamically weighting parts of the input – was pivotal in overcoming RNN limitations and directly inspired the development of the transformer architecture’s self-attention.
2017	Attention Is All You Need	Paper	Ashish Vaswani et al.	Proposed the original Transformer architecture, relying entirely on self-attention (no RNNs or CNNs) (Understanding Large Language Models). It introduced multi-head attention, scaled dot-product attention, and positional encodings in an encoder–decoder framework. Transformers achieved better parallelization and performance on translation tasks, and this paper became the foundation for virtually all modern LLMs, demonstrating that attention mechanisms alone can capture long-range dependencies effectively.
2018	Deep Contextualized Word Representations (ELMo)	Paper	Matthew E. Peters et al.	Demonstrated that word representations greatly improve if they are contextualized (dependent on the sentence) rather than static. ELMo used a bidirectional LSTM language model to produce word embeddings on the fly, yielding state-of-the-art results on many NLP tasks. This showed the importance of context in word meaning and was a precursor to transformer-based contextual models like BERT, proving the value of pre-trained language representations.
2018	Universal Language Model Fine-tuning for Text Classification (ULMFiT)	Paper	Jeremy Howard & Sebastian Ruder	Established the paradigm of pretrain-then-fine-tune for NLP (Understanding Large Language Models) (Understanding Large Language Models). ULMFiT pre-trained an LSTM language model on a large general corpus and fine-tuned it on target tasks, achieving excellent results in text classification. It was the first to show that transfer learning (so successful in vision) can work in NLP, kicking off the trend of large pretrained models (later transformers like BERT/GPT) being fine-tuned for downstream applications, which is now standard practice.
2018	BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding	Paper	Jacob Devlin et al. (Google)	Introduced BERT, a transformer encoder pre-trained via masked language modeling and next-sentence prediction (Understanding Large Language Models). BERT’s bidirectional training (conditioning on both left and right context) produced powerful language understandings and set new state-of-the-art on 11 NLP tasks. Its release popularized transformer encoder models for language understanding, and techniques like masked LM pretraining, ushering in the era of large “foundation” models for NLP (with many variants like RoBERTa, ALBERT, etc.).
2018	Improving Language Understanding by Generative Pre-Training (GPT)	Paper	Alec Radford et al. (OpenAI)	Presented the first Generative Pre-trained Transformer (GPT-1), a unidirectional transformer decoder pre-trained on unlabeled text to predict the next word (Understanding Large Language Models). GPT demonstrated the effectiveness of generative pretraining (without task-specific supervision) for improving downstream NLP tasks after fine-tuning. This work introduced the decoder-only transformer paradigm that underlies subsequent GPT-2/3 models and showed that pretraining on large text corpora learns versatile language representations.
2019	Language Models are Unsupervised Multitask Learners (GPT-2)	Paper	Alec Radford et al. (OpenAI)	Scaled up the GPT model to 1.5 billion parameters and showed it can perform many tasks in a zero-shot fashion by properly prompting it (without fine-tuning). GPT-2’s capacity to generate coherent paragraphs of text and handle tasks like translation or summarization by prompt alone illustrated the emergent zero/few-shot abilities of LLMs (Understanding Large Language Models). Its release also sparked discussion on the societal impact of powerful text generators (OpenAI initially withheld the full model), underscoring real-world considerations of LLM deployment.
2019	Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context	Paper	Zihang Dai et al. (CMU/Google)	Addressed the issue of fixed context lengths in transformers by introducing a recurrence mechanism across segments. Transformer-XL caches hidden states from previous segments and uses relative positional embeddings, enabling substantially longer context for language modeling. This achieved better performance on long text and discourse tasks, showing how architectural tweaks can push transformers beyond previous sequence-length limits – a critical step for LLMs to handle long documents.
2019	XLNet: Generalized Autoregressive Pretraining for Language Understanding	Paper	Zhilin Yang et al. (CMU/Google)	Proposed an alternative to BERT by using an autoregressive pretraining with permutation-based masking. XLNet learns from all parts of sentences (like BERT) but without corrupting input with masks, by predicting words in random order. It outperformed BERT on several tasks, highlighting a different way to leverage bidirectional context. XLNet’s success reinforced the value of innovative pretraining objectives and briefly held state-of-the-art, influencing further research into pretraining methods.
2019	BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension	Paper	Mike Lewis et al. (Facebook)	Combined BERT and GPT ideas in one encoder–decoder transformer (Understanding Large Language Models). BART is pretrained by corrupting text (with noise like token masking, deletion, permutation) and training the model to reconstruct the original text. By using a bidirectional encoder (like BERT) and an autoregressive decoder (like GPT), BART achieved strong results on text generation and understanding tasks. It showed the benefit of a unified seq2seq pretraining approach and became a foundation for many NLP tasks (e.g. summarization).
2019	Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer (T5)	Paper	Colin Raffel et al. (Google)	Introduced the T5 model and “text-to-text” framework, where all tasks are cast as text inputs to text outputs. T5 is a transformer trained on a massive dataset (C4) with a span-denoising objective (similar to fill-in-the-blank). It achieved state-of-the-art on a wide array of benchmarks using the same model and loss for all tasks. T5’s “unified” approach simplified NLP pipelines and demonstrated the versatility of large pretrained transformers, influencing multi-task learning and leading to variants like mT5 and instruction-tuned T5 (FLAN).
2020	Language Models are Few-Shot Learners (GPT-3)	Paper	Tom B. Brown et al. (OpenAI)	Presented GPT-3, a 175-billion parameter model that pushed the limits of scale and showed remarkable few-shot learning abilities (Understanding Large Language Models). Without any gradient updates, GPT-3 could perform tasks from translation to question answering by being given a few examples in its prompt, essentially learning from context. This emergent capability demonstrated the power of scale and surprised the community with how a single huge model could excel across many tasks. GPT-3’s impact was enormous – it became the backbone of many AI applications and inspired a race toward ever-larger models (as well as investigations into their limitations and capabilities).
2020	Scaling Laws for Neural Language Models	Paper	Jared Kaplan et al. (OpenAI)	Empirically discovered scaling laws that relate model size, dataset size, and compute to performance. Kaplan et al. showed that cross-entropy loss decreases as a power-law with increasing model parameters and training data, and identified regimes of diminishing returns. These insights made it clear that larger models do keep improving and informed predictions of performance at future scales (Understanding Large Language Models). This guided research and industry to focus on scaling up models (and datasets), and laid a theoretical foundation later refined by projects like Chinchilla.
2020	On Layer Normalization in the Transformer Architecture	Paper	Ruibin Xiong et al.	Investigated the best placement of Layer Normalization in transformers (Post-LN as in the original paper vs. Pre-LN). The authors found that applying layer norm before the attention and feed-forward sublayers (Pre-LN) leads to more stable training and better gradient flow (Understanding Large Language Models), whereas the original “Post-LN” design could suffer from training difficulties in deep transformers. This work resolved a discrepancy between the Transformer paper and its official code, influencing most modern transformer implementations to adopt the Pre-LN setup for improved training stability.
2021	Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity	Paper	William Fedus et al. (Google)	Demonstrated a Mixture-of-Experts (MoE) architecture to massively scale model size while keeping computation per token constant. By routing each input to one of many expert sub-networks (layers of feed-forward networks), Switch Transformers reached >1 trillion parameters effectively, yet only activated a fraction of those for each input. It achieved competitive results to dense models with far fewer FLOPs, validating MoE as a strategy for scaling LLMs. This influenced subsequent sparse models and showed a path to even bigger models beyond dense scaling limits.
2021	LoRA: Low-Rank Adaptation of Large Language Models	Paper	Edward J. Hu et al. (Microsoft)	Introduced a popular method for parameter-efficient fine-tuning of large models. LoRA injects a pair of low-rank trainable weight matrices into each layer of a pre-trained model and freezes the original weights (Understanding Large Language Models). This drastically reduces the number of parameters one needs to update (and store) to adapt an LLM to a new task. LoRA showed that large models have a low “intrinsic dimension” for task-specific adjustments, and it became widely used to fine-tune models like GPT-3, BERT, and others with limited compute, greatly lowering the barrier to customizing LLMs.
2022	Scaling Language Models: Methods, Analysis & Insights from Training Gopher	Paper	Jack W. Rae et al. (DeepMind)	Reported on Gopher, a 280B-parameter transformer, and provided extensive analysis of scaling effects. This work introduced refinements like RMSNorm (root mean square norm) in place of LayerNorm to stabilize very deep networks, and examined performance across 152 tasks (Understanding Large Language Models). They found that scaling improves performance in knowledge-intensive tasks (reading comprehension, fact-checking, toxicity identification) much more than in logical or arithmetic tasks (Understanding Large Language Models). These insights clarified where simply making models bigger helps versus where fundamental limitations remain, guiding research into areas like reasoning and prompting (since scaling alone wasn’t solving those as much).
2022	PaLM: Scaling Language Modeling with Pathways	Paper	Aakanksha Chowdhery et al. (Google)	Introduced PaLM, a 540-billion parameter transformer trained using Google’s Pathways system, and achieved breakthrough results on many benchmarks. PaLM’s sheer scale yielded emergent behaviors – for example, when prompted with chain-of-thought reasoning, a PaLM model (540B) achieved state-of-the-art on math word problems (GSM8K), outperforming even fine-tuned smaller models ([2201.11903] Chain-of-Thought Prompting Elicits Reasoning in Large Language Models). PaLM demonstrated the frontier of what purely scaling up can achieve, contributing an understanding of emergent capabilities (and also raising concerns about data and compute efficiency, later addressed by models like Chinchilla).
2022	Training Compute-Optimal Large Language Models (Chinchilla)	Paper	Jordan Hoffmann et al. (DeepMind)	Challenged the race for ever-larger models by showing many existing LMs were underequipped with data. The Chinchilla study posited that for a given compute budget, there is an optimal model size vs. training token trade-off. It introduced Chinchilla (70B params) which, being 4× smaller than GPT-3 but trained on 4× more tokens (1.4T tokens), outperformed the 175B GPT-3 on language tasks (Understanding Large Language Models). This “compute-optimal” approach reset the field’s direction: instead of just scaling parameters, ensure models see enough data. It has had real-world impact by guiding organizations to train moderate-sized models on larger corpora (rather than extremely large models on limited data) for better results.
2022	Chain-of-Thought Prompting Elicits Reasoning in Large Language Models	Paper	Jason Wei et al. (Google)	Demonstrated a simple yet powerful technique to boost LLM reasoning: chain-of-thought (CoT) prompting. By showing a model examples of step-by-step reasoning (breaking a problem into intermediate steps), the authors achieved dramatic improvements on complex tasks like arithmetic and commonsense reasoning ([2201.11903] Chain-of-Thought Prompting Elicits Reasoning in Large Language Models). Notably, a 540B model prompted with CoT solved math word problems at a level that surpassed even fine-tuned 175B models ([2201.11903] Chain-of-Thought Prompting Elicits Reasoning in Large Language Models). This revealed that sufficiently large LMs have latent reasoning abilities that can be unlocked via prompting alone. CoT prompting has since become a standard approach in applications requiring reasoning or explanation from LLMs.
2022	Training Language Models to Follow Instructions with Human Feedback (InstructGPT)	Paper	Long Ouyang et al. (OpenAI)	Marked a milestone in alignment by fine-tuning GPT-3 using Reinforcement Learning from Human Feedback (RLHF). The authors used human-written instruction–response pairs and preference rankings to train GPT-3 to produce helpful, obedient outputs (Understanding Large Language Models) (Understanding Large Language Models). The result, InstructGPT, was far more useful and less likely to produce unwanted content than the base model. This paper essentially unveiled the recipe behind ChatGPT – showing that large GPT models can be steered by human feedback to follow instructions and align with user intentions. It had immense real-world impact, as ChatGPT (built on these principles) exploded in popularity and demonstrated the practicality of RLHF-aligned LLMs.
2022	Constitutional AI: Harmlessness from AI Feedback	Paper	Yuntao Bai et al. (Anthropic)	Proposed an alternative approach to alignment using an AI feedback loop guided by a fixed set of human-written principles (a “constitution”). Instead of relying on extensive human reward labeling, the model itself generates and evaluates responses according to the constitution’s rules, and a reinforcement learning procedure (AI self-training) adjusts the model (Understanding Large Language Models). This method produced a more harmless AI assistant without direct human intervention for each example, illustrating a scalable way to align models with human values. The work is influential in exploring how to bake in ethical principles and safety through self-supervision, and Anthropically fine-tuned models (like Claude) use these ideas to reduce toxic or biased outputs.
2022	Self-Instruct: Aligning Language Models with Self-Generated Instructions	Paper	Yizhong Wang et al. (Allen Institute/UW)	Tackled the scarcity of instruction-response data by having the LM generate its own training data for instruction following. Starting with a small set of human-written prompts, the authors used a GPT-3 model to produce new instructions and example answers, filtering for quality, and iteratively grew a large training set (Understanding Large Language Models). Fine-tuning a model on this synthetic instruction data yielded impressive instruction-following performance, comparable to models trained on thousands of real human examples. Self-Instruct showed a path to bootstrap aligned instruction-tuned models (like ChatGPT-like behavior) with minimal human effort. This idea inspired many projects (e.g. Stanford’s Alpaca) to create ChatGPT-like models by leveraging existing LLMs to generate training data.
2022	Cramming: Training a Language Model on a Single GPU in One Day	Paper	Jonas Geiping & Tom Goldstein (University of Maryland)	Explored how far one can push efficient training to make language models accessible. The authors managed to train a BERT-sized model in 24 hours on a single GPU, using tricks to maximize memory and computational efficiency. They found that while smaller models train faster per step, larger models are more parameter-efficient learners – meaning a larger model can reach a given accuracy in about the same wall-clock time as a smaller one by virtue of learning more per step (Understanding Large Language Models). This work has practical impact for budget-limited scenarios, showing that with clever optimizations (learning rate schedules, initialization, etc.), one can “cram” LM training into limited hardware, and also highlighted that bigger models aren’t always slower to get good results if you can handle them in memory.
2022	FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness	Paper	Tri Dao et al. (Stanford)	Introduced an optimized algorithm to compute transformer self-attention significantly faster and using less memory, without approximations. FlashAttention reorders operations to better utilize GPU memory hierarchy and avoid redundant reads/writes, providing up to 10x speedups for the attention step. This method allows training with longer sequences and larger batches on the same hardware. Given that attention is the core of transformer models, FlashAttention’s improvements have been widely adopted in LLM training for better throughput (Understanding Large Language Models) (for example, PaLM, LLaMA, and others use or cite this technique), making large-model training and inference more efficient in practice.
2022	Scaling Down to Scale Up: A Guide to Parameter-Efficient Fine-Tuning	Paper	Sergey Lialin et al.	A comprehensive survey of parameter-efficient fine-tuning (PEFT) methods for LLMs (Understanding Large Language Models). It reviewed and compared 40+ techniques (e.g. adapters, prefix tuning, LoRA, etc.) that enable adapting large models with fewer trainable parameters. By systematizing these approaches, the paper provided clarity on when and how to use PEFT to reduce compute and memory costs. This guide has been influential for practitioners who need to fine-tune large models on specific tasks or domains, as it summarizes best practices to achieve strong results without updating every parameter. The survey’s impact is in accelerating the adoption of efficient fine-tuning in real-world LLM applications (e.g., customizing models like GPT-3, T5 for specific industries or languages).
2023	Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling	Paper	Stella Biderman et al. (EleutherAI)	Released a suite of open-source LLMs (70M–12B) along with their intermediate checkpoints to facilitate research on how model behavior evolves during training. Pythia allowed the community to study phenomena like memorization, scaling, and the effects of data duplication in a controlled way (Understanding Large Language Models). Key findings included that training on duplicate data (multiple epochs) did not significantly harm or help and that token frequency affected few-shot performance. While Pythia models are smaller than state-of-the-art, this project’s influence lies in transparency: it gave researchers a platform to dissect training dynamics of LLMs, which in turn helps guide the development of larger models (e.g., understanding at what point a model starts memorizing data or how scaling laws manifest across training).
2023	LLaMA: Open and Efficient Foundation Language Models	Paper	Hugo Touvron et al. (Meta AI)	Released a family of high-performance LLMs (7B, 13B, 33B, 65B parameters) trained on trillions of tokens and made available to researchers. LLaMA-13B, for instance, was shown to outperform GPT-3 (175B) on most benchmarks, and LLaMA-65B is competitive with DeepMind’s Chinchilla (70B) and Google’s PaLM (540B) despite fewer parameters (Review — LLaMA: Open and Efficient Foundation Language Models). By optimizing training (data quality, longer context, advanced tokenization), Meta achieved excellent results at smaller scale. The open release (under research license, though later leaked) of LLaMA had massive real-world impact: it sparked an explosion of innovation in the open-source community, from fine-tuned chatbots to domain-specific LLMs, effectively “democratizing” access to GPT-3-level models and accelerating LLM research and applications outside big tech labs.
2023	GPT-4 Technical Report	Paper	OpenAI (multiple authors)	Detailed the capabilities of GPT-4, a large-scale (estimated hundreds of billions of parameters) multimodal transformer that accepts text and image inputs. GPT-4 achieved human-level performance on many academic and professional benchmarks – for example, scoring in the top 10% of test-takers on the Uniform Bar Exam and demonstrating expert-level proficiency in biology, math, and coding challenges. It represents the cutting edge of aligned AI systems, incorporating advances in instruction following and safety (via RLHF and other techniques). GPT-4’s release had significant real-world impact, powering a new wave of applications (from advanced chatbots to aiding education and healthcare) and confirming that carefully scaling models and aligning them can produce AI systems with unprecedented capability and reliability.
2023	Harnessing the Power of LLMs in Practice: A Survey on ChatGPT and Beyond	Paper	Qingyang Yang et al.	Provided a broad survey of recent LLMs and their fine-tuning techniques, with a focus on ChatGPT and similar instruction-following models (Understanding Large Language Models). It charts the evolution of LLM architectures from the transformer through GPT-3 and beyond, and reviews how models like ChatGPT are built (pretraining, RLHF, instruction tuning). The survey also discusses practical considerations (efficiency, deployment, ethical concerns) and the emerging ecosystem of applications. By aggregating and distilling the fast-moving LLM landscape of 2022–2023, this paper became a go-to reference for practitioners and researchers to quickly get up to speed on state-of-the-art LLM technology and its implications, underscoring the field’s key developments and real-world usage.
Feb 2023	LLaMA: Open and Efficient Foundation Language Models	arXiv:2302.13971	Meta AI (Touvron et al.)	Introduced a suite of 7B–65B parameter LLMs trained on public data. Demonstrated that smaller models (e.g. LLaMA-13B) can outperform GPT-3 (175B) on many benchmarks, and LLaMA-65B is competitive with state-of-the-art models like Chinchilla-70B ([2302.13971] LLaMA: Open and Efficient Foundation Language Models). All models were released to researchers, spurring a wave of open LLM innovation.
Feb 2023	Toolformer: Language Models Can Teach Themselves to Use Tools	arXiv:2302.04761	Meta AI (Schick et al.)	Showed that LLMs can self-supervise the use of external tools via API calls. Toolformer learns when and how to call calculators, search engines, translators, etc., which greatly improves zero-shot performance on tasks requiring arithmetic or factual knowledge ([2302.04761] Toolformer: Language Models Can Teach Themselves to Use Tools). It achieves results competitive with much larger models, without sacrificing core language modeling abilities ([2302.04761] Toolformer: Language Models Can Teach Themselves to Use Tools).
Feb 2023	Language Is Not All You Need: Aligning Perception with Language Models (Kosmos-1)	arXiv:2302.14045	Microsoft Research (Huang et al.)	Introduced Kosmos-1, a multimodal LLM that integrates vision and language. Trained on web-scale data of interleaved text and images, Kosmos-1 can perform not only language tasks but also image captioning, visual QA, and even read text from images without OCR ([2302.14045] Language Is Not All You Need: Aligning Perception with Language Models). It was a first step toward general-purpose AI that “sees” and “reads,” showing cross-modal transfer improves both language and vision understanding.
Mar 2023	GPT-4 Technical Report	arXiv:2303.08774	OpenAI	Announced GPT-4, a large-scale multimodal model accepting text and images as input. GPT-4 achieved near human-level performance on many academic and professional benchmarks (e.g. scoring in the top 10% of a simulated bar exam) ([2303.08774] GPT-4 Technical Report). OpenAI’s report details its Transformer-based architecture and an alignment process via human feedback that improved factuality and adherence to desired behaviors.
May 2023	PaLM 2 Technical Report	arXiv:2305.10403	Google (Anil et al.)	Introduced PaLM 2, a state-of-the-art LM with greatly improved multilingual understanding and reasoning skills. PaLM 2 is more compute-efficient than its predecessor (PaLM) and shows significantly higher downstream task performance across model sizes ([2305.10403] PaLM 2 Technical Report). It achieved strong results on BIG-Bench reasoning tasks and features responsible AI improvements (e.g. controllable toxicity in outputs) ([2305.10403] PaLM 2 Technical Report).
May 2023	QLoRA: Efficient Finetuning of Quantized LLMs	arXiv:2305.14314	U. Washington / Meta (Dettmers et al.)	Proposed QLoRA, a technique to fine-tune large models using 4-bit quantization and Low-Rank Adapters. QLoRA allows a 65B model to be finetuned on a single 48GB GPU without performance loss ([2305.14314] QLoRA: Efficient Finetuning of Quantized LLMs). Using QLoRA, the authors produced the Guanaco family of models that reached 99.3% of ChatGPT’s quality on the Vicuna benchmark after just 24 hours of training (), dramatically lowering the cost of LLM specialization.
May 2023	The Falcon Series of Open Language Models	arXiv:2311.16867	TII UAE (Almazrouei et al.)	Open-sourced the Falcon models (7B, 40B, 180B) trained on high-quality RefinedWeb data. Notably, Falcon-40B, trained on 1T tokens, topped many open-source LLM benchmarks—outranking LLaMA-65B and other models on the HuggingFace leaderboard ([Technology Innovation Institute trains the state-of-the-art Falcon LLM 40B foundation model on Amazon SageMaker
May 2023	Introducing 100K Context Windows (Claude 2)	Anthropic Blog	Anthropic	Announced that Claude 2 can handle a 100,000-token context window (~75,000 words). This breakthrough allows the model to ingest and analyze hundreds of pages of text in a single query (Introducing 100K Context Windows \ Anthropic). In practice, Claude can read an entire novel or hours-long conversation and then answer questions or follow instructions across that extremely long context, vastly expanding LLM capabilities for long documents.
Jun 2023	Orca: Progressive Learning from Complex Explanation Traces of GPT-4	arXiv:2306.02707	Microsoft Research (Mukherjee et al.)	Introduced Orca, a 13B model taught to imitate GPT-4’s reasoning process. Orca learns from rich explanation traces and step-by-step examples generated by GPT-4, rather than just final answers ([2306.02707] Orca: Progressive Learning from Complex Explanation Traces of GPT-4). This “explanation distillation” yields remarkable gains in complex reasoning: Orca surpassed other 13B models by >100% on challenging benchmarks (Big-Bench Hard) and even reached parity with ChatGPT on those hard tasks ([2306.02707] Orca: Progressive Learning from Complex Explanation Traces of GPT-4).
Jul 2023	Llama 2: Open Foundation and Fine-Tuned Chat Models	arXiv:2307.09288	Meta AI (Touvron et al.)	Released Llama 2, ranging from 7B to 70B parameters, including chat-optimized versions. The fine-tuned Llama 2-Chat models were trained with human feedback and safety techniques, making them more helpful and harmless. They outperform other open-source chat models on most benchmarks, and human evaluators found Llama 2-Chat competitive with some closed models in helpfulness and safety ([2307.09288] Llama 2: Open Foundation and Fine-Tuned Chat Models). The models were released for both research and commercial use, greatly impacting the open LLM ecosystem.
Sep 2023	Textbooks Are All You Need II: phi-1.5 Technical Report	arXiv:2309.05463	Microsoft Research (Li et al.)	Showed that a small 1.3B model can exhibit capabilities of much larger LLMs by training on high-quality, synthesized data. phi-1.5 (an improved version of the earlier phi-1) was trained on “textbook quality” instruction data ([2309.05463] Textbooks Are All You Need II: phi-1.5 technical report), giving it strong common-sense reasoning and even coding abilities. It performs on par with models ~5× larger and outperforms most open models on complex tasks like grade-school math and basic coding ([2309.05463] Textbooks Are All You Need II: phi-1.5 technical report), highlighting the power of data quality over sheer size.
Oct 2023	Mistral 7B	arXiv:2310.06825	Mistral AI (Jiang et al.)	Announced a 7B-parameter model that outperforms LLaMA-2 13B on all evaluated benchmarks and even beats LLaMA 34B on reasoning, math, and code tasks ([2310.06825] Mistral 7B). Mistral 7B uses architectural optimizations like Grouped-Query Attention and Sliding Window attention for efficient inference ([2310.06825] Mistral 7B). Importantly, it was released under an open license (Apache 2.0), delivering top-tier performance in a small, accessible model.
Dec 2024	DeepSeek-V3 Technical Report	arXiv:2412.19437	DeepSeek-AI Research Team	Described DeepSeek-V3, a 671B-parameter Mixture-of-Experts LLM (with 37B active per token). It introduced Multi-head Latent Attention and an auxiliary-loss-free load balancing technique, enabling efficient training on 14.8 trillion tokens (DeepSeek-V3 Technical Report). DeepSeek-V3 achieved state-of-the-art performance among open models and reached parity with leading closed models on many benchmarks (DeepSeek-V3 Technical Report), despite using significantly less compute than previous top-tier LLMs.
Jan 2025	DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via RL	arXiv:2501.12948	DeepSeek-AI Research Team	Introduced DeepSeek-R1, a model focused on advanced reasoning through reinforcement learning. The team first trained R1-Zero purely via RL to induce chain-of-thought reasoning, then refined it with a “cold start” supervised stage to improve clarity (DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning). The final DeepSeek-R1 model achieves GPT-4-level performance on many reasoning benchmarks (math, coding, QA) (DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning). DeepSeek also open-sourced R1 and several smaller distilled models, contributing greatly to accessible, high-reasoning LLMs.