The Mathematics of DeepMind Models

Miquel Noguer i Alonso

Department Of Mathematics
Artificial Intelligence Finance Institute
November 9, 2024

Abstract
DeepMind has been at the forefront of artificial intelligence research, making groundbreaking ad-
vancements across various domains, including reinforcement learning, game-playing AI, protein struc-
ture prediction, generative modeling, neural architecture design, and natural language processing. This
extensive overview presents a comprehensive examination of DeepMind’s models, beginning with the
seminal ”Alpha” series—AlphaGo, AlphaGo Zero, AlphaZero, AlphaStar, and AlphaFold. Each model
is dissected in detail, highlighting their innovative architectures, training methodologies, and the math-
ematical principles underpinning their success.
The document delves into the mathematical formulations of these models, providing insights into how
deep learning and reinforcement learning techniques are combined with search algorithms like Monte
Carlo Tree Search (MCTS) to achieve superhuman performance in complex tasks. It explores how
AlphaGo leveraged policy and value networks to master the game of Go, how AlphaGo Zero advanced
this by learning from scratch without human data, and how AlphaZero generalized the approach to other
board games. AlphaStar’s extension to real-time strategy games and AlphaFold’s revolutionary impact
on protein structure prediction are also thoroughly analyzed.
By providing detailed explanations, mathematical formulations, practical insights, and discussions
on ethical and safety considerations, this comprehensive overview serves as a valuable resource for un-
derstanding DeepMind’s contributions to artificial intelligence and their profound impact on the field.
In October 2024, Demis Hassabis and John Jumper of Google DeepMind were awarded the Nobel Prize
in Chemistry for their development of AlphaFold, an artificial intelligence system capable of accurately
predicting protein structures.

Contents
1 Introduction 4

2 The Alpha Series 4

2.1 AlphaGo . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
2.1.1 Model Explanation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
2.1.2 Mathematical Formulation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
2.1.3 Significance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
2.2 AlphaGo Zero . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
2.2.1 Model Explanation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
2.2.2 Mathematical Formulation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
2.2.3 Significance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
2.3 AlphaZero . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
2.3.1 Model Explanation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
2.3.2 Mathematical Formulation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
2.3.3 Significance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
2.4 AlphaStar . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
2.4.1 Model Explanation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
2.4.2 Mathematical Formulation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7

1
 2.4.3 Significance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
2.5 AlphaFold . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
2.5.1 Model Explanation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
2.5.2 Mathematical Formulation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
2.5.3 Significance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8

3 Other Reinforcement Learning Models 9

3.1 Deep Q-Networks (DQN) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
3.1.1 Model Explanation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
3.1.2 Mathematical Formulation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
3.1.3 Significance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
3.2 Asynchronous Advantage Actor-Critic (A3C) . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
3.2.1 Model Explanation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
3.2.2 Mathematical Formulation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
3.2.3 Significance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
3.3 Soft Actor-Critic (SAC) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
3.3.1 Model Explanation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
3.3.2 Mathematical Formulation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
3.3.3 Significance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
3.4 MuZero . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
3.4.1 Model Explanation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
3.4.2 Mathematical Formulation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
3.4.3 Significance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12

4 Generative Models 12
4.1 Variational Autoencoders . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
4.1.1 VQ-VAE and VQ-VAE-2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
4.1.2 Significance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
4.2 Generative Adversarial Networks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
4.2.1 BigGAN . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
4.2.2 Significance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13

5 Neural Network Architectures 13

5.1 Attention Mechanisms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
5.1.1 Transformer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
5.1.2 Significance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
5.2 Memory-Augmented Networks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
5.2.1 Neural Turing Machines (NTM) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
5.2.2 Significance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
5.2.3 Differentiable Neural Computers (DNC) . . . . . . . . . . . . . . . . . . . . . . . . . . 14
5.2.4 Significance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15

6 Language Models 15
6.1 Gopher . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
6.1.1 Model Explanation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
6.1.2 Significance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
6.2 Chinchilla . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
6.2.1 Model Explanation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
6.2.2 Significance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
6.3 RETRO . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
6.3.1 Model Explanation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
6.3.2 Significance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16

2
7 Implementation Considerations 16
7.1 Hardware Requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
7.1.1 Compute Resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
7.2 Distributed Training Strategies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
7.2.1 Data Parallelism . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
7.2.2 Model Parallelism . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
7.2.3 Pipeline Parallelism . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
7.3 Memory Optimization Techniques . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
7.3.1 Gradient Checkpointing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
7.3.2 Mixed Precision Training . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
7.4 Practical Optimization Techniques . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
7.4.1 Hyperparameter Tuning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
7.4.2 Cost-Benefit Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17

8 Multi-Modal Capabilities 17
8.1 Vision-Language Models . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
8.1.1 Flamingo . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
8.1.2 Significance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
8.2 Cross-Modal Learning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
8.2.1 Joint Embedding Spaces . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
8.2.2 Contrastive Learning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
8.3 Architectural Considerations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
8.3.1 Modality-Specific Encoders . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
8.3.2 Fusion Strategies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18

9 Real-World Applications 18
9.1 Case Studies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
9.1.1 AlphaFold in Drug Discovery . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
9.1.2 Language Models in Healthcare . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
9.2 Deployment Challenges . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
9.2.1 Scalability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
9.2.2 Adaptation Strategies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
9.3 Performance Metrics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
9.3.1 Evaluation Frameworks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19

10 Ethics and Safety Considerations 19

10.1 Safety Mechanisms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
10.1.1 Alignment Techniques . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
10.1.2 Monitoring and Intervention . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
10.2 Ethical Considerations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
10.2.1 Bias and Fairness . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
10.2.2 Transparency and Explainability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20
10.3 Risk Mitigation Strategies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20
10.3.1 Adversarial Testing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20
10.3.2 Policy and Governance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20

11 Model Evaluation and Testing 20

11.1 Evaluation Methodologies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20
11.1.1 Benchmarking . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20
11.1.2 Ablation Studies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20
11.2 Testing Strategies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20
11.2.1 Unit Testing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20
11.2.2 Integration Testing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20
11.2.3 Validation Techniques . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20
11.3 Performance Metrics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21

3
 11.3.1 Quantitative Metrics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
11.3.2 Qualitative Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21

12 Model Compression and Efficiency 21

12.1 Quantization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
12.2 Pruning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
12.3 Knowledge Distillation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
12.4 Efficiency-Performance Trade-Offs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21

13 Integration and Interoperability 21

13.1 Integration Patterns . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
13.1.1 API-Based Integration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
13.1.2 Microservices Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
13.2 API Design Principles . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22
13.3 Interoperability Standards . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22

14 Conclusion 22

A Glossary of Terms 22

B References 22

1 Introduction
DeepMind has been a pioneer in artificial intelligence (AI) research, contributing significantly across various
domains such as reinforcement learning, game-playing AI, protein structure prediction, generative modeling,
neural architecture design, natural language processing, and more. This document provides an extensive
overview of DeepMind’s models, organized into categories for clarity. Each model is first explained in
detail, followed by its mathematical foundations, key contributions to the field, and discussions on practical
implementations, ethical considerations, and future directions.

2 The Alpha Series

The ”Alpha” series represents significant milestones in AI, showcasing breakthroughs in reinforcement learn-
ing, game-playing AI, and protein structure prediction. These models demonstrate the power of combining
deep learning with search algorithms and have had profound impacts across multiple domains.

2.1 AlphaGo
2.1.1 Model Explanation
AlphaGo was the first program to defeat a professional human Go player, marking a historic milestone in AI.
It combines deep neural networks with Monte Carlo Tree Search (MCTS) to evaluate positions and select
moves. The neural networks guide the search process, making it more efficient and effective.
Key innovations:

• Policy Network: Trained to predict the probability distribution over possible moves given a board
state. It narrows down the search space by focusing on promising moves.
• Value Network: Estimates the probability of winning from a given board state. It provides a heuristic
evaluation without simulating the entire game.

• Monte Carlo Tree Search (MCTS): An advanced search algorithm that explores possible moves
using the guidance of the policy and value networks.

4
2.1.2 Mathematical Formulation
Policy Network Training The policy network π(a | s; θp ) is trained using supervised learning on expert
human games to predict the next move a given a board state s:
X
Lpolicy = − log π(a | s; θp ), (1)
(s,a)

where (s, a) are state-action pairs from the dataset.

Value Network Training The value network v(s; θv ) is trained to predict the outcome z ∈ {−1, 1} (loss
or win) of games from positions s:
X 2
Lvalue = (v(s; θv ) − z) . (2)
s

Monte Carlo Tree Search (MCTS) MCTS uses simulations guided by the policy and value networks
to evaluate the potential of moves. The search algorithm balances exploration and exploitation using the
Upper Confidence Bounds for Trees (UCT) formula:
pP
b N (s, b)
Q(s, a) + U (s, a) = Q(s, a) + cpuct · π(a | s) · , (3)
1 + N (s, a)
where:

• Q(s, a): Estimated value of taking action a in state s.

• N (s, a): Visit count for action a in state s.
• cpuct : Exploration constant.

2.1.3 Significance
AlphaGo demonstrated that deep neural networks and reinforcement learning could master complex, intuitive
tasks previously thought to be uniquely human [Silver et al., 2016]. It spurred a new wave of research into
combining deep learning with search and planning algorithms.

2.2 AlphaGo Zero

2.2.1 Model Explanation
AlphaGo Zero represents a significant advancement over AlphaGo by learning to play Go entirely from
scratch, without any human data or prior knowledge beyond the game rules. It starts with random play
and improves solely through self-play reinforcement learning. It employs a single neural network to estimate
both the policy and value functions, streamlining the architecture.
Key features:

• Self-Play: The agent plays against itself, continually improving by learning from the outcomes.
• Unified Network Architecture: Combines policy and value estimation into a single network, re-
ducing complexity.
• Tabula Rasa Learning: Starts without human biases or preconceptions, potentially discovering novel
strategies.

2.2.2 Mathematical Formulation

At each time step, the agent uses MCTS guided by the current neural network to select moves.

5
Loss Function The neural network parameters θ are updated to minimize the loss:
2
L(θ) = (z − v(s; θ))2 − π(a | s)⊤ log p(a | s; θ) + c ∥θ∥ , (4)
where:

• z is the game outcome from self-play (+1 for a win, −1 for a loss).
• v(s; θ) is the predicted value of state s.
• π(a | s) is the improved policy from MCTS (visit counts normalized).
• p(a | s; θ) is the policy output from the neural network.
2
• c ∥θ∥ is a regularization term to prevent overfitting.

Monte Carlo Tree Search Adaptations The MCTS in AlphaGo Zero uses the neural network to
evaluate leaf nodes, replacing the need for rollouts. The selection policy within MCTS is based on the
PUCT (Predictor + UCT) algorithm, which balances exploration and exploitation.

2.2.3 Significance
AlphaGo Zero surpassed all previous versions of AlphaGo and demonstrated that superhuman performance
in complex domains could be achieved without human data [Silver et al., 2017]. It revealed that reinforcement
learning and self-play could lead to the discovery of new knowledge and strategies.

2.3 AlphaZero
2.3.1 Model Explanation
AlphaZero generalizes the approach used in AlphaGo Zero to other two-player, perfect-information games
such as Chess and Shogi. It uses the same algorithm and neural network architecture across different games,
emphasizing the generality of the method.
Key features:

• Game-Agnostic Algorithm: Applies the same learning process to different games without game-
specific modifications.

• Self-Play Reinforcement Learning: Continually improves by playing against itself, learning from
successes and failures.
• Unified Network and MCTS: Integrates the policy and value networks into a single model used
within MCTS.

2.3.2 Mathematical Formulation

AlphaZero uses the same loss function and MCTS adaptations as AlphaGo Zero but applies them to different
game environments.

Loss Function The loss function remains:

2
L(θ) = (z − v(s; θ))2 − π(a | s)⊤ log p(a | s; θ) + c ∥θ∥ . (5)

Policy Update The policy is updated to approximate the improved policy derived from MCTS, encour-
aging the neural network to focus on moves that are promising according to search.

6
2.3.3 Significance
AlphaZero achieved superhuman performance in Chess and Shogi, defeating top traditional programs like
Stockfish and Elmo [Silver et al., 2018]. It demonstrated the potential for a single algorithm to master
multiple complex domains, highlighting the power of general reinforcement learning methods.

2.4 AlphaStar
2.4.1 Model Explanation
AlphaStar extends the principles of the Alpha series to real-time strategy games, specifically StarCraft II.
This environment introduces additional challenges such as imperfect information, real-time decision-making,
and the need for long-term strategic planning.
Key components:

• Neural Network Architecture: Processes raw game observations, including spatial and non-spatial
features, using a combination of convolutional and transformer networks.
• Multi-Agent Learning: Trains a league of agents with diverse strategies to promote robustness and
prevent overfitting to specific tactics.

• Reinforcement Learning with Imitation: Combines supervised learning from human replays with
reinforcement learning to accelerate training.

2.4.2 Mathematical Formulation

Policy Gradient with Baseline The policy is trained using a variant of the policy gradient method,
maximizing expected rewards while using a baseline to reduce variance:
" #
X
∇θ J(θ) = Eτ ∼πθ ∇θ log πθ (at | st ) (Rt − b(st )) , (6)
t

where:

• τ is a trajectory generated by the policy πθ .

• Rt is the return from time t.

• b(st ) is a baseline function, often the value function V (st ; θv ).

Value Function Training The value function is trained to minimize the temporal-difference (TD) error:
h i
2
Lvalue (θv ) = Est ,Rt (V (st ; θv ) − Rt ) . (7)

League Training AlphaStar’s league training involves multiple agents with different roles (main agents,
exploiters, and league exploiters) to ensure diversity. The objective is to minimize the exploitability of agents
within the league.

2.4.3 Significance
AlphaStar achieved Grandmaster level in StarCraft II, outperforming professional human players [Vinyals
et al., 2019]. It demonstrated that deep reinforcement learning could handle complex, dynamic environments
with imperfect information, significantly advancing the field.

7
2.5 AlphaFold
2.5.1 Model Explanation
AlphaFold addresses the long-standing ”protein folding problem” by predicting the three-dimensional struc-
ture of proteins from their amino acid sequences. Accurate protein structure prediction has immense impli-
cations for biology and medicine.
Key innovations:

• Attention Mechanisms: Uses attention to capture complex interactions between amino acids.
• Evoformer Module: Processes multiple sequence alignments (MSAs) and templates to incorporate
evolutionary information.
• Structure Module: Generates the 3D coordinates of atoms in the protein structure.

2.5.2 Mathematical Formulation

Input Representation AlphaFold uses an input representation consisting of:

• Sequence Features: One-hot encoding of amino acid sequences.

• Multiple Sequence Alignments (MSAs): Representations capturing evolutionary relationships.
• Template Structures: Information from known related protein structures.

Evoformer Module Processes the inputs using blocks of attention layers:

QK ⊤
 
Attention(Q, K, V ) = softmax √ V, (8)
d
where Q, K, and V are query, key, and value matrices derived from the input representations.

Structure Module Predicts the 3D coordinates by iterative refinement, minimizing a potential function:
X 2
E(R) = wij (∥ri − rj ∥ − dij ) , (9)
i<j

where:

• R = {ri } are the atomic coordinates.

• dij are the predicted distances between residues.
• wij are weights reflecting confidence.

Loss Function The loss function includes terms for distance and angle predictions, violation penalties,
and alignment with experimental data:

L = Ldist + Langle + Lviolations + Lexp . (10)

2.5.3 Significance
AlphaFold achieved unprecedented accuracy in protein structure prediction, significantly outperforming pre-
vious methods [Jumper et al., 2021]. It has accelerated research in drug discovery, understanding diseases,
and biological functions, marking a transformative moment in computational biology.

8
3 Other Reinforcement Learning Models
Beyond the Alpha series, DeepMind has developed numerous other models that have advanced the field of
reinforcement learning (RL), introducing new algorithms, architectures, and applications.

3.1 Deep Q-Networks (DQN)

3.1.1 Model Explanation
The Deep Q-Network (DQN) algorithm combines Q-learning with deep neural networks to approximate the
optimal action-value function in high-dimensional state spaces. It was a significant breakthrough, enabling
RL algorithms to learn directly from raw pixel inputs in environments like Atari games.
Key innovations include:

• Experience Replay: Stores past experiences (s, a, r, s′ ) in a replay memory. Sampling mini-batches
from this memory breaks correlations between sequential observations and stabilizes training.

• Target Networks: Uses a separate target network to compute target Q-values, which is updated less
frequently. This reduces oscillations and divergence during training.

3.1.2 Mathematical Formulation

Bellman Equation The optimal action-value function Q∗ (s, a) satisfies the Bellman optimality equation:
h i
Q∗ (s, a) = Es′ r + γ max
′
Q∗ ′ ′
(s , a ) | s, a , (11)
a

where:

• s is the current state.

• a is the action taken.

• r is the reward received.
• s′ is the next state.
• γ is the discount factor.

Loss Function The loss function minimized during training is:

h i
2
L(θ) = E(s,a,r,s′ )∼D (y − Q(s, a; θ)) , (12)

with the target y defined as:

y = r + γ max
′
Q(s′ , a′ ; θ− ), (13)
a

where θ are the parameters of the online network and θ− are the parameters of the target network.

3.1.3 Significance
DQN was the first algorithm to achieve human-level performance on a suite of challenging control tasks,
demonstrating the power of combining deep learning with reinforcement learning [Mnih et al., 2015].

9
3.2 Asynchronous Advantage Actor-Critic (A3C)
3.2.1 Model Explanation
A3C is an actor-critic algorithm that uses asynchronous gradient descent for policy and value function
updates. Multiple agents run in parallel, each interacting with its own copy of the environment, which
stabilizes and accelerates training.
Key features:

• Asynchronous Updates: Agents update shared global parameters asynchronously, reducing the
correlation between data and stabilizing training.
• Advantage Function: Uses the advantage function to reduce variance in policy gradient updates.
• Entropy Regularization: Encourages exploration by adding an entropy term to the objective func-
tion.

3.2.2 Mathematical Formulation

Policy Gradient The policy parameters θ are updated using:

∇θ = ∇θ log π(at | st ; θ)A(st , at ) + β∇θ H(π(st ; θ)), (14)

where:

• A(st , at ) = Rt − V (st ; θv ) is the advantage function.

• Rt is the discounted return from time t.
• V (st ; θv ) is the value estimate.

• H(π(st ; θ)) is the entropy of the policy at state st .

• β is the entropy regularization coefficient.

Value Function Update The value function parameters θv are updated to minimize:
2
Lvalue (θv ) = (Rt − V (st ; θv )) . (15)

3.2.3 Significance
A3C achieved state-of-the-art results on various benchmarks, including Atari games and continuous control
tasks, while being computationally efficient and scalable [Mnih et al., 2016].

3.3 Soft Actor-Critic (SAC)

3.3.1 Model Explanation
SAC is an off-policy actor-critic algorithm designed for continuous action spaces. It maximizes a trade-
off between expected return and policy entropy, promoting exploration by encouraging stochasticity in the
policy.
Key features:

• Maximum Entropy Framework: Incorporates an entropy term into the objective, balancing explo-
ration and exploitation.
• Stochastic Policies: Learns stochastic policies that can capture multiple modes of optimal behavior.

• Sample Efficiency: Being off-policy, it can reuse experience data effectively.

10
3.3.2 Mathematical Formulation
Objective Function The policy aims to maximize:
X
J(π) = E(st ,at )∼ρπ [r(st , at ) + αH(π(· | st ))] , (16)
t
where:
• ρπ is the state-action marginal induced by policy π.
• H(π(· | st )) is the entropy of the policy at state st .
• α is the temperature parameter controlling the trade-off.

Policy Update The policy is updated by minimizing the Kullback-Leibler divergence between the policy
and an exponentiated advantage:

!
exp α1 Q(st , ·)
πnew = arg min DKL π(· | st )∥ , (17)
π Z(st )
where Z(st ) is a normalizing constant.

3.3.3 Significance
SAC achieves state-of-the-art performance on continuous control benchmarks with high sample efficiency
and stability [Haarnoja et al., 2018].

3.4 MuZero
3.4.1 Model Explanation
MuZero builds upon the successes of AlphaZero by learning both the environment model and the policy/value
functions from raw observations. It is capable of planning in environments without known dynamics.
Key components:
• Representation Function h: Encodes the observation ot into a hidden state s0 = h(o0 ).
• Dynamics Function g: Predicts the next hidden state and immediate reward (sk+1 , rk ) = g(sk , ak ).
• Prediction Function f : Outputs the policy and value estimate (pk , vk ) = f (sk ).

3.4.2 Mathematical Formulation

Loss Function The loss function combines multiple components:
K
X 2
ℓvalue (vk , zk ) + ℓpolicy (pk , πk ) + ℓreward (rk , rktarget ) + c ∥θ∥ ,
 
L(θ) = (18)
k=0
where:
• vk is the predicted value at step k.
• zk is the target value (e.g., n-step return).
• pk is the predicted policy.
• πk is the target policy from MCTS.
• rk is the predicted reward.
• rktarget is the actual reward.
• ℓ denotes the loss for each component (e.g., mean squared error, cross-entropy).

11
Monte Carlo Tree Search MuZero uses the learned dynamics model within MCTS to simulate future
states and evaluate actions, effectively planning using its own understanding of the environment.

3.4.3 Significance
MuZero demonstrates that an agent can achieve high performance in complex environments without prior
knowledge of the dynamics, highlighting the potential of model-based reinforcement learning [Schrittwieser
et al., 2020].

4 Generative Models
Generative models aim to learn the underlying distribution of data to generate new, realistic samples.
DeepMind has developed several influential generative models that have advanced the state of the art in
image and audio synthesis.

4.1 Variational Autoencoders

4.1.1 VQ-VAE and VQ-VAE-2
Model Explanation Vector Quantized Variational Autoencoders (VQ-VAE) introduce discrete latent
variables by incorporating a codebook of embeddings. This allows for more powerful generative models that
can be combined with autoregressive models to capture complex data distributions.
Key features:

• Discrete Latent Space: Uses vector quantization to map continuous embeddings to discrete codes.
• Codebook Learning: The codebook embeddings are learned during training.

• Hierarchical Modeling: VQ-VAE-2 introduces a hierarchy of latent variables to model data at

multiple scales.

Mathematical Formulation
1. Encoder: Maps input x to latent representation ze = E(x).
2. Quantization: Maps ze to the nearest codebook vector zq .

3. Decoder: Reconstructs x from zq .

The loss function includes:

2 2 2
L = ∥x − D(zq )∥ + ∥sg[ze ] − zq ∥ + β ∥ze − sg[zq ]∥ , (19)

where:

• D is the decoder.
• sg denotes the stop-gradient operator.
• β is a hyperparameter balancing the commitment loss.

4.1.2 Significance
VQ-VAE models can generate high-fidelity images and have been used in state-of-the-art speech synthesis
systems [Van Den Oord et al., 2017]. They enable powerful autoregressive models to be applied over the
discrete latent space instead of raw data, improving efficiency.

12
4.2 Generative Adversarial Networks
4.2.1 BigGAN
Model Explanation BigGAN scales up Generative Adversarial Networks (GANs) to achieve high-fidelity
image synthesis. It introduces techniques like class-conditional batch normalization and the truncation trick
to improve sample quality.
Key features:

• Large-Scale Training: Utilizes large batch sizes and model capacities.

• Class Conditioning: Incorporates class labels to generate images of specific categories.
• Truncation Trick: Controls the trade-off between sample fidelity and variety by adjusting the sam-
pling distribution of latent variables.

Mathematical Formulation

Discriminator Loss Uses the hinge loss:

LD = Ex∼pdata [max(0, 1 − D(x, y))] + Ez∼pz [max(0, 1 + D(G(z, y), y))], (20)

Generator Loss

LG = −Ez∼pz [D(G(z, y), y)], (21)

where:

• D is the discriminator.
• G is the generator.
• z is the latent vector.
• y is the class label.

4.2.2 Significance
BigGAN achieves state-of-the-art image generation on datasets like ImageNet, demonstrating the potential
of large-scale GANs [Brock et al., 2019].

5 Neural Network Architectures

DeepMind has contributed to the development of novel neural network architectures that have significantly
influenced AI research.

5.1 Attention Mechanisms

5.1.1 Transformer
Model Explanation Transformers utilize self-attention mechanisms to process sequences, allowing for
parallel computation and capturing long-range dependencies without the need for recurrence. They have
become foundational in natural language processing.

13
Mathematical Formulation The scaled dot-product attention is defined as:

QK ⊤
 
Attention(Q, K, V ) = softmax √ V, (22)
dk
where:

• Q, K, V are query, key, and value matrices.

• dk is the dimensionality of the key vectors.

Multi-Head Attention Combines multiple attention mechanisms:

MultiHead(Q, K, V ) = Concat(head1 , . . . , headh )W O , (23)

where each head is an attention function.

5.1.2 Significance
Transformers have revolutionized NLP and have been extended to other domains like vision and reinforcement
learning [Vaswani et al., 2017].

5.2 Memory-Augmented Networks

5.2.1 Neural Turing Machines (NTM)
Model Explanation NTMs augment neural networks with external memory resources that can be read
from and written to, allowing them to simulate algorithmic behaviors and manipulate data structures.

Mathematical Formulation

Memory Read X
read
rt = wt,i Mt,i , (24)
i

Memory Write
write write


Mt,i = Mt−1,i 1 − wt,i et + wt,i at , (25)
where:

• Mt,i is the memory content at position i and time t.

• wt,i
read write
and wt,i are read and write weights.
• et is the erase vector.
• at is the add vector.

5.2.2 Significance
NTMs can learn tasks requiring external memory and sequential processing, such as copying and sorting
[Graves et al., 2014].

5.2.3 Differentiable Neural Computers (DNC)

Model Explanation DNCs enhance NTMs by improving memory addressing mechanisms, including tem-
poral links and dynamic memory allocation, allowing for more complex data structures and reasoning tasks.

Mathematical Formulation

14
Temporal Memory Links Captures temporal ordering:
write write write write
Lt,i,j = (1 − wt,i − wt,j )Lt−1,i,j + wt−1,i wt,j , (26)

Usage Vector Tracks memory usage:

ut = ut−1 + wtwrite − ut−1 ⊙ wtwrite ,

(27)
where ⊙ denotes element-wise multiplication.

5.2.4 Significance
DNCs can solve complex tasks like graph traversal and question answering that require flexible memory
usage [Graves et al., 2016].

6 Language Models
DeepMind has developed large-scale language models that contribute significantly to natural language pro-
cessing (NLP).

6.1 Gopher
6.1.1 Model Explanation
Gopher is a Transformer-based language model with up to 280 billion parameters, trained on a diverse
dataset to achieve strong performance across a wide range of NLP tasks.

6.1.2 Significance
Gopher demonstrates that scaling up models leads to improvements in tasks such as reading comprehension,
reasoning, and knowledge recall [Rae et al., 2021].

6.2 Chinchilla
6.2.1 Model Explanation
Chinchilla is a compute-optimal language model that balances model size and training data. By following
compute-optimal scaling laws, Chinchilla achieves better performance than larger models trained on less
data.

6.2.2 Significance
Chinchilla challenges the notion that simply increasing model size yields the best performance, emphasizing
the importance of sufficient training data [Hoffmann et al., 2022].

6.3 RETRO
6.3.1 Model Explanation
RETRO (Retrieval-Enhanced Transformer) integrates a retrieval mechanism into the Transformer architec-
ture, allowing the model to access external documents during generation.

Retrieval Mechanism During training and inference, the model retrieves relevant text passages from a
large database based on the current context.

15
Mathematical Formulation The model computes:
Y
P (y | x) = P (yt | y<t , retrievedt ), (28)
t
where retrievedt are the documents retrieved at time t.

6.3.2 Significance
RETRO improves language modeling by incorporating information from trillions of tokens without increasing
model size significantly [Borgeaud et al., 2022].

7 Implementation Considerations
Implementing large-scale models involves addressing challenges related to computational resources, memory
limitations, efficient training strategies, and practical optimization techniques.

7.1 Hardware Requirements

7.1.1 Compute Resources
Training large models requires significant computational power, often involving clusters of GPUs or TPUs.
Considerations include:

• Memory Capacity: To accommodate large model parameters and batch sizes.

• Compute Throughput: For efficient training within reasonable time frames.
• Interconnect Bandwidth: High-speed communication between devices is critical for distributed
training.

7.2 Distributed Training Strategies

7.2.1 Data Parallelism
Explanation Each device holds a copy of the model, and different mini-batches of data are processed in
parallel. Gradients are aggregated and averaged across devices.

7.2.2 Model Parallelism

Explanation The model is partitioned across multiple devices. Layers or parts of layers are assigned to
different devices to handle models that exceed single-device memory limits.

7.2.3 Pipeline Parallelism

Explanation Combines data and model parallelism by dividing the model into stages, with each stage
processed by a different device. Micro-batches are used to keep all devices busy.

7.3 Memory Optimization Techniques

7.3.1 Gradient Checkpointing
Explanation Reduces memory usage by not storing all intermediate activations during the forward pass.
Instead, recomputes them during the backward pass as needed.

7.3.2 Mixed Precision Training

Explanation Uses lower-precision data types (e.g., FP16) for computations, reducing memory footprint
and increasing computational speed. Care must be taken to maintain numerical stability.

16
7.4 Practical Optimization Techniques
7.4.1 Hyperparameter Tuning
Efficient hyperparameter tuning strategies include:

• Bayesian Optimization: Models the objective function and selects hyperparameters that are ex-
pected to yield better performance.

• Population Based Training (PBT): Simultaneously optimizes hyperparameters and model param-
eters by evolving a population of models.
• Grid and Random Search: Systematic or random exploration of hyperparameter spaces.

7.4.2 Cost-Benefit Analysis

Balancing computational cost against performance gains involves:

• Efficiency Metrics: Measuring training and inference efficiency in terms of FLOPS, energy consump-
tion, and wall-clock time.

• Performance Metrics: Evaluating model accuracy, generalization, and robustness.

8 Multi-Modal Capabilities
Integrating multiple modalities, such as text, images, and audio, enables models to understand and generate
rich, context-aware content.

8.1 Vision-Language Models

8.1.1 Flamingo
Model Explanation Flamingo is a visual language model that combines images and text through cross-
attention mechanisms. It can perform tasks like image captioning, visual question answering, and image-
based dialogue.

Model Architecture Extends the Transformer architecture with:

• Gated Cross-Attention Layers: Integrate visual features into the language model.
• Perceiver Resampler: Processes high-dimensional visual inputs into a fixed-size representation.

8.1.2 Significance
Flamingo achieves strong performance on few-shot learning tasks across various vision-language benchmarks
[Alayrac et al., 2022].

8.2 Cross-Modal Learning

8.2.1 Joint Embedding Spaces
Explanation Learning shared representations where data from different modalities are mapped into the
same space, facilitating cross-modal retrieval and understanding.

8.2.2 Contrastive Learning

Explanation Uses contrastive loss functions to bring representations of corresponding modalities closer
while pushing apart non-corresponding pairs.

17
8.3 Architectural Considerations
8.3.1 Modality-Specific Encoders
Explanation Employing specialized encoders (e.g., CNNs for images, Transformers for text) to extract
modality-specific features before fusion.

8.3.2 Fusion Strategies

Early Fusion Combines modalities at the input level, feeding concatenated data into the model.

Late Fusion Processes each modality separately and combines outputs at a higher level.

Hierarchical Fusion Integrates modalities at multiple levels within the model to capture interactions at
different granularities.

9 Real-World Applications
Deploying AI models in real-world scenarios involves practical considerations, performance evaluations, and
addressing domain-specific challenges.

9.1 Case Studies

9.1.1 AlphaFold in Drug Discovery
AlphaFold’s accurate protein structure predictions enable:

• Target Identification: Understanding protein functions and interactions.

• Structure-Based Drug Design: Developing molecules that interact with specific protein sites.
• Disease Mechanism Elucidation: Exploring the molecular basis of diseases.

9.1.2 Language Models in Healthcare

Applications include:

• Medical Record Analysis: Extracting insights from unstructured clinical notes.

• Patient Communication: Assisting in answering patient queries.

• Diagnostic Support: Providing recommendations based on symptom descriptions.

9.2 Deployment Challenges

9.2.1 Scalability
Ensuring models can handle:

• High Throughput: Serving a large number of requests concurrently.

• Low Latency: Providing fast responses for real-time applications.
• Resource Management: Optimizing computational and memory resources.

18
9.2.2 Adaptation Strategies
Methods include:

• Fine-Tuning: Adapting pre-trained models to specific domains or tasks.

• Transfer Learning: Leveraging knowledge from related tasks.
• Continuous Learning: Updating models with new data over time.

9.3 Performance Metrics

9.3.1 Evaluation Frameworks
Assessing models using:

• Accuracy Measures: Task-specific metrics like BLEU scores, F1 scores, etc.

• Robustness Tests: Evaluating performance under adversarial conditions or data shifts.

• User Satisfaction: Collecting feedback from end-users to gauge effectiveness.

10 Ethics and Safety Considerations

Ensuring that AI models are developed and deployed responsibly involves addressing ethical concerns and
implementing safety mechanisms.

10.1 Safety Mechanisms

10.1.1 Alignment Techniques
Methods to align model outputs with human values:

• Reinforcement Learning from Human Feedback (RLHF): Training models using feedback from
human evaluators.
• Rule-Based Constraints: Enforcing hard constraints to prevent undesirable outputs.

10.1.2 Monitoring and Intervention

Implementing:

• Content Filters: Detecting and filtering inappropriate content.

• Human Oversight: Incorporating human-in-the-loop systems for critical decisions.

10.2 Ethical Considerations

10.2.1 Bias and Fairness
Addressing:

• Data Biases: Ensuring training data is representative and balanced.

• Algorithmic Fairness: Designing models that do not perpetuate or amplify biases.

19
10.2.2 Transparency and Explainability
Providing:

• Model Interpretability: Enabling understanding of how models make decisions.

• Documentation: Clearly explaining model capabilities and limitations.

10.3 Risk Mitigation Strategies

10.3.1 Adversarial Testing
Conducting:

• Security Audits: Identifying vulnerabilities to adversarial attacks.

• Stress Testing: Evaluating model performance under extreme conditions.

10.3.2 Policy and Governance

Establishing:

• Ethical Guidelines: Defining principles for responsible AI development.

• Regulatory Compliance: Adhering to laws and regulations in relevant jurisdictions.

11 Model Evaluation and Testing

Robust evaluation and testing methodologies are crucial for assessing model performance and ensuring reli-
ability.

11.1 Evaluation Methodologies

11.1.1 Benchmarking
Using standardized datasets and tasks to compare models.

11.1.2 Ablation Studies

Systematically removing or altering components to assess their impact.

11.2 Testing Strategies

11.2.1 Unit Testing
Testing individual components for correctness.

11.2.2 Integration Testing

Ensuring that different components work together as intended.

11.2.3 Validation Techniques

• Cross-Validation: Evaluating models on different subsets of data.
• Holdout Sets: Reserving data for final evaluation.

20
11.3 Performance Metrics
11.3.1 Quantitative Metrics
Measuring accuracy, precision, recall, F1-score, ROC-AUC, etc.

11.3.2 Qualitative Analysis

Human evaluation of outputs, error analysis, and case studies.

12 Model Compression and Efficiency

Optimizing models for deployment involves reducing resource requirements without significantly sacrificing
performance.

12.1 Quantization
Explanation Reducing the precision of model parameters (e.g., from 32-bit floats to 8-bit integers) to
decrease memory usage and increase computational efficiency.

12.2 Pruning
Explanation Removing redundant or less important weights or neurons from the network based on criteria
such as magnitude or contribution to loss.

12.3 Knowledge Distillation

Explanation Training a smaller ”student” model to mimic the behavior of a larger ”teacher” model, often
by matching output distributions.

12.4 Efficiency-Performance Trade-Offs

Analyzing:

• Latency vs. Accuracy: Balancing speed with model performance.

• Resource Constraints: Adapting models to hardware limitations.

13 Integration and Interoperability

Ensuring models can be integrated into existing systems and work seamlessly with other components.

13.1 Integration Patterns

13.1.1 API-Based Integration
Exposing model functionalities through APIs for use by other applications.

13.1.2 Microservices Architecture

Deploying models as independent services that communicate over network protocols.

21
13.2 API Design Principles
• Consistency: Uniform interfaces and response formats.
• Versioning: Managing changes without disrupting clients.

• Security: Implementing authentication and authorization.

13.3 Interoperability Standards

Adhering to standards such as:

• ONNX: Open Neural Network Exchange format for model representation.

• TensorFlow Serving: Serving models using standardized protocols.

14 Conclusion
DeepMind’s contributions to AI encompass a wide range of models and technologies, from the ground-
breaking Alpha series to advanced language and generative models. By extensively exploring these models’
mathematical foundations, practical implementations, ethical considerations, and future directions, we gain
a comprehensive understanding of their impact and potential. Continued research and responsible develop-
ment are essential to harness the full benefits of AI while mitigating risks. In October 2024, Demis Hassabis
and John Jumper of Google DeepMind were awarded the Nobel Prize in Chemistry for their development of
AlphaFold, an artificial intelligence system capable of accurately predicting protein structures.

A Glossary of Terms
• Reinforcement Learning (RL): A learning paradigm where agents learn to make decisions by
interacting with an environment to maximize cumulative rewards.
• Policy Network: A neural network that outputs a probability distribution over possible actions.

• Value Network: A neural network that estimates the expected return from a given state.
• Monte Carlo Tree Search (MCTS): A heuristic search algorithm for decision processes that uses
random sampling and tree structures.
• Transformer: A neural network architecture based on self-attention mechanisms, widely used in NLP.

• Entropy Regularization: A technique to encourage exploration by adding an entropy term to the

objective function.
• Upper Confidence Bounds for Trees (UCT): A method used in MCTS to balance exploration
and exploitation.

• Perceiver Resampler: A module for processing high-dimensional inputs by mapping them to a

fixed-size representation.

B References
References
Jean-Baptiste Alayrac, Jeff Donahue, Paul Luc, Antoine Miech, Ian Barr, Yana Hasson, Thomas Leute,
Katie Millican, Mickael Rouvier, Trevor Ryder, et al. Flamingo: a visual language model for few-shot
learning. arXiv preprint arXiv:2204.14198, 2022.

22
Sebastian Borgeaud, Arthur Mensch, Jordan Hoffmann, Trevor Cai, Ethan Rutherford, Katie Millican,
George van den Driessche, Jean-Baptiste Lespiau, Bogdan Damoc, Aidan Clark, et al. Improving language
models by retrieving from trillions of tokens. arXiv preprint arXiv:2112.04426, 2022.
Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale gan training for high fidelity natural image
synthesis. In International Conference on Learning Representations, 2019.
Alex Graves, Greg Wayne, and Ivo Danihelka. Neural turing machines. arXiv preprint arXiv:1410.5401,
2014.
Alex Graves, Greg Wayne, Malcolm Reynolds, Tim Harley, Ivo Danihelka, Agnieszka Grabska-Barwińska,
Sergio Gómez Colmenarejo, Edward Grefenstette, Tiago Ramalho, et al. Hybrid computing using a neural
network with dynamic external memory. Nature, 538(7626):471–476, 2016.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum
entropy deep reinforcement learning with a stochastic actor. In International Conference on Machine
Learning, pages 1861–1870, 2018.
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Katie Millican, Trevor Cai, Ethan Rutherford,
Danila Casas, Aurelia Guy, Simon Osindero, Karen Simonyan, et al. Training compute-optimal large
language models. arXiv preprint arXiv:2203.15556, 2022.
John Jumper, Richard Evans, Alexander Pritzel, Tim Green, Michael Figurnov, Olaf Ronneberger, Kathryn
Tunyasuvunakool, Russ Bates, Augustin Žı́dek, Anna Potapenko, et al. Highly accurate protein structure
prediction with alphafold. Nature, 596(7873):583–589, 2021.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex
Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, and Stig Petersen. Human-level control
through deep reinforcement learning. Nature, 518(7540):529–533, 2015.
Volodymyr Mnih, Adrià Puigdomènech Badia, Mehdi Mirza, Alex Graves, Timothy P Lillicrap, Tim Harley,
David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In Inter-
national Conference on Machine Learning, pages 1928–1937, 2016.
Jack W Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Y Song, John
Aslanides, Sarah Henderson, Roman Ring, Susannah Young, et al. Scaling language models: Methods,
analysis & insights from training gopher. arXiv preprint arXiv:2112.11446, 2021.
Julian Schrittwieser, Ioannis Antonoglou, Thomas Hubert, Karen Simonyan, Laurent Sifre, Simon Schmitt,
Arthur Guez, Edward Lockhart, Demis Hassabis, Thore Graepel, et al. Mastering atari, go, chess and
shogi by planning with a learned model. Nature, 588(7839):604–609, 2020.
David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian
Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, et al. Mastering the game of go
with deep neural networks and tree search. Nature, 529(7587):484–489, 2016.
David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas
Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al. Mastering the game of go without human
knowledge. Nature, 550(7676):354–359, 2017.
David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Marc
Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, et al. A general reinforcement learning
algorithm that masters chess, shogi, and go through self-play. Science, 362(6419):1140–1144, 2018.
Aaron Van Den Oord, Oriol Vinyals, and Koray Kavukcuoglu. Neural discrete representation learning. In
Advances in Neural Information Processing Systems, pages 6306–6315, 2017.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser,
and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems,
pages 5998–6008, 2017.

23
Oriol Vinyals, Igor Babuschkin, Wojciech M Czarnecki, Michaël Mathieu, Andrew Dudzik, Junyoung Chung,
David H Choi, Richard Powell, Timo Ewalds, Petko Georgiev, et al. Grandmaster level in starcraft ii using
multi-agent reinforcement learning. Nature, 575(7782):350–354, 2019.

24