Transformers Explained: Simplified Homework

Notes
These notes break down the concepts from the YouTube video "Transformers Explained | Simple
Explanation of Transformers" in a clear, concise way, following the video’s flow. Perfect for
understanding the Transformer architecture as if you’re jotting down homework notes.

1. Introduction to Transformers
What are Transformers?
Transformers are a deep learning architecture powering modern AI, like ChatGPT (uses GPT,
based on Transformers).
They’re behind the AI boom due to their ability to handle language tasks effectively.
Goal: Predict the next word in a sentence (e.g., Gmail’s word prediction or ChatGPT’s
responses).
Language Models
Language models predict the next word based on input. Examples:
Google’s BERT (Bidirectional Encoder Representations from Transformers).
GPT (Generative Pre-trained Transformer), used in ChatGPT, with billions of parameters.
Large language models (like GPT) are trained on massive datasets, making them highly
capable.
How It Works
You type a question → model predicts the next word → uses the question + predicted word to
predict the next → continues to form a full response.
Sounds like magic, but it’s all about predicting the next word!

2. Word Embeddings
Why Embeddings?
Machine learning models don’t understand text; they work with numbers.
Words need to be converted into numerical representations that capture their meaning.
Static Word Embeddings
Example: Represent the word "King" as a vector (list of numbers).
Ask questions like: Has authority? (Yes = 1), Has a tail? (No = 0), Is rich? (Yes = 1),
Gender? (Male = 1).
Result: A vector like [1, 0, 1, 1, …] for "King."
Similarly, "Queen," "Horse," or "Battle" can be represented as vectors.
These vectors allow math operations, e.g., King - Man + Woman ≈ Queen (amazing, right?).
Real-World Embeddings
Models like Google’s Word2Vec use 300 dimensions (not just 5 like the example).
Trained on massive text (Wikipedia, books, internet) using neural networks to capture word
relationships.
 Each dimension represents a "feature" of the word, but we don’t know exactly what each
number means.
Visualizing Embeddings
Imagine a 3D space (humans can’t visualize 300D!):
"King" might be at (3, 8, 2).
"Queen" is nearby, with a "gender direction" vector connecting them.
You can add this gender vector to "Uncle" to get "Aunt" or to "Father" to get "Mother."
GPT uses 12,000-dimensional embeddings for richer representations.
Static vs. Contextual Embeddings
Static embeddings (e.g., Word2Vec, GloVe) give fixed vectors for words, ignoring context.
Problem: "Track" in "train on the track" vs. "track my package" has different meanings.
Contextual embeddings adjust the vector based on surrounding words (e.g., "rice dish" vs.
"cheese dish").
Example: In "I made a sweet Indian rice dish," the embedding for "dish" changes with
adjectives like "sweet" or "Indian," making it more accurate for predicting words like "kheer"
or "biryani."

3. Transformer Architecture Overview

Two Main Components
Encoder: Takes an input sentence and creates contextual embeddings for each word/token.
Decoder: Uses those embeddings to predict the next word or generate translated text.
Tasks Transformers Handle
Predicting the next word (e.g., autocomplete in ChatGPT).
Translating sentences (e.g., English to Hindi).
Inference vs. Training
Inference: When the model is already trained and predicts words in real-world tasks.
Training: Like teaching a baby—model learns from massive text data (Wikipedia, books, etc.) to
predict words.
BERT vs. GPT
BERT: Uses only the encoder part to create contextual embeddings.
GPT: Uses only the decoder part to predict the next word.
Both are based on the Transformer architecture but implemented differently.

4. How Transformers Work: Step-by-Step

Step 1: Tokenization
What’s a Token?
Tokens are like words or parts of words (e.g., "playing" = "play" + "ing").
BERT has ~30,000 tokens; GPT has ~50,000.
Process:
Input sentence (e.g., "I made kheer") → tokenized into tokens (e.g., "I," "made," "kheer").
 Special tokens added:
CLS: Marks the start (used in BERT).
SEP: Separates sentences.
Each token gets an ID from the vocabulary (e.g., "made" = ID 2532).

Step 2: Static Embeddings

What Happens:
Each token gets a static embedding (vector) from a pre-trained static embedding matrix.
BERT: 768 dimensions per token.
GPT: 12,228 dimensions.
This matrix is created during training and maps each token ID to a vector.

Step 3: Positional Embeddings

Why Needed?
Transformers process all words in parallel (unlike older models like RNNs, which go word-by-
word).
Word order matters (e.g., "I made kheer" vs. "Kheer made I").
Positional embeddings add a small vector to each token’s embedding to encode its position
(1st, 2nd, etc.).
How It Works:
A formula (from the original Transformer paper) generates positional vectors.
Static embedding + positional embedding = a vector that knows the word and its position.

Step 4: Attention Mechanism

What’s Attention?
From the 2017 paper "Attention is All You Need" (by Google researchers).
Words "pay attention" to other words in the sentence to create contextual embeddings.
Example: In "I made a sweet Indian rice dish," "dish" pays attention to "sweet," "Indian," and
"rice" to understand its meaning.
Attention Scores:
Each word gets an attention score for how much it influences another word.
Example: "Sweet" might influence "dish" by 36%, "Indian" by 14%, "rice" by 18%.
Less relevant words (like "I" or "made") have lower scores (e.g., 7%).
Query, Key, Value (Q, K, V):
Analogy: In a library, you (query) ask for a book on quantum physics. The librarian uses book
labels (key) to find the book content (value).
In Transformers:
Query: What a word (e.g., "dish") wants to know about (e.g., its modifiers like "sweet").
Key: What other words describe themselves as (e.g., "sweet" says, "I’m an adjective for
taste").
Value: The actual contribution of each word (e.g., "sweet" contributes a "sweetness" vector).
How It’s Computed:
 Each token’s embedding is multiplied by three matrices: WQ (query), WK (key), WV (value).
These matrices are learned during training and stay fixed during inference.
Example: For "dish," its embedding (E7) × WQ = query vector (Q7). Similarly, E7 × WK =
key vector (K7), E7 × WV = value vector (V7).
Attention Formula:
Compute dot product between query and key vectors to get attention scores.
Pass scores through a softmax function to turn them into probabilities (summing to 1).
Multiply these probabilities by value vectors and sum them to get the contextual embedding.
Formula: Attention = softmax(QK^T / √d_k)V, where d_k is the key vector dimension
(e.g., 128 for GPT).

Step 5: Multi-Head Attention

Why Multiple Heads?
A single attention head focuses on one aspect (e.g., adjectives like "sweet").
Multiple heads (e.g., 96 in GPT) focus on different aspects (e.g., verbs, pronouns, cultural
context).
Example: One head might focus on "sweet Indian rice" modifying "dish," another on the verb
"made," another on pronouns like "I."
How It Works:
Each head computes its own contextual embedding using Q, K, V.
Combine (add) the outputs of all heads to get a richer contextual embedding.
GPT splits its 12,228 dimensions into 96 heads, so each head handles 12,228 ÷ 96 ≈ 128
dimensions.

Step 6: Feed-Forward Network

Purpose:
Attention gives contextual relationships, but language is complex and nonlinear.
A feed-forward neural network (FFN) applies nonlinear transformations to each token’s
embedding independently.
How It Works:
Input: Contextual embedding (e.g., 768D for BERT, 12,228D for GPT).
Passes through a neural network with a hidden layer (can have many neurons) and an output
layer (same size as input).
Output: Even more refined, contextually rich embedding.
Training:
FFN weights are learned during training, adjusting to capture complex language patterns.

Step 7: Residual Connections & Normalization

Residual Connections:
Add the original embedding to the output of attention or FFN.
Helps with training stability and gradient flow (a deep learning concept).
Layer Normalization:
Normalizes values to have zero mean and one standard deviation.
 Ensures stable training and better gradient flow.

Step 8: Stacking Layers (Nx Layers)

What’s an Nx Layer?
One Transformer block = normalization + multi-head attention + FFN + residual connections.
Stack multiple blocks (e.g., 12 for BERT base, 24 for BERT large, different for GPT).
Each block refines the embeddings further, making them more contextually rich.

Step 9: Decoder
Purpose:
Takes contextual embeddings from the encoder and predicts the next word or generates a
translated sentence.
Cross Attention:
In tasks like translation (e.g., English "I made kheer" to Hindi):
Encoder processes the English sentence, producing key (K) and value (V) vectors.
Decoder generates the Hindi sentence (e.g., "Maine kheer banai").
Query (Q) comes from the decoder’s output (e.g., "Maine"), while K and V come from the
encoder (English sentence).
This is called cross attention because queries and keys/values come from different
sources.
Process:
Starts with a special start token.
Predicts the first word (e.g., "Maine"), then uses it as input to predict the next ("kheer"), and so
on.
Outputs a probability distribution over the vocabulary (e.g., 30,000 words for BERT).

5. Training Transformers
How Models Learn:
Trained on massive text (Wikipedia, books, internet) using self-supervised learning.
No human labeling needed! Example: Input = "I made a sweet Indian rice," target = "dish."
Model predicts the next word, computes errors (e.g., predicted "Mexican" instead of "Indian"),
and updates weights via backpropagation.
What’s Learned:
Static embedding matrix (for all tokens).
WQ, WK, WV matrices for attention.
Feed-forward network weights.
Example:
In training, the model sees "developing an advanced crude" and learns "spacecraft" or "vehicle"
are likely next words, not "banana."
It builds a vocabulary (e.g., 30,000 tokens for BERT) and learns contextual relationships.
6. Visualizing Transformers
Tool Recommendation:
Visit poloclub.github.io/transformer-explainer to see an interactive visualization of the
Transformer architecture.
Example: Input "As the spaceship was approaching the" → predicts "station."
Shows token embeddings, positional embeddings, Q/K/V computation, FFN, and softmax
probabilities.
Key Components Visualized:
Token embeddings (e.g., 768D for BERT).
Positional embeddings added.
Q, K, V computation in attention heads.
Feed-forward network and residual connections.
Multiple Transformer blocks (e.g., 11 layers).

7. Additional Resources
3Blue1Brown YouTube Channel:
Watch videos #5, #6, #7 on Transformers for deeper understanding.
Great for visualizing complex concepts (credits to 3Blue1Brown for inspiration!).

8. Summary
Transformers:
Power AI like ChatGPT by predicting the next word or translating sentences.
Use encoders (create contextual embeddings) and decoders (generate output).
Key Steps:
1. Tokenize input sentence and assign token IDs.
2. Get static embeddings from a pre-trained matrix.
3. Add positional embeddings to encode word order.
4. Use multi-head attention to compute contextual embeddings (Q, K, V).
5. Pass through a feed-forward network for nonlinear transformations.
6. Apply residual connections and normalization for stability.
7. Stack multiple Transformer blocks (Nx layers).
8. Decoder uses cross attention for tasks like translation.
Training:
Uses self-supervised learning on massive text data to learn embeddings and weights.
Explore More:
Use the Poloclub Transformer Explainer tool and 3Blue1Brown videos to deepen your
understanding.