The Sequence Modeling Problem

In 2017, a paper called "Attention Is All You Need" introduced the Transformer architecture. It replaced recurrent neural networks (RNNs) with a simpler, faster, and more powerful design. Today, every major LLM — GPT, Claude, Llama, Gemini — is built on Transformers.

This tutorial explains how Transformers work, from the ground up. We use a simple sentence example with real numbers so you can follow every step.

Table of Contents

• The Sequence Modeling Problem
• Word Embeddings and Positional Encoding
• Self-Attention: The Core Innovation
• Query, Key, Value (Q, K, V)
• The Attention Formula
• Worked Example: "The cat sat on the mat"
• Multi-Head Attention
• The Transformer Block
• Encoder vs Decoder
• RNN vs LSTM vs Transformer
• FAQ
• Exercises
• Conclusion

1. The Sequence Modeling Problem

Language is sequential. The meaning of a word depends on the words around it:

RNNs tried to solve this by processing tokens one at a time, carrying a "hidden state" forward. But this sequential approach had two critical problems:

1. Slow training: Cannot parallelize — must wait for token 1 before processing token 2
2. Vanishing gradients: Information from early tokens fades by the time you reach later tokens

Transformers solve both problems by processing all tokens simultaneously using self-attention.

2. Word Embeddings and Positional Encoding

Word Embeddings

Before processing, each token is converted to a numerical vector (embedding). This captures the token's meaning.

Real models use 768–12,288 dimensions. Words with similar meanings cluster together in this high-dimensional space.

Positional Encoding

Transformers process all tokens simultaneously — they have no inherent sense of order. Positional encoding adds position information so the model knows "cat" comes before "sat."

3. Self-Attention: The Core Innovation

Self-attention answers one question for every token: "Which other tokens in this sequence are most relevant to me?"

The key insight: every token attends to every other token simultaneously. This is computed in parallel, making it much faster than RNNs.

4. Query, Key, Value (Q, K, V)

Self-attention uses three vectors for each token, derived from its embedding:

The attention score between two tokens is computed as the dot product of the query of one token with the key of another. High dot product = high relevance.

5. The Attention Formula

Step by step:

1. Q·K^T — Compute dot product of every query with every key → raw attention scores
2. / √d_k — Scale by the square root of the key dimension (prevents large values)
3. softmax — Convert scores to probabilities (sum to 1.0)
4. · V — Multiply probabilities by values → weighted sum → output

6. Worked Example: "The cat sat on the mat"

Attention Score Matrix

After computing Q·K^T / √d_k (where d_k=2):

After Softmax → Attention Weights

Reading the table: Row "cat" shows that "cat" attends 19.6% to "mat" (strongest), 19.4% to itself, and 19.1% to "sat". These weights determine how much information each token contributes to the representation of "cat."

7. Multi-Head Attention

Real transformers do not use a single attention mechanism. They use multiple "heads" — each learning different types of relationships:

The outputs of all heads are concatenated and projected through a linear layer to produce the final attention output.

8. The Transformer Block

Each transformer layer (block) has two main sub-layers:

Add & Norm (residual connection + layer normalization) is critical — it allows gradients to flow through deep networks and stabilizes training.

9. Encoder vs Decoder

Transformers come in three configurations:

Key difference: Decoder-only models (GPT, Claude) use masked self-attention — each token can only attend to previous tokens, not future ones. This prevents "cheating" during text generation.

10. RNN vs LSTM vs Transformer

11. FAQ

12. Exercises

Exercise 1: Attention Intuition

For the sentence "The dog chased the cat because it was fast," which token should "it" attend to most strongly? Why? Write down the attention weight you would expect.

Exercise 2: Dot Product

Given Q=[1.0, 0.5] and K=[0.8, 0.3], compute the raw attention score (dot product). Then compute the scaled score if d_k=2. What does this number represent?

Exercise 3: Softmax

Given raw scores [2.0, 1.0, 0.5], compute the softmax probabilities by hand. Which token is most likely? What happens if you divide all scores by 2 (temperature=2.0)?

Exercise 4: Why Parallel?

If a sentence has 100 tokens, how many sequential steps does an RNN need? How many does a transformer need? What does this mean for GPU training?

Exercise 5: Multi-Head

If a transformer has 8 attention heads, each with d_k=64, what is the total dimension of the concatenated output? Why use multiple heads instead of one large head?

Try These BestWordz Tools

• Regex Tester — Practice pattern matching, analogous to attention patterns
• All BestWordz Tools — Explore the complete tool library

Continue Learning

• What Is an LLM? Beginner's Guide — The foundational concepts behind transformers
• How LLMs Generate Text — Tokens, probabilities, and next-token prediction
• Prompt Engineering Tutorial — How to write effective prompts for transformer-based models
• Context Engineering Explained — Controlling AI behavior through context
• How AI Coding Agents Work — Transformers as the brain of coding agents
• LLM Quantization Explained — Reducing model size while preserving quality
• RAG Architecture Explained — Extending transformer context with retrieval