AI & Machine Learning

The Complete Pipeline

Neural Networks LLMs RAG Prompt Engineering AI Agents Git Redis Transformers Embeddings Vector Search Quantization

2,261 words

LLM text generation pipeline showing 8 steps from tokenization through autoregressive loop, with sampling methods comparison for temperature, top-k, and top-p

📌 Key Takeaway

LLMs generate text one token at a time through a repeating cycle: tokenize input → compute probabilities for every possible next token → sample one token → append it → repeat. Temperature, top-k, and top-p control how that sampling happens — from deterministic to creative.

You press Enter after typing a prompt. Within seconds, a coherent paragraph appears. But what actually happened inside the model? This tutorial walks through every step — with real numbers, not hand-waving.

We use a tiny artificial vocabulary of 24 tokens so you can follow every calculation. Real LLMs have vocabularies of 30,000–100,000 tokens, but the mechanism is identical.

The Complete Pipeline
Step 1: Tokenization
Step 2: Embedding
Step 3: Transformer Processing
Step 4: Logits — Raw Scores
Step 5: Softmax — Probability Distribution
Step 6: Temperature
Step 7: Top-K Sampling
Step 8: Top-P (Nucleus) Sampling
Step 9: The Autoregressive Loop
Full Worked Example
Sampling Methods Compared
FAQ
Exercises
Conclusion

1. The Complete Pipeline

Every LLM text generation follows the same 8-step pipeline:

① TOKENIZE    Split input into tokens → assign integer IDs

     ↓

② EMBED       Convert token IDs → numerical vectors

     ↓

③ TRANSFORM   Self-attention computes context-aware representations

     ↓

④ LOGITS      Output layer produces a score for every token in vocabulary

     ↓

⑤ SOFTMAX     Convert logits → probability distribution (sums to 1.0)

     ↓

⑥ SAMPLE      Select one token using temperature, top-k, or top-p

     ↓

⑦ APPEND      Add selected token to the sequence

     ↓

⑧ REPEAT      Feed updated sequence back to step ②, until [EOS] or max length

Steps ①–⑤ happen in parallel on the full input. Steps ⑥–⑧ repeat one token at a time. This is why LLMs are called autoregressive — each new token depends on all previous tokens.

2. Step 1: Tokenization

Tokenization splits your text into tokens — the atomic units the model processes. Each token maps to an integer ID.

OUR TINY VOCABULARY (24 tokens):

the  a  cat  dog  sat  on  mat  big  small

happy  ran  is  was  very  in  and  to  quick

brown  fox  jumps  over  lazy  [EOS]

INPUT:  "The cat sat"

TOKENS: ["the", "cat", "sat"]

IDS:    [0, 2, 4]

Key detail: Real LLMs use subword tokenization (Byte Pair Encoding). "Unhappiness" might become ["un", "happy", "ness"] — 3 tokens. This lets the model handle any word, even misspelled ones, by decomposing into known subwords.

For a deeper explanation, see What Is an LLM? A Complete Beginner's Guide.

3. Step 2: Embedding

Neural networks process numbers, not text. Embedding converts each token ID into a numerical vector — a list of numbers that captures the token's meaning.

INPUT TOKENS:  ["the", "cat", "sat"]

TOKEN IDS:     [0, 2, 4]

EMBEDDINGS (simplified 3D):

"the" → [0.79,  0.33, -0.23]

"cat" → [0.11,  0.10, -0.27]

"sat" → [0.40,  0.10, -0.04]

Real models use vectors of 768–12,288 dimensions.

Why embeddings matter: Words with similar meanings end up near each other in vector space. "Cat" and "dog" have similar embeddings. "Cat" and "car" do not. This is how the model "understands" that certain words are related.

4. Step 3: Transformer Processing

The transformer is the core architecture. It processes all input tokens simultaneously using self-attention — a mechanism that determines how much each token should "attend to" every other token.

What self-attention does for "The cat sat":

• "cat" attends strongly to "The" (determines it is a noun phrase)
• "sat" attends to "cat" (knows who is doing the sitting)
• Each token gets a context-aware representation that includes information from all other tokens

A real transformer has 32–128 attention layers, each refining these representations.

The transformer does not predict the next token directly. It produces a rich representation of the entire input. The next step converts that representation into scores for each possible next token.

5. Step 4: Logits — Raw Scores

After transformer processing, the model produces a logit (raw score) for every token in the vocabulary. Higher logit = the model thinks this token is more likely to come next.

CONTEXT: "The cat sat"

MODEL OUTPUT (logits for next token):

on:      2.50  ██████████████████████████

mat:     1.80  ██████████████████

in:      1.20  ████████████

[EOS]:   1.00  ██████████

is:      0.90  █████████

a:       0.80  ████████

was:     0.70  ███████

happy:   0.60  ██████

and:     0.50  █████

ran:     0.40  ████

... 14 more tokens with lower scores

Important: Logits are not probabilities. They can be any real number (negative or positive). They need to be converted to probabilities using the softmax function.

6. Step 5: Softmax — Probability Distribution

The softmax function converts logits into a probability distribution — a set of values that sum to 1.0.

SOFTMAX FORMULA:

P(token_i) = exp(logit_i / T) / Σ exp(logit_j / T)

where T = temperature (default 1.0)

Applied to our logits with T=1.0:

Token	Logit	exp(logit)	Probability
on	2.50	12.18	27.0%
mat	1.80	6.05	13.4%
in	1.20	3.32	7.3%
[EOS]	1.00	2.72	6.0%
is	0.90	2.46	5.4%
a	0.80	2.23	4.9%
was	0.70	2.01	4.5%
happy	0.60	1.82	4.0%
Total (24 tokens)			100.0%

The model assigns 27% probability to "on" — the most likely next token. But it does not always pick the most likely token. That is where sampling comes in.

7. Step 6: Temperature

Temperature controls how "spread out" or "focused" the probability distribution is. It divides each logit before softmax.

TEMPERATURE FORMULA:

P(token_i) = exp(logit_i / T) / Σ exp(logit_j / T)

Temperature	Effect on "on"	Entropy	Behavior
0.1	99.9%	0.007	Nearly deterministic — always picks "on"
0.7	43.1%	2.113	Focused but allows alternatives
1.0	27.0%	2.629	Balanced — the "natural" distribution
1.5	16.3%	2.937	Creative — more tokens become viable

Analogy: Restaurant Choice

Low temperature (0.1): You always go to the most popular restaurant. Safe, predictable.
Medium temperature (0.7): You usually go to the popular place but sometimes try something new.
High temperature (1.5): You pick randomly from any restaurant. Exciting but risky.

8. Step 7: Top-K Sampling

Top-K sampling keeps only the K most probable tokens and discards the rest. The remaining probabilities are renormalized to sum to 1.0.

TOP-K with K=5:

BEFORE (24 tokens, probabilities from softmax at T=1.0):

on: 27.0%  mat: 13.4%  in: 7.3%  [EOS]: 6.0%  is: 5.4%

a: 4.9%    was: 4.5%   happy: 4.0%  and: 3.6%  ran: 3.3%

... 14 more tokens

AFTER (keep top 5, renormalize):

on:  45.6%  ████████████████████

mat: 22.6%  █████████

in:  12.4%  █████

[EOS]: 10.2%  ████

is:   9.2%  ███

19 tokens eliminated. Probability mass redistributed among top 5.

Advantage: Prevents the model from picking very unlikely tokens. Disadvantage: Fixed K may be too small (cuts good options) or too large (allows bad options).

9. Step 8: Top-P (Nucleus) Sampling

Top-P (nucleus) sampling keeps the smallest set of tokens whose cumulative probability exceeds p. Unlike top-K, the number of candidates is adaptive.

TOP-P with p=0.85:

Sorted by probability, accumulate until ≥ 85%:

on:    27.0%  → cumulative: 27.0%  ✓ keep

mat:   13.4%  → cumulative: 40.4%  ✓ keep

in:     7.3%  → cumulative: 47.7%  ✓ keep

[EOS]:  6.0%  → cumulative: 53.7%  ✓ keep

is:     5.4%  → cumulative: 59.1%  ✓ keep

a:      4.9%  → cumulative: 64.0%  ✓ keep

was:    4.5%  → cumulative: 68.5%  ✓ keep

happy:  4.0%  → cumulative: 72.5%  ✓ keep

and:    3.6%  → cumulative: 76.1%  ✓ keep

ran:    3.3%  → cumulative: 79.4%  ✓ keep

the:    3.1%  → cumulative: 82.5%  ✓ keep

to:     3.1%  → cumulative: 85.6%  ✓ keep ← reached 85%, stop

12 tokens kept (adaptive), 12 eliminated.

Renormalize the 12 kept tokens so they sum to 100%.

Advantage: Adapts to the model's confidence. When the model is very sure (one token dominates), few candidates survive. When uncertain, many candidates survive. This is why top-p is often preferred over top-k.

10. The Autoregressive Loop

After sampling one token, the model feeds the updated sequence back and repeats the entire process:

Iteration 1: Input "The cat sat"     → Sample "on"   → Sequence: "The cat sat on"

Iteration 2: Input "The cat sat on"  → Sample "the"  → Sequence: "The cat sat on the"

Iteration 3: Input "The cat sat on the" → Sample "mat" → Sequence: "The cat sat on the mat"

Iteration 4: Input "...the mat"     → Sample "[EOS]" → STOP

Critical insight: The model does not plan the entire response in advance. It builds the output one token at a time, with each decision influenced by everything that came before. This is why LLMs sometimes "lose track" in long outputs — earlier context gradually fades.

11. Full Worked Example

Let us trace through one complete generation from start to finish.

PROMPT: "The cat"

SETTINGS: temperature=0.8, top_k=5

STEP 1: Tokenize → ["the", "cat"] → IDs [0, 2]

STEP 2: Embed → [[0.79, 0.33, -0.23], [0.11, 0.10, -0.27]]

STEP 3: Transformer → context-aware representations

STEP 4: Logits → {sat: 1.8, is: 1.2, was: 0.9, ran: 0.7, ...}

STEP 5: Softmax(T=0.8) → {sat: 38%, is: 19%, was: 14%, ran: 10%, ...}

STEP 6: Top-K(5) → keep [sat, is, was, ran, happy], renormalize

STEP 7: Sample → "sat" (38% chance, most likely picked)

STEP 8: New sequence → "The cat sat"

NEXT ITERATION: feed "The cat sat" back...

STEP 4: Logits → {on: 2.5, mat: 1.8, in: 1.2, ...}

STEP 5: Softmax(T=0.8) → {on: 35%, mat: 17%, in: 9%, ...}

STEP 6: Top-K(5) → keep [on, mat, in, [EOS], is]

STEP 7: Sample → "on"

STEP 8: New sequence → "The cat sat on"

... continues until [EOS] or max tokens ...

12. Sampling Methods Compared

Method	How It Works	Best For	Risk
Greedy (T=0)	Always pick highest probability	Code, math, factual answers	Repetitive, no variation
Temperature	Scale logits before softmax	Balancing focus vs creativity	High T → incoherent text
Top-K	Keep K most likely, renormalize	Preventing unlikely tokens	Fixed K may be too strict/loose
Top-P	Keep until cumulative p% reached	Adaptive candidate selection	May include too many/few tokens
Top-K + Top-P	Apply both, take intersection	Production systems	Requires careful tuning

💡 PRACTICAL TIP: Most production systems use temperature between 0.0–0.7 combined with top-p of 0.9–0.95. This balances quality with enough variation to avoid repetition.

13. FAQ

Does the model "decide" what to write before generating?

No. LLMs do not plan ahead. They generate one token at a time, with each token determined by the previous tokens. The model does not "know" the full answer before it starts writing. This is fundamentally different from how humans write, where we usually have an idea of the conclusion before we start.

Why does the same prompt give different answers each time?

Because of sampling. At each step, the model has a probability distribution over possible next tokens. Unless you use temperature=0 (greedy decoding), the selected token varies. This is by design — it gives outputs diversity and creativity. Set temperature=0 for deterministic output.

What is the difference between tokens and words?

Tokens are the units the model processes. A token can be a whole word, part of a word, or a character. In English, 1 token ≈ 0.75 words. "Unhappiness" might be 3 tokens (un + happy + ness). The model's vocabulary is typically 30K–100K tokens.

Why does the model sometimes produce wrong answers confidently?

The model predicts statistically likely tokens — it does not look up facts. If the training data contained incorrect information, or if the question is ambiguous, the model generates plausible-sounding but wrong text. This is called a hallucination. Always verify important information.

Can I use both temperature and top-p together?

Yes, and this is common in production. Temperature adjusts the shape of the distribution, then top-p clips the tail. A typical setting is temperature=0.7 with top-p=0.9. This gives focused but not deterministic output.

What is the [EOS] token?

End of Sequence. When the model selects [EOS], generation stops. It is the model's way of saying "I am done." Without it, the model would keep generating indefinitely until hitting the maximum token limit.

How many tokens can an LLM generate in one response?

This depends on the model's maximum output length (often 4K–16K tokens) and the API configuration. The context window must fit both input and output tokens. If your prompt is 1,000 tokens and the context window is 8K, you can generate up to ~7K output tokens.

Why is temperature=0 called "greedy"?

Because it always picks the locally most probable token — the "greedy" choice. It never explores alternatives. This is deterministic but can produce repetitive text and miss better overall sequences.

14. Exercises

Exercise 1: Trace a Token

Given the logits {cat: 1.5, dog: 1.2, fish: 0.3} and temperature=1.0, calculate the softmax probability for each token by hand. Which token is most likely? What happens if you set temperature=0.5?

Exercise 2: Top-K with K=2

Given probabilities {on: 35%, mat: 17%, in: 9%, is: 8%, was: 7%, happy: 6%, ...}, apply top-k with K=2. What are the renormalized probabilities? What is the probability of picking "in" after top-k?

Exercise 3: Temperature Comparison

Ask an LLM the same question 10 times at temperature=0.0, temperature=0.7, and temperature=1.5. Record: How many unique responses do you get at each temperature? Which is most useful?

Exercise 4: Autoregressive Dependency

Start with "The" and generate 5 tokens at temperature=0.1. Then start with "The" and generate 5 tokens at temperature=1.5. How does the starting context constrain what comes next?

Exercise 5: Why Sampling Matters

If an LLM always picked the most likely token (greedy/temperature=0), predict what would happen to a 500-word essay generated this way. Would it be coherent? Would it be repetitive? Why?

Try These BestWordz Tools

Regex Tester — Understand pattern matching, similar to tokenization rules
JSON Formatter — Useful when working with LLM API responses and logits
All BestWordz Tools — Explore the complete tool library

Continue Learning

What Is an LLM? Beginner's Guide — The foundational concepts behind this tutorial
Prompt Engineering Tutorial — How to write prompts that produce better outputs
Context Engineering Explained — Controlling AI behavior through context, not just prompts
How AI Coding Agents Work — LLMs as the brain of autonomous coding tools
LLM Quantization Explained — How model size affects generation quality
RAG Architecture Explained — Giving LLMs access to external knowledge