What Are Embeddings?

When you search for "pet care tips" and find results about "how to train a puppy," the system did not match exact words. It matched meaning. That is the power of embeddings.

This tutorial explains how AI converts meaning into numbers, from tokenization through vector space to semantic search — with a complete Python implementation.

Table of Contents

• What Are Embeddings?
• Text → Tokens → Embeddings → Vectors
• Understanding Vector Space
• Cosine Similarity
• Worked Example with Python
• Nearest Neighbors
• Semantic Search
• How Many Dimensions?
• Real-World Applications
• Mini-Project: Document Search Engine
• FAQ
• Conclusion

1. What Are Embeddings?

An embedding is a numerical representation of a word, sentence, or document in a continuous vector space. Each piece of text becomes a list of numbers (a vector), and the positions of these vectors encode meaning.

Embeddings are learned during training. The model discovers that "cat" and "dog" appear in similar contexts, so their vectors end up close together. No one manually assigns these positions — the model learns them from data.

2. Text → Tokens → Embeddings → Vectors

The complete pipeline from raw text to usable vectors:

Word embeddings represent individual words. Sentence embeddings represent entire sentences by combining (pooling) the word vectors. Modern embedding models like OpenAI's text-embedding-3 or sentence-transformers produce sentence-level vectors directly.

3. Understanding Vector Space

Embeddings live in a high-dimensional space where each dimension captures some aspect of meaning:

Important: These dimensions are not explicitly labeled. The model discovers useful dimensions automatically during training. In real models with 768–3,072 dimensions, the learned patterns are far more nuanced than simple categories.

See Transformers Explained for how embeddings are computed within transformer architectures.

4. Cosine Similarity

Cosine similarity measures the angle between two vectors — how "pointing in the same direction" they are.

Why cosine, not Euclidean distance? Cosine similarity is insensitive to vector magnitude — it only cares about direction. Two vectors pointing the same way but with different lengths will have high cosine similarity. This makes it ideal for text, where the "direction" encodes meaning.

Visual Intuition

5. Worked Example with Python

Here is a complete, runnable Python implementation of cosine similarity:

import math

def cosine_similarity(a, b):
    """Compute cosine similarity between two vectors."""
    dot = sum(x * y for x, y in zip(a, b))
    mag_a = math.sqrt(sum(x * x for x in a))
    mag_b = math.sqrt(sum(x * x for x in b))
    return dot / (mag_a * mag_b)

# Example word embeddings (4-dimensional)
cat  = [0.9, 0.8, 0.1, 0.2]
dog  = [0.85, 0.78, 0.15, 0.25]
car  = [0.1, 0.2, 0.9, 0.8]

print(f"cat ↔ dog: {cosine_similarity(cat, dog):.4f}")
print(f"cat ↔ car: {cosine_similarity(cat, car):.4f}")

# Output:
# cat ↔ dog: 0.9977  (very similar — both animals)
# cat ↔ car: 0.3331  (dissimilar — different domain)

Step-by-Step Calculation

6. Nearest Neighbors

Finding the most similar items to a query is called nearest neighbor search. For each candidate, compute cosine similarity and rank by score:

At small scale, brute-force comparison works. At large scale (millions of vectors), you need approximate nearest neighbor (ANN) algorithms like FAISS, Annoy, or HNSW for efficient search.

7. Semantic Search

Semantic search uses embeddings to find results based on meaning, not just keywords:

Semantic search is the foundation of RAG (Retrieval-Augmented Generation) systems. When you ask a chatbot about your documents, it embeds your query, finds the most similar document chunks, and feeds them to the LLM.

Learn more in RAG Architecture Explained.

8. How Many Dimensions?

More dimensions ≠ always better. 384 dimensions from a well-trained model can outperform 3,072 from a poorly trained one. Quality depends on training data and architecture, not just dimension count.

9. Real-World Applications

10. Mini-Project: Document Search Engine

Build a simple document search engine using embeddings and cosine similarity:

import math

def cosine_similarity(a, b):
    dot = sum(x * y for x, y in zip(a, b))
    mag_a = math.sqrt(sum(x * x for x in a))
    mag_b = math.sqrt(sum(x * x for x in b))
    return dot / (mag_a * mag_b)

# Document embeddings (in production, use an embedding model)
docs = [
    {"title": "Intro to Python", "vec": [0.8, 0.7, 0.2, 0.1]},
    {"title": "Data Science",    "vec": [0.75, 0.65, 0.35, 0.15]},
    {"title": "Web Dev with Flask", "vec": [0.5, 0.4, 0.6, 0.3]},
    {"title": "Docker Guide",      "vec": [0.2, 0.15, 0.8, 0.5]},
]

def search(query_vec, docs, k=3):
    results = []
    for doc in docs:
        sim = cosine_similarity(query_vec, doc["vec"])
        results.append((doc["title"], sim))
    results.sort(key=lambda x: -x[1])
    return results[:k]

# Search
query = [0.82, 0.72, 0.18, 0.12]  # "learn Python basics"
for title, score in search(query, docs):
    print(f"[{score:.3f}] {title}")

# Output:
# [1.000] Intro to Python
# [0.984] Data Science
# [0.915] Web Dev with Flask

In production, you would replace the hand-crafted vectors with real embeddings from a model like sentence-transformers or OpenAI's text-embedding-3-small, and use a vector database like Chroma, Pinecone, or Weaviate for efficient search at scale.

11. FAQ

Try These BestWordz Tools

• Standard Deviation Calculator — Understand vector magnitude and distance concepts
• Regex Tester — Pattern matching, analogous to embedding-based similarity
• All BestWordz Tools — Explore the complete tool library

Continue Learning

• RAG Architecture Explained — How embeddings power retrieval-augmented generation
• Transformers Explained — The architecture behind modern embedding models
• What Is an LLM? Beginner's Guide — Foundational concepts
• How LLMs Generate Text — Tokens, probabilities, and generation
• Context Engineering Explained — Using embeddings in context design
• Prompt Engineering Tutorial — Better prompts for better results
• How AI Coding Agents Work — Embeddings in agent workflows