What Is an Embedding?

Key Takeaway: Embeddings transform text into numerical vectors that capture meaning. Semantically similar texts produce vectors that are close together in a high-dimensional space — enabling semantic search, recommendation systems, and modern AI applications. You can build and experiment with embeddings using nothing more than Python and basic mathematics.

Every time you search for something online, get a product recommendation, or use an AI assistant that understands your question — even when you phrase it imperfectly — embeddings are likely working behind the scenes.

But what are embeddings? How can a string of numbers capture the meaning of a sentence? And why do semantically similar texts end up close together in a mathematical space?

This article explains embeddings from the ground up — no deep learning prerequisites required. By the end, you will understand how text is transformed into vectors, how similarity is measured, and how to build a practical embedding comparison using Python.

What Is an Embedding?

An embedding is a fixed-length numerical vector that represents a piece of text — a word, sentence, paragraph, or document. The vector captures semantic information: what the text means, not just what characters it contains.

Consider two sentences:

Sentence A: "The cat sat on the mat"
Sentence B: "A feline rested on the rug"

A naive text comparison would find almost no character overlap between these sentences. But a human immediately recognizes they describe the same scenario. Embeddings bridge this gap — they represent both sentences as vectors that are close together in a high-dimensional space.

From Text to Vector: The Pipeline

The transformation follows a consistent pipeline:

1. Tokenization: The input text is split into tokens (words, subwords, or characters).
2. Token Embedding: Each token is mapped to a vector using the model's learned parameters.
3. Aggregation: Token vectors are combined (via mean pooling, attention, or other mechanisms) into a single sentence or document vector.
4. Output: A fixed-length numerical vector — the embedding.

For example, the sentence "I love programming" might produce the vector [0.85, 0.20, 0.90, 0.15, 0.80, ...] — a sequence of floating-point numbers that encodes the sentence's meaning.

Why Numbers Can Represent Meaning

This is the core insight: embeddings encode statistical and learned relationships from massive text corpora. During training, the model learns that words appearing in similar contexts tend to have similar meanings.

Consider how often these words appear near similar contexts:

"The ___ is a popular pet"
  → cat, dog, rabbit (similar context patterns)

"Python is a popular ___"
  → language, programming, framework (similar context patterns)

The model learns that cat and dog are semantically similar because they appear in nearly identical sentence structures. This statistical pattern is captured as vectors that are close together in the embedding space.

Important: Embeddings do not "understand" language the way humans do. They encode statistical patterns — distributional similarities learned from text data. This is powerful enough for many practical applications, but it is not comprehension.

Why Similar Meanings Cluster Together

In an embedding space, words and sentences with similar meanings occupy nearby positions. The distance between vectors reflects their semantic relationship:

• "happy" and "joyful" → very close (nearly synonymous)
• "happy" and "sad" → far apart (antonyms)
• "cat" and "dog" → close (similar category)
• "cat" and "programming" → far apart (unrelated domains)

This clustering property is what makes embeddings so useful — it allows machines to reason about meaning using mathematics.

Measuring Similarity

Cosine Similarity

The most common metric for comparing embeddings is cosine similarity, which measures the angle between two vectors:

cosine_similarity(A, B) = (A · B) / (||A|| × ||B||)

Where:
  A · B    = sum of element-wise products
  ||A||    = magnitude (Euclidean norm) of A
  ||B||    = magnitude of B

Cosine similarity ranges from -1 (opposite directions) to 1 (identical direction). For most embedding models, values range from 0 to 1, where:

• 1.0 = identical meaning
• 0.8–0.99 = very similar
• 0.5–0.8 = somewhat related
• 0.0–0.3 = unrelated

Euclidean Distance

Another common metric is Euclidean distance — the straight-line distance between two vectors in the embedding space. Unlike cosine similarity, it considers the magnitude of vectors, not just their direction.

euclidean_distance(A, B) = sqrt(sum((a_i - b_i)²))

Smaller distance = more similar

For most embedding applications, cosine similarity is preferred because it measures directional similarity regardless of vector magnitude.

Types of Embeddings

Embeddings exist at different granularities, each suited to different tasks:

Word Embeddings represent individual words. Models like Word2Vec, GloVe, and FastText produce vectors of 50–300 dimensions. They are excellent for word-level similarity and analogy tasks, but they assign the same vector to a word regardless of context. Sentence Embeddings capture the meaning of an entire sentence in one vector. Models like Sentence-BERT and all-MiniLM-L6-v2 produce 384–768 dimensional vectors. These are the workhorses of semantic search and retrieval-augmented generation (RAG). Document Embeddings represent entire documents. Approaches include mean pooling over sentence embeddings, specialized models like Doc2Vec, or chunking long documents into segments and embedding each separately. Static vs. Contextual Embeddings: Early models like Word2Vec produce static embeddings — the word "bank" gets the same vector whether it refers to a financial institution or a river bank. Modern transformer-based models produce contextual embeddings, where the vector changes based on surrounding words.

Dimensions Explained

An embedding's dimension is the length of its vector. A 384-dimensional embedding has 384 floating-point numbers. Each dimension captures some aspect of the text's meaning — though individual dimensions are rarely human-interpretable.

# 3-dimensional example
vector_3d = [0.3, 0.7, 0.1]

# Same concept, more dimensions
vector_8d = [0.3, 0.7, 0.1, 0.0, 0.0, 0.0, 0.0, 0.0]

# Real models: 384–1536 dimensions
# More dimensions = richer representation (up to a point)

Higher-dimensional embeddings can capture more nuanced semantic relationships, but they also require more memory and computation. The choice of dimension is a practical trade-off between accuracy and efficiency.

How Embedding Models Are Trained

Embedding models learn from vast amounts of text through self-supervised training. The key insight is the distributional hypothesis: words that appear in similar contexts have similar meanings.

Modern sentence embedding models typically use a two-stage process:

1. Pre-training: A transformer model (like BERT or RoBERTa) learns general language understanding from massive text corpora.
2. Fine-tuning: The model is specialized for producing high-quality sentence embeddings, often using contrastive learning — training on pairs of similar and dissimilar sentences.

After training, the model can convert any text into a meaningful vector without further learning. The vectors encode the statistical patterns discovered during training.

Multilingual Embeddings

Some embedding models are trained on multilingual text corpora and can produce vectors for text in dozens of languages. Remarkably, semantically equivalent sentences in different languages end up close together in the same embedding space:

"Hello, how are you?"  → [0.12, -0.34, 0.87, ...]  # English
"Hola, ¿cómo estás?"  → [0.14, -0.31, 0.85, ...]  # Spanish
"Bonjour, comment vas-tu?" → [0.11, -0.33, 0.86, ...]  # French

# These vectors are close together — same meaning, different languages

Multilingual embeddings enable cross-language search, translation retrieval, and multilingual recommendation systems.

Limitations and Bias

Embeddings are powerful, but they have important limitations:

• Bias: Embeddings reflect biases present in training data. If the training corpus contains gender or racial stereotypes, the embeddings will encode them.
• Context blindness (static models): Word-level embeddings cannot distinguish different meanings of the same word.
• Length limitations: Most models have maximum token limits (typically 512–8192 tokens).
• No reasoning: Similarity scores indicate surface-level semantic closeness, not logical equivalence or factual accuracy.
• Domain sensitivity: Models trained on general text may produce poor embeddings for specialized domains like medical or legal text.

Privacy Considerations

Embeddings raise several privacy considerations:

• Data embedded in vectors: While embeddings abstract away raw text, they can sometimes be partially inverted — recovering approximate original text from embeddings.
• Cloud processing: Most commercial embedding APIs send your text to external servers.
• Local alternatives: Open-source models like all-MiniLM-L6-v2 can run locally, keeping data on your machine.
• Inference from similarity: If you know the embedding of a sensitive document, an attacker could compare it against known documents.

For privacy-sensitive applications, consider running embedding models locally. Our guide on Running AI Locally on CPU shows how to set up a private local AI environment.

Practical Example: Sentence Similarity Ranking

Let's build a working example that compares sentences and ranks them by semantic similarity using pure Python and NumPy. This example uses synthetic embeddings to demonstrate the mechanics.

import numpy as np

def cosine_similarity(a, b):
    """Compute cosine similarity between two vectors."""
    return np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b))

# Synthetic sentence embeddings (5-dimensional)
sentences = {
    "I love programming in Python":         np.array([0.85, 0.20, 0.90, 0.15, 0.80]),
    "Python is my favorite language":        np.array([0.82, 0.22, 0.88, 0.18, 0.78]),
    "I enjoy coding every day":              np.array([0.78, 0.25, 0.82, 0.20, 0.75]),
    "The weather is nice today":             np.array([0.15, 0.80, 0.12, 0.85, 0.18]),
    "Machine learning uses math models":     np.array([0.60, 0.45, 0.55, 0.40, 0.50]),
}

# Query sentence
query = np.array([0.80, 0.23, 0.85, 0.17, 0.76])

# Rank by similarity
scores = [(sent, cosine_similarity(query, vec))
          for sent, vec in sentences.items()]
scores.sort(key=lambda x: x[1], reverse=True)

print("Query: \"I like writing code in Python\"")
print("\nRanking by semantic similarity:")
for rank, (sent, score) in enumerate(scores, 1):
    print(f"  {rank}. [{score:.4f}] {sent}")

This produces a ranking where programming-related sentences rank highest and the weather sentence ranks lowest — exactly what we would expect from a semantic comparison.

From Embeddings to Semantic Search

Embeddings are the foundation of semantic search. The workflow is simple:

1. Embed your documents: Convert each document (or chunk) into a vector.
2. Store vectors: Keep them in a list, array, or vector database.
3. Embed the query: Convert the search query into a vector using the same model.
4. Compare: Compute similarity between the query vector and all document vectors.
5. Rank and return: Return the most similar documents.

For a complete tutorial on building a local vector store, see our guide on Building a Private Vector Store in Pure Python. For understanding the Model Context Protocol that connects AI agents to data, see our MCP guide.

When to Use Embeddings

Embeddings are particularly valuable when keyword search is insufficient:

• Semantic search: Finding documents by meaning, not just keywords.
• Recommendation systems: Suggesting similar content.
• Clustering: Grouping similar documents automatically.
• Retrieval-Augmented Generation (RAG): Providing relevant context to LLMs.
• Duplicate detection: Finding near-duplicate content.
• Anomaly detection: Identifying outliers in text data.
• Cross-language search: Finding equivalent content across languages.

Key Takeaways

• Embeddings are numerical vectors that capture the meaning of text, not just its characters.
• Semantically similar texts produce vectors that are close together in embedding space.
• Cosine similarity is the most common metric for comparing embeddings.
• Word, sentence, and document embeddings serve different purposes at different granularities.
• Modern models produce contextual embeddings that adapt to surrounding words.
• Embeddings are the foundation of semantic search, RAG, and many AI applications.
• For privacy-sensitive work, open-source models can run entirely locally.

Related BestWordz Resources

• How to Build a Private Vector Store in Pure Python — Build vector search from scratch
• Running AI Locally on CPU — Set up a private local AI environment
• Model Context Protocol (MCP) Guide — Connect AI agents to data and tools
• Vector Store Tutorial — Implement cosine similarity in pure Python

Further Reading

• Word2Vec: Mikolov et al., "Efficient Estimation of Word Representations in Vector Space" (2013)
• Sentence-BERT: Reimers & Gurevych, "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks" (2019)
• all-MiniLM-L6-v2: Hugging Face Model Card
• OpenAI Embeddings: OpenAI Embeddings Guide
• Multilingual Models: Reimers & Gurevych, "Making Monolingual Sentence Embeddings Multilingual using Knowledge Distillation" (2020)