What is an embedding model and how does vector dimension affect search quality?

Short answer: An embedding model converts unstructured text into dense numerical floating-point vectors ($\mathbb{R}^d$) capturing semantic meaning; higher embedding dimensions (e.g. 1536 or 3072) store finer semantic granularity but increase vector database storage, memory, and indexing latency.

Detail

Embedding models place text chunks into an $N$-dimensional semantic vector space where semantically similar concepts reside physically close to each other.

"Machine learning algorithm" ──► Embedder ──► [0.012, -0.045, 0.891, ... 1536 Dimensions]
"Artificial intelligence code" ──► Embedder ──► [0.015, -0.041, 0.885, ... 1536 Dimensions]

Dimensionality Trade-Offs

Matryoshka Representation Learning (MRL)

Modern embedding models support Matryoshka embeddings, allowing developers to truncate 1536-dim vectors down to 256 dimensions with $<2\%$ drop in retrieval accuracy.

Example

Truncating Matryoshka embeddings in Python:

import numpy as np

# Simulate a 1536-dimensional L2-normalized vector
full_vector = np.random.randn(1536)
full_vector /= np.linalg.norm(full_vector)

# Truncate to 256 dimensions and re-normalize
truncated_vector = full_vector[:256]
truncated_vector /= np.linalg.norm(truncated_vector)

print("Original Dim:", len(full_vector), "| Truncated Dim:", len(truncated_vector))

Interview tips

• Explain memory math: 1 million 1536-dim vectors in FP32 require $1,000,000 \times 1536 \times 4\text{ bytes} \approx 6.14\text{ GB}$ of raw VRAM/RAM.
• Discuss scalar quantization (SQ8 / SQ4) to compress vector RAM requirements by $4\times$.

Related Concepts

• [[How does GGUF format enable quantization and CPU/GPU offloading in llama.cpp?]] (#147): How does GGUF format enable quantization and CPU/GPU offloading in llama.cpp?
• [[How to detect semantic drift in user queries using embedding clustering over time?]] (#167): How to detect semantic drift in user queries using embedding clustering over time?

⬅ Back to RAG and Vector Databases · All topics