Skip to content

What is document metadata filtering in vector databases?

Beginner · 2 min read · RAG & Vector Databases

Open in study mode to save this question and track progress

Short answer: Document metadata filtering attaches structured key-value attributes (e.g. author, creation_date, tenant_id) to vector embeddings, constraining vector similarity searches to match specific structured criteria.

Detail

Pure vector similarity search searches across all vectors in an index. In real-world enterprise applications, searches must be restricted by tenant boundaries or document tags.

Vector Entry:
- ID: "doc_991"
- Vector: [0.012, -0.451, ...]
- Metadata: {"tenant_id": "corp_42", "department": "HR", "year": 2024}

Pre-filtering vs Post-filtering

Post-Filtering (Slow / Low Recall):
[ANN Vector Search top 100] ──► Filter out non-matching metadata ──► May return 0 items

Pre-Filtering (Fast / Exact Recall):
[Filter HNSW Graph by tenant_id=="corp_42"] ──► Perform ANN Vector Search ONLY on valid nodes

Example

Pinecone/Qdrant metadata query format in Python:

python
def format_vector_query(user_query_vector: list[float], tenant_id: str, dept: str):
    return {
        "vector": user_query_vector,
        "top_k": 5,
        "filter": {
            "tenant_id": {"$eq": tenant_id},
            "department": {"$eq": dept}
        }
    }

query = format_vector_query([0.1] * 1536, tenant_id="acme_corp", dept="Engineering")
print("Structured Vector Filter:", query["filter"])

Interview tips

  • Emphasize why pre-filtering inside vector index graph traversal is critical for multi-tenant data privacy.
  • Discuss index payload indexing (building B-tree/bitmap payload indexes alongside HNSW vector graphs).

Related Concepts

⬅ Back to RAG and Vector Databases · All topics