Scribd
Upload
5 views1 page

03 Vector Databases and Embeddings

Vector databases are essential for efficiently storing and searching high-dimensional vectors (embeddings) used in AI applications like recommendation systems and semantic search. Embeddings are generated by models such as Sentence-BERT and OpenAI's text-embedding-ada-002, with similarity metrics like cosine similarity and Euclidean distance influencing retrieval quality. Popular vector databases include Chroma, FAISS, Pinecone, Weaviate, Milvus, and Qdrant, each with unique features and indexing strategies to optimize performance.

Uploaded by

arosan966
Copyright
© All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
5 views1 page

03 Vector Databases and Embeddings

Vector databases are essential for efficiently storing and searching high-dimensional vectors (embeddings) used in AI applications like recommendation systems and semantic search. Embeddings are generated by models such as Sentence-BERT and OpenAI's text-embedding-ada-002, with similarity metrics like cosine similarity and Euclidean distance influencing retrieval quality. Popular vector databases include Chroma, FAISS, Pinecone, Weaviate, Milvus, and Qdrant, each with unique features and indexing strategies to optimize performance.

Uploaded by

arosan966
Copyright
© All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd

Vector Databases and Embeddings

Vector databases are specialized systems designed to store, index, and search high-dimensional vectors
(embeddings) efficiently. They are foundational to modern AI applications including RAG, recommendation
systems, and semantic search.

Embeddings: Dense numerical representations of data where semantically similar items are close in vector
space. Text embeddings are generated by models like Sentence-BERT, OpenAI's text-embedding-ada-002,
or open-source alternatives like BGE and E5. Typical dimensions range from 384 to 1536.

Similarity Metrics: Cosine similarity measures the angle between vectors (most common for text). Euclidean
distance measures straight-line distance. Dot product is used when vectors are normalized. The choice
affects retrieval quality and should match the embedding model's training objective.

Approximate Nearest Neighbor (ANN): Exact search is O(n) and impractical for large datasets. ANN
algorithms trade accuracy for speed: HNSW (Hierarchical Navigable Small World) builds a multi-layer graph
for fast navigation; IVF (Inverted File Index) partitions vectors into clusters; PQ (Product Quantization)
compresses vectors for memory efficiency.

Popular Vector Databases: Chroma (lightweight, embedded, great for prototyping); FAISS (Facebook's
library, highly optimized, not a full database); Pinecone (managed cloud service); Weaviate (open-source with
GraphQL); Milvus (distributed, cloud-native); Qdrant (Rust-based, high performance).

Indexing Strategies: HNSW offers excellent speed-accuracy tradeoff with parameters M (connections per
layer) and ef_construction (build-time search width). Query-time parameter ef_search controls accuracy vs
speed. IVF uses nlist (number of clusters) and nprobe (clusters to search).

Best Practices: Normalize vectors before storage if using cosine similarity. Batch insertions are faster than
single inserts. Monitor index build time and memory usage. Regularly update embeddings as models
improve. Consider hybrid search combining dense and sparse (BM25) retrieval.

You might also like