A vector database stores embeddings — long lists of numbers that represent the meaning of text, images, or audio — and finds the closest matches to a query in milliseconds. It exists because regular databases search for exact matches, and meaning doesn't work that way. If you ask a traditional database for "cheap flights to Tokyo" it looks for those literal words. A vector database looks for the idea, even if the source document says "affordable airfare to Japan."
This matters because every retrieval-augmented generation (RAG) system depends on it. An AI model can't hold your entire company wiki in its context window, so it needs a way to pull the ten most relevant paragraphs before answering a question. That retrieval step runs on a vector database. Get it wrong and the model hallucinates with confidence. Get it right and the model sounds like it actually read your documents.
An embedding model turns a chunk of text into a vector — typically 384 to 1536 numbers, depending on the model. Each number captures a dimension of meaning the model learned during training. Two sentences about dogs end up close together in that numerical space. A sentence about dogs and a sentence about tax law end up far apart.
The database's job is to store millions of these vectors and answer one question fast: given a new vector, which stored vectors sit closest to it? That's it. No joins, no complex schemas, no foreign keys. Just distance calculations across high-dimensional space, repeated at a scale that would choke a spreadsheet.
Most vector databases also store the original text alongside the vector, plus metadata like a document ID, a timestamp, or a source URL. The vector gets you the match. The metadata tells you what you actually found and lets you filter results before or after the search runs.
Keyword search, the kind built into most relational databases, matches strings. It's fast and cheap, and for decades it was good enough. It breaks the moment a user phrases a question differently than the document does. Search for "how to cancel my subscription" and keyword search misses a help article titled "ending your membership," because not one word overlaps.
Embeddings don't care about word overlap. They care about meaning. "Cancel my subscription" and "end my membership" land close together in vector space because the embedding model was trained on enough text to learn they mean roughly the same thing. This is the entire reason vector databases exist: they make semantic search computationally tractable at scale.
The tradeoff is real, though. Keyword search is exact and explainable — you can see exactly why a document matched. Vector search is probabilistic and a little opaque. It will sometimes return a result that's topically close but factually wrong for the question asked. That's a feature of the math, not a bug you can patch away.
A RAG pipeline runs in four steps, and the vector database only appears in the middle two. First, you chunk your source documents into pieces small enough to be useful — usually 200 to 800 tokens each. Second, you embed every chunk and store the vectors. Third, when a user asks a question, you embed the question itself and search the database for the closest chunks. Fourth, you stuff those chunks into the model's prompt and let it generate an answer grounded in real text.
The quality of step three determines everything downstream. If the database returns the wrong chunks, the model will write a fluent, confident answer based on garbage context. This is why chunking strategy and embedding model choice matter more than which specific database product you pick. Teams obsess over database benchmarks and ignore the fact that their chunks are cutting sentences in half.
Most production RAG systems also run a re-ranking step after the vector search, using a smaller, slower model to re-score the top 20 or so candidates before picking the final 5. Vector search is fast but approximate. Re-ranking trades speed for precision on a shortlist, and it catches a lot of the near-miss errors that pure vector similarity lets through.
There are four categories worth knowing, and the right pick depends almost entirely on scale and what you already run.
| Option | Best for | Main drawback |
|---|---|---|
| pgvector (Postgres extension) | Teams already on Postgres, under ~10M vectors | Slower at very large scale than purpose-built engines |
| Pinecone | Fast production launch, managed infrastructure | Ongoing cost scales with vector count |
| Weaviate | Hybrid search (keyword + vector) out of the box | Heavier to self-host and tune |
| Milvus / Zilliz | Billion-scale vector workloads | Operational complexity is real |
Here's the opinion most vendors won't tell you: if you're under ten million vectors, pgvector handles it fine, and it saves you from running a separate database just to store numbers. Teams jump to a dedicated vector database before they've proven they need one, usually because a blog post told them Postgres "doesn't scale." For the vast majority of internal tools, customer-support bots, and document search projects, it scales plenty.
Once you cross tens of millions of vectors, or you need sub-20-millisecond latency under heavy concurrent load, a purpose-built engine starts earning its keep. That's the point where index types like HNSW and IVF, which are built into dedicated vector databases and tuned for exactly this problem, start to matter more than convenience.
Not every AI feature needs semantic search. If your dataset is small enough to fit in a model's context window, skip the vector database entirely and just paste the whole thing into the prompt. A 50-page policy document fits comfortably in most modern context windows, and retrieval adds latency and failure points for no real benefit at that size.
If your search problem is actually a keyword problem in disguise — users searching for exact product SKUs, order numbers, or legal citations — a vector database will underperform a plain old index. Exact-match problems want exact-match tools. Forcing every search feature through an embedding model because it's the trendy architecture is how teams end up with slower, worse search than what they replaced.
Embedding every document costs money, and re-embedding costs money again every time you change embedding models. Switching from one embedding model to a better one doesn't just mean swapping an API call. It means every vector already stored becomes incompatible, because different models place meaning in different numerical spaces. You have to re-embed your entire corpus and rebuild the index from scratch.
Storage adds up too, though less dramatically than compute. A million 1536-dimension vectors takes roughly 6GB just for the raw numbers, before indexes and metadata. That's manageable for most businesses, but it's not free, and managed vector database pricing often charges per vector stored plus per query, which compounds faster than teams expect once a product takes off.
The real cost is maintenance attention, not dollars. A vector index degrades in quality as your source documents change and nobody updates the embeddings. Stale vectors quietly return stale answers, and because the failure mode is a slightly-wrong answer rather than a crash, nobody notices until a customer complains. Treat the embedding pipeline like you'd treat any other data pipeline: monitored, scheduled, and owned by someone whose job it is to keep it current.
A vector database stores numerical representations of text, images, or audio called embeddings and finds the closest matches to a new query. It powers semantic search, recommendation systems, and retrieval-augmented generation (RAG) pipelines for AI models. The core use case is finding content that means the same thing as a query, even when the wording is completely different.
Not necessarily. If your dataset is under a few million chunks, a Postgres extension like pgvector handles the job without adding a separate system to maintain. Dedicated vector databases earn their cost at larger scale or when you need very low query latency under heavy concurrent load.
A regular database matches exact values, like a specific ID or a literal string. A vector database measures distance between numerical representations of meaning, returning results that are conceptually similar even if no words overlap. Regular databases excel at structured lookups; vector databases excel at fuzzy, meaning-based search.
AI models have a limited context window and can't hold an entire knowledge base in a single prompt. A vector database lets the system retrieve only the most relevant chunks of text for a given question before passing them to the model. Without this retrieval step, the model either runs out of room or answers from memory alone, which increases hallucination risk.
Switching embedding models makes every previously stored vector incompatible, because different models place meaning in different numerical spaces. You have to re-embed the entire dataset and rebuild the vector index from scratch. This is a real operational cost, so it's worth picking an embedding model carefully before building a large production index around it.