What a vector database does in RAG
Chunk
Split documents into passages.
Embed
Turn each passage into a vector.
Index
Store vectors for approximate nearest-neighbour search.
Retrieve
Embed the query and fetch the top k passages.
Generate
Give the passages to the model with the question.
Kinds of options
| Kind | Examples of the category | Fits |
|---|---|---|
| Extension to your database | Vector types in Postgres and other general databases | You already run that database; moderate scale |
| Dedicated vector database | Purpose-built engines, managed or self-hosted | Large scale, high query rates, filtering needs |
| Search engine with vectors | Full-text engines with vector fields | Hybrid keyword + semantic search |
| Embedded library | In-process indexes | Prototypes, edge, single-node |
Five selection questions
- How many vectors, and how fast does the corpus change?
- Do you need metadata filters (tenant, date) at query time?
- Do you need hybrid keyword + vector search?
- Who operates it, and where must data live?
- What does it cost at your query volume?
Common RAG failure modes
- Chunks split the answer across boundaries.
- An outdated document version scores higher than the current one.
- Tenant filters missing, leaking another customer’s data.
- Top-k too small for multi-part questions.
Static documents and changing history
Vector databases are a good fit for documentation and other mostly-static corpora. For a stream of events where facts change — CRM updates, tickets, conversations — pair retrieval with a memory layer that tracks current state and time.
Frequently asked questions
Can I store chat history in a vector database?
Yes, but similarity search will return old and new statements alike. You need extra logic for updates.
Does Auld replace my vector database?
For changing facts about entities, often yes; for static document search, keep it.