Vector search tuning: measure first, then tune one setting at a time

TL;DR: Pick the index by memory budget: HNSW if you have RAM, IVF with compression if you don't. Build a test set from real queries, including hard ones. Change one setting at a time, tune ef_search last, and track p99 latency and recall together.

Vector search looks easy. You load your embeddings, run a query and get results back. Then traffic grows, the corpus changes, and some queries take far too long.

Tuning fixes that. The trick is to measure first, change one thing at a time, and test the way you'll run in production.

Here's the short version. Pick the index type that fits your memory budget. Build a proper test set. Then tune one setting at a time.

Define "better" before you touch a setting

Three numbers matter.

• Recall. How many of the true nearest neighbours your search returns.
• Latency. Track p50, p95 and p99, not the average. The slow tail is what users feel.
• Throughput and memory. Queries per second under load, plus RAM and index size.

Raising one usually costs another. So write down your targets first. Then build a test set from real production queries, with the filters you really use.

Automated tuning can beat guesswork by a long way. The VDTuner paper reports a 14.12% faster search and a 186.38% better recall rate than default settings. That's one study on its own test setups, so treat it as a sign of what's possible, not a promise.

Pick the index type first

The index type decides which settings you even get. Our guide to vector databases covers the basics. Here are the two you'll meet most.

• HNSW. A graph index. It gives high recall at low latency when you have the RAM to hold it. It also handles new inserts without a full rebuild.
• IVF. It splits vectors into clusters and only searches the closest few. Compressed versions, like IVFPQ, use far less memory and cost you some recall.

Ask three questions. How much memory can the index use? Does your data change daily? How long can a rebuild take?

My rule of thumb is simple. If you have the memory, start with HNSW. If memory is tight and you can live with a little less recall, use IVF with compression.

The settings that matter

Each index gives you a few dials.

• M (HNSW). Connections per node. Higher means better recall, plus more memory and a slower build.
• ef_construction (HNSW). How hard the build works. Higher means a better graph and a longer build.
• ef_search (HNSW). How many candidates each query checks. Higher means better recall and slower queries.
• nlist and probes (IVF). How many clusters you make, and how many you check per query.

One of these is cheap to change. You can adjust ef_search any time, because it's a query-time setting. M and ef_construction need a rebuild. So lock those in carefully, then tune ef_search last.

The settings also interact. Research on constraint-aware HNSW tuning shows that M, ef_construction and ef_search don't behave like independent knobs. That's why a blind grid search wastes so many builds.

The method in that paper, called CHAT, works in stages. It picks a few candidate values for M. It drops any ef_construction value that would break your memory or build-time budget, before building anything. It builds only what's left. Then it runs a binary search on ef_search.

You can copy the idea by hand. Never test ef_construction across a wide range. Each value needs a fresh build, so cut the list down with a quick resource estimate first.

Memory is where production breaks

Compression helps. Product and scalar quantisation shrink vectors, and both cost some recall. Half-precision floats sit in the middle: a smaller memory hit than heavy compression, with a smaller recall loss. Embedding quality matters too. Our post on semantic search explains why.

The bigger trap is disk. When the index doesn't fit in RAM, cold queries read from storage. People running pgvector have reported IVFFlat throughput dropping sharply once the index outgrew memory.

A few habits help:

• Prewarm the index after a restart, so the first users don't pay the cold-cache cost.
• Size the server for the index. The index can use more memory than the raw vectors.
• Split large data sets so the hot part stays in memory.
• Set defaults on the index, not just in the session. A session setting is easy to forget.

Test with hard queries

Average numbers hide the queries that hurt. Research on hardness-controlled benchmark generation shows that how a benchmark is built can change which method looks best. Two settings that look the same on easy queries can split apart on hard ones.

So build your test set with care.

• Pull queries from production logs where you can.
• Split the set. One part for tuning, one part held back to confirm the result.
• Keep a small "hard query" group. A few tough queries often drive your p99.
• Rebuild the set when the data or the embedding model changes.

Retune when the data drifts, not on a calendar. Changing the embedding model is a trigger on its own, because the shape of your vectors changes.

When to use an automated tuner

Once you've got M, ef_construction, ef_search, nlist, probes and compression all in play, hand tuning gets slow. Tools like VDTuner prune bad options early and search more cleverly than a grid. MINT covers the harder case of multi-vector search.

Three rules before you trust one:

• Check it understands your database's settings.
• Set a budget up front, in builds or hours.
• Always check its answer against your own test set.

Treat the output as a strong starting point, not the final call.

A tuning recipe you can run this week

1. Set targets and build your test set, including the hard group.
2. Record a baseline: recall, p50, p95, p99, index size and build time.
3. Change one setting at a time and log the result in one table.
4. Binary search ef_search on an index you've already built.
5. If gains flatten out, run an automated tuner inside a fixed budget.
6. Set per-index defaults, then alert on p99 and recall drift.

Decide your rollback rule first. If p99 breaches your limit or recall drops below your floor, go back to the last good setup. Keep the last three good configs on hand. Roll changes out to a slice of traffic before everyone gets them.

Also remember what sits downstream. Retrieval feeds your RAG pipeline, and a small recall drop can mean the model gets the wrong documents. Good chunking helps just as much as index tuning. If your agents store memory in vectors, agent memory design is the next read. For more on the group's take, see vector databases for RAG and what a vector database is.

What I'd do first

Most advice chases recall. I'd chase tail latency and memory stability, because those cause the real incidents. And I'd build the hard-query test set before touching a single setting. Every other choice is only as good as the set you judge it against.

Production search also needs tests around it. This piece on missing tests in AI-built code makes the case well.

If your search layer needs to go from prototype to something your business can run on, our AI platform development team can help. We also build the apps around it through AI app development.

Frequently asked questions

What is vector search?

Vector search finds the items whose embeddings sit closest to your query's embedding. It uses a distance measure like cosine similarity instead of exact keywords. Approximate indexes such as HNSW and IVF make it fast at scale by trading a little accuracy for a lot of speed.

Should I use HNSW or IVF?

Use HNSW when you have the memory and want high recall at low latency. Use IVF with compression when memory is tight and a small recall drop is fine. Your memory budget and how often your data changes decide it.

Which HNSW setting is safe to change after the index is built?

ef_search. It's a query-time setting, so you can change it without a rebuild. M and ef_construction shape the graph, so changing them means building the index again.

How often should I retune vector search?

Retune when your data or query mix drifts, or when you change the embedding model. A fixed calendar date is a weak trigger. Track p99 latency and recall and let drift tell you.