Semantic Search

How it works

Semantic search has two phases: an offline index build and an online query.

1. Embed the corpus. Each item — a document chunk, a tool description, a past trajectory — is passed through an embedding model that maps text to a fixed-length vector. Similar meanings land near each other in that space.
2. Index the vectors. You store them in a vector index that supports approximate nearest-neighbor (ANN) search, so lookups stay fast as the corpus grows.
3. Embed the query. At request time the same model embeds the incoming query into the same space.
4. Rank by distance. You compute a similarity (usually cosine) between the query vector and candidates, then return the top-k closest.
5. Optionally re-rank. A cross-encoder or a cheaper heuristic re-scores the shortlist for precision before it reaches the model.

The defining property is that matching happens on meaning, not tokens. A query for "cancel a running job" can surface a chunk titled "aborting an in-flight task" that shares no words. The cost is that relevance is a continuous score with no natural cutoff: everything returns something, and the something is only as good as your embedding model, your chunking, and your threshold.

Why it matters in an agent harness

Inside a harness, semantic search is rarely the product — it is the thing that decides what an agent sees. That makes it a control point, not a convenience.

• It shapes context. Retrieval selects which chunks enter the prompt, and the prompt determines the agent's behavior. A bad retriever silently degrades every downstream decision, so retrieval quality is part of your evaluation surface, not an afterthought.
• It is a probabilistic component. The same query can return different neighbors as you re-embed, re-chunk, or swap models. If you want reproducible runs, you have to pin the embedding model version and index snapshot the same way you pin a model or a seed.
• It is an injection channel. Retrieved text is untrusted evidence. A poisoned document can rank highly for a benign query and carry instructions into context. Semantic relevance does not imply trust, so retrieved content still needs to be treated as data, not commands.
• It needs to be observable. When an agent does the wrong thing, the first question is "what did it retrieve?" Log the query, the returned IDs, and their scores so a run is legible after the fact.

Treat the retriever as a bounded subsystem with its own evals, its own failure modes, and its own budget.

Semantic search vs keyword search

The two are complementary, and the choice changes how you build the index and how you debug misses.

Most production retrievers run hybrid: lexical for precise identifiers, semantic for conceptual recall, then a fused rank. If your agent has to look up an exact function name or a config key, do not rely on embeddings alone — they will happily return a plausible neighbor instead of the exact match.

The Rifty take

We treat retrieval as a controllable component, not a magic recall layer. We optimize for legibility over raw top-k relevance: a retriever whose queries, IDs, and scores are logged is one you can debug and eval; one that only "feels" relevant is a liability. We accept a hybrid, more boring index over a pure-vector one because exact matches matter, and we enforce a hard boundary that retrieved text is evidence, never instruction.

Common failure modes

• No threshold. Top-k always returns k results, even when nothing is actually relevant. Add a minimum-similarity floor so "no good match" is a valid, handled outcome.
• Stale index. The corpus changed but the embeddings did not. Re-index on write, or your agent retrieves a world that no longer exists.
• Model drift on upgrade. Swapping the embedding model invalidates every stored vector; query and corpus must share one model version, or distances are meaningless.
• Chunking mismatch. Chunks too large dilute the signal; too small lose context. This is often the real cause of "bad" retrieval blamed on the model.
• Confident near-misses on identifiers. Semantic search returns a similar-looking key or error code instead of the exact one. Route exact lookups through lexical search.
• Unfiltered trust. Ranking by relevance is not ranking by trustworthiness. Keep provenance on every chunk and treat retrieved content as prompt-injection surface.
• Unbounded cost. Embedding every request and every write adds latency and spend. Cache query embeddings and give retrieval a budget like any other tool.

Frequently asked questions

What is the difference between semantic search and RAG?

Semantic search is the retrieval step; RAG is the full pattern that feeds retrieved results into a model to ground its answer. Semantic search finds relevant items by vector similarity, then RAG places them in context. You can run semantic search alone, but RAG almost always depends on some retriever underneath it.

When should I use keyword search instead of semantic search?

Use keyword search when queries hinge on exact tokens: error codes, function names, SKUs, config keys, or legal identifiers. Embeddings return plausible near-misses for these, which is worse than a clean miss. In practice most harnesses run a hybrid index so lexical precision and semantic recall cover each other's blind spots.

Does semantic search prevent hallucination?

No. Semantic search improves the odds that relevant evidence reaches the model, but it does not guarantee correctness or force the model to use what it retrieved. A bad chunk, a stale index, or an ignored result still yields wrong answers, so retrieval quality must be evaluated as its own component, not assumed.

How do I make semantic search reproducible across runs?

Pin the embedding model version and the index snapshot, and log the query, returned IDs, and scores for every retrieval. Re-embedding with a new model changes distances and results, so treat the retriever like any other stochastic component: version it, snapshot it, and make its output legible after the fact.

Is retrieved content a security risk in an agent?

Yes. Retrieved text is untrusted evidence and a prime prompt-injection channel — a poisoned document can rank highly and carry instructions into context. Semantic relevance says nothing about trust. Keep provenance on each chunk, treat retrieved content as data rather than commands, and constrain what the agent can do with it.