Semantic Retrieval with Embeddings and Similarity Measures

Hello again. In the previous lesson, you established that embeddings are numeric representations that make language comparable, but that semantic similarity is neither factual proof nor permission to use a document. We also saw why a finite context window makes it impractical and risky to place an entire knowledge base in every prompt.

This lesson turns that foundation into a retrieval model. You will be able to explain how a user’s question and a collection of document passages become vectors, how a similarity measure ranks candidate passages, and why semantic retrieval is useful—but fallible. This is the retrieval step that will later supply evidence to a grounded RAG prompt.

---

From text to a searchable geometric space

A semantic-retrieval system uses an embedding model to turn a piece of text into a fixed-length vector:

$$\mathbf{v}=[v_1,v_2,\dots ,v_n]$$

Each number is one coordinate in a space with many dimensions. The number of dimensions is model-specific and commonly far beyond what we can draw or intuit directly. What matters is not the meaning of coordinate 217 in isolation. Meaning is represented by the vector’s pattern as a whole.

Suppose an internal reliability knowledge base includes these passages:

1. “For a customer-impacting severity-one incident, the incident commander activates the executive liaison within fifteen minutes.”
2. “Engineering leadership reviews quarterly reliability trends and investment priorities.”
3. “Create an incident chat channel and assign communications ownership.”

A user searches:

“How do we notify leadership during a major production outage?”

A keyword engine may favor passages containing the literal words leadership or notify. An embedding-based retriever is designed to recognize that major production outage is related to customer-impacting severity-one incident, and that notify leadership is related to activate the executive liaison. It can therefore surface the first passage even where the wording differs.

This is the central promise of semantic retrieval: retrieve by learned contextual relatedness rather than literal word overlap alone.

The diagram is deliberately simplified. In production, documents are not laid out on a visible plane. The application stores their high-dimensional vectors and asks an index to find the nearest candidates to the query vector.

There is also an important distinction from the prior lesson:

• An LLM has internal, contextual token representations while it generates text.
• A retrieval system typically uses a dedicated embedding model to create one vector for each searchable unit of text: a title, paragraph, policy section, code fragment, or later, a carefully prepared chunk.

The first supports language generation; the second supports comparison and search. They are related ideas, but they are not interchangeable artifacts.

---

Similarity: comparing direction, not reading coordinates

Once query and document are represented as vectors in the same embedding space, the system needs a rule for saying which document is closest. A widely used choice is cosine similarity:

$$\mathrm{cosine}(\mathbf{q},\mathbf{d})=\frac{\mathbf{q}\cdot \mathbf{d}}{\Vert \mathbf{q}\Vert \Vert \mathbf{d}\Vert }$$

Here:

• $\mathbf{q}$ is the query embedding.
• $\mathbf{d}$ is a candidate document embedding.
• $\mathbf{q}\cdot \mathbf{d}$ is their dot product.
• $\Vert \mathbf{q}\Vert$ and $\Vert \mathbf{d}\Vert$ are their vector lengths.

Cosine similarity measures the angle between vectors. Vectors pointing in the same direction have a cosine similarity of $1$. Orthogonal vectors have a similarity of $0$. Negative scores are mathematically possible when vectors point in broadly opposing directions, although the range observed for a particular embedding model and corpus is an empirical matter.

The normalization in the denominator matters. It means cosine similarity focuses on the vector’s direction rather than its raw magnitude. For text retrieval, that is usually helpful: a longer document should not automatically win merely because its representation has a larger magnitude.

A retrieval service applies this calculation between the query and every candidate vector, then sorts descending by score. Conceptually:

Query	Candidate passage	Interpretation
“Notify leadership during a major outage”	“Activate executive liaison for severity-one customer impact”	Likely a strong semantic match despite different vocabulary
Same query	“Quarterly reliability review for engineering leadership”	Topically related, but may not answer the operational question
Same query	“How to request a new laptop”	Usually unrelated and should rank lower

The key word is likely. Similarity does not ask, “Does this passage answer the question completely and correctly?” It asks a narrower question: “Under this model’s learned representation, which passages are most related to this query?”

That distinction explains an important RAG failure mode: a retrieval result can be semantically close yet operationally unusable. An old escalation policy, a policy belonging to another business unit, or a discussion of the same concept at the wrong level of detail may all score well.

Cosine similarity, dot product, and Euclidean distance

You may encounter several similarity or distance measures in architecture discussions:

Measure	What it compares	Practical note
Cosine similarity	Angle between vectors	Common for text embeddings; higher score means closer directional alignment.
Dot product	Alignment including vector magnitude	Efficient and equivalent in ranking to cosine similarity when all vectors are normalized to length $1$.
Euclidean distance	Straight-line distance between vector endpoints	Lower distance means closer vectors; often used by vector indexes.

When both vectors are normalized to unit length, these measures are closely related:

$$\Vert \mathbf{q}-\mathbf{d}{\Vert }^2=2-2\mathrm{cosine}(\mathbf{q},\mathbf{d})$$

So, for normalized vectors, maximizing cosine similarity also minimizes Euclidean distance. This is why a system may describe its index as retrieving “nearest neighbors” while its application displays a cosine-style similarity score.

Do not assume, however, that every model provider normalizes vectors, or that every vector store uses the same metric by default. The embedding model, similarity metric, and index configuration form a compatible set. Changing one without checking the others can silently damage ranking quality.

---

The semantic retrieval loop

At a high level, semantic retrieval has two different time horizons.

Before users search, the system prepares the corpus:

1. Selects the searchable units of content.
2. Creates an embedding for each unit using a chosen embedding model.
3. Stores each vector alongside an identifier and metadata such as source, version, owner, date, and access classification.
4. Builds an index that can compare a query vector with the stored vectors.

When a user searches, the system performs a smaller, online operation:

1. Receives the user’s query.
2. Embeds the query with the compatible query representation.
3. Applies permission and metadata constraints where appropriate.
4. Finds the nearest vectors under the chosen similarity measure.
5. Returns the top $k$ candidate passages, their scores, and their source metadata.

At this stage, retrieval has not generated an answer. It has returned a ranked evidence candidate set.

That separation is operationally valuable. If an assistant gives a poor response, the team can inspect whether the necessary passage was retrieved at all, whether it was ranked too low, or whether the later generation step mishandled good evidence. Without that observability, every failure gets vaguely labelled “the AI was wrong.”

The following concise documentation frames the pattern and introduces a choice that matters for RAG.

---

Symmetric and asymmetric retrieval

The document’s distinction between symmetric and asymmetric search is easy to overlook, but it affects model selection.

In symmetric search, the query and candidate items have comparable form and length. For example, a user might search an archive of prior support tickets to find tickets similar to:

“My account is locked after changing my phone number.”

The desired result might be another short issue report. In principle, reversing query and candidate makes conceptual sense.

In asymmetric search, a short query searches for a longer passage that contains an answer or useful evidence. This is the normal RAG case:

Query: “Who approves a severity-one incident escalation?”
Candidate passage: “For customer-impacting severity-one incidents, the incident commander initiates escalation; the executive liaison coordinates leadership communications.”

Reversing those roles does not make much sense. The short question and the explanatory passage serve different purposes.

Embedding models may be trained for one or the other pattern. Some libraries expose separate query and document encoding operations for this reason. A Director does not need to choose a model from a benchmark table alone, but should insist that the chosen model resembles the actual retrieval task: user questions against internal passages, code questions against code snippets, or similar-ticket matching against similar tickets.

---

What semantic search solves—and what it does not

Semantic retrieval is particularly useful when language varies:

• Synonyms: “outage,” “service interruption,” and “customer-impacting incident.”
• Abbreviations: “SEV-1” and “severity one.”
• Paraphrase: “Who owns communications?” and “Who sends stakeholder updates?”
• Natural-language questions: where the relevant document contains an answer but not the exact query phrasing.

It is not a replacement for lexical search or structured controls. Semantic retrieval can struggle with:

• exact identifiers, error codes, product SKUs, or legal clause numbers;
• a meaningful negation or qualifier, such as “not eligible” versus “eligible”;
• freshness, authority, and document version;
• a passage that is related to the topic but does not answer the specific question;
• access boundaries, which similarity scores cannot enforce.

For example, a retired incident policy can be nearly identical in semantic meaning to the current policy. Its embedding may be an excellent match while the document itself is an unacceptable source. This is why metadata is not merely a convenience. Source version, business unit, effective date, and authorization scope often determine whether a candidate should be eligible for retrieval at all.

A mature system will commonly combine approaches: semantic retrieval for meaning, keyword retrieval for exact terminology, metadata filtering for authority and access, and later, a reranking or evaluation layer. You will revisit those quality improvements in the RAG-quality module. For now, retain the fundamental principle:

Vector similarity identifies potentially relevant content; the application must still determine whether that content is authorized, current, and sufficient evidence.

---

Scale, ranking, and the Director’s questions

For a small corpus, an application can compare a query against every document vector. This is exact nearest-neighbor search. As the corpus grows to hundreds of thousands or millions of passages, exhaustive comparison can become too slow or costly for an interactive user journey.

Vector indexes often use approximate nearest-neighbor methods. These search a structured subset of the vector space to return high-quality candidates quickly, accepting that an exact best match can occasionally be missed. That creates a deliberate systems trade-off among latency, cost, memory, and retrieval recall.

The implementation details can wait. At a leadership level, the relevant questions are more direct:

Review question	Why it matters
What is the retrieval unit: whole document, section, or passage?	Granularity changes relevance, citation quality, and how much context is later sent to the LLM.
Are the query and corpus encoded with a compatible model and intended search mode?	An incompatible model or query-document mismatch can produce systematically poor rankings.
What evidence and metadata accompany each result?	Teams need to inspect sources, effective dates, ownership, and authorization—not only a numeric score.
Does the system filter inaccessible or obsolete content before generation?	Retrieval must respect permissions and authority boundaries.
What proportion of representative queries retrieve the needed evidence in the top results?	Averages and anecdotes cannot establish retrieval quality.
What happens when no result is sufficiently relevant?	A safe assistant should be able to say it lacks evidence rather than force a plausible answer.

A similarity threshold is not a universal constant. A score that seems “high” in one embedding model or corpus may be weak in another. Thresholds should be calibrated using a representative set of real queries with known acceptable sources, not chosen because a number looks intuitively persuasive.

---

Key takeaways

An embedding model maps queries and document passages into vectors in a shared high-dimensional space. Semantic retrieval embeds a user query, compares it with stored document vectors, and returns the nearest, highest-scoring candidates.

Cosine similarity compares vector direction and is widely used for text embeddings. With normalized vectors, cosine similarity, dot-product ranking, and Euclidean nearest-neighbor ranking are mathematically closely related. The score is a ranking signal, not a factual-confidence score.

For RAG, retrieval is usually asymmetric: a short question searches longer passages. Retrieval returns candidate evidence, not an answer, and it must be complemented by metadata, permissions, source authority, and evaluation.

Next, you will use retrieved evidence more deliberately by constructing prompts with explicit context, instructions, constraints, and a requested output format.