Semantic Retrieval and Relevance Ranking

Hello again. Your local index now contains normalized vectors and a position-preserving list of chunk records. This lesson makes that artifact useful: you will embed a user query, search the FAISS index, recover the corresponding source records, and inspect whether the ranked results are genuinely useful evidence.

The goal is not merely to get a list of scores. For a RAG feature, retrieval is successful only when the leading chunks are relevant to the user’s question, traceable to an appropriate source, and suitable to pass on to a later grounded-generation step. Plan for roughly 40 minutes: 8–10 minutes with the resources, 15–20 minutes modifying the notebook, and 10 minutes inspecting several searches.

---

Retrieval is nearest-neighbor search over meaning

Semantic retrieval places two things into the same embedding space:

1. Your prepared document chunks, already indexed in the previous lesson.
2. A new user query, embedded using the compatible model and preprocessing policy.

The index returns the chunk vectors nearest to the query vector. With the normalized-vector, inner-product design you built, higher returned scores mean greater cosine similarity.

The most common RAG pattern is asymmetric semantic search: a short question such as “When do we notify customers of an incident?” is used to find a longer chunk of policy text. This differs from finding a duplicate question, where both sides are similarly sized and interchangeable. That distinction affects model selection and, in some embedding frameworks, which encoding method you call.

The compatibility requirement from the previous lesson remains non-negotiable:

• Use the same embedding model for documents and queries.
• Apply the same normalization policy.
• Preserve the same vector dimension and similarity metric.
• If a model provides separate document and query encoders, rebuild document embeddings using the document method and search using the corresponding query method. Do not mix those methods into an existing generic index without rebuilding it.

For your current notebook, the safest path is to use the existing embed_texts() wrapper that produced the indexed vectors. It guarantees the query enters the same embedding space as the chunk corpus.

---

Implement a retrieval function

The function below assumes you still have these artifacts from the previous lesson:

• index: your faiss.IndexFlatIP index
• records: the sidecar list of chunk records aligned with vector positions
• manifest: the record of model, dimension, metric, and normalization policy
• embed_texts(texts): your provider-neutral embedding wrapper

Add this retrieval code to the notebook:

import faiss
import numpy as np


def embed_query_for_index(query_text, index):
    """Embed and normalize one query for the existing FAISS cosine-similarity index."""
    if not isinstance(query_text, str) or not query_text.strip():
        raise ValueError("Query text must be a non-empty string.")

    raw_vector = np.asarray(embed_texts([query_text]), dtype=np.float32)

    if raw_vector.ndim != 2 or raw_vector.shape[0] != 1:
        raise ValueError(
            f"Expected one query embedding with shape (1, dimension); got {raw_vector.shape}."
        )

    if raw_vector.shape[1] != index.d:
        raise ValueError(
            f"Query dimension {raw_vector.shape[1]} does not match "
            f"index dimension {index.d}."
        )

    if not np.isfinite(raw_vector).all():
        raise ValueError("Query embedding contains NaN or infinite values.")

    query_vector = np.ascontiguousarray(raw_vector)
    faiss.normalize_L2(query_vector)

    return query_vector


def retrieve_chunks(query_text, index, records, k=5):
    """
    Return the top-k chunk records ranked by cosine similarity.

    The index is assumed to be IndexFlatIP containing L2-normalized vectors.
    """
    if index.ntotal != len(records):
        raise RuntimeError(
            "The index and record list do not have the same number of entries."
        )

    if not isinstance(k, int) or k < 1:
        raise ValueError("k must be a positive integer.")

    if index.ntotal == 0:
        return []

    query_vector = embed_query_for_index(query_text, index)
    actual_k = min(k, index.ntotal)

    scores, positions = index.search(query_vector, actual_k)

    ranked_results = []
    for rank, (score, position) in enumerate(
        zip(scores[0], positions[0]),
        start=1,
    ):
        if position < 0:
            continue

        record = records[int(position)]
        metadata = record.get("metadata", {})

        ranked_results.append(
            {
                "rank": rank,
                "score": float(score),
                "vector_position": int(position),
                "id": record["id"],
                "text": record["text"],
                "metadata": metadata,
            }
        )

    return ranked_results

A few details in this implementation are worth noticing.

First, retrieval uses index.search(), which returns two parallel arrays:

• scores: similarity scores for the leading results
• positions: the zero-based positions of their vectors in the FAISS index

Because your records were stored in exactly the same order as the vectors, records[position] is the source record that belongs to that retrieved vector. This is the mapping that makes evidence, metadata checks, and later citations possible.

Second, the code limits $k$ to the number of vectors in the index. That avoids a confusing edge case in which an index is asked to return more results than it contains.

Third, the query is normalized immediately before searching. This mirrors the previous document-vector normalization. Without that symmetry, the scores from IndexFlatIP would no longer represent cosine similarity as intended.

Run a first retrieval using a question that should be covered by the source documents you indexed:

query = "When should customers be notified about a service incident?"

results = retrieve_chunks(
    query_text=query,
    index=index,
    records=records,
    k=5,
)

print(f"Retrieved {len(results)} results for: {query!r}")

Now make the ranking readable. Add the following display helper:

from textwrap import shorten


def show_ranked_results(results, preview_characters=450):
    for result in results:
        metadata = result["metadata"]

        print("=" * 88)
        print(
            f"Rank {result['rank']} | "
            f"score={result['score']:.4f} | "
            f"vector_position={result['vector_position']}"
        )
        print("Chunk ID:", result["id"])
        print("Document:", metadata.get("document_id", "not-recorded"))
        print("Section:", metadata.get("section_path", "not-recorded"))
        print("Version:", metadata.get("document_version", "not-recorded"))
        print("Status:", metadata.get("content_status", "not-recorded"))
        print("Text:")
        print(shorten(result["text"], width=preview_characters, placeholder=" ..."))


show_ranked_results(results)

If the top chunk looks promising, inspect it without truncation:

print(results[0]["text"])

This apparently simple act—reading the retrieved text—is a critical engineering practice. A plausible score is not evidence that the result answers the question.

---

Inspect relevance rather than trusting the score

A semantic similarity score is a ranking signal, not a probability of correctness. A score of $0.74$ does not mean “74% likely to answer correctly.” Its useful meaning is comparative: for this query, result 1 was judged more semantically similar by the embedding model than results 2 through 5.

Scores also should not be compared casually across unrelated queries. A narrow query about a specific policy clause and a broad query about incident management can produce different score distributions even when both retrieve good evidence.

For every result in the top $k$, make four quick judgments:

Check	Question to ask	Why it matters
Semantic relevance	Does this chunk address the user’s actual intent?	A matching topic may still fail to answer the question.
Answer usefulness	Would this chunk help an LLM produce a specific, supported answer?	Retrieval is a means to grounded generation, not an end in itself.
Provenance	Is the document, section, version, and status appropriate?	A stale or draft source can be semantically relevant but operationally wrong.
Ranking quality	Are the strongest chunks near the top, rather than buried below irrelevant ones?	A useful result at rank 20 is often unusable in a small-context RAG workflow.

For example, a customer-notification query might retrieve chunks about incident severity, internal escalation, customer status pages, and external communications. They may all be broadly related. The best result should specifically contain notification conditions, timing, approval, audience, or channel information. A generic severity definition may be relevant background but weak evidence for the question.

Use a small set of deliberately different queries against your own corpus:

test_queries = [
    "When should customers be notified about a service incident?",
    "Who approves external incident communications?",
    "What are the expectations for internal incident escalation?",
    "How do I request a new employee laptop?",
]

for query in test_queries:
    print("\n" + "#" * 88)
    print("QUERY:", query)

    query_results = retrieve_chunks(
        query_text=query,
        index=index,
        records=records,
        k=5,
    )

    show_ranked_results(query_results, preview_characters=260)

The first three queries should test paraphrases and nearby policy concepts. The last query is an absence test if the corpus contains no laptop policy. A good system may still return semantically adjacent chunks for an unsupported request; nearest-neighbor search must return something. That is why a later RAG workflow needs instructions for handling insufficient evidence rather than treating retrieval as proof that an answer exists.

Keep a lightweight retrieval observation log as you inspect results:

Query	Best relevant rank	Useful chunks in top 5	Observation
Customer-notification question	1	2	External-notice chunk is first; severity policy gives supporting context
Approval question	3	1	Correct section exists but is ranked below generic communication guidance
Unsupported laptop request	None	0	System returns unrelated operational policy; generation must not invent an answer

At this stage, these notes are not a formal evaluation suite. They are an engineering diagnostic habit: turn a vague concern such as “the RAG result seems off” into a query, a ranked result set, source evidence, and an observable failure pattern.

---

Recognize common retrieval failure patterns

When retrieval looks poor, avoid immediately changing prompts or models. First classify what you actually observe.

Relevant content is absent from all top results

Possible causes include:

• The source document was never ingested.
• Cleaning removed meaningful content.
• Chunking separated the relevant condition from its explanation.
• The indexed corpus is stale or incomplete.
• The question is genuinely outside the corpus’s scope.

Start by searching your raw prepared chunks directly for terms or concepts you expected. If the answer is absent before embedding, the problem is ingestion or corpus coverage—not semantic ranking.

The correct chunk appears, but too low in the ranking

This indicates a ranking-quality issue rather than complete failure. Likely contributors include broad chunks, an ambiguous query, weak embedding-model fit for the domain, or results dominated by repeated boilerplate. Keep the actual query and ranked IDs. In the next module, you will examine targeted improvements such as metadata filtering, hybrid lexical-semantic retrieval, and reranking.

Top results are semantically related but do not answer the question

This is one of the most common RAG failure modes. The embedding model recognizes topical proximity, but it does not reason about whether a chunk contains the required detail. For example, a query about “who approves notices” can retrieve many passages about notices without retrieving the approval authority.

Treat this as evidence that semantic retrieval has found a useful neighborhood, not that it has completed the user’s task.

Top results are correct but unsuitable to use

A chunk may have the right content but the wrong governance status: draft, superseded, restricted, or outside the requester’s authorization. Vector similarity cannot decide whether a user is permitted to see a source. Metadata makes the problem visible; application-level policy enforcement must decide what content may be returned or supplied to a model.

For a director-level review, this distinction matters: retrieval relevance, source quality, and access authorization are separate controls. A design that reports only “top-$k$ similarity” has not yet demonstrated that it can safely support a business workflow.

---

Key takeaways

You have now completed the retrieval stage of the first RAG pipeline:

• Embed each user query with the same compatible embedding setup used for the indexed corpus.
• Normalize the query before searching your normalized IndexFlatIP index.
• Use the FAISS-returned vector position to recover the corresponding chunk record and metadata.
• Interpret higher cosine-similarity scores as a ranking signal, not a correctness probability.
• Inspect the complete top-ranked chunks for semantic relevance, answer usefulness, provenance, and ranking quality.
• Test queries that are paraphrased, specific, broadly related, and unsupported; each exposes a different kind of weakness.
• Record observed failures before proposing retrieval changes.

Next, you will take the retrieved chunks and construct a grounded generation prompt that gives the model explicit evidence, behavior constraints, an output format, and source citations.