Creating Embeddings and Building a Simple Vector Index

Hello. Last lesson turned raw sources into coherent, traceable chunk records: each chunk has text suitable for retrieval plus metadata that preserves its source, section, version, and governance context. Now we make those records searchable by converting their text into embeddings and placing the vectors in a small local index.

By the end of this lesson, you will be able to build a simple, inspectable RAG index in Python: embed every prepared chunk, validate the resulting vectors, store them in a FAISS index, and maintain the mapping from each vector back to its chunk record. Plan for about 40 minutes: a short visual review, 8–10 minutes of reading, about 20 minutes implementing the notebook pattern, and a final review of the operational choices.

---

What an index actually stores

An embedding model maps a text chunk to a fixed-length list of numbers, called a vector. The individual numbers do not have human-readable meanings. What matters is the model’s learned geometry: texts with related meanings should be positioned near one another in a high-dimensional space.

The prior lesson gave us chunk records such as:

prepared_chunk = {
    "id": "incident-comms:v3:external-notices:002",
    "text": "Incident Communications Standard ...",
    "metadata": {
        "document_id": "incident-comms",
        "section_path": "Communications > External Customer Notices",
        "document_version": "3.0",
        "content_status": "approved",
    },
}

This lesson adds a vector representation of text. Conceptually, the process has four parts:

1. Keep the chunks in a stable, deliberate order.
2. Send the chunk texts to one embedding model.
3. Validate and normalize the resulting vectors.
4. Add the vectors to an index while preserving the connection between vector position and chunk record.

The index does not make the source content authoritative or correct. It only makes a collection of embeddings efficient to compare. A wrongly parsed, outdated, or unauthorized chunk can still be retrieved very efficiently. That is why the preparation and metadata work from the previous lesson remains essential.

A useful way to think about the implementation is that the vector index and the chunk records are a matched pair:

Asset	Holds	Purpose
FAISS index	Numeric vectors	Finds close vectors efficiently
Chunk records	Text, IDs, metadata, source references	Lets the application display evidence, apply policy, and cite the source
Index manifest	Model, dimension, metric, normalization, pipeline version	Makes the index reproducible and diagnosable

For a first notebook implementation, the key invariant is simple: vector position $i$ must always correspond to chunk record $i$.

---

The geometry choice: cosine similarity with normalized vectors

Suppose a chunk has embedding vector $v$, and a future user query has embedding vector $q$. A common measure of semantic similarity is cosine similarity:

$$\mathrm{cosine}(q,v)=\frac{q\cdot v}{\Vert q\Vert \Vert v\Vert }$$

It measures directional alignment rather than raw vector length. If we normalize every vector to unit length,

$$\hat{v}=\frac{v}{\Vert v\Vert }$$

then the inner product of normalized vectors equals their cosine similarity:

$$\hat{q}\cdot \hat{v}=\mathrm{cosine}(q,v)$$

This gives a clean implementation choice:

• normalize all document vectors with L2 normalization;
• use a FAISS inner-product index, IndexFlatIP;
• later, normalize query vectors the same way before searching;
• interpret larger scores as more similar.

FAISS’s simplest documented example uses IndexFlatL2, which compares vectors using squared Euclidean distance. That is also a legitimate choice, but smaller distances are better. For unit-normalized vectors, Euclidean distance and cosine similarity give the same ranking because:

$$\Vert \hat{q}-\hat{v}{\Vert }^2=2-2(\hat{q}\cdot \hat{v})$$

The important leadership-level point is not that one metric is universally superior. It is that the embedding model, preprocessing, normalization policy, and similarity metric are a compatibility contract. They must be documented and applied consistently.

FAISS supports many index types. For this first implementation, use an exact flat index:

• Flat means it compares a query against every stored vector.
• It is straightforward to inspect and has no approximation-induced recall loss.
• It requires no training phase.
• It becomes expensive at large scale, but it is exactly the right baseline for a small corpus, proof of concept, or evaluation dataset.

Do not prematurely optimize this notebook with an approximate index. Before changing index types, a team should know that retrieval quality, corpus size, latency, and cost actually justify that complexity.

---

Build the index from prepared chunks

The following code assumes the previous notebook already contains an embed_texts(texts) wrapper. That wrapper should call your selected embedding provider or local model and return one numerical vector per input text, in exactly the same order as the input list. Keep credentials in environment variables or the provider’s credential mechanism; neither credentials nor API keys belong in the notebook or index files.

Install FAISS in the notebook environment if it is not already available:

# In a notebook cell, if needed:
# %pip install faiss-cpu

Now create the index. This version deliberately performs validation before modifying the index, which makes failure easier to diagnose.

import json
from pathlib import Path

import faiss
import numpy as np


def create_vector_index(prepared_chunks, embedding_model_id):
    """
    Creates an exact cosine-similarity index over prepared chunk text.

    Assumes embed_texts(texts) has already been defined in the notebook.
    It must return one embedding vector per input text, in input order.
    """
    if not prepared_chunks:
        raise ValueError("Cannot create an index from an empty chunk list.")

    texts = [chunk["text"] for chunk in prepared_chunks]

    if any(not text.strip() for text in texts):
        raise ValueError("Every indexed chunk must contain non-empty text.")

    # Embed all chunk texts. The client wrapper should batch internally if needed.
    vectors = np.asarray(embed_texts(texts), dtype=np.float32)

    # Validate the embedding response before creating an index.
    if vectors.ndim != 2:
        raise ValueError(
            f"Expected a two-dimensional embedding matrix; got shape {vectors.shape}."
        )

    if vectors.shape[0] != len(prepared_chunks):
        raise ValueError(
            "The embedding client returned a different number of vectors than texts."
        )

    if not np.isfinite(vectors).all():
        raise ValueError("Embeddings contain NaN or infinite values.")

    norms = np.linalg.norm(vectors, axis=1)
    if np.any(norms <= 1e-12):
        raise ValueError("At least one embedding has near-zero length.")

    # Unit-normalize in place. Inner product now acts as cosine similarity.
    vectors = np.ascontiguousarray(vectors)
    faiss.normalize_L2(vectors)

    dimension = vectors.shape[1]

    # Exact search: higher inner-product score means more semantically similar.
    index = faiss.IndexFlatIP(dimension)
    index.add(vectors)

    # Keep source records in precisely the same order as the added vectors.
    records = [
        {
            "id": chunk["id"],
            "text": chunk["text"],
            "metadata": chunk["metadata"],
        }
        for chunk in prepared_chunks
    ]

    if index.ntotal != len(records):
        raise RuntimeError("Index and chunk-record counts do not match.")

    manifest = {
        "embedding_model_id": embedding_model_id,
        "embedding_dimension": dimension,
        "similarity_metric": "cosine via normalized vectors and inner product",
        "normalization": "L2 unit normalization",
        "index_type": "faiss.IndexFlatIP",
        "chunk_count": len(records),
        "pipeline_versions": sorted(
            {
                record["metadata"].get("pipeline_version", "not-recorded")
                for record in records
            }
        ),
    }

    return index, records, manifest

Run it using your chunk list and the exact identifier for the embedding model configured in your client wrapper:

EMBEDDING_MODEL_ID = "replace-with-your-configured-embedding-model"

index, records, manifest = create_vector_index(
    prepared_chunks=prepared_chunks,
    embedding_model_id=EMBEDDING_MODEL_ID,
)

print(f"Indexed chunks: {index.ntotal}")
print(f"Embedding dimension: {manifest['embedding_dimension']}")
print(f"Similarity policy: {manifest['similarity_metric']}")

A successful run confirms several things at once:

• all chunk texts were non-empty;
• the embedding client returned a rectangular matrix;
• every vector has the same dimension;
• the vectors are valid finite float32 values;
• the index contains exactly one vector for each source record.

Why the code keeps records outside FAISS

IndexFlatIP stores vectors and internally assigns their positions in the order they are added: the first vector has position 0, the second position 1, and so on. It does not understand your chunk["id"], document title, or data classification.

That is why this must remain true:

assert index.ntotal == len(records)
assert records[0]["id"] == prepared_chunks[0]["id"]

When the next lesson searches the index, FAISS will return positions such as 12, 4, and 31. The application will use those positions to recover records[12], records[4], and records[31].

For a small prototype, parallel ordering is transparent and adequate. In a production design, explicit vector IDs and a durable metadata store are usually preferable. But the underlying requirement does not change: there must be a reliable, testable path from each returned vector to the exact source chunk and its access-relevant metadata.

---

Verify, persist, and govern the index as one artifact

Before using an index for retrieval, carry out a small structural smoke test. This is not yet a relevance evaluation; it simply verifies that the stored vector matches what was added.

first_stored_vector = index.reconstruct(0)

assert first_stored_vector.shape == (manifest["embedding_dimension"],)
assert np.allclose(first_stored_vector, first_stored_vector.astype(np.float32))

print("Smoke test passed: the first vector is present with the expected dimension.")
print("First chunk ID:", records[0]["id"])
print("First source section:", records[0]["metadata"].get("section_path"))

The test above checks storage mechanics. It does not establish that the retrieved answer will be relevant. Relevance depends on chunking, source quality, model choice, query formulation, metric choice, and the actual corpus. The next lesson will search this index and inspect ranked evidence.

For a notebook that you want to reopen later, persist the binary index and its sidecar data together:

output_dir = Path("rag_artifacts")
output_dir.mkdir(exist_ok=True)

faiss.write_index(index, str(output_dir / "chunks.faiss"))

with open(output_dir / "chunks_records.json", "w", encoding="utf-8") as file:
    json.dump(
        {
            "manifest": manifest,
            "records": records,
        },
        file,
        indent=2,
    )

print("Saved index and chunk records to:", output_dir)

Treat these files as a single versioned release artifact. Replacing chunks.faiss without updating chunks_records.json breaks the vector-to-source mapping. Updating records without rebuilding the index can create equally serious provenance errors.

A lightweight decision record for an initial implementation might state:

Decision	Initial choice	Why it matters
Embedding model	One named, versioned model	Queries and documents must occupy the same vector space
Vector type	float32	Required by the basic FAISS workflow and efficient in memory
Similarity	Cosine via normalization and inner product	Scores are intuitive: higher means more similar
Index type	IndexFlatIP	Exact, no training, easy to validate
Metadata linkage	Record list aligned to vector position	Enables citations, debugging, and policy checks
Persistence	Index plus JSON sidecar and manifest	Prevents silent model, corpus, and provenance drift

There are also several guardrails worth carrying forward:

• Do not mix embedding models in one index. A vector from a different model may have a different dimension or an incompatible semantic space.
• Embed future queries with the same model and normalization policy used for indexed chunks.
• Do not rely on an index for authorization. Metadata may identify a chunk as internal or restricted, but application-level access checks must occur before content reaches the user or LLM.
• Rebuild after material corpus or chunking changes. A revised chunking policy creates a different retrieval corpus, not merely a metadata update.
• Record the model and pipeline versions. When retrieval quality changes, these details turn a vague incident into an investigable change history.

---

Key takeaways

You have now moved from readable chunk records to a searchable retrieval asset:

• An embedding is a fixed-length numerical representation of chunk text; its usefulness comes from the relative geometry among vectors.
• A simple vector index stores vectors, not the source text and governance context needed to use those vectors responsibly.
• FAISS expects a two-dimensional float32 matrix in which every row has the same embedding dimension.
• L2-normalizing vectors and using IndexFlatIP gives cosine-similarity ranking, where higher scores indicate closer semantic matches.
• The index and its sidecar chunk records must preserve a strict position-by-position mapping.
• The embedding model, dimension, metric, normalization policy, chunking version, and corpus contents should be captured in a manifest.
• An exact flat index is an appropriate baseline because it is easy to inspect and has no approximate-search behavior to obscure retrieval quality.

Next, you will implement semantic retrieval against this index and inspect the ranked chunk records for relevance, provenance, and potential retrieval failures.