Text Embeddings: Capturing Semantic Meaning Numerically

Welcome back. In our last lesson, we established the critical difference between an agent's fleeting short-term memory and its durable long-term memory. You learned that to build truly capable agents for tasks like e-commerce analysis, we need a way to create a persistent, searchable knowledge base. We concluded by noting that this long-term memory can store facts (semantic), events (episodic), and skills (procedural).

This lesson marks our first step in building that long-term memory system. Before an agent can "remember" a product catalog or a customer's history, we must first convert that human-readable text into a format a machine can understand and compare. This lesson focuses on that translation process. Our learning outcome is to explain how text embeddings capture semantic meaning as numerical vectors. This concept is the bedrock of modern information retrieval and the Retrieval-Augmented Generation (RAG) architecture we aim to build.

From Words to Numbers

At their core, machine learning models, including the Large Language Models we've been using, are mathematical functions. They operate on numbers, not on the abstract concepts of words and sentences. The fundamental challenge, then, is to represent text numerically in a way that preserves its meaning.

A text embedding is the solution to this problem. It is a process that converts a piece of text—a word, a sentence, or an entire document—into a list of numbers called a vector. This isn't just a simple encoding; the vector is designed to capture the semantic essence or meaning of the text.

The following video from IBM Technology provides a concise, high-level introduction to this idea.

{"type":"video","title":"What are Word Embeddings?","learning_duration":42,"video_id":"wgfSDrqYMJ4","par_intro":"This video defines word embeddings and explains why they are necessary for machine learning applications.","par_directions":"Please watch the first part of the video, from the beginning until <span data-type=\"resource_video_timerange\" data-resource-subitem-id=\"7f0da8da\" data-range-start=\"0\" data-range-end=\"42\">the explanation</span> of why we need to transform words into numbers.","video_duration":518,"isV2":true,"blockId":"50f39ff6-5c5d-4da1-a43f-a6489d45d7fa","lessonId":"089649e4-7d07-41c0-ac9d-5a6f72db85a4"}

As the video explains, embeddings are required because ML algorithms cannot process raw text. They need numerical input. The magic of embeddings lies in how they assign these numbers.

The Geometry of Meaning

The power of embeddings comes from the structure of the "space" they create. This is a high-dimensional space where each dimension corresponds to one of the numbers in the vector. While we can't visualize the hundreds or thousands of dimensions used in practice, the core principle is simple: distance in this space represents semantic similarity.

Texts with similar meanings will have vectors that are close to each other. Texts with different meanings will have vectors that are far apart.

{"type":"image","url":"https://www.scribbledata.io/wp-content/uploads/2023/06/word-vectorization-12.png","caption":"This diagram illustrates the core concept of embeddings. Words are converted into high-dimensional vectors. After a process called dimensionality reduction (for visualization only), we can see that semantically related words like \"cat\" and \"kitten\" are positioned close together in the 2D space.","isV2":true,"blockId":"4ac1bbdf-391a-48ee-8377-b39757897c55","lessonId":"089649e4-7d07-41c0-ac9d-5a6f72db85a4"}

The Stack Overflow blog post "An intuitive introduction to text embeddings" offers one of the best explanations of this concept.

{"type":"reading","par_intro":"This article breaks down the core concepts of embeddings in a clear, developer-friendly way.","par_directions":"First, read the section <span data-type=\"resource_reading_textrange\" data-resource-subitem-id=\"9a78decf\" data-range-start=\"What’s an embedding?\" data-range-end=\"to the user’s message?\">What's an embedding?</span> for a formal definition.\n\nNext, read the section on **Distance**. Pay close attention to the <span data-type=\"resource_reading_textrange\" data-resource-subitem-id=\"5f0702f8\" data-range-start=\"Imagine a two-dimensional\" data-range-end=\"(dogx, caty) coordinate system.\">library analogy</span>—it's a great way to build intuition for how proximity equals similarity. Also, note the two key metrics mentioned for measuring this proximity: <span data-type=\"resource_reading_textrange\" data-resource-subitem-id=\"d8e494f5\" data-range-start=\"In this case\" data-range-end=\"cosine distance.\">Euclidean and cosine distance</span>. Cosine distance (or cosine similarity) is especially common in this field.","learning_duration":"10 minutes","url":"https://stackoverflow.blog/2023/11/09/an-intuitive-introduction-to-text-embeddings/","title":"An intuitive introduction to text embeddings","isV2":true,"blockId":"9f48a987-7236-4388-ac21-42cc59f5e627","lessonId":"089649e4-7d07-41c0-ac9d-5a6f72db85a4"}

This geometric relationship allows for a form of "vector arithmetic" that captures complex semantic relationships. The most famous example is the equation: vector('king') - vector('man') + vector('woman') ≈ vector('queen').

By subtracting the concept of 'man' from 'king', we are left with a vector representing 'royalty'. Adding the vector for 'woman' to this 'royalty' vector moves us to the point in the space that represents 'queen'. This demonstrates that the embedding space isn't just a random assortment of points; it has a rich, learned structure.

The following video segment explains this vector arithmetic with a simple 2D visualization.

{"type":"video","title":"Tokens vs Embeddings – what are they + how are they different?","learning_duration":119,"video_id":"izbifbq3-eI","par_intro":"This video from Annie Sexton provides a clear, animated explanation of vector math with embeddings.","par_directions":"Watch the segment from <span data-type=\"resource_video_timerange\" data-resource-subitem-id=\"e4576c18\" data-range-start=\"30\" data-range-end=\"149\">the beginning of the explanation</span> on what embeddings are, focusing on the `king - man + woman = queen` example.","video_duration":412,"isV2":true,"blockId":"f5579cae-2ddd-40d4-ba08-df533b34e951","lessonId":"089649e4-7d07-41c0-ac9d-5a6f72db85a4"}

{
  "type": "exercise",
  "id": "bfe2dbe5-cfa9-43fb-aea9-8a17cd3d9f4b"
}

How are Embeddings Created?

So how does a model learn to place words in this meaningful geometric space? The process has evolved significantly.

Early methods like bag-of-words simply counted word occurrences. This was a crude approach that treated "dog bites man" and "man bites dog" as the same, as it ignored word order. A step up from this was Latent Semantic Analysis (LSA), which used linear algebra techniques (specifically, singular value decomposition) to find "latent" or hidden topics within a collection of documents.

Modern embedding models, however, are based on neural networks. These models are trained on vast amounts of text from the internet. The general training process, often using a method called triplet loss, works like this:

1. The model is given three pieces of text: an anchor, a positive (semantically similar to the anchor), and a negative (dissimilar to the anchor).
2. The model generates an embedding for each of the three texts.
3. The training objective is to adjust the model's internal parameters to minimize the distance between the anchor and positive embeddings while maximizing the distance between the anchor and negative embeddings.
4. By repeating this process billions of times with different texts, the model learns to organize its embedding space to reflect the complex semantic relationships in human language.

The Stack Overflow article you read earlier provides a great high-level progression of these techniques, from LSA through to the Transformer architecture that powers modern LLMs.

{"type":"reading","par_intro":"This reading will walk you through the evolution of embedding generation techniques.","par_directions":"Start by reading about <span data-type=\"resource_reading_textrange\" data-resource-subitem-id=\"1bc05065\" data-range-start=\"Latent information\" data-range-end=\"fancy linear algebra, basically)\">latent information and LSA</span> to understand the pre-neural network approach.\n\nThen, move to the section on <span data-type=\"resource_reading_textrange\" data-resource-subitem-id=\"3a05a322\" data-range-start=\"Neural methods\" data-range-end=\"useful signal.\">Neural methods</span>, which explains the modern training paradigm like triplet loss. Pay attention to the description of <span data-type=\"resource_reading_textrange\" data-resource-subitem-id=\"816d357c\" data-range-start=\"Word2vec\" data-range-end=\"a surprisingly rich latent space.\">Word2vec</span>, a classic model that learned word embeddings by analyzing words in a \"sliding window.\"\n\nFinally, read the section on <span data-type=\"resource_reading_textrange\" data-resource-subitem-id=\"35566da9\" data-range-start=\"Dealing with sequences\" data-range-end=\"the models just keep getting better and smarter.\">Dealing with sequences</span>. This is the most important part, as it connects embeddings directly to the models you are using. It traces the path from simple Recurrent Neural Networks (RNNs) to the **Transformer architecture** and its **attention mechanism**, which is the core innovation behind models like GPT.","learning_duration":"15 minutes","url":"https://stackoverflow.blog/2023/11/09/an-intuitive-introduction-to-text-embeddings/","title":"An intuitive introduction to text embeddings","isV2":true,"blockId":"d4ac7ea8-d7ac-4096-ba21-508f06fce6a1","lessonId":"089649e4-7d07-41c0-ac9d-5a6f72db85a4"}

The Crucial Role of Context

A key limitation of early models like Word2Vec was that they produced a single, static embedding for each word. The word "bank" would have the same vector whether it appeared in "river bank" or "investment bank."

Modern models, built on the Transformer architecture, generate contextual embeddings. The vector representation for a word changes based on the surrounding words in the sentence. This is what allows LLMs to disambiguate meaning so effectively. The model doesn't just embed the word "bank"; it embeds the phrase "I deposited money at the bank," and the resulting vector for "bank" is pushed towards the 'finance' region of the embedding space.

{"type":"image","url":"https://tutlinks.com/wp-content/uploads/2025/12/compressed-A-Visual-Guide-to-Vector-Embeddings.png-scaled.jpg","caption":"This infographic summarizes the core concepts and applications of vector embeddings. The top section illustrates how unstructured data is converted into vectors where similar concepts are grouped and mathematical relationships can be exploited.","isV2":true,"blockId":"fbe81839-b325-41eb-9104-c3abfdf8daaa","lessonId":"089649e4-7d07-41c0-ac9d-5a6f72db85a4"}

{
  "type": "exercise",
  "id": "d3fb5f98-38b0-4bf3-886b-13aa32b71186"
}

Tokens vs. Embeddings

In previous lessons, you encountered the term "tokens" in the context of API pricing and context window limits. It's vital to understand the distinction between a token and an embedding.

• A token is a piece of a word (like embed and ##ding) that is assigned a unique numerical ID from a vocabulary list. Token ID: 1037 might represent the word a. It is just an identifier and carries no semantic meaning itself.
• An embedding is the rich, multi-dimensional vector that represents the meaning of that token, often in the context of the surrounding tokens.

The token is the input to the model, which then looks up a starting embedding for that token and subsequently uses the attention mechanism to refine that embedding based on the full context of the prompt.

The video you watched earlier has a perfect segment clarifying this distinction.

{"type":"video","title":"Tokens vs Embeddings – what are they + how are they different?","learning_duration":200,"video_id":"izbifbq3-eI","par_intro":"This video clearly distinguishes between tokens and embeddings and explains how context is used to generate the final, meaningful vector.","par_directions":"Please watch the segment from <span data-type=\"resource_video_timerange\" data-resource-subitem-id=\"854a3699\" data-range-start=\"167\" data-range-end=\"231\">Tokens vs Embeddings</span>, and then continue watching the final, crucial part of the explanation from <span data-type=\"resource_video_timerange\" data-resource-subitem-id=\"cff38f66\" data-range-start=\"232\" data-range-end=\"368\">how context modifies embeddings</span>. The closing line is particularly insightful: \"Embeddings don't store meaning. They map where those meanings sit in relation to one another.\"","video_duration":412,"isV2":true,"blockId":"7adba765-5f62-4052-b4f5-861e69731ac8","lessonId":"089649e4-7d07-41c0-ac9d-5a6f72db85a4"}

{
  "type": "exercise",
  "id": "284fb8ac-20e8-43aa-b252-23f4af170403"
}

Conclusion

In this lesson, we have demystified the concept of text embeddings, the foundational technology for enabling long-term memory in AI agents. You now have the conceptual framework to understand how an agent can "understand" a product catalog or customer query.

Here are the key takeaways:

• Embeddings are numerical vectors that represent the semantic meaning of text, allowing machines to process language.
• The core principle is that semantic similarity corresponds to proximity in a high-dimensional vector space.
• Modern embeddings are generated by Transformer-based neural networks that are trained on vast datasets to learn the complex relationships in language.
• Crucially, these embeddings are contextual, meaning the vector for a word is influenced by the words surrounding it.
• A token is a simple numerical ID for a word piece, whereas an embedding is the rich, multi-dimensional vector that captures its meaning in context.

You now understand the "what" and "why" of embeddings. In our next lesson, we will move to the "how." You will learn to generate text embeddings using an API like OpenAI's, taking your first practical step towards building a Retrieval-Augmented Generation (RAG) system for your e-commerce agent.