Symbols and Units for Key Performance Metrics

Hello again. In the previous lesson, you separated the role of a quantity from its numerical value: an input varies from case to case, an output is produced by a process, and parameters configure the relationship. Now we add the information that makes those quantities usable in an engineering model: a symbol, a unit, and a precise operational definition.

This matters in AI workflows because a statement such as “the agent is expensive and slow” blends together several distinct measurements. Is “expensive” measured in $\mathrm{USD}$ per request, $\mathrm{USD}$ per day, or $\mathrm{USD}$ per successful task? Is “slow” time to first token, full completion time, or latency at the 95th percentile? A mathematical model begins by refusing to leave those distinctions implicit.

By the end of this lesson, you will be able to create a compact measurement vocabulary for token usage, cost, latency, branching, and success. This vocabulary will become the raw material for formulas, plots, experiments, and agent-workflow analysis later in the course.

---

A quantity needs more than a number

Consider these statements:

• “The prompt used 1,900 tokens.”
• “The request cost 0.002 dollars.”
• “The system took 800 milliseconds.”
• “The supervisor delegated to two agents.”
• “The workflow succeeded 92 percent of the time.”

Each includes a number and an implied or explicit unit. The number alone is incomplete:

• $1900$ could be tokens, requests, milliseconds, or dollars.
• $800$ could be milliseconds, seconds, tokens per second, or successful runs.
• A success value of $0.92$ might mean a proportion, a probability estimate, or a score on some unrelated scale.

A good model records each quantity in a form like this:

$$\text{symbol}\text{definition}\text{unit}\text{scope}$$

For example:

$$n_{\mathrm{in}}=1900\text{ tokens per request}$$

This says that $n_{\mathrm{in}}$ is the input-token count for one request. The subscript “in” is simply a label meaning input; next lesson will examine this notation more systematically.

The scope is crucial. Compare these two quantities:

$$n_{\mathrm{in}}=1900\text{ tokens per request}$$ $$N_{\mathrm{in},\mathrm{day}}=15,200,000\text{ tokens per day}$$

They both concern input tokens, but they answer different questions:

• $n_{\mathrm{in}}$ describes one typical interaction.
• $N_{\mathrm{in},\mathrm{day}}$ describes total volume across a day.

Using different symbols prevents an easy and costly mistake: multiplying a daily total by the number of daily requests a second time.

A helpful convention is to write units in square brackets when discussing the structure of an equation:

$$[\ell ]=\mathrm{ms}$$

Read this aloud as: “the unit of $\ell$ is milliseconds.”

---

Units participate in the calculation

Units are not decorative labels placed on an answer after the arithmetic is done. They follow the same multiplication and division as the numbers.

For a familiar example, if a task takes 5 seconds and processes 500 tokens, its average processing rate is:

$$\frac{500\text{ tokens}}{5\text{ seconds}}=100\frac{\text{tokens}}{\text{second}}$$

The number is $100$, but the full result is 100 tokens per second. If you reversed the division, you would get seconds per token, which is a different but still meaningful quantity.

The same practice lets you inspect AI cost equations before relying on them. Suppose:

$$p_{\mathrm{in}}=0.0008\frac{\mathrm{USD}}{\text{token}}$$

and

$$n_{\mathrm{in}}=1900\text{ tokens}$$

Then input cost is:

$$C_{\mathrm{in}}=p_{\mathrm{in}}{n}_{\mathrm{in}}$$

Checking units gives:

$$[C_{\mathrm{in}}]=\left[\frac{\mathrm{USD}}{\text{token}}\right][\text{token}]=\mathrm{USD}$$

The token labels cancel, leaving a cost measured in $\mathrm{USD}$. That is exactly what we should expect.

This technique is called dimensional analysis. In physical sciences it is often used for meters, kilograms, and seconds; in AI systems it is just as valuable for tokens, requests, model calls, compute time, and currency.

Two cautions will serve you well:

1. The same unit does not guarantee the same meaning. Time to first token and full-response latency can both be measured in milliseconds, but they measure different parts of the user experience.
2. Some quantities are ratios and therefore have no physical unit, yet still need a label. A success rate is technically dimensionless, but writing “successful runs per evaluated run” makes its meaning much clearer than writing only $0.92$.

---

A practical symbol-and-unit dictionary for AI workflows

There is no universal law requiring one particular symbol for every AI-system quantity. Different papers may use $T$, $L$, $n$, or $x$ for token counts. What matters is being consistent, defining each symbol once, and choosing names that make the model readable.

Here is a useful starter dictionary for an agent or LLM workflow.

Quantity	Suggested symbol	Unit	Operational definition
Input token count	$n_{\mathrm{in}}$	tokens	Tokens sent to a model in one call
Output token count	$n_{\mathrm{out}}$	tokens	Tokens generated by a model in one call
Request volume	$q$	requests/day	Requests received during one day
Input-token price	$p_{\mathrm{in}}$	$\mathrm{USD}$/token	Provider charge for input tokens
Output-token price	$p_{\mathrm{out}}$	$\mathrm{USD}$/token	Provider charge for output tokens
Cost of one call	$C_{\mathrm{call}}$	$\mathrm{USD}$	Monetary charge associated with one model call
End-to-end latency	$\ell$	$\mathrm{ms}$	Elapsed wall-clock time from request start to final usable result
Time to first token	${\ell }_{\mathrm{first}}$	$\mathrm{ms}$	Elapsed time from request start until the first streamed token arrives
Branching factor	$b$	child executions/parent execution	Number of child calls spawned by a parent call, often averaged across runs
Successful runs	$S$	successful runs	Number of evaluated runs meeting a stated success criterion
Evaluated runs	$R$	runs	Total number of runs included in the evaluation
Success rate	$r_{\mathrm{success}}$	successful runs/evaluated runs	Fraction of evaluated runs that satisfy the success criterion

The subscripts distinguish related quantities:

• $\mathrm{in}$ and $\mathrm{out}$ distinguish prompt-side and generation-side token counts.
• $\mathrm{call}$ identifies a single model call.
• $\mathrm{first}$ identifies the first-token timing metric.
• $\mathrm{success}$ identifies what the rate measures.

You may also see uppercase symbols used for totals and lowercase symbols used for one instance:

$$n_{\mathrm{in}}\text{for tokens in one call}$$ $$N_{\mathrm{in}}\text{for total input tokens in a chosen period}$$

This is a convention, not a requirement. The definition beside the symbol is always more important than capitalization.

---

Tokens and costs: distinguishing a count from a rate

A token is the unit an LLM processes internally. A token is not reliably the same as a word: depending on the tokenizer and the text, a word may correspond to one token, multiple tokens, or part of a token sequence. Therefore, when estimating model usage or API cost, record tokens, not just word count.

For one model call, a basic cost model is:

$$C_{\mathrm{call}}=p_{\mathrm{in}}{n}_{\mathrm{in}}+p_{\mathrm{out}}{n}_{\mathrm{out}}$$

Read the terms carefully:

• $n_{\mathrm{in}}$ and $n_{\mathrm{out}}$ are counts measured in tokens.
• $p_{\mathrm{in}}$ and $p_{\mathrm{out}}$ are prices, measured in $\mathrm{USD}$ per token.
• $C_{\mathrm{call}}$ is an output of the cost model, measured in $\mathrm{USD}$.

A provider may publish a price per thousand or per million tokens rather than per token. That is fine, but the unit must be written honestly. For example:

$$p_{\mathrm{in}}=0.80\frac{\mathrm{USD}}{1,000,000\text{ tokens}}$$

is not the same numerical rate as:

$$0.0000008\frac{\mathrm{USD}}{\text{token}}$$

They describe the same price with different unit scales.

The AWS cost example below shows why keeping input and output quantities separate matters. It lists prompt template, retrieved context, chat history, and the current query as distinct contributors to input-token volume. In a real RAG or recursive workflow, those sources can grow independently.

For example, if a workflow handles $q$ requests per day and has a typical cost $C_{\mathrm{call}}$ per request, then daily model cost can be estimated as:

$$C_{\mathrm{day}}=q{C}_{\mathrm{call}}$$

The unit check is:

$$[C_{\mathrm{day}}]=\left[\frac{\text{requests}}{\text{day}}\right][\mathrm{USD}/\text{request}]=\frac{\mathrm{USD}}{\text{day}}$$

Notice the difference between cost accumulated during one day and a daily cost rate. In ordinary product discussion, both may be described as “daily cost.” In a mathematical model, write a definition that removes ambiguity.

---

Latency is a time measurement, not one universal metric

“Latency” usually means elapsed time, so its basic unit is seconds or milliseconds:

$$[\ell ]=\mathrm{ms}$$

But an AI workflow has several legitimate latency measurements:

Metric	Symbol	Unit	What it measures
Time to first token	${\ell }_{\mathrm{first}}$	$\mathrm{ms}$	When a streaming user first sees progress
Model-call latency	${\ell }_{\mathrm{model}}$	$\mathrm{ms}$	Duration of one provider call
Tool latency	${\ell }_{\mathrm{tool}}$	$\mathrm{ms}$	Duration of a search, browser, database, or code tool call
End-to-end latency	${\ell }_{\mathrm{end}}$	$\mathrm{ms}$	Total time from user request to final workflow result

The symbols do not make these quantities different; the definitions do. A dashboard that reports only “latency” without saying which one it means cannot support a clear design decision.

The graph below uses the label TTFT P95, meaning time to first token at the 95th-percentile level. You do not need the statistics of percentiles yet; for now, notice that this is a specifically defined latency measure and that its vertical axis is in milliseconds.

The horizontal axis measures input token count:

$$[n_{\mathrm{in}}]=\text{tokens}$$

The vertical axis measures latency:

$$[{\ell }_{\mathrm{first},95}]=\mathrm{ms}$$

The red trendline has the form:

$${\ell }_{\mathrm{first},95}={\ell }_0+a{n}_{\mathrm{in}}$$

where:

• ${\ell }_0$ is a baseline delay in milliseconds;
• $a$ measures added latency per input token, with units $\mathrm{ms}$ per token;
• $n_{\mathrm{in}}$ is input-token count;
• ${\ell }_{\mathrm{first},95}$ is the resulting latency estimate in milliseconds.

The unit structure confirms that this formula is sensible:

$$[{\ell }_0]=\mathrm{ms}$$ $$[a{n}_{\mathrm{in}}]=\left[\frac{\mathrm{ms}}{\text{token}}\right][\text{token}]=\mathrm{ms}$$

Thus both terms can be added. You may not add a token count directly to a latency value, because tokens and milliseconds are different kinds of quantities.

This is the first glimpse of a linear model. Later, you will interpret its slope and intercept in depth; for now, the important result is that the graph’s axes determine the units of the formula’s terms.

---

Branching factor and success rate: ratios that need definitions

Agent workflows introduce quantities that do not look like physical measurements but still need careful units and scope.

Branching factor

Suppose a supervisor agent sometimes delegates work to specialized agents. Let:

$$b=\frac{\text{child agent executions}}{\text{parent agent execution}}$$

If the supervisor delegates to two specialized agents on average, write:

$$b=2\frac{\text{child executions}}{\text{parent execution}}$$

Because both numerator and denominator are counts, this ratio is often casually called “unitless.” Mathematically, that is reasonable. Operationally, however, the label is essential: it says the value is delegations per parent execution, not tokens per request or dollars per day.

An individual request may have $0$, $1$, or $3$ delegations, while the measured average across many requests is:

$$b=1.7$$

A fractional branching factor does not mean an agent made 0.7 of a call. It means the average number of child executions per parent execution was 1.7 across the chosen set of runs.

Success rate

A success rate is usually modeled as:

$$r_{\mathrm{success}}=\frac{S}{R}$$

where:

$$[S]=\text{successful runs}$$ $$[R]=\text{evaluated runs}$$

For example:

$$r_{\mathrm{success}}=\frac{92}{100}=0.92=92\%$$

The arithmetic is easy. The difficult and consequential part is defining success and defining the population of evaluated runs.

For a coding agent, “success” might mean:

The generated patch passes a predefined test suite without human edits.

For a retrieval workflow, it might mean:

The final answer cites at least one relevant source and is judged correct by a blinded evaluator.

For an operational agent, it might mean:

The workflow completes within a 30-second latency limit, remains under a cost limit, and takes no prohibited tool action.

Each definition can yield a different success rate. Therefore, never write “the success rate is 92 percent” without being able to state:

1. what counts as success;
2. which runs were included;
3. whether retries count as new runs or part of one attempt;
4. whether the reported number is a proportion, a percentage, or a model-estimated probability.

Probability will later give us a deeper language for uncertainty. At this stage, treat success rate as an observed ratio with a carefully stated denominator.

---

Build a measurement ledger before building formulas

Before modeling a workflow, create a small ledger. It can live in a notebook, a design document, a benchmark specification, or code comments.

Symbol	Meaning	Unit	Measurement scope
$n_{\mathrm{in}}$	Input tokens sent to one model call	tokens	One call, using a named model/tokenizer
$n_{\mathrm{out}}$	Tokens generated by one model call	tokens	One call
${\ell }_{\mathrm{end}}$	End-to-end elapsed time	$\mathrm{ms}$	From request receipt to usable final response
$b$	Mean child-agent executions	child executions/parent execution	Across a stated evaluation set
$r_{\mathrm{success}}$	Workflow success fraction	successful runs/evaluated runs	Under a stated success criterion
$C_{\mathrm{day}}$	Model and infrastructure cost	$\mathrm{USD}$/day	A specified deployment and traffic pattern

This ledger does three jobs:

• It gives every quantity a stable name.
• It exposes when two people are measuring different things with the same word.
• It lets you check future equations by checking their units.

Here is a small optional Python calculation that preserves the same distinction between quantities. Python will not enforce units for you, so the variable names and comments carry part of the modeling discipline.

# Token counts for one model call
input_tokens = 1900
output_tokens = 160

# Prices in USD per token
input_price_per_token = 0.0008 / 1000
output_price_per_token = 0.0032 / 1000

# Result: USD for one model call
cost_per_call_usd = (
    input_price_per_token * input_tokens
    + output_price_per_token * output_tokens
)

requests_per_day = 8000

# Result: USD accumulated across one day
daily_cost_usd = cost_per_call_usd * requests_per_day

print(cost_per_call_usd)
print(daily_cost_usd)

The first printed number is a cost associated with one call. The second is a total cost across a day under the stated request-volume assumption. Altering input_tokens changes the workload; altering input_price_per_token changes a pricing parameter; altering requests_per_day changes the deployment scale. All three affect cost, but they represent different causes.

---

You now have a disciplined way to name AI workflow measurements: give each quantity a meaningful symbol, an explicit unit, and a scope that says what was measured and over what population or time period.

The central habits are:

• Treat tokens, milliseconds, currency, calls, and requests as distinct units.
• Keep per-call values separate from daily, monthly, or workflow-wide totals.
• Separate related metrics such as input versus output tokens and first-token versus end-to-end latency.
• Treat branching factor and success rate as ratios whose operational definitions matter as much as their numerical values.
• Check whether the units on both sides of an equation agree.

Next, you will learn to read the compact notation that makes these definitions manageable in ML papers: subscripts, superscripts, Greek letters, set membership, and comparison symbols.