Using Logarithms and Log Probabilities to Prevent Numerical Underflow

Hello again. In the previous lesson, you used conditional probability and Bayes’ rule to update a belief after observing evidence. Those calculations involved products such as $P(B\mid A)P(A)$. In speech and language systems, the same operation appears hundreds or thousands of times while scoring a sequence of audio frames, characters, or tokens. That creates a practical computer-arithmetic problem: the correct probability can become too small for a floating-point number to represent.

This final lesson in the mathematics module introduces the standard remedy: work in log space. You will learn why probability products underflow, how logarithms turn products into sums without changing score ordering, and when the log-sum-exp operation is needed to safely handle sums of probabilities.

---

Why sequence probabilities become numerically impossible

A probability lies in the interval

$$0\le p\le 1.$$

Most nontrivial probabilities are strictly below $1$. Multiplying many such values therefore produces a rapidly shrinking result.

Suppose a simplified sequence model assigns probability $0.01$ to each of $200$ successive events. The joint probability is

$$(0.01{)}^{200}=10^{-400}.$$

Mathematically, this is a valid nonzero number. But standard 64-bit floating-point arithmetic cannot represent values that small; it typically rounds them to zero. This is numerical underflow.

For a sequence $w_1,\dots ,w_T$, a language model uses the chain rule to assign a joint probability:

$$P(w_1,\dots ,w_T)=\prod _{t=1}^{T}P(w_t\mid w_1,\dots ,w_{t-1}).$$

The conditionals do not need to be independent for this product to apply. Each conditional probability accounts for the preceding context. But regardless of how the individual probabilities were generated, multiplying hundreds of values smaller than $1$ makes the final joint probability extremely small.

This matters throughout speech ML:

• An ASR decoder scores many possible transcription sequences.
• A language model scores a succession of output tokens.
• A CTC model aggregates probabilities across frame-level alignment paths.
• A probabilistic classifier may calculate a likelihood over many observations.

In each case, a computed zero would be wrong: it says the event is impossible, not merely extraordinarily unlikely.

The plot’s vertical axis is logarithmically scaled, which is why the decline looks like a straight line even though the raw probability is shrinking exponentially.

---

The logarithm converts products into sums

The key identity is

$$\log (ab)=\log (a)+\log (b).$$

Repeated application gives

$$\log \left(\prod _{t=1}^T{p}_t\right)=\sum _{t=1}^T\log (p_t).$$

So rather than compute a sequence probability as a product, store and add its log probabilities:

$$\log P(w_1,\dots ,w_T)=\sum _{t=1}^T\log P(w_t\mid w_1,\dots ,w_{t-1}).$$

A probability such as $10^{-400}$ is too small to store directly in many standard formats. Its natural log, however, is entirely manageable:

$$\log (10^{-400})=-400\log (10)\approx -\mathrm{921.03.}$$

The number $-921.03$ is ordinary-sized as a floating-point value, even though its exponential is not representable.

What log-probability values mean

For a valid probability $p$ with $0<p\le 1$,

$$\log (p)\le 0.$$

Some useful reference points are:

Probability	Natural log probability	Interpretation
$1$	$0$	Certain event
$0.5$	approximately $-0.693$	Moderate probability
$0.01$	approximately $-4.605$	Low probability
$10^{-100}$	approximately $-230.259$	Extremely small but tractable in log space
$0$	$-\infty$	Impossible event

A log probability closer to zero is larger. For example,

$$-2>-10,$$

so a log probability of $-2$ represents a more probable event than one of $-10$.

This initially feels backwards because the scores are negative. Keep the underlying relationship in view:

$$\log (p_1)>\log (p_2)\text{if and only if}{p}_1>p_2.$$

The logarithm is strictly increasing, so it preserves rankings. If one transcription has the highest probability, it also has the highest log probability. A decoder can therefore select the best candidate entirely in log space.

---

A concrete calculation in Python

Here is the $200$-event example in NumPy.

import numpy as np

probabilities = np.full(200, 0.01)

raw_probability = np.prod(probabilities)
log_probability = np.log(probabilities).sum()

print(raw_probability)
print(log_probability)

On a typical system, the first result is:

0.0

The second is approximately:

-921.0340371976183

The raw result has underflowed, while the log-space result retains the information needed for comparison, decoding, and optimization.

You should only convert back when you genuinely need a reported probability and when the result is within the representable range:

recovered_probability = np.exp(log_probability)
print(recovered_probability)

For this example, conversion back also yields 0.0, because the probability is still too small for the machine to represent. That is not a failure of the log-space calculation. It is exactly why downstream computation should remain in log space whenever possible.

A safe working rule

When a model combines probabilities by multiplication:

1. Take the log of each nonzero probability.
2. Add log probabilities instead of multiplying probabilities.
3. Compare candidates using their total log probabilities.
4. Convert with $\exp$ only at a boundary where an ordinary probability is truly required.

This convention is common enough that a variable called log_prob, log_likelihood, or score is often intended to remain in log space throughout most of a pipeline.

A small but important distinction: a probability is normalized and lies between $0$ and $1$. A likelihood is a quantity used to compare how well parameter settings or models explain observed data. In ML practice, both are frequently handled with logarithms because the relevant computation involves products over many observations.

---

Logs solve multiplication, not addition

There is one crucial rule to avoid misusing logarithms:

$$\log (a+b)\ne \log (a)+\log (b).$$

Logs make products easy, but they do not make sums into ordinary addition.

This becomes relevant whenever a model must combine several alternative ways for an event to occur. For instance, suppose two alignment paths contribute probabilities $p_1$ and $p_2$. Their total probability is

$$p_1+p_2.$$

If you store

$$x_1=\log (p_1)$$

and

$$x_2=\log (p_2),$$

then the desired log total is

$$\log (p_1+p_2)=\log \left(\exp (x_1)+\exp (x_2)\right).$$

This operation is called log-sum-exp:

$$\mathrm{LSE}(x_1,\dots ,x_N)=\log \left(\sum _{i=1}^N\exp (x_i)\right).$$

Directly calculating the exponentials can still underflow for highly negative values, or overflow if the values are large unnormalized scores. The stable form first subtracts the maximum:

$$m=\underset{i}{\max }{x}_i,$$ $$\mathrm{LSE}(x_1,\dots ,x_N)=m+\log \left(\sum _{i=1}^N\exp (x_i-m)\right).$$

Subtracting $m$ changes no final answer because the factor $\exp (m)$ is accounted for by adding $m$ back afterward. But it ensures that the largest exponent is

$$\exp (0)=1,$$

and every other exponent is at most $1$. That is the numerical-stability benefit.

---

Stable log-sum-exp in code

Consider three log weights:

$$[-1000,-1001,-1003].$$

Each corresponding ordinary-space value underflows if computed directly with typical floating-point arithmetic. But their relative sizes are still meaningful.

import numpy as np

log_weights = np.array([-1000.0, -1001.0, -1003.0])

maximum = np.max(log_weights)
log_total = maximum + np.log(np.sum(np.exp(log_weights - maximum)))

normalized_log_probs = log_weights - log_total
normalized_probs = np.exp(normalized_log_probs)

print(log_total)
print(normalized_probs)

The normalized probabilities are approximately:

[0.70538451 0.25949646 0.03511903]

Although the original values were too small to represent individually in ordinary probability space, their normalized proportions are computed safely.

For production code, prefer a tested library operation rather than rewriting this repeatedly. In NumPy, np.logaddexp.reduce handles a one-dimensional log-sum-exp reduction; later, PyTorch provides torch.logsumexp.

The same pattern is used to turn a vector of unnormalized model scores, often called logits, into normalized log probabilities:

$$\log p_i=x_i-\mathrm{LSE}(x_1,\dots ,x_N).$$

Exponentiating those final log probabilities gives ordinary probabilities that sum to $1$, if they are needed for display or a decision threshold.

---

How this will appear in speech systems

The direct payoff is that sequence decoders can work with additions rather than fragile products. A candidate transcription that has token-level probabilities $p_1,\dots ,p_T$ receives the log score

$$\sum _{t=1}^T\log (p_t).$$

A larger total log score means a more probable candidate under the model. No raw product is required.

Later in the course, this convention will show up repeatedly:

• CTC loss sums probabilities over many valid alignments, requiring stable log-sum-exp operations.
• Beam search retains and compares hypotheses using accumulated log scores.
• Cross-entropy loss is naturally expressed in terms of log probabilities.
• Softmax classifiers use a stable log-sum-exp normalization internally.

The practical habit to build now is simple: if a calculation combines many probabilities, ask whether it should be represented as a sum of log probabilities instead.

---

Key takeaways

A long sequence probability is a product of many values below $1$:

$$P(\text{sequence})=\prod _t{p}_t.$$

That product can underflow to zero in floating-point arithmetic even when the true mathematical probability is nonzero.

Logging the probability transforms multiplication into addition:

$$\log \left(\prod _t{p}_t\right)=\sum _t\log (p_t).$$

Because the logarithm preserves order, comparing log probabilities selects the same best candidate as comparing ordinary probabilities.

When probabilities must be summed, use log-sum-exp rather than adding log values directly:

$$\log \left(\sum _i\exp (x_i)\right)=m+\log \left(\sum _i\exp (x_i-m)\right),$$

where

$$m=\underset{i}{\max }{x}_i.$$

You have now completed the applied-mathematics foundation for the course. Next, the focus shifts from mathematical tools to supervised learning: defining inputs, targets, model parameters, and predictions for a concrete ML problem.