Quantization and Bit Depth: Foundations of Audio Fidelity

Hello! Welcome to the third lesson in our course on audio AI.

In the previous lesson, we explored the Nyquist-Shannon Sampling Theorem, which explains how we can discretize the continuous time axis of an analog signal into a series of discrete samples. You learned that if we sample at a rate greater than twice the highest frequency in the signal, we can perfectly reconstruct the original wave. This answers the question of how often to measure the wave's amplitude.

Now, we must address the second part of the analog-to-digital conversion puzzle: what about the amplitude values themselves? An analog signal has a continuous, infinite range of possible amplitude values at any given point. To store these on a computer, we must also discretize the amplitude axis.

This lesson covers the process of quantization and the crucial role of bit depth in determining the fidelity of digital audio. We will explore how mapping continuous amplitudes to a finite set of values introduces a specific type of error and how this directly impacts the dynamic range and perceived quality of the sound.

---

1. From Continuous Amplitude to Discrete Levels

In the last lesson, we took samples at discrete points in time. However, the value of each sample's amplitude is still a real number with potentially infinite precision (e.g., 0.5, 0.51, 0.512, 0.5123...). A computer cannot store numbers with infinite precision. We need to map this continuous range of amplitudes to a finite set of discrete levels. This process is called quantization.

Think of it as rounding. If you can only use integers, a value like 3.7 gets rounded to 4, and 3.2 gets rounded to 3. Quantization applies this concept to the amplitude of an audio signal.

Let's start with a video that introduces this concept and clarifies its role in the digitization process.

The core idea is that each sample's continuous amplitude is approximated by the nearest available discrete level. The number of available levels is determined by the bit depth.

2. Bit Depth: The Language of Levels

Your computer science background gives you a head start here. Digital information is stored in bits. The number of bits used to represent the amplitude of a single sample is the bit depth.

The relationship is straightforward: an n-bit number can represent $2^n$ distinct values.

• 8-bit audio uses $2^8=256$ levels.
• 16-bit audio (CD quality) uses $2^{16}=65,536$ levels.
• 24-bit audio (studio quality) uses $2^{24}=16,777,216$ levels.

Most audio signals are signed, meaning they represent both positive and negative amplitudes (the wave above and below the zero line). For this, computers use a representation called two's complement, which allows an n-bit integer to represent values from $-2^{n-1}$ to $2^{n-1}-1$.

The following video explains how bit depth defines the number of quantization levels.

This image provides a clear visual comparison. On the left, a low bit depth results in a coarse, "blocky" approximation of the analog wave. On the right, a higher bit depth allows for a much finer and more accurate representation because there are more discrete amplitude steps available.

3. The Inevitable Consequence: Quantization Error

Quantization is inherently a lossy process. The difference between the original analog amplitude and the rounded, quantized amplitude is called quantization error.

This error isn't random. It manifests as a distinct type of distortion or noise that is added to the signal. We call this quantization noise. The lower the bit depth, the larger the rounding errors, and the more audible the noise becomes.

This is best understood by listening. The next resource provides a powerful demonstration.

As you heard, with very low bit depth (4-bit), the noise is overwhelming. With a higher bit depth (8-bit), it becomes much less prominent but is still correlated with the signal, especially noticeable in quieter parts. This raises the central question: how do we measure this effect on audio fidelity?

4. Quantifying Fidelity: Dynamic Range and SQNR

The primary measure of fidelity affected by bit depth is dynamic range.

Dynamic Range is the ratio, measured in decibels (dB), between the loudest possible undistorted signal and the quietest perceptible signal. In digital audio, the "quietest" signal is limited by the noise floor, which is created by quantization error.

A higher bit depth results in smaller quantization steps, which lowers the noise floor and therefore increases the dynamic range.

The Math Behind the Noise

We can formalize this relationship using the Signal-to-Quantization Noise Ratio (SQNR). For a sine wave quantized with n bits, the theoretical maximum SQNR is given by:

$${\text{SQNR}}_{\text{dB}}=20{\log }_{10}(2^n)+10{\log }_{10}\left(\frac{3}{2}\right)\approx 6.02\cdot n+1.76$$

This formula gives rise to a very famous rule of thumb in digital audio:

Each additional bit of depth adds approximately 6 dB to the dynamic range.

Let's walk through a simplified derivation for this rule, as presented in the "Digital Signals Theory" text.

Let's summarize the derivation:

1. The dynamic range is the ratio of the maximum possible value ($v_{+}$) to the minimum non-zero value ($v_{-}$).
2. For an n-bit signed integer, the maximum value is proportional to $2^{n-1}$, while the smallest non-zero value is proportional to 1.
3. The ratio is therefore $v_{+}/v_{-}=2^{n-1}$.
4. Converting this ratio to decibels for an amplitude-like quantity gives: $$R_{\text{dB}}=20\cdot {\log }_{10}(2^{n-1})=(n-1)\cdot 20{\log }_{10}(2)\approx (n-1)\cdot 6.02\text{ dB}$$

This simple approximation tells us the theoretical dynamic range for common bit depths:

• 8-bit: $\approx 7\cdot 6.02\approx 42$ dB (Very limited, noisy)
• 16-bit (CD): $\approx 15\cdot 6.02\approx 90$ dB (Good for listening, as it exceeds the dynamic range of most environments)
• 24-bit (Studio): $\approx 23\cdot 6.02\approx 138$ dB (Exceeds the dynamic range of human hearing, $\approx 120-130$ dB)

5. Quantization in Practice: Integers vs. Floats

Now we can put this all into the context of your work in AI and machine learning.

• Storage (Integer Formats): Audio is typically stored in files (like WAV) as 16-bit or 24-bit signed integers. This provides a good balance of fidelity and file size.
• Processing (Floating-Point Formats): When you load an audio file into a library like torchaudio or librosa, the integer values are almost always converted to 32-bit floating-point numbers, typically scaled to the range [-1.0, 1.0].

Why the conversion? 32-bit floating-point numbers offer a practically immense dynamic range ($>1500$ dB). This provides massive "headroom," preventing clipping and the accumulation of quantization noise during complex mathematical operations like filtering, mixing, or the forward passes of a neural network. You perform all your processing in this high-precision domain and only convert back to an integer format when saving the final output.

---

Conclusion

In this lesson, we completed the second half of the analog-to-digital conversion process. You have now seen how a continuous wave is transformed into a series of numbers that a computer can store and process.

Key Takeaways:

• Quantization is the process of mapping a continuous range of amplitudes to a finite set of discrete levels. It is a lossy process that discretizes the amplitude axis.
• Bit Depth (n) determines the number of available levels ($2^n$) and thus the precision of the amplitude representation.
• Quantization Error is the rounding error introduced by this process, which manifests as audible quantization noise.
• Dynamic Range is the primary measure of fidelity affected by bit depth. Each additional bit adds approximately 6 dB of dynamic range, effectively lowering the noise floor.
• In practice, audio is stored as 16-bit or 24-bit integers but processed as 32-bit floating-point numbers to maintain precision and prevent errors during computation.

We have now defined a digital audio signal by its sampling rate (from the last lesson) and its bit depth. In the next lesson, we will see how these two parameters are used in common digital audio file formats like WAV, FLAC, and MP3, and explore the trade-offs between uncompressed, lossless, and lossy formats.