STFT: Derivation and Windowing Trade-offs

Hello! Welcome to the fifth lesson in our module on Spectral Analysis.

In our last two lessons, we explored the Discrete Fourier Transform (DFT) and the highly efficient Fast Fourier Transform (FFT) algorithm used to compute it. We established that an FFT gives us a powerful snapshot of a signal's frequency content. However, this snapshot is static—it averages the frequencies over the entire duration of the signal. For a non-stationary signal like audio, where notes, words, and sounds are constantly changing, this is a significant limitation.

This lesson introduces the fundamental tool for analyzing time-varying frequency content: the Short-Time Fourier Transform (STFT). Our learning outcome is to derive the Short-Time Fourier Transform (STFT) and explain the trade-offs of different windowing functions. We will build the STFT from the ground up, starting with the DFT you already know. You'll learn how it creates a time-frequency representation of a signal and understand the critical role of window functions in controlling its behavior.

1. From a Still Image to a Movie of Frequencies

Audio signals are non-stationary; their spectral content is dynamic. A single DFT of an entire song tells you which notes were played on average, but not when they were played or in what sequence. We need to move from a static frequency "photograph" to a "movie" that shows how frequencies evolve over time.

The STFT achieves this through a simple and powerful strategy: divide and conquer. Instead of analyzing the whole signal at once, we break it into small, overlapping chunks (or "frames") and apply an FFT to each one. This assumes that over a very short duration (e.g., 20-40 milliseconds), the signal's properties are relatively stable, or "quasi-stationary."

This process can be visualized as a sliding window moving across the signal, with an FFT computed at each step.

2. Deriving the STFT Formula

Let's formalize this process mathematically. Recall the DFT formula for a signal $x[n]$ of length $N$:

$$X[k]=\sum _{n=0}^{N-1}x[n]e^{-j\frac{2\pi kn}{N}}$$

To adapt this for the STFT, we need to incorporate the concepts of framing and windowing. We introduce two key modifications:

1. A time index, $m$, which denotes the current frame number.
2. A window function, $w[n]$, which is a finite-length sequence that we multiply our signal frame by.

The STFT, denoted as $X[m,k]$, is a function of both the frame index $m$ and the frequency bin $k$. It is calculated by taking the DFT of the product of the signal and a window function that is shifted to the $m$-th frame.

The precise mathematical formulation can be a bit dense at first. The following video segment does an exceptional job of comparing the DFT formula side-by-side with the STFT formula, explaining each new component.

Based on the video, the discrete STFT is defined as:

$$X[m,k]=\sum _{n=0}^{N_w-1}x[n+mH]w[n]e^{-j\frac{2\pi kn}{N_{FFT}}}$$

Let's unpack this:

• $X[m,k]$ is the complex STFT value for frame $m$ and frequency bin $k$.
• $x[n+mH]$ is the input signal, shifted to the start of the $m$-th frame. $H$ is the hop size (or hop length), the number of samples you slide the window forward for each new frame.
• $w[n]$ is the window function of length $N_w$.
• The summation is over the length of the window, $N_w$.
• $N_{FFT}$ is the size of the FFT used. This is often equal to the window length $N_w$, but can be larger (by zero-padding) to get a higher-resolution frequency axis.

The output is a 2D matrix where one axis represents time (frame index $m$) and the other represents frequency (frequency bin $k$).

3. The Crucial Role of Window Functions

The choice of the window function, $w[n]$, is not a minor detail; it is fundamental to the quality of your spectral analysis. The simplest choice is a rectangular window, which is 1 inside the frame and 0 everywhere else. However, the sharp transitions at the edges of this window introduce artifacts.

Multiplying a signal by a window in the time domain is equivalent to convolving their spectra in the frequency domain. The spectrum of a rectangular window is a sinc-like function, which has a narrow main lobe but fairly high "side lobes". This convolution causes energy from a single frequency to "leak" into adjacent frequency bins, an effect called spectral leakage.

To mitigate this, we use tapered windows that smoothly approach zero at their edges. This reduces the abruptness of the truncation, leading to much lower side lobes in the frequency domain.

The high side lobes of the rectangular window (-13 dB) mean that a strong frequency component can create leakage that is louder than a weak, nearby frequency component, completely obscuring it. This leads to a fundamental trade-off in window selection.

The Main-Lobe vs. Side-Lobe Trade-off

By choosing a smoother, more tapered window, you can dramatically reduce the height of the side lobes, but it comes at a cost: the main lobe gets wider.

• Narrow Main Lobe: Better frequency resolution. You can distinguish between two sinusoidal components that are very close in frequency.
• Low Side Lobes: Better dynamic range. You can detect a weak sinusoidal component in the presence of a strong one nearby.

Let's compare two of the most common tapered windows: Hann and Hamming.

Here is a summary of the trade-offs:

Window	Main-Lobe Width	Highest Side Lobe	Side-Lobe Roll-off	Best For...
Rectangular	$\sim 1\times$ Base	-13 dB	$\sim$ -6 dB/octave	Strong signals with no close interferers.
Hann	$\sim 2\times$ Base	-31 dB	$\sim$ -18 dB/octave	General purpose, good balance.
Hamming	$\sim 2\times$ Base	-43 dB	$\sim$ -6 dB/octave	Excellent side-lobe rejection near main lobe.
Blackman	$\sim 3\times$ Base	-58 dB	$\sim$ -18 dB/octave	Applications needing very high dynamic range.

This trade-off is not just theoretical. The following reading shows a practical example of analyzing an oboe tone, where the choice of window directly impacts what you can "see" in the spectrum.

4. The Time-Frequency Uncertainty Principle

There is a second, equally important trade-off in STFT, controlled not by the window shape, but by the window size (or frame duration).

• A short window gives you good time resolution. You can pinpoint when a frequency event occurred with high precision. However, it gives you poor frequency resolution because the analysis window is too short to capture enough cycles of a low-frequency wave.
• A long window gives you good frequency resolution. By analyzing a longer segment, you can resolve closely spaced frequencies. However, this comes at the cost of poor time resolution, as any events that happen within that long window are blurred together in time.

This is a fundamental constraint of signal processing, often related to the Heisenberg Uncertainty Principle in physics. You cannot simultaneously have perfect resolution in both time and frequency.

5. An Alternative View: The STFT as a Filter Bank

Given your background, you may appreciate another perspective on the STFT. The entire operation is mathematically equivalent to passing the signal through a bank of parallel, overlapping bandpass filters.

• Each filter is centered on a different frequency bin.
• The shape of each filter's frequency response is determined by the Fourier transform of the window function, $w[n]$.
• The output of the STFT at frame $m$ and frequency $k$ corresponds to the output of the $k$-th filter at time $m$.

From this viewpoint, a long window in time has a narrow Fourier transform, resulting in a bank of very narrow, selective filters (good frequency resolution). A short window has a wide Fourier transform, resulting in a bank of wide, less selective filters (good time resolution). This is just another way to understand the same time-frequency trade-off.

---

Conclusion

In this lesson, we have moved from the static world of the DFT to the dynamic, time-varying analysis of the STFT. You've seen how it's constructed and have learned about the critical trade-offs involved in its application.

Key Takeaways:

• The STFT overcomes the limitation of the DFT by analyzing short, overlapping frames of a signal to see how its frequency content changes over time.
• The STFT formula is essentially a DFT applied to a windowed signal segment, with the result being a 2D matrix of time vs. frequency.
• Window functions are applied to each frame to reduce spectral leakage. Tapered windows (like Hann, Hamming) are used to minimize this effect.
• There is a fundamental trade-off in window shape: better frequency resolution (narrower main lobe) vs. better dynamic range (lower side lobes).
• There is a fundamental trade-off in window size: better time resolution (short window) vs. better frequency resolution (long window).
• The STFT can also be interpreted as a filter bank, where the window function's transform defines the shape of each bandpass filter.

In our next lesson, we will put this theory into practice. We will use Python libraries like torchaudio and librosa to compute and visualize spectrograms—the visual representation of the STFT's magnitude—which are the cornerstone of modern audio AI.