Bidirectional RNNs for Sequence Processing

Hello! Welcome back to our exploration of sequence modeling.

In our last lesson, we delved into LSTMs and GRUs, powerful architectures that use gating mechanisms to overcome the vanishing gradient problem and capture long-range dependencies. However, these models, by default, process information chronologically—they only consider the past to understand the present.

Today, we will overcome that limitation. This lesson is dedicated to building models that can look both backward and forward in a sequence. The learning outcome is to build bidirectional RNNs to process sequence information from both directions. We will explore why this is crucial for many tasks, understand the architecture, and implement it in code.

1. The Limitation of Unidirectional Processing

Imagine you encounter the sentence: "The main character, April, won the tournament."

A standard, unidirectional RNN processing this sentence would read "April" having only seen "The main character,". It might guess "April" is a name. But consider this sentence: "After a long winter, April is the most pleasant month." Here, the context needed to correctly identify "April" as a month comes after the word itself.

Unidirectional RNNs, by their very nature, cannot access this future context. Their prediction at time step $t$ is based solely on the sequence from $1$ to $t$. For many applications in natural language processing (NLP), such as named entity recognition, sentiment analysis, and machine translation, this is a significant handicap. True understanding often requires knowing what comes before and what comes after.

To see this demonstrated clearly, let's watch a short video that uses a similar example to highlight the need for bidirectionality.

2. The Bidirectional RNN (BiRNN) Architecture

The solution to this problem is elegantly simple: process the sequence in both directions. A Bidirectional RNN achieves this by using two separate, independent RNNs:

1. A Forward RNN: This is a standard RNN (which can be a simple RNN, LSTM, or GRU) that processes the input sequence from start to end (left to right).
2. A Backward RNN: This is another RNN that processes the same input sequence but in reverse, from end to start (right to left).

At each time step $t$, the BiRNN's output is formed by combining the hidden states from both the forward and the backward RNNs. This is typically done by concatenating them.

Let's watch a continuation of the previous video, which visualizes this architecture and the flow of information.

3. Mathematical Formulation

Let's formalize this. Given an input sequence ${\mathbf{X}}_1,{\mathbf{X}}_2,\dots ,{\mathbf{X}}_T$, a BiRNN computes the forward hidden states ${\overrightarrow{\mathbf{H}}}_t$ and backward hidden states ${\overleftarrow{\mathbf{H}}}_t$ for each time step $t$.

As you read, here are the key equations to focus on:

• Forward hidden state update:

  $${\overrightarrow{\mathbf{H}}}_t=\varphi ({\mathbf{X}}_t{\mathbf{W}}_{\text{xh}}^{(f)}+{\overrightarrow{\mathbf{H}}}_{t-1}{\mathbf{W}}_{\text{hh}}^{(f)}+{\mathbf{b}}_{\text{h}}^{(f)})$$

  This is the standard RNN update, moving from $t-1$ to $t$.

• Backward hidden state update:

  $${\overleftarrow{\mathbf{H}}}_t=\varphi ({\mathbf{X}}_t{\mathbf{W}}_{\text{xh}}^{(b)}+{\overleftarrow{\mathbf{H}}}_{t+1}{\mathbf{W}}_{\text{hh}}^{(b)}+{\mathbf{b}}_{\text{h}}^{(b)})$$

  Note that this update depends on the next backward hidden state, ${\overleftarrow{\mathbf{H}}}_{t+1}$, as it processes the sequence in reverse. The weights and biases for the backward pass are separate from the forward pass.

• Concatenated hidden state:
  The hidden state of the BiRNN at time $t$, ${\mathbf{H}}_t$, is the concatenation of the forward and backward states:

  $${\mathbf{H}}_t=[{\overrightarrow{\mathbf{H}}}_t,{\overleftarrow{\mathbf{H}}}_t]$$

  If both the forward and backward hidden states have a dimension of $h$, the concatenated hidden state ${\mathbf{H}}_t$ will have a dimension of $2h$.

• Output:
  The final output at time $t$ is then computed from this rich, bidirectional hidden state:

  $${\mathbf{O}}_t={\mathbf{H}}_t{\mathbf{W}}_{\text{hq}}+{\mathbf{b}}_{\text{q}}$$

4. Implementation

Since the learning outcome is to build a BiRNN, let's get into the code. We'll look at it from two perspectives: a conceptual "from scratch" implementation and a practical one using PyTorch's high-level API.

4.1. Conceptual Implementation from Scratch

To truly solidify your understanding, it helps to see how you would construct a BiRNN from its fundamental components: two separate unidirectional RNNs. The logic is straightforward:

1. Initialize two RNN instances (e.g., forward_rnn and backward_rnn).
2. Pass the input sequence to forward_rnn.
3. Pass a reversed version of the input sequence to backward_rnn.
4. For each time step, concatenate the corresponding hidden states from both RNNs. Remember to reverse the sequence of backward hidden states so they align correctly with the forward ones.

4.2. Practical Implementation in PyTorch

In practice, you will almost always use the built-in functionalities of deep learning frameworks. In PyTorch, creating a BiRNN is as simple as setting a boolean flag. However, it's crucial to understand the consequences of setting this flag for the rest of your network architecture.

The following video provides a clear, code-along example of implementing a BiLSTM in PyTorch.

Let's summarize the key takeaways from the implementation:

• Enabling Bidirectionality: When defining your nn.RNN, nn.LSTM, or nn.GRU layer, simply set the argument bidirectional=True.
• Output Shape: The output of a bidirectional layer will have a feature dimension of 2 * hidden_size, reflecting the concatenation of the forward and backward hidden states. You must account for this in any subsequent layers (e.g., your final nn.Linear layer).
• Hidden State Shape: When you manually initialize the hidden state h0 (and cell state c0 for LSTMs), its first dimension must be num_layers * 2. The framework needs separate initial states for the forward and backward passes for each layer.

5. Training and Limitations

Training: Backpropagation in a BiRNN works just as you'd expect. Because the forward and backward passes are independent, the gradients are calculated for each pass separately. The forward pass gradients are backpropagated from the end of the sequence to the beginning, and the backward pass gradients are backpropagated from the beginning to the end.

Limitations:

• Computational Cost: A BiRNN has roughly twice the parameters of a unidirectional RNN, leading to increased memory usage and longer training times.
• Inapplicability for Real-Time Prediction: The most significant limitation is that a BiRNN requires the entire input sequence to be available before it can make a prediction for any time step. The backward pass cannot start until it has seen the last element. This makes BiRNNs unsuitable for online tasks or real-time forecasting, where you need to make predictions as data arrives sequentially.

Conclusion

In this lesson, you have learned how to augment sequence models to consider context from both the past and the future. This is a fundamental technique for a wide range of tasks where a complete understanding of the sequence is necessary.

Key Takeaways:

• Unidirectional RNNs are limited because they can only use past information to make a prediction.
• Bidirectional RNNs solve this by using two independent RNNs: one processing the sequence forwards and one backwards.
• The hidden states from both RNNs at each time step are concatenated to create a rich, context-aware representation.
• In PyTorch, a BiRNN is implemented by setting bidirectional=True, which doubles the output feature dimension and the first dimension of the initial hidden state.
• The main trade-off is increased computational cost and the requirement of having the full sequence available, making it unsuitable for real-time applications.

Preview of the next lesson:
The BiRNN architecture we've just built is an incredibly powerful tool for creating a single vector representation that summarizes an entire input sequence. In our next lesson, we will leverage this idea to build Sequence-to-Sequence (seq2seq) models. A BiRNN will serve as the perfect encoder in a seq2seq model, reading an input sequence (like a sentence in French) and compressing it into a context vector that a decoder can then use to generate a new sequence (like the sentence's translation in English).