Implementing a DCGAN

Hello! Welcome back.

In our previous lesson, we built a basic Generative Adversarial Network using simple multi-layer perceptrons. We saw how the adversarial game between a generator and a discriminator could produce new data, but we also discovered that this process is often unstable and prone to issues like mode collapse.

Today, we'll address those stability issues head-on. Your learning outcome is to implement a Deep Convolutional GAN (DCGAN). A DCGAN isn't a new theoretical framework like the jump from VAEs to GANs; rather, it's a set of powerful architectural guidelines that made GANs practical and effective for image generation tasks.

We will cover:

• The key architectural principles introduced by the DCGAN paper.
• The role of convolutional and transposed convolutional layers in the discriminator and generator.
• How components like Batch Normalization and specific activation functions contribute to training stability.
• A complete, from-scratch implementation of a DCGAN in PyTorch to generate images.

By the end of this lesson, you'll have built a much more robust and powerful GAN, moving from the simple "vanilla" model to a cornerstone architecture in the history of generative models.

From Vanilla GAN to DCGAN: A Recipe for Stability

The 2015 paper, "Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks," was a landmark publication. It didn't change the fundamental minimax game of GANs, but it provided a concrete recipe for building GANs that train stably and produce high-quality images.

These architectural guidelines are the "secret sauce" of DCGANs. Let's explore them before we start coding.

To summarize and expand on the key points from the video and the original paper:

1. Replace Pooling with Strided Convolutions:

   • Discriminator: Instead of using max-pooling or average-pooling layers to downsample the image, DCGANs use strided convolutions (stride > 1). This allows the network to learn its own spatial downsampling, making it more powerful than a fixed function like max-pooling.
   • Generator: To upsample from the latent vector, the generator uses transposed convolutions (often called fractionally-strided convolutions). This is the inverse of a convolution, allowing the network to learn how to intelligently increase spatial dimensions.

2. Use Batch Normalization: Batch Normalization is applied in both the generator and discriminator. It stabilizes learning by normalizing the input to each layer to have zero mean and unit variance. This helps prevent issues like mode collapse and allows gradients to flow more effectively, which is critical in the delicate GAN training dynamic. There are exceptions: it's not used in the generator's output layer or the discriminator's input layer.

3. Eliminate Fully Connected Layers: The networks are fully convolutional. This reduces the number of parameters and improves architectural stability. The final layer of the discriminator is flattened and fed to a single sigmoid output, but there are no dense hidden layers.

4. Choose the Right Activation Functions:

   • Generator: Uses the ReLU activation function for all layers except the output layer. The output layer uses Tanh, which scales the output to the range [-1, 1]. This pairs well with normalizing the input training images to the same range.
   • Discriminator: Uses LeakyReLU for all layers. Unlike a standard ReLU which outputs zero for any negative input, LeakyReLU allows a small, non-zero gradient to flow through for negative values. This prevents "dying ReLUs" and ensures the discriminator's gradients don't vanish, providing a more consistent signal to the generator.

5. Specific Optimizer Settings: The paper recommends the Adam optimizer with a learning rate of 0.0002 and a beta1 parameter of 0.5 (instead of the default 0.9). This slower, more stable update rule was found to be crucial for GAN convergence.

Let's visualize the architectures we are about to build.

Implementing a DCGAN in PyTorch

Now, let's translate these architectural guidelines into code. We will follow a detailed, step-by-step implementation. Your background in software engineering will be helpful here as we construct these models as nn.Module classes, paying close attention to the input and output tensor shapes at each layer.

The following sections will guide you through the implementation, using Aladdin Persson's video as a code-along reference.

1. The Discriminator

The discriminator acts as a binary classifier. It takes an image and outputs a single value indicating the probability that the image is real. We'll build it using a sequence of Conv2d, BatchNorm2d, and LeakyReLU layers.

2. The Generator

The generator's job is to do the reverse: take a small latent vector z and upsample it into a full-sized image. This is achieved using ConvTranspose2d layers.

Test your understanding!

In the discriminator, we use nn.Conv2d with stride=2 to downsample. In the generator, we use nn.ConvTranspose2d with stride=2 to upsample. Why is ConvTranspose2d a better choice for the generator than, for example, using a simple upsampling method (like nearest-neighbor) followed by a standard nn.Conv2d?

Show answer

While using an upsampling layer followed by a standard convolution is a valid approach (and used in some modern architectures), the ConvTranspose2d layer has a key advantage in the DCGAN context: its weights are learnable. This means the network doesn't just apply a fixed upsampling algorithm (like duplicating pixels); it learns the optimal way to fill in the details and reverse the downsampling process performed by the discriminator's convolutions. This allows the generator to learn how to create more detailed and less "blocky" structures during upsampling.

3. Weight Initialization and Unit Testing

The DCGAN paper specifies a particular weight initialization scheme. It's a small but important detail for stability. After defining our models, it's also good practice to run a quick sanity check to ensure our tensor shapes are correct.

4. The Training Loop

The training logic for a DCGAN is identical to the vanilla GAN we saw in the previous lesson. We'll alternate between training the discriminator and the generator.

• Part 1: Train the Discriminator
  1. Feed it a batch of real images and calculate the loss against labels of 1.
  2. Feed it a batch of fake images (from the generator) and calculate the loss against labels of 0.
  3. Sum the two losses and update the discriminator's weights.
• Part 2: Train the Generator
  1. Feed a batch of fake images to the discriminator.
  2. Calculate the generator's loss based on the discriminator's output, but this time using labels of 1 (the generator's goal is to make the discriminator think its fakes are real).
  3. Update the generator's weights.

The following video sections set up the hyperparameters and implement this training loop.

After running the code, you'll be able to generate images of celebrities (if using the CelebA dataset) or other objects, which are of significantly higher quality and stability than what a simple MLP-based GAN could produce.

Bonus: An Alternative Perspective (Keras & Anime Faces)

Given your interest in anime, you might find this alternative implementation interesting. It uses Keras/TensorFlow to train a DCGAN on a dataset of anime faces. While the core principles are the same, the implementation details offer a different perspective.

Conclusion

Congratulations! You've just implemented a Deep Convolutional GAN, a foundational architecture in generative modeling. We've moved beyond the toy examples of fully-connected GANs and built a model capable of learning from complex image datasets.

Key Takeaways:

• DCGANs are not a new theory but a set of architectural best practices for building stable GANs.
• Strided and transposed convolutions allow the networks to learn their own spatial upsampling and downsampling.
• Batch Normalization is a key stabilizer, helping to regulate gradient flow and prevent mode collapse.
• Specific activation functions (LeakyReLU in the discriminator, Tanh output in the generator) and optimizer settings are crucial for successful training.
• The implementation requires careful management of tensor shapes as data flows through the convolutional layers of the generator and discriminator.

Preview of the next lesson:
The DCGAN architecture significantly improves training stability. However, the underlying loss function (Binary Cross-Entropy) still has issues. It can saturate, cause vanishing gradients, and doesn't correlate well with the visual quality of the generated images. In our next lesson, we will explore Wasserstein GANs (WGANs), which introduce a new, theoretically grounded loss function to address these problems, leading to even more stable training and a meaningful loss metric.