Actor-Critic Architectures: A2C Implementation

Hello! Let's dive into the next stage of our reinforcement learning journey.

In our last lesson, we implemented REINFORCE, our first policy gradient algorithm. We saw that while it works by directly optimizing the policy, it suffers from high variance because it relies on the noisy, full-episode Monte Carlo return $G_t$ as its learning signal. A single unlucky outcome could unfairly penalize a whole sequence of good actions. We concluded by introducing the idea of subtracting a baseline, like the state-value function $V(s_t)$, to create a more stable learning signal: the advantage.

This brings us directly to today's topic. Instead of just assuming we have a value function, we are going to learn it alongside the policy. This powerful combination is the core of Actor-Critic methods. Your goal for this lesson is to build an Actor-Critic architecture, specifically the popular variant known as Advantage Actor-Critic (A2C).

1. The Actor-Critic Framework

The name "Actor-Critic" perfectly describes the architecture. We have two distinct components, typically implemented as two neural networks or two "heads" of a single network:

• The Actor is the policy, ${\pi }_{\theta }(a|s)$. It observes the state and decides which action to take. This is the component we had in REINFORCE.
• The Critic is a value function, typically the state-value function $V_{\varphi }(s)$. It doesn't choose actions; it only evaluates the states visited by the Actor, judging "how good" they are.

The two work in a feedback loop: the Actor performs, and the Critic provides feedback, which the Actor uses to improve.

To get a more intuitive feel for this dynamic, let's start with a short conceptual overview.

2. From Policy Gradients to Advantage Updates

Recall the REINFORCE update rule with a baseline:
${\nabla }_{\theta }J(\theta )\approx \mathbb{E}[(G_t-V(s_t)){\nabla }_{\theta }\log {\pi }_{\theta }(A_t|S_t)]$
The term $(G_t-V(s_t))$ is an estimate of the advantage function $A(s_t,A_t)$.

Actor-Critic methods refine this in two key ways:

1. We explicitly learn the value function $V_{\varphi }(s_t)$ with a parameterized Critic network.
2. Instead of waiting for the full Monte Carlo return $G_t$, we use a one-step Temporal Difference (TD) target, $r_{t+1}+\gamma V_{\varphi }(s_{t+1})$.

This gives us the TD error, ${\delta }_t$, which serves as a low-variance, single-step estimate of the advantage:

$${\delta }_t=\underset{\text{TD Target}}{\underbrace{r_{t+1}+\gamma V_{\varphi }(s_{t+1})}}-\underset{\text{Current Value Estimate}}{\underbrace{V_{\varphi }(s_t)}}$$

This ${\delta }_t$ becomes the learning signal for both the Actor and the Critic.

• Critic's Goal: To make its value estimates more accurate. It is trained to minimize the TD error. A common loss function is the mean squared error of the TD error: $$L_{\text{critic}}={\delta }_t^2$$
• Actor's Goal: To take actions that lead to positive advantage. Its loss is the negative log-probability of the action, scaled by the advantage. We use .detach() on the advantage term because we only want the Critic to provide a numeric score, not to receive gradients from the Actor's loss. $$L_{\text{actor}}=-\log {\pi }_{\theta }(A_t|S_t)\cdot {\delta }_t.\text{detach()}$$

This structure is the foundation of the Advantage Actor-Critic (A2C) algorithm.

3. Building an A2C Agent from Scratch

Now, let's translate this theory into a working implementation. We will build an A2C agent step-by-step. Since you're proficient in Python, walking through a detailed coding tutorial is the most effective way to master this.

The following video provides an exceptionally thorough guide to building an Actor-Critic agent using TensorFlow 2. It covers the theory, the network architecture, the agent logic, and the final training loop. We will work through it in sections.

Test your understanding!

In the video at timestamp 11:55, the delta term is defined as:

$$\delta =r+\gamma \cdot V(s^{\prime })-V(s)$$

And the loss functions are given for the Critic (${\delta }^2$) and the Actor ($-\delta \cdot \log \pi (a|s)$).

Why do we want to make the Actor's loss negative? What would happen if we tried to minimize $+\delta \cdot \log \pi (a|s)$?

Show answer

Gradient descent minimizes a loss function. Our goal is to make actions that lead to a positive advantage ($\delta >0$) more likely.

The term $\nabla \log \pi (a|s)$ is the direction in parameter space that increases the probability of taking action $a$ in state $s$.

• If $\delta >0$ (a good action), we want to move our parameters in the direction of $\nabla \log \pi (a|s)$. Minimizing $-\delta \cdot \log \pi (a|s)$ is equivalent to performing gradient ascent on $+\delta \cdot \log \pi (a|s)$, which achieves this.
• If we tried to minimize $+\delta \cdot \log \pi (a|s)$, the optimizer would push the parameters in the opposite direction of $\nabla \log \pi (a|s)$ when $\delta >0$. This would mean punishing good actions and rewarding bad ones, preventing the agent from learning.

Network and Agent Implementation

With the theory in place, let's implement the model. A common and efficient design is to use a single neural network with a shared "body" for feature extraction and two separate "heads":

1. A policy head that outputs logits for the action distribution (the Actor).
2. A value head that outputs a single number representing the state value (the Critic).

This allows the Critic to benefit from the representations learned for the Actor, and vice-versa.

The code in the video provides a complete, working example. For reference, here is the core logic from the learn() function, annotated to match our theoretical discussion:

# Inside the learn() method, within a tf.GradientTape context:

# 1. Get value estimates from the Critic for current and next states
state_value, _ = self.actor_critic(state)
new_state_value, _ = self.actor_critic(new_state)

# 2. Calculate the one-step advantage (TD Error)
# Note: The video uses (1 - int(done)) to zero out the value of the next state if it's terminal.
advantage = reward + self.gamma * new_state_value * (1 - int(done)) - state_value

# 3. Calculate the Critic loss
critic_loss = advantage ** 2

# 4. Calculate the Actor loss
# Get log probability of the action that was actually taken
action_probs = tfp.distributions.Categorical(probs=probs)
log_prob = action_probs.log_prob(self.action) # self.action was saved from choose_action()

actor_loss = -log_prob * advantage # The video uses tape.stop_gradient(advantage) implicitly

# 5. Sum the losses and apply gradients
total_loss = actor_loss + critic_loss
grads = tape.gradient(total_loss, self.actor_critic.trainable_variables)
self.actor_critic.optimizer.apply_gradients(zip(grads, ...))

This implementation directly maps our A2C update rules into TensorFlow code.

4. A2C vs. A3C and Practical Enhancements

The algorithm we've built is often called A2C (Advantage Actor-Critic). It is a synchronous, deterministic version of the original A3C (Asynchronous Advantage Actor-Critic). A3C uses multiple parallel workers, each with its own copy of the environment, to gather experience asynchronously. This helps to de-correlate the agent's experience, which stabilizes learning. A2C, in contrast, often waits for a batch of experience from all workers before performing a single update. With modern GPUs, this synchronous approach is often more efficient.

To further improve stability and encourage exploration, a common addition to the A2C loss function is an entropy bonus. Entropy is a measure of randomness in the policy's action distribution. By adding the policy's entropy to the objective function, we encourage the agent to maintain a more stochastic policy and avoid collapsing to a suboptimal deterministic action too early.

The final loss function often looks like this:
$L=\underset{\text{Actor Loss}}{\underbrace{-\log {\pi }_{\theta }(A_t|S_t)\cdot {\delta }_t.\text{detach()}}}+{\beta }_v\underset{\text{Critic Loss}}{\underbrace{{\delta }_t^2}}-{\beta }_e\underset{\text{Entropy Bonus}}{\underbrace{H({\pi }_{\theta }(s_t))}}$
where ${\beta }_v$ and ${\beta }_e$ are coefficients to weight the different loss components.

Conclusion

Congratulations! You have now moved from the high-variance REINFORCE algorithm to the much more stable and powerful Actor-Critic paradigm. By training a Critic to provide a learned baseline, we can generate a much more reliable learning signal—the advantage—to guide our Actor.

Key Takeaways:

• Actor-Critic methods combine a policy network (the Actor) and a value network (the Critic).
• The Actor decides which actions to take, while the Critic evaluates the states visited by the Actor.
• Advantage Actor-Critic (A2C) uses the TD error (${\delta }_t=r_{t+1}+\gamma V(s_{t+1})-V(s_t)$) as an estimate for the advantage.
• The Critic is trained to minimize the TD error (e.g., $L_{critic}={\delta }_t^2$).
• The Actor is updated using a policy gradient scaled by the advantage from the Critic (e.g., $L_{actor}=-\log \pi (a|s)\cdot {\delta }_t$).
• Practical implementations often use a shared network with two heads and add an entropy bonus to the loss to encourage exploration.

Preview of the Next Lesson:

A2C is a huge step forward, but it still has a vulnerability. The policy updates are "unconstrained." A single bad batch of data could lead to a large, destructive update to the policy from which it might not recover. How can we ensure that we improve the policy without risking a catastrophic performance drop?

This is the problem solved by our next algorithm: Proximal Policy Optimization (PPO). PPO is one of the most widely used and robust RL algorithms today, and it builds directly on the Actor-Critic foundation you've established in this lesson.