Designing Effective Reward, Success, and Termination Criteria

Welcome back. In the previous lesson, you specified the policy interface: what an embodied agent can observe and what physical commands it can issue. That interface tells us what the agent can do. This lesson defines what counts as doing the right thing.

In reinforcement learning, reward, success, and episode ending are related but distinct design choices. Confusing them is one of the fastest ways to create a policy that optimizes a convenient number while failing the physical task you actually care about. We will make those choices concrete using an inverted-pendulum balancing task, then develop a process for checking a task specification for loopholes.

---

Three signals with three jobs

At each control step, the environment returns a scalar reward $r_t$. A learning algorithm typically seeks a policy that maximizes expected discounted return,

$$G_t=\sum _{k=0}^{\infty }{\gamma }^k{r}_{t+k+1},$$

where $\gamma$ is a discount factor between zero and one. Reward is therefore a training signal: it gives the optimization process a direction.

But training reward is not the same thing as a statement that the task has truly been achieved. A robust environment usually defines three separate concepts.

Concept	Main question	Typical form	Primary use
Reward	How useful was this transition for learning?	Scalar at every step	Optimize a policy
Success	Did the completed behavior meet the task specification?	Boolean criterion	Report evaluation performance
Termination	Has the environment reached a true terminal state?	terminated=True/False	End an episode because of success or irreversible failure
Truncation	Did an externally imposed limit stop the episode?	truncated=True/False	Stop an episode due to a time or data-collection budget

A simple grid-navigation task can align all of them: reaching the goal gives reward $+1$, counts as success, and terminates the episode. In physical AI, this neat alignment is often too crude.

For example, consider a robot arm asked to place an object into a bin:

• A reward may encourage the object to move closer to the bin throughout the attempt.
• Success should mean the object is inside the bin, at rest, and released by the gripper.
• Termination may occur after success, after an object breaks or leaves the workspace, or after a dangerous collision.
• Truncation may occur when a 20-second evaluation budget expires, even though the arm could in principle continue.

The key discipline is this:

Write the desired physical outcome first. Then define a measurable success test. Only afterward design a reward that makes learning that outcome feasible.

The Gymnasium documentation frames this ordering as an environment-design problem rather than a coding detail.

---

A reward is a proxy, not your intent

Suppose you want a mobile robot to reach a target. The actual goal might be:

Reach the target safely, without collision, in a reasonable time, and remain there stably.

A simple numerical proxy is negative Euclidean distance:

$$r_t=-\Vert p_t-p_{\text{goal}}{\Vert }_2.$$

This is often useful—but it is not the full objective. It says nothing directly about collisions, unsafe speed, arrival direction, whether the robot has actually stopped, or whether it crossed forbidden terrain.

This mismatch is an instance of Goodhart’s law: when a measure becomes a target, optimizing pressure can make it cease to measure what you cared about. The policy is not being stubborn or “misunderstanding.” It is doing exactly what the reward function makes advantageous.

The issue is especially vivid in embodied environments, because the agent can exploit dynamics, geometry, sensors, software state, resets, and termination logic.

A useful rule follows:

Never ask only, “What behavior does this reward encourage when things go well?” Also ask, “What behavior does it encourage when the agent discovers an unusual state, a boundary, or a simulator flaw?”

---

A worked specification: balancing an inverted pendulum

Consider the cart-pole system shown above. The cart moves along a rail, and its acceleration changes the pole’s angle. We want the system to learn controlled balancing, not merely to delay an obvious failure.

Let:

• $x_t$ be cart position along the rail;
• ${\dot{x}}_t$ be cart velocity;
• ${\varphi }_t$ be the pole angle relative to upright;
• ${\dot{\varphi }}_t$ be pole angular velocity;
• $a_t$ be the normalized motor command.

State the task in operational language

A good task statement is more specific than “keep the pole upright”:

Starting from a perturbed but recoverable state, maintain the pole near upright while keeping the cart near the center of its allowed rail region, using bounded control effort.

This task contains several distinct requirements:

1. Uprightness: the pole must not merely pass through vertical.
2. Stability: it must remain controlled for a sustained interval.
3. Workspace constraint: the cart must not race to a rail boundary.
4. Control constraint: actions must stay within actuator limits.
5. Initial-state policy: resets must be challenging enough that “do nothing” is not a universal solution.

Define success before reward

Here is one possible evaluation success criterion. At the end of an episode, success is true only if all of the following conditions have held for the last $H=250$ control steps:

$$|{\varphi }_t|\le 5^{\circ },$$ $$|{\dot{\varphi }}_t|\le 0.5\text{ rad/s},$$ $$|x_t|\le 0.20\text{ m},$$ $$|{\dot{x}}_t|\le 0.25\text{ m/s}.$$

If the control interval is $0.02$ seconds, then $250$ steps represents five seconds of stable balance. This rules out a policy that briefly swings through the upright state at high speed.

Notice that the success test does not say “the pole looks upright in the rendered frame.” Rendering is not a reliable measurement channel. It uses the simulator’s physical state. For a hardware deployment, the equivalent criterion would need stated sensor or estimator sources, tolerances, and timing rules.

You should also decide whether success ends the episode. For this episodic balancing task, ending on success is sensible: the agent has demonstrated the target behavior. A continuing task could instead record a success event and issue a new reference target without resetting, but that is a different task definition.

Define true terminal failures

A terminal failure is a state from which the task episode should not continue. For this environment, possible failure conditions are:

$$|{\varphi }_t|\ge 12^{\circ },$$

or

$$|x_t|\ge 0.80\text{ m}.$$

The exact numbers are design parameters, but the conceptual separation matters:

• $5^{\circ }$ is a success-quality tolerance;
• $12^{\circ }$ is a failure boundary;
• the region between them gives the policy room to recover.

If the pole falls beyond the recoverable region or the cart reaches the rail limit, set terminated=True. The episode has reached a terminal task state, not merely exhausted a convenient computational budget.

Define truncation separately

Suppose each balancing attempt is allowed at most $500$ control steps, or ten seconds. If neither success nor failure has occurred by then, set:

terminated = False
truncated = True

The time limit is not a physical success or failure by itself. It is an external episode limit chosen for training throughput, benchmark consistency, or safety.

This distinction matters for learning implementations: many value-based and actor-critic methods treat terminal transitions differently from time-limit truncations when estimating what may happen after the final recorded step.

Gymnasium-Robotics’ Point Maze documentation illustrates this distinction directly: reaching the goal can terminate an episodic task, while an episode-length limit causes truncation.

---

Designing a reward that supports, rather than replaces, success

A sparse reward for the pendulum could be:

$$r_t=\{\begin{array}{ll}1,& \text{if success occurs at time }t+1,\\ 0,& \text{otherwise.}\end{array}$$

This specification is clean: it directly rewards the defined outcome. It may nevertheless be hard to learn from, because random exploration is unlikely to balance stably for five seconds.

A dense reward can offer a more informative learning signal. First define a normalized distance-to-desired-balance measure:

$$d(s_t)=w_{\varphi }{\left(\frac{{\varphi }_t}{{\varphi }_{\text{safe}}}\right)}^2+w_{\dot{\varphi }}{\left(\frac{{\dot{\varphi }}_t}{{\dot{\varphi }}_{\text{safe}}}\right)}^2+w_x{\left(\frac{x_t}{x_{\text{safe}}}\right)}^2+w_{\dot{x}}{\left(\frac{{\dot{x}}_t}{{\dot{x}}_{\text{safe}}}\right)}^2.$$

The constants $w_{\varphi }$, $w_{\dot{\varphi }}$, $w_x$, and $w_{\dot{x}}$ express relative importance. They are not arbitrary cosmetic settings. Large $w_{\varphi }$ says pole angle matters far more than cart centering; large $w_x$ says the cart should stay centered even if a larger excursion would aid recovery.

One reward design is:

$$r_t=20{\mathbb{I}}_{\text{success}}-20{\mathbb{I}}_{\text{failure}}+c\left[d(s_t)-d(s_{t+1})\right]-\eta \Vert a_t{\Vert }_2^2.$$

Its terms have different purposes:

Term	Intended role	Design risk
Success bonus	Makes the desired completed behavior valuable	Must be tied to the full success test, not a visual proxy
Failure penalty	Makes unrecoverable falls and rail violations undesirable	Too large a penalty can discourage exploration
Progress term	Rewards reducing distance from stable balance	Can cause oscillation if poorly scaled or incomplete
Action penalty	Discourages needless chattering and extreme effort	Too large a penalty produces passive, underpowered control

The progress term rewards improvement rather than simply surviving for another step. In an undiscounted sum, moving away from the target and then returning generally cancels its own progress reward:

$$[d(s_t)-d(s_{t+1})]+[d(s_{t+1})-d(s_{t+2})]=d(s_t)-d(s_{t+2}).$$

That does not prove the reward is safe. Discounting, termination bonuses, numerical clipping, and unmodeled simulator behavior can still create incentives to cycle or exploit thresholds. It does, however, reveal why “rewarding progress” is often safer than giving the same positive reward merely for remaining alive.

For a more principled shaping method, define a potential $\Phi (s)$ and add:

$$F(s_t,s_{t+1})=\gamma \Phi (s_{t+1})-\Phi (s_t).$$

Under the usual Markov assumptions, this potential-based shaping preserves the optimal policy of the original discounted task. It does not eliminate all implementation or simulator loopholes, but it is preferable to adding arbitrary bonuses with unknown strategic effects.

Do not hide essential requirements in tiny penalties

Suppose keeping the cart inside its workspace is physically essential. It should appear in the terminal-failure and success specification, not only as a small position penalty. Otherwise the optimizer can rationally decide that a rail collision is worth the reward gained from temporarily improving pole angle.

Likewise, if a manipulation task requires placing rather than merely touching an object:

• define success using object pose, containment, release, and post-release stability;
• do not rely only on a reward for decreasing gripper-to-object distance;
• terminate or penalize clearly defined damaging contacts if they are safety-critical;
• test whether the object can be pushed, thrown, or visually occluded to create a false positive.

A reward may include auxiliary terms for distance, smoothness, or energy. The non-negotiable task constraints must be specified independently.

---

Implement the criteria in a transparent order

Within step(), calculate quantities from the post-action state. Then derive success, failure, termination, truncation, and reward in an order that makes each dependency inspectable.

def step(self, action):
    action = np.clip(action, -1.0, 1.0)
    self._apply_action(action)
    self._simulate_substeps()

    state = self._get_physical_state()
    distance = self._balance_distance(state)

    success = self._stable_for_required_duration(state)
    failure = self._pole_has_fallen(state) or self._cart_left_workspace(state)

    terminated = bool(success or failure)
    truncated = bool(
        self.elapsed_steps >= self.max_episode_steps and not terminated
    )

    progress = self.previous_distance - distance
    reward = (
        20.0 * float(success)
        - 20.0 * float(failure)
        + 0.2 * progress
        - 0.001 * np.square(action).sum()
    )

    self.previous_distance = distance

    info = {
        "is_success": bool(success),
        "failure": bool(failure),
        "pole_angle": state.pole_angle,
        "cart_position": state.cart_position,
        "balance_distance": distance,
    }

    return self._get_obs(), reward, terminated, truncated, info

Several details here are deliberate:

• success is exposed in info for evaluation, even when the reward is dense.
• failure is logged separately, so a timeout is not silently counted as a fall.
• Time-limit truncation does not override a simultaneous success or failure event.
• The reward is calculated from named components that can be plotted and audited.
• Clipping action commands prevents undefined actuator inputs, but clipping should also be logged during debugging; frequent saturation may mean the action scaling or task is poorly chosen.

The exact coefficients should be treated as hypotheses, not facts. Inspect rollouts and plots of angle, position, reward components, action magnitude, success rate, and termination reasons. A reward that looks reasonable on paper can still induce a bizarre local strategy.

---

A specification audit for unintended incentives

Before investing in training, conduct an adversarial review. Imagine that the policy is exceptionally good at exploiting your simulator but has no common sense.

1. Audit the success measurement

Ask whether success can be triggered without achieving the physical result.

For a pick-and-place task, “gripper is near object” is vulnerable. “Object centroid is inside the target region, gripper is open, and object speed remains below a threshold for $K$ steps” is much stronger.

For visual evaluation, ask whether occlusion, camera placement, lighting, or a hand placed in front of the object can create a false measurement. Use independent sensors or simulator state for evaluation where possible, and make the measurement pathway explicit.

2. Audit the reward proxy

For each reward component, identify the behavior it favors in isolation.

Reward component	Intended behavior	Plausible exploit
Per-step alive bonus	Avoid falling	Delay failure without accomplishing the task
Distance-to-object reward	Approach object	Hover near or push object away after partial progress
Contact bonus	Make grasp contact	Repeatedly tap or collide
Velocity reward	Move forward	Fall, slide, or exploit initial potential energy
Visual cleanliness score	Remove mess	Obscure the camera or reward sensor

An exploit is not necessarily a reason to remove a term. It is a prompt to add missing success conditions, constraints, better measurements, or a revised environment.

3. Audit episode boundaries and resets

The policy will learn from the distribution of initial states and the rules at episode boundaries.

Check whether:

• an episode can begin in a success state;
• a reset creates free momentum, favorable object placement, or stored potential energy;
• the policy can accumulate reward near a terminal boundary;
• success is checked before the environment resets;
• a timeout is being misreported as success or failure;
• a continuing task accidentally lets the agent collect the same reward repeatedly without doing new work.

4. Separate training optimization from final evaluation

Train with dense shaping if needed, but evaluate with the task’s explicit success metric and constraint-violation rates. A report containing only mean training return is weak evidence: the policy may have learned to exploit shaping terms.

For the cart-pole, meaningful evaluation reports include:

• fraction of episodes satisfying the five-second balance criterion;
• fraction ending in pole failure;
• fraction ending at the cart workspace limit;
• fraction truncated by time limit;
• mean and peak action magnitude;
• performance under varied initial angles and cart positions.

These measures make it much harder for a high reward to conceal a poor physical behavior.

---

Key takeaways

Reward, success, termination, and truncation must be designed as separate signals:

• Reward makes learning possible, but is only a proxy for the desired behavior.
• Success is a measurable definition of genuine task completion and should drive evaluation.
• Termination represents success or irreversible task failure.
• Truncation represents an external limit, such as a maximum episode length.

For the inverted pendulum, a robust specification requires sustained uprightness, low motion, bounded cart position, clear fall limits, and a separate time budget. A dense reward can guide learning through progress and modest control regularization, but it must not replace those explicit success and safety conditions.

Next, we will return to the observation side of the problem: how to recognize when the current observation cannot distinguish states requiring different actions, and why an embodied policy then needs memory.