When Embodied Policies Need Memory

Good to see you again. In the previous lesson, you separated the physical task you want from the reward proxy used to train it. That distinction matters here: a perfectly specified reward still cannot produce good behavior if the policy lacks the information needed to choose correctly.

Earlier in this module, we described embodied tasks as POMDPs: the world has a physical state, but the agent receives only observations produced by its sensors. We now make the practical consequence precise. You will learn how to identify perceptual aliasing—different physical situations that look the same to the policy—and determine when a policy must retain history rather than react to the latest observation alone.

---

The central question: can one observation support one correct decision?

Let the physical state of the world at time $t$ be $s_t$, and let the observation available to the agent be $o_t$. The physical state might include robot pose, velocity, contacts, object locations, battery state, actuator temperatures, or an unobserved task instruction. The observation might include an RGB image, depth image, joint encoders, force readings, and a goal vector.

A reactive or memoryless policy maps only the current observation to an action:

$$a_t\sim \pi (a\mid o_t).$$

This is appropriate when the current observation contains all decision-relevant information. The same visible input always permits the same best response.

A memory-based policy instead conditions on a history, or on an internal summary of that history:

$$h_t=(o_1,a_1,o_2,a_2,\dots ,a_{t-1},o_t),$$ $$a_t\sim \pi (a\mid h_t).$$

In practice, the full history is rarely stored literally. The agent maintains a compact internal memory $m_t$:

$$m_t=f(m_{t-1},o_t,a_{t-1}),$$ $$a_t\sim \pi (a\mid o_t,m_t).$$

The crucial distinction is not whether a task is technically a POMDP. Almost every physical system is partially observed to some degree. The decisive question is:

Do two reachable situations produce the same current observation while requiring different actions for good task performance?

If the answer is yes, the current observation is insufficient. A policy needs either memory, a better observation space, or an information-gathering action before it can make the decision reliably.

---

Perceptual aliasing: identical input, incompatible actions

Perceptual aliasing occurs when multiple true states are mapped to the same observation. The policy sees one input, even though the physical world is in meaningfully different situations.

Consider the T-maze in the Prediction model diagram. At the beginning of a trial, the robot sees a red or blue cue. Later it reaches a junction whose local geometry looks the same regardless of which cue appeared. Suppose red means “turn left” and blue means “turn right.”

At the junction:

• the current camera observation can be identical in red-cue and blue-cue trials;
• the required action differs;
• the cue is no longer visible.

A reactive policy must choose the same action distribution whenever it receives the junction image. If it always turns left, it fails all blue trials. If it randomizes evenly, it succeeds only about half the time under balanced contexts. Randomness can be a reasonable compromise when the agent truly cannot distinguish cases, but it does not recover the missing information.

The required memory can be minimal. One internal bit is sufficient:

$$m_t=\{\begin{array}{ll}0,& \text{if the earlier cue was red},\\ 1,& \text{if the earlier cue was blue}.\end{array}$$

The policy at the junction can then use both the present observation and that bit.

This is fundamentally different from a task in which the two hidden states produce the same image but also have the same optimal action. Hidden variables do not automatically require memory. For example, a navigation robot need not remember the exact color of a wall behind it if that color never affects dynamics, reward, safety, or later observations.

---

A formal test for whether memory is necessary

A useful formal test begins with two histories, $h_t$ and ${\tilde{h}}_t$, that lead to the same current observation:

$$o_t(h_t)=o_t({\tilde{h}}_t).$$

Now ask whether the best action differs after those two histories. Let $Q^{\ast }(h_t,a)$ denote the expected future return from taking action $a$ after history $h_t$, assuming optimal behavior thereafter.

A reactive policy is inadequate for optimal decision-making if there are reachable histories for which:

$$\underset{a}{\mathrm{argmax}}{Q}^{\ast }(h_t,a)\ne \underset{a}{\mathrm{argmax}}{Q}^{\ast }({\tilde{h}}_t,a),$$

while their present observations are the same.

Put less formally:

1. Find two physically possible situations that look the same now.
2. Check whether choosing the same action in both cases causes failure, unnecessary risk, or avoidable loss of reward.
3. Identify information earlier in the trajectory that distinguishes them.
4. If that earlier information must affect the present action, the policy needs access to memory.

The underlying POMDP need not be solved exactly to perform this analysis. In environment design, a carefully constructed counterexample is often enough to reveal a missing input.

The following short video gives the POMDP perspective: physical states exist, but the agent acts from observations; when the last observation is incomplete, previous observations and internal state can improve the decision.

---

Three common embodied cases

Motion hidden by a single image

A first-person camera image can show that a mobile robot is centered in a corridor, but it may not reveal its velocity. The robot could be at the same visible location while:

• moving rapidly toward a wall;
• moving slowly toward it;
• stopped;
• moving away from it.

For a velocity-controlled or force-controlled robot, these cases can require different actions. Braking hard may be correct in the first case, while accelerating may be correct in the last.

Two images separated by a known control interval can provide a velocity estimate:

$${\hat{v}}_t\approx \frac{{\hat{p}}_t-{\hat{p}}_{t-1}}{\Delta t}.$$

A frame stack can therefore solve this particular kind of aliasing when camera motion is sufficiently informative. If the dynamics are more complex, past commanded actions can matter as well: the same current image can follow either a recent acceleration command or a recent braking command.

The paper below uses this exact navigation example and introduces the idea that some POMDPs become effectively observable when a short observation-action history is available.

A task context that occurred in the past

The T-maze cue is a simple version of a common robotics pattern: an early observation establishes a context that is needed later.

Examples include:

• A robot receives the instruction “put the red object in the left bin,” then later sees both objects but no longer receives the language instruction as input.
• An assembly robot scans a part at the start of a cycle to determine which variant it has; later stages look visually similar but require a different insertion force.
• A delivery robot observes which room contains a requested item, travels through visually repetitive corridors, and later reaches a junction.

The essential variable is not necessarily geometric. It may be a goal identity, object category, task phase, prior contact event, or an instruction. If the policy must use it later, it belongs either in the persistent observation input or in memory.

Occlusion and contact state

A robot hand can lose visual contact with an object while moving it behind an obstacle. Suppose the camera view at the next instant looks the same whether the object is securely grasped or has slipped. Continuing the planned arm motion may be safe in the first state and damaging in the second.

Before adding a recurrent policy, inspect the sensor design:

• Does the gripper-width encoder indicate whether the fingers closed?
• Can motor current or tactile sensing distinguish a firm grasp from empty closure?
• Is there a wrist force signal that indicates unexpected load loss?
• Is the object visible from another viewpoint?

If a current force, tactile, or proprioceptive measurement can disambiguate the states, adding it to $o_t$ may be clearer and more robust than asking a network to infer grasp state from image history. Memory is not a substitute for an omitted high-value sensor.

---

How much memory is enough?

The required memory horizon depends on how far back the distinguishing evidence occurred.

For some tasks, a fixed window of recent data is sufficient. Define a suffix containing the last $m$ observations and relevant preceding actions:

$$z_t=(o_{t-m+1},a_{t-m+1},\dots ,a_{t-1},o_t).$$

If this suffix determines the decision-relevant latent state on all reachable trajectories, then an $m$-step policy can act on $z_t$ rather than the entire history.

A useful interpretation is:

Situation	Minimum useful memory
Estimate visible motion from camera frames	Usually one or several earlier frames
Account for actuator delay or momentum	Recent actions plus recent observations
Remember a start-of-episode cue at a late junction	Potentially the whole episode, compressed into a small context variable
Track pose from noisy landmarks	A recursively updated state estimate
Complete a multistage task	A task-phase variable plus any unresolved physical uncertainty

There are three standard ways to supply this information.

1. Augment the current observation

This is the simplest solution when you know the missing variable can be measured or computed directly. For example, append joint velocities, gripper force, a goal ID, elapsed task phase, or an estimated object pose.

This changes the policy interface so that the agent can again be reactive:

$$a_t\sim \pi (a\mid {\tilde{o}}_t),$$

where ${\tilde{o}}_t$ is an enriched observation. It is often easier to diagnose than learned memory.

2. Stack a fixed history window

Frame stacking provides the policy with recent inputs explicitly:

$${\tilde{o}}_t=(o_{t-k+1},\dots ,o_t).$$

For visual control, the policy should normally receive the recent actions too when they help explain state evolution. A stack of observations alone cannot always distinguish “I am moving because I accelerated” from “I am moving because something pushed me.”

Fixed windows are effective when the information is local in time, such as velocity estimation. They become inefficient when a cue must be remembered for hundreds of steps.

3. Maintain a learned or model-based internal state

A recurrent network can update an internal state:

$$m_t=f_{\theta }(m_{t-1},o_t,a_{t-1}).$$

An LSTM or GRU is intended to learn which observations to retain and which to forget. This is flexible, but it is not magic: long-range information can still be forgotten, and training a recurrent policy requires preserving sequence order during data collection and learning.

When a reasonably accurate state-transition and sensor model is available, a belief state is often more interpretable. It represents a probability distribution over possible physical states:

$$b_t(s)=P(s_t=s\mid h_t).$$

After applying action $a_t$, prediction uses the dynamics model:

$${\bar{b}}_{t+1}(s^{\prime })=\sum _{s}P(s^{\prime }\mid s,a_t)b_t(s).$$

After the next observation arrives, the observation model updates that prediction:

$$b_{t+1}(s^{\prime })=\frac{P(o_{t+1}\mid s^{\prime },a_t){\bar{b}}_{t+1}(s^{\prime })}{{\sum }_{\hat{s}}P(o_{t+1}\mid \hat{s},a_t){\bar{b}}_{t+1}(\hat{s})}.$$

The belief does not need to identify one exact state. It summarizes the uncertainty that matters for action. In later modules, this idea will reappear as localization and Kalman filtering.

---

A practical diagnostic workflow

When designing or debugging an embodied environment, use this workflow before deciding that a recurrent policy is necessary.

1. List the latent variables that could affect the decision

Include variables that influence:

• future dynamics, such as velocity, contact mode, friction, or actuator lag;
• reward and success, such as the active goal or task phase;
• safety, such as unseen obstacles, collision state, or grasp stability;
• future sensing, such as camera pose, occlusion, or map location.

Do not list every unmeasured detail. Focus on variables whose different values could make a different action preferable.

2. Write down the actual policy input

Be literal. If the observation consists of a $128\times 128$ RGB image, two joint angles, and a goal coordinate, then it does not contain velocity unless velocity is inferable from those quantities at the same time step. A rendered image may show a robot near an object, but it does not necessarily reveal contact force or whether the object is attached.

3. Construct an aliasing table

For each suspected ambiguity, make the cases concrete.

Same current observation	Hidden difference	Why it matters	Information that resolves it
Robot centered in corridor	Fast approach versus slow retreat	Brake versus accelerate	Earlier frames and prior actions
Identical T-junction	Red versus blue start cue	Left versus right turn	Stored cue or persistent goal input
Closed gripper out of view	Object held versus object dropped	Transport versus recovery behavior	Tactile, force, or earlier contact history
Similar visual scene	Different map location	Different route to goal	Localization estimate and map history

If you cannot create a plausible row with incompatible good actions, memory may not be essential.

4. Identify the shortest sufficient history

Ask how far back the resolving evidence lies. If two frames distinguish velocity, do not use a huge recurrent model merely because the environment is partially observed. If a one-bit instruction from episode start matters at the end, two-frame stacking is not enough.

Include actions in this analysis. Actions change the world, so they are evidence about the state. They are especially important under delayed control, partial actuation, or unmodeled disturbances.

5. Choose the least complicated reliable remedy

A useful ordering is:

1. Add a direct, physically justified measurement when one exists.
2. Add an explicit state estimator when a model and sensors support it.
3. Use a short observation-action stack when the needed horizon is known and short.
4. Use recurrence when relevant information is variable-length, high-dimensional, or difficult to hand-engineer.

This is a design preference, not a prohibition against recurrence. A learned memory may be exactly right for long-horizon visual navigation or language-conditioned tasks. The point is to make the information pathway explicit.

---

Verify that memory is truly helping

A policy’s failure does not by itself prove that memory is needed; it may be poorly trained, poorly rewarded, or constrained by the wrong action space. Test the information hypothesis directly.

In simulation, create paired rollouts:

1. Initialize two states with nearly identical current observations.
2. Alter only the suspected hidden variable, such as velocity, prior cue, object attachment state, or map location.
3. Determine whether the task-optimal action should differ.
4. Measure whether a reactive policy must compromise while a history-based policy can act differently.

Then compare architectures under equivalent training conditions:

• current observation only;
• observation-action stack of several lengths;
• recurrent policy;
• explicit estimator plus reactive controller, when feasible.

Evaluate on the specific hidden-state variations that motivated memory. For the camera-velocity case, test changed initial velocities. For the T-maze, balance cue colors and randomize corridor layout. For occluded manipulation, vary whether the object is held, displaced, or dropped while controlling the visible scene.

Two implementation details are easy to miss:

• Reset recurrent state at episode boundaries. Otherwise, information from a prior episode can leak into the next one.
• Preserve temporal order in training data. Randomly sampling isolated transitions destroys the very dependencies the recurrent model is meant to learn.

A strong diagnosis combines task reasoning with evidence: demonstrate the aliased cases, specify the minimum information needed to disambiguate them, and show that supplying this information improves the relevant failure mode.

---

Key takeaways

An embodied policy requires memory when the same current observation can arise from different reachable situations that require different actions. This is perceptual aliasing: the observation collapses decision-relevant distinctions in the physical state.

The practical test is to find two same-observation histories and ask whether their optimal actions conflict. If they do, a reactive policy can only compromise or randomize. The remedy may be an added sensor, an explicit state estimate, a short history stack, or a learned recurrent state.

Memory should retain decision-relevant information, not the entire past. Recent frames may be enough to infer velocity; a long-delayed task cue may require a compact persistent context; noisy localization may call for a belief state.

This completes the Embodiment and Sensorimotor Intelligence module. Next, the course turns to spatial reasoning: representing points, poses, and robot motion consistently across coordinate frames.