Formulating Embodied Tasks as Partially Observable Markov Decision Processes

Hello again. In the previous lesson, we traced the perception-action loop: an embodied agent senses an incomplete world, acts through a body, and uses the consequences to guide later actions. The formal question now is: how do we describe that loop precisely enough to reason about decisions under uncertainty?

A partially observable Markov decision process (POMDP) is one answer. It represents the world’s actual state, which the agent cannot fully see; the actions available to the agent; the noisy observations it receives; and the task objective. This lesson develops a POMDP formulation for a small robot manipulation task, then shows why an agent must maintain a belief rather than simply react to its latest sensor reading.

---

From an embodied loop to a decision model

An ordinary Markov decision process (MDP) assumes that, at each time step, an agent observes the true state $s_t$ of its environment, chooses an action $a_t$, receives a reward, and reaches a new state $s_{t+1}$.

The key assumption is the Markov property:

$$P(s_{t+1}\mid s_{0:t},a_{0:t})=P(s_{t+1}\mid s_t,a_t)$$

In words: once the current state and action are known, earlier history does not add information needed to predict the next state. A state should therefore contain all variables relevant to the future at the chosen time scale.

For a robot, this is already a modeling choice. Its physical state might include:

• robot pose, joint positions, and gripper state;
• object positions, orientations, velocities, and whether an object is grasped;
• relevant contact conditions;
• other agents or obstacles;
• hidden properties that matter, such as whether a container is full, an object is slippery, or a person intends to cross a robot’s path.

An MDP makes a strong information assumption: the agent gets this state. That is rarely realistic. A camera supplies pixels, not exact object poses. A force sensor reports a noisy signal, not a definitive statement that a grasp is stable. An occluded object may have no direct measurement at all.

A POMDP preserves the MDP’s state dynamics and objective but adds an observation process. The agent acts in the true state, but decides using its observations and memory of prior interaction.

The practical distinction is worth making sharply:

• A state is the complete task-relevant condition assumed by the environment model.
• An observation is the information actually delivered to the agent’s decision system.
• A state variable can be directly observed, partially observed, or unobserved.

For example, a robot’s measured joint encoder values may be nearly direct observations of its joint angles, while the exact pose of a partly hidden object remains uncertain. A POMDP can contain both kinds of variables. “Partially observable” does not mean that every part of the state is hidden.

---

The POMDP tuple

For a finite-horizon task, write a POMDP as

$$\mathcal{P}=⟨H,\mathcal{S},\mathcal{A},\Omega ,T,O,R,\gamma ,b_0⟩$$

Its components are:

Component	Meaning	Embodied-AI interpretation
$H$	Horizon	Maximum number of decision steps before the episode ends
$\mathcal{S}$	State space	Complete, task-relevant world and robot configurations
$\mathcal{A}$	Action space	Commands the robot can issue at the model’s chosen abstraction level
$\Omega$	Observation space	Possible sensor readings or processed perception outputs
$T(s^{\prime }\mid s,a)$	Transition model	Probability that action $a$ changes state $s$ into $s^{\prime }$
$O(o\mid s^{\prime },a)$	Observation model	Probability of receiving observation $o$ after reaching $s^{\prime }$
$R(s,a)$	Reward function	Immediate task value or cost of an action in a state
$\gamma$	Discount factor	Relative importance of future reward, where $0\le \gamma \le 1$
$b_0$	Initial belief	Distribution over possible initial states before sensing

The exact convention for the observation model varies. Some texts use $O(o\mid s^{\prime })$, while others include the previous action, $O(o\mid s^{\prime },a)$. Including the action is useful when sensing depends on what the robot did—for instance, moving a wrist camera, changing illumination, or probing an object with a gripper.

The reward function may also be written with next-state dependence, such as $R(s,a,s^{\prime })$. What matters is not the notation, but that the reward captures the task consequences that the agent should care about.

A POMDP is a problem formulation, not a commitment to a particular solution method. Defining a POMDP does not imply that an exact solver can handle the resulting model. Real robots often have continuous states, continuous actions, and high-dimensional images. Still, the formulation forces useful questions:

1. What physical facts determine what can happen next?
2. What does the robot actually get to measure?
3. How do its actions alter both the world and its later information?
4. What outcomes should it prefer?

Those questions remain useful whether the eventual controller is a planner, a learned policy, a model-predictive controller, or a hybrid system.

---

Worked formulation: searching before grasping

Consider a tabletop robot assigned to place a blue target block into a bin. Two opaque or partially occluding containers, left and right, are on the table. The target is in one of them. The robot can inspect a container with a wrist camera, move an occluding item, attempt a grasp, and place a grasped object in the bin.

This is deliberately a small, high-level model. Its aim is not to simulate every motor torque or camera pixel. Instead, it captures the uncertainty that changes the decision: where is the target, and did the robot successfully acquire it?

1. Define the state space

One possible state is

$$s=(l,q,c,g)$$

where:

• $l\in \{\text{left},\text{right}\}$ is the hidden target location;
• $q\in \{\text{clear},\text{occluded}\}$ represents whether the target can be seen in its current container;
• $c\in \{\text{left},\text{right},\text{home}\}$ is the robot camera or arm viewpoint;
• $g\in \{\text{empty},\text{target grasped},\text{wrong object grasped}\}$ is gripper status.

This state includes facts about the robot and environment. It need not include every physical variable in the room. It should include the variables that affect the consequences of available actions and the observations likely to follow.

The target location $l$ is hidden initially. The arm viewpoint $c$ may be directly inferable from robot encoders. Gripper status $g$ may be partially inferred from tactile sensors and finger position. The visibility condition $q$ is uncertain because an occluder can hide the target even when the camera is pointed at the correct side.

2. Define an action space at one coherent level

Suppose the high-level planner can choose

$$\mathcal{A}=\{\text{inspect left},\text{inspect right},\text{clear left},\text{clear right},\text{grasp left},\text{grasp right},\text{place in bin},\text{stop}\}$$

These are not raw joint torques. Each action could be implemented by a lower-level motion planner and feedback controller. At this level, “inspect left” means the robot executes a safe camera motion toward the left container and waits for the resulting visual estimate.

The abstraction is appropriate only if each listed action has reasonably predictable, modelable consequences. A vague action such as “solve the task” is too broad. Conversely, modeling every millisecond of motor torque may be unnecessary when the decision is simply which container to inspect first.

3. Define observations, not states

The camera does not report the true hidden location. It might return observations such as

$$\Omega =\{\text{target detected left},\text{target detected right},\text{no target visible},\text{ambiguous detection},\text{grasp contact},\text{no grasp contact},\text{target in bin}\}$$

In a real system, these observations might be outputs of perception and contact-estimation components rather than raw RGB images and force traces. This is a common and useful POMDP abstraction: a high-dimensional measurement is converted into a task-relevant, lower-dimensional observation.

The observation model captures sensor limitations. For example:

$$O(\text{target detected left}\mid l=\text{left},q=\text{clear},a=\text{inspect left})$$

should be high, but it need not equal one. Detection can fail due to poor lighting, segmentation error, or an unexpected object pose. The same observation should have low probability if the target is actually on the right, but false positives can make it nonzero.

4. Define state transitions

The transition model describes how actions physically affect the world:

$$T(s^{\prime }\mid s,a)=P(s_{t+1}=s^{\prime }\mid s_t=s,a_t=a)$$

Some actions leave most of the state unchanged. Inspecting a container may primarily change camera pose. Clearing an occluder may change $q$ from $\text{occluded}$ to $\text{clear}$, perhaps with a small probability of moving the target unintentionally. A grasp may change $g$, but not always successfully.

For example, a grasp action can fail even when the robot chooses the correct side:

$$T(g^{\prime }=\text{target grasped}\mid l=\text{left},g=\text{empty},a=\text{grasp left})=0.85$$

The remaining probability accounts for a failed approach, a slip, or a collision-avoidance abort. This uncertainty belongs in the transition model because it concerns the physical outcome of an action.

5. Define reward and episode completion

A minimal task reward might assign:

• a substantial positive reward for placing the target in the bin;
• a substantial negative reward for unsafe collision, dropping an object outside the workspace, or placing a wrong object;
• a small time cost for each action;
• a modest cost for clearing an occluder or performing a risky grasp.

The task ends when one of the following occurs:

• the target is correctly placed;
• an unsafe or irreversible failure occurs;
• the action horizon $H$ is reached;
• the robot explicitly stops after determining that it cannot proceed safely.

These outcomes can be represented either by terminal states or by an explicit horizon together with termination rules. The detailed shaping of reward and the risk of unintended incentives will be the focus of a later lesson; for now, the important point is that a POMDP defines what the agent should optimize as well as what it can observe.

6. Specify initial uncertainty

Before the robot looks inside either container, it may start with

$$b_0(l=\text{left})=0.5$$ $$b_0(l=\text{right})=0.5$$

If task setup usually places the target on the left, a different prior may be justified. A prior is not a guess inserted to make the mathematics work. It is an explicit representation of what the agent knows before new evidence arrives.

---

Belief state: acting without seeing the state

Because the true state is not available, a POMDP policy cannot generally be written as $a_t=\pi (s_t)$. It instead conditions on an action-observation history:

$$h_t=(o_0,a_0,o_1,a_1,\dots ,a_{t-1},o_t)$$

However, retaining the entire history becomes inconvenient as an episode grows. Under the POMDP assumptions, the history can be summarized by a belief state:

$$b_t(s)=P(s_t=s\mid h_t)$$

A belief is a probability distribution over possible true states. It represents uncertainty, not an alternative physical world. The robot is in one actual state; the belief encodes which states remain plausible from the agent’s perspective.

For the search task, perhaps the robot initially assigns equal probability to the target being left or right. It performs inspect left and receives target detected left. Suppose the detection model is:

$$P(\text{target detected left}\mid l=\text{left})=0.9$$ $$P(\text{target detected left}\mid l=\text{right})=0.1$$

With equal initial probabilities, the new belief in the left location is

$$P(l=\text{left}\mid \text{target detected left})=\frac{0.9\cdot 0.5}{0.9\cdot 0.5+0.1\cdot 0.5}=0.9$$

The robot should become much more confident that the target is on the left, but it should not treat this as certainty. Depending on the cost of a failed grasp, it may grasp immediately, make another inspection, or use an action that improves visibility.

Belief updating has two conceptual stages.

First, predict how the action may have changed the state:

$${\bar{b}}_{t+1}(s^{\prime })=\sum _{s\in \mathcal{S}}T(s^{\prime }\mid s,a_t)b_t(s)$$

Second, correct that prediction using the new observation:

$$b_{t+1}(s^{\prime })=\frac{O(o_{t+1}\mid s^{\prime },a_t){\bar{b}}_{t+1}(s^{\prime })}{{\sum }_{\tilde{s}\in \mathcal{S}}O(o_{t+1}\mid \tilde{s},a_t){\bar{b}}_{t+1}(\tilde{s})}$$

The denominator simply normalizes the distribution so that its probabilities sum to one.

This is where embodiment matters. Action does not only pursue a direct physical goal. It can improve the next observation and hence improve the belief. Inspecting, repositioning a camera, moving an occluder, or gently probing a surface can be valuable because it changes what the robot can infer.

The POMDP belief-tree progression shows this closed-loop structure. Starting from $b^1$, the agent selects either $a_1$ or $a_2$. Each action can produce $o_1$ or $o_2$, and each resulting observation produces a distinct belief at the next horizon, such as $b_{11}^2$ or $b_{12}^2$. The policy can then choose different next actions for those different beliefs.

A fixed open-loop plan such as “inspect left, grasp left, place” ignores this branching. A POMDP policy is conditional: it can inspect left, then either grasp, inspect again, clear an occlusion, or redirect attention to the right depending on what the robot actually observed.

---

How to tell whether a formulation is sound

When formulating a new embodied task, use the following discipline.

State must support prediction

Ask: if two situations have the same modeled state and the robot takes the same action, should their distributions over future states and observations be the same?

If not, the state omits a relevant variable. For example, a grasping model that includes object pose but omits whether an object is slippery may fail the Markov assumption if slipperiness changes grasp success and later tactile signals.

Observations must reflect what sensors provide

Do not write “true object pose” as the observation merely because it is convenient in a simulator. If the deployed robot estimates pose from a camera with occlusion and noise, the POMDP should represent a measurement or estimate with appropriate uncertainty.

Using privileged simulator state is sometimes acceptable for training a critic or diagnosing a system, but it is not a faithful observation model for a policy expected to operate from physical sensors.

Separate dynamics uncertainty from sensing uncertainty

A missed visual detection is usually an observation-model issue. An object slipping from the gripper is usually a transition-model issue. Both create uncertainty, but they have different causes and demand different remedies.

Put actions at the decision level you intend to control

At a task-planning level, an action might be “inspect left” or “grasp left.” At a low-level controller level, it might be a joint velocity or torque command. Both can be valid POMDP actions, but mixing the two levels casually makes models difficult to interpret and solve.

Make the initial belief explicit

A robot often begins with uncertainty about object locations, human intent, map layout, friction, or the state of its own sensors. Writing $b_0$ makes these assumptions inspectable rather than implicit.

---

Key takeaways

A POMDP formalizes embodied decision-making when the robot cannot directly observe the complete state relevant to its task. It extends an MDP with an observation space and an observation model.

A complete formulation specifies:

$$⟨H,\mathcal{S},\mathcal{A},\Omega ,T,O,R,\gamma ,b_0⟩$$

The state describes the complete task-relevant situation; the observation describes what the robot actually senses. The transition model represents uncertain physical consequences of action, while the observation model represents uncertain sensing.

Because observations rarely identify the true state, the agent maintains a belief distribution and chooses actions based on that belief. This allows genuinely closed-loop behavior, including information-gathering actions that improve future decisions.

Next, we will examine how a robot’s morphology—its sensors, joints, geometry, actuator limits, and contact surfaces—constrains what can be observed, what actions are possible, and which behaviors can realistically be learned.