observations, and the $W : S \rightarrow \Omega$ function that produces observations $x$ based on the unobservable state $s$ of the process. Two different states, requiring two different optimal actions, may produce the same observation. This makes POMDPs remarkably challenging for reinforcement learning algorithms, as memoryless policies, that select actions or options based only on the current observation, typically no longer suffice.

Finite State Controllers
------------------------

Finite State Controllers (FSCs) are commonly used in POMDPs. An FSC $\langle {\ensuremath{\mathcal{N}}}, \psi, \eta, \eta^0 \rangle$ is defined by a finite set [$\mathcal{N}$]{} of nodes, an action function $\psi (n_t, a_t) \in [0, 1]$ that maps nodes to a probability distribution over actions, a successor function $\eta (n_{t-1}, x_t, n_t) \in [0, 1]$ that maps nodes and observations to a probability distribution over next nodes, and an initial function $\eta^0 (x_1, n_1) \in [0, 1]$ that maps initial observations to nodes [@Meuleau1999].

At the first time-step, the agent observes $x_1$ and activates a node $n_1$ by sampling from $\eta^0 (x_1, \cdot)$. An action is performed by sampling from $\psi (n_1, \cdot)$. At each time-step $t$, a node $n_t$ is sampled from $\eta (n_{t-1}, x_t, \cdot)$, then an action $a_t$ is sampled from $\psi 