Inspired by the "Cognitive Hour-glass" model presented in <cit.>, we propose a new framework for developing cognitive architectures aimed at cognitive robotics. The purpose of the proposed framework is foremost to ease the development of cognitive architectures by encouraging and mitigating cooperation and re-use of existing results. This is done by proposing a framework dividing the development of cognitive architectures into a series of layers that can be considered partly in isolation, and some of which directly relate to other research fields. Finally, we give introductions to and review some topics essential to the proposed framework.

§ INTRODUCTION

Research in cognitive robotics originates from a need to perform and automate tasks in dynamic environments and in close or direct interaction with humans. Uncertainty about the environment and complexity of the tasks require robots with the ability to reason and plan while being reactive to changes in their environment. To achieve such behavior, robots cannot rely on predefined rules of behavior <cit.> and inspiration is taken from cognitive architectures.

Cognitive architectures provide a model for information processing that can capture robot functionalities. In combination with acquired sensory data, they can potentially generate intelligent autonomous behavior <cit.>. Cognitive architectures dates back to the 1950s <cit.> with a grand goal of implementing a full working cognitive system <cit.>. From this considerable challenge, an abundance of architectures have evolved, and a recent survey suggests that the number of existing architectures has reached several hundred <cit.>. Some are aimed towards robotics application, e.g., Robo-Soar <cit.>, CARACaS <cit.>, and RoboCog <cit.>. Unfortunately, most of these architectures take wildly different approaches to model cognition and are implemented in different programming languages. Furthermore, most of these architectures are constructed from a diverse set of specialized modules, e.g., special purpose speech, vision, and simultaneous localization and mapping modules, interacting with each other and have taken decades to develop <cit.>. Thereby making it non-trivial to expand upon, combine, and re-use parts of these architectures. Again, this makes it hard for other researchers and practitioners to contribute to or adopt the architectures for specific problems and applications. A unifying and standardized framework for developing new cognitive architectures aimed at cognitive robotics could potentially remedy these issues and ease the development of cognitive robotics.

In recent years a community consensus has begun to emerge about a standard model of humanlike minds, i.e., computational entities whose structures and processes are substantially similar to those found in human cognition <cit.>. While this "Standard Model of the Mind" spans key aspects of structure and processing, memory and content, learning, and perception and motor, it is agnostic to the best practice for modeling and implementing these things <cit.>.
In line with the idea proposed in <cit.>, we firmly believe that the evolution of the scientific field of cognitive robotics would benefit from anchoring new implementations around a common theoretical elegant base separating a specific model of a part of cognition from the algorithm that implements it. Furthermore, we believe that this theoretical base should allow new functionalities to evolve hierarchically just like software libraries build on top of each other. We believe so since this would allow the discussions and development to flourish at different levels of abstractions, and allow for better synergy with other research fields.

In <cit.> the authors present a Cognitive Hourglass model for their cognitive architecture called "Sigma" based on the following four desiderata:

* Grand Unification, spanning all of cognition,

* Generic Cognition, spanning both natural and artificial cognition,

* Functional Elegance, achieving generically cognitive grand unification with simplicity and theoretical elegance,

* Sufficient Efficiency, efficient enough to support the anticipated uses in real-time.

While Grand Unification and Sufficient Efficiency aligns perfectly with the needs of cognitive robotics, the need for Generic Cognition and Functional Elegance is subtle for cognitive robotics. Although the end goal of cognitive robotics might only be to develop functional artificial intelligence, building
upon something that potentially is also able to model natural intelligence would allow artificial intelligence to more easily benefit from insights obtained by the modeling of natural intelligence and vice versa. Similarly, Functional Elegance is not a goal of cognitive robotics per se. Still, it would allow researchers and practitioners working on different levels of cognition to obtain a common reference point and understanding at a basic level, thereby easing co-operation and re-use of results and innovative ideas.

In an attempt to obtain all four of these desiderata the authors of <cit.> place the so-called "graphical architecture" at the waist of their Cognitive Hourglass model as such a potential theoretical elegant base glueing everything together just like the Internet Protocol (IP) in the Internet-hourglass model <cit.>. Functional elegance is obtained by recognizing and developing general architectural fragments, and based on these defining idioms that can be re-used in modelling different parts of cognition. Having defined sufficiently general idioms, the authors of <cit.> hope to be able to develop full models of cognition from a limited set of such idioms and thereby achieve functional elegance, while at the same time achieving the three other desiderata.

With roots in the given desiderata, this Cognitive Hour-glass model in many ways could constitute a unifying and standardized framework for cognitive robotics. However, as we will elaborate on in <ref> the model commits to specific architectural decisions, which hinders the utilization of new technology and ideas. E.g., their commitment to the sum-product algorithm prevents the use of new algorithms for efficient probabilistic inference.
Therefore, based on the observation that the layers of the Cognitive Hour-glass model presented in <cit.> conceptually can be divided into more and generalized layers, in this paper, we propose a new generalized Cognitive Hour-glass model based on recent advancements within the machine learning community that makes no such commitments.
We believe that the presented generalized model constitutes an appropriate framework for guiding and discussing the future development of cognitive robotics.

In <ref> we briefly introduce the Cognitive Hour-glass model presented in <cit.> in more detail. Based on this, we present our generalized Cognitive Hour-glass model as a framework for developing new cognitive architectures aimed at cognitive robotics in <ref>. In <ref> we give a brief introduction to probabilistic programs since the presented framework is built around them. Explaining the functionality of probabilistic programs with conventional methods can be difficult, therefore in <ref> we present a new graphical representation of probabilistic programs which we call "generative flow graphs". We do so in the hope that it will ease the dissemination of new models of parts of cognition developed within the proposed framework. Being fundamental to achieving functional elegance within the proposed framework, we formally introduce the concept of probabilistic programming idioms in <ref> and explain how "generative flow graphs" can aid the identification of such idioms. In <ref> we discuss an intrinsic problem of performing approximate inference in complex probabilistic programs and present some modern algorithms to tackle this problem for cognitive robotics. Finally, as probabilistic programming languages form the foundation of the present framework, we give a brief survey of probabilistic programming languages relevant to the framework in <ref>.

§ SIGMA'S COGNITIVE HOUR-GLASS MODEL

<ref> illustrates how the dimensions of the model from <cit.> relate to the four desiderata. The top layer of the hour-glass represents all the knowledge and skills implemented by the cognitive system. This includes high level cognitive capabilities such as reasoning, decision making, and meta cognition, as well as low level cognitive capabilities such as perception, attention, and the formation of knowledge and memory that could potentially be inspired by human cognition. But it also includes artificial cognitive capabilities such as, e.g., the creation of a grid-maps common in robotics. As such, the extend of this layer corresponds to the achievable extend of Grand Unification and Generic Cognition.

The "cognitive architecture" layer defines central architectural decisions such as the utilization of the cognitive cycle and tri-level control structure for information processing, and the division of memory into a perceptual buffer, working memory and long-term memory. But also defines other architectural concepts such as "functions", "structures", "affect/emotion", "surprise, and attention. Thereby, the cognitive architecture induces what can be considered an "cognitive programming language" in which all of the knowledge and skills in the top layer can be embodied and learned. As an intermediate layer, cognitive idioms provide design patterns, libraries, and services that ease the implementation of knowledge and skills.

Below the Cognitive architecture and at the waist of the model, they have the graphical architecture constituting a small elegant core of functionality. Functional elegance is thereby obtained by compilation of knowledge and skills through a series of layers into a common representation in the graphical architecture. This graphical architecture primarily consists of probabilistic inference over graphical models, more specifically factor graphs, utilizing the sum-product algorithm <cit.> plus the following extensions:

* each variable node is allowed to correspond to one or more function variables,

* special purpose factor nodes,

* and the possibility of limiting the direction of influence along a link in the graph.

Of these extensions, the two first are merely special-purpose optimizations for the inference algorithm, i.e., a part of the implementation layer in <ref>. According to the authors of <cit.> the third extension has "a less clear status concerning factor graph semantics". Finally, the graphical architecture is implemented in the programming language LISP. In this model, sufficient efficiency is achieved as the cumulative efficiency of all layers. I.e., an efficient implementation in LISP is futile if models of knowledge and skills are inefficient for a given task.

To summarize, the model shown in <ref> commits to multiple more or less restrictive decisions such as the utilization of factor graphs and the sum-product algorithm at its core, the "cognitive cycle", the tri-level control structure, and LISP as the exclusive implementation language. While these commitments may be suitable for the specific cognitive architecture Sigma, they would hinder the exploration of new ideas and the utilization of new technologies, making this model unsuitable as a general framework.

§ GENERALIZED COGNITIVE HOUR-GLASS MODEL

Loose re-drawing of figures of the cognitive hour-glass model presented in <cit.>. Layers with dashed borders are not recognized as distinguishable layers by the authors of <cit.>.

Our proposal for a generalized cognitive hour-glass model. Dashed borders indicates layers that are not necessarily recognized as distinguishable layers, but could help in the development of the other layers.

While the cognitive hour-glass model presented in <cit.> has an advantageous structure with roots in highly appropriate desiderata, it is not suitable as a general framework, due to some exclusive structural commitments
We argue that these structural commitments are mostly artefacts of the limited expressibility of factor graphs and the sum-product algorithm.

Consider, for instance, the "cognitive cycle" dividing processing into an elaboration and adaption phase. The elaboration phase performs inference over the factor graph, while the adaption phase modifies the factor graph before further inference. We argue that the need for this two-phase division of processing is caused by the need for the sum-product algorithm to operate on a static factor graph. This cognitive cycle further makes the tri-level control structure necessary to make cognitive branching and recursion possible.
Similarly, we argue that the third extension of the factor graph semantics employed in <cit.> is nothing more than a simple control flow construct over the information flow in the graphical model and inference algorithm. It is straightforward to imagine how other control flow constructs such as recursion, loops, and conditionals could also be advantageous in modeling cognition.

Basically, we believe that special-purpose implementations of architectural constructs such as the two-phase "cognitive cycle" employed by Sigma and similar cognitive architectures have previously been necessary due to the limitations of the available modeling tools. The flexibility of probabilistic programs provided by the possibility of incorporating I/O operations, loops, branching, and recursion into a probabilistic model, should permit representing such constructs as either Probabilistic Programming or Cognitive Idioms instead.

In <ref> we present our proposal for a more general cognitive hour-glass model having probabilistic programs at its waist as the theoretical modeling base.
Just like the model in <ref>, our model is composed of a series of layers that expands away from the waist of the hour-glass. On top of the pure probabilistic program, we might be able to recognize program fragments that are sufficiently general to be considered idioms. From these idioms, it might be possible to construct dedicated programming languages for expressing cognitive behavior, knowledge, and skills, such as the "Cognitive Language" presented in <cit.>. In this framework, functional elegance above the probabilistic program is obtained via compilation of knowledge and skills through appropriate cognitive programming languages into probabilistic programs.
Below the probabilistic program's different inference algorithms can carry out the necessary inference in the probabilistic program. Different versions of these inference algorithms can potentially be implemented in different probabilistic languages. Both the probabilistic program and probabilistic programming language can be situated in standard deterministic programming languages. Furthermore, as expatiated on in <cit.> one needs not even use the same deterministic programming language for both. Finally, the deterministic programming languages allow us to execute a model of cognition on different types of hardware doing the actual computations.

We believe that this model is sufficiently general to be considered a framework for research in, and development of, cognitive robotics. In fact, as probabilistic programs can be considered an extension of deterministic programs, we believe that it should even be possible to situate both emergent, symbolic, and hybrid approaches to cognitive architectures in this framework, thereby covering the full taxonomy considered in<cit.>. In our framework constructs such as the cognitive cycle and tri-level control structure could potentially be expressed as probabilistic programming idioms rather than special-purpose architectural implementations. Similarly, incorporation of results from other research areas such as deep learning is only limited to the extent that a given probabilistic programming language and corresponding inference algorithms can incorporate essential tools used in these research areas, i.e., automatic differentiation for deep learning. Furthermore, this framework gives a satisfying view on the foundational hypothesis in artificial intelligence about substrate independence <cit.>, by cleanly separating the model of cognition, i.e. the probabilistic program and everything above it, from the organic or inorganic substrate that it exists on, i.e., everything below the probabilistic program.

While the above might sound promising, the choice of probabilistic programs as a focal point also has an important ramification. In general, we cannot guarantee the existence of an analytic solution for all models, and even if a solution exists it might be computationally intractable <cit.>. Therefore, we have to endure approximate solutions. Though this might sound restrictive, this is also the case for most other complex real-world problems. In fact, it can be considered a form of bounded rationality consistent with the concept of "satisficing" stating that an organism confronted with multiple goals does not have the senses nor the wits to infer an "optimal" or perfect solution, and thus will settle for the first solution permitting satisfaction at some specified level of all of its needs. <cit.>.

It is important to stress that the layers of the proposed framework are not independent. On the contrary, as the technological possibilities and community knowledge evolves, changes in one layer might open new possibilities in the layers above. Similarly, the need for new features in one layer might guide the research directions and development of the layers below. However, this structure is exactly what would allow further discussions in cognitive robotics to evolve at different levels of abstractions, and benefit from other research fields related to the layers below probabilistic programs.

§ PRELIMINARIES

In this paper, we do not distinguish between probability density functions and probability mass functions, and jointly denote them as probability functions. The symbol $\int $ is used to denote both integrals and summations depending on the context. In general, we use $z$ to denote latent random variables, $x$ to denote observed random variables, $p(...)$ to denote "true" probability functions, $q(...)$ to denote approximations to "true" probability functions, $\theta$ to denote parameters of "true" probability functions, $p(...)$, and $\phi$ to denote parameters of approximations to "true" probability functions, $q(...)$. When a probability function directly depends on a parameter we write the parameter in a subscript before the parenteses, e.g., $p_{\theta}(...)$ and $q_{\phi}(...)$. We use line over a value, parameter, or random variable to denote that it is equal to a specific value, e.g., $\overline{z} = 1,432$. We use a breve over a parameter or random variable to denote that it should be considered a fixed parameter or random variable within that equation, e.g., $\breve{\theta}$ or $\breve{z}$. For parameters this means that they attain a specific value, $\overline{\theta}$, i.e., $\breve{\theta}$ means that $\theta = \overline{\theta}$. For random variables it means that the probability functions that this variable is associated with is considered fixed within a given equation. We use capital letters to denote sets, e.g., $A=\left\{1,...,\overline{n}\right\}$. We use a superscript with curly brackets to denote indexes. E.g. $z^{\left\{i\right\}}$ would denote the i'th latent random variable. Similarly, we use a superscript with curly brackets and two numbers separated by a semicolon to denote a set of indexes values, i.e., $z^{\left\{1;\overline{n}\right\}} = z^{\left\{A\right\}} = \left\{ z^{\left\{1\right\}},..., z^{\left\{\overline{n}\right\}}\right\}$. We use a backslash, $\setminus$, after a set followed by a value, random variable, or parameter to denote the exclusion of that value, random variable, or parameter from that set, i.e., $z^{\left\{A\right\}}\setminus z^{\left\{\overline{n}\right\}} = \left\{ z^{\left\{1\right\}},..., z^{\left\{\overline{n}-1\right\}}\right\}$. We use capital $C$ to denote a collection of latent random variables, observed random variables, and parameters. Furthermore, we will specify such a collection by enclosing variables and parameters with curly brackets around and with a semi-colon separating latent random variables, observed random variables, and parameters in that order, e.g., $C=\left\{Z;X;\Theta\right\}$. We will use $\text{Pa}$, $\text{Ch}$, $\text{An}$, and $\text{De}$ as abbreviations for parent, child, ancestors, and descendants, respectively, and use, e.g., $\text{Pa}\Theta(C)$ to denote the set of parameters parent to the collection $C$, and $\text{Ch}X(Z)$ to denote the set of observed variables that are children of the latent random variable $Z$.

§ PROBABILISTIC PROGRAMS

At the heart of our framework, we have chosen to place probabilistic programs.
One definition of probabilistic programs is as follows:

“Probabilistic programs are usual functional or imperative programs with two added constructs: (1) the ability to draw values at random from distributions, and (2) the ability to condition values of variables in a program via observations.” <cit.>

With these two constructs, any functional or imperative program can be turned into a simultaneous representation of a joint distribution, $p_{\Theta}(Z,X)$, and conditional distribution, $p_{\Theta}(X|Z)$, where $X$ represent the conditioned/observed random variables, $Z$ the unconditioned/latent random variables, and $\Theta$ represents other parameters in the program that are not given a probabilistic treatment. Thereby allowing us to integrate classical control constructs familiar to any programmer such as if/else statement, loops, and recursions into probabilistic models. As such probabilistic programs can express exactly the same functionality as any deterministic programs can and even more. As elaborated on in <cit.>, these two constructs are usually provided as extensions to a given programming language through special sample and observe functions or keywords. Thus it would be natural to represent such probabilistic programs by pseudo-code. However, based on experiences it can be hard to follow the generative flow of random variables in such pseudo-code. Alternatively, such generative flows have classically been represented by directed graphical models <cit.>. Unfortunately, we have also found that the semantics of classical directed graphical models neither provide an appropriate presentation.

§.§ Generative Flow Graphs

We have found that combining the semantics of classical directed graphical models with the semantics of flowcharts into a hybrid representation is a good visual representation. Directed graphical models represent the conditional dependency structure of a model and flowcharts represent the steps in an algorithm or workflow. The hybrid representation illustrates the order in which samples of random variables in a probabilistic program are generated and how these samples influence the distributions used to generate other samples. For this reason, we denote this hybrid representation by the name Generative Flow Graph.

[Graph idiom for the classical simultaneous localization and mapping (SLAM) problem <cit.>. $z^{\left\{t\right\}}_s$ is the state at time $t$, $z^{\left\{t\right\}}_a$ is the action at time $t$, $z^{\left\{i\right\}}_{map}$ is the $i$'th pixel in a grid map, and $x^{\left\{t\right\}}_{p}$ is the perceived information at time $t$. ]

[Graph idiom for a Markov Decision Process <cit.>. $z^{\left\{t\right\}}_s$ is the state at time $t$, $z^{\left\{t\right\}}_a$ is the action at time $t$, and $x^{\left\{t\right\}}_{O}$ is a "observed" optimality variable at time $t$.

Examples of two directed graphical models developed in different research areas.

To exemplify the utility of the Generative Flow Graph representation consider the graphical model for a classical Markov Decision Process and the simultaneous localization and mapping (SLAM) problem depicted in <ref>. With the classical semantics of directed graphical models, it is often the case that size limitations of figures coerce authors to remove some variables from the figure and represent them indirectly by, e.g., dashed arrows as the case in both <ref> and <ref>. Similarly, the classical semantics of directed graphical models does not represent the influence from other parameters or variables that are not given a probabilistic treatment, even though such variables and parameters might have equal importance for a model. This is especially true if they are not fixed and have to be learned, e.g., if one wants to incorporate artificial neural networks into a model. The classical semantics of directed graphical models also cannot represent dependency structures depending on conditionals giving the illusion that a variable always depends on all of its possible parents, and that all variables in the graph are relevant in all situations. Furthermore, while the semantics of directed graphical models allows us to represent the structure of the joint distribution, $p(Z,X)$, its ability to explicitly express the structure of the posterior distribution, $p(Z|X)$, is limited. Finally, there is no standardized ways of representing a fragment of a graphical model, which hinders discussions at different levels of abstraction. Probabilistic programs easily allow us to incorporate the above in our models and thus a more appropriate representation is needed. The semantics of generative flow graphs shown in <ref> in <ref> alleviate these problems. Utilizing these semantics we can redraw the directed graphical model in <ref> in multiple ways with different levels of information as in <ref>. Notice, that the choice of node collections is not unique.

Three semantically equivalent generative flow graphs with different levels of abstractions corresponding to the directed graphical model in <ref>.

One advantage of the semantics of directed graphical models is that for graphs with no cycles <cit.> such models represents a specific factorization of the joint probability of all the random variables in the model of the form:

\begin{align}
&p( x^{\left\{1;\overline{n}\right\}} ,z^{\left\{1;\overline{m}\right\}}) \label{eq:DAG_fact}\\
&\quad=\prod _{n=1}^{\overline{n}} p( x^{\left\{n\right\}} |\text{Pa}Z( x^{\left\{n\right\}}))\prod _{m=1}^{\overline{m}} p( z^{\left\{m\right\}} |PaZ(z^{\left\{m\right\}})) \nonumber
\end{align}

where $x^{\left\{n\right\}}$ and $z^{\left\{m\right\}}$ are the n'th observed and the m'th latent random variable in the model, respectively. This is in principle also true for the generative flow graph representation if it neither contains any cycles, just with the additional explicit representation of dependency on parameters. For generative flow graphs, we can similarly to <ref> write up a factorization by including a factor of the form

\begin{align*}
p_{\text{Pa}\theta( z^{\left\{m\right\}})}( z^{\left\{m\right\}} |\text{Pa}Z(z^{\left\{m\right\}}))
\end{align*}

for each latent random variable node, $z^{\left\{m\right\}}$, in the graph, and a factor of the form

\begin{align*}
p_{\text{Pa}\theta( x^{\left\{n\right\}})}( x^{\left\{n\right\}} |\text{Pa}Z( x^{\left\{n\right\}}))
\end{align*}

for each observed random variable node, $x^{\left\{n\right\}}$, in the graph, and finally a factor of the form

\begin{align}
& p_{\Theta ,\text{Pa}\Theta(C^{\left\{k\right\}})} (Z,X|\text{Pa}Z(C^{\left\{k\right\}})) \label{eq:node_collection_factor}\\
& =p_{\Theta ,\text{Pa}\Theta(C^{\left\{k\right\}})} (X|Z,\text{Pa}Z(C^{\left\{k\right\}}))p_{\Theta ,\text{Pa}\Theta(C^{\left\{k\right\}})} (z|\text{Pa}Z(C^{\left\{k\right\}})) \nonumber
\end{align}

for each node collection, $C^{\left\{k\right\}} = \{Z;X;\Theta\}$. If a parent node of $y$ is a node collection $\{Z;X;\Theta\}$ then $\text{Pa}Z(y)=z$ and $\text{Pa}\theta(y)=\theta$ unless a subset of the variables or parameters in the node collection is explicitly specified next to the parent link. If the internal structure of a node collection is known from somewhere else, the factor in <ref> can of course be replaced by the corresponding factorization. The catch, however, is that a probabilistic program, and thus also generative flow graphs, potentially can denote models with an unbounded number of random variables and parameters making it impossible to write up the full factorization explicitly. On the other hand, this just emphasizes the need for alternative ways of representing probabilistic programs other than pseudo-code.

Two generative flow graphs representations of a simple model with two parameters, two latent variables, and two observed variables. In both graphs $\theta_1$ and $z_1$ are needed to generate $z_2$, however, in (b) we have explicitly constrained the inference of $z_2$ to not influence the learning of $\theta_1$ and inference of $z_1$. Thus, the evidence provided by $x_2$ neither is allowed to have an influence on $\theta_1$ and $z_1$. Thereby, the model represented by the nodes on the left-hand side of the dashed line in (b) can be seen as an independent problem.

Besides the possibility of expressing a factorization of the joint prior distribution, the detached link allows us to express additional structure for the posterior distribution, $p(z|x)$. Consider the two generative flow graphs in <ref>. By applying standard manipulations we can obtain the factorization in <ref> for the graph in <ref>.

\begin{align}
& p_{\theta _{a} ,\theta _{b}}( z_{a} ,z_{b} |x_{a} ,x_{b}) \nonumber\\
& \quad =p_{\theta _{a} ,\theta _{b}}( z_{b} |z_{a} ,x_{a} ,x_{b}) p_{\theta _{a} ,\theta _{b}}( z_{a} |x_{a} ,x_{b}) \nonumber\\
& \quad =p_{\theta _{a} ,\theta _{b}}( z_{b} |z_{a} ,x_{b}) p_{\theta _{a} ,\theta _{b}}( z_{a} |x_{a} ,x_{b}) \label{eq:detached_1}
\end{align}

wheres from the definition of the detached link we can write the factorization in <ref> for the graph in <ref>.

\begin{align}
& p_{\theta _{a} ,\theta _{b}}( z_{a} ,z_{b} |x_{a} ,x_{b}) \nonumber\\
& \quad =p_{\breve{\theta }_{a} ,\theta _{b}}( z_{b} |\breve{z}_{a} ,\breve{x}_{a} ,x_{b}) p_{\theta _{a}}( z_{a} |x_{a}) \nonumber\\
& \quad =p_{\breve{\theta }_{a} ,\theta _{b}}( z_{b} |\breve{z}_{a} ,x_{b}) p_{\theta _{a}}( z_{a} |x_{a}) \label{eq:detached_2}
\end{align}

The main difference between these two factorizations is the distribution over the latent variable $z_{a}$. In <ref> the distribution over the latent variable $z_{a}$ depends on the evidence provided by both observations $x_{a}$ and $x_{b}$, and is influenced by both parameters $\theta_{a}$ and $\theta_{b}$. In <ref> the distribution over $z_{a}$ depends only on the evidence provided by the observations $x_{a}$, and is only influenced by the parameter $\theta_{a}$. As such the inference problem of obtaining the posterior distribution over $z_a$ is independent of the inference problem of obtaining the posterior distribution over $z_b$, but not conversely. In general, for model consisting of $\overline{a}$ node collections, $C^{( a)} =\left\{Z^{( a)} ;X^{( a)} ;\Theta ^{( a)}\right\}$, connected only by detached links we can write the factorization of the posterior as

\begin{align}
p_{\Theta } (Z|X)=\prod _{a=1}^{\overline{a}} p_{\Theta ^{\{a\}} ,Pa\breve{\Theta }\left( C^{\{a\}}\right)}\left( Z^{\{a\}} |Pa\breve{Z}\left( C^{\{a\}}\right) ,X^{\{a\}}\right) \label{eq:collections_factorization}
\end{align}

the breves are used to emphasize that the variables and parameters are related through a detached link. The possible benefit of being able to express such structure will become clear in <ref>.

Another added benefit of the generative flow graph representation is to express models by different levels of abstraction. As an example consider the three different factorization of the simultaneous localisation and mapping problem given in <ref>, <ref>, and <ref>.

\begin{align}
& p\left( z_{s}^{\{0;t\}} ,z_{a}^{\{0;t-1\}} ,z_{p}^{\{1;t\}} ,z_{m}^{\{0;I\}}\right)\nonumber \\
& \quad =p\left( z_{p}^{\{1;t\}} |z_{s}^{\{0;t\}} ,z_{a}^{\{0;t-1\}} ,z_{m}^{\{0;I\}}\right)\label{eq:slam_fact_1}\\
& \quad\quad\quad \cdot p\left( z_{s}^{\{0;t\}} ,z_{a}^{\{0;t-1\}} |z_{s}^{\{0\}}\right) p\left( z_{s}^{\{0\}}\right)\prod _{i=1}^{I} p\left( z_{m}^{\{i\}}\right)\nonumber \\
& \quad =p\left( z_{s}^{\{0\}}\right)\prod _{i=1}^{I} p\left( z_{m}^{\{i\}}\right)\label{eq:slam_fact_2}\\
& \quad\quad \quad \cdot \prod _{\tau =1}^{t}\left(\begin{matrix*}[l]
p\left( z_{p}^{\{\tau \}} |z_{s}^{\{\tau \}} ,z_{a}^{\{\tau -1\}} ,z_{m}^{\{0;I\}}\right)\\
\quad \cdot p\left( z_{s}^{\{\tau \}} ,z_{a}^{\{\tau -1\}} |z_{s}^{\{\tau -1\}}\right)
\end{matrix*}\right)\nonumber  \\
& \quad =p\left( z_{s}^{\{0\}}\right)\prod _{i=1}^{I} p\left( z_{m}^{\{i\}}\right)\label{eq:slam_fact_3}\\
& \quad\quad \quad \cdot \prod _{\tau =1}^{t}\left(\begin{matrix*}[l]
p\left( z_{p}^{\{\tau \}} |z_{s}^{\{\tau \}} ,z_{a}^{\{\tau -1\}} ,z_{m}^{\{0;I\}}\right)\\
\quad \cdot p\left( z_{s}^{\{\tau \}} |z_{a}^{\{\tau -1\}} ,z_{s}^{\{\tau -1\}}\right) p\left( z_{a}^{\{\tau -1\}}\right)
\end{matrix*}\right)\nonumber
\end{align}

The generative flow graphs in <ref>, <ref>, and <ref>
corresponds directly to the factorization in <ref>, <ref>, and <ref>, respectively. Thereby, they represents different levels of abstractions for the same model. As such generative flow graphs simply yield better expressibility over their directed graphical model counterparts.

§.§ Probabilistic Programming Idioms

Two semantically equivalent generative flow graphs corresponding to the directed graphical model in <ref>.

We have already discussed how probabilistic programming idioms can be seen as a means to achieve functional elegance. In this section, we describe how such idioms can be discovered by inspecting generative flow graphs. We define probabilistic programming idioms as follows:

“Probabilistic programming idioms are reusable code fragments of probabilistic programs sharing an equivalent semantic role in their enclosing probabilistic programs.”

To identify such probabilistic programming idioms, we can look for node collections containing the same nodes and with the same internal structure in at least two different probabilistic programs. Consider for example the node collection $\left\{z_{s}^{\{\tau \}} ,z_{a}^{\{\tau -1\}} ;;\right\}$ highlighted with a green border in the generative flow graph for both the simultaneous localization and mapping problem and Markov Decision Process depicted in <ref> and <ref>, respectively. From <ref> and <ref> it is clear that the internal structure of this node collection is identical in both graphs, and that it represents the factorization

\begin{equation*}
p\left( z_{s}^{\{\tau \}} |z_{s}^{\{\tau -1\}} ,z_{a}^{\{\tau -1\}}\right) p\left( z_{a}^{\{\tau-1 \}}\right).
\end{equation*}

Assuming that the distributions $p\left( z_{s}^{\{\tau \}} |z_{s}^{\{\tau-1 \}} ,z_{a}^{\{\tau -1\}}\right)$ and $p\left( z_{a}^{\{\tau-1 \}}\right)$ are the same in both models, we could possible create a probabilistic program for this node collection once, and then reuse it in both models. This probabilistic program should then take a sample $z_{s}^{\{\tau-1 \}}$ as input. From this input the program could sample both $z_{s}^{\{\tau \}}$ and $z_{a}^{\{\tau-1 \}}$ from "hard-coded" distributions $p\left( z_{s}^{\{\tau \}} |z_{s}^{\{\tau-1 \}} ,z_{a}^{\{\tau-1 \}}\right)$ and $p\left( z_{a}^{\{\tau-1 \}}\right)$ using the sample function or keyword of the probabilistic programming language. Finally, the program should return both of these samples. While this approach might work perfectly for some applications, the two distributions $p\left( z_{s}^{\{\tau-1 \}}|z_{s}^{\{\tau-1 \}} ,z_{a}^{\{\tau -1\}}\right)$ and $p\left( z_{a}^{\{\tau -1\}}\right)$ are usually application specific, limiting the usability for an idiom in which they are "hard-coded". A far more general approach would be to allow the probabilistic program to instead take the two distributions as input or having these distributions as free variables, allowing us to re-use the code fragment even for problems where these distributions are not necessarily the same. Rather than fully defining a model of a part of cognition, such a probabilistic program would constitute a template method for the generative flow of that part of cognition. Specific utilization of the model could then be done via a function closure specifying the free distributions. While the benefits of the above example arguably might be limited since the internal structure of the node collection is relatively simple, it is not hard to imagine more complex structures. Consider for instance the node collection highlighted with a blue border in <ref>. By constructing an appropriate probabilistic program for this node collection we have defined a probabilistic programming idiom constituting the foundation for optimal control and reinforcement learning.

A combination of the generative flow graphs for the simultaneous localization and mapping problem and the Markov Decision Process shown in <ref> and <ref>, respectively, could potentially constitute an end-to-end navigation behavior for a mobile robot.

When we have developed such probabilistic programming idioms it empowers us to mix and match them to construct higher-level intelligence without worrying about all details of the underlying models.
E.g. <ref> implies that the output of a specific model for the simultaneous localization and mapping problem is used as the input to a Markov decision process, but leaves out details about their internal structures.

§ INFERENCE ALGORITHMS

As stated in <ref> a probabilistic program is a simultaneous representation of a joint distribution, $p_{\Theta}(Z,X)$, and a conditional distribution, $p_{\Theta}(X|Z)$. Having defined a model as such distributions we are usually interested in answering queries about the unconditioned/latent random variables, $Z$, given information about the conditioned/observed random variables, $X=\overline{X}$. In the combined navigation problem illustrated in <ref> we are interested in determining which action to take, $z_a^{\{\tau \}}$, given prior perceived information, $x_p^{\{\tau \}}$ for $\tau \in 1,...,t$, and future optimality variables, $x_O^{\{\tau \}}$ for $\tau \in t+1,...,T$.
Often queries of interest are statistics such as the posterior mean and variance of specific random variables, or the posterior probability of a random variable being within a given set. Still, it could also simply be to sample from the posterior, $p_{\Theta}\left(Z|X=\overline{X}\right)$. All of these queries are somehow related to the posterior distribution given by

\begin{align}
p_{\Theta}\left( { Z }|{ X=\overline { X }  } \right) &=\frac { p_{\Theta} \left( X=\overline { X } ,Z \right)  }{ { p_{\Theta} }\left( X=\overline { X }  \right)  } \nonumber \\
&=\frac { { p_{\Theta} }\left( X=\overline { X } ,Z \right)  }{ \int { { p_{\Theta} }\left( X=\overline { X } ,Z \right) dZ }  }. \label{eq:intractable}
\end{align}

Unfortunately, the marginalization by the integral in the denominator of <ref> in general does not have an analytical solution or is intractable to compute in most realistic problems and thereby the reason why one often has to resort to approximate inference <cit.>. Through time, an abundance of algorithms has been developed to find an approximation to the posterior in specific problems. Unfortunately, many of these algorithms cannot be applied to general probabilistic programs mainly due to the possible unbounded number of random variables <cit.>. Possible applicable inference algorithms can roughly be divided into Monte Carlo based algorithms such as Sequential Monte Carlo, Metropolis-Hastings, and Hamiltonian Monte Carlo, and the optimization based Variational Inference algorithms such as Stochastic Variational inference. As the size and complexity of models of cognition increase, the computational efficiency of inference algorithms becomes a paramount necessity to achieve sufficient efficiency of the framework presented in <ref>. While Monte Carlo methods often converge to the true posterior in the limit, convergence can be slow. Conversely, Variational Inference algorithms are often faster even though they can suffer from simplified posterior approximations <cit.>. Also, as Variational Inference methods are based on optimization they provide a natural synergy with data-driven discriminative techniques such as deep learning. By accepting that robots are not necessarily supposed to behave optimally, but rather should behave as agents with bounded rationality, the above characteristic makes Variational Inference algorithms an especially interesting choice for cognitive robotics. Therefore, <ref> is devoted to giving the reader an introduction to the overall concept of Variational Inference. <Ref> and <ref> present two specific solution approaches commonly used in variational inference, namely Message-passing algorithms and stochastic variational inference, respectively. Both approaches have their weaknesses. Therefore, in <ref> we outline a way of combining these two approaches to overcome the weaknesses.

§.§ Variational inference

Variational inference uses optimization to approximate one distribution, ${ p }\left(  Z   \right)$, by another simpler distribution, ${ q }\left( Z \right)$, called the variational distribution. Notice that, in general, ${ p }\left(  Z   \right)$ does not need to be a conditional distribution, ${ p }\left( { Z }|{ X=\overline { X }  } \right)$, as in <ref>. However, for the sake of the topic in this paper we will focus on the conditional distribution case. Thus, we will concentrate on solving a variational inference problem on the form

\begin{equation}
q^{ * }\left( Z \right) =\arg\min _{ q\left( Z \right) \in Q }{ { D }\left( { p_{\Theta} \left( { Z }|{ X=\overline { X }  } \right)  }||{ q\left( Z \right)  } \right)  } \label{eq:opt_problem}
\end{equation}

where $D$ is a so-called divergence measure, measuring the similarity between $p$ and $q$, and $Q$ is the family of variational distributions that we want to find our approximation from. The notation $D(p||q)$ denotes that we are dealing with a divergence measure and that the order of arguments, $p$ and $q$, matters.

The family of variational distributions, $Q$, is usually selected as a compromise between how good an approximation one wants and computational efficiency, and should be chosen such that the given queries can be easily answered. It is important to stress that any variational inference method is more or less biased via the choice of the family of variational distributions, $Q$. As a consequence we cannot view the original model in isolation, and have to consider the variational distribution, $q\left( Z \right) $, as an implicit part of the cognitive model. Furthermore, the divergence measure, $D$, can have a rather large impact on the approximation. However, empirical results suggest that for the family of $\alpha$-divergences, subsuming the commonly used Kullback–Leibler divergence, all choices will give similar results as long as the approximating family, $Q$, is a good fit to the true posterior distribution <cit.>.

§.§ Message-Passing

The overall idea behind message-passing algorithms is to take a possible complicated problem as defined by <ref> and break it down into a series of more tractable problems <cit.>.
This way of solving a variational inference problem is known as message-passing because the solution of each sub-problem can be interpreted as a message sent to the other sub-problems. This is done by assuming that the model of our problem, $p(Z|X)$, naturally factorizes into a product of probability distributions

\begin{align}
p(Z|X) & =\prod_{ a\in A }^{  } p^{\{ a\}}( Z|X).
\end{align}

Notice that the factorization need not be unique and that each probability distribution, $p^{\{ a\}}( Z|X)$, can depend on any number of the variables of $p(Z|X)$. Similarly, we can choose a variational distribution, $q\left( Z \right)$, that factorizes into a similar form

\begin{align}
q\left( Z \right) =\prod_{ a\in A }^{  }{ q^{ \left\{ a \right\}  }{ \left( Z \right)  } }. \label{eq:msg_passing_q_factor}
\end{align}

Now by defining the product of all other than the $a$'th factor of $q\left( Z \right)$ and $p(Z|X)$, respectively as

\begin{align}
q^{ \backslash a }\left( Z \right) &=\prod _{ b\in A\backslash a }^{  }{ q^{ \left\{ b \right\} }{ \left( Z \right)  } } ,\\
p^{ \backslash a }\left( Z|X \right) &=\prod _{ b\in A\backslash a }^{  }{ p^{ \left\{ b \right\}  }{ \left( Z|X \right)  } }
\end{align}

and by further assuming that $q^{ \backslash a }\left( Z \right) \approx p^{ \backslash a }\left( Z|X \right) $ is in fact a good approximation,
it is possible to rewrite our full problem in <ref> into a series of approximate sub-problems on the form

[1]
Initialize $q^{ \left\{ a \right\} * }{ \left( Z \right)  }$ for all $a\in A$
Pick a factor $a\in A$
Solve <ref> to find $q^{ \left\{ a \right\} * }{ \left( Z \right)  }$
$q^{ \left\{ a \right\} * }{ \left( Z \right)  }$ converges for all $a\in A$

The generic message-passing algorithm. The loop in line 2 can potentially be run in parallel and in a distributed fashion.

\begin{align}
& q^{ \left\{ a \right\} * }{ \left( Z \right)  }\approx  \label{eq:msg_optimization}\\
& \quad \arg\min _{ q^{ \left\{ a \right\}  \in Q^{ \left\{ a \right\} } } }{ D\left[ p^{ \left\{ a \right\}  }{ \left( Z|X  \right)  }{ q^{ \setminus a }\left( Z \right)  }  || q^{ \left\{ a \right\}  }{ \left( Z \right)  }{ q^{ \setminus a }\left( Z \right)  } \right] }  \nonumber
\end{align}

where $q^{ \backslash a }\left( Z \right)$ is kept fixed.
Assuming a sensible choice of factor families, $Q^{ \left\{ a \right\} }$, from which $q^{ \left\{ a \right\}  }$ can be chosen, the problem in <ref> can be more tractable than the original problem, and by iterating over these coupled sub-problems as shown in <ref>, we can obtain an approximate solution to our original problem. In principle, we can even use different divergence measures for each sub-problem to do mismatched message-passing, which could make some of the sub-problems easier to solve as described in <cit.>

The approach is not guaranteed to converge for general problems. Furthermore, <ref> might still be a hard problem to solve, and thus previously in practice, the approach has been limited to problems for which <ref> can be solved analytically such as fully discrete or Gaussian problems <cit.>. However, besides breaking the original problem into a series of more tractable sub-problems, this solution approach gives a principle way of solving the original problem in a distributed and parallel fashion, which could be a huge benefit for large models. Furthermore, depending on the dependency structure of the problem, not all sub-problems depend on the solution of the other sub-problems, which can significantly reduce the amount of communication needed due to sparsely connected networks.

§.§ Stochastic Variational inference

Stochastic Variational inference (SVI) reformulates the minimization problem of a variational inference problem, e.g., <ref> or <ref>, into a dual maximization problem with an objective, $L$, that is suited for stochastic optimization <cit.>. To use stochastic optimization we need to assume that the variational distribution, $q$, is parameterized by some parameters, $\Phi $. We will denote the parameterized variational distribution by, $q_{\Phi }$. The steps and assumptions to obtain this dual problem and the objective function, $L$, of the resulting maximization problem of course depends on whether we have chosen the Kullback–Leibler divergence <cit.> or $\alpha$-divergences <cit.>, however, the resulting maximization problem ends up being on the form

\begin{equation}
\Phi^{*} = \arg \max _{ \Phi }{ { \quad { \underbrace { L\left( { p_{ \Theta }\left( Z,X=\overline { X }  \right)  },{ { q }_{ \Phi  }\left( Z \right)  } \right)  }_{ { E }_{ Z\sim q_{ \Phi  }\left( Z \right)  }\left[ l\left( Z, \Theta, \Phi   \right)  \right]  }  } } }. \label{eq:opt_problem_param}
\end{equation}

This dual objective function, $L$, does not depend on the posterior, $p_{ \Theta }\left( { Z }|{ X=\overline { X }  } \right)$, but only the variational distribution, $q_{ \Phi  }\left( Z \right)$ and the unconditional distribution $p_{ \Theta }\left( Z,X=\overline { X }  \right)$ making the problem much easier to work with. Besides being dual to <ref> it turns out that for the family of alpha-divergences with $\alpha>0$, $L$ is also an lower bound on the log evidence, $log \left(p_\Theta\left(Z\right)\right)$ <cit.>. Since the log evidence is a measure of how well a model fits the data, we can instead consider the optimization problem <cit.>

\begin{equation}
\Theta^{*},\Phi^{*} = \arg \max _{ \Theta, \Phi }{ { \quad { \underbrace { L\left( { p_{ \Theta }\left( Z,X=\overline { X }  \right)  },{ { q }_{ \Phi  }\left( Z \right)  } \right)  }_{ { E }_{ Z\sim q_{ \Phi  }\left( Z \right)  }\left[ l\left( Z, \Theta, \Phi  \right)  \right]  }  } } }. \label{eq:opt_problem_param_2}
\end{equation}

that allows us to simultaneously fit the posterior approximation, $q_\Phi$, and model parameters, $\Theta$, to the data, $\overline{X}$. Furthermore, by utilizing the reparameterization trick <cit.> or the REINFORCE-gradient <cit.> it is possible to obtain an unbiased estimate of the gradient, $\overline{{ \nabla  }_{ W }L}$, of the dual objective $L$ where $W=\left\{\Theta, \Phi\right\}$. Stochastic gradient ascent can then be used to iteratively optimize the objective through the update equation

\begin{align}
{ W }^{ \{l\}  }={ W }^{ \{l-1\}  }+\rho ^{ \{l-1\}  }\overline{{ \nabla  }_{ W }L}^{ \{l\}  }\left( W^{ \{l-1\}  } \right)
\end{align}

where superscript $\{l\}$ is used to denote the $l$'th iteration. If the sequence of learning rates, $\rho ^{ \{l-1\} }$, follows the Robbins-Monro conditions,

\begin{align}
\sum _{ l=1 }^{ \infty  }{ \rho ^{ \{l\}  } } =\infty, \qquad \qquad
\sum _{ l=1 }^{ \infty  }{ { \left( \rho ^{ \{l\}  } \right)  }^{ 2 } } <\infty
\end{align}

then stochastic gradient ascent converges to a maximum of the objective function $L$, and since <ref> is dual to the original minimization problem it provides a solution to the original problem. An unbiased gradient estimator with low variance is pivotal for this method and variance reduction methods are often necessary. However, a discussion of this subject is outside the scope of this paper and can often be done automatically by probabilistic programming libraries/languages such as Pyro <cit.>.

One benefit of solving variational inference problems with stochastic optimization is that noisy gradient estimates are often relatively cheap to compute due to, e.g., subsampling of data. Another benefit is that the use of noisy gradient estimates can cause algorithms to escape shallow local optima of complex objective functions <cit.>. The downside of Stochastic variational inference is that it is inherently serial and that it requires the parameters to fit in the memory of a single processor <cit.>. This could potentially be a problem for cognitive robotics where large models with lots of variables and parameters presumably are necessary to obtain a high level of intelligence, and where queries have to be answered within different time scales. I.e. signals to motors have to be updated frequently while high-level decisions can be allowed to take a longer time.

§.§ Stochastic Message-Passing

To summarize the previous sections Message-passing algorithms exploits the dependency structure of a given variational inference problem to decompose the overall problem into a series of simpler variational inference sub-problems, that can be solved in a distributed fashion <cit.>. Message-passing algorithms do not give specific directions on how to solve these sub-problems, and thus classically required tedious analytical derivations, that effectively limited the usability of the method. On the other hand, modern Stochastic variational inference methods directly solve such variational inference problems utilizing stochastic optimization while simultaneously learning model parameters. By fusion of these two approaches, we could potentially overcome the serial nature of Stochastic variational inference to solve large-scale complex problems in a parallel and distributed fashion. However, to do so we need to find an appropriate factorization of a given problem. Again we can make use of the semantics of generative flow graphs. Assuming that we can divide all the nodes of a given generative flow graph into a set, $C^{\{A\}}$, of node collections $C^{\{a\}} =\left\{Z^{\{a\}} ;X^{\{a\}} ;\Theta ^{\{a\}}\right\}$ and a set of "global" observed variable nodes, $X_{G}$, having more than one node collection as parent, we can write the posterior factorization

\begin{align}
& p_{\Theta } (Z|X) \nonumber\\
& \quad=p_{\Theta } (Z^{\{A\}} |X^{\{A\}} ,X_{G} )\nonumber\\
& \quad=\frac{1}{p(X_{G} |X^{\{A\}} )} p_{\Theta } (Z^{\{A\}} ,X_{G} |X^{\{A\}} )\nonumber\\
& \quad=\frac{p(X_{G} |Z^{\{A\}} ,X^{\{A\}} )}{p_{\Theta } (X_{G} |X^{\{A\}} )} p_{\Theta } (Z^{\{A\}} |X^{\{A\}} )\nonumber\\
& \quad=\frac{p(X_{G} |Z^{\{A\}} )}{p_{\Theta } (X_{G} |X^{\{A\}} )} p_{\Theta } (Z^{\{A\}} |X^{\{A\}} )\label{eq:msg_factorization_2}\\
& \quad=\frac{p(X_{G} |Z^{\{A\}} )}{p_{\Theta } (X_{G} |X^{\{A\}} )}\label{eq:msg_factorization_1}\\
& \quad\quad\quad \cdot \prod _{a\in A} p_{\Theta ^{\{a\}} ,\text{Pa}\breve{\Theta }\left( C^{\{a\}}\right)}\left( Z^{\{a\}} |\text{Pa}\breve{Z}\left( C^{\{a\}}\right) ,X^{\{a\}}\right)\nonumber
\end{align}

where <ref> follows from conditional independence between $X_{G}$ and $X^{\{A\}}$ given $Z^{\{A\}}$, and <ref> follows from <ref>.
Following the procedure of message-passing we choose a variational distribution that factorizes as

\begin{align}
q_{\Phi}( Z) & =\prod _{a\in A} q_{\Phi ^{\{a\}}}\left( Z^{\{a\}}\right)\label{eq:S-msg_passing_q_factor}
\end{align}

Notice, that in <ref> we have exactly one factor for each node collection and that this factor only contains the latent variables of that node collection. This is unlike <ref> where a latent variable could be present in multiple factors. By combining <ref> and <ref> we can write an approximate posterior distribution related to the a'th node collection $ p( Z|X) \approx \tilde{p}^{\{a\}}( Z|X)$ where

\begin{align*}
& \tilde{p}_{\Theta ^{\{a\}}}^{\{a\}} (Z|X)=\\
& \quad p_{\Theta ^{\{a\}} ,\text{Pa}\breve{\Theta }\left( C^{\{a\}}\right)}\left( Z^{\{a\}} |\text{Pa}\breve{Z}\left( C^{\{a\}}\right) ,X^{\{a\}}\right)\\
& \quad \quad \cdot \frac{p(X_{G} |Z^{\{A\}} )}{\tilde{p}^{\{a\}} (X_{G} |X^{\{a\}} )}\prod _{b\in A\setminus a} q_{\breve{\Phi }^{\{b\}}}\left( Z^{\{b\}}\right)
\end{align*}

Where $\tilde{p}^{\{a\}} (X_{G} |X^{\{a\}} )$ is defined in <ref> in <ref>. Based on <ref> we can then define approximate sub-problems as

\begin{equation}
\underset{\Phi ^{\{a\}}}{\min} \ D\left[ q_{\Phi ^{\{a\}}}( Z) ||\tilde{p}_{\Theta ^{\{a\}}}^{\{a\}}( Z|X)\right] \label{eq:S-msg_passing_sub_problems}
\end{equation}

Each of these sub-problems can then be solved successively or in parallel potentially on distributed compute instances as outlined in <ref> and utilizing Stochastic variational inference as described in <ref>. To see how this choice of factorization affects the posterior approximations and the learning of model parameters, $\Theta$, consider the KL-divergence as divergence measure. Considering the KL-divergence we can rewrite the objective in <ref> as shown in  <ref> through  <ref> in <ref> to obtain the following local dual objective for stochastic variational inference

\begin{align}
& L_{KL}^{\{a\}}\left( \Theta ^{\{a\}} ,\Phi ^{\{a\}}\right) = \label{eq:s_kl_rewritten_2}\\
& \quad \quad E_{Z\sim \tilde{q}_{\text{Pa}\breve{Z}}^{\{a\}}}\left[ LogEvd_{X_{G} ,\ X^{\{a\}}}^{\{a\}}\left( \Theta ^{\{a\}}\right)\right] -C \nonumber\\
& \quad \quad \quad \quad \quad \quad \quad -D_{KL}\left[ q_{\Phi ^{\{a\}}} (Z)||\tilde{p}_{\Theta ^{\{a\}}}^{\{a\}} (Z|X)\right] \nonumber
\end{align}

Where $C$ is a constant with respect to $\Theta ^{\{a\}}$ and $\Phi ^{\{a\}}$, and $LogEvd^{\{a\}}\left( X_{G} ,\ X^{\{a\}}\right)$ is the joint log-evidence over global observed variables, $X_{G}$, and observed variables, $\ X^{\{a\}}$, local to the a'th node collection. Since the first term on the right-hand side is constant with respect to $\Phi ^{\{a\}}$, maximizing this local dual objective with respect to $\Phi ^{\{a\}}$ will minimize the KL-divergence. Furthermore, since $D_{KL}\left[ q_{\Phi ^{\{a\}}} (Z)||\tilde{p}_{\Theta ^{\{a\}}}^{\{a\}} (Z|X)\right] \geq 0$ by definition it follows from <ref> that

\begin{align*}
& E_{Z\sim \tilde{q}_{\text{Pa}\breve{Z}}^{\{a\}}}\left[ LogEvd^{\{a\}}\left( X_{G} ,\ X^{\{a\}}\right)\right] -C\\
& \quad \quad \quad \quad \quad \quad \quad \quad \quad \geq L_{KL}^{\{a\}}\left( \Theta ^{\{a\}} ,\Phi ^{\{a\}}\right)
\end{align*}

Therefore, by maximizing the local dual objective, $L_{KL}^{\{a\}}\left( \Theta ^{\{a\}} ,\Phi ^{\{a\}}\right)$, with respect to the local model parameters, $\Theta ^{\{a\}}$, we will push the expected joint log-evidence over the global, $X_{G}$, and the local, $X^{\{a\}}$, observed variables higher, where the expectation is taken with respect to the joint variational distribution over latent variables parent to the a'th node collection. In summary, this means that we can simultaneously fit our local model parameters, $\Theta ^{\{a\}}$, to the evidence and obtain an approximate local posterior distribution, $q_{\Phi ^{\{a\}}}( z^{\{a\}})$. While the above derivations were made for the KL-divergence, similar derivations can be done for the more general family of $\alpha$-divergences.

To evaluate this local dual objective, we only need information related to the local node collection, its parents, and other node collections having the same global observed variables as children. Thereby providing substantially computational speedups for generative flow graphs with sparsely connected node collections and global observed variables. To use this procedure with a standard probabilistic programming language, we would have to create a probabilistic program fragment for each node collection, their corresponding variational distribution, and the global observed variables. These fragments would then have to be composed together to form the local objectives potentially in an automated fashion.

So far within this section, we have assumed that all sub-problems are solved through a variational problem as in <ref>. However, there are in principle no reasons why we could not use estimates of sub-posteriors, $q( z^{\{b\}})$, obtained through other means in <ref>, as long as we can sample from these sub-posteriors. Thus, making the outlined method very flexible to combine with other methods, albeit analysis of the results obtained through the combined inference becomes more difficult. It is also important to stress that the factorization used above is not unique. It would be interesting to investigate if other factorizations could be employed, and for which problems these factorizations could be useful.

In summary, if we can divide a generative flow graph representing an overall model of cognition into node collections and global observed variables, then we can utilize the combination of Message-Passing and Stochastic Variational inference presented within this section, to distribute the computational burden of performing inference within this model. At the same time, we can learn local model parameters. Thus, yielding a very flexible tool allowing us to fully specify the part of a model that we are certain about, and potentially learn the rest.

§ PROBABILISTIC PROGRAMMING LANGUAGES

So far, our focus has been on the representation of models defined by probabilistic programs, and on how to answer queries related to these models via modern probabilistic inference. However, we have not considered how this is made possible by probabilistic programming languages and their relation to deterministic programming languages. Here we will not give a detailed introduction to probabilistic programming and refer interested readers to <cit.>, <cit.>, or <cit.>. Instead, we will give a short overview of languages relevant to modeling cognition.

As already mentioned in <ref> the main characteristics of a probabilistic program is a construct for sampling randomly from distributions and another construct for condition values of variables in the program. The purpose of probabilistic programming languages is to provide these two constructs and to handle the underlying machinery for implementing inference algorithms and performing inference from these constructs. As with any other programming language, design decisions are not universally applicable or desirable, and different trade-offs are purposefully made to achieve different goals. This fact combined with theoretical advancements has resulted in several different probabilistic programming languages. For an extensive list see <cit.>. Some of these are domain-specific aimed at performing inference in a restricted class of probabilistic programs, such as STAN <cit.>. These restrictions are usually employed to obtain more efficient inference. More interesting for the framework presented in <ref>, however, are languages self-identifying as universal or general-purpose, such as Pyro <cit.> and Venture <cit.>. These languages aim at performing inference in arbitrary probabilistic programs. Thus maximizing the flexibility for modeling cognition.

A recent trend has been to build probabilistic programming languages on top of deep-learning libraries such as PyTorch <cit.> and TensorFlow <cit.>. This is done both to use the efficient tensor math, automatic differentiation, and hardware acceleration that these libraries provide and to get tighter integration of deep-learning models within probabilistic models. Examples of such languages are Pyro <cit.> and ProbTorch <cit.> build upon PyTorch, and Edward <cit.> build upon TensorFlow. Again, when considering the use within the framework presented in <ref>, the languages based on PyTorch or TensorFlow 2.0 could potentially have the edge over others due to the dynamic approach to constructing computation graphs. This is because the dynamic computation graphs, more easily allow us to define dynamic models which include recursion and unbounded numbers of random choices <cit.>. Constructs potentially being indispensable for models of higher-level cognition supposed to evolve.

Python as the high-level general-purpose programming language it is makes modeling effortless in these languages. However, being based on python the computational efficiency of these languages is potentially limited by the need for interpretation. For this reason the relatively new project called NumPyro <cit.> is  under active development. NumPyro provides a backend to Pyro based on NumPy <cit.> and JAX <cit.> which enables just-in-time compilation, and thus potentially could provide much better computational efficiency, which again is essential for any practical robotic system.

To summarize, the choice of which probabilistic programming language to use depends on the end flexibility needed to model cognition. However, universal or general-purpose languages based on deep-learning libraries possibly with just-in-time compilation and hardware acceleration seem promising for general modeling of cognition, and especially for cognitive robotics.

§ CONCLUSION

Inspired by the work in <cit.>, we have outlined a new framework for developing cognitive architectures for cognitive robotics. With probabilistic programs at the center, this framework is sufficiently general to span the full spectrum of emergent, symbolic, and hybrid architectures. By dividing cognitive architectures into a series of layers this framework provides levels of abstractions between models of cognition and the algorithms that implement them on computational devices.
Some of these layers also directly relate to other fields of research, thereby encouraging better cooperation.

We also presented a novel graphical representation of probabilistic programs which we call generative flow graphs. We showed how such generative flow graphs can help identify important universal fragments of probabilistic programs and models. Fragments that could potentially be re-used in the development of other cognitive architectures. Thereby, again encouraging cooperation and easier re-use of existing results.

Furthermore, we introduced the problem of inference within probabilistic programs. We briefly reviewed possible approaches and argued that variational inference approaches seems interesting for cognitive robotics. We introduced two commonly used approaches called Message-Passing and Stochastic Variational Inference. We also outline the weaknesses of each approach and proposed a combined approach that we call Stochastic Message-Passing. The proposed approach provides a principle way of distributing the computational burden of inference and parameter learning.

Finally, we reviewed existing probabilistic programming languages providing the necessary machinery to implement inference algorithms for and perform inference in probabilistic programs.

The main topics within this paper have been the framework itself, the representation of cognitive models, and the computational burden.  These topics are indeed essential ingredients of the framework and pose interesting research directions by themselves.

§ DISCUSSION/FUTURE WORK

Symbol     Description     Meaning
0.6!          Circle without colored background            A node symbolizing a probabilistic variable, corresponding to a "sample" function or keyword in the probabilistic program. We denote this as a "Latent Variable Node".
0.6!       Circle with colored background         A node symbolizing an observed probabilistic variable, corresponding to a "observe" function or keyword in the probabilistic program. We denote this as an "Observed Variable Node".
0.6!          Square without colored background              A node symbolizing learn-able parameters in the probabilistic program. That is model parameters that can change at run-time. We denote this as a "Variable Parameter Node"
0.6!          Square with colored background           A node symbolizing fixed parameters in the probabilistic program. Use-full for representing parameters that cannot change at run-time such as tuning parameters. We denote this as a "Fixed parameter node"
0.8!      simple arrow        Link showing the generative path in a Probabilistic Program. The arrow can start in Latent Variable Nodes and Parameter nodes, and only point towards latent variable nodes and observed variable nodes. In large graphs or in cases where the origin of a link can be uncertain the readability can be improved by adding the name of the node from which the link originates next to the link. We denote this as a "Generative Link"
0.8!      Half circle on arrow         Symbolizing that operations downstream of this link in the probabilistic program should not influence nodes upstream to this link in the generative path. I.e. information such as accumulated gradients in a backward pass of automatic differentiation should not be propagated back through this link, corresponding to a ".detach()" and ".stop_gradient" call in PyTorch and TensorFlow, respectively. We denote this as a "Detached Link". A "variable parameter node" having only a detached link, can be considered a "fixed parameter node" or vice versa.
0.8!      Simple arrow pointing towards another simple arrow        Depending on the context, samples from different latent variable nodes may be used to generate the next sample. The generative link from the most recent sampled latent variable node in the generative path is the new active link. Used, e.g., to represent for loops.
1.0!          Polygon with rounded corners           Collection of nodes with internal dependency structure. The full structure of links between nodes can be shown inside the polygon. Alternatively, the names of the nodes can be written between curly brackets, $\{z;x;\theta\}$, with a semi-colon separating variables nodes, $z$, observed nodes, $x$, and parameter nodes, $\theta$, in that order. Finally, a node collection can also simply be defined somewhere else, $C = \{z;x;\theta\}$, and referred to by, e.g., a single letter, in which case we encourage the use of capital letters to emphasize the difference from the other types of nodes. In cases where one or more types of nodes are not present in a collection, both semi-colons should still be there. E.g., $\{;x_1,x_2;\}$ for a collection the two observed nodes $x_1$ and $x_2$. Such a collection of nodes directly corresponds to the factor $p_\theta(x_1,x_2|z)$ in a factorization of the joint distribution over all variables in a model. We denote this as a "Node Collection". A node is only allowed to be within one node collection unless it is within a node collection fully nested within another.
1.0!        Index specification in one corner of a polygon with rounded corners             Collection of nodes with indexed names. When the index is used to name nodes it is good practice to use the index in a superscript and encapsulate it in round brackets to emphasize that it is an index. We denote such a node collection an "Indexed Node Collection". The valid index should be clear from the context. E.g., in the case of a loop around the indexed collection, the index is incremented each time the loop enters the indexed node collection. When no such loops exist around the indexed node collection that could potentially cause ambiguity, it can be used instead of writing all the variables in a node collection with the curly brackets. For an example see <ref>
1.0!        Stacked polygons with rounded corners        Explicitly representation of multiple identical collections of nodes conditionally independent given their parents or simply independent if there are no parents. Here the one index is used for each of the independent collections.
0.8!      Dashed arrow         A link symbolizing an indirect relation between nodes. Such a link is non-generative, meaning that it is not directly used as a parameter in the generation of other samples, but only influences the generative path of other samples. We denote this as an "Influence Link"
0.95!        A Polygon with generative links connected at vertices, influence links towards the polygon connected at edges, and conditional values nearby generative links pointing away from the polygon       Node representing a condition changing the "direction" of the generative flow in a probabilistic program. We denote this as "conditioned generative branching".
0.95!         A Polygon with generative links connected at vertices, influence links towards the polygon connected at edges, and conditional values nearby generative links pointing towards the polygon            Node representing a condition selecting one out of two or more possible generative flows from parent nodes. We denote this as "conditioned generative selection".

Semantics of the Generative Flow Graph representation of probabilistic programs.

§.§ Rewriting KL-divergence for Stochastic Message-Passing

In this section, we will derive the dual objective for the combination of message-passing and stochastic variational inference presented in <ref>. Start by considering

\begin{align*}
p_{\Theta } (X_{G} |X^{\{A\}} ) & =\int p_{\Theta }( X_{G} ,Z^{\{A\}} |X^{\{A\}} )dZ\\
& =\int p(X_{G} |Z^{\{A\}} ,X^{\{A\}} )p_{\Theta } (Z^{\{A\}} |X^{\{A\}} )dZ\\
& =\int p(X_{G} |Z^{\{A\}} )p_{\Theta } (Z^{\{A\}} |X^{\{A\}} )dZ\\
& =\int p(X_{G} |Z^{\{A\}} )\prod _{a\in A} p_{\Theta ^{\{a\}} ,\text{Pa}\breve{\Theta }\left( C^{\{a\}}\right)}\left( Z^{\{a\}} |\text{Pa}\breve{Z}\left( C^{\{a\}}\right) ,X^{\{a\}}\right) dZ
\end{align*}

By replacing $p_{\Theta ^{\{b\}} ,\text{Pa}\breve{\Theta }\left( C^{\{b\}}\right)}\left( Z^{\{b\}} |\text{Pa}\breve{Z}\left( C^{\{b\}}\right) ,X^{\{b\}}\right)$ with their corresponding variational distributions $q_{\breve{\Phi }^{\{b\}}}\left( Z^{\{b\}}\right)$ for $b\in A\setminus a$ we obtain

\begin{align}
\tilde{p}^{\{a\}} (X_{G} |X^{\{a\}} )= & \int p(X_{G} |Z^{\{A\}} )p_{\Theta ^{\{a\}} ,\text{Pa}\breve{\Theta }\left( C^{\{a\}}\right)}\left( Z^{\{a\}} |\text{Pa}\breve{Z}\left( C^{\{a\}}\right) ,X^{\{a\}}\right)\prod _{b\in A\setminus a} q_{\breve{\Phi }^{\{b\}}}\left( Z^{\{b\}}\right) dZ \label{eq:marginal_likelyhood_approx}
\end{align}

where we have used $\tilde{p}^{\{a\}} (X_{G} |X^{\{a\}} )$ instead of $\tilde{p}^{\{a\}} (X_{G} |X^{\{A\}} )$ to emphasize the conditional independence between $X_{G}$ and $X^{\{b\}}$ given $ Z^{\{b\}}$ for $ b\in A\setminus a$ implicitly assumed by the approximation. Furthermore, also notice that we can rewrite the distribution, $p(X_{G} |Z^{\{A\}} )$, as follows

\begin{align*}
p(X_{G} |Z^{\{A\}} ) & =p(\text{Ch} X_{G}\left( Z^{\{a\}}\right) |Z^{\{A\}} )p(X_{G} \setminus \text{Ch} X_{G}\left( Z^{\{a\}}\right) |Z^{\{A\}} )\\
& =p\left(\text{Ch} X_{G}\left( Z^{\{a\}}\right) |\text{Pa} Z\left(\text{Ch} X_{G}\left( Z^{\{a\}}\right)\right)\right) p\left( X_{G} \setminus \text{Ch} X_{G}\left( Z^{\{a\}}\right) |Z^{\{A\}} \setminus Z^{\{a\}}\right)
\end{align*}

by separating the global observed variables, $X_G$, into those who are direct children of $Z^{\{a\}}$, and those who are not.
With the definition above and the ones given in <ref>, we can rewrite the KL-divergence as follows:

\begin{align}
& D_{KL}\left[ q_{\Phi ^{\{a\}}} (Z)||\tilde{p}_{\Theta ^{\{a\}}}^{\{a\}} (Z|X)\right] \label{eq:msg_svi_kl_rewritten1}\\
& \quad =\int _{Z} q_{\Phi ^{\{a\}}} (Z)\log\left(\frac{q_{\Phi ^{\{a\}}} (Z)}{\tilde{p}_{\Theta ^{\{a\}}}^{\{a\}} (Z|X)}\right) dZ\\
& \quad =\int _{Z} q_{\Phi ^{\{a\}}} (Z)\log\left(\frac{q_{\Phi ^{\{a\}}}\left( Z^{\{a\}}\right)\prod _{b\in A\setminus a} q_{\breve{\Phi }^{\{b\}}}\left( Z^{\{b\}}\right)}{\frac{p(X_{G} |Z^{\{A\}} )}{\tilde{p}^{\{a\}} (X_{G} |X^{\{a\}} )} p_{\Theta ^{\{a\}} ,\text{Pa}\breve{\Theta }\left( C^{\{a\}}\right)}\left( Z^{\{a\}} |\text{Pa}\breve{Z}\left( C^{\{a\}}\right) ,X^{\{a\}}\right)\prod _{b\in A\setminus a} q_{\breve{\Phi }^{\{b\}}}\left( Z^{\{b\}}\right)}\right) dZ\\
& \quad =\int _{Z} q_{\Phi ^{\{a\}}} (Z)\log\left(\frac{q_{\Phi ^{\{a\}}}\left( Z^{\{a\}}\right)}{\frac{p(X_{G} |Z^{\{A\}} )}{\tilde{p}^{\{a\}} (X_{G} |X^{\{a\}} )} p_{\Theta ^{\{a\}} ,\text{Pa}\breve{\Theta }\left( C^{\{a\}}\right)}\left( Z^{\{a\}} |\text{Pa}\breve{Z}\left( C^{\{a\}}\right) ,X^{\{a\}}\right)}\right) dZ\\
& \quad =\int _{Z} q_{\Phi ^{\{a\}}} (Z)\log\left(\frac{q_{\Phi ^{\{a\}}}\left( Z^{\{a\}}\right)}{\frac{p(X_{G} |Z^{\{A\}} )}{\tilde{p}^{\{a\}} (X_{G} |X^{\{a\}} )}\frac{p_{\Theta ^{\{a\}} ,\text{Pa}\breve{\Theta }\left( C^{\{a\}}\right)}\left( Z^{\{a\}} ,X^{\{a\}} |\text{Pa}\breve{Z}\left( C^{\{a\}}\right)\right)}{p_{\Theta ^{\{a\}} ,\text{Pa}\breve{\Theta }\left( C^{\{a\}}\right)}\left( X^{\{a\}} |\text{Pa}\breve{Z}\left( C^{\{a\}}\right)\right)}}\right) dZ\\
& \quad =\int _{Z} q_{\Phi ^{\{a\}}} (Z)\log\left(\frac{q_{\Phi ^{\{a\}}}\left( Z^{\{a\}}\right)}{\frac{p\left(\text{Ch} X_{G}\left( Z^{\{a\}}\right) |\text{Pa} Z\left(\text{Ch} X_{G}\left( Z^{\{a\}}\right)\right)\right) p\left( X_{G} \setminus \text{Ch} X_{G}\left( Z^{\{a\}}\right) |Z^{\{A\}} \setminus Z^{\{a\}}\right)}{\tilde{p}^{\{a\}} (X_{G} |X^{\{a\}} )}\frac{p_{\Theta ^{\{a\}} ,\text{Pa}\breve{\Theta }\left( C^{\{a\}}\right)}\left( Z^{\{a\}} ,X^{\{a\}} |\text{Pa}\breve{Z}\left( C^{\{a\}}\right)\right)}{p_{\Theta ^{\{a\}} ,\text{Pa}\breve{\Theta }\left( C^{\{a\}}\right)}\left( X^{\{a\}} |\text{Pa}\breve{Z}\left( C^{\{a\}}\right)\right)}}\right) dZ\\
& \quad =\int _{Z} q_{\Phi ^{\{a\}}} (Z)\log\left(\frac{q_{\Phi ^{\{a\}}}\left( Z^{\{a\}}\right)}{p\left(\text{Ch} X_{G}\left( Z^{\{a\}}\right) |\text{Pa} Z\left(\text{Ch} X_{G}\left( Z^{\{a\}}\right)\right)\right) p_{\Theta ^{\{a\}} ,\text{Pa}\breve{\Theta }\left( C^{\{a\}}\right)}\left( Z^{\{a\}} ,X^{\{a\}} |\text{Pa}\breve{Z}\left( C^{\{a\}}\right)\right)}\right) dZ \nonumber\\
& \quad \quad \quad \quad +\int _{Z} q_{\Phi ^{\{a\}}} (Z)\log\left(\tilde{p}^{\{a\}} (X_{G} |X^{\{a\}} )p_{\Theta ^{\{a\}} ,\text{Pa}\breve{\Theta }\left( C^{\{a\}}\right)}\left( X^{\{a\}} |\text{Pa}\breve{Z}\left( C^{\{a\}}\right)\right)\right) dZ\\
& \quad \quad \quad \quad -\int _{Z} q_{\Phi ^{\{a\}}} (Z)\log\left( p\left( X_{G} \setminus \text{Ch} X_{G}\left( Z^{\{a\}}\right) |Z^{\{A\}} \setminus Z^{\{a\}}\right)\right) dZ \nonumber\\
& \quad =\underbrace{E_{Z\sim \tilde{q}_{\Phi ^{\{a\}}}^{\{a\}}}\left[\log\left(\frac{q_{\Phi ^{\{a\}}}\left( Z^{\{a\}}\right)}{p\left(\text{Ch} X_{G}\left( Z^{\{a\}}\right) |\text{Pa} Z\left(\text{Ch} X_{G}\left( Z^{\{a\}}\right)\right)\right) p_{\Theta ^{\{a\}} ,\text{Pa}\breve{\Theta }\left( C^{\{a\}}\right)}\left( Z^{\{a\}} ,X^{\{a\}} |\text{Pa}\breve{Z}\left( C^{\{a\}}\right)\right)}\right)\right]}_{-L_{KL}^{\{a\}}\left( \Theta ^{\{a\}} ,\Phi ^{\{a\}}\right)} \nonumber\\
& \quad \quad \quad \quad +E_{Z\sim \tilde{q}_{\text{Pa}\breve{Z}}^{\{a\}}}\left[\underbrace{\log\left(\tilde{p}^{\{a\}} (X_{G} |X^{\{a\}} )p_{\Theta ^{\{a\}} ,\text{Pa}\breve{\Theta }\left( C^{\{a\}}\right)}\left( X^{\{a\}} |\text{Pa}\breve{Z}\left( C^{\{a\}}\right)\right)\right)}_{LogEvd_{X_{G} ,\ X^{\{a\}}}^{\{a\}}\left( \Theta ^{\{a\}}\right)}\right]  \label{eq:msg_svi_kl_rewritten2}\\
& \quad \quad \quad \quad -\underbrace{E_{Z\sim \prod _{b\in A\setminus a} q_{\breve{\Phi }^{\{b\}}}\left( Z^{\{b\}}\right)}\left[\log\left( p\left( X_{G} \setminus \text{Ch} X_{G}\left( Z^{\{a\}}\right) |Z^{\{A\}} \setminus Z^{\{a\}}\right)\right)\right]}_{C\ } \nonumber
\end{align}

\begin{align*}
\tilde{q}_{\Phi ^{\{a\}}}^{\{a\}} & =q_{\Phi ^{\{a\}}}\left( Z^{\{a\}}\right)\prod _{Z^{\{b\}} \in \text{Pa}\breve{Z}\left( C^{\{a\}}\right) \cup \text{Pa}\breve{Z}\left(\text{Ch} X_{G}\left( Z^{\{a\}}\right)\right) \setminus Z^{\{a\}}} q_{\breve{\Phi }^{\{b\}}}\left( Z^{\{b\}}\right)
\end{align*}

is the joint variational distribution over the latent variables, $Z^{\{a\}}$, local to the a'th node collection, and the latent variables parent to the a'th node collection, $\text{Pa}\breve{Z}\left( C^{\{a\}}\right)$, or having the same child global observed variables as the a'th node collection, $Z^{\{b\}} \in \text{Pa}\breve{Z}\left(\text{Ch} X_{G}\left( Z^{\{a\}}\right)\right) \setminus Z^{\{a\}}$. Furthermore,

\begin{align*}
\tilde{q}_{\text{Pa}\breve{Z}}^{\{a\}} & =\prod _{Z^{\{b\}} \in \text{Pa}\breve{Z}\left( C^{\{a\}}\right)} q_{\breve{\Phi }^{\{b\}}}\left( Z^{\{b\}}\right)
\end{align*}

is the joint variational distribution over the latent variables parent to the a'th node collection, $\text{Pa}\breve{Z}\left( C^{\{a\}}\right)$. Finally, $LogEvd_{X_{G} ,\ X^{\{a\}}}^{\{a\}}\left( \Theta ^{\{a\}}\right)$ denotes the joint log-evidence for $X_{G}$ and $ X^{\{a\}}$, and $C$ is constant with respect to $ \Theta ^{\{a\}} ,\Phi ^{\{a\}} \ $. By simple rearranging terms we obtain the dual objective

\begin{align*}
L_{KL}^{\{a\}}\left( \Theta ^{\{a\}} ,\Phi ^{\{a\}}\right) = & E_{Z\sim \tilde{q}_{\text{Pa}\breve{Z}}^{\{a\}}}\left[ LogEvd_{X_{G} ,\ X^{\{a\}}}^{\{a\}}\left( \Theta ^{\{a\}}\right)\right] -D_{KL}\left[ q_{\Phi ^{\{a\}}} (Z)||\tilde{p}_{\Theta ^{\{a\}}}^{\{a\}} (Z|X)\right] -C
\end{align*}