# Probabilistic Programming 2: Message Passing & Analytical Solutions

#### Goal 
  - Understand when and how analytical solutions to Bayesian inference can be obtained.
  - Understand how to perform message passing in a Forney-style factor graph.

#### Materials        
  - Mandatory
    - This notebook
    - Lecture notes on factor graphs
    - Lecture notes on continuous data
    - Lecture notes on discrete data
  - Optional
    - Chapters 2 and 3 of [Model-Based Machine Learning](http://www.mbmlbook.com/LearningSkills.html).
    - [Differences between Julia and Matlab / Python](https://docs.julialang.org/en/v1/manual/noteworthy-differences/index.html).

Note that none of the material below is new. The point of the Probabilistic Programming sessions is to solve practical problems so that the concepts from Bert's lectures become less abstract.


```julia
using Pkg
Pkg.activate("./workspace/")
Pkg.instantiate();
IJulia.clear_output();
```


```julia
using LinearAlgebra
using SpecialFunctions
using ForneyLab
using PyCall
using Plots
pyplot();
```

We'll be using the toolbox [ForneyLab.jl](https://github.com/biaslab/ForneyLab.jl) to visualize factor graphs and compute messages passed within the graph.

## Problem: A Job Interview

After you finish your master's degree, you will need to start looking for jobs. You will get one or more job interviews and some will be fun while others will be frustrating. The company you applied at wants a talented and skilled employee, but measuring a person's skill is tricky. Even a highly-skilled person makes mistakes and people with few skills can get lucky. In this session, we will look at various ways to assess skills using questions and test assignments. Along the way, you will gain experience with message passing, factor graphs and working with discrete vs continuous data.

### 1: Right or wrong

Suppose you head to a job interview for a machine learning engineer position. The company is interested in someone who knows Julia and has set up a test with syntax questions. We will first look at a single question, which we treat as an outcome variable $X_1$. You can either get this question right or wrong, which means we're dealing with a Bernoulli likelihood. The company assumes you have a skill level, denoted $\theta$, and the higher the skill, the more likely you are to get the question right. Since the company doesn't know anything about you, they chose an uninformative prior distribution: the Beta(1,1). We can write the generative model for answering this question as follows:

$$\begin{align*}
p(X_1, \theta) =&\ p(X_1 \mid \theta) \cdot p(\theta) \\
=&\ \text{Bernoulli}(X_1 \mid \theta) \cdot \text{Beta}(\theta \mid \alpha = 1, \beta=1) \, .
\end{align*}$$

The factor graph for this model is:

<!--  -->


where $f_b(X_1, \theta) \triangleq \text{Bernoulli}(X_1 \mid \theta)$ and $f_a(\theta) \triangleq \text{Beta}(\theta \mid 1,1)$. We are now going to construct this factor graph using the toolbox ForneyLab.


```julia
# Start building a model by setting up a FactorGraph structure
factor_graph1 = FactorGraph()

# Add the prior over 
@RV θ ~ Beta(1.0, 1.0, id=:f_a)

# Add the question correctness likelihood
@RV X1 ~ Bernoulli(θ, id=:f_b)

# The outcome X1 is going to be observed, so we set up a placeholder for the data entry
placeholder(X1, :X1)

# Visualize the graph
ForneyLab.draw(factor_graph1)
```


<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<!DOCTYPE svg PUBLIC "-//W3C//DTD SVG 1.1//EN"
 "http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd">
<!-- Generated by graphviz version 2.40.1 (20161225.0304)
 -->
<!-- Title: G Pages: 1 -->
<svg width="231pt" height="395pt"
 viewBox="0.00 0.00 231.00 395.00" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink">
<g id="graph0" class="graph" transform="scale(1 1) rotate(0) translate(4 391)">
<title>G</title>
<polygon fill="#ffffff" stroke="transparent" points="-4,4 -4,-391 227,-391 227,4 -4,4"/>
<!-- 12298408599198300 -->
<g id="node1" class="node">
<title>12298408599198300</title>
<polygon fill="#d3d3d3" stroke="#000000" points="147.5,-387 74.5,-387 74.5,-333 147.5,-333 147.5,-387"/>
<text text-anchor="middle" x="111" y="-357.8" font-family="Times,serif" font-size="9.00" fill="#000000">placeholder_X1</text>
</g>
<!-- 6701433761736062521 -->
<g id="node5" class="node">
<title>6701433761736062521</title>
<polygon fill="none" stroke="#000000" points="147,-288 75,-288 75,-216 147,-216 147,-288"/>
<text text-anchor="middle" x="111" y="-254.8" font-family="Times,serif" font-size="9.00" fill="#000000">Ber</text>
<text text-anchor="middle" x="111" y="-244.8" font-family="Times,serif" font-size="9.00" fill="#000000">f_b</text>
</g>
<!-- 12298408599198300&#45;&#45;6701433761736062521 -->
<g id="edge3" class="edge">
<title>12298408599198300&#45;&#45;6701433761736062521</title>
<path fill="none" stroke="#000000" d="M111,-332.747C111,-319.2495 111,-302.7693 111,-288.2253"/>
<text text-anchor="start" x="111" y="-308.6" font-family="Times,serif" font-size="8.00" fill="#ff0000">X1</text>
<text text-anchor="start" x="92" y="-290.8253" font-family="Times,serif" font-size="8.00" fill="#000000">1 out </text>
<text text-anchor="start" x="92" y="-326.347" font-family="Times,serif" font-size="8.00" fill="#000000">1 out </text>
</g>
<!-- 18163720509575022497 -->
<g id="node2" class="node">
<title>18163720509575022497</title>
<polygon fill="none" stroke="#000000" points="147,-171 75,-171 75,-99 147,-99 147,-171"/>
<text text-anchor="middle" x="111" y="-137.8" font-family="Times,serif" font-size="9.00" fill="#000000">Beta</text>
<text text-anchor="middle" x="111" y="-127.8" font-family="Times,serif" font-size="9.00" fill="#000000">f_a</text>
</g>
<!-- 8221999983495897590 -->
<g id="node3" class="node">
<title>8221999983495897590</title>
<polygon fill="#d3d3d3" stroke="#000000" points="54,-54 0,-54 0,0 54,0 54,-54"/>
<text text-anchor="middle" x="27" y="-24.8" font-family="Times,serif" font-size="9.00" fill="#000000">clamp_2</text>
</g>
<!-- 18163720509575022497&#45;&#45;8221999983495897590 -->
<g id="edge4" class="edge">
<title>18163720509575022497&#45;&#45;8221999983495897590</title>
<path fill="none" stroke="#000000" d="M74.865,-122.2014C56.4168,-113.6125 35.6629,-100.2859 25,-81 20.5515,-72.9541 19.802,-63.1548 20.5713,-54.1238"/>
<text text-anchor="start" x="25" y="-74.6" font-family="Times,serif" font-size="8.00" fill="#ff0000">clamp_2</text>
<text text-anchor="start" x="1.5713" y="-56.7238" font-family="Times,serif" font-size="8.00" fill="#000000">1 out </text>
<text text-anchor="start" x="61.865" y="-124.8014" font-family="Times,serif" font-size="8.00" fill="#000000">3 b </text>
</g>
<!-- 11300705378904136951 -->
<g id="node4" class="node">
<title>11300705378904136951</title>
<polygon fill="#d3d3d3" stroke="#000000" points="223,-54 169,-54 169,0 223,0 223,-54"/>
<text text-anchor="middle" x="196" y="-24.8" font-family="Times,serif" font-size="9.00" fill="#000000">clamp_1</text>
</g>
<!-- 18163720509575022497&#45;&#45;11300705378904136951 -->
<g id="edge1" class="edge">
<title>18163720509575022497&#45;&#45;11300705378904136951</title>
<path fill="none" stroke="#000000" d="M139.5402,-98.7371C151.0188,-84.1526 164.0257,-67.6262 174.6598,-54.1146"/>
<text text-anchor="start" x="158" y="-74.6" font-family="Times,serif" font-size="8.00" fill="#ff0000">clamp_1</text>
<text text-anchor="start" x="155.6598" y="-56.7146" font-family="Times,serif" font-size="8.00" fill="#000000">1 out </text>
<text text-anchor="start" x="126.5402" y="-92.3371" font-family="Times,serif" font-size="8.00" fill="#000000">2 a </text>
</g>
<!-- 6701433761736062521&#45;&#45;18163720509575022497 -->
<g id="edge2" class="edge">
<title>6701433761736062521&#45;&#45;18163720509575022497</title>
<path fill="none" stroke="#000000" d="M111,-215.9415C111,-201.7071 111,-185.3833 111,-171.1409"/>
<text text-anchor="start" x="111" y="-191.6" font-family="Times,serif" font-size="8.00" fill="#ff0000">θ</text>
<text text-anchor="start" x="92" y="-173.7409" font-family="Times,serif" font-size="8.00" fill="#000000">1 out </text>
<text text-anchor="start" x="98" y="-209.5415" font-family="Times,serif" font-size="8.00" fill="#000000">2 p </text>
</g>
</g>
</svg>



Code notes:
- @RV is a macro that lets you add Random Variables as nodes to your factor graph.
- The symbol ~ means "is distributed as". For example, $\theta \sim \text{Beta}(1,1)$ should be read as "$\theta$ is distributed according to a Beta($\theta$ | $a$=1, $b$=1) probability distribution".

----

Above you can see the factor graph that ForneyLab has generated. It is not as clean as the ones in the theory lectures. For example, ForneyLab generates nodes for the clamped parameters of the Beta prior ($\alpha = 1$ and $\beta = 1$), while we ignore these in the manually constructed graphs. Nonetheless, ForneyLab's version is very useful for debugging later on. 

We are now going to tell ForneyLab to generate a message passing procedure for us.


```julia
# Indicate which variables you want posteriors for
q = PosteriorFactorization(θ, ids=[:θ])

# Generate a message passing inference algorithm
algorithm = messagePassingAlgorithm(θ, q)

# Compile algorithm code
source_code = algorithmSourceCode(algorithm)

# Bring compiled code into current scope
eval(Meta.parse(source_code))

# Visualize message passing schedule
pfθ = q.posterior_factors[:θ]
ForneyLab.draw(pfθ, schedule=pfθ.schedule);
```


<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<!DOCTYPE svg PUBLIC "-//W3C//DTD SVG 1.1//EN"
 "http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd">
<!-- Generated by graphviz version 2.40.1 (20161225.0304)
 -->
<!-- Title: G Pages: 1 -->
<svg width="151pt" height="293pt"
 viewBox="0.00 0.00 150.93 293.00" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink">
<g id="graph0" class="graph" transform="scale(1 1) rotate(0) translate(4 289)">
<title>G</title>
<polygon fill="#ffffff" stroke="transparent" points="-4,4 -4,-289 146.9312,-289 146.9312,4 -4,4"/>
<!-- 18163720509575022497 -->
<g id="node1" class="node">
<title>18163720509575022497</title>
<polygon fill="none" stroke="#000000" points="110.4312,-120 38.4312,-120 38.4312,-48 110.4312,-48 110.4312,-120"/>
<text text-anchor="middle" x="74.4312" y="-86.8" font-family="Times,serif" font-size="9.00" fill="#000000">Beta</text>
<text text-anchor="middle" x="74.4312" y="-76.8" font-family="Times,serif" font-size="9.00" fill="#000000">f_a</text>
</g>
<!-- 8221999983495897590 -->
<g id="node4" class="node">
<title>8221999983495897590</title>
</g>
<!-- 18163720509575022497&#45;&#45;8221999983495897590 -->
<g id="edge4" class="edge">
<title>18163720509575022497&#45;&#45;8221999983495897590</title>
<path fill="none" stroke="#000000" stroke-dasharray="5,2" d="M45.5126,-47.9665C34.1264,-33.779 22.2874,-19.0273 16,-11.1929"/>
<text text-anchor="middle" x="8" y="-13.7929" font-family="Times,serif" font-size="8.00" fill="#000000">1 out</text>
<text text-anchor="middle" x="40.5126" y="-41.5665" font-family="Times,serif" font-size="8.00" fill="#000000">3 b</text>
</g>
<!-- 11300705378904136951 -->
<g id="node5" class="node">
<title>11300705378904136951</title>
</g>
<!-- 18163720509575022497&#45;&#45;11300705378904136951 -->
<g id="edge2" class="edge">
<title>18163720509575022497&#45;&#45;11300705378904136951</title>
<path fill="none" stroke="#000000" stroke-dasharray="5,2" d="M103.3497,-47.9665C114.7359,-33.779 126.5749,-19.0273 132.8623,-11.1929"/>
<text text-anchor="middle" x="124.8623" y="-13.7929" font-family="Times,serif" font-size="8.00" fill="#000000">1 out</text>
<text text-anchor="middle" x="98.3497" y="-41.5665" font-family="Times,serif" font-size="8.00" fill="#000000">2 a</text>
</g>
<!-- 6701433761736062521 -->
<g id="node2" class="node">
<title>6701433761736062521</title>
<polygon fill="none" stroke="#000000" points="110.4312,-237 38.4312,-237 38.4312,-165 110.4312,-165 110.4312,-237"/>
<text text-anchor="middle" x="74.4312" y="-203.8" font-family="Times,serif" font-size="9.00" fill="#000000">Ber</text>
<text text-anchor="middle" x="74.4312" y="-193.8" font-family="Times,serif" font-size="9.00" fill="#000000">f_b</text>
</g>
<!-- 6701433761736062521&#45;&#45;18163720509575022497 -->
<g id="edge1" class="edge">
<title>6701433761736062521&#45;&#45;18163720509575022497</title>
<path fill="none" stroke="#000000" d="M74.4312,-164.9415C74.4312,-150.7071 74.4312,-134.3833 74.4312,-120.1409"/>
<text text-anchor="start" x="74.4312" y="-140.6" font-family="Times,serif" font-size="8.00" fill="#ff0000">θ</text>
<text text-anchor="start" x="39.4312" y="-123.7409" font-family="Times,serif" font-size="8.00" fill="#000000">1 out </text>
<text text-anchor="start" x="58.4312" y="-123.7409" font-family="Times,serif" font-weight="bold" font-size="8.00" fill="#0000ff">((1))</text>
<text text-anchor="start" x="45.4312" y="-159.5415" font-family="Times,serif" font-size="8.00" fill="#000000">2 p </text>
<text text-anchor="start" x="58.4312" y="-159.5415" font-family="Times,serif" font-weight="bold" font-size="8.00" fill="#0000ff">((2))</text>
</g>
<!-- 12298408599198300 -->
<g id="node3" class="node">
<title>12298408599198300</title>
</g>
<!-- 12298408599198300&#45;&#45;6701433761736062521 -->
<g id="edge3" class="edge">
<title>12298408599198300&#45;&#45;6701433761736062521</title>
<path fill="none" stroke="#000000" stroke-dasharray="5,2" d="M74.4312,-273.8545C74.4312,-266.0501 74.4312,-251.3167 74.4312,-237.1302"/>
<text text-anchor="middle" x="66.4312" y="-239.7302" font-family="Times,serif" font-size="8.00" fill="#000000">1 out</text>
<text text-anchor="middle" x="66.4312" y="-267.4545" font-family="Times,serif" font-size="8.00" fill="#000000">1 out</text>
</g>
</g>
</svg>



Code notes:
- ForneyLab.jl compiles the specified model and inference procedure into a string. This string is human-readable and portable across devices. The functions `eval(Meta.parse())` are used to bring that string into the current scope, so the generated code can be used.
- In `ForneyLab.draw()`, only the edge of interest is shown with the two connecting nodes and their inputs. All other parts of the graph are ignored.

----

ForneyLab's visualization of the message passing procedure for a specific variable isolates that variable in the graph and shows where the incoming messages come from. In this case, we are interested in $\theta$ (your skill level), which receives message ((2)) from the likelihood node (the "Ber" node above $\theta$) and message ((1)) from the prior node (the "Beta" node below $\theta$). 

In the message passing framework, the combination of these two messages produces the "marginal" distribution for $\theta$. We are using message passing to do Bayesian inference, so note that the "marginal" for $\theta$ corresponds to the posterior distribution $p(\theta \mid X_1)$. 

Let's inspect these messages.


```julia
# Initialize data structure for messages
messages = Array{Message}(undef, 2)

# Initalize data structure for marginal distributions
marginals = Dict()

# Suppose you got question 1 correct
data = Dict(:X1 => 1)

# Update coefficients
stepθ!(data, marginals, messages);

# Print messages
print("\nMessage ((1)) = "*string(messages[1].dist))
println("Message ((2)) = "*string(messages[2].dist))
```

    
    Message ((1)) = Beta(a=1.00, b=1.00)
    Message ((2)) = Beta(a=2.00, b=1.00)
    


Code notes:
- A `Dict` is a [dictionary data structure](https://docs.julialang.org/en/v1/base/collections/#Base.Dict). In the `marginals` dictionary we only have one entry: the key is the variable `θ` (as a Symbol, i.e. as `:θ`) and the value is a `ProbabilityDistribution` object. It is the initial distribution for that variable. In the `data` dictionary, we also only have one entry: the key is the variable `X1` and the value is a Float. This is because `X1` is observed. We know its value without uncertainty.
- The `stepθ!` function comes from the algorithm compilation.

----

Alright. So, they are both Beta distributions. Do they actually make sense? Where do these parameters come from?

Recall from the lecture notes that the formula for messages sent by factor nodes is:

$$ \boxed{
\underbrace{\overrightarrow{\mu}_{Y}(y)}_{\substack{ \text{outgoing}\\ \text{message}}} = \sum_{x_1,\ldots,x_n} \underbrace{\overrightarrow{\mu}_{X_1}(x_1)\cdots \overrightarrow{\mu}_{X_n}(x_n)}_{\substack{\text{incoming} \\ \text{messages}}} \cdot \underbrace{f(y,x_1,\ldots,x_n)}_{\substack{\text{node}\\ \text{function}}} }
$$

<p style="text-align:center;"></p>

The prior node is not connected to any other unknown variables and so does not receive incoming messages. Its outgoing message is therefore:

$$\begin{align}
\overrightarrow{\mu}(\theta) =&\ f(\theta) \\
=&\ \text{Beta}(\theta \mid 1,1) \, .
\end{align}$$

So that confirms the correctness of Message ((1)).

Similarly, we can also derive the message from the likelihood node by hand. For this, we need to know that the message coming from the observation $\overleftarrow{\mu}(x)$ is a delta function, which, if you gave the right answer ($X_1 = 1$), has the form $\delta(X_1 - 1)$. The "node function" is the Bernoulli likelihood $\text{Bernoulli}(X_1 \mid \theta)$. Another thing to note is that this is essentially a convolution with respect to a delta function and that its [sifting property](https://en.wikipedia.org/wiki/Dirac_delta_function#Translation) holds: $\int_{X_1} \delta(X_1 - x) \ f(X_1, \theta) \mathrm{d}X_1 = f(x, \theta)$. The fact that $X_1$ is a discrete variable instead of a continuous one, does not negate this. Using these facts, we can perform the message computation by hand:

$$\begin{align}
\overleftarrow{\mu}(\theta) =&\ \sum_{X_1} \overleftarrow{\mu}(X_1) \ f(X_1, \theta) \\
=&\ \sum_{X_1} \delta(X_1 - 1) \ \text{Bernoulli}(X_1 \mid \theta) \\
=&\ \sum_{X_1} \delta(X_1 - 1) \ \theta^{X_1} (1 - \theta)^{1-X_1} \\
=&\ \theta^{1} (1 - \theta)^{1-1} \, .
\end{align}$$

Remember that the pdf of a Beta distribution is proportional to $\theta^{\alpha-1} (1 - \theta)^{\beta-1}$. So, if you read the second-to-last line above as $\theta^{2-1} (1 - \theta)^{1-1}$, then the outgoing message $\overleftarrow{\mu}(\theta)$ is proportional to a Beta distribution with $\alpha=2$ and $\beta=1$. So, our manual derivation verifies ForneyLab's Message ((2)).

Let's now look at these messages visually.


```julia
# Probability density function of a Beta distribution
Beta(θ, α, β) = 1/beta(α,β) * θ^(α-1) * (1-θ)^(β-1)

# Extract parameters from message ((1))
α1 = messages[1].dist.params[:a]
β1 = messages[1].dist.params[:b]

# Extract parameters from message ((2))
α2 = messages[2].dist.params[:a]
β2 = messages[2].dist.params[:b]

# Plot messages
θ_range = range(0, step=0.01, stop=1.0)
plot(θ_range, Beta.(θ_range, α1, β1), color="red", linewidth=3, label="Message ((1))", xlabel="θ", ylabel="p(θ)")
plot!(θ_range, Beta.(θ_range, α2, β2), color="blue", linewidth=3, label="Message ((2))")
```

The marginal distribution for $\theta$, representing the posterior $p(\theta \mid X_1)$, is obtained by taking the product (followed by normalization) of the two messages: $\overrightarrow{\mu}(\theta) \cdot \overleftarrow{\mu}(\theta)$. Multiplying two Beta distributions produces another Beta distribution with parameter:

$$\begin{align}
\alpha \leftarrow&\ \alpha_1 + \alpha_2 - 1 \\
\beta \leftarrow&\ \beta_1 + \beta_2 - 1 \, ,
\end{align}$$

In our case, the new parameters would be $\alpha = 1 + 2 - 1 = 2$ and $\beta = 1 + 1 - 1 = 1$. Let's check with ForneyLab what it computed.


```julia
marginals[:θ]
```




    Beta(a=2.00, b=1.00)




Again, ForneyLab matches our manual derivations. Let's visualize the messages as well as the marginal.


```julia
# Extract marginal's parameters
α_marg = marginals[:θ].params[:a]
β_marg = marginals[:θ].params[:b]

# Plot messages
θ_range = range(0, step=0.01, stop=1.0)
plot(θ_range, Beta.(θ_range, α1, β1), color="red", linewidth=3, label="Message ((1))", xlabel="θ", ylabel="p(θ)")
plot!(θ_range, Beta.(θ_range, α2, β2), color="blue", linewidth=3, label="Message ((2))")
plot!(θ_range, Beta.(θ_range, α_marg, β_marg), color="purple", linewidth=6, linestyle=:dash, label="Marginal")
```

The pdf of the marginal distribution lies on top of the pdf of Message ((2)). That's not always going to be the case; the Beta(1,1) distribution is special in that when you multiply Beta(1,1) with a general Beta(a,b) the result will always be Beta(a,b), kinda like multiplying by $1$. We call prior distributions that have this special effect "non-informative priors".

#### Multiple questions

Of course, you won't be evaluated on just a single question: it's still possible for you to get one question wrong even if you have a high skill level. You would consider it unfair to be rejected based on only one question. So, we are going to add another question. We're also going to change the prior: the company now assumes that you must have _some_ skill if you applied for the position. This is reflected in a prior Beta distributions with $\alpha = 3.0$ and $\beta = 2.0$. 

For now, the second question is also a right-or-wrong question. The outcome of this new question is denoted with variable $X_2$. With this addition, the generative model becomes

$$p(X_1, X_2, \theta) = p(X_1 \mid \theta) p(X_2 \mid \theta) p(\theta) \, ,$$ 

with the accompanying factor graph



where $f_c \triangleq \text{Bernoulli}(X_2 \mid \theta)$ and $f_a, f_b$ are still the same. Notice that we now have an equality node as well. That is because the variable $\theta$ is used in three factor nodes. ForneyLab automatically generates the same factor graph:


```julia
# Start building a model
factor_graph2 = FactorGraph()

# Add the prior
@RV θ ~ Beta(3.0, 2.0, id=:f_a)

# Add question 1 correctness likelihood
@RV X1 ~ Bernoulli(θ, id=:f_b)

# Add question 2 correctness likelihood
@RV X2 ~ Bernoulli(θ, id=:f_c)

# The question outcomes are going to be observed
placeholder(X1, :X1)
placeholder(X2, :X2)

# Visualize the graph
ForneyLab.draw(factor_graph2)
```


<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<!DOCTYPE svg PUBLIC "-//W3C//DTD SVG 1.1//EN"
 "http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd">
<!-- Generated by graphviz version 2.40.1 (20161225.0304)
 -->
<!-- Title: G Pages: 1 -->
<svg width="583pt" height="296pt"
 viewBox="0.00 0.00 583.00 296.00" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink">
<g id="graph0" class="graph" transform="scale(1 1) rotate(0) translate(4 292)">
<title>G</title>
<polygon fill="#ffffff" stroke="transparent" points="-4,4 -4,-292 579,-292 579,4 -4,4"/>
<!-- 3022940587281690172 -->
<g id="node1" class="node">
<title>3022940587281690172</title>
<polygon fill="none" stroke="#000000" points="543.5,-171 471.5,-171 471.5,-99 543.5,-99 543.5,-171"/>
<text text-anchor="middle" x="507.5" y="-137.8" font-family="Times,serif" font-size="9.00" fill="#000000">Ber</text>
<text text-anchor="middle" x="507.5" y="-127.8" font-family="Times,serif" font-size="9.00" fill="#000000">f_c</text>
</g>
<!-- 7324254008702105268 -->
<g id="node2" class="node">
<title>7324254008702105268</title>
<polygon fill="none" stroke="#000000" points="326.5,-288 254.5,-288 254.5,-216 326.5,-216 326.5,-288"/>
<text text-anchor="middle" x="290.5" y="-254.8" font-family="Times,serif" font-size="9.00" fill="#000000">=</text>
<text text-anchor="middle" x="290.5" y="-244.8" font-family="Times,serif" font-size="9.00" fill="#000000">equ_θ_1</text>
</g>
<!-- 7324254008702105268&#45;&#45;3022940587281690172 -->
<g id="edge1" class="edge">
<title>7324254008702105268&#45;&#45;3022940587281690172</title>
<path fill="none" stroke="#000000" d="M326.8209,-232.4168C366.9985,-210.7542 431.3426,-176.0618 471.4223,-154.452"/>
<text text-anchor="start" x="406.5" y="-191.6" font-family="Times,serif" font-size="8.00" fill="#ff0000">θ</text>
<text text-anchor="start" x="458.4223" y="-157.052" font-family="Times,serif" font-size="8.00" fill="#000000">2 p </text>
<text text-anchor="start" x="326.8209" y="-235.0168" font-family="Times,serif" font-size="8.00" fill="#000000">3 3 </text>
</g>
<!-- 15029869380802396516 -->
<g id="node3" class="node">
<title>15029869380802396516</title>
<polygon fill="none" stroke="#000000" points="326.5,-171 254.5,-171 254.5,-99 326.5,-99 326.5,-171"/>
<text text-anchor="middle" x="290.5" y="-137.8" font-family="Times,serif" font-size="9.00" fill="#000000">Beta</text>
<text text-anchor="middle" x="290.5" y="-127.8" font-family="Times,serif" font-size="9.00" fill="#000000">f_a</text>
</g>
<!-- 7324254008702105268&#45;&#45;15029869380802396516 -->
<g id="edge6" class="edge">
<title>7324254008702105268&#45;&#45;15029869380802396516</title>
<path fill="none" stroke="#000000" d="M290.5,-215.9415C290.5,-201.7071 290.5,-185.3833 290.5,-171.1409"/>
<text text-anchor="start" x="290.5" y="-191.6" font-family="Times,serif" font-size="8.00" fill="#ff0000">θ</text>
<text text-anchor="start" x="271.5" y="-173.7409" font-family="Times,serif" font-size="8.00" fill="#000000">1 out </text>
<text text-anchor="start" x="277.5" y="-209.5415" font-family="Times,serif" font-size="8.00" fill="#000000">1 1 </text>
</g>
<!-- 2328518386071655588 -->
<g id="node6" class="node">
<title>2328518386071655588</title>
<polygon fill="none" stroke="#000000" points="106.5,-171 34.5,-171 34.5,-99 106.5,-99 106.5,-171"/>
<text text-anchor="middle" x="70.5" y="-137.8" font-family="Times,serif" font-size="9.00" fill="#000000">Ber</text>
<text text-anchor="middle" x="70.5" y="-127.8" font-family="Times,serif" font-size="9.00" fill="#000000">f_b</text>
</g>
<!-- 7324254008702105268&#45;&#45;2328518386071655588 -->
<g id="edge4" class="edge">
<title>7324254008702105268&#45;&#45;2328518386071655588</title>
<path fill="none" stroke="#000000" d="M254.1715,-232.6798C213.287,-210.9367 147.2731,-175.8293 106.5168,-154.1544"/>
<text text-anchor="start" x="187.5" y="-191.6" font-family="Times,serif" font-size="8.00" fill="#ff0000">θ</text>
<text text-anchor="start" x="106.5168" y="-156.7544" font-family="Times,serif" font-size="8.00" fill="#000000">2 p </text>
<text text-anchor="start" x="241.1715" y="-235.2798" font-family="Times,serif" font-size="8.00" fill="#000000">2 2 </text>
</g>
<!-- 12266119185860459323 -->
<g id="node7" class="node">
<title>12266119185860459323</title>
<polygon fill="#d3d3d3" stroke="#000000" points="232.5,-54 178.5,-54 178.5,0 232.5,0 232.5,-54"/>
<text text-anchor="middle" x="205.5" y="-24.8" font-family="Times,serif" font-size="9.00" fill="#000000">clamp_2</text>
</g>
<!-- 15029869380802396516&#45;&#45;12266119185860459323 -->
<g id="edge5" class="edge">
<title>15029869380802396516&#45;&#45;12266119185860459323</title>
<path fill="none" stroke="#000000" d="M261.9598,-98.7371C250.4812,-84.1526 237.4743,-67.6262 226.8402,-54.1146"/>
<text text-anchor="start" x="248.5" y="-74.6" font-family="Times,serif" font-size="8.00" fill="#ff0000">clamp_2</text>
<text text-anchor="start" x="207.8402" y="-56.7146" font-family="Times,serif" font-size="8.00" fill="#000000">1 out </text>
<text text-anchor="start" x="248.9598" y="-92.3371" font-family="Times,serif" font-size="8.00" fill="#000000">3 b </text>
</g>
<!-- 7308723381236975872 -->
<g id="node8" class="node">
<title>7308723381236975872</title>
<polygon fill="#d3d3d3" stroke="#000000" points="401.5,-54 347.5,-54 347.5,0 401.5,0 401.5,-54"/>
<text text-anchor="middle" x="374.5" y="-24.8" font-family="Times,serif" font-size="9.00" fill="#000000">clamp_1</text>
</g>
<!-- 15029869380802396516&#45;&#45;7308723381236975872 -->
<g id="edge3" class="edge">
<title>15029869380802396516&#45;&#45;7308723381236975872</title>
<path fill="none" stroke="#000000" d="M318.7045,-98.7371C330.048,-84.1526 342.9019,-67.6262 353.4108,-54.1146"/>
<text text-anchor="start" x="338.5" y="-74.6" font-family="Times,serif" font-size="8.00" fill="#ff0000">clamp_1</text>
<text text-anchor="start" x="334.4108" y="-56.7146" font-family="Times,serif" font-size="8.00" fill="#000000">1 out </text>
<text text-anchor="start" x="305.7045" y="-92.3371" font-family="Times,serif" font-size="8.00" fill="#000000">2 a </text>
</g>
<!-- 16533094279308769517 -->
<g id="node4" class="node">
<title>16533094279308769517</title>
<polygon fill="#d3d3d3" stroke="#000000" points="575,-279 502,-279 502,-225 575,-225 575,-279"/>
<text text-anchor="middle" x="538.5" y="-249.8" font-family="Times,serif" font-size="9.00" fill="#000000">placeholder_X2</text>
</g>
<!-- 16533094279308769517&#45;&#45;3022940587281690172 -->
<g id="edge7" class="edge">
<title>16533094279308769517&#45;&#45;3022940587281690172</title>
<path fill="none" stroke="#000000" d="M531.3101,-224.8638C527.0889,-208.9324 521.7004,-188.595 517.1175,-171.2982"/>
<text text-anchor="start" x="524.5" y="-191.6" font-family="Times,serif" font-size="8.00" fill="#ff0000">X2</text>
<text text-anchor="start" x="498.1175" y="-173.8982" font-family="Times,serif" font-size="8.00" fill="#000000">1 out </text>
<text text-anchor="start" x="512.3101" y="-218.4638" font-family="Times,serif" font-size="8.00" fill="#000000">1 out </text>
</g>
<!-- 4439389401710276126 -->
<g id="node5" class="node">
<title>4439389401710276126</title>
<polygon fill="#d3d3d3" stroke="#000000" points="73,-279 0,-279 0,-225 73,-225 73,-279"/>
<text text-anchor="middle" x="36.5" y="-249.8" font-family="Times,serif" font-size="9.00" fill="#000000">placeholder_X1</text>
</g>
<!-- 4439389401710276126&#45;&#45;2328518386071655588 -->
<g id="edge2" class="edge">
<title>4439389401710276126&#45;&#45;2328518386071655588</title>
<path fill="none" stroke="#000000" d="M33.2301,-224.8683C32.7036,-213.5759 33.1943,-200.4544 36.5,-189 38.2509,-182.9329 40.9078,-176.8904 43.9802,-171.1624"/>
<text text-anchor="start" x="36.5" y="-191.6" font-family="Times,serif" font-size="8.00" fill="#ff0000">X1</text>
<text text-anchor="start" x="24.9802" y="-173.7624" font-family="Times,serif" font-size="8.00" fill="#000000">1 out </text>
<text text-anchor="start" x="14.2301" y="-218.4683" font-family="Times,serif" font-size="8.00" fill="#000000">1 out </text>
</g>
</g>
</svg>



We will go through the message passing operations below. First, we generate an algorithm and visualize where all the messages for $\theta$ come from.


```julia
# Indicate which variables you want posteriors for
q = PosteriorFactorization(θ, ids=[:θ])

# Generate a message passing inference algorithm
algorithm = messagePassingAlgorithm(θ, q)

# Compile algorithm code
source_code = algorithmSourceCode(algorithm)

# Bring compiled code into current scope
eval(Meta.parse(source_code))

# Visualize message passing schedule
pfθ = q.posterior_factors[:θ]
ForneyLab.draw(pfθ, schedule=pfθ.schedule);
```


<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<!DOCTYPE svg PUBLIC "-//W3C//DTD SVG 1.1//EN"
 "http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd">
<!-- Generated by graphviz version 2.40.1 (20161225.0304)
 -->
<!-- Title: G Pages: 1 -->
<svg width="548pt" height="245pt"
 viewBox="0.00 0.00 548.20 245.00" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink">
<g id="graph0" class="graph" transform="scale(1 1) rotate(0) translate(4 241)">
<title>G</title>
<polygon fill="#ffffff" stroke="transparent" points="-4,4 -4,-241 544.2025,-241 544.2025,4 -4,4"/>
<!-- 3022940587281690172 -->
<g id="node1" class="node">
<title>3022940587281690172</title>
<polygon fill="none" stroke="#000000" points="540.2025,-120 468.2025,-120 468.2025,-48 540.2025,-48 540.2025,-120"/>
<text text-anchor="middle" x="504.2025" y="-86.8" font-family="Times,serif" font-size="9.00" fill="#000000">Ber</text>
<text text-anchor="middle" x="504.2025" y="-76.8" font-family="Times,serif" font-size="9.00" fill="#000000">f_c</text>
</g>
<!-- 7324254008702105268 -->
<g id="node2" class="node">
<title>7324254008702105268</title>
<polygon fill="none" stroke="#000000" points="351.2025,-237 279.2025,-237 279.2025,-165 351.2025,-165 351.2025,-237"/>
<text text-anchor="middle" x="315.2025" y="-203.8" font-family="Times,serif" font-size="9.00" fill="#000000">=</text>
<text text-anchor="middle" x="315.2025" y="-193.8" font-family="Times,serif" font-size="9.00" fill="#000000">equ_θ_1</text>
</g>
<!-- 7324254008702105268&#45;&#45;3022940587281690172 -->
<g id="edge1" class="edge">
<title>7324254008702105268&#45;&#45;3022940587281690172</title>
<path fill="none" stroke="#000000" d="M351.6436,-178.4412C385.2185,-157.6568 434.7575,-126.9898 468.1943,-106.2908"/>
<text text-anchor="start" x="416.2025" y="-140.6" font-family="Times,serif" font-size="8.00" fill="#ff0000">θ</text>
<text text-anchor="start" x="439.1943" y="-109.8908" font-family="Times,serif" font-size="8.00" fill="#000000">2 p </text>
<text text-anchor="start" x="452.1943" y="-109.8908" font-family="Times,serif" font-weight="bold" font-size="8.00" fill="#0000ff">((3))</text>
<text text-anchor="start" x="351.6436" y="-181.0412" font-family="Times,serif" font-size="8.00" fill="#000000">3 3 </text>
</g>
<!-- 15029869380802396516 -->
<g id="node3" class="node">
<title>15029869380802396516</title>
<polygon fill="none" stroke="#000000" points="351.2025,-120 279.2025,-120 279.2025,-48 351.2025,-48 351.2025,-120"/>
<text text-anchor="middle" x="315.2025" y="-86.8" font-family="Times,serif" font-size="9.00" fill="#000000">Beta</text>
<text text-anchor="middle" x="315.2025" y="-76.8" font-family="Times,serif" font-size="9.00" fill="#000000">f_a</text>
</g>
<!-- 7324254008702105268&#45;&#45;15029869380802396516 -->
<g id="edge3" class="edge">
<title>7324254008702105268&#45;&#45;15029869380802396516</title>
<path fill="none" stroke="#000000" d="M315.2025,-164.9415C315.2025,-150.7071 315.2025,-134.3833 315.2025,-120.1409"/>
<text text-anchor="start" x="315.2025" y="-140.6" font-family="Times,serif" font-size="8.00" fill="#ff0000">θ</text>
<text text-anchor="start" x="280.2025" y="-123.7409" font-family="Times,serif" font-size="8.00" fill="#000000">1 out </text>
<text text-anchor="start" x="299.2025" y="-123.7409" font-family="Times,serif" font-weight="bold" font-size="8.00" fill="#0000ff">((1))</text>
<text text-anchor="start" x="292.2025" y="-159.5415" font-family="Times,serif" font-size="8.00" fill="#000000">1 1 </text>
<text text-anchor="start" x="305.2025" y="-159.5415" font-family="Times,serif" font-weight="bold" font-size="8.00" fill="#0000ff">(4)</text>
</g>
<!-- 2328518386071655588 -->
<g id="node4" class="node">
<title>2328518386071655588</title>
<polygon fill="none" stroke="#000000" points="107.2025,-120 35.2025,-120 35.2025,-48 107.2025,-48 107.2025,-120"/>
<text text-anchor="middle" x="71.2025" y="-86.8" font-family="Times,serif" font-size="9.00" fill="#000000">Ber</text>
<text text-anchor="middle" x="71.2025" y="-76.8" font-family="Times,serif" font-size="9.00" fill="#000000">f_b</text>
</g>
<!-- 7324254008702105268&#45;&#45;2328518386071655588 -->
<g id="edge2" class="edge">
<title>7324254008702105268&#45;&#45;2328518386071655588</title>
<path fill="none" stroke="#000000" d="M279.1995,-183.7362C233.0673,-161.6155 153.5566,-123.4895 107.3533,-101.3346"/>
<text text-anchor="start" x="201.2025" y="-140.6" font-family="Times,serif" font-size="8.00" fill="#ff0000">θ</text>
<text text-anchor="start" x="107.3533" y="-104.9346" font-family="Times,serif" font-size="8.00" fill="#000000">2 p </text>
<text text-anchor="start" x="120.3533" y="-104.9346" font-family="Times,serif" font-weight="bold" font-size="8.00" fill="#0000ff">((2))</text>
<text text-anchor="start" x="266.1995" y="-186.3362" font-family="Times,serif" font-size="8.00" fill="#000000">2 2 </text>
</g>
<!-- 12266119185860459323 -->
<g id="node7" class="node">
<title>12266119185860459323</title>
</g>
<!-- 15029869380802396516&#45;&#45;12266119185860459323 -->
<g id="edge6" class="edge">
<title>15029869380802396516&#45;&#45;12266119185860459323</title>
<path fill="none" stroke="#000000" stroke-dasharray="5,2" d="M286.2839,-47.9665C274.8978,-33.779 263.0588,-19.0273 256.7714,-11.1929"/>
<text text-anchor="middle" x="248.7714" y="-13.7929" font-family="Times,serif" font-size="8.00" fill="#000000">1 out</text>
<text text-anchor="middle" x="281.2839" y="-41.5665" font-family="Times,serif" font-size="8.00" fill="#000000">3 b</text>
</g>
<!-- 7308723381236975872 -->
<g id="node8" class="node">
<title>7308723381236975872</title>
</g>
<!-- 15029869380802396516&#45;&#45;7308723381236975872 -->
<g id="edge5" class="edge">
<title>15029869380802396516&#45;&#45;7308723381236975872</title>
<path fill="none" stroke="#000000" stroke-dasharray="5,2" d="M344.1211,-47.9665C355.5073,-33.779 367.3463,-19.0273 373.6337,-11.1929"/>
<text text-anchor="middle" x="365.6337" y="-13.7929" font-family="Times,serif" font-size="8.00" fill="#000000">1 out</text>
<text text-anchor="middle" x="339.1211" y="-41.5665" font-family="Times,serif" font-size="8.00" fill="#000000">2 a</text>
</g>
<!-- 16533094279308769517 -->
<g id="node5" class="node">
<title>16533094279308769517</title>
</g>
<!-- 16533094279308769517&#45;&#45;3022940587281690172 -->
<g id="edge7" class="edge">
<title>16533094279308769517&#45;&#45;3022940587281690172</title>
<path fill="none" stroke="#000000" stroke-dasharray="5,2" d="M506.1061,-195.3568C505.8767,-181.9367 505.2831,-147.2114 504.8192,-120.0735"/>
<text text-anchor="middle" x="496.8192" y="-122.6735" font-family="Times,serif" font-size="8.00" fill="#000000">1 out</text>
<text text-anchor="middle" x="498.1061" y="-188.9568" font-family="Times,serif" font-size="8.00" fill="#000000">1 out</text>
</g>
<!-- 4439389401710276126 -->
<g id="node6" class="node">
<title>4439389401710276126</title>
</g>
<!-- 4439389401710276126&#45;&#45;2328518386071655588 -->
<g id="edge4" class="edge">
<title>4439389401710276126&#45;&#45;2328518386071655588</title>
<path fill="none" stroke="#000000" stroke-dasharray="5,2" d="M16,-195.3568C22.6527,-181.9367 39.867,-147.2114 53.3199,-120.0735"/>
<text text-anchor="middle" x="45.3199" y="-122.6735" font-family="Times,serif" font-size="8.00" fill="#000000">1 out</text>
<text text-anchor="middle" x="8" y="-188.9568" font-family="Times,serif" font-size="8.00" fill="#000000">1 out</text>
</g>
</g>
</svg>



There are 4 messages, one from the prior ((1)), one from the first likelihood ((2)), one from the second likelihood ((3)) and one from the equality node ((4)). ForneyLab essentially combines messages 2 and 3 into message 4 and then multiplies messages 1 and 4 to produce the marginal. We can see this if we look in the source code:


```julia
println(source_code)
```

    begin
    
    function stepθ!(data::Dict, marginals::Dict=Dict(), messages::Vector{Message}=Array{Message}(undef, 4))
    
    messages[1] = ruleVBBetaOut(nothing, ProbabilityDistribution(Univariate, PointMass, m=3.0), ProbabilityDistribution(Univariate, PointMass, m=2.0))
    messages[2] = ruleVBBernoulliIn1(ProbabilityDistribution(Univariate, PointMass, m=data[:X1]), nothing)
    messages[3] = ruleVBBernoulliIn1(ProbabilityDistribution(Univariate, PointMass, m=data[:X2]), nothing)
    messages[4] = ruleSPEqualityBeta(nothing, messages[2], messages[3])
    
    marginals[:θ] = messages[1].dist * messages[4].dist
    
    return marginals
    
    end
    
    end # block


You can see that `messages[4]` is a function of `messages[2]` and `messages[3]` and that `marginals[:θ]` is the product of `messages[1]` and `messages[4]`. 


Suppose you got the first question right and the second question wrong. Let's execute the message passing procedure and take a look at the functional form of the messages.


```julia
# Initialize a message data structure
messages = Array{Message}(undef, 4)

# Initalize marginal distributions data structure
marginals = Dict()

# Suppose you got question 1 right and question 2 wrong
data = Dict(:X1 => 1,
            :X2 => 0)

# Update coefficients
stepθ!(data, marginals, messages);

# Print messages
print("\nMessage ((1)) = "*string(messages[1].dist))
print("Message ((2)) = "*string(messages[2].dist))
print("Message ((3)) = "*string(messages[3].dist))
println("Message ((4)) = "*string(messages[4].dist))
```

    
    Message ((1)) = Beta(a=3.00, b=2.00)
    Message ((2)) = Beta(a=2.00, b=1.00)
    Message ((3)) = Beta(a=1.00, b=2.00)
    Message ((4)) = Beta(a=2.00, b=2.00)
    


Messages ((1)) and ((2)) are clear, but Message ((3)) and Message ((4)) are new. 

---

#### $\ast$ **Try for yourself**

Try deriving the functional form of Message ((3)) for yourself. 
Tip: the derivation is very similar to that of Message ((2)). The most important change is to use $\delta(X_2 - 0)$ instead of $\delta(X_1 - 1)$.

---

Message ((4)) is the result of the standard message computation formula for the case of an equality node:

$$\begin{align}
\downarrow \mu(\theta) =&\ \sum_{\theta',\ \theta''} \overrightarrow{\mu}(\theta'')\ f_{=}(\theta, \theta', \theta'') \ \overleftarrow{\mu}(\theta') \\
 =&\ \overrightarrow{\mu}(\theta) \cdot \overleftarrow{\mu}(\theta) \\
=&\ \text{Beta}(\theta \mid 2, 1) \cdot \text{Beta}(\theta \mid 1, 2) \\
 =&\ \text{Beta}(\theta \mid 2, 2) \quad .
\end{align}$$

Let's visualize the messages and the marginal again.


```julia
# Extract parameters from message ((2))
α2 = messages[2].dist.params[:a]
β2 = messages[2].dist.params[:b]

# Extract parameters from message ((3))
α3 = messages[3].dist.params[:a]
β3 = messages[3].dist.params[:b]

# Extract parameters from message ((4))
α4 = messages[4].dist.params[:a]
β4 = messages[4].dist.params[:b]

plot(θ_range, Beta.(θ_range, α2, β2), color="black", linewidth=3, label="Message ((2))")
plot!(θ_range, Beta.(θ_range, α3, β3), color="green", linewidth=3, label="Message ((3))")
plot!(θ_range, Beta.(θ_range, α4, β4), color="blue", linewidth=3, label="Message ((4))")
xlabel!("θ")
ylabel!("p(θ)")
```

Message ((2)) and Message ((3)) are direct opposites: ((2)) increases the estimate and ((3)) decreases the estimate of your skill level.  Message ((4)) end up being centered on $0.5$. With one question right and one question wrong, you have essentially been guessing at random.


```julia
# Extract parameters from message ((1))
α1 = messages[1].dist.params[:a]
β1 = messages[1].dist.params[:b]

# Extract parameters from message ((4))
α4 = messages[4].dist.params[:a]
β4 = messages[4].dist.params[:b]

# Extract parameters
α_marg = marginals[:θ].params[:a]
β_marg = marginals[:θ].params[:b]

plot(θ_range, Beta.(θ_range, α1, β1), color="red", linewidth=3, label="Message ((1))")
plot!(θ_range, Beta.(θ_range, α4, β4), color="blue", linewidth=3, label="Message ((4))")
plot!(θ_range, Beta.(θ_range, α_marg, β_marg), color="purple", linewidth=4, linestyle=:dash, label="Marginal")
xlabel!("θ")
ylabel!("p(θ)")
```

If we now combine the prior (Message ((1)) in red above) with the combined message from both likelihood terms (Message ((4)) in blue), we get the new marginal (purple dotted line). The mean of the marginal lies above $0.5$, which is due to the prior assumption that you must have _some_ skill if you applied. 

### 2. Score questions

So far, the models we have been looking at have been quite simple; they are Beta-Bernoulli combinations which is exactly what we did for the Beer Tasting Experiment. We will now move on to more complicated distributions. These will enrich your toolbox and allow you to do much more.

Suppose you are not tested on a right-or-wrong question, but on a score question. For instance, you have to complete a piece of code for which you get a score. If all of it was wrong you get a score of $0$, if some of it was correct you get a score of $1$ and if all of it was correct you get a score $2$. That means we have a likelihood with three outcomes: $X_1 = \{ 0,1,2\}$. Suppose we once again ask two questions, $X_1$ and $X_2$. The order in which we ask these questions does not matter, so that means we choose Categorical distributions for these likelihood functions: $X_1, X_2 \sim \text{Categorical}(\theta)$. The parameter $\theta$ is no longer a single parameter, indicating the probability of getting the question right, but a vector of three parameters: $\theta = (\theta_1, \theta_2, \theta_3)$. Each $\theta_k$ indicates the probability of getting the $k$-th outcome. In other words, $\theta_1$ indicates the probability of getting $0$ points, $\theta_2$ of getting $1$ point and $\theta_3$ of getting two points. A highly-skilled applicant will have a parameter vector of $(0.05, 0.1, 0.85)$, for example. The prior distribution conjugate to the Categorical distribution is the Dirichlet distribution. 

Let's look at the generative model:

$$p(X_1, X_2, \theta) = p(X_1 \mid \theta) p(X_2 \mid \theta) p(\theta) \, .$$ 

It's the same as before. The only difference is that:

$$\begin{align}
p(X_1 \mid \theta) =&\ \text{Categorical}(X_1 \mid \theta) \\
p(X_2 \mid \theta) =&\ \text{Categorical}(X_2 \mid \theta) \\
p(\theta) =&\ \text{Dirichlet}(\theta)
\end{align}$$

The factor graph has the same structure as before. The only change is that the factor nodes $f_a, f_b, f_c$ are now parameterized differently. 


```julia
# Start building a model
factor_graph3 = FactorGraph()

# Add the prior
@RV θ ~ Dirichlet([1.0, 3.0, 2.0], id=:f_a)

# Add question 1 correctness likelihood
@RV X1 ~ Categorical(θ, id=:f_b)

# Add question 2 correctness likelihood
@RV X2 ~ Categorical(θ, id=:f_c)

# The question outcomes are going to be observed
placeholder(X1, dims=(3,), :X1)
placeholder(X2, dims=(3,), :X2)

# Visualize the graph
ForneyLab.draw(factor_graph3)
```


<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<!DOCTYPE svg PUBLIC "-//W3C//DTD SVG 1.1//EN"
 "http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd">
<!-- Generated by graphviz version 2.40.1 (20161225.0304)
 -->
<!-- Title: G Pages: 1 -->
<svg width="583pt" height="296pt"
 viewBox="0.00 0.00 583.00 296.00" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink">
<g id="graph0" class="graph" transform="scale(1 1) rotate(0) translate(4 292)">
<title>G</title>
<polygon fill="#ffffff" stroke="transparent" points="-4,4 -4,-292 579,-292 579,4 -4,4"/>
<!-- 6707493110187205485 -->
<g id="node1" class="node">
<title>6707493110187205485</title>
<polygon fill="none" stroke="#000000" points="326.5,-171 254.5,-171 254.5,-99 326.5,-99 326.5,-171"/>
<text text-anchor="middle" x="290.5" y="-137.8" font-family="Times,serif" font-size="9.00" fill="#000000">Dir</text>
<text text-anchor="middle" x="290.5" y="-127.8" font-family="Times,serif" font-size="9.00" fill="#000000">f_a</text>
</g>
<!-- 8595486343092272081 -->
<g id="node5" class="node">
<title>8595486343092272081</title>
<polygon fill="#d3d3d3" stroke="#000000" points="317.5,-54 263.5,-54 263.5,0 317.5,0 317.5,-54"/>
<text text-anchor="middle" x="290.5" y="-24.8" font-family="Times,serif" font-size="9.00" fill="#000000">clamp_1</text>
</g>
<!-- 6707493110187205485&#45;&#45;8595486343092272081 -->
<g id="edge1" class="edge">
<title>6707493110187205485&#45;&#45;8595486343092272081</title>
<path fill="none" stroke="#000000" d="M290.5,-98.7371C290.5,-84.1526 290.5,-67.6262 290.5,-54.1146"/>
<text text-anchor="start" x="290.5" y="-74.6" font-family="Times,serif" font-size="8.00" fill="#ff0000">clamp_1</text>
<text text-anchor="start" x="271.5" y="-56.7146" font-family="Times,serif" font-size="8.00" fill="#000000">1 out </text>
<text text-anchor="start" x="277.5" y="-92.3371" font-family="Times,serif" font-size="8.00" fill="#000000">2 a </text>
</g>
<!-- 1847487353024847686 -->
<g id="node2" class="node">
<title>1847487353024847686</title>
<polygon fill="#d3d3d3" stroke="#000000" points="575,-279 502,-279 502,-225 575,-225 575,-279"/>
<text text-anchor="middle" x="538.5" y="-249.8" font-family="Times,serif" font-size="9.00" fill="#000000">placeholder_X2</text>
</g>
<!-- 6625773108833308772 -->
<g id="node4" class="node">
<title>6625773108833308772</title>
<polygon fill="none" stroke="#000000" points="543.5,-171 471.5,-171 471.5,-99 543.5,-99 543.5,-171"/>
<text text-anchor="middle" x="507.5" y="-137.8" font-family="Times,serif" font-size="9.00" fill="#000000">Cat</text>
<text text-anchor="middle" x="507.5" y="-127.8" font-family="Times,serif" font-size="9.00" fill="#000000">f_c</text>
</g>
<!-- 1847487353024847686&#45;&#45;6625773108833308772 -->
<g id="edge4" class="edge">
<title>1847487353024847686&#45;&#45;6625773108833308772</title>
<path fill="none" stroke="#000000" d="M531.3101,-224.8638C527.0889,-208.9324 521.7004,-188.595 517.1175,-171.2982"/>
<text text-anchor="start" x="524.5" y="-191.6" font-family="Times,serif" font-size="8.00" fill="#ff0000">X2</text>
<text text-anchor="start" x="498.1175" y="-173.8982" font-family="Times,serif" font-size="8.00" fill="#000000">1 out </text>
<text text-anchor="start" x="512.3101" y="-218.4638" font-family="Times,serif" font-size="8.00" fill="#000000">1 out </text>
</g>
<!-- 11615063546771903839 -->
<g id="node3" class="node">
<title>11615063546771903839</title>
<polygon fill="none" stroke="#000000" points="106.5,-171 34.5,-171 34.5,-99 106.5,-99 106.5,-171"/>
<text text-anchor="middle" x="70.5" y="-137.8" font-family="Times,serif" font-size="9.00" fill="#000000">Cat</text>
<text text-anchor="middle" x="70.5" y="-127.8" font-family="Times,serif" font-size="9.00" fill="#000000">f_b</text>
</g>
<!-- 8122598982465519112 -->
<g id="node6" class="node">
<title>8122598982465519112</title>
<polygon fill="#d3d3d3" stroke="#000000" points="73,-279 0,-279 0,-225 73,-225 73,-279"/>
<text text-anchor="middle" x="36.5" y="-249.8" font-family="Times,serif" font-size="9.00" fill="#000000">placeholder_X1</text>
</g>
<!-- 8122598982465519112&#45;&#45;11615063546771903839 -->
<g id="edge6" class="edge">
<title>8122598982465519112&#45;&#45;11615063546771903839</title>
<path fill="none" stroke="#000000" d="M33.2301,-224.8683C32.7036,-213.5759 33.1943,-200.4544 36.5,-189 38.2509,-182.9329 40.9078,-176.8904 43.9802,-171.1624"/>
<text text-anchor="start" x="36.5" y="-191.6" font-family="Times,serif" font-size="8.00" fill="#ff0000">X1</text>
<text text-anchor="start" x="24.9802" y="-173.7624" font-family="Times,serif" font-size="8.00" fill="#000000">1 out </text>
<text text-anchor="start" x="14.2301" y="-218.4683" font-family="Times,serif" font-size="8.00" fill="#000000">1 out </text>
</g>
<!-- 15229993427491404172 -->
<g id="node7" class="node">
<title>15229993427491404172</title>
<polygon fill="none" stroke="#000000" points="326.5,-288 254.5,-288 254.5,-216 326.5,-216 326.5,-288"/>
<text text-anchor="middle" x="290.5" y="-254.8" font-family="Times,serif" font-size="9.00" fill="#000000">=</text>
<text text-anchor="middle" x="290.5" y="-244.8" font-family="Times,serif" font-size="9.00" fill="#000000">equ_θ_1</text>
</g>
<!-- 15229993427491404172&#45;&#45;6707493110187205485 -->
<g id="edge5" class="edge">
<title>15229993427491404172&#45;&#45;6707493110187205485</title>
<path fill="none" stroke="#000000" d="M290.5,-215.9415C290.5,-201.7071 290.5,-185.3833 290.5,-171.1409"/>
<text text-anchor="start" x="290.5" y="-191.6" font-family="Times,serif" font-size="8.00" fill="#ff0000">θ</text>
<text text-anchor="start" x="271.5" y="-173.7409" font-family="Times,serif" font-size="8.00" fill="#000000">1 out </text>
<text text-anchor="start" x="277.5" y="-209.5415" font-family="Times,serif" font-size="8.00" fill="#000000">1 1 </text>
</g>
<!-- 15229993427491404172&#45;&#45;11615063546771903839 -->
<g id="edge3" class="edge">
<title>15229993427491404172&#45;&#45;11615063546771903839</title>
<path fill="none" stroke="#000000" d="M254.1715,-232.6798C213.287,-210.9367 147.2731,-175.8293 106.5168,-154.1544"/>
<text text-anchor="start" x="187.5" y="-191.6" font-family="Times,serif" font-size="8.00" fill="#ff0000">θ</text>
<text text-anchor="start" x="106.5168" y="-156.7544" font-family="Times,serif" font-size="8.00" fill="#000000">2 p </text>
<text text-anchor="start" x="241.1715" y="-235.2798" font-family="Times,serif" font-size="8.00" fill="#000000">2 2 </text>
</g>
<!-- 15229993427491404172&#45;&#45;6625773108833308772 -->
<g id="edge2" class="edge">
<title>15229993427491404172&#45;&#45;6625773108833308772</title>
<path fill="none" stroke="#000000" d="M326.8209,-232.4168C366.9985,-210.7542 431.3426,-176.0618 471.4223,-154.452"/>
<text text-anchor="start" x="406.5" y="-191.6" font-family="Times,serif" font-size="8.00" fill="#ff0000">θ</text>
<text text-anchor="start" x="458.4223" y="-157.052" font-family="Times,serif" font-size="8.00" fill="#000000">2 p </text>
<text text-anchor="start" x="326.8209" y="-235.0168" font-family="Times,serif" font-size="8.00" fill="#000000">3 3 </text>
</g>
</g>
</svg>



The only difference with the previous graph is the fact that the node called "prior" is a 'Dir', short for Dirichlet, and that the two nodes called "likelihood1" and "likelihood2" are 'Cat' types, short for Categorical. Let's look at the message passing schedule:


```julia
# Indicate which variables you want posteriors for
q = PosteriorFactorization(θ, ids=[:θ])

# Generate a message passing inference algorithm
algorithm = messagePassingAlgorithm(θ, q)

# Compile algorithm code
source_code = algorithmSourceCode(algorithm)

# Bring compiled code into current scope
eval(Meta.parse(source_code))

# Visualize message passing schedule
pfθ = q.posterior_factors[:θ]
ForneyLab.draw(pfθ, schedule=pfθ.schedule);
```


<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<!DOCTYPE svg PUBLIC "-//W3C//DTD SVG 1.1//EN"
 "http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd">
<!-- Generated by graphviz version 2.40.1 (20161225.0304)
 -->
<!-- Title: G Pages: 1 -->
<svg width="548pt" height="245pt"
 viewBox="0.00 0.00 548.20 245.00" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink">
<g id="graph0" class="graph" transform="scale(1 1) rotate(0) translate(4 241)">
<title>G</title>
<polygon fill="#ffffff" stroke="transparent" points="-4,4 -4,-241 544.2025,-241 544.2025,4 -4,4"/>
<!-- 6707493110187205485 -->
<g id="node1" class="node">
<title>6707493110187205485</title>
<polygon fill="none" stroke="#000000" points="351.2025,-120 279.2025,-120 279.2025,-48 351.2025,-48 351.2025,-120"/>
<text text-anchor="middle" x="315.2025" y="-86.8" font-family="Times,serif" font-size="9.00" fill="#000000">Dir</text>
<text text-anchor="middle" x="315.2025" y="-76.8" font-family="Times,serif" font-size="9.00" fill="#000000">f_a</text>
</g>
<!-- 8595486343092272081 -->
<g id="node6" class="node">
<title>8595486343092272081</title>
</g>
<!-- 6707493110187205485&#45;&#45;8595486343092272081 -->
<g id="edge4" class="edge">
<title>6707493110187205485&#45;&#45;8595486343092272081</title>
<path fill="none" stroke="#000000" stroke-dasharray="5,2" d="M315.2025,-47.9665C315.2025,-33.779 315.2025,-19.0273 315.2025,-11.1929"/>
<text text-anchor="middle" x="307.2025" y="-13.7929" font-family="Times,serif" font-size="8.00" fill="#000000">1 out</text>
<text text-anchor="middle" x="310.2025" y="-41.5665" font-family="Times,serif" font-size="8.00" fill="#000000">2 a</text>
</g>
<!-- 11615063546771903839 -->
<g id="node2" class="node">
<title>11615063546771903839</title>
<polygon fill="none" stroke="#000000" points="540.2025,-120 468.2025,-120 468.2025,-48 540.2025,-48 540.2025,-120"/>
<text text-anchor="middle" x="504.2025" y="-86.8" font-family="Times,serif" font-size="9.00" fill="#000000">Cat</text>
<text text-anchor="middle" x="504.2025" y="-76.8" font-family="Times,serif" font-size="9.00" fill="#000000">f_b</text>
</g>
<!-- 6625773108833308772 -->
<g id="node3" class="node">
<title>6625773108833308772</title>
<polygon fill="none" stroke="#000000" points="107.2025,-120 35.2025,-120 35.2025,-48 107.2025,-48 107.2025,-120"/>
<text text-anchor="middle" x="71.2025" y="-86.8" font-family="Times,serif" font-size="9.00" fill="#000000">Cat</text>
<text text-anchor="middle" x="71.2025" y="-76.8" font-family="Times,serif" font-size="9.00" fill="#000000">f_c</text>
</g>
<!-- 15229993427491404172 -->
<g id="node4" class="node">
<title>15229993427491404172</title>
<polygon fill="none" stroke="#000000" points="351.2025,-237 279.2025,-237 279.2025,-165 351.2025,-165 351.2025,-237"/>
<text text-anchor="middle" x="315.2025" y="-203.8" font-family="Times,serif" font-size="9.00" fill="#000000">=</text>
<text text-anchor="middle" x="315.2025" y="-193.8" font-family="Times,serif" font-size="9.00" fill="#000000">equ_θ_1</text>
</g>
<!-- 15229993427491404172&#45;&#45;6707493110187205485 -->
<g id="edge3" class="edge">
<title>15229993427491404172&#45;&#45;6707493110187205485</title>
<path fill="none" stroke="#000000" d="M315.2025,-164.9415C315.2025,-150.7071 315.2025,-134.3833 315.2025,-120.1409"/>
<text text-anchor="start" x="315.2025" y="-140.6" font-family="Times,serif" font-size="8.00" fill="#ff0000">θ</text>
<text text-anchor="start" x="280.2025" y="-123.7409" font-family="Times,serif" font-size="8.00" fill="#000000">1 out </text>
<text text-anchor="start" x="299.2025" y="-123.7409" font-family="Times,serif" font-weight="bold" font-size="8.00" fill="#0000ff">((1))</text>
<text text-anchor="start" x="292.2025" y="-159.5415" font-family="Times,serif" font-size="8.00" fill="#000000">1 1 </text>
<text text-anchor="start" x="305.2025" y="-159.5415" font-family="Times,serif" font-weight="bold" font-size="8.00" fill="#0000ff">(4)</text>
</g>
<!-- 15229993427491404172&#45;&#45;11615063546771903839 -->
<g id="edge2" class="edge">
<title>15229993427491404172&#45;&#45;11615063546771903839</title>
<path fill="none" stroke="#000000" d="M351.6436,-178.4412C385.2185,-157.6568 434.7575,-126.9898 468.1943,-106.2908"/>
<text text-anchor="start" x="415.2025" y="-140.6" font-family="Times,serif" font-size="8.00" fill="#ff0000">θ</text>
<text text-anchor="start" x="439.1943" y="-109.8908" font-family="Times,serif" font-size="8.00" fill="#000000">2 p </text>
<text text-anchor="start" x="452.1943" y="-109.8908" font-family="Times,serif" font-weight="bold" font-size="8.00" fill="#0000ff">((2))</text>
<text text-anchor="start" x="351.6436" y="-181.0412" font-family="Times,serif" font-size="8.00" fill="#000000">2 2 </text>
</g>
<!-- 15229993427491404172&#45;&#45;6625773108833308772 -->
<g id="edge1" class="edge">
<title>15229993427491404172&#45;&#45;6625773108833308772</title>
<path fill="none" stroke="#000000" d="M279.1995,-183.7362C233.0673,-161.6155 153.5566,-123.4895 107.3533,-101.3346"/>
<text text-anchor="start" x="201.2025" y="-140.6" font-family="Times,serif" font-size="8.00" fill="#ff0000">θ</text>
<text text-anchor="start" x="107.3533" y="-104.9346" font-family="Times,serif" font-size="8.00" fill="#000000">2 p </text>
<text text-anchor="start" x="120.3533" y="-104.9346" font-family="Times,serif" font-weight="bold" font-size="8.00" fill="#0000ff">((3))</text>
<text text-anchor="start" x="266.1995" y="-186.3362" font-family="Times,serif" font-size="8.00" fill="#000000">3 3 </text>
</g>
<!-- 1847487353024847686 -->
<g id="node5" class="node">
<title>1847487353024847686</title>
</g>
<!-- 1847487353024847686&#45;&#45;6625773108833308772 -->
<g id="edge5" class="edge">
<title>1847487353024847686&#45;&#45;6625773108833308772</title>
<path fill="none" stroke="#000000" stroke-dasharray="5,2" d="M16,-195.3568C22.6527,-181.9367 39.867,-147.2114 53.3199,-120.0735"/>
<text text-anchor="middle" x="45.3199" y="-122.6735" font-family="Times,serif" font-size="8.00" fill="#000000">1 out</text>
<text text-anchor="middle" x="8" y="-188.9568" font-family="Times,serif" font-size="8.00" fill="#000000">1 out</text>
</g>
<!-- 8122598982465519112 -->
<g id="node7" class="node">
<title>8122598982465519112</title>
</g>
<!-- 8122598982465519112&#45;&#45;11615063546771903839 -->
<g id="edge6" class="edge">
<title>8122598982465519112&#45;&#45;11615063546771903839</title>
<path fill="none" stroke="#000000" stroke-dasharray="5,2" d="M506.1061,-195.3568C505.8767,-181.9367 505.2831,-147.2114 504.8192,-120.0735"/>
<text text-anchor="middle" x="496.8192" y="-122.6735" font-family="Times,serif" font-size="8.00" fill="#000000">1 out</text>
<text text-anchor="middle" x="498.1061" y="-188.9568" font-family="Times,serif" font-size="8.00" fill="#000000">1 out</text>
</g>
</g>
</svg>



That's the same as before as well: 2 messages from the likelihoods, 1 combined likelihood message from the equality node and 1 message from the prior.

If we now setup the message passing procedure, we have to be a little bit more careful. We cannot feed the scores $\{ 0,1,2\}$ as outcomes directly. We have to encode them in one-hot vectors (see Bert's lecture notes on discrete distributions). Suppose you had a score of $1$ for the first question and a score of $2$ for the second one. That translates into a vector $[0, 1, 0]$ and $[0, 0, 1]$, respectively. These we enter into the `data` dictionary:


```julia
# Initialize a message data structure
messages = Array{Message}(undef, 4)

# Initalize marginal distributions data structure
marginals = Dict()

# Enter the observed outcomes in the placeholders
data = Dict(:X1 => [0, 1, 0],
            :X2 => [0, 0, 1])

# Update coefficients
stepθ!(data, marginals, messages);

# Print messages
print("\nMessage ((1)) = "*string(messages[1].dist))
print("Message ((2)) = "*string(messages[2].dist))
print("Message ((3)) = "*string(messages[3].dist))
print("Message ((4)) = "*string(messages[4].dist))
println("Marginal of θ = "*string(marginals[:θ]))
```

    
    Message ((1)) = Dir(a=[1.00, 3.00, 2.00])
    Message ((2)) = Dir(a=[1.00, 2.00, 1.00])
    Message ((3)) = Dir(a=[1.00, 1.00, 2.00])
    Message ((4)) = Dir(a=[1.00, 2.00, 2.00])
    Marginal of θ = Dir(a=[1.00, 4.00, 3.00])
    


Visualizing a Dirichlet distribution is a bit tricky. In the special case of $3$ parameters, we can plot the probabilities on a simplex. As a reminder, a [simplex](https://en.wikipedia.org/wiki/Simplex) in 3-dimensions is the triangle between the coordinates $[0,0,1]$, $[0,1,0]$ and $[1,0,0]$:

<p style="text-align:center;"></p>

Each vector $\theta$ is a point on that triangle and its elements sum to $1$. Since the triangle is 2-dimensional, we can plot the Dirichlet's probability density over it.


```julia
# Import matplotlib 
plt = pyimport("matplotlib.pyplot")

# Include helper function
include("../scripts/dirichlet_simplex.jl")

# Extract parameters of Message ((1))
α1 = messages[1].dist.params[:a]

# Compute pdf contour lines on the simplex
trimesh, pvals = pdf_contours_simplex(α1)

# Plot using matplotlib's tricontour
plt.tricontourf(trimesh, pvals, nlevels=200, cmap="jet");
plt.title("Message ((1)) = "*string(messages[1].dist));
```

Code notes:
- `pyimport` allows you to import Python modules. 
- When you `include()` another julia file, it is as if you wrote it at that point in your script.
- `tricontourf` is a function from Matplolib, where you create a contour plot over a triangulated mesh (here, the simplex).

----

The red spot is the area of high probability, with the contours around indicating increasing uncertainty. The prior, with concentration parameters $[1, 3, 2]$, reflects the belief that applicants are least likely to get the question completely wrong ($\alpha_1$ = 1, score = 0), most likely to get the question partly right ($\alpha_2$ = 3, score = 1) and moderately likely to get the question completely correct ($\alpha_3$ = 2, score = 2).


```julia
# Extract parameters 
α4 = messages[4].dist.params[:a]

# Compute pdf contour lines on the simplex
trimesh, pvals = pdf_contours_simplex(α4)

# Plot using matplotlib's tricontour
plt.tricontourf(trimesh, pvals, nlevels=200, cmap="jet")
plt.title("Message ((4)) = "*string(messages[4].dist));
```

Since we got scores $X_1 = 1$ and $X_2 = 2$, the combined message from both likelihoods has concentration parameters $[1,2,2]$. 


```julia
# Extract parameters 
α_marg = marginals[:θ].params[:a]

# Compute pdf contour lines on the simplex
trimesh, pvals = pdf_contours_simplex(α_marg)

# Plot using matplotlib's tricontour
plt.tricontourf(trimesh, pvals, nlevels=200, cmap="jet")
plt.title("Marginal of θ = "*string(marginals[:θ]));
```

The posterior is the combination of Messages ((1)) and ((4)) and focuses much more strongly in the area where the two messages overlap.

---

#### $\ast$ **Try for yourself**

Play around with the prior parameters and your responses to the questions. See how they change your posterior.

---

### 3. Rating scale

You might want to evaluate someone by an even finer metric. For example, in oral exams you need to provide a score based on a conversation which is hard to quantify. You could do this by taking away the discrete set of responses and replacing it with a continuous response variable. For example, rating scales are forms of continuous response models. You would mark the applicant's performance on a question as a cross on a line:

<p style="text-align:center;"></p>

It is still the case that there is some underlying level of skill, that we'll call $\theta$, and that the performance on each question is a noisy measurement of that skill, that we'll call $X$. We argue that performance noise is symmetric: the probability of performing a little better than their skill level is equal to performing a little worse. We will therefore use Gaussian, or Normal, likelihood functions: $p(X \mid \theta) = \text{Normal}(X \mid \theta, \sigma^2)$. The conjugate prior to the mean in Gaussian likelihoods is another Gaussian distribution: $p(\theta) = \text{Normal}(\theta \mid 60, 20)$. Say that we rate performance on a scale from $0$ to $100$, then it makes sense to use a mean of $60$ and a variance of $20$ for the prior.

We'll keep the same generative model as before, with new definitions for each distribution:

$$\begin{align}
p(X_1 \mid \theta) =&\ \text{Normal}(X_1 \mid \theta, 10) \\
p(X_2 \mid \theta) =&\ \text{Normal}(X_2 \mid \theta, 15) \\
p(\theta) =&\ \text{Normal}(\theta \mid 60, 20)
\end{align}$$

The factor graph will again be the same, but with different parameterizations of factor nodes:


```julia
# Start building a model
factor_graph4 = FactorGraph()

# Add the prior
@RV θ ~ GaussianMeanVariance(60, 20, id=:f_a)

# Add question 1 likelihood
@RV X1 ~ GaussianMeanVariance(θ, 10, id=:f_b)

# Add question 2 likelihood
@RV X2 ~ GaussianMeanVariance(θ, 15, id=:f_c)

# Outcomes are going to be observed
placeholder(X1, :X1)
placeholder(X2, :X2)

# Visualize the graph
ForneyLab.draw(factor_graph4)
```


<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<!DOCTYPE svg PUBLIC "-//W3C//DTD SVG 1.1//EN"
 "http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd">
<!-- Generated by graphviz version 2.40.1 (20161225.0304)
 -->
<!-- Title: G Pages: 1 -->
<svg width="585pt" height="296pt"
 viewBox="0.00 0.00 585.00 296.00" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink">
<g id="graph0" class="graph" transform="scale(1 1) rotate(0) translate(4 292)">
<title>G</title>
<polygon fill="#ffffff" stroke="transparent" points="-4,4 -4,-292 581,-292 581,4 -4,4"/>
<!-- 14551220858376551605 -->
<g id="node1" class="node">
<title>14551220858376551605</title>
<polygon fill="none" stroke="#000000" points="328.5,-288 256.5,-288 256.5,-216 328.5,-216 328.5,-288"/>
<text text-anchor="middle" x="292.5" y="-254.8" font-family="Times,serif" font-size="9.00" fill="#000000">=</text>
<text text-anchor="middle" x="292.5" y="-244.8" font-family="Times,serif" font-size="9.00" fill="#000000">equ_θ_1</text>
</g>
<!-- 9139797860911220071 -->
<g id="node3" class="node">
<title>9139797860911220071</title>
<polygon fill="none" stroke="#000000" points="72.5,-171 .5,-171 .5,-99 72.5,-99 72.5,-171"/>
<text text-anchor="middle" x="36.5" y="-137.8" font-family="Times,serif" font-size="9.00" fill="#000000">𝒩</text>
<text text-anchor="middle" x="36.5" y="-127.8" font-family="Times,serif" font-size="9.00" fill="#000000">f_b</text>
</g>
<!-- 14551220858376551605&#45;&#45;9139797860911220071 -->
<g id="edge9" class="edge">
<title>14551220858376551605&#45;&#45;9139797860911220071</title>
<path fill="none" stroke="#000000" d="M256.3644,-235.4849C207.7121,-213.2493 121.6352,-173.9094 72.8653,-151.6201"/>
<text text-anchor="start" x="172.5" y="-191.6" font-family="Times,serif" font-size="8.00" fill="#ff0000">θ</text>
<text text-anchor="start" x="72.8653" y="-154.2201" font-family="Times,serif" font-size="8.00" fill="#000000">2 m </text>
<text text-anchor="start" x="243.3644" y="-238.0849" font-family="Times,serif" font-size="8.00" fill="#000000">2 2 </text>
</g>
<!-- 12822370800796287258 -->
<g id="node5" class="node">
<title>12822370800796287258</title>
<polygon fill="none" stroke="#000000" points="576.5,-171 504.5,-171 504.5,-99 576.5,-99 576.5,-171"/>
<text text-anchor="middle" x="540.5" y="-137.8" font-family="Times,serif" font-size="9.00" fill="#000000">𝒩</text>
<text text-anchor="middle" x="540.5" y="-127.8" font-family="Times,serif" font-size="9.00" fill="#000000">f_c</text>
</g>
<!-- 14551220858376551605&#45;&#45;12822370800796287258 -->
<g id="edge6" class="edge">
<title>14551220858376551605&#45;&#45;12822370800796287258</title>
<path fill="none" stroke="#000000" d="M328.5613,-234.9872C375.5804,-212.8048 457.3546,-174.2258 504.3951,-152.0333"/>
<text text-anchor="start" x="423.5" y="-191.6" font-family="Times,serif" font-size="8.00" fill="#ff0000">θ</text>
<text text-anchor="start" x="489.3951" y="-154.6333" font-family="Times,serif" font-size="8.00" fill="#000000">2 m </text>
<text text-anchor="start" x="328.5613" y="-237.5872" font-family="Times,serif" font-size="8.00" fill="#000000">3 3 </text>
</g>
<!-- 3415132977026660945 -->
<g id="node10" class="node">
<title>3415132977026660945</title>
<polygon fill="none" stroke="#000000" points="328.5,-171 256.5,-171 256.5,-99 328.5,-99 328.5,-171"/>
<text text-anchor="middle" x="292.5" y="-137.8" font-family="Times,serif" font-size="9.00" fill="#000000">𝒩</text>
<text text-anchor="middle" x="292.5" y="-127.8" font-family="Times,serif" font-size="9.00" fill="#000000">f_a</text>
</g>
<!-- 14551220858376551605&#45;&#45;3415132977026660945 -->
<g id="edge8" class="edge">
<title>14551220858376551605&#45;&#45;3415132977026660945</title>
<path fill="none" stroke="#000000" d="M292.5,-215.9415C292.5,-201.7071 292.5,-185.3833 292.5,-171.1409"/>
<text text-anchor="start" x="292.5" y="-191.6" font-family="Times,serif" font-size="8.00" fill="#ff0000">θ</text>
<text text-anchor="start" x="273.5" y="-173.7409" font-family="Times,serif" font-size="8.00" fill="#000000">1 out </text>
<text text-anchor="start" x="279.5" y="-209.5415" font-family="Times,serif" font-size="8.00" fill="#000000">1 1 </text>
</g>
<!-- 9375574533915942436 -->
<g id="node2" class="node">
<title>9375574533915942436</title>
<polygon fill="#d3d3d3" stroke="#000000" points="577,-279 504,-279 504,-225 577,-225 577,-279"/>
<text text-anchor="middle" x="540.5" y="-249.8" font-family="Times,serif" font-size="9.00" fill="#000000">placeholder_X2</text>
</g>
<!-- 9375574533915942436&#45;&#45;12822370800796287258 -->
<g id="edge1" class="edge">
<title>9375574533915942436&#45;&#45;12822370800796287258</title>
<path fill="none" stroke="#000000" d="M540.5,-224.8638C540.5,-208.9324 540.5,-188.595 540.5,-171.2982"/>
<text text-anchor="start" x="540.5" y="-191.6" font-family="Times,serif" font-size="8.00" fill="#ff0000">X2</text>
<text text-anchor="start" x="521.5" y="-173.8982" font-family="Times,serif" font-size="8.00" fill="#000000">1 out </text>
<text text-anchor="start" x="521.5" y="-218.4638" font-family="Times,serif" font-size="8.00" fill="#000000">1 out </text>
</g>
<!-- 2601218693199014220 -->
<g id="node9" class="node">
<title>2601218693199014220</title>
<polygon fill="#d3d3d3" stroke="#000000" points="61.5,-54 7.5,-54 7.5,0 61.5,0 61.5,-54"/>
<text text-anchor="middle" x="34.5" y="-24.8" font-family="Times,serif" font-size="9.00" fill="#000000">clamp_3</text>
</g>
<!-- 9139797860911220071&#45;&#45;2601218693199014220 -->
<g id="edge4" class="edge">
<title>9139797860911220071&#45;&#45;2601218693199014220</title>
<path fill="none" stroke="#000000" d="M33.1319,-98.7093C32.6151,-89.969 32.3077,-80.6559 32.5,-72 32.6285,-66.2145 32.8498,-60.0136 33.0996,-54.121"/>
<text text-anchor="start" x="32.5" y="-74.6" font-family="Times,serif" font-size="8.00" fill="#ff0000">clamp_3</text>
<text text-anchor="start" x="14.0996" y="-56.721" font-family="Times,serif" font-size="8.00" fill="#000000">1 out </text>
<text text-anchor="start" x="20.1319" y="-92.3093" font-family="Times,serif" font-size="8.00" fill="#000000">3 v </text>
</g>
<!-- 11555386723890532146 -->
<g id="node4" class="node">
<title>11555386723890532146</title>
<polygon fill="#d3d3d3" stroke="#000000" points="568.5,-54 514.5,-54 514.5,0 568.5,0 568.5,-54"/>
<text text-anchor="middle" x="541.5" y="-24.8" font-family="Times,serif" font-size="9.00" fill="#000000">clamp_4</text>
</g>
<!-- 12822370800796287258&#45;&#45;11555386723890532146 -->
<g id="edge7" class="edge">
<title>12822370800796287258&#45;&#45;11555386723890532146</title>
<path fill="none" stroke="#000000" d="M540.8358,-98.7371C540.9708,-84.1526 541.1238,-67.6262 541.2489,-54.1146"/>
<text text-anchor="start" x="540.5" y="-74.6" font-family="Times,serif" font-size="8.00" fill="#ff0000">clamp_4</text>
<text text-anchor="start" x="522.2489" y="-56.7146" font-family="Times,serif" font-size="8.00" fill="#000000">1 out </text>
<text text-anchor="start" x="527.8358" y="-92.3371" font-family="Times,serif" font-size="8.00" fill="#000000">3 v </text>
</g>
<!-- 15684075464608502923 -->
<g id="node6" class="node">
<title>15684075464608502923</title>
<polygon fill="#d3d3d3" stroke="#000000" points="399.5,-54 345.5,-54 345.5,0 399.5,0 399.5,-54"/>
<text text-anchor="middle" x="372.5" y="-24.8" font-family="Times,serif" font-size="9.00" fill="#000000">clamp_2</text>
</g>
<!-- 8809929690116452698 -->
<g id="node7" class="node">
<title>8809929690116452698</title>
<polygon fill="#d3d3d3" stroke="#000000" points="73,-279 0,-279 0,-225 73,-225 73,-279"/>
<text text-anchor="middle" x="36.5" y="-249.8" font-family="Times,serif" font-size="9.00" fill="#000000">placeholder_X1</text>
</g>
<!-- 8809929690116452698&#45;&#45;9139797860911220071 -->
<g id="edge2" class="edge">
<title>8809929690116452698&#45;&#45;9139797860911220071</title>
<path fill="none" stroke="#000000" d="M36.5,-224.8638C36.5,-208.9324 36.5,-188.595 36.5,-171.2982"/>
<text text-anchor="start" x="36.5" y="-191.6" font-family="Times,serif" font-size="8.00" fill="#ff0000">X1</text>
<text text-anchor="start" x="17.5" y="-173.8982" font-family="Times,serif" font-size="8.00" fill="#000000">1 out </text>
<text text-anchor="start" x="17.5" y="-218.4638" font-family="Times,serif" font-size="8.00" fill="#000000">1 out </text>
</g>
<!-- 5022865463634313932 -->
<g id="node8" class="node">
<title>5022865463634313932</title>
<polygon fill="#d3d3d3" stroke="#000000" points="230.5,-54 176.5,-54 176.5,0 230.5,0 230.5,-54"/>
<text text-anchor="middle" x="203.5" y="-24.8" font-family="Times,serif" font-size="9.00" fill="#000000">clamp_1</text>
</g>
<!-- 3415132977026660945&#45;&#45;15684075464608502923 -->
<g id="edge5" class="edge">
<title>3415132977026660945&#45;&#45;15684075464608502923</title>
<path fill="none" stroke="#000000" d="M319.3614,-98.7371C330.1648,-84.1526 342.4065,-67.6262 352.4151,-54.1146"/>
<text text-anchor="start" x="337.5" y="-74.6" font-family="Times,serif" font-size="8.00" fill="#ff0000">clamp_2</text>
<text text-anchor="start" x="333.4151" y="-56.7146" font-family="Times,serif" font-size="8.00" fill="#000000">1 out </text>
<text text-anchor="start" x="306.3614" y="-92.3371" font-family="Times,serif" font-size="8.00" fill="#000000">3 v </text>
</g>
<!-- 3415132977026660945&#45;&#45;5022865463634313932 -->
<g id="edge3" class="edge">
<title>3415132977026660945&#45;&#45;5022865463634313932</title>
<path fill="none" stroke="#000000" d="M262.6167,-98.7371C250.598,-84.1526 236.979,-67.6262 225.8445,-54.1146"/>
<text text-anchor="start" x="248.5" y="-74.6" font-family="Times,serif" font-size="8.00" fill="#ff0000">clamp_1</text>
<text text-anchor="start" x="206.8445" y="-56.7146" font-family="Times,serif" font-size="8.00" fill="#000000">1 out </text>
<text text-anchor="start" x="247.6167" y="-92.3371" font-family="Times,serif" font-size="8.00" fill="#000000">2 m </text>
</g>
</g>
</svg>




```julia
# Indicate which variables you want posteriors for
q = PosteriorFactorization(θ, ids=[:θ])

# Generate a message passing inference algorithm
algorithm = messagePassingAlgorithm(θ, q)

# Compile algorithm code
source_code = algorithmSourceCode(algorithm)

# Bring compiled code into current scope
eval(Meta.parse(source_code))

# Visualize message passing schedule
pfθ = q.posterior_factors[:θ]
ForneyLab.draw(pfθ, schedule=pfθ.schedule);
```


<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<!DOCTYPE svg PUBLIC "-//W3C//DTD SVG 1.1//EN"
 "http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd">
<!-- Generated by graphviz version 2.40.1 (20161225.0304)
 -->
<!-- Title: G Pages: 1 -->
<svg width="550pt" height="245pt"
 viewBox="0.00 0.00 549.55 245.00" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink">
<g id="graph0" class="graph" transform="scale(1 1) rotate(0) translate(4 241)">
<title>G</title>
<polygon fill="#ffffff" stroke="transparent" points="-4,4 -4,-241 545.553,-241 545.553,4 -4,4"/>
<!-- 14551220858376551605 -->
<g id="node1" class="node">
<title>14551220858376551605</title>
<polygon fill="none" stroke="#000000" points="352.553,-237 280.553,-237 280.553,-165 352.553,-165 352.553,-237"/>
<text text-anchor="middle" x="316.553" y="-203.8" font-family="Times,serif" font-size="9.00" fill="#000000">=</text>
<text text-anchor="middle" x="316.553" y="-193.8" font-family="Times,serif" font-size="9.00" fill="#000000">equ_θ_1</text>
</g>
<!-- 9139797860911220071 -->
<g id="node2" class="node">
<title>9139797860911220071</title>
<polygon fill="none" stroke="#000000" points="80.553,-120 8.553,-120 8.553,-48 80.553,-48 80.553,-120"/>
<text text-anchor="middle" x="44.553" y="-86.8" font-family="Times,serif" font-size="9.00" fill="#000000">𝒩</text>
<text text-anchor="middle" x="44.553" y="-76.8" font-family="Times,serif" font-size="9.00" fill="#000000">f_b</text>
</g>
<!-- 14551220858376551605&#45;&#45;9139797860911220071 -->
<g id="edge3" class="edge">
<title>14551220858376551605&#45;&#45;9139797860911220071</title>
<path fill="none" stroke="#000000" d="M280.4335,-185.4633C228.4098,-163.0854 132.9287,-122.0145 80.8219,-99.601"/>
<text text-anchor="start" x="189.553" y="-140.6" font-family="Times,serif" font-size="8.00" fill="#ff0000">θ</text>
<text text-anchor="start" x="80.8219" y="-103.201" font-family="Times,serif" font-size="8.00" fill="#000000">2 m </text>
<text text-anchor="start" x="95.8219" y="-103.201" font-family="Times,serif" font-weight="bold" font-size="8.00" fill="#0000ff">((2))</text>
<text text-anchor="start" x="267.4335" y="-188.0633" font-family="Times,serif" font-size="8.00" fill="#000000">2 2 </text>
</g>
<!-- 12822370800796287258 -->
<g id="node3" class="node">
<title>12822370800796287258</title>
<polygon fill="none" stroke="#000000" points="541.553,-120 469.553,-120 469.553,-48 541.553,-48 541.553,-120"/>
<text text-anchor="middle" x="505.553" y="-86.8" font-family="Times,serif" font-size="9.00" fill="#000000">𝒩</text>
<text text-anchor="middle" x="505.553" y="-76.8" font-family="Times,serif" font-size="9.00" fill="#000000">f_c</text>
</g>
<!-- 14551220858376551605&#45;&#45;12822370800796287258 -->
<g id="edge1" class="edge">
<title>14551220858376551605&#45;&#45;12822370800796287258</title>
<path fill="none" stroke="#000000" d="M352.9941,-178.4412C386.569,-157.6568 436.108,-126.9898 469.5448,-106.2908"/>
<text text-anchor="start" x="416.553" y="-140.6" font-family="Times,serif" font-size="8.00" fill="#ff0000">θ</text>
<text text-anchor="start" x="438.5448" y="-109.8908" font-family="Times,serif" font-size="8.00" fill="#000000">2 m </text>
<text text-anchor="start" x="453.5448" y="-109.8908" font-family="Times,serif" font-weight="bold" font-size="8.00" fill="#0000ff">((3))</text>
<text text-anchor="start" x="352.9941" y="-181.0412" font-family="Times,serif" font-size="8.00" fill="#000000">3 3 </text>
</g>
<!-- 3415132977026660945 -->
<g id="node4" class="node">
<title>3415132977026660945</title>
<polygon fill="none" stroke="#000000" points="352.553,-120 280.553,-120 280.553,-48 352.553,-48 352.553,-120"/>
<text text-anchor="middle" x="316.553" y="-86.8" font-family="Times,serif" font-size="9.00" fill="#000000">𝒩</text>
<text text-anchor="middle" x="316.553" y="-76.8" font-family="Times,serif" font-size="9.00" fill="#000000">f_a</text>
</g>
<!-- 14551220858376551605&#45;&#45;3415132977026660945 -->
<g id="edge2" class="edge">
<title>14551220858376551605&#45;&#45;3415132977026660945</title>
<path fill="none" stroke="#000000" d="M316.553,-164.9415C316.553,-150.7071 316.553,-134.3833 316.553,-120.1409"/>
<text text-anchor="start" x="316.553" y="-140.6" font-family="Times,serif" font-size="8.00" fill="#ff0000">θ</text>
<text text-anchor="start" x="281.553" y="-123.7409" font-family="Times,serif" font-size="8.00" fill="#000000">1 out </text>
<text text-anchor="start" x="300.553" y="-123.7409" font-family="Times,serif" font-weight="bold" font-size="8.00" fill="#0000ff">((1))</text>
<text text-anchor="start" x="293.553" y="-159.5415" font-family="Times,serif" font-size="8.00" fill="#000000">1 1 </text>
<text text-anchor="start" x="306.553" y="-159.5415" font-family="Times,serif" font-weight="bold" font-size="8.00" fill="#0000ff">(4)</text>
</g>
<!-- 2601218693199014220 -->
<g id="node9" class="node">
<title>2601218693199014220</title>
</g>
<!-- 9139797860911220071&#45;&#45;2601218693199014220 -->
<g id="edge7" class="edge">
<title>9139797860911220071&#45;&#45;2601218693199014220</title>
<path fill="none" stroke="#000000" stroke-dasharray="5,2" d="M44.553,-47.9665C44.553,-33.779 44.553,-19.0273 44.553,-11.1929"/>
<text text-anchor="middle" x="36.553" y="-13.7929" font-family="Times,serif" font-size="8.00" fill="#000000">1 out</text>
<text text-anchor="middle" x="39.553" y="-41.5665" font-family="Times,serif" font-size="8.00" fill="#000000">3 v</text>
</g>
<!-- 11555386723890532146 -->
<g id="node6" class="node">
<title>11555386723890532146</title>
</g>
<!-- 12822370800796287258&#45;&#45;11555386723890532146 -->
<g id="edge9" class="edge">
<title>12822370800796287258&#45;&#45;11555386723890532146</title>
<path fill="none" stroke="#000000" stroke-dasharray="5,2" d="M505.553,-47.9665C505.553,-33.779 505.553,-19.0273 505.553,-11.1929"/>
<text text-anchor="middle" x="497.553" y="-13.7929" font-family="Times,serif" font-size="8.00" fill="#000000">1 out</text>
<text text-anchor="middle" x="500.553" y="-41.5665" font-family="Times,serif" font-size="8.00" fill="#000000">3 v</text>
</g>
<!-- 5022865463634313932 -->
<g id="node8" class="node">
<title>5022865463634313932</title>
</g>
<!-- 3415132977026660945&#45;&#45;5022865463634313932 -->
<g id="edge6" class="edge">
<title>3415132977026660945&#45;&#45;5022865463634313932</title>
<path fill="none" stroke="#000000" stroke-dasharray="5,2" d="M332.6189,-47.9665C338.9446,-33.779 345.5218,-19.0273 349.0148,-11.1929"/>
<text text-anchor="middle" x="341.0148" y="-13.7929" font-family="Times,serif" font-size="8.00" fill="#000000">1 out</text>
<text text-anchor="middle" x="326.1189" y="-41.5665" font-family="Times,serif" font-size="8.00" fill="#000000">2 m</text>
</g>
<!-- 15684075464608502923 -->
<g id="node10" class="node">
<title>15684075464608502923</title>
</g>
<!-- 3415132977026660945&#45;&#45;15684075464608502923 -->
<g id="edge8" class="edge">
<title>3415132977026660945&#45;&#45;15684075464608502923</title>
<path fill="none" stroke="#000000" stroke-dasharray="5,2" d="M280.326,-52.7492C261.6897,-36.6729 240.8861,-18.7268 231.149,-10.3273"/>
<text text-anchor="middle" x="223.149" y="-12.9273" font-family="Times,serif" font-size="8.00" fill="#000000">1 out</text>
<text text-anchor="middle" x="275.326" y="-55.3492" font-family="Times,serif" font-size="8.00" fill="#000000">3 v</text>
</g>
<!-- 9375574533915942436 -->
<g id="node5" class="node">
<title>9375574533915942436</title>
</g>
<!-- 9375574533915942436&#45;&#45;12822370800796287258 -->
<g id="edge4" class="edge">
<title>9375574533915942436&#45;&#45;12822370800796287258</title>
<path fill="none" stroke="#000000" stroke-dasharray="5,2" d="M507.4566,-195.3568C507.2272,-181.9367 506.6336,-147.2114 506.1697,-120.0735"/>
<text text-anchor="middle" x="498.1697" y="-122.6735" font-family="Times,serif" font-size="8.00" fill="#000000">1 out</text>
<text text-anchor="middle" x="499.4566" y="-188.9568" font-family="Times,serif" font-size="8.00" fill="#000000">1 out</text>
</g>
<!-- 8809929690116452698 -->
<g id="node7" class="node">
<title>8809929690116452698</title>
</g>
<!-- 8809929690116452698&#45;&#45;9139797860911220071 -->
<g id="edge5" class="edge">
<title>8809929690116452698&#45;&#45;9139797860911220071</title>
<path fill="none" stroke="#000000" stroke-dasharray="5,2" d="M16,-195.3568C19.4411,-181.9367 28.345,-147.2114 35.3034,-120.0735"/>
<text text-anchor="middle" x="27.3034" y="-122.6735" font-family="Times,serif" font-size="8.00" fill="#000000">1 out</text>
<text text-anchor="middle" x="8" y="-188.9568" font-family="Times,serif" font-size="8.00" fill="#000000">1 out</text>
</g>
</g>
</svg>



The message passing schedule is still exactly the same.


```julia
# Initialize a message data structure
messages = Array{Message}(undef, 4)

# Initalize marginal distributions data structure
marginals = Dict()

# Enter the scores in the data dictionary
data = Dict(:X1 => 61.5,
            :X2 => 72)

# Update coefficients
stepθ!(data, marginals, messages);

# Print messages
print("\nMessage ((1)) = "*string(messages[1].dist))
print("Message ((2)) = "*string(messages[2].dist))
print("Message ((3)) = "*string(messages[3].dist))
print("Message ((4)) = "*string(messages[4].dist))
print("Marginal of θ = "*string(marginals[:θ]))
```

    
    Message ((1)) = 𝒩(m=60, v=20)
    Message ((2)) = 𝒩(m=61.50, v=10)
    Message ((3)) = 𝒩(m=72, v=15)
    Message ((4)) = 𝒩(xi=10.95, w=0.17)
    Marginal of θ = 𝒩(xi=13.95, w=0.22)


Message ((4)) has a somewhat unusual form in that uses `xi` as a parameter instead of `m`. When you take the product of Messages ((2)) and ((3)), the resulting mean is the sum of the precision-weighted means of Messages ((2)) and ((3)), normalized by the total precision (see [Bert's lecture](http://nbviewer.ipython.org/github/bertdv/BMLIP/blob/master/lessons/notebooks/The-Gaussian-Distribution.ipynb)). `xi` represents the sum of precision-weighted means of the two messages. Let's look for the mean:


```julia
# Extract parameters from message ((4))
m4 = mean(messages[4].dist)
v4 = var(messages[4].dist)
println("Mean of Message ((4)) = "*string(m4))
println("Variance of Message ((4)) = "*string(v4))
```

    Mean of Message ((4)) = 65.69999999999999
    Variance of Message ((4)) = 5.999999999999999


As you can see, the mean of Message ((4)) lies in between the means of Messages ((2)) and ((3)). Note that the variance is much lower than that of Messages ((2)) or ((3)).


```julia
# Define probability density function for Gaussian distribution
pdf_Normal(θ, m, v) = 1/sqrt(2*π*v) * exp( -(θ - m)^2/(2*v))

# Extract parameters from message ((2))
m2 = messages[2].dist.params[:m]
v2 = messages[2].dist.params[:v]

# Extract parameters from message ((3))
m3 = messages[3].dist.params[:m]
v3 = messages[3].dist.params[:v]

# Extract parameters from message ((4))
m4 = mean(messages[4].dist)
v4 = var(messages[4].dist)

# Define new range for skill level θ
θ_range = range(0.0, step=0.1, stop=100.0)
plot(θ_range, pdf_Normal.(θ_range, m2, v2), color="black", linewidth=3, label="Message ((2))", xlabel="θ", ylabel="p(θ)")
plot!(θ_range, pdf_Normal.(θ_range, m3, v3), color="green", linewidth=3, label="Message ((3))")
plot!(θ_range, pdf_Normal.(θ_range, m4, v4), color="blue", linewidth=3, label="Message ((4))", xlims=[50., 80.])
```

Message ((4)) is really a weighted average of Messages ((2)) and ((3)).


```julia
# Extract parameters from message ((1))
m1 = messages[1].dist.params[:m]
v1 = messages[1].dist.params[:v]

# Extract parameters from message ((4))
m4 = mean(messages[4].dist)
v4 = var(messages[4].dist)

# Extract parameters from marginal
m_marg = mean(marginals[:θ])
v_marg = var(marginals[:θ])

# Define new range for skill level θ
plot(θ_range, pdf_Normal.(θ_range, m1, v1), color="red", linewidth=3, label="Message ((1))", xlabel="θ", ylabel="p(θ)")
plot!(θ_range, pdf_Normal.(θ_range, m4, v4), color="blue", linewidth=3, label="Message ((4))")
plot!(θ_range, pdf_Normal.(θ_range, m_marg, v_marg), color="purple", linewidth=6, linestyle=:dash, label="Marginal", xlims=[50., 80.])
```

The posterior is also a weighted average of two incoming messages. Notice that it is much closer to Message ((4)) than Message ((1)). That is because the variance of Message ((1)) (the prior) is much higher than that of Message ((4)) (the combination of likelihoods). The prior has a smaller weight in the weighted average.

---

#### $\ast$ **Try for yourself**

Play around with different values for the prior's variance and the variance of the likelihoods. What happens when you make the variance of $p(X_1 \mid \theta)$ different from that of $p(X_2 \mid \theta)$?

---

c578cd7216f4ce07e1d6183b59a0e0e894dcef54
ipynb
Jupyter Notebook
lessons/notebooks/probprog/Probabilistic-Programming-2.ipynb
bertdv/BMLIP
58920c0117efb019f2256c9ef3ca6ed65db69544
CC-BY-3.0
2019-09-14T17:34:14.000Z
2022-01-22T18:29:11.000Z
lessons/notebooks/probprog/Probabilistic-Programming-2.ipynb
bertdv/BMLIP
58920c0117efb019f2256c9ef3ca6ed65db69544
CC-BY-3.0
2019-08-09T15:49:10.000Z
2021-11-14T10:19:28.000Z
lessons/notebooks/probprog/Probabilistic-Programming-2.ipynb
bertdv/BMLIP
58920c0117efb019f2256c9ef3ca6ed65db69544
CC-BY-3.0
2020-03-18T14:05:09.000Z
2022-01-04T14:35:32.000Z
Qwen/Qwen-72B
1. YES
2. YES