Help with Novelty Recognition and Binary Classification for Emotion Recognition I’m looking for advice regarding my ML project.

Using a special wristband, I am able to collect a bunch of physiological data from human subjects. I want to develop an application to recognize when these physiological signals change in a meaningful way and only then ask the user how he/she is feeling. This data will later be used for machine learning testing. The problem is, that I am struggling to find appropriate ways to classify current data input as meaningful and ask for information only when relevant user input is to be gathered, not more and not less.

For me, this seems to be a novelty detection problem, combined with a binary classification problem. I have to recognize what values coming from the data stream are to be considered normal, and therefore not bother the user with unnecessary input requests. I would also use novelty detection to recognize the data coming out of the normal zone and ask the user about it. This new data is then not considered novelty anymore, and binary classification will tell if the user is to be asked about his emotions when getting the same data in the future.

So, these are my questions:
-  What do you think about my reasoning of the problem? Do you have other perspectives on how to handle these problems? I have been told this could also be considered an anomaly detection problem, for example.

-  What algorithms would you use to separate normal from more meaningful physiological data? Support Vector Machines perhaps? Maybe some decision theory?

-  Do you know any books or papers on similar matters? Even if I have found some after hours and hours of research, you may be able to point me to something different than those I have.

It is worth noting that data collection is supposed to be done when no other factors are messing with signal readings, such as sport.

Any help would be much appreciated.

Best regards,
Augusto

The Project

It appears from the question that emotional detection and response is the longer term goal of the project and that recognizing potential emotional manifestations in easily detectable physiological metrics is an initial R&D objective.

Mobile device applications are already available to do this, but biometric monitoring via a wristband, excellence in AI design, and marketing excellence could overtake these apps in the marketplace and provide emotional regulation to improve productivity and reduce health and wellness risks.

The (at least initial) goal seems to be basic binary classification, which simplifies the output, but not the detection, which seems to have the following two criteria.

Acquire biological information related to emotion
Avoid drawing the user's attention to perform unnecessary tasks

The Biometric Device Challenge

There is definitely a challenge to classifying biometric trends as meaningful in this context.

  What do you think about my reasoning of the problem? Do you have other perspectives on how to handle these problems? I have been told this could also be considered an anomaly detection problem, for example.

Novelty detection is not the first milestone in this challenge. Detection of useful features is the prerequisite. Novelty comes into play once changes in the organism can be characterized with some reliability and accuracy (few false positives and false negatives) and the particular user has provided subjective reporting to correspond with some change in the organism.

Biological systems have forms of stasis that can be indirectly sensed, including these.

Blood pressure
Metabolic rate
Multi-ionic salinity
Cognitive attention
Blood sugar levels

These affect the dermis at the wrists through perturbations in externally detectable metrics, the inclusion of which depends on the capabilities of the wrist device.

Circumference
Coefficient of electrical resistance
Surface temperature
Multi-ionic salinity
Surface moisture

With blood and urine lab tests, conditions can be controlled, however sampling of those fluids are usually infrequent, leading to the use of ranges to detect anomalies that may be indicative of disease, conditions, or other health risks. There are major challenges to using ranges of metrics at the wrist.

Sensitivity to the general and ongoing physical condition of the individual
Unique individual responses to emotional states
Interference from changes in the temperature, pressure, air movement, and humidity of the environment

One inroad to detection of the internal metrics of the organisms is the fact that biological stasis is not the same thing as perfect regulation. Biological systems are chaotic. Normal signals from time series sampling of biological metrics are neither constant nor periodic. When changes in the attractors and spectra characterizing the chaotic fluctuation of direct metrics (like electrical resistance) or indirect metrics (like systolic blood pressure) occur, a request for subjective information may be asked from the wearer.

These are some chaotic principles to consider.

Spectra can provide much information about the intensity of the force driving the chaotic fluctuation in the time series, which is likely to correlate with stress.
The various ways to evaluate exponents, including the Lyapunov exponent, may be of significant value in correlating patterns in the external, indirect time series to emotional states.
Autocorrelation at the primary frequencies involved may reveal changes indicative of significant mental transitions.

Once a correlation is established, then the novelty detection makes sense, except for an occasional re-acquisition of the correlation, since the wearer may become more internally connected to their physical and emotional states as they use the app, and the answers to inquiries may mature correspondingly.

Additional Questions Within the Question

These additional questions were also asked.

  What algorithms would you use to separate normal from more meaningful physiological data? Support Vector Machines perhaps? Maybe some decision theory?

A recurrent neural network such as B-LSTM or GRU are commonly excellent at characterizing time series, but there may be a need for more than one network.

One to extract the features of the chaotic patterns in the raw data
Another to train to correlate changes in the features of the chaos with subjective reporting

This second training must be done during in situ field use of the first already trained network. Since training is not generally viable on a mobile device, there will need to be a client server arrangement where the training data and its resulting trained network can be communicated through secure RestFUL services so the training processes can be asynchronous and leverage GPUs or other hardware acceleration.

  Do you know any books or papers on similar matters? Even if I have found some after hours and hours of research, you may be able to point me to something different than those I have.

The terms above in a scholarly search will bring up all kinds of study materials. Specifically these

Lyapunov exponent
Strange attractors
Spectral analysis of biometrics
Chaos autocorrelation
Phase space

One good book for all but the middle item is Chaos Theory Tamed, 1997
by Garnett P. Williams.

  It is worth noting that data collection is supposed to be done when no other factors are messing with signal readings, such as sport.

This is where higher level detection involving chaotic analysis is much more reliable than ranges. The chaotic features of athletic activity will be distinctly different from those of fear states or sexual arousal.


Why is Jensen-Shannon divergence preferred over Kullback-Leibler divergence in measuring the performance of a generative network? I have read articles on how Jensen-Shannon divergence is preferred over Kullback-Leibler in measuring how good a distribution mapping is learned in a generative network because of the fact that JS-divergence better measures distribution similarity when there are zero values in either distribution.

I am unable to understand how the mathematical formulation of JS-divergence would take care of this and also what advantage it particularly holds qualitatively apart from this edge case. Could anyone explain or link me to an explanation that could answer this satisfactorily?

Lets start with question 1)  how does JS-divergence handles zeros?

by definition:
\begin{align}
D_{JS}(p||q) &= \frac{1}{2}[D_{KL}(p||\frac{p+q}{2}) + D_{KL}(q||\frac{p+q}{2})] \\
&= \frac{1}{2}\sum_{x\in\Omega} [p(x)log(\frac{2 p(x)}{p(x)+q(x)}) + q(x)log(\frac{2 q(x)}{p(x)+q(x)})]
\end{align}
Where $\Omega$ is the union of the domains of $p$ and $q$. Now lets assume one distribution is zero where the other is not (without loss of generality due to symmetry we can just say $p(x_i) = 0$ and $q(x_i) \neq 0$. We then get for that term in the sum
$$\frac{1}{2}q(x_i)log(\frac{2q(x_i)}{q(x_i)}) = q(x_i)\frac{log(2)}{2}$$
Which isn't undefined as it would be the KL case.

Now onto 2) In GANS why does JS divergence produce better results than KL

The asymmetry of KL divergence places an unfair advantage to one distribution over the other where in this case, its not ideal to consider it this way from an optimization perspective. Additionally KL divergences inability to handle non-overlapped distributions is crushing given that these are approximated through sampling schemes, and therefore there are no guarantees. JS solves both those issues and leads to a smoother manifold which is why its generally preferred. A good resource is this paper where they go more in detail investigating this.


Does the performance of a model increase if dropout is disabled at evaluation time? I know dropout layers are used in neural networks during training to provide a form of regularisation in an attempt to mitigate over-fitting.

Would you not get an increased fitness if you disabled the dropout layers during evaluation of a network?

Dropout is a technique that helps to avoid overfitting during training. That is, one can use dropout only for training.

  units may change in a way that they fix up the mistakes of the other
  units. This may lead to complex co-adaptations. This, in turn, leads to
  overfitting because these co-adaptations do not generalize to unseen
  data.

If you want to evaluate your model, you should turn off all dropout layers. For example, PyTorch's model.eval() does this work.

More about dropout:

Improving neural networks by preventing co-adaptation of feature detectors
Dropout: A Simple Way to Prevent Neural Networks from Overfitting


How can we find the value function by solving a system of linear equations without knowing the policy? An MDP is a Markov Reward Process with decisions, it’s an environment in which all states are Markov. This is what we want to solve. An MDP is a tuple $(S, A, P, R, \gamma)$, where $S$ is our state space, $A$ is a finite set of actions, $P$ is the state transition probability function,
$$P_{ss'}^a = \mathbb{P}[S_{t+1} = s' | S_t = s, \hspace{0.1cm}A_t = a] \label{1}\tag{1}$$
and
$$R_s^a = \mathbb{E}[R_{t+1}| S_t =s, A_t = a]$$
and a discount factor $\gamma$.
This can be seen as a linear equation in $|S|$ unknowns, which is given by,
$$V = R + \gamma PV  \hspace{1mm}  \label{2}\tag{2}$$
$V$ is value of a state vector, $R$ is immediate reward vector, $P$ is transition probability matrix, where each element at $(i,j)$ in $P$ is given by,  $ P[i][j] = P(i \mid j)$ i.e., probability that I am in state $j$ going to state $i$.
As $P$ is given, we treat, equation $\ref{2}$ as a linear equation in $V$.
But $P[i][j] = \sum_a (\pi(a \mid j) \times \mathrm{p}(i \mid j, a) )$. But, $ \pi (a \mid s)$ (i.e., probability that I will take action a in state s) is NOT given.
So, how can we frame this problem as the solution to a system of linear equations in \ref{2}, if we only know $ P^a_{ss'}$ and we do not know $ \pi(a \mid s)$, which is needed to calculate $P[i][j]$?

Your equations all look correct to me.
It is not possible to solve the linear equation for state values in the vector $V$ without knowing the policy.
There are ways of working with MDPs, through sampling of actions, state transitions and rewards, where it is possible to estimate value functions without knowing either $\pi(a|s)$ or $P^{a}_{ss'}$. For instance, Monte Carlo policy evaluation or single-step TD learning can both do this. It is also common to work with $\pi(a|s)$ known but $P^{a}_{ss'}$ and $R^{a}_{s}$ unknown in model-free control algoithms such as Q learning.
However, in your case, you are correct, in order to resolve the simultaneous equations you have presented, you do need to know $\pi(a|s)$
This is not as limiting as you might think. You can construct a control method using simultaneous equations, by starting with the policy set to some arbitrary policy. Either a randomly-chosen deterministic policy or the equiprobable policy are reasonable first guesses. Then, after each solution to linear equations, you improve the policy so that each action choice maximises the expected return. This is essentially the policy iteration algorithm but replacing the policy evaluation step with the linear equations method for calculating the values.


Is there an online tool that can predict accuracy given only the dataset? Is there an online tool that can predict accuracy given only the dataset as input (i.e. without the compiled model)?

That would help to understand how data augmentation/distribution standardization, etc., is likely to change the accuracy.

If I get correctly what you're asking, the answer is no, there is no way of knowing in advance how good a model would perform in a dataset without training a model on it. That's the whole point of data science, you try, you analyse the results, you try again using the knowledge you got from your previous attempts. It would be nice to hack the whole field and know in advance what to do to get the perfect model but it is rather unrealistic.

Anyway, there are some standard steps that usually help understanding if you're going on the right direction. For example creating a random benchmark to see how much your model is better than a random one. To create such benchmark you definitely don't need a specific tool, all main programming languages provide built in function to generate random numbers, and that's basically all that you need. For example in python you could do something like:

import numpy as np

# create 30 true labels
y_true = [1,2,3]*10

# generate 30 random labels
y_random = np.random.randint(0,3,size=30)

# calculate accuracy for random model
acc_random = sum(y_true==y_random)/len(y_true)

Another thing that is worth to mention is that in the academy people are pushing more and more to use the same datasets in order to have comparable results. for example if you're trying to train an architecture that should be used for images classification, some golden dataset you must test your models on are the MNIST ones. It is well known that on these datasets a good model should achieve more than 99% accuracy, therefore this is an establish lower bound for these datasets.


I need to select the image from a predefined dataset that are the closest to the input, is this possible or do I even need to use ML/AI? So as the title states, I have a set of images and I want to process input images and need to select the image that "looks" the most like the input image.

I know I've seen something similar where the code could guess who's face was in a picture, I guess I want something like that but for general images.

Sorry if this is a stupid question, but any suggestions or points at resources would be greatly appreciated.

You will only need to use ML or AI if
a) the dataset is very big.
b) It is difficult to get the meaning or value of the image.(ex: group of ants, photos of stars in night sky)
And many more.

Below is the way I thought it can be done through a machine(with deeplearning, CNN)
1. Train the model
        arrange the images in dataset into clusters (the number of clusters can
        be decided based on the criticality of the application, more critical the
        application more should be the number of clusters) .

Predict about input
    Then predict the nearest image in the dataset to the provided image.

I recently listened a podcast in which, a problem is approached similarly. The podcast is about youtube predictions and the result is to provide closest video from the youtube videos ( here the dataset).Below is the podcast.
    ( https://open.spotify.com/episode/6PcMtVXR58i7iu8kLH40Wd?si=feyu0LjESoiE479xHkaqSA )


Is this ML task possible? What I want to do is from an Internet challenge to transform any given image into the Polish flag using the available filters and crop tool on the iPhone camera app. Here's an example.
There aren't nearly enough of these videos to train a neural network using a labeled dataset, and (while I haven't ruled it out) I don't think automatically inserting a polish flag into an image then adding random filters to it to create my own dataset would work out.
My thinking is that I would feed a neural network the image and it would output a value for each filter & cropping coordinates. Then, I could easily calculate the loss by comparing the resulting picture to a picture of the polish flag. The obvious problem here is that you don't know how each of the neurons in the last layer affects the loss so you can't perform back propagation.
Is my best bet to mathematically calculate the loss (by this I mean as opposed to using high level libraries, which would be difficult but I'm sure it's possible) so I can find the partial derivative of each last layer neuron with respect to the loss function and then backpropagate? Would this even work? Are there any alternatives that you recommend?

I think the best thing to use here is a form of "structured prediction". Our "target" is a sequence of operations. The framework of structured prediction allows us to chain together as many filters as we want.
With a neural network of fixed architecture, you would have to make sure you have enough space for all the filters you might need.


How should the output layer of an LSTM be when the output are word embeddings? I'm having trouble grasping how to output word embeddings from an LSTM model. I'm seeing many examples using a softmax activation function on the output, but for that I would need to output one hot vectors as long as the vocabulary (which is too long). So, should I use a linear activation function on the output to get the word embeddings directly (and then find the closest word) or is there something I'm missing here?

Actually, LSTM is not used to get word2vec. Indeed, word2vec is extracted from corpus of words using MLP (Multi Layer Perceptron). There is a great tutorial on how to extract word2wec:
http://mccormickml.com/2016/04/19/word2vec-tutorial-the-skip-gram-model/

After representing word as vectors, you feed your text to LSTM in a deep architecture which the last layer must be softmax to categorize your text.


What if we took a recursive approach and built a smallest possible robot? What if we took a recursive approach and built a smallest possible first robot (Robot 1) that could transfer information and data about the place it was at and could build itself in a very small size proportional to itself. I understand that it means a higher level of accuracy for this first robot (Robot 1) that its creator i.e. us.  And this first robot (Robot 1) again built a robot (Say Robot 2) that was far smaller but an exact copy of the first robot (Robot 1). And then the second robot (Robot 2) built a third Robot (Robot 3) and so on. So each next level robot was tinier and higher precision than its creator.

With the tiniest robot we could make, we sent them to the mission wherein micro-sized intervention was needed. For example studying the atom structure from inside, how similar it was to our big universe etc. Plus many more applications humankind could ever imagine.

I understand though that the material used to construct such a robot and its properties will be limiting and to explore an atom we may not be able to use an atom as the building block.

However, we could possibly build a robot like this which would be small enough to explore the human body from inside.

Lost Article and Found Articles

MIT Review had an article on nanotechnology for disease eradication, DNA repair, and microsurgery in the 1990s that's probably somewhere among the thousands of entries resulting from a web search of, "MIT Review nanotechnology cell repair," or the few hundred resulting from an academic article search for the same.1 The article I can't find described a recursive nano-robot scheme like the one described in this question.

What seems plausible given the current state of technology is to find a recursive algorithm that will command a 3D printer to make a smaller 3D printer that can print a still smaller one.

Extending the Normal Meaning of Recursion

The algorithm will have to take a step in capability beyond mere recursion.  It doesn't call itself.  It must load itself into the machine it printed and then boot the copy of itself there.  As it loads itself, at each level in size, it must parametrize its child for that geometrically reduced size for each progressive reduction.  It must stop when the desired size is achieved.

Such a paradigm could be called, "Printail Recursion," from the synthesis of, "Printer," and the algorithmic principle of, "tail recursion," from the dawn of LISP.

Applying the Decorator Pattern

Once Printail Recursion works, other robotic or algorithmic features could piggy back on the appropriate components of the progressive micro-children.

Notes

[1] The later of those two may even provide some departmental contacts to open options for partnering between academic institutions, something usually compelling for research oriented students and a great catalyst for scientific collaboration among the next generation of researchers.


What is the difference between on-policy and off-policy for continuous environments? I'm trying to understand RL applied to time series (so with infinite horizon) which have a continous state space and a discrete action space.

First, some preliminary questions: in this case, what is the optimal policy? Given the infinite horizon there is no terminal state but only an objective to maximise the rewards, so I can't run more than one episode, is it correct?

Consequently, what is the difference between on-policy and off-policy learning given this framework?

First, some preliminary questions: in this case, what is the optimal policy?

It is the policy that maximises return from any given time step $G_t$. You need to be careful with your definition of return with continuing environments. The simple expected sum of future rewards is likely to be positive or negative infinity.

There are three basic approaches:

Set an arbitrary finite horizon $h$, so $G_t = \sum_{k=1}^{h} R_{t+k}$
Use discounting, with discount factor $\gamma < 1$, so $G_t = \sum_{k=0}^{\infty} \gamma^k R_{t+k+1}$
Use average reward per time step,  $\bar{R} = \lim\limits_{h \to \infty}\frac{1}{h} \sum_{k=1}^{h} R_{t+k}$, which leads to thinking in terms of differential return $G_t = \sum_{k=1}^{\infty} (R_{t+k} - \bar{R})$

With large enough horizon (so that state is ergodic) or large enough $\gamma$ (close to $1$), these approaches are similar and should result in approximately the same policy. The difference is in how you construct an agent in detail to solve the problem of maximising the value.

  Given the infinite horizon there is no terminal state but only an objective to maximise the rewards, so I can't run more than one episode, is it correct?

The term episode becomes meaningless. It may be simpler to think of this differently though - you are trying to solve a non-episodic problem here, in that there is no natural separation of the process into separate meaningful episodes. No physical process is actually infinite, that's just a theoretical nicety.

In practice, if you can run your environment in simulation, or multiple versions of it for training purposes, then you do start and stop pseudo-episodes. You don't treat them as episodes mathematically - i.e. there is no terminal state, and you can never obtain a simple episodic return value. However, you can decide to stop an environment and start a new one from a different state.

Even if there is only one real environment, you can sample sections of it for training, or attempt to use different agents over time, each of which is necessarily finite in nature.

  Consequently, what is the difference between on-policy and off-policy learning given this framework?

The notions of on-policy and off-policy are entirely separate from episodic vs continuing environments.

On-policy agents use a single policy both to select actions (the behaviour policy), and as the learning target. When the learnt policy is updated with new information that immediately affects the behaviour of the agent.
Off-policy agents use two or more policies. That is one or more behaviour policies that select actions (the behaviour policy), and a target policy that is learned, typically the best guess at the optimal policy given data so far.

These things do not change between episodic and continuing tasks, and many algorithms remain identical when solving episodic vs continuing problems. For example, DQN requires no special changes to support continuing tasks, you can just set a high enough discount factor and use it as-is.

You cannot wait until the end of an episode, so certain update methods won't work. However, value bootstrapping used in temporal difference (TD) learning still works.

In some cases, you will want to address the differences in the definition of return. Using an average reward setting typically means looking at a differential return for calculating TD targets for example.


Is value iteration stopped after one update of each state? In section 4.4 Value Iteration, the authors write

One important special case is when policy evaluation is stopped after just one sweep (one update of each state). This algorithm is called value iteration.

After that, they provide the following pseudo-code

It is clear from the code that updates of each state occur until $\Delta$ is sufficiently small. Not one update of each state as the authors write in the text. Where is the mistake?

Where the author mentions the policy evaluation being stopped after one state, they are referring to the part of the algorithm that evaluates the policy -- the pseudocode you have listed is the pseudocode for Value Iteration, which consists of iterating between policy evaluation and policy improvement.
In normal policy evaluation, you would apply the update $v_{k+1}(s) = \mathbb{E}_\pi[R_{t+1} + \gamma v_k(S_{t+1})|S_t = s]$ until convergence. In the policy iteration algorithm, you perform policy evaluation until the value functions converge in each state, then apply policy improvement, and repeat. Value iteration will perform policy evaluation for one update, i.e. not until convergence, and then improve the policy, and repeat this until the value functions converge.
The line
$$V(s) \leftarrow \max_a \sum_{s', r} p(s',r|s,a)[r + \gamma V(s')]$$
perform both the early stopping policy evaluation and policy improvement. Lets examine how:
The $\sum_{s', r} p(s',r|s,a)[r + \gamma V(s')]$ is the same as the expectation I wrote earlier, so we can see clearly that is policy evaluation for just one iteration. Then, we take a max over the actions -- this is policy improvement. Policy improvement is defined as (for a deterministic policy)
\begin{align}
\pi'(s) &= \arg\max_a q_\pi(s,a) \\ &= \arg\max_a \sum_{s', r} p(s',r|s,a)[r + \gamma V(s')]\;.
\end{align}
Here, we assign the action that satisfies the $\mbox{argmax}$ to the improved policy in state $s$. This is essentially what we are doing in the   line from your pseudo when we take the max. We are evaluating our value function for a policy that is greedy with respect to said value function.
If you keep applying the line from the pseudocode of value iteration it will eventually converge to the optimal value function as it will end up satisfying the Bellman Optimality Equation.


What does the statement with the max do in the recursive best-first search algorithm? I am reading the book "Artificial Intelligence: A Modern Approach" by Stuart and Norvig. I'm having trouble understanding a step of the recursive best-first search (RBFS) algorithm.

Here's the pseudocode.

What does the line s.f <- max(s.g + s.h, node.f) do?

Here's a diagram of the execution of RBFS.

This is probably more easily understood as the collapse/restore macro. The idea is that the previously explored state was collapsed and only the minimum f-cost from the sub-tree was stored. This represents the best unexpanded state in the subtree that was collapsed.

When restoring the portion of the collapsed tree, the f-cost of the restored node could either be the original f-cost (g+h), or it could be the stored f-cost if it is larger. By taking the max, the code ensures that states that are restored maintain at least the cost of the previously best unexpanded state. (If the g+h cost is larger, then we know the state wasn't previously expanded and it wasn't previously the state on the fringe with the minimum edge cost.)

The linked paper gives several examples where similar ideas are used during search.


Can I do state space quantization using a KMeans-like algorithm instead of range buckets? Are there any reference papers where it is used a KMeans-like algorithm in state space quantization in Reinforcement Learning instead of range buckets?

There is this paper Representation and Reinforcement Learning for
Personalized Glycemic Control in Septic Patients, presented in the Machine Learning for Health Workshop in NIPS 2017. Here is a quote from the paper where the authors describe the clustering approach:

  After we generated the state representation,
  we used the k-means clustering algorithm to categorize millions of patient states into 500 clusters
  such that similar clinical states can collapse into the same cluster.


Concrete Example for Q Learning I am not sure if I understood the q learning algorithms correctly.
Therefore I would give a concrete example and ask if someone can tell me how to update the q value correctly.

First I initialized a Neural Network with random weights. It shall henceforth evaluate the Q Value for all possible actions(4) given a State S.

Then the following happens. The agent is playing and is exploring.
For 3 steps the Q Values evaluated were:
(0,-1,-5,0), (0,-1,0,0), (0,-.6,0,0)

The reward given was: 0,0,1
The action took were: (1.,1.,1.)
In the random walk example (same reward given), it was: (1.,2.,3.)

So what are the new Q - Values, assuming a discount factor of 0.99 and the learning rate 0.1?

The States for Simplicity are only one number: 1,1.3,2.4 Where 2.4 is the state who ends the game...

The same example holds for exploiting. Is the algorithm the same here?

Here you see my last implementation:

    public void rlearn(ArrayList<Tuple> tupels, double learningrate, double discountfactor) {

    //newQ = sum of all rewards you have got through
    for(int i = tupels.size()-1; i > 0; i--) {
        MLData in = new BasicMLData(45);
        MLData out = new BasicMLData(5);

        //Add State as in
        int index = 0;
        for(double w : tupels.get(i).statefirst.elements) {
            in.add(index++, w);
        }

        //Now start updating Q - Values
        double qnew = 0;
        if(i <= tupels.size()-2){
            qnew = tupels.get(i).rewardafter + discountfactor*qMax(tupels.get(i+1));
        } else {
            qnew = tupels.get(i).rewardafter;
        }

        tupels.get(i).qactions.elements[tupels.get(i).actionTaken] = qnew;
        //Add Q Values as out
        index = 0;
        for(double w : tupels.get(i).qactions.elements) {
            out.add(index++, w);
        }
         bigset.add(in, out);
    }
}

Edit: This is the qMax - function:

    private double qMax(Tuple tuple) {
    double max = Double.MIN_VALUE;
    for(double w : tuple.qactions.elements) {
        if(w > max) {
            max = w;
        }
    }
    return max;
}

Most Deep Q-learning implementations I have read are based on Deep Q-Networks (DQN).  In DQN, the q-value network maps an input state to a vector of q-values, one for each action:

$$
Q(s, \mathbf{w}) \to \mathbf{v}
$$

where $s$ is the input state from the environment, $\mathbf{w}$ are the parameters of the neural network, and $\mathbf{v}$ is a vector of q-values, where $v_i$ is the estimated q-value of the ith action. In the Sutton and Barto book, the q-value function is written as $Q(s, a, \mathbf{w})$, which corresponds to the network output for action $a$.

Unlike tabular Q-learning, Deep Q-learning updates the parameters of the the neural network according to the gradients of the loss function with respect to the parameters. DQN uses the loss function

$$
L(\mathbf{w}) = [(r + \gamma max_{a'} Q(s', a', \mathbf{w^-})) - Q(s, a, \mathbf{w})]^2
$$

where $\gamma$ is the discount rate, $a$ is the selected action (either greedily or randomly for an $epsilon$-greedy behavior policy), $s'$ is the next state, $a'$ is the argmax action for the next state, and $\mathbf{w^-}$ is an older version of the network weights $\mathbf{w}$ that is used to help stabilize training.

In deep Q-learning, training directly updates parameters, not q-values. Parameters are updated by taking a small step in the direction of the gradient of the loss function

$$
\mathbf{w} \gets \mathbf{w} + \alpha [(r + \gamma max_{a'} Q(s', a', \mathbf{w^-})) - Q(s, a, \mathbf{w})] \nabla_w Q(s, a, \mathbf{w})
$$

where $\alpha$ is the learning rate.

In frameworks like tensorflow or pytorch the derivative is calculated automatically by giving the loss function and model parameters directly to an optimizer class which uses some variation of mini-batch gradient descent. In eagerly executed tensorflow updating the parameters for a mini-batch might look something like

batch = buffer.sample(batch_size)
observations, actions, rewards, next_obervations = batch

with tf.GradientTape() as tape:
    qvalues = model(observations, training=True)
    next_qvalues = target_model(next_obervations)
    # r + max_{a'} Q(s', a') for the batch
    target_qvalues = rewards + gamma * tf.reduce_max(next_qvalues, axis=-1)
    # Q(s, a) for the batch
    selected_qvalues = tf.reduce_sum(tf.one_hot(actions, depth=qvalues.shape[-1]) * qvalues, axis=-1)
    loss = tf.reduce_mean((target_qvalues - selected_qvalues)**2)

grads = tape.gradient(loss, model.variables)
optimizer.apply_gradients(zip(grads, model.variables))

Though I am not familiar with the Encog neural network framework you are using, based on the example Brain.java file from your Github repo and Chapter 5 of the Encog User Manual and the Encog neural network examples on Github it looks like weights are updated as follows:

A training set is constructed from pairs of input and target output.
A Propagation instance, train, is constructed with a network and training set. Different subclasses of Propagation use different loss functions to update the network parameters.
The method train.iterate() is called to run the network on the inputs, calculate the loss between the network outputs and target outputs, and update the weights according to the loss.

For DQN, a training set is constructed from a random sample from the experience replay buffer to help stabilize training. A training set could also be the trajectory of an episode, which is what the tupels argument in the example code of the question appears to be.

The input would be the statefirst member of each element of tupels. Since the network produces a vector of q-values, the target output must also be a vector of q-values.

The target output element for the selected action is $r + \gamma max_{a'} Q(s', a', \mathbf{w^-})$,  In the example code of the question, this is

double qnew = 0;
if(i <= tupels.size()-2){
    qnew = tupels.get(i).rewardafter + discountfactor*qMax(tupels.get(i+1));
} else {
    qnew = tupels.get(i).rewardafter;
}
tupels.get(i).qactions.elements[tupels.get(i).actionTaken] = qnew

The target output elements for actions that were not selected should be $Q(s, b, \mathbf{w})$, where $b$ is one of the non-selected actions.  This should have the effect of ignoring the q-values of non-selected actions by making the network output equal to the target output.

So what are the new Q - Values, assuming a discount factor of 0.99 and the learning rate 0.1?

Assuming you mean target outputs by the new Q - Values, and given the trajectory of actions, (1, 1, 1), and q-value vectors from the question, the concrete target outputs are (0, 0 + 0.99 * 0, -5, 0), (0, 0 + 0.99 * 0, 0, 0), and (0, 1 + 0, 0, 0).


Which books or papers clearly explain the relation between Ising models and deep neural networks? I am looking for a book or paper which clearly explains the relationship between Ising models and deep neural networks.

Can anyone provide any references?

The following articles

Ising models for networks of real neurons (2006) by Gasper Tkacik et al.
Deep neural networks for direct, featureless learning through observation: The case of two-dimensional spin models (2018) by Kyle Mills et al.
Inverse Ising inference by combining Ornstein-Zernike theory with deep learning (2017) by Soma Turi, Alpha A. Lee et al.
Neural-network based general method for statistical mechanics on sparse systems (2019) by Feng Pan, Hai-Jun Zhou, Pan Zhang et al.

may help you understand the "relationship" between Ising models and DNN, assuming you know what the Ising model is and what a DNN is, the similarity should be fairly intuitive to you.

The Ising model is a sort of floating soup of ferromagnetic particles each generating their own small magnetic field either working against or with their neighbor. When many of the particles aline, they create an aligned field we refer to as a dipole moment in magnetism, while in a DNN we refer to the joined effort of a few entities working to cause a larger effect in another entity an 'activation function'. In a fully connected DNN, where the Euclidean distance weights the connections and the nodes are initialized with a certain magnetic polarity in relation to the axis of the magnetic field it generates, the network would be an almost exact representation of the reality of what the Ising model seeks to simplify.


Would this relatively small dataset be enough to train a CNN? Scenario: I am trying to create a dataset with images of choice for different animal classes. I am going to train those images for classification using CNN.
Problem: Let's assume I somehow don't have the privilege to collect too many images and was only able to collect a few of them for each class. Here's the list:
| id | animal       | #     |
|----|--------------|-------|
| 1  | Baboon       | 800   |
| 2  | Fox          | 1000  |
| 3  | Hyena        | 5000  |
| 4  | Giraffe      | 43    |
| 5  | Zebra        | 88    |
| 6  | Hippopotamus | 233   |
| 7  | Yak          | 578   |
| 8  | Polar Bear   | 456   |
| 9  | Lion         | 3442  |
| 10 | Indian Tiger | 40000 |

I have three questions.

Is this a good dataset to train the CNN model? I am worried about the quantity each class has.

Will it be helpful if I augment the data? I think I am going to augment it.

In the future, the above-mentioned dataset is going to increase. So there is a chance that I will train the model again. Should I create a model that fits the data of the present size or should I create a bigger one in order to adjust future data?

I can get data from the Internet. But this question is about the approaches to take when we have a small amount of data, like the one in National Data Science Bowl (classifying Planktons).

It is somewhat risky to discuss data independently with your learning mechanism.  There is actually no such thing as good data or a good learner.  There is only data that is good WITH a particular learner.  That is even true of human intelligence after all the standardized education and testing done today.

There are also exceptional learners that find data to be good when most others fumble with it.

If by good data and deep learning you mean image sets that will lead to proper categorization of unsuspected images presented in production, your intuitive understanding of statistics can provide you with a general answer.  The images on which the deep learner develops its activation weights and meta-parameters to provide adequate production behavior must be representative of the range of images that will be found in the production feeds.

If you intended to do a study of men and women to determine if the old belief that women are more motivated by the prospect of love and men are more motivated by the prospect of sex, you wouldn't pick 43 men and 40,000 women for the study.  The study's value is limited by the lower of the two numbers.

You can train the network with the category frequencies you have, but some deep learners may capitalize fully on feature extraction for Indian Tigers and Hyenas but exhibit an unacceptable level mis-categorization of Zebras and Giraffes.

Returning to the concept above, the skew in category frequency can be accounted for by the deep learner.  It is theoretically possible to create an exceptional learner or one that is well attuned to this kind of frequency skew.  A simple approach is to develop a scheme that recognizes frequency skew and allocates additional computing resources to the training that focuses on the differentiation of similar animals with infrequent labeled training instances.

I don't recall who has done that, but I know it has been done.

There are several ways you can give extra attention to the infrequent categories manually in the code, but then it would be a less general solution and the resulting program would neither be an exceptional learner nor particularly reusable.

It is more cost effective to hunt for a skew resistant deep learning scheme and test its accuracy for infrequent animals than sending a photographer to Africa.  If you can find more images of the less frequent animals without a monumental effort, I would do that too.


Deep Learning models train really slow Jetson Nano I recently bought a Jetson Nano and I'm amazed with everything about it. But I don't know what is happening, because I created a very simple neural network with keras and it's taking way to long. I know is taking to long, because I runned the same ANN in my PC's CPU and it was faster than the jetson nano.

Here's the code:

import numpy as np
import matplotlib.pyplot as plt
import pandas as pd

dataset = pd.read_csv('Churn_Modelling.csv')
X = dataset.iloc[:, 3:13].values
y = dataset.iloc[:, 13].values

from sklearn.preprocessing import LabelEncoder, OneHotEncoder
labelencoder_X_1 = LabelEncoder()
X[:, 1] = labelencoder_X_1.fit_transform(X[:, 1])
labelencoder_X_2 = LabelEncoder()
X[:, 2] = labelencoder_X_2.fit_transform(X[:, 2])
onehotencoder = OneHotEncoder(categorical_features = [1])
X = onehotencoder.fit_transform(X).toarray()
X = X[:, 1:]

from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = 0.2, random_state = 0)

from sklearn.preprocessing import StandardScaler
sc = StandardScaler()
X_train = sc.fit_transform(X_train)
X_test = sc.transform(X_test)

from tensorflow import keras
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Dense

classifier = Sequential()

classifier.add(Dense(units = 6, kernel_initializer = 'uniform', activation = 'relu', input_dim = 11))

classifier.add(Dense(units = 6, kernel_initializer = 'uniform', activation = 'relu'))

classifier.add(Dense(units = 1, kernel_initializer = 'uniform', activation = 'sigmoid'))

classifier.compile(optimizer = 'adam', loss = 'binary_crossentropy', metrics = ['accuracy'])

classifier.fit(X_train, y_train, batch_size = 10, epochs = 100)

y_pred = classifier.predict(X_test)
y_pred = (y_pred > 0.5)

I should mention that of course, I did the correct installation of TensorFlow GPU library and not the normal TensorFlow, in fact I used the resources in this link: TensorFlow GPU Jetson Nano

First of all, you mentioned that you installed the correct version of Tensorflow on Jetson. You can list the available Tensorflow devices with:

from tensorflow.python.client import device_lib
print(device_lib.list_local_devices())

And make sure that you see the GPU available.

More importantly, Jetson Nano has a 128-core NVIDIA Maxwell™ architecture-based GPU. This might not be a powerful GPU for training a neural network with. As you can see here, in NVIDIA's official benchmark, they only tested trained models (i.e. classifications, object detections and segmentations). You can consider training your Keras model using another hardware and use it for labeling, classification, or etc. in your Jetson.


In adversarial machine learning, how does an attacker have access to the test and training dataset in order to poison it? In the field of adversarial machine learning, machine learning models are vulnerable to attacks both on the test and training data set. However, how does the attacker get access to these datasets? How do these datasets get manipulated/tampered with?

They don't have acces to the original training or test dataset. Machine learning environments are build on the premise of a benign environment. The models are trained on real data (real inputs). When someone sends a made up input (fake input) it is very easy to fool the model.

This is used for example in image recognition. Imagine a fotograph of a panda. the model may correctly identify this fotograph as a panda. With knowledge of the model you can now alter some pixels in the fotograph. To the human Eye, the fotographs will appear exactly the same, but the model can be fooled to believe the fotograph is actually of a gibbon.

This is all done after the training of the model and doesn't require the original datasets.

For more info, visit this site:
https://medium.com/@ml.at.berkeley/tricking-neural-networks-create-your-own-adversarial-examples-a61eb7620fd8


What are the types of artificial intelligence and how are they measured? I was reading an interesting book about the role of AI in Cybersecurity, and the author mentioned there being 3-4 types. Each one is dependent on its abilities and understanding. For example, a ‘narrow artificial intelligence’ is only capable of one such ability.

What are these "levels" and how are they gauged?

I believe you are referring to something that Arend Hintze wrote about in his article "Understanding the four types of AI, from reactive robots to self-aware beings".  Here are the four types from his article:

Type I AI: Reactive machines
The most basic types of AI systems are purely reactive, and have the ability neither to form memories nor to use past experiences to inform current decisions. Deep Blue, IBM’s chess-playing supercomputer, which beat international grandmaster Garry Kasparov in the late 1990s, is the perfect example of this type of machine.

Deep Blue can identify the pieces on a chess board and know how each moves. It can make predictions about what moves might be next for it and its opponent. And it can choose the most optimal moves from among the possibilities.

But it doesn’t have any concept of the past, nor any memory of what has happened before. Apart from a rarely used chess-specific rule against repeating the same move three times, Deep Blue ignores everything before the present moment. All it does is look at the pieces on the chess board as it stands right now, and choose from possible next moves.

This type of intelligence involves the computer perceiving the world directly and acting on what it sees. It doesn’t rely on an internal concept of the world. In a seminal paper, AI researcher Rodney Brooks argued that we should only build machines like this. His main reason was that people are not very good at programming accurate simulated worlds for computers to use, what is called in AI scholarship a “representation” of the world.

The current intelligent machines we marvel at either have no such concept of the world, or have a very limited and specialized one for its particular duties. The innovation in Deep Blue’s design was not to broaden the range of possible movies the computer considered. Rather, the developers found a way to narrow its view, to stop pursuing some potential future moves, based on how it rated their outcome. Without this ability, Deep Blue would have needed to be an even more powerful computer to actually beat Kasparov.

Similarly, Google’s AlphaGo, which has beaten top human Go experts, can’t evaluate all potential future moves either. Its analysis method is more sophisticated than Deep Blue’s, using a neural network to evaluate game developments.

These methods do improve the ability of AI systems to play specific games better, but they can’t be easily changed or applied to other situations. These computerized imaginations have no concept of the wider world – meaning they can’t function beyond the specific tasks they’re assigned and are easily fooled.

They can’t interactively participate in the world, the way we imagine AI systems one day might. Instead, these machines will behave exactly the same way every time they encounter the same situation. This can be very good for ensuring an AI system is trustworthy: You want your autonomous car to be a reliable driver. But it’s bad if we want machines to truly engage with, and respond to, the world. These simplest AI systems won’t ever be bored, or interested, or sad.

Type II AI: Limited memory
This Type II class contains machines can look into the past. Self-driving cars do some of this already. For example, they observe other cars’ speed and direction. That can’t be done in a just one moment, but rather requires identifying specific objects and monitoring them over time.

These observations are added to the self-driving cars’ preprogrammed representations of the world, which also include lane markings, traffic lights and other important elements, like curves in the road. They’re included when the car decides when to change lanes, to avoid cutting off another driver or being hit by a nearby car.

But these simple pieces of information about the past are only transient. They aren’t saved as part of the car’s library of experience it can learn from, the way human drivers compile experience over years behind the wheel.

So how can we build AI systems that build full representations, remember their experiences and learn how to handle new situations? Brooks was right in that it is very difficult to do this. My own research into methods inspired by Darwinian evolution can start to make up for human shortcomings by letting the machines build their own representations.

Type III AI: Theory of mind
We might stop here, and call this point the important divide between the machines we have and the machines we will build in the future. However, it is better to be more specific to discuss the types of representations machines need to form, and what they need to be about.

Machines in the next, more advanced, class not only form representations about the world, but also about other agents or entities in the world. In psychology, this is called “theory of mind” – the understanding that people, creatures and objects in the world can have thoughts and emotions that affect their own behavior.

This is crucial to how we humans formed societies, because they allowed us to have social interactions. Without understanding each other’s motives and intentions, and without taking into account what somebody else knows either about me or the environment, working together is at best difficult, at worst impossible.

If AI systems are indeed ever to walk among us, they’ll have to be able to understand that each of us has thoughts and feelings and expectations for how we’ll be treated. And they’ll have to adjust their behavior accordingly.

Type IV AI: Self-awareness
The final step of AI development is to build systems that can form representations about themselves. Ultimately, we AI researchers will have to not only understand consciousness, but build machines that have it.

This is, in a sense, an extension of the “theory of mind” possessed by Type III artificial intelligences. Consciousness is also called “self-awareness” for a reason. (“I want that item” is a very different statement from “I know I want that item.”) Conscious beings are aware of themselves, know about their internal states, and are able to predict feelings of others. We assume someone honking behind us in traffic is angry or impatient, because that’s how we feel when we honk at others. Without a theory of mind, we could not make those sorts of inferences.

While we are probably far from creating machines that are self-aware, we should focus our efforts toward understanding memory, learning and the ability to base decisions on past experiences. This is an important step to understand human intelligence on its own. And it is crucial if we want to design or evolve machines that are more than exceptional at classifying what they see in front of them.


Can LSTM Nets be speed up by GPU? I am training LSTM Nets with Keras on a small mobile GPU. The speed on GPU is slower then on CPU. I found some articles that say that it is hard to train LSTMs (RNNs) on GPUs because the training cannot be parallelized.

What is your experience? Is LSTM training on large GPUs like 1080 Ti faster then on CPU?

From nvidia www (https://developer.nvidia.com/discover/lstm):

Accelerating Long Short-Term Memory using GPUs

The parallel processing capabilities of GPUs can accelerate the LSTM training and inference processes. GPUs are the de-facto standard for LSTM usage and deliver a 6x speedup during training and 140x higher throughput during inference when compared to CPU implementations. cuDNN is a GPU-accelerated deep neural network library that supports training of LSTM recurrent neural networks for sequence learning. TensorRT is a deep learning model optimizer and runtime that supports inference of LSTM recurrent neural networks on GPUs. Both cuDNN and TensorRT are part of the NVIDIA Deep Learning SDK.


Where can we use the Bayes' theorem in artificial intelligence? Th Bayes' rule specifies how an agent should update its belief in a proposition based on a new piece of evidence. Suppose an agent has a current belief in proposition $h$ based on evidence $k$ already observed, given by $P(h \mid k)$, and subsequently observes $e$.

In artificial intelligence, where can we use the Bayes' rule? I am unable to understand this concept.

A toy example:

Suppose an agent has information about the reliability of fire alarms. It may know how likely it is that an alarm will work if there is a fire i.e. $P(alarm | fire)$. But, if the problem is tweaked a bit and if the agent must know the probability that there is a fire, given that there is an alarm, it can use Bayes' rule:

$P(fire | alarm) = [P(alarm | fire) ×P(fire )] / P(alarm )$

where:

$P(alarm | fire)$ is the probability that the alarm worked, assuming that there was a fire. It is a measure of the alarm's reliability.

The expression $P(fire)$ is the probability of a fire given no other information. It is a measure of how fire-prone the building is.

$P(alarm)$ is the probability of the alarm sounding, given no other information.


A* and uniform-cost search are apparently incomplete Consider the following diagram of a graph representing a search space.

If we start at $B$ and try to reach goal state $E$, the lowest-cost first search (LCFS) (aka uniform-cost search) algorithm fails to find a solution. This is because, $B$ selects $A$ over $C$ to expand as $f(A)=g(A)=36 < f(C)=g(C)=70$. $f(n)$ is the cost function of node $n$, and $g(n)$ is the cost of reaching node $n$ from the start state. Continuing further, from $A$, LCFS will now select $B$ to expand, which in turn will select $A$ again over $C$. This leads to an infinite loop. This shows LCFS is incomplete (not guaranteed to find a solution, if one exists).

For A*, we define $f(n)=g(n)+h(n)$, where $h(n)$ is the expected cost of reaching goal state from node $n$. If we define Manhattan distance ($L_0$ norm) for $h(\cdot)$, books (such as Artificial Intelligence: A Modern Approach (3rd Ed) by Stuart Russell and Peter Norvig) says A* is bound to find the solution (since it exists). However, I couldn't find how. Using, A*, $B$ will still select $A$ since $f(A)=36+(h(A)=40)=76 < f(C)=70+(h(C)=30+50)=150$. You see, this means, when $A$ expands back $B$, $B$ will again select $A$, and an infinite loop ensues.

What am I missing here?

You forgot to calculate and take into account the costs of the actual paths. You forgot to accumulate the cost of the edges for going forward and backward multiple times!

The evaluation function of uniform-cost search (UCS) is $f(n) = g(n)$, where $g(n)$ represents the cost of the path from the start node to $n$. The evaluation function of A* is $f(n) = g(n) + h(n)$, where $h(n)$ is an admissible heuristic. UCS is a special case of A*, with the admissible heuristic $h(n) = 0, \forall n$. The evaluation function is used to choose the next node to visit from the fringe, which is the set of nodes that can potentially be visited. Whenever we visit a node, we remove it from the fringe. To expand a node $X$ means to add the children of $X$ to the fringe. Whenever you visit a node, you will also expand it.

Let's apply UCS to your specific example. Initially, we check whether $B$ is a goal node or not. It is not, so we expand $B$. We can add $B$ to a list of visited (or expanded) nodes (graph search) or not (tree search). Let's use the tree search, so we will not be keeping track of the expanded nodes, which means that we could expand a node more than once. $B$ has two children, $A$ and $C$, so we add them to the fringe, $\mathbb{F} = \{A, C\}$. Should we now visit $A$ or $C$? We will visit the one with the smallest value of the evaluation function, which is $A$, given that $f(A)=g(A)=36 < f(C)=g(C)=70$, so we remove $A$ from the fringe, which is now $\mathbb{F} = \{ C \}$. Is $A$ a goal node? No, so we expand it, but it only has one child, $B$, so we add $B$ to the fringe, so $\mathbb{F} = \{ C, B \}$. The cost of the path to reach $B$ by going first to $A$ and then to $B$ is $f(B) = 36 + 36 = 72$ (given that you go back and forward on the same edge, so you need to accumulate the cost of these trips!) and $f(C) = 70$, so we visit $C$, so we remove it from the fringe, which is now $\mathbb{F} = \{ B \}$.

You should be able to work out the remaining search (I haven't actually done it). I suggest you watch the video Uniform Cost Search, by John Levine, who shows a concrete example of how UCS and A* work.


How to interpret this learning curve of my neural network? How to interpret the following learning curves?
Background: The accuracy starts at 50%, because the network has a binary output (0 or 1). I chose an exponentially decreasing learning rate of the optimizer - I believe that this the reason why the network starts learning after 10 epochs or so.
lr_schedule = keras.optimizers.schedules.ExponentialDecay(**h_p["optimizer"]["adam"])
optimizer = keras.optimizers.Adam(learning_rate=lr_schedule)
h_p["Compile"]["optimizer"] = optimizer

It looks like your network is overfitting, because the training loss carries on decreasing to zero even though validation loss levels off, and then starts to increase again.
I would guess that your network is essentially "memorising" the training examples because you're getting a near zero loss in training.
You could try:

applying some form of regularisation
reducing the number of neurons or layers
collecting more training data to see if that helps.

Ideally you want to minimise validation loss, so epoch 20 was about the best point in your graph. Look into early stopping: this is an effective way to avoid overfitting too much.
When the learning rate is too high, you often see the loss 'jumping around' quite rapidly, and when the learning rate is too low, the graph barely changes at all. You could try fixing the learning rate closer to what it is around epoch 10 and see if this is "the sweet spot"; it's mostly a game of experimenting to see which works best.


What is the difference between evolutionary computation and evolutionary algorithms? A book on evolutionary computation by De Jong mentions both the term evolutionary algorithms (EA) as well as evolutionary computation (EC). However, it remains unclear to me what the difference between the two is. According to Vikhar, EA forms a subset of EC. However, it remains unclear to me what sort of topics/algorithms would be considered EC but not EA. Is there a clear difference between the two? If so, what is this difference?

As you can find here:

  Evolutionary algorithms form a subset of evolutionary computation in that they generally only involve techniques implementing mechanisms inspired by biological evolution such as reproduction, mutation, recombination, natural selection, and survival of the fittest.

It means, other types of evolutions, which are not necessarily a biological evolution, are in EC, but not in EA. For example, Learning classifier system is in EC as it is evolutionary, but not completely in EA as it is not biological.


What layers to use in a Neural Network for card game I am currently writing an engine to play a card game and I would like for an ANN to learn how to play the game. The game is currently playable, and I believe for this game a deep-recurrent-Q-network with a reinforcement learning approach is the way to go.

However, I don't know what type of layers I should use, I found some examples of Atari games solved through ANN, but their layers are CNN (convolutional), which are better for image processing. I don't have an image to feed the NN, only a state composed of a tensor with cards in the player's own hand and cards on the table. And the output of the NN should be a card or the action 'End Turn'.

I'm currently trying to use TensorFlow but I'm open to any library that can work with NN. Any type of help or suggestion would be greatly appreciated!

The game state consists of the location of all the hidden cards, so you probably need a softmax layer, 52*n, where n is the number of locations.

I'm not very sure that a NN is a good match.


Suitable reward function for trading buy and sell orders I am working to build an deep reinforcement learning agent which can place orders (i.e. limit buy and limit sell orders). The actions are {"Buy": 0 , "Do Nothing": 1, "Sell": 2}.

Suppose that all the features are well suited for this task. I wanted to use just the standard "Profit & Loss" as a reward, but I hardly thought to get something similar to the above image. The standard P&L will simply place the pair (limit buy order, limit sell order) on every up movement. I don't want that because very often it won't cover the commission and it is not a good indicator to trade manually. I would be interested that the agent can maximize the profit and give me a minimum profit of $100 on every pair (limit buy order, limit sell order). On the picture, I would be interested in something similar to the above picture.

Is there a reward function that could allow me to get such a result? If so, what is it?

UPDATE

Is the following utility function can work with the purpose of that question?

$$
U(x) = \max(\$100, x)
$$

That seems correct, but I don't know how the agent will be penalized if it covers a wrong transaction, i.e. the pair (limit buy order, limit sell order) creates a loss of money.

Generally researchers (Ghandar et al, Michalewicz, Lam) have used the profit or return on investment (ROI) as a reward (fitness) function.
$ROI = \frac{ \left[\sum_{t=1}^T (Price_t - sc) \times I_s(t) \right] - \left[ \sum_{t=1}^T (Price_t + bc) \times I_b(t) \right] }{ \left[ \sum_{t=1}^T (Price_t + bc) \times I_b(t) \right] }$
where $I_b(t)$ and $I_s(t)$ are equal to one if a rule signals a buy and sell, respectively, and zero otherwise; $sc$ represents the selling cost and $bc$ the buying cost. ROI is the difference between final bank balance and starting bank balance after trading.
You are correct, that the machine learning algorithm will then be influenced by spikes just before a sell.
Nicholls et al showed that using the average profit or area under the trade resulted in better performing trading rules. This approach was used by Schoreels et al. This approach focuses on being in the market to capitalize on profit. It does not penalize the trading rule when it is in the market and the market is going down. The accumulated asset value (AAV) is defined as:
$AAV = \frac{\sum_{i=1}^N [(Price_s - sc) - (Price_b + bc)]}{N}$
where $i$ is a buy and sell trading event, $N$ is the number of buy and sell events, $s$ the day the sale took place, and $b$ is the day the purchase took place.
Nicholls MSc thesis [available April 2019] showed that the fitness function used by Allen and Karjalainen is the preferred fitness function when evolving trading rules for the JSE using evolutionary programs.
Allen and Karjalainen used a fitness function based on the compounded excess returns over the buy-and-hold (buy the first day, sell the last day) strategy. The excess return is given by:
$\Delta r = r - r_{bh}$
where the continuously compounded return of the trading rule is computed as
$r = \sum_{t=1}^T r_i I_b(t) + \sum_{t=1}^T r_f I_s(t) + n\log\left(\frac{1-c}{1+c'}\right)$
and the return for the buy-and-hold strategy is calculated as
$r_{bh} = \sum_{t=1}^T r_t + \log\left(\frac{1-c}{1+c'}\right)$
In the above,
$r_i = \log P_t - \log P_{t-1}$
and $P$ is the daily close price for a given day $t$, $c$ denotes the one-way transaction cost; $r_f$ is the risk free cost when the trader is not trading, $I_b(t)$ and $I_s(t)$ are equal to one if a rule signals buy and sell, respectively, and zero otherwise; $n$ denotes the number of trades and $r_{bh}$ represents the returns of a buy-and-hold, while $r$ represents the returns of the trader.
A fixed trading cost of $c = 0.25\%$ of the transaction was defined but this could be anything like a STATE fee + Brocker fee + Tax, and might even be 2 different values, one for buying and one for selling. Which was the approach used by Nicholls. The continuously compounded return function rewards an individual when the share value is dropping and the individual is out of the market. The continuously compounded return function penalises the individual when the market is rising and the individual is out of the market.
I would recommend that you use the compounded excess return over the buy and hold strategy as your reward function.


How is the "right to explanation" reasonable? There has been recent uptick in interest in eXplainable Artificial Intelligence (XAI). Here is XAI's mission as stated on its DARPA page:

  The Explainable AI (XAI) program aims to create a suite of machine learning techniques that:

  Produce more explainable models, while maintaining a high level of learning performance (prediction accuracy); and
  Enable human users to understand, appropriately trust, and effectively manage the emerging generation of artificially intelligent partners.

The New York Times article Can A.I. Be Taught to Explain Itself? does a good job at explaining the need for XAI from a human interest perspective as well as providing a glancing outlook on the techniques being developed for the same. The force behind XAI movement seems to center around the (up and coming?) concept of right to explanation, that is, the requirement that AI applications that significantly impact human lives via their decisions be able to explain to stakeholders the factors/reasons leading up to said decision.

How is the right to explanation reasonable, given the current standards at which we hold each other accountable?

How is the right to explanation reasonable, given the current standards at which we hold each other accountable?

In short, it is quite reasonable.

More specifically, making AI accountable and responsible for explaining the decision seems reasonable because

Humans (DARPA in this case) has chosen to create, raise and evolve AI with the tax-payers money. In our society as well, whenever humans have come together for some purpose (some formal arrangement like government or otherwise), accountability and responsibility is assumed and assigned. Thus, expecting AI (who will take decisions on our behalf) to explain their decisions seems only natural extension of how humans currently operate.
Humans (generally at least) don't have super-power or resources to control and manipulate the rest of the population. But when they do (like our political leaders), we want them to be accountable.
In rare cases, powerful humans become powerful due to things which are accessible to everyone, while AI's super-power won't be. So, our civilization would feel safer and thus less insecure with an AI which doesn't shut the door on face when questions are asked.

Other benefits of AI that offers an explanation on how a decision is reached

Far easier to debug and improve in its early stage.
Can be customized further (and along the way) to amplify certain aspects like social-welfare over profitability, etc.


How do I change the values of a neural net I'm trying to have a go at building a neural net, but I can't seem to figure out how to optimise the connections.

I've tried to have a look online and it came up with "backpropagation". I looked through some pages about it, but I can't seem to understand it.

It seems to be where you decide on a target value for each node of your output, and adjust the weights of the synapses to bring the values closer to their targets.

What about the inactive synapses (synapses giving a value of 0 because the previous neuron wasn't activated (its values didn't pass the threshold))? Do those stay the same?
How would this configure the hidden layers? Do I have to assign target values to them? How?
What are the other ways that the connections can be adjusted? What alternatives are there to using target values?

This is too broad a topic to answer directly.

If you are at the beginner stage with neural networks, you will need to learn some basic theory of the maths of neural networks, before the code will make sense. Although it is possible to write neural network code with only a vague understanding of what is going on, it is not a great way to learn for the future, and more advanced neural network features will likely remain beyond your comprehension.

The maths for back propagation are not that difficult conceptually, it is literally just the Chain Rule from basic calculus applied repeatedly. You do this to get a gradient that tells you the direction that an error function would increase in, then take a small step in the opposite direction. Repeat this over time and the error should reduce.

Despite the simplicity once you already know it, there are a lot of moving parts to training a neural network. The formulae for back propagation include multiple symbols with different meanings, and typically indexed in at least 3 dimensions all at once. It can look like a wall of impenetrable maths, especially if you have got a bit rusty at basic calculus and matrix multiplication, and need to review it.

The answer is to take your time and study the basics carefully. There are many resources out there. Introductory material should cover:

Revision of basic linear algebra. Just vectors, matrices and how to add and multiply them.
Revision of basic calculus. Just differentiation and gradients. The chain rule is handy for later.
Linear regression and logistic regression models trained using gradient descent. These are introductory models that naturally lead to layered neural networks.
Neural networks and back propagation.

There are many courses and books that attempt to teach you these things. It is hard to say in general which one would work best for a particular student. The best thing to do would be to go to a learning resource that you already like, such as Coursera or EDX, and search for "neural networks beginner" or similar.

Here are a couple of courses that I can personally recommend. I cannot create a comprehensive list, so if these do not apply on a quick browse (or if the links finally go stale after a few years), then you should search for something more suitable bearing in mind the syllabus suggestions above:

Andrew Ngs' Machine Learning covers more than just neural networks, but the extra parts apply to other aspects of ML, such as sensible approaches to validating and testing that anyone building predictive models should get to know. Code exercises are in Matlab or Octave.
Andrew Ngs' Introduction to Deep Learning starts with basic regression, and moves very quickly to practical builds of neural networks using TensorFlow. It is part of a five course "Specialisation" that works up to advanced NN models developed in the last three years.
Geoffrey Hinton's Neural Networks for Machine Learning covers some historical models (such as perceptrons) and some more obscure models, and the material is hard to understand, but quite rewarding if you are up to it. Not generally recommended to beginners, but if you have solid maths and computing knowledge already, at degree level, but never looked at a neural network before, this might be the right style of course for you.

They all happen to be Coursera courses . . . I am not affiliated, it is just that the relevant courses that I have personally taken have been on that platform.


Finding the optimal policy from a set of fixed policies in reinforcement learning This is an open-ended question.Suppose I have a reinforcement learning task that is being solved using many different fixed policies, one of which is optimal. The goal of the agent is not to figure out what the optimal policy is but rather which policy (from a set of predefined fixed policies) is the optimal one.
Are there any algorithms/methods that handle this?
I was wondering if meta learning is the right area to look into?

The quickest way to do this would be to use policy evaluation methods. Most of the standard optimal control algorithms consist of policy evaluation plus a rule for updating the policy.
It may not be possible to rank arbitrary policies by performance when considering all states. So you will want to rank them according to some fixed distribution of state values. The usual distribution of start states would be a natural choice (this is also the objective when learning via policy gradients in e.g. Actor-Critic).
One simple method would be to run multiple times for each policy, starting each time according to the distribution of start states, and calculate the return (discounted sum of rewards) from each one. A simple Monte Carlo run from each start state would be fine, and is very simple to code. Take the mean value as your estimate, and measure the variance too so you can establish a confidence for your selection.
Then simply select the policy with the best average value in start states. You can use the variance to calculate a standard error for this, so you will have a feel for how robust your selection is.
If have a large number of policies to select between, you could do a first pass through with a relatively low number of samples, and try to rule out policies that perform badly enough that even adding say 3 standard errors to the estimated value would not cause them to be preferred. Other than that, the more samples you can take, the more accurate your estimates of mean starting value for each policy will be, and the more likely you will be to select the right policy.

I was wondering if meta learning is the right area to look into?

In general no, but you might want to consider meta learning if:

You have too many policies to select between by testing them all thoroughly.

The policies have some meaningful low dimension representation that is driving their behaviour. The policy function itself would normally be too high dimensional.

You could then use some form of meta-learning to predict policy performance directly from the representation, and start to skip evaluations from non-promising policies. You may need your fixed policies to number in the thousands or millions before this works though (depending on the number of parameters in the representation and complexity of mapping between parameters and policy function), plus you will still want to thoroughly estimate performance of candidates selected as worth evaluating by the meta-learning.
In comments you suggest treating the list of policies as context-free bandits, using a bandit solver to pick the policy that scores the best on average. This might offer some efficiency over evaluating each policy multiple times in sequence. A good solver will try to find best item in the list using a minimal number of samples, and you could use something like UCB or Gibbs distribution to focus more on the most promising policies. I think the main problem with this will be finding the right hyperparameters for the bandit algorithm. I would suggest if you do that to seed the initial estimates with an exhaustive test of each policy multiple times, so you can get a handle on variance and scale of the mean values.


Are there deep networks that can differentiate object class from individual object? We usually categorize objects in a hierarchy of classes. Let us say crow vs bird. In addition, classes can be "messy", for instance a crow can be also a predator, but not all birds are predators.

My question is, can deep networks represent these hierarchies easily? Has anybody studied that? (I could not find anything at all).

In this case, you have an ontology and want to learn the ontology. There are many researches in this topic that you can find. However, the data could be the most challenging part. Some of the researches:

Ontology learning for the Semantic Web
Ontology Learning

Also, as these are some frameworks to ontology learning, you can use deep networks such as RNN to learn the task.


How to use a Generative Adversarial Network to generate images for developmental analysis? I want to generate images of childrens' drawings consistent with the developmental state of children of a given age.  The training data set will include drawings made by real children in a school setting.  The generated images will be used for developmental analysis.

I have heard that Generative Adversarial Networks are a good tool for this kind of problem. If this is true, how would I go about applying a GAN to this challenge?

A generative adversarial network is probably not the best approach for generating the images desired.  We can assume from the comments that the data is not collected.  That's a good thing, because a set of rasterized images, labeled with student age or grade is an inferior input form.

It appears that access to a student population is planned or already negotiated, which is also good.

Although the drawing, as it is being drawn, is seen through each student's eyes, the primary features correlated with drawing skill development is motor control, shape formation, and color choice.  If the sheet of paper is placed over a drawing tablet, the tablet's incoming USB stream events are captured to a file, and the color selection is somehow recorded or automatically determined by having students hold the pencil or crayon up to the computer's camera before using it, a much better in natura input stream can be developed.

Pre-processing can lead to an expression of each drawing experience as a sequence of events arranged in temporal order with the following dimensions for each event.

Relative time from the instruction to draw in seconds
Color
Nearest x grid
Nearest y grid
Pressure

Determining color from camera input may be developed using LSTM approaches.

The dimensions of the label for each of these sequences would be those demographics and rankings that would most closely correlated with developmental stages.

Student age
Student gender
Curriculum grade (-1, 0, 1, 2, ... 12, where -1 is preschool and 0 is kindergarten)
Identifier of the drawing instructions given to the class
Grade ranking of the student in the class

The micro-analysis attached to each ELEMENT in the sequence includes these additional dimensions.

Drawing rate of the utensil given by $r = \frac {\sqrt{(x - x_p)^2 + (y - y_p)^2}} {t - t_p}$ where the subscript p indicates the values are drawn from the previous event in the sequence.
Drawing direction given by $\theta = \arctan (x - x_p, \; y - y_p)$
Curvature $\kappa$ calculated using cubic splines or some other data fitting approach
FFT spectrum $\vec{a}$ and Lyapunov exponent $\lambda$ applied to auto-correlation results

This is a modification of the system Google uses to synthesize speech, based on the WaveNet design. In the diagram, the residual function is defined as follows.

$z = \tanh \, (W_{f,k} x + V_{f,k} y) \, \odot \, \sigma \, (W_{g,k} x + V_{g,k} y)$

The development required is that the $\vec{a}$ must now be accompanied with scalars $r, \theta, \kappa, and \lambda$, but the resulting drawings are likely to have many of the hand-eye developmental features of the examples.


How can I create an embedding layer to convert words to a vector space from scratch? For an upcoming project, I am trying to build a neural network for classifying text from scratch, without the use of libraries. This requires an embedding layer, or a way to convert words to some vector representation. I understand the gist, but I can't find any deep explanations or tutorials that don't start with importing TensorFlow. All I'm really told is that it works by context using a few surrounding words, but I don't understand exactly how.
Is it much different from a classic network, with weights and biases? How does it figure out the loss?
If someone could point me towards a guide to how these things work exactly I would be very grateful.

Word2vec embedding are trained by simple auto-encoder model that takes a word and tries to predict one word form the window of surrounding words.

You could define it like this:
num_of_words = 50000
# one hot encoded word
input = Input(num_of_words)
# You could use non linear activation
w2v = Dense(300, activation=”linear”)(input)
output = Dense(num_of_words, activation=”softmax”)(w2v)

But in practice, the model is redefined and takes two words as input and predicts next words. And it outputs a probability score for all the words it knows (the model’s “vocabulary”, which can range from a few thousand to over a million words).

It is trained both ways from beginning to end of sentence and reverse. Loss is a regular categorical_crossentropy Detailed explenation can be found here [http://jalammar.github.io/illustrated-word2vec/]


Why do we need both the validation set and test set? I know that this has been asked a hundred times before, however, I was not able to find a question (and an answer) which actually answered what I wanted to know, respectively, which explained it in a way I was able to understand. So, I'm trying to rephrase the question…

When working with neural networks, you typically split your data set into three parts:

Training set
Validation set
Test set

I understand that you use the training set for, well, train the network, and that you use the test set to verify how well it has learned: By measuring how well the network performs on the test set, you know what to expect when actually using it later on. So far, so good.

Now, a model has hyper parameters, which – besides the weights – need to be tuned. If you change these, of course, you get different results. This is where in all explanations the validation set comes into play:

Train using the training set
Validate how well the model performs using the validation set
Repeat this for a number of variants which differ in their hyperparameters (or do it in parallel, right from the start)
Finally, select one and verify its performance using the test set

Now, my question is: Why would I need steps 2 and 3? I could as well train multiple version of my model in parallel, and then run all of them against the test set, to see which performs best, and then use this one.

So, in other words: Why would I use the validation set for comparing the model variants, if I could directly use the test set to do so? I mean, I need to train multiple versions either way. What is the benefit of doing it like this?

Probably, there is some meaning to it, and probably I got something wrong, but I can't figure out what. Any hints?

The difference between the validation and test set in my opinion should be explained in this way:

the validation set is meant to be used multiple times.
the test set is meant to be used only once.

I think that the misunderstanding here arise because machine learning is mostly taught focusing only on a specific part of a large pipeline, which is the model training. In every tutorial standard datasets are used so that you don't have to worry about data collection, data labelling (it's really sad to see that lot of people have not a clue about what the inter annotator agreement is), data pre-processing and especially, all the part about the real application of the model is almost never mentioned.

The importance of having a set of instances that you can use for fine tuning (validation) and a set of instances that your model never encountered neither in training nor during fine tuning (test) becomes particularly clear if you focus on the subsequent deployment of the model you trained. No one expect a model to have the same performance scores in training and when applied to some unknown data. And the crucial point is that the performance of a model on the validation set are not representative either of the behaviour of a model with unknown data, because the same validation data have been used to fine tuned the model! So here's why having a set of data completely new to the model is important, because it gives you a much more unbiased view about the model performance on a real use case scenario.


How should I design the LSTM architecture for multivariate time series forecasting problems? There is plenty of literature describing LSTMs in a lot of detail and how to use them for multi-variate or uni-variate forecasting problems. What I couldn't find though, is any papers or discussions describing time series forecasting where we have correlated forecast data.

An example is best to describe what I mean. Say I wanted to predict number of people at a beach for the next 24 hours and I want to predict this in hourly granularity. This quantity of people would clearly depend on the past quantity of people at the beach as well as the weather. Now I can make an LSTM architecture of some sort to predict these future quantities based upon what happened in the past quite easily. But what if I now have access to weather forecasts for the next 24 hours too? (and historical forecast data too).

The architecture I came up with looks like this:

So I train the left upper branch on forecast data, then train the right upper branch on out-turn data, then freeze their layers to and join them to form the final network in the picture and train that on both forecasts and out-turns. (when I say train, the output is always the forecast for the next 24 hours). This method does in fact have better performance than using forecasts or out-turns alone.

I guess my question is, has anyone seen any literature around on this topic and/ or knows a better way to solve these sort of multivariate time series forecasting problems and is my method okay or completely flawed?

I have not come across a model like this yet. BUT

If you have not tried smaller models I'd recommend trying that first.

Justification: this lets you use learning curves to diagnose what to do next.

Also, you might try starting with a GRU (the LSTM overhead may not be needed).

One Idea for a starting model

Observe that turnout does not affect weather. So the weather forecast model will be independent of the turnout model but not vice versa.

(Note: I'm using RNN to denote which ever recurrent architecture you choose)

Formulation for simultaneous prediction:

Weather

$\text{RNN}_w$ is your weather prediction model

$\hat w_t = \text{RNN}_w(\hat w_{t-1})$ be the predicted weather at time $t$

Turnout: The idea here is that people will or will not go to the beach for various reasons (overcrowded, too hot or stormy etc). So in the beach population prediction task we use all these as features to predict the population at the next time-step. This reduces the problem to any one of the classical models already developed.

$\text{RNN}_p$ is your turnout (beach population) prediction model

$\hat p_t$ be the predicted turnout at time $t$

$\hat c_t=[\hat p_t, \hat w_t]$ is the concatenation at time t

$\hat p_{t+1} = \text{RNN}_p(\hat c_t)$

Formulation in the presence of weather forecast:

Simply replace $\hat w_t$ with the true weather forecasts $\hat f_t$.

A final warning

Unless you think your RNN is better at weather prediction than the forecasting systems I would not recommend using $\text{RNN}_w$ in a production application.

I hope this helps.


If I trained a model to perform semantic segmentation on images with only one object, would it also work on images with multiple objects? I'm working on semantic segmentation tasks in the medical space using the U-Net. Let's say that I train a U-Net model on medical images with the goal of segmenting out, say, ligaments, from a medical image. If I train that model on images that contain just a single labelled ligament, it will be able to segment out single ligaments pretty well, I assume. If I present it with an image with multiple ligaments, should it also be able to segment the multiple ligaments well too?
Based on my understanding, semantic segmentation is just pixel-wise classification. As a result, shouldn't the number of the objects in the image not be relevant since it's only looking at individual pixels? So, as long as a pixel matches that of a ligament, it should be able to segment it equally right?
Or am I misunderstanding some piece?
Basically, if I train a U-Net on images with just single ligaments, will it also be able to segment images with multiple ligaments equally as well based on my logic above?

Without experimental evidence to back me up, I can not answer this with 100% confidence. However, I am fairly certain that this will cause issues depending on the model.

U-net is essentially an auto-encoder, and due to the fact that it is all just one big neural network, it is likely it will learn the easiest pattern (as all NN do), and that is to find one single instance of an object and shade that region.

Now why does this depend on the model? Well let's say you are using something slightly different, where region proposals are generated by a determinisitc algorithm we've predefined, then these regions are run through a CNN to segment them. In this case, as each region is without context of the entire image, the difference between 2 objects in an image and 1 is indistinguishable to the network (as regions may overlap), and as such, only using images with 1 object will not pose any problems (there is a name for early models like these, though it escapes me).

So assuming I am correct, what should you do? The models that use a deterministic algorithm for region proposals are slow and old, so I wouldn't suggest that. Instead, I would think that you should first do some testing, to see if it actually does cause issues. Assuming it does, a good option could be to tamper with the training data and separate segments by a few pixels to sort of "force" multiple objects into existence.

Regardless of such, I would still suggest using U-net. Fixing this issue (if it does arrise) should be relatively easy to do, so there's little to lose by using U-net and just trying the training.


What does the parameter $y$ stand for in function $g(y,\mu,\sigma)$ related to REINFORCE algorithm? I am wondering what the parameter $y$ in the function $g(y,\mu,\sigma)=\frac{1}{(2\pi)^{1/2}\sigma}e^{-(y-\mu)^{2/2\sigma^2}}$ stands for in Section 6 (page 14) of the paper introducing the REINFORCE family of algorithms.
Drawing an analogy to Equation 4 of the same paper, I would guess that it refers to the outcome (i.e. sample) of sampling from a probability distribution parameterized by the parameters $\mu$ and $\sigma$. However, I am not sure whether that is correct or not.

If you take a look at the Wikipedia page related to the normal distribution, you will see the definition of the Gaussian density
$$
{\displaystyle f(x)={\frac {1}{\sigma {\sqrt {2\pi }}}}e^{-{\frac {1}{2}}\left({\frac {x-\mu }{\sigma }}\right)^{2}}} \label{1}\tag{1}
$$
and you will see that the $y$ in your formula corresponds to the $x$ in equation \ref{1}.
I've seen this notation in the context of computer vision and image processing, where the Gaussian kernel is used to blur images.
So, as pointed out by someone in a comment, $y$ should indeed be the point where you evaluate the density.
Maybe the confusing part is that all parameters are treated equally in terms of their purpose, while $\mu$ and $\sigma$ are clearly the parameters that define the specific density, so they are not the inputs to the specific density.
After having read the relevant section of the paper, I now understand why you're confused. The author refers to $y$ as the output (not yet sure why: maybe it's the output of another unit that feeds this Gaussian unit?), but I think that this explanation still applies. The output of the Gaussian density $g$ is not $y$, but the density that corresponds to $y$. In fact, in appendix $B$ of the paper, the author says that $Y$ is the support of $g$ and $y$ is an element of $Y$.


Dimension after multiple convolutions in ConvNets I'm trying to understand exactly what does a convnet do to what, and I have trouble finding the dimensions alongside the convolutions.

If we take VGG 16 architecture, how do I get from 224x224x3 to 112x112x64 ? (The 112 is understandable, it's the last part I don't get)

I thought the CNN was to apply filters/convolutions to layers (for instance, 10 different filters to channel red, 10 to green... : are they the same filters between channels ?), but obviously 64 is not divisible by 3.

And then, how do we get from 64 to 128 ? Do we apply new filters to outputs of previous filters ? (in this case we only have 2 filters applied to previous outputs) Or is it something different ?

The 64 here is the number of filters that are used.
The picture is kind of misleading in that it leaves out the transition of the maxpool.

Below is a text description of the size of the features as they go through the network with the number of filters in bold.

The first 2 layers in the diagram you posted contain 64 3x3 convs
resulting in a 224x224x64 matrix of features.
This is then fed into a maxpool which reduces the size to a 112x112x64 matrix.
This is then fed to 3 layers of 128 3x3 convs resulting in a 112x112x128
matrix.
Then another maxpool giving a 56x56x128 matrix.
Feeding that to 3 layers of 256 3x3 convs results in a 56x56x256 matrix.
This is then fed into another maxpool giving a 28x28x256 matrix,
Which is then fed into 3 layers of 512 3x3 convs resulting in a
28x28x512 matrix.
Another maxpool gives a 14x14x512 matrix which is fed to 3 layers of 512 3x3 convs giving a matrix of 14x14x512 features.
Another maxpool reduces this to a 7x7x512 which is then
given to 3 fully connected layers of 4096 units each before being
sent to a softmax.