What is the role of the hidden vectors in restricted Boltzmann machines? I'm learning about the restricted Boltzmann machine (RBM), and I just came up with two naive understandings of this model. But it seems these two understandings are so different.

My first understanding goes like this.

The  hidden units are just structural support and we don't care about what those hidden vectors really are. We introduce those hidden units just for RBM to gain more expressive power for a probability distribution.

Let's say I have only one image, and I transform this image into a binary vector and feed this vector into an RBM with random variables (all the weights and biases are chosen randomly). Then, by turning on the machine, the first hidden vector would be constructed. But this hidden vector does not tell us anything, it would be only used to reconstruct a visible vector.
(My understanding for this reconstructed visible vector is this vector is a vector encoded in the defined RBM in the first place, we are not really construct something new, but we just happen to sample this vector from the defined RBM).
And we just run this loop of construction and reconstruction for infinitely many times. Finally, what we will get is just the probability distribution encoded in this RBM with random variables.

My second understanding goes like this.

RBM can be used to perform dimensionality reduction, and those hidden vectors are some abstract representations of the raw inputs:

Given an RBM, each hidden unit of the RBM would be a classifier, and what it does is to check the input vector lies in which side of the hyperplane defined by these hidden units (by its weights and bias). So, if we input an image into this RBM, the RBM will project this input vector onto many hyperplanes defined by all the hidden units, and thus for an input vector, the corresponding hidden vector is very important. It is some abstract representation. And we can further feed this representation into other learning models.

What is the role of the hidden vectors in restricted Boltzmann machines?

If you can answer this question by explaining how RBM is used for MNIST that would be extremely helpful for me.

Generative models

  The hidden units are just structural support and we don't care about what those hidden vectors really are.

Generative modeling is concerned about $P(X)$, to be able to compute it, we use representation learning (aka deep learning) to identify and disentangle the underlying explanatory factors hidden in the observed data,
because if we could separate them out like this

$$P(X) = P(X \mid Z) P(Z),$$

where $Z$ could encode not just the digit identity but the angle that the digit is drawn, the stroke width, and also abstract stylistic properties.

Any classifier for some task will decide which of those $z$ elements could help him to make a better decision, i.e. just the part that holds the digit identity and ignore the rest, this kind of modeling is called distributed representation, where each element is a simple indicator about some feature.

No conditional dependency within a layer

  (my understanding for this reconstructed visible vector is this vector is a vector encoded in the defined RBM in the first place, we are not really construct something new, but we just happen to sample this vector from the defined RBM)

RBM is an undirected graphical model and no conditional dependency within a layer. Thus, the conditional distribution over the hidden units $h$ given the input image $v$ factorizes, that is

$$p(h \mid v) = \prod_i p(h_i \mid v) $$

And the conditional distribution over the visible units $v$ given the hidden units $h$ also factorize.

$$p(v \mid h) = \prod_j p(v_j \mid h) $$

Inference is tractable

Inference in RBM is tractable and we could compute $p(z|x)$, but the tractability of the RBM does not extend to its partition function.

To minimize the negative log-likelihood of the data, consider the following equation.

$$-\frac{\partial \log p(x)}{\partial \theta} = \frac{\partial \mathcal{F}(x)}{\partial \theta}-\sum_{\tilde{x}} p(\tilde{x}) \frac{\partial \mathcal{F}(\tilde{x})}{\partial \theta}$$

The first term $\frac{\partial \mathcal{F}(x)}{\partial \theta}$ increases the probability of training data. The second term $\sum_{\tilde{x}} p(\tilde{x}) \frac{\partial \mathcal{F}(\tilde{x})}{\partial \theta}$ decreases the probability of samples generated by the model.

The second term is nothing less than an expectation over all possible configurations of the input $x$ (under the distribution $P$ formed by the model).

Monte Carlo-based algorithms

So, we use Monte Carlo based algorithms to replace the expectation with an average sum. Samples of $p(x)$ from the model can be obtained by running a Markov chain to convergence, using Gibbs sampling as the transition operator.

For RBMs, the set of visible and hidden units, since they are conditionally independent, one can perform block Gibbs sampling. In this setting, visible units are sampled simultaneously given fixed values of the hidden units. Similarly, hidden units are sampled simultaneously given the visible units.

A step in the Markov chain is thus taken as follows:

$v$: input values
$h$: hidden features, latent space, a.k.a $Z$
$W$: the weight parameters; note that I am using the transposed $W$,
since it's an undirected graphical model.

So, we have the formulas.

$$h^{(n+1)} \sim \operatorname{sigm}\left(W^{\prime} v^{(n)}+c\right)$$

and

$$v^{(n+1)} \sim \operatorname{sigm}\left(W h^{(n+1)}+b\right)$$

where $h^{n+1}$ refers to the set of all hidden units at the $n$th step of the Markov chain. What it means is that, for example, $h^(n+1)_i$ is randomly chosen to be $1$ (versus $0$) with the probability of the sigmoid function and similarly for $v^(n+1)_j$.

In theory, each parameter update in the learning process would require running one such chain to convergence. It is needless to say that doing so would be prohibitively expensive. As such, several algorithms have been devised for RBMs, see Contrastive Divergence.

We initialize the Markov chain with a training example (i.e., from a distribution that is expected to be close to $p$, so that the chain will be already close to having converged to its final distribution $p$).
CD does not wait for the chain to converge. Samples are obtained after only $k$-steps of Gibbs sampling. In practice, $k=1$ has been shown to work surprisingly well.

RBM for dimensionality reduction

  RBM can be used to perform dimensionality reduction, and those hidden vectors are some abstract representations of the raw inputs:

According to the paper Reducing the Dimensionality of Data with Neural Networks

  In 2006, a breakthrough in feature learning and deep learning took place (Hinton et al., 2006; Bengio et al., 2007; Ranzato et al., 2007), A central idea, referred to as greedy layerwise unsupervised pre-training, was to learn a hierarchy of features one level at a time, using unsupervised feature learning to learn a new transformation at each level
  to be composed with the previously learned transformations; essentially, each iteration of unsupervised feature learning adds one layer of weights to a deep neural network. Finally, the set of layers could be combined to initialize a deep supervised predictor, such as a neural network classifier.

Summary

Train single layer RBM to reconstruct to input and then keep adding layer after layer until we reach some good low-level representation as depicted in the figure below, W1, W2, W3 are the matrices that have some lower dimensionality of the input space.
Transpose those matrices $W_1, W_2, W_3$ to be the decoder part.
Fine-tune the whole encoder-decoder stack to reconstruct the input. image

RBM vs VAE and GAN

Side note: RBM is now obsolete, see VAE, GAN they are way better at modeling the manifold in which the data is concentrated, yet you could sample from the manifold directly with a single inference step.


How to make meaningful sentences from a set of words? I have set of topics generated using LDA and  like {code, language, test , write, function}, {class, public, method, string, int} etc and I want to make meaningful sentence/sentences from these words using  api or libraries.  How do I implement this with NLTK and(or) Machine Learning? Any suggestions as to how I should go about this?

How do you define "meaningful"? Generally, you would start from concepts and meanings, and then realise them in syntactic structures using lexical items (words). You seem to want to start in the middle somehow.

For turning a semantic representation into a valid sentence, you would use a generator; these are often based on grammars. Examples exist which take a grammar, fill in random words, and create a syntactically well-formed sentence; often they will, however, be rather non-sensical or meaningless. Have a look at this site which describes the Syntax Construction Kit. The author, Mark Rosenfelder, links to a number of toy programs which do exactly that. Just substitute his lexicon with the list of words created by your LDA process. See for example this generator based on generative grammar.


Given a sequence of states followed by the agent, is it possible to find the Q-value for a state-action pair not in this sequence? Assume you are given a sequence of states followed by the agent, generated by a random policy, $[s_0, s_1, s_2, \dots,  s_n]$. Furthermore, assume the MDP is fully observable and time is discrete.
Is it possible to find the Q-value for a state-action pair $(s_j, a_j)$ which was not encountered along this sequence?
From my understanding of the MDP, yes, it would be possible. However, I'm unsure how to get this Q-value.

My understanding from your question is that you have the following data generated from a random policy:
$$[s_0, s_1, s_2 . . . s_n]$$
That is, the state observed at each time step.
You know nothing more about the MDP, such as the transition or reward functions. Although the MDP is discrete and fully observable (and thus usual RL theory is supported), you do not have any observations other than a single list of states.
You wish to obtain an estimate for $Q(s_j, a_j)$, given a specific state/action pair that has not been observed before. This is not possible. You do not have any data from your observations regarding rewards, nor the actions taken by the random agent. At minimum, you would need data about rewards and actions that were observed. Observed rewards are needed to calculate returns, and value functions are expected returns. Observed actions are needed to assign returns to the correct $(s,a)$ pair when calculating estimates.
In addition, if you want to make a meaningful estimate for an as-yet unseen state/action pair, then you would need to be using some form of function approximation for the action value function Q. Otherwise your estimate will be whatever default estimate you had assigned at the beginning of learning.
What inferences can you make given the data you have? Very roughly:
$$p(s_{k+1} | s_k, \pi ) \gt 0, \forall k \in [0,n-1]$$
i.e. the state sequence that was observed gives you proof of non-zero transition probabilities between those states under the given policy. This is not enough to learn a value function, but might have other uses.
With repeated observations - multiple lists of observed states - you could infer what those probabilities were, but would not be able to separate the effect of action choice from state transition rules (because the action was not observed). If you added reward observations then you could learn a state value function.


Does GraphSage use hard attention? I was reading the recent paper Graph Representation Learning via Hard and Channel-Wise Attention Networks, where the authors claim that there is no hard attention operator for graph data.

From my understanding, the difference between hard and soft attention is that for soft attention you're computing the attention scores between the nodes and all their neighbors while for hard attention you have a sampling function that selects only the most important neighbors. If that is the case, then GraphSage is an example of hard attention, because they apply the attention only on a subset of each node's neighbors.

Is my understanding of hard and soft attention wrong, or the claim that the authors made does not hold?

GraphSage does not have attention at all. Yes, it randomly samples (not  most important as you claim) a subset of neighbors, but it does not compute attention score for each neighbor.


What are the differences between a knowledge base and a knowledge graph? Throughout my readings, I have seen many authors using interchangeably the two terms to refer to the same thing. However, we all know about Google's first quotation of "knowledge graph" to refer to their new way of making use of their knowledge base. Afterwards, other companies are claiming using knowledge graphs.

What are the technical differences between the two? Concrete examples will be very useful to understand better the nuances.

Based on the related Wikipedia, a knowledge base (KB) is:

  a technology used to store complex structured and unstructured information used by a computer system. The initial use of the term was in connection with expert systems which were the first knowledge-based systems.

As there are different representation model for a KB, we can find different terminology in different domains. For example, in some AI articles, it's called ontology.

Knowledeg graph (KG) is another object model to KB realization which is introduced by Google for its search engine (as you have mentioned). Hence, KG is a specification of KB. You can find more information in the paper Knowledge Graphs, such as more history about the KG or a formal definition of that:

  knowledge graph is a graph of data intended to accumulate and convey knowledge of the real world, whose nodes represent entities of interest and whose edges represent relations between these entities.

Moreover, you can find some articles about contextual KG (CKG) in the paper Learning Contextual Embeddings for Knowledge Graph Completion and KG$^2$: Learning to Reason Science Exam Questions with Contextual Knowledge Graph Embeddings.


Are Markov Random Fields and Conditional Random Fields still used in computer vision? Back before deep learning, there were a lot of different attempts at computer vision. Some involved Conditional Random Fields and Markov Random Fields, which were both computationally difficult and hard to understand/implement.
Are these areas still being developed in the computer vision domain? What was the end result of this line of study? I haven't seen any papers on this topic be cited in top-performing benchmarks, so I assume nobody cares about them anymore, but I wanted to ask.

In the Image-to-Image Translation with Conditional Adversarial Networks paper (popularly known as pix2pix), they used a Markovian Discriminator to effectively model the image as a Markovian Random Field.
There were some papers in the last 5 years concerning Markov Fields. Here are some of them:-

Combining Markov Random Fields and Convolutional Neural Networks for Image Synthesis

Precomputed Real-Time Texture Synthesis with Markovian Generative Adversarial Networks


Can I do image classification with Multi Layers Perceptron (MLP)? I'm seeking guidence here.
Can I use Multi Layers Perceptron (MLP), e.g regular flat neural networks, for image classification?

Will they perform better than Fisher Faces?

Is it difficult to do image classification with a MLP network?

It's on basic level like classifying objects and not detailed structures and patterns.

Important to me is that the MLP need to be trained with pictures that can have noise in background and different light shadows.

Multi Layers Perceptron(MLP) can be used for image classification, but it has a lot of deficiency than Convolutional Neural network(CNN). But if you compare MLP and Fisher Faces , the better one is MLP, because Fisher Faces will be increasingly difficult if adding more individuals or classes. You can make a simple MLP model, because it just has 3 layers which are an input layer, hidden layer and output layer, here a source code that you can try:

https://towardsdatascience.com/multi-layer-neural-networks-with-sigmoid-function-deep-learning-for-rookies-2-bf464f09eb7f

if you make a model, it will be based on training data, I think if make data trained composed based on noise in background and different light shadows on your image I think it will have a better performance, but remember if you are using MLP for image classification it can just predict an image on one spot, for example:
"if you train a model with the object in the middle of an image, your model can not predict it when the image is moved to the different spot".

here is pdf to see Fisher Faces performance:

https://core.ac.uk/download/pdf/84801048.pdf


How does implementation of VAE's objective function equate to ELBO? For a lot of VAE implementations I've seen in code, it's not really obvious to me how it equates to ELBO.
$$L(X)=H(Q)-H(Q:P(X,Z))=\sum_ZQ(Z)logP(Z,X)-\sum_ZQ(Z)log(Q(Z))$$
The above is the definition of ELBO where $X$ is some input, $Z$ is a latent variable, $H()$ is the equation for entropy. $Q()$ is a distribution being used to approximate distribution $P()$, which in the above case both $P()$ and $Q()$ are discrete distributions, because of the sum.
A lot of the times when VAEs are built for reconstructing discrete data types, let's say for example an image, where each pixel can be black or white or $0$ or $1$. The objective function written in code I always see is written as follows:
$Encoder(Y) \rightarrow Z_u, Z_{\sigma}$
$ReparamterizationTrick (Z_\mu, Z_\sigma) \rightarrow Z$
$Decoder(Z) \rightarrow \hat{Y}$
$L(Y)= CrossEntropy(\hat{Y}, Y) - 0.5*(1+Z_{\sigma}-Z_{\mu}^2-exp(Z_\sigma))$
$Z$ represents the latent embedding of the AutoEncoder and $Z_\mu$ and $Z_\sigma$ represent the mean and standard deviation for sampling for $Z$ from a gaussian distribution. $Y$ represents the binary image trying to be reconstructed and $\hat{Y}$ represents it's reconstruction from the VAE. As we can see from ELBO it's the entropy of the latent distribution being learned $Q()$, which is gaussian as with VAE they usually are and the cross entropy of the latent distribution being learned $Q()$ and the actual distribution $P()$ with $Z$ intersected with $X$.
The main points that confuse me is how the cross entropy(CE) of $Y$ with $\hat{Y}$ equates to the CE of the distribution for generating latents and it's gaussian approximation and how $(0.5*(1+Z_{\sigma}-Z_{\mu}^2-exp(Z_\sigma)))$ equates to the entropy? Is it just assumed the CE of $Y$ with $\hat{Y}$ also leads to the CE of the latent distribution with it's approximation cause they're part of $\hat{Y}$'s generation? Still seems a bit off cause you're getting the cross entropy of $Y$ with it's reconstruction, not the gaussian distribution for learning latents $Z$.
Note: $Z_\sigma$ is usually not softplused to be strictly positive as required by a Gaussian dist so I think that's what $exp(Z_\sigma)$ is for.

I don't want to think about the correctness of your supposed ELBO equation now. Nevertheless, it's true that the ELBO can be rewritten in different ways (e.g. if you expand the KL divergence below, by applying its definition, you will end up with a different but equivalent version of the ELBO). I will use the most common (and definitely most intuitive, at least to me) one, which you can find in the paper that originally introduced the VAE, which you should use as a reference, when you are confused (although that paper may require at least 2 readings before you fully understand it).
Here's the most common form of the ELBO (for the VAE), which immediately explains the common VAE implementations (including yours):
$$\mathcal{L}\left(\boldsymbol{\theta}, \boldsymbol{\phi} ; \mathbf{x}^{(i)}\right)=
\underbrace{
\color{red}{
-D_{K L}\left(q_{\boldsymbol{\phi}}\left(\mathbf{z} \mid \mathbf{x}^{(i)}\right) \| p_{\boldsymbol{\theta}}(\mathbf{z})\right)
}
}_{\text{KL divergence}}
+
\underbrace{
\color{green}{
\mathbb{E}_{q_{\boldsymbol{\phi}}\left(\mathbf{z} \mid \mathbf{x}^{(i)}\right)}\left[\color{blue}{ \log p_{\boldsymbol{\theta}}\left(\mathbf{x}^{(i)} \mid \mathbf{z}\right)}\right]
}
}_{\text{Expected log-likelihood}},
$$
where

$p_{\boldsymbol{\theta}}(\mathbf{z})$ is the prior over $\mathbf{z}$ (i.e. the latent variable); in practice, the prior is not actually parametrized by $\boldsymbol{\theta}$ (i.e. it's just a Gaussian with mean zero and variance one, or whatever, depending on your assumptions about $\mathbf{z}$!), but in the paper they assume that $\mathbf{z}$ depends on $\boldsymbol{\theta}$ (see figure 1).

$q_{\boldsymbol{\phi}}\left(\mathbf{z} \mid \mathbf{x}^{(i)}\right)$ is the encoder parametrized by $\boldsymbol{\phi}$

$p_{\boldsymbol{\theta}}$ is the decoder, parametrized by $\boldsymbol{\theta}$

If you assume that $p_{\boldsymbol{\theta}}(\mathbf{z})$ and $q_{\boldsymbol{\phi}}\left(\mathbf{z} \mid \mathbf{x}^{(i)}\right) $ are Gaussian distributions, then it turns out that the KL divergence has an analytical form, which is also derived in the VAE paper (appendix B)
$$
\color{red}{
-D_{K L}\left(q_{\boldsymbol{\phi}}\left(\mathbf{z} \mid \mathbf{x}^{(i)}\right) \| p_{\boldsymbol{\theta}}(\mathbf{z})\right)}
=
\color{red}{
\frac{1}{2} \sum_{j=1}^{J}\left(1+\log \left(\left(\sigma_{j}\right)^{2}\right)-\left(\mu_{j}\right)^{2}-\left(\sigma_{j}\right)^{2}\right)
}
$$
Hence this implementation, which should be equivalent to your term $0.5*(1+Z_{\sigma}-Z_{\mu}^2-exp(Z_\sigma))$
The expected log-likelihood is actually an expectation, so you cannot compute it exactly, in general. Hence, you can approximate it with Monte Carlo sampling (aka sampling averages: remember the law of large numbers?). More concretely, if you assume that you have a Bernoulli likelihood, i.e. $p_{\boldsymbol{\theta}}\left(\mathbf{x}^{(i)} \mid \mathbf{z}\right)$ is a Bernoulli, then its definitions is (again from the VAE paper, Appendix C.1)
$$
\color{blue}{ \log p(\mathbf{x} \mid \mathbf{z})}= \color{blue}{ \sum_{i=1}^{D} x_{i} \log y_{i}+\left(1-x_{i}\right) \cdot \log \left(1-y_{i}\right) }\tag{1}\label{1},
$$
where $\mathbf{y}$ is the output of the decoder (i.e. the reconstruction/generation of the original input).
This formula should be very familiar to you if you are familiar with the cross-entropy. In fact, minimizing the cross-entropy is equivalent to maximizing the log-likelihood (this may still be confusing because of the flipped signs in the ELBO above, but just remember that maximizing a function is equivalent to minimizing its negative!). Hence this loss.
To answer/address some of your questions/doubts directly

and how $(0.5*(1+Z_{\sigma}-Z_{\mu}^2-exp(Z_\sigma)))$ equates to the entropy?

I answered this above. That's just the analytical expression for the KL divergence (by the way, the KL divergence is also known as relative entropy: lol?): yes, but resolve me the logs and exponentials, though!

Still seems a bit off cause you're getting the cross-entropy of $Y$ with its reconstruction, not the Gaussian distribution for learning the latent $Z$.

As you can see from the definition of the ELBO above (from the VAE paper), the expected log-likelihood is, as the name suggests, an expectation, with respect to the encoder (i.e. the Gaussian, in case you choose a Gaussian). However, the equivalence is between the log-likelihood (which is the term inside the expectation) and the cross-entropy, i.e. once you have sampled from the encoder, you just need to compute the term inside the expectation (i.e. the cross-entropy). Your term $CrossEntropy(\hat{Y}, Y)$ represents the CE but after you have sampled a latent variable from the encoder (or Gaussian), otherwise, you could not have obtained the reconstruction $\hat{Y}$ (i.e. the reconstruction depends on this latent variable, see figure 1).
In equation \ref{1}, note that there is no expectation. In fact, in the implementations, you may just sample once from the encoder, and then immediately compute the ELBO. I have seen this also in the implementations of Bayesian neural networks (basically, normal neural networks with the same principles of the VAE). However, in principle, you could sample multiple times from the Gaussian encoder, compute \ref{1} multiple times, then average it, to compute a better approximation of the expected log-likelihood.
Hopefully, some of the information in this answer is clear. Honestly, I don't have much time to write a better answer now (maybe I will come back later). In any case, I think you can find all answers to your questions/doubts in the VAE paper (although, as I said, you may need to read it at least twice to understand it).
By the way, the simplest/cleanest implementation of the VAE that I have found so far is this one.


Which function $(\hat{y} - y)^2$ or $(y - \hat{y})^2$ should I use to compute the gradient? The MSE can be defined as $(\hat{y} - y)^2$, which should be equal to $(y - \hat{y})^2$, but I think their derivative is different, so I am confused of what derivative will I use for computing my gradient. Can someone explain for me what term to use?

The derivative of $\mathcal{L_1}(y, x) = (\hat{y} - y)^2 = (f(x) - y)^2$ with respect to $\hat{y}$, where $f$ is the model and $\hat{y} = f(x)$ is the output of the model, is

\begin{align}
\frac{d}{d \hat{y}} \mathcal{L_1}
&= \frac{d}{d \hat{y}} (\hat{y} - y)^2 \\
&= 2(\hat{y} - y) \frac{d}{d \hat{y}} (\hat{y} - y) \\
&= 2(\hat{y} - y) (1) \\
&= 2(\hat{y} - y)
\end{align}

The derivative of $\mathcal{L_2}(y, x) = (y - \hat{y})^2 = (y - f(x))^2$ w.r.t $\hat{y}$ is

\begin{align}
\frac{d}{d \hat{y}} \mathcal{L_2}
&= \frac{d}{d \hat{y}} (y - \hat{y})^2 \\
&= 2(y -\hat{y}) \frac{d}{d \hat{y}} (y -\hat{y}) \\
&= 2(y - \hat{y})(-1)\\
&= -2(y - \hat{y})\\
&= 2(\hat{y} - y)
\end{align}

So, the derivatives of $\mathcal{L_1}$ and $\mathcal{L_2}$ are the same.


How are edge features implemented in Geometric Deep Learning? The work I've seen so far have the nodes containing features. Any resources for how to use a GCN on a graph where the edges are the ones that contain features rather than the nodes?

In the paper Neural Message Passing for Quantum Chemistry (2017), the authors (from Google, Google Brain and Google DeepMind) introduce a framework called message passing neural network (MPNN), which generalizes previously proposed geometric deep learning models. In section 2 of the paper, they describe this MPNN framework and they state that edge features can also be learned, using the MPNN framework, by introducing hidden states for all edges in the graph, $\mathbf{h}_{e_{vw}}^t$, where $t$ is the iteration number and $e_{vw}$ is the edge from node $v$ to node $w$.

See also the paper Machine learning prediction errors better than DFT accuracy, which is cited in the MPNN paper as an example where the authors also learn the edge features. In section 2 of the MPNN paper, they briefly describe this specific instantiation of the MPNN framework, called the Molecular Graph Convolutions.


When training a CNN, what are the hyperparameters to tune first? I am training a convolutional neural network for object detection. Apart from the learning rate, what are the other hyperparameters that I should tune? And in what order of importance? Besides, I read that doing a grid search for hyperparameters is not the best way to go about training, and that random search is better in this case. Is random search really that good?

Firstly when you say an object detection CNN, there are a huge number of model architectures available. Considering that you have narrowed down on your model architecture a CNN will have a few common layers like the ones below with hyperparameters you can tweak:

Convolution Layer:- number of kernels, kernel size, stride length, padding
MaxPooling Layer:- kernel size, stride length, padding
Dense Layer:- size
Dropout:- Percentage to keep/drop


How to use BERT as a multi-purpose conversational AI? I'm looking to make an NLP model that can achieve a dual purpose. One purpose is that it can hold interesting conversations (conversational AI), and another being that it can do intent classification and even accomplish the classified task.

To accomplish this, would I need to use multimodal machine learning, where you combine the signal from two models into one? Or can it be done with a single model?

In my internet searches, I found BERT, developed by Google engineers (although apparently not a Google product), which is an NLP model trained in an unsupervised fashion on 3.3 billion words or more and seems very capable.

How can I leverage BERT to make my own conversational AI that can also carry out tasks? Is it as simple as copying the weights from BERT to your own model?

Any guidance is appreciated.

Think of BERT (or similar models) as as good starting place for understanding context.

A couple options to make BERT contextualize dialogue:

Concatenate all messages with a seperator embedding and finetune a language model like BERT

This has shown good results in this paper, but understand it has weaknesses like struggling to determine order or author comprehension or conversation disentanglement. Additionally as threads grow longer this becomes more and more expensive

Use A BERT like model as an RNN and pass along author/memory information as the hidden state

Weaknesses here is that you are putting a lot of strain on the hidden state of the model, and ideally you want to train each comment block separately so the gradient wouldnt need to unroll or traverse the network in backprop (I actually implemented a variant of this for my work, and had good results)

The 2 modes above are just baseline examples that you can alter for your own needs, and im sure there is also others if you brainstorm about it.

Note if you want generation capabilities, you would want to use GPT2 or XLNet instead of BERT for unidirectional embeddings. Hope this helped.


Formal definition of the Object Detection problem For many problems in computer science, there is a formal, mathematical problem defition.
Something like: Given ..., the problem is to ...
How can the Object Detection problem (i.e. detecting objects on an image) be formally defined?
Given a set of pixels, the task is to decide

which pixels belong to an object at all,
which pixels belong to the same object.

How can this be put into a formula?

This is just an idea
Given a set of pixels, the task is to decide:

Which pixel is the center of an object?
What is the size of the bounding boxes with the center is the pixel in part 1?

Formula, consider this is a 2D image, call $(x,y)$ is the horizontal and vertical coordinate and $(w_i,h_i)$ is the size of bouding box of object $i$:
$\text{For }m \in[x,x+w_i] \text{ and } n\in[y,y+h_i]$
$c_i(m,n) = \begin{cases}
    1, \text{if pixel at position (m,n) is belongs to object i,}\\
    0, \text{else}
\end{cases}$


Can optimizing for immediate reward result in a policy maximizing the return? The goal of a reinforcement learning agent is to maximize the expected return which is often a discounted sum of future rewards. The return indeed is a very noisy random variable as future rewards depend on the state-transition-probabilities and the often stochastic policy. Lots of trajectories have to be sampled to approximate its expected value.

The immediate reward indeed does not have these dependencies. Therefore the questions:

If we train a policy to maximize the immediate reward, will it also perform well in the long term? What properties would the reward function need to fulfill?

If we train a policy to maximize the immediate reward, will it also perform well in the long term?

In general, no. The delay of long term reward in real world problems, and often a lack of easy-to-compute heuristics, is a key motivation for developing reinforcement learning in the first place.

It is easy to construct a counter-example to demonstrate this. Any state where the transitions into it are high and positive, but the transitions out of it are higher and negative would "trap" an agent that only considered immediate reward. More complex traps include high immediate gains but ending an episode vs lower gains that continue for longer.

Many real-world environments have sparse rewards where it is not possible to tell the difference between two action choices by immediate reward, but the consequences of being in one part of the state space rather than another early in a trajectory are critical. Consider any two-player strategy board game for instance, where the only goal is to win at the end. Only the last move in such a game is associated with an immediate reward, but there are often important differences between early moves.

  What properties would the reward function need to fulfill?

In all states, the expected immediate reward for taking the correct long term action would need to be higher than the expected immediate reward for any other action choice.

Solving a problem framed in this way could be done with discount factor $\gamma=0$. If the action choices were always the same and valid in each state, then the problem could also be simplified to a contextual bandit, where the fact that the choices exist within a larger trajectory is not relevant.

In practice you can construct environments like this. Simple ones are possible to do manually. Doing that is similiar to adding a heuristic function for search, but with different restrictions. For many search algorithms, admissible heuristic functions are allowed to over-estimate future gains (or under-estimate costs), because a planning/search algorithm will resolve longer-term differences. In your case, you can maybe consider stochastic reward functions, but the expected reward for the correct action must always be highest.

Needing to know the correct optimal action in the first place is clearly a circular problem - if you knew it already you would have no need to perform reinforcement learning to discover the optimal policy. An exception might be if you constructed an easy environment in order to test an algorithm, and prove that it could find the optimal policy. Although even then usually you are interested in the algorithm solving a harder variant of your problem than one you have deliberately constructed to be easy.

In brief, there is no way to create a shortcut here and avoid the need to solve a harder RL problem.


Recurrent Neural Network to track distance from origin I have a game/simulation that takes a vector of encoded sequences of moves (up, down, left, right). Let's say that these are sequential step taken by an ant moving in a 2D space, starting from the origin. The moves are generated randomly.

I want to know for any game, if the ant gets farther than a certain distance y from the origin (although it might even be closer than y at the end of the game). I would like to classify games into "ant gets further away than y" with value of one, or zero for "ant does not get further away than y". I don't need an AI for this task, I have set this objective as a training goal for myself.

I am able to tell if the last position is past y or not, using a regular feed forward network, I believe it is easier because it is as easy as summing up all the moves, regardless of the order. But to tell if the ant got past y and then got back, that still needs to return one.

I thought I might be able to reach my objective through an RNN, encoding the moves as a sequence of one-hot encoded sequential directions to move towards. Currently, I am using one hidden layer (I tried with different sizes ranging from 10 to 100), backpropagating the loss only at the last step of a single training on a vector, but it seems like the RNN total loss doesn't decrease at all.

Is there any obivious flaw in my simulation, or in the neural network model? Is there a category of problems this could belong to?

This kind of problem does not really have a name other than "toy problem" since no-one needs to teach an AI to add up, multiply or divide* - there are already far more reliable and far faster ways to achieve that on any computer. What you are doing here is essentially vector addition, applying a distance metric then setting a true/false value based on a comparison. It would be 2 or 3 lines of code in most high-level programming languages.

Neural networks can learn any function though, so in theory what you want to do is possible. You should not expect results to be perfect, a statistical learner never actually learns the analytical form of a function or process, just the rough "shape" of it.

I have not done your experiment. However, your idea to use a RNN seems reasonable. With the details you have given, I can offer a few pieces of general advice:

Use a modern RNN gated architecture, either LSTM or GRU. That's because the point in the sequence of moves where you want to set a "distance exceeded" toggle could be many steps away from the end of a game. The simplest RNNs (with direct loop backs from output to input within a layer) can easily suffer from vanishing gradients in this situation, whilst LSTM and GRU architectures are designed to deal with it.
Generate a lot of training data. You will need many examples of both categories before any neural network will home in on what is causing one class or the other to occur. The learning is based on statistics, not reason.
Take a look at related LTSM examples that learn to add binary numbers. Repeat those even simpler experiments first, then move on to your own problem. This will avoid some beginner mistakes with poor choices of hyper-parameters, bad implementations etc.

* Humans of course do use intelligence, reasoning and logic to learn reliable procedures for addition, multiplication and division. No doubt someone could be interested in how an AI can replicate that, without starting from built-in capabilties of a CPU (which of course the humans designed and built those procedures into the system at a low level). However, that's at a higher level of AI research than we are dealing with here.


From what aspect to measure the performance of an object detector? I am on the hook to measure the prediction results of an object detector. I learned from some tutorials that when testing a trained object detector, for each object in the test image, the following information is provided:

    <object>
    <name>date</name>
    <pose>Unspecified</pose>
    <truncated>0</truncated>
    <difficult>0</difficult>
    <bndbox>
        <xmin>451</xmin>
        <ymin>182</ymin>
        <xmax>695</xmax>
        <ymax>359</ymax>
    </bndbox>
</object>

However, it is still unclear to me 1) how does these information is taken by the object detector to measure the accuracy, and 2) how does the "loss" is computed for this case. Is it something like a strict comparison? For instance, if for the object "date", I got the following outputs:

    <object>
    <name>date</name>
    <pose>Unspecified</pose>
    <truncated>0</truncated>
    <difficult>0</difficult>
    <bndbox>
        <xmin>461</xmin>  <---- different
        <ymin>182</ymin>
        <xmax>695</xmax>
        <ymax>359</ymax>
    </bndbox>
</object>

Then I will believe that my object detector made something wrong? Or they tolerant some small delta such that if the bounding box has a small drifting, then it's acceptable. But if the "label" is totally wrong, then that's wrong for sure?

This is like a "blackbox" to me and it would be great if someone can shed some lights on this. Thank you.

This is a high level explanation: So most object detectors are generally deep neural networks that return a set of boxes:

box1 : coordinates : confidence
 box2 : coordinates : confidence
 etc...

and loss is computed different model to model. But i think your specifically curious how most do box comparison and loss. Generally they first check if the box is even relevant to the target. To do this often they calculate the Intersection over Union (IOU) as known as the Jaccard index wikipedia link. This checks how overlapped they are biased on the box sizes. Now if its above some threshold (example: boxes have IOU over .5 caluclate a loss, otherwise loss is just 0) they will compute some objective. Objectives can differ, but ideally its something differentiable. Examples include vectorizing and approximiating the IOU, dice loss, etc..

After a quick google search, heres a good tutorial explaining some components commonly used in the field (region proposals, feature extraction, etc) [guide]


Equation not satisfied in Policy Iteration Algorithm In equation 4.9 of Sutton and Barto's book on page 79, we have(for policy iteration algo):
$\pi ^{'}(s) = arg \max_{a}\sum_{s',r}p(s',r|s,a)[r+\gamma v_{\pi}(s')]$
where $\pi$ is the previous policy and  $\pi ^{'}$ is the new policy. Hence in iterations $k$ it must mean
$\pi _{k+1}(s) = arg \max_{a}\sum_{s',r}p(s',r|s,a)[r+\gamma v_{\pi_{k}}(s')]$
But in the example given in the same book on page 77 we have:

Now for the concerned state marked in red -
So $v_{\pi_{1}}$ = -1 for all four surrounding states
r = -1 for all four surrounding states
p(s',r|s,a) = 1 for all four surrounding states
$\pi _{2}(s) = arg \max_{a}[1*[-1+1*-1],1*[-1+1*-1],1*[-1+1*-1],1*[-1+1*-1]]$
$\pi _{2}(s) = arg \max_{a}(-2,-2,-2,-2)$
Hence this should give us a criss-cross symbol(4 directional arrow) in $\pi_{2}$(s) but here a left arrow symbol is given.
What's wrong with my calculations.

Your calculations are correct, but you have misinterpreted the equations and the diagram. The index $k$ in $v_k$ for the diagram refers to the policy evaluation update iteration only, and is not related to the policy update step (which uses the notation $\pi'$ and does not mention $k$).

Policy improvement consists of multiple sweeps through states to fully evaluate the current policy and estimate the value function for it. After that, it updates the policy in a separate policy improvement step. There are two loops - an inner loop indexed by $k$ in the equations and diagram, plus an outer loop which is not given an index notation.

The diagram is not showing incremental $\pi'$ policies from outer loops over policy iteration. Instead it is showing "Greedy Policy w.r.t. $v_k$" steps in the inner loop - you can think of that as the policy $\pi'$ that you would get inside the first outer loop, if you terminated the policy evaluation stage after that iteration $k$ of the inner loop.

The diagram only shows behaviour of policy iteration for a single outer loop. It demonstrates at least two interesting things:

In the case of this very simple environment, if you ran a single outer loop with long enough policy evaluation stage ($k \ge 3$) you would find the optimal policy.
Even before the value function estimate is close to convergence (with a high $k$), the policy that could be derived from the new estimates could be used to improve the policy. That leads to the idea of the value iteration method.


Which machine learning technique can I use to match one set of data points to another? I have two measuring devices. Both measure the same thing. One is accurate, the other is not, but does correlate with a non-fixed offset, some outliers, and some noise.
I won't always be using the accurate device. The nonfixed offset makes things difficult, but I'm certain there is sufficient similarity to make a link using a machine learning (or AI) technique and to convert one set of numbers to a good approximation of the other.
One is a footbed power meter and gives power in Watts every second. The other is a crank-based power meter, also outputting Watts at 1Hz. The footbed power is much less than the crank (which I know to be accurate), but it does track the increases and decreases in power, just with more noise and, as I say, a non-fixed offset (and by non-fixed I mean, at low power the offset is different to that at high power, I don't mean it isn't consistent, it is consistent). Both measure cadence which may be a useful metric to help find a pattern.
I will be collecting sets of data from both and hoped to plug the footbed data in as a column of values with the crank data as another column representing the truth, so after training, the model would be able to transform the footbed data to an approximation of the crank data.
Anyway, I'm completely lost as to how to begin. I've tried searching, but, clearly, I'm using the wrong keywords. Does anyone have any pointers, please?

OK, so I found the answer - it is to use multiple linear regression. I think this can be marked a solved, but I don't have enough rep to do that.


Feature selection of an acoustic voice dataset for classification I'm trying to gain some insight into acoustic voice data composed of 19 features. I want to understand what features contribute most for classification.

ADDED: Most features are related with the fundamental frequency stability. In particular I'm using voice shimmer and jitter and some related calculations.

I'm trying to use MRMR (max relevance min redundancy), but I would like to compare with some other options.
ADDED: I have tried to use FeatureMiner (http://featureselection.asu.edu/index.php) which provides some interesting algorithms implementations. However many of them use deprecated Python functions and require some effort to work properly.

Are there any popular tools for these purposes?

It isn't obvious whether the 19 features are spectral and, if so, whether they contain detected transients for consonant phonics.  It's not obvious whether features were extracted using RBMs (restricted Boltzmann machines), from auto-correlation in the time domain, or some other way.  The intention of the classification, which would probably drive the nature of differentiation, isn't stated either.  If more details are added to the question, the answers may be more detailed.  Here are a few general options to compare.

Principal component analysis (PCA)
Factor analysis
Principal axis factoring
Cluster analysis
Regression or gradient descent if category labels exist and then checking early layer parameters for zeroes
Feature selection
Independent component analysis
Feature Selection: A Data Perspective, Jundong Li et. al. 2016


Why is the max a non-expansive operator? In certain reinforcement learning (RL) proofs, the operators involved are assumed to be non-expansive. For example, on page 6 of the paper Generalized Markov Decision Processes: Dynamic-programming and Reinforcement-learning Algorithms (1997), Csaba Szepesvari and Michael L. Littman state

  When $0 \leq \gamma < 1$ and $\otimes$ and $\oplus$ are non-expansions, the generalized Bellman equations have a unique optimal solution, and therefore, the optimal value function is well defined.

On page 7 of the same paper, the authors say that max is non-expansive. Moreover, on page 33, the authors assume $\otimes$ and $\oplus$ are non-expansions.

What is a non-expansive operator? Why is the $\max$ (and the $\min$), which is, for example, used in Q-learning, a non-expansive operator?

In laymen's terms, a non-expansive operator is a function that brings points closer together or at least no further apart.

An example of a non-expansive operator is the function $f(x) = x/2$. The two numbers $0$ and $5$ are a distance of $5$ apart. The two output numbers $f(0) = 0$ and $f(5) = 2.5$ are 2.5 apart (which is smaller than $5$ apart). It is easy to see that $f$ brings everything closer together except when the two input numbers are the same: in which case, the distance between the outputs of the function at those numbers is at least no further apart than distance between the two input numbers.

$\max$  is a two-input function (or n-input, but the intuition should be clear from the 2-input case).  We can think of max as a function that maps pairs of numbers $(x, y)$ to single numbers (picking whichever of $x$ and $y$ is larger).

Suppose that we chose to measure the distance between pairs using Euclidean distance, and the distance between single numbers using the Euclidean distance as well. Here's an example:

The distance between (0,0) and (3,3) is $\sqrt{3^2 + 3^2} = \sqrt{18}$. The distance between $\max(0,0) = 0$ and $\max(3, 3) = 3$ is $\sqrt{{(\max(0,0) - \max(3, 3))}^2} = \sqrt{9} = 3$.

Let's consider the general case. The Euclidean distance between the 2D points $(a, b)$ and $(c, d)$ is $\sqrt{(a-c)^2 + (b-d)^2}$. There are four cases to consider:

Suppose that $a\geq b$ and $c \geq d$. In this case, the distance between max(a,b) and max(c,d) is just |a-c|, which is clearly at most $\sqrt{(a-c)^2 + (b-d)^2}$.
Suppose that $a\leq b$ and $c \leq d$. In this case, the distance is |b-d|, which is also at most the original distance.
Suppose that $a\geq b$ but $c \leq d$. Then the distance is |a-d|. Suppose that $a > d$. Since $d > c$, then $|a-d| <= \sqrt{(a-c)^2 + (b-d)^2}$ since |a-d|<=$\sqrt{(a-c)^2}$, and a symmetric argument holds for the case $d > a$.
$a\leq b$ but $c \geq d$, we can construct an argument identical to the one for case 3 above.

Since max is always bringing things closer together, or at least, no further apart, it is a non-expansive operator.


Why is it possible to eliminate this branch with alpha-beta pruning? Can someone explain to me why it is possible to eliminate the rest of the middle branch in this image for alpha-beta pruning? I am confused because it seems the only information you know is that Helen would pick at least a 2 at the top (considering if we iterate from left to right in DFS), and Stavros would absolutely not pick anything above 7. This leaves 5 possible numbers that the rest of the branch could take on that Helen could potentially end up picking, but can't because we've eliminated those possibilities via pruning.

You can't prune the nodes that are cross out if we search from left-to-right in the tree using alpha-beta pruning. To do this analysis we can pretend the right branch of the tree doesn't exist. (Branch C from the root.)

In the left branch (A) of the root Helen will get 2 or more.

In the middle branch (B) from the root after going down the left, Stavros will get 7 or less.

Now, we can ask what happens if we put different values for the crossed out branches. If putting different values doesn't change the value at the root, then pruning is correct.

Suppose that after the 7 on branch A at the second level we make the value of branch B to be -10. In this case the value of the second branch will be -10, and Helen will prefer the first branch at the root.

Suppose that after the 7 on branch A at the second level we make the value of branch B to be 100. In this case the value of the second branch will be 7, and Helen will prefer this branch.

Thus, the value of the crossed off nodes matters, and they cannot be pruned.

You may find this tool useful for exploring alpha-beta on binary trees:

https://movingai.com/ab/


Add training data to YOLO post-training (Cross-posting here from the data science stack exchange, as my question didn't get any replies. I hope it's okay!)

I've been playing around with YOLOv3 and obtaining some good results on the ~20 custom classes I trained. However, one or two classes look like they can use some additional training data (not a lot, say about 10% more data), which I can provide.

What is the most efficient way to train my model now? Do I need to start training from scratch? Can I just throw in my additional data (with the appropriate changes to the config files etc.) and run the training based on the weight matrix I already acquired, but for a small number of iterations? (1000?) Or is this more like a transfer learning problem now?

Thanks for all tips!

Assuming that you do have a dataset (images + labels/bounding boxes) in a format that's required by the training model, you can fine-tune your existing model. You can choose to unlock the final few layers or leave all the layers unlocked during the training process. When I had performed such an experiment with RetinaNet I chose to unlock final few conv layers and was able to achieve slightly higher accuracy.


3D environment for RL research in Academia I'm doing my thesis on Reinforcement Learning. My focus on Partially Observable Environments like 3D Games. I want to choose a 3D platform for testing and doing research.

I know some of them. DeepMind Lab and OpenAi Universe. But my question is that which of these environments is good for me? Is there any environment for this purpose that is benchmark and reliable?

I want a platform that accepted in Academia and reliable. For example DeepMind is not a standard or Open Source friendly, Is it rational to use their platform for research in academia?

What i have to do?

For example DeepMind [Lab] is not a standard or Open Source friendly

I'm not sure where you got that info from... as far as I'm aware, DeepMind Lab is definitely used in various publications (maybe primarily publications from DeepMind, but still). Considering the github repo has the GNU GPL 2 license, it also seems Open Source-friendly to me.

Another framework of which I'm sure that it would widely be considered suitable within academia would be the Unity ML-Agents Toolkit, which uses the Unity game engine.

I suppose you could also consider using ViZDoom, which is also used in various publications, but (as far as I'm aware) it only supports one specific game (Doom).

I do not have enough experience with using any of the above personally to be able to recommend one of them over the others... but they would all seem suitable to me.


Could AI kill the joy of competitive sports and games? Lee Sedol, former world champion, and legendary Go player today announced his retirement with the quote "Even if I become the No. 1, there is an entity that cannot be defeated".

Is it possible that AIs could kill the joy of competitive games(Go, chess, Dota 2, etc.) or (thinking more futuristic with humanoid AIs) in sports?

What happens if AIs gets better than us at painting and making music. Will we still appreciate it in the same way we do now?

Unlikely!

Chess has been "solved" by AI much longer than GO (chess engines even before AI are way too strong for human players) and still people are playing and competing.

Simply put competition and sports live from the human element. Humans competing against each other will still create the same joy for most people regardless of the fact that all involved players might lose against a computer.

Some select individuals on the highest level might be put off by the new reality but it won't be the end of competition.

No human is faster than a car and yet we still celebrate running competitions.

Indeed I think long-term we will gain entertainment by watching different AIs and models compete against each other in chess or Go.


What does it mean to train a model? We hear this many time for different problems

  Train a model to solve this problem!

What do we really mean by training a model?

In machine learning, when you train a model, you adjust (or change) the parameters (or weights) of the model so that its performance in solving a certain task increases.

There's little difference between the idea of training a model and the idea of training an animal. In fact, here's the dictionary definition of the verb to train

  teach (a person or animal) a particular skill or type of behaviour through practice and instruction over a period of time

If you train a model, you also teach a skill or type of behavior through practice and instruction. For example, if you train a model to solve an object classification problem, then you teach the model to classify certain objects according to their properties (which is the skill that the model learns).

There are different ways to train a model, depending on the problem you want to solve, the algorithms that you use to train the model, and the available data.

If you have a labeled dataset, then you train a model with a supervisory signal (the labels), i.e. you explicitly tell the model the output that it is supposed to produce for each input, and, if it does not produce it, then you adjust its parameters so that next time it is more likely to produce the correct output for that input. This is called supervised learning (or training).

In certain cases, you do not have the correct output that the model is supposed to produce for each input, but you only have a reward (or reinforcement) signal. So, your training (or learning) algorithm needs to adjust the parameters of the model only based on the reward signal. This is called reinforcement learning (or training).

Finally, there's also unsupervised learning (or training), where you are given a dataset without labels or rewards, but you want to learn e.g. a probability distribution that this data was likely sampled from or separate this data into groups. For example, in k-means (a clustering algorithm), you want to split the data into groups so that similar objects belong to the same group and dissimilar objects belong to different groups. Note that k-means is a learning algorithm, so it's not a model, but you could consider the centroids of the clusters the parameters of the model (a clustering model).

There are variations or subcategories of these learning paradigms and you can also combine them, so sometimes the difference between them is not so clear.

There are also different types of models. There are parametric (e.g. a linear regression model) and non-parametric (e.g. neural networks) models.


Do we know what the units of neural networks will do before we train them? I was learning about back-propagation and, looking at the algorithm, there is no particular 'partiality' given to any unit. What I mean by partiality there is that you have no particular characteristic associated with any unit, and this results in all units being equal in the eyes of the machine.
So, won't this result in the same activation values of all the units in the same layer? Won't this lack of 'partiality' render neural networks obsolete?
I was reading a bit and watching few videos about backpropagation and, in the explanation given by Geoffrey Hinton, he talks about how we're trying to train the hidden units using the error derivatives w.r.t our hidden activities rather than using desired activities. This further strengthens my point about how by not adding any difference to the units, all units in a layer become equal since initially the errors due to all of them are the same and thus we train them to be equal.

In reverse order to how you asked:

  all units in a layer become equal since initially the errors due to all of them are the same and thus we train them to be equal

This actually happens if you initialise the weights equally (e.g. all zero). Gradients in that case are the same to each neuron in the same layer, and everything changes in lockstep. A neural network without random weight initialisation will simply not work.

  So won't this result in the same activation values of all the units in the same layer? Won't this lack of 'partiality' render neural networks obsolete?

No, because random weight initialisation causes the gradients to be different, and the neuron activations will typically diverge to represent different "hidden" features that activate differently depending on the input.

  What I mean by partiality there is that, you have no particular characteristic associated with any unit and this results in all units being equal in the eyes of the machine.

One interesting side-effect of this behaviour is that the "partiality" will often be assigned effectively randomly too, as the neural network will converge to features that somehow work. These features have no guarantee to be meaningful to a human being in the context of a problem to solve. They might be something that can be mapped to the problem, they might be some linear combination of something that can be understood, but often there is no obvious interpretation.


How can I compress the states of a reinforcement learning agent? I'm working on a problem that involves an RL agent with very large states. These states consist of several pieces of information about the agent. The states are not images, so techniques like convolutional neural networks will not work here.
Is there some general solutions to reduce/compress the size of the states for reinforcement learning algorithms?

Compression will be lossy, some detailed features in the state will be off calculation.
A common technique might be using max-pool function or layer (before feeding to policy network if RL here is deep RL).
Max-pooling is very lossy, use some other classic compression algos such as Zip, Rar but using these classic no-loss compression is weird and extremely slow in the model pipeline.
Possible solution if allowing lossy data, commonly: Use max-pool (giving out high contrast data), average-pool (giving out blurred data).
To keep data intact, TensorFlow can compress tensors: "only sacrificing a tiny fraction of model performance. It can compress any floating point tensor to a much smaller sequence of bits."
See: https://github.com/tensorflow/compression


Is there a ReLU-like activation function that concatenates positive and negative values? Is there a ReLU-like activation function that concatenates positive and negative values? What is its name? Apparently, it doubles the output dimension.

It seems I have found it. It is called concatenated ReLU (CReLU).

  Concatenated ReLU has two outputs, one ReLU and one negative ReLU, concatenated together. In other words, for positive x it produces [x, 0], and for negative x it produces [0, x]. Because it has two outputs, CReLU doubles the output dimension.

Understanding and Improving Convolutional Neural Networks via Concatenated Rectified Linear Units (the original paper that introduced CReLU).
A TensorFlow implementation

There is also Negative CReLU. It seems that the difference is only the sign.

$$\text{NCReLU}(x) = (\rho(x) , −\rho(−x) )$$


What information should be cached in experience replay for actor-critic? Experience replay is buffer (or a "memory") of transactions $e_t = (s_t, a_t, r_t, s_{t+1})$.

The equations for calculating the loss in actor critic are an

actor loss (parameterized by $\theta$) $$log[\pi_\theta(s_t,a_t)]Q_w(s_t,a_t)$$ and a critic loss (parameterized by $w$) $$r(s_t,a_t) + \gamma Q_w(s_{t+1},a_{t+1}) - Q_w(s_{t},a_{t}).$$

As I see it, there are two more elements that need to be saved for later use:

The expected Q value at the time $t$: $Q_w(s_{t},a_{t})$
The log probability for action $a_t$: $log[\pi_\theta(s_t,a_t)]$

If they are not saved, how will we be able to later calculate the loss for learning? I didn't see anywhere stating to save those, and I must be missing something.

Do these elements need to be saved or not?

The loss function is estimated in every batch training cycle. Gradients of the loss are computed and propagation back to the network happens in every cycle. This means that you use a small batch (e.g. 100 instances) from the replay memory, and by having the states you can input them to the respective network and have the $Q(s)$ for every state in your batch. Then you estimate the loss and run backpropagation and the networks' weights are getting updated. You continue gathering experience by interacting with the environment and after a threshold that you specify you repeat the cycle by re-sampling a new batch from the memory.

Just a suggestion: Start moving towards Asynchronous/synchronous methods for RL and use one network with different "heads" for Actor and Critic. Then you use one loss function plus the experience that now is collected from multiple instances of your agent-environment interaction (in contrast to your description in which one agent collects and stores experience from one instance).


Why is it a problem if the outputs of an activation function are not zero-centered? In this lecture, the professor says that one problem with the sigmoid function is that its outputs aren't zero-centered. Are the explanation provided by the professor regarding why this is bad is that the gradient of our loss w.r.t. the weights $\frac{\partial L}{\partial w}$ which is equal to $\frac{\partial L}{\partial \sigma}\frac{\partial \sigma}{\partial w}$ will always be either negative or positive and we'll have a problem updating our weights as she shows in this slide, we won't be able to move in the direction of the vector $(1,-1)$. I don't understand why since she only talks about one component of our gradient and not the whole vector. if the components of the gradient of our loss will have different signs which will allow us to adjust to different directions I'm I wrong? But the thing that I don't understand is how this property generalizes to non zero-centered functions and non-zero centered data?

Yes, if the activation function of the network is not zero centered, $y = f(x^{T}w)$ is always positive or always negative. Thus, the output of a layer is always being moved to either the positive values or the negative values. As a result, the weight vector needs more update to be trained properly and the number of epochs needed for the network to get trained also increases. This is why the zero centered property is important, though it is NOT necessary.
Zero-centered activation functions ensure that the mean activation value is around zero. This property is important in deep learning because it has been empirically shown that models operating on normalized data––whether it be inputs or latent activations––enjoy faster convergence.
Unfortunately, zero-centered activation functions like tanh saturate at their asymptotes –– the gradients within this region get vanishingly smaller over time, leading to a weak training signal.
ReLU avoids this problem but it is not zero-centered. Therefore all-positive or all-negative activation functions whether sigmoid or ReLU can be difficult for gradient-based optimization. So, To solve this problem deep learning practitioners have invented a myriad of Normalization layers (batch norm, layer norm, weight norm, etc.). we can normalize the data in advance to be zero-centered as in batch/layer normalization.
Reference:
A Survey on Activation Functions and their relation with Xavier and He Normal Initialization


Are ANNs just highly distributed lossy compression schemes? Conceptually speaking, aren't artificial neural networks just highly distributed, lossy compression schemes?

They're certainly efficient at compressing images.

And aren't brains (at least, the neocortex) just compartmentalized, highly distributed, lossy databases?

If so, what salient features in RNNs and CNNs are necessary in any given lossy compression scheme in order to extract the semantic relations that they do? Is it just a matter of having a large number of dimensions/variables?

Could some kind of lossy Bloom filter be re-purposed for the kinds of problems ANNs are applied to?

ANNs don't compress, they generalise.

Often this leads to compression, i.e. the internal generalised representation is smaller than the original input, but not necessarily. Imagine a ANN that is trained to use the screen input of a computer game to play it. If the game is very rich and conceptually deep the internal representation of a single screen input might be a lot bigger than the input itself, because ANNs put the single data points into the context of the overall data. Which leads us to the second point:

ANNs (and the neocortex) model data hierarchically.

This is what makes them so powerful. So it is not just about having a large number of parameters, they also have to be arranged in such a way that they capture the structure of the data (or the world), which very often seems to be hierarchical. Just look a two different pictures of a duck. On the pixel level they might be as different as random images, all the similarities emerge in higher levels of the hierarchy, when enough pixels combined give you the patterns of feathers, beak and webs. Bloom filters obviously lack this property. They would only give you a pixel by pixel account of whether you have seen (almost) exactly this picture before.


How much computing power does it cost to run GPT-3? I know it cost around $4.3 million dollars to train, but how much computing power does it cost to run the finished program? IBM Watson chatbot AI only costs a few cents per chat message to use, OpeenAI Five seemed to run on a single gaming PC setup. So I'm wondering how much computing power does it need to run the finished ai program.

I can't anwser your question on how much computing power you might need, but you'll need atleast a smallgrid to run the biggest model just looking at the memory requirments (175B parameters so 700GB of memory). The biggest gpu has 48 GB of vram  I've read that gtp-3 will come in eigth sizes, 125M to 175B parameters. So depending upon which one you run you'll need more or less computing power and memory. (https://lambdalabs.com/blog/demystifying-gpt-3/)
For an idea of the size of the smallest, "The smallest GPT-3 model is roughly the size of BERT-Base and RoBERTa-Base."


What's a good resource for getting familiar with reinforcement learning? I am familiar with supervised and unsupervised learning. I did the SaaS course done by Andrew Ng on Coursera.org.

I am looking for something similar for reinforcement learning.

Can you recommend something?

To the good answers here, I would add

A brief overview of RL: Most essential concepts in one place.
Another brief overview, in presentation format.
Ben Recht's An outsider's tour of RL is pretty comprehensive and accessible.
The Bellman equations: central to the whole RL theory.
Policy gradients explained by Andrej Karpathy (mentioned in other answers as "pong from pixels", this is the link).

These barely scratch the surface of RL, but they should get you started.


What size of neural networks can be trained on current consumer grade GPUs? (1060,1070,1080) is it possible to give a rule of thumb estimate about the size of neural networks that are trainable on common consumer grade GPUs?
For example:
The Emergence of Locomotion (Reinforcement) paper trains a network using tanh activation of the neurons. They have a 3 layer NN with 300,200,100 units for the Planar Walker. But they don’t report the hardware and time ...
But could a rule of thumb be developed? Also just based on current empirical results, so for example:
$X$ Units using sigmoid activation can run $Y$ learning iterations per hour on a 1060.
Or using activation function $a$ instead of $b$ causes a $n$ times decrease in performance.
If a student/researcher/curious mind is going to buy a GPU for playing around with these networks, how do you decide what you get? A 1060 is apparently the entry level budget option, but how can you evaluate if it is not smarter to just get a crappy netbook instead of building a high power desktop and spend the saved $ on on-demand cloud infrastructure.
Motivation for the question: I just purchased a 1060 and (clever, to ask the question afterwards huh) wonder if I should have just kept the $ and made a Google Cloud account. And if I can run my master thesis simulation on the GPU.

Usually the problem is to fit the model into video RAM.
If it does not, you cannot train your model at all without big efforts (like training parts of the model separately).
If it does, time is your only problem. But the difference in training time between consumer GPUs like the Nvidia 1080 and much more expensive GPU accelerators like the Nvidia K80 are not very large. Actually the best consumer cards are faster than GPU accelerators, but lack other properties like VRAM. Random comparisons and benchmarks: Which GPU(s) to Get for Deep Learning: My Experience and Advice for Using GPUs in Deep Learning, A Comparison between NVIDIA’s GeForce GTX 1080 and Tesla P100 for Deep Learning.
To calculate if your models fits into VRAM, you just approximate how much data and which hyperparameters you have (inputs, outputs, weights, layers, batch size, which datatype and so on).


Is there an efficient way to implement a random crossover of individuals stored in a matrix? I am using a GA to optimise an ANN in Matlab. This ANN is pretty basic (input, hidden, output) but the input size is quite large (10,000) and the output size is 2 since I have to classes of images to be classified.

The weights are in the form of 2 matrices (10,000*m) and (m * 2). I am now trying to do the genetic cross over with mutation.

Since the weights are in a matrix, is there an efficient way to implement a random crossover (with mutation) without doing it in a point-wise fashion?

Firstly, before we commence I will recommend that you refer to a similar questions on the network https://stackoverflow.com/questions/828486/neural-net-optimize-w-genetic-algorithm

The majority of ML studies focus on gradient algorithms, usually a variation of back-propagation to obtain the weights of the model.

However since genetic algorithms are powerful searching algorithms they can be utilized to tune and optimize the structure and parameters of neural networks.

Genetic algorithms are stochastic search techniques that guide a population of solutions towards an optimum using the principles of evolution and natural genetics.

Genetic algorithms are especially capable of handling problems in which the objective function is discontinuous/non differentiable, nonconvex, multimodal or noisy. (Dharmistha D)

Algorithms, which combine genetic algorithms and error back-propagation, have been shown to exhibit better convergence properties than pure back-propagation.

To facilitate the crossover operation for the exchange of information between two parents we need to define four components within a genetic algorithm.

A way of coding solutions to the problem on chromosomes;
A fitness function which returns a value for each chromosome given to it;
A way of initializing the population of chromosomes;
Operators that may be applied to parents  to alter their genetic composition i.e. mutation.

In these hybrid systems weights in different layers of the network are optimized using a genetic algorithm. Experimental results demonstrate that the genetic algorithms can in some cases optimize the hyper-parameters of an ANN better than a hand tuned model.

Regarding an efficient way to implement a random crossover, here is something to get you started
(Code courtesy of Dan Golding 2013)

%Randomly choose 2 individuals

n = size(new_pop, 1);
l = size(new_pop, 2);

breeders = new_pop(randperm(n,2),:);

%Choose a crossover point

cp = randperm(l, 1);

%Crossover

b1 = [breeders(1, 1:cp), breeders(2, cp+1:end)];
b2 = [breeders(2, 1:cp), breeders(1, cp+1:end)];

For further reference I recommend that you kindly look at the below links

Matlab crossover genetic algorithm

https://stackoverflow.com/questions/17696323/matlab-crossover-genetic-algorithm

Single point ordered crossover in matlab

https://stackoverflow.com/questions/16302382/single-point-ordered-crossover-in-matlab

To optimize a neural network of multiple inputs using a genetic algorithm.

https://www.mathworks.com/matlabcentral/answers/180513-to-optimize-a-neural-network-of-multiple-inputs-using-a-genetic-algorithm?s_tid=gn_loc_drop

How can I use the Genetic Algorithm (GA) to train a Neural Network in Neural Network Toolbox

https://www.mathworks.com/matlabcentral/answers/100323-how-can-i-use-the-genetic-algorithm-ga-to-train-a-neural-network-in-neural-network-toolbox


How is the cost of the path to each node computed in A*? How is the cost of the path to each node $n$ computed in the A* algorithm? Do we need to add the cost of the path to the parent node $p$ to the cost of the path of the child node $n$?

The evaluation function in A* is $f(n) = g(n) + h(n)$, where $g(n)$ is the cost of the path from the starting node to $n$ and $h(n)$ is an estimate of the distance from $n$ to the goal node. To compute $g(n)$, you simply do $g(n) = g(p) + c(p, n)$, where $p$ is the parent node, and $c(p, n)$ is the $c$ost of the edge between $p$ and $n$. So, yes, to compute $g(n)$ you use $g(p)$.


What is the purpose and benefit of applying CNN to a graph? I'm new to the graph convolution network. I wonder what is the main purpose of applying data with graph structure to CNN?

There are some problems that involve graphs and manifolds (sometimes collectively called non-Euclidean data), such as molecule design and generation, drug repositioning, social networks analysis, brain imaging, fake news detection, recommender systems, neutrino detection, computer vision and graphics and shape (e.g. hand or face) completion or generation (generative models).

The main benefit of geometric deep learning (deep learning applied to graphs and manifolds) is that you do not lose the information encoded in the graphs (or manifolds), which, otherwise, you would likely lose because you would need to convert your graphs (or manifolds) to an equivalent vector representation that you can feed into the existing CNN or other standard neural networks.

Note that you cannot directly apply the usual convolution operation to graphs, because, for example, graphs do not have the notion of relative positions of the nodes. Furthermore, note that graph networks have little to do with CNNs, even if they are sometimes called graph convolution networks.


Understanding the intuition behind Content Loss (Neural Style Transfer) I'm trying to understand the intuition behind how the Content Loss is calculated in a Neural Style Transfer. I'm reading from an articles: https://medium.com/mlreview/making-ai-art-with-style-transfer-using-keras-8bb5fa44b216 ,  that explains the implementation of Neural Style Transfer, from the Content loss function:

The article explains that:

F and P are matrices with a number of rows equal to N and a number of columns equal to M.
N is the number of filters in layer l and M is the number of spatial elements in the feature map (height times width) for layer l.

From the code below for getting the features/content representation from particular Conv layers, I didn't quite understand how it works. Basically I printed out the output of every line of code to try to make it easier, but it still left a number of questions to be asked, which I listed below the code:

def get_feature_reps(x, layer_names, model):
    """
    Get feature representations of input x for one or more layers in a given model.
    """
    featMatrices = []
    for ln in layer_names:
        selectedLayer = model.get_layer(ln)
        featRaw = selectedLayer.output
        featRawShape = K.shape(featRaw).eval(session=tf_session)
        N_l = featRawShape[-1]
        M_l = featRawShape[1]*featRawShape[2]
        featMatrix = K.reshape(featRaw, (M_l, N_l))
        featMatrix = K.transpose(featMatrix)
        featMatrices.append(featMatrix)
    return featMatrices

def get_content_loss(F, P):
    cLoss = 0.5*K.sum(K.square(F - P))
    return cLoss

1- For the line featRaw = selectedLayer.output, when I print featRaw, I get the output:
Tensor("block4_conv2/Relu:0", shape=(1, 64, 64, 512), dtype=float32).

a- Relu:0 does this mean Relu activation has not yet been applied?
b- Also I presume we're outputing the feature maps outputs from block4_conv2, not the filters/kernels themselves, correct?
c- Why is there an axis of 1 at the start? My understanding of Conv layers is that they're simply made up from the number of filters/kernels (with shape-height, width, depth) to apply to the input.
d- Is selectedLayer.output simply outputs the shape of the Conv layer, or does the output object also hold other information like the pixel values from the output feature maps of the layer?

2- With the line: featMatrix = K.reshape(featRaw, (M_l, N_l) where printing featMatrix would output: Tensor("Reshape:0", shape=(4096, 512), dtype=float32).

a- This is where I'm confused the most. So to get the feature/content representation of a particular Conv layer of an image, we simply create a matrix of 2 dimensions, the first being the number of filters and the other being the area of the filter/kernel (height * width). That doesn't make sense! How do we get unique feature of an image from just that?!! We're not retrieving any pixel values from a feature map. We're simply getting the area size of filter/kernel and the number of filters, but not retrieving any of the content (pixel values) itself!!
b- Also the final featMatrix is transposed - i.e. featMatrix = K.transpose(featMatrix) with the output Tensor("transpose:0", shape=(512, 4096), dtype=float32). Why is that (i.e. why reverse the axis)?

3 - Finally I want to know, once we retrieve the content representation, how can I output that in both as a numpy array and save it as an image?

Any help would be really appreciated.

For the line featRaw = selectedLayer.output, when I print featRaw, I get the output: Tensor("block4_conv2/Relu:0", shape=(1, 64, 64, 512), dtype=float32).

  a) Relu:0 does this mean Relu activation has not yet been applied?

It has been applied.

  b) Also I presume we're outputing the feature maps outputs from block4_conv2, not the filters/kernels themselves, correct?

Yes.

  c) Why is there an axis of 1 at the start? My understanding of Conv layers is that they're simply made up from the number of filters/kernels (with shape-height, width, depth) to apply to the input.

The first dimension in keras is used for representing batch dimension. When training the model, you aren't passing the images one by one but in batches. The number of images that are processed in parallel before the model calculates its loss and makes its update is called the batch_size. When computing the content loss, though, you have a single image you want to perform style transfer to, so batch_size=1. That's why you see the unit in the beginning of the tensor's shape.

  d) Is selectedLayer.output simply outputs the shape of the Conv layer, or does the output object also hold other information like the pixel values from the output feature maps of the layer?

It outputs the tensor containing all the pixel values that are outputted from that layer.

  With the line: featMatrix = K.reshape(featRaw, (M_l, N_l) where printing featMatrix would output: Tensor("Reshape:0", shape=(4096, 512), dtype=float32).

  a) This is where I'm confused the most. So to get the feature/content representation of a particular Conv layer of an image, we simply create a matrix of 2 dimensions, the first being the number of filters and the other being the area of the filter/kernel (height * width). That doesn't make sense! How do we get unique feature of an image from just that?!! We're not retrieving any pixel values from a feature map. We're simply getting the area size of filter/kernel and the number of filters, but not retrieving any of the content (pixel values) itself!!

I think you're confusing things a bit. You are getting the pixel values, just in a different shape. Instead of stacking them in a $64 \times 64 \times 515$ array you are stacking them in a $1096 \times 515$ array. Exactly the same information is stored in a different shape. Think of it like transforming a $10 \times 2$ matrix to a $5 \times 4$ one. It contains the same information, in a different shape.

  b) Also the final featMatrix is transposed - i.e. featMatrix = K.transpose(featMatrix) with the output Tensor("transpose:0", shape=(512, 4096), dtype=float32). Why is that (i.e. why reverse the axis)?

I'm not sure why exactly he does this, but from a bit that I saw the code I think it has something to do with the style loss (not the content loss). Probably something didn't match up with the dimensions of some operation, maybe a matrix multiplication...

  Finally I want to know, once we retrieve the content representation, how can I output that in both as a numpy array and save it as an image?

If you have the output image as a tensor, let's say y. You can convert it to numpy through keras.backend.eval:

import keras.backend as K

y_arr = K.eval(y)

There are many ways to save the image, including scikit-image, scipy, opencv, PIL, matplotlib, etc. I'll use the last:

import matplotlib.pyplot as plt

plt.imsave('my_image.png', y_arr)

Note that in order to work y_arr should have a shape of either (height, width, 3) (if it is an RGB image) or (height, width) (if it is grayscale).


How to implement an Automatic Learning Rate for a Neural Network? I'm learning Neural Networks, and everything works as planned but, like humans do, adjusting themselves to learn more efficiently, I'm trying to understand conceptually how one might implement an auto adjusting learning rate for a Neural Network.

I have tried to make it based on error, something like how bigger is error learning rate is getting bigger as well. [Could use some clarification here--not entirely sure what you're saying.  If can clarify, I'm happy to clean up the English. -DukeZhou]

*If you want give me an example give it on a C based language or math because I don't have experience with Python or Pascal.

Adjusting for learning rate is a common scenario in machine learning. There is rich literature about it, countless papers. The most common implementation:

AdaGrad
RMSProp
Adam

There are many more variants. You'll need to do some research. Please take a look at:

  https://en.wikipedia.org/wiki/Stochastic_gradient_descent


Seq2Seq dialogs predicts only most common words like `you` after couple of epoches I'm training Seq2Seq model on OpenSubtitles dialogs - Cornell-Movie-Dialogs-Corpus.
My work based on the following papers (but currently I'm not implemented Attention yet):

Sequence to Sequence Learning with Neural Networks, Sutskever et al. 2014
A Neural Conversational Model, Vinyals, Le, 2015

The loss I received is quite high and sucked in variation ~6.4 after 3 epoches. The model predicts the most common words with some times other not significant words (but 99.99% is just 'you'):

I’ve experimented with 128 - 2048 hidden units and with 1 or 2 or 3 LSTM layers per encoder and decoder. The outcomes are more or less the same.

SEQ1: yeah man it means love respect community and the dollars too the package the unk end
SEQ2: but how did you get unk 82 end
PREDICTION: promoting 16th dashboard be of the the the you you you you you you you you you you you you you you you you you you you you you you you you

I'm using here greedy prediction, meaning - after I receive logit I do argmax(..) on all its value for first-3 mini-batch-elements (here I present only first element). For convenient - SEQ1 and SEQ2 are also printed - to know the actual dialog which was presented to the model.
The pseudo-code of my architecture looks like this (I'm using Tensorflow 1.5):
seq1 = tf.placeholder(...)
seq2 = tf.placeholder(...)

embeddings = tf.Variable(tf.random_uniform([vocab_size, 100],-1,1))

seq1_emb = tf.nn.embedding_lookup(embeddings, seq1)
seq2_emb = tf.nn.embedding_lookup(embeddings, seq1)

encoder_out, state1 = tf.nn.static_rnn(BasicLSTMCell(), seq1_emb)
decoder_out, state2 = tf.nn.static_rnn(BasicLSTMCell(), seq2_emb,
                                                        initial_state=state_1)
logit = Dense(decoder_out, use_bias=False)

crossent = tf.nn.saparse_softmax_cross_entropy_with_logits(logits=logit,
                                                         labels=target)
crossent = mask_padded_zeros(crossent)
loss = tf.reduce_sum(crossent) / number_of_words_in_batch

train = tf.train.AdamOptimizer(learning_rate=0.00002).minimize(loss)

I'm also wonder if I pass well state1 to decoder, which in general looks like this:
# reshape in pseudocode: state1 = state[1:]
new_state1 = []
for lstm in state1:
    new_lstm = []
    for gate in lstm:
        new_lstm.append(gate[1:])
    new_state1.append(tuple(new_lstm))
state1 = tuple(new_state1)

Should I use some projection layer between states of encoder and decoder ?

So if seq1 has 32 words, seq2 has 31 (since we will not predict nothing after the last word, which is the tag <END>).

To answer my own question - it was because of 2 things:

Too small number of batches - Model just started to gain statistical knowledge about language dialogs. I needed to train it longer.
I masked sequence too early (wrongly removed the < END > tag) - Because on each sentence the last world is just < END > tag - I removed it on all training examples, which prevented Model from learn what does it mean "the end of the sentence".

The last condition probably caused that strange pattern even further, because if model doesn't know what word to put in (and because of the lack of the < END > tag) it must fill each sentence till the end of max_sequence_len.

So the Model inputed in loop-like-manner - one of the most common word (where there was no signal from target sentence, because it simply ended).


Does use of GPU reduces time of execution of .predict() method in Keras/TF model? Does using GPU instead of CPU with tensorflow and keras model reduces the time of .predict() method on a trained network or maybe it only reduces time of training of the network?

Yes, Keras will use the GPU for inference (predictions) as well as training.

TensorFlow code, and tf.keras models will transparently run on a single GPU with no code changes required.

For more information, you can read the TensorFlow GPU guide, but in effect you shouldn't need to do anything to get inference to run on the GPU.


Can feature engineering change the selection of the model according to the minimum description length? The definition of MDL according to these slides is:

  The minimum description length (MDL) criteria in machine learning says that the best description of the data is given by the model which compresses it the best. Put another way, learning a model for the data or predicting it is about capturing the regularities in the data and any regularity in the data can be used to compress it. Thus, the more we can compress a data, the more we have learnt about it and the better we
  can predict it.

  MDL is also connected to Occam’s Razor used in machine learning which states that "other things being equal, a simpler explanation is better than a more complex one." In MDL, the simplicity (or rather complexity) of a model is interpreted as the length of the code obtained when that model is used to compress
  the data.

To put it in short according to MDL principle, we prefer predictors with relatively smaller description length (i.e can be described within a certain length) for a given description language (this definition is without delving into exact technical details as it is not necessary to the question).

Since MDL is dependent on the description language we use, can we say feature engineering can cause a change in the selection of the predictor?
For example as this picture shows:

To me, it seems that, in the first picture, we will require a longer description length predictor in Cartesian coordinates, as compared to a predictor in polar co-ordinates (just a single discerning radius needs to be specified). So, to me, it seems feature engineering changed the selection of the predictor to a relatively simple one (in the sense that it will have a shorter description length). Thus feature engineering has changed the description length required for out predictor. Did I make any wrong assumptions? If so why?

I think the wrong assumption here is that you've forgotten the cost of encoding the new features!

MDL should be considered relative to the original or raw dataset. The idea is that you want to find an expression you could send to someone else that encodes the structure of the dataset in terms of the original variables. If you define new features, you need to send a description of those features along with your model.

To make this clearer, imagine you call me up on the phone, and we're both looking at your left hand image. You say to me 'Yeah, you just draw a line through it at $p=a$'. The natural question for me to ask is 'What's p?'. If you have to tell me what $p$ is, then it's part of the description length.

A circular model for your lefthand image does have a MDL for its decision boundary of something like $(x−a)^2+(y−b)^2=c$. However, the feature transformation you've selected has a description length of $p=(x-a)^2 + (y-b)^2$. It should be clear that the description lengths for the linear model through $p$ and the circular model through $x$ and $y$ are identical.


Why is DDPG an off-policy RL algorithm? In DDPG, if there are no $\epsilon$-greedy and no action noise, is DDPG an on-policy algorithm?

DDPG is an off-policy algorithm simply because of the objective taking expectation with respect to some other distribution that we are not learning about, i.e. the deterministic policy gradient can be expressed as
$$\nabla _{\theta^\mu} J \approx \mathbb{E}_{s_t \sim \rho^\beta} \left[ \nabla _{\theta^\mu} Q(s,a|\theta^Q) | s=s_t, a=\mu(s_t ; \theta ^\mu)  \right]\;.$$
We are interested in learning about the policy parameters of $\mu$, denoted by $\theta$, but we take expected with respect to some discounted state distribution induced by a policy $\beta$, which we will denote as $\rho^\beta$.
To summarise, we are learning off-policy as the expectation of the gradient is taken with respect to some state distribution that occurs under some policy that we are not learning about.
Given that on-policy learning is a special case of off-policy learning, if the replay buffer had a size of one, i.e. we use only the most recent experience tuple to perform parameter updates, then DDPG would be on-policy.


Analysis of Training Loss and Validation Loss Graph Here I am Showing Two Loss graphs of an Artificial Neural Network.

Model 1

Model 2

Blue -training loss

Red  -val training loss

Can you help me to analyse these graphs? I read some articles and post but doesn't give me any sense.

Simply model 1 is a better fit compared to model 2.

Graph for model 1

We notice that the training loss and validation loss aren't correlated. This means the as the training loss is decreasing, the validation loss remains the same of increases over the iterations. This means that the model is not exactly improving, but is instead overfitting the training data. This isn't what we are looking for.
Graph for model 2

In this case, there is clearly a health correlation between training loss and the validation loss. They both seem to reduce and stay at a constant value. This means that the model is well trained and is equally good on the training data as well as the hidden data.

You should stick with model 2. In case you're going ahead with model 1, make sure to use the chechpoint where both the losses are at a similar value (at around 100 -150 epochs)


How to handle infeasibility caused due to crossover and mutation in genetic algorithm for optimization? I have chromosomes with floating-point representation with values between $0$ and $1$. For example
Let $p_1 = [0.1, 0.2, 0.3]$ and $p_2 = [0.5, 0.6, 0.7]$ be two parents. Both comply with the set of constraints. In my case, the major constraint is $$ p_1[1]*p_1[2] - k*p_1[0] \geq 0 $$ for any chromosome $p_1$. For the example above we can take $k=0.3$, which renders $c_2$ infeasible.
However, the children produced by 1 point crossover, we get $c_1 = [0.1, 0.6, 0.7]$ and $c_2 = [0.5, 0.2, 0.3]$ out of which 1 or both may not comply with the given constraints.
A similar scenario can also occur with a small perturbation of values due to mutation strategy. Correct me if I am wrong in the belief that such kind of scenarios might arise irrespective of the strategy employed for crossover and mutation.
What are the options to handle such kinds of cases?

You have two broad categories of options, prevention and repair.

Prevention means defining a crossover and mutation operator that try to be more intelligent about respecting the constraints. Suppose you have an encoding where each individual is a list of integers, and the constraint is that there can't be duplicates. You might define a crossover operator that did something like the following. For each position in the offspring, choose randomly from one parent such that, if possible, you choose a value that hasn't already appeared in the offspring. If both parents have values at that position that are already used in the offspring, choose a random value.

That operator would avoid violating the constraint in the first place, while attempting as much as possible to have the offspring inherit information from the parents.

Another option is to just let the operator violate constraints and then deal with the ramifications afterward. You don't go into details about what your constraints actually are, just that c1=[0.1, 0.6, 0.7] violates them. Let's say the constraint is that the third position should not be more than 4x greater than the first one. OK, so then let's take this offspring and adjust either the first or third item. Maybe we make the new individual into c1=[0.2, 0.6, 0.7].

Often, you want some element of randomness in either option so that you don't strongly bias the production of new search points. In my example, don't always make the first element larger. Sometimes, make the third element smaller or produce some random combination of both to repair the constraint violation.

Finally, both options typically strongly benefit from domain knowledge. Design an operator that understands your problem and tries to intelligently solve the problem.