Can you help me understand how weight normalization works? I am trying to dissect the paper Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networks.
Unfortunately, because my math is a little bit rusty, I got a little bit stuck with the proof. Could you provide me with some clarification about proof of the topic?
What I understand is that we introduce, instead of the weight vector $w$ a scalar $g$ (magnitude of original $w$?) and $\frac{v}{\|v\|}$ (direction of original $w$?) of the vector.
$$\nabla_{g} L=\frac{\nabla_{\mathbf{w}} L \cdot \mathbf{v}}{\|\mathbf{v}\|}$$
and
$$\nabla_{\mathbf{v}} L=\frac{g}{\|\mathbf{v}\|} \nabla_{\mathbf{w}} L-\frac{g \nabla_{g} L}{\|\mathbf{v}\|^{2}} \mathbf{v}$$
What I am not really sure about is:
If the gradients are noisy (does this mean that in some dimension we have small and in some high curvature or that error noise differs for very similar values of $w$?) the value will quickly increase here and effectively limit the speed of descent by decreasing value of ($\frac{g}{\|v\|}$). This means that we can choose larger learning rates and it will somehow adjust the effect of the learning rate during the training.
And what I completely miss is:
$$\nabla_{\mathbf{v}} L=\frac{g}{\|\mathbf{v}\|} M_{\mathbf{w}} \nabla_{\mathbf{w}} L$$
with
$$\mathrm{M}_{\mathrm{w}}=\mathrm{I}-\frac{\mathrm{w} \mathrm{w}^{\prime}}{\|\mathrm{w}\|^{2}}$$
It should somehow explain the reasoning behind the idea of the final effects. Unfortunately, I don't really understand this part of the paper, I probably lack some knowledge about Linear Algebra.
Can you verify that my understanding of the paper is correct?
Can you recommend some sources (books/videos) to help me to understand the second part of proof (related to the second set of formulas?)

If the gradients are noisy (does this mean that in some dimension we have small and in some high curvature or that error noise differs for very similar values of w?)

Gradients being noisy means that they are "inconsistent" across different epochs / training steps. With that I mean that they'll sometimes point in one direction, later in a different (maybe the opposite) direction, etc., that the gradients at different time steps give inconsistent/conflicting information, that we don't consistently keep following the same direction through gradient descent but keep jumping all over the place. Note that this is across epochs, it's not across different dimensions within the same epoch.
So, for example, we might have a vector of gradients that looks like $[1, 1, 1, \dots, 1]$ at one time step (i.e., positive gradients in all dimensions), and the next training step get a gradient more like $[-1, -1, -1, \dots, -1]$ (this example is pretty much the "most extreme" kind of noise you can have, completely conflicting directions, in reality it'd generally be less extreme).
Such "noisy gradients" can exist because, in practice, we almost always use estimates of the gradient of our loss function, rather than the true gradients (which we'll generally not even be able to compute). For example, if we have a very large training dataset in a supervised learning setting, we'd ideally use the gradient of the loss function computed across the complete dataset. That tends to be computationally expensive (requires forwards and backwards passes for every single instance in the dataset), so in practice we'll often only use a small minibatch to estimate the gradient; this can result in widely different estimates of the gradient for different minibatches. (note: there can also be different reasons for using minibatches rather than full dataset, such as avoiding overfitting, but that's not too important to consider for this specific question)
A different example of a setting where we can have noisy gradients is in Deep Reinforcement Learning. There, computing the loss function itself is often rather noisy (in the sense that our own predictions, which tend to still be incorrect during the training phase, are a component of the targets that we're updating towards), so our estimates of the gradients will also be noisy.

Derivation for your second question, about the gradient of the loss with respect to $\mathbf{v}$:
We start with the following, from Equation (3) in the paper (this gradient you understand, right?):
$$\nabla_{\mathbf{v}} L = \frac{g}{\vert \vert \mathbf{v} \vert \vert} \nabla_{\mathbf{w}} L - \frac{g \nabla_g L}{\vert \vert \mathbf{v} \vert \vert^2} \mathbf{v}$$
Note that Equation (3) also gives:
$$\nabla_{g} L = \frac{\nabla_{\mathbf{w}} L \cdot \mathbf{v}}{\vert \vert \mathbf{v} \vert \vert}$$
If we plug that into the previous Equation we get:
\begin{aligned}
\nabla_{\mathbf{v}} L &= \frac{g}{\vert \vert \mathbf{v} \vert \vert} \nabla_{\mathbf{w}} L - \frac{g \frac{\nabla_{\mathbf{w}} L \cdot \mathbf{v}}{\vert \vert \mathbf{v} \vert \vert}}{\vert \vert \mathbf{v} \vert \vert^2} \mathbf{v} \\
&= \frac{g}{\vert \vert \mathbf{v} \vert \vert} \nabla_{\mathbf{w}} L - \frac{g \nabla_{\mathbf{w}} L \cdot \mathbf{v}}{\vert \vert \mathbf{v} \vert \vert^3} \mathbf{v} \\
\end{aligned}
The $\frac{g}{\vert \vert \mathbf{v} \vert \vert} \nabla_{\mathbf{w}} L$ term that shows up before the minus can also be isolated in the term after the minus:
\begin{aligned}
\nabla_{\mathbf{v}} L &= \frac{g}{\vert \vert \mathbf{v} \vert \vert} \nabla_{\mathbf{w}} L - \frac{g}{\vert \vert \mathbf{v} \vert \vert} \nabla_{\mathbf{w}} L \cdot \frac{\mathbf{v}}{\vert \vert \mathbf{v} \vert \vert^2} \mathbf{v} \\
&= \frac{g}{\vert \vert \mathbf{v} \vert \vert} \nabla_{\mathbf{w}} L - \frac{g}{\vert \vert \mathbf{v} \vert \vert} \nabla_{\mathbf{w}} L \cdot \frac{\mathbf{v}}{\vert \vert \mathbf{v} \vert \vert} \cdot \frac{\mathbf{v}}{\vert \vert \mathbf{v} \vert \vert} \\
\end{aligned}
Equation (2) tells us that $\mathbf{w} = \frac{g}{\vert \vert \mathbf{v} \vert \vert} \mathbf{v}$, which means that $\frac{\mathbf{w}}{g} = \frac{\mathbf{v}}{\vert \vert \mathbf{v} \vert \vert}$. Plugging this into the above leads to:
\begin{aligned}
\nabla_{\mathbf{v}} L &= \frac{g}{\vert \vert \mathbf{v} \vert \vert} \nabla_{\mathbf{w}} L - \frac{g}{\vert \vert \mathbf{v} \vert \vert} \nabla_{\mathbf{w}} L \cdot \frac{\mathbf{w}}{g} \cdot \frac{\mathbf{w}}{g} \\
\end{aligned}
Below Equation (2), they also explain that $\vert \vert \mathbf{w} \vert \vert = g$, so we can rewrite the above to:
\begin{aligned}
\nabla_{\mathbf{v}} L &= \frac{g}{\vert \vert \mathbf{v} \vert \vert} \nabla_{\mathbf{w}} L - \frac{g}{\vert \vert \mathbf{v} \vert \vert} \nabla_{\mathbf{w}} L \cdot \frac{\mathbf{w}}{\vert \vert \mathbf{w} \vert \vert} \cdot \frac{\mathbf{w}}{\vert \vert \mathbf{w} \vert \vert} \\
&= \frac{g}{\vert \vert \mathbf{v} \vert \vert} \nabla_{\mathbf{w}} L - \frac{g}{\vert \vert \mathbf{v} \vert \vert} \nabla_{\mathbf{w}} L \cdot \frac{\mathbf{w} \mathbf{w}'}{\vert \vert \mathbf{w} \vert \vert^2} \\
\end{aligned}
In that last step, the $'$ symbol in $\mathbf{w}'$ denotes that we transpose that vector.
Finally, because we have that common $\frac{g}{\vert \vert \mathbf{v} \vert \vert} \nabla_{\mathbf{w}} L$ before and after the minus symbol, we can pull it out, multiplying it once by the identity matrix $\mathbf{I}$, and once by the remaining negated term. This is very similar to how you would simplify something like $a - ax$ to $a (1 - x)$ in "standard" algebra; the identity matrix $\mathbf{I}$ plays a very similar role here to the number $1$ in "standard" algebra:
\begin{aligned}
\nabla_{\mathbf{v}} L &= \frac{g}{\vert \vert \mathbf{v} \vert \vert} \nabla_{\mathbf{w}} L \cdot \left( \mathbf{I} - \frac{\mathbf{w} \mathbf{w}'}{\vert \vert \mathbf{w} \vert \vert^2} \right) \\
\end{aligned}
Note: in all of the above, I kind of neglected to pay attention to order of matrix/vector multiplications and their dimensions. Probably the $\nabla_{\mathbf{w}} L$ term should by now already have been moved to after those large brackets, instead of being before those brackets. In this case, it should work out fine regardless because $\mathbf{I}$ as well as $\mathbf{w} \mathbf{w}'$ are square and symmetric matrices. The only resulting difference is in whether you get a row or a column vector out at the end.
This can now finally be rewritten as the Equation you mentioned that you didn't understand yet, Equation (4) in the paper.


What happens to the channels after the convolution layer? I wonder what happens to the 'channels' dimension (usually 3 for RGB images) after the first convolution layer in CNNs?

In books and other sources, it is always said that the depth of the output from convolutional layers is the number of kernels (filters) in that layer.

But, if the input image has 3 channels and we convolve each of them with $K$ kernels, shouldn't the depth of the output be $K * 3$? Are they somehow 'averaged' or in other way combined with each other?

Answer to my question is that values obtained from convolutions among different channels sum up together, therefore 3 channels after convolution with one filter give one output.

Best explanation delivered by Andrew: https://www.coursera.org/lecture/convolutional-neural-networks/convolutions-over-volume-ctQZz


What do the vectors of the center and outside word look like in word2vec? In word2vec, the task is to learn to predict which words are most likely to be near each other in some long corpus of text. For each word $c$ in the corpus, the model outputs the probability distribution $P(O=o|C=c)$ of how likely each other word $o$ in the vocabulary is to be within a certain number of words away from $c$. We call $c$ the "center word" and $o$ the "outside word".

We choose the softmax distribution as the output of our model: $$P(O=o|C=c) = \frac{\exp(\textbf{u}_{0}^{T} \textbf{v}_{c})}{\sum_{w \in \text{Vocab}} \exp(\textbf{u}_{w}^{T} \textbf{v}_c)}$$

where $\textbf{u}_0$ and $\textbf{v}_c$ are vectors that represent the outside and center words respectively.

  Question. What do the vectors  $\textbf{u}_0$ and $\textbf{v}_c$ look like? Are they just one-hot-encodings? Do we need to learn them
  too? Why is this useful?

No, the word vectors are not one-hot encodings. Yes, they are learned.

The purpose of the word2vec model is actually to learn dense, semantically meaningful encodings for words. That is, if your words are $d$-dimensional vectors, then each word's position in this vector space says something about what that word means. This is because word2vec learns to represent words in similar ways if they are frequently close together in your corpus. It implements the idea of distributional similarity.

The task of predicting an "outside word" given a "center word" accomplishes all of this in an indirect way.

A naive objective function to maximize for word2vec is
$$J = \prod_{t=1}^L \prod_{-m \leq j \leq m\\ \quad j\neq 0} p(\textbf{u}_{t+j}|\textbf{v}_t)$$

where $L$ is the length of your corpus, $m$ is the "radius" from each center word you want to consider, $\textbf{u}_{t+j}$ is an outside word, and $\textbf{v}_t$ is a center word.

If we let $p(\textbf{u}_{t+j}|\textbf{v}_t)$ be the softmax distribution, then maximizing $J$ means maximizing the inner product $\textbf{u}_{t+j}^T\textbf{v}_t$ in the softmax's numerator. Maximizing that inner product means making center words as close as possible to their neighboring words, giving you some semantically meaningful word vectors to use in your downstream NLP tasks.

This lecture from Stanford's CS224N goes into more detail.


What is the difference between out of distribution detection and anomaly detection? I'm currently reading the paper Likelihood Ratios for Out-of-Distribution Detection, and it seems that their problem is very similar to the problem of anomaly detection. More precisely, given a neural network trained on a dataset consisting of classes $A,B,$ and $C$, then they can detect if an input to the neural network is anomalous if it is different than these three classes. What is the difference between what they are doing and regular anomaly detection?

You observation is correct although the terminology needs a little explaining.
The term 'out-of-distribution' (OOD) data refers to data that was collected at a different time, and possibly under different conditions or in a different environment, then the data collected to create the model.  They may say that this data is from a 'different distribution'.
Data that is in-distribution can be called novelty data.  Novelty detection is when you have new data (i.e. OOD) and you want to know whether or not it is in-distribution.  You want to know if it looks like the data you trained on.
Anomaly detection is when you test your data to see if it is different than what you trained the model.  Out-of-distribution detection is essentially running your model on OOD data.  So one takes OOD data and does novelty detection or anomaly detection (aka outlier detection).
Below is a figure from What is anomaly detection?

In time series modeling, the term 'out-of-distribution' data is analogous to 'out-of-sample' data and 'in-distribution' data is analogous with 'in-sample' data.


Why do we need floats for using neural networks? Is it possible to make a neural network that uses only integers by scaling input and output of each function to [-INT_MAX, INT_MAX]? Is there any drawbacks?

TL;DR

Not only it's possible, it even gets done and is commercially available. It's just impractical on a commodity HW which is pretty good at FP arithmetic.

Details

It is definitely possible, and you might get some speed for a lot of troubles.

The nice thing about floating point is that it works without you knowing the exact range. You can scale your inputs to the range (-1, +1) (such a scaling is pretty commonplace as it speeds up the convergence) and multiply them by 2**31, so they used the range of signed 32-bit integers. That's fine.

You can't do the same to your weights as there's no limit on them. You can assume them to lie in the interval (-128, +128) and scale them accordingly.

If your assumption was wrong, you get an overflow and a huge negative weight where a huge positive weight should be or the other way round. In any case, a disaster.

You could check for overflow, but this is too expensive. Your arithmetic gets slower than FP.

You could check for possible overflow from time to time and take a corrective action. The details may get complicated.

You could use saturation arithmetic, but it's implemented only in some specialized CPUs, not in your PC.

Now, there's a multiplication. With use of 64-bit integers, it goes well, but you need to compute a sum (with a possible overflow) and scale the output back to same sane range (another problem).

All in all, with fast FP arithmetic available, it's not worth the hassle.

It might be a good idea for a custom chip, which could do saturation integer arithmetic with much less hardware and much faster then FP.

Depending on what integer types you use, there may be a precision loss when compared to the floating point, which may or may not matter. Note that TPU (used in AlphaZero) has 8-bit precision only.


Who first coined the term "deep learning"? AFAIK, deep learning became popular in 2012 with the victory of ImageNet Competition - Large Scale Visual Recognition Challenge 2012 where winners of this contest actually used deep learning techniques for optimizing the solution for object recognition.
Who first coined the term deep learning? Is there any published research paper that first used that term?

The term was introduced to the machine learning and computer science community by Rina Dechter in Learning While Searching in Constraint-Satisfaction-Problems (1986) [1], where she writes

Discovering all minimal conflict-sets amounts to acquiring all the possible information out of a dead-end. Yet, such deep learning may require considerable amount of work.

When deep learning is used in conjunction with restricting the level of learning we get deep first-order learning (identifying minimal conflict sets of size 1) and deep second-order learning (i.e. identifying minimal conflict-sets of sizes 1 and 2).

Our experiments (implemented in LISP on a Symbolits LISP Machine) show that in most cases both performance measures improve as we move from shallow learning to deep learning and from first-order to second order.

However, note that the term deep learning was used before 1986 in other contexts, for example, in [2]. Moreover, note that Rina Dechter did not use the term in the context of neural networks, which was probably used later.


How does publishing in the deep learning world work, with respect to journals and arXiv? Let's say I implemented a new deep learning model that pushed some SOTA a little bit further, and I wrote a new paper about for publication.

How does it work now? I pictured three options:

Submit it to a conference. Ok, that's the easy one, I submit it to something like NeurIPS or ICML and hope to get accepted. At that point, how do you make your paper accessible? Are there problems in uploading it to arXiv later, in order to get read by more people?
Upload it on arXiv directly. If I do that it would not be peer-reviewed, and technically speaking it would be devoid of "academic value". Right? It could easily be read by anyone, but there would be no formal "proof" of its "scientific quality". Correct me if I'm wrong.
Submit it to a peer-reviewed journal. Avoid desk rejection, avoid reviewers' rejection, after a long painful process it ends up on some international scientific journal. At that point, since the article is formally the editor's property, can you still upload it on arXiv, or on your blog, so that it can be accessible by many people?
How do the big stars of deep learning research do when they have some hot new paper ready for publication? And what publications are the most valued in the professional and the academic world?

Let me answer your questions one by one.

Submit it to a conference

Let's start with the optimistic case. Say your paper gets accepted! You can upload your preprint on arXiv with the "arXiv.org perpetual, non-exclusive license to distribute this article (Minimal rights required by arXiv.org)". It is a non-Creative Common License that does not provide any exclusive rights to arXiv as per. This is the default permission of arXiv (which does not interfere with the license of the conferences). You can easily find papers from ICML, NeurIPS, and AAAI on their websites as well as arXiv. For, e.g., NeurIPS 2017 began on 4th Dec 2017 and this paper has the latest submission on 6th Dec 2017.

If you say your work was rejected or you are about to submit it, you need to check for the restrictions on uploading the manuscript online before submitting it to the conference. Some conference allows papers to be submitted only if they are not uploaded to sites like arXiv within say 30 days prior to the paper submission deadline.

AAAI does not have restrictions on arXiv as per their policy. ACM also allows non-profit organizations like arXiv as per their policy.
Among all the info on the major publications I could find, I found a common phrase that the paper should not be uploaded to a for-profit digital library. ArXiv is certainly not among them. The reason why editors do not oppose arXiv papers could be that there are a lot of papers that are uploaded to arXiv in order to claim to be first, so banning them would be a big loss for the editors.

Say you comply with the restrictions of the conference and have uploaded your preprint then it brings the risk that reviewers can know who the authors are. They might be biased once they know who the authors are. This works in the favor of renowned/superstars scientists of the field. The reviewer might be a Ph.D. who admires this superstar and hence would try to get it accepted. Or the author might be a known colleague of the author and would hence be biased into accepting it. On the other hand, if the reviewer has some personal bias towards the author then that may be reflected in their scores.

Upload it on arXiv directly

Only uploading to arXiv is of not much academic value because it is not peer-reviewed as you pointed out. Anyone can write an absurd paper with false results and get it published on arXiv. That is why peer-reviewed papers are important.

One reason to upload to arXiv first is to make your work safe from other people coming up with similar approaches. It is not rare at all for people coming up with similar ideas within a duration of 3 months. Deep Learning is a fast-paced domain as it is a hot research field. So getting your paper out first in the name of work in progress saves you from potential rejects from the conference pointing to a similar work done available online before you.

Submit it to a peer-reviewed journal

Same thing as a conference. You can upload your preprint after acceptance of your manuscript with the default license of arXiv. For submitting it before on arXiv needs cross-checking with the rules of the journal. For, e.g., IEEE allows the work to be submitted to arXiv as per.

How do big stars publish?

Coming to your question of how big stars publish. They either publish directly in a conference and then upload to arXiv or they first upload on arXiv, get publicity and citations and then submit to a top conference. Having the preprint available online can also bias the reviewer into accepting their research (as mentioned before). This way they reduce their chance of rejection.

Fun fact

Interestingly, if the preprints become very famous then the reviewer might get confused that your work is a derivative of a popular work which in reality is the preprint of your submitted work! In such cases, you would need to imply this diplomatically. For example, Music Transformer by the Google Brain team (which consisted of authors of the famous Transformer paper "Attention is all you need") was available on arXiv in 2018 and was already being cited by others. When it was submitted to ICLR 2019, a reviewer mistakenly took it as a derivative work of the 2018 arXiv paper and suggested a rejection of the paper. However, after further inspection, he/she realized the confusion which blew their mind! Immediately the paper was given a suitable high score. Source: https://openreview.net/forum?id=rJe4ShAcF7

Where to publish?

Unlike in core electronics and other branches where a journal is much more important than conference papers, in machine learning, conference papers are at par or better than any journal out there. Mostly the quality depends on the Impact Factor of the journal or conference. For example, NeurIPS is a top venue to publish and it's a conference and not a journal.

One reason for conferences to be so important is that they are more popular among researchers. Conferences are fast with their review as compared to the lengthy journal review. This allows researchers to meet and discuss the progress of the field with like-minded people much faster. This is crucial for a rapidly evolving field like artificial intelligence. This makes conference popular which attracts the best researchers to publish their cutting edge which in turn makes the conference more lucrative for other researchers to publish in the same venue as the big stars.

Conferences also allow people from the industry to meet the researchers and hire them. The quick meeting opportunity provided by the conference is beneficial for both the academic researchers as well as the industry for attracting talent.


Machine Learning hardware usage in embedded applications I've been reading a lot about hardware development and implementation for AI/ML, mainly about Deep Learning, and I have a question about its usage.
From what I understand, there are 2 stages for DL: first is training and second is inference. The first is often done on GPUs because of their massive parallelism capabilities among other things, and inference, while can be done on GPUs, it's not used that much, because of power usage, and because the data presented while inferring are much less so the full capabilities of GPUs won't be much needed. Instead FPGAs and CPUs are often used for that.

My understanding also is that a complete DL system will have both, a training system and an inferring system.

My question is that: are both systems required on the same application? Let's assume an autonomous car or an application where visual and image recognition is done, will it have both training system to be trained and an inference system to execute? Or it has only the inference system and will communicate with a distant system which is already trained and has built a database?

Also, if the application has both systems, will it have a big enough memory to store the training data? Given that it can be a small system and memory is ultimately limited.

Deep learning seems mostly to be a buzzword for what is essentially a neural network. You train with a data set to recognize a pattern, then input new data which is then classified by the trained network.

So you train a neural network with 10 different kinds of animals using thousands of pictures. Then you show the network say 100 new images and have the network "guess"  what each animal is.

The point here is that training a neural network would required code for feedback that an application using the trained network would not need. So an application just using the trained network would be a bit more streamlined than an application which could allow additional training data.

What is missing is the ability for machine learning to form hierarchical or otherwise interrelated rules from the trained network so that the trained network is closer to being able to rationally explain why the classification works. Going back to the 10 animal trained network, without formulation of rules during training, there is no practical way for the network to reveal why any of the animals was classified the way it was.


How can neural networks approximate any continuous function but have $\mathcal{VC}$ dimension only proportional to their number of parameters? Neural networks typically have $\mathcal{VC}$ dimension that is proportional to their number of parameters and inputs. For example, see the papers Vapnik-Chervonenkis dimension of recurrent neural networks (1998) by Pascal Koirana and Eduardo D. Sontag and VC Dimension of Neural Networks (1998) by Eduardo D. Sontag for more details.

On the other hand, the universal approximation theorem (UAT) tells us that neural networks can approximate any continuous function. See Approximation by Superpositions of a Sigmoidal Function (1989) by G. Cybenko for more details.

Although I realize that the typical UAT only applies to continuous functions, the UAT and the results about the $\mathcal{VC}$ dimension of neural networks seem to be a little bit contradictory, but this is only if you don't know the definition of $\mathcal{VC}$ dimension and the implications of the UAT.

So, how come that neural networks approximate any continuous function, but, at the same time, they usually have a $\mathcal{VC}$ dimension that is only proportional to their number of parameters? What is the relationship between the two?

I think DuttaA is right in his comment. VC Dimension of Neural Networks establishes VC bounds depending on the number of weights, whereas the UAT refers to a class of neural networks in which the number of weights a particular network can have is not bounded, although it needs to be finite.
I think that we can show, from theorem 2 and the observations below theorem 3 in Approximation by Superpositions of a Sigmoidal Function, that the VC dimension of
$$S=\left\{\sum_{i=1}^N \alpha_i\sigma(y_i^T x + \theta_i) : N\in\mathbb N, \alpha_i, \theta_i \in\mathbb R, y_i\in\mathbb{R}^n \right\}$$
is infinite.
Let $\{(x_i, y_i)\}_{i=1}^k$ be a sample of arbitrary size $k\in\mathbb N$, and let us see that there is a function in $S$ which can correctly classify it, i.e., $S$ shatters $\{x_i\}_{i=1}^k$.
We note $B(x, \varepsilon) := \{ y\in\mathbb{R}^n : d(x,y) < \varepsilon \}$ (this is just standard notation to denote a ball).
First, let $\varepsilon > 0$ be such that $B(x_i, \varepsilon)\cap B(x_j, \varepsilon) = \emptyset$ every time that $i \ne j$.
Now define $D = \cup_{y_i=1} B(x_i, \varepsilon)$. Define $f_{\varepsilon}(x)$ as in the observations below theorem 3 of Cybenko's paper, and use theorem 2 to find a function $G(x)$ in $S$ that classifies correctly all points at least $\varepsilon$ away from the boundary of $D$, i.e., all points in the sample.


Two Models vs One Model for Person Detection and Object Detection Is it possible to do person detection and object detection within one model? The training data would be images annotated with bounding boxes for objects and people.  Because normally object detection and person detection are done separately? Is there any research about models that simultaneously detect both people and objects?

Yes you can classify people bounding box with object detection. State of the art object detection model have people as one of the class in the object detection, as shown here:

As you can see the image have both object bounding box and people bounding box.


"Porpoising" in latter stages of validation loss and MSE charts in Keras Performing a prediction of a continuous y target using Keras, the simple structure of the code revolves around;
model = Sequential()
model.add(Dense(200, input_dim=15, activation= "relu"))
model.add(Dense(750, activation= "relu"))
model.add(Dense(500, activation= "relu"))
model.add(Dense(750, activation= "relu"))
model.add(Dense(500, activation= "relu"))
model.add(Dense(200, activation= "relu"))
model.add(Dense(100, activation= "relu"))
model.add(Dense(50, activation= "relu"))
model.add(Dense(1))

model.compile(loss= 'mse' , optimizer='adam', metrics=['mse','mae'])
history=model.fit(X_train, y_train, batch_size=50,  epochs=150,
                  verbose=1, validation_split=0.2)

This has resulted in the following metric chart;

What might be causing these, and how to eliminate (or greatly reduce) them?
UPDATE: Just reduced the learning rate to 0.0001 per Neil Slater's suggestion, and the loss curve may have had the spikes reduced, though the scale of the graph has changed. The training loss has increased from 0.00007 to 0.00037, the validation loss from 0.0014 to 0.002, and the prediction error increased from 0.037 to 0.046.

I then changed epsilon from it's value of 1e-07 to 0.1 and increased the epochs from 150 to 500. The validation loss increased to 0.0082 and the prediction error increased to 0.093, with the corresponding model loss shown below.

While not an overall improvement at either step, this did remove the spikes as I requested, hence Neil's advice gives me additional considerations to explore and measure within the Adam optimizer (along with other optimizers), so I consider this to have been an important learning experience. One such exploration uncovered this more detailed explanation of optimizers than I had been exposed to before, as well as this 3D visualization of loss topologies and the effect differing optimizers and parameters have on finding the optimal minima (keep a sharp eye on the options being chosen in the upper right corner).

What might be causing these, and how to eliminate (or greatly reduce) them?

It is difficult to be sure just from the graph, but I note you are using the Adam optimiser. It shares with a few other optimisers (most notably RMSProp) that it divides current gradients by a rolling mean of recent gradients to set step sizes. This can cause some minor instability when gradients get close to zero for a long while before growing again. This might happen at a saddle point where only some fraction of the network parameters are critical to results for a few iterations before hitting some other direction of slope where changing the "quiet" parameters becomes important again. This is more likely to occur as loss values become small, and close to perfect convergence.
There are a couple of hyperparameters in Adam that may reduce the effect:

Reduce learning rate. The default learning rate in Adam is often set to 0.001, but you will find a lot of researchers will reduce that to magnitudes around 0.0001 or 0.00001

Increase epsilon. This is effectively the minimum rolling average gradient. The default is 1e-7 but it sometimes needs to be increased significantly, it depends on the loss surface. The official documentation suggests that it may even be useful to increase it up to 1.0


Are there any disadvantages to using a variable population size in genetic algorithms? When implementing a genetic algorithm, I understand the basic idea is to have an initial population of a certain size. Then, we pick two individuals from a population, construct two new individuals (using mutation and crossover), repeat this process X number of times and the replace the old population with the new population, based on selecting the fittest.
In this method, the population size remains fixed. In reality in evolution, populations undergo fluctuations in population sizes (e.g. population bottlenecks, and new speciations).
I understand the disadvantages of variable populations sizes from a biological view are, for example, a bottleneck will reduce the population to minimal levels, so not much evolution will occur. Are there disadvantages to using variable population sizes in genetic algorithms, from a programming perspective? I was thinking the numbers per population could follow a distribution of some sort so they don't just randomly fluctuate erratically, but maybe this does not make sense to do.

Population size is a tricky issue even in pure biological models. Biological population sizes obviously vary. The two great protagonists of the argument were Ronald Fisher and Sewell Wright, with argument being between Fisher favouring few large populations against Wright’s many small interconnected populations. There is evidence that evolution occurs more rapidly in Wright’s model but the evidence is inconclusive. The theory concentrates on the probability that a mutation will occur and then become dominant in a population. In a small population a beneficial mutation is more likely to be selected for reproduction, but premature convergence is a serious danger.  While in a larger population a mutant is less likely to be removed from the population during reproduction. I would strongly recommend a read of Games of life by Karl Sigmund.


Clarifying representation of Neural Nerwork input for Chess Alpha Zero In the Alpha Zero paper (https://arxiv.org/pdf/1712.01815.pdf) page 13, the input for the NN is described. In the beggining of the page, the authors state that:
"The input to the Neural Network is an N x X x (MT + L) image stack [...]"
From this, I understand that (for one training example) each input feature is an 8x8 plane. (Technically speaking, every value of every 8x8 plane is a feature, but for the purpose of the question let's suppose that a plane is an input feature).
In the description of the table on top of the image, the following statement is made:
"[...] Counts are represented by a single real-valued input; other input features are represented by a one-hot encoding using thespecified number of binary input planes. [...]"
I understand how they convert the P1 and P2 pieces to one-hot encodings. My questions are:

When they say single real-valued input, since every input feature should be an 8x8 plane, do they mean that they create an 8x8 plane where every entry has the same single-real value? For example, for the 'Total move count' plane, if 10 moves had been played in the game so far, it would look like the one below?

  move_count_plane = [[10, 10, 10, 10, 10, 10, 10, 10],
                      [10, 10, 10, 10, 10, 10, 10, 10],
                      [10, 10, 10, 10, 10, 10, 10, 10],
                      [10, 10, 10, 10, 10, 10, 10, 10],
                      [10, 10, 10, 10, 10, 10, 10, 10],
                      [10, 10, 10, 10, 10, 10, 10, 10],
                      [10, 10, 10, 10, 10, 10, 10, 10],
                      [10, 10, 10, 10, 10, 10, 10, 10]]

For the 'Repetitions' plane, is it the same case as above? They mean a plane where every value is the number of times a specific board setup has been reached? For example, if a specific position has been reached 2 times, then the repetitions plane for that position would be

  # for a specific timestep in the T=8 step history
  repetitions_plane = [[2, 2, 2, 2, 2, 2, 2, 2],
                       [2, 2, 2, 2, 2, 2, 2, 2],
                       [2, 2, 2, 2, 2, 2, 2, 2],
                       [2, 2, 2, 2, 2, 2, 2, 2],
                       [2, 2, 2, 2, 2, 2, 2, 2],
                       [2, 2, 2, 2, 2, 2, 2, 2],
                       [2, 2, 2, 2, 2, 2, 2, 2],
                       [2, 2, 2, 2, 2, 2, 2, 2]]

? Also, why do they keep 2 repetitions planes? Is it one for every player? (8 repetition planes for the past T=8 moves for P1, and more 8 repetition planes for the past T=8 moves for P2?)
Thanks in advance.

For anyone wondering, I believe to have found the answer:

Yes, it will be an 8x8 plane where all the entries are the same, the number of moves (or mpves with no progress).

There are two repetitions planes (for each position from the most recent T=8 positions):
a) The first repetition plane will be a plane where all the entries are 1's if the position is being repeated for the first time. Else 0's.
b) The second repetition plane will be a plane where all the entries are 1's if the current position is being repeated for the second time. Else 0's.


How can the convolution operation be implemented as a matrix multiplication? How can the convolution operation used by CNNs be implemented as a matrix-vector multiplication? We often think of the convolution operation in CNNs as a kernel that slides across the input. However, rather than sliding this kernel (e.g. using loops), we can perform the convolution operation "in one step" using a matrix-vector multiplication, where the matrix is a circulant matrix containing shifted versions of the kernel (as rows or columns) and the vector is the input.

How exactly can this operation be performed? I am looking for a detailed step-by-step answer that shows how the convolution operation (as usually presented) can be performed using a matrix-vector multiplication.

Is this the usual way the convolution operations are implemented in CNNs?

To show how the convolution (in the context of CNNs) can be viewed as matrix-vector multiplication, let's suppose that we want to apply a $3 \times 3$ kernel to a $4 \times 4$ input, with no padding and with unit stride.
Here's an illustration of this convolutional layer (where, in blue, we have the input, in dark blue, the kernel, and, in green, the feature map or output of the convolution).

Now, let the kernel be defined as follows
$$
\mathbf{W} =
\begin{bmatrix}
w_{0, 0} & w_{0, 1} & w_{0, 2} \\
w_{1, 0} & w_{1, 1} & w_{1, 2} \\
w_{2, 0} & w_{2, 1} & w_{2, 2}
\end{bmatrix}
\in
\mathbb{R}^{3 \times 3}
$$
Similarly, let the input be defined as
$$
\mathbf{I} =
\begin{bmatrix}
i_{0, 0} & i_{0, 1} & i_{0, 2} & i_{0, 3} \\
i_{1, 0} & i_{1, 1} & i_{1, 2} & i_{1, 3} \\
i_{2, 0} & i_{2, 1} & i_{2, 2} & i_{2, 3} \\
i_{3, 0} & i_{3, 1} & i_{3, 2} & i_{3, 3} \\
\end{bmatrix}
\in
\mathbb{R}^{4 \times 4}
$$
Then the convolution above (without padding and with stride 1) can be computed as a matrix-vector multiplication as follows. First, we redefine the kernel $\mathbf{W}$ as a sparse matrix $\mathbf{W}' \in \mathbb{R}^{4 \times 16}$ (which is a circulant matrix because of its circular nature) as follows.
$$
{\scriptscriptstyle
\mathbf{W}' =
\begin{bmatrix}
w_{0, 0} & w_{0, 1} & w_{0, 2} & 0 & w_{1, 0} & w_{1, 1} & w_{1, 2} & 0 &  w_{2, 0} & w_{2, 1} & w_{2, 2} & 0 & 0 & 0 & 0 & 0 \\
0 & w_{0, 0} & w_{0, 1} & w_{0, 2} & 0 & w_{1, 0} & w_{1, 1} & w_{1, 2} & 0 & w_{2, 0} & w_{2, 1} & w_{2, 2} & 0 & 0 & 0 & 0  \\
0 & 0 & 0 & 0 & w_{0, 0} & w_{0, 1} & w_{0, 2} & 0 & w_{1, 0} & w_{1, 1} & w_{1, 2} & 0 &  w_{2, 0} & w_{2, 1} & w_{2, 2} & 0 \\
0 & 0 & 0 & 0 & 0 & w_{0, 0} & w_{0, 1} & w_{0, 2} & 0 & w_{1, 0} & w_{1, 1} & w_{1, 2} & 0 &  w_{2, 0} & w_{2, 1} & w_{2, 2}
\end{bmatrix}
}
$$
Similarly, we reshape the input $\mathbf{I}$ as a 16-dimensional vector $\mathbf{I}' \in  \mathbb{R}^{16}$.
$$
{\scriptstyle
\mathbf{I}' =
\begin{bmatrix}
i_{0, 0} & i_{0, 1} & i_{0, 2} & i_{0, 3} & i_{1, 0} & i_{1, 1} & i_{1, 2} & i_{1, 3} & i_{2, 0} & i_{2, 1} & i_{2, 2} & i_{2, 3} & i_{3, 0} & i_{3, 1} & i_{3, 2} & i_{3, 3}
\end{bmatrix}^T
}
$$
Then the convolution of $\mathbf{W}$ and $\mathbf{I}$, that is
$$\mathbf{W} \circledast \mathbf{I} = \mathbf{O} \in \mathbb{R}^{2 \times 2},$$
where $\circledast$ is the convolution operator, is equivalently defined as $$\mathbf{W}' \cdot \mathbf{I}' = \mathbf{O}' \in \mathbb{R}^{4},$$ where $\cdot$ is the matrix-vector multiplication operator. The produced vector $\mathbf{O}'$ can then be reshaped as a $2 \times 2$ feature map.
You can easily verify that this representation is correct by multiplying e.g. the 16-dimensional input vector $\mathbf{I}'$ with the first row of $\mathbf{W}'$ to obtain the top-left entry of the feature map.
$$w_{0, 0} i_{0, 0} + w_{0, 1} i_{0, 1} + w_{0, 2} i_{0, 2} + 0 i_{0, 3} + w_{1, 0} i_{1, 0} + w_{1, 1} i_{1, 1} + w_{1, 2}i_{1, 2} + 0 i_{1, 3} +  w_{2, 0} i_{2, 0} + w_{2, 1}i_{2, 1} + w_{2, 2} i_{2, 2} + 0 i_{2, 3} + 0 i_{3, 0} + 0 i_{3, 1} + 0 i_{3, 2} + 0 i_{3, 3} = \\
w_{0, 0} i_{0, 0} + w_{0, 1} i_{0, 1} + w_{0, 2} i_{0, 2} + w_{1, 0} i_{1, 0} + w_{1, 1} i_{1, 1} + w_{1, 2}i_{1, 2} + w_{2, 0} i_{2, 0} + w_{2, 1}i_{2, 1} + w_{2, 2} i_{2, 2} = \\
\mathbf{O}'_{0}  \in \mathbb{R} ,$$ which is equivalent to an element-wise multiplication of $\mathbf{W}$ with the top-left $3 \times 3$ sub-matrix of the input followed by a summation over all elements (i.e. convolution), that is
$$
\sum \left(
\begin{bmatrix}
w_{0, 0} & w_{0, 1} & w_{0, 2} \\
w_{1, 0} & w_{1, 1} & w_{1, 2} \\
w_{2, 0} & w_{2, 1} & w_{2, 2}
\end{bmatrix}
\odot
\begin{bmatrix}
i_{0, 0} & i_{0, 1} & i_{0, 2} \\
i_{1, 0} & i_{1, 1} & i_{1, 2}  \\
i_{2, 0} & i_{2, 1} & i_{2, 2}
\end{bmatrix}
\right) = \mathbf{O}_{0, 0} = \mathbf{O}'_{0} \in \mathbb{R},
$$
where $\odot$ is the element-wise multiplication and $\sum$ is the summation over all elements of the resulting matrix.
The advantage of this representation (and computation) is that back-propagation can be computed more easily by just transposing $\mathbf{W}'$, i.e. with $\mathbf{W}'^T$.
Take also a look at this Github repository that explains how the convolution can be implemented as matrix multiplication.


What is the difference between exhaustive nearest neighbor search and k-nearest neighbour search? I have two lists of feature vectors calculated from pre-trained CNN for image retrieval task:

Query: FV_Q and Reference FV_R.

>>> FV_R.shape
(3450, 128)

>>> FV_Q.shape
(3450, 128)

I am a little confused between the concept of exhaustive nearest neighbor search and k-nearest neighbor search.

In python, I use from sklearn.neighbors import KDTree to extract top k = 5 similar images from the reference database, given the query image!

Can somebody explain if there might be any similarities/differences between these two concepts?

Am I making a mistake somewhere in my feature vector comparison?

The exhaustive nearest neighbor search performs an exhaustive search of the nearest neighbor (i.e. the closest image or matches, depending on your application). Here an exhaustive search means that you will compare your query image with any other image, or, in the case of feature matching, you will compare every feature of the query image with every other feature of the target/reference image.

The $k$-nearest neighbor search finds the $k$ closest images. You typically do that by first building a KD-tree with your query (or target) image, so that to speed up the search for the $k$-nearest neighbors of your query image.

(Btw, in case you are interested in the concept of feature matching, in OpenCV, the class that you can use to perform an exhaustive search for the matches between two images is the BFMatcher (which stands for Brute Force Matcher), while the class to perform the search with a KD-tree for the k-nearest neighbhours is FlannBasedMatcher).


How can I find what does an specific neuron do in neural network? How can I know what each neuron does in NN?
Consider the Playground from Tensorflow, there are some hidden layers with some neurons in each. Each of them shows a line(horizontal or vertical or ...). Where these shapes come from. I think they are understandable for nn not a person!

In TensorFlow Playground, the horizontal line show where each class is separated for each neuron. What happens when you take any intermediate neuron to make the decision? You can see the answer by the line provided by that neuron. And this decision is a result of the weighted sum from the decisions of the previous neurons (up to activation).

Take the middle-top neuron in the link you share, which is an almost horizontal line - slightly tilted to the right. This neuron classifies everything above it as a blue, and everything below it as an orange. Hover over the neuron to see a larger picture on the output.

You can also see how this is actually calculated by hovering over the line coming from the neurons in the previous layer to the neuron you are looking at. For the case of the same neuron (center-top), the weight coming from the first input ($x_1$) is 0.091, while from the second one ($x_2$), it is 0.49. The neuron ends up being almost horizontal because the contribution from the horizontal input ($x_2$) is so much larger compared to the vertical one ($x_1$).

Of course you need to take into account the nonlinearity coming from the activation function but the idea presented above is the essence of it. The example uses tanh activation, which behaves very linear in its intermediate region so one can ignore this issue to some extend for this particular case.

Edit: It appears that the values for the weights change at every browser session, so the neuron I describe might look a little different to you. To get the same configuration, simply click on the colored lines between neurons to edit them and use the values above for the connections.


How can a neural network learn to play sudoku? I'm just beginning to understand neural networks and I've performed a couple of successful tests with numerical series where the NN was trained to find the odd one or a missing value. It all works pretty well.

The next test I wanted to perform was to approxmimate the solution of a Sudoku which, I thought could also be seen as a special kind of numerical series. However, the results are really confusing.

I'm using an MLP with 81 neurons in each of the three layers. All output neurons show a strong tendency to yield values that are close to either 0 or 1. I have scaled and truncated the output values. The result can be seen below:

Expected/Actual Solution:     Neural Net's Solution:

6 2 7 3 5 0 8 4 1             9 0 9 9 9 3 0 0 3
3 4 8 2 1 6 0 5 7             0 9 9 0 0 0 9 9 0
5 1 0 4 7 8 6 2 3             0 9 1 9 9 0 2 0 4
1 6 4 0 2 7 5 3 8             0 0 5 0 0 9 0 0 7
2 0 3 8 4 5 1 7 6             0 0 0 0 0 9 9 0 9
7 8 5 1 6 3 4 0 2             9 9 9 9 0 6 2 9 0
0 5 6 7 3 1 2 8 4             0 0 0 0 9 9 0 9 0
4 3 1 5 8 2 7 6 0             9 9 0 0 0 0 9 0 9
8 7 2 6 0 4 3 1 5             9 9 0 9 9 0 9 0 9

The training set size is 100000 Sudokus while the learning rate is a constant 0.5. I'm using NodeJS/Javascript with the Synaptic library.

I don't expect a perfect solution from you guys, but rather a hint if that kind of behavior is a typical symptom for a known problem, like too few/many neurons, small training set, etc.

I think it is the wrong way to frame sudoku as a regression problem in neural networks.
Firstly, you have to understand what regression is. "Regression" is when you predict a value given certain parameters, where the parameters are related to the value you have to predict. This happens because at the core neural networks are "function approximators", they model the function by adjusting their weights using lots of data. They tend to form a highly non-linear boundary to separate classes internally in a high dimensional data-space.

The sudoku doesn't fit in this scenario, the combinatorial complexity of sudoku is way too high for a neural network even if you add many layers to it, it is a totally different problem in its own right. You simple can't "regress" the right values of a perfect sudoku here, they are not numbers like "pixel " intensities in images.

However, you could apply reinforcement learning techniques to learn an optimal policy to solve sudoku.

And you have mentioned an "approximate" solution for the sudoku, what do you mean by "approximate"?  If you mean by this that only a few squares are out of place, then it is a wrong assumption, because neural networks are proven to be good image classifiers, as they are robust to translational invariance in this case, that is not what you need.

You could, however, do a small experiment to see what the neural network actually learns, replace the numbers by pixel value intensities and train a generative adversarial network on the sudoku images and see the images of sudokus produced by it, to see what actually the network can't learn.


Why tf object detection api needs so few pictures? I am wondering why tf object detection api needs so few picture samples for training while regular cnns needs many more?

What I read in tutorials is that tf object detection api needs around 100-500 pictures per class for training (is it true?) while regular CNNs need many many more samples, like tens of thousands or more. Why is it so?

I guess that they need so little data because their models are already trained on huge datasets, and they are just transferring the learning (using those pre-trained models as starting point).


Are Asimov's Laws flawed by design, or are they feasible in practice? Isaac Asimov's famous Three Laws of Robotics originated in the context of Asimov's science fiction stories. In those stories, the three laws serve as a safety measure, in order to avoid untimely or manipulated situations from exploding in havoc.

More often than not, Asimov's narratives would find a way to break them, leading the writer to make several modifications to the laws themselves. For instance, in some of his stories, he modified the First Law, added a Fourth (or Zeroth) Law, or even removed all Laws altogether.

However, it is easy to argue that, in popular culture, and even in the field of AI research itself, the Laws of Robotics are taken quite seriously. Ignoring the side problem of the different, subjective, and mutually-exclusive interpretations of the laws, are there any arguments proving the laws themselves intrinsically flawed by their design, or, alternatively, strong enough for use in reality? Likewise, has a better, stricter security heuristics set being designed for the purpose?

Asimov's laws are not strong enough to be used in practice. Strength isn't even a consideration, when considering that since they're written in English words would first have to be interpreted subjectively to have any meaning at all. You can find a good discussion of this here.

To transcribe an excerpt:

  How do you define these things? How do you define "human", without first having to take a stand on almost every issue. And if "human" wasn't hard enough, you then have to define "harm", and you've got the same problem again. Almost any really solid unambiguous definitions you give for those words—that don't rely on human intuition—result in weird quirks of philosophy, leading to your AI doing something you really don't want it to do.

One can easily imagine that Asimov was smart enough to know this and was more interested in story-writing than designing real-world AI control protocols.

In the novel Neuromancer, it was suggested that AIs could possibly serve as checks against each other. Ray Kurzweil's impending Singularity, or the possibility of hyperintelligent AGIs otherwise, might not leave much of a possibility for humans to control AIs at all, leaving peer-regulation as the only feasible possibility.

It's worth noting that Eliezer Yudkowsky and others ran an experiment wherein Yudkowsky played the role of a superintelligent AI with the ability to speak, but no other connection outside of a locked box. The challengers were tasked simply with keeping the AI in the box at all costs. Yudkowsky escaped both times.


Is it possible to guide a reinforcement learning algorithm? I have just started to study reinforcement learning and, as far as I understand, existing algorithms search for the optimal solution/policy, but do not allow the possibility for the programmer to suggest a way to find the solution (to guide their learning process). This would be beneficial for finding the optimal solution faster.

Is it possible to guide the learning process in (deep) reinforcement learning?

The programmer already guides the RL algorithm (or agent) by specifying the reward function. However, the reward function alone may not be sufficient to learn efficiently and fast, as you correctly noticed.

To attempt to solve this inefficiency problem, one solution is to combine reinforcement learning with supervised learning. For example, the paper Deep Q-learning from Demonstrations (2017) by Todd Hester et al. describes an approach to achieve this.

The paper Active Reinforcement Learning (2008) by Arkady Epshteyn et al. also tries to solve this problem but by incorporating approximations (given by domain experts) of the MDP.

There are probably many other possible solutions. In fact, all model-based RL algorithms could probably fall into this category of algorithms that estimate or incorporate the dynamics of the environment to find a policy more efficiently.


What are "development test sets" used for? This is a theoretical question. I am a newbie to artificial intelligence and machine learning, and the more I read the more I like this. So far, I have been reading about the evaluation of language models (I am focused on ASR), but I still don't get the concept of the development test sets.
The clearest explanation I have come across is the following (taken from chapter 3 of the book Speech and Language Processing (3rd ed. draft) by Dan Jurafsky and James H. Martin)

Sometimes we use a particular test set so often that we implicitly tune to its characteristics. We then need a fresh test set that is truly unseen. In such cases, we call the initial test set the development test set or, devset.

In any case, I still don't understand why an additional test has to be used. In other words, why aren't training and test sets enough?

In machine learning, you normally split your data into 3 parts (80-10-10%).
The first part (80% of your initial data) is for the training of your ML model: this is known as the training dataset.
The second part (10%) is the development set (or dataset), aka validation set. This is used as measuring your performance with various hyperparameters (e.g. in neural networks: layer size).
After you found your best hyperparameters, you learn the model again on the training set, and then test it on your test dataset (10%), which the model has never seen before. Your result on the test data is now a good indicator how your model prediction quality is in the real world (because it was never optimized for this test data).


How to get good results with GAN and some thousands of images? I'm trying to generate images at minimum of size 128 x 128 with a Generative Adversarial Network. I already tried a SAGAN pytorch implementation, but I'm not very happy with results. The images look cool but and I see some correct shape but without explanation you wouldn't know what the images are about. I have a dataset of 4000 images. Lightness, colors and shapes vary a lot, but they are similar in style and on what they portray.

With a Google Cloud V100 GPU the GAN would run a week to two with default parameters. Does this sound realistic time for this kind of dataset? It's definitely not feasible for me.
Is 4000 images enough to train a GAN from scratch?
Is there any implementation with pytorch/keras that would be good to get nice results with?

With a Google Cloud V100 GPU the GAN would run a week to two with
  default parameters. Does this sound realistic time for this kind of
  dataset? It's definitely not feasible for me.

Yes, V100s are quite beefy. You shouldn't even need a week. Obviously this is based on my experience with various problems, rather than a concrete calculation.

  Is 4000 images enough to train a GAN from scratch?

For the size you want to generate, it is still on the edge of what would constitute a decent training set. You will get some results(depending on architecture) but will probably want to grab more data if at all possible.

  Is there any implementation with pytorch/keras that would be good to
  get nice results with?

I would check out this link: https://github.com/eriklindernoren/Keras-GAN. It has some nice implementations in Keras. As far as the particular GAN to use, I would start out with a vanilla GAN that fits your purposes and focus on toying with hyperparameters, and if that doesn't work, look into one of the other variations that correlate well with your problem.


How are mujoco environments used for meta-rl? Afaik, investigating meta reinforcement learning algorithms requires a collection of two or more environments which have similar structure but are still different enough.
When I read this paper it was unclear to me what the meta-training and meta-testing environments were.
For eg., a graph is given for Ant-Fwd-Bkwd showing its performance with number of gradient steps. I'm guessing these are the meta-testing performances. So, which environment was it 'meta-trained' on?
Was it meta-trained on the same Ant-Fwd-Bkwd environment?

According to this paper (PEARL):

These locomotion task families require adaptation across reward
functions (walking direction for Half-CheetahFwd-Back, Ant-Fwd-Back,
Humanoid-Direc-2D, target velocity for Half-Cheetah-Vel, and goal
location for Ant-Goal2D) or across dynamics (random system parameters
for Walker-2D-Params).

It looks like different versions of the same environment with differing reward functions are used. For eg., Forward direction might be rewarded positively in one version, Negative in another, Both directions rewarded positively in yet another.


What is the most accurate pretrained sentiment analysis model by 2019? I've been using OpenAI's 2017 Sentiment Neuron implementation (https://github.com/openai/generating-reviews-discovering-sentiment) for a while, because it was easy to set up and was the most accurate on benchmarks. What is the most accurate alternative now that I should use?

For best results, I'd recommend Google Cloud Machine Learning. It has [Natural Language Processing API] (https://cloud.google.com/natural-language/docs/basics) with Sentiment, Entity, and Entity-Sentiment analysis.

You can implement in C++, PHP, Python, or other languages. This does require running a virtual machine instance on Google Cloud. TensorFlow can also be used for NLP sentiment analysis. The only drawback is it's not economical.

As an alternative, this is something I came across recently by Fast Text : http://fasttext.cc.
They have some pre-trained models trained on yelp and some other datasets. They support a lot of languages


Can a vanilla neural network theoretically achieve the same performance as CNN? I perfectly understand that CNN takes into account the local dependency of each pixel to the nearby pixels. In addition, CNNs are spatially invariant which means that they are able to detect the same feature anywhere in the image. These qualities are useful in image classification problems given the nature of the problem.

How does a vanilla neural net exactly falls short on these properties? Am I right in claiming that a vanilla neural net has to learn a given feature in every part of the image? This is different than how a CNN does it, which learns the feature once and then detects it anywhere in the image.

How about local pixel dependancy? Why can't a vanilla neural network learn the local dependency by relating one pixel to its neighbors in the 1D input?

In other words, is there more information present while training a CNN that are simply absent when training a normal NN? Or is a CNN just better at optimizing in the space of image classification problems?

All CNNs can be represented as vanilla networks on the flattened image data. Just to do so, you would need A LOT of parameters (most of which would be 0) for what CNNs do freely. You can think of a CNN as reusing a filter on a masked input (whichever receptive field it's looking at whatever point during the convolution) repetitively.

In other words, fully connected layers use all the information, so it can still learn spatial dependence as a CNN does, while CNNs for each neuron only look at a specific receptive field and will reuse that filter for all neurons in that channel. This constraint saves computation and allows wider and deeper models under some budget.

This is nice because the hypothesis of why CNN's work are, is that at each point in the network we care about looking at localized features rather than global ones and that creating a composition of these makes it so even if each neuron only relates to a handful of neurons in the previous layer, the receptive field from the initial image can still be quite large if not the whole thing.

Take away: CNNs are an efficient implementation of a vanilla NN, given the locality constraint that each neuron only looks at a small localized subset of neurons from the previous layer.


How to deal with changing video frame sizes in a CNN? How to deal with videos where the frame sizes are not the same frame to frame?

For example this video moves up and down and when it does, the video part of the screen has a different amount of pixels vertically.

How to deal with different frame sizes in a CNN?

Well, the easiest way in order to make different frame sizes work with a convolutional neural network, is to process them using a different operations such as scaling, and or cropping in order to make them the same size. I do not know of any way currently where different frame sizes can be inputted directly into the neural network, as in order to do that you would need to almost completely change the neural network architecture. If the frames have a small difference in size, you may be able to get away with scaling the frame to the dimensions of the largest frame. This also depends upon what kind of problem you are trying to solve. While this method would work if you are trying to classify an image into a couple categories, it would not work if your are trying to do tasks such as object location. If you are doing one of the latter tasks, you should be able to pad the image with blank pixels in order to make the frame you are trying to process the dimensions of the largest one. This would work quite well, especially for object location detection. Finally, another solution is to crop the larger images to the size of the smallest one. While this will work just as well as padding, if you need any image data from the very edges of the frame, this will not work very well. So in conclusion, there are three different methods you could apply. Me personally, I would go with padding because it does not remove any important image data, and it will work very well for tasks such as object location detection. If you are doing a classification task though, scaling would work also. Finally, you could crop the images, but this would have to be evaluated for your specific task.


Can Q-learning working in a multi agent environment where every agent learns a behaviour independently? I am currently exploring multi-agent reinforcement learning. I have multiple agents that communicate with each other and a central service that maintains the environment state.

The central service dispatches some information at regular intervals to all the agents (Lets call this information as energy). The information can be very different for all the agents.

The agents on reception of this information select a particular action. The execution of the action should leave the agent as well as the environment in a positive state. The action requires a limited amount of energy which might change on every timestep. If a agent does not have sufficient energy to it may request for energy from other agents. The other agents may grant or deny this request.

If all the agents are able to successfully perform their actions and leave the environment in a positive state they get a positive reward.

As the environment is stochastic, where a agent's behavior is dependent on another agent can approximate Q Learning be used here?

Not particularly sure what you are asking, so the question that I will be answering is this:

  Can Q learning be used to estimate a value that depends on another value in the Q Learning Matrix even though there is a certain amount of unpredictability involved?

The answer is yes!

I will use the example of a robotic arm trying to reach a point in space since that is what I am most familiar with.

Imagine a robotic arm with a shoulder, elbow, and wrist joint. The desired elbow value depends very much on the shoulder value which is also being learned. Given enough iterations, the Q learning algorithm will come up with a solution (out of possibly many) for the elbow joint based on where the shoulder joint is at that time.

The intrinsic unpredictability (air drag, motor power, etc in this case) is countered intuitively by the Q-learning algorithm iteratively learning what works best.


Alpha Zero queen promotion "The final 9 planes encode possible underpromotions for pawn moves or captures in two possible diagonals, to knight, bishop or rook  respectively.   Other  pawn  moves  or  captures  from  the  seventh  rank  are  promoted  to  a queen."
Doesn't this mean that the network does not know that it can promote to a queen?

It means that there is no explicit coding of action choices to promote to queen, it is the default assumption if the underpromotion actions are not taken.

The Alpha Zero chess implementation can represent promotion to queen by not selecting an underpromotion action, whilst moving a pawn so that it qualifies for promotion.


Why does each component of the tuple that represents an action have a categorical distribution in the TRPO paper? I was going through the TRPO paper, and there was a line under Appendix D "Approximating Factored Policies with Neural Networks" in the last paragraph which I am unable to understand

The action consists of a tuple $(a_1, a_2..... , a_K)$ of integers $a_k\in\{1, 2,......,N_k\} $ and each of these components is assumed to have a categorical distribution.

I can't seem to get how each component has a categorical distribution. I think it should be the tuple that has a categorical distribution.
I think I am getting something wrong.

I'm not sure specifically which Atari games present this type of action space, but you can imagine a game in which you can perform multiple different types of actions at the same timestep (i.e. the different "factors" they mention in the paper).
As an example, imagine a game in which you can both move and jump at the same time. In that case, you might have a 4-dimensional discrete action space for moving (NSWE), and a 2-dimensional discrete action space for jumping (yes/no jump), both of which will require a Categorical distribution which has the size of that factor ($N_k$ in the paper).
So in this case, you would need to have a categorical distribution for each factor, unless you were to turn these two factors into 1 joint 4*2-dimensional factor and learn a single categorical distribution on that (which would likely be less efficient).


How can I prove that all the a-cuts of any fuzzy set A defined on $R^n$ are convex? How can I prove that all the a-cuts of any fuzzy set A defined on $R^n$
are convex if and only if

$$\mu_A(\lambda r + (1-\lambda)s) \geq min \{\mu_A(r), \mu_A(s)\}$$

such that $r, s \in R^n$, $\lambda \in [0, 1]$ ?

That's a fuzzy question on my assignment. Any idea on how to start with?

We can assume without loss of generality that
\begin{equation}
\min\{\mu_A(r), \mu_A(s)\} = \mu_A(r) = \alpha.
\end{equation}
$\implies$

a-cut of fuzzy set $A$ is on $R^n$ is convex. A-cut can be defined as
\begin{equation}
A = \{x \in R^n| \mu_A(x) \geq \alpha\}
\end{equation}
If we take two elements $r$ and $s$, by the definition of convex set, number $\lambda r + (1 + \lambda)s$ is also an element of that set. Since it's an element of that set that means
\begin{equation}
\mu_A(\lambda r + (1 + \lambda)s) \geq \alpha
\end{equation}

$\impliedby$

$\mu_A(\lambda r + (1 + \lambda)s) \geq \alpha$.

We know from $\min\{\mu_A(r), \mu_A(s)\} = \mu_A(r) = \alpha$ that $\mu_A(s) > \alpha$. We have an affine combination $\lambda r + (1 + \lambda)s$ for which also $\mu_A(\lambda r + (1 + \lambda)s) \geq \alpha$ so we know that all numbers $\lambda r + (1 + \lambda)s$ satisfy inequality $\mu_A(\cdot) \geq \alpha$ (belong to the same set as $r$ and $s$) which means this is a convex set again by the definition of a convex set.


How are Artificial Neural Networks and the Biological Neural Networks similar and different? I've heard multiple times that "Neural Networks are the best approximation we have to model the human brain", and I think it is commonly known that Neural Networks are modelled after our brain.

I strongly suspect that this model has been simplified, but how much?

How much does, say, the vanilla NN differ from what we know about the human brain? Do we even know?

We all know that artificial neural networks (ANNs) are inspired by but most of them are only loosely based on the biological neural networks (BNNs).
We can analyze the differences and similarities between ANNs and BNNs in terms of the following components.
Neurons
The following diagram illustrates a biological neuron (screenshot of an image from this book).

The following one illustrates a typical artificial neuron of an ANN (screenshot of figure 1.14 of this book).

Initialization
In the case of an ANN, the initial state and weights are assigned randomly. While for BNNs, the strengths of connections between neurons and the structure of connections don't start as random. The initial state is genetically derived and is the byproduct of evolution.
Learning
In BNN, learning comes from the interconnections between myriad neurons in the brain. These interconnections change configuration when the brain experiences new stimuli. The changes result in new connections, strengthening of existing connections, and removal of old and unused ones.
ANNs are trained from scratch usually using a fixed topology (remember topology changes in case of BNNs), although the topology of ANN can also change (for example, take a look at NEAT or some continual learning techniques), which depends on the problem being solved. The weights of an ANN are randomly initialized and adjusted via an optimization algorithm.
Number of neurons
Another difference (although this difference is always smaller) is in the number of neurons in the network. A typical ANN consists of hundreds, thousands, millions, and, in some exceptional (e.g. GPT-3), billions of neurons. The BNN of the human brain consists of billions. This number varies from animal to animal.
Further reading
You can find more information here or here.


Why is an embedding of dimension 400 enough to represent 70000 words? I am learning PyTorch on Udacity. In lesson 8, section 11: Training the Model, the instructor writes:

Then I have my embedding and hidden dimension. The embedding dimension is just a smaller representation of my vocabulary of 70k words and I think any value between like 200 and 500 or so would work, here. I've chosen 400. Similarly, for our hidden dimension, I think 256 hidden features should be enough to distinguish between positive and negative reviews.

There are more than 70000 different words. How could those more than 70000 unique words be represented by just 400 embeddings? How does an embedding look like? Is it a number?
Moreover, why would 256 hidden features be enough?

The specific term you are looking for is "word embedding" and not just "embedding".
How to numerically represent textual data?
Neural networks (typically) require as inputs (and produce as outputs) numerical data (i.e. numbers, vectors, matrices, or higher-dimensional arrays). So, when processing textual data, we first need to encode (or convert) the text into a numerical representation. There are different ways to do it, such as

one-hot encoding (in that case, if you have 70000 words, you would have sparse vectors with 70000 entries where only one of those entries is equal to $1$ and all other entries are $0$: see this article  for more info)

map each word to a number (in this case, you would have 70000 numbers, one for each word)

word embeddings

Each of these representations has different benefits and drawbacks. For instance, if you map each word to a number, then you just need to keep track of $70000$ numbers. In the case of one-hot encoding or word embeddings, you will need more memory. However, nowadays, word embeddings are widely used in natural language processing/understanding/generation tasks (and given that your question is about word embeddings), so let me briefly describe them.
Word embeddings
There are different word embedding techniques (such as word2vec). However, they are all based on the same ideas

Words that are similar (or related) in meaning should be mapped to vectors (i.e. the "word embeddings") that are also similar in some sense (for instance, their cosine similarity should be high). For instance, the words "man" and "boy" should be mapped to vectors that are similar.

These word embeddings are learned (rather than hard-coded or manually specified) given the data

The size of the word embeddings is a hyper-parameter (this should answer your question!)

Hyper-parameters
To answer your question(s) more directly, the choice of the dimension of the embeddings or the number of "hidden features" (which are both hyper-parameters) was probably more or less arbitrary or based on the instructor's experience. In general, it is difficult to determine the optimal choice of any hyper-parameter. Sometimes you can just use numbers that other people have used in the past and have noticed that work "well enough". If you really want to find more appropriate values of the hyper-parameters, you could use some hyper-parameter optimization technique, such as Bayesian optimization or a simple grid search.
Further reading
You can find many resources online that explain the concept of "word embeddings" more in detail. For instance

What Are Word Embeddings for Text? (by Jason Brownlee)
Word embeddings (A TensorFlow article/tutorial on word embeddings).


How powerful a computer is required to simulate the human brain? How much processing power is needed to emulate the human brain? More specifically, the neural simulation, such as communication between the neurons and processing certain data in real-time.
I understand that this may be a bit of speculation and it's not possible to be accurate, but I'm sure there is some data available or research studies that attempted to estimate it based on our current understanding of the human brain.

H+ magazine wrote an estimate in 2009 that seems broadly comparable to other things I've seen; they think the human brain is approximately 37 petaflops. A supercomputer larger than that 37 petaflop estimate exists today.

But emulation is hard. See this SO question about hardware emulation or this article on emulating the SNES, in which they require 140 times the processing power of the SNES chip to get it right. This 2013 article claims that a second of human brain activity took 40 minutes to emulate on a 10 petaflop computer (a 2400-times slowdown, not the 4-times slowdown one might naively expect).

And all this assumes that neurons are relatively simple objects! It could be that the amount of math we have to do to model a single neuron is actually much more than the flops estimate above. Or it could be the case that dramatic simplifications can be made, and if we knew what the brain was actually trying to accomplish we could do it much more cleanly and simply. (One advantage that ANNs have, for example, is that they are doing computations with much more precision than we expect biological neurons to have. But this means emulation is harder, not easier, while replacement is easier.)


What is the double sample problem in reinforcement learning? According to the SBEED: Convergent Reinforcement Learning with
Nonlinear Function Approximation for convergent reinforcement learning, the Smoothed Bellman operator is a way to dodge the double sample problem? Can someone explain to me what the double sample problem is and how SBEED solves it?

The double sampling problem is referenced in Chaper 11.5 Gradient Descent in the Bellman Error in Reinforcement Learning: An Introduction (2nd edition).
From the book, this is the full gradient descent (as opposed to semi-gradient descent) update rule for weights of an estimator that should converge to a minimal distance from the Bellman error:

$$w_{t+1} = w_t + \alpha[\mathbb{E}_b[\rho_t[R_{t+1} + \gamma\hat{v}(S_{t+1},\mathbf{w})] - \hat{v}(S_{t},\mathbf{w})][\nabla\hat{v}(S_{t},\mathbf{w})- \gamma\mathbb{E}_b[\rho_t\nabla\hat{v}(S_{t+1},\mathbf{w})]]$$
[...] But this is naive, because the equation above involves the next state, $S_{t+1}$, appearing in two
expectations that are multiplied together. To get an unbiased sample of the product,
two independent samples of the next state are required, but during normal interaction
with an external environment only one is obtained. One expectation or the other can be
sampled, but not both.

Basically, unless you have an environment that you can routinely re-wind and re-sample to get two independent estimates (for $\hat{v}(S_{t+1},\mathbf{w})$ and $\nabla\hat{v}(S_{t+1},\mathbf{w})$) then the update rule that naturally arises from gradient descent on the Bellman error does will work any better than other approaches, such as semi-gradient methods. If you can do this rewind process on every step, then it may be worth it because of the guarantees of convergence, even in off-policy with non-linear approximators.
The paper proposes a workaround for this issue, keeping the robust convergence guarantees, but dropping the need to collect two independent samples of the same estimate on each step.


How is the reward in reinforcement learning different from the label in supervised learning problems? How is the notion of immediate reward used in the reinforcement learning different from the notion of a label we find in the supervised learning problems?

Reward in reinforcement learning (RL) is entirely different from a supervised learning (SL) label, but can be related to it indirectly.
In a RL control setting, you can imagine that you had a data oracle that gave you SL training example and label pairs $x_i, y_i$ where $x_i$ represents a state and $y_i$ represents the correct action to take in that state in order to maximise the expected return. For simplicity I will use $G_t = \sum_{k=1}^{\infty} \gamma^k R_{t+k+1}$ for return here (where $G_t$ and $R_t$ are random variables), there are other definitions, but the argument that follows doesn't change much for them.
You can use the oracle to reduce the RL training process to SL, creating a policy function $\pi(s): \mathcal{S} \rightarrow \mathcal{A}$ learned from a dataset that the oracle output. This clearly relates SL with RL, but how do $x_i, y_i$ from SL relate to $s_t, a_t$ from RL in terms of reward values?
The states can relate directly (as input):
$$x_i \equiv s_t$$
The action from the policy function is more indirect, if you want to see how reward is involved:
$$y_i \equiv \pi^*(s_t) = \text{argmax}_a \mathbb{E}_{A \sim \pi^*}[\sum_{k=1}^{\infty} \gamma^k R_{t+k+1} | S_t=s_t, A_t=a]$$
Note the oracle is represented by the optimal policy function $\pi^*(s_t)$, and the expectation is conditional both on the start conditions of state and action plus following the optimal policy from then on (which is what $A \sim \pi^*$ is representing).
In practice the optimal policy function is unknown when starting RL, so the learning process cannot be reduced to a SL problem. However, you can get close in some circumstances by creating a dataset of action choices made by an expert at the problem. In that case a similar relationship applies - the label (of which action to take) and immediate reward are different things but can be related by noting that the expert behaviour is close to the $\text{argmax}$ over actions of expected sums of future reward.
Another way to view the difference:

In SL, the signal from the label is an instruction - "associate these two values". Data is supplied to the learning process by some other independent process, and can be learned from directly

In RL, the signal from the reward is a consequence - "this is the value, in context, of what you just did", and needs to be learned from indirectly. Data is not supplied separately from the learning process, but must be actively collected by it - deciding which state, action pairs to learn from is part of the agent's learning task


Why do you need to retrain GPT-2? I'm following this tutorial, and I wonder why is there a train-step - why is it necessary? I thought the whole idea of GPT-2 is that you do not need to train it on specific text domain, as it's already pre-trained on a large amount of data.

I've come to learn about GPT-2 through Robert Miles AI safety Youtube channel and intend to look into it in more detail.

From my current understanding, GPT-2 is pre-trained to "understand" "natural" language (for any definition of the words in quotes). However you would want it to not only understand general text but generate text similar to some specific "genre", e.g. scientific articles, youtube comments, twitter messages, you name it.

So using its pre-trained understanding, it analyzes the structure of sample texts and replicates this structure.
For scientific articles this structure could be:

Abstract
Context of research topic
Introduction of researchers
Explanation of methods/experiments/discoveries
Results and interpretation
Future research and application

For Youtube comments the structure is probably more chaotic but could include a vague reference to former comments, insults, nonsensical bar-grade philosophy, internet slang and smileys.

TL;DR: The domain specific text is only used to tell GPT-2 what you're looking for. You basically hand it context to work with, instead of prompting "Say something clever" (my least favorite line at parties, when I've been introduced as clever).

P.S.: Take this with a grain of salt. It's 90% conjecture from incomplete information.


Does "transition model" alone in an MDP imply it's non-deterministic? I am looking at a lecture on POMDP, and the context is that, when the quadcopter can't see the landmarks, it has to use reckoning. And then he mentions the transition model is not deterministic, hence the uncertainty grows.
Can transition models in MDP be deterministic?

Yes, in which case, it will be more like a search problem, if it is not POMDP and have finite number of states. Or you can use the same framework (used for POMDP) with constrained (deterministic) transition matrix modeling for model-based systems.
If you think about it after you train any model/agent with an MDP modeling, during test time the optimal strategy is generally deterministic, i.e., given a feature/state you will take a particular action even if it has more than one non-zero element in each row of the transition matrix.


How does vanish gradient restrict RNN to not work for long range dependencies? I am really trying to understand deep learning models like RNN, LSTMs etc. I have gone through many tutorials of RNN and have learned that RNN cannot work for long Range dependencies, like:
Consider trying to predict the last word in the text “I grew up in France… I speak fluent French.” Recent information suggests that the next word is probably the name of a language, but if we want to narrow down which language, we need the context of France, from further back. It’s entirely possible for the gap between the relevant information and the point where it is needed to become very large. Unfortunately, as that gap grows, RNNs become unable to learn to connect the information.
it comes due to vanish gradient problem. However, I could not understand that how to vanish gradient creates an issue for RNN to not work for long-range dependencies.
Since, as I know that vanish gradient usually comes when we have many hidden layers and the gradient for the first layer usually produced too low and that affects the training process. However, everyone connects this issue with vanish gradient, technically what is the relationship RNN (long-range dependencies) with vanish gradient?
I am really sorry if it is a weird question

Vanishing gradient is: as the gradient starts to flow from the end of the network (right side of the network) to the start of the network (left side of the network), it will be multiplied by numbers less than 1 and gradually it will become weaker and weaker and when it arrives to the first layers, it's so weak that makes almost no change in initial layers parameters.
Now in case of RNNs you can unroll the network, and now you can see that it is like a deep network. For clarity look at the image below (taken from a course by Andrew NG and edited):

Red arrows show the way of gradient backpropagation and you can see that in each step they are multiplied by a number (actually a matrix is multiplied in another matrix). If this number is less than one, then it results in vanishing gradient. If this number is greater than one it will result in exposure (which can be controlled by simply clipping it to a max value). The higher the steps, the more of vanishing or exploding effect.
But they have found a solution for this problem, LSTM or GRU. What these units do is that they will make a highway for gradient to backpropagate through (in each state it will be multiplied by 1). So gradient can travel for longer distance (from French into France). Even though, the problem still exists for very long relations.
If you know what is a ResNet and how it works, you can find the same concept behind LSTM or GRU and ResNets as both make a highway for gradient to flow back.
You can have an intuition of how LSTM or GRU works by following the forward pass and assuming what they are doing as locking the concepts in memory cells. Like when the network sees the word France it will understand that it's an important word and maybe it will become handy later. So it will put the France word in one of it's memory cells and keeps it there for several steps and when it wants to guess the language it will use this memory cell to predict French not English or Persian. Then it can release that memory cell and use this cell for something else.
If you want to learn more and get better intuition, I highly recommend you to look at this link: CS231n of MIT - lecture 10


How TensorFlow will know if the prediction is true or false? I'm completely new at ML, but really interested. To be honest, read many articles about it, but still don't understand the workings of it.

I just started to understand this example: https://storage.googleapis.com/tfjs-examples/mnist/dist/index.html

My thinking about it is that TF has some resources, some examples of how numbers look like, and try to match them with the ones in the test. I saw that sometimes the test changes a right prediction to a wrong, but makes better and better predictions. But how? I think that the program doesn't know the right predictions (and this way it won't know the wrong ones). In the training how it makes better predictions? Test by test, from what exceptions it will change it's predictions? What happens in a new test?

During the training process, the machine learning program is shown example inputs (in this case pictures of digits) and also what the answer should be for that input. That is how it learns the correct answer.
However, it does not learn an exact mapping between the input data and required output, instead it learns a series of patterns that generally map the input to the required value. The advantage of this approach is that it can perform the same mapping on new inputs that were not included in the training data but are similar (but not necessarily identical) to examples that it has already seen.


How to use DQN when the action space can be different at different time steps? I would like to employ DQN to solve a constrained MDP problem. The problem has constraints on action space. At different time steps till the end, the available actions are different. It has different possibilities as below.

0, 1, 2, 3, 4
0, 2, 3, 4
0, 3, 4
0, 4

Does this mean I need to learn 4 different Q networks for these possibilities? Also, correct me if I am wrong, it looks like if I specify the action size is 3, then it automatically assumes the actions are 0, 1, 2, but, in my case, it should be 0, 3, 4. How shall I implement this?

There are two relevant neural network designs for DQN:

Model q function directly $Q(s,a): \mathcal{S} \times \mathcal{A} \rightarrow \mathbb{R}$, so neural network has concatenated input of state and action, and outputs a single real value. This is arguably the more natural fit to Q learning, but can be inefficient.

Model all q values for given state $Q(s,\cdot): \mathcal{S} \rightarrow \mathbb{R}^{|\mathcal{A}|}$, so neural network takes input of current state and outputs all action values related to that state as a vector.

For the first architecture, you can decide which actions to evaluate by how you construct the minibatch. You pre-filter to the allowed actions for each state.
For the second architecture, you must post-filter the action values to those allowed by the state.
There are other possibilities for constructing variable-length inputs and outputs to neural networks - e.g. using RNNs. However, these are normally not worth the extra effort. A pre- or post- filter on the actions for a NN that can process the whole action space (including impossible actions) is all you usually need. Don't worry that the neural network may calculate some non-needed or nonsense values.


What is the advantage of using cross entropy loss & softmax? I am trying to do the standard MNIST dataset image recognition test with a standard feed forward NN, but my network failed pretty badly. Now I have debugged it quite a lot and found & fixed some errors, but I had a few more ideas. For one, I am using the sigmoid activation function and MSE as an error function, but the internet suggests that I should rather use softmax for the output layer, and cross entropy loss as an error function. Now I get that softmax is a nice activation function for this task, because you can treat the output as a propability vector. But, while being a nice thing to have, that's more of a convinience thing, isn't it? Easier to visualize?
But when I looked at what the derivative of softmax & CEL combined is (my plan was to compute that in one step and then treat the activation function of the last layer as linear, as not to apply the softmax derivative again), I found:
$\frac{δE}{δi}$ = $t$ − $o$
(With $i$ being the input of the last layer, $t$ the one hot target vector and $o$ the prediction vector).
That is the same as the MSE derivative. So what benefits does softmax + CEL actually have when propagating, if the gradients produced by them are exactly the same?

Short answer: larger gradients
That is not the derivative of the softmax function. $t - o$ is the combined derivative of the softmax function and cross entropy loss. Cross entropy loss is used to simplify the derivative of the softmax function. In the end, you do end up with a different gradients. It would be like if you ignored the sigmoid derivative when using MSE loss and the outputs are different. Using softmax and cross entropy loss has different uses and benefits compared to using sigmoid and MSE. It will help prevent gradient vanishing because the derivative of the sigmoid function only has a large value in a very small space of it. It is similar to using a different cross entropy loss where the combined derivative of the loss and sigmoid is $t - o$.
Information on derivatives of cross entropy with sigmoid function and with softmax function. I would also suggest some more research on cross entropy loss functions beyond my links.


How should I define the action space for a card game like Magic: The Gathering? I'm trying to learn about reinforcement learning techniques. I have little background in machine learning from university, but never more than using a CNN on the MNIST database.
My first project was to use reinforcement learning on tic-tac-toe and that went well.
In the process, I thought about creating an AI that can play a card game like Magic: The Gathering, Yu-gi-oh, etc. However, I need to think of a way to define an action space. Not only are there thousands of combinations of cards possible in a single deck, but we also have to worry about the various types of decks the machine is playing and playing against.
Although I know this is probably way too advanced for a beginner, I find attempting a project like this, challenging and stimulating. So, I looked into several different approaches for defining an action space. But I don't think this example falls into a continuous action space, or one in which I could remove actions when they are not relevant.
I found this post on this stack exchange that seems to be asking the same question. However, the answer I found didn't seem to solve any of my problems.
Wouldn't defining the action space as another level of game states just mask the exact same problem?
My main question boils down to:
Is there an easy/preferred way to make an action space for a game as complex as Magic? Or, is there another technique (other than RL) that I have yet to see that is better used here?

There are several different ways you can model the state and action spaces in such sequential (extensive-form) environments/games. For environments with small action spaces or those typically introduced to beginning-RL students, the state space and action space remains constant along an agent's trajectory (termed normal form games when there are multiple agents). In sequential games which can be illustrated as trees, a "state" is analogous to "information set" which is defined as the sequence (tuple) of actions and observations since the beginning of the game's episode. Terminal states (leaf nodes) exist, and the action space $\mathcal{A}[x]$ at an information set $x$ can be defined as a union of action sequences that can be taken to each terminal state, not counting terminal states that cannot be reached from the current information set.
In the above examples, I discussed games such as the examples you stated where more than one agent can interact with the environment, but this is a generalization over RL and can be applied to when only one agent is maximizing its reward.


How to design a NLP algorithm to find a food item in menu card list? I am new to NLP and AI in general. I am just expecting springboard information so that I can skip all the introduction to NLP websites.  I have just started studying NLP and want to know how to go about solving this problem. I am creating a chatbot that will take voice input from customers ordering food at restaurants. The customer input I am expecting as;

I want to order Chicken Biryani

Can I have a Veg Pizza, please

Coca-cola  etc

I want to write an algorithm that can separate the name of the food item from the user input and compare it with the list of food items in my menu card and come up with the right item.
I am new to NLP, I am studying it online for this particular project, I can do the required coding, I just need help with the overall algo or sort of flow chart. It will save my time tremendously. Thanks.

Since you want a shortcut use the spoonacular API.  Below is a test with your words.  You can see it had trouble with 'Coca' and 'veg'.

What you need is 'named-entity recognition' for food.  This is not a new thing but clearly not a solved problem.  The Foodie Favorites repository attempts to solve the problem from scratch.
If want to do some research and dig deeper see FoodBase corpus: a new resource of annotated food entities.  From the abstract:

It consists of 12,844 food entity annotations describing 2105 unique food entities. Additionally, we provided a weakly annotated corpus on an additional 21,790 recipes. It consists of 274,053 food entity annotations, 13,079 of which are unique.