How do we prove the n-step return error reduction property? In section 7.1 (about the n-step bootstrapping) of the book Reinforcement Learning: An Introduction (2nd edition), by Andrew Barto and Richard S. Sutton, the authors write about what they call the "n-step return error reduction property":

But they don't prove it. I was thinking it should not be too hard but how can we show this? I was thinking of using the definition of n-step return (eq. 7.1 on previous page):

$$G_{t:t+n} = R_{t+1} + \gamma*R_{t+2} + ... + \gamma^{n-1}*R_{t+n} + \gamma^{n}*V_{t+n-1}(S_{t+n})$$

Because then this has the $V_{t+n-1}$ in it already. But in the definition above of the n-step return it uses $V_{t+n-1}(S_{t+n})$ but on the right side of the inequality (7.3) that we want to prove it is just little s $V_{t+n-1}(s)$ ? So kind of confused here which state s it is using? And then I guess after this probably pull out a $\gamma^{n}$ term or something, how should we go from here?

This is the newest Sutton Barto book (book page 144, equation 7.3):
https://drive.google.com/file/d/1opPSz5AZ_kVa1uWOdOiveNiBFiEOHjkG/view

Let's start by looking at:

$$\max_s \Bigl\lvert \mathbb{E}_{\pi} \left[ G_{t:t+n} \mid S_t = s \right] - v_{\pi}(s) \Bigr\rvert.$$

We can rewrite this by plugging in the definition of $G_{t:t+n}$:

\begin{aligned}
& \max_s \Bigl\lvert \mathbb{E}_{\pi} \left[ G_{t:t+n} \mid S_t = s \right] - v_{\pi}(s) \Bigr\rvert \\
%
=& \max_s \Bigl\lvert \mathbb{E}_{\pi} \left[ R_{t + 1} + \gamma R_{t + 2} + \dots + \gamma^{n - 1} R_{t + n} + \gamma^n V_{t + n - 1}(S_{t + n}) \mid S_t = s \right] - v_{\pi}(s) \Bigr\rvert \\
%
=& \max_s \Bigl\lvert \mathbb{E}_{\pi} \left[ R_{t:t+n} + \gamma^n V_{t + n - 1}(S_{t + n}) \mid S_t = s \right] - v_{\pi}(s) \Bigr\rvert,
\end{aligned}

where $R_{t:t+n} \doteq R_{t + 1} + \gamma R_{t + 2} + \dots + \gamma^{n - 1} R_{t + n}$.

If you go all the way back to page 58 of the book, you can see the definition of $v_{\pi}(s)$:

\begin{aligned}
v_{\pi}(s) &\doteq \mathbb{E}_{\pi} \left[ \sum_{k = 0}^{\infty} \gamma^k R_{t + k + 1} \mid S_t = s \right] \\
%
&= \mathbb{E}_{\pi} \left[ R_{t:t+n} + \gamma^n \sum_{k = 0}^{\infty} \gamma^k R_{t + n + k + 1} \mid S_t = s \right] \\
%
&= \mathbb{E}_{\pi} \left[ R_{t:t+n} \mid S_t = s \right] + \gamma^n \mathbb{E}_{\pi} \left[ \sum_{k = 0}^{\infty} \gamma^k R_{t + n + k + 1} \mid S_t = s \right]
\end{aligned}

Using this, we can continue rewriting where we left off above:

\begin{aligned}
& \max_s \Bigl\lvert \mathbb{E}_{\pi} \left[ R_{t:t+n} + \gamma^n V_{t + n - 1}(S_{t + n}) \mid S_t = s \right] - v_{\pi}(s) \Bigr\rvert \\
%
=& \max_s \Bigl\lvert \mathbb{E}_{\pi} \left[ \gamma^n V_{t + n - 1}(S_{t + n}) \mid S_t = s \right] - \gamma^n \mathbb{E}_{\pi} \left[ \sum_{k = 0}^{\infty} \gamma^k R_{t + n + k + 1} \mid S_t = s \right] \Bigr\rvert \\
%
=& \gamma^n \max_s \Bigl\lvert \mathbb{E}_{\pi} \left[ V_{t + n - 1}(S_{t + n}) - \sum_{k = 0}^{\infty} \gamma^k R_{t + n + k + 1} \mid S_t = s \right] \Bigr\rvert
\end{aligned}

Because the absolute value function is convex, we can use Jensen's inequality to show that the absolute value of an expectation is less than or equal to the expectation of the corresponding absolute value:

$$\left| \mathbb{E} \left[ X \right] \right| \leq \mathbb{E} \left[ \left| X \right| \right].$$

This means that:

\begin{aligned}
\max_s \Bigl\lvert \mathbb{E}_{\pi} \left[ G_{t:t+n} - v_{\pi}(s) \mid S_t = s \right] \Bigr\rvert &\leq \gamma^n \max_s \mathbb{E}_{\pi} \left[ \Bigl\lvert V_{t + n - 1}(S_{t + n}) - \sum_{k = 0}^{\infty} \gamma^k R_{t + n + k + 1} \Bigr\rvert \mid S_t = s \right] \\
%
\end{aligned}

Now, the important trick here is to see that:

$$\max_s \mathbb{E}_{\pi} \left[ \Bigl\lvert V_{t + n - 1}(S_{t + n}) - \sum_{k = 0}^{\infty} \gamma^k R_{t + n + k + 1} \Bigr\rvert \mid S_t = s \right] \leq \max_s \mathbb{E}_{\pi} \left[ \Bigl\lvert V_{t + n - 1}(S_{t}) - \sum_{k = 0}^{\infty} \gamma^k R_{t + k + 1} \Bigr\rvert \mid S_t = s \right]$$

I'm skipping the formal steps to show that this is the case to save space, but the intuition is that:

The left-hand side of this inequality involves finding an $S_t = s$ such that some function of $S_{t + n}$ is maximized, whereas the right-hand side involves finding an $S_t = s$ such that exactly the same function of $S_{t}$ is maximized.
In the left-hand side, selecting an $S_t = s$ implicitly induces a probability distribution over multiple possible states $S_{t + n}$, given by $S_t$, the environment's transition dynamics, and the policy $\pi$. Intuitively, this is more "restrictive" for the $\max$ operator, it does not have the "freedom" to directly select a single state $S_{t + n}$ such that the function of $S_{t + n}$ is maximized. The right-hand side is free to choose any single state $S_t = s$ such that $S_t$ in the right-hand side were equal to an "optimal" $S_{t+n}$ on the left-hand side, but it is also free to make even better choices which might never be uniquely reachable after $n$ steps on the left-hand side.

We can use this to rewrite the previous inequality we had (where we might be making the right-hand side a bit bigger than it was, but that's fine, it already was an upper bound anyway so that inequality will still hold):

\begin{aligned}
\max_s \Bigl\lvert \mathbb{E}_{\pi} \left[ G_{t:t+n} - v_{\pi}(s) \mid S_t = s \right] \Bigr\rvert &\leq \gamma^n \max_s \mathbb{E}_{\pi} \left[ \Bigl\lvert V_{t + n - 1}(S_{t}) - \sum_{k = 0}^{\infty} \gamma^k R_{t + k + 1} \Bigr\rvert \mid S_t = s \right] \\
%
&= \gamma^n \max_s \mathbb{E}_{\pi} \left[ \Bigl\lvert V_{t + n - 1}(S_{t}) - v_{\pi}(s) \Bigr\rvert \mid S_t = s \right].
\end{aligned}

After this rewriting we've got a hidden $\mathbb{E}_{\pi}$ "inside" another $\mathbb{E}_{\pi}$ (because the definition of $v_{\pi}(s)$ contains an $\mathbb{E}_{\pi}$), which I suppose is kind of ugly... but mathematically meaningless.

The maximum of a random variable is an upper bound on the expectation of that random variable, so we can get rid of the expectation in the right-hand side (again potentially increasing the right-hand side, which again is still fine since it's already an upper bound anyway):

\begin{aligned}
\max_s \Bigl\lvert \mathbb{E}_{\pi} \left[ G_{t:t+n} - v_{\pi}(s) \mid S_t = s \right] \Bigr\rvert &\leq \gamma^n \max_s \Bigl\lvert V_{t + n - 1}(s) - v_{\pi}(s) \Bigr\rvert,
\end{aligned}

which we can finally rewrite to Equation (7.3) in the book by moving the subtraction of $v_{\pi}(s)$ outside of the expectation on the left-hand side of the inequality (which is fine because, as I already mentioned above, the definition of $v_{\pi}(s)$ itself contains another $\mathbb{E}_{\pi}$ anyway).


Is a switch from R to Python worth it? I just finished a 1-year Data Science master's program where we were taught R. I found that Python is more popular and has a larger community in AI.
What are the advantages that Python may have over R in terms of features applicable to the field of Data Science and AI (other than popularity and larger community)? What positions in Data Science and AI would be more Python-heavy than R-heavy (especially comparing industry, academic, and government job positions)? In short, is Python worthwhile in all job situations or can I get by with only R in some positions?

I want to reframe your question.

Don't think about switching, think about adding.

In data science you'll be able to go very far with either python or r but you'll go farthest with both.

Python and r integrate very well, thanks to the reticulate package. I often tidy data in r because it is easier for me, train a model in python to benefit from superior speed and visualize the outcomes in r in beautiful ggplot all in one notebook!

If you already know r there is no sense in abandoning it, use it where sensible and easy to you. But it is 100% a good idea to add python for many uses.

Once you feel comfortable in both you'll have  a workflow that fits you best dominated by your favorite language.


What are the pros and cons of using a spatial transformation network to predict the next video frame? I've read through a few papers on next frame prediction from a sequence of frames and several of them use spatial transformations (STNs). See this as an example. I want to know what are the pros and cons of using an STN to predict the next frame. Are there any assumptions that must be made about the data besides "Consecutive frames are all approximately affine transformations of each other"?

Using an STN alone to predict the next frame assumes that there is some linear translation between the current frame and the next frame. In some domains this is true but usually there are more complicated transitions from frame to frame (Eg. An occlusion, a new entity, light difference, etc). So, although STNs may be useful to resizing and translating inputs for a CNN, they should be used together with other techniques when predicting a new frame of a sequence.


In RL as probabilistic inference, why do we take a probability to be $\exp(r(s_t, a_t))$? In section 2 the paper Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Review the author is discussing formulating the RL problem as a probabilistic graphical model. They introduce a binary optimality variable $\mathcal{O}_t$ which denotes whether time step $t$ was optimal (1 if so, 0 otherwise). They then define the probability that this random variable equals 1 to be
$$\mathbb{P}(\mathcal{O}_t = 1 | s_t, a_t) = \exp(r(s_t, a_t)) \; .$$
My question is why do they do this? In the paper they make no assumptions about the value of the rewards (e.g. bounding it to be non-positive) so in theory the rewards can take any value and thus the RHS can be larger than 1. This is obviously invalid for a probability. It would make sense if there was some normalising constant, or if the author said that the probability is proportional to this, but they don't.
I have searched online and nobody seems to have asked this question which makes me feel like I am missing something quite obvious so I would appreciate if somebody could clear this up for me please.

After doing some further reading, it turns out that negative rewards are an assumption for this distribution to hold. However, the author notes that as long as you don't receive a reward of infinity for any action then it is possible to re-scale your rewards by subtracting the maximum value of your potential rewards so that they are always negative.


What is the basic purpose of local search methods? I read about the hill climbing algorithms, the simulating annealing algorithm, but I am confused. What is the basic purpose of local search methods?

Hill climbing, Simulated annealing, genetic algorithms are different variants of local search algorithm. Hill Climbing use to find good solution for NP-hard problems. Simulated annealing was first used extensively to solve VLSI layout problems. It has been applied widely to factory scheduling and other large-scale optimization tasks. Genetic algorithm have had a wide-spread impact on optimization problems, such as circuit layout and job-shop scheduling.


What if there's a game where all the AI people actually lived? Imagine the fictitious scenario in a role playing game (RPG) where the non-playing characters (NPCs) within the RPG are conscious of their own surrounding and consider the developer to be god.  The people inside can also live a normal life in that they can create new life, eat, die, and perform other functions humans experience.

Will this qualify them to be an AI?

Will the children of the NPCs qualify as an AI?

Might this be realized in the future?

Yes, a baby can be considered an AI. Will this be our future? I don't know. That is exactly what some people are looking for, to create an AI that can live. We have several AIs that each time more surprises us. But none of them question their own existence, none of them wants to know or require attention from their god (developer) because of their reflections. I believe that what we create today can be equated with a mosquito of our real life: a brainless "being" that performs simple actions.

The cool thing about believing in such a future with Artificial Intelligence is that we do not have to repeat our mistakes. We do not need an AI evolving according to age, it can be born with a superintelligence, knowing everything, learning only new things that are not foreseen.

Some humans (and I) have a passion for the human, for every detail and imperfection, and dream of re-creating something as mysterious and complex as through an AI. It seems something impossible, some believe that it would be impossible only because who created us would be a perfect being (god). But others (me) do not stop at this issue, and continue the quest to create an AI with some characteristics of the human, such as: feelings, reflecting on one's own acts, ...

But all this, as far as I know, is surreal and only projection of our mind. It is already something interesting, for creating new challenges for humanity.


What is the difference between a receptive field and a feature map? In a CNN, the receptive field is the portion of the image used to compute the filter's output. But one filter's output (which is also called a "feature map") is the next filter's input.

What's the difference between a receptive field and a feature map?

I hope this illustration will help you:

Receptive field(s): is a small portion of the input to produce only one node in a feature map.

Feature map(s): is a convolutional process output, a feature map can be said as a feature representation of filter's input. One feature map consists of many filter's outputs (from different receptive fields) from one kernel. The number of feature maps depends on the number of the kernel.

So even feature maps are the next filter's input, but the next receptive fields is not a feature map. The next receptive field consists of a small portion node from different feature maps (not only one feature map).

And also, we can see from the illustration above, a feature map has two-dimensional size $(46 \times 46)$, then a receptive field size will always three-dimensional $(5 \times 5 \times \text{Number Of Feature Maps})$.


Can the (sparse) categorical cross-entropy be greater than one? I am using AlexNet CNN to classify my dataset which contains 10 classes and 1000 data for each class, with 60-30-10, splits for train, validation, and test. I used different batch sizes, learning rates, activation functions, and initializers. I'm using the sparse categorical cross-entropy loss function.
However, while training, my loss value is greater than one (almost equal to 1.2) in the first epoch, but until epoch 5 it comes near 0.8.
Is it normal? If not, how can I solve this?

Both the sparse categorical cross-entropy (SCE) and the categorical cross-entropy (CCE) can be greater than $1$. By the way, they are the same exact loss function: the only difference is really the implementation, where the SCE assumes that the labels (or classes) are given as integers, while the CCE assumes that the labels are given as one-hot vectors.
Here is the explanation with examples.
Let $(x, y) \in D$ be an input-output pair from the labelled dataset $D$, where $x$ is the input and $y$ is the ground-truth class/label for $x$, which is an integer between $0$ and $C-1$. Let's suppose that your neural network $f$ produces a probability vector $f(x) = \hat{y} \in [0, 1]^C$ (e.g. with a softmax), where $\hat{y}_i \in [0, 1]$ is the $i$th element of $\hat{y}$.
The formula for SCE is (which is consistent with the TensorFlow implementation of this loss function)
$$
\text{SCE}(y, \hat{y}) = - \ln (\hat{y}_{y}) \label{1}\tag{1},
$$
where $\hat{y}_{y}$ is the $y$th element of the output probability vector $\hat{y}$ that corresponds to the probability that $x$ belongs to class $y$, according to $f$.
Actually, the equation \ref{1} is also the definition of the CCE with one-hot vectors as targets (which behave as indicator functions). The only difference between CCE and SSE is really just the representation of the targets, which can slightly change the implementation under the hood. Moreover, note that this is the definition of the CE for only $1$ training pair. If you have multiple pairs, you have to compute the CE for all pairs, then average these CEs (for a reference, see equation 4.108, section 4.3.4 Multiclass logistic regression of Bishop's book PRML).
Let's have a look at a concrete example with concrete numbers. Let $C=5$, $y = 3$, $\hat{y} = [0.2, 0.2, 0.1, 0.4, 0.1]$, then the SCE is
\begin{align}
\text{SCE}(y, \hat{y})
&=
- \ln (0.4) \approx 0.92,
\end{align}
If $\hat{y} = [0.2, 0.2, 0.2, 0.1, 0.3]$, so $\hat{y}_{y} = 0.1$, and we still have $y = 3$, then the CCE is $2.3 > 1$.
You can execute this Python code to check yourself.
import numpy as np
import tensorflow as tf # Install TensorFlow 2.3!

y = 1
y_true = [3] # sparse label (integer)
y_true2 = [0, 0, 0, 1, 0] # one-hot vector

for y_y in [0.4, 0.1]:
    sce_np = -(y * np.log(y_y))
    print("SCE (NumPy) =", sce_np)

y_preds = [[0.2, 0.2, 0.1, 0.4, 0.1],
           [0.2, 0.2, 0.2, 0.1, 0.3]]
for y_pred in y_preds:
    sce_tf = tf.keras.losses.sparse_categorical_crossentropy(y_true, y_pred)
    cce_tf = tf.keras.losses.categorical_crossentropy(y_true2, y_pred)

    print("SCE (TensorFlow) =", sce_tf)
    print("CCE (TensorFlow) =", cce_tf)

To answer the following question more directly.

However, while training, my loss value is greater than one (almost equal to 1.2) in the first epoch, but until epoch 5 it comes near 0.8. Is it normal? If not, how can I solve this?

Yes. It can happen, as explained above. (However, this does not mean that you do not have mistakes in your code.)


Why is the Backpropagation algorithm used to train the Multilayer Perceptron? I've read on the book NND by Martin Hagan et al (chapter 11), that to train the feed-forward neural network: Multilayer Perceptron one uses the Backpropagation algorithm.
Why this algorithm?
Could someone please explain in simple terms and detailed terms?
Thank you in advance.

According to wikipedia of backpropagation:

In fitting a neural network, backpropagation computes the gradient of the loss function during supervised learning with respect to the weights of the network for a single input–output example, and does so efficiently, unlike a naive direct computation of the gradient with respect to each weight individually.

Backpropagation is a special case of reverse accumulation of automatic differentiation, and it was announced in Rumelhart, Hinton & Williams (1986). Automatic differentiation has 2 modes: forward accumulation and backward accumulation.

Automatic differentiation is distinct from symbolic differentiation and numerical differentiation (the method of finite differences). Symbolic differentiation can lead to inefficient code and faces the difficulty of converting a computer program into a single expression, while numerical differentiation can introduce round-off errors in the discretization process and cancellation. Both classical methods have problems with calculating higher derivatives, where complexity and errors increase. Finally, both classical methods are slow at computing partial derivatives of a function with respect to many inputs, as is needed for gradient-based optimization algorithms. Automatic differentiation solves all of these problems.

Usually, two distinct modes of AD are presented, forward accumulation (or forward mode) and reverse accumulation (or reverse mode). Forward accumulation specifies that one traverses the chain rule from inside to outside, while reverse accumulation has the traversal from outside to inside...

Forward accumulation is more efficient than reverse accumulation for functions $ f:ℝ^n → ℝ^m$ with $m ≫ n$ as only $n$ sweeps are necessary, compared to $m$ sweeps for reverse accumulation.

Reverse accumulation is more efficient than forward accumulation for functions $ f:ℝ^n → ℝ^m$ with $m ≪ n$ as only $m$ sweeps are necessary, compared to $n$ sweeps for forward accumulation.

Under this historical computational differentiation techniques context, since the loss function in a MLP maps $ℝ^n \rightarrow ℝ^m$ with $m ≪ n$ for most cases, that's one of the main reasons why backpropagation is most often used to train a MLP as a special case of reverse accumulation introduced above.


When we have multiple traces, do we average over traces or the total number of times we have visited that state? I am confused about the workings of the first- and every-visit MC.
My first question is, when we have multiple traces, do we average over traces or the total number of times we have visited that state?
So, in the example:
$$\tau_1 = \text{House} +3, \text{House} +2, \text{School} -4, \text{House} +4, \text{School} -3, \text{Holidays}$$
$$\tau_2 = \text{House} -2, \text{House} +3, \text{School} -3, \text{Holidays},$$
where we have states of either House, Holidays, or School, with the numerical values being the immediate rewards.
For every-visit MC to find the state value of HOUSE, with $\gamma$=1, my intuition would be to create a return list, R, that looks like the following
$$R_1=[3+2−4+4−3, 2−4+4−3, 4−3]= [2, −1, 1]$$
$$R_2=[−2+3−3, 3−3]=[−2, 0]$$
$$R_1+R_2=[2,−1, 1,−2, 0]$$
which, when averaged over 5 visits, is 0 and the correct answer, but I would like if you could confirm if the methodology is correct?
However, another approach would be to compute the average returns for each trace. Which is correct?

For every visit MC you create a list for each state. Every time you enter a state you calculate the returns for the episode and append these returns to a list. Once you have done this for all the episode you want to average over you simply calculate the value of a state to be the average of this list of returns for the state.
First visit MC is almost the same except that you only append the returns to the state returns list for the first time you visit the state in an episode.
Your workings are correct, you average over the number of times you have visited the state (in every visit MC). So, in your example, you would get the value of HOUSE to be 0, as you stated. If you were doing first visit MC then the returns for episode 1 would be 2 and the returns for episode 2 would be -2, which you would then average over the two first visits to again give you 0 (this would not always be the case that they are equal after 2 episodes but in the limit both methods do converge to the true state value function).


What weights should I use while back-propagating? I've started to learn about neural networks recently and I can't find the answer to this question.

Let's assume there's a neural network (fig. 1)

So if the loss function is:

and the derivative  is:

if I want to use this to find  what k and l (well there's only one neuron with index l here, but what if there would be more?) should i use in  and ?

I've also found "other" way of backpropagating it's described here, but I can't understand how they came up with that method from the original equation w -= step * dE/dw.

Sorry if I failed to explain my problem. If something isn't clear please ask in comments.

First I will assume you notate $y$ as the models output and $z$ as the ground-truth. Second, I am assuming this is a linear model (No activation functions). Then the gradient math goes as so:

$
\begin{align*}
\frac{dE}{dw_{ij}^1} &= \frac{dE}{dy}\frac{dy}{dw_{ij}^1} \\
&= \frac{dE}{dy}\sum_k\frac{\partial y}{\partial n_{k}^3}\frac{dn_{k}^3}{dw_{ij}^1} \\
&= \frac{dE}{dy}\sum_k\frac{\partial y}{\partial n_{k}^3} \frac{\partial n_{k}^3}{ \partial n_{j}^2} \frac{dn_{j}^2}{ dw_{ij}^1}  \\
&= -2*(z-y)\sum_kw_{kl}^3w_{jk}^2 x_i  \\
\end{align*}
$

So the reason you are having trouble to figure out which $k$ index to use, the answer is because you need to use both and sum over them. The $l$ index is just the only $l$ index that exists because you only have one node in that layer.


How to add a pretrained model to my layers to get embeddings? I want to use a pretrained model found in [BERT Embeddings] https://github.com/UKPLab/sentence-transformers and I want to add a layer to get the sentence embeddings from the model and pass on to the next layer. How do I approach this?

The inputs would be an array of documents and each document containing an array of sentences.

The input to the model itself is a list of sentences where it will return a list of embeddings.

This is what I've tried but couldn't solve the errors:

def get_embeddings(input_data):

    input_embed = []
    for doc in input_data:
      doc = tf.unstack(doc)
      doc_arr = asarray(doc)
      doc = [el.decode('UTF-8') for el in doc_arr]
      doc = list(doc)
      assert(type(doc)== list)

      new_doc = []
      for sent in doc:
        sent = tf.unstack(sent)
        new_doc.append(str(sent))
        assert(type(sent)== str)

      embedding= model.encode(new_doc)  # Accepts lists of strings to return BERT sentence embeddings
      input_embed.append(np.array(embedding))

    return tf.convert_to_tensor(input_embed, dtype=float)

sentences = tf.keras.layers.Input(shape=(3,5)) #test shape
sent_embed = tf.keras.layers.Lambda(get_embeddings)

x = sent_embed(sentences)

I think you should use Keras embedding layer. It will be too easier than what you are doing.

Steps

Create Embedding Matrix
add matrix to embedding layer while building model.

You will find detailed article

https://www.cs.uaf.edu/2011/spring/cs641/lecture/04_05_modeling.html


What is the difference between the notations $\|x\|_1, \|x\|_2$ and $|x|$? What is the difference between the notations $\|x\|_1, \|x\|_2$ and $|x|$? I think $|x|$ is the magnitude of $x$.

$\|x\| = |x|$ denotes the absolute value norm, which is a special case of the $L_1$ norm defined on the 1-D vector spaces formed by real or complex numbers.

$\|\textbf{x}\|_1 = \sum_{i=1}^n|x_i|$ denotes the Taxicab / Manhattan norm, relating to how a Taxi would drive along a rectangular grid of roads to reach a point $(x, y)$ from $(0,0)$.

$\|\textbf{x}\|_2 = \sqrt{x_1^2 + \dots + x_n^2}$ denotes the Euclidean norm on an N-D Euclidean space, which is a result of the Pythagorean theorem (the shortest distance between two points).


Why is it easier to construct adversarial examples relative to training neural networks? I was having looking at this lecture by Ian Goodfellow and my doubt is around 18:00 timestamp where he explains generation of adversarial examples using FGSM.
He mentions that the there is a linear relationship between the input to the model and the output as the activation functions are piece-wise linear with a small number of pieces. I'm not very clear what he means by input and output. Is he referring to inputs and outputs of a single layer or the input image and final output?
He states that the relation between the parameters (weights) of a model and the output are non-linear which is what makes it difficult to train a neural network, thus it is much easier to find an adversarial example.
Could someone explain what is linear in what? and how linearity helps in adversarial example construction?
EDIT: As per my understanding FGSM method relies on the linearity of the loss function with respect to the input image. It constructs an adversarial example by perturbing the input in the direction of the gradient of the loss function w.r.t image. I am not able to understand why this works?

input to output is linear refers to the input X i.e image and output is the output logits/softmax from the network.
So how does linearity help in constructing adversarial examples? Imagine a simple logistic regressor and a simple 2D space. There is a definite boundary beyond which the label that the model(i.e logisitc regressor in this case) changes. So if we move perpendicular to the boundary (i.e the line represented by the model in this case) we can get to another class's space. So if we perturb the input in this direction, the model outputs wrong class. { Refer the slide with the title Adversarial Examples from Excessive Linearity for diagram }
Now imagine the neural network trained on imagenet, there are so many boundaries and a small change can just shift change the class the model would predict. Now it is important to note that these subspaces of the image remain nearly the same if we train VGG or ResNet etc. So this explains how an adversarial example on one network effects another.
You may ask how such small change can effect. this is because the vectors we deal it is not 2d or 3d, it is very large and small changes add up.


Can you let specific data impact a neural network more than other data? I have a lot of empty values in my dataset, so I want to let my neural network 'learn more' on the rows that have no empty values because these rows are of higher importance.

Is there a way to do this?

Yes, you can weight the loss function for each example, so that instead of your cost function being

$$J = \sum_i \mathcal{L}(y_i, \hat{y}_i)$$

It will be

$$J = \sum_i w_i\mathcal{L}(y_i, \hat{y}_i)$$

Where $i$ iterates over your data set, $\mathcal{L}$ is the loss function you are using, $y_i$ is ground truth for each example and $\hat{y}_i$ is prediction for each example.

You can set your relevance weighting according to any criteria you like based on the example, and your dataset/goals. So for instance you could set it to $1.0$ for complete examples and $0.1$ for incomplete examples. Depending on your NN framework, it may already offer per-example weighting, and even if it does not, auto-differentiation means that typically you only need to implement the forward logic into the cost function for each minibatch, and the weighting will be applied correctly to gradients with no more work required. If you do need to implement gradient calculations yourself, you simply multiply the initial gradient of each example by the example's weight $w_i$.

Once you make a change like this, you need to take care on how you set up and interpret your test set. When your model is used in production, if there are still inputs with missing details, you probably don't want to weight the metrics (e.g. accuracy rating) in the same way, but to report it as-is against a correctly sampled dataset of unseen examples from the population of expected inputs.


How could artificial intelligence harm us? We often hear that artificial intelligence may harm or even kill humans, so it might prove dangerous.

How could artificial intelligence harm us?

tl;dr

There are many valid reasons why people might fear (or better be concerned about) AI, not all involve robots and apocalyptic scenarios.

To better illustrate these concerns, I'll try to split them into three categories.

Conscious AI

This is the type of AI that your question is referring to. A super-intelligent conscious AI that will destroy/enslave humanity. This is mostly brought to us by science-fiction. Some notable Hollywood examples are "The terminator", "The Matrix", "Age of Ultron". The most influential novels were written by Isaac Asimov and are referred to as the "Robot series" (which includes "I, robot", which was also adapted as a movie).

The basic premise under most of these works are that AI will evolve to a point where it becomes conscious and will surpass humans in intelligence. While Hollywood movies mainly focus on the robots and the battle between them and humans, not enough emphasis is given to the actual AI (i.e. the "brain" controlling them). As a side note, because of the narrative, this AI is usually portrayed as supercomputer controlling everything (so that the protagonists have a specific target). Not enough exploration has been made on "ambiguous intelligence" (which I think is more realistic).

In the real world, AI is focused on solving specific tasks! An AI agent that is capable of solving problems from different domains (e.g. understanding speech and processing images and driving and ... - like humans are) is referred to as General Artificial Intelligence and is required for AI being able to "think" and become conscious.

Realistically, we are a loooooooong way from General Artificial Intelligence! That being said there is no evidence on why this can't be achieved in the future. So currently, even if we are still in the infancy of AI, we have no reason to believe that AI won't evolve to a point where it is more intelligent than humans.

Using AI with malicious intent

Even though an AI conquering the world is a long way from happening there are several reasons to be concerned with AI today, that don't involve robots!
The second category I want to focus a bit more on is several malicious uses of today's AI.

I'll focus only on AI applications that are available today. Some examples of AI that can be used for malicious intent:

DeepFake: a technique for imposing someones face on an image a video of another person. This has gained popularity recently with celebrity porn and can be used to generate fake news and hoaxes. Sources: 1, 2, 3
With the use of mass surveillance systems and facial recognition software capable of recognizing millions of faces per second, AI can be used for mass surveillance. Even though when we think of mass surveillance we think of China, many western cities like London, Atlanta and Berlin are among the most-surveilled cities in the world. China has taken things a step further by adopting the social credit system, an evaluation system for civilians which seems to be taken straight out of the pages of George Orwell's 1984.
Influencing people through social media. Aside from recognizing user's tastes with the goal of targeted marketing and add placements (a common practice by many internet companies), AI can be used malisciously to influence people's voting (among other things). Sources: 1, 2, 3.
Hacking.
Military applications, e.g. drone attacks, missile targeting systems.

Adverse effects of AI

This category is pretty subjective, but the development of AI might carry some adverse side-effects. The distinction between this category and the previous is that these effects, while harmful, aren't done intentionally; rather they occur with the development of AI. Some examples are:

Jobs becoming redundant. As AI becomes better, many jobs will be replaced by AI. Unfortunately there are not many things that can be done about this, as most technological developments have this side-effect (e.g. agricultural machinery caused many farmers to lose their jobs, automation replaced many factory workers, computers did the same).
Reinforcing the bias in our data. This is a very interesting category, as AI (and especially Neural Networks) are only as good as the data they are trained on and have a tendency of perpetuating and even enhancing different forms of social biases, already existing in the data. There are many examples of networks exhibiting racist and sexist behavior. Sources: 1, 2, 3, 4.


Is there a neural network that accepts both the current input and previous output? I am quite new to neural networks. I am trying to implement in Python a neural network having only one hidden layer with $N$ neurons and $1$ output layer.
The point is that I am analyzing time series and would like to use the output layer as the input of the next unit: by feeding the network with the input at time $t-1$ I obtain the output $O_{t-1}$ and, in the next step, I would like to use both the input at time $t$ and $O_{t-1}$, introducing a sort of auto-regression. I read that recurrent neural network are suitable to address this issue.
Anyway I cannot imagine how to implement a network in Keras that involves multilayer recurrence: all the references I found are linked to using the output of a layer as input of the same layer in the next step. Instead, I would like to include the output of the last layer (the output layer) in the inputs of the first hidden layer.

You want to look at recurrent neural networks.


What exactly do people mean when they say that the computer "learns" to do something? I'm watching a youtube video where a guy is talking about how computers can learn to go from some point A to some point B.

However, the way he does it is very disappointing: all he does is generate thousands of objects that go from A to some random destination, and then he updates all the random destinations by taking the average of the original destination + the destination of the object that got closest to B. This process is then repeated until convergence.

I don't see what's so impressive about this. If you wanted the machine to go from A to B, why did you not just ... you know .... tell it to go to B?. What's the point of essentially generating random instructions, and hoping one of those random instructions just happens to be: go to B, when you could just as well tell that instruction to the computer yourself?

Is this really what people mean when they say that a computer "learns" to do some task? And if so, what's so impressive about it? What am I missing?

Your confusion is a result of Moravec’s Paradox; the just tell it to... step is enormously more difficult than it sounds. So what researchers do is attempt to find a general approach to that problem. The problem is expressed as a matrix of numbers (a grid that may be a million on each side these days) and “learning” is the process of computing the values for each of those numbers. Obviously no human can solve an equation like that, so a machine solver is used. Therefore the work of the AI researcher is to find better ways to generate these large matrices i.e. to express go to X and to create better solvers. The holy grail is a matrix and a solver that will work for any X.


What are some examples of Classical AI? I believe that Classical AI uses deductive thought processes. For example, given as a set of constraints, deduce a conclusion.  What are some examples of successfully applying Classical AI to real world problems.

The term classical AI refers to the concept of intelligence that was broadly accepted after the Dartmouth Conference and basically refers to a kind of intelligence that is strongly symbolic and oriented to logic and language processing. One basic point is the duality body vs. mind. It's in this period that the mind starts to be compared with computer software.
Two classical historical examples of this conception of intelligence

Deep Blue, whose aim in life was to be the master of chess, ruling over the (not-so) intelligent mankind

Eliza a computer-based therapist that turned out to trigger a critic to the classical AI

Two technical examples of classical AI

Expert systems, which are computer programs that strongly rely on the type of constraints and conclusions that you refer to, in order to accomplish feats of apparent intelligence

Fuzzy logic, which is an extension of multivalued logic, but with continuous values instead of discrete ones

Note that in all cases the hardware (once compared with the body) does not play any role: Intelligence is abstract and independent from the material world.


Occupation detection using Face API For my university project, I am planning to build a face recognition/ occupation recognition programme. However, rather than using the existing Haar cascade(for age and gender) I am planning to use Face API which seems far more accurate than the former. My question is is it possible to somehow combine my trained data for Haar cascade(for occupation) with Face API since Face API doesn't have the option to recognize occupation(such as students/office workers from their appearance)?

In short, it's possible to achieve occupation detection via face but it's highly doubtful that it can be done with Face API that you mentioned.

Now some details,

Face preception is a very complex job. You have so many parameters to consider before even beginning to design the model.

  Many different types of important information are visible in faces and
  the processes and mechanisms involved in extracting this information
  are complex and can be highly specialized.

and

  Face perception serves as the basis for much of human social exchange.
  Diverse information can be extracted about an individual from a single
  glance at their face, including their identity, emotional state, and
  direction of attention.

Extracting that much information is not easy.

Face API:

Face API gives you

Face Similarity (checks how similar two or more faces are)
Face Identification (gives you 27 landmarks like age, gender, facial hair, smile etc confidence level)
Emotion detection (check what kind of emotion the face has)

However, in my view, occupation detection via face will going to need a lot more than these basic parameters.

However, previous work utilizing single person’s nearly frontal clothing information and fore/background context preliminarily proves that occupation recognition is computationally feasible in computer vision.

However, in practice, multiple people with arbitrary poses are common in a photo, and recognizing their occupations is even more challenging.

You have to appropriately built visual attributes, co-occurrence, and spatial configuration model that is learned through structure SVM or deep neural nets to recognize multiple people’s occupations in a photo simultaneously.

I am attaching some research papers so you can review them to get more insight.

Reference Papers:

A Comparative View of Face Perception

The many faces of research on face perception

What Do You Do? Occupation Recognition in a Photo via Social Context

Note:

Please state your question clearly (like: tell us your problem in detail, and your approach to solve it), and also tell what research you have done before asking the question. Please see how much research effort is expected of stackoverflow users before they post any new questions.


What are general ideas behind OpenCog? OpenCog is an open source AGI-project co-founded by the mercurial AI researcher Ben Goertzel. Ben Goertzel writes a lot of stuff, some of it really whacky. Nonetheless, he is clearly very intelligent and has thought deeply about AI for many decades.

What are the general ideas behind OpenCog? Would you endorse it as a insightful take on AGI?

I'm especially interested in whether the general framework still makes sense in the light of recent advances.

What is OpenCog?

OpenCog is a project with the vision of creating a thinking machine with human-level intelligence and beyond.

In OpenCog's introduction, Goertzel categorically states that the OpenCog project is not concerned with building more accurate classification algorithms, computer vision systems or better language processing systems. The OpenCog project is solely focused on general intelligence that is capable of being extended to more and more general tasks.

Knowledge representation

OpenCog's knowledge representation mechanisms are all based fundamentally on networks. OpenCog has the following knowledge representation components:

AtomSpace: it is a knowledge representation database and query engine. Data on AtomSpace is represented in the form of graphs and hypergraphs.

Probabilistic Logic Networks (PLN's): it is a novel conceptual, mathematical and computational approach to handle uncertainty and carry out effective reasoning in real-world circumstances.

MOSES (Meta-Optimizing Semantic Evolutionary Search): it implements program learning by using a meta-optimization algorithm. That is, it uses two optimization algorithms, one wrapped inside the other to find solutions.

Economic Attention Allocation (EAA): each atom has an attention value attached to it. The attention values are updated by using nonlinear dynamic equations to calculate the Short Term Importance (STI) and Long Term Importance (LTI).

Competency Goals

OpenCog lists 14 competencies that they believe AI systems should display in order to be considered an AGI system.

Perception: vision, hearing, touch and cross-modal proprioception

Actuation: physical skills, tool use, and navigation physical skills

Memory: declarative, behavioral and episodic

Learning: imitation, reinforcement, interactive verbal instruction, written media and learning via experimentation

Reasoning: deduction, induction, abduction, causal reasoning, physical reasoning and associational reasoning

Planning: tactical, strategic, physical and social

Attention: visual attention, behavioural attention, social attention

Motivation: subgoal creation, affect-based motivation, control of emotions

Emotion: expressing emotion, understanding emotion

Modelling self and other: self-awareness, theory of mind, self-control

Social interaction: appropriate social behavior, social communication, social inference and group play

Communication: gestural communication, verbal communication, pictorial communication, language acquisition and cross-modal communication

Quantitative skills: counting, arithmetic, comparison and measurement.

Ability to build/create: physical, conceptual, verbal and social.

Do I endorse OpenCog?

In my opinion, OpenCog introduces and covers important algorithms/approaches in machine learning, i.e. hyper-graphs and probabilistic logic networks. However, my criticism is that they fail to commit to a single architecture and integrate numerous architectures in an irregular and unsystematic manner.

Furthermore, Goertzel failed to recognize the fundamental shift that came with the introduction of deep learning architectures so as to revise his work accordingly. This puts his research out of touch with recent developments in machine learning


Which work originally introduced gradient clipping? The Deep Learning book mentions that it's been used for years but the oldest sources it mentions are from 2012:

  A simple type of solution has been in use by practitioners for many years: clipping the gradient. There are diﬀerent instances of this idea (Mikolov, 2012; Pascanu et al., 2013). One option is to clip the parameter gradient from a mini-batch element-wise (Mikolov, 2012), just before the parameter update. Another is to clip the $||g||$ of the gradient $g$ (Pascanu et al., 2013) just before the parameter update

But I find it hard to believe that the first uses and mentions of gradient clipping are from 2012. Does anyone know the origins of the solution?

Short Answer

Tomas Mikolov's mention of gradient clipping in a  single paragraph of his PhD thesis in 2012 is the first appearance in the literature.

Long Answer

The first source (Mikolov, 2012) in the Deep Learning book is Mikolov's PhD thesis and can be found here. The end of section 3.2.2 is where gradient clipping is discussed, only it's called turncating.

  ... The exploding gradient problem has
  been described in [4].

  A simple solution to the exploding gradient problem is to truncate values of the gradients. In my experiments, I did limit maximum size of gradients of errors that get accumulated in the hidden neurons to be in a range < −15; 15 >. This greatly increases stability
  of the training, and otherwise it would not be possible to train RNN LMs successfully on
  large data sets.
  ...
  [4] Y. Bengio, P. Simard, P. Frasconi. Learning Long-Term Dependencies with Gradient Descent is Difficult. IEEE Transactions on Neural Networks, 5, 157-166, 1994.

A search of the referenced paper [4] shows that it does describe the problem as Mikolov said, but it does not present gradient clipping as a solution.

So I had a look at the second source Deep Learning mentioned: On the difficulty of training Recurrent Neural Networks. It directly cites Mikolov as having proposed clipping:

  We would make a final note about the approach proposed by Tomas Mikolov in his PhD thesis (Mikolov, 2012) (and implicitly used in the state of the art results on language modelling (Mikolov et al., 2011)). It involves clipping the gradient’s temporal components element-wise (clipping an entry when it exceeds in absolute value a fixed threshold). Clipping has been shown to do well in practice and it forms the backbone of our approach.

I thought about emailing Mikolov to verify that his thesis was the origin of the idea. But then I noticed that he is a co-author of this paper which cites him as proposing it! Though I still wonder if it was commonly used in practice before
even though it had not been published.


What does learning mean? Can someone explain what is the process of learning? What does it mean to learn something?

That is a wonderfully fundamental question.

  Learning is the use of a system to change another system so that, instead of doing what it did before, which may have been nothing, it does something else.

In the human brain, the system is the way that genetic expression caused the directed mutability of that brain so that human intentions and responses to external stimuli would alter. The direction of alteration is based on incentives and deterrents. In classes of children, there are things a teacher may do or a culture in the school that motivate learning. Misbehavior is deterred through other mechanisms. In that case, the first system is the educational process and the second system is the brain's ability to use what it was taught.

Even without classes, children learn things that create comfort, so comfort is the internal incentive. It is wired into us as a kind of teacher to dislike groin moisture, so we learn to use bathroom receptacles. It is wired into us to like praise from parents and teachers, so we tend to perform to get it. Things like that are complex losses and rewards and primary systems to incentivize learning.

Learning in artificial networks is not nearly as complex as in human brains. In some cases the artificial results are better. In other cases the abilities of the human brain cannot be approached by artificial networks yet.

The functioning of an artificial network often begins arbitrary and completely useless, but it's a parametric function, meaning the function can be changed by changing numbers called parameters. Each time the function is used to process an example drawn from a data set, the result is evaluated and the evaluation is used to modify the parameters. Care is taken to not over-modify the parameters, otherwise confusion can occur. Mathematicians call this type of system confusion chaos.

Repeating this carefully directed and moderated process ideally leads to something called convergence. The learning system is set up so that the result of convergence is a set of parameters that minimize losses, maximize gains, or both.

Sometimes initial learning is not enough and there are other related things to do later.

Adjusting learned behavior to adapt to new conditions or information
Unlearning things that no longer produce benefit so that new learning can replace them
Relearning things that had faded from memory
Developing greater confidence in what was learned so that unlearning requires greater impetus

There are additional technical terms for the above concepts, more details that can be grasped, many categories of learning system types, varieties within those, and the mathematics that was used as a foundation to construct all this and make it work. Because the question was fundamental, those details and technicalities were omitted.


Applications of Information Theory in Machine Learning How is information theory applied to machine learning, and in particular to deep learning, in practice? I'm more interested in concepts that yielded concrete innovations in ML, rather than theoretical constructions.
Note that, I'm aware that basic concepts such as entropy is used for training decision trees, and so on. I'm looking for applications which use slightly more advanced concepts from information theory, whatever they are.

Apart from the entropy and the cross-entropy, which are widely used in deep learning and you seem to be aware of, there is also the Kullback-Leibler divergence (also known as relative entropy), which is widely used in the context of variational Bayesian neural networks and variational auto-encoders, given that it's often part of the loss function that is minimized, i.e. the Evidence Lower BOund, which is a proxy objective function for the KL divergence between the prior and posterior distributions (which actually corresponds to the minimum description length needed to encode the data: huh?). See this answer for more details. There is also the mutual information, which has also been used as a measure of uncertainty in the context of Bayesian neural networks.


Predict value from image set I have a large dataset of skin images, each one associated with a hydration value (percentage).

Now I'm looking into predicting the hydration value from an image. My thinking: train a CNN on the dataset and evaluate the model with a mean square error regression.

First, does this sound like a sensible way to try this?

Second, I'd like to run the model on mobile. Can you recommend any examples with Caffe2 (or alternatively TensorFlow) or diagrams that might explain a similar task?

Your initial idea seems about right. Before creating your own classifier you might want to try transfer learning, using some pretrained network like VGG16 that is incorporated in most of machine learning frameworks.

As to inference on mobile devices, TensorFlow offers some tutorials in this subject:
https://www.tensorflow.org/mobile/tflite/


Can an Markov decision process be dependent on the past? As far as I know MDP are independent from the past. But the definition says that the same policy should always take the same action depending on the state.

What if I define my state as the current "main" state + previous decisions?

For Example in Poker the "main" state would be my cards and the pot + all previous information about the game.

Would this still be a MDP or not?

It's not totally clear from your description, but it sounds like you may be onto something like an Additive Markov Chain.


Is the 'direction' considered, when determining the branching factor in bidirectional search? If I am correct, the branching factor is the maximum number of successors of any node.
When I am applying bidirectional search to a transition graph like this one below

If 11 is the goal state and I start going backwards, is 10 considered as a successor of 5? Even if it does not leads me further to my start state 1?

If I am correct, the branching factor is the maximum number of successors of any node

You are correct, they should also be the immediate ones:

If 11 is the goal state and I start going backwards, is 10 considered as successor of 5? Even if it do not leads me further to my start state 1?

No, there is also a bit of misunderstanding of bidirectional search: In bidirectional search you run 2 simultaneous searches, one forward from the initial state, and another one backwards from the goal(hoping they meet in the middle and save you steps), if actions are reversible ( going from node to node), the successor nodes become predecessors in one search and vice versa, and your goal becomes your initial state, in your case:

Reference/source
Artificial Intelligence: A Modern Approach, by S. Russell and P. Norvig.


Why do current models use multiple normalization layers? In most current models, the normalization layer is applied after each convolution layer. Many models use the block $\text{convolution} \rightarrow \text{batch normalization} \rightarrow \text{ReLU}$ repeatedly. But why do we need multiple batch normalization layers? If we have a convolution layer that receives a normalized input, shouldn't it spit out a normalized output? Isn't it enough to place normalization layers only at the beginning of the model?

One issue is that a normalized set of initial weights may not stay normalized as learning progresses; so given that we adjust weights proportionately according to their relative values and also when working on a subset of the learning data the model may become convinced that one subset of features is important and others not, this can result in the weights becoming unbalanced again. In the learning curve we may see this as a plateau, where the learning becomes convinced that a few features are much more important than they really are and fails to find new features that can contribute even more because tiny proportionate changes did not move them far, or quickly, enough into a noticeable range.

So when we re-sample from the database to get the next batch we need to be fully open to the possibility that the previous batch learning was too heavily biased towards its own favoured set. In effect we are trading accuracy for openness to new feature combinations, and at the same time assisting generalization.


What does the number of required expert demonstrations in Imitation Learning depend on? I just read the following points about the number of required expert demonstrations in imitation learning, and I'd like some clarifications. For the purpose of context, I'll be using a linear reward function throughout this post (i.e. the reward can be expressed as a weighted sum of the components of a state's feature vector)

The number of expert demonstrations required scales with the number of features in the reward function.

I don't think this is obvious at all - why is it true? Intuitively, I think that as the number of features rises, the complexity of the problem does too, so we may need more data to make a better estimate of the expert's reward function. Is there more to it?

The number of expert demonstration required does not depend on -

Complexity of the expert’s optimal policy $\pi^{*}$
Size of the state space

I don't see how the complexity of the expert's optimal policy plays a role here - which is probably why it doesn't affect the number of expert demonstrations we need; but how do we quantify the complexity of a policy in the first place?
Also, I think that the number of expert demonstrations should depend on the size of the state space. For example, if the train and test distributions don't match, we can't do behavioral cloning without falling into problems, in which case we use the DAGGER algorithm to repeatedly query the expert and make better decisions (take better actions). I feel that a larger state space means that we'll have to query the expert more frequently, i.e. to figure out the expert's optimal action in several states.
I'd love to know everyone's thoughts on this - the dependence of the number of expert demonstrations on the above, and if any, other factors. Thank you!

Source: Slide 20/75

The answer to your question can be found in the original paper that introduced the max-margin and projection imitation learning (IL) algorithms: Apprenticeship Learning via Inverse Reinforcement Learning (by Abbel and Ng, 2004, ICML). Specifically, theorem 1 (section 4, page 4) states

Let an $\text{MDP} \setminus R$, features $ \phi : S \rightarrow [0, 1]^k$, and any $\epsilon > 0$ be given. Then the apprenticeship learning algorithm (both max-margin and projection versions) will terminate with $t^{(i)} \leq \epsilon$ after at most
$$n=O\left(\frac{k}{(1-\gamma)^{2} \epsilon^{2}} \log \frac{k}{(1-\gamma) \epsilon}\right)$$
iterations.

Here $k$ is the dimension of the feature vectors, so it's clear that the number of iterations needed for these algorithms to terminate scales with $k$. The proof of this theorem can be found in appendix A of the same paper (and all other terms are defined in the paper, which you should read to understand all the details). Of course, this result holds (only) for these specific IL algorithms (which are the algorithms the author of your slides, Abbel, is referring to). See also theorem 2 and the experiments section (in particular, figure 4, which shows the performance as a function of the number of trajectories) of the same paper.
These slides provide a nice overview of the contents of this paper, so I suggest that you read them too.


What are some programming-oriented resources for reinforcement learning? I have been reading :
Reinforcement Learning: An Introduction by Sutton and Barto.
I admit it's a good read for learning RL whereas it's more theoretical with detailed algorithms.
Now, I want something more programming oriented resource(s) maybe a course, book, etc.
I have been exploring Kaggle, Open-source RL projects.
I need this to learn and grasp a deeper understanding of RL from the perspective of a developer i.e optimized way of writing code, explanation about using the latest RL libraries, cloud services, etc.
Thank you in advance.

Arthur Juliani has some interesting Medium articles on reinforcement learning with TensorFlow backed up with code on GitHub.

Part 0 — Q-Learning Agents
Part 1 — Two-Armed Bandit
Part 1.5 — Contextual Bandits
Part 2 — Policy-Based Agents
Part 3 — Model-Based RL
Part 4 — Deep Q-Networks and Beyond
Part 5 — Visualizing an Agent’s Thoughts and Actions
Part 6 — Partial Observability and Deep Recurrent Q-Networks
Part 7 — Action-Selection Strategies for Exploration
Part 8 — Asynchronous Actor-Critic Agents (A3C)

As nbro pointed out Denny Britz has a good repository: https://github.com/dennybritz/reinforcement-learning
As you have seen with Sutton & Barto's book the code is mostly in Lisp.  Shangtong Zhang has replicated the code in Python: https://github.com/ShangtongZhang/reinforcement-learning-an-introduction
Sudharsan and Ravichandiran wrote a book "Hands-On Reinforcement Leraning with Python" which uses OpenAI Gym and TensorFlow.  You can find more information on the book along with their code repository at Hands-On Reinforcement Learning with Python


Why are neural networks always trained "by themselves"? In the current rush of artificial intelligence research, fueled by NN, independent of the paper I choose, the NN are always trained by themselves. Sure, there are architectures that combine CNN and RNN or LSTMs in a way that can help to solve multiple problems interacting (like labeling images with human readable text snippets), but independent of this, they always learn by themselves.

Supervised networks just get a bunch of examples to learn from.
Reinforcement algorithms run around virtual spaces falling over hundreds of times before learning to walk properly.

This might sound silly but no one helps them and no one plays with them. Children, researchers, puppies, dolphins, ... all intelligent beings we are aware of interact with each other. There are many forms of learning, one very important one being social learning which includes observational learning. We imitate, copy and learn from others all the time. There is also a new idea of looking at knowledge that suggests that we all aren't as independent as our culture might have taught us to think we are. That knowledge really lies in the connections between individuals instead of inside of each agent itself, just like the connections of neurons are what makes the brain work, not the neurons themselves.

Couldn't such interaction between learning agents be enormously valuable? If you throw 5 agents in a room (NN a Knowledge Based Agent a simple pre-programmed one etc) and let them learn from each other through observation and inspiration, what effects could one expect?

I am not talking about ensemble learning as this just throws several hypotheses together and takes the average over all results. I am talking about a complex way of agents interacting with each other during learning.

Is there any research on this idea? If so, what were the results? This is the only one I found:

Learning to Communicate with Deep Multi-Agent Reinforcement Learning

The way children learn is in many ways supervised. It is true that certain abilities are there genetically (visual system, object recognition, to large extent voice recognition), but a lot of human experience is gained as a result of response, either from the environment or from the mentor (parent, teacher, etc).

Social interaction is possible only when a human already learned a lot (things we consider trivial, but in fact pretty complex), in this sense this is equivalent to AlphaGo learning to play Go by playing with a copy network. NNs that can only perform object or voice recognition can't interact, they are too primitive for this. And a nearly all of todays AI is on this stage right now.

However, it is true that NN learning is much less efficient than human learning and current models are far from AGI. I think RL is the closest area that studies real multi-agent environments.


How to build my own dataset and model for an LSTM neural network I have a sort of mathematical problem and I'm not sure which model I should choose to make an LSTM neural network.

Currently in my country, there is a system in which certain groups of researchers upload information on products of scientific interest, such as research articles, books, patents, software, among others. Depending on the number of products, the system assigns a classification to each group, which can be A1, A, B and C, where A1 is the highest classification and C is the minimum.

The classification is done through a mathematical model whose entries are, the total number of each product, the total sum of all products, number of authors, among other indices that are calculated with the previous values.

Once the entries are obtained, these values are processed by a set of formulas and the final result is a single number.

This number is located in a range provided by the mathematical model and this is how the group is classified.

What I want to do is given the current classification of a group, give suggestions of different values to improve their classification.

For example, if there is a group with classification C, suggest how many products it should have, how many authors, what value should its indexes have, so that its category would be finally B.

I think the structure of my network should be:
-1 input, which would be the classification you want to get.
-Multiple output, one for each product and indexes.

But I do not understand how to make the network take into account the current classification of the group, in addition to the number of products and the value of the current indexes.

If you have further questions about the problem, please feel free to ask.

I appreciate your suggestions.

why do you want to use LSTM network?
lstm is a variant of a recurrent neural network , recurrent neural networks are used for "sequential" tasks , i.e the dataset should have some sequential structure , like poems , songs etc.

Your model should be a simple classifier , once you fit a simple classifier , like a decision forest on the categories dataset  products ,authors etc , you will have a  model that will predict the class based on these attributes then you can say from the decision boundaries of the model what values you must have . if the relationship between the attributes is even more simpler you could try plotting distribution plots.


Connection between the Bellman equation for the action value function $q_\pi(s,a)$ and expressing $q_\pi(s,a) = q_\pi(s, a,v_\pi(s'))$ When deriving the Bellman equation for $q_\pi(s,a)$, we have
$q_\pi(s,a) = E_\pi[G_t | S_t = s, A_t = a] = E_\pi[R_{t+1} + \gamma G_{t+1} | S_t = s, A_t = a]$ (1)
This is what is confusing me, at this point, for the Bellman equation for $q_\pi(s,a)$, we write $G_{t+1}$ as an expected value, conditioned on $s'$ and $a'$ of the action value function at $s'$, otherwise, there is no recursion with respect to $q_\pi(s,a)$, and therefore no Bellman equation. Namely,
$ = \sum_{a \in A} \pi(a |s) \sum_{s' \in S} \sum_{r \in R} p(s',r|s,a)(r + \gamma E_\pi[G_{t+1}|S_{t+1} = s', A_{t+1} = a'])$ (2)
which introduces the recursion of $q$,
$ = \sum_{a \in A} \pi(a |s) \sum_{s' \in S} \sum_{r \in R} p(s',r|s,a)(r + \gamma q_\pi(s',a'))$ (3)
which should be the Bellman equation for $q_\pi(s,a)$, right?
On the other hand, when connecting $q_\pi(s,a)$ with $v_\pi(s')$, in this answer, I believe this is done
$q_\pi(s,a) = \sum_{a\in A} \pi(a |s) \sum_{s' \in S}\sum_{r \in R} p(s',r|s,a)(r + \gamma E_{\pi}[G_{t+1} | S_{t+1} = s'])$ (4)
$q_\pi(s,a) = \sum_{a\in A} \pi(a |s) \sum_{s' \in S}\sum_{r \in R} p(s',r|s,a)(r + \gamma v_\pi(s'))$ (5)
Is the difference between using the expectation $E_{\pi}[G_{t+1} | S_{t+1} = s', A_{t+1} = a']$ in (3) and the expectation $E_{\pi}[G_{t+1} | S_{t+1} = s']$ in $(4)$ simply the difference in how we choose to express the expected return $G_{t+1}$ at $s'$ in the definition of $q_\pi(s,a)$?
In $3$, we express the total return at $s'$ using the action value function

leading to the recursion and the Bellman equation, and in $4$, the total return is expressed at $s'$ using the value function

leading to $q_\pi(s,a) = q_\pi(s,a,v_\pi(s'))$?

Your understanding of the Bellman equation is not quite right. The state-action value function is defined as the expected (discounted) returns when taking action $a$ in state $s$. Now, in your equation (2) you have conditioned on taking action $a'$ in the inner expectation - this is not what happens in the state-action value function, you do not condition on knowing $A_{t+1}$, it is chosen according to the policy $\pi$ as per the definition of a bellman equation.
If you want to see a 'recursion' between state action value functions, note that
$$v_\pi(s) = \sum_a \pi(a|s)q_\pi(s,a)\;,$$
Your equation (5) is incorrect -- you need to drop the outter sum over $a$ as we have conditioned on knowing $a$. I will drop the $\pi$ subscripts for ease on notation, and we can see a 'recursion' for state-action value functions as:
$$q(s,a) = \sum_{s',r}p(s',r|s,a)\left(r + \gamma \left[\sum_{a'} \pi(a'|s')q(s',a')\right]\right)\;.$$


How do I design the rewards and penalties for an agent whose goal it is to explore a map I am trying to train an agent to explore an unknown two-dimensional map while avoiding circular obstacles (with varying radii). The agent has control over its steering angle and its speed. The steering angle and speed are normalized in a $[-1, 1]$ range, where the sign encodes direction (i.e. a speed of $-1$ means that it is going backwards at the maximum units/second).
I am familiar with similar problems where the agent must navigate to a waypoint, and in which case the reward is the successful arrival to the target position. But, in my case, I can't really reward the agent for that, since there is no direct 'goal'.
What I have tried
The agent is penalised when it hits an obstacle; however, I am not sure how to motivate the agent to move. Initially, I was thinking of having the agent always move forward, meaning that it only has control over the steering angle. But, I want the ability for the agent to control its speed and be able to reverse (since I'm trying to model a car).
What I have tried is to reward the agent for moving and to penalise it for remaining stationary. At every timestep, the agent is rewarded ${1}/{t_\text{max}}$ if the absolute value of the speed is above some epsilon, or penalised that same amount if otherwise. But, as expected, this doesn't work. Rather than motivating the agent to move, it simply causes it to jitter back and forth. This makes sense since 'technically' the most optimal strategy if you want to avoid obstacles is to remain stationary. If the agent can't do that then the next best thing is to make small adjustements in the position.
So my question: how can I add in an exploration incentive to my agent? I am using proximal policy optimization (PPO).

Measure what you want to achieve as directly as possible, and reward that. Later you can add more sophisticated incentives for the type of motion etc, but the key to a good reward signal is that it measures the quality of a solution at a high level, without specifying how to achieve that solution.
If you want your simulated car to explore, you will want to give it a reward signal based on it encountering new unexplored areas. There are lots of reasonable choices here. I suspect a good one will depend on what sensors you can reasonably code for the car, and what you consider to count as exploration - e.g. is it a thorough search of an area, moving far from the original position, experiencing different "views"?
One likely component you will need to give your agent and incorporate into the state representation is a memory. In order to understand whether the agent is exploring, something will need to know whether the agent has experienced something before and how much. A very simple kind of memory would be to add counters to a grid map and allow the agent to know how many time steps it has spent in each position on the map. The reward signal can then be higher when the agent enters a point on the map that it has not been in recently. If you want a non-episodic or repeating tour of exploration you might decay the values over time, so that an area that has not been visited for a long time counts the same as a non-visited one.
A related concept that you might be able to borrow ideas from is curiousity. There have been some interesting attempts to encourage an agent to seek new/interesting states by modifying action selection. Curiosity-driven Exploration by
Self-supervised Prediction is one such attempt, and might be of interest to you. In that case, the authors use an intrinsic model of curiousity that can solve some environments even when there is no external reward signal at all!
Alternatively, if you don't care to get involved in technical solution, you could create a maybe acceptable behaviour for your vehicle by setting a random goal position, then granting a reward and moving it to a new random location each time the car reaches it.


How to solve Snake Game with a Hamiltonian graph algorithm? I wonder if there is a way to solve snake game using Hamiltonian algorithm?
if there is a way

how to apply it?
what data structure will be used with algorithm?
time complexity and space complexity?
is this algorithm an optimal solution or there is a better way?

A Hamiltonian path in a graph is a path that visits all the nodes/vertices exactly once, a hamiltonian cycle is a cyclic path, i.e. all nodes visited once and the start and the endpoint are the same.
If we want to solve the snake game using this, we could divide the playable space in a grid and then try to just keep traversing on a hamiltonian cycle, this means you would eventually get all the rewards and never hit yourself unless you are longer than can fit on the screen.
You can look at the code given in this Github repo.

Note that this method, as you can tell, is not going to be optimal by time, there could be better solutions than this.
You will have to use a graph to store all the nodes and the complexity is supposed to be O(n!).


What is the difference between LSTM and fully connected LSTM? I'm currently trying to understand the difference between a vanilla LSTM and a fully connected LSTM. In a paper I'm reading, the FC-LSTM gets introduced as

  FC-LSTM may be seen as a multivariate version of LSTM where the input, cell output and states are all 1D vectors

But is not really expanded further upon. Google also didn't help me much in that regard as I can't seem to find anything under that keyword.

What is the difference between the two? Also, I'm a bit confused by the quote - aren't inputs, outputs, etc. of a vanilla LSTM already 1D vectors?

Based on the citations in the ConvLSTM paper I have come to the conclusion that they mean the Peephole LSTM when they say fully connected LSTM. In the paper that they have taken the encoder-decoder-predictor model from, where they refer to a fully connected LSTM, a Peephole LSTM is used. Also they take their fully connected LSTM definition from this paper, which again uses the Peephole LSTM.

With that the difference would be the added "peephole connections", that lets the gate layers look at the cell states and access the constant error carousel.


Does using the softmax function in Q learning not defeat the purpose of Q learning? It is my understanding that, in Q-learning, you are trying to mimic the optimal $Q$ function $Q*$, where $Q*$ is a measure of the predicted reward received from taking action $a$ at state $s$ so that the reward is maximised.

I understand for this to be properly calculated, you must explore all possible game states, and as that is obviously intractable, a neural network is used to approximate this function.

In a normal case, the network is updated based on the MSE of the actual reward received and the networks predicted reward. So a simple network that is meant to chose a direction to move would receive a positive gradient for all state predictions for the entire game and do a normal backprop step from there.

However, to me, it makes intuitive sense to have the final layer of the network be a softmax function for some games. This is because in a lot of cases (like Go for example), only one "move" can be chosen per game state, and as such, only one neuron should be active. It also seems to me that would work well with the gradient update, and the network would learn appropriately.

But the big problem here is, this is no longer Q learning. The network no longer predicts the reward for each possible move, it now predicts which move is likely to give the greatest reward.

Am I wrong in my assumptions about Q learning? Is the softmax function used in Q learning at all?

However, to me, it makes intuitive sense to have the final layer of the network be a softmax function for some games. This is because in a lot of cases (like Go for example), only one "move" can be chosen per game state, and as such, only one neuron should be active.

You are describing a network that approximates as policy function, $\pi(a|s)$, for a discrete set of actions.

  It also seems to me that would work well with the gradient update, and the network would learn appropriately.

Yes there are ways to do this, based on the Policy Gradient Theorem. If you read it you will probably discover this is more complex to understand than you first thought, the problem being that the agent is never directly told what the "best" action is in order to simply learn in a supervised manner. Instead, it has to be inferred from rewards observed whilst acting. This is a bit harder to figure out than the Q learning update rules which are just sampling from the Bellman optimality equation.

You can split Reinforcement Learning methods broadly into value-based methods and policy gradient methods. Q learning is a value-based method, whilst REINFORCE is a basic policy gradient method. It is also common to use a value based method within a policy gradient method in order to help estimate likely future return used to drive the polcy gradient updates - this combination is called Actor-Critic where the actor learns a policy function $\pi(a|s)$ and the critic learns a value function e.g. $V(s)$.

  But the big problem here is, this is no longer Q learning. The network no longer predicts the reward for each possible move, it now predicts which move is likely to give the greatest reward.

This is true, but it is not a big problem. The main issue is that policy gradient methods are more complex than value based methods. They may or may not be more effective, it depends on the environment you are tryng to create an optimal agent for.

  Is the softmax function used in Q learning at all?

I cannot think of any non-contrived environment in which this function would be useful for an action value approximation.

However, it is possible to use a variant of softmax to create a behaviour policy for Q learning. This uses a temperature hyperparameter $T$ to weight the Q values, and provide a probability of selecting an action, as follows

$$\pi(a_i|s) = \frac{e^{Q(s,a_i)/T}}{\sum_j e^{Q(s,a_j)/T}}$$

when $T$ is high all the probabilities of actions will be similar, when it is low even a small difference in $Q(s,a_i)$ will make a big difference to probability of selecting action $a_i$. This is quite a nice distribution for exploring whilst avoiding previously bad decisions. It will tend to focus the agent on exploring differences between similarly high rated actions. The main issue with it is that it introduces hyperparameters for deciding starting $T$, ending $T$ and how to move between them.


If features are always positives, why do we use RELU activation functions? When does it happen that a layer (either first or hidden) outputs negative values in order to justify the use of RELU?

As far as I know, features are never negative or converted to negative in any other type of layer.

Is it that we can use the RELU with a different "inflection" point than zero, so we can make the neuron start describing a lineal response just after this "new zero"?

The fact that features are always positive values don't guarantee that outputs of hidden layers are positive too.

Due to multiplication, output of an hidden layer could contain negative values, i.e., a hidden layer can contain weights that have opposites signs as its input.
Remember that only layer outputs, not their weights, are passed through ReLu, so, weights of a model could contain negative values.


Are the dialogs at Sophia's (the robot) appearings scripted? I talk about the robot from: Hanson Robotics, which was granted the right to citizenship from Saudi Arabia.

I have found the following articles:

Your new friend is a humanoid robot

Source: theaustralian.com.au

  Like Amazon Echo, Google Assistant and Siri, Sophia can ask and answer questions about discrete pieces of information, such as what types of movies and songs she likes, the weather and whether robots should exterminate humans.

  But her general knowledge is behind these players and she doesn’t do maths. Her answers are mostly scripted and, it seems, from my observation, her answer are derived from algorithmically crunching the language you use.

  Sometimes answers are close to the topic of the question, but off beam. Sometimes she just changes the subject and asks you a question instead.

  She has no artificial notion of self. She can’t say where she was yesterday, whether she remembers you from before, and doesn’t seem to amass data of past interactions with you that can form the basis of an ongoing association.

  Questions such as: “What have you seen in Australia?”, “Where were you yesterday?”, “Who did you meet last week?” and “Do you like Australia?” are beyond her.

Why Sophia the robot is not what it seems

Source: smh.com.au

  You can often fool this sort of software by introducing noise. That could be literal noise – machines aren't great at filtering out background noise, as anyone with a hearing aid will tell you – or it could be noise in the sense of irrelevant information or limited context. You could ask "what do you think of humans?" and then follow up with "can you tell more about it?" The second question requires the robot to define "it", remember what it said last time, and come up with something new.

  In the case of the ABC interview, the questions were sent to Sophia's team ahead of time so they were possibly pre-scripted. Just like an interview with a human celebrity!

Pretending to give a robot citizenship helps no one

Source: theverge.com

  Sophia is essentially a cleverly built puppet designed to exploit our cultural expectations of what a robot looks and sounds like. It can hold a stilted conversation, yes, but its one-liners seem to be prewritten responses to key words. (As Piers Morgan commented during an interview with Sophia, “Obviously these are programmed answers.”)

Here are a few more references.

Inside the mechanical brain of the world’s first robot citizen
Sophia, the Uncanny Robot AI, Is Not What You Think.
How much of Robot Sophia’s speech is likely scripted with your understanding of the progress in NLP?

Sophia uses ChatScript.

You can read about what ChatScript can do here.

  ChatScript keeps track of conversations with each user; can record
  where it is in a conversational flow and what facts it has learned
  about a user (you have to tell it what facts to try to learn). You can
  optionally keep logs of the conversations (either on a ChatScript
  server or locally on a freestanding device).


What are some suitable positive functions as activations of neural networks? I am working on a deep Q-learning project. My project is different than normal deep Q-learning. The rewards of my neural network must be positive because I need their values to importance sample actions. I know that I can't use ReLU as the activation function of my neural network. So the only suitable functions which I know are sigmoid, softmax and exponential function. I tried working with sigmoid and softmax but they generate wrong results and the loss function diverges.
There are two terminal states in my model. Their rewards are 1 and 0. All other states don't have any immediate rewards.

First of all: an activation function is usually placed after a linear operation and you can have a lot of them (maybe different) in your nn. That's why it would be better to say $\bf a$ activation function of the neural net and not $\bf the$ activation function. So if you meant by activation function the last operation which is going to make your outputs non-negative, then you're right, ReLU isn't a good choice. Usually when people need to output some positive values, they take exponent as the last operation. I used it several times and everything was just fine.


Compressing text using AI by sending only prediction rank of next word Is there any effort made to compress text (and maybe other media) using prediction of next word and thus sending only the order number of the word/token which will be predicted on the client side
i.e
Server text: This is an example of a long text example, custom word flerfom inserted to confuse, that may appear on somewhere
Compressed Text transmitted : This [choice no 3] [choice no 4] [choice no 1] [choice no 6] [choice no 1] [choice no 3] [choice no 1], custom word flerfom [choice no 4] inserted [choice no 4] confuse [choice no 5] [choice no 4] [choice no 6] [choice no 5] on somewhere
(Note: of course [choice no 3] will be shortened to [3] to save bytes and also maybe we can do much better in some cases by sending the first letter of the word)
of course it means that the client side neural network has to be static or only updated in a predictable fasion, so the server knows for sure that the client neural network's predictions will follow the given choice orders. I tried example with https://demo.allennlp.org/next-token-lm, but the prediction is not that good. maybe gpt-3 can do better . but its too heavy for use in a normal pc / mobile device
In more details, the process is
Deploy the same model on both sides
Predict the next word after the starting word
Keep the prediction limit say 100
For any word which have more than 2 characters we do the prediction
If the current word is predicted within the top 100 predictions of the model , we can essentially replace it with a numeric char between 0-99 (inclusive) so we are replacing a say , 5 character word with a 2 character numerical char..
And if the word is not predicted in top 100 we send the word as it is..
As much better the model predicts, that much better the compression
And under no scenario it will work worse than the existing method..

If you have a fixed predictor, then yes. If the predictor is not fixed but deterministic, the feasibility depends on the effort needed to update the predictor, and ensuring that messages include a time stamp to ensure the correct version of the predictor is used for compression and inflation.
You get a really nice property if the prediction order is in order of probability that the word occurs, and if you use fewer bits for lower numbers. You would get something that is pretty close to Shannon coding, which is not optimal but is still valid.


What is the $\ell_{2, 1}$ norm? I'm reading this paper and it says:

  In this paper, we present a multi-class embedded feature selection method called as sparse optimal scoring with adjustment (SOSA), which is capable of addressing the data heterogeneity issue. We propose to perform feature selection on the adjusted data obtained by estimating and removing the unknown data heterogeneity from original data. Our feature selection is formulated as a sparse optimal scoring problem by imposing $\ell_{2, 1}$-norm regularization on the coefficient matrix which hence can be solved effectively by proximal gradient algorithm. This allows our method can well handle the multi-class feature selection and classification simultaneously for heterogenous data

What is the $\ell_{2, 1}$ norm regularization? Is it L1 regularization or L2 regularization?

$l_{2,1}$ is a matrix norm, as stated in this paper. For a certain matrix $A \in \mathbb{R}^{r\times c}$, we have  $$||A||_{2,1} = \sum_{i=1}^r \sqrt{\sum_{j=1}^c A_{ij}^2}$$
You first apply $l_2$ norm along the columns to obtain a vector with r dimensions. Then, you apply $l_1$ norm to that vector to obtain a real number. You can generalize this notation to every norm $l_{p,q}$.


Is data leakage relevant when scaling across samples? I have a question about data leakage when pre-processing data for a neural network and whether data leakage actually applies in my instance.
I have variance stabilising transformed genomic data. Because it is genomic data we know apriori that lower numbers translate to lower levels of a gene being made and vice versa. Before input into the neural network, the data are squashed to between 0 and 1 using sklearn:
preprocessing.minmax_scale(data, feature_range=(0,1), axis=1)

The min_max scaling needs to be done across sample (axis=1) as opposed to features because of this apriori assumption of gene levels - low genes need to remain low and vice-versa...
Because of this, my question is: do training samples still need to be scaled separately from test samples as it doesn't seem there is a risk of data leakage here? Is this the correct assumption to make?

Considering that you are making a minmax scaling, the only time in which there would be no risk of data leakage is if the minimum value on your training set equals the minimum value of the test set, and your maximum value on the training set equals the maximum value on the test set.
In that circumstance the result of your scaling would be exactly the same as fitting the scaler to the training set, and applying it to the test set.
There is basically no advantage in scaling them separately, and you take the risk of data leakage if their value ranges differ. I don't know much about genomic data, but even if your assumption was correct I would recommend not to do that.


How is the percentage or the probablity calculated using Loss function in Facenet Model? This question is related to What is the formula used to calculate the accuracy in the FaceNet model? . I know how loss is calculated in the FaceNet model , but how the loss function is used to calculate probability that this unknown person is , say Bob (0.70). Also  we don't know which is positive or negative image , we only know the Anchor (so how FaceNet finds which image is positive or negative ?) . How probability is calculated in FaceNet Model using triplet loss ?

Can we know what is the exact formula or CNN is like black box which uses some unknown method to calculate probability  ?

The facenet model is just the head of the model. The architecture is similar to the enocdr part of an autoencoder, but it uses supervised learning instead of unsupervised learning. The network is called a siamese network The triplet loss helps make the embeddings more representative of the input image/person, with the embedding distance being as large as possible for difference people and vice versa.

However the embeddings is just the representation of the people. It doesn't contain information directly mapping to what person it is. The classification head is used after the Face Net feature extraction top is used.

One-shot learning method for person classfication

This method saves the calculated embeddings of people in a database. The face recognition system first calculates the embeddings of the new, unknown image to be classified. The system loop through the embeddings database and calculates the Euclidean distance of the unknown embeddings and the embeddings in the database.

After computing, it chooses the smallest distance of all and compares it to a threshold set by you. If the distance is larger than the threshold, the classified class is unknown , or else the resulting class will be the embeddings which results to the lowest distance.

Example code from the deeplearning.ai coursera course (great course on AI btw):
Example code of one shot face recognition

Advantages and disadvantages of the system

The system have advantages ranging from no training needed and can classify unknown class. However, once the number of people in the database becomes huge, the system will run slower. It's performance may also differ depending on the quality of the reference image.

SVM (Support Vector Machine) method

This is the method used in the github repository of your previous posts. It uses a one layer nn classifying head to output the classified class.

Advantages and disadvantages of the system

The system have the flexibility of adapting to multiple photo environment and no affecting of performance due to reference images. This method also works well for larger database of people. Triplet loss can also be removed using this method. You can train the network directly from propagating the loss with the probability predicted. However, This method requires re-training when a new person is added to the system. Multiple images of a person is also needed.


Simulated Annealing: Why is e-function used as propability function to decide to accept a worse solution Why is the e-function used to decide whether to accept a worse solution or not?
To be more specific: Why was $e$ chosen as basis?

The propability to accept a worse solution is described with:
$p=e^{-\frac{E(y)-E(x)}{kT}}$

$E(y)$ is the energy from the old solution
$E(x)$ is the energy from new solution $T$ is a constant temprature decreasing with a constant factor k in every iteration.

You can find the explanation by asking some question about the function. Suppose, the value of $\frac{E(y)-E(x)}{kT} >> 0$ is much more greater than zero. What does it mean? It means the value of $E(y)$ is much greater than $E(x)$ related to the $kT$ that is as a measure of temperature decreasing. Now, you want in this situation a probability which is near to zero. Hence, $e^{-\frac{E(y)-E(x)}{kT}}$ could be a good value for the probability of selection of worse solutions!

Why $e$ instead of $2$ or other values greater than $1$? Because it could be a good function in optimization problems as its derivative is more simple than others!


Multi class text classification with imbalanced data I am dealing with intent classification task on an Italian customer service data set.

I've more or less 1.5k sentences and 29 classes (imbalanced).

According to the literature, a good choice is to generate synthetic data, oversampling, or undersampling the training data, using for example the SMOTE algorithm.

I also want use a cross-validation mechanism (stratified k-fold) to be more confident in the obtained result.

I also know that accuracy is not the right metric to take into account, I should use precision, recall and confusion matrix.

Is it possible to combine cross validation k-fold and oversampling or undersampling techniques?

It is straightforward to combine k-fold cross-validation with a technique like oversampling or undersampling.

First, apply the balance-restoration technique to your training data. Then parametrize a model using k-fold cross-validation on the re-balanced training data. In Scikit learn, I believe you can even bundle these actions together into a single 'pipeline' object to make it easier to manipulate.

Precision/recall is probably a fine starting place for measuring performance.


Intelligence over simple data Let's say we already have an intelligent system (as a computer program). We give it some simple data from outside world, and system tries to predict it. The question would be: what kind of patterns this system should be able to recognize?

For example, system gets binary data stream(zeroes and ones).
1) Should it recognize patterns like: 1 0 11 0 111 0 ....?
2) Should this system be able to recognize prime numbers in ascending order written with "1" with "0" between them?

Is there any research about it?

It depends on the algorithm.

A classical algorithm would not be able to recognize the pattern of primes unless it was programmed to do so.  This pre-programming would not have to be exclusive to primes, but the algorithm would have to "understand" the nature of primes including 1 (the number can only be divided by 1 and itself.)

Because computers are good at number crunching, it should be quite possible to program an algorithm that looks for numerical relationships, such as primes, parity and factors, without being taught those specific concepts.

Extending this to alternate representations of numbers, 1=1, 11=2, 111=3 requires a type of abstract thinking (i.e. the literal representation is not the true meaning. In an 8-bit byte 11101101 = 237. In Base-2 1,11,111 are the primes 1,3,7;)

However, if the algorithm was superintelligent, figuring out the example would be trivial, and, ostensibly, correctly deriving patterns of any kind, no matter how opaque, would be possible, given enough time and memory.


How can I use a trained CNN to predict a new image label? I was applying this CNN fine-tuning example from Matlab.

The example shows how to fine-tune a pre-trained CNN on letters to classify images of digits. Now I would like to use this new fine-tuned CNN on new images of digits that i have on my computer. How can I do that?

As the question has been left unanswered, for future readers of the questions:

The documentation you link gives the answer to your question.

Given the fact you have a pre-trained model as you say:

YPred = classify(netTransfer,testDigitData);

where netTansfer is the pre-trained model and testDigitData is your test image that you want to predict the label.

Your image(s) need to be loaded following the instruction also given in the documentation:

digitDatasetPath = fullfile(matlabroot,'toolbox','nnet','nndemos', ...
    'nndatasets','DigitDataset');

digitData = imageDatastore(digitDatasetPath, ...
    'IncludeSubfolders',true,'LabelSource','foldernames');

(digitData will become testDigitData if the images that you load are for testing purpose only, which seems to be your case)