What is a good way of identifying volatile positions for a checkers game? I am implementing an AI for a mobile checkers game, and have used alpha-beta pruning with Minimax.
Now I have the problem of horizontal effect, and need to do Quiesence search to avoid that.
Any advice on what makes a position volatile for a checkers game?
I want to consider the cases when player can take a piece, and also when any piece can be taken by opponent a volatile position, and continue searching for another depth.
Anything else?

When combatting the horizon affect, you want to consider any short term actions that will greatly affect your position evaluation. Thus, in addition to captures, you will also want to include:

When the opponent can make a king next move
When the current player can make a king next move
When the only legal moves left will lead to capture the turn after for the opponent
When the only legal moves left will lead to capture the turn after for the current player

3 and 4 are commonly called Zugzwang and play a very prominent role in high level checkers, but may be a bit more difficult to implement. However, they will contribute perhaps most of all when combatting the horizon affect.


What are examples of applications of the Fourier transform to AI? The (discrete and continuous) Fourier transform (FT) is used in signal processing in order to convert a signal (or function) in a certain domain (e.g. the time domain) to another domain (e.g., frequency domain). There are several resources on the web that attempt to explain the FT at different levels of complexity. See e.g. this answer or this and this Youtube videos.

What are examples of (real-world) applications of the Fourier transform to AI? I am looking for answers that explain the reason behind the use of the FT in the given application. I suppose that there are several applications of the FT to e.g. ML (data analysis) and robotics. I am looking for specific examples.

The filtering (or convolution) process in Convolutional Neural Networks can be implemented using Fourier Transforms. Convolutions in image (or spatial) space are equivalent to multiplications in frequency space, and multiplications can be performed much more efficiently than convolutions.

The algorithm for using Fourier Transforms to calculate a convolution between an image and a filter kernel is as follows:

Assuming a 100x100 input image, and a 3x3 filter kernel (such as a Sobel Edge-Detection filter).

Calculate the 2D Fourier Transform (FT) of the image (C library, Python library)
Pad the filter kernel so that it is the same size as the image, and that the filter kernel is centred at image coordinate (0, 0). This means that some parts of the kernel will be placed in the other corners of the padded filter image. For example, consider a 3x3 kernel where the filter coefficients are [[a, b, c], [d, e, f], [g, h, i]], in this case the coefficients map to pixel positions on the padded kernel as follows:

a -> (99, 99)    //bottom right
b -> (0, 99)     //bottom left
c -> (1, 99)
d -> (99, 0) //top right
e -> (0, 0) //top left
f -> (1, 0)
g -> (99, 1)
h -> (0, 1)
i -> (1, 1)

Set all the other padded filter kernel pixels to zero
Calculate the 2D Fourier Transform of the padded filter kernel.
Perform an element-wise multiplication of the image FT and the padded filter kernel FT. Note that this is a complex multiplication as the FT consists of complex numbers. The result of this multiplication is therefore also an array of complex numbers.
Calculate the Inverse Fourier Transform of the result from step 4. This will return a real valued image which is the original image convolved with the filter kernel.

The advantage for performing the kernel convolutions in this way is that it is much faster than performing an element-wise convolution directly. This speeds up the training process for neural networks with Convolutional layers.


In gradient descent's update rule, why do we use $\sigma(z^{l-1})\frac{\delta C_0}{ \delta w^{l}}$ instead of $\frac{\delta C_0}{\delta w^{l}}$? I am trying to code a two layered neural network simple NN as I have described here https://itisexplained.com/html/NN/ml/5_codingneuralnetwork/
I am getting stuck on the last step of updating the weights after calculating the gradients for the outer and inner layers via back-propagation
#---------------------------------------------------------------

# Two layered NW. Using from (1) and the equations we derived as explanations
# (1) http://iamtrask.github.io/2015/07/12/basic-python-network/
#---------------------------------------------------------------

import numpy as np
# seed random numbers to make calculation deterministic
np.random.seed(1)

# pretty print numpy array
np.set_printoptions(formatter={'float': '{: 0.3f}'.format})

# let us code our sigmoid funciton
def sigmoid(x):
    return 1/(1+np.exp(-x))

# let us add a method that takes the derivative of x as well
def derv_sigmoid(x):
   return x*(1-x)

# set learning rate as 1 for this toy example
learningRate =  1

# input x, also used as the training set here
x = np.array([ [0,0,1],[0,1,1],[1,0,1],[1,1,1]  ])

# desired output for each of the training set above
y = np.array([[0,1,1,0]]).T

# Explanaiton - as long as input has two ones, but not three, ouput is One
"""
Input [0,0,1]  Output = 0
Input [0,1,1]  Output = 1
Input [1,0,1]  Output = 1
Input [1,1,1]  Output = 0
"""

input_rows = 4
# Randomly initalised weights
weight1 =  np.random.random((3,input_rows))
weight2 =  np.random.random((input_rows,1))

print("Shape weight1",np.shape(weight1)) #debug
print("Shape weight2",np.shape(weight2)) #debug

# Activation to layer 0 is taken as input x
a0 = x

iterations = 1000
for iter in range(0,iterations):

  # Forward pass - Straight Forward
  z1= x @ weight1
  a1 = sigmoid(z1)
  z2= a1 @ weight2
  a2 = sigmoid(z2)

  # Backward Pass - Backpropagation
  delta2  = (y-a2)
  #---------------------------------------------------------------
  # Calcluating change of Cost/Loss wrto weight of 2nd/last layer
  # Eq (A) ---> dC_dw2 = delta2*derv_sigmoid(z2)
  #---------------------------------------------------------------

  dC_dw2  = delta2 * derv_sigmoid(a2)

  if iter == 0:
    print("Shape dC_dw2",np.shape(dC_dw2)) #debug

  #---------------------------------------------------------------
  # Calcluating change of Cost/Loss wrto weight of 2nd/last layer
  # Eq (B)---> dC_dw1 = derv_sigmoid(a1)*delta2*derv_sigmoid(a2)*weight2
  # note  delta2*derv_sigmoid(a2) == dC_dw2
  # dC_dw1 = derv_sigmoid(a1)*dC_dw2*weight2
  #---------------------------------------------------------------

  dC_dw1 =  (np.multiply(dC_dw2,weight2.T)) * derv_sigmoid(a1)
  if iter == 0:
    print("Shape dC_dw1",np.shape(dC_dw1)) #debug

  #---------------------------------------------------------------
  #Gradinent descent
  #---------------------------------------------------------------

  #weight2 = weight2 - learningRate*dC_dw2 --> these are what the textbook tells
  #weight1 = weight1 - learningRate*dC_dw1

  weight2 = weight2 + learningRate*np.dot(a1.T,dC_dw2) # this is what works
  weight1 = weight1 + learningRate*np.dot(a0.T,dC_dw1)

print("New ouput\n",a2)

Why is
  weight2 = weight2 + learningRate*np.dot(a1.T,dC_dw2)
  weight1 = weight1 + learningRate*np.dot(a0.T,dC_dw1)

done instead of
  #weight2 = weight2 - learningRate*dC_dw2
  #weight1 = weight1 - learningRate*dC_dw1

I am not getting the source of the equation of updating the weights by multiplying with the activation of the previous layer
As per gradient descent, the weight update should be
$$
  W^{l}_{new} = W^{l}_{old} - \gamma * \frac{\delta C_0}{\delta w^{l}}
$$
However, what works in practice is
$$
 W^{l}_{new} = W^{l}_{old} - \gamma * \sigma(z^{l-1})\frac{\delta C_0}{ \delta w^{l}},
$$
where $\gamma$ is the learning rate.

Okay - the answer is here  https://explained.ai/matrix-calculus/#sec6.2 and it is pretty involved. Basically, there is a difference when you derive the equation for one neuron and when you have to do practically for a set of neurons. The answer is matrix calculus. Here goes from what I could make out. Feel free to correct if I am wrong
Gradient Vector/Matrix/2D tensor of Loss function wrto Weight
$$
C = \frac{1}{2} \sum_j (y_j-a^L_j)^2
$$
Assuming a neural net with 2 layers, we have the final Loss as
$$
C = \frac{1}{2} \sum_j (y_j-a^2_j)^2
$$
Where
$$
a^2 = \sigma(w^2.a^1)
$$
We can then write
$$
C = \frac{1}{2} \sum_j v^2 \quad \rightarrow (Eq \;A)
$$
Where
$$
v= y-a^2
$$
Partial Derivative of Loss function wrto Weight
For the last layer, lets use Chain Rule to split like below
$$
\frac {\partial C}{\partial w^2} = \frac{\partial v^2}{\partial v} * \frac{\partial v}{\partial w^2} \quad \rightarrow (Eq \;B)
$$
$$
 \frac{\partial v^2}{\partial v} =2v \quad \rightarrow (Eq \;B.1)
$$
$$
\frac{\partial v}{\partial w^2}=  \frac{\partial (y-a^2)}{\partial w^2} =
0-\frac{\partial a^2}{\partial w^2} \quad \rightarrow (Eq \;B.2)
$$
$$
\frac {\partial C}{\partial w^2} = \frac{1}{2} *2v(0-\frac{\partial a^2}{\partial w^2}) \quad \rightarrow (Eq \;B)
$$

Now we need to find $\frac{\partial a^2}{\partial w^2}$
Let
$$
a^2= \sigma(sum(w^2 \otimes a^1 )) = \sigma(z^2)
$$
$$
z^2 = sum(w^2 \otimes a^1 )
$$
$$
z^2 = sum(k^2) \; \text {where} \; k^2=w^2 \otimes a^1
$$
We now need to derive an intermediate term which we will use later
$$
\frac{\partial z^2}{\partial w^2} =\frac{\partial z^2}{\partial k^2}*\frac{\partial k^2}{\partial w^2}
$$
$$
=\frac {\partial sum(k^2)}{\partial k^2}* \frac {\partial (w^2 \otimes a^1 )} {\partial w^2}
$$
$$
\frac{\partial z^2}{\partial w^2} = (1^{\rightarrow})^T* diag(a^1) =(a^{1})^T \quad \rightarrow (Eq \;B.3)
$$
How the above is, you need to check this  in https://explained.ai/matrix-calculus/#sec6.2
Basically though these are written like scalar here; actually all these are partial differention of vector by vector, or vector by scalar; and a set of vectors can be represented as the matrix here.
Note that the vector dot product $w.a$ when applied on matrices becomes the elementwise multiplication $w^2 \otimes a^1$ (also called Hadamard product)
Going back to  $Eq \;(B.2)$
$$
\frac {\partial a^2}{\partial w^2} = \frac{\partial a^2}{\partial z^2} * \frac{\partial z^2}{\partial w^2}
$$
Using $Eq \;(B.3)$ for the term in left
$$
=  \frac{\partial a^2}{\partial z^2} * (a^{1})^T
$$
$$
=  \frac{\partial \sigma(z^2)}{\partial z^2} * (a^{1})^T
$$
$$
\frac {\partial a^2}{\partial w^2} =   \sigma^{'}(z^2) * (a^{1})^T \quad \rightarrow (Eq \;B.4)
$$
Now lets got back to partial derivative of Loss function wrto to weight
$$
\frac {\partial C}{\partial w^2} = \frac {1}{2}*2v(0-\frac{\partial a^2}{\partial w^2}) \quad \rightarrow (Eq \;B)
$$
Using $Eq \;(B.4)$ to substitute in the last term
$$
= v(0- \sigma^{'}(z^2) * (a^{1})^T)
$$
$$
= v*-1*\sigma^{'}(z^2) * (a^{1})^T
$$
$$
= (y-a^2)*-1*\sigma^{'}(z^2) * (a^{1})^T
$$
$$
\frac {\partial C}{\partial w^2}= (a^2-y)*\sigma^{'}(z^2) * (a^{1})^T
$$


Why feed actions in later layer in Q network? I read the DDPG paper, in which the authors state that the actions are fed only later to their Q network:

  Actions were not included until the 2nd hidden layer of Q. (Sec 7, Experiment Details)

So does that mean, that the input of the first hidden layer was simply the state and the input of the second hidden layer the output of the first hidden layer concatenated with the actions?

Why would you do that? To have the first layer focus on learning the state value independent of the selected action? How would that help?

Is this just a small little tweak or a more significant improvement?

So does that mean, that the input of the first hidden layer was simply the state and the input of the second hidden layer the output of the first hidden layer concatenated with the actions?

Yes.

  Why would you do that? To have the first layer focus on learning the state value independent of the selected action? How would that help?

Neural networks hidden layers learn representations that get progressively closer to a linear relationship with the target, layer by layer.

So the first layer would not be learning the state value per se, but some representation of the state that was better related to the action value at the output.

Neural network architectures are often established by experimentation, so I expect here they tried the idea and the performance was OK. If they do not give alternative architectures in the paper, then the precise reason is not clear.

I can try a few guesses:

Concatenating the actions with the states in the input layer would result in more parameters for the neural network to achieve the same accuracy, running slightly slower. That's because more weights would be required to link the input layer with the first hidden layer.
Separating the layers of state and action inputs is a form of regularisation, as the first layer has to produce features that are useful for all possible actions.
You don't want the neural network to construct a reverse map $\pi(a|s) \rightarrow Q(s,a)$, you want it to independently assess the action values to do its job as a critic. By having the states and actions presented in different laters, this reduces the chance of the neural network finding shortcuts due to "recognising the policy" to predict the values. That's because the state and action pairings may repeat (so be recognised), but the first layer activations change over time (so even with repeated state/action pairs, this is a new representation to learn from).

  Is this just a small little tweak or a more significant improvement?

I don't know, and suggest looking through previous work by the same authors in case they describe the approach in more detail.


Does it make sense to apply softmax on top of relu? While working through some example from Github I've found this network (it's for FashionMNIST but it doesn't really matter).

Pytorch forward method (my query in upper case comments with regards to applying Softmax on top of Relu?):

def forward(self, x):
    # two conv/relu + pool layers
    x = self.pool(F.relu(self.conv1(x)))
    x = self.pool(F.relu(self.conv2(x)))

    # prep for linear layer
    # flatten the inputs into a vector
    x = x.view(x.size(0), -1)

    # DOES IT MAKE SENSE TO APPLY RELU HERE
    **x = F.relu(self.fc1(x))

    # AND THEN Softmax on top of it ?
    x = F.log_softmax(x, dim=1)**

    # final output
    return x

Does it make sense?

In general, yes it is interpretable, back propagation will work, and the NN can be optimised.

By using ReLU, the default network has a minimum logit of $0$ for the softmax input, which means at least initially that there will be higher minimum probabilities associated with all classes (compared to allowing negative logits which would happen randomly with usual weight initialisation). The network will need to learn to produce higher logit values for correct answers, because it has no ability to produce lower logit values for incorrect answers. This is like training a network to produce the highest regression value on one output, whilst clipping all values to be 0 or above, so it does not have the option of making one output e.g. $-1.0$ and the rest $-100.0$

It can probably be thought of as a type of regularisation, as it puts constraints on activation values that will work.

Is it needed?

That is less clear. You can try training with and without the line, and using cross-validation or a test set to see if there is a significant difference.

If the network has been designed well, then I'd expect to see a slight improvement with the added ReLU.

If it is a mistake, then I'd expect to see no difference, or better performance without the ReLU.


Choosing a policy improvement algorithm for a continuing problem with continuous action and state-space I'm trying to decide which policy improvement algorithm to use in the context of my problem. But let me emerge you into the problem
Problem
I want to move a set of points in a 3D space. Depending on how the points move, the environment gives a positive or negative reward. Further, the environment does not split up into episodes, so it is a continuing problem. The state space is high-dimensional (a lot of states are possible) and many states can be similar (so state aliasing can appear), also states are continuous. The problem is dense in rewards, so for every transition, there will be a negative or positive reward, depending on the previous state.
A state is represented as a vector with dimension N (initially it will be something like ~100, but in the future, I want to work with vectors up to 1000).
In the case of action, it is described by a matrix 3xN, where N is the same as in the case of the state. The first dimension comes from the fact, that action is 3D displacement.
What I have done so far
Since actions are continuous, I have narrowed down my search to policy gradient methods. Further, I researched methods, that work with continuous state spaces. I found a deep deterministic policy gradient (DDPG) and the Proximal Policy Gradient (PPO) would fit here. Theoretically, they should work but I'm unsure and any advice would be gold here.
Questions
Would those algorithms be suitable for the problem (PPO or DDPG)?
There are other policy improvement algorithms that would work here or a family of policy improvement algorithms?

I'm using my own implementation of A2C (Advantage Actor Critic) in an industrial application based on Markov Process (present state alone provides sufficient knowledge to make an optimal decision). It's simple and versatile, its performance proven in many different applications. The results so far have been promising.
One of my colleagues had issues with solving a simple task of mapping images to coordinates with OpenAI's Stable Baselines implementations of PPO and TRPO. Hence I'm biased against this framework.
My suggestion is to try the simplest model and if that doesn't satisfy your expectations for performance, then try something fancier. Once you've made a pipeline for learning, switching to a different algorithm is relatively time inexpensive.
Here is a list of algorithms for continuous action and state space from the wikipedia article about RL:


How to use mixed data for image segmentation? I have a task for which I have to do image segmentation (cancer detection on MRIs). If possible, I would also like to include clinical data (i.e. numeric/categorical data which comes in the form of a table with features such as age, gender, ...).
I know that for classification purposes, it's possible to create a model that uses both numeric data as well as image data (as mentioned in the paper by Huang et al. : "Multimodal fusion with deep neural networks for leveraging CT imaging and electronic health record: a case‑study in pulmonary embolism detection"
The problem I have is that, for image segmentation tasks, it doesn't really make sense to me as to how to use both types of data.
In the above-mentioned paper, they create one model with only the image data and another with only the numeric data, and then they fuse them (there are multiple strategies for fusing them together). For classification tasks, it makes sense. However, for my task, it does not make sense to have a model which only uses the clinical data for image segmentation and that's where I get confused.
How do you think I should proceed with my task? Is it even possible to mix both types of data for image segmentation tasks?

You can try doing image segmentation the traditional way, just using the image data. If you want to use the non-image data, then, you can introduce classification as another task for your network. It will provide some regularization to your model. But, this is one way you can still use non-image data whilst still working with image outputs.


Does the Bayesian MAP give a probability distribution over unseen t*? I'm working my way through the Bayesian world. So far i've understood that the MLE or the MPA are point estimates, therefore using such models just output one specific value and not an distribution.
Moreover, vanilla neuronal networks do in fact s.th. like MLE, because minimizing the squared-loss or the cross-entropy is similar to find parameters that maximize the likelihood. Moreover, using neuronal networks with regularisation is comparable to the MAP estimates, as the prior works like the penalty term in error functions.
However, i've found this work. It shows that the weights $W_{PLS}$ gained from a penalized least-squared are the same as the weights $W_{MAP}$ gained through Maximum a posterori:

However the paper says: "The first two approaches result in similar predictions, although the MAP Bayesian model does give a probability distribution for $t_*$ The mean of this
distribution is the same as that of the classical predictor $y(x_*; W_{PLS})$, since $W_{PLS} = W_{MAP}$
What I don't get here is how can the MAP Bayesian give a proability distribution over $t_*$, when it is only a point estimate? Consider a neuronal network - a point estimate would mean some fixed weights so how can there be a output probability distribution? I thought that this is only achieved in the True Bayesian where we integrate out the unknown weights, therefore building something like the weight averaged of all outcomes, using all possible weights.
Can you help me?

You're correct: the MAP estimate is a point estimate (specifically, MAP is used to estimate the mode of a probability distribution).
I think that the paper is referring to the (output) probability distribution over the possible targets/labels, given the point estimate. However, in the case of MLE, you can also have that probability distribution, so I'm not sure why the paper's author is emphasizing that with MAP you can build that probability distribution (so maybe this/my interpretation of that excerpt is wrong!).
That table also shows that the MAP estimate is used to produce a probability distribution, but the $\sigma$ there should be unknown, but I didn't read that article, so maybe I am missing some info or assumption.
In any case, you could also find a point estimate of a parameter of a probability distribution, but this does not imply that MAP produces a probability distribution. For instance, you can show that, if you place a Gaussian prior on the weights of a neural network, this leads to the $L_2$ loss function, but training a (normal) neural network with $L_2$ does not lead to a probability distribution over the weights.
You should try reading only information from reliable references and books. For MAP, check out chapter 5. (p. 149) of the book "Machine Learning: A Probabilistic Perspective" by Murphy.


What is the difference between genetic algorithms and evolutionary game theory algorithms? What is the difference between genetic algorithms and evolutionary game theory algorithms?

Philip's answer is good, but I'll add to it.

In a GA, a population of individuals (typically represented by bit strings) is evaluated for its fitness on a particular task. Each individual is evaluated separately by a fitness function than can determine its quality. In the Traveling Salesman Problem, the bit string might represent a sequence of numbers, for instance, corresponding to an order in which cities are visited during the tour. The fitness function would inspect a single individual, compute the total cost of the tour, and assign that individual a fitness based on that value. Low scoring individuals are removed, high scoring individuals generate variants on themselves, and the process repeats.

In Evolutionary Game Theory, a population of individuals is also evaluated on some task, but usually the task involves interaction between the individuals. For example, you could use an EGT simulation to study what happens in a game like Iterated Prisoner's Dilemma. Here, an individual's fitness doesn't just depend on the rules of the task, but on the behaviors and strategies of the other players in the population. A strategy that is highly effective at first (like always cooperate) will quickly die out once defectors appear. Defectors are effective as long as there are some cooperators to prey upon, but are quickly defeated by strategies like Tit-for-Tat. Usually researchers are not interested in the specific strategies that emerge so much as in the population dynamics over the course of the simulation, and in what kinds of population equilibria can emerge. Check out some of Dan Ashlock's papers on Game Theory for more.


On-policy distribution for Emphatic TD The book by Sutton and Barto discussed in section 11.8 that the convergence of off-policy TD function approximation can be improved by correcting for distribution of states encountered. The section seems to be written in haste and doesn't do a good job in explaining why will $M_t$, the emphasis, help in getting a state distribution closer to the target policy.

My understanding of on-policy distribution is not clear at the moment. I think it is the distribution of states encountered under the target policy (the policy for which we want to state-action/state values). The importance sampling ratio corrects for update distribution (by multiplying the correction term with the ratio), but how is $M_t$ helping in correcting for state distribution?

I don't think the section was written in haste. I think they just didn't have space to include the whole proof. It's a bit involved, so they just gave concepts.

An Emphatic Approach to the Problem
of Off-policy Temporal-Difference Learning gives a proof of stability.
At least parts of it should seem familiar if you've read Sutton and Barto's proof of the convergence of linear TD(0) on page 206 of their 2nd edition RL book.

On Convergence of Emphatic Temporal-Difference Learning gives a proof of convergence.

I confess that I don't understand these papers well enough to give a summary. If you eventually do, I would greatly appreciate an update.


Is logistic regression more free from the conditional independence assumption than naive Bayes? To my understanding, logistic regression is an extension of naive Bayes.

Suppose $X = \{x_1, x_2, \dots, x_N \}$ and $Y = \{0, 1\}$, each $x_i$ is i.i.d and $P(x_i \mid Y=y_k) \sim \mathcal{N}(\mu, \sigma^2)$ is a Gaussian distribution.

In order to create a linear decision surface, we take the assumption of each pdf $P(x_i \mid y_k)$ having variance $\sigma^2$ independent of the value of $Y$, i.e.
$\sigma_{i, k} = \sigma_i$ ($i \rightarrow x_i, k \rightarrow y_k$).

Finally, we end up learning the coefficients $(\omega_0, \omega_i)$  that represent the linear decision surface in following equation (linear decision surface):

$$
\frac{P(Y=0 \mid X)}{P(Y=1 \mid X)} = \omega_0 + \sum_{i} \omega \cdot x_i
$$

Even though the derivation of Linear Regression coefficients$(\omega_0, \omega_i)$ involves the assumption of conditional independent $x_i$ given $Y$, why is it said that learning these coefficients from training data are "somewhat more free" from conditional independence assumption, as compared to learning the regular Bayesian Distribution coefficients $\mu, \sigma$?

I came across this while following this course here.

I refer you to Prof. Tom Mitchell's (the lecturer in the video) draft chapter as the best explanation I could find. I will try to explain it in layman's terms here.

Given a boolean problem, our logistic regression classifier will assign the label $Y=0$ iff $w_0 + \sum_{i=1}^{n}w_{i}X_{i} > 0$ (from the video you posted; eqn. 18 in the draft chapter). Otherwise, it will assign the label $Y=1$. Remember the goal of logistic regression is to learn the weights $w_0, w_1, ..., w_n$ that will give us a linear function that would seperate our data best.

It can be shown that $w_0=\ln\frac{1-\pi}{\pi} + \sum_{i=1}^{n} \frac{\mu_{i1}^2 - \mu_{i0}^2}{2\sigma_{i}^2}$ and $w_i = \frac{\mu_{i0} - \mu_{i1}}{\sigma_{i}^2}$ (eqns. 19-23 in the draft chapter), where:

$\pi$ is $P(Y=1)$,
$\mu_{i0}$ and $\mu_{i1}$ are the means of feature $X_i$ in classes $Y=0$ and $Y=1$, respectively, and
$\sigma_i$ is the standard deviation of feature $X_i$

This means that we can express our weights $w_0, w_1, ..., w_n$ in terms of the Gaussian Naive Bayes classifier's parameters. Now, here is the difference:

Our goal is to find $P(Y=y_k|X_1 ... X_n)$. Assuming conditional independence, $P(Y=y_k|X_1 ... X_n) \propto P(Y=y_k) \prod_{i}^n P(X_i | Y=y_k)$.

If we estimate $\pi$, $\mu_{i0}$, $\mu_{i1}$, and $\sigma_i$, then use those values to estimate $w_0, w_1, ..., w_n$, then we use the conditional independence property directly in our calculations. This is what the Naive Bayes classifier does.

If we estimate $w_0, w_1, ..., w_n$ directly, by repeatedly minimizing the error of an assumed linear function, we are not strictly abiding by the conditional independence property, because we don't explicitly calculate an estimate of the conditional probability $P(X_i|Y=y_k)$. We completely bypass that step. This is what the Logistic Regression classifier does.

From the section 4 of the draft chapter (GNB is short for Gaussian Naive Bayes):

  ... Logistic Regression directly estimates the parameters of $P(Y|X)$, whereas Naive Bayes directly estimates parameters for $P(Y)$ and $P(X|Y)$... if the GNB assumptions hold, then asymptotically (as the number of training examples grows
  toward infinity) the GNB and Logistic Regression converge toward identical classifiers.

Another way to understand this is that it has to do with the fact that Logistic Regression is a discriminative classifier, whereas Naive Bayes is a generative classifier. For more on that, read Prof. Andrew Y. Ng and Prof. Michael I. Jordan's paper, On Discriminative vs. Generative classifiers: A comparison of logistic regression and naive Bayes.


What is the intuition behind how word embeddings bring information to a neural network? How is it that a word embedding layer (say word2vec) brings more insights to the neural network compared to a simple one-hot encoded layer?
I understand how the word embedding carries some semantic meaning, but it seems that this information would get "squashed" by the activation function, leaving only a scalar value and as many different vectors could yield the same result, I would guess that the information is more or less lost.
Could anyone bring me insights as to why a neural network may utilize the information contained in a word embedding?

Shakespeare once said "A rose by any other name would smell as sweet" (Romeo and Juliet). Words are just labels we attach to ideas for convenience. By using one hot we remain tied to the letter sequence r,o,s,e, and some other structure must take on the responsibility of attaching the context of sweetness to it.

Word embeddings learn a multi-dimensional context. What exactly the context is of each dimension of the embedding is something of a mystery and simply emerges from the learning. The larger the number of dimensions, the greater the possibility that some combination of the dimensions will represent the sweetness context, but it might be quite hard to tease out.

So you can attach the idea of sweetness to one member of a one-hot structure, but it must necessarily be a part of a rules-based approach. Embeddings, when they are working well, will not need the rules.


Why can the Bellman equation be turned into an update rule? In chapter 4.1 of Sutton's book, the Bellman equation is turned into an update rule by simply changing the indices of it. How is it mathematically justified? I didn't quite get the initiation of why we are allowed to do that?

$$v_{\pi}(s) = \mathbb E_{\pi}[G_t|S_t=s]$$

$$ = \mathbb E_{\pi}[R_{t+1} + \gamma G_{t+1}|S_t=s]$$

$$= \mathbb E_{\pi}[R_{t+1} + \gamma v_{\pi}(S_{t+1})|S_t=s]$$

$$ = \sum_a \pi(a|s)\sum_{s',r} p(s',r|s,a)[r+ \gamma v_{\pi}(s')]$$

from which it goes to the update equation:

$$v_{k+1}(s) =  \mathbb E_{\pi}[R_{t+1} + \gamma v_{k}(S_{t+1})|S_t=s]$$

$$=\sum_a \pi(a|s)\sum_{s',r} p(s',r|s,a)[r+ \gamma v_{k}(s')]$$

Why are we allowed to convert the Bellman equations into update rules?

There is a simple reason for this: convergence. The same chapter 4 of the same book mentions it. For example, in the case of policy evaluation, the produced sequence of estimates $\{v_k\}$ is guaranteed to converge to $v_\pi$ as $k$ (i.e. the number of iterations) goes to infinity. There are other RL algorithms that are also guaranteed to converge (e.g. tabular Q-learning).

To conclude, in many cases, the update rules of simple reinforcement learning (or dynamic programming) algorithms are very similar to their mathematical formalization because algorithms based on those update rules are often guaranteed to converge. However, note that many more advanced reinforcement learning algorithms (especially, the ones that use function approximators, such as neural networks, to represent the value functions or policies) are not guaranteed or known to converge.


In NN, as iterations of Gradient descent increases, the accuracy of Test/CV set decreases. how can i resolve this? As mentioned in the title I'm using 300 Dataset example with 500 feature as an input.

As I'm training the dataset, I found something peculiar.  Please look at the data shown below.

  Iteration 5000 | Cost: 2.084241e-01

  Training Set Accuracy: 100.000000

  CV Set Accuracy: 85.000000

  Test Set Accuracy: 97.500000

  Iteration 3000 | Cost: 2.084241e-01

  Training Set Accuracy: 98.958333

  CV Set Accuracy: 85.000000

  Test Set Accuracy: 97.500000

  Iteration 1000 | Cost: 4.017322e-01

  Training Set Accuracy: 96.875000

  CV Set Accuracy: 85.000000

  Test Set Accuracy: 97.500000

  Iteration 500 | Cost: 5.515852e-01

  Training Set Accuracy: 95.486111

  CV Set Accuracy: 90.000000

  Test Set Accuracy: 97.500000

  Iteration 100 | Cost: 8.413299e-01

  Training Set Accuracy: 90.625000

  CV Set Accuracy: 95.000000

  Test Set Accuracy: 97.500000

  Iteration 50 | Cost: 8.483802e-01

  Training Set Accuracy: 90.277778

  CV Set Accuracy: 95.000000

  Test Set Accuracy: 97.500000

The trend is that as the Iteration(cost) increases(cost decreases), the training set accuracy increases as expected, but the CV Set/Test Set Accuracy decreases. My initial thought is that this has to do with precision/bias issue, but I really can't buy it.

Anyone know what this entails? Or any reference?

Training scores improving (loss decreasing and accuracy increasing) whilst the opposite happens with cross validation and test data is a sign of overfitting to the training data. Your neural network is getting worse at generalising and no amount of further training will improve it - in fact the situation will get worse the more you train.

This is the main reason you have CV data sets, to show you when this happens before you try to use your model against the test set or real world data. So it is not "peculiar" at all, but the CV set doing its job for you, allowing you notice something has gone wrong.

To improve on the situation, you need to use some form of regularisation. The simplest approach here would be to take your model from around 100 epochs (because it has best CV score that I can see). This is early stopping, and is a simple valid regularisation approach.

Alternatives for neural networks include L2 weight regularisation (also called weight decay) and dropout.

In addition to this, your question states that you only have 300 examples, and more features than examples. This is really tricky to generalise (and hard to tell if you have - for instance it looks like you only have 20 CV examples and 40 test examples, these numbers are very low and prone to giving you inaccurate estimates simply due to chance of what entries are in the data sets). I recommend you look into K-fold cross-validation in order to get more accurate scores, and help you choose the best model for generalisation.


How does 'Bright score' assess how strong the connection is between users and their jobs? In 2014 Linkedin acquired Bright.com, for $120 million and it is using AI and big data algorithms to connect users.

Bright also throws in a little Klout, ranking people by a “Bright score” which it uses to assess how strong the chemistry is between a user and a particular job.
It also takes into account historical hiring patterns into its matching, along with account location, a user’s past experience and synonyms.

In brief, is it known (based on some research papers) how such algorithm works which aiming at scoring 'chemistry' between users and their jobs?

Take a look at this 2012 paper by three people at Bright. (Sadly, it's paywalled and I couldn't easily find a ungated copy, so I don't have a summary for you.)

The abstract:

  Bright has built an automated system for ranking job candidates against job descriptions. The candidate's resume and social media profiles are interwoven to build an augmented user profile. Similarly, the job description is augmented by external databases and user-generated content to build an enhanced job profile. These augmented user and job profiles are then analyzed in order to develop numerical overlap features each with strong discriminating power, and in sum with maximal coverage. The resulting feature scores are then combined into a single Bright Score using a custom algorithm, where the feature weights are derived from a nation-wide and controlled study in which we collected a large sample of human judgments on real resume-job pairings. We demonstrate that the addition of social media profile data and external data improves the classification accuracy dramatically in terms of identifying the most qualified candidates.


How to define Agar.io state and action space? I am trying to implement an AI bot for my Agar.io clone using deep neural network.
However, I am struggling with the state and action space of the AI bot.
Because the bot can take real number for position and velocity, can I say the state space is continuous?
For the action space, I am thinking something like (velocityX, velocityY, "split to half", "eject mass").
What should be the number of input nodes in the input layer for my Neural network? And what are those input(observations, rewards)?
As the number of players and AI bots are changing, how can I train a dynamic network with changing input node number?
For the outputs, how can I get a continuous action output like velocity?
As a reference, you can learn about the game rules from this short youtube video:
20 Rules and Game Mechanism of Agar (How to Play Agar.io)

The state space is certainly continuous, assuming that you can somehow feed that AI exact coordinates. You may have to resort to CNNs if you do not have access to this information. For the action space, you should consider how the game actually plays. Since you use a mouse to simply show the direction, you could use (x,y) positions of the mouse as an action, or even just the angle $\theta$ of the mouse cursor in a circle around the agent. If you are playing on the site, then your observations would have to be from a CNN, it should be possible to use the score as your reward, as well as the possibility for eating things or distance from opponents that are bigger or smaller as intermediate rewards. The number of nodes in your network is something that you must find experimentally, you may like to research what kind of architectures other people have used in this field. You shouldn't need to account for different numbers of players. Nothing special needs to be done to get a single node giving a continuous output to represent the angle $\theta$, or alternatively two nodes representing x and y. You can then use tanh or sigmoid to limit the output node values for eject and split actions.


What is an objective function? Local search algorithms are useful for solving pure optimization problems, in which the aim is to find the best state according to an objective
function.
My question is what is the objective function?

The "objective function" is the function that you want to minimise or maximise in your problem.

The expression "objective function" is used in several different contexts (e.g. machine learning or linear programming), but it always refers to the function to be maximised or minimised in the specific (optimisation) problem. Hence, this expression is used in the context of mathematical optimisation.

For example, in machine learning, you define a model, $\mathcal{M}$. To train $\mathcal{M}$, you usually define a loss function $\mathcal{L}$ (e.g., a mean squared error), which you want to minimise. $\mathcal{L}$ is the "objective function" of your problem (which in this case is to be minimised).

In the context of search algorithms, the objective function could represent e.g. the cost of the solution. For example, in the case of the travelling salesman problem (TSP), you define a function, call it $C$, which represents the "cost" of the tour or Hamiltonian cycle, that is, a function which sums up the weights of all edges in the tour. In this case, the "objective" of your problem is to minimise this function $C$, because, essentially, you want to find an inexpensive tour, which is associated with either a local (or global) minimum of $C$. This function $C$ is the "objective function".

It should now be easy to memorise the expression "objective function", as it contains the term "objective", and the "objective" (or goal) in your (optimisation) problem is to minimise (or maximise) the corresponding function.


What are daily life examples of SAT problems? What are (good) daily life examples of SAT problems?

I've thought about this one. The problem of placing a bunch of different kinds of glasses in a shared cabinet in such a way that some constraints would be satisfied, such as putting the longer ones in the back of the shorter ones, so it will be easier to take them when we need.

Is it a good one, or can you think of any other better one?

SAT problems are decision problems that can be categorised as NPC. This informally means although there has not been any solution that can solve these problem in the polynomial order, the solutions of such problems can be satisfied in $O(n^c)$.

About your question, first, you should see your problem has exponential space and cannot be solved in polynomial order; after that, you have to investigate whether its solutions can be satisfied in polynomial time or not. An easy way to model this is to consider the boolean table and an expression to satisfy. Your problem can be considered as the boolean equation and different possibilities of variables can be possible solutions that you may want to verify.


Reinforcement Learning over an MDP that is actually a POMDP Look at Breakout:

We know that the underlying world behaves like an MDP, because for the evolution of the system it just need to know which is the current state, i.e. position, speed and speed direction of the ball, positions of the bricks and the paddle etc. But considering only single frames as the state space we have a POMDP because we lack informations about the dynamics [1], [2].

Question:
What could happen if we wrongly assume that the POMDP is a MDP and do reinforcement learning with this assumption over the MDP?

Obviously the question is more general, not limited to Breakout and Atari games.

[1] https://arxiv.org/pdf/1507.06527.pdf
[2] https://storage.googleapis.com/deepmind-media/dqn/DQNNaturePaper.pdf

Question: What could happen if we wrongly assume that the POMDP is a MDP and do reinforcement learning with this assumption over the MDP?

It depends on a few things. The theoretical basis of reinforcement learning needs the state descriptions to have the Markov property for guarantees of convergence to optimal or approximately optimal solutions. The Markov property is a requirement that the state defines 100% of the controllable variation of reward and next state (given the action) - the rest must be purely stochastic.

An MDP can be "nearly Markov", and a lot of real-world physical systems are like that. For instance, pole-balancing and acrobot tasks can be implemented as physical systems using motors, wheels, joints etc. In those real systems, there are limits to accuracy of measurement of the state, and many hidden variables, such as variable temperature (affecting length of components), friction effects, air turbulence. Those hidden variables taken strictly by formal definition would make the system a POMDP. However, their influence compared to the key state variables is low, and in some cases effectively random from the perspective of the agent. In practice RL works well in the real physical systems, despite state data being technically incomplete.

In Atari games using multiple frame images as states, there are varying degrees of to which those states are already non-Markovian. In general a computer game's state may include many features that are not displayed on the screen. Enemies may have health totals or other hidden state, there can be timers controlling appearance of hazards, and in a large number of games the screen only shows a relatively small window into the total play area. However, the Deep Mind DQN network did well on a variety of scrolling combat and platform games.

One game where DQN did notably badly - no better than a default random player - was Montezuma's Revenge. Not only does that platform puzzler game have a large map to traverse, but it includes components where state on one screen affects results on another.

It is hard to make a general statement about where an MDP with missing useful state information would benefit from being treated as a POMDP more formally. Your question is essentially the same thing expressed in reverse.

The true answer for any non-trivial environment would be to try an experiment. It is also possible to make some educated guesses. The basis for those guesses might be the question "If the agent could know hidden feature x from the state, how different would expected reward and policy be?"

For the breakout example using each single frame as a state representation, I would expect the following to hold:

Value estimates become much harder since seeing the ball next to a brick - compared to seeing a ball progressively get closer to a brick over 4 frames - gives much less confidence that it is about to hit that brick and score some points.
It should still be possible for the agent to optimise play, as one working strategy is to position the "bat" under the ball at all times. This will mean less precise control over angle of bounces, so I would expect it to perform worse than the four-frame version. However, it should still be significantly better than a default random action agent. A key driver for this observation is that seeing the ball close to the bottom of the screen, and not close to the bat, would still be a good predictor of a low expected future reward (even averaged over chances of ball going up vs going down), hence the controller should act to prevent such states occurring.


To what extent can artificially intelligent agents reliably predict trends in financial markets? I've gotten curious about this topic and am wondering what the stack exchange community has to say about it. Also, does anyone know of any professors/researchers who have published papers pertaining to this?

This is a highly relevant question as market trends have become more emphasized over the fundamentals of individual companies, and algorithmic trading has proven to be quite effective, particularly in areas such as high-frequency micro-trading.

This 2013 Forbes article estimated nearly 80% of stock trading volume in the U.S. is conducted by automated trading systems.  More recently, Bloomberg published an article on the subject: The U.S. Stock Market Belongs to Bots.

The massive adoption of algorithmic trading in a fairly short span is a pretty good indicator that these systems are effective.  However,

My feeling with any predictive system is that it works until it doesn't

Imperfect and Incomplete information are persistent in real-world scenarios.  The implication is that there are always unknown factors that can diminish quality of analysis, and even lead to disastrous outcomes.

No matter how complex an algorithm becomes, reality is more complex, (at least for the foreseeable, pre-singularity future;)

Adding to that, systems in general tend to be imperfect, and when engaged in massively repetitive functions, small errors can grow exponentially and produce catastrophic effects.

"Flash Crashes" are a recent phenomena, and a result of algorithmic, high-frequency, black-box trading.


In variational autoencoders, what does p(x|z) mean? If $x \sim \mathcal{N}(\mu,\,\sigma^{2})$, then it is a continuous variable, and therefore $P(x) = 0$ for any x. One can only consider things like $P(x<X)$ to get a probability greater than 0.
So what is the meaning of probabilities such as $P(x|z)$ in variational autoencoders? I can't think of $P(x|z)$ as meaning $P(x<X|z)$, if $x$ is an image, since $x<X$ don't really make sense (all images smaller than a given one?)

Whilst you're right that for any continuous distribution $P(X = x) = 0 \;; \forall x \in \mathcal{X}$ where $\mathcal{X}$ is there support of the distribution, they are not referring to probabilities here, rather they are referring to density functions (though this should really be denoted with a lower case $p$ to avoid confusion such as this).
$p(x|z)$ is a conditional distribution, which is also allowed in the continuous case -- you can also 'mix and match', i.e. $x$ could be continuous and $z$ could be discrete, and vice-versa.
In the paper, all the authors are meaning when they write $p(x|z)$ is the density of $x$ conditioned on $z$; in VAE's with an image application this is the conditional density of the image $x$ given your latent vector $z$.


Can an artificial intelligence be unbeatable at simple games? There are (two-players, perfect information) combinatorial games for which, at any configuration of the game, a winning move (if there is one) can be quickly computed by a short program. This is the case of the following game that starts with a bunch of matches and each player alternatively removes 1,2 or 3 matches, until the player that removes the last one wins.
This is also the case of the Nim game.

On the other hand, understanding the winning strategy of games like Go or chess seems hopeless.
However, some machine-learning based programs like alphaGo zero are able "learn the strategy" of complex games, using as input data only the rules of the game. I don't really know how these algorithms work but here is my vague question:

For simple game like Nim, can such an algorithm be able to actually find a winning move in any winning configuration of the game ?

The number of configurations of Nim is infinite, but the algorithm will consider during its "training" only a finite number of configurations. It seems imaginable that if this training phase is long enough, then the program will be able to capture the winning strategy, like a human would do.

There is actually a github project about 'solving' Nim that implements certain type of Q-learning reinforcement algorithm (described in undergraduate thesis of Erik Jarleberg (Royal Institute of Technology) entitled "Reinforcement learning on the combinatorial game of Nim") that supposedly finds that optimal strategy lying down there inside in the game that also human can find out.

The project uses a code in Python, which is 'only' 114 lines of code, so it can be run on your own machine if you got interested and want to test it. Github page tells also:

  There is a known optimal strategy for the game which is credited to Charles L. Bouton of Harvard which can serve as a benchmark for evaluating the performance on our Q-learning agent.

(That is the quote I refer for it getting enough good / optimal results).

Q-learning is part of the reinforcement learning algorithm family where the so called agent learns how to play the game by getting rewards from its actions and it tries to maximize the amount of total reward with following strategy:

  The goal of the agent is to maximize its total reward. It does this by adding the maximum reward attainable from future states to the reward for achieving its current state, effectively influencing the current action by the potential future reward. This potential reward is a weighted sum of the expected values of the rewards of all future steps starting from the current state.

Quote and more information in Q-learning Wikipedia-page.

For further interest on easy games tackled with Reinforcement Learning, please have a look on following paper: Playing Atari with Deep Reinforcement Learning. There is also an Udemy class on that paper and its findings.


Is it feasible to train a Machine Learning Model (with image inputs) in an average personal computer? There are lots of examples of machine learning systems that can recognize objects and extract other information from images with very high precision. To train the models of such systems is necessary (I guess) a computer with a lot of computational power.

My question is: For a system with images as inputs, depending on the complexity of the problem, is feasible to train a model in an average laptop? It will take to much time?

I know the time taken to train any machine learning model will be a function of lots of variables. I really don't expect a quantitative answer here, I just want to know whether I will be forced to upgrade my computer to develop and train machine learning models that have images as inputs.

You may play around on an average laptop but training will be very slow and you will be limited on the size of your model.
Once you try to build something more serious you will run out of memory very fast. A system with a GPU is recommended if you want to really do things like image recognition. If you buy something I would not go for any GPU with less than 8 GB memory and for an X99 motherboard to keep the flexibility to add a second GPU on full speed later if needed.
Don't mix up image recognition running on phones etc. Those are trained models and actually just being applied that is much easier process. Training is way much more expensive.


Is the Decoder mask (triangular mask) applied only in the first decoder block, or to all blocks in Decoder? The Decoder mask, also called "look-ahead mask", is applied in the Decoder side to prevent it from attending future tokens. Something like this:
[0, 1, 1, 1, 1]
[0, 0, 1, 1, 1]
[0, 0, 0, 1, 1]
[0, 0, 0, 0, 1]
[0, 0, 0, 0, 0]

But is this mask applied only in the first Decoder block? Or to all its blocks?

The masking should be applied to all Decoder blocks, otherwise in some blocks, past words can attend to future words, which would be cheating during training.
This is reflected in The Annotated Transformer as well. Notice that in the Decoder class, the forward function applies the same mask to each layer of the decoder:
class Decoder(nn.Module):
    "Generic N layer decoder with masking."
    def __init__(self, layer, N):
        super(Decoder, self).__init__()
        self.layers = clones(layer, N)
        self.norm = LayerNorm(layer.size)

    def forward(self, x, memory, src_mask, tgt_mask):
        for layer in self.layers:
            x = layer(x, memory, src_mask, tgt_mask)
        return self.norm(x)


How is $\Delta$ updated in true online TD($\lambda$)? In the RL textbook by Sutton & Barto section 7.4, the author talked about the "True online TD($\lambda$)". The figure (7.10 in the book) below shows the algorithm.

At the end of each step, $V_{old} \leftarrow V(S')$ and also $S \leftarrow S'$. When we jump to next step, $\Delta \leftarrow V(S') - V(S')$, which is 0. It seems that $\Delta$ is always going to be 0 after step 1. If that is true, it does not make any sense to me. Can you please elaborate on how  $\Delta$ is updated?

Let us denote the state we are in at time $t$ by $S_t$. Then at iteration $t$ we create a placeholder $V_{old} = V(S_{t+1})$ for the state we will transition into. We then update the value function $V(s) \; \forall s \in \mathcal{S}$ - i.e. we update the value function for all states in our state space. Let us denote this updated value function by $V'(S)$.

At iteration $t+1$ we calculate $\Delta = V'(S_{t+1}) - V_{old} = V'(S_{t+1}) - V(S_{t+1})$, which does not necessarily equal 0 because the placeholder $V_{old}$ was created using the value function before the last update.


To what does the number of hidden layers in a neural network correspond? In a neural network, the number of neurons in the hidden layer corresponds to the complexity of the model generated to map the inputs to output(s).  More neurons creates a more complex function (and thus the ability to model more nuanced decision barriers) than a hidden layer with less nodes.

But what of the hidden layers?  What do more hidden layers correspond to in terms of the model generated?

I believe, this post and this post written by me well address your question.

EDIT

In fact, this is a very interesting question that you ask. It deserves some more explanation.

1. Fully connected networks

The more layers you add, the more "nonlinear" your network becomes. For instance, in the case of two spirals problem, which requires a "highly nonlinear separation", the first known architecture to solve the problem was pretty advanced for that time: it had 3 hidden layers and also it had skip-connections (very early ResNet in 1988). Back then, computing was a way less powerful and training methods with momentum were not known. Nevertheless, thanks to the multilayer architecture, the problem was solved. Here, however, I was able to train a single-hidden layer network to solve the spirals problem using Adam.

2. Convolutional nets (CNNs)

An interesting partial case of neural networks are CNNs. They restrict the architecture of the first layers, known as convolutional layers, so that there is a much smaller number of trainable parameters due to the weights sharing. What we have learned from computer vision, moving towards the end of CNNs layers, their receptive fields become larger. That means that the subsequent CNN layers "see" more than their predecessors. Conceptually, first CNN layers can recognize simpler features such as edges and textures, whereas final CNN layers contain information about more abstract objects such as trees or faces.

3. Recurrent nets (RNNs)

RNNs are networks with layers which receive some of their outputs as inputs. Technically, a single recurrent layer is equivalent to an infinite (or at least large) number of ordinary layers. Thanks to that recurrence, RNNs retain an internal state (memory). Therefore, it is much more difficult to answer your question in the case of recurrent nets. What is known, due to their memory, RNNs are more like programs, and thus are in principle more complex than other neural networks. Please let me know if you find an answer to your question in the last case.

To conclude, the higher number of hidden layers may help to structure a neural network. Thanks to the recent developments such as ResNets and backpropagation through time, it is possible to train neural networks with a large number of hidden layers.


How to incorporate a symmetry constraint in the loss function to train a CNN? I have a task of extremely sparse binary segmentation, i.e. the segmentation mask contains either 0 or 1, and there are ~95% zeros and only ~5% ones. I use the focal loss to address the sparseness (which is equivalent in my case to imbalances). I have another piece of information that I want to incorporate in the loss term.
The desired output is always symmetric over the diagonal. I was searching for a way to use this information in the loss, but I couldn't find a solution. How would I do this?
For some example of the symmetry in the segmentation maps, I added an arrow to show the axis of symmetry:

If you know it is symmetric, then you could do a couple things.

Zero out a half.

Don't bother learning both halves of the image. Just put a zero mask over the upper or lower half of the output matrix and just have the network regress the other half. Just don't make the network do more work than it needs to do.

Learn both, but add symmetric loss

In your case, it looks like you could create two loss functions added together.
$focal(x, y) + focal(x^T, y)$
This will help the network learn both halves equally.

L1 between $x$ and $x^T$

This might be silly but adding a Huber loss between $x$ and $x^T$ might help promote symmetry, but I'm not as much if a fan of it. Personally I'm more partial to (1).

Combine (1) and (2)

You could take the loss function of (2) at train time but at test time just use whatever half had better metrics and copy that to the bottom half.

Post Processing

Add a post processing custom layer that takes the average of the two halves so that you guarantee symmetry.
$x' = \frac{1}{2}(x + x^T)$
Then do your normal focal loss. Personally I like this the best since it always guarantees a symmetric output and is a pretty easy custom layer.


What are the most common deep reinforcement learning algorithms and models apart from DQN? Recently, I have completed Atari Breakout (https://arxiv.org/pdf/1312.5602.pdf) with DQN.

Similar to DQN, what are the most common deep reinforcement learning algorithms and models in 2020? It seems that DQN is outdated and policy gradients are preferred.

There are several common deep reinforcement algorithms and models apart from deep Q networks (or deep Q learning). I will list some of them below (along with a link to the paper that introduced them), but note that some of these may not be state-of-the-art (at least, not anymore, and it's likely that all of these will be replaced in the future).

Double DQN (DDQN) (2015)
Duelling DQN (2015)
Trust Region Policy Optimization (TRPO) (2015)
Deep Deterministic Policy Gradient (DDPG) (2016)
Asynchronous Advantage Actor-Critic (A3C) (2016)
Hindsight Experience Replay (HER) (2017)
Proximal policy optimization (PPO) (2017)
Twin Delayed Deep Deterministic policy gradient algorithm (TD3) (2018)
Soft Actor-Critic (SAC) (2018)

For an exhaustive overview of deep RL algorithms and models, maybe take a look at this pre-print Deep Reinforcement Learning (2018) by Yuxi Li.


Why do authors track $\gamma_t$ in Prioritized Experience Replay Paper? In the original prioritized experience replay paper, the authors track $\gamma_t$ in every state transition tuple (see line 6 in algorithm below):

Why do the authors track this at every time step?  Also, many blog posts and implementations leave this out (including I believe the OpenAI implementation on github).

Can someone explain explicitly how $\gamma_t$ is used in this algorithm?

Note: I understand the typical use of $\gamma$ as a discount factor. But typically gamma remains fixed. Which is why I’m curious as to the need to track it.

In some cases we may wish to have a discount factor $\gamma_t$ which depends on time $t$ (or depends on state $s_t$ and/or action $a_t$, leading to an indirect dependence on time $t$). Indeed we do not usually do this, but it does happen sometimes.

I guess that, from a theoretical point of view, it was very easy of the authors to make their algorithm more flexible/general and also support this (somewhat rare) case of time-varying discount factor. If it had been very complicated for them to support this option, they may have chosen not to; but if it's trivial to do so, well, why not?

Practical implementations will often indeed ignore that possibility if they're not using it, and can avoid including $\gamma_t$ values in the replay buffer altogether if it is known to be a constant $\gamma_t = \gamma$ for all $t$. As far as I can see, in the experiments discussed in this paper they also only used a fixed, constant $\gamma$.


How to approach this handwritten digit recognition? I have multiple pictures that look exactly like the one below this text. I'm trying to train CNN to read the digits for me. Problem is isolating the digits. They could be written in any shape, way, and position that person who is writing them wanted to. I thought of maybe training another CNN to recognize the position/location of the digits, but I'm not sure how to approach the problem. But, I need to get rid of that string and underline. Any clue would be a great one. Btw. I would love to get the 28x28 format just like the one in MNIST.

Thanks up front.

I think one approach you can try to segment the digits and Connected Components Labeling (https://en.wikipedia.org/wiki/Connected-component_labeling).
With it, you'll end up with a label for each letter and then you can try to find the convex hull of the letter.
After that, just crop a square for each convex hull and input it to your CNN.
Notice that it will only if there is at least one pixel between the letters...


What was the average decision speed pf Alpha Zero in the recent Stockfish match? The match got a lot of press, and I doubt anyone is surprised that Alpha Zero crushed Stockfish.

See: AlphaZero Destroys Stockfish in 100 Game Match

To me, what's really salient is that "much like humans, AlphaZero searches fewer positions that its predecessors. The paper claims that it looks at "only" 80,000 positions per second, compared to Stockfish's 70 million per second."

For those who remember Matthew Lai's GiraffeChess:

  However, it is interesting to note that the way computers play chess is very different from how
  humans play. While both humans and computers search ahead to predict how the game will go on,
  humans are much more selective in which branches of the game tree to explore. Computers, on the
  other hand, rely on brute force to explore as many continuations as possible, even ones that will be
  immediately thrown out by any skilled human. In a sense, the way humans play chess is much more
  computationally efficient - using Garry Kasparov vs Deep Blue as an example, Kasparov could not
  have been searching more than 3-5 positions per second, while Deep Blue, a supercomputer with
  480 custom ”chess processors”, searched about 200 million positions per second 1 to play at
  approximately equal strength (Deep Blue won the 6-game match with 2 wins, 3 draws, and 1 loss).

  How can a human searching 3-5 positions per second be as strong as a computer searching 200
  million positions per second? And is it possible to build even stronger chess computers than what
  we have today, by making them more computationally efficient? Those are the questions this
  project investigates.

[Lai was tapped by DeepMind as a researcher last year]

But what I'm interested in at the moment is the decision speed in these matches:

- What was the average time to make a move in the AlphaZero vs. Stockfish match?

No, ALL computer chess experts were surprised about the outcomes of the match. If you require references, please start a new question.
Your question is simple...

https://arxiv.org/pdf/1712.01815.pdf
... We evaluated the fully trained instances of AlphaZero against Stockfish, Elmo and the previous
version of AlphaGo Zero (trained for 3 days) in chess, shogi and Go respectively, playing
100 game matches at tournament time controls of one minute per move ...

One minute per move. Stockfish would use the one minute if there is more than a single legal move, otherwise it'd move immediately. So the average time for a move for Stockfish is about 57s to 60s.
There is no source code for AlphaZero. However, it's hard to believe the system wouldn't take advantage of the minute it was given. The expected time for AlphaZero should be also 57s to 60s.


How should I get started with artificial intelligence? What is the mathematical background required to start learning AI? What else should I also learn?

Start with Andrew Ng's introduction to Machine Learning course on Coursera.
There are not many prerequisites for that course, but you will learn how to make some useful things. And, more importantly, it will clearly show you which subjects you need to learn next.


Can we ever achieve hypercomputation using recurrent neural networks? It is proved that a recurrent neural net with rational weights can be a super-Turing machine. Can we achieve this in practice ?

I presume the proof the OP is referring to can be found in this monograph by Hava Siegelmann?

In his article 'The Myth of Hypercomputation', the eminent computer scientist Martin Davis explains (p8-9) that there is nothing 'super Turing' about this formulation.

EDIT: It's looking like the claim about rational weights being super-Turing is made in this  more recent paper by Siegelmann, which introduces an additional assumption of plasticity, i.e. that weights can be dynamically updated.


Is this referring to the true underlying distribution, or the distribution of our sample? I am currently studying the paper Learning and Evaluating Classifiers under Sample Selection Bias by Bianca Zadrozny. In the introduction, the author says the following:

One of the most common assumptions in the design of learning algorithms is that the training data consist of examples drawn independently from the same underlying distribution as the examples about which the model is expected to make predictions. In many real-world applications, however, this assumption is violated because we do not have complete control over the data gathering process.
For example, suppose we are using a learning method to induce a model that predicts the side-effects of a treatment for a given patient. Because the treatment is not given randomly to individuals in the general population, the available examples are not a random sample from the population. Similarly, suppose we are learning a model to predict the presence/absence of an animal species given the characteristics of a geographical location. Since data gathering is easier in certain regions than others, we would expect to have more data about certain regions than others.
In both cases, even though the available examples are not a random sample from the true underlying distribution of examples, we would like to learn a predictor from the examples that is as accurate as possible for this distribution. Furthermore, we would like to be able to estimate its accuracy for the whole population using the available data.

It's this part that I am confused about:

In both cases, even though the available examples are not a random sample from the true underlying distribution of examples, we would like to learn a predictor from the examples that is as accurate as possible for this distribution.

What exactly is "this distribution"? Is it referring to the true underlying distribution, or the distribution of our sample (which, as was said, is not necessarily a "good" reflection of the underlying distribution, since it is not a random sample)?

In both cases, even though the available examples are not a random sample from the true underlying distribution of examples, we would like to learn a predictor from the examples that is as accurate as possible for this distribution.

"The true underlying distribution" is the closest "distribution" that is explicitly mentioned as such in the part of the text preceding the phrase "this distribution", so that's what it's referring to. For clarity, I've put the two things that are "the same" in bold in the above quote.


How can I make my network treat rotations of the input equally? I'm attempting to program my own system to run a neural network. To reduce the number of nodes needed, it was suggested to make it treat rotations of the input equally.

My network aims to learn and predict Conway's Game of Life by looking at every square and its surrounding squares in a grid, and giving the output for that square. Its input is a string of 9 bits:

The above is represented as 010 001 111.

There are three other rotations of this shape however, and all of them produce the same output:

My network topology is 9 input nodes and 1 output node for the next state of the centre square in the input. How can I construct the hidden layer(s) so that they take each of these rotations as the same, cutting the number of possible inputs down to a quarter of the original?

Edit:

There is also a flip of each rotation which produces an identical result. Incorporating these will cut my inputs by 1/8th. With the glider, my aim is for all of these inputs to be treated exactly the same. Will this have to be done with pre-processing, or can I incorporate it into the network?

If I understand well your single output node will be the next status of the square in the middle. You don't need to worry about the number of nodes in the hidden layers while you have sufficient resources to train the model. This problem is very easy to learn for a neural network so no size concerns.

You need to do supervised training that means you need to feed in input data and the matching expected output. You need to be sure that in your training data all 4 rotations are assigned to the same output. This way your network should learn to treat all of these the same way.

You made me curious so I tried myself. My solution could learn 100% correct in about 20 epochs running within a few seconds on my old laptop. I only slightly changed the output to be categorical either [0,1] or [1,0] but that gives the same result that you are looking for. Just for reference here is the code written in python:

from keras.models import Sequential
from keras.layers import Input, Dense
from keras.models import Model
from keras import optimizers
from keras.utils.np_utils import to_categorical
import helper

x_,y_ = helper.fnn_csv_toXY("conway.csv","output",False)
y_binary = to_categorical(y_)

model = Sequential()
model.add(Dense(100, activation='relu', kernel_initializer='glorot_uniform',input_shape =(9,)))
model.add(Dense(20, activation='relu', kernel_initializer='glorot_uniform'))
model.add(Dense(2, activation='softmax'))
adam=optimizers.Adam()
model.compile(optimizer=adam,
              loss='categorical_crossentropy',
              metrics=['acc'])
model.fit(x_, y_binary, epochs=100)


Traveling salesman problem variant: which algorithm to choose? I have an industrial problem which I'm trying to cast as a Traveling Salesman problem (TSP) in 3D euclidian space. There are physical limitations which implies that some subpaths may or may not be valid based on simple rules.

What algorithm is best to deal with the TSP given that there are rules/model/constraints?

It could be done with Genetic algorithms for example, but the only way i see how to incorporate those rules is by including them somehow within the fitness function. But i feel there should be more suitable approaches.

Would reinforcement Q-learning or other algorithms be more appropriate for a rule-based euclidian TSP?

If you have lots of rules then you should be able to come up with lots of heuristics for A*. If I were you I would try A* first, and come up with as many heuristics as I could. You can also use Deep Q learning. I don't think you want to just throw coordinates into a Deep Q network. You want a simpler representation. I would probably give the nodes symbolic identifiers like sparse matrices, integers or sigmoids, and then craft the loss function to reflect the actual cost of moving from node to node and ascribe a high cost to illegal moves.


How to train for own dataset really really fast while debugging How to train darkflow for my custom object really really fast during debugging in quad core PC and without GPU? (Can I train with about 10 images and test with only those images, just to check if all convolutions are working as expected. And with 20 epoch?). There is only once class and all license plates are similar in pattern with varying in angle and its digits in plate.
I am using tiny yolo config and weight. So, what all parameters I should tune in yolo based .cfg file to do it? I feel if TV like object can be detected with same training weight and config for tiny yolo then license plate too.

Overview:
I am training for object, license plate, with darkflow. I tried with about 100 custom datasets I had created. This is only for initial POC, actual implementation will have more number of images. And upon testing with test images with trained graph, object is not highlighted rather highlighting squares are shown at random location within image and random in number, starting with 2-3 square boxes to lot many number. But non of those were highlighting to the actual license plate object. It took me 20 hour of training time to verify it. I used training images for testing as well, and also a plane black screen images to test what's going on. But highlighting squares are still random in number and location even on blank screen image.

Assuming that you are using DarkFlow, you can train your model with --train flag and don't use --gpu flag since you don't want to use the GPU memory. And yes it's possible to train with just 10 images (basically over-fitting) just to confirm the working of your model. Place those 10 images in train/images directory and the annotations of these images in train/annotations directory (in PASCAL VOC format).
For faster performance, use pre-trained weights file for tuning your model so that you don't need to train for initial same layers (and since you just want to detect a single object - license plates - use tiny-yolo-voc.weights.
Further more, use --epoch flag to reduce the number of epochs.

flow --model cfg/tiny-yolo-voc-new.cfg --load bin/tiny-yolo-voc.weights --train --annotation train/Annotations --dataset train/Images --epoch 20
Change the number of classes and the number of filters as defined in the readme first. Then you can change the height and width of the input image the model will be trained on. Higher resolution for more accuracy while lower resolution for higher performance and speed. You can also change the activation functions for the layers, but I recommend leaving them unchanged.


What are the most pressing fundamental questions and gaps in AI research? White House published the information about AI which requests mentions about the most important research gaps in AI that must be addressed to advance this field and benefit the public.

What are these exactly?

According to IBM Research organization in the response to the White House as part of preparing for the future of Artificial Intelligence, AI depends upon many long-term advances, not only from AI researchers but from many interdisciplinary teams of experts from many disciplines, including the following challenges:

Machine learning and reasoning.
Currently, AI systems use supervised learning using a huge amount of dataset of labelled data for training. This is very different to how humans learn by creating concepts, relationship, common sense reasoning which gives the ability to learn much without too much data. Therefore machine learning with common-sense reasoning capabilities should be researched further more.

Decision techniques.
Current AI-based systems have very limited ability for making decisions, therefore new techniques must be developed (e.g. modeling systemic risks, analyzing tradeoffs, detecting anomalies in context, analyzing data while preserving privacy).

Domain-specific AI systems.
The current AI-based system is lack of abilities to understand the variety of domains of human expertise (such as medicine, engineering, law and many more). The systems should be able to perform professional-level tasks such as designing problems, experiments, managing contradictions, negotiating, etc.

Data assurance and trust.
The current AI-based systems require huge amounts of data and their behaviour directly depends on the quality of this data which can be biased, incomplete or compromised. This can be expensive and time-consuming especially where it is used for safety-critical systems which potentially can be very dangerous.

Radically efficient computing infrastructure.
The current AI-based systems require unprecedented workloads and computing power which require the development of new computing architectures (such as neuromorphic).

Interpretability and explanations.
For people to follow AI suggestions, they need to trust systems, and this is only when they are capable of knowing users' intents, priorities, reasoning and they can learn from their mistakes. These capabilities are required in many business domains and professionals

Value alignment and ethics.
Humans can share the common knowledge of how the world function, the machine cannot. They can fail by having unintended and unexpected behaviour only because humans did not specify the right goals for them or they omitted essential training details. The systems should be able to correct the specification of the goals and avoid unintended and undesired consequences in the behaviour.

Social AI.
The AI-based systems should be able to work closely with humans in their professional and personal life, therefore they should have significant social capabilities because they can impact our emotions and our decision-making capabilities. Also, sophisticated natural language capabilities will need to be developed to allow a natural interaction and dialogue between humans and machines.

Source: Fundamental questions in AI research, and the most important research gaps (RFI questions 5 and 6). (link no longer available)


Interpretation of inverse matrix in mean calculation in Gaussian Process The formula for mean prediction using Gaussian Process is $k(x_*, x)k(x, x)^{-1}y$, where $k$ is the covariance function. See e.g. equation 2.23 (in chapter 2) from Gaussian Processes for Machine Learning (2006) by C. E. Rasmussen & C. K. I. Williams.

Oversimplifying, the mean prediction of the new point $y_*$ is the weighted average of previously observed $y$, where the weights are calculated by the $k(x_*,x)$ and normalized by $k(x,x)^{-1}$.

Now, the first part $k(x_*, x)$ is easy to interpret. The closer the new data point lies to the previously observed data points, the greater their similarity, the higher will be the weight and impact on the prediction.

But how to interpret the second part $k(x, x)^{-1}$? I presume this makes the weight of the points in the clusters greater than the outliers. Am I correct?

Mathematical Interpretation
Note that equation (2.23) is simply calculating the conditional distribution of equation (2.21) and then finding the mean. Your question reduces to:
"Given normal variables $X$ and $Y$, why is $\mathbb{E}[Y|X] = \mu_y + Cov(X, Y)Cov(Y, Y)^{-1}(x - \mu_x)$? (note: in the book, $\mu_x = \mu_y = 0$)
Deriving the conditional probability mean is complicated (see The Bivariate Normal Distribution, page 3). A more intuitive look can be seen in the first graph in this page.
Here, the mean of $Y|X$ is linear in what value $X$ takes. The line starts at the intercept $\mu_y$, and increases with slope $\rho (\frac{\sigma_y}{\sigma_x}) = \frac{Cov(X,Y)}{\sigma_x \sigma_y}\left(\frac{\sigma_y}{\sigma_x}\right)=\frac{Cov(X,Y)}{\sigma_x^2} = \frac{Cov(X,Y)}{Var(x)} = \frac{Cov(X,Y)}{Cov(X,X)}$. So the mathematical interpretation of $Cov(X,Y)Cov(X,X)^{-1}$ is that it is the slope of the relationship between the mean of $Y|X$ and the value of $x$ that you are given. As you are given a higher value $x$, say $x + \delta$, then the mean of $Y|X$ raises by $Cov(X,Y)Cov(X,X)^{-1}\delta$.
Why is there even a $Cov(X,Y)Cov(X,X)^{-1}$ term there? For some reason, multiplying $Y|X$ by a $Cov(X,Y)Cov(X,X)^{-1}$ term makes $Y|X$ completely independent of $X$ (which makes sense as the definition of "conditional probability", because you are already given a value of $X$). This is just a mathematical property, I don't know if there's an intuitive explanation as to why.
Human Interpretation
In case your post just want an intuition as to why there is a $Cov(X,X)^{-1}$ in the prediction of a Gaussian process (and ignoring the conditional probability fluff), I don't think there's a real basis for this, it would only be coincidental as the authors simply used the conditional probability mean formula, but I would guess $Cov(X,X)^{-1}$ somehow normalizes the values of covariance matrix $Cov(X_*, X)$.
For example, if the training set $X$ has a lot of outliers and therefore extremely high variance (e.g. all non-diagonal entries in millions), then it is very likely that $Cov(X_*, X)$ would also be extremely high as $X_*$ follows the same distribution as $X$ (unless each data in $X_*$ matches the exact same variance in $X$). It doesn't make sense to multiply $y$ by millions though, as $y$ is already a somewhat decent estimator/prior.
It makes more sense to normalize $Cov(X_*, X)$ by dividing it with the training data variance $Cov(X,X) = Var(X)$ so that the ratio $Cov(X_*, X)Cov(X, X)^{-1}$ tends to be more closer to 1 when $X_*$ follows the same distribution as $X$ (which should be the case). If the ratio is exactly 1, then $X_*$ has the exact same distribution as $X$, so you just return the prior estimate $y$. If the ratio is far away from $1$, then the test set distribution is wildly different than the training set distribution, so you return a number far away from $y$.


Which library / algorithm should I use to recognise a specific facial expression (C#) I have a friend who, after having a stroke, now has locked-in syndrome who and can now only move his eyes and eyelids. I've been working on some C# software (written in Unity3D) to help him to communicate, the ones he has tried so far have not been successful due to the level of his disability.

I am in the process of implementing DLib for facial feature recognition. What I would like to do is to ask the user to repeatedly alternate between a neutral face and a face that is indicating. I don't want to limit the indication gesture specifically to winking/blinking/opening eyes wide, etc as I want other people to be able to freely download the app and indicate in whatever way they find easiest due to their incapacity.

I could really do with some pointers as to how to have my app learn what the user's facial indicator is by comparing the facial landmarks of the two states.

At this point, I don't even know what it is I need to know. I don't know the names of any AI algorithms that might be relevant or anything.

I know posts like this are often closed for being "too vague", but due to the nature of the request, I'd appreciate it if it could be left open as any suggestions at all could potentially be life-changing.

OpenCV would be an excellent choice as I already have a licence for a library in Unity3D.

Since you are

A C# developer already
Just getting started and not sure where to go next

I would suggest trying the Emotion API which is now part of the general Face API. This has the benefit of being pre-trained on a very large dataset. You can perform 30,000 recognitions/month for free.


Is the discount not needed in a deterministic environment for Reinforcement Learning? I'm now reading a book titled as "Deep Reinforcement Learning Hands-On" and the author said the following on the chapter about AlphaGo Zero:

Self-play
In AlphaGo Zero, the NN is used to approximate the prior probabilities of the actions and evaluate the position, which is very similar to the Actor-Critic (A2C) two-headed setup. On the input of the network, we pass the current game position (augmented with several previous positions) and return two values. The policy head returns the probability distribution over the actions and the value head estimates the game outcome as seen from the player's perspective. This value is undiscounted, as moves in Go are deterministic. Of course, if you have stochasticity in the game, like in backgammon, some discounting should be used.

All the environments that I have seen so far are stochastic environments, and I understand the discount factor is needed in stochastic environment.
I also understand that the discount factor should be added in infinite environments (no end episode) in order to avoid the infinite calculation.
But I have never heard (at least so far on my limited learning) that the discount factor is NOT needed in deterministic environment. Is it correct? And if so, why is it NOT needed?

The motivation for adding the discount factor $\gamma$ is generally, at least initially, based simply in "theoretical convenience". Ideally, we'd like to define the "objective" of an RL agent as maximizing the sum of all the rewards it gathers; its return, defined as:

$$\sum_{t = 0}^{\infty} R_t,$$

where $R_t$ denotes the immediate reward at time $t$. As you also already noted in your question, this is inconvenient from a theoretical point of view, because we can have many different such sums that all end up being equal to $\infty$, and then the objective of "maximizing" that quantity becomes quite meaningless. So, by far the most common solution is to introduce a discount factor $0 \leq \gamma < 1$, and formulate our objective as maximizing the discounted return:

$$\sum_{t = 0}^{\infty} \gamma^t R_t.$$

Now we have an objective that will never be equal to $\infty$, so maximizing that objective always has a well-defined meaning.

As far as I am aware, the motivation described above is the only motivation for a discount factor being strictly necessary / needed. This is not related to the problem being stochastic or deterministic.

If we have a stochastic environment, which is guaranteed to have a finite duration of at most $T$, we can define our objective as maximizing the following quantity:

$$\sum_0^T R_t,$$

where $R_t$ is a random variable drawn from some distribution. Even in the case of stochastic environments, this is well-defined, we do not strictly need a discount factor.

Above, I addressed the question of whether or not a discount factor is necessary. This does not tell the full story though. Even in cases where a discount factor is not strictly necessary, it still might be useful.

Intuitively, discount factors $\gamma < 1$ tell us that rewards that are nearby in a temporal sense (reachable in a low number of time steps) are more important than rewards that are far away. In problems with a finite time horizon $T$, this is probably not true, but it can still be a useful heuristic / rule of thumb.

Such a rule of thumb is particularly useful in stochastic environments, because stochasticity can introduce greater variance / uncertainty over long amounts of time than over short amounts of time. So, even if in an ideal world we'd prefer to maximize our expected sum of undiscounted rewards, it is often easier to learn how to effectively maximize a discounted sum; we'll learn behaviour that mitigates uncertainty caused by stochasticity because it prioritizes short-term rewards over long-term rewards.

This rule of thumb especially makes a lot of sense in stochastic environments, but I don't agree with the implication in that book that it would be restricted to stochastic environments. A discount factor $\gamma < 1$ has also often been found to be beneficial for learning performance in deterministic environments, even if afterwards we evaluate an algorithm's performance according to the undiscounted returns, likely because it leads to a "simpler" learning problem. In a deterministic environment there may not be any uncertainty / variance that grows over time due to the environment itself, but during a training process there is still uncertainy / variance in our agent's behaviour which grows over time. For example, it will often be selecting suboptimal actions for the sake of exploration.


What AI applications exist to solve sustainability issues? The Sustainable Development Goals of the United Nations describe a normative framework which states what future development until 2030 should strive for. On a more abstract level a basic definition describes sustainable development as

  development that meets the needs of the present without compromising the ability of future generations to meet their own needs.

Alone through the consumption of energy, AI technologies already have a (negative) impact on sustainability questions.

What AI applications already exist, are researched or are at least thinkable from which sustainability would benefit?

The paper The role of artificial intelligence in achieving the Sustainable Development Goals (2020, published in Nature) should contain the information you're looking for.

In the introduction, the authors write

  Here we present and discuss implications of how AI can either enable or inhibit the delivery of all 17 goals and 169 targets recognized in the 2030 Agenda for Sustainable Development. Relationships were characterized by the methods reported at the end of this study, which can be summarized as a consensus-based expert elicitation process, informed by previous studies aimed at mapping SDGs interlinkages. A summary of the results is given in Fig. 1 and the Supplementary Data 1 provides a complete list of all the SDGs and targets, together with the detailed results from this work.

For example, as stated in the supplementary data, goal 1 is

  End poverty in all its forms everywhere

and the first target (1.1.) of goal 1 is

  By 2030, eradicate extreme poverty for all people everywhere, currently measured as people living on less than $1.25 a day

Then the authors suggest that, according to their studies, AI may act as an inhibitor or enabler (i.e. it may be used to fight poverty) for this target.

  We identified in the literature studies suggesting that AI may be an inhibitor for this target, due to the potential increase in inequalities which would hinder the achievement of this goal (1). Other references however identify AI as an enabler for this goal, in the context of using satellite data analysis to track areas of poverty and to foster international collaboration (2).

Therefore, techniques for satellite data analysis are one of the AI techniques that may be used to tackle suistainability issues.


How should I take into consideration the number of steps in the reward function? I am currently implementing the paper Active Object Localization with Deep Reinforcement Learning in Python. While reading about the reward scheme I came across the following:

Finally, the proposed reward scheme implicitly considers the number of steps as a cost because of the way in which Q-learning models the discount of future rewards (positive and negative).

How would you implement this "number of steps" cost? I am keeping track of the number of steps that have been taken, therefore would it be best to use an exponential functions to discount the reward at the current time step?
If anyone has a good idea or knows the standard in regard to this I would love to hear your thoughts.

How would you implement this "Number of Steps" cost?

What the paper is referring to is the reward discounting process which is a standard way of formulating RL problems, either continuous ones, or episodic ones where the goal is to complete a task in the least time (in the episodic version, a fixed cost per time step will also achieve this).
As such, this usually is implemented in the formulation of the value function calculations. The discount factor is usually represented as gamma, $\gamma$.
For Q-learning, the factor should be in the TD target calculation:
$$G_{t:t+1} = R_{t+1} + \gamma \max_{a'}Q(S_{t+1},a')$$
For Monte Carlo control, the factor appears more like this in the calculation of a return:
$$G_t = \sum_{k=0}^{T-t} \gamma^k R_{t+k+1}$$

would it be best to use an exponential functions to discount the reward at the current time step?

Essentially that is what normal discounting is - an exponential decay of future reward. But if you have implemented "normal" Q-learning from equations like those above, it should already be there.


Deep Ranking/Best way to classify book covers? I recently came across a paper on Deep Ranking. I was wondering whether this could be used to classify book covers as book titles. (For example, if I had a picture for the cover of the second HP book, the classifier would return Harry Potter and the Chamber of Secrets.)

For example, say I have a dataset of book covers along with the book titles in text. Could that data set be used for this deep ranking algorithm, or is there a much better way to approach my problem? I'm quite new to this whole thing, and this one of my first projects in this field.

What I'm trying to create is a mobile app where people can take a picture of a book cover, have an algorithm/neural net classify the title of the book, and then have some other algorithm connect that to the book's Goodreads page.

Thanks for the help!

I read some papers talked about it you can take a look on them maybe help you.

Deep Neural Network Learns to Judge Books by Their Covers

Classification of Book Genres By Cover and Title


Can a neural network be used to predict a sequence of integers based on dataset of previously produced random numbers? What i really want to do, is to predict an integer sequence of (5 numbers with values from 1 to 50) for example based on a big dataset of other 5 numbers sequences with same values range created by the same random number generator. I suppose there is a way to train based on the dataset and the program will find a pattern or based on the most common numbers predict the next number sequence. The more numbers will predict in the sequence correctly the better of course. Any help, directions and preferably python code would be greatly appreciated.

I recently read the following  can-a-neural-network-be-used-to-predict-the-next-pseudo-random-number and i am new to the AI field. The proposed code while it creates a sequence of 25 numbers it ends showing 20 numbers i do not understand why. It seems they try to do something similar if i understand correctly

I tried The code here can-a-neural-network-be-used-to-predict-the-next-pseudo-random-number

It shows always the same numbers no matter how many epochs and or iterations i do is that normal?
Is the last code close to what i want to accomplish?

Thanks in advance.

The post you linked to clearly states that pseudo random number cannot be predicted. Their randomness is made to be nearly perfect, and if you ever found a way to even predict a pseudo random number with 20% chance of correct, the security of the entire world would be vulnerable to attacks, as things ranges from cryptocurrency and secure data transfer is all protected by pseudo random number.