Which Openai Gym environment should I use to test a Temporal Difference RL Agent? Up to now, I have been using (my version of) open AI's code, with the suggested CartPole.

I have been using Monte Carlo methods, which, for cartpole, seemed to work fine.

Trying to move to temporal difference, Cartpole seems to fail to learn (with simple TD method) (or I stopped it too soon, but still unacceptable).

I assume that is the case because in Cartpole, for every timestamp, we get a reward of 1, which has very little immediate information about weather or not the action was good or not.

Which gym environment is the simplest that would probably work with TD learning?

by simplest I mean that there is no need for a large NN to solve it. No conv nets, no RNNS. just a few small layers of a fully connected NN, just like in cartpole. something I can train on my home cpu, just to see it starting to converge.

Cartpole will work just fine with a single-step TD method - e.g. Q learning - and a simple neural network. Configured well, a simple DQN model should solve it in a few minutes. You can also forgo the neural network and use discretised states, or tile coding etc.

What you cannot do is plug a neural network estimator into basic Q learning and make no other adjustments. That is because the bootstrap process in TD learning will create a runaway feedback driven from the random and incorrect return estimates in the initialised NN. You have to use something like DQN to get it to work at all. This is different to Monte Carlo approach, where you can generally just plug in a NN in place of a Q table.

The basics of DQN are:

Use experience replay. Store $(S, A, R, S')$ values from each step, and when training the NN, draw a small random batch (e.g. maybe 16 samples) from this table, re-estimate their Q values to calculate TD target e.g.  $R + \gamma\text{max}_{a'} Q(S',a')$ and use that to create a minibatch to train the NN
Use a "delayed" target network for estimating TD target. This can just be a snapshot of your learning network, taken once every N steps, where N is typically anything from 100 to 100000.

Further details of what configured well means varies from problem to problem. With DQN you will want to play with hyper parameters of experience history size, time to keep frozen copy of network (for bootstrap estimates), batch size of experience table samples, amount of exploration.

In CartPole though, there will be a wide range of acceptable hyperparameter values. So I suspect you have just plugged in a NN in place of the Q table and wonder why it doesn't work.

  I assume that is the case because in Cartpole, for every timestamp, we get a reward of 1, which has very little immediate information about weather or not the action was good or not.

Yes this is part of what makes some learning goals become harder challenges in sequential-decision control systems. However, this was also the case for your Monte Carlo method, and in general this relates to the credit assignment problem in RL.


How are weights updated in a genetic algorithm with neural network? Suppose an AI is to play the game flappy bird. And the fitness function is how long the bird has traveled before the game ends.

Would we have multiple neural networks initialized at the beginning with random weights (as in each bird has its own network) and then we determine the neural networks that have lasted the longest for the game and then we perform a selection of weights from the "better" neural networks followed by mutation? Those will then be used as the new weights of a brand new neural network (ie the offspring from two "better" neural networks?)?

If that is the case, does that mean there is no backpropagation because there isn't a cost function?

I found my answer in a different post: How to evolve weights of a neural network in Neuroevolution?. Note that the genetic algorithm is a subcategory of the neuroevolution algorithm. Short answer, my original thoughts were correct.


Is there any scientific/mathematical argument that prevents deep learning from ever producing strong AI? I read Judea Pearl's The Book of Why, in which he mentions that deep learning is just a glorified curve fitting technology, and will not be able to produce human-like intelligence.

From his book there is this diagram that illustrates the three levels of cognitive abilities:

The idea is that the "intelligence" produced by current deep learning technology is only at the level of association. Thus the AI is nowhere near the level of asking questions like "how can I make Y happen" (intervention) and "What if I have acted differently, will X still occur?" (counterfactuals), and it's highly unlikely that curve fitting techniques can ever bring us closer to a higher level of cognitive ability.

I found his argument persuasive on an intuitive level, but I'm unable to find any physical or mathematical laws that can either bolster or cast doubt on this argument.

So, is there any  scientific/physical/chemical/biological/mathematical argument that prevents deep learning from ever producing strong AI (human-like intelligence)?

Judea Pearl's 2018 comment on ACM.org, in his To, Build Truly Intelligent Machines, Teach Them Cause and Effect is piercing truth.

All the impressive achievements of deep learning amount to just curve fitting.

It may be less sensational and more technically correct to state that it is not, "Just curve fitting," but rather, "sophisticated surface fitting." Nonetheless, his general assessment indicates the need to look beyond tuning nonlinear functions to fit a surface in $\mathbb{R}^n$ and consider whether cognition is achievable with a deep network. The split in answers to this question is odd. We have two conflicting assertions, often strongly stated.

Artificial networks cannot perform logic.
Artificial networks are the best approach to AI.

How can rationality be excluded from the list of important human features of intelligence, which is what these two assertions are taken together would mean?
Is the human brain a network of sophisticated curve fitters?  Marvin Minsky's famous quote, "The brain happens to be a meat machine," was offered without proof, and neither a proof of his trivialization of the human brain nor a proof that the brain is beyond the reach of Turing computability has been offered since.
When you read these words, are your neural networks doing the following sequence of curve fits?

Edges from retinal rods and cones
Lines from edges
Shapes from lines
Letters from shapes
Linguistic elements from groups of letters
Linguistic structures from elements
Understanding from linguistic structures

The case is strong for the affirmation that the first five is a convergence mechanism on a model, and all the machine learning structure is just a method to fit the data to the model.
Those last two bullet items are where the paradigm breaks down and where many AI researchers and authors have correctly stated that machine learning has significant limitations when based solely on layers of multi-layer perceptrons and convolution kernels.  Furthermore, the last bullet item is grossly oversimplified in its current state, probably by orders of magnitude.  Even if Minsky is correct that a computer can perform what the brain does, the process of reading and understanding this paragraph could easily have a thousand different kinds of unique process components in patterns of internal workflow with massive parallelism.  Imaging technology indicates this probability.  We have computers modelling only the simplest peripheral layers.

Is there any scientific/mathematical argument that prevents deep learning from ever producing strong AI? — No. But there is no such argument that guarantees it either.

Other questions here investigate whether these sophisticated curve fitters can perform elements of cognition or reasoning.

Could a neural network detect primes?
Can neural networks be used to prove conjectures?

The totem of three in the question's image, seeing, doing, and imagining, is not particularly complete, accurate, or insightful.

There are at least five sensory paradigms in humans, not one
Doing preceded human senses by billions of years — bacteria do
Imagining is not a significantly higher process than scenario replay from models of past experience with some method to apply set functions to combine them and inject random mutations
Creativity may just be imagining in the previous bullet item followed by weeding out useless imagination results with some market-oriented quality criteria, leaving the impressive creative products that sell

The higher forms are appreciation, a sense of realities beyond the scope of scientific measurement, legitimate doubt, love, sacrifice for the good of others or humanity.
Many recognize that the current state of AI technology is nowhere near the procurement of a system that can reliably answer, "How can I make Y happen?" or "If I have acted differently, will X still occur?"
There is no mathematical proof that some combination of small curve fitting elements can or cannot achieve the ability to answer those questions as well a typical human being can, mostly because there is insufficient understanding of what intelligence is or how to define it in mathematical terms.
It is also possible that human intelligence doesn't exist at all, that references to it are based on a religious belief that we are higher as a species than other species.  That we can populate, consume, and exterminate is not actually a very intelligent conception of intelligence.
The claim that human intelligence is an adaptation that differentiates us from other mammals conflicts with whether we adapt well.  We have not been tested.  Come the next meteoric global killer with a shock wave of the magnitude of that of the Chicxulub crater's meteor, followed by a few and a thousand years of solar winter and we'll see whether it is our 160,000-year existence or bacteria's 4,000,000,000 year existence that proves more sustainable.  In the timeline of life, human intelligence has yet to prove itself significant as an adaptive trait.
What is clear about AI development is that other kinds of systems are playing a role along with deep learners based on the multi-layer perceptron concept and convolution kernels which are strictly surface fitters.
Q-learning components, attention-based components, and long-short term memory components are all strictly a surface fitter too, but only by stretching the definition of surface fitting considerably.  They have real-time adaptive properties and state, so they can be Turing complete.
Fuzzy logic containers, rules-based systems, algorithms with Markovian properties, and many other component types also play their role and are not surface fitters at all.
In summary, there are points made that have a basis in more than plausibility or a pleasing intuitive quality, however, many of these authors do not provide a mathematical framework with definitions, applications, lemmas, theorems, proofs, or even thought experiments that can be scrutinized in a formal way.


Is there a naming convention for network weights for multilayer networks? In the diagram below, although the flow of information happens from the input to output layer, the labeling of weights appears reverse. Eg: For the arrow flowing from X3 to the fourth hidden layer node has the weight labeled as W(1,0) and W(4,3) instead of W(0,1) and W(3,4) which would indicate data flowing from the 3rd node of the 0'th layer to the 4th node of the 1st layer.

One of my neural networks teachers did not emphasize on this convention at all. Another teacher made it a point to emphasize on it.

Is there a reason there is such an un-intuitive convention and is there really a convention?

When the system grows matrix notation is used, as a=Wx, being a (input to activation function in hidden layer) and x (values from input layer) column vectors, transpose of (a1,a2,...a_m) and (x1,x2,...,x_n), and W a m-by-n matrix of dimensions m rows and n columns. The standard way to denote matrix elements is w(i,j) where "i" is the row number and "j" column number:

(from wiki)

For this reason, the weight that applies to h4 from x3 is element in row 4 column 3 of the matrix W, that is, W(4,3) ( as your teachers advocates but with a sad lack of ability to explain ).

In your example:

Note: things are a few more complex when x1, x2, ... are itself vectors, but final conclusion is the same.

( PS: URGENT to allow latex notation on this stack exchange ! )


What are some examples of classical AI methods? All of the attention in the AI field over the past few years has been toward statistical AI, now that Machine Learning has been validated on hard problems.

However, I want to learn more about classical AI (also known as "symbolic AI" and "GOFAI--good old fashioned AI", and "expert systems".)

What are some examples of classical AI methods?

Please provide a brief description for each method listed.

If you're looking for AI systems:

Deep Blue
Eliza

If you're looking for specific algorithms:

Bayesian Networks
Decision Trees
Any search algorithm


TensorFlow estimator DNNClassifier fails to fit simple data The ready-to-use DNNClassifier in tf.estimator seems not able to fit these data:

X = [[1,2], [1,12], [1,17], [9,33], [48,49], [48,50]]
Y = [ 1,     1,      1,      1,      2,       3     ]

I've tried with 4 layers but it's fitting to 83% (=5/6 sampes) only:

hidden_units = [2000,1000,500,100]
n_classes    = 4

The sample data above are supposed to be separated by 2 lines (right-click image to open in new tab):

It seems stuck be cause of Y=2 and Y=3 are too close. How to change the DNNClassifier to fit to 100%?

Normalise your inputs.

Neural networks work poorly outside of relatively small numerical ranges on input. An ideal range is for each feature to be drawn from $\mathcal{N}(0,1)$ i.e. a Normal distribution with mean $0$ and standard deviation $1$. In your case, divide both parts of $\mathbf{x}$ by $25$ and subtract $1$ would probably suffice.

Your neural network architecture is completely overblown for the problem at hand. That may be because you were trying to force it to fit this data (and failing because of lack of normalisation). Try something more like: hidden_units = [20,10]


Why does the policy $\pi$ affect the Q value? From my understanding, the policy $\pi$ is basically how the agent acts (i.e. the actions it will take in each state).

However, I am confused about the Q value and how it is "affected" by a policy. This answer says

  $Q^\pi(s, a)$ is the action-value function. It is the expected return starting from state $s$, following policy $\pi$, taking action $a$. It's focusing on the particular action at the particular state.

From this, I infer that the $Q$ value (the action-value function) will be affected by the policy $\pi$. Why? So, why does the Q value change according to policy $\pi$?

Shouldn't the Q value be constant, because the same action taken in the same state will always give the same yield (and hence remain constantly good/bad)?

All the policy does is find out the max Q values and bases its policy on that information.

First of all, $Q_\pi(s, a)$ IS DEFINED AS the value (i.e. the expected return) of taking some action $a$ in some state $s$, AND THEN following some given policy $\pi$ (until e.g. the end of the game or your life). In other words, suppose that you take action $a$ in state $s$, AND THEN use the policy $\pi$ to behave in the world until you die, then $Q_\pi(s, a)$ would represent the value that you would obtain.

So, we are DEFINING $Q_\pi(s, a)$ in a certain way.  This is a DEFINITION! It's not an algorithm. In the algorithms (e.g. Q-learning), things will typically change, but that's a different story that you should investigate later.

  From this, I infer that the $Q$ value (the action-value function) will be affected by the policy $\pi$.

So, $Q_\pi(s, a)$ will not keep changing. You could say that $Q_\pi(s, a)$ (which is a function) is "affected by" $\pi$ ONLY in the sense that it is "defined in terms of" $\pi$. To be precise, $Q_\pi(s, a)$ is actually an expectation (which is a mathematical concept similar to an ideal average). If you are not familiar with the concept of expectation, I suggest you get familiar with it first, before studying reinforcement learning.

  Shouldn't the Q value be constant, because the same action taken in the same state will always give the same yield (and hence remain constantly good/bad)?

Again, there's the distinction between the algorithm that you actually use to find the function $Q_\pi(s, a)$ and the definition of the same function. In case you are estimating the function with an algorithm, you will not necessarily find "constant Q values". It depends on different aspects, which I would like to avoid discussing here, so that this post doesn't become an open discussion (I suggest you first learn about the basic Bellman equations and then you study the algorithms from the book Reinforcement learning: an introduction by Sutton and Barto).


How to change this RNN text classification code to become text generation code? I can do text classification with RNN, in which the last output of RNN (rnn_outputs[-1]) is used to matmul with output layer weight and plus bias. That is getting a word (class name) after the last T in the time dimension of RNN.

The matter is for text generation, I need a word somewhere in the middle of time dimension, eg.:

t0  t1    t2  t3
The brown fox jumps

For this example, I have the first 2 words: The, brown.

How to get the next word ie. "fox" using RNN (LSTM)? How to convert the following text classification code to text generating code?

Source code (text classification):

import tensorflow as tf;
tf.reset_default_graph();

#data
'''
t0      t1      t2
british gray    is => cat (y=0)
0       1       2
white   samoyed is => dog (y=1)
3       4       2
'''
Bsize = 2;
Times = 3;
Max_X = 4;
Max_Y = 1;

X = [[[0],[1],[2]], [[3],[4],[2]]];
Y = [[0],           [1]          ];

#normalise
for I in range(len(X)):
  for J in range(len(X[I])):
    X[I][J][0] /= Max_X;

for I in range(len(Y)):
  Y[I][0] /= Max_Y;

#model
Inputs   = tf.placeholder(tf.float32, [Bsize,Times,1]);
Expected = tf.placeholder(tf.float32, [Bsize,      1]);

#single LSTM layer
#'''
Layer1   = tf.keras.layers.LSTM(20);
Hidden1  = Layer1(Inputs);
#'''

#multi LSTM layers
'''
Layers = tf.keras.layers.RNN([
  tf.keras.layers.LSTMCell(30), #hidden 1
  tf.keras.layers.LSTMCell(20)  #hidden 2
]);
Hidden2 = Layers(Inputs);
'''

Weight3  = tf.Variable(tf.random_uniform([20,1], -1,1));
Bias3    = tf.Variable(tf.random_uniform([   1], -1,1));
Output   = tf.sigmoid(tf.matmul(Hidden1,Weight3) + Bias3);

Loss     = tf.reduce_sum(tf.square(Expected-Output));
Optim    = tf.train.GradientDescentOptimizer(1e-1);
Training = Optim.minimize(Loss);

#train
Sess = tf.Session();
Init = tf.global_variables_initializer();
Sess.run(Init);

Feed = {Inputs:X, Expected:Y};
for I in range(1000): #number of feeds, 1 feed = 1 batch
  if I%100==0:
    Lossvalue = Sess.run(Loss,Feed);
    print("Loss:",Lossvalue);
  #end if

  Sess.run(Training,Feed);
#end for

Lastloss = Sess.run(Loss,Feed);
print("Loss:",Lastloss,"(Last)");

#eval
Results = Sess.run(Output,Feed);
print("\nEval:");
print(Results);

print("\nDone.");
#eof

I found out how to switch it (the code) to do text generation task, use 3D input (X) and 3D labels (Y) as in the source code below:

Source code:

import tensorflow as tf;
tf.reset_default_graph();

#data
'''
t0       t1       t2
british  gray     is  cat
0        1        2   (3)  <=x
1        2        3        <=y
white    samoyed  is  dog
4        5        2   (6)  <=x
5        2        6        <=y
'''
Bsize = 2;
Times = 3;
Max_X = 5;
Max_Y = 6;

X = [[[0],[1],[2]], [[4],[5],[2]]];
Y = [[[1],[2],[3]], [[5],[2],[6]]];

#normalise
for I in range(len(X)):
  for J in range(len(X[I])):
    X[I][J][0] /= Max_X;

for I in range(len(Y)):
  for J in range(len(Y[I])):
    Y[I][J][0] /= Max_Y;

#model
Input    = tf.placeholder(tf.float32, [Bsize,Times,1]);
Expected = tf.placeholder(tf.float32, [Bsize,Times,1]);

#single LSTM layer
'''
Layer1   = tf.keras.layers.LSTM(20);
Hidden1  = Layer1(Input);
'''

#multi LSTM layers
#'''
Layers = tf.keras.layers.RNN([
  tf.keras.layers.LSTMCell(30), #hidden 1
  tf.keras.layers.LSTMCell(20)  #hidden 2
],
return_sequences=True);
Hidden2 = Layers(Input);
#'''

Weight3  = tf.Variable(tf.random_uniform([20,1], -1,1));
Bias3    = tf.Variable(tf.random_uniform([   1], -1,1));
Output   = tf.sigmoid(tf.matmul(Hidden2,Weight3) + Bias3); #sequence of 2d * 2d

Loss     = tf.reduce_sum(tf.square(Expected-Output));
Optim    = tf.train.GradientDescentOptimizer(1e-1);
Training = Optim.minimize(Loss);

#train
Sess = tf.Session();
Init = tf.global_variables_initializer();
Sess.run(Init);

Feed   = {Input:X, Expected:Y};
Epochs = 10000;

for I in range(Epochs): #number of feeds, 1 feed = 1 batch
  if I%(Epochs/10)==0:
    Lossvalue = Sess.run(Loss,Feed);
    print("Loss:",Lossvalue);
  #end if

  Sess.run(Training,Feed);
#end for

Lastloss = Sess.run(Loss,Feed);
print("Loss:",Lastloss,"(Last)");

#eval
Results = Sess.run(Output,Feed).tolist();
print("\nEval:");
for I in range(len(Results)):
  for J in range(len(Results[I])):
    for K in range(len(Results[I][J])):
      Results[I][J][K] = round(Results[I][J][K]*Max_Y);
#end for i
print(Results);

print("\nDone.");
#eof


What are the current approaches for AI to learn a foreign language just from English books? I'm aware this could be a complex topic, however I'm interested in existing research projects or studies where people are attempting or have succeeded in teaching an AI a foreign language just by training/teaching it from English books. By reading, analysing and understanding, so that it knows the foreign language's rules (such as grammar, spelling, etc.), the same way as a human would learn. The language doesn't have to be Chinese, which is difficult for even humans to learn.

Current approaches for learning a language require having a large corpus of that language; it also doesn't seem reasonable to expect that it will ever be possible to learn about language A by extracting information from a corpus from an unrelated language B.

Even if you want to learn about human languages in general (what sorts of things are true about grammar, vocabulary, and so on), that relies having many languages as training data, so that you can see the different ways of doing things instead of assuming that the way they're done in English is the way they're done in every language.

(There is work in automatic translation that goes from a language to 'concept-space', then goes from that 'concept-space' to another language, so that you can build an English-Chinese translator by building two separate English-Concept and Chinese-Concept translators, instead of ever needing material that directly links English and Chinese. The obvious benefit of this is scalability; in order to make translators for a new language to any other language, you just need to learn that language and the models build themselves.)


Should I apply ReLU to non negative output? Suppose I want to predict the position of a sensor based on its reading.

I can first predict the unit vector and predict the distance to be multiplied to this vector.
And I know that distance will never be negative because all the negative parts are inside unit vector already.

Should I apply ReLU to the distance before multiplying it to the unit vector?

I'm thinking that this can be helpful to eliminate the network from needing too much training data by restricting the output ranges the network could give. But I also think that it could make the learning slower when the ReLU unit dies (value=0) so the gradient doesn't flow properly somehow.

Applying a ReLU activation function to a non-negative number is the same as applying the identity activation function.

What you may wish to do is apply $\ln(|\vec{x}|)$, $\arctan(x[0], x[1])$, ..., $\arctan(x[0], x[N-1])$, where $N$ is the number of dimensions to $\vec{x}$, and pass this directly into a parameter matrix multiplication (which involves deliberately using an identity activation in the input layer in some frameworks). This typically improves speed, accuracy, and reliability of convergence when the magnitudes exhibit a distribution close to an exponential of Gaussian and the directional components are near Gaussian when expressed in radians.


Why is it so common to initialize weights with a Guassian distribution divided by the square root of number of neurons in a layer? I have seen in several jupyter notebooks people initializing the NN weights using:

np.random.randn(D, M) / np.sqrt(D)

Other times they just do:

np.random.randn(D, M)

What is the advantage of dividing the Gaussian distribution by the squared root of the number of neurons in the layer?

Thanks

I think they use the Xavier/Glorot's initialization method. You can read from the original paper:

We initialized the biases to be 0 and the weights $W_{ij}$ at each layer with the following commonly used heuristic:
$W_{ij} \sim U [ -\frac{1}{\sqrt{n}}, \frac{1}{\sqrt{n}}] $
where $U[−a, a]$ is the uniform distribution in the interval $(−a, a)$ and $n$ is the size of the previous layer (the number of columns of $W$)

Some people use this as some reports said this initialization method lead to better result


Finding an optimum back propagation algorithm I recently started working on very simple machine learning codes in Python and I came across a big problem: teaching the system to improve on its guesses.

So this is what the code is about:
I will have a list of organisms with their features stated in numerical values. I want to write a code that identifies that whether the organism is a cat or a fish or neither based on their characteristics. (For example an organism with a high fur value and 4 legs is more probable to be a cat.)

My idea for the neural network is to have five input nodes(for the five characteristics) and 2 output nodes (one for how cat it is and one for how fish it is). The input nodes are multiplied by a weight value and then all summed together to produce one of the output nodes. This repeats itself for the other output. How much the system got wrong is just the difference between the value of the output nodes and how cat/fish the being actually is.

But how can I use this information to correct the weights of the input nodes? Since the weights are randomly generated they can be in the "wrong" directions to begin with. For example if the subject is a cat then we should be expecting high fur and leg values. But what if the weight for the leg value is negative while the weight for the fur value is positive? Adding or multiplying the weight by the error won't bring us any closer to accurately determining the being. Is my neural network flawed to begin with? Or is there a rule of thumb in choosing back propagation algorithms?

Thanks.

When you are training a neural network, you use an algorithm called back propagation. This algorithm uses partial derivatives to determine the optimal values for weights. Partial derivatives are a calculus based method which tell you how far you need to adjust the weights in order to get to an optimum value. However, when you have neural networks with many different weights you need to be able to decide which ones to update in order to achieve a more optimal network. This is where stochastic gradient descent comes in. This algorithm is able to take many different gradients and traverse through many dimensional space into a lower error rate. Also, from your diagram of your neural network, I do not see any hidden neurons, which are neurons between the input and output layers which do most of the actual work. While you might be able to train your neural network optimally without hidden neurons, it might improve your accuracy depending upon your dataset. Finally, implementing a back propagation algorithm is a lot of work which is why many people use pre built libraries. Unless you are doing research, I recommend you do the same. I think for a problem like this, pybrain would be perfect, and if you want to use deep neural networks, tensorflow would be good for that.


What is the meaning of the terms in this evaluation function for chess? I'm trying to improve my evaluation and I saw this here
materialScore = kingWt  * (wK-bK)
              + queenWt * (wQ-bQ)
              + rookWt  * (wR-bR)
              + knightWt* (wN-bN)
              + bishopWt* (wB-bB)
              + pawnWt  * (wP-bP)

How do I get the value, let's say wK? Do I get the position of the king and score it relative to the board? For example, wK is more safe than the bK, so let's say wK - bK = 1 - 0.5. So, the result will be 90 * (0.5). Is this really how it works?

The function you are showing (from this website) only calculates the score of a position based on how many pieces are on the board. It does not take into account where the pieces are located. So wK, bK, wQ, bQ, etc. are simply the number of the specific pieces that are on the board. The weights rank the pieces according to their importance.
So for example if white has 8 pawns and black has 6 pawns, the last term of the score would be pawnWt * (wP-bP) = 1 * (8-6) = 2. (assuming the value of a pawn to be 1) That leads to a relative advantage for white only based on pawns, but according to how many of the other pieces are on the board, black might still have an advantage.


What is the meaning of these equations in Noise2Noise paper? I am trying to understand what is meant by following equations in the Noise2Noise paper by Nvidia.

What is meant by the equation in this image? What is $\mathbb{E}_y\{y\}$? And how should I try to visualize these equations?

The equation you are referring to is called Mean Squared Error (or $L_2$ loss) and it is used for regression tasks, where the goal is to predict a real value given some input.
In your case, the inputs are measurements of temperature $y$, either at a certain point in time or point in space or both or none, this is not clear from the image. Now, the goal would be to predict the temperature at a new point in space, time, or both, where we don't have access to a measurement. That is we would like to find a function $f$ (e.g. a simple linear function) which we can use for prediction. But how can we measure which function is "best"? We introduce a loss function $L(f,y)$, another function which tells us how good our proposed function is.
Visually it looks like this (image source):

Red crossed are measurements, the black line is our function we use for prediction and the green dotted lines are the errors (the distance from our prediction to the real measurement). In this example salary depends on experience.
Now, the paper introduces the constant mean of all measurements as $y$, $z = \mathbb{E}_y\{y\} = \frac{1}{N}\sum_i^N y_i$, as our function $f$, which is known to be the minimizer for the $L_2$ loss in the case where there is no dependence on other variables (e.g. time or space).


How could I convolve a 4D image and a 4D filter with stride? I want to create a CNN in Python, specifically, only with NumPy, if possible. For optimizing the time of convolution (actually correlation) in the network, I wanna try to use FFT-based convolution. The data that needs to be convoluted (correlated) is a 4D image tensor with shape [batch_size, width, height, channels] and 4D filter tensor [filter_width, filter_height, in_channel, out_channel]. I read a lot of articles about FFT-based convolution, but they aren't doing it in my way. Thus, I need your help.
How could I fft-convolve a 4D image and a 4D filter with stride?

I think this should work for you:
scipy.signal.correlate | SciPy
I used it myself while I was writing a CNN in numpy.


Has anyone attempted to train an AI to learn all languages? It seems that most projects attempt to teach the AI to learn individual, specific languages.

It occurs to me that there are relations in written and spoken words and phrases across languages - most of use have a much easier time learning more languages after we learn a second language, and we start to understand the relations between words and phrases in different languages.

Has anyone attempt to train an AI to learn all languages?

Wouldn't this potentially be a much simpler problem than trying to teach an AI a single, specific language with all of the specifics and details of that single language? Since you're actually omitting a lot of related data in other languages from the training set?

There are approaches in machine translation that try to capture this kind of synergy between languages. The idea is that if you train your architecture to be able to translate English-Japanese, Japanese-English, Korean-English, English-Korean it will also be able to translate from Japanese to Korean without ever having seen a single such training example. Here, you can read about this so-called zero-shot translation.

It is also possible to train wordvectors on several languages at once, which might give you better wordvectors for a language with few training examples.

Of course for true language understanding you have to solve the grounding problem and I'm not sure using several language will help with that.


How do you encode a chess move in a neural network? In a neural network for chess (or checkers), the output is a piece or square on the board and an end position.

How would one encode this?

As far as I can see choosing a starting square is 8x8=64 outputs and an ending square is 8x8=64 outputs. So the total number of possible moves is 64x64 4096 outputs. Giving a probability for every possible move.

Is this correct? This seems like an awful lot of outputs!

The number is 4672 from Google.

  https://arxiv.org/pdf/1712.01815.pdf

A move in chess may be described in two parts: selecting the piece to move, and then
selecting among the legal moves for that piece. We represent the policy π(a|s) by a 8 × 8 × 73
stack of planes encoding a probability distribution over 4,672 possible moves. Each of the 8×8
positions identifies the square from which to “pick up” a piece.

4672 might sound a big number but it's nothing to what Google (and many other competitors) has been doing to deep learning for image analysis.


What is the relation between an environment, a state and a model? In particular, I would like to have a simple definition of "environment" and "state". What are the differences between those two concepts? Also, I would like to know how the concept of model relates to the other two.

There is a similar question What is the difference between an observation and a state in reinforcement learning?, but it is not exactly what I was looking for.

Environment
This is the manifestation of the problem being solved. It might be a real physical situation (a road network and cars), or virtual on a computer (a board game on a computer). It includes all the machinery necessary to resolve what happens. E.g. in the real world the objects involved, how the agent exerts its control when taking actions, and the applicable real-world laws of physics. Or, in a simulated world, things like the rules of a board game, implemented in code.
State
This is the representation of a "position" at a certain time step within the environment. It may be something the agent can observe through sensors, or be provided directly by the computer system running a simulation.
For RL theory to hold, it is important that the state representation has the Markov Property, which is that the state accurately foretells the probabilities of rewards and following state for each action that could be taken. You do not need to know those probabilities in order to run RL algorithms (in fact it is a common case that you don't know). However, it is important that the dependency between the state+action and what happens next holds reliably.
The state is commonly represented by a vector of values. These describe positions of pieces in a game, or positions and velocities of objects that have been sensed. A state may be built from observations, but does not have to match 1-to-1 with a single observation. Care must be taken to have enough information to have the Markov Property. So, for instance, a single image from a camera does not capture velocity - if velocity is important to your problem, you may need multiple consecutive images to build a useful state.
Model
In reinforcement learning, the term "model" specifically means a predictive model of the environment that resolves probabilities of next reward and next state following an action from a state. The model might be provided by the code for the environment, or it can be learned (separately to learning to behave in that environment).
Some RL algorithms can make use of a model to help with learning. Planning algorithms require one. So called "model free" algorithms do not because they do not make use of an explicit model, they work purely from experience.
There are broadly two types of model:

A distribution model which provides probabilities of all events. The most general function for this might be $p(r,s'|s,a)$ which is the probability of receiving reward $r$ and transitioning to state $s'$ given starting in state $s$ and taking action $a$.

A sampling model which generates reward $r$ and next state $s'$ when given a current state $s$ and action $a$. The samples might be from a simulation, or just taken from history of what the learning algorithm has experienced so far.

In more general stats/ML, the term "model" is more inclusive, and can mean any predictive system that you might build, not just predictions of next reward and state. However, the literature for RL typically avoids calling those "model", and uses terms like "function approximator" to avoid overloading the meaning of "model".


Understanding why the expectation is over the new policy $\pi'$ in the proof of the Policy Improvement Theorem In reinforcement learning, policy improvement is a part of an algorithm called policy iteration, which attempts to find approximate solutions to the Bellman optimality equations.
Pages 84 and 85 in Sutton and Barto's book on RL mentions the following theorem:

Policy Improvement Theorem
Given two deterministic policies $\pi$ and $\pi'$, then $$v_\pi(s) \leq q_\pi(s, \pi'(s)), \forall s \in S.$$
where $S$ is the set of all states.

In the the right-hand side of the inequality, the agent acts according to policy $\pi'$ (given that $\pi'(s)$ is used in the inequality), in the current state $s$, and for all subsequent states acts according to policy $\pi$.
In the left-hand side of the inequality, the agent acts according to policy $\pi$ (hence the subscript $_\pi$ of $v_\pi(s)$), starting from the current state $s$.
The claim is the following

$$v_\pi(s) \leq v_{\pi'}(s), \forall s \in S$$

In other words, $\pi'$ is is an improvement over $\pi$.
However, I have a difficulty in understanding the proof. This is discussed below.

Proof
$$v_\pi(s) \leq q_\pi(s, \pi'(s)) = \mathbb{E}_{\pi'}[R_{t+1} + \gamma v_\pi(S_{t+1}) \mid S_t = s]$$

I am stuck here. The q-function is evaluated over the policy $\pi$ (note the subscript $_\pi$ in $q_\pi(s, \pi'(s))$). That being the case, how is the expectation over the policy $\pi'$?
My guess is the following. In the proof given in Sutton and Barto, the expectation is unrolled in time. At each time step, the agent follows the policy $\pi'$  for that particular time step, and then follows $\pi$ from then on. In the limit of this process, the policy transforms from $\pi$ to $\pi'$. As long as the expression for the return inside the expectation is finite, the governing policy should be $\pi$; only in the limit of this process does the governing policy transform to $\pi'$.

The expectation is over the policy $\pi'$ because the action at the state $S_t = s$ is taken according to $\pi'$, and, for the proof, the book text (2nd edition, paragraph below Equation 4.8) defines $\pi'$ to be a policy that is identical to $\pi$ except that $\pi'(s) = a \neq \pi(s)$, where $s$ is one particular state.

So, essential, the book text tries to prove that, for such a changed policy $\pi'$, if $q_\pi(s, a) = q_\pi(s, \pi'(s)) > v_\pi(s)$, then the changed policy is better than $\pi$. Note that $\pi'(s) = a$ in this case.

That is why $q_\pi(s, \pi'(s)) = \mathbb{E}_{\pi'}[R_{t+1} + \gamma v_\pi(S_{t+1}) \mid S_t = s]$.


What is the difference between local search and global search algorithms? What is the difference between local search and global (or complete) search algorithms?

The difference between a local search algorithm (like beam search) and a complete search algorithm (like A*) is, for the most part, small.

Local search algorithms will not always find the correct or optimal solution, if one exists. For example, with beam search (excluding an infinite beam width), it sacrifices completeness for greater efficiency by ordering partial solutions by some heuristic predicting how close a partial solution is to a complete one. Beam search is a greedy algorithm.

Complete search algorithms will always find the correct or optimal solution if there is one, given enough time. An algorithm like A* uses heuristics to prune the tree as it goes along, but it will not converge to a sub-optimal solution. In a lot of practical cases, this is inefficient, but how much so is problem dependent.


Unable to understand the second iteration update in value iteration algorithm for solving MDP I am trying to understand the value iteration method for Markov Decision Process(MDP) and I was referring ot UC Berkeley's slides titled Markov Decision Processes
and
Exact Solution Methods

On slide no.9, we start with the first step :

Ok! So, we have the information about the transition function (desribed elaborately in slide no.5 as well), the resting reward is 0 and discount of 0.9.

Using this, I am able to compute the utility value of the cell left to terminal state with R = +1 (Green cell). The action that is going to be most rewarding at this cell is moving forward, so putting the values in the equation as:

0.0 + 0.9(0.8*1 + 0.1*0 + 0.1*0)
=0.72

which seems to be correct:

Now, using the same algorithm I am able to compute the value of the cells adjacent to this newly obtained utility cell value. However, I really do not know how did they update the value from

  0.72 -> 0.78

in the next slide:

I have tried searching at various sites and seen some videos but most of them stop at first iteration assuming the next step is same as it is a recursive equation (And it should have been so!) but I am stuck at this!

First thing to know is that, in this case, values for the gridworld in new iteration are completely calculated with respect to the old values from the previous iteration. Value of $0.78$ is got like this:

$0.9 \cdot (0.8 \cdot 1 + 0.1 \cdot 0.72 + 0.1 \cdot 0) = 0.7848 \approx 0.78$

term $0.8 \cdot 1$ is for going to the right with probability of $0.8$ and getting reward of $1$.

term $0.1 \cdot 0.72$ is for going up with probability of $0.1$, we hit the wall and stay in the same field which value is $0.72$ (from previous iteration)

term $0.1 \cdot 0$ is for going down with probability of $0.1$, even though value of that field in the image is $0.43$ we take the value from previous iteration which is $0$.


Book(s) for text embedding Text here refers to either character or word or sentence.
Is there any recent textbook that encompasses from classical methods to the modern techniques for embedding texts?
If a single textbook is unavailable then please recommend a list of books covering the whole spectrum as mentioned above.
Modern textbooks that are similar to Christopher D. Manning, Prabhakar Raghavan and Hinrich Schütze, Introduction to Information Retrieval, Cambridge University Press. 2008 are highly encouraged.
This question asks for textbook/research paper on word embedding only.

I propose you try this. It's about modern Natural Language Processing, Computational Linguistics και Speech Recognition, including Embeddings methods.


CNN based model poor result My goal is to train and evaluate The German Traffic Sign Recognition Benchmark (GTSRB) dataset using Pytorch.
I downloaded the datasets from the official site GTSRB_Final_Training_Images.zip and GTSRB_Final_Test_Images.zip.
But I get very poor results when I train the model.
I am using a simple CNN model.
The code:
import torch
import torch.nn as nn
import torch.optim as optim

from torchvision import datasets, transforms
from torch.utils.data import DataLoader

device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")

class Flatten(nn.Module):
   def forward(self, x):
     return x.view(x.shape[0], -1)

odel_cnn = nn.Sequential(nn.Conv2d(3, 32, 3, padding=1), nn.ReLU(),
                      nn.Conv2d(32, 32, 3, padding=1, stride=2), nn.ReLU(),
                      nn.Conv2d(32, 64, 3, padding=1), nn.ReLU(),
                      nn.Conv2d(64, 64, 3, padding=1, stride=2), nn.ReLU(),
                      nn.Flatten(),
                      nn.Linear(4096, 100), nn.ReLU(),
                      nn.Linear(100, 44)).to(device)

data_transforms = transforms.Compose([
    transforms.Resize((32,32)),
    transforms.ToTensor(),
    #transforms.Normalize((0.5,), (1.0,))
    ])

train_data_path = "GTSRB/Train"
test_data_path = "GTSRB/Test"

train_data = datasets.ImageFolder(root = train_data_path,transform = data_transforms)
test_data = datasets.ImageFolder(root = test_data_path, transform = data_transforms)

train_loader = DataLoader(train_data, batch_size = 100, shuffle=True)
test_loader = DataLoader(test_data, batch_size = 100, shuffle=False)

def epoch(loader, model, opt=None):
total_loss, total_err = 0.,0.
for X,y in loader:
    X,y = X.to(device), y.to(device)
    yp = model(X)
    loss = nn.CrossEntropyLoss()(yp,y)
    if opt:
        opt.zero_grad()
        loss.backward()
        opt.step()

    total_err += (yp.max(dim=1)[1] != y).sum().item()
    total_loss += loss.item() * X.shape[0]
return total_err / len(loader.dataset), total_loss / len(loader.dataset)

opt = optim.SGD(model_cnn.parameters(), lr=1e-1)

for t in range(20):
    train_err, train_loss = epoch(train_loader, model_cnn, opt)
    test_err, test_loss = epoch(test_loader, model_cnn)
    if t == 4:
        for param_group in opt.param_groups:
            param_group["lr"] = 1e-2
    print(*("{:.6f}".format(i) for i in (train_err, train_loss, test_err, test_loss)), sep="\t")

The output:
0.518248    0.851532    1.000000    11.033573
0.514448    0.821411    0.999921    12.305183
0.515213    0.801951    1.000000 12.332444
0.519358    0.787077    1.000000    12.669552
0.511706    0.775102    0.997783    12.443266
0.509730    0.732133    0.999842    13.915797
0.508263    0.724747    0.999842    14.257156
0.510163    0.722937    1.000000    14.508054
0.510980    0.722053    0.999921    14.173411
0.511617    0.721190    0.999921    14.703455

The dataset contains ppm files. They are RGB.
I tried with another dataset containing png files but the result was also bad.
I have the entire code on Google Colab:
https://colab.research.google.com/drive/1br7sE2vI9Ilu3yKTpJdNABNVEmDZENmc?hl=en#scrollTo=qn3RgfRoAfVB
I don't know if there is problem with the model or the dataset.
Any suggestions?

Have you tried using a different learning rate?
Yours seems to be quite high, you could try 1e-4 or something in that region.
Coming from the computer vision side, you could also try to improve image readability with prior grayscaling or blurring.
Here are some repos on GTSRB:
https://github.com/Junth/Traffic-Sign-Classification-using-ConvNets
https://github.com/surmenok/GTSRB


Is there a mathematical example for Conditional Random Fields? I am learning about probabilistic graphical models and I was wondering if there is an example explaining the math behind conditional random fields. Looking solely on the formula, I have no idea what we actually do. I found a lot of examples for the hidden Markov model. There is a part speech-tag task where we have to find the tags for the sentence "flies like a flower". On these slides (slide 8) Ambiguity Resolution: Statistical Method-Prof. Ahmed Rafea, an HMM is used to find the correct tags. How would I transform this model into a CRF and how would I apply the math?

CRF is for HMM in sequences classification as Logistic Regression is for Naive Bayes for simple classification. For a in depth difference between them I strongly suggest you to read the classical Sutton article "An Introduction to Conditional
Random Fields". For a more practical example explaining how to apply the math I suggest you to read this article.


Self-driving control logic based on semantic segmentation In the context of autonomous driving, two main stages are typically implemented: an image processing stage and a control stage. The first aims at extracting useful information from the acquired image while the second employs those information to control the vehicle.

As far as concerning the processing stage, semantic segmentation is typically used. The input image is divided in different areas with a specific meaning (road, sky, car etc...). Here is an example of semantic segmentation:

The output of the segmentation stage is very complex. I am trying to understand how this information is typically used in the control stage, and how to use the information on the segmented areas to control the vehicle.

For simplicity, let's just consider a vehicle that has to follow a path.

TL;DR: what are the typical control algorithms for autonomous driving based on semantic segmentation?

This is a great question. Connecting the AI knowledge to the application is a difficult task and its not easy to test and perfect.

There are a few points regarding autonomous driving algorithms that one should keep in mind.

A lot of data is required before actually applying a model to the real world.
The control algorithms are designed for specific tasks, there is no master algorithm that solves the issue.
Time is very important, decisions have to be made in real time.

The image segmentation you have mentioned in the question is rarely used in current autonomous technologies. The main issue is that

The model is slow and requires massive computing resources. Due to this is cannot help in real time decision making.
A very important parameter depth is missing from the information. This is of utmost importance in autonomous driving technology. A 2D image cannot provide this parameter.

So the current technology use

A range of sensors (LIDAR, Ultrasonic, Cameras etc.) that provide readings 100+ times a second.
A lot of the computer vision algorithms and control algorithms are hard coded. There are set parameters within which decisions are made.
The control algorithms are designed to work with each set of sensors and the mechanical components of the car.

To get a better look at the way these algorithms are used, you can have a look at

Stanley vision system winner of the DARPA grand challenge (Stanford)
How a driverless car sees the road TEDx


Are there any advantages of using rules-based approaches versus models for detecting spam? Suppose that we have unlabeled data. That is, all we have are a collection of emails and want to determine whether any of them is spam or not. Let's say we have $1,000$ rules to determine whether a particular email is spam or not. For example, one rule could be that a sender's email address should not contain the text  no_reply . If an email does contain this text, then it would be classified as spam.
What are the advantages/disadvantages of a rules-based approach for detecting spam vs. a non-rules-based approach/unsupervised methods for detecting spam?
Would there even be a point in constructing a non-rules based model given that we already have a rules-based model? Could we use the rules-based model to create some labeled training data and then apply supervised techniques?

Yes, there would be a point. Assuming your rule set is accurate, then you can use data classified with it to train a model. This model can be expected to be more robust and properly categorise emails that your rule-set will not handle.

Why? Machine learning algorithms generally work on features, and identify relationships between those features that lead to a classification decision. A human rule author basically does the same, but they might not notice subtle relationships; a good ML algorithm, however, might pick those up.

So you could have a hybrid model, where you first use your rule-based classifier, and anything that does not get classified is then run through the ML classifier.


Techniques and semantics in better training of deep learning models I'm relatively new to Deep Learning, and trying various models and datasets using Keras. I'm starting to love it!

Through-out my experimentations, I have come into some semantic questions that I don't know how they can affect the overall accuracy of my trained model. My target application is fire detection in videos (fire vs non-fire). So I'm trying to get tips and tricks from those well experienced on Deep learning, and here are my semantic questions:

Given that I have to do detection on videos, I've been mostly adding actuall frames of videos to my dataset, and less photos. Does adding photos from Google ever help (as we largen our dataset) or it's actually more considered noises and shall be removed?
I've trained a deep model (ResNet50) as well as a shallow 5-layer model. I realized the ResNet50 model is more sensitive and has a high recall (all fires are definitely detected), but has false positives as well (strong source of lights like sunlight or lamps are identified as fire). While the shallower model is 10x faster, it can miss fires if it is smaller in the image, so it's less sensitive. But also has low false positives. Is it always true? So what are techniques and tips to fix these issues in each of these models?

For instance, the shallow model doesn't see this fire. Shall I think it's not complex enough to work well when the scene has many objects inside?

The sample code I saw resizes photos to 256x256 for training. What's the effect of bigger sizes vs smaller ones say 300x300? Can I expect while bigger sizes increase computation time, they provide higher accuracy?
The sample code also converts photos to grayscale and uses Antialiasing before passing. Does it have good effects? What if I pass the colored version as fire is mostly about colors?
When I see the model is doing bad on certain scenes (say those sun lights or lamps), I take multiple of those frames and add them to my non-fire dataset. Does it have any positive effects and being taken care of? And is it better to add multiple successive frames or just one frame is enough?
My fire dataset has 1800 images and my non-fire dataset has 4500 images. As a rule of thumb, the bigger each class, the better? Of course the non-fire data should be bigger, but we can not add whatever on earth as non-fire so what should be the distribution of the sizes?

I try to answer the things I know for sure:

One effect of bigger images is the increasing computation time due to more pixels (input to your training)

4.Grayscaling reduces the information, which might decrease training time, but also model performance (accuracy, precision, recall). What I have seen is that grayscaling is used in for example face detection where structural information (shapes) is important and colors can be regarded as secondary. For your example you have to determine whether such structural differences between positives and negatives exist and if color is a valuable information.

If possible, I would somehow alter these copies by clipping the objects (sun lights, lamps) and rotate, scale, make them transparent etc.
In the end your training data should be representative of the data it should generalize to. If you would for example take my user icon as an additional non fire example, this wouldn't help your cause.


Is there a way to perform pattern recognition without a labeled training set? I have a 10GB file of a time series 1D signal. I want to find some patterns within this signal, I know CNN's are great for this but the problem is I don't have any training data.
Now, I could, of course, spend an entire week slowly making 100 versions of a certain pattern to train the CNN with. But maybe there is some other way?
Maybe there is a way for the neural network to work out patterns on its own and simply categorize them? Like this is pattern A, this is pattern B.
My ultimate goal is to look at any size data and find the occurrences of patterns within the data.
Does anyone have any idea how this problem could be solved? I am just starting with machine learning, so I am slowly learning what's possible in this field.

You should look into unsupervised learning, which is machine learning without a human-labeled training set.


Is neural networks training done one-by-one? I'm trying to learn neural networks by watching this series of videos and implementing a simple neural network in Python.

Here's one of the things I'm wondering about: I'm training the neural network on sample data, and I've got 1,000 samples. The training consists of gradually changing the weights and biases to make the cost function result in a smaller cost.

My question: Should I be changing the weights/biases on every single sample before moving on to the next sample, or should I first calculate the desired changes for the entire lot of 1,000 samples, and only then start applying them to the network?

Should I be changing the weights/biases on every single sample before moving on to the next sample,

You can do this, it is called stochastic gradient descent (SGD) and typically you will shuffle the dataset before working through it each time.

  or should I first calculate the desired changes for the entire lot of 1,000 samples, and only then start applying them to the network?

You can do this, it is called batch gradient descent, or in some cases (especially in older resources) just assumed as the normal approach and called gradient descent.

Each approach offers advantages and disadvantages. In general:

SGD makes each update sooner in terms of amount of data that has been processed. So you may need less epochs before converging on reasonable values.
SGD does more processing per sample (because it updates more frequently), so is also slower in the sense that it will take longer to process each sample.
SGD can take less advantage of parallelisation, as the update steps mean you have to run each data item serially (as the weights have changed and error/gradient results are calculated for a specific set of weights).
SGD individual steps make typically only very rough guesses at the correct gradients to change weights in. This is both a disadvantage (performance of the NN against the objective on the training set can decrease as well as increase) and an advantage (there is less likelihood of getting stuck in a local stationary point due the "jitter" these random differences cause).

What happens in practice is that most software allows you to compromise between batch processing and single-sample processing, to try and get the best performance and update characteristics. This is called mini-batch processing, which involves:

Shuffling the dataset at the start of each epoch.
Working through the shuffled data, N items per time where N might vary from maybe 10 to 1000, depending on the problem and any constraints on the hardware. A common decision is to process the largest batch size that the GPU acceleration allows to run in parallel.
Calculate the update required for each small batch, then apply it.

This is nowadays the most common update method that most neural network libraries assume, and they almost universally will accept a batch size parameter in the training API. Most of the libraries will still call simple optimisers that do that SGD; technically it is true, the gradients calculated are still somewhat randomised due to not using the full batch, but you may find this called mini-batch gradient descent in some older papers.


Are there any artificial intelligence systems with an internal dialectical approach and multiple minds which develop within a community of nodes? In my estimation we have two minds which manage to speak to each other in dialectic through a series of interrupts. Thus at any one time one of these systems is controlling master and inhabits our consciousness. The subordinate system controls context which is constantly being "primed" by our senses and our subordinate systems experience of our conscious thought process( see thinking fast and slow by Daniel Kahneman). Thus our thought process is constantly a driven one. Similarly this system works as a node in a community and not as a standalone thing.
 I think what we have currently is "artificial thinking" which is abstracted a long way from what is described above. so my question is "are there any artificial intelligence systems with an internal dialectical approach and with drivers and conceived above and which develop within a community of nodes? "

There are a lot of systems that follow the ancient maxim: "Always two there are; no more, no less. A master and an apprentice."
In reinforcement learning a class of such setups is called Actor-Critic-Method. There you have a master, whose duty it is to create feedback for the actions of the apprentice, who acts in a given environment. This would be comparable to how a human learns some physical activity, like playing table tennis. You basically let your body do its thing, but your consciousness evaluates how good the result is.
The setup of AlphaGo might be even closer to Kahnemann's system 1 and system 2. AlphaGo has two neural networks which provide actions and evaluations (system 1, fast, intuitive, etc.) and the Monte Carlo tree search, which uses these actions and evaluations to prune a search tree and make a decision (system 2, deliberate, logical).
In the end, this kind of structure will pop up again and again because it is often necessary to do some kind of classification or preprocessing on the raw data before your algorithm can be run on it. You could frame the whole history of gofai as the story of how scientists thought system 1 should be easy and system 2 should be doable in a few decades, where the reality is that we have no idea how difficult system 2 is because it turned out that system 1 is extremely difficult.


NoisyNet DQN with default parameters not exploring I implemented a DQN algorithm that plays OpenAIs Cartpole environment. The NN architecture consists of 3 normal linear layers that encode the state, and one noisy linear layer, that predicts the Q value based on the encoded state.
My NoisyLinear layers looks like this:

class NoisyLinear(nn.Module):
  def __init__(self, in_features, out_features):
    super(NoisyLinear, self).__init__()
    self.in_features = in_features
    self.out_features = out_features
    self.sigma_zero = 0.5
    self.weight_mu = torch.empty(out_features, in_features)
    self.weight_sigma = torch.empty(out_features, in_features)
    self.weight_epsilon = torch.empty(out_features, in_features, requires_grad=False)
    self.bias_mu = torch.empty(out_features)
    self.bias_sigma = torch.empty(out_features)
    self.bias_epsilon = torch.empty(out_features, requires_grad=False)
    self.reset_parameters()
    self.reset_noise()

  def reset_parameters(self):
    mu_range = 1 / math.sqrt(self.in_features)
    self.weight_mu.data.uniform_(-mu_range, mu_range)
    self.weight_sigma.data.fill_(self.sigma_zero / math.sqrt(self.in_features))
    self.bias_mu.data.uniform_(-mu_range, mu_range)
    self.bias_sigma.data.fill_(self.sigma_zero / math.sqrt(self.out_features))

  def _scale_noise(self, size):
    x = torch.randn(size)
    return x.sign().mul_(x.abs().sqrt_())

  def reset_noise(self):
    epsilon_in = self._scale_noise(self.in_features)
    epsilon_out = self._scale_noise(self.out_features)
    self.weight_epsilon.copy_(epsilon_out.ger(epsilon_in))
    self.bias_epsilon.copy_(epsilon_out)

  def forward(self, input):
    return F.linear(input, self.weight_mu + self.weight_sigma * self.weight_epsilon, self.bias_mu + self.bias_sigma * self.bias_epsilon)

However, with the default hyperparameters from the (sigma_0 = 0.5), the agent does not explore at all, and even if I crank it up to sigma_0 = 5, it works way worse than epsilon greedy.
(When I use noisy nets I don't use epsilon greedy).

I think the problem is that you are not defining the weights and biases as parameters. So, when you backpropagate, they are not modified.

These lines should do the trick:

self.weight_mu = Parameter(torch.Tensor(out_features, in_features))
self.weight_sigma = Parameter(torch.Tensor(out_features, in_features))
self.bias_mu = Parameter(torch.Tensor(out_features))
self.bias_sigma = Parameter(torch.Tensor(out_features))

In case you are not familiar, the Parameter class must be imported from torch:

from torch.nn.parameter import Parameter


Any comparison between transformer and RNN+Attention on the same dataset? I am wondering what is believed to be the reason for superiority of transformer?
I see that some people believe because of the attention mechanism used, it’s able to capture much longer dependencies. However, as far as I know, you can use attention also with RNN  architectures as in the famous paper attention is introduced(here)).
I am wondering whether the only reason for the superiority of transformers is because they can be highly parallelized and trained on much more data?
Is there any experiment comparing transformers and RNN+attention trained on the exact same amount of data comparing the two?

If you go through the main introductory paper of the transformer ("Attention is all you need"), you can find the comparison of the model with other state-of-the-art machine translation method:

For example, Deep-Att + PosUnk is a method that has utilized RNN and attention for the translation task. As you can see, the training cost for the transformer with self-attention is $2.3 \cdot 10^{19}$ (FLOPs) and $1.0 \cdot 10^{20}$ (FLOPs) for the "Deep-Att + PosUnk" method (the transformer is 4 times faster) on "WMT14 English-to-French" dataset.
Please note that the BLEU is a crucial factor here (not merely training cost). Hence, you can see the BLEU value of the transformer superior to the ByteNet (Neural Machine Translation in Linear Time). Although the ByteNet has not adopted the RNN, you can find the comparison of the ByteNet with other "RNN + Attention" methods in its original paper:

Hence, by transitivity property of the BLEU score, you can find that the transformer has already outperformed other "RNN + Attention" methods in terms of the BLEU score (please check their performance on "WMT14" dataset).


Applying Eligibility Traces to Q-Learning algorithm does not improve results (And might not function well) I am trying to apply Eligibility Traces to a currently working Q-Learning algorithm.

The reference code for the Q-Learning algorithm was taken from this great blog by DeepLizard, but does not include Eligibility Traces. Link to the code on Google Colab.

I wish to add the Eligibility Traces by implementing this pseud code:

Initialize Q(s,a) arbitrarily and e(s,a) = 0, for all s,a
Repeat (for each episode):
    Initialize s,a
    Repeat (for each step of episode):
        Take action a, observe r,s’
        Choose a’ from s’ using policy derived from Q (e.g., ϵ-greedy)
        δ ← r + γ Q(s’,a’) – Q(s,a)
        e(s,a) ← e(s,a) + 1
        For all s,a:
            Q(s,a) ← Q(s,a) + α δ e(s,a)
            e(s,a) ← γ λ e(s,a)
        s ← s’ ; a ← a’
    until s is terminal

Taken from HERE

This is my code as I have implemented the pseudo-code - Link

The part that needs to be improved is here:

#Q learning algorithem
for episode in range(num_episodes):
  state = env.reset()
  et_table = np.zeros((state_space_size,action_space_size))
  done = False
  reward_current_episode = 0

  for steps in range(max_steps_per_episode):
    #Exploration-Explotation trade-off
    exploration_rate_thresh = random.uniform(0,1)
    if exploration_rate_thresh > exploration_rate:
      action = np.argmax(q_table[state,:])
    else:
      action = env.action_space.sample()

    new_state, reward, done, info = env.step(action)

    #Update Q-table and Eligibility table
    delta = reward + discount_rate * np.max(q_table[new_state,:]) - q_table[state,action]
    et_table[state, action] = et_table[state, action] + 1

    for update_state in range(state_space_size):
      for update_action in range(action_space_size):
        q_table[update_state, update_action] = q_table[update_state, update_action] + learning_rate * delta * et_table[update_state, update_action]
        et_table[update_state, update_action] = discount_rate * gamma * et_table[update_state, update_action]

    state = new_state
    reward_current_episode = reward

    if done==True:
      break

  #Exploration rate decay
  exploration_rate = min_exploration_rate + (max_exploration_rate - min_exploration_rate) * np.exp(-exploration_decay_rate*episode)

  rewards_all_episodes.append(reward_current_episode)

For a while, I was getting pure results (avg. rewards for 1000 episodes were around 0.14 while the original NON-ET algorithm was averaging 0.69 on the last 1000 episodes), but now I get these errors:

/usr/local/lib/python3.6/dist-packages/ipykernel_launcher.py:27: RuntimeWarning: overflow encountered in double_scalars
/usr/local/lib/python3.6/dist-packages/ipykernel_launcher.py:22: RuntimeWarning: invalid value encountered in double_scalars


Why is non-linearity desirable in a neural network? Why is non-linearity desirable in a neural network?
I couldn't find satisfactory answers to this question on the web. I typically get answers like "real-world problems require non-linear solutions, which are not trivial. So, we use non-linear activation functions for non-linearity".

Consider what happens if you intend to train a linear classifier on replicating something trivial as the XOR function. If you program/train the classifier (of arbitrary size) such that it outputs XOR condition is met whenever feature a or feature b are present, then the linear classifier will also (incorrectly) output XOR condition is met whenever both features together are present. That is because linear classifiers simply sum up contributions of all features and work with the total weighted inputs they receive. For our example, that means that when the weighted contribution of either feature is sufficient already to trigger the classifier to output XOR condition is met, then obviously also the summed contributions of both features are sufficient to trigger the same response.
To get a classifier that is capable of outputting XOR condition is met if and only if the summed contributions of all input features are above a lower threshold and below an upper threshold, commonly non-linearities are introduced. You could of course also try to employ a quadratic function to solve the two-feature problem, but as soon as the number of variables/features increases again, you run into the same problem again, only in higher dimensions. Therefore, the most general approach to solving this problem of learning non-linear functions, like the XOR, is by setting up large models with enough capacity to learn a given task and equipping them with non-linearities. That simplifies training since it allows for using stochastic gradient descent for training the system/classifier, preventing one from having to solve Higher-Degree polynomial equations analytically (which can get computationally quite expensive quite quickly) to solve some task.
In case you are interested, here's one paper analyzing and dealing with the XOR problem (as one concrete instance of a problem where purely linear models fail to solve some task).
EDIT:
You can consider a layer in a network as a function $y = f(x)$, where $x$ is the input to some layer $f$ and $y$ is the output of the layer. As you propagate $x$, being the network's input, through the network, you get something like $y = p(t(h(g(f(x)))))$, where $f$ is the input layer and $p$ constitutes the output layer, i.e. a set of weights, by which the input to that respective layer gets multiplied.
If $h$, for example, is some non-linear activation function, like ReLU or sigmoid, then $y$, being the network's output, is a non-linear function of input $x$.


Why is Common Lisp, Python and Prolog used in artificial intelligence? What are the advantages/ strengths and disadvantages/weakness of programming languages like Common Lisp, Python and Prolog? Why are these languages used in the domain of artificial intelligence? What type of problems related to AI are solved using these languages?

Please, give me link to papers or books regarding the mentioned topic.

If we talk about applied AI, the choice of a programming language for an AI application has the same points to be taken into account that in any other software area: speed of generated code, expressive, reusable, etc.

By example, as training of a neural net is very expensive in CPU, languages as C/C++, that produces very optimized code, are very convenient. Moreover, there are GPUs librarians in C/C++ that allows use of strong parallelism.

A system with some complexity will combine more than one language, in order to use the best of the language in the points where it is need.

But returning to the list of languages that appears in the question. As all them are Turing complete compare them means talk about its paradigm, features, syntax and available compilers/interpreters. Obviously, something that exceeds the possibilities of a simple answer. Just to show some key points about the ones mentioned in the question:

Prolog is a programming paradigm by itself. Its main advantage was that Prolog sentences are independent from remainder ones and near to the mathematical definitions of the concepts. Moreover, it is itself a database. Its drawback are also well known: very slow, lack of librarians for i/o, ... . Very interesting (even mandatory) to known a few examples of algorithms in Prolog, but I doubt nobody is using it nowadays, except in obsolete university courses (when you reach the "!", cut its study).

Lisp is also a zombie. Its functional paradigm has been now included in lots of very more modern languages, combining it with object oriented paradigm: scala, haskell, ocaml/F#, ... . Being functional allows a syntax that made easier to express logic concepts as recursive definition of logic or types, ... . Something very interesting in AI.

In the category of object oriented paradigm and valid for all applications, we have Python ( easy to learn, fast prototyping, slow, ... ) C/C++ (very optimized code), Java, ... . More or less, all them are adopting also functional features in latest standards.

In AI there are a lot of very interesting language features to be also considered: rule based systems, ... . Librarians for them can be found in all main languages.

Finally, some words about AGI (strong AI): you do not need a computer. In best moments, we are at the stage of pencil and paper, remainder ones looking at ceiling.


Getting loss from each sample in CNN (python) I have to plot a cumulative density graph for the performance of my CNN model.
But the problem is to do that, I have to get all the losses for each sample in the validation set but it seems like tensorflow only shows the final losses for each epoch.
Is there any way I can get all the losses for all the samples?

You may use Keras model which is provided in TensorFlow. Keras model lets you evaluate a batch of different number of samples compared to when training.
For example, when training:
myModel.fit(x=X, y=Y, batch_size=BATCH_SIZE, epochs=EPOCHS, verbose=1)

Evaluate every single sample to get output loss of each sample (https://www.tensorflow.org/api_docs/python/tf/keras/Model#evaluate):
losses = []

for i in range(len(X)):
    sampleX = X[i]
    sampleY = Y[i]
    loss = myModel.evaluate(x=[sampleX], y=[sampleY], batch_size=1, steps=1, verbose=0)
    losses.append(loss)

Another way is to use Keras callback during multiple sample evaluation (https://www.tensorflow.org/api_docs/python/tf/keras/callbacks/Callback#on_batch_end):
losses = []

class myCallback(tf.keras.callbacks.Callback):
    def on_batch_end(batch,log):
        global losses
        losses.append(log["loss"])

myModel.evaluate(x=X, y=Y, batch_size=1, steps=NUM_SAMPLES, callbacks=[myCallback()])
```


How does Monte Carlo Exploring Starts work? I'm having trouble understanding the 5th step in the flowchart.

For the 5th step, the 'update the Q function by taking the average of returns' is confusing.

From what I understand, the Q function is basically the state-action pair values put in a table (the Q table). To update it means to make adjustments to the state-action pair value of the individual states and their respective actions (e.g state 1 action 1, state 3 action 1, state 3 action 2, so on and so forth).

I'm not sure what 'average of returns' means though. Is it asking me to take the average of the returns after $x$ episodes?  From my understanding, returns is the sum of rewards in a full episode (So, AVG=sum of returns for x episodes/x).

And what do I do with that 'average'?

I'm a little confused when they say 'update the Q function' because the Q function consists of many parameters that must be updated (the individual state-action pair value), and I'm not sure which one they are referring to.

What is the point of calculating the average of returns? Since the state-action pair value for a particular state and particular action will always be the same (e.g if I always take action 3 in state 4, I will always get value=2 forever)

each episode you will calculate the return, you will then update the action value or $Q(s,a)$  as the average each episode. Using the blackjack example from open AI gym and using a discount factor of 1, you get the following

episode 1
[{'state': (22, 10, False), 'reward': -1, 'action': 1}, {'state': (17, 10, False), 'reward': 0, 'action': 1}, {'state': (12, 10, False), 'reward': 0.0, 'action': 1}]

$Q((22, 10, False),0)=-1$

$Q((17, 10, False),1)=-1$

$Q((12, 10, False),1)=-1$

episode 2
[{'state': (21, 10, False), 'reward': 1, 'action': 0}, {'state': (17, 10, False), 'reward': 0, 'action': 1}, {'state': (12, 10, False), 'reward': 0.0, 'action': 1}]

$Q((21, 10, False),0)=1$

$Q((17, 10, False),1)=0$

$Q((12, 10, False),1)=0$

For $Q((17, 10, False),1)$ and $Q((12, 10, False),1)$ is the average return
i.e -1 for the first episode and 1 for the second.


Why are state transitions in MDPs probabilistic rather than deterministic? I've read that for MDPs the state transition function $P_a(s, s')$ is a probability. This seems strange to me for modeling because most environments (like video games) are deterministic.

Now, I'd like to assert that most systems we work with are deterministic given enough information in the state (i.e. in a video game, if you had the random number seed, you could predict 'rolls', and then everything else follows game logic).

So, my guess for why would MDP state transitions are probabilities is because the state given to the MDP is typically a subset (i.e. from feature engineering) of total information available. That, and of course to model non-deterministic systems.

Is my understanding correct?

Your understanding is right!

Using a probabilistic transition function allows the model to explore a bigger search space before making a decision. One of the most important use cases of MDP is in POS tagging in NLP using a Hidden Markov Model.

In case of a deterministic model, search space is limited by the number of transitions and hence at each step, a definite decision is made. This does not take into regard the possibility of relationship between the previous states, rather only deals with the current and next state. These model are good for solving a certain range of tasks like decision trees etc.

When it come to tasks such as weather prediction, there is significance to the historical weather data. In such cases, we cannot make use of a deterministic approach. You always predict the chance of rainfall etc.

This example can also be extended to predict the weather for future days


What are the scientific journals dedicated to artificial general intelligence? Apart from Journal of Artificial General Intelligence (a peer-reviewed open-access academic journal, owned by the Artificial General Intelligence Society (AGIS)), are there any other journals (or proceedings) completely (or partially) dedicated to artificial general intelligence?

If you want to share a journal that is only partially dedicated to the topic, please, provide details about the relevant subcategory or examples of papers on AGI that were published in such a journal. So, a paper that talks about e.g. an RL technique (that only claims that the idea could be useful for AGI) is not really what I am looking for. I am looking for journals where people publish papers, reviews or surveys that develop or present theories and implementations of AGI systems. It's possible that these journals are more associated with the cognitive science or neuroscience communities and fields.

There's also the journal Advances in Cognitive Systems. According to their website

Advances in Cognitive Systems (ISSN 2324-8416) publishes research articles, review papers, and essays on the computational study of human-level intelligence, integrated intelligent systems, cognitive architectures, and related topics. Research on cognitive systems is distinguished by a focus on high-level cognition, reliance on rich, structured representations, a systems-level perspective, use of heuristics to handle complexity, and incorporation of insights about human thinking. Advances in Cognitive Systems reviews submissions within approximately three months and publishes accepted papers on the journal Web site immediately upon receipt of final versions. Articles are distributed freely over the internet by the Cognitive Systems Foundation.


How to find the optimal pokemon team Pokemon is a game where 2 players each select 6 Pokemon (a team) at the beginning of the game without knowing the other player's team. Every Pokemon has one or two types. Every type is either weak, neutral or strong against every other type. This means that every 2 Pokemon matchup will either have a winner or be a tie. This also means that any team can be ranked against any other team based on the number of winning matchups they have.
I want to write a program that can find the optimal Pokemon team out of a set of 70 provided Pokemon.
A team is considered optimal if it has the greatest number of winning matchups against any other team. Basically, I want to calculate which team will have the most amount of favorable matchups if you were to battle it against every other possible team.
What algorithm would be best for doing this? It is not feasible to compute matchups for every possible team. Can I do some sort of A* search with enough pruning to make it computationally feasible?

After my initial comment (where I suggest that it might not be enough info) I believe I actually came up with an idea.
Start with the full set of pokemon. For every possible type, identify the count of pokemon that are strong against that type. For this, you'll end up with a List<(pokemonId, types, List<weakAgainst>)>.
Minimize List<weakAgainst>.Count() and from the possible set of pokemonIds, select one at random. Without knowing anything else besides type, this pokemon is as good as any other with the same weakness count (this is the point of my original comment).
From the list of weaknesses that this selected pokemon has, select a pokemon from your list that is strong against the weakness, minimizing the amount of weaknesses again. Likely more than one will match this criteria, again, select one at random.
Keep repeating this pattern until you obtain the 6 in your team. This is, statistically speaking, one of the best teams that you can gather.
For all the combinations that you might find here, some teams will have less weaknesses, since we're "randomly" walking down a tree of possibilities. This very much sounds like a minimax-prunning algorithm, where each pokemon selection (minimizing your weaknesses) can be met with potential opponents that will maximize your weak points.
Simplified, put together:
input: allPokemon: list<pokemonId, weakAgainst, strongAgainst>

var: teamWeakAgainst: []
var: teamStrongAgainst: []
var: selectedTeam: []

while (size(selectedTeam) < 6)
  goodMatches <- allPokemon.max(p -> size(p.strongAgainst.intersect(teamWeakAgainst)))
  goodMatches <- goodMatches.min(p -> size(p.weakAgainst))
  goodMatches <- goodMatches.max(p -> size(p.strongAgainst))

  selectedPokemon <- goodMatches.random()

  teamWeakAgainst -= selectedPokemon.strongAgainst
  teamWeakAgainst += selectedPokemon.weakAgainst # not counting previously selected pokemon because the current one adds another "weakness", even if it was already accounted for

  selectedTeam += selectedPokemon

output: selectedTeam

From this algorithm is it not obvious where the "max" portion is. We're minimizing our losses (weaknesses) but we're considering all possible opponent teams equally, so there is no real maximization of the opponent choices. For a set of ideas, check below.
Note that this algorithm will give you a set of "teams" that are equally good in the sense that they'll have the same amount of minimized weaknesses and maximized strengths against other possible teams. But even if pokemon are different, the numbers will be the same, just different types.
For a more complex approach, you might want to consider how prevalent some pokemon are (you might not need to optimize against a super rare mythical type, but rather the very common types available in the game), how likely is it that certain pokemon can have better / faster attacks, what is the probability of battle IVs, how frequent can a trainer switch pokemon in battle, etc. Again, I know this is not what you asked for, but for the sake of the example, this will become so complex that instead of a search algorithm, a simulation (Monte Carlo?) approach might be simpler to build teams out of statistical testing.


Why my test error is lower then train error I am trying to train a CNN regression model using the ADAM optimizer, dropout and weight decay.

My test accuracy is better than training accuracy.
But as I know, usually train accuracy is better than test accuracy.

So I wonder how this is happening.

You use dropout during traing to reduce overfitting, but this reduces the training accuracy. The dropout will not be used during testing, therefore the accuracy will be higher.

That's normal behavior if you work with dropout.


Is it possible to use a trained neural network to predict a feature, given other features and output? I have a neural network that is already trained to predict two continuous outputs from a set of 7 continuous features.

Is there any way to apply the network to predict one of the input features, given other 6 features and the two outputs?

From your question, it appears that you would like to use other features in your data to predict one of the features. I am not sure I understood your question clearly, but anyways, either, you would be using the feature you want to predict as the output of the network.

Also, if you want to use the output of the network and other features to predict the new output, I think possibly you are trying to use a Recurrent Neural Network based approach, in which the past output is taken into account for future predictions. My personal experience with RNN's is that they are really good at learning such dependencies, as the RNN cells consider the present input, as well as the past output in modelling such tasks. If your problem is a sequence to sequence prediction task, I would definitely suggest trying RNN's (especially LSTMs, ideal for learning long term temporal dependencies) but ultimately, there is No Free Lunch. So try different approaches and see which one works for you best. Good Luck!


Query regarding the minmax loss function formulation of the training of a Generative Adversarial Network (GAN) Just needed a clarification on the training procedure for a standard GAN.
Of my understanding the loss function to optimize is a min max (max min causing mode collapse due to focus on one class generation) problem where the loss function

needs to maximized for the discriminator and minimized for the generator networks -

1.) In this equation are the $E_{z~p_{z(z)}}$ and $E_{x~p_{data(x)}}$ the means of the distributions of the mini batch samples? Also is the optimal case for the discriminator a maximum value of 0? and the optimal case for the generator a minimum value of $log(small~value)$ {basically a massive negative value}? If so, what happens to the first term during the training of the generator - is it taken as a constant or is the discriminator performing badly considered optimal for the generator?

2.) While putting this in code, for every step is the discriminator trained first for one step, keeping the generator constant, followed by the generator being trained for the same step, with the discriminator kept constant?

3.) In every step of training are there multiple latent vectors sampled to produce multiple generator outputs for each step? If so is the loss function an average or sum of all $V(D, G)$ for that step

I'll answer your questions one by one:

  In this equation are the $E_{z \sim p_z(z)}$ and $E_{x \sim p_{data}(x)}$ the means of the distributions of the mini batch samples?

So let's take the first part $E_{x \sim p_{data}(x)}[log \,D(x)]$. This is read as the "expected value of $log \, D(x)$, where $x$ is sampled from $p_{data}(x)$". So, in simpler terms this means that:

You have a distribution $p_{data}(x)$.
You sample a batch of samples $x$ from this distribution.
You feed the batch to the discriminator and get its output $D(x)$.
You compute the log for the batch of predictions.
You average over the samples in the batch.

  Also is the optimal case for the discriminator a maximum value of 0?

Yes, the goal of the discriminator is to maximize $V(D, G)$. This is achieved when $D(x) = 1$ and $D(z) = 0$, where $V(D, G)=0$.

  the optimal case for the generator a minimum value of log(small value) {basically a massive negative value}?

Yes, the generator wants to minimize $V(D, G)$, so it wants $D(x)$ and $D(z)$ to be as small as possible (though it can affect only the second term). The minimum value for $D(z)$ is $1 / M$, where $M$ is the number of classes in the dataset. So the minimum value for $V(D, G)$ is $\log{[(M-1) / M^2]}$ (I think).

  If so, what happens to the first term during the training of the generator - is it taken as a constant or is the discriminator performing badly considered optimal for the generator?

The first term is of no consequence to the Generator, because the generator can't affect it somehow (i.e. like you say it is taken as a constant). Consider the generator $G_{\theta}$ has parameters $\theta$ and the discriminator $D_{\phi}$ has parameters $\phi$. When training $G$:

$$
\frac{\log{D_{\phi}(x)}}{\partial \theta} = 0
$$

So the generator is trained of $V$.

  While putting this in code, for every step is the discriminator trained first for one step, keeping the generator constant, followed by the generator being trained for the same step, with the discriminator kept constant?

In theory, yes. In practice for every step the Generator takes, we train the Discriminator for more steps (e.g. $5$ steps).

  In every step of training are there multiple latent vectors sampled to produce multiple generator outputs for each step? If so is the loss function an average or sum of all V(D,G) for that step.?

Yes, at every step we sample a batch of latent vectors. The equation indicates that we average over these (i.e. the expectation) but in practice it doesn't have any difference (the average is the sum divided by a constant number, which doesn't affect the optimization very much).


Understanding multi-iteration updates of the model in the Proximal Policy Optimization algorithm I have a general question about the updating of the network/model in the PPO algorithm.
If I understand it correctly, there are multiple iterations of weight updates done on the model with data that is created from the environment (with the model before the update). Now, I think that the updates of the model weights are not correct anymore after the first iteration/optimization step, because the model weights changed and therefore the training data is outdated (since the model would now give different actions in the environment and therefore different rewards).
Basically, in the pseudo-code of the algorithm, I don't understand the line "Optimize surrogate L ... with K epochs...". If the update is done for multiple epochs, the data that is learned is outdated already after the first iteration of optimization, since the model's weights changed. In other algorithms, like A2C, there is only one optimization step done, instead of $K$ epochs.

Is this some form of approximation or augmentation on the data by using the data that was created by an older model for multiple iterations or am I missing something here? If yes, where was this idea first introduced or better described? And where is an (empirical) proof that this still leads to a correct weight updating?

Seems this question was asked and answered in openai/baselines github issue. The issue has been closed for a while.
Below is an answer provided by @matthiasplappert which has the most "thumbs up":

To clarify: PPO is an on-policy algorithm so you are correct that going over the same data multiple times is technically incorrect.
However, we found that PPO is actually quite okay with doing this and we still get stable convergence. This is likely due to the proximal trust region constrained that we enforce, which means that the policy cannot change that much anyway when going over the current set of transitions multiple times, making it still approximately on-policy. You can of course get rid of this but then you'll need more samples.


Can a game AI learn the concept of acceleration? If the game had a variable speed and was essential in evolution/gaining score(IDK AI terminologies). Would the AI be able to figure out when to slow down and speed up?

If it is able to solve the problem or complete the level, will it have an equation to relating acceleration, or perhaps a number on when to speed up and down. What if the game environment was dynamic?

Can you even teach math to an AI?

PS: I'm not sure if I should ask separate question?

This answer mostly assumes you are referring to computer-game-playing bots that learn through experiencing play, such as Deep Mind's DQN as used for playing Atari console games. State of the art for these are typically Reinforcement Learning algorithms, used with neural networks to process input and estimate results of next actions. There are other competitive AI technologies too, and the answer applies generally to most learning or evolving optimisers that would learn through trial and error by playing the game.

  If the game had a variable speed and was essential in evolution/gaining score(IDK AI terminologies). Would the AI be able to figure out when to slow down and speed up?

Yes, as long as the game allowed control of something that influenced acceleration, then a learning agent can figure out the consequences of accelerating and braking and use them appropriately within the game.

A well-known toy example that can challenge learning agents is called Mountain Car. In that game, the agent has to learn to accelerate in the correct direction (which is not always towards its objective), in order to escape an area. It is considered challenging because the reward (for escaping) can be significantly delayed compared to the action that best enables it.

One popular proving ground for learning agents is OpenAI gym. This includes several game environments with a physical model of acceleration included, such as Lunar Lander.

  If it is able to solve the problem or complete the level, will it have an equation to relating acceleration?

In general, no. The agent will learn to respond to certain stimulus, by taking an action that accelerates or decelerates the game piece that it controls. There will not be any concept like $s = ut + \frac{1}{2} at^2$ encoded in the agent's parameters.

  or perhaps a number on when to speed up and down?

Typically the agent will learn which stimuli should be responded to by accelerating. For instance, in a game where the agent's game piece is being chased by an enemy piece and the enemy piece is getting closer, the agent should learn that it will get a better reward if it accelerates away from the enemy.

  What if the game environment was dynamic?

Most game environments are dynamic, as in the state changes over time. If you mean would anything change if the game rules themselves varied over time, then this may cause interesting problems for some learning algorithms, but should not change anything about learning use of controls that affect acceleration in a virtual world.

  Can you even teach math to an AI?

Generally, no you cannot teach math to the kind of system that plays games, or interacts with real world objects. These kind of learning systems are not yet advanced enough to learn concepts or establish game world logic from interactions. Instead, they work more akin to perception, muscle memory and either inherent or learned reflexes. Exceptions to this will generally have a world model (with the necessary equations) built in or made available to the agent without it needing to learn anything.

However, there are AI systems that use formal logic that can work on mathematical theories. Some have performed interesting feats such as "discovering" prime numbers, given formal definitions of integers and basic arithmetic. An example of this kind of system is the Automated Mathematician.

There is an intermediate possibility: Some learning agents not only learn the value or best policy for certain actions, they also learn to predict what should happen next to the state of the environment. Such an agent would include a model that could observe objects that were accelerating and predict their future positions. In some ways this is a learned concept of acceleration, although it would not be expressed mathematically like Newton's laws of motion, and is more akin to the kind of intuition that allows a person to track, anticipate and catch a thrown ball.