Is an artificial intelligence software or hardware? Is an artificial intelligence software? (a program or a set of programs) or can it be hardware?

AI or Artificial Intelligence is nothing but intelligence but in its artificial form. Intelligence comes from formation of rules and patterns in the data which is seen or on which it is trained.

For us, we programme or formulate these rules in Mathematics on which an Intelligence is created.
Ex. A neural network shows a lower level of Intelligence. It is created using basic rules of mathematics like gradient descent, derivatives and other arithmetic stuff.
These rules need a base to form themselves. Here's where computers come in. They run these rules on which intelligence thrives in binary ( or quantum sometimes ).
But, every bit of information acquires physical space for processing and storage. Here's where hardware comes. Hardware provides the container in which  information will be processed.

Hence, for an complete AI system, computer programmes provide a soul whereas hardware provides a body/physical form for the soul.

Since, only AI is concerned, it may run on any hardware as it is a software or program or a set of mathematical rules in binary state.
Hence AI is a program.


If $h_1(n)$ is admissible, why does A* tree search with $h_2(n) = 3h_1(n)$ return a path that is at most thrice as long as the optimal path? Consider a heuristic function $h_2(n) = 3h_1(n)$. Where $h_1(n)$ is admissible.
Why are the following statements true?

$A^*$ tree search with $h_2(n)$ will return a path that is at most thrice as long as the optimal path.
$h_2(n)  + 1$ is guaranteed to be inadmissible for any $h_1(n)$

The sketch of the proof for your first question:
for an open node $n$, if $f_1(n) = g(n) + h_1(n)$, in the same situation in using $h_2$, it will be $f_2(n) = g(n) + 3  h_1(n)$. Hence, all the time for any node $n$, $f_2(n) \leqslant 3f_1(n)$. On the other hand, we know that A* with the admissible husritic function $h_1$ will be admissible (from Theorem 2 of Chapter 3 in "Heuristics Intelligent Search Strategies for Computer Problem Solving" book by Judera Pearl), i.e., for the node $n^*$ with optimal value, $f_1(n^*) \leqslant C^*$ that $C^*$ is the optimal value. Therefore, A* with $h_2$ will return a solution in node $n'$ by the cost of $f_2(n') \leqslant 3 f_1(n^*) \leqslant 3C^*$, as $f_1(n') \leqslant 3 f_1(n^*)$ (see more details of the proof in Theorem 13 of Chapter 13 in the same reference).
You can find more about $h_2$ under the title of $\epsilon$-admissibility as it is $(1 + \epsilon) h_1$ that $h_1$ is an admissible heuristic function. In your case, $\epsilon = 2$.


What is the input to AlphaGo's neural network? I have been reading an article on AlphaGo and one sentence confused me a little bit, because I'm not sure what it exactly means. The article says:

  AlphaGo Zero only uses the black and white stones from the Go board as its input, whereas previous versions of AlphaGo included a small number of hand-engineered features.

What exactly is the input to AlphaGo's neural network? What do they mean by "just white and black stones as input"? What kind of information is the neural network using? The position of the stones?

The input to the neural network is a $19 × 19 × 17$ image stack
  comprising $17$ binary feature planes. $8$ feature planes $X_t$ consist of binary values indicating the
  presence of the current player’s stones ($X^i_t = 1$ if intersection $i$ contains a stone of the player’s
  colour at time-step $t$; $0$ if the intersection is empty, contains an opponent stone, or if $t < 0$). A
  further $8$ feature planes, $Y_t$
  , represent the corresponding features for the opponent’s stones. The
  final feature plane, $C$, represents the colour to play, and has a constant value of either $1$ if black
  is to play or $0$ if white is to play. These planes are concatenated together to give input features
  $s_t = [
X_t, Y_t, X_{t−1}, Y_{t−1}, ..., X_{t−7}, Y_{t−7}, C]$.

This and all the other architecture details can be found in the "Neural Network Architecture" section in the paper.


Should image augmentation be applied before or after image resizing? For the purposes of training a Convolutional Neural Network Classifier, should image augmentation be done before or after resizing the training images?
To reduce file size and speed up training time, developers often resize training images to a set height and width using something like PIL (Python Imaging Library).
If the images are augmented (to increase training set size), should it be done before or after resizing the members of the set?
For simplicity sake, it would probably be faster to augment the images after resizing, but I am wondering if any useful data is lost in this process. I assume it may depend on the method used to resize the images (cropping, scaling technique, etc.)

As it is told in PIL documentation
It uses some filters to resize images.And those filters are explained here
uses mostly numerical methods as I see. So it is approximating the image data input. Which means you are right about data loss. But here might be the question would it change the data so much If it is done after augmentation or before?
Since in numerical methodic approaches the more values means more valid approximation in general.
So it might be beneficial to augment and then resize.
But one should look for its real answer. Mine is just a thought experiment. You will get better proven answers


Is Experience Replay like dreaming? Drawing parallels between Machine Learning techniques and a human brain is a dangerous operation. When it is done successfully, it can be a powerful tool for vulgarisation, but when it is done with no precaution, it can lead to major misunderstandings.
I was recently attending a conference where the speaker described Experience Replay in RL as a way of making the net "dream".
I'm wondering how true this assertion is. The speaker argued that a dream is a random addition of memories, just as experience replay. However, I doubt the brain remembers its dream or either learns from it. What is your analysis?

The speaker argued that a dream is a random addition of memories, just as experience replay.

The speaker is taking some liberties due to a general lack of scientific understanding of what dreams are. We don't even have strong consensus on why sleep is a necessary feature of animals, let alone what part dreaming plays in it. However, there are some widely-accepted theories, with supporting evidence, that dreams are part of a learning and memorisation process. Studies that manipulate sleep or dreaming have shown changes in the speed that skills are learned for example.

Experience replay in reinforcement learning is a far more precise and well-understood affair, whereby individual time steps that occurred in the past are visited and re-assessed in light of current knowledge about long-term value, at random. If dreams were really like experience replay as it is practiced in RL today, then they would consist of a random jumble of tiny seemingly inconsequential events strung together, and all taken very exactly from the events of the past day. Sometimes dreams do contain content like this, but typically the content is far more varied.

Taken with a large dose of artistic license, then yes, the speaker is referring to real theories and conjectures about dreaming, that do have scientific support. Although it is equally good to draw parallels between dreams and a higher-level management of the memory or experience replay data - which items to replay, and which to keep, depending on what is salient about the information. For instance, there is good evidence that dreams help filter what is forgotten, and also evidence that events associated with strong emotional state are more likely to feature in dreams.

It is important to separate the speaker's analogy, and any suggestion that a current reinforcement learning agent has a subjective experience. We are still a long way away from anything like that, and other similar use of a dreaming metaphor in machine learning - e.g. "Deep Dream" - is equally not an assertion that the devices are having an experience of any kind.


Why is the generation of deep style images so slow and resource-hungry? Consider these neural style algorithms which produce some art work:

Neural Doodle
neural-style

Why is generating such images so slow and why does it take huge amounts of memory? Isn't there any method of optimizing the algorithm?

What is the mechanism or technical limitation behind this? Why we can't have a realtime processing?

Here are few user comments (How ANYONE can create Deep Style images):

  Anything above 640x480 and we're talking days of heavy crunching and an insane amount of ram.

  I tried doing a 1024pixel image and it still crashed with 14gigs memory, and 26gigs swap. So most of the VM space is just the swapfile. Plus it takes several hours potentially days cpu rendering this.

  I tried 1024x768 and with 16gig ram and 20+ gig swap it was still dying from lack of memory.

  Having a memory issue, though. I'm using the "g2.8xlarge" instance type.

Real time style transfer and neural doodle is very much possible and is an active topic I see users working on to improve upon. The basic idea is to do only feed forward propagation at test time and train with appropriate loss functions at train time.

Perceptual Losses for Real-Time Style Transfer
and Super-Resolution is a good starting point to understand a methodology for this purpose.


Finding patterns in binary files using deep learning I am a newbie in deep learning and wanted to know if the problem I have at hand is a suitable fit for deep learning algorithms. I have thousands of fragments each of about 1000 bytes size (i.e. numbers in the range of 0 to 255). There are two classes in the fragments:

Some fragments have a high frequency of two particular byte values appearing next to one another: "0 and 100". This kind of pattern roughly appears once every 100 to 200 bytes.
In the other class, the byte values are more randomly distributed.

We have the ability to produce as many numbers of instances of each class as needed for training purposes.  However, I would like to differentiate with a machine learning algorithm without explicitly identifying the "0 and 100" pattern in the 1st class myself. Can deep learning help us solve this? If so, what kind of layers might be useful?

As a preliminary experiment, we tried to train a deep learning network made up of 2 hidden layers of TensorFlow's "Dense" layers (of size 512 and 256 nodes in each of the hidden layers). However, unfortunately, our accuracy was indicative of simply a random guess (i.e. 50% accuracy). We were wondering why the results were so bad. Do you think a Convolutional Neural Network will better solve this problem?

Your network is essentially memorizing data but not extracting features. You need to apply CNN.

That said the CNN architecture will need to be kind of unusual, each bit will need to be turned into representing a positional element.


LSTM is not converging I am writing my first LSTM network and I would really appreciate if someone can tell me if it is right (the loss seems to go down very slowly and before playing around with hyper parameters I want to make sure that the code is actually doing what I want). The code is meant to go through some time series and label each point according to some categories. In the version I am putting here there are just 2 categories: 0, if the value of the point is 1 and 1 otherwise (I know it’s a bit weird, but I didn’t choose the labels). So this is the code:

import torch
import torch.nn as nn
import torch.nn.functional as F
from torch.autograd import Variable
import torch.optim as optim
import numpy as np
from fastai.learner import *

torch.manual_seed(1)
torch.cuda.set_device(0)

bs = 2

x_trn = torch.tensor([[1.0000, 1.0000],
        [1.0000, 0.9870],
        [0.9962, 0.9848],
        [1.0000, 1.0000]]).cuda()

y_trn = torch.tensor([[0, 0],
        [0, 1],
        [1, 1],
        [0, 0]]).cuda()

n_hidden = 5
n_classes = 2

class TESS_LSTM(nn.Module):
    def __init__(self, nl):
        super().__init__()
        self.nl = nl
        self.rnn = nn.LSTM(1, n_hidden, nl)
        self.l_out = nn.Linear(n_hidden, n_classes)
        self.init_hidden(bs)

    def forward(self, input):
        outp,h = self.rnn(input.view(len(input), bs, -1), self.h)
        return F.log_softmax(self.l_out(outp),dim=1)

    def init_hidden(self, bs):
        self.h = (V(torch.zeros(self.nl, bs, n_hidden)),
                  V(torch.zeros(self.nl, bs, n_hidden)))

model = TESS_LSTM(1).cuda()

loss_function = nn.NLLLoss()

optimizer = optim.Adam(model.parameters(), lr=0.01)

for epoch in range(10000):
    model.zero_grad()
    tag_scores = model(x_trn)
    loss = loss_function(tag_scores.reshape(4*bs,n_classes), y_trn.reshape(4*bs))
    loss.backward()
    optimizer.step()

    if epoch%1000==0:
        print("Loss at epoch %d = " %epoch, loss)

print(model(x_trn), y_trn)

The (super reduced in size) time series should be [1,1, 0.9962,1], with labels [0,0,1,0] and [1, 0.9870, 0.9848,1] with labels [0,1,1,0] and the batch size should be 2. I really hope I didn’t mess up the dimensionalities, but I tried to make it in a shape accepted by the LSTM. This is the output:

Loss at epoch 0 = tensor(1.3929, device='cuda:0', grad_fn=&lt;NllLossBackward&gt;)
Loss at epoch 1000 = tensor(0.8939, device='cuda:0', grad_fn=&lt;NllLossBackward&gt;)
Loss at epoch 2000 = tensor(0.8664, device='cuda:0', grad_fn=&lt;NllLossBackward&gt;)
Loss at epoch 3000 = tensor(0.8390, device='cuda:0', grad_fn=&lt;NllLossBackward&gt;)
Loss at epoch 4000 = tensor(0.8339, device='cuda:0', grad_fn=&lt;NllLossBackward&gt;)
Loss at epoch 5000 = tensor(0.8288, device='cuda:0', grad_fn=&lt;NllLossBackward&gt;)
Loss at epoch 6000 = tensor(0.8246, device='cuda:0', grad_fn=&lt;NllLossBackward&gt;)
Loss at epoch 7000 = tensor(0.8202, device='cuda:0', grad_fn=&lt;NllLossBackward&gt;)
Loss at epoch 8000 = tensor(0.8143, device='cuda:0', grad_fn=&lt;NllLossBackward&gt;)
Loss at epoch 9000 = tensor(0.8108, device='cuda:0', grad_fn=&lt;NllLossBackward&gt;)

(tensor([[[-9.0142e-01, -1.2631e+01],
          [-9.3762e-01, -9.6707e+00]],

         [[-1.3467e+00, -3.9542e+00],
          [-2.2005e+00, -7.6977e-01]],

         [[-2.4500e+01, -1.9363e-02],
          [-2.3349e+01, -6.2210e-01]],

         [[-1.0969e+00, -2.1953e+01],
          [-6.9776e-01, -1.8608e+01]]], device='cuda:0',
        grad_fn=<LogSoftmaxBackward>), tensor([[0, 0],
         [0, 1],
         [1, 1],
         [0, 0]], device='cuda:0'))

The loss doesn’t go down too fast (I expected it to overfit and go really close to zero). The actual values are okish (the smaller one is always the right one), but they can be definitely improved. Can someone tell me if my code is doing what I want (and maybe suggest why is the loss still big - maybe I need a smaller LR?)

I tried to play with your code and found changing loss function to the cross_entropy alternate of negative log-likelihood makes the difference between 2000th epoch's loss and  9000th epoch's loss is greater about 0.2 alternate of 0.09

I also tried to change optimizer and learning rate but no loss didn't improve.

you can explore the modified code may help you with another idea


What are applications of object/human tracking in autonomous cars? Objects tracking is finding the trajectory of each object in consecutive frames. Human tracking is a subset of object tracking which just considers humans.

I've seen many papers that divide tracking methods into two parts:

Online tracking: Tracker just uses current and previous frames.
Offline tracking: Tracker uses all frames.

All of them mention that online tracking is suitable for autonomous driving and robotics, but I don't understand this part. What are the applications of object/human tracking in autonomous driving?

Do you know some related papers?

There are two important parts:

When you track, you have a trajectory you can expect. So it becomes easier to "look into the future" is you understand the past
When you loose an object you had a trajectory of before, you might be able to come up with an explanation that can influence your behavior


Interesting examples of discrete stochastic games SGs are a generalization of MDPs to multiple agents. Like this previous question on MDPs, are there any interesting examples of zero-sum, discrete SGs—preferably with small state and action spaces? I'm hoping to use such examples as benchmarks, but couldn't find much in the literature. One example I can think of is a pursuit-evasion game on a graph.

Some of the domains in the International Probabilistic Planning Competition, such as the Wildlife Preserve benchmark, fit quite well the constraints you have given. Note that the problems are modeled with a high-level declarative language, RDDL. This means that you can define problems as big or as small as your heart desires with relative ease, since you can parametrize state description in terms of functions describing properties of an arbitrary number of objects.

There's also a quite useful project that allows to instance OpenAI gym environments from the declarative description of the environment, states and actions.


Using True Positive as a Cost Function I wanted to use True Positive (and True Negative) in my cost function to make to modify the ROC shape of my classifier. Someone told me and I read that it is not differentiable and therefore not usable as a cost function for a neural network.

In the example where 1 is positive and 0 negative I deduce the following equation for True Positive ($\hat y = prediction, y = label$):

$$ TP = \bf(\hat{y}^Ty) $$
$$ \frac{\partial TP}{\partial \bf y} = \bf y $$

The following for True Negative:
$$ TN = \bf(\hat{y}-1)^T(y-1) $$
$$ \frac{\partial TN}{\partial \bf y} = \bf \hat y^T -1    $$

The False Positive:
$$ FP = - \bf  (\hat y^T-1) y $$
$$ \frac{\partial FP}{\partial \bf y} = - \bf ( \hat y^T - 1)    $$

The False Negative:
$$ FN = \bf  \hat y^T (y-1) $$
$$ \frac{\partial FN}{\partial \bf y} = \bf \hat y    $$

All equations seem differentiable to me. Can someone explain where I went wrong?

The vector functions for true positive, false positive etc all make use of the "magic" numbers $0$ and $1$ used to represent Boolean values. They are convenience methods that you can use in a numerical library, but you do need to be aware of the fundamental Boolean nature of the data. The $0$ and $1$ values allow the maths for calculating TP et al, but are not fundamental to it, they are a representation.

Your derivations of gradients for the functions you give seem correct, barring the odd typo. However, the gradient doesn't really apply to the value of $\mathbf{y}$, because all components of $\mathbf{y}$ are either $0$ or $1$. The idea that you could increase $\mathbf{y}$ slightly where  $\mathbf{\hat{y}}$ is $1$ in order to increase the value of the TP metric slightly has no basis. Instead the only valid changes to make an improvement are to modify FN values from $0$ to $1$ exactly.

You could probably still use your derivations as a gradient for optimisation (it would not be the only time in machine learning that something does not quite apply theoretically but you could still use it in practice). However, you then immediately hit the problem of how the values of $\mathbf{y}$ have been discretised to  $0$ or $1$ - that function will not be differentiable, and it will prevent you back propagating your gradients to the neural network weights that you want to change. If you fix that follow-on problem using a smoother function (e.g. a sigmoid) then you are likely to end up with something close to either cross-entropy loss or the perceptron update step.

In other words, although what you have been told is an over-simplification, you will not find a way to improve the performance of your classifier by adding cost functions based directly on TP, FP etc. That is what binary cross-entropy loss is already doing. There are other, perhaps more fruitful avenues of investigation - hyperparameter searches, regularisation, ensembling, and if you have an unbalanced data set then consider weighting the true/false costs.


How to fill in missing transitions when sampling an MDP transition table? I have a simulator modelling a relatively complex scenario. I extract ~12 discrete features from the simulator state which forms the basis for my MDP state space.
Suppose I am estimating the transition table for an MDP by running a large number of simulations and extracting feature transitions as the state transitions.
While I can randomize the simulator starting conditions to increase the coverage of states, I cannot guarantee all states will be represented in the sample, i.e. states which are possible but rare.
Is there a rigorous approach to "filling in the gaps" of the transition table in this case?
For example:

For each state that was unrepresented in the sample, simply transition to all other states with equal probability, as a "neutral" way to fill in the gap?

As above, but transition only to represented states (with equal probability)?

Transition to the same state with probability 1.0?

Ignore unrepresented states during MDP solving entirely, and simply have a default action specified?

I assume you use the 12 discrete features as state variables, and, for each of these variables, you will have at least two values. So, the minimum number of states will be $2^{12} = 4096$, which gives $(2^{12})^{2} = 16777216$ possible transitions. In order to reach this, you will need a huge amount of simulations, also taking into account that this number is a minimum since you might have more values per state variable, and you probably have more than one action per state transition.
How to fill the gaps depends on your problem, in a problem where I did this, I filled the gaps using a uniformly random transition to its neighbor states. However, since I had to fill in such a a large amount, there was no significant difference between using this estimated transitions probabilities with a predefined transition table.
In your case, it might be better to use Q-Learning, which is a model-free method that does not require the transition probabilities and uses directly the rewards and states obtained.


Is it possible to use deeplearning with spark (with a distributed databases as HDFS or Cassandra)? If it is possible, will it be really useful or the model will end up converging very early(with a typical optimum learning rate) ? Any content on this topic will be helpful for me.

Yes, it is possible to use deep learning architecture with Apache Spark now. Databricks have Spark-deep-learning which is a pipeline based in python and uses tensorflow and keras.

https://github.com/databricks/spark-deep-learning

You can check this. There is also BigDL by Intel analytics.
https://github.com/intel-analytics/BigDL


Binary mode or Multi-label mode is correct when using binary crossentropy and sigmoid output function on multi-label classification I would like to ask a question about the relationship of accuracy with the loss function.
My experiment is a multiclass text classification problem, and I have built a Keras neural network to tackle it. My labels are something like
array([array([0, 0, 0, 0, 0, 1, 0, 1]), array([0, 1, 1, 0, 0, 0, 0, 1])])
For the final output layer I use the 'sigmoid' activation function and for loss the 'binary crossentropy', however, I am a bit confused about the metric. I am using the F1_score metric because Accuracy it's not a metric to count on when there are many more negative labels than positive labels. So, since the problem is multilabel classification shall I use the multi-label mode like tfa.metrics.F1_score(mode="micro"). Is that correct? or since I use binary_crossentropy and sigmoid activation function should I use the standard binary f1-score because every label-tag Is independent to the others and has a different Bernoulli distribution?
I would really like to get your input on this. My humble opinion is that I should you the binary standard mode of binary f1-score and not the multi-label micro approach even though my experiment is multi-label text classification.
My current approach (using micro F1-score since my y_train is Multi-label
model_for_pruning.compile(optimizer='adam',
                          loss='binary_crossentropy',
                          metrics=[tfa.metrics.F1Score(y_train[0].shape[-1], average="micro")])

My alternative approach (based on the binary_crossentropy and the sigmoid activation function, despite I have multi-label y_train)
model_for_pruning.compile(optimizer='adam',
                          loss='binary_crossentropy',
                          metrics=[tfa.metrics.F1Score(y_train[0].shape[-1], average=None)])

Τhe reason why I use sigmoid and not softmax as the output layer
relevant link
Why Sigmoid and not Softmax in the final dense layer?
In the final layer of the above architecture, sigmoid function as been used instead of softmax. The advantage of using sigmoid over Softmax lies in the fact that one synopsis may have many possible genres. Using the Softmax function would imply that the probability of occurrence of one genre depends on the occurrence of other genres. But for this application, we need a function that would give scores for the occurrence of genres, which would be independent of occurrences of any other movie genre.
relevant link 2
Binary cross-entropy rather than categorical cross-entropy.
This may seem counterintuitive for multi-label classification; however, the goal is to treat each output label as an independent Bernoulli distribution and we want to penalize each output node independently.
Please check my reasoning behind this and I would be happy if we can contradict this explanation. To better explain my experiment, I want to predict movie genres. So a movie can belong to 1 or more genres ['Action', 'Comedy', 'Children'], so when I use softmax the probability sum to 1, while when I use sigmoid its single probability of a class has a range between (0,1). Thus, if the predictions are correct the genres with the highest probabilities are those assigned to the movie. So imagine that my vector of prediction probabilities are something like [0.15, 0.12, 0.54, 0.78, 0.99] sum()> 1, and not something like [0.12, 0.43, 11, 0.32,  0.01, 0.01) sum() = 1.

Since you have a multiclass classification problem rather than a binary classification problem (i.e. a two-class problem), I recommend to adjust your architecture and use softmax instead of sigmoid as final activation function and categorcal_crossentropy instead of binary_crossentropy.
Softmax will ensure all your outputs are valid probabilities. This is crucial in calculating the loss of your model. Using sigmoid will only ensure this if you have binary classes. Further categorical crossentropy is used for multiclass problems, again this is to make sure you calculate the correct loss. This is just a general model characteristic:
binary -> sigmoid & binary_crossentropy
multi-class -> softmax & (sparse_)categorical_crossentropy
To answer you original question, I think you should use average = micro, macro or weighted when calculating the F1-score. This depends on what you like your model to put more emphasis on and how severe your class imbalances are.


How do I get started with multi-agent reinforcement learning? Is there any tutorial that walks through a multi-agent reinforcement learning implementation (in Python) using libraries such as OpenAI's Gym (for the environment), TF-agents, and stable-baselines-3?
I searched a lot, but I was not able to find any tutorial, mostly because Gym environments and most RL libraries are not for multi-agent RL.

After checking the Internet, you will probably find several resources such as

https://github.com/mohammadasghari/dqn-multi-agent-rl
https://rlss.inria.fr/files/2019/07/RLSS_Multiagent.pdf
https://arxiv.org/abs/2011.00583
and others.

Try to understand the principles first (see above). After some reasonable amount of coding you can adapt OpenAI gym.
Good luck!


How does the crossover operator work when my output contains only 2 states? I'm currently working on a project where I am using a basic cellular automata and a genetic algorithm to create dungeon-like maps. Currently, I'm having an incredibly hard time understanding how exactly crossover works when my output can only be two states: DEAD or ALIVE (1 or 0).

I understand crossover conceptually - you find two fit members of the population and they exchange genetic material, hopefully producing a fitter offspring. I also understand this is usually done by performing k-point crossover on bit strings (but can also be done with real numbers).

However, even if I encode my DEAD/ALIVE cells into bits and cross them over, what do I end up with? The cell can only be DEAD or ALIVE. Will the crossover give me some random value that is outside this range?

And even if I were to work on floating-point numbers, wouldn't I just end up with a 1 or 0 anyway? In that case, it seems like it would be better to just randomly mutate DEAD cells into ALIVE cells, or vice versa.

I've read several papers on the topic, but none seem to explain this particular issue (in a language I can understand, anyway). Intuitively, I thought maybe I can perform crossover on a neighbourhood of cells - so I find 2 fit neighbourhoods, and then they exchange members (for example, neighbourhood A gives 4 of its neighbours to neighbourhood B). However, I have not seen this idea anywhere, which leads me to believe it must be fundamentally wrong.

Any help would be greatly appreciated, I'm really stuck on this one.

How do you calculate fitness for your organism if there are only two states? I would think it would be a solution to work on combinations of cells:

If you have blocks of, say, 4x4 cells, then each organism would be encoded using 16 bits. A crossover then is like overlaying two 4x4 blocks, picking the new block from either the first or the second parent. Effectively you are doing this:

0011                   1111                 xx11
0001  combined with    0000  would become   000x
1001                   0000                 x00x
1111                   0000                 xxxx

(this would be a bottom-right corner combined with a top wall segment)

or, as a single bit-string:

0011000110011111 +
1111000000000000 =
xx11000xx00xxxxx

where 'x' can be either 0 or 1, as this is where the two organisms are different.


How is the transformers' output matrix size arrived at? In this tensorflow article, the comments in the code say that MHA should output with one of the dimensions being the sequence length of the query/key. However, that means that the second MHA in the decoder layer should output something with one of the dimensions being the input sequence length, but clearly it should actually be the output sequence length! From all that I have read on transformers, it seems that the output of the left side of the SHA should be a matrix with dimensions q_seq_length x q_seq_length, and the output of the right side of the SHA should be v_seq_length x d_model. These matrices can't even multiply when using the second MHA in the decoder to incorporate the encoder output! Please help. I would appreciate a clear-cut explanation. Thanks

Aha, I understand now! In the paper, the diagram for SHA has its inputs in the order Q, K, V, while the diagram for MHS has its inputs in the order V, K, Q! In the grand diagram for the entire net, I thought the arrows for the inputs were in the order for SHA, but they are actually in the order for MHA. It is a bit confusing that when changing to the MHA diagram, they decided to swap the order of inputs in the paper, but it all makes sense now. output_seq x d_q multiplies d_q x input_seq to get output_seq x input_seq, which finally multiplies with input_seq x d_v to get output_seq x d_v, which then concatenates to output_seq x hd_v, so after the final linear layer it becomes output_seq x d_model, the standard in the decoder layer. Hopefully, anyone else who mixed up orders from the diagram will understand now.


Does AlphaZero use Q-Learning? I was reading the AlphaZero paper Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithm, and it seems they don't mention Q-Learning anywhere.

So does AZ use Q-Learning on the results of self-play or just a Supervised Learning?

If it's a Supervised Learning, then why is it said that AZ uses Reinforcement Learning? Is "reinforcement" part primarily a result of using Monte-Carlo Tree Search?

Note: you mentioned in the comments that you are reading the old, pre-print version of the paper describing AlphaZero on arXiv. My answer will be for the "official", peer-reviewed, more recent publication in Science (which nbro linked to in his comment). I'm not only focusing on the official version of the paper just because it is official, but also because I found it to be much better / more complete, and I would recommend reading it instead of the preprint if you are able to (I understand it's behind a paywall so it may be difficult for some to get access). The answer to this specific question would probably be identical for the arXiv version anyway.

No, AlphaZero does not use $Q$-learning.

The neural network in AlphaZero is trained to minimise the following loss function:

$$(z - \nu)^2 - \pi^{\top} \log \mathbf{p} + c \| \theta \|^2,$$

where:

$z \in \{-1, 0, +1\}$ is the real outcome observed in a game of self-play.
$\nu$ is a predicted outcome / value.
$\pi$ is a distribution over actions derived from the visit counts of MCTS.
$\mathbf{p}$ is the distribution over actions output by the network / the policy.
$c \| \theta \|^2$ is a regularisation term, not really interesting for this question/answer.

In this loss function, the term $(z - \nu)^2$ is exactly what we would have if we were performing plain Monte-Carlo updates, based on Monte-Carlo returns, in a traditional Reinforcemet Learning setting (with function approximation). Note that there is no "bootstrapping", no combination of a single-step observed reward plus predicted future rewards, as we would have in $Q$-learning. We play out all the way until the end of a game (an "episode"), and use the final outcome $z$ for our updates to our value function. That's what makes this Monte-Carlo learning, rather than $Q$-learning.

As for the $- \pi^{\top} \log \mathbf{p}$ term, this is definitely not $Q$-learning (we're directly learning a policy in this part, not a value function).

  If it's a Supervised Learning, then why is it said that AZ uses Reinforcement Learning? Is "reinforcement" part primarily a result of using Monte-Carlo Tree Search?

The policy learning looks very similar to Supervised Learning; indeed it's the cross-entropy loss which is frequently used in Supervised Learning (classification) settings. I'd still argue it is Reinforcement Learning, because the update targets are completely determined by self-play, by the agent's own experience. There are no training targets / labels that are provided externally (i.e. no learning from a database of human expert games, as was done in 2016 in AlphaGo).


Could a neural network be capable to diferentiate between two boards of a game? Let's propose, that I can define the state of a board in a board game, with 234 neurons. In theory, could I be able to train a neural network, with 468 inputs (two game boards), and 1 output, to tell me which board state is 'better'? The output should give me ~-1 if the second board is better than the first, ~0 if they are equal, and ~1 if the first board is better than the second.

If yes, what could be the number of ideal neurons on the hidden layers? What could be the ideal number of hidden layers?

For optimal performance, the network complexity should fit the complexity of the game. Since we do not know the latter, your question is not answerable.


How would an AI learn language? I was think about AIs and how they would work, when I realised that I couldn't think of a way that an AI could be taught language. A child tends to learn language through associations of language and pictures to an object (e.g., people saying the word "dog" while around a dog, and later realising that  people say "a dog" and "a car" and learn what "a" means, etc.). However, a text based AI couldn't use this method to learn, as they wouldn't have access to any sort of input device.

The only way I could come up with is programming in every word, and rule, in the English language (or whatever language it is meant to 'speak' in), however that would, potentially, take years to do.

Does anyone have any ideas on how this could be done? Or if it has been done already, if so how?

By the way, in this context, I am using AI to mean an Artificial Intelligence system with near-human intelligence, and no prior knowledge of language.

The general research area is known as grammar induction.

It is generally framed as a supervised learning problem, with the input presented as raw text, and the desired output the corresponding parse tree.
The training set often consists of both positive and negative examples.

There is no single best method for achieving this, but some of the techniques that have been used to date include:

Bayesian approaches
Genetic Algorithms
Genetic Programming
Blackboard Architectures
The UpWrite Predictor


What ML algorithm should I use that suits this data? What if I have some data, let's say I'm trying to answer if education level and IQ affect earnings, and I want to analyze this data and put in a regression model to predict earnings based on the IQ and education level. My confusion is, what if the data is not linear or polynomial? What if it's a mess but there are still patterns that the linear plane algorithm can't capture? How do I figure out if plotting all of the independent variables will form a line or a polynomial curve like here?

I mean, with one dependent and one independent variable it's easy because you can plot it and see, but in a situation with multiple independent variables... how do I figure out if the relationship is linear or something like this? How do I figure out if I should use a regression model?
Let's say I want to predict a store's daily revenue based on the day of the week, weather and the number of people arrived in the city. My data would look something like this:
+-----------+---------+----------------+---------+
| DAY       | WEATHER | PEOPLE ARRIVED | REVENUE |
+-----------+---------+----------------+---------+
| Monday    | Sunny   | 1115           | $500    |
+-----------+---------+----------------+---------+
| Tuesday   | Cloudy  | 808            | $250    |
+-----------+---------+----------------+---------+
| Wednesday | Sunny   | 450            | $300    |
+-----------+---------+----------------+---------+

I'm a bit confused about what ML algorithm I should use in such a scenario. I can represent the days of the week as (Monday - 1, Tuesday - 2, Wednesday - 3, etc.) and the weather as (Sunny - 1, Cloudy - 2, Normal - 3, etc.) but would a regression model work? I'm skeptical because I'm not sure if there's a linear relationship between the variables and I'm not sure if a hyperplane can create accurate representation of what's going on.

Regression Model will definitely work on That problem.
You only need to change shape of predicting variables like(Day, Weather, people arrived) into 1D array if you got error..
Otherwise you can simply apply Linear Regression, SVM etc to get your output with good accuracy.


Why do we need convolutional neural networks instead of feed-forward neural networks? Why do we need convolutional neural networks instead of feed-forward neural networks?

What is the significance of a CNN? Even a feed-forward neural network will able to solve the image classification problem, then why is the CNN needed?

Why are CNNs useful?
The main property of CNNs that make them more suitable than FFNNs to solve tasks where the inputs are images is that they perform convolutions (or cross-correlations).
Convolution
The convolution is an operation (more precisely, a linear operator) that takes two functions $f$ and $h$ and produces another function $g$. It's often denoted as $f \circledast h = g$, where $\circledast$ represents the convolution operation and $g$ is the function that results from the convolution of the functions $f$ and $h$.
In the case of CNNs,

$f$ is a multi-dimensional array (aka tensor) and it represents an image (or a processed version of an image, i.e. a feature map)
$h$ is a multi-dimensional array and it is called kernel (aka filter), which represents the learnable parameters of the CNN, and
$g$ is a processed version (with $h$) of $f$ and it is often called the feature map, so it's also a multi-dimensional array

Images as functions
To be consistent with the initial definition of the convolution, $f, h$, and $g$ can indeed be represented as functions.
Suppose that the input image is a greyscale (so it is initially represented as a matrix), then we can represent it as a function as follows
$$f: [a, b] \times [c, d] \rightarrow [0, 1],$$
i.e. given two numbers $x \in [a, b]$ and $y \in [c, d]$, $f$ outputs a number in the range $[0, 1]$, i.e. $f(x, y) = z$, where $z$ is the grayscale intensity of the pixel at coordinates $x$ and $y$. Similarly, the kernel $h$ and $g$ can also be defined as a function $h: [a, b] \times [c, d] \rightarrow [0, 1]$ and $g: [a, b] \times [c, d] \rightarrow [0, 1]$, respectively.
To be more concrete, if the shape of the image $f$ is $28 \times 28$, then it is represented as the function $f: [0, 28] \times [0, 28] \rightarrow [0, 1]$.
Note that the domain of the images doesn't have to range from $0$ to $28$ and the codomain doesn't have to range from $[0, 1]$. For example, in the case of RGB images, the codomain can also equivalently range from $0$ to $255$.
RGB images can also be represented as functions, more precisely, vector-valued functions, i.e.
$$
f(x, y) =
\begin{bmatrix}
r(x, y) \\
g(x, y) \\
b(x, y)
\end{bmatrix}
$$
where

$r: [a, b] \times [c, d] \rightarrow [0, 1]$ represents the red channel,
$g: [a, b] \times [c, d] \rightarrow [0, 1]$ represents the green channel, and
$b: [a, b] \times [c, d] \rightarrow [0, 1]$ represents the blue channel

Or, equivalently, $f: [a, b] \times [c, d] \times [0, 1]^3$.
Why is the convolution useful?
The convolution of an image with kernels (e.g. the median kernel) can be used to perform many operations.
For example, the convolution of a noisy image with the median filter can be used to remove noise from that image.

This is a screenshot of an image from this article, which you should read if you want to understand more about noise removal. So, on the left, there's the noisy image, and, on the right, there's the convolution of the median filter with the noisy image, which removes (at least, partially) the initial noise (i.e. those dots, which are due to the so-called "pepper and salt" noise).
The convolution of any image with the Sobel filter can be used to compute the derivatives of that image (both in the $x$ and $y$ directions, from which you can compute the magnitude and orientation of the gradient at each pixel of the image). See this article for more info.
So, in general, the convolution of an image with a kernel processes the image and the results (i.e. another image, which, in the case of CNNs, is called a feature map) can be different depending on the kernel.
This is the same thing as in CNNs. The only difference is that, in CNNs, the kernels are the learnable (or trainable) parameters, i.e. they change during training so that the overall loss (that the CNN is making) reduces (in the case CNNs are trained with gradient descent and back-propagation). For this reason, people like to say that CNNs are feature extractors or are performing feature extraction (aka feature learning or representation learning).
(Moreover, note that the convolution and cross-correlation are the same operations when the kernels are symmetric (e.g. in the case of a Gaussian kernel). In the case of CNNs, the distinction between convolution and cross-correlation doesn't make much sense because the kernels are learnable. You can ignore this if you are a beginner, but you can find more details here.)
Other useful properties
There are other useful properties of CNNs, most of them are just a consequence of the use of the convolution

Translation invariance (or equivariance), i.e. they can potentially find the same features (if you think of them as feature extractors) in multiple places of the image independently of their position, orientation, etc. See this answer for more details.

The equivalent FFNN has a lot more parameters (so CNNs may be less prone to overfitting)

They often use a sub-sampling operation  (known as pooling) to further reduce the number of parameters (which can possibly help to avoid overfitting) and introduce non-linearity.

Notes
Note that the FFNN can also be used to process images. It's just that the CNN is more suited to deal with images for the reasons described above.


Training an AI to play Starcraft 2 with superhuman level of performance? I'm interested in working on challenging AI problems, and after reading this article (https://deepmind.com/blog/deepmind-and-blizzard-open-starcraft-ii-ai-research-environment/) by DeepMind and Blizzard, I think that developing a robust AI capable of learning to play Starcraft 2 with superhuman level of performance (without prior knowledge or human hard-coded heuristics) would imply a huge breakthrough in AI research.

Sure I know this is an extremely challenging problem, and by no means I pretend to be the one solving it, but I think it's a challenge worth taking on nonetheless because the complexity of the decision making required is much closer to the real world and so this forces you to come up with much more robust, generalizable AI algorithms that could potentially be applied to other domains.

For instance, an AI that plays Starcraft 2 would have to be able to watch the screen, identify objects, positions, identify units moving and their trajectories, update its current knowledge of the world, make predictions, make decisions, have short term and long term goals, listen to sounds (because the game includes sounds), understand natural language (to read and understand text descriptions appearing in the screen as well), it should probably be endowed also with some sort of attention mechanism to be able to pay attention to certain regions of interest of the screen, etc. So it becomes obvious that at least one would need to know about Computer Vision, Object Recognition, Knowledge Bases, Short Term / Long Term Planning, Audio Recognition, Natural Language Processing, Visual Attention Models, etc. And obviously it would not be enough to just study each area independently, it would also be necessary to come up with ways to integrate everything into a single system.

So, does anybody know good resources with content relevant to this problem? I would appreciate any suggestions of papers, books, blogs, whatever useful resource out there (ideally state-of-the-art) which would be helpful for somebody interested in this problem.

Thanks in advance.

StarCraft II is a real time strategy game that combines fast paced micro actions with the need for high level planning and execution. StarCraft II being a popular game with millions of users it proceeds that defeating top players becomes a meaningful and measurable long term objective in AI research.

Computer games provide a compelling solution to the issue of evaluating and comparing different learning and planning approaches on standardized tasks. They are an important source of challenges for research in AI.

Game playing AI agents i.e. deepmind's Atarinet and DQN alongside Open AI's Dota 2 bot represent the first demonstration of a General Purpose Agent that is able to continually adapt behavior without any human intervention, a major technical step forward in the quest for general AI (Source deepmind blog).

Computer games offer numerous advantages in AI research i.e:

They have clear objective measures of success.
Computer games typically output rich streams of observational data, which are ideal inputs for deep networks.
They are externally defined to be difficult and interesting for a human to play. Therefore they provide an excellent test for intelligence.
Games are designed to be able to run anywhere with the same interface and game dynamics. This enables running many simulations in parallel. Sharing and updating the same table throughout training.
In some cases pools of superb human players exist, making it possible to benchmark against highly skilled humans.

The Starcraft challenge for reinforcement learning, introduces a taxing set of problems because it is a multi-agent problem with multiple players interacting. There is imperfect information due to a partially observed map, it has a large state space, it has delayed credit assignment requiring long term strategies.

Tools

The SC2LE Environment

DeepMind and Blizard games have collaborated to release the SC2LE, which exposes StarCraft II as a research environment.

The SC2LE consists of three sub-components.

A Linux Starcraft II binary.
StarCraft II API which allows programmatic control of StarCraft II. The API can be used to start the game, get observations, take actions and review replays.
PySC25 which is an open source environment written in Python. It includes some mini-games and visualization tools

Open source Open AI RL environments

Universe - Universe is a software platform by Open AI for measuring and training an AI's general intelligence across games, websites and other applications.

Gym - Open AI Gym is a toolkit for developing and comparing reinforcement learning algorithms. It makes no assumptions about the structure of your agent and is compatible with any numerical computation library such as Tensorflow or Theano.

Supervised Classification Approach

Consider this, we could decide to screen capture game sessions from expert players and use it as input to a model. The output could be the direction in which the AI agent could move. This would be a supervised classification approach.

However, this is not an elegant solution because we are training a model not on a static dataset but a dynamic one (game environment). The training data from a game environment is stochastic/continuous meaning any number of events can occur.
Furthermore, humans learn most effectively by interacting with the environment. Not by watching others interact with the environment.

Markov Decision Process

Markov Decision Processes (MDPs) provide a mathematical framework for modeling decision making in situations where outcomes are partly random and partly under the control of a decision maker i.e. game environments.

Reinforcement Learning with Deep Q-Learning

Q-Learning is a strategy that has been proven to find an optimal action selection policy for any Markov Decision Process (MDP). In Q-Learning we choose an action that maximizes future reward. The further in the future we go, the further the rewards can diverge, we resolve this by adding a discount in future rewards.

  Unlike policy gradient methods, which attempt to learn functions which directly map an observation to an action, Q-Learning attempts to learn the value of being in a given state, and taking a specific action there. (Arthur J 2016)

The formula for Q-Learning is:

  Where:

  R = Reward

  s = State

  a = Action

  Experience during learning is based on (s, a) pairs

  One has an array Q and uses experience to update it directly

  (Source wikipedia https://en.wikipedia.org/wiki/Markov_decision_process)

One of the strengths of Q-learning is that it is able to compare the expected utility of the available actions without requiring a model of the environment.

For further reference, I recommend you look at Siraj Ravals tutorial on Deep Q-Learning

https://www.youtube.com/watch?v=79pmNdyxEGo and source code for the same available here https://github.com/llSourcell/deep_q_learning

Additionally I recommend the following references for more information on computer game playing AI agents.

StarCraft II: A New Challenge for Reinforcement Learning https://arxiv.org/abs/1708.04782

Playing Atari with Deep Reinforcement Learning https://www.cs.toronto.edu/~vmnih/docs/dqn.pdf

Human-level control through deep reinforcement learning  https://web.stanford.edu/class/psych209/Readings/MnihEtAlHassibis15NatureControlDeepRL.pdf


Minimum video quality for facial recognition I am currently working on a small project where I am trying to automate some stuff at home. I am building a model capable of identifying my face with OpenCV. This will be a live feed.
I am making the project's estimations and have a really low budget. Therefore I am trying to identify what could be the minimum quality video feed I can pass to my algorithm to identify any face. For now I am just trying to identify mine.
I understand facial recognition works primarily on the unique pattern that could be found in the face. What is the minimum video resolution I need to identify anyone with facial recognition?

On page 2 of Axis' web page Identification and Recognition there is an estimate of the minimum number of pixels needed for identification, recognition and detection.


Will a .h5 file trained with Xception model work with Resnet50? I have been running my 2013 server box since 2 weeks ago for training an AI model.
I set up 30 epochs to run but since than it only ran 1 epoch as my PC config is super slow. But it generates 1 .h5 file.

My question is will this .h5 file that I trained with Xception model work for Resnet 50?

If I understand your question correctly, you are asking whether you could load the saved weights of a trained model with the Xception architecture on a a Resnet50 architecture.

Short answer: No

Long answer: Xception and Resnet50 have very different architectures.

Here is a paper comparing multilple CNNs including Xception and Resnet50:
https://www.researchgate.net/publication/330478807_Deep_Feature-Based_Classifiers_for_Fruit_Fly_Identification_Diptera_Tephritidae/figures?lo=1

As you can see, the architectures are quiet different between Xception and ResNet50. Thus when you train a model, you are changing the weights and bias of the different layers of the model. If you switch models, and they are similar like let's say VGG16 and VGG19, you could import part of the weights for the layers which are similar between the two models.
As far as I know, I don't know of a function which can do this operation in tensorflow or keras.
But in your case, the architectures are extremely different between Xception and ResNet and there seem not to have any layers in common.

In general, one train a model, save the weights of the training, import back the weights, and use the same model for the next round of training/testing/predictions.

Hope that helps!


How do we find the length (depth) of the game tic-tac-toe in adversarial search? When we perform the tic-tac-toe game using adversarial search, I know how to make a tree. Is there a way to find the depth of the tree, and which level is the last level?

If you mean minimax search as adversarial, you can take a number of free spaces on the board as an upper bound of the tree depth.


How do we know the classification boundaries of the data? Consider an image classification problem. Conceptually, we then have some high dimensional space where all the images can be represented as points, and having large enough labeled data set we can build a classifier. But how do we know that our data in this space has some structure? Like this one in two-dimensional case:

If we have a data set with images of, say, cats and dogs, why these two classes are not just uniformly mixed with each other but have some distribution or shape in appropriate space? Why it cannot be like this:

This is the classic question of what structure is or can be. It relates directly to the concepts of generalization, pattern recognition, over-fitting in surface fitting strategies, and learning tabula rasa, Latin for blank slate. The underlying questions are these:

How can it be determined whether the organization of data in a set, which appears to correlate well with model $\mathbb{A}$, isn't just a random data set that merely appears to have $\mathbb{A}$ organization.

How can it be determined whether the organization of data in a set, which appears to correlate well with model $\mathbb{A}$, isn't generated by a phenomenon that generally exhibits organization that correlates well with $\mathbb{B}$?

How can we determine the interrelationship between $\mathbb{A}$ and $\mathbb{B}$, especially since inclusion and overlap relates to the above question?

In the second image in the question, if $\mathbb{A}$ is defined as clustering of a type the smallest boundaries of which are not breached by instances of another type with at least five members per cluster, then several false positives would occur if the locations of the points were generated with quantum or thermal noise and the model was intended to illuminate clustering. In such a case there may be the appearance of clusters, but proximity would not be indicative of organization.
In the first image in the question, if $\mathbb{A}$ is the compound angular projection of the orbital paths of the planets in this solar system, then the image would appear as noise.
If a third image existed of the moon's sky, the same model would likely produce a false negative if the tolerance of inaccuracy in input was high.
It is difficult to reliability detect organization in the measured properties of a complex system. The goal of feature extraction, using RBMs or some other approach is to detect patterns. Whether those patterns are organized or merely apparent is a hypothesis that may be strengthened or weakened as the amount of data analyzed increases.
If an AI system is trained to detect rats but knows nothing of cats and dogs, one cannot assume it will differentiate cats from dogs on the basis of the closer relationship between rodent and cat features than between rodent and dog features unless the AI system was meta-trained or programmed statically with a specific conception of what a feature is.


Would a neuromorphic SNN of the same complexity as the human nervous system be 'smarter'? If the nervous system is wired up such that there are no well defined layers, how does this compare to a neatly stacked artificial net? If between my sensory and motor side I had a neatly designed SNN with well defined layers, how would I see the world?
I get that there are some evolutionary advantages to a system where information can sometimes take a shortcut from sensory cell to motor cell (reflex action) bypassing brain processing but for arguments sake let's talk only about intelligence.

The human nervous system is an extremely dynamic entity, having been formed by the processes of embryogenesis, mediated by various markers guiding neurons to grow to specific areas, and all of it laid down by millions of years of evolution. There are many different varieties of neurons in the human brain, indeed more so than pretty much any other animal on this planet.

It is hard to see that a spiking neural network (SNN) would be able to get all these details correct when we don't yet fully understand the biological analog.

I think for this to be successful, we need to understand much more about not only the variety of neurons in the brain, but details of the the embryogenic dynamics as well. And not just the neurons, but the glial cells as well.

Having said that, there is something to be said for using evolutionary approaches to resolve "a solution" that may work anyway. Taking this approach will require many more resources, but is perhaps doable.

All in all, I would not expect a naive attempt of the neuromorphic SNN to succeed. There is a lot of complexity involved in what the brain does, and it involves the glial cells to a large degree. Do we understand enough about the role of glial cells in the brain? We cannot ignore them. Are they only performing "housekeeping" operations, like, for example, the uptake of "spent" neurotransmitters? Or are they doing more, taking part in the computational and / or memory aspects of the brain?

There is much research in this and other pertinent areas that one should look into. And expect a lot of surprises.


Is the following statement about neural networks overclaimed? Is the following statement about neural networks overclaimed?

  Neural networks are iterative methods that minimize a loss function defined on the output layer of neurons.

I wrote this statement in the Introduction section of a conference paper. Reviewer got back to me saying that, "this statement is over claimed without proper citations". How is this statement over claimed? Is it not obvious that neural networks are iterative methods that try to minimize a cost function?

I think I would never say that neural networks are iterative methods. I would say that iterative methods (e.g. gradient descent) are used to train neural networks (which can be thought of as linear and non-linear models, but mainly non-linear), which is quite different. Maybe you should or wanted to say that deep learning is an area of study where iterative methods are used to train neural networks.

It is possible the reviewer is just telling you that this or similar statements have been used a lot and don't really provide any insight, they are misleading, or simply useless or trivial in the context of the paper or conference.

Without further clarifications by the reviewer and about your paper and conference, it is difficult to provide more explanations.


How is the actor-critic algorithm guaranteed to converge? From my understanding, the critic evaluates the policy (actor) following dynamic programming (DP) or approximate dynamic programming (ADP) scheme, which should converge to the optimal value function after sufficient iterations. The policy (actor) then updates its parameter w.r.t the optimal value function using gradient methods. This policy evaluation and improvement circle are repeated until neither the critic nor the actor changes anymore.

How's guaranteed to converge as a whole? Is there any mathematical proof? Is it possible that it may converge to a local optimal point instead of a global one?

There are different actor-critic (AC) algorithms with different convergence guarantees. For example, AC algorithms where the critic is tabular have different convergence guarantees than AC algorithms where the critic is a neural network (function approximation). Most convergence proofs assume that the actor and the critic operate at different time scales, but, for example, in the paper A Convergent Online Single Time Scale Actor-Critic Algorithm (2010) this assumption is not made.

In the paper Incremental Natural Actor-Critic Algorithms (2007), the authors propose four different AC algorithms that use function approximation (neural networks) to represent the critic. Three of these proposed AC algorithms are based on natural policy gradients. In section 6 of the extended and more technical version of the mentioned paper, Natural Actor-Critic Algorithm (2007), the authors prove the convergence of the parameters of the policy and value function to a local maximum of an objective (or performance) function, which corresponds to the average reward (equation 2) plus a measure of the temporal-difference (TD) error inherent in the function approximation.


How can I change observation states' values in OpenAI gym's cartpole environment? I am learning with the OpenAI gym's cart pole environment.
I want to make the observation states discrete (with small stepsize) and for that purpose, I need to change two of the observations from [$
-\infty, \infty$] to some finite upper and lower limits. (By the way, these states are velocity and pole velocity at the tip).
How can I change these limits in the actual gym's environment?
Any other suggestions are also welcome.

I don't recommend changing the rules of the environment.
What you could do:
Perform a method called bucketing i.e. take a value from a continuous state space see which discrete bucket it should go into and then let your agent use the bucket number as the observation.
e.g. Say I do have a continuous state space with one variable in range $[-\infty,\infty]$
The buckets can be as follows:
0). x < -1000
1). -1000 $\le$ x $<$ -500
2). -500 $\le$ x $<$ -100
3). -100 $\le$ x $<$ -50
4). -50 $\le$ x $<$ 0
5). 0 $\le$ x $<$ 50
6). 50 $\le$ x $<$ 100
7). 100 $\le$ x $<$ 500
8). 500 $\le$ x $<$ 1000
9). x > 1000
Therefore in this example scenario there are 9 buckets. Hence, the observations can be in range [0, 9] discretely.


Can a basic CNN (Conv2D, MaxPooling2D, UpSampling2D) find a good approximation of a product of its input channels? Let's assume I want to teach a CNN some physics. Starting with a U-Net, I input images A and B as separate channels. I know that my target (produced by a very slow Monte-Carlo code) represents a signal such as f(g(A) * h(B)), where f, g and h are fairly "convolutional" operations -- meaning, involving mostly blurring and rescaling operations.

I feel safe to state that this problem would not be too difficult for the case of f(g(A) + h(B)) -- but what about f(g(A) * h(B))? Can I expect a basic CNN such as the U-Net to be able to represent the * (multiplication) operation?

Or should I expect to be forced to include a Multiply layer in my network, somewhere where I expect that the part before can learn the g and h parts, and the part after can learn the f part?

I think U-Net is already a quite complex network that probably should (as of my experience) be able to approximate this multiplication. However this would still be an approximation that might not be accurate and maybe only solves a function that looks like a multiplication for the range of the shown input samples defined by your dataset (therefore potentially overfitting on your training dataset).

So in general if you know that your target function does have this multiplication then you should definetly excplicitly enforce it in your network. If you know so much about the wanted function it's always better to build a good fitting architecture than using a general neural network architecture. That will ease the optimization and should generalize much better.

However, it's hard or impossible to tell you for sure how much depth or complexity you need to solve this task within your wanted accuracy. Eventually you should just try it out.


How important will statistical learning be to a conscious AI? Deep learning is based on getting a large number of samples and essentially making statistical deductions and outputting probabilities.

On the other hand we have formal programming languages like PROLOG which don't involve probability.

Is there any essential reason why an AI could be called conscious without being able to learn in a statistical manner? i.e. by only being able to make logical deduction alone. (It could start with a vast number of innate abilities).

Or is probability and statistical inference a vital part of being conscious?

Computational Learning Theory gives us an interesting framework to understand what statistical learning is doing.

The gist of it is, we can model the process of statistical learning as one of formal deduction. The learning itself does not require a random element.

This shouldn't be too surprising. Consider a classic decision tree learner like C4.5 or ID3: the algorithm works through the data deterministically, and no random decisions are made. When asked to make a prediction, the learned model returns the frequency of each possible label within the most similar subpopulation from its training data, with similarity defined according to the algorithm's rules for partitioning data.

There's no reason you can't write a decision tree learner (or even a deep learning algorithm) in Prolog, it just might not be very efficient, or very practical.


Why do my rewards fall using tabular Q-learning as I perform more episodes? Using the tutorial from: SentDex - Python Programming I added Q Learning to my script that was previously just picking random actions. His script uses the MountainCar Environment so I had to amend it to the CartPole env I am using. Initially, the rewards seem sporadic but, after a while, they just drop off and oscillate between 0-10. Does anyone know why this is?

Learning_rate = 0.1
Discount_rate = 0.95
episodes = 200

# Exploration settings
epsilon = 1  # not a constant, qoing to be decayed
START_EPSILON_DECAYING = 1
END_EPSILON_DECAYING = episodes//2
epsilon_decay_value = epsilon/(END_EPSILON_DECAYING - START_EPSILON_DECAYING)

env = gym.make("CartPole-v0") #Create the environment. The name of the environments can be found @ https://gym.openai.com/envs/#classic_control
#Each environment has a number of possible actions. In this case there are two discrete actions, left or right

#Each environment has some integer characteristics of the state.
#In this case we have 4:

#env = gym.wrappers.Monitor(env, './', force=True)

DISCRETE_OS_SIZE = [20, 20, 20, 20]

discrete_os_win_size = (env.observation_space.high - env.observation_space.low)/ DISCRETE_OS_SIZE

def get_discrete_state(state):
    discrete_state = (state - env.observation_space.low)/discrete_os_win_size
    return tuple(discrete_state.astype(np.int))

q_table = np.random.uniform(low = -2, high = 0, size = (20, 20, 20, 20, env.action_space.n))

plt.figure() #Instantiate the plotting environment
rewards_list = [] #Create an empty list to add the rewards to which we will then plot
for i in range(episodes):
    discrete_state = get_discrete_state(env.reset())
    done = False
    rewards = 0
    frames = []

    while not done:
        #frames.append(env.render(mode = "rgb_array"))

        if np.random.random() > epsilon:
            # Get action from Q table
            action = np.argmax(q_table[discrete_state])

        else:
            # Get random action
            action = np.random.randint(0, env.action_space.n)

        new_state, reward, done, info = env.step(action)

        new_discrete_state = get_discrete_state(new_state)

        # If simulation did not end yet after last step - update Q table
        if not done:

            # Maximum possible Q value in next step (for new state)
            max_future_q = np.max(q_table[new_discrete_state])

            # Current Q value (for current state and performed action)
            current_q = q_table[discrete_state, action]

            # And here's our equation for a new Q value for current state and action
            new_q = (1 - Learning_rate) * current_q + Learning_rate * (reward + Discount_rate * max_future_q)

            # Update Q table with new Q value
            q_table[discrete_state, action] = new_q

        else:
            q_table[discrete_state + (action,)] = 0

        discrete_state = new_discrete_state

        rewards += reward
        rewards_list.append(rewards)
    #print("Episode:", i, "Rewards:", rewards)
    #print("Observations:", obs)

    # Decaying is being done every episode if episode number is within decaying range
    if END_EPSILON_DECAYING >= i >= START_EPSILON_DECAYING:
        epsilon -= epsilon_decay_value

plt.plot(rewards_list)
plt.show()
env.close()

It becomes even more pronounced when I increase episodes to 20,000 so I don't think it's related to not giving the model enough training time.

If I set START_EPSILON_DECAYING to say 200 then it only drops to < 10 rewards after episode 200 which made me think it was the epsilon that was causing the problem. However, if I remove the epsilon/exploratory then the rewards at every episode are worse as it gets stuck in picking the argmax value for each state.

The problem here is likely related to the state approximations you are using.

Unfortunately, OpenAI's gym does not always give reasonable bounds when using env.observation_space, and that seems to be the case for CartPole:

>>> env = gym.make('CartPole-v0')
>>> env.observation_space.high
array([4.8000002e+00, 3.4028235e+38, 4.1887903e-01, 3.4028235e+38],
      dtype=float32)
>>> env.observation_space.low
array([-4.8000002e+00, -3.4028235e+38, -4.1887903e-01, -3.4028235e+38],
      dtype=float32)

Processing this, similarly to your code:

>>> discrete_os_win_size = (env.observation_space.high - env.observation_space.low)/ DISCRETE_OS_SIZE
__main__:1: RuntimeWarning: overflow encountered in subtract
>>> discrete_os_win_size
array([0.48000002,        inf, 0.0418879 ,        inf])

>>> discrete_state = (state - env.observation_space.low)/discrete_os_win_size
>>> discrete_state
array([11.27318768,  0.        , 19.50682776,  0.        ])

That means that all the velocities will get squashed down to $0$ in your approximation. Your agent cannot tell the difference between a nearly static balancing position (generally the goal) and transitioning through it really fast - it will think that both are equally good. It is also not able to tell difference between moving towards balance point, or moving away from it.

I suggest you check for what reasonable bounds are on the space (a quick look suggests +/- 2.0 might be a reasonable starting point) and use that instead.

The approximation approach of discrete grid is also very crude, although it does allow you do use tabular approaches. If you want to stick with a linear system (and avoid trying neural networks and DQN) then the next step up would be some form of tile coding, which uses multiple offset grids to obtain smoother interpolation between states.


Understanding the n-step off-policy SARSA update In Sutton & Barto's book (2nd ed) page 149, there is the equation 7.11

I am having a hard time understanding this equation.

I would have thought that we should be moving $Q$ towards $G$, where $G$ would be corrected by importance sampling, but only $G$, not $G-Q$, therefore I would have thought that the correct equation would be of the form

$Q \leftarrow Q + \alpha (\rho G - Q)$

and not

$Q \leftarrow Q + \alpha \rho (G - Q)$

I don't get why the entire update is weighted by $\rho$ and not only the sampled return $G$.

Multiplying the entire update by $\rho$ has the desirable property that experience affects $Q$ less when the behavior policy is unrelated to the target policy. In the extreme, if the trajectory taken has zero probability under the target policy, then $Q$ isn't updated at all, which is good. Alternatively, if only $G$ is scaled by $\rho$, taking zero probability trajectories would artificially drive $Q$ to zero.


Use mobile device camera for moving pattern recognition For instance, one task would be to detect if with an Android phone in hand, I'm panning the camera toward a circle shape in a 2D space.

What is the best technology set and embedded application approach can be used for these types of motion independent pattern recognition tasks?

Firstly, before we commence I will recommend that you refer to similar questions on the network i.e. https://stackoverflow.com/questions/6499880/ios-gesture-recognition-utilizing-accelerometer-and-gyroscope and https://stackoverflow.com/questions/6368618/store-orientation-to-an-array-and-compare
Your problem can be divided into three parts.
How to gather sensor data.
How to use the gathered data to train a model
How to use the trained model to make a prediction.
A modern smartphone contains around six sensors packed into one device. To implement your application I recommend that you use raw sensor data from either the gyroscope or the accelerometer.
On the android platform, you can access these sensors and acquire the raw sensor data by using the Android sensor framework. The Android sensor framework is part of the android.hardware package.

To capture the raw sensor data, take regular samples (> 20 Hz) and save the maximum values of x, y and z in an array each (to recognize in all 3 planes). If you want the gesture to be 5 seconds long, keep 100 samples (at 20 Hz). Then analyze if any of the three arrays has values which change sinusoidally. If it does, you've got a gesture.
You could store these values into an array, if the user is in 'record mode'. And when the user tries to replicate that movement, the model could predict the replicated movement array from the recorded one's. The thing is, how can you compare the arrays in a smart way? (Randy M 2011)

This leads us to the next step which is applying ML.
To train your model, you can choose to either train the model on the cloud or train it locally on the device. In most cases the problem of training a model on a mobile device is about computational limitations. Machine Learning algorithms running on a mobile device need to be carefully designed and implemented since most mobile devices have weak processors and small RAM's.
Its quite a challenge to squeeze a large neural network (NN) into the small RAM's that smartphones's have since NN's require that the model is fully loaded into memory. In many cases it is advisable to slim the model down and set some weights near zero to zero. (Chomba B 2017)
Incase you decide to utilize the cloud. Your mobile app is required to simply send an HTTPS request to a cloud web service along with the required raw sensor data, and within seconds the service replies with the prediction results. Though there are several cloud services i.e. Microsoft Azure Cognitive Services, Clarifai and Google Cloud Cognition that you can leverage to host the server side of your application, personally I recommend that you consider using reality.ai which is specifically an AI tool for engineers working with signals and sensors.
The next step will be to select an appropriate algorithm to be used in classifying the gesture. Here you can employ either Logistic Regression, Support Vector Machines, Random Forest or Extremely Randomized Trees depending on your app's use case. In order to train your model we then provide the learning algorithm with 'labeled examples'. The ML algorithm then extracts the features and constructs a mathematical model that can accurately describe gestures i.e. roll, pan from raw sensor data.


How did the OpenAI 5 for Dota concatenate units? I am no expert in the field of AI so I apologize if this is a simple/easy question. I was trying to implement a network similar to OpenAI's for another game and I noticed that I did not fully understand how the network worked.
Below is the image of OpenAI five's network

Basic question
How is the data concatenated/ what is the dimension of the data right after the concatenation before the LSTMs or just before entering the LSTMs?
Below are my thoughts which I provide for clarification's sake.
1st Interpretation
In the blue area for units, my initial understanding was that for each visible unit, the output of the max-pool is concatenated along the columns. So, assuming the number of rows is 1(as a 1d max-pool is being applied for each unit), the number of columns is n and there are N visible units, the final size of the matrix when concatenated is $(1,n\cdot N)$ with few extra columns given by the pickups and the like as shown on the left-hand side of the model.
Problem with this interpretation
As the number of units a player can see per each turn is not constant, under this interpretation, I suspect that the fully connected layer after the concatenation layer cannot do its job as matrix multiplication becomes impossible with a variable number of columns.
Possible solution
One possible solution to this is to set a maximum to the number of observed units as $N_{max}$ and pad with constants if some units are not observed. Is this the case?
2nd Interpretation
My 2nd interpretation is that the data is concatenated along the rows. In this case, I can see that the data can pass through a fully connected layer because the number of columns can remain constant. Under this assumption, I decided that right before going through the LSTM, the data is reshaped to (batch size, number of rows, number of columns).
Problems with this interpretation
While I found this interpretation to be more appealing, I noticed that under this train of thought, the LSTM is used just to associate the input data and is not associated with time(the time step for the LSTM is simply the next row of data rather than actual time). I know that this is not especially a problem but I thought that there is no special need to use an LSTM here as in this second interpretation, the order of the data holds no special meaning. But is this the case?
I apologize in advance for any unclear points. Please tell me in the comments and I'll try to clarify as best as I can!

Tl;dr max-pool
You can see in the diagram, everywhere there are a variable number of inputs (pickups, units, hero modifiers/abilities/items), a max-pool follows, though I don't know the specifics of the max-pool implementation.
From https://neuro.cs.ut.ee/the-use-of-embeddings-in-openai-five :

Notice that while the number of modifiers, abilities and items is variable, the network max-pools over each of those lists. This means that only the highest value in all those dimensions actually gets through. At first it does not seem to make sense – it might give the impression that you have an ability that is combination of all existing abilities, e.g. ranged passive heal. But it seems to work for them.

Above processing is done separately for each of the nearby units, the results from general attributes, hero modifiers, abilities and items are all concatenated together. Then different post-processing is applied depending if it was enemy non-hero, allied non-hero, neutral, allied hero or enemy hero.

Finally the results of post-processing are max-pooled over all units of that type. Again this seems to be questionable at the first sight, because different qualities of nearby units would be combined, e.g. if one of the dimensions would represent the health of a unit, then the networks sees only the maximum health over the same type of units. But, again, it seems to work fine.


How to extract parameters from a text using AI/NLP lets say I have three texts:

"make a heading that says hello word"
"make a heading of hello world"
"create heading consist of hello world"

How can I fetch those groups of words using AI which is referring to heading i.e hello world in this case. Which AI frameworks or libraries can do that?
in all examples heading is pointing to hello world (which i am referring as group of words). so basically i want those words which will be a part of heading or in other word there is a relationship between them. another example i can give is "I am watching Breaking bad" so there is a relationship between watching and breaking bad and i want to extract what are you watching.
What's the best approach? Do I have to train a model for that or there are some other techniques that can get it done?

I think U are looking for PLSA
for PLSA either U find out those topics(catogeries) with EM or NNMF
Personally I recommend NNMF
or u can use LDA which is Bayesian version of PLSA
here is code for PLSA:
https://github.com/Man-ash/Probabilistic-Latent-semantic-analysis
which use NNMF
for EM method I code it by myself but i am not sure if it is right
https://stackoverflow.com/questions/65783880/problem-about-m-step-in-probabilistic-latent-semantic-analysis-with-code


How do you measure multi-label classification accuracy? Multi-label assignment is the task in machine learning to assign to each input value a set of categories from a fixed vocabulary where the categories need not be statistically independent, so precluding building a set of independent classifiers each classifying the inputs as belong to each of the categories or not.
Machine learning also needs a measure by which the model may be evaluated. So this is the question how do we evaluate a multi-label classifier?
We can’t use the normal recall,  accuracy and  F measures since they require a binary is it correct or not measure of each categorisation. Without such a measure we have no obvious means to evaluate models nor to measure concept drift.

Your intuition is correct. We do use other metrics for multi-label classification. The meaning of evaluation itself changes. Apart from grading the classifier on whether it classifies correctly or not, we also have to penalize it, if it chooses the wrong class appropriately. You could use the following metrics:

micro/macro averaging Recall/Precision, etc.
Hamming Loss
Subset Accuracy


What are some online courses for deep reinforcement learning? What are some (good) online courses for deep reinforcement learning?

I would like the course to be both programming and theoretical. I really liked David Silver's course, but the course dates from 2015. It doesn't really teach deep Q-learning at this time.

This answer will point the reader to potentially useful resources, but I can't ensure that the courses are good (because I have never followed them).
Free

Reinforcement Learning in the Open AI Gym (a small course that you can find in the YouTube channel suggested in the other answer) by Phil Tabor

The free course Advanced Deep Learning & Reinforcement Learning by DeepMind can be useful. See video 14, which discusses deep RL topics (e.q. DQN).

The free course CS234: Reinforcement Learning Winter 2020 can also be useful. In particular, lesson 6 is about deep reinforcement learning. Here's the course schedule.

Paid

Advanced AI: Deep Reinforcement Learning in Python (at Udemy) seems to be completely dedicated to DRL.

Machine Learning: Beginner Reinforcement Learning in Python  (at Udemy)

Deep Reinforcement Learning 2.0 (at Udemy)

Modern Reinforcement Learning: Deep Q Learning in PyTorch by Phil Tabor

Modern Reinforcement Learning: Actor-Critic Methods by Phil Tabor

In any case, if you are familiar with RL and deep learning topics, I encourage you to directly read the DQN papers (both by DeepMind folks)

Playing Atari with Deep Reinforcement Learning (2013)
Human-level control through deep reinforcement learning (2015)

Of course, deep RL isn't just DQN, but these are two very important papers that you should read.
You should also note that deep RL isn't anything special. It's just RL concepts combined with function approximators and deep learning tricks or deep neural networks. So, if you are really familiar with deep learning and RL (including the usage of neural networks to approximate policies and value functions), you don't need any course to understand deep RL concepts. You can just pick any deep RL paper and you can potentially understand it, although you may require more than 1-2 iterations (but that depends on the person).


What is the simplest classification problem which cannot be solved by a perceptron? What is the simplest classification problem which cannot be solved by a perceptron (that is a single-layered feed-forward neural network, with no hidden layers and step activation function), but it can be solved by the same network if the activation function is swapped out to a differentiable activation function (e.g. sigmoid, tanh)?
In the first case, the training would be done with the perceptron training rule, in the second case with the delta rule.
Note that regression problems cannot be solved by perceptrons, so I'm interested in classification only.

Anything that is not linearly separable cant be solved perceptrons, unless you use feature maps on data to map them to a higher dimension in which it is linearly separable.

As a simple, concrete example, perceptron cant learn the XOR function.

This page might help you further.


Cropping image using ML? Is there a way to train AI to find aspecific line or symbol in a image and crop it?

OpenCV scripts finds a face and crops it: how can I add my annotations?

Lest say I have a image like this:

 +------+
 |  *  *|
 |      |
 |  *  *|
 |      |
 +------+

I want it to find the * and crop it.

Yes, this is possible.

There is actually a pretty easy way that doesn't even require machine learning and can be implemented with a small amount of code. You just use a framework for image processing (e.g. PIL for Python), find the marks by going over your image with an appropriate filter and use the implemented crop function, that the framework hopefully provides (or write i yourself with 2 nested for loops copying the area). Your filter will react strongest in the areas of your image that look like the mark you defined. This could actually be implemented like a single layer conv+RELU layer in a CNN (see this introduction for details about conv and RELU layers).

Because you are asking explicitly about machine learning, I will also give you the hard version to solve this problem using a full CNN to identify the bounding box. Once you have found the bounding box, the actual cropping can be done like in the example above and is not part of the CNN. You could call it post processing if you like.

The technique you are looking for is called Object Localization and Detection. The linked article should tell you all the details you need. You basically build and train your CNN to localize and identify interesting objects in your image. The easy way is to train your CNN to identify the 4 marks individually and return the coordinates of the 4 areas with the highest likelihood to contain your marks. You calculate the center of those 4 individual points and feed it to your cropping function. The hard way is to find the smallest bounding box containing all 4 marks. This is a much harder problem, because the neural network needs to learn that it should ignore everything except those 4 marks during detection, although they will be a small portion of the bounding box. It can be done, but looking for the 4 marks individually would be much easier.

Which approach is best for you? Depends on your goal. If you need a robust and efficient algorithm for this problem, forget about ML and implement it with the straight forward approach I described first. No need for fancy learning CNNs. You can implement this with a few dozen lines of code.

If this is a research or training project to study ML, take my second approach. The easy and the hard way are both fine, the hard one obviously more challenging for your CNN. If you are new to CNN, I recommend following this Stanford Course. It teaches you all you need for your project in 25 to 30 hours (excluding homework).

Edit: Concerning your question, how do you add your own marks. I would simply do it using with the same framework you use for cropping. Your program needs some kind of image manipulation capabilities and this will allow pixel manipulation as well (at least the framework PIL which I suggested has this capability). Just draw your mark using this method. You can also copy a sample mark with a transparent background using a tool like Gimp manually, if you prefer that.


Once the environments are vectorized, how do I have to gather immediate experiences for the agent? My main purpose right now is to train an agent using the A2C algorithm to solve the Atari Breakout game. So far I have succeeded to create that code with a single agent and environment. To break the correlation between samples (i.i.d), I need to have an agent interacting with several environments.

class GymEnvVec():

    def __init__(self, env_name, n_envs, seed=0):
        make_env = lambda: gym.make(env_name)
        self.envs = [make_env() for _ in range(n_envs)]
        [env.seed(seed + 10 * i) for i, env in enumerate(self.envs)]

    def reset(self):
        return [env.reset() for env in self.envs]

    def step(self, actions):
        return list(zip(*[env.step(a) for env, a in zip(self.envs, actions)]))

I can use the class GymEnvVec to vectorize my environment.

So I can set my environments with

envs = GymEnvVec(env_name="Breakout-v0", n_envs=50)

I can get my first observations with

observations = envs.reset()

Pick some actions with

actions = agent.choose_actions(observations)

The choose_actions method might look like

def choose_actions(self, states):
        assert isinstance(states, (list, tuple))

        actions = []
        for state in states:
            probabilities  = F.softmax(self.network(state)[0])
            action_probs = T.distributions.Categorical(probabilities)
            actions.append(action_probs.sample())

        return [action.item() for action in actions]

Finally, the environments will spit the next_states, rewards and if it is done with

next_states, rewards, dones, _ = env.step(actions)

It is at this point I am a bit confused. I think I need to gather immediate experiences, batch altogether and forward it to the agent. My problem is probably with the "gather immediate experiences".

I propose a solution, but I am far from being sure it is a good answer. At each iteration, I think I must take a random number with

nb = random.randint(0, len(n_envs)-1)

and put the experience in history with

history.append(Experience(state=states[nb], actions[nb], rewards[nb], dones[nb]))

Am I wrong? Can you tell me what I should do?

class ExperienceSource():
    def __init__(self, env, agent, reward_steps):
        self.env = env
        self.agent = agent
        self.reward_steps = reward_steps

    def __iter__(self):
        histories = [deque(maxlen=self.reward_steps) for i in range(len(self.env.envs))]
        states = self.env.reset()

        while True:

            for idx, env in enumerate(self.env.envs):
                action = self.agent.choose_action(states[idx])
                state, reward, done, _ = env.step(action)

                current_rewards[idx] += reward
                histories[idx].append(Experience(state, action, reward, done))

                if len(histories[idx]) == self.reward_steps:
                    yield tuple(histories[idx])

                if done:
                    yield tuple(histories[idx])
                    state = env.reset()

Be aware that self.reward_steps is simply the value defined by N-1 in the following formula $$Q(s,a) = \sum_{i=0}^{N-1} \gamma^i r_i
 + \gamma^N V(s_N)$$ and self.env is simply an instance of GymEnvVec class from the question.