Does training happen during NEAT? When one uses NEAT to evolve the best fitting network for a task, does training take place in each epoch as well?

If I understand correctly, training is the adjustment of the weights of the neural network via backpropagation and gradient descent. During NEAT, say, a generation runs for 1000 iterations. During that time, is there any training involved, or does each genome randomly poke around and the winner takes it to the next stage?

I've used NEAT, but the fact that neural networks are not trained does not make sense to me. At the same time, I can't find any code in my framework (Neataptic.js) that would train the generation during the epoch.

NEAT uses genetic algorithms both to search for improved connection weights and for improved architectures.

Whilst it is possible to train a NEAT-generated neural network using backpropagation of error gradients, libraries implementing "original" NEAT will not implement that.

There are a couple of reasons:

There is often no training data, in a supervised learning sense. The fitness function of a NEAT system can be measured arbitrarily by performance at some task, and in the general case this consists of some environment simulation that interacts with an agent controlled by the NN, and not training data per se.
Evolved network topologies do not conform to stacked layers models preferred by ML frameworks designed to run typical deep learning architectures.

Both of these issues can be resolved with a little effort (for instance for the second issue there are frameworks which will work with arbitrary feed forward connection graphs, they are just a little more niche than e.g. Keras). However, NEAT is often used because it can solve problems without needing to frame them as supervised learning or reinforcement learning.

Other than the hard work of putting it together, there is nothing stopping you creating a train-on-data stage which could alternate between the two approaches, perhaps with controllable weighting from evolution to gradient-based training. To add the gradient-based training, then either:

a) Your original problem is one of fitting to a classifier or regression problem. In which case your fitness function and training loss function could be the same.

b) Your original problem is one of controlling an agent in an environment, and you could potentially use something like the REINFORCE algorithm based on most-recent assessments, in order to provide gradients to train the NN. Other policy gradient methods could also be a good fit, as the NEAT network typically outputs a policy as opposed to a value prediction.

I have never tried these (nor much with NEAT at all other than demos and a bit of theory). For (a) I would expect the combination to be successful, but wonder why you were bothering with NEAT in the first place. For (b) I am less sure whether you would get useful results, because REINFORCE relies on multiple runs with the same network, whilst NEAT relies on stochastic search across multiple networks. Applying REINFORCE training across a whole population could be very CPU intensive.


Why does L1 regularization yield sparse features? In contrast to L2 regularization, L1 regularization usually yields sparse feature vectors and most feature weights are zero.

What's the reason for the above statement - could someone explain it mathematically, and/or provide some intuition (maybe geometric)?

In L1 regularization, the penalty term you compute for every parameter is a function of the absolute value of a given weight (times some regularization factor).
Thus, irrespective of whether a weight is positive or negative (due to the absolute value) and irrespective of how large the weight is, there will be a penalty incurred as long as weight is unequal 0. So, the only way how a training procedure can considerably reduce the L1 regularization penalty is by driving all (unnecessary) weights towards 0, which results in a sparse representation.
Of course, the L2 regularization will also only be strictly 0 when all weights are 0. However, in L2, the contribution of a weight to the L2 penalty is proportional to the squared value of the weight. Therefore, a weight whose absolute value is smaller than 1, i.e. $abs(weight) < 1$, will be much less punished by L2 than it would be by L1, which means that L2 puts less emphasis on driving all weights towards exactly 0. This is because squaring a some value in (0,1) will result in a value of lower magnitude than taking the un-squared value itself: $x^2 < x\ for\ all\ x\ with\ abs(x) < 1$.
So, while both regularization terms end up being 0 only when weights are 0, the L1 term penalizes small weights with $abs(x) < 1$ much more strongly than L2 does, thereby driving the weight more strongly towards 0 than L2 does.


How to know if my DQN is optimized? I made a DQN that controls a traffic light. The observation states are the number of vehicles of each lane in the intersection. I trained it for 500 episodes and saved the model every 50th episode. I plotted the reward curve of the model after the training and found out that around the 460th episode, the reward curve become unstable. Does it mean that the optimized DQN model is the 450th model? If not, how do I know if the my DQN is really optimized?

There is a good chance that your DQN is already optimized but you would have to take a look at its performance to really check and see whether its actions seem up to par.
Reasons it may not be optimized: If you are tracking the reward after every episode, unstable rewards are very common just due to random chance, but if you are averaging the reward over the past 50 episodes or so, then it may also be your learning rate or epsilon.
If your learning rate is too high or low you may either never be able to reach a fully optimized DQN or be stuck in a local minimum. An easy way to solve a problem like this would be to just add a simple learning rate decay, so that the learning rate would start off high as to not get stuck in local minima yet decay to a small enough number where you know that the agent has found the global minimum.
The other problem could be that your epsilon may be too high or low. A high epsilon will never allow the agent to fully optimize while a low epsilon doesn't allow the agent to explore and discover better strategies, so it may be a good idea to mess around with this as well.
The only way to really gauge the agent's performance would be to look at it making some decisions through a video or by analyzing some of its predictions. And, if it seems to be doing well, then it may very well be optimized, but, if the agent is not performing as well as it should, then it may be a good idea to try out some of the strategies above.


Selecting the right technique to predict disease from symptoms I'm trying to come up with the right algorithm for a system in which the user enters a few symptoms and the system has to predict or determine the likelihood that a few selected symptoms are associated with those existing in the system. Then after associating them, the result or output should be a specific disease for the symptoms.

The system is comprised of a series of diseases with each assigned to specific symptoms, which also exist in the system.

Let's assume that the user entered the following input:

A, B, C, and D

The first thing the system should do is check and associate each symptom (in this case represented by alphabetical letters) individually against a data-table of symptoms that already exist. And in cases where the input doesn't exist, the system should report or send feedback about it.

And also, let's say that A and B was in the data-table, so we are 100% sure that they're valid or exist and the system is able to give out the disease based on the input. Then let's say that the input now is C and D where C doesn't exist in the data-table, but there is a possibility that D exists.

We don't give D a score of 100%, but maybe something lower (let's say 90%). Then C just doesn't exist at all in the data-table. So, C gets a score of 0%.

Therefore, the system should have some kind of association and prediction techniques or rules to output the result by judging the user's input.

Summary of generating the output:

If A and B were entered and exist, then output = 100%
If D was entered and existed but C was not, then output = 90%
If all entered don't exist, then output = 0%

What techniques would be used to produce this system?

I think you're coming at your problem slightly wrong... what you're essentially talking about is a belief network.
You may want to look into existing Bayesian Learning techniques to get your head around this, but belief networks commonly use the exact scenario you're talking about; using a set of known (or uncertain facts) statements to produce some inferred probability of a particular output.
Even more, they often express this through disease-symptom based examples in tutorials! Try here.
My point is that it would be better to use a belief network as the theoretical groundwork is all already there for you, instead of an ANN.


Why is the change in cost wrt bias in neural network equal to error in the neuron? While reading the book on neural networks by Michael Nielson, I had a problem understanding equation (BP3), which is
$$
\frac{\partial C}{\partial b_{j}^{l}}=\delta_{j}^{l} \tag{BP3}\label{BP3},
$$
which can be translated to plain English as follows

The change in cost, $C$, with respect to the bias of a neuron, $b_{j}^{l}$, is equal to the error in that neuron, $\delta_{j}^{l}$.

Why would this be the case?

This is just an application of the chain rule. The same chapter has "Proof of the four fundamental equations" section, which proves BP1-2, while PB3-4 are left as exercise to the reader. I agree that it's a good exercise indeed, that's why I encourage you to stop here and try to prove it yourself using a chain rule.

Now, if you decided to read further, here's the sketch of a proof.

Recall equations (25) and (29), both definitions:

$z^l$ is a linear transformation of $a^{l-1}$: $z^l = w^l a^{l-1} + b^l$
$\delta^l$ is a partial derivative of $C$ with respect to $z^l$: $\delta^l  = \dfrac{\partial C}{\partial z^l}$

The chain rule itself:

the partial derivative of $C$ with respect to $b^l$:

$$\frac{\partial C}{\partial b^l_j} = \sum_k \frac{\partial C}{\partial z^l_k} \frac{\partial z^l_k}{\partial b^l_j}$$

Almost all elements in this sum will be zero, except for one when $k=j$.
The first term in it is by definition $\delta^l_j$, the second term $\dfrac{\partial z^l_j}{\partial b^l_j}$ is $1$, because $z$ is linear with respect to $b$.

The derivative with respect to weights is taken the same way, and differs only in the last term: $z$ is linear with respect to $w$ as well, but with a coefficient $a^{l-1}$.


How does visual cortex share convolution weight TL;DR
If we buy into the idea visual cortex functions like a convolutional neural network, then there's a problem makes me scratch my head: how does brain force weight sharing as in convolutional network?
Okay, explain more
Obviously, there's no way for left visual cortex to directly tell the right visual cortex "hey, I've learned some new stuff, copy me!!" (or is there?). Then, if the learned features are diverse across visual field, how does it keep the translation invariance property?
For example, you already know English characters, you can recognize them with your both eyes. Now that you wanna learn some Chinese and you  excercise your right brain at the same time, so you closed your right eye and memorized a new character. After that, certainly you can recognize the new character with solely your right eye. But why?
The answer may be, the object / higher-level feature detection happens in a higher level cortex, which receives entire visual field. There may be also some transfer/one-shot learning taking place. But then, if a newborn baby trying to learn the low level visual features, he/she would definitely face the weight sharing problems.
A possible explanation would be, the baby will be exposed to very large amount of data and eventually learn invariance. Large amount of data reduces overfitting but doesn't guarantee deterministic convergence. If we train the same CNN model on the same dataset, however using different random generator seeds, there's a big chance the same feature detector will appear in a different channel, or a difference set of features appear as linear recombination.
If there's no way to share weights, the brain would learn a lot different feature combinations across the entire visual field, how does it still able to consistently solve visual invariance problem?

In the human brain every common pattern is recognised by a multitude of pattern recognisers (be they neurons or microcolumns). That's pretty obvious because neurons die all the time, but we don't wake up and have forgotten how to recognise the letter 'A'. In fact we have to take out rather big chunks of the neocortex until functionality degrades, which is why Alzheimer's patients show symptoms only when the brain is already visibly messed up.

Translational invariance within a level of the hierarchy has to be learned. Basically you need filters for the same edge all over your V1. An object moving across your field of vision only becomes invariant in it's representation on a higher level. Unfortunately we don't magically learn a representation that we can easily turn and twist, our ability to recognise objects from an unusual angle degrades with how uncommon the angle is.

A nice thought experiment to illustrate the point: Imagine a cube sitting in front of you. Now pull one of the corners of the cube upwards until the cube is dangling in front of you, one corner pointing straight to the ceiling, the opposite corner pointing to the ground. Now indicate with your finger where the rest of the corner are.

If you are anything like me, this is really difficult. I think the first time I did't even realise I had to point out six corners!

Of course this is 3D and we might still have inbuilt 2D invariance, but it also turns out that faces that are turned upside down have to be processed much higher into the cortex to be recognised as faces and so on ...

Concerning the learning of a lot of different feature vectors across the field of vision: This might be prevented by the fact that the neocortex learns sequences of input by predicting (via depolarisation) the next pattern. So you might have the situation that a higher level tells the lower level that there is an object moving from left to right and the lower level will predict that the edge it is detecting at point y will reoccur at point y+x.

This setup differs from the training of NNs in two ways: The data is already translational and the translation is predicted, which facilitates learning the same features. Basically two pattern recogniser close to each other get pretty much the same pattern one after the other whenever something moves in a certain direction and the second occurrence is predicted, which means it will be more likely detected, which means it will be learned. (I don't want to dole out a lecture about the cortex, but the prediction is a big deal because it allows neurons to fire quicker which means they beat other neurons before they are laterally inhibited and only if they actually fire will they learn.)

As a disclaimer I want to add that this is just my current understanding of the issue and I'm pretty sure the actual explanation is more complicated. For example I've read that one of the layers of the cortex is important for translational invariance and this layer only exists in the levels close to the sensory input. It is conjectured that this layer (L4) doesn't do the sequence prediction stuff, so maybe having the same kind of input is enough or learning different feature vectors is not actually a problem. There is also the complicating issue that there is poorly understood interplay between the different layers and different levels of the cortex. I would recommend to ask a neuroscientist, except I don't think they figured this out yet.


What should we do when we have equal observations with different labels? Suppose we have a labeled data set with columns $A$, $B$, and $C$ and a binary outcome variable $X$. Suppose we have rows as follows:

 col  A B C X
  1   1 2 3 1
  2   4 2 3 0
  3   6 5 1 1
  4   1 2 3 0

Should we throw away either row 1 or row 4 because they have different values of the outcome variable X? Or keep both of them?

The problem you are portraying looks like a modified XOR problem. You can't throw away the lines with a label of 1 because a the model won't be able to learn this class.


Does the "lowest layer" refer to the first or last layer of the neural network? People sometimes use 1st layer, 2nd layer to refer to a specific layer in a neural net. Is the layer immediately follows the input layer called 1st layer?

How about the lowest layer and highest layer?

Lowest layer generally refers to the layer closest to the input. This comes from the idea that layers closer to the input represent low-level features such as gradients and edges, while layers closer to the output represent high-level features such as parts and objects.


Is there a way of computing a prominence score based on the prevalence of features in an image? Is there any previous work on computing some sort of prominence score based on the prevalence of features in an image?

For example, let's say I am classifying images based on whether or not they have dogs in them. Is there a way to compute how prominent that feature is?

Saliency is typically discussed more in object detection or scene understanding than prominence.  There are lots of papers on saliency models.  "What do different evaluation metrics tell us about saliency models?" is a good paper on various metrics on saliency models.  It covers the following metrics:

Similarity or histogram intersection (SIM)
Pearson's Correlation Coefficient (CC)
Normalized Scanpath Saliency (NSS)
Area Under ROC Curve (AUC)
Information Gain (IG)
Kullback-Leibler divergence (KL)
Earth Mover's Distance (EMD)

Some other papers you may find interesting:

Saliency, attention, and visual search: An information
theoretic approach
Towards Instance Segmentation with Object Priority:
Prominent Object Detection and Recognition


Optimum Discriminator for label smoothed GAN I was reading the paper called Improved Techniques for Training GANs. And, in the one-sided label smoothing part, they said that optimum discriminator with label smoothing is
$$ D^*(x)=\frac{\alpha \cdot p_{data}(x) + \beta \cdot p_{model}(x)}{p_{data}(x) + p_{model}(x)}$$
I could not understand where this is come from. How did we get this result?
Note: By the way, I knew how to find optimal discriminator in vanilla GAN i.e.
$$ D^*(x) = \frac{p_{r}(x)}{p_{r}(x) + p_g(x)} $$

The equation most likely comes from one of the following references:

David Warde-Farley and Ian Goodfellow. Adversarial perturbations of deep neural networks. In Tamir Hazan, George Papandreou, and Daniel Tarlow, editors, Perturbations, Optimization, and Statistics, chapter 11. 2016. Book in preparation for MIT Press.

C. Szegedy, V. Vanhoucke, S. Ioffe, J. Shlens, and Z. Wojna. Rethinking the Inception Architecture for Computer Vision. ArXiv e-prints, December 2015.

I was not able to retrieve the book but Szegedy et al. discuss on page 6 (see "Model Regularization via Label Smoothing").
Ian Goodfellow discusses this topic on page 31 of his tutorial:

I. Goodfellow. NIPS 2016 Tutorial: Generative Adversarial Networks. ArXiv e-prints April 2017.

A similar question was asked and answered in the Data Science community: https://datascience.stackexchange.com/questions/28764/one-sided-label-smoothing-in-gans


Turn photos right-side up? I'm looking for either an existing AI app or a pre-trained NN that will tell me if a photograph is right-side up or not. I want to use this to create an application that automatically rotates photos so they are right-side-up. This doesn't seem hard.

If it doesn't exist, presumably I can create it with Tensorflow, and just use a ton of photos to train it, and assume they are all correctly oriented in the training set. Would that work?

I don't know if there is an existing pretrained NN that does this but it wouldn't be very hard to modify one to do this.

First, I'd take a pretrained image classification NN (e.g. VGG, ResNet), drop its final layer and replace it with one with 4 neurons, representing the 4 orientations (so that you know which way to rotate it).

Then I'd take again a dataset of regular images (e.g. a subset of ImageNet) and assume that they are correctly oriented. I'd make three more duplicate datasets with the same images rotates by 90, 180 and 270 degrees respectively. These 4 datasets would be the 4 classes I'd fine tune the model on.

By training your model on this dataset, you'll be training it to recognize which side your image is facing. Since it's a pretrained net and its a fairly simple task, I think that after a few iterations, your model will have converged. Then you could write a script that uses this model to predict an image's orientation and rotate it accordingly.


Trying to understand why nonlinearity is important for neural networks by analogy Is the reason why linear activation functions are usually pretty bad at approximating functions the same reason why combinations of hermitian polynomials or combinations of sines and cosines are better at approximating a function than combinations of linear functions?
For example, regardless of the amount of terms in this combination of linear functions, the function will always be some form of $y = mx + b$. However, if we're summing sines, you absolutely cannot express a combination of sines and cosines as something of the form $A \sin{bx}$. For example, a combination of three sinusoids cannot be simplified further than $A \sin{bx} + B \sin{cx} + D \sin{ex}$.
Is this fact essentially why the Fourier series is able to approximate functions (other than obviously the fact that $A \sin{bx}$ is orthogonal to $B \sin{cx}$)? Because if it could be simplified into one sinusoid, it could never approximate an arbitrary function because it's lost its robustness? Because with other terms combined, whereas linear functions summed up gain no further ability to approximate, things like sinusoids actually begin to approximate really well with enough terms and with the right constants.
In that vein, is this the reason why non-linear activiation functions (also called non-linear classifiers?) are generally valued more than linear ones? Because linear activation functions simply are lousy function approximators, while, with enough constants and terms, non-linear activation functions can approximate any function?

Your analogy is correct, except it is not really an "analogy". Sin is an activation function - in past works (before modern deep learning boom) it was rather standard to see it listed as a possible activator.
So your expression $\sigma(x) = A\sin ax + B \sin bx + D \sin ex$ is of a neural network with one 3-neuron layer and a single output linear neuron:
$$\sigma(x) = \sum_i V_i \sin\left(\sum_jW_{ij} x_j + b_i\right)+\beta$$
With all biases being zero $b_i=\beta=0$, output weights are $V_i = \left(A,B,C\right)$ and the inner weight matrix is diagonal:
$$W_{ij} = \begin{pmatrix}a&0&0\\0&b&0\\0&0&e\end{pmatrix}$$


How would DeepMind's new differentiable neural computer scale? DeepMind just published a paper about a differentiable neural computer, which basically combines a neural network with a memory.
The idea is to teach the neural network to create and recall useful explicit memories for a certain task. This complements the abilities of a neural network well, because NNs only store knowledge implicitly in the weights and the information used to work on a single task is only stored in the activation of the network and degrades quickly the more information you add. (LSTMs are one try to slow down this degradation of short term memories, but it still happens.)
Now, instead of keeping the necessary information in the activation, they presumably keep the addresses of memory slots for specific information in the activation, so these should also be subject to degradation.
Why would this approach scale? Shouldn't a somewhat higher number of task-specific information once again overwhelm the network's capability of keeping the addresses of all the appropriate memory slots in its activation?

Examining the architecture of the DNC indeed shows many similarities to the LSTM. Consider the diagram in the DeepMind article that you linked to:

Compare this to the LSTM architecture (credit to ananth on SlideShare):

There are some close analogs here:

Much like the LSTM, the DNC will perform some conversion from input to fixed-size state vectors (h and c in the LSTM)
Likewise, the DNC will perform some conversion from these fixed-size state vectors to potentially arbitrarily-lengthed output (in the LSTM we repeatedly sample from our model until we are satisfied/the model indicates we are done)
The forget and input gates of the LSTM represent the write operation in the DNC ('forgetting' is essentially just zeroing or partially zeroing memory)
The output gate of the LSTM represents the read operation in the DNC

However, the DNC is definitely more than an LSTM. Most obviously, it utilizes a larger state which is discretized (addressable) into chunks; this allows it to make the forget gate of the LSTM more binary. By this I mean that the state is not necessarily eroded by some fraction at every time step, whereas in the LSTM (with the sigmoid activation function) it necessarily is. This might reduce the problem of catastrophic forgetting that you mentioned and thus scale better.

The DNC is also novel in the links that it uses between memory. However, this might be a more marginal improvement on the LSTM than it seems if we re-imagine the LSTM with complete neural networks for each gate instead of just a single layer with an activation function (call this a super-LSTM); in this case, we can actually learn any relationship between two slots in memory with a sufficiently powerful network. While I don't know the specifics of the links that DeepMind is suggesting, they imply in the article that they are learning everything just by backpropagating gradients like a regular neural network. Therefore whatever relationship they are encoding in their links should theoretically be learnable by a neural network, and so a sufficiently powerful 'super-LSTM' should be able to capture it.

With all that being said, it is often the case in deep learning that two models with the same theoretical capability for expressiveness perform vastly different in practice. For example, consider that a recurrent network can be represented as a huge feed-forward network if we just unroll it. Similarly, the convolutional network isn't better than a vanilla neural network because it has some extra capacity for expressiveness; in fact, it is the constraints imposed on its weights that makes it more effective. Thus comparing the expressiveness of two models isn't necessarily a fair comparison of their performance in practice, nor an accurate projection of how well they will scale.

One question I have about the DNC is what happens when it runs out of memory. When a classical computer runs out of memory and another block of memory is requested, programs start crashing (at best). I'm curious to see how DeepMind plans to address this. I assume it will rely on some intelligent cannibalization of memory currently in use. In some sense computers currently do this when an OS requests that applications free up non-critical memory if memory pressure reaches a certain threshold.


If deep learning is a black box, then why are companies still investing in it? If deep learning is a black box, then why are companies still investing in it?

Easy answer: utility.

The strength and applicability of "black box" NNs has been regularly validated in the past few years, and business is concerned with results.  (i.e. they don't care how the sausage is made, so long as it gets made.)


Complete deep learning text classification with Python example I would like to know if there is a complete text classification with deep learning example, from text file, csv, or other format, to classified output text file, csv, or other. I have seen tens of tutorials and they mostly focus on the model and its performance, but I have not been able to find one that shows how to apply the model to a set of text strings and how to output a document with the classified(labeled) text.

You may look at these github repositories :

Text classification in NLP with deep learning
Convolutional Neural Networks for Sentence Classification


Is the expert system still in use today? In my country, the Expert System class is mandatory, if you want to take the AI specialization in most universities. In class, I learned how to make a rule-based system, forward chaining, backward chaining, Prolog, etc.
However, I have read somewhere on the web that expert systems are no longer used.
Is that true? If yes, why? If not, where are they being used? With the rise of machine learning, they may not be as used as before, but is there any industry or company that still uses them today?
Please, provide some references to support your claims.

I would say Expert Systems is still being taught. For instance, if you look at some of the open courses like MIT's, there are still lectures on it.

Also, looking at the CLIPS documentation, you will find a couple of examples of usage from 2005.

What I suspect is that Expert Systems are now embedded with "normal systems" in practice. Hence it may be difficult to distinguish from systems used on a daily basis for diagnostics, etc. and not as popular as before.


Is there a neural network method for time-varying directed graphs? I want to study NN for time-varying directed graphs. However, as this field has developed relatively recently, it is difficult to find new ways. So the question is, is there any NN that can handle such data?

I'm seeing recent trend of combining RNN/CNN with GNN(graph neural networks) so that both time dependency and topology are captured. I would suggest you to start by looking at DCRNN (Yaguang Li et al.), it's a strong baseline that everyone uses nowadays. Other good resources:

Graph wavenet for deep spatial-temporal graph modeling (Zonghan Wu et al.)
Spatio-Temporal Graph Convolutional Networks: A Deep Learning Framework for Traffic Forecasting (Bing Yu et al.)
STANN: A Spatio–Temporal Attentive Neural Network for Traffic Prediction (Zhixiang He et al.)


Is a VGG-based CNN model sometimes better for image classfication than a modern architecture? I have an image classification task to solve, but based on quite simple/good terms:

There are only two classes (either good or not good)
The images always show the same kind of piece (either with or w/o fault)
That piece is always filmed from the same angle & distance
I have at least 1000 sample images for both classes

So I thought it should be easy to come up with a good CNN solution - and it was. I created a VGG16-based model with a custom classifier (Keras/TF). Via transfer learning I was able to achieve up to 100% validation accuracy during model training, so all is fine on that end.

Out of curiosity and because the VGG-based approach seems a bit "slow", I also wanted to try it with a more modern model architecture as the base, so I did with ResNet50v2 and Xception. I trained both similar to the VGG-based model, tried it with several hyperparameter modifications, etc. However, I was not able to achieve a better validation accuracy than 95% - so much worse than with the "old" VGG architecture.

Hence my question is:

  Given these "simple" (always the same) images and only two classes, is the VGG model probably a better base than a modern network like ResNet or Xception? Or is it more likely that I messed something up with my model or simply got the training/hyperparameters not right?

VGG is a more basic architecture which uses no residual blocks. Reset usually perform better then VGG due to it's more layers and residual approach. Given that resnet-50 can get 99% accuracy on MNIST and 98.7% accuracy on CIFAR-10, it probably should achieve better than VGG network. Also, the validation accuracy should not be 100%. You could try increasing the size of your validation set to improve accuracy on validation. VGG network should perform worst than ResNet in most scenario, but experimenting is the way to go. Try and experiment more to get a method that works for your data. Hope that I can help you and have a nice day!


How exactly does MICE imputation combine multiple datasets into one? I'm trying to understand Multiple Imputation with Chained Equation (MICE) imputation process (a statistical method for imputing missing data). I have read some articles and I have understood how the imputation happens, but I didn't get the "pooling" step.
After analyzing the resulting datasets with Rubin's rules, how to pool these datasets? How to get only one dataset?
In the end, do I combine all these datasets? If yes, how? Or do I compare every dataset's estimators with Rubin's estimators and choose one dataset?

MICE is a Multiple imputation method, it contains three phases:

Imputation phase: The missing data are estimated M times from a specific model to obtain M complete and potentially different data sets.
Separate analysis phase: The selected analysis is performed separately on each of m = 1, ..., M sets of imputed data to obtain estimates (central tendency and variance).
Combined analysis phase: The results obtained  are combined according to the rules established by Rubin to obtain a single final estimate.


What make a CNN suitable for image classification or for semantic segmentation? I've just started with CNN and there is something that I haven't understood yet:

How do you "ask" a network: "classify me these images" or "do semantic segmentation"?

I think it must be something on the architecture, or in the loss function, or whatever that makes the network classify its input or do semantic segmentation.

I suppose its output will be different on classification and on semantic segmentation.

Maybe the question could be rewritten to:

What do I have to do to use a CNN for classification or for semantic segmentation?

Disclaimer: This question is very broad, my answer is admittedly partial and is intended to just give an idea of what's out there and how to find out more.

  How do you "say" a network: "classify me these images" or "do semantic segmentation"?

You're mixing two very different problems there. Although there are SO many variations of problems people are applying CNNs to, for this example we can focus on the "classification of something in the image" subset ad identify 4 key tasks:

Image Classification answers the question "What is this image about" (the answer is an image where each pixel is assigned to one of the given classes)
Semantic Segmentation answer he question "What areas of this image are part of a Cat?" (e.g.,
Object Detection answers the question "Where are the objects in the image AND what objects are they?" (e.g. of answer: "Cat in bounding box at x,y,w,h [10,20,50,60]")
Semantic Segmentation answers the question "Where are the individual objects in this image AND what class are they AND give me the pixels that belong to each object". You may guess from the number of ANDs there, this is the hardest of the four. The output here would be a set of class, bounding_box, mask tuples where the mask is typically defined in relation to the returned bounding box.

So, how do we build networks capable of solving one problem or the other?
We build architectures towards one specific problem, exploiting reusable parts where possible. For example, typically classification and object detection are based on a deep "backbone" that extracts highly complex features from the image, that finally are used by a classifier layer interprets to make a prediction (for image classification) or a box prediction head to predict where objects lie in the image (very big simplification, look up object detection architectures and how they work for the proper description!).

  What do I have to do to use a CNN for classification or for semantic segmentation?

In principle you can't just take a network built for classification and just "ask" it to do semantic segmentation (think of it as trying to use a screwdriver as scissors... it just was not built for that!). You need changes in the architecture, which necessarily imply new training, at the very least for the new parts that were added.


When discounted MAB is useful? Many of multi-armed bandit(MAB) algorithms are used when the total reward is the sum of all rewards. However, in RL, the discounted reward is mainly used. Why is the discounted reward not prevailing in MAB problem, and in what cases is this type of modeling valid and might be better?

One of the reasons a discount factor is used, is to make sure the reward maximization is a well-defined problem and to make the sum of all rewards convergent.

In the MAB problem, the number of trials is typically finite owing to some sort of budget in the number of trials. Hence, this is less of problem. However, by all means discounts are still valid and helpful in cases where the analysis in asymptotic in the number of trials.


Would you term Google's Captchas as Turing Test? Quoting from Wikipedia page on Turing Test

  The Turing test, developed by Alan Turing in 1950, is a test of a machine's ability to exhibit intelligent behaviour equivalent to, or indistinguishable from, that of a human. Turing proposed that a human evaluator would judge natural language conversations between a human and a machine designed to generate human-like responses. The evaluator would be aware that one of the two partners in conversation is a machine, and all participants would be separated from one another. The conversation would be limited to a text-only channel such as a computer keyboard and screen...

The definition made me wonder if we would term the captchas that show up on Google here it gives textual instructions to the user, bot or not, asking them to select images containing a specific object out of the given set of images.

In the standard Turing test (or imitation game), the interrogator can ask multiple arbitrary questions, while, in the case of captchas, usually, there's only one question or problem. Additionally, in the Turing test, the interrogator interactively communicates with both the human and the machine. Furthermore, captchas do not test the conversational skills but only the visual ones of the AI. Therefore, the existing captchas are not an example of the standard Turing test.


Are AI researchers more likely to follow "conservative anthropocentrism" or "post-human fundamentalism" in deciding whether AI has moral standing? "Conservative anthropocentrism": AI are to be judged only in relation to how to they resemble humanity in terms of behavior and ideas, and they gain moral worth based on their resemblance to humanity (the "Turing Test" is a good example of this - one could use the "Turing Test" to decide whether AI is deserving of personhood, as James Grimmelmann advocates in the paper Copyright for Literate Robots).

"Post-human fundamentalism": AI will be fundamentally different from humanity and thus we require different ways of judging their moral worth  (People For the Ethical Treatment of Reinforcement Learners is an example of an organization that supports this type of approach, as they believe that reinforcement learners may have a non-zero moral standing).

I am not interested per se in which ideology is correct. Instead, I'm curious as to what AI researchers "believe" is correct (since their belief could impact how they conduct research and how they convey their insights to laymen). I also acknowledge that their ideological beliefs may change with the passing of time (from conservative anthropocentrism to post-human fundamentalism...or vice-versa). Still..what ideology do AI researchers tend to support, as of December 2016?

Many large deployments of AI have carefully engineered solutions to problems (ie self driving cars). In these systems, it is important to have discussions about how these systems should react in morally ambiguous situations. Having an agent react "appropriately" sounds similar to the Turing test in that there is a "pass/fail" condition. This leads me to think that the current mindset of most AI researchers falls into "Conservative anthropomorphism".

However, there is growing interest in Continual Learning, where agents build up knowledge about their world from their experience. This idea is largely pushed by reinforcement learning researchers such as Richard Sutton and Mark Ring. Here, the AI agent has to build up knowledge about its world such as:

  When I rotate my motors forward for 3s, my front bump-sensor activates.

and

  If I turned right 90 degrees and then rotated my motors forward for 3s, my front bump-sensor activates.

From knowledge like this, an agent could eventually navigate a room without running into walls because it built up predictive knowledge from interaction with the world.

In this context, lets ignore how the AI actually learns and only look at the environment AI agents "grow up in". These agents will be fundamentally different from humans growing up in homes with families because they will not have the same learning experience as humans. This is much like the nature vs nurture argument.

Humans pass their morals and values on to children through lessons and conversation. As RL agents would lack much of this interaction (unless families adopted robot babies I guess), we would require different ways of judging their moral worth and thus "Post-human fundamentalism".

Sources:
5 years in the RL academia environment and conversations with Richard Sutton and Mark Ring.


Convergence of semi-gradient TD(0) with non-linear function approximation I am looking for a result that shows the convergence of semi-gradient TD(0) algorithm with non-linear function approximation for on-policy prediction. Specifically, the update equation is given by (borrowing notation from Sutton and Barto (2018))

$$\mathbf w \leftarrow \mathbf w +\alpha [R + \gamma \hat v(S', \mathbf w) - \hat v(S, \mathbf w)] \nabla \hat v(S, \mathbf w)$$

where $\hat v(S, \mathbf w)$ is the approximate value function parameterized by $\mathbf w$.

Sutton and Barto (2018) mention that the above update equation converges when $\hat v$ is linear in $\mathbf w$. But I couldn't find a similar result for non-linear function approximation. Any help would be greatly appreciated.

Apparently there is an example of non-convergence for semi-gradient sarsa, according to Rich Sutton (check slide 35). I guess TD(0) is not so different. So, probably your approximator will need to satisfy certain conditions to proof convergence.

Maybe this paper will be useful for you. It seems that they show that constraining your network to have relu activation functions allow you to show some convergence properties.


Why is an average of all returns used to update the value in the first-visit MC control? In Sutton & Barto's Reinforcement Learning: An Introduction,  in page 83 (101 of the pdf), there is a description of first-visit MC control. In the phase where they update $Q(s, a)$, they do an average of all the returns $G$ for that state-action pair.

Why don't they just update the value with a weight for the value from previous episodes $\alpha$ and a weight $1- \alpha$ for the new episode return as it is done in TD-Learning?

I have also seen other books (for example, Algorithms for RL (page 22) where they update it using $\alpha$. What is the difference?

Why don't they just update the value with a weight for the value from previous episodes $\alpha$ and a weight $1- \alpha$ for the new episode return as it is done in TD-Learning?

In my opinion, this is a mistake in the book. I went back and checked that this is still the same in the finished second edition, and it is still there.

Keeping all returns and taking averages works fine in a prediction scenario with a fixed policy, and it is somewhat simpler and more intuitive to explain as "this is the mean value" (efficiency can come later, after comprehension). However, the pseudo-code is incorrect for a control scenario where older returns (that you still refer in the list) will not reflect the current policy, so will be biased.

In practice, no-one really uses the MC Control algorithms as written. Nearly always a learning rate parameter, $\alpha$ is used to update estimates.

  What is the difference?

In a prediction scenario, where you want to evaluate a fixed policy, it would be slightly more efficient to use a count of samples for each $s,a$ pair and variable $\alpha = \frac{1}{N(s,a)}$ which is mathematically identical to keeping a list and taking the average as needed.

In a control scenario using a list of old return values is not only memory inefficient, it is also sample inefficient as you would need samples from each improved policy to outnumber the older samples in order to remove sampling bias due to using returns from initial worse policies. A fixed, or slowly decaying $\alpha$ is a simple and efficient way of "forgetting" the old return values as they become less relevant to the task.

Another way to achieve similar forgetting may be to have a maximum size for each $Returns(s,a)$ list. However, that is not mentioned in the MC Control pseudocode.

Perhaps I am missing something, as the book has been through significant review process. However, it may be that this detail is overlooked because the basic MC Control scenarios are not as interesting as all the extensions and combinations with TD, where you will find the book does use a learning rate $\alpha$ - including when making comparison charts of MC vs TD approaches etc. The explicit list of returns and averaging over them for control only occurs in a couple of places in the text in this pseudocode, and is not mentioned again.


Measuring Object size using Deep Neural Network I have a large dataset of vehicles with the ground truth of their lengths (Over 100k samples). Is it possible to train a deep network to measure/estimate vehicle length ? I haven't seen any papers related to estimating object size using deep neural network.

Yes! This most certainly can be done. Since you have a labeled dataset, that makes it all the more simple!

I would take a look at this project and that should get you where you need to go.

The implementation details should be pretty straightforward. Let me know if I can help further.


Are neural networks invertible? I am interested in learning about the inverse of neural networks and I would like to understand about the invertibility of neural networks, as for example described in On the Invertibility of Invertible Neural Networks.
Researchers who are working on this domain, can you help me understand these two questions.

Are all neural network invertible ?
What exactly qualifies a neural network to be invertible ?

The meaning of invertible here is the standard definition of invertibility for a mathematical function $f \colon X \to Y$. Invertible simply means "the function has an inverse map $f^{-1} \colon Y \to X$". Equivalently the function $f$ is bijective, which means the following two conditions hold:

$f$ is injective: for any two distinct $x_1, x_2 \in X$, $f(x_1) \ne f(x_2)$.

$f$ is surjective: for any $y \in Y$, there exists an $x \in X$ such that $f(x) = y$.

If this is unfamiliar, you should be able to find some helpful references on Google using these terms.
Most obvious neural network architectures cannot possibly be invertible. Consider for example a classifier which takes an image or some other high-dimensional input, and outputs a classification label. This network could only be invertible if there was only one possible input1 which corresponds to each label, which is not the goal of the network.

1 I mean this rather literally: if one pixel is even slightly different, this would be a different input to the network and would have to have a different output.


What makes a transformer a transformer? Transformers are modified heavily in recent research. But what exactly makes a transformer a transformer? What is the core part of a transformer? Is it the self-attention or the parallelism or something else?

There is not one answer to this question, but one could argue that transformers heavily rely on

transforming each input into latent subspaces of queries, keys and values in order to generate attention score
a pool of transformations of the attention vectors (multi-head) according to which models can capture richer interpretations as different sections of the input embedding can attend different per-head subspaces that link back to each input


When using neural networks to detect features in an image, how can locate that specific feature in the original image? I understand how a neural network can be trained to recognise certain features in an image (faces, cars, ...), where the inputs are the image's pixels, and the output is a set of boolean values indicating which objects were recognised in the image and which weren't.

What I don't really get is, when using this approach to detect features and we detect a face for example, how we can go back to the original image and determine the location or boundaries of the detected face. How is this achieved? Can this be achieved based on the recognition algorithm, or is a separate algorithm used to locate the face? That seems unlikely since to find the face again, it needs to be recognised in the image, which was the reason of using a NN in the first place.

This problem is called object detection.

If you have a trainings set of images with boxed objects you can just train a neural network to directly predict the box. I.e. the output has the same dimension as the input and the NN learns to assign each pixel the probability of belonging to a certain object.

If you don't have such a convenient dataset you could just recursively narrow the location down by feeding parts of the image to the network until you find the smallest part that still fully activates a certain classification.

In this paper they try a mixture of these two approaches.


Why is there more than one way of calculating the accuracy? Some sources consider the true negatives (TN) when computing the accuracy, while some don't.

Source 1:
https://medium.com/greyatom/performance-metrics-for-classification-problems-in-machine-learning-part-i-b085d432082b

Source 2:https://www.researchgate.net/profile/Mohammad_Sorower/publication/266888594_A_Literature_Survey_on_Algorithms_for_Multi-label_Learning/links/58d1864392851cf4f8f4b72a/A-Literature-Survey-on-Algorithms-for-Multi-label-Learning.pdf

which can be translated as

which one of these must be considered for my multi-label model.

In machine learning, the accuracy is usually defined as the number of correct predictions divided by the total number of predictions. The correct predictions are the true positives ($\mathrm {TP}$) and true negatives ($\mathrm {TN}$), so the usual formula to calculate the accuracy is the following one (your first one).

\begin{align}
\text{Accuracy}=\frac {\mathrm {TP} + \mathrm {TN}}{\mathrm {TP} + \mathrm {TN} + \mathrm {FP} + \mathrm {FN}} \label{0}\tag{0}
\end{align}

The following formula corresponds to the threat score ($\mathrm{TS}$) or critical success index ($\mathrm {CSI}$).

\begin{align}
{\displaystyle \mathrm {TS} ={\frac {\mathrm {TP} }{\mathrm {TP} +\mathrm {FN} +\mathrm {FP} }}} \label{1} \tag{1}
\end{align}

In section 7 of the paper A Literature Survey on Algorithms for Multi-label Learning, it is written

  In traditional classification such as multi-class problems, accuracy is the most common evaluation criteria. Additionally, there exists a set of standard evaluation metrics that includes precision, recall, F-measure, and ROC area defined for single-label multi-class classification problems. However, in multi-label classification, predictions for an instance is a set of labels and, therefore, the prediction can be fully correct, partially correct (with different levels of correctness) or fully incorrect. None of these existing evaluation metrics capture such notion in their original form.

Then they say

  Depending on the target problem, evaluation measures for multi-label data can be grouped into at least three groups: evaluating partitions, evaluating ranking and using label hierarchy. The first one evaluates the quality of the classification into classes, the second one evaluates if the classes
  are ranked in order relevance and the third one evaluates how effectively the learning system is able to take into account an existing hierarchical structure of the labels.

In section 7.1, the authors state

  To capture the notion of partially correct, one strategy is to evaluate the average difference between the predicted labels and the actual labels for each test example, and then average over all examples in the test set. This approach is called example-based evaluations.

To capture the notion of partially incorrect in multi-label classification problems, they then use the following definition of accuracy (proposed in section 5.2 of the paper Discriminative Methods for Multi-labeled Classification)

\begin{align}
\frac{1}{n} \sum_{i=1}^{n} \frac{|Y_i \cap Z_i|}{|Y_i \cup Z_i|} \label{2} \tag{2}
\end{align}

where $Y_i$ is the true set of labels and $Z_i$ the predicted set of labels for the single instance (or observation) $i$, so $\frac{|Y_i \cap Z_i|}{|Y_i \cup Z_i|}$ is the accuracy for the instance $i$. If $|Y_i|=1$ and $|Z_i|=1$, then we have a single-label classification problem.

The metric \ref{2} does not correspond to the metric \ref{1}. In \ref{2}, the accuracy is calculated for each example (or instance) and then we average all these accuracies, because, for each example, you may have that one predicted label is correct, two predicted labels are correct, and so on (also depending on the number of labels you need to predict for each of the examples). In \ref{1}, there is no notion of multiple labels.


What is the reward system of reinforcement learning? Can you describe this reward system in more detail?
I understand that the environment sends a signal indicating whether or not the action taken by the agent was 'good' or not, but it seems too simple.
Basically, can you detail the nitty-gritty workings of this system? I dunno, I may just be overthinking things.

In this case, the word "system" refers to a Markov decision process (MDP), which is the mathematical model used to represent the reinforcement learning (RL) problem or, in general, a decision making problem. Recall that, in RL, the problem consists in finding an (optimal) policy, which is a policy that allows the agent to collect the highest amount of reward (in the long run). Hence, in RL, the MDP is the problem and the optimal policy (for that MDP) is the solution.

The MDP is composed of the set of states of the environment $S$, the set of possible actions that the RL agent can take $A$, a transition function, $P(s_{t+1}=s'\mid s_{t}=s, a_t = a)$, which is a probability distribution and describes the dynamics of the environment, and a reward function $R_a(s', s)$, which is a function that describes the "reward system" of the environment. $R_a(s', s)$ can be thought of as the reward (signal) that the agent receives after having taken action $a$ in state $s$ and having landed in state $s'$.

In other words, the reward system is just this function $R_a(s', s)$, which is often the hardest function to define when modelling an RL problem. However, in certain cases, this function can be easily defined. For example, in the case of chess, you could define $R_a(s', s) = 1$, if $s'$ is the terminal state (checkmate), else $R_a(s', s) = 0$. Nonetheless, we could define the reward function for a chess environment differently. For example, we could give a reward of $0.5$ for certain "clever" moves. So, the reward function needs to be defined (by the programmer) in order to solve an RL problem and it highly affects the way the agent will learn.

The reward function can also be denoted by $R_a(s)$, which can be thought of as the reward that the agent receives after having taken action $a$ in state $s$ (no matter which state it lands in), or $R(s)$, which can be thought of as the reward the agent receives either when it enters or exits state $s$.


In AlphaZero, do we need to store the data of terminal states? I have a question about the training data used during the update/back-propagation step of the neural network in AlphaZero.
From the paper:

The data for each time-step $t$ is stored as ($s_t, \pi_t, z_t$) where $z_t = \pm r_T$ is the game winner from the perspective of the current player at step $t$. In parallel (Figure 1b), new network parameters $\Theta_i$ are trained from data ($s,\pi, z$) sampled uniformly among all time-steps of the last iteration(s) of self-play

Regarding the policy at time $t$ ($\pi_t$), I understood this as the probability distribution of taking some action that is proportional to the visit count to each child node, i.e. during MCTS, given some parent node (state) at time $t$, if some child node (subsequent state) $a$ is visited $N_a$ times and all children nodes are visited $\sum_b N_b$ times, then the probability of $a$ (and its corresponding move) being sampled is $\frac{N_a}{\sum_b N_b}$, and this parametrizes the distribution $\pi_t$. Is this correct? If this is the case, then for some terminal state $T$, we can't parametrize a distribution because we have no children nodes (states) to visit. Does that mean we don't add ($s_T, \pi_T, z_T$) to the training data?
Also, a followup question regarding the loss function:

$l = (z-v)^2 - \pi^T log\textbf{p} + c||\Theta||^2$

I'm confused about this $\pi^T$ notation. My best guess is that this is a vector of actions sampled from all policies in the $N$ X $(s_t, \pi_t, z_t)$ minibatch, but I'm not sure. (PS the $T$ used in $\pi^T$ is different from the $T$ used to denote a terminal state if you look at the paper. Sorry for the confusion, I don't know how to write two different looking T's)

I'm not 100% sure whether or not they added any data for terminal game states, but it's very reasonable to indeed make the choice not to include data for terminal game states. As you rightly pointed out, we don't have any meaningful targets to update the policy head towards in those cases, and this is not really a problem because we would also never actually make use of the policy output in a terminal game state. For the value head we could provide meaningful targets to update towards, but again we would never actually have to make use of such outputs; if we encounter a terminal game state in a tree search, we just back up the true value of that terminal game state instead of making a call to the network to obtain a value function approximation.
In theory, I could imagine some cases where training the value head on terminal game states might be slightly beneficial despite not being strictly necessary; it could enable generalisation to similar game states that are not terminal (but close to being terminal), and speed up learning for those. For example, if you have a game where the goal is to complete a line of $5$ pieces, training the value head on terminal states where you actually have a line of $5$ pieces and have entirely won the game might generalise and speed up learning for similar game states where you may not yet have $5$ pieces in a line, but are very close to that goal. That said, intuitively I really don't feel like this would provide a big benefit (if any), and we could probably also come up with cases where it would be harmful.

In the $\pi^{\text{T}}$ notation, $\pi$ is a vector (for any arbitrary time step, the time step is not specified here) containing a discrete probability distribution over actions (visit counts of MCTS, normalised into a probability distribution), and the $\text{T}$ simply denotes that we take the transpose of that vector. Personally I don't like the notation though, I prefer something like $\pi^{\top}$ which is more clearly distinct from a letter $T$ or $\text{T}$.
Anyway, once you understand that to denote the transpose, you'll see that $\pi^{\top}\log(\mathbf{p})$ is a dot product between two vectors, which then ends up being a single scalar.


How does A* search work given there are multiple goal states? When I have read through the fundamentals of AI, I saw a situation (i.e., a search space) which is illustrated in the following picture.

These are the heuristic estimates:

h(B)=9
h(D)=10
h(A)=2
h(C)=1

If we use the A* algorithm, the node $B$ will be expanded first because $f(B)=1+9=10$, while node $A$ having $f(A)=9+2=11$ and $f(B)<f(A)$, right?

After that, the search tree will go on in the order R -> B -> D -> G2. Will the search go on to also find the goal state G1?

Kindly let me know the order of the search if I am wrong.

Yes. If you leave A* running (i.e. do not impose a goal condition on a newly-encountered state), all states will be explored, just as they would be in breadth- or depth- first search.


Using word embedding to extend words for searching POI names I am developing my own mobile app related to digital map. One of the functions is searching POIs (points of interest) in the map according to relevance between user query and POI name.

Besides the POIs whose names contain exact words in the query, the app also needs to return those whose names are semantically related. For example, searching 'flower' should return POI names that contain 'flower' as well as those that contain 'florist'. Likewise, searching 'animal' should return 'animal' as well as 'veterinary'.

That said, I need to extend words in the query semantically. For example, 'flower' has to be extended to ['flower', 'florist']. I have tried to use word embeddings: using the words corresponding to most similar vectors as extensions. Due to the fact I don't have user review data right now and most of the POI names are very short, I used trained word2vec model published by Google. But the results turn out to be not what I expect: most similar words of 'flower' given by word2vec are words like 'roses'and 'orchid', and 'florist' is not even in the top 100 most similar list. Likewise, 'animal' gives 'dog', 'pets', 'cats' etc. Not very useful for my use case.

I think simply using word embedding similarity may not be enough. I may need to build some advanced model based on word embedding. Do you have any suggestions?

I think word embeddings are overkill in this particular case.

My suggestion would be to go for a simple dictionary based approach: compose sets of semantically related words, and then use those to expand your query terms. This might take a bit longer to set up, but has several advantages:

simplicity: you can't make many mistakes with this
transparency: you know exactly why a certain term matches, and another one doesn't
accuracy: you have tight control over the whole process; if some term is wrong, you remove it from the set. You cannot do that with embeddings
resources: a dictionary-based approach is far simpler and needs less storage

'Old tech' doesn't sound as sexy as the latest deep learning stuff, but unless you want to have this as a toy project to learn about how to do things with embeddings I would say the latter are the wrong tool for the job. At least you can be sure that it works, and if it doesn't you can easily fix it.


Training Neural Network with 'fake' data? I have a neural network with 2 inputs and one output, like so:

input    | output
____________________
 a    | b   |  c
 5.15 |3.17 | 0.0607
 4.61 |2.91 | 0.1551

etc..

I have 75 samples and I am using 50 for training and 25 for testing.

However, I feel that the training samples are not enough. Because I can't provide more real samples (due to time limitation), I would like to train the network using fake data:

For example, I know that the range for the a parameter is from 3 to 14, and that the b parameter is ~65% of the a parameter. I also know that c is a number between 0 and 1 and that it increases when a & b increase.

So, what I would like to do is to generate some data using the above restrictions (about 20 samples). For example, assume a = 13 , b = 8 and c= 0.95, and train the network with these samples before training it with the real samples.

Has anybody studied the effect of doing this on the neural network? Is it possible to know if the effect will be better or worse on the networks? and are there any recommendations/guidelines if I want to do this?

Yes, this is not advisable. If you train your model with random data your model is not learning anything useful, because there is no information to gain from those examples. Even worse it may (and likely is) trying to generalize off of your incorrect examples, which will lessen the effect your real examples have. Essentially, you are just dampening your training set with noise.

You are moving in the right direction though. 75 examples will not be enough if your problem has any complexity at all. And unless, you know some correlation between the inputs a,b and the output c you don't want to generate data (and even if you did know some correlation, it is not always suggested to generate data). If it is impossible to get any more data you might want to consider a statistical model, rather than a neural network.


What is graph clustering? There are several (family of) algorithms that can be used to cluster a set of $d$-dimensional points: for example, k-means, k-medoids, hierarchical clustering (agglomerative or divisive).

What is graph-based clustering? Are we clustering the nodes or edges of a graph instead of a set of $d$-dimensional (as e.g. in k-means)? Couldn't we just use k-means to also cluster a set of nodes?

In graph clustering, we want to cluster the nodes of a given graph, such that nodes in the same cluster are highly connected (by edges) and nodes in different clusters are poorly or not connected at all.

A simple (hierarchical and divisive) algorithm to perform clustering on a graph is based on first finding the minimum spanning tree of the graph (using e.g. Kruskal's algorithm), $T$. It then proceeds in iterations. At each iteration, we remove from $T$ the edge with the highest weight. Given that $T$ is a tree, the removal of an edge from $T$ will create a forest (with connected components). So, after the removal of the edge of highest weight from $T$, we will have two connected components. These two connected components will represent two clusters. So, after one iteration, we will have two clusters. At the next iteration, we remove the edge with the second highest weight, and this will create other connected components, and so on, until, possibly, all nodes are in their own cluster (that is, all edges have been removed from $T$).

There are several limitations of this algorithm. For example, it only considers the edges of the initial graph that are shared with $T$ (that is, it only considers the edges of the minimum spanning tree). It also requires the edges of the graph to be weighted. It does not require the number of clusters to be known in advance (like any other hierarchical clustering algorithm), but we still need to choose the optimal number of clusters (after the algorithm has terminated). We can do that in several ways. A way to do it would be to have a threshold value $t$ that is used to decide when we should stop removing edges from $T$: more specifically, we will keep removing edges from $T$ until the next highest weight is higher than this threshold $t$.

There are numerous applications of this type of clustering. For example, we might want to discover certain groups of people in social networks.

The paper Graph clustering (by Satu Elisa Schaeffer, 2007) provides a readable and quite detailed overview of this field.

There are algorithms based on k-means that can also work on graphs. See e.g. Graph-based k-means Clustering: A Comparison of the Set Median versus the Generalized Median Graph (by Ferrer et al.).


Is the Q value the same as the state-action pair value? Am I right to say that the Q value of a particular state and action is the same as the state-action pair value of that same state and action?

I don't understand your question very clearly.

Q-value of a particular state-action pair (s,a) under policy $\pi$ is the total reward you would expect to collect if you start from the state s, take the action a, and follow policy $\pi$ from then on.

In the literature, this is referred to as state-action values.


Is it useful to eliminate the less relevant filters from a trained CNN? Imagine I have a tensorflow CNN model with good accuracy but maybe too many filters:

Is there a way to determine which filters have more impact in output? I think it should be possible. At least, if a filter A has a 0, that only multiples the output of a filter B, then filter B is not related to filter A. In particular, I'm thinking in 2d data where 1 dimension is time-related and the other feature related (like one-hot char).
Is there a way to eliminate the less relevant filters from a trained model, and leave the rest of the model intact?
Is it useful or there are better methods?

NOTE: All the observations and results are from the paper The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks.

To answer your questions one by one:

Yes there are ways to determine which filters have more impact on the output. Its a very naive way but works very good in practice. Filters with small weights impact output less (according to empirical evidence), which basically means neurons whose weights lie in the switching region i.e ~$0$ in ReLu and ~$-1$ to $1$ (say) have less impact on final output.
Yes, just eliminating these lower weight filters eliminate the unnecessary noise and indecisiveness introduced by these filters and suprisingly makes the model perform better (observed empirically).
The concept a relatively old paradigm but has been a new twist by the simplicity of the method of elimination of unnecessary weights in the aforementioned paper and thus winning it the best paper award in ICLR 2019.

TL;DR: Eliminating unnecessary weights makes the model perform better than the original model.

Also here is the TensorFlow code.


Is there bidirection sequence-to-sequence neural machine translation? I have heard about bidirectional RNN LSTM units (endcoders-decoders), but my question is - is there bidirectional neural machine translation, that uses A->B weights for the translation in the opposite direction B->A? If not, then what are the obstacles to such system?

Bidirectional Mapping of Arbitrary Functions

Interchange between what properties are considered labels and what properties are considered features can support bidirectional map learning.

Unidirectional training represents the objective of functional approximation $f_p$ in terms of features $\mathcal{X}$ and labels $\mathcal{Y}$ through convergence of parameters $p$, ideally toward an accurate and reliable approximation.

$$ \mathcal{Y} \Leftarrow f_p(\mathcal{X}) $$

Because only a small proportion of arbitrary functions are monotonic (have only one value in the domain for any given value in the range), the existence of an inversion of $f_p$ is unlikely. This is certainly true of natural language processing functions. This fact will be further illuminated below.

Bidirectional training of a non-monotonic translation function $f_p$ and a reverse translation $g_p$ can be represented as two adjacent training objectives.

$$ \mathcal{Y} \Leftarrow f_p(\mathcal{X}) $$
 $$ g_q(\mathcal{Y}) \Rightarrow \mathcal{X} $$

Function $g_q$ is not a guaranteed inversion of $f_p$ and may rarely be. Parameters resulting from convergence $q$ may not be derivable from parameters resulting from convergence $p$ or vice versa either.

Temporal Bidirectionality to Minimize Round Trip Inaccuracy

For reasons further illuminated below, temporal bidirectionality may be important in some use cases. For this reason, use two B-LSTM networks.

There may be some efficiency gains by training both B-LSTM networks in the same training phase, especially if more than one host, GPU, parallel signal processor, or AI accelerator is available at training time. The same is true of test and validation.

The ontology of network types is as follows.

RNNs (recurrent artificial networks) are a stateful extension of MLPs (multi-layer perceptrons).
LSTM (long short term memory) networks are an extension of RNNs, and use gating within each cell to adaptively regulate retention of state and its dissipation.
B-LSTMs provide bidirectionality in the temporal direction, which allows more variety in what causality and reactance can be modeled.

Importance of Contemporary Concepts in Linguistics

Since the natural-language-processing tag was included with the posting of the question and the term, "Machine translation," was mentioned, it appears that bidirectional natural language translation using AI is the project goal. If that is the case, a contemporary linguist's understanding of words sequences in text and the sequence of phonetic elements in speech will help with developing an effective solution using AI. An engineer should be aware of the interplay between expectations placed on project results and technical feasibility at the current state of NLP technology.

Human translation experts rely on cognition to accurately represent text that may contain cultural references, analogies, colloquialisms, and other literary or oratory devices.
Because of rhyme, alliteration, cadence and other phonetic devices, prose and poetry may not have an equivalent in the target language.
Linguistic elements are not strictly words. A single word may have multiple linguistic elements because of tenses, plurality, gender, prefixes, suffixes, and other modifiers. For example, "Multicolored," is an adjective made up of three linguistic elements: {"multi-", "color", "-ed"}. A single idea is sometimes represented as multiple words, the individual meanings and juxtaposition of which does not imply the meaning of the word group. For example, "I don't wish to hitch my apple-cart to that wobbly wagon," has nothing to do with apples, carts, mechanical stability, or wagons.
The translation $\mathcal{A} \Rightarrow \mathcal{B} \Rightarrow \mathcal{C}$ may not lead to $\mathcal{A} = \mathcal{C}$.
When a series of linguistic elements are communicated, the semantics represented are not often a series. For instance, "Flying birds can also jump," is equivalent to, "If a bird is able to fly, it can jump."  This works in natural language because the associations and modifying aspects between linguistic elements is a semantic network that correlates with a cognitive ontology. The serialization of this network and its reconstitution in another mind cannot be formalized in a way that eliminates all ambiguity. Communication $\mathcal{K}_o \Rightarrow \mathcal{L} \Rightarrow \mathcal{K}_r$, where $\mathcal{K}$ are cognitions at originating intelligence $o$ and target intelligence $t$ respectively and $\mathcal{L}$ is a language expression in between, may not lead to $\mathcal{K}_t = \mathcal{K}_o$.
For the above reasons, formal grammar has been discarded as a set of principles that can facilitate NLP. Humans neither write nor speak in conformance with any formal grammar on a regular basis.

Expected Dominance of Recursive Network Technology

For this reason, LSTM networks may not be the direction of NLP technology development. Recursive network designs (often using RNN as an acronym but referring to a distinct subset of recurrent networks) can train semantic network models based on serial streams of linguistic elements. That appears to bear closest resemblance to language centers in the human mind, and may emerge as a dominant NLP component in emerging AI systems.


Continuous state and continuous action Markov decision process time complexity estimate: backward induction VS policy gradient method (RL) Model Description: Model based(assume known of the entire model) Markov decision process.
Time($t$): Finite horizon discrete time with discounting factor
State($x_t$): Continuous multi-dimensional state
Action($a_t$): Continuous multi-dimensional action (deterministic action)
Feasible(possible action) ($A_t$): possible action is a continuous multi-dimensional space, no discretization for the action space.
Transition kernel($P$): known and have some randomness associated with the current state and next state
Reward function: known in explicit form and terminal reward is know.
The method I tried to solve the model:

Discretize the state space and construct multi-dimensional grid for state space, starting from the terminal state, I used backward induction to reconstruct value function from the previous period. By using the Bellman equation, I need to solve an optimization problem, selecting the best action that gives me the best objective function.

$$V_t(x_t) = max_{a_t \in A_t}[R_t(x_t,a_t) + E[\tilde{V}_{t+1}(x_{t+1})|x_t, a_t]]$$
where $\tilde{V}_{t+1}$ here is an approximation using interpolation method, since the discrete values are calculated from the previous time episode. In other words: $\tilde{V}_{t+1}$ is approximated by some discrete value: $$V_{t+1}(x_0),V_{t+1}(x_1),V_{t+1}(x_2)\cdots V_{t+1}(x_N)$$ where $x_0,x_1,x_2\cdots x_N$ are grid points from discretizing the state space.
In this way, for every time steps t, I could have a value function for every grid point and my value function could be approximated by using some interpolation method(probably cubic spline). But here are some of the problems: 1. what kind of interpolation is suitable for high dimensional data. 2. Say we have five dimension for the state, then I discretize the state by giving 5 grid points to every dimension, then there are 5^5 = 3125 discrete state values I need to calculate through the optimization.(Curse of the dimensionality). 3. What kind of optimizer should I use? Since I do not know the shape of the objective function, I do not know if it is a smooth function and I do not know if the function is concave. So I may have to use a robust optimizer, probably some evolutionary optimizer. So eventually I end up with this computational complexity and it takes too long for the computation.
And recently I learned the techniques of policy gradient from OpenAI: https://spinningup.openai.com/en/latest/spinningup/rl_intro3.html
This method avoid using this backward induction and approximating the value function by using interpolation. And obtained the approximated policy by firstly guessing a functional form of the policy and take approximated gradient of the policy function by using sampling(simulation) method. Since the model is known, every time it could sample new trajectories and use that to update the policy by using the stochastic gradient assent method. In this way updating the policy until it gets some sort of convergency.
I am wondering if this type of technique could potentially reduce the computational complexity significantly. Any advice helps, thanks a lot.

Discretizing state/action space will always be a very expensive strategy, in fact I don't think you can do better than exponential time in state/action dimension that way.
Now, you still haven't really explained your algorithm with the discretized states and actions. Given your known models, it sounds like you're planning on doing dynamic programming. This would definitely be an exponential time algorithm. You also haven't explained how you'd use your knowledge of the model in your policy gradient implementation. If you're just using it to sample some trajectories to have more data for updates, policy gradient methods will be much more efficient (poly-time). Otherwise, if you're planning on using your model for some sort of MCTS planning, depending on how you implement that, it could be very inefficient as well.


Q-learning problem wrong policy I am coding out a simple 4x4 grid game whereby the agent starts at a particular state and his aim is to reach the terminal state. The agent is supposed to avoid traps along the way and reach the end goal with high reward. The below picture illustrates the environment.

The code that I am running is shown below:

# 4x4 Grid
import random

gamma = 1
grid = [[-0.1 for i in range(4)] for j in range(4)]
episodes = 500000
epsilon = 1 # start greedy
decay = 0.999
min_epsilon = 0.1
alpha = 0.65
# set terminal states
grid[1][0] = -1
grid[2][2] = -1
grid[0][3] = 1

# Set up Q tables
# 0: up, 1: down, 2: left, 3: right
# Q = {(0,0): {0: z, 1: x, 2: c, 3: v}, ... }}
Q = {}

# 4 rows
for row in range(4):
    # 4 columns
    for column in range(4):
        Q[(row,column)] = {}
        # 4 actions
        for k in range(4):
            Q[(row,column)][k] = 0

def isTerminal(state):
    if state == (1,0) or state == (2,2) or state == (0,3):
        return True
    return False

def get_next_state_reward(state, action):

    row = state[0]
    col = state[1]
    #print(row, col)
    if action == 0: # up
        # out of grid
        if (row - 1) < 0:
            return (state, grid[row][col])

    if action == 1: # down

        # out of grid
        if (row + 1) > len(grid) - 1:
            return (state, grid[row][col])

    if action == 2: # left

        if (col - 1 < 0):
            return (state, grid[row][col])

    if action == 3: # right

        if (col + 1 > len(grid[row]) - 1):
            return (state, grid[row][col])

    if action == 0:

        row -= 1
        return ((row,col), grid[row][col])

    if action == 1:

        row += 1
        return ((row,col), grid[row][col])

    if action == 2:

        col -= 1
        return ((row,col), grid[row][col])

    if action == 3:

        col += 1
        return ((row,col), grid[row][col])

state_visit = {}
for row in range(4):
    # 4 columns
    for column in range(4):
        state_visit[(row,column)] = 0

for episode in range(episodes):

    # let agent start at start state
    state = (3,0)

    while not isTerminal(state):

        r = random.uniform(0,1)

        if r < epsilon:
            action = random.randint(0,3)
        else:
            action = max(Q[state], key=lambda key: Q[state][key])

        next_state, reward = get_next_state_reward(state, action)

        TD_error = reward + gamma * max(Q[next_state]) - Q[state][action]

        Q[state][action] = Q[state][action] + alpha * TD_error

        state = next_state

        state_visit[next_state] += 1
        epsilon = max(min_epsilon, epsilon*decay)

        #input()

policy = {}
# get optimal policies for each state
for states in Q:
    policy[states] = max(Q[states], key=lambda key: Q[states][key])

When I finish running the algorithm however, I am unable to achieve the optimal policy no matter how many tweaks I do to the number of episodes, or epsilon decay, or the alpha value.

Particularly, the Q values that I attain for state (2,0), (0,1) and (0,0) have Q values that are equal values for three directions except for the last direction which brings the agent to the terminal state.

For example, these are the Q-values that I get for state (0,0), (0,1) and (2,0) respectively.

(0,0): {0: 2.0, 1: 2.9, 2: 2.9, 3: 2.9}

(0,1): {0: 2.9, 1: 2.0, 2: 2.9, 3: 2.9}

(2,0): {0: 2.9, 1: 2.9, 2: 2.9, 3: 2.9}

I am not sure why the Q-values for the 3 directions should be the same because each extra step that the agent takes incurs a negative reward.

Would anyone be able to help ? Thank you so much !

You have a simple mistake in your TD Error function:

TD_error = reward + gamma * max(Q[next_state]) - Q[state][action]

You have made Q[next_state] a Python dict, so this will take the maximum key which is 3 for all your Q table entries. This is why you end up with values very close to 3 at the end, which is impossible for your problem, the maximum return will be 1.0 when stepping from adjacent grid points to the positive terminal state.

The correct code is:

TD_error = reward + gamma * max(Q[next_state].values()) - Q[state][action]

Alternatively, since your actions are all numeric, you could use a list instead of a dict in your Q table.


How can I be sure that the final model, trained on all data, is correct? The 'by the book' method of delivering final machine learning models is to include all data in the final training (including validation and test sets). To check robustness of my model I use randomly chosen population for training and validation sets with each training (no set random seed). The results on validation and then test sets are pretty satisfactory for my case however they are always different each time, precision spans between 0.7 and 0.9. This is due to fact that each time different data points fall to set with which model is trained.
My question is: how do I know that final training will also generate good model and how to estimate its precision when I do not have anymore unseen data?

The purpose of the test set is to test your model before deploying, otherwise, you would not need the test set in the first place. If you retrain your model by also including the validation and test datasets, of course, you cannot test your model anymore. You need to leave the test dataset separate and not use it for retraining, unless you have more data for testing.


Is increasing software complexity the most likely bottleneck to the AI singularity? From Wikipedia:

According to the most popular version of the singularity hypothesis, called intelligence explosion, an upgradable intelligent agent will eventually enter a "runaway reaction" of self-improvement cycles, each new and more intelligent generation appearing more and more rapidly, causing an "explosion" in intelligence and resulting in a powerful superintelligence that qualitatively far surpasses all human intelligence.

But what if the complexity of the problem of self-improving the software grows at a faster rate than the AGI intelligence self-improvement?
From experience we know that problems tend to be harder to solve at every iteration, with diminishing returns. Take as an example the theory of gravitation. Newtonian physics is relatively easy to formulate and covers the majority of high level gravitation phenomenas.
A more refined picture, like General Relativity, fill few holes in the theory with a huge increase in complexity. To describe black holes and primordial cosmology we need a theory of quantum gravity, which appears to require a further step in complexity.
What "saved" us so far is the economic growth of our civilisation, which allowed more and more scientist to focus on solving the next problems. It's true that the first AGI will have the luxury of being duplicated, being a software base intelligence, but at the same time is likely that the first AGI will be extremely compute intensive. But even assuming that we (or maybe it would better to say, they) have the hardware to run $10^{2}$ instances, if the complexity of every substantial self-improvement grows say by $10^{d}$x with $d=3$ while the improvement to the intelligence is only $10^{l}$x with $l=1$, the self-improvement cycle will quickly slow down.
So is increasing software complexity the most likely bottleneck to the AI singularity? And what are likely values for $d$ and $l$?

What if the complexity of the problem of self-improving the software grows at a faster rate than the AGI intelligence self-improvement?

If this turns out to be the case, then the complexity would indeed be the bottleneck.
There's a formal term for this: kolmogrov complexity or "k-complexity".  (The best formal definition of intelligence I've come across is Hutter & Legg's, which is broken down in the article "On the definition of intelligence" by our own nbro.  It will shed shed some light on the formal structure of the question at hand.

Any attempt to provide values to your d & l would speculative, since we're still so far from AGI. The question would be what is the effect of any given (d,l)

The subject can definitely be treated formally in a mathematical sense the and values analyzed.  (There may even been some formulae out there, although, they are likely to be more complex, per the Hutter&Legg.)
Sorry I can't give you a more formal answer--it's a good question!

Notes:
Another bottleneck would be the speed of light, essentially, "bits and bytes across a pipeline".  This would be a factor in both self-replication and networked processing and memory.  You can expand bandwidth/throughput, but not infinitely.
This relates to space complexity (how much volume/memory does the algorithm require?) in addition to the k-complexity, which itself is likely to be non-trivial for anything approaching AGI.
AI Threat:
From the standpoint of neoluddism, I'm skeptical that "infinitely" self-optimizing superintelligence is the greatest threat.  Evolution, and economic theory, suggest that the greater danger would come from not what is most expensive, complex, & capable, but that which is "just good enough", cheaper and more easily replicable.  This is because decision time becomes a critical factor in a competitive environment, as does replication time.
I always bring up grey goo because it only has to be good at 1 thing to eat everything else!


Is artificial intelligence vulnerable to hacking? The paper The Limitations of Deep Learning in Adversarial Settings explores how neural networks might be corrupted by an attacker who can manipulate the data set that the neural network trains with. The authors experiment with a neural network meant to read handwritten digits, undermining its reading ability by distorting the samples of handwritten digits that the neural network is trained with.

I'm concerned that malicious actors might try hacking AI. For example

Fooling autonomous vehicles to misinterpret stop signs vs. speed limit.
Bypassing facial recognition, such as the ones for ATM.
Bypassing spam filters.
Fooling sentiment analysis of movie reviews, hotels, etc.
Bypassing anomaly detection engines.
Faking voice commands.
Misclassifying machine learning based-medical predictions.

What adversarial effect could disrupt the world? How we can prevent it?

AI is vulnerable from two security perspectives the way I see it:

The classic method of exploiting outright programmatic errors to achieve some sort of code execution on the machine that is running the AI or to extract data.
Trickery through the equivalent of AI optical illusions for the particular form of data that the system is designed to deal with.

The first has to be mitigated in the same way as any other software. I'm uncertain if AI is any more vulnerable on this front than other software, I'd be inclined to think that the complexity maybe slightly heightens the risk.

The second is probably best mitigated by both the careful refinement of the system as noted in some of the other answers, but also by making the system more context-sensitive; many adversarial techniques rely on the input being assessed in a vacuum.


Why exactly do neural networks require i.i.d. data? In reinforcement learning, successive states (actions and rewards) can be correlated. An experience replay buffer was used, in the DQN architecture, to avoid training the neural network (NN), which represents the $Q$ function, with correlated (or non-independent) data. In statistics, the i.i.d. (independently and identically distributed) assumption is often made. See e.g. this question. This is another related question. In the case of humans, if consecutive data points are correlated, we may learn slowly (because the differences between those consecutive data points are not sufficient to infer more about the associated distribution).
Mathematically, why exactly do (feed-forward) neural networks (or multi-layer perceptrons) require i.i.d. data (when being trained)? Is this only because we use back-propagation to train NNs? If yes, why would back-propagation require i.i.d. data? Or is actually the optimisation algorithm (like gradient-descent) which requires i.i.d. data? Back-propagation is just the algorithm used to compute the gradients (which is e.g. used by GD to update the weights), so I think that back-propagation isn't really the problem.
When using recurrent neural networks (RNNs), we apparently do not make this assumption, given that we expect consecutive data points to be highly correlated. So, why do feed-forward NNs required the i.i.d. assumption but not RNNs?
I'm looking for a rigorous answer (ideally, a proof) and not just the intuition behind it. If there is a paper that answers this question, you can simply link us to it.

There is an assumption behind the theory training a neural network, that also applies to many other supervised learning methods, that a training sample is representative of the data set as a whole - that it has been sampled fairly from the population that the learning algorithm has been set up to approximate.
The term i.i.d. stands for "independent and identically distributed". If you pick trajectories from RL then the sampling (per individual* record as used to train with) is not independent. Even if you choose the start of the trajectory randomly, you will have made one random choice then your remaining choices are made according to the trajectory - technically they are chosen by the policy and environment dynamics over a single step, which is usually not enough to make the second, third etc steps from a trajectory fully independent random samples from the training population. To make the sampled observations independent, you should make a new random choice, and to be identically distributed that choice has to be made fairly over the whole dataset.
If a subset of samples drawn for a mini-batch (or as consecutive individual samples) are correlated for an online algorithm like gradient descent, it causes a problem. For the algorithm to converge towards a globally optimal solution, it needs errors and the gradients they generate to be unbiased samples of the "true" function gradients across the loss function. Whilst correlated data from a trajectory does not do this at all - it exhibits a strong sampling bias, causing weight updates to move consistently in the wrong direction.
You can demonstrate this effect trying to learn any simple function like $y = x^2$ with $x$ from $-1$ to $+1$. Training a NN with randomly selected $x,y$ pairs results in far better results than training it serially with ~2000 records starting $(-1.000, 1.000) (-0.999, 0.998) (-0.998, 0.996)$ etc

So, why do feed-forward NNs required the i.i.d. assumption but not RNNs?

RNNs do need i.i.d. data, but the "unit" of sampling here is each sequence.

* By "individually" I mean in terms of how they are used in the training process. A "unit" here is the smallest set of records that leads to a complete measure of loss in your optimisation.
It is ok to draw longer trajectories at random if your smallest unit of measurable error is from a trajectory, as in Monte Carlo or TD($\lambda$) for RL, or for RNNs. There can be subtle problems with this - the trajectories still have to be i.i.d. as in they represent fair samples from the assumed population of trajectories.
If you are working at the level of longer trajectories, typically this reduces systemic bias: In RL it reduces bias from initial conditions due to bootstrap process. For RNNs I am less sure what this would mean, but suspect there is some equivalent. Typically, it also increases variance, meaning you may need more training samples. There is a bias/variance trade-off, which is why setting $\lambda$ somewhere between 0 and 1 in TD($\lambda$) is often the optimal choice. Note in RL this is a different source of bias and variance than discussed when considering number of parameters and regularisation in supervised learning.


Can current AI techniques distinguish a fake old paper from a real one? It is an easy matter to make a paper look old, for example, using any of the techniques explained on this page of WikiHow: https://www.wikihow.com/Make-Paper-Look-Old.

Is current AI sufficient to distinguish a fake old paper from a real one?

Is current AI sufficient to distinguish a fake old paper from a real one?

It depends. Specifically, it depends on how much information you (or any automated system that feeds data to the AI), gathers about the paper you want to test, and how good that information is at distinguishing between real and fake old paper.

A model that distinguishes between cases in an input domain (which can be any consistent data, such as images of a subject) is called a classifier. The current state of the art classifiers working with sensory data, if fed enough training data of sufficient quality, can out-perform humans at certain tasks. This is a success of "narrow AI" over humans. This is not some magical ability of machine learning that always applies though, there are some strong caveats:

The task must be feasible given the data collected. Low-resolution photos of samples of the paper might not contain enough information for your example task at all.
When comparing with a human performing the same task, they must be given identical data. If a human would gain more data by interacting with the paper in your example, it can give the human an advantage that a narrow AI cannot easily replicate.
The quality of training data "ground truth" must be higher than achievable by a single human. In your example, this could be achieved simply by knowing the provenance of all the training data.
There often needs to be a large amount of training data to achieve better-than-human ability, although the actual amount required depends heavily on the task.

Expanding on the first point, image classifiers can sometimes be surprisingly good, they have different strengths and weaknesses compared to human vision, so it is possible for them to sometimes be good at making classifications that humans are bad at. However, there is no guarantee of getting that kind of result for any specific task.

Expanding on the second point, if interacting with a subject is allowed in order to classify it, this moves from the domain of "narrow AI" classifiers into active rational agents.

If you gave a piece of paper to a human expert on old paper, and ask them to classify it, they might first look at the paper, then maybe sniff it, touch it . . . still undecided, so get out a magnifying glass to check a couple of features that might be good telltale signs . . . etc.

To build an equivalent artificial agent is harder than a single classifier: It might be possible to purpose-build a collection of narrow AI classifiers, that output a class and confidence levels that decided whether certain tests should be carried out. It might even be possible to give a more general AI the same toolkit plus a very long time to experiment and it could learn for itself which tests to do in which order and how much to rely on the separate results. Or some hybrid of the two approaches. However, any such sophisticated system would still be bound by the limits of individual classifiers.